Parameter grids can fit random walks

What the in-sample scorecard reports

Cheap historical data and desktop search tools made it routine to vary every rule input across a large grid and then keep the single in-sample set that looked best. That workflow is system-optimization: varying every rule input across a historical window and keeping the single set that maximizes a chosen in-sample score.

That search only shows a close fit to one preselected window. It does not show that the rules captured lasting price dynamics.

A polished ledger on a random-price-path

The demonstration used a relative-strength threshold rule whose inputs were length, a buy zone, and a sell zone. The search enumerated those inputs and skipped cases where the buy zone exceeded the sell zone. It retained one combination as the set that maximized the chosen in-sample scores on the fitted sample.

The tested path was an artificial random-price-path, a manufactured series with no recoverable market structure. Each next print was the prior print adjusted by a random percentage held inside a bound, continued across a long run of prints, with only a later date slice shown in the test.

A random series has no repeatable history to recover, so an attractive fitted scorecard on that series cannot establish that the same rules have later value. The result is curve-fitting: forcing a rule set to match one fixed window so closely that the ledger describes the fit rather than a repeatable process.

Freeze the hunt before scoring unseen prints

A basic robustness-testing step is walk-forward-analysis: freezing those chosen rules and scoring the same entry, exit, and abstention logic on a reserved window that never entered the search. That reserved slice is a holdout-window: price data withheld from the parameter hunt so the locked rules can be checked once on unseen prints.

Robustness-testing checks whether a fitted procedure still behaves when the path, the window, or the assumed participant behavior is no longer the one that was tuned. A clean holdout still does not guarantee later behavior, because future dynamics can differ and the reserved window may, by chance, resemble the fitted window.

Hypothetical in-sample ledgers do not authorize an assumption of later profit. Continued ability has to be examined on unseen data, and even a pass only improves probability.

Flattering episodes are the same error

Picking a few flattering chart episodes is a narrower form of the same fitting error, because it withholds the remaining signals the identical rules would have issued. That selection is anecdotal-selection: showing only a few flattering signals while hiding the rest of the trades the same rules would have issued.