Walk-forward and neighborhood tests after optimization

Optimization proposes a joint procedure

System-optimization is a search over rule inputs that picks the historically strongest joint entry, exit, and abstention procedure. A search for the highest historical profit can still report a profitable combination when the series is random or the indicator has no skill. The in-sample peak does not by itself show that the procedure will remain reliable.

A mechanical procedure assumes both a recurring price pattern and an indicator that can exploit it. If the pattern is too infrequent, or the indicator also fires on similar but unwanted setups, losing trades can outnumber winning ones.

Score the same rules on a later unused window

Walk-forward-analysis fits the procedure on one window, then scores the same rules on a later window that was not used in fitting. The later results are not produced by hindsight.

The method needs enough history to both fit and retest. When possible the illustrated design used at least five to 10 years. A shorter fit window can be chosen so that more years remain for unused-window tests.

An illustrated rolling and expanding design

One illustrated walk-forward design on T-bond futures used 40 expired contracts over 10 years. It traded the last three months of each contract, optimized on 12 contracts covering three years, tested the next contract, then rolled that window through contract 40, producing a seven-year unused-window record on 28 contracts.

An expanding-window variant adds each new contract to the optimization sample instead of dropping the oldest. That variant can suit longer-horizon systems that need more history in every fit.

Second-order-optimization can still fit chance

Repeatedly trying procedures until one looks good on the unused window is second-order-optimization and can still fit chance. The expanding design also re-optimizes early contracts many times, up to 28 times for contracts 1 through 12.

Judge nearby profits, not only the peak

Robustness-testing asks whether profits stay orderly across nearby parameter values instead of appearing only at the single best fit.

A profit-distribution-chart plots profit against one varied parameter. A smooth rise and fall around a peak is treated as a sign that the parameter influences results, while sharp irregular spikes are treated as a sign of little consistent effect. The chart can change only one parameter at a time, so it isolates inputs better than it certifies a multi-parameter system. Reading the shape is a judgment used mainly to reject obviously unstable settings.

Cluster-spotting inspects results around the best system-optimization point. A peak whose neighboring parameter sets show similar profits is treated as a stronger robustness-testing signal than an isolated spike.