Grade backtested signals with holdouts and optimization plateaus

What must agree before a result is a signal

Editorial framing: treat this archive material as a three-gate classroom audit of system design. A backtested result is treated as a signal only when a walk-forward holdout, a short statistically meaningful input list, and a wide profitable plateau on the optimization curve all support the same entry, exit, and abstention procedure.

System optimization searches rule inputs, market state, and execution constraints as one backtested procedure whose output is a signal over the system holding period. Robustness testing checks whether that same procedure still holds when inputs stay few, statistically meaningful, and profitable across a wide value range. Walk-forward analysis withholds unused historical segments from the optimization search so the signal can be judged on market conditions that did not select its inputs.

Hold unused history out of the search

System optimization is described as searching for effective input values. Unused historical segments are to be held out of that search so the finished rules can be checked on market conditions that did not choose the inputs.

Walk-forward analysis is presented as reserving unused history, including leaving out the most recent year, while optimizing earlier data. The archive presents this check as serving the same function as observing a completed system for a year before implementation.

Keep the input list short and meaningful

A low input count is identified as a main defense against curve-fitting, because adding many variables makes it easier to overfit even a random price series.

Robustness testing is framed as preferring statistically significant, predictive market drivers over stacked smoothed price indicators, which are described as easy to fit to a single series.

Non-price candidates listed for investigation include volatility, interest rates, volume, intermarket and sector links, gaps, bar relationships, breadth, up and down volume, tape measures, put/call ratios, and cash-versus-futures relationships.

Read the optimization curve for a wide plateau

An optimization curve is a results plot across a range of input values, used to see whether profitability is isolated in spikes or persists over a broad band. Isolated spikes of profitability on that curve are treated as evidence that an input lacks predictive ability, and selecting one of those spikes is labeled curve-fitting.

A robust input is described as one whose results stay profitable across a wide band of reasonable values, with the stated design goal that the optimization curve remain above the zero line.

Reasonable input values are those that produce a commonsense trade count for the study horizon and average net per trade. That band is the reasonable input range. Overtrading is associated with slippage, and undertrading with too few observations to be statistically meaningful.

One procedure has to pass every gate

A sports-rule metaphor shows robustness shrinking as extra conditions are added. A filter that wins only when jersey numbers sum to 9, 21, or 42 is treated as no more predictive than a 9-, 21-, or 42-bar moving average.

Editorial classroom rule: the walk-forward holdout, the short input list, and the wide profitable plateau must all support the same entry, exit, and abstention procedure. If one gate fails, the backtested result is not treated as a signal.