1997issue C011-2
When a holdout fails, discard the rule set
If a fitted rule set fails unused-sample testing, the formulation should be abandoned rather than retuned until the holdout looks acceptable. Unequal buy and sell search ranges and lookbacks were treated as a lopsided specification and a logical error in system design.
- If a rule set and its fitted inputs fail unused-sample testing, abandon the formulation instead of adjusting it until it passes.
- Repeating parameter changes after a holdout failure until the unused sample looks acceptable folds that sample back into the optimization.
- Unequal buy and sell search ranges or lookbacks are a lopsided specification and were treated as a logical error in system design.
- A passing unused-sample test still does not guarantee later results, because chance or a poorly chosen holdout window can produce a pass.
A system is a complete procedure
A published reply framed a trading system as a complete set of rules and procedures that can be checked before live outcomes supply the last test.
System-optimization is a search over rule inputs under stated market-state and execution constraints to produce one testable holding-period procedure. Walk-forward-analysis optimizes that complete signal procedure on one window, then scores the same entry, exit and abstention rules on a later unused window.
A failed unused sample ends the formulation
A published reply said that if a rule set and its fitted inputs fail unused-sample testing, the formulation should be abandoned rather than adjusted until it passes.
Another letter described repeating parameter changes after a holdout failure until the unused sample also looks acceptable, which folds that sample back into the optimization. That practice is holdout-retuning: changing parameters after a failed unused-sample test until that unused sample also looks acceptable.
Unequal buy and sell rules are a specification error
A letter treated optimization of buy-side and sell-side zones over unequal search ranges as a logical error in system design. The same letter used unequal buy and sell lookbacks as an example of lopsided rule specification.
Asymmetrical-rules are a design in which buy conditions and sell conditions use different thresholds, lookbacks or search ranges. Robustness-testing checks whether a fitted procedure still holds when the sample, search ranges or buy-versus-sell specification change.
A passing holdout does not guarantee later results
The reply stated that a passing unused-sample test still does not guarantee later results, because chance or a poorly chosen holdout window can produce a pass.
The reply distinguished curve-fitted hypothetical illustrations from the claim that carefully applied optimization procedures cannot work.
All readings on this track · 51 readings
- 1986Degrees of freedom in trading system optimization
- 1988Walk-forward and neighborhood tests after optimization
- 1988Undisclosed rules block system robustness tests
- 1988Testing re-optimization calendars against random parameter controls
- 1989Binary search limits on multi-peak average grids
- 1989Parameter neighborhoods that survive a shift
- 1990Use profit mapping to keep a cycle and stop plateau
- 1990Why popular indicator optimization fails robustness
- 1991Retesting weighted indicator balances across horizons
- 1992Constructing forecast models with regression, walk-forward, and robustness
- 1992Diagnose regimes before you lock parameters
- 1992When stops change system timing
- 1993Walk-forward halt rules for forecast models
- 1994Walk-forward evaluation of genetic index rules
- 1995Input pruning as walk-forward system evaluation
- 1995Critiquing neural nets as incomplete trading systems
- 1996Rebuild the equity-path ratio before it ranks a designed system
- 1996Parameter grids can fit random walks
- 1996Walk-forward analysis belongs in the design of a mechanical trading system
- 1997When a holdout fails, discard the rule set
- 1997Test rewarded rule breaks before replacing the system
- 1997Walk-forward rules keep system research from rewriting live trades
- 1999Keep a channel-breakout to two lookbacks and test neighbor stability
- 1999Constant investment size in stock system evaluation
- 2000Forcing optimization maps mechanical system failure boundaries
- 2000Robust parameter selection with surface charts
- 2001A two-gate classroom test for a two-window momentum trend filter
- 2002How a two-sided continuation factor becomes a testable trend rule
- 2002Evaluating two-window trend intensity as a reversal rule
- 2003Discounting speculative bubbles in system robustness tests
- 2003Walk-forward evaluation of locked stochastic oscillator rules
- 2003Critiquing mechanical system design after extreme price regimes
- 2004Evaluating a two-window trend trigger
- 2005Grade backtested signals with holdouts and optimization plateaus
- 2006Reserved-sample evaluation of trading system design
- 2006Walk-forward critique of hindsight crossover systems
- 2008Condition-matched walk-forward evaluation for mechanical systems
- 2011Session-split evaluation of regular and overnight systems
- 2012Walk-forward evaluation as operator rehearsal
- 2013Two-window evaluation of mechanical trading systems
- 2013Walk-forward filter selection for repeated-median velocity
- 2014Walk-forward evaluation for fading-memory velocity systems
- 2015Test oscillator events before tuning rules
- 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
- 2016Walk-forward optimization without curve fitting
- 2017Optimization without overfitting in trend-system evaluation
- 2017Parameter stability is a better guide than a larger crossover grid
- 2018Point-in-time universes for system evaluation
- 2018Walk-forward robustness evaluation for optimized systems
- 2018Critiquing breakout systems through robustness tests
- 2018A critique of parameter fitting in system design