2006issue C131-5
Walk-forward critique of hindsight crossover systems
A common design error is to certify entry and exit rules on the same historical series used to choose them. A Moving-average crossover found in hindsight is not a certificate that the procedure will persist on unseen data.
- A common design error is to certify entry and exit rules after those rules were chosen in hindsight on the same historical series later treated as proof.
- Searching a historical series for a short and long moving-average pair that would have triggered when the shorter average crossed above the longer one for a long, or the reverse for a short, is presented as a typical hindsight example.
- Rules that appear to have worked in hindsight are not guaranteed to persist on unseen future data, and a reliable repeat of past market behavior is treated as a weak basis for certifying a system.
- Even a system described as good can produce a consecutive string of four or five losses, so capital planning must assume losing streaks rather than uninterrupted repeats of a backtest.
The hindsight certification error
A common design error is to certify entry and exit rules after those rules were chosen in hindsight on the same historical series later treated as proof.
Searching a historical series for a short and long moving-average pair that would have triggered when the shorter average crossed above the longer one for a long, or the reverse for a short, is presented as a typical hindsight example.
Rules that appear to have worked in hindsight are not guaranteed to persist when applied to unseen future data.
Fitted history is a weak certificate
If later markets do not resemble the fitted history, a backtested procedure can fail even when commissions are paid to execute it.
A reliable repeat of past market behavior is treated as virtually impossible and therefore a weak basis for certifying a system.
Walk-forward analysis and robustness testing
Editorial view: Walk-forward analysis is how a finished set of entry, exit, and abstention rules is kept testable as one procedure, using rule inputs, market state, and execution constraints over the system holding period. That task is not a search for a prettier in-sample Moving-average crossover.
A named evaluator tool is described as a way to judge whether a trading procedure can survive its own loss sequence, which is a Robustness testing question rather than a search for prettier in-sample crossovers.
Capital planning for losing streaks
Even a system described as good can produce a consecutive string of four or five losses, so capital planning must assume losing streaks rather than uninterrupted repeats of a backtest.
A planned derivational market analysis system
A planned system labeled derivational market analysis is described as combining a chosen dependent market with positively and negatively correlated independent markets plus multi-year commercial positioning statistics.
All readings on this track · 51 readings
- 1986Degrees of freedom in trading system optimization
- 1988Walk-forward and neighborhood tests after optimization
- 1988Undisclosed rules block system robustness tests
- 1988Testing re-optimization calendars against random parameter controls
- 1989Binary search limits on multi-peak average grids
- 1989Parameter neighborhoods that survive a shift
- 1990Use profit mapping to keep a cycle and stop plateau
- 1990Why popular indicator optimization fails robustness
- 1991Retesting weighted indicator balances across horizons
- 1992Constructing forecast models with regression, walk-forward, and robustness
- 1992Diagnose regimes before you lock parameters
- 1992When stops change system timing
- 1993Walk-forward halt rules for forecast models
- 1994Walk-forward evaluation of genetic index rules
- 1995Input pruning as walk-forward system evaluation
- 1995Critiquing neural nets as incomplete trading systems
- 1996Rebuild the equity-path ratio before it ranks a designed system
- 1996Parameter grids can fit random walks
- 1996Walk-forward analysis belongs in the design of a mechanical trading system
- 1997When a holdout fails, discard the rule set
- 1997Test rewarded rule breaks before replacing the system
- 1997Walk-forward rules keep system research from rewriting live trades
- 1999Keep a channel-breakout to two lookbacks and test neighbor stability
- 1999Constant investment size in stock system evaluation
- 2000Forcing optimization maps mechanical system failure boundaries
- 2000Robust parameter selection with surface charts
- 2001A two-gate classroom test for a two-window momentum trend filter
- 2002How a two-sided continuation factor becomes a testable trend rule
- 2002Evaluating two-window trend intensity as a reversal rule
- 2003Discounting speculative bubbles in system robustness tests
- 2003Walk-forward evaluation of locked stochastic oscillator rules
- 2003Critiquing mechanical system design after extreme price regimes
- 2004Evaluating a two-window trend trigger
- 2005Grade backtested signals with holdouts and optimization plateaus
- 2006Reserved-sample evaluation of trading system design
- 2006Walk-forward critique of hindsight crossover systems
- 2008Condition-matched walk-forward evaluation for mechanical systems
- 2011Session-split evaluation of regular and overnight systems
- 2012Walk-forward evaluation as operator rehearsal
- 2013Two-window evaluation of mechanical trading systems
- 2013Walk-forward filter selection for repeated-median velocity
- 2014Walk-forward evaluation for fading-memory velocity systems
- 2015Test oscillator events before tuning rules
- 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
- 2016Walk-forward optimization without curve fitting
- 2017Optimization without overfitting in trend-system evaluation
- 2017Parameter stability is a better guide than a larger crossover grid
- 2018Point-in-time universes for system evaluation
- 2018Walk-forward robustness evaluation for optimized systems
- 2018Critiquing breakout systems through robustness tests
- 2018A critique of parameter fitting in system design