Skip to main content
Track Walk-forward analysis
34 / 50
Search the archiveLibrary
Library

2012issue C0112-17

Walk-forward analysis and out-of-sample tests for a mechanical trading system

Historical evaluation reserved unseen slices at the start and at the end of a dataset before treating a mechanical trading system as ready. Walk-forward analysis and those out-of-sample checks were still treated as unable to guarantee live-trading results.

  • A mechanical trading system was treated as unfinished until it was forced to trade periods withheld from development, first at the start of the sample and then at the end.
  • One design reserved the first 15 percent of a dataset and built the system on the remaining 85 percent; another developed on the first 80 percent and reserved the final 20 percent.
  • Larger trade counts in these withheld-data tests were treated as making conclusions about whether a mechanical trading system is tradable more trustworthy.
  • Walk-forward analysis and pre- or post-out-of-sample testing were treated as unable to guarantee that a system will make money in live trading.
Entries in this reading3 entries
System optimization85 readingsCurrent trackWalk-forward analysis95 readingsMechanical trading system158 readings

A check at the start of the sample

One evaluation design reserved the first 15 percent of a dataset for out-of-sample testing and built the mechanical trading system on the remaining 85 percent as if that later slice were a new problem.

Editorial: Keep System optimization inside that later 85 percent. The first 15 percent is the period the rules are not allowed to see while they are being built.

A check at the end of the sample

A second evaluation design limited development to the first 80 percent of the original dataset and reserved the final 20 percent as previously unseen out-of-sample data.

Editorial: The same limit applies in reverse. System optimization may use only the earlier window. The final 20 percent is held back until the mechanical trading system is already specified.

What the withheld slices showed

In the first-15-percent out-of-sample check, an intraday emini Russell futures system produced hypothetical gains on data withheld from development. The accompanying equity path was described as having a low drawdown and no wild swings.

In the final-20-percent out-of-sample check, a 10-year emini Russell swing-trade system produced hypothetical gains on data it had not been developed with and had not previously encountered.

Larger trade counts in these withheld-data tests were treated as making conclusions about whether a mechanical trading system is tradable more trustworthy.

Editorial: Treat Walk-forward analysis as another withheld-data check in the same sequence, not as a separate kind of proof.

Closed-trade equity after holding back the last fifth of the sample

A trader should see a long, mostly orderly rise in closed-trade equity on this emini Russell swing system through the first 320 trades, then a blank band where the final 20 percent of the 10-year file was withheld. Dollar levels were read from the published plot against the printed scale running from -$20,000 to $120,000 versus trade number.
A trader should see a long, mostly orderly rise in closed-trade equity on this emini Russell swing system through the first 320 trades, then a blank band where the final 20 percent of the 10-year file was withheld. Dollar levels were read from the published plot against the printed scale running from -$20,000 to $120,000 versus trade number.E-mini Russell futures · Swing trades · 2001-10-31T00:00:00.000Z to 2011-11-01T00:00:00.000Z

Development used only the first 80 percent of 31 October 2001–1 November 2011. The extracted raster does not resolve a line inside the reserved band, so those trades are omitted rather than guessed.

Educational research material, not investment advice. Historical source context does not establish present-day performance.
34 of 50 in the Walk-forward analysis track
201250-54 pp.Next on Walk-forward analysisPersonality-first trading system designAssess how you think, learn, and work before you adopt tactics, or the plan may not match your personality or available attention.
All readings on this track · 50 readings
  1. 1990Three-window walk-forward system evaluation
  2. 1990Building the construction layer of a mechanical trading system
  3. 1991Constructing walk-forward neural trading rules
  4. 1991Constructing neural trading systems from facts to walk-forward
  5. 1992Walk-forward evaluation of stop overlays on average crossovers
  6. 1992Audit mechanical system tests for fills and regimes
  7. 1993Walk-forward evaluation of monthly yield and real-rate forecasts
  8. 1993Constructing walk-forward forecasts with linear and moving-average baselines
  9. 1993Walk-forward hybrid rules for intermarket forecast stacks
  10. 1994Neural-net construction as a mechanical trading-system problem
  11. 1995Constructing an intermarket neural net trading system
  12. 1996Weekly market breadth as one procedure on an unused window
  13. 1996Walk-forward evaluation of gold-index bond-fund rules
  14. 1996Evaluating weekday-in-month filters for index day trades
  15. 1996Require both a trend filter and a cycle oscillator before entry
  16. 1997Walk-forward windows as a diagnostic of parameter instability
  17. 1997Walk-forward validation of a market-breadth timing rule
  18. 1997Sunspot spikes and walk-forward evaluation of an adaptive cycle rule
  19. 1997A walk-forward check for bond-breadth timing
  20. 1998Walk-forward audit of regression trend forecasts
  21. 1998Evaluating a cubic least-squares currency trend with walk-forward segments
  22. 1998Walk-forward evaluation of recursive yen trend signals
  23. 1999Personal system design under crowd psychology
  24. 1999Walk-forward evaluation of a polynomial price forecast
  25. 2000Walk-forward optimization of regression-slope-angle rules
  26. 2001Construct a winter seasonal window as one procedure
  27. 2001Inspectable rules when system write-ups dry up
  28. 2002Evaluating mechanical systems before position sizing
  29. 2003Walk-forward construction of rule-based market-position systems
  30. 2007Evaluating metal seasonal windows across regimes
  31. 2007Evaluating mechanical timing systems against hold baselines
  32. 2011Walk-forward reoptimization as a system design gate
  33. 2011Evaluate generated systems on holdouts, then add stops
  34. 2012Walk-forward analysis and out-of-sample tests for a mechanical trading system
  35. 2012Personality-first trading system design
  36. 2012Scorecard-first mechanical system construction
  37. 2012Constructing an advancer-decliner moving average for market breadth
  38. 2012Formula search as mechanical system construction
  39. 2013Identity-first system construction
  40. 2013Construct a swing system from bias rules to walk-forward
  41. 2014Evaluate mechanical stock systems with stops and walk-forward
  42. 2014Walk-forward velocity filters on noisy intraday trends
  43. 2015Event-predictability versus position-constrained rules
  44. 2015Constructing mechanical systems for walk-forward tests
  45. 2016When a tested system must be retired
  46. 2016Walk-forward metric filters and chance-level checks for selected inputs
  47. 2018Evaluate mechanical trading systems without catalog rankings
  48. 2019Phased stop construction from entry risk to trailing exit
  49. 2020Stockpiling simple ideas for mechanical system construction
  50. 2020A pretty first draft is not a walk-forward waiver
By fieldSystem design117 readingsBy year201281 readings
All 95 readings tagged Walk-forward analysis
Also on Walk-forward analysis5 readings
1990Building the construction layer of a mechanical trading systemSystem optimization1991Constructing neural trading systems from facts to walk-forwardSystem optimization1991Constructing walk-forward neural trading rulesSystem optimization1994Neural-net construction as a mechanical trading-system problemSystem optimization1995Constructing an intermarket neural net trading systemSystem optimization