Skip to main content
Track Robustness testing
49 / 51
Search the archiveLibrary
Library

2018issue C0444-45

Walk-forward robustness evaluation for optimized systems

A combined indicator and neural-network system is evaluated as one locked procedure: choose which rule inputs may be searched, reserve later data the search cannot see, and keep the rules only if they still emit coherent signals after costs and competing objectives.

  • Choose which rule inputs may be searched, reserve later data the search cannot see, and keep the rules only if they still emit coherent signals after costs and competing objectives.
  • Each parameter can be left fixed or included in the search, and turning parameters out of the search or tightening their ranges is presented as a control against curve-fitting.
  • Walk-forward analysis locks parameters after an earlier sample, then inspects them on a paper-trading window and later unused data before a further forward stretch.
  • Robustness testing checks whether locked rules still behave acceptably when costs, scored objectives, or unused data change.
Entries in this reading3 entries
Current trackRobustness testing58 readingsSystem optimization85 readingsWalk-forward analysis95 readings

A single system under test

Conventional indicators and neural-network models can be combined into one trading system that is then backtested and optimized.

System optimization is a controlled search over selected rule parameters, with other inputs held fixed and search bounds set in advance. Each parameter can be left fixed or included in the search, and the allowed range for a searched parameter can be chosen or changed.

Turning selected parameters out of the search or tightening their ranges is presented as a control against curve-fitting and over-optimization. Curve-fitting is fitting rules so closely to historical data that later unused data no longer support the same behavior.

Unused later data

A system can be backtested and optimized, then examined on later unused data, including using a paper-trading interval to select parameters for a further forward stretch. Walk-forward analysis locks parameters after an earlier sample and then generates signals on later unused data. A paper-trading window is a reserved interval after optimization used to inspect locked rules before a later forward stretch.

One illustrated evaluation used eight years of optimization data, two years of paper-trading data, and two subsequent years of later data. A system can be placed on a price chart, optimized, inspected through an equity-curve indicator, and then reviewed on a later interval by extending the chart. An out-of-sample check reviews signals and equity after the tested range is extended past the optimization window.

Walk-forward windows reserved on the AAPL test

The review locks eight years for the optimizer, then holds out two years of paper trading and two years of live trading that the search is not allowed to see. Those lengths are stated in the article and shown on the strategy Dates tab. A trader should treat only the last four years as an honest check that the locked entry and exit rules still fire after costs.
The review locks eight years for the optimizer, then holds out two years of paper trading and two years of live trading that the search is not allowed to see. Those lengths are stated in the article and shown on the strategy Dates tab. A trader should treat only the last four years as an honest check that the locked entry and exit rules still fire after costs.AAPL · Daily · 2006-01-01T00:00:00.000Z to 2018-04-15T00:00:00.000Z

The tester kept the parameter set that did best on the later paper-trading slice, then applied it through the final trading window. The sample starts in 2006; the last chart date shown is 15 April 2018.

Costs and competing objectives

Backtests can accept explicit inputs for order-size costs, pyramiding, brokerage fees, and slippage. The search can be scored on different objectives, including the gap between winning and losing trades, the largest drawdown, account return, and account return multiplied by equity-curve correlation.

An optimization objective is the scored goal that ranks candidate parameter sets during a search. Robustness testing checks whether locked rules still behave acceptably when costs, scored objectives, or unused data change.

Inputs and markets

The same evaluation workflow can use built-in indicators, custom indicators, or values returned from an external compiled library, and can be applied across stocks, foreign exchange, commodities, indexes, and options.

Educational research material, not investment advice. Historical source context does not establish present-day performance.
49 of 51 in the Robustness testing track
201832-35 pp.Next on Robustness testingCritiquing breakout systems through robustness testsSystem work is said to begin by combining incomplete ideas, conditions, risk techniques, and indicators, then rejecting most of them through robustness-testing.
All readings on this track · 51 readings
  1. 1986Degrees of freedom in trading system optimization
  2. 1988Walk-forward and neighborhood tests after optimization
  3. 1988Undisclosed rules block system robustness tests
  4. 1988Testing re-optimization calendars against random parameter controls
  5. 1989Binary search limits on multi-peak average grids
  6. 1989Parameter neighborhoods that survive a shift
  7. 1990Use profit mapping to keep a cycle and stop plateau
  8. 1990Why popular indicator optimization fails robustness
  9. 1991Retesting weighted indicator balances across horizons
  10. 1992Constructing forecast models with regression, walk-forward, and robustness
  11. 1992Diagnose regimes before you lock parameters
  12. 1992When stops change system timing
  13. 1993Walk-forward halt rules for forecast models
  14. 1994Walk-forward evaluation of genetic index rules
  15. 1995Input pruning as walk-forward system evaluation
  16. 1995Critiquing neural nets as incomplete trading systems
  17. 1996Rebuild the equity-path ratio before it ranks a designed system
  18. 1996Parameter grids can fit random walks
  19. 1996Walk-forward analysis belongs in the design of a mechanical trading system
  20. 1997When a holdout fails, discard the rule set
  21. 1997Test rewarded rule breaks before replacing the system
  22. 1997Walk-forward rules keep system research from rewriting live trades
  23. 1999Keep a channel-breakout to two lookbacks and test neighbor stability
  24. 1999Constant investment size in stock system evaluation
  25. 2000Forcing optimization maps mechanical system failure boundaries
  26. 2000Robust parameter selection with surface charts
  27. 2001A two-gate classroom test for a two-window momentum trend filter
  28. 2002How a two-sided continuation factor becomes a testable trend rule
  29. 2002Evaluating two-window trend intensity as a reversal rule
  30. 2003Discounting speculative bubbles in system robustness tests
  31. 2003Walk-forward evaluation of locked stochastic oscillator rules
  32. 2003Critiquing mechanical system design after extreme price regimes
  33. 2004Evaluating a two-window trend trigger
  34. 2005Grade backtested signals with holdouts and optimization plateaus
  35. 2006Reserved-sample evaluation of trading system design
  36. 2006Walk-forward critique of hindsight crossover systems
  37. 2008Condition-matched walk-forward evaluation for mechanical systems
  38. 2011Session-split evaluation of regular and overnight systems
  39. 2012Walk-forward evaluation as operator rehearsal
  40. 2013Two-window evaluation of mechanical trading systems
  41. 2013Walk-forward filter selection for repeated-median velocity
  42. 2014Walk-forward evaluation for fading-memory velocity systems
  43. 2015Test oscillator events before tuning rules
  44. 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
  45. 2016Walk-forward optimization without curve fitting
  46. 2017Optimization without overfitting in trend-system evaluation
  47. 2017Parameter stability is a better guide than a larger crossover grid
  48. 2018Point-in-time universes for system evaluation
  49. 2018Walk-forward robustness evaluation for optimized systems
  50. 2018Critiquing breakout systems through robustness tests
  51. 2018A critique of parameter fitting in system design
By fieldSystem design117 readingsBy year201898 readings
All 58 readings tagged Robustness testing
Also on Robustness testing5 readings
1986Degrees of freedom in trading system optimization1988Testing re-optimization calendars against random parameter controls1988Walk-forward and neighborhood tests after optimization1992Diagnose regimes before you lock parameters1993Walk-forward halt rules for forecast models