1990issue C051-2
Three-window walk-forward system evaluation
A historical search that ranks parameter sets by in-sample trading results is treated as incomplete unless the selected rules are later examined on a period that was not used to pick them. Window lengths are design variables, and a later indicator-selection window can choose among already-optimized indicators instead of locking one family from the entire record.
- A search that ranks parameter sets by in-sample trading results is incomplete unless the selected rules are later examined on a period that was not used to pick them.
- The lengths of the optimization window and the evaluation window are design variables, described for interday work as ranging from a few months to several years as market volatility changes.
- When several indicators are available, each can be optimized and compared on later windows, and a further indicator-selection window can choose which already-optimized family is used next.
- Editorial: reliability is a property of the whole walk-forward procedure, not of the single best historical parameter set.
In-sample ranking is incomplete
A historical search that ranks parameter sets by in-sample trading results is treated as incomplete unless the selected rules are later examined on a period that was not used to pick them. Curve fitting is the failure to isolate that later stretch: it judges rules on the same history used to select them, so later unused performance is never isolated.
System optimization is the search step in that workflow. It searches over candidate rule inputs and ranks parameter sets on a designated fit sample before any later unused sample is examined. A parameter set is one complete combination of lookback length and threshold values that defines a single testable version of a rule.
Two windows as design variables
Walk-forward analysis is a rolling test that selects rules on one span of data and then applies those same rules on a later span that was not used to choose them. The optimization window is the historical interval on which candidate parameter sets are ranked. The evaluation window is the later interval on which the selected parameter set is applied without reusing the ranking sample.
The length of the fit window and the length of the later evaluation window are themselves design variables. For interday work those lengths are described as ranging from a few months to several years as market volatility changes.
A relative strength index walk-forward pass
The relative strength index is a price-based oscillator whose lookback length and buy and sell thresholds form a discrete grid that can be searched, then applied forward. One relative strength index search enumerated 16,008 combinations from lookbacks 4 through 60 in steps of 2, sell thresholds 96 through 50 in steps of 2, and buy thresholds 4 through 48 in steps of 2.
In that illustration the winning relative strength index settings from March 2, 1980 through March 1, 1981 were then applied from March 2, 1981 through June 1, 1981. After each holdout the first window rolls by dropping the oldest contract and adding a newer one, the relative strength index search is rerun, and the new winner is applied to the next contract.
A third window for indicator choice
When several indicators are available, each can be optimized and then compared on later windows so that indicator choice is scored out of sample rather than only inside its own historical search. A further window after the parameter-evaluation window can select which already-optimized indicator is used next, so the indicator family is not fixed by inspecting the entire history first.
That further interval is the indicator-selection window: a still later interval used to choose among several already-optimized indicators instead of locking one indicator family in advance. That extra selection stage increases the research load by the number of additional indicators considered and is offered as a way to change rules with market conditions without fitting one specification to the entire record.
All readings on this track · 50 readings
- 1990Three-window walk-forward system evaluation
- 1990Building the construction layer of a mechanical trading system
- 1991Constructing walk-forward neural trading rules
- 1991Constructing neural trading systems from facts to walk-forward
- 1992Walk-forward evaluation of stop overlays on average crossovers
- 1992Audit mechanical system tests for fills and regimes
- 1993Walk-forward evaluation of monthly yield and real-rate forecasts
- 1993Constructing walk-forward forecasts with linear and moving-average baselines
- 1993Walk-forward hybrid rules for intermarket forecast stacks
- 1994Neural-net construction as a mechanical trading-system problem
- 1995Constructing an intermarket neural net trading system
- 1996Weekly market breadth as one procedure on an unused window
- 1996Walk-forward evaluation of gold-index bond-fund rules
- 1996Evaluating weekday-in-month filters for index day trades
- 1996Require both a trend filter and a cycle oscillator before entry
- 1997Walk-forward windows as a diagnostic of parameter instability
- 1997Walk-forward validation of a market-breadth timing rule
- 1997Sunspot spikes and walk-forward evaluation of an adaptive cycle rule
- 1997A walk-forward check for bond-breadth timing
- 1998Walk-forward audit of regression trend forecasts
- 1998Evaluating a cubic least-squares currency trend with walk-forward segments
- 1998Walk-forward evaluation of recursive yen trend signals
- 1999Personal system design under crowd psychology
- 1999Walk-forward evaluation of a polynomial price forecast
- 2000Walk-forward optimization of regression-slope-angle rules
- 2001Construct a winter seasonal window as one procedure
- 2001Inspectable rules when system write-ups dry up
- 2002Evaluating mechanical systems before position sizing
- 2003Walk-forward construction of rule-based market-position systems
- 2007Evaluating metal seasonal windows across regimes
- 2007Evaluating mechanical timing systems against hold baselines
- 2011Walk-forward reoptimization as a system design gate
- 2011Evaluate generated systems on holdouts, then add stops
- 2012Walk-forward analysis and out-of-sample tests for a mechanical trading system
- 2012Personality-first trading system design
- 2012Scorecard-first mechanical system construction
- 2012Constructing an advancer-decliner moving average for market breadth
- 2012Formula search as mechanical system construction
- 2013Identity-first system construction
- 2013Construct a swing system from bias rules to walk-forward
- 2014Evaluate mechanical stock systems with stops and walk-forward
- 2014Walk-forward velocity filters on noisy intraday trends
- 2015Event-predictability versus position-constrained rules
- 2015Constructing mechanical systems for walk-forward tests
- 2016When a tested system must be retired
- 2016Walk-forward metric filters and chance-level checks for selected inputs
- 2018Evaluate mechanical trading systems without catalog rankings
- 2019Phased stop construction from entry risk to trailing exit
- 2020Stockpiling simple ideas for mechanical system construction
- 2020A pretty first draft is not a walk-forward waiver