2020issue C0345
A pretty first draft is not a walk-forward waiver
A repeatable development sequence is useful because changing how candidates are evaluated makes later results hard to compare. Process consistency is still not enough unless the same procedure can be shown to produce acceptable real-time results. Editorial view: a clean first-pass historical path is a reason to inspect walk-forward analysis and system optimization, not a reason to disable those gates.
- A repeatable development sequence is useful because changing how candidates are evaluated makes later results hard to compare.
- Process consistency is not enough; the same procedure must also be shown to produce acceptable real-time results.
- A first-pass mechanical candidate that skips the usual walk-forward and explicit-tuning steps still has no comparison except its own historical test.
- After more than one candidate has been tested and the better one kept, that selection itself counts as system optimization even if no parameter was tuned.
A mechanical system is one procedure
A mechanical trading system is a complete set of entry, exit, and abstention rules meant to be applied as one procedure over the holding period.
Walk-forward analysis is a sequential check that estimates rules on earlier data and evaluates the same procedure on later unseen periods. System optimization is any search that prefers one rule set over another, including informal selection among candidates.
Editorial view: treat mechanical-system design as factory quality control. A clean first-pass historical path is incoming inspection, not a finished release.
Process consistency is not a live result
Process consistency means using the same development sequence for every candidate so later live results can be compared.
A repeatable development sequence is useful because changing how candidates are evaluated makes later results hard to compare.
Process consistency is not enough. The same procedure must also be shown to produce acceptable real-time results.
A skipped gate leaves only one comparison
A first-pass mechanical candidate that skips the usual walk-forward and explicit-tuning steps still has no comparison except its own historical test.
Keeping the better candidate is still optimization
Hidden optimization is bias created by testing more than one candidate and keeping the more attractive result.
After more than one candidate has been tested and the better one kept, that selection itself counts as system optimization even if no parameter was tuned.
Irregular paths need a larger set or live time
Judging a nonstandard development path statistically would require a large set of similarly produced strategies, on the order of 50 to 100.
One alternative to treating an irregular candidate as confirmed is to observe it in real time for three to six months or longer.
All readings on this track · 50 readings
- 1990Three-window walk-forward system evaluation
- 1990Building the construction layer of a mechanical trading system
- 1991Constructing walk-forward neural trading rules
- 1991Constructing neural trading systems from facts to walk-forward
- 1992Walk-forward evaluation of stop overlays on average crossovers
- 1992Audit mechanical system tests for fills and regimes
- 1993Walk-forward evaluation of monthly yield and real-rate forecasts
- 1993Constructing walk-forward forecasts with linear and moving-average baselines
- 1993Walk-forward hybrid rules for intermarket forecast stacks
- 1994Neural-net construction as a mechanical trading-system problem
- 1995Constructing an intermarket neural net trading system
- 1996Weekly market breadth as one procedure on an unused window
- 1996Walk-forward evaluation of gold-index bond-fund rules
- 1996Evaluating weekday-in-month filters for index day trades
- 1996Require both a trend filter and a cycle oscillator before entry
- 1997Walk-forward windows as a diagnostic of parameter instability
- 1997Walk-forward validation of a market-breadth timing rule
- 1997Sunspot spikes and walk-forward evaluation of an adaptive cycle rule
- 1997A walk-forward check for bond-breadth timing
- 1998Walk-forward audit of regression trend forecasts
- 1998Evaluating a cubic least-squares currency trend with walk-forward segments
- 1998Walk-forward evaluation of recursive yen trend signals
- 1999Personal system design under crowd psychology
- 1999Walk-forward evaluation of a polynomial price forecast
- 2000Walk-forward optimization of regression-slope-angle rules
- 2001Construct a winter seasonal window as one procedure
- 2001Inspectable rules when system write-ups dry up
- 2002Evaluating mechanical systems before position sizing
- 2003Walk-forward construction of rule-based market-position systems
- 2007Evaluating metal seasonal windows across regimes
- 2007Evaluating mechanical timing systems against hold baselines
- 2011Walk-forward reoptimization as a system design gate
- 2011Evaluate generated systems on holdouts, then add stops
- 2012Walk-forward analysis and out-of-sample tests for a mechanical trading system
- 2012Personality-first trading system design
- 2012Scorecard-first mechanical system construction
- 2012Constructing an advancer-decliner moving average for market breadth
- 2012Formula search as mechanical system construction
- 2013Identity-first system construction
- 2013Construct a swing system from bias rules to walk-forward
- 2014Evaluate mechanical stock systems with stops and walk-forward
- 2014Walk-forward velocity filters on noisy intraday trends
- 2015Event-predictability versus position-constrained rules
- 2015Constructing mechanical systems for walk-forward tests
- 2016When a tested system must be retired
- 2016Walk-forward metric filters and chance-level checks for selected inputs
- 2018Evaluate mechanical trading systems without catalog rankings
- 2019Phased stop construction from entry risk to trailing exit
- 2020Stockpiling simple ideas for mechanical system construction
- 2020A pretty first draft is not a walk-forward waiver