2020issue C0246-47
Data construction as a mechanical system input
Walk-forward analysis and robustness testing can make a mechanical entry, exit, and abstention procedure testable as one process only after the historic file is built the same way the live signal will see dividends, membership, futures rolls, and venue quotes.
- Split and dividend accounting in the historic file must be known before a mechanical strategy is tested, because the same price path can be assembled in more than one coherent way.
- A test that keeps only the names still listed at the end of an index window can omit losing paths a live membership rule would have taken.
- Futures rollover choices and broker-specific foreign-exchange quotes must be encoded if the tested signal is to match a tradable series.
- Taking a data feed as given, without inspecting how the series was built, undermines later checks of a mechanical system.
The file is a rule input
A mechanical trading system is a fully specified procedure that turns rule inputs, market state, and execution constraints into entry, exit, or abstention signals over the system holding period.
In the historical workflow, taking a data feed as given, without inspecting how the series was built, is a habit that undermines later checks of a mechanical system.
Splits, dividends, and joined contracts
Historic equity series may embed dividend effects or omit them, and either construction is usable only if the tester knows which convention built the file and codes the mechanical rules to match.
Split and dividend accounting in the historic file must be known before a mechanical strategy is tested, because the same price path can be assembled in more than one coherent way. Series construction is the set of conventions used to embed or omit splits, dividends, and contract joins in a price history.
Membership during the window, not after it
A typical year can see 10 to 30 stocks replaced in a broad large-cap equity index. Names are often removed from that index after poor results, including after bankruptcy, so a test that keeps only the names still listed at the end can omit losing paths a live membership rule would have taken.
Survivorship bias is building a test universe from names that remain in an index at the end of the sample, which drops names that left after poor results. A historical test that claims to cover every index member during a window needs the names that belonged during that window, not only the names present afterward.
Omitting departed names can bias a diversified membership backtest toward overly optimistic conclusions about the mechanical rules.
Rolled futures and broker quotes
Futures histories introduce contract-rollover construction choices that a mechanical procedure must encode if the tested signal is to match a tradable series.
Foreign-exchange series can differ by broker, so a mechanical test on quotes that do not match the executing broker's data is not a test of the live input stream. Venue-quote mismatch is testing on a series that does not match the quotes of the broker or venue that will fill live orders.
Later windows still need the same construction
Walk-forward analysis is a sequential out-of-sample check that reruns the same mechanical procedure on later unseen windows so entry, exit, and abstention rules stay testable as one process. Robustness testing stresses that same procedure across alternative universes and data-construction choices to see whether the signal depends on a single historical artifact.
Editorial interpretation: those checks do not repair an unspecified file. They make the full entry, exit, and abstention procedure testable as one process only after the live handling of dividends, membership, rolls, and quotes has already been written into the input.
All readings on this track · 57 readings
- 1986Degrees of freedom in trading system optimization
- 1988Walk-forward and neighborhood tests after optimization
- 1988Undisclosed rules block system robustness tests
- 1988Testing re-optimization calendars against random parameter controls
- 1989Binary search limits on multi-peak average grids
- 1989Parameter neighborhoods that survive a shift
- 1990Use profit mapping to keep a cycle and stop plateau
- 1990Why popular indicator optimization fails robustness
- 1991Retesting weighted indicator balances across horizons
- 1992Constructing forecast models with regression, walk-forward, and robustness
- 1992Diagnose regimes before you lock parameters
- 1992When stops change system timing
- 1993Walk-forward halt rules for forecast models
- 1994Walk-forward evaluation of genetic index rules
- 1995Input pruning as walk-forward system evaluation
- 1995Critiquing neural nets as incomplete trading systems
- 1996Rebuild the equity-path ratio before it ranks a designed system
- 1996Parameter grids can fit random walks
- 1996Walk-forward analysis belongs in the design of a mechanical trading system
- 1997When a holdout fails, discard the rule set
- 1997Test rewarded rule breaks before replacing the system
- 1997Walk-forward rules keep system research from rewriting live trades
- 1999Keep a channel-breakout to two lookbacks and test neighbor stability
- 1999Constant investment size in stock system evaluation
- 2000Forcing optimization maps mechanical system failure boundaries
- 2000Robust parameter selection with surface charts
- 2001A two-gate classroom test for a two-window momentum trend filter
- 2002How a two-sided continuation factor becomes a testable trend rule
- 2002Evaluating two-window trend intensity as a reversal rule
- 2003Discounting speculative bubbles in system robustness tests
- 2003Walk-forward evaluation of locked stochastic oscillator rules
- 2003Critiquing mechanical system design after extreme price regimes
- 2004Evaluating a two-window trend trigger
- 2005Grade backtested signals with holdouts and optimization plateaus
- 2006Reserved-sample evaluation of trading system design
- 2006Walk-forward critique of hindsight crossover systems
- 2008Condition-matched walk-forward evaluation for mechanical systems
- 2011Session-split evaluation of regular and overnight systems
- 2012Walk-forward evaluation as operator rehearsal
- 2013Two-window evaluation of mechanical trading systems
- 2013Walk-forward filter selection for repeated-median velocity
- 2014Walk-forward evaluation for fading-memory velocity systems
- 2015Test oscillator events before tuning rules
- 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
- 2016Walk-forward optimization without curve fitting
- 2017Optimization without overfitting in trend-system evaluation
- 2017Parameter stability is a better guide than a larger crossover grid
- 2018Point-in-time universes for system evaluation
- 2018Walk-forward robustness evaluation for optimized systems
- 2018Critiquing breakout systems through robustness tests
- 2018A critique of parameter fitting in system design
- 2019Noise-matched rules still need trend filters and robustness tests
- 2019Three gates for evaluating a trading system
- 2020Data construction as a mechanical system input
- 2020Hidden optimization in ported relative-strength systems
- 2020When mechanical historical tests decay after optimization
- 2025Add a second procedure before you retune the first