Skip to main content
Track Walk-forward analysis
46 / 50
Search the archiveLibrary
Library

2016issue C0513-16

Walk-forward metric filters and chance-level checks for selected inputs

A walk-forward search can apply an in-sample metric filter to choose one input row, then test the associated rules on the matching out-of-sample window. Chance-level mirrors that draw a random row in each weekly file give a reference distribution for the filter’s total out-of-sample net profit.

  • Walk-forward analysis can search many input combinations on in-sample data, then apply a metric filter to pick one input row whose rules are evaluated on the matching out-of-sample window.
  • Ranking candidate inputs by a single simple metric such as highest net profit or best profit factor is described as rarely producing good out-of-sample results.
  • A composite filter first drops in-sample rows with more than five consecutive losers, then keeps the twenty remaining rows with the largest median-win-to-median-loss ratios so a few large trades do not distort the ranking the way an average would.
  • Bootstrap resampling builds five thousand mirror filters that, in each weekly file, draw a random row’s out-of-sample net profit and sum those draws into a chance-level reference for total net profit.
Entries in this reading3 entries
Bootstrap resampling1 readingSystem optimization85 readingsCurrent trackWalk-forward analysis95 readings

Search many inputs, then test one selected row

A walk-forward procedure can search many input combinations on in-sample data, then apply a chosen metric filter to pick one input row whose associated rules are evaluated on the matching out-of-sample window.

System optimization is the in-sample search over those input combinations. Walk-forward analysis is the procedure that carries the selected row onto the matching out-of-sample window.

Simple ranking versus a composite metric filter

Ranking candidate inputs by a single simple metric such as highest net profit or best profit factor is described as rarely producing good out-of-sample results.

A more composite filter first drops in-sample rows with more than five consecutive losers, then keeps the twenty remaining rows with the largest median-win-to-median-loss ratios.

Median win and loss statistics are used in the filter so a few large, possibly nonrepeatable trades do not distort the ranking the way an average would.

The reported walk-forward explorer run

The reported walk-forward explorer run covers two hundred two weekly in-sample and out-of-sample windows of thirty-minute SPY bars for a parabolic SAR strategy, with first and last out-of-sample week-end dates of April 6, 2012 and February 12, 2016.

Out-of-sample evaluation in the reported run is framed on a one-hundred-share SPY size after subtracting a four-dollar round-trip cost and slippage from total net profit.

Walk-forward out-of-sample equity for the SPY SAR metric filter

Net equity after the $4 round-trip charge climbs from near zero to about $10,400 over 202 weekly out-of-sample windows, tracking gross equity with a widening cost gap and holding an uptrend through 2015–16 volatility. That ending net level is the path behind the filter’s reported $10,406 total out-of-sample net profit. Values were read from the published walk-forward plot for 100 SPY shares.
Net equity after the $4 round-trip charge climbs from near zero to about $10,400 over 202 weekly out-of-sample windows, tracking gross equity with a widening cost gap and holding an uptrend through 2015–16 volatility. That ending net level is the path behind the filter’s reported $10,406 total out-of-sample net profit. Values were read from the published walk-forward plot for 100 SPY shares.SPY · 202 weekly walk-forward windows on 30-minute bars · 2012-04-06T00:00:00.000Z to 2016-02-12T00:00:00.000Z

The in-sample screen kept rows with at most five consecutive losers, took the 20 highest median-win to median-loss ratios, then chose the smallest median losing trade. Second-order polynomial overlays on the source plot are omitted. Point values are approximate readings from the printed chart.

Chance-level mirrors and filter output columns

Chance-level comparison is constructed by building five thousand mirror filters that, in each weekly file, draw a random row’s out-of-sample net profit instead of the filter-selected row, then summing those draws to form a reference distribution of total net profit. That construction is the Bootstrap resampling step in the reported workflow.

Filter output columns include student-t on weekly out-of-sample profits, efficiency as average daily out-of-sample profit divided by average daily in-sample profit, and a probability that the filter’s total out-of-sample net profit arose by chance.

Educational research material, not investment advice. Historical source context does not establish present-day performance.
46 of 50 in the Walk-forward analysis track
201859-59 pp.Next on Walk-forward analysisEvaluate mechanical trading systems without catalog rankingsA mechanical trading system replaces discretionary interpretation at the decision point by issuing buy, sell, or stand-aside actions from stated rules.
All readings on this track · 50 readings
  1. 1990Three-window walk-forward system evaluation
  2. 1990Building the construction layer of a mechanical trading system
  3. 1991Constructing walk-forward neural trading rules
  4. 1991Constructing neural trading systems from facts to walk-forward
  5. 1992Walk-forward evaluation of stop overlays on average crossovers
  6. 1992Audit mechanical system tests for fills and regimes
  7. 1993Walk-forward evaluation of monthly yield and real-rate forecasts
  8. 1993Constructing walk-forward forecasts with linear and moving-average baselines
  9. 1993Walk-forward hybrid rules for intermarket forecast stacks
  10. 1994Neural-net construction as a mechanical trading-system problem
  11. 1995Constructing an intermarket neural net trading system
  12. 1996Weekly market breadth as one procedure on an unused window
  13. 1996Walk-forward evaluation of gold-index bond-fund rules
  14. 1996Evaluating weekday-in-month filters for index day trades
  15. 1996Require both a trend filter and a cycle oscillator before entry
  16. 1997Walk-forward windows as a diagnostic of parameter instability
  17. 1997Walk-forward validation of a market-breadth timing rule
  18. 1997Sunspot spikes and walk-forward evaluation of an adaptive cycle rule
  19. 1997A walk-forward check for bond-breadth timing
  20. 1998Walk-forward audit of regression trend forecasts
  21. 1998Evaluating a cubic least-squares currency trend with walk-forward segments
  22. 1998Walk-forward evaluation of recursive yen trend signals
  23. 1999Personal system design under crowd psychology
  24. 1999Walk-forward evaluation of a polynomial price forecast
  25. 2000Walk-forward optimization of regression-slope-angle rules
  26. 2001Construct a winter seasonal window as one procedure
  27. 2001Inspectable rules when system write-ups dry up
  28. 2002Evaluating mechanical systems before position sizing
  29. 2003Walk-forward construction of rule-based market-position systems
  30. 2007Evaluating metal seasonal windows across regimes
  31. 2007Evaluating mechanical timing systems against hold baselines
  32. 2011Walk-forward reoptimization as a system design gate
  33. 2011Evaluate generated systems on holdouts, then add stops
  34. 2012Walk-forward analysis and out-of-sample tests for a mechanical trading system
  35. 2012Personality-first trading system design
  36. 2012Scorecard-first mechanical system construction
  37. 2012Constructing an advancer-decliner moving average for market breadth
  38. 2012Formula search as mechanical system construction
  39. 2013Identity-first system construction
  40. 2013Construct a swing system from bias rules to walk-forward
  41. 2014Evaluate mechanical stock systems with stops and walk-forward
  42. 2014Walk-forward velocity filters on noisy intraday trends
  43. 2015Event-predictability versus position-constrained rules
  44. 2015Constructing mechanical systems for walk-forward tests
  45. 2016When a tested system must be retired
  46. 2016Walk-forward metric filters and chance-level checks for selected inputs
  47. 2018Evaluate mechanical trading systems without catalog rankings
  48. 2019Phased stop construction from entry risk to trailing exit
  49. 2020Stockpiling simple ideas for mechanical system construction
  50. 2020A pretty first draft is not a walk-forward waiver
By fieldSystem design117 readingsBy year201678 readings
All 95 readings tagged Walk-forward analysis
Also on Walk-forward analysis5 readings
1986Degrees of freedom in trading system optimizationSystem optimization1988Testing re-optimization calendars against random parameter controlsSystem optimization1988Walk-forward and neighborhood tests after optimizationSystem optimization1990Building the construction layer of a mechanical trading systemSystem optimization1990Three-window walk-forward system evaluationSystem optimization