Skip to main content
Track Robustness testing
50 / 51
Library

2018issue C0932-35

Critiquing breakout systems through robustness tests

The archive workflow presents a breakout-system as transferable across markets and holding periods, then uses robustness-testing to reject most early candidates as overfit material. Extra false-breakout filters are said to worsen that overfitting. Timing-layers and scenario-based dynamic-risk are the stated way to stand aside.

  • System work is said to begin by combining incomplete ideas, conditions, risk techniques, and indicators, then rejecting most of them through robustness-testing.
  • A breakout-system is one procedure for entry, exit, and standing aside, not a growing stack of false-breakout filters.
  • Failed follow-through is framed as a timing problem across general, system-level, volatility, and market views, while dynamic-risk can cut size or skip a trade.
  • After the beginner stage, attention moves to a small portfolio of low-correlation robust systems rather than a search for a currently winning rule set.
Entries in this reading3 entries

How breakout work is said to start

Breakout systems are presented as transferable across markets and timeframes, including both day and swing holding periods. System work is said to start by combining incomplete ideas, conditions, risk techniques, and indicators, then checking those candidates on historical data.

About 99% of early candidates are characterized as overfit material, so robustness-testing is the step that must separate unusable ideas from potentially viable systems. In the terms used here, robustness-testing is a demanding validation procedure used after ideation to separate overfit strategy candidates from systems that may remain usable.

Where systems are said to fail

The main stated reason systems fail is the absence of a comprehensive robustness-testing procedure. Commonly used validation methods are described as no longer sufficient. Most of the described development time, about 80%, is spent improving robustness-testing rather than searching for a currently winning rule set.

Later windows test the same procedure

Walk-forward-analysis is a sequential out-of-sample check that re-estimates rules on later data windows to test whether the same procedure still produces usable signals. TradersWeek editorial view: apply that later-window check to the same entry, exit, and abstention rules. A later window is not a reason to attach another filter.

False-breakout filters inflate the pile

A student survey is cited as finding false breakouts the top breakout-system problem, and filtering them out is said to increase overfitting instead of fixing the procedure. A false-breakout is a breakout signal that fails to follow through. Stacking extra filters is presented as a path to overfitting rather than a remedy.

Four timing-layers, not more entries

Reducing false breakouts is framed as a timing problem across four layers: general, system-level, volatility, and market. Those four timing views are the timing-layers used to reduce failed breakouts without adding signal filters. TradersWeek editorial view: if the procedure cannot stand aside, another entry condition is the wrong repair.

Scenario-based risk includes a skip

Risk is varied by scenario through an automated procedure: as much as 1.5 to 2% when conditions look favorable, about 0.5% or less when unpersuasive, and a full skip when highly unfavorable. That automated change in per-trade risk, including a full skip, is dynamic-risk driven by how favorable the current market scenario appears. TradersWeek editorial view: the skip belongs inside the breakout-system rather than beside it.

A small robust set after the beginner stage

Futures are preferred over equities and indexes because dissimilar contracts are described as easier to combine into lower-correlation portfolios. After the beginner stage, attention is directed to a portfolio of a few low-correlation robust systems, with three to five named as a sufficient starting set. That intermediate shift from single-strategy tinkering to a small set of low-correlation robust systems is the portfolio-stage.

Educational research material, not investment advice. Historical source context does not establish present-day performance.
50 of 51 in the Robustness testing track
201834-37 pp.Next on Robustness testingA critique of parameter fitting in system designSearching every combination of rule inputs is described as recovering the hindsight-best set, not a procedure that can be trusted on unseen data.
All readings on this track · 51 readings
  1. 1986Degrees of freedom in trading system optimization
  2. 1988Walk-forward and neighborhood tests after optimization
  3. 1988Undisclosed rules block system robustness tests
  4. 1988Testing re-optimization calendars against random parameter controls
  5. 1989Binary search limits on multi-peak average grids
  6. 1989Parameter neighborhoods that survive a shift
  7. 1990Use profit mapping to keep a cycle and stop plateau
  8. 1990Why popular indicator optimization fails robustness
  9. 1991Retesting weighted indicator balances across horizons
  10. 1992Constructing forecast models with regression, walk-forward, and robustness
  11. 1992Diagnose regimes before you lock parameters
  12. 1992When stops change system timing
  13. 1993Walk-forward halt rules for forecast models
  14. 1994Walk-forward evaluation of genetic index rules
  15. 1995Input pruning as walk-forward system evaluation
  16. 1995Critiquing neural nets as incomplete trading systems
  17. 1996Rebuild the equity-path ratio before it ranks a designed system
  18. 1996Parameter grids can fit random walks
  19. 1996Walk-forward analysis belongs in the design of a mechanical trading system
  20. 1997When a holdout fails, discard the rule set
  21. 1997Test rewarded rule breaks before replacing the system
  22. 1997Walk-forward rules keep system research from rewriting live trades
  23. 1999Keep a channel-breakout to two lookbacks and test neighbor stability
  24. 1999Constant investment size in stock system evaluation
  25. 2000Forcing optimization maps mechanical system failure boundaries
  26. 2000Robust parameter selection with surface charts
  27. 2001A two-gate classroom test for a two-window momentum trend filter
  28. 2002How a two-sided continuation factor becomes a testable trend rule
  29. 2002Evaluating two-window trend intensity as a reversal rule
  30. 2003Discounting speculative bubbles in system robustness tests
  31. 2003Walk-forward evaluation of locked stochastic oscillator rules
  32. 2003Critiquing mechanical system design after extreme price regimes
  33. 2004Evaluating a two-window trend trigger
  34. 2005Grade backtested signals with holdouts and optimization plateaus
  35. 2006Reserved-sample evaluation of trading system design
  36. 2006Walk-forward critique of hindsight crossover systems
  37. 2008Condition-matched walk-forward evaluation for mechanical systems
  38. 2011Session-split evaluation of regular and overnight systems
  39. 2012Walk-forward evaluation as operator rehearsal
  40. 2013Two-window evaluation of mechanical trading systems
  41. 2013Walk-forward filter selection for repeated-median velocity
  42. 2014Walk-forward evaluation for fading-memory velocity systems
  43. 2015Test oscillator events before tuning rules
  44. 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
  45. 2016Walk-forward optimization without curve fitting
  46. 2017Optimization without overfitting in trend-system evaluation
  47. 2017Parameter stability is a better guide than a larger crossover grid
  48. 2018Point-in-time universes for system evaluation
  49. 2018Walk-forward robustness evaluation for optimized systems
  50. 2018Critiquing breakout systems through robustness tests
  51. 2018A critique of parameter fitting in system design
All 58 readings tagged Robustness testing
Also on Robustness testing5 readings