1997issue C071-3
Test rewarded rule breaks before replacing the system
A carefully specified mechanical trading system can still lose in the short run, and a deliberate rule break can still be followed by a favorable short-horizon outcome. Editorial view: treat that rewarded exception as a robustness-testing ticket and keep executing the incumbent rules until the exception is specified, compared and either discarded or written in.
- Short-run losses on a specified system, and short-horizon wins after a rule break, can start a pattern of abandoning the procedure.
- Substituting peer advice for system signals, or following an accidental rewarded exception with more rule breaks, can strip trust from the incumbent rules and leave the account largely depleted.
- Design a candidate change in planning mode, with contingencies, and keep the incumbent rules in force until the exception is tested and written in.
- Editorial view: a rewarded exception is a robustness-testing ticket, not a license to abandon the mechanical trading system.
Short-horizon results can retire the procedure
A carefully specified mechanical trading system can still post losses in the short run. A deliberate rule break can still be followed by a favorable short-horizon outcome. That pairing can start a pattern of abandoning the procedure.
A mechanical trading system is a single written procedure that turns rule inputs, market state and execution constraints into entry, exit or abstention signals over the system holding period. The trading psychology process is the habit loop in which short-horizon wins and losses train the operator to follow, outsource or sabotage that pre-specified procedure.
Peer advice and sequential breaks
A new operator who substitutes peer advice for a purchased, tested system's signals can receive intermittent favorable outcomes. Those outcomes can further reduce trust in the original rules, including unused system signals that later looked favorable.
A long-tenured operator who accidentally violates a major rule and receives an unusually large gain can then break remaining rules in sequence until the account is largely depleted. In both cases the first rewarded exception, a rule violation followed by a favorable short-horizon outcome, tempts the operator to treat the violation as a new method.
Specify the exception in planning mode
A candidate rule change should be designed in planning mode, with contingencies for the consequences of the change, while the incumbent rules continue to be followed. Planning mode is the separate design step in which that draft is specified without substituting it for live signals. The incumbent rules are the current written procedure that remains in force while any candidate change is planned and tested.
A favorable result after a bad decision is not sufficient reason to keep the exception unless that exception is tested and written into the rules.
Trust comes after the long sample
New rules earn trust through later losing stretches only after they have been backtested and shown to function over a long sample. Commitment through drawdown is the willingness to keep executing a rule set after testing has shown why that set is the procedure to follow when results turn unfavorable.
Keep decision ownership with the operator
The operator should retain responsibility for trading decisions, because insecurity about oneself or the system makes it easy to hand those decisions to other traders. Decision ownership is the requirement that the operator, not a more confident peer, issue the entry, exit and abstention signals.
Emotional discharge should happen outside the trading process. Following the rules can be paired with a personally valued reward so adherence carries a positive association.
When the incumbent rules were never tested
When incumbent rules were never seriously backtested, simultaneous testing of those rules and the recent exceptions can decide which procedure to adopt as the committed rule set. Robustness testing is that side-by-side historical testing of an incumbent rule set against a specified exception before the operator is asked to trust either set through later losing stretches.
A rule set should be believed because it has been tested over time and shown to be reliable. If it cannot pass that test, the operator should check whether the exceptions can themselves be written as committed rules.
All readings on this track · 51 readings
- 1986Degrees of freedom in trading system optimization
- 1988Walk-forward and neighborhood tests after optimization
- 1988Undisclosed rules block system robustness tests
- 1988Testing re-optimization calendars against random parameter controls
- 1989Binary search limits on multi-peak average grids
- 1989Parameter neighborhoods that survive a shift
- 1990Use profit mapping to keep a cycle and stop plateau
- 1990Why popular indicator optimization fails robustness
- 1991Retesting weighted indicator balances across horizons
- 1992Constructing forecast models with regression, walk-forward, and robustness
- 1992Diagnose regimes before you lock parameters
- 1992When stops change system timing
- 1993Walk-forward halt rules for forecast models
- 1994Walk-forward evaluation of genetic index rules
- 1995Input pruning as walk-forward system evaluation
- 1995Critiquing neural nets as incomplete trading systems
- 1996Rebuild the equity-path ratio before it ranks a designed system
- 1996Parameter grids can fit random walks
- 1996Walk-forward analysis belongs in the design of a mechanical trading system
- 1997When a holdout fails, discard the rule set
- 1997Test rewarded rule breaks before replacing the system
- 1997Walk-forward rules keep system research from rewriting live trades
- 1999Keep a channel-breakout to two lookbacks and test neighbor stability
- 1999Constant investment size in stock system evaluation
- 2000Forcing optimization maps mechanical system failure boundaries
- 2000Robust parameter selection with surface charts
- 2001A two-gate classroom test for a two-window momentum trend filter
- 2002How a two-sided continuation factor becomes a testable trend rule
- 2002Evaluating two-window trend intensity as a reversal rule
- 2003Discounting speculative bubbles in system robustness tests
- 2003Walk-forward evaluation of locked stochastic oscillator rules
- 2003Critiquing mechanical system design after extreme price regimes
- 2004Evaluating a two-window trend trigger
- 2005Grade backtested signals with holdouts and optimization plateaus
- 2006Reserved-sample evaluation of trading system design
- 2006Walk-forward critique of hindsight crossover systems
- 2008Condition-matched walk-forward evaluation for mechanical systems
- 2011Session-split evaluation of regular and overnight systems
- 2012Walk-forward evaluation as operator rehearsal
- 2013Two-window evaluation of mechanical trading systems
- 2013Walk-forward filter selection for repeated-median velocity
- 2014Walk-forward evaluation for fading-memory velocity systems
- 2015Test oscillator events before tuning rules
- 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
- 2016Walk-forward optimization without curve fitting
- 2017Optimization without overfitting in trend-system evaluation
- 2017Parameter stability is a better guide than a larger crossover grid
- 2018Point-in-time universes for system evaluation
- 2018Walk-forward robustness evaluation for optimized systems
- 2018Critiquing breakout systems through robustness tests
- 2018A critique of parameter fitting in system design