1991issue C051-12
Retesting weighted indicator balances across horizons
A timing system can be audited as one procedure. Each live directional call is scored by later market direction, the hit-rate becomes a statistical-weight, several horizons merge into a composite-balance, and the chi-square-screen is rerun when market structure changes.
- Each bullish or bearish call is checked against later market direction over a fixed horizon, then hits and misses become a period hit-rate.
- The hit-rate's significance is mapped to a 0-to-9 statistical-weight and placed in a bullish or bearish pan only while the same-horizon indicator is live.
- The same tally at 5-, 13-, 26-, and 52-week horizons is merged into one composite-balance and read as an inverted risk gauge, not a statement of how high or low prices will go.
- Assigned weights change when the sample is refreshed, and a high hit-rate whose chi-square-screen stays reasonably stable across reruns was preferred over ranking rules by total profit.
From a live call to a statistical-weight
An indicator can be scored by checking each bullish or bearish call against whether the market later moved that way over a fixed horizon, then converting hits and misses into a period hit-rate.
That hit-rate's statistical significance can be mapped to a 0-to-9 statistical-weight, which is then placed in a bullish or bearish pan whenever the same-horizon indicator is live.
Four horizons in one composite-balance
The same weighting tally is run at 5-, 13-, 26-, and 52-week horizons and then merged into one composite-balance.
The composite-balance is meant to be read as an inverted risk gauge that rises near bottoms and falls near tops, so a turn down after a peak was treated as the buy-side condition and a turn up as the sell-side condition.
A short-horizon cluster and a dated risk map
A cluster of three or four 100% short-horizon readings inside three or four weeks was used as a discrete trigger after earlier episodes had been reviewed against subsequent upside breakouts within a month.
Averaging every horizon balance that already speaks to the same future date yields a dated risk map that updates as new readings arrive and does not state how high or low prices will go. That overlay is the multi-horizon-roadmap.
Refreshing the chi-square-screen
Robustness checking compared each indicator's buys and sells with later market direction at those four horizons, then applied a chi-square-screen to ask whether the hit-rate was distinguishable from chance.
Assigned weights change when the sample is refreshed. A long stretch in which about 75% of weeks were up made downside accuracy harder to establish, which is why a later window with a more even mix of up and down weeks was planned after market structure had changed.
Structural-decay and a repeatable relationship
Indicators lose weight or are removed when the activity they measure fades, is distorted by new trading mechanics, or is redefined by reporting changes, while accidental correlations not tied to market behavior are treated as unreliable. That loss of accuracy is structural-decay.
Ranking rules by total profit can favor one large winning episode amid many misses. A high hit-rate whose chi-square-screen result stays reasonably stable across reruns was preferred as evidence of a repeatable relationship.
All readings on this track · 51 readings
- 1986Degrees of freedom in trading system optimization
- 1988Walk-forward and neighborhood tests after optimization
- 1988Undisclosed rules block system robustness tests
- 1988Testing re-optimization calendars against random parameter controls
- 1989Binary search limits on multi-peak average grids
- 1989Parameter neighborhoods that survive a shift
- 1990Use profit mapping to keep a cycle and stop plateau
- 1990Why popular indicator optimization fails robustness
- 1991Retesting weighted indicator balances across horizons
- 1992Constructing forecast models with regression, walk-forward, and robustness
- 1992Diagnose regimes before you lock parameters
- 1992When stops change system timing
- 1993Walk-forward halt rules for forecast models
- 1994Walk-forward evaluation of genetic index rules
- 1995Input pruning as walk-forward system evaluation
- 1995Critiquing neural nets as incomplete trading systems
- 1996Rebuild the equity-path ratio before it ranks a designed system
- 1996Parameter grids can fit random walks
- 1996Walk-forward analysis belongs in the design of a mechanical trading system
- 1997When a holdout fails, discard the rule set
- 1997Test rewarded rule breaks before replacing the system
- 1997Walk-forward rules keep system research from rewriting live trades
- 1999Keep a channel-breakout to two lookbacks and test neighbor stability
- 1999Constant investment size in stock system evaluation
- 2000Forcing optimization maps mechanical system failure boundaries
- 2000Robust parameter selection with surface charts
- 2001A two-gate classroom test for a two-window momentum trend filter
- 2002How a two-sided continuation factor becomes a testable trend rule
- 2002Evaluating two-window trend intensity as a reversal rule
- 2003Discounting speculative bubbles in system robustness tests
- 2003Walk-forward evaluation of locked stochastic oscillator rules
- 2003Critiquing mechanical system design after extreme price regimes
- 2004Evaluating a two-window trend trigger
- 2005Grade backtested signals with holdouts and optimization plateaus
- 2006Reserved-sample evaluation of trading system design
- 2006Walk-forward critique of hindsight crossover systems
- 2008Condition-matched walk-forward evaluation for mechanical systems
- 2011Session-split evaluation of regular and overnight systems
- 2012Walk-forward evaluation as operator rehearsal
- 2013Two-window evaluation of mechanical trading systems
- 2013Walk-forward filter selection for repeated-median velocity
- 2014Walk-forward evaluation for fading-memory velocity systems
- 2015Test oscillator events before tuning rules
- 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
- 2016Walk-forward optimization without curve fitting
- 2017Optimization without overfitting in trend-system evaluation
- 2017Parameter stability is a better guide than a larger crossover grid
- 2018Point-in-time universes for system evaluation
- 2018Walk-forward robustness evaluation for optimized systems
- 2018Critiquing breakout systems through robustness tests
- 2018A critique of parameter fitting in system design