1997issue C031-3
Audit the holdout and the exposure before the fitted entries
Grade a published mechanical system as a laboratory procedure before you inspect tidy entries. Ask which sample chose the signal thresholds, whether any unused market left those thresholds untouched, and whether the comparison book sat in the market for the same hours with the same stops.
- Entries that look unusually well placed on the same history used to tune the rules do not show how those entries would behave on unused data.
- A mechanical-system write-up that reports fitted signals without a holdout comparison leaves readers no separate yardstick for the procedure.
- A constant long book over the same test window is not an equal-risk contrast to a design that uses tight exits, forbids overnight holds, and spends only a small share of the window in positions.
- Winner share is a descriptive count, not a complete score: a book can remain economically interesting with a low winner share if losses are tightly capped and a few large wins dominate.
Grade the procedure first
Editorial view: critique a published mechanical system as a laboratory procedure before you look at tidy entries. The object of review is the harness: which sample chose the thresholds, whether any unused market left those thresholds untouched, and whether the comparison book sat in the market for the same hours with the same stops. How orderly the fitted signals appear is not that review.
Entries that look unusually well placed on the same history used to tune the rules do not show how those entries would behave on unused data. A mechanical-system write-up that reports fitted signals without a holdout comparison leaves readers no separate yardstick for the procedure.
Holdout is unused history
Curve-fitting searches rule constants so that reported signals look strong on the same sample that was searched. A holdout is price history or a second market that did not help choose the rule thresholds and is used only to check those unchanged rules.
Leaving the constants unchanged and applying the same mechanical rules to a second market was offered as a form of holdout, alongside the claim that many trades relative to few tuned constants lessens fit to the original series.
Match exposure before you compare
A constant long book over the same test window is not an equal-risk contrast to a design that uses tight exits, forbids overnight holds, and spends only a small share of the window in positions. Time in market is the share of the test window during which a position is actually open.
An exposure-matched benchmark is a comparison book that shares the system's stops, overnight limits, and fraction of time in the market. An exposure-matched control was described as many random-entry trade sets that keep the same stops, flatten on the close, and spend a similar fraction of time in the market. A random-entry control keeps exits and risk limits fixed while choosing entries without the system's signal logic.
Winner share is a count, not the score
A mechanical book can remain economically interesting with a low winner share if losses are tightly capped and a few large wins dominate, so hit rate alone is a weak score for the procedure. Winner share is the fraction of closed trades that ended with a gain: a descriptive count, not a complete score for the book.
A critic applied a two-outcome chance model to a reported winner share and a stated trial count, arguing that a higher share remained likely and that the published hit rate was therefore not decisive.
Undefined constants are not a testable procedure
Published optimization tables and signal constants cannot be rebuilt as a testable procedure when lookbacks, buy and sell cutoffs, and column codes are left undefined. An optimization report is a table of candidate constants and summary scores produced while searching the rule space.
Those cutoffs were later described as fixed development constants: a buy is allowed only after a named strength-style indicator reading exceeds its buy threshold. A signal threshold is a fixed cutoff, set in development, that an indicator reading must cross before a buy or sell is allowed.
All readings on this track · 23 readings
- 1984Evaluating mechanical systems beyond net profit
- 1987Mechanical systems need more than pretty charts
- 1987Evaluating silver moving averages and RSI by scoring rule
- 1988Simple systems over optimized complexity
- 1988Cycle rule inputs until the parameter search stops
- 1990Split-sample tests for stochastic parameter grids
- 1993Four-subsystem trading system construction
- 1993Randomized price paths for stock system parameters
- 1994Mechanical timing with out-of-sample and filter checks
- 1995Evaluating asymmetric percent-reversal trading rules
- 1996Evaluating a daily volume-strength and new-high new-low timing procedure
- 1997Rule-based intraday system design with genetic search
- 1997Evolving three-rule systems with genetic templates
- 1997Audit the holdout and the exposure before the fitted entries
- 2001Inner weights are part of the candlestick signal
- 2003Mechanical short-term rules for news-sensitive rallies
- 2004Band construction as a testable trading procedure
- 2006Layered day-session automation from brackets to rules
- 2006Why mechanical rules must be retuned per security
- 2006A linear-regression next-bar forecast needs a searchable cutoff
- 2015Construct if-then rules, then backtest and optimize
- 2015Mechanical system design as a critique of discretion
- 2025Joint channel design with a trend filter and stop-loss orders