1988issue C061-2
Undisclosed rules block system robustness tests
A mechanical system sold without its entry, exit, and stay-out rules cannot be checked, versioned, or diagnosed. Editorial view: disclosure is the control condition that separates an inspectable procedure from vendor theater.
- An undisclosed system withholds both the trading rules and the preferred parameter values, so the buyer cannot run an inspectable procedure.
- Without published rules, result diagnosis cannot tell a normal outcome from a market change or a defective design.
- Hidden logic also permits silent revision, so a later update may no longer be the procedure that was bought.
- Editorial view: ignore undisclosed systems, or treat them only as entertainment software, because parameter search and robustness checks are unverifiable.
What an undisclosed system withholds
Some commercial mechanical systems were sold with both the trading rules and the preferred parameter values withheld. The buyer paid first and then had to reconstruct the procedure afterward. That offering is an undisclosed system: a mechanical procedure sold without publishing the rules or the preferred parameter values that generate its signals.
Vendors were reported to sell such opaque systems successfully. Raising the price itself was treated as a demand tactic, including services that began in a 25 to 40 range.
An inspectable procedure can be priced modestly
Openly described idea packages priced under 1000, and sometimes under 100, were judged comparable to expensive disclosed packages and better than any hidden-rule package. An inspectable procedure is a complete, auditable set of entry, exit, and stay-out rules that can be run, revised, and version-checked as one method.
Result diagnosis needs published rules
If the rules stay hidden, a trader cannot tell whether a given gain or loss is a normal system outcome, an unforeseen market change, or a defective design. Result diagnosis is that classification, and it requires a known procedure.
Hidden logic also blocks version control. One currency system was reported to change signals with every update until updates stopped, so buyers could not confirm they still held the same procedure. That pattern is silent revision: a vendor update that can alter hidden signal logic so live behavior no longer matches the procedure the buyer thought was tested.
Why secrecy is not required
Without disclosure, buyers cannot separate crooked vendors from reputable ones. That gap was treated as sufficient reason to ignore undisclosed systems.
Full rule disclosure is urged because data capture, signal generation, and parameter-search software can cost more to rebuild than the purchase price, so secrecy is not required to protect the product. Parameter search is the adjustment of rule inputs against historical market state. The search is unverifiable if the rules themselves are secret.
The later conclusion is not to buy a system whose interior is withheld, unless the price is in the range usually paid for entertainment software.
All readings on this track · 51 readings
- 1986Degrees of freedom in trading system optimization
- 1988Walk-forward and neighborhood tests after optimization
- 1988Undisclosed rules block system robustness tests
- 1988Testing re-optimization calendars against random parameter controls
- 1989Binary search limits on multi-peak average grids
- 1989Parameter neighborhoods that survive a shift
- 1990Use profit mapping to keep a cycle and stop plateau
- 1990Why popular indicator optimization fails robustness
- 1991Retesting weighted indicator balances across horizons
- 1992Constructing forecast models with regression, walk-forward, and robustness
- 1992Diagnose regimes before you lock parameters
- 1992When stops change system timing
- 1993Walk-forward halt rules for forecast models
- 1994Walk-forward evaluation of genetic index rules
- 1995Input pruning as walk-forward system evaluation
- 1995Critiquing neural nets as incomplete trading systems
- 1996Rebuild the equity-path ratio before it ranks a designed system
- 1996Parameter grids can fit random walks
- 1996Walk-forward analysis belongs in the design of a mechanical trading system
- 1997When a holdout fails, discard the rule set
- 1997Test rewarded rule breaks before replacing the system
- 1997Walk-forward rules keep system research from rewriting live trades
- 1999Keep a channel-breakout to two lookbacks and test neighbor stability
- 1999Constant investment size in stock system evaluation
- 2000Forcing optimization maps mechanical system failure boundaries
- 2000Robust parameter selection with surface charts
- 2001A two-gate classroom test for a two-window momentum trend filter
- 2002How a two-sided continuation factor becomes a testable trend rule
- 2002Evaluating two-window trend intensity as a reversal rule
- 2003Discounting speculative bubbles in system robustness tests
- 2003Walk-forward evaluation of locked stochastic oscillator rules
- 2003Critiquing mechanical system design after extreme price regimes
- 2004Evaluating a two-window trend trigger
- 2005Grade backtested signals with holdouts and optimization plateaus
- 2006Reserved-sample evaluation of trading system design
- 2006Walk-forward critique of hindsight crossover systems
- 2008Condition-matched walk-forward evaluation for mechanical systems
- 2011Session-split evaluation of regular and overnight systems
- 2012Walk-forward evaluation as operator rehearsal
- 2013Two-window evaluation of mechanical trading systems
- 2013Walk-forward filter selection for repeated-median velocity
- 2014Walk-forward evaluation for fading-memory velocity systems
- 2015Test oscillator events before tuning rules
- 2016Walk-forward evaluation of a five-parameter parabolic stop-and-reversal
- 2016Walk-forward optimization without curve fitting
- 2017Optimization without overfitting in trend-system evaluation
- 2017Parameter stability is a better guide than a larger crossover grid
- 2018Point-in-time universes for system evaluation
- 2018Walk-forward robustness evaluation for optimized systems
- 2018Critiquing breakout systems through robustness tests
- 2018A critique of parameter fitting in system design