A critique of parameter fitting in system design

Fitting recovers a hindsight-best set

The archive describes searching every combination of several rule inputs as recovering the hindsight-best set rather than establishing a procedure that can be trusted on unseen data.

Color-coded buy and sell cues that cannot be independently validated are contrasted with entry, exit, and abstention rules a trader can test on historical data as one procedure.

Freeze the rules, then test a band of inputs

Optimization is treated as useful when inputs chosen on one data segment are then applied, unchanged, to a later unused segment. Walk-forward-analysis is that step: select the inputs on one historical segment, then apply the frozen procedure to a later unused segment so the same signal process is judged after the fitting data ends.

Testing a band of input values to see whether behavior holds across that band is presented as a robustness check, distinct from keeping a single winning setting. Robustness-testing asks whether the same entry, exit, and abstention procedure still produces coherent signals across a range of inputs rather than only at one winning setting.

Let the equity path constrain later signals

A bar-by-bar engine that runs the same rules on each symbol as history advances is described as letting the live equity path affect later decisions, unlike per-symbol tests that are merged only after the fact.

Stop-losses and rules that inspect the equity curve while the test is still unfolding are listed as ways to keep risk control inside the same procedure rather than adding it afterward. An equity-curve constraint inspects the procedure's own open equity path while the test is still running and can change later entries, exits, or abstention.

Specify the rules yourself

Risk tolerance and preferred markets are described as personal enough that a self-specified rule set is preferred to adopting a generic packaged system.

A simple, familiar rule such as a moving-average crossover is recommended as a first idea to encode, test in the trader's market of interest, and refine from those results.

Reviewing how a rule set behaved in the past is justified by the claim that participant psychology is relatively stable, so similar events may draw similar reactions, while conceding that this does not always occur.