Reserved-sample evaluation of trading system design

The procedure is what must be judged

A finished backtest result is not treated as a sufficient evaluation of a trading system. The procedure used to obtain that result is what must be judged.

Concept, initial rules, and degrees of freedom are written down first. Degrees of freedom are the rule inputs a developer is allowed to vary during a project; anything not declared stays fixed. Only those declared degrees of freedom may be varied later in the project.

A stated split of the historical pool

One stated split of the historical pool is 5% build data, 40% test data, 40% walk-forward data, and 15% unseen data, with a single symbol preferably assigned to at least three of those roles.

Build data are used only to confirm that coded orders match the intended rules before system-optimization begins. System-optimization is a search over declared degrees of freedom on a test sample to choose rule inputs before any later reserved-data evaluation.

Choose a robust set on the test sample

System-optimization on the test sample chooses a robust set of degrees of freedom. Robustness-testing is a check that acceptable rule inputs appear across a band of nearby settings rather than at a single tuned point. A wide band of acceptable settings is treated as more informative than a single tuned combination.

Walk-forward analysis on unused data

After the test-sample backtest is judged acceptable enough to continue evaluation, walk-forward analysis applies those locked degrees of freedom to unused walk-forward data. Walk-forward analysis is a locked-parameter evaluation on reserved data that asks whether the same entry, exit, and abstention rules still produce an acceptable signal after optimization has finished. If the reserved result is not acceptable, the project returns to refinement.

When reserved samples become contaminated

After five refine-test-walk-forward loops, the walk-forward sample is treated as contaminated by hindsight and is no longer trusted as an independent evaluation. Data-contamination is the loss of independence that occurs when reserved data are reused after their outcomes have already been seen.

Unseen data are a last reserved sample held out so repeated walk-forward trials do not consume every unused observation. Unseen data function as a second reserved walk-forward sample so earlier loops need not spend the last unused history. A negative result there ends the project because remaining observations are then considered spoiled.

Rejection is the expected reserved-data outcome

A development process is expected to reject most coded systems at the reserved-data stages. Keeping hindsight out of the final survive-or-abandon decision is treated as more important than producing a passing result on every project.