Skip to main content
Track Hypothesis testing
13 / 20
Search the archiveLibrary
Library

1997issue C081-6

Walk-forward tests after system optimization

A locked-parameter run on unused data is one test with no further adjustment. After a search, walk-forward analysis records trades from the next unseen interval so hypothesis testing on those trades needs no multiplicity correction.

  • A single locked-parameter run on unused data is the same evaluation setup as an unoptimized system: one test and no further parameter adjustment.
  • Simulated trades from a fixed-parameter run are treated as a sample, so a t-statistic compares the mean trade result with a hypothesized mean of zero or worse, and a win-rate interval is formed from the binomial count of winning trades.
  • After a search, the best in-sample t-statistic is multiplicity-adjusted; once parameters are locked, one out-of-sample run does not need that correction, and neither do walk-forward trades from the next unseen interval.
  • Statistical checks during development are used to reject models whose observed behavior could be explained by chance or by excessive fitting on an inadequate sample.
Entries in this reading3 entries
Current trackHypothesis testing26 readingsWalk-forward analysis119 readingsSystem optimization131 readings

One unused run after lock-in

A single run of a locked-parameter system on unused data is the same evaluation setup as an unoptimized system: one test and no further parameter adjustment. Simulated trades from that fixed-parameter run can be treated as a sample from the system's trade population, so inference targets the mean trade result versus a hypothesized population mean of zero or worse.

The t-statistic is the sample mean trade result minus that stated baseline, divided by the standard error of that mean. Its tail probability is read from the t distribution with degrees of freedom equal to the number of trades minus one. An interval for the population win rate can be formed by inverting the cumulative binomial distribution around the observed count of winning trades.

How a search changes the reading

After a search over parameter combinations, the significance of the best in-sample t-statistic is adjusted to one minus the quantity one minus that significance, raised to the number of trials. That adjusted figure estimates the chance of seeing at least one result that good. In-sample means the data window used while parameters are searched.

Treating every trial as independent overstates how many distinct tests were run when nearby parameter values produce similar results, so an independence-based multiplicity adjustment is conservative. Once parameters are locked and only one out-of-sample run is made, statistics computed on those trades do not require an optimization correction. Out-of-sample means a later window used only after parameters are locked.

Walk-forward analysis reoptimizes on a moving window, records trades from the next unseen interval, then advances the window. Those trades remain out-of-sample, and t-tests on them need no multiplicity correction.

Hypothesis testing during development is used to reject models whose observed behavior could be explained by chance or by excessive fitting on an inadequate sample.

Educational research material, not investment advice. Historical source context does not establish present-day performance.
13 of 20 in the Hypothesis testing track
19971-4 pp.Next on Hypothesis testingTesting breakout rate versus rally speedA 10-period moving-average channel of highs and lows located reactionary turning points that started and stopped later break and rally segments.
All readings on this track · 20 readings
  1. 1988Serial dependence, trade excursions, and belief revision
  2. 1989Score an indicator before anyone writes a buy or sell rule
  3. 1990Evaluating safety labels in a crash window
  4. 1991Keep a decade-year only when it differs from the overall-mean
  5. 1991Audit indicator forecasts with chi-square hit rates
  6. 1992A lag-regression audit of stochastic forecast skill
  7. 1992Constructing detrended tests for calendar seasonality
  8. 1992Building a shared workbook for calendar seasonality tests
  9. 1995T-test evaluation of simulated trade returns
  10. 1995Constructing a two-proportion Z-test for hit rates
  11. 1996Treat a volume-price surge as a contingency table
  12. 1997Testing bond and stock seasonality before you trade
  13. 1997Walk-forward tests after system optimization
  14. 1997Testing breakout rate versus rally speed
  15. 2000Score the misses before a pattern is a system
  16. 2002Leave a moving-average crossover unfinished until chance cannot explain the ranges
  17. 2013Keep or drop an RSI threshold with a complementary-sample test
  18. 2013Constructing a null and a t-test for sample returns
  19. 2017Partial correlation and the market-adjusted stock trend
  20. 2017Constructing market-adjusted stock trend scores
By fieldSystem design203 readingsBy year199791 readings
All 26 readings tagged Hypothesis testing
Also on Hypothesis testing5 readings
1986Chi-square test for clustered price-change histograms1986Constructing price-change density histograms and symmetry tests1988Serial dependence, trade excursions, and belief revision1989Score an indicator before anyone writes a buy or sell rule1990Evaluating safety labels in a crash window