1986issue C021-2
Skill score versus a coin-flip forecast baseline
Outcomes that look successful can still arise from chance. Log each forecast beside a contemporaneous coin-flip call, score both under one correctness rule, and compute a skill score only after a large set of paired decisions.
- A run of correct calls does not identify skill, because successful-looking outcomes can still arise from chance-dominated processes.
- Log each forecast beside a contemporaneous coin-flip no-skill call and mark both under one prechosen correctness rule.
- A skill score subtracts correct no-skill calls from both the forecast hit count and the total decision count, then takes the ratio of those two differences.
- Compute the score only after 50 to 100 paired decisions. Zero or below means no skill beyond luck; a positive score is only a crude measure of how much skill was present.
A run of correct calls is not skill
Outcomes that look successful can still arise from chance-dominated processes. A run of correct calls does not by itself identify skill.
Keep a paired decision log
A no-skill comparison call can be generated by flipping a coin and assigning one face to sell and the other to buy. That coin-flip no-skill baseline is a same-period random buy-or-sell call, scored with the same correctness rule as the forecast.
Each forecast should be logged beside a contemporaneous no-skill call and later marked correct or incorrect under one prechosen scoring rule applied to both. The paired decision log stores each forecast, its contemporaneous coin-flip call, and the later correctness outcome of both.
The correctness rule is a predeclared test of whether a buy or sell call was right. Apply that same rule to the forecast and to the baseline.
How the skill score is formed
A skill score subtracts the count of correct no-skill calls from both the count of correct forecast calls and the total number of forecast calls, then takes the ratio of those two differences.
A skill score of zero or below indicates that the forecasts added no skill beyond luck. A positive skill score indicates that some skill was present, while the size of the score is only a crude measure of how much.
Editorial reading
Editorial reading: treat every forecast log as an evaluation problem. Pair each call with a same-period coin-flip baseline and compute a skill score before a hit rate is mistaken for skill. This is a TradersWeek interpretation of the archive workflow, not a claim made by the archive.
All readings on this track · 6 readings
- 1986Skill score versus a coin-flip forecast baseline
- 1991Evaluating a trailing stop against a coin-flip entry
- 2004Evaluating trend rules against no-skill baselines
- 2005Evaluating systems with walk-forward analysis, robustness testing, and coin-flip baselines
- 2015Trade-tape entropy versus a coin-flip no-skill baseline
- 2017A coin-flip timed exit as the skill floor for trend and mean-reversion