Evaluating mechanical systems beyond net profit

Mechanical rules as a complete procedure

Mechanical trading systems existed before computers. They generate buy, sell, hold, or exit plans from price-tracking rules, including moving-average trend followers.

The rules are predetermined and applied without discretionary override during the test or the trade.

Computerizing mechanical rules reduces calculation and transcription error and can include bounds checks on incoming data. Computerization does not remove doubt that a system has stopped working after consecutive losses.

Optimization ranks combinations

Optimization consists of testing many parameter combinations, such as moving-average length, so the computer can rank which combination performed best on a chosen measure.

Static analysis searches a long series, described as at least three years, for one parameter set expected to remain useful. Dynamic analysis reoptimizes more often, generally over less than one year, because market character may change.

Define what best means

What counts as best must be defined explicitly, because highest net profit, highest probability of a profitable trade, and lowest risk-to-return are different evaluation targets.

Five core evaluation criteria are total net profit, average net profit, the coefficient of variation of average net profit, the risk-to-return ratio, and the probability of a profitable trade. These five expand into eighteen evaluative concepts.

Average profit, variation, and capital path

Average net profit, also called the expected value of a trade, supports comparison across unequal periods. A high average can come from few large trades and can be distorted by outliers.

The coefficient of variation of average net profit is formed by dividing the standard deviation of the mean by the mean. It is interpreted as risk relative to average return.

After ranking parameter sets, evaluation should inspect whether top combinations cluster in similar ranges. Capital needs can be estimated from consecutive gain and loss strings and cumulative profit plots. Several criteria may be combined into a composite measure.

What the ranking does not finish

TradersWeek editorial: clustering of top parameter sets, inspection of the capital path, and a composite of several criteria belong to the design of a testable system. They are not optional notes attached after a profit ranking.