Limitations of Quantitative Claims About Trading Strategy Evaluation

Overfitting is a chief concern for most systematic traders, and rightfully so. Traders use quantitative analysis methods to evaluate their backtests and tweak their strategies for better performance, however oftentimes adjusting a strategy based on the results of a backtest leads to an overfit strategy. Further, argues Michael Harris, determining when market conditions change is “in many cases fundamentally more important” than performance analysis results, and while prior literature has addressed overfitting and selection bias it has not addressed the limited ability of quantitative analysis methods to determine when a strategy will stop working due to a market regime change.

To illustrate this point, Harris presents two strategies, a trend-following strategy and a mean-reversion strategy, and compares the performance of each strategy between 1950-1997 and 1998-2016. The trend following strategy performed well in the 47-year period before 1998 but from 1998-2016 it yielded negative returns. The mean reversion strategy generated negative returns in the 47-year period before 1998 but performed well in the period 1998-2016. Had a trader at the end of 1997 not have been able to predict the market regime change, he may have found himself on the wrong side of the trade.

It is common practice among traders and researchers to use quantitative models, artificial intelligence and machine learning to evaluate a strategy. However, putting a strategy through dozens of quantitative backtests and optimizations leads to an overfit strategy that may have been successful during past market conditions but does not account for future market conditions. While the proper tool for timing a market regime shift is still a subject of much debate, Harris suggests reducing the number of backtests and optimizations and limiting the reuse of data to avoid overfitting. Read the full paper by Michael Harris here.

Get your free API token and start algo trading today!

To generate your token:

  1. Register for a free practice account here.
  2. Log into fxcm.com and click on the account ID in the upper right corner.
  3. Click token management and generate your token.

Risk Warning: The FXCM Group does not guarantee accuracy and will not accept liability for any loss or damage which arise directly or indirectly from use of or reliance on information contained within the webinars. The FXCM Group may provide general commentary which is not intended as investment advice and must not be construed as such. FX/CFD trading carries a risk of losses in excess of your deposited funds and may not be suitable for all investors. Please ensure that you fully understand the risks involved.