Limitations of Quantitative Claims About Trading Strategy Evaluation

Overfitting is a chief concern for most systematic traders, and rightfully so. Traders use quantitative analysis methods to evaluate their backtests and tweak their strategies for better performance. However oftentimes adjusting a strategy based on the results of a backtest leads to an overfit strategy. Further, argues Michael Harris, determining when market conditions change is “in many cases fundamentally more important” than performance analysis results. While prior literature has addressed overfitting and selection bias, it has not addressed the limited ability of quantitative analysis methods to determine when a strategy will stop working due to a market regime change.

To illustrate this point, Harris presents two strategies, a trend-following strategy and a mean-reversion strategy. He compares the performance of each strategy between 1950-1997 and 1998-2016. The trend following strategy performed well in the 47-year period before 1998 but from 1998-2016 it yielded negative returns. The mean reversion strategy generated negative returns in the 47-year period before 1998 but performed well in the period 1998-2016. Had a trader at the end of 1997 not have been able to predict the market regime change, he may have found himself on the wrong side of the trade.

It is common practice among traders and researchers to use quantitative models, artificial intelligence and machine learning to evaluate a strategy. However, putting a strategy through dozens of quantitative backtests and optimizations leads to an overfit strategy that may have been successful during past market conditions but does not account for future market conditions. The proper tool for timing a market regime shift is still a subject of much debate. However, Harris suggests reducing the number of backtests and optimizations and limiting the reuse of data to avoid overfitting. Read the full paper by Michael Harris here.

Get your free API token and start algo trading your strategy today!

To generate your token:

  1. Register for a free practice account here.
  2. Log into and click on the account ID in the upper right corner.
  3. Click token management and generate your token.

Risk Warning: The FXCM Group does not guarantee accuracy and will not accept liability for any loss or damage which arise directly or indirectly from use of or reliance on information contained within the webinars. The FXCM Group may provide general commentary which is not intended as investment advice and must not be construed as such. FX/CFD trading carries a risk of losses in excess of your deposited funds and may not be suitable for all investors. Please ensure that you fully understand the risks involved.

Demo Account: Although demo accounts attempt to replicate real markets, they operate in a simulated market environment. As such, there are key differences that distinguish them from real accounts; including but not limited to, the lack of dependence on real-time market liquidity, a delay in pricing, and the availability of some products which may not be tradable on live accounts. The operational capabilities when executing orders in a demo environment may result in atypically, expedited transactions; lack of rejected orders; and/or the absence of slippage. There may be instances where margin requirements differ from those of live accounts as updates to demo accounts may not always coincide with those of real accounts.