11 Reasons Why Backtesting Frameworks are Nothing like the Real World

Photo by Sarah Kilian on Unsplash.com

“I don’t know who dropped this in Leeds city centre but I feel their disappointment…” the photographer commented on her photo above. Well guess what?! That’s how I feel about my latest backtest when I started paper trading. But… but… (lower lip quivering) the backtest results looked so great… 🥶🥶🥶

Unfortunately, it’s a sad reality in algo trading that backtesting doesn’t provide a foolproof method to ensure that a trading strategy will be profitable when you let the beast out of the cage so to say and start live trading.

Backtests are basically over simplistic simulations of real world trading using historic data with in hopes that historic results will reflect future results and conditions when live trading. In this post we are going to take a look under the hood of popular backtest frameworks and count the ways in which there are gotchas between the simulation and real world trading that can significantly f@#%k with your strategy results.

This story is solely for general information purposes, and should not be relied upon for trading recommendations or financial advice. Source code and information is provided for educational purposes only, and should not be relied upon to make an investment decision. Please review my full cautionary guidance before continuing.

1. Hindsight Is Always 20/20: Historic Results May Not Indicate Future Results

Have you ever heard the saying "past performance is not indicative of future results"? That's like the golden rule in backtesting. When backtesting, most commonly historical data is used to develop trading strategies and put them through the wind tunnel. However, historic market conditions may unfortunately not reflect the market-, economic- political- and asset-specific conditions that will arise in the future. As a result, a strategy that may have worked wonders in the past, may be a dud when running it live.

So, it’s good to keep in mind that backtesting tools like QuantConnect or Zipline let you test a strategy hypothesis but they don’t “let you see around the corners”. A good recent example is the COVID pandemic, which completely altered the market conditions that threw us traders a real curveball.

🔑 Tip: Diversify your backtest to include different market conditions and backtest with sufficiently long periods of time.

2. Guess What? Foresight is always 20/20 as well: The Dreaded Lookahead Bias

Lookahead Bias, also known as “Peeking” or “Snooping Bias” has bitten me in the butt more times than I can count: Ichimoku Cloud, calculating pivots with Kernel Regression, and shifting data back and forth. In simple terms, this type of bias can occur when you use any data to make a trading decision that is not available at the point in time when the backtest is executed which can lead to overoptimistic results. An inherent issue of backtesting is that while processing historic data, data of the “future horizon” is available but we need to be extremely conscious not to use it.

Lookahead bias can be introduced by using indicators, mathematical or statistical methods that use future data, by shifting data to include future data or by simply “looking ahead” to check future data.

🔑 Tip: Always double-check the indicator and signal calculations and make sure you're not accidentally using info from the future.

3. When the Backtest Pulls the Plug: Automatic Position Closing

Some backtesting frameworks like backtesting.py automatically close out positions at the end of the backtest, which can seriously screw with your backtest results. This feature is designed to provide a clear final valuation of the portfolio and accurately calculate backtest stats. However, when positions are closed automatically, they are closed by force at conditions that may not be in line with the strategy rules.

For example, if you are testing a swing trading strategy that relies on holding a stock while the through intermediate downturns and the backtesting.py closes out the position during a downturn at a loss, then this may invalidate your strategy if this case occurs often enough.

🔑 Heads-Up: Be aware of how your backtesting platform handles position closing at the end of the test and exclude backtest results that may affect the whole picture. You may also add additional strategy conditions to close these trades out early.