FXMasterCourse

Trading Order Out of Chaos

  • Facebook
  • LinkedIn
  • RSS
  • Twitter
  • About
  • Testimonials
  • Contact
  • Member Login
You are here: Home / Trading Strategies / Improving your Python Backtesting – From DataFrames to Cython [Part 2]

February 23, 2021 by Corvin Codirla Leave a Comment

Improving your Python Backtesting – From DataFrames to Cython [Part 2]

Improving your Python Backtesting – From DataFrames to Cython [Part 2]

In Part 1 we used simple Python to Improve our Backtesting times.  Starting out with DataFrames we took a simple RSI strategy over a period of 7000 days and reduced from 7.3 seconds (which is a complete joke; sorry Pandas!) down to 0.003 seconds by converting everything to lists. But if you recall the comparison table at the end of Part 1

ImplementationTime for RSI2 Backtests
Python – Lists0.003s
Java0.00005s
C0.00002s

there was still a great gap between the Python list implementation and a simple Java or C implementation by as much as a factor of 100!

We promised to get back to this and start looking at Cython as bridge between the two worlds of interpreted language and bare metal implementation. You still get to live in the Python world with all the fancy machine learning and graphing facilities and fast backtesting!

So, by how much can Cython improve our timings?  Here is a sneak peak, so you don’t have to plough through all the text: all the way down to 0.00056s, which is six times faster and actually pretty close to bare metal.

If you want to repeat these numbers [which will be of course highly dependent on the machine you run it out] you can grab the code at this github repository: FXMC/backtest.

You might say, so who gives?  Why squeeze it all the way down.  The answer is simple.  When you’re doing research and you want to try your idea over many assets and with lots of different parameters, to get a feel for how stable or random your results are, or even just generate a useful sample of P&L paths for your Monte Carlo analysis; well, you’d probably like to do that within a couple of minutes.  Not a couple of days!

Table of Contents

Toggle
  • Intro to Cython
  • Cython Implementation
  • Python Backtesting Results
  • Conclusion

Intro to Cython

The focus here is on what I did to get it running.  Show you the specifics, and then you can copy paste in your own work.

First what is Cython?  In a nutshell: it is a transpiler that takes annotated Python source [hence it’s a pyx file and not a py file] and converts it to a C file.  This file is then compiled as a module that Python can load. 

The speed happens because your stuff has now been ported to C, and the load of the data is the only bottle neck. 

Why? Because when you run your stuff in Python, it has to be passed to your C(P)ython module and that conversion of data from one language to another takes time.  Specifically, because you’re going from a very lenient data language like Python to a very stringent data language like C.

However, if you include specific data annotations to your .pyx file you can bypass a lot of machinery.

It’s the annotations that make the magic.  Without them, it does really look like you’re not adding much to the speed improvement.

So, that’s what we will focus on from a syntax perspective (for completeness, the repo has the other non-annotated version as well).

Cython Implementation

Here it goes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cpdef void run_strategy(list close, list ma_long, list ma_short,
                       list rsi_n, float lower_rsi, int start_indx,
                       list posn, list cash):
    """Cythonized function with explicit type definitions"""
    cdef bint long_posn = False
    cdef int idx = 0
    cdef double shares = 0.0

    for idx in range(start_indx, len(close)):
        posn[idx] = posn[idx-1]
        cash[idx] = cash[idx-1]

        if long_posn:
            if close[idx] > ma_short[idx]:
                long_posn = False
                shares = posn[idx]
                posn[idx] = 0
                cash[idx] = cash[idx] + shares * close[idx]

        if not long_posn:
            if (close[idx] > ma_long[idx]) and (rsi_n[idx] < lower_rsi):
                long_posn = True
                shares = math.floor(cash[idx] / close[idx])
                posn[idx] = shares
                cash[idx] = cash[idx] - shares * close[idx]

Let’s go through how this differs from normal python code:

  1. First on the import side: nothing, we are not importing a thing
  2. The function has a cpdef tag with a void specifying a no return in this case.  This is because the results are passed by reference in the lists posn and cash
  3. Inside the function we also specify the types of all variables we will be using before jumping into the body of the function.
  4. The rest is standard python.

In terms of building this .pyx file we need a setup.py file which is in the repo.  The details of providing the correct C/C++ toolchain is left for the reader to check in the PSF website.  But it’s pretty straightforward.

Python Backtesting Results

How do the results stack up?

Here is the table again with the Cython results!

ImplementationTime for RSI2 Backtests
Python – DataFrame, date indexing7.3 s
Python – DataFrame, iterrows1.3s
Python – DataFrame, itertuples0.03s
Python – Lists0.003s
Python -- Cython no Type Hints0.0015s
Python -- Cython with Type Hints0.00056s
Java0.00005s
C0.00002s

As promised the Cython implementation gets down to 0.00056, with type hints included, boosting your Python backtesting!

This is a pretty awesome performance reduction even if you compare it only to the pure list implementation.

It appears that running simulations over many different configurations won’t be as long a wait or as psychologically tortuous, as it looked to be at the start!

Conclusion

It is true that you shouldn’t be obsessed with speed.  But sometimes it’s definitely worthwhile, especially when it boils down to the “research” loop: from idea to result and back to idea.  If the delays are significant, the psychology is different, and you’ll have an additional obstacle to overcome.

The big advantage here is that ALL the other modules in the Python eco-system are at your fingertips.  It might be worthwhile to compare the Python implementations to the C and Java ones in the repo, just to remind yourselves how much we’ve progressed in the last 40 years in terms of programming languages!

Both Part 1 and this second part are much more computer / programming focused than usual.  How does this relate to trading?  It does in as far as if you lack the tools to test and analyze, you’ll be stabbing in the dark.  And given that Python is the go-to language, it’s worthwhile being able to perform fast backtesting with it.

Filed Under: Trading Strategies Tagged With: Backtest, Python

Subscribe to Get the One Exercise which will Improve Your Foreign Exchange Trading Straight Away

  • Get rid of your fear of losing
  • Learn to stick to the rules
  • Achieve a Zen like state when you trade

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Browse Topics

Tags

ADP AUDUSD Backtest Bid Ask Bonds Breakout Commodities ECB Economic Indicators Economic Releases Emerging Markets Engulfing Candles Equities EURGBP EURJPY EURSEK EURUSD FOMC FX G10 GBPUSD Gold HFT Kelly Market Timing Mean Reversion Moon News Events News Trading NFP NZDUSD PBOC Performance Measures Python Range Expansion Risk Parity Seasonality Sharpe Ratio Slippage Solar Eclipse SP500 Trading Systems USDCAD USDCHF USDJPY

Subscribe to Get the One Exercise which will Improve Your Foreign Exchange Trading Straight Away

  • Get rid of your fear of losing
  • Learn to stick to the rules
  • Achieve a Zen like state when you trade

Search Site

Testimonials

Very clear, engaging and insightful.
CASS Seminar
The bar in trader education has been set and I highly recommend anyone wishing to pursue a career in trading to invest in this program.         Jehan Jabar
One-on-One Coaching
I really do rate everything with a “5.” Very good talk
CASS Seminar
Very insightful from an intelligent and talented individual. Life changing.
CASS Seminar
Quick thank you for the suggestion to look at shorting the CHFJPY. Just closed at 122 pips profit. With only 9 pips Max loss. Nice.  Michele Russell
One-on-One Coaching
Excellent, funny. Very knowledgeable!
CASS Seminar
Great teaching style. Very engaging. Material was presented in interesting manner and focused more on real life than theory.
CASS Seminar
Very well structured, interesting and insightfull
CASS Seminar

© 2025 · FXMasterCourse · Privacy Policy · Terms & Conditions · Earnings Disclaimer