FXMasterCourse

Trading Order Out of Chaos

  • Facebook
  • LinkedIn
  • RSS
  • Twitter
  • About
  • Trading Approach
  • Resources
  • Forex Tools
    • MT4 Code
    • Tick Data
    • Useful Tools
  • In the Media
  • Testimonials
  • Contact
  • Member Login
You are here: Home / Trading Strategies / Improving your Python Backtesting – From DataFrames to Cython [Part 2]

February 23, 2021 by Corvin Codirla Leave a Comment

Improving your Python Backtesting – From DataFrames to Cython [Part 2]

Improving your Python Backtesting – From DataFrames to Cython [Part 2]

In Part 1 we used simple Python to Improve our Backtesting times.  Starting out with DataFrames we took a simple RSI strategy over a period of 7000 days and reduced from 7.3 seconds (which is a complete joke; sorry Pandas!) down to 0.003 seconds by converting everything to lists. But if you recall the comparison table at the end of Part 1

ImplementationTime for RSI2 Backtests
Python – Lists0.003s
Java0.00005s
C0.00002s

there was still a great gap between the Python list implementation and a simple Java or C implementation by as much as a factor of 100!

We promised to get back to this and start looking at Cython as bridge between the two worlds of interpreted language and bare metal implementation. You still get to live in the Python world with all the fancy machine learning and graphing facilities and fast backtesting!

So, by how much can Cython improve our timings?  Here is a sneak peak, so you don’t have to plough through all the text: all the way down to 0.00056s, which is six times faster and actually pretty close to bare metal.

If you want to repeat these numbers [which will be of course highly dependent on the machine you run it out] you can grab the code at this github repository: FXMC/backtest.

You might say, so who gives?  Why squeeze it all the way down.  The answer is simple.  When you’re doing research and you want to try your idea over many assets and with lots of different parameters, to get a feel for how stable or random your results are, or even just generate a useful sample of P&L paths for your Monte Carlo analysis; well, you’d probably like to do that within a couple of minutes.  Not a couple of days!

Intro to Cython

The focus here is on what I did to get it running.  Show you the specifics, and then you can copy paste in your own work.

First what is Cython?  In a nutshell: it is a transpiler that takes annotated Python source [hence it’s a pyx file and not a py file] and converts it to a C file.  This file is then compiled as a module that Python can load. 

The speed happens because your stuff has now been ported to C, and the load of the data is the only bottle neck. 

Why? Because when you run your stuff in Python, it has to be passed to your C(P)ython module and that conversion of data from one language to another takes time.  Specifically, because you’re going from a very lenient data language like Python to a very stringent data language like C.

However, if you include specific data annotations to your .pyx file you can bypass a lot of machinery.

It’s the annotations that make the magic.  Without them, it does really look like you’re not adding much to the speed improvement.

So, that’s what we will focus on from a syntax perspective (for completeness, the repo has the other non-annotated version as well).

Cython Implementation

Here it goes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cpdef void run_strategy(list close, list ma_long, list ma_short,
                       list rsi_n, float lower_rsi, int start_indx,
                       list posn, list cash):
    """Cythonized function with explicit type definitions"""
    cdef bint long_posn = False
    cdef int idx = 0
    cdef double shares = 0.0

    for idx in range(start_indx, len(close)):
        posn[idx] = posn[idx-1]
        cash[idx] = cash[idx-1]

        if long_posn:
            if close[idx] > ma_short[idx]:
                long_posn = False
                shares = posn[idx]
                posn[idx] = 0
                cash[idx] = cash[idx] + shares * close[idx]

        if not long_posn:
            if (close[idx] > ma_long[idx]) and (rsi_n[idx] < lower_rsi):
                long_posn = True
                shares = math.floor(cash[idx] / close[idx])
                posn[idx] = shares
                cash[idx] = cash[idx] - shares * close[idx]

Let’s go through how this differs from normal python code:

  1. First on the import side: nothing, we are not importing a thing
  2. The function has a cpdef tag with a void specifying a no return in this case.  This is because the results are passed by reference in the lists posn and cash
  3. Inside the function we also specify the types of all variables we will be using before jumping into the body of the function.
  4. The rest is standard python.

In terms of building this .pyx file we need a setup.py file which is in the repo.  The details of providing the correct C/C++ toolchain is left for the reader to check in the PSF website.  But it’s pretty straightforward.

Python Backtesting Results

How do the results stack up?

Here is the table again with the Cython results!

ImplementationTime for RSI2 Backtests
Python – DataFrame, date indexing7.3 s
Python – DataFrame, iterrows1.3s
Python – DataFrame, itertuples0.03s
Python – Lists0.003s
Python -- Cython no Type Hints0.0015s
Python -- Cython with Type Hints0.00056s
Java0.00005s
C0.00002s

As promised the Cython implementation gets down to 0.00056, with type hints included, boosting your Python backtesting!

This is a pretty awesome performance reduction even if you compare it only to the pure list implementation.

It appears that running simulations over many different configurations won’t be as long a wait or as psychologically tortuous, as it looked to be at the start!

Conclusion

It is true that you shouldn’t be obsessed with speed.  But sometimes it’s definitely worthwhile, especially when it boils down to the “research” loop: from idea to result and back to idea.  If the delays are significant, the psychology is different, and you’ll have an additional obstacle to overcome.

The big advantage here is that ALL the other modules in the Python eco-system are at your fingertips.  It might be worthwhile to compare the Python implementations to the C and Java ones in the repo, just to remind yourselves how much we’ve progressed in the last 40 years in terms of programming languages!

Both Part 1 and this second part are much more computer / programming focused than usual.  How does this relate to trading?  It does in as far as if you lack the tools to test and analyze, you’ll be stabbing in the dark.  And given that Python is the go-to language, it’s worthwhile being able to perform fast backtesting with it.

Filed Under: Trading Strategies Tagged With: Backtest, Python

Subscribe to Get the One Exercise which will Improve Your Foreign Exchange Trading Straight Away

  • Get rid of your fear of losing
  • Learn to stick to the rules
  • Achieve a Zen like state when you trade

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Browse Topics

Tags

ADP AUDUSD Backtest Bid Ask Bonds Breakout CHFJPY Commodities ECB Economic Indicators Economic Releases Emerging Markets Engulfing Candles Equities EURCHF EURSEK EURUSD FOMC FX G10 GBPUSD HFT Kelly Market Timing Mean Reversion Moon News Events News Trading NFP ORB Performance Measures PPP Python Range Expansion Recession Risk Parity Sharpe Ratio Slippage Solar Eclipse SP500 Trading Systems USD USDCAD USDCHF USDJPY

Subscribe to Get the One Exercise which will Improve Your Foreign Exchange Trading Straight Away

  • Get rid of your fear of losing
  • Learn to stick to the rules
  • Achieve a Zen like state when you trade

Search Site

Testimonials

The bar in trader education has been set and I highly recommend anyone wishing to pursue a career in trading to invest in this program.         Jehan Jabar
One-on-One Coaching
Very well structured, interesting and insightfull
CASS Seminar
I really do rate everything with a “5.” Very good talk
CASS Seminar
Great teaching style. Very engaging. Material was presented in interesting manner and focused more on real life than theory.
CASS Seminar
Excellent, funny. Very knowledgeable!
CASS Seminar
Very clear, engaging and insightful.
CASS Seminar
Quick thank you for the suggestion to look at shorting the CHFJPY. Just closed at 122 pips profit. With only 9 pips Max loss. Nice.  Michele Russell
One-on-One Coaching
Very insightful from an intelligent and talented individual. Life changing.
CASS Seminar

© 2023 · FXMasterCourse · Privacy Policy · Terms & Conditions · Earnings Disclaimer

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish.Accept Read More
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT