FXMasterCourse

Trading Order Out of Chaos

  • Facebook
  • LinkedIn
  • RSS
  • Twitter
  • About
  • Trading Approach
  • Resources
  • Forex Tools
    • MT4 Code
    • Tick Data
    • Useful Tools
  • In the Media
  • Testimonials
  • Contact
  • Member Login
You are here: Home / Trading Strategies / Improving your Python Backtesting – From DataFrames to Cython [Part 2]

February 23, 2021 by Corvin Codirla Leave a Comment

Improving your Python Backtesting – From DataFrames to Cython [Part 2]

Improving your Python Backtesting – From DataFrames to Cython [Part 2]

In Part 1 we used simple Python to Improve our Backtesting times.  Starting out with DataFrames we took a simple RSI strategy over a period of 7000 days and reduced from 7.3 seconds (which is a complete joke; sorry Pandas!) down to 0.003 seconds by converting everything to lists. But if you recall the comparison table at the end of Part 1

ImplementationTime for RSI2 Backtests
Python – Lists0.003s
Java0.00005s
C0.00002s

there was still a great gap between the Python list implementation and a simple Java or C implementation by as much as a factor of 100!

We promised to get back to this and start looking at Cython as bridge between the two worlds of interpreted language and bare metal implementation. You still get to live in the Python world with all the fancy machine learning and graphing facilities and fast backtesting!

So, by how much can Cython improve our timings?  Here is a sneak peak, so you don’t have to plough through all the text: all the way down to 0.00056s, which is six times faster and actually pretty close to bare metal.

If you want to repeat these numbers [which will be of course highly dependent on the machine you run it out] you can grab the code at this github repository: FXMC/backtest.

You might say, so who gives?  Why squeeze it all the way down.  The answer is simple.  When you’re doing research and you want to try your idea over many assets and with lots of different parameters, to get a feel for how stable or random your results are, or even just generate a useful sample of P&L paths for your Monte Carlo analysis; well, you’d probably like to do that within a couple of minutes.  Not a couple of days!

Intro to Cython

The focus here is on what I did to get it running.  Show you the specifics, and then you can copy paste in your own work.

First what is Cython?  In a nutshell: it is a transpiler that takes annotated Python source [hence it’s a pyx file and not a py file] and converts it to a C file.  This file is then compiled as a module that Python can load. 

The speed happens because your stuff has now been ported to C, and the load of the data is the only bottle neck. 

Why? Because when you run your stuff in Python, it has to be passed to your C(P)ython module and that conversion of data from one language to another takes time.  Specifically, because you’re going from a very lenient data language like Python to a very stringent data language like C.

However, if you include specific data annotations to your .pyx file you can bypass a lot of machinery.

It’s the annotations that make the magic.  Without them, it does really look like you’re not adding much to the speed improvement.

So, that’s what we will focus on from a syntax perspective (for completeness, the repo has the other non-annotated version as well).

Cython Implementation

Here it goes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cpdef void run_strategy(list close, list ma_long, list ma_short,
                       list rsi_n, float lower_rsi, int start_indx,
                       list posn, list cash):
    """Cythonized function with explicit type definitions"""
    cdef bint long_posn = False
    cdef int idx = 0
    cdef double shares = 0.0

    for idx in range(start_indx, len(close)):
        posn[idx] = posn[idx-1]
        cash[idx] = cash[idx-1]

        if long_posn:
            if close[idx] > ma_short[idx]:
                long_posn = False
                shares = posn[idx]
                posn[idx] = 0
                cash[idx] = cash[idx] + shares * close[idx]

        if not long_posn:
            if (close[idx] > ma_long[idx]) and (rsi_n[idx] < lower_rsi):
                long_posn = True
                shares = math.floor(cash[idx] / close[idx])
                posn[idx] = shares
                cash[idx] = cash[idx] - shares * close[idx]

Let’s go through how this differs from normal python code:

  1. First on the import side: nothing, we are not importing a thing
  2. The function has a cpdef tag with a void specifying a no return in this case.  This is because the results are passed by reference in the lists posn and cash
  3. Inside the function we also specify the types of all variables we will be using before jumping into the body of the function.
  4. The rest is standard python.

In terms of building this .pyx file we need a setup.py file which is in the repo.  The details of providing the correct C/C++ toolchain is left for the reader to check in the PSF website.  But it’s pretty straightforward.

Python Backtesting Results

How do the results stack up?

Here is the table again with the Cython results!

ImplementationTime for RSI2 Backtests
Python – DataFrame, date indexing7.3 s
Python – DataFrame, iterrows1.3s
Python – DataFrame, itertuples0.03s
Python – Lists0.003s
Python -- Cython no Type Hints0.0015s
Python -- Cython with Type Hints0.00056s
Java0.00005s
C0.00002s

As promised the Cython implementation gets down to 0.00056, with type hints included, boosting your Python backtesting!

This is a pretty awesome performance reduction even if you compare it only to the pure list implementation.

It appears that running simulations over many different configurations won’t be as long a wait or as psychologically tortuous, as it looked to be at the start!

Conclusion

It is true that you shouldn’t be obsessed with speed.  But sometimes it’s definitely worthwhile, especially when it boils down to the “research” loop: from idea to result and back to idea.  If the delays are significant, the psychology is different, and you’ll have an additional obstacle to overcome.

The big advantage here is that ALL the other modules in the Python eco-system are at your fingertips.  It might be worthwhile to compare the Python implementations to the C and Java ones in the repo, just to remind yourselves how much we’ve progressed in the last 40 years in terms of programming languages!

Both Part 1 and this second part are much more computer / programming focused than usual.  How does this relate to trading?  It does in as far as if you lack the tools to test and analyze, you’ll be stabbing in the dark.  And given that Python is the go-to language, it’s worthwhile being able to perform fast backtesting with it.

Filed Under: Trading Strategies Tagged With: Backtest, Python

Subscribe to Get the One Exercise which will Improve Your Foreign Exchange Trading Straight Away

  • Get rid of your fear of losing
  • Learn to stick to the rules
  • Achieve a Zen like state when you trade

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Browse Topics

Tags

ADP AUDUSD Backtest Bid Ask Bonds Breakout CHFJPY Commodities ECB Engulfing Candles EURCHF EURGBP EURJPY EURUSD FOMC FX GBPUSD Gold HFT Kelly Market Timing Moon News Events News Trading NFP NZDUSD ORB PBOC Performance Measures PPP Python Range Expansion Recession Risk Parity Seasonality Sharpe Ratio Slippage Solar Eclipse SP500 Tick Volume Trading Systems USD USDCAD USDCHF USDJPY

Subscribe to Get the One Exercise which will Improve Your Foreign Exchange Trading Straight Away

  • Get rid of your fear of losing
  • Learn to stick to the rules
  • Achieve a Zen like state when you trade

Search Site

Testimonials

Would you like to build consistently profitable trading systems from scratch? This article series does exactly that.  Here are the steps which we’ll work through:
  • Define the underlying principles of a successful trading approach
  • Determine the assets which obey these principles
  • Determine rule sets for trading these assets
  • Work out a straightforward portfolio construction method
That’s the theory. In this article-series we’ll go a step further and look at how to implemen… Read more
Building Consistently Profitable Trading Systems – Principles of a Successful Trading Approach, Part 1
Python Backtesting

Intro

Backtesting is every systematic trader’s basic tool. And Python is becoming the lingua franca of programming. So putting Python into Backtesting to get fast results should be possible!

Yes and no!

In this article, we’ll cover how to really improve your Python backtesting and boost your speeds by several orders of magnitude!

Welcome
Equity Mean Reversion
At last!  We’ve made it to the mean-reverting part of the series, starting with our focus on equities and their mean-reversion habits. So, here’s the deal.  We’re going to keep it simple, just like in the previous three posts, and start from the ground up.  Over the following series we’ll culminate in a simple, straightforward (and well known) system that still works. In detail for this article:
  • What do we mean by mean-reversion and how can we measure it?
  • What are some natural meth… Read more
Equities and Their Mean Reversion Habits
Short update on AUDUSD and the AIG Construction index on the 7th January, 2016, 22:30 GMT. There is a reason ForexFactory labelled this as yellow (which seems to be the lowest importance rating they give to news releases). The market was completely comatose, both from a tick count per minute basis, as well as from the perspective of the bid/ask: was wide / stayed wide / couldn’t be bothered. And of course AUDUSD stayed where it was as well. However, both of these are poor indications to dra… Read more
More on AUDUSD dynamics and News Events: what happens when the event is not important?
More Money Than God: Hedge Funds and the Making of the New Elite is simply WOW! All I can say is WOW! Mallaby apparently held in excess of 1000 interviews and was given access to emails by some of the main players in the history of hedge funds. If you have read the Market Wizards trilogy, then this book actually fleshes out some of these individuals and provides the historical setting. The book is made up of 14 Chapters, which track the history of hedge funds both chronologically and stylist… Read more
Book Review: More Money Than God
Many people ask if there are any particular technical setups that I like to look at and what the best way to trade them is. My all-time favorite heavily involves Support and Resistance levels, and using these as breakout levels. The art lies in finding them and then managing trades appropriately. In this article we’ll focus on using Bullish and Bearish Engulfing Bars to identify support and resistance levels, and present the analysis that goes behind implementing such a system. Implementati… Read more
Bullish & Bearish Engulfing Bars (Part I)
Taking Control of Your Trading Numbers is Taking Control of Your Profits
Taking Control of Your Trading Numbers – that means actually being able to replicate the results that others produce.  It’s the only way to be sure that you know what’s going on. Well, many people got back after last week’s article asking about the details of the calculations as well as about some of the more technical jargon:
  • What do I mean by asset?
  • What’s an asset bias?
  • What does it mean to risk-adjust returns?
  • How can I go about calculating the charts in the article?
  • How … Read more
Taking Control of Your Trading Numbers

© 2022 · FXMasterCourse · Privacy Policy · Terms & Conditions · Earnings Disclaimer

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish.Accept Read More
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.

SAVE & ACCEPT