The Alternative Investment Management Association

Alternative Investment Management Association Representing the global hedge fund industry

Algorithmic Execution: Does backtesting really help?

Nicholas Hallam

TraderServe Limited

Q3 2008

Algorithmic strategies are being used increasingly to execute orders in electronic financial markets. Algorithmic execution strategies break up an order, which is large in comparison with available volume (or bid), into manageable slices which are then worked upon over time to reduce the total cost of execution.

What is the algo objective?

Historically, in the equity markets, execution performance has been measured against a post-trade benchmark such as VWAP (the volume-weighted average price in the market over the day or period). The attraction of such a benchmark is that it indicates how well one market participant has performed in comparison with the rest of the market. However, even if the VWAP benchmark could be relied upon to give a fair picture of how well an algorithm or trader has performed compared with the market, it does not begin to address the question of whether that market VWAP itself was more costly than it needed to be.

More recently, there has been a trend towards the use of pure pre-trade measures and in particular to what is called “implementation shortfall” – the slippage between the price current at the generation of the trading decision and the volume-weighted average execution price actually achieved in the market. Following this trend we are disposed to regard “a good execution strategy” as one that produces competitive numbers for this measure.

Backtesting as a means of improvement

Backtesting is a venerable tradition in automated trading. Even before the advent of electronic markets, systematic traders evaluated trading strategies on the basis of historical data. This has proved fruitful for some, though there are a number of well-known pitfalls:

• The tendency to “fit” to historical data (over-optimisation);
• the vulnerability to the assumption that the future will be sufficiently similar to the past (stationarity);
• the dependence on realistic assumptions about the cost of execution (slippage).

Today’s algorithmic trading environments facilitate backtesting and allow an execution strategy to be evaluated against historical data. Once tested, modifications can be made to the strategy – e.g., rules added, thresholds changed, etc., and the revised model backtested. If the performance is improved, there would appear to be a rationale for preferring the new strategy. At least that is the idea …

However, the old pitfalls of backtesting have not gone away.

Fitting will always be a problem. Even with low parameter strategies and large volumes of historical data, it is still possible (and almost irresistible for the unwary) to greatly over-estimate the efficacy of a strategy – particularly if excessive use is made of backtesting and if no clear separation is made between in-sample training data and out-of-sample testing data. The more you look at data, the more patterns you will appear to find but the more of these will be spurious. This is really a law of statistics but it is something that most modellers learn only from brutal experience in the market when actual returns prove very different from the expected returns based on backtesting.

Additionally, the assumption that the future will be like the past is particularly dangerous in connection with algorithmic execution. Market microstructural changes are much more common than macrostructural ones and these can cause an execution strategy to come to grief. For the sake of illustration, let us suppose we have a strategy with opportunistic elements where trades are generated because the strategy is exploiting inefficiency in the market. It may look like a world-beater on historical data but to what extent can we rely on it? It is important not to lose sight of the fact that other modellers are capable of identifying similar opportunities. If they incorporate features designed to exploit the same inefficiencies in their strategies, then our own strategy will be taking part in a competition that it had not seen in history. Its performance is bound to be adversely affected.

These are not the worst problems. The fundamental problem of drawing conclusions about execution strategies from backtesting results is that you cannot tell from historical data how the market would have behaved if you had been trading. The historical data used for simulation is of the market without your interaction. There is no warrant to believe that the market would behave the same had you been participating. In fact there is every reason to think it would behave differently. Each order placed in the market changes the balance in weight between buying and selling interest and thus affects price formation. This impact is something that can be mitigated by judicious strategy selection but it cannot be entirely removed and should not be overlooked.

When trading in any substantial size – and this could be little more than one or two percent of the daily volume – the trader expects to move the market and, moreover, to do so to an extent which is likely to dwarf any small improvements achievable on market VWAP. The key question for the execution strategy builder should then be this: How do I minimise the impact of my trades? Or more precisely, how do I minimise the implementation shortfall of which the market impact is a sizeable part? This is a question for transaction cost analysis, not backtesting.

Transaction Cost Analysis

The key to improving algorithmic execution lies in analysis of the actual impact of a strategy’s order slices on the underlying market. Designing and operating high-frequency real-time trading environments tells one that micro-level analysis of actual order impact on a slice-by-slice basis is at least as important as backtesting.

A strategy can be constructed out of numerous substrategies exploiting different aspects of market behaviour. Every time an order slice is placed in the market, the slice details are stored along with information about which substrategywas responsible for the generation of that slice Historical data can then be retrieved along with the saved slices, and the market movement subsequent to the placing of the slice is measured. The data is of course very noisy, since our order is only one among many factors responsible for price movement: Nevertheless, dynamic patterns are detectable over thousands of slices and it is possible to build up a picture of the dependence of market impact on size, substrategy, time of day and any other factor deemed significant. In response to the analysis, the strategy can be modified, shifting greater emphasis onto the substrategies that have historically leaned least heavily on the market and perhaps avoiding certain times of day and orders larger than a certain size.
One should not assume that the markets will continue to behave exactly as they have in the past. Indeed, one should expect them to change and for some of those changes to happen virtually overnight, especially as new algorithms come online. The Transaction Cost Analysis must be ongoing; its aim, in addition to improving execution performance, is to identify secular microstructural shifts in market responses to order slices. The execution strategies may need to be modified very quickly to continue performing acceptably.

In all this, backtesting has a very minor role to play.

Back to Listing

Main Menu

  1. Home
  2. About
    1. Our Core Objectives
    2. AIMA's Policy Principles
    3. Meet the team
    4. AIMA Council
    5. Global Network
    6. Sponsoring Members
    7. Global Partners
    8. FAQs
    9. Opportunities at AIMA
  3. Join AIMA
    1. Benefits of Membership
    2. Membership Fees
    3. Application form
  4. Members
    1. AIMA Annual Reports
    2. AIMA Governance
    3. AIMA Logo
      1. Policy note
    4. AIMA Members' List
    5. AIMA Review of the Year
    6. Committees and Working Groups
    7. Weekly News
    8. Update Profile
  5. Investors
    1. AIMA Investor Services
    2. AIMA Members' List
    3. Investor Steering Committee
    4. Update Profile
  6. Regulation
    1. AIMA's Policy Principles
    2. Asset Management Regulation
      1. EU Asset Management Regulation
        1. AIFMD
        2. European Capital Markets Regulation
        3. MiFID / MiFIR
        4. UCITS
          1. ETFs and Structured UCITS
        5. Venture Capital
        6. Shareholder Rights Directive
      2. US Hedge Fund Adviser Regulations
        1. Registration and Reporting
        2. Incentive-Based Compensation
        3. JOBS Act
      3. Asia Pacific Asset Management regulation
      4. Other Jurisdictions’ Asset Management Regulation
      5. Regulation of NBFCs / SIFIS
      6. Supervision
        1. UK regulatory reform
        2. European Supervisory Authorities
        3. US Self-Regulatory Organisations
      7. Remuneration
        1. UK
        2. US
        3. CRD IV and CRR
        4. AIFMD
        5. MiFID
      8. Shadow Banking
      9. Volcker Rule
      10. Other
    3. Markets Regulation
      1. Bank/Capital Regulation
        1. Capital Requirements Directive
        2. EU Bank Structural Reforms
      2. Derivatives/Clearing
        1. EMIR
        2. MiFID / MiFIR
        3. MAD / MAR
        4. Dodd-Frank Act Title VII
        5. Hong Kong
        6. IOSCO
        7. Singapore
      3. High Frequency Trading
        1. ESMA Guidelines
        2. MiFID / MiFIR
        3. MAD / MAR
        4. Flash Crash
        5. IOSCO
        6. Germany
        7. CFTC Automated Trading
      4. Insurance Regulation
        1. Solvency II
      5. Market Abuse
        1. MAD / MAR
        2. Indices as Benchmarks
      6. Position Limits
        1. MiFID / MiFIR
        2. CFTC Position Limits
      7. Resolution of Financial Institutions
        1. Europe
          1. EU Bank Recovery and Resolution Directive
          2. EU Non-Bank Recovery and Resolution
        2. CPSS-IOSCO
        3. Financial Stability Board
        4. UK
        5. USA
      8. Short Selling
        1. EU Short Selling Regulation
        2. Hong Kong Short Selling Regulation
        3. US Short Selling Regulation
        4. Short Selling Bans
      9. Securities Settlement
      10. Shadow Banking
        1. International Shadow Banking
        2. EU Shadow Banking
      11. Trading
        1. MiFID / MiFIR
        2. Dodd-Frank Act
    4. Tax Affairs
      1. EU Savings Directive
      2. FAIFs and FINROFs
      3. FATCA
      4. FIN 48 and IAS 12
      5. Financial Transaction Tax
      6. Investment Manager Exemption
      7. UK Offshore Funds Regime
      8. Other
    5. Search
    6. Resources
      1. Guidance Notes
      2. Jurisdictional Guides
      3. Noticeboard
        1. BEPS
        2. Dealing Commission
        3. Derivatives
        4. FATCA
        5. FTT
        6. High Frequency Trading
        7. MiFID / MiFIR
        8. Other Hot Asset Management Topics
        9. Other Hot Markets Topics
        10. Position Limits
        11. Trading
        12. UK Partnership Tax Review
        13. Volcker Rule
      4. Hedge Fund Manager Training
      5. Quarterly Regulatory Update
      6. Webinar Programme
      7. Regulatory Compliance Association
        1. About the Regulatory Compliance Association
        2. RCA Curricula and initiatives for alternative investment firms
        3. Meet the regulators and Sr. Fellows
  7. Education
    1. AIMA Journal
      1. Recent issues
      2. Search AIMA Journal articles
      3. AIMA Journal Archive
    2. The Extra Mile: Partnerships between Hedge Funds and Investors
    3. 'Apples and apples' - How to better understand hedge fund performance
    4. AIMA/KPMG reports on state of global hedge fund industry
    5. Roadmap to Hedge Funds
    6. ‘Capital Markets and Economic Growth – Long-Term Trends and Policy Challenges'
    7. Guides for institutional investment
    8. Industry-standard DDQs
    9. Sound Practices
    10. Industry Guides
    11. CAIA
      1. FAI
    12. Regulatory Compliance Association
      1. About the Regulatory Compliance Association
      2. RCA Curricula and initiatives for alternative investment firms
      3. Meet the regulators and Sr. Fellows
    13. Services to Start-up Managers
    14. Useful Websites
    15. 'The Alternative Answer'
    16. Glossary
    17. AIMA's Investor Steering Committee Paper
  8. Events
    1. AIMA Events
    2. AIMA webinars
    3. Industry events
  9. Media
    1. Press Releases & Statements
    2. AIMA's blog
    3. Media Coverage
      1. Articles by AIMA
        1. Archive
      2. AIMA in the news
      3. Video interviews
      4. Industry news
    4. Media Contacts
    5. Press Materials

Sub Menu

  1. Education
    1. AIMA Journal
    2. Bibliography
    3. CAIA Designation
    4. Research
    5. Roadmap to Hedge Funds
    6. AIMA's Investor Steering Committee Paper
    7. Glossary
  2. Regulatory, Tax, Policy & Government Affairs
    1. AIMA Position Papers
    2. AIMA Responses
      1. Australian Tax Office
      2. Authority for the Financial Markets
      3. Committee of European Banking Supervisors
      4. Committee of European Securities Regulators
      5. Commodity Futures Trading Commission
      6. Dubai Financial Services Authority
      7. European Commission
      8. European Securities and Markets Authority
      9. Swiss Financial Market Supervisory Authority
      10. Financial Services Authority (UK)
      11. Financial Services and the Treasury Bureau
      12. Guernsey Financial Services Commission
      13. HM Revenue & Customs
      14. HM Treasury
      15. Independent Commission on Banking
      16. IOSCO
      17. Monetary Authority of Singapore
      18. Securities and Exchange Board of India
      19. Securities and Exchange Commission (USA)
      20. Securities and Futures Commission
      21. Singapore Exchange
      22. The Takeover Panel
      23. US House of Representatives / Senate
      24. Federal Deposit Insurance Corporation
      25. Financial Stability Oversight Council
      26. Financial Stability Board
      27. US Treasury
      28. Internal Revenue Service
      29. US Federal Reserve
      30. Financial Industry Regulatory Authority (FINRA)
      31. Council of European Union
      32. Hong Kong Exchanges and Clearing
      33. House of Lords
    3. AIMA Summaries
      1. CESR
      2. European Commission
      3. Financial Services Authority (UK)
      4. HM Revenue & Customs
      5. HM Treasury
      6. IOSCO
      7. Securities and Exchanges Commission
      8. FSOC
      9. CFTC
    4. Guidance Notes
    5. Jurisdictional Resource
    6. AIMA Noticeboard
      1. EU Directive on Alternative Investment Fund Managers
      2. FSA Remuneration Code
      3. Short Selling
      4. US Dodd-Frank Wall Street Reform and Consumer Protection Act
      5. UK Stewardship Code
      6. Securities Law Directive
      7. EU Directive on Alternative Investment Fund Managers - Level II
      8. EU Directive on Markets in Financial Instruments (MiFID)
      9. International Financial Centres
      10. Bribery Act
      11. Market Abuse Directive
      12. MF Global
      13. FATCA
      14. FTT
      15. Other Tax Issues
    7. AIMA Regulatory Update
  3. Sound Practices
    1. Due Diligence Questionnaires
    2. Guides to Sound Practices
  4. Start-Up Service Providers
  5. Useful Websites