The Alternative Investment Management Association

Alternative Investment Management Association Representing the global hedge fund industry

Algorithmic Execution: Does backtesting really help?

Nicholas Hallam

TraderServe Limited

Q3 2008

Algorithmic strategies are being used increasingly to execute orders in electronic financial markets. Algorithmic execution strategies break up an order, which is large in comparison with available volume (or bid), into manageable slices which are then worked upon over time to reduce the total cost of execution.

What is the algo objective?

Historically, in the equity markets, execution performance has been measured against a post-trade benchmark such as VWAP (the volume-weighted average price in the market over the day or period). The attraction of such a benchmark is that it indicates how well one market participant has performed in comparison with the rest of the market. However, even if the VWAP benchmark could be relied upon to give a fair picture of how well an algorithm or trader has performed compared with the market, it does not begin to address the question of whether that market VWAP itself was more costly than it needed to be.

More recently, there has been a trend towards the use of pure pre-trade measures and in particular to what is called “implementation shortfall” – the slippage between the price current at the generation of the trading decision and the volume-weighted average execution price actually achieved in the market. Following this trend we are disposed to regard “a good execution strategy” as one that produces competitive numbers for this measure.

Backtesting as a means of improvement

Backtesting is a venerable tradition in automated trading. Even before the advent of electronic markets, systematic traders evaluated trading strategies on the basis of historical data. This has proved fruitful for some, though there are a number of well-known pitfalls:

• The tendency to “fit” to historical data (over-optimisation);
• the vulnerability to the assumption that the future will be sufficiently similar to the past (stationarity);
• the dependence on realistic assumptions about the cost of execution (slippage).

Today’s algorithmic trading environments facilitate backtesting and allow an execution strategy to be evaluated against historical data. Once tested, modifications can be made to the strategy – e.g., rules added, thresholds changed, etc., and the revised model backtested. If the performance is improved, there would appear to be a rationale for preferring the new strategy. At least that is the idea …

However, the old pitfalls of backtesting have not gone away.

Fitting will always be a problem. Even with low parameter strategies and large volumes of historical data, it is still possible (and almost irresistible for the unwary) to greatly over-estimate the efficacy of a strategy – particularly if excessive use is made of backtesting and if no clear separation is made between in-sample training data and out-of-sample testing data. The more you look at data, the more patterns you will appear to find but the more of these will be spurious. This is really a law of statistics but it is something that most modellers learn only from brutal experience in the market when actual returns prove very different from the expected returns based on backtesting.

Additionally, the assumption that the future will be like the past is particularly dangerous in connection with algorithmic execution. Market microstructural changes are much more common than macrostructural ones and these can cause an execution strategy to come to grief. For the sake of illustration, let us suppose we have a strategy with opportunistic elements where trades are generated because the strategy is exploiting inefficiency in the market. It may look like a world-beater on historical data but to what extent can we rely on it? It is important not to lose sight of the fact that other modellers are capable of identifying similar opportunities. If they incorporate features designed to exploit the same inefficiencies in their strategies, then our own strategy will be taking part in a competition that it had not seen in history. Its performance is bound to be adversely affected.

These are not the worst problems. The fundamental problem of drawing conclusions about execution strategies from backtesting results is that you cannot tell from historical data how the market would have behaved if you had been trading. The historical data used for simulation is of the market without your interaction. There is no warrant to believe that the market would behave the same had you been participating. In fact there is every reason to think it would behave differently. Each order placed in the market changes the balance in weight between buying and selling interest and thus affects price formation. This impact is something that can be mitigated by judicious strategy selection but it cannot be entirely removed and should not be overlooked.

When trading in any substantial size – and this could be little more than one or two percent of the daily volume – the trader expects to move the market and, moreover, to do so to an extent which is likely to dwarf any small improvements achievable on market VWAP. The key question for the execution strategy builder should then be this: How do I minimise the impact of my trades? Or more precisely, how do I minimise the implementation shortfall of which the market impact is a sizeable part? This is a question for transaction cost analysis, not backtesting.

Transaction Cost Analysis

The key to improving algorithmic execution lies in analysis of the actual impact of a strategy’s order slices on the underlying market. Designing and operating high-frequency real-time trading environments tells one that micro-level analysis of actual order impact on a slice-by-slice basis is at least as important as backtesting.

A strategy can be constructed out of numerous substrategies exploiting different aspects of market behaviour. Every time an order slice is placed in the market, the slice details are stored along with information about which substrategywas responsible for the generation of that slice Historical data can then be retrieved along with the saved slices, and the market movement subsequent to the placing of the slice is measured. The data is of course very noisy, since our order is only one among many factors responsible for price movement: Nevertheless, dynamic patterns are detectable over thousands of slices and it is possible to build up a picture of the dependence of market impact on size, substrategy, time of day and any other factor deemed significant. In response to the analysis, the strategy can be modified, shifting greater emphasis onto the substrategies that have historically leaned least heavily on the market and perhaps avoiding certain times of day and orders larger than a certain size.
One should not assume that the markets will continue to behave exactly as they have in the past. Indeed, one should expect them to change and for some of those changes to happen virtually overnight, especially as new algorithms come online. The Transaction Cost Analysis must be ongoing; its aim, in addition to improving execution performance, is to identify secular microstructural shifts in market responses to order slices. The execution strategies may need to be modified very quickly to continue performing acceptably.

In all this, backtesting has a very minor role to play.

Back to Listing

Main Menu

  1. Home
  2. About
    1. Our Core Objectives
    2. AIMA's Policy Principles
    3. Meet the team
    4. AIMA Council
    5. Global Network
    6. Sponsoring Partners
    7. Opportunities at AIMA
    8. AIMA’s 25th anniversary in 2015
  3. Join AIMA
    1. Benefits of Membership
    2. Membership Fees
    3. Application form
  4. Members
    1. AIMA DDQs
    2. AIMA Annual Reports
    3. AIMA Governance
    4. AIMA Logo
      1. Policy note
    5. AIMA Members' List
    6. AIMA Review of the Year
    7. Committees and Working Groups
    8. Weekly News
    9. Update Profile
  5. Investors
    1. AIMA Investor Services
    2. AIMA Members' List
    3. Investor Steering Committee
  6. Regulation
    1. Asset Management Regulation
      1. EU Asset Management Regulation
        1. AIFMD
        2. European Capital Markets Regulation
        3. MiFID / MiFIR
        4. UCITS
        5. European Venture Capital Directive
        6. Shareholder Rights Directive
        7. European Long Term Investment Fund Regulation
        8. Loan Origination Funds
        9. Capital Raising
        10. AIFMD-Related Events
      2. US Hedge Fund Adviser Regulations
        1. Registration and Reporting
        2. Incentive-Based Compensation
        3. JOBS Act
      3. Asia Pacific Asset Management regulation
      4. Other Jurisdictions’ Asset Management Regulation
      5. Private Placement Regime
        1. Canada
        2. Dubai
        3. Finland
        4. Germany
        5. Hong Kong
        6. Japan
        7. Saudi Arabia
        8. Sweden
        9. United Arab Emirates
      6. Systemically Important Financial Institutions ('SIFIs')
      7. Remuneration
        1. UK
        2. United States
        3. CRD IV and CRR
        4. AIFMD
        5. MiFID
      8. Shadow Banking
      9. Volcker Rule
      10. Other
      11. Systemic Risk Reporting
      12. Dealing Commission
      13. Corporate Governance
      14. Securitisation
    2. Markets Regulation
      1. Bank/Capital Regulation
        1. Capital Requirements Directive
        2. EU Bank Structural Reforms
      2. Capital Markets Union
      3. Derivatives/Clearing
        1. EMIR
        2. MiFID II / MiFIR - Derivatives
        3. MAD / MAR
        4. Dodd-Frank Act Title VII
        5. Hong Kong
        6. IOSCO
        7. Singapore
      4. High Frequency Trading
        1. EU automated trading
          1. ESMA Guidelines
          2. Germany
          3. MiFID II / MiFIR - HFT
        2. US automated trading
          1. SEC Regulation SCI
          2. CFTC Automated Trading
        3. IOSCO
        4. Flash Crash
      5. Insurance Regulation
        1. Solvency II
      6. Market Abuse
        1. MAD / MAR
        2. Indices as Benchmarks
      7. Position Limits
        1. MiFID II - Commodities
        2. CFTC Position Limits
      8. Resolution of Financial Institutions
        1. Europe
          1. EU Bank Recovery and Resolution Directive
          2. EU Non-Bank Recovery and Resolution
        2. CPSS-IOSCO
        3. Financial Stability Board
        4. UK
        5. USA
      9. Shadow Banking
        1. International Shadow Banking
        2. EU Shadow Banking - SFT reporting & transparency
      10. Short Selling
        1. EU Short Selling Regulation
        2. Hong Kong Short Selling Regulation
        3. US Short Selling Regulation
      11. Trading
        1. Dodd-Frank Act
        2. MiFID Portal
        3. REMIT
        4. Securities Settlement
    3. Tax Affairs
      1. Automatic Exchange of Information (AEOI)
        1. FATCA
        2. EU - AEFI
        3. OECD - Global Standard on AEFI
      2. Australia - Investment Manager Regime (IMR)
      3. Base Erosion - Profit Shifting (BEPS)
      4. FIN 48 and IAS 12
      5. Financial Transaction Tax (FTT)
      6. UK Investment Management Exemption (IME)
      7. UK Offshore Funds Regime
      8. Other
    4. AIMA's Policy Principles
    5. Search
    6. Resources
      1. Guidance Notes
      2. Jurisdictional Guides
      3. Noticeboard
        1. AEOI: FATCA and other regimes
        2. AIFMD
        3. Bank/Capital Regulation (including NSFR)
        4. BEPS
        5. CFTC Registration and Exemptions
        6. Corporate Governance
        7. Dealing Commission
        8. Derivatives
        9. FTT
        10. High Frequency Trading
        11. MiFID / MiFIR
        12. Other Hot Asset Management Topics
        13. Other Hot Markets Topics
        14. Other Hot Tax Topics
        15. Position Limits
        16. Trading
        17. UCITS
        18. UK Partnership Tax Review
        19. US State and Local Taxes
        20. Volcker Rule
      4. Hedge Fund Manager Training
      5. Quarterly Regulatory Update
      6. Webinar Programme
      7. Regulatory Compliance Association
        1. About the Regulatory Compliance Association
        2. RCA Curricula and initiatives for alternative investment firms
        3. Meet the regulators and Sr. Fellows
  7. Education
    1. Research
      1. AIMA Research
      2. Industry research
      3. Search research documents
    2. "The Case for Hedge Funds"
      1. Global Hedge Fund Industry Paper: The value of our industry
      2. The Value of the Hedge Fund Industry to Investors, Markets and the Broader Economy: Research commissioned by AIMA and KPMG
      3. The Evolution of an Industry: KPMG/AIMA Global Hedge Fund Survey
      4. Contributing to Communities: A global review of charitable and philanthropic activities by the hedge fund industry
      5. Beyond 60-40: The evolving role of hedge funds in institutional investor portfolios
      6. The Cost of Compliance: Global hedge fund survey by AIMA, MFA and KPMG
      7. Capital Markets and Economic Growth: Long-term trends and policy challenges
      8. Apples and Apples: How to better understand hedge fund performance
      9. The Extra Mile: Partnerships between hedge funds and investors
      10. Key articles by AIMA on the case for hedge funds
    3. AIMA Journal
      1. Recent issues
      2. Search AIMA Journal articles
      3. AIMA Journal Archive
    4. AIMA Guides to Sound Practices
    5. AIMA guides for institutional investors
    6. CAIA Association pages
      1. Fundamentals of Alternative Investments
    7. Regulatory Compliance Association pages
      1. About the Regulatory Compliance Association
      2. RCA Curricula and initiatives for alternative investment firms
      3. Meet the regulators and Sr. Fellows
    8. Certified Investment Fund Director programme
    9. Services to Start-up Managers
    10. Glossary
  8. Events
    1. AIMA Events
      1. AIMA Annual Conference
        1. AIMA 25th Anniversary AGM & Annual Conference
      2. AIMA's Global Policy and Regulatory Forum
        1. 2015 Forum - Review
        2. 2015 Forum - Photos
        3. 2015 Forum - Agenda
        4. 2015 Forum - Sponsors and Supporting Organisations
    2. AIMA webinars
    3. Industry events
  9. Media
    1. Press Releases & Statements
    2. AIMA's blog
    3. Media Coverage
      1. Articles by AIMA
        1. Archive
      2. AIMA in the news
      3. Video interviews
      4. Industry news
    4. Media Contacts
    5. Press Materials

Sub Menu

  1. Education
    1. AIMA Journal
    2. Bibliography
    3. CAIA Designation
    4. Research
    5. Roadmap to Hedge Funds
    6. AIMA's Investor Steering Committee Paper
    7. Glossary
  2. Regulatory, Tax, Policy & Government Affairs
    1. AIMA Position Papers
    2. AIMA Responses
      1. Australian Tax Office
      2. Authority for the Financial Markets
      3. Committee of European Banking Supervisors
      4. Committee of European Securities Regulators
      5. Commodity Futures Trading Commission
      6. Dubai Financial Services Authority
      7. European Commission
      8. European Securities and Markets Authority
      9. Swiss Financial Market Supervisory Authority
      10. Financial Services Authority (UK)
      11. Financial Services and the Treasury Bureau
      12. Guernsey Financial Services Commission
      13. HM Revenue & Customs
      14. HM Treasury
      15. Independent Commission on Banking
      16. IOSCO
      17. Monetary Authority of Singapore
      18. Securities and Exchange Board of India
      19. Securities and Exchange Commission (USA)
      20. Securities and Futures Commission
      21. Singapore Exchange
      22. The Takeover Panel
      23. US House of Representatives / Senate
      24. Federal Deposit Insurance Corporation
      25. Financial Stability Oversight Council
      26. Financial Stability Board
      27. US Treasury
      28. Internal Revenue Service
      29. US Federal Reserve
      30. Financial Industry Regulatory Authority (FINRA)
      31. Council of European Union
      32. Hong Kong Exchanges and Clearing
      33. House of Lords
    3. AIMA Summaries
      1. CESR
      2. European Commission
      3. Financial Services Authority (UK)
      4. HM Revenue & Customs
      5. HM Treasury
      6. IOSCO
      7. Securities and Exchanges Commission
      8. FSOC
      9. CFTC
    4. Guidance Notes
    5. Jurisdictional Resource
    6. AIMA Noticeboard
      1. EU Directive on Alternative Investment Fund Managers
      2. FSA Remuneration Code
      3. Short Selling
      4. US Dodd-Frank Wall Street Reform and Consumer Protection Act
      5. UK Stewardship Code
      6. Securities Law Directive
      7. EU Directive on Alternative Investment Fund Managers - Level II
      8. EU Directive on Markets in Financial Instruments (MiFID)
      9. International Financial Centres
      10. Bribery Act
      11. Market Abuse Directive
      12. MF Global
      13. FATCA
      14. FTT
      15. Other Tax Issues
    7. AIMA Regulatory Update
  3. Sound Practices
    1. Due Diligence Questionnaires
    2. Guides to Sound Practices
  4. Start-Up Service Providers
  5. Useful Websites