AIMA
The Alternative Investment Management Association
Welcome to AIMA
Alternative Investment Management Association
Algorithmic Execution: Does backtesting really help?
Nicholas Hallam
TraderServe Limited
2008
Algorithmic strategies are being used increasingly to execute orders in electronic financial markets. Algorithmic execution strategies break up an order, which is large in comparison with available volume (or bid), into manageable slices which are then worked upon over time to reduce the total cost of execution.
What is the algo objective?
Historically, in the equityThe net worth of a company. This represents the ownership interest of the shareholders (common and preferred) of a company. For this reason, shares are often known as equities. markets, execution performance has been measured against a post-trade benchmarkAny basis of measurement (security or index), used as a reference point, against which a comparison and evaluation of performance of an investment portfolio can be made. such as VWAP (the volume-weighted average price in the market over the day or period). The attraction of such a benchmarkAny basis of measurement (security or index), used as a reference point, against which a comparison and evaluation of performance of an investment portfolio can be made. is that it indicates how well one market participant has performed in comparison with the rest of the market. However, even if the VWAP benchmarkAny basis of measurement (security or index), used as a reference point, against which a comparison and evaluation of performance of an investment portfolio can be made. could be relied upon to give a fair picture of how well an algorithm or trader(1) A merchant involved in cash commodities; (2) a professional speculator who trades for his own account and who typically holds exchange trading privileges. has performed compared with the market, it does not begin to address the question of whether that market VWAP itself was more costly than it needed to be.
More recently, there has been a trendThe general direction, either upward or downward, in which prices have been moving. towards the use of pure pre-trade measures and in particular to what is called “implementation shortfall” – the slippageThe difference between the sample or target price for buying or selling an asset and the actual price at which the transaction takes place. between the price current at the generation of the trading decision and the volume-weighted average execution price actually achieved in the market. Following this trendThe general direction, either upward or downward, in which prices have been moving. we are disposed to regard “a good execution strategyThe particular investment process employed by a manager in the application of an investment style.” as one that produces competitive numbers for this measure.
Backtesting as a means of improvement
Backtesting is a venerable tradition in automated trading. Even before the advent of electronic markets, systematic traders evaluated trading strategies on the basis of historical data. This has proved fruitful for some, though there are a number of well-known pitfalls:
• The tendency to “fit” to historical data (over-optimisation);
• the vulnerability to the assumption that the future will be sufficiently similar to the past (stationarity);
• the dependence on realistic assumptions about the cost of execution (slippageThe difference between the sample or target price for buying or selling an asset and the actual price at which the transaction takes place.).
Today’s algorithmic trading environments facilitate backtesting and allow an execution strategyThe particular investment process employed by a manager in the application of an investment style. to be evaluated against historical data. Once tested, modifications can be made to the strategyThe particular investment process employed by a manager in the application of an investment style. – e.g., rules added, thresholds changed, etc., and the revised model backtested. If the performance is improved, there would appear to be a rationale for preferring the new strategyThe particular investment process employed by a manager in the application of an investment style.. At least that is the idea …
However, the old pitfalls of backtesting have not gone away.
Fitting will always be a problem. Even with low parameter strategies and large volumes of historical data, it is still possible (and almost irresistible for the unwary) to greatly over-estimate the efficacy of a strategyThe particular investment process employed by a manager in the application of an investment style. – particularly if excessive use is made of backtesting and if no clear separation is made between in-sample training data and out-of-sample testing data. The more you look at data, the more patterns you will appear to find but the more of these will be spurious. This is really a law of statistics but it is something that most modellers learn only from brutal experience in the market when actual returns prove very different from the expected returns based on backtesting.
Additionally, the assumption that the future will be like the past is particularly dangerous in connection with algorithmic execution. Market microstructural changes are much more common than macrostructural ones and these can cause an execution strategyThe particular investment process employed by a manager in the application of an investment style. to come to grief. For the sake of illustration, let us suppose we have a strategyThe particular investment process employed by a manager in the application of an investment style. with opportunistic elements where trades are generated because the strategyThe particular investment process employed by a manager in the application of an investment style. is exploiting inefficiency in the market. It may look like a world-beater on historical data but to what extent can we rely on it? It is important not to lose sight of the fact that other modellers are capable of identifying similar opportunities. If they incorporate features designed to exploit the same inefficiencies in their strategies, then our own strategyThe particular investment process employed by a manager in the application of an investment style. will be taking part in a competition that it had not seen in history. Its performance is bound to be adversely affected.
These are not the worst problems. The fundamental problem of drawing conclusions about execution strategies from backtesting results is that you cannot tell from historical data how the market would have behaved if you had been trading. The historical data used for simulation is of the market without your interaction. There is no warrantA contract that gives an investor the rights to purchase a security at a specific price (usually above the current price) on a future date. It is usually issued with a bond or preferred stock to provide additional incentive to the buyer. Warrants are similar to options contracts, but unlike options, they can stay in effect for a period ranging from a few years to eternity. to believe that the market would behave the same had you been participating. In fact there is every reason to think it would behave differently. Each order placed in the market changes the balance in weight between buying and selling interest and thus affects price formation. This impact is something that can be mitigated by judicious strategyThe particular investment process employed by a manager in the application of an investment style. selection but it cannot be entirely removed and should not be overlooked.
When trading in any substantial size – and this could be little more than one or two percent of the daily volume – the trader(1) A merchant involved in cash commodities; (2) a professional speculator who trades for his own account and who typically holds exchange trading privileges. expects to move the market and, moreover, to do so to an extent which is likely to dwarf any small improvements achievable on market VWAP. The key question for the execution strategyThe particular investment process employed by a manager in the application of an investment style. builder should then be this: How do I minimise the impact of my trades? Or more precisely, how do I minimise the implementation shortfall of which the market impact is a sizeable part? This is a question for transactionThe entry or liquidation of a trade. cost analysis, not backtesting.
Transaction Cost Analysis
The key to improving algorithmic execution lies in analysis of the actual impact of a strategyThe particular investment process employed by a manager in the application of an investment style.’s order slices on the underlying market. Designing and operating high-frequency real-time trading environments tells one that micro-level analysis of actual order impact on a slice-by-slice basis is at least as important as backtesting.
A strategyThe particular investment process employed by a manager in the application of an investment style. can be constructed out of numerous substrategies exploiting different aspects of market behaviour. Every time an order slice is placed in the market, the slice details are stored along with information about which substrategywas responsible for the generation of that slice Historical data can then be retrieved along with the saved slices, and the market movement subsequent to the placing of the slice is measured. The data is of course very noisy, since our order is only one among many factors responsible for price movement: Nevertheless, dynamic patterns are detectable over thousands of slices and it is possible to build up a picture of the dependence of market impact on size, substrategy, time of day and any other factor deemed significant. In response to the analysis, the strategyThe particular investment process employed by a manager in the application of an investment style. can be modified, shifting greater emphasis onto the substrategies that have historically leaned least heavily on the market and perhaps avoiding certain times of day and orders larger than a certain size.
One should not assume that the markets will continue to behave exactly as they have in the past. Indeed, one should expect them to change and for some of those changes to happen virtually overnight, especially as new algorithms come online. The Transaction Cost Analysis must be ongoing; its aim, in addition to improving execution performance, is to identify secular microstructural shifts in market responses to order slices. The execution strategies may need to be modified very quickly to continue performing acceptably.
In all this, backtesting has a very minor role to play.
