Alternative Investment Management Association
Algorithmic strategies are being used increasingly to execute orders in electronic financial markets. Algorithmic execution strategies break up an order, which is large in comparison with available volume (or bid), into manageable slices which are then worked upon over time to reduce the total cost of execution.
What is the algo objective?
Historically, in the equity markets, execution performance has been measured against a post-trade benchmark such as VWAP (the volume-weighted average price in the market over the day or period). The attraction of such a benchmark is that it indicates how well one market participant has performed in comparison with the rest of the market. However, even if the VWAP benchmark could be relied upon to give a fair picture of how well an algorithm or trader has performed compared with the market, it does not begin to address the question of whether that market VWAP itself was more costly than it needed to be.
More recently, there has been a trend towards the use of pure pre-trade measures and in particular to what is called “implementation shortfall” – the slippage between the price current at the generation of the trading decision and the volume-weighted average execution price actually achieved in the market. Following this trend we are disposed to regard “a good execution strategy” as one that produces competitive numbers for this measure.
Backtesting as a means of improvement
Backtesting is a venerable tradition in automated trading. Even before the advent of electronic markets, systematic traders evaluated trading strategies on the basis of historical data. This has proved fruitful for some, though there are a number of well-known pitfalls:
• The tendency to “fit” to historical data (over-optimisation);
• the vulnerability to the assumption that the future will be sufficiently similar to the past (stationarity);
• the dependence on realistic assumptions about the cost of execution (slippage).
Today’s algorithmic trading environments facilitate backtesting and allow an execution strategy to be evaluated against historical data. Once tested, modifications can be made to the strategy – e.g., rules added, thresholds changed, etc., and the revised model backtested. If the performance is improved, there would appear to be a rationale for preferring the new strategy. At least that is the idea …
However, the old pitfalls of backtesting have not gone away.
Fitting will always be a problem. Even with low parameter strategies and large volumes of historical data, it is still possible (and almost irresistible for the unwary) to greatly over-estimate the efficacy of a strategy – particularly if excessive use is made of backtesting and if no clear separation is made between in-sample training data and out-of-sample testing data. The more you look at data, the more patterns you will appear to find but the more of these will be spurious. This is really a law of statistics but it is something that most modellers learn only from brutal experience in the market when actual returns prove very different from the expected returns based on backtesting.
Additionally, the assumption that the future will be like the past is particularly dangerous in connection with algorithmic execution. Market microstructural changes are much more common than macrostructural ones and these can cause an execution strategy to come to grief. For the sake of illustration, let us suppose we have a strategy with opportunistic elements where trades are generated because the strategy is exploiting inefficiency in the market. It may look like a world-beater on historical data but to what extent can we rely on it? It is important not to lose sight of the fact that other modellers are capable of identifying similar opportunities. If they incorporate features designed to exploit the same inefficiencies in their strategies, then our own strategy will be taking part in a competition that it had not seen in history. Its performance is bound to be adversely affected.
These are not the worst problems. The fundamental problem of drawing conclusions about execution strategies from backtesting results is that you cannot tell from historical data how the market would have behaved if you had been trading. The historical data used for simulation is of the market without your interaction. There is no warrant to believe that the market would behave the same had you been participating. In fact there is every reason to think it would behave differently. Each order placed in the market changes the balance in weight between buying and selling interest and thus affects price formation. This impact is something that can be mitigated by judicious strategy selection but it cannot be entirely removed and should not be overlooked.
When trading in any substantial size – and this could be little more than one or two percent of the daily volume – the trader expects to move the market and, moreover, to do so to an extent which is likely to dwarf any small improvements achievable on market VWAP. The key question for the execution strategy builder should then be this: How do I minimise the impact of my trades? Or more precisely, how do I minimise the implementation shortfall of which the market impact is a sizeable part? This is a question for transaction cost analysis, not backtesting.
Transaction Cost Analysis
The key to improving algorithmic execution lies in analysis of the actual impact of a strategy’s order slices on the underlying market. Designing and operating high-frequency real-time trading environments tells one that micro-level analysis of actual order impact on a slice-by-slice basis is at least as important as backtesting.
A strategy can be constructed out of numerous substrategies exploiting different aspects of market behaviour. Every time an order slice is placed in the market, the slice details are stored along with information about which substrategywas responsible for the generation of that slice Historical data can then be retrieved along with the saved slices, and the market movement subsequent to the placing of the slice is measured. The data is of course very noisy, since our order is only one among many factors responsible for price movement: Nevertheless, dynamic patterns are detectable over thousands of slices and it is possible to build up a picture of the dependence of market impact on size, substrategy, time of day and any other factor deemed significant. In response to the analysis, the strategy can be modified, shifting greater emphasis onto the substrategies that have historically leaned least heavily on the market and perhaps avoiding certain times of day and orders larger than a certain size.
One should not assume that the markets will continue to behave exactly as they have in the past. Indeed, one should expect them to change and for some of those changes to happen virtually overnight, especially as new algorithms come online. The Transaction Cost Analysis must be ongoing; its aim, in addition to improving execution performance, is to identify secular microstructural shifts in market responses to order slices. The execution strategies may need to be modified very quickly to continue performing acceptably.
In all this, backtesting has a very minor role to play.