We know that doing a correct Back-Testing of our system is very important. Whether we optimize or not. Back-Testing will tell us roughly whether the system works or not, how much money we can expect to earn from it or not, etc. But … is this really so? Is the Back-Testing of our system reliable regarding the future performance of our system? It depends.
In the trading industry the phrase “Past performance does not guarantee future performance” is used a lot, which is also true for trading systems.
What really matters about any system is the Robustness, its ability to keep making money in the future, as indicated by Back-Testing. Probably many veteran readers and trader will have found a system that has an excellent Back-Testing which when used to trade in real time does not behave the way expected.
Of course. You can argue that there are ways and ways to do a Back-Testing. It is true. In the same way there are techniques to improve Robustness, both in the design and in the evaluation or optimization of a trading system. True too. What’s more, a good Walk-Forward or external test analysis can help us assess whether our system maintains its behavior in non-optimized data. We optimize a piece of history and check the results in another piece not optimized. Perfect, true too.
But in reality, the external test we have to do in different historical periods, so that we can obtain several Out-of-Sample (OOS) results. For example:
It is fine … but it has its limitations and in any case, doing well this analysis requires a really precious time and work. It is a very complex and laborious procedure that we have usually done in Excel … Until now.
The technology advances and this puts in our reach better and better tools. The TradeStation Walk Forward Optimizer (WFO) is the first tool that allows us to objectively measure the robustness of our system.
The WFO is a process that performs successive optimizations in certain periods other than historical, uses In-Sample (INS) samples and apply the best parameters to successive non-optimized periods that we call the Out-of-Sample (OOS) sample.
It is a stress test that tests our system with various tests all performed with data out of sample. Actually WFO starts where the rest of Back-Testing methods end. It is the closest thing to testing our system in “future” data because it is able to build even an Equity or Performance Report completely obtained with OOS data.
WFO will realistically answer the following questions:
- Will our system continue to make money after optimization?
- How will changes in market behavior affect performance?
- How often should we re-optimize our system?
The software performs the optimization with all the chosen combinations. Choose the best INS and apply OOS according to the chosen objective function. This allows us to obtain a huge number of INS and OOS segments, different from each other, which tests the logic behind the system.
Thus, we obtain an array of data as the following:
The matrix shows whether the system passes the tests or not on each cluster. WFO applies 5 tests to the OOS data of each cluster and the system has to pass the 5 to obtain the PASS. The values to be overcome by default are the following, although they are configurable:
- Total Profit: OOS data must add up to more than 0, meaning they must earn money.
- Walk-Forward Efficency> 50: The annualized OOS benefit must be at least 50% of the earned INS.
- Consistency of Profits: At least 50% of runs must have earnings.
- Distribution of Profits: No run can contribute with more than 50% of total profit obtained by OOS.
- DrawDown: No run can have a DD greater than 40%.
The purpose of WFO is to obtain the greatest number of PASS possible, reflecting in green the central area of the data. A Robust system must pass the tests in most Clusters, but there are many systems that do not get it because we really stress the logic of our system. It must be able to adapt to numerous combinations of OOS segments and number of executed runs.
But the potential of WFO does not end there. The matrix is a summary of the tests performed but we can see the detail of each one of them as well as see the INS and OOS runs with their main ratios. This is the analysis that corresponds to the cluster 10% OOS and 5 runs:
There are many data to be studied in each of these tables, especially in OOS, which allow us to analyze in detail the robustness of our system.
Another possibility that allows WFO is to choose the parameters that would be used to trade in real time with our system, ie, would be run 6 in the example shown. The purpose is to stress the strategy and verify whether or not the system passes the test. If the system pass it, you can use this technique or not to set the operative parameters, which even tells us when to re-optimize our system in a fully programmed way.
And there is more … We can perform sensitivity tests, MonteCarlo, see the Equity of any combination, see the performance of any combination and in any range of dates … As an example we show the Equity of the Cluster shown in the tables INS and OOS:
The first time I saw this application I thought it would be only within the reach of the institutional or that it would be very expensive. The reality, however, is very different. WFO is included in the TradeStation platform and is completely free for its customers.
For those who do not know Tradestation, it is one of the most advanced trading software platforms on the market in addition to an efficient and competitive broker, making it a complete solution for trading.
