Tell me one thing: how do you validate your trading system? How can you be sure that by following these entry and exit signals from the market you will have profitable results? Just because you have executed a backtest and it has given you good results?
In today’s article I propose to rethink your way of working and begin to apply the scientific method as a procedure of development and validation of trading systems.
Application of the scientific method to investment and trading systems
This is the main idea developed by David Aronson in his book Evidence Based Technical Analysis. Well, specifically he proposes to use the hypothetical-deductive method to evaluate the effectiveness of signals in a trading system.
The theory says that the hypothetico-deductive method consists of 5 phases:
- Observation
- Hypothesis
- Prediction
- Verification
- Conclusions
How does Aronson want to apply this method to test a trading system? Let’s see an example
Example
Observation:
We analyze the behavior of the SP500. We note that when the index exceeds its 200-day moving average, the index generally continues to rise during the following X months.
HYPOTHESIS:
Here comes the inductive phase. Based on our observations from the previous phase, we formulated the following hypothesis 200-H.
– 00-H: “When the price crosses its moving average of 200 periods, we can obtain gains taking long positions during the following 3 months”.
PREDICTION:
If the hypothesis 200-H is true, then the backtest will show good results. However, if we directly seek to corroborate this hypothesis, we will find a logical fallacy (see wikipedia article “Affirming the consequent“).
To overcome this drawback we will rely on the work method of inferential statistics and we will propose two mutually exclusive hypotheses: a null hypothesis and another alternative hypothesis. Only one of them can be true.
In statistics, the null hypothesis is the basic assumption. Our objective is to propose an alternative hypothesis (which is what really interests us), to examine the data and in the case that there is enough evidence, to reject the null hypothesis.
Now comes the key issue: following the bases of inferential statistics, the data will never allow us to accept our alternative hypothesis. If we follow the scientific method, the only thing we can do is to refute, or not to refute, the null hypothesis.
Following with the previous example:
Null Hypothesis (H0): The bullish crosses of the price above its 200 SMA do NOT generate long position gains during the following 3 months. If this hypothesis is true, then the results of the backtest will NOT be profitable.
As you can see, the statement is the reverse of what you do (or we do because I do it too) when we validate a trading system. With this method we start by postulating that our system does not make money and this is the hypothesis that we try to refute.
If through the evidence we can show that our system gives results greater than zero, then we can refute H0 and show that we have a profitable system.
Verification
How can we verify the hypotheses? With the statistical analysis of the results of the backtest.
The rejection of the null hypothesis occurs when the average profitability obtained in the tests is statistically significant.
Let’s suppose that the backtest shows positive results. This result may be a consequence of an error in the sample (that particular backtest is simply luck) or it may be because H0 is wrong.
Following the example, what we need to falsify the null hypothesis is to obtain such a high profitability (so improbable) that it is not reasonable to attribute it simply to luck.
If we continue with the example of Aronson, in order to reject H0, profitability must be so high that it only has 0.05 or less probability of occurring within the framework of a true null hypothesis.
Put another way: a statistical test with 95% confidence means that there is a 5% chance that this return is just luck and that the null hypothesis is true.
CONCLUSIONS
This method to analyze trading systems is based on indirect evidence. To corroborate that the hypothesis that interests us is true, we must show that the opposite hypothesis is false.
What are the advantages of applying this method
This method forces us to combine rational reflection (the formation of hypothesis and deduction) with the observation of reality (observation and verification). Obviously, it forces us to confront our ideas with real data. We work a quantitative trading system in the sense that we have a model and rules that we can contrast.
We minimize the effect of our cognitive biases.
