optimizing rules and variables over an entire data set and obtaining a profitable result does not say anything about how that set of rules will perform on unseen, future data. It's possible - no, it's relatively easy, to optimize a single variable like a trailing stop to turn a trading system that should perform no better than chance, to an awe inspiring, hand rubbing, $ signs in the eyes gold mine. And that's just evil.
First, I'll show you perform this black magic. Then, I'm going to show you how to do it correctly - how to look for the TRUTH - how we do it here at AlgoLab.
Black Magic Recipe:
I built a simple trend line break technical pattern system that has 1 variable which is a following stop. The following stop is optimized using 4 steps from 4 standard deviations to 16 standard deviations.
The rules are simple, If price breaks down through a 10 bar pivot up trend line, the system generates a sell signal. If price pushes up through a down trend line, the system generates a buy signal. 4 following stop values are tested. Th total number of variables testing on the data therefore, is 4.
The data set is historical data for 20 continuous, back adjusted futures symbols, 30 minute per bar time frame from 1/1/2008 to 10/1/2015 (about 1/2 million bars total). According to "degrees of freedom" rules intended to help prevent curve fitting when optimizing rules for a trading system, this would be very close to 100% degrees of freedom, and the recommended minimum DOF to avoid curve fitting is 90%.
For this test, no trading costs such as commissions or fees have been deducted. Costs for slippage is also not included.
Here is the resulting equity curve:
Looks pretty good doesn't it? Would you say that this system, using the optimized following stop for each market would likely continue to perform on future, unseen data?
Well, look at this equity curve that was run on a different basket of historical data (same number of bars):
Also looks pretty good right? The profit was higher, and profit to drawdown ratio was slightly better, and the average trade was better. This second back test is using RANDOM data. That's right, historical data for 20 symbols that has been generated using a random number generator. To the human eye, this data looks like it's underlying symbol, but the bar by bar returns are 100% random.
And no, it is NOT POSSIBLE to generate a profit using a good exit and/or entry strategy on RANDOM DATA!
Let me break this down, and explain exactly what is happening here with this random data. Since the data is random, the trading system rules are just random dart throws into the data. So, essentially, we randomly pick a price bar in the random data, and then randomly say we are either going to sell at the close, or buy at the close, and this is also chosen randomly. Random bar, random bias signal. The trend line break means nothing in this case. So, now that we are in a long or short position, we can test 4 following stop amounts. The chances are good that at least one of these four stops will result in an overall profit across the entire symbol - but even better because we can select the most profitable stop as our optimized variable for this symbol. Now we can do it all over again for the next symbol. As it turns out, out of the 20 symbols, we found an over all profitable stop value in 16 of these symbols, and the remaining 4 lost money, but again, we got to select the stop value that lost the least amount of money for these remaining 4. Now, when you merge all of these 20 sets of trades together, the equity dips and peaks are smoothened out, and you end up with a very pretty and smooth, 45 degree up-sloping equity curve!
So... it appears that testing trading systems by optimizing the variables - even on VAST quantities of historical data, is NOT VIABLE, because promising results can easily be obtained using random data. Therefore, we have no way of knowing if the system is really working!
Out of sample testing is not the solution, because as soon as you attempt an out of sample test more than one time using different variables (which we ALL inevitably do when we don't get the results we are looking for after the first out of sample run), you are optimizing the variables again, and simply repeating the original in-sample variable optimization (but 'calling' it "out of sample) !!
Now for the good stuff:
>> still working on it! Maybe a future post