The need for off-sample data testing in quantitative strategies

Big data in real life. The need for off-sample data testing in quantitative strategies.

• NO：01

  The life of a person, from young to old, from old to old, is in fact a process of making mistakes, correcting and making mistakes, almost no one can be an exception. Maybe you have made many mistakes, which now seem very low-level; or maybe you missed many opportunities to get on the train, such as: real estate, Internet, digital currency, etc...

  So that the hearer may say: "I shouldn't have"... "If... I would... "

  I had this question for a long time, and I could not get it out of my head until I realized it later. In fact, it was not something to be afraid of, because at that time every choice, right or wrong, would lead us away from a predetermined outcome and towards an unknown; and our reflection, just outside the historical data, would open up the perspective of God.

• NO：02

  I have seen many trading systems with a success rate of more than 50% when retested. With such a high win-loss ratio, there can also be a win-loss ratio of more than 1: 1. However, without exception, these systems are basically loss-making once they are put on the table. There are many reasons for the loss, including, when retested, unintentionally, looking from right to left, opening the God's perspective.

  

  However, transactions are such a tangled thing that in retrospect it is clear that if we do not go back to the beginning without the lens of God's perspective, we will still be uncertain. This is a limitation of quantitative roots and historical data. It is difficult to avoid the problem of looking at the rear-view mirror when testing a trading system with limited historical data.

• NO：03

  But how can one make the most of the limited data to fully test a trading strategy in the face of limited data? There are usually two approaches: push and cross checking.

  The basic principle of a regression test is to train the model with the previous longer historical data, and then test the model with the relatively shorter data that follows, and then continuously move the data window backwards, repeating the training and testing steps.

  

  1, Training data: 2000 to 2001, test data: 2002; 2, Training data: 2001 to 2002, test data: 2003; 3 Training data: 2002 to 2003, testing data: 2004; 4 Training data: from 2003 to 2004, test data: from 2005; 5 Training data: 2004 to 2005, testing data: 2006;

  ...and so on and so forth...

  Finally, the results of the tests (in 2002, 2003, 2004, 2005, 2006...) were statistically evaluated to provide a comprehensive assessment of the strategy's performance.

  The following diagram illustrates an intuitive explanation of the principle behind the push-up test:

  

  The diagram above shows two methods of push-up testing.

  The first is that each test has a shorter test run and a larger number of tests. The second is that the test data is longer and the number of tests is smaller.

  In practical applications, multiple tests can be performed by varying the length of test data to determine the stability of the model in the face of non-equilibrium data.

• NO：04

  The basic principle of cross-checking is to divide all the data into N parts, train with N-1 parts each time, and test with the rest.

  

  Divide the years 2000 to 2003 by year into four parts. The operation of the cross-checking is as follows: 1, training data: 2001-2003, test data: 2000; 2, Training data: 2000-2002, test data: 2003; 3, Training data: 2000, 2001, 2003, test data: 2002; and 4 Training data: 2000, 2002, 2003, test data: 2001, and the following:

  

  As shown above, the biggest advantage of cross-checking is the full use of limited data, each training data is also test data. However, there are also obvious disadvantages when cross-checking is applied to the testing of strategy models:

  1, when price data is not stable, the model test results are often unreliable. For example, training with 2008 data and testing with 2005 data. It is likely that the market environment in 2008 has changed significantly compared to 2005, so the model test results are unreliable.

  2, Similar to the first, in cross-checking, it is not logical to train the model with the latest data and test the model with older data.

• NO：05

  Additionally, when testing a quantitative strategy model, both push and cross checks encounter data overlap issues.

  

  When developing a trading strategy model, most of the technical indicators are based on historical data of a certain length. For example, using a trend indicator to calculate the historical data of the last 50 days, and the next trading day, the indicator is calculated from the data of the previous 50 days, then the data of the calculation of the two indicators is the same for 49 days, which will result in the change of the indicator every two adjacent days.

  

  Overlapping data can have the following effects:

  1, slow changes in model predicted outcomes lead to slow changes in holdings, which is what we often call lagging indicators.

  2, some statistical values are not available for the model results test, and the results of some statistical tests are unreliable because of the sequence relatedness caused by the repetition of data.

• NO：06

  Good trading strategies should be profitable in the future. Out-of-sample testing, in addition to objectively detecting trading strategies, can more effectively save time for the broadcaster.

  In most cases, it is very dangerous to engage in real-world combat by directly using the optimal parameters of the entire sample.

  If all historical data prior to the time point of optimization is separated into in-sample and out-sample data, and the parameters are optimized using the in-sample data, then the out-of-sample data is used for out-of-sample testing, this error can be sorted out, while also checking whether the optimized strategy is applicable to future markets.

• NO：07

  Just like trading, we can never travel through time and make a correct decision for ourselves without making a single mistake. If there is a hand of God or the ability to travel back from the future, then without testing, we can trade directly online, and the pot is full. And we mortals must test our strategy in historical data.

  However, even with a history of vast amounts of data, history is extremely scarce in the face of an endless and unpredictable future. So a trading system based on history pushed up and down will eventually sink over time. Because history cannot end in the future.

  

• NO：08

  We (inventors of Quantitative Trading Platforms) aim to change the current quantum cycle to a purer quantum cycle, one without dry goods, exchange-closures, and scammers. No one has ever created knowledge and theories in this world, they just exist and are waiting to be discovered.

  

  Sharing is an attitude, but it's also wisdom!

Loose guest is online Written by Hukybo