Generator : Out of sample

[WassB]

Hello,

I watched the video tutorial, but I have a question and can’t find the answer in tutorials. I want in the Express generator, take into account an out of sample of 30% in the generator like we can do on EA Studio :

So I have seen in the gen.setting file those commands,

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Forward testing          ;;

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Validate strategies on new unseen data.

use_forward_testing = false

preload_data_bars   = 0

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Out of sample            ;;

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

; Use these parameters to set the data range.

; The program cuts data after applying Data Horizon.

data_start_percent = 0

data_end_percent   = 100

 

How can I use it to make an out of sample of 30% ? I tried “use_forward_testing = true”, “data_start_percent = 70” and “data_end_percent   = 100” (to make 30% OOS), but it seems to be not that because when I load the collection on the EA Studio, I don’t see the green part that represent the OOS.

Do you have an idea how can I do that correctly ?

 

Thank you,

Regards,

Wassim

[Alan Northam]

The way I do it for example is as follows: In expressgenerator I will create strategies for 01/01/2020 through 10/01/2023. Then I will import the strategies in EA Studio with the date set from 1001/2023 through today. The result is the OOS test. It will not be green but it is an OOS result since the strategy was created from 01/01/2020 through 10/01/2023 and EA Studio is testing out of that sample.

Alan,

[WassB]

Ok thank you for you answer thats clear.

I’m a big fan of automation so tell if I’m wrong but :

If I want to do it via Express Generator, I understand that I can create strategies for 01/01/2020 through 10/01/2023, save the collection then re-use this collection using the command –input …. ^ to load it and test it on the new periode representing the OOS, that’s it ?

I see just one problem : On EA studio I use the Monte Carlo Validation which normally is including the OOS, so it is done on the whole periode. So now with this method I will use Monte Carlo to validate the created strategies (In Sample), then use it again for the OOS simulation. Using Monte Carlo twice is equivalent to use a Monte Carlo Validation once on the whole periode ? (In Sample + Out of Sample)

Thank you,

Regards,

Wassim

[Alan Northam]

I do the Monte Carlo using ExpressGenerator (EG).  When I import into EA Studio I do not do the Monte Carlo as it was already done in EG.  The reason I do the OOS in EA studio is because you still have to import the strategies into EA Studio to create the EAs so why not just do the OOS there.  Plus, it allows me to do various filtering on the collection to find the strategy I want to use.  Anyway, that’s the way I use it.

Alan,

 

[WassB]

Ok I see. You don’t use the Monte Carlo on OOS, so you only use it once, when you create the strategies. Clear for me , thanks!

Concerning the way I work, I try to use EA Studio as little as possible because I want to try a lots of different ways to find strategies (playing on timeframe, in sample size, indicators , selecting best profit/Rsquared etc … ). I want to find the best research method that increases the probability of finding good strategies, so I need to work fast, everything being automated as much as possible, to save time and explore as many possibilities as I can.
And OOS was a criteria I wanted to explore 🙂

Well that’s the way I work. 🙂
Thank you for your answers.

Wassim

[Alan Northam]

That is a good to try different things to see what works best.  Good exercise.  Happy to hear you are being creative and not just following what someone else does.  Good job.

Alan,

[WassB]

Yes thanks !

So how to use those settings :

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Out of sample ;;

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

data_start_percent = 0

data_end_percent = 100

What are they for ?  I’m not sure to use it in a the good way. Anyone know how to use it?

 

Thanks,

Rds,

Wassim

[Alan Northam]

Lets say you have the horizon settings set to great strategies from January through October a period of 10 months.

With Out of sample set to data_start_percent = 0 and data_end_percent = 100 then EG will use all 10 months to create strategies.

Now with Out of sample set to data_start_percent = 10 and data_end_percent = 90 then EG will use 8 months from February through September to create strategies with January and October not being used in creating strategies.  These two months are not in the sample being used to create strategies, therefore, they are out of sample.

Test it and while EG is creating strategies you can view the range of dates it is using.

Alan,

[WassB]

Alan, thank you for your quick and clear responses, I appreciate. I just ran the test, and it works as you explained.

Do you know if there is an automatique check for Acceptance Criteria on the Out of Sample part?

For example I run the generator for the dates : 01/01/2023 to 01/10/2023 with data_end_percent = 70, so it creates strategies from 01/01/2023 to ~01/07/2023. Then is there a check of acceptance criteria on the period 01/07/2023 to 01/10/2023 ?

Thanks,

Rds,

Wassim

[Alan Northam]

Only the in sample date range will have Acceptance Criteria applied!

Alan,

[WassB]

Ah ok so It’s not like in EA Studio where you can add different acceptance criteria between In-Sample and Out-of-Sample date range.

Thank you,

 

Wassim

[Alan Northam]

To do this in EG you would have to take the collection and run it during the OOS time period (example 80% to 100%) to test against Acceptance Criteria.  You may have to change some Acceptance Criteria settings such as minimum count of trades may not be met over a short OOS time period.

Alan,

[WassB]

Ok so I will do that.

Thank you!

Rds,
Wassim

[Author]

Posts