Automated Optimization of a Trading Robot

Introduction

It is supposed that an Expert Advisor having inputs adjusted to the history will trade to a profit for the first (rather short) time. Indirect confirmations of this suggestion appeared after I had watched the Automated Trading Championship 2006. When the Championship started there were much more profitable Expert Advisors than later, when some of them turned to be noncompetitive. This is why I suppose the most of those Expert Advisors that had not come to the finish were adjusted to the history.

The idea to check this supposition in practice was born on the Russian forum of this website, in the section of Ideal Automated Trading System. The main idea is to start optimization of an EA automatically once a day and then analyze the obtained optimization results and record them in the EA's variables.

To implement this idea, we decided to take the ready-made Expert Advisor, MACD Sample, from the MetaTrader 4 Client Terminal and insert our own function of automated optimization into it. A bit later, the code of that automated optimizer was ready and uploaded in the same forum, in the section of Automated Optimizer. After some more time, the first confirmations of the idea appeared in the branch of Automated Optimizer. Later on, the optimizer was transformed into an mqh-library for better usability.

Installation of the Automated Optimizer

Below is what must be done for performing this task:

• copy MACD Sample_1.mq4 into the 'expert' folder of the MetaTrader 4 Client Terminal preliminarily installed and connected to the Internet; and
• copy the folder together with the installed MetaTrader 4 Client Terminal to a new location.

For better usability, we will hereinafter name the original terminal "Terminal", whereas the copy will be named "Terminal-Tester". We will perform a checking test on EURUSD at timeframe H1 using the Expert Advisor initially provided by the terminal, but slightly changed - MACD Sample_1.mq4.

Terminal-Tester Setup

Please don't forget to compile MACD Sample_1.mq4 in the Terminal-Tester. Let's start the client terminal first, then strategy tester, and set it up as shown in the screenshot below.

Optimization will be performed for three days. This is quite enough to check the automated optimizer. We will select the optimization date "From" according to the following formula - the current date minus three days. During optimization, the necessary history for the selected symbol (in our case, it is EURUSD) must be downloaded.

Those who perform optimization for the first time can find the description of necessary procedures in the Help menu of the MetaTrader 4 Client Terminal: . Or they can read the article named Testing of Expert Advisors in the MetaTrader 4 Client Terminal: An Outward Glance.

Then let's check variables to be optimized as shown in the screenshot below.

Automated optimization is limited to four variables, but it will be enough to have only three for our purposes and in order to save time. After the variables have been selected, let us store the optimization settings in the set-file named as MACD Sample_1.set. This file must be saved in the 'tester' folder of the Terminal-Tester. Then launch the pre-optimization and memorize the start time. This is necessary to calculate the time period needed for automated optimization with the preset parameters. After the optimization is finished, we will calculate the necessary waiting time. Then we have to close this terminal, since otherwise we won't be able to start it programmatically.

Setup of the Expert Advisor Located in the Terminal

For this, let's open the test Expert Advisor MACD Sample_1.mq4 in MetaEditor and perform the following:

- set the start time for auto optimization, for example, at 00:01 every day:

        datetime SetHour    = 0;  // Optimization starting hour; datetime SetMinute  = 1;  // Optimization starting minute.

- set the amount of days for optimization (it must be the same as that for pre-optimization):

        int TestDay = 3;

- set the optimization end waiting time, in minutes, that we have calculated before, for example, 4 minutes:

        int TimeOut = 4;

- type in the Expert Advisor name:

        string NameMTS = "MACD Sample_1";  // EA's name

- type in the set-file name with the settings:

        // Set-file name with the settings        string NameFileSet = "MACD Sample_1.set"; 

- type in the path to the folder containing the installed Terminal-Tester, for instance:

        // Path to the testerstring PuthTester = "D:\Program Files\Forex Best Trade Station";

- set the priority for filtering:

        // Sorting by Maximal profit        int Gross_Profit = 1;// Sorting by Maximal profit factor                int Profit_Factor = 2;// Sorting by Maximal expected payoff        int Expected_Payoff = 3;

- write variable names for optimization:

        string Per1 = "FastEMA";string Per2 = "SlowEMA";string Per3 = "SignalSMA";string Per4 = "";

- copy attached file auto_optimization.mqh to the 'include' folder';
- include the library file in the Expert Advisor:

//--- Including the auto optimizer's library#include 

- it is only remains to copy the code below to the beginning of the start() function of your Expert Advisor. MACD Sample_1.mq4 contains it already.

// Not to be launched at testing and optimizing   if(!IsTesting() && !IsOptimization()){// Compare the current hour with that preset for launchingif(TimeHour(TimeLocal()) == SetHour){// Protection against restartingif(!StartTest){// Compare the minute range to the minute// preset for launchingif(TimeMinute(TimeLocal()) > SetMinute - 1){// the range is necessary, in case that for some reasons // no new tick is available for a long timeif(TimeMinute(TimeLocal()) < SetMinute + 1){// Flag of tester launchingStartTest    = true;TimeStart    = TimeLocal();Tester(TestDay, NameMTS, NameFileSet, PuthTester,TimeOut, Gross_Profit, Profit_Factor,Expected_Payoff, Per1, Per2, Per3, Per4);}}}}}FastEMA      = GlobalVariableGet(Per1);SlowEMA      = GlobalVariableGet(Per2);SignalSMA    = GlobalVariableGet(Per3);TrailingStop = GlobalVariableGet(Per4);// If the tester launching is flagged  if(StartTest){// If more time has elapsed the launching than it was set // to be the test waiting timeif(TimeLocal() - TimeStart > TimeOut*60){// Zeroize the flagStartTest = false;}}

That's all. After the automated optimizer has been recompiled, it can be launched, but it may be used for only the same symbol and timeframe, for which the pre-optimization was performed. In our case, it is EURUSD on H1. To check the auto optimizer, you can insert the code given below in the int init () function, then the auto optimizer will be launched at the Expert Advisor's start.

Tester(TestDay,NameMTS,NameFileSet,PuthTester,TimeOut, Gross_Profit,Profit_Factor,Expected_Payoff, Per1,Per2,Per3,Per4);

How Automated Optimizer Works

The automated optimizer works on the basis of using Terminal-Tester to optimize parameters of the Expert Advisor attached to the chart in Terminal. For this, the program sends to the Terminal-Tester a file containing optimization parameters (optimise.ini) and launches the Terminal Tester in the optimization mode. Then it copies the obtained results of "FileReport........htm" back to the Terminal and filters the best values from the obtained results.

More Details about How Auto Optimizer Works

At the preset time, for example, at 00.01, the automated optimizer should be launched. Variables should be filled out with values.

 // Path to the terminal string PuthTerminal = TerminalPath() + "\experts\files";// Name of the ini file for the testerstring FileOptim    = "optimise.ini";string FileOptim1   = "\optimise.ini";// Calculation of the starting datedatetime DayStart   = TimeLocal()-86400*TestDay;// Optimization starting datestring DateStart    = TimeToStr(DayStart,TIME_DATE);// Optimization ending datestring DateStop     = TimeToStr(TimeLocal(),TIME_DATE);// Tester report file namestring FileReport   = "FileReport_" + Symbol() + "_" + DateStop + ".htm";string FileReport1  = "\FileReport_" + Symbol() + "_" + DateStop + ".htm";// Limitation for the minimum amount of trades per daydouble MinTr        = TestDay - 2;// Limitation for maximal amount of trades per daydouble MaxTr        = (60 / Period()*TestDay) + 2;// The amount of attempts to copy the report fileint    KvoPptk      = 10;// The amount of lines for sortingint    StepRes      = 12;

Then the parameters of the ini file are written in the string array:

// Prepare the ini file for optimizationArrayOpttim[0] = ";optimise strategy tester";// Enable/Disable Expert AdvisorsArrayOpttim[1] = "ExpertsEnable = false";// Name of the EA fileArrayOpttim[2] = "TestExpert=" + NameMTS;// Name of the file containing parametersArrayOpttim[3] = "TestExpertParameters=" + NameFileSet;// SymbolArrayOpttim[4] = "TestSymbol=" + Symbol();// TimeframeArrayOpttim[5] = "TestPeriod=" + Period();// Modeling modeArrayOpttim[6] = "TestModel=" + 0;// RecalculateArrayOpttim[7] = "TestRecalculate=false";// OptimizationArrayOpttim[8] = "TestOptimization=true";// Use dateArrayOpttim[9] = "TestDateEnable=true";// FromArrayOpttim[10] = "TestFromDate=" + DateStart;// ToArrayOpttim[11] = "TestToDate=" + DateStop;// Report file nameArrayOpttim[12] = "TestReport=" + FileReport;// Rewrite the report fileArrayOpttim[13] = "TestReplaceReport=true";// Shut down the terminal upon completionArrayOpttim[14] = "TestShutdownTerminal=true";

The optimization parameters are recorded into the ini file from the array. You can also read about how to create an ini file in the MetaTrader 4 Client Terminal Help, see .

// Write data into the ini file                // Find out about the array sizeOptimArraySize = ArraySize(ArrayOpttim);// Open a file to writeopttim = FileOpen(FileOptim, FILE_CSV|FILE_WRITE, 0x7F);if(opttim > 0){for(int i = 0; i < OptimArraySize; i++){// from the array into the variableini = ArrayOpttim[i];// from the variable into the fileFileWrite(opttim, ini);}// close the fileFileClose(opttim);}else{Print("Failed writing data into the ini file. Error No ",GetLastError());return(0);}

After the parameters have been recorded in the ini file, the shell32.dll included in the standard Windows delivery is connected and function ShellExecuteA is launched.

#import  "shell32.dll"               //Connect a dll (provided with Windows)       int ShellExecuteA(int hwnd,string Operation,string                    File,string Parameters,string Directory,int ShowCmd);#import

File containing the parameters will be sent to the Terminal Tester folder.

// copy the ini file into the tester folder copyini = ShellExecuteA(0,"Open","xcopy", "\"" + PuthTerminal +FileOptim1 + "\" \"" + PuthTester + "\" /y","", 3);// wait until the file is copiedSleep(1200);if(copyini < 0){Print("Failed copying ini file");return(0);}

Then the tester is launched and starts to optimize the predefined variables. The Expert Advisor is in the halted state during optimization.

// Start Tester start = ShellExecuteA(0, "Open", "terminal.exe", FileOptim,PuthTester, 3);if(start < 0){Print("Failed starting Tester");return(0);}Comment("Wait until optimization is complete");// wait until optimization is completeSleep(60000*TimeOut);

After optimization is complete, Tester will automatically record the results in the report file. That file is copied to the folder that contains the terminal.

 for(Pptk = 0; Pptk < KvoPptk; Pptk++){//Start a cycle attempting to compy the resport fileComment("Attempt # " + Pptk + " to copy the report file");ShellExecuteA(0, "Open", "xcopy", "\"" + PuthTester + FileReport1 +"\" \"" + PuthTerminal + "\" /y", "", 3);// wait until the file is copiedSleep(1200);// Try to open the report filefile = FileOpen(FileReport, FILE_READ, 0x7F);if(file < 0){// if it fails to open, wait some more and try againSleep(60000);}elsebreak;}if(file < 0){Print("Failed copying the report file");return(0);}

Then the data from the report file will be placed in the string array for further processing.

// Read from file into the array// Cycle, until the file endswhile(FileIsEnding(file) == false){// Read a string from the report fileFileLine = FileReadString(file);// Find the necessary string and set the reference point thereindex = StringFind(FileLine, "title", 20);if(index > 0){// Increase the array in sizeArrayResize(ArrayStrg, NumStr + 1);// Record the strings from the file in the arrayArrayStrg[NumStr] = FileLine;NumStr++;}}// Close the fileFileClose(file);// Delete the file in order not to produce too many copiesFileDelete(FileReport);// Set the array size by the amount of data read from the fileArrayResize(ArrayData, NumStr); strings.

Then the necessary values are selected in the array:

  for(text = 0; text < NumStr; text++){select = ArrayStrg[text];//-------------------------------------------------------------------------//   Reporting text processing (These are apples and oranges)              | //-------------------------------------------------------------------------// Position Pass ClStep=StringFind(select, "; \">",20)+4;// Find the end of positionClStepRazm = StringFind(select, "td>", ClStep);// Read the valueCycleStep = StringSubstr(select, ClStep, ClStepRazm - ClStep);// Position Profit // Find the beginning of the positionGrProf = StringFind(select, "", ClStepRazm);// Find the end of positionGrProfRazm = StringFind(select, "td>", GrProf);// Read valueGrossProfit = StringSubstr(select, GrProf+4, GrProfRazm - (GrProf + 4));// Position Total Trades// Find the beginning of positionTotTrad = StringFind(select, "", GrProfRazm);// Find the end of positionTotTradRazm = StringFind(select, "td>", TotTrad);// Read the valueTotalTrades = StringSubstr(select, TotTrad+4, TotTradRazm -(TotTrad + 4));// Position Profitability// Find the beginning of positionProfFact = StringFind(select, "", TotTradRazm);// Find the end of positionProfFactRazm = StringFind(select, "td>", ProfFact);// Read the valueProfitFactor = StringSubstr(select, ProfFact + 4, ProfFactRazm -(ProfFact + 4));// Position Expected Payoff // Find the beginning of positionExpPay = StringFind(select, "", ProfFactRazm);// Find the dn of positionExpPayRazm=StringFind(select, "td>", ExpPay);// Read the valueExpectedPayoff = StringSubstr(select, ExpPay+4, ExpPayRazm -(ExpPay + 4));// Variables' positions starting with the second one// Find the beginning of positionP1 = StringFind(select, Per1, 20);// Find the end of positionP1k = StringFind(select, ";", P1);// Read the VariablePerem1 = StringSubstr(select, P1 + StringLen(Per1) + 1, P1k -(P1 + 1 + StringLen(Per1)));// Find the beginning of positionP2 = StringFind(select, Per2, 20);// Find the end of positionP2k = StringFind(select, ";", P2);// Read the VariablePerem2 = StringSubstr(select, P2 + StringLen(Per2) + 1, P2k -(P2 + 1 + StringLen(Per2)));// Find the beginning of positionP3 = StringFind(select, Per3, 20);// Find the end of positionP3k = StringFind(select, ";", P3);// Read the VariablePerem3 = StringSubstr(select, P3 + StringLen(Per3) + 1, P3k -(P3 + 1 + StringLen(Per3)));// Find the beginning of positionP4 = StringFind(select, Per4, 20);// Find the end of positionP4k = StringFind(select, ";", P4);// Read the Variable Perem4 = StringSubstr(select, P4 + StringLen(Per4) + 1, P4k -(P4 + 1 + StringLen(Per4)));Comment("The obtained results are being analyzed");

After that, the obtained results, before they are transformed into number format, have been filtered by the minimal and the maximal amount of trades. Zero in the value of Profit_Factor is replaced with 1000 for correct sorting and subsequent sifting.

// Transform into number formatTotalTradesTransit = StrToDouble(TotalTrades);GrossProfitTransit = StrToDouble(GrossProfit);ExpectedPayoffTran = StrToDouble(ExpectedPayoff);nodubl = true;if(MinTr < TotalTradesTransit && MaxTr > TotalTradesTransit){// Filter by the amount of tradesPrFactDouble = StrToDouble(ProfitFactor);// Replace 0 in the Profit_Factor for proper analysisif(PrFactDouble == 0){PrFactDouble = 1000;}

Then the values are checked for duplications and filtered out.

// Filter data having identical valuesfor(Dubl = 0; Dubl <= ResizeArayNew; Dubl++){// Start the loop searching for identical valuesif(GrossProfitTransit == ArrayData[Dubl][1]){// check whether the results for maximal profit coincideif(TotalTradesTransit == ArrayData[Dubl][2]){// check whether the results for the amount of trades coincideif(PrFactDouble == ArrayData[Dubl][3]){// check whether the results for Profit Factor coincideif(ExpectedPayoffTran == ArrayData[Dubl][4]){// check whether the results for expected payoff coincidenodubl=false;// If everything coincides, flag it as coincided}}}}}

Then the values prepared for sorting are written in the array.

// Write the filtered data in the arrayif(nodubl){ArrayData[text][1] = GrossProfitTransit;ArrayData[text][2] = TotalTradesTransit;ArrayData[text][3] = PrFactDouble;ArrayData[text][4] = ExpectedPayoffTran;ArrayData[text][5] = StrToDouble(Perem1);ArrayData[text][6] = StrToDouble(Perem2);ArrayData[text][7] = StrToDouble(Perem3);ArrayData[text][8] = StrToDouble(Perem4);ResizeArayNew++;}

Then the data start to be analyzed in the preset priority order. The analysis is performed as follows:

• the loop is launched and, at the first pass, the values are sorted by the first parameter, for instance, by maximal profit; several best values are selected (12, by default), others are cut off;
• at the second pass, the values are sorted by the second parameter, for example, by Profit Factor; some best values are selected, a half after the first sorting, others are cut off;
• at the third pass, the last sorting is performed for the third parameter, for example, by expected payoff; a half of values is taken after the second sorting, others are cut off.

// Analyzer// Analyzing principle is the sequential checking of maximal // values according to the predefined filtering priority   ArrayResize(ArrayTrans, ResizeArayNew - 1);for(int PrioStep = 1; PrioStep < 4; PrioStep++){for(PrCycle = 0; PrCycle < ResizeArayNew; PrCycle++){Sort     = ArrayData[PrCycle][0];Prior1   = ArrayData[PrCycle][1];transit  = ArrayData[PrCycle][2];Prior2   = ArrayData[PrCycle][3];Prior3   = ArrayData[PrCycle][4];transit1 = ArrayData[PrCycle][5];transit2 = ArrayData[PrCycle][6];transit3 = ArrayData[PrCycle][7];transit4 = ArrayData[PrCycle][8];if(PrioStep == 1){//Prepare for the 1st sortingif(Gross_Profit ==1){SortTrans = Prior1;}if(Profit_Factor == 1){SortTrans = Prior2;}if(Expected_Payoff == 1){SortTrans = Prior3;}}if(PrioStep == 2){// Restoreif(Gross_Profit ==1){Prior1 = Sort;}if(Profit_Factor == 1){Prior2 = Sort;}if(Expected_Payoff == 1){Prior3 = Sort;}//Prepare for the 2nd sortingif(Gross_Profit == 2){SortTrans = Prior1;}if(Profit_Factor == 2){SortTrans = Prior2;}if(Expected_Payoff == 2){SortTrans = Prior3;}}if(PrioStep == 3){// Restoreif(Gross_Profit == 2){Prior1 = Sort;}if(Profit_Factor == 2){Prior2 = Sort;}if(Expected_Payoff == 2){Prior3 = Sort;}//Prepare for the 3rd sortingif(Gross_Profit ==3){SortTrans = Prior1;}if(Profit_Factor == 3){SortTrans = Prior2;}if(Expected_Payoff == 3){SortTrans = Prior3;}}ArrayTrans[PrCycle][0] = SortTrans;ArrayTrans[PrCycle][1] = Prior1;ArrayTrans[PrCycle][2] = transit;ArrayTrans[PrCycle][3] = Prior2;ArrayTrans[PrCycle][4] = Prior3;ArrayTrans[PrCycle][5] = transit1;ArrayTrans[PrCycle][6] = transit2;ArrayTrans[PrCycle][7] = transit3;ArrayTrans[PrCycle][8] = transit4;}ArraySort(ArrayTrans,StepRes, 0, MODE_DESCEND); // Sort the arrayArrayResize(ArrayTrans, StepRes);               // Cut off the unnecessary thingsfor(int CopyAr = 0; CopyAr < StepRes; CopyAr++){ArrayData[CopyAr][0] = ArrayTrans[CopyAr][0];ArrayData[CopyAr][1] = ArrayTrans[CopyAr][1];ArrayData[CopyAr][2] = ArrayTrans[CopyAr][2];ArrayData[CopyAr][3] = ArrayTrans[CopyAr][3];ArrayData[CopyAr][4] = ArrayTrans[CopyAr][4];// Per1    Variable 1ArrayData[CopyAr][5] = ArrayTrans[CopyAr][5];// Per2    Variable 2ArrayData[CopyAr][6] = ArrayTrans[CopyAr][6];// Per3    Variable 3ArrayData[CopyAr][7] = ArrayTrans[CopyAr][7];// Per4    Variable 4ArrayData[CopyAr][8] = ArrayTrans[CopyAr][8];}StepRes = StepRes / 2;}

The values filtered in this manner are written in global variables. The values from global variables will be substituted in the EA.

     // Write the obtained results in variablesdouble Peremen1 = ArrayTrans[0][5];double Peremen2 = ArrayTrans[0][6];double Peremen3 = ArrayTrans[0][7];double Peremen4 = ArrayTrans[0][8];// If the variable name is specified, write the result in // global variablesif(Per1 != ""){GlobalVariableSet(Per1, Peremen1);}if(Per2 != ""){GlobalVariableSet(Per2,Peremen2);}if(Per3 != ""){GlobalVariableSet(Per3,Peremen3);}if(Per4 != ""){GlobalVariableSet(Per4,Peremen4);}Comment(Per1, " ", Peremen1, "  | ", Per2, " ", Peremen2, "  | ", Per3," ", Peremen3, "  | ", Per4, " ", Peremen4);Print(Per1, " ", Peremen1, "  | ", Per2, " ", Peremen2, "  | ", Per3," ", Peremen3,"  | ",Per4," ",Peremen4);}  // Function ends. That's all, automated optimization is complete.

Auto Optimizer Operation Results

The operation results of the automated optimizer can be observed using messages that appear in the upper left corner of the chart as shown in the screenshot below:

Optimization Completeness Reference Time.


Analysis of Values Obtained after Optimization.


Resulting Values of Variables.

If the optimization results appear in the message, it means that optimization is complete and data have been received.

To estimate the automated optimizer's work independently, one may look through all files containing intermediate data and saved during the working process. Tester stores data in the file named "FileReport_EURUSD_2007.03. 12. htm" where the symbol and the date are substituted in the file name according to the selected symbol and the current date of optimization. This file can be found in the Terminal-Tester folder. These file with reports are not deleted automatically, so one may use them to check the changes of parameters.



The next file, FileTest1.csv, is saved after the values have been filtered by the amount of trades and the copies have been deleted. The file is saved in: D:\Program Files\terminal_folder_ame\experts\files



Then values obtained after each sifting step are saved to FileTest2.csv. The file is saved in this folder, too: D:\Program Files\terminal_folder_name\experts\files



The tables above show how the obtained values are filtered. The filtering order was set by default as: 1- Gross_Profit, 2- Profit_Factor, 3- Expected_Payoff.

The automated optimizer's code contains detailed comments and you can fit the most suitable variable parameters, if necessary. For example, you want to optimize your EA for a period of time other than the latest days, or you are planning to increase/decrease the amount of trades within the optimization period. For this, you just should change the corresponding variables directly in auto_optimization.mqh.

 // Limitation of minimal amount of trades per daydouble MinTr   = TestDay - 2;// Limitation on maximal amount of trades per daydouble MaxTr   = (60 / Period()*TestDay) + 2;// The amount of attempts to copy the report fileint    KvoPptk = 10;// The amount of strings to be sortedint    StepRes = 12;

Conclusion

This article is not aimed at teaching newbies elements of optimization, so it is highly recommended to learn normal optimization before setting up automated optimization of your Expert Advisor. It is better to use the automated optimizer after you have chosen the basic variables that will influence your Expert Advisor differently at different times. I.e., it is better to use this automated optimizer to fit parameters of the variables the changes in which influence the EA operation more than those in other variables, depending on the market volatility.

Besides, it is better not to set a very large automated optimization period. Suppose the Expert Advisor has been optimized for 6-12 hours every day. Then a question occurs: When will it trade? In other words, optimization is not necessary as itself. It is recommended to set optimization periodicity (the periodicity of optimizer launching is meant) considering the timeframe, on which the EA is supposed to trade. This means that it is necessary to consider that the historical data will be pumped when the Tester-Terminal is started and it is possible that the broker just does not possess the necessary historical data for the specified period of time. To verify the hypothesis described at the beginning of this article, you will need 24-hour and stable internet connection.

The automated optimization programs developed are located in the attached files: auto_optimization.mqh - the library itself, MACD Sample_1.mq4 - slightly changed Expert Advisor included in the MetaTrader 4 Client Terminal standard delivery set