periodgen

Download script
Author: Igor Morozov
periodgen

This script is designed to automate the generation of historical trading data across multiple timeframes using a single source of minute-by-minute data. Instead of requiring you to manually calculate or download every different timeframe (like 5-minute, 30-minute, or 4-hour charts), this tool "builds" them for you from a raw M1 (1-minute) data file.

How the Logic Works

1. Data Input and Setup The script starts by looking into a specific folder on your computer (the "databank") to find a historical record of 1-minute price movements for a specific asset. It also allows for a "Time Shift" adjustment, which ensures the data aligns correctly with your specific server's clock.

2. Creating Multiple Outputs The script is programmed to simultaneously create seven different timeframes: M1, M5, M15, M30, H1, H4, and D1. For each of these, it creates a new individual data file.

3. Processing the Data (The Conversion Engine) The core logic works by reading the 1-minute data line by line and "grouping" the information:

  • Aggregation: As it reads every minute, it checks if that minute belongs to a new candle for the larger timeframes (for example, every 5 minutes for an M5 chart).
  • Price Calculation:
    • If the current minute is still part of an ongoing candle (like the 3rd minute of a 5-minute block), it updates the "High" price if the current price is higher, the "Low" price if it is lower, and adds to the total "Volume."
    • Once a minute crosses into a new candle boundary, it "saves" the final data for the previous candle (closing price, final high, and final low) into its respective file.
  • Progress Tracking: While processing large amounts of data, the script provides feedback by calculating the percentage of completion based on the total number of records found.

4. Logging and Completion Throughout the process, the script maintains a log file. This log records:

  • The specific asset and files being processed.
  • Header information (like versioning and time stamps).
  • The exact start and end times of the data.
  • A final summary showing exactly how many records were created for each timeframe.

Summary for the User

Essentially, this script acts as a data factory. You provide it with one large "raw" file of 1-minute data, and it automatically "repackages" that information into all the standard timeframes you need for your analysis, ensuring that every high, low, and closing price is mathematically accurate based on the original minute-by-minute movements.

Miscellaneous
Uses files from the file systemIt writes information to fileIt reads information from a fileIt reads information from a fileIt reads information from a fileIt writes information to fileIt writes information to fileIt writes information to fileIt reads information from a fileIt writes information to file
8 Views
0 Downloads
0 Favorites
periodgen 
//+------------------------------------------------------------------+
//|                                                    periodgen.mq4 |
//|                                                     Igor Morozov |
//+------------------------------------------------------------------+
//| Quickly create historical data from M1 data. Change TIMESHIFT to |
//| shift time for your server.                                      |
//| Steps to create data:                                            |
//| 1. create subfolder .../experts/files/databank/                  |
//| 2. copy M1 data into databank and call M1_XXXXXX.hst             |
//|    (alpari format)                                               |
//| 3. attach script 'periodgen' to the same chart XXXXXX, frame     |
//|    doesn't matter                                                |
//| 4. It will create subfolder .../experts/files/databank/XXXXXX    |
//|    in which you will find all timeframes up to D1 will be        |
//|    generated                                                     |
//+------------------------------------------------------------------+
#property copyright "Igor Morozov"
#property link      ""
//----
#include <stdlib.mqh>
#define NUM_FILES 7
#define DATA_FOLDER "databank"
#define BS "\\"
#define TIMESHIFT 3 // timeshift in hours
//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {
// initialize
	  int i;
	  int iFile, lFile, pFile[7];
	  string iFileName = "M1_" + Symbol() + ".hst"; // alpari input file
	  string lFileName = Symbol() + BS + Symbol() + ".log";
	  int pTime[NUM_FILES];
	  double pOpen[NUM_FILES];
	  double pLow[NUM_FILES];
	  double pHigh[NUM_FILES];
	  double pClose[NUM_FILES];
	  double pVolume[NUM_FILES];
	  int pCount[NUM_FILES];
	  int pPeriod[NUM_FILES];
	  pPeriod[0] = PERIOD_M1;
	  pPeriod[1] = PERIOD_M5;
	  pPeriod[2] = PERIOD_M15;
	  pPeriod[3] = PERIOD_M30;
	  pPeriod[4] = PERIOD_H1;
	  pPeriod[5] = PERIOD_H4;
	  pPeriod[6] = PERIOD_D1;
	  string pFileName[NUM_FILES];
	  for(i = 0; i < NUM_FILES; i++)
		     pFileName[i] = Symbol() + BS + Symbol() + pPeriod[i] + ".hst";
   int f_Unused[13];
// open files
	  iFile = FileOpen(DATA_FOLDER + BS + iFileName, FILE_BIN|FILE_READ);
	  if(iFile < 0)
	    {
		     Print(iFileName, " Open Error: ", ErrorDescription(GetLastError()));
		     return(0);
	    }
	  lFile = FileOpen(DATA_FOLDER + BS + lFileName, FILE_CSV|FILE_WRITE, " ");
	  if(lFile < 0)
	    {
		     Print(lFileName, " Open Error: ", ErrorDescription(GetLastError()));
		     return(0);
	    }
	  if(FileSize(iFile) < 192)
	    {
		     Print(iFileName, " has no data");
		     return(0);
	    }
	  for(i = 0; i < NUM_FILES; i++)
	    {
		     pFile[i] = FileOpen(DATA_FOLDER + BS + pFileName[i], FILE_BIN|FILE_WRITE);
		     if(pFile[i] < 0)
		       {
			        Print(pFileName[i], " Open Error: ", ErrorDescription(GetLastError()));
			        return(0);
		       }
	    }
	  FileWrite(lFile, TimeToStr(TimeCurrent()));
	  FileWrite(lFile, "Creating all timeframes for", Symbol(), "from", 
	            DATA_FOLDER + BS + iFileName);
	  FileWrite(lFile, "Time Shift ", TIMESHIFT);
	  // read header
	  int f_Version = FileReadInteger(iFile);
	  string f_CopyRight = FileReadString(iFile, 64);
	  string f_Symbol = FileReadString(iFile, 12);
	  int f_Period = FileReadInteger(iFile);
	  int f_Digits = FileReadInteger(iFile);
	  int f_TimeStamp = FileReadInteger(iFile);
	  int f_Sync = FileReadInteger(iFile);
	  FileReadArray(iFile, f_Unused, 0, 13);
	  FileWrite(lFile, "*------------------------------------------------");
	  FileWrite(lFile, "Header Information copied:");
	  FileWrite(lFile, "   Version:", "\t", f_Version);
	  FileWrite(lFile, " CopyRight:", "\t", f_CopyRight);
	  FileWrite(lFile, "    Symbol:", "\t", f_Symbol);
	  FileWrite(lFile, "    Period:", "\t", f_Period);
	  FileWrite(lFile, "    Digits:", "\t", f_Digits);
	  FileWrite(lFile, "Time Stamp:", "\t", TimeToStr(f_TimeStamp));
	  FileWrite(lFile, "      Sync:", "\t", f_Sync);
	  FileWrite(lFile, "*------------------------------------------------");
	  // write output headers
	  for(i = 0; i < NUM_FILES; i++)
	    {
	      FileWriteInteger(pFile[i], f_Version, LONG_VALUE);
   	   FileWriteString(pFile[i], f_CopyRight, 64);
   	   FileWriteString(pFile[i], f_Symbol, 12);
   	   FileWriteInteger(pFile[i], pPeriod[i], LONG_VALUE);
   	   FileWriteInteger(pFile[i], f_Digits, LONG_VALUE);
   	   FileWriteInteger(pFile[i], TimeCurrent(), LONG_VALUE);       //timesign
   	   FileWriteInteger(pFile[i], f_Sync, LONG_VALUE);       //last_sync
   	   FileWriteArray(pFile[i], f_Unused, 0, 13);
	    }
	  // init timeframes
	  int iCount;
	  int tCount = (FileSize(iFile) - 148) / 44;
   int f_Time = FileReadInteger(iFile) + TIMESHIFT*3600;
   double f_Open = FileReadDouble(iFile);
   double f_Low = FileReadDouble(iFile);
   double f_High = FileReadDouble(iFile);
   double f_Close = FileReadDouble(iFile);
   double f_Volume = FileReadDouble(iFile);
   iCount++;
	  FileWrite(lFile, "Start Time: ", TimeToStr(f_Time));
   for(i = 0; i < NUM_FILES; i++)
     {
 	     pTime[i] = f_Time - f_Time % (pPeriod[i]*60);
 	     pOpen[i] = f_Open;
 	     pLow[i] = f_Low;
 	     pHigh[i] = f_High;
 	     pClose[i] = f_Close;
 	     pVolume[i] = f_Volume;
     }
	  // creation loop	
   //	int testcount;
	  while(true)
	    {
		     bool isEndOfFile = FileTell(iFile) >= FileSize(iFile) - 1;
		     if(!isEndOfFile)
		       {
	   	      f_Time =FileReadInteger(iFile) + TIMESHIFT*3600;
	   	      f_Open =FileReadDouble(iFile);
   		      f_Low =FileReadDouble(iFile);
   		      f_High =FileReadDouble(iFile);
	   	      f_Close =FileReadDouble(iFile);
   		      f_Volume =FileReadDouble(iFile);
   		      iCount++;
	   	      if(iCount % 100000 == 0)
   			         Print(DoubleToStr(iCount*100.0 / tCount, 0), "% processed");
   	     }
   	   for(i = 0; i < NUM_FILES; i++)
   	     {
   		      int newtime = f_Time - f_Time % (pPeriod[i]*60);
   		      if(newtime != pTime[i] || isEndOfFile)
   		        {
	              FileWriteInteger(pFile[i], pTime[i]);
   	           FileWriteDouble(pFile[i], pOpen[i]);
      	        FileWriteDouble(pFile[i], pLow[i]);
         	     FileWriteDouble(pFile[i], pHigh[i]);
	              FileWriteDouble(pFile[i], pClose[i]);
   	           FileWriteDouble(pFile[i], pVolume[i]);
   	           pCount[i]++;
   	           if(pCount[i] % 100000 == 0)
   	      	        FileFlush(pFile[i]);
   	           if(!isEndOfFile)
   	             {
		   		            pTime[i] = newtime;
   				            pOpen[i] = f_Open;
		   		            pLow[i] = f_Low;
   				            pHigh[i] = f_High;
		   		            pClose[i] = f_Close;
   				            pVolume[i] = f_Volume;
   			           }
   		        }
   		      else
   		        {
   			         pClose[i] = f_Close;
   			         if(pLow[i] > f_Low)
   				            pLow[i] = f_Low;
   			         if(pHigh[i] < f_High)
   				            pHigh[i] = f_High;
   			         pVolume[i] += f_Volume;
   		        }
   	     }
   	   if(isEndOfFile)
   		  break;
	    }
	  // finish up
	  FileWrite(lFile, "  End Time: ", TimeToStr(f_Time));
	  FileWrite(lFile, "*------------------------------------------------");
	  FileWrite(lFile, "Records Processed (" + Symbol() + "1): ", iCount);
	  FileWrite(lFile, "Records Created:");
	  for(i = 0; i < NUM_FILES; i++)
		     FileWrite(lFile, Symbol() + pPeriod[i] + ":", pCount[i]);
   FileClose(iFile);
   FileClose(lFile);
   for(i = 0; i < NUM_FILES; i++)
 	     FileClose(pFile[i]);
   return(0);
  }
//+------------------------------------------------------------------+

Similar scripts

data_to_csv

Iteratively captures historical price action and volume metrics into a delimited text format for external analytics.

BrainTrendALP1Sig

BrainTrend1Sig_v2

CSV producer

This script extracts historical price action, volume, and temporal coordinates from a specified bar range to generate a structured comma-separated text file.

ClosingPriceReport

Comments