Matrix4

Download script
Author: Copyright � 2008, MetaQuotes Software Corp.
Matrix4

This script is designed to handle data preparation for machine learning models, specifically focusing on a process called "Normalization." In non-technical terms, normalization is the act of scaling different types of data so that they can be compared fairly on the same scale.

Core Logic Overview

The script manages two distinct sets of data:

  1. Sample Data: Think of this as the "Training Set"—the historical information the system uses to learn patterns.
  2. Test Data: Think of this as the "Live" or "Validation" data—new information that the system needs to evaluate based on what it learned from the training set.

How the Data is Processed

The script performs several mathematical operations to ensure these datasets are uniform:

  • Statistical Analysis: The code calculates key metrics for each column of data, including the Average (Mean), the Standard Deviation (how much the numbers vary), the Range (the distance between the highest and lowest numbers), and the Midrange (the center point between the high and low).
  • Data Scaling: Once these metrics are identified, the script transforms the raw numbers. It shifts the data so it centers around a specific point (like zero) and shrinks or stretches it so that it fits within a consistent scale. This prevents "large" numbers from drowning out "small" numbers that might be equally important for a prediction.

Versatile Normalization Modes

The script is highly flexible, offering four different ways to handle this scaling depending on the user's needs:

  1. Independent Scaling: The system scales both the training and testing data using their own unique averages and variations.
  2. Reference Scaling (Standard): The system scales both datasets using the statistics derived from the training data. This is a common practice to ensure that new data is interpreted exactly like the data used for learning.
  3. Independent Range Scaling: Similar to the first mode, but uses the high/low boundaries rather than standard deviation to set the scale.
  4. Reference Range Scaling: Similar to the second mode, but uses the boundaries from the training data to define the scale for the new data.

Execution Flow

When run, the script automatically generates a pool of random data points to simulate real-world inputs. It then performs the normalization cycles sequentially and outputs the "Before" and "After" results into a text format, allowing a user to see exactly how the raw values were transformed into standardized data.

2 Views
0 Downloads
0 Favorites
Matrix4 
//+------------------------------------------------------------------+
//|                                                      Matrix1.mq4 |
//|                      Copyright © 2008, MetaQuotes Software Corp. |
//|                                        http://www.metaquotes.net |
//+------------------------------------------------------------------+
#property copyright "Copyright © 2008, MetaQuotes Software Corp."
#property link      "http://www.metaquotes.net"
#define DBL_MAX 99999999999999999999
#define LIN1 5 //sample lines
#define LIN2 4 //test lines
#define COL 3  //attributes
#property show_inputs
extern   int normalize=1;

double M_sample_attr[LIN1][COL];   
double M_sample_class[LIN1][1];   

double M_test_attr[LIN2][COL];   
double M_test_class[LIN2][1];   

string m1;

int init()
  {
    m1="";
    MathSrand(TimeLocal());
    return(0);
  }                
//+------------------------------------------------------------------+
//| script program start function                                    |
//+------------------------------------------------------------------+
int start()
  {

//----
  for(int i=0;i<LIN1;i++)
    for(int j=0;j<COL;j++)
      M_sample_attr[i][j]=MathRand();   

  for(i=0;i<LIN1;i++)
    for(j=0;j<1;j++)
      M_sample_class[i][j]=MathRand();         

  for(i=0;i<LIN2;i++)
    for(j=0;j<COL;j++)
      M_test_attr[i][j]=MathRand(); 

  for(i=0;i<LIN2;i++)
    for(j=0;j<1;j++)
      M_test_class[i][j]=0; 
      
  PrintMatrixToStr(M_sample_attr);  
  svm_normalize_sample(M_sample_attr,normalize);
  PrintMatrixToStr(M_sample_attr);  

  PrintMatrixToStr(M_sample_class);  
  svm_normalize_sample(M_sample_class,normalize);
  PrintMatrixToStr(M_sample_class);  

  PrintMatrixToStr(M_test_attr);  
  svm_normalize_test(M_sample_attr,M_test_attr,normalize);		
  PrintMatrixToStr(M_test_attr);    

 // PrintMatrixToStr(M_test_class);  
  
   Comment(m1);	  
//----
   return(0);
  }
//+------------------------------------------------------------------+

void PrintMatrixToStr(double Matrix[][])
{
	int Line = ArrayRange(Matrix,0);
	int Col = ArrayRange(Matrix,1);

	for (int i=0;i<Line;i++)
	{
	  for (int j=0;j<Col;j++)
	   m1=m1+Matrix[i][j]+",";
	  
	  m1=m1+"\n"; 
	}   
	m1=m1+"\n";
}	

//+------------------------------------------------------------------+
//|                                                                  
//+------------------------------------------------------------------+
double MAX(double x, double y)
{
  if(x>y)
    return(x);
  else
    return(y);  
}

//+------------------------------------------------------------------+
//|                                                                  
//+------------------------------------------------------------------+
double MIN(double x, double y)
{
  if(x<y)
    return(x);
  else
    return(y);  
}

//+------------------------------------------------------------------+
//| Normalize matrix 'fmx' using precomputed 'xmean', 'std'. The                                                                  
//| normalization is: for each column, subtract 'xmean' and, if the
//| column 'std' is non-zero, divide by 'std'.
//| d=nr_of_attrib
//| nv=nr_of_sample_lines 
//| double **x --> double **svm_samples_attrib_scaled_dyn_
//| double *xmean = fmx_mean(x2, nv, d);
//| double *std = fmx_std(x2, nv, d);
//| fmx_prenorm(x, nv, d, xmean, std);
//|       X  - mean
//|        i
//|   S = ---------
//|    i     std
//+------------------------------------------------------------------+
void fmx_prenorm(
                 double &fmx[][], 
                 double xmean[], //  double *xmean = fmx_mean(x2, nv, d);
                 double std[]   //  double *std = fmx_std(x2, nv, d);
                 )
{
  int    i;
  int    j;
  double  f;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);

    for (i = 0; i < columns; i++)
      for (j = 0; j < rows; j++)
	  {
	    f = fmx[j][i];
	      if (std[i] != 0)
		     fmx[j][i] = (f-xmean[i])/std[i];
	      else
		     fmx[j][i] = f-xmean[i];
	 }
	 

	 
}

//+------------------------------------------------------------------+
//|              max X  +  min X
//|               i   i     i   i
//|   midrange = ----------------
//|                     2
//|   range = max X  -  min X
//|            i   i     i   i
//|    
//|       X  - midrange
//|        i
//|   S = -------------
//|    i     range / 2                                                                     
//+------------------------------------------------------------------+
void fmx_prenorm_range(
                       double &fmx[][], 
                       double xrange[], 
                       double xmidrange[] 
                       )
{
  int    i;
  int    j;
  double  f;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);

  for (i = 0; i < columns; i++)
    for (j = 0; j < rows; j++)
    {
	    f = fmx[j][i];
	      if (xrange[i] != 0)
		    fmx[j][i] = (f-xmidrange[i])/(xrange[i]/2);
	      else
		    fmx[j][i] = f-xmidrange[i];

	  }
}

//+------------------------------------------------------------------+
//|                                                                  
//+------------------------------------------------------------------+
void fmx_midrange(double  &xmidrange[],double fmx[][])
{
  int    i;
  int    j;
  double smax = -DBL_MAX;
  double smin = DBL_MAX;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);
  ArrayResize(xmidrange,columns);
  for (i = 0; i < columns; i++)
  {
     smax = -DBL_MAX;
     smin = DBL_MAX;
     for (j = 0; j < rows; j++)
	  {
	      smax=MAX(fmx[j][i],smax);
         smin=MIN(fmx[j][i],smin);
     }  
	  xmidrange[i] = (smax+smin)/2;
  }
}
//+------------------------------------------------------------------+
//|                                                                  
//+------------------------------------------------------------------+
void fmx_range(double  &xrange[], double fmx[][])
{
  int    i;
  int    j;
  double smax = -DBL_MAX;
  double smin = DBL_MAX;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);
  ArrayResize(xrange,columns);
  for (i = 0; i < columns; i++)
  {
     smax = -DBL_MAX;
     smin = DBL_MAX;
     for (j = 0; j < rows; j++)
	  {
	     smax=MAX(fmx[j][i],smax);
        smin=MIN(fmx[j][i],smin);
     }  
	  xrange[i] = smax-smin;
   }

}


//+------------------------------------------------------------------+
//|                                                                  
//+------------------------------------------------------------------+
void fmx_max(double &xrange[],double fmx[][])
{
  int    i;
  int    j;
  double smax = -DBL_MAX;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);

  ArrayResize(xrange,columns);
  for (i = 0; i < columns; i++)
  {
    smax = -DBL_MAX;
	 for (j = 0; j < rows; j++)
	 {
	   smax=MAX(fmx[j][i],smax);
    }  
    xrange[i] = smax;
  }
}

//+------------------------------------------------------------------+
//|                                                                  
//+------------------------------------------------------------------+
void fmx_min(double &xrange[],double fmx[][])
{
  int    i;
  int    j;
  double smin = DBL_MAX;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);

  ArrayResize(xrange,columns);
  for (i = 0; i < columns; i++)
  {
    smin = DBL_MAX;
    for (j = 0; j < rows; j++)
	 {
      smin=MIN(fmx[j][i],smin);
    }  
    xrange[i] = smin;
  }
}

//+------------------------------------------------------------------+
//| Return mean values of columns in 'fmx'.                                                                  
//+------------------------------------------------------------------+
void fmx_mean(double &xmean[], double fmx[][])
{
  int    i;
  int    j;
  double s;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);

  ArrayResize(xmean,columns);
  if (rows > 1)
  {
    for (i = 0; i < columns; i++)
	 {
	   s = 0.0;
	   for (j = 0; j < rows; j++)
	     s += fmx[j][i];
		  xmean[i] = s/rows;
	 }
  }
  else
  {
    for (i = 0; i < columns; i++)
	   xmean[i] = fmx[0][i];
  }
}

//+------------------------------------------------------------------+
//| Set elements of 'vec' to 'value'.                                                                  
//+------------------------------------------------------------------+
void fvec_set(double &vec[], int len, double value)
{
  int i;
  ArrayResize(vec,len);  
  for (i = 0; i < len; i++)
    vec[i] = value;
}

//+------------------------------------------------------------------+
//|  Return standard deviations of columns in 'fmx'.
//+------------------------------------------------------------------+
void fmx_std(double  &std[], double fmx[][])
{
  int    i;
  int    j;
  double  xmean;
  double s;
  int rows = ArrayRange(fmx,0);
  int columns = ArrayRange(fmx,1);
  
  ArrayResize(std,columns);
  if (rows > 1)
  {  
    for (i = 0; i < columns; i++)
    {
	      s = 0.0;
	      for (j = 0; j < rows; j++)
		s += fmx[j][i];
	      xmean = s/rows;
	      s = 0.0;
	      for (j = 0; j < rows; j++)
		s += (fmx[j][i]-xmean)*(fmx[j][i]-xmean);
	      std[i] = MathSqrt(s/(rows-1));
    }
  }  
  else
   fvec_set(std, columns, 0.0);
}

//+------------------------------------------------------------------+
//| 1-normalize TRAIN lines from std and mean calculated from TRAIN lines
//| 1-normalize TEST lines from std and mean calculated from TEST lines
//| ..................................................................
//| 2-normalize TRAIN lines from std and mean calculated from TRAIN lines  
//| 2-normalize TEST lines from std and mean calculated from TRAIN lines    
//| ..................................................................
//| 3-normalize TRAIN lines from range and midrange calculated from TRAIN lines   
//| 3-normalize TEST lines from range and midrange calculated from TEST lines 
//| ..................................................................
//| 4-normalize TEST lines from range and midrange calculated from TRAIN lines                                                          
//+------------------------------------------------------------------+
void svm_normalize_sample(
								double &svm_samples_attrib[][],							
								int normalize
							 )
{

  double xmean[];   
  double std[];  
  double xrange[];  
  double xmidrange[];  

  
  if(normalize==1 || normalize==2)
  { //normalize TRAIN lines from std and mean calculated from TRAIN lines
    fmx_mean(xmean,svm_samples_attrib);
    fmx_std(std,svm_samples_attrib);
    fmx_prenorm(svm_samples_attrib,xmean, std);
  } 
  else if(normalize==3)
  { //normalize TRAIN lines from range and midrange calculated from TRAIN lines
    fmx_range(xrange,svm_samples_attrib);
    fmx_midrange(xmidrange,svm_samples_attrib);       
    fmx_prenorm_range(svm_samples_attrib,xrange, xmidrange);
  }

}

//+------------------------------------------------------------------+
//| 1-normalize TEST lines from std and mean calculated from TEST lines
//| 2-normalize TEST lines from std and mean calculated from TRAIN lines     
//| 3-normalize TEST lines from range and midrange calculated from TEST lines
//| 4-normalize TEST lines from range and midrange calculated from TRAIN lines                                                             
//+------------------------------------------------------------------+
void svm_normalize_test(
	                     double svm_samples_attrib[][],
								double &svm_test_attrib[][],								
								int normalize
							)		
							
{

  if(normalize==1)
  {  //normalize TEST lines from std and mean calculated from TEST lines
    double xmean2[]; 
    fmx_mean(xmean2,svm_test_attrib);
    double std2[];
    fmx_std(std2,svm_test_attrib);
    fmx_prenorm(svm_test_attrib,xmean2, std2);
  }                 
  else if(normalize==2)
  { //normalize TEST lines from std and mean calculated from TRAIN lines
    double xmean1[];
    fmx_mean(xmean1,svm_samples_attrib);
    double std1[];
    fmx_std(std1,svm_samples_attrib);
    fmx_prenorm(svm_test_attrib, xmean1, std1);
  }
  else if(normalize==3)
  { //normalize TEST lines from range and midrange calculated from TEST lines
    double xrange2[];
    fmx_range(xrange2,svm_test_attrib);
    double xmidrange2[];
    fmx_midrange(xmidrange2,svm_test_attrib);   
    fmx_prenorm_range(svm_test_attrib, xrange2, xmidrange2);
  }
  else if(normalize==4)
  { //normalize TEST lines from range and midrange calculated from TRAIN lines
    double xrange1[];
    fmx_range(xrange1,svm_samples_attrib);
    double xmidrange1[];
    fmx_midrange(xmidrange1,svm_samples_attrib);                                       
    fmx_prenorm_range(svm_test_attrib, xrange1, xmidrange1);
  }
  
}

Similar scripts

MatrixInd_1

The code implements a multi-dimensional data normalization engine designed to standardize training and testing matrices using Z-score (mean/standard deviation) or Min-Max range scaling techniques for Support Vector Machine compatibility.

MatrixInd_3

The indicator extracts historical price data across multiple currency pairs to perform multi-dimensional matrix normalization using standard deviation and range-based scaling techniques.

MatrixInd_2

This indicator implements multi-dimensional matrix normalization techniques to standardize price data across multiple assets for Support Vector Machine (SVM) compatibility.

Matrix2

This script performs multi-dimensional matrix normalization using Z-score standard deviation or midrange range scaling to prepare raw numerical data for Support Vector Machine (SVM) inputs.

AttribInd_4e

AttribInd_3e_1

AttribInd_3e

AttribInd_2e

Matrix3

This script performs multi-dimensional array normalization using Z-score standardization and min-max range scaling to prepare training and testing datasets for Support Vector Machine architectures.

Comments