This script is designed to perform data normalization—a mathematical process used to bring different sets of numbers into a common scale. This is a standard procedure in machine learning and statistical analysis to ensure that variables with large values do not disproportionately influence the results over variables with smaller values.
The logic follows a structured workflow:
1. Data Initialization
The script begins by creating two distinct groups of data: "Samples" (intended as training data) and "Test" data. Each group contains multiple rows of information, where each row represents a set of attributes. For the sake of demonstration, the script populates these with randomized numerical values.
2. Normalization Methods
The core functionality relies on two different mathematical approaches to "level" the data:
- Standard Deviation Method (Mean/Std): This centers the data around an average (mean) and scales it based on how much the numbers vary (standard deviation). It essentially tells you how many "steps" away from the average a specific value is.
- Range Method (Midrange/Range): This identifies the highest and lowest values in a set to determine the total spread. It then centers the data around the midpoint of that range and scales it relative to the size of the total spread.
3. Logic Execution Modes
The script provides four specific modes of operation, governed by a user-selectable setting. These modes determine how the "Test" data is handled in relation to the "Sample" data:
- Independent Normalization (Mode 1 & 3): The test data is treated as its own entity. It calculates its own averages and ranges independently before being scaled.
- Relative Normalization (Mode 2 & 4): The test data is transformed using the mathematical "rules" derived from the sample data. This is a crucial step in many predictive models where you want to ensure that new, unseen data is measured against the exact same benchmarks used during the training phase.
4. Operational Workflow
When executed, the script performs the following sequence:
- Generates the raw datasets.
- Displays the initial "Sample" data.
- Applies the chosen normalization logic to the "Sample" data and displays the result.
- Applies the corresponding normalization logic to the "Test" data (either independently or relative to the samples) and displays the final transformation.
In summary, this script is a utility for preparing raw numerical information so that it can be processed accurately by more complex mathematical models, ensuring that all attributes are compared on a fair and consistent playing field.
5
Views
0
Downloads
0
Favorites
Matrix3
//+------------------------------------------------------------------+
//| Matrix1.mq4 |
//| Copyright © 2008, MetaQuotes Software Corp. |
//| http://www.metaquotes.net |
//+------------------------------------------------------------------+
#property copyright "Copyright © 2008, MetaQuotes Software Corp."
#property link "http://www.metaquotes.net"
#define DBL_MAX 99999999999999999999
#define LIN1 5 //sample lines
#define LIN2 4 //test lines
#define COL 3 //attributes
#property show_inputs
extern int normalize=1;
double M_sample[LIN1][COL];
double M_test[LIN2][COL];
string m1;
int init()
{
m1="";
MathSrand(TimeLocal());
return(0);
}
//+------------------------------------------------------------------+
//| script program start function |
//+------------------------------------------------------------------+
int start()
{
//----
for(int i=0;i<LIN1;i++)
for(int j=0;j<COL;j++)
M_sample[i][j]=MathRand();
for(i=0;i<LIN2;i++)
for(j=0;j<COL;j++)
M_test[i][j]=MathRand();
PrintMatrixToStr(M_sample);
svm_normalize_sample(M_sample,normalize);
PrintMatrixToStr(M_sample);
PrintMatrixToStr(M_test);
svm_normalize_test(M_sample,M_test,normalize);
PrintMatrixToStr(M_test);
Comment(m1);
//----
return(0);
}
//+------------------------------------------------------------------+
void PrintMatrixToStr(double Matrix[][])
{
int Line = ArrayRange(Matrix,0);
int Col = ArrayRange(Matrix,1);
for (int i=0;i<Line;i++)
{
for (int j=0;j<Col;j++)
m1=m1+Matrix[i][j]+",";
m1=m1+"\n";
}
m1=m1+"\n";
}
//+------------------------------------------------------------------+
//|
//+------------------------------------------------------------------+
double MAX(double x, double y)
{
if(x>y)
return(x);
else
return(y);
}
//+------------------------------------------------------------------+
//|
//+------------------------------------------------------------------+
double MIN(double x, double y)
{
if(x<y)
return(x);
else
return(y);
}
//+------------------------------------------------------------------+
//| Normalize matrix 'fmx' using precomputed 'xmean', 'std'. The
//| normalization is: for each column, subtract 'xmean' and, if the
//| column 'std' is non-zero, divide by 'std'.
//| d=nr_of_attrib
//| nv=nr_of_sample_lines
//| double **x --> double **svm_samples_attrib_scaled_dyn_
//| double *xmean = fmx_mean(x2, nv, d);
//| double *std = fmx_std(x2, nv, d);
//| fmx_prenorm(x, nv, d, xmean, std);
//| X - mean
//| i
//| S = ---------
//| i std
//+------------------------------------------------------------------+
void fmx_prenorm(
double &fmx[][],
double xmean[], // double *xmean = fmx_mean(x2, nv, d);
double std[] // double *std = fmx_std(x2, nv, d);
)
{
int i;
int j;
double f;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
for (i = 0; i < columns; i++)
for (j = 0; j < rows; j++)
{
f = fmx[j][i];
if (std[i] != 0)
fmx[j][i] = (f-xmean[i])/std[i];
else
fmx[j][i] = f-xmean[i];
}
}
//+------------------------------------------------------------------+
//| max X + min X
//| i i i i
//| midrange = ----------------
//| 2
//| range = max X - min X
//| i i i i
//|
//| X - midrange
//| i
//| S = -------------
//| i range / 2
//+------------------------------------------------------------------+
void fmx_prenorm_range(
double &fmx[][],
double xrange[],
double xmidrange[]
)
{
int i;
int j;
double f;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
for (i = 0; i < columns; i++)
for (j = 0; j < rows; j++)
{
f = fmx[j][i];
if (xrange[i] != 0)
fmx[j][i] = (f-xmidrange[i])/(xrange[i]/2);
else
fmx[j][i] = f-xmidrange[i];
}
}
//+------------------------------------------------------------------+
//|
//+------------------------------------------------------------------+
void fmx_midrange(double &xmidrange[],double fmx[][])
{
int i;
int j;
double smax = -DBL_MAX;
double smin = DBL_MAX;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
ArrayResize(xmidrange,columns);
for (i = 0; i < columns; i++)
{
smax = -DBL_MAX;
smin = DBL_MAX;
for (j = 0; j < rows; j++)
{
smax=MAX(fmx[j][i],smax);
smin=MIN(fmx[j][i],smin);
}
xmidrange[i] = (smax+smin)/2;
}
}
//+------------------------------------------------------------------+
//|
//+------------------------------------------------------------------+
void fmx_range(double &xrange[], double fmx[][])
{
int i;
int j;
double smax = -DBL_MAX;
double smin = DBL_MAX;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
ArrayResize(xrange,columns);
for (i = 0; i < columns; i++)
{
smax = -DBL_MAX;
smin = DBL_MAX;
for (j = 0; j < rows; j++)
{
smax=MAX(fmx[j][i],smax);
smin=MIN(fmx[j][i],smin);
}
xrange[i] = smax-smin;
}
}
//+------------------------------------------------------------------+
//|
//+------------------------------------------------------------------+
void fmx_max(double &xrange[],double fmx[][])
{
int i;
int j;
double smax = -DBL_MAX;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
ArrayResize(xrange,columns);
for (i = 0; i < columns; i++)
{
smax = -DBL_MAX;
for (j = 0; j < rows; j++)
{
smax=MAX(fmx[j][i],smax);
}
xrange[i] = smax;
}
}
//+------------------------------------------------------------------+
//|
//+------------------------------------------------------------------+
void fmx_min(double &xrange[],double fmx[][])
{
int i;
int j;
double smin = DBL_MAX;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
ArrayResize(xrange,columns);
for (i = 0; i < columns; i++)
{
smin = DBL_MAX;
for (j = 0; j < rows; j++)
{
smin=MIN(fmx[j][i],smin);
}
xrange[i] = smin;
}
}
//+------------------------------------------------------------------+
//| Return mean values of columns in 'fmx'.
//+------------------------------------------------------------------+
void fmx_mean(double &xmean[], double fmx[][])
{
int i;
int j;
double s;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
ArrayResize(xmean,columns);
if (rows > 1)
{
for (i = 0; i < columns; i++)
{
s = 0.0;
for (j = 0; j < rows; j++)
s += fmx[j][i];
xmean[i] = s/rows;
}
}
else
{
for (i = 0; i < columns; i++)
xmean[i] = fmx[0][i];
}
}
//+------------------------------------------------------------------+
//| Set elements of 'vec' to 'value'.
//+------------------------------------------------------------------+
void fvec_set(double &vec[], int len, double value)
{
int i;
ArrayResize(vec,len);
for (i = 0; i < len; i++)
vec[i] = value;
}
//+------------------------------------------------------------------+
//| Return standard deviations of columns in 'fmx'.
//+------------------------------------------------------------------+
void fmx_std(double &std[], double fmx[][])
{
int i;
int j;
double xmean;
double s;
int rows = ArrayRange(fmx,0);
int columns = ArrayRange(fmx,1);
ArrayResize(std,columns);
if (rows > 1)
{
for (i = 0; i < columns; i++)
{
s = 0.0;
for (j = 0; j < rows; j++)
s += fmx[j][i];
xmean = s/rows;
s = 0.0;
for (j = 0; j < rows; j++)
s += (fmx[j][i]-xmean)*(fmx[j][i]-xmean);
std[i] = MathSqrt(s/(rows-1));
}
}
else
fvec_set(std, columns, 0.0);
}
//+------------------------------------------------------------------+
//| 1-normalize TRAIN lines from std and mean calculated from TRAIN lines
//| 1-normalize TEST lines from std and mean calculated from TEST lines
//| ..................................................................
//| 2-normalize TRAIN lines from std and mean calculated from TRAIN lines
//| 2-normalize TEST lines from std and mean calculated from TRAIN lines
//| ..................................................................
//| 3-normalize TRAIN lines from range and midrange calculated from TRAIN lines
//| 3-normalize TEST lines from range and midrange calculated from TEST lines
//| ..................................................................
//| 4-normalize TEST lines from range and midrange calculated from TRAIN lines
//+------------------------------------------------------------------+
void svm_normalize_sample(
double &svm_samples_attrib[][],
int normalize
)
{
double xmean[];
double std[];
double xrange[];
double xmidrange[];
if(normalize==1 || normalize==2)
{ //normalize TRAIN lines from std and mean calculated from TRAIN lines
fmx_mean(xmean,svm_samples_attrib);
fmx_std(std,svm_samples_attrib);
fmx_prenorm(svm_samples_attrib,xmean, std);
}
else if(normalize==3)
{ //normalize TRAIN lines from range and midrange calculated from TRAIN lines
fmx_range(xrange,svm_samples_attrib);
fmx_midrange(xmidrange,svm_samples_attrib);
fmx_prenorm_range(svm_samples_attrib,xrange, xmidrange);
}
}
//+------------------------------------------------------------------+
//| 1-normalize TEST lines from std and mean calculated from TEST lines
//| 2-normalize TEST lines from std and mean calculated from TRAIN lines
//| 3-normalize TEST lines from range and midrange calculated from TEST lines
//| 4-normalize TEST lines from range and midrange calculated from TRAIN lines
//+------------------------------------------------------------------+
void svm_normalize_test(
double svm_samples_attrib[][],
double &svm_test_attrib[][],
int normalize
)
{
if(normalize==1)
{ //normalize TEST lines from std and mean calculated from TEST lines
double xmean2[];
fmx_mean(xmean2,svm_test_attrib);
double std2[];
fmx_std(std2,svm_test_attrib);
fmx_prenorm(svm_test_attrib,xmean2, std2);
}
else if(normalize==2)
{ //normalize TEST lines from std and mean calculated from TRAIN lines
double xmean1[];
fmx_mean(xmean1,svm_samples_attrib);
double std1[];
fmx_std(std1,svm_samples_attrib);
fmx_prenorm(svm_test_attrib, xmean1, std1);
}
else if(normalize==3)
{ //normalize TEST lines from range and midrange calculated from TEST lines
double xrange2[];
fmx_range(xrange2,svm_test_attrib);
double xmidrange2[];
fmx_midrange(xmidrange2,svm_test_attrib);
fmx_prenorm_range(svm_test_attrib, xrange2, xmidrange2);
}
else if(normalize==4)
{ //normalize TEST lines from range and midrange calculated from TRAIN lines
double xrange1[];
fmx_range(xrange1,svm_samples_attrib);
double xmidrange1[];
fmx_midrange(xmidrange1,svm_samples_attrib);
fmx_prenorm_range(svm_test_attrib, xrange1, xmidrange1);
}
}
Comments