Methods of Solving the Methods of Solving the Problem of Construction the Problem of Construction the Optimum Regression Model Optimum Regression Model as a Discrete Optimization as a Discrete Optimization

TaskTaskIvan MelnykIvan Melnyk

The International Research and Training Centre of The International Research and Training Centre of Information Technologies and Systems of the Information Technologies and Systems of the National Academy of Sciences of Ukraine and National Academy of Sciences of Ukraine and Ministry of Science and Education of UkraineMinistry of Science and Education of UkraineAkademik Glushkov prospect, 40, Kyiv 03680, Akademik Glushkov prospect, 40, Kyiv 03680,

UkraineUkraine astridastrid@@irtcirtc..orgorg..uaua

1. Introduction1. Introduction Construction the optimum regression model is a Construction the optimum regression model is a

special task to select from the set of independent special task to select from the set of independent variables (regressors) a subset for constructing the variables (regressors) a subset for constructing the regression model using the proper function. This is regression model using the proper function. This is a multiextremal task of the discrete optimization. a multiextremal task of the discrete optimization. There are no effective algorithms for decision such There are no effective algorithms for decision such a task.a task.

In this article the problem is formulated as a task of In this article the problem is formulated as a task of discrete optimization on a special graph. The exact discrete optimization on a special graph. The exact method of solving this task is proposed as the method of solving this task is proposed as the special task for searching the shortest path on this special task for searching the shortest path on this graph, and also as a heuristic method based on the graph, and also as a heuristic method based on the ideas of classical genetic algorithm proposed in the ideas of classical genetic algorithm proposed in the Michigan University by J.H. Holland.Michigan University by J.H. Holland.

2. The Tasks of Research2. The Tasks of Research A complex system having input (independent) A complex system having input (independent)

variables and one output (dependent) variable is variables and one output (dependent) variable is considered to have a stochastic character and considered to have a stochastic character and given by a sample of given by a sample of n n observations. The observations. The structures of linear regression models with structures of linear regression models with parameters estimated by the least-squares parameters estimated by the least-squares method are generated in the process of method are generated in the process of identication. Some functional of evaluation the identication. Some functional of evaluation the model Fmodel F ( . ) ( . ) is set as the criterion for selection is set as the criterion for selection the optimum model. This functional can depend the optimum model. This functional can depend from the model complexity (numbers of the from the model complexity (numbers of the estimated parameters) and/or the disparity of a estimated parameters) and/or the disparity of a regression equation on some subset of the regression equation on some subset of the statistical sample and other parameters of model. statistical sample and other parameters of model.

For example, let us designate the For example, let us designate the coefficients of a regression model found coefficients of a regression model found by least-squares method with subset of by least-squares method with subset of numbers of input variables numbers of input variables and and the the “charges” for receiving the statistical “charges” for receiving the statistical information about the independent information about the independent variable value. The following functional variable value. The following functional of model estimation is considered of model estimation is considered ((11))::

m

k Ji

kii

k

Jiii xJayCJixFJF

1

2 min))((),()(

Second option, when the functional is defined as a Second option, when the functional is defined as a maximal value of the residual sum of squares of the maximal value of the residual sum of squares of the regression model on a part of the set. A next criterion regression model on a part of the set. A next criterion to choose regressive model is considered furtherto choose regressive model is considered further. . Great number of points of statistical selection Great number of points of statistical selection is is broken-down into two partsbroken-down into two parts: : and and , , that is typical for that is typical for GMDHGMDH. . On the first part of selection a regressive On the first part of selection a regressive model is built, and the value of functional of model is built, and the value of functional of estimation of the built model is calculated on the estimation of the built model is calculated on the second part second part ((22))::

Ji

kii

k

SkxJayJF 2))((max)(

2 where where – – part from the great number of numbers of part from the great number of numbers of

independent variables independent variables , , and is and is - - part of input part of input variables which a regressive model is built onvariables which a regressive model is built on. .

The value to functional (2) is calculated as The value to functional (2) is calculated as a maximal value to the square of a maximal value to the square of differences between the statistical values differences between the statistical values of input variable and value of regression of input variable and value of regression model on all observations of values of input model on all observations of values of input variables of variables of ..

It is necessary to select such a subset of It is necessary to select such a subset of variables from the whole set of input variables from the whole set of input variables when the regression model built variables when the regression model built on the basis of this subset produce the on the basis of this subset produce the optimum (minimum or maximum) of a optimum (minimum or maximum) of a given functional of model estimation F( . ). given functional of model estimation F( . ). We determine the We determine the optimum modeloptimum model as the as the proper regression model which contains proper regression model which contains only the variables (regressors) from the only the variables (regressors) from the chosen subset.chosen subset.

3. The Task Solving3. The Task Solving3.1. The exact solving3.1. The exact solving methodsmethods

It is proposed to formulate and solve the task of It is proposed to formulate and solve the task of selection the optimum regression model as a task selection the optimum regression model as a task of discrete optimization with Boolean variables on of discrete optimization with Boolean variables on special graph special graph (I, U).(I, U). The set I of the graph nodes The set I of the graph nodes is formed in such a way. The graph node i is set in is formed in such a way. The graph node i is set in accordance with any independent variable accordance with any independent variable Xi Xi (i=1,...m).(i=1,...m). Two other nodes are added: the node Two other nodes are added: the node (zero) and the node (zero) and the node m+1m+1. The set of arcs . The set of arcs UU of the of the graph is built as follows. The node graph is built as follows. The node 00 is linked to is linked to every node every node i i (i=1,...m)(i=1,...m) and arc and arc (0,i).(0,i). Further, Further, every node i is connected with every node every node i is connected with every node j, j>ij, j>i, , arc arc (i, j)(i, j). .

It is considered that a node It is considered that a node ii Є Є I (i=1,...,m)I (i=1,...,m) corresponds to the situation when an independent corresponds to the situation when an independent variable Xi is included into regression model. So, any variable Xi is included into regression model. So, any variant of construction of regression model variant of construction of regression model corresponds to the path from the node 0 to the corresponds to the path from the node 0 to the node m+1. The “length” of every such path μ node m+1. The “length” of every such path μ corresponds to the value of the estimation functional corresponds to the value of the estimation functional of the proper model F( of the proper model F( .. ). The initial task of ). The initial task of construction of optimum regression model (task of construction of optimum regression model (task of selection subset of regressors) reduces to finding selection subset of regressors) reduces to finding the shortest path from the node 0the shortest path from the node 0 to the node to the node m+1m+1 on the graph on the graph (I, U).(I, U).

Graph Graph (I, U)(I, U) allows to organize the searching allows to organize the searching variants for selection the optimum regression model variants for selection the optimum regression model consistently by an economical procedure. Finding consistently by an economical procedure. Finding the shortest path on the built graph the shortest path on the built graph (I, U)(I, U) gives the gives the solution of basic tasks of selection the optimum solution of basic tasks of selection the optimum regression model. regression model.

It is considered that a node corresponds to the It is considered that a node corresponds to the situation when an independent variable is included situation when an independent variable is included into regression model. So, any variant of into regression model. So, any variant of construction of regression model corresponds to construction of regression model corresponds to the path from the node to the node . The “length” the path from the node to the node . The “length” of every such path corresponds to the value of the of every such path corresponds to the value of the estimation functional of the proper model F( estimation functional of the proper model F( .. ). The ). The initial task of construction of optimum regression initial task of construction of optimum regression model (task of selection subset of regressors) model (task of selection subset of regressors) reduces to finding the shortest path from the node reduces to finding the shortest path from the node to the node on the graph . to the node on the graph .

Graph allows to organize the searching variants Graph allows to organize the searching variants for selection the optimum regression model for selection the optimum regression model consistently by an economical procedure. Finding consistently by an economical procedure. Finding the shortest path on the built graph gives the the shortest path on the built graph gives the solution of basic tasks of selection the optimum solution of basic tasks of selection the optimum regression model. regression model.

3.2. The genetic algorithm 3.2. The genetic algorithm of the task solvingof the task solving

The genetic algorithm of decision short cut search The genetic algorithm of decision short cut search task is offered on a graph as a method of task is offered on a graph as a method of decision of task of construction of optimum decision of task of construction of optimum regression model, based on the ideas of classic regression model, based on the ideas of classic genetic algorithmgenetic algorithm. . According to the terminology According to the terminology and definitions of the genetic algorithms theory, and definitions of the genetic algorithms theory, every path from the node to the node on graph every path from the node to the node on graph (that is variant of solving the regression model (that is variant of solving the regression model selection), which the Boolean vector of dimension selection), which the Boolean vector of dimension corresponds to, is the chromosome of proper corresponds to, is the chromosome of proper population and every element of this Boolean population and every element of this Boolean vector is the gene of this chromosome. vector is the gene of this chromosome.

We will define the function of adaptabilityWe will define the function of adaptability PP()() for for every chromosome on the basis of the special every chromosome on the basis of the special goal function F( . ) of all the chromosomes of goal function F( . ) of all the chromosomes of population as the following:Ppopulation as the following:P()() = F( )/ (SUM F( = F( )/ (SUM F( .. ) – ) – FF(()), (3) )), (3)

where SUM F( where SUM F( . . ) is an operator of summing for all ) is an operator of summing for all the chromosomes of population.the chromosomes of population.

It is possible to proof that if some genotype It is possible to proof that if some genotype (chromosome) corresponds to a local (global) (chromosome) corresponds to a local (global) optimum for the basic task of selection the optimum for the basic task of selection the optimum regression model (for the special goal optimum regression model (for the special goal function Ffunction F( . )( . ))) then this genotype then this genotype ( (chromosomes) chromosomes) gives the local (global) optimum for the gives the local (global) optimum for the adjustment function Padjustment function P( . ) ( . ) also and vice versa. also and vice versa. And also if And also if 11 and and 22 are the chromosomes of are the chromosomes of proper population and F(proper population and F(11) < F() < F(22) then F) then F((11)/ F()/ F(22) ) << P P ((11) / P () / P (22)). .

The offered genetic method is an iterative algorithmic The offered genetic method is an iterative algorithmic process. Lets describe iteration process. Lets describe iteration (t=1,2,...)(t=1,2,...) calculation calculation process of this algorithm process of this algorithm . . After implementation of After implementation of previous iteration previous iteration t-1t-1, , for implementation of iteration for implementation of iteration we have the great numbers of population we have the great numbers of population mm of types of types:. :.

W1(t-1), W2(t-1),...,Wk(t-1),...Wm(t-1).W1(t-1), W2(t-1),...,Wk(t-1),...Wm(t-1). In the offered models of genetic algorithm the In the offered models of genetic algorithm the

procedure of constructions of initial great number of procedure of constructions of initial great number of population of type population of type Wk(0), (k=1,...,m) Wk(0), (k=1,...,m) is important is important. . Great numbers of population Great numbers of population W1(0), W1(0), W2(0),...,Wk(0),...Wm(0) W2(0),...,Wk(0),...Wm(0) for realization of first for realization of first iteration iteration (t=1)(t=1) are determined by the proper are determined by the proper procedure of determination of these initial great procedure of determination of these initial great numbers, which is carried out by the special variant numbers, which is carried out by the special variant of method marks [5] on partial graphs (Iof method marks [5] on partial graphs (Ikk, U, Ukk), with ), with the use of probable sensors for „equalizing” of the use of probable sensors for „equalizing” of variants of decisions (chromosomes) for these great variants of decisions (chromosomes) for these great numbers [6]numbers [6]..

STEP І (STEP І (Large cycle).Large cycle).

Choice of the type of population . Choice of the type of population . The population is the set of The population is the set of chromosomes (paths) which chromosomes (paths) which necessarily contain a node (variable) necessarily contain a node (variable) with the numbers and larger than .with the numbers and larger than .

STEPSTEP ІІ ІІ ((Small cycleSmall cycle)).. Step 1Step 1. Presentation for iteration . Presentation for iteration tt of calculation of calculation

genetic process of great number genetic process of great number Wk(t-1)Wk(t-1) of population of population of type of type kk, which contains , which contains NkNk persons (chromosomes). persons (chromosomes). Calculation of values of the goal function F( . ) and the Calculation of values of the goal function F( . ) and the function of adaptability P(.) for populations function of adaptability P(.) for populations (chromosomes) from the great number (chromosomes) from the great number Wk(0).Wk(0).

Step 2Step 2. Choice of „parents” from the great number . Choice of „parents” from the great number Wk(t-1)Wk(t-1) of populations (chromosomes) of type of populations (chromosomes) of type kk on on the basis of combination of two principlesthe basis of combination of two principles: : near and near and distant „cognation”. Under „cognation” of distant „cognation”. Under „cognation” of chromosomes „parents” means distance between the chromosomes „parents” means distance between the members of population in the value of measure members of population in the value of measure Hamming (hamming distance) between the sets (by Hamming (hamming distance) between the sets (by the vectors of Boolean variables) of chromosomes the vectors of Boolean variables) of chromosomes „parents”.„parents”.

STEPSTEP ІІ ІІ ((Small cycleSmall cycle)).. Step 3Step 3.. The crossing over pairs of chromosomes The crossing over pairs of chromosomes

of “parents” for the generation of chromosomes of “parents” for the generation of chromosomes of “posterity’s”.of “posterity’s”.

Step 4Step 4. Calculation of values of the goal function . Calculation of values of the goal function F( . ) and the function of adaptability P( . ) for the F( . ) and the function of adaptability P( . ) for the chromosomes of “posterity’s”.chromosomes of “posterity’s”.

Step 5Step 5. Excepting the procedure of mutation for . Excepting the procedure of mutation for the chromosomes of “posterity’s” for the the chromosomes of “posterity’s” for the generation of “mutants” chromosomes.generation of “mutants” chromosomes.

Step Step 66. Calculation of values of goal function F. Calculation of values of goal function F( . ( . )) and function of adaptability P and function of adaptability P(.)(.) for the for the “mutants” chromosomes.“mutants” chromosomes.

Step Step 77. Checking end of process of calculating of . Checking end of process of calculating of the procedures of “crossing over pairs” and the procedures of “crossing over pairs” and iteration pf the type .iteration pf the type .

STEPSTEP ІІ ІІ ((Small cycleSmall cycle))..

Step 8Step 8. Conducting of the procedure of migration . Conducting of the procedure of migration between by the great numbers of population of between by the great numbers of population of type type kk and other types. and other types.

Step 9Step 9. Calculation for chromosomes „migrants” . Calculation for chromosomes „migrants” of function having a special purpose and function of function having a special purpose and function of adaptability P( . ). of adaptability P( . ).

Step 10Step 10. . Conducting the selection procedure for Conducting the selection procedure for the generation of great numberthe generation of great number Wk(t)Wk(t)..

Step 11Step 11. Transition to begin small cycle for . Transition to begin small cycle for choosing next type of population (with verification choosing next type of population (with verification of condition of condition kk<<mm). ). If If k=mk=m, , we pass to the large we pass to the large cycle for realization of next iteration cycle for realization of next iteration (t+1).(t+1).

The model of genetic algorithm offered The model of genetic algorithm offered above is a so called „island model” [7], in above is a so called „island model” [7], in which the general great number of which the general great number of population is broken down into several population is broken down into several different parts (islands) of types of different parts (islands) of types of population. Each of these parts of general population. Each of these parts of general great number of population will separately great number of population will separately develop on the basis of a genetic develop on the basis of a genetic algorithm. The proper procedure of algorithm. The proper procedure of migration will take place between them . migration will take place between them . The ”island„ model allows to start a The ”island„ model allows to start a genetic algorithm at once several times for genetic algorithm at once several times for the best global decision. the best global decision.

44. . ConclusionConclusion Presentation of task of search of optimum regression Presentation of task of search of optimum regression

model on special graph allows the task of discrete model on special graph allows the task of discrete optimization effectively to use the tool of technology optimization effectively to use the tool of technology of search of short cut on this graph.of search of short cut on this graph.

For arbitrary numbers of primary variables For arbitrary numbers of primary variables (regressors), including the enough large number, a (regressors), including the enough large number, a special heuristic method is offered. The proposed special heuristic method is offered. The proposed genetic algorithm of solving the task of selection the genetic algorithm of solving the task of selection the optimum regression model does not guarantee the optimum regression model does not guarantee the exact solution but gives an enough satisfactory one.exact solution but gives an enough satisfactory one.

The offered genetic algorithm of decision of task of The offered genetic algorithm of decision of task of construction of optimum regression model from the construction of optimum regression model from the class of so called «island models» allows effectively to class of so called «island models» allows effectively to decide the general global task of optimization by decide the general global task of optimization by decomposition of calculation process of its decision decomposition of calculation process of its decision into process of decision of separate local tasks.into process of decision of separate local tasks.

ReferencesReferences1.1. Holland J. H.: Adaptation in Natural and Artificial Systems. – Ann Holland J. H.: Adaptation in Natural and Artificial Systems. – Ann

Arbor: Michigan University PressArbor: Michigan University Press, 1975., 1975.2.2. Goldberg D. S.: Genetic algorithms in search, optimization, and Goldberg D. S.: Genetic algorithms in search, optimization, and

machine learning. – Reading, MA: Addison-Wesley, 1989. machine learning. – Reading, MA: Addison-Wesley, 1989. 3.3. Melnyk I.MMelnyk I.M.., Stepashko V.S.: , Stepashko V.S.: On an approach to the optimum On an approach to the optimum

regression model selection using the discrete optimization regression model selection using the discrete optimization method. – Proceedings of the International Workshop on Inductive method. – Proceedings of the International Workshop on Inductive Modelling. Modelling. – – KyivKyiv: : IRTC ITSIRTC ITS, 2005. – , 2005. – PP. 223-229.. 223-229. (In Ukrainian) (In Ukrainian)

4.4. Ivakhnenko A. G., Stepashko V.S.: Pomekhoustoichivost Ivakhnenko A. G., Stepashko V.S.: Pomekhoustoichivost modelirovania (Noise-immunity of modeling). Kiev: Naukova modelirovania (Noise-immunity of modeling). Kiev: Naukova Dumka, 216 p, 1985. (In Russian)Dumka, 216 p, 1985. (In Russian)

5.5. Yermoliev Yu.MYermoliev Yu.M., ., Melnyk I.MMelnyk I.M. . Extremalnye zadachi na grafakh Extremalnye zadachi na grafakh ((Extreme tasks on the graphsExtreme tasks on the graphs). – ). – Kiev: Naukova Dumka, Kiev: Naukova Dumka, 1967. – 1967. – 159 с.159 с. (In Russian)(In Russian)

6.6. Melnyk I.MMelnyk I.M. . Genetic algorithm for solving the task of an optimum Genetic algorithm for solving the task of an optimum regression model construction as a discrete optimization problem . regression model construction as a discrete optimization problem . – Problems of Control and Informatics– Problems of Control and Informatics. – 2008. . – 2008. –– №3. – №3. – PP.30-43. .30-43. (In (In Russian)Russian)

7.7. Whitley W.D., Rana S.B., and Heckendorn R.B. Island model Whitley W.D., Rana S.B., and Heckendorn R.B. Island model genetic algorithms and linearly separable problems. – Selected genetic algorithms and linearly separable problems. – Selected Papers from AISB Workshop on Evolutionary Computing. – London: Papers from AISB Workshop on Evolutionary Computing. – London: Springer-Verlag, 1997. – P. 109-125. Springer-Verlag, 1997. – P. 109-125.

Thank you!Thank you!