Upload
lauren-simon
View
217
Download
0
Embed Size (px)
Citation preview
Chapter 13.Chapter 13.General Least-Squares General Least-Squares
and Nonlinear and Nonlinear RegressionRegression
Gab-Byung ChaeGab-Byung Chae
13.1 Polynomial 13.1 Polynomial RegressionRegression
-> poorly represented by a straight line.
As discussed in Chap. 12, one method to accomplish this objective is to use transformations.
Another alternative is to fit polynomials to the data using polynomial regression.
Least Squares Least Squares RegressionRegression
Minimize some measure of the difference between Minimize some measure of the difference between the approximating function and the given data the approximating function and the given data points.points.
In least squares method, the error is measured as :In least squares method, the error is measured as :
The minimum of The minimum of EE occurs when the partial occurs when the partial derivatives of derivatives of EE with respect to each of the with respect to each of the variables are 0.variables are 0.
2
1
))(( ii
n
i
yxfE
,...0,0
b
E
a
E
Linear Least Squares Linear Least Squares RegressionRegression
f(x) f(x) is in a linear form : is in a linear form : f(x)=ax+bf(x)=ax+b the error :the error :
Is minimized when :Is minimized when :
2
1
)( ii
n
i
ybxaE
01)(2
0)(2
ii
iii
ybaxb
E
xybaxa
E
ii
iiii
ybxa
yxxbxa
1
2
xaybxxn
yxyxna
ii
iiii
,
)( 22
Quadratic Least Squares Quadratic Least Squares ApproximationApproximation
f(x) f(x) is in a quadratic form : is in a quadratic form : f(x)=axf(x)=ax22+bx+c+bx+c the error :the error :
Is minimized when :Is minimized when :
22
1
)( iii
n
i
ycbxxaE
0)(2
0)(2
01)(2
22
2
2
iiii
iiii
iii
xycbxaxa
E
xycbxaxb
E
ycbxaxc
E
iiiii
iiiii
iii
yxxaxbxc
yxxaxbxc
yxaxbnc
2432
32
2
Cubic Least Squares Cubic Least Squares ApproximationApproximation
f(x) f(x) is in a cubic form : is in a cubic form : f(x)=axf(x)=ax33+bx+bx22 +cx+d +cx+d the error :the error :
Is minimized when :Is minimized when :
223
1
)( iiii
n
i
ydcxbxxaE
01)(2
0)(2
0)(2
0)(2
23
23
223
323
iiii
iiiii
iiiii
iiiii
ydcxbxaxd
E
xydcxbxaxc
E
xydcxbxaxb
E
xydcxbxaxa
E
This case can be easily extended to an mth-order polynomial.
Determining the coefficients of an mth-order Determining the coefficients of an mth-order polynomial is equivalent to solving a system polynomial is equivalent to solving a system of m+1 simutaneous linear equations.of m+1 simutaneous linear equations.
The standard error is formulated as The standard error is formulated as
(m+1) data-drived coefficients- a(m+1) data-drived coefficients- a0, 0, aa1, … 1, … aam, m, - - were used to compute Swere used to compute Sr r ..
)1(/
mn
SS r
xy
Example 13.1Example 13.1Fit a second-order polynomial to the data Fit a second-order polynomial to the data
in the first two columns of Table 13.1in the first two columns of Table 13.1
Sol> Sol>
8.2488
6.585
6.152
97922555
2255515
55156
2
1
0
a
a
a
>> N = [6 15 55; 15 55 225; 55 225 979];>> N = [6 15 55; 15 55 225; 55 225 979];
>> r = [152.6 585.6 2488.8]>> r = [152.6 585.6 2488.8]
>> a = N\r>> a = N\r
a = a =
2.47862.4786
2.35932.3593
1.8607 1.8607
28607.13593.24786.2 xxy
99925.0
99851.039.2513
74657.339.2513
1175.1)12(6
74657.3
2
/
r
r
S xy
)( 2 yyS it22
1
)( iii
n
ir ycbxxaS
The standard error
The coefficient of determination
The correlation coefficient
t
rt
S
SSr
2
Sum of the squares of the residuals between the data points(yi) and the mean
Sum of the squares of the residuals between the data points(yi) and regression curve
Fit of a second-order polynomial.Figure 13.2
13.2 Multiple Linear 13.2 Multiple Linear RegressionRegression
An extension of linear regression : y is a An extension of linear regression : y is a linear function of two or more linear function of two or more independent variables.independent variables.
For this two-dimensional case, the For this two-dimensional case, the regression line becomes a plane(Fig. regression line becomes a plane(Fig. 13.3).13.3).
The sum of the squares of the residuals:The sum of the squares of the residuals:
exaxaay 22110
2,22,110
1
)( iii
n
ir xaxaayS
Graphical depiction of multiple linear regression where y is a linear function of x1 and x2.
Figure 13.3
0)(2
0)(2
0)(2
,22,110,22
,22,110,11
,22,1100
iiiir
iiiir
iiir
xaxaayxa
S
xaxaayxa
S
xaxaaya
S
ii
ii
i
iiii
iiii
ii
yx
yx
y
a
a
a
xxxx
xxxx
xxn
,2
,1
2
1
0
2,2,2,1,2
,2,12,1,1
,2,1
Example 13.2 Multiple Example 13.2 Multiple Linear RegressionLinear Regression
Use multiple Use multiple linear regression linear regression to fit this data. to fit this data.
ii
ii
i
iiii
iiii
ii
yx
yx
y
a
a
a
xxxx
xxxx
xxn
,2
,1
2
1
0
2,2,2,1,2
,2,12,1,1
,2,1
3,4,5
100
5.243
54
544814
4825.765.16
145.166
210
2
1
0
aaa
a
a
a
Which gives us
Extension to m Extension to m dimensionsdimensions
exaxaxaay mm 22110
)1(/
mn
SS r
xy
Power equations of the form Power equations of the form
mm
am
aa
xaxaxaay
xxxay m
logloglogloglog 22110
21021
Standard error
13.3 General Linear Least 13.3 General Linear Least SquaresSquares
mm
mm
m
mm
xzxzxzz
xzxzxzz
functionsbasismarezzwhere
ezazazazay
,,,,1
,,,,1
.1,.....,
)7.13(
2210
22110
0
221100
When
We have simple or multiple linear regression.
When
We have polynomial regression.
The functions can be highly nonlinear.The functions can be highly nonlinear. For example:For example:
OrOr
)1(
)sin()cos(
10
210
xaeay
xaxaay
Equation (13.7) can be expressed in matrix Equation (13.7) can be expressed in matrix notation as notation as
where m is the number of variables in the where m is the number of variables in the model and n is the number of data points. model and n is the number of data points.
mnnn
m
m
zzz
zzz
zzz
Z
functionsbasistheof
valuescalculatedtheofmatrixaisZwhere
eaZy
10
21202
11101
Because n>m, mostly Z is not a square Because n>m, mostly Z is not a square matrix.matrix.
n
i
m
jjijir
mT
mT
nT
zayS
residualtheofsquarestheofsumThe
eeee
evectorcolumnThe
yaaa
avectorcolumnThe
yyyy
yvectorcolumnThe
1
2
0
21
21
21
)(
:
:
:
:
By taking its partial derivative with respect By taking its partial derivative with respect to each of the coefficients and setting the to each of the coefficients and setting the resulting equation equal to zero.resulting equation equal to zero.
aZy
asformvectorinwrittenbecanyy
fitsquaresleasttheofpredictiontheywhere
yy
yyr
S
S
S
SSr
yZaZZ
i
i
i
t
r
t
rt
TT
.
)(
)(1
1
2
22
2
Example 13.3 Example 13.3 Polynomial Regression with Polynomial Regression with
MATLABMATLABRepeat example 13.1 Repeat example 13.1 >> x=[0 1 2 3 4 5]';>> x=[0 1 2 3 4 5]';>> y=[2.1 7.7 13.6 27.2 40.9 61.1]';>> y=[2.1 7.7 13.6 27.2 40.9 61.1]';>> Z=[ones(size(x)) x x.^2]>> Z=[ones(size(x)) x x.^2]
Z =Z =
1 0 01 0 0 1 1 11 1 1 1 2 41 2 4 1 3 91 3 9 1 4 161 4 16 1 5 251 5 25
>> Z'*Z>> Z'*Z
ans =ans =
6 15 556 15 55 15 55 22515 55 225 55 225 97955 225 979
>> a=(Z'*Z)\(Z'*y)>> a=(Z'*Z)\(Z'*y)
aa = =
2.47862.4786 2.35932.3593 1.86071.8607
>> Sr = sum((y-Z*a).^2)>> Sr = sum((y-Z*a).^2)
Sr =Sr =
3.74663.7466
>> r2=1-Sr/sum((y-mean(y)).^2)>> r2=1-Sr/sum((y-mean(y)).^2)
r2 =r2 =
0.99850.9985
>> Syx= sqrt(Sr/(length(x)-length(a)))>> Syx= sqrt(Sr/(length(x)-length(a)))
Syx =Syx =
1.11751.1175
13.4 QR factorization and 13.4 QR factorization and the backslash operator the backslash operator
QR factorization and singular value QR factorization and singular value decomposition : beyond the scope of this decomposition : beyond the scope of this book but we can use it in MATLAB which is book but we can use it in MATLAB which is implemented as polyfit and backslashimplemented as polyfit and backslash
{y} = [Z]{a} : general model Eq.(13.8) {y} = [Z]{a} : general model Eq.(13.8) >> x = [ 0 1 2 3 4 5]' ;>> x = [ 0 1 2 3 4 5]' ;>> y=[2.1 7.7 13.6 27.2 40.9 61.1]‘;>> y=[2.1 7.7 13.6 27.2 40.9 61.1]‘;>> z=[ones(size(x)) x x.^2];>> z=[ones(size(x)) x x.^2];>> a=polyfit(x,y,2)>> a=polyfit(x,y,2)>> a =z\y>> a =z\y
13.5 Nonlinear 13.5 Nonlinear regressionregression
Ex : Ex :
The sum of the square The sum of the square
Find aFind a0 0 and aand a1 1 that minimize the function f that minimize the function f Matlab’s fminsearch function can be used for Matlab’s fminsearch function can be used for
this purpose.this purpose.
[x, fval] =fminsearch(fun, x0, options, p1, p2,[x, fval] =fminsearch(fun, x0, options, p1, p2,…)…)
eeay xa )1( 10
20
110 )]1([),( 1 ixa
n
ii eayaaf
Example 13.4 Example 13.4 Nonlinear Regression with Nonlinear Regression with
MATLABMATLAB Recall example 12.4 with the table 12.1: Recall example 12.4 with the table 12.1:
we have we have
This time use nonlinear regression. This time use nonlinear regression. Employ initial guesses of 1 for the Employ initial guesses of 1 for the coefficient.coefficient.
9842.12741.0 vF
SolSol M-file M-file function f = fSSR(a, xm, ym)function f = fSSR(a, xm, ym)yp = a(1)*xm.^a(2);yp = a(1)*xm.^a(2);f =sum((ym-yp).^2);f =sum((ym-yp).^2);
>> x=[10 20 30 40 50 60 70 80 ];>> x=[10 20 30 40 50 60 70 80 ];>> y = [25 70 380 550 610 1220 830 1450];>> y = [25 70 380 550 610 1220 830 1450];>> fminsearch(@fSSR, [1,1],[], x,y)>> fminsearch(@fSSR, [1,1],[], x,y)
ans =ans =
2.53839730236869 1.435853174785852.53839730236869 1.43585317478585 4359.15384.2 vF
Comparison of transformed and untransformed model fits for force versus velocity data from Table 12.1.
Figure 13.4