This article was downloaded by: [University of California Santa Cruz]On: 26 November 2014, At: 13:10Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Applied Economics LettersPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/rael20

An iterative method for flattering the ridge in theRidge regressionMiyoung Lee aa Department of MIS, School of Business , Konkuk University , Seoul, Korea E-mail:Published online: 18 Jun 2007.

To cite this article: Miyoung Lee (2007) An iterative method for flattering the ridge in the Ridge regression, AppliedEconomics Letters, 14:7, 529-531

To link to this article: http://dx.doi.org/10.1080/13504850500425840

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Applied Economics Letters, 2007, 14, 529–531

An iterative method for flattering

the ridge in the Ridge regression

Miyoung Lee

Department of MIS, School of Business, Konkuk University, Seoul, Korea

E-mail: yura@konkuk.ac.kr

In many cases of multiple regression in an undefined system, some

independent variables may not be orthogonal to each other. In such

cases, the systems either are not solvable or induce incorrect results

which could vary with the data used. Such problems are often

overcome by using the Ridge regression method. This article

proposes an alternative way of getting an exact least square

estimator by using an iterative method. We prove the solvability

of the proposed algorithm and demonstrate that our method

outperforms traditional approaches.

I. Introduction

There has been a surge in studies on the Ridge

regression since the seminal work by Hoerl and

Kennard (1968). Hoerl and Kennard (1970) point out

that the ordinary least squares estimator based on

data exhibiting near extreme multicollinearity would

be subject to a number of errors and this problem can

yield one or a few small eigenvalues in its correlation

matrix. In fact, the difficulty of obtaining analytical

results for this kind of estimator is well-defined in

extant literature. Riley (1955) explores such a linear

system with symmetric but almost singular matrix.

James and Stein (1961) adopt a method of permitting

a little of error to overcome these kinds of difficulties.

Heorl (1964) also explores the potential drawback of

the Ridge method. In a standard regression model for

multiple linear form, Y¼X�þ ", where Y is k� 1, X

is k� n and of rank(X )¼ n, � is n� 1 and the random

vector ", normal with E(")¼ 0, the Ridge regression

estimator evaluates �*¼ (X 0Xþ kI )�1X 0Y using a

slightly biased matrix X 0Xþ kI for a positive value k

instead of using �¼ (X 0X)�1X 0Y in an unbiased

original linear system. Also this Ridge regression

method is applied for other studies for regression

analysis as in Seo (1999) for dynamic plots for a

higher order regression model. However this Ridge

regression estimator is still subject to some error dueto the biased model of the matrix.

X 0Xþ k

By eliminating the aforementioned errors prevailingin the Ridge regression estimator this article suggestsan alternative way of achieving exact least squaredsolutions for the system Y¼X�þ " even for theill-conditioned correlation matrix system X 0X.

The remainder of the article proceeds as follows. InSection II, we derive an iterative method of solvingfor an exact least squared estimator. Section IIIexplores a numerical example to highlight thedifference between our method and the simpleRidge regression model. Concluding remarks aremade in Section IV.

II. An Iterative Method

Consider a standard regression model for multiplelinear form. The ordinary least square estimator �,which minimizes the sum of squares of residuals isequivalent to maximum likelihood estimator;

� ¼ ðX 0XÞ�1X 0Y ð1Þ

Applied Economics Letters ISSN 1350–4851 print/ISSN 1466–4291 online � 2007 Taylor & Francis 529http://www.tandf.co.uk/journalsDOI: 10.1080/13504850500425840

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 13:

10 2

6 N

ovem

ber

2014

We suggest a computational algorithm to obtain anunbiased solution for Y¼X�þ ", which can bedenoted as in Equation 1 whenX 0X is almost singular.

At the first step of the iteration, we use a Ridgeregression estimator, B0 as an initial value,

B0 ¼ ðX 0Xþ kIÞ�1X 0Y ð2Þ

Then, at the mth step of the iteration, derive asolution Bm by the recursive relation;

Bm ¼ ðX 0Xþ kIÞ�1ðkBm�1 þ X 0YÞ ð3Þ

As we can see in Golub and Van Loan (1989,Theorem 10.1.1, p. 508), with the following theorem,the spectral radius of any n� n matrix G can bedefined as �ðGÞ ¼ maxfj�j: � 2 �ðGÞg where �(G)stands for the entire set of eigenvalues of matrix G.

Theorem 1: Suppose that b is n� 1 and A¼M�N isa n� n nonsingular matrix. If M is nonsingular and thespectral radius of M�1N satisfies the inequality�(M�1N)<1, then the iterate Bm defined byMBm ¼ NBm�1 þ C converges to B¼A�1C for anystarting B0, which is n� 1.

Proof: See Golub and Van Loan (1989,Theorem 10.1.1, p. 508). œ

Theorem 2: For k� 1 matrix, Y, k� n matrix X withrank(X)¼ n, the iterative method specified as inEquation 3 with a initial guess given in Equation 2converges to a problem solving the systems of equations(X 0X)B¼X 0Y and enables us to obtain the unique leastsquares estimator �¼ (X 0X)�1X 0Y to Y¼X�þ ", " isn� 1, a random variate.

Proof: For any k>0 in Equation 3, the mth step ofthe iteration is denoted by

ðX 0Xþ kIÞBm ¼ kBm�1 þ X 0Y ð4Þ

Then denote M¼X 0Xþ kI and N¼ kI. And wearrange the eigenvalues of matrix X0X as follows;

�max ¼ �n � �n�1 � � � � � �2 � �1 ¼ �min > 0 ð5Þ

Since X has a full column rank, the positiveeigenvalues of correlation matrix X 0X can be assumedusing the definition of positive definite as in Strang(1976, definition 6B, p. 331). Then G. Strang (1988,p. 258) shows that eigenvalues of M�1N are

k

�i þ kj i ¼ 1, 2, . . . , n

� �ð6Þ

From Equations 5 and 6, the spectral radius of M�1Nis (k/(�minþ k))<1 for any k>0.

Therefore, by Theorem 1, the iterates defined byEquation 5 converge to B¼ (X 0X)�1X 0Y for anyn� 1 matrix Y. œ

III. Numerical Illustration

Our proposed iteration method is coded in matlab5.3.

As an example, the random variables x1, x2 and x3are used with 50 observations generated according to

the model,

y ¼ x1 � 2x2 þ x3 þNð0, 0:12Þ

The corresponding correlation matrix system X 0X has

eigenvalues �3¼ 12.0, �2¼ 10 and �3>0 (but close

too). Fig. 1 and Fig. 2 illustrates the regression

estimator of the Ridge regression and the

proposed iteration method, respectively. Figure 1

evidently demonstrates that the Ridge regression

estimation suffers from bias, which occurs due to

the bias in the correlation matrix of the regressors.

The graph with the Ridge regression estimator

Data givenThe Ridge regression resultExact least squares estimator

Data index

Val

ue o

f the

func

tions

on

each

dat

a in

dex

0 10

10

5

0

−5

−1020 30 40 50

Fig. 1. The Ridge regression estimator

The graph with the estimator from the iteration result

Data givenIteration resultExact least squares estimator

Data index

Val

ue o

f the

func

tions

on

each

dat

a in

dex

0 10

10

5

0

−5

−1020 30 40 50

Fig. 2. The estimator from the iteration method

530 M. Lee

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 13:

10 2

6 N

ovem

ber

2014

In Fig. 2, however, the bias is shown to be

eliminated completely when the iteration scheme is

added.

IV. Concluding Remark

This article demonstrates that adding an

iteration scheme to the Ridge regression method

successfully eliminates bias inherent in the stan-

dard Ridge regression by adopting an exact least

square solution to the correlation matrix instead

of the biased Ridge regression estimator. In

principle, the iterative method proposed in this

article eliminates the ridge on the matrix in the

Ridge regression through iteration. Put differently,

the matrix that we solve for in each iterative step

is not singular unlike the matrix used in the extant

Ridge method.

References

Golub, G. H. and Van Loan, C. F. (1989) MatrixComputations, 2nd edn., The Johns HopkinsUniversity Press, Baltimore and London.

Hoerl, A. E. (1964) Ridge analysis, Chemical engineeringprogress symposium series, 60, 67–77.

Hoerl, A. E. and Kennard, R. W. (1968) On regressionanalysis and biased estimation, Technometrics, 10,422–3.

Hoerl, A. E. and Kennard, R. W. (1970) Ridge Regression:Based Estimation for Nonorthogonal Problems.Technometrics 12, 55–67.

James, W. and Stein, C. M. (1961) Estimation withquadratic loss, Proc. 4th Berkeley Symposium, 1,361–79.

Riley, J. D. (1955) Solving systems of linear equationswith a positive definite. Sysmetric, but possiblyill-conditioned matrix, Mathematics of Computation,9, 96–101.

Strang, G. (1988) Linear Algebra and its Application,3rd edn., Harcourt Brace Jovanovich, Inc.

Seo, H. S. (1999) A dynamic plot for the specification ofcurvature in linear regression, Computational Statisticsand Data Analysis, 30, 221–8.

An iterative method for flattering the ridge in the Ridge regression 531

Dow

nloa

ded

by [

Uni

vers

ity o

f C

alif

orni

a Sa

nta

Cru

z] a

t 13:

10 2

6 N

ovem

ber

2014