Averaged and weighted average partial least squares

Analytica Chimica Acta 504 (2004) 279–289

Averaged and weighted average partial least squares

M.H. Zhanga, Q.S. Xub,1, D.L. Massarta,∗a Department of Pharmaceutical and Biomedical Analysis, Pharmaceutical Institute, Vrije Universiteit Brussel,

Laarbeeklaan 103, B-1090 Brussels, Belgiumb Hunan University, Changsha, PR China

Received 24 June 2003; accepted 21 October 2003

Abstract

Two alternative partial least squares (PLS) methods, averaged PLS and weighted average PLS, are proposed and compared with the classicalPLS in terms of root mean square error of prediction (RMSEP) for three real data sets. These methods compute the (weighted) average ofPLS models with different complexity. The prediction abilities of the alternative methods are comparable to that of the classical PLS but theydo not require to determine how many components should be included in the model. They are also more robust in the sense that the quality ofprediction depends less on a good choice of the number of components to be included. In addition, weighted average PLS is also comparedwith the weighted average part of LOCAL, a published method that also applies weighted average PLS, with however an entirely differentweighting scheme.© 2003 Elsevier B.V. All rights reserved.

Keywords: Partial least squares (PLS); Averaged partial least squares (APLS); Weighted average partial least squares (WPLS); Multivariate calibration;LOCAL

1. Introduction

Partial least squares (PLS) regression is widely appliedfor multivariate calibration in many fields[1]. Using PLSrequires however expertise and more user-friendly PLS ver-sions are highly called for. For instance, in classical PLS,models are constructed with different numbers of PLS com-ponents and it must be decided which of them is most ad-equate. If the number of components selected in a model istoo small, the prediction ability may be bad since some rel-evant information is not included. If the number is too large,overfitting may occur, i.e. the prediction ability of the modelis poor because the last components contain much noise. Inorder to select an adequate number of components, the pre-dictive ability is determined, i.e. validation is carried out.There is a considerable literature[2] on how to make thebest choice of the number of components. Methods based oncross validation (CV)[3,4], randomisationt-test[5], MonteCarlo cross validation (MCCV)[6,7], the Durbin Watsoncriterion [8], the adjusted Wold’s R criterion[4], the jack-

∗ Corresponding author. Tel.:+32-2-477-4734; fax:+32-2-477-4735.E-mail address: [email protected] (D.L. Massart).1 On leave.

knife [9] and the bootstrap[10] statistics, are used to decideon the number of components. These methods often lead todifferent conclusions, so that the selection of a PLS modelis far from evident. When a calibration model is updated thenumber of components has to be determined at each update,each time requiring the attention of the expert. In this study,we introduce a methodology in which the selection of thenumber of components requires less attention and compareit with the classical PLS method. The proposed method isbased on averaging the different PLS models. Shenk et al.[11] proposed similar ideas as part of a patented softwarecalled LOCAL. The major originality claims of this softwareare the use of local PLS models and of a weighted average ofPLS models. Only the latter is relevant in the context of thepresent study. From the literature[12] it seems that Shenkand Westerhaus’s method performs well for the very largedatabases applied for the analysis of agronomic and foodcommodities and this prompted us to study the averaging ofPLS models in more detail. It should be emphasised that inthis article we study only the weighted averaging part of LO-CAL, which for convenience we will call here WA-LOCAL,and that therefore this study is not an evaluation of LOCALas such. The averaging process proposed by us is different(and simpler), the main difference being that the weighting

0003-2670/$ – see front matter © 2003 Elsevier B.V. All rights reserved.doi:10.1016/j.aca.2003.10.056

280 M.H. Zhang et al. / Analytica Chimica Acta 504 (2004) 279–289

process in WA-LOCAL depends on the characteristics of thesample being predicted, while in our proposed method this isnot the case. A (weighted) averaged PLS model is developedthat is used as such for the prediction of all new samples.

2. Theory

Suppose a data set{X, y}, whereX is a matrix withnobjects andp variables andy is the corresponding dependentvariable vector with sizen×1. Classical PLS (CPLS) makeslinear regression models connectingX andy by:

y = b0 + Xb + e (1)

whereb is the coefficient matrix with sizep × number ofPLS components,e the residual vector andb0 is the offsetvector:

b0 = y − Xb (2)

For prediction,

yj = b0,j + Xbj (3)

wherej is the complexity of the model.The correspondingroot mean square error of prediction (RMSEP) for a newdata set is calculated as

RMSEPj =√√√√ 1

nv

nv∑i=1

(yi − yi,j)2 (4)

wherenv is the sample number of the new data set.Assume models withc andd (d = c+1) PLS components

were obtained. For these models we can write, respectively

yc=b0,c + x1b1,c + x2b2,c + · · · + xpbp,c = b0,c + Xbc

yd=b0,d + x1b1,d + x2b2,d + · · · + xpbp,d=b0,d + Xbd

Suppose now that it is not clear which of the models isto be preferred because different criteria lead to differentconclusions. A possibility would then be to make the averageof both models:

¯yA(c,d)= 1

2(b1,c + b1,d)x1 + 12(b2,c + b2,d)x2

+ · · · + 12(bp,c + bp,d)xp + 1

2(b0,c + b0,d)

We call this averaged PLS (APLS). More generally themean of theb from themth to thekth component could bemade:

¯yA(m,k)=

[b1,m + · · · + b1,k

k − m + 1

]x1+

[b2,m + · · · + b2,k

k − m + 1

]x2

+ · · · +[bp,m + · · · + bp,k

k − m + 1

]xp

+b0,m + · · · + b0,k

k − m + 1(5)

In APLS all models averaged have the same weight, althoughsome models are clearly better than others. Therefore, the

models can be weighted. The weighted average PLS (WPLS)method proposed here gives weightswj to each coefficientbj.

wj = 1

RMSECVj

(6)

where root mean square error of cross validation (RMSECV)is

RMSECVj =√√√√1

n

n∑i=1

(yi − y\i,j)2 (7)

wherey\i,j is the prediction ofyi when the model is con-structed without samplei and with the complexity ofj.

The WPLS can be written as

¯yW(m,k) =[wmb1,m + · · · + wkb1,k

wm + · · · + wk

]x1

+[wmb2,m + · · · + wkb2,k

wm + · · · + wk

]x2

+ · · · +[wmbp,m + · · · + wkbp,k

wm + · · · + wk

]xp

+wmb0,m + · · · + wkb0,k

wm + · · · + wk

(8)

Different weighting procedures can be proposed. For in-stance, instead of weighting as inEq. (6), variance weight-ing can be applied, i.e.

wj = 1

MSECVj

(9)

where mean square error of cross validation (MSECV) is

MSECVj = 1

n

n∑i=1

(yi − y\i,j)2 (10)

Shenk et al.[11] proposed to use weights in LOCAL[13],which are calculated as

weighti,j = 1

rms residuali,j × rms betaj(11)

where rmsresiduali,j is the root mean square of spectralresiduals of the unknown samplei when 1–j PLS compo-nents are used, rmsbetaj is the root mean square of thejth regression coefficients of calibration model andj is thecomplexity of the model.

The weights inEq. (11) are, according to the authors,“used to de-emphasise solutions that do not explain enoughof the X variation (usually considered to be a sign of un-derfitting) and solutions that have large regression coeffi-cients (usually considered to be a sign of overfitting)”[13].The spectral residuals are the part ofX that are not ex-plained by the model and rmsresidual is the unexplainedX variation. The rmsresidual decreases as the number ofthe PLS components increases. It should be noted that theweighting used here depends on the spectrum of the new

M.H. Zhang et al. / Analytica Chimica Acta 504 (2004) 279–289 281

sample to be predicted, and is therefore different for eachsample, which is not the case in our proposal. The regres-sion coefficients are indications of the influence ofX on y

and rmsbeta is an indication of the variance of the coeffi-cients. As the number of PLS components increases, morenoise may be included, resulting in unstable estimated co-efficients and therefore in a larger variation, i.e. a largerrms beta.

Replacingwj with weighti,j in Eq. (8), WA-LOCAL givesthe prediction ofy for the unknown samplei. The perfor-mance of this weighting method will be investigated andcompared with our proposed method.

Fig. 1. The RMSECV curve (–) of the constructed PLS model of green tea data set and the prediction ability (RMSEP) (�) of the model for the test set.

Fig. 2. The RMSEP values of the green tea test set using classical PLS (–), averaged PLS (+) and weighted average PLS (�). The five points selected,in the order from left to the right, are using the averaged or weighted coefficients of the PLS components (4, 5), (4, 5, 6), (7, 8), (8–13) and (12, 13),respectively.

3. Experimental

The following three data sets were studied:

• Green tea data set[14] includes 123 NIR spectra of greentea samples measured in reflectance mode every 2 nm be-tween 1100 and 2500 nm. The total antioxidant capacityis the measured response. The data set is divided into twosubsets using the DUPLEX[15] method. Three objectsof the test set identified as outliers in the previous workare removed, resulting in 100 objects in the calibration setand 20 objects in the test set.


• Wheat data set[16] consists of 100 NIR spectra of wheatsamples measured in reflectance mode every 2 nm be-tween 1100 and 2500 nm. The response is the moisturecontent. The spectra are pre-treated by offset correction.Two clusters are observed in the PC1–PC3 plot. The dataset is randomly separated into two subsets with 70 objectsin the calibration set and 30 objects in the test set.

• Hydrogen data set[17] consists of 239 NIR spectraof gas oil samples measured every 2 cm−1 between4900–9000 cm−1. The response is the percent of hydro-gen determined by NMR. Two clusters are observed in

Fig. 3. The RMSEP values of the green tea test set using classical PLS (–), averaged PLS (+) and weighted average PLS (�). The RMSEP values areobtained by using the coefficients averaged or weighted from the first PLS component to the indicated PLS component.

Fig. 4. The RMSEP values of the green tea test set using classical PLS (–), averaged PLS (+) and weighted average PLS (�). The RMSEP values areobtained by using the coefficients averaged or weighted from the fourth PLS component to the indicated PLS component.

PC1–PC2 plot. The data set is randomly split into twosubsets: 169 in the calibration set and 70 in the testset.

4. Results

4.1. Green tea data set

Classical PLS models up to 20 components are obtainedby using leave-one-out cross validation of the calibration set.


Fig. 1 shows that the first minimal RMSECV is obtainedwith five PLS components. A second minimum is obtainedwith 8 components and a third is with 13 components.

The RMSEP values of the test set with different PLScomponents using CPLS models are shown inFig. 1.

Let us first consider some simple applications in whichonly two or a few models are averaged. APLS models wereconsidered around the three RMSECV minimums using sev-eral consecutive components. APLS models were made byaveraging (1) the model with 4 components and the modelwith 5 components (2) the models with 4, 5 and 6 compo-nents (3) the models with 7 and 8 components (4) the mod-els with 8,9,10,11,12 and 13 components and (5) the modelswith 12 and 13 components. The individual CPLS modelsselected have similar RMSECV values to each other and itis not clear which one should be preferred. The results ofthe averaged models are compared with those of the corre-sponding CPLS models.

Most RMSEP values of the APLS models fall as expectedwithin the range of RMSEP values obtained by CPLS mod-els included in the average. In some cases the RMSEP val-ues of the averaged PLS models are even slightly betterthan any RMSEP values obtained by the CPLS models in-volved in the average (Fig. 2). This is the case for the modelthat averages the models with four and five PLS compo-nents, which is somewhat better than the individual modelswith four and five components and for the APLS model re-placing the individual models with four, five and six PLScomponents.

The APLS models obtained by averaging models withcomponents from the first to a selected number were there-fore also compared with the corresponding CPLS models.These APLS models always show intermediate quality. They

Fig. 5. The RMSECV curve of the wheat data set.

are worse than the best CPLS model included in the average,but better than the worst CPLS model (Fig. 3). However,when the randomisation test[5] is applied no significant dif-ference is observed between the APLS model and the bestCPLS model included in the average.

APLS models were also constructed by using the modelswith PLS components from the fourth to a selected number,i.e. by eliminating from the process the models with one tothree components that are clearly inadequate.

Fig. 4shows that the RMSEP values of the APLS modelsare always within the range of the RMSEP values of theCPLS models involved in the average. Since the PLS modelswith only a few components, which have higher RMSEPvalues, are not taken into account, the APLS models giveindeed lower RMSEP values than those obtained when theyare included in the APLS model.

WPLS models are compared with the CPLS models andthe APLS models. When using the PLS models situatedaround the three minimal RMSECV values, there is no dif-ference between the WPLS models and the correspondingAPLS models (Fig. 2). When the weighting and the averag-ing starts from the PLS model with only one component, theWPLS model is always worse than the best CPLS model,but the WPLS model yields better results than the APLSmodel (Fig. 3). Similar trends are obtained when weightingstarts from the model with four PLS components (Fig. 4) butthe latter leads to lower RMSEP values. Because the modelswith only one to three PLS components are excluded, theresults of the WPLS models become more similar to thoseof the corresponding APLS models. The WPLS modelcombining classical models with 4–20 components is nownot significantly worse than the best classical model with17 components. It should be noted that the selection of the


number of PLS components has now become simpler. Thereis no statistically nor practically significant difference be-tween the models with 14–20 components. The RMSEPcurve decreases very slowly and regularly. In contrast theRMSEP curve for classical PLS is much more irregular anddepends much more on the way the data were split into cal-ibration and validation set.

4.2. Wheat data set

The first minimal RMSECV value of the wheat data setis obtained for seven PLS components (Fig. 5). Checked byrandomisation test, there is no significant difference betweenthe models built with three components and seven compo-

Fig. 6. The RMSEP values of the wheat test set using classical PLS (–), averaged PLS (+) and weighted average PLS (�). The average or weightingis (a) from the first PLS component to the indicated PLS component, (b) from the third PLS component to the indicated PLS component.

nents and it is therefore not surprising that the first mini-mal RMSEP of the test set is with three PLS components(Fig. 6). Practitioners would be divided over the questionwhether the models with three or seven components shouldbe preferred.

When the average is taken from the model with one PLScomponent to models with a larger selected number of com-ponents, the RMSEP values of the APLS models for thewheat data set can be better than the best CPLS models(Fig. 6a). This is also true for the WPLS results. The WPLSis better than the APLS when only a few models with lowcomplexity are averaged. When models with higher com-plexity are included into the average, the difference betweenthe WPLS and the APLS becomes less and less.


Fig. 7. The RMSECV curve of the hydrogen data set.

When the average starts from the model with threePLS components, the difference between the averaged andweighted models becomes smaller (Fig. 6b) and most ofthe RMSEP values of the test set are lower than those forthe corresponding best CPLS models. It is interesting toobserve that as for the green tea data set the fluctuationsin the RMSEP curve of the CPLS model are larger thanfor the APLS and the WPLS RMSEP curves. For practi-cal purpose, the RMSEP from 4 to 20 components do notchange (RMSEP range 0.22–0.23). This again suggests thatthe APLS or WPLS models are less sensitive to the numberof PLS components, and therefore more robust towards thenumber selected.

4.3. Hydrogen data set

The RMSECV curve of the hydrogen data set rises moresteeply when the number of components increases thanthe other data sets, indicating a larger tendency to overfit(Fig. 7). The model with five components gives the minimalRMSECV value but the RMSECV of the model with fourcomponents is only a little higher and might be chosen bypractitioners that prefer less complex models.

When the averaging or weighting process includes themodels starting with one component, both APLS and WPLSmodels with at least eight components are better than CPLSmodels with the same number of terms (Fig. 8a). The WPLSmodel is a little better than the APLS model when thenumber of components is small but the difference becomessmaller as it increases. Unlike the CPLS model for whichthe RMSEP curve is quite unstable and rises much higherat the end, both APLS and WPLS models are flat from theeighth component on and are not affected much with the in-clusion of models with a larger number of components. The

overfitting problem due to selecting too many componentsis minimised by the averaging process. From the inclusionof the model with eight components on, there is no signifi-cant difference between APLS and WPLS and there is alsono significant difference with the best CPLS model, whichuses five components.

When averaging or weighting starts the model with fourcomponents, both APLS and WPLS models have lowerRMSEP values than those for which averaging or weight-ing starts from the model with one component (Fig. 8b),due to the exclusion of the first three components, which

Table 1The weights of the three data sets

PLS components Data set

Green tea Wheat Hydrogen

1 0.29 0.85 2.912 0.34 1.13 4.293 0.35 3.22 4.554 0.38 3.32 4.935 0.44 4.13 4.956 0.42 4.38 4.867 0.48 4.55 4.768 0.51 4.45 4.739 0.51 4.37 4.65

10 0.49 3.81 4.7911 0.49 3.72 4.7812 0.50 3.47 4.6913 0.51 3.09 4.5114 0.48 2.80 4.3615 0.48 2.76 4.1616 0.47 2.66 3.9217 0.49 2.80 3.8618 0.47 2.82 3.5519 0.46 2.83 3.5520 0.44 2.67 3.55


Fig. 8. The RMSEP values of the hydrogen test set using classical PLS (–), averaged PLS (+) and weighted average PLS (�). The average or weightingis (a) from the first PLS component to the indicated PLS component and (b) from the fourth PLS component to the indicated PLS component.

have higher RMSEP. Both are also better than all CPLSmodels.

4.4. Comparison of WPLS with WA-LOCAL

LOCAL is mainly applied for large databases[11,18,19,20].It is reported that LOCAL generally performs better thanglobal PLS[11,20]. In order to achieve better prediction,LOCAL includes several algorithms, such as for instancean algorithm for the selection of the type and the num-ber of samples used in calibration. Our study focuses ononly one aspect namely the weighting method. The perfor-mance of our proposed method is compared with that ofWA-LOCAL. The results obtained for the three data sets

indicate that when weighting starts from the first PLS com-ponent, WPLS is often better than WA-LOCAL (Fig. 9a-1,b-1, c-1). This is due to the fact that in WA-LOCAL theweights of the first few PLS components are often muchlarger than those of the following ones (Fig. 10). These firstPLS components usually have relatively poor predictioncharacteristics. Therefore, the LOCAL method proposedby Shenk recommends to exclude some of the first PLScomponents as is also the case in our method and this wasdone in the comparison performed here. The results showthat WPLS and WA-LOCAL have comparable predictionabilities (Fig. 9a-2, b-2, c-2). Compared with CPLS, bothWPLS and WA-LOCAL are less sensitive to the number ofPLS components included.


Fig. 9. The RMSEP values of the three data sets using WA–LOCAL (�), classical PLS (–) and weighted average PLS (�). (a) Green tea data, theweighting is (a-1) from the first PLS component to the indicated PLS component, (a-2) from the fourth PLS component to the indicated PLS component;(b) wheat data, the weighting is (b-1) from the first PLS component to the indicated PLS component, (b-2) from the third PLS component to the indicatedPLS component; (c) hydrogen data, the weighting is (c-1) from the first PLS component to the indicated PLS component, (c-2) from the fourth PLScomponent to the indicated PLS component.


Fig. 10. The mean weights of all test samples of the (a) green tea data,(b) wheat data and (c) hydrogen data in WA-LOCAL.

5. Discussion and conclusions

The prediction ability of APLS and WPLS are found tobe comparable to the classical PLS model. Indeed whenthe average includes the models from 1–20 components,the RMSEP values of the APLS and the WPLS models arealways comparable to that of the best CPLS model.

The WPLS model yields better prediction than the APLSmodel when only a few models of low complexity are av-eraged. This difference is diminished when the complexityof the averaged models increases. This is because in APLSthe b of each model has the same contribution to the fi-nal averaged model but in WPLS theb of models with asmaller RMSECV have a larger effect on the final model.Thus, the RMSEP of WPLS is expected to be better thanthat of the corresponding APLS model. When the complex-ity of the models that are averaged increases, it is found thatthe weights of the individual models (Table 1) included haveless effect on the final result, so that the difference betweenAPLS and WPLS becomes smaller.

In industrial PLS application with NIR data, the number ofcomponents seldom exceeds 20 and in most cases studied byus, the RMSEP curves of the APLS/WPLS models becomestable before the models with the first 15 components areincluded. In order to achieve better prediction results, thefirst PLS model included could be the one that yields thefirst minimum in the RMSECV curve or the PLS model withone component less number.

Our study indicates that APLS and WPLS are at least ac-ceptable alternatives to classical PLS from the point of viewof prediction ability. They may have some additional advan-tages. To start with, the difficulty of selecting a classical PLSmodel with a specific complexity is avoided. Moreover, bothAPLS and WPLS models are less sensitive to the numberof PLS components included than the classical PLS, sug-gesting that they are robust methods, which is important inindustrial applications. Since it is not necessary anymore todecide on a specific complexity, one difficulty of updatingmultivariate calibration models is also avoided: it is less nec-essary to study the complexity of the model at each update.For updating purposes, APLS may be preferred to WPLS.Both have the same prediction ability but APLS is computa-tionally simpler than WPLS, since the cross validation stepneeded to compute the weights is not necessary.

Our study also indicates that WPLS is comparable in per-formance to WA-LOCAL. It should be emphasised that thiscomparison included only one of the original steps of LO-CAL and our conclusions are therefore limited to that step. Afair comparison of LOCAL with other complete PLS strate-gies should include also the selection of samples step andthe local properties of that method.

References

[1] S. Wold, M. Sjöström, L. Eriksson, Chemom. Intell. Lab. Syst. 58(2001) 109–130.


[2] H. Martens, T. Naes, Multivariate Calibration, Wiley, Chichester,1989.

[3] D.W. Osten, J. Chemom. 2 (1988) 39–48.[4] S. Wold, Technometrics 20 (1978) 397–405.[5] H. van der Voet, Chemom. Intell. Lab. Syst. 25 (1994) 313–323.[6] R.R. Picard, R.D. Cook, J. Am. Stat. Assoc. 79 (1984) 575–583.[7] Q.S. Xu, Y.Z. Liang, Chemom. Intell. Lab. Syst. 56 (2001) 1–11.[8] N. Draper, H. Smith, Applied Regression Analysis, second edition,

Wiley, New York, 1981, p. 162.[9] C. Duchesne, J.F. MacGregor, J. Process Control 11 (2001) 553–564.

[10] B. Efron, J. Am. Stat. Assoc. 81 (1986) 461–470.[11] J.S. Shenk, M.O. Westerhaus, P. Berzaghi, J. Near Infrared Spectrosc.

5 (1997) 223–232.[12] P. Dardenne, G. Sinnaeve, V. Baeten, J. Near Infrared Spectrosc. 8

(2000) 229–237.

[13] J.S. Shenk, M.O. Westerhaus, US Patent No. 5,798,526 (1998).[14] M.H. Zhang, J. Luypaert, J.A. Fernandez Pierna, Q.S. Xu, D.L.

Massart, Talanta, in press.[15] R.D. Snee, Technometrics 19 (1977) 415–428.[16] J.H. Kalivas, Chemom. Intell. Lab. Syst. 37 (1997) 255–259.[17] F. Wahl, personal communication.[18] P. Berzaghi, P.C. Flinn, P. Dardenne, M. Lagerholm, J.S. Shenk,

M.O. Westerhaus, I.A. Cowe, in: Proceedings of the 10th Interna-tional Conference on Near infrared Spectroscopy, 2002, pp. 107–111.

[19] P. Dardenne, I.A. Cowe, P. Berzaghi, P.C. Flinn, M. Lagerholm, J.S.Shenk, M.O. Westerhaus, in: Proceedings of the 10th InternationalConference on Near infrared Spectroscopy, 2002, pp. 23–28.

[20] P. Berzaghi, J.S. Shenk, M.O. Westerhaus, J. Near Infrared Spectrosc.8 (2000) 1–9.

Documents

Averaged and weighted average partial least squares