Y.Zhang T.T. Lie C.B.Soh

Indexing terms: Parameter estimation, Least-squares method, Consistent estimation, System identification, Signal processing

Abstract: The paper presents a new bias- compensating least-squares (BCLS) method which can eliminate the noise induced estimation bias even without modelling the noise. By employing a composited signal in the parameter estimation, two estimates for the same part of the system parameter are obtained so that the noise induced bias can be extracted from the difference between these two estimates. With the bias removed, the consistent parameter estimate is obtained. It is shown in the paper that the BCLS method is a member of the class of weighted-instrumental- variable methods. Therefore, the BCLS method also provides a particular way to construct instrumental variables for the identification of a general system.

In the area of system identification and IIR filter design by using least squares (LS) methods, a common concern and important problem is how to eliminate the estimation bias induced by the noise acting on the system. Over the decades, much effort has been devoted to this problem and many kinds of modified least-squares methods have been developed [ 1-31. In the early proposed methods, such as the generalised least-squares (GLS) method, the extended least-squares (ELS) method and prediction error (PE) method, the noise model needs to be estimated at the same time as the system parameters are being estimated. Thus the results of these methods are inevitably dependent upon the accuracy of the noise model. In addition, some restrictive conditions on the noise model must be satisfied in these methods to obtain consistent parameter estimates. For instance, in some cases where the noise model is not selected appropriately, the GLS method may give biased estimates due to the possibility of local minima. In the ELS method and the PE method, the noise must be assumed to satisfy the strictly positive real (SPR) condition to guarantee the convergence of these methods. In general, however, it is 0 IEE, 1997 IEE Proceedings online no. 19970948 Paper first received 25th March and in revised form 20th September 1996 The authors are with the School of Electrical & Electromc Engineering, Nanyang Technological University, Nanyang Avenue, Singapore 639798

40

very difficult to model the noise accurately and it is also hard to know a priori whether the noise model satisfies these conditions.

To overcome the bias problem without modelling the noise, the instrumental variables (IV) method was developed methodologically in [4]. Under a unified framework, the IV method provides a promising way to obtain consistent estimates which have certain optimal properties. However, it seems at the initial stage of the development of the IV method that there has not been any general and efficient technique to choose suitable instrument variables for the identification of linear systems. Recently a new approach called the bias eliminated least-squares method (BELS) was proposed [5] and further developed to be an efficient method to treat bias problem in system identification [6, 71. Without modelling the noise, the BELS method can achieve unbiased parameter estimation for linear systems described by a general model. In recent literature [8], it has been shown that this method is just a member of the class of weighted-instrumental-variables (WIV) methods. Thus a concrete way is available to construct general optimal instrumental variables for identification of linear systems.

This paper presents another much simpler approach to constructing a class of general instrumental variables for the same identification purpose. First, a new bias- compensating least-squares method is proposed to treat the identification problem without modelling the noise. Then this method is shown to be a WIV method. In the proposed method, a composited signal made of the sum of the system output and delayed input is used so that two estimates of the same part of the system parameter are obtained. The noise induced bias is then calculated from the difference between these two esti- mates based on an asymptotical analysis. With the bias removed from the ordinary LS estimate, an unbiased parameter estimate is obtained. In comparison with the BELS method, the proposed method can also achieve consistent parameter estimation without modelling the noise, but it does not need to filter the system input sig- nal. The composited signal employed here is just the simple sum of the output and the old recorded system input. Thus the identification procedure of the pro- posed method is simpler than that of the BELS method. It is shown that the new developed method is also a member of the class of WIV methods by choos- ing an appropriate weighting matrix. Therefore, a much simpler method for constructing IV is obtained.

IEE Proc -Control Theory Appl , Vol 144, No 1, January 1997

Numerical studies are given to illustrate the perform- ance of the proposed method.

2 Problem formulation

Consider a linear system described by the following model:

(1) y(t) + a& - 1) + ' . . + a,y(t - n)

= bou(t) + b1u(t - 1) + ' * ' + b,u(t - m) + .(t) where u(t) and y( t ) are the system input and output, respectively. v(t) is an unknown coloured noise disturb- ing the system and it is uncorrelated with the system input. Denote:

T a = [a1,a2,. . . ,a,]

- aT = [aT; bT]

= [Y@ - 11,. . . ,u(t - 4IT

b = [ b l , b 2 , . . . , bnIT

(2)

4 ( t ) = [U@),. . . , u(t - m)]T -U

-

Then the system eqn. 1 can be expressed by the follow- ing regressive equation:

The problem under study is to obtain a consistent parameter estimate for the parameter _a from the avail- able observed data {y( t ) , u( t )}y . Since our attention is concentrated on unbiased parameter estimation, the system order n and m are assumed to be known.

Following the notations in [l, 21, the LS estimate of - a and its asymptotical property are given by

where

and

= [Tyv(l),...,Tyw(n);~u,(O),...,~,v(m)lT (8) Since v(t) is uncorrelated with u(t), ruv(i) = 0 for i = 0, 1, ..., m. Thus eqn. 8 can be rewritten as

R4v - = [ ~ ~ ~ ( 1 ) , ~ ~ ~ ( 2 ) , . . . , T ~ ~ ( ~ ) ; O , . . . ,OIT = [R:v;O]T (9)

It can be seen from eqn. 6 that the LS estimate QLdc;(N) is biased due to the appearance of Ryv. Thus if the con- sistent estimate of RYr is obtained, the consistent esti- mate of a can be obtained by

! L W ( W = G L S ( N ) - R,,(wl[R;v; - 0lT (10)

This is just the bias-correction principle often used in

IEE Proc -Contsol Theosy Appl, Vol 144, No 1, January 1997

many modified LS methods [5, 6, 8, 91. The proposed method in this paper is also developed by using this principle.

3 Developmlent of proposed method

Define a new signal regressive vector and a new param- eter vector as follows:

where hux(t) = [u(t - m - l), u(t - m ~ 2), ..., u(t - m - n)lT and M is an invertible weighted matrix of Rnxn It will be shown later that the matrix M will not affect the statistical accuracy of the obtained estimates. Thus we can take it to be an identity matrix. For the com- pleteness of analysis, however, M is still used in the development of the new method.

The system eqn. 1 can be rewritten as

Y ( t ) = $(t)Te + v ( t ) (13) It is to be noted that a composited signal &(t) + M&,,(t) is used in the new signal-regressive vector g. As a result, the parameter a of the original system appears twice in the new parameter vector 8 to be esti- mated. This property will be shown to be useful to determine the noise-induced bias in the LS estimate.

Similarly to elqn. 5 and 6, the LS estimate of 8 and its asymptotical property are obtained by

where

Since v(t) is uncorrelated with input signal u(t), we have

R$?J = [ryv(l),...,Tyv(n);O,...,O]T - - [ItT . ()IT E J p + m + l ) x 1 (16)

-

Y V '

where Ryv E Rnxl.

form: Thus eqn. 151 can be rearranged into the following

= [ b" ] +R&[Rf] (17)

where &(A') and d3(N) are vectors in Rnxl. They are estimates for the parameter a.

It is obvious that two estimates of the parameter a are obtained in d ( N ) and their asymptotic difference iFm (dl(N) - e3(A')) depends only upon the noise induced bias item Ryv. Thus the difference between the

MTa

41

two estimates of a can be used to extract the noise induced bias from d(N>.

To extract the difference between the two estimates of a, we define the following matrix:

(18)

zTe = 0 (19)

(20)

ZT = [ M T ; Q; -11 E $(,nx(zn+m+l)

Then

Multiplying ZT on the both sides of eqn. 17 gives

zT N-iCO- lim 8 ( ~ ) = Z~R&QR,, -

where Q = [I; O]* E R(2ncm+1)xn. This equation gives an explicit expression of the relation between the noise- induced bias Ryv and the difference of the two estimates of a.

Thus, Ryv can be determined by

R,, = (Z'R;iQ)-'Z' - N+CO- lim @ N ) (21)

R,, = (ZTR,,(N)-'Q)-'ZTB(N) - (22)

and the consistent estimate of Ryy is given by

Using the bias-correction principle given in eqn. 10, the unbiased estimate of the parameter vector &can be obtained as follows:

BBBCLS(N)

Define

I (24) H = 1 0 '1 E R(2"+"+1)x(n+m+I)

LMT Q It follows from eqn. 2 and 12 that

H a = @ (25) Since rank(H) = IZ + m + 1, the consistent estimate of parameter vector a is given by

T H - 1 ~ T 4 &?CLS(N) =(H 1 - -BCLS(N)

= ( H ~ H ) : ~ H ~ ( I - R++(N)-~Q -

x ( Z ~ R ~ ~ ( N ) - ' Q ) - ~ Z ~ ) ~ ( N ) - (26) Eqn. 26 gives a batch algorithm for obtaining the con- sistent parameter estimates for system eqn. 1. As was discussed in [l, 21, d ( N ) and &,,Q(N> can be obtained recursively. Therefore, the above batch algorithm can be easily transformed into the following on-line recur- sive algorithm:

where P, = %$(t). Performing the same procedures as that in [5 ] , it is

easily verified that both the obtained estimates - B,,,(N) and CiBCLS(t) are consistent estimates of the

42

parameter a. So far, we have established the new bias- compensating least-squares (BCLS) method to achieve consistent parameter estimation without modelling the noise.

4 BCLS method is a WIV method

In this Section, we will show that the BCLS method presented above is just a member of the class of weighted-instrumental-variables (WIV) methods. The analysis procedures in [SI will be employed here to reach that conclusion. From eqns. 3 and 11, it is easily verified that $ has the following properties:

Property PI : The last m + n + 1 elements of $(t) are uncorrelated with the coloured noise v( t ) :

Property P2:

where

E R(2nfmfl)X(n+m+l),

rank(H) = m + n + l and Onx(n+m+ll is a matrix of Rnx(n+m+l) in which all ele- ments are zeros. Define

If u(t) is a persistently exciting signal of order 2n + m + 1, z( t ) obviously satisfies

rank ( E [ ~ ( t ) $ ( t ) ~ ] ) = m + n + 1, E[z(t)zr(t)J = 0 (31) which just meet with the requirements of instrument variables. Thus, the WIV estimate of the parameter a in eqn. 4 can be obtained by

-

hIV = (GTwrVG)-lGTWIVp - (32) where

N - 1 - N p = - C ( z ( t ) y ( t ) ) E B(n+m+l)xl

t=l

and WIv is a weighted matrix of R(n+m+l)x(n+m+l). To establish the equivalence between a,, and QBCLS(w, it should be better to express aBCLS(N) i,n

Grms of 4. Using the definitions of z( t ) , *(t) and e, 8(N) given by eqn. 14 can be rewritten into the fol-

lowing form:

B(N) = R$,(N)-1R,&V) - - -

(33)

IEE Proc.-Control Theory Appl., Vol. 144, No. 1, January 1997

where

t=l Thus it readily follows from eqn. 26 that:

&CL#) = ( H ~ H ) - ~ H * ( I - R,,(N)-~Q -

x (ZTR,,(N)-lQ)-lZT)R,,(N)-l - - [ i] = ( H ~ H ) - ~ H ~ ( I - R,+(N)-~Q -

x (zTRi+ - ( N ) - ~ Q ) - ~ z ~ ) R ~ + ( N ) - ~ Q ~ .-

+ (HTH)-lHT(I - R,, (N)- lQ -

Eqn. 34 can be rewritten into the following form:

a B C L S ( N ) = (K - C D - l F ) ; - (39) In the following theorem, it is verified that QBCLS(N) is just a WIV estimate. The similar procedures as those in [8] are performed to obtain the conclusion of the theorem.

Theorem I : Let u(t) be a persistently exciting signal of order 2n + m + 1. Then

Li-IvIwIv=I = L i - B C L S ( N (40) Proof: Let

s = [o; I ] R + , ( N ) z ( z T z ) - ' - E R("+m+l)xn

p = R++((N)[Z(ZTZ)-1; H] E R(2n+mfl)X(2n+m+l) -

Then from the definition of G, P can be expressed by

If, = R$$(N) [Z(ZTZ)- ' ;H] - = [ $ 1 (41)

Since

it follows from eqns. 35-38 and the definition of p that

Comparing eqn. 41 with eqn. 43 gives

R Q D F [ s G ] - l = [ C K ] (44)

Then the standard formula for the inverse of a parti- tioned matrix can be applied to yield

(45)

(46)

(47)

i;i - &-'F

__ &BcLs(N) = G-1; -

- cilwIv=~ = G - l p -

Thus,

On the other hand,

Comparing eqns. 46 and 47 concludes the theorem. Since &--, .is a member of WIV estimators, it cer-

tainly has the good properties presented in [2] and [SI. Especially, it can achieve a consistent parameter esti- mate with the minimum asymptotic covariance matrix. It is further noticed from the above theorem that the weighted matrix WIv is independent of M. Therefore, the matrix M does not have any impact on the statisti- cal accuracy of BCLS(IV), which confirms theoretically the statement in. Section 3.

5 Numerical examples

This Section presents two parameter estimation exam- ples using the presented BCLS method.

Example I : Consider the following system ~ ( t ) + 1.6y(t - 1) + 0.7y(t - 2)

= ~ ( t ) -I- 1.5u(t - 1) + 0 . 5 6 ~ ( t - 2) + ~ ( t ) where u(t) is a pseudo-binary random sequence (PRBS) with unit amplitude. The coloured noise v(t) is simu- lated by a MA model

v ( t ) = e( t ) - l . le( t - 1) + 0.2e(t - 2) where e(t) is a white noise sequence with zero mean and unit variance.

The above batch algorithm is used for ten runs with the sample data size varying from 500 to 1000 and to 1500 for each rim. Table 1 shows the mean values and the standard deviations of the estimated parameters. The corresponding results obtained by the LS method, the BELS method and the IV method are also listed in Table 1 for comparison. It can be seen from the Table that the proposed method shows a high accuracy of parameter estimates which can be comparable to the BELS method and the IV method.

Example 2: To illustrate the performance of the presented metlhod under different noise models, Example 1 is again considered but the coloured noise is simulated by the following more complex model:

v ( t ) =z 1.0 + 1 . 5 ~ ~ ' + 0 . 7 5 ~ ~ 1.0 - 0.9z-l + 0 . 9 5 ~ ~ ~

The experiment is performed under the same conditions as in Example 1 and the estimates obtained are summarised in 'Table 2. Since in this example the signal to noise ratio (SNR) is 1.0 on average, the standard deviations of the BCLS estimates are obviously larger than those in Example 1. However, the obtained estimates are quite acceptable. The results of Example 1 and Example 2 show that the proposed BCLS method can still achieve a high accu- racy of parameter estimates even under different noise models. This observation just tallies with the theoretical conclusion that the results of the BCLS method does not depend upon the accuracy of modelling the noise.

43 IEE Proc-Control Theory Appl., Vol. 144, No. 1, January 1997

Table 1: Results of example 1

N Method a,

500 LS 1.235i0.046

BELS 1.587t0.069

IV 1.612t0.041

BCLS 1.59220.078 1000 LS 1.22720.039

BELS 1.59520.036 IV 1.61020.030 BCLS 1.59720.055

1500 LS 1.23220.042

BELS 1.58820.031

IV 1.60720.039 BCLS 1.602-.0.047

a2

0.473i0.037

0.711i0.046

0.689i0.035

0.694i0.056

0.473i0.036 0.709+0.039 0.690+0.031 0.704*0.041 0.436i0.031

0.710i0.047

0.689t0.038 0.706i0.035

bo

1.01820.062 0.99550.053

1.010~0.032

1.007~0.051

1.011~0.050 0.98120.048 1.008*0.035 0.991.-0.053 0.98820.058

0.98750.036 1,00420.028

0,99420.046

bl

1.138i0.047 1.48920.061

1.508+0.053 1.495k0.072

1.185i0.056 1.486i0.048 1.510t0.031 1.49720.063 1.1 1720.041

1.48720.035

1.508i0.048 1.49420.054

b2 0.648rt0.027

0.573*0.054

0.55420.048

0.558rt0.069

0.650rt0.039 0.575t0.043 0.55520.047 0.56320.046 0.642*0.022

0.577rt0.040 0.557k0.035

0.564*0.042

true ~ _ _ _ _ _ ~

1.6 0.7 1 .o 1.5 0.56

Table 2: Results of example 2

N Method a, a2 bo b, b2

500 LS 0.83520.174 0.23820.1 19 1.032~0.110 0.67420.324 1.293rt0.430 BCLS 1.56620.089 0.663i0.093 1.03020.074 1.53520.114 0.540rt0.108

1000 LS 0.840i0.161 0.2 1OiO. 103 0.971 20.095 0.683*0. 127 1.21 5i0.216

BCLS 1.571rt0.084 0.67020.077 1.027i0.063 1.529i0.098 0.54520.086 1500 LS 0.838*0.096 0.229i0.083 1.040~0.105 0.67720.101 1.230i0.092

BCLS 1.569t0.068 0.66820.056 1.034i0.078 1.533t0.084 0.54720.077

true 1.6 0.7

In this paper a new bias-compensating least-squares (BCLS) method has been developed. The basic idea of this method is to extract the noise induced estimation bias from the redundant LS estimates. The weighted sum of the output and the old recorded input is employed as a new regressive signal in the parameter estimation so that two estimates about the same part of the system parameter are obtained. Based on the asymptotical analysis, the noise induced bias can be determined by using the difference between these two estimates. Thus the consistent parameter estimate is obtained by removing the bias in the ordinary LS estimate. Since, in this method, the unbiased estimate is obtained just by modifying the ordinary LS estimates, the convergence of this method is guaranted. Theoretical analysis has demonstrated that this method can achieve consistent parameter estimation even when the noise disturbing the system cannot be modelled accurately. It has also been shown that the proposed BCLS method is a particular weighted instrumental variable (WIV) method. Therefore, the obtained BCLS estimate has the same good statistical properties as those of a WIV estimate. On the other hand, such a

1 .o 1.5 0.56

conclusion also shows that the BCLS method provides a particular way to construct instrumental variables for identification of a general linear system with correlated noise.

7

1

2

3

4

5

6

7

8

9

References

LJUNG, L.: ‘System identification: theory for users’ (Prentice- Hall, 1987) SODERSTROM, T., STOICA, P.: ‘System identification’ (Pren- tice-Hall, 1989) GOODWIN, G.C., and PAYNE, R.L.: ‘Dynamic system identifi- cation: experimental design and data analysis’ (Academic Press, 1977) SODERSTROM, T., and STOICA, P.: ‘Instrumental variable methods for system identification’ (Springer-Verlag, 1983) FENG, C.B., and ZHENG, W.-X.: ‘Robust identification of sto- chastic linear systems with correlated noise’, ZEE Proc. D, 1991, 38, pp. 484-492 ZHENG, W.-X., and FENG, C.B.: ‘Identification of a class of dynamics errors-in-variables models’, Int. J. Aclupt. Contr. Sign. Proc., 1992, 6, pp. 431440 FENG, C.B., and ZHANG, Y.: ‘Unbiased identification of sys- tems with nonparametric uncertainties’, ZEEE Trans., 1995, AC- 5 , pp. 933-936 STOICA, P., SODERSTROM, T., and SIMONYTE, V.: ‘Study of a bias-free least squares parameter estimators’, IEE Proc. D, 1995, 142, pp. 1-6 STOICA, P., and SODERSTROM, T.: ‘Bias correction in least squares identification’, Int J. Control, 1982, 35, pp. 449457

44 IEE Proc.-Control Theory Appl., Vol. 144, No. I , January 1997