Identification of Nonlinear Differential EquationModels from Generalized Frequency Response

FunctionsAkshya K Swain

Department of Electrical & Computer EngineeringUniversity of Auckland,Private Bag-92019,

New Zealand, Email: a.swain@auckland.ac.nz

E.M.A.M.MendesDepartment of Electronic EngineeringUniversidade Federal de Minas Gerais,

Av. Antonio Carlos 6627, BH-MG,31270-901,Brazil, Email: emmendes@cpdee.ufmg.br

Abstract— A new algorithm called the total least squares withstructure selection (TLSSS) has been proposed to identify contin-uous time differential equation models from complex frequencyresponse data. The algorithm combines the advantages of boththe total least squares and orthogonal least squares with structureselection (OLSSS). The error reduction ratio (ERR) feature ofOLSSS are exploited to provide an effective way of detectingthe correct model structure or which terms to include into themodel and the total least squares algorithm provides accurateestimates of the parameters when the data is corrupted withnoise. The performance of the algorithm has been comparedwith the weighted complex orthogonal estimator and has beenshown to be superior.

I. INTRODUCTION

The problem of identifying continuous time systems is offundamental importance in various applications in engineeringand science. The most obvious reasons for working withcontinuous time models is that most physical systems areinherently continuous in time and parameters of such modelsoften have a simple physical interpretation. However, theliterature of system identification is dominated with algorithmswhich fit discrete parametric models to the underlying contin-uous time systems [1] [2]. This is partly due to the suitabilityof discrete models for digital implementation and partly due tothe difficulty associated with continuous time modelling whichis far more challenging than their discrete counterpart.

There has been increasing interest amongst researchers tofit continuous time models in the past few decades [3] [4]. Amajor difficulty of identification of continuous time models isassociated with the numerical errors induced while computingderivatives of input-output from raw (noisy)data. Several tech-niques using Poisson moment functional, piecewise constantblock functions, Hermite and Legendre polynomials, and dif-ferent types of modulating functions have been proposed byresearchers in the past to minimize and possibly to avoid theseerrors during estimation [5] [6] [7].

Another approach to avoid the errors arising due to differen-tiation is to utilize the frequency domain information to fit con-tinuous time models. Several techniques have been developedto estimate continuous time models for linear systems usingfrequency response data which has been comprehensively

reviewed in [8]. Extending these techniques into nonlinearsystems are not without difficulties. Several algorithms havebeen proposed in the past decade to estimate parameters ofnonlinear differential equation models exploiting the invariantfeature of generalized frequency response functions [9] [10][11]. The great advantage of this approach is that directcalculation of derivatives of the input-output signals is avoided.However, in earlier works, emphasis has been given to noisefree case and the the accuracy of the estimates is heavilydependent on the choice of the frequency range and theweighting parameter [10]. Li and Billings [12] suggesteda method based around instrumental variables to produceunbiased estimates from the noisy frequency response data.

The present study introduces a new algorithm based ontotal least squares to obtain accurate estimates of the para-meters under noisy circumstances. This algorithm combinesthe feature of total least squares with the error reductionration test of Swain and Billings [10] to include significantterms into the model. The estimation procedure is progressivewhere the terms associated with each order of nonlinearityis estimated beginning with the estimation of linear terms ,followed by estimation of quadratic nonlinear terms and soon. The performance of the algorithm has been compared withthe weighted orthogonal estimator [10] at different levels ofnoise considering an example of a nonlinear resonator and hasbeen shown to be superior.

The paper is organized as follows. Section-II introducesthe total least squares with structure selection algorithm. Insection-III, the estimation procedure to fit continuous timemodels into the frequency response data is described. Resultsof simulation are included in section-IV to demonstrate theeffectiveness of the proposed algorithm.

II. TOTAL LEAST SQUARES WITH STRUCTURE SELECTION

Since the structure selection of the TLSSS algorithm isdone using an error reduction ratio (ERR) test which is abyproduct of orthogonal least squares algorithm [13] [10],it is appropriate to introduce briefly about the ERR test andorthogonal least squares.

A. Orthogonal Least Squares with Structure Selection

Consider a system which can be modeled as

z(jω) =M∑i=1

θipi(jω) + ξ(jω) (1)

where θi,i = 1, ......M are the real unknown deterministic pa-rameters of the system associated with the complex regressorspi(jω),i = 1, ....M . z(jω) is a complex dependent variable orthe term to regress upon and ξ(jω) represents the modelingerror. If ‘N’ measurements of z(jω) and pi(jω) are availableat ωi, i = 1, .....N , the complex system of equation (1) canbe represented in matrix form (after partitioning into real andimaginary parts) as

Z = Pθ + Ξ (2)

where P is an 2N × M matrix of known regressors and Ξ =2N × 1 real vector.. The matrix ‘P’ will be referred to as theinformation matrix. By using QR factorization, matrix P canbe decomposed into

P = QR (3)

where R is an M × M triangular matrix with 1’s on thediagonal and 0’s below the diagonal and Q is an 2N × Mmatrix with orthogonal columns qi such that, QT Q = D andthe diagonal elements of D are given as

di = qTi qi =

2N∑j=1

qi(ωj)T qi(ωj), 1 ≤ i ≤ M (4)

Since the space spanned by the set of orthogonal basis vectorsqi is the same space spanned by the original set of vectors pi,itis possible to represent equation (2) by the auxiliary equation

Z = Qg + Ξ (5)

The orthogonal least squares solution g can be calculated from

g = D−1QT Z or,

gi =qTi Z

qTi qi

, 1 ≤ i ≤ M (6)

The original parameters θ can be recovered by solving thetriangular system

Rθ = g (7)

Multiplying equation (5) by itself and taking the time averagegives

1N

ZT Z =1N

M∑i=1

g2i qT

i qi +1N

ΞT Ξ (8)

The output energy consists of two terms; the first term isthe part of output energy explained by the regressors whereasthe second term accounts for the unexplained energy. Due tothe orthogonal nature of qis, the contribution that each termmakes to the output energy can be computed independentlyas g2

i qTi qi. Expressing this as a fraction of the overall output

energy gives the Error Reduction Ratio (ERRi)

[ERR]i =giq

Ti qi

ZT Z, 1 ≤ i ≤ M (9)

Thus the value of ERR indicates the significance of a candidateterm and is used to include relevant terms into the modelin accordance with their contribution to the overall outputvariance (energy). Normally at the beginning all availablecandidate terms are examined and the term which contributesthe maximum ERR is included in the model. This is repeateduntil all candidate terms have been exhausted or the sum ofERR approaches 100 percent.

B. Total Least Squares Solution

Consider the system of equation (2). Most of the standardleas squares based algorithm assume that the output (depen-dent) variable Z contains error and the regressor matrix Pis error free. However, as it will be shown in the followingsection, the regressors in the frequency domain formulationare also corrupted by noise. The total least squares algorithm[14] [15] [16] estimates the parameters in such cases byminimizing the following Frobenius norm :

minimize||∆P ; ∆Z||F (10)

where ∆P and ∆Z represent errors in P and Z respectively.The algorithm uses singular value decomposition and is com-bined with the ERR test to yield TLSSS [17]:

C. TLS solution with Structure Selection

Given P ∈ RN×M and Z ∈ RN , the following algorithmcomputes the total least squares estimate of θ denoted as θTLS

such that equation (10) is satisfied. The relevant terms areselected using the ERR test.

1) Apply the orthogonal least squares or weighted orthogo-nal least squares algorithm to select the significant terms.

2) Apply SVD algorithm to the upper triangular part of thematrix [P (1 : M, 1 : M); Z(1 : M ] resulting from theQR factorization.

3) Save the matrix V where V is obtained from the SVDalgorithm

4) Determine the rank η5) Compute a Householder matrix Γ such that V = V Γ,

then V (M + 1, M − η + 1 : M) = 0if vM+1,M+1 �= 0for i = 1 : Mθi=−vi,M+1./vM+1,M+1

endend

III. GENERALIZED FREQUENCY RESPONSE FUNCTIONS

Before formulating the estimation problem in the frequencydomain, the concept of generalized frequency response func-tion (GFRF) is introduced.

The traditional method of representing the input-outputbehaviour of a wide class of nonlinear systems is given bythe Volterra series [18].

y(t) =N∑

n=1

yn(t) (11)

where the n-th order output of the system is given by

yn(t) =∫ ∞

−∞. . .

∫ ∞

−∞hn(τ1, . . . , τn)

n∏i=1

u(t−τi)dτi, n > 0

(12)where hn(τ1, . . . , τn) is called the n-th order Volterra kernelor generalized impulse response function of order n. Themultidimensional (n > 1) Fourier transform of the n-th orderimpulse response yields the n-th order transfer function orgeneralized frequency response function(GFRF)

Hn(jω1, jωn)

=∫ ∞

−∞

∫ ∞

−∞hn(τ1, ., τn)e(−j(ω1τ1+.+ωnτn))dτ1, .dτn

(13)

Note that when n=1, equation (13) gives the linear transferfunction.

A. Problem Formulation

Consider a system whose dynamics are described by thenonlinear differential equation

Nl∑m=1

m∑p=0

L∑l1,lp+q=0

cp,q(l1, ..lp+q)p∏

i=1

Dliy(t)p+q∏i=p

Dliu(t) = 0

(14)where ‘L’ is the maximum order of differential; p+q = m, andm = 1, ....Nl corresponds to various orders of nonlinearity.The operator ‘D’is the differential operator. Once ‘m’ takesa specific number,all the m-th-order terms, each of whichcontains a pth-order factor in ‘Dliy(t)’ and qth-order factorin ‘Dliu(t)’, subject to p + q = m, are included and eachterm is multiplied by a coefficient cp,q(l1, ...lp+q) while themultiple summation over the li(li = 0...L) generates all thepossible permutations of differentiations. For example, thegeneral linear differential equation is included in the aboveform by setting the order of nonlinearity ‘m’ as 1 to give:

L∑l1=0

c1,0(l1)Dl1y(t) +L∑

l1=0

c0,1(l1)Dl1u(t) = 0. (15)

As an example the differential equation

d2y

dt2+ a1

dy

dt+ a2y + b1

du

dt+ b2u + c1y

2

+d1u2 + c2y

3 + d2(d2y

dt2)2u = 0

(16)

would be represented as by the model of equation (14) withfollowing definitionsc1,0(0) = a2, c1,0(1) = a1, c1,0(2) = 1.0, c0,1(0) = b2,c0,1(1) = b1, c2,0(0, 0) = c1, c0,2(0, 0) = d1, c3,0(0, 0, 0) =c2, c2,1(2, 2, 0)

The frequency domain equivalent of equation (14) is basedon the generalized or higher order frequency response func-

tions which are given by mapping the time-domain represen-tation into the frequency domain [19].

−[

L∑l1=0

c1,0(l1)(jω1 + ....jωn)l1

]Hasym

n (jω1, .jωn)

=L∑

l1,ln=0

c0,n(l1, ...ln)(jω1)l1 ..(jωn)ln

+n−1∑q=1

n−q∑p=1

L∑l1,ln=0

cp,q(l1, ...lp+q)

× (jωn−q+1)lp+1 ..(jωn)lp+qHn−q,p(jω1, .jωn−q)

+n∑

p=2

L∑l1,lp=0

cp,0(l1, ..lp)Hn,p(jω1, .jωn). (17)

where the recursive relation is given by

Hasymn,p (.) =

n−p+1∑i=1

Hi(jω1, ...jωi)

× Hn−i,p−1(jωi+1, ..jωn)(jω1 + .. + jωi)lp

(18)

The recursion finishes with p = 1 and Hn,1(jω1, ....jωn) hasthe property

Hn,1(jω1, ....jωn) = Hn(jω1, ...jωn)(jω1 + .... + jωn)l1 .(19)

Equations (17-19) for the GFRF will be used to derive theregression equations for estimating the unknown parameterscp,q(.) in the continuous time model of equation (14).

Note that the computation of the n-th order frequencyresponse functions is a recursive procedure where each lowerorder of GFRF contains no effects from higher order terms.This offers a distinct advantage since the parameters corre-sponding to different nonlinearities or terms can be estimatedindependently, beginning with first order and then building upto include the nonlinear terms.

B. Estimation of Linear Terms : Synthesis of Linear TransferFunctions

The first order frequency response is only related to thelinear input-output terms. Setting n = 1 in equation (17) gives

−[

L∑l1=0

c1,0(l1)(jω1)l1

]H1(jω1) =

L∑l1=0

c0,1(l1)(jω1)l1 .

(20)Notice that multiplying by a constant on both sides has noeffect on the form of the equation except all the parameterswill be changed. This suggests that all the parameters can notbe uniquely estimated. Hence it is necessary to fix a parameterbefore the estimation begins. Without loss of generality if it isassumed that the parameter corresponding to the linear output

term c1,0(0) is unity, then equation (20) gives

−H1(jω1) =L∑

l1=1,l1 �=0

c1,0(l1)(jω1)l1H1(jω1)

+L∑

l1=0

c0,1(l1)(jω1)l1 . (21)

which is a linear in the parameters expression. Equation(20)can be put in the form of equation (1) where z(jω) =−H1(jω1)θ1 = c1,0(1), p1(jω) = (jω1)H1(jω1)θ2 = c1,0(2), p2(jω) = (jω1)2H1(jω1)...θL = c1,0(L), pL(jω) = (jω1)LH1(jω1)θL+1 = c0,1(0), pL+1(jω) = pM (jω) = (jω1)0 = 1θL+2 = c0,1(1), pL+2(jω) = (jω1)...θ2L+1 = c0,1(L), p2L+1(jω) = pM (jω) = (jω1)L

The proposed estimation algorithm can now be applied toequation (21) to identify the unknown parameters by replacingthe frequency response function H1(jω1) by estimates ofthis function. Note that when the frequency response data iscorrupted by noise, both the dependent variable and the inde-pendent regressors associators associated with the output termsare contaminated with noise. Therefore, the total least squaresalgorithm is a possible solution to get accurate estimates ofthe parameters in such cases.

C. Estimation of Second Order Nonlinearities

To estimate the parameters associated with second ordernonlinear terms, set n = 2 in equation (17) so that[

L∑l1=0

c1,0(l1)(jω1 + jω2)l1

]Hasym

2 (jω1, jω2)

=L∑

l1,l2=0

c0,2(l1, l2)(jω1)l1(jω2)l2

+L∑

l1,l2=0

c1,1(l1, l2)(jω2)l2H1,1(jω1)

+L∑

l1,l2=0

c2,0(l1, l2)H2,2(jω1, jω2). (22)

By the recursive relation

H1,1(jω1) = H1(jω1)(jω1)l1 . (23)

Hasym2,2 (jω1, jω2) = H1(jω1)H1,1(jω2)(jω1)l2

= H1(jω1)H1(jω2)(jω2)l1(jω1)l2 . (24)

So that finally a linear regression equation is obtained as

−[

L∑l1=0

c1,0(l1)(jω1 + jω2)l1

]Hasym

2 (jω1, jω2)

=L∑

l1,l2=0

c0,2(l1, l2)(jω1)l1(jω2)l2

+L∑

l1,l2=0

c1,1(l1, l2)(jω2)l2H1(jω1)(jω1)l1

+L∑

l1,l2=0

c2,0(l1, l2)H1(jω1)H1(jω2)(jω1)l2(jω2)l1

(25)

Notice that the parameters c1,0(l1), l1 = 0, ...L on the lefthand side have been estimated as linear terms initially whileall the parameters on the right hand side of the equation can beestimated from equation (25) by replacing the first and secondorder frequency response functions by their estimates andapplying the new estimator. The procedure can be followedfor estimating the higher order nonlinear terms [10].

IV. SIMULATED EXAMPLE

The effectiveness of the new estimator is demonstratedconsidering an example of the nonlinear resonator whosedynamics is governed by

y + 2ξωny + ω2ny + 0.01ω2

nu2 = ω2nu (26)

where ξ is the damping ratio, ωn is the natural frequency. Thelinear and second order GFRF of the system is given as

H1(jω1) =ω2

n

ω2n + 2ξωn(jω1) + (jω1)2

(27)

H2(jω1, jω2) =−0.01ω2

n

ω2n + 2ξωn(jω1 + jω2) + (jω1 + jω2)2

(28)

The gain and phase of both linear and second order GFRFfor ξ = 0.2 and ωn = 100π/3 are shown in Fig-1. In orderto construct the linear part of the model 100 equally spacedfrequency response data was generated from equation (27) inthe frequency range of 0-30 Hz. To demonstrate the structureselection feature of the algorithm, an over-parameterized (8thorder) transfer function was initially specified. The algorithm,which follows closely the structure selection feature of theweighted complex orthogonal estimator [10], selected thecorrect model structure. The parameters associated with thelinear terms were estimated using both total least squaresand orthogonal least squares. To estimate the parametersof nonlinear terms, 25 equally spaced frequency responsedata were used in the frequency range of 0-2 Hz. An overparameterized model of second order nonlinearity with modelterms [u2, y2, uy, y2, uy] was initially specified. The proposedalgorithm detects the correct model structure and the parameterestimates coincides with the true parameters (not shown in theresult tables).

0 10 20 30 40−20

−15

−10

−5

0

5

10

f1 (Hertz) (a)

Ga

in (

dB

)

0 10 20 30 40 50−200

−150

−100

−50

0

f1 (b)

Ph

ase

(d

eg

ree

)

0

20

40 −40−20

020

40

−60

−40

−20

f2 f

1 (c)

Ga

in (

dB

)

0

20

40 −50

0

50−200

0

200

f2f

1 (d)

Ph

ase

Fig. 1. Magnitude and phase response of nonlinear resonator (a). Gain oflinear transfer Function (b). Phase of linear transfer function (c). Gain ofsecond order GFRF (d). Phase of second order GFRF

Since in practice, the frequency response data could becorrupted by noise, the performance of the estimator wasstudied and compared with the orthogonal estimator by addingcomplex white Gaussian noise of varying SNR to the clean fre-quency response data. The magnitude response of both linearand nonlinear frequency response functions of the resonatorfor SNR=10 is shown in Fig-2. The results of the parameterestimates are summarized in Table-I and II. The error in the

Terms θ θ θ θ θSNR=30 SNR=20 SNR=10 SNR=5

y 1.0 0.9992 0.9945 0.9585 0.899y 41.9 41.8089 41.559 40.259 38.764u 10975.06 10959 10979 11389 12657u2 -109.75 -109.7511 -109.7539 -109.784 -109.845

TABLE I

PARAMETER ESTIMATES USING TOTAL LEAST SQUARES

Terms θ θ θ θ θSNR=30 SNR=20 SNR=10 SNR=5

y 1.0 0.9990 0.9926 0.9428 0.8616y 41.9 41.76 41.1043 36.2304 28.45u 10975.06 10947 10859 10249 9292u2 -109.75 -109.75 -109.7529 -109.7734 -109.8101

TABLE II

PARAMETER ESTIMATES USING ORTHOGONAL LEAST SQUARES

estimates are calculated as

ε =|θ − θ|

θ× 100 (29)

where θ and θ represents the true and estimated value of theparameters respectively. From the tables it is obvious thatas the noise level increases performance of the orthogonal

0 5 10 15 20 25 30 35 40 45 50−30

−20

−10

0

10

f1 (a)

Gain

(dB

)

010

2030

40 −40−20

020

40−100

−50

0

f2

f1 (b)

Gain

(dB

)

Fig. 2. Magnitude response of nonlinear resonator under the effects of noise(SNR=10) (a). Gain of linear transfer Function (b). Gain of second orderGFRF (d)

Terms ε ε ε εSNR=30 SNR=20 SNR=10 SNR=5

y 0.0838 0.5497 4.1457 10.1100y 0.2174 0.8140 3.9155 7.4842u 0.1476 0.0332 3.7715 15.3234u2 0.1174 0.3901 3.4031 9.5253

TABLE III

PERCENTAGE OF ERRORS IN PARAMETER ESTIMATES USING TOTAL LEAST

SQUARES AT DIFFERENT LEVELS OF NOISE.

Terms ε ε ε εSNR=30 SNR=20 SNR=10 SNR=5

y 0.1028 0.7358 5.7184 13.8358y 0.3277 1.899 13.5313 32.0811u 0.2579 1.0611 6.6133 15.3366u2 0.1042 0.2878 2.3439 6.006

TABLE IV

PERCENTAGE OF ERRORS IN PARAMETER ESTIMATES USING

ORTHOGONAL LEAST SQUARES AT DIFFERENT LEVELS OF NOISE.

algorithm degrades. However, the total least squares algorithmperforms better under noisy conditions.

V. CONCLUSIONS

A new algorithm for estimating continuous time nonlineardifferential equation models using generalized frequency re-sponse function data have been introduced.The structure selec-tion property of the algorithm helps to determine the correctmodel structure i.e. which terms to include into the model.The method does not require differentiation of the input-output data. Moreover, nonlinear models are fitted sequentiallyby building in the linear model terms followed by quadraticterms and so on. The performance of the algorithm has beencompared with the weighted complex orthogonal estimatorconsidering an example of nonlinear resonator and has beenshown to give better parameter estimates under the effect ofsignificant noise.

ACKNOWLEDGEMENTS

Akshya Swain acknowledges that part of this work wassupported by University of Auckland grant ref 3605678/9273.E. Mendes acknowledges the support of CNPq under the grant301313/96-2.

REFERENCES

[1] L. Ljung, System Identification: Theory for the User. Englewood Cliffs,New Jersey: Prentice Hall, 1999.

[2] T. Soderstrom and P. Stoica, System Identification. London: PrenticeHall, 1989.

[3] P. Young, “Parameter estimation of continuous time models-a survey,”Automatica, vol. 17, pp. 23–29, 1981.

[4] H. Unbehauen and G. P. Rao, “A review of identification of continuoustime systems,” Annual Rev. Control, vol. 22, pp. 145–171, 1998.

[5] ——, Identification of continuous time systems. Amsterdam: NorthHolland, 1987.

[6] M. Shinbrot, “On the analysis of linear and nonlinear systems,” Trans.ASME Eng., vol. 79, pp. 547–552, 1957.

[7] S. Daniel-Berhe and H. Unbehauen, “Physical parameter estimation ofthe nonlinear continuous time dynamics of a dc motor using hartleymodulating function method,” Journal of Franklin Institute, vol. 336,pp. 481–501, 1999.

[8] R. Pintelon, P. Guillaume, Y. Rollain, J. Schoukens, and H. V. Hamme,“Parameter identification of transfer functions in frequency domain-asurvey,” IEEE Transactions on Automatic Control, vol. 39, pp. 2245–2260, 1994.

[9] K. Tsang and S. A. Billings, “Reconstruction of linear and nonlinearcontinuous time models from discrete time sampled data systems,”Mechanical Systems and Signal Processing, vol. 6, no. 1, pp. 69–84,1992.

[10] A. K. Swain and S. A. Billings, “Weighted complex orthogonal estimatorfor identifying linear and nonlinear continuous time models from gen-eralized frequency response functions,” Mechanical Systems and SignalProcessing, vol. 12, no. 2, pp. 269–292, 1998.

[11] S. A. Billings and A. K. Swain, “Reconstruction of multiple-input-multiple-output nonlinear differential equation models from generalizedfrequency response function matrix,” ProcIMech (Journal of Systemsand Control), vol. 214, pp. 35–51, 2000.

[12] L. M. Li and S.A.Billings, “Continuous time nonlinear system iden-tification in the frequency domain,” International Journal of Control,vol. 74, no. 11, pp. 1052–1061, 2001.

[13] S.A.Billings, S. Chen, and M. J. Korenberg, “Identification of MIMOnonlinear systems using a frward regressor orthogonal estimator,” Inter-national Journal of Control, vol. 49, no. 6, pp. 2157–2189, 1989.

[14] G. Golub and C. F. V. Loan, “An analysis of total least squares problem,”SIAM Journal of Numerical Analysis, vol. 17, pp. 403–420, 1980.

[15] G. H. Golub and C. F. V. Loan, Matrix Computation, 2nd ed. Baltimoreand London: John Hopkins, 1989.

[16] S. V. Huffel, “Analysis of total least squares and its use in parameterestimation,” Ph.D. dissertation, Department of Electrical Engineering,Katholieke Universiteit Leuven, Belgium, June 1987.

[17] E. M. A. M. Mendes, “Identification of nonlinear discrete systemswith intelligent structure detection,” Ph.D. dissertation, Department ofAutomatic Control and Systems Engineering, University of Sheffield,June 1995.

[18] M. Schetzen, The Volterra and Wiener theories of Nonlinear Systems.New York: Wiley, 1980.

[19] S. A. Billings and J. C. P. Jones, “Mapping nonlinear integro-differentialequations into the frequency domain,” International Journal of Control,vol. 52, no. 4, pp. 863–879, 1990.