IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 3, MAY 1998 1125

Input Recovery from Noisy Output Data, UsingRegularized Inversion of the Laplace Transform

Aswini K. Dey, Clyde F. Martin,Fellow IEEE, and Frits H. Ruymgaart

Abstract—In a dynamical system the input is to be recoveredfrom finitely many measurements, blurred by random error, ofthe output of the system. As usual, the differential equation de-scribing the system is reduced to multiplication with a polynomialafter applying the Laplace transform. It appears that there existsa natural, unbiased, estimator for the Laplace transform of theoutput, from which an estimator of the input can be obtained bymultiplication with the polynomial and subsequent applicationof a regularized inverse of the Laplace transform. It is possible,moreover, to balance the effect of this inverse so that ill-posednessremains restricted to its actual source: differentiation. The rateof convergence of the integrated mean-square error is a positivepower of the number of data. The order of the differentialequation has an adverse effect on the rate which, on the otherhand, increases with the smoothness of the input as usual.

Index Terms—Ill-posed problem, inverse estimation, Laplacetransform, regularized inverse.

I. INTRODUCTION AND PRELIMINARIES

L ET us consider the simple but—for our purposes—repre-sentative example of a dynamical system where the input

function is related to the output function through thedifferential equation

with (1.1)

where and are given numbers. The outputis measured at a finite number of time points with randommeasurement error and the problem is to recover the inputfrom this finite set of corrupted data. We will see that extensionto higher order linear differential equations is straightforward.

Usually one considers the identification of parameters (see,e.g., [7]). The problem of recovering the input in a nonpara-metric manner, however, is of interest in its own right. Thisproblem seems to have been first considered in Brockett’sthesis as reported in [5]. Although these authors consideredthe multivariable problem, the analysis remains the same. Theyshow that in order to recover the input from the output of asystem it is necessary to be able to take a certain number ofderivatives of the output. (We will further comment on thisbelow.) In the case of single input–single output, the numberof derivatives that must be taken equals the relative degreeof the system. This problem arises in various forms in much

Manuscript received December 21, 1995; revised July 23, 1997. The workof A. K. Dey was supported by the NSF under Grant DMS-92-04950. Thework of C. F. Martin was supported by the NSF under Grants ECS-97-05312and ECS-97-20357. The work of F. H. Ruymgaart was supported by NSFunder Grants DMS-92-04950 and DMS-95-04485.

The authors are with the Department of Mathematics and Statistics, TexasTech University, Lubbock, TX 79409 USA.

Publisher Item Identifier S 0018-9448(98)02708-4.

of modern control theory. For example, there is currentlya large literature on systems driven by exogenous systems,i.e., It is often important to beable to understand the characteristics of the exogenous system

which can only be seen through the state(see,e.g., [6]). The problem of input reconstruction also arises incryptology [12] and in the theory of convolution codes [15],albeit that there the systems are over fields of characteristicrather than over the real numbers. The parameter estimationproblem, as described in [7], is a special case of the inputrecovery problem; by letting the input be an impulse thesystem is excited as it would be with input zero and nonzeroinitial conditon.

Yet, one of our objectives is also to draw attention to aspecial way of solving this problem, which involves manip-ulation of the Laplace transform as an injective operatormapping the Hilbert space into itself. Considering(1.1) it seems that we only need to estimatefrom the databy means of a sufficiently smooth random function, say, andthen obtain an estimator for according to

(1.2)

At this point, however, two problems emerge. The first ishow we should obtain a suitable interpolationfrom thedata. Statistical techniques from nonparametric regression (see,e.g., [9]) will be pertinent. Besides the usual averaging thesetechniques involve the choice of an extraneous kernel anda bandwidth. The second problem is the ill-posedness ofdifferentiation which requires regularization.

By preconditioning we will understand here replacing anintegral equation with an equivalent one, typically easier todeal with, by applying an injective operator to either side. Themethod that we propose is based on preconditioning with theLaplace transform, which is very natural here since it replaces(1.1) with the equivalent equation

where

(1.3)

This intrinsic way of preconditioning subsumes the choice ofa kernel. Because is a smoothing operator, an estimatorof of unquestionable quality can be constructedin a straightforward manner as we will see below. Morespecifically, turns out to be unbiased and -consistent.It is important to note that, when solving (1.3) for, the

0018–9448/98$10.00 1998 IEEE

1126 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 3, MAY 1998

real ill-posedness is due to the multiplication bybecausethe operator appears on both sides of the equation. Thismultiplication represents differentiation. More precisely, wepropose to estimate by

(1.4)

where is a suitably regularized version of (withregularization parameter ), to be specified below. We willsee that the effect of is balanced by the implicit presenceof in the expression for

A common way to assess the quality of an estimator likeis to look at its integrated mean-square error (IMSE)

(1.5)

where denotes “expectation.” This entails that we shouldmodel the problem in the framework of the Hilbert space

It has been noted in [10] that the Laplace transformis customarily considered as a transform whose range is aHardy space, but that in certain applications it is important toconsider it as an operator on Here is an instance ofsuch an application. In Chauveauet al. [8], this somewhatdifferent approach has been followed. It is based on therelation of the Laplace transform and a convolution operatoron , where is considered as a multiplicativegroup. Exploiting this a regularized inverse could be con-structed. Here we essentially follow the method of the latterpaper.

At this point some comments are in order. Berteroetal. ([3], [4]) consider the problem of Laplace inversion,mainly employing sampling points that are either equidistant orforming a geometrical progression. These authors, moreover,either assume the functions to have support in a boundedinterval or employ a weighted -norm. In the case of boundedsupport the Laplace transform may be restricted accordinglyto yield a finite Laplace transform which is compact andtherefore possesses a singular value decomposition. In [2] itis shown that the problem of parameter identification (inputrecovery with impulse control) can be solved in a numericallystable manner when the data points are equally spaced. This isessentially a variant of the classical method of Prony, see [11].In [13], however, it is shown that the problem of parameteridentification is in essence unsolvable in an exact manner ifthe points are not equally spaced. In that paper the problem ofdiscrete observability of linear systems is considered.

There are many relations between input recovery, linear con-trol theory, inversion of the Laplace transform, and numericalanalysis on the one hand, and the moment problem on the otherhand. It is difficult to exploit these relations explicitly due tothe extreme numerical instability of the moment problem. Forthese relations see, e.g., [14] and [2]; see also [1].

In this paper, the problem is considered in the statisticalcontext of indirect regression, where random noise is pertinent.Our primary aim, however, is to recover the source term ina differential equation, not the inversion of the Laplace trans-form which is only a tool. The actual ill-posedness is inducedby the required differentiation. In order to make sure that theIMSE is finite, we will also focus on input functions with a

given bounded support. Our sampling points, however, willbe chosen at random from a known density which is boundedaway from zero on that support. Randomly selected samplingpoints arise naturally when there are multiple observationsmade by observers with uncoordinated clocks. In the currentliterature this is part of the problem of data fusion (see, e.g.,the January 1997 Special Issue of the PROCEEDINGS OF THE

IEEE [17]).Our construction of the estimators is based on a regularized

inverse of the unrestricted Laplace transform, which does notrequire modification when we have to deal with another inputsignal having a different support. This is an advantage aboveusing the finite Laplace transform tailored to the support ofthe specific input considered, since the spectral propertiesdepend on the restriction. It should be noted, moreover, thatthe regularized inverse is given in a constructive manner andbased on a number of explicitly defined operators (see (2.10)and (2.15) in Section II).

The model will be further specified in Section II, wherewe also briefly review the above mentioned construction ofa regularized inverse of the Laplace transform onThe main theorem regarding the rate at which the IMSE tendsto zero as the number of data tends to infinity is containedin Section III. This rate is of polynomial order in the numberof data, which corroborates the claim that the ill-posedness ofthe inversion of the Laplace transform does not really play arole: this ill-posedness is rather bad and would have entaileda much slower rate which is polynomial in the logarithm ofthe number of data. In Section IV, we briefly sketch howthis approach might work for more general systems, and givesome conclusions in Section V.

II. M ODEL SPECIFICATION AND

INVERSE LAPLACE TRANSFORM

In this paper, we shall use the random design wherethe output is measured at observable random time points

that are independent and identically distributed(i.i.d,) with common continuous type densityhaving support

contained in If a deterministic design isused these random variables should be replaced by givennumbers The unobservable randomerrors are supposed to be i.i.d. with meanandfinite variance The random errors are also supposed to bestochastically independent of the random time points.

Our data consist of the random variables

(2.1)

where also the are observable. It will be convenient to letand be stochastically independent random variables with

the same distribution as and , and to letWe have the useful relation

(2.2)

in terms of conditional expectations.In order to make sure that (1.5) makes sense we will assume

that

(2.3)

DEY et al.: INPUT RECOVERY FROM NOISY DATA, USING INVERSION OF THE LAPLACE TRANSFORM 1127

where for some is the class of all functions onthat have bounded support and that are bounded

on that support, and is the class of all probability densitieswith support that are bounded away from zero on thatsupport. From the form of the exact solution of (1.1) it thenfollows that also Since the Laplace transform isa bounded injective operator mapping into itself itfollows that To estimate let us propose

where

(2.4)

The norm in will be written

Theorem 2.1:The estimators are in withprobability . Provided that we have, moreover,

(2.5)

where the equality is referred to as unbiasedness.Proof: The first statement is immediate, and it suffices

to prove (2.5) for one of the Let be the finite numberbounding the function and recall that(because has zero mean). In terms of the generic randomvariables , , and we now have

(2.6)

exploiting the independence of and , and (2.1). This settlesthe finiteness of the second expectation on the right in (2.6).The finiteness of the third one can be shown in a similarmanner.

For the unbiasedness note that, for

using (2.3).

Anticipating the construction of a regularized inverseof the Laplace transform that, for each , is a boundedoperator mapping into itself, note that (1.4) entails

(2.8)

so that is a random element in with finiteexpected squared norm.

To construct these regularized inverses we recall that theHaar measure on is given by

(2.9)

where denotes Lebesgue measure on The Hilbertspace of functions that are square-integrable with respect to

is denoted by , and the Hilbert space of functionsthat are square-integrable with respect to Lebesgue measureon by We also need the following three isometries:

(2.10)

and the function

(2.11)

It is possible [8] to represent as an operator onin the form

(2.12)

where for each See also Gilliametal [10] for related observations. Writing

C (2.13)

let us define

(2.14)

where for each Letbe the number such that It follows from [8, eq.(3.4)] that the expressions for the operators in (2.10) yield theexplicit expression

(2.15)

for the input estimator as defined in (1.4).

Theorem 2.2:For each the operator is bounded,and

as (2.16)

1128 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 3, MAY 1998

Proof: Let us write for the norm in Sincethe three operators in (2.10) are isometries, according to (2.12)and (2.14) we have

(2.17)

The convergence in (2.16) follows because

(2.18)

so that is strictly positive for each

III. CONVERGENCE RATE OF THE IMSE

As usual, the rate at which the IMSE in (1.5) converges tozero as the sample sizetends to infinity will depend on thesmoothness of Let us introduce the class

is continuous on IN

(3.1)

For we have

(3.2)

where

Because we have and the smooth-ness of entails that has a continuousth derivative. Sincethe last integral in (3.2) is the Fourier transform ofthis willoften boil down to

for some as

(3.3)

We are now in a position to present the main theorem aboutthe order of magnitude of the IMSE in terms of the samplesize , using the estimators as in (1.4) with definedin (2.14), in (1.3), and in (2.4)

Theorem 3.1:Suppose that , such that (3.3) issatisfied, and Choosing

the IMSE satisfies

as (3.4)

Proof: The IMSE has the usual decomposition into avariance part and a bias part. Because is an unbiasedestimator of this decomposition assumes the form

(3.5)

Let us now first consider, for arbitrary

(3.6)

For the above equality we use, again, that the operators in(2.10) are isometries. In terms of the generic, , andwe have

(3.7)

where the random functions and are given by

(3.8)

and where in the last line of (3.7) we use the property

C

It should be noted that

and

are independent of Condition (2.3), relation (2.1), and theindependence of and entail that

(3.9)

Exploiting (3.9), (2.14), and the property Cwe arrive at

DEY et al.: INPUT RECOVERY FROM NOISY DATA, USING INVERSION OF THE LAPLACE TRANSFORM 1129

(3.10)

where is the positive number such thatLet us next consider the second term in the last expression

in (3.5) for arbitrary Using (2.16) and (3.3) we obtain

as

(3.11)

where is defined as above.Combining (3.5) with (3.6), (3.10), and (3.11) we see

that the IMSE is of order Choosing suchthat yields Thecorresponding equals , of course,and the ensuing order of magnitude of the IMSE equals

, as claimed in the theorem.

As we see from the proof the effect of introducing theLaplace transform for preconditioning has been eliminated.The resulting rate of convergence, polynomial in the samplesize, relates to the ill-posedness of the differentiation which isrepresented by the multiplication with the functionin (1.3),after taking the Laplace transform.

IV. SOME POSSIBLE GENERALIZATIONS

A. Generalization of the Order of the Differential Equation

Rather than the first-order linear differential equation (1.1),the starting point might be a linear differential equation

(4.1)

of arbitrary order IN, for suitable values of the constantsand for suitable initial conditions. Again the data are givenby (2.2) and the problem is to recover Taking the Laplacetransform leads to an equation like (1.3) wherehas to bereplaced with a polynomial of order This generalizationbecomes effective in (3.7) where an expression involvingterms will be obtained, each relating to a corresponding termin the polynomial This means that (3.10) will consist of

terms, the overall order of which will be

There is no change in (3.11) so that the order of the IMSE willbe Choosing such thatyields the following result.

Theorem 4.1:Under suitable conditions, the IMSE of theestimator of the input variable , given the system (4.1),satisfies

as

provided that we take

B. Generalization to Partial Differential Equations

A more challenging extension might be to functions of morevariables where (1.1) is to be replaced by some partial differ-ential equation. Zemanian [16] introduces the-dimensionalLaplace transform

(4.3)

and gives an application to solving a partial differential equa-tion. Since this generalized Laplace transform seems to shareall the relevant properties with its univariate version, it isvery likely that a regularized inverse can be constructed alongsimilar lines.

V. CONCLUSION

In this paper, we recover the input from finitely manynoisy output data, where the system is driven by ath-orderdifferential equation. The problem is solved in the frameworkof indirect nonparametric regression. Regularized inversion ofthe unrestricted Laplace transform is used as a tool, withoutintroducing extra undue ill-posedness. We have been focusingon output functions with given bounded support and randomsampling designs, drawn according to a known density on thatsupport. The rate of convergence of the IMSE was determined.

We conjecture that modifications of this method are possibleso as to include unknown design densities as well as suitabledeterministic designs. We also conjecture that the procedureextends to systems with transfer functions whose reciprocalis rational rather than polynomial, provided that this functionhas no zeros on the positive real line. Extension to arbitrarytransfer functions, with decay to zero at infinity of specifiedorder, might even be possible.

ACKNOWLEDGMENT

The authors wish to thank the Associate Editor and thereferees for useful comments.

REFERENCES

[1] N. Akhiezer, The Classical Moment Problem and Some Related Ques-tions in Analysis. New York: Hafner, 1965.

[2] G. Ammar, W. Dayawansa, and C. Martin, “Exponential interpolation:Theory and numerical algorithms,”Appl. Math. Comp., vol. 41, pp.189–232, 1991.

[3] M. Bertero, P. Briani, and E. R. Pike, “On the recovery and resolutionof exponential relaxation rates from experimental data. III. The effectof sampling and truncation of data on the Laplace transform inversion,”Proc. Roy. Soc. London A, vol. 398, pp. 23–44, 1985.

[4] , “On the recovery and resolution of exponential relaxationrates from experimental data: Laplace transform inversion in weightedspaces,”Inverse Problems, vol. 1, pp. 1–15, 1985.

1130 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 3, MAY 1998

[5] R. Brocket and M. Mesarovic, “The reproducibility of multivariablesystems,”J. Math. Anal. Appl., vol. 11, pp. 548–563, 1965.

[6] C. Byrnes and A. Isidori, “Output regulation of nonlinear systems,”IEEE Trans. Automat. Contr., vol. 35, pp. 131–140, 1990.

[7] P. Caines,Linear Stochastic Systems. New York: Wiley, 1988.[8] D. E. Chauveau, A. C. M. van Rooij, and F. H. Ruymgaart, “Regularized

inversion of noisy Laplace transforms,”Adv. Appl. Math., vol. 15, pp.186–201, 1994.

[9] R. L. Eubank,Spline Smoothing and Nonparametric Regression. NewYork: Marcel Dekker, 1988.

[10] D. S. Gilliam, J. R. Schulenberger, and J. L. Lund, “Spectral represen-tation of the Laplace and Stieltjes transforms,”Math. Comp., vol. 7, pp.101–107, 1988.

[11] F. Hildebrand,Introduction to Numerical Analysis, 2nd ed. New York:Dover, 1987.

[12] Z. Li and C. Martin, “An inverse problem of a linear dynamic system

with noise,” Internat. J. Contr., to be published.[13] C. Martin and J. Smith, “Approximation, interpolation and sampling,”

in Differential Geometry: The Interface between Pure and AppliedMathematics(Contemporary Mathematics Series), W. Shadwick, M.Luksic, and C. Martin, Eds. Providence, RI, Amer. Math. Soc., 1987,pp. 227–251.

[14] G. Talenti, “Recovering a function from a finite number of moments,”Inverse Probl., vol. 1, pp. 1–15, 1987.

[15] E. York, J. Rosenthal, and J. Schumacher, “On the relationship betweenalgebraic systems theory and coding theory: Representation of codes,” inProc. 34th IEEE Conf. Decision and Control(New Orleans, LA, 1995),pp. 3271–3276.

[16] A. H. Zemanian,Generalized Integral Transformations. New York:Dover, 1987.

[17] P. K. Varshney,Guest Editor, Proc. IEEE(Special Issue on Data Fusion),vol. 85, pp. 3–197, Jan. 1997.