11
1. INTRODUCTION Tracking of Crossing Targets with Imaging Sensors HEMCHANDRA M. SHERTUKDE, Member, IEEE University of Hartford University of Connecticut YAAKOV BAR-SHALOM, Fellow, IEEE We presenl an algorilhm lor the lracking of crossing targets using the cenlroid measuremenl and lhe centroid offset measuremenl of (he distributed image formed by lhe lye& The measurements are obtained by a forward-looking hf&d (FLIR) imaging sensor. Overlap of target images occurs when lhe lines of slght of the largets cross The resulting centroid measurement of the dktribuled image is a single merged measurement which is a linear combination of the cemlroids of lhe individual largets under consideralion Also the ensuing image correlalion coefflcient matrix between two frames is mullimodal and care has lo be laken (0 properly associute the derived offset measurements lo a particular 1argeL The overlap of the images for several sampling limes causes a dependence of (he slale eslimation errors for (he two targels, which has lo be laken into accounL The join1 probabilistic &la association (JPDA) mrged-measurement coupled filter is enployed for dale estimation which perform filtering in a coupled manner for the targets wllh common measurements. ’ho fllters are examined one assuming the displacement noise white and the other one modeling it correctly as autocorrelated. The latter is slwwn to yield substantially better performenee. The simulation results presenled validate the performance predictions of (he proposed algorithm Manuscript received April 29,1989; revised October 1, 1990 and February 4,1991. IEEE Log No. 911oo901. This work was supported in part by the Faculty Research Release lime awarded by the Electrical Engineering Department of the University of Hartford, W. Hartford, CT and by the Office of Naval Research under Contract N00014-K-0059. Authors’ addresses: H. Shertukde, Electrical Engineering Dep’t., University of Hartford, W. Hartford, CT 06117; Y. Bar-Shalom, Dep’t. of Electrical and Systems Engineering, University of Connecticut, U-157, Stons, CT 06269-3157. 0018-9251/91/0700-05&2 $1.00 @ 1991 IEEE Over the past decade considerable research was done to examine several methods of tracking using forward-looking infrared (FUR) sensors. In 112-151 an extended Kalman filter was used for tracking with nonlinear measurements being the observed pixel intensities. This requires an accurate model of the target intensity distribution parameters in the focal plane of the sensor, and an adaptive scheme to estimate these parameters was also proposed. This approach is costly since it utilizes a 64dimensional measurement vector. Other work [16,19] used linear measurements (centroid offset (displacement) measurement derived from correlation of adjacent frames), but the measurement noise was assumed to be white and its variance assumed known. As shown in [3], the offset measurement noise is autocorrelated and is the output of a moving average system driven by white noise. The simulation results in [3] validated the theoretically derived statistical characterization of the measurement noises and showed that the best estimation performance is obtained accounting for the autocorrelated offset measurement noise. In air or space defence, the problem of crossing targets with measurements obtained with a F U R sensor is critical. A practical situation would be where the tracking is done based on a small image of a jet engine exhaust or a missile exhaust plume. The measurement errors are due to sensor noise, optics and the irregularity of the exhaust in addition to the atmospheric jitter (the latter is discussed in [12]). These errors get compounded in the case of two targets whose lines of sight from the sensor cross. The crossing of lines of sight results in ambiguity of the offset measurements from the two targets over several sampling times. The overlap of the two images also generates a distributed target image with a centroid that can be assumed a linear combination of the centroids of the individual targets. In light of this, proper association of the measurements to targets can be difficult. Furthermore, this “sharing” of the measurements over several sampling times, causes a dependence of the state estimation errors for the two targets. This aspect of the problem has been handled in the sonar tracking of multiple targets using the joint probabilistic data association (JPDA) technique as in [2, 91. This was further augmented with a measurement merging model in [SI. An extension of the JPDA called the JPDA coupled filter algorithm [4, 61 which performs filtering in a coupled manner for targets with common measurements is employed for state estimation. The merged measurement is accounted for with a model as in [SI. The resulting algorithm, which is the main result of this work, is the JPDA merged-measurement coupled filter (JPDAMCF). state estimation algorithm for the two targets under Section I1 deals with the problem formulation. The 582 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991

Tracking of crossing targets with imaging sensors

Embed Size (px)

Citation preview

Page 1: Tracking of crossing targets with imaging sensors

1. INTRODUCTION

Tracking of Crossing Targets with Imaging Sensors

HEMCHANDRA M. SHERTUKDE, Member, IEEE University of Hartford

University of Connecticut YAAKOV BAR-SHALOM, Fellow, IEEE

W e presenl an algorilhm lor the lracking of crossing targets using the cenlroid measuremenl and lhe centroid offset measuremenl of (he distributed image formed by lhe l y e & The measurements are obtained by a forward-looking hf&d (FLIR) imaging sensor. Overlap of target images occurs when lhe lines of slght of the largets cross The resulting centroid measurement of the dktribuled image is a single merged measurement which is a linear combination of the cemlroids of lhe individual largets under consideralion Also the ensuing image correlalion coefflcient matrix between two frames is mullimodal and care has lo be laken (0 properly associute the derived offset measurements l o a particular 1argeL The overlap of the images for several sampling limes causes a dependence of (he slale eslimation errors for (he two targels, which has lo be laken into accounL The join1 probabilistic &la association (JPDA) mrged-measurement coupled filter is enployed for dale estimation which perform filtering in a coupled manner for the targets wllh common measurements. ’ h o fllters are examined one assuming the displacement noise white and the other one modeling it correctly as autocorrelated. The latter is slwwn to yield substantially better performenee. The simulation results presenled validate the performance predictions of (he proposed algorithm

Manuscript received April 29,1989; revised October 1, 1990 and February 4,1991.

IEEE Log No. 911oo901.

This work was supported in part by the Faculty Research Release l ime awarded by the Electrical Engineering Department of the University of Hartford, W. Hartford, CT and by the Office of Naval Research under Contract N00014-K-0059.

Authors’ addresses: H. Shertukde, Electrical Engineering Dep’t., University of Hartford, W. Hartford, CT 06117; Y. Bar-Shalom, Dep’t. of Electrical and Systems Engineering, University of Connecticut, U-157, Stons, CT 06269-3157.

0018-9251/91/0700-05&2 $1.00 @ 1991 IEEE

Over the past decade considerable research was done to examine several methods of tracking using forward-looking infrared (FUR) sensors. In 112-151 an extended Kalman filter was used for tracking with nonlinear measurements being the observed pixel intensities. This requires an accurate model of the target intensity distribution parameters in the focal plane of the sensor, and an adaptive scheme to estimate these parameters was also proposed. This approach is costly since it utilizes a 64dimensional measurement vector. Other work [16,19] used linear measurements (centroid offset (displacement) measurement derived from correlation of adjacent frames), but the measurement noise was assumed to be white and its variance assumed known. As shown in [3], the offset measurement noise is autocorrelated and is the output of a moving average system driven by white noise. The simulation results in [3] validated the theoretically derived statistical characterization of the measurement noises and showed that the best estimation performance is obtained accounting for the autocorrelated offset measurement noise.

In air or space defence, the problem of crossing targets with measurements obtained with a F U R sensor is critical. A practical situation would be where the tracking is done based on a small image of a jet engine exhaust or a missile exhaust plume. The measurement errors are due to sensor noise, optics and the irregularity of the exhaust in addition to the atmospheric jitter (the latter is discussed in [12]). These errors get compounded in the case of two targets whose lines of sight from the sensor cross. The crossing of lines of sight results in ambiguity of the offset measurements from the two targets over several sampling times. The overlap of the two images also generates a distributed target image with a centroid that can be assumed a linear combination of the centroids of the individual targets. In light of this, proper association of the measurements to targets can be difficult. Furthermore, this “sharing” of the measurements over several sampling times, causes a dependence of the state estimation errors for the two targets. This aspect of the problem has been handled in the sonar tracking of multiple targets using the joint probabilistic data association (JPDA) technique as in [2, 91. This was further augmented with a measurement merging model in [SI. An extension of the JPDA called the JPDA coupled filter algorithm [4, 61 which performs filtering in a coupled manner for targets with common measurements is employed for state estimation. The merged measurement is accounted for with a model as in [SI. The resulting algorithm, which is the main result of this work, is the JPDA merged-measurement coupled filter (JPDAMCF).

state estimation algorithm for the two targets under Section I1 deals with the problem formulation. The

582 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991

Page 2: Tracking of crossing targets with imaging sensors

’7

I t I

8 x 8 array - INACE PLANE 1 5 I

Fig. 1. Images of jet exhausts of two targets in 8 x 8 array.

consideration with overlapping images is presented in Section 111. Section IV illustrates the procedure with simulation results. Section V presents a summary and conclusions.

II. PROBLEM FORMULATION

A. Modeling Assumptions

It is assumed that there are two targets moving with nearly constant velocities and their tracks are already established. The two targets which yield Gaussian plume images [3, 121, are at distances R1 and R2 from the lens system as shown in Fig. 1. The respective images of the two targets as projected on the focal plane have signal intensity distributions Sj(5,q) j = 1,2. Following [lo, 11, 17 it is assumed that the radiation source (jet exhaust) is not opaque, such that, when the lines of sight of the two targets from the sensor cross, the resultant intensity in each pixel of the two-dimensional array is approximately the addition of the two individual intensities.

indexing is used for simplicity) is The intensity measured in pixel i (single-argument

4 = sli +sz + ni, i = 1, ..., m (1)

where sji is the signal (target) intensity of target j , j = 1,2 in pixel i and ni is the noise intensity in pixel i after subtracting its mean; m is the total number of pixels in the two-dimensional mt x m, array and is given by

As in [3] the image noise is modeled as a set of independently and identically distributed (iid) random variables with first and second moments given by

m = mtm,. (2)

E[ni] = 0 (3)

(4)

The total target related intensity in the two-dimensional array is

B. Centroid of Combined Image Versus Individual Target Centroids

The estimate of the t coordinate of the location of the centroid of the entire distributed image at time k, is, in pixel units [3],

where 5 is the 5 coordinate of the center of pixel i and the measured intensity 4 corresponding to this pixel is given by (1). The summations above are over the entire frame. As shown in Appendix A, the estimated centroid (6) of the combined image relates to the centroids of the two targets according to the equation

i c = ate, + (1 - a) tc2 + w c (7)

where tCi is the true centroid of target i, (Y is the “image-mixing parameter” discussed in Appendix A, and wc is the measurement noise which, as shown in [3] is zero-mean, white, and with variance 0,’ detailed in Appendix B. The image-mkhg parameter has to be estimated before the targets overlap which is also discussed in Appendix A.

C. Evaluation of Offset Measurement From Multimodal Image Correlation Matrix Due to Two Targets

The correlation coefficient between two frames from times k and k - 1 with displacement (offset) 6

SHERTUKDE & BAR-SHALOM. TRACKING O F CROSSING T4RGETS WITH IMAGING SENSORS

~

_ _ _ - _ _ _ _ - _ _ -

Page 3: Tracking of crossing targets with imaging sensors

t I *I

Fig. 2 Intensities for two targets in frame at three consecutive times.

C[Ij(k) - J(k)][Ij, (k - 1) - J(k - l)] {C[I i (k) - l (k) ]2C[I; . , (k - 1) - r (k - 1)]2}1/2 16(k) =

(8)

d = [dt a,]‘ (9) where

and Ii(k) is the intensity of pixel i at time k given by (1); i6 is the index of the pixel in the image at k - 1 corresponding to pixel i at k when the image from k - 1 has been shifted by d pixels (d is a 2dimensional vector of integers). The summations are only over the overlying pixels and the denomeator is %normalizing factor. The average intensities I ( k ) and I(k - 1) are also based only on the overlying pixels.

image plane, the correlation function (8) exhibits a multimodal distribution. Fig. 2 shows three consecutive sensor frames using some numerical values for the intensity distributions and the velocities for the two targets These intensities correspond to the images depicted in Fig. 3. The image of target i is moving with heading of 135’ with a nearly constant speed of one pixel per sampling time and the image of target 2

Due to the existence of two targets in the

! P

(k-11

l k l

- Targe t 1 f l i g h t p a t h

*;HC Targe t 2 f l i g h t p a t h

Fig. 3. Drget images in 8 x 8 array at three consecutive times.

is moving with heading of 27’ with a nearly constant speed of two pixels per sampling time. The target intensities in the focal plane of the sensor are Gaussian plumes as in [3], given by

, j = 1,2 -1/w - € e j ,“a: + ( W l C j Y/dj I

(10) Si(& 7) = Sj.,e

with the centers at (<cj,r]cj), semiaxes (footprint) utj , uqj, j = 1,2, assumed, for simplicity, oriented along the coordinates of the sensor.

The image correlation matrix elements between frames at k - 1 and k, and between frames at k and k + 1 are shown in Tables I and 11, respectively. The displacement measurements are obtained using parabolic interpolation for each peak with its neighbors [3]. The resulting displacement measurements contain an additive zero-mean noise wd that is not white; the error in the observed displacement between frames k - 1 and k and the error in the one between frames k and k + 1 are correlated since both are affected by the noise in frame k. The autocorrelation function of wd(k) is given in Appendix B.

From both these tables it is clear that, based on the rule of eight nearest neighbors, there are two peaks which correspond to the two targets. While this eliminates false peaks, one does not know which peak should be associated with which target to

584 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991

~ ---

Page 4: Tracking of crossing targets with imaging sensors

TABLE I Image Correlation Coefficient Matrix Between Noise Frames at k - 1 and k

0.52 -0.07 -0.15 0.42 0.13 -0.16 -0.03 0.19 0.13 -0.20 -0.16 10.721 0.51 -0.05 -0.25 0.12

-0.20 -0.24 -0.16 0.23 0.38 0.00 -0.32 -0.27 -0.18 -0.26 -0.12 0.00 0.02 -0.18 -0.23 -0.17 -0.12 -0.18 -0.06 0.09 0.07 -0.07 -0.21 -0.09 -0.10 -0.08 -0.04 0.17 0.46 0.29 -0.06 -0.10 -0.19 -0.19 -0.08 0.07 0.42 0.28 0.05 -0.27 -0.18 0.26 -0.04 -0.05 0.01 -0.21 -0.26

TABLE I1 Image Correlation Coefficient Matrix Between Noisy Frames at k and k + 1

-0.10 -0.32 -0.26 -0.19 -0.04 -0.08 -0.17 -0.16 -0.32 -0.28 -0.29 -0.15 -0.09 -0.13 -0.36 -0.31 -0.12 -0.22 -0.22 0.10 0.41 0.21 0.17 -0.28 0.21 -0.08 -0.03 0.09 0.32 10.451 0.15 -0.19 0.03 -0.14 -.09 0.35 0.36 0.26 0.04 -0.17

-0.17 -0.22 -0.10 0.24 10.481 0.17 -0.20 -0.15 -0.30 -0.32 -0.20 -0.09 0.18 0.10 -0.29 -0.40 -0.25 -0.28 -0.27 -0.17 -0.11 0.01 -0.15 -0.22

evaluate the correct displacement. Thus there exists an ambiguity in the derived displacement measurements which is accounted for by using the data association technique described in the sequel. The estimation algorithm assumes two peaks in displacement. If the signal-to-noise ratio is significantly lower one might have additional peaks and the algorithm can be modified.

D. Summary of Available Measurements

From the discussion of Subsections IIB and IIC it follows that there are two kinds of measurements available during the overlap period for the filter to process: a single merged measurement for the centroid of the distributed image, and two centroid displacement measurements with ambiguous origin.

for the filter described in the sequel. The merged single centroid measurement has white noise. The centroid displacement measurements have colored noise. First a simple tracking filter is developed ignoring the autocorrelatedness of the displacement measurement noise and, following this, a full-fledged filter is developed that accounts for this autocorrelated noise. Later a comparison between the two is carried out to show the benefits of the latter.

These are used to develop the measurement model

Ill. STATE ESTIMATION MODEL

A. Estimation with White Measurement Noise Model

Assuming that the displacement measurement noise is white, the state estimation for two crossing targets

with nearly constant velocities is done stagewise as follows.

image (obtained from segmentation), the states of the two targets under consideration are estimated separately using two filters (one associated with each target as in [3]).

is evaluated by the segmentation algorithm) the measurements available are: a) a single merged measurement for the centroid of the distributed image and b) two displacement measurements which are “shared” by both the targets (i.e., have ambiguous origin) during the overlap period. In this situation the two targets can no longer be tracked reliably with separate filters using a “nearest neighbor” assignment rule for the ambiguous measurements [2, 41. The problem can be handled by using the JPDAMCF which is presented below.

1) If there are distinct regions in the distributed

2) If the images at the sensor overlap (this

The state model is

(11) x(k + 1) = Fx(k) + Gv(k)

where the stacked vector of the states of the two targets under consideration is

with the bIock-diagonal state transition matrix, stacked process noise vector and its coefficient matrix in (11) given by

SHERTUKDE & BAR-SHALOM: TZiACKING OF CROSSING TARGETS WITH IMAGING SENSORS 585

~~ - r - - _ _ - - _ _ _ ~ _ _ _ _ _ _ ~ - _ _ _ _ _ ~

Page 5: Tracking of crossing targets with imaging sensors

The quantities x'(k), F', G', v'(k); i = 1,2 are as in [3] for the white displacement measurement noise case, where the state vector is made up of the present position, current velocity, and previous position.

The measurement model has to be modified due to the merged centroid measurement. For each coordinate we have the measurement vector given by

z (k ) =

a 0 0 1-a 0 1 0 - 1 0 0 :] 0 0 0 1 0 - 1

(14)

where the subscripts c and dj, i = 1,2, represent the centroid measurement and the centroid displacement measurements, respectively.

The parameter a which depends on the relative intensities of the targets [SI has to be found experimentally and the method to do this is illustrated in Appendix A. The stacked measurement vector in (14) has the first component which is the merged centroid measurement while components two and three which are from targets 1 and 2, or 2 and 1 (in which case they are switched around), respectively, result in the ambiguous displacement measurements. It is assumed that there are two peaks in the image correlation, otherwise the algorithm can be modified to account for false peaks [2, Sect. 9.31.

The parallel procedure in which the state is updated with the stacked vector of all measurements simultaneously can be used but is computationally more expensive [4, 201 and difficult with respect to setting up the validation matrix [S, 91 since the ambiguity is associated with only two of the three components of the measurement vector. Instead, the sequential procedure across measurements for estimating the state of the system (11) is used as follows.

sampling time. The measurement j at time k is There are two measurements available at a given

where

z(k, l ) = zc(k)

H ( k , l ) = [ a 0 0 1-a 0 01

and

0 0 0 1 0 - 1 OI- [ 1 0 - 1 0 0

H(k,2) =

Thus (16) and (17) represent the merged measurement component and the displacement measurement components in (14), respectively. The sequential updating procedure is described next.

Centroid (Merged) Measurement: Denote the predicted state at time k and its covariance as

%(k I k,O) k%(k I k - 1)

P(k I k,O) kP(k I k - 1).

(18)

(19)

The updated state with the measurement (6) at time k is

%(k I k, l ) = %(k I k,O) + W(k,l)[z(k,l) - 2(k,l)]

(20) where

i (k , l ) = H(k, 1)%(k I k,O)

W(k,l) = P(k I k,O)H(k,l)'S(k,l)-'

S(k,l) = H(k,l)P(k I k,O)H(k,l)'

(21)

(22)

+ R(k,l); R(k,l) = U," (23)

and

P(k I k,l) = P(k I k,O) - W(k, l)S(k, 1)W(k, 1)'. (24)

Centroid Displacement Measurement: ?b complete the state update with (17) we proceed as follows

is the measurement (17) and j l ( 0 ) is the index of the measurement associated with target t in event 8 at time k and

i(k,2) = H(k,2)f(k I k,l). (27)

Note from (26) the possible switching around of the measurements: one has jl(f31) = 1, j2(f31) = 2 and jl(f32) = 2, j2(f32) = 1.

586 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991

_ _ _ _ - --- ~ ~~ --

Page 6: Tracking of crossing targets with imaging sensors

The conditional probabfity for a joint association event 8 [4, ch. 121 now reduces to

1 ~ ( 0 I z k ) = Ffijl,tb [zj,(e)(k,2),~j*(e)(k,2)1 (28)

where fijl,tb is the joint probability density function (pdf) of the measurements of the targets under consideration; tj is the target to which zj(k,2) is associated in event 8 E {81,82} and c is a normalizing const ant.

The joint probabilities are not mapped into marginal association probabilities as in [SI for use in the decoupled JPDA filters. Instead, they are used directly in a coupkd filfeter [4, ch. 12; 61.

The filter gain in (25) is

W(k,2 ) = P(k I k , l)H(k,2)'S(k,2)-' (29) where the innovation covariance S(k,2) is given by

S(k,2) = H(k,2)P(k I k,l)H(k,2)' + R(k,2) (30)

where H(k,2) is as given in (17) and

is the (diagonal) noise covariance matrix for the two targets under consideration. The covariance update equation is

P(k I k,2) = P(k I k,l)- W(k,2)s(k,2)-'w(k,2)'

g P ( k I k ) (32)

with the fmal updated state being

The above summarizes the JPDAMCF with sequential updating for the centroid and displacement measurement, with the latter assumed to be white. The next subsection presents the corresponding filter for the displacement measurement model as autocorrelated.

B. Model for Estimation with Autocorrelated M eas u re men t Noise

As shown in [3] the autocorrelated displacement measurement noise wd is the output of the following white noise driven subsystem (of the moving average type)

wi(k) = Awh(k) + ,&wh(k). (34) Augmenting the target state equation with the above leads to a state equation with the measurement noise and the process noise sequences for the two targets that are white but the process noise v'(k) and the measurement noise w'(k) are correlated, as pointed out in [3]. The augmented state models for each target

are, for one coordinate

(35)

and the measurement model is

[xi (k) 1

t = 1,2 (36)

where v i@) is the motion process noise, v i (k) is the white noise input to the colored noise in the displacement measurement [denoted in (34) as wh(k), with variance IT^)^)], w',(k) is the noise in the centroid measurement, and

w$(k) = PlWh(k) = PlVi(k). (37)

The second component of the measurement vector in (35) corresponds to the displacement measurement. As in Subsection IIIA until the targets do not overlap the estimation is done separately for each target with individual filters described based on (35) and (36) [5]. The filter with autocorrelated displacement measurement noise model to be used for each target is presented below.

C. Filtering with Autocorrelated Noise Model for a Single Target

The augmented state model ( 3 9 , (36) that accounts for the colored measurement noise is denoted (without the target superscript) as

x(k + 1) = F x ( k ) + G v ( k )

z ( k ) = H x ( k ) + w(k) . (3) (39)

The measurement and process noise sequences are white but v(k) and w(k) are correlated. This is described as follows

E[Gv(k)vU)'G'] = GQG'dkj = QGG'dkj (4)

SHERTUKDE & BAR-SHALOM: TRACKING OF CROSSING TARGETS WITH IMAGING SENSORS

.____ - _ _ - _ _ _ _ _ _ _ _

587

Page 7: Tracking of crossing targets with imaging sensors

where, foUowing (37), the last equation indicates cross-correlation between the two noise sequences, i.e.,

The technique described in [l, 181 was used to obtain the state estimation filter. The plant equation can be rewritten such that it has a new process noise

arbitrary matrix T, to be determined later, one can write and

where the modified block-diagonal state transition matrix is

(52) it is not a standard filtering problem. F*(k) = F(k) - T(k)H(k).

The modified stacked input vector in (51) is

uncorrelated with the measurement noise. Using an (53)

~ ( k + 1) = Fx(k) + Gv(k) + T[z(k) - Hx(k) - ~ ( k ) ] vl' ( k )

= (F - TH)x(k) + Gv(k) - Tw(k) + Tz(k). The elements of the stacked vector in [53] are given by

(43)

Denote the new transition matrix ui' (k) = T' C p ( e I zk)Zjj(q(k), i = 1,2, j = 1,2

F * ~ F - T H (55) e

where the notation from (28) is used and

T'(k) = Gi(k)Ui(k)Ri(k)-', i = 1,2 (56) new process noise

v*(k) e Gv(k) - Tw(k) (45) and the stacked block diagonal covariance matrix between the measurement and process noise is

and new input

U* (k) Tz(k). (46) U = [U1 0 u2 0 1 . (57)

Q = [f 12]. (59)

Then one can write the new state equation Finally the modified block diagonal Q-matrix is

x(k 4- l ) = F*x(k) + v*(k) + (47) Q*(k) = Q(k) - G(k)U(k)[R(k)]-'U(k)'G(k)' (58)

where Setting the crosscorrelation between the new process noise and the measurement noise to zero

E[v*(k)w(k)'] = 0 (48) The measurement model is yields

T = GUR-'. (49) [ z d k ) 1 z (k) = Zdl(k)

Lz4(kA The covariance of the new process noise can be shown to be

E[V* (k)v*(q'] = Q - GUR-~G' 4 Q*. (50)

The state estimation problem based on (47) and (39) with the noise covariances given by (41), (a), and (50) is now a standard one and can be solved using the Kalman filter technique, e.g., as in [2].

a 0 0 0 1 - ( Y o 0 0

0 0 0 0 1 0 - 1 1 O OI 0 0

D. Coupled Filtering for Overlapping Targets with Autocorrelated Noise Model

When the targets overlap the filtering is done in a coupled manner as described below. The stacked vector of the states of the two targets under consideration is obtained similarly to Section IIIA.

The modified state transition equation that has process noise uncorrelated from the measurement noise is

x ( k + 1) = F* (k)x(k) + U* (k ) + V * ( k ) (51)

+[;;I where the subscripts c and di , i = 1,2, represent the centroid measurement and the centroid displacement measurements, respectively. The association probabilities for each target are evaluated in a similar manner as in Subsection IIIA.

588 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991

Page 8: Tracking of crossing targets with imaging sensors

WLE I11 Average Errors in < Direction and Average NSES for Entire State for %get 1 (100 Monte Carlo Runs)

A v g . A v g . A V A v g . . Norm . Vel. e&r Time Pos. Po Norm ve7:

k error 9.6: Pos. erior error 0 0.195 0.276 1 -0.004 0.14 -0.025 0.0086 0.088 0.098 2 0.013 0.13 0.098 0.006 0.07 0.09 3 0.017 0.122 0.136 0.007 0.066 0.11 4 -0.01 0.118 -0.11 0.01 0.064 0.156 5 0.002 0.118 0.016 -0.005 0.0639 -0.08 6 -0.0004 0.1175 -0.003 -0.0016 0.0637 -0.025 7 0.007 0.1175 0.062 0.003 0.0637 0.05 8 0.027 0.1175 0.23 0.011 0.0637 0.22 9 -0.004 0.1175 -0.03 -0.004 0.0637 -0.06

5.90 5.96 6.33 6.10 5.82 5.86 6.03 6.4 4 6.11

TABLE IV Average Errors in < Direction and Average NSES for Entire State for %get 2 (100 Monte Carlo Runs)

Time k 0 1 2 3 4 5 6 7

A v g . A v g . Pos. Po Norm. error s.2 os. error

0.195 -0.005 0.14 -0.013 0.13 -0.01s 0.122 0.009 0.118 -0.02 0.118 0.004 0.1175 0.013 0.1175

-0.008 0.1175 -0.01 0.1175

-0.032 -0.1

-0.123 0.07

-0.173 0.003 0.113 -0.07 -0.09

A v g . . Norm . Vel. e+or

The JPDAMCF for the state model (51) and measurement model (60) is implemented by a corresponding sequential updating procedure as discussed in Subsection IIIA, considering first the update due to the centroid merged measurement and then the update due to the ambiguous centroid displacement measurements.

IV. SIMULATION RESULTS

The simulations were carried out for two filters: one assuming the displacement measurement noise white and the other one modeling it correctly as autocorrelated.

A. Filter with White Measurement Noise Model

%o targets with Gaussian plume images as in Subsection IIC and with motion described by secondader kinematic models (see [2]) as in Subsection IIIA with white noise acceleration with variance q = (0.05)2 in each of the two coordinates were considered as shown in Fig. 3. The image of target 1 is moving with a heading of 135' with a nearly constant speed of one pixel per sampling period, and image of target 2 is moving with a heading of 27' with a nearly constant speed of two pixels per sampling

0.276 0.016 0.088 0.005 0.07 -0.007 0.066 0.007 0.064

-0.006 0.0639 0.015 0.0637 0.013 0.02

-0.006 0.0637 -0.001 0.0637

0.19 0.07 -0.1 0.11 -0.2

0.024 0.094 -0.094 -0.02

6.63 6.05 5.85 5.57 5.66 6.12 6.20 5.93 6.26

period. The sampling period is T = 1. The filters were initialized using two-point differencing [4] and the initial state vectors of the targets in the image (pixel) frame were

'hrget 1 : [-3.5 1 -4.5 3.5 -1 4.51'

'hrget 2 [-3.5 1 -4.5 -8.5 2 -10.51'.

The target state estimation was done for ten sampling times. The crossing of the target images takes place at times 3, 4, and 5. The measurement noise had variance U,' = (0.195)2, U; = (0.104)2. The results of 100 Monte Carlo runs are shown for target 1 and target 2 in 'hbles I11 and IV, respectively. The results indicate that the filters are consistent. The results are given only for the < coordinate but the average normalized state error squared (NSES) pertains to the entire state.

B. Filter with Autocorrelated Measurement Noise Model

The simulations were repeated for the two targets with the displacement measurement noise modeled as autocorrelated. The state equation was (51) and the measurement equation was (60) with correlation coefficient a in (68) equal to -0.5. All the other parameters were as in Subsection IVA. The results

SHEJXIUKDE & BAR-SHALOM: TRACKING OF CROSSING TARGETS WITH IMAGING SENSORS 589

Page 9: Tracking of crossing targets with imaging sensors

TimLE V Average Errors in C Direction and Average NSES for Entire State for Wget 1 (100 Monte Cado Runs)

A v g . A v g . A V A v g . 1 Time Pos. Pos. Norm. Ve?: Vel. Norm. k error s.d. Pos. error error s.d. Vel. error ( 8

0 1 2 3 4 5 6 7 6

9

0.195 -0.006 0.13 -0.012 0.12 -0.05 0.11 -0.014 0.099 -0.015 0.094 -0.007 0.092 -0.001 0.088 -0.0001 0.008 -0.024 O.O8t?

-0.047 -0.105 -0.048 -0.142 -0.159 -0.076 -0.012 -0.001 -0.027

0.005 0.07 -0.027 0.06 -0.007 0.057 0.005 0.056 0.002 0.056 0.004 0.056 0.001 0.056 -0.0001 0.056 -0.006 0.056

0.276 0.072 -0.033 -0.122 0.080 0.034 0.079 0.107 -0.002 -0.107

9.14 8.06 7.99 8.46 8.40 8.06 7.95 6.33 8.35

TABLE VI Average Errors in < Direction and Average NSES for Entire State for 'Ihrget 2 (100 Monte Carlo Runs)

A v g . Time Pos. Pos.

IC error s.d.

0 0.195 1 -0.006 0.13 2 -0.013 0.12 3 -0.011 0.11 4 0.012 0.099 5 -0.017 0.094 6 -0.02 0.092 7 0.009 0.088 8 -0.008 0.088 9 -0.005 0.088

A v g . AV Norm. Ve?:

POS. error error

-0.047 -0.114 -0.11 0.121 -0.181 -0.217 0.102 -0.086 -0.061

0.005 0.005 0.002 0.007 -0.005 -0.009 -0.003 -0.001 -0.006

of N = 100 Monte Carlo runs are shown for target 1 and target 2 in Thbles V and VI, respectively. The average normalized errors are seen to be within the bounds of zero-mean Gaussian with standard deviation l/m = 0.1 and the values of the average NSES are also close to their theoretical value of 8 (it has a standard deviation of 0.4). The results show that the filters are consistent and best performance is achieved by incorporating the model of the autocorrelated measurement noise (cf. Table V versus 111) and rms tracking accuracies under a tenth of pixel are obtained. The results are indicated only for the coordinate but the average NSES pertains to the entire state. Since the covariance equations are independent of the measurements, they can be iterated forward off-line. Thus one can focus only on the theoretical standard deviation values given in Tables I11 and V.

V. SUMMARY AND CONCLUSIONS

The proposed algorithm demonstrates the usefulness of the JF'DAMCF for tracking crossing targets in combination with the recently developed models for the centroid and offset measurements. Even though the centroid offset measurement requires more computations and a more complex model for estimation, it yields significantly better results, namely

::I% A v g .

Vel. Norm. s.d. Vel . error (8 states)

0.07 0.06 0.057 0.056 0.056 0.056 0.056 0.056 0.056

0.276 0.072 0.083 0.035 0.125 -0.089 -0.161 -0.054 -0.018 -0.104

9.01 7.78 8.08 8.03 8.32 8.67 8.37 8.22 7.97

50 percent mean square error (MSE), if the filter accounts for its colored measurement noise.

APPENDIX A. EVALUATION OF "IMAGE-MIXING" PARAMETER

Let us consider the image plane intensity distribution as in Fig. 3 (with two Gaussian plumes shown by their 3u contours at time k - 1) such that the distributed image has two distinct regions, indicating the existence of two targets with their lines of sight from the sensor not crossing. Since the target images in the array are separated it is assumed that the array has been segmented into two nonoverlapping regions 3 and 7 2 corresponding to each image such that

I I n Z = O , Zu72=7 (61)

where 7 is the set of pixels in the entire array. It is assumed that the target signal is negligible outside its segmented region 3, j = 1,2.

the centroid of the distributed image at time k, is, in pixel units [3]

The estimate of the < coordinate of the location of

(62) ,. CSiIi Q = - E4

where is the < coordinate of the center of pkel i and the measured intensity 4 corresponding to this

590 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991

- -- _ _ - __ - -~

Page 10: Tracking of crossing targets with imaging sensors

pixel is given by (1). The summations above are over 7 . Equation (62) can be approximately written as

t i ( s ~ i + ni) + CiE72&(~2i + ni) j c =

CiE7(~1i + ~ 2 i + ni) CiE7(s1i + s2i + ni) (63)

which after some manipulations yields

c = i c , Q + i c z ( l - Q) (64) where

is the image-mixing parameter. The estimate of this parameter is obtained as

when the targets do not overlap, to be used later when the targets overlap.

APPENDIX B. STATISTICAL CHARACTERIZATION OF THE MEASUREMENT NOISES

The centroid measurement noise for the merged image is zero-mean, white with covariance [3]

where m is the total number of pixels in the array, s is the total signal available (from both targets) and u is the variance of the video noise in each pixel.

The centroid displacement measurements contain an additive zero-mean noise with autocorrelation [3]

~ [ ~ d ( k ) ~ d ( j ) l = (6kj + a b k , j - l + a6k , j+ l )g ; (68) where a is the correlation coefficient of two displacement measurement errors that are adjacent in time and

and

where s is the total target-related intensity due to both targets and S is the average target intensity over the entire frame. As shown in [3], a = -0.5. The constants a1 and a2 are

(71) A *

Ql=pat l - /%- l

Q2=2(2pa A -p6-1 - p 6 t 1 )

where p6-1 < > p6+1 (73)

are the highest image correlation with its neighbors (for displacements in one direction, the subscript omitted for simplicity). The estimated displacement (displacement measurement) is obtained from a parabolic interpolation between the three displacements 6 - 1, 6, 6 + 1 with the peak in the center and (69) is the variance of the resulting error

or 7, with

[31.

REFERENCES

Anderson, B. D. O., and Moore, J . B. (1979) Optimal Filtering. Englewood Cliffs, NJ: Prentice-Hall, 1979.

Tracking and Data Association. New York: Academic Press, 1988.

Extraction and optimal use of measurement from an imaging sensor for precision target tracking. In Proceedings of the 1989 IEEE ICCON, Jerusalem, Israel, Apr. 1989; Also in IEEE Transactwns on Aerospace and Electronic Systems, 25, 6 (Nov. 1989), 863472.

Multitarget-multisensor tracking. Short course notes, University of California at Los Angeles and University of Maryland, 1988-1989.

Precision target tracking for small extended objects. In Proceedings of the I989 Society of Photo-Optical Instrumentation Engineers Conference, Orlando, FL, Mar. 1989. Also in Optical Engineering Journal, 29, 2 (Feb.

Bar-Shalom, Y., and Fortmann, T. E. (1988)

Bar-Shalom, Y., Shertukde, H. M., and Pattipati, K. R. (1989)

Bar-Shalom, Y. (1988)

Bar-Shalom, Y., Shertukde, H. M., and Pattipati, K. R. (1989)

1990), 121-126. Blake, S., and Watts, S. C. (1987)

A multitarget track-while scan filter. In Proceedings of the IEE Radnr-87 Conference, London, UK, Oct. 1987.

Estimation of object motion parameters from noisy images. IEEE Pansactiom on Pattern Anllrysis and Machine Intelligence, PAMI-8 (Jan. 1986), 90-99.

Joint pmbabilistic data association for multi-target tracking with possibly unresolved measurements and maneuvers. IEEE Transactiom on Automatic Control, AC-29 (July

Fortmann, T. E., Bar-Shalom, Y., and Scheffe, M. (1983)

Broida, T. J., and Chellappe, R. (1986)

Chang, K. C., and Bar-Shalom, Y. (1984)

1984), 585-594.

Sonar tracking of multiple targets using joint probabilistic data association. IEEE Journal of Oceanic Engineering, OE-8, 3 (July 1983), 173-184.

Introduction to Fourier Optics. New York McGraw-Hill, 1968.

Goodman, J. W. (1985) Statistical Optics. New York: Wiley, 1985.

A target tracker using spatially distributed infrared measurements. IEEE Transactions on Automatic Control, AC-25, 2 (Apr. 1980), 222-225.

Goodman, J. W. (1968)

Maybeck, F'. S., and Mercier, D. E. (1980)

SHERTUKDE & BAR-SHALOM: TRACKING O F CROSSING TARGETS WITH IMAGING SENSORS

1 - -~

591

Page 11: Tracking of crossing targets with imaging sensors

[13] Maybeck, I? S., Jensen, R. L., and Hernly, D. A. (1981) An adaptive extended Kalman filter for target image tracking. IEEE Transactions on Aermpace and Electronic Systems, AES-17 (Mar. 1981), 173-180.

Adaptive tracking of multiple hot-spot target IR images. IEEE Transactions on Automatic Control, AC-28, 10 (Oct.

[14] Maybeck, F! S., and Rogers, S. K. (1983)

1983), 937-943. [15] Maybeck, P. S., and Suizu, R. I. (1985)

Adaptive tracker field-of-view variation via multiple model filtering. IEEE Transactions on Aerospace and Electronic S y s t m , AES-21, 4 (July 1985), 529-539.

[16] Mitzel, J. H., and Elsey, J. C. (1988) Ttacking and automatic target recognition with an imaging sensor. In Muhitarget-Multisensor Tracking 11: Advanced Applications, Y. Bar-Shalom, Coordinator, University of California at Los Angeles Extension Short Course, Jan. 1988.

[17] Pohlig, S . C. (1987) An algorithm for detection of moving optical targets. IEEE Transactions on Aerospace and Electronic Systems, AES-25 (Jan. 1987), 5643.

Estimation Theory with Applications to Communications and Control. New York McGraw-Hill, 1971.

Enhancements to a multiple model adaptive estimator/image tracker. IEEE Transactions on Aerospace and Electronic Systems, AES-24 (July 1988), 417-426.

Kalman filter algorithms for a multisensor system. In Proceedings of the IEEE Conference on Decision and Control, Clearwater Beach, FL, Dec. 1976.

[18] Sage, A. I?, and Melsa, J. L. (1971)

[19] Tobin, D. M., and Maybeck, P. S . (1988)

[ZO] Willner, D., Chang, C. B., and Dunn, K. P. (1976)

Hemhandra M. Shertukde (S’84, M’89) was born in Bombay, India, on April 29, 1953. He received the Bachelor of Technology (HONS) degree with distinction from the Indian Institute of Technology, Kharagpur, India, in 1975, in electrical engineering, the Diploma in Management Studies from the Bombay University in 1980, and the Master of Science and Ph.D. degrees in electrical engineering from the University of Connecticut in 1985 and 1989, respectively.

He is an Assistant Professor at the University of Hartford, West Hartford, CT, since the Fall of 1988. From August 1975 to November 1977 he worked for Thta Engineering and Locomotive Company in Pune, India, as an Assistant Engineer in the Maintenance Department. From December 1977 to August 1983 he worked for Crompton Greaves Limited, Bombay, India, and was a Senior Executive in the Design and Production Departments of the lkansformer Division (extra high voltage). He received his training in the manufacture and design of EHV gappedcore shunt reactors at Westinghouse Canada Ltd., Hamilton, Canada, and Westinghouse Electric Corporation, Pennsyivania, in 1981. He offers short courses in HV dc Theory and Operations for power systems operators. His recent research interests are in estimation and filtering, information theory, and digital signal processing. He has published several papers dealing with multi-sensor tracking in clutter using the “track before detect” approach and has also conducted a short course in Multisensor Data Fusion for Tracking at the 1989 SPIE Symposia on Aerospace Sensing. He recently co-authored a book Muhitarget-Multisensor Tracking: Applications and Advances (Edited by Yaakov Bar-Shalom, Artech House, Vol. I1 1991).

He is a member of B u Beta Pi and Eta Kappa Nu honor societies. Dr. Shertukde received a fellowship for the doctoral dissertation at the University of Connecticut.

Yaakov Bar-Shalom (S’63-M’66SM’&F84) was born on May 11, 1941. He received the B.S. and M.S. degrees from the Technion, Israel Institute of Technology, in 1963 and 1967 and the Ph.D. degree from Princeton University in 1970, all in electrical engineering.

From 1970 to 1976 he was with Systems Control, Inc., Palo Alto, California. Currently he is Professor of Electrical and Systems Engineering at the University of Connecticut. His research interests are in estimation theory and stochastic adaptive control and has published over 100 papers in these areas. He coauthored the monograph Tracking and Data Association (Academic Press, 1988) and edited the books Muhitarget-Muhisensor Packing: Applications and Advances (Artech House, Vol. I 1990 Vol. I1 1991). He has been elected Fellow of IEEE for “contributions to the theory of stochastic systems and of multitarget tracking”. He has been consulting to numerous companies, and originated a series of Multitarget-Multisensor lkacking short courses offered via UCLA Extension, University of Maryland, at Government Laboratories, private companies and overseas. He has also developed the interactive software packages MULTIDAT for automatic track formation and tracking of maneuvering or splitting targets in clutter, PASSDAT for data association from multiple passive sensors and BEARDAT for target localization from bearing and frequency measurements in clutter.

During 1976 and 1977 he served as Associate Editor of the IEEE Transactions on Automatic Control and from 1978 to 1981 as Associate Editor of Automalica. He was Program Chairman of the 1982 American Control Conference, General Chairman of the 1985 ACC, and Co-chairman of the 1989 IEEE International Conference on Control and Applications. During 1983-87 he served as Chairman of the Conference Activities Board of the IEEE Control Systems Society and during 1987439 was a member of the Board of Governors of the IEEE CSS. In 1987 he received the IEEE CSS Distinguished Member Award.

592 IEEE TRANSACTIONS ON AEROSPACE AND ELECTRONIC SYSTEMS VOL. 27, NO. 4 JULY 1991