Embed Size (px)
Citation preview
1
Formation et Analyse d’ImagesSession 6
Daniela Hall
18 November 2004
2
Course Overview
• Session 1: – Homogenous coordinates and tensor notation– Image transformations– Camera models
• Session 2:– Camera models– Reflection models– Color spaces
• Session 3:– Review color spaces– Pixel based image analysis
• Session 4:– Gaussian filter operators– Scale Space
3
Course overview
• Session 5:– Contrast description– Hough transform
• Session 6:– Kalman filter– Tracking of regions, pixels, and lines
• Session 7:– Stereo vision – Epipolar geometry
• Session 8: exam
4
Session overview
1. Kalman filter
2. Robust tracking of targets
3. Tracking of point neighborhoods1. using SSD
2. using CC and NCC
3. using Gaussian receptive fields
5
Kalman filter
• The Kalman filter ist a optimal recursiv estimator. • Kalman filtering has been applied in areas such as
– aerospace,
– marine navigation,
– nuclear power plant instrumentation,
– manufactring, and many others.
• The typical problem tries to estimate position and speed from the measurements.
Tutorial:http://www.cs.unc.edu/~welch/kalman/kalmanIntro.html G. Welch, G. Bishop: An Introduction to the Kalman Filter, TR 95-041, Univ of N. Carolina, USA
6
Kalman filter
• The Kalman filter tries to estimate the state x of a discrete time controlled process that is governed by the equation
• with measurement• Process noise• Measurement noise
• Process noise covariance Q and measurement noise covariance R might in practise change over time, but are assumed constant
• Matrix A relates the state at time k to the state at previous time k-1, in absence of noise. In practise A might change over time, but is assumed constant.
• Matrix B relates the optional control input to the state x. We let it aside for the moment.
• Matrix H relates the state to the measurement.
),0(~)(
),0(~)(
111
RNvp
QNwp
vHxz
wBuAxx
kkk
kkkk
7
Notations
• Measurement
• Measurement noise covariance
• Process noise
• Process noise covariance
• Kalman gain K
Q
w
R
z
k
k
8
Kalman filter notations
A priori state estimate
A posteriori state estimate
A priori estimate error
A posteriori estimate error
A priori estimate error covariance
A posteriori estimate error covariance
}{
}{
ˆ
ˆ
ˆ
ˆ
T
kkk
T
kkk
kkk
kkk
k
k
eeEP
eeEP
xxe
xxe
x
x
9
Kalman filter
• Goal: find a posteriori state estimate ^xk as a linear combination of an a priori estimate ^xk
- and a weighted difference between the actual measurement zk and a measurement prediction H^xk
-
• The difference (zk – H^xk-) is called innovation or
residual• K is the gain or blending factor that minimizes the
a posteriori error covariance.
)ˆ(ˆˆ kkkk xHzKxx
10
Kalman gain K
• Matrix K is the gain that minimizes the a posteriori error covariance. – The equations that need to be minimized
– How to minimize}{
ˆ
)ˆ(ˆˆ
Tkkk
kkk
kkkk
eeEP
xxe
xHzKxx
Kfor solve ,0)(
}))ˆ(ˆ))(ˆ(ˆ{(}{
)ˆ(ˆˆ
k
Tkkkkkkkk
Tkkk
kkkkkkk
PtracedK
d
xHzKxxxHzKxxEeeEP
xHzKxxxxe
11
Kalman gain K
• One form of the result is
• Measurement cov small
weights residual heavier
• A priori estimate error
small weights residual little
0lim
lim
)(
0
1
0
1
kP
kR
Tk
Tkk
K
HK
RHHPHPK
k
12
Kalman gain K
• When the error covariance R approaches 0, the actual measurement zk is trusted more, while the predicted measurement ^xk
- is trusted less
• When the a priori estimate error covariance approaches 0, the actual measurement zk is trusted less, while the predicted measurement ^xk
- is trusted more.
13
Discrete Kalman filter algorithm
• The Kalman filter estimates a process by using a form of feedback control.
• The filter estimates the process state at some time and then obtains feedback in form of noisy measurements.
• The Kalman filter equation fall in 2 groups– time update equations– measurement update equations
• Time update equations project forward in time the current state and error covariance estimates to obtain a priori estimates.
• The measurement update equations implement feedback. They incorporate a new measurement into the a priori estimate to form an improved a posteriori estimate.
14
Kalman filter algorithm
Time Update(« Predict »)
Measurement update(« Correct »)
The time update projects current state estimate ahead in time.The measurement update adjusts the projected estimate by an actual measurement.
15
Kalman filter
• Time update equations (predict)
– Project state and covariance estimates forward in time
• Measurement update equations (correct)
– Compute Kalman gain K– Measure process zk– Compute a posteriori
estimate xk– Compute a posteriori error
covariance estimate Pk
QAAPP
xAxT
kk
kk
1
1ˆˆ
kkk
kkkkk
Tk
Tkk
PHKIP
xHzKxx
RHHPHPK
)(
)ˆ(ˆˆ
)( 1
Initial estimates for^xk-1 and Pk-1
16
Filter parameters and tuning
• R: measurement noise covariance can be measured a priori to the filter operation (off-line)
• Q: process noise covariance. Can not be measured, because we can not directly observe the process we are measuring. If we choose Q large enough (lots of uncertainty), a poor process model can produce acceptable results.
• Parameter tuning: We can increase filter performance by tuning the parameters R and Q. We can even use a distinct Kalman filter for tuning.
• If R and Q are constant, the estimation error cov Pk and the Kalman gain Kk will stabilize quickly and stay constant. In this case, Pk and Kk can be precomputed.
17
Session overview
1. Kalman filter
2. Robust tracking of targets
3. Tracking of point neighborhoods1. using SSD
2. using CC and NCC
3. using Gaussian receptive fields
18
Robust tracking of objects
Trigger regions
Detection New targets
List of targets
PredictList of predictions
Correct
Detection
Measurements
19
Robust tracking of objects
• Measurement• State vector
• State equation
• Source: M. Kohler: Using the Kalman Filter to track Human interactive motion, Research report No 629/Feb 1997, University of Dortmund Germany
0010
0001,
',',,
HvHxz
yxyxx
y
xz
kkk
Tk
k
20
Robust Tracking of objects
• Measurement noise error covariance
• Temporal matrix
• Process noise error covariance
• a affects the computation speed (large a increases uncertainty and therefore the search regions)
ItI
tItItaQ
t
t
A
R
k
ye
xe
k
63
3)(2
6
1000
0100
010
001
0
0
22
2,
,2
21
Form of the temporal matrix A
• Matrix A relates a posteriori state estimate ^xk-1 to the a priori state estimate ^xk
-
• The new a priori state estimate requires the temporal derivative
• According to a Taylor serie we can write
k
xxx
dk
dx kkk
kkk
ˆˆlimˆˆ
0
1000
0100
010
001
with ,
'
'
'
'
ˆˆˆ^^
k
k
A
y
x
y
x
A
y
x
y
x
rkxxx
kkk
kkkkk
1ˆˆ kk xAx
22
Kalman filter notations
A priori state estimate
A posteriori state estimate
A priori estimate error
A posteriori estimate error
A priori estimate error covariance
A posteriori estimate error covariance
}{
}{
ˆ
ˆ
ˆ
ˆ
T
kkk
T
kkk
kkk
kkk
k
k
eeEP
eeEP
xxe
xxe
x
x
23
Kalman filter
• Time update equations (predict)
– Project state and covariance estimates forward in time
• Measurement update equations (correct)
– Compute Kalman gain K– Measure process zk
– Compute a posteriori estimate xk
– Compute a posteriori error covariance estimate Pk
QAAPP
xAxT
kk
kk
1
1ˆˆ
kkk
kkkkk
Tk
Tkk
PHKIP
xHzKxx
RHHPHPK
)(
)ˆ(ˆˆ
)( 1
Initial estimates for^xk-1 and Pk-1
24
Example
• A 1D point moves with a certain speed on a continuous scale
• We have a sensor that gives only integer values
• Compute a Kalman filter for the process.
25
Example results
p 2.2 6.4 10.6 14.8 19 true position
p’ 4.2 4.2 4.2 4.2 true speed
z 2 6 11 15 19 measured pos
z’ 4 5 4 4 measured grad
^x 2 5.6 10.9 15.1 estimated pos
^x’ 0 4.2 5.3 4.2 estimated grad
K and P converge quickly
K
26
Session overview
1. Kalman filter
2. Robust tracking of targets
3. Tracking of point neighborhoods1. using SSD
2. using CC and NCC
3. using Gaussian receptive fields
27
Tracking of point neighborhoods
• When we have additive noise, the euclidean norm is the optimal method for neighborhood matching, because it minimises the error probability.
• Goal: which position (i,j) of the image I(i,j) is the most similar to the pattern X(i,j).
• Hypotheses:– additive Gaussian noise– No image rotation (2D)– No rotation in space (3D)– No scale changes
• The euclidean norm is known as SSD (« sum of squared distances »)• The method is efficient and precise, but sensible.
28
Sum of squared distances (SSD)
• Definition:– Let X(m,n) be the pattern with 0<m<M-1, 0<n<N-1
– Let I(i,j) be the image with 0<i<I-1, 0<j<J-1, (M<<I, N<<J)
• The position (i,j) of the image I(i,j) is the most similar to the pattern X(i,j) is computed as
1
0
1
0
222
2
22
)),(),((
),(),(),(M
m
N
n
NM
NM
nmXnjmiP
nmXnjmiPjiSSD
1
0
1
0
222,
)),(),((minargM
m
N
n
NM
jinmXnjmiP
29
Sum of squared distances
• Searching a pattern X(m,n) within an image I(i,j) corresponds to placing the pattern at all possible positions (i,j) and computing the SSD(i,j).
• Depending on the size of the pattern and the image, this can be costly.
• SSD is sensitive to rotations, scale and noise.
30
Pattern as a feature vector
• Any image patch can be seen as a vector.
• To transform an 2D image patch to a vector, you need to concatenate the lines one after another. For an image of size MxN, you obtain a vector with M*N dimensions.
mnMkXXnmX k with ,),(
31
SSD using feature vectors
• Transform the pattern X(m,n) and the neighborhood of size MxN at the position (i,j) of the image I to vectors.
• SSD is the norm of the difference of these two vectors.
1
0
22
))()((),(MN
m
mXmPPXjiSSD
32
Cross Correlation (CC)
• Another method for pattern matching is cross correlation (scalar product). The best match is characterised by maximising the product.
• In the case of normalised vectors, the scalar product is the cos of the angles between the vectors. This is the definition of the normalised cross correlation (NCC). -1 <NCC<1
1
0
))()((,),(MN
m
mXmPPXjiCC
X
mX
P
mPPXjiNCC
X
X
X
XX
MN
muu
MN
mm
ku
)()(,),(
)(
1
0
12
33
Relation of SSD and NCC
• The best match minimises SSD and maximises NCC.
• We note:
uuuu
uuuuuu
PXPX
PPXXPX
,221,21
2222
34
Tracking by correlation
• Computation time of tracking by correlation depends on the size of the pattern (target) and the size of the image.
• When all possible positions in the image are tested, this is slow.
• How can we optimise tracking by correlation (reduce the computation time):– Reduce the number of tests by testing only one position out of
two. Increases speed by 4, reduces precision of the result. Problem: if too little positions are tested, the target might be missed.
– Reduce the number of tests by restricting the search to a small search region (region of interest, ROI).
35
Speed up of tracking
• The search region can be determined from the position of the target at time t-1 and its maximum speed. This is measured in pixels/delta t.
• If we can reduce the search region, we can process more images (reduce delta t), which allows us to reduce the search region more, ....
• Problem: speed depends on the distance of the object to the camera. Close objects have higher speeds than objects far away.
36
Example
37
Example
• Person traverses entry hall in 5.2s (130 frames*25frames/s)
• Distance is 288 pixels, target size is 45x35 pixels• Speed 55.4pixels/s• Let maximum speed be twice the measured speed
110.8pixels/s• Then we need a search region of size target size +
• ROI = target size +/- 4.4 pixels = 54 x 44 pixels.
framepixels
sframes
spixels
4.425
8.110
38
Example
• Number of tests exhaustive search (searching whole image of size 384x288 pixels)
• (384-45)(288-35)=85767 tests• Number of tests using search region (54x44
pixels)• (54-45)(44-35)=81 tests• Speed up factor 85767/81= 1090
39
Session overview
1. Kalman filter
2. Robust tracking of targets
3. Tracking of point neighborhoods1. using SSD
2. using CC and NCC
3. using Gaussian receptive fields
40
Discussion of pattern matching using SSD and NCC
• Matching is fast (when we use small search region) and precise (when we test pixelwise positions)
• Sensitivity to rotation and scale changes.– Matching fails– Approach regular update of target: reduces
sensitivity to rotation.
41
Regular target update
For each frame1. Compute search region2. Detect best matching position3. Update the target by copying the target from the
current frame4. Update the target position
• Tracking must be fast and slow changes in rotation.
• It can also compensate for scale changes, but we do not know if we need to increase or diminish the target size.
42
Point neighborhood description using Gaussian receptive fields
• Approach that uses scale and orientation invariant description of point neighborhoods.
• Matching of point neighborhoods possible by evaluating distance in feature space.
• Computationally more expensive than SSD or NCC due to scale and orientation normalisation.
• This argument is less critical nowadays because we have sufficient computing power to do real-time point matching using receptive fields.
43
Target localisation using receptive fields
• Model target by a grid of local receptive fields
• Tracking– within ROI extract the
scale and rotation invariant receptive field response
– Search point neighborhoods that are the most similar to the model by evaluating the distance in feature space.
44
Extraction of invariant features
For all positions within the ROI1. Determine the dominant orientation for each
point neighborhood.
2. Determine the intrinsic feature scale for each point neighborhood.
3. Project the point neighborhood on the scale and orientation normalised receptive field vector. Result is the feature signature.
See session 4 for details.
45
Search matching candidates • Matching candidates are found by evaluating the distance in feature space.
• All elements within a sphere with radius from the query vector are matching candidates.
• Tree structures allow efficient nearest neigbor search.
46
Discussion of matching using receptive field vectors
• Allows matching invariant to position, orientation and size. Target can be found, even when its scale and orientation (2d) have changed.
• Computationally more expensive.
• Real-time tracking and identification possible by restriction to search region.
