Namrata Vaswani, Univ. of Maryland, College Park Change Detection in Shape Dynamical Models & Application to Activity Recognition Namrata Vaswani Dept

Namrata Vaswani, Univ. of Maryland, College Park

Change Detection in Shape Dynamical Models & Application

to Activity Recognition

Namrata VaswaniDept. of Electrical & Computer Engineering

University of Maryland, College Parkhttp://www.cfar.umd.edu/~namrata


Overview

• The Group Activity Recognition Problem

• Slow and Drastic Change Detection

• Landmark Shape Dynamical Models

• Experiments and Results


The Group Activity Recognition Problem


Problem Formulation• The Problem:

– Model activities performed by a group of moving and interacting objects (which can be people or vehicles or robots or diff. parts of human body). Use the models for abnormal activity detection and tracking

• Our Approach: – Treat objects as point objects: “landmarks”.– Changing configuration of objects: deforming shape– ‘Abnormality’: change from learnt shape dynamics

• Related Approaches for Group Activity:– Co-occurrence statistics, Dynamic Bayes Nets


Bayesian Approach

• Define a Stochastic State-Space Model (a continuous state HMM) for shape deformations in a given activity, with shape & motion forming the hidden state vector and configuration of objects forming the observation.

• Use a particle filter to track a given observation sequence, i.e. estimate the hidden state given observations.

• Define Abnormality as a slow or drastic change in the shape dynamics with unknown change parameters. We propose statistics for slow & drastic change detection.


Human Action Tracking

Cyan: ObservedGreen: Ground Truth Red: SSA Blue: NSSA


Slow and Drastic Change Detection in Continuous State

HMMs


The Problem• General Hidden Markov Model (HMM): State seq. {Xt},

Observation seq. {Yt}

• Finite duration change in system model which causes a permanent change in probability distribution of state

• Change parameters unknown: Log LRT(Xt) LL(Xt)

• Noisy Observations: LL(Xt) E[LL(Xt)|Y1:t)] = ELL

• Nonlinear dynamics: Particle filtered estimate

• Slow or drastic change: ELL for slow, OL/TE for drastic

indep. }{ },{ ,)( ,)( 1 tttttttt nwnXfXwXhY


Related Work• Change Detection in Nonlinear Systems using PF

– Known change parameters, sudden change• Log-LRT of current observation given past observations• Multimode system detect change in mode

– Unknown change parameters, sudden change• Generalized LRT• Tracking Error (TE)• Neg. Log likelihood of current observation given past (OL)

• Avg. Log Likelihood of i.i.d. observations used often– But ELL = E[LL(Xt)|Y1:t] (MMSE of LL given observations) in

context of HMMs is new


Particle Filtering

2 step toGo 1,tSet t 4.

)},~({~ , :(b)

~

~ ,~ :(a)

~ ,

,

,Pr Evaluate

11

|

1

1~|

11

~1|

0001

0|0

0

(Update) Step Selection 3.

n)(Predictio Step Sampling Importance2.

tionInitializa 1. :Aim

0

ni

(i)t

(i)t

(i)t

n

ix

ntt

n

i

(i)ttt

(i)ttt(i)

t

n

ix

(i)t

ntt

(i)t-t

(i)t

n

ix

ntt

|(i)

n

ix

n

:ttnt

wxlMultinomiax(dx)δ(dx)π

)x|(Y

)x|(Yw(dx)δw(dx)π

)(dx|x~Qx(dx)δ(dx)π

(dx)~πx(dx)δ(dx)π

t)dx|Y(X(dx)π

(i)t

(i)t

(i)t

(i)


Change Detection Statistics• Drastic Change:

– Tracking Error (TE). If Gaussian noise, TE ≈ OL– Neg. of Current Observation Likelihood given past (OL)

OL = -log [Pr(Yt|Y0:t-1,H0) ] = -log[<Qt t-1 , ψt>]

• Slow Change: Propose Expected Log Likelihood (ELL)– ELL = Kerridge Inaccuracy b/w πt and pt

0

ELL(Y1:t )=E[-log pt0

(Xt)|Y1:t]=Eπ[-log pt0

(Xt)]=K(πt: pt0)

• “Detectable changes” using ELL:– E[ELL(Y1:t

0)] = K(pt0:pt

0)=H(pt0), E[ELL(Y1:t

c)]= K(ptc:pt

0)– Chebyshev Ineq: With false alarm & miss probabilities of 1/9,

ELL detects all changes s.t. K(pt

c:pt0) -H(pt

0)>3 [√Var{ELL(Y1:tc)} +√Var{ELL(Y1:t

0)}]


OL & ELL: Slow & Drastic Change• Problem of TE or OL: Fail to detect slow changes

– Particle Filter tracks slow changes “correctly” – Assuming change till t-1 was tracked correctly (error in posterior

small), OL only uses change introduced at t– ELL works because it uses total change in posterior till time t &

since PF tracks the posterior “correctly” for a slow change, ELL is approximated “correctly”

• Problem of ELL: Fails to detect drastic changes– Approximating posterior for changed system using a PF for

unchanged system: error large for drastic changes– OL relies on the error introduced due to change to detect it, so

works for drastic changes

• ELL detects change before loss of track, OL/TE: after


A Simulated Example• Change introduced in system model from t=5 to t=15

ELL Tracking Error (or OL)


Practical Issues

• Defining pt0(x):

– Either use part of state vector which has linear Gaussian dynamics: can define pt(x), in closed form.

– Assume a parametric family for pt(x), learn parameters using training data (assume pt(x) piecewise constant over time)

• Declare a change when either ELL or OL/TE exceed their respective thresholds.– Set ELL threshold to H(pt

0) +3√Var{ELL(Y1:t0)}

– Set OL threshold to a little above E[OL0,0]=H(Yt|Y1:t-1)

• Single frame estimates of ELL or OL/TE may be noisy– Average the statistic or average no. of detects or modify CUSUM


Approximation Errors

• Total error < Bounding error + Model error + PF error– Bounding error: Stability results hold only for bounded fns but

LL is unbounded. BE=|ELLtc,c-ELLt

c,c,M|

– Model error: Error b/w exact filtering with original system model & with changed model, ME=|ELLt

c,c,M-ELLtc,0,M|

– PF Error: Error b/w exact filtering with changed model & particle filtering with changed model, PE=|ELLt

c,0,M-ELLtc,0,M,N|

}]),(log[min{E )],(log[E 00,

t

,

t

MXpELLXpELL tc,c,Mtt

c,ct cccc

}]),(log[min{E 000,

t

MXpELL t,Mc,

t c

}]),(log[min{E 0,0,0,

t

MXpELL tN,Mc,

t Nc


Asymptotic Stability, Stability with t

• The error in ELL estimation averaged over observation sequences & PF realizations is asymptotically stable if– Change lasts for a finite time– “Unnormalized filter kernels are uniformly “mixing”– Certain boundedness assumptions hold

• Stability & monotonic decrease of error if the kernels are only “mixing”

• Analysis generalizes to errors in MMSE estimate of any function of state evaluated using a PF with system model error


“Unnormalized filter kernel” “mixing”• “Unnormalized filter kernel”, Rt, is state transition kernel,Qt,

weighted by observation likelihood given state

• “Mixing”: measures the rate at which the transition kernel “forgets” its initial condition or eqvtly. how quickly the state sequence becomes ergodic. Mathematically,

• Example: State transition Xt =Xt-1+nt + a : not mixing. But if Yt=h(Xt)+wt, wt is truncated noise, then Rt is mixing

)'|()'( ),',()'()',( ,, xYgxdxxQxdxxR tYttYtt tt

xx E,ExAAxKA

K

A subsets Borel ),(),()(

s.t. , measure nonneg. a and 0 if mixing is Kernel1


Complementary Behavior of ELL/OL

• We have shown that etc,0=|ELLt

c,c- ELLtc,0,N| is upper

bounded by an increasing function of OLkc,0, tc<k<t

• Implication:

• Assume “detectable” change i.e. ELLc,c large

– OL fails => OLkc,0,tc<k<t small => ELL error, et

c,0 small=> ELLc,0 large =>ELL detects

– ELL fails => ELLc,0 small =>ELL error, etc,0 large

=> at least one of OLkc,0,tc<k<t large => OL

detects

),()exp( 1,

0,0,k

c

kQck

t

tk

ct DOLe


“Rate of Change” Bound• The total error in ELL estimation is upper bounded by

increasing functions of the “rate of change” (or “system model error per time step”) with all increasing derivatives.

• OLc,0 is upper bounded by increasing function of “rate of change”.

• Metric for “rate of change” (or equivalently “system model error per time step”) for a given observation Yt : DQ,t is

'|)',()',(|)'(sup),( '0

,0 dxxxqxxqxQQD Ex t

ctYtxt

ctQ t


Applications/ Possible Applications• Surveillance: abnormal activity detection

• Medical applications: Detect motion disorders by modeling normal human actions using “Shape Activity” models

• ELL + “PSSA model” for activity segmentation

• Neural signal processing: detecting changes in stimuli

• Congestion Detection

• System model change detection in target tracking problems without the tracker loses track


Landmark Shape Dynamical Models


What is Shape?

• Shape is the geometric information that remains when location, scale and rotation effects are filtered out [Kendall]

• Shape of k landmarks in 2D– Represent the X & Y coordinates of the k points as

a k-dimensional complex vector: Configuration– Translation Normalization: Centered Configuration– Scale Normalization: Pre-shape– Rotation Normalization: Shape


“Activities” on the Shape Sphere in Ck-1

Abnormal Activity

Tangent space(hyper-plane)

μ

Normal Activity

Observation vector: k-dim complex vector representing locations of k point objects at time t

Shape space


Related Work

• Related Approaches for Group Activity– Co-occurrence Statistics– Dynamic Bayesian Networks

• Shape Analysis/Deformation:– PDMs,Thin plate splines, Principal & Partial warps– Active Shape Models: affine deformation in configuration space– ‘Deformotion’: Euclidean motion of avg. shape + deformation – Piecewise geodesic models for tracking on Grassmann manifolds

• Particle Filters for Multiple Moving Objects: – JPDAF (Joint Probability Data Association Filter): difficult to define

complicated interactions b/w objects


Motivation• Obtain a generic & sensor invariant approach for “activities”

performed by multiple moving objects. Easy to fuse sensors

• Why “shape”: invariant to translation, zoom & in-plane rotation of camera

• Single global framework for modeling motion & interactions, co-occurrence statistics requires individual & joint histograms.

• New framework to track a group of interacting & moving objects: know that the group is “constrained” to move in a certain fashion defined by the “activity”.

• Active Shape Models good for approximately rigid objects (small nonrigidity introduced by camera motion)


The HMM• Observation, Yt: Centered configurations • State, Xt=[γt, ct, st, θt ]

– Current Shape (γt), – Shape “Velocity” (ct): Tangent coordinate w.r.t. γt

– Scale (st), – Rotation angle (θt)

• Use complex vector notation to simplify equations

• Use a particle filter to approximate the optimal non-linear filter, t(dx) = Pr(Xtєdx|Y0:t) = posterior state distribution conditioned on observations upto time t, by an N-particle empirical estimate of t


State Dynamics

Shape Dynamics:• Define shape “velocity” at time t in the tangent space

w.r.t. the current shape, γt

• Tangent space is a vector space: define a linear Gaussian-Markov model for shape “speed”, ct

• “Move” γt by an amount ct on shape manifold to get γt+1

Motion Dynamics:

• Linear Gauss-Markov dynamics for log st, unwrapped θt

),(~ , ntttct NnncAc 01

tttttttttt zvvvzcUv ,)*( ,)( / 1211


HMM Equations

tttttjs

tttt

obstttt

vvvzevzXh

outliersNwwXhY

tt

211

0/log )*(,),()(

),(~ ,)(

Observation Model: Map [Shape,Motion]Centered Config.

•Linear Gauss-Markov models for log st and θt

•Can be stationary or non-stationary

System Model : Shape and Motion Dynamics

tv

ttv

tv

tz

Ctt

Ibasist

Ut

ct

Ut

vn

Nt

nt

nt

cAt

ct

zt

c

211

01

1

/)*(

)*]([)( ,)(

),(~ ,

Shape Dynamics: Motion Dynamics:


Special Cases

• Stationary Shape Activity (SSA): γt = μ, constant• Models shape variation in a single tangent space

w.r.t mean shape. • Track normal behavior, detect abnormality

• Non-Stationary Shape Activity: γt changes for all t• Tangent space changes at every time instant • Most flexible: Detect abnormality and also track it

• Piecewise Stationary Shape Activity: γt p.w. constant• Change time can be fixed or decided on the fly

using ELL • PSSA + ELL: Activity Segmentation


Stationary Shape Activity

• Mean shape is constant, so set γt= μ (Procrustes mean), for all t, γt not part of state vector, learn mean shape using training data.

• Define a single tangent space w.r.t. μ : shape dynamics simplifies to linear Gauss-Markov model in tangent space

• Since shape space is not a vector space, “data mean” may not lie in shape space, evaluate “Procrustes mean”: an intrinsic mean on the shape manifold.


What is Procustes Mean?

• “Proc mean” μ, minimizes sum of squares of “Proc distances” of the set of pre-shapes from itself

• “Proc distance” is Euclidean distance between “Proc fit” of one pre-shape onto another

• “Proc fit”: scale or rotate a pre-shape to “optimally” align it with another pre-shape

• “Optimally” : minimum Euclidean distance between the two pre-shapes after alignment


Abnormal Activity Detection

• Define abnormal activity as – Slow or drastic change in shape statistics with

change parameters unknown.– System is a nonlinear HMM, tracked using a PF

• This motivated research on slow & drastic change detection in general HMMs– Tracking Error detects drastic changes. We

proposed a statistic called ELL for slow change.– Use a combination of ELL & Tracking Error and

declare change if either exceeds its threshold.


Applications/Possible Applications

• Modeling group activity to detect suspicious behavior – Airport example– Lane change detection in traffic

• Model human actions: track a given sequence of actions, detect abnormal actions (medical application to detect motion disorders)

• Activity sequence segmentation: unsupervised training

• Sensor independent: IR/Radar/Seismic

• Robotics, Medical Image Processing


Experiments and Results


Experiments• Group Activity:

– Normal activity: Group of people deplaning & walking towards airport terminal: used SSA model

– Abnormality: A person walks away in an un-allowed direction: distorts the normal shape

• Simulated walking speeds of 1,2,4,16,32 pixels per time step (slow to drastic distortion in shape)

• Compared detection delays using TE and ELL• Plotted ROC curves to compare performance

• Human actions: – Defined NSSA model for tracking a figure skater– Abnormality: abnormal motion of one body part– Able to detect as well as track slow abnormality


Normal/Abnormal ActivityNormal Activity Abnormal Activity


Abnormality• Abnormality introduced at t=5• Observation noise variance = 9• OL plot very similar to TE plot (both same to first order)

ELL Tracking Error (TE)


ROC: ELL• Plot of Detection delay against Mean time b/w False Alarms

(MTBFA) for varying detection thresholds• Plots for increasing observation noise

Slow Change: ELL Works Drastic Change: ELL Fails


ROC: Tracking Error(TE)• ELL: Detection delay = 7 for slow change , Detection delay = 60

for drastic• TE: Detection delay = 29 for slow change, Detection delay = 4

for drastic

Slow Change: TE Fails Drastic Change: TE Works


ROC: Combined ELL-TE• Plots for observation noise variance = 81 (maximum)• Detection Delay < 8 achieved for all rates of change

Slow Change: Works! Drastic Change: Works!


Human Action Tracking

Cyan: ObservedGreen: Ground TruthRed: SSA Blue: NSSA


NSSA Tracks and Detects Abnormality

Red: SSA, Blue: NSSA

ELL Tracking Error


Contributions

• Slow and drastic change detection in general HMMs using particle filters. We have shown– Asymptotic stability / stability of errors in ELL approximation– Complementary behavior of ELL & OL for slow & drastic changes– Upper bound on ELL error is an increasing function of “rate” of

change, with all increasing derivatives

• Stochastic state space models (HMMs) for simultaneously moving and deforming shapes.– Stationary, non-stationary & p.w. stationary cases– Group activity & human actions modeling, detecting abnormality– NSSA for tracking slow abnormality, ELL for detecting it– PSSA + ELL : Apply to activity segmentation


Other Contributions

• A linear subspace algorithm for pattern classification motivated by PCA– Approximates the optimal Bayes classifier for Gaussian pdfs with

unequal covariance matrices.– Useful for “apples from oranges” type problems.– Derived tight upper bound on its classification error probability – Compared performance with Subspace LDA both analytically &

experimentally– Applied to object recognition, face recognition under large pose

variation, action retrieval.

• Fast algorithms for infra-red image compression


Ongoing and Future Work

• Change Detection– Bound on Errors is increasing fn. of rate of change:

Implications– CUSUM algorithm, applications to other problems

• Non-Stationary & Piecewise Stationary Shape Activities– Application to sequences of different kinds of actions– PSSA + ELL for activity segmentation

• Time varying number of Landmarks?– What is “best” strategy to get a fixed no. ‘k’ of landmarks?– Can we deal with changing dimension of shape space?

• Sequence of Activities, Multiple Simultaneous Activities • Multi-Sensor Fusion, 3D Shape, General shape spaces

Documents

Namrata Vaswani, Univ. of Maryland, College Park Change Detection in Shape Dynamical Models & Application to Activity Recognition Namrata Vaswani Dept