Action RecognitionA general survey of previous works on

Sobhan Naderi Parizi

September 2009

List of papers

Statistical Analysis of Dynamic Actions

On Space-Time Interest Points

Unsupervised Learning of Human Action Categories Using Spatial-Temporal

Words

What, where and who? Classifying events by scene and object recognition

Recognizing Actions at a Distance

Recognizing Human Actions: A Local SVM Approach

Retrieving Actions in Movies

Learning Realistic Human Actions from Movies

Actions in Context

Selection and Context for Action Recognition

Non-parametric Distance Measure for Action Recognition

Paper info: Title:▪ Statistical Analysis of Dynamic Actions

Authors:▪ Lihi Zelnik-Manor▪ Michal Irani

TPAMI 2006 A preliminary version appeared in CVPR

2001▪ “Event-Based video Analysis”

“Statistical Analysis of Dynamic Actions”

Overview: Introduce a non-parametric distance measure Video matching (no action model): given a reference

video, similar sequences are found Dense features from multiple temporal scales (only

corresponding scales are compared) Temporal extent of videos in each category should be

the same! (a fast and slow dancing are different) New database is introduced▪ Periodic activities (walk)▪ Non-periodic activities (Punch, Kick, Duck, Tennis)▪ Temporal Textures (water)▪ www.wisdom.weizmann.ac.il/~vision/EventDetection.html

“Statistical Analysis of Dynamic Actions”

Feature description: Space-time gradient of each pixel Threshold the gradient magnitudes Normalization (ignoring appearance) Absolute value (invariant to dark/light

transitions)▪ Direction invariant

▪ 222 )()()(

|)||,||,(|),,(

lt

ly

lx

lt

ly

lxl

tly

lx

SSS

SSSNNN

“Statistical Analysis of Dynamic Actions”

Comments: Actions are represented by 3L independent 1D

distributions (L being number of temporal scales) The frames are blurred first▪ Robust to change of appearance e.g. high textured

clothing Action recognition/localization▪ For a test video sequence S and a reference sequence of

T frames:▪ Each consequent sub-sequence of length T is compared to the

reference

▪ In case of multiple reference videos:▪ Mahalanobis distance

Space-Time Interest Points (STIP)

Paper info: Title:▪ On Space-Time Interest Points

Authors:▪ Ivan Laptev: INRIA / IRISA

IJCV 2009

“On Space-Time Interest Points”

Extends Harris detector to 3D (space-time) Local space-time points with non-constant

motion: Points with accelerated motion: physical forces

Independent space and time scales Automatic scale selection

“On Space-Time Interest Points”

Automatic scale selection procedure: Detect interest points Move in the direction of optimal scale Repeat until locally optimal scale is

reached (iterative) The procedure can not be used in

real-time: Frames in future time are needed There exist estimation approaches to

solve this problem

Unsupervised Action Recognition

Paper info: Title:▪ Unsupervised Learning of Human Action

Categories Using Spatial-Temporal Words Authors:▪ Juan Carlos Niebles: University of Illinois▪ Hongcheng Wang: University of Illinois▪ Li Fei-Fei: University of Illinois

BMVC 2006

“Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words”

Generative graphical model (pLSA) STIP detector is used (piotr dollár et al.)

Laptev’s STIP detector is too sparse Dictionary of video words is created The method is unsupervised Simultaneous action

recognition/localization Evaluations on:

KTH action database Skating actions database (4 action classes)

“Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words”

Overview of the method:

K

kkijkjkjiji zwPdzPdPzdwPdwP

1

),()|()(),,(),(

w: video word d: video sequence z: latent topic (action category)

“Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words”

Feature descriptor: Brightness gradient + PCA Brightness gradient found equiv. to Optical Flow

for motion capturing

Multiple action can be localized in the video:

Average classification accuracy: KTH action database: 81.5% Skating dataset: 80.67%

K

l jlli

jkkijik

dzPzwP

dzPzwPdwzP

1)|()|(

)|()|(),|(

Event recognition in sport images

Paper info: Title:▪ What, where and who? Classifying events by

scene and object recognition Authors:▪ Li-Jia Li: University of Illinois▪ Li Fei-Fei: Princeton University

ICCV 2007

“What, where and who? Classifying events by scene and object recognition”

Goal of the paper: Event classification in still images Scene labeling Object labeling

Approach: Generative graphical model Assumes that objects and scenes are

independent given the event category Ignores spatial relationships between objects

“What, where and who? Classifying events by scene and object recognition”

Information channels: Scene context (holistic representation) Object appearance Geometrical layout (sky at infinity/vertical

structure/ground plane)

Feature extraction: 12x12 patches obtained by grid sampling (10x10) For each patch:▪ SIFT feature (used both for scene and object models)▪ Layout label (used only for object model)

“What, where and who? Classifying events by scene and object recognition”

The graphical model E: event S: scene O: object X: scene feature A: appearance feature G: geometry layout

“What, where and who? Classifying events by scene and object recognition”

A new database is compiled: 8 sport even categories (downloaded from

web) Bocce, croquet, polo, rowing,

snowboarding, badminton, sailing, rock climbing

Average classification accuracy over all 8 event classes = 74.3%

“What, where and who? Classifying events by scene and object recognition”

Sample results:

Action recognition in medium resolution regimes

Paper info: Title:▪ Recognizing Actions at a Distance

Authors:▪ Alexei A. Efros: UC Berkeley▪ Alexander C. Berg: UC Berkeley▪ Greg Mori: UC Berkeley▪ Jitendra Malik: UC Berkeley

ICCV 2003

“Recognizing Actions at a Distance”

Overall review: Actions in medium resolution (30 pix tall) Proposing a new motion descriptor KNN for classification Consistent tracking bounding

box of the actor is required Action recognition is done only

on the tracking bounding box Motion in terms of as relative

movement of body parts No info. about movements is given by the tracker

“Recognizing Actions at a Distance”

Motion Feature: For each frame, a local temporal

neighborhood is considered Optical flow is extracted (other alternatives:

image pixel values, temporal gradients) OF is noisy: ▪ half-wave rectifying + blurring

To preserve motion info:▪ OF vector is decomposed to its

vertical/horizontal components

“Recognizing Actions at a Distance”

Similarity measure: i,j: index of frame T: temporal extent I: spatial extent A: 1st video sequence = B: 2nd video sequence =

Tt c Iyx

tjc

tic yxbyxajiS

4

1 ,

),(),(),(

},,,{ 4321iiii aaaa

},,,{ 4321iiii bbbb

“Recognizing Actions at a Distance”

New Dataset: Ballet (stationary camera):▪ 16 action classes▪ 2 men + 2 women▪ Easy dataset (controlled environment)

Tennis (real action, stationary camera):▪ 6 action classes (stand, swing, move-left, …)▪ different days/location/camera position▪ 2 players (man + woman)

Football (real action, moving camera):▪ 8 action classes (run-left 45˚, run-left, walk-left, …)▪ Zoom in/out

“Recognizing Actions at a Distance”

Average classification accuracy: Ballet: 87.44% (5NN) Tennis: 64.33% (5NN) Football: 65.38% (1NN)

What can be done?

“Recognizing Actions at a Distance”

Applications: Do as I Do:▪ Replace actors in videos

Do as I Say:▪ Develop real-world motions in computer

games 2D/3D skeleton transfer Figure Correction:▪ Remove occlusion/clutter in movies

KTH Action Dataset

Paper info: Title:▪ Recognizing Human Actions: A Local SVM

Approach Authors:▪ Christian Schuldt: KTH university▪ Ivan Laptev: KTH university

ICPR 2004

“Recognizing Human Actions: A Local SVM Approach”

New dataset (KTH action database): 2391 video sequences 6 action classes (Walking, Jogging, Running,

Handclapping, Boxing, Hand-waving) 25 persons Static camera 4 scenarios:▪ Outdoors (s1)▪ Outdoors + scale variation (s2): the hardest scenario▪ Outdoors + cloth variation (s3)▪ Indoors (s4)

“Recognizing Human Actions: A Local SVM Approach”

Features: Sparse (STIP detector) Spatio-temporal jets of order 4

Different feature representations: Raw jet feature descriptors Exponential kernel on the histogram of jets Spatial HoG with temporal pyramid

Different classifiers: SVM NN

2

“Recognizing Human Actions: A Local SVM Approach”

Experimental results: Local Feature (jets) + SVM performs the

best SVM outperforms NN HistLF (histogram of jets) is slightly better

than HistSTG (histogram of spatio-temporal gradients)

Average classification accuracy on all scenarios = 71.72%

Action Recognition in Real Scenarios

Paper info: Title:▪ Retrieving Actions in Movies

Authors:▪ Ivan Laptev: INRIA / IRISA▪ Patrik Perez: INRIA / IRISA

ICCV 2007

“Retrieving Actions in Movies”

A new action database from real movies Experiments only on Drinking action vs.

random/Smoking Main contributions:

Recognizing unrestricted real actions Key-frame priming

Configuration of experiments: Action recognition (on pre-segmented seq.) Comparing different features Action detection (using key-frame priming)

“Retrieving Actions in Movies”

Real movie action database: 105 drinking actions 141 smoking actions Different scenes/people/views www.irisa.fr/vista/Equipe/People/Laptev/actiondetection.html

Action representation: R = (P, ΔP) P = (X, Y, T): space-time coordinates ΔP = (ΔX, ΔY, ΔT):▪ ΔX: 1.6 width of head bounding box▪ ΔY: 1.3 height of head bounding box

“Retrieving Actions in Movies”

Learning scheme: Discrete AdaBoost + FLD (Fisher Linear Discriminant) All action cuboids are normalized

to 14x14x8 cells of 5x5x5 pixels(needed for boosting)

Slightly temporal-randomized sequences is added to training

HoG(4bins)/OF(5bins) is used Local features:▪ Θ=(x,y,t, δx, δy, δt, β, Ψ)▪ Β Є{plain, temp-2, spat-4}▪ ΨЄ{OF5, Grad4}

“Retrieving Actions in Movies”

HoG captures shape, OF captures motion Informative motions: start & end of action Key-frame:

When hand reaches head Boosted-Histogram on HOG No motion info

around key-frame Integration of

motion & key-frameshould help

“Retrieving Actions in Movies”

Experiments: OF/OF+HoG/STIP+NN/only key-frame OF/OF+HoG works best on hard test (drinking vs.

smoking) Extension of OF5 to OFGrad9 does not help!

Key-frame priming: #FPs decreases significantly (different info.

channels) Significant overall accuracy:▪ It’s better to model motion and appearance separately

Speed of key-primed version: 3 seconds per frame

“Retrieving Actions in Movies”

Possible extensions: Extend the experiments to more action

classes Make it real-time

Automatic Video Annotation

Paper info: Title:▪ Learning Realistic Human Actions from Movies

Authors:▪ Ivan Laptev: INRIA / IRISA▪ Marcin Marszalek: INRIA / LEAR▪ Cordelia Schmid: INRIA / LEAR▪ Benjamin Rozenfeld: Bar-Ilan university

CVPR 2008

“Learning Realistic Human Actions from Movies”

Overview: Automatic movie annotation:▪ Alignment of movie scripts▪ Text classification

Classification of real action Providing a new dataset Beat state-of-the-art results on KTH

dataset Extending spatial pyramid to space-time

pyramid

“Learning Realistic Human Actions from Movies”

Movie script: Publicly available textual description about:▪ Scene description▪ Characters▪ Transcribed dialogs▪ Actions (descriptive)

Limitations:▪ No exact timing alignment▪ No guarantee for correspondence with real actions▪ Actions are expressed literally (diverse descriptions)▪ Actions may be missed due to lack of conversation

“Learning Realistic Human Actions from Movies”

Automatic annotation: Subtitles include exact time alignment Timing of scripts is matched by subtitles Textual description of action is done by a text classifier

New dataset: 8 action classes (AnswerPhone, GetOutCar, SitUp, …) Two training sets (automatically/manually annotated) 60% of the automatic training set is correctly

annotated http://www.irisa.fr/vista/actions

“Learning Realistic Human Actions from Movies”

Action classification approach: BoF framework (k=4000) Space-time pyramids▪ 6 spatial grids: {1x1, 2x2, 3x3, 1x3, 3x1,

o2x2}▪ 4 temporal grids: {t1, t2, t3, ot2}

STIP with multiple scales HoG and HoF

“Learning Realistic Human Actions from Movies”

Feature extraction: A volume of (2kσ x 2kσ x 2kτ) is taken

around each STIP where σ/τ is spatial/temporal extent (k=9)

The volume is divided to grid

HoG and HoF for each grid cell is calculated and concatenated together

These concatenated features are concatenated once more according to the pattern of spatio-temporal pyramid

233 tyx nnn

“Learning Realistic Human Actions from Movies”

Different channels: Each spatio-temporal template: one channel Greedy search to find the best channel combination Kernel function = Chi2 distance

Observations: HoG performs better than HoF No temporal subdivision is preferred (temporal grid = t1) Combination of channels improves classification in real scenario Mean AP on KTH action database = 91.8% Mean AP on real movies database:▪ Trained on manually annotated dataset : 39.5%▪ Trained on automatically annotated dataset : 22.9%▪ Random classifier (chance) : 12.5%

C

channel channelKernelDist1

“Learning Realistic Human Actions from Movies”

Future works: Increase robustness to annotation noise Improve script to video alignment Learn on larger database of automatic annotation Experiment more low-level features Move from BoF to detector based methods The table shows:▪ effect of temporal division when combining channels (HMM based methods

should work)▪ Pattern of spatio-temporal pyramid changes so that context is best captured

when the action is scene-dependent

Image Context in Action Recognition

Paper info: Title:▪ Actions in Context

Authors:▪ Marcin Marszalek: INRIA / LEAR▪ Ivan Laptev: INRIA / IRISA▪ Cordelia Schmid: INRIA / LEAR

CVPR 2009

“Actions in Context”

Contributions: Automatic learning of scene classes from video Improve action recognition using image

context and vice versa Movie scripts is used for automatic training For both action and scene: BoF + SVM New large database:

12 action classes 69 movies involved 10 scene classes www.irisa.fr/vista/actions/hollywood2

“Actions in Context”

For automatic annotation, scenes are identified only from text

Features: SIFT (modeling scene)

on 2D-Harris HoG and HoF (motion)

on 3D-Harris (STIP)

“Actions in Context”

Features: SIFT: extracted from 2D-Harris detector▪ Captaures static appearance▪ Used for modeling scene context▪ Calculated for single frame (every 2 seconds)

HoG/HoF: extracted from 3D-Harris detector▪ HoG captures dynamic appearance▪ HoF captures motion pattern

One video dictionary per channel is created Histogram of video words is created for each channel

Classifier: SVM using chi2 distance Exponential kernel (RBF) Sum over multiple channels

)),(1

exp(),( jichannelchannel channel

ji xxDxxK

“Actions in Context”

Evaluations: SIFT: better for context HoG/HoF: better for action Only context can also classify

actions fairly good! Combination of the 3 channels

works best

“Actions in Context”

Observations: Context is not always good▪ Idea: The model should control

contribution of context for each action class individually

Overall, the gain of accuracyis not significant using context:▪ Idea: other types of context should

work better

Object Co-occurrence in Action Recognition

Paper info: Title:▪ Selection and Context for Action Recognition

Authors:▪ Dong Han: University of Bonn▪ Liefeng Bo: TTI-Chicago▪ Cristian Sminchisescu: University of Bonn

ICCV 2009

“Selection and Context for Action Recognition”

Main contributions: Contextual scene descriptors based on:▪ Presence/absence of objects (bag-of-detectors)▪ Structural relation between objects and their parts

Automatic learning of multiple features▪ Multiple Kernel Gaussian Process Classifier (MKGPC)

Experimental results on: KTH action dataset Hollywood1,2 Human Action database (INRIA)

“Selection and Context for Action Recognition”

Main message: Detection of a Car and a Person in its proximity increases

probability of Get-Out-Car action

Provides a framework to train a classifier based on combination of multiple features (not necessarily relevant) e.g. HOG+HOF+histogram intersection, …

Similar to MKL but here Parameters are learnt automatically i.e. (weights + hyper-

parameters)

Gaussian Process scheme is used for learning

T

t

ttj

tijim xxkexxk

t

1

);,(),;,(

“Selection and Context for Action Recognition”

Descriptors: Bag of Detectors▪ Deformable part models are used (Pedro)▪ Once object BBs are detected, 3 descriptors are built:▪ ObjPres (4D)▪ ObjCount (4D)▪ ObjDist (21D): pair-wise distances of object parts for all of

Person detector (7 parts)

HOG (4D) + HOF (5D) from STIP detector (Ivan)▪ Spatial grids: 1x1, 2x1, 3x1, 4x1, 2x2, 3x3▪ Temporal grids: t1, t2, t3

3D gradient features

“Selection and Context for Action Recognition”

Experimental results: KTH dataset▪ 94.1% mean AP vs. 91.8% reported by Laptev▪ Superior to state-of-the-art in all but Running class

HOHA1 dataset▪ Trained on clean set only▪ The optimal subset of features is found greedily

(addition/removal) based on test error▪ 47.5% mean AP vs. 38.4% reported by Laptev

HOHA2 dataset▪ 43.12% mean AP vs. 35.1% reported by Marszalek

“Selection and Context for Action Recognition”

Best feature combination