View
215
Download
0
Category
Preview:
Citation preview
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 1/20
Regression based Learning ofHuman Actions from Videousing HOF-LBP Flow patterns
Binu M Nair, Vijayan K Asari
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 2/20
Motivation and Objectives
• Motivation: To recognize a human action from a surveillance video feed at long
distance.
• Objectives: To develop a human action recognition framework
– Which is invariant to sequence length normalization
–
Can classify human actions from 10-15 frames (for real time operation) – To account for variation in speed of an action
• Different people wave with different speeds
– To be invariant to initialization of the starting/ending points of an action cycle
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 3/20
Overview of proposed algorithm• Define and extract suitable motion descriptors based on the optical flow at each
frame
• Using the extracted motion descriptors, define action manifolds for each class.
– Contains variations of motion with respect to the sequence
• Learn a neural network to characterize each action manifold.
•Classify the test sequence using the learned neural networks.
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 4/20
Proposed Methodology
1. Motion Representation using Histogram of Oriented Flow and Local Binary Flow patterns(HOF-
LBP).
– Motion descriptor computed from optical flow for each frame of the video sequence
2. Computation of Reduced Posture Space using PCA
– Computing an action manifold for each action class using Principal Component Analysis
3. Modeling of Action Manifolds using Generalized Regression Neural Networks
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 5/20
Motion Representation using HOF-LBPFlow Patterns
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 6/20
Motion Representation using Histogram ofFlow Patterns
• Gives information about the extent of motion on a local scale and the direction of
motion
• Algorithm
– Compute Optical Flow < , > between consecutive frames at location (,) – Compute the magnitude and direction images from optical flow.
– Divide them into blocks
• At each block, histogram of flow is computed
• Histogram of flow: weighted histogram of the flow direction with the weights being the
corresponding magnitude.
• Concatenate across blocks to get the HOF descriptor
• These are local distributions which change during the course of an action
sequence.
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 7/20
Motion Representation using Local Binary FlowPatterns
• To extract relationship between the flow vectors in different regions of the body
• This “textural” context can be extracted by using the Local Binary Pattern
encoding on optical flow magnitude and direction.
, =
2( − )• A sampling grid of (P,R) = (16,2) where P refers to the number of neighbors and R
refers to the radius of the neighborhood.
•
The concatenation of HOF and LBP constitutes the action feature set
5 6 743
0 0 00
1
0 1 0
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 8/20
Feature Extraction - Optical Flow
HOF (5,5)
LBP(16,2)
+ Action Feature
LBP(16,2)
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 9/20
Computation of Reduced Posture Space
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 10/20
Computation of Reduced Posture Space usingPCA• Aim is to perform regression analysis on the set of action features
– Action features will be considered as the regressors/input variables to a
regression function.
– Selection of the response/output variable should
• Bring out the variations in the regressors w.r.t to time
• Be invariant to the time : selecting time will not be the solution
• A multivariate time-series set of (regressors,responses) for each action class would
correspond to an action manifold(Reduced posture space).
• The frames of an action sequence is then considered as points on a particular manifold.
– One method to treat a multi-variate time series data
• Prinicipal Component Analysis or Empirical Orthogonal Function Analysis
• Time series data is represented as a linear combination of time-independent orthogonal
basis functions(Eigen vectors) with time varying amplitude(Eigen coefficients).
dim 1
dim 2
Frame 1
Frame k
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 11/20
Computation of Reduced Posture Space usingPCA for action class • EOF Analysis
– Let … . ∈ and is observed at , , 3… . , then
=
. () ; − ; −
– Extending this to our motion feature set
[ , … . ]of the action
class having a total of frames and ∈ ,
• We get time independent basis functions which are Eigen vectors V [ , , … . d]• We get time dependent coefficients [ , … . ] and ∈ • Establishes one-to-one correspondences between motion feature set and coefficients
XK(m)xD PCA
EigenVectors(× )
Coefficients
( × )
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 12/20
Modelling the action posture space usingGRNN
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 13/20
Modeling of Action Manifolds usingGeneralized Regression Neural Networks
• Generalized Regression Neural Networks
–Used to learn the functional mapping between and for an action class .
– Based on the radial basis function network
– Faster training scheme which is one-pass algorithm
– The number of input nodes depends on number of training samples.
– K-Mean clustering is used before training so to reduce training sample size
• { : 1 ≤ ≤ • { : 1≤≤ ( )• GRNN Model learns the mapping : →
()• The neural network models
.( − ) ( − )
d l f f ld
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 14/20
Modeling of Action Manifolds usingGeneralized Regression Neural Networks
• If there are () clusters from training pairs , ,
=() , .exp(,
2)
=(),2
; , − , − ,
Where ( , , ,) : set of clusters for action class
3
exp(,
2 )
exp((),
2)
,
1
,
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 15/20
Classification of test sequence
• Algorithm (Testing)
– Compute HOF-LBP motion feature for each frame of test sequence(partial – 15 frames or full –
60-80 frames)
– Project the test features on Eigen basis for each action class – Estimate the projections of each action by applying the feature set onto the trained GRNN
model
– Correct class ∗ argmin(projections −estimations)• The model which gives the smallest difference between the eigen space projections and the GRNN
estimations is the correct class.
R l (W i d b (10 i 9
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 16/20
Results (Weizmann database (10 actions, 9individuals)
•
Testing strategy:- Leave 9 sequences out of training• Partial Sequence :- 15 frames with overlap of 10 frames
a1 a2 a3 a4 a5 a6 a7 a8 a9 a10
a1 100a2 3 75 22
a3 100
a4 88 12
a5 93 5
a6 78 21
a7 100
a8 100
a9 1 99
a10 100
a1-bend a2-jump p a3-jjack
a4-jump f a5-run a6-side
a7-wave1 a8-skip a9-wave2
a10-walk
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 17/20
Robustness Test (Test for Deformity)With bag With dog Knees Up Limping Moonwalk
Legs
Occluded
Normal
Walk
With
Briefcase
With Pole With Skirt
Test Seq 1st Best 2nd Best Median to all
actions
Swinging a
bag
Walk 2.508 Skip 3.094 3.9390
Carrying a
briefcase
Walk 1.866 Skip 2.170 3.6418
Walking with
a dog
Walk 1.806 Skip 2.338 3.8249
Knees Up Walk 2.894 Side 3.270 4.0910
Limping Man Walk 2.224 Skip 2.922 3.8217
Sleepwalking Walk 1.892 Skip 2.132 3.6633
Occluded
Legs
Walk 1.883 Skip 2.594 2.6249
Normal Walk Walk 1.886 Skip 2.624 3.6338
Occluded by
a pole
Walk 2.149 Skip 2.945 3.8801
Walking in a
skirt
Walk 1.855 Skip 2.159 3.5401
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 18/20
Robustness Test (View Invariance)
Test Seq 1st Best 2nd Best Median to all actions
Dir. 0 Walk 1.7606 Skip 2.3435 3.6550Dir. 9 Walk 1.6975 Skip 2.3138 3.6286
Dir. 18 Walk 1.7342 Skip 2.2600 3.6066
Dir. 27 Walk 1.7314 Skip 2.3225 3.5359
Dir. 36 Walk 1.7721 Skip 2.3296 3.5050
Dir. 45 Walk 1.7750 Skip 2.2099 3.4217
Dir. 54 Walk 1.7796 Skip 2.1169 3.3996
Dir. 63 Walk 1.9683 Skip 2.3181 3.2095
Dir. 72 Walk 2.2900 Skip 2.4930 3.3460
Dir. 81 Side 2.6917 Side 2.8095 3.7771
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 19/20
Conclusions/Inferences
• Motion Information is used.
• Misclassifications are not spread across action classes.
– Occurs between at most two actions.
• Does not rely too much on the silhouette mask
– Only an approximate mask is required
• Can identify actions from a set of 10-15 frames
• Can be used in a higher level activity recognition system where the scores
for the primitive actions is available.
8/10/2019 SMC 2013 Presentation
http://slidepdf.com/reader/full/smc-2013-presentation 20/20
Thank You
Questions?
Recommended