Moving Object Detection and Moving Object Detection and Tracking for Intelligent Outdoor Tracking for Intelligent Outdoor
SurveillanceSurveillanceAssoc. Prof. Dr. Kanappan Palaniappan [email protected]
Dr. Filiz Bunyak [email protected]. Sumit Nath [email protected]
Department of Computer ScienceUniversity of Missouri-Columbia
Visual Surveillance and MonitoringVisual Surveillance and Monitoring• Mounting video cameras is cheap, but finding available human resources to
observe the output is expensive.According to study of US Nat’l Institute of Justice:
• A person can not pay attention to more than 4 cameras.
• After only 20 minutes of watching and evaluating monitor screens, attention of most individuals falls below acceptable levels.
• Although surveillance cameras are already prevalent in banks, stores, and parking lots, video data currently is used only "after the fact".
What is needed
• Continuous 24-hour monitoring of surveillance video to alert security officers to a burglary in progress, or to a suspicious individual loitering in the parking lot, while there is still time to prevent the crime.
Intelligent SurveillanceIntelligent Surveillance A visual surveillance system combined with visual event detection methods to analyze movements, activities and high
level events occurring in an environment.
Event recognition module detects
unusual activities, behaviors, events
based on visual clues.
Sends an alarm to operators when a suspicious activity is
detected.
Visual Event Detection ApplicationsVisual Event Detection ApplicationsSurveillance and Monitoring:• Security (parking lots, airports, subway stations, banks, lobbies etc.)• Traffic (track vehicle movements and annotate action in traffic scenarios with natural
language verbs.)• Commercial (understanding customer behavior in stores)• Long-Term Analysis (statistics gathering for infrastructure change i.e. crowding
measurement)
Broadcast Video Indexing: Sports video indexing for newscasters and coaches.
Interactive Environments: environment that responds to the activity of occupants
Robotic Collaboration: robots that can effectively navigate their environment and interact with other people and robots.
Medical:• Event based analysis of cell motility• Gait analysis, etc.
Event TypesEvent TypesReal Time AlarmsLow level alarms:
Movement detectors, long term change detectors etc.
Feature based spatial alarms: Specific object detection in monitored areas
Behavior-related alarms: Anomalous trajectories, agitated behaviors, etc.
Complex event alarms: Detection of scenarios related to multiple relational events
Long Term and Large Scale Analysis Learning activity patterns of people or vehicles in a given environment over a long period of time can be used to:– retrieve events of interest – make projections– identify security holes– control the traffic or crowd– make infrastructure decisions– monitor behavior patterns in urban
environments
Issues in High Level Video AnalysisIssues in High Level Video Analysis
1-Analysis:• Segmentation of motion blobs (background models, shadow).• Object tracking (prediction, correspondence, occlusion
resolution etc.)
2-Representation: • Video object representations (shape, color descriptors,
geometric models). • High-level event representations.
3-Access: • Efficient data structures for high-dimensional feature space. • Efficient and expressive query interface for query manipulation.
Visual Event Detection FrameworkVisual Event Detection Framework
Feature Extraction
Motion Analysis
Event Detection
ObjectClassification
-Objects-Relationships
-Events
ContextObject, Scene &Event Libraries
Events
right turn
cross road
Constraints
Controlled Environment versus Controlled Environment versus Far-view Outdoor SurveillanceFar-view Outdoor Surveillance
Controlled Environment
(i.e. indoor)
Uncontrolled Environment
(i.e. far-view outdoor)
Illumination Controlled generally static except light switch which cause a global change.
Highly dynamic especially in cloudy days.
Shadows Smooth Darker and Sharper
Object Size Large Small (difficult to learn an appearance model)
Perspective Distortion
Low High
Color saturation High Low
Background Static Dynamic (wind etc.)
Type of motion Articulated Whole-body
Our Current CapabilitiesOur Current Capabilities
Moving Object Detection Moving Object Tracking
Sudden Illumination Change Detection
Trajectory Filtering and Discontinuity Resolution
Moving Cast Shadow Detection/Elimination
Our Current CapabilitiesOur Current Capabilities1. Moving Object Detection - Using Mixture of Gaussians
method or Flux tensors
2. Moving Cast Shadow Elimination
3. Sudden Illumination Change Detection
4. Moving Object Tracking – Multi-hypothesis testing using appearance and motion
5. Trajectory filtering - Temporal consistency check, spatio-temporal cluster check
6. Discontinuity resolution - Kalman filter, appearance model (color and spatial layout)
} Combined photometric invariants
Moving Object DetectionMoving Object DetectionGoal: Segment moving regions from the rest of the image
(background).
Rationale: Provide focus of attention for later processes such as tracking, classification, event detection/recognition.
Background SubtractionBackground Subtraction• By comparing incoming frames to a reference image (background
model), regions in the incoming frame that have significantly changed are located.
Feature Extraction
PostprocessingBG/FG
ClassificationComparisonPreprocessing
BG Modeling
BG/FG masks
Frames
BG model
Preprocessing-Spatial smoothing-Temporal smoothing-Color space conversions
Features-Luminance-Color-Edge maps-Albedo (reflectance)image-Intrinsic images-Region statistics
Comparison-Differentiation -Likelihood ratioing
Postprocessing-morphological filtering-connectivity analysis-color analysis-edge analysis-shadow elimination
Classification-Thresholding-Clustering
Problems with Basic Methods
Challenging Situations in Moving Challenging Situations in Moving Object DetectionObject Detection
1. Moved objects: A background object that moved should not be considered part of the foreground forever after.
2. Gradual illumination changes alter the appearance of the background (time of day).
3. Sudden illumination changes alter the appearance of the background (cloud movements).
4. Periodic movement of the background: Background may fluctuate, requiring models which can represent disjoint sets of pixel values (waving trees).
5. Camouflage: A foreground objects' pixel characteristics similar to modeled background.
6. Bootstrapping: A training period absent of foreground objects is not always available.
7. Foreground aperture: When an homogeneously colored object moves, change in interior pixels can not be detected.
8. Sleeping person: When a foreground object becomes motionless it cannot be distinguished from a background.
9. Waking person: When an object initially in the background moves, both the object and the background appear to change.
10. Shadows: Foreground objects' cast shadows appear different than modeled background.
Background ModelBackground Model
Color history of the specified pixel Color distribution of the specified pixel
inte
nsity
Mixture of Gaussians ModelThe recent history of each pixel, X(1),...,X(t), is modeled by a mixture of K Gaussian distributions. Each distribution is characterized by its
•mean μ,•variance σ2, •weight w (indicates what portion of the previous values did get assigned to this distribution).
Performance of Mixture of Performance of Mixture of Gaussians MethodGaussians Method
1. Moved objects √ 2. Gradual illumination changes √3. Sudden illumination changes X4. Periodic movement of the
background √ X5. Camouflage X6. Bootstrapping √7. Foreground aperture √8. Sleeping person √9. Waking person √10.Shadows X
Since MoG is adaptive & multi-modal, it is robust to:
• Gradual illumination changes• Repetitive motion of the background
(such as waving trees) • Slow moving objects• Introduction and removal of scene
objects (sleeping person & waking person problems)
– when something is allowed to become part of the background, the original background color remains in the mixture until it becomes the least probable and a new color is observed.
Moving Object Detection using Moving Object Detection using Flux TensorsFlux Tensors
Input sequence obtained from OTCBVS Benchmark Dataset Collection http://www.cse.ohio-state.edu/otcbvs-bench/
Color image sequence Thermal image sequence Moving Objects Detected using Flux Tensors
merges separateobjects
creates “new” objects
static shadow
Shadow ProblemShadow Problem
Shadow Detection by Combined Photometric Shadow Detection by Combined Photometric
Invariants for Improved Foreground SegmentationInvariants for Improved Foreground Segmentation
Moving Object Detection
Identification of Darker Regions
Normalized ColorComparison
Reflectance RatioComparison
CombinationPost
Processing
Shadow Detection
New Frame
FGmask
BG model
FGmask
Shadow Mask
FGmask
Shadow Mask
Combine the MasksCombine the MasksProblems with photometric invariants:• An invariant expression may not be unique to a particular
material.• There may be singularities and instabilities for particular
values. (normalized color is not reliable around black vertex).
For a robust result:• Combine results from two invariants based on two different
properties– Normalized color : spectral properties.– Reflectance ratio: spatial properties.– At shadow boundaries, same illuminant assumption fails.
different reflectance ratios for neighbor pixels misclassification of shadow pixels as foreground dilate shadow mask.
Example: Intelligent Room SequenceExample: Intelligent Room Sequence
Input Image Frame #100 MOG Model #1
MOG Model #2 MOG Model #3 MOG Model #4
Shadow MasksShadow Masks
Normalized Color MaskReflectance Ratio Mask
Shadow Mask Post processed shadow mask
Foreground & Shadow MasksForeground & Shadow Masks
Foreground Mask Post Processed Foreground Mask
Shadow Mask Post Processed Shadow Mask
Example: Walk-in SequenceExample: Walk-in Sequence
Input Frame Walk-in #14 Model 1
Model 2 Model 3 Model 4
Shadow MasksShadow Masks
Normalized Color Masks Reflectance Ratio Mask
Shadow Mask Shadow Mask Post Processed
Foreground & Shadow Masks
Foreground Mask Foreground Mask Post Processed
Shadow Mask Shadow Mask Post Processed
Sudden Illumination ChangesSudden Illumination Changes(Cloud Movements, Light switch etc.)(Cloud Movements, Light switch etc.)
Sudden illumination changes completely alter the color characteristics of the background, thus increase the deviation of background pixels from the background model in color or intensity based subtraction.
Result: • Drastic increase in false detection (in the worst case the whole
image appears as foreground).• This makes surveillance under partially cloudy days almost
impossible.
Moving Object TrackingMoving Object Tracking
Object States
Moving ObjectDetection &
Feature Extraction
Data Association(Correspondence)
Prediction
Update
Context
Tracking
Steps:
1. Predict locations of the current set of objects of interest.
2. Match predictions to actual measurements.
3. Update object states.
Tracking Tracking (as a Dynamic State Estimator)(as a Dynamic State Estimator)
Dynamic System State EstimatorMeasurement
System
System state
Measurements State estimate
Stateuncertainties
System Error Source•Agile motion•Distraction/clutter•Occlusion•Changes in lighting•Changes in pose•Shadow
(Object or background models are often inadequate or inaccurate))
Measurement Error Source•Camera noise•Grabber noise•Compression artifacts•Perspective projection
States•Position•Appearance
•Color •Shape•Texture etc.
•Support map
System noise Measurement noise
Our Tracking MethodOur Tracking Method• Detection-based• Probabilistic
Features Used in Data Association:– Proximity– Appearance
Data Association Strategy: Multi-hypothesis testingGating Strategies: Absolute and RelativeDiscontinuity Resolution:
– Prediction (Kalman filter) – Appearance models
Filtering: – Temporal consistency check– Spatio-temporal cluster check
Trajectory FilteringTrajectory Filtering
• Some artifacts can not be totally removed by image or object level processing.
• These artifacts produce spurious segments.
Temporal Consistency CheckTemporal Consistency CheckSource of the Problem: Segments resulting from • Temporarily fragmented parts of an object• Un-eliminated cast shadows
Effect: Short segments that split from or merge to a longer segment.
Proposed Solution: Pruning short split or merge segments by temporal consistency check.
Elimination of short disconnected segments are delayed until after discontinuity resolution.
Spatio-Temporal Cluster CheckSpatio-Temporal Cluster CheckSource of the Problem:• Repetitive motion of the background (i.e. moving branches or
their cast shadows).• Spectral reflections (i.e. reflections from car windshields).
Effect: Temporally consistent and spatially clustered trajectories.
Proposed Solution:• Average Displacement to Length Ratio (ADLR)• Diagonal to Length Ratio (DLR)
Discontinuity ResolutionDiscontinuity ResolutionDiscontinuities occur especially in low resolution outdoor sequences.
Source of the problem:• Temporarily undetected objects due to
– Low contrast– Partial or total occlusions
• Incorrect pruning in data association due to significant change in appearance or size caused by– Partial occlusion– Fragmentation
Discontinuity ResolutionDiscontinuity Resolution1. Define source and sink locations
where the objects are expected to appear and disappear.
2. Identify – Segdis :Segments disappearing
unexpectedly (at a non-sink location) -> possible start of a discontinuity.
– Segapp :Segments appearing unexpectedly (at a non-source location) -> possible end of a discontinuity.
3. Identify possible matches based on time constraint.
4. Use Kalman filter to predict future positions of disappearing and past positions of appearing segments.
5. Check direction and position consistencies on
– Disappearing segment– Appearing segment– Joining segment
6. Check Color similarity.7. Multiple possible matches for a single
disappearing segment-> select appearing segment starting earliest.
8. Multiple possible matches for a single appearing segment-> select disappearing segment ending latest.
9. Match-> appearing segment inherits disappearing segment’s label and propagates this new label to its children.
Challenges in TrackingChallenges in Trackingfor Visual Event Detectionfor Visual Event Detection
• Shadows -false detections, shape distortions, merges
• Sudden illumination changes (e.g. due to cloud movements) -difficulty in object detection especially in partly cloudy days
• Glare from specular surfaces (e.g. car windshields)-spurious detections and trajectory segments
• Perspective distortion (objects far away from the camera look smaller and appear to move slower)
-difficulty in filtering false detections• Occlusion
-discontinuities in trajectories• Poor video quality (low resolution, low color saturation)
-difficulty in moving object detection-difficulty in appearance modeling
Some Experimental Results-1Some Experimental Results-1
a) All segments b) Pruned segments
c) Predictions d) After discontinuity resolution
Some Experimental Results-2Some Experimental Results-2
a) All segments b) Pruned segments
c) Predictions d) After occlusion handlingUPS
Some Experimental Results-3Some Experimental Results-3
a) All segments b) Pruned segments
c) Predictions d) After discontinuity resolution
Potential Collaborations in Potential Collaborations in Visual Event DetectionVisual Event Detection
• New moving object detection methods– Flux tensor (especially in the
presence of global motion, clutter and illumination changes)
– Weather (i.e. snow, rain, wind) • Trajectory analysis
– Trajectory validation– Feature extraction – Trajectory annotation
• Extraction of primitive events based on
– Trajectory properties– Trajectory to trajectory interactions– Agent types
• Complex event detection/recognition through temporal combination of primitive events – Hierarchical approach
• Low-level : probabilistic methods
• High-level : structural methods
• Incorporation of learning to event modeling and recognition.
• Video event mining
Questions?Questions?