Upload
neil-carroll
View
213
Download
0
Embed Size (px)
Citation preview
Visual Event Visual Event Detection & Detection & RecognitionRecognition
Filiz Bunyak Ersoy, Ph.D. Filiz Bunyak Ersoy, Ph.D. studentstudent
Smart Engineering Systems Smart Engineering Systems LabLab
Research InterestResearch Interest
Recognition of visual events from video Recognition of visual events from video sequences. sequences.
Use of visual event information for intelligent Use of visual event information for intelligent video surveillance and semantic video video surveillance and semantic video indexing & retrieval.indexing & retrieval.
Incorporation of learning to event modeling Incorporation of learning to event modeling and recognition.and recognition.Very few event detection systems that are currently available Very few event detection systems that are currently available are not flexible, work for a very limited domain for very limited are not flexible, work for a very limited domain for very limited number of predefined, mostly hand-coded events. They are not number of predefined, mostly hand-coded events. They are not designed to be extended or modified. Learning will enable designed to be extended or modified. Learning will enable adaptability and extensibility of an event detection systemadaptability and extensibility of an event detection system..
Content of Video DataContent of Video Data
Low Level Low Level
Visual Visual FeaturesFeatures
ColorColor TextureTexture ShapeShape MotionMotion Shot Shot
BoundariesBoundaries
Mid LevelMid Level
Semantic Semantic ContentContent
People/People/ObjectsObjects
LocationLocation ActionsActions TimeTime
High LevelHigh Level
Semantic Semantic ContentContent
StoryStory ConceptConcept EventEvent
ChallengesChallenges1-Representation:1-Representation: Modeling visual Modeling visual
features features Image/video object Image/video object
representation representation Description of spatio-Description of spatio-
temporal relationships temporal relationships High-level event High-level event
representationrepresentation2-Analysis: 2-Analysis: Segmentation of video Segmentation of video
objects objects Adaptive grouping of Adaptive grouping of
features & objects features & objects Compressed-domain Compressed-domain
feature extraction feature extraction
3-Indexing: 3-Indexing: Efficient indexing Efficient indexing
algorithms for high-algorithms for high-dimensional feature dimensional feature space space
Robust, scalable Robust, scalable indexing algorithms for indexing algorithms for spatio-temporal queries spatio-temporal queries
4-Summarization:4-Summarization: Automated Automated
summarization of visual summarization of visual contentcontent
Visualization of content Visualization of content at different levels at different levels
Proposed FrameworkProposed Framework
Feature Extraction
Motion Analysis
Event Inference
ObjectClassification
ObjectsRelationships
Events
ContextObject, Scene &Event Libraries
Context is any a priori informationprovided to the system.
Events
Object Information
Object Trajectories & Spatio-temporal relationships
Motion AnalysisMotion Analysis
Moving Object DetectionMoving Object Detection Temporal differencingTemporal differencing Background subtractionBackground subtraction Optical FlowOptical Flow
TrackingTracking Region-based methodsRegion-based methods Contour-based methodsContour-based methods Feature-based methodsFeature-based methods Model-based methodsModel-based methods
Moving ObjectDetection
Feature Extraction
CorrespondenceAnalysis
Prediction
Update
Context
Object States
Tracking
Single-view methods Single-view methods (single camera)(single camera)
Multi-view methods Multi-view methods (multiple cameras)(multiple cameras)
Event InferenceEvent InferencePossible Event Inference Possible Event Inference
MethodsMethods Rule based Rule based Logical FormalismsLogical Formalisms
Temporal LogicTemporal Logic Event LogicEvent Logic Fuzzy LogicFuzzy Logic
Bayesian belief networkBayesian belief network Hidden Markov modelHidden Markov model Petri-netPetri-net GrammarGrammar
Event Inference Methods
InformationAbout Objects
Spatio-temporal Relationships
A priori InformationAbout the Application, Goal, Scene, Objects. Event descriptions.
EVENTS
Applications of Event Applications of Event Detection/Recognition Detection/Recognition
Surveillance and MonitoringSurveillance and Monitoring Traffic (track vehicle Traffic (track vehicle
movements and annotate movements and annotate action in traffic scenarios.)action in traffic scenarios.)
Detection of accidents, traffic Detection of accidents, traffic violations, congestions.violations, congestions.
Gather statistics about human Gather statistics about human activities, road utilization etc.activities, road utilization etc.
Surveillance of public places / Surveillance of public places / shops / offices etc. shops / offices etc.
Detection of atypical Detection of atypical incidents, theft, vandalism, incidents, theft, vandalism, shoplifting, abandoning shoplifting, abandoning (possibly dangerous) objects.(possibly dangerous) objects.
Indexing of Broadcast Indexing of Broadcast VideoVideo Sports video indexing for Sports video indexing for
newscasters or trainers.newscasters or trainers. Semantic indexing for Semantic indexing for
automated annotation for automated annotation for content retrieval.content retrieval.
Interactive environments:Interactive environments: environment that respond to environment that respond to the activity of occupants.the activity of occupants.
Robotic collaboration:Robotic collaboration: creating robots that can creating robots that can effectively navigate their effectively navigate their environment and interactenvironment and interact with other people and robotswith other people and robots..
Some Interesting Systems Some Interesting Systems from the Industry-1from the Industry-1
Realtime Video Analysis Realtime Video Analysis Group @ Siemens Group @ Siemens Corporate ResearchCorporate Research
Subway monitoring Subway monitoring SystemSystem
Real time segmentation of Real time segmentation of people in subway people in subway platforms for the purpose platforms for the purpose of congestion (crowding) of congestion (crowding) detectiondetection
Honeywell Laboratories:Honeywell Laboratories:
Cooperative Camera Network Cooperative Camera Network (CCN): Indoor(CCN): Indoor
Reports the presence of visually Reports the presence of visually tagged individual throughout a tagged individual throughout a building structure. building structure.
Meant to be used for Meant to be used for monitoring potential shoplifters monitoring potential shoplifters in department stores.in department stores.
Detection of Events for Threat Detection of Events for Threat Evaluation & Recognition Evaluation & Recognition (DETER): Outdoor(DETER): Outdoor
Monitor large open spaces like Monitor large open spaces like parking lots and reports parking lots and reports unusual moving patterns by unusual moving patterns by pedestrians & vehicles.pedestrians & vehicles.
Some Interesting Systems Some Interesting Systems from the Industry-2from the Industry-2
Mitsubishi Electric Mitsubishi Electric Research Research LaboratoriesLaboratories
ApplicationsApplications Detecting accidents through Detecting accidents through
analysis of traffic surveillance analysis of traffic surveillance video. video.
Detection of traffic jams using Detection of traffic jams using our MPEG-7 motion activity our MPEG-7 motion activity descriptor. descriptor.
Extraction of semantic features Extraction of semantic features from low-level features of soccer from low-level features of soccer games. games.
http://www.merl.com/projects/event-http://www.merl.com/projects/event-detection/detection/
Some Interesting Systems Some Interesting Systems from the Industry-3from the Industry-3
ASCOM: INVIS Traffic ASCOM: INVIS Traffic DetectDetect Traffic speed and traffic Traffic speed and traffic
flow density flow density Congestion and slow Congestion and slow
traffic traffic Stationary vehicles Stationary vehicles
(possibly an accident)(possibly an accident)
Copyright © 2002 Ascom Copyright © 2002 Ascom www.ascom.comWrong Way Early Warning
System Recognizes vehicle patterns and
compares subsequent images to determine vehicle and direction.
Thus it can detect any car driving the wrong way into a lane against the flow.