Visual Surveillance in Retail Stores Visual Surveillance in Retail Stores and Homeand Home

Authors:Authors: Tomas Brodsky, Robert Cohen, Eric Cohen-Solal, Tomas Brodsky, Robert Cohen, Eric Cohen-Solal, Srinivas Gutta, Damian Lyons, Vasanth Philomin, Miroslav Srinivas Gutta, Damian Lyons, Vasanth Philomin, Miroslav Trajkovic.Trajkovic.

Philips Research USA.Philips Research USA.

Course: Course: CIS 750 - Video Processing and Video MiningCIS 750 - Video Processing and Video Mining

Semester:Semester: Spring 2003Spring 2003

Presenter:Presenter: Nilesh Ghubade (nileshg@temple.edu)Nilesh Ghubade (nileshg@temple.edu)

Advisor: Advisor: Dr Longin Jan LateckiDr Longin Jan Latecki

AgendaAgenda AbstractAbstract IntroductionIntroduction PTZ Camera CalibrationPTZ Camera Calibration Intruder detection and tracking with a PTZ Intruder detection and tracking with a PTZ

cameracamera Video Content AnalysisVideo Content Analysis Indexing and retrievalIndexing and retrieval Residential intruder detectionResidential intruder detection Object classification Object classification Radial-basis networks.Radial-basis networks. ConclusionsConclusions ReferencesReferences

AbstractAbstract Professional security market Professional security market Retail stores monitoring. Retail stores monitoring. Low-cost automated residential security.Low-cost automated residential security. Pan-Tilt-Zoom (PTZ) camera:Pan-Tilt-Zoom (PTZ) camera:

Intruder trackingIntruder tracking Calibration, enhanced camera control.Calibration, enhanced camera control.

Video Content Analysis, Detection of security related Video Content Analysis, Detection of security related objects and events.objects and events.

This system does real time video processing and This system does real time video processing and provides immediate alarms to alert the security operator.provides immediate alarms to alert the security operator.

Relevant information stored in database for later Relevant information stored in database for later retrieval.retrieval.

Residential monitoring Residential monitoring Intruder detection system Intruder detection system Robust to changes in lightingRobust to changes in lighting Object classification scheme based on radial-basis networks.Object classification scheme based on radial-basis networks.

IntroductionIntroduction Traditional commercial video surveillance systems:Traditional commercial video surveillance systems:

Capture several hours or days worth of video.Capture several hours or days worth of video. Manual search Manual search tedious job. tedious job.

Set Alarms Set Alarms Improvement over manual search method, but … Improvement over manual search method, but … Alarms usually must be defined before capturing video.Alarms usually must be defined before capturing video. Search limited to predefined binary alarms.Search limited to predefined binary alarms. Cumbersome search if alarms are too simplistic or too many false alarms.Cumbersome search if alarms are too simplistic or too many false alarms.

This systemThis system Detects a whole range of events like ‘enter’, ‘people met’, ‘deposit object’, Detects a whole range of events like ‘enter’, ‘people met’, ‘deposit object’,

‘leave’, etc…‘leave’, etc… Semantic indexing and retrieval process used for search.Semantic indexing and retrieval process used for search.

Residential environment Residential environment low cost factor introduces constraints: - low cost factor introduces constraints: - Grayscale cameras (instead of colored ones).Grayscale cameras (instead of colored ones). Limited computational power.Limited computational power. No supervision.No supervision. Robustness to environmental changes.Robustness to environmental changes.

PTZ Camera PTZ Camera CalibrationCalibration Pan-tilt-zoom (stationary, but rotating and zooming) camera advantage: -Pan-tilt-zoom (stationary, but rotating and zooming) camera advantage: -

One camera used for surveillance of large area.One camera used for surveillance of large area. Closely look at points of interest.Closely look at points of interest.

Knowledge of camera position and orientation is crucial for geometric Knowledge of camera position and orientation is crucial for geometric reasoning:reasoning:

Automatically pointing the camera to certain location, by clicking on its position on Automatically pointing the camera to certain location, by clicking on its position on the area map.the area map.

Displaying current field of view of the camera.Displaying current field of view of the camera. Knowledge of internal camera calibration parameters is important for:Knowledge of internal camera calibration parameters is important for:

Tracking with a rotating camera.Tracking with a rotating camera. Obtaining metric measurementsObtaining metric measurements Knowing how much to zoom to achieve a desired view, etc…Knowing how much to zoom to achieve a desired view, etc…

Goal Goal Automatic calibration of surveillance cameras. Automatic calibration of surveillance cameras.

Assumptions:Assumptions: Camera principal point and the center of rotation of the pan and tilt units coincide.Camera principal point and the center of rotation of the pan and tilt units coincide. The skew factor is zero.The skew factor is zero. The principal point does not move while the camera is zooming.The principal point does not move while the camera is zooming. Maximum zoom-out factor Maximum zoom-out factor ss of the camera is known. of the camera is known.

PTZ Camera Internal PTZ Camera Internal CalibrationCalibration

Point the camera to a texture rice area in the room.Point the camera to a texture rice area in the room. Camera zooms in and out completely to acquire two images ICamera zooms in and out completely to acquire two images I11

and Iand I22

Principal point is then determined by scaling down for factor Principal point is then determined by scaling down for factor ss image Iimage I11 and finding the best match for so obtained template in and finding the best match for so obtained template in

the image Ithe image I22

Take two images at fixed pan and different tilt settings.Take two images at fixed pan and different tilt settings. f = -(d / tan f = -(d / tan ) where ) where

f = focal length at particular zoom setting,f = focal length at particular zoom setting, d = displacement of the principal point w.r.t the two images.d = displacement of the principal point w.r.t the two images. = difference in the tilt angle.= difference in the tilt angle.

Compute mapping between zoom settings and focal length, by Compute mapping between zoom settings and focal length, by fitting inverse of focal length (lens power) to the second order fitting inverse of focal length (lens power) to the second order polynomial in zoom ticks. It can be shown that this fitting not only polynomial in zoom ticks. It can be shown that this fitting not only has desirable numerical properties (i.e. stability), but also yields has desirable numerical properties (i.e. stability), but also yields linear solution.linear solution.

PTZ Camera External PTZ Camera External CalibrationCalibration Assumes known camera height and allows the installer to determine camera Assumes known camera height and allows the installer to determine camera

position and orientation in the following manner:position and orientation in the following manner: The user points the camera at several points in the area and clicks on their The user points the camera at several points in the area and clicks on their

respective position on the area map shown in the GUI in Fig 1.respective position on the area map shown in the GUI in Fig 1. Each time the user clicks in a point in the map, the system acquires current Each time the user clicks in a point in the map, the system acquires current

camera position and location on the map camera is point to.camera position and location on the map camera is point to. The algorithm then computes camera position and orientation using data The algorithm then computes camera position and orientation using data

acquired at step 2.acquired at step 2.

Intruder detection and Intruder detection and trackingtracking Target selection: First step of tracking process.Target selection: First step of tracking process.

Placing ‘Target Rectangle’ (TR) on torso, head and part of Placing ‘Target Rectangle’ (TR) on torso, head and part of trousers.trousers.

Hue and saturation color model. Model gray colors separately.Hue and saturation color model. Model gray colors separately. Represent TR by its combined color/gray histogram.Represent TR by its combined color/gray histogram.

Motion detection: This system has Motion detection: This system has procedure for procedure for recursive and fast histogram matchingrecursive and fast histogram matching..

Issue velocity commands so that camera moves towards Issue velocity commands so that camera moves towards the TR and acquire next image.the TR and acquire next image.

This system has improved procedure for feature based This system has improved procedure for feature based image alignment that does not require any information image alignment that does not require any information on camera calibration and camera motion.on camera calibration and camera motion.

Video Content AnalysisVideo Content Analysis System processes video in real-time:System processes video in real-time:

Extracts relevant objects and event Extracts relevant objects and event information.information.

Indexes this information into a database.Indexes this information into a database. Issue alarms to alert the operator.Issue alarms to alert the operator. Separate retrieval software used to search Separate retrieval software used to search

for specific data in the database and to for specific data in the database and to quickly review associated video content.quickly review associated video content.

Assume stationary camera and use Assume stationary camera and use background subtraction technique.background subtraction technique.

Each video frame is compared with the Each video frame is compared with the background model. Foreground pixels background model. Foreground pixels extracted and grouped into connected extracted and grouped into connected components and tracked.components and tracked.

Event detectionEvent detection:: Simple events like enter/leave and merge/split are based on appearance and Simple events like enter/leave and merge/split are based on appearance and

disappearance of foreground regions.disappearance of foreground regions. Event reasoning module generates more complicated events derived from simple Event reasoning module generates more complicated events derived from simple

event stream, based on user provided rules which specify sequence of events, event stream, based on user provided rules which specify sequence of events, length of time intervals, etc…length of time intervals, etc…

Hierarchies of events constructed using Hierarchies of events constructed using feedback feedback strategy.strategy.

Indexing and Indexing and RetrievalRetrieval

Query types:Query types: Merge/Split: Show all people that an identified pickpocket interacted with.Merge/Split: Show all people that an identified pickpocket interacted with. Color model: Group of employees talking without entertaining a customer.Color model: Group of employees talking without entertaining a customer.

New event: Theft New event: Theft Person hiding an object. Person hiding an object.

Residential Intruder Residential Intruder DetectionDetection

Detection of moving objects proceeds Detection of moving objects proceeds in two steps:in two steps:

First, a background subtraction First, a background subtraction technique used to detect pixels that technique used to detect pixels that differ from the background model.differ from the background model.

Additional filter applied to classify Additional filter applied to classify such pixels into real objects or such pixels into real objects or lighting changes. Example:lighting changes. Example:

• Separate person from his/her Separate person from his/her shadow.shadow.

• Moving flashing light on a sofa in Moving flashing light on a sofa in living room produces moving bright living room produces moving bright spot. Current system detects this as a spot. Current system detects this as a lighting change.lighting change.

Compare the gray-level structure of 3x3 or 5x5 neighborhood around each Compare the gray-level structure of 3x3 or 5x5 neighborhood around each detected pixel (using detected pixel (using normalized cross-correlation filtersnormalized cross-correlation filters) with the ) with the corresponding region in the reference (background) image. If the comparison corresponding region in the reference (background) image. If the comparison is similar then the pixel is marked as caused by lighting changes.is similar then the pixel is marked as caused by lighting changes.

Foreground pixels grouped into objects.Foreground pixels grouped into objects. Objects classified into people and animals, so that the system can suppress Objects classified into people and animals, so that the system can suppress

alarms caused by pets (false alarms).alarms caused by pets (false alarms).

Object Object classificationclassification

Objects are classified based on horizontal and vertical Objects are classified based on horizontal and vertical gradient gradient featurefeature that captures shape information. that captures shape information.

The extracted gradients are used to train The extracted gradients are used to train Radial Basis Function Radial Basis Function (RBF) classifier(RBF) classifier Architecture very similar to that of a traditional Architecture very similar to that of a traditional three-layer back-propagation (neural) network.three-layer back-propagation (neural) network.

Overall classification performance Overall classification performance 93.5% 93.5%

Graphical User Graphical User InterfaceInterface

ConclusionsConclusions Automated camera calibration and PTZ Automated camera calibration and PTZ

tracking.tracking. Easy to use GRAPHICAL user interface.Easy to use GRAPHICAL user interface. Efficient indexing and retrieval of video Efficient indexing and retrieval of video

content.content. Improved object classification technique.Improved object classification technique. Surveillance/Security system for Surveillance/Security system for

professional market (retail stores) and low-professional market (retail stores) and low-end market (residential).end market (residential).

ReferencesReferences C. Stauffer, W.E.L. Grimson, “C. Stauffer, W.E.L. Grimson, “Adaptive Background Mixture Models for Adaptive Background Mixture Models for

Real-time TrackingReal-time Tracking”, Proc. Computer Vision and Pattern Recognition.”, Proc. Computer Vision and Pattern Recognition.

F. Bremond, M. Thonnat, “F. Bremond, M. Thonnat, “Object Tracking and Scenario Recognition for Object Tracking and Scenario Recognition for Video SurveillanceVideo Surveillance”, Proc. IJCAI, 1997.”, Proc. IJCAI, 1997.

E. Stringa and Carlo S. Regazzoni, “E. Stringa and Carlo S. Regazzoni, “Real-Time Video-Shot Detection for Real-Time Video-Shot Detection for Scene Surveillance ApplicationsScene Surveillance Applications”, IEEE Trans. Image Processing, Jan ”, IEEE Trans. Image Processing, Jan 20002000

R. P. Lippmann and K. A. Ng, “R. P. Lippmann and K. A. Ng, “Comparative Study of the Practical Comparative Study of the Practical Characteristic of Neural Networks and Pattern ClassifiersCharacteristic of Neural Networks and Pattern Classifiers”, MIT ”, MIT Technical Report 894, Lincoln Labs, 1991.Technical Report 894, Lincoln Labs, 1991.

D.M. Lyons, T. Brodsky, E. Cohen-Solal and A. Elgammal, “D.M. Lyons, T. Brodsky, E. Cohen-Solal and A. Elgammal, “Video Video Content Analysis for Surveillance Applications”Content Analysis for Surveillance Applications”, Philips Digital Video , Philips Digital Video Technologies Workshop 2000.Technologies Workshop 2000.

Thank youThank you