ETISEO, Nice, May 11-12 2005
PETSPETSInternational Workshops onInternational Workshops on
Performance Evaluation of Tracking and SurveillancePerformance Evaluation of Tracking and Surveillance
James FerrymanJames FerrymanComputational Vision GroupComputational Vision Group
Department of Computer ScienceDepartment of Computer ScienceThe University of Reading, UKThe University of Reading, UK
ETISEO, Nice, May 11-12 2005
Supported by
PETSPETSInternational Workshops onInternational Workshops on
Performance Evaluation of Tracking and SurveillancePerformance Evaluation of Tracking and Surveillance
ETISEO, Nice, May 11-12 2005
IntroductionIntroduction
Theme - Performance Evaluation of Tracking and Surveillance
Successful tracking of object motions key to visual surveillance
PETS started in Grenoble, France in 2000 as satellite workshop of FG2000
Not a competition http://visualsurveillance.org ftp://pets.rdg.ac.uk
ETISEO, Nice, May 11-12 2005
PETS - HistoryPETS - History
PETS’2000 was held at FG’2000, 31 March 2000, Grenoble, France.
PETS’2001 at CVPR’01. PETS’2002 at ECCV, Copenhagen, Denmark, June
1 2002. PETS2003 at ICVS, Graz; VS-PETS at ICCV2003 PETS2004 at ECCV04 WAMOP-PETS, CO, USA (Jan 05) as part of IEEE
Winter Workshop Series 2005: VS-PETS at ICCV’05
ETISEO, Nice, May 11-12 2005
Datasets – Example – PETS2001Datasets – Example – PETS2001
Five separate sets of training and test sequences.
All datasets are multiview (frame sychronised).
Datasets were significantly more challenging than PETS2000 (significant lighting variation, occlusion, scene activity and use of multiview data)
ETISEO, Nice, May 11-12 2005
DatasetsDatasets
Dataset 2
Dataset 1
Dataset 3
ETISEO, Nice, May 11-12 2005
Dataset 1Dataset 1
ETISEO, Nice, May 11-12 2005
Dataset 2Dataset 2
ETISEO, Nice, May 11-12 2005
Dataset 4Dataset 4
ETISEO, Nice, May 11-12 2005
PETS - PrerequisitesPETS - Prerequisites
Tracking results reported– should be performed using the test sequences, but the
training sequences may optionally be used if the algorithms require it (for learning etc.)
– may be based on a single camera view of the scene, or using multiple view data.
– can be based on the entire test sequence, or a portion of it; the images may be converted to any other format and/or subsampled.
– results must be submitted in XML format.
ETISEO, Nice, May 11-12 2005
PETS – Workshop OverviewPETS – Workshop Overview
XX contributed papers~3 sessions: e.g. appearance-based
tracking, people and vehicle tracking, multiview tracking
Y invited speakersDemonstration sessionOverall evaluation and discussion
ETISEO, Nice, May 11-12 2005
Quantitative PE - XMLQuantitative PE - XML
XML provides mechanism of setting up “syntax” file in form of schema
Schema used to automatically validate object tracking results
For PETS’2001, two schemas were used:– low-level tracking results– high-level surveillance
(understanding object motions and interactions)
ETISEO, Nice, May 11-12 2005
Quantitative PE - XMLQuantitative PE - XML<?xml version='1.0' encoding='ISO-8859-1' ?>
<!-- -->
<!-- Example file for visual surveillance reporting -->
<!-- -> scene understanding with multiple cameras. -->
<!-- Edited by PETS2001.Reading.JMF ([email protected]) -->
<people_tracker xmlns="http://www.cvg.cs.reading.ac.uk/PETS2001"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://www.cvg.cs.reading.ac.uk/PETS2001
http://www.cvg.cs.reading.ac.uk/PETS2001/XML/surveillance.xsd">
<!-- this is a comment ... add more as appropriate -->
<header>
<recording site="PETS2001 (Reading)" session="1" date="01/06/01">
<list_cameras num_cameras="2">
<camera camera_id="1"/>
<camera camera_id="2"/>
</list_cameras>
</recording>
ETISEO, Nice, May 11-12 2005
Quantitative PE - XMLQuantitative PE - XML
<video start_frame="1" end_frame="1450" step="2" fps="25"/>
<!-- step: stepping to read list of processed frames below -->
<!-- (for (i=start; i<=end; i+=step); i++) -->
<!-- fps: frames per second of original video -->
<image xdim="768" ydim="576" colour="1"/>
<!-- dimensions of video images and whether it is in colour (0,1) -->
<software name="Reading People Tracker" platform="Linux" version="0.03"
run_date="12/07/00">
<!-- information about the software this file originates from -->
<object_detector name="Reading People Tracker" platform="Linux"
version="0.03" run_date="08/06/01"/>
</software>
</header>
Quantitative PE - XMLQuantitative PE - XML<sequence>
<!-- the actual data: a sequence of one or more frames. -->
<frame id="2" num_targets="1">
<target id="8" start_frame="1" end_frame ="2">
<!-- a "target" is any object which moves or may move, usually a
person, group of people, or a vehicle.
The target's id is GLOBAL to all the cameras defined in
"list_cameras" -->
<!-- start_frame and end_frame indicate when the target has been
tracked. end_frame may be unknown because it is in the future;
in this case the longest known time where the object was
tracked will be given -->
<track status="4" location="0" speed="200" trajectory="1" t_confidence="0.5" num_parents="2">
<!-- track is part of a graph representing tracks of all targets -->
<!-- the status of a graph node explains how the node of current
target has been created or tracked. The following values may
be used and added together as appropriate:
0 : default value, already tracked
1 : new track (id did not exist before)
2 : re-appearing object (id copied from last occurrence)
4 : merging (more than one parent in graph)
8 : splitting (at least one parent in graph has more than 1 child)
16 : lost (object NOT found in current image, given position etc
are estimates (if available) or previous values)
32 : out of field of view (tracked object not "visible" as per
definition (see elsewhere)) -->
<!-- location values are defined as the sum of the following:
0 : undefined
1 : roadway
2 : in close proximity to vehicle (parking lot)
4 : on grass/verge
8 : other -->
ETISEO, Nice, May 11-12 2005
D1C1: XML outputD1C1: XML output
ETISEO, Nice, May 11-12 2005
D1 C1 - 1
ETISEO, Nice, May 11-12 2005
D1 C1 - 1
ETISEO, Nice, May 11-12 2005
D1 C1 - 1
ETISEO, Nice, May 11-12 2005
D1 C1 - 1
ETISEO, Nice, May 11-12 2005
D1 C1 - 1
ETISEO, Nice, May 11-12 2005
D1 C1 - 1
ETISEO, Nice, May 11-12 2005
D1 C1 - 2
ETISEO, Nice, May 11-12 2005
D1 C1 - 2
ETISEO, Nice, May 11-12 2005
D1 C1 - 2
ETISEO, Nice, May 11-12 2005
D1 C1 - 2
ETISEO, Nice, May 11-12 2005
D1 C1 - 2
ETISEO, Nice, May 11-12 2005
D1 C1- 3
ETISEO, Nice, May 11-12 2005
D1 C1- 3
ETISEO, Nice, May 11-12 2005
D1 C1- 3
ETISEO, Nice, May 11-12 2005
D1 C1- 3
ETISEO, Nice, May 11-12 2005
D1 C1- 3
ETISEO, Nice, May 11-12 2005
D1 C1- 3
ETISEO, Nice, May 11-12 2005
D1C1: XML output 1D1C1: XML output 1
ETISEO, Nice, May 11-12 2005
D1C1: XML output 2D1C1: XML output 2
ETISEO, Nice, May 11-12 2005
D1C1: XML output 3D1C1: XML output 3
ETISEO, Nice, May 11-12 2005
D1C1: XML output 4D1C1: XML output 4
ETISEO, Nice, May 11-12 2005
D1C1: XML output 5D1C1: XML output 5
ETISEO, Nice, May 11-12 2005
Performance EvaluationPerformance Evaluation
Evaluation of surveillance system can be judged in a number of ways:– object detection lag– object centroid position error– object area error– track incompleteness factor– accuracy of semantics of interaction– object identity error
• maintenance of identity through occlusion• …
ETISEO, Nice, May 11-12 2005
384 x 288 (JPEG)
10 fps
384 x 288 25-30 fps 600 MHz
Dual PIII
850 MHz
384 x 288 (AVI)
384 x 288 5 fps
768 x 576 5 fps
384 x 288 6.25 fps 800 MHz PIII
320 x 240 29.97 fps 1 GHz PIV
5fps 1.7 GHz PIV
768 X 576 5fps
Image Format Processing Speed Processor
ETISEO, Nice, May 11-12 2005
DiscussionDiscussion
Evaluation criteria are application dependent
Training data – required or not?representative exampleshow much?
Semantics of XML schema Ground truth
difficult to obtainautomatic evaluation - how?
ETISEO, Nice, May 11-12 2005
PETS EvaluationPETS Evaluation
+ve: “Mindset” – engaging the community –change of culture
+ve: Repository of data (PETS01 most frequently accessed)
+ve: Discussion/presentation of methodologies, metrics, tools …
+ve: Filters through to conferences/published literature
-ve: For workshop, choice of dataset(s)+ annotation
-ve: More quantitative evaluation needed
ETISEO, Nice, May 11-12 2005
PETS, ETISEO and the future …PETS, ETISEO and the future … Online web-based evaluation service
(Semi-)automatic validation of XML against ground truth
Repository of algorithms (incl. “strawman”), and tabulated results (rank?)
Methodology for evaluationMetrics
More challenging datasets (e.g. multiview) Live workshop sessions on “unseen” data Expectation that ETISEO will support PETS
ETISEO, Nice, May 11-12 2005
PETS’05PETS’05
ICCV ’05, Beijing, China15-16 October 2005http://www.cbsr.ia.ac.cn/
conferences/VS-PETS-2005 http://visualsurveillance.org/PETS2005