102-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures

102-OCT-02CANDELA presentation STSI Confidential information/proprietary

CANDELA Content Analysis for Networked DELivery

Architectures

http://www.extra.research.philips.com/euprojects/candela/

OverviewFrench ConsortiumSE Demonstrator

MAM DemonstratorFrench Partners Presentation



Architectures





CANDELA

General goals:

Intelligent video delivery applications

Development of innovative content analysis technology

Involved technologies

• Computer vision• Networks, storage


Consortium: 35 Partners

Project leaderWP leaderFrench

Industry:• Bosch CSI NL• Philips SLE NL• LuTech I• Philips• Philips PDSL NL• Philips Res. NL• Siemens D• Thomson F• Thales F• Telecom Italia I• Ibermatica E• TXT I

263,5 MY in 7 countries:• Belgium (31), Finland (46), France (49), Germany (36), Italy (35,5),

Netherlands (40), Spain (23)

Universities:•U Paderborn D•TU Berlin D•Free U Brussels B•KU Leuven B•TU Eindh. EESI NL•Cefriel I•ESI E•INRIA F•VTT Fi•Multitel B

SME:•Solid Fi•Hantro Fi•capVidea B•Ciaolab I•IT Optic B•MicroGenesis E•RheaSystem B•Retevision Movil E

•AlgoVision/ LuraTec D•Empolis D•TecMath D•Vitec F•Quadrox B•Vartec B


Costs: 33,8 Mio EUR

Belgium

11%

Germany14%

Spain7%

France

19%

Finland

19%

Italy15%

Netherlands

15%


CANDELA Structure

Video server

Terminal

metadatabase

ContentAnalysis

Network

WP3: Architectures

WP1: Content Analysis WP2: Networked Delivery


CANDELA Structure

Content analysis

Coding

Integration

Platform

Application

Validation

Requirements

Delivery

Security

WP1WP2

WP3

StorageRetrieval


CANDELA Working Packages

WP0: Project management• WP leader: Bosch CSI (NE), Paul Merkus

WP1: Content Analysis• WP leader: Philips SLE (NE), Ronald Begeer

WP2: Networked Delivery• WP leader: Solid (SME, Fi), Pauli Berg

WP3: Architectures• WP leader: LuTech (SME, I), Simona Costa


WP 1: Content Analysis

Object tracking (TMM, F)• Object-based metadata generation

Feature extraction (CapVidea, B)• Metadata generation from video

Video structure (Vitec, F)• Metadata generation on temporal video structure

Semantic metadata (INRIA, F)• High-level metadata• Metadata representation language

Semantic video encoding (VUB, B)• metadata-driven video encoding


WP 2: Networked Delivery

Efficient video storage and retrieval technology (Philips Research, NL)• User storage, provider storage, distributed storage• Scalable and reliable storage• Database retrieval by mobile user, by provider

Intelligent video delivery technologies (TU Eindhoven, NL)• Adaptive video delivery• Profiled delivery• Video delivery technologies

Video network delivery (Thales, F)• Video Network Delivery architecture• Video delivery and data management• Adaptive and Intelligent video services


WP 3: Architectures

Applications (TXT, I)• Application requirements• Demonstrator specifications

Platform & security (Cefriel, E)• middleware• standard compliant implementations

Methodology (ESI, S)

Integration & validation (Lutech, I)• Standard compliance• Test, demo

Exploitation, operation (Rhea, B)


Deliverables

2004

Application Spec.

2003

Progress report

IntegrationFinal Demonstrator

Technology Demonstrators

Technologies

Progress report

Progress report

Progress report

2005

Technology Spec.

Platform



Architectures


Overview

French ConsortiumSE Demonstrator



CANDELA: French Consortium

Total French effort: 49 MY, 6.5 Mio EUR

0

0

6

INRIA

1

9

8

Thales

00WP 2Networked Delivery

22WP 3Architectures

714WP 1Content Analysis

VITECTMMMY

618 916Total

9

5

35

Total

49 MY


French Consortium

May 2003: 40 MY (Thales, TMM)

June 2003: 36 MY (Thales, TMM)

July 2003: 56 MY (Thales, TMM, VITEC, INRIA)

August 2003: 49 MY (Thales, TMM, VITEC, INRIA)

Reduction

Equilibration

Reduction


MAM DemonstratorSE Demonstrator

French Demonstrators

Terminal

Video server

metadatabase

ContentAnalysis

Network


French Demonstrators

TMM Thales INRIA VITEC

MAMMutimedia

ContentManagement

SESmart

Encoding

MAMSE

MAMSE

New partner for MAM: content provider?



Architectures


OverviewFrench Consortium

SE DemonstratorMAM demonstrator

French partners presentation


Smart Encoding (SE)Demonstrator


Smart Encoding (SE)Demonstrator Context

Video-surveillance application• Requires video coding scheme adapted to the video-surveillance needs

– Video coders have been developed mainly for broadcast like applicationsStatic details are privilegedMoving objects are coded with reduced quality and resolution

– In contradiction with video-surveillance requirements where the best quality and resolution has to be ensure for mobile objects.

• Need of enhanced network delivery to cope with video-surveillance network constraints

–Heterogeneous secured networks

–QoS

–Fast and easy/automatic reconfiguration

This demonstrator results from WP1, WP2 and WP3 works


Smart Encoding (SE)Issues

Smart video encoding• Region of interest adapted video coding

– Object tracking and scene identification

To segment and track moving objects in video scenes such as persons. The information on moving objects will provide new type of semantic metadata for

multimedia asset management (MAM)Metadata on moving objects will allow object-based bandwidth and system resource

allocation in video surveillance and mobile applications.

– Evolution and adaptation of existing video coding schemes (MPEG-4, H.264 ….)

To exploit high-level information extracted from the audiovisual content analysis to drive the subsequent lossy compression mechanisms

Target solutions that are compliant with standard compressed video stream formats, (MotionJPEG 2000, MPEG and/or H26x …), to ensure a widespread dissemination of the project’s results.


Smart Encoding (SE)Region of Interest Adapted Video Coding

Mobile object detection &

tracking

Video encoding

ROI driven better quality

Background lower quality

Metadata

Coded video

Merging• Region of interest adapted

video coding • Object detection and tracking


Smart Encoding (SE)Networking Issues

The necessity of delivering with good quality different video content on different type of network implies addressing :• Real time networking• Tight traffic control• Optimisation of the link utilisation

The need to deliver information in a secure way means addressing the following security issues:• Confidentiality, integrity, authentication, protection against replay, access

control

Necessity for reconfiguration and reactivity implies• Simple and homogeneous management of multiple network services • Dynamic network management (security, QoS)


Smart Encoding (SE)Issues

Core Network

QoS &Securitymanagement

Reconfigurability management


Smart Encoding (SE) Tasks & Partners Involved

Object-based metadata generation (WP1 Task 1.1, 1.2 and 1.4)• Event detection (Scenes identification and object tracking)• French partners involved

–TMM

–THALES

–INRIA

Metadata-driven video encoding (WP1 Task 1.5) • Evolution and adaptation of existing encoding schemes

–MPEG-4/H.264

–MotionJPEG 2000

– …..

• French partners involved–VITEC

–THALES

Other european partners involved• VUB, Multitel, AlgoVision...


Smart Encoding (SE) Tasks & Partners Involved

Video delivery network management (WP2 Task 2.2)

• Qos and security network architecture

• Qos and security Policy based management

Intelligent video services (WP2 Task 2.3)

• End to end QoS

• Dynamic security routing

European partners involved

• VUB, Solid, TUE, Ciaolab, Philips NL, Telecom Italia,...


Smart Encoding (SE) Expected Results

Contributions to Standards• MPEG-4/H.264, MotionJPEG 2000

–Additional Guidelines Annexes–Amendements

• Following IETF recommendations (draft, FRC)Technology

• Merge of two different/separate technological domains (image coding & Object detection/tracking ).

• Enhanced network cope with video-surveillance constraintsProducts

• VITEC–Know How for product line evolutions

• THALES–New functionalities for video surveillance systems



Architectures



MAM demonstratorFrench partners presentation


Multimedia Asset Management (MAM)Demonstrator


Multimedia Asset Management (MAM)

RepeaterEditing

Ingestion

Playout serverVideo server

Low qualityvideo serverAnnotation

terminal

metadatabase

playout

Internet

Encoder

Logging

Ingestion

Creation

Retrieval


CANDELA

MAM Workflow

Ingest• Development• Digitizing• Automatic technical metadata (date, time, format• Incoming content:

–Dailies from film production

–Live events

–reused content

Logging• Manual Annotation

Creation• Editing, cutting• Color correction

Archiving/Retrieval• Storage

Ingestion

Video server

Annotationterminal

metadatabase

Logging

Ingestion

Retrieval

Editing

Creation


MAM Issues

Automatic metadata very limited

Time costly manual annotation

Inefficient re-use of stored content

Need of cost-efficient metadata generation

Need of semantic metadata to enable content re-use


New MAM Technologies

Intelligent ingest:Automatic generation of semantic-level metadata

• Temporal video summary video chapters• Semantic categories keywords• Description language interpretation

Object-based logging:Semi-automatic object-based annotations

• Object segmentation object-based annotations• Object recognition keywords

Task 1.3 Video structure (VITEC)

Task 1.2 Object description (INRIA)

Task 1.4 Semantic Metadata (INRIA)

Task 1.1 Object tracking (TMM)


Video Structure Generation

unstructured video

Shot detection

Keyframeextraction Classification

Scenedetection

keyframes

Video chapters


Object Tracking

Imageregion

tracking

Objectannotation

Objectdetection

video frame

Annotations: horse



Architectures



MAM demonstrator

French partners presentation


Thomson MultimediaContribution


… through content distribution…

From contentProduction…

Video Image Chain

> Movies: Production of distribution copies, marketing support, digital movie systems > Video: Production and delivery of VHS and DVD packaged media; integrated services> Networks: Professional video equipment for satellite, cable, telecommunications and ground networks; associated services

> Receiver products: TV sets, VCRs, DVDs, audio systems, decoders, etc. > Interactive services: Electronic program guides, « back office » services for broadcasters / advertisers, interactive TV > Components: Flat screens, high-definition, plasma, optical components, electronics, etc.

Thomson all along the Video Chain

> Development of film negatives> Post-production services: image processing (digitalization, colorization, audio and video montage, etc.)> Management of digital content: indexing, compressing, formatting, archiving, routing images > Studio services and equipment: cameras, integrated studio video servers, etc.

… until content access

Group businesses involved: SBU Digital Media SolutionsSBU Patents and LicensingResearch and Innovation

Group businesses involved: SBU Broadband Access Products and SystemsSBU New Media ServicesSBU Patents and LicensingResearch and Innovation

Group businesses involved: SBU TV, Accessories and AftersalesSBU Audio, Video & ATLINKSSBU Displays and ComponentsSBU Patents and LicensingSBU New Media ServicesResearch and Innovation


Thomson Goals in CANDELA

Goals• Acquire technologies for automatic metadata generation • Exploit intra-image information (object segmentation)• Rise the semantic level of metadata

Technologies• Image classification (e.g. sports, indoor, family)• Object tracking (e.g. cars and faces)• Object recognition (e.g. faces)

Aimed applications: MAM• Ingest with automatic semantic annotation• Object tracking for semi-automated logging• Semantic browsing of multimedia assets


TMM Contribution

WP1 Content Analysis• Task 1.1 Object Tracking (task leader) 10 MY• Task 1.3 Video Structure 4 MY

WP3 Architecture

• Task 3.1 Applications 1.5 MY• Task 3.4 Integration & Validation 0.5 MY

16 MY2200 kEUR


TMM in Task 1.1 Object Tracking

Task leadership• State-of-the-art report• Requirements• Object tracking for simple scenes• Object tracking for complex scenes• Demonstration, report


TMM in Task 1.1 Object Tracking

Object Tracking Simple Scenes• Tracking of object silhouettes

Object Tracking Complex Scenes• Motion-based object tracking (general objects)• Pattern-based object tracking (faces, logos)• Combined tracking (persons, cars)

Innovation• Domain specific knowledge• Combination of motion (image analysis)

and pattern clues (statistics)

video frame

Annotations: horse


TMM in Task 1.3 Video Structure

Video structure

Problem: Semantic gapUser:

Show me the whole forest sequence!

User metric:

Semantic similarity

MAM System:Here are all green frames

System metric:

Visual similarity

videokeyframes

Video chapters


TMM in Task 1.3 Video Structure

Innovation: Semantic classification of video scenes

OUTDOOR

INDOOR


TMM in WP3 Architectures

Task 1.1 Applications• MAM application specification• MAM application requirements

Task 3.4 Integration & validation• MAM demonstrator integration

TMM skills• Business units in professional video• Customers in content creation and archiving


Thales Contribution


THALES Presentation

Thales Corp.• Leading European & worldwide positions in defence and aerospace electronics Leading-edge technologies in

IT&S for high-tech B2B markets• 3 business Areas:

DefenseAerospace

Information Technology & Services

THALES CommunicationsWe design and supply global information and communications systems solutions

Seamless Secured Standard and interoperable

• Thales Security &System:• Design, development and turnkey installation of integrated security systems dedicated to the protection of

sensitive civil and military sites• High-tech systems integrator and prime contractor for large-scale security projects• 20 years of extensive experience and technical expertise in electronic security (access control, video surveillance,

intrusion detection, perimeter protection, global supervision


THALES Contribution :Goals & Points of Interest

Thales Goals in CANDELA• Goals

Acquire technologies for Smart Video encoding Exploit standardized Technologies (Video &Network) for specific uses

• TechnologiesObject tracking Object recognition (e.g. person)Content delivery (Network)

• Aimed applications: videosurveillance...Thales Points of Interest:

• WP1 : Content analysisContent Analysis for Event detection and Flexible compression

• WP2 : Content Delivery Network secured delivery and QoS Management

• WP3 : ArchitectureDefinition of user requirements (for video surveillance)Definition of the network architecture


THALES Contribution WP1 Content Analysis

Focus on Smart Encoding

• Task 1.1 (Object tracking)– State of the art

– Requirements

– Object tracking in simple scenes (specially in out door condition)

– Participation in object tracking in complex scenes

– Demonstration

• Task 1.5 (Semantic video encoding)– State of the art

– Smart encoding

– Flexible lossy compression

– Scalable encoding

– RC Region Of Interest (ROI)

– Demonstration


THALES Contribution WP2 Content Delivery

Focus on network management• Task 2. 3.1(Video Network delivery management)

–State of the art report concerning QoS and security on network side and Policy based network management for QoS

–Participation to the definition of the Multimedia network reference architecture for QoS and security

–QoS and Security Policy based management

–Deployment of this architecture in a research platform

–Test with WP1 streaming applications

• Task 2.3.2 (Intelligent Video Delivery Technologies)–Develop new mechanisms management for providing end to end QoS

–Develop new mechanisms to assure the mobile user to benefit of permanent security

–Participation to the “intelligent delivery solutions” report (presentation of the QoS and security solutions)

–Deployment of those solutions in a research platform


MultimediaReceiver

=> Bad Reception

Network Load

THALES Contribution WP 2 QoS Themes

Classification in the edge (EF)

PHB in core routers

=> Good reception

Step 2 : dynamic management

Step 3 : RSVP


THALES Contribution WP 2 Security Themes

PKIPKI

: Clear text

: Encrypted text

Authentication Encryption Tunnel

IKE Session

Negotiation of the security parameters

Exchange of the certificates

PKI : Public Key InfrastructurePM : Policy ManagerAD : Access Device

PM

PM

Step 2: Dynamic management

Step 3 : Dynamic tunnel rebuildin case of mobility


THALES Contribution WP3 Architecture

Application definition and user requirement analysis (in the context of video surveillance)

Definition of the platform (in term of network architecture)

Integration and validation of a video surveillance demonstrator

Participation to the security task (by bringing the results obtained in the WP2 security task)


VITEC MultimediaContribution


VITEC Presentation

VITEC Multimedia

Company established in 1988Independent companyRevenues: 3M€Capital: 1,5M€25 people


Market of VITEC Multimedia

Digitalvideo

Video

Computers Telecom


VITEC Objectives in CANDELA

Productivity tools for mass video image databases (MAM & SE):

- image content extraction and classification(content editing)

- video structuring and data reduction(scalable content delivery)

- sequence identification and retrieval(content query)

- MPEG encoding enhancement(content-driven coding)


WP1 VITEC Contribution (1)

Task 1.3: Video Structure (Task Leader: 3.50 Mys)

- Automatic chapter generation :- Scene identification ( video shot grouping)- Chapter generation (story-telling principles)

- Video data reduction:- Video summarization- Video digest generation


WP1 VITEC Contribution (2)

Task 1.4: Semantic description (1.50 Mys)

Image sequence segmentation and indexing based on content extraction and classification

Task 1.5: Semantic video encoding (2.00 Mys)

MPEG encoding enhancement:image sequence coding using a compression rate

driven by image content


WP3 VITEC Contribution

Task 3.1: Requirements and Overall Application (1.00 Mys)

- user’s requirements gathering from one’s own customers base

- design of architecture and video processing modules

Task 3.4: Integration and Validation (1.00 Mys)

- video structuring and semantic description softwareat MAM demonstration disposal

- participating to integration and validation of SE demonstrator


INRIAContribution


ORION team: Research directions

Intelligent Reusable Systems for Cognitive Vision

Intelligent: explicit knowledge, reasoning and learning capabilities

Reusable Systems: for different applications or problems

Cognitive Vision: image understanding

Multidisciplinary team: artificial intelligence, software engineering, computer vision

INRIA Activities Monique THONNAT Sophia Antipolis


Objectives:

• real time and automated analysis of video sequences

• video understanding = from people detection and tracking to behavior

recognition

Examples:

• recognition of bank agencies scenes for visual surveillance

• recognition of metro scenes for visual surveillance

INRIA Video Understanding



Our goal:

to model the interpretation process of video sequences from pixel up to behaviour.

time

..

Recognised scenario

Video stream

Mobile object detection &

tracking

“Vandalism ?”

“Two people are fighting ?”

Scenario recognition


Behaviour recognition: detection of an agitated group in metro


lively

Agitated behaviour


Behaviour recognition:

• approach based on a priori knowledge

model of the empty scene (3D geometry and semantics)

models of predefined scenarios

• a language for representing scenarios based on combination of states and

events

10 states and 21 events can be used

• a reasoning mechanism for real time detection of states, events and

scenarios (e.g. temporal reasoning, constraints solving techniques)




Description of a bank attack scenario Scenario(Attack,

Characters((cashier : Person), (robber : Person))

Constraints((exists ( state(s1, inside_zone, cashier, "Back_Branch")

event(e, changes_zone, robber,"Gate","Infront_Branch") state(s2, inside_zone, cashier, "Safe_zone") state(s3, inside_zone, robber, "Safe_zone") )

((s1 before s2) (s1 before s3) (e before s3) ) ) )

Production((name of sc : Scenario Attack) (Start of sc := Start of s1) (End of sc = End of s2)))



The recognition of the scenario “Attack” in a bank agency


Description of a vandalism scenario.

Scenario(vandalism_against_ticket_machine, Characters ((p : Person), (eq : Equipment, Name = “Ticket_Machine”) )

Constraints((exist ((events1: p move_close_to eq)

(state s2: p stay_at eq) (event s3: p move_away_from eq) (event s4: p move_close_to eq) (state s5: p stay_at eq) ) (s1 before s2) (s2 before s3) (s3 before s4) (s4 before s5) ) ) )

Production( (sc : Scenario) ( (Name of sc := "vandalism_against_ticket_machine")

(StartTime of sc := StartTime of s1) (EndTime of sc := EndTime of s5) ) ) )





INRIA Activities

INRIA will participate to WP1 on video content analysis• task 1.1 Object Tracking 2 mys• task 1.2 Feature Extraction/Object Description 1 my• task 1.4 Semantic Metadata (task leader) 3 mys

INRIA activities in SE demonstrator

• to detect interesting events and regions of interest for smart encoding using intelligent video understanding techniques for fixed visual surveillance (task 1.1, 1.2 and 1.4)

INRIA activities in MAM demonstrator• to help metadata generation (logging) by:

+ proposing a standard vocabulary for video annotation (task 1.4)

+ attempt automating video description (task 1.2 and 1.4)


April 3/4 2003kick-off meeting

INRIA Sophia Antipolis

Documents

102-OCT-02CANDELA presentation STSIConfidential information/proprietary CANDELA Content Analysis for Networked DELivery Architectures