3 May, 2015 Richard Verhoeven, [email protected] TU/e Computer Science, System Architecture and Networking 1 Richard Verhoeven The Candela Project

April 18, 2023 Richard Verhoeven, [email protected]/e Computer Science, System Architecture and Networking

1

Richard Verhoeven

The Candela ProjectBosch demonstrator perspective


2

Outline

• Introduction• Content Analysis• Networked DELivery• Architectures• Bosch demonstrator• Demo


3

Introduction

• ITEA project 02013• Content Analysis and Networked DELivery

Architectures• Partners– Philips, TU/e, Bosch (NL)– Multitel, IT-Optics, VUB,

Traficon (BE)– Hantro, Solid, VTT (FI)– ESI, Amena, Ibermatica,

CanalMicro (ES)


4

Introduction

• Focus– Home Multimedia– Surveillance– Medical systems

• Issues– Large multimedia collections– Distributed storage

• Solution– Automatic metadata extraction– Content delivery framework

Acquisition

CompressionContentAnalysis

Search & Retrieve

Deliver

Present

Contentdatabase

Metadatadatabase

MPEG7MPEG2/4


5

Introduction

User Interface

Cable

WWW

AV

Content & descriptors

Content Management

HDD

Database

MPEG7Storage


6

Content Analysis

• AV segmentation– Chaptering of video content

• Object tracking– Traffic movement, suspicious behavior

• Idle object detection– Airport and public transport surveillance

• Face detection & recognition– Photo & video content management

• Sports analysis– Statistics, improved video compression


7

Content Analysis

• Voice recognition– Video content management

• Optical character recognition (OCR)– License plates, news tickers, prints on trucks

• Classification– Traffic type based on speed and size


8

Content Analysis - Issues

• Light conditions– Shadow, clouds, head lights, back light

• Merging objects– At traffic light, near crossings

• Processing intensive– Different & variable frame rates


9

Network delivery

• Content standardization– Video content – MPEG4– Metadata – MPEG7

• Control protocol– SOAP– UPnP

• Distribution and load balancing• Reliability and robustness


10

Network delivery

• Database storage– Allows persistent storage of features– Allows nonlinear features browsing and

searching– Allows non real-time features processing– Enables features distribution using metadata

standards– Enables new applications– Serves as a reference to evaluate new algorithms


11

Network delivery

video1

VCA MDA

VCA DB

VCA

ACA

VCA

video 2

audio

Home multimediademonstrator

Networkedinfrastructure

Mobilephone

ACQ VCA DB

VCA

Medical systemsdemonstrator

ACQ

VCA MDA

VCA DB

VCA

alarm

Surveillance demonstrator

retrieval

DBXML/XLS

XHTMLpresent

Videostreaming

Interoperability of media and devices:

eTourism


12

Architectures

• Centralized & static (Bosch)– Content analysis modules known in advance– Connections between modules fixed– Central controller to manage data flows– Not easy to configure, not very robust

• Distributed & dynamic (Philips)– Content analysis modules can be added– Connections created on demand– Flexible and robust – a.k.a. Cassandra platform


13

Bosch Demonstrator

• Content analysis– Acquisition (Bosch)– Object tracking (Bosch)– Metadata analysis (Bosch)– Face recognition (TU/e, VCA group)– Idle object detection (Multitel)– Alarming (Bosch, Solid)

• Network delivery – Controller & Middleware (Multitel)– MPEG7 (VUB)

• Application / Architecture– Database (Bosch)– Retrieval client (TU/e, SAN group)


14

Bosch Demonstrator


15

Middleware

• Communication transparency• Video & metadata router (controller) in

star-structure• Simple connect(), getData(), and setData() functions

• Decoding of MPEG-4 under API• Buffering in case of computational

overflow• Implementation for Windows and Linux

PCs


16

Middleware

ControllerACQ

Track MDA

Idle Face

DB

MPEG4

MPEG7


17

VCA Modules

• Acquisition– Live video grabbing with real-time

compression – MPEG-4 (colour, CIF res., 1 MBit, 12.5 fps)

• Object tracking– Algorithm provided by BOSCH

• Shape and bounding box of moving objects• Assignment of object IDs• Sensitivity difficult to control

– Shape information ignored– Filter on short-term objects– Thumbnail selection


18

VCA Modules

• Metadata analysis– Analyse bounding boxes from object tracking– Calculate size and speed based on camera info – Classify object based on size

• Truck/bus, car, person/other

– Extract trajectories from center of bounding boxes

– Create events for certain trajectories• Turn left, turn right

– Future extension• Color analysis in thumbnail (yellow car, blue truck)


19

VCA Modules

• Face recognition– Provided by Fei Zuo from VCA group– Based on neural networks– 2 - 4 fps– Difficult to combine with

other VCA modules


20

VCA Modules

• Idle object detection– Provided by Multitel– Changes with respect to background image

• Database storage– Video: MySQL database with GOVs (16

frames)– Metadata: PostgreSQL database with

OpenGIS for trajectory• Alarming– Receive metadata from face recognition– Inform mobile terminal of recognized person– Display video on mobile terminal


21

Integration issues

• Windows & Linux– Middleware library cross-platform– MPEG7 library platform specific

• Memory leaks in middleware libraries– Unclear who had to free memory– MPEG7 data is in XML

• Stability of the system– Endurance test– Fill the database with useful content


22

Integration issues

• Different frame rates and buffering– Face recognition at 2 fps, rest at 12.5 fps

• Variable frame rate once a face is recognized

– Middleware doesn’t provide buffer info– Possible fix: separate thread to retrieve frames– Used fix: check delay in frame retrieval

• Video codecs and formats– YUV vs RGB vs RBG


23

Retrieval client

• Search through metadata and watch video• Most visible part of the demonstrator– Only way to show the content of the database

• Ad-hoc application– Sketch on one A4 and a list of nice features– Integration of work-in-progress parts

• Trajectory search by Rob Albers

– Debugging– Tight deadline

• ITEA review in Helsinki, all PCs shipped

– Not the SEP approach


24


25

Retrieval client - Issues

• Testing difficult without a filled database– Data was recorded one week before shipping

• Metadata stored in multiple tables– Very complex SQL queries– Combined selections might not make sense

• Integration of existing and changing parts– GUI for video playback from a database– GUI and engine for trajectory search

• Synchronization of video and metadata– Date&time in MySQL, PostgreSQL, ODBC, Qt


26

Retrieval client – video playback

GOV buffer 1 GOV buffer 2

Bounding box metadata


27

Demo


28

Questions?

Documents

3 May, 2015 Richard Verhoeven, [email protected] TU/e Computer Science, System Architecture and Networking 1 Richard Verhoeven The Candela Project