View
214
Download
0
Category
Tags:
Preview:
Citation preview
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
1
Richard Verhoeven
The Candela ProjectBosch demonstrator perspective
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
2
Outline
• Introduction• Content Analysis• Networked DELivery• Architectures• Bosch demonstrator• Demo
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
3
Introduction
• ITEA project 02013• Content Analysis and Networked DELivery
Architectures• Partners– Philips, TU/e, Bosch (NL)– Multitel, IT-Optics, VUB,
Traficon (BE)– Hantro, Solid, VTT (FI)– ESI, Amena, Ibermatica,
CanalMicro (ES)
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
4
Introduction
• Focus– Home Multimedia– Surveillance– Medical systems
• Issues– Large multimedia collections– Distributed storage
• Solution– Automatic metadata extraction– Content delivery framework
Acquisition
CompressionContentAnalysis
Search & Retrieve
Deliver
Present
Contentdatabase
Metadatadatabase
MPEG7MPEG2/4
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
5
Introduction
User Interface
Cable
WWW
AV
Content & descriptors
Content Management
HDD
Database
MPEG7Storage
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
6
Content Analysis
• AV segmentation– Chaptering of video content
• Object tracking– Traffic movement, suspicious behavior
• Idle object detection– Airport and public transport surveillance
• Face detection & recognition– Photo & video content management
• Sports analysis– Statistics, improved video compression
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
7
Content Analysis
• Voice recognition– Video content management
• Optical character recognition (OCR)– License plates, news tickers, prints on trucks
• Classification– Traffic type based on speed and size
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
8
Content Analysis - Issues
• Light conditions– Shadow, clouds, head lights, back light
• Merging objects– At traffic light, near crossings
• Processing intensive– Different & variable frame rates
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
9
Network delivery
• Content standardization– Video content – MPEG4– Metadata – MPEG7
• Control protocol– SOAP– UPnP
• Distribution and load balancing• Reliability and robustness
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
10
Network delivery
• Database storage– Allows persistent storage of features– Allows nonlinear features browsing and
searching– Allows non real-time features processing– Enables features distribution using metadata
standards– Enables new applications– Serves as a reference to evaluate new algorithms
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
11
Network delivery
video1
VCA MDA
VCA DB
VCA
ACA
VCA
video 2
audio
Home multimediademonstrator
Networkedinfrastructure
Mobilephone
ACQ VCA DB
VCA
Medical systemsdemonstrator
ACQ
VCA MDA
VCA DB
VCA
alarm
Surveillance demonstrator
retrieval
DBXML/XLS
XHTMLpresent
Videostreaming
Interoperability of media and devices:
eTourism
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
12
Architectures
• Centralized & static (Bosch)– Content analysis modules known in advance– Connections between modules fixed– Central controller to manage data flows– Not easy to configure, not very robust
• Distributed & dynamic (Philips)– Content analysis modules can be added– Connections created on demand– Flexible and robust – a.k.a. Cassandra platform
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
13
Bosch Demonstrator
• Content analysis– Acquisition (Bosch)– Object tracking (Bosch)– Metadata analysis (Bosch)– Face recognition (TU/e, VCA group)– Idle object detection (Multitel)– Alarming (Bosch, Solid)
• Network delivery – Controller & Middleware (Multitel)– MPEG7 (VUB)
• Application / Architecture– Database (Bosch)– Retrieval client (TU/e, SAN group)
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
14
Bosch Demonstrator
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
15
Middleware
• Communication transparency• Video & metadata router (controller) in
star-structure• Simple connect(), getData(), and setData() functions
• Decoding of MPEG-4 under API• Buffering in case of computational
overflow• Implementation for Windows and Linux
PCs
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
16
Middleware
ControllerACQ
Track MDA
Idle Face
DB
MPEG4
MPEG7
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
17
VCA Modules
• Acquisition– Live video grabbing with real-time
compression – MPEG-4 (colour, CIF res., 1 MBit, 12.5 fps)
• Object tracking– Algorithm provided by BOSCH
• Shape and bounding box of moving objects• Assignment of object IDs• Sensitivity difficult to control
– Shape information ignored– Filter on short-term objects– Thumbnail selection
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
18
VCA Modules
• Metadata analysis– Analyse bounding boxes from object tracking– Calculate size and speed based on camera info – Classify object based on size
• Truck/bus, car, person/other
– Extract trajectories from center of bounding boxes
– Create events for certain trajectories• Turn left, turn right
– Future extension• Color analysis in thumbnail (yellow car, blue truck)
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
19
VCA Modules
• Face recognition– Provided by Fei Zuo from VCA group– Based on neural networks– 2 - 4 fps– Difficult to combine with
other VCA modules
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
20
VCA Modules
• Idle object detection– Provided by Multitel– Changes with respect to background image
• Database storage– Video: MySQL database with GOVs (16
frames)– Metadata: PostgreSQL database with
OpenGIS for trajectory• Alarming– Receive metadata from face recognition– Inform mobile terminal of recognized person– Display video on mobile terminal
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
21
Integration issues
• Windows & Linux– Middleware library cross-platform– MPEG7 library platform specific
• Memory leaks in middleware libraries– Unclear who had to free memory– MPEG7 data is in XML
• Stability of the system– Endurance test– Fill the database with useful content
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
22
Integration issues
• Different frame rates and buffering– Face recognition at 2 fps, rest at 12.5 fps
• Variable frame rate once a face is recognized
– Middleware doesn’t provide buffer info– Possible fix: separate thread to retrieve frames– Used fix: check delay in frame retrieval
• Video codecs and formats– YUV vs RGB vs RBG
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
23
Retrieval client
• Search through metadata and watch video• Most visible part of the demonstrator– Only way to show the content of the database
• Ad-hoc application– Sketch on one A4 and a list of nice features– Integration of work-in-progress parts
• Trajectory search by Rob Albers
– Debugging– Tight deadline
• ITEA review in Helsinki, all PCs shipped
– Not the SEP approach
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
24
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
25
Retrieval client - Issues
• Testing difficult without a filled database– Data was recorded one week before shipping
• Metadata stored in multiple tables– Very complex SQL queries– Combined selections might not make sense
• Integration of existing and changing parts– GUI for video playback from a database– GUI and engine for trajectory search
• Synchronization of video and metadata– Date&time in MySQL, PostgreSQL, ODBC, Qt
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
26
Retrieval client – video playback
GOV buffer 1 GOV buffer 2
Bounding box metadata
April 18, 2023 Richard Verhoeven, P.H.F.M.Verhoeven@tue.nlTU/e Computer Science, System Architecture and Networking
27
Demo
Recommended