Embed Size (px)
Citation preview
Watskeburt?! Onder de motorkap
Technology Update 6 oktober, 2016
Founded by Geert Vos and Joost de Wit in 2014
We still can’t search in video content!
Adding metadata used to be manual labour
We use advanced data mining to analyse video content
Photo courtesy of Google
Both broadcast & on-line sources
Teletext Speech Subtitles Logos Faces Filmstrips
Ingest
Analysis
Storage
Presentation / interaction
API
cask still well
foundation
Large Scale Video Storage
Online Video Service
StorageService
RSS feed Video scraper
RabbitMQ
SpeechRecognizer
SubtitleExtractor Filmstrip Creator …
individual frames chunked videoannotations
well still cask foundation
SpeechRecognizer
SubtitleExtractor Filmstrip Creator …
RabbitMQ Indexer Elasticsearch
ContentService
Notification Service
SearchService
AuthenticationService
FeedService
well still cask foundation
annotations Large Scale Video Storage
Large Vocabulary Automatic Speech Recognition (ASR)
Photo courtesy of IBM
r eh k ao g n ay z s p iy ch
"recognize speech"
Speech / non-speech
Speaker diarisation
Phonemes (acoustic model)
Vocabulary (language model)
Dialect de NS > Dennis
SjoemelsoftwarePhone calls
Subtitle extraction
ROI selection
Text extraction
OCR
Utterance detection
Garbage detection
Logo recognition
Voor het kijken Tijdens het kijken
While browsingWhile watching
Research questions
• How to “visually summarise” a video clip as a filmstrip? • #frames to show (fixed / variable)? • Which frames to show? • Size of the frames to show (fixed / variable)? • Part of the frame to show? • How to present the filmstrip? • How should users interact with it?
Frame sampling
Shot detection
Frame selection
Merge
[Bar clipping]
Videoclip
Filmstrip & manifest
Features for frame/bar selection
• Gezichten (en de beweging ervan over het scherm)
• Gezichtsuitdrukking • Open ogen • Tekst • Infographics • Scene-overgangen • Scherpte in het beeld • Rule-of-thirds • Aanwezigheid van muziek / spraak • Ondertiteling
• Visuele eigenschappen frame (sharpness, saturation, kleurhistogram)
• Programma specifiek (studio, naambordjes, …)
Filmstrip as (eventually) tested
• Variable number of frames (one per shot) • Selected frame just after shot changed • Fixed width, no clipping • Focused on the presentation & interaction
Examples
• Short clip • One long shot taken from a helicopter • No voice-over or text present
Examples
• The subtitling tells the story • Selected images don’t contain much information
Examples
• Great example • Reads like a comic book • Lucky shot
Alternative filmstrip
• Variable number of frames • “bar” width based on length of shot • “bar” cut with respect to frame’s center
Storing the filmstrip
• We store parts of the filmstrip up to a maximum length separately • Parts are stored in Cassandra • Efficient retrieval based on program ID and start time of the strip • Scalability and redundancy are build-in in Cassandra
Part 1
Part 2
The API
• REST API (created using Jersey, running in Apache Tomcat) • JSON & JPG as output
{ "height" : 480, // The height of the filmstrip "strips" : [ { "program_id" : "b214753f-5e17-46ec-afeb-c1b3d9ee6565", // The Id of the program from which // the filmstrip was created "start" : 0, // Start time of the strip (in milliseconds) "stop" : 12000, // Stop time of the strip (in milliseconds) "url" : "<base>/strip_001.jpg", // The URL of this strip's image "width" : 6700, // The width of this strip "segments" : [ // Segments are the tiles in the strip { "start_offset" : 0, // The start offset of this segment (in pixels) "stop_offset" : 100, // The stop offset of this segment (in pixels) "start_timestamp" : 0, // The start time of this segment (in milliseconds) "stop_timestamp" : 1234 // The stop time of this segment (in milliseconds) }, { "start_offset" : 101, "stop_offset" : 305, "start_timestamp" : 1235, "stop_timestamp" : 2345 } ] } ], "previous" : false, // A reference to the previous filmstrip file (when present) "next" : "<base>/next_strip.json" // A reference to the next filmstrip file (when present) }
/api/{version}/filmstrip/{program_id}
Flexibility of the API (for other applications)
Parameter Required Description Defaultprogram_id yes The id of the program for which the filmstrip should
be returned-
start no The start time (in milliseconds) in the video for which the filmstrip is constructed
0
max_duration no The maximum duration (from {start}) of the video covered by the filmstrip
-1 (complete clip)
max_width no The maximum width (in pixels) of one single filmstrip
15000
size no The size of the filmstrip, either small, medium or large
small
/api/{version}/filmstrip/{program_id}/strip
UI challenges
• Cross platform & responsive (is a hassle) • We had to ‘fold’ the frames to shorten the strip • Interaction between the strip and the player
• Move strip while watching? • Interact with strip while watching?
• Equally sized frames represent different shot durations
• Implementing ‘the wiggle’ was challenging, but required to stimulate swiping
• IOS only plays video in fullscreen mode
The filmstrip UI
• Runs in the (mobile) browser • Responsive design (using Bootstrap) • Heavily using jQuery + libraries
• jQuery UI • jQuery Mobile (Events) • Kinetic • SmoothDivScroll • Handlebars
• Thin backend that’s basically a proxy to the Media Distillery platform
Any questions?
