Integrating media playback on embedded devices. DTG HTML5
workshop, London, 3 rd July 2015
Slide 2
Who am I? [email protected] uk.linkedin.com/in/asrashley
github.com/asrashley
Slide 3
Page composition
Slide 4
The user sees
Slide 5
The browser is compositing
Slide 6
The hardware is compositing
Slide 7
Typical QtWebKit Media Pipeline
Slide 8
Our Device Media Pipeline
Slide 9
One decoder === one video element Or does it?
Slide 10
Mapping to HTML media elements Direct mapping between elements
and hardware resources is problematic Removing a video element from
the DOM doesnt instantly clean up its resources Application might
want to preload a video while another is playing
Slide 11
Removing a video element $(vidElt).off(); /* unbind all event
listeners */ vidElt.pause(); vidElt.src=''; /* remove the source */
try { delete vidElt.src; } catch(e){ } vidElt.load(); /* force
source change */ /* remove from DOM */ $(vidElt).remove();
Slide 12
Back-to-back videos Use cases Pre-roll branding idents Advert
insertion Seamless start of next episode Options Removing one video
before loading next Changing source URL of video element Media
Source Extension Multiple video elements
Slide 13
HbbTV 2.0 multiple video elements The terminal shall support at
least one HTML5 media element that is playing together with at
least two HTML5 media elements in a paused state The terminal may
use hardware audio and video decoders to decode and render and
HTML5 media elements.
Slide 14
Multiple video elements, single decoder Main video HTML
application Advert h/w decoder h/w
Slide 15
Slide 16
Video resizing Not all platforms support moving / resizing
video elements There might be hardware limitations on accuracy
Granularity of 4, 8, 16 pixels Probably a good idea to avoid
animated moves or resizes Sync between video and graphics planes
might not be perfect Might trip over race conditions in underlying
implementation
Slide 17
Position reporting Ad insertion needs accurate reporting of
playback position To find the correct point to pause main content
HTML position reporting needs to match position of video on screen
From hardware presentation timestamp Needs to work for live streams
Where timecodes dont start at zero, but HTML5 requires that they
do
Slide 18
Seek For live streams, need to accurately report seekable range
Needs support from HLS/DASH/PDL implementation HTML5 applications
can be quite challenging in their use of trick play Overlapping
seeks Multiple pause/play transitions during seek Seek to end while
paused Difficult to create test harnesses
Slide 19
Other media events stalled Binding to FIFO empty signal from
hardware error Network errors Codec errors DRM errors Metadata
events loadedmetadata canplay durationchange resize
Slide 20
Slide 21
Recommendations Have only one media element in the playing
state There might be only one hardware decoder Avoid using opacity
on a video element Graphics under video might not be supported
Avoid video inside canvas elements Platform might not allow access
to decoded video Clean up after yourself Media elements consume
scarce resources Keep it simple when moving/resizing video elements
Try to avoid lots of state changes during seek If possible, store
up the changes until the seeked event