Efficient multimediasupport in QtWebKit

on Raspberry Pi

Talk outline

• Raspberry Pi

• WebKit architecture(s) with emphasis on Media playback

• Platform-level improvements for QtWebKit

Raspberry Pi - Model B

• 700MHz single-core ARMv6 CPU• Broadcom BCM2835 VideoCore IV• 512 MB RAM (partly allocated for GPU)

GStreamer OpenMax (gst-omx)

• Hardware decoders for H.264, VP8, Theora

• meta:EGLImage

• Custom audio sinks: HDMI and analog

• Very good integration within playbin

Premises of the project

• Fullscreen browser, no tabs, no WM

• Focus on media consumption web-apps and Youtube TV

• OpenGL for rendering

• Qt5 platform, EGLFS QPA

WebKit rendering - from HTML to the screen

WebKit rendering - the forest

WebKit1 rendering (with OpenGL)

• GraphicsLayers are backed by GL textures

• The video player has its own GraphicsLayer

• RenderLayers are traversed and painted recursively (to their GraphicsLayers): children below -> self -> children above

• Composition: GraphicsLayers are stacked to produce the final result

WebKit2 rendering (OpenGL) - processes

• Rendering split into 2 processes: UI and Web

• Web process renders the layers (similar to WebKit1 rendering)

• UI process composites and shows them

• GraphicsLayers shared using GraphicsSurfaces

• A GraphicsSurface encapsulates a method to share gl textures among processes (EGLImages)

WebKit2 rendering (OpenGL) - Coordinated graphics

The MediaPlayer in WebCore

MediaPlayer - platform level

Video rendering - the inefficient way

• Sink negotiates xRGB or ARGB video caps

• Optional buffer copy in ::render, conversion to Cairo's ARGB pre-multiplied alpha

• Buffer data copy in MediaPlayerPrivate::paint to a WebCore::BitmapImage

• BitmapImage rendered in a QPixmap

• Another approach: GLTextureUpload meta (CPU -> GPU buffer copy)

WebKit2 video rendering (inefficient)

Video sink - improved integration withinthe platform

• Sink negotiates meta:EGLImage with upstream video decoder

• Buffer pool and EGLImage allocator (like in glimagesink)

• EGLImage encapsulated in a GstBuffer passed to MediaPlayerPrivate

• Efficient rendering in the player, different approaches for WebKit1 and WebKit2

Improvements in WebKit2 rendering

Improving media resources loading

Using libsoup within QtWebKit

• The Qt-based resource loader doesn't cope well with large media resources

• The libsoup-based resource loader is more memory efficient and better integrated in WebCore

• Pre-allocation of GstBuffers within our HTTP source element

WebRTC / getUserMedia

Local devices listing/playback

• In JS: webkitGetUserMedia({audio: true}, successCallback, failureCallback);

• function successCallback(stream) { video.src = URL.createObjectURL(stream); video.play(); }

• Pipeline should also allow encoding and RTP payloading if required by webapp (WebRTC PeerConnection)

WebRTC - RPiCam integration

• rpicamsrc: emits H.264 byte stream

• Leverage H.264 hardware rendering

• rpicamsrc ! h264parse ! omxh264dec ! sink

• with a tee downstream rpicamsrc to allow encoding/streaming later on

WebAudio

WebAudio - basics

• JS API for processing and synthesizing audio in web applications

• Pipeline based. Examples of nodes: BufferSource, GainNode, BiQuadFilter, Destination...

• Head-related transfer function: tricking the human ear to spatial sound

WebAudio - decoding incoming files

• First scenario: local file. Leverage decodebin and appsink to create an in-memory representation of the audio file PCM data (FloatArray).

• Second scenario: memory buffer. Decodebin again! With giostreamsrc

• No RPi-specific optimisations

WebAudio - playback

• Playback of WebAudio FloatArrays PCM samples to soundcard

• Custom GStreamer source element encoding internal WebCore audio buffers to WAV

• Again, no RPi-specific optimisations

So which part of the backend needed optimisations?

WebAudio - HRTF Database

• Loaded at AudioContext creation, from a separate thread

• 240 very small (256 frames) WAV files => lots of context switches

• Improvement: concatenate all samples and internally split when building the database

• => only one file to read, less context switches!

WebAudio - FFT experiments

• default WebAudio FFTFrame backend using GstFFT (itself based on KissFFT)

• math functions hot on perf profiles

• GPUFFT library (from rpi-userland): leveraging VideoCore for FFT calculations

• Major drawback: requires allocated RAM, so less RAM for the other libraries

Next tasks

• video/canvas and video/webgl integration

• WebRTC

• ORC ARMv6 backend?

Demo rendering from youtube at 720p

Demo rendering from youtube at 1080p

Flickr pictures

• http://bit.ly/1p8YHJA

• http://bit.ly/1yz8ctT

• http://bit.ly/1v8nwLh

This is the last slide, let's have lunch now!