Embed Size (px)
DESCRIPTION
By Xabier Rodríguez Calvar. WebKit is a well known open source browser engine used by Apple, GNOME, EFL, Blackberry and others and shares quite a big codebase with Blink. It was the result of a fork from KHTML and KJS. As a simplification we can say that it has three main layers, the API, the core and the backend. The different ports implement their API and their backends and the core is shared. WebKit 2 is robust and transparent multiprocess architecture allowing safer interactions with webpages. GStreamer is the multimedia backend of some WebKit ports and in this talk we will see its architecture, status, challenges and future.
Citation preview
WebKit and GStreamerBringing lolcatz videos to your desktop
Xabier Rodríguez CalvarIgalia
Who is this dude?
Born in the Basque CountryGrown and living in Galicia
Studied Computer Science at Universidade da CoruñaPartner and developer at Igalia
··
One of the seven Celtic Nations, like Scotland-··
Working in Multimedia and WebKit-
Index
Quick review of WebKitWebKit MultimediaPower features
···
WebAudioHardware accelerationAccelerated video compositing
---
5/36
Quick Review
WebKit
Web content engineProcesses and renders web contentStarted as a fork of KTHML and KJSOpen source since 2005Goals: open source, compatibility, compliance, stability,performance, security, portability, usability, hackabilityNon-goals: being a full web browser, being a science project,having reusable components, unlimited scopeForked by Google as BlinkSplit into ports: GTK+, Qt, EFL, Mac, Chromium, Windows
·····
·
··
7/36
Architecture
8/36
WebKitGTK+
Each WebKit port is composed of
WebKitGTK+ platform layer:
API layer is a GtkWidget and a set of GObject APIsWebKitGTK+ is used by Epiphany, Midori, yelp, devhelp
·Platform interfacesAPI layer
--
·libsoup for networkingcairo for rasterizationOpenGL for making the scene graph and WebGLGStreamer for mediaVarious GTK+ APIs for talking with the system
-----
··
9/36
WebKit2
Give the web rendering parts of WebKit their own process
Sandbox web rendering
·Page crashes don't crash the browserCan put vulnerable data into a separate address space
··
·Prevent pages from accessing the disk and operating systeminterface
·
10/36
WebKit2 Architecture
11/36
The Near Future
12/36
WebKit Multimedia
WebKit Multimedia
Mac port: QuickTime, QtKit, AVFoundationsApple Win port: QuickTimeGTK+ and EFL ports: GStreamer 1.0 backendBlackberry port: Bridges over to its own Media playback frameworkChromium: bridges over to WebKit/Chromium backend whichrelays to FFMPeg implementationsQt port: GStreamer on Linux, QtKit on Mac, QuickTime onWindows
·····
·
14/36
WebKit Multimedia Architecture
WebKitWebCore
MediaPlayer
Platform- - - - - - - - - - - -
GStreamer15/36
GStreamer MediaPlayer - Overview
Mainly developed and tested on Linux platformSuccessfully ported to MacOS/Win platfoms (not fully upstreamyet though)Playbin-based: the best GStreamer element for playback use-cases
MediaPlayer registers and uses two WebKit-specific GStreamerelements: webkitwebsrc and a video-sink
··
·
Playbin2 with GStreamer 0.10-·
16/36
GStreamer MediaPlayer - Main Features
On-disk-bufferingFrame-accurate seekingBasic metrics reportingTrick-modesWebAudioHardware decoding support with VA-API (gst 1.2.x)Video accelerated compositingCodec installer support
········
17/36
WebKit and GStreamer Architecture
-mediaPlayer
MediaPlayerClient
+playbackChanged()+volumeChanged()
-mediaPlayerClient
-m_privateHTMLMediaElement+play()+playbackChanged()+setVolume(in volume:float)+volumeChanged()
MediaPlayer+play()+setVolume(in volume:float)
GstPlaybin -playbinHTMLVideoElement HTMLAudioElement
MediaPlayerPrivateInterface
+play()+setVolume(in volume:float)
MediaPlayerPrivateGStreamerBase
+setVolume(in volume:float)
MediaPlayerPrivateGStreamer+play()
18/36
GStreamer MediaPlayer - HTTP source element
a GStreamer bin implementing the URI handler interface forhttp(s):// and blob:// uris.A single source pad provides raw data access to downstreamelements.appsrc is used internally to interface with the WebCoreSubResourceLoader objectElement is registered with high rank in the GStreamer registry soplaybin is more prone to pick it up to read http:// URIs.
·
·
·
·
19/36
GStreamer MediaPlayer - Video rendering
Element inheriting from GstVideoSink
Takes video GstBuffers as inputPass those to the MediaPlayerPrivateGStreamer via a signal
·In practical sense, it is a GstAppSink with video capabilitiesnegotiation
-
··
20/36
GStreamer MediaPlayer - Audio rendering
Use autoaudiosinkCheck for pulseaudio for playback request messagesPitch preservationPreserve audio passthrough when possibleVolume managed by the sink
·····
21/36
GStreamer MediaPlayer - Next steps
Improvements and bug fixes in the player:
MediaStream (getUserMedia) and WebRTC spec implementationMediaSource adaptive streaming API spec implementationIntegration of WebAudio with media elements and mediastreamstackFinish porting to 1.2Remove GStreamer 0.10 code path
·Store on-disk buffered media in WebCore's cache directoryMake the webkitwebsrc element private to WebKit
--
···
··
22/36
WebKit Multimedia Power Features
WebAudio
A JavaScript API allowing to create a new generation of Webgames and applicationsEffects, convolution engine for room simulation, etcRealtime analysis, playback and visualizationW3C spec driven and implemented mainly by Chromium andFirefoxIn WebKit, currently implemented for:
·
···
·MacChromiumGStreamer based Ports: GTK+, EFL, Qt
---
24/36
WebAudio GStreamer - Reading Audio
Spec allows two ways to read audio data:
Pipeline: src ! decodebin2 ! audioconvert ! audioresample !capsfilter ! deinterleave ! queue ! appsinkdeinterleave separates multi-channel audio to n audio monochannels (Pipeline above can contain multiple queue ! appsinkchains)appsink used to collect mono audio buffers, store them in a list andconvert data to floating point valuesData is stored in a WebCore::AudioBus, containing one or multiplechannel Float arrays
·from memory: we can use giostreamsrc GStreamer elementfrom a file path: we can use the filesrc GStremer element
--
·
·
·
·
25/36
WebAudio GStreamer - Playing Audio
The WebAudio stack processed or generated some audio data to besent to the sound cardSimilarly to the "Reading Data" step, a WebCore::AudioBus is usedto store the dataCustom GStreamer element to convert an AudioBus to aninterleaved audio data flow: webkitwebaudiosrcwebkitwebaudiosrc: a bin with n internal pads pushing to aninterleave element. Source pad outputs a single, interleave, wav-encoded data flowPipeline: webkitwebaudiosrc ! wavparse ! audioconvert !autoaudiosink
·
·
·
·
·
26/36
WebAudio GStreamer - Work in Progress
Some layout tests don't pass yetNot yet enabled by default in release buildsSupport for stereo+ streams in the reader side. Playback sidealready handles those.Integration with <video> and <audio> elements: WebAudioMediaElementSourceNode
···
·
27/36
Hardware Accelerated Video
We use GStreamer VA-API. (1.2.x)That's it! Hardcore comes when compositing that.
··
28/36
Accelerated Video Compositing - Acceleration inWebKit
From Joone Hur
29/36
Accelerated Video Compositing
Element inheriting from GstVideoSink
Takes video GstBuffers as input.Video frame is transferred to the MediaPlayerPrivateGStreamer viaa GLib signal.MediaPlayerPrivateGStreamer uploads it to a GL texture andrenders it asynchronously in the WebKit context.
·In practical sense, it is a GstAppSink with video capabilitiesnegotiation
-
··
·
30/36
Accelerated Video Compositing - Architecture
GstPlaybin -playbin
MediaPlayerPrivateInterface
MediaPlayerPrivateGStreamerBase
+paint()
MediaPlayerPrivateGStreamer
GstElement*
WebKitVideoSink <<creates>>
TextureMapperPlatformLayer+paint()
...
31/36
Accelerated Video Compositing - Rendering paths
Software rendering path
Accelerated compositing paths
·It uses Cairo-
·TextureMapperLayer:
TextureMapperGL
-Light-weight implementation of a scene-graphThe TextureMapperLayer tree is equivalent to theGraphicsLayer tree.
--
-Provides a GPU accelerated implementation of thedrawing primitives, using shaders compatible with GL/ES2.0
-
32/36
Accelerated Video Compositing - Next steps
More video decoders with GstVideoGLTextureUploadMeta supportSupport EGLImage in WebKit as video frames handlers
Handle the vertical blank interval in WebKit
··
And video decoders that deliver this kind of capabilities.-·
GTK+ 3.8 seems to fix this issue.-
33/36
Thank You!(q&a)
twitter @calvariswww blogs.igalia.com/xrcalvar
