WebKit, HTML5 mediaand GStreamer onmultiple platforms
Spreading GStreamerawesome in WebKit
Philippe Normand
Talk outlineAbout WebKitHTML5 Media playback in WebKit withGStreamerWebKit/GStreamer: opening the door to moreplatformsEnd goal: spread the usage of GStreameron more and more WebKit-basedapplications!
About WebKitA content rendering/editing engineNot a browser!Highly standards-compliant
WebKit EcosystemApple, Nokia, Google, RIM, Adobe, Sony-EriccsonSamsung, Palm, Motorola, Electronic Arts, ...=> lots of devices using WebKit in the market=> wide range of applications using thevarious WebKit ports
High level WebKitArchitecture
JavaScriptCore: Javascript engine andplatform-specific bitsWebCore: platform-independant blocks andtheir counter part platform implementationsfor:
parsing, layoutingnetwork, paintingmedia playback...
WebKit: platform-specific APIs => used byapplication developersPorts: GTK+, Qt, EFL, Chromium, Mac,WinCairo, Apple Windows, ...
HTML5 Mediaplayback in WebKit
3 layers: DOM/HTML level(HTMLMediaElement), WebCore/graphics(MediaPlayer), platform-dependantimplementation (MediaPlayerPrivate)Some MediaPlayerPrivate implementations:
GStreamer: GTK+ and EFL portsQt/Phonon: Qt portQuicktime: Apple Mac/Win ports
GStreamer mediaplayer
playbin2 of course!Custom video sink and source elementsBasic trick-modes supportOn-disk bufferingFullscreen video display
The GStreamer playerin other WebKit ports?
GStreamer cross-platform already :)
the MediaPlayer layer allows multiple Privateinstances
choice based on media mime-type andcodecs:
<video src=test.ogv type="video/ogg; codecs="theora,vorbis&qu
Targets: Mac and WinCairo ports
Porting check-listGStreamer platform packagesGLib main context integrationwebkit src elementwebkit video sinkFullscreen supportWebKit build :)
GStreamer platformpackages
on Windows: OSSBuildon Mac OSX: OSSBuild too! soon!For now on Mac: scripts developed based ongtk-osx-build
GStreamer build onMac
jhbuild to the rescueadd -headerpad_max_install_names toLDFLAGSuse install_name_tool to change theshared library paths in the various .dylib filesabsolute paths -> relative paths to@loader_path
GLib main contextintegration
GstBus signal watch used in the player =>need periodic calls ofg_main_context_iteration()on Mac: NSRunLoop observer triggering theglib main context iterationson Windows: integration in the message loop
WebKit src elementBased on appsrc, work by Sebastian Dröge!using WebCore Resource loader (=> HTTPsession data: cookies, referer)Contains some platform-specific code topause/resume data downloadFor now: #ifdefs :( Better approach: useWebCore resource defersLoading API.
Video paintingvideo sink exports buffers to the player("repaint-requested" signal)ImageGStreamer: GstBuffer ->WebCore::Imageactual painting deferred to theImageGStreamer implementations:
ImageGStreamerCairo: used by GTK+ andWinCairo portsImageGStreamerCG: CoreGraphicsimplementation for the Mac port
Fullscreen support(Overview)
webkitEnterFullscreen() VideoElement DOM API=> WebKit WebView private API=> => FullscreenVideoController
enterFullscreen() hooks intoMediaPlayer::platformMedia()MediaPlayerPrivate-specifics: to create awidget where the video is paintedController gets the widget, adds controls,goes fullscreen
Fullscreen support(GStreamer side)
Our playbin2 video-sink is a bin!
fullscreen off:
tee name=t t. ! queue ! webkitvideosink
fullscreen on:
tee name=t t. ! queue ! webkitvideosink t. ! queue !ffmpegcolorspace ! videoscale ! autovideosink
Fullscreen support(Video overlay)
GstXOverlayPlatformVideoWindow : window on whichautovideosink overlays the video
PlatformVideoWindowGtkPlatformVideoWindowEFLPlatformVideoWindowCocoa...
WebKit buildmodifications
(note: WTF here means Web TemplateFramework, obviously!)add JavaScriptCore/wtf/gobject/ inyour buildaddWebCore/platform/graphics/gstreamer/in your buildenable WTF_USE_GSTREAMER andENABLE_GLIB_SUPPORT definesport-specific FullscreenVideoControlleradaptations
Implementationstatus : Mac OSX
Most of the patches merged upstreamalready, excepted:
XCode config additionsgst webkit src patchPlatformVideoWindowCocoaimplementationFullscreenVideoController patch
Packaging: WebKit-nightly DMG includingGStreamer and its dependencies.
Implementationstatus : WinCairo
HTML5 media implemented from scratch forWinCairo:
WinCairo still using Visual Studio 5 => bigblocker :(Patches in a separate public git repository
Basic media controlsFullscreen video
Recap!1. GStreamer build2. Periodic calls to
g_main_context_iteration()3. Patching gst webkit src element and/or your
ResourceHandle implementation4. ImageGStreamerMyPort implementation5. PlatformVideoWindowMyPort
implementation6. Patching your WebKit
FullscreenVideoController7. Build WebKit!
Some bit of adviceUse playbin2 :-)Facade design-pattern for your app platform-specific bitsSingle build-system == less maintenanceburden (gyp, cmake, ...)Continuous integration on multiple platforms:BuildBot
DemoWebM playback, in Safari! With GStreamer! :-)
Future plansFinish upstream merge of the patchesWebKit/GStreamer build slavesFullscreen video improvementsWebKitGTK+ API to allow fine-grained controlon the player like:
per-webview volume controlcodec installer
Luke, may the sourcebe with yougst-mac and win-gst WebKit branches:
http://git.igalia.com/cgi-bin/gitweb.cgi?p=webkit.githttp://trac.webkit.org/wiki/GStreamerOnMac
That would be all,folks
Work sponsored by:IgaliaR&D funds from Galicia governmenta private Igalia client
Contact:http://[email protected]: philn-tp on #gstreamer, #webkit and#webkit-gtk
Questions?