Song and Dance and TWD using GStreamer Thomas Vander Stichele

Song and Dance and TWDusing GStreamer

Thomas Vander Stichele

Overview

● TWD and USP● TMS● VKC of GST● GPE prototyping● LAP for SAD celebrating FTA● Roadmap

software desktops

● Desktops have targets and philosophies– GNOME: less options, less bloat, HIG– KDE: full configurability– MacOS: eye candy, artistic personality– Windows: applications

● Desktops need a USP for better marketing● After-hours business school applied to

GNOME● TMS done by Korean firm

Thorough Market Study

GNOME KDE Mac Windows Other0

10

20

30

40

50

60

70

80

90

100

Which desktop are you currently using ?

which desktop ?

Thorough Market Study

GNOME KDE MacOS Windows0

10

20

30

40

50

60

70

80

90

100

What do you want in your desktop ?

More options

Less bloat

Stability

More appsKaraoke

Thorough Market Study

Yes No0

10

20

30

40

50

60

70

80

90

100

Would you switch to GNOME if it had karaoke ?

well, would you ?

Next year at LWE

Karaoke

● What do we need ?● Why use GStreamer ?● What is GStreamer ?● What needs to be done ?

What do we need ?

● Audio playback● Visualization● Video● Subtitling

Why use GStreamer for it ?

● Lot of elements already present● Framework is there for things you need

– Output– Synchronization– Mixing multiple formats

● Prototyping is easy– gst-launch for quick tests– gst-editor for experimenting with pipelines– Gst-python for testing application structure

GStreamer

● What is it ?● Why is it ?● Why is it not ?● Elements and pads● Graph-based pipelines

What is it ?

● Core: streaming media framework– Arbitrary filter graphs– Extensible pluggable architecture– Data-agnostic, basic elements (GObjects)

● Plug-ins : provide functionality– Audio : filters, codecs, analysis, visuals– Video : filters, codecs, (de)muxers– Sources/Sinks : video/audio/media in/out

What is it ?

● GStreamer is first of all a library– Compare it to GTK

● We provide a few applications– A player as a test bed for features (~ gtk-

demo)– An editor used in debugging (~ glade)

● We provide plug-ins● We provide documentation

Why is it ?

● Linux has bad media handling● many hacked single-purpose apps and libs,

code copies and forks, ...– avifile, Broadcast2000, ...

● other OS's have such a framework– DirectShow, Quicktime framework, Be

Media, RealMedia SDK● LGPL core gives companies the chance to

make binary plug-ins

What is it not ?

● It's NOT a sound server– one could be written with it (maybe)– not important for an app

● It's NOT Gnome-only● It's NOT 100% ABI-stable (partial)● It's NOT perfect

Elements and Pads●element has pads

●chain- or loop-based

●buffer of data comes in on the sink pad

●element processes data

●Processed buffer goes out on the source pad

●This one decodes Ogg/Vorbis audio

Graph-based pipelines

●Pipeline managed by a bin (not visible)

●Links elements by their pads●elements are chain-based or loop-based

●Data will flow left to right

What needs to be done ?

● Experiment– gst-launch– gst-editor

● Write elements● Fix bugs

– In elements– In applications like gst-editor

Let's get to it !

● Audio playback– Guadec-editor 1

● Visualization– Guadec-editor 2

● Subtitling– Guadec-editor 3

Combining the three

● Normal:– Guadec-editor 4

● Threaded:– Guadec-editor 5

Using video footage

● Threaded MPEG playback:– Guadec-editor 6

● With subtitles– -guadec-editor 7

Ideas for improvement

● Use pitchshifter so Stallman can sing● Use festival to replace Stallman

– Play: Guadec-editor 8– Save to vorbis: Guadec-editor 9– Use with subtitles: guadec-editor 10

● Personalise– Find video of friends: guadec-editor 11

Ideas for improvement

● Use live input– Mix in audience audio and video

● Use backing music– Mod files, midi files, ...

● Use color font rendering● Add a bouncing ball plugin

Gkaraoke

● Truth is stranger than fiction

Roadmap

● Actual stuff to work on● What can you do ?

Actual stuff to work on

● Media-info library:– Save and load metadata and streaminfo– Cache it (-> thumbnails)– Playback parameters (mixing, volume)– Shared by all apps

Actual stuff to work on

● Nautilus:– Hoverplay for audio and video– Further audio view improvement

● Encoding profiles– Used by recorder, sound-juicer, ...– Using human-readable names

● Editor:– Educational GStreamer tool

What can you do ?

● Users– Test our applications:

● Gst-totem, gst-player, gst-editor● Rhythmbox, quark, nautilus-media, sound-juicer● Use media files on http://gstreamer.net/media/

– Suggest new applications– Run the upcoming test suite on your computer– Harass developers

What can you do ?

● Developers– Help us fix bugs and implement new features– Document– Write applications– Write/fix elements– Test and use the bindings– Use the tools we provide

● Build scripts● Template tarballs

– Go to Ronald's talk