ID B32L: Graphical Application Development under …...... Graphical Application Development under Linux/Implementing Video ... VLC Media Player ... Command line application that can

1

Renesas Electronics America Inc.

© 2010 Renesas Electronics America Inc. All rights reserved.

ID B32L: Graphical Application Developmentunder Linux/Implementing VideoSolutions using G-Streamer

Chris Brandt

13 October 2010

Version: 1.0

Staff Application Engineer

2

2 © 2010 Renesas Electronics America Inc. All rights reserved.

Chris Brandt

� Applications Engineering

� Currently supporting SH4A and SH-Mobile

Linux applications (among other things)

� Worked for Renesas/Mitsubishi 12 years (I’ve

done a little of everything in that time)

3


Renesas Technology and Solution Portfolio

Microcontrollers

& Microprocessors#1 Market share

worldwide *

Analog and

Power Devices#1 Market share

in low-voltage

MOSFET**

Solutions

for

Innovation

Solutions

for

InnovationASIC, ASSP

& MemoryAdvanced and

proven technologies

* MCU: 31% revenue basis from Gartner

"Semiconductor

Applications Worldwide

Annual Market Share:

Database" 25

March 2010

** Power MOSFET: 17.1%

on unit basis from

Marketing Eye 2009

(17.1% on unit basis).

In the session 110C, Renesas Next Generation Microcontroller and Microprocessor Technology

Roadmap, Ritesh Tyagi introduces this high level image of where the Renesas Products fit. The big picture.

4


4

Renesas Technology and Solution Portfolio

Microcontrollers

& Microprocessors#1 Market share

worldwide *

Analog and

Power Devices#1 Market share

in low-voltage

MOSFET**

ASIC, ASSP

& MemoryAdvanced and

proven technologies

* MCU: 31% revenue basis from Gartner

"Semiconductor

Applications Worldwide

Annual Market Share:

Database" 25

March 2010

** Power MOSFET: 17.1%

on unit basis from

Marketing Eye 2009

(17.1% on unit basis).

Solutions

for

Innovation

Solutions

for

Innovation

This is where our session, B32L is focused within the ‘Big picture of Renesas Products’, Microcontroller

and Microprocessors.

5


5

Microcontroller and Microprocessor Line-up

Superscalar, MMU, Multimedia� Up to 1200 DMIPS, 45, 65 & 90nm process

� Video and audio processing on Linux

� Server, Industrial & Automotive

� Up to 500 DMIPS, 150 & 90nm process

� 600uA/MHz, 1.5 uA standby

� Medical, Automotive & Industrial

� Legacy Cores

� Next-generation migration to RX

High Performance CPU, FPU, DSC

Embedded Security

� Up to 10 DMIPS, 130nm process

� 350 uA/MHz, 1uA standby

� Capacitive touch


� 190 uA/MHz, 0.3uA standby

� Application-specific integration

� Up to 25 DMIPS, 180, 90nm process

� 1mA/MHz, 100uA standby

� Crypto engine, Hardware security



� Ethernet, CAN, USB, Motor Control, TFT Display

High Performance CPU, Low Power

Ultra Low PowerGeneral Purpose

Here are the MCU and MPU Product Lines, I am not going to cover any specific information on these families, but rather I want to show you where this session is focused

6


6

Microcontroller and Microprocessor Line-up

Superscalar, MMU, Multimedia� Up to 1200 DMIPS, 45, 65 & 90nm process

� Video and audio processing on Linux

� Server, Industrial & Automotive

� Up to 500 DMIPS, 150 & 90nm process


� Medical, Automotive & Industrial

� Legacy Cores

� Next-generation migration to RX

High Performance CPU, FPU, DSC

Embedded Security


� 350 uA/MHz, 1uA standby

� Capacitive touch


� 190 uA/MHz, 0.3uA standby

� Application-specific integration

� Up to 25 DMIPS, 180, 90nm process

� 1mA/MHz, 100uA standby

� Crypto engine, Hardware security



� Ethernet, CAN, USB, Motor Control, TFT Display

High Performance CPU, Low Power

Ultra Low PowerGeneral Purpose

SuperH

Notes for Devcon Positioning Slide: There’s a lot of vital information on this slide, which spotlights

the Renesas MCU/MPU product lines recommended for new designs. Perhaps the best way to discuss this material is to cover it from a very high level.

Since the merger, we have scrutinized the needs of our global markets, reassessed our strengths, and

implemented a business strategy focusing on supporting the ‘ubiquitous computing’ paradigm. This insightful concept — often abbreviated as ‘ubicomp’, and sometimes termed ‘pervasive computing’ or ‘ambient intelligence’ — was introduced by Mark Weiser of Xerox in 1988.

Ubiquitous computing refers to a new genre of computing, a worldwide electronic environment in which computer-controlled products completely permeate the life of end users around the globe. Obviously,

many types of products and an enormous range of applications are encompassed by this paradigm, all driven by human ingenuity, engineering creativity and marketing expertise. To one extent or another, people everywhere are already beginning to enjoy the first wave of benefits of the concept’s reality.

7


Innovation

Mobile Media is everywhere…and it’s not going away any time soon.

The perception in the market place is that embedded Ethernet is hard to achieve. The cost of TCP/IP stacks, development tools, and the time to learn the technology all make it very difficult to design

products that talk to each other over Ethernet. However, with a low cost Ethernet enabled MCU and the software freely available in the Open Source community embedded Ethernet is very easy to achieve. Renesas has a complete set of solution for H8S/2472 Ethernet connectivity.

8


Renesas Multimedia solutions

Renesas has hardware & software solutions to increase

performance and decrease development time.

The perception in the market place is that embedded Ethernet is hard to achieve. The cost of TCP/IP stacks, development tools, and the time to learn the technology all make it very difficult to design

products that talk to each other over Ethernet. However, with a low cost Ethernet enabled MCU and the software freely available in the Open Source community embedded Ethernet is very easy to achieve. Renesas has a complete set of solution for H8S/2472 Ethernet connectivity.

9


Agenda

� Introduction of GStreamer software

� Features / Usage / Pros / Cons

� Lab Overview

� Q&A

10


What is GStreamer?

� What is says on their web site:“GStreamer is a library for constructing graphs of media-handling

components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.”

“Applications can take advantage of advances in codec and filter technology transparently. Developers can add new codecs and filters by writing a simple plugin with a clean, generic interface. Read more ...”

� My summary:

“A quick and easy way to add audio & Video support to your product for all kinds of media formats without you ever having to dig up and digest a bunch of media specification……and it’s free.

The purpose of this slide is to have a relevant summary of what GStreamer is that can’t be found on

any webpage

11


GStreamer Basics

� Website http://gstreamer.freedesktop.org/

� GNU LGPL (GNU Lesser General Public License)� May used (or be used by) proprietary software without source

disclosure

� Example Applications� VLC Media Player

� Mplayer

� Many more…

With the GNU LGPL license, even though the code that you get is free and open source, you can still

use it as part of a project that will not be free/open source. Compared to GPL where any software that uses GPL code will automatically be required to also be open source. The importance of this LGPL license is that you get to use free open source code, but it doesn’t hinder you from making a closed

source product.

VLC and MPlayer are very widely used. Those applications run in both Linux and Windows. Compared to Windows Media Player, VLC and MPlayer can play many more media formats and have more advanced

features.

One thing to note is that while GSTreamer can play all type of media formats, it’s not really the GStreamer software doing the decoding. It’s mostly made up of other open source (and closed source)

codecs. GStreamer is essentially a ‘wrapper’ around all those different codecs and video/sound manipulators making it easy to integrate into a product and switch elements in and out as you needed.

12


How does it work?

� The framework is a ‘pipeline’ of plugins

Example of a typical pipleline

Looking at this image, the idea of a pipeline is pretty apparent. Data flows from stage to stage where

each stage has a relevant task and prepares the data for the next stage.

13


How does it work?

� The framework is a ‘pipeline’ of plug-ins

� Essentially you combine plug-ins to get the desired result

� Plug-in outputs are referred as ‘Source’, and inputs are referred to as ‘Sink’

� Plug-ins have unique options that can be set

Element

(Input)

Source

Element

(Decoder)

Source

Sink

Element

(Output)

Sink

Example of a typical pipeline

The terms ‘plug-in’ and ‘element’ mean the same thing.

Knowing which plug-ins are available and which need to be ‘piped’ together and how is the trick about learning GStreamer.

Filters are used to change the default format of data as they flow from element to element. You can think of them as like a type cast in programming language.

In Linux, the term “Sink” is also used to define the final stage of an output. A screen sink for example would be a video frame buffer. An audio sink could be ALSA (a common Linux audio player)

Each plug-in can define as many options as they want. Some plug-ins (like the one Renesas made) even take a configuration file location as an argument because the number of options are greater than

you really want to specify as individual parameters.

14


How the pieces fit together

http://www.gstreamer.net

Provided by

GStreamer

End

Application

The boxes here in blue represent items that are distributed with gstreamer. The orange one are boxes

that represent applications or utilities that can be created using gstreamer.

15


GStreamer Tools

� gst-inspect

� Displays all properties, inputs, output types, parameters.

� Example: gst-inspect fakesrc

� gst-launch

� Command line application that can be used to test pipelines.

� Example:

� gst-edit

� A set of widgets to display a

Graphical representation of

a pipeline.

gst-launch audiotestsrc ! audioconvert ! audio/x-raw-int,channels=2 ! Alsasink

GStreamer comes with a default set of command-line utilities that can help in application development.

We will discuss only gst-launch and gst-inspect here.

gst-inspect

gst-inspect can be used to inspect all properties, signals, dynamic parameters and the object hierarchy

of

an element. This can be very useful to see which GObject properties or which signals (and using what

arguments) an element supports. Run gst-inspect fakesrc to get an idea of what it does.

gst-launch

gst-launch is a simple script-like commandline application that can be used to test pipelines. For

example, the command gst-launch audiotestsrc ! audioconvert ! audio/x-raw-int,channels=2 !

alsasink will run a pipeline which generates a sine-wave audio stream and plays it to your ALSA audio

card.

The command line of the application uses exclamation points ( ‘!’ seen in red ) to separate the plug-ins, kind of like the pipe symbol ( | ) on a shell command line.

Plug-ins can also take arguments (seen in blue).

16


Packages

� GStreamer� The core include some essential elements,

documentation

� Base� Well-maintained collection of GStreamer plug-ins and elements

� The Good (collection of plug-ins)� Well supported, good license, good documentation� demuxers, ffmpegcolorspace, flac

� The Bad (collection of plug-ins)� less supported, needs testing/fixes� swfdec, xvid, faad, celt, jack, fbdevsink

� The Ugly (collection of plug-ins)� Well supported, good code, but…watch out for license issues� mad, lame, dvdread, realmedia

� Others…� ffmpeg plugin, access to most ffmpeg library� Python bindings

The ‘GStreamer’ and ‘Base’ are essentially your starting point for the overall framework.

There is a master list on the gstreamer web site that show what media support is located in which package so you know what to download. By putting the software in these categories, it at least give you a general idea of what type of software you’re getting. For the ‘Ugly’ for example, you may not have

been aware of the licensing issues for some of the media formats, so this gives you a good heads up in that sense.

The ‘Others…’ is referring to the fact that there are about 15 download packages in total. Some codes

(like FFmpeg) or other specialty utilities have a module package completely for themselves.

17


Pros and Cons

� Pros� Lots of media formats, codecs, parsers, protocols and filters

� Simple projects to large audio/video editing software� Bindings in multiple languages

– C++, Python, Perl, .NET, Guile, Java, Ruby, Vala� Command line tool for pipeline prototyping� Features like auto-detection of pipelines� Standard Linux API support (V4L2)� Pluggable design allows proprietary plugins

� Cons� Trading convince for performance� Indirect use of media libraries

Cons: Since at it’s core, gstreamer is essentially a set of wrappers around codecs and filters. So, we could probably hypothesize that the software flow is not as optimized as it could be. But to that effect, you would really have to know the codecs inside and out to really get that optimization anyway, so maybe even this point is moot.

18


SH7724 Ports & Custom Plug-ins

� gst-sh-mobile

� Hardware decoder and framebuffer display

– Much faster than using ffmpeg

� Camera capture

� Encode arbitrary video format to h.264 using hardware

� Combine with existing gstreamer demuxers to play h.264 from containers

� https://oss.renesas.com/modules/document/?GStreamer

Camera I/F

Blending Scaling

SoundJPEG

LCDC

NTSC

H.264/MP4 D1@60fps

2D Accel

SH-Mobile SH7724

SH-4A500MHz900MIPs

The SH7724 is a SH4A superscalar SH-Mobile device with FPU.

There is hardware supporting 2 camera capturing devices.

The LCD controller output (while can be used to drive an LCD) can also be used to drive an DVI or HDMI display (given the appropriate interface chip)

The reference to NTSC output (and input by the way) also means it supports PAL.

The sound is a 24-bit Sound processing unit

The major importance of the SH7724 is the hardware video processing units. The blending and scaling units can be tied together so that the data flows through them without software intervention. The 2-D

acceleration is for quickly drawing shapes in a frame buffer. The JPEG engine can be used for decode or encode, and could also be used to make a motion JPEG video file (with additional appropriate software of course).

The H.264 block is the most important. It can encode/decode 60 frames per second at 640x480. It will also do 1280x720 at 30 fps (also knows as HD 720p).

Besides H.264 (also known as AVC or MPEG-4 part 10), the block also supports MPEG-4 (MPEG-4 part 2) and VC-1 (SMPTE 421M-2006).

The oss.renesas.com site contains everything from Linux BSPs, drivers and utilities to presentations by Renesas people.

19


SH7724 Evaluation Board Features

USB mini-AB

(USB1)

USB Host

(USB0) LAN (PoE)

RMII

USB mini-B for

debug console

(SCIF0)

DC Power

Power Switch

POR Reset

Manual Reset

NMI

MFI Bus

(BSC)

DVI

(LCDC)

LCD Board

(LCDC)

DIP

Switches

Push Button

(KEYSC)

IrDAMic Jack

(FSBI)HP Jack

(FSBI)NTSC/PAL

OUT (DV)

NTSC/PAL

IN (DV)

Camera

1 (VIO0)

Camera

2 (VIO1)

JTAG

SD/MMC

(SHDI1)

SD

(SHDI0)

20


Lab Overview

� Look at the GStreamer ‘gst-launch’ tool.

� Examine how ‘plug-ins’ are chosen and used

� Container de-muxing

� Camera h.264 encoding

� Streaming h.264 video

� HW vs SW decoding comparison (SH7724)

� Build an application from source

� Modify an application

21


Checking Progress

� We are using the die to keep track of where everyone is in the lab. Make sure to update it as you change sections.

� When done with the lab, your die will have the 6 pointing up as shown here.

22


Questions?

23


Lab Question Answers

1. Since the custom output plug-in ‘gst-sh-mobile-sink’ does not work with other general GStreamer plug-ins, what you could do for adding effect?

2. Besides available CPU bandwidth, what would also affect flawless playback/recording.

3. Building this application is simple because all the GStreamer libraries and headers were already installed on your board. What do you think it would take rebuild the GStreamer libraries?

24


Innovation

Mobile Media is everywhere…and it’s not going away any time soon.

The perception in the market place is that embedded Ethernet is hard to achieve. The cost of TCP/IP

stacks, development tools, and the time to learn the technology all make it very difficult to design products that talk to each other over Ethernet. However, with a low cost Ethernet enabled MCU and the software freely available in the Open Source community embedded Ethernet is very easy to achieve. Renesas has a complete set of solution for H8S/2472 Ethernet connectivity.

25

© 2010 Renesas Electronics America Inc. All rights reserved.

25

Thank You!

26

Renesas Electronics America Inc.

Documents

ID B32L: Graphical Application Development under …...... Graphical Application Development under Linux/Implementing Video ... VLC Media Player ... Command line application that can