View
5
Download
0
Category
Preview:
Citation preview
© Copyright Khronos Group, 2007 - Page 1
Streaming Media Streaming Media PortabilityPortability
March, 2007
© Copyright Khronos Group, 2007 - Page 2
Introduction/AgendaIntroduction/Agenda• Jim van Welzen
- Chair of OpenMax AL- Co-editor of OpenMax IL- Active in Khronos for 2 years- Manager of Embedded Multimedia Software for NVIDIA Corporation
• Agenda- Motivate OpenMax- Explain the three layers
- Development Layer- Integration Layer- Application Layer
- Illustrate through examples- Pointers for learning more- Q & A
© Copyright Khronos Group, 2007 - Page 3
What is What is OpenMaxOpenMax??• A set of open royalty-free APIs for abstracting multimedia
functionality on embedded devices• OpenMax Participation
© Copyright Khronos Group, 2007 - Page 4
Why Why OpenMaxOpenMax??• Classic reason for a standard: fragmentation compromises portability• Multiple pieces to a multimedia ecosystem
- General purpose processors with varying ISA- Media co-processors, DSP, dedicated HW- Codecs, readers/parsers, renderers, sources, post processors- Multimedia frameworks/middleware- Applications
• Multiple implementations of each piece multimedia processing• Multiple (often proprietary) API solutions between any two pieces• Portability problem:
- How do you write a single implementation of a codec, multimedia framework, or application across different platforms?
© Copyright Khronos Group, 2007 - Page 5
Multimedia EcosystemMultimedia Ecosystem
Media Engines - CPUs, DSP, Hardware Accelerators etc.
“Development Layer”Defines media primitives and
concurrency constructs
Audio Components
e.g. MP3
Media components can be integrated into flexible media graphs for advanced
streaming media processing
Media components can be written using primitives for portability across diverse parallel and serial silicon architectures
Application
Platform Media Framework
“Application Layer”Defines high-level playback and
recording interface API
Media applications can be written portably, independent of the underlying media platform
IL
DL
“Integration Layer”Defines media component interfaces
Video Components
e.g. H.264Image
Components e.g. JPEG
AL
OpenMAX layers can be implemented together or independently from the other layers
© Copyright Khronos Group, 2007 - Page 6
3DSmall footprint 3D for embedded systems
Complete Khronos Media StackComplete Khronos Media Stack
Vector 2DLow-level vectoracceleration API
Media Engines – CPUs, DSP, Hardware Accelerators etc.
Platform MediaFrameworks
IL
SOUNDLow-level
gaming audio acceleration
API Image Libraries, Video Codecs,Sound Libraries
Accelerated media primitives for codec
developmentDL
Component interfaces for codec integration
AL
Playback and recording interfaces
Khronos defines low-level, FOUNDATION-level APIs.“Close to the hardware” abstraction provides portability AND flexibility
The Khronos API family provides a complete ROYALTY-FREE, cross-platform media acceleration platform
Applications or middleware libraries (JSR 184 engines, Flash players, media players etc.)
EGLAbstracted Access to
OS ResourcesFast mixed mode 2D/3D
rendering
© Copyright Khronos Group, 2007 - Page 7
Development LevelDevelopment Level
© Copyright Khronos Group, 2007 - Page 8
OpenMAX DL OpenMAX DL –– OverviewOverviewProblem: Porting media components to new platforms is costly and time consuming
Media Engines - CPUs, DSP, Hardware Accelerators etc.
“Development Layer”Defines media primitives and concurrency constructs
Audio Components e.g. MP3
Video Components e.g. H.264
Image Components e.g. JPEG
DL
• Software components are not portable across processors- Exacerbated by proliferation of media standards and increasing silicon complexity
• Software component & silicon vendors need a reliable way to accelerate diverse codecs on diverse silicon
Solution: OpenMAX DL – media components are written using primitives for rapid portability across diverse parallel and serial silicon architectures
• Focus on what one does best- Component vendors provide advanced functionality without worrying about hardware specific optimization
- Silicon vendors provide optimized implementations of DL primitives to reduce cost to market [DL APIs are IP free]
© Copyright Khronos Group, 2007 - Page 9
OpenMAX DL OpenMAX DL –– OverviewOverview
DL Domains• Video Domain
- MPEG-4 SP/H.263 BL (encode and decode)- H.264 (encode and decode)
• Image Processing Domain- Color space conversion- Pixel packing/unpacking- De-blocking / de-ringing- Rotation, scaling, compositing, etc.
• Image Codec Domain- JPEG (encode and decode)
• Multimedia Audio Domain- MP3- AAC
• Signal Processing Domain- FIR- IIR- FFT- Dot Product
A wide range of media acceleration silicon using
many diverse architecturesSilicon vendors supply optimized
OpenMAX DL library covering 80% of processing requirement
An increasing number of multimedia codecs for video, audio, graphics and images
DL
© Copyright Khronos Group, 2007 - Page 10
OpenMAX DL OpenMAX DL –– OverviewOverviewDL API Examples (Video)
OMXResult omxVCCOMM_Average_8x(...)
OMXResult omxVCCOMM_Copy8x8(...)
OMXResult omxVCCOMM_SAD_16x(...)
OMXResult omxVCM4P2_IDCT8x8blk(...)
OMXResult omxVCM4P2_DecodeVLCZigzag_Inter(...)
OMXResult omxVCM4P10_InvTransformResidualAndAdd(...)
OMXResult omxVCM4P10_DeblockLuma_I(...)
Advanced Concurrency Mechanisms• aDL (Asynchronous DL)
- Chain together multiply DL primitives to create one API- Each building block could run aBetter optimization on some platforms
• iDL (Integrated DL)- Integration of DL APIs in OpenMAX Integration Layer (IL)- DL APIs are mapped into IL structures; execution is controlled by IL state machine
© Copyright Khronos Group, 2007 - Page 11
Integration LayerIntegration Layer
© Copyright Khronos Group, 2007 - Page 12
OpenMaxOpenMax ILIL• Componentized architecture for abstracting blocks of multimedia
functionality- Could be implemented in SW or HW
• Building blocks categorized by- Domain: Audio, video, image, of some combination thereof- Function: Encode, decode, apply an effect, capture, render, split, mix, etc
• Allows blocks from different sources to work together• Allows clients to build arbitrary multimedia pipelines by pluging blocks
togetherPortable & Re-usable ‘building blocks’
© Copyright Khronos Group, 2007 - Page 13
OpenMaxOpenMax IL IL ““ComponentComponent””• A component is a building block encapsulating one function• Plumbing
- Each component port is either the entrypoint or exitpoint for one particular stream of data of the component
- A port may be connected to the client or to a port on another component- A client exchanges buffers with a port or two ports (from different componts) exchange
buffers with each other• Knobs
- A component parameter is a value that is set prior to component execution- A component config is a value that may be set during component execution- OpenMax defines an independent set of “knobs”- Five groups: audio, video, image, other (time), common- Examples: rate, volume, resolution, scaling, bitrate, error correction, brightness, etc
• Control- Via a standard interface common to all components (i.e. a structure of function pointers)- The connection of ports- Data flow in and out of ports- State management- Query/set configs and parameters
© Copyright Khronos Group, 2007 - Page 14
Component State MachineComponent State Machine• The client controls the operation of each component by manipulating its
state.- Loaded: alive without resources- Waiting for resources: deficient of resources and actively waiting for them to become available- Idle: has resources but not transferring buffers- Paused: has resources, transferring buffers, but not processing data- Executing: has resources, transferring buffers, and processing data
LOADED
IDLE
EXECUTING
PAUSED
INVALID
UNLOADED
WAIT FORRESOURCES
© Copyright Khronos Group, 2007 - Page 15
OpenMaxOpenMax IL Standard ComponentsIL Standard Components• OpenMax IL 1.1 (just released!) defines a set of standard components• Examples
- Readers/writers: 3gp, asf, image, video, audio- Audio decoders/encoders: AAC, AMR, MP3, WMA, Real Video- Audio post-processor: stereo widening, equalizer, reverb- Image decoders/encoders: JPEG- Input devices: camera, audio input- Output devices: audio renderer, video renderer- Synchronization: clock component, video scheduler
• Each component definition is a recipe consisting of:- Functional description- Which ports of which type are required- Which configs/parameters and which settings on those configs/parameters are required
• Standard set of building blocks allows for more portability
© Copyright Khronos Group, 2007 - Page 16
OpenMAX IL Example GraphOpenMAX IL Example Graph• Standardized component interfaces enable flexible media graphs• Includes multi-stream synchronization• Allows for custom plug-ins
*.mp4 / *.3gpFile Reader
*.mp4 / *.3gpFile Reader
Clock for AV Sync
Clock for AV Sync
AudioDecoderAudio
Decoder AudioRendererAudio
Renderer
VideoScheduler
VideoScheduler Video
RendererVideo
Renderer
Speakers
Display
AAC Audio
TimeData
MPEG4/H.264 Video
VideoDecoderVideo
Decoder DecompressedVideo
OpenMAX IL Component InterfacesEnables components to be flexibly connected in any graph topology
Example: MPEG-4 video synchronized with AAC audio decode
Plug-inPlug-in
E.g. send dance beatsto application for visualization
© Copyright Khronos Group, 2007 - Page 17
Example: Video TextureExample: Video Texture• OpenMax is designed to work with other Khronos APIs, e.g. OpenGL-ES• EGL is a Khronos API for abstracting surfaces
- EGLImage surfaces may be shared between OpenGL and OpenMax• Thus OpenMax IL and OpenGL-ES may be used in concert to implement a
video texture without unnecessary copies
Without EGLImage the game must copy the OpenMax buffer to an OpenGL-ES
textureA client may use an EGLimage as both the
destination of the decode and a texture withoutcopying the datacopies
© Copyright Khronos Group, 2007 - Page 18
Example: Recording Example: Recording GameplayGameplay• OpenMax IL and OpenGL-ES may be used in concert to record gameplay
without unnecessary copies
Without EGLImage the game must copy the pixels from GL to an OpenMax buffer
A client may use an EGLimage as both the destination of the rendering and the source of the
encoding without copying the data
© Copyright Khronos Group, 2007 - Page 19
OpenMAX Architecture & FeaturesOpenMAX Architecture & Features
• Objectives & Profiles• System Architecture• Component Architecture• Component Registration• Component States• In-Context/Out-Context Behavior• Buffer Allocation & Sharing• Port Reconnection
• Buffer Queue Flush• Buffer Marking• Buffer Payload• Buffer Flags• Synchronization• Rate Control• Resource Management• Future Features
© Copyright Khronos Group, 2007 - Page 20
Application LayerApplication Layer
© Copyright Khronos Group, 2007 - Page 21
MotivationMotivation• OpenMax IL has more complexity than most application developers require• Instead of building a multi-component playback graph and coordinating the
numerous pieces most app developers just want to- Specify where the content comes from- Where the content should be rendered to- Manipulate a few playback controls- Have simple configurability
• Likewise for recording use cases• That’s what AL provides…
- A simple high level multimedia API for playback and recording use cases
© Copyright Khronos Group, 2007 - Page 22
Relationship with Relationship with OpenSLOpenSL--ES (1/2)ES (1/2)
• Working groups collaborating to define the common API functionality.
3D Audio
AudioEffects
AdvancedMIDI
AudioPlayback
AudioRecording
MIDI
Videoplayback
Camera
Videorecording
Image capture& displayBuffer
queues
Analog Radio & RDS
(Enhanced audio API) (Multimedia API)
© Copyright Khronos Group, 2007 - Page 23
Relationship with Relationship with OpenSLOpenSL--ES (2/2)ES (2/2)
• Independent- There is no dependency between the APIs.- A device can support a combination of the APIs that most suits the device:
- OpenMAX AL + OpenSL ES (Music)- OpenMAX AL + OpenSL ES (Game)- OpenMAX AL only- OpenSL ES (Phone, Game, Music) only
• Compatible- Working groups collaborated to make sure the APIs work well together.
• Consistent- Identical API architecture.- Identical APIs for same functionality.
© Copyright Khronos Group, 2007 - Page 24
Use CasesUse Cases• OpenMax AL was designed around satisfying a few specific use case yet
also to have a model general enough to be extensible to other use cases• Targeted by 1.0:
- Audio player- Audio recorder- Image displayer- Image capture- Synchronized audio/video playback- Synchronized audio/video recording- Analog Radio
© Copyright Khronos Group, 2007 - Page 25
AL Architectural ConstructsAL Architectural Constructs• AL consists of a set of standard objects and interfaces.
- An interface is a collection of logically related methods.- An particular interface is not exclusive to a particular object
• AL objects- Engine
- The one entity that represents an application’s OpenMax AL session- Devices
- Input and output peripherals, e.g. camera, audio input, audio output, radio- Output Mix: a collection of audio devices
- Media Object- An entity the does some high level multimedia task by performing some operation on an
input and emitting the result as output• AL interfaces (not an exhaustive list)
- Play, Seek, Rate, Audio, DisplayRegion, Metadata Extraction - Record, Camera, Video Encoder, Audio Encoder, Metadata Insertion- Radio, MIDI
© Copyright Khronos Group, 2007 - Page 26
Media ObjectsMedia Objects• Two types:
- Media Player- Media Recorder
• Media object defined by:- Function/Exposed interfaces- Inputs/Outputs
• The input and output are flexible- Inputs include: camera, audio input, URI, memory, content pipe- Outputs include: output mix, display window, URI, memory, content pipe
Media ObjectURI
memory
Content pipe
Camera
Audio Input
URI
memory
Content pipe
Audio Mix
Display Window
? ?
© Copyright Khronos Group, 2007 - Page 27
Simple Call Sequence for PlaybackSimple Call Sequence for Playback• Create Engine object• Get engine interface
- method on engine’s object interface• Create Audio Output Mix object
- method on engine interface- the mix points to particular audio output devices
• Create Media Player object (method on engine interface)
- input parameter: e.g. URI pointing to a local 3gp file
- output parameters: e.g. native display handle, audio output mix
• Get play interface- method on players object interface
• Register event callback- method on player interface
• Set PlayState to Playing- method on play interface
• Wait for end of file event- via register callback
AL Engine
Media Player
Application
Output Mix
EngineItf
PlayItf
Play EventCallback
© Copyright Khronos Group, 2007 - Page 28
Content Pipe ExampleContent Pipe Example• Problem:
- Most parsers expect synchronous pull access to data- Most multimedia frameworks asynchronously push data to parsers- Having each parser accommodate this is inefficient
• A content pipe looks like a file i/o API…- i.e. synchronous read/write calls
• …but it accounts for asynchronously streamed data - i.e. a CheckAvailableBytes call to ensure that enough bytes are available to do a series of
reads or writes• Thus OpenMax implementations can treat all sources (or all sinks) the
same- local files, streamed remote files, broadcasts, etc…
• Both OpenMax IL and OpenMax AL support content pipes• An application can write its own content pipe implementation
- This leads to extremely efficient transfer of data: less latency, less copies
© Copyright Khronos Group, 2007 - Page 29
Content Pipe Example DiagramContent Pipe Example Diagram
3gp parser AAC Decoder Audio Renderer
Application
OpenMax AL
OpenMax IL
Media Player
Depacketizer
Content PipeInterface
Network
© Copyright Khronos Group, 2007 - Page 30
OpenMaxOpenMax –– Relevance to Game DevelopersRelevance to Game Developers• Industry acceptance
- Write code once and it works on a variety of platforms• Interaction with other Khronos APIs
- OpenMax, OpenSL-ES, EGL, and other APIs designed to work well together• OpenMax DL
- Although a game developer will likely not leverage DL it should be viewed as an enabler for rich cross platform multimedia
• OpenMax IL- Graph building – either native or via another framework (e.g. GStreamer)- A means to create a new plug-in- Also as an enabler
• OpenMax AL- The game developers main point of contact with OpenMax functionality- Simple high-level playback- Simple high-level recording- Main use case: any video playback in a game
© Copyright Khronos Group, 2007 - Page 31
Next StepsNext Steps• Visit the Khronos booth
- See the OpenMax demo from TI• Availability
- OpenMax DL 1.0 available now- OpenMax IL 1.1 available now- OpenMax AL available Q2 2007
• Learn more at www.khronos.org- Specification- Headers- White papers
© Copyright Khronos Group, 2007 - Page 32
Questions?Questions?
Recommended