Mobile APIs BOF - Khronos Group · API A little 30 year History of APIsDevloper OS Release Date API Developer OS Release Date ... 1,107 1,062 Mobile Devices WW annual shipments (M

© Copyright Khronos Group, 2009 - Page 1

Mobile APIs BOF

SIGGRAPH 2010


Agenda

- …nothing works without them- Jon Peddie, JPR

The Khronos Handheld API Ecosystem- Neil Trevett, NVIDIA

EGL- Working Group report – Jon Leech, Khronos

OpenGL ES- Working Group report – Tom Olson, ARM- KTX: A light-weight file format for OpenGL and OpenGL ES textures – Mark Callow, HI Corp- Real-time, on-device OpenGL ES profiling using gDebugger – Yaki Tabaka, Graphic Remedy- OpenGL ES 2.0: 3D Graphics for In-Vehicle Infotainment – Petri Talala, Symbio

OpenVG- OpenVG Implementations and Applications – Hwanyong Lee, HUONE

Jon

Ped

die

Re

sea

rch

- nothing works without them

3

Jon

Ped

die

Re

sea

rch

Jon Peddie Research

Jon

Ped

die

Re

sea

rch

Agenda - a bunch of overviews

Status and Momentum of current APIs

• (It’s been a long, hard trail…)

Future developments and trends

• (You may be surprised, or maybe not)

Trends in the mobile graphics markets

• (Some interesting, others FYI)

Jon

Ped

die

Re

sea

rch

A little 30 year History of APIsAPI Devloper OS Release Date API Developer OS Release Date

Plot 10 Tektronix Tek 4010 & HG 1971

CORE ACM Various 1972

GKS ACM Various 1977

PHIGS ACM Various 1979

IRIS--GL SGI Various 1981

ADI Autodesk Various 1988 8514 IBM MS DOS 1988

PEX Various 1989

S3G S3 MS DOS 1991

Fusix PX/GL Du Pont Various 1991 GL Compaq (ACE)

MESA Various 1991 Glide 3Dfx MS DOS & others 1995

Open GL SGI Various 1992 QuickDraw 3D Apple Apple 1995

Direct3D Microsoft MS DOS 1995

PIKS ISO/IEC Various 1995 DirectX 1.0 Microsoft Windows 3.1 Sept. 1995

DirectX 2.0 Microsoft 95 and NT 4.0 Jun. 1996

DirectX 3.0 Microsoft NT 4.0 SP3 Sept. 1996

Fahrenheit Project DirectX 4.0 Microsoft n/a Never released

DirectX 5.0 Microsoft 98 Jul. 1997

GDI Microsoft 98 Jul. 1998

DirectX 6.0 Microsoft 98 SE and ME Aug. 1998

GLX SGI Various 1999 DirectX 7.0 Microsoft 2000 Sept. 1999

DirectX 8.0 Microsoft 2000 Nov. 2000

DirectX 8.1 Microsoft XP Nov. 2001

DirectX 9.0 Microsoft XP Dec. 2002

DirectX 9.0a Microsoft XP Mar. 2003

DirectX 9.0b Microsoft XP Aug. 2003

WGF (LDDM) Microsoft XP 2004

DirectX 9.0c Microsoft XP Aug. 2004

DirectX 10 Microsoft Vista Nov. 2006

PS3-RSX Sony/Nvidia PS3 2006

DirectX 10.1 Microsoft Vista SP1 Feb. 2008

DirectX 11 Microsoft Windows 7 Oct. 2009

Jon

Ped

die

Re

sea

rch

Future Trends

Application

Driver

Operating System

Hardware

API and Graphics Libraries

Jon

Ped

die

Re

sea

rch

Jon

Ped

die

Re

sea

rch

The Mobile Market Overall

0

500

1,000

1,500

2,000

2,500

20092010

20112012

20132014

2015

598 691 784 895 1,031 1,197 1,335

1,0261,091

1,1141,141

1,129

1,1071,062

Mobile Devices WW annual shipments (M units)

All other phones

Mobile (non PC) devices

BILLIONS of thingies get suppliers excited…

Jon

Ped

die

Re

sea

rch

The Suppliers( There’s a lot… )

Main Suppliers Others

Broadcom *

Freescale *

Intel **

Marvell *

NEC *

Nvidia **

Qualcomm **

Renesas *

Samsung **

ST-Ericsson *

Telechips

TI **

Zoran

Anyka

MediaTek

Mtekvision *

Sigma Design

NetLogic *

Wondermedia/VIA *

IP Providers ARC/Virage/Synopsys

ARM **

DMP *

Imagination Technologies **

MIPS

Takumi *

Vivante ** KHRONOS Member

Jon

Ped

die

Re

sea

rch

The Sub or Adjacent Markets

0

200

400

600

800

1,000

1,200

1,400

20092010

20112012

20132014

2015

TOTAL DEVICE SHIPMENTS (M units)

Tablets & Smartbooks Smartphones

DAB Personal Media Devices (PMP)

Navigation (PND) Mobile Game Console - Gen 2

MIDs & Gadgets e-Books

Digital Still Cameras (DSC) Digital Picture Frames (DPF)

Jon

Ped

die

Re

sea

rch

The Sub or Adjacent Markets

0% 5% 10% 15% 20% 25% 30% 35%

Laptop

Smartphone

MP3 music player

Desktop

Netbook

Tablet PC

Consumer future purchase plans (PriceGrabber.com)

Jon

Ped

die

Re

sea

rch

Thank you

[email protected]

Chasing pixels – finding gems


OpenGL ES Update

Tom Olson, ARM

Chair, OpenGL ES Working Group


What is OpenGL ES?

• OpenGL for mobile devices- Gets rid of redundant / legacy features- Extensions to make it mobile-friendly- All the functionality of the desktop API

• Versions- ES 1.0 / 1.1: fixed function- ES 2.0: shader-based via GLSL ES

• A huge success!- The dominant native 3D API for mobile devices- Widely used in other markets: STB, printers, NAV…- Now shipping > 100M units a year


• Available in all the latest high-end handsets

• SDKs, books are now available

• Starting point for other APIs, e.g. WebGL

OpenGL ES 2.0 is here!

…and the fun is just starting


Working Group Activities

• Next Generation OpenGL ES- Working Group’s main focus since mid-2009- Will be released when market needs it- Will not disrupt the growing OpenGL ES 2.0 ecosystem

• ARB / ES Convergence Group- Joint subcommittee of OpenGL and OpenGL ES working groups- Goal is to align roadmaps and avoid unnecessary divergence

• Ecosystem Support- OpenGL ES 2.0 Reference Card - KTX file format


KTX

Mark Callow, HI Corporation

July 2010

© Copyright Khronos Group, 2010 -

Page 19

The Khronos Group Announces

http://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/

• KTX – Khronos TeXture

http://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/


Page 20

KTX

• A new file format for storing texture images

• Oh no! Not Another image file format? Groan...!

• NO

• It is a lightweight format specifically for storing OpenGL®

textures


Page 21

KTX

• Designed around how textures are loaded in OpenGL.

• KTX files contain all the parameters needed for texture loading.

• The loader is very simple.


Page 22

Compression

• Image data can be stored in any of the compressed formats

supported by OpenGL family APIs and extensions

- e.g. ETC1, ATITC, S3TC, etc.

• Or can be stored uncompressed.

• A single .ktx file can contain any type of GL texture from a 1D

texture to an array texture with all mipmap levels.


Page 23

E.G. Mipmap Pyramid in Single KTX file

} my_mipmap.ktx


Page 24

E.G. Cube Map + Mip Levels in Single KTX File

}my_cubemap.ktx


Page 25

Tools

• libktx

- Library of functions for writing ktx files and loading textures from

them

- Loader decompresses ETC1 compressed textures when hardware

does not support ETC1.

- Easy to use.

- E.g. app. can change from using a base level 2D texture to

providing a full mipmap pyramid simply by changing the

file it loads.

• toktx

- Create KTX files from a list of .pam, .pgm or .ppm files

- Does not compress images

• Both are open source under BSD-like license


Page 26

• Specification, downloads, source and documentation can all be

found on the KTX home page in new OpenGL ES SDK at:

http://www.khronos.org/opengles/sdk/tools/KTX/

http://www.khronos.org/opengles/sdk/tools/KTX/


Page 27

More Tools

• etcpack

- Creates KTX files for RGB textures compressed with ETC1.

• Available from Ericsson’s developer site:

http://devtools.ericsson.com/etc

http://devtools.ericsson.com/etc


Page 28

A Note About Orientation

• etcpack and toktx by default map the first pixel in the input file

to s0, t0.

• First pixel in a netpbm file is logical top-left of the image

- so image will be inverted if displayed with GL defaults

• This behavior was chosen to match existing compression tools

• etcpack and toktx provide options to change the mapping

• toktx inserts a KTXorientation key-value pair into the file that

indicates the logical orientation of the image allowing smart

applications to display the image correctly.


Page 29

Future

• We expect other compression tools such as The

Compressionator to add an option to output the KTX format.

• Applications will then be able to use a single file format across

all vendors proprietary texture compression schemes.

© 2004 – 2009 Graphic Remedy. All Rights Reserved

Debugging and Profiling OpenGL ES based applications using gDEBugger ES

Yaki Tebeka, CTO

Graphic Remedy

Smart phones and devices era

• Today’s smart phones and devices are

equipped with GPUs, offering high

performance graphics and low power

consumption

• Implementations are available from

PowerVR, ARM, Qualcomm, NVIDIA, …

• The OpenGL ES API is widely used over

these platforms


Debugging and profiling 3D apps

• Debugging and profiling 3D applications are

hard and time consuming tasks

• Companies find it extremely hard to deliver

robust and bug-free 3D applications

• It is almost impossible to optimize a 3D

application to fully utilize the graphic system

performance


• OpenGL, OpenGL ES and Open CL Debugger,

Profiler and Memory Analyzer

• Exposes the internal graphic system information

needed to find bugs, optimize render performance

and memory usage


Debugging and Profiling Demo


Automatic errors detection

mechanism

gDEBugger can automatically break on:

• OpenGL ES errors

• gDEBugger detected errors: a unique and more

comprehensive errors detection mechanism

• Graphic memory leaks

• Redundant functions

• And more


Call stack and Source code views

• View the call stack and source code that

led to the error / OpenGL ES function call


Statistical and Redundant calls viewer

• Displays the number

of times each

OpenGL ES function

was called

• Display redundant

state changes

• Break on redundant

state change


Performance graph

Displays performance counters graphs from different

sources

• Operating Systems: Windows, Mac OS X and Linux

• GPU and driver: SGX Graphic driver, etc

• gDEBugger OpenGL ES Server


Performance Analysis Toolbar

• Turn off the graphic pipeline stages one after

the other

• If the performance improves when turning off a

certain stage, you have found a graphic

pipeline bottleneck


These commands include

• Eliminate draw commands

• Eliminate raster operations

• Eliminate fixed pipeline lights

• Eliminate texture data fetch operations

• Eliminate geometry shader operations

• Eliminate fragment shader operations


Textures and Buffers viewer

Displays all texture types, FBOs, pbuffers, Depth,

Stencil, VBOs, etc.


Shader's Source Code Editor

• Displays shader's source code

• Displays programs active uniform values

• Edit shader's source code, re-compile, link and

validate "on the fly"


Graphic Memory analysis viewer

• Displays information

about graphic memory

allocated objects

• Displays graphic

memory leaks

• Use the Object

Creation Calls Stack

to see the scenario

that led to each

object's creation


Statistical and Redundant calls viewer

• Displays a statistical

view of application

activities

• Warns about none best

practice usage

• Suggest best practice

alternatives


Questions?

www.gremedy.com

Info [at] gremedy.com


http://www.gremedy.com/

9/3/2010Confidential46

SERIOUS ABOUT SOFTWARE

OpenGL ES 2.0: 3D Graphics and In-Vehicle Infotainment (IVI)

July 29th, 2010 Petri Talala, Symbio

Audi V8.... 20 Years Ago

9/3/2010

Audi R8... Today

9/3/2010

Audi e-Tron Concept 2010

9/3/2010

In Vehicle Infotainment (IVI)

Direction

• IVI systems are becoming a key differentiator between

automotive OEMs

• Huge need for graphics design and programming

expertise

• IVI systems are moving towards open platforms

– Genivi platform

– Ubuntu – Android

• IVIs heavily based on same technology as mobile

graphics

– In majority cases… same HW platforms, same OS base, same

tools

– OpenGL ES x.x, OpenVG for graphics 9/3/2010

9/3/2010

Use Cases

• Different forms of graphics and multimedia acceleration are becoming more common in automotive IVI systems– UI’s for IVI and entertainment systems

– Instrumentation• Gauges and information screens rendered with real-time graphics

• Heads Up Displays

– Video compress/decompress• Parking cameras

– Video postprocessing• Graphics overlay

• Night vision

• Navigation

– Content playback for entertainment• Video decode

• Games

9/3/2010

Use cases

Entertainment cluster:

• UI (content browsing, climate control, etc)

• Video decode for 1 or more screens from 1 or more sources

• Multiple application content

Instrumentation cluster:

• Gauges (2D/OpenVG/OpenGL ES 2.0)

• Information system UI (2D/OpenVG/OpenGL ES 2.0)

• HUD (graphics overlay)

• Night vision (graphics overlay, composition, video postprocessing)

3D Navigation DemoSource: Kishonti Informatics (www.navigenie.com)

9/3/2010

Ecosystem Overview

9/3/2010

HW Platforms

OpenGL ES 2.0

OS /

Frameworks

Content Creation

Toolchain:

-> Model, animate

-> Export / convert-> Integrate-> Run it on target

Adaptation Work

• Customization• Optimized

booting• Optimized

usability• Scheduling

• Layering• Smooth

animations• Smooth

transitions• Automotive

Grade

Few Words about

• Non-profit industry alliance committed to IVI reference

platform based on open source IVI community

– Members such as ARM, BMW, Continental, Freescale, GM,

Intel, Marvell, NVIDIA, Renesas, Symbio, and Texas Instruments

• Genivi platform is a common SW architecture

– Linux based core services, middleware and open application

layer API

– OpenGL ES 2.0 is preferred for graphics

– Allows new compelling automotive applications and new

business models

9/3/2010

9/3/2010

Content Creation Example

HW Platform: NVIDIA Tegra2OS: LinuxOpenGL ES 2.0 based contentFull HD resolutionFPS > 30

Final Notes to Takeaway

1. IVI market is a good business opportunity with

potentially long term commitments

2. Automotive OEMs and Tier-1 suppliers urgently need

graphics expertise

3. IVIs will be mainly based on familiar HW platforms, OS,

OGLES 2.0 graphics from mobile industry

4. Usage of familiar Digital Content Creation Tools

5. Importance to select a trusted and experienced SW

Integration Partner

9/3/2010

9/3/2010Confidential

www.symbio.com

Symbio’s value is based on an extensive track record plus years of experience in demanding software integration and user interface implementation projects in mobile and embedded industry. We know how to optimize graphics rendering solutions for the best

possible visual performance and user experience, while minimizing power consumption through an optimized software stack.

Petri TalalaDirector, Business Development

[email protected]: +1 408 888 8927

mailto:[email protected]

Applications and Implementations

CTO and Technical Marketing Director

HUONE

Hwanyong LEE

60

The Khronos Ecosystem

Embedded 3D

Cross platform desktop 3D

3D Asset Interchange Format

High-level Enhanced Audio

Vector 2D

Surface and synch abstraction

Mobile OS Abstraction

3D asset interchange, 3D web content and Mobile OS portability

High-level streaming media and audio frameworks

Low-level silicon acceleration of graphics, media and parallel computation

Hundreds of man years invested by industry experts to create a coordinated visual computing ecosystem for accelerated parallel computation, 3D, video, audio and image processing on desktop,

embedded and mobile systems

Heterogeneous Parallel Programming

WebGL

3D Web Content

Streaming Media andImage Processing

High-level Steaming MediaRecording and Playback

61

OpenVG

Royalty-free open standard API

Low-level 2D vector graphics rendering API

OpenGL-style programming model

Advanced feature set enables SVG, Flash, PDF, Postscript, Java (JSR 287, 271, 226) etc.

Portable content

Map Applications

Hardware Acceleration

Applications and UI

Hardware Acceleration

SVG/Flash/Font Packages etc..

62

OpenVG Rendering Pipeline

63

CPU sending data and commands to OVG HW. OpenVG rendering pipeline is in the hardware

OpenVG with Native Graphics Processor

CPU Native Vector

Graphics Hardware

64

• Many implementation alternatives

• Everything is processed on the CPU (or +FPU)

- Power consumption is often the CPU maximum power

Software Implementation

CPU

65

• 3D hardware is not tailored for tessellating arbitrary paths to triangles which causes significant pre-processing to the CPU

• Frame-rates can be discontinuous when tessellation is computed infrequently

OpenVG with a 3D GPU (tessellation approach)

Tessellation-based

implementations require complex

polygon tessellation in the software driver

CPU

GPU

66

• Path subdivision is computationally simpler than path tessellation

• Stencil buffer rasterization consumes a lot of memory bandwidth

OpenVG with a 3D GPU (the stencil approach)

Stencil-based implementations require path subdivision in the software driver

CPU

GPUTriangles are

rasterized into stencil buffer to determine path

areas

67

OpenVG with Multimedia Hardware

• Using double buffer, Fast image processing, Bilblt

Selective hardware acceleration

CPU

MM

Most of rasterization (Vector Graphics)

Implementations

Conformant Products

Takumi GV series

DMP dandelion VG,

Vivente GC series

ARM Mali series

Inst. for Information Industry DeltaVG

Imagination PowerVR SGX, MBX with VGP

Broadcom – VideoCore III

HUONE – AlexVG engine

NVIDIA – NVIDIA AP

68

Implementations

Not conformant but based on OpenVG

ShivaVG – Open Source OpenVG on OpenGL

Mazatech – AmanithVG (Software and on OpenGL ES)

Hooked Wireless – Hooked OpenVG (on OpenGL ES)

HUONE – AlexVG forge (on OpenGL ES)

69

Applications

GUI

Mobile Devices GUI – Samsung

Bitmap Graphics, Flash lite, OpenVG, Flash lite on OpenVG

Qt Rendering on OpenVG

Mapping

GPS map drawing

Flash lite player

SVG Mobile Player MMS, DCD, MPEG4-LASeR, WAP

Java Bindings - JSR226, JSR271, JSR287

70

OpenVG Roadmap

OpenVG Inari – Future Version

Investigations

Various kind of features including “shading language”

OpenVG Express – A Streamlined, refactored

Investigations

Add no new functionality

Reduce HW and SW complexity

Fix Color Space

Objectives - Supports SVGT, Flash Lite

CTS - Making it more effective and more accessible to new vendor

71

BOOK & Study Materials

AlexVG™ StudyKit – Sep. 2010 (www.alexvg.com)

Book “OpenVG Programming”

Korean Version - Jan. 2010

English Version – Jan. 2011?

Quick Ref. Card

KHRONOS Booth

72

http://www.alexvg.com/

Documents

Mobile APIs BOF - Khronos Group · API A little 30 year History of APIsDevloper OS Release Date API Developer OS Release Date ... 1,107 1,062 Mobile Devices WW annual shipments (M