TIDC 2008 : Technologies for Real-world Problems

1

Building advanced user interfaces using SGX Graphics core on OMAP3

Prabindh Sundareson (prabu@ti.com)

DSPS

Texas Instruments

2

Outline• Overview of OpenGL

• Developing an UI with 3D components

• Advanced 3D features – texturing, rotation, materials

• Integrating the whole system

• Performance and Power

• Call For Action

• Q & A

3

Video vs GraphicsVideo vs Graphics

Video OpenGL based Graphics

Based on motion between frames

It’s a state machine - Setup the stage, lighting, actors… Then draw it !

4

The OpenGL ES 2.0 PictureThe OpenGL ES 2.0 Picture

OPENGL Full version

ES version

Common

Common-Lite

GLSL companion

GLSL-ES companion

What we miss in ES compared to desktop version:Polygons, Display lists, Accumulation buffers,…

Currently in 3.0

Currently in 2.0Currently in 1.0.16

Currently in 1.20

EGLCurrently in 1.3

Core GL Spec Vertex/Fragshaders

Platform interface and Configuration setup

EGLCurrently in 1.3

5

ES2.0,ES1.1 and Full FunctionES2.0,ES1.1 and Full Function

• ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1– Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write

code to generate these effects.– More flexibility, little more work, but lot of available code– Android uses ES 1.1

• ES2.0 NOT backward compatible with ES1.1– See Companion paper for definitive comparison of available functions in ES1.1

and ES2.0

• See “Difference Specification” documents in Khronos OpenGL website, for summary of changes compared to openGL “full” function specification

• Ok, so what exactly is openGL ?

6

Primitives, Vertices …

D

B

C

A

*Note the orientation

7

The pipeline…The pipeline…

Vertex shader outputs position of vertex, calculates orientationFragment shader outputs colour

8

Sample OpenGL ES 2.0 program

Handle = get_platform_specific_window_handle();eglGetDisplay(handle);eglInitialize();

eglBindAPI(EGL_OPENGL_ES_API);eglChooseConfig();eglCreateWindowSurface();eglCreateContext();eglMakeCurrent();

//Compile and Load the shaders before this step …float afVertices[] = {G,H,F, F,H,E, E,A,F, F,A,B, …};glEnableVertexAttribArray(0);glVertexAttribPointer(VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices);glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

eglSwapBuffers(eglDisplay, eglSurface);

Setup

Actors

Show

Keep showing the actors, in a loop, change view/position/texture ..

9

Simple ShaderSimple Shader

• char* pszFragShader = "\

• uniform sampler2D sTexture; \

• varying mediump vec2 TexCoord; \

• void main (void)\

• {\

• gl_FragColor = texture2D(sTexture, TexCoord);\

• }";

• char* pszVertShader = "\

• attribute highp vec4 inVertex;\

• attribute mediump vec2 inTexCoord;\

• uniform mediump mat4 MVPMatrix;\

• varying mediump vec2 TexCoord;\

• void main()\

• {\

• gl_Position = MVPMatrix * inVertex;\

• TexCoord = inTexCoord;\

• }";

Note: Read the GL ES Shading Language specification for details

10

OMAP3 + SGX530

• The OMAP3 Graphics Engine supports – OpenGL ES1.1 + EGL 1.3 Fixed function pipeline

– OpenGL ES2.0 + EGL 1.3 Programmable function pipeline (shaders)

– OpenVG 1.0.1

– Arbitrary texture sizes

• Several useful application libraries provided for – Font support (Fixed fonts)

– Loading textures

– Loading compressed textures and with mip-maps

– Loading effects files

• See the Graphics SDK for details

11

OMAP3 from a UI point of view

• High performance ARM Cortex-A8

• VFP Lite with NEON – Floating point acceleration– Use the below GCC flags on Linux when compiling, to get best performance

on Graphics applications“-mcpu=cortex-a8 -mfpu=neon -mfloat-abi=softfp”

• SGX core

• Display subsystem that supports– Output resolution upto 1280 x 720 (24 bit Digital)– Graphics/Video planes

12

OMAP3 from a UI point of view

• Linux/WinCE– QT/GTK based toolkits– Other 3Ps

• Browser as an OS independent User Interface – Opera, Firefox, other Webkit based browsers

– Many of these can use Cairo (See companion paper) as backend for 2D rendering

• Flash, Flash-lite

• Java – M3G

• How to integrate 3D graphics efficiently into these ?

13

Typical UI development with OpenGL ES

• Shapes – Convert to meshes based on primitives (ex, using 3DStudio plugins)

• Artwork – Texture, Mip Maps, filtering modes, lighting (in compressed or uncompressed

formats)

• Animation – Path description, timing, dynamic changes, rotation (Write shader logic)

• Transparency – Alpha settings

• Programming the display path – Display driver interfaces and HW

• Performance tuning –– Pipelining– Compressed textures

14

Alpha Blending with SGX

• glEnable(GL_BLEND); //disabled by default

• glBlendEquation(GL_FUNC_ADD); //many other equations available

• glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other modes available

15

TexturingTexturing

• Gives more realistic views to otherwise “dull” objects– Ex, Lighting effects

• Textures can be calculated programmatically, or can be taken from images specified by application at runtime (ex, live video thumbnails)

• Can be applied to lines, or surfaces

• Coordinates mapped in Vertex shaders

• “Applying the texture” done in Fragment shaders

• Can have multiple textures that can be combined– One for lighting effects, one for material properties (ex. Gloss vs Matt …)

16

Adding Texturing using OpenGL

• Pass Texture coordinates– glEnableVertexAttribArray();

– glVertexAttribPointer();

• Use glTexImage2D() to load image buffer for texture

• Apply texture in the fragment shader program– gl_FragColor = texture2D();

• OMAP3 supports ARGB 8888 textures

17

Texturing with SGX - Example

An exampleWindow managerusing texturing and SGX only

18

Rotation with OpenGL

• Rotation is accomplished by modifying the viewing angle

• Update the Model View Projection Matrix before drawing

• Update the vertex position in the vertex shader

– gl_Position = MVPMatrix * inVertex; // Recall the shader code

• Along with texturing, can be used to create “wet floor” effects– Apply texture – Rotate object to make back side visible, or apply texture in reverse way

(remember the orientation of primitives ?)– Need to disable CULLing to make back side visible if first method is used

19

Key Decision Points

• Decide - Compositing also on SGX, or only 3D on SGX ?– Decision should be based on ARM loading for the target UI

– Benchmark before deciding – What other applications run on ARM while running the UI ?

• Use indexed vertices instead of strips for large number of vertices– How many > 500 vertex objects are being submitted for rendering ?

• Avoid dependency on CPU for SGX, and SGX for CPU

• Mip-Maps increase the size of textures– But increase the quality discernibly, especially for near-far cases

20

Getting the final image to the display

• Case 1 (Partner mode – work with other applications):– OpenGL allows reading back the composited image from the gl frame buffer

• glReadPixels(xpos, ypos, xwidth, yheight, format, type, *userdataptr);• Caveat: This flushes the pipeline. Benchmark for your target application• Now the data is in the external application/driver• Regular alpha blending/compositing operations can be performed on this data

– OMAP3 Display System HW supports 3 display planes• 2 video planes with up/down scaling, 1 Graphics plane

– Any of these planes can be used for blending with outputs from other applications

• Case 2 (Full ownership mode)– The final image can be directly written to the framebuffer in full screen mode

by EGL itself

21

Application Integration

• Multiple options available

• Write separate application, directly use EGL for display to frame buffer– Use SGX for all compositing and window operations

– No other window management required

• Use application framework like Clutter to build your applications. Configure Clutter to use OpenGL-ES as backend– More application support becomes available with this option, ex. Browsers ..

– Clutter is still under active development

• Use GTK to manage the Graphics output created by SGX application

22

Performance and Power

• The SGX core is connected to the L3 interface (Refer SPRU66B)– Internal DMA, MMU for operating without interfering ARM

• Tops at 111 MHz, other variants in the future

• Power management handled internally in the SGX HW/driver

• Power numbers comparable to ARM at equivalent MHz (contact TI support for details)

23

Performance and Power

• Target upto 30 fps with 512x512 (window like) textures in 2D

• Frame rate goes down as number/size of textures, and complexity of objects and operations goes up– Use compressed PVRTC mode textures for static textures

• Rendering full screen with OpenGL allows ARM to do other stuff– Network stack, Browsers, Multimedia, …

24

Call For Action

• Download the OMAP3 Graphics SDK

• Use Beagle or EVM or other suitable OMAP3 platform for development

• Use the examples provided to start, download examples from net

• Use mailing list for support

• Engage at the TI open-source symposium (7/11/08)

25

Support

• For support on Graphics related queries on OMAP3, write to

omap35x_3p_support@list.ti.com

• We will be glad to support you to create that great UI !!

26

ReferencesReferences

• TRM for SGX core

– http://focus.ti.com/lit/ug/spruff6b/spruff6b.pdf

• PC version of PowerVR SDK– http://www.imgtec.com/powervr/insider/sdk/KhronosOpenGLES2xSGX.asp

• TI DVEVM software (Needs EVM serial #, and my.ti.com account for registration)– https://www-a.ti.com/extranet/cm/product/dvevmsw/dspswext/general/v1_30_00.shtml

• The OpenGL Redbook– http://www.glprogramming.com/red/chapter01.html

• OpenGL ES2.0 programming handbook– OpenGL® ES 2.0 Programming Guide, Aaftab Munshi; Dan Ginsburg; Dave Shreiner

• Khronos organisation– http://Khronos.org

• Brief description of OMAP3 Graphics usage (Needs my.ti.com acc)– https://focus.ti.com/general/docs/event/accesseventaction.tsp?actionId=414