OpenGL ES based UI Development on TI Platforms

  • View
    7.807

  • Download
    4

Embed Size (px)

DESCRIPTION

This presentation made at TI Developer Conference 2008, introduces the options available for developers to create User Interfaces on TI SGX based platforms.

Text of OpenGL ES based UI Development on TI Platforms

  • Building advanced user interfaces using SGX Graphics core on OMAP3 Prabindh Sundareson ( [email_address] ) DSPS Texas Instruments
  • Outline
    • Overview of OpenGL
    • Developing an UI with 3D components
    • Advanced 3D features texturing, rotation, materials
    • Integrating the whole system
    • Performance and Power
    • Call For Action
    • Q & A
  • Video vs Graphics Its a state machine - Setup the stage, lighting, actors Then draw it ! Based on motion between frames OpenGL based Graphics Video
  • The OpenGL ES 2.0 Picture OPENGL Full version ES version Common Common-Lite GLSL companion GLSL-ES companion What we miss in ES compared to desktop version: Polygons, Display lists, Accumulation buffers, Currently in 3.0 Currently in 2.0 Currently in 1.0.16 Currently in 1.20 EGL Currently in 1.3 Core GL Spec Vertex/Frag shaders Platform interface and Configuration setup EGL Currently in 1.3
  • ES2.0,ES1.1 and Full Function
    • ES2.0 offers a Programmable pipeline vs Fixed function in ES1.1
      • Ex, Lighting, Fog functions in ES1.1 not present in ES2.0, so need to write code to generate these effects.
      • More flexibility, little more work, but lot of available code
      • Android uses ES 1.1
    • ES2.0 NOT backward compatible with ES1.1
      • See Companion paper for definitive comparison of available functions in ES1.1 and ES2.0
    • See Difference Specification documents in Khronos OpenGL website, for summary of changes compared to openGL full function specification
    • Ok, so what exactly is openGL ?
  • Primitives, Vertices D B C A *Note the orientation
  • The pipeline Vertex shader outputs position of vertex, calculates orientation Fragment shader outputs colour
  • Sample OpenGL ES 2.0 program Handle = get_platform_specific_window_handle(); eglGetDisplay (handle); eglInitialize (); eglBindAPI ( EGL_OPENGL_ES_API ); eglChooseConfig (); eglCreateWindowSurface (); eglCreateContext (); eglMakeCurrent (); //Compile and Load the shaders before this step float afVertices [] = {G,H,F, F,H,E, E,A,F, F,A,B, }; glEnableVertexAttribArray (0); glVertexAttribPointer ( VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, (const void*)afVertices); glDrawArrays(GL_TRIANGLE_STRIP, 0, 4) ; eglSwapBuffers(eglDisplay, eglSurface); Setup Actors Show Keep showing the actors, in a loop, change view/position/texture ..
  • Simple Shader
    • char* pszFragShader = "
    • uniform sampler2D sTexture;
    • varying mediump vec2 TexCoord;
    • void main (void)
    • {
    • gl_FragColor = texture2D(sTexture, TexCoord);
    • }";
    • char* pszVertShader = "
    • attribute highp vec4 inVertex;
    • attribute mediump vec2 inTexCoord;
    • uniform mediump mat4 MVPMatrix;
    • varying mediump vec2 TexCoord;
    • void main()
    • {
    • gl_Position = MVPMatrix * inVertex;
    • TexCoord = inTexCoord;
    • }";
    Note: Read the GL ES Shading Language specification for details
  • OMAP3 + SGX530
    • The OMAP3 Graphics Engine supports
      • OpenGL ES1.1 + EGL 1.3 Fixed function pipeline
      • OpenGL ES2.0 + EGL 1.3 Programmable function pipeline (shaders)
      • OpenVG 1.0.1
      • Arbitrary texture sizes
    • Several useful application libraries provided for
      • Font support (Fixed fonts)
      • Loading textures
      • Loading compressed textures and with mip-maps
      • Loading effects files
    • See the Graphics SDK for details
  • OMAP3 from a UI point of view
    • High performance ARM Cortex-A8
    • VFP Lite with NEON
      • Floating point acceleration
      • Use the below GCC flags on Linux when compiling, to get best performance on Graphics applications
      • -mcpu=cortex-a8 -mfpu=neon -mfloat-abi=softfp
    • SGX core
    • Display subsystem that supports
      • Output resolution upto 1280 x 720 (24 bit Digital)
      • Graphics/Video planes
  • OMAP3 from a UI point of view
    • Linux/WinCE
      • QT/GTK based toolkits
      • Other 3Ps
    • Browser as an OS independent User Interface Opera, Firefox, other Webkit based browsers
      • Many of these can use Cairo (See companion paper) as backend for 2D rendering
    • Flash, Flash-lite
    • Java M3G
    • How to integrate 3D graphics efficiently into these ?
  • Typical UI development with OpenGL ES
    • Shapes
      • Convert to meshes based on primitives (ex, using 3DStudio plugins)
    • Artwork
      • Texture, Mip Maps, filtering modes, lighting (in compressed or uncompressed formats)
    • Animation
      • Path description, timing, dynamic changes, rotation (Write shader logic)
    • Transparency
      • Alpha settings
    • Programming the display path
      • Display driver interfaces and HW
    • Performance tuning
      • Pipelining
      • Compressed textures
  • Alpha Blending with SGX
    • glEnable(GL_BLEND); //disabled by default
    • glBlendEquation(GL_FUNC_ADD); //many other equations available
    • glBlendFunc(GL_SRC_ALPHA, GL_DST_ALPHA); //many other modes available
  • Texturing
    • Gives more realistic views to otherwise dull objects
      • Ex, Lighting effects
    • Textures can be calculated programmatically, or can be taken from images specified by application at runtime (ex, live video thumbnails)
    • Can be applied to lines, or surfaces
    • Coordinates mapped in Vertex shaders
    • Applying the texture done in Fragment shaders
    • Can have multiple textures that can be combined
      • One for lighting effects, one for material properties (ex. Gloss vs Matt )
  • Adding Texturing using OpenGL
    • Pass Texture coordinates
      • glEnableVertexAttribArray ();
      • glVertexAttribPointer ();
    • Use glTexImage2D () to load image buffer for texture
    • Apply texture in the fragment shader program
      • gl_FragColor = texture2D( );
    • OMAP3 supports ARGB 8888 textures
  • Texturing with SGX - Example An example Window manager using texturing and SGX only
  • Rotation with OpenGL
    • Rotation is accomplished by modifying the viewing angle
    • Update the Model View Projection Matrix before drawing
    • Update the vertex position in the vertex shader
      • gl_Position = MVPMatrix * inVertex; // Recall the shader code
    • Along with texturing, can be used to create wet floor effects
      • Apply texture
      • Rotate object to make back side visible, or apply texture in reverse way (remember the orientation of primitives ?)
      • Need to disable CULLing to make back side visible if first method is used
  • Key Decision Points
    • Decide - Compositing also on SGX, or only 3D on SGX ?
      • Decision should be based on ARM loading for the target UI
      • Benchmark before deciding What other applications run on ARM while running the UI ?
    • Use indexed vertices instead of strips for large number of vertices
      • How many > 500 vertex objects are being submitted for rendering ?
    • Avoid dependency on CPU for SGX, and SGX for CPU
    • Mip-Maps increase the size of textures
      • But increase the quality discernibly, especially for near-far cases