Mobile 3D Graphics

Mobile 3D GraphicsMobile 3D Graphics

Anders HenryssonAnders HenryssonNVIS/VITA, Linköpings universitetNVIS/VITA, Linköpings universitet

TNCG03 – Real Time Rendering

2

OutlineOutline

• Mobile DevicesMobile Devices• Mobile 3D GraphicsMobile 3D Graphics• APIs for Mobile 3D GraphicsAPIs for Mobile 3D Graphics

– OpenGL ESOpenGL ES– M3G (JSR-184)M3G (JSR-184)

• Fixed-Point ArithmeticFixed-Point Arithmetic• Mobile GPUsMobile GPUs• ResearchResearch

– Texture CompressionTexture Compression– Augmented RealityAugmented Reality

• ReadingsReadings• ResourcesResources

3

Need for 2D graphicsNeed for 2D graphics

• EvolutionEvolution– CUI to GUICUI to GUI– Photo and VideoPhoto and Video– Web BrowsingWeb Browsing– GamesGames– TVTV– Navigation (GPS)Navigation (GPS)

1998 2007

4

Need for 3D Graphics?Need for 3D Graphics?

• Games!Games!– The main The main

motivation but motivation but not the only onenot the only one

• Mobile devices:Mobile devices:– On-site On-site

VisualizationVisualization– Augmented Augmented

RealityReality– Small screen Small screen

-3DUI?-3DUI?

5

Example Device – Nokia N73Example Device – Nokia N73

• 220 MHz ARM CPU220 MHz ARM CPU• 42 MB memory42 MB memory• 2.4” TFT2.4” TFT• 240 x 320 @ 16 bpp240 x 320 @ 16 bpp• Button interfaceButton interface• Symbian OS 9.1Symbian OS 9.1• C++ and Java SDKsC++ and Java SDKs

6

Example Device – iPhoneExample Device – iPhone

• 620 MHz ARM* 620 MHz ARM* • Jazelle* (Java)Jazelle* (Java)• PowerVR MBX* (3D)PowerVR MBX* (3D)• 3.5” Touch screen3.5” Touch screen• 480 x 320480 x 320• OS XOS X• SDK for Web-based appsSDK for Web-based apps

* = Unconfirmed* = Unconfirmed

7

TrendsTrends

• Faster processorsFaster processors• More memoryMore memory• Increasing screen resolutionIncreasing screen resolution

– Handheld displays are closer to the eye than desktop Handheld displays are closer to the eye than desktop displays hence need higher ppi displays hence need higher ppi

• GPUs (more on these later)GPUs (more on these later)

8

ChallengesChallenges

• Battery Battery – Development lag behindDevelopment lag behind– More power consumption with camera, GPS etc.More power consumption with camera, GPS etc.– 3D graphics need memory access, which consumes power3D graphics need memory access, which consumes power

• No cooling possibleNo cooling possible• Device diversityDevice diversity

– OS (Many are proprietary)OS (Many are proprietary)– Floating-points (No FPU means big decrease in performance Floating-points (No FPU means big decrease in performance

due to emulation)due to emulation)– Screens have varying resolutions and proportionsScreens have varying resolutions and proportions– Hardware is not PnP – custom chipsHardware is not PnP – custom chips– Big difference between low-end and high-endBig difference between low-end and high-end

• Limited input capabilities for 3D graphicsLimited input capabilities for 3D graphics

9

3D APIs3D APIs

[Image courtesy of Kari Pulli, Nokia]

• Two levels:Two levels:– Low level controlLow level control– High level functionalitiesHigh level functionalities

• Two environments:Two environments:– Open operating systems allowing native applicationsOpen operating systems allowing native applications– Closed systems allowing JavaClosed systems allowing Java

10

OpenGL ESOpenGL ES

• BackgroundBackground• Differences to OpenGLDifferences to OpenGL• Versions Versions • ExamplesExamples

11

OpenGL ES - BackgroundOpenGL ES - Background

• Desktop OpenGL is a good startDesktop OpenGL is a good start– Widely adopted and well documentedWidely adopted and well documented– Low levelLow level– Orthogonal functionsOrthogonal functions

• Too big for mobile phonesToo big for mobile phones• Redundant functionsRedundant functions

– E.g. many variations of glColor() E.g. many variations of glColor()

• Expensive functionsExpensive functions– 3D textures3D textures– SplinesSplines

12

OpenGL ES - OpenGL for Embedded SystemsOpenGL ES - OpenGL for Embedded Systems

• Subset of desktop OpenGLSubset of desktop OpenGL– Smaller memory footprint (50 KB)Smaller memory footprint (50 KB)

• Khronos GroupKhronos Group– >100 companies>100 companies– Nokia, Nvidia, ATI, Ericsson, Sun, Intel, Google…Nokia, Nvidia, ATI, Ericsson, Sun, Intel, Google…– … … or everyone but Microsoftor everyone but Microsoft

• Used by Sony for Playstation 3Used by Sony for Playstation 3– Designed for interactive 3DDesigned for interactive 3D

13

OpenGL ES 1.0OpenGL ES 1.0

• Based on OpenGL 1.3 pipelineBased on OpenGL 1.3 pipeline– Fixed-function hardwareFixed-function hardware

• Double precision floating-points removedDouble precision floating-points removed• Fixed-point added (more on this later)Fixed-point added (more on this later)• Two profilesTwo profiles

– Common / Common-Lite - EntertainmentCommon / Common-Lite - Entertainment– Safety Critical – Industrial, military, medicalSafety Critical – Industrial, military, medical

14

OpenGL 1.3 – OpenGL ES 1.0 differencesOpenGL 1.3 – OpenGL ES 1.0 differences

• Primitives Primitives – Supported: Supported: POINTS, LINES, LINE STRIP, LINE LOOP,

TRIANGLES, TRIANGLE STRIP, TRIANGLE FAN– Not supported: Not supported: QUADS, QUAD STRIP, POLYGON

• No Begin/End!No Begin/End!• Vertex arrays onlyVertex arrays only

15

16

Example: How to draw a cube Example: How to draw a cube

static const GLshort cubeVertices[] =static const GLshort cubeVertices[] = 8 vertices8 vertices{{

-40, 40, 40,-40, 40, 40,……..-40, -40, -40-40, -40, -40

};};

static const GLbyte cubeNormals[] =static const GLbyte cubeNormals[] = 8 normals8 normals{{

-1, 1, 1,-1, 1, 1,......-1, -1, -1-1, -1, -1

};};

static const GLubyte cubeIndices[] =static const GLubyte cubeIndices[] = 6 polygons (i.e. 2 triangles)6 polygons (i.e. 2 triangles){{

0, 3, 1, 2, 0, 1, /* front */0, 3, 1, 2, 0, 1, /* front */……3, 7, 1, 7, 5, 1 /* bottom */3, 7, 1, 7, 5, 1 /* bottom */

};};

glEnableClientState (GL_VERTEX_ARRAY);glEnableClientState (GL_VERTEX_ARRAY);glEnableClientState (GL_NORMAL_ARRAY);glEnableClientState (GL_NORMAL_ARRAY);glVertexPointer (3, GL_SHORT, 0, cubeVertices);glVertexPointer (3, GL_SHORT, 0, cubeVertices);glNormalPointer (GL_BYTE, 0, cubeNormals);glNormalPointer (GL_BYTE, 0, cubeNormals);

glDrawElements (GL_TRIANGLES, 36, GL_UNSIGNED_BYTE, cubeIndices);glDrawElements (GL_TRIANGLES, 36, GL_UNSIGNED_BYTE, cubeIndices);

17


• No support for operation on pixel imagesNo support for operation on pixel images– ConvolutionConvolution– DrawPixelsDrawPixels– HistogramHistogram

• 2D textures only – 1D and 3D are out2D textures only – 1D and 3D are out• Special functions not supportedSpecial functions not supported

– Evaluators mostly for CADEvaluators mostly for CAD– Selection not applicable in gamingSelection not applicable in gaming– Display list not supported in 1.0 Common but in SCDisplay list not supported in 1.0 Common but in SC

18


• Core AdditionsCore Additions– Fixed-point functions e.g.:Fixed-point functions e.g.:

19


• Derived from OpenGL 1.5Derived from OpenGL 1.5• Enhanced texturingEnhanced texturing• Some differences to 1.0 :Some differences to 1.0 :

– MultitextureMultitexture– Auto mipmap generationAuto mipmap generation– Vertex buffer objectsVertex buffer objects– Point sprite arrays – different sizes makes particle rendering Point sprite arrays – different sizes makes particle rendering

easiereasier– User clip planesUser clip planes

• Backward compatible with 1.0Backward compatible with 1.0

20


• Based on OpenGL 2.0Based on OpenGL 2.0• Programmable hardwareProgrammable hardware

– Vertex and fragment shadersVertex and fragment shaders

• Shaders have replaced fixed Shaders have replaced fixed functions to keep the API size functions to keep the API size smallsmall

• Not fully backward compatible Not fully backward compatible with 1.xwith 1.x

• Products expected before end Products expected before end of 2007of 2007– Two drivers: 1.x and 2.0Two drivers: 1.x and 2.0

21


• OpenGL ES Shading LanguageOpenGL ES Shading Language– Same as GLSL but adapted to mobile devicesSame as GLSL but adapted to mobile devices

• Vertex ArraysVertex Arrays– In: VertexAttribPointerIn: VertexAttribPointer– Out: VertexPointer, NormalPointer etc.Out: VertexPointer, NormalPointer etc.

• Draw a triangle:Draw a triangle:GLfloat pfVertices[] = {-0.4f,-0.4f,0.0f, +0.4f,-0.4f,0.0f, 0.0f,0.4f,0.0f};GLfloat pfVertices[] = {-0.4f,-0.4f,0.0f, +0.4f,-0.4f,0.0f, 0.0f,0.4f,0.0f};

glEnableVertexAttribArray(VERTEX_ARRAY);glEnableVertexAttribArray(VERTEX_ARRAY);

glVertexAttribPointer(VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, pfVertices);glVertexAttribPointer(VERTEX_ARRAY, 3, GL_FLOAT, GL_FALSE, 0, pfVertices);

glDrawArrays(GL_TRIANGLES, 0, 3);glDrawArrays(GL_TRIANGLES, 0, 3);

22

OpenGL ES RoadmapOpenGL ES Roadmap

Mobile 3D graphics is catching up fast!Mobile 3D graphics is catching up fast!

23

EGLEGL

• No GLU or GLUT on mobile devicesNo GLU or GLUT on mobile devices• EGL is the interface between OpenGL ES and EGL is the interface between OpenGL ES and

the underlying window systemthe underlying window system• EGL handles EGL handles

– graphics context – RGB bits, stencil bits etc.graphics context – RGB bits, stencil bits etc.– Render surface – window, buffer, pixmapRender surface – window, buffer, pixmap– Synchronization – e.g. native 2D and OpenGL ESSynchronization – e.g. native 2D and OpenGL ES

24

EGL Example CodeEGL Example Code

attributes:attributes:static const EGLint attribList[] =static const EGLint attribList[] =

{{EGL_BUFFER_SIZE,EGL_BUFFER_SIZE, 32,32,EGL_ALPHA_SIZE,EGL_ALPHA_SIZE, 8,8,EGL_BLUE_SIZE,EGL_BLUE_SIZE, 8,8,EGL_GREEN_SIZE,EGL_GREEN_SIZE, 8,8,EGL_RED_SIZE,EGL_RED_SIZE, 8,8,EGL_DEPTH_SIZE,EGL_DEPTH_SIZE, 16,16,EGL_SURFACE_TYPE,EGL_SURFACE_TYPE, EGL_PIXMAP_BIT,EGL_PIXMAP_BIT,

EGL_NONEEGL_NONE };};

return an EGL display connection: return an EGL display connection: m_display = eglGetDisplay(EGL_DEFAULT_DISPLAY); m_display = eglGetDisplay(EGL_DEFAULT_DISPLAY);

return a list of all EGL frame buffer configurations for a display:return a list of all EGL frame buffer configurations for a display:if(eglGetConfigs(m_display, NULL, 0, &numConfigs) == EGL_FALSE) if(eglGetConfigs(m_display, NULL, 0, &numConfigs) == EGL_FALSE)

return a list of EGL frame buffer configurations that match specified attributes:return a list of EGL frame buffer configurations that match specified attributes:if(eglChooseConfig(m_display, attribList, &m_config, 1, &numConfigs) == EGL_FALSE) if(eglChooseConfig(m_display, attribList, &m_config, 1, &numConfigs) == EGL_FALSE)

create a new EGL rendering context:create a new EGL rendering context:m_context = eglCreateContext(m_display, m_config, NULL, NULL);m_context = eglCreateContext(m_display, m_config, NULL, NULL);

create a new EGL pixmap surface:create a new EGL pixmap surface:m_surface = eglCreatePixmapSurface(m_display, m_config, m_pixmap, NULL);m_surface = eglCreatePixmapSurface(m_display, m_config, m_pixmap, NULL);

25

M3G – Mobile 3D Graphics (JSR-184)M3G – Mobile 3D Graphics (JSR-184)

• What? 3D graphics for Java ME (Micro Edition)What? 3D graphics for Java ME (Micro Edition)• Why? Java ME is cross platform and supported by Why? Java ME is cross platform and supported by

most device manufacturers – C/C++ is notmost device manufacturers – C/C++ is not• Much slower than C/C++ thoughMuch slower than C/C++ though

– Graphics processing moved to native codeGraphics processing moved to native code• M3G is about 150KB (Java3D is about 100 times larger)M3G is about 150KB (Java3D is about 100 times larger)• Not Java3D – redesigned for mobilesNot Java3D – redesigned for mobiles

– Support for animationSupport for animation– Import meshes from animation packagesImport meshes from animation packages– Scene graphScene graph

• Two modesTwo modes– ImmediateImmediate– RetainedRetained

26

M3G – Immediate ModeM3G – Immediate Mode

• Low-level - similar to OpenGL ESLow-level - similar to OpenGL ES• Rendering engine execute commands Rendering engine execute commands immediatelyimmediately • Code sample: Code sample:

private static final byte[] VERTEX_POSITIONS = {private static final byte[] VERTEX_POSITIONS = { -1, -1, 1, 1, -1, 1, -1, 1, 1, 1, 1, 1,-1, -1, 1, 1, -1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, -1, -1, -1, 1, -1, 1, 1, -1-1, -1, -1, 1, -1, -1, -1, 1, -1, 1, 1, -1 };};

VertexArray vertexPositions = new VertexArray(VERTEX_POSITIONS.length/3, 3, 1);VertexArray vertexPositions = new VertexArray(VERTEX_POSITIONS.length/3, 3, 1);vertexPositions.set(0, VERTEX_POSITIONS.length/3, VERTEX_POSITIONS);vertexPositions.set(0, VERTEX_POSITIONS.length/3, VERTEX_POSITIONS);_cubeVertexData.setPositions(vertexPositions, 1.0f, null);_cubeVertexData.setPositions(vertexPositions, 1.0f, null);

protected void paint(Graphics graphics)protected void paint(Graphics graphics) {{ _graphics3d.bindTarget(graphics);_graphics3d.bindTarget(graphics); _graphics3d.clear(null);_graphics3d.clear(null); _graphics3d.render(_cubeVertexData, _cubeTriangles, new Appearance(), null);_graphics3d.render(_cubeVertexData, _cubeTriangles, new Appearance(), null); _graphics3d.releaseTarget();_graphics3d.releaseTarget(); }}

27

M3G – Retained ModeM3G – Retained Mode

• High-level information is High-level information is retainedretained from frame to from frame to frameframe

• While immediate mode operates on 3D objects, While immediate mode operates on 3D objects, retained mode operates on a world of 3D retained mode operates on a world of 3D objectsobjects

• World is the root node in a scene graphWorld is the root node in a scene graph

28

Scene GraphScene Graph

• Tree structure – organize Tree structure – organize data in a hierarchydata in a hierarchy

• A transform (translation, A transform (translation, rotation etc.) on one node rotation etc.) on one node applies to all child nodes – applies to all child nodes – convenient for animationconvenient for animation

• Nodes are objectsNodes are objects– Example: Group, world, Example: Group, world,

camera, mesh, Sprite3Dcamera, mesh, Sprite3D

[Image courtesy of Tomi Aarnio, Nokia]

Group allMeshes = (Group)_world.find(USER_ID_ALL_MESHES);Group allMeshes = (Group)_world.find(USER_ID_ALL_MESHES);allMeshes.postRotate(2, 0.0f, 0.0f, 1.0f);allMeshes.postRotate(2, 0.0f, 0.0f, 1.0f);

29

M3G Example Code (IBM)M3G Example Code (IBM)

_world = new World();_world = new World();

Mesh cubeMesh = new Mesh(cubeVertexData, cubeTriangles, new Appearance());Mesh cubeMesh = new Mesh(cubeVertexData, cubeTriangles, new Appearance()); _world.addChild(cubeMesh);_world.addChild(cubeMesh);

Camera camera = new Camera();Camera camera = new Camera(); camera.setPerspective(30.0f, aspect, 1.0f, 1000.0f);camera.setPerspective(30.0f, aspect, 1.0f, 1000.0f); camera.setTranslation(0.0f, 0.0f, 10.0f);camera.setTranslation(0.0f, 0.0f, 10.0f); _world.addChild(camera);_world.addChild(camera);

protected void paint(Graphics graphics)protected void paint(Graphics graphics) {{ _graphics3d.bindTarget(graphics);_graphics3d.bindTarget(graphics); _graphics3d.render(_world);_graphics3d.render(_world); _graphics3d.releaseTarget();_graphics3d.releaseTarget(); }}

30

M3G File Format .m3gM3G File Format .m3g

• Export scene with animation from animation Export scene with animation from animation packages – 3D Studio MAX, MAYA etc.packages – 3D Studio MAX, MAYA etc.

• From individual objects to entire scene graphsFrom individual objects to entire scene graphs• Saves a lot of timeSaves a lot of time

roots = Loader.load("http://www.example.com/myscene.m3g");roots = Loader.load("http://www.example.com/myscene.m3g");

myWorld = (World) roots[0];myWorld = (World) roots[0];

31

M3G - AnimationM3G - Animation

• Some animation classesSome animation classes– AnimationController - position, speed, and weightAnimationController - position, speed, and weight– KeyframeSequence - animation data as a sequence of KeyframeSequence - animation data as a sequence of

time-stamped, vector-valued keyframestime-stamped, vector-valued keyframes– MorphingMesh – morph weight can be animatedMorphingMesh – morph weight can be animated– SkinnedMesh - scene graph node that represents a SkinnedMesh - scene graph node that represents a

skeletally animated polygon meshskeletally animated polygon mesh

32

M3G 2.0M3G 2.0

• JSR-297 submitted 2007JSR-297 submitted 2007• Purpose: Make OpenGL ES 2.0 hardware Purpose: Make OpenGL ES 2.0 hardware

accessible to Java ME developersaccessible to Java ME developers

33

Other StandardsOther Standards

• OpenVG OpenVG – 2D vector graphics for UI, presentation etc.2D vector graphics for UI, presentation etc.

• COLLADA COLLADA – Asset management. XML based. Can be translated to Asset management. XML based. Can be translated to

.m3g.m3g• MD3DM (Managed Direct3D Mobile) MD3DM (Managed Direct3D Mobile)

– For Microsoft Mobile. Supports files (.md3dm) For Microsoft Mobile. Supports files (.md3dm) • OpenKODE - Source Portability.OpenKODE - Source Portability.

– OpenGL ES, OpenVG OpenMAXOpenGL ES, OpenVG OpenMAX– Nvidia UI demo at 3GSM (check it out on youtube)Nvidia UI demo at 3GSM (check it out on youtube)

• Brew, Mascot Capsule etc.Brew, Mascot Capsule etc.– Proprietary M3G alternatives Proprietary M3G alternatives

34

Fixed-Point ArithmeticFixed-Point Arithmetic

• Background – Mobile phones have no FPUBackground – Mobile phones have no FPU– Consumes valuable chip space and battery powerConsumes valuable chip space and battery power– Not needed for most mobile phone softwareNot needed for most mobile phone software

• Software emulation is up to two order of Software emulation is up to two order of magnitude slowermagnitude slower

• 3D graphics need precision3D graphics need precision– Blending, depth test, rotationBlending, depth test, rotation

• 32 bit integers: 32 bit integers: -2147483648 … 2147483648-2147483648 … 2147483648– No physical equivalenceNo physical equivalence

• Idea: Use some bits for precisionIdea: Use some bits for precision

35


• OpenGL ES – 16:16 formatOpenGL ES – 16:16 format– 16 bits for integer value and 16 bits for precision16 bits for integer value and 16 bits for precision– Max integer: 32768 (2Max integer: 32768 (21515) ) – Max resolution: 0,0000152587890625 (2Max resolution: 0,0000152587890625 (2-16-16))

• Macros (bits = 16):Macros (bits = 16):#define#define float_to_fix(val, bits) ( float_to_fix(val, bits) (intint)(val * (1<<bits)))(val * (1<<bits))

#define#define fix_to_float(val, bits) ( fix_to_float(val, bits) (floatfloat)val / (1<<bits))val / (1<<bits)

• Addition and subtraction as with integersAddition and subtraction as with integers

36


• Multiplication and division needs 64 bit representations Multiplication and division needs 64 bit representations not to loose significant bits:not to loose significant bits:

• Multiplication:Multiplication:res = (res = (intint) (((I64) x * (I64) y) >> bits);) (((I64) x * (I64) y) >> bits);

• Division:Division: res = (res = (intint) ((((I64)x)<<bits)/(I64)y);) ((((I64)x)<<bits)/(I64)y);

• Assembler twice as fast. ARM example - x * y:Assembler twice as fast. ARM example - x * y: asm volatile ("SMULL r6, r7, %0, %1" : : "r"(x), asm volatile ("SMULL r6, r7, %0, %1" : : "r"(x),

"r"(y) : "r6", "r7" );"r"(y) : "r6", "r7" );

• For trigonometry – use tablesFor trigonometry – use tables• Also useful: Newton-Raphson for inversionAlso useful: Newton-Raphson for inversion

37

Mobile Graphics Hardware - MotivationMobile Graphics Hardware - Motivation

• Increased 3D rendering performance but also Increased 3D rendering performance but also faster UI, video, imagesfaster UI, video, images– Many UIs are annoyingly slowMany UIs are annoyingly slow

• Hardware floating-pointHardware floating-point• Programmable pipelineProgrammable pipeline• Dedicated GPU more power efficient than Dedicated GPU more power efficient than

general CPUgeneral CPU

38

AMDAMD

• Imageon 2300Imageon 2300– OpenGL ES 1.0OpenGL ES 1.0

• Imageon 238xImageon 238x– OpenGL ES 1.1OpenGL ES 1.1– < 3M Triangles/sec< 3M Triangles/sec– <100M pixels/sec<100M pixels/sec– Features:Features:

• Vertex SkinningVertex Skinning• PostprocessingPostprocessing• Dynamic Cube Mapping Dynamic Cube Mapping • Advanced Reflection/Refraction Advanced Reflection/Refraction • Projective LightingProjective Lighting• Per Pixel Specular HighlightsPer Pixel Specular Highlights• Point SpritesPoint Sprites

Videos: ati.amd.com/products/imageon238x/Videos: ati.amd.com/products/imageon238x/

39

NVIDIANVIDIA

• GoForce 4800GoForce 4800– OpenGL ES 1.1OpenGL ES 1.1

• GoForce 5500GoForce 5500– OpenGL® ES with NVIDIA Pixel Shading Extensions OpenGL® ES with NVIDIA Pixel Shading Extensions – 200 million pixels/second 3D fill rate 200 million pixels/second 3D fill rate – 2.67 million drawn triangles/second 2.67 million drawn triangles/second – 128-bit interface to internal memory 128-bit interface to internal memory – 32-bit interface to stacked memory 32-bit interface to stacked memory – Transform engine Transform engine – 40-bit color pipeline 40-bit color pipeline – 5 simultaneous textures 5 simultaneous textures – Signed, overbright color Signed, overbright color – 7 surfaces (color, Z, texture 1..5) 7 surfaces (color, Z, texture 1..5) – 16 4-bit palettes or one 8-bit palette 16 4-bit palettes or one 8-bit palette – Transform engine Transform engine – Setup engine Setup engine – Programmable pixel shading engine Programmable pixel shading engine – Bilinear/Trilinear texture filtering Bilinear/Trilinear texture filtering – Fixed point & floating point data Fixed point & floating point data – 3D rendering to XGA [1024x768] and smaller displays3D rendering to XGA [1024x768] and smaller displays

Siege OpenGL ES Demo

40

Imagenation TechnologiesImagenation Technologies

• PowerVR MBXPowerVR MBX– OpenGL ES 1.1OpenGL ES 1.1– 300 million pixels/sec 300 million pixels/sec – 3.7 million triangles/sec3.7 million triangles/sec

– Example devices: Nokia N95, SonyEricsson P1 (Lite)Example devices: Nokia N95, SonyEricsson P1 (Lite)

• PowerVR SGXPowerVR SGX– OpenGL ES 2.0OpenGL ES 2.0– 200Mpix/sec to 1200Mpix/sec200Mpix/sec to 1200Mpix/sec– 2Mpoly/sec to 13.5Mpoly/sec2Mpoly/sec to 13.5Mpoly/sec– VideosVideos

41

ARMARM

• Mali 55Mali 55– OpenGL ES 1.1OpenGL ES 1.1

• Mali 200Mali 200– OpenGL ES 2.0OpenGL ES 2.0

• FeaturesFeatures– 4X / 16X FSAA4X / 16X FSAA– Flat / Gouraud ShadingFlat / Gouraud Shading– Perspective Correct TexturingPerspective Correct Texturing– Point Sampling / Bilinear / Trilinear FilteringPoint Sampling / Bilinear / Trilinear Filtering– MipmappingMipmapping– Multi TexturingMulti Texturing– Dot3 Bump MappingDot3 Bump Mapping– Alpha BlendingAlpha Blending– Stencil Buffering (4-bit)Stencil Buffering (4-bit)– JSR184JSR184– Point SpritesPoint Sprites– 2 bit per texel Texture Compression (FLXTC)2 bit per texel Texture Compression (FLXTC)– 4 bit per texel Texture Compression (ETC)4 bit per texel Texture Compression (ETC)

• VideoVideo

42

SummarySummary

• APIsAPIs– OpenGL ES for low-level performanceOpenGL ES for low-level performance– M3G for high-level animation control M3G for high-level animation control

and scene graphand scene graph– Rapidly approaching desktop Rapidly approaching desktop

functionality (programmable pipeline)functionality (programmable pipeline)

• HardwareHardware– Major hardware players (AMD, Major hardware players (AMD,

NVIDIA, ARM, Intel)NVIDIA, ARM, Intel)– Focus on power efficiency rather than Focus on power efficiency rather than

pure performancepure performance

43

SummarySummary

44

Research on mobile 3D graphicsResearch on mobile 3D graphics

• Power efficient algorithms Power efficient algorithms – Minimize memory reading (can cost 10 times more than Minimize memory reading (can cost 10 times more than

internal chip power consumption)internal chip power consumption)– Lund University: Texture compressionLund University: Texture compression

• Benchmarking and modeling requirementsBenchmarking and modeling requirements• Remote visualizationRemote visualization• Augmented RealityAugmented Reality

45

Augmented RealityAugmented Reality

• Mix real and virtual Mix real and virtual information (i.e. not VR)information (i.e. not VR)

• Registered in 3DRegistered in 3D• Interactive (i.e. not FX)Interactive (i.e. not FX)• Why? - Avoid domain Why? - Avoid domain

switchingswitching• The world through the The world through the

computercomputer

46

Augmented Reality on Mobile PhonesAugmented Reality on Mobile Phones

• Camera feedback makes the screen Camera feedback makes the screen “transparent”“transparent”

• Track display using camera and/or Track display using camera and/or GPS + accelerometer + gyrosGPS + accelerometer + gyros

• ARToolKit:ARToolKit:

47

Example 1: Sensor Visualization (exjobb)Example 1: Sensor Visualization (exjobb)

• Purpose:Purpose:– Visualize humidity data on-siteVisualize humidity data on-site

• Idea: Idea: – Synthesize an image from a grid of sensors. Register Synthesize an image from a grid of sensors. Register

image with element. image with element.

48

Example 2: Collaboration (exjobb)Example 2: Collaboration (exjobb)

• Share a scene graph Share a scene graph over Bluetoothover Bluetooth

• Manipulate objects Manipulate objects using motion of phoneusing motion of phone

• PC-client outputs to PC-client outputs to projector or plasma projector or plasma screen for high quality screen for high quality graphics and group graphics and group viewview

49

Papers for you to readPapers for you to read

• Kari Pulli, Tomi Aarnio, Kimmo Roimela, Jani Vaarala, “Designing Graphics Kari Pulli, Tomi Aarnio, Kimmo Roimela, Jani Vaarala, “Designing Graphics Programming Interfaces for Mobile Devices”, IEEE Computer Graphics and Programming Interfaces for Mobile Devices”, IEEE Computer Graphics and Applications, Volume 25, Issue 8, Nov.-Dec. 2005.Applications, Volume 25, Issue 8, Nov.-Dec. 2005.

• Kari Pulli, “New APIs for Mobile Graphics”, SPIE Electronic Imaging 2006 Kari Pulli, “New APIs for Mobile Graphics”, SPIE Electronic Imaging 2006 Multimedia on Mobile Devices II, San Jose, USA, 2006.Multimedia on Mobile Devices II, San Jose, USA, 2006.

[Above papers at http://people.csail.mit.edu/kapu/resume.html][Above papers at http://people.csail.mit.edu/kapu/resume.html]

• Tomas Akenine-Möller, Jacob Ström, "Graphics for the Masses: A Hardware Tomas Akenine-Möller, Jacob Ström, "Graphics for the Masses: A Hardware Rasterization Architecture for Mobile Phones", ACM Transactions of Rasterization Architecture for Mobile Phones", ACM Transactions of Graphics (Proceedings of ACM SIGGRAPH), vol. 22, no. 3, pp. 801-808, Graphics (Proceedings of ACM SIGGRAPH), vol. 22, no. 3, pp. 801-808, July 2003.July 2003.

[Above paper at http://graphics.cs.lth.se/research/mobile/][Above paper at http://graphics.cs.lth.se/research/mobile/]

50

ResourcesResources

• Mobile Computer Graphics Course in LundMobile Computer Graphics Course in Lund– http://www.cs.lth.se/EDA075/http://www.cs.lth.se/EDA075/– Lots of interesting material!Lots of interesting material!

• OpenGL ES SpecificationsOpenGL ES Specifications– http://www.khronos.org/opengles/http://www.khronos.org/opengles/

• SIGGRAPH course: Developing Mobile 3D Applications With SIGGRAPH course: Developing Mobile 3D Applications With OpenGL ES and M3GOpenGL ES and M3G– http://people.csail.mit.edu/kapu/siggraph_course/http://people.csail.mit.edu/kapu/siggraph_course/

• SIGRAD05 – Swedish conference with mobile computer graphics SIGRAD05 – Swedish conference with mobile computer graphics as special topicas special topic– http://graphics.cs.lth.se/sigrad05/program.shtmlhttp://graphics.cs.lth.se/sigrad05/program.shtml

51

Resources Resources

• Mobile Data Visualization at Quinn Lab:Mobile Data Visualization at Quinn Lab:– http://mobile.sdsc.edu/http://mobile.sdsc.edu/

• SDKs and documentation for developing OpenGL ES and M3G on SDKs and documentation for developing OpenGL ES and M3G on Nokia phones (or emulator)Nokia phones (or emulator)– http://www.forum.nokia.com/http://www.forum.nokia.com/

• Benchmarks for M3G and OpenGL ES Benchmarks for M3G and OpenGL ES – http://www.glbenchmark.com/http://www.glbenchmark.com/

• IBM DeveloperWorks on M3GIBM DeveloperWorks on M3G– Immediate mode: http://www-Immediate mode: http://www-

128.ibm.com/developerworks/wireless/library/wi-mobile1/128.ibm.com/developerworks/wireless/library/wi-mobile1/– Reatined mode: http://www-Reatined mode: http://www-

128.ibm.com/developerworks/wireless/library/wi-mobile2/128.ibm.com/developerworks/wireless/library/wi-mobile2/

Documents

Mobile 3D Graphics