Java.sun.com/javaone/sf | 2004 JavaOne SM Conference | Session TS-2121 1 Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli

java.sun.com/javaone/sf

| 2004 JavaOneSM Conference | Session TS-21211

Advanced Game Development with the Mobile 3D Graphics API

Tomi Aarnio, Kari PulliNokia Research Center

| 2004 JavaOneSM Conference | Session TS-2121 2

Gain an insight to the Mobile 3D Graphics API and take your games to the next level


Prerequisites

• Fundamentals of 3D graphics

• OpenGL® or some other modern 3D API

• Java™ technology, preferably MIDP


Objectives

• Get a quick overview of the whole API

• Gain a deeper understanding of selected topics

• Learn practical tricks not found in the spec


Agenda

Why should I care?

Getting started

Low level features

Bitmaps and texturing

Scene graph

Dynamic meshes

Animation


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


Mobile 3D Graphics API (M3G)

• Also known as JSR-184

• Designed for mobile devices─ Primarily CLDC / MIDP─ But also CDC

Mobile 3D Graphics API

Java Virtual Machine (CLDC 1.1)

MIDP

Midlets

Graphics Hardware


Overcome the performance barrier

00.10.20.30.40.50.60.70.80.9

1

Vertex transformation Image downsampling

Rel

ativ

e sp

eed

Native code

CLDC 1.1 HI

Jazelle™

KVM

Benchmarked on an ARM9 processor

Native (C/C++) vs. Java on mobiles


Leverage hardware acceleration

• Benefits for developers─ Mask hardware differences (3D HW, FPU, DSP,…)─ Exploit any new hardware automatically

• Benefits for hardware vendors─ Gives a concrete target which features to offer


Speed up development cycles

• Separate content from code─ Quick prototyping without programming─ Artists create looks and behavior─ Export all in a binary file format─ Load the file, populate scene graph

• Commonly needed functionality built in─ High-level functionality such as scene graph,

keyframe animation


Why a new standard?

• OpenGL (ES) is too low-level─ Lots of Java code needed for simple things

• Java 3D™ API is too bloated─ A hundred times larger than M3G─ Does not fit together with MIDP─ Tried and failed, but…

• Now knew what we wanted!─ Basic Java 3D™ ideas: nodes, scene graph─ Add file format, keyframe animation─ Remain compatible with OpenGL ES


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


Key classes

World

Graphics3D

Loader

3D graphics context Performs all rendering

Scene graph root node

Can load individual objects and entire scene graphs (M3G and PNG files)


Graphics3D

• Contains global state─ Target surface, viewport, depth buffer─ Camera, light sources─ Rendering quality hints

• Each renderable object has its own local state─ Geometry and appearance (material, textures, etc.)─ Transformation relative to parent or world


Graphics3D: Rendering modes

• Retained mode─ Render a scene graph, rooted by the World─ Take the Camera and Lights from the World

• Immediate mode─ Render a branch or an individual node at a time─ Explicitly give the Camera and Lights to Graphics3D


Graphics3D: How-To-Use

• Bind a target to it, render, release the target

• Tip: Never mess with a bound target

• Tip: Graphics3D is a singleton (threads!)

void paint(Graphics g) {

myGraphics3D.bindTarget(g);

myGraphics3D.render(world);

myGraphics3D.releaseTarget();

}


Graphics3D: Rendering targets

Graphics Canvas

Image

CustomItem

Graphics3D Image2D

World

Can render to textures also

M3G

MIDP


• Everything is synchronous─ A method returns only when it’s done─ No separate thread for renderer or loader

• There are no callbacks─ No abstract methods, interfaces, or events─ Tip: Do not even try to override any methods

• Scene update is decoupled from rendering─ render : draws the scene, no side-effects─ animate : updates the scene to the given time─ align : applies auto-alignments, e.g., billboards

Things to keep in mind


“Hello, World”

import javax.microedition.midlet.MIDlet;import javax.microedition.lcdui.Display;import javax.microedition.lcdui.game.GameCanvas;import javax.microedition.m3g.*;

public class Player extends MIDlet{ public void pauseApp() {}

public void destroyApp(boolean b) {}

public void startApp() { PlayerCanvas canvas = new PlayerCanvas(true); Display.getDisplay(this).setCurrent(canvas); try { canvas.run(); } catch (Exception e) {} notifyDestroyed(); }}

A simplified animation player


“Hello, World”

class PlayerCanvas extends GameCanvas { PlayerCanvas(boolean suppress){super(suppress);}

public void run() throws Exception { Graphics3D g3d = Graphics3D.getInstance(); World w = (World) Loader.load("/file.m3g")[0]; long start, elapsed, time = 0; while (getKeyStates() == 0) { start = System.currentTimeMillis(); g3d.bindTarget(getGraphics()); try { w.animate(time); g3d.render(w); } finally { g3d.releaseTarget(); } flushGraphics(); elapsed = System.currentTimeMillis()-start; time += (elapsed < 100) ? 100 : (int)elapsed; if (elapsed < 100) Thread.sleep(100-elapsed); } }}


Demo

Animation playback

21


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


Renderable Objects

Mesh

Sprite3D

Made of triangle stripsBase class for meshes (two subclasses)

2D image placed in 3D spaceGood for labels, etc.


Mesh

Mesh VertexBuffer coordinates

normals

colors

texcoords

IndexBuffer

Appearance

Composed of submeshes

• Common buffer of vertex data

• Submesh has triangle strip indices to vertices

• One Appearance for each submesh


Appearance components

CompositingMode

Material colors for lightingCan track per-vertex colors

PolygonMode

Fog

Texture2D

MaterialBlending, depth bufferingAlpha testing, masking

Winding, culling, shadingPerspective correction hint

Fades colors based on distanceLinear and exponential mode

Texture matrix, blending, filteringMultitexturing: One Texture2D for each unit


Sprite3D

• Scaled mode for billboards, trees, etc.

• Unscaled mode for text labels, icons, etc.

Image2D

2D image with a position in 3D space

Sprite3D Appearance

Image2D

CompositingMode

Fog


Rendering tricks

• Use layers to impose rendering order─ Appearance contains a layer index (integer)─ Defines a global ordering for submeshes and sprites─ Tip: Disable z-buffering for sky boxes, use layers

• Tip: Maximize triangle strip length─ Even if it requires adding degenerate triangles─ Better benefits from vertex caching


Example: Rotating cube

// Corners of a cube as (X,Y,Z) tripletsstatic short[] cubeVertices = { -1, -1, 1, 1, -1, 1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, -1, -1, -1, 1, -1, 1, 1, -1};

// A color for each corner as an (R,G,B) tripletstatic byte[] cubeColors = { (byte) 255, (byte) 0, (byte) 0, // red … (byte) 0, (byte) 255, (byte) 0, // green};

// Define the cube as a single triangle stripstatic int[] indices = { 6,7,4,5,1,7,3,6,2,4,0,1,2,3 };static int[] stripLen = { 14 };

Define raw data for a cube



// Fill in VertexArrays from short[] and byte[]int numVertices = vertices.length/3;VertexArray pos = new VertexArray(numVertices, 3, 2);VertexArray col = new VertexArray(numVertices, 3, 1);pos.set(0, numVertices, cubeVertices);col.set(0, numVertices, cubeColors);

// Attach the VertexArrays to a VertexBuffer// Note the scale (1.0) and bias (0,0,0)vertices = new VertexBuffer();vertices.setPositions(pos, 1.0f, null);vertices.setColors(col);

// Fill in the triangle striptriangles = new TriangleStripArray(cubeIndices, stripLen);

// Create a Mesh with default Appearancecube = new Mesh(vertices, triangles, new Appearance());

Copy raw data into Objects



Camera cam = new Camera();

// 60-degree field of view, screen aspect ratio// Near clipping plane at 1.0, far plane at 1000

float aspect = (float) getWidth() / (float) getHeight();cam.setPerspective(60.0f, aspect, 1.0f, 1000.0f);

// Place the camera at z=5.0 in world space// View vector is along the negative Z axis

transform = new Transform();transform.postTranslate(0.0f, 0.0f, 5.0f);

g3d = Graphics3D.getInstance();g3d.setCamera(cam, transform);

Set up a Camera



g3d = Graphics3D.getInstance();g3d.bindTarget(getGraphics());

try { g3d.clear(null); g3d.render(cube, transform);} finally { g3d.releaseTarget(); }

xAngle += 1; yAngle += 2; zAngle += 3;transform.setIdentity();transform.postRotate(xAngle, 1.0f, 0.0f, 0.0f);transform.postRotate(yAngle, 0.0f, 1.0f, 0.0f);transform.postRotate(zAngle, 0.0f, 0.0f, 1.0f);

Clear the buffers, render, animate


Demo

Rotating cube

32


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


Add bitmap graphics

• 3D graphics is mostly vector graphics─ Mathematical shape definitions─ Define camera, lights, simulate─ “Computer graphics look”, pretty boring

• Bitmap graphics adds visual richness─ Background images─ Texture maps on surfaces─ Sprites


Texturing tricks

• Tip: Use light maps for lighting effects─ Usually faster than per-vertex lighting─ Use luminance textures, not RGB─ Multitexturing is your friend

• Tip: Use the perspective correction hint─ Almost always faster than adding more vertices─ But first check if the implementation supports it!


More bitmap tricks

• Tip: Use background images ─ Can be scaled, tiled and scrolled very flexibly─ Generally much faster than sky boxes or similar

• Tip: Use sprites as impostors, particles─ Generally (much) faster than textured quads─ Unscaled mode is (much) faster than scaled

• Tip: Load your images with the Loader─ Loader.load(“/img.png”) outputs Image2D─ Going through MIDP Image is a waste of memory


Demo

Backgrounds and texturing

37


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


The scene graph

SkinnedMesh

Group

Group

Group

Mesh

Sprite

Light

World

Group Camera

Group MorphingMesh

Actually, it’s just a tree

Not allowed!


VertexArray

Background

Image2D

VertexBuffer

IndexBuffer

Appearance

Image2D

Texture2D

MeshGroup

Mesh Mesh

Group Light Camera

World

How to find an object in the scene

0

12 3 4

4

5 6

1

0

6

7 89 10

11

UserID defaults to 0

UserIDs not necessarily unique

find(4)

Diagram courtesy of Sean Ellis, Superscape

Mesh4


• From this node to the parent node─ Ignored for the root node

• Composed of four parts─ Translation T─ Orientation R─ Non-uniform scale S─ Generic 3x4 matrix M

• Composite: C = T R S M

Node transformations

Group

Group

Group

Mesh

Sprite

C

CC

C C

WorldC


Node transformations

• Tip: Keep the transformations simple─ Favor the T R S components over M─ Avoid non-uniform scales in S

• Tip: How to scale along an arbitrary axis─ There is no direct support (C = T R N-1 S N M)─ Method 1: R’ = R N-1 and M’ = N M─ Method 2: Use an extra Group node

• Tip: How to rotate about an arbitrary point─ Again, no direct support (C = T P-1 R P S M)─ Method 1: Use an extra Group node─ Method 2: If S is identity, combine into T and M


Example: Set up a hierarchy

private Group group1, group2;private Mesh cube1, cube2, cube3;…cube1 = new Mesh(cubeVB, triangles, appearance);cube2 = new Mesh(cubeVB, triangles, appearance);cube3 = new Mesh(cubeVB, triangles, appearance);

group1 = new Group();group2 = new Group();

group1.addChild(cube1);group1.addChild(cube2);group1.addChild(group2);group2.addChild(cube3);

group2

group1

cube1

cube3

cube2


Animate and render the hierarchy

g3d.render(group1, null);…cube1.setOrientation( yAngle, 0.0f, 1.0f, 0.0f );cube1.setTranslation( 0.0f, 2.0f, 0.0f );

cube2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f );cube2.setTranslation( 0.0f,-2.0f, 0.0f );

group2.setOrientation(-yAngle, 0.0f, 1.0f, 0.0f );group2.setTranslation( -2.0f, 0.0f, 0.0f );

cube3.setOrientation( zAngle, 0.0f, 0.0f, 1.0f );cube3.setTranslation( -2.0f, 0.0f, 0.0f );cube3.setScale( 0.5f, 0.5f, 0.5f );


Demo

Hierarchic transformations

45


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


MorphingMesh

• A base mesh and several morph targets

• Result = weighted sum of morph deltas

• Can morph any vertex attribute─ Vertex positions─ Colors─ Normals─ Texture coordinates

i

iiw BTBR

For vertex morphing animation


MorphingMesh

Base Target 1eyes closed

Target 2mouth closed

Animate eyesand mouth

independently

Example: Animating a rabbit’s face


SkinnedMesh

i

iii vwv BM'

Articulated characters without cracks at joints

• Stretch a mesh over a hierarchic “skeleton”─ The skeleton consists of scene graph nodes─ Each node (“bone”) defines a transformation─ Each vertex is linked to one or more bones


SkinnedMesh

Neutral pose, bones at rest

Bone BBone A

"skin"shared vertex,

weights = (0.5, 0.5)non-sharedvertex

Weighted skinning


SkinnedMesh

Bone B rotated 90 degrees

Weighted skinning


SkinnedMesh

No skinning Local skinningone bone per vertex

Smooth skinningtwo bones per vertex

Example: Animating an arm


Agenda

Why should I care?

Getting started

Low level features


Scene graph

Dynamic meshes

Animation


Animation: Relevant classes

KeyframeSequence

AnimationController

AnimationTrack

A link between sequence, controller and target

Object3D

Base class for all objects that can be animated

Controls the playback of one or more animations

Storage for keyframesDefines interpolation mode


KeyframeSequence

Keyframe is a time and the value of a property at that time

Can store any number of keyframes

Several keyframe interpolation modes

Can be open or closed (looping)

sequence timet

v

KeyframeSequence



AnimationController

Can control several keyframed animations together

Determines relationship between world time and sequence time

0 dsequence time

world timet

0

t

s

0 dsequence time0 dsequence time

AnimationController



Animation

Relates animation controller, keyframe sequence, and object property together.

0 dsequence time

Identifies animated property on this object

Call toanimate(worldTime)

sv

Calculate sequence time from world time

Look up value at this sequence time

Apply value to animated property


Object3D

AnimationTrack

AnimationController

KeyframeSequence


Animation

• Tip: You can read back the animated values─ Use the animation engine for anything you want─ Much faster than doing interpolation in Java─ Especially in case of quaternions and splines


Example: Set up keyframes

KeyframeSequence createKeyframeSequence() {

// 10 keys, 3 components, spline interpolation KeyframeSequence ks; ks = new KeyframeSequence(10,3,KeyframeSequence.SPLINE);

// x grows linearly, y wiggles up-down, z remains 0.0f float[] tb = { 0.0f, 0.0f, 0.0f }; tb[0] = -4.0f; tb[1] = 0.0f; ks.setKeyframe(0, 0, tb); tb[0] = -3.0f; tb[1] = 2.0f; ks.setKeyframe(1, 10, tb); … tb[0] = 5.0f; tb[1] = 0.0f; ks.setKeyframe(9, 90, tb);

ks.setDuration(100); // sequence duration in local time ks.setValidRange(0, 9); // all 10 keyframes are valid ks.setRepeatMode(KeyframeSequence.LOOP); return ks;}


Example: Set up controller

controller = new AnimationController();

// The animation is active between these world timescontroller.setActiveInterval(0, 1500);

// Set speed, scale “around” given time (0)controller.setSpeed(2.0f, 0);

// Create animation track with keyframes & controllerks = createKeyframeSequence();at = new AnimationTrack(ks,AnimationTrack.TRANSLATION); at.setController(controller);

// Attach it to our cubecube.addAnimationTrack(at);


Demo

Keyframe animation

61


Summary

• M3G is the 3D API for J2ME™ technology─ Will be supported in millions of devices

• Not just an API─ M3G also defines a binary file format

• M3G is flexible─ Using the immediate mode gives you full control─ High-level features make your code faster and smaller


Demo

3D snowboarding

63


For More Information

• M3G specificationwww.forum.nokia.com/java/jsr184

• SDKs, additional documentationwww.forum.nokia.com/ (click “Series 40”)developer.superscape.com/ (registration needed)

• M3G content creation toolswww.discreet.com/www.superscape.com/


Q&A

Thanks: Kari Kangas, Sean Ellis, Nokia M3G team, JSR-184 Expert Group

65

java.sun.com/javaone/sf


Advanced Game Development with the Mobile 3D Graphics API

Tomi Aarnio, Kari PulliNokia Research Center

Documents

Java.sun.com/javaone/sf | 2004 JavaOne SM Conference | Session TS-2121 1 Advanced Game Development with the Mobile 3D Graphics API Tomi Aarnio, Kari Pulli