Tricks to Making a Realtime SurfaceView Actually Perform in Realtime - Maarten Edger

Tricks to Making a Realtime SurfaceView Actually Perform in Realtime

Maarten Edgar

Hello, my name is …

Hello, my name is …

Maarten Edgar

What we’ll cover

SurfaceViews:• Why• When• What• How• Hard earned lessons

Why use a SurfaceView?

SurfaceViewGL_SurfaceView

TextureViewSurfaceTexture

View

What is a SurfaceView?

A View which gives you access to a Surface using .getHolder(), which is drawn on a seperate thread and is double/triple buffered behind the scenes.

It cuts holes and displays underneath the window it is in.

How to use it:• Setup• Threads vs Runnables and other

control mechanisms• Loops• UI communication• Tips

Setup

• Activity/View• SurfaceView and its Thread

Setup: Activity and View@Overrideprotected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState);

// set flags as needed getWindow().setFormat(PixelFormat.RGBA_8888);

this.setVolumeControlStream(AudioManager.STREAM_MUSIC);

setContentView(R.layout.activity_game);

// get handles to the View from XML, and its Thread mCSurfaceView = (MySurfaceView) findViewById(R.id.surfaceview);

setSurfaceType(View.LAYER_TYPE_SOFTWARE);

mSurfaceViewThread = mSurfaceView.getThread();

createInputObjectPool();

Your SurfaceView class

public class ChiBlastSurfaceView extends SurfaceView implements SurfaceHolder.Callback {

public ChiBlastSurfaceView(Context context) { super(context);

mSurfaceCreated = false; touchBool = true;

// register our interest in hearing about changes to our surface SurfaceHolder holder = getHolder(); holder.addCallback(this);

myHandler = new MyInnerHandler(this);

// create thread only; it's started in surfaceCreated() thread = new ChiBlastSurfaceViewThread(holder, context, myHandler);

setFocusable(true); // make sure we get key events}

Your SurfaceView callbacks 1/3

SurfaceHolder.Callback:

@Overridepublic void surfaceCreated(SurfaceHolder holder) { // start the thread here so that we don't busy-wait in run() waiting for the surface to be created if (mSurfaceCreated == false) { createThread(holder); mSurfaceCreated = true; touchBool = true; }}



@Overridepublic void surfaceDestroyed(SurfaceHolder holder) { mSurfaceCreated = false; cleanupResource(); terminateThread();}



@Overridepublic void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

thread.setSurfaceSize(width, height);}

Setup: driving the SurfaceView

Runnables, thread and loops, oh my!

Setup: Threadpublic class ChiBlastSurfaceViewThread extends Thread { public ChiBlastSurfaceViewThread(SurfaceHolder surfaceHolder,

Context context, Handler handler) { // get handles to some important objects mSurfaceHolder = surfaceHolder; mSurfaceHolder.setFormat(PixelFormat.RGBA_8888); mContext = context;

res = context.getResources();

//any other initialization:

ops = new BitmapFactory.Options(); ops.inPurgeable = true; ops.inDensity = 0; ops.inDither = false; ops.inScaled = false; ops.inPreferredConfig = Bitmap.Config.ARGB_8888; ops.inJustDecodeBounds = false;

}

@Overridepublic void run() { while (mRun) { Canvas c = null; try { // update game state processInput(); //if (mMode == STATE_SCROLL_MAP) if (mMode != STATE_PAUSE) { updatePhysics(timeDiff); }

c = mSurfaceHolder.lockCanvas(null);

synchronized (mSurfaceHolder) { doDraw(c); }

} finally { // do this in a finally so that if an exception is thrown // during the above, we don't leave the Surface in an // inconsistent state if (c != null) { mSurfaceHolder.unlockCanvasAndPost(c); } } }}

Setup: Thread

The Thread and your Activity

What does this now mean for your Activity?

orHow do we make this fit into the

Android Lifecycle?


@Overrideprotected void onPause() { super.onPause(); // pause game when Activity pauses mSurfaceView.getThread().pause(); mSurfaceView.terminateThread();

System.gc();}


@Overrideprotected void onResume(){ super.onResume(); if (mSurfaceView.mSurfaceCreated) { mSurfaceView.createThread(mSurfaceView.getHolder()); setSurfaceType(View.LAYER_TYPE_SOFTWARE); } mSurfaceView.SetTouch(true);}


@Overrideprotected void onRestoreInstanceState(Bundle inState) { // just have the View's thread load its state from our Bundle if (mSurfaceView.mSurfaceCreated) { mSurfaceView.createThread(mSurfaceView.getHolder()); setSurfaceType(View.LAYER_TYPE_SOFTWARE); } mSurfaceViewThread.restoreState(inState);}

The main loop

• AFAFP• Fixed step

@Overridepublic void run() {

long beginTime; // the time when the cycle begun long timeDiff; // the time it took for the cycle to execute int sleepTime; // ms to sleep (<0 if we're behind) int framesSkipped; // number of frames being skipped

timeDiff = System.currentTimeMillis()+50; sleepTime = 0;

while (mRun) { Canvas c = null; try { beginTime = System.currentTimeMillis(); framesSkipped = 0; // resetting the frames skipped // update game state processInput(); //if (mMode == STATE_SCROLL_MAP) if (mMode != STATE_PAUSE) { updatePhysics(timeDiff); }

c = mSurfaceHolder.lockCanvas(null);

synchronized (mSurfaceHolder) { doDraw(c); }

The main loop 1/3

The main loop 2/3 // calculate how long did the cycle take

timeDiff = System.currentTimeMillis() - beginTime; // calculate sleep time sleepTime = (int)(FRAME_PERIOD - timeDiff);

if (sleepTime > 0) { // if sleepTime > 0 we're OK try { // send the thread to sleep for a short period // very useful for battery saving Thread.sleep(sleepTime); } catch (InterruptedException e) {} }

while (sleepTime < 0 && framesSkipped < MAX_FRAME_SKIPS) { // we need to catch up // update without rendering processInput(); if (mMode != STATE_PAUSE) { updatePhysics(timeDiff); } // add frame period to check if in next frame sleepTime += FRAME_PERIOD; framesSkipped++; }

The main loop 3/3

} finally { // do this in a finally so that if an exception is thrown // during the above, we don't leave the Surface in an // inconsistent state if (c != null) { mSurfaceHolder.unlockCanvasAndPost(c); } } }}

UI Communication

Use a Handler:

static class MyInnerHandler extends Handler { private final WeakReference<ChiBlastSurfaceView> mView;

MyInnerHandler(ChiBlastSurfaceView aView) { mView = new WeakReference<ChiBlastSurfaceView>(aView); }

@Override public void handleMessage(Message m) { ChiBlastSurfaceView theView = mView.get(); theView.mStatusText.setText(m.getData().getString("text")); if (m.getData().getInt("viz") == View.VISIBLE) { theView.mStatusText.setVisibility(View.VISIBLE); //mStatusText.setAnimation(displayTextAnim); //mStatusText.startAnimation(displayTextAnim); } else { if (m.getData().getInt("viz") == View.INVISIBLE) { theView.mStatusText.setVisibility(View.INVISIBLE); theView.mStatusText.setAnimation(null); } else if (m.getData().getInt("viz") == View.GONE) { theView.mStatusText.setVisibility(View.GONE); } } theView.mStatusText.invalidate(); }}

Setup: Cleanuppublic void terminateThread (){ boolean retry = true; thread.setRunning(false); while (retry) { try { thread.join(); retry = false; } catch (InterruptedException e) { } //break; //THIS BREAKS IT ON PUSHING HOME } //thread = null; //THIS BREAKS IT ON PUSHING HOME}

Tips

• Input buffer• Object creation• Scaling• Drawing, bitmaps and other dirty

things

Tips: input buffer in SVActivity

private void createInputObjectPool() { inputObjectPool = new ArrayBlockingQueue<InputObject>(INPUT_QUEUE_SIZE); for (int i = 0; i < INPUT_QUEUE_SIZE; i++) { inputObjectPool.add(new InputObject(inputObjectPool)); }}

public class InputObject { public static final byte EVENT_TYPE_KEY = 1; public static final byte EVENT_TYPE_TOUCH = 2; public static final int ACTION_KEY_DOWN = 1; public static final int ACTION_KEY_UP = 2; public static final int ACTION_TOUCH_DOWN = MotionEvent.ACTION_DOWN; public static final int ACTION_TOUCH_POINTER_DOWN = MotionEvent.ACTION_POINTER_DOWN; //public static final int ACTION_TOUCH_POINTER_2_DOWN = MotionEvent.ACTION_POINTER_2_DOWN; public static final int ACTION_TOUCH_MOVE = MotionEvent.ACTION_MOVE; public static final int ACTION_TOUCH_UP = MotionEvent.ACTION_UP; public static final int ACTION_TOUCH_POINTER_UP = MotionEvent.ACTION_POINTER_UP; //public static final int ACTION_TOUCH_POINTER_2_UP = MotionEvent.ACTION_POINTER_2_UP; public ArrayBlockingQueue<InputObject> pool; public byte eventType; public long time; public int action; public int keyCode; public int x; public int y; public int x2; public int y2; public int pointerID; public int pointerIndex; public int pointerIndex2;

InputObject 1/5

InputObject 2/5public InputObject(ArrayBlockingQueue<InputObject> pool) { this.pool = pool;}

public void useEvent(KeyEvent event) { eventType = EVENT_TYPE_KEY; int a = event.getAction(); switch (a) { case KeyEvent.ACTION_DOWN: action = ACTION_KEY_DOWN; break; case KeyEvent.ACTION_UP: action = ACTION_KEY_UP; break; default: action = 0; } time = event.getEventTime(); keyCode = event.getKeyCode();}

public void useEvent(MotionEvent event) { eventType = EVENT_TYPE_TOUCH; int a = event.getAction(); switch (a) { case MotionEvent.ACTION_DOWN: action = ACTION_TOUCH_DOWN; break; case MotionEvent.ACTION_POINTER_DOWN: action = ACTION_TOUCH_POINTER_DOWN; break; case MotionEvent.ACTION_POINTER_2_DOWN: action = ACTION_TOUCH_POINTER_DOWN; break; case MotionEvent.ACTION_MOVE: action = ACTION_TOUCH_MOVE; break; case MotionEvent.ACTION_UP: action = ACTION_TOUCH_UP; break; case MotionEvent.ACTION_POINTER_UP: action = ACTION_TOUCH_POINTER_UP; break; case MotionEvent.ACTION_POINTER_2_UP: action = ACTION_TOUCH_POINTER_UP; break; default: action = -1; }

InputObject 3/5

InputObject 4/5time = event.getEventTime(); pointerIndex = (event.getAction() & MotionEvent.ACTION_POINTER_ID_MASK) >> MotionEvent.ACTION_POINTER_ID_SHIFT; pointerID = event.getPointerId(pointerIndex); x = (int) event.getX(pointerIndex); y = (int) event.getY(pointerIndex); if (event.getPointerCount() > 1) { pointerIndex2 = pointerIndex== 0 ? 1 : 0; x2 = (int)event.getX(pointerIndex2); y2 = (int)event.getY(pointerIndex2); }}

InputObject 5/5 public void useEventHistory(MotionEvent event, int historyItem) { eventType = EVENT_TYPE_TOUCH; action = ACTION_TOUCH_MOVE; time = event.getHistoricalEventTime(historyItem); pointerIndex = (event.getAction() &

MotionEvent.ACTION_POINTER_ID_MASK) >> MotionEvent.ACTION_POINTER_ID_SHIFT;

pointerID = event.getPointerId(pointerIndex); x = (int) event.getHistoricalX(pointerIndex, historyItem); y = (int) event.getHistoricalY(pointerIndex, historyItem); if (event.getPointerCount() > 1) { pointerIndex2 = pointerIndex== 0 ? 1 : 0; x2 = (int) event.getHistoricalX(pointerIndex2, historyItem); y2 = (int) event.getHistoricalY(pointerIndex2, historyItem); } } public void returnToPool() { pool.add(this); }

@Overridepublic boolean onTouchEvent(MotionEvent event) { try { // history first int hist = event.getHistorySize(); if (hist > 0) { // add from oldest to newest for (int i = 0; i < hist; i++) { //for (int i = hist-1; i > -1; i--) { InputObject input = inputObjectPool.take(); input.useEventHistory(event, i); mSurfaceViewThread.feedInput(input); } } // current last InputObject input = inputObjectPool.take(); input.useEvent(event); mSurfaceViewThread.feedInput(input); } catch (InterruptedException e) { } // don't allow more than 60 motion events per second try { Thread.sleep(16); } catch (InterruptedException e) { } return true;}

Back to the activity:

public void feedInput(InputObject input) { synchronized(inputQueueMutex) { try { inputQueue.put(input); } catch (InterruptedException e) { //Log.e(TAG, e.getMessage(), e); } }}

private void processInput() { synchronized(inputQueueMutex) { ArrayBlockingQueue<InputObject> inputQueue = ChiBlastSurfaceView.inputQueue; while (!inputQueue.isEmpty()) { try { InputObject input = inputQueue.take(); if (input.eventType == InputObject.EVENT_TYPE_KEY) { //processKeyEvent(input); } else if (input.eventType == InputObject.EVENT_TYPE_TOUCH) { processMotionEvent(input); } input.returnToPool(); } catch (InterruptedException e) { //Log.e(TAG, e.getMessage(), e); } } }}

And in the SurfaceView.Thread:

Tips: object creation


Just don’t do it.


Or do it up front.

No matter how odd that sometimes may seem.

Tips: scaling

Two types of scaling:• Realtime whole view SV scaling

only works from Android N• Fixed scaling (as done in Unreal

Tournament 3)

Tips: scaling

@Overridepublic void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

//thread.setSurfaceSize(width, height);

if (mCanvasWidth != width) { int scaledWidth = (int)(width*0.75f); int scaledHeight = (int)(height*0.75f); if (scaledHeight != height) { yRatio = (float)(scaledHeight / (float)height); xRatio = (float)(scaledWidth / (float)width); } holder.setFixedSize(scaledWidth, scaledHeight); thread.setSurfaceSize(scaledWidth, scaledHeight); }}

Tips: drawing, bitmaps and other dirty things

Tips: drawing, bitmaps and other dirty things

In SurfaceView.Thread doDraw():

canvas.drawBitmap(mBackgroundImage, null, fullscreenRect, mPicPaint);

Q&A

Thank you!

Maarten [email protected]

Resources:https://source.android.com/devices/graphics/architecture.htmlhttps://github.com/google/grafika

mailto:[email protected]

https://source.android.com/devices/graphics/architecture.html

https://github.com/google/grafika

Technology

Tricks to Making a Realtime SurfaceView Actually Perform in Realtime - Maarten Edger