Testing Sucks, But It Doesn't Have To

TESTING SUCKS

NOVEMBER 2011

NICHOLAS FITZROY-DALE

TESTING IS LIKE FLOSSING

© 2011 -‐ CONFIDENTIAL






3 ©2011 Apkudo Inc. www.apkudo.com

61% CRASHES

54% WON’T INSTALL

99%


“I TEST MY APP!”

99%


“I TEST MY APP!”

O RLY?

85% MANUAL

6 ©2011 apkudo, inc. Confidential www.apkudo.com

OH NOES


AD HOC TESTING

Sucks Keeping up with device versions Testing across various devices

AUTOMATED TESTING Tedious JUnit, Robotium Test plan


MONKEY! A good start. Simulates inputs. Completely automated.


MONKEY OVERVIEW

Ac$vity Manager Start / stop ac:vi:es Get list of running ac:vi:es

Window Manager Send touch and key events to foreground app

Package Manager Install and uninstall packages Get list of package ac:vi:es

ocean-‐flisch.deviantart.com


MONKEY OVERVIEW

Ac$vity Manager setActivityController()

Window Manager inject*Event()

Package Manager queryIntentActivities()


MONKEY SEE MONKEY DO •  Monkey injects “events” - what is an event?

– KeyEvent: Hardware buttons and keyboard – MotionEvent: X, Y, and touches – FlipEvent, NetworkEvent, PowerEvent (mostly

legacy)

Security

•  Monkey uses standard Android permissions... that no user apps can use!

public static final String SET_ACTIVITY_WATCHER Since: API Level 1 Allows an application to watch and control how activities are started globally in the system. Only for is in debugging (usually the monkey command).

public static final String INJECT_EVENTS Since: API Level 1 Allows an application to inject user events (keys, touch, trackball) into the event stream and deliver them to ANY window. Without this permission, you can only deliver events to windows in your own process. Very few applications should need to use this permission.


SECURITY

Monkey vs Monkey Runner

•  Monkey is for randomized testing, and runs on the device

•  Monkey Runner is for scripting your own tests, and runs on a laptop / desktop

•  In fact, Monkey Runner uses Monkey internally

MONKEY VS MONKEY RUNNER

Monkey on Angry Birds

adb shell monkey –v -p com.rovio.angrybirds -s 39 --throttle 10 --pct-touch 100 5000

adb shell monkey -p com.rovio.angrybirds 500014 ©2011 apkudo, inc. Confidential www.apkudo.com

THE BIRDS, THEY ARE ANGRY!

What can Monkey find?

•  Key and motion events •  Crash information if applicable, as traceback


MONKEY RESULTS :AllowPackage: com.rovio.angrybirds :IncludeCategory: android.intent.category.LAUNCHER :IncludeCategory: android.intent.category.MONKEY :Switch: #Intent;ac:on=android.intent.ac:on.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10000000;component=com.rovio.angrybirds/com.rovio.ka3d.App;end // Allowing start of Intent { act=android.intent.ac:on.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.rovio.angrybirds/com.rovio.ka3d.App } in package com.rovio.angrybirds :Sending Pointer ACTION_DOWN x=554.0 y=357.0 :Sending Pointer ACTION_UP x=558.0 y=350.0 // Rejec:ng start of Intent { act=android.intent.ac:on.CALL_BUTTON cmp=com.android.contacts/.DialtactsAc:vity } in package com.android.contacts :Sending Pointer ACTION_DOWN x=696.0 y=280.0 :Sending Pointer ACTION_UP x=709.0 y=271.0 :Sending Pointer ACTION_MOVE x=-‐1.0 y=1.0 :Sending Pointer ACTION_DOWN x=348.0 y=10.0

What can Monkey find?

•  Weird UI sequences – Press button 2 before button 1

•  Unexpected inputs – E.G. keyboard input during a Toast

•  User-interface race conditions •  Activity life-cycle errors


WHAT CAN MONKEY FIND?

Weird UI sequences

•  Monkey presses UI controls in an unexpected way

•  Example: Comic Rack: selecting “Options” immediately after application start

•  Resolution: Always check preconditions in UI callback functions


WEIRD UI SEQUENCES

Unexpected inputs

•  Monkey simulates keypresses that the application doesn’t expect

•  Example: multiple keypresses on the same button

•  Resolution: Again, be paranoid when entering UI callback functions


UNEXPECTED INPUTS

Race condi:ons

•  Monkey can press UI elements very quickly, triggering race conditions

•  Example: I Heart Radio: PlayerController not initialized

•  Resolution: limit use of threads. When they are necessary, perform appropriate locking.

•  Same applies to networking.


RACE CONDITIONS

Ac:vity life-‐cycle errors

•  Monkey performs unexpected activity life-cycle actions

•  Example: Shazam: Unable to pause activity •  Resolution: Become tiresomely familiar

with the Android activity life cycle.

ACTIVITY LIFE CYCLE ERRORS

Mul:-‐device Monkey

•  Monkey is really good at testing across multiple devices:

•  Correct Market permissions – appropriate-architecture native code, eg armv7

•  Unexpected hardware properties •  “Weird” hardware failures

MULTI-DEVICE MONKEY

Mul:-‐device Monkey

•  OpenGL ES version and features •  Dependencies on system libraries •  Incorrect or missing minSdkVersion

CORRECT MARKET FILTERS

Unexpected hardware proper:es

•  The device has an unexpected hardware feature (e.g. a strange screen resolution)

•  Examples: RDefense Free, Zombie Killer •  Best resolution: Test on multiple devices

UNEXPECTED HARDWARE PROPERTIES

Weird hardware failures

•  Not all devices “just work” •  Example: PaperCamera doesn’t handle

RuntimeError from takePicture() •  Resolution: Test on multiple devices. Be

wary of Java unchecked errors.


WEIRD HARDWRE FAILURES

Limita:ons

•  False positives – “Impossible” sequences of actions / speeds

•  Very limited testing strategy – What if most of your app is hidden behind a login

screen? •  Some Android pain points not covered

– E.G. screen rotation, layout issues


MONKEY LIMITATIONS

Apkudo

•  “See your app run on every Android device” – Monkey test on 289 phones and tablets – Free – Private beta right now: email

[email protected] if interested


AND APKUDO

Thank you. NFD@ .COM

Education

Testing Sucks, But It Doesn't Have To