Ensuring quality in cloud and mobile applications

1 Copyright 2012 Gerard Meszaros InfoQ Hangzhou 2012

Ensuring Quality in Cloud and Mobile Applications

Gerard Meszaros [email protected]


Product & I.T.

I.T.

Embedded Telecom

My Background

Gerard Meszaros [email protected]

• Software developer

• Development manager

• Project Manager

• Software architect

• OOA/OOD Mentor

• Requirements (Use Case) Mentor

• XP/TDD Mentor

• Agile PM Mentor

• Test Automation Consultant & Trainer

• Lean/Agile Coach/Consultant

80’s ----- 90’s ----- 00’s


Agenda •  What’s Special About Cloud and Mobile Apps? •  What is Quality? •  A Quality Strategy •  Testing Cloud-Based Apps •  Testing Mobile Apps •  No DOH! •  Ensuring Design for Testability •  Tools, Frameworks and Patterns


Current Challenges

The Good Old Days •  Captive Audience

– Have no real choice

•  Stable requirements – Can have a single release

•  Long lead times – Can have long manual test

cycle

•  Relatively simple environment –  Just a few inputs (user, DB) – Mostly synchronous

Current Reality

•  Users Have Real Choice – Many alternatives

•  Rapid change – Highly iterative; multi releases

•  Rapid change – No time for extensive manual

testing

•  Highly Complex Environment – Many sources of inputs –  Sync and Async


DB Server Desktop

Other System

Other System

User Interface

Business Logic

OS Services

Other System

Typical Desktop Application •  UI Logic and

Business Logic run on desktop

•  Server hosts the database.

Database


Desktop Browser

Server Browser

Other System

UI Logic

Business Logic Database

Container Services

Other System

Typical Cloud-Based Application •  UI Logic runs in

browser •  Business Logic is split

between Server and Browser

•  Both use browser services

–  Widgets, Cookies, SSL,DOM rendering

•  Business logic may also interact with other systems.

–  E.g. IP-based location •  Browser-based BL

may receive data from server: –  synchronous

(request-reply) –  asynchronous

(event-based)

Business Logic

Browser Services

OS Services

(A.K.A. Software-as-a-Service or SaaS)

Dynamic scripting languages •  won’t report compile errors until run.


Device Browser

Sensors GPS Acc

Gyro

OthrApps Contacts Msg’ing Gyro Batt

Server Browser

Other System

UI Logic


Container Services

Other System

Typical Web-Based Mobile Application •  UI Logic runs in

browser •  Business Logic is split

between Server and Browser

•  Both use browser services

–  Widgets, Cookies, SSL,DOM rendering





(event-based)

Business Logic

Browser Services

Platform Services



Device

Server

Other System

UI Logic


Container Services

Other System

Typical Mobile Application •  UI Logic runs in

device •  Business Logic is split

between Device and Servers

•  Both use Platform services

–  Config, Prefs, Widgets,SSL,





(event-based)

Business Logic

Platform Services

Sensors GPS Acc

Gyro

OthrApps Contacts Msg’ing Gyro

Config Prefs

Time Prefs

Batt


Outputs / Controlables •  Screen (LCD) •  LED indicators •  Vibration motor •  Camera •  Wifi, Bluetooth, 3G,

NFC Radios (on/off)

•  Other Applications •  Add event to

Calendar •  Add/edit/read contact


Other static inputs Physical Configuration •  Screen size •  Screen density •  Roles of Physical

buttons •  Camera(s)

capabilities

User Preferences •  Language / locale •  Keyboard prefs •  Application prefs


Sources of Events Sensors •  Proximity •  Accelerometer •  GPS •  Signal Strength (Wifi,

3g, BT, NFC) •  Compass

Services •  Send message (e-mail,

SMS, MMS, IAM) •  Voice interpretation •  Output Audio •  IP Location •  Wifi Location •  GPS Location •  Get/Show Advertising •  In-App purchasing


Events & Reponses •  Low Battery •  Lost Data

Connection •  Change in data

capabilities •  Geofence crossed

•  Valid Responses •  Data returned sync •  Data returned async •  Error responses •  Low resources •  Can’t communicate


Example – Screen Layout •  Large number of available screen sizes •  Need to verify app uses screen space well •  Four different orientations •  For each screen size and orientation:

– need to verify how various screen elements are sized and placed

•  May be possible to do manually for IOS –  iPhone 1, iPhone 4, iPhone 5, iPad1/2, iPad3, iPod Touch

•  Not possible to do well manually for Android – Automation is essential; best done at component level


Example – Screen Layout •  What about changes to orientation of the

device? •  That’s 4P2 or 12 transitions we need to test

North ->South North -> East North -> West

•  And we have to run this test at various states of the app

•  So, 12 tests in 10’s or 100’s of states •  On 10’s to 100’s of device configurations

South -> North South -> East South -> West

East -> North East -> West East -> South

West -> North West -> East West -> South




Google Quality Checklists Core App Quality Quality Criteria •  Design and Interaction •  Functionality •  Performance and Stability •  Google Play Testing •  Setting Up a Test

Environment •  Test Procedures You Should Also Read •  Tablet App Quality Checklist •  Improving App Quality

Tablet App Quality 1. Test for Core App Quality 2. Optimize your layouts 3. Use the extra screen area 4. Use assets designed for

tablets 5. Adjust fonts and touch targets 6. Adjust HomeScreen widgets 7. Offer the app's full feature set 8. Don’t require hardware

features 9. Declare tablet screen support 10. Follow best practices for

publishing in Google Play http://developer.android.com/distribute/googleplay/quality/core.html http://developer.android.com/distribute/googleplay/quality/tablet.html


What Exactly is Quality?

Building the Right Thing •  Fitness-for-purpose •  Understand the users •  And their usage env’ts

Practices: •  UxD •  Usability testing •  Lean Startup stuff

Building the Thing Right •  Good architecture •  Well implemented in

code •  Well tested at all levels

Practices: •  Automated Unit Testing •  Test-Driven

Development •  ATDD •  Exploratory Testing


Traditional Test Cycle

Development Verification & Acceptance

Software

Requirements Requirements




How to Achieve Quality?


What Should We Test? •  Product X consists of components A, B, and C •  Components A and B are well tested but have

never been used together. •  Component C is not well tested

•  We need to Component-test C, •  Interaction test A vs. B •  Interaction test A vs C


A Quality Strategy •  Use automated tests at unit and component

level – to maximize confidence in quality of components.

•  Use system-level tests focusing on the logic not already tested – mostly integration tests – automated if possible, manual otherwise

•  Use Exploratory testing to find out what else could possibly go wrong. – Let testers follow their hunches to find problem areas


A word about Terminology •  Testing vs. Checking

– Checking is verifying that a behaviour occurs as expected

– Testing is seeing what happens when I do something – Automated Testing is therefor an oxymoron; it should be

automated checking. •  Direct vs Indirect Inputs (and outputs)

SUT methodA( )

methodB( ) Client

Client

Client Objects (Users)

DOC

DOC

DOC

Depended-on Components

Client

Indirect Output

Input

Output

Direct Input

Indirect Input

Direct Output

System Under Test


Test Automation Pyramid

Unit Tests

Component Tests

System Tests

Pyramid originally proposed by Mike Cohn

Exploratory Tests •  Tools to support effective

exploratory testing •  A small number of tests

for the entire application & workflow – Ensure application(s) support

users’ requirements •  Medium number of

functional tests for major components – Verify integration of units

•  Large numbers of very small unit tests – Ensures integrity of code




Server Browser

Other System

UI Logic


Container Services

Other System

Testing Cloud-Based Application •  Natural interface

between Browser & Server facilitates testing –  Server testing –  Browser testing with

Server stubbed out •  Component tests for

Browser Business Logic bypassing UI – Requires API on BL

•  Also need unit tests for Business Logic code

Business Logic

Browser Services

OS Services



Server Browser

Other System

UI Logic


Container Services

Other System






Business Logic

Browser Services

OS Services


Server Test


Server Browser

Other System

UI Logic


Container Services

Other System






Business Logic

Browser Services

OS Services


Whole Client Test

Server Test Stub


Server Browser

Other System

UI Logic


Container Services

Other System






Business Logic

Browser Services

OS Services


Unit Test

Test Stub Unit

Unit Test

Test Stub Unit

Unit Test

Test Stub Unit

Unit Test

Test Stub Unit


Server Browser

Other System

UI Logic


Container Services

Other System






Business Logic

Browser Services

OS Services


Business Logic Test

Server Test Stub

Platform Services Test Stub


Server Browser

Other System

UI Logic


Container Services

Other System






Business Logic

Browser Services

OS Services


Server Logic Test

DB Test Stub



Server Browser

Other System

UI Logic


Container Services

Other System






Business Logic

Browser Services

OS Services


Whole System

Test


Special Considerations •  Different Browsers and Browser-Versions

– Eg. Rendering behaviour, widget behavior, etc. •  Different Host-OSes

– For OS-based services such as file choosers, etc. •  Different languages & locales •  Browser resizing


Server Device

Other System

UI Logic


Container Services

Other System

Testing Browser-Based Mobile Application Similar to testing

Cloud-based apps but:

•  Different browsers & Screen sizes

•  Need to test with events from relevant sensors –  E.g. Orientations

and changes

Business Logic

Platform Services

Sensors GPS Acc

Gyro


Config Prefs

Time Prefs

Batt


Server Device

Other System

UI Logic


Container Services

Other System

Testing Browser-Based Mobile Application Similar to testing

Cloud-based apps but:

•  Different browsers & Screen sizes

•  Need to test with events from relevant sensors –  E.g. Orientations

and changes

Business Logic

Platform Services

Sensors GPS Acc

Gyro


Config Prefs

Time Prefs

Batt

Business Logic Test

Server Test Stub



Server Device

Other System

UI Logic


Container Services

Other System

Testing Native Mobile Application •  To test device BL,

need to: –  Provide API

bypassing UI –  Trap outgoing

reqests to sensors and services

–  Inject responses from same

•  Need ability to: –  Stub out platform

services, sensors and other apps

•  Need unit tests for all code

Business Logic

Platform Services

Sensors GPS Acc

Gyro


Config Prefs

Time Prefs

Batt


Server Device

Other System

UI Logic


Container Services

Other System









Business Logic

Platform Services

Sensors GPS Acc

Gyro


Config Prefs

Time Prefs

Batt

Business Logic Test

Server Test Stub



Server Device

Other System

UI Logic


Container Services

Other System









•  Characterization Tests to detect changes in dependencies

Business Logic

Platform Services

Sensors GPS Acc

Gyro


Config Prefs

Time Prefs

Batt

Dependency Characterization

Tests


Agenda • What’s Special About Cloud and Mobile

Apps? • What is Quality? • A Quality Strategy •  Testing Cloud-Based Apps •  Testing Mobile Apps • No DOH! •  Ensuring Design for Testability •  Tools, Frameworks and Patterns


DOH! •  DOH! = Debugging on Hardware •  Should be avoided because •  Slow, tedious, non-deterministic •  Complete code coverage is impossible

because we cannot cause on-demand: – Hardware errors – “Impossible” return codes or exceptions – All but most common events from sensors

You have to ask yourself the following question: Do I feel lucky?


James Grenning on DOH! •  Reasons why DOH is not feasible:

– There is no hardware, it’s being concurrently developed. – The hardware is expensive, so team members have to

share it. – DOH is slow, with tools that may be less capable than

the development system native tools. – The hardware has bugs, as if DOH! isn’t slow enough;

buggy hardware makes it extra difficult. – Building code for the target is slow. – Downloading to the target is slow. – The embedded processor is probably slow too.


Test in IDE (Automated)

Test in Emulator

(Automated)

Test on Hardware (Automated)

Test in Acceptance

(manual)

How No DOH? •  How do we avoid Debugging on Hardware?

Unit Tests Component Tests

System Tests

Which tests:

When: Day 1 As soon as Available Weekly


Test in IDE (Automated)

Test in Emulator

(Automated)

Test on Hardware (Automated)

Test in Acceptance

(manual)

No DOH How •  How do we avoid Debugging on Hardware?

Frequency: Every few minutes Daily Weekly+

Each code change Per component test Per user story





How Do We Ensure Testability? “Design for Testability”, then code •  Write tests last •  And hope it’s easy

Or •  Code for Testability •  Via Test-Driven Development (TDD)


TDD Rhythm

Test Code •  First Test •  Second Test •  Third Test •  Fourth Test

Product Code •  Hard-coded method •  Introduce variable •  Introduce conditional •  Surround with a loop

Just Like Double-Entry Booking: •  An entry on the test side for each entry on

the Prod side


Test - After Architecture

System Under Test

•  Must test through User Interface

Should we Notify? Process

Transaction

Configure Notification Threshold

Notification Rules

Transaction Interface

Configuration User

Interface

Notification Log

Workflow Test

Do Notification.


System Under Test

Test-Driven Architecture •  Need to provide API’s to invoke functionality

directly

•  And ways to stub out dependencies

Should we Notify?

Do Notification.

Process Transaction


Notification Rules

Notification Log


Configuration User

Interface Workflow

Test Test Stub


Test-Driven Architecture •  “What kind of tests do I want to be able to automate?” •  “Which component would be responsible for that part

of the behavior?”

Should we Notify?

Do Notification.

Process Transaction


Notification Rules

Notification Log


Configuration User

Interface

Configuration TX Test


Test-Driven Architecture •  “What other components would that component

depend on?” •  “How can I break that dependency when component

testing this component?”

•  With the right architecture, automating these tests is trivial

Should we Notify?

Do Notification.

Process Transaction


Notification Rules

Notification Log


Configuration User

Interface Notification Rule Test

Notification Method Test

Notification Rules Fake


Unit Tests

Component Tests

System Tests

Workflow

Transactions

Behavior Specification at Right Level •  Specify broad scope at minimum detail

– E.g. Use least detail when specifying workflow •  Specify most detailed req’ts at narrowest scope

– E.g. Don’t use workflow when specifying business rules

Make examples / tests easy to

understand and easy to write

Broad Narrow Scope

Det

ail

High

Lo

w

Business Rules

Too vague

Too much detail Unmaintainable

Unit Tests





Tools, Frameworks & Patterns •  Tools come & go

– Different ecosystems (Android, IOS, BBOS, WP, etc.) – New, better tools appear regularly

•  Strategies persist – Need to define strategy first

•  Then choose tools to support the strategy •  And Patterns for testable designs

Avoids “Impedance Mismatch” between tools and strategy


Framework Issues •  Many frameworks not designed for testability •  May need to introduce level of indirection to

API (via Proxy or Adapter) •  May need to extract Plain Ordinary Testable

Object (POTO) leaving a Humble Object that subclasses framework base class


Extract POTO - Before

Framework Plug-in

ActionBase

MyAction

doAction()

MyActionTest

testDoAction()

tests Rules

ActionContext

+rules

getRules()


Framework Plug-in

Extract POTO - After

Runs in device; Tests “wrapping”

(Optional)

Runs in IDE; Tests “business logic”

MyActionHOTest

testDoAction()

ActionBase

MyActionHO

doAction()

tests

MyActionPoto

doAction(rules)

MyActionPotoTest

testDoAction() tests

Rules

ActionContext

+rules

Humble Object

Testable Object

Rules

Fake Rules

Simple design change; Huge testability improvement!

getRules()


Conclusions •  Mobile and Cloud-Based Applications come

with special challenges – While business logic tends to be simpler, constraints

and variations make design and testing more challenging

•  Testing-in Quality is not an option – There are too many potentially interacting dimensions – Many are not controllable from outside the black box

•  Need to avoid Debugging on Hardware – Test as much as possible “off device” – Only deploy code you know works correctly


Conclusions •  Design-for-Testability is essential for success

– Otherwise, tests will be too expensive to execute and/or automate

•  Beware tools/frameworks that make DfT difficult – May need to decouple from framework to ensure

testability – But extra effort is compensated for by better quality and

productivity


What Can You Do? •  Product Owner:

– Break work into manageble chunks – Provide examples/tests for each chunk to drive

development – Insist that team automates the tests

•  Tester/BA: – Provide tests to developers Before they start dev – Insist on seeing the results of the automate tests before

you start testing – Focus manual testing on areas not covered by

automated tests


What Can You Do? Developer: •  Ask for tests from PO/Testers before agreeing to build •  Discuss the kinds of tests that they will provide •  Include the work to automate the tests in any

estimates you provide •  Implement the automation before you develop the

code, not after! •  Use the automated tests to drive development •  Don’t recreate provided tests at unit test level; find

ways of reusing business rule tests as unit tests •  Think/Do Design-for-Testability •  If you find yourself in an interactive debugger, ask

yourself: “What unit tests would tell me the same information and prevent this from happening again?”


References •  James Grenning “Don’t Let Embedded Tool Chain

Slow You Down” –  http://www.renaissancesoftware.net/blog/archives/41

•  Michael Bolton “Testing vs. Checking” –  http://www.developsense.com/blog/2009/08/testing-vs-checking/

•  Dion Johnson “Cyber-dueling Over ‘Check’ vs. ‘Test’ and Other Semantics” –  http://www.automatedtestinginstitute.com/home/index.php?

option=com_content&task=view&id=1373 •  Gerard Meszaros “Humble Object”

–  http://xunitpatterns.com/Humble%20Object.html

•  Homer Simpson “Doh!” –  http://en.wikipedia.org/wiki/D%27oh!


Resources •  http://www.mobileapptesting.com/ •  For Success, Build Record/Playback into Your

Application - StarEast 2008 – http://builtinrecordandplayback.xunitpatterns.com

– http://mobile.testAutomationPatterns.com


Thank You! Gerard Meszaros

[email protected] http://www.xunitpatterns.com

Slides:

http://infoQ2012hz.testautomationpatterns.com Call me when you: •  Want to transition to Agile or Lean •  Want to do Agile or Lean better •  Want to teach developers how to test •  Need help with test automation strategy •  Want to improve your test automation

Jolt Productivity Award winner - Technical Books

Coming Soon:

Documents

Ensuring quality in cloud and mobile applications