Webinar app testing and distribution

1  

30 MINUTEN Welcome! How to build your Mobile App Competence Center? Webinar Series.. #5 App Testing and Distribution Remco de Vos, Project Manager Peter Broekroelofs, CTO

Agenda

2

•  A Mobile Maturity Model for Your App Center - Summary •  App Test Process and various elements and roles in testing •  Automatic Testing •  Our Approach to testing in various phases (definition, development, acceptance) •  App Distribution •  App Lifecycle Management

Companies struggle in transition to strategic mobility

3

Opportunistic Strategic Mobile-First

The majority of companies A small minority of companies Very few, if any, companies

A reactive IT department

Mobility Center of Excellence: C-level attention, self-empowered lines of business, a responsive IT department

Low

Organization-wide strategic focus

Medium High

Most of the time customers only. Some-times siloed employee classes, typically field and sales forces and applications addressing basic informational interactions

Limited extendibility of architecture

Addresses large subsets of both dedicatedly and occasional mobile workers and more sophisticated offerings to customers. Transactional driven

Affects all mobile workers and internal activities and sophisticated customer engagement

Common architecture for mobility

Common architecture for mobility and integrated into most IT business processes

Siloed point solutions

Sophisticated administration and management tools; voice, data and integrated communications services

Integrated platform capabilities and ubiquitous connectivity

Few formal policies with decisions heavily user-influenced

Policy-driven approach for management, security and compliance

Policy-driven and 'factory' approach to mobile innovation, re-casting business workflows

Proportion of companies

Mobile strategy center of gravity

Level of business model innovation

Users

Architecture

Technologies

Policies

The Opportunistic Market in Transition

Experimentation is giving way to a more thoughtful approach to mobility. Organizations are taking a step back and rethinking how best to maximize the value of mobility. 3  

1  "The early days of mobile adoption were characterized by experimentation and unfettered departmental demand for mobile apps. These apps, funded by the business, tend to be natively developed, and are built quickly and cheaply and often without coordination with the rest of the organization or a view to long-term sustainment and value maximization.

2  Such experimentation and piloting are necessary for organizations to test and learn about mobility, with the lessons learned that enterprise mobility can show significant business value, and that not embracing enterprise mobility will put the organization at a competitive disadvantage.

Source:  Gartner,  2013  

Quick Introduction to our Mobile Maturity Model

5

Disclaimer J

How to grow your mobile maturity?

6

1.  What is your Mobile Strategy Maturity goal? 2.  How are you performing currently? 3.  What are the gaps? 4.  Define gaps that matter most 5.  Prioritize and close the gaps

7

1.  Strategy and Organization 2.  Initiation and Design 3.  Security and Management 4.  Development and Maintenance 5.  Test and Distribution 6.  Backend and Integration

Mobile Maturity Model - Assessment Axes

8

Mobile Maturity Model - Level determination

OPPORTUNISTIC

Developing

Aware

STRATEGIC

Optimising

Practicing

MOBILE FIRST

Leading

Opportunistic Strategic Mobile First

Strategy & Organisation

Practice observed 1 ✔ x

Practice observed 2 ✔ x

Initiation & Design

Practice observed 1 ✔ x

Practice observed 2 ✔

Development & Maintenance

Practice observed 1 x

9

Mobile Maturity Model - Gap Identification

Chapter 5:

10

App testing & Distribution

Advanced app solutions | 11

Advanced app solutions | 12

Imagine.. Your customer calls within 5 minutes after delivering the Release Candidates…

Our Bloopers

Advanced app solutions | 13

‘Testing is not a phase in a project, it has focus in every phase!’

Advanced app solutions | 14

Preparation Definition Development Rework /Acceptance (post)-GoLive

Acceptance Criteria

Test plan Unit Testing Functional Acceptance Tests

Full sanity check

Reference Set Use Cases Integration Testing User Acceptance Tests

Graphical & interaction designs

End-to-End Functionality testing

Design Verification

Style Guides Usability Testing

Design Verification

System Testing

Performance Testing

Load testing & Stress testing

Testing in our projects.. Executing Preparation

Roles vs Testing

Advanced app solutions | 15

Designer Client Developer

Back-end Developer

Tester Customer

Unit Testing ✔ ✔

Integration Testing ✔ ✔ ✔

End-to-End Functionality testing ✔ ✔

✔

Usability Testing ✔ ✔

✔ ✔

Design Verification ✔ ✔

✔ ✔

System Testing ✔ ✔

✔

✔

Performance Testing ✔

✔

✔

✔

Load testing & Stress testing ✔ ✔ ✔ ✔

Functional Acceptance Tests ✔

User Acceptance Tests ✔

App Test and Distribution Axis

16

App Testing

Opportunistic Strategic Mobile First

•  Testing is performed on emulators and a few mainstream devices.

•  A test process focusses on requirements and features in happy flow.

•  Testing is performed on a representative subset of devices.

•  Client has a decicated testing discipline. Testers are involved from project start and part of the scrum team.

•  More attention for exceptions.

•  Testing is partly automated with (cross platform) unit - and functional test tools on a representative subset of devices.

•  Client has implemented testing at various levels, including e.g. design/prototype usability testing, integration testing and performance testing.

Practices Observed

Test definition

Test tooling

Test set

Test process

Automated testing

Advanced app solutions | 17

•  Unit testing •  Functional testing

Unit testing Example function TestSuiteByteArray:UnitTest_createWithOddHexNumber()

local data = ByteArray.newFromHex("0xA20"); Assertions.assertEquals("0x0a20", data:toHex());

end

function TestSuiteByteArray:UnitTest_createWithCapitalAndNonCapitalLettersHex()

local data = ByteArray.newFromHex("0xA20f34");

Assertions.assertEquals("0xa20f34", data:toHex()); end

function TestSuiteByteArray:UnitTest_createByteArraynewFromBase64String()

local data = ByteArray.newFromBase64("og80");

Assertions.assertEquals("og80", data:toBase64());

end

function TestSuiteByteArray:UnitTest_bytesExtremeValuesWorkOk()

local data = ByteArray.newFromHex("0x0080FF"); -- 0, 128 , 255

Assertions.assertEquals(0, data:get(1));

Assertions.assertEquals(128, data:get(2));

Assertions.assertEquals(255, data:get(3)); End

Functional testing Example Scenario: Test the button listener

Given I am on the mainpage

And I go to the "Manual/UI/Button/Button listener" menu

Then I should see "M2AT.08.06.030"

Then I drag from "clickButton" to "pressButton"

And I should not see "Press OK to continue"

Then I drag from "pressButton" to "clickButton"

And I should see "Press OK to continue"

And I press the "OK" button

Manual versus automatic testing Automatic testing Pro's

•  Fast verification on every code commit possible for large percentage of functionality

•  Promotes test early and often policy, especially for unit testing

•  Investing in testscripts gets stronger on multiple OS-s and form factors from the same source

Advanced  app  solu8ons    |   20

Automatic testing Cons:

•  Additional costs, continuously (every large change)

•  Flexibility loss in GUI (costs) •  Requires (regression) test setup

for multiple devices / OS’s •  Covers likely only most

important flows •  Hardly ‘waterproof’; manual

verification remains. •  Break even point for apps with

short lifetime hard to reach

How dynamic to

setup which tests?

How do we perform our testing?

Advanced app solutions | 21

•  Definition Phase •  Development Phase •  Acceptance Phase

When the app goes into beta and the first bug reports arrive…

Advanced app solutions | 22

Phases in a project – Definition Phase 1/2

Advanced app solutions | 23

Contract / Project plan •  Definition of target devices /

operating systems •  Definition of ‘Reference Set’ # •  Severity Matrix •  Definition of Severity

# A ‘reference set’ is a selected set of devices to meet the specifications and behavior of the most commonly used devices of a specific target group.

Phases in a project – Definition Phase 2/2

Advanced app solutions | 24

Testplan •  Which tests? (integration, performance, method of acceptance, etc) •  Process of reporting & tools •  Test arrangements •  Distribution during test phase Scrum Team •  Defines ‘Definition of Ready for test’ •  Defines ‘Definition of Done’ •  Creates Use Stories with Acceptance criteria, Business Logic & Exceptions for validation in testing

Advanced app solutions | 25

Our bloopers (2)

Imagine.. Right Facebook key, wrong copy / paste…

Phases in a project – Development phase 1/3

Advanced app solutions | 26

•  List of devices (‘reference set’) – testing on devices & compliancy. •  Compliancy list & ticking your devices

•  Builds are created and published on internal fast installation location

•  Usage of the tool ‘M2Borrow’ •  A library of all our devices with OS’s •  Barcode based •  Easy track & trace from borrowers

Advanced app solutions | 27

Phases in a project – Development phase 2/3

Advanced app solutions | 28

•  Working in JIRA with tickets •  Open > In Progress > Resolved > Closed

•  Important: signalize defects as soon as possible. Example  ‘Defini.on  of  Ready’  

Phases in a project – Development phase 3/3

Advanced app solutions | 29

•  ‘Resolved > Closed’

•  Note: ‘Resolved > Closed’ is in some cases assigned to the Product Owner of the customer.

Example  ‘Defini.on  of  Done’  

Phases in a project – Acceptance phase

Advanced app solutions | 30

•  Optional specific tests (performance, load/stress, etc) •  Acceptance builds are distributed to the customer (published on fast installation location and/or

TestFlight and in near future on new S2M Appstore.)

•  Customer verifies functionality on variety of users (‘Family & Friends’ testing)

•  Submission preparation

Advanced app solutions | 31

Imagine.. Mismatching team logo for Steaua Bucharest

Blooper intermezzo (3)

App Test and Distribution Axis

32

App Distribution

Opportunistic Strategic Mobile First

Ad hoc distribution to the app stores. Internal distribution via mail or website.

Partial automation of the distribution, including monitoring. A process with measureable gate before 'going live' and in sync with multi channel approach and compelling events. Use of enterprise stores

App distribution policy fully integrated in the Customer Lifecyle Focus on customer engagement. B2E distribution via integrated enterprise store for all OS-s. Centrally managed with policy based device and application management

Practices Observed

B2C distribution: App stores

App Lifecycle Management

B2E distribution: Enterprise store

MAM integration

Advanced app solutions | 33

App Distribution

•  Final verification testing for submission

•  App Stores •  Apple iTunes, Google Play, Windows Market place, Blackberry World

•  MDM / MAM (Good, MobileIron, XenMobile) •  Mobile Device/Application Management systems & enterprise app stores

•  Lean, fast and easy app distribution mechanism •  Via Service2Media enterprise app store with In House Certificates

Add-on: App Lifecycle Management

Advanced app solutions | 34

•  Suggestion: use a version control mechanism apart from the stores, like our M2Version service.

•  Verifies on every start-up with a back-end if a business rule applies for a specific audience (app version, OS (version), language/country, etc)

•  Allows pop-up messages to specific group of users on start-up for i.e. •  Voluntary updates with a link to a (different) app or URL •  Mandatory updates with a link to a (different) app or URL •  Maintenance or outage messages with optional URL •  Retirements of an app •  Etc.

Are you interested in a Mobile Strategy Maturity Assessment?

anette@service2media.com

35

Next Webinar:

June 12th

36

#6. Backend and integration