Infosys & Experitest Co-webinar: Total Quality in Mobile Apps … Webinar... · 2014-12-17 · Total Quality in Mobile Apps -Continuous Integration & Performance Testing Nov 2014

Infosys & Experitest Co-webinar:

Total Quality in Mobile Apps -Continuous

Integration & Performance Testing

Nov 2014

Part I

How To Use Continuous Integration for Mobile

Speaker: Guy Arieli , Experitest CTO

Agenda

Why is it important?

How is it different from regular Continuous Integration?

What is the recommended flow for Continuous Integration in mobile?

Case study: major bank

Demo: mobile continuous integration Jenkins

Continuous Integration is especially important for mobile

1

2

3

4

5

What Is Continuous Integration?

Software

development practice

where members of a

team integrate their

work frequently

Significantly reduces

integration problems

and allows a team to

develop cohesive

software rapidly

Each person

integrates at least

daily – leading to

multiple integrations

per day

CI significantly reduces integration problems and allows a team to develop cohesive software rapidly

YOURAPP

Developers Source Code

Repository

CI Server

Why is CI especially important for mobile apps?

Mobile app development is faster

End-user expectation for fast response time

Frequent external environment changes

New OS versions

New devices

High adoption of Agile methodology in R&D

Fast app release

Many (hundreds) devices to be tested

Several mobile OS to be tested

Huge device matrix

Higher ROI because of mega matrix

How is mobile CI different?

Parallel execution

Complex management

of Continuous

Integration

Smart device allocation

Dual application mode

(instrumented, non-instrumented)

Continuous Integration For Mobile

Developers

Source Code

Repository

CI Server

YOURAPPExecuter Agent

Execute Test On Devices

x

Test Results

Recommended process for mobile Continuous Integration

Build the application

Prepare the application for automation

Reserve the devices to be used

Deploy application and device preparation

Tests Execution

View and analyze reports

Build the Application

For example: If it’s an Android

build, you will get a URL for

downloading this APK file

Build the Application

Source Code

Repository

CI Server

YOURAPP

New Build of Your App

Trigger the entire

execution






Tests Execution


Prepare the Application

YOURAPP

New Build of Your App

YOURAPP

YOURAPP

Instrumentedmore control and more

capabilities over the

application

Non- instrumented

(optional step)

Application under test

may have two modes:

Tips for

Implementation

Unattended, on-the-fly

instrumentation

Test automation tool should enable:

1

2 Perform the instrumentation

on the same machine that is

being used for automation

3 Windows machine to perform all

instrumentation (inc. on-the-fly)

(also in the case of an iOS app)

Dynamic, smart reservation of the Devices to be used

@os=‘android’ and @versionnumber > 4.2

and

@manufacture=‘htc’

Dynamic,

smart

reservation

Execute your regression of tests on as many devices as you can

Cloud of Devices

Smart Device Reservation






Tests Execution


Deploy Application and Device Preparation

**These operations should be available and should be able to run entirely unattended.

Device reboot

Application clear data

Uninstallation of applications

Installation of applications

Launch of applications

Have full control of

the device and of

the application

installation

Working with the Settings






Tests Execution


Tests Execution

CI Server

Executer Agent

Execute the Devices

Remote Mobile

device Lab

T1

T2

T2

T2

…

Tn

Actions for making

Test execution

efficient:

Capability to run the tests in parallel on different devices

Decupling between the CI server and the execution machine

Queue of tests that are waiting for devices the be available






Tests Execution



Demo How to use Jenkins for mobile

Continuous integration

Case Study

Headquartered in APAC

Spans over 50 countries across Europe, America, Africa & Asia

~200,000 employees

Mobile applications in tens of languages

Ranked in top 10 largest banks worldwide

Revenue more than $30 Bn.

Customer Profile:

Multinational banking and financial services company

Major global financial services provider engaged in personal

banking, credit cards, corporate and investment banking and

wealth and investment management.

The bank’s Personal and Business

accounts management apps

Wallet apps

Trading apps

Mortgages apps

BankCard

Many more…

The Need

Testing end users' apps Testing the Internal bank

Employees’ Apps

Loan qualification form

Mortgage requirements form

Etc.

Applications used by The bank

employees providing services to

customers such as:

The Requirements

On-premise devices, all within the bank VPN

Cross-platform testing on all OS: Android, iOS,

WindowsPhone8 & BlackBerry

No need for access to application source code for testing

(on-the-fly instrumentation)

Support for multi-field form filling including scrolling in

lists, invisible objects, etc.

Ability to test on devices located in one of the bank facility

by testers located in other locations

</>

Implementation

Years working with the tool: 3

Over 100 SeeTestAutomation stations

Number of Automation engineers working on the project: 200

4 geographical locations around the world (US, Europe, Africa and China)

Number of tests before: 0

Number of tests today: ~20,000

Number of executed tests: ~200,000 (each test can run on more than one device)

Onsite deployment with integration into the existing test environments

QTP, Junit, Eclipse

QC and Jenkins

Parallel execution of tests

Coverage

Total coverage: ~90%.

Number of executions of the full cycle: Once a day

Number of apps tested: +20, (different application in different regions)

Coverage of both customer ’s apps and internal systems apps

Full app test coverage for all OS-Cross platform testing of

Android

iOS

WindowsPhone8

BlackBerry

Full end-to-end coverage of test scenario

Support for multiple languages (including English, Spanish, French)

What if we did it manually?

A single QA engineer: 56 man years to finish the cycle (lifetime).

For one day we would need 20,000 trained manual QA engineers.

Another Advantage

The R&D engineers can work faster and take more risks

According to the bank R&D managers the development speed is

ROI

3-4 times faster!

The Tools Used

• Automation tool

• Same test runs on different devices

• Plugs into all testing & CI environments:

• UFT/Selenium/ Microsoft Visual Studio/

RFT/Java/C#/TestComplete//Perl/Python/Ruby

• QC/TFS/Jenkins/TeamCity/RQM

• Manual testing tool

• Mini-recorder for automating short,

repetitive, manual scenarios

• Report with screenshots of each step

• Device Lab located onsite (in the customer VPN)

• Devices can be reserved and accessed by any remote user

• Used in conjunction with SeeTestAutomation/Manual for testing of remote devices

SeeTestAutomation SeeTestManual

Free

Tool!

SeeTestCloud

Manual to Automation Script Convertor

Thank you

Infosys Point of View (POV) on

Mobile Application Performance Testing

17 December, 2014

Performance is key, for a Mobile user

Poor App Store ratings

… are often linked with poor app

performance

• Frequent crashes and

glitches

• Slow loading times

• High battery drain

• Unreliability

• Inefficient start and exits …

Forrester: “Mobile apps live

and die by their ratings in an

App Store… When the rating

suffers, customer adoption

suffers.”

Analysis reveal that

majority of 1-star

reviews are related

to performance

issues, proving that

Quality is key for an

end-user.

Apigee: “44% of users delete

a mobile app immediately if it

didn’t perform as expected;

Performance is reported to be

especially important for

banking (74%) apps

IMS, Goldsmiths: "Users

experience a lot of negative

emotions and frustrations

when trying to complete some

digital tasks and apps are slow

to load and respond"

realBusiness: “Consumers are

unforgiving towards slow and

poor – performing apps.”

52%

0

76%

48%

0

15%

0

9%

CRITICAL, 1-STAR REVIEWS

PERFORMANCE RELATED, CRITICAL, 1-STAR

REVIEWS

% of Critical, 1-Star Reviews For Top iOS Apps

Performance Related

Other

Crash, Bug

Lagging, Performance, Hot, Battery DrainSlow, Long Load, Load Fails

Source: appurify

http://appurify.com/blog/avoid-bad-reviews/

If left unaddressed, the impacts are significant

Lost transactions not only result in lost revenue for banks, but have a detrimental impact on customer loyalty and brand reputation – it's a lose-lose situation.

38 percent try another app when faced with a problem, 34 percent stop using the app, and 19 percent complain to friends and family

Majority users delete or uninstall mobile app because of performance problems. About 18% delete the app immediately if it froze for just 5 sec. 38% delete the app if it freezes for longer than 30 seconds

30% of users would spend more money with an organization that had a good mobile app

29 percent would pay more for a product or service if the organization’s app performed better than its competitors

Nearly one third of smartphone and tablet owners would change banks if a mobile app wasn’t up to par

Addressing Customer Reviews on Mobile apps..33

• One in six customers who switch banks indicate a poor

mobile banking experience prompted the switch

• Between 51% and 76% of mobile users use their mobile

apps to check balances, transfer money, pay bills, and

look up branch locations and hours. Hence reduced

performance leads to greater & immediate customer dissatisfaction

Problem Cause Solution

Application is buggy (doesn’t

open, crashes, freezes often).

Sloppy code, varying internet

connection, poor memory

management.

Thorough testing of the app

under multiple conditions and

variables to ensure that an

app is devoid of issues

Infosys’ Performance Testing Solution

Client Application Performance Server Side Performance Network Impact on Performance

Device Impact Testing / Client Application Performance34

Key Measurement Parameters:

• Battery Consumption

• Device CPU and Memory constraints

• Device I/O – Assess speed for read and

write to devices

Tools:

- Xcode Instruments (memory

allocation, leaks, CPU usage …)

- Android ADB (battery consumption,

CPU usage / processing speed)

- Android DDMS ( memory usage,

device I/O, track running thread info ..)

- Quadrant (only for Android)

- Neoload

Server Side Performance / Load Testing35

Key Measurement Parameters:

• Response Time

• Error Rate

• Throughput

• Server CPU utilization

• Server Memory utilization

Tools:

- LoadRunner

- Neoload

- CloudTest

Mobile Performance Tool Recommendation:

• Ability to support multiple devices

• Ability to support native applications as well as browser-based applications

• Ability to record through the device using agents, or through emulators, as

well as directly from a browser to reduce complexity

• Ease of use and fast turnaround time

Network Impact Testing36

Network Emulation Options:

Network Coverage - Occasional disconnection of the network.

Packet Loss & Duplication – Packet loss & duplication for an application

session by dropping packets based on preset values

Delay and Jitter - Delay of data originating from app .

Multi-Network testing (Roaming) –Switching from one type of network to

another type of network e.g. (3G, 4G, LTE)

Key Measurement Parameters:• Response Time

• Exception Handling

• UI Issues

• Consistency

Tools: - Infosys Network Impact Testing Solution (iNITS)

- Shunra

- Neoload

Mobile Performance Tool Recommendation:• Ability to dynamically emulate various network conditions

• Ability to generate and or emulate load from various locations, leveraging the

cloud

• Ability to test regardless of the protocol the application uses

Some of the Performance Testing Best Practices for Mobile Apps

Test early: Do quick check on mobile web application performance, using page analysis tools like

‘PageSpeed Insights’, Yslow etc. For native apps, developers can perform regular checks with

Instruments / DDMS tools

Verify & validate the performance on a real device

Perform both modular and system performance tests

Correlate server-side metrics with client-side response times. It helps nailing down the root cause

May look at reusing the functional scripts for load Testing (e.g. SeeTestLoad -LoadRunner)

Look for consistency in test results, during multiple test iterations

Ensure complete performance test coverage, executing Load, Endurance, Stress and Single-user

test cycles

37

© 2013 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to changewithout notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Exceptas expressly permitted, neither this documentation nor any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing,photocopying,recording or otherwise, without the prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.

Thank You

Documents

Infosys & Experitest Co-webinar: Total Quality in Mobile Apps … Webinar... · 2014-12-17 · Total Quality in Mobile Apps -Continuous Integration & Performance Testing Nov 2014