April 2015

Hexaware & Experitest Co-webinar:

Performance Testing- Is Your Mobile Application Ready for Scale?

Speakers

Guy ArieliCTO

Experitest

Vinayak Gaur

Head of Mobile

Application Testing

Hexaware

Agenda

Optimal testing strategy

1

2

3

5

6

Zoom into Mobile Network Virtualization

Why performance testing is important?

Experitest Solution

7

Case Study

Demo

Importance

Basic terms

Challenges

Q & A

Mobile (App) Experience

60%of mobile users will abandon your app if it

doesn’t respond within three seconds or

provide the desired user experience in any

circumstance (Google)

>65%Over 65% Of the apps on App Store and

Play Store have poor ratings (2 or less stars)

BrandA poor mobile experience can have a

negative impact on the transaction; and in

the long term, it could be even more

detrimental to a brand’s reputation.

Composite Testing Strategy = Optimal Testing StrategyIs this the new Trend?

Compatibility

Regression Test

Automation

Network

Virtualization

Functional SecurityDevice level

PerformanceUsability /

PlatformServer Level

Performance

Compliance

> Segregation of Horizontal and vertical Test types

> Combination of 2 or more test types

> Implementation Strategy defines the parameters for combining Test types

> Network Virtualization provides critical coverage across test types

Network Virtualization Testing

Why is it Important for your testing Strategy?

30% of bugs found by end users were caused by running the application in "non optimal" network conditions

Functionality

FragmentationPerformance

Network Virtualization

Drilling Down Into Network Virtualization

Network Virtualization- Basic Terms

If your network is

a water pipe your

data is the water

ThroughputThe width of the pipe, or how

match water it can move in a

second.

LatencyThe length of the pipe , or how much time it will take for a drop of water to path the pipe.

What have Changed When We Moved to Mobile ?

Built in disconnection

Different Connectivity

Technologies

Different physical geography

conditions

Built in latency Operator/Provider

In the mobile age the connection is dynamic.

Wi Fi GSMLTE

Lab Network Conditions

Latency= <2 ms

Throughput=~30,000 Kb/sec

• Your lab device is connected to a high speed wi-fi connection

• The device connects through the Wi fi to the application server

Wi fi

Real Life Network Conditions

So the difference is huge! • Your customers are using a slow dynamic GSM network.

• The client device is on a different network in India

• The servers are in the US.

Throughput=~50 Kb

Latency= ~250 ms

GSM

Why does my application

care about the network

condition?

Your application assumes the data requested from the server

will be delivered in a reasonable amount of time.

A part of the quality of your application is related to tuning the

timeouts and verifying the error message are appropriate.

Other aspects of your application quality related

to the user experience.

If your application is not responding after a few

seconds your user is likely to delete it.

SeeTest Network Virtualization Architecture

N

a b c e

d

End-to-End Product Suite - Manual, Automation, Monitoring, Load, Network Virtualization, Manager

• Manual testing tool

• Mini-recorder for automating short,

repetitive, manual scenarios

• Report with screenshots of each step

• Manual to Automation Script Convertor

• Device Lab located onsite (in the customer VPN)

• Devices can be reserved and accessed by any remote user

• Used in conjunction with SeeTestAutomation/Manual for testing of remote devices

SeeTestManual

SeeTestCloud

• Automation tool

• Same test runs on different devices

• Plugs into all testing & CI environments:

SeeTestAutomation

SeeTestNetworkVirtualization

• Device performance under varying network conditions

• Simulating mobile network bandwidth latency and reception while on the move

SeeTestLoad

• Real device generated mobile load

• Full integration to LoadRunner, Jmeter, etc.

Mobile Add-On for UFT• Full & complete integration into HP UFT:

• Record a test, edit it, run it, and view results

• Integration of object repository and more…

SeeTestManager

Free

Tool!

N

L

G

U

Demo

Call for action

Providing this information to the developers will enable them to use few technics to solve the issues:

Reduce the data size

Use CDM storage where part of the data (images) will be served dependent on your user device location.

Cache the data

Reduce the number of round-trips

Analyzing the Completeness of a Mobile

Testing Strategy

0%

10%

20%

30%

40%

50%

Itera

tion

1

Itera

tion

2

Itera

tion

3

Itera

tion

4

Functionality

Compatibility

Usability

Compliance

Performance-Server Level

Performance-Device Level

Security

Change in Distribution of Test Types across Sprints

Performance – Server Level

Compliance

Security

> Functionality

> Compatibility

> Usability

> Performance – Device Level

Perc

enta

ge o

f to

tal

Test

ing

Challenges – Including Network Virtualization in

your Testing Strategy

Test Strategy Components Impacted Test

Infrastructure

Test Planning &

Implementation

Efficiency in Test

ExecutionTest Coverage

1 Distribution of Testing teams X X

2Support for multiple Mobile Platforms –

versions, devices, configurations X X X X

3 Organizational Info-security constraints X

4 Need for network coverage across regions X X

5Application availability restricted to client

devices X

6Cost of infrastructure setup and

sustenance; Tools & Licenses X X X

7Lack of adequate support from Mobile

Test tools X X

Case Study

Case Study:

End User Performance Testing of a Mobile Financial App

CUSTOMERLeading global financial institution (140+ countries)

End to End Testing of Banker

Application on Mobile Platform

(aimed at providing the end

user with information to

conduct banker business

activities)

• Business need: Improve Time to

Market & Exceptional User

Experience

Speedy Delivery of the

Application to the

business Users in limited

time frame

User Experience

surpassing the business

needs for users in

different countries

SDLC – Custom Agile SCRUM

BACKGROUND TESTING IMPERATIVES

iPad Banker

ApplicationUsability

Testing

Compatibility

Testing

Sync Testing

User Security

Testing

Platform

Testing

Functional

Testing

Performance

Testing

Device Build

Management

Test Data

Management

Adherence to human

interface principles

for iPad platform

Supported iPad

& network

configurations

Web <_>Mobile cliet

synchronization

Validation of

user level access

controls

Support of iPad

features in conjunction

with the App

Server & Device level

performance testing

Built distribution

strategy on mobile

devices

Setup & creation of business

specific test data for different

levels of testing

Case Study:

End User Performance Testing of a Mobile Financial App…

SAMPLE

SCENARIOS

App performance in different network configurations – WiFi, 3G, 4G

Mobile workloads during various business hours

Performance characteristics based on geographical domain specific latencies

CHALLENGESValidating app user experience in varying network bandwidth & conditions

Performance benchmarking for real life usage scenarios (driving, low signal, etc.)

WORKAROUNDS /

TACTICAL SOLUTION

Usage of Performance Testing – NV integrated solution for for Server Performance testing

Distribution of testing between Nearshore, Onsite & Offshore at different locations

Manually vary parameters to enable repetitive testing in varying network conditions – support from test automation

Thank you