Mobile App Testing Strategy by RapidValue Solutions

© RapidValue Solutions

Mobile Application

Testing Strategy


Mobile App Testing Strategy – Adopted to Deliver Quality Service

and Provide Utmost Consumer Satisfaction with the Perfect App !

There has been an increase in the adoption of smartphones, tablets and several mobile

devices with the passage of time. And it has led to an enormous growth of mobile

applications in recent years. Mobile device is considered to be the primary medium of

interaction for the customers and also, businesses worldwide. And mobile applications are,

actually, driving the communication.

People, generally, do not give much importance to mobile application testing because of its

expensive nature. But it is very much essential to ensure that the consumers have a great

experience, every time they use the application.

The aim of mobile application testing should be to acquire knowledge about the quality of the

service that you are offering. Does it work properly or not? Will it provide services as per the

expectation of the customer? These questions need to be answered to ensure that the

customer comes back to you, for your service again.

Mobile testing is becoming more and more complicated and complex with each passing day.

Strategies are invented and used to simplify the mobile application testing.

This presentation primarily addresses, the strategy to be adopted in mobile app testing, the

types of mobile app testing and the stages to be followed, before the application is set live.


The main objective of mobile application testing is to help customers to gain assurance that the

software applications are performing and functioning as expected with the devices or with the

several other applications of the vital players in the mobility ecosystem.

Objectives of Mobile Application Testing

Parameters Details

Functionality UI responsiveness, Peak load performance, Operational times

Performance Network type (Wi-Fi, 2G, 3G, 4G), Impact of connectivity issues

Network Security of data in motion and data at rest

Security Mobile platform compatibility, Device model compatibility,

Backward compatibility with previous app version

Compatibility Marketplace guidelines’ compliance for Apple and Google stores

Conformance Compare the developed product with user stories

Usability User Experience

Installation and

Provisioning

Installation process, Un-installation process, User provisioning and de-

provisioning

Service Testing Check that the services are up and running and also measure the

performance

Resource Testing Battery performance, Memory usage, Local database growth and Garbage in

memory, come under this process

Load Testing on Mobile With heavy loads of images, videos and also loading the app with more and

heavy functions, use third party application like blaze meter to generate load

Localization With this, the QA team checks the application’s multilingual capability


Emulators Cloud Testing Solution Physical Device

o All applications can be

deployed and tested on

emulator without

investing in mobile

handset for various OS

o Emulators are mostly

available free, and we

can also perform UI,

Stress and performance

testing on that

o 30-40% test

o The Mobile devices can be

accessed through web

interface i.e. the browser

o Application can be

deployed, tested and

managed

o Automation module is

available and the solution

is secure, if private cloud is

used with no maintenance

o Up to 100% test

coverage can be

achieved

o Real device testing will

give most realistic view

of test results

o All possible types of

testing activities can be

performed including

those, that are

dependent on hardware

Mobile Testing Strategy is an essential element to ensure the smooth functioning of any

application. Emulators are considered to be vital testing platforms while Cloud testing solution is

very much secure. Real Device Testing is said to yield the most realistic and practical results.

Mobile Testing Strategy


• Different Form Factors

• Wide Variety of devices from different manufacturers, incase of Android

• Different versions of Mobile OS

• Different Screen Density

• Infrastructure and Connectivity

• Mobile testing tool availability – still on nascent stage

• Cloud testing platform solves some of these issues, however:

o It is very expensive

o Lock – in with the platform

o Scripts cannot be used outside of the platform

o OCR based platform. Because of automation, it is time consuming

Challenges in Mobile Testing

Mobile applications are considered to be “game-changing” technologies, which have further led to

the development of various unique and beneficial testing challenges. The growth of the application

market has increased the customer’s expectation of obtaining quality service. Companies are also

coming up with new apps in the marketplace. Hence, challenges of mobile testing are also

increasing each day. Whether it is an enterprise app or a consumer app, companies are finding

out ample opportunities to boost their revenues and reaching out to a vast range of consumers.

This ambition leads to a greater development of mobile testing strategies which face certain

challenges:


• iOS 7 and above

• iPhone 4, iPhone 4s

• iPhone 5, iPhone 5s, iPhone 5c

• iPhone 6

• iPad Air, iPad2

• iPod touch 4th generation

• iPod 5th generation

• Android 4.x and above

• Samsung galaxy s4, s5

• Samsung Note 3

• Nexus 5

• HTC desire 816,one

• Moto X

• Sony Xperia Z

Popular Mobile Devices for Testing Applications

A great application certainly needs a greater customer service solution. And it is recommended

for the application to support all the iOS and Android devices. The following are some of the

devices and versions, on which its good to test your application.


The following statistical chart gives a clear picture of the screen sizes and densities for Android:

Screen Sizes and Densities for Android


Simulator

Testing

Device Testing

(On Wi-Fi Networks +

Network Simulation

Tools)

Device Testing

(In-Network)

• Supports only device dependent use cases

like camera, accelerometer, GPS etc.

Device Testing (In –Network)

• Ideal during the development phase

• Best and economical option for functionality

testing

Simulator Testing

Device Testing ( Wi-Fi )

• Use proxy tool Charles proxy to monitor the

load and performance of the application

There are certain distinctive features of Simulator and Device Testing. They are summarized

below:

Simulator vs. Device Testing


Advantages of Cloud Testing:

• Variety of plans available, such as hourly, monthly or yearly subscriptions

• Test incoming calls and sms without having any cellphone connection

• Wide variety of devices

• Automated test execution, can be played as videos

• Test can run on several devices, in parallel

• Build integration

The performance testing is conducted on the applications and services provided via cloud

computing. Cloud testing ensures optimal performance and scalability under a wide variety of

conditions. Cloud testing monitors and reports on the real-world user traffic conditions.

Cloud Testing

User interacts with the device through studio or web front

provided by the cloud provider

Server receives the request and

delegates that to the actual device

Device sends back the result and that would be displayed

on the studio or web front


The Application testing commences with Unit Testing and then proceeds with Functional Testing,

Regression Testing and UI Testing. This is further followed by Automation Testing, Performance

& Load Testing and the final stage is the Security Testing.

Application Testing Flow

Unit Testing

Functional Testing

Regression Testing

UI Testing

Automation Testing

Performance & Load Testing

Security Testing

Performed by developers before releasing the build

QA team perform the functional testing based on the test cases that are created using user stories

Regression testing is a suite of testing, that include smoke test and execution of regression test cases

UI testing is a usability testing, focusing on the UI design of the application, carried out by UI & QA teams

Once the development is in shape, all the regression test cases should be automated

QA team should check the UI response, peak load, transaction time as part of the performance testing. This should be conducted after every 3 sprints

Security testing should be considered for data security, authentication and authorization


App Store Play Store

Terms & Conditions

Functionality/Reliability

Metadata/Ratings & Location

Intellectual Property Rights

UI conformance & Contents

Spam & Placement

Ads & System Interference

Contents

Personnel & Confidential Information

Content Ratings

Conformance testing is considered to be critical. Conformance is essential and crucial to

determine whether the product or the service meets the specified standards. Validation of the

App Store and provision of the Google Store Guidelines are the integral part of the effective

process, which plays a pivotal role.

Conformance – Validate App Store & Google Play Store Guidelines


• The network testing should be carried out in physical device by the QA team to ensure the

network performance under various technologies like GSM and CDMA

• Cloud testing should also be leveraged to test it in the real environment

• Separate test case suite should be designed to test the various signal strength under tele-

signals using 4G,3G,2G etc. and the Wi-Fi signals

• Charles proxy, a third-party tool can be used to control the network bandwidth by creating

device and Charles proxy operate under one network

• Applications should be tested under multiple signal strength and quality performance under

live environment is ensured

The most important thing is to ensure smooth and seamless network performance which can be

evaluated with the help of efficient Network Testing. Performance is evaluated under

technologies like CDMA and GSM. The applications are tried and tested under live environment

to ensure utmost customer satisfaction.

Network Testing


• Usability testing should be carried out by the following testing:

o Validation of gray scale wire frames

o Testing the compatibility with all devices and browsers

o UI testing that focuses on user acceptance

o UI comparison testing with benchmarked applications & standards

o Review testing before every UAT by customer

• Usability test should be conducted by UI/UX and QA team

Another important testing procedure is the Usability Testing which should be ideally conducted

by UI/UX team and the QA team. It is essential to test the compatibility factor with the various

devices and browsers. UI Comparison Testing is also needed to know whether the benchmarked

App is in conformity with the existing standards.

Usability


• SOAP UI is a free testing tool, based on open-source standards as XML, XPATH and

GROOVY (Java). It is stable and robust too.

• Using SOAP UI, tester are able to perform

o Web service regression testing

o Web service test automation

o Web service load testing

• SOAP, REST service will be covered as a part of service and all the responses will be

evaluated using tools like GET, POST etc.

Service testing will enable the providers to evaluate the performance of the application and work

on it, according to the test results. It is essential to deliver a comprehensive testing as a service

platform. There has been a rapid growth in the demand for effective application testing tools and

services that will leverage mobility services. Service testing is performed to deliver effective and

high-quality applications.

Service Testing


• Resource utilization like memory, CPU, etc. is measured while doing Performance

Testing.

• Set criteria for Application/Device performance, Server Performance and Network

performance

o Quadrant and Linpack is used for measuring performance

o Lint (come with Android SDK), static code analyzer is used for performance and

optimization

o OC Lint (IOS), Static code analyzer is used for performance and optimization

o iOS instrumentation tool is used to find the leaks and memory foot print in iOS

application

Testing is, indeed, an important aspect of development. It is crucial and critical for both, creating

new applications and modifying the existing ones. Utilization of resources is measured

effectively while going through the process of Performance Testing. Linpack and Quadrant prove

to be effective in measuring performance of the application.

Resource Testing


There are two methods to perform load testing - Manual & Automation

Automation Testing:

o Load Testing is trying to simulate the work load of various user scenarios to the

applications with third Jmeter

o Testing the load within a specific pagination by monkey testing

Manual Testing:

o The application under test are loaded with large images (high resolution) and videos

to evaluate the peak point where the application can handle the maximum load

o Load testing are performed under various signal bandwidth to understand the

minimum network signal strength to run the application with standard usage

Load Testing


Automation testing is the key testing strategy to automate Regression testing which include all

functionalities:

o Appium is a good tool for Automation testing. And automation should start after the

development is in the right stage

o Appium is an open source, cross-platform test automation tool for native, hybrid and

mobile web apps. It is tested on simulators (iOS, FirefoxOS), emulators (Android),

and real devices (iOS, Android, FirefoxOS)

o Automation testing should be a part of every sprint to reduce the effort of Regression

testing and Smoke testing

o Automation frame should be designed after couple of sprints

One of the best characteristic of Automation Testing is that it does not require any manual

intervention while executing the particular test. The test has certain benefits like greater

consistency and the reduced nature of the repetitive work. Automation testing can be relied upon

to provide the ease of accessibility to information about the tests.

Automation Testing


• Automation of regression suite can be incremental starting from 1st sprint

• Automated regression suite to check the impact of latest sprint deliverables on the

previous sprint(s) deliverables, for continuous integration

• Re-use the same (with additional manual\automated tests) for hardening sprint before

application is launched

Regression Testing is another essential procedure which deals in rerunning test cases just to

ensure that the software changes do not have any side-effects. Regression Testing has become

more effective and efficient with time. This testing is critical to check or detect whether new

issues have cropped up after the change has been applied. Regression Testing is critical for

large applications as it is a little difficult to figure out whether any part of the application has

given rise to any issue or problem due to the change.

Regression Testing


• Supporting types: Text, Images, Videos etc.

• Size: Maximum size supported for each category

• Size: Beyond maximum supported size for each category to see the stability / Large volume

of data and measure performance

• Mix of different data types with maximum size

• Negative Data (Invalid data): The application is not intented to support

• Data and network (2G,3G,Wifi) combination

• Data and Device type/OS combinations

The following gives a detailed description of the data types, data sizes, blend of different data

types and sizes, supporting formats etc. It also provides information regarding the data and

device/network combinations. One obtains knowledge about negative data also, which is a

particular data type or data size that the application does not support.

Test Data – Type and Size


• The app diagnostics and crashes on iOS and Android can be monitored using Crittercism

• Google Analytics tool helps to monitor the exit points and identify the reason. It collects

user interaction data, manages how the data is processed and provides a report.

The post-production stage is equally important as the pre-production and the production stage.

A report has to be provided after collecting user-interaction data and managing the data

procedure. Crittercism and Google Analytics provide immense support for monitoring the

application.

Post Production


• Hardening sprint(s) - QA only sprint(s) for

Go Live

• On actual environment and configurations

• Regression suite to focus on system level,

Integration, Impacted (changes) areas,

performance and security

• Real-time data, text, videos and images

with maximum supporting size

• Go Live check lists based on the

sprint level confidence

o Entry Criteria

o List of devices and configurations

o Actual environment &

Infrastructure, Test Data

o Regression Suite and Use cases

o Installation/ Uninstallation

o Exit criteria

Testing stage holds utmost importance. It gives the clear picture of the task that has been carried

out with such great effort. Testing is done in actual environment and configurations to know

whether the application supports it or not.

Go Live – Testing


The growth of the application market has increased the customer’s expectation of obtaining

quality service. Companies are also coming up with new applications in the marketplace.

Hence, challenges of mobile testing are also increasing day-by-day. Mobile application

testing is, indeed, an integral and essential part of the production process. It is a procedure

by which the application software, that is developed for the mobile devices, is tested for its

supportive feature, its functionality, usability and for its consistency. The main objective of

mobile application testing is to help and assist customers to gain assurance that the software

applications are performing and functioning as expected. Mobile testing strategy is an

essential element to ensure the smooth functioning of any application.

• The most important thing is to ensure smooth and seamless network performance which

can be evaluated with the help of efficient Network Testing.

• Service testing enables the providers to evaluate the performance of the application and

work on it, according to the test results.

• The performance testing is conducted on the applications and services provided via

cloud computing.

• The Usability testing is equally essential to test the compatibility factor with the various

devices and browsers.

• Service testing, Load testing, Regression testing and Automation Testing - all the testing

procedures form the significant part of the application testing strategy which is crucial

and vital for the production of an application.

Conclusion


Thank You


About RapidValue

RapidValue is a leading provider of end-to-end mobility solutions to enterprises worldwide.

Armed with a team of 250+ experts in mobility consulting and application development,

along with experience delivering over 400 mobility projects, we offer a range of mobility

services across industry verticals. RapidValue delivers its services to the world’s top

brands and Fortune 1000 companies, and has offices in the United States and India.

www.rapidvaluesolutions.com www.rapidvaluesolutions.com/blog

+1 877.690.4844 [email protected]

This document contains information that is confidential and proprietary to RapidValue Solutions Inc. No part of it may be used, circulated, quoted, or reproduced for distribution

outside RapidValue. If you are not the intended recipient of this report, you are hereby notified that the use, circulation, quoting, or reproducing of this report is strictly prohibited and

may be unlawful.

Disclaimer:

Presentation by

Rinish KN

CTO, RapidValue Solutions

Software

Mobile App Testing Strategy by RapidValue Solutions