Practical_Guide_WP-2012 - Mobile Testing Solution

Perfecto Mobile | Mobile Market Overview 1

MOBILE MARKET OVERVIEW

The mobile boom presents tremendous business opportunities for enterprises. With a total of over 300

million smartphones shipped in 2011 and with mobile app store sales exceeding $6 billion, the market for

mobile applications is ready for prime time. Those enterprises able to get to market quickly with high-

quality mobile apps will reap the rewards.

Figure 1 - Mobile phone market dynamics: Android, iOS, Windows and BlackBerry releases and new devices in a

single year

2011

Practical Guide for Choosing the Right

Mobile Testing Solution for Your Enterprise

Practical Guide for Choosing the Right Mobile Testing December 1, 2011

Perfecto Mobile | Mobile Testing Is More than Software Testing 2

As more and more enterprises "go mobile", they are discovering that traditional software testing

methods are no longer suitable to ensure the quality and performance of their new mobile apps and

websites. In today's extremely fragmented and dynamic mobile market, there is a need for a new testing

approach designed to address the unique needs and challenges of the mobile environment. Choosing the

right mobile testing solution is a key criterion for success as enterprises roll out their mobile business

strategies.

Perhaps the easiest way to underline the challenge of mobile application testing is to consider this

staggering fact: over 260 Android devices have been released since the beginning of 2011. New

smartphones and tablets (a segment which didn't even exist two years ago!) are released to the market

on a weekly basis, each with a unique form factor or set of features. With three major operating systems

on the market and an ever-growing number of versions, models, screen sizes and network technologies,

it's no surprise that enterprises find it virtually impossible to keep their mobile apps and services in sync

with ever-changing market trends.

How does all of this affect the way enterprises should conduct their mobile testing? What are the

implications of today's fragmented, dynamic and localized mobile environment on your mobile QA

strategy? When you release a mobile app (or website), what do you need to consider?

This whitepaper provides practical "how to" information that will assist enterprises in choosing the right

mobile testing solution. It will help you to understand how mobile testing is different than traditional

software testing, as well as outlining key criteria that enterprises should consider when selecting their

solution. Lastly, we present a real world example of a mobile testing solution built to meet today's market

challenges, along with the results achieved by an enterprise using this solution to test its mobile

applications.

MOBILE TESTING IS MORE THAN SOFTWARE TESTING

Make no mistake – mobile applications are software, and mobile testing is by definition a type of software

testing. In other words, all your existing quality standards and ALM methodologies need to be applied in

accordance with business objectives. In fact, a key guideline when “going mobile” is to change as little

as possible in terms of ALM infrastructure and resources.

But there's one catch. Mobile testing involves added complexity due to several new factors inherent in

mobile that do not exist in traditional software QA. When choosing a mobile testing solution you will need

to take these factors into account:

Multiple platforms and networks – As mentioned above, mobile applications must be tested on at least three major operating systems (i.e., iOS, Blackberry, Android). Each OS has multiple versions and Android, for example, rolls out a major release every 6-9 months (Honeycomb, Ice Cream Sandwich etc.). Device types, form factors and network technologies also need to be taken into account.

Device logistics , procurement and security – Enterprises require solutions that minimize the hassles and logistics associated with procuring, deploying and maintaining dozens of devices in multiple geo-locations. Solutions that facilitate the logistics of offshoring mobile testing (shipments, network availability, regulations, etc.) can save substantial time and money. In addition, the security aspect of


Perfecto Mobile | Mobile Testing Is More than Software Testing 3

those mobile devices is critical - you must make sure those new "testing devices" do not breach corporate security policy.

Collaboration among globally distributed teams – Testing might be performed outside the target region of the application, i.e., off-shore testers require access to mobile devices from abroad. Globally distributed teams require real-time collaboration tools that enable a tester to share a device in "live" testing with one or more distant colleagues for their input or approval. If, for example, your mobile testing is offshored to India, your R&D team is in Eastern Europe and your target market is in the US, you need a solution that enables handsets to be tested and shared among these teams.

Adapting to rapid changes –"Here today, gone tomorrow" is probably the best way to characterize the pace of change in the mobile market. Approximately 30% of handsets are replaced every quarter. These extreme market dynamics dictate the need for continuous mobile testing to ensure that your applications remain relevant to the devices being used in your market.

Figure 2 –Android devices compatible with specific Android versions (Source: Android Developers)

http://theunderstatement.com/post/11982112928/android-orphans-visualizing-a-sad-history-of-support


Perfecto Mobile | Understanding the Mobile Development Cycle 4

UNDERSTANDING THE MOBILE DEVELOPMENT CYCLE

The illustrations below depict the various testing activities required throughout the mobile application

lifecycle. While many of these activities correspond to the classic software development model, the

unique characteristics of the mobile market impact each and every one of these activities. Moreover,

there are aspects of mobile testing (e.g., interoperability, porting) related to platforms and networks that

are unique to the mobile application lifecycle (depicted in orange) as explained in the next sections.

Figure 3 - Testing activities by phase, software QA vs. mobile QA (simplified)



Think Platforms – Know Your Market

In order to address market dynamics, mobile applications need to be developed and tested on multiple

platforms. You cannot assume that an application developed on Android will run smoothly on iOS. Thus, it

is important to identify 4-6 "must" devices and run rigorous nightly sanity testing and regression testing

on these devices. During the QA phase, it is recommended to extend the testing to cover ~10 "major"

devices to achieve a better representation of the market. The bulk of the functional and regression testing

would be performed against these devices. Automation in both of these phases is critical in order to allow

the release of new apps and functionality to the market in a timely fashion. Regardless of the phase,

around 30% of the devices will need to be replaced each quarter to account for new devices being

introduced to the market.

Figure 4 – "Must", "Major" and "Market" devices used on different testing stages (simplified)

Think Connected - Interoperability

Since the mobile device is a connected device, its interface with the environment is different than the

average web or desktop application. Network-related events could affect the quality of your application.

What happens, for instance, when a call or text message comes into your device in the middle of a

transaction? Different scenarios must be tested since a slight variation in timing could have a major effect

on the results. In addition, different network technologies (e.g., HSPDA, WCDMA, UMTS, LTE) might

change the way your application behaves and must be taken into account.



Automation can greatly reduce the risk involved in interoperability testing. While eliminating the errors

that inevitably result from manual testing, an automated system can run these tests systematically and

record the details of each iteration.

Continuous Testing - Compatibility

Compatibility testing (porting) requires continuous testing on an ever-changing stream of new devices

and tablets being introduced to the market. Compatibility testing is aimed at making sure your application

operates properly on the 30-40 most relevant devices in your market. This stage is more conducive to

manual testing since the testing will be carried out on multiple carriers, form factors and OS versions.

The most important aspect of this testing is to keep your selection of "market" devices up to date by

replacing ~10 of these devices every three months. This means that you will need a logistical solution

which will enable you to ensure availability of these devices when you need them,

as well as collaboration between testers and R&D.

Consider the security aspects as well, since "loose" management of these devices

could inadvertently lead to IT security breaches. Examples include enabling devices

to be in unknown locations, giving them access to your internal network, or

allowing access to your USB ports (some corporate policies disable this possibility

by default to prevent information leakage).

Shorten Development Cycle to Minimize Requirement Gaps

In traditional software development, new software versions are released to meet ongoing market

requirements. During the time between releases, the gap between the software functionality and the

market requirements begins to grow until the next version closes the gap. In the mobile world, due to the

extreme dynamics and new technologies being introduced to the market, the gaps grow much faster than

in the standard software world. To address this issue, it is very important to release software versions as

frequently as possible, avoiding large gaps and allowing for applications that successfully address the

needs of users.

Figure 5 - Shorter mobile development cycles reduce risk, every release closes the gap from existing requirements

RiskGap from

requirements

Mobile Dev.

TimeSW Cycles

Short Cycle

Mobile dev.


Perfecto Mobile | Selection Criteria for Your Enterprise Mobile Testing Solution 7

SELECTION CRITERIA FOR YOUR ENTERPRISE MOBILE TESTING SOLUTION

With an understanding of the particular business and technological challenges of the mobile environment,

as well as the key testing activities in the mobile development cycle, you are now ready to select the right

mobile testing solution for your enterprise. This section outlines the key selection criteria that you should

consider.

Automation is Imperative

As development cycles become shorter over time, the need for automation in regression testing has

turned from “nice to have” into “mandatory”, particularly when using agile or iterative development

methodologies. By automating the functional and regression testing of your mobile applications, you can

ensure the quality and performance of your applications on any device, OS, network and location.

Automation allows more tests to be performed on more devices in less time and reduces the

"requirements gap." The result is a shortened and systematic ALM cycle that allows for continuous QA,

better coverage, easier re-creation of problems and substantial cost savings.

Be careful not to underestimate the complexity of launching mobile apps. It is common to experience a

cycle of over-optimism following development on the first platform (OS), followed by disillusionment

when difficulties are encountered once the second platform and associated devices are added to the mix.

The realities of an extremely dynamic market require a well-planned and methodical approach.

In light of this fragmentation, it is highly recommended to adopt a device-agnostic testing approach that

allows you to write test scripts once and then re-use them on multiple platforms. Script automation

should support low level functionality (e.g., touching a key, touching the screen) as well as logical

abstraction which enables the execution of virtual functions (e.g., login) that are not dependent on a

particular device or platform.

Cloud-Based Access to Devices

To ensure application compatibility on the platforms that matter to your users, best practices indicate

that you'll need access to 30-40 fully functional devices at any given time, and you'll have to replace

approximately 10 of these devices each quarter to stay up to date. The logistics of managing this number

of handsets in diverse geo-locations is challenging, to say the least. The amount of supported devices

should be considered. With the rate of introduction of new devices and OS versions, assume that for the

first year the amount of devices will grow significantly as your user base builds “legacy” support.

Cloud-based access to REAL handsets located in live networks allows you to avoid the hassle and costs of

procuring and managing new devices (i.e., shipments, network availability, active plans, SIMs, etc.). Since

all devices are managed in the cloud, you no longer have to worry about complicated logistics. Devices

can be easily added or replaced within a few hours in response to market needs. In addition, a cloud-

based approach facilitates offshoring and enables globally distributed teams to share a device in "live"

testing and to replicate issues for improved troubleshooting.


Perfecto Mobile | Selection Criteria for Your Enterprise Mobile Testing Solution 8

The table below presents a comparison of cloud-based mobile testing with other types of solutions:

Figure 6 - Comparison: Different approaches to mobile testing

Use Existing ALM Resources

For enterprises with an ALM infrastructure already in place, the goal is to minimize the cost of new tools

and skill sets required for mobile testing. Rather than creating a new separate ALM environment, you can

leverage existing tools and processes by extending your current ALM framework to support mobile

testing.

By managing mobile, desktop and web application testing through a single integrated ALM platform,

organizations can centrally manage and track all application projects and ensure consistent workflows and

processes. This will help to eliminate the costs associated with maintaining, managing and synchronizing

separate systems for desktop and mobile testing.

The integration of mobile testing within your existing ALM enables centralized and complete test

management and visibility for your mobile apps. Using a single platform for managing and automating the

application lifecycle helps to ensure mobile quality and performance while reducing total cost of

ownership.

Natural Extension of Your ALM Platform

As opposed to a simple exchange of data items between two disparate applications, creating a natural

extension of your ALM platform to mobile will ensure a seamless user experience. By truly embedding the

mobile testing capabilities within your ALM environment (e.g., HP ALM, IBM, etc.), testers can work within

the existing application, manipulating mobile functions from within the system using known language and

syntax. Mobile testing solutions that use a familiar user interface, scripting language and workflows will

reduce learning curves and allow for a rapid implementation. This enables you to maintain the same

functions and staff you already have in the organization, while preserving the knowledge and experience

of your team.


Perfecto Mobile | Case Study: Extending HP Automated Testing to Mobile 9

Secure Environment

To meet your enterprise's security requirements, it is essential that mobile application testing can be

performed on a dedicated bank of handsets (i.e., private cloud) that can only be accessed by your

organization. This ensures that the handsets you need are always available for testing, and that your

application and data are always secure. The private cloud should be configurable to comply with your

organization's security policies, including firewall requirements and other needs.

CASE STUDY: EXTENDING HP AUTOMATED TESTING TO MOBILE

Perfecto Mobile and HP have collaborated to develop a complete and fully integrated solution for

automated testing of mobile applications within QuickTest Professional (QTP). Already deployed by a

number of enterprises, MobileCloud for QTP integrates the core scripting language and flow control of

QTP with Perfecto Mobile's comprehensive cloud-based automated mobile testing solution. This proven

solution enables enterprises to naturally extend their existing HP Functional Testing and ALM

environment to support mobile applications.

HOW IT WORKS

MobileCloud for QTP gives testers immediate access to hundreds of devices located anywhere in the

world from inside QTP. To create a mobile test script, testers login to QTP, connect to the MobileCloud

and choose one or more devices to work on. Users work on the device directly from the familiar QTP user

interface, using the rich feature set and scripting environment provided by QTP. There is no need to learn

any additional applications. Tight integration at the UI level allows testers to use the same UI for normal

QTP scripts, mobile scripting and hybrid scripting.

MobileCloud for QTP enables high-level intelligent mobile specific functions that mimic the actions that a

real user would take when operating a mobile device (e.g., press keys, open browser, select image, type

text, etc.), allowing fast, easy and maintainable mobile scripting. Perfecto Mobile's ScriptOnce™

technology enables advanced functions, such as dynamic search for text and images (including scrolling)

on screen and visual relationship association that facilitate the creation of device-agnostic scripts during

the recording phase.

Complete and detailed testing information from executed mobile commands are embedded directly into

the QTP report. These enhanced QTP reports include active screenshots and video recording of the test

run, as well as standard information such as test status and description.

The MobileCloud for QTP solution enables enterprises to cost-effectively extend their functional and

regression testing to mobile through the following capabilities:

Automated mobile testing that leverages your existing QA knowledge, platforms and workflows

Secure cloud-based access to hundreds of real devices in live networks

Device-agnostic mobile scripting enabling re-use and porting of scripts to multiple platforms


Perfecto Mobile | Case Study: Extending HP Automated Testing to Mobile 10

Native QTP scripting for mobile eases script creation and maintenance

Rich QTP reports with embedded screenshots and video improve collaboration among teams

Full back-end integration with HP Quality Center enables centralized test management

Figure 7- HP QuickTest Professional UI with MobileCloud integration

REAL-WORLD BENEFITS

MobileCloud for QTP was implemented at a large financial institution to test mobile applications (native,

mobile web and web) that process financial transactions on a variety of iPhone, Android and BlackBerry

devices. One of the test scenarios called for the transaction to begin on a mobile device and then verify it

on the website. This complex scenario demonstrates the unique value of the integrated solution, i.e., the

ability to leverage existing QTP assets and to call mobile assets as needed directly from QTP.

The financial institution achieved the following tangible results using the MobileCloud for QTP solution:

Reduction in Mobile regression testing time – From 3 weeks to 2 days to execute more than 30 end-to-end test cases test on more than 10 handsets

Improved Device Coverage - Ability to change and support new devices for automated testing in a "plug and play" manner as needed.

Improved Test Coverage – Broadened testing to 30+ test cases across 10+ handsets (compared to 4 handsets previously)

Improved Quality – QTP based automated enabled end-to-end testing for transactions across web and mobile systems

Debugging support – QA team can provide video of the defect to developers and provide access to remote device in private cloud for debugging and issue resolution

Better QA Visibility for Management – Automated mobile test results are available in HP Quality Center for consolidated reporting across platforms and end-to-end quality measurements


Perfecto Mobile | Conclusions 11

CONCLUSIONS

To address the complexities of mobile application testing in a highly fragmented and dynamic mobile

market, enterprises need to implement a well-designed mobile test automation strategy. This strategy

should reflect the unique needs and challenges of the mobile environment, as well as new aspects related

to the mobile development cycle (e.g., interoperability, porting) that do not exist in standard software

ALM.

Unlike desktop application development, in the mobile world it is critical to think platforms. With over

580 different mobile devices released to the market in the first eleven months of 2011, mobile

applications need to be developed and tested on multiple platforms and operating systems. Moreover,

since new devices are constantly being released to the market, enterprises must continuously test their

applications to ensure compatibility with the most relevant devices in their market. This is an ongoing

process since one-third of these devices will need to be replaced each quarter to keep pace with market

changes. Given the extreme market dynamics, shortening the mobile ALM cycle can help to avoid large

gaps between versions and market requirements, allowing for applications that adapt to the changing

needs of your users.

As mobile apps and services become business-critical, your choice of a mobile testing solution will have a

major impact on business operations as well as your organization's internal QA activities. When choosing

your solution, it is recommended to consider the following main selection criteria (dubbed "ACE"):

Automation – mobile test automation will enable you to shorten the ALM cycle, increase

coverage, facilitate re-creation of problems and save costs

Cloud-based platform - Cloud-based access to REAL handsets located in live networks allows you

to avoid the hassle and costs of procuring and managing new devices, while facilitating offshoring and collaboration

Use Existing ALM Resources - Leverage existing tools, processes and knowledge by extending

your current ALM framework to support mobile testing

Solutions that meet these criteria will help your enterprise to address the complexities of multi-platform

development in a highly fragmented mobile market, while enabling you to cost-effectively extend your

ALM environment to mobile.

ABOUT PERFECTO MOBILE

Perfecto Mobile is a global leading provider of cloud-based testing, automation and monitoring solutions

for mobile applications and websites, utilizing a wide selection of REAL and emulated mobile devices

accessible via the web. Perfecto Mobile’s MobileCloud™ Platform enables developers and testers to

access and control a comprehensive range of the latest mobile devices connected to live networks around

the world. Users can rely on Perfecto Mobile’s handsets and tablets to develop, test, deploy and monitor

their mobile applications and services without having to physically obtain them.


Perfecto Mobile | About Perfecto Mobile 12

Perfecto Mobile and HP have collaborated to develop the world's first complete solution for automated

testing of mobile applications within HP's QuickTest Professional (QTP) product. MobileCloud for QTP

integrates the core scripting language and flow control of QTP with Perfecto Mobile's comprehensive

cloud-based automated mobile testing solution. This ground-breaking solution enables enterprises to

naturally extend their existing HP Functional Testing and Application Lifecycle Management (ALM)

environment to support mobile applications with both real devices and emulators.

Documents

Practical_Guide_WP-2012 - Mobile Testing Solution