Template - NFI...Making your business mobile is no longer a matter for debate. • Today’s mobile users – your customers, your employees – your suppliers – even your shareholders

Mobile Testing

April 2013

© Sogeti

Mattias Bergströmner

• Mattias has 12 years of experience within the QA and test domain. Mattias helps and support leading companies in various questions and challenges related to quality assurance and testing.

• Past – Ericsson Software Technology AB

– Ericsson Mobile Platforms AB

– System verification

– Capgemini Sweden

• Present – Mattias is head of the business area QA &

Testing in Sogeti Sweden.

– Mattias belongs Sogeti´s global team of experts

within QA & Testing.

• You reach Mattias by

– Email: [email protected]

– Tel: +46 730 88 40 26

– Twitter: @mabergst

mailto:[email protected]

© Sogeti

Agenda

• Mobility – Your Business. Always On.

• Mobile Testing Priorities and Challenges

• What is different with Mobile Testing

• 10 Mobile Testing Success Factors

• Test Tools for Mobile Testing

• Summary

© Sogeti

This is Sogeti QA & Testing Practice

• 9500 test professionals worldwide, 400 in Nordics and 200 in Sweden

• Test service portfolio that ranges from test strategy development to outsourced testing operations across the whole world

• Consultants trained in own methods TMAP & TPI and also ISEB/ISTQB certified

• Alliance partners with HP, IBM-Rational, Microsoft, Oracle and Microfocus

• Analysts such as Nelson & Hall, Ovum and IDC confirms our position as the leading company within testing in the world

© Sogeti

Our Methods, Tools And Accelerators Create

Immediate Efficiency And Quality Benefits

Client

dedicated test lines

Market leading methodologies and accelerators for testing and test process improvement

© Sogeti

Mobile Solutions and Testing

• From mobile solution design to mobile testing

• Mobile Solutions

– 1,500+ mobile experts world wide

– Mobile Solutions Center of Excellence, Mumbai

• Testing

– World leading testing provider

– 10 000 Test Professionals

– Mobile testing is a joint initiative between the Mobile solutions and Testing global service lines

– Mobile Testing Center of Excellence, Mumbai

© Sogeti

Agenda






• Summary

© Sogeti

Mobility – Your Business. Always On.

Enterprises ask:

“How can we enhance and extend our products and services?”

Making your business mobile is no longer a matter for debate.

• Today’s mobile users – your customers, your employees – your suppliers – even your shareholders – expect interaction with you at their fingertips

• Enterprise mobility goes beyond mobile apps – it influences how you run your business

• Mobility has reset expectations of what constitutes application quality – robust performance and usability on the move – adding a further layer of complexity

Business travelers: the past 12 months, • 26% have used an app to check flight schedules • 11% have used an app to book flights Source: Forrester Research

One of Europe’s largest travel operators generates 200 million Euro per year in mobile channel

Mobile bookings reaches 15% of total bookings by end of 2012 Source: Wall Street Journal

© Sogeti

Business Critical Mobility

• Mobile and consumerization of IT

– Majority of mobile phones sold are app centric

– >50 billion app downloads

– >1 million apps in App Stores

• Mobile business is in exponential growth

– Transactional features in responsive web sites and apps

– Products and services

• Accelerated internal processes

– Mobile process management including field and sales service, approval and time management

– Improved social collaboration using social enterprise platforms

– Mobile access to business intelligence solutions

© Sogeti

Mobile evolution

© Sogeti

Agenda






• Summary

© Sogeti

Quality Assurance and Mobile Testing

The latest annual World Quality Report 2012-13* research found that:

• Organizations are struggling to manage the mobile challenges, with only one-third (31%) of those surveyed currently formally testing their mobile applications

* Based on 1,554 telephone interviews conducted for report

sponsors Capgemini, Sogeti and HP across 25 countries

© Sogeti

Mobile Testing Priorities: Performance

WQR 2012-13 Survey:

• Performance testing is a market priority (64% of respondents)

Mobile Transactions Survey*:

• +1 second delay: Up to 8% of users stop using the app

• 85% of users expect the mobile experience to be better than or equal to online using a laptop or desktop computer

*Source: Harris Interactive Mobile Transactions Survey, 2011

© Sogeti

Mobile Testing Priorities: Usability

• App users expect an easy to use, beautiful user interface, which is both distinctive and enjoyable

• 47% expect the mobile experience to be better than the in-store experience

• 85% expect the mobile experience to be better than or equal to online using a laptop or desktop computer

© Sogeti

Mobile Testing Priorities: Compatibility

• 52% of respondents do not have the necessary devices available for testing (WQR 2012-13)

• Companies consider compatibility testing almost as important as functional testing (46% vs 48%)

Challenges of proliferation and fragmentation: 3,997 Android models*

*Source: opensignalmaps.com study, May 2012

© Sogeti

Mobile Testing Priorities: Security

• Explosion of mobile applications creates new security challenges such as:

– Lost or stolen devices

– Mobile malware

– Targeted attacks on devices

• Security issues include:

– Confidentiality: Does your app keep your private data private?

– Integrity: Can the data from your app be trusted and verified?

– Authentication: Does your app verify you are who you say you are?

– Authorization: Does your application properly limit user privileges?

– Availability: Can an attacker take the app offline?

– Non-Repudiation: Does your app keep records of events?

© Sogeti

Summary Mobile Testing Challenges

• Lack of devices and operating systems for test

• Rapidly changing technology landscape, such as device types and operating systems, leading to continuous need for updates

• Lack of methods, solutions and experts specific for mobile testing

• Lack of the right tools

• Lack of re-usable assets and frameworks

• Performance, usability and security require specific attention

© Sogeti

Agenda






• Summary

© Sogeti

Mobile Testing is business as usual...

© Sogeti

... but Mobile Testing adds additional hot spots

© Sogeti

..Screen sizes today

© Sogeti

Device fragmentation creates a need to perform

device level testing & UAT provisioning for mobile apps

Testing cycle for desktop application

Requirements Analysis

Execute regression testing /

automation

Test report analysis Test Plan

Test case creation /

automation scripting

Execute new features testing

/ manual testing

Requirements Analysis

Device Level validation /

manual testing

Device provisioning &

device UAT

Test report analysis

Test Plan

Test Case Creation

Execute new features testing

/ manual testing

Execute regression testing /

automation

Testing cycle for mobile application

Source: Forrester - Strategies For Testing Mobile Apps

Unlike the desktop, the mobile space includes multiple platforms and form factors. These complexities lead many organizations to test on real devices, and most do this manually

© Sogeti

The complete eco system must be considered

in your mobile testing strategy

© Sogeti

Agenda






• Summary

© Sogeti

10 Mobile Testing Success Factors

Mobile

Connectivity

Physical characteristics

Location

Test automation

Platform

Fragmentation

User experience

Third parties

Solution

Integration

Performance

Security

© Sogeti

Success Factor #1:

Testing for Connectivity

• Mobile solutions depend on network connectivity

• Solution design and testing need to address:

– variable bandwidth

– offline and flight mode scenarios

– user sessions moving between different network conditions

• To ensure consistent behavior, it is recommended to conduct a combination of:

– automated network simulations

– manual testing in real network conditions

© Sogeti

Success Factor #2:

Understanding Physical Characteristics

• Physical characteristics differ significantly from non-mobile solutions

• Mobile testing needs to take several characteristics into consideration:

– Screen size

– Touch and gesture capabilities

– Orientation (vertical or horizontal) and movements in 3 dimensions

– Camera

– GPS

© Sogeti

Success Factor #3:

Location Simulation

• An increasing number of solutions utilize location data and GPS integration

• Features may depend on location or distance to other locations

• Successful mobile testing ensures quality across different GPS implementations and enables efficient location simulation

– Make use of devices in the cloud, located in different parts of the world

– Simulate localization scenarios in emulators

© Sogeti

Success Factor #4:

Test Automation

• Test automation using scripting or recording makes mobile testing significantly more efficient

• Smart test automation utilizes parameterized scripts for both device types and user interaction, and is enabled to run on devices in-house and in the cloud

• Some sophisticated user interactions cannot be fully automated, hence there is need for planning for what tests can be automated and what tests must remain manual

© Sogeti

Success Factor #5:

Addressing Fragmentation

• Fragmentation of operating systems and device types continues to challenge solution design and testing

• Most solutions need to support hundreds of device types and several versions of operating systems

• A relevant mobile testing tool box includes:

– physical access to most major combinations of device type and OS

– ability to run manual and automated tests across both physical devices and emulators

– access to cloud based platforms to maximize number of test devices

© Sogeti

Success Factor #6:

User Experience

• User experience is a main criteria for success

• Testing user experience is an art that consists of both objective and subjective measurements

• Key elements include:

– User interaction – level of intuitiveness and efficiency

– Navigation – adherence to platform rules and guidelines

– Signup and login – clarity and ease of use

– Layout and user interface design – look and feel, aesthetics, precision in layout

– Exception handling – graceful, returning user friendly messages

© Sogeti

Success Factor #6:

User Experience

© Sogeti

Success Factor #6:

User Experience

Not so good!

© Sogeti

Success Factor #6:

User Experience

© Sogeti

Success Factor #7:

Engaging with Third Parties

• Most apps are distributed through open and public app stores, each with its own set of guidelines

• Apps that fail to adhere to guidelines may be rejected

• Given the third party review process, there is an inherent bug fix latency in updates

• Operating systems are frequently updated. New releases may break existing apps.

• Guidelines: – Use common testing checklists

based on latest app store rules and guidelines

– Rapid testing cycles for updates in order to minimize impact of bug fix latencies

– Test existing apps on beta versions of operating systems

© Sogeti

Success Factor #8:

End-to-End Integration Testing

• Transactional apps integrate with back end servers and applications

• System integration is often the most challenging aspect of any solution design

• Mobile testing becomes highly relevant as it provides a foundation for end-to-end integration testing

• Mobile Testing should be considered as an integrated activity in the overall solution testing

© Sogeti

Success Factor #9:

Performance

• Mobile solutions can overload servers, impacting performance for all users, for several reasons:

– Rapid growth of number of users

– Average life span of transactions increase due to variations in bandwidth and latency

• There are significant performance differences across devices. Tests should be conducted on high-end and low-end devices.

• Successful mobile testing focuses on network capabilities, system integration and back end layers, as well as the app itself

© Sogeti

Success Factor #10:

Security

• Mobile solutions drive an increasing number of scenarios where sensitive systems are accessed, and private data is in transit and at rest, i.e. stored in mobile devices

• Given the physical nature of mobile devices, they are more easily forgotten, lost, or stolen

• Mature mobile testing aims at

– Securing confidentiality and integrity of data

– Validating whether authentication and authorizing is secure

– Verifying that systems are keeping records of events (non-repudiation)

© Sogeti

Agenda






• Summary

© Sogeti

Perfecto Mobile – Test Automation (UFT Mobile)

Keyword-based scripting

ScriptOnce® - Patented

device agnostic automation

Device Replacement takes

minutes not weeks…

Native HP ALM Extension

© Sogeti

Performance Test Tool

• Performance Testing

− End-to-end testing

− Simulation of user types

− Network virtualization

− Partnership with Neotys

© Sogeti

Agenda






• Summary

© Sogeti

Summary

Mobile testing needs to be addressed in the company’s overall Mobile strategy

As apps become increasingly business relevant and business critical – do not neglect the importance of quality assurance

Make sure you understand why mobile testing is different and apply the right actions to meet them

Make sure your testing efforts take the device’s characteristics into consideration

Mobile apps is “Everywhere and IMMEDIATLY”

Your business is always on

Documents

Template - NFI...Making your business mobile is no longer a matter for debate. • Today’s mobile users – your customers, your employees – your suppliers – even your shareholders