STeP-IN SUMMIT 2014 · 3/26/2014 · 1 Cost – open source / commercial 2 Technologies supported 3 Scripting Language 4 Integration with Test Management Tools 5 Reusability, Maintainability

STeP-IN SUMMIT 2014

11th

International Conference on Software Testing

June 2014 at Bangalore, Hyderabad, Pune - INDIA

Effective Tool Selection Criterion for Mobile Automation

by

Krishnaveni M and Pavan Kumar YG, Test Automation Lead, Tata Consultancy Services Limited

Copyright: STeP-IN Forum

Published with permission for restricted use in ‘STeP-IN SUMMIT 2014’ in agreement with full copyrights from owner(s) / author(s) of material. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the prior consent of the owner(s) / author(s). This edition is manufactured in India and is authorized for distribution only during ‘STeP-IN SUMMIT 2014’ as per the applicable conditions.

Practices Experience Knowledge Automation

Produced By Hosted By

www.stepinforum.org www.qsitglobal.com

http://www.stepinforum.org/

http://www.qsitglobal.com/

1 Copyright © 2013 Tata Consultancy Services Limited

March 26, 2014

Mobile Applications Functional Testing Automation Tool Selection Criteria

An Original White Paper for Submission to STeP-In Forum 2014 Authors:

Pavan Kumar YG Krishnaveni M

2

Agenda

Mobile Application Trends

Mobile Automation Key Considerations

Mobile Automation – How Different?

Mobile Automation Strategy – Key Parameters

Automation Tool Selection Key Parameters

Key Parameters

Best Practices

STeP-IN SUMMIT 2014 Effective Tool Selection Criterion for Mobile Automation Paper Publication

1

3

Mobile Application Trends

Increased usage of

Smart Phones and

Tablets

Global mobile phone sales are on pace

Rapidly growing

market of mobile

applications

Increased mobile app downloads

Time lag between

mobile device launch and app launch

Mobile Application Assurance - The key focus area for IT organizations

*Source - www.Gartner.com

4

Mobile Automation Key Considerations

Multiple Device Models

Multiple Platforms –

Android, iOS

Multiple Browsers

Frequent Releases

Regular platform upgrades

Mobile Networks

Increased Regression Scope and

Cycles


2

5

Mobile Automation – How Different?

One script across Platforms (Android, iOS), Devices (Smart Phones, Tablet) and Mobile Networks Infra dependency (Platforms, Devices, Networks, Device Emulator, Wi-Fi) Frequent Object changes – Platform Updates Frequent Object changes – Application upgrades Continuous Integration Testing (CIT) Interoperability Testing

1

6

Mobile Automation - How Different?

Localized Networks and Devices dependency Scripting Languages Cross Browser testing Test Data automation Development and Maintenance overhead Integration with Test management tools Unattended execution

Thorough Automation Strategy is Key to Success

2


3

7

Mobile Automation Strategy – Key Parameters

1 Automation Objectives

Automation Test Suite Profiling

Automation Feasibility Study

Tool compatibility Analysis

Automation Fitment Analysis

Return on Investment (RoI) Estimation and PoC findings

Environment – Platforms, Devices, Emulators, Networks

2

3

4

5

6

7

Selection of automation tool - a critical parameter for automation success

8


Jail Breaking / rooting 1

Device emulators and Real devices 2

Support for Platforms (Android, iOS, Windows, Blackberry), Device (Smart Phone, Tablet), Browsers and Device emulator

3

Both WiFi and USB support 4

Object identification technology (Native, Image, Optical Character Recognition etc,..) 5

IDEs supported - can you use your current automation tool as IDE? 6

1


4

9


Application code level instrumentation 7

Supports cloud based environment – Mobile devices 8

CIT tools support 9

Concurrent execution on devices 10

Unattended execution 11

Application Programming Interface (API) – Technologies supported 12

2

10

Key Parameters – Emulators/Real Devices

Device Emulators • Network conditions can’t be emulated • Can’t be used for Interoperability testing

scenarios • Requires high end configuration – i5 processor

and 8GB RAM • All the latest versions may not be available • Could be leveraged for smoke testing

Real Devices • Most realistic environment and results • Takes longer time to yield RoI – depending on

usage pattern and device upgrades • Test across different network conditions • Suitable for Interoperability and localization

testing (carriers, devices)


5

11

Key Parameters – Supports cloud based environment – Mobile devices

•Limited scope of application, device and duration? •Comparatively less cost Public Cloud

•Secured •Better Compliance •Comparatively costlier

Private Cloud

•Highly Secured •Enterprise wide usage •Increasing number of applications •Plan to roll out multiple releases over a period of time

On Premises Cloud

Combination of Cloud, Emulator and Real devices?

• No upfront investment for devices • No maintenance overhead – QA team can focus on primary objective • Availability of latest OS, Devices

12

Key Parameters – Object Identification Technology

•Recognize the objects based on how they are defined than how they appear on the screen

•The objects are identified by the logical properties (for example: element ID)

•Relatively robust object identification

Native

•The objects are identified as images •Works well with most of the platforms and objects •No performance issues either in identifying the objects or execution •Maintenance could be high - in some cases changes in screen resolution may lead to script failure

Image based

•Works only for text recognition •Works well for specific languages •Very easy to get the text

Optical Character

Recognition (OCR)


6

13

Is Open Source Mobile Automation Tool A Right Option?

Point solutions offered by open source tools 1

Dependency on technical experts 2

Permits high degree of customization 3

Suitable for real devices and device emulators 4

Good choice if cost reduction is primary objective 5

14

Key Parameters - Integration

1 Suitable for the existing automation and manual testing processes?

2 Supports current automation framework?

3 Plugs-in to the current reporting framework?

4 The suite of tools support performance testing and Integration testing?

5 Ability to Integrate with CI tools – for example Jenkins, Hudon

6 Ability to export to multiple scripting languages – for example Java, VB, Python or Perl


7

15

Automation Tool Selection Criteria - Commonly Known Criteria

Cost – open source / commercial 1

Technologies supported 2

Scripting Language 3

Integration with Test Management Tools 4

Reusability, Maintainability 5

Support for data-driven execution 6

1

16

Automation Tool Selection Criteria - Commonly Known Criteria

Learning curve 7

Tool Support 8

Frequent or regular tool upgrades 9

Debugging Feature 10

Error Handling and Recovery Scenarios 11

Execution Performance 12

2


8

Best Practices

18

Best Practices

Identify upfront, the key application changes (GUI) from

– Platform to platform (for example: Android to iOS)

– Smart phone to tablet (for example: iPhone to iPad)

– Smart phone to Smart phone ( Samsung 3.2” display phone to

Samsung 5” display phone)

Calculate the reusability % of test cases and scripts across

platforms and devices

Identify upfront, the range of devices and models to be tested

1


9

19

Best Practices

Scope the feasibility study to each type of test case, screen,

object, platform, device – as specific as possible

Calculate automatability % and RoI for each platform

individually

Identify the high level reusable scripts (keywords) during the

feasibility study

Establish test data strategy upfront

Automate test data creation, where ever possible

2

Questions ???


10

IT Services Business Solutions Consulting

Thank You

For queries please contact - [email protected] [email protected]


11

Documents

STeP-IN SUMMIT 2014 · 3/26/2014 · 1 Cost – open source / commercial 2 Technologies supported 3 Scripting Language 4 Integration with Test Management Tools 5 Reusability, Maintainability