Mobile Application Testing af Frederik Carlier, Jamo Solutions

Mobile Application TestingFrederik Carlierww Technology Strategist

11th September 2013

Agenda

• The Mobile Market• Testing for Desktop and Mobile Apps: Now and Then• Methods for Test Automation• Functional Testing• Performance Testing• Frameworks• Conclusion

Agenda


2013 Gartner CIO PrioritiesRanking 2013 2012 2011 2010 2009 2008

Analytics and Business Intelligence 1 1 5 5 1 1

Mobile Technologies 2 2 3 6 12 12

Cloud Computing 3 3 1 2 16 *

Collaboration Technologies (workflow) 4 4 8 11 5 8

Legacy Modernization 5 6 7 15 4 4

IT Management 6 7 4 10 * *

Customer Relastionship Management 7 8 18 * * *

Virtualization 8 5 2 1 3 3

Security 9 10 12 9 8 8

ERP Applications 10 9 13 14 2 2

* Not an option this yearSource: http://www.gartner.com/imagesrv/cio/pdf/cio_agenda_insights2013.pdf

http://www.gartner.com/imagesrv/cio/pdf/cio_agenda_insights2013.pdf

http://www.gartner.com/imagesrv/cio/pdf/cio_agenda_insights2013.pdf

2013 Gartner Strategic Technology Trends

Source: http://www.gartner.com/technology/research/top-10-technology-trends/

http://www.gartner.com/technology/research/top-10-technology-trends/

http://www.gartner.com/technology/research/top-10-technology-trends/

Tablets

Android63%

iOS32%

Windows Phone4%

Android iOS Windows PhoneWindows RT Others

Smartphone

Android79%

iOS13%

Windows Phone4%

BlackBerry3%

Linux1%

Android iOS Windows PhoneBlackBerry Linux SymbianOthers

Mobile Market Share

Source: http://www.idc.com/getdoc.jsp?containerId=prUS24253413

Source: http://www.idc.com/getdoc.jsp?containerId=prUS24257413

http://www.idc.com/getdoc.jsp?containerId=prUS24253413





Agenda


Waterfall approach for testing

• Party• Project Definition• Project Realization• Testing (used a project buffer)• Deployment to production• Party

Result: low return on investment on testing effort

Test Case Implementation & Execution – ROI?• Reusablity across

• Different mobile OS versions (new)• Different mobile OS vendors (new)• Different hardware (new)• Different versions of the app under test• Different browser

• Independent of• Screen resolution• Keyboards• Localization

• Re-use of existing• Testing tools• Automation expertise• Scripting / Programming language

Agenda


Manual Testing

• Not reproducible• Time consuming• On Premise?• Test Coverage

Jamo Solutions - Mission

The mission of Jamo Solutions is to provide technology and expertise that enables successful testing and monitoring solutions around the mobile end user experience.

Object Attributes & Identification

• Objects are identified by a unique combination of attributes

• This combination can be set seperately for every object or globaly for all objects of the same type

• Access to all attributes of an object

• Key advantage of object recognition based approach

• Depending on the types of application you are creating — mobile browser, native, hybrid and the level of interactivity (use of gesture sets, access to platform features), this lower-level access will be the stronger path.

Gartner, Magic Quadrant for Integrated Software Suites, 11 July 2013

A typical UI

Consider the differences between testing this application using OCR (screen scraping/bitmap recognition) and object recognition based approach

• All buttons share the same label• All buttons share the same form• All buttons share the same size

• Object recognition: direct access• Reliable• Reusable

• OCR: Need for more intelligence• Error prone• Reduced reusability

Portrait view vs. landscape view

Native, Web & Hybrid Applications

• Native app: packaged application deployed to the phone

• Web app: accessed through the browser

• Hybrid app: native app with embedded web views

Native App• Java • C# • C++• Objective C• ....

Web App• HTML• CSS• Java Script• Runs inside a

browser

Hybrid App

Agenda


Functional Testing - Overview

Mobile Test Automation• M-eux Test allows you to automate your mobile testing

scenarios• Based on future-proof, object recognition based

technology (no jail breaking / rooting, no OCR, iOS7 compatible)

• Wide range of support across mobile platforms (iOS, Android, Windows, BlackBerry)

Demo

Functional Testing - Overview

Integration with exiting tools & processes• Integration with industry-standard development tools

(HP UTP/QTP, Microsoft,...)• Integration with application life cycle management tools

(HP ALM/QC, Microsoft, IBM RQM, ...)• Allows effective re-use of and integration with your

existing testing processes and tools• Minimizes the total investment, increases the ROI

Browser-Based Functional Testing

What?• Allows you to automate the testing of web applications on mobile platforms

• Based on our proven object recognition based technology

• Allows for seamless cross-browser, cross-platform, cross-device scripting

• You test the real app; on real devices or simulators/emulators

Demo

Browser-Based Functional Testing

Cross Browser Script

Agenda


26

Performance/Capacity Management

Aspects of performance/capacity management• Performance Testing

• Load Generation• Response Time Measurement

• Performance Monitoring

04/11/2023 © Jamo Solutions

27


Load Generation• M-eux Test allows you to create automated test cases• You can replay these devices on multiple devices

simultaneously (physical or virtual)• We integrate with load generation tools (LoadRunner,

VS,…) through UFT/VS integration


28


Response Time Measurement• M-eux Test allows you to create scripts that execute an

action and wait for the operation to complete , and report timing data

• You can integrate this data with ALM tools (HP, Microsoft,…) to track performance over time


29


Performance Monitoring• M-eux Test allows you to create scripts that execute an

action and wait for the operation to complete or fail, and report timing data

• You can integrate this data feed with your NOC operational tools (HP, Microsoft,…)


Agenda


31

Best Practices Management

• Re-use scripting best practice from PC world:• Descriptive programming• Exception handling• Object reconfiguration• Frameworks ( Linear framework, function decomposion

framework, keyword-driven framework, data-driven framework, etc.)

• Best pracitices on setup• Lab setup• Scripting setup

Agenda


Conclusion

• Let your test be reproducable• Manage frequent test cycles• Create re-usable tests• Object recognition is the way to go• Change your system and app as less as possbile by

instrumentation• Note that the app is not the only thing to test

[email protected]+32 485 869 861

www.jamosolutions.com

Thank [email protected]+32 485 869 861

www.jamosolutions.com