101 – How To Deliver Winning Mobile Apps

Perfecto Mobile &

Toronto Association of System and Software

Eran Kinsbruner is a director & mobile technical evangelist at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at ek121268.wordpress.com.

What To Do?

Great News – Your first Mobile Project has arrived!

Now What?

Presentation Objectives

• Mobile Market State

• Mobile QA/Dev Objectives

• Sustaining a Mobile QA Plan Based on CQL• Functional Testing

• Non Functional Testing

• Regression Testing/CI

• Pillars Of A Continuous Quality Lab

• Mobile QA Best Practices

Mobile Market State

Digital experience drives innovation

• IOT

• Complex use cases

Fragmentation becomes more painful

• Devices

• OS’s

Tool mix as a key for agile adoption

• Commercial, Open Source, Proprietary

Mobile QA & Dev Objectives

Source: OpenXcell

Qualit

y

Velocity

Increase Quality

Increase Velocity

Mobile QA & DevObjectives

QA & Dev Testers Vision (aspiration)

Increase mobile app velocity without

compromising on quality

Keep up with mobile market changes and

meet end user requirements

Develop & Test against the real end

user profiles

Leverage unattended automation as part of

any nightly build through CI workflow

Receive actionable mobile app quality insights per build

RBC – QA Objectives

v

Improved Quality Optimized Cost Faster Time to Market

Operational Effectiveness

Operational Efficiency

Business Value

• Increase Test Coverage• Test Effectiveness – reduce test

duplication between QA & business • Reduce defect leakage to production

• Reuse test automation during UAT test cycles

• Decrease QA test cycles – via test automation

• Near zero downtime/production failures

• Applications perform as expected

• Test automation cost savings• Manual and automation test

management savings 10% using BDD Approach

• Onsite to offshore ratio 40:60• QA Technical Solution Lead across all

projects• Leverage Perfecto Mobile cloud solution

to save on devices and data plans

• Reduction in Cost of Testing due to productivity improvements-Maintenance and Projects

• Reduce regression test cycle by 5-10 days per project

• Perfecto Mobile UFT

• Continuous improvement in test design & execution

• Maximize Test Automation coverage• Increase re-use - BDD

• Enable business to respond to market change

• Proactive feedback on market changes to existing production mobile applications

SUSTAINING A MOBILE QA PLAN

Sufficient Coverage

• Choose the right devices and OS platforms for your project

• Get Perfecto Mobile’s Q1 2015 Mobile Test Coverage Index

Q1-2015 Mobile Test Coverage Index (Preview)

What’s Coming? Why It’s Important?

Manual & Interoperability

• Starting point for Pre-Automation phase (foundation)

• Candidates to keep manual:– Incoming events

• Calls

• Text Messages

• Notification pop ups

– Complex use cases to automate

Functional Automation – That Works

• Reusable, Repeatable, Predictable Tests– Unattended

– Continuous

– Easy to maintain

• Object Support On Any Mobile App Style– Dom, Visual, Native

– Instrumentation (Yes/No)

– System Level Control

• Flexible Frameworks

100% accurate native object rec.

Much faster than OCR

Support all languages

Does not often detect GUI defects

Imperative for enduser’s experience

Imperative for GUI glitches

Implement theWYSIWYG

Slower then objectlevel analysis

Dependent on 3rd party

Does not support all languages

Object level analysis

Visual analysis (OCR)

May require Instrumentation

Levels of Test Automation

Level 1

Level 2

Level 3

Level 4

Level 5

Navigation flows

Content validation

Error Validation

Data validation(if feasible)

UI validation(if feasible)

Check automation feasibility Identify scenarios for basic navigation & successful function flow

Navigation flows are automatedCheck automation feasibility of screen content

Level 2 Scenarios are readyCheck automation feasibility of error validation

Level 3 scenarios are readyCheck automation feasibility of data validation

Level 4 scenarios are readyCheck automation feasibility of UI validation and other complex features

Automated scenarios of basicNavigation % happy path are ready

Content validation scripts are ready and plugged in with exiting automated navigation BPT component

Error validation components are ready and plugged in with automated components of Level 2

Data validation scripts are written and plugged in with existing automated components of Level 3

Automated scripts for UI screen & other complex scenarios are ready and plugged with existing scripts

Data D

riven A

uto

matio

n Fram

ewo

rk

Non Functional Testing

• Cover key business transactions response time & general mobile app availability

• Early actionable performancefeedback & insights

• Assures high end user experiencein various conditions (aka: Wind Tunnel)– Various network profiles – Various locations– Various load conditions– Incoming events – Per app insights

Mobile App Monitoring

• The only Efficient Way to Monitor Mobile Apps

– Browsers and Emulators - irrelevant for mobile monitoring

• Capture Your True Mobile End User Experience

– Monitor mobile engagement scenarios, including interoperability on real devices on real networks

• Early Warnings and Fast Correction

– Real-time alerting triggered when transaction availability or response times are compromised

• Extend your existing quality assets for monitoring

– Automation test code

– Performance test code

What Makes A True CQL

Quality Lab

What Makes A Complete Continuous Quality Lab

Real Device Lab (Device as a Service)

Functional Manual Testing

Functional Automation

Performance Testing

Real environments & Profiles

Production Monitoring

Hybrid Cloud (Local, Public, Private)

Operations, Security & Support

3rd Party API’s & Integrations

Mobile QA Best Practices

Mobile Continuous Integration Recommended Work Flow

Dev Team

Source Control

Code Commit

CI Server

ReportingUnit Tests

Launch Perfecto Job

(Release Package)

Support today:

PM Native & Selenium

Automated App Deployment On Real Devices

Functional Testing

Integration Testing

Performance Testing

Continuous Feedback

Test Planning & Test Execution

Bug Reporting

Build Your Mobile App Test Plan Properly

Functional

• Validation of Functionality

• Smoke / Regressions Testing

• Offline access testing

• Negative Testing

Non Functional

• Network Strength / Outage / Recovery

• Different NW Types

• Peripheral Testing

• Services

Interoperability (IOP)

• Voice / SMS interrupts

• Notifications

• Battery /Cable Removal

Memory Leak

• Memory Usage

• Memory Leaks

• Garbage Collection

Installation Testing

• New App Install

• Uninstall and Reinstall

• Upgrade testing

Language Testing

• Validation for Locales (French & English)

• Images and Text

• Currencies, time zones etc.

• Context

Performance Testing

• CPU Usage testing

• Network Usage

• Page Render time or activity Render time

Security Testing

• OWASP Vulnerabilities

• Sensitive data (App/Device)

• Dynamic Testing

• Static Code Analysis

• Data Encryption

Usability Testing

• User Experience

• Competitive Analysis

• Expert Review

RBC Mobile Quality Lab

Team Size –

45 (Onshore/Offshore Mix)

>100 devices - Blackberry, Android & iOS platforms

Using Selenium

& UFT

Continuous Automation Regression Model

• Executing automated test scripts 24*7 (consider offshore team)

– Private Perfecto Cloud – used for project development• Execute automated testing suite to reduce project

timelines

– Internal Perfecto Cloud –used for project and maintenance testing on supported devices

– Perfecto Public Cloud – test non supported devices (new market devices/ mimic customer behaviour). Leverage Perfecto public metrics i.e. what is currently in use by public (device and configuration)

Hybrid Cloud

Q&A