It's a Phone First! How to Test Your Five-star Mobile Apps

BW4 Concurrent Session 11/7/2012 10:15 AM

"It's a Phone First! How to Test Your Five-star Mobile Apps"

Presented by:

Will Hurley Quality and Security Services

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Will Hurley Quality and Security Services

A practice leader of quality management and security for Idea Integration Corporation, Will Hurley has been at the crest of numerous software development waves during his twenty-four year career. Will represented the US Air Force in the DoD’s first joint service system architecture, led CMM assessments, and supported the development of the CMMi. After leaving the Air Force, he consulted for industry-leading companies, helping them adopt and apply new development methods. In his current position, Will consults with clients in North America on achieving desired system and product lifecycle characteristics.

28/09/2012

1

©2012 Idea. All rights reserved 1

It’s a Phone First! How to Test Your Five‐star Mobile Apps

Today’s discussionIdea and I

Acknowledgements

Five Star Mobile Application


improving mobile project outcomes and time to market.

28/09/2012

2

Since 1986, we’ve helped some of the world’s largest and most respected brands thrive in the digital spacedigital space.


The Presenter

Will HurleyPractice ExecutiveQuality Mgmt. & SecurityIdea

USAF Systems EngineerTBMCS Core Systems LeadJoint Service Architect GCCS


Idea

[email protected](781) 831-0389

Joint Service Architect GCCSUSAF Data AdministratorConsultantSmall Business OwnerPractice Leader

28/09/2012

3

This discussion captures mobile testing insights developed at Idea’s Global Solution Centers in Nashua, NH and Houston, TX.

Idea’s insights are supported by more than 10 years of mobile

Acknowledgements

g pp y ydevelopment and testing.

Insights are not SDLC or tool specific.

Data is updated on the quarter through a verity of sources.

ComScore®

Netbiscuits®


Harris Interactive®

OPA

Crittercism

5

5 Star Mobile applications


6

28/09/2012

4

5 Star Mobile radar chart


Ecosystem insights

o Capture more than 80% of smartphone users i h 3 k i

Understanding the mobile ecosystem and trends increase test effectiveness and product quality characteristics.

with 3 network carriers.

o Capture more than 85% of smart phone users with Android and iOS platforms.

o Application use is growing faster than browser use.

o Android users spend more time on carrier networks.


o iOS users spend more time on wireless networks.

28/09/2012

5

Ecosystem carrier

As of Aug 2012, 114 million Americans own smart phones.

US Carrier Smart Phone Market Share

p

60 percent of all mobile devices acquired in the U.S. are smartphones.

Verizon market share has grown fastest driven by iOS.

Consider market share when signing contracts for cloud services.

Verizon 31.60%

AT&T 26.30%

SprintT‐Mobile 9 50%

Other 15%


Sprint 10.50%

9.50%Tracfone 7.10%

Ecosystem oem

Samsung Galaxy S III leading 2012 sales may be passed by iPhone 5.

US Smartphone OEM

iPhone 4 took the top US device sales spot in 2011.

iPhone 3GS took 2nd place.

iPhone 4S took 3rd place.

BlackBerry Curve 8530 4th place.

HTC Evo 4G 5th place.

Samsung, 25.6%

LG, 18.4%

Apple, 16 3%

RIM, 6.3%

Other, 15.8%


10

HTC is the most active OEM for Android smartphones.

16.3%

Motorola, 11.2%

HTC, 6.4%

28/09/2012

6

Ecosystem platform

66 percent of Americans ages 24-35 own a smartphone

US OS Share

own a smartphone.

Apple and Android market share are growing.

Market share for all other platforms is shrinking.

Android devices out number iPhones by 3-to-2 in the U.S. market.

Android52.2%

Apple33.4%

RIM9.5%

Microsoft3.6%

Symbian0.8%

Other0.5%


Platform fragmentation

Fragmentation is a real but manageable issue for all platforms.

A d id f i iAndroid faces more issues since it resides in an open ecosystem.

Android OS adoption is slower than iOS.

Trades reporting >½ of all iPhone users that can upgrade to iOS5 have.

iOS6 expected to drive largest upgrade population to date.


upg ade popu at o to date

Crash data are normalized as crashes as a percentage of each application launch.

28/09/2012

7

Ecosystem tablet

An estimated 74.1 million consumers are using tablets.

Tablet Share (Platform)

iPads account for more than 70% of tablet network traffic.

More than 80% of tablet traffic is wireless, not cellular.

Tablet adoption is balanced between males and females, trending older with households earning $50K or more.

Apple46%

Android42%

Other12%


Kindle Fire 28%

iPad 2 31%

Customer Experience insights

o Good design is not accidental. Get help if d i

Mobile users expect more.

you need it.

o Good design is sticky.

o Unstructured or unsolicited feedback has more action value than a survey.

o Analytics portray the customer voice.


28/09/2012

8

Experience components

How easy is it for your customer to use your application.

How does the li ti k

The application does what your customer wants it to do.


application make your customer feel.

Customer experience

Four in ten mobile users are disappointed with applications from their favorite brands.

Text

Smartphone Usage

Three-quarters of users surveyed believe a company's mobile app should be easier to use than its Web site.

Users spend more time with mobile applications than their mobile browser.

82% of iOS and Android Apps crash once out of every 10 loads

Text message, 74.60%

Used browser, 48.60%

Used App, 48.60%Social net

or Blog,

Games, 31.80%

Music, 24.50%


Less than 5% of mobile apps are still in use 3 months after downloading.

35.70%

28/09/2012

9

Experience sticky


Experience design


28/09/2012

10

Experience themes

Based on conversations, we found several common themes:

• All purchasing because it’s required by their apartment complexapartment complex.

• All participants were referred to site via a brochure.

• Participants pleased overall with the simplicity of the online process

• Liked being able to select from presented options in order to simplify and speed up the process.

• Appreciated the ability to get additional


Appreciated the ability to get additional details where necessary.

• A majority didn’t use these features. • All participants pay annually for their

policy.

o Functional requirements are only part of the customer experience.

o Key moments in customer interaction demand good

Experience maturity

design.o If you don’t have experienced designers on your team

bring them in. o Understand accessible and emotional goals at every key

moment.o Don’t predict how people will respond to a new

experience test it


experience, test it. o Allot the time to make incremental changes on all key moments.

28/09/2012

11

Functional insights

o Test while preforming common smart phone k

It’s a phone first.

tasks.

o It’s cheaper, faster and easier to test in emulated environments first.

o Testing types are the same as Web based n-tier applications

o Manage functional and non-functional insights across projects.


g p j

o Create a mobile test strategy document.

o Reduce tool and process resistance between groups.

Manage scope

Carriers

Operating Systems

OEM Devices


Com & Sensors

28/09/2012

12

Manage knowledge


• Display and Graphics• Input Devices

Manage modularity

• Sensors• Permissions• Intents• Phone and messaging• Deploy and Upgrade

A l i


• Analytics• Your application’s key moments

28/09/2012

13

Emulation(Simulation) testing

Does not require the physical device.

Can still use network services.

Emulation requires experience with specific platform development tools.

Static security and performance testing can be performed on source code.

Emulation is best for:

Investigating architectures and application layers

Agile testing of key features and


g g yinteractions

Evaluating code quality characteristics

Research HTML5 characteristics.

Planning for automation

Device testing

Does require a physical device.

Testing on the device is best for:

U E i th l t 100 ilUser Experience – the last 100 miles.

Acceptance testing – working with gestures, sensors and hardware.

Soak Testing – how will the device perform over time.

Provisioning – installation and removal.


Cloud providers and plugins for popular automation tools support both functional and performance testing.

No need to run out and buy all platforms.

28/09/2012

14

Automated testing

Browser automationTools can be used for ‘closest‐match’, browser testing on or off the device.

‘Closest‐match’ browser automation possible on some devices ‐ Safari 3 for the iPhone driven on a PC.

User‐Agent header data to drive compatibility testing logic.

Work closely with development team to manage and control Web 2.0 features.

Third party libraries can speed


implementations.

Physical devices and Cloud services can be used to test physical platforms when defects are detected.

Automated testing

Application automationTools can be used in emulated environments and on the device.

Installation of some type of automation controller is required on the device.

Controllers consume resources.

Plug‐in and Cloud support is growing and stable.

Cloud services may not provide object recognition.

Bluetooth/Near Fiel


Additional fixtures may be required for sensor (GPS, Accelerometer) and hardware (camera, battery) automation.

Applications may incorporate browser engine.

Field

28/09/2012

15

Functional details

Create a mobile test strategy document.Tight control of manifest and OS support.

Identify development and testing tool stack.

Explain how you plan to manage fragmentation – OS, device, sensors, UI, marketplace.

Test teams should be working in emulated, physical and cloud based environments.

Test teams should manage insights across releases. HTML5 platform insights

Carrier and Device insights


OS / Platform insights

Security insights

Marketplace

Security insights

o Address security issues programmatically and start now!

Protect your brand.

o Static analysis is a highly effect approach to address coding issues.

o Dynamic analysis, Security Penetration testing, is your last line of defense.

o SD memory should be tested


28/09/2012

16

Security testing

Static Application Security Testing (SAST)

Static analysis tools can address unsafe

Dynamic Application Security Testing

Mobile devices are susceptible toStatic analysis tools can address unsafe practices earlier in the SDLC.

Clang; AppValidator; AnalytiX

SAST should be integrated with build and reporting practices.

Evaluate application permissions.

Don’t build on or deliver on compromised platforms

Mobile devices are susceptible to common browser exploits and more.

Emulators or devices can be used to ensure applications don’t bleed sensitive data on to the device or to the network.

Keep up to date with known exploits and add test cases to qualify the impact.


platforms.

Performance insights

o Static analysis is a effect approach to address performance issues.

Protect your brand.

p

o Dynamic analysis, Security Penetration testing is your last line of defense before delivery.

o SD memory should be tested.


28/09/2012

17

Performance testing

Static Application Performance Testing (SAPT)

Static Analysis tools find issues earlier in

Performance Testing

Load stress and longevity testing areStatic Analysis tools find issues earlier in the SDLC and improve coding practices.

Tools address code that slow application performance.

In a 53K line project we found 328 instances of code that would contribute to performance issues.

SAPT should be integrated with build and

Load, stress and longevity testing are critical to the customer experience.

Load and stress testing are focused on middle and backend service layers.

Longevity testing should look at both backend services and the mobile device.

Performance tools should support bandwidth throttling and common RESTful


reporting. protocols.

Final thoughts

o Cloud based testing services are valuable but…

o Will not achieve expected ROI without proper investigationproper investigation.

o Lead to Scrumfall application lifecycles.

o Can slow delivery times.

o Functional, performance, security and test automation should be considered for each mobile project.

o Simplify the testing space through active


p y g p gmanagement.

o Diagram test environments.

o Support agile approaches for all team members.

28/09/2012

18

Final thoughts

o Develop sticky applications with the 5 star insights.

o Fill out our five start survey and see how your team stacks up!how your team stacks up!


Technology

It's a Phone First! How to Test Your Five-star Mobile Apps