The Best of Both Worlds - Combining Performance and Functional Mobile App Testing

HouseKeeping rules

Presenters

Speaker

Henrik Rexed

Performance Engineer

Neotys

Speaker

Lingkai Shao

Marketing Analyst

Bitbar

Speaker

Dave Horner

Director of Sales

Bitbar

Neotys Continuous Performance Testing Solution

System Testing

Integrated Component Testing/System-wide Testing

ReleaseComponent Testing

Continuous Integration & Component Testing

Proactive Production Monitoring End-User

Experience

“Continuously ship quality apps with a high velocity delivery pipeline”

Bitbar - Mobile DevOps

Dev Ops

Continuous

Release &

Deployment

Collaborative

Development

Continuous

TestingContinuous

Monitoring

Continuous

Integration

Continuous

Feedback

DevOps is no longer a

unicorn’s discipline.TestDev Ops

Shift Left Shift Right

DevOps

According to Adobe 2016 Online Shopping Data, mobile drove 55% of visits to retail websites on Black Friday (45% coming from smartphones, 10% from tablets) and accounted for 36% of sales (25% from smartphones, 11% from tablets) *

Why Functional and Performance Testing for Holiday Season

*=http://news.adobe.com/press-release/marketing-cloud/media-alert-adobe-data-shows-black-friday-breaks-online-sales-record-3

Performance matters

53% of visits to mobile sites are

abandoned after 3 secondsaccording to research from Google

According to Google doubleClick faster websites has 70% longer average session lengths, 35% lower bounce rates and 25% higher ad

viewability than their slower counterparts.

Etam reduces its average page load time from 1.2s to 500msand increased conversions by 20%

Walmart saw up to a 2% increase in conversions for every 1 second of improvement in load time.

Every 100ms improvement also resulted in up to a 1% increase in revenue.

Top Failures During Peak Days

Concrete study

When Revenue is Generated

● Preparation: typically 10 weeks ahead of peak days● Functional test suite including

Stage One: Functional & Compatibility

Do you know the behavior of your app/web users?

Defining the right user flows

Stage One: Functional & Compatibility

Stage One: Functional & Compatibility

Stage Two: Performance & Load

Standard performance testing workflow

Build the performance strategy

Create the testing scenarios

Prepare

Execute

AnalyseTuning

Performance design is the key

Mobile Application Architecture Landscape

Native app

Hybrid app

Web app

Web appWeb Model

Mobile Model

Rest Interface

Business logic

Data Access logic

XML

Legacy

Relational

Presentation Business Data

Third party

Mobile Performance Testing Touchpoints

BACK-END

EXECUTION PLATFORM

RIA/FRAMEWORKS

DATABASE

NETWORK

3G, 4G / LTE, Wi-Fi

VIDEO

HTTP Adaptive StreamingThroughput

LatencyPacket loss

SERVERINFRASTRUCTURE

OS

CLIENTAPP

DEVICE

iPhone 4, 5, 6SamsungBlackberry…(7,000 different device types are used to access Facebook)

Screen SizeCPUMemory

Native Browser-BasedHybrid

Impact of Mobile Network

The most important network constraints are:• Latency

• Bandwidth

At a certain point, increasing the bandwidth will only have a minor effect.

Latency & Bandwidth . . .

Packet Loss Consequences

• Packet Retransmission

• End user experience degraded because packet loss increases jitter

• Errors could occur if packet loss is in pure data transmission

Packet Loss is Inevitable with Mobile

• Mobility: Quickly switching to the next cell tower may result in temporary signal loss due to congestion on the current tower

Mobile Protocols Try to Reduce Packet Loss Rate

Packet Loss . . .

Lots of users

• Mobile applications often have large user bases

• High growth in popularity can create the need for highly scalable load generation infrastructure

Geographical distributed load

• Users are accessing apps from locations around the globe

• Distance and location can greatly affect performance

Mix of Web & Mobile Users

• Many mobile apps have a corresponding web app with a shared backend

• High load on the web app can affect performance on mobile

High volume, Geographically Distributed

Don’t forget the User Experience…..

What are is really being measured?

… Applying the Bitbar Testing Integration…

00:00:00 00:00:30 00:01:00 00:01:30

Load

NeoLoadApplication

Server

App Error Rate

Virtual Users Load

UserExperience

RUM (Passive) vs Synthetic (Active)

Stage Three: Proactive Production Monitoring

Why Synthetic Monitoring?

● Alarm and Alert● Better MTTR! (Mean Time To Resolution)● Find and fix issues before impact to users● Baselining and Benchmarking (Pre-production Staging & Post Production)● Prepare for peak traffic and new markets● Monitor Complex Transactions● Measure and Adhere to SLAs● Hold 3rd Party Vendors Accountable● Test from end user perspective● Competitive Benchmarking

Stage Three: Proactive Production Monitoring

Understand Performance of Complex User Transactions and Hold 3rd Party Services Accountable (SLAs)

Benefits of Synthetic Monitoring

Faster Mean Time to Repair… Identify Issues Before Your Users Do

Benefits of Synthetic Monitoring

Understand Performance In New Markets, Optimize Content Delivery, Benchmark Pre & Post Change

Benefits of Synthetic Monitoring

Competitive Benchmarking:

Benefits of Synthetic Monitoring

Purpose Built Synthetic Monitoring Solution

Global Points of Presence

Use Your Existing Functional Test Scripts For Monitoring

Solution:

Stage Three: Proactive Production Monitoring

DEMO

Demonstration

Questions?

AQ