1

TESTING 4.0

THE NEXT GENERATION OF QUALITY

Powered by Crowd & Cloud Technologies

2

Agenda

Introduction

Functional testing

(Example 1)

Usability test

(Example 2)

3

Testbirds

Introduction

UK Country Manager

s.holland@testbirds.com

Vision: “The World’s Leading Company for Innovative

Testing Services”

Founded in 2011

Origins in the enterprise software testing market

Customer satisfaction 9.2 out of 10

Quality assurance and UX / usability testing utilising

crowd and cloud technologies

4

Device Diversity

Do You Have All of the Devices to Test On?

5

Device Diversity

How Do You Validate Localisation?

6

Target Group Relevance

Do You Know Your Target Groups‘ Needs?

7

Current Challenges in the Market

Insufficient QA, Poor Market Analysis and Time Constraints

Organisational blindness

Is everything really fine? Unbiased

users can test differently.

Device diversity

Do we really have all of these

devices to test with?

Target group relevance

Target groups have different

expectations and abilities.

Lack of resources

Time pressure, limited budget and

priority of other projects.

8

CROWD

150,000+ people in 193 countries

380,000+ devices

110,000+ found bugs

Any target audience (60+ criteria / demographics)

Real world testing without artificial environments

Client specific feedback even prior to release

Flexible and efficient ad-hoc availability

CLOUD

Any combination of virtual machine

(Win, Mac, Linux) and emulator (iOS, Android)

Modern technologies exclusively created by

Testbirds

High scalability and individual configuration

Flexible, efficient and dynamic

No maintenance or development costs

Servers only hosted in Germany for maximum

data security

Testing 4.0

Powered by Crowd & Cloud Technologies

9

Nest Demo

10

Scenarios for Crowdtesting in the Automotive Industry

Multiple Possibilities for Boosting Quality

Brand portal

Car configurator

Content

Connected car app

(Example 2)

Connected car

web portal (Example 1)

Used car portal

Other apps

…

11

User-centric Testing in the Automotive Industry

Assuring Functionality and Improving Usability in a Connected Cars Project

Objectives

Functional test on various

devices/browser combinations before

Go-Live

Localization check across different global

markets

Generation of user input for requirement

definition phase

Benefits

Highly efficient coverage of device/browser

matrix

Low effort for project integration / internal

resources

Comprehensive view of customer needs

Higher customer satisfaction

Setup

Three semi-structured functional tests

runs before release (Test Pre-Live,

Retest Pre-Live, Test Live)

Integrated localization tests on up to 70

country – language combinations

Functionality checked on up to 80 device

/ browser combinations

UAT outside the development/rollout

process with 80 vehicle owners (model

year > 2009) across all major models and

important Go-Live markets (DE, FR, UK,

NL, US, IT)

Results

List of functional issues after each test that

were fixed before the next test run

Bug-Retesting in test run 2 and 3

List of spelling and translation issues from

up 70 country – language combinations

Feedback from actual vehicle owners

regarding the usability and content of the

web portal

Summary with all the feedback and

information from the testers in a compact

and clear final report

Customer

OEM

Headquarter

Location

Germany

Industry

Automotive

Product

Connected

car web

portal

12

User-centric Testing in the Automotive Industry

Assuring Functionality and Improving Usability in a Connected Cars Project

Functionality

Example

1

Usability

Product: Connected car service portal

Three semi-structured tests runs before

each release (Test Pre-Live, Retest Pre-

Live, Test Live)

Functionality checked on up to 80 device /

browser combinations

Integrated localisation tests on up to 70

country – language combinations

Product: Connected car app

Usability study outside the

development/rollout process with 81

vehicle owners, across all major models

and important Go-Live markets (DE, FR,

UK, NL, US, IT)

Exploratory (use cases)

Structured (use cases)

Localisation

End-to-End

Beta / Live

Example

1

13

Process Overview

A

Preparation

B

Testing phase

C

Reporting

D Follow-ups

1 – 3 days

Phase

Time-

frame 1 – 3 days 1 day ongoing

14

Agenda

Introduction

Functional testing

(Example 1)

Usability test

(Example 2)

15

A - PreparationFunctionality

Topic Contact person

Test environment On which environment will be tested?

How is it accessible?

XY

Test date When will be tested?

What results will be needed by when?

XY

Test content What will be tested?

Which test cases / use cases?

Is there a requirements document?

In scope / out of scope

XY

Country/language combinations Which native languages and which countries? XY

Devices and browsers Desktop / Smartphones / Tablets

+ Browser (versions)

XY

Testers Which tester demographics? XY

VINs Which VINs to use? Pre-live, Live etc. XY

Release plan Are there ongoing deployments? XY

…

16

B Test Case Test

Testbirds Platform

Test Cases implemented directly on the platform

Results documented per device

Several steps per Test Case – if one step fails the whole test case is highlighted and a bug is recorded

Device the Test Cases were

performed on

Test Case 1 failed on Device 1

Functionality

17

B Test Case Test

Testbirds Platform

Test case steps:

• Pass = green

• Fail = red > bug is submitted

• N/A = yellow > not possible to perform due to previous bug

Functionality

Same steps all worked on

different device

Step 2 failed – bug was submitted

Step 3 could not be performed because of

step 2

18

B Test Case Test

How Bugs Are Displayed

Overview of all bugs submitted by testers

Possibility to export to xls. or bug tracking tools

Functionality

Export function

19

B Test Case Test

Bug Example

Functionality

Bug

description

Visual

documentation

General information

(device, category, severity)

Title

20

C Results

Testbirds Platform

Detailed list with all devices tested

• 49 devices (device type / OS /

browser) – 28 different devices

Device #

Windows 7 (Chrome) 4

Windows 7 (Firefox 28) 3

Windows 7 (Firefox 29) 1

Windows 7 (Firefox 32) 1

Windows 7 (Firefox 34) 1

Windows 7 (Firefox 35) 3

Windows 7 (IE 11) 3

Windows 7 (IE 9) 1

Windows 8 (Firefox 28) 1

Windows 8 (IE 11) 1

Windows 8.1 (Chrome) 5

Windows 8.1 (Firefox 35) 2

Windows 8.1 (IE 11) 1

Windows 8.1 Touchscreen (IE 11) 2

Mac OS 10.10 (Firefox 35) 1

Mac OS 10.10 (Retina) (Chrome) 1

Mac OS 10.10 (Retina) (Safari) 2

Mac OS 10.10 (Safari) 1

Mac OS 10.10.1 (Chrome) 1

Mac OS 10.10.1 (Safari) 1

Mac OS 10.6 (Safari) 1

Mac OS 10.7 (Chrome) 1

Mac OS 10.9 (Chrome) 1

Device #

Tablet

Apple iPad 2, iOS 7.0.3 1

Apple iPad 4, iOS 8.1.0 1

Apple iPad Air (iPad 5), iOS 8.1.2 1

Smartphone

Apple iPhone 5S, iOS 8.1.0 1

Samsung Galaxy S i9000, Android 2.3.3 1

Total 49

Functionality

21

C Results

Technical Issues – Examples of Defects

Functionality

Defect ID Defect Title

Spelling / Grammar mistake

ID-29354 (FR/FR) Service overview - Bad translation of "Board book" tab

Translation issue

ID-29331 (FR/FR) My vehicles - Status message not translated

ID-29344 (ES/ES) Service overview - Service offers in English instead of Spanish

ID-29352 (NL/NL) Service overview - Service offers in English not Dutch

ID-29381 (FR/FR) Service overview - Service offers in English instead of French

ID-30433 (GB/EN) Service overview - Dealer Search in German (was already closed but apparently is not fixed ID 29179)

ID-30557 (Belgium/Dutch) Service overview - tabs in English instead of Dutch

Other

ID-29167 Service overview - Google link in Service Offers available

ID-29170 (US/EN) Dealer Search - Pop up not translated to English

ID-29304 Dealer Search - Encoding Problem in Print Version

ID-29334 (FR/FR) Dealer Search - GUI issue in result details of dealer search

ID-29422 (Canada/EN) Service overview - Dealer Search in English

ID-29660 Dealer search - Position and layout of error message not aligned

ID-29683 (NL/Dutch) Dealer Search - GUI issue in result details of dealer search

ID-30434 Service Overview - Service offers: Price in Euro's whilst text in English

22

Agenda

Introduction

Functional testing

(Example 1)

Usability test

(Example 2)

23

A Usability Test

Initial Situation

This usability study focused on the current production version of the client’s app / service

Content: 12 questions to establish the users’ opinion of the app, what he / she likes, what he / she misses

Goal: Gauge user acceptance and optimise the app before local market launch

Participants: 81 car-owning testers in 14 locations

Usability

Country # of participants Country # of participants

Austria 7 Netherlands 3

Belgium 4 Portugal 4

Canada 5 Romania 2

France 3 Singapore 2

Germany 28 Spain 5

Hungary 3 Sweden 4

Mauritius 1 United Kingdom 7

24

B Usability TestUsability

Use case

documentation

Visual

documentation

Export

Example

25

C Usability Test

Results

Goal: Identifying potential for improvement of the app / needs of the customer

Setup:

• 81 customers from 14 countries

• 24 hours

• 12 questions

Results:

• Overall Satisfaction

• Registration Process

• Visual Appearance

• Navigation

• Loading Times

• Recommendation

• Content

Usability

3,5

5,1

4,5

4,3

3,3

3,6

4,4

Is there anything you miss on the current app?

• “Information about my last service check. When should I change

the oil, brakes and other parts.”

• “Information about my own car. Next Service, last Bills, ... There

are only marketing Info about selling new products or add-ons

what I could buy.”

• “Guidance/Ideas/Services that help me to either update my

vehicle, unlock new features etc.”

• “The portal gives a great overview about my vehicle, but no real

function to configure/improve my car.”

• “I miss recall information.”

• “Only the basic contact details are show: No picture of the dealer's

building or some additional services or special offers (e.g. free

winter check up from 1.10. - 15.12.) are shown here. Also a rating

for the dealer could be shown like XXX does.”

26

B Usability Test

Results - Report

Usability

The results are

delivered in a

detailed final

report

27

Discussions

Any remarks / questions /

comments?

Looking for a PoC!