Session 1 - Customer Experience Threshold KPI Benchmarking

Customer Experience Threshold: Benchmarking

mobile network operators – who wins?

ITU Workshop on Quality of Service and Quality of Experience of Multimedia Applications and Services

Haarlem, 9.5.2016 – Arnold van Holten

©Omnitele Ltd. 2016 2

Introduction of an on Customer

Experience focussed KPI

% Above Average

Customer Experience


ag

end

a1. Introduction

2. Network Performance vs. Customer Experience

3. Customer Experience Threshold

>


Our Company

Pioneeringfounded in 1988 to design and rollout world’s first GSM network,

1000+ projects in 80+ countries since then

Independentowned by Finnish telecom investors, independent

of all operator groups and equipment vendors

Consultingconsulting and professional services for telecom service

providers and regulators

for mobile industr y since 1988


Our Services

Technology

Strategy

Design and

Optimisation

Audit and

Benchmark

Predictive

Analytics

maximised customer experience

minimised network cost


Our Experience

1000+ Projects in 80+ Countries

Iceland Finland Isle Of Man

QatarJordan

United Arab Emirates ZimbabweRwanda

Kuwait Greenland

Haiti Sweden

Faroe IslandsDRC Switzerland

Estonia SurinameNetherlands

SloveniaBahrain

Angola Jamaica Jersey

network infrastructure swapcompetitive network benchmark

HSPA optimisation network design audit and optimisationfrequency refarming

performance management network sharing POC

capacity utilisation audit RAN modernisation support end-user QOS benchmark

customer experience optimisationDVB-T2 network design LTE700 network design

LTE vendor selection PM vendor selection and sourcing

GSM-R planningradio network capacity expansion strategy

HSPA network nominal plan TETRA indoor design post-hurricane network audit

frequency management strategy LTE800 frequency licence verification


ag

end

a1. Introduction


3. Customer Experience Threshold

>


Network Performance vs. Customer Experience

…pre-HSPA networks

General user experience depends on

two basic services

Voice Services Data Services

MOS, Success Rates, Drop Rates,

Setup Time

Throughput

Generally available bitrates are not sufficient for data services. As

throughput is the bottleneck, it becomes the measure to define the QoS


Customer Experience Challenge

0

2

4

6

8

10

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Qo

S: W

WW

pag

e w

aitin

g t

ime [

s]

NW Performance: Bitrate [Mbit/s]

1000KB WWW Page Waiting Time vs. Bitrate

HSPA LTE

Bitrate is no longer the ultimate mobile network quality

indicator


Vodafone higher bitrate,

but KPN faster YouTube

0

10

20

30

40

T-Mobile Vodafone KPN

bitrate [Mbit/s]

big difference in NW performance…

0

0.5

1

1.5

2

T-Mobile Vodafone KPN

buffering time [s]

…thin margins in Quality of Service

Insta

Customer Experience Challenge


Customer Experience Benchmark

”Be the Customer”


ag

end

a1. Introduction


3. Customer Experience Threshold>


Benchmarking MNOs – who wins?

In mature networks average results show very similar QoS performance for OTT services

KPN 1.1

VODAFONE 1.1

T-MOBILE 1.0

TELE2 2.0

0 1 2 3 4 5

TWITTER SESSION WAITING TIME (s)

Netherlands

KPN 2.8

VODAFONE 3.4

T-MOBILE 3.3

TELE2 3.5

0 1 2 3 4 5

YOUTUBE BUFFERING TIME (s)

Netherlands

Identified issues:1. How to assess the competitive advantage/disadvantage between operators

By default operators give good QoS performance - differentiate on how often a substandard customer experience is

provided

2. What is the real QoE experienced by mobile subscribers?Defining good/bad performance falls into QoE domain, which is not straightforward to measure and enviromentally

dependent


Customer Experience Threshold

KPN 79.8%

VODAFONE 78.6%

T-MOBILE 84.6%

TELE2 63.3%

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

TWITTER % ABOVE AVERAGE CUSTOMER EXPERIENCE

Netherlands

The Customer Experience Threshold is defined as the average figure from the samples obtained from all operators

– it is an indicator of the average performance expected by customers in that country

’% Above Average Customer Experience’ depicts the amount of samples above/below the Customer Experience

Threshold – it is an indicator of how often substandard experience is delivered

Insta


% Above Average Customer Experience All application tests are analysed in terms of following key performance indicators (KPI):

Accessibility (successful tests / all tests),

Usability (time-to-content)

% Above Average Customer Experience

The “% Above Average Customer Experience” combines Accessibility and Usability into a single KPI. It illustrates the share of each

operator’s test samples being faster than the average time-to-content of all samples.

The average time-to-content is calculated over all successful samples of all operators. Failed test samples are treated as slower than

the average.

The objective is to show how consistent an operator is. That is, how often an operator delivers quality better than the typical quality

in the market (country average result).

The “% Above Average Customer Experience” KPI is binomially distributed which allows for straightforward statistical analysis of

results and comparison of results between operators.


Drive test and Hotspot measurements

The benchmark was conducted with Omnitele’s internally standardised be-the-

customer methodology. Rather than only evaluating the technical network

performance (data rate, signal strength), the analysis focuses on the usability of mobile

services and smartphone applications.

In this project the scope consisted of Mobile Voice, Web browsing, Twitter, Facebook

and YouTube customer experience measurements.

Each network was measured in the very same locations simultaneously with identical

measurement equipment.

Measurements have been executed where (location), when (time) and how (services,

devices) real subscribers typically use mobile services

Execution: February 2016

Drive tests

70 cities (> 40 K inhabitants)

All A-roads

Most important N-roads

Example CE Benchmark in the Netherlands


Nemo Invex

Nemo

Walker Air

Samsung

Galaxy

Note 4

Samsung

Galaxy

S6 Edge+

Example CE Benchmark –

Measurement Equipment


Results – Summary Data

KPN 99.3%

VODAFONE 98.4%

T-MOBILE 99.5%

TELE2 91.3%

50%

60%

70%

80%

90%

100%

DATA TEST CASE SUCCESS RATE

Netherlands

50%

60%

70%

80%

90%

100%

WWW Facebook Twitter YouTube

% Above Average Customer Experience, All Drive Tests

KPN 2.8

VODAFONE 3.4

T-MOBILE 3.3

TELE2 3.5

0 0.5 1 1.5 2 2.5 3 3.5 4

YOUTUBE BUFFERING TIME (s)

Netherlands


Results – Summary Telephony

KPN

99.6

Vodafone

99.0

T-Mobile

99.7

Tele2

99.695

96

97

98

99

100

CALL SUCCESS RATE (%)

Netherlands

KPN

51.8

Vodafone

19.9

T-Mobile

95.1

Tele2

69.80

20

40

60

80

100

CALL SETUP TIME < 6.9s (%)

Netherlands

KPN 7.0

Vodafone 7.7

T-Mobile 5.2

Tele2 7.6

0 1 2 3 4 5 6 7 8

CALL SETUP TIME (s)

Netherlands

KPN

3.8

Vodafone

3.7

T-Mobile

4.0

Tele2

3.30

1

2

3

4

5

Voice Audio Quality MOS

Netherlands

KPN

71.8

Vodafone

62.6

T-Mobile

90.4

Tele2

0.00

20

40

60

80

100

Voice Audio Quality MOS > 3.7 (%)

Netherlands


Web Portal Presentation




w w w . o m n i t e l e . c o m



w w w . o m n i t e l e . c o m

http://www.omnitele.com/




Documents

Session 1 - Customer Experience Threshold KPI Benchmarking