Benchmarking Service Sum [Compatibility Mode]

Benchmarking Service Summary

2 © Nokia Siemens Networks Benchmarking Service Summary

Contents

Benchmark Service Overview

Benchmarking Principles

Reporting Examples


Contents



Reporting Examples


Network Optimization and Assurance Services

Perf

orm

an

ce C

on

su

ltin

g

Develop

Assure

Optimize

Analyze

Pro

ject

Man

ag

em

en

t

Network performance

assessment & analysis

Radio

interface

Benchmarking

Transport &

backbone

Core network

Fixed / mobile

Access network

Fixed / mobile

Network Optimization and

Assurance Services

Capability extension

managementPerformance

New feature

introduction

Network

evolution

The offered optimization services will be customized to meet your specific needs.


What is Benchmarking?

• Benchmarking is drive & static testing of voice and data services using a standardized test set-up and evaluation method

• 100+ operators have been measured in 30+ countries

• OSS KPI related “benchmarking” is rather covered under Assessment


100+ operators measured in 30+ countries

*) other technologies are: GSM/GPRS, EDGE, CDMA1X, TDMA

EraCentertelPolkomtel

P4

PL

OITIMClaroTIM TDMAClaro TDMAVivo CDMATelemig TDMA

BR

ConecelConecel TDMABellSouth CDMABellSouth TDMATelcel CDMA

EC

MobilinkUfone Warid

Telenor

PK

AWSCingularT-MobileSprint CDMAVerizon CDMA

USA

E-PlusO2T-MobileD2-Vodafone

D

SFROrangeBouygues

F

AirtelAmenaMovistar

E

OrangeSunriseSwisscom

CH

TIMWindVodafone

I

BaseMobistarProximus

B

OptusTelstraVodafone

AUS

IndosatExcelcomTelkomsel

RI

GlobeSmartDigitel

RP

CMCCUnicom GSMUnicom CDMA

CHN

CSLOrangeNew WorldSmartone

HK

FETViboTCC

Chunghwa

TWN

A1ONEMaxmobil

TeleRingThree-UMTS

A

PannonT-MobileVodafone

H

O2OrangeT-Mobile

Vodafone‘3‘ - UMTS

GB KPNOrangeBenVodafoneTelfort

NL TDCTeliaSonofon

DK TeliaCOMVIQVodafone

‘3’ - UMTS

S

DNASoneraRadiolinja

FIN

other technologies *)

UMTS & othertechnologies

ConnexOrange

ROGlobulM-tel

BG

SunComCingular Centennial CDMA

Movistar CDMASprint CDMAVerizon CDMA

PR

AstelitKyiv StarUMC

URS (Moby)

UA

AirtelRelianceHutch

IdeaBSNL

IND

VodafoneMobinil

ET


Value proposition – Benchmarking

• Benchmarking is the least intrusive service to get insight into the customer’s and his competitors’ network and service performance.

• Little or even no involvement from the operator is needed.

• It is relatively cost limited - depending on scope costs are: 10 – 19k€ for a normal campaign (one metropolitan area – approximately 500km drive testing incl. static tests).

• Service lead time is 2 weeks.

• It is recommended as a door opener service or a service which is done in regular intervals to establish a continuous communication link to your customer.

• Continuous / volume benchmarking is usually delivered through partner companies, consulting companies takes care of the data analysis, reporting and report presentation.


Value proposition – Benchmarking

• National and international best-in-class comparison allows to position yourself and support strategic decisions (e.g. marketing or network evolution related)

• Rich portfolio of services tested (Coverage Quality, Voice, FTP, Email, HTTP, WAP, SMS, MMS)

• Complete spectrum of technologies (GSM, GPRS, EDGE, TDMA, CDMA1X, EV-DO *), UMTS, HSDPA)

• Clear standards for measurements and high-end tools ensure true comparability of results

• Monitor global and local trends over time

• Unbiased view on service performance

• Benefit from global network of expertise (>1400 Nokia Siemens Networks performance experts around the world)

*) EV-DO requires a slightly longer lead time for service delivery (upgrade required).


Contents



Reporting Examples


Drive Test• >500 voice calls per operator.• >200 data tests

• FTP up/download• HTTP download• Email send/receive

• Test route 500 to 700 km

Static Tests• Tests done at 4 static locations• 200 SMS • 100 MMS• 240 WAP tests • 480 Ping jobs• up to 70 FTP up/downloads

Test Overview – Standard Campaign


Urban

Special

Sub

Sub

Sub Urban

Rural

Rural

Rural

Urban

Special

Sub

Sub

Sub Urban

Rural

Rural

Rural

The measurements in each survey area have the following characteristics:

• Drive test routes are defined such that a defined proportion of test time is carried out in specific area types (urban, rural, motorway) to allow fair comparison between survey areas.

• The overall length of all drive test routes in one survey area is at minimum 500km.

• A survey area is defined as the catchment area of a major city. The exact measurement plan will be discussed prior to the campaign start.

Route planning


Drive Test - Overview

• Voice and Data services are measured simultaneously during drive test

Data Service

Start of Drive TestVoice Call

Idle30 sec Pause

Data Session

Voice Service

End of Drive Test

Voice Session

Data Session

Data Session

30 sec Pause

Voice Call

Voice Call

Idle


• Four locations are selected for static measurement

• The exact locations are agreed in advance with the customer.

• SMS and MMS tests are done

mobile to mobile with both

phones residing at the same

location.

• For WAP Tests only one mobile is necessary

Static Measurement - Overview


From End-user perception to Engineering …

VCQI / DCQI

Service Performance

Performance Indicator

Charts, plots & tables

en

gin

ee

rin

g…

… t

op

ma

na

ge

me

nt

en

gin

ee

rin

g…

en

d-u

se

r p

erc

ep

tio

n

60% FTP success DL [%]

40% FTP throughput DL [kbit/s]

e.g.

FTP DL Service (20%)

VCQI / DCQI


Voice Connection Quality Index (VCQI)

Call Setup Time (CST)

10% Call Setup Time [%]

Call Setup Time (10%)

Call Setup Success Rate (20%)

Call Completion Rate (30%)

Mean Opinion Score (40%)

Call Setup Success Rate

(CSSR)

20% Call Setup Success

Rate [%]

Mean Opinion Score (MOS)

40% MMS success [%]

Call Completion Rate (CCR)

30% Call Completion Rate [%]


Data Connection Quality Index (DCQI)

GPRS Attach (5%):

50% GPRS attach success [%]

50% GPRS attach delay [s] PDP Context Activation (5%):

50% PDP context success [%]

50% PDP context delay [s]

FTP UL Service (5%):

60% FTP success UL [%]

40% FTP throughput UL [kbit/s]

FTP DL Service (20%):

60% FTP success DL [%]

40% FTP throughput DL [kbit/s]

HTTP Browser (40%):

60% HTTP success [%]

40% HTTP download time [s]

Email Service (10%):

80% Email success [%]

20% Email Transfer Time [s]

SMS Service (10%):

70% SMS success [%]

30% SMS delivery time [s]

GPRS Attach (5%)

PDP Context Activation (5%)

FTP UL Service (5%)

FTP DL Service (20%)

HTTP Browser (40%)

Email Service (10%)

SMS Service (10%)

MMS Service (5%):

80% MMS success [%]

20% MMS delivery time [s]

MMS Service (5%)


FTP DLThroughput

[kbit/s]

15.2

28.8

1

3

Bar Charts

The “Max” value shows the best result in all tested networks. In this case the best performing

operator had a FTP DL Throughput of 28.8kbit/s

“low 20% - percentile” shows that 20% of all operators are on

this level or lower.

The “Min” value shows which network showed the lowest

performance in this category. Here the lowest performing

operator had a FTP DL Throughput of 15.2kbit/s

“80% - percentile” draws the borderline to the global top 20%

performers.

50% of all operators are above this “Median” value and 50% of

all operators are below this value.

Here you find individual operators’ results.

2

4


Contents



Reporting Examples


Introduction

• The following slides shall illustrate the look & feel of a standard benchmarking report.

• The shown slides are only an extract of a full report.

• Reports can be customized according to your wishes.


Service Performance in XX City at a Glance

top 20% performers internationally

low 20% performers internationally

international average

VCQI

1

2

3

4

5

DCQI

1

2

3 4

5

1 OP1

2 OP2

3 OP3

4 OP45 OP5


Executive summary - XX City

Voice:

With voice service OP1 is behind local competition. When roaming to OP2, OP1 shows better performance than in its own network.

The OP1 network shows low Speech Quality (MOS) and Call Setup Success, but the Call Setup Time is the shortest measured amongst 3G operators in XX City.

Data:

In its own network OP1 shows an overall excellent performance for data services.

When roaming to OP2 the overall performance of data services is the worst in XX City and below international average. This is mainly due to the bad stability of FTP, HTTP and Email services during roaming.

The drive test indicates a low performance of the GPRS attach in the OP1 network.


Some facts – Benchmarking Project in XX City

Benchmarking of the UMTS networks of: OP1, OP2, OP3 and OP4

Tests done: voice service, data services (HTTP, FTP, Email, MMS, SMS, WAP, Ping)

Test phone: all tests were executed with N6680

Measurement period xxth – xxth January 2010

Per operator approximately:

• 500 two-minute voice calls evaluated

• 200 FTP up- and downloads, 200 Emails send and received, 200 HTTP requests, 200 SMS, 80 MMS, 80 WAP requests, 200 Ping tests

• 500 km of routes driven

Comparison to xx operators in yy countries


OP1 - CPICH RSCP, Drops & Fails, XX City


Voice Service Performance - XX City= Call Setup Time

= Call Setup Success Rate

= Call Completion Rate

= Mean Opinion Score

CST

CSSR

CCR

MOS

VCQI

1

2

3

4

5

CST

6.0

5.0

4.2

2.7

2.3

1

2

3

4

5

CSSR

99.6%

98.6%

96.4%

90.3%

100%

1

23

4

5

CCR

99.4%

98.7%

97.3%

91.0%

100%

1 2

3

4

5

MOS

3.78

3.72

3.70

3.66

3.52

1

2

3

4

5

1 OP1

2 OP2

3 OP3

4 OP45 OP5


Call Setup & Completion, XX City

With Accessibility of voice calls OP1 is behind local competition and below international standards in its own network.

With respect to Retainability OP1‘s own network shows comparable results to the national market.

90

91

92

93

94

95

96

97

98

99

100

90 92 94 96 98 100

Call Setup Success Rate [%]

Call C

om

ple

tio

n R

ate

[%

]

Other Networks OP1

OP2 OP3

OP4 OP5

int. Median Call Completion Rate [%] int. Median Call Setup Success Rate [s]


Call Setup Time

OP1 shows the fastest call setup time in their own network.

Two lower peaks can be seen for OP1 roaming to OP2.

Cumulative Distribution Function CST

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 2000 4000 6000 8000 10000 12000 14000

CST [ms]

Perc

en

tag

e

OP1 OP2 OP3 OP4 OP5 International Average

Probability Distribution Function CST

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

0 2000 4000 6000 8000 10000 12000 14000

CST [ms]

Perc

en

tag

e

OP1 OP2 OP3 OP4 OP5 International Average


Speech quality (MOS) made “tangible”

Click icons to hear what your subscribers experience!

The man says: "Screen the porch with woven straw mats“

The woman replies: "The lamp shone with a steady green flame."

1,0

1,1

1,2

1,3

1,4

1,5

1,6

1,7

1,8

1,9

2,0

2,1

2,2

2,3

2,4

2,5

2,6

2,7

2,8

2,9

3,0

3,1

3,2

3,3

3,4

3,5

3,6

3,7

3,8

3,9

4,0

4,1

4,2

4,3

4,4

>4

,4

MOS score

Bad Poor Fair Good Excellent


Voice Quality, XX City

Downlink Speech Quality (DL MOS) and Uplink Speech Quality (UL MOS) of OP1 is somewhat behind local competition.

OP2 shows the best DL MOS, OP4 has the best UL MOS.

3.3

3.35

3.4

3.45

3.5

3.55

3.6

3.65

3.7

3.75

3.8

3.3 3.35 3.4 3.45 3.5 3.55 3.6 3.65 3.7 3.75 3.8

UL MOS [-]

DL

MO

S [-]

Other Networks OP1 OP2 OP3 OP4 OP5


Data Service Performance - XX City

DCQI

1

2

3 4

5

HTTP

1

2

3

4

5

FTP DL

1

2

3

4

5

FTP UL

1

2

3

4

5

SMS

1

2

3

4 5

Email

1

2

3

4

5

GPRS Attach

1

2

3

4

5

PDP Context

1

2

3

4

5

1 OP1 2 OP2 3 OP3 4 OP4 5 OP5


FTP Download, XX City 90% Confidence Interval *.)

OP1 shows excellent service quality with own network.

When roaming to OP2, OP1 shows relatively weak results with respect to FTP Download success.

With average and peak FTP Throughput OP1 and OP2 are equally strong and well above international standards.

*.) more details about the Confidence Interval can be found in the back-up material

FTP DL Success Rate

100.0% 96.5% 100.0% 96.8% 98.4%90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

99.0%

100.0%

OP1 OP2 OP3 OP4 OP5

Su

cc

ess

Rate

[%

]

Test Success Rate [%] int. Median Test Success Rate [%]

+

-

FTP DL Throughput

255.9 253.7 257.5

211.0 215.7

294.1 299.3 301.8

274.8

298.5

0

50

100

150

200

250

300

350

OP1 OP2 OP3 OP4 OP5

Th

rou

gh

pu

t [k

bit/s

]

Avg Throughput [kbit/s] int. Median Throughput [kbit/s] Max Throughput [kbit/s]

+

-


Email Send, XX City

90% confidence interval

Email sending from OP1’s own network and from OP3 appears to be most reliable.

Apart from OP3 all operators showed pretty fast email sending times.

In average the transfer time in OP1‘s own network is about 0.5s faster than when roaming to OP2.

Email Send Success Rate

100.0% 97.0% 99.1% 98.1% 100.0%90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

99.0%

100.0%

OP1 OP2 OP3 OP4 OP5

Su

cc

ess

Rate

[%

]


+

-

Email Send Transfer Time

2.1

2.72.6

1.8

3.1

1.6

1.0

1.31.2

1.0

0

0.5

1

1.5

2

2.5

3

3.5

OP1 OP2 OP3 OP4 OP5

Tra

ns

fer

Tim

e [s

]

Avg Transfer Time [s] int Median Transfer Time [s] Min Transfer Time [s]

-

+


HTTP Browser, XX City


For OP1 roaming to OP2 also HTTP Browsing was lower in stability than for the other networks.

Apart from OP3 web download times were comparable and faster than international average with OP1(OP1) being the best.

None of the operators is using Performance Enhancement Protocol (PEP)

HTTP Success Rate

100.0% 98.0% 99.5% 100.0% 100.0%90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

99.0%

100.0%

OP1 OP2 OP3 OP4 OP5

Su

cc

ess

Rate

[%

]


+

-

HTTP Download Time

7.3

11.611.0

10.0

16.1

5.2

7.1 6.9

6.0

9.4

0

2

4

6

8

10

12

14

16

18

OP1 OP2 OP3 OP4 OP5

Do

wn

loa

d T

ime

[s]

Avg Download Time [s] int Median Download Time [s] Min Download Time [s]

-

+


SMS End-To-End, XX City

OP1 provides the fastest SMS service

OP2 lost one out of 147 SMS

99

99.1

99.2

99.3

99.4

99.5

99.6

99.7

99.8

99.9

100

02468101214

Delivery Duration [s]

Su

cc

ess

Ra

te [

%]

Other Networks OP1 OP2

OP3 OP4 OP5

int. Median Success Rate int. Median Delivery Duration


WAP, XX City

OP1 provides best WAP service with respect to download times and success rates of WAP homepage.

Operator URL Page Size [bytes] Num of Tests

Download

Success Rate [%]

Avg Download

Time [s]

OP1 http://wap.op1.com 18375 80 100.0% 11.4

OP2 http://wap.op2.com 11450 60 88.3% 22.8OP3 http://wap.op3.com 14490 80 77.5% 16.9

OP4 http://wap.op4.com 24242 80 68.8% 14.7


GPRS Attach, XX City


GPRS Attach success rate for OP1 is behind local competition

GPRS Attach times were by far the shortest for OP4.

With 2,6 seconds OP1’s Attach Time is fairly long.

GPRS Attach Success Rate

98.4% 100.0% 99.6% 100.0% 100.0%90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

99.0%

100.0%

OP1 OP2 OP3 OP4 OP5

Su

cc

ess

Rate

[%

]


+

-

GPRS Attach Time

2608

2300

1997

1277

1709

1736 1730

1460

670

1450

0

500

1000

1500

2000

2500

3000

OP1 OP2 OP3 OP4 OP5

Att

ach

Tim

e [m

s]

Avg Attach Time [ms] int Median Attach Time [ms] Min Attach Time [ms]

-

+


PDP Context Activation, XX City


Excellent PDP Activation success rate for OP1 in own network.

OP4 shows significant problems with PDP context activation.

OP1 is on average level (national and international) for the activation duration.

PDP Success Rate

100.0% 99.5% 100.0% 94.6% 100.0%90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

99.0%

100.0%

OP1 OP2 OP3 OP4 OP5

Su

cc

es

s R

ate

[%

]


+

-

PDP Activation Time

18051895

1437

3003

2043

1143

1330

143

900

720

0

500

1000

1500

2000

2500

3000

3500

OP1 OP2 OP3 OP4 OP5

Ac

tiv

atio

n T

ime

[m

s]

Avg Activation Time [ms] int Median Activation Time [ms] Min Activation Time [ms]

-

+

Thank you

Documents

Benchmarking Service Sum [Compatibility Mode]