CRA NETWORK AUDIT 2020 QOS MEASUREMENTS SUMMARY

CRA NETWORKAUDIT 2020QOS MEASUREMENTS SUMMARY

Communications Regulatory Authority

2

TABLE OF CONTENT

1. INTRODUCTION........................................................................................................................... 3

2. QUALITY OF SERVICE AUDIT.................................................................................................... 3

2.1 Measurement Methodology.............................................................................................................. 3

2.1.1 Voice Service Quality Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.2 Short Message Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.3 Data Service Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2 Equipment Used...................................................................................................................................... 4

2.3 Measurement Timeline........................................................................................................................ 4

2.4 Samples (Rounded)................................................................................................................................. 4

2.5 Voice Service.............................................................................................................................................. 4

2.5.1 Call Completion Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.5.2 Unsuccessful Call Attempt Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.5.3 Call Setup Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.5.4 Voice Quality (MOS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.6 Short Message Service.......................................................................................................................... 7

2.7 Data Services to Operator’s Server................................................................................................. 8

2.7.1 Data Download Throughput – Maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.7.2 Data Download Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.7.3 Data Upload Throughput – Maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.7.4 Data Upload Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.7.5 Webpage Download Success Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.8 Data Services to International Servers....................................................................................... 10

2.8.1 Data Download Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.8.2 Data Upload Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

CRA Network Audit 2020 QoS Measurements

3

1. INTRODUCTION

As per the mandate provided to the Communications Regulatory Authority, to safeguard the interest of telecoms consumers, to publish information regarding the performance including comparisons of Quality of Service of the Service Providers, CRA has conducted the Quality of Service (QoS) Network Audit for the year 2020. The summary of the findings of the audit are as below.

2. QUALITY OF SERVICE AUDIT

The QoS audit is conducted to benchmark the Key performance Indicators of the services viz. Voice calls, Short message service (SMS) and Data service offered to the consumers. The samples are collected based on normal consumer behavior and from areas weighted based on the population density for the State of Qatar.

2.1 Measurement Methodology

2.1.1 Voice Service Quality TestingA voice measurement is a call to a Speech Quality Server for one minute duration and Mean Opinion Score (MOS) is recorded using POLQA algorithm.

2.1.2 Short Message ServiceA 52 character sample SMS is send automatically and the received timings are recorded. Tests are conducted within and across networks.

2.1.3 Data Service Testing• Data throughput measurement is carried out in a stationary environment by calculating

the average throughput for completion of an entire session from/to a server located within the operator’s network. Different file size and their respective maximum permissible timeouts as mentioned in the below table are used while collecting the samples.

Measurement Size / Timeout

Dowload (MB) 10 50 100

Upload (MB) 5 25 50

Timeouts (seconds) 150 300 300

• Webpage accessibility measurement is carried out in drive test and stationary environment which includes download a standard test page from the server.


4

2.2 Equipment Used• 4 UE’s - LTE CAT 16 device (Smartphone).• 2 UE’s - LTE CAT 18 device (Smartphone).• 2 UE’s - 5G enable device CAT 24 (Smartphone).• R&S Bench marker II Drive test software.• Handheld device - LTE CAT 16 device with Qualipoc software.

2.3 Measurement Timeline• 07th September 2020 till 31st December 2020.

2.4 Samples (Rounded)• Voice Service – 30900• Voice (Speech Quality) – 12700• Short Message Service – 5600• Data Service – 5800 • Web Service – 78800

KPI Definition

Call Completion Rate

Probability that a successful call attempt is maintained for 1 minute until it is released intentionally by user.

Number of normally ended callsCall completion rate = x100

All successful call attempts by all users

Unsuccessful Call Attempt Rate

The call set-up time is the time period between sending of complete address information and receipt of call set up notification.

Unsuccessful telephony service attempts by all users when service shown as available

Unsuccessful call attempt rate = x100All telephony service attempts by all users

Call Set Up Time The call set-up time is the time period between sending of complete address information and receipt of call set up notification.

Call setup time=t (Connect established) - t (User pressed button on terminal)

Voice Quality (MOS)

Voice quality in mobile networks is measured with algorithms based on ITU-T P.863 (POLQA), which covers the overall listening speech quality from narrowband (300 to 3’400 Hz) to super wideband (50 to 14’000 Hz) telecommunication scenarios as perceived by the user. The Mean Opinion Score (MOS) in both uplink and downlink, averaged for a call duration of 1 minute is reported.

2.5 Voice Service


5

Call Completion Rate

Statistical Accuracy ±0.03% ±0.02%

2.5.1 Call Completion Rate

2.5.2 Unsuccessful Call Attempt Rate

Unsuccessful Call Attempt Rate


50%

60%

70%

80%

90%

100%

99.91% 99.96%

0.00%

0.20%

0.40%

0.60%

0.80%

0.34%

0.60%


6

2.5.3 Call Setup Time

2.5.4 Voice Quality (MOS)

Average Call Setup Time (Sec)

Average Voice Quality (MOS)

0.00

1.00

2.00

3.00

4.00

3.81

3.20

0.00

1.00

2.00

3.00

4.00

5.00 3.82 3.79


7

SMS Successful Completion RateSMS Successful Completion Rate(Own Network)(Own Network)


KPI Definition

SMS Successful Completion Rate (Own & Cross Network)

Number of SMS transfer completedSMS success rate = x100

Number of SMS sent successfully

SMS End to End Transfer Time

The SMS end to end delivery time is the period of time between sending a short message to the network and the message being received at the distant terminal (user device)

SMS end to end delivery time= t (B,received) - t (A,send initiate)

2.6 Short Message Service

SMS Successful Completion RateSMS Successful Completion Rate(Cross Network*)(Cross Network*)


SMS Avg End to End Transfer Time (Sec)SMS Avg End to End Transfer Time (Sec)(Cross Network*)(Cross Network*)

SMS Avg End to End Transfer Time (Sec)SMS Avg End to End Transfer Time (Sec)(Own Network)(Own Network)

*Note: The originating operator’s does not have absolute control on Cross network KPI’s which are terminated in the other network.

95%

96%

97%

98%

99%

100%

99.61% 99.86%

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1.56

2.46

0.0

0.5

1.0

1.5

2.0

2.5

3.02.46

1.64

95%

96%

97%

98%

99%

100%99.61% 99.44%


8

KPI Definition

Data Download Throughput

Downloading a file via HTTP.

The Average throughput to download the entire session is calculated.

2.7 Data Services to Operator’s Server

Average Download Throughput - Maximum

Average Download Throughput - Local Server

2.7.1 Data Download Throughput – Maximum

2.7.2 Data Download Throughput – Distribution & Cumulative

CDFCDF

0

100

200

300

400

500

600

700

800

743.89 Mbps

600.79 Mbps

1.8% 2.8%4.4%

6.3% 8.2% 10.6%15.4%

22.8%

39.8%

54.6%73.6%

90.0%96.4%

100.0%

0.8% 2.4%6.5% 11.4%

16.9% 21.8%28.0% 38.6%

53.7%62.3%

76.2%

89.7%96.4% 100.0%

0%10%20%30%40%50%60%70%80%90%100%

0

50

100

150

200

250

300

>500 450 400 350 300 250 200 150 100 75 50 25 10 <10

# S

ampl

es

Mbps


9

Average Upload Throughput - Maximum

KPI Definition

Data Upload Throughput

Uploading a file via HTTP.

The Average throughput to upload the entire session is calculated.

Average Upload Throughput - Local Server

2.7.3 Data Upload Throughput – Maximum

2.7.4 Data Upload Throughput – Distribution & Cumulative

CDFCDF

0.7% 2.9%11.1%

30.5%

59.3%

83.1%94.3%

100.0%

0.6% 5.1%14.5%

30.1%

46.4%

69.5%

89.1%100.0%

0%10%20%30%40%50%60%70%80%90%100%

050

100150

200250300350400450

>70 60 50 40 30 20 10 <10

# S

ampl

es

Mbps

010

20304050607080

78.46 Mbps 75.42 Mbps


10

Successful Webpage Download Rate


2.7.5 Webpage Download Success Rate

KPI Definition

Successful Webpage Download Rate

Number of completed download sessionSuccessful webpage download rate = x100

Number of session started successfully

CRA conducted test to multiple international servers located in different locations outside Qatar, to portray typical user experience for data sessions. The average throughputs across different test samples are summarized below. The results are indicative and represents throughputs to the International servers, chosen by CRA and the identity of the locations of the servers are kept anonymous to the service providers. The throughput rates can vary depending on various factors for e.g. Internet traffic, Location of the server which may be beyond the operator’s absolute control.

2.8 Data Services to International Servers

50%

60%

70%

80%

90%

100%

99.98% 99.93%


11

2.8.1 Data Download Throughput – Distribution & Cumulative

2.8.2 Data Upload Throughput – Distribution & Cumulative

Average Download Throughput - International Server

Average Upload Throughput - International Server

CDFCDF

CDFCDF

0.0% 3.8% 6.0% 9.6%14.7%

21.3%27.8%

39.2%

56.4%

79.6%92.3%

100.0%

10.2%18.3% 19.3% 20.1%

24.4%35.3% 45.8%

53.4%

67.9%

85.4%95.1% 100.0%

0%10%20%30%40%50%60%70%80%90%100%

050

100150

200250300350

>150 100 90 80 70 60 50 40 30 20 10 <10

# S

ampl

es

Mbps

0.6% 1.3%5.6%

19.2%

43.4%

86.6%100.0%

0.8% 3.8%

10.0%21.4% 41.7%

67.1%

100.0%

0%10%20%30%40%50%60%70%80%90%100%

0

200

400

600

800

>60 50 40 30 20 10 <10

# S

ampl

es

Mbps


12www.cra.gov.qa

Documents

CRA NETWORK AUDIT 2020 QOS MEASUREMENTS SUMMARY