4G network quality test - · PDF file• More than 100 KPIs reported. ... The four Huawei Honor 6C used for the test ... 4G 3G 2G no network Legend:

4G networkquality test

test on the high-speed train line rennes-Paris

july 2017

SmartViSer - teSting expert

teSt on the high-Speed train line

the reSultS

contents

Page 3

Page 4 and 5

Page 7 to 19

Page 7 to 11

Page 12 to 16

Page 7 and 8

Page 17

Page 12 and 13

Page 9 and 11

Page 18

Page 14 to 16

Page 19

Page 17 to 19

1 - from rennes to Paris

2 - from Paris to rennes

4G networkweb Performance

4G networkweb Performance

4G networkweb Performancebattery life

3 - results’ averaGe

Page 4Page 5Page 5 and 6

objectivetests’ conditionstest Protocol

• End User at the heart of innovation.Based on our Virtual User concept, our embedded powerful test engine acts as a test bot for Smart-phones and Networks.

• More than 100 KPIs reported.From battery performance to web speed and network coverage viSerTM application records accurately the relevant performance indicators in the lab or in the field...

• Answer to challenges.Our mission is to support operators, manufacturers, regulators, retailers or specialized media in their tes-ting activities with the objective to reach the best end user satisfaction and to empower their brand reputa-tion.

SmartViser - All rights reserved - Paris-Rennes high-speed train line test - July 2017

smartviser Go further in the customer exPerience

3/20

• ObjEctIvE

The report’s objective is to give a compari-son overview, of the 4G network quality, of the four French operators: Bouygues Tele-com, Free, Orange and SFR.

Now it only takes 1h25 to travel from Rennes to Paris but which operator offers the best 4G netwrok quality? In France the opening of new high-speed train lines is a crucial is-sue for operators to ensure they meet the users’ expectations.

This is why this report focuses on the cus-tomer experience by testing the network quality in the new high-speed train line Rennes-Paris.

• tEsts’ cOndItIOns

The test protocol chosen is in accordance with ARCEP, the French High Regulation Authority of the Telecommunication.

three main KPIs were measured:- 4G coverage- Web Performance, (connectivity in mobility)- Battery life

The testing was carried out using four de-vices installed with viserTM, SmartViser’s au-tomated solution and each had a different SIM card from the four networks under test.

All four devices carried out the testing simul-taneously for the duration of the return jour-ney from Rennes to Paris and then Paris to Rennes.

tHe test4G network test on the hiGh-sPeed train line

4/20Map from SNCF, the French train company: https://tinyurl.com/yagxdbwd


• date: 6th july 2017

• train numbers: N° 8606 et N° 8085

• devices:Huawei Honor 6C

• IMEI:863734031539927863734030775522863734031526403863734031538507

• software:Diego-L21HNC432B121

• speed and duration:380 km - 1h25 one way.

• test scenario:The complete scenario lasted 11 minutes. The scenario was started at Rennes’s de-parture and stopped at Paris’s arrival.

For the return journey, the test was started at Paris’s departure and stopped at Rennes station.

5/20

To execute the entire test, the viSer app, developed by SmartViser, was used.Each device used exactly the same scenarios and parameters to execute the testing and were placed on top of a table in the same carriage of the train. No manual action from a team member was required.

All the results presented in the report were collected by visertM.

• tEst PrOtOcOl



6/20

The four Huawei Honor 6C used for the test

• User actions:

- 1 Web navigation every 20 seconds - 30 different websites - Actions must be completed in 10 seconds

- Amazon- Fnac- Paypal- Service Public- Facebook- Pole Emploi- Instagram- Twitter- Voyages SNCF- Skype

- Yelp- Caisse d’Epargne- Youtube- Wikipédia- Darty- Live- La Banque Postal- Credit Mutuel- CAF- Le boncoin

- Pages jaunes- Snapchat- Whatsapp- France Info- Trip Advisor- Waze- Twitch- La fourchette- Yahoo-Google

• tEst PrOtOcOl


1 // bar graph: The bar graph below indicates the 4G coverage for each operator during the test on the high- speed train line.

2 // Map:The maps indicate the change of different network technology during the journey. It gives a clear visibility of the geographic zones with good or insufficient network cover.

7/20

4G 4G quality network network coveraGe

❷

❹

❶

❸

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Couverture en service 4G (LGV Paris-‐Rennes) 6 juillet 2017

22%17%15%

91%

4

1

3

tHe results from rennes to Paris

8/20

2G3G

22%

15%

91%

17%

4G no network

4G network quality detailed maPs

legend:



The web performance results are detailed with three values:

1 // bar Graph:The bar graph below indicates the pass rate of the web-pages loaded in less than 10 seconds as per ARCEPs’ specifications.

2 // Map:The maps indicate the successful actions in green and the failed ones in red. It gives a clear view of the geographic zones with good or insufficient network coverage.

3 // column chart:The column chart indicates the throughput during the web-pages loading.

9/20

web Performance webPaGes loaded in less than 10 seconds

❸

❹

❶

❷

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Pages Web chargées en moins de 10 secondes 6 juillet 2017

24%

13%

74%

34%2

3

10/20


failedsucceededLegend:

24%

13%

74%

34%



11/20



1 // bar Graph:The bar graph below indicates the 4G coverage for each operator during the test on the high-speed train line.

2 // Map:The maps indicate the change of different network technology during the journey. It gives a clear view of the geographic zones with good or insufficient network coverage.

12/20


❷

❹

❶

❸

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


34%

6%3%

87%

2

4

tHe results from Paris to rennes


13/20


34%

3%

87%

6%

2G3G4G no networkLegend:



14/20

The web performance results are detailed with three values:

1 // bar Graph:The graphic below indicates the pass rate of the web-pages loaded in less than 10 seconds as per ARCEP’s specifications.

2 // Map:The maps indicate the successful actions in green and the failed one in red. It gives a clear view of the geographic zones with good or insufficient network coverage.

3 // column chart:The column chart indicates the throughput during the web pages loading.

34%

11%

76%

26%

1

2

3

4


15/20

34%

11%

76%

26%


failedsucceededLegend:


web naviGation webPaGes loaded in less than 10 seconds

16/20


• The average of the two journeys between Paris and Rennes clearly shows that one operator gives better results. Thanks to a good 4G quality network, Oranges’ suscribers get the best customer experience in mobility.

tHe results averaGe of the journeys

4G quality network network coveraGe

17/20

❷

❹

❶

❸

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


28%

14%10%

89%


web naviGation webPaGes loaded in less than 10 seconds

18/20

Average based on the return trip Rennes-Paris.

• With 75% pass rate for the web-page downloaded, the French operator Orange offered the best customer experience during the testing journey.

Webpages loaded in less than 10 seconds

Failed download in ArcEPs’ conditions

128381

355154

361148

44861

In the scenario, SmartViser has automated the download of 509 web-pages, (261 on the Rennes- Paris journey and 248 on the return journey).

❷

❹

❶

❸

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


29%

12%

75%

3

30%


battery life network quality imPact on the battery life

Average rate of the two journeys.

• The network quality has a direct impact on the smartphone’s battery life, a poor quality will directly slow down the web-page download and use more battery. Orange is the operator with the best per-formance and it offered 10% more battery life during the testing journey.

19/20

Consomma'on ba*erie du mobile 6 juillet 2017

37%34%

44%3

21

4

42%

Testing innovation

About SMARtVISER. Founded in 2014, SmartViser is the forerunner of embedded testing solution for smartphones, tablets and networks, based on the virtual user concept. SmartViser evolves into B2B market and is dedicated to network operators, mobile manufacturers, retailers, benchmark experts and all organizations that evaluate Customer Experience, (devices and networks).

To this date SmartVisers’ solutions have already conquered major telecom operators, mobile manufacturers, retailers and medias. The company’s ambitions is to become the reference partner for connectivity challenges, based on its unique customer experience approach.

www.smartviser.com

[email protected]

+ 33 2 99 314 208

SmartViser@

contact us

Documents

4G network quality test - · PDF file• More than 100 KPIs reported. ... The four Huawei Honor 6C used for the test ... 4G 3G 2G no network Legend: