A preliminary test for broadband speed

measurementToshiya JitsuzumiKyushu University

Jitsuzumi@econ.kyushu-u.ac.jp

Jan. 19, 2014

Question

•How large is the variation in speed measurement results and how does it affect the measurement?

•Is it necessary to measure things other than download speed?

Part 1: Fixed (& Wi-Fi)

broadband

• Connected to the speedtest.net website and measured the download/upload speeds and latency as frequently as possible during 30 minutes.

Test Setting

• Date and timeJan. 18, 2014 14:00-14:30 (Japan Time)

• Measurement place and Internet setting

My office at Kyushu University equipped with the university’s wired/wireless LAN.

Router: LYNKSIS Wireless-N Home Router (WRT150N)

Measurement site

• Speedtest server at Tokyo, hosted by Alocac, Inc.• Network path from my office [133.5.240.168]

1 203 ms 1 ms 1 ms 192.168.1.12 1 ms 1 ms 1 ms router.en.kyushu-u.ac.jp [133.5.240.254]3 1 ms 1 ms 2 ms 133.5.69.2414 2 ms 2 ms 2 ms 133.5.9.2305 1 ms 1 ms 1 ms 150.99.191.416 13 ms 13 ms 13 ms osaka-dc-rm-ae2-vlan10.s4.sinet.ad.jp [150.99.2.41]7 13 ms 13 ms 13 ms osaka-dc-gm1-ae0-vlan10.s4.sinet.ad.jp [150.99.2.58]8 21 ms 17 ms 17 ms 103.246.232.949 52 ms 15 ms 15 ms obpjbb206.int-gw.kddi.ne.jp [118.155.199.73]

10 22 ms 22 ms 47 ms otejbb206.int-gw.kddi.ne.jp [59.128.4.181]11 23 ms 22 ms 23 ms cm-ote232.int-gw.kddi.ne.jp [118.155.197.13]12 23 ms 22 ms 22 ms 111.87.16.4213 * * * Request timed out.14 * * * Request timed out.15 * * * Request timed out.16 23 ms 22 ms 23 ms kvps-27-34-160-198.secure.ne.jp [27.34.160.198]

Test parameters

• Three PCs were used for speed measurement.• DELL XPS 8500 [Wireline 1]

• Windows 7 Ultimate Service Pack1, Intel® Core™ i7-3770 CPU @3.40GHz• RAM 16.0GB, 64bit Operating System• Connected to the university’s LAN via cable

• DELL XPS 420 [Wireline 2]• Windows Vista Home Premium Service Pack2, Intel® Core™ 2 QuadCPU Q9550

@2.83GHz• RAM 4.0GB, 32bit Operating System• Connected to the university’s LAN via cable

• MacBook Air [Wi-Fi]• OS X version 10.9.1, 1.6GHz Intel Core 2 Duo• RAM 4GB 1067MHz DDR3• Connected to the university’s LAN via Wi-Fi

0

10

20

30

40

50

60

70

0 5 10 15 20 25 30

Download Speed (Mbps)(2013/01/18 14:00-14:30)

Wireline 1 Wireline 2 Wi-Fi

Result 1: DL SPD Wireline 1 Wireline 2 Wi-Fi

N 59 61 60

Average 38.1Mbps 36.0 17.9

Standard dev. 8.0Mbps 8.4 2.1

Coefficient of variation 0.209 0.233 0.116

0

10

20

30

40

50

60

0 5 10 15 20 25 30

Upload Speed (Mbps)(2013/01/18 .14:00-14:30)

Wireline 1 Wireline 2 Wi-Fi

Result 2: UL SPD Wireline 1 Wireline 2 Wi-Fi

N 59 61 60

Average 34.8Mbps 24.8 16.9

Standard dev. 9.1Mbps 7.0 0.8

Coefficient of variation 0.262 0.280 0.048

0

5

10

15

20

25

30

35

40

45

0 5 10 15 20 25 30

Latency (ms) (2013/01/18 14:00-14:30)

Wireline 1 Wireline 2 Wi-Fi

Result 3: Latency Wireless 1 Wireless 2 Wi-Fi

N 59 61 60

Average 25.6ms 16.6 31.4

Standard dev. 1.78ms 3.73 2.13

Coefficient of variation 0.070 0.224 0.068

Result 4: DL vs. UL

0

10

20

30

40

50

60

0 10 20 30 40 50 60 70

Upl

oad

Spe

ed (

Mbp

s)

Download Speed (Mbps)

Download vs. Upload

Wireline 1 Wireline 2 Wi-Fi

UL= 33.3+ 0.04ถ𝑝= 0.58DL− 9.91D𝑤2 − 17.1D𝑤𝑖𝑓𝑖 Note: UL=upload speed (Mbps); DL=download speed (Mbps); Dw2 = 1 if

the data comes from Wireline 2, or = 0, otherwise; Dwifi = 1 if the

data comes from Wi-Fi, or = 0, otherwise; and p is p-value.

Result 5: DL vs. Latency

10

15

20

25

30

35

40

0 10 20 30 40 50 60 70

Late

nvy

(ms)

Download Speed (Mbps)

Download vs. Latency

Wireline 1 Wireline 2 Wi-Fi

LAT= 27.4− 0.05ถ𝑝= 0.10DL− 9.06D𝑤2 + 4.84D𝑤𝑖𝑓𝑖 Note: LAT=latency (ms); DL=download speed (Mbps); Dw2 = 1 if the

data comes from Wireline 2, or =0, otherwise; Dwifi = 1 if the data

comes from Wi-Fi, or =0, otherwise; and p is p-value.

Preliminary findings and implications• Variation in broadband speed measurement was quite large and therefore

it is important to gather a large number of measurements to obtain robust results.

• Surprisingly, Wi-Fi connection generated more stable results than wired connection.• Latency measurement had less variation than speed measurement.

• In measuring QoS of fixed broadband, Wi-Fi or wired must be clearly addressed.

• Wi-Fi connection downgraded the speed by half, and the latency by 20 to 90%.

• Measuring download speed only may not be sufficient to describe the overall QoS.

• Correlation between download speed and the other measures was not significant.

Part 2: Mobile broadband

• Install the iPhone App “OOKLA SPEEDTEST” and measured the download/upload speeds and latency as frequently as possible during 30 minutes.

Test Setting

• Date and timeJan. 19, 2014 14:00-14:30 (Japan Time)

• Measurement settingiPhone 5 and iPad 2 on my desk at Kyushu University.

Test parameters and measurement site• Two mobile terminals were used for speed measurement.

• iPhone 5 [LTE]• iOS 7.0.4• 16GB model• Disabled the Wi-Fi function and connected to the LTE network of Softbank

• iPad 2 [3G]• iOS 7.0.4• 64GB model• Disabled the Wi-Fi function and connected to the 3G network of Softbank

• Speedtest server at Tokyo, hosted by Alocac, Inc.

Path to the site1 1.479ms 0.772ms 0.793ms 172.20.10.12 48.269ms 50.246ms 57.111ms 192.168.120.653 53.922ms 40.201ms 54.193ms 192.168.120.664 63.959ms 58.92ms 81.512ms 192.168.120.1165 78.297ms 53.911ms 78.864ms 192.168.120.1136 81.238ms 79.041ms 77.41ms 192.168.120.1137 76.054ms 57.761ms 96.612ms 192.168.120.1248 * * * *9 102.022ms 61.808ms 80.319ms pw126211191058.5.panda-world.ne.jp

10 * * * *11 113.069ms 60.569ms 80.519ms 172.28.62.1212 79.172ms 60.716ms 64.358ms 172.28.62.2713 54.858ms 78.124ms 78.042ms 172.28.62.2714 * * * *15 112.238ms 62.091ms 78.308ms pw126211191113.5.panda-world.ne.jp16 81.309ms 63.087ms 95.298ms pw126211186130.5.panda-world.ne.jp17 60.359ms 61.717ms 80.3ms pw126211184097.5.panda-world.ne.jp18 78.349ms 79.309ms 61.638ms pw126211184041.5.panda-world.ne.jp19 81.181ms 62.049ms 79.951ms 101.110.21.12120 81.927ms 64.81ms 80.356ms 10.7.196.2121 79.257ms 61.105ms 74.447ms 10.7.61.2122 91.198ms 57.532ms 78.262ms 10.2.194.12523 77.631ms 62.634ms 81.464ms 10.9.194.14224 78.086ms 59.822ms 79.421ms 210.132.127.16525 79.301ms 47.306ms 79.344ms obpjbb206.int-gw.kddi.ne.jp26 96.986ms 99.193ms 79.359ms otejbb206.int-gw.kddi.ne.jp27 79.105ms 98.464ms 80.246ms cm-ote232.int-gw.kddi.ne.jp28 56.267ms 95.245ms 78.588ms 111.87.16.4229 * * * *30 * * * *31 * * * *32 116.654ms 88.744ms 80.311ms kvps-27-34-160-198.secure.ne.jp

0

5

10

15

20

25

30

0 5 10 15 20 25 30

Download speed (Mbps) (2013/01/19 14:00-14:30)

DL_3G DL_LTE

Result 6: DL SPD LTE 3G

N 63 18

Average 13.46Mbps 6.08

Standard dev. 4.88Mbps 1.04

Coefficient of variation 0.362 0.171

0

2

4

6

8

10

12

14

16

0 5 10 15 20 25 30

Upload speed (Mbps) (2013/01/19 14:00-14:30)

UL_3G UL_LTE

Result 7: UL SPD LTE 3G

N 63 17

Average 7.55Mbps 0.75

Standard dev. 2.96Mbps 0.34

Coefficient of variation 0.392 0.456

Result 8: Latency

0

20

40

60

80

100

120

0 5 10 15 20 25 30

Latency (ms) (2013/01/19 14:00-14:30)

Latency_3G Latency_LTE

LTE 3G

N 63 18

Average 57.67ms 79.22

Standard dev. 6.85ms 9.30

Coefficient of variation 0.119 0.117

Result 9: DL vs. UL

0

2

4

6

8

10

12

14

16

0 5 10 15 20 25 30

Upl

oad

Spe

ed (

Mbp

s)

Download Speed (Mbps)

Download vs. Upload

UL_3G UL_LTE

UL= −0.88+ 0.27ถ𝑝<0.01 DL+ 4.83D𝐿𝑇𝐸

Note: UL=upload speed (Mbps); DL=download speed (Mbps); DLTE = 1

if the data comes from LTE or = 0, otherwise; and p is p-value.

Result 10: DL vs. Latency

0

20

40

60

80

100

120

0 5 10 15 20 25 30

Late

ncy

(ms)

Download Speed (Mbps)

Download vs. Latency

Latency_3G Latency_LTE

LAT= 82.4− 0.46ถ𝑝= 0.02DL− 18.5D𝐿𝑇𝐸

Note: LAT=latency (ms); DL=download speed (Mbps); DLTE = 1 if the

data comes from LTE or = 0, otherwise; and p is p-value.

Preliminary findings and implications• Variation in LTE broadband measurement was larger than that of fixed

broadband in terms of download/upload speed, and therefore it needs a larger number of samples.

• Because the path to the measurement site was much longer in the mobile broadband, there seems to exist more factors that affected the E2E performance than in the fixed counterpart.

• Compared to fixed broadband, download speed figures have more representative values in mobile broadband.

• Correlation between download speed and the other measures was highly significant.

Agenda for future research

I need three things to move forward:1. more data, 2. more data, and3. more data.