CheesePi: Measuring Home Network Performance Using Dedicated Hardware

Devices

Abstract Internet users may not get the service quality promised by their providers, and also may not know what service they can receive. When users experience poor Internet connection performance, it is not easy to identify the source of the problem. We develop CheesePi, a distributed measurement system that measures the Internet connection experience of home users based on some network performance attributes (e.g. latency, packet loss rate, and WiFi signal quality). The CheesePi runs on a Raspberry Pi (a credit card sized computer) connected to the user’s home network as a measurement agent. It is important to measure the network performance from the user’s side since it is difficult to measure each individual’s link from the operator (provider) side. Each measurement agent conducts measurement periodically without disturbing the user’s Internet quality. Measurements are conducted during big media events from SICS (Swedish Institute of Computer Science) labs and student accommodations. The measurement results show customers with an Ethernet connection experienced significantly better latency and packet loss compared to WiFi users. In most of the measurements users at SICS lab perceived better latency and packet loss compared to the users at the student accommodation. We also quantify how customers experienced lower performance when streaming from websites which do not use CDN technology compared to the websites which do use CDN, particularly during big media events.

1 Introduction This chapter gives an overview of the master’s thesis focus area to help readers understand what the project is essentially about. Next the existing problems that motivate this work will be discussed, followed by the project’s goal. Thereafter the specific goals and contribution of this master thesis in the project is going to be presented. At last the organization of the report will be explained.

1.1 Motivation The Internet is one of the very essential needs in the modernised world. We use the Internet for communication, sharing information, watching videos and TV, shopping, and many other activities. According to the Internet World Statistics [22], there are about 9.2M Internet users in Sweden, 604M in Europe, and 3.3B around the world.

1

Approximately 85% of Internet users watch online videos [1]. Users become frustrated when videos pause while the system downloads additional data before resuming video playback. According to a new study from the University of Massachusetts Amherst and Akamai Technologies [23], many broadband Internet users give up on an online video if it doesn’t display within two seconds (probably not true in developing countries). Most Internet users, when they perceive a connection problem, they have little idea about its root cause. The problem can be due to a variety of reasons, sometimes it can be a local problem due to poor WiFi connection in the user’s environment. It could be also due to the Internet Service Provider’s (ISP) lack of bandwidth, or several other reasons. Measuring the Internet connection is important for the home user to ascertain if they are receiving the promised service. It is also important from the service providers and regulators point of view, to characterize the whole network and assessment of potential degradation in the services they provide. In other words it is monitoring the network and services they provide. Researchers have been trying to measure end-to-end quality-of-service (QoS) for many years, and we want to add our contribution for users’ better experience.

1.2 Goals The main goal of this project is to build a measurement platform (CheesePi) that help home Internet users to characterize their connections and with collated data to help ISPs and operators to fix network problems. The main goals of this master thesis are:

1. Design the database of the different measurement tools in the MA (Raspberry Pi) and in the central server.

2. Build test modules for ping, httping, and traceroute network measurement tools.

3. Run several scheduled measurements periodically. 4. Perform specific measurements during big media events to popular streaming

websites and integrate the results into the database. 5. Conduct analysis on the collected results and draw conclusion on the

observations.

1.3 Organisation of the report Chapter one introduces the motivation and objective of the project, and the contribution of the master thesis. Chapter two introduces the area of the thesis, and introduce the network tools used on this project. Chapter three discusses the related projects. Chapter five is going to explain the methodology followed for building this project and conducting the experiments. Chapter five presents the results of the experiment and small discussion about the observations. Chapter six discusses the observations and the followed approach during the master thesis. Chapter seven

2

draws conclusion about this master thesis. Chapter eight presents the possible future works of this project. Chapter nine presents the lessons learned during this master thesis. The last part of this report presents the references of articles and technical reports used for this master thesis and an appendix.

3

2 Background First in this chapter, the general purpose of the Internet and how it works will be explained first. Then, different way of accessing the Internet are going to be introduced. After that, different network performance measurement approaches followed by the tools used for measuring network performance will be discussed. At the end, different content delivery techniques used by different media streaming services will be introduced.

2.1 Internet Protocol (IP) Networks The Internet is a network of many networks. The connected devices and networks are differ in hardware, software and design, so they need a common protocol in order to communicate with each other. The Transmission Control Protocol/Internet Protocol (TCP/IP) protocol enables inter-connected networks to perform end-to-end communication, so that any node on the Internet should have the ability to communicate with any other node.

Figure 2.1: Internet building blocks [2]

The TCP/IP protocol suite is divided into four layers for division of labor and easiness of implementation and is called a protocol stack [3]. The four layers are: Application, Transport, Network, and Link layers. The application layer provides a service that enables a user to communicate over the TCP/IP model, for example services such as: HTTP (web), FTP (file transfer), and SMTP (mail). The transport layer provides an end-to-end data transfer between applications or peers, and the most used protocols are the Transport Control Protocol (TCP) and the User Datagram Protocol (UDP). Data is multiplexed up the network stack at this level from information in the

4

port numbers. This is because network flows arriving at this machine, but are being used by different applications. The Network layer transfers data from source to destination by performing routing functions, and the IP is the most prominent protocol at this layer. In order to demultiplex flows at the router level, the IP addresses are used. And the Link layer is the interface to the network hardware, for example, 802.11, ATM, and X.25. For example, a WiFi user connected to the home router wirelessly wants to send data to a target server. The frame is send to the first hope (WiFi router), then the router sends it to the next router, and it goes like that till it reach the target server. The frame contains source and the next hop MAC (Media Access Control) address (Link Layer), source and destination IP address (Network or Internet Layer), the source and destination port (Transport Layer), and the data. The destination port tells about the used application layer protocol, for example, if the destination port is 80, it is an HTTP protocol for accessing the web. Unlike the IP addresses, the source and destination MAC addresses changed every time during the path till it reaches the destination host.

Figure 1.2: The TCP/IP Model

IP defines an addressing schema to uniquely identify a host and devices on a network. When a packet is sent from the source to a targeted destination, it should include the IP address of the sender and the receiver. The routers in the network select where to forward the packet based on the destination IP address. The IP version 4 (IPv4), a widely deployed protocol. IPv4 uses a 32 bit addressing schema to represent a device in the network, which allows more than four billion (2^ 32) addresses, whilst IP version 6 (IPv6), created to resolve shortage of addresses in IPv4, uses a 128 bit addressing schema. IP does not provide delivery guarantees,

5

and hence is called a ‘best effort’ delivery, this means packets can be lost, corrupted, and received out of order. It is up to the upper layers like Transmission Control Protocol (TCP) to guarantee the (eventual) delivery.

2.2 Access technologies Users link to the network via different technologies. There are several choices if a user wants to access the Internet namely: WiFi, Ethernet (typically via a cable), and Mobile Data (Cellular Network) are common available choices in 2015.

2.2.1 Ethernet We are not going to discuss the details of the Ethernet protocol in the link layer of the TCP/IP stack. Ethernet is used to connect computers in a home or company to a modem (or router) for Internet access. It typically uses a cable to connect a computer to a modem (router) and does not suffer from medium contention [5]. Some users prefer Ethernet for speed, security or even (simpler) configuration reasons.

2.2.2 WiFi The popularity of laptops and smartphones make WiFi a widely used access method to the Internet. Many people at homes, offices, and public areas use WiFi technology to access the Internet. In WiFi, there is no physical connection between the mobile device and a fixed device that has permanent, often non-mobile and stationary connection to the Internet. Most home users are familiar with a wireless router or an access point. When a user connects to the Internet all the data transferred to the access point is sent over unlicensed radio spectrum. A user should be within the coverage range in order to have access. A periodic beacon sent by the router, needs to be seen to join the wireless network, via a secure protocol exchange. For example, a wireless router that runs the 802.11n standard, a user should be in the range of 70 meters indoors or 250 meters outdoors [6].

2.2.3 Mobile/Cellular connection In a cellular network the data from a mobile device is transmitted on licensed spectra via a radio-to-air interface [7]. The signal is received at a fixed base station. Each base station is responsible for a physical area, often called a cell. Frequency allocation is an important issue, so not to interfere with adjacent cells and competing providers in the same physical area. 3G/4G networks use CDMA division to achieve separation (as well as frequency bands), however 5G technologies will use CDMA and BDMA. The coverage area of the base station depends on different factors: frequency, number of users, antenna, location (existence of mountains and buildings), and other parameters, but they cover larger areas compared to access points using WiFi technology.

2.3 Network performance measurements It is not easy for customers to monitor the quality of the service they are provided. Currently, from the customer perspective, the performance of the Internet typically

6

means measuring the speed, “how fast is my network?” or “how quickly can I download the data?”. But does speed mean everything? The answer is simply no, because speed is not the performance requirement of every online application. The most popular performance measurement metrics for different applications are:

1. The upload and download speeds 2. The delay or latency between two end points 3. The variation of the delay between selected packets, which is called jitter 4. The ratio of discarded packets to the total number of packets sent (packet

loss ratio) 5. To what extent a sequence of packets is reordered within the network, or

even duplicated by the network (packet error ratio) The purpose of network monitoring is to observe and quantify the network measurement results based on the performance metrics. The two common network monitoring approaches are based on passive and active measurements. The active measurement approach injects test packets into the network by sending packets to servers (testing network reachability and performance) and applications (network reachability and server software, e.g. Apache). The passive approach does not inject test traffic into the network and does not rely on test servers. Instead, passive monitoring uses a device in promiscuous mode to sense existing data traffic. Both approaches have advantages and disadvantages. In active measurements, the traffic is artificial and creates extra load on the network, whilst passive monitoring measures the real traffic originated from the user and does not add extra traffic to the network. Since the passive approach may require sensing all packets on the network, there can be privacy or security issues about how to access/protect the data gathered [12]. The active monitoring provides control on generating the traffic which is not possible in the passive approach, and it helps to have a full control on what and when to conduct measurement.

2.4 Network performance measurement tools This section introduces some of the tools and device used in a network performance measurement.

2.4.1 Ping Ping is used for measuring the network delay or the round trip time (RTT) taken for a packet from a client to a chosen destination. It is supported by most operating systems. Ping uses the ICMP protocol to send a so called ‘echo’ request and reply packets. It puts a timestamp in the packets to measure the RTT, and a sequence number to calculate the packet loss for the unreceived packets. The main problem of ping is: it can be ignored by some servers due to the suspicious nature of malicious attacks, albeit of a high frequency nature, significantly higher than the default one second inter packet time.

7

2.4.2 Httping Httping is used for measuring the latency of the network path and behaviour of a web server. Since the httping requests a full HTTP response from the server, the returned size of the data might be different for each Uniform Resource Locator (URL). Httping works on most operating systems, plus there is an Android version. Httping can include the DNS (Domain Name System) lookups delays as well.

2.4.3 Traceroute Traceroute is a network diagnostic tool that tracks the route (path) of a packet from a source to a destination and measures its transit delays across an IP network. It is used to trace the path a packet follows in a network, and can be used to identify connectivity problems. By default it sends a sequence of UDP packets addressed to a destination host; ICMP Echo Request or TCP SYN packets can also be used [13]. Traceroute starts by sending a packet with a Time-To-Live (TTL), also known as hop limit, at a value of one. The first router decrements the TTL value, and sends an ICMP "time exceeded" reply to the source due to the TTL value in the packet being zero. Then the source sends the next packet with a TTL value incremented by one. The previous router forwards the packet (as the value is now valid) but the next router drops the packet as the previous one decremented the counter to zero. In other words, traceroute acts like an expanding ring search, increasing the scope of the packets. This continues until the destination is reached and returns an ICMP "Port Unreachable" reply or a max (30 or 64) is reached. This is because traceroute packets should not circulate the Internet indefinitely. Three probes (by default) are sent at each TTL setting, and the terminal (console) displays the address of the gateway or the router in the path and the round trip time of each probe. It is supported (included) by most of the modern operating systems. Some routers block ICMP/UDP packets.

2.4.4 Measurement Device There are a number of ways to host a measurement suite. Some of them will be discussed in the state of the art section. We have chosen a standard hardware device, with our dedicated software platform, CheesePi. The Raspberry Pi is a credit card sized single board small computer developed in UK by the Raspberry Pi foundation. The price of a Raspberry Pi is very affordable and about 5 million units have been sold [11]. A Raspberry Pi2 models cost around $35. Due to its robustness, low price, low power consumption, size, and other factors Raspberry Pis are becoming popular and start to be used in many projects including network measurement projects (NetBeez, ARK) see below.

2.4.5 Storage One can store measurement data in a number of different ways. Common log or CSV files are still the predominant format. However within network measurements

8

there has been a trend towards more structured data (M-lab for example). A database is one way to store the data. The data is organised in a way it can easily be accessed, managed and updated by other computer programs. Relational databases, a table-based database represents entities as a schema and allows users to relate data fields and tables with others stored in the database [15]. Recently, non-relational databases, document based and key-value pairs with wide column fields are becoming a popular alternative technology for analytics of large-scale data.

2.5 Content Delivery Approaches When users visit a website to watch a movie or read a news, they don’t know how the content is delivered to their device. In this section we are going to discuss different content delivery approaches.

2.5.1 Single Server The website has only one server, the request for every service from the connected clients is processed by a single server.

2.5.2 Multiple Servers One of the most commonly used applications of load balancing is to provide a single Internet service from multiple servers. For Internet services, the load balancer is usually a software program that is listening on the port where external clients connect to access services. The load balancer forwards requests to one of the "backend" servers, which usually replies to the load balancer. This allows the load balancer to reply to the client without the client ever knowing about the internal separation of functions.

2.5.3 Content Delivery Network (CDN) A Content Delivery Network or Content Distribution Network is a system that replicates digital content over mirrored servers, that are located in suitable places in the Internet [8]. The goal of the CDN is to deliver fast and reliable data by distributing the content to caching servers located closer to the end consumers than the original servers. Most of the popular web streaming like BBC, SVT, YouTube, and Netflix use CDN technology to deliver their content. If a user requests an object or a file from a website that uses CDN technology, instead of the hosting server the CDN takes care of it. This is typically done through the DNS request returning an address to the CDN server, despite the user typing in the commonly known address. In the UNIX system the command “host” can reveal where a CDN is being used. The CDN will consider the geo-location of the visitor and the object will be served from the closest caching server. If the assigned cache server does not have the requested object, it asks other cache servers, then the original (host) server [9]. Note, sometimes the CDN might have to remove some content, if the total catalog size it holds is large and the popularity of items is highly variable.

9

3. State of the art within network monitoring This chapter introduces network monitoring projects and how they are related to CheesePi. It also explains the network tools used during the project. We begin with monitoring systems that use a hardware-based measurement agent. Then it explains the software based network monitoring systems.

3.1 RIPE ATLAS RIPE Atlas is a project measuring Internet quality around the world using measurement probes. The goal is to measure Internet traffic while providing users with information about their connection [24]. Probes perform measurements using ping and traceroute with support from DNS and SSLcert. Users with active hardware probes produce rendezvous points that can be used to set up user defined measurements involving other probes. The source code and RIPE data is available to the public. Around eight thousand probes have been deployed as of 2015. An open source approach is common with CheesePi, but RIPE uses dedicated hardware whilst CheesePI uses commodity hardware.

3.2 ARK The Archipelago Measurement Infrastructure (ARK) uses 120 Raspberry Pis deployed to perform network monitoring to quantify topology (connectivity) and dynamics (congestion) using active measurements for both IPv4 & IPv6 [25]. As of August 2015, the developers UCSD, have deployed 120 nodes. Quantifying the congestion is done by a Time-Sequence Ping (TSP), a crafted sequence of pings along a selected network path. The project conducts measurements on the global ARK overlay network predominantly using traceroute. Variations in the diurnal variation in delay have been quantified. ARK project is a pure networking research project, but CheesePi is a measurement system targeting home Internet users. As said both ARK and CheesePi use Raspberry Pi as the measurement agents.

3.3 SamKnows whitebox It is a project that measures broadband performance for consumers, ISPs, and government (regulators) around the world [26]. They use Whiteboxes (a modified router) to measure the performance of users’ broadband. The hardware is connected to the user’s home router and run tests based on the client’s interest without disturbing the Internet experience. It also sends measurement data to a central server for permanent storage. They also use a web-based measurement similar to the hardware based measurements (using Whitebox), which run tests from the user’s computer. The Whitebox is configured to run several performance measurements. It measures download and upload speed (throughput) between the

10

Whitebox and closely located test node (server). It also measures VoIP (Voice over IP) quality, UDP packet loss and latency under normal and loaded conditions, FTP throughput, DNS resolution latency, peer to peer throughput, Internet connection lost, web page loading latency, email relaying latency, bitrate reliability and startup latency during video streaming, video quality of experience for popular media streaming web servers (e.g. YouTube, Netflix, and BBC iPlayer), performance of multicast IPTV over RTP. They use a dedicated device for continuous measurement and a dashboard for visualizing the results based on user’s interest, which makes it similar to CheesePi. But customers have no freedom on controlling the Whitebox, and can not use the hardware for other purposes. Whereas the Raspberry Pi which runs the CheesePi platform can be used for many different purposes. They have one or more test nodes (servers) in the popular cities around the world from network operators [27], and the stored data in the central server is used by users, ISP, Internet regulators and academia. Samknows has a good collaboration with different organizations, such as Ofcom in the UK and EU research projects.

3.4 NetBeez NetBeez is a distributed network monitoring system, which is mainly targeted at companies with offices in different locations and a centralized IT department [28]. The NetBeez agent (Beez) is deployed behind a switch in every remote location as if it were a normal user. The agents then simulate users in the organisation and monitors the network, if a problem occurs it makes troubleshooting easier by using the diagnostic information from the Raspberry Pi. The measurement agent, which is a Raspberry Pi, is configured to run several tests. It runs ping to verify the connectivity and round trip time between two agents (or other chosen host), the DNS resolution latency, traceroute to check if there is a change in the network path and observe the bottleneck hops, Iperf to test the stress of the bandwidth, and measure the performance of an HTTP server by evaluating the HTTP test packets. All the measurement tests are controlled by the NetBeez server, it also collects the test results from the NetBeez agents. The users are able to control the functionalities through a NetBeez web based Dashboard, and they can visualize the performance on different aspects of the network (locations, targets, historical data). Similar to CheesePi, NetBeez also uses Raspberry Pis as measurement agent. However, CheesePi targets are home users and NetBeez customers are companies who have offices in different locations. CheesePi is free measurement platform while NetBeez offers subscription based services.

11

3.5 Software-based network monitoring systems In this section monitoring systems, which do not use dedicated hardware for conducting the measurements, or follow a software based approach are going to be presented. Deploying software is somewhat easier than a hardware-based approach and can achieve larger scale measurements and coverage. We explain those that are either well known, or are similar in vain to CheesePi.

3.5.1 SpeedTest Speedtest is a web service that provides free analysis of Internet access parameters [29]. Probably from the user perspective it is the most well known one, globally. It was founded by Ookla in 2006, The service measures the bandwidth and latency of a visitor's Internet connection against closest host servers. As of October 2015 [30],They have 4085 distributed host servers around world. This makes it somewhat similar to Bredbandskollen (a Swedish web-based speed test) and SamKnows. Each test measures the data rate for the download direction, i.e. from the server to the user computer, and the upload data rate, i.e. from the user's computer to the server. The tests are performed within the user's web browser using the HTTP protocol. As of 2015, over 7 billion speed tests have been completed. The site also offers detailed statistics based on test results. This data has been used by numerous ISPs and researchers in the analysis of Internet access around the world.

3.5.2 ICSI Netalyzr Netalyzr is a network measurement and debugging service/tool that evaluates users’ Internet connectivity [31]. The tool aims to be both comprehensive in terms of the properties measures, and easy to use as it is web-based for users with little technical background. Netalyzr is a signed Java applet that communicates with a suite of measurement-specific servers. Traffic between the two then probes for a diverse set of network properties, including outbound port filtering, hidden in-network HTTP caches, DNS manipulations, NAT behavior, path MTU issues, IPv6 support, and access-modem buffer capacity. In addition to reporting results to the user, Netalyzr also forms the foundation for an extensive measurement of edge-network properties. Along with describing Netalyzer’s architecture and system implementation, they presented a detailed study of 130,000 measurement sessions from 99,000 public IP addresses that the service has recorded since being made publicly available in June 2009.

3.5.3 Fathom Fathom has been used in home network environment [32] which has similar goals to ours, but differs in the software/hardware and the external/embedded measurement approaches. Fathom is a Firefox extension that implements the measurement primitives of Netalyzer to measure network characteristics using JavaScript. It adds

12

3% to the times to load a web page and can timestamp network events to within one millisecond.

3.5.4 PingER The PingER (Ping End-to-end Reporting) project monitors the end-to-end performance of Internet links worldwide [33]. Developed by the IEPM group at the SLAC National Accelerator Lab in California, PingER has a decade old data repository of network performance measurements from and to sites around the world. Currently, there are over 60 active monitoring hosts in 23 countries monitoring over 300 sites in over 165 countries that cover 99% of the world’s Internet connected 3.2 billion population [22].

13

4. Methodology: a monitoring approach In this chapter we will discuss the research methodology used in the CheesePi project. We applied a quantitative methodology [14] to conduct the research. We used several networking tools to measure the quality of the Internet service and design a database to store the results. Then we conducted analysis on the collected results. This chapter will discuss how the measurement tools and the database have been designed and the important techniques used for the sampling and scheduling the performance measurements.

4.1 CheesePi Architecture The MA runs on a Raspberry Pi connected to a home network. This measurement agent run tests periodically without disturbing the perceived Internet experience of the user. In practice this means using as little capacity as possible. A controller distributes schedules on how tests should run on the MAs. There is a database for storing the measurement result of the four tools namely; ping, httping, traceroute, and VoIP in both the MA and the central server. A collector organizes the results collected from the MAs and stores it in the central server database. A tool recorded the WiFi contention is also part of the system. User can visualize the real-time and history of the experienced Internet performance on a web-based dashboard, which is hosted locally on the Raspberry Pi. The detail of each test modules will be explained later.

14

Figure 4.1: CheesePi Architecture

4.2. ICMP-based measurements We use the default installed ping on Raspbian, an optimized Debian based operating system for Raspberry Pi. The controller in the central server schedules the MA to do the delay measurements. The schedule consists of the number of measurements to perform, the landmarks (the target destinations), and when to start the measurements. In CheesePi, the controller decides how and when the measurements should be conducted. For this master’s thesis, we use a crontab, a wall-clock job scheduler in Unix operating systems to schedule the measurements. For example, the cron job schedule can be, every five minutes conduct ten ping measurements to BBC web server with an interval of one second. “*/5 * * * * /home/pingMeasurement.py ” The default ping packet size we used was 64 bytes, which is eight bytes of the ICMP header and 56 bytes of (null) payload. Before we choose the ping packet size we conducted an experiment to see how the packet size affects the network performance? We sent 100 probes each to www.bbc.com with packet sizes of 64, 128, 256, and 512 bytes in parallel. The experiment was taken at SICS lab, which the download and upload speed is around 100 Mbps (Megabits per second). The average round trip times (RTTs) were 30.75, 30.79, 30.88, and 31.44 ms for the ping measurements with 64, 128, 256, and 512 byte packet sizes. The results indicate

15

that the latency is almost similar for the different packet sizes and we decided to use 64 bytes. We conduct the experiment by considering users with limited data plans, or some form of constrained access. The cron job runs the Python module, that conducts delay and packet loss measurements, according to a the schedule, for example every five minutes. The Python module works as follows:

1. It runs a number of probes, for example, “ping -c 10 www.facebook.com” 2. It refines and organizes the ping results in a suitable format for store in the

database. 3. It stores the necessary measurement metric results and the MA information,

which helps to uniquely identify the MA in the central database, in a local database. For example, average RTT, IP address of the MA, destination domain name, MAC address of the MA, and other information.

4. It also stores the tool output in a file (optional) According to the schedule provided by the controller, the uploader module sends collected ping measurement results to the CheesePi central server. For example, if we decide to send the data every 23:00 UTC (Coordinated Universal Time), at 23:00 the program checks if there are accumulated results since the last submission and it transmits the data to the central server. The controller is responsible for the distributing of the data submission time. Because if all MAs (we assume the number of MAs is large) send their data at the same time, the central server can be loaded and this can lead to improper data handling.

4.3 HTTP-based measurements We installed httping on all Raspberry Pis for measuring the performance of a web server. Similar to ping, the controller in the central server schedules when the HTTP measurements should be conducted. For this master thesis, the cron job runs the python module that performs httping measurements at the same time as the ping measurements. By running both the httping and ping measurements parallely, we can ascertain whether the network or the web server is the source of the low performance during visiting to particular web server. Unlike ping, we do not choose the packet size for the httping probes. Since the httping program returns the HTTP header of the target web server, the packet size can be different for different destination (landmarks). The httping python module runs httping measurements in a very similar way to ping measurements, which organizes the results in a suitable format for the database, stores them in a local database and a file (optional). Then the module uploads the results to the CheesePi central server in a similar manner as ping results uploaded.

16

4.4 Traceroute delay measurements We use the default installed traceroute in Raspbian on all MAs. Unlike ping and httping, traceroute sends three packets at a time and the packets may follow different paths. Therefore the Python module, that does traceroute measurements, has to take into account this potential hurdle. The traceroute module works as follows:

1. It runs traceroute command e.g. “traceroute www.facebook.com”. 2. It refines and organizes the traceroute results in a suitable format to be stored

in the database as in Figure 4.3. 3. It stores the traceroute results and other necessary information, which

uniquely identifies this measurement in the central database, in the local database.

4. It also stores the results in a log file (optional). Then, according to the schedule chosen by the controller, the upload module sends the accumulated results to the CheesePi central server.

4.5 Database design This section describes how the database was designed in the MA and the central server. Choosing an appropriate database for CheesePi was evaluated and relational (SQL, MySQL) and non-relational (Time series, Influx) databases were evaluated during the development of this project. The different databases considered in this project and the scientific approach followed for choosing them will be discussed next. Since CheesePi is a system that helps users to characterize their real-time network performance, how these databases in the MA were designed and what are the advantages and disadvantages of each will be discussed next.

4.5.1 Measurement Agent design I Relational databases are well suited to carrying out complex queries. And MySQL, an open source relational database was chosen for CheesePi because of its open source freedom, online support, scalability and flexibility, high performance, strong security options for data protection, popularity, and management ease. Below are some cases where a single query is used to retrieve data from multiple tables. These are used during a visualization task and analysing the network performance.

17

Figure 4.2: Ping and httping database design

In the MA the ping and httping measurements are each represented by a single table. The table designs of httping and the ping are the same since the metrics used by both tools are similar. Within each tool there can be differences in the output, however to keep the tables manageable we use similar metrics. Some of the attributes like delay (RTT) and packet loss (%) are performance metrics. Other attributes like timestamp and starting and ending time are used in management of the data and measurements. The source and destination addresses plus the target hostname identifies the two end points. And the Ethernet MAC address is used to identify an MA in the central server.

18

Figure 4.3: Traceroute database design As one can see in Figure 4.3 the traceroute measurement result is represented by three tables due to its different behaviour from the previously discussed tools. The tables: Traceroute_Seq, Traceroute, and Hop are related by an attribute ID. The Hop table which stores the details of the results, the routers in the path and their delay, should be related with the Traceroute table which stores the measurement information about the time when the measurement took place, and the details about source and destination nodes. So a unique ID (Ethernet MAC address of the MA + auto-increment integer) is the primary key in the Traceroute table to uniquely identify every traceroute measurement and a foreign key in the Hop table to relate the details for each measurement. We make the ID a combination of the Ethernet MAC address and auto-increment integer to uniquely identify the MA’s result in the central server. Since it is impossible to auto-increment a text (varchar), before a new row is inserted into the Traceroute table, a trigger auto-increment the integer ID in the Traceroute_Seq table and concatenate it with the Ethernet MAC address of the MA. For example, if the last inserted ID in the Traceroute_Seq and Traceroute was 9 and b8:27:eb:16:7d:a6-9, in the next traceroute probe before the results inserted the ID in the Traceroute_Seq table incremented by one and the ID in the Traceroute table becomes b8:27:eb:16:7d:a6-10.

19

4.5.2 Measurement Agent design II We don’t want to limit users to store the measurement results only on mysql. It is a good approach to let users store their results in any database or other file formats. CheesePi uses Grafana, an open source application for visualizing large-scale metrics data, as a dashboard for displaying the measurement results. Currently, Grafana supports a few time series databases (Graphite, InFluxDB, and Open TSDB). Even though InFluxDB is a schemaless time series database, which is incapable of retrieving results from multiple tables with a single query, there is no need of complicated queries for displaying the results in the dashboard. So, the CheesePi project decided to use InFluxDB as a default database for storing the measurement results, and a user has a decent looking (and editable) dashboard. The attributes (fields) in all the performance measurement tools are similar to the MA design I (MySQL).

Figure 4.4: The CheesePi Dashboard

4.5.3 The central server design Most of the measurement analytics are done in the central server, so it is important to design the database carefully in order to create relations among tables so that data can be retrieved with relatively simple efficient SQL queries. For example, a query might want to retrieve the number of users in Stockholm who experience a network delay (RTT) value greater than 250 ms during measurements to www.bbc.com on 20th of May 2015 between 13:00-14:00 UTC. So we need a number of tables which are related by key fields to get this information.

20

Figure 4.5: Central server database design

The tables are PI, User, Connection, Operation, Ping, Traceroute, Traceroute_Hop, and Httping. The PI table stores the Pi ID and its Ethernet MAC address to identify the Pi (MA) uniquely. The User table stores the Pi ID, email address of the user, and alias (temporary name) to identify the user of each MA, as a user can have multiple MAs. Another table is a Connection table, which stores the connection ID, Pi ID, and MAC address. During measurements the MA can use different access technologies, either WiFi or Ethernet, and the connection table identifies the connection type during the measurements. We need to maintain which operation was run, which tool was used during the measurement, the connection type, the start and end time and the success/failure, and the Operation table is responsible to storie such information. The ping, httping, and traceroute tables in the central server have similar design to the tables in the MA’s, except the additional Pi ID and Op ID attributes. We want CheesePi to be robust and scalable, the MA was built to store the measurement data in many formats. As one can see in Figure 4.1, the MA can store the measurement results in any format based on user’s choice, but in the central server the collector will organize the data in a format optimised for complicated queries.

4.6 Sampling of network packets In passive monitoring real data is used to measure the performance of the network, while active monitoring generates its own data for measurement. Ideally, the sampled measurement probes, the packet size and the frequency of the measurements, should represent the real user experience. But CheesePi should not

21

disturb the browsing experience of the user, so active sampling should be as lightweight as possible, while still gathering useful data. As was mentioned in section 4.2 an experiment was conducted to choose the minimum packet size for network performance measurement with a minimum extra load on the network. The result shows that no significant difference in the delay and packet loss for the different chosen packet sizes, which are the key metrics of CheesePi for network performance measurement. So the default packet size for a network (ping) measurement in CheesePi is 64 Bytes, but the user can change the default size. In the HTTP measurement it is not possible to select the packet size. The only alternatives are the HEAD and GET HTTP requests. If the httping requests an HTTP GET, the complete page/file is returned. However, the main purpose of using httping in this project is to measure the latency and packet loss during the connection, not the throughput of the web server. So head is used as a default HTTP request option since its packet size is less than the get one.

4.7 Measurement frequency scheduling CheesePi runs the user's Internet performance measurement periodically. The frequency of the measurements and the sampling duration is different for different applications, so it is important to consider what can happen within the measurement duration and in the interval between the two successive measurements. In CheesePi's case, the system is trying to measure the general trend of network performance at the Transport, Network and Link layers of the TCP/IP stack. If the interval between two successive measurements is large, it is possible to miss important performance information, and it is also possible to disturb the user’s quality of experience if the interval is small. We conducted an experiment in order to choose a measurement schedule that balances the capturing of the connection behaviour and not disturbing the user's’ Internet connection. As one can see in Table 4.1 and Table 4.2 First we ran a continuous one hour network measurement of 3600 probes, and the interval between each probe was one second (1 X 3600). This is good to capture the connection behaviour but such large probes can disturb the user’s connection experience. We decided to minimize the number of the number of probes, so the next step is comparing the 60 probes of every minute (first minute, second minute, third minute and so on) to sample the network behaviour by smaller probes. And as one can see in Table 4.1, the results show the average network delay and packet loss of every minute (60 probes) in a five minute measurement is almost similar to the average five minute (300 probes), which is around 32 ms delay and 0% packet loss. Then we decide to represent the five minute network performance by one minute

22

measurement. But if the measurement is to several different targets, still the number of probes can influence the user experience. So, is it possible to sample the network behaviour with less number of probes? We took the first ten probes of every minute measurement and we compare them with the average five minute measurement, and can be seen in Table 4.2. The average delay and packet loss of the ten probes of every minute is almost the same (around 32 ms and 0%) to the average delay and packet loss of the total five minute measurement (300 probes). At the end we decided upon ten probes at an interval of one per second to sample the network performance over a 5 minute interval. The path of the network doesn’t change often and we decided to only run the path (traceroute) measurement once every 15 minutes.

Average ping RTT (ms) Ping packet loss (%)

1 hour (3600 probes) 32.42 0

first 5 minutes (300 probes) 32.37 0

first minute (60 probes) 32.38 0

second minute (60 probes) 32.36 0

third minute (60 probes) 32.34 0

fourth minute (60 probes) 32.39 0

fifth minute (60 probes) 32.36 0

Table 4.1: Network measurement from an MA in SICS lab to www.bbc.com in different time intervals.

23

first 10 probes of every minute

Average Ping RTT (ms) Ping Packet loss (%)

first minute (10 probes) 32.32 0

second minute (10 probes)

32.30 0

third minute (10 probes) 32.33 0

fourth minute (10 probes) 32.48 0

fifth minute (10 probes) 32.32 0

Table 4.2: First few probes of the first five minutes network measurement from an MA in SICS lab to www.bbc.com

4.8 Data analysis tools In this master thesis we used different Python packages for plotting and analysis. For example, we used numpy package for analysis the results, and matplotlib for plotting the graphs. All the latency vs. time graphs in this report are smoothed by a window size of 50. 4.9 Measurement agent's location All the measurement for this master’s thesis were conducted in Stockholm. Most of the experiments were done at the SICS lab and student apartments in Kista Galleria and Lappis. The Internet provider in the student apartments and SICS are Bredbandsbolaget and SUNET respectively.

Measurement Agent Connection Type Location

rpi1 Ethernet SICS lab

rpi2 Ethernet SICS lab

rpi3 WiFi SICS lab

rpi4 WiFi Student Accommodation (Kista)

rpi5 Ethernet Student Accommodation (Lappis)

rpi6 Ethernet Student Accommodation (Kista)

rpi7 Mobile Data (3G) Home (Stockholm)

Table 4.3 Measurement agent's location and Internet connection type.

24

25

5. Network Performance Evaluation This chapter will discuss the motivations behind the conducted measurements. The measurement results gathered from several locations in Stockholm are also going to be discussed. We are also going to look how the Internet performance of home users behaves during big media events.

5.1 Measurement motivation In order to determine the source of a user’s Internet performance degradation, the quality measurement should consider different access technologies (cable, WiFi, mobile data) allowing us to determine their differing performance characteristics. We can then identify if the performance problems are due to the access technology. As an example, we can measure the Internet performance of two neighbors who have the same service provider with similar service level agreements, but one of them uses a WiFi router to access the Internet and the other one is directly connected through Ethernet. Suppose the network and HTTP measurements are taken from the neighbours at the same time, if the delay and packet loss rate is high in the user who uses WiFi technology, then the problem can be narrowed down to the WiFi network. Further diagnosis such as channel interference, speed limit of the router, contention and so on can be a separate measurement effort. Another approach is to determine the Internet performance of users while targeting different server destinations. In this case, different popular media streaming websites (servers) are chosen as a destination landmarks for measurement from our MAs. The target server can influence the user’s Internet performance due to several reasons:

● The distance (network topological) between the MA and the target server also. The more the MA is closer (less number of hops) to the target the better the performance.

● The technology used by the server (Single server or CDN). If the server uses CDN technology, its performance will probably be more stable. But a single server’s performance will be highly influenced by a large number of visitors.

● The memory and processing capacity of the target server also has an influence on the performance. The better processing capacity the server has the better performance can be obtained.

It is also good to compare the measurements collected from different service providers. For example, the measurement results of two users from the same geographic location but with different service providers. We assume the target server for measurements use a CDN technology, both the users are connected to the Internet through Ethernet cable, and both of them are located in the same area. If one of the users has higher delay and packet loss rate compared to the other one, it

26

is highly probable the service provider to be the cause. During this master thesis we didn’t conduct experiment from home Internet users with different operators (service providers), but the service providers of SICS and the student accommodation are different operators. Finally, we want to observe, how the Internet performance or popular streaming servers behave during big media events: sports, music contests, etc. To see whether Internet users experience more during big sport games than any other normal days on video buffering (i.e. pauses). The streaming servers become very busy serving the huge number of customers. The performance experienced by the Internet user during such events depends on the network and how well the server handles the requests and what technology it supports. Later on this chapter, we are going to discuss on the measurement results obtained from some popular media events. 5.2 Major Media Events We were lucky during this master’s thesis, quite a few big media events happened such as: a boxing match between Mayweather and Pacquiao (‘Fight of the century’), Eurovision 2015, the Champions League final between Barcelona and Juventus, and an Open House event at SICS with more than 400 guests (results in appendix). Apart from SICS Open House, all the big media event measurements were not conducted with the standard CheesePi tools. As it was explained in section 4.7 we designed CheesePi to sample the network performance by conducting small number of measurements, but in the big media events the measurements were continuous with high frequency (1200 probes per hour), and we were improving the way of measurements based on the lesson learned from the previous media event. So we decided to conduct separate measurements for the big media events and at the end to import the results to CheesePi database (this might not be the ideal approach but I didn’t want to miss the measurements). Some of the more interesting results of the events will be presented in this section. 5.2.1 Boxing - Fight of the Century On the 3rd of May 2015 a boxing match between Mayweather and Pacquiao happened at Las Vegas. The match started at 04:00 UTC and it took 12 rounds, which lasted around one hour. The media calls the fight “Fight of the Century”, reports indicate that Mayweather vs. Pacquiao was the most-watched Pay Per View (PPV) event of all time [16]. For those who live in USA it was costing $89.95 to order "Mayweather vs. Pacquiao" and an additional $10 for the high-definition (HD) experience [17]. In such matches major providers struggle to handle the load of last minute demand, and this was a good opportunity for us to measure the network behaviour. After we heard about this event the next task was to choose which streaming servers

27

should be measured. We choose five landmarks among the listed streaming websites by a promotional article in The Guardian, and the reasons for selecting the landmarks can be explained as follows:

● www.hbo.com ● Home Box Office (HBO) was the organizer of the event. ● One of the top American premium cable and satellite television

network. ● www.sho.com

● Showtime (SHO) is one of the top American premium cable and satellite television network.

● www.philstar.com ● Pacquiao (one of the boxers) is from Philippines, and Philstar is a

Philippine news and entertainment portal for the Filipino global community

● my.sky.com ● UK's largest pay-TV broadcaster with 11 million customers as of 2015.

● www.viaplay.se ● This thesis is conducted in Stockholm, Sweden and Viaplay is a major

broadcaster in Sweden.

The measurement was conducted at Stockholm in a student apartment and SICS laboratory. The measurement in most of the MAs started around 2015-05-02 23:10 UTC and ended around 2015-05-03 21:43 UTC. The ping packet size was the default 64 bytes, but the size of the httping probes were different for all the landmarks based on server replies for the HTTP head request. During the measurement the httping packet size for each target was as follows:

● www.hbo.com (477 bytes) ● www.sho.com (297 bytes) ● www.philstar.com (569 bytes) ● my.sky.com (165 bytes) ● www.viaplay.se (286 bytes)

During this measurement, we were not checking whether the servers are CDN technology enabled or not. But after some event measurements we started to consider CDN as one of the selecting criteria for the target. We were using CDN Finder http://www.cdnplanet.com/tools/cdnfinder/ as a tool to check if the target uses CDN or not (i.e. single server). According to CDN finder and MaxMind https://www.maxmind.com/en/geoip-demo the location of the target servers and their CDN supportability is listed as follows:

28

● www.hbo.com is located in Boardman, Oregon, United States, North America,

and no CDN technology was found behind the hostname. ● www.sho.com located in Amsterdam, North Holland, Netherlands, Europe and

Akamai, a leading CDN service provider, was running behind the hostname. ● www.philstar.com located in Tempe, Arizona, United States, North America

and no CDN technology was found behind the hostname. ● my.sky.com located in Amsterdam, North Holland, Netherlands, Europe and

Akamai is running behind the hostname. ● www.viaplay.se located in Dublin, Dublin City, Leinster, Ireland, Europe and

no CDN technology was found behind the hostname. Among the five targets, the HBO and Viaplay servers were not responding to ping requests. So for those two servers, we only conduct httping measurement. It would have been desirable if we could have run ping tests specifically for HBO, since it was the organizer of the match , probably has the most customers (at least in the US).

Figure 5.1: HTTP latency to different targets from SICS Ethernet during the boxing match between Mayweather and Pacquiao. Except for HBO, the httping latency from SICS Ethernet to all the targets did not behave differently during the match (4-5 AM). The average httping RTT of a probe from SICS Ethernet to HBO before 04:27 AM was around 461.6 ms, but after 04:27 till the end of the measurement there were only 21 probes with RTT of less than 500 ms and the average RTT was 580.9 ms, and this is clearly shown in Figure 5.1. This latency increase is observed in all MAs. As we can see in table 5.1 the average httping latency to HBO was 479, 582, 515 ms in SICS Ethernet, student apartment WiFi, and student apartment Ethernet respectively. After the match began, in all MAs

29

the latency started to increase at the same time. After the match began, the average httping latency increased by 21.6%, 22.5%, and 38.6% for SICS Ethernet, student apartment WiFi, and student apartment Ethernet respectively and it remains almost constant till the end of the measurement. This latency increase can be related to different causes. It could be due to a path change or a congested router on the path creating the delay. It can be also due to the delay created by load on the target server. It was my mistake not conducting traceroute measurement during this event, and this creates complexity in finding out the source of the cause.

RTT (ms) Before the match (21:00 - 04:00)

RTT (ms) During the match (04:00 -05:00)

RTT (ms) After the match (05:00 -22:00)

Academic environment (SICS, Ethernet)

478 ± 86 582 ± 26 581 ± 65

Student apartment (WiFi)

582 ± 364 713.37 ± 86 717 ± 276

Student apartment (Ethernet)

515 ± 141 714 ± 22 714 ± 104

Table 5.1: Httping latency to HBO from different MAs during boxing match between Mayweather and Pacquiao. In order to observe if Boxing had an effect on the obtained performance, I did the same measurement one week after the match , even though I repeat the same mistake by not including traceroute measurements. But as it is shown in Table 5.2 and Figure 5.2, unlike the average latency to HBO during the match day, the average latency to HBO one week later was almost similar in the time before, during, and after the match, and the latency was also less variable. This tells us the source of the latency increase that happened during the match day can be the packet filtering measurement taken by HBO, in order not to disturb the quality experience of the subscribers (It is a personal guess).

Latency (ms) Before the match (21:00-04:00)

Latency (ms) During the match (04:00-05:00)

Latency (ms) After the match (05:00 -22:00)

SICS Ethernet 498 ± 68 499 ± 28 493 ± 47 Table 5.2: HTTP latency to HBO from SICS Ethernet one week later after the boxing match between Mayweather and Pacquiao.

30

Figure 5.2: Httping latency variability to HBO from SICS Ethernet during and one week later after the boxing match between Mayweather and Pacquiao. The HTTP and network latency to my.sky.com and www.sho.com were much smaller compared to the other targets. A traceroute that taken after this measurement showed the SHO and Sky servers were located closer to the MAs, the number of hops in order to reach the destination was less compared to the other targets. The HTTP latency from SICS Ethernet to Philstar started behave differently around 14:00. The RTT value started to increase in the time between 14:38 and 18:00. Then later it stopped increasing and relatively stayed stable in the time between 18:00 and 20:28. Finally, the RTT value started to decrease till it reaches the average value before the change, and it happened between 20:28 and 21:28. Such behaviour is observed in all MAs, and as it is indicated in Figure 5.3, a similar behaviour is also reflected in the ping measurement. Since the behaviour is also reflected in the ping result, the increased latency was likely due to the network. The Philippines time zone is UTC+8, so the local time at which the latency started to increase was approximately 22:00, which may be the time people usually start to watch TV. But did this behaviour reflected one week after the match?

31

Figure 5.3: Network and HTTP latency to Philstar from SICS Ethernet. As it is shown in Table 5.3, unlike the Philstar latency results during the match the latency obtained one week later was relatively smaller and consistent, especially in the time between. So the latency increased to Philstar during the match day could be the congestion created on the links before the target server due to the replay of the match or there was some other live TV program.

Httping latency (ms) (23:00-14:00)

Httping latency (ms) (14:00- 22:00)

Ping latency (ms) (23:00-14:00)

Ping latency (ms) (14:00- 22:00)

During the match

374 ± 87 484 ± 155 175 ± 8 227 ± 37

One week later

352 ± 37 352 ± 48 170 ± 0.5 170 ± 0.7

Table 5.3: Latency to Philstar from SICS Ethernet during and one week later after the boxing match between Mayweather and Pacquiao. The measurement results for HBO in Table 5.1 showed that the latency before the match from the Student apartment connected by a WiFi is higher and greater variance than the latency from the MA in student apartment connected via a cable. And it is similar for all the other targets. The laptops and smartphones of the two students who live in the apartment were connected to the Internet through WiFi. The contention and channel interference in the WiFi network likely created a higher latency compared to the Ethernet connection.

32

HBO SHO Sky Philstar ViaPlay

SICS Ethernet

544 ± 87 79 ± 130 24 ± 62 418 ± 128 113 ± 70

Student appr. Ethernet

649 ± 148 75 ± 105 19 ± 86 351 ± 227 105 ± 71

Student appr. WiFi

668 ± 313 108 ± 233 48 ± 221 No data collected

144 ± 232

Table 5.4: HTTP latency comparison among different MAs during boxing match between Mayweather and Pacquiao. In the Boxing event we observed, the CDN enabled streaming websites are rarely affected by the match. While those which are not CDN enabled, showed different characteristics during the match day. The MA from the student Ethernet experienced better latency than the MA connected by WiFi. The network latency from the wirelessly connected MA was 22.8% larger than the latency of the MA connected by cable, and also less variable. 5.2.2 Eurovision 2015 The next big event we found was Eurovision song contest. Eurovision is the longest running annual TV song competition held, primarily, among the member countries of the European Broadcasting Union (EBU) since 1956. It is also one of the most watched non-sporting events in the world. For example Eurovision 2015 was seen by around 197 million viewers [18]. Vienna was the host of Eurovision 2015 and the contest was held in 3 stages (semi-final one, semi-final two, and grand final) on 19, 21 and 23 of May 2015, respectively. In this event, we measured from different places in order to get a variety of locations and access technologies. One good aspect of this event was the three stages of the contest, we got a chance to adapt and rethink the measurement approach for the grand final contest. For example, the HTTP was crashing during the continuous high frequency measurements in the first and second semi-final contests, and so were were able to follow a better approach for the final contest. 5.2.2.1 Semi final one The Eurovision 2015 semi-final one was on 19th of May 2015. It started at 19:00 UTC and ended at 21:00 UTC. In most of the MAs the measurements started and ended at 17:00 and 23:59 UTC respectively. As it was mentioned before the measurement was conducted from different places (team members apartment and SICS lab) and the event was not during the working hours. So there was a bit

33

difference in the starting time of the measurement, but it does not affect the analysis since all measurement nodes cover the whole events. Choosing targets for the measurement is tricky because it was not easy to identify which website users would use to watch the event. We tried to look for popular streaming websites of many participant countries, and also asked friends to get suggestions and check the popularity of the chosen targets. The semi-final one measurement targets were the following:

● www.eurovision.tv - a European broadcasting union’s streaming website for Eurovision. We choose this target due to its popularity and it was also an organizer of the event.

● www.svtplay.se - SVT (Sveriges Television AB), a Swedish public TV broadcaster. The measurement is conducted in Stockholm (Sweden), and Eurovision is very popular in Sweden. So we decide to include a popular streaming website which broadcast Eurovision in our targets.

● tvthek.orf.at - Österreichischer Rundfunk (ORF) is an Austrian national public broadcaster. The reason we chose the target was: its dominance in Austrian broadcasting media and Austria was also the host of the event.

In order to compare and analyse the results, it is good to know the location of the target servers and the technology they support. The location of the target servers and its CDN technology supportability was as follows:

● www.eurovision.tv - CloudFlare, one of the top CDN providers, distributes the contents of the target. The target server during the measurement was located in Stockholm, Sweden.

● www.svtplay.se - Akamai distributes the content of SVT and the target server during the event was located in Stockholm, Sweden.

● tvthek.orf.at - According to CDN Finder and MaxMind, ORF does not use CDN technology to distribute its content and it is located in Austria.

The ping and httping were running in parallel and the interval between each probe was three seconds. The packet size of a ping probe for each target was 64 bytes, and 479, 206, and 477 bytes for each httping probe of EurovisionTV, SVT, and ORF respectively.

34

Figure 5.4: HTTP latency to different targets from an MA at SICS (Ethernet) during Eurovision 2015 semi-final one. From the previous boxing event, we observed that performance from the Ethernet connection was better than the performance from the WiFi connection. So we decided to first start looking at the results for each target from the MA at SICS lab connected to the Internet via a cable. As it is shown in Figure 5.4 and Table 5.5 the HTTP delay of SVT was consistently small between 6 to 8 ms with a small variance but there were very few large ( > 205 ms ) delays during the contest in the time between 20:00 and 21:00, and after 23:00. The latency of ORF and EurovisionTV vary a lot especially ORF. From the SVT delay results we can understand that the CDN server performance was good, there was no congestion in the path between the MA and the target. Next we are going to look at EurovisionTV and ORF results.

EurovisionTV SVT ORF

Mean Median

Stdv. Mean

Median

Stdv.

Mean

Median

Stdv.

Before Program (17:00 - 19:00)

23.87 8.44 203 7.83 6.43 5.68 181 94 865

During Program

23.75 8.31 241 17 8.09 205 147 94 591

35

(19:00 - 21:00)

After Program (21:00 - 00:00)

8.35 7.97 1.7 11 7.84 93 370 93 1772

Whole Measurement (17:00 - 00:00)

17.21 8.17 169 12 7.67 125 251 94 1287

Table 5.5 Comparison of HTTP latency from SICS Ethernet to different targets before, during, and after the program of Eurovision semi-final one.

Figure 5.5: HTTP latency from a MA at SICS (Ethernet) to EurovisionTV during Eurovision 2015 semi-final one. The HTTP delay of EurovisionTV from SICS Ethernet started to vary many times from 17:20 till the end of the contest (21:00) as it is clearly shown in Figure 5.5 and Table 5.5. This tells the event affected the HTTP delay to EurovisionTV to some extent. But we can not be certain the increase in the RTT value was due to the delay created by the server or traffic created on the network. So to get an idea about the causes we should compare the network delay with the HTTP delay (network and server’s computation latency).

36

Figure 5.6: Comparison of Network and HTTP delays from a MA at SICS (Ethernet) to EurovisionTV during Eurovision 2015 semi-final one. The network delay was 1.84 ms with a tiny variation of 0.53 ms, as it is clearly shown in Figure 5.6 the delay was small and consistent. The larger network delays mostly happened a few minutes after the contest started with peaks 44.44% larger than the average network delay during the whole measurement. While the HTTP delay was varying before and during the contest. As it is shown in Figure 5.6 and Table 5.5 the deviation before and during the contest was around 10 times the average delay. Many of the peaks were more than five times the average latency of the httping measurement to the target. So the results indicate the large HTTP delays were due to the delay created in the server. But what about the large HTTP delay variance to ORF?

37

Figure 5.7: HTTP latency from a MA at SICS (Ethernet) to ORF during Eurovision 2015 semi-final one. There are many servers which do not respond to a ping request, and ORF’s server is one of these servers. It is difficult to draw a conclusion based only on HTTP measurements. But since httping results are the performance of the whole TCP/IP stack, it is possible to have a general understanding of the connection’s performance. As one can see in Figure 5.7 the HTTP delay to ORF are peculiar. It is common for the latency to increase and fluctuate a lot while watching a big media event from a website which uses a single server to handle all the user requests. But the average HTTP latency (147 ms) and its variance (591) during the contest was relatively better compared to the latency and variance before (181 ms, 865 ms) and after (370 ms, 1772 ms) the contest. When we looked the path based measurement results to find the cause, we found that, the IP address of the target was keep changing between 194.232.116.172-176. During this measurement we did not include “-r” parameter in the httping command, which makes it only resolve the hostname once at the start. This tells we were measuring five different ORF servers and this might be the cause of the frequent and large latency variation. So ORF was using several servers for load balancing to handle user request, but still it does not save users from experiencing larger delays for some moments.

38

Figure 5.8: Comparison of HTTP latency to EurovisionTV from different MAs during Eurovision 2015 semi-final one. We compared the latency experienced by different MAs. The HTTP delay to SVT and ORF experienced by the MAs in SICS lab and the student accommodations is more or less similar except some higher delays for the wirelessly connected MAs. But as one can see in Figure 5.8 the HTTP latency experienced by SICS lab and the student accommodations is different. The middle subplot of Figure 5.8 shows, the HTTP delay from both the students apartment have a similar pattern and an average latency. This is due to both MAs served by the same operator, and after the second hop the packets follow a similar path to reach the target. However, there is a big latency difference between the results obtained from SICS lab and the student apartment. Even though the traceroute shows both of them have almost similar number of hops, SICS uses Nordnet network which is directly connected to the designated edge CDN server (Stockholm), whereas due to the students accommodation ISP routing policy, the packets from the student apartment were leaving Sweden network to Norway and came back to Sweden network. So the main reason for the large latency difference from the student apartment and SICS lab was the delay created by the path followed to reach the target. The next step was to look how the event affected the latency in the student apartment and comparing the latency between the WiFi and cable connection. The average HTTP delay to Eurovision from both the Ethernet and WiFi connected MAs in the time from the beginning of the measurement till the end for the contest (17:00 - 21:00) was 311 ± 446 and 324 ± 396 ms respectively. This latency is much higher and deviates more compare to the average HTTP delay experienced after the contest (21:00 - 00:00) by both the Ethernet and WiFi connected MAs, which is 90 ±

39

89 and 109 ± 133 ms respectively. This behaviour is clearly shown in Figure 5.8. Such pattern is also reflected in the network measurement, so this indicates there was higher traffic in the network during and few hours before the contest. We will see in the next contests if this latency variation was due to the Eurovision effect on the Telenor network that goes to the target. From the presented results the Ethernet connection in the student apartment was as bad as the WiFi in terms of delay and deviation. 5.2.2.2 Eurovision 2015 semi-final two The Eurovision 2015 semi-final two was held two days after the semi-final one on 21st of May 2015. It started at 19:00 UTC and ended at 21:00 UTC. In most of the MAs, the measurements started and ended at 16:05 and 23:05 UTC respectively. During this contest, we add one more MA which used 3G to connect to the Internet. All the targets were the same targets as the semi-final one.

Figure 5.9: Comparison of HTTP latency to SVT among different MAs during Eurovision semi-final two. Before the contest, the average HTTP delay to SVT from the student accommodation in Kista with an Ethernet connection was 16 ± 320 ms, the large deviation is due to a single large delay that happened in 18:19 which was around 19 seconds. The average HTTP delay of the SICS Ethernet was 15.4 ± 171 ms which is

40

almost similar to the student accommodation. But in the student accommodation during the contest at around 19:54 the HTTP delay increased by large amount and continued like that till 20:38, the average delay during that time was 215 ± 115 ms. As one can see in Figure 5.9, such behaviour was also observed in the WiFi connection of the student apartment. Similar case but at different time also happened another student accommodation in Lappis, which started at 19:12 and ended at 19:36. smooth except for some moments during the contest in rpi4, rpi5, and rpi6 (the MAs in student apartments). In all the MAs, the cause for the delay increase was a continuous 6 packet loss, and after that the target IP address changed and the latency increased by more than 15 times. The average HTTP delay from the 3G connection was three times than the Ethernet connection in the student apartment.

Figure 5.10: Comparison of HTTP latency to EurovisionTV among different MAs during Eurovision semi-final two. In the MAs at the student accommodation the HTTP delay to EurovisionTV started to increase when the contest was about to begin and it started decreasing after the end of the contest. And becomes relatively stable after 22:00, which is similar to Eurovision semi-final one. During the HTTP delay measurement to EurovisionTV, the MA with a 3G connection experienced a better average delay and deviation (85 ±

41

115 ms) compared to the MA at the student accommodation (148 ± 194 ms). This is clearly shown in Figure 5.10. The latency increase pattern observed in both Eurovision semi-final one and two in the measurements from the student accommodation to EurovisionTV will be analysed later in the next sections. 5.2.2.3 Eurovision 2015 Final The Eurovision 2015 grand final was held on 23rd of May 2015. The program started at 19:00 UTC and ended at 23:00 UTC. The voting started at 21:14 and ended at 21:40 UTC. The result announcement started at 21:52 and ended at 22:49 UTC. In most of the MAs, the measurements started and ended at 16:30 and 04:00 respectively. During this contest we wanted to make some changes in the targets due to ORF website was not streaming the grand final, and both EurovisionTV and SVT were using CDN technology. Thus we decided to remove ORF and add more targets, for example, SBS (Special Broadcasting Service), an Australian public broadcasting, and the BBC’s iPlayer. We added SBS because it was the first time for Australia to participate in Eurovision and SBS is one of the top broadcasting services in Australia. We dedicated a MA at SICS with an Ethernet connection, to measure the performance by probing to SBS, BBC iPlayer, and BBC UK. The other MAs were probing to SVT, BBC UK and EurovisionTV.

Figure 5.11: Network and HTTP latency of different targets form SICS Ethernet during Eurovision 2015 final.

42

As one can see in Figure 5.11, except the large HTTP delay (5042 ms) at 21:12, the network and the HTTP delay to BBC iPlayer from the SICS Ethernet was consistent, which is 9 ± 46 ms and 1.6 ± 0.3 ms average HTTP and network delays respectively. It was also similar for the SVT measurement but when the contest started (21:00 - 21:10) five large delays with an RTT value more than 5 seconds are recorded. This might be related to large number of users request, related with the beginning of the voting. The HTTP delay to EuroVision (8 ± 21 ms) was deviating less after 21:00 compared to the measurement before 21:00 (16 ± 158 ms).

Figure 5.12: Network and HTTP latency of different targets form SICS Ethernet during Eurovision 2015 final. The SBS server was located in Australia and the network and HTTP delay was large 356 ± 3 ms and 713 ± 182 ms respectively. Out of 12600 httping probes to SBS 90 of them recorded with more than 2000 ms delay. No special pattern was observed during the contest, except 3 probes with more than 5 second HTTP delay. But this large delay can affect the quality of different online applications. In the measurements to SBS we observed, the more the target is far from the client in the Network topology the more the performance degraded. We also compared the latency between BBC UK and the BBC iPlayer. The HTTP delay of BBC iPlayer (12 ± 111 ms) was smaller compared to BBC UK (78 ± 65 ms). The small delay of BBC

43

iPlayer is due to it uses Amazon CloudFront for distributing its content. This is clearly shown in Figure 5.12. As one can see in Figures 5.13 and 5.14 the HTTP delay to BBC iPlayer and EurovisionTV from the student apartment is higher than the HTTP delay from SICS lab. As we previously discussed in Eurovision semi-final one, it is due to the routing policy of the service providers. In the SICS network the DNS server returns the Amazon CloudFront server nearest to the MA and the packets were reaching the target after few hops. However the DNS server for the student accommodation was returning different Amazon CloudFront server, which is located in Amsterdam (according to the DB-IP Database) which can not be reached close to the MA. From this delay comparison we can understand that, the routing policy of the service provider or its agreement with CDN providers affects the user’s Internet performance.

Figure 5.13: Comparison of HTTP delay to EurovisionTV from different MAs during Eurovision 2015 final. However the HTTP delay to SVT from the student apartment (16 ± 102 ms) is similar to the HTTP delay from SICS lab (14 ± 171 ms). In this case, even though both the DNS servers in the student accommodation and SICS network return different CDN servers and did not share common path to reach the target, the probes from both networks are able to reach the target quickly. This might be due to 108,507 Akamai servers deployed worldwide in over 110 countries and within more than 1,400

44

networks around the world [35]. The distribution of the CDN servers responsible for distributing the streaming websites content and the agreement between the ISP (operator) and the CDN provider influence the Internet performance of a user.

Figure 5.14: Comparison of HTTP delay to BBC iPlayer from different MAs during Eurovision 2015 final. 5.2.2.4 One Week After Eurovision 2015 Final In order to understand the effects of the big media event on the performance of the Internet quality, we repeated the experiment exactly one week after the Eurovision 2015 final to observe if some of the performance changes during the measurement were due to the media event. The time the measurement conducted, the places, the targets, and the frequency of the measurement was the same as the Eurovision 2015 final. The red graphs in the top part of Figure 5.15 are the HTTP delay of different targets from the SICS Ethernet connection during Eurovision 2015 final, while the bottom measurement graphs in blue are the HTTP delay to the targets after one week. During the Eurovision Final the average HTTP delay to SVT was (14 ± 171 ms), the HTTP delay of 87.12% of the probes were less than 10 ms, and for the measurement conducted one week after 87.85% of the probes recorded less than 10 ms HTTP delay. There were few peak points observed around the beginning of the program (19:00) and voting time (21:15) which did not happen in the measurement conducted after one week. So the graph tells us the large latency of the few probes

45

may indeed be due to the load on the web server or the link during large number of users around the mentioned time. But the average latency in both measurement is almost similar, which is 14 ± 171 ms and 11 ± 60 ms for the Eurovision Final and the one week after measurements respectively .

Figure 5.15: Comparison of HTTP delay from SICS Ethernet to different targets during Eurovision 2015 final and one week after the final. The average HTTP delay to BBC iPlayer from SICS Ethernet during the Eurovision Final was 10 ± 46 ms. However, when the voting was about to start there was a peak with a latency around 250 ms and we initially thought it was due to an increase in number of visitors at that time. In the measurement conducted one week later, until 18:07 the average HTTP delay was 11.9 ms but suddenly it increased to an average of 220 ms and stayed for about 2 hours till 20:03. We try to analyse the reason behind the sudden latency increase. First we checked if it was due to continuous packet drop or a target IP address change, but there was no packet loss and the IP was changing frequently before the latency started to increase. When we checked the traceroute, the path was changed at the moment the latency started to increase. During the smaller latency the httping packets were reaching the target after few hops with a very small RTT. However during the higher latency period the number of hops increased because it was following a different route. When the path to the target returned to the previous path the latency decreased closer to its previous average delay (26 ms). This artifact is clearly shown in the bottom right plot of Figure 5.15. The cause of the sudden latency increase may be due to target IP address

46

changing its load balancing. The new path for reaching the target requires more time and affects the end to end delay. The httping latency from SICS Ethernet to EurovisionTV during the Eurovision Final has fewer peak points than the experiment conducted one week after the final. For example, in the measurement during the Eurovision Final there were 29 HTTP probes recorded with delay greater than 100 ms, whereas one week later there were 60 probes with HTTP delay of more than 100 ms. During the Eurovision Final measurement we thought the high latencies were due to users visiting EurovisionTV to check the programs and watch the contest at the time before the contest started and as the program started. But the measurement that conducted one week after Eurovision Final shows the high latency were probably not due to the event. The high latency in the time between 16:30 and 22:00 can be due to network congestion or the CDN server is relatively busy everyday during the given time interval. In order to determine the cause of the behaviour we should look at the ping latency from SICS to EurovisionTV and other MAs’ httping latency. This helps to check if the behaviour (pattern) is due to the characteristics of the network, or the target server.

(a) From SICS Ethernet.

(b) From Student apartment WiFi.

Figure 5.16: Comparison of HTTP latency to EuroVisionTV from student apartment WiFi and SICS Ethernet between the measurements during Eurovision Final and one week after the final. As one can see in Figure 5.16 (a), the network latency to Eurovision TV from SICS Ethernet during Eurovision final and one week later showed low variation, as the coefficient of variation (the standard deviation / mean) is less than 1, a thumb rule for the variation in dimensionless values, given the measured values of 1.77 ± 0.4 ms and 1.72 ± 0.28 ms respectively. So it is unlikely the network was the cause of the

47

previously observed large httping delays. But if we look at HTTP latency from student apartment (WiFi) to EurovisionTV, in both measurements the latency is higher in the time between 16:30 to 22:00. As it is clearly shown in Figure 5.17 this behaviour is observed in all EuroVision measurements. So this higher latency in the time between 16:30 and 22:00 could be due to the load on the CDN server everyday during the specified time, or it may give a priority to other tasks instead of HTTP head request packets.

Figure 5.17: HTTP latency from student apartment to EurovisionTV during different measurements. 5.2.3 2015 UEFA Champions League Final The UEFA Champions League, known simply as the Champions League, is an annual continental club football competition organised by the Union of European Football Associations (UEFA) and contested by top-division European clubs. The 2015 UEFA Champions League final was the final match of the 2014–15 UEFA Champions League, and it was played at the Olympiastadion in Berlin , Germany, on 6 June 2015, between the Italian side Juventus and the Spanish side Barcelona. A massive global audience on TV and social media confirmed the UEFA Champions League final's status as the world's most watched annual sporting event. We thought this big media event would create a behavioural change in the Internet quality of users, and we conducted measurements during the event to observe how it affect the Internet performance of users.

48

It started at 18:45 UTC and ended at 20:45 UTC. And the trophy ceremony ended at 21:00 UTC. In most of the MAs the measurements started and ended at 16:00 and 04:00 (the next day) respectively. From the previous measurements we learned targets which use a CDN technology are less affected by big media events. So during this measurement we tried to focus more on popular free streaming targets which do not use CDN technology. And we select four targets: one subscription based websites, and three free streaming websites which do not use CDN technology. Unfortunately we realized later two of the free streaming targets were using CloudFlare to distribute their content. The chosen targets were the following:

● www.espnplayer.com - ESPN (Entertainment and Sports Programming Network) is a U.S.-based global cable and satellite television channel owned by ESPN Inc . We choose ESPN as our target because as of February 2015, ESPN has approximately 94,396,000 subscribers, and it was one of the Champions League final broadcasters. The crashing during most of the world cup 2014 watched live streaming [21] was another reason for choosing ESPN. It uses Akamai to distribute the content on large live events but during the Champions League final the target server was not an Akamai server. The server was located in USA.

● www.cricfree.sx - CRiCFREE is a popular free sport streaming portal website. According to [20] it has millions of visitors every month. We chose the target to observe how free streaming websites handle user requests during big media events. It uses CloudFlare to distribute its content and the closest server was in UK.

● www.crichd.in - CricHD is also a poplular free sport streaming portal. After we conducted the measurement, we realized it is another CRiCFREE with different domain name with a small changes in the front page of the website.

● www.atdhe.ws - ATDHE is a another popular free sport streaming portal. It doesn’t use a CDN technology to distribute the content and the target server is located around Netherlands.

In some of the previous httping measurements we had faced difficulties with some conclusions due to frequent IP address change of the target. During this measurement we include the “resolve hostname once” parameter in the httping command, so the measurement was only to one web server. This is helps the measurement to be more stable when compared to a measurement to a hostname with several IP addresses for load balancing.

Based on the knowledge of networking and communication systems, when large number of users visit a specific server at the same time, the server becomes busy handling all the requests and creates high traffic over the link to the target. So during a big media events we expect there will be higher latency and packet loss during the match. Now let’s look at the result obtained from different MAs during the match.

49

First we should see how the Champions League final affected the performance measurement to ATDHE. Before we look on all MAs, let’s look the results collected from SICS Ethernet. Figure 5.18 shows the latency of the HTTP and the network, and its relations to the ping packet loss. The red line graph in the figure represents the HTTP latency, the blue graph in the middle subplot represents the network latency, and the green graph in the bottom subplot of the figure represents the network packet loss rate in every five minute across the whole measurement. The HTTP delay was 94 ± 19 ms. However when the match was about to start at 18:22, the HTTP delay variance increased (104 ± 84 ms) and continued until 20:30. The coefficient of variance for these two events are 0.2 and 0.8 respectively, which again captures, somewhat simply, the change in the experienced network service. This was just before and during the match, which is similar to our expectation. Then it went back to its previous “state” (94 ± 36 ms), and there were few spikes between 21:30 and 22:30 as one can see in Figure 5.18.

Now, let’s look at the ping latency and packet loss to try and understand the cause of the latency (change). A network latency of 44.7 ± 0.9 ms throughout the whole measurement phase except for four probes with delays between 100 and 104 ms, could be taken as “stable”. There were also few ping packet losses throughout the whole measurement. From the results we understand the latency increase during the match are due to the latency created by the server, because the behaviour observed in HTTP delay measurement was not reflected in the network measurements. So this can be due to large number of visitors to the ATDHE website during the Champions League final, and this load on the single server created the HTTP delay increase. This artifact is clearly shown in the top subplot of Figure 5.18.

50

Figure 5.18: Network, HTTP, and network packet loss relation of measurement from SICS Ethernet to ATDHE during UEFA 2015 Champions League Final. From the previous measurements we observed that some behavioural patterns observed only on specific Networks (MAs). So let’s look if the HTTP delay to ATDHE was larger during the match in all other MAs. As it is shown in the blue graph of the top subplot in Figure 5.19, the SICS WiFi also experienced higher latencies (110 ± 142 ms) around the time the match took place from 18:35 - 20:45. Whereas, the average HTTP delay before the match (16:00 - 18:35) was (96 ± 28 ms). The WiFi and Ethernet connection of the student apartment also experienced higher httping delay during the match but there were also latency spikes every few minutes. During the measurement to ATDHE, the student apartment experienced lower HTTP delay (59 ± 93 ms) compared to the delay from SICS lab (96 ± 48 ms). According to the traceroute results, the student apartment packets were passing through fewer hops to reach ATDHE server compared to the packets from SICS. But as it is shown in the middle and bottom subplots of Figure 5.19 the HTTP delay from the student apartment was deviating more both in the WiFi and the Ethernet environments, especially during the match. For example, during time 18:35 - 21:00 the HTTP delay was 79 ± 87 ms and 65 ± 118 ms for the WiFi and Ethernet respectively. We doubt this latency variation was due to the network congestion on the path, and thus we decided to look the latency of ping from the student apartment and SICS lab.

51

Figure 5.19: HTTP latency of different MAs to ATDHE during UEFA 2015 Champions

League Final.

The first subplot of Figure 5.20 shows the network latency from SICS WiFi and Ethernet to ATDHE. The network latency of the Ethernet was very smooth (44 ± 0.9 ms), while the latency from the WiFi had few larger delays (47 ± 10 ms). This tells us the latency on the network was very stable while there were some delay increase during the match and due to the WiFi. The network results tell us, the larger HTTP delay during the match was not due the network congestion, it is due to the load on the single ATDHE server. As it is shown in the middle and bottom subplots of Figure 5.20 the student apartment experienced less ping latency than SICS but there were a lot of spikes throughout the whole measurement, especially from the WiFi. These spikes from the WiFi might be due to high wireless-medium contention. Two students were watching the match and a movie using wirelessly connected laptops, and there were also two other smartphones connected to the Internet through the WiFi. The contention among wireless devices and the channel interference from neighbor routers, who use the same channel as the WiFi router of the student apartment, could be the cause of the spikes.

52

Figure 5.20: Network latency of different MAs to ATDHE during UEFA 2015 Champions League Final. We have examined how the Champions League Final affected the measurement performance to a popular server which does not use CDN technology to distribute its content. What about the targets which use CDN technology to distribute their content and ESPN (supposed to use a CDN)?

Figure 5.21: Httping latency of different MAs to CricHD during UEFA 2015 Champions League Final.

53

Even though CricHD uses CDN technology the average HTTP delay from every MA was more than 200 ms. This is shown in all the subplots of Figure 5.21. The CDN server, which was handling the HTTP request to CricHD, was located in UK and the average network (ping) delay from all the MAs in Stockholm was around 40 ms. We did not expect a big difference among the ping and the httping latencies since CricHD uses CDN (CloudFlare), however, the average latency was 160 ms more. These httping latencies were consistent and never went below 180 ms for any of the MAs. It can’t be also due to the packet size (326 Bytes), because we tried the ping measurement with different packet size up to 526 Bytes but the difference was not significant. It seems there may have been some load balancing or reverse proxy behind the CDN server that makes the HTTP delay this much larger. Beside the consistent large HTTP latency from SICS to CricHD there were also some spikes a few minutes before and during the match. While the latency from student apartment to CricHD was higher (301 ± 461 ms) during the match (18:35 - 21:00) compared to the average HTTP delay (220 ± 363 ms). We try to figure out if the latency increase is due to the network or the target server. As it is shown in the middle and bottom blue subplots of Figure 5.22 the ping latency to CricHD from the student apartment started to increase at around 17:00 and became higher (66 ± 13 ms) between 18:35 and 21:20 (during the match) compared to the average network delay (42 ± 17 ms) and the delay after 22:55 (28 ± 1.5 ms). This tells the larger delay observed in the HTTP performance measurements are influenced by the network performance. Then we tried to find which network link is creating the latency by looking the traceroute results. The router in Norway “tiscali-1.ti.telenor.net” was creating these larger delays. It might be because the router was busy at that time or there were some priority to other packets. When the latency to the “tiscali-1.ti.telenor.net” started to decrease and later became stable, the ping latency to CricHD also became stable. So it seems a specific router on the path can strongly influence the end-to-end delay.

54

Figure 5.22: Network latency of different MAs to CricHD during UEFA 2015 Champions League Final. Some unofficial video stream portals use different domain names with a small front page, designed to keep streaming if the main domain of the website is seized by intellectual property protection organizations. In our case, CRICFREE and CricHD looked the same, and providing similar services with a slight difference on their front page. We also trace the route to both targets, and they follow the same path till the last hop to the target. CRICFREE also uses CloudFlare to distribute the content, and the closest server to our MAs was located in the UK similar to CricHD. As it is shown in Figure 5.23 the general behaviours observed during httping and ping measurements to CricHD also happened while measuring to CRICFREE. However, the average HTTP delay to CRICFREE was smaller compared to the CricHD HTTP delay. For example the average HTTP delay to CRICFREE from student apartment of Ethernet connection (147 ± 167 ms) was smaller than the average HTTP delay to CricHD (220 ± 363 ms). In most of the MAs, the average HTTP delay to CRICFREE was less than 150 ms. This indicates, the delay created in the CRICFREE server was smaller compared to the delay created by CricHD server.

55

Figure 5.23: HTTP delay of different MAs to CRICFREE during UEFA 2015 Champions League Final. Even though ESPN is one of the top on-demand streaming services, the HTTP and the network delays were large. The HTTP and network delays from all MAs were very similar, and it was around 235 and 112 ms for the HTTP and the network respectively. Figure 5.24 shows, the HTTP delay from the SICS Ethernet and WiFi connections were 231 ± 70 ms and 237 ± 180 ms respectively. And in the student accommodation the HTTP delay was 229 ± 87 ms and 246 ± 101 ms from the Ethernet and WiFi connections respectively. As one can observe, the deviation was larger in the wirelessly connected MAs. In the student apartment WiFi there were continuous periodic spikes. We tried to find out the causes of the periodic spikes, and we tried to compare the HTTP and network latencies. As it is clearly shown in Figure 5.25 the periodic latency increase is in both ping and httping. This means, something was periodically happening in the network. In order to find the cause we have to look the ping results carefully. When we looked the ping results the average network delay was around 113 ms before the periodic increase, but every multiple of 5 minutes the delay increased. For example, the RTT was going constantly around 113 ms then at 5th minute it became 500 ms and return to its previous state, then again after multiples of 5 minutes it increases. We tried to examine the ping measurements to other targets from the student apartment WiFi. And the behaviour is observed in all ping measurement of different targets. We did not observe such a behaviour in the results from the Ethernet connection. So this periodical delay increase may be due to the periodic

56

PSM (Power Saving Mode) of the WiFi dongle in the MA. When using PSM the wireless interface is put in a sleep mode when the device is not transmitting a data. If the Access Point (WiFi router) receives data of the client while the client was in a sleep mode, it buffers the data until it receives the next beacon from the client. So such periodic WiFi interface sleep mode of the MA can be the cause of the periodic large RTTs. And in a Raspberry Pi the PSM is enabled by a default.

Figure 5.24: HTTP latency of different MAs to ESPN during UEFA 2015 Champions League Final. One can only watch the match on ESPN if one is a subscribed customer. We think the measurements to ESPN may not represent the traffic during the match at that time. Because, our target was “ www.espnpalyer.com”, which was located in US, but for a subscribed customer it would be different server located closely. So we can not say the performance observed on the measurements to ESPN during the Champions League Final are the performance experienced by the customers.

57

Figure 5.25: HTTP and network latency to ESPN from student apartment WiFi during UEFA 2015 Champions League Final.

58

6. Discussion WiFi experiences medium contention and so can be slower and more variable than Ethernet. Our measurement results confirmed and quantified customers with an Ethernet connection experienced lower latency and packet loss compared to WiFi users. In most of the measurements the SICS lab achieved better performance in terms of latency compared to the student apartment. SICS uses SUNET, which enables Swedish universities and institutions to connect with a well-developed and effective national and international data communication and related services. And the student accommodation uses Bredbandsbolaget ISP owned by Telenor. Before the large media event measurement results were analyzed, we expected that users who connect to a website on a CDN would receive data with a lower delay than a non-CDN solution. However our experiments showed the effectiveness of CDN technology on minimizing the end to end latency depends on how close the server is located to the customer, and the functioning of the Internet service provider’s routing policy (or possibly the CDNs). If the designated edge server is very close (in number of hops and geographically), and the path chosen to reach the desired domain is short, the latency is low. But if the path chosen to reach the target is long, the latency will be highly dependent on the network status. Such behaviour is observed during measurement to EurovisionTV from the SICS lab and student apartment during Eurovision. During our httping experiments to a CDN host, we observed a situation where significant packet loss occurred when the IP address of the target domain changed to a different edge server. This happened during the second Eurovision semi-final, as part of the measurements from a student apartment to Swedish Television (SVT). After six contiguous packet losses the edge target changed, which required a more hops to be reached by the measurement agents (MA). The HTTP delay increased by more than 15 times, and all the ping packets were lost after the IP address change. The ping packets lost were due to the hostname only being resolved once at the beginning of the experiment. In the student accommodation (Kista), this large delay continued for more than 40 minutes until the target address returned to its previous (edge) CDN server. This show how changes made in the network have a great influence on the latency. The target IP address changed and caused communication to be routed to a different destination IP. Following the new path to communicate with the target may require more time and this affects the end to end delay. Such a similar case also happened during measurement to the BBC iPlayer, one week after the Eurovision final. And the average HTTP delay increased from 11.9 ms to 220 ms

59

and this continued for approximately two hours. We can only speculate at this point on the reasons, possibly technical failure, possibly load balancing, but it would require further studies. It is common to observe some patterns in streaming websites with a single server. For example, if we do an HTTP measurement to a website which streams a popular TV show (“Breaking Bad”) on Sunday between 21:00 - 21:50 using a single server, the HTTP delay on Sunday between 20:45 - 21:55 can be larger compared to the HTTP delays on a different time. The HTTP delay increased because of the greater load due to a large number of visitors during that time. But most of the popular streaming websites, which use CDNs, are capable of handling a large number of requests, and it is very rare to observe such delays. However during our measurement to EuroVisionTV which uses CDN, during the time between 16:30 and 22:00 in all Eurovision measurements a higher HTTP delay (for example, 143 ± 136 ms during Eurovision Final) was observed compared to the HTTP delay after 22:00 (for example, 64 ± 33 ms during Eurovision Final). The pattern is not reflected in the ping measurement and it is most likely cause by the server load. Even though It is difficult to precisely know the source of such a pattern since a measurement is not conducted in the server-side, but it can be due to the high load on the CDN server everyday in the specified time or it may give priority to other tasks instead of HTTP head request packets. One of the main purposes of CheesePi is to collect data for operators to troubleshoot the source of problems in a network. During measurement to CRICHD in 2015 UEFA Champions League Final, there was a latency increase before and during the match. By tracing back the results from all the tools the source of the problem was identified as a specific router or the link that connects it to neighboring routers. The delay increase can be related to the congestion created on the link or other some reasons, but the job of CheesePi is to collect and quantify the observations of the problem. It is not easy to measure the exact performance of the Internet due to its complexity, but by following smart approaches and using the right tools, measurements can characterise the performance experienced by a user. All the measurements conducted during this master’s thesis use ping, httping, and traceroute network tools to measure the performance of users’ Internet quality. However, it is difficult to make concrete conclusions on some situations based only on the results obtained from ping, httping, and traceroute measurements. The tools selection and their output data defines the possible conclusions that can be drawn. More advanced tools will allow more advanced conclusions. For example, to observe the behaviour of TCP we could use a tool such as tcpdump. The longer delays experienced by httping, likely caused by packet loss being fixed by TCP re-transmissions would be exposed by this tool.

60

In some of the big media event measurements the httping was crashing several times when an error occurred. During such measurements it became a challenge not to have a complete results from some MAs.

61

7. Conclusions We developed CheesePi, a distributed measurement platform which help users to characterise their Internet experience. The CheesePi measurement agent runs on a Raspberry Pi connected to a home router. The measurement agent conducts ICMP, HTTP, traceroute and VoIP measurements periodically. Additionally, by increasing the number and distribution of the measurement agents the network can be characterized better. A dashboard which is locally hosted on the Raspberry Pi, is used to visualise the experienced connection behaviour. The measurement results show customers with an Ethernet connection experienced better latency and packet loss compared to WiFi users. For example, the average network delay for all targets during Eurovision Final from SICS WiFi was 4 ms and was more than double the average network delay from an Ethernet connection (1.5 ms). And the average network delay for all targets during Eurovision Final from the student accommodation WiFi was 1.4 times than the average network delay from the Ethernet connection. In all the Eurovision measurements, the student apartment experienced significantly (26-57 times) worse performance in terms of latency compared to the SICS site. The relatively larger network delays from the student accommodation during Eurovision were for targets which use CloudFlare CDN service, and this might be the traffic carrying agreement between the student’s accommodation ISP and other ISPs. ISPs prefer to send their traffic through ISPs which do not charge them for carrying their traffic (Peering), and sometimes this might cause relatively larger delay. It can be also the content owners business relationship with ISPs for their CDN. In Boxing and Champions League Final both experienced almost comparable performance, with 1.x times better or worse performance, the .x depending on the particular site we chose. Our measurements showed us that the performance of streaming from most of the CDN enabled websites is not influenced by large number of visitors during big media events, which shows that CDNs are well provisioned. Another factor was the distance of the CDN server to the user, despite being on a highly available and well dimensioned the effect of the network delay, can dominate. As an example, the average HTTP delay from SICS Ethernet to ATDHE, a streaming website which is not CDN enabled, (but close in terms of hops) was 96 ms. The average HTTP delay from SICS Ethernet to CRICHD, which was CDN enabled streaming website, showed an average delay 212 ms, but far in terms of hops. This shows that the network delay, can dominate. Of course the counter case has been seen to be true, too, for example, the average HTTP delay from SICS Ethernet to SBS, a streaming

62

website which is not CDN enabled, was 713 ms. Whereas the average HTTP delay from SICS Ethernet to BBC iPlayer, a CDN enabled streaming website, was 12 ms. The CDN server was close to user, as we found from traceroute. We showed, it is possible to characterise the network performance using network tools such as ping, httping, and traceroute. By comparing ping and httping measurement results, we are able to distinguish whether performance degradation is created by the network or by the server. If the performance issue is known due to the network, by looking the traceroute results, we are able to identify where exactly the source of the problem is. We also able to identify local performance degradations because of the WiFi problem by comparing WiFi with Ethernet measurement results. This project shows that it is possible to characterize most of the user’s Internet experience using a few network tools. By using more additional network tools, and increasing the number of measurement agents, this project can help home Internet users and ISPs on identifying and troubleshooting existing network problems.

63

8. Future work CheesePi is an on-going project, of which my work was only a part. There are many ways it can be improved or extended. The most most important one is to conduct measurements from additional locations in Sweden. As of October, Raspberry PIs, running CheesePi have been installed at Karlstad, Vasterås and Luleå universities. We could also add other parts of the Nordic Internet. Deployment of more CheesePi nodes would enable a more detailed network characterisation. Extending the number of Pis to further home users is the final goal, of course. In the experiments we have done mostly landmark measurements. With more deployed nodes we could expand to intra-Pi measurements. This would help understand the traffic on more links than the landmarks. And specifically between home users, rather than from home users to large servers. We have seen path changes during our measurements and how it affects the user experience, the reasons are not clear at the end of this thesis. Further studies could and should look at this phenomenon, particularly considering longer measurement periods (at least a week) and again during popular events with video traffic. We have done some analysis on the measurement results, however this could be improved, for example, looking at the correlation between variables. Also some more detailed time series work could be done. Is there a trend in the data or are the delay measurements cyclic. We have used standard networking tools. More advanced networking tools and database storage engines could be included. For example tcpdump, paris-traceroute, one-way ping or a short TCP connection. These are done in the other works, such as Fathom or ARK, described briefly earlier.

64

9. Lessons learned ● This master thesis has helped me to have a much better understanding of

TCP/IPs behavior. I read some technical reports and articles which helps me to understand some popular network monitoring systems. I learned how to conduct experiments better. This master thesis was a good opportunity to gain experience with lower level network tools.

● Writing this scientific report was the best experience during this master thesis. The thesis report might not be perfect, but I learned a lot about how to summarize my ideas, how to convey my thoughts in a scientific and accurate way. The continuous feedback from my supervisors helps me a lot to improve my report.

● I improved my programming skills mainly in Python and to some extent in bash scripting. Before this master thesis my Python knowledge was rather basic, but since all the measurement modules are implemented using Python, it helps me to improve my programming skills. It also introduced me to different libraries for data analysis. I also acquired basic bash scripting skills during this project.

● The project was implemented with another master student, and it was a good experience to develop my team working skills. We had to share our knowledge and convey our thoughts properly to each other. Guidance from our supervisors was very helpful, they helped us to see things in different perspectives.

● Holding weekly meeting helped me to plan and present my work. I was presenting what I did every Tuesday and this help me to plan my weekly tasks ahead and to present my weekly work. It was a reasonable time frame to report my work and to consider the feedback I got from my supervisors.

● SICS is a great place, and I had a chance to meet different researchers and students. It was good experience for me to meet people from different backgrounds and cultures, and everyone around was very supportive. There was also a weekly paper review session in the Network Embedded Systems department, the comments from the researchers who supported and opposed the paper were a good lesson in understanding research.

● This master thesis helps me on how to take initiative and be alert in order to accomplish your goals. I took some initiatives to conduct network measurements even not included in this master thesis.

65

References [1] Greg Bobolo, “5 digital video trends to watch in 2015”, http://www.inma.org/blogs/innovative-advertising-solutions/post.cfm/5-digital-video-trends-to-watch-in-2015, 2015, [Online; accessed 10-August-2015]. [2] James F. Kurose, Keith W. Ross, Computer Networking a Top Down Approach 6th Edition, 2013. [3]Lydia Parziale, David T. Britt, Chuck Davis, Jason Forrester, Wei Liu, Carolyn Matthews, Nicolas Rosselot, TCP/IP Tutorial and Technical Overview, IBM, 2006. [4]NetWare TCP/IP Administration Guide, http://www.novell.com/documentation/nw6p/?page=/documentation/nw6p/tcpipenu/data/hbnuubtt.html, 2015, [Online; accessed 12-August-2015]. [5] Andrew S. TanenBaum, David J. Wetherall, Computer Networks 5th Edition, 2011. [6] Olivier Bonaventure, Computer Networking : Principles, Protocols and Practice Release 0.25, 2012 [7] Cellular Network, https://en.wikipedia.org/wiki/Cellular_network, [Online; accessed 17-August-2015]. [8] Content Delivery Network, https://en.wikipedia.org/wiki/Content_delivery_network, [Online; accessed 17-August-2015]. [9] How does a CDN work , http://www.cloudvps.com/community/knowledge-base/how-does-a-cdn-work/, [Online; accessed 18-August-2015]. [10] What is a Raspberry Pi? , https://www.raspberrypi.org/help/what-is-a-raspberry-pi/, [Online; accessed 18-August-2015]. [11] About 6 million Raspberry Pis have been sold, https://blog.adafruit.com/2015/06/08/about-6-million-raspberry-pis-have-been-sold-raspberry_pi-raspberrypi-mattrichardson-twit-newscreensavers/, [Online; accessed 18-August-2015]. [12] Passive Vs. Active Monitoring, https://www.slac.stanford.edu/comp/net/wan-mon/passive-vs-active.html, 2001, [Online; accessed 19-August-2015]. [13] Traceroute(8) Linux Man page, http://linux.die.net/man/8/traceroute, [Online; accessed 20-August-2015]. [14] A. Håkansson, Portal of Research Methods and Methodologies for Research Projects and Degree Projects. WORLDCOMP'13 - The 2013 World Congress in Computer Science, Computer Engineering, and Applied Computing, 22-25 July, 2013 Las Vegas, Nevada; USA. [15] Carlos Ordonez, Yeol Song, Carlos Garcia-Alvarado ,Relational versus Non-Relational Database Systems for Data Warehousing, 2010.

66

[16] Anthony Riccobono, Sports TV Ratings: How Many People Watched Mayweather vs. Pacquiao, The Kentucky Derby And NFL Draft?, http://www.ibtimes.com/sports-tv-ratings-how-many-people-watched-mayweather-vs-pacquiao-kentucky-derby-nfl-1907833, [Online; accessed -9 September-2015]. [17] Thomas Myers, Here's how you can watch Floyd Mayweather fight Manny Pacquiao tonight, http://www.mmamania.com/2015/5/2/8468715/how-to-watch-floyd-mayweather-fight-manny-pacquiao-tonight-ppv-boxing-free-online, [Online; accessed -9 September-2015]. [18] Nearly 200 million people watch Eurovision 2015, http://www.eurovision.tv/page/news?id=nearly_200_million_people_watch_eurovision_2015, [Online; accessed -9 September-2015]. [19] Kevin Ashby, Berlin final captures the world's imagination, http://www.uefa.com/uefachampionsleague/news/newsid=2255318.html, [Online; accessed -21 September-2015] [20] E RNESTO, CRICFREE bounces back after UK police domain seizure, https://torrentfreak.com/cricfree-bounces-back-after-uk-police-domain-seizure-140524/, [Online; accessed -21 September-2015] [21] Streaming 101: ESPN Vs Univision In World Cup, https://www.bizety.com/2014/06/28/streaming-101-espn-vs-univision-world-cup/, [Online; accessed -21 September-2015] [22] Internet World Stats, http://www.internetworldstats.com/stats.htm, [Online; accessed - 07 October-2015] [23] S. Shunmuga Krishnan, Ramesh K. Sitaraman, Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-Experimental Designs, 2012 [24] RIPE Atlas, https://atlas.ripe.net/about/, [Online; accessed - 08 October-2015] [25] The Archipelago Measurement Infrastructure (ARK), http://www.caida.org/projects/ark/, [Online; accessed - 08 October-2015] [26] Samknows, https://www.samknows.com/, [Online; accessed - 08 October-2015] [27] Test Node Briefing Technical information relating to the SamKnows test nodes, https://www.samknows.com/broadband/uploads/methodology/SQ302-001-EN-Test-Node-Briefing-D01.pdf, [Online; accessed - 08 October-2015] [28] NetBeez, https://netbeez.net/, [Online; accessed - 08 October-2015] [29] SpeedTest, http://www.speedtest.net/about, [Online; accessed - 08 October-2015] [30] OOKLA, https://www.ookla.com/, [Online; accessed - 08 October-2015] [31] Christian Kreibich, Nicholas Weaver, Boris Nechaev, Vern Paxson, Netalyzr: Illuminating The Edge Network, 2010. [32] Mohan Dhawan, Justin Samuel, Renata Teixeira, Christian Kreibich, Mark Allman, Nicholas Weaver, Vern Paxson, Fathom: A Browser-based Network Measurement Platform, 2012.

67

[33] PingER, https://www.slac.stanford.edu/grp/scs/net/talk09/pern-pinger-initiative.pdf, [Online; accessed - 08 October-2015]. [34] MLAB, http://www.measurementlab.net/ , [Online; accessed - 08 October-2015]. [35] Facts and figures about Akamai, https://www.akamai.com/us/en/about/facts-figures.jsp, [Online; accessed - 15 October-2015]

68