Evolution of P2P file sharing · Evolution of P2P le sharing Dmitri Moltchanov Department of Communications Engineering Tampere University of Technology [email protected] September

Evolution of P2P file sharing

Dmitri Moltchanov

Department of Communications EngineeringTampere University of Technology

[email protected]

September 10, 2014

Based on slides provided by R. Dunaytsev http://utopia.duth.gr/rdunayts/

Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster

5 GnutellaGnutella version 0.4Gnutella version 0.6Gnutella2

6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes

Dmitri Moltchanov (TUT) ELT-53206, Lecture 3 September 10, 2014 2 / 59

Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


ARPANET

The goal of the original ARPANET :share computing resources around the USA

Initially, the ARPANET: 4 independent hosts with equal status:Stanford Research Institute (SRI)University of UtahUniversity of California, Los Angeles (UCLA)University of California, Santa Barbara (UCSB)


ARPANET: some observations

Observations from the early years:

The early Internet client/server applications

telnet: log-in and use resources of a remote compute serverFTP: send and receive files from a file server

However, notice the following

the whole usage pattern is symmetricevery host could connect to any other hostservers acted as clients as well

What’s important to remember:

the Internet started as a P2P system

later on, it has become more restricted to client/server


Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


Usenet: basic info

What is that?

Usenet (USEr + NETwork): a system for distributing messagesmessages are posted to newsgroupsany user can access the messages, read, and respond to them

Developing and goals:

T. Truscott and J. Ellis, U. of North Carolina, Duke University, 1979goal: use Unix to write and read networked messages

Notice the key differences:

vs. Web forums: no central server, nor central system ownervs. e-mail: Usenet is for public distribution


Usenet: basic operations

Usenet communication consists of 4 basic steps :

1 Article composition: a user creates a Usenet article2 Posting: the user submits the article to a local Usenet server3 Propagation: transferring to remote Usenet servers4 Article access: reading the article

Local server

Local server

Usenet backbone peers

Poster Reader

Client Client

P2Pclient / server

interactionclient / server



interaction


Usenet: networking

Originally: messages over P2P network known as UUCPnet

Later on: some nodes are better connected – the Usenet backbone

How it works nowadays:

backbone servers collect articles from local onestransfer to other backbone servers in P2P fashiondistribute among other local servers

However, Usenet itself:

typical client/server APPsimple clientsmore powerful servers

Usenet: client/server and P2P are not mutually exclusive


Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


WWW: basic facts

Historic notes:

06.08.91, summary of the WWW project was posted on Usenetconsidered as the debut of the WWWgroups.google.com/group/alt.hypertext/msg/395f282a67a1916c

How it happened?

The driving force: disseminate information easily for everyoneResult: a wave of ordinary people began to use the Internet

Note: WWW is based on the client/server model using HTTP

Important consequences of WWW:

Internet became a mass media!!!up to WWW: equal sharing of information over the Internetafter WWW: the downstream paradigm, like TV and newspapers


http://groups.google.com/group/alt.hypertext/msg/395f282a67a1916c

WWW: implications

WWW drastically affected the Internet

the way we think about itmore importantly, technologies

WWW technological implications

asymmetric accessfirewallsnetwork address translators (partially)dynamic IP (partially).

Result: the network is well-suited for WWW or similar service.


WWW: bandwidth asymmetry

Bandwidth asymmetry

more downlink (server to client) than uplink (client to server)affected access technologies...

Example: ADSL2+ (ITU G.992.5, 2003):

downstream rate = 24 Mbit/supstream rate = 1 Mbit/s

Asymmetric access: places severe limitations

symmetric service became almost ”prohibited”bidirectional traffic: TCP performance degradationfor more information, see RFC 3449

Client

Downlink

UplinkServer


WWW: the need for firewalls

The early days of the Internet:any host can access any other hostno harm done: pro were using them

WWW made the Internet a public place:the need to secure the network protecting hosts from unlimited accessfirewalls is a tool to control access to subnetworks networks and hosts

What Firewalls do:stand in-between the internal network and the Internet outsidechoose which traffic to let through and which to deny

Early Internet These days


WWW: features of firewalls

A typical firewall do the following:

allows outgoing connections to any external hostprevents incoming connections from some/all external hosts

A host protected in this way cannot easily function as server

Outgoing connections may be restricted to certain applications

we block traffic to certain ports at the firewalle.g., only HTTP (ports 80 and 8080) and FTP (ports 20 and 21)


WWW: additional implications

IP address assignment:

the early days: static assignmentafter WWW ( DHCP ): dynamic due to too many usersresult: many hosts on the Internet are not easily reachable

Usage of Network Address Translation (NAT)

allows the use private IP addresses within a local networkresult: not only the host address is dynamic, it is not even visible!

Consequence: huge problems for something different than WWW


Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


Napster: shifting back to P2P

Some important notes:the driving force: frustration...digital music files were not easy to find back thenno intentions to make historycreative way of thinking in WWW era

Developer: Shawn Fanning, 18-year-old student, 1999

Functionality: share mp3 files

A year later, in 2000, Napster had 20 million users


Napster: basic functionality

Napster is a hybrid P2P system:

peers (Napster clients) and a central serversearching via central index containing location of all shared filesthe actual file transfer occurs between peers

quer

y / r

eply

Server

mp3 downloadPeer

P2P

client / server interaction

Peer

query / reply


Napster: logic

Napster worked as follows:1 after log-in: list of shared files sent to Napster server2 server maintains an index of files of users who are currently on-line3 index is automatically updated when users log-on and log-off4 searching for content

a client sends the query to the serverquery: pattern for search in filenamesreply: names of files, IP addresses and port numbers

5 a user decides from which peer to download the files6 file download is performed directly between the peers

Napster ran on top of TCP for client/server and P2P communications


Napster: parsed server response


Napster: solving firewalls

Normal operation:

downloader: sends a DOWNLOAD REQUEST to the serverserver: answers with a DOWNLOAD ACK: IP address+port:downloader establish TCP connection with uploadermay fail due to uploader behind firewall

If the uploader is behind firewall:

can be explicitly indicated by a usercan be understood by a serverport 0 is indicated in DOWNLOAD ACK

When the uploader is behind firewall

downloader: sends a DOWNLOAD REQUEST to the serverserver: answers with a DOWNLOAD ACK: IP address+port 0:downloader: ALTERNATE DOWNLOAD REQUEST to the serverserver: ALTERNATE DOWNLOAD ACK to the uploaderuploader establish TCP connection to the downloader.


Napster: some facts

Napster focused exclusively on mp3-encoded music files

Only one client implementation, called Napster in 1999

Windows only for one yearin 2000 Black Hole Media: Macintosh client called MacsterMacster was later used by Napster as the Napster client for Mac

The Napster protocol is a closed-source protocol

i.e., no one knows for sure how searching and transfer is done

How do we know details?

Reverse engineeringProject OpenNap: opennap.sourceforge.net


http://opennap.sourceforge.net

Napster: shutting it down

The reason for Napster’s failure was not technical, but legal:

20 millions of user in 2000Recording Industry Association of America (RIAA)

ban the exchange of copyrighted material and/orshut down the network

Napster deployed various filters in 2000Napster was forced to shut down in 2001

A bit more2002: Napster’s brand and logos were purchased by Roxio

online music service as Napster 2.0

2008: Napster, Inc. was purchased by Best Buy

music.napster.com/napsterhomemain.htm


http://music.napster.com/napsterhomemain.htm

Napster: shortcomings

The most important are:Scalability: reason - central index

one server was still sufficient for 20Mwhat’s about 960M ‘(like Facebook now)?

Reliability: reason - central index

vulnerable to DoS attacksvulnerable to legal prosecution

Security: encryption/description

authenticity of mp3 files was based on file names onlynot fully encrypted (like Skype now)

all these are easy to solve though...


Napster: summary

Developer(s) – Shawn Fanning

Launched – 1999

Protocol – proprietary

Clients – proprietary (Napster)

Architecture – hybrid P2P

Search facility – central server/central index

Shared content – mp3 only


Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


Gnutella v0.4

Aims of Gnutellasharing all types of filesP2P system without any central point of controldecentralized and autonomous, robust and hard to shut downthe latter one was extremely important

Important facts:

developed by J. Frankel and T. Pepper, employees of Nullsoft, in 2000source code and protocol specification are both openmore than 20 clients from different vendors

Decentralized systems: search strategy is a critical part!


Gnutella v0.4: search strategy

Simple and similar to the strategy used by humansif you want a particular thing, you ask one of your friends nearbyif he/she does not have it, he can ask his friendsthis request is conveyed furthera person having this thing is foundthe information about that person is routed to you

In this query flooding we have to ensure that:

everyone is eager to helpthe thing you’re looking for actually existssource routing is possible


Gnutella v0.4: Query propagation

TTL = 7

TTL = 6

TTL = 5

TTL = 4

TTL = 3

TTL = 2

TTL = 1

Query?

QueryHit!


Gnutella v0.4: controlled flooding

What we need to ensure

flooding zone is somehow limited (TTL)nodes do not sent the same Query multiple times (???)

Rules to avoid redundant replications:

facility: 128-bit ID associated with each Query, Ping, QueryHit messageall relaying nodes must remember itif exactly this is received again, it is not forwarded

Additional rules/needs

forwarding messages to all neighbors except for the sendersource routing: Pong, QueryHit, Push take reverse pathsdecrementing TTL to avoid overflooding


Gnutella v0.4: messages

Gnutella v0.4 uses the following messages:

Ping: used to discover peers on the network

a peer receiving Ping is expected to respond with a Pongincludes: IP address and port number

Pong: the response to a Ping message

includes: IP address and port numberincludes: number of files and Kb of data that it shares

Query: used to search files on the network

contains the search queryresponded with QueryHit if a match is found locally

QueryHit: the response to a Query descriptor

contains data to establish a connectionrouted back to the searching node.


Gnutella v0.4: solving the firewalls

Recall: a node behind a firewall

may not be able to receive incoming connectionsmay still be able to establish outgoing ones

Push message: the response to a QueryHit

response to QueryHit messagewhen the originator of QueryHit does not support connections

Peers proceed as follows:

Query (P1) - QueryHit (P2) exchangenormal operation: TCP Syn (P1) to P2avoiding firewall at P2: Push (P1) to P2TCP Syn (P2) to P1TCP Data (P2) to P1


Gnutella v0.4: download via HTTP

The Gnutella protocol just facilitates the search and find capabilities

The file download protocol is based on HTTP (on top of TCP)

PingPing

Pin

gP

ing

Pong

Ping / Pong routing Query / QueryHit routing Push routing

Query

Qu

ery

Qu

ery

Hit

Hit

Hit

Push

Download (HTTP)

Download (HTTP)

Pu

sh

Pu

sh


Gnutella v0.4: performance issues

Performance and scalability concerns of Gnutella v0.4:

Peers with low-speed connections

these peers are usually scattered all over the networkget overloaded with queries until they become unavailableresult: the network is highly fragmented

Signaling traffic

ping/pong takes too muchprevents delivery of the other important messages


Gnutella v0.4: performance issues

TR-2001-26: ”Peer-to-peer architecture case study: Gnutellanetwork”

Gnutella traffic (in bytes) by message types, November 2000:

Query messages (i.e., user-generated traffic) ≈ 35%Ping/Pong messages (i.e., overhead traffic) ≈ 55%


Gnutella v0.4: summary

Developer(s) – Justin Frankel and Tom Pepper

Launched – 2000

Protocol – open source

Clients – freeware and proprietary (Gnotella, Mactella, XoloX, . . . )

Architecture – pure P2P

Search facility – query flooding

Shared content – any


Gnutella v0.6

The biggest concern in v0.4 is the overhead

Gnutella v0.6 in 2001 addressed some of them

What was the idea?

ultrapeers and normal peersin fact, a hierarchial topologyultrapeers store info about connected leafsinfo: files they share + IP addresses + ports

TR-2001-26 ”Peer-to-peer architecture case study: Gnutella network”

Gnutella traffic (in bytes) by descriptor types, June 2001:

query messages (i.e., user-generated traffic) ≈ 92%ping/pong messages (i.e., overhead traffic) ≈ 8%compare with the previous slide


Gnutella v0.6: leafs and ultrapeer functionalities

2 modes of Gnutella v0.6 peers :

LeafUltrapeer

Functionalities:

a peer can switch to/from ultrapeer modea peer cannot be in both modes at the same timeleafs maintain only a single connection to a ultrapeeran ultrapeer maintain 10-100 connections with leafsan ultrapeer maintain ∼10 connections with leafs

How leafs info is used at ultrapeers

recall: file names + IP + portforwarding Query messagesi.e., if a leaf shares nothing no Query is sent


Gnutella v0.6: ultrapeer network

an ultrapeer network


Gnutella v0.6: summary

Developer(s) – Justin Frankel and Tom Pepper

Launched – 2001


Clients – freeware and proprietary (LimeWire, gtk-gnutella, . . . )


Search facility – ultrapeers / query flooding


Notice performance gains of v0.6 vs v0.4


Gnutella 2: overview

Gnutella2 :

AKA G2 or Mike’s Protocol, MPP2P protocol developed by Michael Stokesoffshoot of the Gnutella protocolreleased in 2002

G2 divides nodes into 2 groups: leafs and hubsleafs maintain 1 or 2 connections to hubshubs accept 100s of leaves and many connections to other hubshubs index what files a leaf has using a Query Routing Table

Gnutella v0.6 and G2 are fairly similar

Compatibility notice:

G2 is not supported by many Gnutella clientsmany G2 clients can connect to the Gnutella network


Gnutella 2: summary

Developer(s) – Michael Stokes

Launched – 2002


Clients – freeware and proprietary (Shareaza, Morpheus, . . . )


Search facility – hubs / query flooding

Shared content – music and video


Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


FastTrack: basic info

Very quickly it turned out that hierarchical P2P topologies scaleremarkably well and have been, therefore, adopted by a number ofpopular P2P file sharing systems

FastTrack is a P2P protocol developed by Niklas Zennstrom andJanus Friis in 2001

In 2003, FastTrack became very popular, having at one point moreusers than that Napster on its peakSimilar to Gnutella v0.6, highly connected peers with sufficientresources, called supernodes, perform the search on behalf of moreresource constraint regular peersBut in contrast to Gnutella, FastTrack is proprietary software

In 2003, an open-source alternative, called OpenFT, was developedby the giFT project through reverse engineering

The giFT project: gift.sourceforge.net


http://gift.sourceforge.net

FastTrack: supernode network

Supernode network


FastTrack: bootstrapping

In order to ensure the constant availability of the network, there existsa need for dedicated peers that will monitor and keep track of thenetwork

Such peers are called bootstrapping nodes

When a peer joins the network, it will first contact the bootstrappingnode

The bootstrapping node will then determine if that particular peerqualifies to be a supernode

If it does, then it will be provided with some IP addresses of othersupernodesOtherwise, the bootstrapping node will respond by providing the IPaddress of one of the supernodes


FastTrack: cooperation mechanism

The P2P model heavily depends on the concept of cooperation

See ”Free Riding on Gnutella” by E. Adar and B. Huberman:

∼ 50% of all files for sharing are stored on only 1% of peers∼ 70% of Gnutella users share no files (they are referred to asfreeloaders or free riders)

Some FastTrack clients, such as Kazaa, used a method known asthe reputation system , where the reputation of a certain user is

reflected by their participation level

E.g., in Kazaa if you have a high participation level, you will getpriority over people with a lower participation level when youdownload files

You can increase your participation level by uploading filesWhen you download files, your participation level will go downWhen your participation level is low, downloading will be slower


FastTrack: shutting down

Sharing copyrighted content using the original FastTrack protocol:

KazaaOwned: Consumer Empowerment BV, Sharman Networks, Ltd.Lifetime: 2001 - 2005 (reason - lawsuits from copyright owners)Today: legal music download service, www.kazaa.com

GroksterOwned: Grokster, Ltd.Lifetime: 2001 - 2005 (reason - lawsuits from copyright owners)Today: closed, www.grokster.com

iMeshOwned: iMesh, Inc.Lifetime: 2001 - 2005 (reason - lawsuits from copyright owners)Today: legal music download service, www.imesh.com

MorpheusOwned: StreamCast Networks, Inc.Lifetime: 2001 - 2002 (reason - failed to pay FastTrack license fees)Today: closed, www.morpheus.com


http://www.kazaa.com

http://www.grokster.com

http://www.imesh.com

http://www.morpheus.com

FastTrack: summary

Developer(s) – Niklas Zennstrom and Janus Friis

Launched – 2001

Protocol – proprietary (FastTrack)

Clients – proprietary (Kazaa, Grokster, iMesh, Morpheus)


Search facility – supernodes / query flooding



Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


eDonkey2000: performance improvement

The next problem to solve is how to increase the throughput of P2Pnetworks

With the size of the files being transferred, it might take a long timebefore a new peer would be able to share a new file it is acquiring

A new approach, called swarming , cleverly sidestepped this issue:

If a file can be downloaded in small pieces (aka chunks) and thenimmediately be shared by new peers, this bidirectional transfer reducesthe load on peers that have the complete file, thereby distributing notonly the file but its access as well

iMesh was the first company to introduce swarming

Nowadays, swarming is a must for P2P file sharing

E.g., eDonkey2000, BitTorrent


eDonkey2000: swarming

While every peer has both upload and download capabilities, withoutswarming the upload bandwidth goes wasted until that peer has 100%of the file

With swarming, all peers work bidirectionally with each otherBut if the originator leaves the swarm too early, no one will be able touse the file


eDonkey2000: summary

Developer(s) – Jed McCaleb

Launched – 2000

Protocol – server part is proprietary (the original eD2k byMetaMachine and the alternative Lugdunum eserver)

Clients – freeware (eMule, Shareaza, . . . )


Search facility – servers / index



Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


BitTorrent: basic info

BitTorrent was developed by Bram Cohen in 2001

Nowadays, it is maintained by Cohen’s company – BitTorrent, Inc.

The BitTorrent network is huge

E.g., according to isohunt.com (September 15, 2010):

Peers: 23.53M (+3.1M over the last 5 months)Shared files: 134.05M (+15.68M)The total amount of shared content: 11,322.96 TB (+715.70 TB)

Search is up to users

Efficient cooperation mechanisms

Swarming functionality

Extensions for trackerless operations


http://isohunt.com

BitTorrent: summary

Developer(s) – Bram Cohen

Launched – 2001

Protocol – open source (BitTorrent)

Clients – freeware (µTorrent, BitComet, . . . )


Search facility – trackers / torrents



Outline

1 ARPANET

2 Usenet

3 World Wide Web

4 Napster


6 FastTrack

7 eDonkey2000

8 BitTorrent

9 Learning outcomes


Learning Outcomes

Things to know:

Basics about USENETRestrictions WWW imposedHow Napster worksHow Gnutella worksNew features introduced in FadtTrack/eDonkey/BitTorrent

Be ready to outline the evolution of P2P file sharing


Documents

Evolution of P2P file sharing · Evolution of P2P le sharing Dmitri Moltchanov Department of Communications Engineering Tampere University of Technology [email protected] September