VoD BitTorrent Framework

Background

• BT is a very popular peer to peer protocol with many implementations:– http://en.wikipedia.org/wiki/

Comparison_of_BitTorrent_clients

• The first one was created in Python by Bram Cohen.

Real world motivation

• BitTorrent is common and Free

• Good chance there is a swarm for your movie

• VoD servers are expensive and complicated

• Its better to use P2P for VoD

Some definitions• Delay (D) = How much time I am allowed to hold before

starting to play M (Movie).

• Prefetch Time (P) = How much time before I watch byte B from M, it should arrive.

• Playback Rate (R) = How many bytes/second I watch

• B(t) – the index of the bit that must arrive by t.

• B(t) = R *(P+t-D)

Rules and Examples

• B(t) = R * (P+t-D)

• If R=5 KB/sec and D=6 seconds and P=2 seconds, than:– At t=6 seconds, I must have 10KB continuous from

the start of the movie.– At t=18 seconds I must have 70KB continuous from

the start of the movie.

What does one VOD client do

• Your client tries to get from the Bittorent clients pieces of data needed to play the video.

• Each piece needs to arrive before its deadline.

• If the piece does not arrive by the deadline you can download it “instantly” from the “server”.

• Each VOD client counts the data from server (DFS).

What is my goal

• Each byte taken from the server costs money, therefore you need to minimize DFS.

• Goal - Minimize the accumulated DFS of all the VOD clients that run concurrently.

How do I create my client• Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this

Code” link in the course web page.

– It is also in /users/courses/hanochworkshop/group_vod

• GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to monitor DFS. Any changed section is marked  ###### GROUP VOD ######

• Modify it to minimize acumulated DFS.

• Do not change StreamWatched.py.

Standard BT parameters

• --saveas F

• --max_upload_rate

• F.torrent

VOD client additional params• SET = defined inside run_all.sh, not a parameter.• delay – Wait before prefetch, in seconds. SET TO 20• prefetchT – How much time in advance the byte must arrive. In

percent of movie length. SET TO 5%• rate – VOD rate in KB/sec. SET TO 90 KB.• order – How many run when client spawned. SET BY run_all.sh.

• Parameter of run_multiple.sh.• gap – How much to wait between spawns – unit is pewfetchT.• alg – Name of the algorithm. Default is ORIG.• verbose – Print and kill the whole swarm when complete.• peers_num – how many regular peers. • seeds_num – how many seeds.• Vod_num – how many VoD peers.

How do I test my idea• cd /users/courses/hanochworkshop/voddiversegroup/vod_tests

• run_multiple.sh – your_client - example: ~/GROUP_VOD_CODE/btdownloadheadless.py

– Gap – example: 0– output_location example: ~/bt_out– iteration example: 4– empty/full example: empty– peers_num example: 16– seeds_num example: 2– vod_num example: 14

Execution example

run-all.sh

Tracker

BTBTBT

BTBT

BTBT

BT

BTBT

VOD BT

run_multiple.sh

VOD BTVOD

BTVOD

BT

How does it work?

• Run tracker

• Run the “world”

• Run your clients

How does it work?

• Both tracker and the “world” run on the same computer.

• Running a test:– Reserve time on pc-hanoch-w3/w4 in:

• http://bt-p2p-vod.wikispaces.com/ • Or from course www - "Scheduling computers for tests.“

Output to screen==================================================================================

total time: 1 min 07 sec

share rating: 0.904 (4816.0 KB up / 5328.0 KB down)

saving: picture-100.bmp (27.5 MB)

percent done: 18.9

time left: 4 min 52 sec

download to: /home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp

download rate: 93.4 kB/s

upload rate: 80.6 kB/s

seed status: 2 seen now, plus 0.340 distributed copies

peer status: 28 seen now, 10.7% done at 1220.3 kB/s

--------------------------------StreamWatcher-------------------------------------

Csv stats: ~/out/1/statistics-order-4-gap-0.csv

DFS is: 8372224 bytes

DFS/Total is: 29 %

FullPieces: 63 / 220

DirtyPieces: 1 / 220

FaildHashChecks: 0

Prefetching 5 %

Playing point: 56 / 220 ( 25 %)

Only last VOD client prints

Output to .csv

• In ~/out you will find vod_num directories: /1, /2, …/iteration.

• In each directory, you have vod_num files.

• Statistics in - order-<1..vod_num>-gap-<GAP>.csv

• Order is how many VOD clients ran before.

• <GAP> is the parameter Gap.

.csv filealg,dfs,p2p

ORIG,5,0

ORIG,10,2

ORIG,15,6

ORIG,19,11

ORIG,23,16

ORIG,28,20

ORIG,31,25

ORIG,34,31

ORIG,37,36

ORIG,39,41

.

.

.

total DFS and total P2P GAP = 2

File percentage

GAP = 0

Local DFS GAP = 2

GAP = 0

run_all.sh defaults• Start

– Use in run_multiple.sh

• Stop– Kill all python threads

• Kill – Kill all python threads and scripts.

To run client separately:

• PATH_TO_YOUR_VOD_CLIENT/btdownloadheadless.py

--saveas F

--delay D

--prefetchT P

--rate R

--out_dir O Can add the parameters

--order X in the IDE

--gap G

--group_size S

--alg ‘ALG'

--verbose

F.Torrent

Python

• Very common script language

• IDE: – PyDev – WINGWARE Python IDE – not free, but very good.

• References:– python.org :http://docs.python.org/reference/ – http://wiki.python.org/moin/BeginnersGuide/NonProgrammers– http://wiki.python.org/moin/BeginnersGuide/Programmers

• The book "Think like a Computer Scientist in Python" has a very good reputation:

Getting started• Run the environment

– ssh pc-hanoch-w3 or pc-hanoch-w4– cd /users/courses/hanochworkshop/group_vod/vod_tests– run_multiple.sh MyClient GAP output-directory– Note tracker prints a message for each GET.

• You can connect to the tracker, from the CS network, with a browser at:– http://pc-hanoch-wk:6970/

• Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see downloaded files

Directory Structure

• Root – – high level application

• Root/BitTornado – – services

• Root/BitTornado/BT1 –– Actual work

Client Operation

• btdownloadheadless.py – First to run.– Performs initializations.

• get_response – parse the .torrent• Infohash – the ID of the .torrent• startEngine – Start connection with peers• startRerequester - Start connection with tracker

– Run rawserver.listen_forever from RawServer.py.

– listen_forever is: • Polling port • executing func() from a task queue.

Receiving messages

• From listen_forever loop it goes to:– BT1/Connecter.py/got_message

• CHOKE • UNCHOKE• INTERESTED• NOT_INTERESTED• BITFIELD• REQUEST• CANCEL• PIECE• HAVE

Influencing decisions• Can be done in one of the callbacks which

spring from Connecter.py / got_message:

• Downloader.py– HAVE->got_have->send_interested

• Tune for VoD?

BT tracker

• Send GET request in Rerequester.py / announce

• Receive GET data in thread in Rerequester.py / _rerequest

Policy

• Choker.py/_round_robin – decide who to choke / optimistically un choke / send_have to.

• Downloader.py / _request_more.

• PiecePicker.py / next - rarest piece, etc. Tune for VoD?

• got_unchoke->_request_more->next…

More about the code

• Start your project from “BitTornado - Start with this Code” link.

• Click “BitTornado - Legacy Code Guide” for basic explanations about the code.