VoD BitTorrent Framework
Background
• BT is a very popular peer to peer protocol with many implementations:– http://en.wikipedia.org/wiki/
Comparison_of_BitTorrent_clients
• The first one was created in Python by Bram Cohen.
Real world motivation
• BitTorrent is common and Free
• Good chance there is a swarm for your movie
• VoD servers are expensive and complicated
• Its better to use P2P for VoD
Some definitions• Delay (D) = How much time I am allowed to hold before
starting to play M (Movie).
• Prefetch Time (P) = How much time before I watch byte B from M, it should arrive.
• Playback Rate (R) = How many bytes/second I watch
• B(t) – the index of the bit that must arrive by t.
• B(t) = R *(P+t-D)
Rules and Examples
• B(t) = R * (P+t-D)
• If R=5 KB/sec and D=6 seconds and P=2 seconds, than:– At t=6 seconds, I must have 10KB continuous from
the start of the movie.– At t=18 seconds I must have 70KB continuous from
the start of the movie.
What does one VOD client do
• Your client tries to get from the Bittorent clients pieces of data needed to play the video.
• Each piece needs to arrive before its deadline.
• If the piece does not arrive by the deadline you can download it “instantly” from the “server”.
• Each VOD client counts the data from server (DFS).
What is my goal
• Each byte taken from the server costs money, therefore you need to minimize DFS.
• Goal - Minimize the accumulated DFS of all the VOD clients that run concurrently.
How do I create my client• Download GROUP_VOD_CODE.tar.gz from “BitTornado - Start with this
Code” link in the course web page.
– It is also in /users/courses/hanochworkshop/group_vod
• GROUP_VOD_CODE.tar.gz is a regular BT with StreamWatcher.py to monitor DFS. Any changed section is marked ###### GROUP VOD ######
• Modify it to minimize acumulated DFS.
• Do not change StreamWatched.py.
Standard BT parameters
• --saveas F
• --max_upload_rate
• F.torrent
VOD client additional params• SET = defined inside run_all.sh, not a parameter.• delay – Wait before prefetch, in seconds. SET TO 20• prefetchT – How much time in advance the byte must arrive. In
percent of movie length. SET TO 5%• rate – VOD rate in KB/sec. SET TO 90 KB.• order – How many run when client spawned. SET BY run_all.sh.
• Parameter of run_multiple.sh.• gap – How much to wait between spawns – unit is pewfetchT.• alg – Name of the algorithm. Default is ORIG.• verbose – Print and kill the whole swarm when complete.• peers_num – how many regular peers. • seeds_num – how many seeds.• Vod_num – how many VoD peers.
How do I test my idea• cd /users/courses/hanochworkshop/voddiversegroup/vod_tests
• run_multiple.sh – your_client - example: ~/GROUP_VOD_CODE/btdownloadheadless.py
– Gap – example: 0– output_location example: ~/bt_out– iteration example: 4– empty/full example: empty– peers_num example: 16– seeds_num example: 2– vod_num example: 14
Execution example
run-all.sh
Tracker
BTBTBT
BTBT
BTBT
BT
BTBT
VOD BT
run_multiple.sh
VOD BTVOD
BTVOD
BT
How does it work?
• Run tracker
• Run the “world”
• Run your clients
How does it work?
• Both tracker and the “world” run on the same computer.
• Running a test:– Reserve time on pc-hanoch-w3/w4 in:
• http://bt-p2p-vod.wikispaces.com/ • Or from course www - "Scheduling computers for tests.“
Output to screen==================================================================================
total time: 1 min 07 sec
share rating: 0.904 (4816.0 KB up / 5328.0 KB down)
saving: picture-100.bmp (27.5 MB)
percent done: 18.9
time left: 4 min 52 sec
download to: /home/pc-hanoch-w4/home/hillelav/name/picture.pc-hanoch-w4-4.bmp
download rate: 93.4 kB/s
upload rate: 80.6 kB/s
seed status: 2 seen now, plus 0.340 distributed copies
peer status: 28 seen now, 10.7% done at 1220.3 kB/s
--------------------------------StreamWatcher-------------------------------------
Csv stats: ~/out/1/statistics-order-4-gap-0.csv
DFS is: 8372224 bytes
DFS/Total is: 29 %
FullPieces: 63 / 220
DirtyPieces: 1 / 220
FaildHashChecks: 0
Prefetching 5 %
Playing point: 56 / 220 ( 25 %)
Only last VOD client prints
Output to .csv
• In ~/out you will find vod_num directories: /1, /2, …/iteration.
• In each directory, you have vod_num files.
• Statistics in - order-<1..vod_num>-gap-<GAP>.csv
• Order is how many VOD clients ran before.
• <GAP> is the parameter Gap.
.csv filealg,dfs,p2p
ORIG,5,0
ORIG,10,2
ORIG,15,6
ORIG,19,11
ORIG,23,16
ORIG,28,20
ORIG,31,25
ORIG,34,31
ORIG,37,36
ORIG,39,41
.
.
.
total DFS and total P2P GAP = 2
File percentage
GAP = 0
Local DFS GAP = 2
GAP = 0
run_all.sh defaults• Start
– Use in run_multiple.sh
• Stop– Kill all python threads
• Kill – Kill all python threads and scripts.
To run client separately:
• PATH_TO_YOUR_VOD_CLIENT/btdownloadheadless.py
--saveas F
--delay D
--prefetchT P
--rate R
--out_dir O Can add the parameters
--order X in the IDE
--gap G
--group_size S
--alg ‘ALG'
--verbose
F.Torrent
Python
• Very common script language
• IDE: – PyDev – WINGWARE Python IDE – not free, but very good.
• References:– python.org :http://docs.python.org/reference/ – http://wiki.python.org/moin/BeginnersGuide/NonProgrammers– http://wiki.python.org/moin/BeginnersGuide/Programmers
• The book "Think like a Computer Scientist in Python" has a very good reputation:
Getting started• Run the environment
– ssh pc-hanoch-w3 or pc-hanoch-w4– cd /users/courses/hanochworkshop/group_vod/vod_tests– run_multiple.sh MyClient GAP output-directory– Note tracker prints a message for each GET.
• You can connect to the tracker, from the CS network, with a browser at:– http://pc-hanoch-wk:6970/
• Do ls -l /home/pc-hanoch-wk/home/hillelav/$USER to see downloaded files
Directory Structure
• Root – – high level application
• Root/BitTornado – – services
• Root/BitTornado/BT1 –– Actual work
Client Operation
• btdownloadheadless.py – First to run.– Performs initializations.
• get_response – parse the .torrent• Infohash – the ID of the .torrent• startEngine – Start connection with peers• startRerequester - Start connection with tracker
– Run rawserver.listen_forever from RawServer.py.
– listen_forever is: • Polling port • executing func() from a task queue.
Receiving messages
• From listen_forever loop it goes to:– BT1/Connecter.py/got_message
• CHOKE • UNCHOKE• INTERESTED• NOT_INTERESTED• BITFIELD• REQUEST• CANCEL• PIECE• HAVE
Influencing decisions• Can be done in one of the callbacks which
spring from Connecter.py / got_message:
• Downloader.py– HAVE->got_have->send_interested
• Tune for VoD?
BT tracker
• Send GET request in Rerequester.py / announce
• Receive GET data in thread in Rerequester.py / _rerequest
Policy
• Choker.py/_round_robin – decide who to choke / optimistically un choke / send_have to.
• Downloader.py / _request_more.
• PiecePicker.py / next - rarest piece, etc. Tune for VoD?
• got_unchoke->_request_more->next…
More about the code
• Start your project from “BitTornado - Start with this Code” link.
• Click “BitTornado - Legacy Code Guide” for basic explanations about the code.