1 Computer Networks An Open Source Approach Chapter 6: Internet Services Chapter 6: Application Layer

1

Computer NetworksAn Open Source Approach

Chapter 6: Internet Services

Chapter 6: Application Layer

2

Content

6.1 Issues 6.2 Domain Name System 6.3 E-Mail 6.4 World Wide Web 6.5 File Transfer Protocol 6.6 Simple Network Management Protocol 6.7 Voice over IP (VoIP) 6.8 Streaming 6.9 Peer-to-Peer Applications (P2P) 6.10 Summary


3

6.1 Issues

How ports work? How servers start? Classification of servers Characteristics of application layer protocols


4

How Ports Work?

Well known ports: 0~1023 Registered ports: 1024~49151 Dynamic and/or private ports: 49152~65535

FTP Daemon

Mail Daemon

Telnet Daemon

HTTP Daemon

21

ListeningPorts

23

25

80

Server Daemons

Server MachineClient 1 Machine

FTP Client

Web Browser

2880

8752

User AgentOutbound

Port

User AgentOutbound

Port

Client 2 Machine


5

How Servers Start?

Is a server better to run as standalone or via (x)inetd? Standalone:

requires explicit restarting for configuration changes to take effect

crashed standalone server will probably stay unnoticed and the service will become unavailable

better performance (x)inetd:

reads configuration files each time a client attempts to access their information

fresh copies will be spawned on demand


6

Classification of Servers

Concurrent connectionless server Concurrent connection-oriented server Iterative connectionless server Iterative connection-oriented server


Underlying Transport ProtocolsApplication Application layer protocol Underlying transport protocol

electronic mail SMTP, POP3, IMAP TCP

remote terminal access Telnet TCP

file transfer FTP TCP

web HTTP TCP

web caching ICP typically UDP

name resolution DNS typically UDP

network file system NFS typically UDP

network management SNMP typically UDP

routing protocol RIP typically UDP

Internet telephony proprietary (e.g. Vocaltec) typically UDP

streaming multimedia proprietary (e.g. RealNetworks) typically UDP

P2P proprietary (e.g. BitTorrent, eDonkey) UDP for queries and TCP for data transfer

Chapter 6: Application Layer 7

8

Iterative Connectionless Server socket

bind

recvfrom

sendto

socket

sendto

recvfrom

close

repeatinfinitely

repeat asneeded

Clients

Server

process

requests

responses


9

Concurrent Connection-Oriented Server socket

bind

listen

accept

fork

close(accepting)

close(listening)

read

process

write

Close(accepting)

socket

connect

write

read

close

repeatinfinitely

repeat asneeded

repeat asneeded

connection request

request

reply

Client

Server

child process

server process


10

Characteristics of Application Layer Protocols Variable message formats and length

Message formats and length of application layer protocols vary with different applications and requirements

Variable data types Messages can be transmitted in textual or non-textual

formats. E.g., a Web server replies textual Web pages and binary

images Statefulness

The server retains information of the session with the client E.g., an FTP server remembers the client’s current working

directory


11

6.2 Domain Name System

Domain name space and domain delegation Zones and name servers Zone data files and resource records Resolvers and Name resolution Open source: BIND


12

Top Level DomainsDomain Description

com Commercial organizations, such as Intel (intel.com).

org Non-profit organizations, such as WWW consortium (w3.org).

gov Government organizations, reserved for U.S government such as National Science Foundation (nsf.gov).

edu Educational organizations, such as UCLA (ucla.edu).

net Networking organizations, such as Internet Assigned Numbers Authority which maintains the DNS root servers (gtld-servers.net) .

int Organizations established by international treaties between governments. For example, International Telecommunication Union (itu.int).

Mil Reserved exclusively for the United States Military. For example, Network Information Center, Department of Defense (nic.mil).

Two-letter country code

The two-letter country code top level domains (ccTLDs) are based on the ISO 3166-1 two-letter country codes. Examples are tw (Taiwan), uk (United Kingdom).

arpa Mostly unused now, except for the in-addr.arpa domain, which is used to maintain a database for reverse DNS queries.

Others Such as .biz (business), .name (for individuals), .info (similar with .com).


13

Domain Delegation

root domain

comedutw

educomorg

nctunthu nsysu

ciscsie ee

cis.nctu.edu.tw

uk

co

Easier managementLoad distribution


14

Zones and Name Servers Differences between a zone and a domain

A zone contains some part of the domain name space A domain is a subtree in the domain name space A name server may be authoritative for multiple zones

Zone data files in a name server Contains Resource Records (RR) describing all the hosts

within that zone Six major types of RR: “SOA”, “NS”, “A”, “PTR”, “CNAME” and

“MX”

Master and slave name servers Availability and load balance Data duplication by zone transfer

Request from slave Notify by master


15

Master/Slave Name Servers

slave

1. Redundancy for heavy load

2. Availability achieved

mastermaster (A) | slave (B)

zone A zone B

df of A

df: zone data file

df of Bdf of A and B

A large domain name space

zone transfer

zone transfer

clients clients


16

Resource Records SOA (Start Of Authority)

Specify a best name server to be authoritative (master) for a zone

Ex: cis.nctu.edu.tw. 86400 IN SOA cisserv.cis.nctu.edu.tw.

Options for refresh, retry, expire ..etc are supported NS (Name Server)

Specify name servers for a domain Ex: cis.nctu.edu.tw. 86400 IN NS cisserv.cis.nctu.edu.tw.

A (Address) Map names to addresses, Example (multi-homed):

linux.cis.nctu.edu.tw. 86400 IN A 140.113.168.127linux.cis.nctu.edu.tw. 86400 IN A 140.113.207.127


17

Resource Records (cont) CNAME (Canonical Name)

Create name-to-name alias Ex: www.cis.nctu.edu.tw. 86400 IN CNAME cache.cis.nctu.edu.tw.

PTR (Pointer) Point addresses to names (canonical name only) Ex: 10.23.113.140.in-addr.arpa. 86400 IN PTR laser0.cis.nctu.edu.tw.

MX (Mail Exchanger) Provide mail-routing information The one of best (lowest) preference value is chosen

cis.nctu.edu.tw. 86400 IN MX 0 mail.cis.nctu.edu.tw.cis.nctu.edu.tw. 86400 IN MX 10 mail1.cis.nctu.edu.tw.


18

Name Resolution

Query type Forward query – name-to-address Reverse query – address-to-name

Resolution method Iterative (by referral, commonly adopted) Recursive (site-by-site recursion)

Caching for speeding up the lookup process Expire according to TTL Tradeoff between performance and consistency


19

Name Resolution (Iterative)

client (resolver

)

local name server

query for “www.dti.gov.uk”

root name server

1

2

3

4

6

5

“uk”name server

”gov.uk”name server

other candidate name servers

“dti.gov.uk”name server

7

8

www.dti.gov.uk. 5M IN A 164.36.253.20www.dti.gov.uk. 5M IN A 164.36.164.20


20

Message Format

ID

QR

Opcode AA

TC

RD

RA

Reserved Rcode

QDCOUNT

ANCOUNT

NSCOUNT

ARCOUNT

Question

Answer

Authority

Additional

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 bit

Header


21

Execution Flowchart of Named

Cache initialization

Wait for signals

Command line parsing

chroot and daemonize

Initialize server configurations and data

structures

Create managers

Install a handler for signals

1. Task manager

2. Timer manager

3. Socket manager

For possible restart or termination signals



23

Basic Data Structure in Named

view1 …viewlist

zone_table

zone1

zone3 zone2

zoneN

zone_table

zone3

zone4 zone8

…Red/Black Tree

for zones

RR1

RR2

RRN

Red/Black Tree of RRs for zone4

zone_table

…

viewNview2 NULL


24

Domain Information Gopher (DIG); <<>> DiG 9.2.0 <<>> www.nctu.edu.tw

;; global options: printcmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26027;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:;www.nctu.edu.tw. IN A

;; ANSWER SECTION:www.nctu.edu.tw. 259200 IN A 140.113.250.5

;; AUTHORITY SECTION:nctu.edu.tw. 259200 IN NS ns.nctu.edu.tw.nctu.edu.tw. 259200 IN NS ns2.nctu.edu.tw.nctu.edu.tw. 259200 IN NS ns3.nctu.edu.tw.

;; ADDITIONAL SECTION:ns.nctu.edu.tw. 259200 IN A 140.113.250.135ns2.nctu.edu.tw. 259200 IN A 140.113.6.2ns3.nctu.edu.tw. 259200 IN A 163.28.64.11


25

6.3 E-Mail

Introduction Message Formats Internet Mail Protocols Open Source: Qmail


26

Introduction

What is E-mail? A method of sending messages from one user to

another via computer networks Internet mail addressing:

format: [email protected] e.g. [email protected]


27

Introduction (cont.)

Components of Internet mail system: Mail User Agent (MUA) Mail Transfer Agent (MTA) Mail Delivery Agent (MDA) Mail Retrieval Agent (MRA)

Sending MUA Forwarding MTA Receiving MTA

TCP MDA Local MDA

MRA

Receving MUA

Mailbox

SMTP

System Call

SMTP

System Call

Write

Read/Write

POP/IMAP

Sender's Machine Local Mail Server Remote Mail Server Recipient's Machine


28

Message Formats

Internet Message Format (RFC 822) Multipurpose Internet Mail Extensions (RFC

2045-2049)


29

RFC 822 – Internet Message Format A message consists of:

An envelope which contains information needed to accomplish transport and delivery

Contents which compose the object to be delivered to the recipient


30

RFC 822 – Internet Message Format (cont.) Common message header fields:

Type Field Meaning

From: The person(s) who wished this message to be sent Originator

Reply-To: Provides a general mechanism for indicating any

mailbox(es) to which responses are to be sent

To: The primary recipients of the message

Cc: The secondary recipients of the message

Receiver

Bcc: Additional recipients of the message

Received: A copy of this field is added by each transport service

that relays the message

Trace

Return-Path: This field is added by the final transport system that

delivers the message to its recipient

Message-ID: Contains a unique identifier generated by the mail

transport on the originating system

Reference

In-Reply-To: Previous correspondence which this message

answers.

Other Subject: Provides a summary, or indicate the nature, of the

message.

Date Date: Supplies the date and time the mail was sent

Extension X-anything: It is used to implement additional features that have

not yet made it into an RFC, or never will.


31

RFC 822 – Internet Message Format (cont.) An example of message header:

From: "Dr. Ying-Dar Lin" <[email protected]>

To: [email protected]

Date: Mon, 30 Jul 2001 02:42:34 +0800 (CST)

Subject: paper

Return-Path: <[email protected]>

Received: (from root@localhost) by mail.cis.nctu.edu.tw (8.11.4/8.11.4) id f6TIgct33454 for [email protected]; Mon, 30 Jul 2001 02:42:38 +0800 (CST) (envelope-from [email protected])

Received: from cissun53.cis.nctu.edu.tw ([email protected] [140.113.23.53]) by mail.cis.nctu.edu.tw (8.11.4/8.11.4av) with ESMTP id f6TIgaI33445 for <[email protected]>; Mon, 30 Jul 2001 02:42:36 +0800 (CST) (envelope-from [email protected])

Received: (from ydlin@localhost) by cissun53.cis.nctu.edu.tw (8.9.3/8.9.3) id CAA16191 for juvenia; Mon, 30 Jul 2001 02:42:34 +0800 (CST)

Message-Id: <[email protected]>

X-Virus-Scanned: by AMaViS perl

X-UIDL: F5%#!88!"!>RG"!l!G!! Chapter 6: Application Layer

32

MIME What is MIME?

MIME is a specification for enhancing the capabilities of conventional Internet message format .

MIME enables email messages to have: textual header information and message bodies in

character sets other than 7-bit ASCII multiple objects within a single message binary or application-specific files multimedia files, such as images, audio, and video

files


33

MIME (cont.)

MIME message header:

Field Description

MIME-Version: Describes the version of the MIME message

format

Content-Type: Describes the MIME content type and subtype

Content-Transfer-Encoding: Indicates the encoding method for transmission

Content-ID: Allows a body of information to refer to another

Content-Description: Possible description for a body of information


34

MIME (cont.)

The MIME content type set:

Type Subtype(s) Important parameters

text plain, html charset

multipart mixed, alternative, parallel,

digest

boundary

message RFC 822, partial,

external-body

Id, number, total, access-type,

expiration, size, permission

application octet-stream, postscript, rtf,

pdf, msword

type, padding

image jpg, gif, tiff, x-xbitmap none

audio basic, wav none

video mpeg none


35

MIME (cont.)

Possible values for the Content-Transfer-Encoding: field: Quoted-Printable Base64 7bit 8bit Binary X-Encoding


36

MIME (cont.)

An example of MIME message:

From: 'Ching-Ming Tien' <[email protected]>

To: [email protected]

Subject: Cover

MIME-Version: 1.0

Content-Type: image/jpg;

name=cover.jpg'

Content-Transfer-Encoding: base64

Content-Description: The front cover of the book

<.....base64 encoded jpg image of cover...>Chapter 6: Application Layer

37

Internet Mail Protocols

Simple Mail Transfer Protocol (RFC 2821) Post Office Protocol (RFC 1939) Internet Message Access Protocol (RFC

2060)


38

Simple Mail Transfer Protocol What is SMTP?

A standard host-to-host mail transport protocol and traditionally operates over TCP on port 25


39

Simple Mail Transfer Protocol (cont.) Important SMTP commands:

SMTP replies:

Command Description

HELO Greet the receiver with the name

MAIL FROM: Indicates the sender, could be spoofed too

RCPT TO: Indicates the recipient

DATA Indicates the mail data, terminated by a " . " in a single line

RSET Reset the session

QUIT Close the session


Response Description

2xx Command accepted and processed.

3xx General flow control.

4xx Critical system or transfer failure.

5xx Errors with the SMTP command.

40

Simple Mail Transfer Protocol (cont.) An SMTP transaction scenario:


41

Post Office Protocol

What is POP? A protocol used to retrieve e-mail from a mail server.

POP3 Session States: 1. Authorization:

Must log in with password before entering transaction state.

2. Transaction: Client can request actions of server, get mail for example.

3. Update: Updates mail box to reflect actions taken in transaction

state.


42

Post Office Protocol (cont.) Minimal POP3 commands:

Command Description Session state

USER name Identifies the user to the server AUTHORIZATION

PASS string Enters user password AUTHORIZATION

STAT Gets the number of messages in and

octet size of maildrop

TRANSACTION

LIST [msg] Gets the size of one or all messages TRANSACTION

RETR msg Retrieves a message from the maildrop. TRANSACTION

DELE msg Marks the msg as deleted from the

maildrop.

TRANSACTION

NOOP No operation. TRANSACTION

RSET Resets all messages that are marked as

deleted to unmarked.

TRANSACTION

QUIT Terminates the session. AUTHORIZATION,

UPDATE


43

Post Office Protocol (cont.)

POP3 replies: +OK, -ERR

Example POPsession:


44

Internet Message Access Protocol What is IMAP?

A replacement for the POP3 protocol Differences between IMAP4 and POP3:

IMAP4 allows messages being stored and manipulated on the mail system

POP3 only allows users to download their messages and store and manipulate messages on the client’s machines


45

Internet Message Access Protocol (cont.) Four states in the IMAP4 server:

Non-authenticated state When a connection is established between the server and client

Authenticated state When a pre-authenticated connection starts, When acceptable authentication credentials have been provided After an error in selecting a mailbox

Selected state When a mailbox has been successfully selected

Logout state When the client asks to exit the server


46

Internet Message Access Protocol (cont.) IMAP4 command summary:

Session state Commands

Any CAPABILITY, NOOP, LOGOUT

Non-authenticated AUTHENTICATE, LOGIN

Authenticated SELECT, EXAMINE, CREATE, DELETE, RENAME,

SUBSCRIBE, UNSUBSCRIBE, LIST, LSUB,

STATUS, APPDNED

Selected CHECK, CLOSE, EXPUNCGE, SEARCH, FETCH,

STORE, COPY UID


47

Internet Message Access Protocol (cont.) An IMAP4 transaction scenario:



49

Open Source: qmail

Introduction to qmail: qmail is a secure, reliable, efficient, simple MTA

designed for Unix-like operating systems qmail is a replacement for the sendmail qmail is the second most common SMTP server qmail has by far the fastest growth of any SMTP

server


50

qmail System Structure

Core modules of qmail:Module Description

qmail-smtpd Receive a message via SMTP

qmail-inject Preprocess and send a message

qmail-queue Queue a message for delivery

qmail-send Deliver messages from the queue

qmail-clean Clean up the queue directory

qmail-lspawn Schedule local deliveries

qmail-local Deliver or forward a message

qmail-rspawn Schedule remote deliveries

qmail-remote Send a message via SMTP

qmail-pop3d Distribute message via POP3


51

qmail Data Flow

qmail-smtpd qmail-inject

qmail-queue

qmail-send qmail-clean

qmail-lspawnqmail-rspawn

qmail-localqmail-remote

SMTP

MDA

qmail-pop3d

mailbox/maildir

local mailclient

MRA

remote mailserver

remote mailserver

remote mailclient

MUA

qmail system

MUA

remote mailclient

MTA

MUA

SMTP POP3

MTA

MTA


52

qmail Control Files

Control files are in /var/qmail/control Some control files of qmail:

Control Default Used by Description

me FQDN of

system

various Default for many control files

rcpthosts (none) qmail-smtpd Domains that qmail accept

message for

locals me qmail-send Domains that qmail deliver locally

defaultdomain me qmail-inject Default domain name

plusdomain me qmail-inject Added to any host name that

ends with a plus sign

virtualdomains (none) qmail-send Virtual domains and users


53

qmail Queue Structure

The qmail queue directory is in /var/qmail/quque Subdirectories in the qmail queue and contents :

Subdirectory Contents

bounce Permanent delivery errors

info Envelope sender addresses

intd Envelopes under construction by qmail-queue

local Local envelope recipient addresses

lock Lock files

mess Message files

pid Used by qmail-queue to acquire an i-node number

remote Remote envelope recipient addresses

todo Complete envelopes


54

How Messages Pass Through the qmail Queue


55

6.4 World Wide Web

Introduction Web Naming and Addressing Web Data Formats Hypertext Transfer Protocol Open Source: Apache


56

Introduction

What is WWW? A system of Internet servers that support specially

formatted documents. How does it work?

WebBrowser

Web Server

DNS Server1: DNS query

2: TCP 3-way handshake

3: HTTP request

4: HTTP response


57

Web Naming and Addressing

Uniform Resource Identifier (RFC 2396) Uniform Resource Locator (RFC 1738) Uniform Resource Name (RFC 2141)

http: ftp: gopher: etc.

urn:URLs

URNs

URNs


58

Uniform Resource Identifier What is URI?

A compact string of characters for identifying an abstract or physical resource.

URI syntax: Absolute URI: <scheme>:<scheme-specific-part> Generic URI: <scheme>://<authority><path>?<query>

URI examples: http://speed.cis.nctu.edu.tw/~ydlin/index.html#Books http://www.google.com/search?q=linux ftp://ftp.cis.nctu.edu.tw/Documents/IETF/rfc2300~2399/rfc2396.txt mailto: [email protected] news: comp.os.linux telnet://bbs.cis.nctu.edu.tw/ ../icons/logo.gif


59

Uniform Resource Locator

What is URL? A compact string representation of the location for a

resource that is available via the Internet URL syntax:

<service>//<user>:<password>@<host>:<port>/<url-path>Service Description

ftp File Transfer protocol

http Hypertext Transfer Protocol

gopher The Gopher protocol

mailto Electronic mail address

news USENET news

nntp USENET news using NNTP access

telnet Reference to interactive sessions

wais Wide Area Information Servers

file Host-specific file names

prospero Prospero Directory Service


60

Uniform Resource Locator (cont.) Some URL examples:

http://www.cis.nctu.edu.tw/chinese/ccg/titleMain.gif ftp://john:[email protected]/projects/book.txt nntp://news.cis.nctu.edu.tw/cis.course.computer-

networks/5238 telnet://mail.cis.nctu.edu.tw:110/


61

Uniform Resource Name

What is URN? A name that identifies a resource of unit of

information independent of its location URN syntax:

<URN> ::= "urn:" <NID> ":" <NSS> NID: Namespace Identifier NSS: Namespace Specific String

URN examples: urn:path:/A/B/C/doc.html urn:ans:cis.nctu.edu.tw/ydlin/Resource urn:isbn:0-201-56317-7

URN resolutioin: http://www.isbn.com/0-201-56317-7


62

Web Data Formats

Evolution of Web Data Formats SGML HTML XML XHTML

SGML

HTML XML

XHTML


63

Standard Generalized Markup Language What is SGML?

A system for organizing and tagging elements of a document.

Characteristics of SGML Descriptive Markup Types of Document Data Independence


64

HyperText Makeup Language What is HTML?

The authoring language used to create documents on the World Wide Web.

A brief history of HTML HTML 2.0, 3.0, 3.2, 4.0

HTML document structure HTML version information Header Body


65

Extensible Markup Language

What is XML? A pared-down version of SGML, designed

especially for Web documents. Why XML? How to use XML?

Traditional data processing Document-driven programming (DDP) Archiving Binding


66

Extensible HyperText Markup Language What is XHTML?

A hybrid between HTML and XML specifically designed for Net device displays.

Why XHTML? Using XHTML with other W3C tag sets:

XHTML for structural markup of documents SMIL for multimedia MathML for mathematics SVG for scalable vector graphics XForms for smart web forms


67

Hypertext Transfer Protocol

What is HTTP? HTTP Conversation Client Request Server Response HTTP 1.1


68

What is HTTP?

A comprehensive addressing scheme Client-Server Architecture The HTTP protocol is connectionless and

stateless An extensible and open representation for

data types


69

HTTP Conversation

Client request Request headers Request body Server status Response headers Requested data Disconnected


70

Client Request

HTTP request methods:

Method Meaning OPTIONS A request for information about the communication options available

for the specified URI. GET Requests a document from the server. HEAD Like GET, except only the headers are returned. POST Sends data to some handler indicated by the URI. PUT Requests that the data in the body section be stored at the specified

URI. DELETE Requests that the specified resource be deleted. TRACE For debugging purposes; lets the client see what’s being received on

the other end. CONNECT Reserved for future use.


71

Client Request

Request example:


72

Server Response

Response example:

Server status codes:Class of code Meaning 1xx Informational. 2xx Indicates that a client request was completed successfully. 3xx Indicates that the request was redirected for some reason. 4xx Indicates that there was an error on the client end. 5xx Indicates that there was an error on the server end.



74

HTTP 1.1

HTTP 1.1 features: Persistent connections Chunked encoding byte ranges Caching Non-IP virtual hosting


Web Caching

Web caching is a mechanism to expedite document downloading in WWW.

To achieve maximum satisfaction from Web caching, some aspects need to be considered. Candidates to be cached Content replacement Cache coherence

75Copyright Reserved 2010 75Chapter 6: Application Layer

Transparent Proxy

A cache server can also act as a proxy server With port redirection, transparent proxy does

not require manual configuration Two types of transparent proxy

integrated within a gateway standalone server box

HTTP requests

Squid proxy/cache server

iptables

gateway

router/switchHTTP requests

Squid box

policy route or switch rules

(1)

(2)

dest. port = 80

dest. port = 80

76Copyright Reserved 2010 76Chapter 6: Application Layer

77

Open Source: Apache

Introduction to Apache: Open-Source Web server originally based on

NCSA server Available on over 160 varieties of Unix -- and

Windows NT Over 58% of Internet Web servers run Apache or

an Apache derivative








Ration of sendfile ( ) to ap_invoke_handler ( )

File size 1 kb 4 kb 16 kb 64 kb 256 kb 1024 kb

# of calling sendfile ( ) 1 1 1 2 7 15

Total execution time (μs) of sendfile ( )

37 37 42 78 215 527

Time ratio of sendfile ( ) to ap_invoke_handler ( )

35% 38% 40% 53% 77% 87%


85

Apache Server Life Cycle

On Unix systems, Apache creates multiple processes to handle requests.

The Windows and OS/2 ports are multithreaded..


86

The Request Processing Cycle


87

6.5 File Transfer Protocol File transfer service Operation model Inside the connections Open source: wu-ftpd


88

File Transfer Service Goal

File sharing Data replication for backup Efficiency and reliability during transmission

Use client-server model based on TCP/IP Authenticated and anonymous accesses Relationship between FTP and Telnet Access an ftp server with browser

ftp://ftp.cis.nctu.edu.tw (anony) ftp://[email protected] (authen)


89

Some Application FTP Commands

Command Description

OPEN Connect to a remote host

CAT View a file in a remote host

GET Retrieve files in a remote host

RENAME Change the name of a file in a remote host

RM Delete a file in a remote host

QUIT Terminate an FTP session


90

Operation Model

Client as both control host and receiver

Control connection (initiated by client)

FTP commands"PORT h1,h2,h3,h4,p1,p2"

FTP replies

File System File System

ServerClientData connection

(initiated by server)

Send/Receive data

Listen on port "L" (L=21)

Listen on port "p1,p2"

PORT: Send the IP and port of the client to which the data is retrieved


91

Inside the Connections Establishing control/data connections

Active Mode Control connection initiated by client Data connection initiated by server

Passive Mode When client is behind a firewall Both control/data connections are initiated by client

FTP Reply Example FTP session Error Recovery


92

Active/Passive ModeActive mode

Passive mode

client server

Request (Ask server to listen)

Reply(Listening on port P of host

H)

Initiate a data connection

firewall

client server

Initiate a data connection

firewall

Request(Connect to me at port P of host

H)

Control

Data


93

Some Proper FTP Commands

Command Description Type

USER Send the user name Access Control

PASS Send the password Access Control

PORT Send the IP and port of the client to which the data is retrieved

Transfer Parameter

PASV Tell the server to listen on a data port rather than initiate a data connection

Transfer Parameter

RETR Ask server to transfer a copy of the requested file to the client

File service

STOR Cause the server to accept and receive the data and store it as a file

File service

RNFR Specify the path of a source file to rename from File service

RNTO Specify the path of a destination file to rename to File service

ABOR Tell the server to abort the previous command and the corresponding data transfer

File service


94

FTP Reply

Reply Description Type

1yz The requested action is being initiated; expect another reply before proceeding with a new command.

Positive Preliminary reply

2yz The requested action has been successfully completed. Positive Complete reply

3yz The command has been accepted, but the requested action is being held, waiting for further information from another command.

Positive Intermediate reply

4yz The command is not accepted the and the requested action did not take place. The action can be requested again.

Transient Negative Completion reply

5yz Similar with 4yz, except that the error condition is permanent so that the action cannot be requested again.

Permanent Negative Completion reply


95

Error Recovery

The restart mechanism Sender inserts ‘marker’ (used to identify the checkpoint)

in the data stream Receiver marks the position of the marker and reply the

latest marker position of both sender and receiver to user When error, user issues ‘restart’ with the position of the

marker to the sender

* User (control host) and receiver may/may not exist in the same machine


96

Example FTP SessionSTATUS:> Connecting to www.cis.nctu.edu.tw (ip = 140.113.166.122)STATUS:> Socket connected. Waiting for welcome message...

220 www.cis.nctu.edu.tw FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready.COMMAND:> USER www

331 Password required for www.COMMAND:> PASS ********

230 User www logged in.COMMAND:> TYPE I

200 Type set to I.COMMAND:> REST 100

350 Restarting at 100. Send STORE or RETRIEVE to initiate transfer.COMMAND:> REST 0

350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.COMMAND:> pwd

257 "/home/www" is current directory.COMMAND:> TYPE A

200 Type set to A.COMMAND:> PORT 140,113,189,29,10,27 tell the server where to connect to

200 PORT command successful. COMMAND:> LIST retrieve directory listing

150 Opening ASCII mode data connection for /bin/ls. File status okay; about to open data connection

…….list of files….

COMMAND:> TYPE I200 Type set to I.

COMMAND:> PORT 140,113,189,29,10,31200 PORT command successful.

COMMAND:> RETR test retrieve the file “test”150 Opening BINARY mode data connection for test (5112 bytes).


97

Open Source: wu-ftpd Introduction to wu-ftpd

Originally developed at Washington University Most popular ftp daemon Maintained by WU-FTPD Development Group.

Features and configurations Virtual ftp servers On-the-fly compression Important configuration files

Inside wu-ftpd


98

Inside wu-ftpd

read ACL files

listen to requests

connection accepted?

fork a handler

stand-alone?

fork off (parent exits)

service initialization

reverse DNS check

parse & execute commands

yes

no (under (x)inetd)

yes

no

start ftp server with some options

loop until termination signal is received


99

Virtual FTP Servers1. Manage two or more FTP servers in one machine

2. Support guest groups for different virtual FTP servers

…

FTP server

Configuration file of each virtual server

clients

ftp.site1.com.tw Lookup the ftpaccess file

ftp.site2.com.tw

ftp.site3.com.tw

# Virtual Server setup for ftp.site1.com.tw virtual ftp.site1.com.tw root /var/ftp/virtual/site1 virtual ftp.site1.com.tw banner /var/ftp/virtual/site1/banner.msg

virtual ftp.site1.com.tw logfile /var/log/ftp/virtual/site1/xferlog # Virtual Server setup for ftp.site2.com.tw…

A rule segment in ftpaccess


100

On-the-Fly File Compressions Server compress a file (or directory) right before

retrieved by users Example

User ynlin logged in.Logged in to wwwpc.cis.nctu.edu.tw.ncftp /home/ynlin > ls1.tar.gz Desktop/ ucd-snmp-4.2.1/ncftp /home/ynlin > get ucd-snmp-4.2.1.tar.gzucd-snmp-4.2.1.tar.gz: 7393280 bytes 552.83 kB/sncftp /home/ynlin >lls -ldrwxr-xr-x 24 gis88559 gis88 3584 Oct 8 12:18 .drwxr-xr-x 88 root gis88 2048 Sep 10 17:48 ..-rw-r----- 1 gis88559 gis88 7393280 Oct 8 12:18 ucd-snmp-4.2.1.tar.gz


101

Important Configuration Files

File name Description

ftpaccess Used to configure the operations of the ftp daemon.

ftpconversions Specify the postfix of a retrieved file and its corresponding operations.

ftphosts Used to deny/allow some hosts to login as certain accounts.

ftpservers List the virtual servers and the corresponding directories containing their own configuration files.


102

6.6 Simple Network Management Protocol

Background Architectural framework MIB SMI SNMP, SNMPv2, SNMPv3 Open source: NET-SNMP


103

Background Control over network systems are demanded Small tools: ping, traceroute, netstate..etc (base

on ICMP) SNMP:

Remote control without being physically attached to managed entities

Exchange management information between network devices

Short history SNMPv1 (1989)– Management framework SNMPv2 (1993)– Functionality enhancement SNMPv3 (1998)– Security add-on


104

Architectural Framework

trap or response

management station

MIB

agent & managed

device in one machine

master agent

subagent (managed device )

poll or request

MIB MIB MIB

SNMP

Management station Agent Managed device Managed object (specified in MIB) Management protocols

Protocols simpler than SNMP


105

Management Information Base (MIB)

Tree-like virtual information store Identified by object identifiers Extensible in “experimental” and “private” branches

Register at IANA (http://www.iana.org/) MIB-II: for network management of TCP/IP-based

internets (RFC 1213)


106

MIB-II iso (1)

org (3)

dod (6)

….

internet (1)

directory (1)mgmt (2)experimental (3)private (4)

….

mib-2 (1)

system (1)

interface (2)at (3)

icmp (5)

tcp (6)udp (7)egp (8) ip (4)

….

….

transmission (10)

snmp (11)

OBJECT IDENTIFIER: 1.3.6.1.2.1.4

security (5)snmpv2 (6)

….


107

Object Groups in Mib-2

Group Description

system General information about the managed system

interface Configuration information and statistics of each physical interface

at Address translation between network address and physical address

ip Information of implementation and operation of IP in a local system. For example, routing table, default TTL.

icmp Information about the implementation and operation of ICMP. For example, number of messages ICMP sent and received.

tcp Information about the implementation and operation of TCP. For example, the number of maximum and active connections in the system.

udp Information about the implementation and operation of UDP. For example, the number of datagrams sent.

egp Information about the implementation and operation of EGP.

tranmission Related information and statistics of different transmission schemes.

snmp Information about the accesses and errors of SNMP operations.


108

Structure of Management Information (SMI) Define the structure of a particular MIB

Place restrictions on the types of the objects allowed in the MIB Abstract Syntax Notation one (ASN.1)

Exchange of information between application components in different systems

Syntax – integer, octet string, object identifier Encode the object values (by Basic Encoding Rule)

Define the way data is represented during transmission

Three categories of data types Simple: Primitive ASN.1 data types Application-wide: Special data types used in particular Applications Simply constructed: table, row


109

TCP Connection Table (in MIB-II)

-- the TCP Connection table -- The TCP connection table contains information about this -- entity's existing TCP connections. tcpConnTable OBJECT-TYPE SYNTAX SEQUENCE OF TcpConnEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "A table containing TCP connection-specific information." ::= { tcp 13 } tcpConnEntry OBJECT-TYPE SYNTAX TcpConnEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "Information about a particular current TCP connection. An object of this type is transient, in that it ceases to exist when (or soon after) the connection makes the transition to the CLOSED state." INDEX { tcpConnLocalAddress, tcpConnLocalPort, tcpConnRemAddress, tcpConnRemPort } ::= { tcpConnTable 1 } TcpConnEntry ::= SEQUENCE { tcpConnState INTEGER, tcpConnLocalAddress IpAddress, tcpConnLocalPort INTEGER (0..65535), tcpConnRemAddress IpAddress, tcpConnRemPort INTEGER (0..65535) } tcpConnState OBJECT-TYPE SYNTAX INTEGER {

closed(1), listen(2), synSent(3), synReceived(4) established(5), finWait1(6), finWait2(7), closeWait(8),lastAck(9), closing(10), timeWait(11), deleteTCB(12) }

ACCESS read-write STATUS mandatory DESCRIPTION "The state of this TCP connection.." ::= { tcpConnEntry 1 } tcpConnLocalAddress OBJECT-TYPE SYNTAX IpAddress ACCESS read-only STATUS mandatory DESCRIPTION "The local IP address for this TCP connection. In the case of a connection in the listen state which is willing to accept connections for any IP interface associated with the node, the value 0.0.0.0 is used." ::= { tcpConnEntry 2 } tcpConnLocalPort OBJECT-TYPE SYNTAX INTEGER (0..65535) ACCESS read-only STATUS mandatory DESCRIPTION "The local port number for this TCP connection." ::= { tcpConnEntry 3 } tcpConnRemAddress OBJECT-TYPE SYNTAX IpAddress ACCESS read-only STATUS mandatory DESCRIPTION "The remote IP address for this TCP connection." ::= { tcpConnEntry 4 } tcpConnRemPort OBJECT-TYPE SYNTAX INTEGER (0..65535) ACCESS read-only STATUS mandatory DESCRIPTION "The remote port number for this TCP connection." ::= { tcpConnEntry 5 }


110

TCP Connection Table (tabular view)

tcpConnTable (1.3.6.1.2.1.6.13)

Listen 0.0.0.0 23 0.0.0.0 0

Listen 0.0.0.0 161 0.0.0.0 0

closeWait 127.0.0.1 161 127.0.0.1 1029

established 140.113.88.164 23 140.113.88.174 3082

tcpConnState (x.1.1)

tcpConnLocalAddress(x.1.2)

tcpConnLocalPort(x.1.3)

tcpConnRemoteAddress(x.1.4)

tcpConnRemotePort(x.1.5)

x.1

tcpConnEntry = (x.1)

x.1

x.1

x.1

INDEX


111

Basic Operations in SNMP

PDU Descriptions Version

GetRequest Retrieve the value of a leaf object V1

GetNextRequest Get the object lexicographically next to the one specified

V1

SetRequest Set (update) a leaf object with a value V1

GetResponse Response for GetRequest (value) or SetRequest (ACK) V1

Trap Issued by agent to notify the management station of

some significant event asynchronously

V1

GetBulkRequest Retrieve large blocks of data, such as multiple rows in a

table.

V2

InformRequest Allows one MS to send trap information to another MS

and receive a response

V2

PDU: Basic data unit in SNMP operationsMS : Management StationVariable-binding list: A list of variables and corresponding values in a PDU


112

TCP Connection Table (lexicographical view)tcpConnTable (1.3.6.1.2.1.6.13=x)

tcpConnState (x.1.1)

tcpConnLocalAddress(x.1.2)

tcpConnLocalPort(x.1.3)

tcpConnEntry = (x.1)

Listen (x.1.1.0.0.0.0.23.0.0

.0.0.0)

Listen (x.1.1.0.0.0.0.161.0.0

.0.0.0)

closeWait(x.1.1.127.0.0.1.161.

127.0.0.1.1029)

0.0.0.0 (x.1.2.0.0.0.0.23.0.0

.0.0.0)

established(x.1.1.140.113.88.164.23.

140.113.88.174.3082)

0.0.0.0 (x.1.2.0.0.0.0.161.0.0

.0.0.0)

127.0.0.1(x.1.2.127.0.0.1.161.

127.0.0.1.1029)

140.113.88.164(x.1.2.140.113.88.164.23.

140.113.88.174.3082)

23(x.1.3.0.0.0.0.23.0.0

.0.0.0)

161 (x.1.3.0.0.0.0.161.0.0

.0.0.0)

161(x.1.3.127.0.0.1.161.

127.0.0.1.1029)

23(x.1.3.140.113.88.164.23.

140.113.88.174.3082)

…

Traverse the tree using Depth First Search


113

User-based Security Model (USM, RFC2574) Four major threats

Modification of Information (between agent and MS) Masquerade (pretend to be an authorized user) Disclosure (eavesdropping) Message Stream Modification (reorder, delay, replay)

Solutions Timeliness module (for 1st, 3rd and 4th threats) Authentication protocol—use MD5 (for the 2nd threat) Privacy protocol—Use DES (for 3rd)


114

View-based Access Control Model (VACM RFC 2575) Access control

MIB view -- a collection of MIB objects Elements in the model

Group– Categorization of managers securityLevel– Help distinguish the access rights of a

group Context– A collection of managed object accessible by an

snmp entity. MIB view– A subset of a particular context Access policy– Decide the final access rights of a group

to a context


115

An SNMP Entity and its Component (RFC 2571)

DispatcherMessage

Processing Subsystem

Access Control

Subsystem

Security Subsystem

SNMP engine

Command Generator

Command Responder

Notification Receiver

Notification Originator

Proxy Forwarder

Other

Application(s)

SNMP entity


116

Comparisons Between Three SNMP Versions

Version Description and improvement

SNMPv1 1. Define the SMI (RFC1155)

2. A more concise MIB definition (RFC 1212)

3. SNMP framework and its related operations (RFC 1157)

4. Concept of security (authentication) is proposed

SNMPv2 1. Improved SMI (support 64-bit counter, and other types of address than IP)

2. Inclusion of ‘GetBulkRequest’ to improve the efficiency for retrieving large

blocks of data

3. ‘InformRequest’ for communication between management stations

SNMPv3 1. Security and administration add-on’s (RFC2571)

2. Multi-version SNMP message processing and dispatch capability (RFC 2572)

3. The five type of applications within an SNMP engine (RFC 2573)

4. User-based security model (RFC 2574)

5. View-based access control (RFC 2575)


117

Open Source: Net-SNMP

Introduction to Net-SNMP Some commands for query, set and trap Extensible architecture

Ways of extending the MIB Subagent protocol– AgentX (RFC2741) Build and include a private MIB

Inside Net-SNMP Snmpd and snmptrapd


118

Introduction

Started at CMU, moved to UCDavis (1995) and is now based at SourceForge (2000~)

What it provides: An extensible agent SNMP library for further development Tools to get or set information from SNMP agents Tools to generate and handle SNMP traps

Support SNMP V1, V2 and V3 Runs on many Unix-like systems and Windows


119

Some Commands for Query, Set and Trap

Name Description and example PDU used

SNMPGET Retrieve the value of a leaf object using get GetRequest

SNMPSET Set (update) a leaf object with a value SetRequest

SNMPBULKGET Get multiple objects at a time. Possibly

under different subtreesGetBulkReques

t

SNMPWALK Explore all the objects under a subtree

of the MIBGetNextReques

t

SNMPTRAP Uses the TRAP Request to send information

to a network manager. More than one object

identifiers can be applied as arguments

Trap

SNMPSTATUS Used to retrieve several important statistics

from a network entity. Errors will also be

reported, if any

SNMPNETSTAT Displays the values of various network-

related information retrieved from a remote

system using the SNMP protocol


120

Command Line Examples

$ /usr/local/bin/snmpbulkwalk -v 3 -u ynlin -l authNoPriv –a MD5 -A ynlinpasswd localhost systemsystem.sysDescr.0 = Linux ynlin2.cis.nctu.edu.tw 2.4.14 #5 SMP 週四 11 月 22 23:6system.sysObjectID.0 = OID: enterprises.ucdavis.ucdSnmpAgent.linuxsystem.sysUpTime.0 = Timeticks: (30411450) 3 days, 12:28:34.50system.sysContact.0 = gis88559system.sysName.0 = ynlin2.cis.nctu.edu.twsystem.sysLocation.0 = ynlin2system.sysORLastChange.0 = Timeticks: (0) 0:00:00.00system.sysORTable.sysOREntry.sysORID.1 = OID: ifMIBsystem.sysORTable.sysOREntry.sysORID.2 = OID: .iso.org.dod.internet.snmpV2.snmpBsystem.sysORTable.sysOREntry.sysORID.3 = OID: tcpMIBsystem.sysORTable.sysOREntry.sysORID.4 = OID: ipsystem.sysORTable.sysOREntry.sysORID.5 = OID: udpMIB. . . .

$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0system.sysContact.0 = [email protected]

$ snmpset -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0 s gis88559system.sysContact.0 = gis88559

$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0system.sysContact.0 = gis88559


121

Snmpd and Snmptrapd

Snmpd Process snmp packets and respond with the required

information Can be either master agent or subagent Basic access control with a port/addr pair Load dynamical MIB module at start

Snmptrapd Receive and log trap messages Configured to launch an external program when receiving

a trap matching a particular object identifier


122

Processing flow inside Net-SNMP


Start the server with required options

init_agent()[agent/snmp_vars.c]

count = select(numfds, &fdset, 0, 0, tvp)

snmp_read()[snmplib/snmp_api.c]

receive() [agent/snmpdt.c]

netsnmp_check_outstanding_agent_requests() [agent/snmp_agent.c]

while (netsnmp_running)

init_master_agent() [agent/snmp_agent.c]

snmp_select_info()[snmplib/snmp_api.c]

init_snmp()[agent/snmp_api.c]

123

Data Structure of a Session


124

6.7 VoIP Introduction Elements in a VoIP environment Underlying protocols

H.323 SIP

Challenges for VoIP technology Open source: openphone


125

Introduction

Three ways of transferring voice Circuit switched Voice over frame relay Voice over IP

Pros and cons of VoIP compared with PSTN Inexpensive cost Simplicity Less bandwidth consumption Extensibility


126

VoIP Environment (H.323 adopted)

PSTN Network

Gateway

Telephone

Router

H.323 gatekeeper

H.323 terminal

IP Network

MCU

Packet switched network Circuit switched network


127

H.323 Protocol Stack

Q.931 H.245

G.711

G.723

G.729

H.263RTCP

RAS

(H.225.0)

Control (by GK)

A / V controlAudio VideoControl

TCP UDP

IP

RTP

T.120

Data

Recommendation Description

RAS(H.225.0) Stands for Registration, Admission and Status.

H.245 Control protocol for capability negotiation and channel setup.

Q.931 Used for call setup and teardown between two terminals.

T.120 Data protocols for multimedia conferencing. (application sharing, whiteboarding)

RTP/RTCP Used for real time traffic synchronization and transportation.


128

Setup Procedure of an H.323 Call Registration and admission

Call setup

Terminal capability negotiation ,channel setup and master-slave detection

Stable call established and proceeds

Close channel

Call teardown

Disengagement

RAS

Q.931

H.245

RTP/RTCP

H.245

Q.931

RAS


129

Session Initialization Protocol (SIP)

SIP Targeted to replace the H.323

Simplicity Compatibility

Characterize a multimedia session With supplementary SDP/SAP protocols

HTTP-like characteristics Text-based protocol Message syntax and header fields identical to HTTP/1.1 Client-server scheme


130

Elements in an SIP Environment

Local proxy server

Remote proxy server

Redirect server

Location server

User Agent Server (UAS)

User Agent Client (UAC)

Internet


131

Protocol Stacks of SIP

TCP UDP

IP

SAP/SDP RTP RTCP

SIP Multimedia Traffic

Control plane Data plane


132

Operators and Reply Codes in SIPOperators Description

INVITE Invite a user to a call

ACK Confirmation for the final response

BYE Terminate a call between endpoints

CANCEL Terminate the search for a user or request for a call

OPTIONS Features supported for a call

REGISTER Register current location of the client with location server

INFO Use for mid-session signaling

Reply Code Description

1xx (Informational) Trying, ringing and queued

2xx (Successful) The request was successful

3xx (Redirection) Give information about the receiver’s new location

4xx (Request Failures) Failure responses from a particular server

5xx (Server Failures) Failure responses given when a server itself has erred

6xx (Global Failures) Busy, decline, requests not acceptable


133

Example Operation in SIP

1. UAC INVITE UAS with the SIP URL 2. Destination IP address known => Send the request directly to the destination unknown => Redirected to the local proxy server (with

location server).3. In redirection mode, send back the callee’s new location4. In proxy mode, forward the request to the destination. 5. The UAS answers 200 (OK), then the UAC ACKs6. Session established


134

H.323 V.S. SIP

Simplicity in operation and implementation Message encoding Capability exchange Supported data types Capability of handling a conference


135

Open Source: Asterisk

An integrated PBX (Private Branch eXchange) system


Asterisk PBX Server

PC Softphone Analog PhoneSIP phone

ATA Adaptor

PSTN Networks

136

Block Diagram

Asterisk provides a framework to build a customized VoIP system. Flexible to add/remove modules to establish the basic

transport service

HTTP

Channel

CDR Engine

RTP

PBX

Framer

Module Loader

Asterisk framework

SNMP

136Chapter 6: Application Layer

137

Data Structures

Two concepts, context and extension, are introduced

Context1

Ext1 Ext2

Context2

Ext1 Ext2

. . . . . .

. . .

Inside a Asterisk PBX


138

Call flow

__ast_request_and_dial()[main/pbx.c]

action_originate() [main/manager.c]

init_manager() [main/manager.c]

ast_pbx_outgoing_exten() [main/pbx.c]

First call in this channel?

yes no

ast_pbx_start()[main/pbx.c]

pbx_thread()[main/pbx.c]

__ast_pbx_run() [main/pbx.c]

__ast_pbx_run() [main/pbx.c]

ast_pbx_run()[main/pbx.c]



140

6.8 Streaming

Introduction to streaming Availability Short fetching time Avoid storage overhead Live broadcasting

Architecture and components for streaming Compression scheme Protocol stack for streaming QoS and synchronization mechanism

Open source: Darwin streaming server


141

Architecture and Components for Streaming

Video Files

Audio Files

Transport Protocols

QoS Control

Server

Video Decoder Audio Decoder

Transport Protocols QoS Control

Client

synchronization

Video/Audio raw data

Internet


142

Compression Schemes

Temporal or spatial Temporal – describe changes between frames Spatial – describe similar patterns in a frame

Lossless or lossy Recoverability when decompressed GIF (lossless) and JPG (lossy)

Symmetrical or Asymmetrical The time for compression and decompression According to the load of the server


143

Protocol Stacks of Streaming

TCP UDP

IP

RTSP RTCP RTP

Control plane Data plane

RTSP (Real Time Streaming Protocol) VCR-Style remote control functionalities Establish and control streams

Description file before a session Methods in a session (SETUP, PLAY, TEARDOWN)

Reuse of HTTP concepts (Ex: ASCII messages) HTTP

Mostly by buffering Low performance HTTP


144

QoS and Synchronization Mechanism QoS mechanism

Source-based rate control Receiver-based rate control

Synchronization mechanism Intra-stream synchronization

Continuity of audio/video data frames inside a stream

Inter-stream synchronization Consistency between cooperative streams

Inter-object synchronization Consistency between streams and other objects

like text and still images


145

Open Source: Darwin Streaming Server Open source version of Apple's QuickTime

Streaming Server (QTSS) Deliver streaming media with RTP and RTSP Provide both live and on-demand programs Supports many formats: H.264/MPEG-4 AVC,

MPEG-4 Part 2, 3GP and MP3 Provide Web-based administration, authentication,

relay support, and integrated broadcaster administration, etc.


146

Block Diagram

Two parts: core server and modules Core server provides task scheduling Module provides specific functions


147

Data Structures


148

Call Flow: Task Handling


149

RTSP Handling State Transition Diagram


150

6.9 Peer-to-Peer Applications (P2P)

Introduction P2P Architectures Performance Issues of P2P Applications Case Study: BitTorrent Open source: Vuze (BitTorrent Client)


151

Introduction to P2P A distributed network architecture in which

participants act as both a client and a server Participants construct a virtual overlay network at

the application layer on top of the underlying IP network

P1

R1

R2 R3 P2

P1

P2

Physical Network

Overlay Network

Node: peers

Edge: TCP/UDP connection


Introduction to P2P

Operations in P2P systems Joining the P2P overlay network, Resource discovery

The most challenging problem for P2P applications Resource retrieval


P2P Architectures

Way of forming a P2P overlay network could be classified into three categories Centralized Decentralized and unstructured Decentralized but structured


Centralized P2P

Operation overview


Directory

Peer

1. upload index

Peer Peer

1. upload index2. Query&Response

3. Download file

Centralized P2P Utilizes a central directory server for locating

objects in the P2P network Peers join by registering to the directory server Peers inform the directory server of objects to be

shared A peer sends query to the directory server to

search an object The peer receives reply from the directory server The peer selects one or more peer in the reply to

download the object directly from Example: Napster


Centralized P2P

Advantages Simple Easy to implement Support various kinds of search such as keyword,

full-text, and metadata search Disadvantages

Not a true P2P system as it relies on a central directory server which becomes performance bottleneck, single point of failure, not scalable and vulnerable to DoS attacks


Decentralized and Unstructured P2P To get rid of server, this approach floods query

messages to peers to search for shared objects Limited-scope flooding is adopted to reduce flooding

messages A query hit message is returned along the reverse

path back to the inquirer


Query

Query

QueryQuery

Query

Query Hit

Query Hit

Query Hit

Download

Example: Gnutella

Decentralized and Unstructured P2P Join procedure

A peer needs to know at least one of the peers already on the overlay network.

The peer sends a join message to a peer already on the overlay.

The existing peer then replies its identity as well as a list of its neighbors. It may also forward the join message to its neighbors.

Upon receiving join reply messages, the newcomer knows more peers on the overlay.


Decentralized and Unstructured P2P Hierarchical overlay with super peers

Flooding is apparently not scalable FastTrack adopts a hierarchical overlay A super peer acts as a local directory database

which stores the indexes of objects shared by ordinary peers

Two-level hierarchical overlay The lower level adopts the central

server approach The upper level (super peers) adopts the

decentralized and unstructured approach.


query reply

Decentralized and Unstructured P2P Advantages

Fully distributed Reliable, fault-tolerant No single point of failure

Disadvantages Excessive query traffic make it not scalable May fail to find content that is actually in the

system Super peer may become overloaded or been

attacked


Decentralized but Structured

Combine the distributed directory service with an efficient query routing scheme

Key ideas For distributed directory service, a hash function

maps peers and objects into the same address space so that objects can be deterministically assigned to peers in a distributive manner.

For efficient query routing, peers are organized into a structured overlay based on their positions in the address space.



Operations overview Each peer generates its own node ID by a predefined

hash function. For each object being held and to be shared, the peer

generates the object ID by the same or another hash function.

For each object, the peer sendd a register message to the node that has the node ID same as the object’s ID.

If a peer wants to query an object, it uses the hash function to generate the object ID and sends the query message to the node that hosts the object’s ID.



Message routing (use Chord as an example) Key idea: have each peer maintain a specially

designed routing table such that every peer could forward the arriving message to a neighboring peer with node ID that is further closer to the destination.

Consider a 10-node Chord overlay in a 6-bit address space

Chord views its address space as a one-dimensional circular space such that peers in the space form a ring overlay.


Message Routing in Chord

The routing table in Chord is called a finger table.

For an m-bit address space, the finger table of a node with ID=x consists of at most m entries and the i-th entry points to the first node with ID following the ID of x+2i-1 modulo 2m, for 1≤i≤m.


Finger Table of Chord Finger table of node N8, where m =6.


N8

N15

N20

N30N38

N42

N47

N51

N56 +1

+2

+4

+8

+16+32

Finger table

N8+1 N15

N8+2 N15

N8+4 N15

N8+8 N20

N8+16 N30

N8+32 N42

N1

Routing a Query Message Routing a query message for object 54 from N8


N1

N8

N15

N20

N30N38

N42

N47

N51

N56

Finger table

N8+1 N15

N8+2 N15

N8+4 N15

N8+8 N20

N8+16 N30

N8+32 N42

Finger table

N42+1 N47

N42+2 N47

N42+4 N47

N42+8 N51

N42+16 N1

N42+32 N15

Finger table

N51+1 N56

N51+2 N56

N51+4 N56

N51+8 N1

N51+16 N8

N51+32 N20

K54lookup(54)

167

Performance Issues of P2P Applications

Free Riding Flash Crowd Topology Awareness NAT Traversal Churn Security Copyright Infringement

Copyright Reserved 2010 167Chapter 6: Application Layer

Free Riding Scalability of P2P systems relies on the

contribution from peers If a peer only consumes but contributes little or no

resources, it becomes a free rider of the system. 85% of peers share no files in Gnutella in 2005

A common solution is to implement some incentive mechanisms. For example, tit-for-tat in BitTorrent.

Other solutions: reward-based and credit-based mechanisms


Flash Crowd

A sudden, unanticipated growth in the demand of a particular object e.g., a new release of a DVD video or mp3 file

Issues how to deal with a sudden large amount of query

messages how long to find and download the object within a

short time period Solutions

Cache, duplicating popular objects


Topology Awareness

A virtual link in a P2P overlay could be a long end-to-end connection across continents or a short one within a local area network How to avoid serious topology mismatch

Solutions Many route-proximity and neighbor-proximity

enhancements for P2P overlay systems have been proposed based on RTT measurement, preference of routing domain or ISP, or geographical information.


NAT Traversal

Basic requirement for a P2P system is to provide peers with NAT traversal mechanisms If both peers are behind NAT devices, they cannot

connect to each other without help from other peers or STUN servers

Solutions In most cases, NAT traversal is solved by relay

peers or super peers that have public IP addresses


Churn

Churn refers to the phenomenon that peers dynamically join and leave the system at will. high churn rate seriously affects the stability and

scalability of a P2P system. e.g., a high churn rate may cause a tremendous

overlay maintenance overhead and dramatic routing performance degradation in DHT-based system

Solutions Avoid a rigid structure or relation among peers Peers maintain a list of potential neighbors for quick

and dynamic neighbor replacement


Security

Security issues P2P programs with back hole (Trojan Horse),

spurious content, leaking of files not to be shared. Solutions to content pollution

protect the content with message digest such as MD5 In BitTorrent, the MD5 digest of each piece of a shared

file is stored in the metadata file peer reputation system object reputation system


Copyright Infringement

Sharing copyrighted objects through P2P systems is a serious problem which hinders the promotion and usage of P2P systems.

Not only P2P users are responsible for copyright infringement, so are the companies that host P2P applications Especially in the case where P2P systems will not

be able to exist without their serves, e.g., Napster.


175

Case Study: BitTorrent BitTorrent (BT) is originally designed by Bram Cohen in

2001 Well thought-out protocol with several unique features

use tit-for-tat as an incentive mechanism to cope with free riders

use out-of-band search to avoid copyright infringement issue use pull-based swarming for load balancing use hash check to prevent propagation of spurious pieces after a peer has successfully downloaded a file, it becomes a

seeder to distribute the file


BT Terminologies

A file is cut into pieces of a fixed size. A piece is further divided into chunks, the basic

data unit for a peer to request for content. The integrity of a piece is protected by a SHA-1 A peer becomes a seeder if it has successfully

downloaded the file. A tracker for each file or group of files to be

shared. The tracker tracks the downloading peers and seeds, and coordinates the file distribution among peers.


BT Operation Overview


`

Web page with Link

to .torrent

Web Server

Tracker

Seeder

`

`

`

Source

publish

Downloading peer

Downloading peer

1. get .torrent 2. get

announce

3. response peer

list

4. get piece

5. get piece

Piece Selection

Random first piece selection For the first few pieces, the client just randomly

selects a piece to download. Rarest first policy

Selects the most scarce piece to download first End-game mode

To speed up the completion of a file download at the end, a peer with only a few pieces missing will send requests for all missing pieces to all the peers


Peer Selection Choking/unchoking

Choking refers to a temporal refusal to upload to a peer. At the beginning, all peers are chocked Tit-for-tat algorithm selects a fixed number of peers from which the

peer downloaded most to unchoke

Optimistic unchoking new peer needs to move its first step when initially joined the

system select one peer at random

Anti-snubbing If a peer is choked by all of its peers (snubbed), it is better to run

optimistic unchoking more often to explore more peers that are willing to cooperate.


180

Open source: Vuze (BitTorrent Client)

Many BT client programs are open-source Some popular client programs

Vuze uTorrent BitComet


Core Packages

Most of Vuze’s core packages are located under the .\com\aelitis\azureus\core directory


Data Structure

Classes for the peer and piece objects


Algorithm Implementation

Main program for controlling the piece and peer selection is the PEPeerControlImpl class


Implementation of Peer and Piece Selection Unchoking algorithms are implemented in

DownloadingUnchocker.java and SeedingUnchocker.java tit-for-tat is implemented in calculateUnchokes() Optimistic unchoking is implemented in

UnchokerUtil.getNextOptimisticPeer() getRequestCandidate() defined in

PiecePickerImpl.java is the core method for deciding which block to download


185

6.10 Summary

General issues well-known ports, concurrent connection-oriented

servers and iterative connectionless servers, statefulness/statelessness

Major application protocols DNS, SMTP, POP3, IMAP4, HTTP, FTP, SNMP

Real time, multimedia application protocols SIP, RTP, RTCP, RTSP

P2P protocols Napster, Gnutella, DHT, Chord, BT
