1
Computer NetworksAn Open Source Approach
Chapter 6: Internet Services
Chapter 6: Application Layer
2
Content
6.1 Issues 6.2 Domain Name System 6.3 E-Mail 6.4 World Wide Web 6.5 File Transfer Protocol 6.6 Simple Network Management Protocol 6.7 Voice over IP (VoIP) 6.8 Streaming 6.9 Peer-to-Peer Applications (P2P) 6.10 Summary
Chapter 6: Application Layer
3
6.1 Issues
How ports work? How servers start? Classification of servers Characteristics of application layer protocols
Chapter 6: Application Layer
4
How Ports Work?
Well known ports: 0~1023 Registered ports: 1024~49151 Dynamic and/or private ports: 49152~65535
FTP Daemon
Mail Daemon
Telnet Daemon
HTTP Daemon
21
ListeningPorts
23
25
80
Server Daemons
Server MachineClient 1 Machine
FTP Client
Web Browser
2880
8752
User AgentOutbound
Port
User AgentOutbound
Port
Client 2 Machine
Chapter 6: Application Layer
5
How Servers Start?
Is a server better to run as standalone or via (x)inetd? Standalone:
requires explicit restarting for configuration changes to take effect
crashed standalone server will probably stay unnoticed and the service will become unavailable
better performance (x)inetd:
reads configuration files each time a client attempts to access their information
fresh copies will be spawned on demand
Chapter 6: Application Layer
6
Classification of Servers
Concurrent connectionless server Concurrent connection-oriented server Iterative connectionless server Iterative connection-oriented server
Chapter 6: Application Layer
Underlying Transport ProtocolsApplication Application layer protocol Underlying transport protocol
electronic mail SMTP, POP3, IMAP TCP
remote terminal access Telnet TCP
file transfer FTP TCP
web HTTP TCP
web caching ICP typically UDP
name resolution DNS typically UDP
network file system NFS typically UDP
network management SNMP typically UDP
routing protocol RIP typically UDP
Internet telephony proprietary (e.g. Vocaltec) typically UDP
streaming multimedia proprietary (e.g. RealNetworks) typically UDP
P2P proprietary (e.g. BitTorrent, eDonkey) UDP for queries and TCP for data transfer
Chapter 6: Application Layer 7
8
Iterative Connectionless Server socket
bind
recvfrom
sendto
socket
sendto
recvfrom
close
repeatinfinitely
repeat asneeded
Clients
Server
process
requests
responses
Chapter 6: Application Layer
9
Concurrent Connection-Oriented Server socket
bind
listen
accept
fork
close(accepting)
close(listening)
read
process
write
Close(accepting)
socket
connect
write
read
close
repeatinfinitely
repeat asneeded
repeat asneeded
connection request
request
reply
Client
Server
child process
server process
Chapter 6: Application Layer
10
Characteristics of Application Layer Protocols Variable message formats and length
Message formats and length of application layer protocols vary with different applications and requirements
Variable data types Messages can be transmitted in textual or non-textual
formats. E.g., a Web server replies textual Web pages and binary
images Statefulness
The server retains information of the session with the client E.g., an FTP server remembers the client’s current working
directory
Chapter 6: Application Layer
11
6.2 Domain Name System
Domain name space and domain delegation Zones and name servers Zone data files and resource records Resolvers and Name resolution Open source: BIND
Chapter 6: Application Layer
12
Top Level DomainsDomain Description
com Commercial organizations, such as Intel (intel.com).
org Non-profit organizations, such as WWW consortium (w3.org).
gov Government organizations, reserved for U.S government such as National Science Foundation (nsf.gov).
edu Educational organizations, such as UCLA (ucla.edu).
net Networking organizations, such as Internet Assigned Numbers Authority which maintains the DNS root servers (gtld-servers.net) .
int Organizations established by international treaties between governments. For example, International Telecommunication Union (itu.int).
Mil Reserved exclusively for the United States Military. For example, Network Information Center, Department of Defense (nic.mil).
Two-letter country code
The two-letter country code top level domains (ccTLDs) are based on the ISO 3166-1 two-letter country codes. Examples are tw (Taiwan), uk (United Kingdom).
arpa Mostly unused now, except for the in-addr.arpa domain, which is used to maintain a database for reverse DNS queries.
Others Such as .biz (business), .name (for individuals), .info (similar with .com).
Chapter 6: Application Layer
13
Domain Delegation
root domain
comedutw
educomorg
nctunthu nsysu
ciscsie ee
cis.nctu.edu.tw
uk
co
Easier managementLoad distribution
Chapter 6: Application Layer
14
Zones and Name Servers Differences between a zone and a domain
A zone contains some part of the domain name space A domain is a subtree in the domain name space A name server may be authoritative for multiple zones
Zone data files in a name server Contains Resource Records (RR) describing all the hosts
within that zone Six major types of RR: “SOA”, “NS”, “A”, “PTR”, “CNAME” and
“MX”
Master and slave name servers Availability and load balance Data duplication by zone transfer
Request from slave Notify by master
Chapter 6: Application Layer
15
Master/Slave Name Servers
slave
1. Redundancy for heavy load
2. Availability achieved
mastermaster (A) | slave (B)
zone A zone B
df of A
df: zone data file
df of Bdf of A and B
A large domain name space
zone transfer
zone transfer
clients clients
Chapter 6: Application Layer
16
Resource Records SOA (Start Of Authority)
Specify a best name server to be authoritative (master) for a zone
Ex: cis.nctu.edu.tw. 86400 IN SOA cisserv.cis.nctu.edu.tw.
Options for refresh, retry, expire ..etc are supported NS (Name Server)
Specify name servers for a domain Ex: cis.nctu.edu.tw. 86400 IN NS cisserv.cis.nctu.edu.tw.
A (Address) Map names to addresses, Example (multi-homed):
linux.cis.nctu.edu.tw. 86400 IN A 140.113.168.127linux.cis.nctu.edu.tw. 86400 IN A 140.113.207.127
Chapter 6: Application Layer
17
Resource Records (cont) CNAME (Canonical Name)
Create name-to-name alias Ex: www.cis.nctu.edu.tw. 86400 IN CNAME cache.cis.nctu.edu.tw.
PTR (Pointer) Point addresses to names (canonical name only) Ex: 10.23.113.140.in-addr.arpa. 86400 IN PTR laser0.cis.nctu.edu.tw.
MX (Mail Exchanger) Provide mail-routing information The one of best (lowest) preference value is chosen
cis.nctu.edu.tw. 86400 IN MX 0 mail.cis.nctu.edu.tw.cis.nctu.edu.tw. 86400 IN MX 10 mail1.cis.nctu.edu.tw.
Chapter 6: Application Layer
18
Name Resolution
Query type Forward query – name-to-address Reverse query – address-to-name
Resolution method Iterative (by referral, commonly adopted) Recursive (site-by-site recursion)
Caching for speeding up the lookup process Expire according to TTL Tradeoff between performance and consistency
Chapter 6: Application Layer
19
Name Resolution (Iterative)
client (resolver
)
local name server
query for “www.dti.gov.uk”
root name server
1
2
3
4
6
5
“uk”name server
”gov.uk”name server
other candidate name servers
“dti.gov.uk”name server
7
8
www.dti.gov.uk. 5M IN A 164.36.253.20www.dti.gov.uk. 5M IN A 164.36.164.20
Chapter 6: Application Layer
20
Message Format
ID
QR
Opcode AA
TC
RD
RA
Reserved Rcode
QDCOUNT
ANCOUNT
NSCOUNT
ARCOUNT
Question
Answer
Authority
Additional
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 bit
Header
Chapter 6: Application Layer
21
Execution Flowchart of Named
Cache initialization
Wait for signals
Command line parsing
chroot and daemonize
Initialize server configurations and data
structures
Create managers
Install a handler for signals
1. Task manager
2. Timer manager
3. Socket manager
For possible restart or termination signals
Chapter 6: Application Layer
23
Basic Data Structure in Named
view1 …viewlist
zone_table
zone1
zone3 zone2
zoneN
zone_table
zone3
zone4 zone8
…Red/Black Tree
for zones
RR1
RR2
RRN
Red/Black Tree of RRs for zone4
zone_table
…
viewNview2 NULL
Chapter 6: Application Layer
24
Domain Information Gopher (DIG); <<>> DiG 9.2.0 <<>> www.nctu.edu.tw
;; global options: printcmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26027;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3
;; QUESTION SECTION:;www.nctu.edu.tw. IN A
;; ANSWER SECTION:www.nctu.edu.tw. 259200 IN A 140.113.250.5
;; AUTHORITY SECTION:nctu.edu.tw. 259200 IN NS ns.nctu.edu.tw.nctu.edu.tw. 259200 IN NS ns2.nctu.edu.tw.nctu.edu.tw. 259200 IN NS ns3.nctu.edu.tw.
;; ADDITIONAL SECTION:ns.nctu.edu.tw. 259200 IN A 140.113.250.135ns2.nctu.edu.tw. 259200 IN A 140.113.6.2ns3.nctu.edu.tw. 259200 IN A 163.28.64.11
Chapter 6: Application Layer
25
6.3 E-Mail
Introduction Message Formats Internet Mail Protocols Open Source: Qmail
Chapter 6: Application Layer
26
Introduction
What is E-mail? A method of sending messages from one user to
another via computer networks Internet mail addressing:
format: [email protected] e.g. [email protected]
Chapter 6: Application Layer
27
Introduction (cont.)
Components of Internet mail system: Mail User Agent (MUA) Mail Transfer Agent (MTA) Mail Delivery Agent (MDA) Mail Retrieval Agent (MRA)
Sending MUA Forwarding MTA Receiving MTA
TCP MDA Local MDA
MRA
Receving MUA
Mailbox
SMTP
System Call
SMTP
System Call
Write
Read/Write
POP/IMAP
Sender's Machine Local Mail Server Remote Mail Server Recipient's Machine
Chapter 6: Application Layer
28
Message Formats
Internet Message Format (RFC 822) Multipurpose Internet Mail Extensions (RFC
2045-2049)
Chapter 6: Application Layer
29
RFC 822 – Internet Message Format A message consists of:
An envelope which contains information needed to accomplish transport and delivery
Contents which compose the object to be delivered to the recipient
Chapter 6: Application Layer
30
RFC 822 – Internet Message Format (cont.) Common message header fields:
Type Field Meaning
From: The person(s) who wished this message to be sent Originator
Reply-To: Provides a general mechanism for indicating any
mailbox(es) to which responses are to be sent
To: The primary recipients of the message
Cc: The secondary recipients of the message
Receiver
Bcc: Additional recipients of the message
Received: A copy of this field is added by each transport service
that relays the message
Trace
Return-Path: This field is added by the final transport system that
delivers the message to its recipient
Message-ID: Contains a unique identifier generated by the mail
transport on the originating system
Reference
In-Reply-To: Previous correspondence which this message
answers.
Other Subject: Provides a summary, or indicate the nature, of the
message.
Date Date: Supplies the date and time the mail was sent
Extension X-anything: It is used to implement additional features that have
not yet made it into an RFC, or never will.
Chapter 6: Application Layer
31
RFC 822 – Internet Message Format (cont.) An example of message header:
From: "Dr. Ying-Dar Lin" <[email protected]>
Date: Mon, 30 Jul 2001 02:42:34 +0800 (CST)
Subject: paper
Return-Path: <[email protected]>
Received: (from root@localhost) by mail.cis.nctu.edu.tw (8.11.4/8.11.4) id f6TIgct33454 for [email protected]; Mon, 30 Jul 2001 02:42:38 +0800 (CST) (envelope-from [email protected])
Received: from cissun53.cis.nctu.edu.tw ([email protected] [140.113.23.53]) by mail.cis.nctu.edu.tw (8.11.4/8.11.4av) with ESMTP id f6TIgaI33445 for <[email protected]>; Mon, 30 Jul 2001 02:42:36 +0800 (CST) (envelope-from [email protected])
Received: (from ydlin@localhost) by cissun53.cis.nctu.edu.tw (8.9.3/8.9.3) id CAA16191 for juvenia; Mon, 30 Jul 2001 02:42:34 +0800 (CST)
Message-Id: <[email protected]>
X-Virus-Scanned: by AMaViS perl
X-UIDL: F5%#!88!"!>RG"!l!G!! Chapter 6: Application Layer
32
MIME What is MIME?
MIME is a specification for enhancing the capabilities of conventional Internet message format .
MIME enables email messages to have: textual header information and message bodies in
character sets other than 7-bit ASCII multiple objects within a single message binary or application-specific files multimedia files, such as images, audio, and video
files
Chapter 6: Application Layer
33
MIME (cont.)
MIME message header:
Field Description
MIME-Version: Describes the version of the MIME message
format
Content-Type: Describes the MIME content type and subtype
Content-Transfer-Encoding: Indicates the encoding method for transmission
Content-ID: Allows a body of information to refer to another
Content-Description: Possible description for a body of information
Chapter 6: Application Layer
34
MIME (cont.)
The MIME content type set:
Type Subtype(s) Important parameters
text plain, html charset
multipart mixed, alternative, parallel,
digest
boundary
message RFC 822, partial,
external-body
Id, number, total, access-type,
expiration, size, permission
application octet-stream, postscript, rtf,
pdf, msword
type, padding
image jpg, gif, tiff, x-xbitmap none
audio basic, wav none
video mpeg none
Chapter 6: Application Layer
35
MIME (cont.)
Possible values for the Content-Transfer-Encoding: field: Quoted-Printable Base64 7bit 8bit Binary X-Encoding
Chapter 6: Application Layer
36
MIME (cont.)
An example of MIME message:
From: 'Ching-Ming Tien' <[email protected]>
Subject: Cover
MIME-Version: 1.0
Content-Type: image/jpg;
name=cover.jpg'
Content-Transfer-Encoding: base64
Content-Description: The front cover of the book
<.....base64 encoded jpg image of cover...>Chapter 6: Application Layer
37
Internet Mail Protocols
Simple Mail Transfer Protocol (RFC 2821) Post Office Protocol (RFC 1939) Internet Message Access Protocol (RFC
2060)
Chapter 6: Application Layer
38
Simple Mail Transfer Protocol What is SMTP?
A standard host-to-host mail transport protocol and traditionally operates over TCP on port 25
Chapter 6: Application Layer
39
Simple Mail Transfer Protocol (cont.) Important SMTP commands:
SMTP replies:
Command Description
HELO Greet the receiver with the name
MAIL FROM: Indicates the sender, could be spoofed too
RCPT TO: Indicates the recipient
DATA Indicates the mail data, terminated by a " . " in a single line
RSET Reset the session
QUIT Close the session
Chapter 6: Application Layer
Response Description
2xx Command accepted and processed.
3xx General flow control.
4xx Critical system or transfer failure.
5xx Errors with the SMTP command.
41
Post Office Protocol
What is POP? A protocol used to retrieve e-mail from a mail server.
POP3 Session States: 1. Authorization:
Must log in with password before entering transaction state.
2. Transaction: Client can request actions of server, get mail for example.
3. Update: Updates mail box to reflect actions taken in transaction
state.
Chapter 6: Application Layer
42
Post Office Protocol (cont.) Minimal POP3 commands:
Command Description Session state
USER name Identifies the user to the server AUTHORIZATION
PASS string Enters user password AUTHORIZATION
STAT Gets the number of messages in and
octet size of maildrop
TRANSACTION
LIST [msg] Gets the size of one or all messages TRANSACTION
RETR msg Retrieves a message from the maildrop. TRANSACTION
DELE msg Marks the msg as deleted from the
maildrop.
TRANSACTION
NOOP No operation. TRANSACTION
RSET Resets all messages that are marked as
deleted to unmarked.
TRANSACTION
QUIT Terminates the session. AUTHORIZATION,
UPDATE
Chapter 6: Application Layer
43
Post Office Protocol (cont.)
POP3 replies: +OK, -ERR
Example POPsession:
Chapter 6: Application Layer
44
Internet Message Access Protocol What is IMAP?
A replacement for the POP3 protocol Differences between IMAP4 and POP3:
IMAP4 allows messages being stored and manipulated on the mail system
POP3 only allows users to download their messages and store and manipulate messages on the client’s machines
Chapter 6: Application Layer
45
Internet Message Access Protocol (cont.) Four states in the IMAP4 server:
Non-authenticated state When a connection is established between the server and client
Authenticated state When a pre-authenticated connection starts, When acceptable authentication credentials have been provided After an error in selecting a mailbox
Selected state When a mailbox has been successfully selected
Logout state When the client asks to exit the server
Chapter 6: Application Layer
46
Internet Message Access Protocol (cont.) IMAP4 command summary:
Session state Commands
Any CAPABILITY, NOOP, LOGOUT
Non-authenticated AUTHENTICATE, LOGIN
Authenticated SELECT, EXAMINE, CREATE, DELETE, RENAME,
SUBSCRIBE, UNSUBSCRIBE, LIST, LSUB,
STATUS, APPDNED
Selected CHECK, CLOSE, EXPUNCGE, SEARCH, FETCH,
STORE, COPY UID
Chapter 6: Application Layer
47
Internet Message Access Protocol (cont.) An IMAP4 transaction scenario:
Chapter 6: Application Layer
49
Open Source: qmail
Introduction to qmail: qmail is a secure, reliable, efficient, simple MTA
designed for Unix-like operating systems qmail is a replacement for the sendmail qmail is the second most common SMTP server qmail has by far the fastest growth of any SMTP
server
Chapter 6: Application Layer
50
qmail System Structure
Core modules of qmail:Module Description
qmail-smtpd Receive a message via SMTP
qmail-inject Preprocess and send a message
qmail-queue Queue a message for delivery
qmail-send Deliver messages from the queue
qmail-clean Clean up the queue directory
qmail-lspawn Schedule local deliveries
qmail-local Deliver or forward a message
qmail-rspawn Schedule remote deliveries
qmail-remote Send a message via SMTP
qmail-pop3d Distribute message via POP3
Chapter 6: Application Layer
51
qmail Data Flow
qmail-smtpd qmail-inject
qmail-queue
qmail-send qmail-clean
qmail-lspawnqmail-rspawn
qmail-localqmail-remote
SMTP
MDA
qmail-pop3d
mailbox/maildir
local mailclient
MRA
remote mailserver
remote mailserver
remote mailclient
MUA
qmail system
MUA
remote mailclient
MTA
MUA
SMTP POP3
MTA
MTA
Chapter 6: Application Layer
52
qmail Control Files
Control files are in /var/qmail/control Some control files of qmail:
Control Default Used by Description
me FQDN of
system
various Default for many control files
rcpthosts (none) qmail-smtpd Domains that qmail accept
message for
locals me qmail-send Domains that qmail deliver locally
defaultdomain me qmail-inject Default domain name
plusdomain me qmail-inject Added to any host name that
ends with a plus sign
virtualdomains (none) qmail-send Virtual domains and users
Chapter 6: Application Layer
53
qmail Queue Structure
The qmail queue directory is in /var/qmail/quque Subdirectories in the qmail queue and contents :
Subdirectory Contents
bounce Permanent delivery errors
info Envelope sender addresses
intd Envelopes under construction by qmail-queue
local Local envelope recipient addresses
lock Lock files
mess Message files
pid Used by qmail-queue to acquire an i-node number
remote Remote envelope recipient addresses
todo Complete envelopes
Chapter 6: Application Layer
55
6.4 World Wide Web
Introduction Web Naming and Addressing Web Data Formats Hypertext Transfer Protocol Open Source: Apache
Chapter 6: Application Layer
56
Introduction
What is WWW? A system of Internet servers that support specially
formatted documents. How does it work?
WebBrowser
Web Server
DNS Server1: DNS query
2: TCP 3-way handshake
3: HTTP request
4: HTTP response
Chapter 6: Application Layer
57
Web Naming and Addressing
Uniform Resource Identifier (RFC 2396) Uniform Resource Locator (RFC 1738) Uniform Resource Name (RFC 2141)
http: ftp: gopher: etc.
urn:URLs
URNs
URNs
Chapter 6: Application Layer
58
Uniform Resource Identifier What is URI?
A compact string of characters for identifying an abstract or physical resource.
URI syntax: Absolute URI: <scheme>:<scheme-specific-part> Generic URI: <scheme>://<authority><path>?<query>
URI examples: http://speed.cis.nctu.edu.tw/~ydlin/index.html#Books http://www.google.com/search?q=linux ftp://ftp.cis.nctu.edu.tw/Documents/IETF/rfc2300~2399/rfc2396.txt mailto: [email protected] news: comp.os.linux telnet://bbs.cis.nctu.edu.tw/ ../icons/logo.gif
Chapter 6: Application Layer
59
Uniform Resource Locator
What is URL? A compact string representation of the location for a
resource that is available via the Internet URL syntax:
<service>//<user>:<password>@<host>:<port>/<url-path>Service Description
ftp File Transfer protocol
http Hypertext Transfer Protocol
gopher The Gopher protocol
mailto Electronic mail address
news USENET news
nntp USENET news using NNTP access
telnet Reference to interactive sessions
wais Wide Area Information Servers
file Host-specific file names
prospero Prospero Directory Service
Chapter 6: Application Layer
60
Uniform Resource Locator (cont.) Some URL examples:
http://www.cis.nctu.edu.tw/chinese/ccg/titleMain.gif ftp://john:[email protected]/projects/book.txt nntp://news.cis.nctu.edu.tw/cis.course.computer-
networks/5238 telnet://mail.cis.nctu.edu.tw:110/
Chapter 6: Application Layer
61
Uniform Resource Name
What is URN? A name that identifies a resource of unit of
information independent of its location URN syntax:
<URN> ::= "urn:" <NID> ":" <NSS> NID: Namespace Identifier NSS: Namespace Specific String
URN examples: urn:path:/A/B/C/doc.html urn:ans:cis.nctu.edu.tw/ydlin/Resource urn:isbn:0-201-56317-7
URN resolutioin: http://www.isbn.com/0-201-56317-7
Chapter 6: Application Layer
62
Web Data Formats
Evolution of Web Data Formats SGML HTML XML XHTML
SGML
HTML XML
XHTML
Chapter 6: Application Layer
63
Standard Generalized Markup Language What is SGML?
A system for organizing and tagging elements of a document.
Characteristics of SGML Descriptive Markup Types of Document Data Independence
Chapter 6: Application Layer
64
HyperText Makeup Language What is HTML?
The authoring language used to create documents on the World Wide Web.
A brief history of HTML HTML 2.0, 3.0, 3.2, 4.0
HTML document structure HTML version information Header Body
Chapter 6: Application Layer
65
Extensible Markup Language
What is XML? A pared-down version of SGML, designed
especially for Web documents. Why XML? How to use XML?
Traditional data processing Document-driven programming (DDP) Archiving Binding
Chapter 6: Application Layer
66
Extensible HyperText Markup Language What is XHTML?
A hybrid between HTML and XML specifically designed for Net device displays.
Why XHTML? Using XHTML with other W3C tag sets:
XHTML for structural markup of documents SMIL for multimedia MathML for mathematics SVG for scalable vector graphics XForms for smart web forms
Chapter 6: Application Layer
67
Hypertext Transfer Protocol
What is HTTP? HTTP Conversation Client Request Server Response HTTP 1.1
Chapter 6: Application Layer
68
What is HTTP?
A comprehensive addressing scheme Client-Server Architecture The HTTP protocol is connectionless and
stateless An extensible and open representation for
data types
Chapter 6: Application Layer
69
HTTP Conversation
Client request Request headers Request body Server status Response headers Requested data Disconnected
Chapter 6: Application Layer
70
Client Request
HTTP request methods:
Method Meaning OPTIONS A request for information about the communication options available
for the specified URI. GET Requests a document from the server. HEAD Like GET, except only the headers are returned. POST Sends data to some handler indicated by the URI. PUT Requests that the data in the body section be stored at the specified
URI. DELETE Requests that the specified resource be deleted. TRACE For debugging purposes; lets the client see what’s being received on
the other end. CONNECT Reserved for future use.
Chapter 6: Application Layer
72
Server Response
Response example:
Server status codes:Class of code Meaning 1xx Informational. 2xx Indicates that a client request was completed successfully. 3xx Indicates that the request was redirected for some reason. 4xx Indicates that there was an error on the client end. 5xx Indicates that there was an error on the server end.
Chapter 6: Application Layer
74
HTTP 1.1
HTTP 1.1 features: Persistent connections Chunked encoding byte ranges Caching Non-IP virtual hosting
Chapter 6: Application Layer
Web Caching
Web caching is a mechanism to expedite document downloading in WWW.
To achieve maximum satisfaction from Web caching, some aspects need to be considered. Candidates to be cached Content replacement Cache coherence
75Copyright Reserved 2010 75Chapter 6: Application Layer
Transparent Proxy
A cache server can also act as a proxy server With port redirection, transparent proxy does
not require manual configuration Two types of transparent proxy
integrated within a gateway standalone server box
HTTP requests
Squid proxy/cache server
iptables
gateway
router/switchHTTP requests
Squid box
policy route or switch rules
(1)
(2)
dest. port = 80
dest. port = 80
76Copyright Reserved 2010 76Chapter 6: Application Layer
77
Open Source: Apache
Introduction to Apache: Open-Source Web server originally based on
NCSA server Available on over 160 varieties of Unix -- and
Windows NT Over 58% of Internet Web servers run Apache or
an Apache derivative
Chapter 6: Application Layer
Ration of sendfile ( ) to ap_invoke_handler ( )
File size 1 kb 4 kb 16 kb 64 kb 256 kb 1024 kb
# of calling sendfile ( ) 1 1 1 2 7 15
Total execution time (μs) of sendfile ( )
37 37 42 78 215 527
Time ratio of sendfile ( ) to ap_invoke_handler ( )
35% 38% 40% 53% 77% 87%
Chapter 6: Application Layer 84
85
Apache Server Life Cycle
On Unix systems, Apache creates multiple processes to handle requests.
The Windows and OS/2 ports are multithreaded..
Chapter 6: Application Layer
87
6.5 File Transfer Protocol File transfer service Operation model Inside the connections Open source: wu-ftpd
Chapter 6: Application Layer
88
File Transfer Service Goal
File sharing Data replication for backup Efficiency and reliability during transmission
Use client-server model based on TCP/IP Authenticated and anonymous accesses Relationship between FTP and Telnet Access an ftp server with browser
ftp://ftp.cis.nctu.edu.tw (anony) ftp://[email protected] (authen)
Chapter 6: Application Layer
89
Some Application FTP Commands
Command Description
OPEN Connect to a remote host
CAT View a file in a remote host
GET Retrieve files in a remote host
RENAME Change the name of a file in a remote host
RM Delete a file in a remote host
QUIT Terminate an FTP session
Chapter 6: Application Layer
90
Operation Model
Client as both control host and receiver
Control connection (initiated by client)
FTP commands"PORT h1,h2,h3,h4,p1,p2"
FTP replies
File System File System
ServerClientData connection
(initiated by server)
Send/Receive data
Listen on port "L" (L=21)
Listen on port "p1,p2"
PORT: Send the IP and port of the client to which the data is retrieved
Chapter 6: Application Layer
91
Inside the Connections Establishing control/data connections
Active Mode Control connection initiated by client Data connection initiated by server
Passive Mode When client is behind a firewall Both control/data connections are initiated by client
FTP Reply Example FTP session Error Recovery
Chapter 6: Application Layer
92
Active/Passive ModeActive mode
Passive mode
client server
Request (Ask server to listen)
Reply(Listening on port P of host
H)
Initiate a data connection
firewall
client server
Initiate a data connection
firewall
Request(Connect to me at port P of host
H)
Control
Data
Chapter 6: Application Layer
93
Some Proper FTP Commands
Command Description Type
USER Send the user name Access Control
PASS Send the password Access Control
PORT Send the IP and port of the client to which the data is retrieved
Transfer Parameter
PASV Tell the server to listen on a data port rather than initiate a data connection
Transfer Parameter
RETR Ask server to transfer a copy of the requested file to the client
File service
STOR Cause the server to accept and receive the data and store it as a file
File service
RNFR Specify the path of a source file to rename from File service
RNTO Specify the path of a destination file to rename to File service
ABOR Tell the server to abort the previous command and the corresponding data transfer
File service
Chapter 6: Application Layer
94
FTP Reply
Reply Description Type
1yz The requested action is being initiated; expect another reply before proceeding with a new command.
Positive Preliminary reply
2yz The requested action has been successfully completed. Positive Complete reply
3yz The command has been accepted, but the requested action is being held, waiting for further information from another command.
Positive Intermediate reply
4yz The command is not accepted the and the requested action did not take place. The action can be requested again.
Transient Negative Completion reply
5yz Similar with 4yz, except that the error condition is permanent so that the action cannot be requested again.
Permanent Negative Completion reply
Chapter 6: Application Layer
95
Error Recovery
The restart mechanism Sender inserts ‘marker’ (used to identify the checkpoint)
in the data stream Receiver marks the position of the marker and reply the
latest marker position of both sender and receiver to user When error, user issues ‘restart’ with the position of the
marker to the sender
* User (control host) and receiver may/may not exist in the same machine
Chapter 6: Application Layer
96
Example FTP SessionSTATUS:> Connecting to www.cis.nctu.edu.tw (ip = 140.113.166.122)STATUS:> Socket connected. Waiting for welcome message...
220 www.cis.nctu.edu.tw FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready.COMMAND:> USER www
331 Password required for www.COMMAND:> PASS ********
230 User www logged in.COMMAND:> TYPE I
200 Type set to I.COMMAND:> REST 100
350 Restarting at 100. Send STORE or RETRIEVE to initiate transfer.COMMAND:> REST 0
350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.COMMAND:> pwd
257 "/home/www" is current directory.COMMAND:> TYPE A
200 Type set to A.COMMAND:> PORT 140,113,189,29,10,27 tell the server where to connect to
200 PORT command successful. COMMAND:> LIST retrieve directory listing
150 Opening ASCII mode data connection for /bin/ls. File status okay; about to open data connection
…….list of files….
COMMAND:> TYPE I200 Type set to I.
COMMAND:> PORT 140,113,189,29,10,31200 PORT command successful.
COMMAND:> RETR test retrieve the file “test”150 Opening BINARY mode data connection for test (5112 bytes).
Chapter 6: Application Layer
97
Open Source: wu-ftpd Introduction to wu-ftpd
Originally developed at Washington University Most popular ftp daemon Maintained by WU-FTPD Development Group.
Features and configurations Virtual ftp servers On-the-fly compression Important configuration files
Inside wu-ftpd
Chapter 6: Application Layer
98
Inside wu-ftpd
read ACL files
listen to requests
connection accepted?
fork a handler
stand-alone?
fork off (parent exits)
service initialization
reverse DNS check
parse & execute commands
yes
no (under (x)inetd)
yes
no
start ftp server with some options
loop until termination signal is received
Chapter 6: Application Layer
99
Virtual FTP Servers1. Manage two or more FTP servers in one machine
2. Support guest groups for different virtual FTP servers
…
FTP server
Configuration file of each virtual server
clients
ftp.site1.com.tw Lookup the ftpaccess file
ftp.site2.com.tw
ftp.site3.com.tw
# Virtual Server setup for ftp.site1.com.tw virtual ftp.site1.com.tw root /var/ftp/virtual/site1 virtual ftp.site1.com.tw banner /var/ftp/virtual/site1/banner.msg
virtual ftp.site1.com.tw logfile /var/log/ftp/virtual/site1/xferlog # Virtual Server setup for ftp.site2.com.tw…
A rule segment in ftpaccess
Chapter 6: Application Layer
100
On-the-Fly File Compressions Server compress a file (or directory) right before
retrieved by users Example
User ynlin logged in.Logged in to wwwpc.cis.nctu.edu.tw.ncftp /home/ynlin > ls1.tar.gz Desktop/ ucd-snmp-4.2.1/ncftp /home/ynlin > get ucd-snmp-4.2.1.tar.gzucd-snmp-4.2.1.tar.gz: 7393280 bytes 552.83 kB/sncftp /home/ynlin >lls -ldrwxr-xr-x 24 gis88559 gis88 3584 Oct 8 12:18 .drwxr-xr-x 88 root gis88 2048 Sep 10 17:48 ..-rw-r----- 1 gis88559 gis88 7393280 Oct 8 12:18 ucd-snmp-4.2.1.tar.gz
Chapter 6: Application Layer
101
Important Configuration Files
File name Description
ftpaccess Used to configure the operations of the ftp daemon.
ftpconversions Specify the postfix of a retrieved file and its corresponding operations.
ftphosts Used to deny/allow some hosts to login as certain accounts.
ftpservers List the virtual servers and the corresponding directories containing their own configuration files.
Chapter 6: Application Layer
102
6.6 Simple Network Management Protocol
Background Architectural framework MIB SMI SNMP, SNMPv2, SNMPv3 Open source: NET-SNMP
Chapter 6: Application Layer
103
Background Control over network systems are demanded Small tools: ping, traceroute, netstate..etc (base
on ICMP) SNMP:
Remote control without being physically attached to managed entities
Exchange management information between network devices
Short history SNMPv1 (1989)– Management framework SNMPv2 (1993)– Functionality enhancement SNMPv3 (1998)– Security add-on
Chapter 6: Application Layer
104
Architectural Framework
trap or response
management station
MIB
agent & managed
device in one machine
master agent
subagent (managed device )
poll or request
MIB MIB MIB
SNMP
Management station Agent Managed device Managed object (specified in MIB) Management protocols
Protocols simpler than SNMP
Chapter 6: Application Layer
105
Management Information Base (MIB)
Tree-like virtual information store Identified by object identifiers Extensible in “experimental” and “private” branches
Register at IANA (http://www.iana.org/) MIB-II: for network management of TCP/IP-based
internets (RFC 1213)
Chapter 6: Application Layer
106
MIB-II iso (1)
org (3)
dod (6)
….
internet (1)
directory (1)mgmt (2)experimental (3)private (4)
….
mib-2 (1)
system (1)
interface (2)at (3)
icmp (5)
tcp (6)udp (7)egp (8) ip (4)
….
….
transmission (10)
snmp (11)
OBJECT IDENTIFIER: 1.3.6.1.2.1.4
security (5)snmpv2 (6)
….
Chapter 6: Application Layer
107
Object Groups in Mib-2
Group Description
system General information about the managed system
interface Configuration information and statistics of each physical interface
at Address translation between network address and physical address
ip Information of implementation and operation of IP in a local system. For example, routing table, default TTL.
icmp Information about the implementation and operation of ICMP. For example, number of messages ICMP sent and received.
tcp Information about the implementation and operation of TCP. For example, the number of maximum and active connections in the system.
udp Information about the implementation and operation of UDP. For example, the number of datagrams sent.
egp Information about the implementation and operation of EGP.
tranmission Related information and statistics of different transmission schemes.
snmp Information about the accesses and errors of SNMP operations.
Chapter 6: Application Layer
108
Structure of Management Information (SMI) Define the structure of a particular MIB
Place restrictions on the types of the objects allowed in the MIB Abstract Syntax Notation one (ASN.1)
Exchange of information between application components in different systems
Syntax – integer, octet string, object identifier Encode the object values (by Basic Encoding Rule)
Define the way data is represented during transmission
Three categories of data types Simple: Primitive ASN.1 data types Application-wide: Special data types used in particular Applications Simply constructed: table, row
Chapter 6: Application Layer
109
TCP Connection Table (in MIB-II)
-- the TCP Connection table -- The TCP connection table contains information about this -- entity's existing TCP connections. tcpConnTable OBJECT-TYPE SYNTAX SEQUENCE OF TcpConnEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "A table containing TCP connection-specific information." ::= { tcp 13 } tcpConnEntry OBJECT-TYPE SYNTAX TcpConnEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "Information about a particular current TCP connection. An object of this type is transient, in that it ceases to exist when (or soon after) the connection makes the transition to the CLOSED state." INDEX { tcpConnLocalAddress, tcpConnLocalPort, tcpConnRemAddress, tcpConnRemPort } ::= { tcpConnTable 1 } TcpConnEntry ::= SEQUENCE { tcpConnState INTEGER, tcpConnLocalAddress IpAddress, tcpConnLocalPort INTEGER (0..65535), tcpConnRemAddress IpAddress, tcpConnRemPort INTEGER (0..65535) } tcpConnState OBJECT-TYPE SYNTAX INTEGER {
closed(1), listen(2), synSent(3), synReceived(4) established(5), finWait1(6), finWait2(7), closeWait(8),lastAck(9), closing(10), timeWait(11), deleteTCB(12) }
ACCESS read-write STATUS mandatory DESCRIPTION "The state of this TCP connection.." ::= { tcpConnEntry 1 } tcpConnLocalAddress OBJECT-TYPE SYNTAX IpAddress ACCESS read-only STATUS mandatory DESCRIPTION "The local IP address for this TCP connection. In the case of a connection in the listen state which is willing to accept connections for any IP interface associated with the node, the value 0.0.0.0 is used." ::= { tcpConnEntry 2 } tcpConnLocalPort OBJECT-TYPE SYNTAX INTEGER (0..65535) ACCESS read-only STATUS mandatory DESCRIPTION "The local port number for this TCP connection." ::= { tcpConnEntry 3 } tcpConnRemAddress OBJECT-TYPE SYNTAX IpAddress ACCESS read-only STATUS mandatory DESCRIPTION "The remote IP address for this TCP connection." ::= { tcpConnEntry 4 } tcpConnRemPort OBJECT-TYPE SYNTAX INTEGER (0..65535) ACCESS read-only STATUS mandatory DESCRIPTION "The remote port number for this TCP connection." ::= { tcpConnEntry 5 }
Chapter 6: Application Layer
110
TCP Connection Table (tabular view)
tcpConnTable (1.3.6.1.2.1.6.13)
Listen 0.0.0.0 23 0.0.0.0 0
Listen 0.0.0.0 161 0.0.0.0 0
closeWait 127.0.0.1 161 127.0.0.1 1029
established 140.113.88.164 23 140.113.88.174 3082
tcpConnState (x.1.1)
tcpConnLocalAddress(x.1.2)
tcpConnLocalPort(x.1.3)
tcpConnRemoteAddress(x.1.4)
tcpConnRemotePort(x.1.5)
x.1
tcpConnEntry = (x.1)
x.1
x.1
x.1
INDEX
Chapter 6: Application Layer
111
Basic Operations in SNMP
PDU Descriptions Version
GetRequest Retrieve the value of a leaf object V1
GetNextRequest Get the object lexicographically next to the one specified
V1
SetRequest Set (update) a leaf object with a value V1
GetResponse Response for GetRequest (value) or SetRequest (ACK) V1
Trap Issued by agent to notify the management station of
some significant event asynchronously
V1
GetBulkRequest Retrieve large blocks of data, such as multiple rows in a
table.
V2
InformRequest Allows one MS to send trap information to another MS
and receive a response
V2
PDU: Basic data unit in SNMP operationsMS : Management StationVariable-binding list: A list of variables and corresponding values in a PDU
Chapter 6: Application Layer
112
TCP Connection Table (lexicographical view)tcpConnTable (1.3.6.1.2.1.6.13=x)
tcpConnState (x.1.1)
tcpConnLocalAddress(x.1.2)
tcpConnLocalPort(x.1.3)
tcpConnEntry = (x.1)
Listen (x.1.1.0.0.0.0.23.0.0
.0.0.0)
Listen (x.1.1.0.0.0.0.161.0.0
.0.0.0)
closeWait(x.1.1.127.0.0.1.161.
127.0.0.1.1029)
0.0.0.0 (x.1.2.0.0.0.0.23.0.0
.0.0.0)
established(x.1.1.140.113.88.164.23.
140.113.88.174.3082)
0.0.0.0 (x.1.2.0.0.0.0.161.0.0
.0.0.0)
127.0.0.1(x.1.2.127.0.0.1.161.
127.0.0.1.1029)
140.113.88.164(x.1.2.140.113.88.164.23.
140.113.88.174.3082)
23(x.1.3.0.0.0.0.23.0.0
.0.0.0)
161 (x.1.3.0.0.0.0.161.0.0
.0.0.0)
161(x.1.3.127.0.0.1.161.
127.0.0.1.1029)
23(x.1.3.140.113.88.164.23.
140.113.88.174.3082)
…
Traverse the tree using Depth First Search
Chapter 6: Application Layer
113
User-based Security Model (USM, RFC2574) Four major threats
Modification of Information (between agent and MS) Masquerade (pretend to be an authorized user) Disclosure (eavesdropping) Message Stream Modification (reorder, delay, replay)
Solutions Timeliness module (for 1st, 3rd and 4th threats) Authentication protocol—use MD5 (for the 2nd threat) Privacy protocol—Use DES (for 3rd)
Chapter 6: Application Layer
114
View-based Access Control Model (VACM RFC 2575) Access control
MIB view -- a collection of MIB objects Elements in the model
Group– Categorization of managers securityLevel– Help distinguish the access rights of a
group Context– A collection of managed object accessible by an
snmp entity. MIB view– A subset of a particular context Access policy– Decide the final access rights of a group
to a context
Chapter 6: Application Layer
115
An SNMP Entity and its Component (RFC 2571)
DispatcherMessage
Processing Subsystem
Access Control
Subsystem
Security Subsystem
SNMP engine
Command Generator
Command Responder
Notification Receiver
Notification Originator
Proxy Forwarder
Other
Application(s)
SNMP entity
Chapter 6: Application Layer
116
Comparisons Between Three SNMP Versions
Version Description and improvement
SNMPv1 1. Define the SMI (RFC1155)
2. A more concise MIB definition (RFC 1212)
3. SNMP framework and its related operations (RFC 1157)
4. Concept of security (authentication) is proposed
SNMPv2 1. Improved SMI (support 64-bit counter, and other types of address than IP)
2. Inclusion of ‘GetBulkRequest’ to improve the efficiency for retrieving large
blocks of data
3. ‘InformRequest’ for communication between management stations
SNMPv3 1. Security and administration add-on’s (RFC2571)
2. Multi-version SNMP message processing and dispatch capability (RFC 2572)
3. The five type of applications within an SNMP engine (RFC 2573)
4. User-based security model (RFC 2574)
5. View-based access control (RFC 2575)
Chapter 6: Application Layer
117
Open Source: Net-SNMP
Introduction to Net-SNMP Some commands for query, set and trap Extensible architecture
Ways of extending the MIB Subagent protocol– AgentX (RFC2741) Build and include a private MIB
Inside Net-SNMP Snmpd and snmptrapd
Chapter 6: Application Layer
118
Introduction
Started at CMU, moved to UCDavis (1995) and is now based at SourceForge (2000~)
What it provides: An extensible agent SNMP library for further development Tools to get or set information from SNMP agents Tools to generate and handle SNMP traps
Support SNMP V1, V2 and V3 Runs on many Unix-like systems and Windows
Chapter 6: Application Layer
119
Some Commands for Query, Set and Trap
Name Description and example PDU used
SNMPGET Retrieve the value of a leaf object using get GetRequest
SNMPSET Set (update) a leaf object with a value SetRequest
SNMPBULKGET Get multiple objects at a time. Possibly
under different subtreesGetBulkReques
t
SNMPWALK Explore all the objects under a subtree
of the MIBGetNextReques
t
SNMPTRAP Uses the TRAP Request to send information
to a network manager. More than one object
identifiers can be applied as arguments
Trap
SNMPSTATUS Used to retrieve several important statistics
from a network entity. Errors will also be
reported, if any
SNMPNETSTAT Displays the values of various network-
related information retrieved from a remote
system using the SNMP protocol
Chapter 6: Application Layer
120
Command Line Examples
$ /usr/local/bin/snmpbulkwalk -v 3 -u ynlin -l authNoPriv –a MD5 -A ynlinpasswd localhost systemsystem.sysDescr.0 = Linux ynlin2.cis.nctu.edu.tw 2.4.14 #5 SMP 週四 11 月 22 23:6system.sysObjectID.0 = OID: enterprises.ucdavis.ucdSnmpAgent.linuxsystem.sysUpTime.0 = Timeticks: (30411450) 3 days, 12:28:34.50system.sysContact.0 = gis88559system.sysName.0 = ynlin2.cis.nctu.edu.twsystem.sysLocation.0 = ynlin2system.sysORLastChange.0 = Timeticks: (0) 0:00:00.00system.sysORTable.sysOREntry.sysORID.1 = OID: ifMIBsystem.sysORTable.sysOREntry.sysORID.2 = OID: .iso.org.dod.internet.snmpV2.snmpBsystem.sysORTable.sysOREntry.sysORID.3 = OID: tcpMIBsystem.sysORTable.sysOREntry.sysORID.4 = OID: ipsystem.sysORTable.sysOREntry.sysORID.5 = OID: udpMIB. . . .
$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0system.sysContact.0 = [email protected]
$ snmpset -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0 s gis88559system.sysContact.0 = gis88559
$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0system.sysContact.0 = gis88559
Chapter 6: Application Layer
121
Snmpd and Snmptrapd
Snmpd Process snmp packets and respond with the required
information Can be either master agent or subagent Basic access control with a port/addr pair Load dynamical MIB module at start
Snmptrapd Receive and log trap messages Configured to launch an external program when receiving
a trap matching a particular object identifier
Chapter 6: Application Layer
122
Processing flow inside Net-SNMP
Chapter 6: Application Layer
Start the server with required options
init_agent()[agent/snmp_vars.c]
count = select(numfds, &fdset, 0, 0, tvp)
snmp_read()[snmplib/snmp_api.c]
receive() [agent/snmpdt.c]
netsnmp_check_outstanding_agent_requests() [agent/snmp_agent.c]
while (netsnmp_running)
init_master_agent() [agent/snmp_agent.c]
snmp_select_info()[snmplib/snmp_api.c]
init_snmp()[agent/snmp_api.c]
124
6.7 VoIP Introduction Elements in a VoIP environment Underlying protocols
H.323 SIP
Challenges for VoIP technology Open source: openphone
Chapter 6: Application Layer
125
Introduction
Three ways of transferring voice Circuit switched Voice over frame relay Voice over IP
Pros and cons of VoIP compared with PSTN Inexpensive cost Simplicity Less bandwidth consumption Extensibility
Chapter 6: Application Layer
126
VoIP Environment (H.323 adopted)
PSTN Network
Gateway
Telephone
Router
H.323 gatekeeper
H.323 terminal
IP Network
MCU
Packet switched network Circuit switched network
Chapter 6: Application Layer
127
H.323 Protocol Stack
Q.931 H.245
G.711
G.723
G.729
H.263RTCP
RAS
(H.225.0)
Control (by GK)
A / V controlAudio VideoControl
TCP UDP
IP
RTP
T.120
Data
Recommendation Description
RAS(H.225.0) Stands for Registration, Admission and Status.
H.245 Control protocol for capability negotiation and channel setup.
Q.931 Used for call setup and teardown between two terminals.
T.120 Data protocols for multimedia conferencing. (application sharing, whiteboarding)
RTP/RTCP Used for real time traffic synchronization and transportation.
Chapter 6: Application Layer
128
Setup Procedure of an H.323 Call Registration and admission
Call setup
Terminal capability negotiation ,channel setup and master-slave detection
Stable call established and proceeds
Close channel
Call teardown
Disengagement
RAS
Q.931
H.245
RTP/RTCP
H.245
Q.931
RAS
Chapter 6: Application Layer
129
Session Initialization Protocol (SIP)
SIP Targeted to replace the H.323
Simplicity Compatibility
Characterize a multimedia session With supplementary SDP/SAP protocols
HTTP-like characteristics Text-based protocol Message syntax and header fields identical to HTTP/1.1 Client-server scheme
Chapter 6: Application Layer
130
Elements in an SIP Environment
Local proxy server
Remote proxy server
Redirect server
Location server
User Agent Server (UAS)
User Agent Client (UAC)
Internet
Chapter 6: Application Layer
131
Protocol Stacks of SIP
TCP UDP
IP
SAP/SDP RTP RTCP
SIP Multimedia Traffic
Control plane Data plane
Chapter 6: Application Layer
132
Operators and Reply Codes in SIPOperators Description
INVITE Invite a user to a call
ACK Confirmation for the final response
BYE Terminate a call between endpoints
CANCEL Terminate the search for a user or request for a call
OPTIONS Features supported for a call
REGISTER Register current location of the client with location server
INFO Use for mid-session signaling
Reply Code Description
1xx (Informational) Trying, ringing and queued
2xx (Successful) The request was successful
3xx (Redirection) Give information about the receiver’s new location
4xx (Request Failures) Failure responses from a particular server
5xx (Server Failures) Failure responses given when a server itself has erred
6xx (Global Failures) Busy, decline, requests not acceptable
Chapter 6: Application Layer
133
Example Operation in SIP
1. UAC INVITE UAS with the SIP URL 2. Destination IP address known => Send the request directly to the destination unknown => Redirected to the local proxy server (with
location server).3. In redirection mode, send back the callee’s new location4. In proxy mode, forward the request to the destination. 5. The UAS answers 200 (OK), then the UAC ACKs6. Session established
Chapter 6: Application Layer
134
H.323 V.S. SIP
Simplicity in operation and implementation Message encoding Capability exchange Supported data types Capability of handling a conference
Chapter 6: Application Layer
135
Open Source: Asterisk
An integrated PBX (Private Branch eXchange) system
Chapter 6: Application Layer
Asterisk PBX Server
PC Softphone Analog PhoneSIP phone
ATA Adaptor
PSTN Networks
136
Block Diagram
Asterisk provides a framework to build a customized VoIP system. Flexible to add/remove modules to establish the basic
transport service
HTTP
Channel
CDR Engine
RTP
PBX
Framer
Module Loader
Asterisk framework
SNMP
136Chapter 6: Application Layer
137
Data Structures
Two concepts, context and extension, are introduced
Context1
Ext1 Ext2
Context2
Ext1 Ext2
. . . . . .
. . .
Inside a Asterisk PBX
137Chapter 6: Application Layer
138
Call flow
__ast_request_and_dial()[main/pbx.c]
action_originate() [main/manager.c]
init_manager() [main/manager.c]
ast_pbx_outgoing_exten() [main/pbx.c]
First call in this channel?
yes no
ast_pbx_start()[main/pbx.c]
pbx_thread()[main/pbx.c]
__ast_pbx_run() [main/pbx.c]
__ast_pbx_run() [main/pbx.c]
ast_pbx_run()[main/pbx.c]
138Chapter 6: Application Layer
140
6.8 Streaming
Introduction to streaming Availability Short fetching time Avoid storage overhead Live broadcasting
Architecture and components for streaming Compression scheme Protocol stack for streaming QoS and synchronization mechanism
Open source: Darwin streaming server
Chapter 6: Application Layer
141
Architecture and Components for Streaming
Video Files
Audio Files
Transport Protocols
QoS Control
Server
Video Decoder Audio Decoder
Transport Protocols QoS Control
Client
synchronization
Video/Audio raw data
Internet
Chapter 6: Application Layer
142
Compression Schemes
Temporal or spatial Temporal – describe changes between frames Spatial – describe similar patterns in a frame
Lossless or lossy Recoverability when decompressed GIF (lossless) and JPG (lossy)
Symmetrical or Asymmetrical The time for compression and decompression According to the load of the server
Chapter 6: Application Layer
143
Protocol Stacks of Streaming
TCP UDP
IP
RTSP RTCP RTP
Control plane Data plane
RTSP (Real Time Streaming Protocol) VCR-Style remote control functionalities Establish and control streams
Description file before a session Methods in a session (SETUP, PLAY, TEARDOWN)
Reuse of HTTP concepts (Ex: ASCII messages) HTTP
Mostly by buffering Low performance HTTP
Chapter 6: Application Layer
144
QoS and Synchronization Mechanism QoS mechanism
Source-based rate control Receiver-based rate control
Synchronization mechanism Intra-stream synchronization
Continuity of audio/video data frames inside a stream
Inter-stream synchronization Consistency between cooperative streams
Inter-object synchronization Consistency between streams and other objects
like text and still images
Chapter 6: Application Layer
145
Open Source: Darwin Streaming Server Open source version of Apple's QuickTime
Streaming Server (QTSS) Deliver streaming media with RTP and RTSP Provide both live and on-demand programs Supports many formats: H.264/MPEG-4 AVC,
MPEG-4 Part 2, 3GP and MP3 Provide Web-based administration, authentication,
relay support, and integrated broadcaster administration, etc.
Chapter 6: Application Layer
146
Block Diagram
Two parts: core server and modules Core server provides task scheduling Module provides specific functions
146Chapter 6: Application Layer
150
6.9 Peer-to-Peer Applications (P2P)
Introduction P2P Architectures Performance Issues of P2P Applications Case Study: BitTorrent Open source: Vuze (BitTorrent Client)
150Chapter 6: Application Layer
151
Introduction to P2P A distributed network architecture in which
participants act as both a client and a server Participants construct a virtual overlay network at
the application layer on top of the underlying IP network
P1
R1
R2 R3 P2
P1
P2
Physical Network
Overlay Network
Node: peers
Edge: TCP/UDP connection
151Chapter 6: Application Layer
Introduction to P2P
Operations in P2P systems Joining the P2P overlay network, Resource discovery
The most challenging problem for P2P applications Resource retrieval
Chapter 6: Application Layer 152
P2P Architectures
Way of forming a P2P overlay network could be classified into three categories Centralized Decentralized and unstructured Decentralized but structured
Chapter 6: Application Layer 153
Centralized P2P
Operation overview
Chapter 6: Application Layer 154
Directory
Peer
1. upload index
Peer Peer
1. upload index2. Query&Response
3. Download file
Centralized P2P Utilizes a central directory server for locating
objects in the P2P network Peers join by registering to the directory server Peers inform the directory server of objects to be
shared A peer sends query to the directory server to
search an object The peer receives reply from the directory server The peer selects one or more peer in the reply to
download the object directly from Example: Napster
Chapter 6: Application Layer 155
Centralized P2P
Advantages Simple Easy to implement Support various kinds of search such as keyword,
full-text, and metadata search Disadvantages
Not a true P2P system as it relies on a central directory server which becomes performance bottleneck, single point of failure, not scalable and vulnerable to DoS attacks
Chapter 6: Application Layer 156
Decentralized and Unstructured P2P To get rid of server, this approach floods query
messages to peers to search for shared objects Limited-scope flooding is adopted to reduce flooding
messages A query hit message is returned along the reverse
path back to the inquirer
Chapter 6: Application Layer 157
Query
Query
QueryQuery
Query
Query Hit
Query Hit
Query Hit
Download
Example: Gnutella
Decentralized and Unstructured P2P Join procedure
A peer needs to know at least one of the peers already on the overlay network.
The peer sends a join message to a peer already on the overlay.
The existing peer then replies its identity as well as a list of its neighbors. It may also forward the join message to its neighbors.
Upon receiving join reply messages, the newcomer knows more peers on the overlay.
Chapter 6: Application Layer 158
Decentralized and Unstructured P2P Hierarchical overlay with super peers
Flooding is apparently not scalable FastTrack adopts a hierarchical overlay A super peer acts as a local directory database
which stores the indexes of objects shared by ordinary peers
Two-level hierarchical overlay The lower level adopts the central
server approach The upper level (super peers) adopts the
decentralized and unstructured approach.
Chapter 6: Application Layer 159
query reply
Decentralized and Unstructured P2P Advantages
Fully distributed Reliable, fault-tolerant No single point of failure
Disadvantages Excessive query traffic make it not scalable May fail to find content that is actually in the
system Super peer may become overloaded or been
attacked
Chapter 6: Application Layer 160
Decentralized but Structured
Combine the distributed directory service with an efficient query routing scheme
Key ideas For distributed directory service, a hash function
maps peers and objects into the same address space so that objects can be deterministically assigned to peers in a distributive manner.
For efficient query routing, peers are organized into a structured overlay based on their positions in the address space.
Chapter 6: Application Layer 161
Decentralized but Structured
Operations overview Each peer generates its own node ID by a predefined
hash function. For each object being held and to be shared, the peer
generates the object ID by the same or another hash function.
For each object, the peer sendd a register message to the node that has the node ID same as the object’s ID.
If a peer wants to query an object, it uses the hash function to generate the object ID and sends the query message to the node that hosts the object’s ID.
Chapter 6: Application Layer 162
Decentralized but Structured
Message routing (use Chord as an example) Key idea: have each peer maintain a specially
designed routing table such that every peer could forward the arriving message to a neighboring peer with node ID that is further closer to the destination.
Consider a 10-node Chord overlay in a 6-bit address space
Chord views its address space as a one-dimensional circular space such that peers in the space form a ring overlay.
Chapter 6: Application Layer 163
Message Routing in Chord
The routing table in Chord is called a finger table.
For an m-bit address space, the finger table of a node with ID=x consists of at most m entries and the i-th entry points to the first node with ID following the ID of x+2i-1 modulo 2m, for 1≤i≤m.
Chapter 6: Application Layer 164
Finger Table of Chord Finger table of node N8, where m =6.
Chapter 6: Application Layer 165
N8
N15
N20
N30N38
N42
N47
N51
N56 +1
+2
+4
+8
+16+32
Finger table
N8+1 N15
N8+2 N15
N8+4 N15
N8+8 N20
N8+16 N30
N8+32 N42
N1
Routing a Query Message Routing a query message for object 54 from N8
Chapter 6: Application Layer 166
N1
N8
N15
N20
N30N38
N42
N47
N51
N56
Finger table
N8+1 N15
N8+2 N15
N8+4 N15
N8+8 N20
N8+16 N30
N8+32 N42
Finger table
N42+1 N47
N42+2 N47
N42+4 N47
N42+8 N51
N42+16 N1
N42+32 N15
Finger table
N51+1 N56
N51+2 N56
N51+4 N56
N51+8 N1
N51+16 N8
N51+32 N20
K54lookup(54)
167
Performance Issues of P2P Applications
Free Riding Flash Crowd Topology Awareness NAT Traversal Churn Security Copyright Infringement
Copyright Reserved 2010 167Chapter 6: Application Layer
Free Riding Scalability of P2P systems relies on the
contribution from peers If a peer only consumes but contributes little or no
resources, it becomes a free rider of the system. 85% of peers share no files in Gnutella in 2005
A common solution is to implement some incentive mechanisms. For example, tit-for-tat in BitTorrent.
Other solutions: reward-based and credit-based mechanisms
Chapter 6: Application Layer 168
Flash Crowd
A sudden, unanticipated growth in the demand of a particular object e.g., a new release of a DVD video or mp3 file
Issues how to deal with a sudden large amount of query
messages how long to find and download the object within a
short time period Solutions
Cache, duplicating popular objects
Chapter 6: Application Layer 169
Topology Awareness
A virtual link in a P2P overlay could be a long end-to-end connection across continents or a short one within a local area network How to avoid serious topology mismatch
Solutions Many route-proximity and neighbor-proximity
enhancements for P2P overlay systems have been proposed based on RTT measurement, preference of routing domain or ISP, or geographical information.
Chapter 6: Application Layer 170
NAT Traversal
Basic requirement for a P2P system is to provide peers with NAT traversal mechanisms If both peers are behind NAT devices, they cannot
connect to each other without help from other peers or STUN servers
Solutions In most cases, NAT traversal is solved by relay
peers or super peers that have public IP addresses
Chapter 6: Application Layer 171
Churn
Churn refers to the phenomenon that peers dynamically join and leave the system at will. high churn rate seriously affects the stability and
scalability of a P2P system. e.g., a high churn rate may cause a tremendous
overlay maintenance overhead and dramatic routing performance degradation in DHT-based system
Solutions Avoid a rigid structure or relation among peers Peers maintain a list of potential neighbors for quick
and dynamic neighbor replacement
Chapter 6: Application Layer 172
Security
Security issues P2P programs with back hole (Trojan Horse),
spurious content, leaking of files not to be shared. Solutions to content pollution
protect the content with message digest such as MD5 In BitTorrent, the MD5 digest of each piece of a shared
file is stored in the metadata file peer reputation system object reputation system
Chapter 6: Application Layer 173
Copyright Infringement
Sharing copyrighted objects through P2P systems is a serious problem which hinders the promotion and usage of P2P systems.
Not only P2P users are responsible for copyright infringement, so are the companies that host P2P applications Especially in the case where P2P systems will not
be able to exist without their serves, e.g., Napster.
Chapter 6: Application Layer 174
175
Case Study: BitTorrent BitTorrent (BT) is originally designed by Bram Cohen in
2001 Well thought-out protocol with several unique features
use tit-for-tat as an incentive mechanism to cope with free riders
use out-of-band search to avoid copyright infringement issue use pull-based swarming for load balancing use hash check to prevent propagation of spurious pieces after a peer has successfully downloaded a file, it becomes a
seeder to distribute the file
Copyright Reserved 2010 175Chapter 6: Application Layer
BT Terminologies
A file is cut into pieces of a fixed size. A piece is further divided into chunks, the basic
data unit for a peer to request for content. The integrity of a piece is protected by a SHA-1 A peer becomes a seeder if it has successfully
downloaded the file. A tracker for each file or group of files to be
shared. The tracker tracks the downloading peers and seeds, and coordinates the file distribution among peers.
Chapter 6: Application Layer 176
BT Operation Overview
Chapter 6: Application Layer 177
`
Web page with Link
to .torrent
Web Server
Tracker
Seeder
`
`
`
Source
publish
Downloading peer
Downloading peer
1. get .torrent 2. get
announce
3. response peer
list
4. get piece
5. get piece
Piece Selection
Random first piece selection For the first few pieces, the client just randomly
selects a piece to download. Rarest first policy
Selects the most scarce piece to download first End-game mode
To speed up the completion of a file download at the end, a peer with only a few pieces missing will send requests for all missing pieces to all the peers
Chapter 6: Application Layer 178
Peer Selection Choking/unchoking
Choking refers to a temporal refusal to upload to a peer. At the beginning, all peers are chocked Tit-for-tat algorithm selects a fixed number of peers from which the
peer downloaded most to unchoke
Optimistic unchoking new peer needs to move its first step when initially joined the
system select one peer at random
Anti-snubbing If a peer is choked by all of its peers (snubbed), it is better to run
optimistic unchoking more often to explore more peers that are willing to cooperate.
Chapter 6: Application Layer 179
180
Open source: Vuze (BitTorrent Client)
Many BT client programs are open-source Some popular client programs
Vuze uTorrent BitComet
Copyright Reserved 2010 180Chapter 6: Application Layer
Core Packages
Most of Vuze’s core packages are located under the .\com\aelitis\azureus\core directory
Chapter 6: Application Layer 181
Algorithm Implementation
Main program for controlling the piece and peer selection is the PEPeerControlImpl class
Chapter 6: Application Layer 183
Implementation of Peer and Piece Selection Unchoking algorithms are implemented in
DownloadingUnchocker.java and SeedingUnchocker.java tit-for-tat is implemented in calculateUnchokes() Optimistic unchoking is implemented in
UnchokerUtil.getNextOptimisticPeer() getRequestCandidate() defined in
PiecePickerImpl.java is the core method for deciding which block to download
Chapter 6: Application Layer 184
185
6.10 Summary
General issues well-known ports, concurrent connection-oriented
servers and iterative connectionless servers, statefulness/statelessness
Major application protocols DNS, SMTP, POP3, IMAP4, HTTP, FTP, SNMP
Real time, multimedia application protocols SIP, RTP, RTCP, RTSP
P2P protocols Napster, Gnutella, DHT, Chord, BT
Chapter 6: Application Layer