- 1. AUDIO AND VIDEO STREAMINGWHAT IS STREAMING Streaming
corresponds to playing audio and video files from anInternet
server.This opposes to downloading the corresponding files.Pierre
Nugues As transmission over the Internet is not synchronous,
streaming uses [email protected] buffer to store a small
part of data. www.cs.lth.se/~pierreThis buffer dampens
irregularities in Internet transmission.Streaming imposes
constraints on the network speed: Download speedon the average
should be at least as fast as playback speed.Streaming, May 2005
1Streaming, May 20052 APPLICATIONS OF STREAMINGPROBLEMS WITH
TCP/UDPApplications are numerous. In addition to data:The original
TCP/UDP protocols are based on packet transmission Internet
telephony and video conferencesand have no quality of service.
Radios and Digital TV: ordinary broadcast but through Internet,
Multimedia transmission has to tackle:no frontier, no distance
Delay. Must be less than 300 ms. (Perception threshold: 150 ms)
Audio and video server: on demand movies and concerts. Jitter.
Packets may use different transmission paths that results into
Games and virtual reality time expansion and compression
Interaction Loss. Routers may drop packets when the network load is
too highTriple play: data (IP), audio (VoIP), video (TVoIP).Audio
and video transmission needs buffering and synchronization,It is
made possible because of the growing availability of ADSL and
possibly error correction, for instance by repeating datafast
Internet UDP is just a layer to address ports. It is compatible
with theCompanies in Sweden: NetInsight, etc. requirementsTCP
design makes it less adapted to real-time, fast communications
Streaming, May 2005 3Streaming, May 20054
2. PACKET TRANSMISSION REAL TIME TRANSPORT PROTOCOLThe Real Time
Transport Protocol (RTP) Identifies the content Adds time stamps
Adds sequence numbers RTP does not guarantee a real-time delivery.
RTP needs an application layer to Re-order packets Attenuate jitter
Compensate packet loss Streaming, May 2005 5Streaming, May 20056
ARCHITECTUREREAL TIME TRANSPORT PROTOCOL (RTP)RTP is on top of
UDP:Ethernet IP UDP RTP Media content: MPEG, AIFF, and son onThe
simplified RTP header structure is: Payload SequenceTimestamp Sync.
Source ID Other typeNumber (SSRC) fields RTP RFCs are available
here: http://www.ietf.org/rfc/rfc3550.txt and
http://www.ietf.org/rfc/rfc3551.txt Other reference:
http://csperkins.org/rtp-book.htmlStreaming, May 2005 7Streaming,
May 20058 3. RTP HEADERTIMESTAMP ACCORDING TO THE RFC Payload type
(7 bits) corresponds to the packet content: PCM = 0, The timestamp
reflects the sampling instant of the first octet in the RTP data
packet. DVI4=5, MPEG-2 = 33 (http://www.iana.org/assignments/rtp-
The sampling instant must be derived from a clock that increments
monotonically and linearly in time to allow synchronization and
jitter calculations (see Section 6.3.1). The parameters) resolution
of the clock must be sufficient for the desired synchronization
accuracy and Sequence number (16 bits) is incremented each time a
packet is sent for measuring packet arrival jitter (one tick per
video frame is typically not sufficient). (Nothing guarantees the
arrival order with UDP) The clock frequency is dependent on the
format of data carried as payload and is specified statically in
the profile or payload format specification that defines the
format, The timestamp (32 bits) corresponds to the sampling instant
of theor may be specified dynamically for payload formats defined
through non-RTP means. first octet in the RTP data packet. (Clock
of the sending machine)If RTP packets are generated periodically,
the nominal sampling instant as determined from the sampling clock
is to be used, not a reading of the system clock. As an SSRC (32
bits) is the source of the stream. (A sending machine canexample,
for fixed-rate audio the timestamp clock would likely increment by
one have multiple sessions.)for each sampling period. If an audio
application reads blocks covering 160 sampling periods from the
input device, the timestamp would be increased by 160 for each such
block, regardless of whether the block is transmitted in a packet
or dropped as silent. 8 kHz! 125 !s, 160 samples ! 20 ms Streaming,
May 2005 9Streaming, May 2005 10 REAL-TIME CONTROL PROTOCOL (RTCP)
ENCODING FORMATSPart of the RTP protocol and defined in the same
RFC. Telephone: 8 kHz, 1 octet (256 values), 64 kbit/s It sends
periodically control packets to all participants in the sessionCD:
44.1 kHz, 16 bits, stereo, 1.4 mbits/s and uses a different port
MP3, compressed, 96, 128, 160 kbit/s It provides feedback on the
quality of the data distribution from the G.732, (Internet
telephony in H.323) 5.3 kbit/s or 6.4 kbit/s sender and the
receiver: Statistics on packets sent, received, lost, jitter MPEG-2
used in DVD, 3-6 mbit/s Should limit to 5% of the bandwidth Two
important concepts in encoding methods: Commands Description Bit
rate can be constant (CBR) or variable (VBR) SRSender report From
an original format, encoding compress data with or without
RRReceiver report loss. Lossy compression generally results in
better rates but lower SDESSource descriptionquality: sometimes not
perceptible. Loss rate can be a parameter of BYE Quitthe
method.Streaming, May 2005 11 Streaming, May 2005 12 4. PULSE CODE
MODULATIONJPEG Compression standard for still pictures:Sampling
Digitization Maps RGB images onto YUV coordinates (luminance
andchrominance) Applies a Discrete Cosine Transform (DCT)
Quantizes, which results in a compression with an adjustable loss
100 0 80 120 90 0 40 40 10 30 90 140 170 180 170 140 100 60 0 70
Run-length encoding Digitization can be linear or logarithmic:
"-law, A-lawStreaming, May 2005 13 Streaming, May 2005 14 MPEG
CODECSInitially sequences of frames using JPEG (I Frames)Codec
encode and decode original data streams. 25 or 30 frames/sDepending
on the media you are sending, you must have the Uses temporal
redundancies between images: differences between corresponding
codec. frames (P and B Frames)Formats supported by the RTP
implementation of the Java Media MPEG 2 has possible resolutions:
720 ! 480, 720 ! 576, 1920 ! 1080.Framework:
http://java.sun.com/products/java- Multimedia streams contain audio
and video data that are media/jmf/2.1.1/formats.html#RTPFormats
synchronized in MPEG Codecs can be found from many sources
as:http://jffmpeg.sourceforge.net/index.html
(http://en.wikipedia.org/wiki/MPEG)Streaming, May 2005 15
Streaming, May 2005 16 5. JAVA MEDIA FRAMEWORKJMF
COMPONENTSAccording to Sun, the Java Media Framework (JMF) is an
API for The main components are: incorporating time-based media
into Java applications and applets. The Clock interface keeps track
of time in a media stream It also features RTP capabilities
Managers: Manager, PackageManager, etc. construct Players, JMF
makes it possible to build media applications assembling modules.
Processors, DataSources, etc Just follow the examples JMF programs
report their current state using MediaEvents. Available as an
optional package from Sun Many subclasses include ControllerEvent
and RTPEvent http://java.sun.com/products/java-media/jmf/index.jsp
DataSources manage media transfer. It contains the location of
Native implementations for Windows, Linux, and Solaristhe media
(URL), protocol, and software. Java implementation for the others,
including MacOS Source available under Suns license (SCSL)
Apparently, no active development from Sun. Latest release: 2003,
an MP3 updateStreaming, May 2005 17 Streaming, May 2005 18 JMF
PLAYERS PLAYER STATESA player reads a DataSource and renders
it(From Suns documentation)(From Suns documentation)Streaming, May
2005 19 Streaming, May 2005 20 6. A FIRST EXAMPLE: A PLAYERreturn;
if ((visual = mplayer.getVisualComponent()) != null) { MDIApp.java
available from http://java.sun.com/products/java-Dimension size =
visual.getPreferredSize();videoWidth = size.width;
media/jmf/2.1.1/solutions/index.htmlvideoHeight = size.height; if
((url = new URL(mediaFile)) == null)
{getContentPane().add("Center", visual); } else player =
Manager.createPlayer(url);videoWidth = 320; if ((control =
mplayer.getControlPanelComponent()) != null) mplayer = player;{
mplayer.addControllerListener((ControllerListener)
this);controlHeight = control.getPreferredSize().height;
mplayer.realize();getContentPane().add("South", control); } public
void controllerUpdate(ControllerEvent ce) { setSize(videoWidth +
insetWidth,if (ce instanceof RealizeCompleteEvent) { videoHeight +
controlHeight + insetHeight);mplayer.prefetch(); validate();} else
if (ce instanceof PrefetchCompleteEvent) { mplayer.start();if
(visual != null)Streaming, May 2005 21 Streaming, May 2005 22 }
else if (ce instanceof EndOfMediaEvent) {
PROCESSORSmplayer.setMediaTime(new Time(0));mplayer.start(); A
processor is a specialized player that can output data to a }
DataSource and carry out some processing } Streaming, May 2005 23
Streaming, May 2005 24 7. PROCESSOR STATESSENDING
AUDIOAudioTransmit.java available from
http://java.sun.com/products/java-
media/jmf/2.1.1/solutions/index.html private String
createProcessor() { if (locator == null) return "Locator is null";
DataSource ds; DataSource clone; try { ds =
Manager.createDataSource(locator); } catch (Exception e) { (From
Suns documentation)return "Couldn't create DataSource"; Configure
connects to the DataSource and demultiplexes the stream } // Try to
create a processor to handle the input media locator try {
Streaming, May 2005 25 Streaming, May 2005 26 processor =
Manager.createProcessor(ds);if (tracks == null || tracks.length
< 1) } catch (NoProcessorException npe) {return "Couldn't find
tracks in processor"; return "Couldn't create processor"; boolean
programmed = false; } catch (IOException ioe) { AudioFormat afmt;
return "IOException creating processor";// Search through the
tracks for a Audio track } for (int i = 0; i < tracks.length;
i++) { Format format = tracks[i].getFormat(); // Wait for it to
configure if ( tracks[i].isEnabled() && boolean result =
waitForState(processor,format instanceof AudioFormat &&
Processor.Configured);!programmed) { if (result == false)afmt =
(AudioFormat)tracks[i].getFormat(); return "Couldn't configure
processor";AudioFormat ulawFormat = new
AudioFormat(AudioFormat.DVI_RTP); // Get the tracks from the
processor// afmt.getSampleRate(), TrackControl [] tracks =
processor.getTrackControls();// afmt.getSampleSizeInBits(), //
afmt.getChannels()); // Do we have at least one track? //
8000,4,1);Streaming, May 2005 27 Streaming, May 2005 28 8.
tracks[i].setFormat(ulawFormat); // Realize the processor. This
will internally create a flowSystem.err.println("Audio transmitted
as:"); // graph and attempt to create an output datasource
forSystem.err.println(" " + ulawFormat);ULAW/RTP// Assume
succesful// Audio frames.programmed = true; result =
waitForState(processor, Controller.Realized);} else if (result ==
false)tracks[i].setEnabled(false);return "Couldn't realize
processor"; }if (!programmed) // Get the output data source of the
processor return "Couldn't find Audio track";dataOutput =
processor.getDataOutput();return null; // Set the output content
descriptor to RAW_RTP } ContentDescriptor cd = new
ContentDescriptor(ContentDescriptor.RAW_RTP);
processor.setContentDescriptor(cd); Streaming, May 2005 29
Streaming, May 2005 30 SENDING AUDIO (II) while (p.getState() <
state && !failed) { synchronized (getStateLock()) { private
synchronized boolean waitForState(Processor p, int state) try { {
getStateLock().wait(); p.addControllerListener(new
StateListener()); } catch (InterruptedException ie) { failed =
false; return false; } // Call the required method on the
processor} if (state == Processor.Configured) { } p.configure(); if
(failed) } else if (state == Processor.Realized) { return false;
p.realize(); else } return true; } // Wait until we get an event
that confirms the // success of the method, or a failure event. //
See StateListener inner classStreaming, May 2005 31 Streaming, May
2005 32 9. SENDING AUDIO (III) // for the processor to start
sending data. So we need to start the // Creates an RTP transmit
data sink. This is the easiest way to// output data source of the
processor. We also need to start createthe // an RTP transmitter.
The other way is to use the// processor itself, which is done after
this method returns. RTPSessionManager API.try { // Using an RTP
session manager gives you more control if you wishrtptransmitter =
Manager.createDataSink(dataOutput, tooutputLocator); // fine tune
your transmission and set other parameters.rtptransmitter.open();
private String createTransmitter() {rtptransmitter.start(); //
Create a media locator for the RTP data sink.dataOutput.start(); //
For example: } catch (MediaException me) {
//rtp://129.130.131.132:42050/Audio return "Couldn't create RTP
data sink"; String rtpURL = "rtp://" + ipAddress + ":" + port +
"/audio"; } catch (IOException ioe) { MediaLocator outputLocator =
new MediaLocator(rtpURL);return "Couldn't create RTP data sink";
}// Create a data sink, open it and start transmission. It will
return null; wait} Streaming, May 2005 33 Streaming, May 2005 34
SENDING VIDEO (int)(size.width / 8) * 8); int h = (size.height % 8
== 0 ? size.height : VideoTransmit.java available from
(int)(size.height / 8) * 8); VideoFormat jpegFormat = new
http://java.sun.com/products/java-VideoFormat(VideoFormat.JPEG_RTP,
media/jmf/2.1.1/solutions/index.htmlnew Dimension(w, h), for (int i
= 0; i < tracks.length; i++) { Format.NOT_SPECIFIED, Format
format = tracks[i].getFormat();Format.byteArray, if (
tracks[i].isEnabled() && frameRate); format instanceof
VideoFormat &&tracks[i].setFormat(jpegFormat); !programmed)
{System.err.println("Video transmitted as:"); System.err.println("
" + jpegFormat);// Found a video track. Try to program it to
output// Assume succesful JPEG/RTP programmed = true;// Make sure
the sizes are multiple of 8's. } elseDimension size =
((VideoFormat)format).getSize(); tracks[i].setEnabled(false);float
frameRate = ((VideoFormat)format).getFrameRate(); }int w =
(size.width % 8 == 0 ? size.width : Streaming, May 2005 35
Streaming, May 2005 36 10. RECEIVINGRESOURCES FOR
JMFAVReceive2.java available from
http://java.sun.com/products/java-Documentation:
http://java.sun.com/products/java-
media/jmf/2.1.1/solutions/index.html
media/jmf/reference/docs/index.html It opens RTP sessions for each
track Program examples available from Sun: It receives events from
the RTP streamshttp://java.sun.com/products/java- It creates one
player per stream media/jmf/2.1.1/solutions/index.htmlActive
discussion
list:http://forum.java.sun.com/forum.jspa?forumID=28Codecs
available from SourceForge:
http://jffmpeg.sourceforge.net/Streaming, May 2005 37 Streaming,
May 2005 38 REAL-TIME STREAMING PROTOCOL (RTSP)SESSION INITIATION
PROTOCOL (SIP)http://www.ietf.org/rfc/rfc2326.txtThere is no
procedure to connect to a remote host in UDP or RTP. RTSP is a
protocol to control streaming media. It acts as a sort ofThis is
handled by SIP. Defined by IETF in RFC: remote
control.http://www.ietf.org/rfc/rfc2543.txt Commands Description
SIP enables to set up a call, negotiate the parameters, manage and
SETUPCauses the server to allocate resources for a stream and start
an close the session.RTSP sessionBorrows many ideas from HTTP and
uses UDP or TCP. PLAY Tells the server to start sending dataOnce
the session is established, the media transmission can use RTP
RECORD Records dataor something else. PAUSETemporarily halts a
stream without freeing server resources TEARDOWN Frees resources
associated with the stream. The RTSP sessionceases to exist on the
server RTSP servers typically use RTP for the audio/video
transmission.Streaming, May 2005 39 Streaming, May 2005 40 11. A
SIP EXAMPLE (SIMPLIFIED FROM THE RFC)A CONTENT EXAMPLEAlice sends
an INVITE request addressed to Bobs SIP URI.The session description
protocol, SDP, specifies details of the INVITE
[email protected]">sip:[email protected] SIP/2.0connection using
name-value pairs. Via: SIP/2.0/UDP
pc33.atlanta.com;branch=z9hG4bK776asdhds(http://www.ietf.org/rfc/rfc2327.txt)
Max-Forwards: 70 v(ersion)=0 To: Bob ">sip:[email protected]>;
o(wner)=bell 53655765 2353687637 IN IP4 128.3.4.5 From:
Alicec(onnection)=IN IP4 135.180.144.94
;tag=1928301774">sip:[email protected]>;tag=1928301774
Call-ID: [email protected](edia)=audio 3456 RTP/AVP
0 3 4 5 CSeq: 314159 INVITE Contact:
">sip:[email protected]>; Content-Type: application/sdp
Content-Length: 142Streaming, May 2005 41 Streaming, May 2005 42 A
SIP EXAMPLE (SIMPLIFIED FROM THE RFC)SIP METHODSAliceProxyProxy
BobMethodsDescriptions INVITE F1 -----> INVITE Invites a session
INVITE F2 ----->ACKAcknowledges OPTIONSServer capabilities
