Embed Size (px)
Citation preview
An Empirical Study of RealVideo Performance Across the Internet
Yubing Wang, Mark Claypool and Zheng Zuo
http://perform.wpi.edu/real-tracer
November 2001 SIGCOMM Internet Measurement Workshop
Introduction
• Streaming Audio and Video is growing
• Traditional Internet studies, but ….video is different– Bandwidth and response time not enough
• Care about smooth rate
– Packet loss rate alone is not enough• Often use repair techniques
• RealVideo is big [Jup01]
– RealPlayer, MediaPlayer, QuickTime
Measure RealVideo on Internet
November 2001 SIGCOMM Internet Measurement Workshop
Outline
• Introduction
• RealVideo
• Methodology
• Results
• Analysis
• Conclusions
November 2001 SIGCOMM Internet Measurement Workshop
RealVideo Network Characteristics
• Buffering
• SureStream
• Scalable Video Technology
• Repair
Server
RTSP
Data: TCP or UDP
November 2001 SIGCOMM Internet Measurement Workshop
Outline
• Introduction
• RealVideo
• Methodology
• Results
• Analysis
• Conclusions
November 2001 SIGCOMM Internet Measurement Workshop
Methodology
• Build customized RealVideo player (RealTracer)– Uses RealPlayer core
– Software Development Kit (SDK)
– Records performance stats: frame rate, jitter ….
• Select diverse set of RealVideo servers and videos
• Solicit volunteers to run RealTracer– Many users to Many servers
– Friends, colleagues outside of Mass., U.S.
– rec.video newsgroup and end2end mailing list
– 2 weeks in June 2001
• Analyze results
November 2001 SIGCOMM Internet Measurement Workshop
Outline
• Introduction
• RealVideo
• Methodology
• Results
• Analysis
• Conclusions
November 2001 SIGCOMM Internet Measurement Workshop
TCP UDP
Results
• 11 servers in 8 countries
• 63 users in 12 countries
• 2855 video clips played
• 388 video clips watched and rated
• 10% chance clip unavailable
November 2001 SIGCOMM Internet Measurement Workshop
Geographic Location of Servers and Clients
November 2001 SIGCOMM Internet Measurement Workshop
Clips Played per User
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 20 40 60 80 100
Clips Played per User
Cu
mu
lati
ve
De
nsi
ty
.
November 2001 SIGCOMM Internet Measurement Workshop
Clips Served per Country
126 184 240 260 294 297416
1075
0
200
400
600
800
1000
1200
Can
ada
Japa
n
Italy
Chi
na
Aus
tral
ia
Bra
zil
UK
US
Nu
mb
er
of
Clip
s
November 2001 SIGCOMM Internet Measurement Workshop
Clips Played per Country
8 16 32 47 55 59 84 98 115 131 142
2100
0
500
1000
1500
2000
2500
Egyp
t
Indi
a
New
Zea
land
Rom
ania
UA
E
UK
Can
ada
Aus
tral
ia
Fran
ce
Ger
man
y
Chi
na US
Nu
mb
er
of
Clip
s
November 2001 SIGCOMM Internet Measurement Workshop
0
200
400
600
800
1000
1200
VA WA ME TN CT NH CO IL TX CA WI DE MD MN NC FL MA
State Names
Nu
mb
er
of
Clip
s
Clips Played per State
November 2001 SIGCOMM Internet Measurement Workshop
Outline
• Introduction
• RealVideo
• Methodology
• Results
• Analysis– Frame Rate – Jitter– Perceived Quality
• Conclusions
November 2001 SIGCOMM Internet Measurement Workshop
Frame Rate
• Basic video performance unit– frames per second (fps)
• Guidelines [Rea00a]:
– 24-30 fps: full-motion video
– 15 fps: full-motion video approximation
– 7 fps: choppy
– 3 fps: very choppy
– Less than 3 fps: slide show
November 2001 SIGCOMM Internet Measurement Workshop
Overall Frame Rate
00.1
0.20.30.40.5
0.60.70.8
0.91
0 5 10 15 20 25 30
Average Frame Rate (fps)
Cu
mu
lati
ve D
en
sit
y .
November 2001 SIGCOMM Internet Measurement Workshop
Frame Rate vs. Network Configuration
Modem: 19% (550)DSL/Cable: 30% (874)T1/LAN: 51% (1468)
November 2001 SIGCOMM Internet Measurement Workshop
Bandwidth vs. Network Configuration
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 100 200 300 400 500
Average Bandwidth (Kbps)
Cum
ulat
ive
Den
sity
56k Modem
DSL/Cable
T1/LAN
November 2001 SIGCOMM Internet Measurement Workshop
Frame Rate vs. Transport Protocol
0
0.2
0.4
0.6
0.8
1
0 10 20 30
Frame Rate (fps)
Cu
mm
ula
tive
De
ns
ity
TCP
UDP
November 2001 SIGCOMM Internet Measurement Workshop
Bandwidth vs. Transport Protocol
0
0.1
0.2
0.3
0.40.5
0.6
0.7
0.8
0.9
1
0 100 200 300 400 500 600
Average Bandwidth (Kbps)
Cu
mu
lati
ve
De
ns
ity
TCP
UDP
November 2001 SIGCOMM Internet Measurement Workshop
Frame Rate vs. Server Geographic Location
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 5 10 15 20 25 30
Average Frame Rate (fps)
Cu
mm
ula
tive
De
ns
ity
Asia
Brazil
US/Canada
Australia
Europe
Asia: 344Brazil: 297US/Canada: 1201Australia: 294Europe: 656
November 2001 SIGCOMM Internet Measurement Workshop
Frame Rate vs. User Geographic Location
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 5 10 15 20 25 30
Average Frame Rate (fps)
Cu
mm
ula
tive
De
ns
ity
Australia/Newzealand
US/Canada
Asia
Europe
November 2001 SIGCOMM Internet Measurement Workshop
Frame Rate vs. PC Type
November 2001 SIGCOMM Internet Measurement Workshop
Outline
• Introduction
• RealVideo
• Methodology
• Results
• Analysis– Frame Rate– Jitter – Perceived Quality
• Conclusions
November 2001 SIGCOMM Internet Measurement Workshop
Jitter
• Frame rate alone not enough– Also need smooth playout– High frame rate but choppy playout can be as
bad as frame loss [CT99]
• Variation in frame playout interval called jitter– Standard deviation
• Buffering can reduce jitter [RKTS94, SJ95]
– But can still have residual jitter
• Quantitative impact of residual jitter on perceived quality scarce– 50 ms imperceptible, 300 ms very rough
November 2001 SIGCOMM Internet Measurement Workshop
Overall Jitter
0102030405060708090
100
50 550 1050 1550 2050 2550 3050
Jitter (ms)
Cu
mu
lati
ve
De
ns
ity (%
)
November 2001 SIGCOMM Internet Measurement Workshop
Jitter vs. Network Configuration
0
10
20
30
40
50
60
70
80
90
100
50 550 1050 1550 2050 2550 3050Jitter (ms)
Cu
mu
lati
ve D
en
sit
y (%
)
56K ModemDSL/CableT1/LAN
November 2001 SIGCOMM Internet Measurement Workshop
Jitter vs. Server Geographic Location
0
10
20
30
40
50
60
70
80
90
100
50 1050 2050 3050
Jitter (ms)
Cu
mu
lati
ve
De
ns
ity (
%)
AustraliaUS/Canada
AsiaEuropeBrazil
November 2001 SIGCOMM Internet Measurement Workshop
Jitter vs. User Geographic Location
0
10
20
30
40
50
60
70
80
90
100
50 550 1050 1550 2050 2550 3050
Jitter (ms)
Cum
ulat
ive
Dens
ity (%
)
Australia/New zealand
US/Canada
Asia
Europe
November 2001 SIGCOMM Internet Measurement Workshop
Jitter vs. Observed Bandwidth
0
10
20
30
40
50
60
70
80
90
100
50 550 1050 1550 2050 2550 3050
Jitter (ms)
Cu
mu
lati
ve D
en
sit
y (%
)
< 10K
10K - 100K
> 100K
November 2001 SIGCOMM Internet Measurement Workshop
Outline
• Introduction
• RealVideo
• Methodology
• Results
• Analysis– Frame Rate – Jitter– Perceived Quality
• Conclusions
November 2001 SIGCOMM Internet Measurement Workshop
Perceived Quality
• Frame Rate and Jitter not enough– Best frame rate depends upon content [TC01]
• RealServer varies
– Effects of jitter depend upon content [CT99]
• Perceptual Quality– Numeric rating, 0-10
November 2001 SIGCOMM Internet Measurement Workshop
00.10.20.30.40.50.60.70.80.9
1
0 5 10 15 20 25 30 35
Clips Rated per User
Cu
mu
lati
ve D
en
sit
y
(388 video clips rated)
Clips Rated per User
November 2001 SIGCOMM Internet Measurement Workshop
Quality vs. Network Configuration
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 1 2 3 4 5 6 7 8 9 10
Quality Rating
Cu
mu
lati
ve D
en
sity
LAN/T1
DSL/Cable
56k Modem
November 2001 SIGCOMM Internet Measurement Workshop
Conclusions
• Average RealVideo clip has good quality– 10 frames per second
– Nearly imperceptible jitter
• Network configuration– Modems are a bottleneck
– DSL/Cable modems pushing bottleneck towards server
November 2001 SIGCOMM Internet Measurement Workshop
Future Work
• RealVideo over UDPTCP-Compatible?
• Perceptual quality of streaming video
• Other streaming players and protocols– MediaPlayer, QuickTime
• Effects of caching, CDNs
• Release of customizable RealTracer
• Download IP addresses, Playlist, RealTracer– From Web page
An Empirical Study of RealVideo Performance Across the Internet
Yubing Wang, Mark Claypool and Zheng Zuo
http://perform.wpi.edu/real-tracer
