Video Coding for Streaming Media Delivery on the InternetConklin, G. et al, 2000

Jake Wires

Motivation

Original standards designed for constant bit rate encoding

Constant bit rate encoding does not account for the channel variability inherent in many network applications

Large demand for streaming multimedia—over 350,000 hours of live content broadcast per week

History of Streaming Media Technologies

VivoActive 1.0HTTP server-client transactions via TCPH.263 video codec, G.723 audio codecMultiplexing protocol to combine audio &

videoVery simple

History: VivoActive

ProblemsHTTP’s lack of flow control and TCP’s

guaranteed delivery mechanisms resulted in large fluctuations of fragment delivery times

Necessitated large preroll buffer (5 – 20 sec)Rebuffering (filling empty buffer) diminished user

expereinceSession Control (i.e. f.f., rew.) difficult

History: RealAudio 1.0

Introduced in March 1995Relied on Progressive Networks

Architecture—a suite of dedicated network protocolsUsed 2 connections:

TCP-based session control connectionUDP-based audio transmission connection

History: RealAudio 1.0

Use of UDP-based transmission introduced possibility for lost, delayed, and out of order packets Automatic Repeat-Request mechanism allowed

clients to request retransmission of lost packets Frame interleaving technique minimized perceptual

damage Send packets of interleaved (rather than continuous)

audio frames Dropped packet results in multiple small disruptions rather

than single large disruption

History: RealVideo

RealVideo design features: Produced compressed data that can be streamed

at a fixed bit rate Non-uniform nature of video clips (scene changes,

transitions, etc.) results in unequal distribution of bits between frames when encoded at same level of distortion

To maintain fixed bit rate, encoder must introduce unequal distortion between frames, neglect encoding some frames, or both

Use of preroll buffer increases bit rate flexibility—only average rate must be kept constant

Variable-bit-rate (VBR) rate-control algorithm

History: RealVideo

Facilitated random access by inserting Intra frames

Employed unequal error protectionLarge video frame size precluded simple

interleaving techniques Instead, RealVideo combined forward error

correction codes for the most sensitive data with various error concealment mechanisms

History: RealSystem G2

SureStream technology improved adaptive streaming capabilities: A single SureStream file contains multiple streams

of the original content Different streams optimized for different channel

capacities Client of stream monitors connection performance,

dynamically requests optimal stream from server

Used IETF standardized RTP and RTSP protocols

History: Network Strategies

Multicasting reduces amount of redundant data transferred

Streaming media delivery networks allow for multiple, geographically scattered servers (achieved either by splitting or caching)

Dedicated networks guarantee bandwidth Multiple-access transmissions over Internet—

requested info sent from several different locations simultaneously

Streaming Media

Three modes in which video information can be streamed:Live broadcastingOn-demand streamingConferencing

Streaming Media: Live Broadcasting System Structure

Streaming Media: Live Broadcasting

Live video encoded on-the-fly, passed to server, which disperses compressed info to all splitters and/or clients Splitters are additional servers that can be used to

distribute server workload over a number of machines

Push splitting: initiated by server Pull splitting: initiated by a client connecting to a local

splitter; request propagated to nearest active splitter Multiple-access splitting: lower-tier splitters can connect to

multiple upper-tier splitters

Streaming Media: On-demand System Structure

Streaming Media: On-demand

Distribution system similar to live broadcasting system, except:No direct connection between encoder and

serverRemote proxy servers can cache popular

filesClients allowed to rewind, fast forward, etc.Server proxy transfers only initiated by client

Video Coding Problems

Channel capacity fluctuations Adaptive client-driven serving

Allow client to inform server of network performance Congestion control Error control: use bandwidth prescribed by congestion

control algorithm to minimize effects of packet loss

Dynamic prediction Combined with sufficient preroll buffer, a broad set of

statistical techniques can (possibly) be used to throttle transfer rate to improve user experience

Video Coding Problems

Random accessCan provide random access by inserting

I-frames at the boundaries of small video intervals

Reduces achievable compression ratesProcessing resources

Diverse range of client machinesLive encoding often done on computer with

limited resources

Video Coding Algorithms for Streaming

Original motion-compensated hybrid video coding algorithms produced single-rate encoded versions of input signals Inadequate for many network applications

Need scalable coding algorithms

Scalable Streaming Techniques

Simulcast: produce several, redundant, independently coded streams, each optimized for specific channel conditions Not suitable for splitting, where the combined

representation of all streams should be as small as possible

Scalable coding Based on spatio-temporal pyramid decomposition

of source signal Temporal, spatial, and Signal to Noise Ratio (quantizer

size) scalability modes

Scalable Streaming Techniques

Multiple-description coding for use in multiple-access systemsAchieved by replacing scalar quantization

with multiple-description quantization scheme

RealSystem 8

Adaptive channel encoding of streaming content to match bit rate with bandwidth and loss statisticsChannel statistics not available until time of

streaming, so server must manage adaptive strategies

Infeasible for server to encode on demand, so encoding techniques are required which allow server to efficiently choose between different bit rates

RealSystem 8

SureStream components:Adaptive Stream Management (ASM)SureStream file formatSource and channel coding algorithms

RealSystem 8

ASM Allows clients to communicate which type of

encoding should be ‘synthesized’ by server File format plug-in uses ASM rules to produce

various combinations of encoded streams from compressed media files

Rules expressed by fully programmable syntax Client subscribes to appropriate rule based on

channel characteristics ASM is format-independent

RealSystem 8

RealVideo 8 encoding module:

RealVideo 8 Algorithm

Input filters: remove noise and potential artifacts introduced by edits and conversions of video signal Remove low-energy spatial noise De-interlace analog signals to produce progressive

frames without “jaggy”-shaped artifacts Inverse telecine filter: remove redundant 5.97 frames

introduced by NTSC video

RealVideo 8 Algorithm

Spatial resampler: downscale input frames to spatial resolutions suitable for streaming

Codecs: motion-compensated hybrid scheme Rate control determines which frames to

encode Low bit rates: perceived quality impacted by

relationship between framerate and frame quality High bit rates: must maintain framerate, want to

maintain appearance of uniform high quality Parameterized preroll buffer size Rate control important in providing resiliency and

random access

RealVideo 8 Algorithm

CPU scalability control module: throttles complexity of different components of algorithm to maintain best quality level for encoding live presentations

RealVideo 8 Algorithm

Client-side video postprocessing: filters aimed at improving subjective quality of decoded signals Frame rate upsampler: special temporal filter that

attempts to interpolate intermediate frames Especially useful when many original frames were

regularly skipped to decrease bit rate of encoding

Pointers

Helix Community “The Helix community is a collaborative

effort among Real, independent developers, and leading companies to extend the Helix DNA™ platform, the first open multi-format platform for digital media creation, delivery and playback.”

https://helixcommunity.org/

Conclusion

Streaming multimedia poses many challenges to video encoders Variability of channel performance High server workloads

Need to adapt coded bit rate to match channel capacities; must be done efficiently and on the fly

“RealSystem 8 provides an open and extensible platform, capable of accommodating various future needs of streaming media infrastructure on the Internet, and in particular, new demands for improved video-coding techniques.”