ENEE631 Digital Image Processing (Spring'04) Basics on Video Communications and Other Video Coding Approaches/Standards Spring ’04 Instructor: Min Wu ECE

ENEE631 Digital Image Processing (Spring'04)

Basics on Video Communications andBasics on Video Communications and

Other Video Coding Approaches/StandardsOther Video Coding Approaches/Standards

Spring ’04 Instructor: Min Wu

ECE Department, Univ. of Maryland, College Park

www.ajconline.umd.edu (select ENEE631 S’04) [email protected]

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)

Based on ENEE631 Based on ENEE631 Spring’04Spring’04Section 13Section 13

ENEE631 Digital Image Processing (Spring'04) Lec20 – Video Coding (3) [2]

Quick Review – A Few Basics on VideoQuick Review – A Few Basics on Video

Acquisition, Display, Analog & Digital FormatsAcquisition, Display, Analog & Digital Formats

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Video CameraVideo Camera

Frame-by-frame capturing

CCD sensors (Charge-Coupled Devices)

– 2-D array of solid-state sensors– Each sensor corresponding to a pixel– Store in a buffer and sequentially read out– Widely used

small and light

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Video DisplayVideo Display

CRT (Cathode Ray Tube)– Large dynamic range– Bulky for large display

CRT physical depth has to be similar to screen width

LCD Flat-panel display– Use electrical field to change the optical properties hence the

brightness/color of liquid crystal– Generating the electrical field

by an array of transistors: active-matrix thin-film transistors by plasma

“Active-matrix display” (also known as TFT) has a transistor located at each pixel, allowing display be switched more frequently and less current to control pixel luminance. Passive matrix LCD has a grid of conductors with pixels located at the grid intersections

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Composite vs. Component VideoComposite vs. Component Video

Component video– Three separate signals for tristimulus color representation or luminance-

chrominance representation – Pro: higher quality– Con: need high bandwidth and synchronization

Composite video– Multiplex into a signal signal– Historical reason for transmitting color TV through monochrome

channel– Pro: save bandwidth– Con: cross talk

S-video: luminance sig. + single multiplexed chrominance sig.

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Analog Video RasterAnalog Video Raster

Line-by-line “Raster Scan”– Represent line-by-line image frame with 1-D analog

waveform– Synchronization signal for horizontal and vertical retrace

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Forming Picture on TV Tube (Monochrome)Forming Picture on TV Tube (Monochrome)

How many lines?

From B.Liu EE330S’01 Princeton

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


How Many TV Lines?How Many TV Lines?

Determined by spatial freq. response of HVS

dot

dot

Cannot resolve if

distance > 2000 x separation

(~ 0.03 degree viewing angle)

From B.Liu EE330S’01 Princeton

N = 500 for D=4H

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Review: Progressive vs. Interlaced scanReview: Progressive vs. Interlaced scanFrom B.Liu EE330S’01 Princeton

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Analog Color TV SystemsAnalog Color TV Systems

Historical notes – Color TV system had to be compatible with earlier monochrome TV system

3 formats– NTSC ~ North American + Japan/Taiwan – PAL ~ Western Europe + Asia(China) + Middle East– SECAM ~ Eastern Europe + France– What format in your home country?

From Wang’s Preprint Fig.1.5

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Comparison of Three Analog TV SystemsComparison of Three Analog TV Systems

– Spatial and temporal resolution– Color coordinate– Signal bandwidth– Multiplexing of luminance, chrominance, and audio

(From Wang’s Preprint)

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


NTSCNTSC

4:3 aspect ratio (width:height)

525 lines/frame, 2:1 interlace at field rate 59.94Hz– 483 active lines per frame; vertical retrace takes time of 9 lines– rest for broadcaster’s info. like closed caption

YIQ color coordinate for transmission– RGB primary slightly different from PAL– Orthogonal chrominance

I ~ orange-to-cyan; Q ~ green-to-purple (need less bandwidth)

Multiplexing over 6M Hz total bandwidth– Artifacts due to cross talk between luminance and chrominance

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


NTSC 6MHz Bandwidth NTSC 6MHz Bandwidth From Wang’s Preprint Fig.1.6(b)

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Analog Video RecordingAnalog Video Recording

Comparison of common formats

From Wang’s Preprint Table 1.2

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


Digital Video FormatsDigital Video Formats ITU-R BT.601 recommendation

Downsampled chrominance– Y Cb Cr coordinate and four subsampling formats

Inter. Telecomm. Union – Radio sector

Wang’sPreprint Fig.1.8

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


From Wang’sPreprint

Table 1.3

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

01

)


ResourceResource

Background and Motivation on Background and Motivation on Multimedia Coding / CommunicationsMultimedia Coding / Communications

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Generations of Video CodingGenerations of Video Coding

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)

From R.Liu Seminar Course ’00 @ UMCP


Channel Bandwidth Channel Bandwidth

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)



UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)

Storage CapacityStorage Capacity

UMCP ENEE408G Slides (created by M.Wu & R.Liu © 2002)



Source Video FormatsSource Video Formats

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)



Application RequirementsApplication Requirements

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)



Other Standard and Considerations for Other Standard and Considerations for

Digital Video Coding Digital Video Coding

UM

CP

EN

EE

63

1 S

lide

s (c

rea

ted

by

M.W

u ©

20

04

)


Performance Tradeoff for Video CodingPerformance Tradeoff for Video Coding

From R.Liu’s Handbook Fig.1.2:

“mos” ~ 5-pt mean opinion scale of bad, poor, fair, good, excellent

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


H.26x for Video TelephonyH.26x for Video Telephony Remote face-to-face communication: A dream for years

H.26x – Video coding targeted low bit rate– Through ISDN or regular analog telephone line ~ on the order of 64kbps – Need roughly symmetric complexity on encoder and decoder

H.261 (early 1990s)– Similar to simplified MPEG-1 ~ block-based DCT/MC hybrid coder– Integer-pel motion compensation with I/P frame only ~ no B frames– Restricted picture size/fps format and M.V. range

H.263 (mid 1990s) and H.263+/H.263++ (late 1990s)– Support half-pel motion compensation & many options for improvement

H.264 (latest, 2001-): also known as H.26L / JVT / MPEG4 part10

– Hybrid coding framework with many advanced techniques– Focusing on greatly improving compression ratio at a cost of complexity

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


MPEG-2MPEG-2

Extend from MPEG-1

Target at high-resolution high-bit-rate applications

– Digital video broadcasting, HDTV, …– Also used for DVD

Support scalability

Support interlaced video

– Frame pictures vs. Field pictures– New prediction modes for motion compensation related to interlaced

video Use previously encoded fields to do M.E.-M.C.U

MC

P E

NE

E4

08

G S

lide

s (c

rea

ted

by

M.W

u &

R.L

iu ©

20

02

)


Scalability in Video CodecsScalability in Video Codecs

Scalability: provide different quality in a single stream– Stack up more bits on base layer to provide improved quality

Possible ways for achieving scalabilities– SNR Scalability ~ Multiple–quality video services

Basic vs. premium quality

– Spatial Scalability ~ Multiple-dimension displays Display on PDA vs. PC vs. Super-resolution display

– Temporal Scalability ~ Multiple frame rates– Frequency Scalability ~ Blurred version to sharp, detailed version

Layered coding concept facilitates:– Unequal error protection – Efficient use of resources– Different needs from customers – Multiple services

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


SNR ScalabilitySNR Scalability

Two layers with same spatio-temporal resolution but different qualities

base-layerencoder

base-layerdecoder

enhancement-layerencoder

mul

tipl

exer+ -

Video inBase-layerbitsteam

Enhancement-layerbitsteam

Outputbitsteam

From R.Liu Seminar Course @ UMCP


MPEG-4MPEG-4

Many functionalities targeting a variety of applications

Introduced object-based coding strategy– For better support of interactive applications & graphics/animation video– Require encoder to perform object segmentation

difficult for general applications

Introduced error resilient coding techniques– “Streaming video profile” for wireless multimedia applications

Part-10 is converged into H.264– Focused on improving compression ratio and error resilience– Stick with Hybrid coding frameworkU

MC

P E

NE

E4

08

G S

lide

s (c

rea

ted

by

M.W

u &

R.L

iu ©

20

02

)


Object-based Coding in MPEG-4Object-based Coding in MPEG-4

Interactive functionalities Higher compression

efficiency by separately handling – Moving objects– Unchanged background– New regions– M.C.-failure regions=> “Sprite” encoding

Object segmentationneeded (not easy )– Based on color, motion,

edge, texture, etc.– Possible for targeted

applications

Revised from R.Liu Seminar Course @ UMCPU

MC

P E

NE

E4

08

G S

lide

s (c

rea

ted

by

M.W

u &

R.L

iu ©

20

02

)


Analysis-Synthesis Coding: A General FrameworkAnalysis-Synthesis Coding: A General Framework

From R.Liu Seminar Course @ UMCP


MPEG-7MPEG-7

“Multimedia Content Description Interface”

– Not a video coding/compression standard like previous MPEG– Emphasize on how to describe the video content for efficient

indexing, search, and retrieval

Standardize the description mechanism of content

– Descriptor, Description Scheme, Description Definition Languages– Example of MPEG-7 visual descriptor: Color, Texture, Shape, …

Figure from MPEG-7 Document N4031 (March 2001)

UM

CP

EN

EE

40

8G

Slid

es

(cre

ate

d b

y M

.Wu

& R

.Liu

© 2

00

2)


Documents

ENEE631 Digital Image Processing (Spring'04) Basics on Video Communications and Other Video Coding Approaches/Standards Spring ’04 Instructor: Min Wu ECE