Coding - Khoa Toántatuana/Xu Ly Anh/Ebook Others/CS450 Slide...The theoretical compression limit of entropy coding is the entropy of the ... Predictively encode DC coeff. ... Zig-zag

CS 450 Compression 1

Coding

All information in digital form must be encoded somehow.

Examples:

• Binary numbers• ASCII• IEEE floating-point standard

Coding can be lossless or lossy.

Coding can be efficient or inefficient.

Compression is nothing but efficient coding.


Information

Informationis something unknown.

More probabilistically, it is somethingunexpected.


Example: Information

What’s the missing letter?

H E L O

Now, what do you think is missing?

A Y


Information vs. Data

Data: the actual bits, bytes, letters, numbers, etc.

Information: the content

Redundancy: difference between data and information

redundancy= data− information


Quantifying Information: Entropy

Information content of symbolw measured in bits:

infow = − log2 p(w)

wherep(w) is the probability of symbolw occurring.

The average bits for a language withn symbols is

information =n∑i=1

p(wi) infowi

= −n∑i=1

p(wi) log2 p(wi)

wherep(wi) is the probability of symbolwi occurring.


Context

Information is based on expectation.

Expectation is based on context.

Examples of contexts:

• lastn letters

• lastn values for a time-sampled sequence

• neighboring pixels in an image


Calculating Information in Context

Without considering context, a string ofn symbols has

information=n∑i=1

d− log2 p(wi)e

wherep(wi) is the probability of symbolwi occurring.

Considering context, a string ofn symbols may have

information=n∑i=1

d− log2 p(wi|w0 . . . wi−1)e

wherep(wi|w0 . . . wi−1) is the probability of symbolwi occurring

immediately after symbolsw0 . . . wi−1.

Can do better by considering multiple symbols as one.


Types of Redundancy

Remember:

redundancy= data− information

For images, there are three types of redundancy:

• Coding

• Interpixel

• Visual


Entropy Coding

Entropy coding allocates bits per symbol or group of symbols according

to information content (entropy).

Huffman Coding:

Optimal unique coding on a per-symbol basis.

Arithmetic Coding:

Encodes a sequence of symbols as an infinite-precision real number.

(About 2:1 better than Huffman)

The theoretical compression limit of entropy coding is the entropy of the

language (set of symbols) itself.


Vector Quantization

Divide the signal into sets ofn values and treat as one collectivevector

value.

Quantize the vectors to some finite number of them—this causes loss but

reduces the space of possible values.

Use a unique code for each vector and transmit that code, not the vector.

Most VQ systems intelligently select the quantization based on the signal

content—this means you have to send the codebook with the codes.

Can extend to images usingn x m blocks.

Example:

4 x 4 blocks, 256 levels per pixel =25616 possibilities = 128 bits per block

Choose 4096 possibilties = 12 bits per block


Interpixel Redundancy

The basis of interpixel redundancy is

• repetition

• prediction


Run-Length Encoding

Encode sequences of identical symbols as (symbol,count) pairs.

Can extend to multiple dimensions.

Useful for binary signals and black-and-white images.

2-D RLE is used in the CCITT fax standard.


Lempel-Ziv

Idea: encode longest possible previously-seen sequence

Coding stream is mixture of symbols and back-pointers

Used in most common text compression algorithms, zip, and the

GIF image standard.


Lempel-Ziv: Basic Idea

sequence = empty

get(symbol)

while (symbol)

if sequence + symbol starts a previous sequence

sequence += symbol

get(symbol)

else

if empty(sequence)

write symbol

get(symbol)

else

write pointer to start of previous sequence

sequence = empty


Lempel-Ziv: Actual Algorithm

codebook = empty

sequence = empty

while (get(symbol))

if sequence + symbol is in codebook

sequence += symbol

else

add sequency+symbol to codebook

output(new code)


Example: Lempel-Ziv

Mary had a little lamb,

little lamb, little lamb,

Mary had a little lamb,

its fleece was white as snow.


Predictive Coding

Use one set of pixels to predict another.

Predictors:

• Next pixel is like the last one• Next scanline is like the last one• Next frame was like the last one• Next pixel is the average of the already-known neighbors

The error from the prediction (residual) hopefully has smaller entropy

than the original signal.

The information used to make the prediction is thecontext.


Predictive Coding

Sender and receiver use the same predictive model.

Sender:

1. Make prediction

2. Send the entropy-encoded difference (the residual)

Receiver:

1. Make prediction

2. Add the residual


Delta Modulation

Prediction: next signal value is the same as the last.

Residual is the difference (delta) from the previous one.

Residual is encoded in a smaller number of bits than the original.

Often used in audio systems (phones).

Problem: limited-precision delta can cause under/overshoot.


Predictive Image Coding

Prediction: next pixel is weighted average of the neighbors

that have been previously seen in scanline order.

Can use larger context.

Used in DPCM and the lossless part of the JPEG standard.

Newer algorithms (CALOC, LOCO-I) use multiple contexts.


Visual Redundancy

Eye is less sensitive to

• Color

• High Frequencies

So,

• Allocate more bits to intensity than chromaticity.

• Allocate more bits to low frequencies than to high frequencies


Transform Coding

Use some transform to convert from spatial domain to another

(e.g., a frequency-based one).

Quantize coefficients according to information content

(e.g., quantize high frequencies more coarsely than low ones).

Problem: artifacts caused by imperfect approximation in one place get

spread across the entire image.

Solution: independently transform and quantizeblocksof the image:

block transform encoding.


JPEG

Intensity/Chromaticity Can allocate more bits to intensity

than to chromaticity

8 x 8 block DCT Energy compaction

Predictively encode DC coeff. Takes advantage of redundancy in

the block averages

Quantize AC coefficients Many high frequencies become

zero!

Zig-zag ordering Changes from 2-D to 1-D and puts

similar frequencies together

Run-length encoding Collapses long runs of zeroes

Entropy encode what’s left To more efficiently encode the RLE

sequences


Evaluating Compression

Compression rate is usually expressed in bits-per-symbol (bps).

Compression rate isn’t the only thing!

Must also consider andistortion(error or loss) introduced.

Rate-Distortion Curves


Typical Compression Rates

Application Uncompressed Compressed

Voice 64 kbps 2-4 kbps

(8 ksamples/sec, 8 bits/sample)

Audio conference 64 kbps 16-64 kbps

(8 ksamples/s, 8 bits/sample)

Digital Audio 1.5 Mbps 128 kps–1.5 Mbps

(stereo, 44.1 ksamples/s, 16 bits/sample)

Slow Motion Video 5.07 Mbps 8-16 kbps

(10 fps, frame size 176x120, 8 bits/pixel)


Typical Compression Rates

Application Uncompressed Compressed

Video Conference 30.41 Mbps 64-768 kbps


Video File Transfer 30.41 Mbps 384 kbps


Digital Video on CD-ROM 60.83 Mbps 1.5–4 Mbps


Broadcast Video 248.83 Mbps 3–8 Mbps


HDTV 1.33 Gbps 20 Mbps

(59.94 fps, frame size 1280x720, 8 bits/pixel)


Progressive Methods

Interlaced GIF:

Send every 8 scanlines, then every 4, then 2, then all.

Interpolate intermediate lines until they get there.

Progressive JPEG:

Send DC coefficients

Send all lowest-frequency AC coefficients

Send successively higher AC coefficients


Other Compression Methods

Wavelets:

Similar in principle to block-transform-based methods.

Greater compression/fidelity by exploiting both spatial and frequency

information.

Fractals:

Blocks of the image are represented as rotated, translated, and rescaled

copies of other blocks.

Probably the highest claimed compression rates.

Useful in some applications; very lossy in others.

Some people are researching hybrid methods based on fractal

compression.

Documents

Coding - Khoa Toántatuana/Xu Ly Anh/Ebook Others/CS450 Slide...The theoretical compression limit of entropy coding is the entropy of the ... Predictively encode DC coeff. ... Zig-zag