§3 Discrete memoryless sources and their rate-distortion function §3.1 Source coding §3.2...

§3 Discrete memoryless sources and their rate-distortion function

§3.1 Source coding

§3.2 Distortionless source coding theorem

§3.3 The rate-distortion function

§3.4 Distortion source coding theorem

§3.1 Source coding

§3.2 Distortionless source coding theorem

§3.3 The rate-distortion function

§3.4 Distortion source coding theorem

1. Source coder

§3.1 Source coding

Source coder}1...,1,0{ rA },...,,{ 110 rC

|| iin

}1,...,1,0{ sS

)...,,( 110 rpppp

1 2( ), , ( 1, ..., )ii i i in ij is s s s S j n

Source alphabet

Channel input alphabet

Code

mnm

1 2( ), , ( 1, ..., )ii i i in ij is s s c S j n

Extendedsource coder

mA

}1,...,1,0{ sS

1 2( )mU U U U

iU A

Example 3.1

§3.1 Source coding

2. Examples

1) ASCII source coder

ASCII coder

{0,1}

{English symbol , command} {binary code, 7 bits}

2) Morse source coder

Source coder(1)

Source coder(2)

{0,1}{. , —}

{A,B,…,Z} Binary code

2. Examples

§3.1 Source coding

3) Chinese telegraph coder

“中”

“0022”

“01101 01101 11001 11001”

2. Examples

§3.1 Source coding

Constant-length codes

Variable-length codes

Distortionless codes

Distortion codes

2. Classification of the source coding

Uniquely decodable (UD) codes

Non-UD codes

§3.1 Source coding

in n

in

( ; ) ( )I A C H A

( ; ) ( )I A C H A

The code C is called uniquely decodable (UD) if each string in each Ck arises in only one way as a concatenation of codewords. This means that if say

and each of the τ’s and σ’s is a codeword, then

Thus every string in Ck can be uniquely decoded into a concatenation of codewords.

1 2 1 2* * * * * *k k

1 1 2 2, , , .k k

2. Classification of the source coding

Example 3.2

§3.1 Source coding

Source symbol

si

Symbol probability

P(si) Code

1

Code

2

Code

3

Code

4

Code

5

S1 1／2 0 0 1 1 00 S2 1／4 11 10 10 01 01 S3 1／8 00 00 100 001 10 S4 1／8 11 01 1000 0001 11

3. Parameters about source coding

1) Average length of coding

1

0

1

0

||)(r

iii

r

iii nppn

mr

i

mi

mim npn

1

For extended source coding:

(code/sig)

code/m-sigs

Length of codeword

§3.1 Source coding

mnn

m (code/sig)

2) Information rate of coding

( ) /R H p n

( )

/mm

H pR

n m

(bit/code)

(bit/code)

3. Parameters about source coding

§3.1 Source coding

3) Coding efficiency

smn

pH

m log

)(

Actual rate

Maximum rate

3. Parameters about source coding

§3.1 Source coding

log

R

s

For extended source coding:

s

npH

log

/)(

§3.2 Distortionless source coding theorem

§3 Discrete memoryless sources and their rate-distortion function

§3.1 Source coding

§3.2 Distortionless source coding theorem

§3.3 The rate-distortion function

§3.4 Distortion source coding theorem

Example 3.3

The binary DMS has the probability space:

4

1

4

3)(

21 aa

aP

A

i

§3.2 Distortionless source coding theorem

1) “0” a1, “1” a2

2) a1a1: 0 a1a2: 10 a2a1: 110 a2a2: 111

)/(811.03

4log

4

34log

4

1)( signbitAH

Average length of coding: )/(11 sigcoden

Code efficiency: 811.01

§3.2 Distortionless source coding theorem

“0” a1, “1” a2

Rate:1

1

( )0.811 ( / )

H pR bit code

n

Example 3.3

4

1

4

3)(

21 aa

aP

A

i

Extended source coding

i )( iP code Length of codeword

a1a1 16

90 1

a1a2 16

310 2

a2a1 16

3110 3

a2a2 16

1111 3

Average length of coding :3

16

13

16

32

16

31

16

92 n

Code efficiency:

)/(961.0844.0

811.02 codebitR

Rate:

961.02

)2/(688.1 sigscode

)/(844.02

2 sigcoden

§3.2 Distortionless source coding theorem

Example 3.3

m times extended source coding

m = 3: 985.03

R3 = 0.985 (bit/code)

m = 4: 991.04

R4 = 0.991 (bit/code)

m 1m

§3.2 Distortionless source coding theorem

Example 3.3

§3.2 Distortionless source coding theorem

Distortionless source coding theorem

Theorem 3.1 If the code C is UD, its average length mustexceed the s-ary entropy of the source , that is,

1

0

log)(r

iisis pppHn

(Theorem 11.3 in textbook)

§3.2 Distortionless source coding theorem

Distortionless source coding theorem

Theorem 3.2

1)()()( pHpnpH sss

(Theorem 11.4 in textbook)

§3.2 Distortionless source coding theorem

Theorem 3.3 )()(

1lim pHpn

m sm

sm

(Theorem 11.5 in textbook)

Distortionless source coding theorem

The source can indeed be represented faithfully using s-ary symbols per source symbol.

p

( )sH p

§3.2 Distortionless source coding theorem

Distortionless source coding theorem

corollary

The efficient UD codes are achievable if rate R ≤ C.

(C is the capacity of s-ary lossless channel )

Review

• KeyWords:

Source coder

Variable-length codes

distortionless codes

Uniquely decodable codes

Average length of coding

Information rate of coding

Coding efficiency

Shannon’s TH1

Homework

1. p344: 11.12

2. p345:11.20

§3 Discrete memoryless sources and their rate-distortion function

§3.1 Source coding

§3.2 Distortionless source coding theorem

§3.3 The rate-distortion function

§3.4 Distortion source coding theorem

§3.3 The rate-distortion function

1. IntroductionReview

Distortionless source coding theorem (corollary) The efficient UD codes are achievable if rate R ≤ C.

(C is the capacity of s-ary lossless channel )

Conversely, any sequence of (2nR, n) codes with must have R ≤ C.

0EP

The channel coding theorem (Statement 2 ):

All rates below capacity C are achievable. Specifically,

for every rate R ≤ C, there exists a sequence of (2nR,n) codes

with maximum probability of error .0EP

§3.3 The rate-distortion function

1. Introduction

Review

For distortionless coding: R≤C - (PE→0, R→C - )

But actually……

Given a source distribution and a distortion measure, what is the minimum expected distortion achievable at aparticular rate?what is the minimum rate description required to achieve a particular distortion?

§3.3 The rate-distortion function

2. Distortion measure

coding channelui

vj

AU={u1,u2,…,ur} AV={v1,v2,…,vs}

k

iii vudvud

1

),(),(

kV

kUkk AAvvvuuuvuif ),...,,;,...,,(),( 2121

( , )i jd u v

Source symbol Destination symbol

2. Distortion measure

Average distortion measure:

, ,

( ) [ ( , )] ( ) ( , ) ( ) ( | ) ( , )u v u v

D k E d U V p uv d u v p u p v u d u v

Let the input and output of the channel be U=(U1,U2,…,Uk)and V=(V1,V2,…,Vk) respectively

kV

kUkk AAvvvuuuvu ),...,,;,...,,(),( 2121

where,

§3.3 The rate-distortion function

,

,

( ) [ ( , )] ( ) ( , )

( ) ( | ) ( , )

i j i j i jU V

i j i i jU V

D E d E d u v p u v d u v

p u p v u d u v

Example 3.3.1

AU = AV = {0,1};source statistics p(0) = p, p(1) = q = 1-p,where p ½; and distortion matrix

01

10D

2. Distortion measure

§3.3 The rate-distortion function

Example 3.3.2

AU = {-1,0,+1}, AV = {-1/2, +1/2};source statistics(1/3,1/3,1/3)and distortion matrix

12

11

21

D

2. Distortion measure

§3.3 The rate-distortion function

2. Distortion measure

Fidelity criterion:

§3.3 The rate-distortion function

( )D or D k k ,

Test channel:

Let the source statistics p(u) and distortion measure d(u,v) are fixed.

( | ) :j iB P v u D

( | ) : ( )or B P V U D k k

3. Rate-distortion function

1) Definition

The function is a function of the source statistics(p(u)) ,the distortion matrix D, and the real number .

)(kR

§3.3 The rate-distortion function

The information rate distortion function Rk(δ) for asource U with distortion measure d(U, V) is defined as

1 1, ( ,

( )

) (( , ..., ), ( ,

min{ ( ; )

..., ))

: ( ) }k

k kwhere U V

R I U V D k k

U U V V

The information rate distortion function Rk(δ) for asource U with distortion measure d(U, V) is defined as

3. Rate-distortion function

UAu

vvudup ),(min)(min

21 )()( 21 kk RR ③If , then

§3.3 The rate-distortion function

②The minimum possible value of is ,wheremink( )D k

R(δ) and C

①The function I(U;V) actually achieves its minimum value on the region of ;D

3. Rate-distortion function

2) Properties

Theorem 3.4 is a convex function of .)(kR min (Theorem 3.1 in textbook)

§3.3 The rate-distortion function

R(0)=H(U)

maxmin

( )kR

max

( )kR

maxmin},:);(min{)( DVUIR

max( ) 0, iff.R

3. Rate-distortion function

2) Properties

Theorem 3.4 is a convex function of .)(kR min

Theorem 3.5 For a DMS, for all k and .min )()( 1 kRRk

(Theorem 3.1 in textbook)

(Theorem 3.2 in textbook)

§3.3 The rate-distortion function

3. Rate-distortion function

Example 3.3.1 (continued)

AU = AV = {0,1};source statistics p(0) = p, p(1) = q = 1-p,where p ½; and distortion matrix

01

10D

§3.3 The rate-distortion function

min

max

( ) (0) ( ) ( ),

( ) 0

R R H U H p

R

D

2) Properties

with different

(bit/sig)

0.0

1.0

0.8

0.6

0.4

0.2

0.50.40.30.20.1

0.5p

0.2p

0.1p

( )R

( )R p

0.3p

§3.3 The rate-distortion function

AU = AV = {0,1,…,r-1},

P{U=u}=1/r

Distortions are given by:

vuif

vuifvud

,1

,0),(

0.6 1.0

3.0

2.0

1.0

with different r

(bit/sig)

0.0 0.80.40.2

( )R

( )R

8r4r2r

2) Properties

Example 3.3.3

3. Rate-distortion function

§3 Discrete memoryless sources and their rate-distortion function

§3.1 Source coding

§3.2 Distortionless source coding theorem

§3.3 The rate-distortion function

§3.4 Distortion source coding theorem

1. Distortion source coding theorem

(modified on Theorem 3.4 in textbook)

§3.4 Distortion source coding theorem

Theorem 3.6 (Shannon’s source coding theorem with a fidelity criterion) If , there exists a source code C of length k with M codewords, where:

)(RR

Db

Ma kR

)(

2)(

If ,no such codes exist.)(RR A source symbol can be compressed into R(δ) bits

if a distortion δ is allowable.

2. Relation of shannon’s theorems

§3.4 Distortion source coding theorem

Source Distortion

source coder

Distortionless source coder

Sink Distortion

source decoder

Distortionless source decoder

channel

Channel coder

Channel decoder

A general communication system

Review

• KeyWords:

Distortion measure

Average distortion measure

Fidelity criterion

Test channel

Rate-distortion function

Shannon’s TH3

thinking

Source X has the alphabet set {a1,a2,…,a2n},P{X = ai}=1/2n,i = 1,2,…,2n. The distortion measure is Hamming distortionmeasure ,that is

ji

jid ij ,0

,1

Design a source coding method with δ=1/2.

§3.4 Distortion source coding theorem