Towards Practical Distributed Coding
Bernd GirodBernd Girod
Information Systems Information Systems LaboratoryLaboratory
Stanford UniversityStanford University
B. Girod: Towards Practical Distributed Coding 2
Outline
Distributed lossless compression– Simple examples– Slepian-Wolf Theorem– Slepian-Wolf coding vs. systematic channel coding– Turbo codes for compression
Lossy compression with side information– Wyner-Ziv Theorem– Optimal quantizer design with Lloyd algorithm
Selected applications– Sensor networks– Low complexity video coding– Error-resilient video transmission
B. Girod: Towards Practical Distributed Coding 3
Simple Example
1212
Xx p xabcd
101001000
001011000
Source A
1212
Xx p xabcd
Source B
Source statistics exploited in the encoder.Different statistics Different code.
B. Girod: Towards Practical Distributed Coding 4
Simple Example - Revisited
1212
Xx p xabcd
0101
Source A
1212
Xx p xabcd
Source B
0101
Different statistics Same code. Source statistics exploited in the decoder.“Lossless” compression with residual error rate.
B. Girod: Towards Practical Distributed Coding 5
X
| |121 12 2
12
| |0
00 0
X Y X Yx p x A p x Babcd
Compression with Side Information
SourceA/B
Encoder Decoder
Y
141214
0
Xx p xabcd
X
1.5 bitR H X
X
Y Y | 1 bitR H X Y ??R
B. Girod: Towards Practical Distributed Coding 6
Distributed Compression of Dependent Sources
Source X
Source Y
Encoder X
Encoder Y
JointDecoder
1 8 1 81 8 1 8
1 8 1 81 8 1 8
XYp x a b c dy a
bcd
X
Y
??XR
??YR
B. Girod: Towards Practical Distributed Coding 7
Achievable Rates for Distributed Coding Example
[bits]XR
[bits]YR
1
1
2
2
Separate encodingand decoding of X and Y
Separate encodingand joint decoding of X and Y
B. Girod: Towards Practical Distributed Coding 8
General Dependent i.i.d. Sequences
[bits]XR
[bits]YR
H X
H Y
|H Y X
|H X Y
,X YR R H X Y
Vanishing error probabilityfor long sequences
No errors
[Slepian, Wolf, 1973]
B. Girod: Towards Practical Distributed Coding 9
Distributed Compression and Channel Coding
Idea Interpret Y as a “noisy” version of X with
“channel errors” Encoder generates “parity bits” P to
protect against errors Decoder concatenates Y and P and
performs error-correcting decoding
SourceX|Y
Encoder Decoder
Y
X
YX Y
010011000101010100110101010100000001000000
XY
P
B. Girod: Towards Practical Distributed Coding 10
Practical Slepian-Wolf Encoding
Coset codes [Pradhan and Ramchandran, 1999] Trellis codes [Wang and Orchard, 2001] Turbo codes
[Garcia-Frias and Zhao, 2001] [Bajcsy and Mitran, 2001] [Aaron and Girod, 2002]
LDPC codes [Liveris, Xiong, and Georghiades, 2002]
B. Girod: Towards Practical Distributed Coding 11
12
n
RX
Interleaver length L
1PX
XL bits in
L bitsSystematic Convolutional Encoder Rate n
n 1
1nL
bits
2PX
Systematic Convolutional EncoderRate n
n 1
1nL
bits
L bits
Compression by Turbo Coding
[Aaron, Girod, DCC 2002]
B. Girod: Towards Practical Distributed Coding 12
Interleaver length L
1PX
L bits out
“Channel” probabilities calculations
1nL
bits in 2PX “Channel”
probabilities calculations
Y
1nL
bits in
)|( yxP
SISO Decoder
Pchannel
PextrinsicPa priori
Interleaver length L
Deinterleaver length L
SISO Decoder
Pchannel
Pextrinsic Pa prioriDeinterleaver
length L
Decision X
Pa posteriori
Pa posteriori
[Aaron, Girod, DCC 2002]
Turbo Decoder
B. Girod: Towards Practical Distributed Coding 13
0.15 bit
Results for Compression of Binary Sequences
X,Y dependent binary sequences with symmetric cross-over probabilities
Rate 4/5 constituent convolutional codes;
RX=0.5 bit per input bit
[Aaron, Girod, DCC 2002] |H X Y
B. Girod: Towards Practical Distributed Coding 14
)|( YXHRX
),()(
YXHYHRX
1.3511.445
1.033 1.0551.114
1.404
0.000
0.200
0.400
0.600
0.800
1.000
1.200
1.400
1.600
1 2 3
Results with Puncturing
0.5 bitsxR 0.25 bitsxR 0.125 bitsxR
[Aaron, Girod, DCC 2002]
B. Girod: Towards Practical Distributed Coding 15
Channel Coding vs. Slepian-Wolf Coding
Systematic Channel Coding
Systematic bits and parity bits subject to bit-errors
Mostly memoryless BSC channel
Low bit-error rate of channel
Rate savings relative to systematic bits
Slepian-Wolf Coding
No bit errors in parity bits
General statistics incl. memory
Whole range of “error” probabilities
Rate savings relative to parity bits
Might have to compete with conventional compression (e.g., arithmetic coding)
B. Girod: Towards Practical Distributed Coding 16
Distributed Lossy Compression of Dependent Sources
Source X
Source Y
Encoder X
Encoder Y
JointDecoder
X’
Y’
XR , 'XD E d X X
, 'YD E d Y YYR
, , , ??X Y X Yf R R D D Achievable rate region
B. Girod: Towards Practical Distributed Coding 17
X
Lossy Compression with Side Information
'X
X XR R D
Source Encoder Decoder
Y
, 'XD E d X X
Y Y
|X X Y XR R D | |wz
X X Y X X Y XR R D R D [Wyner, Ziv, 1976]
| |
, i.i.d. jointly Gaussian, mse distortion
wzX Y X X Y X
X Y
R D R D
| | |
General statistics, mse distortion
0.5 bits wzX Y X X Y X X Y XR D R D R D
[Zamir,1996]
B. Girod: Towards Practical Distributed Coding 18
Practical Wyner-Ziv Encoder and Decoder
Wyner-Ziv Decoder
QuantizerSlepian-
Wolf Encoder
Wyner-Ziv Encoder
Slepian-
WolfDecode
r
Minimum Distortion
Reconstruction
Y Y
X 'XQ Q
B. Girod: Towards Practical Distributed Coding 19
Non-Connected Quantization Regions
Example: Non-connected intervals for scalar quantization
Decoder: Minimum mean-squared error reconstruction with side information
x
1q 2q 3q
x
| |X Yf x y 2
ˆ
conditional centroid
ˆ ˆ | ,arg min x
x E X x y q
B. Girod: Towards Practical Distributed Coding 20
Quantizer Reconstruction Function
Side information y
Reconstructionx̂
ˆ 1x
ˆ 2x
ˆ 3x
ˆ 1,x y
ˆ 2,x y
ˆ 3,x y
B. Girod: Towards Practical Distributed Coding 21
Finding Quantization Regions
1 2 3 4 1 2q =
1
2
3 4
x
Lagrangian cost 2
| ˆ, , , |Y Xj x q E x x q Y r q Y X x
,j x q
B. Girod: Towards Practical Distributed Coding 22
Lloyd Algorithm for Wyner-Ziv Quantizers
EndY
Find best reconstruction functions for current quantizers ),(ˆ )( yqx k
(1) ( )q xChoose initial quantizers 1k
Update rate measure for current quantizers ),()( yqr k
Lagrangian cost for current quantizers, reconstructor and rate measure
)(kJ
NFind best quantizers for current reconstruction and rate measure
( 1) ( )kq x
Convergence1kk
[Fleming, Zhao, Effros, unpublished]
[Rebollo-Monedero, Girod, DCC 2003]
B. Girod: Towards Practical Distributed Coding 23
Which Rate Measure?
X
Wyner-Ziv Decoder
QuantizerSlepian-
Wolf Encoder
Wyner-Ziv Encoder
Slepian-
WolfDecode
r
Minimum Distortion
Reconstruction
Y Y
'XQ
Conditional entropy coder H(Q|Y)
2 |
2 |
, | log |
hence, use , log |Q Y
Q Y
E r Q Y H Q Y E p Q Y
r Q Y p Q Y
[Rebollo-Monedero, Girod, DCC 2003]
B. Girod: Towards Practical Distributed Coding 24
Example
Data set: Video sequence Carphone, 100 luminance frames, QCIF
X: pixel values in even frames
Y: motion-compensated interpolation from two adjacent odd frames
0 50 100 150 200 2500
0.2
0.4
0.6
0.8
1
x
p X|Y(x|y)
0 50 100 150 200 2500
0.005
0.01
0.015
0.02
0.025
0.03
x
p X(x)
B. Girod: Towards Practical Distributed Coding 25
Example (cont.)
Quantizer w/ rate constraint H(Q) Quantizer w/ rate constraint H(Q|Y)
PSNR=39 dBH(Q)=3.05 bit H(Q|Y)=0.54 bit
PSNR=37.4 dBH(Q)=1.87 bit H(Q|Y)=0.54 bit
B. Girod: Towards Practical Distributed Coding 26
Example (cont.)
0 1 2 3 4 5 632
34
36
38
40
42
44
46
48
50
Rate [bit]
PS
NR
[dB
]
DistributedUniformConditionalDistributed at R=H(Q|Y)
0 0.5 1 1.5 2 2.532
34
36
38
40
42
44
46
48
50
Rate [bit]
PS
NR
[dB
]DistributedUniformConditional
Quantizer w/ rate constraint H(Q) Quantizer w/ rate constraint H(Q|Y)
Rate [bit] Rate [bit]
PS
NR
[dB
]
PS
NR
[dB
]
B. Girod: Towards Practical Distributed Coding 27
Wyner-Ziv Quantizers: Lessons Learnt
Typically no quantizer index reuse for rate constraint H(Q|Y) and high rates: Slepian-Wolf code provides more efficient many-to-one mapping in very high dimensional space.
Uniform quantizers close to minimum m.s.e., when combined with efficient Slepian-Wolf code
Quantizer index reuse required for rate constraint H(Q) and for fixed-length coding
Important to decouple dimension of quantizer (i.e. scalar) and Slepian-Wolf code (very large)
B. Girod: Towards Practical Distributed Coding 28
Sensor Networks
Central Unit Local Sensor
Remote Sensor
Side Information
Remote Sensor
Remote Sensor
Remote Sensor [Pradhan, Ramchandran, DCC 2000]
[Kusuma, Doherty, Ramchandran, ICIP 2001][Pradhan, Kusuma, Ramchandran, SP Mag., 2002][Chou, Perovic, Ramchandran, Asilomar 2002]
B. Girod: Towards Practical Distributed Coding 29
Video Compression with Simple Encoder
Reconstruction X’
Y
Interframe Decoder
Scalar Quantizer
Turbo Encoder Buffer
Video frameX
Intraframe Encoder
Turbo Decode
rRequest bits
Slepian-Wolf Codec
InterpolationKey frames
previous
next[Aaron, Zhang, Girod, Asilomar 2002][Aaron, Rane, Zhang, Girod, DCC 2003]
B. Girod: Towards Practical Distributed Coding 30
Decoder Side information After Wyner-Ziv Decoding
Video Compression with Simple Encoder
16-level quantization (~1 bpp)
B. Girod: Towards Practical Distributed Coding 31
Decoder Side information After Wyner-Ziv Decoding16-level quantization (~1 bpp)
Video Compression with Simple Encoder
B. Girod: Towards Practical Distributed Coding 32
Video Compression with Simple Encoder
Decoder Side information After Wyner-Ziv Decoding16-level quantization (~1 bpp)
B. Girod: Towards Practical Distributed Coding 33
Performance of Simple Wyner-Ziv Video Coder
7 dB
B. Girod: Towards Practical Distributed Coding 34
Digitally Enhanced Analog Transmission
Forward error protection of the signal waveform Information-theoretic bounds [Shamai, Verdu, Zamir,1998] “Systematic lossy source-channel coding”
AnalogChannel
Wyner-Ziv
Encoder
DigitalChannel
Wyner-Ziv
Decoder
Sideinfo
B. Girod: Towards Practical Distributed Coding 35
Forward Error Protection for MPEG Video Broadcasting
MPEG Encoder
MPEG Decoder with Error
Concealment
Err
or-P
rone
cha
nnel
S S’
Wyner-Ziv Decoder A S*
Wyner-Ziv Encoder A
Wyner-Ziv Decoder B S**Wyner-Ziv
Encoder B
Graceful degradation without a layered signal representation
B. Girod: Towards Practical Distributed Coding 36
Error-resilient Video Transmissionwith Embedded Wyner-Ziv Codec
Carphone CIF, 50 frames @ 30fps, 1 Mbps,1% Random Macroblock loss
[Aaron, Rane, Rebollo-Monedero, Girod, ICIP 2003]
B. Girod: Towards Practical Distributed Coding 37
Towards Practical Distributed Coding:Why Should We Care?
Chance to reinvent compression from scratch– Entropy coding– Quantization– Signal transforms– Adaptive coding– Rate control– . . .
Enables new compression applications– Sensor networks– Very low complexity encoders– Error-resilient transmission of signal waveforms– Digitally enhanced analog transmission– Unequal error protection without layered coding– . . .
The EndThe End