Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Outline
Searching for low weight pseudo-codewords
Michael Chertkov1 & Mikhail Stepanov 2,1
1Theory Division, LANL and 2University of Arizona, Tucson
Jan 30, 2007, ITA workshop, UCSD
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
Outline
Outline
1 IntroductionLDPCBP vs LP & Bethe Free EnergyError-floor
2 Instanton-amoeba (general)
3 Pseudo-Codeword-Search (LP)
4 Dendro-LDPC
5 Simulations: FER vs SNR & pseudo-codeword spectrumTanner codeMargulis p=7 code
6 Conclusions
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Error Correction
Scheme:
Example of Additive White Gaussian Channel:
P(xout |xin) =Y
i=bits
p(xout;i |xin;i )
p(x|y) ∼ exp(−s2(x − y)2/2)
Channelis noisy ”black box” with only statistical information available
Encoding:use redundancy to redistribute damaging effect of the noise
Decoding:reconstruct most probable codeword by noisy (polluted) channel
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Low Density Parity Check Codes
N bits, M checks, L = N − M information bitsexample: N = 10, M = 5, L = 5
2L codewords of 2N possible patterns
Parity check: Hv = c = 0example:
H =
0BBB@
1 1 1 1 0 1 1 0 0 00 0 1 1 1 1 1 1 0 00 1 0 1 0 1 0 1 1 11 0 1 0 1 0 0 1 1 11 1 0 0 1 0 1 0 1 1
1CCCA
LDPC = graph (parity check matrix) is sparse
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Decoding=Statistical InferenceMaximum Likelihood Maximum-a-Posteriori
ML=arg maxσ=codeword
P(xout|σ) MAPi =sign
(∑σ σiP(xout|σ)∑σ P(xout|σ)
)
MAP≈BP=Belief-Propagation (Bethe-Pieirls) Gallager ’61
Exact on a tree
Trading optimality for reduction in complexity: ∼ 2L →∼ L
BP = solving equations on the graph:
ηjα = hj +j∈β∑β 6=α
tanh−1
(i∈β∏i 6=j
tanh ηiβ
)Message Passing = iterative BP
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Decoding=Statistical InferenceMaximum Likelihood Maximum-a-Posteriori
ML=arg maxσ=codeword
P(xout|σ) MAPi =sign
(∑σ σiP(xout|σ)∑σ P(xout|σ)
)
MAP≈BP=Belief-Propagation (Bethe-Pieirls) Gallager ’61
Exact on a tree
Trading optimality for reduction in complexity: ∼ 2L →∼ L
BP = solving equations on the graph:
ηjα = hj +j∈β∑β 6=α
tanh−1
(i∈β∏i 6=j
tanh ηiβ
)Message Passing = iterative BP
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Bethe free energy: variational approach(Yedidia,Freeman,Weiss ’01 - inspired by Bethe ’35, Peierls ’36)
F = −Pi
hiPσi
σibi (σi ) +Pα
Pσα
bα(σα) ln bα(σα)−Pi(qi − 1)
Pσi
bi (σi ) ln bi (σi )
constraints:
∀ i , α : 0 ≤ bi (σi ), bα(σα) ≤ 1
∀ α :X
σα
bα(σα) = 1
∀ i ; α ∈ i : bi (σi ) =X
σα\σi
bα(σα)
Belief-Propagation Equations:
δFδb
���constr.
= 0
• Relaxation to minimum of the Bethe Free energy enforces convergence of iterative BP (Stepanov, Chertkov ’06)
LP decoding Feldman, Wainwright, Karger ’03
LP decoding = minimization of a linear function over a bounded domaindescribed by linear constraints
”Large SNR” limit of BP: F ≈ E = −Pi
hiPσi
σibi (σi )
“small” polytope = get rid of bα
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Bethe free energy: variational approach(Yedidia,Freeman,Weiss ’01 - inspired by Bethe ’35, Peierls ’36)
F = −Pi
hiPσi
σibi (σi ) +Pα
Pσα
bα(σα) ln bα(σα)−Pi(qi − 1)
Pσi
bi (σi ) ln bi (σi )
constraints:
∀ i , α : 0 ≤ bi (σi ), bα(σα) ≤ 1
∀ α :X
σα
bα(σα) = 1
∀ i ; α ∈ i : bi (σi ) =X
σα\σi
bα(σα)
Belief-Propagation Equations:
δFδb
���constr.
= 0
• Relaxation to minimum of the Bethe Free energy enforces convergence of iterative BP (Stepanov, Chertkov ’06)
LP decoding Feldman, Wainwright, Karger ’03
LP decoding = minimization of a linear function over a bounded domaindescribed by linear constraints
”Large SNR” limit of BP: F ≈ E = −Pi
hiPσi
σibi (σi )
“small” polytope = get rid of bα
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LDPCBP vs LP & Bethe Free EnergyError-floor
Error-Floor
T. Richardson, Allerton ’03
BER vs SNR = measure ofperformance
Waterfall ↔ Error-floor
Suboptimal decoding causeserror-floor: at s2 = Es/N0 →∞,
FERML ∼ exp(−dMLs2/2) vs
FERsub ∼ exp(−dsubs2/2) where
dML ≥ dsub
Monte-Carlo is useless atFER . 10−8
Need an efficient method toanalyze error-floor
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
Pseudo-codewords and Instantons
Error-floor is caused by Pseudo-codewords:
Wiberg ’96; Forney et.al’99; Frey et.al ’01;
Richardson ’03; Vontobel, Koetter ’04-’06
Instanton = optimal conf of the noise
BER =
∫d(noise) WEIGHT (noise)
BER ∼ WEIGHT
(optimal confof the noise
)optimal confof the noise
=Point at the ESclosest to ”0”
Instantons are decoded to Pseudo-Codewords
Instanton-amoeba
= optimization algorithmStepanov, et.al ’04,’05
Stepanov, Chertkov ’06
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
“0”-codeword
Error-Surface
dangerouspseudo-codeword
Weighted Median: xinst = argminx P(0|x)|E(x;σ)=0
xinst =σ
2
∑i σi∑i σ
2i
, d =(∑
i σi )2∑
i σ2i
, FER ∼ exp(−d ·s2/2)
Wiberg ’96; Forney et.al ’99; Vontobel, Koetter ’03,’05
Pseudo-Codeword-Search Algorithm Chertkov, Stepanov ’06
Start: Initiate x(0).
Step 1: x(k) is decoded to σ(k).
Step 2: Find y(k) - weighted median
between σ(k), and ”0”
Step 3: If y(k) = y(k−1), k∗ = k End.Otherwise go to Step 2 with
x(k+1) = y(k) + 0.
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
“0”-codeword
Error-Surface
dangerouspseudo-codeword
Weighted Median: xinst = argminx P(0|x)|E(x;σ)=0
xinst =σ
2
∑i σi∑i σ
2i
, d =(∑
i σi )2∑
i σ2i
, FER ∼ exp(−d ·s2/2)
Wiberg ’96; Forney et.al ’99; Vontobel, Koetter ’03,’05
Pseudo-Codeword-Search Algorithm Chertkov, Stepanov ’06
000
xxx(0)
xxx(1)
xxx(2)
xxx(3)
σσσ(0)
σσσ(1)
σσσ(2,3)
xxx(0) start
step 2
step 3
σσσ(2) = σσσ(3) end
Start: Initiate x(0).
Step 1: x(k) is decoded to σ(k).
Step 2: Find y(k) - weighted median
between σ(k), and ”0”
Step 3: If y(k) = y(k−1), k∗ = k End.Otherwise go to Step 2 with
x(k+1) = y(k) + 0.
Multiple repetitions ⇒ instanton frequency spectra
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LP complexity grows exponentially with check degree
Current solutions:
Adaptive LP (Taghavi, Siegel ’06)
BP-style relaxation of LP (Vontobel, Koetter ’06)
Dendro-trick = Graph Modification (our solution)Chertkov,Stepanov’07
MAP solutions are identical
Set of Pseudo-codewords are identical
Instanton spectra are very alike, ≈
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
LP complexity grows exponentially with check degree
Current solutions:
Adaptive LP (Taghavi, Siegel ’06)
BP-style relaxation of LP (Vontobel, Koetter ’06)
Dendro-trick = Graph Modification (our solution)Chertkov,Stepanov’07
MAP solutions are identical
Set of Pseudo-codewords are identical
Instanton spectra are very alike, ≈
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
Tanner codeMargulis p=7 code
1 2 3 4 5
1
10
10
10
10
10
–2
–4
–6
–8
–10
SNR
Fram
e-E
rror
-Rat
e (F
ER
)
5
5 5 5
7
3
3 3
4 4
1 1
BP, 4 iter
BP, 1024 iter
LP
BP, 8 iter
BP, 128 iter
0
0.2
0.4
0.6
0.8
1
10 15 20 25 30di
stri
butio
n fu
nctio
nd
dML
LP
BP, 8 iter
BP, 4 iter
dmin;inst < dML = 20
Dangerous instantons are frequent
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
Tanner codeMargulis p=7 code
1 2 3 4 5
1
10
10
10
10
10
–2
–4
–6
–8
–10
SNR
Fram
e-E
rror
-Rat
e (F
ER
)
BP, 4 iter
LP
BP, 128 iter
BP, 8 iter
BP, 1024 iter 0
0.2
0.4
0.6
0.8
1
10 15 20 25 30 35 40 45 50di
stri
butio
n fu
nctio
nd
dML
BP, 8 iter
BP, 4 iter
LP
dmin;inst;LP > dML = 16
Dangerous codewords are rare ⇒ emergence of a steeptransient asymptotic of FER vs SNR
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
Conclusions:
BP is faster but LP is easier to analyze
Instanton-amoeba and especially Pseudo-codeword-search areof a practical value for the error-floor domain exploration
Dendro-LDPC is a convenient trick reducing complexity of LP
Future Challenges:
Improving LP/BP (Facet Guessing of Dimakis,Wainwright ’06 and
LP-erasure Chertkov,Chernyak ’06 are good candidates)
Analyzing really long codes
Analyzing error-floor in correlated channels (e.g. 1d and 2d ISI
with and without coding)
Design of LPDC codes (with reduced error-floor)
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
Conclusions:
BP is faster but LP is easier to analyze
Instanton-amoeba and especially Pseudo-codeword-search areof a practical value for the error-floor domain exploration
Dendro-LDPC is a convenient trick reducing complexity of LP
Future Challenges:
Improving LP/BP (Facet Guessing of Dimakis,Wainwright ’06 and
LP-erasure Chertkov,Chernyak ’06 are good candidates)
Analyzing really long codes
Analyzing error-floor in correlated channels (e.g. 1d and 2d ISI
with and without coding)
Design of LPDC codes (with reduced error-floor)
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords
IntroductionInstanton-amoeba (general)
Pseudo-Codeword-Search (LP)Dendro-LDPC
Simulations: FER vs SNR & pseudo-codeword spectrumConclusions
Bibliography:
M. Chertkov, M.G. Stepanov, Searching for low weight pseudo-codewords, 2007 Information Theory andApplication Workshop, proceedings.
M. Chertkov, M.G. Stepanov, Pseudo-codeword Landscape, submitted for proceeding of ISIT 2007, June2007, Nice, cs.IT/0701084.
M. Chertkov, M.G. Stepanov, An Efficient Pseudo-Codeword Search Algorithm for Linear ProgrammingDecoding of LDPC Codes, arXiv:cs.IT/0601113, submitted to IEEE Transactions on Information Theory.
M.G. Stepanov, M. Chertkov, Instanton analysis of Low-Density-Parity-Check codes in the error-floorregime, arXiv:cs.IT/0601070, ISIT 2006 (July 2006, Seattle, WA)
M.G. Stepanov, M. Chertkov, Improving convergence of belief propagation decoding, arXiv:cs.IT/0607112,44th Allerton Conference (September 27-29, 2006, Allerton, IL)
M.G. Stepanov, M. Chertkov, The error-floor of LDPC codes in the Laplacian channel, Proceedings of 43rdAllerton Conference (September 28-30, 2005, Allerton, IL), arXiv:cs.IT/0507031
M.G. Stepanov, V. Chernyak, M. Chertkov, B. Vasic, Diagnosis of weakness in error correction: a physicsapproach to error floor analysis, Phys. Rev. Lett. 95, 228701 (2005) [See alsohttp://www.arxiv.org/cond-mat/0506037 for extended version with Supplements.]
V. Chernyak, M. Chertkov, M. Stepanov, B. Vasic, Error correction on a tree: An instanton approach,Phys. Rev. Lett. 93, 198702-1 (2004).
All papers are available at http://cnls.lanl.gov/∼chertkov/pub.htm
Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords