Error correction on a tree: Instanton approach

Misha Chertkov (LANL)

In collaboration with: V. Chernyak (Corning) M. Stepanov (UA, Tucson) B. Vasic (UA, Tucson)

Thanks:I. Gabitov (Tucson/LANL)

Boulder: 04/15/04

Submitted to Phys.Rev.Lett.

•Forward-Error-Correction (FEC). Channel Noise. •Coding. Low Density Parity Check codes (LDPC) - Tanner graph •Decoding. Marginal-A-Posteriori (MAP) – Stat Mech interpretation Belief Propagation (BP) – Message Passing (MP) •Post-Error-Correction Bit-Error-Rate (BER). Optimization. Shannon transition/limit . Error floor - Evaluation.

•Tree as an approximation: BP is exact . From LDPCC to a tree •BER in the center of the tree • High Signal-to-Noise Ratio (SNR) phase. Hamming distance. •Symmetry. * Broken Symmetry. *•Instantons/phases on the tree.

Introduction:

What is next? *

Our objectives

L

L

N

N

Forward-Error-Correction

Coding

Decoding

N > L R=L/N - code rate

)|()|( )(

1

)()()( ini

N

i

outi

inout xxpxxP

22exp)|( 2 syx

syxp

channelwhite

Gaussian symmetricexam

ple

menu

Nxxx ,,1

)()()( inoutin xxxnoise

Low Density Parity Check Codes

menu

N=10variable nodes

M=N-L=5 checking nodes

Parity check matrix

0

0

0

0

0

10

9

8

7

6

5

4

3

2

1

x

x

x

x

x

x

x

x

x

x

H mod 2

Tanner graph

M

ii

ii x

1

1,

112

“spin” variables -

- set of constraintsM

Ni

,,1

,,1

(linear coding)

Decoding (optimal)

N

kkk

M

ii hhFhZ

11}{

exp1,)(exp)(

1sh“magnetic” field(external/noise)

)|( )()( inout xxP

constraints

“free energy”

“statistical sum”

menu

(symbol to symbol) Maximum-A-Posteriori (MAP) decoding

)()( hmsignhoutput j

Efficient but Expensive:requires operationsL2

hhFhm

)()(

“magnetization”

Stat Mech interpretation was suggested byN. Sourlas (Nature ‘89)

To notice – spin glass (replica) approach for random codes:e.g. Rujan ’93, Kanter, Saad ’99; Montanari, Sourlas ’00; Montanari ’01; Franz, Leone, Montanari, Ricci-Tersenghi ‘02

Sub-optimal but efficient decoding

i

jii

j

jj

i

jii

j

jj

hm

h

tanhtanhtanh

tanhtanh

1

1 Belief Propagation (BP) Gallager’63;Pearl ’88;MacKay ‘99

=solving Eqs. on the graph

it

i

i

ji

ti

j

jt

j

h

h

)(

)(1)1( tanhtanh

Iterative solution of BP= Message Passing (MP)

Q*m*N steps instead of Q - number of MP iterations

m - number of checking nodes contributing a variable node

L2

What about efficiency? Why BP is a good replacement for MAP?

* (no loops!)

menu

Post-Error-Correction Bit Error Rate (BER)

2/1

20

1

)2(2

)(exp)( N

N

j j

ii ss

shhmdhdB

measure for unsuccessful decoding

Probability of making an error in the bit “i”

{+1} is chosen for the initial code-word

probability density for givenmagnetic field/noise realization

Foreword-error-correction scheme/optimizationForeword-error-correction scheme/optimization

1. describe the channel/noise --- External2. suggest coding scheme3. suggest decoding scheme4. measure BER/FER5. If BER/FER is not satisfactory (small enough) goto 2

menu

From R. Urbanke, “Iterative coding systems”

menu

SNR, s

BE

R, B

Shannon transition/limit

Error floor

Error floor prediction for some regular (3,6) LDPC Codes using a 5-bit decoder. From T. Richardson “Error floor for LDPC codes”, 2003 Allerton conference Proccedings.

menu

No-go zone for brute-force Monte-Carlo numerics.

Estimating very low BER is the major bottleneck of the coding theory

Our objective:

For given (a) channel (b) coder (c) decoderto estimate BER/FER by means ofanalytical and/or semi-analytical methods.

Hint:

BER is small and it is mainly formed at some very special“bad” configurations of the noise/”magnetic field”

Instanton/saddle-point approach is the right way to identifythe “bad” configurations and thus to estimate BER!

menu

menu

Tree -- no loops -- approximation

}{}{

1

}{

}{}{

1

}{

11}{

exp1,)(

exp1,)(

exp1,)(exp)(

kkk

iij

kkk

iij

N

kkk

M

ii

hhY

hhX

hhFhZ

j

j

i

jii

j

jj h tanhtanh 1

2/)/ln(

)()(2

1)exp(

)()(2

1)exp(

jjj

i

ji

i

jiiiii

j

jj

i

ji

i

jiiiii

j

jj

XY

YXYXhY

YXYXhX

MAP

BP

Belief Propagation is optimal (i.e. equivalentto Maximum-A-Posteriori decoding) on a tree (no loops)

Analogy: Bethe lattice (1937)

Gallager ’63; Pearl ’88; MacKay ’99Vicente, Saad, Kabashima ’00; Yedidia, Freeman, Weiss ‘01

From a finite-size LDPCC to a tree:

1) Fix the variable node where BER needs to be calculated2) Choose shortest loop on the graph coming through the “0”th

node. Length of the loop is (n+1). 3) Count n-generations from the tree center and cut the rest.

Regular graph/tree is characterized by:m - number of checking nodes connected to a variable nodek - number of variable nodes connected to a checking noden - number of generations on the tree

m=2,k=3,n=4menu

BER in the center of the tree

0

tanh2

1tanh

2

1

)(exp

2

1

2

0 0

1

0

1 00

Q

ss

ss

Q

QddB

j

j

k

jkkj

k

kk

jj

Tree is directed thus integrating over the ``magnetic fields” one gets a path-integral over new fields, , defined on the variable nodes.

j

menu

Remarks: 1) Optimal configuration/instanton depends on SNR, s; 2) There are may be many competing instantons; 3) Looking for instantons pay attention to the symmetry

2/1

20

1

)2(2

)(exp)( N

N

j j

ii ss

shhmdhdB

i

jii

j

jj

i

jii

j

jj

hm

h

tanhtanhtanh

tanhtanh

1

1

Instanton equations!

Effective action

High Signal-to-Noise-Ratio (SNR) phase

menu

Original code word = “+1” on the entire tree

The next “closest” code word = “-1” on the colored branches, = “+1” on the remaining variable nodes

Hamming distance between the two code words= number of the colored variable nodes

2exp0

HsB

at s>>1 That is also given by an instanton:

0 node is colored

node is not colored

Analogy with a low-temperature phase in stat-mech:High SNR value of effective action ~ self energy

Low SNR -- symmetric -- phase

menu

Symmetric phase: at any node on the tree depends

primarily on the generation (counted from the center)

11

2112

1

1

0

10

tanhtanh)1(),(

2

),(),())1)(1((

2

)1(

lm

nmjmj

n

j

jn

mssg

sgsglm

lmQ

);,('),(),()1(

),('),(),()1)(1(

1121

11

sgsgsglm

sgsgsglm

nmnmnmn

jmjmjjmj

01

for j=0,…,n-2instantonequations

“zero momentum” configuration/approximation

s

sg jmj

0

1 ),(

guarantees estimation

from above for eff. action

),('1

),(

ccm

ccmc

sg

sg

Shannon’s

transition

0

21

;0 2

)),((Q

sgQ nm

zm

c

c

ss

ss

finite

infinite

“0” “2”“1”

“4”“3”

In general:There are many (!!!) broken symmetryinstanton solutions

Remark:Broken symmetry instantons may be related

to the “near codewords” suggested by Richardson ‘03 in the context of the error-floor phenomenon explanation menu

Broken symmetry

High SNR(low temp)

Low SNR(high temp)

m=4, l=5, n=3. Curves of different colors correspond to

the instantons/phases of different symmetries.

Instanton phases on the tree

menutruth …

sg

msg

sgssk

sss

ms

lm

ccmcck

n

,1

tanhtanh)1(),(

,:11

121

transitions

Full numerical optimization (no symmetry breaking was assumed !!!)

Area of a circle surrounding any variable node is proportional

to the value of the noise on the node.

m=2 l=3 n=3

menu

What is next?

We plan to develop and extend this instanton approach to:

•Regular codes with loops. This task will require developing a perturbation theory with respect to the inverse length of the closed loop and/or with respect to the small density of closed loops. •Other types of codes, e.g. convolutional, turbo, etc.•Calculation of the Frame Error Rate (FER), thereby measuring the probability of making an error in a code word.•Finite-number of iteration in message-passing version of the BP algorithm. The particular interest here lies in testing how BER in general and the error floor phenomena in particular depend on the number of iterations. •Other types of fast but, probably, less efficient decoding schemes.•Other types of uncorrelated channels (noise), e.g. binary eraser channel.•Correlated channels, with both positive and negative types of correlations between neighboring slots. This is particularly relevant for linear and nonlinear (soliton) transmission in fiber optics communications.•Accounting for Gaussian fluctuations (i.e. second order effects) around the instantons.

menu

Truth …

menu

mainslide