Belief Propagation and LDPC Code Design (A Review)alex/lectures/lecture14.pdfBelief Propagation and LDPC Code Design (A Review) Alex Balatsoukas-Stimming Technical University of Crete

Belief Propagation and LDPC Code Design (A Review)

Alex Balatsoukas-Stimming

Technical University of Crete

January 27, 2011

Alex Balatsoukas-Stimming (TUC) Belief Propagation and LDPC Code Design January 27, 2011 1 / 30

Outline

1 Belief PropagationSum-Product AlgorithmBECBI-AWGNCMATLAB Implementation

2 Density EvolutionBECBI-AWGNCCode Optimization

3 EXIT ChartsBECBI-AWGNCCode Optimization


Sum-Product Algorithm

Message passing algorithm.

Used for efficient marginalization of factorizable functions which canbe represented as factor graphs.

Decoding of LDPC Codes can be expressed as such a function.

Optimal on graphs with no cycles, generally suboptimal.

Polynomial time complexity with respect to code length.



Consider the function

f (x1, x2, x3, x4, x5, x6) = f1(x1, x2, x3)f2(x1, x4, x6)f3(x4)f4(x4, x5)

For the marginalization wrt x1, we can write

f (x1) =∑∼x1

f (x1, . . . , x6)

=∑∼x1

f1(x1, x2, x3)f2(x1, x4, x6)f3(x4)f4(x4, x5)

=

[∑x2,x3

f1(x1, x2, x3)

][ ∑x4,x5,x6

f2(x1, x4, x6)f3(x4)f4(x4, x5)

],

simplifying the summation.

The above can be applied recursively.



Bit-wise MAP decoding of LDPC codes can be written as follows

xMAPi (y) = arg max

xi∈{±1}p(xi |y)

= arg maxxi∈{±1}

∑∼xi

p(x|y)

= arg maxxi∈{±1}

∑∼xi

p(y|x)p(x)

= arg maxxi∈{±1}

∑∼xi

n∏j=1

p(yi |xi )[x ∈ C],

where

[x ∈ C] =

{ 1|C| , x ∈ C,0, otherwise.


Sum-Product Algorithm - Factor Graphs

Figure: Factor graph of an LDPC code.


Sum-Product Algorithm - Processing Rules

Figure: Processing rules for variable (left) and check (right) nodes.


Belief Propagation - BEC

The Sum-Product algorithm is usually called Belief Propagation whenthe messages represent beliefs.

For the BEC, BP boils down to the following very simple rules.

At variable nodes, if all of the messages used for the computation ofthe outgoing message are erasures, then the outgoing message is anerasure.At check nodes, if at least one of the messages used for thecomputation of the outgoing message is an erasure, then the outgoingmessage is an erasure.


Belief Propagation - General Binary Channels

For other binary channels, things are a bit more complicated.

Log-domain decoder is used due to simplification of rules andnumerical stability of the log function.

LLRs are used as messages, i.e. each variable node calculates

l = log

(p(yi |xi = +1)

p(yi |xi = −1)

)If l > 0, then xi = +1, if l < 0 then xi = −1.

l 6= 0 with probability 1, so we don’t care about the equality.

In practical scenarios, 0 can be assigned to any value of xi or thedecision can be randomized with no perceivable effect on the results.


Belief Propagation - General Binary Channels

At variable nodes, due to the log function, the outgoing message iscalculated as the sum of the corresponding incoming messages.

At check nodes, the outgoing message is calculated according to thefollowing equation

mcvji = 2 tanh−1

∏k 6=i

tanh(mvckj/2)

,

where mcvji represents the message sent from check node j to variablenode i and mvcij represents the message sent from variable node i tocheck node j .


Belief Propagation - MATLAB Implementation

Note: the following implementation of belief propagation for the AWGNchannel aims at readability and intuitiveness, not at speed.

Data structures used:

m × n matrix H.

m × n matrix variable to check.

m × n matrix check to variable.

1× n matrix for initial LLRs.

1× n matrix c hat for the intermediate hard decisions.


Belief Propagation - MATLAB Implementation

Algorithm:

1 Initialize 1× n matrix for LLRs based on channel observations.2 For each variable node (i.e. for i = 1 : n) calculate the outgoing

message to check node j based on the values incheck to variable(:, i), where j are the indices of all non-zero entriesin H(:, i) and store it in variable to check(j , i).

3 For each variable node also calculate the overall LLR and make a harddecision for each bit based on the sign of the LLR.

4 Calculate c hat·HT . If it is equal to an all-zero 1×m vector,decoding is successful. Else, continue.

5 For each check node (i.e. for i = 1 : m) calculate the outgoingmessage to variable node j based on the values invariable to check(i , :), where j are the indices of all non-zero entriesin H(i , :) and store it in check to variable(i , j).

6 If maximum number of iterations has not been reached, go to step 2,else declare failure.


Density Evolution

In the limit of infinite blocklength and assuming that the all-zerocodeword is transmitted, the performance of an ensemble of LDPCcodes can be predicted by Density Evolution.

Density Evolution tracks the evolution of message pdfs throughoutthe decoding procedure.

For the BEC, a single parameter suffices to track this density, i.e. theaverage bit erasure probability.

For the BI-AWGNC, whole densities have to be tracked as thereceived LLR messages have a continuous pdf.


Density Evolution - BEC

Recall that the degree distribution of a code from an edge perspectiveis denoted as follows:

λ(x) =∑i

λixi−1 and ρ(x) =

∑i

ρixi−1

Also recall that the degree distribution from a node perspectiveassociated with λ(x) is denoted as L(x).

For the BEC(ε), Density Evolution takes on the following simple form:

x` = ελ(1− ρ(1− x`−1)),

Pb` = εL(1− ρ(1− x`−1)),

where x0 = ε and Pb` is the bit erasure probability at iteration `.


Density Evolution - BEC

In order for the probability of erasure to converge to zero, thefollowing has to hold for each `

x` > x`+1 = ελ(1− ρ(1− x`))

Figure: Density evolution for a (3, 6) regular LDPC code.


Density Evolution - BI-AWGNC

DE for the BI-AWGNC tracks the pdf of LLR messages transmittedacross the graph edges throughout the decoding procedure.

At a degree i variable node, the output density is the (i − 1)-foldconvolution of the input density, which is then convoluted with thechannel LLR density.

As a degree i check node, the output density is the (i − 1)-foldconvolution of the input density after a transformation into the socalled G -domain in which convolution is defined in a slightly differentway.

Computationally demanding, not suitable for optimization procedures.

Gaussian Approximation (GA) assumes that all distributions aresymmetric Gaussian (i.e. variance σ2 and mean σ2/2).

Only one parameter needs to be tracked, the mean.



At a variable node of degree i , the mean of the outgoing message µviis (i − 1) times the mean of the incoming messages plus the mean ofthe channel message.

Due to the central limit theorem, the Gaussian assumption is a goodapproximation for the variable node messages.

For the check nodes, we first need to define the function φ(x)

φ(x) =

{1− 1√

4πx

∫ +∞−∞ tanh

(u2

)exp

{− (u−x)2

4x

}du, x > 0

1, x = 0.

φ(x) can either be approximated by another function or precalculated.



At a check node of degree i , the mean of the outgoing message canbe calculated as follows.

µ(`)u,i = φ−1

1−

[1−

∑i

λiφ(µ(`−1)vi

)]j−1Averaging over ρ(x), we will have the overall density µ

(`)u .

In order for decoding to converge to a vanishingly small probability oferror, the following has to hold for all `

µ(`)u > µ

(`−1)u .


Code Optimization

The usual approach is to maximize the rate for given noise variance(or erasure probability), while ensuring a small probability of error.The rate of a code can be expressed as

r = 1−∑

i ρi/i∑i λi/i

.

By fixing ρ(x), the objective function becomes linear, i.e.maximization of

∑i λi/i .

The optimization problems resulting from Density Evolution are notlinear, so they can not be solved optimally with reasonable complexity.

An efficient genetic algorithm, called Differential Evolution, iscommonly used.


Code Optimization

For the BEC, the DE convergence condition can be made slightlymore strict as follows

ελ(1− ρ(1− x)) < x , ∀x ∈ (0, 1).

In fact, it suffices if the above holds in (0, ε].

Now, for fixed ρ(x), the problem is linear and can be solved efficiently.

With the same approach, DE with GA can be rewritten as a linearprogram.


Code Optimization (Example)

Using the linear programming approach for ε = 0.5 and vmax = 15, wewere able to find a code with

λ(x) = 0.4001x + 0.1441x2 + 0.1703x3 + 0.0734x7 + 0.2122x8

and ρ(x) = x5, which has rate r = 0.4846.

Figure: Density Evolution for the code found above.


EXIT Charts - BEC

If we define νε(x) = ελ(x) and c(x) = 1− ρ(1− x), then the stricterDE recursion can be written as

νε(c(x)) < x , ∀x ∈ (0, 1).

Since νε(x) is a polynomial with positive coefficients, it has an inverseand we can write

νε(x)−1 > c(x), ∀x ∈ (0, 1).


EXIT Charts - BEC

Figure: EXIT chart for a (3, 6) regular code over a BEC with ε = 0.43.


EXIT Charts - BI-AWGNC

For the BI-AWGN channel, mutual information between the all-zerocodeword X and the LLR corresponding to the Gaussian RVY = X + V , V ∼ N (0, σ2) is usually tracked.

First, let us define the function J(σ) as follows

J(σ) = 1− 1√2πσ

∫ +∞

−∞e−

(x−σ2/2)2

2σ2 log2(1 + e−x)dx .

J(σ) can either be approximated by another function or precalculated.



Using J(σ), the EXIT chart of an irregular code IEC describing thevariable node function can be computed as follows.

I iEV (IAV , σ2ch) = J

(√(i − 1)J−1(IAV )2 + σ2ch

)IEV (IAV , σ

2ch) =

∑i

λi IiEV (IAV , σ

2ch),

where i is the variable node degree, IAV is the mutual information ofthe message entering the variable node with the transmittedcodeword X , I iEV (IAV , σ

2ch) is called an elementary EXIT chart,

IEV (IAV , σ2ch) is the overall EXIT chart.

For σ2ch we have:

σ2ch =4

σ2w,

where σ2w is the variance of the additive white Gaussian noise.



Using J(σ), the EXIT chart of an irregular code IEC describing thecheck node function can be very well approximated as follows.

I iEC (IAC ) ≈ 1− J(√

(i − 1)J−1(1− IAC ))

IEC (IAC ) ≈∑i

ρi IiEC (IAC ),

where i is the check node degree, IAC is the mutual information ofthe message entering the check node with the transmitted codewordX , I iEC (IAC ) is called an elementary EXIT chart, and IEC (IAC ) is theoverall EXIT chart.


Code Optimization

In order for the decoding to converge to a vanishingly smallprobability of error, the EXIT chart of the variable nodes has to lieabove the inverse of the EXIT chart for the check nodes.

As with DE, the target usually is to maximize the rate for a givennoise variance or channel erasure probability.

The usual approach again is to fix ρ(x) and optimize only λ(x).

This optimization is optimistic for the BI-AWGNC because thevariable (check) EXIT chart curve with a GA is slightly higher (lower)than the actual curve, so the resulting codes may have a slightly lowerthreshold than the one predicted.



We set the maximum variable degree to 100 and the channel noisevariance to σ2 = 0.97869 which is exactly the Shannon limit forrate-1/2 binary codes.

Using an EXIT chart technique we were able to find the followingdegree distributions

λ(x) = 0.1350x + 0.2816x2 + 0.2576x8 + 0.0867x33

+ 0.1204x34 + 0.0447x91 + 0.0740x92

ρ(x) = x10

with associated rate r = 0.49303.

By increasing vmax, we can improve performance. For example, forvmax = 200 we found a code with rate r = 0.49670.



Figure: EXIT chart of the code with vmax = 100.


Bibliography

[1] Tom Richardson and Rudiger Urbanke, “Modern Coding Theory,”Cambridge University Press, 2008

[2] S. Y. Chung, T. J. Richardson, and R. L. Urbanke, “Analysis ofSum-Product Decoding of Low-Density Parity-Check Codes Using aGaussian Approximation””, IEEE Trans. Inf. Theory, Vol. 47, No. 2,pp. 657-670, Feb. 2001

[3] M. Tuchler and J. Hagenauer, Exit charts of irregular codes, inProc. CISS, pp. 748753, 2002.


Documents

Belief Propagation and LDPC Code Design (A Review)alex/lectures/lecture14.pdfBelief Propagation and LDPC Code Design (A Review) Alex Balatsoukas-Stimming Technical University of Crete