Polarization and Polar codes - Institute of Network Coding Polarization and Polar codes Emre Telatar EPFL Hong Kong | January, 2013Published in: Foundations and Trends in Communications

Setting Brick Mortar Polar Speed Complexity Sincerely,

Polarization and Polar codes

Emre Telatar

EPFL

Hong Kong — January, 2013


Extremal Channels

Among all channels, there are two classes for which it is easy tocommunicate optimally:

The perfect channels: the output Y determines the input X .

The useless channels: the output Y is independent of theinput X .

Arıkan’s polar coding is a technique to convert any binary-inputchannel to a mixture of binary-input extremal channels.

The technique is information lossless, and of low complexity?.

I am not the inventor of this technique.


Extremal Channels








Extremal Channels








Extremal Channels








Extremal Channels








Extremal Channels








Building block

Given two copies of a binary input channel W : F2 → Y

SetX1 = U1 + U2

X2 = U2

with U1,U2 i.i.d., uniform on F2.

W

WX1

X2

Y1

Y2

This induces two synthetic channels W− : F2 → Y2

andW + : F2 → Y2 × F2.


Building block


SetX1 = U1 + U2

X2 = U2

with U1,U2 i.i.d., uniform on F2.W

WU1

U2

+ Y1

Y2


andW + : F2 → Y2 × F2.


Building block


SetX1 = U1 + U2

X2 = U2


WU1

U2

+ Y1

Y2


andW + : F2 → Y2 × F2.


Building block


SetX1 = U1 + U2

X2 = U2


WU1

U2

+ Y1

Y2

This induces two synthetic channels W− : F2 → Y2 andW + : F2 → Y2 × F2.


Example: Erasure channel

Suppose W is a BEC(p), i.e., Y = X with probabilty 1− p, Y =?otherwise.

W−

W +

We already begin to see some extremalization: W + is betterthan W , while W− is worse.




W− has input U1, output (Y1,Y2) =

W +





W− has input U1, output (Y1,Y2) = (U1 + U2,U2)

W +





W− has input U1, output (Y1,Y2) = (U1 + U2, ? )

W +





W− has input U1, output (Y1,Y2) = ( ? ,U2)

W +





W− has input U1, output (Y1,Y2) = ( ? , ? )

W +





W− is a BEC(2p − p2).

W +






W + has input U2, output (Y1,Y2,U1) =






W + has input U2, output (Y1,Y2,U1) = (U1 + U2,U2,U1)






W + has input U2, output (Y1,Y2,U1) = ( ? ,U2,U1)






W + has input U2, output (Y1,Y2,U1) = (U1 + U2, ? ,U1)






W + has input U2, output (Y1,Y2,U1) = ( ? , ? ,U1)






W + is a BEC(p2).






W + is a BEC(p2).



Building block

Properties of W 7→ (W−,W +):

I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)

W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W )

≥ I (W−).


Building block


I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)

W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W )

≥ I (W−).


Building block


I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)

W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W )

≥ I (W−).


Building block


I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W )

≥ I (W−).


Building block


I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W )

≥ I (W−).


Building block


I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W )

≥ I (W−).


Building block


I (W−) = I (U1; Y1Y2)

I (W +) = I (U2; Y1Y2U1)

I (W−) + I (W +) = I (U1U2; Y1Y2)

= I (X1X2; Y1Y2)W

WU1

U2

+ Y1

Y2

12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W ) ≥ I (W−).


Building block


12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W ) ≥ I (W−).

‘Guaranteed progress’ unlessalready extremal:

|I (W±)− I (W )| < δ implies

I (W ) 6∈ (ε, 1− ε),

with ε(δ)→ 0 as δ → 0.


Building block


12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W ) ≥ I (W−).



I (W ) 6∈ (ε, 1− ε),



Building block


12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W ) ≥ I (W−).



I (W ) 6∈ (ε, 1− ε),


I (W +)− I (W−)

I (W )0 1

12


Building block


12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W ) ≥ I (W−).



I (W ) 6∈ (ε, 1− ε),


I (W +)− I (W−)

I (W )0 1

12


Building block


12 I (W−) + 1

2 I (W +) = I (W ).

I (W +) ≥ I (W ) ≥ I (W−).



I (W ) 6∈ (ε, 1− ε),


I (W +)− I (W−)

I (W )0 1

12


Polarization construction

What we can do once, we can do many times.

Given W : F2 → Y,

Duplicate W and obtain W−

and W +.

Duplicate W− (and W +),

and obtain W−− and W−+

(and W +− and W ++).

Duplicate W−− (and W−+,W +−, W ++) and obtainW−−− and W−−+ (andW−+−, W−++, W +−−,W +−+, W ++−, W +++).

. . .



What we can do once, we can do many times. Given W : F2 → Y,


and W +.





. . .





and W +.





. . .





and W +.





. . .





and W +.





. . .





and W +.





. . .



n levels into this process, wehave transformed N = 2n usesof channel W to one use eachof N channels

W s1...,sn , sj ∈ {+,−},

these are the channels

Ui → Y NU i−1, i = 1, . . . ,N.

The quantities{I (W s1...sn)

}are

exactly the N quantities

I (Ui ; YnU i−1), i = 1, . . . ,N

U1 Y1

Un Yn



n levels into this process, wehave transformed N = 2n usesof channel W to one use eachof N channels

W s1...,sn , sj ∈ {+,−},

these are the channels

Ui → Y NU i−1, i = 1, . . . ,N.

The quantities{I (W s1...sn)

}are

exactly the N quantities

I (Ui ; YnU i−1), i = 1, . . . ,N

U1 Y1

Un Yn


Are the synthetic channels ‘real’?On successive decoding

The quantities I (Ui ; YNU i−1) are

relevant for a genie-aided decoder:

U1 = φ1(Y N)

U2 = φ2(Y N ,U1)

U3 = φ3(Y N ,U2)

. . .

UN = φN(Y N ,UN−1)

vs

Standalone decoder:

U1 = φ1(Y N)

U2 = φ2(Y N , U1)

U3 = φ3(Y N , U2)

. . .

UN = φN(Y N , UN−1).

If the genie-aided decoder makes no errors, then, the standalonedecoder makes no errors: the synthetic channels are real.





U1 = φ1(Y N)

U2 = φ2(Y N ,U1)

U3 = φ3(Y N ,U2)

. . .


vs

Standalone decoder:

U1 = φ1(Y N)

U2 = φ2(Y N , U1)

U3 = φ3(Y N , U2)

. . .







U1 = φ1(Y N)

U2 = φ2(Y N ,U1)

U3 = φ3(Y N ,U2)

. . .


vs

Standalone decoder:

U1 = φ1(Y N)

U2 = φ2(Y N , U1)

U3 = φ3(Y N , U2)

. . .




Polarization

Polarization refers to the phenomenon that in the limit,almost all channels are extremal, i.e.,

1

N#{i : I (Ui ; Y

NU i−1) ∈ (ε, 1− ε)}→ 0

as N = 2n gets large.

If this happens, the good limiting synthetic channels can beused to transmit uncoded data bits — the inputs to the otherchannels can be frozen to fixed values. Since thetransformation is information lossless, The fraction of goodchannels (i.e., data rate) must be I (W ).

The decoder can decode U1, U2, . . . , UN successively.


Polarization


1

N#{i : I (Ui ; Y

NU i−1) ∈ (ε, 1− ε)}→ 0





Polarization


1

N#{i : I (Ui ; Y

NU i−1) ∈ (ε, 1− ε)}→ 0





Polarization happens

Organize the syntheticchannels as a tree.

Pick a random path.

The I (·) sequence weencounter form a boundedmartingale.

This sequence converges foralmost all paths.

Can’t converge to anon-extremal value becauseof the ‘guaranteed progress’property.

W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++




Pick a random path.




W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++




Pick a random path.




W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++




Pick a random path.




W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++




Pick a random path.




W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++



Theorem

For any ε > 0, as N = 2n gets large

1

N#{

(s1, . . . , sn) : I (W s1...sn) ∈ (ε, 1− ε)}→ 0.

Theorem

1

N#{

(s1, . . . , sn) : I (W s1...sn) ∈ (εN , 1− εN)}→ 0.

even when εN = 2−N0.5−


Polarization happens fast enough

Theorem

For any ε > 0, as N = 2n gets large

1

N#{

(s1, . . . , sn) : I (W s1...sn) ∈ (ε, 1− ε)}→ 0.

Theorem

1

N#{

(s1, . . . , sn) : I (W s1...sn) ∈ (εN , 1− εN)}→ 0.

even when εN = 2−N0.5−


Polarization speed

The polarization speed quoted in the previous slide is morethan sufficient to arrest error propagation.

To show such a fantastic speed, introduce

Z (W ) =∑y

√W (y |0)W (y |1)

as a companion to I (W ). Note that this is an upper bound onprobability of error for uncoded transmission over W .


Polarization speed

The polarization speed quoted in the previous slide is morethan sufficient to arrest error propagation.

To show such a fantastic speed, introduce

Z (W ) =∑y

√W (y |0)W (y |1)

as a companion to I (W ). Note that this is an upper bound onprobability of error for uncoded transmission over W .


Polarization speed

Properties of Z (W ):

Z (W ) ∈ [0, 1].

Z (W ) ≈ 0 iff I (W ) ≈ 1.

Z (W ) ≈ 1 iff I (W ) ≈ 0.

Z (W +) = Z (W )2.

Z (W−) ≤ 2Z (W ).

Since Z (W ) upper bounds on probability of error for uncodedtransmission over W , we can choose the good indices on the basisof Z (W ). The sum of the Z ’s of the chosen channels will upperbound the block error probability. This suggests studying thepolarization speed of Z .


Polarization speed


Z (W ) ∈ [0, 1].

Z (W ) ≈ 0 iff I (W ) ≈ 1.

Z (W ) ≈ 1 iff I (W ) ≈ 0.

Z (W +) = Z (W )2.

Z (W−) ≤ 2Z (W ).

Z (W )

I (W )

1

10

0



Polarization speed


Z (W ) ∈ [0, 1].

Z (W ) ≈ 0 iff I (W ) ≈ 1.

Z (W ) ≈ 1 iff I (W ) ≈ 0.

Z (W +) = Z (W )2.

Z (W−) ≤ 2Z (W ).

Z (W )

I (W )

1

10

0



Polarization speed


Z (W ) ∈ [0, 1].

Z (W ) ≈ 0 iff I (W ) ≈ 1.

Z (W ) ≈ 1 iff I (W ) ≈ 0.

Z (W +) = Z (W )2.

Z (W−) ≤ 2Z (W ).

Z (W )

I (W )

1

10

0



Polarization speed


Z (W ) ∈ [0, 1].

Z (W ) ≈ 0 iff I (W ) ≈ 1.

Z (W ) ≈ 1 iff I (W ) ≈ 0.

Z (W +) = Z (W )2.

Z (W−) ≤ 2Z (W ).

Z (W )

I (W )

1

10

0



Polarization speed


Z (W ) ∈ [0, 1].

Z (W ) ≈ 0 iff I (W ) ≈ 1.

Z (W ) ≈ 1 iff I (W ) ≈ 0.

Z (W +) = Z (W )2.

Z (W−) ≤ 2Z (W ).

Z (W )

I (W )

1

10

0



Polarization speed


On a typical path the Zvalues we encouter aresquared half the time.

On a fraction I (W ) of thepaths Z converges to zero.

On a ‘typical’ such path, Zapproaches zero likeexp(−2n/2).

W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++


Polarization speed





W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++


Polarization speed





W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++


Polarization speed





W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++


Polarization speed





W

W−

W +

W−−

W−+

W +−

W ++

W−−−

W−−+

W−+−

W−++

W +−−

W +−+

W ++−

W +++


Polarization speed

After taking care of the ugly details with ε’s and δ’s,

Theorem

For any β < 1/2,

limn→∞

1

N#{

(s1, . . . , sn) : Z(W s1...sn

)< exp(−Nβ)

}= I (W )


Decoding Complexity

The ‘butterfly’ structure of the encoder ensures O(N log N)encoding complexity.

For the decoding we need to compute

LR(s1, . . . , sn) :=W s1...sn(·|0)

W s1...sn(·|1)

But

LR(s1, . . . , sn−1,+) = LR1(s1, . . . , sn−1) · LR2(s1, . . . , sn−1)1−2ui

LR(s1, . . . , sn−1,−) =LR1(s1, . . . , sn−1) + LR2(s1, . . . , sn−1)

1 + LR1(s1, . . . , sn−1)LR2(s1, . . . , sn−1)

So, the computation of the LR’s is amenable to divide andconquer, and can also be done in N log N effort.


Decoding Complexity



LR(s1, . . . , sn) :=W s1...sn(·|0)

W s1...sn(·|1)

But



1 + LR1(s1, . . . , sn−1)LR2(s1, . . . , sn−1)



Decoding Complexity



LR(s1, . . . , sn) :=W s1...sn(·|0)

W s1...sn(·|1)

But



1 + LR1(s1, . . . , sn−1)LR2(s1, . . . , sn−1)



So far

Polar codes are I (W ) achieving,

encoding complexity is N log N,

with successive decoding, the decoding complexity is N log N,

probability of error decays like 2−√

N .

Moreover, one can identify almost all the ‘good’ channels withN log2+ N computational effort.

. . .


So far





N .


. . .


So far





N .


. . .


So far





N .


. . .


So far





N .


. . .


So far





N .


. . .


So far





N .


. . .


Moreover

For symmetric channels the construction is deterministic.There is no “choose from this ensemble and verify” step.

The error probability guarantees are not based on simulations— interesting for very low error probability applications.

Generalizes to channels with arbitrary discrete inputalphabets: a similar ‘two-by-two’ construction, with samecomplexity and error probability bounds. This allows one toachieve true capacity C (W ) rather than I (W ).

With ‘k-by-k ’ constructions the error probability can be madeto decay almost exponentially in blocklength.


Moreover






Moreover






Moreover






Moreover






Extensions

Dual constructions yield vector quantizers that achieve therate distortion bound.

Yields codes that achieve the secrecy capacity of the Wyner’swiretap channel.

Polarizing each user of a MAC yield sum rate bound achievingcodes.

. . . .


Extensions




. . . .


Extensions




. . . .


Extensions




. . . .


Extensions




. . . .


Remarks

Polar codes are easy to teach.

They give a concrete way to do many things that weotherwise show via indirect arguments.

They may prove practical in certain setups. (E.g., multi-levelmodulation, high reliability applications, rate distortion, ...).

Many basic questions still to answer.


Remarks






Remarks






Remarks






Remarks





Documents

Polarization and Polar codes - Institute of Network Coding Polarization and Polar codes Emre Telatar EPFL Hong Kong | January, 2013Published in: Foundations and Trends in Communications