View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Linear-time Encodable/Decodable Error-
Correcting Codes
Some results from two papers of M. Sipser and D. Spielman1
1Much of the technical wording within is either taken verbatim or paraphrased from [SS96; S96]; see references.
Coding for Low-Power Communication Across Noisy
Channels
• Static on the line / Game of “Operator” (Spielman’s lecture).
• A coding solution: “A” as in “Apple”, “B” as in “Baseball”, …”Z” as in Zebra.
(Encode)
Messages (live in low-d space)
Codewords (live in high-d space) =
Message Bits + Check Bits + (Decode)
Diagram for codeword space inspired by digital signature lecture at MIT earlier this year
minimum distance bound
Motivation
• Recall one question answered in professor Spielman’s lecture:
• Q1: Do there exist families of codes which as block length grows realize: – constant information rate / – bounded relative minimum distance
• A: Yes! (e.g. Justesen’s explicit construction)
• Q2: Can we construct such a family of codes with linear time algorithms for encoding AND decoding a constant fraction of error?
• A: Yes again!
)( n
r n
Goals
1. Describe randomized and explicit constructions of families of asymptotically good error-correcting codes
2. Describe linear-time encoding and decoding algorithms for these codes– Sketch proofs of correctness for the decoding
algorithms– Motivate why these algorithms are in fact
linear-time
Sights and sounds along the way…1. Linear-time decodable error-correcting
codes (built from expander graphs and existing code)
• Randomized construction• Explicit construction
2. Linear-time encodable/error-reducible Error-reduction codes (also from expander graphs and existing code)
• Explicit construction
3. Linear-time encodable/decodable error-correcting codes (from error-reduction codes)
…and mathematical flavors
• Graph theory– Clever correspondences between graphs and codes, graphs
and graphs. – Expander and regularity properties of underlying graphs
used to establish • minimum distance bounds • correctness of decoding algorithms
– Regularity also used to show rate, linear-time
• Probability– The “probabilistic method,” or, a way to prove existence
theorems non-constructively– How to identify properties of randomly constructed objects
Models of Computation for Linear-Time1
• Cost of Memory access / Bit-wise operations (as a function of argument size n):
– Uniform cost model => 1 time unit– Logarithmic cost model => log n time units
• Machines for sequential decoding algorithms– Graph inputted as a set of pointers; each vertex indexes a list
of pointers to its neighbors2
• Pointer Machine (constant degree directed graph)• RAM
• Boolean circuits for parallel decoding algorithms– Size (= # wires)– Depth (= longest path length in circuit digraph)
2 [SS96, p1713]1 [SS96, p1710-11]
Important things to recall• Linear Codes: codewords form a vector space• Gilbert-Varshamov bound: “Good” linear codes exist for sufficiently long
block lengths • Graphs G = (V, E)• Expander graphs (Mohamed’s lecture)
– All minority subsets of vertices have many (external) neighbors (large “boundary”):
Expansion factor =
• Polynomial-time constructible infinite families of k-regular Ramanujan graphs (Mohamed’s lecture)1
– Realize maximum possible “spectral gap” (~expansion)– Explicit construction of Lubotsky-Philips-Sarnak/Margulis using Cayley graphs2
|||)(| UcU c
1 [LL00 p1]
• “Unbalanced”, (c,d)-regular bipartite graphs • Regularity important for showing linear-time; also for counting
arguments used to prove correctness of decoding algorithms.
Six 2-regular Four 3-regular
vertices vertices
• Degree ratio c/d determines bipartite ratio (edge count)
Example: A (2,3)-regular graph:
Defining expansion for (c,d)-regular graphs
• Only consider subsets of the c-regular vertices (LHS)
Definition1: A (c,d)-regular graph is a -expander if every subset of at most an -fraction of the c-regular vertices expands by a factor of at least .
1 [SS96, p1711]
),,,( dc
Building an Expander Code:C(B,S)
• Necessary Ingredients– a (c,d)-regular expander graph (c<d)– a good block-length error-correcting code
(existence by the Gilbert-Varshamov bound)
– A correspondence between the graph and the codewords of .
BS d
B),( SBC
Corresponding
(c-regular) vertices (d-regular) vertices
variables constraints
),( SBCB
},,,{ 21 nvvv
dcn
},,,{ 21 dcnCCC
n
The expander code1 : • A constraint is satisfied if its d neighboring
variables form a codeword of the input code S.• The expander code C(B,S) is the code of block
length whose codewords are the settings on the variables such that all constraints are satisfied simultaneously.
• The (linear) code C(B,S) is the vector subspace which solves the set of all linear constraints
• The constraints collectively determine a check matrix H (from which generator matrix G can be derived)
n
1 [SS96, p1712]
),( SBC
C(B,S) is asymptotically good1
• For:– Graph B with good enough expansion– Code S with bounded rate and minimum distance
• Conclusion:– C(B,S) is a code of (related) bounded rate and minimum distance, for all block lengths n.
Theorem1: Let be a -expander and an error-correcting code of block length , rate , and minimum relative distance . Then C(B,S) has rate at least
and minimum relative distance at least . (note that these quantities are independent of the code’s block length ).
1[SS96, p1712]
• Proof1: Rate Bound (Relies on regularity & rate of S)
– Count the total number of linear restrictions imposed on the subspace of codewords
– Assume all restrictions are mutually independent
– Dimension of solution space = # degrees of freedom left out of block length n
– Corresponds to the # message bits in code => value for rate
1 [SS96, p1712]
• Proof (continued): Min Distance Bound– Sufficient to show that no codeword w exists below a
certain weight.
– Contradiction by regularity and expansion:
– Regularity fixes the number of edges leaving the subset of all 1 (non-zero) variables in w
– Expansion guarantees these edges enter a lot of constraints
– => Average #edges/constraint low
– => there must be some constraint with few 1 variables as neighbors
– By minimum distance bound of underlying code S, entire word can’t be a codeword of C(B,S).
Encoding and Decoding C(B,S)
• Encoding – quadratic time algorithm (multiply message by
generator matrix G)
• Decoding– Brute force– Linear time algorithms
• Challenge: Correctness proof requires constructing a graph B with expansion greater than c/2; no explicit constructions are known for such graphs.
Constructing a (c,d)-regular graph with very good expansion (>c/2)
• Explicit constructions: Best known (1996) can give expansion c/2, but no greater:– (Lubotsky-Phillips-Sarnak, Margulis 1988)1: Ramanujan
graphs: (p+1)-regular, second-largest eigenvalue – (Kahale 1992)2: Lubotsky et al.’s Ramanujan graphs have
quality expansion, but spectral gap can not certify expansion greater than c/2
• Random (c,d)-regular graphs will almost always have greater expansion– Food for thought: How could this random construction be
done?
1 [SS96, p1712]
p2
2[SS96, p1713]
Random (c,d)-regular graphs: How good is their expansion?
• For B a randomly chosen (c,d)-regular bipartite graph between n variables and (c/d)n constraints, for all , with exponentially high probability any set of
variables will have at least
neighbors1. Pf: Bound probability that the # neighbors of any subset is far from its expected # neighbors
eHcn ddc
2log/)(2))1(1(
n10
1[SS96, p1721]
Decoding a Random Expander Code C(B,S)
• Flip variable(s) in more unsatisfied than satisfied constraints until none remain
– Sequential Algorithm– Parallel Algorithm
• Proof of correctness (sequential algorithm): – Relies on expansion and regularity properties of graph B,
as well as nature of code S. Sufficient to show:– As long as there are some corrupt variables (but # below
threshold), then some variable will be flipped.– =>Number of unsatisfied constraints strictly decreases
• Linear time relies only on regularity of graph B.– Total # (unsatisfied) constraints linear in block length– Local graph structure unchanged as block length grows;
fixed code S => constant local cost
Remarks on Construction
• Expansion factor required: for efficient decoding of one example code
• Bound on rate assumed all constraints imposed on variables were independent
• Potential applications for codes with some redundancy in constraints (see Spielman’s thesis1, Ch.5 PCP’s and “checkable” codes)
43c
1http://www-math.mit.edu/~spielman/Research/thesis.html
Explicit Construction of Expander Code C(B,S)
• To construct the code C(B,S), we will take– B the edge-vertex incidence graph of a d-
regular Ramanujan graph G (Lubotsky et al: A dense family of good expander graphs)1
– S a good block length d code
1[S96, p1726]
Edge-vertex incidence graphs
• Definition: Let G be a graph with edge set E and vertex set V. The edge-vertex incidence graph B(G) of G is the bipartite graph with vertex set and edge set:
• Thus the edge-vertex incidence graph B(G) of our d-regular Ramanujan graph G will be a (2,d)-regular bipartite graph, to which we correspond dn/2 variables and n constraints.
VE
evVEve ofendpoint an is :),(
Corresponding
Edges E of G Vertices V of G
(2-regular) vertices (d-regular) vertices
variables constraints
),()( SBCGB
},,,{
221 dnvvv
n
},,,{ 21 nCCC
2dn
Parallel Decoding for C(B,S)1
(where S has block length d, min rel dist )
1. For each constraint, if the variables in that constraint differ from a codeword of S in at most places, then send a “flip” message to each variable that differs.
2. In parallel, flip the value of every variable which receives at least one flip message.
1[SS96, p1716]
4de
Main Theorem1
• There exist polynomial-time constructible families of asymptotically good expander codes in which a constant fraction of error can be corrected in a circuit of size and depth .
• The action of this circuit can be simulated in linear time on a Pointer Machine or a RAM under the uniform cost model. (No natural sequential decoding algorithm is known).
1 [SS96, p1716]
)log( nnO)(log nO
Proof Sketch• Outline similar to the proof above for the random expander case• One important difference:
– Above, it was the known expansion factor of graph B which was used to prove
• Minimum distance bound• Correctness of decoding algorithm
– Here, instead apply a result of Alon-Chung to upper bound the number of edges contained in a subgraph of a graph G with a given spectral gap.
– Convenient, since our edge-vertex incidence graph B(G) is built from a Ramanujan graph G (Lubotsky) that is qualified in terms of spectral gap, not expansion
.
(Alon-Chung)1: Let G be a d-regular graph on n vertices with second largest eigenvalue . Let X be a subset of the vertices of G of size . Then, the number of edges in the subgraph induced by X in G is at most
1(22 d
dn
n
1 [SS96, p1712]
• Could use Alon-Chung bound to quantify the expansion factor of the edge-vertex incidence graph B(G)
• But a direct application of Alon-Chung itself works well in proof
• => each decoding round removes a constant fraction* of error from input words within threshold of nearest codeword. *(fraction <1 for
sufficiently large choices of block length d of S)
• => after O(log n) rounds, no error remains.
Circuit for Parallel Decoding1
• Each parallel decoding round can be implemented by a circuit of linear size and constant depth
– constraint satisfaction: constant # of layers of XOR’s
– variable “flips”: majority of constant # of inputs
• => Total of O(log n) rounds needs a circuit of size O(n log n), depth O(log n).
1 [SS96, p1715]
Linear Time of Sequential Simulation (Uniform Cost Model)
• During each round, compile a list of all constraints which might be unsatisfied at the beginning of next round by including:
– All constraints of previous round’s list which actually were unsatisfied
– All constraints neighboring at least one variable which received a “flip” message in current round
1. Total amount of work performed by algorithm is linear in the combined length of these lists:
– Each round: Check which constraints from input list are actually unsatisfied; do decodes on S; send “flips”
– Regularity, fixed code S => constant local cost
2. Total length of these lists is linear in block length n of the code C(B,S).
– # variable errors decreases by constant ratio each round – Only constraints containing these variables appear in lists– => convergent geometric series, first term O(n)
Linear Time of Sequential Simulation (Logarithmic Cost Model)
Challenge: • Need O(n) bit additions and O(n) memory accesses• Cost per access is O(log n)• Total cost O(n log n)Solution:• Retrieve an O(log n) byte per access, only need to make
O(n/log n) accesses; total cost O(n)• Encode each byte with a good linear error-correcting code• Algorithm will correct a constant fraction of error in
overall code unless there is a constant fraction of error in the encoding of a constant fraction of bytes.
Expander Codes Wrap-Up1
• Error distance from a correct codeword was corresponded to a potential1 (of unsatisfied constraints) in a graph whose expander and regularity properties guarantee the success of a linear-time decoding algorithm which steadily decreases this potential to zero.
• Quadratic time encoding / linear time decoding• Superior speed of sequential decoding on random
expanders• Can tune parallel decoding algorithm for better
performance1[SS96, p1718-20]
Expander Codes Wrap-Up
• Average-case error-correction results in experiments much better than bounds predict (worst-case).
• Introducing random errors in variable flips can further improve performance.
• For random expanders, degree 5 at variables seemed to give best results.
Linear-time Encodable/Decodable Error-Correcting Codes
• Expanders leveraged to produce linear time encodable/decodable error-reduction codes
• Construction involving a recursive assembly of smaller error-reduction & error-correcting codes gives linear time error-correcting codes.
• Decoding algorithm, proof of its correctness are the same flavor as those already encountered.
• Randomized / Explicit constructions– Explicit case shown here is a generalization of the
randomized one
Historical Context1
• Prior constructions did not always use such well-defined models of computation or deal with asymptotic complexity
• Juntesen and Sarwate used efficient implementation of polynomial GCD algorithm to get O(n log n log log n) encoding, O(n log2 n log log n) decoding for RS and Goppa codes.
• Gelfand, Dobrushin, and Pinsker gave randomized constructions for asymptotically good codes with O(n) encoding; but no polynomial-time algorithm for decoding is known.
1 [S96, p1723]
Main Theorem1
• There exists a polynomial-time constructible family of error-correcting codes of rate ¼ that have linear-time encoding algorithms and linear-time decoding algorithms for a constant fraction of error.
• The encoding can be performed by linear-size circuits of logarithmic depth and decoding can be performed by circuits of size O(n log n) and O(log n) depth, and can be simulated sequentially in linear time.
[S96, p1729]
Error-Reduction Codes
• Definition: A code C is an error-reduction code if there is an algorithm ER which, given an input word w with total # errors below threshold, can reduce the # of message bit errors in w to a constant fraction of the # of check bit errors originally present in w.– Note: for zero check bit errors, implies error-correction
of all errors in message bits.
• Can construct such codes from good expanders
Building the Error-Reduction Code: R(B,S)
• Necessary Ingredients– B a (c,d)-regular expander graph (c<d)– S a good block length d error-correcting code
with k message bits (rate = k/d)– A correspondence between the graph B and
the codewords of R(B,S).
Corresponding
(c-regular) vertices (d-regular) vertices
message bits check bit clusters
),( SBRB
},,,{ 21 nvvv
dcn
},,,{ 21 dcnCCC
n
The error-reduction code R(B,S) 1
• The error-reduction code R(B,S) will have n message bits and (c/d)nk check bits.
• The n message bits are identified with the n c-regular vertices.
• A cluster Ci of k check bits is identified with each of the (c/d)n d-regular vertices.
• The check bits of cluster Ci are defined to be the check bits of that codeword in S which has the d neighbors of cluster Ci as its message bits
• R(B,S) can be encoded in linear time, (constant time for generating each check bit cluster; linear # of clusters)
1 [S96, p1726]
Parallel Error-Reduction round of R(B,S)1
(where is the minimum distance of S)
• In parallel, for each cluster, if the check bits in that cluster and the associated message bits are within relative distance of a codeword, then send a “flip” signal to every message bit that differs from the corresponding bit in the codeword.
• In parallel, every message bit that receives at least one “flip” message flips its value.
[S96,p1728]
6
Quality of an Error-Reduction Round on R(B,S)
• Take B as the edge-vertex incidence graph of a d-regular Ramanujan graph G
(Lubotsky et al: A dense family of good expander graphs)1
• Can show every error reduction round removes a constant fraction* of error from the message bits of input words which are within threshold of nearest codeword. *(fraction <1 for sufficiently large choices
of block length d of S)
1[S96, p1726]
Proof of Quality of Error-Reduction
• Analogous to that seen above for error-correction of explicitly constructed expander codes
• Relies on the same graph properties:– Regularity (and fixed code S)– theorem of Alon-Chung (~expansion)
Linear Time of Error-Reduction on R(B,S)
• Argument analogous to that for linear-time simulation of parallel error-correction for the explicitly constructed expander codes above
• For good enough eigenvalue separation (~expansion) of graph B & good enough min distance of code S, after O(log n) rounds, error in message bits will be reduced to within threshold required by recursive decoding of overall error-correcting code Ck.
{Err-Red Rk}+{Err-Cor Ck-1} => Err-Cor Ck
2kR
kMkA kB kC
1kC
1kR
bits2 30
kn bits23 30
kn bits2 20
kn
bits2 20
kn
Recursive Definition of the Error-Correcting Codes Ck
To encode message :
• Use Rk-2 to encode
• Use Ck-1 to encode
• Use Rk-1 to encode
• Codeword of Ck:
kM kA
kB
kC
kA
)()( kkkk CBAM
kk BA
bits2 20
kn bits2 30
kn
bits23 30
kn
bits2 20
kn
bitscheck 23 20
kn bits message2 20
kn
bits2length block total 0knn
) (rate 41
kM
Linear-Time Decoding of codeword: • ER linear-time error-reduction for Rk
=> linear-time error-correction for Ck
– Proof by induction – Hypothesis: Assume linear-time decoding (error-
correcting) algorithm EC which works for all sub-codes Ci, i<k.
1. Run ERx on • Reduces error in
2. Run EC on • Corrects ALL errors in ! (by quality of ERx, I.H.)
3. Run ER0 on • Corrects ALL errors in !!! (by assumption on ER0)
)()( kkkk CBAM
kkk CBA )()( kk BA
kk BA )(kA
kk AM )(
kM
Remarks1
• By modifying this construction one can obtain asymptotically good linear-time encodable and decodable error-correcting codes of any constant rate r.
• Linear time codes of rate ¼ will correct about as many errors on average as expander codes of rate ½
1[S96 p1730-1]
Conclusions1. Graphs - interesting results obtained by choosing
clever correspondences for edges and vertices: E.g.
– Variables-Message bits / Constraints– Edges / Vertices of another graph– Alon’s generalization: correspond variables with all
length-k paths (here k = 2) in another graph
2. Probabilistic Method1
– A nice way of proving existence theorems. E.g. Suppose I can’t show or describe to you a particular object from a set S with property X, but perhaps I can prove that for an object randomly selected from S, Pr[X] > 0, ...
1http://www.math.uiuc.edu/~jozef/math475/description.pdf
Some questions to ponder…1
• Can rate vs. error correction tradeoffs be improved by a more natural (linear-time) decoding algorithm for these codes? (Present algorithm is an unnatural simulation of
a parallel algorithm) • Does there exist a (linear-time encodable?) code
for which a number of errors matching the Gilbert-Varshamov bound can be corrected in linear-time? How good can linear-time codes be?
1[S96 p1730-1]
References (1)
• Berlekamp, Elwyn R. Algebraic Coding Theory. Laguna Hills, CA: Aegean Park Press. 1984.
• Blahut, Richard E. Theory and Practice of Error-Control Codes. Reading, MA: Addison-Wesley Pub. Co. 1983.
• [LR00] Lafferty, John and Dan Rockmore. Codes and Iterative Decoding on Algebraic Expander Graphs. International Symposium on Information Theory and Its Applications. November 5-8, 2000.
References (2)
•[SS96] Sipser, Michael and Daniel A. Spielman. Expander Codes. IEEE Transactions on Information Theory, Vol. 42, No. 6, November 1996 p1710-1722•[S96] Spielman, Daniel A. Linear-Time Encodable and Decodable Error-Correcting Codes. Ibid. p1723-1731•Sudan, Madhu Lecture Notes from “Algorithmic Introduction to Coding Theory”. Fall 2001(2002) MIT.