Serial Concatenation Rate One

8/2/2019 Serial Concatenation Rate One

1/14

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 49, NO. 6, JUNE 2003 1425

The Serial Concatenation of Rate- 1 Codes ThroughUniform Random Interleavers

Henry D. Pfister, Student Member, IEEE, and Paul H. Siegel, Fellow, IEEE

AbstractUntil the analysis of Repeat Accumulate codes byDivsalar et al., few people would have guessed that simple rate- 1codes could play a crucial role in the construction of good binarycodes. In this paper, we will construct good binary linear blockcodes at any rate 1 by serially concatenating an arbitraryouter code of rate with a large number of rate-

1

inner codesthrough uniform random interleavers. We derive the averageoutput weight enumerator (WE) for this ensemble in the limit asthe number of inner codes goes to infinity. Using a probabilisticupper bound on the minimum distance, we prove that long codesfrom this ensemble will achieve the GilbertVarshamov boundwith high probability. Numerical evaluation of the minimumdistance shows that the asymptotic bound can be achieved witha small number of inner codes. In essence, this constructionproduces codes with good distance properties which are also com-patible with iterative turbo style decoding. For selected codes,we also present bounds on the probability of maximum-likelihooddecoding (MLD) error and simulation results for the probabilityof iterative decoding error.

Index TermsRandom coding, rate-1

codes, serial concatena-tion, turbo codes, uniform interleaver.

I. INTRODUCTION

S INCE the introduction of turbo codes by Berrou, Glavieux,and Thitimajshima [2], iterative decoding has made it prac-tical to consider a myriad of different concatenated codes, and

the use of random interleaversand recursive convolutional en-coders has provided a good starting point for the design of newcode structures. Many of these concatenated code structures fitinto a class that Divsalar, Jin, and McEliece call turbo-likecodes [1]. Perhaps the simplest codes in this class are repeat ac-cumulate (RA) codes, which consist only of a repetition code,an interleaver, and an accumulator. Yet, Divsalar et al. provethat the maximum-likelihood decoding (MLD) of RA codes hasvanishing word error probability, for sufficiently low rates andany fixed signal-to-noise ratio (SNR) greater than a threshold,as the block length goes to infinity. This demonstrates that pow-erful error-correcting codes may be constructed from extremelysimple components.

Manuscript received July 31, 2001; revised February 4, 2003. This workwas supported in part by the National Science Foundation under GrantNCR-9612802 and by the National Storage Industry Consortium, now knownas the Information Storage Industry Consortium. The material in this paper waspresented in part at The 37th Allerton Conference on Communication, Control,and Computing, Monticello, IL, September 1999.

Theauthors arewith the Departmentof Electricaland Computer Engineering,University of California, San Diego, La Jolla, CA 92093-0407 USA (e-mail:[email protected]; [email protected]).

Communicated by R. Urbanke, Associate Editor for Coding Techniques.Digital Object Identifier 10.1109/TIT.2003.811907

In this paper, we consider the serial concatenation of an ar-bitrary binary linear outer code of rate with identicalrate- binary linear inner codes where, following the conven-tion of the turbo-coding literature, we use the term serial con-catenation to mean serial concatenation through a random in-terleaver. Any real system must, of course, choose a particularinterleaver. Our analysis, however, will make use of the uniform random interleaver(URI) [3] which is equivalent to av-eraging over all possible interleavers. We analyze this systemusing a probabilistic bound on the minimum distance and showthat, for any fixed block length and large enough , the en-

semble contains some codes whose minimum distance achievesthe GilbertVarshamov bound (GVB) [4].Our work is largely motivated by [1] and by the results of

berg and Siegel [5]. Both papers consider the effect of a simplerate- accumulate code in a serially concatenated system. In[1], a coding theorem is proved for RA codes, while in [ 5], theaccumulatecode is analyzed as a precoder for thedicode mag-netic recording channel. Benedetto et al. also investigated thedesign and performance of double serially concatenated codesin [6].

We also discuss some specific codes in this family, known asconvolutional accumulate- (CA ) codes, which were intro-duced as generalized RA codes in [7] and [8]. A CA code is a

serially concatenated code where the outer code is a terminatedconvolutional code (CC) and the inner code is a cascade ofinterleaved accumulate codes. These codes were studied insome depth for by Jin in [9]. This paper focuses on thecase of , and gives a straightforward Markov chain basedanalysis of the distance properties and MLD performance.

The outline of the paper is as follows. In Section II, wereview the weight enumerator (WE) of linear block codesand the union bound on the probability of error for MLD.We also review the average WE for the serial concatenationof two linear block codes through a URI, and relate serialconcatenation to matrix multiplication using a normalized formof each codes inputoutput WE (IOWE). In Section III, we

introduce our system, shown in Fig. 1, compute its averageoutput WE, and compare this WE to that of random codes.In Section IV, we consider some properties of rate- codeswhich affect the performance of our system. In Section V,we discuss a probabilistic bound on the minimum distance ofany code, taken from an ensemble, in terms of the ensembleaveraged WE. Applying this bound to the WE from Section IIIgives an expression that is very similar to the GVB and that isasymptotically equal to the GVB for large block lengths. Wealso evaluate this bound numerically for various CA codesand observe that three or four accumulate codes seem to be

0018-9448/03$17.00 2003 IEEE
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


2/14
http://-/?-http://-/?-http://-/?-


3/14
http://-/?-http://-/?-


4/14
http://-/?-


5/14


6/14
http://-/?-http://-/?-http://-/?-


7/14
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


8/14

1432 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 49, NO. 6, JUNE 2003

(a) (b)

(c) (d)

Fig. 3. Probabilistic bound on the minimum distance of various CA codes. (a) m = 1 , (b) m = 2 , (c) m = 3 , (d) m = 4 .

be a sequence of code ensembles with information bits andcode bits such that the rate is fixed. We de-

fine asthe largest minimumdistance guaranteed, withprobability , by applying Corollary 1 to the ensemble aver-aged WE of . In the following results, we look at the sequence

using numerically averaged WEs for various code

ensembles. This means that at least half of the codes in each en-semble have a minimum distance of at least . Weconsider 16 ensembles formed by choosing one of four outercodes and the number of accumulate codes .Each outer code is referred to in shorthand: repeat by 2 (R2), re-peat by 4 (R4), rate single parity check (P9), and theextended Hamming code (H8). The results, over a range of code-word lengths, are shown in Fig. 3.

We compare these code ensembles to the uniform ensembleby focusing on the rate at which the minimum distance growswith the block length. It is important to note that, at a fixed rate,a good code is defined by a minimum distance which growslinearly with the block length. When examining these results,

we will focus on whether or not the minimum distance appearsto be growing linearly with block length and on how close the

is to the GVB. For ensembles of CA codes with, it is known that the minimum distance of almost all of

the codes grows like , where is the free dis-tance of the outer terminated CC [14]. Examining Fig. 3 for

, we see that the minimum distance grows slowly forR4 and H8 (which have ) and not at all for R2 and P9(which have ). For , the growth rate of the min-imum distance for R4, H8, and R2 appears distinctly linear. Itis difficult to determine the growth rate of P9 with fromthese results. With , all of the codes appear tohave a min-imum distance growing linearly with the block length. In fact,the apparent growth rates are very close to . Finally,with , the bounds on minimum distance andare almost indistinguishable. These results are very encouragingand suggest that, over a range of rates, even a few accumulatecodes are sufficient to approach the behavior of an asymptoti-cally large number.
http://-/?-http://-/?-


9/14
http://-/?-http://-/?-http://-/?-


10/14


(a) (b)

Fig. 4. Analytical and simulation results for a rate1 = 2

RA code withk = 1 0 2 4

andm = 1 ; 2 ; 3

. Simulations are completed using 50 decoding iterationsand the left plot shows the word-error rate (WER) while the right plot shows the bit-error rate (BER). The label XVV signifies the ViterbiViterbi (VV) boundapplied to the expurgated ensembles.

(a) (b)

Fig. 5. Analytical and simulation results for a rate 1/4 RA code with k = 1 0 2 4 and m = 1 ; 2 ; 3 . Simulations are completed using 50 decoding iterations andthe left plot shows the WER while the right plot shows the BER. The label XVV signifies the VV bound applied to the expurgated ensembles.

B. MLD Performance

In Section V-B, we applied (4) to compute the averagedWEs for several CA code ensembles. Using the WEs and

the ViterbiViterbi (VV) bound [19], we calculated upperbounds on the probability of MLD error for some of theseensembles. The results for the R2, R4, and P9 ensembles aregiven in Figs. 4, 5, and 6, respectively. These figures alsoshow the results of iterative decoding simulations which willbe discussed in the next section. At high SNR, these boundsare dominated by the probability of picking a code with smallminimum distance, as reflected in the pronounced error floorsof the nonexpurgated ensembles in Figs. 4, 5, and 6. Forthis reason, we also considered the expurgated ensembles, asdescribed in Section V-C, with .

The results of applying the ViterbiViterbi (VV) bound tothese ensembles have some characteristics worth mentioning.

In all cases, increasing , the number of accumulate codes,seems to improve the performance both by shifting the cliff re-gion to the left and by lowering the error floor. We also see that,in some cases, the effect of expurgation is negligible, which im-plies that almost all of the codes in the ensemble have smallminimum distance. As we saw in Section V-B, the minimumdistance of the expurgated ensemble depends on the outer codeand the number of accumulate codes. The minimum distanceof the outer code does not seem to completely explain the be-havior though, because the P9 ensemble requires one more ac-cumulatecode than theR2 ensemble in order forexpurgation tomake a significant difference. Of course, at longer block lengthsthis may change.

The axes of the figures were chosen to show details of theperformance curves, but in many cases the error floor of the ex-purgated ensemble is too low to be shown. Consider the R2 en-
http://-/?-http://-/?-


11/14

PFISTER AND SIEGEL: THE SERIAL CONCATENATION OF RATE-1 CODES THROUGH UNIFORM RANDOM INTERLEAVERS 1435

(a) (b)

Fig. 6. Analytical and simulation results for a rate 8/9 P A with k = 1 0 2 4 and m = 1 ; 2 ; 3 . Simulations are completed using 50 decoding iterations and theleft plot shows the WER while the right plot shows the BER. The label XVV signifies the VV bound applied to the expurgated ensembles.

(a) (b)

Fig. 7. Simulation results for rate1 = 2 RA codes for 30 decoding iterations with m = 1 ; 2 and k = 1 0 2 4 ; 2 0 4 8 ; 4 0 9 6 ; 8 1 9 2 ; 1 6 3 8 4 . (a) Word-error rate(WER). (b) Bit-error rate (BER)

semble with ; the word-error rate (WER) of the expur-gated ensemble remains steep until around 10 where it flat-tens somewhat. The expurgated R2 ensemble with hasa WER which remains steep until well below the numerical ac-

curacy of our computations. The expurgated P9 ensemble withalso shows no error-floor region, but the curve losessome steepness at a WER of 10 . These error floors are inter-esting because understanding the performance of these codes athigh SNR, where simulation is infeasible, is important for ap-plications where very low error rates are required.

C. Iterative Decoding Performance

In this subsection, we use computer simulation to evaluatethe performance of iterative decoding for these codes. A singledecoding iteration corresponds to a posteriori proba-bility (APP) decoding operations of an accumulate code (a

backward/forward pass through all accumulate encoders) anda single APP decoding operation of the outer code. It is worthnoting that the complexity of iterative decoding is linear in both

and , making it quite feasible to implement. All simula-

tion results were obtained using between 20 and 50 decodingiterations, depending on the particular code, and modest gainsare observed (but not shown) when the number of iterations isincreased to 200. These results are compared with analyticalbounds in Figs. 46 and shown by themselves in Figs. 7 and 8.

The discrepancies between the simulation results and MLDbounds in Figs. 46 are very pronounced. While the MLDbounds predict uniformly improving performance with in-creasing , it is clear that the performance of iterative decodingdoes not behave in this manner. The optimum depends on thedesired error rate and the minimum distance of the outer code.In general, it appears that increasing moves the cliff region ofthe error curve to the right and makes the floor region steeper.


12/14


(a) (b)

Fig. 8. Simulation results for rate8 = 9 PA codes with k = 1 0 2 4 ; 2 0 4 8 ; 4 0 9 6 ; 8 1 9 2 ; 1 6 3 8 4 and 20 decoding iterations. (a) Word-error rate (WER). (b)Bit-error rate (BER)

This seems reasonable because more rate- decoders (whichhave no coding gain) are applied before the outer code (with allof the coding gain) is decoded. This results in a phenomenonwhere the iterative decoder often does not converge, but rarelymakes a mistake when it does converge.

The expurgated WE can also be used to detect the presenceof bad codes which are chosen with low probability. If the MLDexpurgated bound is better than the nonexpurgated bound, thenthe effect of these bad codes has been reduced. The MLD ex-purgated bound is not shown when it coincides with the nonex-purgated bound. In some cases, iterative decoding is performing

better than the MLD expurgated bound (e.g., the R4 ensemblewith ). This may occur because the use of well designed(e.g., S-random [20]) interleavers can provide a minimum dis-tance which is better than that guaranteed by Lemma 1.

The interleaver gain exponent (IGE) conjecture is based onthe observations of Benedetto and Montorsi [3] and is statedrigorously in [1]. It states that the probability of MLD decodingerror for turbo-like codes will decay as , where de-pends on the details of the coding system. If the IGE conjecturepredicts that the bit-error rate (BER) (resp., WER) will decaywith the block length, then we say that the system has BER(resp., WER) interleaver gain. It is easy to verify that WER in-terleaver gain implies BER interleaver gain. The IGE and the

MLD expurgated bound are quite closely connected. If a systemhas WER interleaver gain, the probability of picking a codewith codewords of fixed weight must decay to zero as the blocklength increases. Therefore, one would expect the MLD expur-gated bound to beat the nonexpurgated bound. On the otherhand, if a system has only BER interleaver gain, then it is likelythat the MLD expurgated bound will equal the nonexpurgatedbound.

Finally, the IGE conjecture predicts that the R2 code will haveno WER interleaver gain (i.e., ) for , butthat it will have WER interleaver gain (i.e., ) for

. In Fig. 7, the WER of the R2 code with doesindeed appear to be independent of block length and the WER of

the R2 code with is clearly decreasing with block length.In Fig. 8, we see similar behavior for the interleaver gain of theP9 codes.

VII. CONCLUSION AND FUTURE WORK

In this paper, we introduce a new ensemble of binary linearcodes consistingof any rate outer codefollowed bya largenumber of uniformly interleaved rate- codes. We show that thisensemble is very similar to the ensemble of uniform randomlinear codes in terms of minimum distance and error exponentcharacteristics. A key tool in the analysis of these codes is a cor-

respondence between inputoutput weight transition probability(IOWTP} matrices and Markov chains (MCs), which allows usto draw on some well-known limit theorems from MC theory.We derive a probabilistic bound on the minimum distance ofcodes from this ensemble, and show it to be almost identical tothe GilbertVarshamov bound (GVB). In particular, our anal-ysis implies that almost all long codes in the ensemble have anormalized minimum distance meeting the GVB.

Next, we consider a particular class of these codes, whichwe refer to as convolutional accumulate- CA codes.These codes consist of an outer terminated convolutional codefollowed by uniformly interleaved accumulate codes. Weevaluate the minimum distance bound for a few specific CA

codes for and observe that these relatively smallvalues may be sufficient to approach the GVB. Finally, we

use computer simulation to evaluate the bit-error rate (BER)and word-error rate (WER) performance of these CA codeswith iterative decoding and compare this to the performancepredicted by union bounds for MLD.

Remark 5: An MLD coding theorem for CA codes canbe found in [8], with numerical estimates of the correspondingnoise thresholds. Also given there are the thresholds which re-sult from applying density evolution [21] to the iterative de-coding of these codes. Finally, a comprehensive treatment ofboth of these subjects can be found in [22].
http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-http://-/?-


13/14


14/14


Fig. 9. The weight mapping graph, G , with the G subgraph drawn in solidlines.

We illustrate the proof techniqueusingthe accumulate codeexample from Section II-C. The impulse response, , of theaccumulate code is the infinite sequence of ones, ,for . The weight mapping graph is shown in Fig. 9and the subgraph is drawn with solid lines. It is easy tosee that is both irreducible and aperiodic; in particular, notethe self-loop at the vertex labeled 1. There is an edge fromvertex 4 to vertex 2, corresponding to the weight- input vector

, and a directed edge from vertex 1 to vertex 4, cor-

responding to the weight- input vector . Togetherwith the irreducibility of , this implies that is irreducible.The self-loop at vertex 1 ensures the aperiodicity and, therefore,the primitivity of .

ACKNOWLEDGMENT

The authors would like to thank the associate editor, R. Ur-banke, and the two anonymous reviewers for their helpful com-ments. We are also very grateful to M. berg for posing theinitial question that led to this research and for many helpfuldiscussions along the way.

REFERENCES

[1] D. Divsalar, H. Jin, and R. J. McEliece, Coding theorems for turbo-like codes, in Proc. 36th Annu. Allerton Conf. Communication, Con-trol, and Computing, Monticello, IL, Sept. 1998, pp. 201210.

[2] C. Berrou, A. Glavieux, and P. Thitimajshima, Near Shannon limiterror-correcting coding and decoding: Turbo-codes, in Proc. IEEE Int.Conf. Communications, vol. 2, Geneva, Switzerland, May 1993, pp.10641070.

[3] S. Benedetto and G. Montorsi, Unveiling turbo codes: Some resultson parallel concatenated coding schemes, IEEE Trans. Inform. Theory,vol. 42, pp. 409428, Mar. 1996.

[4] T. Ericson, Bounds on the Size of a Code (Lecture Notes in Controland Information Sciences). Berlin, Heidelberg, Germany: Springer-Verlag, 1989, vol. 128, pp. 4569.

[5] M. berg and P. H. Siegel, Performance analysis of turbo-equalizeddicode partial-response channel, in Proc. 36th Annu. Allerton Conf.Communication, Control, and Computing, Monticello, IL, Sept. 1998,

pp. 230239.[6] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, Analysis, de-sign, and iterative decoding of double serially concatenated codes withinterleavers,IEEEJ. Select. Areas Commun., vol. 16,pp. 231244,Feb.1998.

[7] H. D. Pfister and P. H. Siegel, The serial concatenation of rate-1 codesthrough uniform random interleavers, in Proc. 37th Annu. AllertonConf. Communication, Control, and Computing, Monticello, IL, Sept.1999, pp. 260269.

[8] , Coding theorems for generalized repeat accumulate codes, inProc. Int. Symp. Information Theory and Its Applications, vol. 1, Hon-olulu, HI, Nov. 2000, pp. 2125.

[9] H. Jin, Analysis and design of turbo-like codes, Ph.D. dissertation,Calif. Inst. Technol., Pasadena, May 2001.

[10] A. J. Viterbi and J. K. Omura, Principles of Digital Communication andCoding. New York: McGraw-Hill, 1979.

[11] E. Seneta, Non-Negative Matrices: An Introduction to Theory and Ap-

plications, 2nd ed. New York: Wiley, 1981.[12] A. Papoulis, Probability, Random Variables, and Stochastic Processes,3rd ed. New York: McGraw-Hill, 1991, ISBN 0-07-048477-5.

[13] R. G. Gallager, Low-Density Parity-Check Codes. Cambridge, MA:MIT Press, 1963.

[14] N. Kahale and R. Urbanke, On the minimum distance of parallel andserially concatenated codes, in Proc. IEEE Int. Symp. InformationTheory, Cambridge, MA, Aug. 1998, p. 31.

[15] E. N. Gilbert, A comparison of signalling alphabets, Bell Syst. Tech.J., vol. 31, pp. 504522, May 1952.

[16] J. N. Pierce, Limit distribution of the minimum distance of randomlinear codes, IEEE Trans. Inform. Theory, vol. IT13, pp. 595599,Oct. 1967.

[17] R. G. Gallager, A simple derivation of the coding theorem and someapplications, IEEE Trans. Inform. Theory, vol. IT11, pp. 317, Jan.1965.

[18] N. Shulman and M. Feder, Random coding techniques for nonrandom

codes,IEEETrans. Inform. Theory, vol. 45,pp. 21012104, Sept. 1999.[19] A. M. Viterbi and A. J. Viterbi, Improved union bound on linear codesfor the input-binary AWGN channel, with applications to turbo codes,in Proc. IEEE Int. Symp. Information Theory, Cambridge, MA, Aug.1998, p. 29.

[20] D. Divsalar and F. Pollara, Turbo codes for PCS applications, in Proc.IEEE Int. Conf. Communications, Seattle, WA, June 1995, pp. 5459.

[21] T. J. Richardson and R. L. Urbanke, The capacity of low-density paritycheck codes under message-passing decoding, IEEE Trans. Inform.Theory, vol. 47, pp. 599618, Feb. 2001.

[22] H. D. Pfister, On the capacity of finite state channels and the analysisof convolutional accumulate- m codes, Ph.D. dissertation, Univ. Calif.,San Diego, La Jolla, 2003.

Documents

Serial Concatenation Rate One