The Application of Sparse ComponentDecomposition in the Over-complete Dictionary

to Signal RepresentationPeng Xu

School of Life Science andTechnology, University of

Electronic Science and Technologyof China, Chengdu 610054, China.E-mail: leisure_xp@l63.com

Dezhong YaoSchool of Life Science andTechnology, University of

Electronic Science and Technologyof China, Chengdu 610054, China.E-mail: dyao@uestc.edu.cn

Huafu ChenSchool of Life Science andTechnology, University of

Electronic Science and Technologyof China, Chengdu 610054, China.E-mail: chenhf@uestc.edu.cn

Abstract-Sparse component analysis (SCA) is a new andpromising method for signal processing. With SCA, a sparseand compact expression of signal can be achieved. In this paper,Matching Pursuit (MP), one of the popularly used SCAmethods, was adopted to decompose the signals in the waveletover-complete dictionary for a sparse expression and high-ratiocompression. By comparison of the decomposition andreconstruction results between wavelet used in the JPEG2000compression and MP, we see that the pulse signal which is notsparse in the wavelet dictionary may have a more sparseexpression in the over-complete dictionary, and when signal isrecovered with the same number of atoms or coefficients, theconstruction result with MP decomposition is superior to thatwith wavelet decomposition.

Index Terms-Sparse component analysis, Matchingpursuit, Over-complete dictionary, Atom

L INTRODUCTION

Hunting for the compact and effective signal expressionis one of the focal researches in the signal processing (SP)because the more compact expression the signal has, thehigher ratio compression the signal can attain [2, 3, 5-7].Sparse component analysis (SCA), which is developedrecently, is an effective signal processing method for signalcompact expression [3-7, 9]. The principle for SCA could bebriefly stated as the following [7,9]. Once a certain signalonly has several non-null entries in time or othertransformation domain, it means that the signal power isfocally concentrated and this signal could be compactlyrepresented in this corresponding domain with only a fewnon-null entries. Sparse degree is adopted to measure howcompact a signal is. Signal sparseness could be measuredwith ip norm or the decaying-to-null speed of sortedabsolute signal entries. The recent SCA transform isgenerally based on the learning in an over-completedictionary because the decomposition in the over-completelibrary is not unique and has great freedom for the basisselection, moreover when the over-complete dictionary islarge enough, the signal must have the sparse representation

in the over-complete dictionary if sparse constraints areemphasized [6-7, 9]. The construction of over-completedictionary is adapted with different application purposes andin this paper, the symmlet wavelet was taken to constructthe over-complete dictionary. There are many approaches tothe SCA decomposition, among which Matching Pursuit(MP) developed by Mallat and Zhang [1] is a popularly usedone. MP decomposes signal into sparse expressioniteratively in the over-complete dictionary and in thedecomposition procedure, the optimal basis (atom) for thesignal expression could be adaptively selected according tothe characteristic of signal. In this paper, MP was taken tosparsely decompose the signal in the symmlet waveletover-complete dictionary and the decomposition differencesbetween wavelet and SCA were compared with experimenttests.

II. METHODS

A. Sparseness ofsignalFor a signal denoted by X, if there are only a few

non-null entries in its coefficients and the number ofnon-null entries is relatively small compared with the lengthof signal, the signal can be supposed to be sparse. Thesparseness of signalX could be measured with 1P norm asfollows [9],

(1)||X||p = (E| Xi |P)j=l

Where 0 < p < 1, and n is the signal dimension of

X. The sparser a signal is, the smaller lp norm is. At thesame time, the sparseness can be expressed in anotherobvious way. Let X and Y be two different signalexpressions, if these two arrays are sorted in the descendingorder by their absolute coefficients values, the one withfaster decaying-to-null speed is sparser than the other one.As shown in Fig.l, Xis sparser than Y [9].

0-7803-9422-4/05/$20.00 ©2005 IEEE1957

In the following experiments, both the above twomeasurements are taken to score the sparseness of MP andwavelet decomposition coefficients.

B. Construction ofwavelet over-complete dictionaryIn signal processing, the usually adopted transformdictionary (matrix) is orthogonal and the decompositionresult is unique. The current SCA methods are mainly basedon the learning in a over-complete dictionary to decomposea signal sparsely. The over-complete dictionary can bederived from the complete dictionary by simplyover-sampling again [6, 9]. The difference between theover-complete dictionary and the complete dictionary is thatthe basis in the complete dictionary is orthogonal to eachother, while it may not be true in an over-completedictionary. According to the research aim, differentover-complete dictionaries such as Wavelet and discretecosine (DC) avail for different research aims. In our work,we adopted the over-complete dictionary based on thewavelet. A wavelet function can be expressed as,

expression in the corresponding dictionary is achieved. Theiterative MP algorithm can briefly be stated as follows.(1) Initialization: Set k1=,S(°) = 0,R() =S_

where S is the signal to be decomposed, R is theresidual signal during the iterations; the superscript isthe current iteration number; initializing the stoppingerror £ with a small positive number.

(2) Atom selection: Calculate inner product betweencurrent residual signal and each atom in theover-complete dictionary, and find the atom withmaximum inner product as follows:ack = max < R(k1),q j >, where j is the jth

Jatom in the dictionary and <-> is the inner productoperator.

(3) Update: Add the weight of the selected atom to thedecomposed signal S(k)corresponding contribution ofresidual signal R(k) as,

and remove thethis atom from the

WY,b (t) = T( 5)S

(2)

Where s,b are the scale and translation parameters,respectively. If over-sampling is implemented for scale andtranslation parameters in the wavelet function, a waveletover-complete dictionary could be expanded from thetraditional orthogonal dictionary, and in our work, thesymmlet wavelet was taken to construct the over-completedictionary and the number of atoms in the over-completedictionary was 8 times of signal length. In Fig.2, threesymmlet wavelet atoms at scale s=64 with differenttranslation b of 38, 45 and 60 are shown. It shows that thesethree atoms in the over-complete dictionary are notorthogonal.

i~~~~~~~~ 2k L £1 {i:isk>

X~~ ~ ~~ A;-''os I

Fig. 1. Signal X is sparser than Y Fig. 2. Three atoms in the symmletwavelet over-complete dictionary

C. Signal decomposed with Matching Pursuit*MP was developed by Mallat and Zhang to decompose

signal into sparse representation iteratively in the dictionary(D . In each iteration, a certain atom with the maximuminner product between itself and the residual signal isselected as the optimal one. After several iterations, whenthe convergence condition is satisfied, a sparse signal

S(k) =5(k-1) +CakOk,R ) =S S(k)

(4) Convergence judgment: If j R (k) II '£, stop theiterative procedure; else k = k + 1, jump to step 2and go on.

When algorithm converges, signal will be sparselydecomposed into a linear combination of some atoms in theover-complete dictionary.

m. RESULTS

A. Sparse representation ofa pulse signalWe took the pulse signal with 256 sample points as

the test signal and adopted the symmlet wavelet to constructover-complete dictionary with 2048 atoms, 8 times of signallength (256). The pulse signal was decomposed andreconstructed with symmlet wavelet and MP respectively,where the signal was decomposed with wavelet to level 5and the convergence error £ ofMP was set with JE-6.The results were shown in Fig.3 and Table I.

U.4 ami4*

0.2

0 "

ji*"q

x. : ~ oc ~ t3.:h~~~~~x

1958

.01Was iiiiiiiwi w-" OK*

Fig 4 0G rcntuo wt 3 am m

Fig. 4. EEG reconstruction with 35 atoms. Error-7%

na.jl 03X ~ 3X 311-C 1 ¶

lI~~~~~~~~~~~~ - 3L

Fig.3. Decomposition and reconstruction results with wavelet and MPrespectively. Grey signal in (c)-(h) is the error and the dark one is thereconstructed signal.

Table I. Reconstruction errors with different atoms and coefficientsAtoms 200 150 50Wavelet 6.3% 16.7% 56.1%MP 0.2% 0.5% 6.8%

The lp norms with p=0.35 for wavelet and MPcoefficients in Fig.3 (a) were 75.88 and 24.36, and therewere 104 and 22 coefficients larger than threshold 0.05 forwavelet and MP coefficients. These above results clearlyindicated that the pulse signal has sparser and morepower-concentrated expression in the over-completedictionary.

B. EEG sparse representationTo test the decomposition effect of SCA to the real

signal, we implemented the SCA decomposition for one realspontaneous EEG signal with 256 sample points. After MPdecomposition, only with 35 atoms the signal could berecovered with only 7% error. In fact, for the signalcorrupted by the noise, just like the denoising principle inthe wavelet, by the suitable selection of atoms, the noisecould be greatly removed with SCA decomposition, too.

IV. DISCUSSION

As shown in Fig.3 (a), the wavelet coefficients of pulsesignal do not decay to null fast, which means that the pulsesignal has no sparse representation in the wavelet orthogonaltransform domain, so there is relatively large error anddistortion for signal reconstruction with 50,150 and 200wavelet coefficients. Whereas only after 100 iterations, theMP decomposition coefficients and the signal iteration errorare nearly close to null. Furthermore, the fact that only 22coefficients are larger than threshold 0.05 among the MPdecomposition coefflicients (104 among the waveletdecomposition coefficients) indicates the signal power isfocally concentrated on only several atoms. The aboveexperiment results show that the pulse signal has the sparserexpression in over-complete dictionary than in theorthogonal wavelet dictionary. The sparseness guaranteesthat there is better reconstruction quality in the waveletover-complete dictionary than that in wavelet completedictionary when reconstructed with the same number ofcoefficients or atoms.

The real EEG signal shows great sparseness in theover-complete dictionary too. Because those coefficientswith relatively little value can be regarded to correspond tothe detail of signal, in the noise case, SCA could denoiseeffectively by reconstructing signal only with those atoms oflarge coefficients.

Currently, the popularly used compression techniqueJPEG2000 [10] is based on wavelet decomposition. Onlyfrom the viewpoint of decomposition and reconstructionresults, decomposition in the over-complete waveletdictionary could provide higher ratio of compression thanthat in the complete wavelet one.

V. CONCLUSION

Because any signal has sparse expression in the largeenough over-complete dictionary [6, 9], the pulse signal,which is not sparse in the wavelet dictionary, has the sparseexpression in the over-complete dictionary instead. At thesame compression ratio, SCA could have betterreconstruction result than wavelet. In our experiment, SCAprovides a promising method for higher ratio signalcompression. Furthermore, after SCA decomposition, thesignal power is more concentrated, so suitable selection of

1959

atoms for signal reconstruction could greatly remove thenoise in the signal. The physiology and neural activityresearch shows that the sparseness exists in the visionimaging structure and process [8], so an effective imagerepresentation with physiology sense may be achieved byimage decomposition and processing with SCA.

But what prevents SCA from practical use is that theover-complete dictionary is too large, which generally isseveral times of the signal dimension. In each MP iteration,it is necessary to find the optimal atom in the wholeover-complete dictionary, and the calculation load isenormous for such a greedy global search. In future work,some optimization methods such as GA, POWELL and SA,etc could be taken to accelerate the speed of searching forthe optimal atom [6].

ACKNOWLEDGMENT

This work was supported by NSFC#90208003, TRAPOYT,the 973 Project No. 2003CB716100, Doctor training Fund ofMOE,PRC, Fok Ying Tong Education Foundation (91041).

REFERENCES

[1] S. Mallat, Z. Zhang, "Matching Pursuit in a time-frequency dictionary,"IEEE Trans. SP, vol. 41, pp. 3397-3415, 1993.

[2] D.L. Donoho, "De-noising by soft-thresholding," IEEE Trans. IF, vol.41, no. 3, pp. 613-627,1995.

[3] R. Adelino, F.D. Silva, "Atomic decomposition with evolutionarypursuit," D.S.P., vol. 13, 317-337, 2003.

[4] A. Ossadtchi, S. Kadambe, "Over-complete blind source separation byapplying sparse decomposition and information theoretic basedprobabilistic approach," Proceedings (ICASSP'O1). IEEE internationalconference on acoustics, speech, and signal processing, vol. 5, pp.2801-2804, 2001.

[5] R. Coifman, M.V. Wickerhauser, "Entropy-based algorithms forbest-basis selection," IEEE Trans. IF, vol. 38, pp. 713-718, 1992.

[6] S. Chen, D.L. Donoho, M. Saunders, "Atomic decomposition by BasisPursuit,", SIAMJ. Sci Comp, vol. 20, no.l, pp. 33-61, 1999.

[7] D.L. Donoho, " Sparse components of images and optimal atomicdecompositions," available online at: http: //www.stat.stanford.edu/donoho/ reports, 1998.

[8] B.A. Olshausen, D.J. Field, "Sparse coding with an overcomplete basisset: a strategy employed by VI," Vision Research, vol. 37, pp.3311-3325, 1997.

[9] X.M. Huo, "Sparse Image Representation via Combined Transforms,"PHD Paper of Stanford univ, available online at:http://www.isye.gatech.edu/xiaoming, 1999.

[10] A.N. Skodras, C.A. Christopoulos, T. Ebrahimi, "JPEG2000: Theupcoming still image compression standard," Pattern RecognitionLetters, vol. 22, no. 12, pp. 1337-1345, 2001.

1960