IMAGE INPAINTING METHODS BYUSING CELLULAR NEURAL NETWORKS

Alexandru Gacsddi* andPter Szolgay'*Electronics Department, University of Oradea, Str. Armatei Romdne, Nr.5, 410087, Oradea, Romania,

e-mail: agacsadi(uoradea.ro,Analogical and Neural Computing Laboratory, Computer and Automation Institute, Hungarian Academy ofSciences, P.O.B.63, Tel: +36-12095265, Fax: +36-12095264, H-1502 Budapest, Hungary, affiliated also to

Image Processing and Neurocomputing Department, Veszprem University, Egyetem u. 10. Veszprem, Hungary,e-mail: szolgaygsztaki.hu.

ABSTRACTSome CNN methods are presented that can be used for thereconstruction of damaged or partially known images. Theproposed methods take the possibility of direct implementationon an existing CNN chip into account, in a single step, by using3*3 dimensional linear reaction templates. Due to completeparallel processing, computational time reduction is achieved.Efficiency of these methods can be increased by combining themwith nonlinear template that ensures the growth of the localproperties spreading area along with regional ones.

where an artificial parameter t has been used and F is theoperator which characterizes the desired processing algorithm(F: R2-+R). In general, function F depends on the initial image,and its first and second order spatial derivatives. The final imageis a solution of this partial differential equation. By using avariational formulation, the same image processing problem canbe obtained as the minimization of a cost function:

arg{Min.E(tD)}, (2)

where E is a given energy function, and F is the first orderderivative of E. Through minimizing E, 'D results from thecondition: F(D)=O, which is a steady state solution of equation:

1. INTRODUCTION (3)

Image inpainting is an interpolation problem where an imagewith missing or damaged parts is restored. An observer notfamiliar with original image, practically will not notice that theimage has been restored. The most often used image inpaintingapplications are for pictures or films known or damagedpartially. In image restoration it is possible to obtain specialeffects, eliminating some unwanted parts, texts or objects[5][6][7].As a first step the user manually selects the portions of the imagethat will be restored. Then image restoration is doneautomatically, by filling these regions in with informationcoming from the surroundings. The output image is of the samedimensions and resolution as the input image [8][IO].

In the case of image restoration, the unknown parts may bebigger, and they generally do not contain any information, whilein the case of denoising, the pixels contain information withadditive noise [9][14].

Complex mathematical models based on partial differentialequations (PDE) are available in the field of reconstruction orinpainting of damaged image [4].

Let us take a gray-scale image 'D(x,y) where (D: R2-R, andQ={(x,y): xe[l,M], ye[l,N], M and N E R+}. The processing ofthis image, with an algorithm based on an operator, can bedescribed by the following partial differential equation:

= F [D(x,y,t)]' (1)at

where t is also an artificially introduced parameter.

Regardless of the chosen formulation for modeling the imageprocessing, two or more obtained solutions allow us to makecombinations of them, resulting in another complex processing.If, e. g., two distinct processing are described by cost functionsEl and E2, another complex image processing can be formulatedminimizing the energy:

aE1±+E2 - (4)

Weighting the terms El and E2, with scalar parameters a and X(a and X ER+), let us balance the complex processing betweenthe limits described by the initial results.

Considerable computing power is necessary to solve the imageprocessing task described by variational computing. For the timebeing serial processing does not provide us with methodsimplementable in real-time. The Cellular Neural Networksproved to be very useful regarding real-time image processing[1]. The reduction of computing time, due to parallel processing,can be obtained only if the processing algorithm can beimplemented on a CNN-UC [3].

Even if variational methods are used, the determination oftemplates ensuring the gray-scale image the desired processingremains a difficult problem, since the fact that the actuallyexisting CNN chip can use only linear templates having 3*3dimension has to be taken into consideration. In some casestemplates satisfy these conditions [11] by using nonlineartemplates. Effective CNN implementation is still possible inCNN algorithms [13].

198

2. THE RECONTRUCTION OFDAMAGED IMAGES BY USING CNN

METHODSRegarding CNN gray-scale image processing, variatiitemplate design is possible if the following design constrainsrespected:

* For ensuring network stability and maintaining it in anysituation, the values of state image xij and the values ofthe output image yij, in the linear domain [-1,1], thenecessary and sufficient condition are:

EA + EB +zZ + SCjj1x1 + YDjjw(uu,xw,yu)CkjeN, CideN, Ck,eN, CkjeN,

for 1< i < M; 1< j < N, where Nr is the neighborhood r radius

N, (i,j)= ta(k,ilzmaxlk-i, -|lX<r)In this case, the output is identical with the network's state:

* In template design the condition to obtain the finalstable state X* is taken into consideration for every

cell where the steady state solution of the CNNsl stateequation is:

In principle, any of the 2D signal CNN interpolation methodscan be used for restoration of damaged or partially known image.Due to the used constraints for determining the methods forimage restoration, only those using linear feedback templates,with 3*3 dimension are first taken into consideration. Regardlessof the restoration, partially known or damaged image the usedtemplate is based on a smoothing process.

For two dimensional signal interpolation the following costfunctions [11] can be used:

(5) E2 = fJj('D-'r)2]dc z4"x

x,y(=Qf

(9)

where (Dr represents the r radius image from every (D pixel'sneighborhood;

E2 = Jf 2+ dxa~\21(6) axylL6x ) ( )j

(7) E3= ~XJ 02E +2( ) +I dxdy

(10)

(11)

The smoothing functionals can also be the total variation integralfrom the Rudin-Osher-Fatemi image denoising model [14]:

E4 = lV(D01) dxdy= TV (12)(8) x yen

EAijlyfi + YBjjlui + C+2jci + EDijj(uX, Y.)= x"CkXer= N , Ct eN, CM,eN,

* Cost or energy functions are used for complete analytictemplate design. Another relation that can help intemplate design is the global minimum search of thesecost functions.

* An important design step is the way how energyfunctions are associated to templates A, B, C or D.Taking the actual CNN chip's characteristics intoaccount, only A and B templates can be used.

* The weighting of cost functions with scalar parametersbetween 0 and 1, is necessary to satisfy condition (5).Obviously, it is recommended that the number ofimposed parameters at the beginning of the algorithmshould be as small as possible.

In CNN designing particularly the following facts should betaken into account:

* Using zero-flux condition, it is not necessary to knowinitial boundary conditions.

* It is necessary to use a mask image that does notchange the elements of the image that are known at thebeginning, but allow the computing of unknownelements. The existence of a mask image presumes thatthe user knows the positions of the elements that needto be computed. The portions that correspond to thedamaged regions have zero as the initial value. Theimage will be processed by (DIN, used at t = 0.

Based on cost functions minimization, templates containing onlyA feedback term resulted, the others being zero (B=0, z-0). Byusing these templates, CNN processing does linear, quadratic,spline-cubical interpolations, as well as image denoising,respectively: (13)

0.125F0.125 0.12 A 0 0.25 0Al = 0. 125 0 0.1 25 A2 = 0.25 0 0.25

P.125.0125.01251 0 i0.25 i0Ainpoll.tem Ainpol2.tem

-0.05 0.3 -0.05 0 a 0A3 = 0.3 ° 0.3 A4 = a O a

1-0.051 0.3 1-0.0-5-1 0 a I0

Ainpol3.tem

where a = Xsign(xi - x, ,XE [O, 1]-

osrufaA.tem

(14)

3. TESTING OF CNN IMAGEINPAINTING METHODS

In this section the experimental results obtained by using the"CadetWin" (CNN Application Development Environment andToolkit under Windows [15]) are presented.

Whatever the chosen image restoring method is, the precision ofthe restoration should be as good as possible, but at the same

time, it is desirable that the dimensions of the image's holes thatare restored should be as large as possible, that is theinterpolation propagation distance should be great. The above is

199

the evaluation basis of the methods proposed for the restorationof the damaged image.

Let us denote the error ER (precision criterion) which occuresbetween the restored image, D, and the original image (DO,(eventually obtained from an ideal interpolation). This is thereason why in the following examples, the damaged images O]Nwere obtained from original real images where the values ofsome elements (pixels) are zeroes. This way, the errors that resultform the use of different methods can be quantitatively measuredas:

M*N I l<iSM; ISjSN-

In figure 2 we may see the results obtained with the proposedCNN methods in the case of an image where the unknownelements are regularly placed in rows and columns (unknownregions are small, being only few-pixel-wide in a direction).Compared with the case of damaged images with large holes, inthis case the insufficient propagation problem does not occur.

(15)

Propagation distance (radius) can be quantitatively analyzed witherror images obtained by using the following relation:

(16)

In order to analyze the efficienicy of the proposed templates forrestoring images with large holes, a synthetic image was used(see figure 1).

Figure 2. a) Original image Oo; b) image to be restored(DIN; c) output image obtained with ainpoll.tem(ER1=0.0573); d) output image obtained withainpol2.tem (ER2=0.0507); e) output image obtainedwith ainpol3.tem (ER3=0.0492); f) output imageobtained with osrufaA.tem (ER4=0.0757); g) outputimage obtained with nel_ainpol3.tem (ER5=0.0497).

Figure 1. a) Original image (O; b) image to be restored(DIN; c) error image (DERI for ainpoll.tem (ERI=0.0217);d) error image (DER2 for ainpol2.tem (ER2=0.0555);e) error image (DER3 for ainpol3.tem (ER3=0.0822);f) error image 'DER4 for osrufaA.tem (ER4-=0.0086);g) error image OEMU for nel_ainpol3.tem (ER5=0.0041).

The two holes are placed in uniform noiseless regions, one withpositive value (0.70) and the other one with negative value(-0.70), the image's pixels having standard CNN domain values[, +1].

Analyzing the images from figure I and ER error values, it maybe observed that in the case of hole reconstruction osrufaAtemplate has the best behavior, the biggest errors being obtainedwith aintpol3 template.

Analyzing the images from Figure 2 and ER error values, it maybe observed that this time the situation is reversed, compared tothe first case from Figure 1: aintpol3 template has the bestbehavior, the biggest errors results from the use of osrufaAtemplate. On the other hand, the error that results from usingaintpol3.tem, aintpol2.tem, aintpoll.tem templates do not differessentially, if we take the CNN processing resolution intoconsideration. Practically, every procedure may be useful if thetype of the damaged image is known.

The efficiency of image restoration can be obtained from thecombination of the two methods, based on relation (4). Thesemethods provide a satisfying behavior for any image, even in oneof the limit-situations. The resulting template is as follows: (17)

0 a

D= a a

nel_ainpol3.tem

where a = Xsign(yij - Xk xE [0,], with (B 0, z=0).

200

jZ.(Oo(ij)-(D(ij))2X1-F.R

OER=O,(ij)-(D(ij), I<i<M; I<j<N.

-0.05 0.3 -0.05A = 0.3 0 0.3

-0.05 0.3 -0.05

By using this template, the results of damaged image restorationare presented in Figures Ig , 2g, and 3.

Figure 3. a) Original image (O; b) image to be restored(IN; c) output image (DouT using nel_ainpol3.tem(ERI=0.0217;. ERI=0.0216; ER2=0.0200; ER3=0.0194;ER4=0.0206; ER5=0.0189).

4. CONCLUSION

Complex mathematical models are available now, models thatcan be used in restoration ofdamaged or partially known images.These image-processing methods sometimes are difficult to beimplemented in real-time, even if a large serial processingcomputing power is available. Proposed methods using linearfeedback templates with 3*3 dimension, can be implementeddirectly on existing CNN. Due to parallel processing, hugecomputing power is achieved, regardless of the dimensions oftheimages. Increasing the efficiency is possible through combiningthese methods with a nonlinear template, which ensures localproperties spreading area growth, making it possible to processeven images with large holes.

[4] V. Casellas, J.M. Morel, G. Sapiro, A. Tannenbaum,"Introduction to the special issue on partial differentialequations and geometry-driven diffusion in imageprocessing and analysis", IEEE Trans. on ImageProcessing, Vol. 7, pages 269-273, 1998.

[5] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester,"Image inpainting", Computer Graphics (SIGGRAPH2000), pages 417-424, 2000.

[6] M.M. Oliveira, B. Bowen, R. McKenna, Y.S. Chang, "FastDigital image inpainting", Proceedings of the Int.Conference on Visualization, Imaging and ImageProcessing (VIIP 2001), pages 261-266, Marbella, 2001.

[7] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, andJ.Verdera, "Filling-in by joint interpolation of vector fieldsand gray levels", IEEE Trans. Image Processing, 10(1),pages 1200-1211, 2001.

[8] M. Bertalmio, A. Bertozzi, G. Sapiro, Navier-Stokes,"Fluid Dynamics, and Image and Video Inpainting",Proceedings of the 2001 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition(CVPR), pages 1355-1362, Hawaii, 2001.

[9] T. Chan, J. Shen, "Mathematical Models for LocalNontexture Inpaintings", SIAM Journal of AppliedMathematics, 62(3), pages 1019-1043, 2001.

[10] J. Shen, "Inpainting and the fundamental problem of imageprocessing," SIAMNews, 36(5), 2003.

[11] A. Gacsadi, P. Szolgay, "Interpolation of 2D signals usingCNN", Proceedings ofthe European Conference on CircuitTheory and Design, (ECCTD'01), pages I 349-352,Espoo, 2001.

[12] A. Gacsadi, P. Szolgay, "A variational method for imagedenoising, by using cellular neural networks", Proceedingsofthe Int. Workshop on Cellular Neural Networks and theirApplications, (CNNA '04), pages 213-218, Budapest, 2004.

[13] Cs. Rekeczky, T. Roska, A. Ushida, "CNN-baseddifference-controlled adaptive on non-linear image filters",Int. Journal of Circuit Theory and Applications, Vol. 26,pages 375-423, 1998.

[14] L. Rudin, S. Osher, E. Fatemi, "Nonlinear total variationalbased noise removal algorithms", Physica D, Vol. 60,pages 259-268, 1992.

[15] *** "CadetWin-CNN application development environmentand toolkit under Windows", Version 3.0, Analogical andNeural Computing Laboratory, Comp. and Aut. Inst.Hungarian Academy of Sciences, Budapest, 1999.

5. REFERENCES[1] L.O. Chua, L. Yang, "Cellular Neural Networks: Theory

and Applications", IEEE Trans. on Circuits and Systems,Vol.35, pages 1257-1290, 1988.

[2] T. Roska, L.O. Chua, "The CNN universal machine: an

analogic array computer", IEEE Trans. on Circuits andSystems, Vol. 40, pages 163-173, 1993.

[3] G. Linan, S. Espejo, R. Dominguez-Castro, E. Roca,A. Rodriguez-Vazquez, "CNNUM3: A mixed-signal 64x64CNN Universal Chip", Proceedings of the Int. Conf. on

Microelectronics for Neural, Fuzzy and Bio-inspiredSystems (MicroNeuro99), pages 61-68, Granada, 1999.

201