A Perspective on Image Inpainting

IJSTE - International Journal of Science Technology & Engineering | Volume 1 | Issue 12 | June 2015 ISSN (online): 2349-784X

All rights reserved by www.ijste.org

260

A Perspective on Image Inpainting

Gaurav Chouhan Mandeep Singh Saini

M.E. Student Assistant Professor

Department of Electronics Communication Engineering Department of Electronics Communication Engineering

LR College of Engineering and Technology, Solan (H.P.)-

173223

LR College of Engineering and Technology, Solan (H.P.)-

173223

Abstract

Removing the uninterested objects or filling in the contaminated portion of an image represent an important photo editing

cumbersome task. Image Inpainting is the process filling in the missing areas or modifying the damaged images. Inpainting

algorithm have numerous applications. It is helpfully used for restoration of old films and object removal in digital photographs.

It is also applied to red-eye correction, super resolution, compression etc. The main goal of the Inpainting algorithm is to modify

the damaged region in an image in such a way that the inpainted region is undetectable to the ordinary observers who are not

familiar with the original image. In this paper a comprehensive review of different techniques for the image inpainting like

diffusion based inpainting, PDE based image inpainting, exemplar based image inpainting, and texture synthesis based image

inpainting has been discussed.

Keywords: Inpainting, Diffusion, Exemplar, PDE, Image Restoration

________________________________________________________________________________________________________

I. INTRODUCTION

The modification of images in a way that is non-detectable for an observer who does not know the original image is a practice as

old as artistic creation itself [3]. The filling-in of missing information is a very important topic in image processing, with

applications including image coding and wireless image transmission (e.g. recovering lost blocks), special effects (removal of

objects) and image restoration (scratch removal). The basic idea behind the algorithms that have been proposed in the literature is

to fill-in these regions with available information from their surroundings [1]. Digital techniques are starting to be a widespread

way of performing inpainting, ranging from attempts to fully automatic detection and removal of scratches in film [3]. Efros and

Freeman [18] present an approach in which texture synthesis is performed using blocks, not pixel by pixel, which significantly

reduces the execution time. The algorithm has proven to be more efficient by copying an entire block when a valid candidate is

found in the source. A digital image inpainting using cellular neural network was proposed by Elango and Murugesan [12]. CNN

has been widely used for image processing, pattern recognition, moving object detection, target classification, signal processing,

augmented reality and solving partial differential equations etc. Kokaram et al. [19] use motion estimation and autoregressive

models to interpolate losses in films from adjacent frames. The basic idea is to copy into the gap the right pixels from

neighboring frames. Hirani and Totsuka [11] combine frequency and spatial domain information in order to fill a given region

with a selected texture. This is a very simple technique that produces incredible good results. On the other hand, the algorithm

mainly deals with texture synthesis (and not with structured background), and requires the user to select the texture to be copied

into the region to be inpainted.

II. DIFFERENT APPROACHES OF IMAGE INPAINTING

Nowadays, there are different approaches to image inpainting are available and can classified into several categories as follows:

1) Diffusion based inpainting 2) Texture Synthesis based Inpainting. 3) PDE based Inpainting 4) Exemplar based Inpainting 5) Convolution based Image Inpainting

Diffusion Based Inpainting A.

Diffusion based Inpainting was the first digital Inpainting approach. In this approach missing region is filled by diffusing the

image information from the known region into the missing region at the pixel level [7]. The problem is usually modeled by

Partial Differential Equation (PDE), so sometimes it is called a PDE-based approach. The diffusion- based Inpainting algorithm

produces superb results or filling the non-textured or relatively smaller missing region. The drawback of the diffusion process is

it introduces some blur, which becomes noticeable when filling larger regions [13].

A Perspective on Image Inpainting (IJSTE/ Volume 1 / Issue 12 / 043)


261

Texture Synthesis Based Inpainting B.

Texture synthesis based algorithms are one of the earliest methods of image inpainting. These algorithms are used to complete

the missing regions using similar neighbourhoods of the damaged pixels. The texture synthesis algorithm synthesize the new

image pixels from an initial seed and then strives to preserve the local structure of the image. All the earlier inpainting techniques

utilized these methods to fill the missing region by sampling and copying pixels from the neighbouring area and new texture is

synthesized by querying existing texture and finding all similar neighbourhoods. These synthesis based techniques perform well

only for a select set of images where completing the hole region with homogenous texture information would result in nature

completion [10]. Main difference between different texture based algorithms is how they maintain continuity between holes pixel and original image pixels [14]. Yamauchi et.al presented algorithm which generate texture under different brightness

condition and work for multi resolution [15]. Texture synthesis approaches can be categorized into three categories [5]:

Statistical (parametric)

Pixel-based (non-parametric)

Patch-based (non-parametric)

Fig. 1: Sample Texture Image And A New Image Is Being Synthesized One Pixel At A Time [5].

As shown in Figure1, to synthesize a pixel, the algorithm first finds all neighborhoods in the sample image (boxes on the left)

that are similar to the pixels neighborhood (box on the right) and then randomly chooses one neighborhood and takes its centre to be the newly synthesized pixel. All texture based methods are different in terms of their capacity to generate texture with

different color, intensity, gradient and statistical characteristics. Texture synthesis based inpainting method not performs well for

natural images. These methods not handle edges and boundaries well [14].

PDE based Inpainting: C.

PDE algorithm aims to restore degraded images that are affected by both noise and missing zones. This algorithm is the iterative

algorithm [2]. First PDE base approach was given by Bertalmio et.al [3]. The main idea behind this algorithm is to continue

geometric and photometric information that arrives at the border of the occluded area into area itself. This is done by propagating

the information in the direction of minimal change using isophote lines. This algorithm will produce good results if missed

regions are small one. But when the missed regions are large, the algorithm take long time and it will not produce good results

[7]. There are many ways to get the nonlinear PDEs. In image processing and computer vision it is very common to obtain them

from some variational problems. The basic idea of any variational PDE technique is the minimization of a energy functional. The

variational approaches have important advantages in both theory and computation, compared with other methods. An influential

variational denoising and restoration model was developed by Rudin et.al [8]. Their technique, named Total Variation (TV)

denoising, is based on the minimization of the TV norm and is remarkably effective at simultaneously preserving boundaries

whilst smoothing away the noise in flat regions. This model uses Euler-Lagrange equation and anisotropic diffusion based on the

strength of the isophotes. Another PDE based technique known as vector valued regularization under anisotropic diffusion

framework presented by Tschumperle et.al [16]. But the drawback of this method is that this method neither connects broken

edges nor handled the pixels on edges properly.

Exemplar Based Inpainting D.

The exemplar based approach is an important technique of inpainting algorithms. Exemplar-based Inpainting approach

iteratively synthesizes the target region by most similar patch in the source region. Basically it consists of two basic steps: in the

first step priority assignment is done and the second step consists of the selection of the best matching patch. The exemplar based

approach samples the best matching patches from the known region, whose similarity is measured by certain metrics, and pastes

into the target patches in the missing region [10]. This method is an efficient approach for reconstructing large target regions.

These algorithms also overcome the drawbacks of PDE based inpainting. Limitations of this technique are: (i) the synthesis of

regions for which similar patches do not exist does not produce reasonable results (ii) the algorithm is not designed to handle

curved structures, (iii) it does not handle depth ambiguities (i.e., what is in front of what in the occluded area?) [17].



262

Fig. 2: Removing Large Objects from Images. (A) Original Photograph. (B)

The region corresponding to the foreground person has been manually selected and then automatically removed [17]. In Figure

2, the horizontal structures of the fountain have been synthesized in the occluded area together with the water, grass and rock

textures.

Convolution based Image Inpainting E.

Convolution based image inpainting algorithms are very rapid, however in many cases, they don't provide adequate results in

sharp details such as edges. In convolution based image inpainting, the mask coefficients are calculated using the gradient of the

image to be inpainted. The algorithm is fast, iterative, simple to implement, and provides very adequate results [4]. A

convolution-filter based algorithm inpaint an image by convolving the neighborhood of the damaged pixels with a proper kernel.

Convolving the image with the averaging filter to compute the weighted averages of pixels neighborhoods is the same as isotropic diffusion.

Oliveira et al. [9] presents a fast image inpainting algorithm which uses convolution operation. In their algorithm, the regions

to be inpainted are convolved with a predefined diffusion mask repeatedly. This model is very similar to isotropic diffusion. In

this method, the central weight of the diffusion mask is considered zero, because its related pixel in the original image is

unknown. The values of a, b, and for both kernels are 0.073235, 0.176765, and 0.125, respectively [9].

Fig. 3: The Convolution Kernels Proposed By Oliviera Et Al. [19].

Hadhoud et al. present a modification of Oliveira algorithm permitting an implementation time reduction [6]. Both above

convolution based algorithms are very quick but they don't have very good results in high contrast damaged edges.

III. CONCLUSION

Image inpainting is an important part of image restoration, which aims to automatically restore missing information of image

according to information around damaged region. It is mainly used for heritage conservation, restoration of old photographs,

removal of occlusions and so on. In this paper a variety of image Inpainting techniques such as texture synthesis based

Inpainting, PDE based Inpainting, Exemplar based Inpainting are studied. For each technique a detailed explanation of the

techniques can be given which are used for filling the missing region building use of image. It is experiential that the PDE based

Inpainting algorithms cannot fill the large missing region and it cannot renovate the texture pattern. The analysis proved that the

exemplar based image Inpainting create better results for Inpainting the huge missing region. This algorithm can inpaint both the

formation and textured image efficiently. But it will work well only if the missing region contains only simple structure and

texture. The convolution algorithm is adaptive and uses gradient to calculate weights of convolving mask at each position.

Results shows that the algorithm is fast, iterative, simple to implement and provides as well results as structure algorithms.



263

REFERENCES

[1] M. Bertalmio, Luminita, V., Guillermo, S. and Osher S. ( 2003). Simultaneous Structure and Texture Image Inpainting. IEEE Transactions on Image Processing. 12 (8): pp. 882-889.

[2] T. Barbu, Ciobanu, A. and Luca, M. (2014). PDE-based Image Restoration using Variational Denoising and Inpainting Models, Institute of Computer Science Romanian Academy Iasi, Romania.

[3] M. Bertalmio, Guillermo, S., Caselles, V. and Ballester C. (2000). Image Inpainting, Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, USA.

[4] H. Noori, Saryazdi, S., Nezamabadi-pour H. (2010). A Convolution Based Image Inpainting. International Conference on Communication Engineering, pp. 130-134.

[5] A. Alexei and Thomas, K. L. (1999). Texture Synthesis by Non-parametric Sampling IEEE International Conference on Computer Vision. [6] M. Mohiy, Hadhoud, K. A. Moustafa and Sameh. Z. Shenoda, Digital Images Inpainting using Modified Convolution Based Method, International

Journal of Signal Processing, Image Processing and Pattern Recognition, pp. 1-10. [7] N. Pandya and Pandya M., (2014). An Image Inpainting Using Patch-Based Synthesis via Sparse Representation. International Journal of Science

Technology & Engineering, 1(1): pp. 11-14.

[8] L.I. Rudin, Stanley, O. and Fatemi E. (1992). Nonlinear total variation based noise removal algorithms.Elsevier Science Publishers, pp. 259-268. [9] M. M. Oliveira, Bowen B., McKenna, R. and Chang Y.S., (2001). Fast Digital Image Inpainting. International Conference on Visualization, Imaging and

Image Processing.

[10] K. Mahajan, and Vaidya M.B, (2012). Image in Painting Techniques: A survey. IOSR Journal of Computer Engineering, 5 (4): pp. 45-49. [11] A. N. Hirani and Totsuka, T. (1996). Combining Frequency and Spatial Domain Information for Fast Interactive Image Noise Removal, ACM. [12] P. Elango and Murugesan, K. (2009). Digital Image Inpainting Using Cellular Neural Network, International Journal of Open Problems in Computer

Science and Mathematics, 2 (3): pp. 439-449. [13] P. Jain and Mishra, A. (2015). A Review of Image Inpainting Techniques. International Journal of Advanced Research in Computer Science and Software

Engineering, 5(2): pp. 604-606. [14] P. Patel, Prajapati A. and Mishra, S. (2012). Review of Different Inpainting Algorithms. International Journal of Computer Applications, 59 (18): pp. 30-

34.

[15] H. Yamauchi, Haber, J. and Seidel, H. P. (2003). Image restoration using multiresolution texture synthesis and image inpainting, ACM Transactions in Computer Graphics International, pp. 120-125.

[16] D. Tschumperl and Deriche, R. (2005). Vector-valued Image Regularization with PDE's : A Common Framework for Different Applications, IEEE Transactions on Pattern Analysis and Machine Intelligence. 27 (4): pp. 506-517.

[17] A. Criminisi, Perez P. And Toyama, K. (2004). Region Filling and Object Removal by Exemplar-Based Image Inpainting, IEEE Transactions on Image Processing, 13 (9): pp. 1-13.

[18] A. A. Efros and Freeman, W.T. (2001). Image Quality for Texture Synthesis and Transfer, 28th Annual Conferrence on Computer Graphics and Interactive Techniques, pp. 341-346.

[19] A. C. Kokaram, Morris, R.D. Fitzgerald, W.Z. and Rayner, P.J.W. (1995). Interpolation of missing data in image sequence, IEEE Transactions on Image Processing, 4 (11): pp. 1509-1519.

Documents

A Perspective on Image Inpainting