Image Inpainting Using Vanishing Point Analysisand Scene Segmentation

Eman T. Hassan∗, Hazem M. Abbas†, Hoda K. Mohamed∗, M. Watheq El-Kharashi∗∗Department of Computer and Systems Engineering

Ain Shams University, Cairo 11517, Egypteman.tarek-ibn-zeyad, hoda.korashy, watheq.elkharashi@eng.asu.edu.eg

†Faculty of Media Engineering and TechnologyGerman University in Cairo, Egypt

hazem.abbas@guc.edu.eg

Abstract—A new image inpainting technique is developed tofit perfectly for special categories of images that contain mainlybuildings. This technique handles the need to obtain an imageof building free from parking cars along sides of the roads. Todo that, one needs to carefully inpaint the roads and the missingparts of images. This can be done by combining vanishing pointsdetections and image segmentation. After detecting vanishingpoints, it is possible to draw the line dividing the missing regionsinto two parts: the road part and the building part. Each partshould be then inpainted independently using a different sourceregion for each one. A segmentation technique, which is basedon color and texture features, is employed to extract a sourceregion for road part inpainting. Simple geometric calculations areemployed to detect the source region for building part inpainting.

I. INTRODUCTION

Image completion, also known as image inpainting, is theprocess of filling the missing regions of an image so that theoutput image appears as if there were no missing regions.This is useful for image retouching and repairing in areas likecomputer graphics, film postproduction and image restoration.This task is a bit challenging in the area of computer visionand that is why it has attracted many research efforts.

Image inpainting problem can be expressed formally asfollows : given an image I with missing region, Ω, it isrequired to fill each pixel in Ω with a value from a regionnamed source region, φ, so that φ ⊆ I\Ω. Many techniqueshave been developed to perform the needed inpainting thatcan be classified into two main categories. The first cate-gory employs the partial differential equation (PDE) methods.Employing PDE in image inpainting was first proposed in[1]. It uses a diffusion process to fill the missing regionΩ. Information is propagated smoothly from the boundaryof Ω into its interior in the direction of the isophotes. Thisdiffusion process is simulated through solving a high orderPDE. The recovered results are highly smooth in the caseof small missing region Ω. However, when Ω becomes large,this technique produces blurry artifacts that lack texture. Thework of [1] has inspired the development of a total variational(TV) inpainting model presented in [2], [3]. In [3], the TVinpainting model employs Euler Lagrange equation inside Ωwhich uses anisotropic diffusion based on the contrast of theisophotes for filling Ω. The curvature driven diffusion (CDD)model is proposed in [2] that extends the TV algorithm. It

considers geometric information of isophotes when definingthe strength of the diffusion process, thus allowing the CDDmethod to proceed over larger areas. Wavelets is combinedwith variational methods in [4] to eliminate blurry edges. In[5] poisson equation is employed to solve the problem oflarge missing region Ω, since the PDE based methods cannotmaintain texture detail in large Ω and often produces blurryartifacts.

The second category is exemplar based approaches. Thesetechniques try to fill the missing region Ω by copying thecontents from another existing part of the image. The most fa-mous exemplar-based image completion method was proposedin [6]. The source region propagates along linear isophotesand the order of filling the pixels inside Ω is determinedby the structure information of the image. This makes theinpainting reasonable regardless of the boundary shape ofΩ. There are several improvement of the technique proposedin [6]. In [7], the energy of wavelet coefficients are usedfor determining region filling order and propagation. Fillingpriority and similarity measure techniques, that are consistentwith human visual judgement, are proposed in [8]. This isbased on a proposed edge term and the energy of distributedcosine transform coefficients DCT of a patch. A new methodwhich is based on color distribution analysis on the boundaryof Ω is proposed in [9] so that the assignment of a high priorityto the structures enables the salient structures to be consistentwith the surrounding regions. In [10], the exemplar-based in-painting is formulated as a global energy optimization problemwhere multiscale graph cuts method is employed to obtaina global minimum. A new inpainting model, presented in[11], employs various techniques to enhance the performance.Copy-and-paste texture synthesis, geometric partial differentialequations (PDEs), and coherence among neighboring pixelsare combined in a variational model to provide a workingalgorithm for image inpainting.

In this paper, a new inpainting algorithm is proposed whichtakes advantage of the unique features of every scene typeand employ these features to enhance the inpainting algorithmfor that type. The algorithm employs the estimated vanishingpoints in images of a specific scene type to effectively fillthe missing region. The estimated vanishing points help todivide the inpainting problem into two independent inpaintingproblems. With the help of segmentation and geometric cal-culations, this ensures the correct propagation of texture and

structure in the missing region. The experimental results showthe effectiveness of this algorithm, since it is designed to usethe main scene characteristic effectively to produce better andplausible output.

The rest of the paper is organized as follows: The proposedalgorithm is explained in section II where the motivation be-hind the idea of developing inpainting techniques for differenttypes of scenes is explained. The detection of vanishing pointsis explained in section III. This is followed by describing Themain characteristics and detection of the scene type handledby of the proposed technique is described in section IV. Insection V, the main steps of the proposed inpainting algorithmare shown. The experimental results in section VI demonstratethe effectiveness of the proposed algorithm. Conclusions andfuture work are shown in sections VII respectively.

II. THE PROPOSED ALGORITHM

Developing an inpainting technique that gives high perfor-mance for different sizes of the missing region or differenttypes of image scenes is very challenging. Each image scenehas its own main dominant characteristics. These characteris-tics can be very useful in the inpainting algorithm. ConsiderFigure 1 that shows different types of image scenes. If wewant to remove the humans from the beach scene in Figure1(a), one may apply either the techniques in [6] or in [1]which will produce very good performance. This is due tothe small missing region and the less complicated structuralinformation in the image. However, these techniques will givepoor results if the parking cars from the image shown inFigure 1(b) need to be removed. The main important featureof this image is the perspective projection of the parallellines of the building that intersect in a vanishing point. Ifthese features are carefully used to formulate an inpaintingalgorithm to these particular types of images, it is expectedthat a far better performance will be achieved. Similarly, theimage in Figure 1(c) contains mainly overlapping faces. Toremove one face we need to re-synthesise the missing face part.This requires developing a specific scene inpainting algorithm.Hence, the basic idea proposed here is to develop a scenespecific inpainting algorithm, and in turn develop a classifier toclassify the scene into a specific category and use its inpaintingalgorithm accordingly.

In this work, an inpainting algorithm for scenes similar tothose shown in Figure 2 is proposed. In these scenes, it isrequired to remove parking cars. The most dominant featureand challenge of this scene type is perspective projection, roadand building identification and inpainting.

To illustrate the idea, in Figure 3, it is required to removethe parking cars from the scene and fill the missing region.To do that efficiently, the road and building into the missingregion are to be propagated. The obvious characteristics ofthese scenes are the parallel lines of the street and the buildingthat intersect into an imaginary vanishing point due to theperspective projection in the image. That is why propagationin the missing region should be consistent with this feature.The proposed technique capitalizes on this feature to producea plausible and consistent inpainted output.

The algorithm can be summarized as follows :

(a) (b)

(c)

Fig. 1: Removing objects using Image Inpainting. (a) Removehuman from the beach scene. (b) Romve cars from the image.(c) Remove the face of one child in the image.

(a) (b)

Fig. 2: Building scenes

(a) (b)

Fig. 3: Removing objects using Image Inpainting. (a) Theoriginal image (b) Mask image to show the area that needsto be inpainted

Fig. 4: Missing Regions : Ω1 and Ω2. Source Regions: φ1 andφ2

1) Estimate the vanishing points of the image using theimage perspective projection. This will be described insection III.

2) Use the estimated vanishing point to divide the missingregion into two parts Ω1 and Ω2 representing the missingroad and building part, respectively, as shown in Figure4. This process is detailed in Section III.

3) Inpaint each missing region Ω1 and Ω2 independentlyusing the examplar-based technique [6]. This will beexplained further in V. To improve the performance, weneed to extract source regions, φ1 and φ2, that will beused to fill in the missing regions, Ω1 and Ω2, respec-tively, as depicted in Figure 4.

4) Obtain the first source region, φ1, which represents theexisting road part in the image. by employing texturalsegmentation as explained in IV-A.

5) Obtain the second source region, φ2, which represents thebuilding part by applying simple geometric calculationsoutlined in IV-B.

III. PERSPECTIVE PROJECTION AND VANISHING POINTSDETECTION

A camera is an image acquisition system that transforms areal 3D space into 2D image space. It can be represented byapin-hole model shown in Figure 5. The pinhole lies betweenthe imaging screen and the observed 3D world scene and anyray of light that is emitted or reflected from a surface patch inthe scene is constrained to travel through the pinhole beforereaching the imaging screen. A mathematical model of theperspective projection through a simple pin-hole camera isillustrated in Figure 6 [12]. The solid angle of rays that issubtended by the pinhole relates the field of view of eachregion on the imaging screen to the corresponding regionimaged in the world. Therefore, a point in 3D with coordinates(x0, y0, z0) is related to the image point (X,Y ) as follows :

Y =f

z0y0, X =

f

z0x0 (1)

where the image plane is parallel to axes x and y and islocated at distance f from the origin (center of projection)in the negative direction of the z axis.

Fig. 5: Pin hole Camera model

Fig. 6: Mathematical model for Perspective Projection

Under perspective projection, straight lines in 3-D areprojected as straight lines in the 2-D image plane. Parallellines in the 3-D are projected into non-parallel lines in theimage that intersect in a vanishing point. Figure 7(a) showsthree group of lines colored by red, green and blue. Each setof lines represents lines that are parallel in 3D but not in theimage. Rather they intersect in a vanishing point. Figure 8shows the estimated vanishing point for each group of lines.Figure 7(b) shows another example of parallel lines groups,red and blue, and each group has its own vanishing point.

(a) (b)

Fig. 7: Lines classes for vanishing point detection. Each colorof the lines represents a class of lines that are parallel in thereality and intersecting into the vanishing point in the imagedomain

Fig. 8: The Vanishing point of the image

For vanishing point detection, the methods proposed in[13]–[15] are employed here. First, line segments are extractedfrom the input image by using the technique proposed in [15].Then each lines is classified to a specific group where eachgroup of lines has specific features. A tailored agglomerativeclustering, called J-linkage, is employed to line group thatbelongs to the same model [13], so that lines can be assignedto different groups as red, green or blue line groups (Figure7). Finally A non-iterative approach is used to estimate thevanishing points for each group of lines [14]. This techniqueestimates a vanishing point for each group of lines thatmaximizes the measure of consistency between a vanishingpoint and the lines of that group.

Now the building/road boundary in the missing region canbe considered as the line connecting the vanishing point andthe geometrical center of the missing region. That line belongsto the same group of lines of this particular vanishing point. Italso divides the missing region into two parts: Ω1 for missingroad part and Ω2 for the missing building part as shown inFigure 4. The equation of this line is expressed as y = mx+c,where x ∈ [xmin, xmax] and y ∈ [ymin, ymax], and thusthe slope, m, and intercept, c, can be found. Eventually,the inpainting problem can be divided into two independentinpainting problems where each one has its own missingregion Ω and source region φ. The following subsections IV-Aand IV-B explain how to define source regions φ1 and φ2

respectively.

IV. SCENE ANALYSIS: ROAD AND BUILDING DETECTION

A. Road Area identification

A texture segmentation technique is applied to detect roadregion. This technique proceeds as follows:

1) Use K-means algorithm to perform color segmentation[16].

(a) Original Image (b) Segmentation result

(c) The identified road part

Fig. 9: Shows the road area identification after image segmen-tation

2) Region merging: pair of regions are merged if they areoverlapped by more than 50% of their area, after applyingdilation operation to each region.

3) Get Gabor texture coefficient for merged regions [17]. UseK-means clustering algorithm to segment each mergedregion.

Figure 9 shows the segmentation result of the technique andthe road region detection. This region is used as a source regionto inpaint the missing road region part.

B. Building Part Identification

The line, that identifies the road/building boundary in themissing region, has the equation of y = mx + c, where xhas a range of xmin to xmax and y has a range of ymin toymax, where the equation parameters has been calculated asexplained in subsection III. The source region φ2 is defined asthe region where y −mx − c < 0 for the defined range of xand y in the existing part of the image (Figure 4).

V. INPAINTING

The exemplar-based inpainting technique proposed in [6] isemployed to propagate texture and structure into the missingregion. There are two independent inpainting problems: thefirst one is to propagate the road into the missing regionΩ1 where the source region is φ1 obtained by the techniquedescribed in IV-A. The second problem is to propagate thebuilding into the missing region Ω2 where the source regionis φ2 obtained as in IV-B.

Let us define the source region φ where φ ∈ φ1, φ2,and the missing region to be Ω where Ω ∈ Ω1,Ω2. Eachpixel on the source has a confidence term, C(.), such that,

(a) Source Region and TargetRegion

(b) Selected patch to be ina-painted

Fig. 10: Image Inpainting algorithm

during initialization, C(p) is assigned to C(p) = 1∀(p ∈ φ)and C(p) = 0∀(p ∈ Ω) as shown in Figure 10(b). The stepsof exemplar-based inpainting algorithm [6] can be summarisedas follows :

1) Identify the filling front ∂Ω , and exit if ∂Ω = 0 as shownin Figure 10(a).

2) Compute (or update) the priorities of every pixel p onthe filling front ∂Ω by P (p) = C(p)D(p) defined in animage block Ψ centered at p and shown in Figure 10(b).C(p) and D(p) are the confidence term and the data termrespectively which are defined as :

C(p) =

∑q∈Ψp

⋂−Ω C(q)

|Ψp|, D(p) =

| 5 I⊥p .np|α

(2)

where |Ψp| is the area of Ψp, α is the normalization factor,np is the unit normal vector which is orthogonal to thefront ∂Ω at p, −Ω is the binary complement of Ω , and5I⊥p is the isophote at p.

3) Find the patch which has the maximum priority amongall patches centered at the filling front ∂Ω.

4) Find the exemplar patch in the source region which bestmatches the query image block. The best matched imageblock is a patch in the source region which minimizesthe Sum of Squared Differences (SSD) between the querypatch at p and the source patch.

5) Copy image data from source patch to missing region.6) Update the priority term. Go to the first step.

VI. EXPERIMENTAL RESULTS

To verify the efficiency of our method and the quality ofoutput images, we compared our results to some exemplar-based inpainting method in [6] and [18]. Results generatedby the proposed approach and these two approaches areshown in Figures 11. It is clear from the Figure that theproposed image completion method has correctly drawn theroad region according to the perspective projection detectedfrom the image. Then it inpaints the road and building regionsseparately. On the other hand, the other techniques have failedto correctly propagate the road and the building structure toproduce a consistent result. The same implementation andcomparisons using two other test images and the producedinpainted results are shown in Figures 12,13. This shows theefficiency of our proposed technique.

(a) The original image (b) The result of our algorithm

(c) The result of algorithm in [6] (d) The result of algorithm in[18]

Fig. 11: Removing parking cars using proposed and otheralgorithms

(a) The original image (b) The result of our algorithm

(c) The result of algorithm in [6] (d) The result of algorithm in[18]

Fig. 12: Removing parking cars using Image Inpainting

(a) The original image (b) The result of our algorithm

(c) The result of algorithm in [6] (d) The result of algorithm in[18]

Fig. 13: Removing parking cars using Image Inpainting. (a)The original image (b) The result of our algorithm (c) Theresult of algorithm in [6] (d) The result of algorithm in [18]

VII. CONCLUSION

An inpainting technique is proposed and developed. Itworks efficiently on this type of scenes which are characterizedby their parallel lines in the 3D while they are not parallelin the image but rather intersect in a vanishing point. Thisalgorithm extracts information about the perspective projectionfrom the image represented by vanishing points detection.The estimated values of the vanishing points are then usedto divide the missing region into two parts in such a waythat the road part and building part will propagate into themissing region independently. To enhance the performanceof the propagation inside the missing regions, the sourceregion for each part is extracted. A segmentation techniqueis employed to extract the road source region, while a simplegeometry is employed to identify the building source region.The experimental results show that propagation of road andbuilding independently produces result that is consistent withthe perspective projection in the image.

This work will be further extended to build a generalframework that takes an input image and classify it into aspecific scene type. Accordingly the most efficient inpaintingalgorithm that is suitable to this type of scenes will beapplied. We need to define suitable features descriptors forthat problem and develop a classifier with high performancein scene classifications.

REFERENCES

[1] V.Caselles M.Bertalmio, G.Sapiro and C.Ballester. Image inpainting. InComputer Graphics (SIGGRAPH00), page 417424, Singapore, 2000.

[2] Tony F. Chan and Jianhong Shen. Mathematical models for localnontexture inpaintings. SIAM J. Appl. Math, 62:1019–1043, 2002.

[3] Tony F. Chan and Jianhong Shen. Non-texture inpainting by curvature-driven diffusions (cdd). J. Visual Comm. Image Rep, 12:436–449, 2001.

[4] Julia A. Dobrosotskaya and Andrea L. Bertozzi. A wavelet-laplacevariational technique for image deconvolution and inpainting. IEEETransactions on Image Processing, 17:657–663, 2008.

[5] Chuan Zhoua Jianbing Shena, Xiaogang Jina and Charlie C.L. Wangb.Gradient based image completion by solving the poisson equation.Computers & Graphics, 31:119126, 2007.

[6] Antonio Criminisi, Patrick Prez, and Kentaro Toyama. Region fillingand object removal by exemplar-based image inpainting. IEEE Trans-actions on Image Processing, 13:1200– 1212, 2004.

[7] C.R. Jung U.A. Igncio. Block-based image inpainting in the waveletdomain. Vis. Comput., 23:733–741, 2007.

[8] V.Sadasivam D.Jemi Florinabel, S.Ebenezer Juliet1. Combined fre-quency and spatial domain-based patch propagation for image com-pletion. Computers & Graphics, 35:10511062, 2011.

[9] Qing Zhang and Jianjun Lin. Exemplar-based image inpainting usingcolor distribution analysis. J. Inf. Sci. Eng., 28:2012, 641-654.

[10] Vicent Caselles Yunqiang Liu. Exemplar-based image inpainting usingmultiscale graph cuts. IEEE Transactions on Imaged Processing,22:1699 – 1711, 2013.

[11] Aurelie Bugeau, Marcelo Bertalmıo, Vicent Caselles, and GuillermoSapiro. A comprehensive framework for image inpainting. Trans. Img.Proc., 19(10):2634–2645, October 2010.

[12] Boguslaw Cyganek and J. Paul Siebert. An Introduction to 3D ComputerVision techniques and algorithms. John Wiley & Sons, Ltd, 2009.

[13] A. Fusiello R. Toldo. Robust multiple structures estimation with j-linkage. In European Conference on Computer Vision(ECCV), 2008.

[14] Tardif J.-P. Non-iterative approach for fast and accurate vanishing pointdetection. In 12th IEEE International Conference on Computer Vision,2009.

[15] J. Morel G. Randall G.Grompone, J.Jakubowicz. Lsd: A fast linesegment detector with a false detection control. IEEE Transactionson Pattern Analysis and Machine Intelligence, 23:722 – 732, 2010.

[16] S. Ranka K. Alsabti and V. Singh. An efficient k-means clusteringalgorithm. In Proc. First Workshop High Performance Data Mining,1998.

[17] D.Gabor. Theory of communication. Journal of the Institution ofElectrical Engineers, 93:429–457, 1946.

[18] Anupam, Pulkit Goyal, and Sapan Diwakar. Fast and enhancedalgorithm for exemplar based image inpainting. Image and VideoTechnology, Pacific-Rim Symposium on, 0:325–330, 2010.