Simultaneous Structure and Texture Image Inpaintingby: Bertalmio, Sapiro, Vese, Osher

Presented by: Shane Brennan

June 7, 2007EE 264 – Spring 2007

What Is Inpainting?

Inpainting can also be used for other purposes including:

Humor Entertainment Improving aesthetic quality of images

And for less playful purposes…

How To Inpaint (yes you can do

it too!)

Identify the regions you want to fill/remove Continue any lines arriving at those regions Fill in the regions with texture/color from the

surrounding areas

Enjoy!

Image Inpaintingby: Marcelo Bertalmio and Guillermo Sapiro

Proceedings of SIGGRAPH 2000

The (very) Basic Idea

Form an iterative algorithm

Each update gets you closer to the desired result

The Update Image

Want to propagate “information” (aka lines) into the region being inpainted

Need information, and direction it is heading

Project information on that direction

Rate of Change of Information

Direction of Information

What Is Information?

Since want result to be smooth, use laplacian of the image to represent information.L = I * 2

The rate of change of the information entering a pixel is the derivative of the laplacian

= [Li,j+1- Li,j-1, Li+1,j – Li-1,j]

Propagation Direction

Tangent to the isophate line (which is the information) arriving at the boundary

Tangent easily computed as:

N = [-IY, Ix]

note: I use the (x,y) coordinate notation

Anisotropic Diffusion Areas where N = [0, 0] will never get updated! These

are smooth regions “Bleed” smooth regions into the region being inpainted Also ensures lines stay smooth and curved BUT! Very problematic! My form of diffusion:

The authors form: ???

Bad Result Due to My Diffusion

How About Texture?

Texture Synthesis by Non-Parametric Samplingby: Alexei Efros and Thomas Leung

IEEE International Conference on Computer Vision, Corfu, Greece, September 1999

How It Works

Take a pixel to be synthesized. Find which pixels near it have already been synthesized (or pre-existed). Define this to be the mask

For every pixel in the image (the candidates), compare the WxW neighborhood to the neighborhood around the pixel to be synthesized using a distance metric, but only on pixels defined by the mask

Keep either the K most similar neighborhoods, or the neighborhoods whose distance is less than T (W, K, and T are user-defined values)

Of the remaining regions, select one at random Assign the intensity of the center pixel of the selected region to

the pixel being synthesized

A Visualization…

How About Structure?

The Problem

Image inpainting works well on regions with structure, but not on regions with texture

Texture synthesis works well on regions with texture, but not on regions with structure

Early papers: for each pixel decide if structure or texture, and perform the appropriate filling method

But there is a better way…

Simultaneous Structure and Texture Image Inpaintingby: Bertalmio, Sapiro, Vese, Osher

IEEE Transactions On Image Processing, Vol. 12, No. 8, August 2003

How It Works

Decompose image into two parts: structure image and texture image

Perform inpainting on the structure image

Perform texture synthesis on the texture image

Recombine the two images to get final result

But how to decompose? no, not like that

Modeling Textures With Total Variation Minimization and Oscillating Patterns in Image Processing

by: Stanley Osher and Luminita Vese

Journal of Scientific Computing, Vol. 19, Nos. 1–3, December 2003

The Initial Version

Considered an image to be some underlying “real image” and then noise added. Want to remove the noise (Rudin, Ohser, and Fatemi. 1992)

Find structure image, u, that minimizes:

Smoothness Term Data Fidelity

Incorporating the Texture

No texture in there! But lets consider image to be f = u + v, where u is structure and v is texture. Note v = f – u

Solutions to this are known! (refer to report for some references on the solution)

Cutting to the Chase

Model texture as x and y components, call them g1 and g2

Some magic... (refer to the paper for more)

Smoothness on u Reconstruction Error Smoothness on v (remove noise)

An Iterative Solution

Break the optimization down into an iterative solution. Although more iterations aren’t always better, in practice need to play with # of iterations

Also need to play with tuning parameters. Though there are default values which work well in a broad array of images

Original Image (f)

Structure Image (u) Texture Image (v)

Original Image (f) Structure Image (u) Texture Image (v)

Back to Inpainting!

The Algorithm

Decompose Iin into sub-images u and v Perform inpainting on u to obtain U Perform texture synthesis on v to obtain V Iout = U + V

And now for some results…

Original Image Structure Image Texture Image

Completed Image Inpainted Structure Image Synthesized Texture Image

Original Image Structure Image Texture Image

Completed Image Inpainted Structure Image Synthesized Texture Image

Original Image Structure Image Texture Image

Completed Image Inpainted Structure Image Synthesized Texture Image

References

M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. 2000.

A. A. Efros and T. K. Leung. Texture synthesis by non-parametric sampling. In ICCV (2), pages1033–1038, 1999.

M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. Simultaneous structure and texture image inpainting, 2002.

L. Vese and S. Osher. Modeling textures with total variation minimization and oscillating patterns in image processing, 2002.