Image Quilting for Texture SynthesisImplementing Techniques from (Efros & Freeman, 2001)

Sam Griesemer samgriesemer@wustl.edu

Washington University in St. Louis

Abstract

Texture synthesis is a process by which largeimages are constructed from smaller texturesamples. In this project we implement mul-tiple techniques used in (Efros & Freeman,2001) as simple yet effective improvementsover basic tiling techniques. We outline thesemethods in detail and took a closer look athow and why they work. Finally, we ana-lyze these methods and their performanceon textures of varying complexity, as well astheir sensitivity across a number of settings.

1. Introduction

Image synthesis encompasses a broad class of algo-rithms that are capable of creating new images. Manytechniques address specific, constrained scenarios (e.g.texture synthesis), while others address larger, moregeneral problems (e.g. synthesizing high-res naturalimages). These approaches vary wildly, from sim-ple random tiling to sophisticated generative modelslike GANs. In this project we focus on texture syn-thesis, the process of producing from a small texturesample a larger, perceptually similar image. Thereare many techniques for approaching such a prob-lem, each with varying degrees of success in termsof output quality and efficiency. For example, sim-pler methods are often more efficient but suffer frompoor output quality due to strong assumptions aboutthe nature of textures. More complex methods of-ten do a better job at producing high-quality outputs(thanks to more flexible assumptions and model fit-ting), but can be very inefficient and thus difficult touse in practice. As a result, the primary challenge isfinding a texture synthesis algorithm that makes aneffective tradeoff between efficiency and output qual-ity.

Texture synthesis is an important problem with a num-ber of practical applications across computer vision.

These include texture mapping, inpainting, image de-noising, and more. Texture mapping may be consid-ered the canonical application of texture synthesis,whereby a texture sample is mapped to a much largerarea while retaining similar visual qualities. Such aprocess could also be used to “cover up” or removeobjects from images using the surrounding texturecontext. This example overlaps with the goal of im-age inpainting, where damaged or missing parts of animage are restored or “filled in” using nearby texturecontext.

For this project we explore three texture synthesis meth-ods discussed in (Efros & Freeman, 2001). These meth-ods include simple random patch selection, a con-strained overlapping patch search, and the so-called“minimum error boundary cut” method introducedby the authors. We seek to implement each of thesemethods and replicate the paper’s texture synthesisresults. We then describe each of the methods in de-tail, and explore their performance in a variety of sit-uations. As a result, the primary goal of this projectis to provide an efficient, working implementation ofthe paper’s proposed methods, expand on the detailsof these algorithms in an intuitive way, and better un-derstand the situations in which these methods per-form successfully.

2. Related Work

Due to its implications and practical uses, texture syn-thesis has received a fair amount of attention over thepast few decades. As a result, a wide variety of algo-rithms and approaches have been published in thistime. There are far too many to address here, buta few notable classes of approaches such as physicalsimulations are worth mentioning. It is no surprisethat surface textures can be synthesized using phys-ical simulations. Although somewhat limited in thekinds of textures that can be represented, patternssuch as fur, skin, and scales can be modeled usingmethods like cellular texturing (Worley, 1996). Addi-

Image Quilting for Texture Synthesis

tionally, the common weathering effects often seen onreal life materials can be reproduced using computersimulation as seen in (J. Dorsey & Pedersen, 1999).

Additionally, we observe work related to the mainpaper of interest. This includes efficiency improve-ments introduced after the papers release such as (Wei& Levoy, 2000) and (Liang et al., 2001), along withthe related prior work from the authors in (Efros &Leung., 1999).

3. Quilting

The process of stitching together texture patches toform an output image is referred to as “image quilt-ing” in the original paper. The image quilting algo-rithm introduced in (Efros & Freeman, 2001) is theresult of multiple successive improvements to a verysimple stochastic tiling method. Here we build upfrom this simple technique and explore the results ofthe increasingly sophisticated modifications as intro-duced by the authors.

3.1. Naı̈ve Tiling Method

The first and most basic quilting algorithm discussedin the paper establishes a performance baseline thatwe should hope to exceed with successive improve-ments. This naive method takes an input texture im-age, along with a block size B, and stitches togetherpatches of size B×B drawn randomly from the sourceimage. The diagram (Figure 1) below attempts to showthis process visually:

Figure 1. Diagram showing the random patch selection al-gorithm during synthesis. The red square represents a ran-domly selected patch of size B×B in the input texture on theleft and its placement onto the synthesized output image onthe right. The light blue region represents the portion of theoutput image that has already been synthesized.

We note from this figure that the sampled patch is

placed directly into its own area in the output im-age; there is no overlap with the previously selectedpatches. This process continues in raster scan orderuntil the output image has filled the desired size.

We now provide an example of the algorithm usingone of the same textures seen in the paper (to allowfor direct comparison):

Input texture

Randomly selected patches

Figure 2. Random patch selection applied to the input tex-ture (left) and the resulting image (right). The input textureis size 64×64, and the synthesized image is a 6×6 stitchingof patches each sized 32× 32.

Looking closely at the synthesized result, we can seedistinct edges between most of the patches. As a re-sult, the output is not a very natural or satisfying ex-tension of the input texture. This, of course, is ex-pected; there are no constraints in place to select “suit-able” patches as the image is constructed. We nowlook to a slight modification of this method that yieldssignificantly better results by enforcing a simple over-lap constraint.

3.2. Overlapping Patches

We can significantly increase the quality of the syn-thesized image by searching the input texture for the“most suitable” patches during construction. The no-tion of a “suitable patch” refers to those patches whichmeet certain criteria according to some measure alonga region of overlap between patches. What exactly isused as this measure is left undefined in the paper,however we believe the authors likely make use ofsquared error here as it’s used elsewhere throughoutthe paper.

This simple “overlapping patches” method works asfollows. Let SB be the set of all possible patches ofsize B×B in the input texture, and let O be some fixedamount of pixel overlap to be used for selecting newpatches. As we iterate over the output image in rasterscan order, we fill it using the patch from SB that min-imizes the squared error along the region of overlap

2

Image Quilting for Texture Synthesis

at the current patch location. That is, if the overlapregion in the output image is to, then we set the cur-rent output location to be the patch p such that

p = argmins∈SB

(so − to)2,

where so is the region of overlap in the patch s ∈ SB.The original paper defines e = (so − to)2 as the “errorsurface” of the patch s and the output image at thecurrent location. Figure 3 below shows this processvisually.

Figure 3. Diagram showing the overlapping patch selectionduring synthesis. The red square represents the patch p ∈SB with the lowest overlap error (as defined above). On theright, we can see the location where the selected patch is tobe placed in the synthesized image. The blue region insideof the red square here represents the overlap region to ofthe currently synthesized image. On the left, the light redregion inside of the patch is the overlap region po for thepatch p. The squared difference between the pixels in thesetwo regions is defined to be the error surface, as seen above.

Now we run this algorithm on the same input textureas before:

Input texture

Constrained overlap

Here we can easily see that the output quality has im-proved; the edges between patches are much morefaint, and there are many places where the textureacross the seams appears to line up. This is due to the

fact we’ve explicitly chosen patches that work wellwithin some overlapping region, and should expectthat patches share some structure with the edges ofneighboring patches. These results are very similar tothose seen in the original paper for this simple over-lap method. However the results still leave room forimprovement; careful inspection makes it easy to spotseparate patches in the image. The fact that patchesmust have straight edges prevents the successful align-ment of features across seams. We will now look atthe final improvement introduced by the authors thatimproves output quality even more.

3.3. Minimum Error Boundary Cut

Making yet another simple modification to the previ-ous algorithm, we arrive at the paper’s primary tex-ture synthesis contribution. This algorithm allowspatches to be “cut” before they are stitched together,meaning patches are no longer required to have a straightboundary. As a result, patches can better align witheach other and stitch features together in a more flex-ible and natural way.

The authors refer to the optimal boundary cut on apatch’s edge as the “minimum error boundary cut”.This cut is defined along the error surface e (presentedin Section 3.2), and can be computed by finding thecumulative minimum error E for all possible pathsthrough this surface:

Ei,j = ei,j + min(Ei−1,j−1,Ei−1,j ,Ei−1,j+1)

E is a matrix of values built up by traversing the errorsurface e and tracking the minimal error path at eachlocation. In our implementation, we used dynamicprogramming to efficiently build up E as a memoiza-tion table (as is recommended in the paper). Oncethis computation is complete, the last row of E holdsthe cumulative costs for the minimal error paths end-ing at each location in the row. It is then trivial totrace back through E and recover the overall minimalerror cut through the error surface.

Once again, we run this method on the same inputtexture for comparison: Here we observe improvedoutput quality once again. The resulting texture ap-pears to be quite consistent throughout the image,and there are no longer any sharp, salient patch edges.Because the algorithm is allowed to have curved patchedges, features are now able to fit together in a waythat is more consistent with the structure present inthe input texture. This isn’t to say that the output isentirely free of artifacts; looking closely at the out-put image, we can spot regions where color or shapedoesn’t quite align properly. Nevertheless, for this

3

Image Quilting for Texture Synthesis

Input texture

Constrained overlap

particular texture input, the image quilting methodproposed by (Efros & Freeman, 2001) performs sig-nificantly better than the naive random approach in-troduced in Section 3.1.

4. Experiments

We now perform a number of experiments with theimage quilting method introduced in Section 3.3. Thisincludes further verification that our implementationmatches that of the original paper, synthesis resultson a number of different textures, and explorationinto potentially troublesome situations for the algo-rithm.

4.1. Implementation Correctness

We first provide a few examples using the same tex-tures as the original paper to verify our implementa-tion’s correctness. Although not definitive, this com-parison allows us to check for possible inconsisten-cies across results. Note that our algorithm was run toproduce output images of the same size as those seenin the original paper. We also use the same reportedsettings: an overlap O 1/6 the size of the block sizeB, and an error tolerance of 0.1 for selected matchingpatches.

4.2. Texture Complexity

An interesting factor of texture synthesis is the levelof randomness present in the input texture. If the tex-ture was perfectly deterministic and repetitive, thena successful image quilting algorithm would requireno more than the equivalent of copy and paste wheretexture repeats. However, for more stochastic tex-tures, there can be inconsistent variations through-out the image, making it difficult to scale in a naturalway. Here we explore the degree to which the pro-posed method can handle varying degrees of stochas-ticity and/or complexity in the input texture. The re-sults seen in Figure 4 are what we might classify as“medium” complexity; they undoubtedly have a de-

Figure 4. Result comparison between our implementationand the original paper using the image quilting algorithm.Left: input texture, Center: our implementation, Right:original paper

4

Image Quilting for Texture Synthesis

gree of repetition, but across the board all share somevarying, non-trivial structure.

To test simple, deterministic textures we run the al-gorithm on a repetitive checkerboard pattern. Theresults, unsurprisingly, look as follows:

Figure 5. Simple checkered result

Here we see the algorithm succeeds in the expectedway. It is a able to identify the simple, checkered pat-tern and repeat it consistently. This is the kind of be-havior we would expect from even the most simple oftexture synthesis algorithms.

For a more difficult pattern, we searched for texturesthat had a large degree of variability and inconsistentstructure. We hoped this would reveal some poten-tial weak points of the algorithm when faced with in-creasingly random textures. Our selected image andalgorithm’s performance look as follows:

Figure 6. Complex input texture

Figure 7. Complex texture synthesized result

Here we find the algorithm is ultimately able to comeup with a reasonable quilting pattern for the seem-ingly complicated input texture. This is somewhatsurprising, as there are a number of “medium-tier”complexity textures that appear to have some minorartifacts. We anticipated these inconsistenices wouldbe magnified in this scenario. One possibility hereis that the inherent curvy nature of the local struc-ture in the image lends itself well to the algorithm’sflexible boundaries, making it fairly easy to hide the“stitches” across patches.

5. Conclusion

In this project we implemented and explored threetexture synthesis methods discussed in (Efros & Free-man, 2001). Among these approaches include the “min-imum error boundary cut” method introduced by theauthors, on which we focus most of our analyses. Wefirst described this method in detail, as a result ofmultiple simple improvements to a naive random tilingmethod. We then implemented this algorithm andcompared our results to those seen in the original pa-per. We then performed a few additional experimentsto better understand how the algorithm performs ontextures of varying complexity. Here we found thatthe algorithm did surprisingly well at what appearedto be a sufficiently stochastic and difficult texture toextend.

ReferencesEfros, A. and Leung., T., 1999. Texture synthesis by

non-parametric sampling.

5

Image Quilting for Texture Synthesis

Efros, Alexei A. and Freeman, William T., 2001. Im-age quilting for texture synthesis and transfer.

J. Dorsey, A. Edelman, J. Legakis H. W. Jensen andPedersen, H. K., 1999. Modeling and rendering ofweathered stone.

Liang, Lin, Liu, Ce, Xu, Ying-Qing, Guo, Baining, andShum, Heung-Yeung, 2001. Real-time texture syn-thesis by patch-based sampling.

Wei, Li-Yi and Levoy, Marc, 2000. Fast texture syn-thesis using tree-structured vector quantization.

Worley, S. P., 1996. A cellular texture basis function.

6