Overview of Texture Synthesis

Ganesh Ramanarayanan

Cornell Graphics Seminar

The Problem

small texture

???

big texture

A Simple Solution …

small texture

???

big texture

???

big texture

A Simple Solution … Tiling

small texture

A Simple Solution … Tiling

Tiling is obvious!big texture?

small texture

Ideal Solution?

The Problem … in Words

Given texture I, generate a texture J which– Looks like the same texture– Has no obvious copying or tiling from I– Difference between I and J should be the same as

the way I “differs from itself” [DeBonet97]

Things to watch for:– ‘Looks the same’: what is the texture model?– ‘Obvious copying’: how is it avoided?– Underlined text: indicates algorithm parameter

Classes of Algorithms

Multiresolution pyramids– [HeegerBergen95]

Pixel-by-pixel synthesis– [EfrosLeung99]

Multiresolution pixel-by-pixel– [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01],

[Ashikhmin01] Patch quilting

– [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching

– [WuYu04], [Liu et.al. 04]

Heeger Bergen 1995

Seminal paper that introduced texture synthesis to the graphics community

Algorithm:– Initialize J to noise– Create multiresolution pyramids for I and J– Match the histograms of J’s pyramid levels with I’s

pyramid levels– Loop until convergence– Can be generalized to 3D

Heeger Bergen 1995 - Algorithm

Image pyramids– Gaussian– Laplacian

Steerable pyramids [SimoncelliFreeman95]

– b): multiple scales of oriented filters

– c): a sample image– d): results of filters in b)

applied to c)

Heeger Bergen 1995 - Results

I J

Successes Failures

I J

Heeger Bergen 1995 - Results

Heeger Bergen 1995 - Verdict

Texture model:– Histograms of responses to various filters

Avoiding copying:– Inherent in algorithm

No user intervention required Captures stochastic textures well Does not capture structure

– Lack of inter-scale constraints

De Bonet 1997

Propagate constraints downwards by matching statistics all the way up the pyramid

Feature vector: multiscale collection of filter responses for a given pixel

Algorithm:– Initialize J to empty image– Create multiresolution pyramids for I and J– For each pixel in level of J, randomly choose pixel

from corresponding level of I that has similar feature vector

De Bonet 1997 - Algorithm

6 feature vectors shown

Notice how they share parent information

De Bonet 1997 - Results

De Bonet 1997 - Verdict

Texture model:– Feature vector containing multiscale responses to

various filters

Avoiding copying:– Random choice of pixels with ‘close’ feature vectors,

but copying still frequent on small scale

Individual per-filter thresholds are cumbersome Feature vectors used in later synthesis work

Efros Leung 1999

Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood

Algorithm:– Initialize J to empty– Synthesize new pixels in a spiral pattern– Select new pixel from I based on conditional

probabilities given the current neighborhood in J and similar neighborhoods in I

Efros Leung 1999 - Algorithm

I J

‘causal’neighborhood

Efros Leung 1999 - Results

Efros Leung 1999 – Verdict

Texture model:– MRF

Avoiding copying:– MRF

Neighborhood size = largest feature size Markov model is surprisingly good

– “I spent an interesting evening recently with a grain of salt.”

Search is very slow with large neighborhoods

Wei Levoy 2000

Adds tree-structured vector quantization speedup and multiresolution matching

Algorithm:– Initialize J to noise– Synthesize new pixels in scanline order– Select new pixel from I that has the closest

matching neighborhood to J– In multiresolution case, look instead at

neighborhood of feature vectors

Wei Levoy 2000 - Algorithm

Scanline synthesis with neighborhoods

– a) input texture– b) algorithm start– c) midway point– d) ending

Wei Levoy 2000 - Results

Wei Levoy 2000 - Verdict

Texture model:– Multiscale filters and MRF

Avoiding copying:– Not explicitly, but in practice inherently avoided;

TSVQ adds additional unpredictability

Results comparable to [EfrosLeung99] TSVQ speedup comes at a price (quality loss)

Ashikhmin 2001

Adds more verbatim image copying and user-constrained synthesis

Algorithm:– New array J* stores location in I of J’s pixels– Initialize J to empty, J* to random– Synthesize new pixels in scanline order– Select new pixel by copying from patches of I, as

determined by locations in J*– In user-constrained case, also include error

comparison with user-specified target image T

Ashikhmin 2001 - Algorithm

J*

x=3y=4

x=4y=4

x=12y=11

x=17y=16

Candidates:-x=4, y=5-x=18, y=16-x=11, y=12

x=18y=16

Ashikhmin 2001 - Results

Ashikhmin 2001 - Verdict

Texture model:– MRF

Avoiding copying:– Actually, here it is encouraged on a small scale, but

in practice it doesn’t occur on a large scale

Recognized that copying preserves fine detail Features can get ‘cut off’ because of the abrupt

jumping from patch to patch

Hertzmann et. al. 2001

Framework for solving image analogies: learning an image filter f by example– Given A, f(A), and B, generate f(B)

To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J

Algorithm:– Bring image histograms into correspondence– Combine [WeiLevoy00] and [Ashikhmin01]

Each search method is weighted appropriately and then the best choice is used

Hertzmann et. al. 2001 - Algorithm

A provides set of constraints

Histogram matching critical because of color differences between A and B sets of images

Hertzmann et. al. 2001 - Results

I J I J

Hertzmann et. al. 2001 - Results

Hertzmann et. al. 2001 - Verdict

Texture model:– Multiscale filters and MRF

Avoiding copying:– See [WeiLevoy00] and [Ashikhmin01]

Unique approach to image manipulation and texture synthesis

Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]

Efros Freeman 2001

Inspired by chaos mosaics [Xu et. al. 00] Make an ‘image quilt’ by pasting together

patches of the input and fixing seams Algorithm:

– Initialize J to empty– Copy new patches of certain size in scanline order,

with fixed overlap width– Randomly select new patch that has error in overlap

region less than given threshold– Copy pixels based on minimal error boundary cut

Efros Freeman 2001 - Algorithm

Input texture

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

Efros Freeman 2001 - Results

Efros Freeman 2001 - Verdict

Texture model:– ??

Avoiding copying:– Randomized patch selection, but still noticeable

Patch size is a hard parameter to understand Results are surprisingly good given algorithm Multiscale goes on a brief hiatus

Kwatra et. al. 2003

Generalizes seam computation in overlap regions as a graph cut problem– Based on [Boykov et. al. 99] (with Ramin Zabih)

Algorithm:– Initialize J to empty– Copy pieces of I to J using a variety of methods– Formulate flow graph in overlap region based on

errors and compute minimum cut– Copy sink-side pixels to J– Variety of strategies to further hide seams

Kwatra et. al. 2003 - Algorithm

(assume cut region is 3x3 for simplicity)

Kwatra et. al. 2003 - Results

Kwatra et. al. 2003 - Results

Kwatra et. al. 2003 - Verdict

Texture model:– MRF

Avoiding copying:– Even with a multitude of patch selection methods,

still noticeable when it happens repeatedly

Paper presents a bag of synthesis tricks without much intuition for when to use what

Graph cut formalization is useful and powerful

Wu Yu 2004

Synthesizes a feature image J* in parallel Tries to match features in overlap region Algorithm:

– Compute feature image I*– Initialize J* to empty– Synthesize J and J* in parallel– When searching for new patch, add in weighted

feature matching error– Prior to copying, warp selected patch to match

features exactly

Wu Yu 2004 - Algorithm

Match feature pixels based on

– Tangent orientation– Distance– Bijectivity

Example:– L1

out matches L1in

– L2out matches L3

in

– Warping will move matching features closer to each other

Feature correspondence

Wu Yu 2004: Results

Wu Yu 2004: Verdict

Texture model:– Feature lines and MRF

Avoiding copying:– No; in fact, mirroring is often very noticeable

Unclear how to combine this metric with others Insight to use warping is valuable, although the

next paper does a much better job with it

Liu et. al. 2004

Regular textures: can be tiled Near-regular textures: statistical distortion of a

regular texture Algorithm (simplified):

– Mark lattice points in I– Transform I into regular texture R– Express transformation as deformation texture I*– Synthesize larger deformation J*– Synthesize J by combining J* with R

Liu et. al. 2004 - Algorithm

RI

Deformation Texture I*

Liu et. al. 2004 - Results

Liu et. al. 2004 - Results

original texture transfer

Liu et. al. 2004 - Verdict

Texture model:– Regular texture with modulated geometry and color

Avoiding copying:– Depends on the synthesis algorithm

Proposes specific model to handle specific types of textures

Interestingly, this method depends on previous texture synthesis methods

Classes of Algorithms - Recap

Multiresolution pyramids– [HeegerBergen95]

Pixel-by-pixel synthesis– [EfrosLeung99]

Multiresolution pixel-by-pixel– [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01],

[Ashikhmin01] Patch quilting

– [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching

– [WuYu04], [Liu et.al. 04]

Evolution of Texture Model

Responses to image filters / decompositions Markov Random Fields Copies of small pieces of image Feature data Various combinations of the above

What’s Next?

Texture synthesis is really hard Rather than creating a silver bullet, focus on

algorithms for more specific problems Superresolution:

+ = ???

What’s Next?

Texture synthesis is really hard Rather than creating a silver bullet, focus on

algorithms for more specific problems Superresolution:

+ =

[Hertzmann et. al. 01]