IMAGE VECTORIZATION USING GRADIENT MESHES

Jian Sun, Lin Liang, Fang Wen, Heung-Yeung Shum

Microsoft Research Asia

ACM SIGGRAPH 2007

Input image Initial mesh

Optimized gradient meshReconstruction

Cutout tool

Outline

Introduction Background Gradient Mesh Optimized Gradient Mesh Result Conclusions

Introduction

Image Vectorization

Goal : convert a raster image into a vector graphics Compact Scalable Easy to animate

Requirements Vector-based contents (eg. Flash or SVG) on

the Internet Vector-based GUIs used in Windows Vista

Gradient mesh

Gradient mesh, a drawing tool of commercial vector graphics editors

http://www.creativebush.com/tutorials/mesh_tutorial.php

Selecting mid value skin tone

Tracing photograph

Sampling colors from photo

Sampling colors from within the mesh

Start adding mesh points Sampling colors from face mesh to hide seam

Finished eye/eye socket

gradient mesh Image represented by gradient mesh

http://www.creativebush.com/tutorials/mesh_tutorial.php

Image vectorization tools

Adobe Illustrator, “Live Trace” Corel CoreDraw, “CorelTrace” AutoTrace, “AutoTrace”

Input image Adobe, Live Trace

Optimized gradient mesh Blend surface colors according to the

control points color as constructing surface by the control points

Optimize the gradient mesh as an energy minimization problem

Advantages Efficiency of use Easy to edit – modify, animation Scalability Compact representation

JPEG, 37.5 KB

Optimized, 7.7KB

Background

Object-based vectorization

Object-based vectorization [Price and Barrett 06] Hierarchically segmentation of object and

sub-objects by a recursive graph cut algorithm

Subdivide meshes until the reconstruct error is below a threshold

Input image Subdivision mesh

RaveGrid [Swaminarayan and Prasad 06]

Constrained Delaunay triangulation of the edge contour set

cont.

Ardeco Automatic Region Detection and

Conversion algorithm [Lecot and Levy 06] Cubic splines Each region filled with a constant color, or a linear or

circular gradient

cont.

Gradient Mesh

Overview

Input raster image

Initial mesh(Coons mesh)

Optimized gradient mesh

(Ferguson patch)

Reconstruction image

min arg Energy(Mesh) non-linear least squares (NULL) problem

Cutout tool

Levenberg-Marquardt (LM) algorithm

Color fitting: monotonic cubic spline interpolation

Coherent matting method: sample object boundary color from the estimated foreground colors

E(M) Constrains:1. Smoothness2. User guided vector3. Boundary

Mesh initialization (manually) Decompose the input image into sub-

objects using Cutout tool [Li et al. 04]

Divide each sub-objects with 4 segments manually as Coons patches

m(u,v): the position vector of a point (u,v) Q : control points el al. F : basis functions

Ferguson patch [Ferguson 1964]

P(0)=A P(1)=B

TA TB

B

A

T

T

B

A

uuuuP

1122

1233

0100

0001

1)( 32

Basic Curve Segmentation

Ferguson patch

Basic Surface Segmentation

0

Ferguson patch

Basic Surface Segmentation

0

Gradient mesh

A gradient mesh consists of topologically planar rectangular Ferguson patches with mesh-lines

For each point q Position: {xq,yq}

Derivatives: {mqu,mq

v, αqumq

u, αqvmq

v} RGB color: cq = {cq(r), cq (g), cq (b)}

TTf VCUCQvuf ),(

1122

1233

0100

0001

C

321 uuuU 321 vvvV

00

0031

20

3131

2020

uu

uu

vv

vv

f

CC

CC

CCCC

CCCC

Q

Ferguson patches are lack of Cv and Cu !

Color interpolation

[Wolberg and Alfy 99] Determine the smoothest possible curve that

passes through its control points and satisfy monotonic constraint

The seven data points are monotonically increasing in f(xi) for 0 i 6, the cubic spline is not monotonic≦ ≦

Monotonic cubic spline

Rendering Ferguson patches

1. Sample color of control points2. Estimate Cu, Cv by Monotoic Cubic

Spline algorithm3. Render Ferguson patches

TTf VCUCQvuf ),(

Scalability

A gradient mesh• original resolution (x

1)

Scaling result (x8)• shape edges are well

preserved

Bi-cubic raster scaling (x8)• Blocky artifacts

appear

Optimized gradient mesh

Minimize E(M)

min arg.

Solve NULL problem using LM algorithm

Minimizing E(M) is a non-linear least squares (NULL) problem Energy function

z: vector form of unknowns in M

Levenberg-Marquardt (LM) algorithm is the most successful solver for NULL [Levenberg 44], [Nocedal and Wright 99]

cont.

Gaussian pyramid from the input image and apply coarse-to-fine optimization for LM

Optimization

Optimized gradient mesh• 0.7/pixel reconstruction

error• 40 iterations

Gradient mesh of Adobe Illustrator

Smooth constraint

Input image Opt. gradient mesh without smooth(err. 1.8/pixel)

Opt. gradient mesh with smooth (err. 0.9 pixel)

vtus 5,5,50 Smooth neighboring patches also,mp(- s, t) = m△ p-1(1- s, t) △

Vector line guided optimized gradient mesh

Initial mesh

User guided vector, V

Opt. gradient mesh with user guided vector(err. 0.5/pixel)

Directly optimized gradient mesh (err. 2.5/pixel)

Vector line guided optimized gradient mesh

Initial mesh Opt. gradient mesh with V

w = 1/5 L

3wv

Boundary constraint

The boundary of a gradient – one or more cubic Bezier spline

The control points on the boundary only move along the splineEx: control point q on the spline S in u direction

Results

Red pepper

Gradient meshes by an artist (354

patches)

Initial gradient meshes

Optimized the highlight and

shadow regions are reconstructed

Sculpture

Input image ReconstructionOptimized

gradient mesh

Face

Input image ReconstructionOptimized

gradient mesh

Conclusions

Introduce the gradient mesh as an image representation tool first

Present optimized gradient mesh

Limitations A fine image details and

highly textured image Boundaries or topologies are

too complicated

Input image

Reconstructed image

Optimized gradient meshes

END