1 Roey Izkovsky Yuval Kaminka Matting Helping Superman fly since 1978

1

Roey Izkovsky

Yuval Kaminka

Matting

Helping Superman fly since 1978

2

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

3

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

4

The matting problem - Motivation

Image and video editing

New backgroundComposite image

5

The matting problem - Motivation

Image and video editing

Input image New image

6

The matting problem

iiiii BFI )1(

– The separation of an image I intoI. Foreground object image F

II. Background image B

III.Alpha matte α – the opacity )10(

– Problem: extract F, B, α from image

hair fur

7

Why is matting challenging?

• Under constrained problem:One equation, 3 unknowns

iiiii BFI )1(

We need to constrain the problem!

8

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

9

Previous work

Two types:

Known background Natural image

matting matting

10

Known Background

• Blue screen Matting• Still under-constrained

– Solution: make more assumptions• “Foreground contains no blue”• Other foreground distribution assumption…

• Use two different backgrounds

• Main flaw: need to know the background…

iiiii BFI )1( Blue background Composite image

11

Natural Image Matting

• The assumptions:– Smoothness of the alpha matte– GMM for the Background and Foreground

colors

• Initial estimate: trimap provided by the user

Input image Trimap

• Background

• Foreground

• Unknown

12

Natural Image Matting

• The algorithms framework:– Estimate F, B distributions from close pixels– Find best α by some method

13

Knockout

– Extrapolate F,B from close neighborhood

– Estimate α from calculated F, B values

14

Bayesian

– Estimate F, B distributions in area

– Find best α matching distributions

CBFPBF

|,,maxarg,,

)(/)()()(,,maxarg,,

CPPBPFPBFCPBF

)()(,,maxarg,,

BPFPBFCPBF

15

Bayesian

– P(F), P(B) from image samples

– P(C|F,B,α) using a distribution for C BFC )1(

)()(,,maxarg,,

BPFPBFCPBF

16

Natural Image Matting

• Main flaw: Accurate trimap required• Tedious to provide manually

• Hard to extract automatically

In particular, not feasible to videos

Binary segmentation

Adding unknown region

Input imageTrimap

17

Great.So let’s get started…

18

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

19

New Approach to Matting

Trimap reduces to scribbles

Two new methods– Iterative optimization approach

• Heuristic algorithmic optimization

– A closed form solution• Mathematical approach

Trimap Scribbles

20

Iterative optimization approach

Jue Wang

Michael F. Cohen

21

Iterative approach

22

Iterative approach

• Score: fit to image data

+alpha matte smoothness

• Iteratively propagating estimated results.

23

Iterative optimization - outline

• Initialize “work pixels” from scribbles

• Repeatedly:• Expand work pixels • Find best alpha matte

• Stop when finished

24

Initialization

• Introducing:– ui - uncertainty variable

– Uc – work pixels

ui = 0

α = 0

Uc = {user scribbles}

ui = 0

α = 1

ui = 1

α = 0.5

)10( iu

25

Optimization

Uc = {user scribbles + 15 pixel radius}

Our goal:

find α matte for Uc that minimizes the energy -

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

DataSmoothness

26

Vd

Score for αp = α

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

N Possible values for F N Possible values for B

BF )1(

2))1(,( BFId p

Fw2

Fw3

Fw4

Fw1

Bw1

Bw2

Bw3

Bw4

Image color Ip

27

Vd

• Fit measure of αp to Ip

• Score for αp = α :

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

2

2

1 12 2

))1(,(exp

1

jip

N

i

N

j

Bj

Fi

BFIdww

N

Fi , Bj – possible values for F, B in the pixel

wFi, wB

j – corresponding weights

28

Vd

F Samples

B Samples

2

2

1 12 2

))1(,(exp

1

jip

N

i

N

j

Bj

Fi

BFIdww

N

Fi , Bj – possible values for F, B in the pixel

wFi, wB

j – corresponding weights

)2/(

),(exp))(1(

2

2

r

ppspuw iii

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

p

α = 0.9

u = 0.2

α = 0.8

u = 0.3

α = 0.4

u = 0.5

α = 0.4

u = 0.4

α = 0.2

u = 0.3

α = 0.3

u = 0.3

α = 0.5

u = 1.0

What happens when there are

not enough F/B samples?

29

Vd

• Score for αp = α :

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

2

2

1 12 2

))1(,(exp

1)(

jipN

i

N

j

Bj

Fip

BFIdww

NL

• Discretize

• and normalize

},...,,{ 2521

j

jp

kpk

pd L

LV

)(

)(1)(

30

Vs

• Matte smoothness :

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

)/)(exp(1),( 222121 ssV

31

Iterative optimization – step 2

Uc = {user scribbles + 15 pixel radius}

Our goal: find α matte for Uc that minimizes the energy -

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

Uc Graph

Nodes = Pixels, Edges by 4-connectivity

32

Iterative optimization – step 2

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

GOAL: Minimize

BELIEF PROPAGATION

33

Iterative optimization – step 2

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

GOAL: Minimize

BELIEF PROPAGATION

αlog p

0-2

0.04-1.7

……

12.3

p q

mpq – message from p to q

t=0

y

0tpqm

)(),(min)(0 pqp

p

q kpd

kq

kps

k

kq

tpq VVm

Vector: p’s “opinion” for eachpossible α for q

34

Iterative optimization – step 2

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

GOAL: Minimize

αlog p

0-1.6

0.04-1.2

……

12.2

p q

t=1

y

1tpqm

αlog p

01

0.040.7

……

1-2.1

0typm

BELIEF PROPAGATION

mpq – new message pq

myp – previous message yp

qrpr

kp

trp

kpd

kq

kps

k

kq

tpq

ppqp

p

q mVVm~

01 )()(),(min)(

35

Iterative optimization – step 2

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

GOAL: Minimize

BELIEF PROPAGATION

p q

t=2,3,4…

y

tpqm

1typm

qrpr

kp

trp

kpd

kq

kps

k

kq

tpq

ppqp

p

q mVVm~

1 )()(),(min)(

36

Iterative optimization – step 2

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

GOAL: Minimize

BELIEF PROPAGATION

p q

t=T (stopping time)

y

αLog p

0-1.7

0.04-1.3

……

11

αLog p

0-1.6

0.04-1.1

……

11.3

αLog p

0-1.7

0.04-1.3

……

11

αLog p

0-1.7

0.04-1.3

……

11

qr

qtrq

kqd kmV q

~

1 )()(

37

Iterative optimization – step 2

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

GOAL: Minimize

BELIEF PROPAGATION

p q

t=T (stopping time)

y

αLog p

0-1.7

0.04-1.3

……

11

αLog p

0-1.4

0.04-1.5

……

11.3

αLog p

0-1.7

0.04-1.3

……

11

αLog p

0-1.7

0.04-1.3

……

11

Best state calculated for each node:

prp

trp

kpd

kp kmV p

p ~

1 )()(minarg

38

Iterative optimization – step 3

),()(

~,

qp

qpUqp

sUp

pd

cc

VVV

Found α matte for Uc that minimizes the energy -

Update F, B and uncertainty:

**

1

))1((minarg,

*

2

,

**

Bj

Fi

jipBF

wwu

BFIBFji

39

Iterative optimization - algorithm

•Initialize Uc, F, B, u and alpha matte from scribbles

•Repeatedly:•Expand Uc by another 15 pixel radius•Find best alpha matte (BP)•Update F,B,u for new matte

•Stop when total uncertainty is minimal

Initial matte Propagation of α matte Final matte

40

Iterative optimization - Results

Input image Extracted matte

41

Iterative optimization - Results

Input image

Extracted matte

Composite image

42

Iterative optimization - Results

The ambiguity bunny

43

Ambiguity bunny with trimap

Iterative optimization - Results

Scribbles result Trimap resultAmbiguity bunny with scribbles

44

Iterative optimization - Summary

• Minimal user input• Applicable to video

• Sensitive to ambiguity in F, B• Uses simple color-model

• Performance:– 15-20 min. on a 640x480 image– Factor 50 reported by better implementation

45

Fantastic.Let’s go on…

46

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

47

Closed form solution

Anat Levin

Dani Lischinski

Yair Weiss

48

Closed form solution

• Assumption: local smoothness in F, B

cancel out unknowns from the matte equs.

• Solve for F,B and alphausing algebraic tricks.

49

Closed form solution

Assumptions:– F,B locally smooth.

treat F,B as constant in a small window w

ww

ww

www

wiwiwiwii

BF

Bb

BFa

bIawiBFI

1

)1(

50

Closed form solution

GOAL:

Minimize -

Ij wiwwiwi

j

jjjabIabaJ 22)(),,(

-Numerical stability

-Bias to smoother matte

wj

wiwi bIawi

ww

ww

www BF

Bb

BFa

1

51

Closed form solution

• GOAL:– Minimize:

Ij wiwwiwi

j

jjjabIabaJ 22)(),,(

211 )( bIa pp

299 )( bIa pp

29 ia

52

Closed form solution

• Minimize:

Ij wiwwiwi

j

jjjabIabaJ 22)(),,(

3N Variables (N = image size)

We can rid a, b by algebraic manipulation

53

Closed form solution

• Minimize:

Ij wiwwiwi

j

jjjabIabaJ 22)(),,(

Theorem: for we have

),,(min)(,

baJJba

k kwjik

kjki

kwkijij II

wL

),|(2

||

))((1

1||

1

LJ T)(

Intuitively, L is some covariance matrix

kwjik k

kjki

kij

II

wL

),|(2

))((

||

1~

54

Closed form solution

• Minimize:

Ij wiwwiwi

j

jjjabIabaJ 22)(),,(

Proof: Rewrite in matrix form:

k ww

w

w kk

k

kb

a

I

I

baJ

2

||

1

2/1||

1

00

1

1

),,(

55

Closed form solution

• Minimize:

Ij wiwwiwi

j

jjjabIabaJ 22)(),,(

Proof: Rewrite in matrix form:

0

1

1

2/1||

1

k

k

ww I

I

G

2

),,(

k

w

w

w

w k

k

k

k b

aGbaJ

0||

1

k

k

ww

By mean-least-squares, best a,b pair

for each window is:

kkkkkkw

Tww

Twww GGGba 1** )(),(

k

wwTww

Tww kkkkkk

GGGG2

1)(

),,(min)(,

baJJba

56

Closed form solution

• Some more manipulation give the required result

LJ T)(

k kwjik

kjki

kwkijij II

wL

),|(2

||

))((1

1||

1

EXCITED?

GET YOUR I LOVE MATHT-SHIRT, NOW FOR ONLY $1999

k

wTww

Tww kkkkk

IGGGGJ2

1 ))(()(

57

Closed form solution

• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.

Use redundancy to allow a “color-line” model per window

21

21

)1(

)1(

wiwii

wiwii

BBB

FFF

Color line model:

OUT: F, B Constant within a window

IN: F, B are on some line

BGRc

wci

cwi bIawi

,,

R

GF1

F2

58

Closed form solution

c

wci

cwi bIawi

• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.

Use redundancy to allow a “color-line” model per window

59

Closed form solution

c

wci

cwi bIawi

Ij wi c

cww

c

ci

cwi

j

jjjabIabaJ 22 )()(),,(

Now, as before, cost is:

And a,b can be cancelled out.

• For color images:– Simple: Do each channel separately– Smart: Assume one alpha for R,G,B.

Use redundancy to allow a “color-line” model per window

60

Closed form solution

Now problem reduced to finding best α for:

LJ T)(

k kwjik

kjki

kwkijij II

wL

),|(2

||

))((1

1||

1

L is Huge size NxN (N = # image pixels)

But Sparse…

61

Closed form solution

• The algorithm:– Compute L– Solve for given the scribbles.

• Solving a sparse set of bilinear equationswith constraints (Lagrange multipliers)

– Find F, B given the matte• Adding smoothness assumptions on F, B

• Improvements:– Use larger environment in low cost by “pyramids”

LTminarg

62

Closed form solution - Results

Input image Extracted matte

63

Closed form solution - Results

Input image with scribbles

Problematic matte

64

Eigenvectors as guides

Small eigenvectors of L are

correlated with minimal matte

L is positive definite.

Eigenbasis: v1,…,vN

Eigenvalues: λ1 > λ2 > … > λN > 0

2211

211 ...)( NNNN

t

ii

LJ

v

65

Eigenvectors as guides

Small eigenvectors of L are

correlated with minimal matte

66

Eigenvectors as guides

Small eigenvectors of L are

correlated with minimal matte

can guide user scribbles

Eigenvectors matching smallest eigenvalues

Guided scribbles Resulting matte

67

Closed form solution - Summary

• Minimal user input• Provable optimality (under assumptions)• Assumes only smooth F,B (no color model)

• Applicable to video (as we speak…)

• Problematic with textures

• Performance:– 20-40 seconds for a 200x300 image– Expensive in memory

68

Superb.Let’s sum up…

69

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

70

Comparison

Iterative approachPoisson Closed form solution

Input image

Matte ground truth

71

Main improvements

Trimap based approaches

New approaches

User inputTrimapScribbles

Complex foreground

Poor results. Exact trimap required

Good results

VideoNot easily applicable.Applicable

72

Comparison

Color ambiguity

Iterative approach Closed form Sensitive Sensitive

Solvable by adding more scribbles

73

Comparison

Improving results…

Iterative approachBayesian Closed form solution

Ambiguity bunny

74

Comparison

Optimality?

Iterative approach Closed form

Uses heuristics

to optimize

Provably optimal

But for the specific

(simplified) cost

75

Comparison

Textures

Iterative approach Closed form

Assumes only

Alpha matte smooth

F,B must satisfy

color-line model

76

Comparison

Rough edges

Iterative approach Closed form

Assumes

Alpha matte smooth

Can handle rough

edges

Input image with scribbles

matte results

77

Comparison

Running time

Iterative approach Closed form

~20 sec. 20/40 seconds

Costly in memory

(For medium size image)

78

Comparison

Tests

Iterative approach Closed form

No quantitative

results reported

Extensively tested

quantitative results

79

Outline

• The matting problem• Previous work• New approaches:

– The iterative approach

Jue Wang, Michael F.Cohen

– Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

• Comparison and summary• Bonus?

80

Environment Matting and Compositing

Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin

81

Environment Matting

C = F + (1- )B + ~ Contribution of light from Environment

that travels through the object

R – reflectance imageT – Texture image

82

Environment Matting?

Alpha Matte Environment Matte Photograph

83

Environment Mattin

Alpha Matte Environment Matte Photograph

84

Summary

• The matting problem• Old methods: require trimap• Two new methods from scribbles:

– Iterative optimization• Assume: matte smooth, F,B locally similar• Use heuristic optimization for alpha

– Close form solution• Assume: F, B locally smooth (color-line model)• Solve linear equations for alpha

85

ANY LAST

86