Appearance Models

• Shape models represent shape variation

• Eigen-models can represent texture variation

• Combined appearance models represent both

Appearance Models

• Statistical model of shape and

texture

• Generative model

– general

– specific

– compact

Building Appearance Models

• For each example extract shape vector

• Build statistical shape model,

Shape, x = (x1,y1, … , xn, yn)T

ssbPxx

Building Appearance Models

• For each example, extract texture vector

Shape, x = (x1,y1, … , xn, yn)T

Texture, gWarp tomeanshape

Warping texture

• Problem:– Given corresponding points in two images,

how do we warp one into the other?

• Two common solutions1. Piece-wise linear using triangle mesh

2. Thin-plate spline interpolation

Interpolation using Triangles

Region of interest enclosed by triangles.

Moving nodes changes each triangle

Just need to map regions between two triangles

),( :points Control ii yx )','( :points Warped ii yx

Barycentric Co-ordinates

cbax

a b

c

x

''' cbax '

'a

'b

'c

'x

1

10 and 10

if triangle theinside is

βα

x

Barycentric Co-ordinates

a b

c

x

)( ab

)( ac cba

cba

acabax

)1(

)()(

cbax

1

Three linear equations in 3 unknowns

1111yyy

xxx

cba

cba

y

x

Interpolation using Triangles

• To find out where each pixel in new image comes from in old image

• Determine which triangle it is in• Compute its barycentric co-ordinates• Find equivalent point in equivalent triangle in

original image

• Only well defined in region of `convex hull’ of control points

Thin-Plate Spline Interpolation

• Define a smooth mapping function (x’,y’)=f(x,y) such that

– It maps each point (x,y) onto (x’,y’) and does something smooth in between.

– Defined everywhere, even outside convex hull of control points

niyxyxf iiii ..1 allfor )','(),(

Thin-Plate Spline Interpolation

• Function has form )),(),,((),( yxfyxfyxf yx

ii

ni

ixixxx rrwxbayxf log),( 2

1

ii

ni

iyiyyy rrwybayxf log),( 2

1

222 )()( where iii yyxxr

)','(),(

bygiven equationslinear thesolvingby found are

),,,,,( parameters The

iiii

yiyyxixx

yxyxf

wbawba

Building Texture Models

• For each example, extract texture vector

• Normalise vectors (as for eigenfaces)

• Build eigen-model

Texture, g

Warp tomeanshape

ggbPgg

Face Texture Model

1b12 12

2b22 22

3b32 32

Textured Shape Modes

Shape variation (texture fixed)

Generate position of control points

Warp mean texture image

(Mean points go to new points, X)

)( ssT bPxX

Textured Shape Model

1b12 12

2b22 22

3b32 32

Combined Models

• Shape and texture often correllated– When smile, shadows change (texture) and

shape changes

• Learning this correlation leads to more compact (and specific) model

Learning Correlations

sb

gbModel assuming shape and texture independent

Model accounting for correlations between shape and texture

Learning Correlations

• For each image in training set we have best fitting shape and texture param.s

• Construct new vector,

• Apply PCA (mean + eigenvec.s of covar.)

gs bb ,

g

s

c b

Wbb

cQ

QQcb

g

sc

too.does thus

set, trainingover themean zero have and Both

c

gs

b

bb

Combined Appearance Models

cQ

WQQc

b

Wbb

g

s

g

sc

ggbPgg ssbPxx

cQgg

cQxx

g

s

Varying c changes both

shape and texture

Combined Appearance Model

• Generate shape, X, and texture, g

• Warp texture so mean control points lie on

new X

1b12 12

Face Appearance Model

1b12 12

2b22 22

Face Appearance Model

3b32 32

4b42 42

Sub-cortical structures

• 72 examples

• 123 points

• 5000 pixel model

Ventricles

Lentiform Nucleus

Caudate Nucleus

Shape and Texture Modes

Shape variation (texture fixed)

Texture variation (shape fixed)

Combined Appearance Model

• Shape and texture correlated

Full brain slice

Shape:

Texture:

Full brain slice

CombinedMode 1

CombinedMode 2

Problems with viewpoint

• Models require all points visible– Sometimes a problem for 2D images of 3D objects

• Small rotations (+/-30o) of face modelled well• Large rotations cause occlusions

– Eg eye hidden behind nose etc

• Solutions1. Use multiple `view based’ 2D models

2. Use a full 3D model

View-Based Models

• Build 3 distinct models– Exploit symmetry

Profile Profile (Reflected)

Frontal

Half-Profile Half-Profile (Reflected)

Face Profile Model

Mode 1:

Mode 2:

Half-Profile Model

Mode 1:

Mode 2:

3D Models

• Use 3D shape model (3n-D vectors)

• Points control a polyhedral mesh

• Texture mapped onto mesh and modelled

• Reconstruct by generating new texture and mapping onto 3D mesh described by shape model

3D Models

=+

Mesh

Texture

Interpreting Images (1)

Place model in image

Measure Difference

Update Model

Iterate