Wangfei Ningbo University

A Brief Introduction to A Brief Introduction to Active Appearance Active Appearance

ModelsModels

Wangfei Ningbo University

Topics of the talkTopics of the talk

IntroductionIntroduction

AAMAAM

Future and Related worksFuture and Related works

ReferenceReference

Wangfei Ningbo University

IntroductionIntroduction

What is AAM?What is AAM?Non-linear, generative, parametric Non-linear, generative, parametric modelsmodels

What can AAM do?What can AAM do?Statistical models Statistical models

Depend on the problemDepend on the problem

Computer Vision Image InterpretationComputer Vision Image InterpretationFace RecognitionFace Recognition

Medical image analysisMedical image analysis

Wangfei Ningbo University

Introduction-ApplicationIntroduction-Application

Face RecognitionFace Recognition

Figure: Example face image annotated Figure: Example face image annotated with landmarkswith landmarks

Wangfei Ningbo University

Introduction-ApplicationIntroduction-Application

Medical image analysisMedical image analysis

Figure: Example MR image of knee with Figure: Example MR image of knee with carilage outlinedcarilage outlined

Wangfei Ningbo University

Introduction-HistoryIntroduction-History

HistoryHistorySnake (Active Contour Models) --1989Snake (Active Contour Models) --1989

ASM (Active Shape Models) --1995ASM (Active Shape Models) --1995

AAM (Active Appearance Models) --1998AAM (Active Appearance Models) --1998

Wangfei Ningbo University

Snake- Active Contour Snake- Active Contour ModelsModels

Start with a curve near the objectStart with a curve near the objectDiscrete snake: spline with n control Discrete snake: spline with n control pointspoints

Evolve the curve to fit the boundaryEvolve the curve to fit the boundaryMinimize the energy functionMinimize the energy function

Original formulationOriginal formulation

Wangfei Ningbo University

SnakeSnake

WeaknessWeaknessweak constraintsweak constraints

high compute costhigh compute cost

can not search inside boundarycan not search inside boundary

not optimal for known shape because of not optimal for known shape because of no prior knowledgeno prior knowledge

Wangfei Ningbo University

ASM-Active Shape ModelsASM-Active Shape Models

Use prior knowledge from the training Use prior knowledge from the training setset

Variable parametersVariable parameters

Statistical Shape ModelsStatistical Shape ModelsAllow formal statistical techniques to be Allow formal statistical techniques to be applied to sets of shapes, making applied to sets of shapes, making possible analysis of shape differences possible analysis of shape differences and changesand changes

Wangfei Ningbo University

ASMASM

Variable parametersVariable parameters

positiopositionn

shape parametersshape parameters

scalescale

orientatioorientationn

Wangfei Ningbo University

ASMASM

ShapeShapedefine the shapes as the coordinates of define the shapes as the coordinates of the v vertices that make up the mesh:the v vertices that make up the mesh:

AAM allow linear shape variation

the shape the shape parametersparameters

Wangfei Ningbo University

ASMASM

The linear shape model of an The linear shape model of an independent AAMindependent AAM

Wangfei Ningbo University

ASMASM

Build the modelBuild the modelGet shapes from a set of annotated Get shapes from a set of annotated images of typical examplesimages of typical examples

NormalizeNormalize

PCA PCA

Wangfei Ningbo University

ASMASM

Use the model for locatingUse the model for locatingGiven a rough starting approximation Given a rough starting approximation instanceinstance

Examine a region around, find the best Examine a region around, find the best nearby match for each pointnearby match for each point

Update the parameters to best fit the Update the parameters to best fit the new pointnew point

Repeat until convergenceRepeat until convergence

Wangfei Ningbo University

ASMASM

Figure: Search using Active Shape Figure: Search using Active Shape Model of a faceModel of a face

Wangfei Ningbo University

AAM-Active Appearance AAM-Active Appearance ModelsModels

ShapeShape

AppearanceAppearance

Model InstantiationModel Instantiation

FittingFitting

Wangfei Ningbo University

AAMAAM

AppearanceAppearanceWarp each example imageWarp each example image

SampleSample

NormalizeNormalize

PCAPCA

Wangfei Ningbo University

AAMAAM

Warp each example imageWarp each example imageIts control points match the mean shapeIts control points match the mean shape

UsingUsingPiecewise affine warping Piecewise affine warping

((Delaunay Delaunay Triangulation algorithm)Triangulation algorithm)

Thin plate splinesThin plate splines

SampleSampleThe intensity information from shape-The intensity information from shape-normalized image to form a texture vectornormalized image to form a texture vector

Wangfei Ningbo University

AAMAAM

Figure: a ‘shape-free’ image patchFigure: a ‘shape-free’ image patch

Wangfei Ningbo University

AAMAAM

NormalizeNormalizeTo minimize the effect of global lighting To minimize the effect of global lighting variationvariation

PCAPCA

The appearance expressionThe appearance expression

the appearance parametersthe appearance parameters

Wangfei Ningbo University

AAMAAM

Figure: The linear appearance Figure: The linear appearance variation of an independent AAMvariation of an independent AAM

Wangfei Ningbo University

AAMAAM

Model InstantiationModel InstantiationThe two equations describe the shape The two equations describe the shape and the appearance variationand the appearance variation

Given the shape parametersGiven the shape parameters

Given the appearance parametersGiven the appearance parameters

Create warping appearance Create warping appearance AA from the from the base mesh base mesh SS00 to the model shape to the model shape SS

Wangfei Ningbo University

AAMAAM

Figure: An example of AAM Figure: An example of AAM instantiationinstantiation

Wangfei Ningbo University

AAMAAM

FittingFittingNaturally, we want to minimize the error Naturally, we want to minimize the error betweenbetween and and

Denote as:Denote as:

Wangfei Ningbo University

AAMAAM

Fitting AlgorithmsFitting AlgorithmsInefficient Gradient Descent AlgorithmsInefficient Gradient Descent Algorithms

Efficient Ad-Hoc Fitting AlgorithmsEfficient Ad-Hoc Fitting Algorithms

Efficient Gradient Descent Image Efficient Gradient Descent Image AlignmentAlignment

Lucas-Kanade Image AlignmentLucas-Kanade Image Alignment

Forwards Compositional Image AlignmentForwards Compositional Image Alignment

Inverse Compositional Image AlignmentInverse Compositional Image Alignment

......

Wangfei Ningbo University

Future and Related worksFuture and Related works

Alignment algorithmsAlignment algorithms

Automatic landmarkAutomatic landmark

View-Based appearance modelsView-Based appearance models

ApplicationsApplications

……

Wangfei Ningbo University

ReferenceReference

T.F. Cootes and C.J. TaylorT.F. Cootes and C.J. TaylorStatistical Models of Appearance for Statistical Models of Appearance for computer visioncomputer vision

Active Appearance ModelsActive Appearance Models

Active Shape Models-Their Training and Active Shape Models-Their Training and ApplicationApplication

Iain Matthews and Simon BakerIain Matthews and Simon BakerActive Appearance Models RevisitedActive Appearance Models Revisited

……