3D Human Face Reconstruction and Expression Modelling

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N 3D Human Face Reconstruction 3D Human Face Reconstruction and Expression Modellingand Expression Modelling

Alexander Woodward

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Outline

AimSystem overviewRelated work3D face reconstructionExpression modellingContributions and future work

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

3

OverviewOverview

Aim: Integrated system for 3D face reconstruction and expression modelling

Vision based not graphics basedLow cost and self-contained

Results can be applied to:Biometrics and securityBiomedical visualisationComputer and video games

FilmTeleconferencingHuman computer interaction

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

System overview

3D video scanner

3D reconstruction

Expression modelling

Active structured lighting

Active & passive binocular stereo

Active photometric stereo

DynamicStatic

Video basedMarker based motion capture

3D data

Muscle inverse kinematics Sequences from 3D video

scanner

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Related workRelated work

20 April 2023 Department of Computer Science 5

Complete systems for face reconstruction and animation are uncommonHigh hardware requirementsData acquisition, motion capture and animation systems are

often provided as disparate packages or only as a service, cf. a stand-alone solution

At least 9 prominent projects aimed toward complete systemsExcluding in-house solutionsLarge body of work in 3D face research

3D reconstruction, expressions, motion capture

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Related workRelated workBorshukov et al (2003 – 2007)

Playable Universal Capture approach 3D scanner, marker based tracking, optical flow, video texture

Ma et al (2007, 2008)

Capture face reflectance 3D scanner, photometric stereo, motion captureLight stage – 156 LED lights over an icosahedron

Image Metrics Inc. & U Sth Carolina Graphics Lab (2008) Digital Emily project

Light stage captures geometry and reflectance 33 expressions captured; creates an animation rigPerformance data mapped to the 3D face

6

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

3D reconstruction

3D video scanner

3D reconstruction


DynamicStatic


3D data


scanner




2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

3D reconstruction requirements3D reconstruction requirements

Off-the-shelf hardware, no special propertiesCameras, PC, projector

Low acquisition time – faces move, esp. childrenControlled lightingVision based

New algorithms

Useful for any type of object

8

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Static 3D reconstruction

3D video scanner

3D reconstruction


DynamicStatic


3D data


scanner




2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

10

Static 3D reconstructionStatic 3D reconstruction

Evaluated approaches:1. Active & passive binocular stereo2. Active structured lighting

3. Active photometric stereo

Ground truth data: 3D scanner

Evaluate effectiveness Accuracy, time complexity

Determine best approach for dynamic 3D reconstruction system 12 algorithms Database of 15 faces

Alternative test set Focus on stereo algorithms

Compared to Middlebury, algorithms rank differently for faces Projected patterns improve and level out performance

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

11

Active binocular stereoActive binocular stereoStrip colour pattern: much higher accuracy

SAD correlation algorithm:

Pattern colour should contrast strongly on skin

Strip pattern SAD - without pattern

SAD - with strip pattern

92%80%

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

12

Statistical resultsStatistical results

Ground truth

Gray code

FCV

SAD SDPSGC

Four PathShapelet

BP + Grad. + Strip DPM + Grad. + StripCM + Grad. + Strip

73% 77% 89% 88% 89% 92% 79% 84% 92%

77% 83% 92% 80% 85% 92% 89% 90% 93%

69% 54% 71% 97%

+ Grad. + Strip + Grad. + Strip + Grad. + Strip

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Dynamic 3D reconstruction

3D video scanner

3D reconstruction


DynamicStatic


3D data


scanner




2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

14

Dynamic 3D Dynamic 3D reconstructionreconstructionReconstruction at video rates →3D video!

From static reconstruction best results: ‘One shot’ active illumination + Symmetric Dynamic Programming (SDPS) Project pattern every other frame to get a clean texture

Monochrome stereo pair of video cameras + 3rd colour web camera

obtains colour texture.

(1)

(2)

(3)

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

15

Colour texture generationColour texture generation

Low resolution colour information combined with high resolution luminance information

Next step: colour video cameras

Final textureColour image (reprojected into same reference

frame)

Monochrome image

+ →

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

16

VideosVideos

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Patternless reconstruction

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Marker based expression modelling

3D video scanner

3D reconstruction


DynamicStatic


3D data


scanner




2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

19

Marker based expression modellingMarker based expression modelling

Data driven: Stereo web-cameras, face

markers. Head motion - rigid Expressions - non-rigid

Tracked 3D points Unique 3D face model

mappingVirtual muscle animation

17 active muscles Muscle inverse kinematics (IK) –

Jacobian Transpose

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

20

Example videosExample videos

Surprise

Happiness – easiest to reproduce

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Anger – needs teeth!

Disgust – pursing of mouth & closing of eyes not represented

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Video based expression modelling

3D video scanner

3D Reconstruction


DynamicStatic


3D data


scanner




2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

23

3D video based expression 3D video based expression modelling modelling

Image blending Novel face expressions from

multiple video sequencesInteractiveLow preparation Not data driven

Dense depth data – cf. marker system

Video based → realistic 3D movement and texture

Reconstruction data directly used for expression modelling

Sub-region masks

11 control pointsControl points

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

24

Synthetic expression resultsSynthetic expression resultsSadness: lower face region, anger: right eye region, surprise: left

eye region

Happiness: lower face region, surprise: left and right eye regions

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Synthetic expression resultsSynthetic expression resultsFear: lower face region, happiness: right eye region, anger: left eye

region

Disgust: lower face region, anger: left and right eye regions

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

26

ContributionsContributions

3D face reconstruction and expression modelling system Unique tool-setLow-cost, off-the-shelfVision based

To 3D face reconstruction: Extensive reconstruction comparison Face database Dynamic reconstruction system for 3D video: SDPS + pattern

To expression modelling: Marker based performance capture system

Muscle based IK animation system, unique mapping approach Video based expression system – realistic, less flexible

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Future work and perspective Many areas for future research

Refine hardware - better reconstructions ( low-cost? ) Markerless motion capture - face ( feature ) tracking

Statistical analysis on video data Active appearance model (AAM)

New animation system (out of scope)Full body → complete character

Synergy of computer vision and computer graphics!Physical models for animationComputer vision tools

Especially 3D video & markerless motion capture

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N Questions?

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Timeline of Experiments

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

31

Universal expressionsUniversal expressionsEkman - 1987Ekman - 1987

Sadness Anger Happiness

Fear Disgust Surprise

Recognisable in every culture! Used as exemplar expressions to judge my results

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Types of binocular stereo algorithm

Local vs global optimisation WTA

SAD, SSD Chen-Medioni –

local method with explicit surface constraints Seed propagation approach

Dynamic programming – 1D optimisation SDPS – markov chain DPM

Cubic algorithms – 2D optimisation Markov random field Energy minimisation Graph-cut (KZ1, RoyCox), Belief Propagation,

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Types of photometric stereo algorithm

Experiment focused on integration methodsAssumes C² continuity – i.e. a smooth second derivative

Local optimisation – based on curve integralsFour path integrationShapelet

Explicit summation of basis functions

Global optimisationFCV – Frankot Chellappa Variant

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Structured lighting techniques

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Body modelling and animationBody modelling and animation

Body: generic skinned animationSkeletal hierarchy, fully articulated

• The body model with underlying skeleton

• Each bone of the skeleton has a region of influence, denoted in green

• The bones of the hand

• Movement of the forearm

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

36

Interactive personalised avatar creatorInput photographRBF mapping

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


ResultsResults

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Results

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Results

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

3D video based expression system 3D video based expression system overviewoverview

Acquire sequences of individual expressions using dynamic 3D face reconstruction system.

Expression sequences start from a neutral state. Test subject’s head remains in the same position for every sequence A reference texture and depth map are taken from the neutral expression

and used as the base for all image regions

11 control points are manually annotated on video sequences.

Future work to automate this process.

Six sub-regions manually defined on the face.A sub-region’s texture and depth updated by dragging a

control point residing in it and its currently chosen expression sequence.


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

System conclusionsSystem conclusions

Sinusoidal interpolation instead of a linear one. This roughly models the biphasic nature of skin

Realistic animations are created as motion is derived from 3D video sequences of real-life test subjects.

A user can create unnatural but interesting looking expressions that can convey a comical feel

Texture maps sourced from video sequences solves the loss of detail in the marker based approach

However, apart from the control points that were manually specified, no points on the face surface are tracked

Results could be refined by improving the quality of 3D video reconstruction.


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Test subject placement

Subject can be placed with knowledge of required view area, sensor size, and camera lens:

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Projector synchronisation

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

RBF mapping approach

Radial Basis FunctionsUser specified point correspondences on generic

model and 3D face dataSpecify divergences between data

For each dimension (in 3D)Find RBF approximation of (1D) displacements within the 3D

space of specified points.

Using this RBF approximation all 3D points from the generic model can be mapped to the 3D face data proportions

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Marker tracking

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Rigid and non-rigid motion

Anchor markers:

Rigid orientation:

Remove rigid motion by using transpose of orientation and centre of gravity of anchors

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Muscle inverse kinematics

Forward kinematics is the calculation of a new position g of an end effector by specifying updates to parameters of a kinematic chain

Inverse kinematics is the calculation of parameters for a kinematic chain to meet a desired goal position g, when starting from an initial position e.

Kinematic chain consists of jointsEach joint has DOF’s – its animitable parameters,

E.g. 3-DOF for position, 1-DOF for orientation around one axis (position of joint implied through kinematic chain transformation)

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Jacobian Transpose approach

FK: IK:

e = current end effector position

g = goal end effector position

d = change in end effector position

First order estimate in positional change:

Change in parameters:

Jacobian Transpose estimate:

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Estimate assured to move closer to the goal g:

Always moving in a direction less than 90 degrees from d

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Interface between Raw Data and Generic Model

User specifies a ‘minimal’ set of correspondences between raw and generic data

Radial Basis Functions (RBF) used as the interpolant

Model with animation system

Correspondences made and mapped via RBF with a final nearest point map and texture projectionDepth map

Results in a custom face with animation system in place

•Feature extraction as a goal

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Face Animation Model Research primarily based on Terzopoulos, Waters, Parke collective work in

the field

Physically based model for skin tissue Mass-Spring system Epidermal – Fascial – Skull levels of tissue Forces are applied to the tissue to simulate muscle contractions Springs bring elasticity, allow forces to propagate-> stretches and pulls!

Abstract muscle definitions Decoupled from model Warped via RBF also Two types

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Face Animation System Forces

Model the behaviour of the tissueReactionary over the evolution of applied muscle forces:

Skull Penetration Constraint

otherwise 0

0 when iniii

ni

i

f nnnfs

jjjjj llc s-g )( Spring Forces

ji gg ,

jc

- rest length of springjl

- biphasic spring constant

jijj l/)( xxs jl - current length of spring

- spring direction vector

Muscle Forces

Applied to fascia nodes based on the abstract muscle definitions………..(explained later)

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Face System Forces

ei

ei

ei

ei nq pp21 kVVk ee

Volume Preservation Force

ein

- current and rest nodal positions with respect to center of mass of element ‘e’

ei

ei p,p

- epidermal normal for volume element ‘e’

21 ,kk - force scaling constants

These forces allows for tissue form restitution

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Linear Muscle

Linear Muscle:Applies forces to nodes inside it’s angular rangeInfluence is weighted by angle and radius from muscle vector

1

1

pv

pvpp akr

Displacement formula:

)cos(

)cos(1

a

)(sector inside for );

2cos(

)(sector inside for );2

1cos(

msrn

1mn1

ppppp

pppvp

sf

s

s

RR

RD

R

D

r

Where and

‘k’ = muscle contraction increment.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Ellipsoid Muscle

Ellipsoid Muscle:Acts like a string bag Application of force weighted by radius onlyDefined by major and 2 minor axesCan generate puckering effects

1

1

pv

pvpp kr

Displacement formula:

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Physical Simulation

Layered Tissue Model is a physically based one Euler integration is used to run the simulation

titi

ti

ti

ti

ei

ti hsqgvfa

~~~~1 i

im

Equations of motion

ti

ti

tti avv t

tti

ti

tti vxx t

Velocity dependent damping co-efficient. Controls the rate of dissipation of kinetic energy which eventually brings the facial mesh to rest.

Velocity

Nodal position

Acceleration

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

6 pre-built expressions

Sadness

Anger

Happiness

Fear Disgust

Surprise

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


General conclusions and future General conclusions and future work (old version)work (old version)

Investigated low-end and cost effective equipment to create self-contained tools that can run entirely on any end user system.A unique solution has been proposed for 3D face

reconstruction and expression modelling with appropriate hardware

Synchronised audio capture of speech sequences would greatly add to the realism.

The attachment of the face model to a body would complete the system, giving a fully realised virtual human.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N Conclusions drawn from the static reconstruction experiment formed the basis of a dynamic 3D face reconstruction system

3D face reconstructions have no notion of higher order surface structure and are just a collection of points.

This structure was addressed in the second part of this thesis which investigated face expression modelling.

Marker motion was combined with a muscle inverse kinematics framework to drive the facial animation system.

A static face texture impacts on the visual result, as illumination cues such as wrinkles and shadowing over the face are lost.


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N To supplement the work on 3D faces, a body model was created

Interactive 3D video expression creation system which ties together 3D face reconstruction and expression modelling.

Main problems faced were dealing with hardware constraints

But focus on low-cost and off-the-shelf solutions

Focused on the computer vision aspects of facial reconstruction and expressions as opposed to computer graphics


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N A combined marker and dense 3D reconstruction system could be developed, to incorporate further information for a muscle inverse kinematics system

Highly detailed face animation is best served by taking advantage of real world data in the form of digital images and computer vision processing

Advanced physical models of faces meets the tools and approaches investigated within this thesis


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N Details in important areas of the face that are not currently modelled include the eyelids, lips, teeth and inner mouth.

Loss of texture detail in the forehead where the wrinkles are lost

Fine tuning of the preset muscle locations and parameters when mapping a new face model was sometimes needed to improve results or correct muscles


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

System analysisSystem analysis

Capable of reproducing facial expressions from marker motion.Low cost hardware, easy retargetting to other models.

Many differences between test-subjectsExpression articulation, muscle control, gross face

movement

Difficulty in performing when no emotional tie involved.

Easy to understand the need for directors in performance capture situations.

Some user direction was needed to describing to a test subject how an expression should be created


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N Issue: potentially multiple solutions for a vertex position when influenced by multiple muscles

Illumination conditions affect coloured marker detection

Reflectance properties of the skin surface are important visual cues.Missing in this systemAddressed in next chapter.


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Body modelling and animation Body modelling and animation systemsystem

Skinned animation system was chosen for real-time capability and ease of creating new body posesA posable skeleton is associated with a body model (skin

surface description), usually in the form of geometric data Forward and inverse kinematics used for animation Skin surface under new pose is determined based on skeletal bone

local coordinate systems and blending between adjacent bones.

Future work: combine with the face reconstructions and animation systems.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Static reconstruction experiment Static reconstruction experiment Evaluated three computer vision approaches to 3-D face reconstruction.

Binocular stereo: passive. Structured lighting: active. Photometric stereo: active.

Two main aims: Determine their effectiveness for 3D facial reconstruction.

Accuracy, time complexity. Provide a new and alternative test set for evaluating algorithms.

Database of faces. We focus on stereo vision algorithms.

Integrated lab environment designed. 12 algorithms tested in total. Results compared to ground truth data obtained from a commercial 3D scanner.

Summary: Active illumination techniques are most accurate. Stereo algorithm rankings were different from that expected. ‘One shot’ active illumination coupled with a traditional stereo algorithm a strong

choice.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Photogrammetry LaboratoryPhotogrammetry LaboratoryOptical ‘Range’.

Integrated.Multiple systems view a

common scene. Stereo bench.

Sideways for face capture!

Projector for structured lighting.

Light sources for photometric stereo.

Commercial 3D scanner. Solutionix Rexcan 400.

Example Data:

Depth map Perspective visualisation

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


System calibration:Estimates intrinsic and extrinsic

camera parameters I.e. camera projection matrices For cameras:

A calibration cube - 63 markings defines a world co-ordinate system

Tsai calibration For the lights:

A calibration sphere - estimates directions to lights

Simple analytic derivation, inaccurate

Could also calibrate the projector using Tsai’s algorithm

CalibrationCalibration

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

World to image co-ordinates

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Rectification: The camera calibration matrices were used to rectify images.

The resultant image pairs meet the epipolar constraint.

Data processing: Data must be compared in a common co-ordinate frame. Alignment done using a semi-automatic process involving 3D object rigid

transformations. Small number of manual correspondences made. Data projected into disparity space.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

73

Database of 15 people createdData acquired

from all systemsRexcan ground

truth

Test-bed for new algorithms

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Binocular StereoBinocular Stereo Approach 1: Binocular stereo (stereo

vision). Passive. Active research area in our department. Textureless regions cause problems.

Remedy via active illumination.

Test a set of local and global algorithms.

Sum of Absolute Differences (SAD)

Dynamic Programming Method (DPM)

Symmetric Dynamic Programming Stereo (SDPS)

Graph Cut (GC)

Belief–Propagation (BP)

Chen and Medioni (CM) – seed based algorithm

System Geometry (side view)

Tested algorithms: Use two Canon digital SLRs –

6 Mpixels 1536 x 1024

resolution.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


•Add texture to face.•Used with standard stereo algorithms.

Structured LightingStructured Lighting Approach 2: Structured Lighting.

Active approach. Depth inferred in the same manner as stereo.

Augment stereo system with a colour projector.

Add structure to scene -> break homogeneity.

Projects 800 x 600 pixel image.Acer PL111 LCD Projector.

Interested in ‘one shot’ patterns over Gray code.

Time-multiplexed structured lighting using Gray code

Direct coding - ‘one shot’ colour gradation pattern.

Direct coding - ‘one shot’ colour strip pattern.

6 of the Gray code projections:

System Geometry (side view)

Tested algorithms:

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Photometric StereoPhotometric Stereo

Approach 3: Photometric Stereo (PSM). Face viewed under 3 different known lighting

conditions. Depth by integrating recovered surface orientation

map. Albedo independent approach used.

Three 150W light sources.

Analysed gradient field integration techniques.

System Geometry (top-down view)

Frankot-Chellappa Variant (FCV) Fourier based integration.

Four-Scan method Local integration paths.

Shapelets Summation of correlated basis functions.

Tested algorithms:

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


A Collection of ReconstructionsA Collection of Reconstructions

Groundtruth

Graycode

FCV SAD SDPS GC CM

Example depth maps:

Binocular Stereo

Photometric Stereo

Structured lighting

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Photometric Stereo ResultsPhotometric Stereo ResultsReconstruction accuracy:

17 test subjects.

Gold standard result for accuracy

97

7154

69

P <=2,%Method

Percentage of errors less than 2 disparity units

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Passive Stereo ResultsPassive Stereo ResultsReconstruction accuracy:

97

89

7977

8073

88

P<=2,%Method

GC

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Stereo + Gradation PatternStereo + Gradation PatternReconstruction accuracy:

97

9084

83

8577

89

P<=2,%Method

GC

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Stereo + Strip PatternStereo + Strip PatternReconstruction accuracy:

97

9392

92

9389

92

P<=2,%Method

GC

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Improvement to Stereo from Active Improvement to Stereo from Active IlluminationIllumination

Addition of the Strip colour pattern.SAD stereo algorithm:

Pattern colour should avoid skin tones.

Strip pattern SAD - without pattern

SAD - with strip pattern

Depth map

P<=2 = 93%P<=2 = 80%

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N

Error map example

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Gray code approach most accurate.Slower acquisition time.

Look to alternative ‘one shot’ approaches.Photometric stereo least accurate.Our test set has high resolution images and large

disparity ranges.O(n3) stereo algorithms – GC, BP – inappropriate.

Long processing time.Parameter setting difficult.

Our results differ from the Middlebury rankings: http://cat.middlebury.edu/stereo/

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


All results contain errors. Need post processing to clean up data. Even for the commercial 3D scanner.

Faces have many unique properties posing a challenge for 3D reconstructionHuman sensitivity to errors in reconstruction - we see faces

all the time.For computer vision:

Specularities. Anistropic reflectance of hair. Sub-surface scattering. Large homogenous regions.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


Static reconstruction conclusion Static reconstruction conclusion and analysisand analysis

Framework and test-bench for active and passive 3-D acquisition systems designed.

Three computer vision approaches tested. 12 algorithms altogether.

Analysed accuracy of algorithms for 3D face reconstruction. Data compared to scanner benchmark.

Provided new alternative test set to Middlebury for testing stereo algorithms High resolution images of faces.

Passive stereo combined with active illumination a promising approach. Want a one shot approach for faces (moving object). SDPS + Strip pattern. Leads to real-time spatio-temporal acquisition.

Acquire 3D face performance.

2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


2009

The

Uni

vers

ity o

f Auc

klan

d | C

ompu

ter S

cien

ce |

New

Zea

land

PR

ES

EN

TA

TIO

N


A generic face model with an abstract muscle animation system was designed during my Master’s thesis.

Refined for PhD thesis.Can be personalised with 3D data and texture information from the

static reconstruction experiment using a custom RBF mapping procedure.

• Generic morphable face with linear and ellipsoid muscles • A biomechanical tissue model

• Example of muscle contraction

Documents

3D Human Face Reconstruction and Expression Modelling