CSCI480/582 Lecture 31 Chap 5.3Performance Driven Facial Animation

Apr, 13, 2009

Outline What differs facial animation from body

animation? Face model Facial motions and FACS Facial motion synchronization with audio

production Performance-driven techniques

Face Model for Facial Animation

In modeling face, Four aspects needs to be considered Structure: vertex distribution and topology to

represent face in geometry

Appearance: color, texture, light reflection of face

Face Model - Geometry Dense mesh around the “holes” The density of the mesh determines the level of details in facial deformation Eyes, ears, and mouth cavity Teeth and tongue

Face Model - Appearance

Using Texture Mapping

Unwinding the “cylinder” surface by concatenating multiple camera views to form the texture map

More Advanced Skin Appearance Model

The skin is translucent Light is scattered through

multiple layers, then reflected from a sub-layer

Bidirectional Reflectance Distribution Function is often used

Bidirectional Reflectance Distribution Function(BRDF)

A 4-dimensional function that defines how light is reflected at an opaque surface

Input:

incoming light direction wi

Outgoing direction wo

Output:

Ratio of reflected radiance along wo to irradiance incident along wi

Facial Motion Expressions involving

All front facial mesh

Eye ball orientation

Speeches involving Lip and jaw

Tongue

The combination of expressions and speeches

Facial Acting Code System

Originally developed by Ekman and Friessen in 70s

Based on the physiology of facial muscles

A common standard to systematically categorize the physical expression of emotions

FACS – The Codes for Action Units

Action Descriptors

Inner Brow Raiser -- Frontalis (pars medialis)

2 Outer Brow Raiser -- Frontalis (pars lateralis)

4 Brow Lowerer -- Corrugator supercilii, Depressor supercilii

5 Upper Lid Raiser -- Levator palpebrae superioris

6 Cheek Raiser -- Orbicularis oculi (pars orbitalis)

7 Lid Tightener -- Orbicularis oculi (pars palpebralis)

9 Nose Wrinkler -- Levator labii superioris alaeque nasi

10 Upper Lip Raiser -- Levator labii superioris

…

66 Action descriptors

Demo

Lip-Sync by Viseme Morphing Use reference video to rotoscope

the shape of the 3D model for key gestures

Use a set of viseme corresponding to phonemes, then interpolate between viseme gestures

Such approach often is not detailed enough for finer visual speech animation

Facial Motion Driven by Mocap Data

Captures subtle details of facial expressions

But, Mocap Data need be to cleaned for good rendering effect. Often this step is labor intensive.

How can we make Mocap more flexible?

FACS-Viseme Solver Use a small set of Mocap data to synthesize a

much larger range of facial motions Each piece of mocap data in the small recording set

represent an Action Unit

Use Action Unit analysis to decompose recorded Mocap data for easy post editing

Use audio-visual mapping analysis to align mocap segment for new speech

N

kkkAUwimocap

1

)(