Facial Tracking and Animation

Project ProposalComputer System Design

Spring 2004

Todd Belote David BrownBrad Busse Bryan Harris

Problem Background

• Speech Driven Facial Animation (PRISM)– Facial animation from processed speech

• Previous Research (Jablonski & Zavala)– Low cost facial motion and speech processing using facial markers– An infrared camera is used with infrared reflectors to pick up facial

markers– Microphone array for audio– Winnov capture card (640x480 at 30 fps)

• Areas of Desired Improvement– There is no current recovery method for point loss– Feature points skewed in perspective create playback artifacts– There are only 22 feature points, which cannot fully describe a face– Initialization requires mouse-clicking the markers on the first frame– The current algorithm is costly

Problem Statement

• Design a low cost computer system which can decode and analyze audio/video data in real-time and save the resulting analysis to disk.

Design Objectives

• Analyze video at 30 fps; generate an FAP(Facial Animation Parameter) file.

• Continue with current audio analysis.

• All processing done in real-time.

Existing Solutions

• Lin et al, from National Taiwan University use a system with mirrors and camera to determine (x,y,z) coordinates of feature points

• Essa et al, from MIT Media Lab use computer recognition to analyze video in non real-time setting

Microphone Array, Camera, and Capture Card

• Audio sample rates range from 8 to 48 kHz

• Video sample rate will be 30 fps, and can be captured at 640x480 resolution

• Capture card brings audio and video into computer in synchrony

IBM Pupil Cam

• Camera designed to identify human pupils by emmitting infrared light using LEDs

• We will use infrared reflective markers to track facial movements

Design Constraints and Feasibility

• Cost and Speed– The system must run in real-time

• Portability– Hardware specific system

• Quicktime would make a more portable system

• Usability– Point initialization– Freedom of movement for each user– Recovery from point occlusion

Alternative Solutions

• Leave system as-is

• Use mirrors to find (x,y,z) coordinates

• Quicktime libraries

• Individual facial templates

Design Validation

• Can generate FAP file in real-time

• Lost points can be recovered

• User can rotate her head without data loss

• Audio is analyzed in real-time

• Automatic point initialization

Societal, Safety and Environmental Analysis

• Primary use as a research tool

• Low bandwidth face to face phones using texture maps

• Low bandwidth phone support for the deaf

• Audio-visual data recording

Management

• Todd Belote – Data Aquisition

• David Brown – Marker Initialization

• Brad Busse – Marker Tracking Algorithms

• Bryan Harris – Facial Relationships

Scheduling

• Mondays – 4:20 to 6:30 PM– With Steve Ortiz

• Wednesdays – 5:30 to 7:00 PM– With Steve Ortiz and Marco Zavala

• Steve Ortiz – Project Advisor

• Marco Zavala – Previous Project Owner

Scheduling – Gant Chart

Pert Chart