Elliptical Head Tracking Using Intensity Gradients and

Color Histograms

Stan Birchfield

Stanford University

Autodesk Advanced Products Group

http://vision.stanford.edu/~birch

PROBLEM

TILT

PAN

ZOOM

CHALLENGES: * rotation * multiple people * zoom

APPLICATIONS: * video conferencing * distance learning

PREVIOUS METHODS

FL

ES

H-

CO

LO

RE

DO

BJE

CT

S

MU

LT

IPL

EM

OV

ING

PE

OP

LE

AR

BIT

RA

RY

CA

ME

RA

MO

VE

ME

NT

OU

T-O

F-P

LA

NE

RO

TA

TIO

N1. TEMPLATE [Hager & Belhumeur, 1996]

Y Y YN

2. FLESH COLOR [Fieguth & Terzopoulos, 1997]

N N YN

3. BACKGROUND DIFFERENCING [Graf et al., 1996]

Y N NY

Method

Criterion

CUES:• COLOR• MOTION• TEXTURE

INTERIOR BOUNDARY

COMPLEMENTARY CRITERIA

• INTENSITY EDGES• DEPTH & MOTION . DISCONTINUITIES

APPLICATION: 1. Interesting, useful 2. Well-connected to other body parts

WHY FOCUS ON THE HEAD?

GEOMETRIC: 1. Nearly rigid 2. Nearly ellipsoid Easy to model

HEADMODEL (x,y)

Ellipse: vertical aspect ratio = 1.2state s = (x,y,)

SEARCH

velocityprediction

}||,||,|:|{ rp

rp

rp yyyxxxS s

*2

*1

2 tt

pt xxx

*2

*1

2 tt

pt yyy

*1t

pt

LOCAL HEAD SEARCH

)}()({maxarg* icigSisss

s

GRADIENT COLOR

SEARCHRANGE

TWO CHOICES:

1. MAGNITUDE

2. DOT PRODUCT

NORMALIZATION

GRADIENT MODULE

N

iNg i1

1 |)(|)( sgs

N

iNg ii1

1 |)()(|)( sgns

)(min)(max

)(min)(

)(igSsigSs

igSsg

ii

ig ss

ss

s

ellipse normal gradient

COLOR MODULE

COLOR SPACE

HISTOGRAMINTERSECTION[Swain & Ballard 1991]

NORMALIZATION

B-G (8 bins) G-R (8 bins)

B+G+R (4 bins)

)(min)(max

)(min)(

)(icSsicSs

icSsc

ii

ic ss

ss

s

Ni

Ni

iI

iMiIc

1

1

)(

))(),(min()(

s

ss

MODEL

CURRENT

INTERSECTION

SKIN HAIR

SUMMARY OF ALGORITHM

OFF-LINE: 1. Manually place head within ellipse 2. Store model histogram

RUN TIME: 1. At each hypothesized location, compute - Sum of gradient around perimeter - Histogram intersection 2. Move ellipse to location that maximizes sum of two criteria

COMPARISON OF MODULES

• Controls pan, tilt, zoom

• Handles textured backgrounds

• More robust• Large basin of

attraction

• Controls pan, tilt• Keeps off neck• Scale in front of

flesh-colored object• Scale when back

turned

COLORGRADIENT

BASIN OF ATTRACTION

Gradient confused, pulls to left Color pulls to right

COMPUTING TIME

0

10

20

30

40

50

60

70m

agni

tude

dot

prod

uct

colo

r

mag

&co

lor

dot

&co

lor

4x4x18x8x1

Real time (30 Hz)

Com

pu

tin

g ti

me

per

fra

me

(ms)

Search range

(on a 200 MHz Pentium Pro)

CONCLUSION

SUCCESSES: 1. Tracks head in real time on standard hardware 2. Insensitive to - full 360-degree out-of-plane rotation - arbitrary camera movement (including zoom) - multiple moving people - severe but brief occlusion - hair/skin color, hair length, facial hair, glasses

FUTURE WORK: 1. Speed (computer speed and NTSC video standard) 2. Color adaptation, but imprecise localization 3. No explicit model of occlusion