Faces and Image-Based Lighting - 國立臺灣大學

Faces and Image-Based Lighting

Digital Visual EffectsgYung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz, Alex Efros, Li-Yi Wei and Paul Debevec

Outline

• Image-based lighting

• 3D acquisition for faces• Statistical methods (with application to face

super-resolution)p )• 3D Face models from single images• Image based faces• Image-based faces• Relighting for faces

Image-based lightingImage based lighting

Rendering

• Rendering is a function of geometry, reflectance lighting and viewingreflectance, lighting and viewing.

• To synthesize CGI into real scene, we have to t h th b f f tmatch the above four factors.

• Viewing can be obtained from calibration or structure from motion.

• Geometry can be captured using 3D y p gphotography or made by hands.

• How to capture lighting and reflectance?• How to capture lighting and reflectance?

Reflectance

• The Bidirectional Reflection Distribution FunctionGiven an incoming ray and outgoing ray– Given an incoming ray and outgoing raywhat proportion of the incoming light is reflected along outgoing ray?

surface normalsurface normal

Answer given by the BRDF:

Rendering equation

)ω,p( iiL

iω

)ω,p( iiL

poω

)ωp,( ooL5D light field

)(L )(L

5D light field

)ωp,( ooL )ω,p( oeL

iiiio ωθcos)ωp,()ω,ωp,(2

dLi iiiio )p,(),p,(2 is

Complex illumination)ωp,( ooL )ω,p( oeL


dLf is

)ωp,( oB iiiio ωθcos)ωp,()ω,ωp,(2

dLf ds)ω( opB iiiiω, ωθcos)ω()ω(

2 odLf ds p

p q

Point lightsClassically, rendering is performed assuming point light sourceslight sources

directional source

Natural illuminationPeople perceive materials more easily under natural illumination than simplified illuminationnatural illumination than simplified illumination.

I t R D d T d Ad lImages courtesy Ron Dror and Ted Adelson

Natural illuminationRendering with natural illumination is more expensive compared to using simplified expensive compared to using simplified illumination

directional source natural illumination

Environment maps

Miller and Hoffman 1984Miller and Hoffman, 1984

HDR lighting

Examples of complex environment light Examples of complex environment light

Complex illumination)ωp,( ooL )ω,p( oeL


dLf is

)ωp,( oB iiiio ωθcos)ωp,()ω,ωp,(2

dLf ds)ω( opB iiiiω, ωθcos)ω()ω(

2 odLf ds p

reflectance lighting

B th h i l f tiBoth are spherical functions

Function approximation

• G(x): the function to approximate• B1(x), B2(x), … Bn(x): basis functions• We want

)()( xBcxG i

n

i )()(1

xBcxG ii

i

• Storing a finite number of coefficients ci gives an approximation of G(x)

Function approximation• How to find coefficients ci?

Mi i i – Minimize an error measure• What error measure?

– L2 error2

2

])()([2

I iiiL xBcxGE

• Coefficients dxxBxGBGc )()(X

iii dxxBxGBGc )()(

Function approximation• Basis Functions are pieces of signal that can be used to

produce approximations to a functionproduce approximations to a function

1c

2c

2

c 3c

Function approximation• We can then use these coefficients to reconstruct an

approximation to the original signalapproximation to the original signal

1c

2c

2

c 3c

Function approximation• We can then use these coefficients to reconstruct an

approximation to the original signalapproximation to the original signal

xBcN

ii i

ii1

Orthogonal basis functions• Orthogonal Basis Functions

Th f ili f f ti ith i l – These are families of functions with special properties

jiji

dxxBxB ji 01

ji0

– Intuitively, it’s like functions don’t overlap each other’s footprint

A bit lik th F i t f b k • A bit like the way a Fourier transform breaks a functions into component sine waves

Integral of product

dxxGxFI xBfxF )( xBgxG )(

i

ii xBfxF )( j

jj xBgxG )(

i jjjii dxxBgxBfdxxGxF )()(

iijiji GFdxgfdxxBxBgf ˆˆ)()( ii j

)ω(B ωθcos)ω()ω( dLf)ω( opB iiiiω, ωθcos)ω()ω(2 o

dLf ds p

Basis functions• Transform data to a space in which we can

capture the essence of the data bettercapture the essence of the data better• Spherical harmonics, similar to Fourier

t f i h i l d i i d i PRTtransform in spherical domain, is used in PRT.

Real spherical harmonics• A system of signed, orthogonal functions over

the spherethe sphere• Represented in spherical coordinates by the

f ti function

02 PK mm

00

,cossin2,coscos2

, mm

PmKPmK

y ml

ml

ml

ml

ml

00

,cos,cossin2,

00 mm

PKPmKy

ll

lll

where l is the band and m is the index within the band

Real spherical harmonics Reading SH diagrams

Thisdi idirection

+–

+

Not thisdirection

Reading SH diagrams

Thisdi idirection

+–

+

Not thisdirection

The SH functions

00y0y

11y 1

1y

12y 2

2y02y1

2y2

2y

The SH functions Spherical harmonics

Spherical harmonics

0 ( )Y m

0 ( , )lmY 1

1

l

1xy z

22xy yz 23 1z zx 2 2x y

-1-2 0 1 2

SH projection• First we define a strict order for SH functions

mlli 1

• Project a spherical function into a vector of• Project a spherical function into a vector ofSH coefficients

ii dssysfc S

ii

SH reconstruction• To reconstruct the approximation to a function

2N

~ N

ii sycsf0i

• We truncate the infinite series of SH functions to give a low frequency approximationg q y pp

Examples of reconstruction

An example• Take a function comprised of two area light

sourcessources– SH project them into 4 bands = 16 coefficients

9080930067903291 ,.

2380042508370317000106420

27800417009400908093006790

,.,.,.,.,.,,.,,.

,.,.,.

238004250 .,,.

Low frequency light source• We reconstruct the signal

U i l th ffi i t t fi d l f – Using only these coefficients to find a low frequency approximation to the original light source

SH lighting for diffuse objects• An Efficient Representation for Irradiance

Environment Maps Ravi Ramamoorthi and Pat Environment Maps, Ravi Ramamoorthi and Pat Hanrahan, SIGGRAPH 2001A ti• Assumptions– Diffuse surfaces– Distant illumination – No shadowing, interreflection

)( op,ωB iiiio ωθcos)ωp,()ω,ωp,(2

dLf dss)n()( Epn)B(p,

irradiance is a function of surface normal

Diffuse reflection

B EB Edi i flradiosity

(image intensity)reflectance

(albedo/texture)irradiance

(incoming light)

×=

k li hquake light map

Irradiance environment mapsL n

Illumination Environment Map Irradiance Environment Map

p p

dnLnE )(

Spherical harmonic expansionExpand lighting (L), irradiance (E) in basis functions

l

( , ) ( , )l

lm lmL L Y

0l m l

l

0( , ) ( , )lm lm

l lE E Y

0l m l

= .67 + .36 + …

Analytic irradiance formula

Lambertian surface acts like low-pass filter 2 / 3

E A LlA / 4

lm l lmE A L / 4

0

l0 1 2cosine term

2 1

22

( 1) !2( 2)( 1) 2 !

l

l l l

lA l evenl l

2

( )( ) 2 !

9 parameter approximation

i Order 0Exact image Order 01 term

m

RMS error = 25 % 0 ( , )lmY l

m

1

2

xy z

-1-2 0 1 22 xy yz 23 1z zx 2 2x y

9 Parameter Approximation

i Order 1Exact image Order 14 terms

m

RMS Error = 8% 0 ( , )lmY l

m

1

2

xy z

-1-2 0 1 22 xy yz 23 1z zx 2 2x y

9 Parameter Approximation

i Order 2Exact image Order 29 terms

m

RMS Error = 1% 0 ( , )lmY l

m

For any illumination, average error < 3% [Basri Jacobs 01]

1

2

xy z

error < 3% [Basri Jacobs 01]

-1-2 0 1 22 xy yz 23 1z zx 2 2x y

Comparison

Incident Irradiance map Irradiance mapillumination

300x300

pTexture: 256x256

Hemispherical

pTexture: 256x256

Spherical HarmonicIntegration 2Hrs Coefficients 1sec

Time 300 300 256 256 Time 9 256 256

Complex geometry

Assume no shadowing: Simply use surface normalAssume no shadowing: Simply use surface normal

y

Natural illuminationFor diffuse objects, rendering with natural illumination can be done quicklyillumination can be done quickly

directional source natural illumination

Video

Acquiring the Light Probe HDRI Sky Probe

Clipped Sky + Sun Source Lit by sun onlyy y

Lit by sky onlyy y y Lit by sun and skyy y

Illuminating a Small Scene

Real Scene Example

• Goal: place synthetic objects on tableGoal: place synthetic objects on table

Light Probe / Calibration Gridg

Modeling the Scene

light-based modellight-based model

real scene

The Light-Based Room Model

Rendering into the Scene

• Background PlateBackground Plate

Rendering into the scene

• Objects and Local Scene matched to SceneObjects and Local Scene matched to Scene

Differential rendering

• Local scene w/o objects, illuminated by modelLocal scene w/o objects, illuminated by model


==-


++

Differential Rendering

• Final ResultFinal Result

Environment map from single image? Eye as light probe! (Nayar et al)

Results Application in “Superman returns”

Capturing reflectance Application in “The Matrix Reloaded”

3D acquisition for faces3D acquisition for faces

Cyberware scanners

face & head scanner whole body scannery

Making facial expressions from photos

• Similar to Façade, use a generic face model and view dependent texture mappingand view-dependent texture mapping

• Procedure1. Take multiple photographs of a person2. Establish corresponding feature points3. Recover 3D points and camera parameters 4. Deform the generic face model to fit points5. Extract textures from photos

Reconstruct a 3D model

input photographs

generic 3D pose more deformedgeneric 3D face model

pestimation features model

Mesh deformation

– Compute displacement of feature pointsApply scattered data interpolation– Apply scattered data interpolation

generic model displacement deformed model

Texture extraction

• The color at each point is a weighted combination of the colors in the photoscombination of the colors in the photos

• Texture can be:– view-independent – view-dependent

• Considerations for weighting– occlusion– smoothness– positional certaintyp y– view similarity

Texture extraction Texture extraction

Texture extraction

view-independent view-dependent

Model reconstruction

Use images to adapt a generic face modelUse images to adapt a generic face model.

Creating new expressions

• In addition to global blending we can use: R i l bl di– Regional blending

– Painterly interface


New expressions are created with 3D morphing:

+ =+

/2 /2

Applying a global blend


+x x+ =

Applying a region-based blend


+ + ++ + +

=

Using a painterly interface

Drunken smile Animating between expressions

Morphing over time creates animation:

“neutral” “joy”

Video Spacetime faces

Spacetime faces

black & white cameras

color cameras

video projectors

time

time

Face surfaceFace surface

time

stereo

time

stereo active stereo

time

spacetime stereostereo active stereo

timeSpacetime Stereo

surface motionsurface motion

time=1



time=2



time=3



time=4



time=5



Better • spatial resolution• temporal stableness

time

• temporal stableness

Spacetime stereo matching Video

Fitting FaceIK

Animation 3D face applications: The one

3D face applications: Gladiator

extra 3Mextra 3M

Statistical methodsStatistical methods

Statistical methods

para observedf(z)+z ypara-

metersobserved

signal

)|(max* yzPz Example: super-resolution)|(max yzPz

z)()|(max zPzyP

super-resolutionde-noisingde-blocking

)(max

yPz de-blocking

Inpainting

)()|(min zLzyLz

…

Statistical methods

para observedf(z)+z ypara-

metersobserved

signal

)()|(min* zLzyLz )()|(min zLzyLzz

2)(zfy data a-priori2evidence knowledge

Statistical methods

There are approximately 10240 possible 1010 There are approximately 10 possible 1010 gray-level images. Even human being has not seen them all yet. There must be a strong seen them all yet. There must be a strong statistical bias.

Takeo KanadeTakeo Kanade

Approximately 8X1011 blocks per day per person.

Generic priors

“S th i d i ”“Smooth images are good images.”

x

xVzL ))(()( x

2)( dd Gaussian MRF )( ddGaussian MRF

Tdd 2

TdTd

TdTTd

d

)(2)( 2Huber MRF

Generic priors Example-based priors

“E i ti i d i ”“Existing images are good images.”

six 200200Images Images 2,000,000pairspairs

Example-based priors

L(z)

Example-based priors

high-resolution

low-resolution

Model-based priors

“Face images are good images whenFace images are good images whenworking on face images …”

Parametric model Z=WX+ L(X)model

)()|(min* zLzyLz )()|(yz

)()|(min* XLWXyLX

**)()|(min

WXzXLWXyLX

x

PCA• Principal Components Analysis (PCA):

approximating a high dimensional data setapproximating a high-dimensional data setwith a lower-dimensional subspace

****

******

**** **

**

** **

**

**First principal componentFirst principal componentSecond principal componentSecond principal component

Original axesOriginal axes

**

**** **

****** **

**** **********

Data pointsData points

PCA on faces: “eigenfaces”

AverageAverageFirst principal componentFirst principal component

AverageAveragefaceface

OtherOthercomponentscomponents

For all except average,For all except average,o a e cept a e age,o a e cept a e age,“gray” = 0,“gray” = 0,

“white” > 0,“white” > 0,“black” < 0“black” < 0black < 0black < 0

Model-based priors

“Face images are good images whenFace images are good images whenworking on face images …”

Parametric model Z=WX+ L(X)model

)()|(min* zLzyLz )()|(yz

)()|(min* XLWXyLX

**)()|(min

WXzXLWXyLX

x

Super-resolution

(a) (b) (c) (d) (e) (f)

(a) Input low 24×32 (b) Our results (c) Cubic B-Spline(a) Input low 24×32 (b) Our results (c) Cubic B Spline

(d) Freeman et al. (e) Baker et al. (f) Original high 96×128

Face models from single imagesFace models from single images

Morphable model of 3D faces

• Start with a catalogue of 200 aligned 3D Cyberware scansCyberware scans

• Build a model of average shape and texture • Build a model of average shape and texture, and principal variations using PCA

Morphable model

shape examplars texture examplars

Morphable model of 3D faces

• Adding some variations

Reconstruction from single image

Rendering must be similar to the input if weguess rightg g

Reconstruction from single image

prior

shape and texture priors are learnt from database

ρ is the set of parameters for shading including camera pose, lighting and so onp , g g

Modifying a single image

Animating from a single image Video

Exchanging faces in images Exchange faces in images

Exchange faces in images Exchange faces in images

Exchange faces in images Morphable model for human body

Image-based faces(lip sync.)

Video rewrite (analysis)

Video rewrite (synthesis) Results

• Video database2 i t f JFK– 2 minutes of JFK

• Only half usable• Head rotation• Head rotation

training video

R d liRead my lips.

I never met Forest Gump.

Morphable speech model Preprocessing

Prototypes (PCA+k-mean clustering)

W fi d I d C f h t t iWe find Ii and Ci for each prototype image.

Morphable model

analysisI α β

analysis

synthesis

Morphable model

analysis

synthesis

Synthesis

Results Results

Relighting facesRelighting faces

Light is additivelamp #1 lamp #2

Light stage 1.0 Light stage 1.0

64x32 lighting directions

Input images Reflectance function

occlusion flare

Relighting Results

Changing viewpoints Results

3D face applications: Spiderman 2 Spiderman 2

real syntheticreal synthetic

Spiderman 2

videovideo

Light stage 3

Light stage 6 Application: The Matrix Reloaded

Application: The Matrix Reloaded References• Paul Debevec, Rendering Synthetic Objects into Real Scenes:

Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography Illumination and High Dynamic Range Photography, SIGGRAPH 1998.

• F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski Synthesizing realistic facial expressions from Szeliski. Synthesizing realistic facial expressions from photographs. SIGGRAPH 1998, pp75-84.

• Li Zhang, Noah Snavely, Brian Curless, Steven M. Seitz, S ti F High R l ti C t f M d li g d Spacetime Faces: High Resolution Capture for Modeling and Animation, SIGGRAPH 2004.

• Blanz, V. and Vetter, T., A Morphable Model for the S th i f 3D F SIGGRAPH 1999 187 194 Synthesis of 3D Faces, SIGGRAPH 1999, pp187-194.

• Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark Sagar, Acquiring the R fl t Fi ld f H F SIGGRAPH 2000 Reflectance Field of a Human Face, SIGGRAPH 2000.

• Christoph Bregler, Malcolm Slaney, Michele Covell, Video Rewrite: Driving Visual Speeach with Audio, SIGGRAPH 1997.

• Tony Ezzat, Gadi Geiger, Tomaso Poggio, Trainable Videorealistic Speech Animation, SIGGRAPH 2002.

Documents

Faces and Image-Based Lighting - 國立臺灣大學