1

Multilinear Representation of Image Ensemblesfor Recognition and Compression

Multilinear Representation of Image Ensemblesfor Recognition and Compression

TensorFaces:

The Problem with Linear (PCA) Appearance Based Recognition

Methods

The Problem with Linear (PCA) Appearance Based Recognition

Methods• Eigenimages work best for recognition when only a single

factor – e.g., object identity – is allowed to vary

• Natural images are the composite consequences of multiple factors (or modes) related to scene structure, illumination and imaging

Decomposition from Pixel Space to Factor Spaces

Decomposition from Pixel Space to Factor Spaces Multilinear Model ApproachMultilinear Model Approach

• Non- linear appearance based technique

• Appearance based model that explicitly accounts for each of the multiple factors inherent in image formation

• Multilinear algebra, the algebra of higher order tensors

• Applied to facial images, we call our tensor technique “TensorFaces” [ Vasilescu & Terzopoulos, ECCV 02, ICPR 02 ]

TensorFaces vs Eigenfaces(PCA)

TensorFaces vs Eigenfaces(PCA)

88%27%

Training: 23 people, 5 viewpoints (0,+17, -17,+34,-34), 3 illuminations

Testing: 23 people, 5 viewpoints (0,+17, -17,+34,-34), 4th illumination

80%61%

Training: 23 people, 3 viewpoints (0,+34,-34), 4 illuminations

Testing: 23 people, 2 viewpoints (+17,-17), 4 illuminations (center,left,right,left+right)

TensorFacesPCAPIE Recognition Experiment

PIE Database (Weizmann)PIE Database (Weizmann)

2

Data OrganizationData Organization

• Linear/PCA: Data Matrix– Rpixels x images

– a matrix of image vectors

• Multilinear: Data Tensor– Rpeople x views x illums x express x pixels

– N-dimensional array– 28 people, 45 images/person– 5 views, 3 illuminations,

3 expressions per person

exilvpp ,,,i

People

Illu

min

atio

nsEx

pres

sions

Views

D

D

Pixe

ls

ImagesD Tensor Decomposition

Complete Dataset

}Learning Stage

Tensor DecompositionTensor Decomposition

Views

Illu

ms.

People

Matrix Decomposition - SVDMatrix Decomposition - SVD

• A matrix has a column and row space

• SVD orthogonalizes these spaces and decomposes

• Rewrite in terms of mode-n products

2xI1IIR∈D

T21 UUD ∑=

2UUD x1x 2 1 ∑=

Pixe

ls

ImagesD

( contains the eigenfaces )1U

D

Tensor DecompositionTensor Decomposition• D is a n-dimensional matrix, comprising N-spaces

• N-mode SVD is the natural generalization of SVD

• N-mode SVD orthogonalizes these spaces & decomposes

D as the mode-n product of N-orthogonal spaces

• Z core tensor; governs interaction between mode matrices

• , mode-n matrix, is the column space of nU )(nD

nn UUUU ... x x x xx 3 21 1 432 ZD = D 321 x xx 32 1 UUU Z=

Multilinear (Tensor) DecompositionMultilinear (Tensor) Decomposition

( ) ( ) ( )ZD vecvec 123 UUU ⊗⊗=

323

32121

3211

3

1

2

1

1

1rrr

rrrr

rr,,, uuu oo∑∑∑=

===

RRR

σ

U1

U2

U3

D Z=

3

N-Mode SVD AlgorithmN-Mode SVD Algorithm

1. For n=1,…,N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD.

2. Solve for the core tensor as follows

)( nDnU

nU

TNN

Tnn

TT UUUU ××××= LL 2211 DZ

Multilinear (Tensor) AlgebraMultilinear (Tensor) Algebra

mode - n tensor flattening

NIII L××ℜ∈ 21A

( )( )Nnnn IIIIII

nLL 1121 +−×ℜ∈A

flattening=

“matricize”

N = 3

pixelsxexpressxillums.xviews xpeoplex 51 UUUUU . ZD 432=

Facial Data Tensor DecompositionFacial Data Tensor DecompositionPeople

Illu

min

atio

nsEx

pres

sions

ViewsD

People

Illu

min

atio

nsEx

pres

sions

ViewsD(views)D

Computing UviewsComputing Uviews

Images

• D(views)- flatten D along the view point dimension• Uviews – orthogonalize the column space of D(views)

Pixe

ls

Images

Computing UpixelsComputing Upixels

D(pixels)

• D(pixels)- flatten D along the pixel dimension

• Upixels – orthogonalize D(pixels)

– eigenimages

N-Mode SVD AlgorithmN-Mode SVD Algorithm1. For n=1,…,N, compute matrix by computing the

SVD of the flattened matrix and setting to be the left matrix of the SVD.

2. Solve for the core tensor as follows

)( nDnU

nU

TNN

Tnn

TT UUUU ××××= LL 2211 DZ

4

I1

J1

x1

• Mode-n product is a generalization of the product of two matrices• It is the product of a tensor with a matrix

• Mode-n product of andNn III x...xx...x1ℜ∈A

I1

I2

I3

I2

J2

J2

I2

J2

I2

= x2

Nnnn IIJII x..xxx.x...x 111 +−ℜ∈B

nn IJ xℜ=M

Mn×= AB

Mode-n ProductMode-n Product

AB M x3

J3

I3

I3

J3

nnNnnn ijiiiii

niNininjniin ma ......

......A

111

111+−∑=×

+−

M

Multilinear (Tensor) AlgebraMultilinear (Tensor) Algebra

N-th order tensor NIII L××ℜ∈ 21A

nn IJ ×ℜ∈Mmatrix (2nd order tensor)

mode - n product:

Mn×= AB ( ) ( )nn AMB =where

( )44444444 344444444 21321321

matrixt coefficien

matrix basis

matrix data

expressillums.viewspeople(pixels)pixels(pixels)

T.UUUUZUD ⊗⊗⊗=

Eigenfaces vs TensorFaces Eigenfaces vs TensorFaces • Multilinear Analysis / TensorFaces:

pixelsxexpressxillums.xviews xpeoplex 51 UUUUU . ZD 432=

( ) T express U(pixels) Z viewsU⊗illums.U⊗ peopleU⊗=

321matrix data

(pixels)D321

matrix basis pixelsU

• Linear Analysis / Eigenfaces:

• TensorFaces subsumes Eigenfaces

TensorFaces:TensorFaces:

Views

Illums.

People

B = Z x5 Upixels

TensorFaces:explicitly representcovariance acrossfactors

PCA:

TensorFaces:

TensorFaces

Mean Sq. Err. = 409.153 illum + 11 people param.

33 basis vectors

PCA

Mean Sq. Err. = 85.7533 parameters

33 basis vectors

Strategic Data Compression = Perceptual Quality

Strategic Data Compression = Perceptual Quality

Original

176 basis vectors

TensorFaces

6 illum + 11 people param.66 basis vectors

• TensorFaces data reduction in illumination space primarily degrades illumination effects (cast shadows, highlights)

• PCA has lower mean square error but higher perceptual error

5

Dimensionality Reduction -Truncation

Dimensionality Reduction -Truncation

∑+∑+∑≤===

− N

NNN

I

Rii

I

Rii

I

Rii σσσ 2222 2

222

1

111

LDD

Iterative Multilinear Model -Dimensionality Reduction

Iterative Multilinear Model -Dimensionality Reduction

• Iterative data reduction approach:– Optimize mode per mode in an iterative way– Alternating Least Squares [Golub & Van Loan] improves data fit

1 - IUUU ... U ... U −Σ+×××××==

Tnnn

N

nNn Nne λ11 ZD

1. Initialize U10, U2

0, …, UN0:

– Compute U1, U2, …, UN using N-Mode SVD and – Truncate each mode matrix

2. Iterate:– U 1

t = D x2 U2t-1T

x3… xN UN

t-1T

U1t = svd( U 1

t(1) )

– U 2t = D x1 U1

tT x3 U3

t-1T … xN UNt-1T

U2t = svd( U 2

t(2) )

3. Z = D x1 U1tT

x2 U2tT

x3… xN UN

tT

Iterative Multilinear ModelIterative Multilinear Model Construction of Projection Basis

Construction of Projection Basis

• Multilinear decomposition allows for the construction of different basis depending on the application needs

• Object Specific TensorFaces: person appearance model; eigenvectors span an individuals set of images

• View Based TensorFaces

• Recognition basis – basis maps an image into people parameter space, Upeople

Person Specific TensorFacesB =Z x1 Upeop. x5 Upixels

Person Specific TensorFacesB =Z x1 Upeop. x5 Upixels

• Basis spanning one person’s set of images

Views

Illumi

natio

ns

Higher -Order Statistics

2nd -Order Statistics(covariance)

Our Nonlinear (Multilinear) Models

Linear Models

Perspective on Our Face Recognition Approach

Perspective on Our Face Recognition Approach

Multilinear PCATensorFaces

PCAEigenfaces

ICA Multilinear ICAIndependent TensorFaces

Vasilescu & Terzopoulos, Learning 2004

6

. ...

... .. .

. .....

... ..

..

.. ......... ..

pixel 1

pixe

l kl

2550

255

. ...

... .. .

. .....

... ..

..

.. ......... ..

PCA ICA

TUSVD =coefficient matrix

basis matrix

UD = TSVCK=

independent components coefficient matrix

pixel 1

pixe

l kl

2550

255

TSV TT −WW

Toy Example: Observed DataToy Example: Observed Data

Toy Example: Hidden VariablesToy Example: Hidden Variables Basis VectorsBasis Vectors

ICA

Hidden Variable RepresentationHidden Variable Representation

• MICA

• MPCA

1. For n=1,…,N, compute matrix by computing the SVD of the flattened matrix and setting to be the left matrix of the SVD. Compute using ICA. Our new mode matrix is

2. Solve for the core tensor as follows

Tnn

Tnn VZWK )(−=

N-Mode ICAN-Mode ICA

)(nDnU

nU

11122

111 −−−− ××××××= NNnn KKKK LL DS

Tnnnn VZUD )()( = ( ) T

n(n)

Tn

Tnn VZWWU −=

TnW

TNN

Tnn

TT −−−− ××××××= WWWW 2211 LL ZS

nK

7

PCA:

TensorFaces:• Multilinear orthog. decomp.• Encodes 2nd order statistics

ICA:

Independent TensorFaces:Multilinear ICA

• Multilinear decomposition• Encodes higher order statistics