15-826: Multimedia Databases and Data Miningchristos/courses/826.F13/... · Title: 250_svd1.ppt Author: Christos Faloutsos school2 Created Date: 10/23/2013 6:15:41 AM

C. Faloutsos 15-826

1

CMU SCS

15-826: Multimedia Databases and Data Mining

Lecture #17: SVD - part I (definitions) C. Faloutsos

CMU SCS

15-826 Copyright: C. Faloutsos (2013) 2

Must-read Material

•  Numerical Recipes in C ch. 2.6; •  MM Textbook Appendix D

CMU SCS


Outline

Goal: ‘Find similar / interesting things’ •  Intro to DB •  Indexing - similarity search •  Data Mining

C. Faloutsos 15-826

2

CMU SCS


Indexing - Detailed outline •  primary key indexing •  secondary key / multi-key indexing •  spatial access methods •  fractals •  text •  Singular Value Decomposition (SVD) •  multimedia •  ...

CMU SCS


SVD - Detailed outline •  Motivation •  Definition - properties •  Interpretation •  Complexity •  Case studies •  Additional properties

CMU SCS


SVD - Motivation •  problem #1: text - LSI: find ‘concepts’ •  problem #2: compression / dim. reduction

C. Faloutsos 15-826

3

CMU SCS


SVD - Motivation •  problem #1: text - LSI: find ‘concepts’

CMU SCS

15-826 Copyright: C. Faloutsos (2013) P1-8

SVD - Motivation •  Customer-product, for recommendation

system:

bread

lettu

ce

beef

vegetarians

meat eaters

tomato

s ch

icken

CMU SCS


SVD - Motivation •  problem #2: compress / reduce

dimensionality

C. Faloutsos 15-826

4

CMU SCS


Problem - specs

•  ~10**6 rows; ~10**3 columns; no updates; •  random access to any cell(s) ; small error: OK

CMU SCS


SVD - Motivation

CMU SCS


SVD - Motivation

C. Faloutsos 15-826

5

CMU SCS



CMU SCS


SVD - Definition (reminder: matrix multiplication

x

3 x 2 2 x 1

=

CMU SCS



x

3 x 2 2 x 1

=

3 x 1

C. Faloutsos 15-826

6

CMU SCS



x

3 x 2 2 x 1

=

3 x 1

CMU SCS



x

3 x 2 2 x 1

=

3 x 1

CMU SCS



x =

C. Faloutsos 15-826

7

CMU SCS


SVD - Definition •  A = U Λ VT - example:

CMU SCS


SVD - Definition A[n x m] = U[n x r] Λ [ r x r] (V[m x r])T

•  A: n x m matrix (eg., n documents, m terms) •  U: n x r matrix (n documents, r concepts) •  Λ: r x r diagonal matrix (strength of each

‘concept’) (r : rank of the matrix) •  V: m x r matrix (m terms, r concepts)

CMU SCS


SVD - Properties THEOREM [Press+92]: always possible to

decompose matrix A into A = U Λ VT , where •  U, Λ, V: unique (*) •  U, V: column orthonormal (ie., columns are unit

vectors, orthogonal to each other) –  UT U = I; VT V = I (I: identity matrix)

•  Λ: singular are positive, and sorted in decreasing order

C. Faloutsos 15-826

8

CMU SCS


SVD - Example •  A = U Λ VT - example:

data inf.

retrieval brain lung

= CS

MD

x x

CMU SCS



data inf.


= CS

MD

x x

CS-concept MD-concept

CMU SCS



data inf.


= CS

MD

x x

CS-concept MD-concept

doc-to-concept similarity matrix

C. Faloutsos 15-826

9

CMU SCS



data inf.


= CS

MD

x x

‘strength’ of CS-concept

CMU SCS



data inf.


= CS

MD

x x

term-to-concept similarity matrix

CS-concept

CMU SCS



data inf.


= CS

MD

x x

term-to-concept similarity matrix

CS-concept

C. Faloutsos 15-826

10

CMU SCS



CMU SCS


SVD - Interpretation #1 ‘documents’, ‘terms’ and ‘concepts’: •  U: document-to-concept similarity matrix •  V: term-to-concept sim. matrix •  Λ: its diagonal elements: ‘strength’ of each

concept

CMU SCS


SVD – Interpretation #1 ‘documents’, ‘terms’ and ‘concepts’: Q: if A is the document-to-term matrix, what

is AT A? A: Q: A AT ? A:

C. Faloutsos 15-826

11

CMU SCS


SVD – Interpretation #1 ‘documents’, ‘terms’ and ‘concepts’: Q: if A is the document-to-term matrix, what

is AT A? A: term-to-term ([m x m]) similarity matrix Q: A AT ? A: document-to-document ([n x n]) similarity

matrix

CMU SCS

15-826 Copyright: C. Faloutsos (2013) P1-32 Copyright: Faloutsos, Tong (2009) 2-32

SVD properties •  V are the eigenvectors of the covariance

matrix ATA

•  U are the eigenvectors of the Gram (inner-product) matrix AAT

Further reading: 1. Ian T. Jolliffe, Principal Component Analysis (2nd ed), Springer, 2002. 2. Gilbert Strang, Linear Algebra and Its Applications (4th ed), Brooks Cole, 2005.

CMU SCS


SVD - Interpretation #2 •  best axis to project on: (‘best’ = min sum of

squares of projection errors)

C. Faloutsos 15-826

12

CMU SCS


SVD - Motivation

CMU SCS


SVD - interpretation #2

•  minimum RMS error

SVD: gives best axis to project

v1

first singular

vector

CMU SCS


SVD - Interpretation #2

C. Faloutsos 15-826

13

CMU SCS


SVD - Interpretation #2 •  A = U Λ VT - example:

= x x v1

CMU SCS



= x x

variance (‘spread’) on the v1 axis

CMU SCS



– U Λ gives the coordinates of the points in the projection axis

= x x

C. Faloutsos 15-826

14

CMU SCS


SVD - Interpretation #2 •  More details •  Q: how exactly is dim. reduction done?

= x x

CMU SCS


SVD - Interpretation #2 •  More details •  Q: how exactly is dim. reduction done? •  A: set the smallest singular values to zero:

= x x

CMU SCS



~ x x

C. Faloutsos 15-826

15

CMU SCS



~ x x

CMU SCS



~ x x

CMU SCS



~

C. Faloutsos 15-826

16

CMU SCS


SVD - Interpretation #2 Exactly equivalent: ‘spectral decomposition’ of the matrix:

= x x

CMU SCS



= x x u1 u2 λ1

λ2

v1 v2

CMU SCS



= u1 λ1 vT1 u2 λ2 vT

2 + +... n

m

C. Faloutsos 15-826

17

CMU SCS




2 + +... n

m

n x 1 1 x m

r terms

CMU SCS


SVD - Interpretation #2 approximation / dim. reduction: by keeping the first few terms (Q: how many?)


2 + +... n

m

assume: λ1 >= λ2 >= ...

CMU SCS


SVD - Interpretation #2 A (heuristic - [Fukunaga]): keep 80-90% of

‘energy’ (= sum of squares of λi ’s)


2 + +... n

m

assume: λ1 >= λ2 >= ...

C. Faloutsos 15-826

18

CMU SCS


Pictorially: matrix form of SVD

– Best rank-k approximation in L2

A m

n

Σ m

n

U

VT

≈

CMU SCS


Pictorially: Spectral form of SVD

– Best rank-k approximation in L2

A m

n

≈ +

σ1u1°v1 σ2u2°v2

CMU SCS


SVD - Detailed outline •  Motivation •  Definition - properties •  Interpretation

–  #1: documents/terms/concepts –  #2: dim. reduction –  #3: picking non-zero, rectangular ‘blobs’

•  Complexity •  Case studies •  Additional properties

C. Faloutsos 15-826

19

CMU SCS


SVD - Interpretation #3 •  finds non-zero ‘blobs’ in a data matrix

= x x

CMU SCS


SVD - Interpretation #3 •  finds non-zero ‘blobs’ in a data matrix

= x x

CMU SCS

15-826 Copyright: C. Faloutsos (2013) P1-57

SVD - Interpretation #3 •  finds non-zero ‘blobs’ in a data matrix = •  ‘communities’ (bi-partite cores, here)

Row 1

Row 4

Col 1

Col 3

Col 4 Row 5

Row 7

C. Faloutsos 15-826

20

CMU SCS


SVD - Interpretation #3 •  Drill: find the SVD, ‘by inspection’! •  Q: rank = ??

= x x ?? ??

??

CMU SCS


SVD - Interpretation #3 •  A: rank = 2 (2 linearly independent rows/

cols)

= x x ??

?? ??

??

CMU SCS


SVD - Interpretation #3 •  A: rank = 2 (2 linearly independent rows/

cols)

= x x

orthogonal??

C. Faloutsos 15-826

21

CMU SCS


SVD - Interpretation #3 •  column vectors: are orthogonal - but not

unit vectors:

= x x

1/sqrt(3) 0 1/sqrt(3) 0 1/sqrt(3) 0

0 1/sqrt(2) 0 1/sqrt(2)

CMU SCS


SVD - Interpretation #3 •  and the singular values are:

= x x

1/sqrt(3) 0 1/sqrt(3) 0 1/sqrt(3) 0

0 1/sqrt(2) 0 1/sqrt(2)

CMU SCS


SVD - Interpretation #3 •  Q: How to check we are correct?

= x x

1/sqrt(3) 0 1/sqrt(3) 0 1/sqrt(3) 0

0 1/sqrt(2) 0 1/sqrt(2)

C. Faloutsos 15-826

22

CMU SCS


SVD - Interpretation #3 •  A: SVD properties:

– matrix product should give back matrix A – matrix U should be column-orthonormal, i.e.,

columns should be unit vectors, orthogonal to each other

–  ditto for matrix V – matrix Λ should be diagonal, with positive

values

CMU SCS



CMU SCS


SVD - Complexity •  O( n * m * m) or O( n * n * m) (whichever

is less) •  less work, if we just want singular values •  or if we want first k singular vectors •  or if the matrix is sparse [Berry] •  Implemented: in any linear algebra package

(LINPACK, matlab, Splus/R, mathematica ...)

C. Faloutsos 15-826

23

CMU SCS


SVD - conclusions so far •  SVD: A= U Λ VT : unique (*) •  U: document-to-concept similarities •  V: term-to-concept similarities •  Λ: strength of each concept •  dim. reduction: keep the first few strongest

singular values (80-90% of ‘energy’) – SVD: picks up linear correlations

•  SVD: picks up non-zero ‘blobs’

CMU SCS


References •  Berry, Michael: http://www.cs.utk.edu/~lsi/ •  Fukunaga, K. (1990). Introduction to Statistical

Pattern Recognition, Academic Press. •  Press, W. H., S. A. Teukolsky, et al. (1992).

Numerical Recipes in C, Cambridge University Press.

Documents

15-826: Multimedia Databases and Data Miningchristos/courses/826.F13/... · Title: 250_svd1.ppt Author: Christos Faloutsos school2 Created Date: 10/23/2013 6:15:41 AM