7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 1/85

Applications and opportunities

Emamnuel J. Candes

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 2/85

Applications and opportunities

New analog-to-digital converters

New optical systemsFast Magnetic Resonance Imaging

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 3/85

Time/Space sampling

Analog-to-digital converters, receivers, ...

Space: cameras, medical imaging devices, ...

Nyquist/Shannon foundation: must sample at twice the highest frequency

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 4/85

Sampling of ultra wideband radio frequency signals

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 5/85

Hardware brick wall

Signals are wider and wider band“Moore’s law:” factor 2 improvement every 6 to 8 years

Extremely fast/high-resolution samplers are decades away

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 6/85

Hardware brick wall

Signals are wider and wider band“Moore’s law:” factor 2 improvement every 6 to 8 years

Extremely fast/high-resolution samplers are decades away

Way out? Analog-to-information (DARPA)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 7/85

New analog to digital converters

Two architectures: joint between Caltech (Candes, Emami) and Northrop

GrummanNonuniform sampler (NUS)

Random modulator pre-integration (RMPI)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 8/85

Random pre-integrator

∫kT1,-1,1,1,-1,...

yk,1

∫kT-1,-1,-1,1,-1,...

yk,2

∫ kT-1,1,-1,1,1,...

yk,P

f(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1!2

!1.5

!1

!0.5

0

0.5

1

1.5

2

RPI architecture ±1 sequence

E l f l

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 9/85

Empirical performance: pulse reconstruction

Joint with M. Wakin and S. Boyd: different algorithm

Two-pulse signal

Random modulation followed by integration

Sampling at 6% of Nyquist rate

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1!2

!1.5

!1

!0.5

0

0.5

1

1.5

2

0 100 200 300 400 500 600!1

!0.8

!0.6

!0.4

!0.2

0

0.2

0.4

0.6

0.8

1

˙

E i i l f l i

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 10/85

Empirical performance: pulse reconstruction

Joint with M. Wakin and S. Boyd: different algorithm

Two-pulse signal

Random modulation followed by integration

Sampling at 6% of Nyquist rate

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1!2

!1.5

!1

!0.5

0

0.5

1

1.5

2

0 100 200 300 400 500 600!1

!0.8

!0.6

!0.4

!0.2

0

0.2

0.4

0.6

0.8

1

˙ 0 100 200 300 400 500 600!1

!0.8

!0.6

!0.4

!0.2

0

0.2

0.4

0.6

0.8

1

A li i d i i

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 11/85

Applications and opportunities

New analog-to-digital converters

New optical systemsFast Magnetic Resonance Imaging

Wh t d ?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 12/85

What do we measure?

Direct sampling: analog/digital photography,mid 19th century

Indirect sampling: acquisition in a

transformed domain, second half of 20thcentury; e.g. CT, MRI

Compressive sampling: acquisition in anincoherent domain

Design incoherent analog sensors ratherthan usual pixelsPay-off: need far fewer sensors

The first photograph?

CT scanner

O i l

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 13/85

One pixel camera

Richard Baraniuk, Kevin Kelly, Yehia Massoud, Don JohnsonRice University, dsp.rice.edu/CS

MIT Tech review: top 10 emerging technologies for 2007

Other works: Brady et al., Freeman et al., Wagner et al., Coifman et al.

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 14/85

Rice “Single-Pixel” CS Camera

random

pattern on

DMD array

DMD

single photon

detector

image

reconstruction

object

Richard Baraniuk,

Kevin Kelly, andstudents

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 15/85

CS Low-Light Imaging with PMT

true color low-light imaging

256 x 256 image with 10:1compression

[Nature Photonics, April 2007]

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 16/85

Dual Visible and Infrared Imaging

dual photodiode sandwich

K cutout in paper

front-litvisible

back-litIR LEDs

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 17/85

Confocal microscopy

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 18/85

L a s e r

Photodiode, PM

slit

Objective

Raster scan microscopy

Move the focus point along the sample

Applications and opportunities

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 19/85

Applications and opportunities

New analog-to-digital converters

New optical systemsFast Magnetic Resonance Imaging

Magnetic Resonance Imaging

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 20/85

Magnetic Resonance Imaging

Trzasko, Manduca, Borisch (Mayo Clinic)

Magnetic Resonance Imaging

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 21/85

Magnetic Resonance Imaging

Trzasko, Manduca, Borisch (Mayo Clinic)

Fully sampled

Magnetic Resonance Imaging

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 22/85

Magnetic Resonance Imaging

Trzasko, Manduca, Borisch (Mayo Clinic)

Fully sampled 6 × undersampledclassical

Magnetic Resonance Imaging

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 23/85

Magnetic Resonance Imaging

Trzasko, Manduca, Borisch (Mayo Clinic)

Fully sampled 6 × undersampled 6 × undersampledclassical CS

MR angiography

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 24/85

g g p y

Trzasko, Manduca, Borisch

Fully sampled 6 × undersampled

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 25/85

Fully sampled 6 × undersampled

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 26/85

Matrix completion

(joint with B. Recht)

Partially filled-out surveys

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 27/85

y y

Questions asked to individuals

Many questions left unanswered

Questions

People

× ×× ×

× ×× ×

× × ×

Partially filled-out surveys

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 28/85

Questions asked to individuals

Many questions left unanswered

Questions

People

× ? ? ? × ?? ? × × ? ?× ? ? × ? ?? ?

×? ?

×× ? ? ? ? ?? ? × × ? ?

ProblemWould like to guess the missing answers

The Netflix problem

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 29/85

Netflix database

About a million users

About 25,000 movies

People rate movies

Sparsely sampled entries

The Netflix problem

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 30/85

Netflix databaseAbout a million users

About 25,000 movies

People rate movies

Sparsely sampled entries

Movies

Users

× ×× ×

× ×× ×

× × ×

Challenge

Complete the “Netflix matrix”

Many such problems → collaborative filtering

Triangulation from incomplete data

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 31/85

Points xj1≤j≤n ∈ RdPartial information about distances

M ij = xi − xj2

Example (A. Singer et al.)

Low-powered wirelessly networked sensors

Each sensor can construct a distance estimatefrom nearest neighbor

j

i

× ×× ×

× ×× ×

× × ×

Triangulation from incomplete data

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 32/85

Points xj1≤j≤n ∈ RdPartial information about distances

M ij = xi − xj2

Example (A. Singer et al.)

Low-powered wirelessly networked sensors

Each sensor can construct a distance estimatefrom nearest neighbor

j

i

× ×× ×

× ×× ×

× × ×

ProblemLocate the sensors

Other problems of this kind

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 33/85

Partially observed covariance matrix

System identification in control theory

Low-rank matrix completion in machine learningStructure-from-motion problem in computer vision

Many others

Linear system identification

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 34/85

Linear time-invariant system

x(t + 1) = Ax(t) + Bu(t)y(t) = Cx(t) + Du(t)

, t = 0, . . . , N

u(t) ∈ Rm is input

y(t)∈R p is output

x(t) ∈ Rn is state vector

System identification (Vandenberghe et al.)

Estimatesystem order n

model matrices A, B, C , D and x(0)

from obs. of inputs and outputs

Fundamental questions

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 35/85

Can we recover a matrix M ∈ Rn1×n2 from m sampled entries, m n1n2?

In general, everybody would agree that this is impossible

Fundamental questions

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 36/85

Can we recover a matrix M ∈ Rn1×n2 from m sampled entries, m n1n2?

In general, everybody would agree that this is impossible

Some surprises

Can recover matrices of interest from “incomplete” sampled entries

Can be done by convex programming

Matrix completion

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 37/85

Matrix M ∈ Rn1×n2Observe subset of entriesCan we guess the missing entries?

× ? ? ? × ?? ? × × ? ?

×? ?

×? ?

? ? × ? ? ×× ? ? ? ? ?? ? × × ? ?

Low-rank matrix completion?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 38/85

Engineering/scientific applications: unknown matrix is often (approx.) low-rank

Movies

Users

× ×× ×

× ×× ×

× × ×

Examples:

Netflix matrix

Low-rank matrix completion?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 39/85

Engineering/scientific applications: unknown matrix is often (approx.) low-rank

j

i

× ×× ×

× ×× ×

× × ×

Examples:

Netflix matrix

Sensor-net matrix:

xi

−xj

2,

xi

∈Rd

Matrix is of rank 2 if d = 2

Matrix is of rank 3 if d = 3

...

Low-rank matrix completion?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 40/85

Engineering/scientific applications: unknown matrix is often (approx.) low-rank

j

i

× ×× ×

× ×× ×

× × ×

Examples:

Netflix matrix

Sensor-net matrix:

xi

−xj

2,

xi

∈Rd

Matrix is of rank 2 if d = 2

Matrix is of rank 3 if d = 3

...

Problems in control (low-order system identification)

Many others (e.g. machine learning)

Multiclass learning

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 41/85

Build multiple classifiers with same training data

K classes to distinguish

Training examples f 1, . . . , f n

Partial labeling

yik =

+1 if f i is of class k

−1 if f i is not of class k? if no information at all

Goal: produce a linear classifier wk for each class

w∗

kf i

> 0 if f i

is of class k

w∗kf i < 0 otherwise

Low-rank fitting

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 42/85

Common hypothesisW = [w1, . . . ,wK ]

has low-rank

Multiclass learningminimize rank(W )

subject to f ∗iWek = yik

References

Srebro, Rennie & Jaakkola

Abernethy, Bach, Evgeniou & Vert

Argyriou, Evgeniou & Pontil

Amit, Fink, Srebro & Ullman

Low-rank matrices

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 43/85

M

∈Rn×n: matrix of rank r

Singular value decomposition

M =

rk=1

σkukv∗k

σk: singular values

uk, vk: singular vectors

M depends upon (2n − r)r degrees of freedom < ambient dimension

Low-rank matrices

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 44/85

M

∈Rn×n: matrix of rank r

Singular value decomposition

M =

rk=1

σkukv∗k

σk: singular values

uk, vk: singular vectors

M depends upon (2n − r)r degrees of freedom < ambient dimension

Do we need to see all the entries to recover M ?

Which matrices?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 45/85

M = e1e∗n =

0 0 · · · 0 10 0 · · · 0 0...

......

......

0 0 · · · 0 0

,

Cannot be recovered from a sampling set

Which matrices?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 46/85

M = e1e∗n =

0 0 · · · 0 10 0 · · · 0 0...

......

......

0 0 · · · 0 0

,

Cannot be recovered from a sampling set

What if M is nonadversarial? Can we recover almost all matrices?

Which sampling set?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 47/85

Rank-1 matrix M = xy∗

M ij = xiyj

× × × × × ×

× × × × × ×× × × × × ×× × × × × ×× × × × × ×

If single row is not sampled → recovery is not possible

If single column is not sampled → recovery is not possible

Which sampling set?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 48/85

Rank-1 matrix M = xy∗

M ij = xiyj

× × × × × ×

× × × × × ×× × × × × ×× × × × × ×× × × × × ×

If single row is not sampled → recovery is not possible

If single column is not sampled → recovery is not possible

What if the sampling set is random?

Ω subset of m entries selected uniformly at random

Which algorithm?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 49/85

Hope: only one low-rank matrix consistent with the sampled entries

Recovery by minimum complexity

minimize rank(X )subject to X ij = M ij (i, j)

∈Ω,

Which algorithm?

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 50/85

Hope: only one low-rank matrix consistent with the sampled entries

Recovery by minimum complexity

minimize rank(X )subject to X ij = M ij (i, j)

∈Ω,

ProblemThis is NP-hard

Doubly exponential in n (?)

Nuclear norm minimization

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 51/85

Nuclear norm (σi(X ) is ith largest singular value of X )

X ∗ =ni=1

σi(X )

Heuristicminimize X ∗subject to X ij = M ij (i, j) ∈ Ω.

Convex relaxation of the rank minimization program

Ball X : X ∗ ≤ 1: convex hull of rank-1 matrices obeying xx∗ ≤ 1

Semidefinite programming (SDP)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 52/85

Special class of convex optimization problems

Relatively natural extension of linear programming (LP)“Efficient” numerical solvers (interior point methods)

Semidefinite programming (SDP)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 53/85

Special class of convex optimization problems

Relatively natural extension of linear programming (LP)“Efficient” numerical solvers (interior point methods)

LP (std. form): x ∈ Rn

minimize c,xsubject to Ax = b

x ≥ 0

Semidefinite programming (SDP)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 54/85

Special class of convex optimization problems

Relatively natural extension of linear programming (LP)“Efficient” numerical solvers (interior point methods)

LP (std. form): x ∈ Rn

minimize c,xsubject to Ax = b

x ≥ 0

SDP (std. form): X ∈ Rn×n

minimize C ,X subject to

Ak,X

= bk

X 0

Standard inner product: C ,X = trace(C ∗X )

Positive semidefinite unknown: SDP formulation

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 55/85

Suppose unknown matrix is positive semidefinite

minn

i=1 σi(X )s. t. X ij = M ij (i, j) ∈ ΩX 0

⇔ min trace(X )s. t. X ij = M ij (i, j) ∈ ΩX 0

Trace heuristics: Mesbahi & Papavassilopoulos (1997), Beck & D’Andrea (1998)

General SDP formulation

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 56/85

SDP characterization: X ∗ = val(P )

(P )min (trace(W 1) + trace(W 2)) /2

s. t. W 1 X

X ∗ W 2 0

Optimization variables: W 1 ∈ Rn1×n1 , W 2 ∈ Rn2×n2

Nuclear norm heuristics: Fazel (2002), Hindi, Boyd & Fazel (2001)

Uncovering shared structure in multiclass classification

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 57/85

Amit, Fink, Srebro & Ullman

Multiclass classificationH : X → Y

Linear classifiers: for each y ∈ Y

H (x) = arg maxy∈Y

wy,x

“SVM-loss”

(W ; (x, y)) = maxy=y

[1 + wy −wy,x]+

e.g. Y = 1, 2, w1 = −w2 = 1

2wsvm ⇒ SVM!

Proposal

min W ∗ + λ

mi=1

(W ; (xi, yi))

i.e. find low-rank classifiers

Matrix recovery

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 58/85

M =

2k=1

σkuku∗k, u1 = (e1 + e2)/√2,u2 = (e1 − e2)/

√2,

M =

∗ ∗ 0 · · · 0 0∗ ∗ 0 · · · 0 0

0 0 0 · · · 0 0......

......

......

0 0 0 · · · 0 0

Cannot be recovered from a small set of entries

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 59/85

M = e∗1x =

x1 x2 x3 · · · xn−1 xn0 0 0 · · · 0 0

0 0 0 · · · 0 0......

......

......

0 0 0 · · · 0 0

Cannot be recovered from a small set of entries

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 60/85

M = e∗1x =

x1 x2 x3 · · · xn−1 xn0 0 0 · · · 0 0

0 0 0 · · · 0 0......

......

......

0 0 0 · · · 0 0

Cannot be recovered from a small set of entries

Intuition: column and row spaces cannot be aligned with basis vectors

Coherence

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 61/85

U ⊂ Rn is a subspace of dimension r

P U is the orthogonal projection onto U

The coherence of U (vis-a-vis the standard basis (ei)) is defined as

coh(U ) ≡ n

rmax1≤i≤n

P U ei2, 1 ≤ coh(U ) ≤ n/r

ei ∈ U ⇒ coh(U ) = n/r

Othobasis [u1, . . . ,ur] with

uk∞ = 1/√n ⇒ coh(U ) = 1

Example: random plane of dimension r (r ≥ log n), coh(U ) = O(1)

Exact matrix completion

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 62/85

min X ∗ s. t. X ij = M ij , (i, j) ∈ Ω

Exact matrix completion

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 63/85

min X ∗ s. t. X ij = M ij , (i, j) ∈ Ω

Theorem (C. and Recht, ’08)

M ∈ Rn1×n2 of rank r (obeying µ0r n1/5)

µ0 := max (coh(col. space), coh(row. space))

random set of entries of size m

Exact matrix completion

M ( ) Ω

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 64/85

min X ∗ s. t. X ij = M ij , (i, j) ∈ Ω

Theorem (C. and Recht, ’08)

M ∈ Rn1×n2 of rank r (obeying µ0r n1/5)

µ0 := max (coh(col. space), coh(row. space))

random set of entries of size m

The minimizer is unique and equal to M w.p. at least 1 − n−3 if

m µ0 n6/5 r log n, n = max(n1, n2)

Similar result for higher values of the rank

Can achieve stronger probabilities, e. g. 1 − O(n−β), β > 0, if log n → β log n

Example I: random orthogonal model (most matrices)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 65/85

M ∈ Rn1×n2 of rank r with col. and row spaces sampled uniformly atrandom

random set of entries of size m

If r n1/5, recovery is exact w. p. at least 1 − O(n−3) if

m n6/5r log n, n = max(n1, n2)

For all values of the rank, recovery is exact w. p. at least 1

−O(n−3) if

m n5/4r log n, n = max(n1, n2)

Example II: bounded orthogonal systems

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 66/85

M =

r

k=1

σkukv∗k

Entries of uk ∈ Rn1 bdd by µB/√

n1

Entries of vk ∈ Rn2 bdd by µB/√

n2

P U ei22 =

1≤k≤r

|uik|2 ≤ µB r/n1 ⇒ µ(U ) ≤ µB

Example II: bounded orthogonal systems

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 67/85

M =

r

k=1

σkukv∗k

Entries of uk ∈ Rn1 bdd by µB/√

n1

Entries of vk ∈ Rn2 bdd by µB/√

n2

P U ei22 =

1≤k≤r

|uik|2 ≤ µB r/n1 ⇒ µ(U ) ≤ µB

Recovery is exact if

m µB · n6/5r log n, n = max(n1, n2)

Formal equivalence

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 68/85

There is a regime in which

the solution to the combinatorially hard rank-minimization problem is unique

the solution to the nuclear-norm minimization problem is uniqueand they are the same!

Methods of proof

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 69/85

Duality in convex optimization

Find a dual Y certifying that min rank solution is solution to SDP

Banach space methods for studying spectra of random matrices

Rudelson selection theorem (Rudelson)

Decoupling of dependent random variables (Bourgain & Tzafiri, de la Pena)

Noncommutative Khintchine inequalities (Lust-Picard, Pisier)

Concentration of measure (Talagrand)

Numerical experiments

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 70/85

m/n2

d r / m

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

m/Dn

d r / m

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

M = Y LY ∗R M = Y LY

∗L

x-axis: m/Dn, Dn = n2 (left), Dn = n(n + 1)/2 (right)

y-axis: d/m, d = r(2n − r) (left), d = r(n − (r − 1)/2))

Y L, Y R with i.i.d. N (0, 1) entries

Extensions I

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 71/85

Two orthonormal bases F = [f 1, . . . ,f n], G = [g1, . . . , gn]

minimize X ∗subject to f ∗iXgj = f ∗iMgj , (i, j) ∈ Ω,

Extensions I

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 72/85

Two orthonormal bases F = [f 1, . . . ,f n], G = [g1, . . . , gn]

minimize X ∗subject to f ∗iXgj = f ∗iMgj , (i, j) ∈ Ω,

Nuclear norm minimization succeeds if column and row spaces of M areincoherent with f i and gi

Extensions I

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 73/85

Two orthonormal bases F = [f 1, . . . ,f n], G = [g1, . . . , gn]

minimize X ∗subject to f ∗iXgj = f ∗iMgj , (i, j) ∈ Ω,

Nuclear norm minimization succeeds if column and row spaces of M areincoherent with f i and gi

Why? Because f ∗i Xgj = e∗i (F ∗XG)ej

Extensions II

A A (orthonormal) basis of Rn×n (N 2)

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 74/85

A1, . . . ,AN (orthonormal) basis of Rn×n (N = n2)

Ω

⊂ 1, . . . , N

minimize X ∗subject to Ak,X = Ak,M , k ∈ Ω

Results should also apply:

small inner products → nuclear norm minimization succeeds

Recht, Parrilo, Fazel, ’07

If each Ak i.i.d. N (0, 1), this succeeds (connected to compressed sensing)

Connections with compressed sensing

General setup

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 75/85

Rank minimization

minimize rank(X )subject to A(X ) = b

Convex relaxation

minimize X ∗subject to A(X ) = b

Connections with compressed sensing

General setup

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 76/85

Rank minimization

minimize rank(X )subject to A(X ) = b

Convex relaxation

minimize X ∗subject to A(X ) = b

Suppose X = diag(x), x ∈ Rn

rank(X ) =

i 1(xi = 0) = x0X ∗ =

i |xi| = x1

Connections with compressed sensing

General setup

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 77/85

Rank minimization

minimize rank(X )subject to A(X ) = b

Convex relaxation

minimize X ∗subject to A(X ) = b

Suppose X = diag(x), x ∈ Rn

rank(X ) =

i 1(xi = 0) = x0X ∗ =

i |xi| = x1

Rank minimization

minimize x0subject to Ax = b

Convex relaxation

minimize x1subject to Ax = b

This is compressed sensing!

Singular Value Thresholding algorithm

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 78/85

Joint with J-F Cai, Z. Shen and S. Becker

Parallel C implementation (MPI)

Machine: 4 core CPU (2.4GHz)

Unknown 30, 000 × 30, 000 matrix of rank 10

Observe 0.4% of the entries

Singular Value Thresholding algorithm

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 79/85

Joint with J-F Cai, Z. Shen and S. Becker

Parallel C implementation (MPI)

Machine: 4 core CPU (2.4GHz)

Unknown 30, 000 × 30, 000 matrix of rank 10

Observe 0.4% of the entries

Near-exact completion in 197 seconds

Demo

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 80/85

Summary

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 81/85

Can recover signals of interest from “incomplete” data

Can recover matrices of interest from “incomplete” sampled entries

Can be done by convex programming

Many applications

Biomedical imaging (fast MRI, fast CT)New optical sensorsNew analog-to-digital convertersMachine learning

Connections with graph theory

For rank 1 matrices

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 82/85

For rank-1 matrices

Need at least one observation /row and column→

coupon collector’s problem

x

x

xx x

xx

xx

x x xx

x x

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 83/85

Example

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 84/85

Observe rank-1 matrix 1 ?? 1

Example

7/31/2019 E_Candes_Applications and Opportunities of CS

http://slidepdf.com/reader/full/ecandesapplications-and-opportunities-of-cs 85/85

Observe rank-1 matrix 1 ?? 1

Two possible completions 1 11 1

1 −1

−1 1