Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter Sparse & Redundant Representation

Sparse & Redundant Representation Modeling of ImagesProblem Solving Session 1: Greedy Pursuit AlgorithmsBy: Matan Protter

Sparse & Redundant Representation Modeling of Images

OMP versus Batch OMP

Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel

These slides were prepared by Matan Protter and used for a

summer school on sparse approximation in PCMI (Park City

Mathematical Institute (2010)

http://upload.wikimedia.org/wikipedia/he/7/70/Technion_Logo.png


2

Recall: The Thresholding Alg.

0 0Set : n 0, 0,S n

2r

n n 1

Initialization

1.

2.

Main Iteration1.2.

3.n nUpdate Residual: r x D

n n n 1nUpdate S : S S {i }

StopYesNo

Thresholding finds the set of atoms approximating the solution of

0

0 2min s.t. x

D

1 2 3 k

T

1 2

i i i i

Compute x and

sort this vector by absolute

decending order i ,i ,...

D

n

n

LS: min x s.t.

supp S

D


3

Recall: The OMP Algorithm

n 1i

zCompute E(i) min z d r for 1 i K

0

0 0

0

n 0, 0

r x x

and S

D

n

2r

n n 1

Initialization

Main Iteration1.

2.

3.

4.

5.

0 0Choose i s.t. 1 i K, E(i ) E(i)

n nLS : min x s.t. supp S

Dn nUpdate Re sidual : r x D

n n n 10Update S : S S {i }

StopYesNo

OMP finds one atom at a time for approximating the solution of

0 2 2

0 2min s.t. x

D

n 1i

z 2

T n 1ii

2 2n 1 T n 12i

2

E(i) min z d r

z d r

E (i) r d r


4

Few Notes

• Stages 1 & 2 in loop:

• To find the next atom to use:– Compute the projection of the

residual onto the dictionary– Select the atom with the largest (magnitude)

projection

• In Stage 4: – One can use Matlab’s “\” operator to solve the

Least-Squares problem, or – Even exploit the “recursive option” of Least-

Squares with growing number of unknowns.

n 1i

z 2

T n 1ii

2 2n 1 T n 12i

2

E(i) min z d r

z d r

E (i) r d r


5

Can we Speed Up the OMP?


6

Observation 1

• In step 1, we compute for each atom in the dictionary

• Suppose during the loop, we have found support S and coefficient

• We can write:

Tid r

j j S

T T T Ti i j j i jj j

j S j S

d r d x d d x d d


7

Observation 1 – Cont.

• If we work on one signal – this doesn’t help

• If we work on many signals (>> number of atoms), we can compute ONCE in advance (this is actually the multiplication

)• We also need to compute once for

each signal


j S j S

d r d x d d x d d

Ti jd d i, j

TD DT xD


8


• # of operations needed for computing – Regular OMP

• n – dimension of the signal– Modified Version

• |S| - the size of the current support (assuming there are enough signals that computing is negligible)

• We term this version “Batch-OMP”

Tid r

TD D


j S j S

d r d x d d x d d


9

Observation 1 - Practicalities

• Pre-compute • Denote as the sub-matrix of obtained

by taking only the columns belonging in S

TD D xDT

SGTD D

S 2,4

TD D

T T T Tj jj j

j S j S

r x d x d

D D D D

SG

TSSxD G


10

Observation 1 - Summary

• Pre-compute and • To find the atom with largest inner

product with the residual, compute:

T TSSr x D D G

T xD

The vector of the coefficients

belonging to the support

TD D


11

Implications

n 1i

zCompute E(i) min z d r for 1 i K

0

0 0

0

n 0, 0

r x x

and S

D

n

2r

n n 1

Initialization

Main Iteration1.

2.

3.

4.

5.

0 0Choose i s.t. 1 i K, E(i ) E(i)

n nLS : min x s.t. supp S

Dn nUpdate Re sidual : r x D

n n n 10Update S : S S {i }

StopYesNo

OMP finds one atom at a time for approximating the solution of

0 2 2

0 2min s.t. x

D

Replaced

Removed


12

Observation 2

• Assuming we have an error threshold as condition, we need to compute the norm of the residual in each stage

• In observation 1, we avoid computing the residual altogether …

• Can we still avoid it?

• Answer: Yes, this is possible …


13


The norm of the residual is cheap to compute using the norm from the previous iteration

Where:

n n 1

2 2 T Tn n 1 n n n 1 n 1

S S2 2r r G G

n The coefficients of the atoms in the support in iteration n A sub-matrix of by taking the rows and columns in the support in the n-th iteration Sn

TD DnSG


14


• Denote

• Then:

• Therefore: compute in each iteration, and update the norm of the residual accordingly

2 2n n 1n n 12 2

r r

n

Tn nn S

G

n


15


Is it cheap to compute ? Yes!

is already computed for the update of

and then,

is simply a dot product between two short vectors of length |Sn| each

n

Tn nn S

G

n

n

SG

T TSSr x D D G

n

Tn nn S

G


16

• Denote the norm of the residual in iteration n by

• Initialize to

• At each stage, compute

• And update the norm of the residual

2 20022

t r x

n

Tn nn S

G

n n 1n n 1tt

nt



17

Batch-OMP Summary

• Start with the regular OMP algorithm• Add pre-processing steps:

– Compute– Compute

• Modify the atom selection stage using

• Modify the residual update stage as shown before – ONLY if you intend to stop by the residual norm

TD DTD X

T TSSr x D D G

Documents

Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter Sparse & Redundant Representation