73
Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video M. Salman Asif Advisor: Justin Romberg

Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic Compressive Sensing:

Sparse recovery algorithms for streaming signals and video

M. Salman Asif

Advisor: Justin Romberg

Page 2: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic compressive sensing

2

Part 1

Dynamic updating

Use previous solutions and dynamics to quickly solve the

recovery problems

Part 2

Dynamic modeling

Use the dynamical signal structure to improve the

reconstruction

“warm start”

final solution

Page 3: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic compressive sensing

Part 2

Dynamic modeling

Use the dynamical signal structure to improve the

reconstruction

3

Part 1

Dynamic updating

Use previous solutions and dynamics to accelerate the

recovery process

...

Page 4: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Compressive sensing

• A theory that deals with signal recovery from underdetermined systems:

y = ©x =

M < N

underdetermined

M £N system

[Candes, Romberg, Tao, Donoho, Tropp, Baraniuk, Gilbert, Vershynin, Rudelson, …]4

=

M < N

underdetermined

M £N system

Incoherent measurement matrix(e.g., subGaussian matrices)

Sparse signal

2. Sparse representation in a redundant dictionary

Similar principles: structure and incoherence

1. Compression with sampling

Page 5: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

• Compressive sensing deals with signal recovery from underdetermined systems:

Compressive sensing

y =©x´ ©ª®

Measurement matrix

Representation basis Sparse signal

x = ª® =

NX

i=1

Ãi®i

=

Sparse representation restricts the space of signals

minimize ¿kªTxk1 +1

2k©x¡ yk22

sparsity data fidelity

tradeoff parameter

5

Page 6: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

• Compressive sensing deals with signal recovery from underdetermined systems:

Compressive sensing

y =©x´ ©ª®

Measurement matrix

Representation basis Sparse signal

x = ª® =

NX

i=1

Ãi®i

=

Sparse representation restricts the space of signals

minimize k W ªTxk1 +1

2k©x¡ yk22

sparsity data fidelity

Weights

6

Page 7: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

minimize kWªTxk1 +1

2k©x¡ yk22

• “Static” compressive sensing:

minimize ¿kªTxk1 +1

2kAx¡ yk22

Fixed signalFixed set of measurements Fixed model for representation

x1 x2 y1 y2

innovations

dynamic model

1 2ª1 ª2

minimize kWtªTt xtk1 +

1

2k©txt ¡ ytk22

y =©x´ ©ª®

“Dynamic” compressive sensing

7Streaming measurementsData-adaptive model

yt =©txt ´©tªt®t dynamics

W1 W2

Iterative reweighting

Page 8: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

minimize kWtªTt xtk1 +

1

2k©txt ¡ ytk22

+kFtxt ¡ xt+1kp + kBtxt ¡ xt¡1kp

• “Static” compressive sensing:

Fixed signalFixed set of measurements Fixed model for representation

y = Ax´Aª®

“Dynamic” compressive sensing

dynamics

x1 x2 x3

...

xT

F1 F2 FT¡1

B2 B3 BT

Spatio-temporal structure (redundancies) in the videos as a dynamic model

yt =©txt ´©tªt®t

Page 9: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic compressive sensing

9

Dynamic updating Quickly update the solution to

accommodate changes

•   homotopy

• Variations:• Streaming signal

• Streaming measurements

• More…

Dynamic modeling Improve reconstruction by exploiting

the dynamical signal structure

1. Low-complexity video compression

2. Accelerated dynamic MRI

“warm start”

final solution

...

Page 10: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Part 1: Dynamic `1 updating

10

“warm start”

final solution

Page 11: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Motivation: dynamic updating in LS

y = ©x

© xy

• System model:

minimize k©x¡ yk2 ! x0 = (©T©)¡1©Ty

• LS estimate

is full rank

x is arbitrary

=

• Updates for a time-varying signal with the same mainly incurs a one-time cost of factorization.

11

Page 12: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Recursive updates

• Sequential measurements:

w

© xy

·y

w

¸=

·©

Á

¸x

Á

=

• Recursive LS

x1 = (©T©+ ÁTÁ)¡1(©Ty + ÁTw)

= x0 +K1(w¡ Áx0)

K1 = (©T©)

¡1ÁT(1+Á(©

T©)

¡1ÁT)

Rank one update

12

Page 13: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamics with the Kalman filter

• Linear dynamical system:

• Update requires few low-rank updates

yi = ©ixi + ei

xi+1 = Fixi + fi

minimize

kX

i=1

¾ik©ixi ¡ yik22 + ¸ikxi ¡ Fi¡1xi¡1k22

Kalman filter equation

13

Page 14: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

• Quickly update the solution of the `1 program as the system parameters change.

• Variations:

– Time-varying signal

– Streaming measurements

– Iterative reweighting

– Streaming signals with overlapping measurements

– Sparse signal in a linear dynamical system (sparse Kalman filter)

Dynamic   updating

14

y = ©¹x+ e ! minimize kWxk1 +1

2k©x¡ yk22

Page 15: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

• Quickly update the solution of the `1 program as the system parameters change.

• Variations:

– Time-varying signal

– Streaming measurements

– Iterative reweighting

– Streaming signals with overlapping measurements

– Sparse signal in a linear dynamical system (sparse Kalman filter)

Dynamic   updating

15

yt = ©t¹xt + et ! minimize kWtxtk1 +1

2k©txt ¡ ytk22

variations

Page 16: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Time-varying signals

y1 =©x1+ e1• System model:

minimize ¿kxk1 +1

2k©x¡ y2k22• New `1 problem:

16

x1! x2 ) y1! y2• Signal varies: “Sparse innovations”

minimize ¿kxk1 +1

2k©x¡ y1k22• `1 problem:

minimize ¿kxk1 +1

2k©x¡ (1¡ ²)y1 ¡ ²y2k22• Homotopy:

Homotopy parameter: 0 ! 1

[A., Romberg, “Dynamic L1 updating”, c. 2009]

Page 17: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Time-varying signals

y1 =©x1+ e1• System model:

minimize ¿kxk1 +1

2k©x¡ y2k22• New `1 problem:

17

x1! x2 ) y1! y2• Signal varies: “Sparse innovations”

minimize ¿kxk1 +1

2k©x¡ y1k22• `1 problem:

minimize ¿kxk1 +1

2k©x¡ (1¡ ²)y1 ¡ ²y2k22

Homotopy parameter: 0 ! 1

bx1

bx2

• Path from old solution to new solution is piecewise linear and it is parameterized by ²: 0 ! 1 ² = 0

² = 1

Page 18: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Time-varying signal

minimize ¿kxk1 +1

2k©x¡ (1¡ ²)y1 ¡ ²y2k22• Homotopy:

©T¡ (©x¤ ¡ (1¡ ²)y1 ¡ ²y2) = ¡¿z

k©T¡c(©x¤ ¡ (1¡ ²)y1 ¡ ²y2)k1 < ¿

• Optimality conditions:Must be obeyed by any solution x* with support ¡ and sign sequence z

Optimality: set subdifferential of the objective to zero

ÿg +©T (©x¤ ¡ (1¡ ²)y1 ¡ ²y2) = 0;

g = @kx¤k1 : kgk1 6 1; gTx¤ = kx¤k1

!

18

Page 19: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Time-varying signal

minimize ¿kxk1 +1

2k©x¡ (1¡ ²)y1 ¡ ²y2k22• Homotopy:

©T¡ (©x¤ ¡ (1¡ ²)y1 ¡ ²y2) = ¡¿z

k©T¡c(©x¤ ¡ (1¡ ²)y1 ¡ ²y2)k1 < ¿

• Optimality conditions:Must be obeyed by any solution x* with support ¡ and sign sequence z

©T¡ (©x¤ ¡ (1¡ ²)y1 ¡ ²y2) + ±©T

¡ (©@x + y1 ¡ y2) = ¡¿z

k©T¡c(©x¤ ¡ (1¡ ²)y1 ¡ ²y2) + ±©T

¡c(©@x + y1 ¡ y2)k1 < ¿

• Change ² to ²+±:

@x =

((©T

¡©¡)¡1©T

¡ (y2 ¡ y1); on ¡

0; on ¡c

19

Page 20: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Results [A., Romberg, “Dynamic L1 updating”, c. 2009]

20

Signal typeDynamicX

(nProdAtA, CPU)

LASSO

(nProdAtA, CPU)

GPSR-BB

(nProdAtA, CPU)

FPC_AS (nProdAtA, CPU)

N = 1024

M = 512

T = m/5, k ~ T/20

Values = +/- 1

(23.72, 0.132) (235, 0.924) (104.5, 0.18) (148.65, 0.177)

Blocks (2.7, 0.028) (76.8, 0.490) (17, 0.133) (53.5, 0.196)

Pcw. Poly. (13.83, 0.151) (150.2, 1.096) (26.05, 0.212) (66.89, 0.25)

House slices (26.2, 0.011) (53.4, 0.019) (92.24, 0.012) (90.9, 0.036)

nProdAtA: avg. number of matrix-vector products with and T

CPU: average cputime to solve¿ = 0:01kATyk1

GPSR: Gradient Projection for Sparse Reconstruction. M. Figueiredo, R. Nowak, S. WrightFPC_AS: Fixed-point continuation and active set. Z. Wen, W. Yin, D. Goldfarb, and Y. Zhang

Page 21: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

“If all you have is a hammer, everything looks like a nail.”

Sparse Kalman filterReweighted `1

Dictionary update

`1 analysis

Streaming signal

Positivity/affine constraint

Dantzig selector

21

`1 decodingSequential measurements

Page 22: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse Kalman filterReweighted `1

Dictionary update

`1 analysis

Streaming signal

Positivity/affine constraint

Dantzig selector

22

`1 decodingSequential measurements

minimize ¿kxk1 +1

2kAx¡ yk22 +

1

2kBx¡ (1¡ ²)Bbxold ¡ ²wk22

• Sequential measurements: Homotopy parameter: 0 ! 1

Dummy vector used to ensure optimality of previous solution

with the changes

=

[A., Romberg, 2010]

Previous work on single measurement update: [Garrigues & El Ghaoui ’08; A., Romberg, ‘08]

Page 23: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse Kalman filterReweighted `1

Dictionary update

`1 analysis

Streaming signal

Positivity/affine constraint

Dantzig selector

23

`1 decodingSequential measurements

minimize k[(1¡ ²)W + ²fW ]xk1 +1

2k©x¡ yk22

• Iterative reweighting :Homotopy parameter: 0 ! 1

Weights are updated

[A., Romberg, 2012]

Adaptive reweighting in Chapter VI in the thesis

Page 24: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse Kalman filterReweighted `1

Dictionary update

Streaming signal

Positivity/affine constraint

Dantzig selector

24

`1 decodingSequential measurements

minimize kWxk1 +1

2k©x¡ yk22 + (1¡ ²)uTx

“ ne homotopy to rule them all”

[A., Romberg, 2013]

Page 25: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

• System model:

• Solve

  is sparse

Page 26: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

• System model:

• Instead, use the following versatile homotopy:

26

  is sparse

Optimization problem to solve Homotopy part

A “dummy” variable that maintains optimality of the starting point at  

Page 27: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

• System model:

• A versatile homotopy formulation:

27

  is sparse

Optimization problem to solve Homotopy part

Optimality: set subdifferential of the objective to zero

ÃWg+©T(©x¤ ¡ y) + (1¡ ²)u = 0;

g = @kx¤k1 : kgk1 6 1; gTx¤ = kx¤k1

!A “dummy” variable that maintains optimality of the starting point at  

Page 28: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

minimize kWxk1 +1

2k©x¡ yk22 + (1¡ ²)uTx• Homotopy:

©T¡ (©x

¤ ¡ y) + (1¡ ²)u = ¡Wz

j©T(©x¤ ¡ y) + (1¡ ²)uj 6 w

• Optimality conditions:Must be obeyed by any solution x* with support ¡ and sign sequence z

28

Page 29: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

minimize kWxk1 +1

2k©x¡ yk22 + (1¡ ²)uTx• Homotopy:

©T¡ (©x

¤ ¡ y) + (1¡ ²)u+ ± (©T¡©@x¡ u) = ¡Wz

jÁTi (©x¤ ¡ y) + (1¡ ²)ui + ± (ÁTi ©@x¡ ui)j 6wi

• Change ² to ²+±:

29

   

©T¡ (©x

¤ ¡ y) + (1¡ ²)u = ¡Wz

j©T(©x¤ ¡ y) + (1¡ ²)uj 6 w

• Optimality conditions:Must be obeyed by any solution x* with support ¡ and sign sequence z

Page 30: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

• Change ² to ²+±:

@x =

((©T

¡©¡)¡1u¡; on ¡

0; on ¡c

30

   

©T¡ (©x

¤ ¡ y) + (1¡ ²)u+ ± (©T¡©@x¡ u) = ¡Wz

jÁTi (©x¤ ¡ y) + (1¡ ²)ui + ± (ÁTi ©@x¡ ui)j 6wi

minimize kWxk1 +1

2k©x¡ yk22 + (1¡ ²)uTx• Homotopy:

©T¡ (©x

¤ ¡ y) + (1¡ ²)u = ¡Wz

j©T(©x¤ ¡ y) + (1¡ ²)uj 6 w

• Optimality conditions:Must be obeyed by any solution x* with support ¡ and sign sequence z

Page 31: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

• Change ² to ²+±:

±¤ = min(±+; ±¡)

±+ = mini2¡c

µwi ¡ pidi

;¡wi ¡ pi

di

+

±¡ = mini2¡

µ¡x¤i@xi

+

31

   

©T¡ (©x

¤ ¡ y) + (1¡ ²)u+ ± (©T¡©@x¡ u) = ¡Wz

jÁTi (©x¤ ¡ y) + (1¡ ²)ui + ± (ÁTi ©@x¡ ui)j 6wi

minimize kWxk1 +1

2k©x¡ yk22 + (1¡ ²)uTx• Homotopy:

©T¡ (©x

¤ ¡ y) + (1¡ ²)u = ¡Wz

j©T(©x¤ ¡ y) + (1¡ ²)uj 6 w

• Optimality conditions:Must be obeyed by any solution x* with support ¡ and sign sequence z

Page 32: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

  -homotopy

minimize kWxk1 +1

2k©x¡ yk22 + (1¡ ²)uTx• Homotopy:

• Change ² to ²+±:

32

   

@x =

((©T

¡©¡)¡1u¡; on ¡

0; on ¡c

±¤ =min(±+; ±¡)

±+ = mini2¡c

µwi ¡ pidi

;¡wi ¡ pi

di

+

±¡ = mini2¡

µ¡x¤i@xi

+

©T¡ (©x

¤ ¡ y) + (1¡ ²)u+ ± (©T¡©@x¡ u) = ¡Wz

jÁTi (©x¤ ¡ y) + (1¡ ²)ui + ± (ÁTi ©@x¡ ui)j 6wi

Page 33: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

The Dantzig selector   -homotopy

• Dantzig selector:

• Primal-dual   -homotopy:

33

minimize kWxk1subject to j©T(©x¡ y)j ¹ q

maximize ¡¸T©Ty¡ kQ¸k1subject to j©T©¸j ¹ w;

minimize kWxk1 + (1¡ ²)uTx

subject to j©T(©x¡ y) + (1¡ ²)vj ¹ q

maximize ¡¸T(©Ty¡ (1¡ ²)v)¡ kQ¸k1subject to j©T©¸ + (1¡ ²)uj ¹ w;

Primal

Dual

Primal homotopy

Dual homotopy

Page 34: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: streaming system

• Signal observations:  

• Sparse representation:  

34

LOTcoefficients

LOT representation basesSignalLOTwindows

Measurement matrices SignalMeasurements Error

Page 35: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: streaming system

• Time-varying signal represented with block basesAND

• Streaming, disjoint measurements

35

DCTcoefficients

DCT representationSignalMeasurement matrices SignalMeasurements Error

Page 36: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: streaming system

• Time-varying signal represented with lapped basesOR

• Streaming, overlapping measurements

36

LOTcoefficients

LOT representation basesSignalLOTwindows

Measurement matrices SignalMeasurements Error

Page 37: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: streaming system

• Time-varying signal represented with lapped basesOR

• Streaming, overlapping measurements

37

LOTcoefficients

LOT representation basesSignalLOTwindows

acti

ve in

terv

al

Measurement matrices SignalMeasurements Error

acti

ve in

terv

al

Page 38: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: streaming system

• Iteratively estimate the signal over a sliding (active) interval:

38

Overlapping system matrix Sparsevector

Error

minimize kW®k1 +1

2k¹©~ª®¡ ~yk22

minimize kW®k1 +1

2k¹©~ª®¡ ~yk22 + (1¡ ²)uT®

Desired

Homotopy

Divide the system into two parts

Page 39: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Streaming signal recovery - Results

39

(Top-left) Linear chirp signal (zoomed in for first 2560 samples. (Top-right) Error in the reconstruction at R=N/M = 4. (Bottom-left) LOT coefficients. (Bottom-right) Error in LOT coefficients

Page 40: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Streaming signal recovery - Results

40

(left) SER at different R from ±1 random measurements at 35 db SNR(middle) Count for matrix-vector multiplications(right) Matlab execution time

SpaRSA: Sparse Reconstruction by Separable Approximation, Wright et al., 2009. http://www.lx.it.pt/~mtf/SpaRSA/YALL1: Alternating direction algorithms for L1-problems in compressive sensing, Yang et al., 2011. http://yall1.blogs.rice.edu/

Page 41: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Streaming signal recovery - Results

41

(Top-left) Mishmash signal (zoomed in for first 2560 samples. (Top-right) Error in the reconstruction at R=N/M = 4. (Bottom-left) LOT coefficients. (Bottom-right) Error in LOT coefficients

Page 42: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Streaming signal recovery - Results

42

(left) SER at different R from ±1 random measurements at 35 db SNR(middle) Count for matrix-vector multiplications(right) Matlab execution time

Page 43: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: dynamical system

• Signal observations:  

• Sparse representation:  

• Linear dynamic model:  

43

Dynamic model

Related work: [Vaswani ’08; Carmi et al. ’09; Angelosante et al. ’09; Zaniel et al. ’10; Charles et al. ’11]

Page 44: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Sparse recovery: dynamical system

• Signal observations:  

• Linear dynamic model:  

• Sparse representation:  

44

minimize kW®k1 +1

2k¹©~ª®¡ ~yk22 +

1

2k¹F~ª®¡ ~qk22 + (1¡ ²)uT®

Page 45: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic signal recovery - Results

45

(Top-left) HeaviSine signal (shifted copies) in image(Top-right) Error in the reconstruction at R=N/M = 4. (Bottom-left) Reconstructed signal at R=4. (Bottom-right) Comparison of SER for the L1- and the L2-regularized problems

Page 46: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic signal recovery - Results

46

(left) SER at different R from ±1 random measurements at 35 db SNR(middle) Count for matrix-vector multiplications(right) Matlab execution time

Page 47: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic signal recovery - Results

47

(Top-left) Piece-Regular signal (shifted copies) in image(Top-right) Error in the reconstruction at R=N/M = 4. (Bottom-left) Reconstructed signal at R=4. (Bottom-right) Comparison of SER for the L1- and the L2-regularized problems

Page 48: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Dynamic signal recovery - Results

48

(left) SER at different R from ±1 random measurements at 35 db SNR(middle) Count for matrix-vector multiplications(right) Matlab execution time

Page 49: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Part 2: Dynamic models in video

49

x1 x2 x3

...

xT

F1 F2 FT¡1

B2 B3 BT

Page 50: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Low-complexity video compression

50

Page 51: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Video compression

Compression is achieved by removing the spatio-temporal redundancies in the videos

time space

51

Page 52: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Video compression

Compression is achieved by removing the spatio-temporal redundancies in the videos 52

Motion comp.

I frame P or B frames

Page 53: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Video coding paradigm

Major blocks in the encoder: Motion estimation, Transform coding, Entropy coding

53

Low complexity

encoder

“Smart”decoder

In Out

Standard compliantbitstream

‘Smart’ encoder

“Dumb”decoder

In Out

Shift processing burden from the encoder to the decoder!

Page 54: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Compressive encoder

video frame with N pixels

vector

vector

Encoder complexity is minimal, while all the

computational load is shifted to the decoder

x y

M £ 1

N £ 1

A yx=

Random Sensing matrix

M £N

54

Page 55: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Structure in video (for recovery)

• Frame-by-frame recovery (same as recovery of independent images).

• Spatial structure:

– Wavelets

– Total-variation

DWT

• Recovery of multiple frames (images are temporally correlated)

• Temporal structure:

– Frame difference

– Inter-frame motion

x1 x2 x3

...

xT

F1 F2 FT¡1

B2 B3 BT 55

Page 56: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Structure…

DWT

Linear dynamical system:

x1 x2 x3...

xT

F1 F2 FT¡1

B2 B3 BT

yi = Aixi + ei

xi = Fi¡1xi¡1 + fi

xi = Bi+1xi+1 + bi

A yx=

Random Sensing matrix

M £N

(Linear measurements)

(forward motion pred.)

(backward motion pred.)

56

Page 57: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Accelerated dynamic MRI

57

Page 58: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Accelerated acquisition in MRI?

• How the cardiac MRI works:

Fourier plane---Phase-encoding step samples rows in the Fourier plane

Temporal resolution

One cardiac cycle

A small spatial area is scanned in every heart beat (e.g., 8 lines per heartbeat)

Acquisition during breathholdsDirect tradeoff between temporal resolution and lines per heart beat segmentNumber of lines per frame are the same 58

Page 59: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Parallel imaging

• Parallel imaging [SENSE, SMASH, SPACE-RIP, GRAPPA, …]:

26664

y1y2...

yT

37775 =

26664

A1 0 : : : 0

0 A2 : : : 0...

.... . .

...

0 0 : : : AT

37775

26664

x1x2...

xT

37775+

26664

e1e2...

eT

37775

´ y = Ax+ e

yi = Aixi + ei

Ai ´

264F­iS1i

...

F­iSCi

375

Receiver coils

MRI data courtesy: Hamilton and Brummer59

Page 60: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Motion-adaptive Spatio-temporal Regularization (MASTeR)

• We model temporal variations in the images using inter-frame motion!

• This way we are not using some fixed global model, instead, we learn the structure directly from the data.

Motion vectors

60

Page 61: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Motion-adaptive Spatio-temporal Regularization (MASTeR)

• Spatial structure: wavelets

• Temporal structure: inter-frame motion

• Linear dynamical system model:

yi = Aixi + ei

xi = Fi¡1xi¡1 + fi

xi = Bi+1xi+1 + bi

(Linear measurements)

(forward motion pred.)

(backward motion pred.)

61

Page 62: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Video recovery

62

Page 63: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Video reconstruction

DWT x1 x2 x3

...

xT

F1 F2 FT¡1

B2 B3 BT

63

DWT

yi = Aixi + ei

xi = Fi¡1xi¡1 + fi

xi = Bi+1xi+1 + bi

(Linear measurements)

(forward motion pred.)

(backward motion pred.)

Page 64: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Pseudocode:0. Find Initial frame estimatesRepeat

1. Calculate motion from frame estimates2. Use the motion information to refine estimates

Video reconstruction

DWT x1 x2 x3

...

xT

F1 F2 FT¡1

B2 B3 BT

Backward motion diff.

Spatial regularityData fidelity

Forward motion diff.

minimizex1;:::;xT

X

k

kAkxk ¡ ykk2 + ¿kkxkkª1+

®kkFkxk ¡ xk+1kª2+ ¯kkBkxk ¡ xk¡1kª3

64

DWT

Page 65: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Comparison of low-complexity encoders

coastguard

foreman

container

hall

65

CS-MC: L1 with motion compensation CS-DIFF: L1 with frame differenceCS: frame-by-frame

qJPEG: mask-based DCT thresholdingldct: linear DCT (zigzag) thresholding

CS measurements: partial wavelets + noiseletscompression ratio = N/M

Page 66: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Results(short axis):

R = 8R = 4

MA

STeRk-t FO

CU

SS with

ME/M

C

(e)

(f)

(b)(b)

(c)

(d)

(a)

y

x

k-t FOCUSS with ME/MC:-Temporal DFT sparsity-Motion residual with a reference frame or temporal average.

66

MASTeR: Motion-adaptive spatio-temporal regularization:

R: acceleration factor Sampling: 8 low-freq. lines + random sampling

[Jung et al., k-t FOCUSS: a general compressed sensing framework for high resolution dynamic MRI. MRM, 2009]

Page 67: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

67

k-t FOCUSS with ME/MC (full)

k-t FOCUSS with ME/MC (ROI)

MA

STeRk-t FO

CU

SS with

ME/M

C

t

y

(a)

(b)

R=4

(c)

(e)

R=8

(d)

(f)

MASTeR (ROI)

MASTeR (full)

Results (short axis)

Page 68: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Conclusion

68

Dynamic modeling Improve reconstruction by exploiting

the dynamical signal structure

• Motion-adaptive dynamical system

1. Low-complexity video compression

2. Accelerated dynamic MRI

Dynamic   updating Quickly update the solution to

accommodate changes

•   homotopy

– Breaks updating into piecewise linear steps

– Simple, inexpensive (rank-one update)

“warm start”

final solution

...

The more we know about the signal (dynamics) the faster and/or more accurately we can reconstruct

Page 69: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Future directions

•   -homotopy:

– Theoretical analysis

– Large-scale streaming problems

• Compressive sensing in videos:

– Adaptive sampling

– Distributed cameras

– Computational imaging (lightfield etc.)

• Medical imaging:

– MRI… (maybe hyper-polarized)

– Ultrasound

– EEG69

Page 70: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Select publications• Dynamic updating

– M. Asif and J. Romberg, ``Sparse recovery algorithm for streaming signals using L1-homotopy,'' Submitted to IEEE Trans. Sig. Proc., June 2013. [Arxiv]

– M. Asif and J. Romberg, ``Fast and accurate algorithm for re-weighted L1-norm minimization,'' Submitted to IEEE Trans. Sig. Proc., July 2012. [Arxiv]

– M. Asif and J. Romberg, ``Dynamic updating for L1 minimization,'' IEEE Journal of Selected Topics in Signal Processing, 4(2) pp. 421--434, April 2010.

– M. Asif and J. Romberg, ``Sparse signal recovery and dynamic update of the underdetermined system,'' Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, November 2010.

– M. Asif and Justin Romberg, ``Basis pursuit with sequential measurements and time-varying signals,'' in Proc. Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), Aruba, December 2009.

– M. Asif and J. Romberg, ``Dynamic updating for sparse time-varying signals,'' Conference on inf. sciences and systems (CISS), Baltimore, March 2009.

– M. Asif and J. Romberg, ``Streaming measurements in compressive sensing: L1 filtering,'' Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, October 2008.

• Dynamic model in video– M. Asif, L. Hamilton, M. Brummer, and J. Romberg, ``Motion-adaptive spatio-temporal

regularization (MASTeR) for accelerated dynamic MRI,'' accepted in Magnetic Resonance in Medicine, November 2012 [Early view DOI: 10.1002/mrm.24524].

– M. Asif, F. Fernandes, and J. Romberg, ''Low-complexity video compression and compressive sensing,'' preprint 2012. 70

Page 71: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Select publications• Sparsity and dynamics

– M. Asif, A. Charles, J. Romberg, and C. Rozell, ``Estimating and dynamic updating of time-varying signals with sparse variations,'' in Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP), Prague, Czech Republic, May 2011.

– A. Charles, M. Asif, J. Romberg, and C. Rozell, ``Sparse penalties in dynamical system estimation,'', Conference on Inf. Sciences and Systems (CISS), Baltimore, Maryland, March 2011.

• Streaming greedy pursuit– P. Boufounos and M. Asif, ``Compressive sensing for streaming signals using the

streaming greedy pursuit,'' in Proc. Military Commun. Conf. (MILCOM), San Jose, California, October 2010.

– M. Asif, D. Reddy, P. Boufounos, and A. Veeraraghavan, ``Streaming compressive sensing for high-speed periodic videos,'' in Proc. IEEE Int. Conf. on Image Processing (ICIP), Hong Kong, September 2010.

– P. Boufounos and M. Asif, ``Compressive sampling for streaming signals with sparse frequency content,'' Conference on Inf. Sciences and Systems (CISS), Princeton, New Jersey, March 2010.

• Channel protection– M. Asif, W. Mantzel, and J. Romberg, ``Random channel coding and blind

deconvolution,'' Allerton Conf. on Communication, Control, and Computing, Monticello, Illinois, October 2009.

– M. Asif, W. Mantzel, and J. Romberg, ``Channel protection: Random coding meets sparse channels,'' Information Theory Workshop, Taormina, Italy, October 2009.

71

Page 72: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Acknowledgements

• Special thanks to my collaborators

– William Mantzel

– Petros Boufonous

– Felix Fernandes

– Adam Charles and Chris Rozell

– Lei Hamilton and Marijn Brummer

• and my advisor, Justin Romberg

72

Page 73: Sparse recovery algorithms for streaming signals and videosasif/talks/asif-phd-slides.pdf · Dynamic Compressive Sensing: Sparse recovery algorithms for streaming signals and video

Thank you!

Questions?

73

Email: [email protected]: http://users.ece.gatech.edu/~sasif/