Compressed Sensing in Measurement How does it work, why … · what \signal processing" means. Is CSusefulin measurement? C. Narduzzi Compressed Sensing in Measurement. Compressed

Compressed Sensing in MeasurementHow does it work, why – and when?

Claudio Narduzzi

Department of Information EngineeringUniversity of Padua – Italy

Scuola “Italo Gorini” 2019

Napoli, Italy - September 4, 2019

Presentation Outline

1 Compressed IntroWhat is compressed sensing?Where’s the problem, really?Beginnings – and some application examples

2 CS Example: Frequency Domain MeasurementMeasuring a sinewave

3 Sensing or Sampling?

4 Solution of CS problemsSparse recovery and the `1-normSensing Scheme DesignRandom sensing matrix – Signal Processing analysis

5 A Look at CS ApplicationsBiomedical ImagingSensor monitoring application

6 Conclusions

Compressed IntroCS Example: Frequency Domain Measurement

Sensing or Sampling?Solution of CS problems

A Look at CS ApplicationsConclusions

What is compressed sensing?Where’s the problem, really?Beginnings – and some application examples

Compressed Intro

C. Narduzzi Compressed Sensing in Measurement





What is compressed sensing?

Let x be a (large) vector of N signal values. Consider matrix Φ,having size M × N with M � N.

The product y = Φ · x yields a (much) smaller, compressed vectorof data:

y = Φ · xCall Φ the sensing matrix, and y the vector of measured values.






Questions

Do the measured values y represent in compressed form thesame information as x, or is anything lost in the process?

Conditions on x?

Conditions on Φ?

Answers – Sparsity

Primary assumption: signal x is sparse “in some domain”

Sensing matrix Φ is “reasonably well-behaved”

If needed, x can be recovered from y “almost surely”

Characterisation and analysis of uncertainty are needed






A look past the “WOW” effect

An engineer typically finds the idea quite cool:

“measure little, learn a lot”

A kind of game of “hide and seek” . . .

. . . but, as soon as one gets into details, the mathematics mayget somewhat scary

or, theoretical details that are important to the understandingof CS may be missed, with disappointing results.

Well, surely that’s what researchers are for, right?






What’s this lecture about

The purpose of this lecture is:

go through CS, and survive;

get somewhat closer to an engineering viewpoint of CS.

We can get there. . .

. . . if you are not afraid of matrices and have at least an idea ofwhat “signal processing” means.

Is CS useful in measurement?






What is a “sparse” signal ?

x = Ψ · aA linear transformation allows to map x into a vector a that issparse, i.e., it has few non-zero elements.






Signal model: the “sparsifying” matrix

x = Ψ · a

In CS parlance, Ψ is called a sparsifying matrix, because it mapsthe signal into a domain where its representation is sparse.

the equation gives a linear relationship between x and a;

Ψ may be a transform matrix (e.g., a discrete Fouriertransform), but needs not be, nor even be orthogonal.

Equation x = Ψ · a can be considered a model of the measurand.






Putting it all together . . .

y = Φ · Ψ · agiven M “compressive” measurements in vector y

with known sensing matrix Φ and sparsifying matrix Ψ

find the solution a of: y = Φ ·Ψ · a = A · a

But:dim[y]=M, dim[a]=N with M�N: underdetermined ?






Where’s the problem, really?

it is assumed that a has only K non-zero elements (K -sparse)

if K < M the equation is actually overdetermined, BUT

we do not know where non-zero elements of a are!

Sparsity constraint: find a as the maximally sparse solution






Formalising the approach

The problem can be formally expressed in mathematical terms:

the count of non-zero elements is a (pseudo)-norm (`0);

recast the problem as constrained optimisation:

mina ‖a‖0 subject to: y = ΦΨa

All nice and clean, but:

this problem has combinatorial complexity;

if a is assumed to have K non-zero elements, there are(NK

)possible combinations






When did it start?

Compressed Sensing (CS) has been developed for over ten years asan attractive and sophisticated mathematical theory

Candés, 2006

[VOLUME 25 NUMBER 2 MARCH 2008]

SP Magazine, 2008






Solution of CS problems

Convex optimisation approach

replace `0 norm by `1 norm and solve:

mina ‖a‖1 subject to: y = ΦΨa

or, using Lagrange multipliers:

mina[‖a‖1 + λ‖y −ΦΨa‖2

]interest in compressed sensing sparked by significant progressin `1–`2 optimisation

“seminal papers” written by mathematicians and algorithmists






Spreading interest . . .

and the measurement community took notice:

Introduzione al campionamento compressoSparsità e l’equazione Ax = y

Emanuele Grossi

DAEIMI, Università degli Studi di Cassinoe-mail: [email protected]

Gorini 2010, Pistoia






Measurement application 1: Single-Pixel Camera

incident light (image) modulated by a digital micromirrordevice (DMD);

digital micromirror device (DMD) DMD structure

each micromirror changes tilt individually, sending light eithertowards or away from detector;

modulated image sent to single pixel detector.






Single-pixel camera - recovery

acquisition of detector output with M different randommodulation patterns (different mirror tilt angles);

M digitised values from one high-resolution ADC;total number of pixels = DMD elements = N � M;

Digital Signal Processing Group, Rice University






Variation on a theme – single-pixel hyperspectral imaging

Single detector makes complex sensing feasible:

high-sensitivity detectors, optical spectrum analyser (OSA).

F. Magalhes, M. Abolbashari, F.M. Arajo, M.V. Correia, F. Farahi, “High-resolution hyperspectral single-pixelimaging system based on compressive sensing”, Opt. Eng. 51(7), (May 25, 2012)






Measurement application 2: Monitoring RF Spectral Bands

Y.C. Eldar, M. Mishali et al. several papers, see:http://webee.technion.ac.il/people/YoninaEldar/index.php

a modulated wideband converter can be realised by a RFhardware circuit implementation (shown);

MWC enables multi-band detection of RF signals.






Modulated wideband converter (MWC)

Y.C. Eldar, M. Mishali et al. several papers, see:http://webee.technion.ac.il/people/YoninaEldar/index.php

Note how signals widely spaced in frequency are aliased intonarrower bands, that require lower sampling rates.





Measuring a sinewave

CS Example: FrequencyDomain Measurement






A simple case: measuring a sinewave

How many samples do we need to represent a sinewave?

let the sinewave frequency be f0

(Shannon) a signal x(t), whose frequency band isupper-limited to fMAX , is completely determined from asequence of samples taken at uniformly spaced intervals:

T ≤ 12fMAX

in principle, the number of samples is infinite

x [n] with: −∞ < n < +∞.






“Degrees of freedom” – do we need infinite samples?

A sampled sinewave can be simply described by the equation:

x [n] = A0 cos(2πf0T · n + φ0)that contains just three unknowns: A0, f0 and φ0.

Other than by infinite observation, there seems to be no wayto distinguish in the time domain a sinewave with frequencyf0 from a sinewave of frequency: f

′0 = f0 + δ for any value of δ.






Linear/non-linear relationships

Using for x [n] = A0 cos(2πf0T · n + φ0) the alternative expression:

x [n] = A0ejφ0 · e j2πf0T ·n + A0e−jφ0 · e−j2πf0T ·n

one may note that samples x [n] are dependent:

linearly, on a complex parameter depending, in turn, on A0and φ0

through a non linear relationship, on frequency f0

If frequency f0 is known, samples x [n] depend linearly on just twoparameters – A0, φ0. Sinewave is sparse in the frequency domain.






Uniform sampling, known frequency – linear equation

T = 12f0 – under-determined system (rank-1 matrix)[x [0]

x [1]

]=

[12

12

−12 −12

][A0e

jφ0

A0e−jφ0

]

coherent sampling, T = 13f0 (exactly three samples perperiod) – A0 and φ0 can be determined (rank-2 matrix) x [0]x [1]

x [2]

=

12

12

12e

j 2π3

12e−j 2π

3

12e

j 4π3

12e−j 4π

3

[ A0e jφ0A0e

−jφ0

]

Required number of samples is finite and very small






Introducing a finite frequency grid

So far, so good:

either f0 unknown, acquire an infinite number of samples;

or, f0 known perfectly, acquire just three samples;

anything in between?

Let’s make life a bit easier:

define a discrete grid with “reasonably fine” step ∆F ;

frequency value can be given as f0 = k0 ·∆F for some k0 ∈ Zassume a finite frequency range: |f | ≤ fMAXand choose N so that fMAX∆F =

N2

the finer the grid, the larger N needs to be






Inverse DFT as “sparsifying matrix”

If T = 1N·∆F , the mathematical expression of sinewave samples

when f = k ·∆F is: x [n] = A02 ejφ0e j2π

k0nN + A02 e

−jφ0e−j2πk0nN

N time-domainsamples

inverse Discrete Fouriertransform (IDFT) matrix

N Fourier (DFT)coefficients

x = Ψ · aC. Narduzzi Compressed Sensing in Measurement





Sparsity and spectral leakage – everybody knows. . .

If f0 6= k0 ·∆F , all DFT coefficients may be non-zero, but:for each frequency component having frequency fx ∼= kx∆F ,DFT coefficient magnitude decreases as |k − kx | gets largerif the signal is compressible, that is:

|ak | ≤ C · |k − kx |−q with:q > 0

it can likewise be recovered from compressive measurements






“Sensing” the sinewave

y = Φ · x

Rather than acquire a (large) vector x of N signal samples, we usethe M × N sensing matrix Φ to acquire M � N measurements:

Write: Φ =

φT

1

φT2...

φTM

Then:

y1

y2...

yM

=φT

1x

φT2

x...

φTM

x

Each measurement is a linear combination of the signal samples.





Sensing or Sampling?





Compressed Sensing or Compressive Sampling?

CS is variously understood to be a short form of either:“compressed sensing”, or: “compressive sampling”

what’s the difference – if any at all?

let the generic signal – or “measurand” x(t) be “sensed”through a linear device, represented by math operator φ(t):

y(t) =

∫ +∞−∞

x(τ)φ(t − τ)dτ

let y(t) be uniformly sampled at suitable instants ti :

y(ti ) =

∫ +∞−∞

x(τ)φ(ti − τ)dτ





Comparison with the sensing matrixConsider a sequence of measurement vectors y

i, y

i+1, . . . and

compare, for ti = iNT :

CS measurement vector:

yi

y1[i ]

y2[i ]...

yM [i ]

=φT

1xi

φT2

xi...

φTM

xi

“linearly sensed” measurands:y1[i ]

y2[i ]...

yM [i ]

=

∫ +∞−∞ x(τ)φ1(iNT − τ)dτ∫ +∞−∞ x(τ)φ2(iNT − τ)dτ

...∫ +∞−∞ x(τ)φM(iNT − τ)dτ

Each element in the vector sequence y

i, y

i+1, . . . is obtained from

the original “signal” through filtering and (down)sampling.





Can’t we always see it this way?

The sensing matrix Φ has been interpreted as the mathematicaldescription of a bank of M linear, finite impulse response (FIR)filters that process input vectors x in parallel, followed by adownsampler.

Is that all it takes?

Actually, no.

multi-dimensional data (e.g., image processing), are alsoarranged into vectors – structure harder to understand/exploit.

CS conditions are usually discussed with regards to Φ and Ψ

examples where x is indeed a one-dimensional signal vectorare still useful and help get a better grasp on CS.





Relation to Sampling Theory

“Shannon theorem” (Cauchy, 1843): If a function f (t) contains nofrequencies higher than W cps, it is completely determined by giving its

ordinates at a series of points spaced 1/2W seconds apart

Can be seen as:

bandlimiting by a linear filter:

y(t) =

∫ +∞−∞

x(τ)φ(t − τ)dτ with: φ(t) = sin 2πWt2πWt

sampling the filtered version of x(t) at instants ti =i

2W :

y(ti ) =

∫ +∞−∞

x(τ)sinπ(i − 2W τ)π(i − 2W τ)

dτ





Sparse recovery and the `1-normSensing Scheme DesignRandom sensing matrix – Signal Processing analysis

Solution of CS problems






“Forcing” sparsity

constrain the solution to be sparse:vector a should have few non-zero elements (`0 norm);it should satisfy the measurement equation: y = Φ ·Ψ · a

replace `0 norm by `1 norm and solve constrainedoptimisation problem:

mina‖a‖1 subject to: y = ΦΨa

Greedy algorithms

iterative, sub-optimal solution

good computational efficiency






Where are the non-zeroes?

If we already knew where non-zero elements are, i.e., if vector ahad known support, solving the problem would be fairly simple.

x = ΨS · a






Known-support reconstruction

Locations of non-zero columns in matrix ΨS coincide withthe support of vector a.Only the reduced vector aS contains non-zero elements.

x = ΨS · aSC. Narduzzi Compressed Sensing in Measurement





Pseudo-inverse solution

Given a measurement vector y with M elements, and K < Mnon-zero elements of a, there are more equations than unknowns:

y = Φ ·ΨS · aS

Values of non-zero elements of a are determined by computing the(Moore-Penrose) pseudo-inverse:

aS =[(ΦΨS)

H (ΦΨS)]−1

(ΦΨS)H y






Sparse recovery and the `1-norm

Different shape of `1 and `2 “balls” in R2






Sparse recovery and the `1-norm, plus uncertainty

`1 norm minimises the number of “large elements”






What makes a good CS scheme?

Sensing matrix design:

what are the requirements of a good sensing matrix?

are there specific design criteria?

since the measurement equation is:

y = ΦΨa

does design of Φ depend on matrix Ψ?






How to set-up a CS problem

Possibly, most difficult part of the job. It requires:

analysis of the signal at hand to:consider a suitable linear signal model

define the sparsifying matrix

analysis/design of a sensing schemefind out corresponding sensing matrix

assess its properties w.r.t. the sparsifying matrix






Recovery and unique solution

Remember the original problem is:

mina ‖a‖0 subject to y = ΦΨa

but we do not want to solve this one.

We require a K -sparse solution to be unique:

assume two solutions exist, a1 and a2, then:

ΦΨa1 6= ΦΨa1 → ΦΨ(a1 − a2) 6= 0for this to hold, any subset of 2K columns of Ω = ΦΨ mustbe linearly independent






Maximal incoherence and random sampling

According to CS theory, Φ should be incoherent with Ψ.Coherence is a measure of similarity between Φ and Ψ, definedas:

µ(Φ,Ψ) = maxi ,j ‖〈φi , ψj〉‖ i = 1, 2, . . .M j = 1, 2, . . .N

where:incoherence 1 ≤ µ ≤

√N maximal coherence

Assume matrix Φ is built from a set of vectors forming anorthonormal random basis: it can be shown that:

µ(Φ,Ψ) =√

2 logN with “overwhelming probability”

(Note: for N = 100, one has:√N = 10, whereas

√2 loge N

∼= 3. Not thelower bound, but a reasonably good value.)






Design of the sensing matrix

Random sensing matrices

this topic is among the most extensively discussed in CS

a well-known claim is that a random sensing matrix Φ allowsto satisfy key CS conditions “with overwhelming probability”

this is helped by the fact that sparsifying matrices are oftenorthonormal or, at least, “very close” to preserving vectornorms

design of deterministic Φ possible, met more limited interest






Random sampling in a CS framework

Consider M � N values extracted from x by random sampling.This is equivalent to premultiplying ΦS (or Φ) by a (0, 1) randommatrix, with a single non-zero element per row, to obtain vector y:

y = Φ · x = Φ ·ΨS · aSC. Narduzzi Compressed Sensing in Measurement





Random Sensing Matrix

simplest: random sampling (0− 1 matrix)other common possibilities:

* random matrix elements drawn from Gaussian probabilitydensity function

* Bernoulli matrix: elements assuming two values, [−1, 1] withequal probability – see: single-pixel camera, modulatedwide-band converter

Why “overwhelming probability”?

some signal vectors x may not always be recoverable

position and magnitude of non-zero elements in a is a prioriunknown, motivating analysis based on success probability

relating such probabilities to uncertainty is desirable






Noise, stability and the Restricted Isometry Property (RIP)

In practice, attention has to be paid to the effects of noise anduncertainty. Consider:

y = ΦΨa + w = Ωa + w

where w is assumed to be additive white Gaussian noise.

Given a positive integer K , the isometry constant δK of Ω is thesmallest number such that, for any K -sparse vector a:

(1− δK )‖a‖22 ≤ ‖Ωa‖22 ≤ (1 + δK )‖a‖22 with δK < 1

δK is a sort of measure of the numerical conditioning of Ω. If Ψ isorthonormal, RIP can be referred to Φ only, becoming one of thesensing matrix design parameters.






Noise effect on CS solutions

The K -sparse vector aK is an approximate solution to the “noisy”equation y = Ωa + w, bounded by:

‖aK − a‖2 < CN · ε+ CK · σK

where:

ε upper bounds sensing noise (on vector y);

σK bounds approximation error (on vector a);

CN , CK are constants

since CN > 1, sensing noise is amplified: employ low-noisesensors.






RIP and its consequences

If Ω satisfies RIP of order 2K it is possible to find a K -sparsesolution aK , such that:

in the noiseless case (i.e., when w ∼= 0) it is always possible torecover a unique K -sparse solution x̂

if δ2K ≤√

2− 1, there exist two positive constants C0, C1that bound the distance of the recovered vector x̂ from theoriginal vector x:

‖x̂− x‖2 ≤ C0‖xK − x‖1√

K+ C1ε.






A “greedy” alternative – orthogonal matching pursuit

define residue r. Given an estimate â, one has: r = y −Ωâcompute the N-size vector ΩHri (where

H denotes conjugate

transposition – the subscript is the iteration index)

the set of indices of the largest non-zero elements of a, calledthe support S , is found iteratively (initially, S0 = ∅):

Si = Si−1 ∪{

arg maxn

∥∥∥[ΩHri]n

∥∥∥2

}where n is the index for elements in ΩHri , as well as in a.

the pseudo-inverse solution then follows:

âSi =[(ΦΨSi )

H (ΦΨSi )]−1

(ΦΨSi )H y






Orthogonal matching pursuit (OMP) – how does it work?

since S0 = ∅, initially âS0 = 0 and r1 = yconsider then the first iteration and look for the index of thelargest non-zero element of ΩHy:

arg maxn

∥∥∥[ΩHri]n

∥∥∥2

∼= arg maxn

∥∥∥[ΩHΩa]n

∥∥∥2

is it also the largest non-zero element of a?

“reasonably” it is, because of the restricted isometry property

then, OMP works like “peeling off” the most significantcontributions to y, one by one, in descending order






How many measurements?

Generally, the number of measurements required to ensure the CSproblem has a unique K -sparse solution is of the order of:

K log

(N

K

)which holds for both `1 − `2 and greedy approaches.

(Note: for N = 100 and K = 2 (e.g., single sinewave), K logeNK∼= 8: less than

M = 10 random measurements may suffice, instead of N = 100 samples. The

number drops to 3 with coherent sampling – quite close in performance.)

If RIP of order 2K holds, there exists a positive constant C suchthat:

M ≥ C · K log(N

K

)C. Narduzzi Compressed Sensing in Measurement





What all this means in practice...

According to CS theory, the acquisition stage:

is linear and can be modelled by a matrix equation;

is totally blind, i.e. no prior information is needed;

can be random, extracted by some probability distribution;

is required to satisfy RIP of order 2K ;

can provide compression, to a rate of nearly one order ofmagnitude.

What changes take place going from x to y?






Random sensing matrix – Signal Processing analysis

Recall the sensing matrix Φ can be interpreted as a bank of M FIRfilters in parallel. Each sensing matrix row contains the coefficientsone FIR filter:

ym[i ] = φTm

xn =N−1∑n=0

x(iNT − nT )φm(nT )

where:

φm(nT ) ∈ {+1,−1}let: ∆F =

1NT

three filter frequencyresponses shown in plot






Random sensing matrix as a filter bank

Each filter randomly weights signal frequencies up to 12T

Binary sequences are uncorrelated, average PSD being uniformbetween 0 and 12T (see plot on the left).






Down sampling → random aliasing

continuously sampled signal, CSoperates on N-lengthconsecutive blocks

each measurement is takenfrom filter outputs at the rate ofone sample every NT seconds

multiple randomly aliasedsamples

any signal “randomly aliased” into frequency band [0, 12NT ]

recovery possible if, collectively, a “de-aliased” signal can bereconstructed from compressed measurements





Biomedical ImagingSensor monitoring application

A Look atCS Applications






Tomography - problem analysis

Main features of a tomographic problem:

medical imaging scanner expected output image:g(n1∆, n2∆), where ∆ is pixel size;

sensors acquire cross-section information from a fewdifferent angular positions only;

acquired signals contain spatially encoded frequencycomponents at a few frequencies.






Tomography case study

Example Test Image (Logan-Shepp Phantom)

acquisition along radials (b)

spatial Fourier transform

finite number of frequencies

E. J. Candés, J. Romberg, T. Tao, Robust UncertaintyPrinciples: Exact Signal Reconstruction from Highly

Incomplete Frequency Information , IEEE Trans. Inform.Theory, 52(2): 489 - 509.






What about noise?

Tomographic imaging requires solving equation:

y = Rg

where linear operator R represents sensing by the scanner (oftengiven as Radon transform), to obtain vector g (that containsimage pixels)

for noisy measurements, y = Rg + n

then, a minimal energy solution would be:

ĝ = ming‖g‖2 s.t. ‖y − Rg‖2 < ε

This produces artifacts - (see image).






Tomography - convex optimization

Artifacts in energy-constrained backpropagation are allowed by thesparsity-indifferent constraint

min total variation (TV): minimise energy of first-orderderivatives along x1, x2 axes:

‖g‖TV =∑n1,n2

√|D1 [g(n1, n2)] |2 + |D2 [g(n1, n2)] |2

ĝ = ming‖g‖TV s.t. ‖y − Rg‖2 < ε

' `1-norm bound on image wavelet coefficients.






Compressive Sensing MRI

MRI scans may employ different patterns;

acquiring enough scans to obtain diagnostic quality images isoften a lengthy process;

motion artifacts;

patient well-being can be affected (long spells of immobility,claustrophobia, difficult breathing);

reduce number of scans, preserve information;

fast MRI.






Sensor monitoring – Electrocardiogram (ECG)

long-term monitoring of ECG plotsis needed for several heart-relateddiseases

high sampling rate (250/360 Hztypical)

high-resolution ADC (16-bittypical) to account for signaldynamics and allow for baselinefluctuations






Random sensing matrix

Bernoulli matrix (randomly distributed ’+1’, ’−1’ values)obtained from pseudo-random binary sequence (PRBS)

fixed-length N-sample blocks

baseline removal

although noise level may increase,compressed measurementshave narrower dynamics (seeplot) – 8-bit digitization suffices






Working with compressed measurements – detection

A. Galli, C. Narduzzi, G. Giorgi, “ECG Monitoring and Anomaly Detection Based on Compressed Measurements”,2018 3rd International Conference on Biomedical Imaging, Signal Processing, Bari, Italy, 11–13 October 2018.

when a specific individual is monitored, basic shapes of ECGQRS-wave, T-wave and P-wave components are knowable

templates can be built in the compressed domain

detection of ECG wave components by simple matching





Ten years later

SP Magazine, 3/2018 SP Magazine, 4/2018

Community: https://nuit-blanche.blogspot.com/ (I. Carron)


Conclusions

analysis and exploitation of structure within data remains akey focus of research

advanced measurement applications are relying on increasinglysophisticated signal processing methods

Compressive Sensing is primarily a way to reduce theamount of sensor data;

sensing and reconstruction can be entirely separated, bothconceptually and, possibly, from the physical viewpoint;

open field of research for efficient reconstructionalgorithms;

enhance data acquisition (faster, less data) withoutsacrificing accuracy.

Thank You

M. Fornasier, H. Rauhut, Compressive sensing, ...

CS-based reconstruction of a fresco in Cappella Ovetari (Eremitani Church, Padua, Italy) from remaining fragments(in colour) and black and white photographs. Frescoes in Cappella Ovetari were hit and almost completely

destroyed in Allied air bombing during World War II. Arguably, an extreme approach to sparsity.

Compressed IntroWhat is compressed sensing?Where's the problem, really?Beginnings – and some application examples

CS Example: Frequency Domain MeasurementMeasuring a sinewave

Sensing or Sampling?Solution of CS problemsSparse recovery and the 1-normSensing Scheme DesignRandom sensing matrix – Signal Processing analysis

A Look at CS ApplicationsBiomedical ImagingSensor monitoring application

Conclusions

Documents

Compressed Sensing in Measurement How does it work, why … · what \signal processing" means. Is CSusefulin measurement? C. Narduzzi Compressed Sensing in Measurement. Compressed