Signal decompositions using trans-dimensional Bayesian methods: Alireza Roodaki's Ph.D. defense

Relabeling and summarizing posterior distributionsProposed approach

ResultsConclusion

Signal decompositions usingtrans-dimensional Bayesian methods

Alireza Roodaki

Ph.D. Thesis Defense

Department of Signal Processing and Electronic Systems

2012, May 14th

Advisors: Julien Bect and Gilles Fleury

Alireza Roodaki (SUPELEC) Signal decompositions using trans-dimensional . . . 1/ 52

ResultsConclusion

Example 1: Detection and estimation of muons in theAuger project

Figure: A conceptual shower(http://auger.org).

Ultra high energy particlescoming from space(E ∼ 1019eV)

How and where?

What is their composition(Proton, Iron)?

ResultsConclusion

How and where?

ResultsConclusion

How and where?

ResultsConclusion

Example 1: Detection and estimation of muons in theAuger project (Contd.)

Figure: A conceptual shower anddetectors (water tanks)(http://auger.org).

muons are generatedwhen particles cross theatmosphere

the number k of muonsand their arrival times tµare indicators of the originand composition of particle

ResultsConclusion

Figure: Water tank detector.

ResultsConclusion

t [ns]

100 200 300 400 500 6000

Figure: Observed signal (n)

Prof. Balázs Kégl from LAL, University of Paris 11.Alireza Roodaki (SUPELEC) Signal decompositions using trans-dimensional . . . 4/ 52

ResultsConclusion

t [ns]

100 200 300 400 500 6000

Figure: Observed signal (n)

Prof. Balázs Kégl from LAL, University of Paris 11.Alireza Roodaki (SUPELEC) Signal decompositions using trans-dimensional . . . 4/ 52

ResultsConclusion

Example 2: Spectral analysis

radial frequency0 0.5 1 1.5 2 2.5 3

0 10 20 30 40 50 60

Figure: Observed signal (top) and itsperiodogram (bottom).

Applications

RADAR / SONAR

Array signal processing

Vibration analysis

ResultsConclusion

Example 2: Spectral analysis (Cont.)

Detection and estimation of sinusoids in white noisemodel the observed signal y by sinusoidal components

ResultsConclusion

observed signal

Mk : y [i] =

aj cos[ωj i] + bj sin[ωj i])

+ n[i].

ResultsConclusion

observed signal

Mk : y [i] =

+ n[i].

Joint model selection and parameter estimation problem

ResultsConclusion

Trans-dimensional problems

Def. The problems in which the number of things that wedon′t know is one of the things that we don′t know [Green,

2003.]

ResultsConclusion

2003.]

space X =⋃

k∈K{k} ×Θk with points x = (k ,θk )

➠ k ∈ K denotes number of components

➠ θk ∈ Θk is a vector of component-specific

parameters

ResultsConclusion

2003.]

space X =⋃

k∈K{k} ×Θk with points x = (k ,θk )

➠ k ∈ K denotes number of components

➠ θk ∈ Θk is a vector of component-specific

parameters

Applications:➠ Spectral Analysis (Array signal processing)

➠ (Gaussian) Mixture modeling & Clustering

➠ Object detection and recognition

ResultsConclusion

Bayesian inference

Likelihood

p(x | y) =p(y | x) p(x)

Xp(y | x ′) p(x ′)dx ′

ResultsConclusion

Bayesian inference

Likelihood

p(x | y) =p(y | x) p(x)

Xp(y | x ′) p(x ′)dx ′

Prior distribution

ResultsConclusion

Bayesian inference

Likelihood

p(x | y) =p(y | x) p(x)

Xp(y | x ′) p(x ′)dx ′

Prior distributionPosterior distribution

ResultsConclusion

Bayesian inference

Likelihood

p(x | y) =p(y | x) p(x)

Xp(y | x ′) p(x ′)dx ′

x = (k ,θk )

ResultsConclusion

Bayesian inference

Likelihood

p(x | y) =p(y | x) p(x)

Xp(y | x ′) p(x ′)dx ′

x = (k ,θk )

➠ both detection and estimation problems

ResultsConclusion

Bayesian inference

Likelihood

p(x | y) =p(y | x) p(x)

Xp(y | x ′) p(x ′)dx ′

x = (k ,θk )

➠ both detection and estimation problems

high-dimensional / intractable integrals

ResultsConclusion

Markov Chain Monte Carlo (MCMC) methods

generate samples from the posterior distribution of interest(target distribution), say, π.

construct a Markov chain (x (1), . . . , x (M)) that under someconditions converges to π.

ResultsConclusion

Famous algorithms:➠ Metropolis-Hastings (MH) sampler [Metropolis, et al.

1953, Hastings, 1970.]

➠ Gibbs sampler [Geman and Geman, 1984.]

➠ RJ-MCMC sampler [Green, 1995.]

[Robert and Casella, 2004.]

ResultsConclusion

Famous algorithms:➠ Metropolis-Hastings (MH) sampler [Metropolis, et al.

1953, Hastings, 1970.]

➠ Gibbs sampler [Geman and Geman, 1984.]

➠ RJ-MCMC sampler [Green, 1995.]

[Robert and Casella, 2004.]

ResultsConclusion

Example 2: spectral analysis (Cont.)

RJ-MCMC sampler ⇒ variable dimensional samples

Iteration number160 170 180 190 200

Outline

1 Relabeling and summarizing posterior distributionsLabel-switching issueVariable-dimensional summarization

Outline

2 Proposed approachAn original variable-dimensional parametric modelEstimating the model parameters (SEM-type algorithms)Robustifying strategies

Outline

3 ResultsDetection and estimation of sinusoids in white noiseDetection and estimation of muons in the Auger project

Outline

4 Conclusion

Outline

4 Conclusion

ResultsConclusion

Label-switching issueVariable-dimensional summarization

summarizing posterior distributions

Posterior = all the information

ResultsConclusion

Posterior = all the information➠ It is a complex mathematical object (not easy to

manipulate)

ResultsConclusion

manipulate)

(RJ)-MCMC sampler

ResultsConclusion

manipulate)

(RJ)-MCMC sampler➠ What to do with the generated samples?

ResultsConclusion

manipulate)

Summarizationhuman readable summaries

ResultsConclusion

manipulate)

interpretable figures (e.g., histograms)

ResultsConclusion

manipulate)

interpretable figures (e.g., histograms)

statistical measures (e.g., mean and variance)

ResultsConclusion

Fixed-dimensional problems

uni-modal uni-variate case

Samples

0 2 40

report location (mean and median) and dispersion(variance and confidence intervals) parameters

µ σ2

2.0 0.25

ResultsConclusion

Label-switching issue

Additive mixture: lack of identifiability

the likelihood is invariant under relabeling of components

ResultsConclusion

the posterior distribution is invariant under permutation ofcomponents

ResultsConclusion

Comp. #1

Comp. #2

Comp. #3

0.5 0.75 10

10Marginal posteriors ofcomponent-specificparameters are identical!

ResultsConclusion

Comp. #1

Comp. #2

Comp. #3

0.5 0.75 10

10Marginal posteriors ofcomponent-specificparameters are identical!How to summarize theposterior information?

ResultsConclusion

strategies to deal with label-switching

imposing artificial “identifiability constraints”Exp: sorting the components [Richardson and Green, 1997.]

Comp. #1

Comp. #2

Comp. #3

0.5 0.75 10

Figure: components are sorted based on ω.

ResultsConclusion

strategies to deal with label-switching

imposing artificial “identifiability constraints”Exp: sorting the components [Richardson and Green, 1997.]

Comp. #1

Comp. #2

Comp. #3

0.5 0.75 10

Figure: components are sorted based on ω.

relabeling algorithms [Celeux, et al. 1998, Stephens, 2000, Jasra,

et al, 2005, Sperrin et al, 2010, Yao, 2011.].

ResultsConclusion

Example 2: spectral analysis (Cont.)Variable-dimensional posterior distribution

p(k |y) ω0.5 0.75 10 0.3 0.6

Figure: Posteriors of k and sorted radial frequencies, ωk , given k .Alireza Roodaki (SUPELEC) Signal decompositions using trans-dimensional . . . 16/ 52

ResultsConclusion

Classical Bayesian approaches

Bayesian Model Selection (BMS)

One model is selected (estimated) by looking at the MAP,i.e. k = argmax p(k |y).

Component-specific parameters are summarized givenk = k .

ResultsConclusion

p(k |y) ω

0.5 0.75 10 0.3 0.6

Figure: The model with k = 2 is selected.

ResultsConclusion

Classical Bayesian approaches

One model is selected (estimated) by looking at the MAP,i.e. k = argmax p(k |y).

Component-specific parameters are summarized givenk = k .

Bayesian Model Averaging (BMA)

Use the information from all possible models:p(∆|y) =

k p(∆|k , y)p(k |y)

However, ∆ cannot be ωk as its size changes from modelto model.

ResultsConclusion

Binned data representation (∆ = N(Bj)):

E(N(Bj) | y) =

kmax∑

E(N(Bj) | k , y) · p(k | y)

wherej = 1, . . . ,Nbinand E(N(Bj)) is theexpected number ofcomponents in binBj .

ω0 0.5 1 1.5 2 2.5 3

Figure: Expected number of componentsusing BMA.

ResultsConclusion

Are BMS and BMA approaches satisfactory?

➠ selects a model ⇒ component-specific parameters

➠ losing information from the discarded models

➠ ignoring the uncertainties about the presence ofcomponents.

ResultsConclusion

Are BMS and BMA approaches satisfactory? (Cont.)

➠ appropriate for signal reconstruction and prediction

➠ does not provide information about component-specificparameters

ResultsConclusion

A novel approach is needed!

ResultsConclusion

A novel approach is needed!

Properties of an “ideal” approach

information from all (plausible) models➠ interpretable summaries for component-specific

parameters

uncertainties about the presence of components

Outline

4 Conclusion

ResultsConclusion

An original variable-dimensional parametric modelEstimating the model parameters (SEM-type algorithms)Robustifying strategies

Big picture: relabeling and summarizing posteriordistributions

True posterior f = p(· | y)

0 1 2 30

Approximate posterior qη

0 1 2 30

Parametric family {qη, η ∈ N}

Measure of “distance”

[Stephens, 2000.]

ResultsConclusion

0 1 2 30

Samples

0 1 2 30

Samples

[Stephens, 2000.]

ResultsConclusion

Fixed-dimensional problems

uni-modal uni-variate case

Samples

0 2 40

report location (mean and median) and dispersion(variance and confidence intervals) parameters

µ σ2

2.0 0.25

ResultsConclusion

variable-dimensional approximate posterior

An original (variable-dimensional) parametric model qη

Four main requirements:1 Must be defined on the same space X =

k∈K{k} ×Θk

ResultsConclusion

k∈K{k} ×Θk

2 Must be permutation invariance

ResultsConclusion

k∈K{k} ×Θk

2 Must be permutation invariance3 Must be “simple” (small number of parameters)

ResultsConclusion

k∈K{k} ×Θk

2 Must be permutation invariance3 Must be “simple” (small number of parameters)4 Must be able to capture the main features of the posterior

distributions typically met in practice.

ResultsConclusion

A generative model point of view

· · ·

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ResultsConclusion

u11 u2

· · ·

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ResultsConclusion

ξ1 u11

ξ2 u22

· · ·

ξL uLL

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ξl ∈ {0, 1}

ResultsConclusion

ξ1 u11

ξ2 u22

· · ·

ξL uLL

{u l | ξl = 1}

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ξl ∈ {0, 1}

ResultsConclusion

ξ1 u11

ξ2 u22

· · ·

ξL uLL

{u l | ξl = 1}

random arrangement

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ξl ∈ {0, 1}

ResultsConclusion

ξ1 u1

1ξ2 u2

· · ·

ξL uL

{u l | ξl = 1}

random arrangement

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ξl ∈ {0, 1}

ξl ∼ B(πl)

ResultsConclusion

ξ1 u1

π1µ1 Σ1

1ξ2 u2

π2µ2 Σ2

· · ·

ξL uL

πLµL ΣL

{u l | ξl = 1}

random arrangement

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ξl ∈ {0, 1}

ξl ∼ B(πl)

ul ∼ N (µl ,Σl)

ResultsConclusion

ξ1 u1

π1µ1 Σ1

1ξ2 u2

π2µ2 Σ2

· · ·

ξL uL

πLµL ΣL

{u l | ξl = 1}

random arrangement

x = (k ,θk ) ∈ X =⋃

k{k} ×Θk

ul ∈ Θ

ξl ∈ {0, 1}

ξl ∼ B(πl)

ul ∼ N (µl ,Σl)

ηl = {πl ,µl ,Σl}

ResultsConclusion

1 for l = 1, . . . , Lgenerate binary number ξl ∼ B (πl) end

ResultsConclusion

2 set k =∑L

l=1 ξl ;3 for each l such that ξl = 1

generate random sample ul ∼ N (µl ,Σl) end

4 Random arrangement ⇒ θk

ResultsConclusion

2 set k =∑L

Example: