Statistical Physics of Inference - Bienvenue · information theory machine learning Tuesday, ... Use gained insight to develop better algorithms. Statistical Physics of Inference

Lenka Zdeborová (IPhT, CNRS & CEA Saclay)

Statistical Physics of Inference

Tuesday, December 9, 14

My research

Physics

signal processing

optimization

information theory

machine learning



An example of a success story:

Compressed sensingwith F. Krzakala, M. Mezard, F. Sausset, Y. Sun,


From 106 wavelet coefficients, keep 25.000

Why do we record a huge amount of data, and then keep only the important bits?

Couldn’t we record only the relevant information directly?

Most signal of interest are sparse in an appropriated basis⇒Exploited for data compression (JPEG2000).


Possible applications- Rapid Magnetic Resonance Imaging- Rapid Computed Tomography- Image acquisition (single-pixel camera)- Detection of rare genes with small number of tests- Enhance resolution in very noisy devices- .... many more .....

Left: image acquired with acceleration by a factor 2.5

Lustig, Donoho, Pauly ’07


Setting of the problemDesign the matrix F such that sparse signal x can be reconstructed efficiently from measurements y.

= F

y

x

yµ =NX

i=1

Fµixi MN

x is sparse, i.e. only elements are non-zero. The linear problem has many solutions, only is one sparse.

⇢N



↵=

M N

⇢ =K

N

N-component signal, K of them nonzero, M measurements,

N ! 1

impossible

easy with linear programing

?

State-of-the-art in 2011


Statistical physics formulation

P (~x|~y) = 1

Z

e

PNi=1 log [(1�⇢)�(xi)+⇢�(xi)]

e

PMµ=1

12�µ (yµ�

PNi=1 Fµixi)2

local magnetic field

spin-variablesxi

N-body interactions

Z: partition function

Optimal signal reconstruction = equilibrium solution.Methods from physics of mean field spin glasses: replica method, cavity method, message passing. 40 years of work in physics (Mezard, Parisi, Zecchina, Nishimori ....)

Boltzmann distribution


1st result of statistical physics analysis:

αL1αBEPα=ρ

0.4 0.5 0.6 0.7 0.8 0.90

0.05

0.1

0.15

0.2

α

Mea

n s

quar

e er

ror

L1

BEP

1 10-5 0.0001 0.001 0.01 0.1

-1

-0.5

0

0.5

1

Mean square error

tanh[4

!(E

)]

α = 0.8

α = 0.6

α = 0.5

α = 0.3

αL1αrBPα=ρ

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

ρ

α

αL1(ρ)αrBP(ρ)S-BEP

α = ρ

0.4 0.5 0.6 0.7 0.8 0.9

30

100

300

1000

3000

10000

α

Num

ber

of iter

atio

ns

rBP

Seeded BEP - L=10

Seeded BEP - L=40

fraction of non-zeros

mea

sure

men

ts p

er s

igna

l ele

men

t

‣ Better performance for a class of signals.‣ Algorithmic barrier = spinodal of a 1st order phase

transition. Algorithms blocked in a metastable state.


Can physics insight bring us even further?


Heating pad or hand warmer:

sodium acetate melts at 58 C

Thanks to: UCGP 2008, Kyoto, Japan

Nucleation for optimality!


Implementing nucleation in compressed sensing:

0

BBBB@

1

CCCCA= ⇥

y F s

: unit coupling

: no coupling (null elements)

: coupling J1: coupling J2

J1J1

J1J1

J1J1

J1

J2J2

J2J2

J2J2

11

11

11

1

1 J2

0

0

0

BBBB@

1

CCCCA

0

BBBBBBBBBBBB@

1

CCCCCCCCCCCCA


Thanks to statistical physics analysis compressed sensing is today computationally tractable down the information theoretic limit! Krzakala, Mezard, Sausset, Sun,

Zdeborova, Phys. Rev. X 2012. Proof: Donoho, Javanmard, Montanari, ISIT 2012.

αL1αBEPα=ρ

0.4 0.5 0.6 0.7 0.8 0.90

0.05

0.1

0.15

0.2

α

Mea

n s

quar

e er

ror

L1

BEP

1 10-5 0.0001 0.001 0.01 0.1

-1

-0.5

0

0.5

1

Mean square error

tanh[4

!(E

)]

α = 0.8

α = 0.6

α = 0.5

α = 0.3

αL1αrBPα=ρ

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

ρ

α

αL1(ρ)αrBP(ρ)S-BEP

α = ρ

0.4 0.5 0.6 0.7 0.8 0.9

30

100

300

1000

3000

10000

α

Num

ber

of iter

atio

ns

rBP

Seeded BEP - L=10

Seeded BEP - L=40


L1

BEP

S-BEP

α = 0.5 α = 0.4 α = 0.3 α = 0.2 α = 0.1

α = ρ ! 0.15

Shepp-Logan phantom, sparse in the Haar-wavelet representation


(1) Pick an important problem. (2) Compute phase transitions in an amenable setting. (3) Use gained insight to develop better algorithms.

Statistical Physics of Inference

Our cook-book:

Our main contribution: Algorithms for practitioners. Conjectures for mathematicians.


Read more: F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Statistical physics-based reconstruction in compressed sensing, Phys. Rev. X (2012).

F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices, J. Stat. Mech. (2012).

Another example of this approach:F. Krzakala, E. Mossel, C. Moore, J. Neeman, A.Sly, LZ, P. Zhang, Spectral Redemption in Clustering Sparse Networks, PNAS (2013).

A. Decelle, F. Krzakala, C. Moore, LZ, Inference and phase transitions in detection of modules in sparse networks, Phys. Rev.lett. (2011)


Thank you for your attention!

Read more: F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Statistical physics-based reconstruction in compressed sensing, Phys. Rev. X (2012).

F. Krzakala, M. Mézard, F. Sausset, Y. Sun, LZ, Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices, J. Stat. Mech. (2012).

Another example of this approach:F. Krzakala, E. Mossel, C. Moore, J. Neeman, A.Sly, LZ, P. Zhang, Spectral Redemption in Clustering Sparse Networks, PNAS (2013).

A. Decelle, F. Krzakala, C. Moore, LZ, Inference and phase transitions in detection of modules in sparse networks, Phys. Rev.lett. (2011)


Documents

Statistical Physics of Inference - Bienvenue · information theory machine learning Tuesday, ... Use gained insight to develop better algorithms. Statistical Physics of Inference