ThinBlinDe: Blind deconvolution for confocal microscopy

8/8/2019 ThinBlinDe: Blind deconvolution for confocal microscopy

1/105

ThinBlinDe

P. Panka-jakshan, L.

Blanc-Fraud, J.

Zerubia

ThinBlinDe: Blind deconvolution for confocalmicroscopy

Praveen Pankajakshan,

Laure Blanc-Fraud, Josiane Zerubia.

Ariana joint research group,INRIA/CNRS/UNS,

Sophia Antipolis, France.

This research work was partially funded byP2R Franco-Israeli project and ANR DIAMOND project. Access to

the Applied optics paper is here.

October 2010

P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 1 of 42
http://www-sop.inria.fr/ariana/Projets/P2R/http://www-syscom.univ-mlv.fr/ANRDIAMOND/http://www.opticsinfobase.org/abstract.cfm?uri=ao-48-22-4437http://www.opticsinfobase.org/abstract.cfm?uri=ao-48-22-4437http://www-syscom.univ-mlv.fr/ANRDIAMOND/http://www-sop.inria.fr/ariana/Projets/P2R/


2/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Road map of presentation

1 IntroductionOptical sectioning fluorescence microscopyLimitations of imaging

2

ThinBlinDe software for deconvolutionBreaking the limit-computationallyBlind deconvolution

3 Results

Phantom dataMicrospheresPlant samples



3/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Holy grail of biological imaging

The holy grail for most cell

biologists

E. Betzig et al. Imaging intracellular fluorescent proteins at nanometer resolution, Sc. Exp., Aug. 2006.



4/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



biologists

dynamically image living cellsnoninvasively,




5/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



biologists


to see at the level of anindividual protein molecule,and




6/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



biologists


to see at the level of anindividual protein molecule,and

to see how these molecules

interact in a cell.




7/105


8/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Confocal laser scanning microscope

Figure 1: Schematic of a confocal laserscanning microscope (Minsky (1988)).

A- Laser

B- Beam expander

C- Dichroic mirror

D- ObjectiveE- In-focus plane

F- Confocal pinhole

G- Photomultiplier tube

(PMT)

Minsky (1988) Memoir on Inventing the Confocal Scanning Microscope. Scanning (10): 128138.P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 4 of 42


9/105

ThinBlinDe

P. Panka-jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

3D imaging by optical sectioning



10/105


11/105

ThinBlinDe


Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Point-spread function

Figure 2: Experimental PSFdetermination by imaging point sources.

Figure 3: 2D Airy disk function.



12/105

ThinBlinDe


Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Illustration: Diffraction effect

Figure 4: Single green fluorescence protein (GFP) if imaged using a widefieldmicroscope. (NA = 1.4, ex = 500nm)



13/105

ThinBlinDe


Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Illustration: Effect of circular pinhole

Airy Unit (AU); 1AU= 1.22ex/NA



14/105


15/105

ThinBlinDe


Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


1AU 3AU




16/105

ThinBlinDe


Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


1AU 3AU Fully open

Figure 5: Effect of changing confocal pinhole size on the out-of-focus fluorescence,contrast and SNR.




17/105

ThinBlinDe


Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

"The triangle of trade-off"


Ill S h l b


18/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Illustration: Spherical aberration

Figure 6: Schematic of spherical aberration. i is the azimuthal angle in the lensimmersion medium, s is the angle in the specimen and d (NFP) is the depthunder the cover slip and into the specimen. The refractive index of the cover slip is

assumed to be either equal to the index of the lens ni or the specimen ns.P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 10 of 42


19/105

Th l i


20/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

The eternal question

Figure 8: How to get the best of your microscope?


P bl t t t


21/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement

Images from fluorescent optical sectioning microscopes (OSM)

are affected by


P bl t t t


22/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement


are affected by blurring, due to diffraction limits and aberrations,


Problem statement


23/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement



when pinhole size is 1 airy units (AU), 30% of photonscollected are from out-of-focus sections.


Problem statement


24/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement




shot noise, occurring as detectable statistical fluctuations,


Problem statement


25/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement




shot noise, occurring as detectable statistical fluctuations, spherical aberrations due to mismatch in medium,


Problem statement


26/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement





exact point-spread function (PSF) is unknown and varieswith experiment.


Problem statement


27/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Problem statement





exact point-spread function (PSF) is unknown and varieswith experiment.

Often only a single observation of the object is given (to avoidphotobleaching) for restoration!


Breaking the diffraction barrier


28/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


Ernst Karl Abbe [1840-1905]

S. Hell et al. (1994) Breaking the diffraction resolution limit by stimulated emission:stimulated-emission-depletion fluorescence microscopy. OL 19 (11): 780782.

M. J. Rust et al. (2006) Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy

(STORM). NM 3 (10): 793

796.P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 14 of 42



29/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Abbes diffraction limitapproximation:

d=ex

2nosinmax



(STORM). NM 3 (10): 793

796.P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 14 of 42



30/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSMLimitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples




d=ex

2nosinmax

Recent fluorescence lightmicroscopy techniques are aimed atsurpassing the Abbe resolutionlimit (Hell, et al.(1994), Betzig, etal. (2006), Rust, et al.(2006)).



(STORM). NM3

(10

):793796

.P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 14 of 42



31/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples




d=ex

2nosinmax

Recent fluorescence lightmicroscopy techniques are aimed atsurpassing the Abbe resolutionlimit (Hell, et al.(1994), Betzig, etal. (2006), Rust, et al.(2006)).

Resulting images are quite similarand differences in the operationaldetails can make these techniquesmore or less suitable for specific

types of biological studies.



(STORM). NM3

(10

):793796

.P. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 14 of 42

Imaging Model


32/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Imaging Model

Image formation statistics is Poissonian, and

i(x) = P((ho+ b)(x)),x s

where denotes 3D deconvolution,

O(s = {o = oxyz : s N3

R}),


Imaging Model


33/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

g g


i(x) = P((ho+ b)(x)),x s



R}), h : s R models the PSF,


Imaging Model


34/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

g g


i(x) = P((ho+ b)(x)),x s




b is the low frequency background fluorescence signal,


Imaging Model


35/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

g g


i(x) = P((ho+ b)(x)),x s




b is the low frequency background fluorescence signal,

1/ is the photon conversion factor, and i(x) is the

observed photon at the detector.


Trends: Deconvolution and CLSM


36/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Jan 2004 Jun 2006 Jan 200910

20

30

40

50

60

70

80

90

100

Time

Normalizedn

umberofsearches

Confocal Microscopy

Deconvolution

Figure 9: Annual search trends forCLSM and deconvolution between2004-2009 (Source: Google).

0 20 40 60 80 100

10

20

30

40

50

60

70

80

90

Relative number of searches for "confocal microscopy"

Relativenumberofse

archesfor"deconvolution"

Figure 10: Scatter plot between thekeyword hits deconvolution andconfocal microscopy (Source:Google). Dotted line is the line of leastsquare (LS) fit.


Trends: scientific interest


37/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

1970 1975 1980 1985 1990 1995 2000 20050

1

2

3

4

5

6

7

8

9

10

Year

NormalizedReferenceVolume

Deconvolution

Blind Deconvolution

Figure 11: Recent scientific trends on deconvolution and blind deconvolution (Datasource: ISI web of knowledge).


Breaking the limitations-ThinBlinDe software


38/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 12: Observed volume section of a convallaria and restoration using the

ThinBlinDe software. cINRA and Ariana-INRIA/CNRS/UNSP. Pankajakshan, L. Blanc-Fraud, J. Zerubia November 5, 2010 page 18 of 42

Deconvolution as Bayesian inference


39/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

From the Bayes theorem, the posterior probability is

Pr(o|i) Pr(i|o)Pr(o)




40/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

MicrospheresPlant samples



Pr(i|o) is the likelihood, Pr(o) is the belief of the object.




41/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data





The equivalent energy function is

J(o|i)




42/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data






J(o|i) = log(Pr(o|i))




43/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data






J(o|i) = log(Pr(o|i)) = Jobs(i|o) +Jreg(o)




44/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data







Maximum a posteriori (MAP) estimate is obtained by




45/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data








o(x) = arg maxo(x)0

Pr(o|i)




46/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data








o(x) = arg maxo(x)0

Pr(o|i) = arg mino(x)0

( log(Pr(o|i)))


State of the art: deconvolution


47/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Assumption Method References

No Noise Nearest neighbors [Agard 84],Inverse filter [Erhardt et al. 85]

Additive Gaussian Reg. lin. least square [Preza et al. 92],white noise Wiener filter [Tommasi et al. 93],

JVC [Agard 84,

Carrington et al. 90],APEX [Carasso 99],

MAP estimate [Levin et al. 09]Poisson noise ML estimate [Holmes 88],

MAP estimate [Verveer et al. 99,

Dey et al. 06,Pankajakshan et al. 08]


Maximum likelihood object estimation


48/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Likelihood of the observation, i(x), knowing the specimen

o(x) is given as




49/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


o(x) is given as

Pr(i|o) (ho+ b)(x)i(x) exp((ho+ b)(x))

i(x)!




50/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


o(x

) is given as


i(x)!

Find o(x) such that

o(x) = arg maxo(x)0

Pr(i|o)




51/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


o(x

) is given as


i(x)!

Find o(x) such that

o(x) = arg maxo(x)0

Pr(i|o)

Approach: Maximum likelihood expectation maximization(MLEM) algorithm [Richardson 72, Lucy 74, Dempster 77,Celeux 85].




52/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-

Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


o(x

) is given as


i(x)!

Find o(x) such that

o(x) = arg maxo(x)0

Pr(i|o)

Approach: Maximum likelihood expectation maximization(MLEM) algorithm [Richardson 72, Lucy 74, Dempster 77,Celeux 85].

Assumption: h(x) is available!


Prior as statistical information


53/105

ThinBlinDe

P. Panka-

jakshan, L.Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Gibbsian distribution with total variation (TV) function is used

as prior on the object [Demoment 1989]

Figure 13: Markov random fieldover a six member neighborhood

x

for a voxel sitex

s.




54/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Figure 13: Markov random fieldover a six member neighborhood

x

for a voxel sitex

s.

Pr(o) = Z1o exp(ox

s

|o(x)|),




55/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Figure 13: Markov random fieldover a six member neighborhoodx

for a voxel sitex

s.

Pr(o) = Z1o exp(ox

s

|o(x)|),

Zo =oO(s)

exp(oxs

|o(x)|) is

the partition function,




56/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Figure 13: Markov random fieldover a six member neighborhoodx

for a voxel sitex

s.

Pr(o) = Z1o exp(ox

s

|o(x)|),

Zo =oO(s)

exp(oxs

|o(x)|) is

the partition function, with o 0.



57/105

Maximum a posteriori object estimate


58/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Combining the likelihood and the prior terms

Pr(o,h|i) Pr(i|o,h)Pr(o)

this posterior probability can be written as

Pr(o,h|i) xs

((ho+ b)(x))i(x) exp((ho)(x))

i(x)!

exp(oxs

|o(x)|)

o

exp(oxs

|o(x)|)


Gradient vector field


59/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples X

Y

50

100

150

200

250

Lateral direction




60/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples X

Y

50

100

150

200

250

X

Z

50

100

150

200

250

Lateral direction Axial direction




61/105

ThinBlinDe

P. Panka-


Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples X

Y

50

100

150

200

250

X

Z

50

100

150

200

250

Lateral direction Axial direction

Figure 14: The gradient vector field is overlapped over a synthetic object. Thefield flow is more prominent along the borders and less along the homogeneousinteriors. The effect of the noise is negligible. cAriana-INRIA/CNRS/UNS.



62/105

Incoherent scalar PSF model


63/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples




64/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

For the case when the acquisition parameters of the

experiments are exactly known, deconvolution can beachieved by using theoretically calculated PSFs.




65/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Computation of the incoherent PSF can be reduced to 2Nz2D Fourier transform of the pupil function

hTh(x;ex,em) = C|hA(x;ex)| |AR(x,y)hA(x,y,z;em)|




66/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Computation of the incoherent PSF can be reduced to 2Nz2D Fourier transform of the pupil function

hTh(x;ex,em) = C|hA(x;ex)| |AR(x,y)hA(x,y,z;em)|

IfP(kx,ky,z) is the 2D complex pupil function and is thewavelength, the amplitude PSF is

hA(x,y,z;) =

kx

ky

P(kx,ky,z)exp(j(kxx+ kyy))dkydkx


Numerically computed PSF


67/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 16: Numerically calculated PSF for a C-Apochromat water immersionobjective. NA 1.2, 63 magnification. cAriana-INRIA/CNRS/UNS.


State of the art: Blind deconvolution


68/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Method References

A priori PSF [Boutet de Monvel 2001],identification

Marginalization [Jalobeanu 2007,

Levin et al. 2009],Joint maximum likelihood [Holmes 1992,

Michailovich & Adam 2007],

Parametric blind deconvolution [Markham& Conchello 1999].


Thi Bli D

Blind deconvolution by alternate minimization


69/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

When the imaging parameters are not known, it isnecessary to estimate o(x) and h(x) from i(x)


Thi Bli D



70/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


Minimizing the cost function w.r.t o(x)

o(x) = arg mino(x)0

J(o(x)|o,h(x))

= arg mino(x)0Jobs(i(x

)|o(x

),h(x

)) +Jreg(o(x

)|o)


ThinBlinDe



71/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


Minimizing the cost function w.r.t o(x)

o(x) = arg mino(x)0

J(o(x)|o,h(x))

= arg mino(x)0Jobs(i(x

)|o(x

),h(x

)) +Jreg(o(x

)|o)

Minimizing the cost function w.r.t h(x)

h(x) = arg minh(x)0

J(h(x)|o,o)

= arg minh(x)0

Jobs(i(x)|o,h(x)) +Jreg(h(x))


ThinBlinDe

Blind deconvolution by PSF parametrization


72/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


ThinBlinDe



73/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

The estimation of the complete PSF from the observationis a difficult problem as the number of unknowns are large.


ThinBlinDe



74/105

ThinBlinDe


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


Since the CLSM PSF is theoretically the Bessel functionraised to the fourth power, an approximation in the spatialdomain is a 3D Gaussian approximation,


ThinBlinDe


Th f h l PSF f h b


75/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples



Diffraction-limited PSF in the LS sense (up to 3AU pinhole

diameter)

h(x;h) = (2) 3

2 ||1

2 exp(1

2(x)T1(x))


ThinBlinDe


Th i i f h l PSF f h b i


76/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples




diameter)

h(x;h) = (2) 3

2 ||1

2 exp(1

2(x)T1(x))

BD is reduced to estimation of spatial parameters,


ThinBlinDe


Th ti ti f th l t PSF f th b ti


77/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples




diameter)

h(x;h) = (2) 3

2 ||1

2 exp(1

2(x)T1(x))

BD is reduced to estimation of spatial parameters,

advantages: few parameters and no DFT necessary.


ThinBlinDe

Why 3D Gaussian?


78/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Properties of the PSF



79/105

ThinBlinDe

Why 3D Gaussian?


80/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Properties of the PSF positivity:

h(x) 0, x s

circular symmetry:

h(x,y,z) = h(x,y,z)


ThinBlinDe

Why 3D Gaussian?


81/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


positivity:

h(x) 0, x s

circular symmetry:

h(x,y,z) = h(x,y,z)

normalization:

xs

h(x) = 1


ThinBlinDe

Why 3D Gaussian?


82/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


positivity:

h(x) 0, x s

circular symmetry:

h(x,y,z) = h(x,y,z)

normalization:

xs

h(x) = 1

CLSM PSF is theoretically the Besselfunction raised to the fourth power.


ThinBlinDe

Why 3D Gaussian?


83/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


positivity:

h(x) 0, x s

circular symmetry:

h(x,y,z) = h(x,y,z)

normalization:

xs

h(x) = 1

CLSM PSF is theoretically the Besselfunction raised to the fourth power.

Figure 17: The Besselfunction raised to the fourthpower loses its side lobes.

cAriana-INRIA/CNRS/UNS.


ThinBlinDe



84/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


ThinBlinDe


In the presence of aberrations, spatial approximation leads


85/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

In the presence of aberrations, spatial approximation leadsto a large number of parameters,


ThinBlinDe




86/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

t e p ese ce o abe at o s, spat a app o at o eadsto a large number of parameters,

a way of handling it is estimating the parameters of thepupil function h = d,ni,ns

Pa(;x) = Pd(;x)exp(j(d,ni,ns))

P Pankajakshan L Blanc-Fraud J Zerubia November 5 2010 page 32 of 42

ThinBlinDe




87/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

p , p ppto a large number of parameters,

a way of handling it is estimating the parameters of thepupil function h = d,ni,ns

Pa(;x) = Pd(;x)exp(j(d,ni,ns))

regularization of the PSF is achieved by using a functionalform of phase from geometrical optics

(d,ni,ns) = d(nicosins coss) dns secs


ThinBlinDe

PSF estimation


88/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples


ThinBlinDe

PSF estimation

Deconvolution requires the PSF h(x) or h(x, h)


89/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

( ) ( )

h,MAP = argmaxh>0

Pr(i(x)|o(x),h)Pr(h)


ThinBlinDe

PSF estimation



90/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

h,MAP = argmaxh>0


Pr(h) is the parameter prior. We assume the parametersto be uniformly distributed in a set: [h,LB,h,UB],


ThinBlinDe

P P k

PSF estimation



91/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

h,MAP = argmaxh>0



and the cost function is

J(o,h(h)) =x

i(x)log((h(h)o+ b)(x))

x

(h(h)o+ b)(x)

P Pankajakshan L Blanc Fraud J Zerubia November 5 2010 page 33 of 42

ThinBlinDe

P P k

PSF estimation



92/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

h,MAP = argmaxh>0



and the cost function is

J(o,h(h)) =x

i(x)log((h(h)o+ b)(x))

x

(h(h)o+ b)(x)

The cost function, J(o,h(h)), could be minimized byusing a gradient-descent type algorithm.


ThinBlinDe

P Panka

Simulation results


93/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Phantom object Observation = 100,xy = 25nm z = 50nm


ThinBlinDe

P Panka

Simulation results


94/105


Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Sans regularization (Naive MLEM) Proposed approach


ThinBlinDe

P Panka-

Microsphere deconvolution


95/105

P. Pankajakshan, L.

Blanc-Fraud, J.

Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 18: A 15m microsphere has a thin layer of fluorescence of thickness500700nm.


ThinBlinDe

P. Panka-

Microsphere deconvolution


96/105

P. Pankajakshan, L.

Blanc-Fraud, J.Zerubia

Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Observed section BD restored (535.58 nm)

Figure 19: Measured thickness of a microsphere, after using our approach, was535.58nm, and was 260nm with the naive MLEM algorithm.cAriana-INRIA/CNRS/UNS.


ThinBlinDe

P. Panka-

Deconvolution on plant samples


97/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 20: Observed section of arabidopsis thaliana in water and observed with aLSM 510 microscope and pinhole 2AU. Lateral sampling is 285.64nm and axialsampling is 845.62nm. cINRA.


ThinBlinDe

P. Panka-



98/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 21: Restoration sans regularization (Naive MLEM).cAriana-INRIA/CNRS/UNS.

P Pankajakshan L Blanc Fra d J Zer bia November 5 2010 page 39 of 42

ThinBlinDe

P. Panka-



99/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 22: Restoration using the proposed approach after 10 iterations.cAriana-INRIA/CNRS/UNS.

P P k j k h L Bl F d J Z bi N b 5 2010 40 f 42

ThinBlinDe

P. Panka-



100/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

Figure 23: Comparison of observed and restored lateral section of convallariamajalis. Lateral sampling is 53.6nm and axial sampling is 129.4nm. The samplewas observed with a Zeiss LSM 510TMmicroscope, and 1.3NA oil immersionobjective. cINRA, Ariana-INRIA/CNRS/UNS.

P P k j k h L Bl F d J Z bi N b 5 2010 41 f 42


101/105

ThinBlinDe

P. Panka-j k h L

Conclusions and discussions

Developed Bayesian framework for blind deconvolution forthin specimens


102/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

thin specimens,


ThinBlinDe

P. Panka-jakshan L




103/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

thin specimens,

experiments on simulated data, microsphere images andreal data show promising results,


ThinBlinDe

P. Panka-jakshan L




104/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

thin specimens,


PSF model chosen initially is a 3D separable Gaussianfunction for thin specimens,


ThinBlinDe

P. Panka-jakshan L


Developed Bayesian framework for blind deconvolution forthin specimens,


105/105

jakshan, L.


Road map

Introduction

OSM

Limitations

ThinBlinDe

Break the limit

Blind

deconvolution

Results

Phantom data

Microspheres

Plant samples

thin specimens,


PSF model chosen initially is a 3D separable Gaussianfunction for thin specimens,

Total variation regularization functional sometimes leads toloss in contrast in restoration but there are methods toovercome this.


Documents

ThinBlinDe: Blind deconvolution for confocal microscopy