Upload
dohanh
View
217
Download
0
Embed Size (px)
Citation preview
Image restoration by deconvolution
17/12/2014
(part) Slides courtesy: Sébastien Tosi (IRB Barcelona)
A few concepts related to the topic
Convolution
Deconvolution
Point Spread Function
Noise
Fourier Transform
Spatial resolution
Pixel size
Rayleigh Criterion
Airy disk
Numerial Aperture
Refractive Index
Wavelength
Image formation (in Fluorescence microscopy)
Image from [9]
In fluorescence microscopy (in all its modes including widefield, confocal,
and multi-photon): the imaging process can be mathematically described
by a convolution
Imaging, Convonlution, Deconvolution
Convolution consists of replacing each point in the original object with its blurred image in all dimensions and summing together overlapping contributions from adjacent points to generate the resulting three-dimensional image
All microscopy techniques that include directly or indirectly a convolution in their image formation processes can benefit from image deconvolution.
2D Convolution
The convolution can also be computed by “stamping”
the kernel on each pixel of the image: the kernel is
scaled (multiplied) by the intensity of the central pixel
and accumulated (summed) in the output image.
11111
12221
12321
12221
11111
Filter kernel
� �
Original imageFiltered image Original imageFiltered image
3D Convolution
Convolution extends to more than two dimensions: in 3D the kernel is a small volume (stack)
and the sum is triple (inside a volume around each voxel).
=
A 3D kernel is a stack holdingthe filter coefficients
=⊗
⊗
Fourier transform (FT)
If we take the FT of the
equation, the is replaced by
multiplication, thus image
restoration might be achievable
by:
dividing the FT of the image by
the FT of the kernel and then
taking the inverse Fourier
transform.
⊗⊗
X
Spatial Domain
FourierDomain
=
=
Image from Sébastien Tosi (IRB
Barcelona)
Point Spread Function (PSF)
More slides from Math-Clinic BioImage Analysis website: http://goo.gl/u52WmC
An image resulting from a single small spherical fluorescent bead (smaller than
the optical resolution, thus forms effective a point source of light)
A record of how much the microscope has spread or blurred a single point
Simplified diagram to visualize how a light-emitting point would be imaged using a widefield microscope, Image from [9] Widefield PSF, Image from [3]
Spatial resolution: distance by which two
objects must be separated to be
distinguished, i.e. the radius of the smallest
point source in the image (defined as the
first minimum of the Airy disk)
the Rayleigh criterion:
2
2axial
nr
NA
λ=0.61radialr
NA
λ= λ : fluorophore emission wavelength
NA : objective numerical aperture
n : refractive index of the objective
lens immersion medium
NA can never exceed n, which itself
has fixed values (e.g. 1.0 for air, 1.33
for water, or 1.52 for oil)
NA = n sinθ
Notes:
1. Rayleigh criterion has not taken into account the effects
of: brightness, pixel size, noise
2. High NAs are possible when the immersion refractive
index is high
PSF (in the focal plane)
Widefield PSFs obtained by imaging 100-nm fluorescent beads (excitation
520nm; emission 617nm), Image from [2]
Image from [9]
Experimental PSF
(measured PSF) Theoretical PSF
Theoretical PSFQuotes from [3]:
Most methods are based on the work of Born and Wolf (1980).
A good description for a confocal PSF is given by van der Voort and Brakenhoff (1990), in
which the PSF is calculated from: the NA of the objective, the illuminating and emitted
wavelengths, and the refractive index of the immersion medium in either (simpler) paraxial
forms or with WF integrals.
Theoretical PSF gives an indication of the best possible resolution for a given objective but
these limits are not achievable.
In our experience, real PSFs are typically >20% bigger than calculated versions.
image from [9]
What else does (measured) PSF tells us?
Asymmetry:
radial (x-y): commonly misalignment of optical
components about the z-axis, either as tilt or decentration
along the optical axis (z-axis): commonly due to spherical
aberration, which may result from refractive index mis-
matches between the objective, immersion medium, and
sample or tube length/coverslip thickness errors.
Image from [3]
Notes:
1. The immersion refractive index should match the
refractive index of the medium surrounding the
sample, to avoid spherical aberration
2. Item 1 is often strongly preferable to using the
highest NA objective available, as it is usually
better to have a larger PSF than a highly irregular
one.
Deconvolution Principle
The deconvolution filter F should “undo” the effect of the microscope PSF H by
processing the sampled image R, ideally D = S.
Assuming H known, F linear (convolution) and no noise (N = 0) leads to:
or
or
In this context “1” is a black image holding a single point in its center
In practice the noise N and the error on the estimation of H (measurement or model) are
impeding a perfect deconvolution and we can at best hope for an approximate solution…
Object (sample)
Microscope PSF
AWGN source
Digital filterDeconvolved image
D H S H⊗ = ⊗( )R F H R⊗ ⊗ =
1F H⊗ =�
( )R F H R⊗ ⊗ =
More/Details
http://blogs.qub.ac.uk/ccbg/fluorescenc
e-image-analysis-intro (Part III)
RECOMMEND! It is a pleasant reading!
Ref [2]
More on reference slide…
When to do deconvolution?
• Wide field microscopy (WFM)
Less affected by out-of-focus light:
• Confocal laser scanning microscopy (CLSM)
• Two-photon excitation microscopy (TPEM)
• Selective plane illumination microscopy (SPIM)
Super-resolution fluorescence microscopy
• Stimulated emission depletion microscopy (STED)
• …
All microscopy techniques that
include directly or indirectly a
convolution in their image
formation processes can benefit
from image deconvolution.
Any 2D or 3D image obtained
from almost any fluorescence
microscope is expected to be
deconvolved before being analysed.
Why to do deconvolution?
• Attenuation of the out of focus light - increase contrast
• Reduce noise
• Increase of the spatial resolution
Ref [2]
Deblurring - subtractive
Nearest neighbour
No neighbours
Linear inverse filter
Regularized Inverse filter
Object smoothness
e.g. Wiener filter
Constrained Iterative
Nonnegative
Blind deconvolution
Not for
quantitative
intensity
measurements
Do not count for
noise
Quantitative
No PSF as input
May not be
absolutely
quantitative
Linear Deconvolution: Inverse Filter Deconvolution
A very simple modelfor the PSF H
(Gaussian std = 1 pixel)
1
4,1·10-8
1
2,4·107
H power spectrum (log display)overlaid with raw values
H-1 power spectrum (log display)overlaid with raw values
1( , ) ( , )F u v H u v−=
As convolution in the spatial domain can be performed as a multiplication in the frequency
domain, inverse filtering can be performed as a division in the frequency domain!
But in practice…
Noise enhancement ruins our efforts!
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
350
400
450
Inverse Filter Deconvolution
Original image Blurred image
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
350
400
450
H +N
H-1
Original image S
H is a Gaussian with std = 2 pixels
50 100 150 200 250 300 350 400 450
50
100
150
200
250
300
350
400
450 Noise std = 10-4 Noise std = 10-12
S after convolution by H
No noise
Second try: Regularized Inverse
A very simple modelfor the PSF H
(Gaussian std = 1 pixel)
1
4,1·10-8
H power spectrum (log display)overlaid with raw values
(H-1)reg (1% clipping)power spectrum (log display)
overlaid with raw values
1
2,4·107
1
100
1 1
1
( , ), ( , )( , )
, ( , )
H u v H u v tF u v
t H u v t
− −
−
≤= >
Regularized Inverse Filter Deconvolution
Original image Blurred image
H +N
(H-1)trunc
Original image S S after convolution by H
Restoration of Blurred, Noisy Image Using regularized inverseNoise std = 10-4
Third Try: Wiener Filter The Golden Linear Deconvolution Trade-off
2*
2 2 2
( , ) ( , )( , )
( , ) · ( , ) ( , )
H u v S u vF u v
H u v S u v N u v=
+
( )estS F S H N= ⊗ ⊗ +
.estE S S= −Minimizing the expectation of ||E|| over all possible noise realizations assuming a
white Gaussian noise:
Coming back to:
Bands free of noise: |N(u,v)| = 0 ���� F(u,v) = H(u,v)-1 (inverse filter)
Strong noise bands: |N(u,v)|���� ∞ ���� F(u,v) ���� 0 (cut-off)
Intermediate bands: best trade-off
Wiener filter
Wiener filter attenuates frequencies dependent on their signal-to-noise ratio.
Wiener Deconvolution
Restoration of Blurred, Noisy Image Using Wiener filter for known noise varianceRestoration of Blurred, Noisy Image Using regularized inverseRegularized inverse filter resultNoise std = 10-4
Wiener filter resultNoise std = 10-4
Non-Linear Deconvolution
The best deconvolution algorithms for 3D microscopy are typically non-linear.
Principle of Maximum A Priori algorithms (MAP):
The second equality comes from Bayes theorem.
In the optimization S is usually constrained to be positive and somehow spatially smooth (TV
regularization term) � Pr(S).
The statistical distribution of the noise has to be known to derive the maximum likelihood
term Pr(R|S) � the algorithm is tuned to a particular noise (e.g. Poisson or Gaussian noise).
There is usually no known analytical solution to the problem, the algorithms proceeds by
iterations (candidate Si at iteration i) to refine the estimate of the data at each iteration.
The Richardson-Lucy algorithm is among the most well known MAP deconvolution algorithm.
Some algorithms also simultaneously estimate the PSF from the sampled image (blind
deconvolution).
( ) Pr( | )Pr( )arg max Pr( | ) arg max .
Pr( )MAP
S S
R S SS S R
R= =
Quantification with deconvolution
• Ideally: relocate signal to the point of origin in 3D, thus conserve the sum of fluorescence signal. It improves quantification!
• In practice: different algorithms have more or less compromises
• Quantitative intensity measurements, e.g. intensity ratio: controls, also report on un-deconvolved data for comparison
• Quantitative positional or structural analysis, e.g. centroid, tracking, volume analysis, (object based) colocalisation, etc: relatively less critical the choice
• For all analysis:
• Deconvolution process comparable between datasets
• Compare with control/un-deconvolved data
• Understand algorithm used and choose most suitable
• Report possible artifacts and confirm it, if possible
Deconvolution tools (not exhaustive!)
Parallel iterative deconvolution (fiji.sc/Parallel_Iterative_Deconvolution): 4 deconvolution algorithms
Parallel spectral deconvolution (fiji.sc/Parallel_Spectral_Deconvolution)
Not iterative, no constraint e.g. nonnegativity
Iterative Deconvolve 3D (fiji.sc/Iterative_Deconvolve_3D) :
non-negative, iterative, similar to WPL algorithm. The execution is way slower on modern (multicore) computers
but the memory requirement is less stringent
DeconvolutionLab (http://bigwww.epfl.ch/algorithms/deconvolutionlab/ ): different algorithms including a custom
version of the thresholded Landweber algorithm
Commercial software
- SVI Huygens
- MC AutoquantX
- …
Fiji plugins
I could not comment on commercial software, due to access issue.
Original AutoquantX (30IT, bead PSF)
Huygens (50IT, bead distilled PSF)
PID (WPL, Wiener Gamma 0.1, 50IT, bead PSF)
Original PID (WPL, 50IT, true PSF) AutoquantX (30IT, true PSF) Huygens (50IT, distilled true PSF)
Original AutoquantX (30IT, bead PSF)
Huygens (50IT, bead distilled PSF)
PID (WPL, 50IT, bead PSF)
Courtesy of Sébastien Tosi (IRB
Barcelona)
+ Microscope specific
PSF
+ depth-varying PSF
+ supports spinning
disk M.
+ Visually appealing
results
- Expensive & Closed
source
+ Free & Open source
& full control
+ Reasonably fast
+ Support for
spatially-variant
PSF (un-tested)
- High memory usage
- Visually less
crispy
+ Fast convergence
+ Robust algorithms
+ Very simple to use
+ Visually appealing
results
+ 2D mode for thin
samples
- Expensive & Closed
source
Examples
Theoretical PSF generator
PSF generator:
http://bigwww.epfl.ch/algorithms/psfgenerator/#download
>15 models
Diffraction PSF 3D:
http://fiji.sc/Diffraction_PSF_3D
using Fraunhofer diffraction
Parallel Iterative Deconvolution
The plugin provides 4 deconvolution
algorithms:
- Wiener Filter Preconditioned Landweber
(WPL)
- Modified Residual Norm
Steepest Descent (MRNSD)
nonnegative
- Conjugate Gradient for Least Squares
(CGLS)
- Hybrid Bidiagonalization Regularization
(HyBR)
regularized
Parallel Iterative Deconvolution
attempts to reduce artifacts from features near the boundary of the imaging volume.
stops the iteration if the changes appear to be increasing. Increase the low pass filter size if this problem occurs. fraction of the largest Fourier
coefficent of the PSF
higher value increases
convergence
affects the scaling of the result
DeconvolvedImage 1 Image 3
Objects look brighter -> higher contrast
Better separation between close objects
Other resources/tools:A MatlabMatlabMatlabMatlab software
• http://www.unife.it/prin/software/sgp_deblurring_boundary.zip
Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy,
Scientific Reports 2013.
Other resources/tools:Fiji Squassh – segmentation / colocalization
• deconvolved (subpixel) segmentations in 2D&3D through prior knowledge of PSF
• Intensity within each object is homogeneous
• Bright foreground & dark background
• Noise model: Gaussian (wide field) or Poisson (confocal)
• (more robust) Joint deconvolution-segmentation procedure
Rizk et al., Segmentation and quantification of subcellular structures in fluorescence microscopy images using Squassh, Nature Protocol, 2014.
Summary
• Deconvolution is a computational technique allowing to (partly) compensate for theimage distortion created by an optical system
• Correct deconvolution should improve:
attenuation of the out of focus light
quantitative measurements
the spatial resolution
• Incorrect deconvolution could:
Introduce (more) artifacts -> reduce image quality
• It works best for thin (<50 um), optically transparent, fixed, bright samples.
• Challenging for live microscopy: short exposure (limit motion blur), objective adaptedto medium (limit spherical aberrations).
References (to name a few…)
Good reviews (overviews):
1. Waters, Accuracy and precision in quantitative fluorescence microscopy, JCB 2009
2. Parton et al., Lifting the fog: Image restoration by deconvolution, Cell biology 2006
3. Pawley, Chapter 25: “Image enhancement by deconvolution”, Handbook of biological confocal microscopy, 2006
4. McNally et al., Three-Dimensional Imaging by Deconvolution Microscopy, Methods 1999
Technical articles:
5. Zanella et al., Towards real-time image deconvolution: application to confocal and STED microscopy, Scientific Reports 2013
6. Bertero et al., Image deconvolution, Proc. NATO A.S.I. 2004
7. Thiébaut, Introduction to image reconstruction and inverse problems, Proc. NATO A.S.I. 2002
On the web:
8. Olympus microscopy center (overview): http://www.olympusmicro.com/primer/digitalimaging/deconvolution/deconvolutionhome.html
9. Textbook: http://blogs.qub.ac.uk/ccbg/fluorescence-image-analysis-intro
10. http://fiji.sc/Deconvolution_tips