Imaging the Developing Embryonic Heart: Combining Fast...

Imaging the Developing Embryonic Heart:

Combining Fast Confocal Microscopes

with 4D Reconstruction Techniques

Michael Liebling1, Arian S. Forouhar1,Mory Gharib1, Scott E. Fraser1, Mary E. Dickinson1,2

1California Institute of Technology2Baylor College of Medicine, Houston TX

http://www.its.caltech.edu/∼liebling/

<< < > >> B F R I Page 1

VBA

SV

A

1 day post fertilization 5 days post fertilization

Goals• Determine genetic & epigenetic factors contributing to heartdevelopment• Dynamic 3D imaging in vivo, over various developmental stages• Measure volume changes, blood flow, shear stress, etc.ChallengeEven state-of-the-art laser scanning confocal microscopes are notfast enough for high-speed, real-time 3D image acquisition

Studying the Developing Zebrafish Heart

<< < > >> B F R I Page 2

A

V

Advantages:• zebrafish are vertebrates• reproduce externally and rapidly• relatively transparent• may be genetically engineered toexpress fluorescent markers in spe-cific tissues [e.g. Tg(gata1::GFP)]

48 hpf (hours post fertilization)mb: midbrainot: otocyste: eyeh: heartyolk: yolk mass

Imaging in the Zebrafish Embryo

<< < > >> B F R I Page 3

Fluid forces contribute to heart formation

Acquisition:Widefield, 440 frames/sAnalysis:Digital Particle Image Ve-locimetry

J.R. Hove, R.W. Köster,A.S. Forouhar, G. Acevedo-Bolton, S.E. Fraser, andM. Gharib,“Intracardiac fluid forces are anessential epigenetic factor forembryonic cardiogenesis,”Nature, 421, 172–177 (2003).

Genetic and Epigenetic Factors in Cardiogenesis

<< < > >> B F R I Page 4

J.R. Hove, R.W. Köster, A.S. Forouhar, G. Acevedo-Bolton, S.E. Fraser, and M. Gharib,“Intracardiac fluid forces are an essential epigenetic factor for embryonic cardiogenesis,”Nature, 421, 172–177 (2003).

Fluid forces contribute to heart formation

<< < > >> B F R I Page 5

Process Speed / Frequency

Microtubule growth 300-500nm/s

Cell migration (EC over fibroblasts) 20-50 μm/h

Cellular tra»cking 20-80 μm/min

(protein transp. by vesicle, . . . )

Chromosome dynamics > 0.5 μm/10 sec

Calcium signaling (sparks) 1 nm/s to 50 μm/sec

Blood flow mm/s

Heart beat 1-10 Hz

Beating ciliated cells 3-40 Hz

Speed of some Biological Processes

<< < > >> B F R I Page 6

Migrating germ cells @ 6 frames/h

C. Readhead, M. Liebling18.5 dpc Tg(oct4::GFP) mouse.Grid: 50μm, 6 frames/h, over 6 hours

Blood circulation @ 12 frames/h

Liz Jones, Mary Dickinson8.5dpc Tg(ε-globin::GFP) mouse, 1 frame/5 minutes[Jones et al. (2002) Genesis 34, 228–235]

Cell migration/division Heart motion, blood flow

Speed 10 μm/s mm/s

Required frame rate 1–10 frames/s 100–200 frames/s

Moving at the Speed of Cells

<< < > >> B F R I Page 7

Blood flow visualization@ 2 frames/s (confocal)

Liz Jones, Mary DickinsonTg(ε-globin:GFP) mouse, Zeiss LSM 5 PASCALTransmitted light / 488nm ex/ LP 505 em, 2 frames/s

@ 30 frames/s (video)

Liz Jones, Mary Dickinson

Insu»cient frame-rates introduce artifacts

<< < > >> B F R I Page 8

Fluorescence: Reminder

<< < > >> B F R I Page 9

From: http://www.olympusfluoview.com/theory/index.html

Confocal Microscopy Principle: Reminder

<< < > >> B F R I Page 10

Adapted from: http://www.olympusfluoview.com/theory/confocalscanningsystems.html

Point Scanning

<< < > >> B F R I Page 11

Line array detector

Point detectorConfocalPinhole

Confocal Slit

Point Scanning

Slit Scanning

MicroscopeObjective

MicroscopeObjective

FocusedLine

FocusedPoint

Making a Fast Scanning Confocal Microscope

<< < > >> B F R I Page 12

Zeiss LSM 5 LIVE

<< < > >> B F R I Page 13

Zeiss LSM 5 LIVEFast confocal microscopy

13

Beam path of the LSM 5 LIVE

1. Optical fibers2. Mirror3. Beam combiner4. Beam shaper5. AchroGate beam splitter6. Zoom optics7. Scanning mirror8. Scanning optics9. Objective lens

10. Specimen11. Secondary beamsplitter12. Detector- and pinhole optics13. Confocal pinhole14. Emission filters15. Detector

Parallel illumination/detection120 frames per s at 512×512

2D slice of zebrafish heart (60 fps)

M. Liebling, J. Vermot[38 hpf wildtype zebrafish, BODIPY FL C5-ceramide]

Parallel Acquisition for Increased Speed

<< < > >> B F R I Page 14

4D Imaging via Gated Image Acquisition

<< < > >> B F R I Page 15

Nongated Imaging Requires Synchronization

<< < > >> B F R I Page 16

Method

Adjacent 2D slices of zebrafish heartBefore synchronization (raw data)

z1 z2

Mary Dickinson, Arian Forouhar

4D Imaging via Post-Acquisition Synchronization

<< < > >> B F R I Page 17

Method

Adjacent 2D slices of zebrafish heartAfter synchronization

z1 z2

Mary Dickinson, Arian Forouhar

4D Imaging via Post-Acquisition Synchronization

<< < > >> B F R I Page 17

Manual 4D Reconstruction

<< < > >> B F R I Page 18

Limitations of manual registrationTedious, non-reproducible, error-prone.

ChallengesLarge data size, limited time, fluorescence imaging artifacts (bleaching,photon/detector noise, etc.).

RequirementsNo a priori information (electrocardiogram, etc.), robust (accurate andreproducible)

X Y Z Time Channels Depth Memory

512 512 128 512 4 16 bits 128GB

512 512 128 512 2 8 bits 32GB

256 256 64 256 1 8 bits 1 GB

256 256 64 128 1 8 bits 512 MB

Automatic Registration Challenges

<< < > >> B F R I Page 19

Sequential z-acquisition:

z1

z1z2

xy

t

t

Measurement model:

Im(x, zk, t) =ZZZI(x′, z, t − sk)

PSF(x − x′, z − zk) dx′dz

for k = 1, . . . , Nz. Unknowns shifts sk .Assumptions:• Periodicity: |I(x, z, t)− I(x, z, t+T )| � Imax• Same period T at all depths• Acquisitions overlap: suppz(PSF) > Éz

Synchronization (time-registration):Error metric

|Im(x, z1, t) − Im(x, z2, t − s1,2)|2

z1

s

z2

z1

z2

t

1,2

4D Confocal Imaging of the Beating Heart

<< < > >> B F R I Page 20

. . .with some help from astronomy

i + 1τ

i - 1τ

iτ

j+1τʼ

j-1τʼ

jτʼ

j+1τʼ

j-1τʼ

jτʼ

T’1

1

T’2

2

10

10

T

t

t

t

Original Sequence

Wrong Guess T’

Correct Guess T’

R. F. Stellingwerf, “Period determination using phase dispersion minimization,” Astrophysical Journal, 224(3),953–960 (1978)M. M. Dworetsky, “A period-finding method for sparse randomly spaced observations or how long is a piece ofstring,” Monthly Notices of the Royal Astronomical Society, 203(3), 917–924 (1983).

Finding the Period. . .

<< < > >> B F R I Page 21

Mean square intensity di«erence

Qk,k′(s) =ZZ

R2

Z L0|Im(x, zk, t) − Im(x, zk′, t − s)|2 dt dx where s ∈ R

Periodicity implies:

Qk,k′(s) =ZZ

R2

Z L0|Im(x, zk, t)|2 + |Im(x, zk′, t − s)|2 dt dx

− 2ZZ

R2

Z L0Im(x, zk, t)Im(x, zk′, t − s) dt dx

= C − 2ZZ

R2

Z L0Im(x, zk, t)Im(x, zk′, t − s) dtj ız ℘

Correlation

dx.

Least-squares error minimization (≡ correlation maximization)

sk,k′ = mins=kT,k=1,...,Nt

Qk,k′(s).

Intensity-based Registration—Periodic Case

<< < > >> B F R I Page 22

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Shift

s1,2

Correlation

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Shift

s1,2

Correlation

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Shift

s1,2

Correlation

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Shift

s1,2

Correlation

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Shift

s1,2

Correlation

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

t

t

Intensity I(x,z ,t)1

Intensity I(x,z ,t)2

t

t

Shift

s1,2

Correlation

Correct Shift Maximizes Correlation

<< < > >> B F R I Page 23

20 μm

Raw Data Wavelet Coefficients Data Reduction

Overcoming Fluorescence Imaging Artifacts

<< < > >> B F R I Page 24

t t t

Original Wavelet Transform Data ReductionThreshold

Features:Cropping→ increase speed, save memoryRemove high pass bands, threshold→ robustness to photon / acquisition noiseRemove low pass band→ robustness to bleaching artifactsTime interpolation→ increase accuracy

M. Liebling, A.S. Forouhar, M. Gharib, S.E. Fraser, M.E. Dickinson, J. Biomedical Optics, 10 (5), 2005.

Thresholded Wavelet Coe»cient Synchronization

<< < > >> B F R I Page 25

Relation between absolute shifts sk and relative shifts sk,k ′ :266666666666666664

1 0 0 0 0

1 −1 0 0 0

0 1 −1 0 0

0 0 1 −1 0

0 0 0 1 −11 0 −1 0 0

0 1 0 −1 0

0 0 1 0 −1

377777777777777775

j ız ℘A

0BBBBBBB∂

s1s2s3s4s5

1CCCCCCCA

j ız ℘t

=

266666666666666664

0

s1,2s2,3s3,4s4,5s1,3s2,4s3,5

377777777777777775

j ız ℘s

Weighted least-squares solution (via normal equations):

A†W†WAt = A†W†Ws

with weights W = diag(1, w1, w1, w1, w1, w1, w2, w2, w2).

Absolute Shift Determination

<< < > >> B F R I Page 26

z1

z2

z3

z4

z1

z2

z3

z4

t

A Special Case. . .

<< < > >> B F R I Page 27

4 Possible Reconstructions

<< < > >> B F R I Page 28

. . . an ill-Posed Problem

<< < > >> B F R I Page 29

Conversion from 2D+time to 3D+time is possible when:

a) Motion is homogeneous along z (→ hypothesis: PSF overlap).

b) Availability of supplementary knowledge:ECG, triggered/gated measurements, etc.

Shear is allowed!

z2

z1

z3

z2

z1

z3

Time

When is reconstruction possible?

<< < > >> B F R I Page 30

Validation method:1. Random shifts definition2. Simulated data acquisition(random periodic deformation)3. Reconstruction4. Comparison

Average error of absolute shiftsε̄ = 0.31 ± 0.08 frames(Calculated over 100 random periodichomogeneous deformations40 frames, 2× time-oversampling)

⇒ Sub-frame accuracy

M. Liebling et al.,J. Biomed. Opt., 10(5), 2005.

Simulated original

Reconstruction

What is the accuracy?

<< < > >> B F R I Page 31

Specific Imaging by Use of Transgenic Fish

<< < > >> B F R I Page 32

<< < > >> B F R I Page 33

Acquisition Model

Im(x, zk, t) =ZZZI [x′, z, τk(t)]h(x − x′, zk − z) dx′dz,

τk(t): unknown warping functions. h(x, z): Microscope point spread function(PSF) Hypothesis

|I [x, z, τ] − I [x, z, τ + T ]| � Imax.Objective criterion (absolute di«erence):

Qk,k ′,wk {w ′k} =Z L0

ZZR2|Im[x, zk, wk(τ)] − Im[x, zk ′ , wk ′ (τ)]|dxdτ.

Goal:Recover warping functions wk ′ (τ) for k

′ = {0, . . . , Nz − 1}\{k̄}.Approach:(a) Interpolation and oversampling of reference sequence⇒ discrete wk(τ)(b) Frame matching via dynamic programming

⇒ Reduce complexity from Oζ(Nr )

Ntηto O(NrNt).

Nonuniform temporal registration

<< < > >> B F R I Page 34

0 10 20 30 40 50 60 70 80−1

−0.5

0

0.5

1

1.5

2

2.5

3

3.5

4ReferenceTest

Nonuniform Registration

<< < > >> B F R I Page 35

0 10 20 30 40 50 60 70 80−1

−0.5

0

0.5

1

1.5

2

2.5

3

3.5

4ReferenceTestWarped

Nonuniform Registration

<< < > >> B F R I Page 35

Nonuniform Dynamic Programming Registration

<< < > >> B F R I Page 36

Nonuniform Dynamic Programming Registration

<< < > >> B F R I Page 36

Nonuniform Dynamic Programming Registration

<< < > >> B F R I Page 36

Nonuniform Dynamic Programming Registration

<< < > >> B F R I Page 36

Nonuniform Dynamic Programming Registration

<< < > >> B F R I Page 36

Nonuniform Dynamic Programming Registration

<< < > >> B F R I Page 36

Slices before synchronizationz = 0 μm z = 10 μm z = 20 μm z = 30 μm z = 40 μm

Slices after synchronization

M. Liebling, J. Vermot [38 hpf wildtype zebrafish, BODIPY FL C5-ceramide]M. Liebling, J. Vermot, A.S. Forouhar, M. Gharib, M.E. Dickinson, S.E. Fraser, Proc. ISBI’06, in press.

Robust and Reproducible Synchronization

<< < > >> B F R I Page 37

Before synchronization After synchronization

M. Liebling, J. Vermot [38 hpf wildtype zebrafish, BODIPY FL C5-ceramide]M. Liebling, J. Vermot, A.S. Forouhar, M. Gharib, M.E. Dickinson, S.E. Fraser, Proc. ISBI’06, in press.

Automatic synchronization allows 4D reconstruction

<< < > >> B F R I Page 38

M. Liebling, J. Vermot [38 hpf wildtype zebrafish, BODIPY FL C5-ceramide]

Synchronized dataset is suited for 3D rendering

<< < > >> B F R I Page 39

0 ms 100 ms 200ms 300 ms 400 ms

AA A A A

VV VVV

0 ms 75 ms

100 μm

150ms 225 ms 300 ms

AA

A A A

V VV

VV

100 μm

Heart Cycle @ 48hpf, gata1::GFP Zebrafish

<< < > >> B F R I Page 40

M. Liebling, A.S. Forouhar, M. Gharib, S.E. Fraser, M.E. Dickinson, J. Biomedical Optics, 10 (5), 2005.

Microscopic Heart

<< < > >> B F R I Page 41

Fast 3D+time imaging is possiblein the living embryonic heart

Automatic reconstruction procedureallows quantitative measurements

in a large number of fish

Automated Reconstruction:• Robust to confocal microscopy artifacts• Adapted to large data size• Accuracy validated in silico and in vivo• Allows for systematic and quantitative measurements

Summary

<< < > >> B F R I Page 42

Contact: Michael Liebling, http://www.its.caltech.edu/~liebling/

CollaborationsCarl Zeiss, Advanced Imaging MicroscopyShuo Lin, UCLA [Tg(gata1::GFP)]Huai-Jen Tsai, Nat’l Taiwan U. [Tg(cmlc2::GFP)]Colophon4D processing & rendering: Matlab, Imaris (Bitplane AG)Postprocessing:ImageMagick, GraphicConverter, pdfLATEXFundingHuman Frontiers, Center for Bioengineering for the Explorationof Space, American Heart Assoc., NSF, NIHSwiss National Science Foundation

Arian S. ForouharMory GharibScott E. FraserMary E. DickinsonSonia CollazoAnna HickersonLiz JonesAbbas MoghaddamCarol ReadheadJulien VermotChris Waters

Index Studying the Developing HeartZebrafish EmbryosGenetic and Epigenetic FactorsSpeed Biological Processes Mov-ing at the Speed of CellsInsu»cient frame-rate artifactsFluorescence: ReminderConfocal Microscopy ReminderPoint ScanningPoints vs line scanZeiss LSM 5 LIVE

Parallel AcquisitionGated 4D ImagingNongated 4D Imaging4D ImagingManual 4D ReconstructionAutomatic Registration ChallengesProblem definitionFinding the Period. . .Intensity-based RegistrationCorrect Shift Maximizes CorrelationWavelets and Fluorescence

Wavelet SynchronizationAbsolute Shift DeterminationA Special Case. . .When is the reconstruction possible?What is the accuracy?cmlc2 3DDev. of Zfish Heart (cmlc2)Nonuniform temporal registrationDP exampleDP RegistrationBodipy Galery

Bodipy SliceBodipy 3DHeart Cycle @ 48hpfDev. of Zfish Heart (gata1)Volume MeasurementsPeristaltic vs Impedance PumpingImpedance Pump: Mechanical ModelEarly embryonic heart tubeIs pump peristaltic?Summary

Acknowledgments

<< < > >> B F R I Page 43