2011-02-18 Jérôme Kieffer - EDNA-Passerelle 1

EDNA:

Scientific workflows for synchrotron

Example of existing pipelines

Developed by ESRF & DLS

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 2

Scientific applications of EDNA

• Full Field X-Ray Absorption Spectroscopy• FF-XANES (ID21)

• Bio-Saxs collaboration• Up-stream data-reduction (ESRF + EMBL)• Down-Stream data processing (DLS + EMBL)

• Diffraction workflows• Diffraction Computed Tomography (ID22)• Material science (ID11)• Fiber diffraction (ID13)

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 3

Full Field X-Ray Absorption Spectroscopy

First pure SciPy workflow

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 4

Scientific background

SampleMonochromatic beam

Scintillator

2D CCD Camera

Attenuated beam

Measure a (couple of) flats at each energy to correct for scintillator's response Align the sample at each position to correct sub-micron position change

Scan in Energy on monochromator around absorption edge of a given element

Non attenuated beam

Mo

ve in

/ou

t o

f b

ea

m

Sample size: 1 mm x 1 mm x 1 µmResolution: 500 nm

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 5

AccumulatorRaw

AccumulatorShift

MeasureOffset

Shift Image

HDF5Image stack

Align Stack

FullField XAS PipelineFullField XAS Pipeline

NormalizeImage v1_0

I – Idark

Iflat – Iflat

dark

_______

Mesure sub-pixeloffset by 2D correlation

(FFT)

Make pairs of images

Calculateabsoluteoffset

scipy.ndimage.shift

Stack imagesIn HDF5

Pure SciPyCode

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 6

FullField XAS Results

Works:• Tested online & offline in December 2010 (ID21)• Towards Nexus compliance

• Problems:• Memory consumption for both EDNA & PyMca• Distinction between feature of the image and artifacts (!)

• Limits of parallelism with the GIL (and/or) serialization of HDF5

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 7

Small Angle Scatteringfor Proteins

Collaboration with EMBL & Diamond

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 8

DLS-EMBL-ESRF BioSaxs Collaboration

Who does what ...• ESRF & DLS collaborate on the maintenance of EDNA Kernel

(port to windows asked by EMBL). http://www.edna-site.org

• EMBL provides SAXS tools (as executable under unix & win)http://www.embl-hamburg.de/biosaxs/software.html

• ESRF and DLS provide execution plugins for EMBL binaries• ESRF focuses on data reduction EDNA plugins (→ AutoRG) • DLS focuses on downstream processing (DAMMIF / DAMMIN)

•

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 9

Overview of the preprocessing pipeline (Existing but not used)

Normalization

Azimuthal integration

Averaging of frames

BSxCube

ESRF ID14-eh3

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 10

Focus on the Normalisation part

Saxs M A C

Saxs Add Metadata

Normalized Image

Wait File

Raw Image + Metadata(Pilatus)

EDPluginBioSaxsNormalizev1_0

edfheaderaddkey

Work aroundfor buggy NFS

at ESRF

Calling an external program toMultiply by a constant an array !!!

Each metadata is added independently

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 11

Focus on Azimuthal integration part

WaitFile

Normalized Image (+Metadata)

(Get) Metadata→ Fabio Saxs Add

(Set) Metadata

SaxsAngle

SaxsCurves

Masked Im

age

Inte

grat

ed Im

age

Integrated Spectrum

EDPluginBioSaxsAzimutIntv1_0

edfheaderaddkey

Again: many programscalled to do trivial tasks

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 12

Focus on data averaging (basic version)

Wait Multi File

Integrated EDF + Metadata

(Get) Metadata→ Fabio

SaxsMAC

SaxsCurves

Ave

rage

d Im

age

Averaged Spectrum

EDPluginBioSaxsAveragev1_0

Saxs Add Metadata

edfheaderaddkey

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 13

Pro / Cons / Future of the BioSaxs pipeline

• Pro:• Reproduces the data of the existing pipeline• Adds proper metadata management

• Cons• Was not adopted because added value was too small• Dozens of external programs called (slow pipeline)

• Future:• Integration of new Fabio which is compatible with saxs* programs• Reduce external calls to a single program: saxs angle• Evaluation of radiation damage & AutoRg

→ Demo in May 2011

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 14

EDNA application around diffraction

Making best use of SPD

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 15

ID22 – Fluorescence-Diffraction Tomography

Fluorescence Detector

DiffractionCCD

Sample

Beam

1cm

zx

y

ω

Courtesy of V.A. Solé

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 16

Ny×Nω Diffraction Images

1cm

y

ω

Sum Sinogram

Sum Pattern

Azimuthal Integrations

Ny×Nω Diffraction Patterns

Fit2d software

Phase Sinograms

ωy

PyMca software

ReconstructionCapillary

Calcite

Ferrite

Cubicsp3

25µm

y

x

Acknowledgements: Pierre Bleuet CEA - Grenoble

Original idea: 20h data acquisition, 30h treatment

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 17

EDNA - Difffraction Computed Tomography

EDPluginControlDiffractionCTv1_2

Input Image

HDF5MapOfSpectra

ExportAscii Powder

EDPluginControlDCTPowderIntegrationv1_1

EDPluginSPDCakev1_5

SPDEDPluginControlDCTReadHeaderv1_0

EDPluginEDFReadHeaderv1_0

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 18

Diffraction tomography on ID22

• Mesh 120 x 220 (26k images 2k x 2k)• All data processed on the fly:

• Meta-data extraction• Dark & flat field correction.• Tilt & distortion corrections• Azimuthal integration (SPD)• Sinogram: HDF5 3D dataset,

• live ROI with PyMca

• Process time: 1.5 s/image

• Throughput (8 cores): 4.3 fps

• Future:

• PyMca will incorporate Algebraic Tomographic reconstruction

• Nexus compliance of HDF5 data (axis, ...)

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 19

Spreading of the SPD-solution

• Soft Condensed Matter Nanofocus beamline (ESRF ID13)• SPD plugin tested with the online mode (EDNA Parallel Execute)• 10 frames per second ( 512 x 512 image)• 2 SPD processes under control on a 4-core computer• Image correction (dark, flat, distortion) + azimuthal integration

• Materials Science beamline (ESRF ID11)• SPD plugin tested with the offline mode (EDNA Parallel Execute)• Images coming from 3 different detectors• Selection of the right process for correction and azimuthal

integration • Pipeline under evaluation

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 20

Acknowledgements

• ID11 (Materials Science): • Jon Wright & Andy King

• ID13 (NanoBeam): • Manfred Burghammer

• ID14-eh3 (BioSaxs): • Louiza Zerad & Adam Round

• ID21 (FF Xanes): • Barbara Fayard & Muriel Salomé

• ID22 (Diffraction computed tomography): • Rémi Tucoulou

• SPD developers: • Peter Boesecke & Rainer Wilcke

• ODA: • Cyril Guilloud, Armando Solé & Olof Svensson

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 21

2011-02-18 Jérôme Kieffer - EDNA-Passerelle 22

List of EDNA Plugins

• Number of Control Plugins: 68• Number of Execution Plugins: 88• Number of Others Plugins: 21