19
UNIVERSITY OF CALIFORNIA

CALIFORNIA UNIVERSITY OF

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: CALIFORNIA UNIVERSITY OF

UNIVERSITY OFCALIFORNIA

Page 2: CALIFORNIA UNIVERSITY OF

The cctbx.xfel GUI: processing serial crystallographic data at massive scales

Aaron S. Brewster

March 17th, 2021

Page 3: CALIFORNIA UNIVERSITY OF

Serial crystallography

• 102-107 images/dataset• Each crystal is independent

and randomly oriented• Rep rates:10-120 Hz• Upcoming: 3 kHz

Page 4: CALIFORNIA UNIVERSITY OF

How much data do I need?

• What is my hitrate?– Solvent rate– Crystal rate– Indexing rate– High quality rate

• What is my completeness?• Can I stop collecting data?

Beamtime is scarce! Fast feedback is critical!

4

Page 5: CALIFORNIA UNIVERSITY OF

cctbx.xfel GUI

• Built on cctbx and DIALSWinter G. (2018). DIALS: implementation and evaluation of a new integration package. Acta Crystallogr D Struct Biol 74, 85-97• Full SX workflow– Each image: import, spot-find, index, integrate, scale,

post-refine, merge– Monitor for new data and submit jobs– Organize processing many datasets– Massively parallel multiprocessing on the biggest

computing clusters– MySQL backend allows scaling to the kHz regime

Footer 5

Page 6: CALIFORNIA UNIVERSITY OF

6

Page 7: CALIFORNIA UNIVERSITY OF

7

Page 8: CALIFORNIA UNIVERSITY OF

Rate tracking

8

Page 9: CALIFORNIA UNIVERSITY OF

Non-isomorphism: two clusters

Axis length (Å)Unit cell plots from 17786 metalloprotein crystals (space group P212121) collected at LCLS beamline MFX.

Page 10: CALIFORNIA UNIVERSITY OF

Heuristics

• How much data do I need?– What is my hitrate?– What is my completeness?– Can I stop collecting data?

Easy answer (rule of thumb):• 10-fold multiplicity at the resolution of interest• Monotonically decreasing CC1/2 to the resolution of

interestThe easy answer lends itself to sampling the data

10

Page 11: CALIFORNIA UNIVERSITY OF

How much data do I need?

• How much data do I need to answer my biological questions?

11

Page 12: CALIFORNIA UNIVERSITY OF

PSII 2Fo-Fc and Fo-Fc at 2.07 Å, S3 state

Kern et al., 2018 Nature 563:421

How much data do I need?

Page 13: CALIFORNIA UNIVERSITY OF

Data analytics at the petascale for free electron lasers

• Processing 1 image takes 30s per hit, 1s per miss, on average• Vastly varies depending on protocol:– Multiple indexing methods– Multiple lattices– Large unit cell– Hit rate• 30 Hz Rayonix at LCLS: only 288 cores available. Cannot keep

up with good hitrates– 100% hitrate = 30s/image * 30 images/s = 900s to process 1s of

data collection– Need 900 cores

Page 14: CALIFORNIA UNIVERSITY OF

Data analytics at the petascale for free electron lasersSolution:

• Sample the data at LCLS for GUI (process 1 in 5 images). At the same time, transfer the data over 100 Gb-ESnet to NERSC, and process all images on Cori– 2388 Intel Xeon Haswell nodes– 9688 Intel Xeon Phi “Knight’s Landing” nodes (68 cores)– 1.8 PB burst buffer (keep the data on solid state drives)

• 9 TB example experiment; 861969 diffraction images (2.0 hrs’ data collection @ 120 Hz)– 150466 useful crystal patterns (17% hit rate)– Required 140 “Knight’s Landing” node-hours to process– Not counting I/O, 70 nodes in continuous use during data collection– About 1% of Cori

Four year Exascale Computing Project funding to scale this for • bigger algorithms• faster LCLS-II pulse rate• exascale machines in a few years’ time• Collaboration with Amedeo Perrazo, SLAC

Page 15: CALIFORNIA UNIVERSITY OF

Data collection at LCLS / MFX December 2018

• Four 12-hour shifts at MFX (LU50 / P143)• 20 – 30 Hz overall collection rate (Rayonix

340)• 594,230 total indexed images• > 8.4 indexed images s-1 (31% of all shots)• Overall 48% collection duty cycle• ~ 54 tr-SFX + tr-XES data sets• ~ 12 types of enzymes • Typically ~ 1 mg protein per dataset• Some data beyond 1.3 Å resolution

Page 16: CALIFORNIA UNIVERSITY OF

Deployment: every XFEL

16

LCLS SACLA PAL-XFEL SwissFEL EuXFEL

Location Stanford, CA USA

Harima, Japan

Pohang, South Korea

Aargau,Switzerland

Schenefeld,Germany

Computing NERSC or SDF

Local KISTI PSI Maxwell

Detectors CSPAD, ePix, Jungfrau, Rayonix

Octal-8 MPCCD,Rayonix

Rayonix Jungfrau 16M

AGIPD

Supports LSF, SGE, Slurm, HTCondor, Docker/ShifterNew use of GUI: DLS beamline I24 (Pilatus3 6M)Best current result: LCLS + NERSC, 5 minute turnaround for fully merged dataset

Page 17: CALIFORNIA UNIVERSITY OF

17

Asmit Bhowmick

Aaron Brewster

NickSauter

Derek Mendez

XFEL @ LBL

DanPaley

ElyseSchriber

Page 18: CALIFORNIA UNIVERSITY OF

AcknowledgementsDiamond Light SourceGwyndaf EvansGraeme WinterDavid StuartJonathan GrimesJames ParkhurstLuis Fuentes-MonteroMarkus GerstelNicholas DevenishAllen OrvillePierre AllerAgata ButrynDanny AxfordTiankun ZhouJos KampsRobert Bosman

CCP4David Waterman

UCLADavid EisenbergDuilio CascioMichael SawayaJose RodriguezLuki GoldschmidtJohan Hattne

Berkeley National LabNicholas SauterAsmit BhowmickDerek MendezDan PaleyElyse SchriberPaul AdamsPeter ZwartVittal YachandraJunko YanoJan KernRuchira ChatterjeeJames HoltonJohannes BlaschkeIn-Sik KimCindy C. PhamJoshua HeinemannLouise LasalleBilly K. Poon

LLNLMichael Wall

BrookhavenHerbert Bernstein

NebraskaLimei Zhang

IBSJacques-Philippe Colletier

Stanford UniversityAxel BrungerBill WeisSoichi WakatsukiHenry van dem Bedem

SSRLMike SoltisAna GonzalezAshley DeaconAina CohenArtem LyubimovScott McPhillips

Charité-Universitätsmedizin BerlinAndrea SchmidtMichal SzczepekPatrick Scheerer

AlumsNat EcholsTara Michels-ClarkLee O'RiordanMuhamed AminOliver ZeldinSheraz GulRobert BolotovskyKyle D. Sutherlin

NIH/NIGMS grants 1R01GM117126DOE/Office of Science contract DE-AC02-05CH11231

LCLSUwe BergmannAlberto LutmanFranklin FullerRoberto Alonso-MoriSebastien BoutetMengning LiangAmedeo PerazzoChuck YoonChris O’GradyMark HunterJason KoglinRay SierraMona UervirojnangkoornAlexander BatyukDiling ZhuAnton D. Loukianov

UCSFJames FraserIris YoungMichael ThompsonAlex WolffRahel Woldeyes

HarvardKevin DaltonDoeke Hekstra

Zoom background: “Black Lotus” from Magic the Gathering

Page 19: CALIFORNIA UNIVERSITY OF

References

Brewster AS, et. al (2019): Processing serial crystallographic data from XFELs or synchrotrons using the cctbx.xfel GUI. Computational Crystallography Newsletter 10, 22-39.

http://www.phenix-online.org/newsletterBrewster AS, et. al (2018): Improving signal strength in

serial crystallography with DIALS geometry refinement. Acta Crystallogr D Struct Biol 74, 877-894.

http://cci.lbl.gov/xfelhttp://dials.github.io

19


University of California, Los Angelestom/papers/ruin.pdf · University of California, Los Angeles

University of California, Los Angelestom/papers/ruin.pdf · University of California, Los Angeles

Documents
University of California, Riverside, California

University of California, Riverside, California

Documents
University of California Strategies for Decarbonization ......5 University of California, Berkeley 6 University of California, Davis 7 University of California, ... 14 National Center

University of California Strategies for Decarbonization ......5 University of California, Berkeley 6 University of California, Davis 7 University of California, ... 14 National Center

Documents
University of Southern California - USC SAASsaas.usc.edu/files/2012/10/Tutorial-Manual.pdf · University of Southern California SAAS Tutorial Manual 1 University of Southern California

University of Southern California - USC SAASsaas.usc.edu/files/2012/10/Tutorial-Manual.pdf · University of Southern California SAAS Tutorial Manual 1 University of Southern California

Documents
UNIVERSITY OF CALIFORNIA, DAVIS, WOMEN’S CARDIOVASCULAR ... · UNIVERSITY OF CALIFORNIA, DAVIS, WOMEN’S CARDIOVASCULAR MEDICINE ... University of California, Davis Women’s Cardiovascular

UNIVERSITY OF CALIFORNIA, DAVIS, WOMEN’S CARDIOVASCULAR ... · UNIVERSITY OF CALIFORNIA, DAVIS, WOMEN’S CARDIOVASCULAR MEDICINE ... University of California, Davis Women’s Cardiovascular

Documents
University of California

University of California

Documents
Lexical Tools Sub-Term Mapping Tools (STMT)...• University of California, Los Angeles • University of California Los Angeles • University of California-Los Angles • university

Lexical Tools Sub-Term Mapping Tools (STMT)...• University of California, Los Angeles • University of California Los Angeles • University of California-Los Angles • university

Documents
Peter Jones, University of California, Irvine Anne Jones, University of California, Riverside

Peter Jones, University of California, Irvine Anne Jones, University of California, Riverside

Documents
University of California, Los Angeles Medical Centerregents.universityofcalifornia.edu/regmeet/nov09/j1ucla.pdf · University of California, Los Angeles ... University of California,

University of California, Los Angeles Medical Centerregents.universityofcalifornia.edu/regmeet/nov09/j1ucla.pdf · University of California, Los Angeles ... University of California,

Documents
UNIVERSITY of CALIFORNIA · UNIVERSITY of CALIFORNIA Implementing AB 2302 (Fong): Associate Degree Pathway to the University of California-Final January 2012 Legislative Report

UNIVERSITY of CALIFORNIA · UNIVERSITY of CALIFORNIA Implementing AB 2302 (Fong): Associate Degree Pathway to the University of California-Final January 2012 Legislative Report

Documents
Code Vendor - Nevada · ubc university of british columbia ucb university of california at berkeley-clark kerr campus ucc university of california at chico ucla university of california

Code Vendor - Nevada · ubc university of british columbia ucb university of california at berkeley-clark kerr campus ucc university of california at chico ucla university of california

Documents
University of California Berkeley, California The Wine

University of California Berkeley, California The Wine

Documents
University of California, Berkeley and University of

University of California, Berkeley and University of

Documents
Choosing an Alfalfa - University of California, Davis · University of California, Davis. Steve B. Orloff. Farm Advisor, University of California Cooperative Extension, Yreka, CA

Choosing an Alfalfa - University of California, Davis · University of California, Davis. Steve B. Orloff. Farm Advisor, University of California Cooperative Extension, Yreka, CA

Documents
University of California, Berkeleybh/pdf/ssch25.pdf · University of California, Berkeley

University of California, Berkeleybh/pdf/ssch25.pdf · University of California, Berkeley

Documents
University of California, IrvineUniversity of California, Irvine

University of California, IrvineUniversity of California, Irvine

Documents
Garrick Blalock, Cornell University Paul Gertler, University of California, Berkeley David Levine, University of California, Berkeley

Garrick Blalock, Cornell University Paul Gertler, University of California, Berkeley David Levine, University of California, Berkeley

Documents
University of California, San Diego · University of California, San Diego

University of California, San Diego · University of California, San Diego

Documents
UNIVERSITY OF CALIFORNIA - University of North Texas

UNIVERSITY OF CALIFORNIA - University of North Texas

Documents
University California - University of California, Berkeley

University California - University of California, Berkeley

Documents
CAPITAL PROJECT COSTS - University of California, … Cost.pdfcost management: construction delivery method. university of california capital project costs ... university of california

CAPITAL PROJECT COSTS - University of California, … Cost.pdfcost management: construction delivery method. university of california capital project costs ... university of california

Documents
University California College of the Arts California ... Accreditation Redesign 2012.pdfAcademy for Jewish Religion California Academy of Art University Alliant International University

University California College of the Arts California ... Accreditation Redesign 2012.pdfAcademy for Jewish Religion California Academy of Art University Alliant International University

Documents
University of California, Irvine University Registrar

University of California, Irvine University Registrar

Documents
University of California, San Francisco (UCSF) - NDM - University of California San...University of California, San Francisco (UCSF) Organization University of California, ... the

University of California, San Francisco (UCSF) - NDM - University of California San...University of California, San Francisco (UCSF) Organization University of California, ... the

Documents
University of California, Berkeley · University of California, Berkeley

University of California, Berkeley · University of California, Berkeley

Documents
University of California, Davisvojin/CLASSES/EEC272/S2005/...University of California, Davis

University of California, Davisvojin/CLASSES/EEC272/S2005/...University of California, Davis

Documents
Northwestern California University SCHOOL OF LAW · B.S.L. Northwestern California University J.D. Northwestern California University Former Assistant Professor – United States

Northwestern California University SCHOOL OF LAW · B.S.L. Northwestern California University J.D. Northwestern California University Former Assistant Professor – United States

Documents
UNIVERSITY OF CALIFORNIA Wendell C. Brase Wendell C. Brase Chair, University of California Chair, University of California Climate Solutions Steering Group

UNIVERSITY OF CALIFORNIA Wendell C. Brase Wendell C. Brase Chair, University of California Chair, University of California Climate Solutions Steering Group

Documents
The University of California and the California State ...The University of California (UC) and the California State University (CSU) are the two public university systems in California

The University of California and the California State ...The University of California (UC) and the California State University (CSU) are the two public university systems in California

Documents
University of California, Berkeley arXiv:1905.13285v3 ... · University of California, Berkeley jordan@cs.berkeley.edu Peter L. Bartlett University of California, Berkeley peter@berkeley.edu

University of California, Berkeley arXiv:1905.13285v3 ... · University of California, Berkeley [email protected] Peter L. Bartlett University of California, Berkeley [email protected]

Documents