CCP4 Chicago 2011: Data Integration with XDS · 2011. 11. 10. · XDS.INP XDS is controlled by one single input ﬁle: XDS.INP . Name cannot be changed Each data set must be run in

Tim Grüne

CCP4 Workshop Chicago 2011Data Integration with XDS

Tim GrüneDept. of Structural Chemistry, University of Göttingen

June 2011http://shelx.uni-ac.gwdg.de

[email protected]

CCP4 Chicago 2011: XDS 1/29

http://shelx.uni-ac.gwdg.de

Tim Grüne

The XDS Distribution

XDS is an integration program for monochromatic X-ray data

originally written by W. Kabschnow co-authored by K. Diederichs

availability:XDS http://xds.mpimf-heidelberg.mpg.de/ main program suitexds-viewer http://xds-viewer.sourceforge.net/ viewer for control images (.cbf-files)Wiki and auxiliary programs http://strucbio.biologie.uni-konstanz.de/xdswiki/, maintained by Kay

DiederichsXDSi (GUI) http://cc.oulu.fi/˜pkursula/xdsi.html


http://xds.mpimf-heidelberg.mpg.de/http://xds-viewer.sourceforge.net/http://strucbio.biologie.uni-konstanz.de/xdswiki/http://cc.oulu.fi/~pkursula/xdsi.html

Tim Grüne

XDS Characteristics

Some of the special features of XDS:

• 3-dimensional spot integration• Correction for Radiation Damage• Optimised for new Pilatus Detector• Command-line program• Parallelised: Fast!• Simple to read documentation


Tim Grüne

Learning Curve

• Why newbies often do not like XDS:1. No GUI, only command line2. error messages may initially appear “cryptic”3. users must read (rather than look at graphics)

• Once understood, the program flow is simple and very easy to set up• Understanding XDS gets you better data quality (also with other integration programs)


Tim Grüne

Documentation

XDS and accompanying programs come with very clean (html-) doc-umentation. Every step of the program and every keyword is de-scribed, and reading is not interrupted by many cross-links.Documentation contains both technical information for those whowant to understand how XDS works in details as well as user in-formation that help setting up the integration process.


Tim Grüne

Other Sources of Information

Wiki http://strucbio.biologie.uni-konstanz.de/xdswiki/ Tips, tricks, tutorials,. . .CCP4bb Sign up at www.ccp4.ac.uk


http://strucbio.biologie.uni-konstanz.de/xdswiki/file:www.ccp4.ac.uk

Tim Grüne

Templates

• Templates of input scripts for all sup-ported detector formats• Only very few adjustments necessary to

get started• Beamlines often generate appropriate in-

put scripts• generate XDS.INP from XDSwiki for

MARCCD, ADSC, and TODO


Tim Grüne

XDS.INP

XDS is controlled by one single input file: XDS.INP.

• Name cannot be changed• Each data set must be run in separate directory to avoid overwriting of files.• Contains about 100 Keywords a of the form

KEYWORD=VALUE

• Only about 10 Keywords must be modified for most data sets (e.g. image names, detectordistance, number of images, etc.)• Most important one:

JOB= XYCORR INIT COLSPOT IDXREF DEFPIX XPLAN INTEGRATE CORRECT

Each name stands for one of the steps XDS carries out during data integration.(XPLAN is optional and corresponds to the BEST [2] or STRATEGY [1] output to report optimaldata collection range(s))

aalso called “cards” for historical reasons


Tim Grüne

The Steps

XYINIT writes files for positional corrections of the detector plane. Most modern detectors providealready corrected images so that these to files are normally flat.

INIT determines initial detector backgroundIDXREF indexing: unit cell dimensions and crystal orientationDEFPIX set active dectector area (exclude resolution cut-off, beam stop shadow, . . . )(XPLAN, optional) generate “strategy” table with data completeness depending on

• starting angle• total scan width

INTEGRATE determine reflection intensitiesCORRECT applies corrections (polarisation, Lorentz-correction, . . . ), scales reflections, reports data

statistics


Tim Grüne

Program Flow

• Each step must be passed at least once - the subsequent steps depend on files produced bythe previous steps.• Each step creates a log-file (XYINIT.LP, INIT.LP,...).• IDXREF is the main hurdle - once unit cell and crystal orientation are determined, integration

usually runs smoothly.• CORRECT summarises the quality of the data.• Mostly IDXREF.LP and CORRECT.LP should be inspected.


Tim Grüne

Step Dependencies

IDXREF

COLSPOT

INIT

XYCORR

CORRECT

INTEGRATE

DEFPIX

XPARM.XDS

INTEGRATE.HKL

XDS_ASCII.HKL

GXPARM.XDS

BKGPIX.cbf

ABS.cbf

FRAME.cbf

FRAME.cbf

Step (Important)

output files

SPOT.XDS

SPOT.XDS

red: Text files with parameters or datacyan: control images (use xds-viewer)black: data files for further processing


Tim Grüne

SPOT.XDS

• written by COLSPOT: Detector coordinates and Intensity of strong spots to be used for indexing:X(pixel) Y(pixel) #image counts

1056.11 1529.51 15.35 2544.

1895.52 1525.49 9.19 2481.

1913.43 1547.90 2.63 1999.

• amended by Miller-Indices IDXREF after determination of unit cell and crystal orientationX(pixel) Y(pixel) #image counts H K L

1056.11 1529.51 15.35 2544. 14 13 18

1895.52 1525.49 9.19 2481. -7 -13 -14

1913.43 1547.90 2.63 1999. 0 0 0

Tim Grüne

“!!! ERROR !!! SOLUTION IS INACCURATE”

Correct indexing is crucial for data integration. If XDS cannot index more than 70 % of all reflections listed inSPOT.XDS, it stops with the above error message.

Most common reasons:

1. Wrong ORGX, ORGY2. Poor data quality

XDS has a very robust indexing method and depending on the situation, one can usually continue integrationby one or a combination of the following means.


Tim Grüne

Wrong ORGX, ORGY

The “detector origin” ORGX, ORGY in XDS.INP is provided in pixels. Typically it is in the centre of the detector, soa good start is ORGX=NX/2, ORGY=NY/2, i.e. half the detector dimensions.

Illustration of the sensitivity to ORGX, ORGYDiffraction image from Thaumatin with a mediumsize cell 58 Å, 58 Å, 150 Å, 90◦, 90◦, 90◦.

Coordinates of the four magnified spots:Coordinates Difference Difference(mm)1505 18001514 1793 9 7 0.66 0.511522 1788 8 6 0.59 0.441530 1780 8 7 0.59 0.51

An offset of the origin of about 0.5 mm means a shiftof 1 in Miller-Indices, mis-indexing.


Tim Grüne

Complaints about ORGX, ORGY

A good indicator of a wrongly set origin:

1. refined detector distance far from real value

2. ORGX, ORGY drift far away from initial value

IDXREF.LP:

***** DIFFRACTION PARAMETERS USED AT START OF INTEGRATION *****

[...]

DETECTOR ORIGIN (PIXELS) AT 1580.99 1506.71 (XDS.INP: 1450 1450)

CRYSTAL TO DETECTOR DISTANCE (mm) 211.85 (XDS.INP: 200)

The detector distance reported during data collection (image header) should be correct within fractions of mmat a proper beamline and not drift by 11 mm.


Tim Grüne

Correcting ORGX, ORGY1. Start with ORGX=NX/2, ORGY=NY/2 (useadxv [3] to read out header information). Of-ten XDS refines these values reasonably well.


Tim Grüne

Correcting ORGX, ORGY2. Check the image for the beam centre.At θ = 0◦ this correpsonds well to ORGX,ORGY. Ice rings are particularly usefule, theyare centred at about the beam centre (use cir-cle fitting option in mosflm).


Tim Grüne

Carry on regardless

The XDS error message SOLUTION IS INACCURATE does not necessarily mean that something is seriouslywrong. XDS is rather careful. The step IDXREF still refines the experimental parameters and writes them intoXPARM.XDS. This is all we need to continue.

Check IDXREF.LP for

• Number of indexed reflections

***** INDEXING OF OBSERVED SPOTS IN SPACE GROUP # 1 *****1909 OUT OF 2506 SPOTS INDEXED.

0 REJECTED REFLECTIONS (REASON: OVERLAP)597 REJECTED REFLECTIONS (REASON: TOO FAR FROM IDEAL POSITION)

with 30 % of indexed reflections the parameters in XPARM.XDS are probably good enough toconverged to the correct values during INTEGRATE and CORRECT.• refined detector distance and detector origin do not shift too much

Set JOB = DEFPIX INTEGRATE CORRECT and integrate your data.

If something did seriously go wrong, the file XPARM.XDS was not written.


Tim Grüne

DEFPIX: active detector mask

DEFPIX sets the area of the detector which is taken into account during integration. It takes into account:

1. INCLUDE_RESOLUTION_RANGE default: 20 Å to detector edge.2. VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS exclude shadowed regions, e.g. beamstop, cryo

stream nozzle3. UNTRUSTED_RECTANGLE exclude gaps of CCD chips e.g. Pilatus detector4. EXCLUDE_RESOLUTION_RANGE exclude ice rings from data integration


Tim Grüne

VALUE RANGE FOR TRUSTED DETECTOR PIXELS

ABS.cbf BKGPIX.cbf , all included

VALUE RANGE FOR TRUSTED DETECTOR PIXELS= 1000 30000


Tim Grüne

VALUE RANGE FOR TRUSTED DETECTOR PIXELS

ABS.cbf BKGPIX.cbf , shadows removed

VALUE RANGE FOR TRUSTED DETECTOR PIXELS= 6100 30000


Tim Grüne

INTEGRATE and CORRECT: Intensity determination and fine tuning

INTEGRATE determine spot intensities based on parameters (saved in XPARM.XDS)CORRECT experimental corrections (e.g. Lorentz- and Polarisation-correction), refine parameters and determine

space group (saved to GXPARM.XDS)CORRECT write FRAME.cbf:• predicted spot positions encircled• check correctness of predictions


Tim Grüne

Recycling

• Parameters (in GXPARM.XDS) depend on measured intensities• Intensities (including corrections) depend on Parameters⇒ rename GXPARM.XDS to XPARM.XDS and rerun XDS (JOB = DEFPIX INTEGRATE CORRECT) to im-

prove results.

This way one should also set the correct high- and low- resolution cut-offs


Tim Grüne

Resolution Cut-Off

The default resolution range in XDS is 20 Å to the detector edge

INCLUDE_RESOLUTION_RANGE=20.0 0.0

• Medium to low resolution data: increase 20 Å to 30 Å or even 50 Å (check BKGPIX.cbf)• After second round of integration: determine high-resolution cut-off. My favourite: xprep - higher

number of resolution shells than e.g. listed in CORRECT.LP

Why after second round?

• Correct space group rather than P1⇒ more symmetry related reflections⇒ more reliable data statistics, especially I/σI


Tim Grüne

High Resolution Cut-OffResolution #Data #Theory %Complete Redundancy Mean I Mean I/s Rmerge RsigmaInf - 7.07 181 218 83.0 4.24 1194.8 34.87 0.0324 0.02937.07 - 4.92 411 415 99.0 5.57 951.6 36.95 0.0341 0.02614.92 - 3.94 594 598 99.3 5.44 824.0 35.08 0.0403 0.02593.94 - 3.45 594 596 99.7 5.54 610.4 33.43 0.0418 0.02623.45 - 3.14 595 597 99.7 5.59 413.7 31.29 0.0441 0.02703.14 - 2.92 583 583 100.0 5.66 176.3 24.63 0.0514 0.03162.92 - 2.75 599 599 100.0 5.74 69.2 17.51 0.0832 0.04712.75 - 2.61 612 612 100.0 5.77 46.2 13.48 0.1018 0.06242.61 - 2.50 562 562 100.0 5.79 32.1 10.49 0.1467 0.08362.50 - 2.40 620 620 100.0 5.77 20.6 7.33 0.1971 0.12322.40 - 2.32 591 591 100.0 5.77 17.8 6.26 0.2213 0.14432.32 - 2.25 573 573 100.0 5.76 11.4 4.23 0.3183 0.22492.25 - 2.18 662 662 100.0 5.74 10.9 3.94 0.3295 0.24142.18 - 2.12 653 653 100.0 5.77 11.0 3.96 0.3439 0.24452.12 - 2.08 436 436 100.0 5.72 9.6 3.48 0.3875 0.28042.08 - 2.03 627 627 100.0 5.74 9.1 3.26 0.4194 0.29892.03 - 1.99 556 556 100.0 5.73 7.0 2.56 0.4740 0.3817 21.99 - 1.95 594 594 100.0 5.72 4.2 1.57 0.6256 0.6229

Tim Grüne

Low Resolution Cut-Off

BKGPIX.cbf BKGPIX.cbf

Default resolution range (20 Å - edge)- includes noise at edge- loose low resolution reflections

Adjusted resolution range (40 Å - 2.85 Å)- I/σI ≈ 2 in outer shell- low resolution reflection important for“shape” of molecule (MR, refinement)


Tim Grüne

Generating XDS.INP

• Download detector specific template and fill in manually.Advantage: Double-check of experimental settings — donever solely rely on correctness of image headers!• Many beamlines offer automatic generation of XDS.INP.

Crucial: Beamline specific chip borders and list of deadpixels for Pilatus detectors.• Script generate XDS.INP by Kay

Diederichs http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Generate XDS.INP– Works with MAR-CCD, ADSC, and Pilatus 6M detec-

tors– Number of images– Reads all information from image header

BKGPIX.cbf from Pilatus detector: Chip bor-ders and dead pixels excluded.


http://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Generate_XDS.INPhttp://strucbio.biologie.uni-konstanz.de/xdswiki/index.php/Generate_XDS.INP

Tim Grüne

XDSSTAT

The program XDSSTAT (Kay Diederichs, also from the XDS Wiki) creates more detailed statistics from XDSintegration.

The output can be viewed with a file browser - similar tables like scala output.

General rule of thumb: developers like encouragment — if you miss a feature point it out to the program authorto increase the chance of the feature being incorporated

(One of the most efficient “examples”: Paul Emsley & Bernd Lohkamp (Coot), Pavel Afonine (phenix)).


Tim Grüne

References

1. Leslie, A.G.W., Recent changes to the MOSFLM package for processing film and image plate data,Joint CCP4 + ESF-EAMCB Newsletter on Protein Crystallography (1992), No. 26

2. A.N. Popov and Bourenkov , Choice of data-collection parameters based on statistic modeling ActaCrystallogr. (2003). D59, 1145-1153

3. Andrew Arvai, http://www.scripps.edu/˜arvai/adxv.html

http://www.scripps.edu/~arvai/adxv.html

Documents

CCP4 Chicago 2011: Data Integration with XDS · 2011. 11. 10. · XDS.INP XDS is controlled by one single input ﬁle: XDS.INP . Name cannot be changed Each data set must be run in