XMM data reduction: part IISAS command-line analysis and scripting

Andy Read

Weds 14th Dec 2005 XMM data reduction: part II 2

Overview Command-line examples/help Why use command-line and scripts? Example scripts : SASmakecalev & SASprepareXMM Imaging scripts & spectral scripts Scripts within scripts Final remarks

Weds 14th Dec 2005 XMM data reduction: part II 3

Setting up the user environment To run on XROA system:> sas-setup-6.5.0 initialises software

> setenv SAS_ODF (path_to_ODF_directory)

e.g. /data/71/tro/xmm_data/0106860101/odf/

> setenv SAS_CCFPATH /data/rb2/sas_ccf

> cifbuild fullpath=yes >& cifbuild.log

builds ccf.cif – calibration index file

> setenv SAS_CCF (path_to_ccf.cif_file)

> odfingest odfdir=$SAS_ODF outdir=$SAS_ODF >& odfingest.log

builds ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing

(from Tim’s talk)Command line examples

Weds 14th Dec 2005 XMM data reduction: part II 4

Command line help

- Simple help – parameters and default values

Weds 14th Dec 2005 XMM data reduction: part II 5

- More - Parameters, parameter types, default values and ‘one-line’ help

Weds 14th Dec 2005 XMM data reduction: part II 6

- Full help – Links to SAS web pages

- Cut and paste link into browser

Weds 14th Dec 2005 XMM data reduction: part II 7

Why use scripts?

Useful scripting cookbook at http://starlink.rl.ac.uk/star/docs/sc4.htx/sc4.html

- Scripts are just a list of command-line calls/commands run sequentially- There are many powerful command line calls

- Can run intensive analysis on many datasets, sources etc.- Can be sure of doing exactly the same thing every time, e.g. running

exactly the same procedures on many sources/datasets- No need for interactive use

- can run overnight (no need to be there)- can run as cron jobs (no need to be even logged in)

- Often need to redo entire analysis, e.g.- New calibration files- New ODF files

- Mistake, change in e.g. source positions, detection threshold etc.

Weds 14th Dec 2005 XMM data reduction: part II 8

Example script : SASmakecalev

Weds 14th Dec 2005 XMM data reduction: part II 9

Executable t-shell scriptComment lines

Runs at low priority

On-screen help – appears whenever incorrect number of arguments is given (e.g. zero)

Number of arguments

If correct number of arguments is given (7) then input arguments (1-7) converted to parameters used in script, and we move into the main body of the script

Weds 14th Dec 2005 XMM data reduction: part II 10

Check that ODF exists (otherwise exit)

Get SAS version (for directory naming)

Create (if necessary) output directory, and (if allowed) enter

Weds 14th Dec 2005 XMM data reduction: part II 11

Check ODF

Set SAS_ODF environment variable

Check for ***SUM.SAS file

(Check for ***SUM.ASC file)

Create ccf.cif – calibration index file Set SAS_CCF environment variable

Create ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing

Output all SAS environment variables to screen – useful check

Weds 14th Dec 2005 XMM data reduction: part II 12

Run emproc (if required)

Run emchain (if required)

Run epproc (if required)

Run epchain (if required)

List created final calibrated event lists and exit

Weds 14th Dec 2005 XMM data reduction: part II 13

Run script

Weds 14th Dec 2005 XMM data reduction: part II 14

Example script : SASprepareXMM

Weds 14th Dec 2005 XMM data reduction: part II 15

Soft Proton flaring in an observation of a Galaxy Group

Weds 14th Dec 2005 XMM data reduction: part II 16

Executable t-shell scriptComment lines

Runs at low priority

On-screen help – appears whenever incorrect number of arguments is given (e.g. none)

Number of arguments

Weds 14th Dec 2005 XMM data reduction: part II 17

If correct number of arguments is given (9) then input arguments (1-9) converted to parameters used in script

If requested :

Check ODF

Check for ***SUM.SAS file

Create ***SUM.SAS file in ODF

Set SAS_ODF environment variable

Weds 14th Dec 2005 XMM data reduction: part II 18

If requested :

Create ccf.cif – calibration index file

Set SAS_CCF environment variable

Output all SAS environment variables to screen – useful check

Weds 14th Dec 2005 XMM data reduction: part II 19

Convert patcode parameter into an expression for evselect

Weds 14th Dec 2005 XMM data reduction: part II 20

MOS1 example (comment)

Weds 14th Dec 2005 XMM data reduction: part II 21

If $m1 does not equal 0, OK – continue to next line

If $m1 equals 0, ignore this whole section

Weds 14th Dec 2005 XMM data reduction: part II 22

If $m1 equals F, then we attempt to find the file

Else, the file name is given by the parameters

Weds 14th Dec 2005 XMM data reduction: part II 23

Remove a temporary file if it already exists

Weds 14th Dec 2005 XMM data reduction: part II 24

Long list all the MOS1 event files in the directory to temporary file

Weds 14th Dec 2005 XMM data reduction: part II 25

Count the number of entries (MOS1 event files) in the temporary file

Weds 14th Dec 2005 XMM data reduction: part II 26

If only one MOS1 event file, use this

Weds 14th Dec 2005 XMM data reduction: part II 27

If more than one MOS1 event file, try to find the largest

Weds 14th Dec 2005 XMM data reduction: part II 28

Initiate loop through ‘i’ (i=0)

Weds 14th Dec 2005 XMM data reduction: part II 29

loop through ‘i’ until ‘i’ equals number of event files, then end

Weds 14th Dec 2005 XMM data reduction: part II 30

Increment ‘i’ by +1 (i=1)

Weds 14th Dec 2005 XMM data reduction: part II 31

Get size of ‘i’th file :

Type the file, extract the top (head) ‘i’ lines, then extract the bottom (tail) line – file size is the 5th string in long listing

Weds 14th Dec 2005 XMM data reduction: part II 32

If size of ‘i’th file is larger than largest recorded, note ‘i’ (‘ibig’) and size as largest recorded

Weds 14th Dec 2005 XMM data reduction: part II 33

Loop round to next ‘i’

Weds 14th Dec 2005 XMM data reduction: part II 34

Get name of largest file (i.e. file corresponding to ‘ibig’) :

Type the file, extract the top (head) ‘ibig’ lines, then extract the bottom (tail) line – file name is the 9th string in long listing

Set $m1 to this filename

Weds 14th Dec 2005 XMM data reduction: part II 35

Create (evselect) lightcurve :

High-energy, single events, ‘good’ flags, 100s bins

Weds 14th Dec 2005 XMM data reduction: part II 36

Output help to screen: What is being done, what the user should do

Weds 14th Dec 2005 XMM data reduction: part II 37

Plot (dsplot) the light curve to the screen for user to examine

Weds 14th Dec 2005 XMM data reduction: part II 38

Request upper and lower threshold values from user

Weds 14th Dec 2005 XMM data reduction: part II 39

Use threshold values to create (tabgtigen) good times GTI file

Weds 14th Dec 2005 XMM data reduction: part II 40

Create (evselect) intermediate event file : filter on low-E threshold, pattern, flags, etc.

Weds 14th Dec 2005 XMM data reduction: part II 41

Create (evselect) final event file : filter intermediate file for good times using GTI file

Weds 14th Dec 2005 XMM data reduction: part II 42

Same for MOS2

Weds 14th Dec 2005 XMM data reduction: part II 43

Same for pn

Weds 14th Dec 2005 XMM data reduction: part II 44

Weds 14th Dec 2005 XMM data reduction: part II 45

Enter MOS1 thresholds

Same for MOS2

Same for pn

New filtered event files (r1***EV***)

smaller than originals

Weds 14th Dec 2005 XMM data reduction: part II 46

Already seen xmmselect

GUI - clean data, and produce science products

Only handles one dataset at a time

Essentially a handy GUI ‘wrapper’ to the SAS-task ‘evselect’

evselect is the command-line workhorse of SAS analysis tasks

Many scripts are command-line ‘wrappers’ of evselect and other SAS-tasks

Weds 14th Dec 2005 XMM data reduction: part II 47

Script to create images

- Essentially an evselect wrapper, plus other tasks (e.g. asmooth)

Weds 14th Dec 2005 XMM data reduction: part II 48

MOS1

MOS1 smoothed

MOS2

MOS2 smoothed

pn

pn smoothed

e.g. 300-2000eV sky (X/Y) 2′′ images (raw and smoothed) [MOS-sdtq, pn-sd]

Weds 14th Dec 2005 XMM data reduction: part II 49

Script to create spectra + ARF, RMF and do grouping etc.

Again, evselect wrapper, plus arfgen, rmfgen, grppha etc…

SAS-task equivalent especget now much improved

Weds 14th Dec 2005 XMM data reduction: part II 50

createspectrum to create BG spectrum

Note: can set parameters (here, co-ordinates from Tim’s region files) on the command lineRun script (using parameters)

Weds 14th Dec 2005 XMM data reduction: part II 51

createspectrum to create source spectrum, ARF, RMF and group final spectrum

Weds 14th Dec 2005 XMM data reduction: part II 52

Weds 14th Dec 2005 XMM data reduction: part II 53

Can put each step/script into a file (e.g. analysis) – A script to call other scripts

Can then run whole ‘analysis’:

> source analysis

Weds 14th Dec 2005 XMM data reduction: part II 54

Homework (2005) Extract NGC 1313 data (Oct 2000) from XSA Set up analysis environment Filter data for background flares Extract 0.3 – 10 keV images for all three

detectors Extract spectra of NGC 1313 X-1 Compare your results to

/data/71/tro/xmm_data/0106860101/work/

Homework (2006) – use scripts (and write a script of your scripts) to do all the analysis

Weds 14th Dec 2005 XMM data reduction: part II 55

Resources This talk (and others):

http://www.star.le.ac.uk/~sav2/stats/ The scripts (and others):

http://www.star.le.ac.uk/~amr30/Scripts/ XMM-Newton SAS web pages

Via http://xmm.vilspa.esa.es Particularly useful documentation:

HEASARC ABC guide SAS user’s guide

Experienced users!