29
28 February 2008 COS Training Series IV ([email protected]) 1 COS Training Series IV: COS Post-observation M.E. Kaiser P. E. Hodge 28 February 2008 COS Spectrograph (Colorado, BASG)

COS Training Series IV: COS Post-observation

  • Upload
    janice

  • View
    34

  • Download
    0

Embed Size (px)

DESCRIPTION

COS Training Series IV: COS Post-observation. M.E. Kaiser P. E. Hodge 28 February 2008. COS Spectrograph (Colorado, BASG). COS Pipeline Verification. STScI : Mary Elizabeth Kaiser Phil Hodge Tony Keyes Tom Ake Cristina Oliveira Dave Sahnow Brittany Shaw Alessandra Aloisi Rosa Diaz - PowerPoint PPT Presentation

Citation preview

Page 1: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 1

COS Training Series

IV: COS Post-observation

M.E. Kaiser

P. E. Hodge

28 February 2008

COS Spectrograph (Colorado, BASG)

Page 2: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 2

COS Pipeline Verification

STScI :• Mary Elizabeth Kaiser • Phil Hodge• Tony Keyes• Tom Ake• Cristina Oliveira• Dave Sahnow• Brittany Shaw• Alessandra Aloisi• Rosa Diaz• Charles Proffitt

Colorado:• Jim Green• Stephane Beland• Steve Penton• Eric Burgh• Cyndi Froning• Steve Osterman• Eric Wilkinson

Page 3: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 3

COS Pipeline

OutlineOutline

• Pipeline DescriptionPipeline Description• PythonPython

• Data Formats and File StructureData Formats and File Structure

• Calibration Reference Files & TablesCalibration Reference Files & Tables• Reduced Data Products, Formats, etc. Reduced Data Products, Formats, etc. • CumCumulative Image & Pulse Height Maps

• Headers and KeywordsHeaders and Keywords

Page 4: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 4

CALCOS Foundation

CALCOS data requirements specified by AV03 (COS Calibration Requirements & Procedures)

- Outlines final data products for COS observations- Specifies the pre-flight calibration requirements- Specifies the reference files required- Specifies the calibration steps required to obtain the COS data product

Reminder:

Detector Wave Range(A)

Plate Scale(m/arcsec)

Active Area (mm)

Active Area(pixels)

Pixel Size(m)

Resel Size

(pixels)

XDL (FUV) L: 1230 - 2378M: 1132 - 1798

~265 2 @85 x 10

2 @~ 14160 x 400

6 x 24 6 x 10

MAMA (NUV) L: 1334 - 35604 osms, 3 stripes, Non-contiguous

~970 25.6 x 25.6 1024 x 1024 25 x 25 3 x 3

CALCOS: Python program to calibrate COS FUV & NUV TIME_TAG event lists & ACCUM images, producing a wavelength & flux calibrated spectrum.

Page 5: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 5

CALCOS Products

TIME-TAG data:TIME-TAG data:• Raw events table: time, x, y, pulse height amplitude (FUV)Raw events table: time, x, y, pulse height amplitude (FUV)

• Produce corrected events table: time, corrected x, corrected y, weight, dq, pha Produce corrected events table: time, corrected x, corrected y, weight, dq, pha

• Produce calibrated ACCUM image from corrected event listProduce calibrated ACCUM image from corrected event list

ACCUM data:ACCUM data:• Produce calibrated ACCUM image from raw ACCUM imageProduce calibrated ACCUM image from raw ACCUM image

• Produce flux-calibrated 1-D spectrum Produce flux-calibrated 1-D spectrum (wavelength calibrated)(wavelength calibrated)• Combine FP-POS or REPEATOBS spectraCombine FP-POS or REPEATOBS spectra

•2-d data: FITS images - single imset consisting of science, error, & data quality images

•1-d data and TIME-TAG lists: FITS tables - extracted spectra contain fluxes, wavelengths, errors, data quality, and other related quantities (e.g. background)

Page 6: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 6

Input Files and Products

All associated data (e.g., wavecals, FP-POS components, REPEATOBS, PHA) use separate files (unlike STIS) & are linked via an association table.

CALCOS files CALCOS files • Science spectrum (suffix designates detector)Science spectrum (suffix designates detector)

“_a” (or “_b” ): FUV detector segment A (or B); NUV: no “_” • Pulse-height histogram if FUV ACCUM (A and B)Pulse-height histogram if FUV ACCUM (A and B)• Target-acquisition image Target-acquisition image • Support file (contains engineering information)Support file (contains engineering information)• Association file controls calibration processing (science exposures, wavecals)Association file controls calibration processing (science exposures, wavecals)

Primary TIME-TAG file products: “rawtag” (raw photon list), “corrtag” (corrected photon list), “flt” (2-D corrected count-rate image), “x1d” (1-D flux-calibrated extracted spectrum).

Primary ACCUM file products: “rawimage” (raw image), “flt”, “x1d”

1-D spectrum for each FP-split location or repeatobs exposure; Sum of 1-D spectra1-D spectrum for each FP-split location or repeatobs exposure; Sum of 1-D spectra

Page 7: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 7

What is Python & Why should we use it?

CALCOS is Written in Python

• Extremely productive Extremely productive (Many powerful libraries available)(Many powerful libraries available)• Easy to learn and read & well-documented.Easy to learn and read & well-documented.• Available on just about any computing platform. Available on just about any computing platform. Portable with robust implementationPortable with robust implementation• Wide user and developer baseWide user and developer base• Supports both procedural and object-oriented coding.Supports both procedural and object-oriented coding.

• Unlike Tcl or Perl, scales well to larger programming projectsUnlike Tcl or Perl, scales well to larger programming projects• Extendable with C, C++, or Fortran.Extendable with C, C++, or Fortran.

• The Science Software Group successfully developed a new Command The Science Software Group successfully developed a new Command Language for IRAF using Python.Language for IRAF using Python.

• Programs more easily written, tested, and modified than C programs because of Programs more easily written, tested, and modified than C programs because of Python’s interpreted, dynamic and concise nature Python’s interpreted, dynamic and concise nature

• Programs easier to adapt to unanticipated cal requirements than C programs.Programs easier to adapt to unanticipated cal requirements than C programs.

• Better at bookkeeping and set-up than CBetter at bookkeeping and set-up than C

• Easier to diagnose Python errors than C errors (esp. memory and pointer errors).Easier to diagnose Python errors than C errors (esp. memory and pointer errors).

Python is an Open Source, freely available “scripting” language Python is an Open Source, freely available “scripting” language for astronomy data reduct’n & analysis for astronomy data reduct’n & analysis

Page 8: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 8

CALCOS Overview

COS ASSOCIATIONS: will generally follow NICMOS design

- associated obs are in separate datasets linked via an association table•Each spectroscop sci expos will have at least 1 wavecal assoc wavecals may be shared by diff sci exps•Only formal associat’ns in pipeline process & HAD: science/wavecal groupings, FP-POSs, & REPEATOBS

•User supplies association table nameUser supplies association table name•Read and interpret association table:Read and interpret association table:

• get a list of the input filesget a list of the input files• open one input file to get header keywords:open one input file to get header keywords:

• identify calibration steps to performidentify calibration steps to perform• obtain list of reference file namesobtain list of reference file names

• verify that all required files are presentverify that all required files are present

•Calibrate:Calibrate:• Process associated wavecalsProcess associated wavecals• determine wavecal offset from ref wavecaldetermine wavecal offset from ref wavecal• calibrate each input science filecalibrate each input science file

•Average the individual FP-split or repeatobsAverage the individual FP-split or repeatobs

Reference FilesReference Files ref_flat.fits ref_flat.fits flat field imageflat field image

ref_geo.fits ref_geo.fits geom distort’n filegeom distort’n file

ref_dead.fits ref_dead.fits livetime factr tablelivetime factr table

ref_bpix.fits ref_bpix.fits dqi tabledqi table

ref_brf.fits ref_brf.fits baseline (stim loc) tablebaseline (stim loc) table

ref_pha.fits ref_pha.fits pulse height tablepulse height table

ref_1dx.fits ref_1dx.fits spectrum loc tablespectrum loc table

ref_lamp.fits ref_lamp.fits template spectrm tbltemplate spectrm tbl

ref_phot.fits ref_phot.fits sensitivity ref tablesensitivity ref table

ref_dsp.fits ref_dsp.fits dspn relation poly tbldspn relation poly tbl ref_wcp.fits ref_wcp.fits wave params tablewave params table

Page 9: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 9

Outline of Calibration Steps

•TIME-TAG dataTIME-TAG data•Copy input data into output arrayCopy input data into output array•Add pseudo-random number to Add pseudo-random number to each X and Y position (FUV only)each X and Y position (FUV only)•Thermal correction (FUV only)Thermal correction (FUV only)•Geometric correction (FUV only)Geometric correction (FUV only)•Flat field correctionFlat field correction•Livetime correctionLivetime correction•Data quality assignmentData quality assignment•Filter by pulse height (FUV only)Filter by pulse height (FUV only)•Filter by timeFilter by time•Orbital & heliocentric Doppler correctionsOrbital & heliocentric Doppler corrections•Create 2-D ACCUM imagesCreate 2-D ACCUM images

Cal Step Description

BRSTCORR Search for bursts (FUV)

BADTCORR Flag bad time intervals (FUV)

PHACORR Filter by pulse height (FUV)

RANDCORR Dither to reduce aliasing (FUV)

TEMPCORR Temp-dependent distortion (FUV)

GEOCORR Correct nonlinear distortion (FUV)

DQICORR Includes DQ flag in DQ extension

DOPPCORR Correct for orbital Doppler shift

HELCORR Correct for heliocentric Doppler shift

FLATCORR Divide by flat field

DEADCORR Correct for deadtime

WAVECORR Correct for OSM uncertainty

X1DCORR Extract 1-D spectrum

BACKCORR Subtract background from spectrum

FLUXCORR Convert count rate to flux units

Page 10: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 10

Outline of Calibration Steps (cont.)

• ACCUM dataACCUM data• Verify good pulse-height distribution (FUV only)Verify good pulse-height distribution (FUV only)• Find stim pulses and compute thermal distortion coefficients (FUV only)Find stim pulses and compute thermal distortion coefficients (FUV only)• Convert ACCUM to temporary coordinate list (FUV only)Convert ACCUM to temporary coordinate list (FUV only)• Add pseudo-random number to each X and Y position (FUV only)Add pseudo-random number to each X and Y position (FUV only)• Thermal correction (FUV only)Thermal correction (FUV only)• Geometric correction (FUV only)Geometric correction (FUV only)• Data quality assignmentData quality assignment• Compute orbital and heliocentric Doppler shiftsCompute orbital and heliocentric Doppler shifts• Create 2-D temporary count rate ACCUM imageCreate 2-D temporary count rate ACCUM image• Convolve flat field with Doppler shift and divide into count rate imageConvolve flat field with Doppler shift and divide into count rate image• Livetime correctionLivetime correction• Create 2-D effective count rate imageCreate 2-D effective count rate image

Page 11: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 11

Outline of Calibration Steps (cont.)

• Steps common to TIME-TAG and ACCUMSteps common to TIME-TAG and ACCUM• Extract 1-D spectrum:Extract 1-D spectrum:

• Compute net count rateCompute net count rate• Compute error estimateCompute error estimate• Compute maximum and average data qualityCompute maximum and average data quality• Compute wavelengthsCompute wavelengths• Compute flux from count rateCompute flux from count rate

• Combine FP-split or repeatobsCombine FP-split or repeatobs

Page 12: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 12

Running CALCOS: An Example Input

Running CALCOS (2 options)• Run calcos.py from the UNIX command line• Run from Pyraf (Python w/ IRAF imported) pyraf import calcos stsdas calcos.calcos (“file.fits”)

FUV spectrum: - source: external Pt/Ne lamp - exposure sequence: 3 FP-POS positions - tagflash exposures through WCA using internal Pt/Ne lamp

Science Exposure: External Pt/Ne lamp (PSA aperture)

Tagflash Wavecal Exposure: Internal Pt/Ne lamp (WCA aperture)

Page 13: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 13

Running CALCOS

Page 14: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 14

The Details: Thermal Correction

Electronic stim pulse signals are used to correct the FUV XDL data for thermal drift (The x & y positions of detected photon events are obtained from analog electronics --> susceptible to thermal changes.)

• Reference Stim positions used to correct the data to the reference frame• Must be done at the outset before any reference files are appliedNUV MAMA has physical pixels. No stim pulse capability exists or required.

2 stim pulsesfor each detector Segment, locatedat the lower left& upper right

Thermal correction done after screening for bursts (outliers in bkg_counts)

m

m

Page 15: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 15

The Details: Geometric Correction

Corrects for optical distortions at the detector image • FUV geometric correction reference file has been delivered by IDT dx is multiplied by 10 in the geometric distortion map below for segments A and B to illustrate the distortion characteristics• NUV geometric correction need under evaluation by IDT - effect is small

Page 16: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 16

The Details: Flat Field Correction

Top panel: flat field taken pre-TV no grid wiresMiddle panel: 2003 TV flat field note grid wiresLower panel: same as middle panel but with zoomed y-axis note blemish

Corrects for pixel-pixel variations • FUV flat field under evaluation - in-flight validation planned

Grid wires: A negatively charged, primarily vertical grid that increases detector efficiency by inhibiting electrons from scattering out of the MCP pores.

Page 17: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 17

The Details: Flat Field Correction

Full flat field

Corrects for pixel-pixel variations • NUV flat field delivered by IDT

zoomed region - note hex structure

Page 18: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 18

The Details: Data Quality Assignment

Note location & flux level depression for:• grid wire shadows• dead spots• blemishes

Identifies detector regions of less than optimal performance • Detector dead-zones, hot spots, grid wire shadows, blemishes• Data drop-outs

Page 19: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 19

The Result

FUV spectrum: source: external Pt/Ne lamp exposure sequence: 3 FP-POS positions (PSA aperture) tagflash exposures through WCA aperture using internal Pt/Ne lamp

Page 20: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 20

Lifetime Charge Extraction Detector lifetime is a function of the charge extracted.

A cumulative signal image of the FUV & NUV detectors will be maintained to monitor the extracted charge.

FUV detector has 5 default positions to mitigate effects of charge extraction (gain sag). (non-concurrent)

FUSE cumulative signal imageLeft: Lyman Beta regionRight: Background region

Light stripes are grid wire shadows

Page 21: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 21

FUV Pulse Height Maps

The Pulse Height of the electron cloud can be used to• discriminate between science and spurious sources • monitor the lifetime of the microchannel plate (MCP) - identify regions with large charge extractions (gain sag)

Red histogram: Gain sag of the FUSE detector

is shown for the Lyman Beta region - which has had large charge extraction due to both airglow and science targets.

Black histrogram: Background region with minimal

gain sag due to charge extraction.

Page 22: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 22

Data File Naming Conventions

Product Suffix Type Contents ---------------------------------------------------------------------------------------------------------------

•Uncalibrated _rawimage image Raw ACCUM counts image • _rawtag table Raw TIME-TAG event list • _asn table Association file • _trl table Trailer file • _jit table • _jif image • _pha table Pulse-height (FUV ACCUM only)---------------------------------------------------------------------------------------------------------------

•Calibrated _corrtag table Calibrated TIME-TAG event list • _flt image Flat-fielded, corrected counts • _fltsum image Summed flat-fielded corrected-counts • _x1d table 1-D extracted spectra • _x1dsum table Combined 1-D extracted spectrum • _trl table Trailer file (output)

• _spt image Support file (header) •----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Wavecal expos produce same calibrated files as science, except no flux calibration applied to the _x1* files. Dark, flat, & image exposures produce no _x1* files. Acquisition exposures produce no calibrated products.

Page 23: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 23

Data Processing Steps

Exposure typeExposure type

raw tag

raw tag

rawunag

rawunagee

corrtagcorrtag

phapha

countscounts

fltflt

fltsumfltsum

x1dx1d

x1dsumx1dsum

asnasn

trltrl

sptspt

TIME-TAG No FP FUVTIME-TAG No FP FUVNUVNUV

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

TIME-TAG FP FUVTIME-TAG FP FUVNUVNUV

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

ACCUM No FP FUVACCUM No FP FUVNUVNUV

XXXX

XX XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

ACCUM repeatobs No FP ACCUM repeatobs No FP FUVFUVNUVNUV

XXXX

XX XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

ACCUM FP FUVACCUM FP FUVNUVNUV

XXXX

XX XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

XXXX

IMAGE TIME-TAG IMAGE TIME-TAG XX XX XX XX XX XX XX

IMAGE ACCUMIMAGE ACCUM XX XX XX XX XX XX

IMAGE ACCUM repeatobsIMAGE ACCUM repeatobs XX XX XX XX XX XX XX

ACQ and PEAKUPsACQ and PEAKUPs XX XX XX

WAVECALWAVECAL XX XX XX XX XX XX XX XX

FLAT and DARKFLAT and DARK XX XX XX XX XX XX

Page 24: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 24

COS Typical File Sizes

Exposure Exposure typetype

PODPOD

(Mb)(Mb)

rawtagrawtagrawimagerawimage

(Mb)(Mb)

corrtagcorrtag

(Mb)(Mb)

fltflt

(Mb)(Mb)

countscounts

(Mb)(Mb)

OPUSOPUS(Mb)(Mb)

intointoHDAHDA(Mb)(Mb)

TIME-TAGTIME-TAGFUVFUV 1818 = = 4545 = =

115115

160x2160x2 160x2160x2 818818 6363

TIME-TAGTIME-TAGNUVNUV 1818 4040 110110 1010 1010 188188 5858

ACCUMACCUMFUVFUV 1616 160x2160x2 ------ 160x2160x2 160x2160x2 976976 336336

ACCUMACCUMNUVNUV 22 1010 ------ 1010 1010 3232 1212

WAVECALWAVECAL < 18< 18 < 45< 45 < <

115115

160x2160x2 160x2160x2 640-640-

818818

< 63< 63

Page 25: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 25

COS Configuration Keywords

• Keyword: some possible values Keyword: some possible values – OBSTYPE: SPECTROSCOPIC, IMAGING OBSTYPE: SPECTROSCOPIC, IMAGING – OBSMODE: ACCUM, TIME-TAGOBSMODE: ACCUM, TIME-TAG– EXPTYPE:EXPTYPE: ACQ,ACQ, PEAKUP/XDISP, PEAKUP/DISP, SCI,PEAKUP/XDISP, PEAKUP/DISP, SCI, WAVE,WAVE,

FLAT, DARK,FLAT, DARK, PHAPHA– DETECTOR: FUV , NUVDETECTOR: FUV , NUV– SEGMENT: FUVA, FUVBSEGMENT: FUVA, FUVB– OPT_ELEM:OPT_ELEM: G130M, G160M, G185M, G225M, G285M, G140L,G130M, G160M, G185M, G225M, G285M, G140L,

G230L, MIRRORA, MIRRORBG230L, MIRRORA, MIRRORB– CENWAVE:CENWAVE: 1298, 1309, 1320, etc.1298, 1309, 1320, etc.– APERTURE :APERTURE : PSA, BOA, WCA, FCAPSA, BOA, WCA, FCA

(Indicates a COS unique keyword or value)(Indicates a COS unique keyword or value)

Page 26: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 26

Keyword specification

– keyword namekeyword name– default valuedefault value– possible valuespossible values– unitsunits– datatypedatatype

– short comment for headershort comment for header– long descriptionlong description– header positionheader position– DADS tableDADS table– keyword sourcekeyword source

• The following information is being generated by STScI Science Instrument team and DST for all COS specific keywords.

Page 27: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 27

Association Table

• Column names MEMNAME, MEMTYPE, MEMPRSNTColumn names MEMNAME, MEMTYPE, MEMPRSNT• MEMNAME is the rootname for a file or set of filesMEMNAME is the rootname for a file or set of files• MEMTYPE distinguishes science from wavecal, and exposure from MEMTYPE distinguishes science from wavecal, and exposure from

productproduct• MEMPRSNT = true implies that input files with the specified rootname MEMPRSNT = true implies that input files with the specified rootname

existexist

Page 28: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 28

COS Associations

• Only associations required for data processing will be Only associations required for data processing will be constructed by TRANSconstructed by TRANS

• There will be 1 product, and only 1 product, for each There will be 1 product, and only 1 product, for each association.association.

• COS associations will consist of COS associations will consist of – FP-split (always 4) or repeat obs. exposuresFP-split (always 4) or repeat obs. exposures– WavecalsWavecals

• Wavecals executed after each grating, central wavelength, mirror, Wavecals executed after each grating, central wavelength, mirror, or aperture change; and at least once each orbit.or aperture change; and at least once each orbit.

• Wavecals can be shared between associationsWavecals can be shared between associations• An association table will contain all information about An association table will contain all information about

associated datasetassociated dataset• Observations will complete Generic Conversion as individual Observations will complete Generic Conversion as individual

exposures, be held by a Data Collector until all member exposures, be held by a Data Collector until all member exposures are present, and the product will be created by exposures are present, and the product will be created by CALCOS.CALCOS.

Page 29: COS Training Series IV:  COS Post-observation

28 February 2008COS Training Series IV ([email protected]) 29

NUV Science Data Format

CC BB AA CC BB AA

PtNePtNeWavecalWavecal

ExternalExternalScienceScience

NUVNUVMAMAMAMA