JWST Pipeline/Analysis Tools

Perry GreenfieldScience Software Branch

Running the JWST pipeline

• Uses a python framework (stpipe) to connect processing steps into a pipeline.

• Framework handles:– Command line interface– Configuration files– Logging and error handling– Input and output

• One can run a pipeline– From the shell command line

• Directly or by invoking a configuration file

– Interactive Python prompt• Multiple ways to customize:

– Command line options to stop after a specified step– And restart at a step– Insert one’s own shell level command inside a pipeline (i.e., non-Python)– Define one’s own Python step and insert it, or modify existing ones

A (very) Simple Stepfrom jwst_lib.stpipe import Step, Cmdlinefrom jwst_lib import models

class FlatFieldStep(Step):

reference_file_types = [‘flat’]

def process(self, input):

with models.ImageModel(input) as im: self.flat = self.get_reference_file(im, ‘flat’).name flat_model = models.FlatModel(self.flat) result = flat_field.correct(im, flat_model)

return result

Executing a Step

> strun jwst_pipeline.FlatFieldStep myimage.fits

From the command line:

> strun flat_field.cfg myimage.fits

or

From Python:>>> from jwst_pipeline import FlatFieldStep>>> result = FlatFieldStep.call(‘myimage.fits’, config_file=‘flat_field.cfg’)

Configuration (.cfg) files use ini-file format and stpipe uses the ConfigObj library to parse them

A (very) Simple Pipelinefrom jwst_lib.stpipe import Pipelinefrom jwst_pipeline.bias import bias_stepfrom jwst_pipeline.dark import dark_stepfrom jwst_pipeline.reset import reset_step

class SimplePipeline(Pipeline):

step_defs = {“bias”: bias_step.SuperBiasStep, “dark”: dark_step.DarkCurrentStep, “reset”: reset_step.ResetStep}

def process(self, input): input = self.bias(input) input = self.dark(input) if input.meta.instrument.name == “MIRI”: input = self.reset(input) return input

Pipeline Config filename = "SloperPipeline"class = "jwst_pipeline.pipeline.SloperPipeline"save_calibrated_ramp = True

    [steps]      [[dq_init]]        config_file = dq_init.cfg      [[saturation]]        config_file = saturation.cfg      [[ipc]]        config_file = ipc.cfg      [[superbias]]        config_file = superbias.cfg      [[bias_drift]]        config_file = bias_drift.cfg      [[reset]]        config_file = reset.cfg      [[lastframe]]        config_file = lastframe.cfg      [[linearity]]        config_file = linearity.cfg      [[dark_current]]        config_file = dark_current.cfg      [[jump]]        config_file = jump.cfg      [[ramp_fit]]        config_file = ramp_fit.cfg

Step Config File

name = "jump"class = "jwst_pipeline.jump.JumpStep"rejection_threshold = 5.0do_yintercept = Falseyint_threshold = 1.0

Calibration Reference Data System

• Figures out what reference files to use for data set

• JWST pipelines automatically will download rules and any needed reference files to local computer when run remotely

• CRDS allows specifying different rules configuration than current operation system is using

• Rules can be locally customized to use user-provided reference files

Tools being developed for JWST

• Priorities driven by JWST Data Analysis Development Forum– Organized by Harry Ferguson and Susan Kassin

• Currently time series tools not high on the list of priorities– If you want to change that, you should get

involved (even if outside STScI)• Pipeline needs may impact data analysis

resources.

Sprint-oriented Framework• 1-d interactive spectral analysis • Image utilities (imarith, imstat, etc.)• Non-interactive fitting of 1-d spectra• Signal processing: filtering and interpolation• Tools for cubes I• Tools for cubes II• MOS visualization, data-quality inspection• 2-d visualization • Release preparations for imutils & imexam• Parameter files • PSF and PSF-kernel tools I• JWST data file registry, I/O, and associations • 1D spectral tools II (and initial release preparations) • Photometry on non-resampled data • Background estimation toolbox & tutorials (Spectroscopic and imaging) • Image & spectral geometry, registration & resampling I• Synthetic photometry • MOS tools II (and initial release preparations) • JWST Ramp inspection and analysis tools • PSF/LSF and PSF/LSF-kernel tools II • Visualization of resampled and non-resampled data • Spectral fitting on non-resampled data • Cube tools III (and initial release preparations)• Astrometry on non-resampled data • Improved source extraction tools (remaining SExtractor-like functionality, particularly overlapping source segmentation)

• Image & spectral geometry, registration & resampling II • 2D/3D spectral data combination • Tools to go from quick-look to publication-quality plots• Grism analysis and visualization tools • Optimal spectral extraction tools (optimal S/N extraction)• Coronagraphy tools to augment pipeline tools • NRM analysis tools• Time-series tools (e.g., transit analysis)

Astropy

• Community based astronomical core library for Python

• Started in 2011• >140 code contributors to date• http://astropy.org

Astropy Core

• Works with both Python 2 and 3• Regression tests• Documentation (sphinx)• Hosted on github• Goal is to foster better interoperability of

software tools written in Python

Astropy contents

• Units, quantities, physical constants• Tables• I/O (FITS, text tables, VOTables)• Time and Dates• Astronomical Coordinate Systems• Generalized World Coordinate Systems• Much more flexible than FITS WCS• Models and Fitting• Analytic functions• Convolution and filtering• Cosmological calculations• Astrostatistics• VO cone search

Affiliated packages

• Use astropy core libraries• Same python coverage, tests, documentation• Selected examples:– APLpy (image + plotting)– Ginga (image viewer)– Astroquery– Photutils– Sncosmo– Spectral-cube– WCSAxes