WEAVE - Rijksuniversiteit Groningenweaveahm2017/talks/WAHM2017_NWalto… · includes management and QA activities • Level Q: quick-look analysis at telescope • Level 0 : basic

WEAVEScience Processing and Analysis (SPA):

Core Processing System (CPS)

Nicholas Walton, Mike Irwin, Jim Lewis, David Murphy, Luis Peralta & Clare Worley

1

2

Science Processing and Analysis(SPA)

• Core Processing System (CPS) L0/L1 and On-Island Pipeline• Infrastructure: basic elements, e.g. MEF FITS, language etc – this WP

includes management and QA activities• Level Q: quick-look analysis at telescope

• Level 0 : basic data quality control checks at CASU

• Level 1 : full image processing and spectral extraction

• Operational repository and survey progress monitoring

• Advanced Processing System (APS) L2 [Cristina Zurita talk]• Higher level stellar science pipeline

• Higher level galaxy and AGN science pipeline

• WEAVE Archive System (WAS) [Emilio Molinari talk]• Science archive systems

21 Nov 2017 Nic Walton - SPA/CPS @ WEAVE AHM17 - Groningen

3Nic Walton - CPS PDR@IAC 3

Science Processing and Analysis System


4

End-to-End Data Flow


QL, CPS and OR at CASU

STL

OCS

APS

WAS

CPS

5

WEAVE Data Flow (simplified)


OCS

SWG

6

WEAVE Data Flow (less simplified)Operational Rehearsal 3 example


David Murphy talk earlier

7

Science Requirements: SPA

• Design reference surveys developed• Define success criteria for each of the major surveys• Halo LR/HR, Disk/Bulge LR/HR, SCIP, Clusters, AGN, BAO

• Set the scientific requirements on the SPA• Cover parameters such as: line EW’s, redshifts, velocity

dispersions, radial velocities, abundances, Log g, Teff, etc• Translates to requirements on wavelength and flux

calibration and achieved s:n, and requirements on cross-talk and scattered light corrections and sky subtraction (L1)

• Scoped processing steps required (e.g. telluric correction) • Scoped survey speed and no. of spectra to achieve main

science goals


8Nic Walton - CPS PDR@IAC

WEAVE Science Data Flow

8

Stacking within an OB is done during the extraction phase. The 1-D spectra are then the individual spectra plus the stacked spectrumStacking of same object spectra from different OBs is done on a monthly basis.

Throughput spec for a peak of 45 Gbyte/day


9

Weave Automated Submission Platform (WASP)

• Simple submission and automated checking• In development, but many features are already

accessible• Designed to facilitate exchange of non-

spectroscopic data• Catalogues• XML field-fibre configuration files• OB files

• Verification checking, version controlling• Email alerts for users who submit data


10

WEAVE Operational Archivemodelled on the GES archive


survey progress monitoring à data reduction progress trend analysis à QC monitoring à planning feedback

11

Quick Look and L1 Pipelines

• QL composed of three components• QL_reduce: Does basic reduction• QL_harvest: Collects QC data from QL_reduce data

products• QL_display: GUI for display of results from QL_harvest

• QL components communicate through FITS files• QL_reduce and QL_harvest at advanced stage of

development and integration• QL_display GUI in development• L1 Pipeline at an advanced stage of development and

integration• Main processing in C routines wrapped in Python


12

Quick Look Pipelinequality assurance metrics

• Global QC outputs• missing/broken fibres• excessive stray/scattered/sky variation light (monitor with sky spectra)• unexpected losses in overall throughput• monitoring focus from “arcs”• flat field illumination level; uniformity• monitoring detector noise; pixel-pixel rms; cosmic ray hits• changes in cross-talk level

• Fibre level QC outputs• wavelength solution statistics • FWHM of “arc” lines/ sky emission lines• average flux in fibres from calibration observations• target flux -v- expected in wavebands [e.g. g,r,i] • continuum signal:to:noise measures• emission line fluxes


13

Simulationsvital input to test campaigns


Blue arm: 2 x 6kx6k CCD: here a simulated 1200s exposure

14

Spectra Extraction


K Giant: mg=17.06blue and red arms: simulated 1200s exposure

15

Spectra Extraction


B Dwarf: mg=17.65blue and red arms: simulated 1200s exposure

16

Spectra Extractionmany more examples from OpR2/2.5


M Supergiant: mg=16.0 blue and red arms: simulated 1200s exposure

17

SPA: Calibration for WEAVE• data characterisation:• biases, linearity, darks, fibre flats, arcs, twilight flats• detector (2D) flats, “salsa” flats & arcs à PSFs

• physical calibration:• wavelength calibration à arcs, skylines, telluric lines• relative flux calibration à fibre flats; spectral types • absolute flux calibration à Gaia BP/RP; photometry

• survey verification & external calibration:• Gaia benchmark stars à fairly infrequent if at all• RV standards à ditto (+ e.g M67)• standard fields e.g. Kepler, open clusters …


18Nic Walton - CPS PDR@IAC

Cal

ibra

tion

Dat

a Fl

ow

18

Pipeline composed of modules

Possible to reorder data flow if required


19

WEAVE arcs

191 Nov 2017 Nic Walton - SPA/CPS @ WEAVE AHM17 -Groningen

20

WEAVE arcs

1 Nov 2017 Nic Walton - SPA/CPS @ WEAVE AHM17 -Groningen

21

WEAVE - “salsa” arc line PSFs LR blue

wavelength

spatialdirection

1 Nov 2017 Nic Walton - SPA/CPS @ WEAVE AHM17 - Groningen 21

22

L0/L1 Issues under consideration

• extraction issues• “optimal” extraction in a close-packed MOS• characterise (PSFs) and remove crosstalk• handling cosmic-rays single-v-multiple exposures• dealing with scattered light and ghosts

• spectrum issues• telluric lines -> modelling MOLECFIT; reference stars• sky emission line removal• sky continuum and scattered light removal • resampling and pixel noise covariance

• quality measures• quality of various processing/extraction stages (rms, FWHM … )• estimates of s:n, flux levels continuum/emission lines• standard fields for inter-survey comparison• flux calibration


Current focus of development

23

Current Development Activities

• Use of simulations, WHT representative data (AF2 + WYFFOS) and high res data (Gaia-ESO survey)


AF2 Ursa Minor K giants: sky subtraction GES: HR21 sky (one month of data) – decomposition into eigen vectors

1.0%

2.1%

7.7%

72.4%

Av sky

1.4%

24



• Sky subtraction examples• Based on OpR2.5

simulations and pipeline extractions

• EO gal spectrum at r mag = 21.0• Here before sky

subtraction

25



• EO gal spectrum at r mag = 21.0• This shows after subtraction• Green: continuum-subtracted

version where residuals are normalised by the predicted (Poisson + r/o noise) pipeline spectrum noise relative to the average (median) noise• Demonstrates that the

leftover sky line residual features are close to the theoretical Poisson level • The red curve is the predicted

pipeline noise sigma.

26

WEAVE SPA/ CPS Summary

• CPS passed Preliminary (June 2014) and Final (April 2015) Design reviews

• Currently in implementation phase• Operational Rehearsals (involving input from the

science teams) • OpR2 Jun 17, OpR2.5 Aug 17, OpR3: Jun 18• Testing of full data flow

• Operational readiness review Q4/2018• Support to on-sky commissioning Q2/2019• OpR4 (on sky test of L2) Q3/2019

• Science operations from Q4/2019261 Nov 2017 Nic Walton - SPA/CPS @ WEAVE AHM17 - Groningen

OpR3 Talk: David Murphy

Documents

WEAVE - Rijksuniversiteit Groningenweaveahm2017/talks/WAHM2017_NWalto… · includes management and QA activities • Level Q: quick-look analysis at telescope • Level 0 : basic