OONE OF THE GOALS for theVLT Data Flow System(Quinn 1996) is to providecalibrated data productsready for science analysis

for the instrument modes used most heavilyduring service observing. Achieving thisgoal depends on a complex interactionbetween the science goals of the astronomer,the operational model of the observatory,and the implementation of instrument cali-bration pipelines. The VLT Science Policydocument establishes the following guide-lines for instrument calibration and dataproducts:

ESO will execute and maintain a cali-bration plan for all VLT/VLTI instru-ments. The calibration data resultingfrom this plan shall be made available tothe ESO community. ESO will also usethese data to monitor the long-term evo-lution of instruments and to producedata products. These data products willresult from pipeline processing using theVLT/VLTI Data Flow System. Dataproducts will consist of data with theinstrumental signature removed as wellas data calibrated into physical units.The accuracy of the instrumentation andphysical unit calibration will be moni-tored and maintained by ESO.In order to allow the ESO community toreproduce and modify the output ofinstrument calibration pipelines, ESOwill make the pipeline reduction recipesand code available to the ESO commu-nity for all supported instrument modes.Over time, the number of supportedmodes for each instrument will beincreased to ensure that the most active-ly used modes have calibrated dataproducts available.The accuracy of ESO pipelines shouldbe such to satisfy a major fraction of thescientific needs of the users and ESOwill attempt to increase their accuracyover time following the guidance of itscommunity and in-house scientists.

In this article, the current status ofVLT/VLTI science products delivered byinstrument calibration pipelines is presentedwith an emphasis towards science readiness,i.e. whether the currently delivered scienceproducts can be used for science analysiswithout further processing. For the mostheavily used instrument modes over the lasttwo years, we believe the answer is “yes”,depending on the science objective and therequired calibration precision. In the end,however, the usefulness and quality of anygiven science data product is in the eye ofthe beholder, i.e. it strongly depends on thetechnical needs and science objectives of theend-user astronomer or data analysis team.We hope this article will not only provideinformation to the ESO user community butalso elicit feedback to help us improve ourservice.

In the following article, VLT is used as a

metaphor for all science instruments operat-ed on Cerro Paranal. Messenger articles byComerón et al. (2003) and Mathys (2004)provide recent overviews of the VLT serviceobserving operational model. Discussion ofinstrument calibration pipelines and scienceproducts for La Silla instruments are outsidethe scope of this article.

IINSTRUMENTNSTRUMENT CALIBRACALIBRATIONTION PIPELINESPIPELINESOperational fundamentalsOperational fundamentalsAll VLT science observations are executedusing Observation Blocks (OBs) that are inturn composed of observation templates.Automatic instrument calibration pipelineshave been implemented for the most heavilyused instrument modes. These pipelines pro-duce master calibration products used tomonitor instantaneous and long-term instru-ment performance. Using the master calibra-tion products, the pipelines also process raw

The Messenger 1182

VLVLT ST SCIENCECIENCE PPRODUCTSRODUCTS PPRODUCEDRODUCED

BYBY PPIPELINESIPELINES: A S: A STTAATUSTUS RREPOREPORTT

USING THE DATA FLOW SYSTEM, ESO IS PRODUCING CALIBRATED SCIENCE DATA PRODUCTS FOR ALL VLT AND

VLTI INSTRUMENTS. FOR MANY USERS, THESE PRODUCTS CAN BE USED DIRECTLY FOR SCIENTIFIC ANALYSIS

WITHOUT FURTHER PROCESSING. IN THE END, THE SCIENTIFIC USEFULNESS OF THESE PRODUCTS DEPENDS

STRONGLY ON THE NEEDS OF THE INDIVIDUAL INVESTIGATORS.

DDAAVIDVID R. SR. SILILVVAA ANDAND MMICHÈLEICHÈLE PPERONERON (ESO)

Figure 1: Science integration hours per VLT instrument, Period 70 – 73. The FORS1, FORS2,ISAAC, and UVES instruments were operational during this entire two-year period. VIMOS andGIRAFFE operations did not start until February 2003. Exact dates are 2002 October 1 to 2004September 30. Guaranteed Time Observations (GTO) data have been excluded.

Tele

scop

es a

nd In

stru

men

tatio

n

science data into science products. Data areprocessed on a template-by-template basis.Data from different templates (or differentOBs) are not processed together. In general,only so-called standard instrument configu-rations are supported.

The automatic production of scienceproducts using instrument calibrationpipelines requires an electro-mechanicallystable instrument (including detector), theacquisition of sufficient calibration data, andthe implementation of proper software algo-rithms, where proper is defined as an algo-rithm that does not degrade scientific useful-ness. How is ESO doing in these areas?

• Stability: at present, most of our instru-ments are stable (FORS1, FORS2, UVES,GIRAFFE), some are semi-stable (ISAAC,NACO) and one is (mechanically) unstable(VIMOS). Despite various mechanical prob-lems, high quality VIMOS science productsare often produced in automatic mode if suf-ficient night-time calibrations are available.

• Calibration data: calibration plansexist for all instruments and calibration dataare obtained on a regular basis. These dataare used to monitor instrument health andprocess science data. The target photometriccalibration accuracy is 5–10% in magnitudefor imaging and in the relative flux calibra-tion for spectroscopy. The achieved photo-metric accuracy is often much better in bothcases. Nevertheless, the achieved accuracymay not be sufficient for all science pro-grammes.

• Proper algorithms: for imaging,essentially no problems exist today. Forspectroscopy, the situation is more complex.While UVES pipeline results are universallyproclaimed as science-ready for most users,this is not necessarily true for other instru-ments.

Instrument usage for science observa-

tions during the last two years is summa-rized in Figure 1. During this interval, morescience data was produced in Service Modethan in Visitor Mode for all instruments.This information is broken down by instru-ment mode in Figure 2. Modes withoutpipeline support are indicated by verticalstripes. The vast majority of science dataobtained during this period came frominstrument modes with pipeline support. Themost glaring exception is FORS2-MXU.Unfortunately, it was necessary in the lasttwo years to shift pipeline developmentresources originally scheduled for FORS2-MXU to VIMOS.

Recipe development and availabilityRecipe development and availabilityAt the core of each instrument calibrationpipeline are a set of instrument-specific soft-ware data processing engines known aspipeline recipes. For FORS1, FORS2,VIMOS, GIRAFFE, and MIDI, the instru-ment consortia that built and commissionedthose instruments developed the originalpipeline recipes. Since their delivery, theserecipes have been maintained by ESO andmodified to meet ESO operational goals.Recipes for the other currently operationalinstruments (VIMOS-IFU, ISAAC, UVES,and NACO) were developed within ESOwhere they are still actively maintained andextended as needed.

Recipes are released to the communityas soon as possible after instrument opera-tions begin, but not before their performancehas been verified using data from the as-built, as-operated instrument in all standardconfigurations. In the past, this verificationprocess has taken as long as a year but hasrecently accelerated. Recipe upgrades arereleased as needed. For information abouthow to download available recipes, seehttp://www.eso.org/pipelines/.

All released pipeline recipes come withtechnical documentation – some good andsome not so good. More useful forastronomers are the science reduction dis-cussions maintained for each of the VLTinstruments on the ESO QC Web pages(http://www.eso.org/qc). But astronomer-ori-ented cookbooks are really needed, prefer-ably written by experienced astronomersfamiliar with the pipeline recipes. Such aguidebook exists for ISAAC. ESO is tryingto find ways to facilitate the creation of suchcookbooks for other instruments, but unfor-tunately no direct resources for this activityexist right now.

Recent pipeline recipes (VIMOS,GIRAFFE) were built partially or in wholeusing the Common Pipeline Library (CPL)developed by ESO. All future recipes will bebased on CPL (http://www.eso.org/cpl). Thishas two benefits for end-user astronomers.First, all CPL-based recipes will have com-mon Unix shell behavior, creating a moreunified and stable user interface. Suchbehavior allows rapid development of user-specific processing scripts using the Unixshell or such common scripting languages asPerl or Python. The tools dfits and fitsort(see http://archive.eso.org/saft) can be used toextract and organize FITS header informa-tion within such scripts. Second, CPL-basedrecipes can be executed within the Gasganodata organization and management tooldeveloped by ESO (available fromhttp://www.eso.org/gasgano/). Gasgano pro-vides a user-friendly Java-based environ-ment for data organization and processingwithin a single graphical user interface.

At ESO, the pipeline recipes are embed-ded in an infrastructure that automaticallyclassifies raw data frames, associates themwith appropriate master calibration prod-ucts, and then executes the appropriate

© ESO - December 2004 3

Figure 2: Science integrationtime per VLT instrument mode,

Period 70 – 73. Modes withactive instrument calibration

pipelines are shown with solidcolors, while modes without

pipelines are indicated by verticalstripes. Exact dates are 2002

October 1 to 2004 September 30.Guaranteed Time Observations

(GTO) data have been excluded.

pipeline procedure. This infrastructure istuned to the ESO operational environmentand cannot be easily exported to externalsites. As our pipeline infrastructure evolvesthis situation may change, and ESO willexplore ways of providing more automaticdata organization tools to the community.

SSCIENCECIENCE PRODUCTSPRODUCTSCreation of science productsCreation of science productsFor instrument modes with pipeline support(see Figure 2), science products can be pro-duced automatically on Paranal, if appropri-ate calibration data are available.Immediately after an OB template is com-pleted, the Paranal-based pipelines attemptto create science products using pre-loadedcalibration data. These pre-loaded data arenot necessarily the best possible and are typ-ically updated only at 1 – 3 month intervals.On a best effort basis, Paranal ScienceOperations can sometimes pre-load moreappropriate calibration data for specific pro-grammes. If appropriate calibration data arenot available (e.g. mask-specific flats formulti-object spectroscopy reductionsobtained in morning after science observa-tions or non-standard instrument configura-tion used), no science products are created.If created, science products are made avail-able to visiting astronomers at the appropri-ate off-line workstation. Visitors also havethe option of using the off-line workstationsto process (or re-process) their science data.

In Garching, all Service Mode sciencedata are processed (or re-processed) usingthe same pipeline recipes, although some-times with different control parameters (e.g.average vs. median combination, block-average vs. optimal spectral extraction).Furthermore, the best available and mostappropriate calibration data are used, even ifthey were obtained after the science obser-vation. Service Mode science products aredelivered in due time to the principal inves-tigator. At this time, Visitor Mode data arenot re-processed and delivered due tostaffing limitations, but such a service maybe provided in the future. In general, thequality of the science products produced inGarching are higher than the products pro-duced on Paranal because better calibrationdata are used and the pipelines are moreclosely monitored by ESO staff for outputquality.

Science data product quality: what is it?Science data product quality: what is it?The VLT Science Policy guidelines indicatethat pipelines should remove the instrumentsignature and calibrate the data into physicalunits to an accuracy that is satisfactory for amajor fraction of users. What is the currentgeneral situation?

For imaging, the current goal is to pro-duce photometrically flat images with astro-metrically accurate World CoordinateSystem (WCS) coefficients and a nightly

zeropoint with 5 – 10% accuracy if a stan-dard filter was used. Zeropoint accuracy islimited by the number of standard fieldobservations per night (typically, 2). Nightlyreports from the Paranal Astronomical SiteMonitor (available from http://archive.eso.org/asm/ambient-server) can be used to constrainwhether or not the night was photometric.Color terms and extinction coefficients aremeasured several times a year but not in allfilters. Values are published on the ESOQuality Control Web pages (http://www.eso.org/qc). If a non-standard filter is used orhigher photometric transformation accuracyis required, the user must obtain additionalcalibration observations. However, ESOleaves the responsibility for building objectcatalogs in the hands of the user. So, strictlyspeaking, the objects on delivered images

have not been calibrated into physical fluxunits – roaming over an image with a WCSaware image browser will show the celestialcoordinates of each object but not their mag-nitudes. But ESO provides the informationnecessary to convert instrumental flux meas-urements into physical units once (projectspecific) object identification and extractionis complete.

In the most general sense, the currentgoal for spectroscopy is to produce back-ground corrected one or two-dimensionalspectra with linear scales in the spatial anddispersion direction and their true relativecontinuum shapes. WCS coefficients shouldbe set to relevant wavelength units (e.g.Ångstroms or microns) in the dispersiondirection and arc seconds in the spatialdirection. In the future, the spatial WCS

The Messenger 1184

Table 1: Summary of Science Product Status By Instrument Mode

Instrument Mode ScienceReady?

Product Remarks

FORS1/2 IMG Yes No jitter combination

FORS1/2 LSS Maybe

Polynomial correction for distortion and disper-sion too low (?), no correction for slit responseor sensitivity, not all grisms supported.Spectrophotometric standards observed butnot applied by pipeline.

FORS1/2 MOS Maybe Same comment as FORS/LSS.

FORS1 PMOS/IPOL No Not supported yet

FORS2 MXU No Not supported yet

FORS2 HIT No Not supported yet

GIRAFFE All MaybeScience pipeline still under development.Products produced with flat-field correction oroptimal extraction. Not all setups supported.

ISAAC SWI/LWI Yes For data before April 2003, image stacksshould be reprocessed with the latest recipe.

ISAAC SWS/LWS Maybe

Polynomial correction for distortion and disper-sion too low (?), no correction for slit responseor sensitivity. Telluric standards can be used todetermine sensitivity function.

ISAAC SW-POL No Not supported yet

NACO IMG Yes

NACO Other No No pipeline support

MIDI HIGH_SENS Yes Dispersed modes not yet supported. SeeBallester et al. (2004)

UVES ECH Yes

UVES SLICER Yes

UVES FIBRE YesFibre-to-fibre correction but no sensitivity cor-rection. No automatic background correction,since no dedicated sky monitoring fibre.

VIMOS IMG Yes No jitter combination.

VIMOS MOS Maybe

Flat-field not applied, pending further investi-gation of internal reflections and fringing. Nosensitivity correction. Dispersion correctionquality variable due to problems related tomask insertion and mechanical flexure. If pres-ent, more than one object extracted per slit.

VIMOS IFU Maybe See VIMOS-MOS comments as well as Izzo etal. (2004)

coordinates may be updated to support truecelestial coordinates. The flux units are rela-tive counts. If users require a conversion toabsolute flux units (e.g. ergs s–1 cm–2 Å–1),they must request additional observations offlux standards with appropriate slit sizes andat appropriate time intervals. For all instru-ments that produce two-dimensional spectra,ESO provides the processed 2D spectra andan extracted 1D spectrum for the brightestobject in each slit. If spectra are extracted,signal-to-noise should be optimized. Ofcourse, details of spectral extraction (or evendeciding what to extract) are usually sci-ence-driven, so many users may choose tore-extract their objects.

Nevertheless, the usefulness of anygiven science product for a specific pro-gramme may be limited by the quality of theavailable calibration data and the choice ofparameters used when running the recipes(e.g. type of spectral extraction, order ofpolynomial used for spatial correction). Bydesign, these automatic pipelines processscience type of similar type in a similar wayin all cases. In the end, astronomers need todecide for themselves and for every projectif the science products produced by ESO areright for them or if they need to reprocesstheir raw data.

Specific goalsSpecific goalsOver time, the following goals for scienceproduct creation have emerged:

• All instrument modes: as appropriatefor the specific instrument and detector, cor-rections for bias, dark, small scale pixel-to-pixel sensitivity variations and large scaleillumination variations are applied. WCScoefficients are updated appropriately forinstrument mode and situation. Instrumentspecific quality control parameters are pro-vided.

• Imaging (IMG): all frames acquiredthrough the same filter within a single OBtemplate are combined. If the image group isa jitter stack, they are co-aligned and co-added using average with some form of out-lier rejection; a byproduct weight map isproduced. For mosaic imagers (WFI,OmegaCam), the entire field-of-view isplaced on a common astrometric grid; thisrequires geometric rectification of imagesfrom the individual detectors. For imagingpolarimetry, images are grouped by relevantoptical element, and processed; but Stokesparameters are not computed. Final product:single corrected image. Instruments:FORS1/FORS2 IMG, ISAAC SWI/LWI,NACO, VIMOS IMG, WFI, OmegaCam.

• Spectroscopy (general): correctionsare applied for optical distortion, wave-length dispersion, slit response function(only long-slit spectroscopy), fibre responsefunction (fibre-fed instruments only), spec-tral cross-talk (for densely packed spectra,e.g. IFUs), and relative sensitivity (flux). No

corrections are made for fringing, atmos-pheric differential refraction, telluric absorp-tion, or airmass. For instrument modes withslits, the background is subtracted; thederived background spectrum used for cor-rection is provided to the user. The brightest

object (or objects in some cases) within eachslit are extracted into 1D spectra.

• Long-slit spectra (LSS) and echelle(ECH): Final product: two-dimensional(2D) frame with linear spatial and dispersionscales, extracted 1D spectrum of brightest

© ESO - December 2004 5Silva D. & Peron M., VLT Science Products produced by Pipelines

Figure 3: ExampleFORS MOS Science

Product. From FORS1MOS using 150I grism

at λcen = 7100 Å. Noticealignment of skylines.

Courtesy of R. Mignani.

Figure 4: ExampleVIMOS IMGScience Product.B-band image ofthe center of Cen A(NGC 5128) – quad-rant 3 only.Courtesy of P.Sartoretti.

object, and various related spectra depend-ing on instrument (e.g. background spec-trum). Instruments: FORS1, FORS2,ISAAC SW/LW, UVES (2D spectra byrequest only).

• Multi-object spectra (MOS) withslits: Final product: corrected two-dimen-sional slit images with linear spatial and dis-persion scales, aligned in wavelength space,and stacked into one plane of a FITS image.Also: 1D spectra for brightest objects in slitswith various supporting spectra as appropri-ate for instrument (e.g. background spectra).Instruments: FORS1/FORS2 MOS, FORS2MXU, VIMOS MOS.

• Multi-object spectra (MOS) withfibres: Final product: extracted, flux-cor-rected one-dimensional spectra with lineardispersion scales, aligned in wavelengthspace, and stacked into one plane of a FITSimage. Various supporting spectra (e.g. errorvector). Instruments: GIRAFFE MEDUSA,UVES FIBRE.

• Integral-field spectroscopy (IFU):Final product: same as fibre MOS plusreconstructed (broad-band) image correctedfor fibre efficiencies. Instruments: VIMOSIFU, GIRAFFE ARGUS, GIRAFFE IFU,SINFONI.

At this time, none of our pipelines makespecific corrections for fringing, an opticalphenomena caused by interference betweenincident light and internal reflectionsbetween the thin layers in a detector. Internaldiscussions about this complex topic contin-ue but are beyond the scope of the currentarticle. Likewise, no corrections for telluricabsorption lines in spectral regions beyond7000 Å are made, although spectra of tel-

luric absorption standard stars are obtainedas part of the standard ISAAC and NACOcalibration plans and delivered to users.

The goal of VLT interferometry is todeliver calibrated visibilities. For furtherdetails, see Ballester et al. (2004).

StatusStatusScience products created by pipelines duringthe first years of Paranal operations had vari-able usefulness for science analysis. WithinESO, higher priority was given initially toproducing quick-look data products neces-sary to confirm field acquisition and qualitycontrol parameters to monitor instrumenthealth, rather than science-ready data prod-ucts. The number one priority for ESO was(and remains) to assure that the raw sciencedata stream is fundamentally sound and pro-duced in a controlled manner. Nevertheless,the quality of pipeline-produced scienceproducts has steadily improved and today isquite good in many cases.

Table 1 summarizes current pipelineproduct status. In the Remarks column, devi-ations from the goals listed above aredescribed. For a brief summary with moredetails, see Pipeline Status Web page(http://www.eso.org/qc/pipeline-status.html). Forcomplete details, see the instrument-specificsections of the Quality Control Group Website (http://www.eso.org/qc/). In particular,readers are urged to read the KnownProblems information provided for eachinstrument.

A few example science products areshown in Figures 3 – 5. Examples for allsupported instrument modes can be found onthe QC Group Web pages.

Future DevelopmentFuture DevelopmentAlthough the current pipeline recipesreceive maintenance as time permits, imple-mentation of instrument calibrationpipelines for new instruments using CPL-based recipes remains the highest priority.Thanks to early and fruitful collaborationwith the instrument development teams dur-ing commissioning and science verification,the SINFONI and VISIR pipelines are closeto completion and should be operational atthe start of Period 75. Next will comeOmegaCam, Hawk-I and CRIRES, followedby the various VLT 2nd generation instru-ments. As soon as possible, FORS1, FORS2(including MXU), ISAAC, NACO andUVES pipelines will be re-implemented,using CPL-based modules written for otherinstruments. Realistically, it may take morethan two years to complete this conversionwork, given higher priority projects.

In parallel, ESO is investigating ways ofproviding science products produced fromdata with expired proprietary periods to thecommunity. Possibilities include ingestingexisting Service Mode science products kepton off-line archival media or re-created sci-ence products produced by re-processingraw science data with improved pipelinerecipes. A pilot project involving UVES datais active. In the future, ESO will also ingestinto the Science Archive Facility scienceproducts produced by ESO for ServiceMode users immediately after their creationand then make them available to the commu-nity after the proprietary period is complet-ed. If resources permit, it may also be possi-ble to start created products from VisitorMode data on a regular basis. Finally, as

The Messenger 1186

Figure 5: Example GIRAFFE IFU Science Product. Left panel: reducedscience spectra from 15 GIRAFFE IFU units. Right panel: reconstruct-ed images for each IFU unit. For VIMOS IFU example, see Izzo et al.(2004). Courtesy of R. Hanuschik.

appropriate, on-the-fly recalibration systemswill be activated for appropriate instrumentsmodes. Many of these efforts will fall underthe responsibility of the Virtual ObservatorySystems department created recently withinthe Data Management and OperationsDivision.

AACKNOWLEDGMENTSCKNOWLEDGMENTSThe development of instrument calibrationpipelines to produce science products has been acollaborative effort involving many people overmany years. We thank everyone for their efforts

on behalf of the ESO community. The originalpipeline recipes for FORS1, FORS2, GIRAFFE,SINFONI, UVES-Fibre, and VIMOS were devel-oped by their respective external instrument con-sortia. Of course, the consortia are not responsiblefor how these recipes have been maintained orrevised over time by ESO to suit our operationalneeds. The DMD Data Flow Systems departmentdeveloped all other pipeline recipes, as well as theassociated infrastructure wrapped around allrecipes, in close collaboration with astronomersfrom the Instrument Division, the ParanalObservatory, and the DMD Data Flow Operationsgroup. In particular, we thank sincerely the vari-

ous cross-divisional Instrument Operations Teamsfor their past and on-going efforts. Finally, wethank Fernando Comerón, Reinhard Hanuschik,Andreas Kaufer, Bruno Leibundgut, ChrisLidman, Peter Quinn, and the DFO/QC team fortheir comments on earlier drafts of this article.

RREFERENCESEFERENCESBallester, P. et al. 2004, Messenger, 116, 4Comerón, F. et al. 2003, Messenger, 113, 32Izzo,C. et al. 2004, Messenger,117, 33Mathys, G. 2004, Messenger, 116, 8Quinn, P.J. 1996, Messenger, 84, 30

© ESO - December 2004 7

Table 2: On-line Resources Related to Science Products

Resource URL Remarks

ESO Quality Control Group http://www.eso.org/qc/ Instrument-specific information about qualitycontrol, data processing, and science products

Gasgano http://www.eso.org/gasgano Java-based tool for file organization and recipeexecution

Common Pipeline Library (CPL) http://www.eso.org/cpl ISO-C libraries to build pipeline recipes

Pipeline recipes http://www.eso.org/pipelines Released recipes and documentation

Pipeline status http://www.eso.org/qc/pipeline-status.html Brief overview of pipeline status, updatedbefore every ESO Period

Site Monitor http://archive.eso.org/asm/ambient-server Daily meteorological data for Paranal and La Silla

Standalone FITS Tools http://archive.eso.org/saft/ Tools for working with FITS files

H.

Heyer

(ES

O)

Signature in Copenhagen of the agreement for the construction of the X-shooter, a second generation VLT instrument scheduled to go into oper-ation in 2008. From left to right, sitting: Jens Hjorth (Astronomy Professor, Copenhagen University Observatory), Henning Jørgensen (AstronomyProfessor, Copenhagen University and Danish ESO Council Member), Jørgen Olsen (Vice-Chancellor, Copenhagen University), CatherineCesarsky (Director General of ESO), Per Kjærgaard Rasmussen (P.I. for the X-shooter, Copenhagen University Observatory); standing: Nils O.Andersen (Director of the Niels Bohr Institute for Astronomy, Physics and Geophysics), Sandro D’Odorico (X-shooter P.I. at ESO) and Jens ViggoClausen (Director of Copenhagen Observatory). (see Page 71)