Data processing and science products at CASU...fov 1 deg x 1 deg 32 CCD mosaic 2k x 4k e2v 44-82 256...

Data processing and science products at CASU

Mike IrwinJim Lewis

Eduardo Gonzalez-SolaresMarco Riello

Aybuke Kupcu Yoldas

Simon Hodgkin

Nic WaltonCarlos Gonzalez-Fernandez

Greg Madsen

.. .. ..,

Internet transfer ~ 500GB/night

Processing Database ingestion

and QC

Data serving

CASU

VST @ ParanalVISTA @ ParanalUKIRT @ Mauna Kea

! ground-based imaging surveys

– all data taken on UKIRT WFCAM (Z,Y,J,H,K.....)

– all data taken on VISTA VIRCAM (Z,Y,J,H,Ks.....)

– all VST public survey data (ATLAS, VPHAS, KiDS - u,g,r,i,z,H )

! processing for other large area/deep imaging instruments

– CFH, CTIO, INT WFC, Subaru .......

– all UT4 HAWKI data 2008 ->

– all UT3 VIMOS imaging data 2002 ->

! wide-field multi-object spectroscopic surveys

– Gaia-ESO survey

– data management for WEAVE, 4MOST?, MOONS?

α

10TB

50TB

500TB

Optical

WFCAM

/year now now

90TB

VISTA

400TB

Data products volume

75TB

50TB

500TB

VST WFCAM

/year now now

90TB

VISTA

400TB

Data products volume

VST now

Jim Lewis

Pipeline processing NIR data

CASU mantra

• MEFs as container -> simplifies bookkeeping

• use lossless Rice-compression -> (x 2-4 less space)

• FITS images and catalogue binary tables (CFITSIO)

• FITS headers record processing details

- derived QC parameters

- WCS astrometric calibration

- photometric calibration

- table/image fluxes in ADU, x,y positions

- versioning and software details

• modular software -> C & perl/python scripts

• minimise external software dependencies

!"#$%&'&$#()*+',-.$"//*(/

0 "#$)('(+'-1/$&1($,.2(1(.'34$5-1($

*./'&63($'2&.$-7',+&3$88%9

0 /-5($-))$(3(+'1-.,+$(::(+'/

0 .-'$&/$;--)$+-/5(',+&334$&/$88%/

0 9<4$6&+<;1-*.)$=>??@$61,;2'(1$'2&.$

5-/'$-6A(+'/$-:$,.'(1(/'

0 9<4$6&+<;1-*.)$,/$B&1,&63($'(57-1&334$

&.)$/7&',&334

0 C@7-/*1($',5(/$&1($/2-1'D$/-$.,;2'34$)&'&$

1&'(/$&1($B(14$2,;2$EFGH?IH??JKL

Time Variable Sky

NIR image processing steps

• Raw data transfer, ingest and organisation• Monthly calibration updates (flats, linearity, masks)

• Nightly processing: – reset correction – dark correction– linearity correction– flat field correction– sky background correction *****– destripe - controller level pickup

• Crosstalk, persistence, fringing corrections

Before And After Background Correction

“Sky” background Product image

VIRCAM ‘Group’ Processing Steps

• combining dithers -> stacked pawprints

• catalogue generation

– morphological classification

– astrometric calibration (< 100 mas)

– photometric zeropoint calibration (< 2%)

• nebulising pawprint images

• tiling/mosaicing of pawprints

• tile catalogue generation + calibration

• retiling original stacked pawprints

!(6*3,/()$7&M71,.'/$&.)$',3(/

!(6*3-/,'4$:,3'(1$&773,()$'-$/'&+</$6(:-1($5-/&,+,.;

Post-Processing Steps

• monthly zeropoint updates

– stack 2MASS photometric residuals

– compute detector-level ZP offsets

– and residual spatial illumination correction

• tile grouting

– fix catalogues for variable PSFs

– and for varying pawprint-level ZPs

– and for individual detector-level ZPs

• reclassify and update tile catalogue calibration

Typcial data products

• products consist of:

– calibrated single exposure images– shifted ”average” stack frames (pawprints) + conf maps

– calibrated stacked pawprint catalogues– filled area tile images + confidence maps

– calibrated tile object catalogues

– sky background images, flats, darks, bad pixel mask

• science products are MEF files (images Rice-compressed)

• all QC parameters are stored in MEF headers

• headers contain provenance information

• processing web page updates– http://casu.ast.cam.ac.uk/surveys-projects/vista

Eduardo Gonzalez-Solares

Web-based user interfaces

http://casu.ast.cam.ac.uk

UKIRT/WFCAM

Start Apr 2005

ESO/VISTA

Start Oct 2009

ESO/VST

Start Oct 2011

1.5 petabytes

7,047,786 single CCD images*

48,438,579,284 single band detections

raw processed raw processed

Other

WFCAM

VISTA

VST

21/07/2011 09:19ESO Ambient Conditions Database

Page 1 of 2http://archive.eso.org/asm/ambient-server?site=paranal&night=24+Jun+2011

ESO Observatories

Ambient Conditions Database

Archive Facility HOME ESO HOME INFO

show previous Night show next Night

make Printable Click on a graph to zoom inJava enabled browsers only

Night starting on 24 Jun 2011 Site Paranal Display

Circumstances for Paranal on the night starting on 24 Jun 2011 (times in UT)

Midnight SUN Twilight LMST MOON

LMST set twi.end twi.beg rise eve morn rise set %illum RA DEC

ESO monitoring of ambient conditions

Monitoring Paranal NIR sky surface brightness

N.$(@&573($-:$O8$'1(.)$&.&34/,/

• Variable zero-point (and hence survey depth) with time

0 silvered mirror degradation

0 aluminised mirror

0 a bit of window cleaning

A search by position returnsimages that contain that positionand allows preview of postagestamps,catalogue sources anda view of provenance images.

!"#$%&'()*+,-./0!"#$%&'()**%&+&,-./

!"#$%&'()&'*+,

!"#$%&'()*+,-%./0.,(1.2

!"#"$"%"&"'"()**+,"-)(./01*)23+(4"42)*21."3("50*36"7889

:&;<=="4>*?1@4"'"4A)66"B*)C23+("+B"D;"0*+E*)A4F3432+*"A+.1">(236"1(."+B"788GH1A+216@"+01*)21."B*+A"$IJK

<)2)"0*+C1443(E"-@"#$3'"L.44,5.6.M=C31(C1"5*CN3?1"3("O.3(->*EN"LPQ5:M:&;<=="*),".)2)")?)36)-61"L)64+M"B*+A"O=/")*CN3?1

!"#$%&'()&*%+,&%(-$.(%/+01

%0&.)2(0

!"#$3&/$-&%&$%0&.)2(00(-$20+,$)*,,4%$%+$546+789$%+$9:"7$;+.<(0)4+.$'(0$;&,(0&

#"%&=0+.$>+?$'*66)$&66$.(/$-&%&$'(0$.4@A%$&.-$;&,(0&$BCDE$A0)FG))(,?6($H*6%4DIJ%(.)4+.$9:"7$246()

8(%(;%4+.$+2$4.;+,'6(%($4,&@()$K,4))4.@L-(&-$;&,(0&M8(%(;%4+.$&.-$26&@@4.@$+2$;+00*'%$A(&-(0)$L$4,&@()N(.(0&%4+.$+2$+?)(0<&%4+.$6+@G--$.4@A%$%+$O9=GH$-&%&?&)($K/(?$'&@()M

P*)A$QR:877$S$=&64?0&%4+.)$%+$I7#T&;1*'$0&/$-&%&$K!"#$%&'()M

'())*"(+,-.+/0

!"#"$%&'($)$%*'+$,"-."$/0"$#'$1"+2*3450

!"#"$%&'($)$%*'+$*"($#'$,-./0-/1*/"23$2"#"

!"#$%"%&'()!'$#*')#!"(+#,,')-

!(,&.$"(+#,,')-%)/01%*'/%&'()

2&("%-#

3*%&03'#*/045()6'/#)+#07%$,

!"#$%&'

8)+(7$"#,,90/%":,90$"(+#,,')-0,;77%"<

!"#.$"(+#,,')-90,:<0+"#%&'()0=0,;>&"%+&'()90!"#$%&$'(!")90,&%+:')-90(>?0/#&#+&'()90@+,

A50B-%">%-#0/',$(,%*90$C(&(7#&"<90@+,D90E2F90+(7$"#,,'()

G**;7')%&'()05(""#+&'(),H"%),6#"0&(0E3F8

()*$"*+#,-%$.

/"#/(001#$&'

7'+"(,&#$$')-0=0')&#"*#%1')-

!"#$%&'

!"#"$%&'()*#+

! "#$%&'()*+,-./! 0.$-1&+'2#$%&/'3'-*452+&4-&'#$6/

! "4.&)7&$8&+'2#$%&/'3'-*452+&4-&'#$6/

! "4+282+,$7'2#$%&/

! 9/.)*#&.)2-$77:';'6<*.*#&.)2-$77:'-$72=)$.&+>

! ?=@&-.'-$.$7*%,&/! ?=@&-.'-7$//252-$.2*4';'6$)$#&.&)/

! A&$+&)B=$/&+'#&.$+$.$

" 9/.)*#&.)2-$77:';'(<*.*#&.)2-$77:'-$72=)$.&+>

" C,$72.:'-*4.)*7'DCEF'6$)$#&.&)/

>',/24%'GH900

!"#$%&'()*+,+*&%&'*&'-*

! "#$$%&'()*+%,&-%./0&*'&(

! 1,*,%2/3045

! 6,78%9:;$%<=

! >6/.5++5-8%9?$$%<=

! ./4>65++'/&%@,.*/68%9AB

! C.'5&.5D65,-E%-,*,%>6/-0.*+8

! 9%FF$$$$%C*,.G5-%'4,(5+

! 9%FHF%4'33'/&%>,7D>6'&*+

! 9%I%='33'/&%-5*5.*'/&+

!"#$%"$&$'()*+$,-./0!!

!"#$%&'()'*+',"-'.#(./0

VISTA processing

Aybuke Kupcu Yoldas

OrionM42

region

colour composite

J,H,Ks

16kx13kpixels/

waveband

mosaic of96 2kx2kstacked images/

waveband

.. .. ..

,

optical

optical NIR

VISTA focal plane

VISTA Surveys

Survey Area (deg2) Filters and Depth Measure Depth (mag)

Ultra-VISTA 0.73 (ultra-deep) 5!, AB Y=26.7 J=26.6 H=26.1 Ks=25.6 NB=26.0

VIKING 1500 5!, AB Z=23.1 Y=22.3 J=22.1 H=21.5 Ks=21.2

VMC 184 10!, Vega Y=21.9 J=21.4 Ks=20.3

VVV 520 5!, Vega Z=21.9 Y=21.2 J=20.2 H=18.2 Ks=18.1

VHS 20 000 5!, AB Y=21.2 J=21.2 H=20.6 Ks=20.0

VIDEO 12 5!, AB Z=25.7 Y=24.6 J=24.5 H=24.0 Ks=23.5

VISTA focal plane

18/07/2011 12:06VISTA Focal Plane

Page 1 of 1http://casu.ast.cam.ac.uk/surveys-projects/vista/vistafocalplane.png/image_view_fullscreen

Back to site

VIRCAM focal plane

@PQQQQQQ

R$+-.:,)(.+($5&7

R$+&'&3-;*(

R$+-.:,)(.+($5&7

R$+&'&3-;*(

SP$+2,7/

T"9UN

7&M71,.'/

),'2(1

/'&+<()

5-/&,+

',3(

Challenges when dealing with tiles

! imperfect sky subtraction pawprint matching + artefacts

! variable PSF across pawprint detectors

! variable seeing conditions for entire pawprint

! variable saturation levels of each detector

! variable extinction (ZP) during tile observations

! astrometric distortion = photometric distortion

! interpolation = varying correlated noise patterns

! “interesting” Modified Julian Date pattern

MJD variation across tiles

VMC survey

SMC west

and 47Tuc

VMC survey

SMC west

and 47Tuc

VMC survey

SMC west

and 47Tuc

VMC survey SMC west & 47Tuc -> pipeline CMD

Science examples“Milky Way Demographics with the VVV Survey I. The 84 Million Star Colour–Magnitude Diagram of the Galactic Bulge“ by R. K. Saito et al.,A&A, 544, A147

http://www.eso.org/public/news/eso1242/

VST processing

Mike Irwin

fov 1 deg x 1 deg

32 CCD mosaic 2k x 4k e2v 44-82

256 Mpixels

0.214 arcsec sampling

co-planar to +/-20um

r/o time 40sr/o noise 6e-

2 x 6 filter stackchanges 65s - 115s

gain ~2.9e-/ADU

PA 0, 90, 180, 270

OmegaCAM

User Manual VST-MAN-OCM-23100-3110 Version 2.5 (44 pages) 13

Figure 9: Throughput of the OmegaCAM Sloan filters, times the average quantum e!ciency of theCCDs. For comparison the standard Johnson-Cousins UBVRI filters are also shown.

VST processing schema

[pre-commissioning plan based on processing for other optical mosaic cameras]

VSTVST public surveys

ESO Revised Survey Management Plan Form Phase 1

In terms of survey area we shall be following the recommendations of the ESO Public Survey Panel in revisingthe area of ATLAS to reduce overlap with DES. The SGC area is thus now defined between 21h30 < RA < 04h00and −40 < Dec < −10. The NGC area is defined between 10h00 < RA < 15h30 and −20 < Dec < −2.5 plus10h00 < RA < 15h00 and −30 < Dec < −20. The total area of the survey is 4711 deg2 with 2087 deg2 in theNorthern Galactic Cap (NGC) and 2624 deg2 in the Southern Galactic Cap (SGC). Fig. 1(a) shows the revisedsurvey area.

Figure 1: (a) The revised ATLAS footprint approved by the Public Surveys Panel and OPC. The dashed linesmark galactic latitudes b = 30◦ and b = −30◦. (b) The current status of the VISTA Hemisphere Survey (VHS).

We also aim to coordinate as fully as possible with the VISTA Hemisphere Survey (VHS). In the NorthernGalactic Cap (NGC), VHS intends to cover the sky with YJHK at galactic latitudes, |b| > 30deg, and JHK at|b| < 30deg. In the SGC, VHS will cover the DES area in JHK, and the remaining |b| > 30deg area with YJHKand the remaining |b| < 30deg area with JK. Current VHS coverage is shown in Fig. 1(b).

Our basic observing strategy for each VST ATLAS tile is to obtain 2 dithered exposures in each of the ugrbands in dark time and then to obtain 2 dithered exposures in each of the iz bands in grey/bright time. A 1hrOB will require no filter changes as has been recommended by ESO. We shall be grouping together the 3 ‘dark’OBs and the 2 ‘gray/bright’ OBs in different filters on a given area so that the 5 filters may be done as closetogether in time as possible, and preferably within a single lunation to minimise the impact that point-sourcetime variability might otherwise have on derived colours. To some extent, this strategy will also increase thelikelihood of equivalent image quality (in terms of PSF, photometric conditions) in each field, simplifying finalflux calibration of the data.

Our new observing strategy accommodates the decision of the ESO Survey Team not to allow 2 × 2 on-chipbinning as well as the recommendation to minimise filter changes in a 1hr OB. Our strategy also takes intoaccount the results from VST+OmegaCAM commissioning, that guiding will not be needed for exposures ofless than 60s and that the telescope can dither/slew/preset when the CCDs are reading out.

We therefore aim to make 2 sub-exposures per filter on each ATLAS tile. This will allow us to remove cosmicrays effectively (CR rate is 1790± 200/CCD/hr) and also help fill in interchip gaps. This leaves us with a planfor each tile as shown in Table 2. Table 3 shows the number of sky electrons per 0.��214 pixel according toVOCET, assuming the darkest sky. Given the read-out noise of 5.5 electrons, we have therefore increased theoverall exposure times from 1×60s in ugriz to 2×60s in u, 2×50s in g and 2×45s in riz. These new exposuretimes return almost exactly the same S/N as would have been achieved with 0.��4 pixels, as used by SDSS orby binning 2 × 2 with OmegaCAM. These assume the same (VOCET) sky counts as given in Table 3 and 5.5e− read-out noise.

The two sub-exposures tiling plan means that the overall efficiency of observation is now 55%, still comparableto 60% with a single exposure, unbinned, strategy.

We also have agreed to make a calibration pass in ugri(z) in the ≈ 700deg2 KiDS-S area. This will comprise

ESO-USD (usd-help@eso.org) page 2 of 10

VST public surveys

Illumination correction issues

! radial concentration of scattered light in optics

! detector level zero-point differences

! non-uniform non-astronomical scattered light

! scattering off masking strips and edges

! PSF variations over calibration frames

! solution -> APASS (now v6) + monthly stacks

! achieving robust +/-1% accuracy at a resolution of < 1 arcmin directly is difficult -> but current method achieves this

illumination correction anatomy i-band (jan2012)

illumination correction anatomy i-band (jan2012)

illumination correction anatomy i-band (jan2012)

illumination correction anatomy i-band (jan2012)

illumination correction anatomy i-band (jan2012)

illumination correction anatomy i-band (jan2012)

APASS jan2012

g

r

i

z

u

Ha

APASS