Embed Size (px)
Citation preview
Data processing and science products at CASU
Mike IrwinJim Lewis
Eduardo Gonzalez-SolaresMarco Riello
Aybuke Kupcu Yoldas
Simon Hodgkin
Nic WaltonCarlos Gonzalez-Fernandez
Greg Madsen
.. .. ..,
Internet transfer ~ 500GB/night
Processing Database ingestion
and QC
Data serving
CASU
VST @ ParanalVISTA @ ParanalUKIRT @ Mauna Kea
! ground-based imaging surveys
– all data taken on UKIRT WFCAM (Z,Y,J,H,K.....)
– all data taken on VISTA VIRCAM (Z,Y,J,H,Ks.....)
– all VST public survey data (ATLAS, VPHAS, KiDS - u,g,r,i,z,H )
! processing for other large area/deep imaging instruments
– CFH, CTIO, INT WFC, Subaru .......
– all UT4 HAWKI data 2008 ->
– all UT3 VIMOS imaging data 2002 ->
! wide-field multi-object spectroscopic surveys
– Gaia-ESO survey
– data management for WEAVE, 4MOST?, MOONS?
α
10TB
50TB
500TB
Optical
WFCAM
/year now now
90TB
VISTA
400TB
Data products volume
75TB
50TB
500TB
VST WFCAM
/year now now
90TB
VISTA
400TB
Data products volume
VST now
Jim Lewis
Pipeline processing NIR data
CASU mantra
• MEFs as container -> simplifies bookkeeping
• use lossless Rice-compression -> (x 2-4 less space)
• FITS images and catalogue binary tables (CFITSIO)
• FITS headers record processing details
- derived QC parameters
- WCS astrometric calibration
- photometric calibration
- table/image fluxes in ADU, x,y positions
- versioning and software details
• modular software -> C & perl/python scripts
• minimise external software dependencies
!"#$%&'&$#()*+',-.$"//*(/
0 "#$)('(+'-1/$&1($,.2(1(.'34$5-1($
*./'&63($'2&.$-7',+&3$88%9
0 /-5($-))$(3(+'1-.,+$(::(+'/
0 .-'$&/$;--)$+-/5(',+&334$&/$88%/
0 9<4$6&+<;1-*.)$=>??@$61,;2'(1$'2&.$
5-/'$-6A(+'/$-:$,.'(1(/'
0 9<4$6&+<;1-*.)$,/$B&1,&63($'(57-1&334$
&.)$/7&',&334
0 C@7-/*1($',5(/$&1($/2-1'D$/-$.,;2'34$)&'&$
1&'(/$&1($B(14$2,;2$EFGH?IH??JKL
Time Variable Sky
NIR image processing steps
• Raw data transfer, ingest and organisation• Monthly calibration updates (flats, linearity, masks)
• Nightly processing: – reset correction – dark correction– linearity correction– flat field correction– sky background correction *****– destripe - controller level pickup
• Crosstalk, persistence, fringing corrections
Before And After Background Correction
“Sky” background Product image
VIRCAM ‘Group’ Processing Steps
• combining dithers -> stacked pawprints
• catalogue generation
– morphological classification
– astrometric calibration (< 100 mas)
– photometric zeropoint calibration (< 2%)
• nebulising pawprint images
• tiling/mosaicing of pawprints
• tile catalogue generation + calibration
• retiling original stacked pawprints
!(6*3,/()$7&M71,.'/$&.)$',3(/
!(6*3-/,'4$:,3'(1$&773,()$'-$/'&+</$6(:-1($5-/&,+,.;
Post-Processing Steps
• monthly zeropoint updates
– stack 2MASS photometric residuals
– compute detector-level ZP offsets
– and residual spatial illumination correction
• tile grouting
– fix catalogues for variable PSFs
– and for varying pawprint-level ZPs
– and for individual detector-level ZPs
• reclassify and update tile catalogue calibration
Typcial data products
• products consist of:
– calibrated single exposure images– shifted ”average” stack frames (pawprints) + conf maps
– calibrated stacked pawprint catalogues– filled area tile images + confidence maps
– calibrated tile object catalogues
– sky background images, flats, darks, bad pixel mask
• science products are MEF files (images Rice-compressed)
• all QC parameters are stored in MEF headers
• headers contain provenance information
• processing web page updates– http://casu.ast.cam.ac.uk/surveys-projects/vista
Eduardo Gonzalez-Solares
Web-based user interfaces
UKIRT/WFCAM
Start Apr 2005
ESO/VISTA
Start Oct 2009
ESO/VST
Start Oct 2011
1.5 petabytes
7,047,786 single CCD images*
48,438,579,284 single band detections
raw processed raw processed
Other
WFCAM
VISTA
VST
21/07/2011 09:19ESO Ambient Conditions Database
Page 1 of 2http://archive.eso.org/asm/ambient-server?site=paranal&night=24+Jun+2011
ESO Observatories
Ambient Conditions Database
Archive Facility HOME ESO HOME INFO
show previous Night show next Night
make Printable Click on a graph to zoom inJava enabled browsers only
Night starting on 24 Jun 2011 Site Paranal Display
Circumstances for Paranal on the night starting on 24 Jun 2011 (times in UT)
Midnight SUN Twilight LMST MOON
LMST set twi.end twi.beg rise eve morn rise set %illum RA DEC
ESO monitoring of ambient conditions
Monitoring Paranal NIR sky surface brightness
N.$(@&573($-:$O8$'1(.)$&.&34/,/
• Variable zero-point (and hence survey depth) with time
0 silvered mirror degradation
0 aluminised mirror
0 a bit of window cleaning
A search by position returnsimages that contain that positionand allows preview of postagestamps,catalogue sources anda view of provenance images.
!"#$%&'()*+,-./0!"#$%&'()**%&+&,-./
!"#$%&'()&'*+,
!"#$%&'()*+,-%./0.,(1.2
!"#"$"%"&"'"()**+,"-)(./01*)23+(4"42)*21."3("50*36"7889
:&;<=="4>*?1@4"'"4A)66"B*)C23+("+B"D;"0*+E*)A4F3432+*"A+.1">(236"1(."+B"788GH1A+216@"+01*)21."B*+A"$IJK
<)2)"0*+C1443(E"-@"#$3'"L.44,5.6.M=C31(C1"5*CN3?1"3("O.3(->*EN"LPQ5:M:&;<=="*),".)2)")?)36)-61"L)64+M"B*+A"O=/")*CN3?1
!"#$%&'()&*%+,&%(-$.(%/+01
%0&.)2(0
!"#$3&/$-&%&$%0&.)2(00(-$20+,$)*,,4%$%+$546+789$%+$9:"7$;+.<(0)4+.$'(0$;&,(0&
#"%&=0+.$>+?$'*66)$&66$.(/$-&%&$'(0$.4@A%$&.-$;&,(0&$BCDE$A0)FG))(,?6($H*6%4DIJ%(.)4+.$9:"7$246()
8(%(;%4+.$+2$4.;+,'6(%($4,&@()$K,4))4.@L-(&-$;&,(0&M8(%(;%4+.$&.-$26&@@4.@$+2$;+00*'%$A(&-(0)$L$4,&@()N(.(0&%4+.$+2$+?)(0<&%4+.$6+@G--$.4@A%$%+$O9=GH$-&%&?&)($K/(?$'&@()M
P*)A$QR:877$S$=&64?0&%4+.)$%+$I7#T&;1*'$0&/$-&%&$K!"#$%&'()M
'())*"(+,-.+/0
!"#"$%&'($)$%*'+$,"-."$/0"$#'$1"+2*3450
!"#"$%&'($)$%*'+$*"($#'$,-./0-/1*/"23$2"#"
!"#$%"%&'()!'$#*')#!"(+#,,')-
!(,&.$"(+#,,')-%)/01%*'/%&'()
2&("%-#
3*%&03'#*/045()6'/#)+#07%$,
!"#$%&'
8)+(7$"#,,90/%":,90$"(+#,,')-0,;77%"<
!"#.$"(+#,,')-90,:<0+"#%&'()0=0,;>&"%+&'()90!"#$%&$'(!")90,&%+:')-90(>?0/#&#+&'()90@+,
A50B-%">%-#0/',$(,%*90$C(&(7#&"<90@+,D90E2F90+(7$"#,,'()
G**;7')%&'()05(""#+&'(),H"%),6#"0&(0E3F8
()*$"*+#,-%$.
/"#/(001#$&'
7'+"(,&#$$')-0=0')&#"*#%1')-
!"#$%&'
!"#"$%&'()*#+
! "#$%&'()*+,-./! 0.$-1&+'2#$%&/'3'-*452+&4-&'#$6/
! "4.&)7&$8&+'2#$%&/'3'-*452+&4-&'#$6/
! "4+282+,$7'2#$%&/
! 9/.)*#&.)2-$77:';'6<*.*#&.)2-$77:'-$72=)$.&+>
! ?=@&-.'-$.$7*%,&/! ?=@&-.'-7$//252-$.2*4';'6$)$#&.&)/
! A&$+&)B=$/&+'#&.$+$.$
" 9/.)*#&.)2-$77:';'(<*.*#&.)2-$77:'-$72=)$.&+>
" C,$72.:'-*4.)*7'DCEF'6$)$#&.&)/
>',/24%'GH900
!"#$%&'()*+,+*&%&'*&'-*
! "#$$%&'()*+%,&-%./0&*'&(
! 1,*,%2/3045
! 6,78%9:;$%<=
! >6/.5++5-8%9?$$%<=
! ./4>65++'/&%@,.*/68%9AB
! C.'5&.5D65,-E%-,*,%>6/-0.*+8
! 9%FF$$$$%C*,.G5-%'4,(5+
! 9%FHF%4'33'/&%>,7D>6'&*+
! 9%I%='33'/&%-5*5.*'/&+
!"#$%"$&$'()*+$,-./0!!
!"#$%&'()'*+',"-'.#(./0
VISTA processing
Aybuke Kupcu Yoldas
OrionM42
region
colour composite
J,H,Ks
16kx13kpixels/
waveband
mosaic of96 2kx2kstacked images/
waveband
.. .. ..
,
optical
optical NIR
VISTA focal plane
VISTA Surveys
Survey Area (deg2) Filters and Depth Measure Depth (mag)
Ultra-VISTA 0.73 (ultra-deep) 5!, AB Y=26.7 J=26.6 H=26.1 Ks=25.6 NB=26.0
VIKING 1500 5!, AB Z=23.1 Y=22.3 J=22.1 H=21.5 Ks=21.2
VMC 184 10!, Vega Y=21.9 J=21.4 Ks=20.3
VVV 520 5!, Vega Z=21.9 Y=21.2 J=20.2 H=18.2 Ks=18.1
VHS 20 000 5!, AB Y=21.2 J=21.2 H=20.6 Ks=20.0
VIDEO 12 5!, AB Z=25.7 Y=24.6 J=24.5 H=24.0 Ks=23.5
VISTA focal plane
18/07/2011 12:06VISTA Focal Plane
Page 1 of 1http://casu.ast.cam.ac.uk/surveys-projects/vista/vistafocalplane.png/image_view_fullscreen
Back to site
VIRCAM focal plane
@PQQQQQQ
R$+-.:,)(.+($5&7
R$+&'&3-;*(
R$+-.:,)(.+($5&7
R$+&'&3-;*(
SP$+2,7/
T"9UN
7&M71,.'/
),'2(1
/'&+<()
5-/&,+
',3(
Challenges when dealing with tiles
! imperfect sky subtraction pawprint matching + artefacts
! variable PSF across pawprint detectors
! variable seeing conditions for entire pawprint
! variable saturation levels of each detector
! variable extinction (ZP) during tile observations
! astrometric distortion = photometric distortion
! interpolation = varying correlated noise patterns
! “interesting” Modified Julian Date pattern
MJD variation across tiles
VMC survey
SMC west
and 47Tuc
VMC survey
SMC west
and 47Tuc
VMC survey
SMC west
and 47Tuc
VMC survey SMC west & 47Tuc -> pipeline CMD
Science examples“Milky Way Demographics with the VVV Survey I. The 84 Million Star Colour–Magnitude Diagram of the Galactic Bulge“ by R. K. Saito et al.,A&A, 544, A147
http://www.eso.org/public/news/eso1242/
VST processing
Mike Irwin
fov 1 deg x 1 deg
32 CCD mosaic 2k x 4k e2v 44-82
256 Mpixels
0.214 arcsec sampling
co-planar to +/-20um
r/o time 40sr/o noise 6e-
2 x 6 filter stackchanges 65s - 115s
gain ~2.9e-/ADU
PA 0, 90, 180, 270
OmegaCAM
User Manual VST-MAN-OCM-23100-3110 Version 2.5 (44 pages) 13
Figure 9: Throughput of the OmegaCAM Sloan filters, times the average quantum e!ciency of theCCDs. For comparison the standard Johnson-Cousins UBVRI filters are also shown.
VST processing schema
[pre-commissioning plan based on processing for other optical mosaic cameras]
VSTVST public surveys
ESO Revised Survey Management Plan Form Phase 1
In terms of survey area we shall be following the recommendations of the ESO Public Survey Panel in revisingthe area of ATLAS to reduce overlap with DES. The SGC area is thus now defined between 21h30 < RA < 04h00and −40 < Dec < −10. The NGC area is defined between 10h00 < RA < 15h30 and −20 < Dec < −2.5 plus10h00 < RA < 15h00 and −30 < Dec < −20. The total area of the survey is 4711 deg2 with 2087 deg2 in theNorthern Galactic Cap (NGC) and 2624 deg2 in the Southern Galactic Cap (SGC). Fig. 1(a) shows the revisedsurvey area.
Figure 1: (a) The revised ATLAS footprint approved by the Public Surveys Panel and OPC. The dashed linesmark galactic latitudes b = 30◦ and b = −30◦. (b) The current status of the VISTA Hemisphere Survey (VHS).
We also aim to coordinate as fully as possible with the VISTA Hemisphere Survey (VHS). In the NorthernGalactic Cap (NGC), VHS intends to cover the sky with YJHK at galactic latitudes, |b| > 30deg, and JHK at|b| < 30deg. In the SGC, VHS will cover the DES area in JHK, and the remaining |b| > 30deg area with YJHKand the remaining |b| < 30deg area with JK. Current VHS coverage is shown in Fig. 1(b).
Our basic observing strategy for each VST ATLAS tile is to obtain 2 dithered exposures in each of the ugrbands in dark time and then to obtain 2 dithered exposures in each of the iz bands in grey/bright time. A 1hrOB will require no filter changes as has been recommended by ESO. We shall be grouping together the 3 ‘dark’OBs and the 2 ‘gray/bright’ OBs in different filters on a given area so that the 5 filters may be done as closetogether in time as possible, and preferably within a single lunation to minimise the impact that point-sourcetime variability might otherwise have on derived colours. To some extent, this strategy will also increase thelikelihood of equivalent image quality (in terms of PSF, photometric conditions) in each field, simplifying finalflux calibration of the data.
Our new observing strategy accommodates the decision of the ESO Survey Team not to allow 2 × 2 on-chipbinning as well as the recommendation to minimise filter changes in a 1hr OB. Our strategy also takes intoaccount the results from VST+OmegaCAM commissioning, that guiding will not be needed for exposures ofless than 60s and that the telescope can dither/slew/preset when the CCDs are reading out.
We therefore aim to make 2 sub-exposures per filter on each ATLAS tile. This will allow us to remove cosmicrays effectively (CR rate is 1790± 200/CCD/hr) and also help fill in interchip gaps. This leaves us with a planfor each tile as shown in Table 2. Table 3 shows the number of sky electrons per 0.��214 pixel according toVOCET, assuming the darkest sky. Given the read-out noise of 5.5 electrons, we have therefore increased theoverall exposure times from 1×60s in ugriz to 2×60s in u, 2×50s in g and 2×45s in riz. These new exposuretimes return almost exactly the same S/N as would have been achieved with 0.��4 pixels, as used by SDSS orby binning 2 × 2 with OmegaCAM. These assume the same (VOCET) sky counts as given in Table 3 and 5.5e− read-out noise.
The two sub-exposures tiling plan means that the overall efficiency of observation is now 55%, still comparableto 60% with a single exposure, unbinned, strategy.
We also have agreed to make a calibration pass in ugri(z) in the ≈ 700deg2 KiDS-S area. This will comprise
ESO-USD ([email protected]) page 2 of 10
VST public surveys
Illumination correction issues
! radial concentration of scattered light in optics
! detector level zero-point differences
! non-uniform non-astronomical scattered light
! scattering off masking strips and edges
! PSF variations over calibration frames
! solution -> APASS (now v6) + monthly stacks
! achieving robust +/-1% accuracy at a resolution of < 1 arcmin directly is difficult -> but current method achieves this
illumination correction anatomy i-band (jan2012)
illumination correction anatomy i-band (jan2012)
illumination correction anatomy i-band (jan2012)
illumination correction anatomy i-band (jan2012)
illumination correction anatomy i-band (jan2012)
illumination correction anatomy i-band (jan2012)
APASS jan2012
g
r
i
z
u
Ha
APASS
