NIR Transient Surveys

Nicholas CrossWFAU, Edinburgh

Nigel Hambly, Mike Read, Ross Collins, Eckhard Sutorius, Rob Blake, Mark Holliman

IAU S285, Oxford 2

NIR Variability Science Drivers

• NIR, smaller detectors, higher backgrounds and more expensive detectors than optical– Only do multi-epoch work where it is not practical for

optical detectors– Looking through the dense dusty regions of the MW to

the far side– Young Stellar Objects in star-forming regions– Low mass stars / brown dwarfs– High z galaxies / Snae– Can get better RR Lyrae / Cepheid distances in NIR

19/09/2011

IAU S285, Oxford 3

NIR Variability Surveys

• UKIRT WFCAM – UKIDSS – DXS/UDS (Deep surveys, multi-epoch), – WFCAM Transit Survey,– Calibration/Standard Stars, – Surveys of YSOs in Orion/Ophiuchus

• VISTA– VISTA Variables in Via-Lactea (VVV), (RR Lyrae, Cepheids)– VISTA Magellanic Cloud (VMC), (RR Lyrae, Cepheids)– VIDEO (Deep Extragalactic – SNae)

19/09/2011

IAU S285, Oxford 4

WFCAM• 3.8 m UKIRT telescope

on Mauna Kea.• 4 2k x 2k Rockwell

Hawaii 2 detectors.• Spaced 94% apart. • 0.4” pixels.• 13.65’ across each side.• 60% of time on UKIRT in

2005b• 100% for 2009a

19/09/2011

IAU S285, Oxford 5

VIRCAM

19/09/2011

• 4.1m VISTA telescope at Cerro Paranel.• 16 2k x 2k Raytheon VIRGO detectors• Spaced 90% in x and 42.5% in y.• 0.34” pixels• Tile is 1.5° • VIRCAM has 100% of time. • > 3 times area WFCAM• 2 * QE

IAU S285, Oxford 6

VISTA Public Surveys

19/09/2011

IAU S285, Oxford 7

VISTA Variables in Via-Lactea (VVV)

• Very high density ~106 sources / sq. deg.– Issues with

deblending• 500 sq. deg• ~100 epochs

(currently ~10)• ~ few 1010 detections

19/09/2011

IAU S285, Oxford 8

Processing of WFCAM and VISTA data

• VDFS: VISTA Data Flow System (System for processing of UKIRT WFCAM and VISTA data.– CASU (Cambridge): Data reduction, processing of

observing blocks, photometric and astrometric calibration

– WFAU (Edinburgh): Archive, processing of multiple observing blocks – deep stacks, multi-band tables, links to external tables, MULTI-EPOCH.

– For VISTA, data goes through ESO and final products go to ESO too.

19/09/2011

IAU S285, Oxford 9

Constraints from VDFS• >=6 week time lag before data at WFAU– Data needs to be transferred to Cambridge (with VISTA this

includes disk drive to Garching and then to Cambridge)– Accurate photometric calibration (including scattered light

corrections uses 1 month of data.– VoEvent alerts are too late from WFAU

• Reprocessing of OB data requires retransfer between CASU and WFAU and reingest of data at WFAU. – Detection tables are used by many curation processes –

reingestion into these slows later stages.

19/09/2011

IAU S285, Oxford 10

Stages of multi-epoch processing

• Stack epochs to create deep images and extract catalogues

• Create master list (Source table) from band-merged catalogues from deep images.

• Recalibrate each epoch image compared to the deep image in that filter and pointing.

• Create table linking sources to each observation • Calculate the noise properties of each pointing and filter• Calculate astrometric and photometric statistics for each

source.

19/09/2011

IAU S285, Oxford 11

Analysing Variables

• Calculate mean, rms of magnitudes.

• Bin in magnitude and calculate clipped median

• Fit empirical noise model • (m)=a+b10-0.4m+c10-0.8m

• Classify as variable or non-variable

19/09/2011

IAU S285, Oxford 12

Archival Databases• Curation of WFCAM and VISTA data occurs in a

RDBMS using Microsoft SQL Server. – Dynamic database, updated with new data,

improved calibrations and reprocessed data when necessary.

– Static releases to the science teams and world for science purposes.

• Curation controlled by comparing current state of DB with requirements

19/09/2011

IAU S285, Oxford 13

Programme Requirements• Pointing, filter and table requirements are setup by

grouping the metadata and using specifications for each survey.

• Schema updated if necessary• Stack / tile products made for a particular release

number• Source table created for particular pointings• Each stage of multi-epoch processing checks the

whether the previous table has changed in that pointing – higher curation event ID.

19/09/2011

IAU S285, Oxford 14

VISTA tiles• Most surveys require tiles to reach expected depth, and

tiles are standard ESO product.• PSF and sky vary on short time scales < integration time• Images filtered to remove large spatial variations (>30”)• Tile catalogues are inferior to pawprints:

– Not as accurate astrometry– Do not deal with saturation correctly– Extended (>30”) sources are missing or have incorrect

photometry• Catalogues from tiles and pawprints

– Need to be able to compare – multiple layers and linking tables.

19/09/2011

IAU S285, Oxford 15

Problems / bottlenecks /solutions• Reprocessing of OB data.

– 1st year of VISTA – 2 sets of full reprocessing• Ingesting new data while curating later products

– Put VVV on separate server and synchronise metadata tables• BUT foreign key constraints to vvvDetection cause major holdups if

metadata is deleted.– Split vvvDetection into semesters / months so new data can be

ingested into new semester.• Has not been implemented yet

• Users want to use both tile and pawprint detections– Produce linking tables

• BUT some queries that join these can join several tens of tables and SQL does not handle these joins well.

• Enhancements to user interface allow users to save intermediate results

19/09/2011

IAU S285, Oxford 16

Problems / bottlenecks /solutions• Checking non-detections of sources

– Using half-space method of Budavari, major improvement• Dealing with very long processing times of VVV

– Break curation into chunks with software testing to see what has already been done

– Make sure memory never exceeds ~40% – BUT this adds additional overheads at beginning of each run

• Variability table curation is dominated by DB reads (85% for VVV)– Use Query Analyser and other tools to optimise queries [OPTION

(MAXDOP 1)], adding removing indexes.– Split detection tables into parts?

• I/O limited between servers and disks– SQL Server “cluster” linked by infiniband 10Gbs-1

19/09/2011

IAU S285, Oxford 17

Other issues• Classification

– DB has simple classification (variable or not) and some other statistical quantities. VVV will have ~106 variables

– Chilean teams working on NIR templates for different types of variables

– Trend analysis (Istvan Dekany)• Accuracy

– VSA/WSA, simple ZP recalibration – rms ~0.005mag• Good enough for most variables• Planetary Transits require (prefer) ~0.001 mag.

• Confusion– Difference Imaging Analysis (Eamonn Kerins), will probably be applied

to densest 40 sq. deg of VVV bulge.

19/09/2011