NEON Obs School 11-Aug-2005 Archival Data and Virtual Observatories 1 Virtual Observatories...or how...

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 1

Virtual Observatories

...or how to do your research from a beach in the Bahamas rather than stuck up a cold

mountain in the middle of the night...

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 2

Who I amJohn Taylor: jdt@roe.ac.ukWork on the AstroGrid and VOTech Virtual Observatory projectsMember of Edinburgh's Wide Field Astronomy Unit – curates catalogue and image data from SuperCOSMOS and (soon) WFCAM and (less soon) VISTASoftware engineer NOT Astronomer

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 3

OutlineThe data deluge Why science is changing The scale of the problem and technical issues

Catalogues How to use them VOTables, FITS files Visualisation tools

The Virtual Observatory AstroGrid Demonstration

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 4

I: The astronomer's world is changing...

Previously an Astronomer was a “radio astronomer” or a “x-ray astronomer”. Now you need to be an expert in all wavelengths.The move towards Survey AstronomyFederating data from different instruments.Vast quantities of auto-generated dataSee: http://grist.caltech.edu/sc4devo/What_is_the_VO.pdf

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 5

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 6

Supernova Remnant Cassiopeia-A – a 300

year old Supernova

Shocks seen in the X-ray Chandra image

Heavy elementsseen in the optical

Dust shows in the IR

Images from Chandra Science CentreSlide: Nic Walton, AstroGrid Project Scientist

Mapping e¯s in the magnetic field as revealed by Radio data

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 7

Data Deluge

(Ex-)Concorde(15 km)

CD stack with1 year LHC data(~ 20 km)

SuperCOSMOS: 2.5 Tb

WFCAM: 14Tb per yearVISTA: 4Tb per weekPhysicists, Astronomers, Biologists all facing the same problemSteve Lloyd: http://www.gridpp.ac.uk/talks/Data_Deluge_Brunel_2005.ppt

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 8

Nearby Spiral Galaxy

x-ray optical radio

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 9

II: Catalogue and Image ServersCone searchesData formats: FITS and VOTablesVisualising the data: Topcat and AladinMore sophisticated searches using SQLDemo of the SuperCOSMOS Science Archive

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 10

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 11

Example: SuperCosmos Science Archive

Scans of the UK Schmidt telescope platesCatalogue data & Image data3 wavebands, 2 epochsOver 1 billion objects2.5 terabytes in size

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 12

Data FormatsFITS – the astronomer's jpeg images/spectra/tables Includes metadata

VOTable “xml” format for

tables The VO standard

UCDs describe what a

quantity “is”

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 13

Visualising the dataTOPCAT For viewing and

manipulating tables, especially VOTables

Create new columns Filter the sources Output the table in

different formats

Aladin Good for viewing

image files Overlay catalogue

tables Create RGB images Access data archives

such as Vizier Aims to be a general

“Virtual Observatory” portal

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 14

Complex queries: SQLSQL is a general language for querying databases

select ra,dec,mag FROM sources where ra>0 and ra<1 and dec>-45 and dec<-44 and mag>23

give me ra, dec and mag values(use * to get everything)

table name(each db will have several) the following

conditions are true

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 15

SQL IIPlenty of resources on the web for learning SQL e.g. http://www.w3schools.com/sql/default.aspThere is an astronomy variant of SQL called ADQL (astronomy data query language)It's basically SQL with a few extra bits thrown in (a CIRCLE keyword for doing cone searches, and an XMATCH keyword for cross-matching data between databases)[Cross Matching: Taking two or more different database tables from (maybe) different sources and joining them together based on their RA and DEC]

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 16

Some databasesSIMBAD - Basic data on objectshttp://simbad.u-strasbg.fr/Simbad SuperCOSMOS Science Archive http://surveys.roe.ac.uk/ssa/ SDSS (Sloan Digital Sky Survey) http://www.sdss.org/2mass (2 micron all sky survey) http://www.ipac.caltech.edu/2mass/XMM http://xmm.vilspa.esa.es/external/xmm_data_acc/xsa/index.shtml

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 17

Databases of databasesVizier http://vizier.u-strasbg.fr/viz-bin/VizieRDatascope http://heasarc.gsfc.nasa.gov/cgi-bin/vo/datascope/init.plNED (Nasa Extragalactic Database) http://nedwww.ipac.caltech.edu/OpenSkyQuery http://openskyquery.net/Sky/skysite/

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 18

This presentation will be available at:

http://wiki.astrogrid.org/pub/Main/JohnTaylor/neon.sxi

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 19

A recap on yesterdayIs the SSA now behaving?Last night's observing group: how to use SQL to get your list of galaxies from SDSS

select g.ra,g.dec,g.objId, s.z from Galaxy g, ELRedShift s where g.specObjID=s.specObjID and s.z>0 and s.z<0.01 and g.r<10 and g.ra>0 and .........

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 20

III: Virtual Observatories :

What's a virtual observatory?What do you need to make one?AstroGrid – glue for the Euro-VODemo

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 21

The International Virtual Observatory

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 22

Virtual Observatory“The VO vision can be summed up as the desire to make all archives speak the same language − all searchable and analysable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.”Andy Lawrence, 09/2003

Take all the world's Astronomical data, and all the world's applications and computing power and make it appear that they reside on the user's desktop.

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 23

Science Cases – Brown Dwarf Location

Brown Dwarfs are cool, faint and (if can see them) closeRoughly speaking: Query optical and infrared catalogues for objects

that are “bright” in the infra red, but not in the optical and have large proper motions.

Requires use of more than one catalogueMore: “The AstroGrid 10” http://wiki.astrogrid.org/bin/view/Astrogrid/ScienceProblems

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 24

AstroGrid Components

PAL

PALPAL

CEA CEA

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 25

Example workflow: calculating photometric redshifts

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 26

AstroGrid Components

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 27

Demo

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 28

Contacts and info

Contacts: John Taylor jdt@roe.ac.uk Institute for Astronomy,

Royal Observatoy of Edinburgh

Background Info on AstroGrid http://www.astrogrid.org

Download the Euro-VO AstroGrid software from: http://software.astrogrid.org

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 29

What now?

AstroGrid Release 1 is up and running now

http://www.astrogrid.org/release

http://wiki.astrogrid.org/bin/view/Astrogrid/ReleaseV1Pages

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 30

Locating data and applications

IVO-standard registryRegistries harvest from each otherXml-based query language ADQL

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 31

WorkflowOrchestrate servicesQueries and workflows are designed using the portal – other clients on the wayWork is run remotely and asynchronously Archives searched and results manipulatedResults are stored in a virtual file system Queries and workflows can be re-used and shared

A PPARC funded project

AstroGrid Intro & Demo

John TaylorInstitute for Astronomy,

Edinburgh

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 33

Building a VO - Layer Diagram