NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 1
Virtual Observatories
...or how to do your research from a beach in the Bahamas rather than stuck up a cold
mountain in the middle of the night...
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 2
Who I amJohn Taylor: [email protected] on the AstroGrid and VOTech Virtual Observatory projectsMember of Edinburgh's Wide Field Astronomy Unit – curates catalogue and image data from SuperCOSMOS and (soon) WFCAM and (less soon) VISTASoftware engineer NOT Astronomer
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 3
OutlineThe data deluge Why science is changing The scale of the problem and technical issues
Catalogues How to use them VOTables, FITS files Visualisation tools
The Virtual Observatory AstroGrid Demonstration
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 4
I: The astronomer's world is changing...
Previously an Astronomer was a “radio astronomer” or a “x-ray astronomer”. Now you need to be an expert in all wavelengths.The move towards Survey AstronomyFederating data from different instruments.Vast quantities of auto-generated dataSee: http://grist.caltech.edu/sc4devo/What_is_the_VO.pdf
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 5
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 6
Supernova Remnant Cassiopeia-A – a 300
year old Supernova
Shocks seen in the X-ray Chandra image
Heavy elementsseen in the optical
Dust shows in the IR
Images from Chandra Science CentreSlide: Nic Walton, AstroGrid Project Scientist
Mapping e¯s in the magnetic field as revealed by Radio data
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 7
Data Deluge
(Ex-)Concorde(15 km)
CD stack with1 year LHC data(~ 20 km)
SuperCOSMOS: 2.5 Tb
WFCAM: 14Tb per yearVISTA: 4Tb per weekPhysicists, Astronomers, Biologists all facing the same problemSteve Lloyd: http://www.gridpp.ac.uk/talks/Data_Deluge_Brunel_2005.ppt
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 8
Nearby Spiral Galaxy
x-ray optical radio
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 9
II: Catalogue and Image ServersCone searchesData formats: FITS and VOTablesVisualising the data: Topcat and AladinMore sophisticated searches using SQLDemo of the SuperCOSMOS Science Archive
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 10
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 11
Example: SuperCosmos Science Archive
Scans of the UK Schmidt telescope platesCatalogue data & Image data3 wavebands, 2 epochsOver 1 billion objects2.5 terabytes in size
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 12
Data FormatsFITS – the astronomer's jpeg images/spectra/tables Includes metadata
VOTable “xml” format for
tables The VO standard
UCDs describe what a
quantity “is”
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 13
Visualising the dataTOPCAT For viewing and
manipulating tables, especially VOTables
Create new columns Filter the sources Output the table in
different formats
Aladin Good for viewing
image files Overlay catalogue
tables Create RGB images Access data archives
such as Vizier Aims to be a general
“Virtual Observatory” portal
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 14
Complex queries: SQLSQL is a general language for querying databases
select ra,dec,mag FROM sources where ra>0 and ra<1 and dec>-45 and dec<-44 and mag>23
give me ra, dec and mag values(use * to get everything)
table name(each db will have several) the following
conditions are true
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 15
SQL IIPlenty of resources on the web for learning SQL e.g. http://www.w3schools.com/sql/default.aspThere is an astronomy variant of SQL called ADQL (astronomy data query language)It's basically SQL with a few extra bits thrown in (a CIRCLE keyword for doing cone searches, and an XMATCH keyword for cross-matching data between databases)[Cross Matching: Taking two or more different database tables from (maybe) different sources and joining them together based on their RA and DEC]
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 16
Some databasesSIMBAD - Basic data on objectshttp://simbad.u-strasbg.fr/Simbad SuperCOSMOS Science Archive http://surveys.roe.ac.uk/ssa/ SDSS (Sloan Digital Sky Survey) http://www.sdss.org/2mass (2 micron all sky survey) http://www.ipac.caltech.edu/2mass/XMM http://xmm.vilspa.esa.es/external/xmm_data_acc/xsa/index.shtml
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 17
Databases of databasesVizier http://vizier.u-strasbg.fr/viz-bin/VizieRDatascope http://heasarc.gsfc.nasa.gov/cgi-bin/vo/datascope/init.plNED (Nasa Extragalactic Database) http://nedwww.ipac.caltech.edu/OpenSkyQuery http://openskyquery.net/Sky/skysite/
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 18
This presentation will be available at:
http://wiki.astrogrid.org/pub/Main/JohnTaylor/neon.sxi
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 19
A recap on yesterdayIs the SSA now behaving?Last night's observing group: how to use SQL to get your list of galaxies from SDSS
select g.ra,g.dec,g.objId, s.z from Galaxy g, ELRedShift s where g.specObjID=s.specObjID and s.z>0 and s.z<0.01 and g.r<10 and g.ra>0 and .........
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 20
III: Virtual Observatories :
What's a virtual observatory?What do you need to make one?AstroGrid – glue for the Euro-VODemo
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 21
The International Virtual Observatory
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 22
Virtual Observatory“The VO vision can be summed up as the desire to make all archives speak the same language − all searchable and analysable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.”Andy Lawrence, 09/2003
Take all the world's Astronomical data, and all the world's applications and computing power and make it appear that they reside on the user's desktop.
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 23
Science Cases – Brown Dwarf Location
Brown Dwarfs are cool, faint and (if can see them) closeRoughly speaking: Query optical and infrared catalogues for objects
that are “bright” in the infra red, but not in the optical and have large proper motions.
Requires use of more than one catalogueMore: “The AstroGrid 10” http://wiki.astrogrid.org/bin/view/Astrogrid/ScienceProblems
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 24
AstroGrid Components
PAL
PALPAL
CEA CEA
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 25
Example workflow: calculating photometric redshifts
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 26
AstroGrid Components
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 27
Demo
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 28
Contacts and info
Contacts: John Taylor [email protected] Institute for Astronomy,
Royal Observatoy of Edinburgh
Background Info on AstroGrid http://www.astrogrid.org
Download the Euro-VO AstroGrid software from: http://software.astrogrid.org
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 29
What now?
AstroGrid Release 1 is up and running now
http://www.astrogrid.org/release
http://wiki.astrogrid.org/bin/view/Astrogrid/ReleaseV1Pages
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 30
Locating data and applications
IVO-standard registryRegistries harvest from each otherXml-based query language ADQL
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 31
WorkflowOrchestrate servicesQueries and workflows are designed using the portal – other clients on the wayWork is run remotely and asynchronously Archives searched and results manipulatedResults are stored in a virtual file system Queries and workflows can be re-used and shared
A PPARC funded project
AstroGrid Intro & Demo
John TaylorInstitute for Astronomy,
Edinburgh
NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 33
Building a VO - Layer Diagram