Introduction to Observational Catalogues & Databases
Duncan Law-Green(LEDAS Archive Scientist)
21st February 2007
IntroductionIntroduction
What this seminar is aboutWhat this seminar is aboutIntroduction to selection of online astronomical Introduction to selection of online astronomical data resources to help with your research. data resources to help with your research. Outline of data formats and catalogue tools. Outline of data formats and catalogue tools. Practical demos.Practical demos.
What this seminar is not aboutWhat this seminar is not aboutData reduction and analysis techniques.Data reduction and analysis techniques.Computer science of databases, SQL Computer science of databases, SQL programming, how to build your own databases.programming, how to build your own databases.
Introduction to Observational Catalogues & Databases: 21/02/2007
Use of Catalogues/ArchivesUse of Catalogues/Archives Literature searchesLiterature searches
Build science case for your proposalBuild science case for your proposal Archive retrievalArchive retrieval
Previous observations of your objectsPrevious observations of your objects Feasibility studiesFeasibility studies
Is the observation possible with your instrument?Is the observation possible with your instrument? Observation planningObservation planning
Source positions, instrument FOV, guide stars etc.Source positions, instrument FOV, guide stars etc. Class studiesClass studies
Filter catalogues for interesting sources/outliersFilter catalogues for interesting sources/outliers Cross-correlationCross-correlation
Compare source positions in different cataloguesCompare source positions in different catalogues
Data & MetadataData & Metadata
DataDataA set of measured parameters for a source.A set of measured parameters for a source.
MetadataMetadata Data Data aboutabout the dataset, provides vital context info. the dataset, provides vital context info.
Examples include: coordinate epoch, date of Examples include: coordinate epoch, date of observations, observing mode, filter bandpass, observations, observing mode, filter bandpass, pipeline processing performed etc.pipeline processing performed etc.
Data without metadata is Data without metadata is uselessuseless..Some data formats preserve metadata, some don't.Some data formats preserve metadata, some don't.
Introduction to Observational Catalogues & Databases: 21/02/2007
Data Formats IData Formats I
ASCII ASCII (CSV/TSV: Comma/Tab-Separated Variables)(CSV/TSV: Comma/Tab-Separated Variables)
Advantages:Advantages: Easy to generate, easy to read. Easy to generate, easy to read. Will ingest directly into, e.g. ExcelWill ingest directly into, e.g. Excel
Disadvantages:Disadvantages: No direct metadata support. No direct metadata support. No integrity support. No integrity support. No documentation. No documentation. Bulky.Bulky.
Data Formats IIData Formats II
FITS (NOAO), HDS, NDF (Starlink)FITS (NOAO), HDS, NDF (Starlink)
Advantages:Advantages: Structured formats, include metadata. Structured formats, include metadata. Multi-dimensional (tables, images, datacubes). Multi-dimensional (tables, images, datacubes). Well-defined formats, good software support. Well-defined formats, good software support. Binary format, compact.Binary format, compact.
Disadvantages:Disadvantages: Varying compatibility between FITS, HDS, NDF.Varying compatibility between FITS, HDS, NDF.Conversion may affect metadata.Conversion may affect metadata.No semantics.No semantics.
Data Formats IIIData Formats III
VOTable (IVOA, AstroGrid)VOTable (IVOA, AstroGrid)
Advantages:Advantages: Structured format, metadata+Structured format, metadata+semanticssemantics. . Human-readable, supported by modern software. Human-readable, supported by modern software. Uses XML, existing tools to generate and check Uses XML, existing tools to generate and check integrity.integrity.
Disadvantages:Disadvantages: Uses XML – very bulky. Uses XML – very bulky. Multidimensional support awkward. Multidimensional support awkward. Standards evolving.Standards evolving.
SemanticsSemantics
What a data column means in physical termsWhat a data column means in physical terms Unified Content Descriptors (UCDs)Unified Content Descriptors (UCDs)
Created by IVOA (International Virtual Observatory Created by IVOA (International Virtual Observatory Alliance) as standard controlled vocabulary of Alliance) as standard controlled vocabulary of keywords to describe physical nature of table keywords to describe physical nature of table columns. Current system “UCD1+”columns. Current system “UCD1+”
R.A. (Main):R.A. (Main): pos.eq.ra;meta.main pos.eq.ra;meta.main Source ID:Source ID: meta.id;srcmeta.id;srcRadio flux ratio:Radio flux ratio: phot.flux;em.radio;arith.ratiophot.flux;em.radio;arith.ratio
UCDs are feature of well-constructed VOTables. UCDs are feature of well-constructed VOTables. Intended to ease automated data handling, “workflows”Intended to ease automated data handling, “workflows”
VOTableVOTable<?xml version="1.0"?><VOTABLE version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/VOTable/v1.1"> <COOSYS ID="J2000" equinox="J2000." epoch="J2000." system="eq_FK5"/> <RESOURCE name="myFavouriteGalaxies"> <TABLE name="results"> <DESCRIPTION>Velocities and Distance estimations</DESCRIPTION> <PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel" unit="m" value="3.6"/> <FIELD name="RA" ID="col1" ucd="pos.eq.ra;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Dec" ID="col2" "pos.eq.dec;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Name" ID="col3" ucd="meta.id;meta.main" datatype="char" arraysize="8*"/> <FIELD name="RVel" ID="col4" ucd="src.veloc.hc" datatype="int" width="5" unit="km/s"/> <FIELD name="e_RVel" ID="col5" ucd="stat.error;src.veloc.hc" datatype="int" width="3" unit="km/s"/> <FIELD name="R" ID="col6" ucd="phys.distance" datatype="float" width="4" precision="1" unit="Mpc"> <DESCRIPTION>Distance of Galaxy, assuming H=75km/s/Mpc</DESCRIPTION> </FIELD>
continued on next slide...
VOTable (cont.)VOTable (cont.)
<DATA> <TABLEDATA> <TR> <TD>010.68</TD><TD>+41.27</TD><TD>N 224</TD><TD>-297</TD><TD>5</TD><TD>0.7</TD> </TR> <TR> <TD>287.43</TD><TD>-63.85</TD><TD>N 6744</TD><TD>839</TD><TD>6</TD><TD>10.4</TD> </TR> <TR> <TD>023.48</TD><TD>+30.66</TD><TD>N 598</TD><TD>-182</TD><TD>3</TD><TD>0.7</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE></VOTABLE>
continued from previous slide...
TreeviewTreeview File format viewerFile format viewer
Can read multiple file formats, display hierarchical Can read multiple file formats, display hierarchical structures, expand and collapse nodes with click of structures, expand and collapse nodes with click of mouse. Some basic plotting, image, stats routines.mouse. Some basic plotting, image, stats routines.
Literature SearchLiterature Search ADS: NASA Astrophysics Data ServiceADS: NASA Astrophysics Data Service
3 bibliographic databases3 bibliographic databasesAstronomy & AstrophysicsAstronomy & Astrophysics (1.2 million) (1.2 million)PhysicsPhysics (3.6 million) (3.6 million)ArXiv e-printsArXiv e-prints (400,000) (400,000)
Searchable by author, subject, title, object, Searchable by author, subject, title, object, abstract text, full-text...abstract text, full-text...
Links to full PDFs of articles, object catalogues, Links to full PDFs of articles, object catalogues, data tables.data tables.
MyADS Update Service: subscribe to updatesMyADS Update Service: subscribe to updates
Not peer-reviewed!
adsabs.harvard.edu
““Telegrams”Telegrams”
IAU Circulars: Central Bureau for IAU Circulars: Central Bureau for Astronomical TelegramsAstronomical TelegramsCentral clearinghouse for info on transient events Central clearinghouse for info on transient events (comets, solar system bodies, novae, supernovae (comets, solar system bodies, novae, supernovae etc.). Subscribe by email. etc.). Subscribe by email. Search via ADSSearch via ADS
Astronomers' TelegramAstronomers' TelegramPrimarily high-energy transient events (GRBs etc.). Primarily high-energy transient events (GRBs etc.). Subscribe via email or RSS. Searchable web Subscribe via email or RSS. Searchable web interface, mirror @ LEDASinterface, mirror @ LEDAS
Data ServersData Servers Catalogue ServersCatalogue Servers
Surveys and article data. Searchable by position, filter by Surveys and article data. Searchable by position, filter by various parameters, output data in ASCII, FITS, VOTable. various parameters, output data in ASCII, FITS, VOTable. (examples: NED, HEASARC, LEDAS, ViZieR)(examples: NED, HEASARC, LEDAS, ViZieR)
Image ServersImage ServersImages of the sky at various wavelengthsImages of the sky at various wavelengths.. Output data in Output data in bitmap (GIF,JPG,PNG) or FITS image. May or may not be bitmap (GIF,JPG,PNG) or FITS image. May or may not be “science grade”. (examples: DSS-I/II, SDSS, SkyView, Aladin)“science grade”. (examples: DSS-I/II, SDSS, SkyView, Aladin)
Archive ServersArchive ServersRepository of public data from particular observatory or Repository of public data from particular observatory or mission. Various formats, may need specialist software or mission. Various formats, may need specialist software or training to interpret. (examples: Hubble, MERLIN, Chandra, training to interpret. (examples: Hubble, MERLIN, Chandra, LEDAS).LEDAS).
AladinAladin Advanced image and catalogue search systemAdvanced image and catalogue search system
Plot catalogue search results directly on imagePlot catalogue search results directly on image
Highly versatile, write scripts for repetitive Highly versatile, write scripts for repetitive operations.operations.
Launch via CDS website, LEDAS site or directly on Launch via CDS website, LEDAS site or directly on desktop.desktop.
RTFM!RTFM!
Introduction to Observational Catalogues & Databases: 21/02/2007
AladinAladin
Introduction to Observational Catalogues & Databases: 21/02/2007
Plane stack
Object data
Preview
Button bar
Main window
AladinAladin
Introduction to Observational Catalogues & Databases: 21/02/2007
Multiview option
Splitscreen button
TOPCATTOPCAT Catalogue plotting, editing and filtering toolCatalogue plotting, editing and filtering tool Cross-correlations between cataloguesCross-correlations between catalogues
Introduction to Observational Catalogues & Databases: 21/02/2007
TOPCATTOPCAT
Cross-correlationCross-correlation
Search for matching positions in 2 or more Search for matching positions in 2 or more catalogues (e.g. “does this X-ray source have a catalogues (e.g. “does this X-ray source have a radio counterpart?”)radio counterpart?”)
Consider positional uncertainties, statistical Consider positional uncertainties, statistical probability of chance coincidenceprobability of chance coincidence
Convenient tool for 2, 3, 4-way catalogue Convenient tool for 2, 3, 4-way catalogue matches in TOPCAT (Joins -> Pair Match etc.)matches in TOPCAT (Joins -> Pair Match etc.)
Introduction to Observational Catalogues & Databases: 21/02/2007
Virtual Observatory (VO)Virtual Observatory (VO)
International project to simplify access to International project to simplify access to astronomical catalogues and archives. astronomical catalogues and archives. Coordinated by IVOA.Coordinated by IVOA.
Standard set of access commands (“protocols”), Standard set of access commands (“protocols”), all databases “appear” the same on the network.all databases “appear” the same on the network.
UK VO project UK VO project AstroGridAstroGrid, developed additional , developed additional software for distributed search, distributed software for distributed search, distributed storage, workflows etc.storage, workflows etc.
Introduction to Observational Catalogues & Databases: 21/02/2007
AstroGrid WorkbenchAstroGrid Workbench Workbench: Desktop Java application for VO Workbench: Desktop Java application for VO
searches and data processing. searches and data processing.
Check availability of dataCheck availability of data
Execute simultaneous searches across multiple Execute simultaneous searches across multiple catalogues/serverscatalogues/servers
Construct Construct “workflows”“workflows”: drag+drop editing of : drag+drop editing of data gathering/reduction/analysis pipelines.data gathering/reduction/analysis pipelines.
Save results to “MySpace”, temp scratch spaceSave results to “MySpace”, temp scratch space
Introduction to Observational Catalogues & Databases: 21/02/2007
VOPlotVOPlot
Product of VO India projectProduct of VO India project
Reading, interactive plotting and filtering tool for Reading, interactive plotting and filtering tool for catalogue data (primarily VOTables)catalogue data (primarily VOTables)
Introduction to Observational Catalogues & Databases: 21/02/2007
And finally...And finally...
Stellarium (Stellarium (http://www.stellarium.org/http://www.stellarium.org/))Free planetarium software, impress your friends!Free planetarium software, impress your friends!
..
..
..
Questions to Questions to [email protected]@star.le.ac.uk
Seminar slides, URLs to appear on my webspaceSeminar slides, URLs to appear on my webspacehttp://www.star.le.ac.uk/~dlg/http://www.star.le.ac.uk/~dlg/
Introduction to Observational Catalogues & Databases: 21/02/2007