Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility

Aus-VO:Aus-VO: Progress in the Australian Progress in the Australian Virtual ObservatoryVirtual Observatory

Tara MurphyTara MurphyAustralia Telescope National Australia Telescope National FacilityFacility

Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune Tara Murphy Australia Telescope National Facility IVOA Small Projects Meeting, Pune 20042004

Aus-VO: who we areAus-VO: who we are

Science infrastructure effort, funded by ARC LIEF Science infrastructure effort, funded by ARC LIEF grantsgrants

The Aus-VO Partner Institutes are:The Aus-VO Partner Institutes are:– CSIRO Australian Telescope National FacilityCSIRO Australian Telescope National Facility– The University of MelbourneThe University of Melbourne– The University of SydneyThe University of Sydney– Anglo-Australian ObservatoryAnglo-Australian Observatory– Swinburne University of TechnologySwinburne University of Technology– The University of QueenslandThe University of Queensland– Monash UniversityMonash University– The University of New South WalesThe University of New South Wales– Victorian Partnership for Advanced ComputingVictorian Partnership for Advanced Computing– ANU Mount Stromlo Observatory & Supercomputing FacilityANU Mount Stromlo Observatory & Supercomputing Facility

Plus collaborators:Plus collaborators:– CSIRO Mathematical and Information SciencesCSIRO Mathematical and Information Sciences– Australian Partnership for Advanced ComputingAustralian Partnership for Advanced Computing


APAC Astronomy GridAPAC Astronomy Grid

Grid infrastructure effortGrid infrastructure effort Funded by the Australian Partnership Funded by the Australian Partnership

for Advanced Computing (APAC)for Advanced Computing (APAC) Active members & partners are:Active members & partners are:

– Melbourne and SwinburneMelbourne and Swinburne– UNSW and SydneyUNSW and Sydney– ANUANU

Project Leader: David BarnesProject Leader: David Barnes Research Leader: Rachel WebsterResearch Leader: Rachel Webster


Aus-VO: our focusAus-VO: our focus

Bring legacy astronomy archives on-line Ensure future project compliance Provide access to archived realizations of

simulations Describe all data fully, and support a small

and well-chosen set of interoperability protocols

Develop tools and interfaces to find, acquire, process and visualize data

Build national grid infrastructure to host the data, tools and interfaces


Aus-VO: what we doAus-VO: what we do

Science contentScience content– providing dataproviding data– providing softwareproviding software– metadatametadata– registry contentregistry content

Science services– source findingsource finding– coordinate transformationscoordinate transformations– catalogue matchingcatalogue matching– visualizationvisualization– interfaces to legacy environmentsinterfaces to legacy environments


APAC Grid: what we doAPAC Grid: what we do

Data grids– data store– data access protocols– authentication & authorisation– network facilities

Compute grids– workflow– job configuration, submission & monitoring– code-to-data paradigms


Aus-VO: current Aus-VO: current researchresearch Data mining

– catalogue matching– searches for new empirical relationships & rare

objects Legacy software integration

– using legacy code in VO services– using VO services in legacy software environments– transparent access to Web and Grid Services from

the desktop Visualisation

– remote visualisation service (legacy s/w drawing to remote “canvases” via Corba, SOAP, …)

– VOTable to VRML service– volume rendering


Aus-VO: projects 2004Aus-VO: projects 2004

Common format on-line archive projects:– HIPASS catalog: HI Parkes All Sky Survey: neutral

Hydrogen spectral line survey, 4,300 sources with ~30 parameters and 1024-channel radio spectra

– SUMSS catalog: Sydney University Molonglo Sky Survey: radio continuum survey at 843 MHz, 107,000 sources with ~15 parameters

– 2dFGRS QSO catalog: 2-degree Field Galaxy Redshift Survey: optical spectra of >20,000 southern quasi-stellar objects

– ATCA archive: Australia Telescope Compact Array archive: all observations since 1988, circa 1.5 TB of more than 1,000 separate observing projects! Substantial exercise in describing data with metadata.

– MACHO, RAVE, Pulsar timing archives, Gemini archive and more in 2004!


Aus-VO: projects 2004Aus-VO: projects 2004

Server- and client-based visualisation tools:– client canvas for legacy software package AIPS++

to draw upon from a remote server.– 3d visualisation of catalogues

Data reduction pipelines for:– Molonglo Observatory Synthesis Telescope– AT Compact Array archive– Gemini, for Australian-used instruments

Portal for theoretical astrophysics jobs:– configuration and execution– monitoring– result retrieval and initial analysis and display


The Australia The Australia Telescope Online Telescope Online ArchiveArchive Consists of:Consists of:

1.1. an an archivearchive of all data observed with the of all data observed with the Australia Telescope Compact ArrayAustralia Telescope Compact Array

2.2. a data a data reduction pipelinereduction pipeline to process the to process the data for ‘quick and dirty’ imagesdata for ‘quick and dirty’ images

3.3. a remote a remote visualisationvisualisation system to inspect system to inspect the images on the client machinethe images on the client machine

People:People:– Peter Lamb, Tara Murphy, Anil Chandra, Peter Lamb, Tara Murphy, Anil Chandra,

Praveena Tokachichu, Chris OwenPraveena Tokachichu, Chris Owen


1. The archive1. The archive

All ATCA data since All ATCA data since 19901990

Total of 1.9 Tb Total of 1.9 Tb covering over 1000 covering over 1000 projectsprojects

Hosted on a machine Hosted on a machine with 2GB memory with 2GB memory and 6TB of disk and 6TB of disk

Back end is a 20 Back end is a 20 node Beowulf clusternode Beowulf cluster


1. The archive1. The archive

Challenges:Challenges:– Unifying metadata to fully describe Unifying metadata to fully describe

datadata– Capturing the intentions of the Capturing the intentions of the

observersobservers determining the calibration strategydetermining the calibration strategy

– Dealing with the significant Dealing with the significant minority of non-standard minority of non-standard observationsobservations


2. The pipeline2. The pipeline

Automatically runs complete data Automatically runs complete data reduction process for ATCA datareduction process for ATCA data

Underlying work done by AIPS++Underlying work done by AIPS++ Methods exposed as web services Methods exposed as web services

using Glish/Javausing Glish/Java Uses metadata from archive to Uses metadata from archive to

make decisions about reductionmake decisions about reduction


2. The pipeline2. The pipeline

Challenges:Challenges:– Dealing with legacy codeDealing with legacy code– Dealing with legacy data (via the Dealing with legacy data (via the

archive)archive)– Getting the pipeline to make Getting the pipeline to make

sensible decisions about how to sensible decisions about how to process the dataprocess the data


3. Visualization system3. Visualization system

A software system that allows scientists to view and A software system that allows scientists to view and analyse astronomical data that are stored in remote analyse astronomical data that are stored in remote locations.locations.

RVSServer

User

Data Centre

User connects to view data

Visualisation Data (Image) sent to user RVS response

Dataset sent to RVS

Browse data


3. Visualization system3. Visualization system

Challenges:Challenges:– PerformancePerformance

RVS Image data formatRVS Image data format Connection speeds between servicesConnection speeds between services

– Session ManagementSession Management How to coordinate data centres and How to coordinate data centres and

clients?clients? Caching of images for future sessionsCaching of images for future sessions


ATOA WalkthroughATOA Walkthrough

1.1. Query the archiveQuery the archive

2.2. Query resultsQuery results

3.3. Automatic data reductionAutomatic data reduction

4.4. Visualise dataVisualise data


Aus-VO: linksAus-VO: links

www.aus-vo.orgwww.aus-vo.org www.atnf.csiro.au/vowww.atnf.csiro.au/vo

RVS will be public in ~2 monthsRVS will be public in ~2 months The ATOA will be public in ~2 The ATOA will be public in ~2

monthsmonths The prototype pipeline will be public The prototype pipeline will be public

in ~3 monthsin ~3 months

Documents

Aus-VO: Progress in the Australian Virtual Observatory Tara Murphy Australia Telescope National Facility