AstroGrid: Past, Present & FutureAstroGrid: Past, Present & Future

Bob Mann Bob Mann Institute for AstronomyInstitute for Astronomy & & National e-Science National e-Science CentreCentre

University of EdinburghUniversity of Edinburgh

My backgroundMy background

Research astronomer, not CS or IT Research astronomer, not CS or IT – Interested in evolution of galaxies & Interested in evolution of galaxies &

clustersclusters Needs multiwavelength data Needs multiwavelength data

Finding & matching catalogues is Finding & matching catalogues is tedious!tedious!– Want to automate data integrationWant to automate data integration

Hence interest in the nascent AstroGrid projectHence interest in the nascent AstroGrid project Half-time post in AstroGrid (2001-2002)Half-time post in AstroGrid (2001-2002) Research LeaderResearch Leader at National e-Science at National e-Science

Centre (2002-2004), also half-timeCentre (2002-2004), also half-time

Two talks at this Two talks at this workshopworkshop Now:Now:

– AstroGrid: Past, Present & FutureAstroGrid: Past, Present & Future

Tomorrow:Tomorrow:– Astronomy and e-Science:Astronomy and e-Science:

Opportunities & ChallengesOpportunities & Challenges

AstroGridAstroGrid

Past, Present and FuturePast, Present and Future

Outline of talkOutline of talk

PastPast– Origins in UK e-Science ProgrammeOrigins in UK e-Science Programme

PresentPresent– Completion of AstroGrid-1 at end of 2004:Completion of AstroGrid-1 at end of 2004:

a working data grid for UK astronomya working data grid for UK astronomy FutureFuture

– AstroGrid-2 and VOTechAstroGrid-2 and VOTech How might AstroGrid and Aus-VO How might AstroGrid and Aus-VO

mesh? mesh?

PastPast

Origins in the UK e-Science Origins in the UK e-Science ProgrammeProgramme

The UK e-Science The UK e-Science ProgrammeProgramme

Major government initiativeMajor government initiative– 2001-2003: £118M = A$283M2001-2003: £118M = A$283M– 2004-2006: £115M = A$276M2004-2006: £115M = A$276M

Funding:Funding:– e-Science Core Programmee-Science Core Programme– Programmes for each Research CouncilProgrammes for each Research Council

Particle Physics & Astronomy: Particle Physics & Astronomy: £56M=A$134M£56M=A$134M

– Astronomy: ~£12M=A$29MAstronomy: ~£12M=A$29M AstroGrid: ~£8M=A$19MAstroGrid: ~£8M=A$19M

e-Science Core e-Science Core ProgrammeProgramme

Network of Regional e-Science Centres and Centres of Excellence

Open Middleware Infrastructure Institute

Digital Curation Centre

AstroGrid AstroGrid

Funded by PPARC e-Science Funded by PPARC e-Science programmeprogramme– £3.7M = A$8.9M in 2001-2004 (AG1)£3.7M = A$8.9M in 2001-2004 (AG1)– £4M = A$9.6M in 2005-2007 (AG2)£4M = A$9.6M in 2005-2007 (AG2)

““to build a to build a data griddata grid for UK astronomy, for UK astronomy, which will form the which will form the UK contribution UK contribution to a global Virtual Observatoryto a global Virtual Observatory””

AstroGrid is about VO AstroGrid is about VO infrastructureinfrastructure

AstroGrid-1: 2001-AstroGrid-1: 2001-20042004 AG1 Phase A: Sept 2001 – Dec AG1 Phase A: Sept 2001 – Dec

20022002– Analysis, R&D, ArchitectureAnalysis, R&D, Architecture

AG1 Phase B: Jan 2003 – Dec 2004AG1 Phase B: Jan 2003 – Dec 2004– Build, test & deliverBuild, test & deliver

~26 staff (23.4 FTE)~26 staff (23.4 FTE) Edinburgh, Leicester, Cambridge, Edinburgh, Leicester, Cambridge,

MSSL, Jodrell Bank, RAL, QUBMSSL, Jodrell Bank, RAL, QUB Part of EU-funded Astrophysical Part of EU-funded Astrophysical

Virtual Observatory (AVO) projectVirtual Observatory (AVO) project

Development Development ApproachApproach IterativeIterative

– full cycle each iterationfull cycle each iteration– 3 month(-ish) iterations3 month(-ish) iterations

IncrementalIncremental– add functionality each iterationadd functionality each iteration– refactor wherever necessaryrefactor wherever necessary

Use-case drivenUse-case driven– from ‘key 10’ science casesfrom ‘key 10’ science cases

Architecture ledArchitecture led

Some technical choicesSome technical choices

Open Source throughoutOpen Source throughout Modular architectureModular architecture Write everything in JavaWrite everything in Java Everything browser-basedEverything browser-based

– no standalone applicationsno standalone applications Build with Web Services firstBuild with Web Services first

– then move to Grid services (?)then move to Grid services (?) Open projectOpen project

– all docs & most debate on wiki & forumall docs & most debate on wiki & forum

PresentPresent

Completion of AstroGrid-1 at end of Completion of AstroGrid-1 at end of 2004:2004:a working data grid for UK a working data grid for UK astronomyastronomy

Where are we now?Where are we now?

Yellow = AG1Yellow = AG1 Mauve = AG2Mauve = AG2 Blue = externalBlue = external

We do have We do have the yellow bits the yellow bits working nowworking now…sort of …sort of

The pre-VO worldThe pre-VO world

NED DB DB DB CDS

Locally installed tools & local storage

AstroGrid SchematicAstroGrid Schematic

NED DB DB DB CDS

Locally installed tools & local storage

RegistryRegistry

Portal

MySpace

MSMSMS MS MS

Compute Node

Compute Node

MS

MS

Tool

AstroGrid SchematicAstroGrid Schematic

NED DB DB DB CDS

Locally installed tools & local storage

RegistryRegistry

Portal

MySpace

MSMSMS MS MS

Compute Node

Compute Node

MS

MS

Tool

Job Execution

AstroGrid SchematicAstroGrid Schematic

NED DB DB DB CDS

Locally installed tools & local storage

RegistryRegistry

Portal

MySpace

MSMSMS MS MS

Compute Node

Compute Node

MS

MS

Tool

Job Execution

MySpaceMySpace

Distributed file management systemDistributed file management system– In the VO, you won’t just use local diskIn the VO, you won’t just use local disk– Allows third-party delivery of dataAllows third-party delivery of data

User sees single logical view of all User sees single logical view of all filesfiles– Manipulates them with file browserManipulates them with file browser

Implementation a little clunky nowImplementation a little clunky now– Needs automation and scalability Needs automation and scalability

Community & Community & Auth/AuthAuth/Auth Most astronomical data is open accessMost astronomical data is open access

– But the VO must support proprietary rightsBut the VO must support proprietary rights VO workflow model needs single sign-onVO workflow model needs single sign-on

– Credentials must pass between servicesCredentials must pass between services AstroGrid needs some Authentication and AstroGrid needs some Authentication and

Authorization for the writable MySpaceAuthorization for the writable MySpace System is based on “Communities” from System is based on “Communities” from

which individuals’ rights are derivedwhich individuals’ rights are derived– Basic: more work on this in AstroGrid-2Basic: more work on this in AstroGrid-2

Common Execution Common Execution ArchitectureArchitecture

Remember: AstroGrid don’t do toolsRemember: AstroGrid don’t do tools– But we want to use existing onesBut we want to use existing ones

CEA provides web service wrapper to CEA provides web service wrapper to existing command line toolsexisting command line tools– CEA toolkit wraps and registers toolsCEA toolkit wraps and registers tools

All CEA-wrapped services have same WSDLAll CEA-wrapped services have same WSDL Easy to use – even by non-expertsEasy to use – even by non-experts

Developing this as an IVOA standardDeveloping this as an IVOA standard

FutureFuture

AstroGrid-2 and VOTechAstroGrid-2 and VOTech

AstroGrid-2 & VOTechAstroGrid-2 & VOTech

AstroGrid-2AstroGrid-2– £4M = A$9.6M for 2005-2007£4M = A$9.6M for 2005-2007– New partners, bringing in new expertise: New partners, bringing in new expertise:

Bristol, Exeter, Leeds, PortsmouthBristol, Exeter, Leeds, Portsmouth

VOTechVOTech– €€3.3M = A$5.5M for 2005-20073.3M = A$5.5M for 2005-2007– AstroGrid + ESO + French & Italian AstroGrid + ESO + French & Italian

groupsgroups

Completing the pictureCompleting the picture

AstroGrid-2 plansAstroGrid-2 plans

ExtensionExtension– InfrastructureInfrastructure– Grid (GT4/WSRF/?) integrationGrid (GT4/WSRF/?) integration– Intelligent resource discovery (semantic Intelligent resource discovery (semantic

grid)grid)– Full security: Full security:

Shibboleth/Internet2 integration?Shibboleth/Internet2 integration?

Additional componentsAdditional components– Data Mining & VisualizationData Mining & Visualization– Client-based access: Client-based access:

VO Workbench – not just web-based portalVO Workbench – not just web-based portal

VOTech plansVOTech plans

Design studies for a European VODesign studies for a European VO– New InfrastructureNew Infrastructure– New User ToolsNew User Tools– Intelligent Resource DiscoveryIntelligent Resource Discovery– Data ExplorationData Exploration

Coupling data mining and visualizationCoupling data mining and visualization

How might AstroGrid How might AstroGrid and Aus-VO mesh?and Aus-VO mesh?

AstroGrid AstroGrid vsvs Aus-VO Aus-VO

AstroGridAstroGrid– One of the big VO playersOne of the big VO players– Aims to deliver full VO infrastructureAims to deliver full VO infrastructure– Doesn’t do toolsDoesn’t do tools

Aus-VOAus-VO– Limited size – can’t do everythingLimited size – can’t do everything– World-leading strengths in some areasWorld-leading strengths in some areas

e.g. radio, optical spectroscopy, visualizatione.g. radio, optical spectroscopy, visualization

The projects seem very complementaryThe projects seem very complementary

How do we progress How do we progress this?this?

Download AstroGrid-1 (from mid-Dec)Download AstroGrid-1 (from mid-Dec)– ““AstroGrid-in-a-box” or the full releaseAstroGrid-in-a-box” or the full release– Deploy the bits you want – it’s all modularDeploy the bits you want – it’s all modular– Improve them (and give them back to us!)Improve them (and give them back to us!)

It’s all open sourceIt’s all open source

Set up an Aus-VO registry Set up an Aus-VO registry – of published datasets and of of published datasets and of

tools/servicestools/services What else?What else?

Summary & Summary & ConclusionsConclusions AstroGrid-1 finishes in December 2004AstroGrid-1 finishes in December 2004

– It will deliver a working VO infrastructureIt will deliver a working VO infrastructure AstroGrid-2 and VOTech will enhance AstroGrid-2 and VOTech will enhance

thatthat– And add tools – e.g. for data explorationAnd add tools – e.g. for data exploration

All AstroGrid code is open sourceAll AstroGrid code is open source– Take what you want; tell us/fix what’s badTake what you want; tell us/fix what’s bad

We should collaborate more than we doWe should collaborate more than we do

www.astrogrid.org www.astrogrid.org rgm@roe.ac.ukrgm@roe.ac.uk

Portal and WorkflowPortal and Workflow

Supports workflow paradigm (from the Supports workflow paradigm (from the simple to the complex)simple to the complex)

Queries and workflows are designed Queries and workflows are designed using the portalusing the portal

Work is run remotely & asynchronously Work is run remotely & asynchronously Archives searched & results manipulatedArchives searched & results manipulated Results are stored in a virtual file system Results are stored in a virtual file system Workflows can be re-used and sharedWorkflows can be re-used and shared

Enabling Resource DiscoveryEnabling Resource Discovery

Types of Registries:– Full, Publish, SpecialTypes of Registries:– Full, Publish, Special Registry is the main focal point for all Astrogrid Registry is the main focal point for all Astrogrid

componentscomponents Agreed Standards with IVOAAgreed Standards with IVOA

– Search and harvest interface Search and harvest interface – OAI standard for harvest interfaceOAI standard for harvest interface

Adheres to the Astrogrid CEA standardAdheres to the Astrogrid CEA standard Types of resourceTypes of resource

– Generic services, web services, applications, …Generic services, web services, applications, …– Data collectionsData collections– AstroGrid-specific resources (e.g. MySpace servers)AstroGrid-specific resources (e.g. MySpace servers)

Use of XQuery language with eXist XML database.Use of XQuery language with eXist XML database. Harvesting Harvesting

– US NVOUS NVO– VizieRVizieR

Access ControlAccess Control AstroGrid has taken the lead in AAA because our AstroGrid has taken the lead in AAA because our

system is based on writable storagesystem is based on writable storage Implement single-sign-on (SSO) authenticationImplement single-sign-on (SSO) authentication

– Using a community service (so, account can be used Using a community service (so, account can be used with a number of portals or other UIs)with a number of portals or other UIs)

Programme-to-programme authentication based on Programme-to-programme authentication based on message security with SOAP headers and WS-Securitymessage security with SOAP headers and WS-Security

Facade API to hide details from implementationsFacade API to hide details from implementations– Underneath, use JAX-RPC/Axis handlers to secure Underneath, use JAX-RPC/Axis handlers to secure

messages.messages. Little authorization support in 2004, more in 2005Little authorization support in 2004, more in 2005

– groups, quotas, community authorization, file groups, quotas, community authorization, file sharingsharing

Currently specific to AstroGridCurrently specific to AstroGrid– Working with IVOA to develop common standardWorking with IVOA to develop common standard

Common Execution Common Execution ArchitectureArchitecture Uniform interface for running applications in VObsUniform interface for running applications in VObs Set of WSDL and schemaSet of WSDL and schema

– Define the interactions in language independent wayDefine the interactions in language independent way Adapters (ExecutionControllers) implementedAdapters (ExecutionControllers) implemented

– Using Java Web servicesUsing Java Web services– To run legacy command-line applicationsTo run legacy command-line applications– To run legacy HTTP GET/POST apps within the frameworkTo run legacy HTTP GET/POST apps within the framework

ExecutionControllers ExecutionControllers – Allow asynchronous operation of the applicationsAllow asynchronous operation of the applications– Callback and polling mechanisms for delivery of resultsCallback and polling mechanisms for delivery of results

Tightly integrated with JES (Job Execution System)Tightly integrated with JES (Job Execution System)– Allows programmable workflowsAllows programmable workflows

Publisher's AstroGrid Publisher's AstroGrid LibraryLibrary

Task: What's involved in publishing data to the VO?Task: What's involved in publishing data to the VO? Library of components Library of components

– Make a data owners life a (relatively) straight-forward Make a data owners life a (relatively) straight-forward transition to a data owner/publishertransition to a data owner/publisher

Variety of ‘plugins’Variety of ‘plugins’– Cope with different collections of data (modes of storage)Cope with different collections of data (modes of storage)

To make life easier for installers To make life easier for installers and test how they might be presented to the VOand test how they might be presented to the VO

Ready-made web UIReady-made web UI– Suitable for astronomers to use, Suitable for astronomers to use, – Includes useful configuration & monitoring featuresIncludes useful configuration & monitoring features

VO-compatible interfaces for tools to access the dataVO-compatible interfaces for tools to access the data Features to communicate with other VO services Features to communicate with other VO services

– E.g. RegistriesE.g. Registries