AstroGrid status SOFT the VO the future AstroGrid presentation to GSC Andy Lawrence July 2003

AstroGrid AstroGrid

• status

• SOFT

• the VO

• the future

AstroGrid presentation to GSC Andy Lawrence July 2003

AstroGrid presentation to GSC Andy Lawrence July 2003

status status

• six months into Phase B • team of 26 people (23.4 FTEs)• spent 1.5M (40% of budget) • 10,000 lines of code • working s/w released • working on infrastructure, standards

– not tools

Delivered in Phase-A Delivered in Phase-A

• requirements analysis• technology evaluation• comprehensive Phase A report• demonstration s/w (with AVO)• initial VO standards (with AVO and NVO)• respected position in UK e-sci programme• world ranking position in VO work

Delivered in Itn-01,02 Delivered in Itn-01,02 • basic engine room components

– registry schema, implementation, and query service– data access and job control methods– MySpace manager, explorer, and registry

• preliminary interface and contents– simple portal, no tools– populated with a handful of DCs and datasets

• further international/esci visibility– use of OGSI, OGSA-DAI, formed GGF-RG– defined IVOA standards process

• OGSI experiments– intercontinental visualisation with grid services

Planned for Itn03,04 Planned for Itn03,04

• engine room continued– consolidate Registry, Data Access, MySpace – add Auth/Auth, workflow tool (poss Triana)– Registry harvesting

• interface and contents– astronomer useable interface– meaningful contents– data federation services

• international programme– standards for Registry, VOQL

demonstrations demonstrations

• Jan 2003 : with AVO– modest technical advance – new tools, well integrated, excellent PR

• July 2003 : AGOC and Sydney– large technical advance– no new tools, minimal interface

• next steps– feedback from AGSAG beta testers crucial– wrap tools for next demo

TargetTarget

CurrentCurrent

successes successes

• well motivated hardworking team

• rigorous s/w eng processes

• code cut and released on schedule

• basic engine room components in place

• status in international VO

• status in astronomical community

opportunities opportunities

• our infrastructure could become standard

• additional tool access via AVO collaboration

failures failures

• portal work failed : GUI hacked fixed

• process take-up slow fixed

threats threats

• US diverging on registry standard• OGSA-DAI vs standard Grid• too little Data Centres effort

– incorporating datasets– writing services

Progress on AstroGrid goalsProgress on AstroGrid goals(1) develop standards for data, metadata, data exchange and provenance GOOD(2) develop a software infrastructure for data services GOOD(3) establish a physical grid of resources GOOD but DC effort worrying(4) construct and maintain a Service and Resource Registry GOOD(5) implement a working VO system of real scientific use to astronomers TBD NEXT SIX MONTHS(6) provide a user interface to that VO system BEHIND(7) provide or adapt a set of science user tools to work with the VO LACK OF FUNDING(8) establish a leading position for the UK in VO work GOOD

The VO is a system that will ..The VO is a system that will ..• allows users to interrogate multiple data centres in a

seamless and transparent way

• provide new powerful analysis and visualisation tools within that system

• give data centres a standard framework for publishing and delivering services using their data.

AstroGrid Phase A ReportOctober 2002

multi- views of a Supernova Remnant

Shocks seen in the X-ray

Heavy elementsseen in the optical

Dust seen in the IR

Relativistic electrons seen in the radio

What happens to the Earth's magnetosphere during a coronal mass ejection ?

Event imaged by space-based solar observatory

Effect detected later bysatellites and ground radar

needles in a haystackneedles in a haystack Hambly et al 2001

- faint moving object is a cool white dwarf- may be solution to the dark matter problem- but hard to find : one in a million- even harder across multiple archives

The VO is a system that will ..The VO is a system that will ..• allows users to interrogate multiple data centres in a

seamless and transparent way

• provide new powerful analysis and visualisation tools within that system,

• give data centres a standard framework for publishing and delivering services using their data.

VO-1

VO-2

VO-3

The VO is a system that will ..The VO is a system that will ..• allows users to interrogate multiple data centres in a

seamless and transparent way

• provide new powerful analysis and visualisation tools within that system,

• give data centres a standard framework for publishing and delivering services using their data.

VO-1

VO-2

VO-3

looking good

need effort for DCs to "publish and deliver"

almost untouched

the future : data servicesthe future : data services

• VO : geometry– not a warehouse; not a hierarchy; not P2P – small set of service centres + large popn of end-users

• VO : content– not software monolith : framework + standards – needs content : data services and user tools

• implies work for Data Centres– writing services; framework uptake; resource management– upcoming opportunities internationally competitive – establish a few centres as professionally competitive

yesterdayyesterday

browserfrontend

CGIrequest

html

web page

DBengine

SQL

data

todaytoday

appl

icat

ion

webservice

SOAP/XML request

SOAP/XML data

DBengine

SQL

nativedata

anyt

hin

g

standard formats

tomorrowtomorrow

appl

icat

ion

webservice

job

results

anyt

hin

g

webservice

webservice

webservice

webservice

webservice

Registry Workflow

GLUE AstroPass MySpace

standard semantics

publ

ish W

SDL

day after tomorrowday after tomorrow

appl

icat

ion

gridservice

job

results

anyt

hin

g

gridservice

gridservice

gridservice

gridservice

gridservice

Registry

Workflow GLUE AstroPass MySpace

poo

led

res

ourc

e

standard semantics

agents

ontology

work neededwork needed

appl

icat

ion

gridservice

job

results

anyt

hin

g

gridservice

gridservice

gridservice

gridservice

gridservice

Registry

Workflow GLUE AstroPass MySpace

poo

led

res

ourc

e

standard semantics

agents

TOOLS

STANDARDS

INFRASTRUCTURE

TECHNOLOGYRESEARCH

DATA SERVICES(access and analysis)

GRID UPTAKE

DA

TA

PIP

EL

INE

S

ontology

expected status 2004expected status 2004

• tools : ok but minimal

• standards : good but evolving

• glue : good but could be better

• data access services : ok but under-resourced

• data analysis services : almost untouched

• grid uptake : ready to deploy but resource worry

• technology research : treading water

AstroGrid-2 AstroGrid-2

• priority areas– core : evolve and improve, build tools

– establish content : data services, resource grid

– analysis services : visualisation, datamining : research and deploy

– next technology wave : ontology, agents

• expertise needed– existing AstroGrid developer team

– alliance of Data Centres

– add Computer Science expertise

CS expertiseCS expertise

• Leeds : visualisation experts

• Portsmouth : kd-tree algorithms

• Exeter : Intelligent Agents for astronomy

• commercial experience : Linde, Noddle

• local links : Bristol, Edinburgh, Cambridge

• external links : OGSA-DAI, GGF, MS, IBM

InfrastructureInfrastructure

• framework largely in place but..

• needs to be robust and reliable

• needs continuing work on standards– IVOA and Grid

• needs suite of tools

• key idea is personalisation– tool requests from competitive calls

delivers VO-1

R&DR&D

• really mostly D– assess, customise, integrate

• keep on the next wave– ontology, agents

• keep promise that everybody can be a power user– large volume and multi-D visualisation and datamining services

delivers VO-2and re-newed VO-1

Data CentresData Centres

• Data Centres are the heart of the VOthere is no UK VO without well resourced DCs

• top priority is support for VO/uptake

• second priority is core support for DCs– not mission specific resources

– not pipeline or archive development

– basic physical resource management and data curation

delivers VO-1

delivers VO-3

Costed Model OptionsCosted Model Options

• M1 : full programme £9.25M• M2 : full research £6.68M

– lose DCA core support

– world beating VO infrastructure but UK content compromised

• M3 : partial research £5.28M– lose some lines and trim others; minimise new algorithms

– good working VO but loses opportunity for UK lead in new technology

– approximately flat continuation of AG1-PhaseB

• M4 : minimum viable £3.75M– no research; unlikely to lead any third wave of VO development

– working VO but relies on external components

FIN