OGSA-DAI Users’ Meeting

Introduction

Malcolm AtkinsonDirector

www.nesc.ac.uk

7th April 2004

Overview

Today’s MeetingUsers meet each otherWe hear Users’ Complaints, Requirements & PrioritiesUsers understand better the limits of one team’s workWe jointly identify the way ahead

OGSA-DAI VisionWhat we aspire to achieveShort, medium and long termRelationship with GGF DAIS WG Standardisation effort

OGSA-DAI TodayCurrent and Imminent Releases

The Road MapWhich you — the Users — should influence

OGSA-DAI History LessonDB & BiGUM meetings

August 2001 — Paul Watson paper — prime moverDatabase Task Force

01Q3 – 02Q3 Norman Paton, Dave Pearson & Paul Watson http://www.cs.man.ac.uk/grid-db/dbtf.html

Architecture Task Force01Q3 – 02Q2

Urged UK engagement in OGSA OGSI Sep 01 to Sep 03 http://www.nesc.ac.uk/teams/atf/01-03/

OGSA-DAI projectStart Feb 02 – end Sept 03Produced releases R1 to R3.1Produced Alpha Release of DQPhttp://www.ogsadai.org.uk/

DAIT projectStart Nov 03 – end ~ March 06Continues OGSA-DAI development

• GGF4 Toronto March 02•DAIS BoF •OGSI announced

• GGF5 Edinburgh July 02•DAIS WG launched

• GGF10 Berlin March 04•WS-Resource Framework

Oxford

Glasgow

Cardiff

Southampton

London

Belfast

Daresbury Lab

RAL

OGSA-DAI Partners

EPCC & NeSC

Newcastle

IBMUSA

IBM Hursley

Oracle

Manchester

Cambridge

Hinxton

$5 million, 20 months, started February 2002

Additional 24 months, starts October 2003

OGSA-DAI Vision 1

There are many sources of DataStructure is the essence of using dataStructure supported by DBMS: Relational & XMLStructure also recognised in files: BinX & GGF DFDL

Sources of Data are Autonomously ManagedVary in model, structure, schemaVary in contentVary in Access & Use policiesVary in Platform

Discovery & Design enabled by Combining Data

From multiple sourcesRecorded in multiple data resources

OGSA-DAI Vision 2

Developing applications using heterogeneous data sources is difficult:

Dealing with all those variationsNegotiating with data owners and data platformsCommunication protocols and data movementData translationDistributed operationsScale, reliability, coping with change, failure recovery, security, privacy…

Currently solved incrementally on a per application basis

OGSA-DAI to the rescue

OGSA-DAI Vision 3OGSA-DAI will provide a generic framework

Standard components for dealing with common requirements Query, update, bulk load, schema edit, metadata access, transactions, …

Agreed protocols and standard agreements with DBMSEstablished mechanisms for handling owner policy requirementsProvides data translationSupports distributed operationsDelivers

Scale, reliability, coping with change, failure recovery, security, privacy

…Costs of developing a robust OGSA-DAI amortised over many applicationsDevelopment of Applications significantly acceleratedAccepted de facto standard by DBMS vendors and data owners

Efficiency from direct support and system or resource knowledge

Collaborating International Community Extending the Framework

E.G. the work of OGSA-WebDB

OGSA-DAI Vision 4

Progressively develop the functionalityProgressively raise level of operationsProgressively improve quality

Scale, Performance, Reliability, Flexibility, UsabilityAdapt to changing contextSupport user community

Training, Documentation, Consultation, Collaboration

Place High Value on Users’ & Application Developers’ Ingenuity

They know about the data resourcesThey invent new ways of combining and analysing data

Therefore Extensibility and Flexibility MUST always be supported

OGSA-DAI today1919 Downloads so far from http://www.ogsadai.org.uk/

Release 4: April 04 from http://www.ogsadai.org.uk/

Release 5: October 04 from http://www.ogsadai.org.uk/

Releases 6&7: 2005: Many options — You must help us choose

Download analysis

Downloads by Domain

unresolved, 674

uk, 487

at, 44

gov, 13

net, 73

de, 87

jp, 126

ru, 9br, 11

it, 12

com, 170

edu, 133

And Now …

Users’ PresentationsWhat are you doing with our “baby”What do you likeWhat do you want changed, extended or addedWhat consensus is there

Team PresentationsMore details of current and planned releaseThe issues that face us

Choosing Interoperation Platforms Choosing which requirements get priority

Working GroupsCollation of the Groups’ conclusionsIdeas for Future User Group Activities

URGENTQuestions

?

Reserve Slides Follow

Database (Xindice, MySQL

Oracle, DB2)

Request to Registry for sources of data about “x”

Registry responds with Factory handle

Request to Factory for access to database

Factory creates GridDataService

Factory returns handle of GDS to client

Client queries GDS with SQL, XPath, XQuery etc

GDS interacts

with database

Query results

returned XML

SOAP/HTTP

service creation

API interactions

Analyst

RegistryGDSR

FactoryGDSF

Grid Data Service

GDS

Consumer

ORdelivered to consumer

as XML

OGSA-DAI Behaviours

Delivery Patterns

A G

Q

S + R

AG

Q + D

S

RC

A G

Q + U

S

A G

Q

S

UP

Retrieve Update/Insert Pipeline

A

G1 = PQ1 + D

S1

U/RA

G

Q

S

DC

R

A G

Q + D

S

IP

U

I

G2 = CS2

Q2

A

G1 = PQ1

S1

U/R

G2 = CS2

Q2 + DI

1.

2.

3.

4.

6.

5.

7.

8.

Statement Operations

Receives statementPerforms statement against a data resourceReturns results

Statement Component and Database Connections

Data Resource

Statement Results

<?xml version="1.0" encoding="UTF-8" ?> <GridDataServiceScript><Header> … Name, versioning, configuration specification </Header> <Body> <Documentation>Update with data delivered with the script</Documentation> <DefineParameter name="table">myimages</DefineParameter> <DefineParameter name="id">id = 2</DefineParameter> <Statement name="xyz" dataResource="MyDataResource"> SELECT * FROM <UseParameter reference="table" /> WHERE <UseParameter reference="id" /> </Statement> <Delivery name="delivery1"> <Mechanism type="bulk" /> <Mode type="full" /> <From>xyz</From> <To>response</To> </Delivery> <Execute name="execute1">xyz</Execute> </Body> </GridDataServiceScript>

Statement Document

Statement Architecture

Engine/GDS-S

JDBC Connection XMLDB Connection

JDBC API Relational DB

XML DBXML:DB API

Statement

Statement Activity

Statement Handlers

JDBC Statement Handlers

XMLDB Statement Handlers

OGSA-DAI Security

Grid Security Infrastructure AuthenticationThe role mapper takes credentials and a database name and returns a dbRoledbRole contains username and password

Role Mapper

dbRoleCredentials

Database Name

username

password

1a. Request to Registry for sources of data about “x”

1b. Registry responds with

Factory handle2a. Request to Factory for access to database

2c. Factory returns handle of GDS to client

3a. Client queries GDS with XPath, SQL, etc

3b. GDS interacts with database

3c. Results of query returned to client as XML

SOAP/HTTP

service creation

API interactions

Registry

Factory

2b. Factory creates GridDataService to manage access

Grid Data Service

Client

XML / Relational database

Data Access & Integration Services

GDTS 2 GDS 3

GDS2

GDTS 1

S x

S y

1a. Request to Registry for sources of data about “x” & “y”

1b. Registry responds with

Factory handle

2a. Request to Factory for access and integration from resources Sx and Sy

2b. Factory creates GridDataServices network

2c. Factory returns handle of GDS to client

3a. Client submits sequence of scripts each has a set of queries to GDS with XPath, SQL, etc

3c. Sequences of result sets returned to analyst as formatted binary described in a standard XML notation

SOAP/HTTP

service creation

API interactions

Data Registry

Data Access& Integrationmaster

Client

Analyst XML database

Relational database

GDS

GDS

GDS

GDTS

GDTS

3b. Client tells analyst

GDS1

Future DAI Services?

“scientific”Applicationcodingscientificinsights

ProblemSolving

Environment

SemanticMeta data

Application Code