iSERVOiSERVOInternational Solid Earth International Solid Earth

Research Virtual ObservatoryResearch Virtual Observatory Grid/Web Services and Portals Supporting Grid/Web Services and Portals Supporting

Earthquake ScienceEarthquake Science

December 15 2004 December 15 2004 AGU Fall Meeting San Francisco AGU Fall Meeting San Francisco

Geoffrey Fox, Marlon PierceGeoffrey Fox, Marlon Pierce(Community Grids Lab, Pervasive Technologies Laboratories, (Community Grids Lab, Pervasive Technologies Laboratories,

Indiana University)Indiana University)John Rundle John Rundle (UC Davis),(UC Davis),

Andrea Donnellan, Robert Granat, Greg Lyzenga, Jay Parker Andrea Donnellan, Robert Granat, Greg Lyzenga, Jay Parker (JPL)(JPL)Don McLeod Don McLeod (USC),(USC), Lisa Grant Lisa Grant (UC Irvine)(UC Irvine)

Database Database

Analysis and VisualizationPortal

RepositoriesFederated Databases

Data Filter

Services

Field Trip DataStreaming Data

Sensors

?DiscoveryServices

SERVOGrid

ResearchSimulations

Research Education

CustomizationServices

From Research

to Education

EducationGrid ComputerFarmGrid of Grids: Research Grid and Education Grid

GISGrid

Sensor GridDatabase Grid

Compute Grid

iSERVO in a nutshelliSERVO in a nutshell Designed to link Designed to link data-setsdata-sets (repositories and real time), (repositories and real time),

computationscomputations and and earthquake scientistsearthquake scientists in ACES (Asia in ACES (Asia Pacific) CooperationPacific) Cooperation• Australia China Japan USAAustralia China Japan USA

Exemplified by Exemplified by SERVOGrid SERVOGrid in USA led by in USA led by JPLJPL Supports Supports simulationsimulation and and datamining datamining as servicesas services Adopts conservative Adopts conservative WS-I+ Web Service InteroperabilityWS-I+ Web Service Interoperability

standardsstandards Builds full “Grid” in a library fashion as a Builds full “Grid” in a library fashion as a Grid of GridsGrid of Grids

• GISGIS (Geographic Information System) (Geographic Information System) GridGrid built as a set of built as a set of OGCOGC compatible Web Services “talking” compatible Web Services “talking” GMLGML

• iSERVO federatesiSERVO federates separate Grids in each separate Grids in each country/organization/functioncountry/organization/function

• A Grid is A Grid is “just” a collection of Services“just” a collection of Services aka distributed programs aka distributed programs Multi-scaleMulti-scale simulations supported by simulations supported by Grid workflowGrid workflow Portals Portals based on NSF Middleware Initiative NMI based on NSF Middleware Initiative NMI Open Open

Grid Computing EnvironmentGrid Computing Environment OGCE OGCE

Characteristics of Computing for Characteristics of Computing for Solid Earth ScienceSolid Earth Science

Widely distributed datasetsWidely distributed datasets in various formats in various formats• GPS, Fault data, Seismic data sets, InSAR satellite dataGPS, Fault data, Seismic data sets, InSAR satellite data• Many available in state of art tar files that can be FTP’dMany available in state of art tar files that can be FTP’d• Provenance problems: faults have controversial parameters like Provenance problems: faults have controversial parameters like

slip rates which have to be estimated.slip rates which have to be estimated. Distributed models and expertiseDistributed models and expertise

• Lots of codes with different regions of validity, ranging from Lots of codes with different regions of validity, ranging from cellular automata to finite element to data mining applications cellular automata to finite element to data mining applications (HMM) (HMM)

• Simplest challenges are just making these codes useable for Simplest challenges are just making these codes useable for other researchers.other researchers.

• And hooking this codes to data sourcesAnd hooking this codes to data sources• Some codes also have export or IP restrictionsSome codes also have export or IP restrictions• Other codes are highly specialized to their deployment Other codes are highly specialized to their deployment

environments.environments. Decomposable problemsDecomposable problems requiring interoperability for linking full requiring interoperability for linking full

modelsmodels• The fidelity of your fault modeling can vary considerablyThe fidelity of your fault modeling can vary considerably• Link codes (through data) to support multiple scalesLink codes (through data) to support multiple scales

(i)SERVO Web (Grid) Services(i)SERVO Web (Grid) Services ProgramsPrograms: : All applications wrapped as Services using proxy strategyAll applications wrapped as Services using proxy strategy Job SubmissionJob Submission:: support remote batch and shell invocations support remote batch and shell invocations

• Used to execute simulation codes (VC suite, GeoFEST, etc.), mesh Used to execute simulation codes (VC suite, GeoFEST, etc.), mesh generation (Akira/Apollo) and visualization packages (RIVA, generation (Akira/Apollo) and visualization packages (RIVA, GMT).GMT).

File managementFile management::• Uploading, downloading, backend crossloading (i.e. move files Uploading, downloading, backend crossloading (i.e. move files

between remote machines) between remote machines) • Remote copies, renames, etc.Remote copies, renames, etc.

Job monitoringJob monitoring WorkflowWorkflow: Apache Ant-based remote service orchestration: Apache Ant-based remote service orchestration

• For coupling related sequences of remote actions, such as RIVA For coupling related sequences of remote actions, such as RIVA movie generation.movie generation.

Data servicesData services: support remote data bases and query construction: support remote data bases and query construction• XML data model being adopted for common formats with XML data model being adopted for common formats with

translation services to “legacy” formats.translation services to “legacy” formats.• Migrating to Geography Markup Language (GML) descriptions.Migrating to Geography Markup Language (GML) descriptions.

Metadata ServicesMetadata Services: for archiving user session information.: for archiving user session information.

SERVOGrid ApplicationsSERVOGrid Applications Codes range from simple “rough estimate” codes to Codes range from simple “rough estimate” codes to

parallel, high performance applications.parallel, high performance applications.• DislocDisloc: handles multiple arbitrarily dipping dislocations : handles multiple arbitrarily dipping dislocations

(faults) in an elastic half-space.(faults) in an elastic half-space.• SimplexSimplex: inverts surface geodetic displacements for fault : inverts surface geodetic displacements for fault

parameters using simulated annealing downhill residual parameters using simulated annealing downhill residual minimization. minimization.

• GeoFESTGeoFEST: Three-dimensional viscoelastic finite element : Three-dimensional viscoelastic finite element model for calculating nodal displacements and tractions. model for calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, Allows for realistic fault geometry and characteristics, material properties, and body forces. material properties, and body forces.

• Virtual CaliforniaVirtual California: Program to simulate interactions : Program to simulate interactions between vertical strike-slip faults using an elastic layer over between vertical strike-slip faults using an elastic layer over a viscoelastic half-space a viscoelastic half-space

• RDAHMMRDAHMM: Time series analysis program based on Hidden : Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities Markov Modeling. Produces feature vectors and probabilities for transitioning from one class to another. for transitioning from one class to another.

Preprocessors, mesh generators: Preprocessors, mesh generators: AKIRA suiteAKIRA suite Visualization tools: Visualization tools: RIVARIVA, , GMTGMT,, IDL IDL

SERVOGrid Codes, RelationshipsSERVOGrid Codes, Relationships

Elastic DislocationPattern Recognizers

Fault Model BEM

Viscoelastic Layered BEM

Viscoelastic FEMElastic Dislocation Inversion

This linkage called Workflow in Grid/Web Service parlance

Role of WorkflowRole of Workflow

Programming the GridProgramming the Grid: Workflow describes linkage : Workflow describes linkage between servicesbetween services

As distributed, As distributed, linkage must be by messageslinkage must be by messages Linkage is two-way and has both control and dataLinkage is two-way and has both control and data Apply to multi-scale (Apply to multi-scale (complexitycomplexity) linkage, multi-) linkage, multi-

program linkage, link program linkage, link visualization to simulationvisualization to simulation, GIS to , GIS to simulations and viz filters to each othersimulations and viz filters to each other

Microsoft-IBM specification Microsoft-IBM specification BPELBPEL is current preferred is current preferred Web Service XML specification of workflowWeb Service XML specification of workflow

SERVOGrid uses ANT (well known XML build tool) to SERVOGrid uses ANT (well known XML build tool) to perform workflow and this works well in our relatively perform workflow and this works well in our relatively simple cases)simple cases)

Service-1 Service-3

Service-2

Applications and Observational DataApplications and Observational Data Several SERVO codes work Several SERVO codes work directly with observational directly with observational

datadata. . Scenarios includeScenarios include

• GeoFEST, VirtualCalifornia, Simplex, and Disloc all depend GeoFEST, VirtualCalifornia, Simplex, and Disloc all depend upon fault models.upon fault models.

• RDAHMM and Pattern Informatics codes use seismic RDAHMM and Pattern Informatics codes use seismic catalogs.catalogs.

• RDAHMM primarily used with GPS dataRDAHMM primarily used with GPS data ProblemProblem: We need to provide a way to integrate these : We need to provide a way to integrate these

codes with the online data repositories.codes with the online data repositories.• QuakeTables QuakeTables Fault DatabaseFault Database• Existing GPS and Earthquake Catalogs Existing GPS and Earthquake Catalogs

SolutionSolution: use databases to store catalog data; use XML : use databases to store catalog data; use XML ((GMLGML) as exchange data format; use ) as exchange data format; use OGC and WS-I+ OGC and WS-I+ Compatible Web ServicesCompatible Web Services for data exchanges, invoking for data exchanges, invoking queries, and filtering data.queries, and filtering data.• Use Web Feature Service, Web Map Service from OGCUse Web Feature Service, Web Map Service from OGC• Use UDDI (Discovery), WS-DAI (Database),WS-Context Use UDDI (Discovery), WS-DAI (Database),WS-Context

(Dynamic metadata) from WS-I+(Dynamic metadata) from WS-I+

SERVOGrid and Semantic GridSERVOGrid and Semantic Grid SERVOGrid has many types of metadataSERVOGrid has many types of metadata We are designing RDFS descriptions for the We are designing RDFS descriptions for the

following components:following components:• Simulation codes, mesh generators, etc.Simulation codes, mesh generators, etc.• Visualization toolsVisualization tools• Data typesData types• Computing resourcesComputing resources• ……

These are easily expressed as These are easily expressed as RDFSRDFS (actually (actually DAML) “nuggets” of information.DAML) “nuggets” of information.• Create instances of theseCreate instances of these• Use properties to link instances.Use properties to link instances.

Some Sample RelationshipsSome Sample Relationships

Danube

DislocApplication

Computer

FaultDataType

Stress MapDataFormat

USC Fault DB

Data Storage

GMTViz Appl

installedOn

storedInusesInput

createsOutput

visualizedBy

installedOn

Expanding to iSERVO Strategy• Agree on what (type of) resources and capabilities need to put on the

ISERVO Grid– Computers, instruments, databases, visualization, maps, job submittal

….• Agree on interfaces to resources from OGSA-DAI (databases) to

particular data structures (GML/OpenGIS) – specify in XML• Implement Resources and Capabilities as Services

– User Interface should be a portlet that can be integrated by the portal into web interface

• Make certain overarching Grid capabilities such as workflow, federation and metadata are sufficient

• SERVO Grid is a prototype of this strategy using several US sites rather than several countries– Can be naturally extended to iSERVO, education, emergency response

by extending resources• WS-I+ Web Service Architecture ensures continued interoperability and

extensibility

Grid Syntax Controversies• There are several proposals for the Web Service extensions

needed for Grids – OGSI (GT3), WSRF (GT4), WS-GAF (Newcastle)– We adopt a wait and see philosophy

• We use WS-I+ Pure Web Services approach that adopts minimum set of ~7 Web Service specifications choosing from 60 or so proposed in last few years– Those adopted by Industry wide WS-I Web Service

Interoperability group– Those declared by IBM and Microsoft– Any extra absolutely essential – This approach adopted by next phase of UK e-Science Program

Performance and Streaming

• Web Services are meant to exchange messages using SOAP which is very interoperable but very slow– Drastically reduces effective bandwidth

• Most real programs exchanges data via reading and writing binary files– Increases latency

• All Control Messages should use classic SOAP• All data messages use optimal binary

– Respect “SOAP Infoset” (Header and Body of Message)• Use streaming not file-based infrastructure to give better latency

and same technology for files and streaming sensors– Similar to using UNIX Pipes not directly files– http://www.naradabrokering.org

WS-1 WS-2

SERVOGrid Web PortalSERVOGrid Web Portal Package every Web Service Package every Web Service

with its own user interface with its own user interface as a document fragmentas a document fragment

Portlets are underlying Portlets are underlying technologytechnology

OGCE Open Grid OGCE Open Grid Computing Environment is Computing Environment is developing lots of useful developing lots of useful portletsportlets• ComputingComputing• GISGIS• Access Grid etc.Access Grid etc.

Core Web Services

Portlet User InterfaceComponents

Application Web Servicesand Workflow

Aggregate Portals

Portal ArchitecturePortal ArchitectureC

lient

s (P

ure

HTM

L, J

ava

App

let .

.)

Agg

rega

tion

and

Ren

derin

g

PortalInternalServices

Portlet Class

Portlet Class

Portlet Class

Portlet Class:WebForm

SERVOGrid(IU)

Web/Gridservice

Web/Gridservice

Web/Gridservice

Computing

Data Stores

Instruments

GridPortetc.

(Java)COG Kit

Clients Portal Portlets Libraries Services Resources

LocalPortlets

Remoteor ProxyPortlets

Hierarchical arrangement

OGCEOGCEConsortium

Individual portlet for the Proxy Manager

Use tabs or choose different portlets to navigate through interfaces to different services

2 Other Portlets

Each Servicehas its own portlet

SERVOGrid Portal Screen SERVOGrid Portal Screen ShotsShots