Implementing Geographical Implementing Geographical Information System Information System

Services for SERVOGridServices for SERVOGrid

Marlon PierceMarlon Pierce

Community Grids LabCommunity Grids Lab

Indiana UniversityIndiana University

SERVOGrid ComponentsSERVOGrid Components Component (“portlet”)-based portals.Component (“portlet”)-based portals.

• OGCE mentioned by Chris HillOGCE mentioned by Chris Hill Web Services for “execution grid” servicesWeb Services for “execution grid” services

• Ant-based job specificationAnt-based job specification• File transferFile transfer• Distributed session management (“context”).Distributed session management (“context”).

Geographic Information System (GIS) services Geographic Information System (GIS) services for “data grid” services.for “data grid” services.• Web Map ServiceWeb Map Service• Web Feature ServiceWeb Feature Service• GIS-compatible information services.GIS-compatible information services.

Support for streaming, real-time data.Support for streaming, real-time data. Distributed service management/orchestrationDistributed service management/orchestration

• Using events and data streams.Using events and data streams.

Guiding PrinciplesGuiding Principles Grids are composed of families of servicesGrids are composed of families of services

• Data, execution, information, …Data, execution, information, … Use “WS-I+” approach to building service families.Use “WS-I+” approach to building service families.

• Build Grids out of Web Service standards conservatively.Build Grids out of Web Service standards conservatively. WS-Interoperability is the starting point.WS-Interoperability is the starting point.

• See position paper See position paper http://grids.ucs.indiana.edu/ptliupages/publications/WebShttp://grids.ucs.indiana.edu/ptliupages/publications/WebServiceGrids.pdferviceGrids.pdf

SOAP and WSDL provide universal messaging SOAP and WSDL provide universal messaging framework and service definition language.framework and service definition language.• All services should communicate with the same message All services should communicate with the same message

format.format.• Message delivery is left as an exercise.Message delivery is left as an exercise.

Implementations are interesting.Implementations are interesting.

Pattern Informatics (PI)Pattern Informatics (PI) PI is a technique developed at University of PI is a technique developed at University of

California, Davis for analyzing earthquake seismic California, Davis for analyzing earthquake seismic records to forecast regions with high future records to forecast regions with high future seismic activity.seismic activity.• They have correctly forecasted the locations of 15 of last They have correctly forecasted the locations of 15 of last

16 earthquakes with magnitude > 5.0 in California.16 earthquakes with magnitude > 5.0 in California. See Tiampo, K. F., Rundle, J. B., McGinnis, S. A., & See Tiampo, K. F., Rundle, J. B., McGinnis, S. A., &

Klein, W. Pattern dynamics and forecast methods Klein, W. Pattern dynamics and forecast methods in seismically active regions. in seismically active regions. Pure Ap. Geophys. Pure Ap. Geophys. 159, 2429-2467 (2002). 159, 2429-2467 (2002). • http://citebase.eprints.org/cgi-bin/fulltext?format=applichttp://citebase.eprints.org/cgi-bin/fulltext?format=applic

ation/pdf&identifier=oai%3AarXiv.org%3Acond-mat%2Fation/pdf&identifier=oai%3AarXiv.org%3Acond-mat%2F01020320102032

PI is being applied other regions of the world, and PI is being applied other regions of the world, and John has gotten a lot of press.John has gotten a lot of press.• Google “John Rundle UC Davis Pattern Informatics”Google “John Rundle UC Davis Pattern Informatics”

Pattern Informatics in a Grid Pattern Informatics in a Grid EnvironmentEnvironment

PI in a Grid environment:PI in a Grid environment:• Hotspot forecasts are made using publicly available seismic records.Hotspot forecasts are made using publicly available seismic records.

Southern California Earthquake Data CenterSouthern California Earthquake Data Center Advanced National Seismic System (ANSS) catalogsAdvanced National Seismic System (ANSS) catalogs

• Code location is unimportant, can be a service through remote Code location is unimportant, can be a service through remote executionexecution

• Results need to be stored, shared, modifiedResults need to be stored, shared, modified• Grid/Web Services can provide these capabilitiesGrid/Web Services can provide these capabilities

Problems:Problems:• How do we provide programming interfaces (not just user interfaces) How do we provide programming interfaces (not just user interfaces)

to the above catalogs?to the above catalogs?• How do we connect remote data sources directly to the PI code.How do we connect remote data sources directly to the PI code.• How do we automate this for the entire planet? How do we automate this for the entire planet?

Solutions:Solutions:• Use GIS services to provide the input data, plot the output dataUse GIS services to provide the input data, plot the output data

Web Feature Service for data archivesWeb Feature Service for data archives Web Map Service for generating mapsWeb Map Service for generating maps

• Use HPSearch tool to tie together and manage the distributed data Use HPSearch tool to tie together and manage the distributed data sources and code.sources and code.

WFS+

Seismic Rec.

WSDL

WFS+

State Bounds

WSDL

WMS+

OnEarth

“REST”

…

AggregatingWMS

Stubs

WFS+

Seismic Rec.

Stubs

WSDL

SOAPHTTP

GIS Behind the ScenesGIS Behind the Scenes The web features are served up by a Web Feature Service.The web features are served up by a Web Feature Service. Web Map Service aggregates mapsWeb Map Service aggregates maps

• NASA OnEarth + our own renderings.NASA OnEarth + our own renderings. We re-implement Open Geospatial Consortium standards We re-implement Open Geospatial Consortium standards

using Web Service Standards.using Web Service Standards.• SOAP messages, WSDL service definitions.SOAP messages, WSDL service definitions.• Will allow us to separate messages from HTTP transport layer Will allow us to separate messages from HTTP transport layer

in future.in future. More WMS Info:More WMS Info:

• http://grids.ucs.indiana.edu/ptliupages/publications/acm-gis-http://grids.ucs.indiana.edu/ptliupages/publications/acm-gis-sayar.pdf.sayar.pdf.

• http://grids.ucs.indiana.edu/ptliupages/publications/http://grids.ucs.indiana.edu/ptliupages/publications/Geoinformatics05_asayar.pdf.Geoinformatics05_asayar.pdf.

More WFS Info: More WFS Info: • http://grids.ucs.indiana.edu/ptliupages/publications/http://grids.ucs.indiana.edu/ptliupages/publications/

gwpap243.pdfgwpap243.pdf More general info, software, demos: More general info, software, demos:

http://www.crisisgrid.orghttp://www.crisisgrid.org

Tying It All Together: HPSearchTying It All Together: HPSearch HPSearchHPSearch is an engine for orchestrating is an engine for orchestrating

distributed Web Service interactionsdistributed Web Service interactions• It uses an event system and supports both file transfers It uses an event system and supports both file transfers

and and data streams.data streams.• Legacy nameLegacy name

HPSearch flows can be scripted with JavaScriptHPSearch flows can be scripted with JavaScript• HPSearch engine binds the flow to a particular set of HPSearch engine binds the flow to a particular set of

remote services and executes the script.remote services and executes the script. HPSearch engines are Web Services, can be HPSearch engines are Web Services, can be

distributed interoperate for load balancing.distributed interoperate for load balancing.• Boss/Worker modelBoss/Worker model

ProxyWebService:ProxyWebService: a wrapper class that adds a wrapper class that adds notification and streaming support to a Web notification and streaming support to a Web Service.Service.

More info: http://www.hpsearch.orgMore info: http://www.hpsearch.org

Data Filter(Danube)

PI Code Runner(Danube) Accumulate Data Run PI Code Create Graph Convert RAW -> GML

WFS(Gridfarm001)

WMS

HPSearch(TRex)

HPSearch(Danube)

HPSearch hosts an AXIS service for remote deployment of scripts

GML(Danube)

WS Context(Tambora)

NaradaBroker network: Used by HPSearch engines as well as for data transfer

Actual Data flow

HPSearch controls the Web services

Final Output pulled by the WMS

HPSearch Engines communicate using NB Messaging infrastructure

Virtual Data flow

Data can be stored and retrieved from the 3rd part repository (Context Service)

WMS submits script execution request (URI of script, parameters)

Support for Real Time Support for Real Time ApplicationsApplications

RDAHMM: GPS Time Series SegmentationRDAHMM: GPS Time Series SegmentationSlide Courtesy of Robert Granat, JPLSlide Courtesy of Robert Granat, JPL

Complex data with subtle signals is difficult for Complex data with subtle signals is difficult for humans to analyze, leading to gaps in analysishumans to analyze, leading to gaps in analysis

HMM segmentation provides an automatic way to HMM segmentation provides an automatic way to focus attention on the most interesting parts of focus attention on the most interesting parts of the time seriesthe time series

GPS displacement (3D) length two years.

Divided automaticallyby HMM into 7 classes.

Features:• Dip due to aquifer

drainage (days 120-250)

• Hector Mine earthquake (day 626)

• Noisy period at end of time series

Towards Real-Time RDAHMMTowards Real-Time RDAHMM

A real-time version of RDHAMM could A real-time version of RDHAMM could potentially be used to detect state change potentially be used to detect state change events in live data from a GPS station.events in live data from a GPS station.

SCIGN maintains 125+ GPS stations, so SCIGN maintains 125+ GPS stations, so trivially parallel RDAHHM clones can trivially parallel RDAHHM clones can monitor state changes in the entire monitor state changes in the entire network.network.

But first we must get the data to RDAHMM.But first we must get the data to RDAHMM.

NaradaBrokering: Message NaradaBrokering: Message Transport for Distributed ServicesTransport for Distributed Services

NB is a distributed NB is a distributed messaging software messaging software system.system.• http://http://

www.naradabrokering.orgwww.naradabrokering.org NB system virtualizes NB system virtualizes

transport links between transport links between components.components.• Supports TCP/IP, parallel Supports TCP/IP, parallel

TCP/IP, UDP, SSL.TCP/IP, UDP, SSL. See e.g. See e.g.

http://grids.ucs.indiana.eduhttp://grids.ucs.indiana.edu/ptliupages/publications/All/ptliupages/publications/AllHands2005NB-Paper.pdf Hands2005NB-Paper.pdf for trans-Atlantic parallel for trans-Atlantic parallel tcp/ip timings.tcp/ip timings.

SOPAC GPS ServicesSOPAC GPS Services

NaradaBrokering topicsNaradaBrokering topics

More InformationMore Information

Contact: Contact: mpierce@cs.indiana.edumpierce@cs.indiana.edu GIS Work at CGL: GIS Work at CGL: www.crisisgrid.orgwww.crisisgrid.org

• Software, demos, publicationsSoftware, demos, publications• Several recent manuscript submissions are/will Several recent manuscript submissions are/will

be posted soon.be posted soon. HPSearch at CGL: www.hpsearch.orgHPSearch at CGL: www.hpsearch.org SERVOGrid Web SitesSERVOGrid Web Sites

• Our fine parent projectOur fine parent project• http://servo.jpl.nasa.gov/http://servo.jpl.nasa.gov/• http://quakesim.jpl.nasa.gov/http://quakesim.jpl.nasa.gov/

AcknowledgementsAcknowledgements

Geoffrey Fox, Community Grids Lab Geoffrey Fox, Community Grids Lab director.director.

Shrideep Pallickara: NaradaBrokering Shrideep Pallickara: NaradaBrokering design/development leaddesign/development lead

Grad Students: Ahmet Sayar, Galip Grad Students: Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawadhan Aydin, Mehmet Aktas, Harshawadhan GadgilGadgil

Backup SlidesBackup Slides

SERVO Apps and Their DataSERVO Apps and Their Data GeoFEST:GeoFEST: Three-dimensional viscoelastic finite element model for Three-dimensional viscoelastic finite element model for

calculating nodal displacements and tractions. Allows for realistic calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, material properties, and body fault geometry and characteristics, material properties, and body forces. forces. • Relies upon fault models with geometric and material properties.Relies upon fault models with geometric and material properties.

Virtual California:Virtual California: Program to simulate interactions between vertical Program to simulate interactions between vertical strike-slip faults using an elastic layer over a viscoelastic half-space.strike-slip faults using an elastic layer over a viscoelastic half-space.• Relies upon fault and fault friction models.Relies upon fault and fault friction models.

Pattern Informatics:Pattern Informatics: Calculates regions of enhanced probability for Calculates regions of enhanced probability for future seismic activity based on the seismic record of the regionfuture seismic activity based on the seismic record of the region• Uses seismic data archivesUses seismic data archives

RDAHMM:RDAHMM: Time series analysis program based on Hidden Markov Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities for Modeling. Produces feature vectors and probabilities for transitioning from one class to another. transitioning from one class to another. • Used to analyze GPS and seismic catalog archives.Used to analyze GPS and seismic catalog archives.• Can be adapted to detect state change events in real time.Can be adapted to detect state change events in real time.

We will focus on the latter two.We will focus on the latter two.

Some SERVOGrid Some SERVOGrid Research ChallengesResearch Challenges

Problems with Conventional Web Problems with Conventional Web ServicesServices

Transport: HTTP Request/Response is a poor Transport: HTTP Request/Response is a poor choice for non-trivial data transport.choice for non-trivial data transport.• Much better to stream out data without knowing Much better to stream out data without knowing

the content-length.the content-length. Representation: ASCII XML is inefficient in Representation: ASCII XML is inefficient in

obvious and not so obvious ways.obvious and not so obvious ways.• For example, WS security depends upon For example, WS security depends upon

canonicalization to make reproducible message canonicalization to make reproducible message digests.digests.

Efficiency and performance is not just a high Efficiency and performance is not just a high performance computing problem. performance computing problem. • Needed to support PDAs and other devices Needed to support PDAs and other devices

NaradaBrokering and Web NaradaBrokering and Web ServicesServices

SOAP 1.2 defines a message routing across SOAP 1.2 defines a message routing across distributed SOAP Nodes.distributed SOAP Nodes.• Naturally maps to an NB implementation.Naturally maps to an NB implementation.• This has just been released from This has just been released from www.naradabrokering.orgwww.naradabrokering.org

NB also has support for WS-Eventing and WS-NB also has support for WS-Eventing and WS-ReliableMessaging.ReliableMessaging.

More generally, we argue for the use of software More generally, we argue for the use of software messaging substrates to provide/implement messaging substrates to provide/implement desirable “quality of service” featuresdesirable “quality of service” features• Transport, routing/addressing, reliability, security, Transport, routing/addressing, reliability, security,

discovery, etc. discovery, etc. • Specific service capabilities (like “run job”, “move file”, Specific service capabilities (like “run job”, “move file”,

“query data”) are decoupled from the substrate capabilities. “query data”) are decoupled from the substrate capabilities.

Efficient XML RepresentationEfficient XML Representation The XML Infoset provides an abstract data model.The XML Infoset provides an abstract data model.

• SOAP 1.2 is defined using the Infoset.SOAP 1.2 is defined using the Infoset. This separates XML from “angle bracket notation” This separates XML from “angle bracket notation”

restrictions.restrictions.• Infoset-compliant binary representations are possible.Infoset-compliant binary representations are possible.• No loss of data, so you can translate between binary and No loss of data, so you can translate between binary and

ascii representations.ascii representations. Current lab research investigates hand-held Current lab research investigates hand-held

applications.applications.• See See

http://grids.ucs.indiana.edu/ptliupages/publications/OptSOAPhttp://grids.ucs.indiana.edu/ptliupages/publications/OptSOAP_CTS05.pdf_CTS05.pdf

But easily extensible to high performance transport But easily extensible to high performance transport problems.problems.

More InformationMore Information

Contact: mpierce@cs.indiana.eduContact: mpierce@cs.indiana.edu GIS Work at CGL: www.crisisgrid.orgGIS Work at CGL: www.crisisgrid.org

• Software, demos, publicationsSoftware, demos, publications• Several recent manuscript submissions are/will Several recent manuscript submissions are/will

be posted soon.be posted soon. HPSearch at CGL: www.hpsearch.orgHPSearch at CGL: www.hpsearch.org SERVOGrid Web SitesSERVOGrid Web Sites

• Our fine parent projectOur fine parent project• http://servo.jpl.nasa.gov/http://servo.jpl.nasa.gov/• http://quakesim.jpl.nasa.gov/http://quakesim.jpl.nasa.gov/

A Big Picture for SERVOGridA Big Picture for SERVOGrid

Figure 1: Science, Critical Infrastructure Protection (CIP) and Emergency Preparedness and Response (EPR) Grids built as a Grid of Web Service (WS) Grids

EarthquakeResponse WS

Physical Network

Registry Metadata

EarthquakeForecast WS

Science Grid EPR/CIP Grid… TsunamiEPR/CIP …

Data Access/Storage

Security WorkflowNotification Messaging

Portals Visualization GridCollaboration Grid

Sensor/Database Grid Compute GridGIS Grid

Core Grid Services

RDAHMM: SCIGN GPS Network AnalysisRDAHMM: SCIGN GPS Network AnalysisSlide Courtesy of Robert Granat, JPLSlide Courtesy of Robert Granat, JPL

Have found a way to detect Have found a way to detect regional aseismic signalsregional aseismic signals This software is being integrated with the Quakesim web portalThis software is being integrated with the Quakesim web portal Scenarios for use with real time streaming data through the web Scenarios for use with real time streaming data through the web

portal are currently being investigatedportal are currently being investigated

Now segment all 127GPS stations

In blue: Numberof stations that changestate on a given day

In red: Seismic activity

Days with many state changes often do notcorrelate with largeearthquakes.

Support for Streaming DataSupport for Streaming Data We use NaradaBrokering messaging software to manage We use NaradaBrokering messaging software to manage

data streams and filters.data streams and filters.• Open source, Java-based software from the Community Grids Open source, Java-based software from the Community Grids

LabLab• Based on topic-based publication/subscription for delivery of Based on topic-based publication/subscription for delivery of

messages from/to multiple endpoints.messages from/to multiple endpoints.• ““Message” can be anything, including SOAP and binary data Message” can be anything, including SOAP and binary data

streams.streams.• We use this for audio/video collaboration.We use this for audio/video collaboration.• More recently using it to build Web Service messaging More recently using it to build Web Service messaging

substratessubstrates SOAP 1.2 routing model, WS-Reliability, WS-Eventing SOAP 1.2 routing model, WS-Reliability, WS-Eventing

NB ensures NB ensures reliable deliveryreliable delivery of events in the case of broker of events in the case of broker or client failures and prolonged entity disconnects. or client failures and prolonged entity disconnects. • Also supports Also supports replayreplay..

Implements high-performance protocols (message transit Implements high-performance protocols (message transit time of 1 to 2 ms per hop) time of 1 to 2 ms per hop)

GPS StationsGPS Stations Current implementation provides real-time access to Current implementation provides real-time access to

GP messages to following stations in RYO, ASCII and GP messages to following stations in RYO, ASCII and GML formats:GML formats:

SSC-A

SC-1

SC-2

SC-3

e

g

c4 56

b

f

d

aSSC-C

SC-7

SC-8

SC-9

s

u

o

q

t

r

p

SSC-DSC-11

y

z

SC-10w

x

v

SSC-BSC-4

SC-5

SC-6

l

n

i

j

m

k

h

SOPAC GPS Services SOPAC GPS Services As a case study we implemented services to provide real-As a case study we implemented services to provide real-

time access to GPS position messages collected from time access to GPS position messages collected from several SOPAC networks.several SOPAC networks.

Next step is to couple data assimilation tools (such as Next step is to couple data assimilation tools (such as RDAHMM) to real-time streaming GPS data.RDAHMM) to real-time streaming GPS data.

Next stepsNext steps• Programming APIs: currently we assume the subscriber speaks Programming APIs: currently we assume the subscriber speaks

NaradaBrokering Java APIs (either NB’s native API or Java NaradaBrokering Java APIs (either NB’s native API or Java Messaging Service).Messaging Service).

Need to investigate appropriate Web Service standards and C/C++ Need to investigate appropriate Web Service standards and C/C++ bindings.bindings.

• SOAP enveloping of the GML message stream.SOAP enveloping of the GML message stream.• A Sensor Collection Service will be implemented to provide A Sensor Collection Service will be implemented to provide

metadata about GPS sensors in SensorML.metadata about GPS sensors in SensorML.

Position MessagesPosition Messages

SOPAC provides 1-2Hz real-time SOPAC provides 1-2Hz real-time position messages from various GPS position messages from various GPS networks in a binary format called networks in a binary format called RYO.RYO.

Position messages are broadcasted Position messages are broadcasted through RTD server ports.through RTD server ports.

We have implemented tools to We have implemented tools to convert RYO messages into ASCII convert RYO messages into ASCII text and another that converts ASCII text and another that converts ASCII messages into GML.messages into GML.

Real-Time Access to Position Real-Time Access to Position MessagesMessages

We have a Forwarder tool that connects to We have a Forwarder tool that connects to RTD server port to forward RYO messages RTD server port to forward RYO messages to a NB topic.to a NB topic.

RYO to ASCII converter tool subscribes this RYO to ASCII converter tool subscribes this topic to collect binary messages and topic to collect binary messages and converts them to ASCII. Then it publishes converts them to ASCII. Then it publishes ASCII messages to another NB topic.ASCII messages to another NB topic.

ASCII to GML converter subscribes this ASCII to GML converter subscribes this topic and publishes GML messages to topic and publishes GML messages to another topic.another topic.