1

SERVOGrid Basics• SERVOGrid is our project to build a distributed computing infrastructure to

support earthquake simulation codes.– We use Web Services for this.– We build portal interfaces to these web services out of portlets.

• Portlets for Java portals are standardized (JSR 168)• All major commercial vendors, many open source projects support.• GEON, NEESGrid, LEAD portal use also.

• Distributed Computing Infrastructure– Services for remotely executing applications, managing distributed files in logical

units, monitoring applications, archiving user projects, and so on.– Variously called “Grids” (DOE), “Cyberinfrastructure” (NSF), “e-Science” (UK e-

Science program), ….• The above is sometimes referred to as an “Execution Grid”.

– We designed this around a web service architecture.– See links at the end for more information.

• But what about the data?– Need ways to access remote data sources through programming interfaces.– Practice often is to download (via FTP) entire catalogs and hand edit to get data

you want in the format you want.– We need to automate this.

2

Some SERVO Apps and Their Data

• Disloc: handles multiple arbitrarily dipping dislocations (faults) in an elastic half-space.– Relies upon geometric fault models.

• GeoFEST: Three-dimensional viscoelastic finite element model for calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, material properties, and body forces. – Relies upon fault models with geometric and material properties.

• Virtual California: Program to simulate interactions between vertical strike-slip faults using an elastic layer over a viscoelastic half-space.– Relies upon fault and fault friction models.

• RDAHMM: Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities for transitioning from one class to another. – Used to analyze GPS and seismic catalogs.

3

Where Is the Data?• QuakeTables Fault Database

– SERVO’s fault repository for California.– Compatible with GeoFEST, Disloc, and VirtualCalifornia– http://infogroup.usc.edu:8080/public.html

• GPS Data sources and formats (RDAHMM and others).– JPL: ftp://sideshow.jpl.nasa.gov/pub/mbh– SOPAC: ftp://garner.ucsd.edu/pub/timeseries– USGS: http://pasadena.wr.usgs.gov/scign/Analysis/plotdata/

• Seismic Event Data (RDAHMM and others)– SCSN: http://www.scec.org/ftp/catalogs/SCSN– SCEDC: http://www.scecd.scec.org/ftp/catalogs/SCEC_DC– Dinger-Shearer:

http://www.scecdc.org/ftp/catalogs/dinger-shearer/dinger-shearer.catalog

– Haukkson: http://www.scecdc.scec.org/ftp/catalogs/hauksson/Socal

4

What Do We Want to Do?• Web Services provide a programming interfaces to data so that you can

build remote clients.• SERVO portal work houses these Web Service clients.• Portal also houses the clients for the remote simulation services (RDAHMH,

GeoFEST, Potts model codes, etc).– As we have seen, these codes work directly with data catalogs

• Typically the codes must process and reformat the data catalog entries in code-specific ways.

– Done by hand by most developers, which is fine for development.– But not fine for mature codes or automated applications.

• Currently, our solution is to harvest data from online catalogs and transform into GIS compatible formats in DBs.

– Discuss a couple of examples• But we would like to experiment with building services that are closer to the

source.– Finer resolution data products– Academic exercise in implementing reliable sensor network services and

streaming applications.• SensorML family from Open Geospatial Consortium provides the standards.

5

Open GIS Services• GML abstract data models can encode data but you need services

to interact with the remote data.• Some example OGC services include

– Web Feature Service: for retrieving GML encode features, like faults, roads, county boundaries, GPS station locations,….

– Web Map Service: for creating maps out of Web Features – Sensor Grid Services: for working with streaming, time-stamped data.

• Problems with OGC services– Not (yet) Web Service compliant

• “Pre” web service, no SOAP or WSDL• Use instead HTTP GET/POST conventions.

– Often define general Web Service services as specialized standards• Information services• Notification services in sensor grids

6

Using GML to Encode GPS• Collected GPS data for each

site is made available through online catalogs.– Using various text formats.

• We have developed GML descriptions of GPS using Feature.xsd schema.

• Full GML GPS schema: http://www.crisisgrid.org/files/gps.xsd.

• Supports JPL, SOPAC, and USGS formats.

• Other efforts: Feature encodings of Seismic and Fault catalogs.– http://www.crisisgrid.org/html/

servo_-_gps.html

7

Anatomy of Web Feature Service• GML provides the data model, but we must still provide a remote

programming interface to the data.• Web Feature Service provides three major services as described in

OGC specification:• GetCapabilities: The clients (WMS servers or users) starts with

requesting a document from WFS which describes it’s abilities. When a getCapabilities request arrives, the server dynamically creates a capabilities document and returns this.

• This is OGC’s formalization of metadata, so important to GEON, ESG, etc.• DescribeFeatureType: After the client receives the capabilities

document he/she can request a more detailed description for any of the features listed in the WFS capabilities document.

• The WFS returns an XML schema that describes the requested feature.• Metadata about a specific entry.

• GetFeature: The client can ask the WFS to return a particular portion of any feature data.

• GetFeature requests contain some property names of the feature and a Filter element to describe the query.

• The WFS extracts the query and bounding box from the filter and queries the feature databases.

• The results obtained from the DB query are converted the feature’s GML format and returned to the client as a FeatureCollection.

8

Web Map Services

• OGC Web Map Servers create digital maps from abstract data sets retrieved from Web Feature Services.– Layer features are overlayed.

• WMS servers can be daisy chained.• Our research efforts here are to build Web Service

version of the WMS as well as portlet clients.• We are developing backward compatibility with existing

WMS.– Purdue CAAGIS for Indiana maps.– NASA JPL OnEarth WMS.

• Future plans are to replace current GMT portal tools with WMS, combine maps with GeoFEST and Virtual California application output.

9

Next Steps and Questions (Jan)• We are very interested in

starting a new Sensor Grid project.– Implement Open

Geospatial Consortium standards for Sensor Grids (see supplemental slides).

– We are outlining a more ambitious integration of Robert Granat’s RDAHMM with GPS data.

• I will be visiting JPL in early February.

• Some questions we have are on the right.

• Direct (or at least closer) access to GPS stations?

• Any documentation on the filtering process? • What kind of filters are applied on the raw

data? Can it be automated?• How is data stored, in a DB or just in files? • How do the stations communicate with the

servers? • What protocols are being used for transport,

etc? • Are there any web-based interfaces to

stations or to the data stores? • Are the GPS stations configurable? • How often do they send out data, can you

change the interval? • What are the most important data formats

for you? (JPL,SOPAC,USGS) • Are there any other formats we need to

support? • What (other) kinds of applications are there

to use the GPS data?

10

HPSearch Overview

• HPSearch is a system for scripting and controlling web services and controlling data streams.– Uses JavaScript (Rhino) for scripting– Uses NaradaBrokering for controlling communications

between services.– Name is historical

• Basic premise is that it can be used to hook together streams of data to successive filters (services) that are controlled through a client environment.

11

Basic HPSearch Parts

• Kernel Components– Shell: processes the extended JavaScript– Service Manager: this is a scheduler for load balancing between

several services.– RequestHandler: monitors individual remote resources (may

include web services but also DBs or plain old URLs).

• WSProxy facilitates wrapping legacy applications as streaming web services– Manages both the control invocations from the shell and the NB-

based streaming data connections – Comes with two interfaces

• Runnable: you extend this Java interface for new services.• Wrapped: this is a legacy/non-WS service such as a perl or matlab

script.

12

HPSearch ArchitectureRequest Handler

Javascript Shell

Service Manager

Web Service EP

Other Objects

HPSearch Kernel

URIHandler

DBHandler

WSDLHandler

WSProxyHandler

Request Handler

HPSearch Kernel

HPSearch

Kernel

Broker Network

DataBase

Web Servic

e

Files, Sockets,Topics

WSProxy

Service

WSProxy Servi

ce

Stream ProxyKernel connects to

nearest broker and communicates with other kernels with messages

Handlers steer web services via simple SOAP requests

Data Flows as a stream of messages between WSProxies through the brokering network

Data is read / written from / to various data resources

13

HPSearch for RDAHMM + GPS

Kernel manages (steers) the various services

Sensor Network / GPS Database

RDAHMM

Data Filter

Matlab Script

WSProxy wrapped service

Start Activity, Web Service

HPSearch Kernel

Service distribution

Narada Broker

Virtual Data Flow between various components

Links between Brokers

Actual Data Flow between components takes place through the brokering network

14

More Information• My email:

– mpierce@cs.indiana.edu• Overview (submitted to ACES special issue of PAGEOPH).

– http://www.servogrid.org/slide/GEM/SERVO/ISERVO_ACES_PAGEOPH.doc• QuakeSim Project Page:

– http://quakesim.jpl.nasa.gov/• QuakeSim Portal:

– http://www.complexity.ucs.indiana.edu:8282.• QuakeTables Fault Database:

– http://infogroup.usc.edu:8080/public.html• Community Grids Lab GIS Grid Development:

– http://www.crisisgrid.org/.– See particularly http://www.crisisgrid.org/html/servo.html for GML information.– See http://www.crisisgrid.org/html/wfs.html for more on Web Feature Service.– WMS and Information Services information coming soon.

15

Supplemental Slides: Sensor Web Work

Slides Prepared by Galip Aydin

16

Sensor Web

• Goal: To create Web-based sensor networks by exploiting web-connected sensors (flood gauges, air pollution monitors etc.).

• This would make all sensors and repositories of sensor data discoverable, accessible and where applicable controllable via the WWW.

• OGC describes 5 important encoding and service standards for a Sensor Web Implementation (see next slide).

17

OGC-Sensor Web Enablement1. SensorML: XML encoding language for sensors. Used to discover, query

and control Web-resident sensors. 2. Observations & Measurements: The general models and an XML

encoding for what a sensor observes or measures (The value returned by or derived from a sensor observation -e.g. quantity, count, boolean, category, ordered category, position-).

3. Sensor Collection Service: A service to fetch observations from a sensor or constellation of sensors. Provides real time or archived observed values. Clients can also obtain information that describes the associated sensors and platforms.

4. Sensor Planning Service: A service by which a client can determine collection feasibility for a desired set of collection requests for one or more mobile sensors/platforms, or the client may submit collection requests directly to these sensors/platforms. SPS enables sensor tasking, acquisition requests, processing and simulation requests, and registration for alert notification.

5. Web Notification Service: A service by which a client may conduct a dialog with one or more other services. Provides a means for Sensor Planning Services to alert people, software, or other sensor systems of SPS results or alerts regarding phenomena of interest.

18

OGC-SWE Pieces

Sensor/Platform & Observable

Registries

User Environment In Situ Collection Environment

SWE Client

ObservationArchive

&Catalog

Need for Water Quality Data

SensorPlanningService

SensorCollection

Service

BaseStations & Sensors

Water Quality Observations & Metadata

SCS, WFS,WCS,CPS,WMS

Discover SCSFor Water Quality

ServiceRegistries

ObservationRequest

UserAvailable SCSFor Water Quality

ObservationResponse

Confirmation ofValid CollectionRequest or exceptions+ Session Handle +Collection Feasibility Plan

LocalArchive

&Catalog

Get Water Quality Observations & Metadata

Sensor/Platform & Observable

RegistriesCollection Feasibility Request orCollection Request

19

Sensor Web Enhancements 1

• We think that with some changes and additions a minimal and high performance Sensor Web with live sensor data capability can be realized.

• We realize that some of these service specifications are trying to solve problems common to the Web Services community such as notification.

• There are several Notification specifications for Web Services such as WS-Notification and WS-Eventing. Considering the industrial backing and huge WS community support we can expect these specifications to evolve into standards.

• Community Grids lab has considerable experience on Notification issues and currently a WS-Notification enabled version of NaradaBrokering is being developed.

20

Sensor Web Enhancements 2

• Sensor Planning service is designed as an asset management service. For our purposes existence of such a service is not crucial instead an Information Service with search and discovery capabilities is preferable.

• We are developing a High Performance Fault Tolerant Information Service for GIS applications and with little modifications this service can handle the functionality expected from Sensor Planning Service.

• The Information Service will provide search and discovery interfaces and WSDL addresses for Sensor Collection Services.

21

Sensor Web Enhancements 3

• In some cases immediate access to the sensor data is crucial. In such cases we expect to have some sort of a streaming data channel for live sensor data available to the clients.

• We plan to use Narada Brokering to provide us such a capability.

• Narada Brokering allows message producers to publish messages to topics and subscribers of these topics immediately receives the published messages.

• We can create topics according to the types of the sensors and subscribe clients to these topics according to their preferences. Thus Narada Brokering can provide a streaming channel between sensors and the clients.

22A sample Sensor Web that utilizes Information Service and Narada Brokering.

23

NaradaBrokering Overview

• NB is software for routing messages that has been developed by Shrideep Pallickara at CGL for several years.– Topic-based publish/subscribe system

• Implements JMS, JXTA, and its own APIs.• Allows you to construct hierarchical broker networks for

efficient routing.• Can route “anything”

– HTTP, TCP/IP, UDP – Binary data for collaborative shared display (www.anabas.com)– GridFTP via proxy– SOAP messages– WS-Eventing and WS-ReliableMessaging

• Moving toward supporting SOAP 1.2-style messaging.