Integrating Geographical Information Systems and Grid ApplicationsMarlon Pierce (mpierce@cs.indiana.edu)

Contributions: Yili Gong, Ahmet Sayar, Galip Aydin, Mehmet Aktas, Harshawardhan Gadgil, Zhigang QiCommunity Grids LabIndiana UniversityProject Funding: NASA AIST

Some Project Organizational Details We need a project-wide mailing list or two.

I can set these up really quickly at IU. Code repositories?

I have started using SourceForge and SVN for several projects, have been generally happy.

I think its good for visibility of the project, good to show program managers.

SF also has project management stuff like bugzillas. But licensing model may not work for JPL.

I’ve also become a recent convert to using Wikis for group editable web pages. We should do this or its equivalent. I have one at www.crisisgrid.org but it is trivial to make a new one

on www.servogrid.org also. Very low maintenance.

Something New: Using the TeraGrid The NSF TeraGrid is an administrative federation of

supercomputing facilities across the country. SDSC, NCSA, IU, ANL, PU, ORNL, TACC, PSC.

Four useful TG facts Almost any US researcher can apply to get 30,000 hours

(somewhat painful web forms to fill out). You can get more hours if you apply.

This researcher can share his allocation with others (1 page form--I used to give John, Gleb, Terry and others accounts).

All TG machines try to have the same software environments.

All come with Globus installed.

Problems with TeraGrid

TeraGrid is still broken up into fiefdoms Articles of confederation instead of constitution.

There is no way to do the following query: “Dear TG, I want to run the follow GeoFEST

simulation. It will require the following resources. Please submit to the best available machine. Love, Marlon.”

You still have to login to a specific machine and submit to its specific queuing system. PBS, LoadLeveler, LSF, etc.

Our Solution: Condor-G

Condor is a famous scheduler/cycle scavenger from U Wisconsin. To use it, run condor software on all nodes. Has a “matchmaker” component that matches a user’s request to

available resources. “Classads” in condor-speak. Condor-G is a bit different

It is a condor client interface that can submit Globus jobs. Globus in turn can hide differences between queuing systems.

You ony need Condor-G installed on one machine Can be anywhere.

Both Condor and Condor-G have a Web Service interface called Birdbath. We have built portlets out of these things.

CondorMaster

Condor

Condor

Condor

Condor

Condor Only Condor-G and Globus

(Portal)Client

Condor-G

LSFPBS

GlobusGlobus

(Portal)Client

What’s the Problem?

The problem is that the Condor matchmaker only works for Condor. Condor daemons on various machines report back to the

collector at regular intervals. Condor-G needs an external provider since Condor

is only installed in one place. We are solving this problem by using GPIR (a

resource monitoring tool installed on the TG) to construct classads and publish to the matchmaker.

We have prototyped this for GeoFEST, but need to take it to some sort of “production” level.

Bigger Research Issue: Generalized Matchmaking Condor matchmaking is only good for running

jobs. More generally you want to do Web Service

matchmaking on a Grid. May be “find me best machine to run GeoFEST”. May be “find me QuakeTables service with

Australian faults” Workflow also needs matchmaking, and

matchmaking should be decoupled from workflow execution.

Workflow and Matchmaking

QuakeTablesService

NCSA’s CobaltQuakeTables

CaliforniaQuakeTables

Australia

VC Service

IU’s Big Red

User LayerWorkflowMatchmaking

QuakeSim/SERVO IT/CS Development Overview Portlet-based portal components allow different portlets to be

exchanged between projects. Form-based portlets --> Interactive Maps These are clients to services

Sensor Grid: Topic based publish-subscribe systems support operations on streaming data.

Web services allow request/response style access to data and codes. GIS services (WMS, WFS) “Execution grid” services for running codes and moving files. Information services (WS-Context) and Web Service workflow

(HPSearch)

Portlets and Portals

Portlets and Portals

Portlets are a standard way for Java web applications to be shared between different portal containers. A portlet may be a web application such as a Google map client

that I want to put into container. Will inherit login, access control, layout management, etc.

We will show some demos for RDAHMM and ST-Filter later. We use Java Server Faces for development, so there may be

some solvable interoperability issues. The main point is that portlets allow REASON and QuakeSim

to exchange user interface components. We still need to develop client libraries and Web Services

Sensor Grid Overview

Sensor Grid Overview

QuakeTables and Web Feature Service provide access to archival data. Faults, GPS time series, Seismic records

Our Sensor Grid architecture supports access to real-time data. Integrated with all 70 stations of CRTN.

Consists of chains of filters communicating on a network through a publish/subscribe broker.

Each filter does a single task and passes the data along. Filters are also web services, but the communication is currently proprietary. Could be adapted to use SOAP and Axis 2 one way communication model, but

this is an academic exercise. Filters can be applications, like RDAHMM. Scripps collaborators have a prototype command line client if you want to

pipe and grep. Or you can develop your own stream sink.

SensorGrid Architecture

Real-Time Services for GPS Observations Real-time data processing is supported by

employing filters around publish/subscribe messaging system.

The filters are small applications extended from a generic Filter class to inherit publish and subscribe capabilities.

Input Signal Output SignalFilter

Filter Chains

NaradaBrokering Topics

Real-Time positions on Google maps

Real-Time Station Position Changes

RDAHMM + Real-Time GPS Integration