ANABAS

August 25 2005 Briefing [email protected] [email protected] [email protected] 1

ANABAS

Use of Grids in DoD Applications

Geoffrey Fox, Alex Ho, Marlon PierceSBIR Briefing August 25, 2005


What do Web Services Prescribe?• The specify interfaces for system services (and generally useful

services like database) • They specify an interface language (WSDL) for all services• They develop containers and frameworks to use to host services• They specify a message format (SOAP) for ALL messages that

defines both application and system actions precisely• They imply a process be started to define domain specific

services• There are multiple competing activities from Microsoft and IBM

to Apache, IU and Anabas (for example) developing system and application services

• Unlike for RTI and CORBA, services from different vendors should interoperate

H1 H4H3H2 Body F1 F2 F3 F4 Service

Container Handlers

Container System Processing


What do Grids Add?• Grids use all of the Web Services • They address management and deployment of large

distributed systems of services • They address security and management issues of virtual

organizations crossing multiple administrative domains• GGF is developing specific services of relevance including

job management, many aspects of data and scheduling• GGF has a good process for developing new higher level

specifications– For example GGF15 will address Cross enterprise security,

Campus Grids, Enterprise Grids, High Performance messaging, GIS Grids, Portals and continue OGSA work integrating data, compute and management


The Grid and Web Service Institutional Hierarchy

1: Container and Run Time (Hosting) Environment

2: System Services and FeaturesHandlers like WS-RM, Security, Programming Models like BPEL

or Registries like UDDI

3: Generally Useful Services and FeaturesSuch as “Access a Database” or “Submit a Job” or “ManageCluster” or “Support a Portal” or “Collaborative Visualization”

4: Application or Community of InterestSpecific Services

such as “Run BLAST” or “Look at Houses for sale”

OGSAand otherGGF/W3C/ ………

WS-* fromOASIS/W3C/Industry

Apache Axis.NET etc.


The Grid and Web Service Functional Hierarchy

B: Resources

C: Electronic Proxy Services for Resources

G: User Interface

F: Portal: Aggregation, Profiles

E: Manipulating and Linking Services

A: Pervasive System Services: Security, Collaboration, Messaging, Metadata

D: Brokering Monitoring and Managing Resources and Services


Composing Functionality and Resourcesin the Grid of Grids

Overlayand ComposeGrids of Grids

Methods Services Functional Grids

CPUs Clusters ComputeResource Grids

MPPs

DatabasesFederatedDatabases

Sensor Sensor Nets

DataResource Grids


Critical Infrastructure (CI) Grids built in composite fashion and linked to an NCOW (GiG) Grid

DoD Servicesand Filters

Physical Network

Registry Metadata

Flood Servicesand Filters

Flood CIGrid NCOW Grid… Electricity CIGrid …

Data Access/Storage

Security WorkflowNotification Messaging

Portals Visualization GridCollaboration Grid

Sensor Grid Compute GridGIS Grid


Mediation and Transformation in a Grid of Grids and Simple Services

Po

rtP

ort

Port PortInternal

Interfaces

Subgrid or service

Po

rtP

ort

Port PortInternal

Interfaces

Subgrid or service

Po

rtP

ort

Port PortInternal

Interfaces

Subgrid or service

Messaging

Mediation andTransformationServices

External facingInterfaces


DoD Services as Part of Grid of Grids

PortalServicesJSR168 Portlets

GridService1

XMSF RTIBridge

GridService2HPCMOGateway

GridService4Sensor(Net)

HLA Federation

e.g. SAFIRE

GridService5

XMSF RTIBridge

HLA Federate

GridService3

OGC MapService

GridService6

Discovery

GridService7

Visualization

Aggregation/Customization Portal

Resource Level

Grid Messaging Infrastructure

ANABAS


The Global Information Grid Core Enterprise Services

Core Enterprise Services Service Functionality

CES1: Enterprise Services Management (ESM)

including life-cycle management

CES2: Information Assurance (IA)/Security

Supports confidentiality, integrity and availability. Implies reliability and autonomic features

CES3: Messaging Synchronous or asynchronous cases

CES4: Discovery Searching data and services

CES5: Mediation Includes translation, aggregation, integration, correlation, fusion, brokering publication, and other transformations for services and data. Possibly agents

CES6: Collaboration Provision and control of sharing with emphasis on synchronous real-time services

CES7: User Assistance Includes automated and manual methods of optimizing the user GiG experience (user agent)

CES8: Storage Retention, organization and disposition of all forms of data

CES9: Application Provisioning, operations and maintenance of applications.


The Ten areas covered by the core WS-* Specifications

WS-* Specification Area Examples

1: Core Service Model XML, WSDL, SOAP

2: Service Internet WS-Addressing, WS-MessageDelivery; Reliable Messaging WSRM; Efficient Messaging MOTM

3: Notification WS-Notification, WS-Eventing (Publish-Subscribe)

4: Workflow and Transactions BPEL, WS-Choreography, WS-Coordination

5: Security WS-Security, WS-Trust, WS-Federation, SAML, WS-SecureConversation

6: Service Discovery UDDI, WS-Discovery

7: System Metadata and State WSRF, WS-MetadataExchange, WS-Context

8: Management WSDM, WS-Management, WS-Transfer

9: Policy and Agreements WS-Policy, WS-Agreement

10: Portals and User Interfaces WSRP (Remote Portlets)


Activities in Global Grid Forum Working Groups

GGF Area Standards Activities

1: Architecture High Level Resource/Service Naming (level 2 of fig. 1),Integrated Grid Architecture

2: Applications Software Interfaces to Grid, Grid Remote Procedure Call, Checkpointing and Recovery, Interoperability to Job Submittal services, Information Retrieval,

3: Compute Job Submission, Basic Execution Services, Service Level Agreements for Resource use and reservation, Distributed Scheduling

4: Data Database and File Grid access, Grid FTP, Storage Management, Data replication, Binary data specification and interface, High-level publish/subscribe, Transaction management

5: Infrastructure Network measurements, Role of IPv6 and high performance networking, Data transport

6: Management Resource/Service configuration, deployment and lifetime, Usage records and access, Grid economy model

7: Security Authorization, P2P and Firewall Issues, Trusted Computing


Core Services in Action INCOW Service or Feature WS-* Service area GGF and Others Demo Status

A: General Principles

Use Service Oriented Architecture Core Service Model (#1)

Build Grids on Web Services

YES

Grid of Grids Composition DTA7

B: NCOW Core Services (to be continued)

CES 1: Enterprise Services Management

WS-* #8 Management GGF #6: ManagementCIM

Phase II (DTA9)

CES 2: Information Assurance(IA)/Security

WS-* #5WS-Security

GGF #7, Grid-Shib, Permis Liberty Alliance etc.

Phase II (Not in DTA)

CES 3: Messaging WS-* #2, #3 JMS, MQSeries,Streaming /Sensor Technologies

SBIR (DTA1)

CES 4: Discovery WS-* #6 SBIR (DTA8)

CES 5: Mediation WS-* #4 workflow Treatment of Legacy systems. Data Transformations

SBIR (DTA6 DTA7)

CES 6: Collaboration VO GGF VO. XGSP, Shared Web Service ports

SBIR (DTA2 DTA3)

CES 7: User assistance WS- * #10 Portlets, JSR168NCOW Capability Interfaces

SAB (DTA10)


Core Services in Action IINCOW Service or Feature WS-* Service area GGF and Others Demo Status

B: NCOW Core Services Continued

CES 8: Storage (not real-time streams)

GGF #4 DataNCOW Data Strategy

Phase II (Not in DTA)

CES 9: Application GGF #2 ; Best Practice in building Grid/Web services

SAB (DTA6 DTA7)

Environmental Control Services ECS

WS-*, #9 Phase II (Not in DTA)

Resource Infrastructure GGF #5; Ad-hoc networks important

NO

C: Key NCOW Capabilities not directly in CES

Meta-data WS-* #7 SAB (DTA8)

Resource/Service Matching/Scheduling

Distributed Scheduling and SLA’s (GGF # 3)

GGF scheduling work extended to networks

Phase II with scheduled messaging (DTA1)

Sensors (real-time data) OGC Sensor standards SBIR (DTA6)

GIS OGC GIS standards SBIR (DTA4 DTA5)


Analysis of Grid Technology for DoD• We developed four significant documents• DoD Grid Opportunities for the GiG and NCOW

– Grids for the GiG and Real Time Simulations (Proceedings of Ninth IEEE International Symposium DS-RT 2005 on Distributed Simulation and Real Time Applications' Montreal October 10-12 2005) is subset

• Appendix on Grid and Web Services covering all basic Grid and Web services

• Grid Technology Overview and Status reviews Grids• Grid Application Areas within DoD is a general study of

applications of Grids in DoD• Complete list of References for Reports on Grids for GiG and

NCOW contains all references in above reports• Implementing some Grid Application Areas within NCOW 1.1 of

DoD is not started but is intended to write up specific applications such as demos


Major Conclusions I• One can map “broadly” NCOW and GiG core capabilities

into Web Service (WS-*) and Grid (GGF) architecture and core services– Analysis of Grids in NCOW document inaccurate (confuse Grids

and Globus and only consider early activities)• Important “mismatches” on both NCOW and Grid sides• Grid/WS-* do not have messaging and collaboration• NCOW does not have system metadata and

resource/service scheduling and matching• Higher level services of importance include GIS

(Geographical Information Systems), Sensors and data-mining

• Criticisms of Web services in a recent paper by Birman seem to be addressed by Grids or reflect immaturity of initial technology implementations


Major Conclusions II• NCOW does not seem to have any analysis of how

to build their systems on WS-*/Grid technologies in a layered fashion; they do have a layered service architecture so this can be done– In particular do not clearly endorse critical header/body

structure of SOAP messages

• Grid of Grids allows modular architectures and natural treatment of legacy systems


Areas in Birman Paper• Information Architecture and Service Description

• Document-Centric Nature

• Time-critical Events

• Life Cycle Support for Services

• Reliable Messages

• Security

• Scalability


Grids and HLA/RTI I• HLA through IEEE1516 has specified the interfaces for its key services

that are supported by RTI (Run Time Infrastructure)• HLA does not specify each message semantics or core system services

– RTI implementations are NOT interoperable although each one should support any HLA federation

– RTI implementations become a full distributed system environment as need metadata, reliable messaging etc. with simulation support only a small part

• Grids can be used in HLA with– Dynamic assignment of compute resources to support federates– Building web service interfaces to federates (XMSF)– Infrastructure to build a new generation of RTI that will use Web

system services and just add simulation support


Grids and HLA/RTI II• HLA specifies

– Declaration management – achieved through use of publish/subscribe Grid Messaging (NaradaBrokering)

– Data Distribution management – corresponds to geometry sensitive publish and subscribe model (add to NaradaBrokering)

– Time management – corresponds to simulation framework (use best event driven and time stepped models – as infrastructure generic, one can support broad range of simulations including classic parallel computing and agent-based simulations)

– Object management - Very specific to HLA and should be built as per IEEE1516

– Ownership management - could use Grid virtualization and use metadata catalog catalogs to handle properties – might be generalizable

– Federation management - Could generalize to support of general simulation models (federates and federations are a general concept)


Demo Technology Areas DTA1) NaradaBrokering: Grid Messaging2) Anabas Collaborative Applications3) GlobalMMCS/XGSP Collaborative Stream Management

Framework and A/V Conferencing4) Geographical Information Systems (GIS) Web Feature

Services (WFS)5) GIS Web Map Services (WMS) and Clients 6) GIS Sensor Enablement Services7) Wrapping of non Grid Applications as Grids of grids or

Grids of Services8) Information and meta-data Services 9) HPSearch Workflow and Management10) Portals and Portlets


IN1 Data Mining and GIS Grid

WMS handlingClient requests

WMS Client

UDDI

WFS2

Databases withNASA, USGS features

SERVOGrid Faults

WFS1 NASA WMS

HTTP

SOAP

WFS3

Data Mining Grid

WMS Client


California fault data from Quake Tables fault database via Web Feature Service.

Standard Open Geospatial Consortium WMS Clients


Get Feature Info allows users to get map information. This can also be used to read feature info off the map when creating input data for applications

Standard Open Geospatial Consortium WMS Clients


IN1 Data Mining Grid

HPSearchWorkflow

UDDI

Databases withNASA,USGS features

SERVOGrid FaultsWFS4

SOAP

WS-Context

WFS3

PI Data Mining

Filter

GIS Grid

Filter

NaradaBrokering

Pipeline

System Services


PI demo combines WFS, WMS, and HPSearch for service orchestration.

Tool bar items allows you to adjust maps

Users set up problems by adding filtered seismic archives from WFS as map layers.


Hot spots calculations--areas of increased earthquake probability in the forecast time-- calculations are re-plotted on the map as features.


Typical use of Grid Messaging in NASA

Datamining Grid

Sensor Grid

Grid Eventing GIS Grid



Typical use of Grid Messaging

HPSearchManages

NaradaBrokering

Sensor Grid

WS-ContextStores dynamic data

Filter orDatamining

WFS

Post beforeProcessing

Post afterProcessing

Notify

SubscribeDatabaseArchivess


IN2 Google Map Client

Google Central

Google Map Client

UDDI

WFS2

Databases withSERVOGrid Faults

WFS1

SOAP

Sensor Grid

HTTP

Helper Services

Archived Real Time


IN2: Real Time GPS and Google Maps

Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients.

Select and zoom to GPS station location, click icons for more information.


Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.

IN2: Integrating Archived Web

Feature Services and Google Maps


IN2: Integrating Archived Web

Feature Services and Google Maps

Google maps can be integrated with Web Feature Service Archives to browse earthquake fault records.

Faults are typically stored by segment number, so map interfaces are convenient for both verifying continuity and setting up input files for computing problems.


IN2: Same Screens with Hybrid Maps


IN3 Google Maps as a WMS Web service

Google Central


WMS Client

UDDI

WFS2 …..

Databases withNASA,USGS features

SERVOGrid Faults

WFS1Google Maps

WMS (Gateway)

HTTP

SOAPOther

Servicesusing results


Google Maps as Service

accessed from our WMS

Client


CollaborativeStreaming

Grid

IN4 Server-side Streaming from a WMS


UDDI

SOAP

XGSP MediaService

NaradaBrokering

WFS2


SERVOGrid Faults

WFS1 NASA WMSWFS3

Data Mining Grid


CollaborativeStreamingGrid

IN4 Shared Display Streaming from a WMS


UDDI

SOAP

XGSP MediaService

NaradaBrokering

WFS2


SERVOGrid Faults

WFS1 NASA WMSWFS3

Data Mining Grid

WMS Client

GMC

GMC = GlobalMMCS Client


Web Service Collaboration

Web Service NaradaBrokering

WS1

WS2

WS3

NaradaBrokering

Shared Input Port with replicated services

Shared Output port with replicated recipients


Pipelined Web Service Collaboration

• In a workflow, one can invoke collaborative streams on any flow and this splitting is between output port of one and input of next Web Service in chain

WS1

WS2

WS3

NaradaBrokering

WS4

WS5

WS6

WS-BWS-A

Shared Input Port

Shared Output Port


Collaboration Grid

UDDI NaradaBroker

HPSearch

WS-Context

Gateway

WS-Security

NaradaBroker

NaradaBroker

Gateway

Gateway

Gateway

XGSP MediaService

Video Mixer

Transcoder

Audio Mixer

Replay

Record

Annotate

Thumbnail

WhiteBoard

SharedDisplay

SharedWS


Annotation of GIS Maps from WMS

converted into H261 Video Stream

August 25 2005 Briefing [email protected] [email protected] [email protected] 44New GlobalMMCS Client

Chat

TV

WebcamVideo Mixer

GIS


Collaborative Map Pages• Shared Input Port

Anabas Endpoint

XML Sharedlet interfacesimilar to WSDL Narada

Brokering

Anabas Endpoint

Anabas Endpoint

Google Central

Input to Googleweb page

August 25 2005 Briefing [email protected] [email protected] [email protected] 46Impromptu Google Interface


Impromptu Google Map on Whiteboard


Clients and Services• Thick clients can be Grid (WSDL) endpoints and

act as services

• Can break thick client up as MVC with Model becoming a service – Message-based MVC

ClientGrid ServiceSOAP

Model asService

Grid Service

SOAP

Portal

HTTP

Portlet View


Collaborative “Legacy” Object• Shared Output Port with Anabas Endpoint providing

Service wrapping of RMI Object

Anabas Endpoint


Brokering

Anabas Endpoint

Anabas Endpoint

Java Distributed Object

RMI

Output from Java Object


Collaborative Annotation of Map Pages• Shared Input Port with replicated white board


Brokering

Anabas WB Service

Google Central

White Boardchanges

Anabas Google

Endpoint

WB Service

Anabas WB Service


Collaborative Search• Shared Input Port for annotation with replicated search

display (shared output port for search service)

NaradaBrokering

AnabasAnnotatable Text Service

Search Resultsand annotation

Anabas Search

Endpoint

Annotatable Text Service

AnabasAnnotatable Text Service

Google SearchYahoo Search MetaSearch


Impromptu Yahoo Search


Impromptu GoogleSearch


Impromptu MetaSearch

Documents

ANABAS