Upload
shae
View
32
Download
1
Tags:
Embed Size (px)
DESCRIPTION
SBIR Review Grid of Grids for Information Management. Anabas May 4, 2007. Outline. Revisit Problem Statement Revisit Research Goals Revisit Research Methodology Part 1: Grid of Grids System And Tools Architecture, Implementation and Status Demo Scenario Summary - PowerPoint PPT Presentation
Citation preview
SBIR ReviewGrid of Grids for
Information Management
Anabas
May 4, 2007
Outline• Revisit Problem Statement• Revisit Research Goals• Revisit Research Methodology• Part 1: Grid of Grids System And Tools• Architecture, Implementation and Status• Demo Scenario• Summary• Part 2: Additional Collaboration Grid Services• I. Collaborative Group Design, Implementation and
Status• II. Hybrid Shared Display Design, Implementation
and Status
Problem Statement
• Information and communication have played increasingly critical roles in our nation’s security
• The GiG is not one global seamless construct– Different pieces have different stakeholders with different
missions– Each has own name and unique vision of network-centric
operations– Many operations have been done independently
• Unable to satisfy interoperability, scalability, and security information management requirements for Net-Centric Operations without an advanced grid-based scalable service oriented framework
Challenges
• Operational tasks (network management, enterprise services management, information staging, and dissemination management) need to be done across the network using common tactics, techniques and procedures
• To integrate global grid technology with collaboration technology to provide a framework for network-centric operations to examine and derive warfighter requirements on the GiG
General Goals
• Build Net Centric Core Enterprise Services in fashion compatible with GGF/OGF and Industry
• Add key additional services including advance collaboration services and those for sensors and GIS
• Support Systems of Systems by federating Grids of Grids supporting a heterogeneous software production model allowing greater sustainability and choice of vendors
• Build tool to allow easy construction of Grids of Grids
• Demonstrate the Grid of Grids capability through one or more relevant applications
Research Methodology
• Our solution builds upon existing technology and infrastructure currently being developed across the grid and web services communities
• The major innovation is a systematic mapping between NCOW Core Enterprise Services and Grid and Web Service Architectures
Approach
• Develop the Grid of Grids architecture and information management system to address federation of legacy and new DoD enterprise systems with service oriented mediation between component collaboration, sensor, information, and computing Grids
• Develop static and dynamic Grid Builder Tool compatible with Web service workflow standards
• Demonstrate for Earthquake Science and DoD applications
Part 1:
Grid of Grids System and Tools
High-level Architecture of the Grid of Grids Prototype System
Management System
Metadata Service
Grid Builder
Template Grids
Deploy a grid via Condor
Interface for specific resource
Grid Middleware
QueryRetrieve
InputDiscover
Output
Resources
Ready
WIP
Workflow designer
Current Status• Key Accomplishment
– Our work has led to the realization that one can combine static tools like the Grid Builder with dynamic systems management
• Individual modules that are ready for use:– Template grids (function complete)– BPEL Workflow designer (function complete)– Resource viewer (function complete)– Management system (development near completion)
• Work in progress– The systems management metadata service configuration for application-specific
functions (integration in progress)– The integration of management component with Grid Builder (integration in
progress)– Customizable management interface for specific resource (being designed)– Integrate workflow designer with workflow execution engine (engine available from
OMII but not tested by us yet)
Grid Builder
• Grid Builder provides a graphical user interface with functionalities
• Template Grid – Grid Builder takes abstract specifications of Grids (i.e., Template Grids) and instantiates them on real infrastructure
• Management System - It provides needed management/monitoring, mediation (between sub Grids and Services), and configuration
• Workflow Designer - It uses Business Process Execution Language (BPEL) to link services/grids in a workflow and deploy them through a workflow engine
Grids linked as a workflow
Portal
SOAP messages
Grid of Grids (e.g., Earthquake Grid)
Template Grids
• Grid Builder can instantiate template grids taking into account scale of events and currently available resources– Template Grids corresponding to anticipated
scenarios– It can copy (with perhaps no changes) a Grid from
one deployment to another
– It can generate the needed management (fault tolerance, monitoring, firewall strategies)
Template Grids • Extended WS-BPEL 2.0 to include the
information below, to describe grids :• WS-BPEL 2.0 specifies the workflow of
services that are federated to complete a common task
Attributes Possible Values
Service Type
Application Services
GIS, Sensor, Filter, etc
System Services
Security, Management, Registry, etc
Operating System Windows, Linux, Solaris, etc
Server container IIS, Apache, etc
Context Firewalls, Running environment, SOAP types, NATs, etc
Database JDBC,Oracle,MS SQL, etc
Client Portal, Matrix, Anabas, Dashboard, etc
BPEL workflow
An Example of Template Grid in WS-BPEL 2.0 Specification
<Grid name="Earthquake"targetNamespace="http://cgl.com/ws/earthquakegrid"xmlns="http://schemas.xmlsoap.org/ws/earthquake-process/"xmlns:lns="http://manufacturing.org/wsdl/earthquake"><services><applications_services> name="GISGrid"targetNamespace="http://acme.com/ws-bp/gisgrid"myRole="GISInfo"/><application_services> name="SensorGrid"targetNamespace="http://cgl.com/ws /sensorgrid"myRole="SensorInfo"/><system_services> name="Security"targetNamespace="http://acme.com/ws-bp/security"myRole="SSH"/><system_services> name="Registry"targetNamespace="http://acme.com/ws-bp/registry"myRole="UDDI"/><system_services> name="Management"targetNamespace="http://acme.com/ws-bp/management"myRole="QoS"/></services><variables><variable name="OS" Type="Linux"/><variable name="Container" Type="tomcat"/><variable name="Message" Type="SOAP"/><variable name="Database" Type="ODBC"/><variable name="Client" Type="Matrix"/> </variables><contexts><firewalls Type="" Status = "on"/><nats type ="" Status = "enabled"<message type="SOAP"/></contexts>
<bpel_workflow><faultHandlers><catch faultName="lns:cannotComplete"faultVariable="Fault"><reply partnerLink="administrator"portType="lns:fault"operation="sendError"variable="POFault"faultName="cannotComplet"/></catch></faultHandlers><sequence><receive partnerLink="GIS"portType="lns:Filter1"operation="CheckGISInfo"variable="PO"></receive><links><link name="switch_to_Sensor"/></links><assign name="Sensor_Info"></assign><invoke partnerLink="switching" operation = "submitJob"….</invoke><receive partnerLink="switching"portType="lns:earthquakeinfo"operation="send_location_of_earthquakes"variable="filter"></receive></sequence><bpel_workflow></Grid>
BPEL Workflow Designer
• Define or edit template grids
• A GEF-based editor to design a workflow of grids/services
• As a precursor to workflow systems. – Fill gap between installing a grid and collecting things
into a workflow.
BPEL Workflow Designer Interface
A workflow describing the Servo Grid
BPEL Workflow Designer Interface
The workflow is associated with resource view
Management SystemKey functions:
• Configuration and lifecycle operations (function complete)
• A global view of all accessible resources including their links (work-in-progress)
• Resource status monitor (e.g., tracking logs) (work-in-progress)
• System status maintenance (recovery, fault tolerance etc) (function complete)
• Resource-specific features (being designed)– Input & output interfaces– Unique functionalities– Others
The Integration of the Management System and Grid Builder (1)
• Service wrapper for a manageable resource (function complete)
• Need to set up a metadata service for handling different types of resources (integration in progress)
• Predictable input and output interfaces are defined by metadata (integration in progress)
The Integration of the Management System and Grid Builder (2)
• Two schemas for each resource type– Essential info, such as name, type, operating system, security,
availability, etc – Non-essential but useful info, such as location, ownership, etc
• Metadata specifies both generic and non-generic features of a resource instance– Generic features: life-cycle operations, heartbeat,
communication, etc– Non-generic features: specific interface, unique functions, etc
User Interface
The current interface is designed for specific resources, which are messaging nodes (brokers)
Grid of Grids Demo Scenario
• Building Earthquake Grid - QuickSim2. It provides services such as:– AnalyzeTseri Service (Time Series Filter)– AntVisco Service (Workflow)– Gnuplot Service (Plot) – RDAHMM Service (Datamining by HMM)– STFilter Service (Time Series Filter)
Grid of Grids Demo Scenario
• Building Earthquake Grid - QuickSim2. It provides services such as:– AnalyzeTseri Service (Time Series Filter)– AntVisco Service (Workflow)– Gnuplot Service (Plot) – RDAHMM Service (Datamining by HMM)– STFilter Service (Time Series Filter)
Grid of Grids Demo Scenario
• These form component Grids– User Assistance (Portal)– Sensors– Filter (Datamining, Compute)– Geographical Information System– Management (being added as part of project)
QuakeSim2
• A representative Web Service Grid application, which includes– Web services: provide access to data and codes– Portlet: acts as an aggregation of client interfaces
• We can build web services from the sketch or use those built by others
• Data can be retrieved from archives or from real-time filters
Typical use of Grid Messaging in NASA
Datamining Grid
Sensor Grid
Grid Eventing GIS Grid
SERVOGrid has a portalThe Portal is built from portlets
– providing user interface fragments for each service that are composed into the full interface – uses OGCE technology as does planetary science VLAB portal with University of Minnesota
Grid Workflow Datamining in Earth Science Work with Scripps Institute Grid services controlled by workflow process real time
data from ~70 GPS Sensors in Southern California
Streaming DataSupport
TransformationsData Checking
Hidden MarkovDatamining (JPL)
Display (GIS)
NASA GPS
Earthquake
Real Time
Archival
Service Description
Job Management SERVO wraps Apache Ant as a web service and uses it to launch jobs. For a particular application, we design a build.xml template. The interface is simply a string array of build properties called for by the template. We’ve also built a simple generic “template engine” version of this.
Specific Applications: Virtual California, Geofest, Park, RDAHMM ..
These can be all launched by a single Job Management service or by custom instances of this with metadata preset to a particular application
Context Data Service We store information gathered from users’ interactions with the portal interface in a generic, recursively defined XML data structure. Typically we store input parameters and choices made by the user so that we can recover and reload these later. We also use this for monitoring remote workflows. We have devoted considerable effort into developing WS-Context to support the generalization of this initial simple service.
Application and Host Metadata Service
We have an Application and a Host Descriptor service based on XML schema descriptors. Portlet interfaces allow code administrators to make applications available through the browser.
File Services We built a file web service that could do uploads, downloads, and crossloads between different services. Clearly this supports specific operations such as file browsing, creation, deletion and copying.
Portal We use an OGCE based portal based on portlet architecture
Authentication and Authorization This uses capabilities built into portal. Note that simulations are typically performed on machines where user has accounts while data services are shared for read access
Information Service We have built data model extensions to UDDI to support XPath queries over Geographical Information System capability.xml files. This is designed to replace OGC (Open Geospatial Consortium) Web registry service
Web Map Service We built a Web Service version of this Open Geospatial Consortium specification. The WMS constructs images out of abstract feature descriptions.
Web Feature Service We’ve built a Web Service version of this OGC standard. We’ve extended it to support data streaming for increased performance.
QuakeSim Services
Workflow/Monitoring/Management Services
The HPSearch project uses HPSearch Web Services to execute JavaScript workflow descriptions. It has more recently been revised to support WS-Management and to support both workflow (where there are many alternatives) and system management (where there is less work). Management functions include life cycle of services and QoS for inter-service links
Sensor Grid Services We are developing infrastructure to support streaming GPS signals and their successive filtering into different formats. This is built over NaradaBrokering (see messaging service). This does not use Web Services as such at present but the filters can be controlled by HPSearch services.
Messaging Service This is used to stream data in workflow fed by real-time sources. It is based on NaradaBrokering which can also be used in cases just involving archival data
Notification Service This supplies alerts to users when filters (data-mining) detects features of interest
QuakeTables Database Services The USC QuakeTables fault database project includes a web service that allows you to search for Earthquake faults.
Scientific Plotting Services We are developing Dislin-based scientific plotting services as a variation of our Web Map Service: for a given input service, we can generate a raster image (like a contour plot) which can be integrated with other scientific and GIS map plot images.
Data Tables Web Service We are developing a Web Service based on the National Virtual Observatory’s VOTables XML format for tabular data. We see this as a useful general format for ASCII data produced by various application codes in SERVO and other projects.
Key interfaces/standards/software Used
GML WFS WMS WSDL XML Schema with pull parser XPP SOAP with Axis 1.xUDDI WS-Context JSR-168 JDBC ServletsWS-Management VOTables in Research
Key interfaces/standards/software NOT Used (often just for historical reasons as project predated standard)
WS-Security JSDL WSRF BPEL OGSA-DAI
QuakeSim Services Continued
Key GIS and Related ServicesComponent Description
HPSearch Support for streaming data between services; supports scriptable workflows so not limited to DAGs; implementation of WS-Distributed Management
WS-Context Contexts can be used to hold arbitrary content (XML, URIs, name-value pairs); can be used to support distributed session state as well as persistent data; currently researching scalability.
Web Feature Service
Supports both streaming and non-streaming returns of query results.
Web Map Services
Supports integration of local and remote map services; treats Google maps as an OGC-compliant map server;
Sensor Grid Publish/subscribe system allows data streams to be reorganized using topics.
Critical Infrastructure (CI) Grids built as Grids of Grids
Command & Control Services
Physical Network
Registry Metadata
Earthquake Data& Simulation Services
Earthquake Grid DoD CIGrid… Electricity CIGrid …
Data Access/Storage
Security WorkflowNotification Messaging
Portals Visualization GridCollaboration Grid
Sensor Grid Compute GridGIS Grid
Core Grid Services
Limitations
• Lack of management of services such as:– Monitoring of machines and services– Recovery– Hot deployment– Fault tolerance, etc
• More flexible approaches to access services are expected– Allowing users to compose collections of services and
sub-Grids– The sub-Grids and services can be mapped to
resources appropriately
Our Approach• We have developed an extensible framework for managing
resources (services included)– A demo of monitoring resources– Further useful extensions will be addressed– Based on WS-Management (WS-Enumeration, WS-Transfer, WS-
Eventing, etc)
• We use BPEL to represent system and demonstrate a BPEL workflow designer– A workflow engine will be integrated for executing workflows– The management system can provide useful information such as load
balancing in the deployment
• The prototype system gives an overview of the earthquake grid and allows the user to select services/grids based on the situation assessment
QuakeSim2 Demo
Management System Demo
Workflow Designer Demo
The Infrastructure for the Demo
(Windows XP)
Grid Builder tool
- Grid template
- Workflow
- Resource monitor
(Windows XP)
Management System for
- Broker nodes
- Simulated services
(Linux)
-QuakeSim2 Portals
- ServoGrid Services
(Linux)
- Other Services
Network/Grid Middleware
QuickSim2 Results
Plot of X values
Plot of Y values
Plot of Z values
*X, Y, Z are 3D coordinates of the station spot
Lessons Learned (1)• We demonstrated that one could combine static tools
like the Grid Builder with dynamic systems management
• The end user, who serves as the administrator, can select a grid template based on the current situation
• The extended workflow designer’s Graphical User Interface – enables the user to edit the template and resource requirement– enables accuracy and data integrity improvement of user input
• The selected grid will be deployed on available resources automatically and dynamically– saves time and resources
Lessons Learned (2)
• The management system keeps monitoring resource status automatically and dynamically– saves time and resources– enables performance improvement
• The user can access services through portals which are customizable– enables information access from different devices
• Distributed, different services/grids are federated and interoperable in a seamless way
Summary• What we have done
– A management system for monitoring resources– A visual workflow editor– Portals with a set of web services– The design of a more complete framework
• Why they are important for achieving our objectives– Provide the foundation for integrating for further capabilities of Grid
Builder– The demonstration of the prototype system shows the potentials of
achieving our objectives successfully
• The scope– We leverage the technology actively developed and supported in the
grid community– Not aimed to compete for specific systems such as Condor, etc.
Goals and Plan (1)
• Short-term goals– Management system for more general
resources (in 1 ~ 2 months)• Recovery• Hot deployment• Fault-tolerance, etc
– BPEL workflow (in 3 months)• A workflow engine to execute BPEL scripts• Mapping services in a workflow to resources,
based on the status info from the management system.
• Long-term goals– Management system (in 7~9 months)
• Metadata service for different types of resources• Predictable input and output interfaces
– BPEL workflow (in 7~9 months)• Enhance the workflow execution engine with the
capability of match making to optimize the resource allocation
Goals and Plan (2)
Part 2:
Key Additional Services- Advances in
Collaboration Grid Services
I: Collaborative GroupII: Hybrid Shared Display
Advances in Collaboration Grid Services
I: Collaborative Group – Design, Implementation and Status
Advances in Collaboration Grid Services
II: Hybrid Shared Display – Design, Implementation and Status
Collaborative Groups Illustrated In Anabas Impromptu
Examples of applications: private discussions in conference/lecture simultaneous breakout groups Multiple broadcasting in the same session (e.g.
audio/voice or video/TV channels for user-defined, such as particular need-to-know, groups)
Group & Sharedlets An Anabas Sharedlet is a shared application, e.g. TextChat, VoIP,
Video Conferencing, Shared Applications, Whiteboard GroupManager provides preliminary Group information to each
sharedlet, include joined sessions, active session, session participants, participant privileges (e.g. host, presenter) in each session
Each Sharedlet has its own specific method to handle Group. E.g. Text Sharedlet stores all conversations in every sessions Video Sharedlet displays the videos in the active session only Audio Sharedlet plays the audio in the active session only Shared Display Sharedlet may store data in every sessions or
in the active session only The Sharedlet specific method depends on network bandwidth
requirement (e.g. Is the network bandwidth sufficient?) and usage difference (e.g. Can past data be disposed? Who can share information?)
GroupGroup
AudioAudio TextText
VideoVideo
HSD – Hybrid Shared Display
HSD builds on a combination of Classic Shared Display (CSD) and Video Shared Display (VSD)
Problem: Video sharing using lossless encoding scheme consumes very high network bandwidth
Motivation of HSD: Find the video or fast changing regions in the shared application, and encode them using video codec e.g. H.261 and MPEG4 to save network bandwidth while retaining good visual quality
Illustration of Hybrid Shared Display on the sharing of a browser window with a fast changing region.
Screen capturing
Region finding
Video encoding SD screen data encoding
Network transmission (RTP) Network transmission (TCP)
Video Decoding (H.261) SD screen data decoding
Rendering Rendering
Screen display
HSD Flow
Presenter
Participants
Through NaradaBrokering
VSD CSD