Upload
marianna-cain
View
213
Download
0
Embed Size (px)
Citation preview
A Swift Talk about Globus Technology:
What Can It Do for Me?OOI Cyberinfrastructure Design Meeting,
San Diego, 17-19 October
The Globus Team
(presented by Kate Keahey)
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Overview
Globus Toolkit overview What is WSRF and how does it work? Defining community: membership and laws Workflow management with Swift Globus as project: how we work
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
What Is the Globus Toolkit? The Globus Toolkit is a collection of solutions to problems that
frequently come up when trying to build collaborative distributed applications
Heterogeneity To date (v1.0 - v4.0), the Toolkit has focused on simplifying
heterogeneity for application developers We are increasingly including more “vertical solutions” that
implement typical application patterns Security
The Grid Security Infrastructure (GSI) allows collaborators to share resources without blind trust
Standards Our goal has been to capitalize on and encourage use of existing
standards (IETF, W3C, OASIS, GGF) The Toolkit also includes reference implementations of new/proposed
standards in these organizations
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
What’s In the Globus Toolkit? A Grid development environment
Develop new OGSA-compliant Web Services Develop applications using Java or C/C++ Grid APIs Secure applications using basic security mechanisms
A set of basic Grid services Job submission/management File transfer (individual, queued) Database access Data management (replication, metadata) Monitoring/Indexing system information
Tools and Examples The prerequisites for many Grid community tools
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Globus Technology Areas
Core runtime Infrastructure for building new services WSRF = Web Service Resource Framework
Security Apply uniform policy across distinct systems
Execution management Provision, deploy, & manage services
Data management Discover, transfer, & access large data
Monitoring Discover & monitor dynamic services
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
IncubatorProjects
SecurityExecution
MgmtInfo
ServicesCommonRuntime
Globus Projects
Other
MPICH G2
GridWay
Data Mgmt
IncubationMgmt
Cog WF
LRMA
GAARDS
OGROGDTE UGP
HOC-SAPURSE
GridShib
Introduce
Dyn Acct
WEEP
Gavia JSC
Gavia MS
DDM
Virt WkSp
SGGC
Metrics
ServMark
GridFTP
ReliableFile
Transfer
OGSA-DAI
GRAM
MDS4CAS
DataRepDelegation
ReplicaLocation
Java Runtime
C Runtime
Python Runtime
C Sec GT4 Docs
MEDICUS
GSI-OpenSSH
MyProxy
Swift MonMan
NetLogger
GEMLCA
GlobusToolkit
Web Services Resource Framework (WSRF)
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
“Stateless” vs. “Stateful” Services
Without state, how does client: Determine what happened (success/failure)? Find out how many files completed? Receive updates when interesting events arise? Terminate a request?
Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state
Client
FileTransferService
move (A to B)move
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
FileTransferService (without WSRF)
Developer reinvents wheel for each new service Custom management and identification of state: transferID Custom operations to inspect state synchronously (whatHappen) and
asynchronously (tellMeWhen) Custom lifetime operation (cancel)
Client
FileTransferService
move (A to B) : transferIDmove
statewhatHappen
tellMeWhen
cancel
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
WSRF in a Nutshell Service State representation
Resource Resource Property
State identification Endpoint Reference
State Interfaces GetRP, QueryRPs,
GetMultipleRPs, SetRP Lifetime Interfaces
SetTerminationTime ImmediateDestruction
Notification Interfaces Subscribe Notify
ServiceGroups
RPs
Resource
ServiceGetRP
GetMultRPs
SetRP
QueryRPs
Subscribe
SetTermTime
Destroy
EPREPR
EPR
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
FileTransferService (w/ WSRF)
Developer specifies custom method to createResource and leaves the rest to WSRF standards:
State exposed as Resource + Resource Properties and identified by Endpoint Reference (EPR)
State inspected by standard interfaces (GetRP, QueryRPs) Lifetime management by standard interfaces (Destroy)
ClientFileTransferService
createResource (A to B) : EPRcreateResource
RPs
Transfer getRP
queryRPs
destroy
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
The Introduce Authoring Tool Define service Create skeleton Discover types Add operations Configure security Modify service
Introduce: Hastings, Saltz, et al., Ohio State University
Generates GT4-compatible WebServices
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Defining Community: Membership and Laws
Identify VO participants and roles For people and services
Specify and control actions of members Empower members delegation Enforce restrictions federate policy
A
1 2
B
1 2
A B
1
10
1
10
1
16
Access granted by community
to user
Site admission-
control policies
EffectiveAccess
Policy of site to
community
Courtsey : DOE report : LBNL-41349 : Authorization & Attribute Certificates for Widely Distributed Access Control
XACML
SAML
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Globus Authorization Framework
VOMS Shibboleth LDAP PERMIS…
GT4 ClientGT4 Server
PDP
AttributesAuthorization
Decision
PIP PIP PIP
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Swift System Clean separation of logical/physical concerns
XDTM specification of logical data structures
+ Concise specification of parallel programs SwiftScript, with iteration, etc.
+ Efficient execution on distributed resources Karajan threading, Falkon provisioning, Globus
interfaces, pipelining, load balancing
+ Rigorous provenance tracking and query Virtual data schema & automated recording
Improved usability and productivity Demonstrated in numerous applications
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Workflow Language - SwiftScript
Goal: Natural feel to expressing distributed applications Variables (basic, data structures) Conditional operators (if, foreach, ) Functions (atomic / compound)
Used to connect outputs to inputs It does not specify invocation order, only dependencies It can be seen as a metadata for expressing
experiments
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Execution Engine
Karajan engine (event-based execution) Has a scheduler to map tasks to resources
Score-based planning Recovers from failures (retries)
Falkon resource manager creates a “private cluster” Uses Globus GRAM4 (PBS/Condor/Fork) to
acquire resources from Grid systems
Virtual Node(s)
SwiftScript
Abstractcomputation
Virtual DataCatalog
SwiftScriptCompiler
Specification Execution
Virtual Node(s)
Provenancedata
ProvenancedataProvenance
collector
launcher
launcher
file1
file2
file3
AppF1
AppF2
Scheduling
Execution Engine(Karajan w/
Swift Runtime)
Swift runtimecallouts
C
C CC
Status reporting
Provisioning
ResourceProvisionersFalkon/VWS
AmazonEC2
Dynamic Provisioning:Swift Architecture
Yong Zhao, Mihael Hatigan, Ioan Raicu, Mike Wilde, Ben Clifford
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Globus Philosophy
Globus was first established as an open source project in 1996
The Globus Toolkit is open source to: Allow for inspection
for consideration in standardization processes
Encourage adoption in pursuit of ubiquity and interoperability
Encourage contributions harness the expertise of the community
The Globus Toolkit is distributed under the (BSD-style) Apache License version 2
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
dev.globus Governance model based on Apache Jakarta
Consensus based decision making Globus software is organized as several
dozen “Globus Projects” Each project has its own “Committers”
responsible for their products Cross-project coordination through shared
interactions and committers meetings A “Globus Management Committee”
Overall guidance and conflict resolution
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
http://dev.globus.org
Guidelines(ApacheJakarta)
Infrastructure(CVS, email,
bugzilla, Wiki)
ProjectsInclude
…
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Open Source != “Free time”
Globus development is well-funded The open source model facilitates contributions NSF and DOE sponsor Globus development at several
institutions via multiple grants, totaling >$5M/yr Non-U.S. science agencies also contribute to Globus
development Corporations also sponsor developers
NSF explicitly funds Globus improvements CDIGS: Community-Driven Improvements to Globus
Software
10/18/07, ORION meeting Virtual Workspaces: http://workspace.globus.org
Parting Thoughts
There is a wealth of available Grid technology that can be leveraged for this project Security mechanisms, workflow, and other
services The Globus Toolkit is funded by NSF to
support scientific communities.