Ian Foster

Computation Institute

Argonne National Lab & University of

Chicago

A Sociology of the Grid?

CarlKesselman

Information Sciences Institute, University

of Southern California

3

“When the network is as fast as the

computer’s internal links,

the machine disintegrates across the net into a set of

special purpose appliances”

(George Gilder, 2001)

4

“The Anatomy of the Grid,” 2001 The … problem that underlies the Grid concept is

coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations. The sharing that we are concerned with is not primarily file exchange but rather direct access to computers, software, data, and other resources, as is required by a range of collaborative problem-solving and resource -brokering strategies emerging in industry, science, and engineering. This sharing is, necessarily, highly controlled, with resource providers and consumers defining clearly and carefully just what is shared, who is allowed to share, and the conditions under which sharing occurs. A set of individuals and/or institutions defined by such sharing rules form what we call a virtual organization (VO).

5

Examples (from AotG, 2001)

“The application service providers, storage service providers, cycle providers, and consultants engaged by a car manufacturer to perform scenario evaluation during planning for a new factory”

“Members of an industrial consortium bidding on a new aircraft”

“A crisis management team and the databases and simulation systems that they use to plan a response to an emergency situation”

“Members of a large, international, multiyear high-energy physics collaboration”

6

Quantitative medicine is the key to reducing healthcare costs and

improving healthcare outcomes

Patients with same diagnosis

7

Quantitative medicine is the key to reducing healthcare costs and

improving healthcare outcomes

Patients with same diagnosis

Misdiagnosed

Non-responders,toxic responders

Non-toxic responders

8

Virtual organizations in medicine span multiple dimensions

Basic Research

Clinical Practice

Clinical Trials

trial subjects, outcomes

library

Outco

mes

,

tissu

e ba

nksc

reen

ing

test

s

ongoing

investigative

studies

pathways

9

From the organizational behavior and management community

“[A] group of people who interact through interdependent tasks guided by common purpose [that] works across space, time, and organizational boundaries with links strengthened by webs of communication technologies”

— Lipnack & Stamps, 1997

Yes—but adding cyber-infrastructure: People computational agents & services Communication technologies IT infrastructure

Collaboration based on rich data & computing capabilities

10

NSF Workshops on

Building Effective Virtual

Organizations

[Search “BEVO 2008”]

11

Two perspectives Technology used to enhance collaboration

(Computer Supported Collaborative Work)

Collaboration used to enhance technology

12

What is an organization?

A organization has an identity and a purpose, which it seeks to fulfill within its environment

The organization’s purpose influences its participants, structure, activities, and deliverables, whether products or services

The organization’s performance can be evaluated with respect to various metrics

Is a virtual organization any different?

13

Identity Legal aspects. Credentials.

Purpose [Let’s assume] anything legal

EnvironmentAvailable service & resource providers. Legal & organizational constraints

ParticipantsIdentity-based or attribute-based.People, services, resources, sensors.

Structure Centralized, decentralized, …

Activities Business processes. Workflows.

DeliverablesData products. Services. Instrument operations. …

PerformanceThroughput, responsiveness, growth, happiness, security, …

14

“I can’t solve this problem alone—I

need to involve my buddies Sebastien

and Krishna”

It looks like you’re creating a VO

Get help withcreating the VO

Just create the

VO without help

“Where am I going to eat

tonight?”

From: IanTo: Krishna, SebastienSubject: Help me find a restaurant

15

Identity Geneva-Dinner-VO. Informal, so no legal status.

Purpose Find a good restaurant in Geneva. Avoid one that speaks to you in English.

Environment News feeds. Satellite data feeds. Analytic services. EGEE, OSG, TeraGrid.

Participants Ian, Krishna, Sebastien. Anyone else we trust to make good recommendations.

Structure Central database and associated services (membership, …)

Activities Restaurant identification workflow. English-speaking staff tracking workflow.

Deliverables Restaurant recommendation for today. Recommendation service. Business plan?

Performance Food quality. Cost. Service response time. VO setup time.

16

Building a Virtual Organization Define &implement policy

Negotiation, trust management, credentials Determine and implement membership & roles

Terms of engagement Virtualization & integration of providers

Create VO-wide services Global behaviors

Manage work Collaborative problem solving, workflow

management Manage the VO

Monitor performance, report metrics

17

Defining community: Membership & laws Identify VO participants and roles

And map participants to attributes and roles Specify and control actions of members

Empower members delegation Enforce restrictions federate policy

A

1 2

B

1 2

A B

1

10

1

10

1

16

20

A set of core security mechanisms

Attribute Assertions C asserts that S has attribute A with value V

Authentication and digital signature Allows signer to assert attributes

Delegation C asserts that S can perform O on behalf of C.

Attribute mapping {A1, A2… An}VO1 {A’1, A’2… A’m}VO2

Policy Entity with attributes A asserted by C may

perform operation O on resource R

21

Trust in VOs

Do I “believe” an attribute assertion Used to evaluate cost vs. benefit of performing

an operation E.g., perform untrusted operation with extra

auditing Look at attributes of assertion signer Rooting trust

Externally recognized source, e.g., CA Dynamically via VO structure delegation Dynamically via alternative sources, e.g.,

reputation

22

Building blocks

Attribute Authority (ATA): Issue signed attribute assertions

(including identity, delegation, mapping)

Authorization Authority (AZA) Makes decisions based on assertions & policy

ATA

User A is an admin

User B is a member

User B can use service

X

23

VO policy at a service

GT4 authorization and delegation services provide implementations

ResourceATA

WSResource

ResourceAZA

VO ATA

WS-Subject

ATA: Attribute AuthorityAZA: Authorization Authority

24

Establishing VO-Wide policy

ResourceATA

WSResource

ResourceAZA

VO ATAVO AZA

SubjectATA

SubjectAZA

WS-Subject

ATA: Attribute AuthorityAZA: Authorization Authority

GT4 authorization and delegation services provide implementations

26

Protected health information problem

What do we want? Use clinical data for research Share clinical data, make research data available Reuse same infrastructure Image exchange between health providers

Patient authorizes use of data – consent process Intact unmodified DICOM workflow for diagnostics De-identified DICOM workflow for research

(Modality profiles) Group authorization problem: Patient data–to-user

(Physician/Researcher) relationship not manageable!

27

Patient

Primary Care Physician

HealthGrid

Pathologist

Radiologist HospitalPhysician

Visit

Medical Images Personal Health Record

Neurosurgeon

Treatment Planning

Pathology Report

Medical Record

PHR Vendor

28

Patient-authorized grid image workflow

Patient

Healthcare Provider

Internet2 IdP

Globus OGSA-DAIMeta Catalog Service

PHI safe entries

2.1

2.2

Hippocratic Verification ServicePolicy Decision Point (PDP)

2.3

Globus GridFTPStorage Service Provider

Compressed DICOM Series Records

4.1

4.3

4.4

Physician

Globus RLSReplica Location Service

4.2

GridShib

5.1

1.1

3.1

MEDICUS(Erberich et al.)

29

HIPAA-compliant research access

Research Center

Globus OGSA-DAIMeta Catalog Service

PHI safe entries

2.1

Hippocratic Verification ServicePolicy Decision Point (PDP)

2.2

Globus GridFTPStorage Service Provider

Compressed DICOM Series Records

3.2

4.2

Researcher

Globus RLSReplica Location Service

3.1

4.1

DICOM De-Identification ServiceModality Profiles

1.1

MEDICUS(Erberich et al.)

30

PHI-safe workflow with patient-centric authorization

Healthcare Provider Healthcare Provider

MCS

HVS RLS

SSP

HVS

Los Angeles CountyCalifornia

Columbia CountyNew York

3.2

4.1

4.2 2.1

IdP

2.2

2.3

3.1

1.1

31

Online-CAAuthN Svc

Application Client + PKIClient

App Svc 1.LoginUsername/Password

0. Trusted CA/CRLs

AuthenticationDB

ProvisioningDatabase

Attribute Service

MyProxyLogin with

Provisioning

32

Online-CAAuthN Svc

Application Client + PKIClient

App Svc

2. Authentication and Attributes retrieval

0. Trusted CA/CRLs

AuthenticationDB

ProvisioningDatabase

Attribute Service

MyProxyLogin with

Provisioning

33

Online-CAAuthN Svc

Application Client + PKIClient

App Svc

3. Short term X509 credentials with

attributes, CAs, CRLs

0. Trusted CA/CRLs

AuthenticationDB

ProvisioningDatabase

Attribute Service

MyProxyLogin with

Provisioning

34

Online-CAAuthN Svc

Application Client + PKIClient

App Svc

0. Trusted CA/CRLs

Authentication DB

ProvisioningDatabase

Attribute Service

4. Access using X509 Credentials

MyProxyLogin with

Provisioning

35

Online-CAAuthN Svc

Application Client + PKIClient

App Svc

0. Trusted CA/CRLs

Authentication DB

ProvisioningDatabase

Attribute Service

5. Update trust roots

MyProxyLogin with

Provisioning

36

Browser

Web SSO using OpenID

Application

ServerService

Provider (SP/RP)

IdentityProvider

(IdP)Authentication

DB

Site Attribute Service

37

Browser

Web SSO using OpenID

Application

ServerService

Provider (SP/RP)

IdentityProvider

(IdP)Authentication

DB

Site Attribute Service

1. Client access application

server

38

Browser

Web SSO using OpenID

Application

ServerService

Provider (SP/RP)

IdentityProvider

(IdP)Authentication

DB

Site Attribute Service

2. Redirected to Identity Provider

39

Browser

Web SSO using OpenID

Application

ServerService

Provider (SP/RP)

IdentityProvider

(IdP)Authentication

DB

Site Attribute Service

3. User authenticates

with IdP

40

Browser

Web SSO using OpenID

Application

ServerService

Provider (SP/RP)

IdentityProvider

(IdP)Authentication

DB

Site Attribute Service

4. AuthN completed,

user identity.

41

Browser

Web SSO using OpenID

Application

ServerService

Provider (SP/RP)

IdentityProvider

(IdP)Authentication

DB

Site Attribute Service

5. Authenticated Call.

42

Making it easy: Social VPNs

Alice

CarolBobSocialNetwork

Web interface

Social network(e.g. Facebook)

Overlay network(IPOP)

carol.facebook.ipop10.10.0.2 node0.alice.facebook.ipop

10.10.0.3

SocialNetwork API

Social network Information system

Alice’s public key certificateBob’s public key certificate

Carol’s public key certificate

Identities are managed with web-based interface profiles;Public key certificates retrieved through API

Symmetric keys exchanged and point-to-point private tunnels created on demand;

Multicast-based resource discovery

Bob: browses Alice’s SMB share

RenatoFigueiredo

43

Globus

44

As of Oct19, 2008:

122 participants105 services

70 data35

analytical

45

Registries(E.g., caBIG)

Core Services

Grid Service

Uses TerminologyDescribed In

Cancer Data Standards Repository

Enterprise Vocabulary

Services

References ObjectsDefined in

Index Service

Service Metadata

Publishes

Subscribes Toand Aggregates

Queries ServiceMetadata Aggregated In

Registers To

Discovery Client API

46

ApplnService

Create

Index service

StoreRepository

ServiceAdvertize

Discover

Invoke;get results

Introduce

Container

Transfer GAR

Deploy

Ohio State University and Argonne/U.Chicago

Service oriented medicine:caGrid, Introduce, and gRAVI

Introduce Define service Create skeleton Discover types Add operations Configure security

Grid Remote Application Virtualization Infrastructure Wrap executables

Globus

47

Microarray clustering using Taverna

1. Query and retrieve microarray data from a caArray data service:cagridnode.c2b2.columbia.edu:8080/wsrf/services/cagrid/CaArrayScrub

2. Normalize microarray data using GenePattern analytical service node255.broad.mit.edu:6060/wsrf/services/cagrid/PreprocessDatasetMAGEService

3. Hierarchical clustering using geWorkbench analytical service: cagridnode.c2b2.columbia.edu:8080/wsrf/services/cagrid/HierarchicalClusteringMage

Workflow in/output

caGrid services

“Shim” servicesothers

Wei Tan

48

VO as a Service (VOaaS)

Virtual organizations integrate participants and resource providers Participants are selected or self assemble Select “best of breed” providers for VO services

Much of this process can be automated Provisioning of enabling services, at least

Function

Resource

49

Community

Services Provider

Content

Services

Capacity

1

3

4

5

VOs assemble services

Integrate services from various sources Virtualize external services as VO services

Deploy new services for the VO

Capacity Provider

2

51

Providing VO services

Integrate existing services

Delegate and deploy capabilities/services Provision service to deliver defined capability Configure execution environment Host higher-level functions GRAM, Nimbus, EC2, …

Coordinate and compose Build new functions from individual services

52

ApplnService

Create

Index service

Store

Repository ServiceAdvertize

Discover

Invoke;get results

Introduce

Container

Transfer GAR

Deploy

gRAVI: Ravi Madduri et al., Argonne/U.Chicago & OSU

Service authoring and deployment grid Remote

Application Virtualization Infrastructure

Builds on Introduce Define service Create skeleton Discover types Add operations Configure security

Wrap arbitrary executables

RaviMadduri

53

Pull “missing” files to a storage system

List of required Files

GridFTPLocalReplicaCatalog

ReplicaLocationIndex

Data Replicati

on Service

Reliable File

Transfer Service Local

ReplicaCatalog

GridFTP

Service Composition:Data Replication Service

“Design and Implementation of a Data Replication Service Based on the Lightweight Data Replicator System,” Chervenak et al., 2005

ReplicaLocationIndex

Data MovementData Location

Data Replication

AnnChervenak

54

Decomposition EnablesSeparation of Concerns & Roles

User

ServiceProvider

“Provide access to data D at S1, S2,

S3 with performance P”

ResourceProvider

“Provide storage with performance P1, network with

P2, …”

D

S1

S2

S3

D

S1

S2

S3Replica catalog,User-level multicast, …

D

S1

S2

S3

55

Policy, revisited

Traditionally policy is enforced at end points, integrated with application E.g., PDP call-out in Globus container

We can also apply policy at the VO level Define interactions between services at the

organizational level Factor policy out of service implementations

56

Policy-driven service oriented architecture

Need stand-alone policy engine to coordinate at VO level

Connection between application policy and infrastructure policy (dynamic provisioning)

Policy extension points designed into services allow Coordination at VO level Dynamic policy enforcement

across services and service oriented infrastructure

Web Services 2.0: Policy-driven Service Oriented Architectures Thomas B Winans and John Seely Brown

57

2001 view of the “grid problem”

Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations

Too limited a view

58

We need an end-to-end perspective

A organization has an identity and a purpose, which it seeks to fulfill within its environment

The organization’s purpose influences its participants, structure, activities, and deliverables, whether products or services

The organization’s performance can be evaluated with respect to various metrics

Then focus on clear identification of roles, separation of concerns, isolation of policy