Slide 1

The Grid Ian Foster Argonne National Laboratory University of Chicago Globus Alliance www.mcs.anl.gov/~foster

Slide 2

2 foster@mcs.anl.gov ARGONNE CHICAGO Grid Computing is in the News

Slide 3

3 foster@mcs.anl.gov ARGONNE CHICAGO Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology

Slide 4

4 foster@mcs.anl.gov ARGONNE CHICAGO Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology

Slide 5

5 foster@mcs.anl.gov ARGONNE CHICAGO The Power Grid: On-Demand Access to Electricity Time Quality, economies of scale Decouple production & consumption, enabling l On-demand access l Economies of scale l Consumer flexibility l New devices

Slide 6

6 foster@mcs.anl.gov ARGONNE CHICAGO But Computing Isnt Really Like Electricity! l How about access computing resources like we access Web content? u We have no idea where a website is, or on what computer or operating system it runs Two interrelated opportunities 1) Enhance economy, flexibility, access by virtualizing computing resources 2) Deliver entirely new capabilities by integrating distributed resources

Slide 7

7 foster@mcs.anl.gov ARGONNE CHICAGO Virtualization Servers: Execution Application Services: Distribution Applications: Delivery Application Virtualization Automatically connect applications to services Dynamic & intelligent provisioning Infrastructure Virtualization Dynamic & intelligent provisioning Automatic failover Source: The Grid: Blueprint for a New Computing Infrastructure (2 nd Edition), 2004

Slide 8

8 foster@mcs.anl.gov ARGONNE CHICAGO Distributed System Integration NY Financial Institution Insurance Group UK Financial Institution NY Financial Institution Capital Markets Group

Slide 9

9 foster@mcs.anl.gov ARGONNE CHICAGO And Ultimately

Slide 10

10 foster@mcs.anl.gov ARGONNE CHICAGO The Grid Resource sharing & coordinated problem solving in dynamic virtual organizations 1. Enable integration of distributed service & resources 2. Using general-purpose protocols & infrastructure 3. To achieve useful qualities of service The Anatomy of the Grid, Foster, Kesselman, Tuecke, 2001

Slide 11

11 foster@mcs.anl.gov ARGONNE CHICAGO Terminology (1) Cluster Grid Enterprise Grid Global Grid

Slide 12

12 foster@mcs.anl.gov ARGONNE CHICAGO Terminology (2) Autonomic Computing Service- Oriented Architecture Utility Computing Utility Computing Autonomic Computing Service- Oriented Architecture Grid

Slide 13

13 foster@mcs.anl.gov ARGONNE CHICAGO Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology

Slide 14

14 foster@mcs.anl.gov ARGONNE CHICAGO Grid Addresses Pain Points Now l Low utilization of enterprise resources l High cost of provisioning for peak demand l Inadequate resources prevent use of advanced applications l Lack of information integration

Slide 15

15 foster@mcs.anl.gov ARGONNE CHICAGO Grid Deployment Trends Mission Criticality Department Enterprise Collaboration Internet Corporate Scientific Corporate

Slide 16

16 foster@mcs.anl.gov ARGONNE CHICAGO Gridified Infrastructure Early Commercial Applications Financial Services Derivatives Analysis Statistical Analysis Portfolio Risk Analysis Derivatives Analysis Statistical Analysis Portfolio Risk Analysis Manufacturing Mechanical/ Electronic Design Process Simulation Finite Element Analysis Failure Analysis Mechanical/ Electronic Design Process Simulation Finite Element Analysis Failure Analysis LS / Bioinformatics Cancer Research Drug Discovery Protein Folding Protein Sequencing Cancer Research Drug Discovery Protein Folding Protein Sequencing Other Web Applications Weather Analysis Code Breaking/ Simulation Academic Web Applications Weather Analysis Code Breaking/ Simulation Academic Sources: IDC, 2000 and Bear Stearns- Internet 3.0 - 5/01 Analysis by SAI Grid Services Market Opportunity 2005 Energy Seismic Analysis Reservoir Analysis Seismic Analysis Reservoir Analysis Entertainment Digital Rendering Massive Multi-Player Games Massive Multi-Player Games Streaming Media Leading adopters (Oct 2003) * Financial services: 31% Life sciences: 26% Manufacturing: 18% *Grids 2004: From Rocket Science To Business Service, The 451 Group

Slide 17

17 foster@mcs.anl.gov ARGONNE CHICAGO Grid Deployment Strategies l A range of excellent commercial & open source products for resource federation u Federate enterprise computing resources u Federate enterprise information resources u Globus Toolkit : inter-enterprise sharing l But, Grids are built, not bought u Integration with other enterprise systems is needed to deliver complete solution l Start small & with well-defined ROI case u Grow based on experience

Slide 18

18 foster@mcs.anl.gov ARGONNE CHICAGO Novartis PC Grid links 3,700 desktop systems u Research & development applications u Potentially mainstream business computing u > 5 teraflop/s computing power u Estimate savings of $200M over 3 years We have projects we calculate would take 6 years on a single supercomputer. Today, the run time is 12 hours. Peter Sany, Novartis CIO

Slide 19

19 foster@mcs.anl.gov ARGONNE CHICAGO University of Texas

Slide 20

20 foster@mcs.anl.gov ARGONNE CHICAGO Royal Dutch/Shell l Improve accuracy and speed of summarization & scientific modeling applications u More robust, scalable IT infrastructure that adjusts as volumes fluctuate u Open standards ease integration of software u Cut processing time of seismic data, while improving the quality of the output u Focus employees on key scientific, not IT problems Grid computing is important to Shell because it offers the potential to create a truly unlimited resource, with a uniform interface to a variety of services. This is a significant opportunity for Shell to engage its independent companies in closer cooperation. J.N. Buur, Principal Research Physicist, Shell International Exploration and Production B.V.

Slide 21

21 foster@mcs.anl.gov ARGONNE CHICAGO Kansai Electric Power Co. l Japans second largest electric utility company has various information in a heterogeneous, distributed database environment l Integrate information across departments and affiliated companies to enable information sharing Federated Database DB2 Server ADABAS Server NOTES Server Oracle Server Application DB Client CRM Accounting Workflow

Slide 22

NEES (Network for Earthquake Engineering Simulation) Collaboratory U.Nevada Reno www.neesgrid.org

Slide 23

23 foster@mcs.anl.gov ARGONNE CHICAGO NCSA Computational Model m1m1 f1f1 UIUC Experimental Model f1f1 m1m1 f2f2 f2f2 U. Colorado Experimental Model NEESgrid at Work

Slide 24

24 foster@mcs.anl.gov ARGONNE CHICAGO CMS High Energy Physics l Generate data for a physics experiment u Simulate 1.5 million events for physics studies @ ~500 sec per event u 2 months continuous running across 5 sites u Managed by a single person

Slide 25

25 foster@mcs.anl.gov ARGONNE CHICAGO Why Should You Care? 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology

Slide 26

26 foster@mcs.anl.gov ARGONNE CHICAGO Grid Vision, Marketing, and Reality l Vision u Computing resources can be shared like content on the Wb l Marketing u Have we got a [Data, compute, knowledge, information, desktop, PC, enterprise, cluster, ] Grid for you! l Reality u Commercial products mostly noninteroperable u Open source tools offer de facto standards, but are also far from a complete solution

Slide 27

27 foster@mcs.anl.gov ARGONNE CHICAGO Standards Matter! l Open, standard protocols u Enable interoperability u Avoid product/vendor lock-in u Enable innovation/competition on end points u Enable ubiquity l In Grid space, must address how we u Describe, discover, & access resources u Monitor, manage, & coordinate, resources u Account & charge for resources For many different types of resource

Slide 28

28 foster@mcs.anl.gov ARGONNE CHICAGO Increased functionality, standardization Custom solutions 1990199520002005 Open Grid Services Arch Real standards Multiple implementations Web services, etc. Managed shared virtual systems Research Globus Toolkit Defacto standard Single implementation Internet standards Developing Grid Standards 2010

Slide 29

29 foster@mcs.anl.gov ARGONNE CHICAGO Open Grid Services Architecture Adopt service-oriented architecture u Key to virtualization, discovery, composition, local-remote transparency + Standard service description & access u Leverage industry standard Web services + Distributed service management protocols u A component model for Web services = A framework for creating, managing, & delivering interoperable services The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration, Foster, Kesselman, Nick, Tuecke, 2002

Slide 30

30 foster@mcs.anl.gov ARGONNE CHICAGO Web Services: Basic Functionality OGSA Open Grid Services Architecture (GGF, OASIS, W3C) OGSI: Interface to Grid Infrastructure Applications in Problem Domain X Compute, Data & Storage Resources Distributed Application & Integration Technology for Problem Domain X Users in Problem Domain X Virtual Integration Architecture Generic Virtual Service Access and Integration Layer - Structured Data Integration Structured Data Access Structured Data RelationalXMLSemi-structured Transformation Registry Job Submission Data TransportResource Usage Banking BrokeringWorkflow Authorisation

Slide 31

31 foster@mcs.anl.gov ARGONNE CHICAGO Open Grid Services Infrastructure Implementation Service data element Other standard interfaces: factory, notification, collections Hosting environment/runtime (C, J2EE,.NET, ) Service data element Service data element GridService (required) Data access Lifetime management Explicit destruction Soft-state lifetime Introspection: What port types? What policy? What state? Client Grid Service Handle Grid Service Reference handle resolution

Slide 32

32 foster@mcs.anl.gov ARGONNE CHICAGO OGSA Standards: Status l WSDL 1.2 in progress in W3C l OGSI 1.0 completed July 2003 in GGF l WS-Agreement draft in GGF l WS-Management drafts in OASIS l OGSA Data Access & Integration in GGF l WS-Security specifications in OASIS l SAML & XACML in OASIS l WS-Addressing status unclear l Other work in progress

Slide 33

33 foster@mcs.anl.gov ARGONNE CHICAGO Standards: Summary l Grid and Web Services are merging u Grid is an aggressive use case of Web Services l Web Services standards landscape is in flux u OGSI/A will need to evolve with it u Uncertain status of security & policy standards continues to be a big source of concern l Grid services standards landscape heating up l W3C, OASIS, GGF are key standards orgs l Open source software important for adoption

Slide 34

34 foster@mcs.anl.gov ARGONNE CHICAGO OGSA Status: Implementations l Globus Toolkit v3: Linux for the Grid u Open source middleware, commercial support u A range of computation & data management, registry, and security functions l Some nice announced OGSI-based products u IBM, Avaki, Platform, Sun, NEC, HP, UD, Entropia, DataSynapse, Insors, Oracle, etc. u Read the fine print: Intent is to use OGSI- based products, OGSA-compliant software, Embraces fundamental OGSA concepts

Slide 35

35 foster@mcs.anl.gov ARGONNE CHICAGO Globus Toolkit History: An Unreliable Memoir DARPA, NSF begin funding Grid work NASA initiates Information Power Grid Globus Project wins Global Information Infrastructure Award MPICH-G released The Grid: Blueprint for a New Computing Infrastructure published GT 1.0.0 Released Early Application Successes Reported GT 1.1.1 Released GT 1.1.2 Released GT 1.1.3 Released NSF & European Commission Initiate Many New Grid Projects GT 1.1.4 and MPICH-G2 Released Anatomy of the Grid Paper Released First EuroGlobus Conference Held in Lecce Significant Commercial Interest in Grids NSF GRIDS Center Initiated GT 2.0 beta Released Physiology of the Grid Paper Released GT 2.0 Released GT 2.2 Released Only Globus.Org; not downloads from: NMI UK eScience EU DataGrid IBM Platform etc.

Slide 36

36 foster@mcs.anl.gov ARGONNE CHICAGO HP Grid Product Offerings l Globus Toolkit for HP-UX, Tru64 Unix, Linux u Globus Toolkit 2.4 u Tested and optimized for HP platforms u Commercial support! l HP Utility Data Center (UDC) u Data Center operation & construction services u Service-centric design u Intent is to construct UDC products using OGSI-compliant services l Enterprise Grid consulting

Slide 37

37 foster@mcs.anl.gov ARGONNE CHICAGO HP and Planetary Computing shared, traded resources value clusters grid-enabled systems programmable data center virtual data center Open VMS clusters, TruCluster, MC ServiceGuard Tru64, HP-UX, Linux switch fabric computestorage UDC computing utility or GRID today

Slide 38

38 foster@mcs.anl.gov ARGONNE CHICAGO Pop Quiz: The Grid Is a)A collaboration & resource sharing infrastructure for scientific applications b)A distributed service integration and management technology c)A disruptive technology that enables a virtualized, collaborative, distributed world d)An open source technology & community e)A marketing slogan f)All of the above

Slide 39

39 foster@mcs.anl.gov ARGONNE CHICAGO Summary: Why You Should Care 1) Grid is a disruptive technology [Vision] u It ushers in a virtualized, collaborative, distributed world 2) Grid addresses pain points now [Reality] u Grids are built not bought, but are delivering real benefits in commercial settings 3) An open Grid is to your advantage [Future] u Standards are being defined now that will determine the future of this technology

Slide 40

40 foster@mcs.anl.gov ARGONNE CHICAGO To Learn More Deep tech & strategic info Jan 20-23, 2004, San Fran www.globusworld.org Working, research groups Three meetings a year www.ggf.org Weekly email & web newsletter www.gridtoday.com Commercial conf & expo May 24-26, Philadelphia www.gridtoday.com 2nd Edition www.mkp.com/grid2 Background information www.mcs.anl.gov/ ~foster Become a Globus corporate affiliate