Grid Computing Products and Services - IBM Redbooks · 8.3.9 Grid Clash Analysis ... B-3 World Community Grid: Architecture ... Joao Almeida Fabiano Lucchese Grid . Grid Computing

ibm.com/redbooks

Grid Computing Products and Services

Luis FerreiraMariano BatistaSebastien Fibra

Chin Yau LeeCarlos Alexandre Queiroz Silva

Joao AlmeidaFabiano Lucchese

Nam Keung

Grid computing functionality

Products for grid computing

Services and solutions

Front cover

http://www.redbooks.ibm.com/



August 2005

International Technical Support Organization

SG24-6650-00

© Copyright International Business Machines Corporation 2005. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.

First Edition (August 2005)

This edition applies to the capability of the IBM, ISVs, and open source products used to build a grid computing solution.

Note: Before using this information and the product it supports, read the information in, “Notices” on page xiii.

Contents

Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvThe team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviBecome a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi

Part 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1.1 Grid computing as an evolution of distributed computing . . . . . . . . . . 41.1.2 Grid computing models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.1.3 Grid and its relationship with other IT components . . . . . . . . . . . . . . 11

1.2 Categories and functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.2.1 Categorization of components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131.2.2 Functionality on grid computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.2.3 Grid computing products and players . . . . . . . . . . . . . . . . . . . . . . . . 20

Part 2. Functionality and products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Chapter 2. Web services core/hosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.2 Grid standards and toolkits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.2.1 Grid standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.2.2 OGSA Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.3 Grid application environments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.3.1 WebSphere Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.3.2 WebSphere Extended Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . 322.3.3 WebSphere Portal Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.3.4 WebSphere MQ Family. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.3.5 WebSphere Messaging Resources. . . . . . . . . . . . . . . . . . . . . . . . . . 372.3.6 WebSphere Business Integration Messaging Broker . . . . . . . . . . . . 38

2.4 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

© Copyright IBM Corp. 2005. All rights reserved. iii

Chapter 3. Workload virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.1 Scheduling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.2 Grid scheduling and policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.3 How to choose the appropriate scheduler. . . . . . . . . . . . . . . . . . . . . . . . . 443.4 Mapping user requirements into products . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.4.1 Tivoli Workload Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.4.2 LoadLeveler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.4.3 DataSynapse GridServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483.4.4 Platform LSF (Load Sharing Facility) . . . . . . . . . . . . . . . . . . . . . . . . 503.4.5 Platform LSF MultiCluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.4.6 Platform LSF HPC (High-Performance Computing) . . . . . . . . . . . . . 513.4.7 Platform Symphony. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523.4.8 United Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523.4.9 Altair PBS Professional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.4.10 OpenPBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553.4.11 Condor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.4.12 Community Scheduler Framework (CSF) . . . . . . . . . . . . . . . . . . . . 573.4.13 Maui Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

3.5 Architecture examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.5.1 Desktop grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.5.2 Server grid. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Chapter 4. Information virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.2 Information virtualization layers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4.2.1 Block data virtualization layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2.2 File system virtualization layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2.3 Information virtualization layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.3 Requirements and products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694.3.1 Defining requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694.3.2 Block data layer products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.3.3 Files system layer products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.3.4 Information layer products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.4 Case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 814.4.1 Problem overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824.4.2 Grid solution planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824.4.3 Federation of data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Chapter 5. Other disciplines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 865.2 Provisioning and orchestration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

5.2.1 Provisioning and orchestration products . . . . . . . . . . . . . . . . . . . . . . 885.2.2 Provisioning and orchestration case study . . . . . . . . . . . . . . . . . . . . 89

iv Grid Computing Products and Services

5.2.3 Correlated products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915.3 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.3.1 Firewalls, VPNs, and virtual organizations . . . . . . . . . . . . . . . . . . . . 935.3.2 Security products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.3.3 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

5.4 Directory Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 985.4.1 Directory Services products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 985.4.2 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.5 Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995.5.1 Licensing products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1015.5.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.6 Billing and metering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Part 3. Bundles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Chapter 6. Virtualization Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1056.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.2 Virtualization Engine suites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

6.2.1 IBM Virtualization Engine Suite for Servers . . . . . . . . . . . . . . . . . . 1086.2.2 IBM Virtualization Engine Suite for Storage . . . . . . . . . . . . . . . . . . 110

6.3 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Chapter 7. ISV suites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1137.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1147.2 Platform Computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1147.3 DataSynapse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1167.4 United Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Part 4. Services and solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Chapter 8. Services and solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1218.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1228.2 Services portfolio for grid computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

8.2.1 Grid Innovation Workshop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1248.2.2 Grid Strategy Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1258.2.3 Grid Design Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1268.2.4 Grid Software Implementation Services . . . . . . . . . . . . . . . . . . . . . 1278.2.5 Grid Training and Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1288.2.6 IBM support line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

8.3 Solutions portfolio for grid computing . . . . . . . . . . . . . . . . . . . . . . . . . . . 1298.3.1 IBM Grid Value at Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1308.3.2 Analytics Acceleration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1318.3.3 Design Collaboration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

Contents v

8.3.4 Engineering Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1338.3.5 Geophysical Processing and Analysis . . . . . . . . . . . . . . . . . . . . . . 1348.3.6 Information Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1368.3.7 IT Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1378.3.8 Research Collaboration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1378.3.9 Grid Clash Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1398.3.10 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

8.4 Services approach to Grid Projects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Part 5. Appendixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Appendix A. Surrounding initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145World Community Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146Globus Alliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146Global Grid Forum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146OASIS and WSRF TC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147W3C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147TeraGrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Appendix B. World Community Grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150Technical features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153High level design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154Products used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158More information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

World Community Grid Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161Example of the World Community Grid Screen Saver . . . . . . . . . . . . . . . 162www.worldcommunitygrid.org . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

Appendix C. Products reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169Big picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170DB2 Content Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173GPFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174LoadLeveler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176Tivoli Access Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177Tivoli Enterprise Console®. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179Tivoli Intelligent Orchestrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180Tivoli License Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181Tivoli Management Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183Tivoli Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184Tivoli NetView. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186Tivoli Provisioning Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

vi Grid Computing Products and Services

Tivoli SAN FS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189Tivoli Service Level Advisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190Tivoli Workload Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192TotalStorage SAN Volume Controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Virtualization Engine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195WebSphere Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196WebSphere Extended Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198WebSphere Information Integrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199WebSphere MQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201WebSphere Portal Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

Contents vii

viii Grid Computing Products and Services

Figures

1-1 Grid stack products’ portfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131-2 Product evaluation criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171-3 WebSphere Information Integrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182-1 OGSA architecture model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262-2 Web Services based Grid Application Model . . . . . . . . . . . . . . . . . . . . . 272-3 Globus-based Grid Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313-1 Focus of scheduling environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433-2 LoadLeveler components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483-3 GridServer architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493-4 Simplistic view of Condor scheduling. . . . . . . . . . . . . . . . . . . . . . . . . . . 573-5 Proposed Desktop Grid setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603-6 Current environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613-7 Proposed Architecture for “xyz” Corporation . . . . . . . . . . . . . . . . . . . . . 634-1 Virtualization layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664-2 Block data virtualization layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674-3 File layer virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684-4 Information virtualization layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684-5 Example for a product overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704-6 Block data layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714-7 File system layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754-8 Information layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784-9 “xyz” Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824-10 Federated using WebSphere Information Integration . . . . . . . . . . . . . . 845-1 A typical Internet banking infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . 895-2 A bank VPN infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906-1 Groups of virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068-1 Grid computing services roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1248-2 Analytics Acceleration architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 1328-3 Engineering Design architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1348-4 Geophysical Processing and Analysis architecture . . . . . . . . . . . . . . . 1368-5 Research Collaboration architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 138B-1 World Community Grid - Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153B-2 World Community Grid: High Level Solution Design . . . . . . . . . . . . . . 155B-3 World Community Grid: Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 158B-4 World Community Grid: Architecture with products . . . . . . . . . . . . . . . 161B-5 World Community Grid Agent Interface . . . . . . . . . . . . . . . . . . . . . . . . 162B-6 World Community Grid Screen Saver . . . . . . . . . . . . . . . . . . . . . . . . . 163B-7 World Community Grid Web Site - Homepage . . . . . . . . . . . . . . . . . . 164

© Copyright IBM Corp. 2005. All rights reserved. ix

B-8 World Community Grid Web Site - Project Statistics . . . . . . . . . . . . . . 165B-9 World Community Grid: Member Area. . . . . . . . . . . . . . . . . . . . . . . . . 166C-1 DB2 Content Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173C-2 GPFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175C-3 LoadLeveler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176C-4 Tivoli Access Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178C-5 Tivoli Enterprise Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179C-6 Tivoli Intelligent Orchestrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180C-7 Tivoli License Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182C-8 Tivoli Management Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183C-9 Tivoli Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185C-10 Tivoli NetView . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186C-11 Tivoli Provisioning Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188C-12 Tivoli SAN FS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189C-13 Tivoli Service Level Advisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191C-14 Tivoli Workload Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192C-15 TotalStorage San Volume Controller . . . . . . . . . . . . . . . . . . . . . . . . . . 194C-16 Virtualization Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195C-17 WebSphere Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197C-18 WebSphere Extended Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 198C-19 WebSphere Information Integrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . 200C-20 WebSphere MQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201C-21 WebSphere Portal Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

x Grid Computing Products and Services

Tables

2-1 Comparing WBI-MQ and WAS-MR . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398-1 Grid computing services overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1238-2 Grid solutions overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1298-3 Offering availability by industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140C-1 Web Services Core/Hosting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171C-2 Workload virtualization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171C-3 Information virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171C-4 Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172C-5 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172C-6 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172C-7 License management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172C-8 DB2 Content Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174C-9 GPFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175C-10 LoadLeveler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177C-11 Tivoli Access Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178C-12 Tivoli Enterprise Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179C-13 Tivoli Intelligent Orchestrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181C-14 Tivoli License Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182C-15 Tivoli Management Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184C-16 Tivoli Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185C-17 Tivoli NetView . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187C-18 Tivoli Provisioning Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188C-19 Tivoli SAN FS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190C-20 Tivoli Service Level Advisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191C-21 Tivoli Workload Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193C-22 TotalStorage San Volume Controller . . . . . . . . . . . . . . . . . . . . . . . . . . 194C-23 Virtualization Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196C-24 WebSphere Application Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197C-25 WebSphere Extended Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . 199C-26 WebSphere Information Integrator. . . . . . . . . . . . . . . . . . . . . . . . . . . . 200C-27 WebSphere MQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202C-28 WebSphere Portal Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

© Copyright IBM Corp. 2005. All rights reserved. xi

xii Grid Computing Products and Services

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

© Copyright IBM Corp. 2005. All rights reserved. xiii

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

AFS®AIX 5L™AIX®CICS®Domino®DB2 Universal Database™DB2®Eserver®Eserver®FlashCopy®HACMP™IBM®i5/OS™

iSeries™LoadLeveler®Lotus Notes®Lotus®MQSeries®NetView®Notes®OmniFind™OS/2®pSeries®QuickPlace®Redbooks™Redbooks (logo) ™

RS/6000®Storage Tank™Tivoli Enterprise™Tivoli Enterprise Console®Tivoli®TotalStorage®TCS®Virtualization Engine™WebSphere®World Community Grid™xSeries®z/OS®zSeries®

The following terms are trademarks of other companies:

EJB, Java, JavaBeans, Jini, JDBC, JSP, J2EE, Solaris, Sun, Sun Microsystems, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Nina, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

xiv Grid Computing Products and Services

Preface

This IBM® Redbook is part of a series of documents related to grid computing that IBM is presenting to the IT community to it and all its players: clients, industry leaders, emerging enterprises, universities, and producers of technology. It is mainly oriented to IT architects or those who have the responsibility of analyzing the capabilities of the products to build a grid solution.

We describe the functions of grid computing and the categorizations of the components within it. IBM, ISVs, and open source products will be explained. Some of these products are not specific for grid, but they can be used in a grid computing environment. We hope that this redbook helps you select functions and products and shows you how grid can fit into your IT picture as new products and services for grid are introduced.

The redbook is organized into the following parts:

Part 1, “Introduction” on page 1Introduces the concept of grid computing and provides the terminology the readers will use in this redbook.

Part 2, “Functionality and products” on page 21Explores the functionality associated with certain products and product families. It also presents some examples of the utilization of these products in the grid computing environment.

Part 3, “Bundles” on page 103Presents grid computing product families that can be used to build a complete grid solution. These bundles, or suites, often implement most of the core grid disciplines.

Part 4, “Services and solutions” on page 119Gives an overview of the grid computing offerings and services that IBM provides.

Part 5, “Appendixes” on page 143Explorers the following additional grid computing related topics:

� Appendix A, “Surrounding initiatives” on page 145� Appendix B, “World Community Grid” on page 149� Appendix C, “Products reference” on page 169

© Copyright IBM Corp. 2005. All rights reserved. xv

The team that wrote this redbookThis redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center.

Luis Ferreira, also known as “Luix”, is a Senior Software Engineer at IBM Americas, working on grid computing architecture projects. He has 20 years of experience with UNIX®-like operating systems in design, architecture and implementation, and holds a Master of Science degree in Systems Engineering from Universidade Federal do Rio de Janeiro in Brazil. Before joining IBM Americas, Luis worked at International Technical Support Organization as a Linux® and Grid Computing Project Leader, Tivoli® Systems as a Certified Tivoli Consultant, at IBM Brazil as a Certified IT Specialist, and at Cobra Computadores as a kernel developer and operating systems designer.

Mariano Batista is a Certified IT Architect. He has been working for IBM for more than 11 years. He is usually called to do pre-sales activities in multi-brand projects that have strong integration and complex technologies. Mariano got his certification as ITA in Middleware and Application Services in December 2002 and was invited to be part of the ITA Certification Board at the same time. In the last six years, he has worked for Sales and Distribution as a Solution IT Architect doing pre-sales activities for multi-industry clients, mainly in Argentina, with coverage on SSA (Spanish South America) and Latin America. He has been part of the on demand pre-sales IT Architects team since 2003 and the e-business Solutions Technical Sales unit since 2001 in the Americas. His current responsibilities include a broad range of IT solutions and technologies; grid computing is one of them. Mariano represents IBM in internal and external conferences, events, and articles published in newspapers and magazines in Argentina related to grid computing. Before becoming a formal IT architect, Mariano was part of the Industry Systems Business Unit for Financial Services Sector in Argentina. He was responsible for important engagements in e-business solutions for Financial Services Sector industries. He was part of the team that implemented the first Interactive Financial Services project in Latin America, and he was the IT Architect and technical leader for the second phase of that project. He started at IBM in 1993 as a C/C++ developer in the Advanced Solutions Unit of IBM Argentina. He also provided AIX® and system management support for that unit. Mariano received his Bachelor of Computer Sciences degree with honors at the Universidad del Salvador in Argentina in 1994.

Sebastien Fibra is a grid specialist at the IBM Advanced Technical Support (ATS) Design Center for on demand business, within the Product and Solutions Support Center (PSSC) in Montpellier, France. He received a M.S. degree from a top-ranked French computing engineering school (ENSEEIHT, Toulouse) in 1996. After his graduation, he worked for Matra, providing consulting and support

xvi Grid Computing Products and Services

for CAD solutions for space applications and aeronautics, spending 16 months in Boston, Massachusetts. Prior to his current position in IBM, he was a senior technical consultant for a Web agency in Paris. He joined IBM in 2003 and is the lead for the deployment and operations of the grid connecting the network of IBM Design Centers worldwide.

Chin Yau Lee works as an Advisory Technical Specialist in grid computing for IBM ASEAN/South Asia. He holds a Honours Degree in Computing and Information System from the University of Staffordshire. He has been using Linux since 1996 and had a few years of experience as a UNIX and Linux engineer before joining IBM. His areas of expertise includes High Performance Linux and UNIX, UNIX Systems Administration, High Availability solutions, Internet based solutions, and grid computing architectures, which he has been actively working on for the last four years. He is also an IBM Certified Advance Technical Expert on AIX, Sun™ Certified System/Network Administrator and Red Hat Certified Engineer. He co-authored the redbook Deploying Linux on IBM pSeries Clusters, SG24-7014.

Carlos Alexandre Queiroz Silva is an independent consultant working for Alex Microsystems. He has been working with grid computing, Jini™, and J2EE™ technologies since 2000. Currently, he is earning a Master's degree at Universidade de São Paulo as a distributed systems and network specialist. He has published articles at several congresses, such as middleware2003, SBRC, grid computing, and parallel applications events. Carlos is an active developer of InteGrade, which can be found at http://gsd.ime.usp.br/integrade.

Joao Almeida is an IT Specialist with IBM Portugal. He is a Red Hat Certified Engineer and has four years of experience in Linux AIX and pSeries®. Working in implementation services and technical support, his areas of expertise include Linux, AIX, HACMP™, WebSphere®, and grid computing.

Fabiano Lucchese is the business director of Sparsi Computing in Grid (http://www.sparsi.com) and works as a grid computing consultor in a number of nation-wide projects. In 1994, Fabiano was admitted to the Computer Engineering undergraduate course of the State University of Campinas, Brazil and in mid-1997, he moved to France to finish his undergraduate studies at the Central School of Lyon. Also in France, he pursued graduate-level studies in Industrial Automation. Back in Brazil, he joined Unisoma Mathematics for Productivity, where he worked as a software engineer on the development of image processing and optimization systems. From 2000 to 2002, he joined the Faculty of Electrical and Computer Engineering of the State University of Campinas as a graduate student and acquired an M.S. degree on Computer Engineering for developing a task scheduling algorithm for balancing processing loads on heterogeneous grids. Fabiano took part in publishing the redbook Grid Services Programming and Application Enablement, SG24-6100.

Preface xvii

http://gsd.ime.usp.br/integrade

http://www.sparsi.com

Nam Keung is a senior technical consultant for IBM in Austin, Texas. He has worked in the area of AIX ISDN communications, AIX SOM/DSOM development, AIX multimedia development, NT clustering technology, and Java™ performance. His current assignment involves helping IBM business partners and solution providers in their efforts to port and deploy applications to the pSeries platform. He also consults in performance tuning and other educational needs for the pSeries platform.

AcknowledgementsThanks to the following people for their contributions:Joanne Luedtke, Lupe Brown, Cheryl Pecchia, Arzu Gucer, Chris Blatchley, Wade Wallace, Ella BuslovichInternational Technical Support Organization, IBM

Tony WhiteWorldwide Grid Computing Technical Sales Business Unit Executive, IBM

Ronald WatkinsWorldwide Grid Computing Business Development Executive, Public Sector, IBM

Matthew P HaynosProgram Director, Grid Computing Strategy, IBM Somers

Matthew FriedmanWorldwide Grid Marketing Executive

David ChisholmManager Grid/VE, Linux, and Digital Media, IBM Americas

Stephen GordonIndustry Solutions Executive, IBM Beaverton

Al HamidExecutive IT Architect and STSM, Grid/OSS Worldwide Leader, BCS

Peter NielsenLinux & Grid EBO, IBM Global Services

Chris Reech, Viktors Berstis, Richard Strysniewicz, Jeff Mausolf, Dennis SpexetIBM Global Services / e-Technology Center, Grid Computing Initiative

Nina® WilnerGrid Computing - IT Technical Architect LifeSciences, IBM Austin

Bill BovermannWorld Community Grid™ Executive, BCS Seattle

xviii Grid Computing Products and Services

Lee B WilsonTechnical Sales Specialist, IBM Dallas

Susan MalaikaSTSM (Web, XML, Grid, DB2®, Data), IBM Silicon Valley Laboratory

Allen LuniewskiInformation Management On Demand Strategy, IBM Almaden

Chris DawsonIT Architect, Grid Community of Practice Lead, IBM Bethesda

Alexei ChirokikhConsulting IT Architect, IBM Rochester

Robert VrablikGrid Computing Strategy and Planning, IBM Poughkeepsie

Barbara DeLiberoWorldwide Sales Executive - Grid, IBM Waltham

Dr. Alexei ChirokikhPublic Sector Business Consulting Services IBM

Judith AdolinoWW Linux and Grid Services EBO, IBM Global Services Boca Raton

Joe ZhouGrid Computing, XML and Relational conversion, IBM Southbury

Dikran S MeliksetianSTSM - High Performance On Demand Solutions, IBM Southbury

Takanori SekiDistinguished Engineer - Grid Computing Business, Asia Pacific, IBM Japan

Tomoari YasudaGrid Specialist, IBM Japan

Yann GuerinEMEA Grid Computing TSM, IBM France

Herve SabrieManager, EMEA Design Center & Solution Centers, IBM France

Ruth HaradaAlliances Manager, IBM Brazil

Preface xix

Katia PessanhaUniversities Alliances Manager, IBM Brazil

Luiz Roberto RochaGrid Specialist, IBM Brazil

Srikrishnan SundararajanIBM India Software Labs

Clive HarrisSenior Architect, IBM UK

Dr. Manfred OeversLife Science Solutions Development and Grid Computing, IBM UK

Cliff VarsInformation Integration Technology Solutions, IBM Austin

Ellen StokesSTSM, Grid Computing, IBM Austin

Pete HenryClient Executive, ISV & Developer Relations IBM

Dr. Carl KesselmanUniversity of Southern California/Information Sciences Institute

Antonio Saverio Rincon MungioliEscola Politécnica of Universidade de São Paulo, Brazil

Elton MinettoUniversidade Comunitária Regional de Chapecó, Brazil

Edward Walker, Ph.D.Texas Advanced Computing Center, University of Texas at Austin

Karen MillerUniversity of Wisconsin - Madison

Michael HumphreyAltair Engineering, Inc.

Wayne Slater, David Smith, Marnie BilesPlatform Computing Inc.

Mary Bass, Piush PatelUnited Devices Inc.

xx Grid Computing Products and Services

Tony Bishop, Jennifer Smith, Pat AughavinDataSynapse Inc.

Become a published authorJoin us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners, and clients.

Your efforts will help increase product acceptance and client satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcomeYour comments are important to us!

We want our Redbooks™ to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways:

� Use the online Contact us review redbook form found at:

ibm.com/redbooks

� Send your comments in an e-mail to:

[email protected]

� Mail your comments to:

IBM Corporation, International Technical Support OrganizationDept. JN9B Building 003 Internal Zip 283411400 Burnet RoadAustin, Texas 78758-3493

Preface xxi

http://www.redbooks.ibm.com/residencies.html

http://www.redbooks.ibm.com/residencies.html



http://www.redbooks.ibm.com/contacts.html

xxii Grid Computing Products and Services

Part 1 Introduction

Part 1

© Copyright IBM Corp. 2005. All rights reserved. 1

2 Grid Computing Products and Services

Chapter 1. Introduction

This chapter discusses the following:

� Concepts of grid computing

� Terminology

� Classification of the grid disciplines that will be the basis of the analysis of the products and scenarios

1


1.1 IntroductionGrid computing has been evolving since its formal definition in the early 90's. At that time, grid computing was a concept and started to have incipient implementations in universities. Today, it is a reality that can be used not only in academic or scientific environments, but in commercial bids to provide leading edge solutions to clients. As stated in the Introduction to Grid Computing with Globus, SG24-6895, “grid computing is distributed computing taken to the next evolutionary level”.

Several things have happened from the time of initial concept and academic implementation. Over the following years, grid was used in laboratories and gained maturity as the concept became a model. That model started a new way of thinking in the IT community and helped solve requirements for end users and the consumers of the IT industry. Definitions like server grid, desktop grid, data/information grid, and others appeared.

The grid paradigm inspired several technical people outside the places where the concept was born. It was integrated into products of both incipient and well established IT enterprises. The paradigm opened the way to real pieces of technology that could be used for commercial purposes.

With grid computing, we can observe a very interesting evolution: What started in papers and in projects led by universities became a conceptual architecture that can be used with commercial targets to enrich a user’s experience and to help with the requirements of the enterprises that consume IT.

1.1.1 Grid computing as an evolution of distributed computingGrid computing is an evolution of distributed computing that allows you, through open standards, to see independent and physically scattered computing resources as though they were a unique large virtual computer. This is a simple definition of grid computing, but good enough for explaining grid in a few words to a new comer, or to someone who does not know very much about IT.

In a more formal and complete definition, grid computing can be seen as a distributed computing model that supports the concept of virtual dynamic organizations by providing secure and coordinated access and sharing of heterogeneous and geographically distributed resources, such as applications, data, processor power, network bandwidth, storage capacity and others, over a network and across organizational boundaries, using a set of open standards and protocols. Grid users see these resources as a large virtual computer.


If we analyze this definition deeper, we will find several important elements:

Distributed computing model...Grid is based on distributed computing. It means that the infrastructure is not centralized, because its resources are geographically distributed. This implies some kind of communication between them.

...over a network...The network is the physical structure that interconnects the distributed resources. A network generally only provides the pipes to connect some component with other components. However, in order to create comprehensive and interesting communications, it is necessary to adopt some protocols and open standards.

...using open standards and protocols...Open standards and protocols provide the mechanisms needed to create communications between components developed by different vendors. They also allow the developers to concentrate on the business logic rather than on the method of programming communication routines.

...using shared resources...A distributed computing infrastructure provides the ability to share resources. Initially, we see resources as CPU, disks, and memory. However, resources can also involve other elements, such as networks, databases, storage devices, and so on. Sharing resources through a network allows you to have a homogeneous system composed of heterogeneous resources. A request can use a resource that is outside of the organization to which it belongs. It brings forth the concept of "virtual dynamic organizations".

... inside a virtual organization...An example of a virtual organization is a department within an university or an university in a community. When a user requires a service from the grid, it can get the solicited resource from a second party that belongs to another organization. For a certain time, these two parties will belong to the same organization, a virtual organization intended to provide a resource to the requester. A virtual organization needs to be dynamic, because if the requester needs more resources than the provider can provide, then the grid should search for more resources, expanding that organization to deliver what the requester is asking. This dynamism applies as well when the requester asks again for the same kind of resources and the grid provides what is available at that moment.

Chapter 1. Introduction 5

...with secure access...The implementation of virtual dynamic organizations requires secure mechanisms to ensure protection of all shared elements. If a user requires CPU time to run a certain program, it should not jeopardize any grid component.

Security also implies a mechanism to guarantee the identity of the actors, meaning authentication of the consumers and providers. Security is also related to authorization, which means that as soon as a consumer or provider has been authenticated, it will use only the resources for which it has proper rights. There are additional implications related to security, like keeping up-to-date the list of users and resources, or the Access Control Lists (ACLs).

The integrity of the information is needed to ensure that the information that is transferred in the network is not modified after delivery. Security has to enforce privacy as well, to make sure that only those who have the right to see the information will be allowed to.

... and coordinated access...The notion of coordination brings forth the idea of a scheduler for jobs, processes, or higher level units of processing like tasks. Implementing mechanisms to provide quality of service (QoS) comes with coordination. Although QoS is not an original quality of grid, it is obvious that if a grid can ensure QoS, it will become more effective in fulfilling business objectives. For example, it is possible to implement a grid with regular users and premium users. These premium users, who are willing to benefit from a higher QoS, can be billed for the exclusive services provided by the grid, which guarantees them that their jobs will be scheduled on time by increasing their priority. These are only a few qualities needed to provide an acceptable QoS.

Coordinated access is required in many ways. If someone submits a job to the grid that can be parallelized, it might be desirable to have the grid split the job into a number of subjobs, process these smaller chunks of computation, and return a unique consolidated result to the requester. On the other hand, if the requester prefers to receive the results as they are coming, the grid should allow it. Both examples require coordination features for a single actor. But grids must provide coordination to all the requesters, producers, consumers, and donors.

Grids must also implement policies to satisfy the requirements of the consumers and to use the capabilities of the donors without exceeding the levels that they stated. Coordination requires the implementation of policies, the registration of the capabilities of the donors, some metering abilities, and some intelligence to optimize the usage of the whole grid. It should plan for potential cases, such as a machine turned off during non-working hours or days. If the grid has premium users, it has to give a higher priority to their requests


1.1.2 Grid computing modelsIt is possible to identify models of grids based on the type of service that their donors give and that their requesters demand.

There are also some fundamental grid models based on the type of basic services provided. Resources can basically be computing power, provided by servers or individual computers, data storage capacity, provided by information and data repositories, or network bandwidth, provided by networked infrastructures. Of course, a grid can be conceived to handle several types of resources. It is also possible to implement higher, value-add services based on one or more of these models. These grids expand the principles of the basic models, leveraging and combining them in innovative way to provide more advanced services.

Computational gridA computational grid is an infrastructure that allows resources to donate computing power to the grid whenever the workload demands. This infrastructure is suitable for applications that demand:

� As much processing power as possible or additional processing power during certain periods of time and a single machine cannot provide it or at too much cost.

� It is usually associated with resource scavenging in desktops machines and underused servers.

Such a grid takes advantage of the idle resources in the virtual organization. Just think about the cycles of CPU that are unused when a typical user browses on the Internet, reads an e-mail, or creates office documents like presentations or word processing files. Additionally, it would allow to reuse the infrastructure and the client application for more than one project at the same time.

Examples of server-oriented gridsLet us consider a person who uses complex spreadsheets to execute heavy financial calculations. Sometimes this kind of spreadsheet needs to iterate as many times as possible to get closer to the desired result. The more iterations, the more accurate the results of the execution will be. That process is often applied to carry out profitability analysis.

Profitability analysis and stocks evaluations for shareholders, or loan approvals, can be carried out with a server grid as well. Usually, these operations are executed in the customer assistant’s desktop machines and it is possible to improve the response time to the client by using a server-oriented grid to run this kind of application. The users who are in contact with the client could submit a


job to the grid, it would use idle resources to fulfill the job, and it would return results faster than if the user had used only his own computer.

Another interesting application area is the petroleum industry. Geologists gather information in oil fields, which are almost always away from the data centers. Therefore, these users have very powerful workstations with high level graphics and processing capabilities, since they need to analyze the composition, geology, and streams in the reservoir locally. This reservoir study, which encompasses seismic and geology analysis, requires high processing power. A fast response from the system is important to the geologists and engineers on the field, who have to make important real-time decisions for the oil company. Despite the great high-end workstations that exist today, more processing power is sometimes needed to be more effective on the field. A server grid provides the ability to submit jobs from the field that are executed in the main, regional, or local datacenter. In that way, the activity of the professionals in the oil field can be improved and the time to take decisions that impact on business is reduced.

Examples of desktop-oriented gridsThis model is used in World Community Grid, and it could be implemented with commercial objectives within an enterprise as well. An enterprise could implement a desktop grid to free some server resources. It might be possible to identify some activities that usually run on servers and could be run on other computers to free the servers from these activities. It would allow the servers to have more processing power during the peak activities that can be run only on this server.

In this kind of grid, it is more complex to guarantee the time needed to get a result. For this reason, it is primarily intended for applications that can afford to deliver results with no critical time commitment. Although it is called a desktop grid, some distinct coordination points, typically deployed on server machines, play an important role in this solution; scheduling, directory services, dispatching, disassembly, and assembly mechanisms should be provided. Another important characteristic of this grid that could be very appreciated by its adopters is the provision of a toolkit or framework to create applications.

Note: World Community Grid's mission is to create the largest public computing grid benefiting humanity. More information about World Community Grid can be found in Appendix B, “World Community Grid” on page 149 and at the following Web site:

http://www.worldcommunitygrid.org/



Data gridWithin a data grid infrastructure are the components used to provide grid capabilities to the data and information virtualization disciplines. It provides the ability to supply homogeneous access to heterogeneous repositories of data. It allows the data consumers to see an unified image of the respective information or data spread across different resources, potentially based on different technologies.

Information InfrastructureInformation is usually defined as "meaningful data". Meaningful data is often associated with the unit of information that means something useful to the end user. The vision of different repositories of information as though they were a single one illustrates the idea of information integration. Information grid is intended to integrate different sources of information in a comprehensive way. It allows applications and users to see a single database that hides the complexity of accessing multiple databases. The information spread across them is published by the grid as though it were really centralized.

This grid should implement connectors to the final databases, which surely are heterogeneous and geographically distributed. It is desirable as well to have replication and caching mechanisms that make the management and usage of the IT infrastructure more efficient.

File system and block data infrastructureFile system and block data infrastructure for grid follows the main principle of the information infrastructure but at a lower level. While the information infrastructure provides unified access to information, the file system and block data infrastructure provides unified access to files or chunks of data.

Data is not always meaningful to the end user. However, it is the unit of work of operating systems and hardware devices. Data in this context means blocks of data or files.

SAN-FS is one of these technologies. It implements mechanisms to see storage components from different vendors as a single disk in a Storage Area Network. SAN-FS is an example of a basic data grid implemented at the hardware level.

Network gridIn a typical corporate network, computers are very often permanently connected to it while using only a portion of its bandwidth. Every machine, servers and desktops, has underused network bandwidth, which can be considered as an idle resource. When a given user or machine requires more resources from the network, a bottleneck is reached.


ExamplesIn a file transfer using FTP, the server side of it allows no more than two or three simultaneous connections from the same requester. The traffic may be limited by the server itself just to avoid the situation where a given user gets a great portion of the server’s bandwidth. This is necessary because there could be other requesters asking the same server for its resources. The server has to implement some way to ensure a balanced competition among their requesters. This is fair from the server point of view, but from the requester standpoint, things may be seen a bit differently. If the desktop tries to download a file from a FTP server that limits the bandwidth just in case there are other requesters competing for the same resource, some idle network resources will appear on the requester side. Imagine a desktop connected through a 100 Mbps connection to a server that perhaps provides one or more 1 Gbps connections. If the limitation is set up to 1 Mbps per connection, the desktop will have theoretically 99 Mbps free, ready to use. The server has to leave some free network resources to satisfy other users’ requirements. But the desktop could use its free resources. We know that there could be additional factors to consider, like WAN bandwidth limitations, routers, and so on. But, just to keep it simple, we are showing this example in a LAN environment, or in an environment without other physical restrictions than the bandwidth of the nodes.

In this example, the desktop could connect to other servers in the same infrastructure to simultaneously download another portion of the file. This way, the first server would not modify its policies to attend users’ requests, the second server could provide more data to the desktop, and the desktop would more effectively use its own network interface.

This implementation requires some intelligence in the desktop side, perhaps a fancier file transfer protocol or a smarter FTP client.

From the server side, this requires the duplication of the pool of files on each server. Perhaps the servers are geographically distributed, in a WAN environment. In this case, it is obvious that storage is cheaper than network. Therefore, it is possible to have the files downloaded from several servers.

This type of grid is sometimes called “distributed peer to peer” grid, network grid communication grid, or “grid delivery”. We can see some resemblance to the well known peer-to-peer (P2P) programs. However, a network grid has some more efficient mechanisms to provide services. For example, it can provide a centralized directory of files, or the directory itself can be part of the client side.

IBM has implemented an internal network grid called IBM downloadGrid with services similar to the one described above, which allowed IBM to reduce the transatlantic network traffic.


In order to implement a well balanced network grid, it is necessary to analyze the whole infrastructure, the network topology, the behavior of the users, and the cost of the most critical connections, as well as the way the contents are used. It is recommended that the CPU that performs strong processing activities is as close as possible to the repository of contents.

Multipurpose gridThe multipurpose grid is perhaps the more common implementation in the future of grid computing. The infrastructure of this grid should be adaptive enough to provide any of the grid models. It could be implemented as well as a meta-grid with abilities to route the requests to the grid that supports the right model to fulfill them.

1.1.3 Grid and its relationship with other IT componentsThis section discusses the relationship between grid computing and other components of IT. Some of those components are technologies, while others are concepts or paradigms. It is important to see that grid is related to all of them in some way. As we will see, none of the following items is exclusively grid. In most of the cases, grid leverages them and is leveraged by them.

Grid and deep computingDeep computing, as well known as high performance computing (HPC), is a technology that uses clusters of machines to execute high demanding CPU tasks. A deep computing cluster is generally implemented with machines connected through high speed networks. In these cases, the job that the cluster runs can be split and parallelized into small pieces that run in the cluster’s nodes.

A deep computer infrastructure can be part of a grid, and a grid can be the mechanism to provide deep computing resources. Grid provides something that a typical deep computing cluster does not: access to heterogeneous resources.

Grid and on demandOn demand business is IBM’s vision of the IT industry and market. It acknowledges that the flexibility of the markets will demand flexibility in enterprises’ business processes. Flexible, adaptable, and resilient business processes require a flexible IT operating environment. The operating environment has to be virtualized; it should be based on open standards to be

Note: More information about the IBM downloadGrid can be found at the following Web site:

http://www.research.ibm.com/journal/sj/434/meliksetian.html



integrable and must have autonomic capabilities. Grid computing leverages on demand because it is a fundamental component to achieve the highest degree of virtualization, one of the key factors of the on demand operating environment. Grid is also leveraged when its implementation complies with open standards; it is integrated into the business processes of the organization, and its components have autonomic abilities.

Grid and UMIIn an on demand enterprise, once the business and the IT infrastructure are flexible, it is possible to think of new ways to make the IT services and other enterprise services profitable. UMI (Utility Management Infrastructure) is a model or specification of how the IT services can be acquired and paid as though they were a service provided by a traditional utility. Utilities use the “pay per usage” payment method. UMI conceptually establishes what components a utility must have to provide IT services and their consumers pay per usage. It means variable costs for the clients. It also means, for the IT utility provider, that the infrastructure can be reused and shared. Grid computing capabilities are a reasonable option to implement an IT utility, but not the only choice. Grid provides provisioning, metering, billing, and mechanisms to use idle resources, sharing of portions of resources, and security and management. All of them are required qualities to implement an infrastructure aligned with UMI.

Grid and virtualizationVirtualization is the ability to provide a unified vision of a set of resources. These resources may be geographically distributed, run on different technologies, and developed by different vendors. Virtualization is one of the key elements of the on demand operating environment, and essential on grid implementations. More information about on demand operating environment can be found at:


Grid and provisioningAs explained in “Provisioning” on page 86, provisioning mechanisms can provide on demand additional resources to the grid. Alternately, the grid can provide resources as well. Both can live independently, coexist in the same infrastructure, and be leveraged if they are well integrated.

1.2 Categories and functionsGrid computing is a concept that can be studied in several ways. The information contained in this redbook has been classified and analyzed according to cross criteria, categories, and functions.



Categories are related to the layers in a grid stack products’ portfolio, as shown in Figure 1-1. Categories help the reader relate products with their role and scope within a grid. IT Architects will find the categories to be a technical reference to help them find the right products for their solutions. Categories will also be useful to understand the scope of the professional services that should be implemented to satisfy the requirements of a solution. IT architects must consider business, functional, and non-functional requirements that should be fulfilled by the infrastructure and the services that a product and a category of the grid layer can provide.

1.2.1 Categorization of componentsThe layer structure, or grid stack (Figure 1-1), of the grid components is an creation of the traditional IT industry.

Figure 1-1 Grid stack products’ portfolio

ApplicationThis layer describes an environment for developed applications to take advantage of the whole set of features provided by the grid. Applications can make use of all layers in the stack, through classes, APIs, frameworks, toolkits, or Software Development Kit (SDKs).

Grid Middleware

File and Block Data Virtualization

Structured Data Virtualization

Job Scheduling

Software licensing

Systems Management

Workload Management

Task Scheduling

Orchestration and Provisioning

Billing and Metering

Content Management

Applications

Grid Middleware

File and Block Data Virtualization

Structured Data Virtualization

Job Scheduling

Software licensing

Systems Management

Workload Management

Task Scheduling

Orchestration and Provisioning

Billing and Metering

Content Management

Applications


The application layer is also associated with the applications developed for a specific industry.

Content managementContent management provides another degree of virtualization to data and information within a grid. It is related to the handling of digital media, like video, audio, images, or streams. These are not structured data, and need to be treated in a different way. It usually requires specific features, such as searching an image database for the most similar picture to the one given in input.

Billing and meteringMetering is a desirable service in a grid. It helps quantify the efficiency and performance of the grid, and the usage of its resources. It is the base of billing activities, which affect a price to the resource consumption units delivered by the grid. Provisioning products have some basic capabilities of metering and billing to register the events related to the resource’s provision.

Orchestration and provisioningProvisioning is the ability to transform a basic inactive resource into an mobilized resource for applications, by installing and configuring all the required software in an automated way. Orchestration is designed to define and optimize the concurrent provisioning activities in an infrastructure, including the return to the initial state of the resources once their utilization has been ended.

Task schedulingTask scheduling provides an environment to run small pieces of execution that, combined, create the unit of work that the requester needs to execute. The decomposition of this unit of work into smaller pieces can consist of parallel, short running tasks or of tasks that have to be run in an special order within an automated workflow.

Workload managementWorkload management is a mechanism designed to balance the workload among different resources. It is defined during the setup and used at run time. The workload management can be dynamic depending on the real time workload of the involved components and the policies applied to them. This layer may be associated with a Service Level Agreement (SLA). In fact, if a SLA exists, the workload management layer should be aware of it in order to provide the right power to the right resource at the right time.


System managementManagement is vital in every IT infrastructure. Distributed computing environments require even more sophisticated tools and policies.

Several of the activities behind the scenes are related to management and administration. Almost all layers in the grid stack have a strong relationship with management. The middleware has to be configured and enabled. Security has to be set up and security logs have to be sent to the system management. Block, file, and structured data virtualization have to be configured and adapted to optimize their utilization. License and Workload management are obvious management activities. Provisioning, orchestration, billing, metering, and the rest of the layers in the grid stack have to be configured and managed as well. The Tivoli Suite is an example of management infrastructure. It is intended for a traditional IT environment, but can clearly be integrated into a grid.

Software licensingLicense management is a technology that allow you to control the licenses that are in use for a certain period of time. It provides elements to determine which machine is the most suitable candidate for a task, based on the software installed on it.

As grid is based on a dynamic infrastructure, it may require that certain software products run for a limited time, then stay on standby for a certain period.

Job schedulingJob scheduling is used to optimize the execution of jobs onto a grid. In this context, a job can be understood as the computational work needed to deliver a meaningful result within the context of an application. Its execution is governed by policies that rule the basic scheduling and prioritizing of jobs to be run.

Structured data virtualizationStructured data virtualization provides a unified access to heterogeneous databases as though they were a large and single database. It supplies grid-based access to structured data sources of information, like databases, documental databases, XML, or flat files with certain structure and objects.

This is a higher level of data virtualization. A grid can create an unified vision of the different repositories of information. The consumers of this service should see a single database that in fact is composed of aggregations and associations of other databases.


File and block data virtualization layerFile and block data virtualization mechanisms are related to the lowest layers of data services provided by the grid. They can live at the hardware level, associated with blocks of data managed by storage devices.

In the block data virtualization case, the actor is the hardware itself and its drivers. The components create the unified vision at the operating system level, which, in turn, will create a file system on top of the block data grid. File data virtualization is provided by file system implementations.

Grid middlewareThe grid middleware provides the integration among the grid components. It is the keystone software that provides basic services to grid components. These basic services are integration mechanisms based on standards, description, and creation of services. The middleware also includes the elements that allow the grid to interact with its own components.

What about grid security?Security is a service that is present across the whole grid stack. This explains why it does not appear in the Figure 1-1 on page 13. Its mission is to guarantee the integrity of the grid, of its resources, and of its users interactions, by preventing unauthorized access. Security provides authentication, authorization, integrity, and privacy to the operations carried out by the grid.

1.2.2 Functionality on grid computingFunctions group high level product roles to better understand the details and the capabilities of the products. A classification per function, also known as disciplines, will be used throughout all chapters of this redbook. They are the attributes an IT Architect will look for to understand the main features a product brings to the grid. This classification, as shown in Figure 1-2 on page 17, presents four elements:

� Web Services Core/Hosting� Workload virtualization� Information virtualization � Other Disciplines

Examples of other disciplines would be security, provisioning, billing, and metering.


Figure 1-2 Product evaluation criteria

Products charts based on their disciplinesProducts covered in this redbook support the functions described above to a certain point. The width of each bar (0 to 3) on Figure 1-2 reflects our perception of the product in a grid computing environment, as follows:

� 0: The product has no features of this discipline.

� 1: The product has some features of this discipline.

� 2: The product has most of the features of this discipline.

� 3: The product was developed and designed to completely cover the discipline.

As an example, you will find in Figure 1-3 on page 18 the product chart for the WebSphere Information Integrator, which illustrates our understanding of the support it provides for the functions specified in this redbook. The chart is to be read as follows:

� Information virtualization

3: The product was developed and designed to completely cover the discipline. WebSphere Information Integrator is a data federation software that provides structured access to heterogeneous types of data.

Other Disciplines

InformationVirtualization

WorkloadVirtualization

Web ServicesCore/Hosting

Functionality on Grid Computing


Figure 1-3 WebSphere Information Integrator

Product charts of most of the products described in this redbook are in Appendix C, “Products reference” on page 169.

Web services core/hostingThis is the application environment, the arena where applications live, provide services to requesters, and consume services from the grid. The environment is intended to facilitate the life cycle of the grid-enabled applications, in other words, how they are developed, run, and maintained.

Examples of the capabilities that an application environment can provide are a run-time environment for the application itself, monitoring tools, security features, and APIs to access the lower layers of the grid stack (toolkits). These toolkits should provide a common set of utilities to improve the development time experience. For example, a toolkit could provide a set of tools to build applications that can be parallelized in an easy way, either to be further deployed on a desktop grid or on a server grid, or high-level interfaces to handle grid resources and submit jobs to them. Other tools include components to split the unit of works and consolidate the results.

Other Disciplines






For more information about Web Services Core/Hosting, see Chapter 2, “Web services core/hosting” on page 23.

Workload virtualizationScheduling, the main matter of workload virtualization, is the ability to execute a unit of work at the most favorable moment under certain policies. Scheduling can be defined at a single machine level (within a grid) and at grid level (over several computation pools); this is what meta-scheduling is about. The unit of work and terminology are defined according to the type of application environment. It is normally possible to schedule jobs, tasks, or services. Just to simplify matters, we can generically consider it as an “activity”.

A scheduler’s main role is to ensure that some activity will be carried out over a certain period of time. The scheduler has to consider the workload of the resources that are candidates to run some activity. It would not make sense to submit a consequent job to a machine that is already heavily loaded, while other machines just sit in the grid with idle resources. If the machine accepts that workload, it should delay the execution of its current and new activities with a lower priority.

For more information about Workload virtualization, see Chapter 3, “Workload virtualization” on page 41.

Information virtualizationInformation virtualization encompasses the different types of data infrastructure for grid and the different types of products available to implement them, in order to enable the unified vision of different repositories of data. The virtualization can be done at the block data, file, or information levels.

The block data virtualization provides the unified access directly from hardware or from a driver of a given hardware that runs on a given operating system. The consumer of this data is usually the operating system itself, and the unit of work is a raw block of data in a given storage device. It allows operating systems to see physical resources that are not in the same device as a single logical volume or file system. Then the operating system can emulate a regular file system on top of the services provided by the lower layers. These components create a logical low level structure on top of the physical structure of the storage and publish it to the operating systems.

The file virtualization level is associated with file systems. In this case, the unit of work is a file. A file can be meaningful by itself, like a spreadsheet or a word processing document; however, applications usually need a combination of files to map the data into meaningful information.


The information layer is the highest level of the information virtualization discipline. It is also known as the structured data layer and it provides the ability to integrate information from heterogeneous resources with heterogeneous formats by accessing them using heterogeneous protocols and publishing them to the applications in a homogeneous way, thereby enabling the applications, or consumers of this information, to have a unified vision of the information repository.

For more information about information virtualization, see Chapter 4, “Information virtualization” on page 65.

Other disciplinesExamples of other disciplines are security, provisioning, billing, and metering. Explanations about these disciplines can be found in Chapter 5, “Other disciplines” on page 85.

1.2.3 Grid computing products and playersIBM Business Partners — resellers and distributors, consultants and integrators, and developers — all play critical roles in bringing the benefits of grid computing to both large enterprise and mid-market clients, across a variety of industries.

The products presented in this redbook are the main products suitable for a grid computing environment to date. We will not discuss the displayed products as grid products, or grid-enabled products, but as products with useful and strong capabilities that are qualified to be part of a grid computing solution. Some of these products are IBM products, but some of them are also developed by Independent Software Vendor (ISVs) or Open Source communities. These ISVs, also presented in Chapter 7, “ISV suites” on page 113, can be integrated through IBM professional services as well.

IBM has established key business relationships with leading middleware ISVs to provide clients with the most robust grid solutions in the industry. Each plays an important role in IBM Grid Computing solutions. More information can be found at the following Web site:

http://www.ibm.com/grid/grid_partners/index.shtml



Part 2 Functionality and products

Part 2



Chapter 2. Web services core/hosting


� Grid application development toolkits

� Grid application hosting environment products

� Grid standards, as elaborated by the Grid community, and how Grid services are merging with Web services

2


2.1 IntroductionThis chapter discusses products and technologies that can be used to support the execution of grid applications, or constitute a hosting environment for these applications.

The Grid Standards and Toolkits section focuses on low-to-middle level software toolkits that support the building of grids, but are not meant to be usable right “out of the box”. Their purpose is to make grid oriented programming easier by providing developers with APIs they can call from within their own code, in order to implement grid features.

The Grid Application Environments section aims at describing what products are eligible for grid application deployment, according to grid standards, such as Open Grid Services Architecture (OGSA).

2.2 Grid standards and toolkitsA development toolkit, also known as a software developer's kit (SDK), is a set of programs used by a computer programmer to write application programs. Typically, an SDK can include a visual screen builder, a code editor, a compiler, a linker, and some other facilities.

A grid application development toolkit focuses on providing fundamental enabling technology, for grid users to securely share computing power, data, and other resources across corporate, institutional, and geographic boundaries, without sacrificing local autonomy. Underlying features to achieve these tasks include information infrastructure, dynamic resource discovery and allocation, data management, communication, fault detection, security, and portability.

Most grid application development toolkits already abide by grid standards, as defined by the Global Grid Forum (GGF) and main contributors, such as the Globus Alliance, OASIS, and IBM. The latest contributions, the Web Services - Resource Framework and Notification (WSRF and WS-Notification, respectively) are currently being reviewed by the Oasis consortium for standardization.

2.2.1 Grid standardsGrid computing uses open standards and protocols to enable the virtualization of distributed computing resources to create a single system image across heterogeneous, geographically dispersed IT environments. In order to guarantee grid components’ and applications’ portability and interoperability, a number of


rules and open standards, which new grid developments are expected to comply with, have been published and refined.

To better understand how OGSA and WSRF combine to become a grid standard, one has to consider what steps are required to design a technology:

1. Defining the technology: What is the technology all about? What need is it expected to fulfill? What are its limitations? What is needed to build this technology?

2. Specifying the technology: Specifications for the components, definition of properties and attributes for the objects, and detailed description of the communication protocols and interfaces involved.

3. Implementation: A reference implementation to support the specifications and to demonstrate both the feasibility and the relevance of the technology.

As far as grid technologies are concerned, OGSA addresses the first issue (definition), WSRF addresses the second one (specifications), and the Globus Toolkit 4.0, for example, is a reference implementation of WSRF.

Open Grid Services Architecture (OGSA)OGSA was designed as an attempt to standardize interfaces and behaviors for distributed systems by:

1. Refactoring the Globus protocol suite to enable a common base and expose key capabilities.

This was achieved by generalizing and redesigning protocols and mechanisms already identified as key for grid technologies, such as authentication, policy and lifetime management, notification, service naming, and reliable invocation.

2. Extending new technical requirements, such as concurrent access to resources and management tools.

3. Introducing the notion of “Grid Service”.

The goal here was to define grid entities by their interface and behavior, to unify access to programs and resources, and make it easier for grid architects to achieve virtualization.

4. Embracing key Web services technologies to adopt a standard Interface Definition Language.

The best way to look at OGSA is to consider it as a layered model, just like the OSI reference model (see Figure 2-1 on page 26).

Chapter 2. Web services core/hosting 25

Figure 2-1 OGSA architecture model

Web Services: Resource Framework (WSRF)“The purpose of the Web Services Resource Framework (WSRF) TC is to define a generic and open framework for modeling and accessing stateful resources using Web services. This includes mechanisms to describe views on the state, to support management of the state through properties associated with the Web service, and to describe how these mechanisms are extensible to groups of Web services.” -- OASIS Web Services Resource Framework Technical Committee.

Note: WSRF aims to solve the concerns that the Grid community has expressed about former OGSA specifications, namely OGSI. Its main purpose is to achieve the convergence of Grid services and Web services. It was announced by the Globus Alliance and IBM in conjunction with HP on January 2004. In March 2004, the WSRF specifications was submitted to OASIS.

More information can be found at:

http://www.globus.org/wsrf/http://www.oasis-open.org/home/index.php

Network

OGSA Enabled

Storage

OGSA Enabled

Servers

OGSA Enabled

MessagingOGSA Enabled

DirectoryOGSA Enabled

File SystemsOGSA Enabled

DatabaseOGSA Enabled

WorkflowOGSA Enabled

SecurityOGSA Enabled OGSA Enabled









SecurityOGSA Enabled


OGSA Architected Grid Services

Applications

Web Services-Resource Framework

Network

OGSA Enabled

Storage

OGSA Enabled

Servers

OGSA Enabled

MessagingOGSA Enabled





SecurityOGSA Enabled OGSA Enabled





















OGSA Architected Grid Services

Applications

Web Services-Resource Framework


http://www.globus.org/wsrf/

http://www.oasis-open.org/home/index.php

The main improvements consisted of:

� Splitting the monolithic OGSI into a more meaningful subset of specifications, in order to provide a collection of specifications that can be used either individually or in combinations

� Improving compatibility of Grid services with existing Web services tooling, hereby defining an architecture more clearly aligned with the general evolution of Web services

� Refining Grid services specifications to make a clear distinction between the service that operates a resource and the resource itself

More to the point, WSRF introduces both a design pattern to specify how to use Web services to access “stateful” components, and a message-based publish-subscribe mechanism to Web services.

From a Grid perspective, this means all components of a Grid architecture can be described via WSRF as stateful resources accessible through one or more Web services, whether they are physical entities (processor, disk drive, communication link, and so on) or logical construct (authorization, job, subscription, and so on), real or virtual, static or dynamic (through lifetime management).

Figure 2-2 gives an overview of the Web Services based Grid Application Model.

Figure 2-2 Web Services based Grid Application Model

Web Services Messaging, Security, …

Web Services-Resource Framework (WS-RF)

Grid Services

Program Execution Data Services

Application-Specific Services

Web Services Messaging, Security, …

Web Services-Resource Framework (WS-RF)

Grid Services

Program Execution Data Services

Application-Specific Services


For further informtion about Grid standards, check the following Web sites:

http://www.gridforum.orghttp://www.globus.orghttp://www.oasis-open.org

2.2.2 OGSA ImplementationsOGSA implementations are grid development toolkits that enable programmers to build OGSA-compliant grid solutions. Toolkits are available for the following programing languages, some of them discussed in this section:

� Java/C++ (the Globus Toolkit)

http://www.globus.org/toolkit/

� Perl (WSRF::Lite)

http://www.sve.man.ac.uk/Research/AtoZ/ILCT

� Python (pyGlobus)

http://www-itg.lbl.gov/gtg/projects/pyGlobus/

� .Net (WSRF.Net)

http://www.cs.virginia.edu/~gsw2c/wsrf.net.html

The Globus ToolkitThe Globus Toolkit provides an open source reference implementation of basic services for building a grid infrastructure and for deploying grid applications.

The Toolkit relies on the open standard protocols described above, and provides APIs to allocate and manage shared resources in a secure framework. It provides the core services necessary to address the Grid Computing challenges introduced earlier.

The IBM Grid Toolbox V3 for MultiplatformsThe IBM Grid Toolbox V3 for Multiplatforms is a platform for grid programmers to develop and test grid services and applications, and for grid administrators to host them.

Note: The Globus Toolkit is an open source software developed by the Globus Alliance. More information can be found at the following Web site:

http://www.globus.org/


http://www.gridforum.org

http://www.globus.org

http://www.oasis-open.org

http://www.globus.org/toolkit/


http://www-itg.lbl.gov/gtg/projects/pyGlobus/



In addition to GT 3.0 features, the Toolbox provides the following add-ons:

� An integrated wizard-based installer (InstallShield technology)

� A grid service Java/J2EE run-time environment based on the GGF Grid Service specification for hosting grid services

This environment replaces the open source stand-alone container with an embedded WebSphere Express Application Server (see 2.3.1, “WebSphere Application Server” on page 30 for more information).

� A Web-based management application used to manage services within the run-time environment

� A set of configuration and administration commands

� Development tools to build, package, and use grid services

� Common (base) grid services for:

– Discovery via Service Group

– Policy Management

– Common Management Models (CMM)

� An Information Center, including tutorials to assist with the education and understanding of the technologies and capabilities packaged within the product

� Sample grid services and applications demonstrating key capabilities

Besides, IBM Grid Toolbox includes an embedded relational database used for:

� Reliable File Transfer service journaling, which is required for recoverability

� Backing store for Policy services

� Backing store for Service Group service

Finally, IBM Grid Toolbox embeds OpenJMS for the JMS based notification framework.

The Platform Globus ToolkitThe Platform Globus Toolkit is a commercial distribution of the Globus Toolkit 3.0.

Note: The IBM Grid Toolbox is a commercial release of the Globus Toolkit 3.0 with IBM added value. More information can be found at:

http://www.ibm.com/grid/solutions/grid_toolbox.shtml



Here is a list of features provided by the Platform Globus Toolkit:

� Round-robin job scheduling

� Advance reservation booking, query, and control

� Reservation-based job scheduling

� Job throttling support for increased reliability

� File staging support

� Interoperability with a wide range of third-party workload management systems and grid service providers

� Scalability

� Job persistence

� Load balancing between multiple Platform LSF clusters

� Intelligent scheduling based on host type specification

� Kerberos Security Support

2.3 Grid application environmentsThe application environment is the arena where applications live, provide services to requesters, and consume services from the grid. The environment is intended to facilitate the life cycle of the grid-enabled applications or, in other words, how they are developed, run, and maintained.

2.3.1 WebSphere Application ServerWebSphere Application Server provides an operating environment with advanced performance and availability capabilities in support of dynamic application environments.

Note: In addition to the base set of components and services available in GT3, Platform Globus Toolkit includes significant extensions based upon the Community Scheduler Framework (CSF), and connectors that provide interoperability with a wide variety of third-party workload management systems and grid offerings. More information can be found at:

http://www.platform.com/Products/Platform.Globus.Toolkit/Product.Information/Features.Benefits.htm/




More specifically, and from a grid standpoint, WebSphere Application Server is bringing programmers a Java run-time environment to support grid implementations, and monitoring capabilities to automatically manage the application workload and route traffic to one server or another according to its workload at a given time.

In addition to its support of Web services standards, WebSphere Application Server comes with a set of tools and Java APIs that enable, for example, the development of portal-type Web applications, such as a job submission portal. On top of the job submission features, a programmer could implement a full grid environment by enforcing security with user and project authentication, policy-based authorization, proxy delegation and certificates management, and by adding capabilities such as input data staging, computing resource selection, and multi-format job output retrieval, all of it accessible through a simple Web browser, as described in the Figure 2-3.

Figure 2-3 Globus-based Grid Portal

Note: WebSphere Application Server is the foundation of the WebSphere software family, the industry's premier Java-based application platform, integrating enterprise data and transactions for the dynamic e-business world. More information can be found at:

http://www.ibm.com/software/websphere/index.html

Java Server Page

Java Servlet

Java Web Service

Enterprise Java Bean

GlobusJava

CoG Kit

WebSphere Application Server

View ModelController

Database

Certificate Authority

GridResource

User Browser

HTTP

HTTP



In this case, features such as the Grid Registration service are implemented as a Web application and based on the Model View Controller (MVC) design pattern. The separation of presentation logic and business logic is made by defining three parts, Model, View, and Controller, hosted by WebSphere Application Server.

The Controller manages and controls all interactions between the user and the application. Usually, the Controller is a servlet that receives an HTTP request when the user clicks on a link in the browser. Then, the Controller passes the input parameters to the Model that does the work. This Grid Registration service is made of several Controller servlets that implement methods specific to each user action.

The Model encapsulates the business logic, rules, and data, and does the business processing, usually implemented by Java classes, Java Beans, or Enterprise Java Beans (EJBs). The Grid Registration service can use a session EJB™ to store and fetch data from the database.

Called by the Controller, the View part uses the results of the business process and constructs the response to be presented to the user, usually implemented through Java Server Pages (JSPs). JSPs build dynamic HTML code and send it as an HTTP response to the user's browser.

Here, WebSphere Application Server interacts with the Globus Toolkit through the Globus Java Commodity Grid (CoG) Kit, which provides a set of APIs to handle Globus resources; it is especially useful here to create certificates signed by the Certificate Authority, and exchange information with the grid resource manager in order to submit a job.

Of course, these are only a few hints about what can be created using WebSphere Application Server as the hosting keystone; further developments could extend the reach of the application to other products, in order for it to include job scheduling, resource provisioning, and other grid features.

2.3.2 WebSphere Extended DeploymentWebSphere Extended Deployment (WAS-XD) is an extension of the WebSphere Application Server to support intensive workloads. WAS-XD allows the splitting of a given task into several tasks, and run each piece in different application servers that belong to the same domain. This is an implementation of grid computing, specifically, a server grid with parallel tasks. Applications have to include specific calls to WAS-XD services and have to be adapted to this paradigm in order to be split in smaller and parallel pieces.

Once the WAS-XD engine is running, it finds those applications that can be placed or split in different application servers. Based on policies, it assigns


pieces of execution to the right application server. It allows you to use underutilized application servers, to provide additional capabilities during workload peaks, and to improve the quality of services for the applications running in the WAS-XD domain.

WAS-XD provides dynamic allocation of resources, and virtualizes pool of resources and applications. It has autonomic characteristics, since it provides real-time information about the current status in regards to the policies, and acts according to them. WAS-XD allows you to apply business goals to WebSphere Application Server applications. The operator has a graphic console that shows the current status of the WAS-XD. It gives a new design pattern of development that is specially designed for high performance and reliable transaction processing.

2.3.3 WebSphere Portal ServerWebSphere Portal Server is IBM’s platform to create, run, and maintain corporate portals. This type of portals manages transactional information, and integrates information originating from different sources, like databases, existing systems, content providers, business partners, and so on.

This kind of platform requires a sophisticated degree of integration to provide comprehensive Web access to heterogeneous sources of information. This integration is achieved through the use of “portlets”, which are standardized objects, loosely coupled, that have the required logic to look for some source of information and publish that information to the portal. The portal engine provides the required connectors and run-time services to run and contain the portlet itself, to manage the portlet’s connector with the foreign source of information, and to get the portlet’s output and display it in the Web page. WebSphere Portal Server provides services to implement this programming model.

In terms of hosting, WebSphere Portal Server maintains the environment where the portlets live. This container is a specialized portlet’s application server. All portlets share the same communication protocols with the portlet container.

Note: WebSphere Extended Deployment is an extension of the WebSphere Application Server to support intensive workloads. More information can be found at:

http://www.ibm.com/software/webservers/appserv/extend/features/

Note: WebSphere Portal Server is part of the WebSphere software family. More information can be found at:

http://www.ibm.com/software/genservers/portal/




http://www.ibm.com/software/webservers/appserv/extend/features/

WebSphere Portal Server also provides common services to the portlets on top of those provided by the application server itself, like authentication, authorization, connection pooling, and so on. For example, WebSphere Portal Server provides personalization of content, and interaction between the user and applications, processes, and content. WebSphere Portal Server implements a single point of interactive access to all the information that a user needs and the enterprise publishes through a comprehensive and unified Web interface.

Information and the path to this information can be personalized according to the user and type of user who is accessing it. Personalization services also take into account the kind of device that is being used to access the portal, and their policy can be modelled as a matrix of users and devices. Devices can be typical browsers, low end workstation browsers, public Web browsers (which impacts the information’s confidentiality), cell phones and PDAs using WAP, pagers, interactive voice response systems (IVRs), scripts automatically retrieving content, and so on. Defining how the information is accessed and formatted depending on the device is crucial to enable a user to have different profiles and preferences based on the device that is being used.

Additional services provided to enhance the portal are, for example, tools to search topics or certain content in information managed by the portal. Portal Document Manager is a portlet that allows users to share, view, and organize files. On the other hand, Productivity Components allow users to create, edit, convert, and view documents like word processing files and spreadsheets. WebSphere Portal Server also includes collaborative tools, like Instant Messaging or People Finder Portlet, which bring advanced navigation capabilities over a corporate directory. My Lotus® Workspace (QuickPlace®) portlet manages workplaces, which are virtual desktops managed by the portal. WebConferencing Portlet provides a suite to manage online meetings. Users can schedule or join meetings directly from the portlet, with a few mouse clicks. Finally, WebSphere Portal Server can provide external search capabilities in DB2, Lotus Notes®, and Lotus Domino® databases, HTML and text documents, and external Web search engines.

Note: More information about DB2 can be found at:

http://www.ibm.com/software/data/db2/

Note: More information about Lotus Notes, Lotus Domino, and Lotus Workspace can be found at:

http://www.lotus.com/



http://www.lotus.com/

What does a Web portal bring to a grid?In the same way, a Web portal is an entry point to access and use information, or, to existing systems, a portal can be an entry point to a grid, as soon as interaction is required between the user of the grid and the infrastructure itself. A given user can submit a job to the grid from its Web browser through the portal, or schedule a time and book resources to run it. The output of this job can be returned by the grid to a PDA, or a cell phone. The grid itself can notify a user about the execution of a job by e-mail, by a SMS message, or by a message on voice mail. These proactive features are brought to the grid by the portal, in addition to more classical notifications like a Web page.

WebSphere Portal Server can implement high level services to track specific actions taken by the users. This may be particularly useful to prevent abusive usage of the grid, like repetitive high priority jobs whose results are not retrieved immediately. It is vital to know how relevant a job submitted to the grid is, so that more important jobs can have precedence if need be. As a result, grid schedulers and workload managers implement sophisticated policies to assign job priorities and book resources in advance, and dynamically adapt their policies. This is an example of how sophisticated but typical portal characteristics can improve the user experience and the grid efficiency.

In some cases, portals are necessary rather than just desirable for a grid. For example, in a IBM Grid Toolbox or Globus Toolkit grid, all grid nodes must have the software installed to be registered to the grid and submit jobs. A way to plug these machines into a IBM Grid Toolbox or Globus Toolkit grid is to build “submission clients” that can run on the desktops and connect them to the grid. Another more scalable and easy to administer solution would be to create a Web portal that connects the user with the grid. It would be necessary to build a portlet that allows you to submit jobs, to schedule their execution and inquire about their status, to book CPU time in advance, and so on.

All the features described above are typical qualities of the Web Services Core and Hosting ecosystem from the user interaction standpoint. WebSphere Portal Server, IBM Grid Toolbox, and the WebSphere Application Server family provide all the tools necessary to implement them and integrate people, besides applications, with the grid.


2.3.4 WebSphere MQ FamilyThe WebSphere MQ family includes many products, offering a range of capabilities. It allows you to exchange information across different platforms, integrating new and existing business applications.

WebSphere MQ MessagingWebSphere MQ Messaging (formerly MQSeries®) provides any-to-any connectivity from desktop to mainframe, through business quality messaging, supporting several platforms. MQ supplies a reliable messaging system and APIs for several languages on the supported platforms. By using MQ, an application can interact with another, whatever protocols and operating systems are used by either one. The developer only needs to create the interface to implement the syntax and semantic of the message. MQ provides the connectivity, and the application brings the logic to create and interpret messages.

What do WebSphere MQ products bring to grid?WebSphere MQ products are components that are part of a traditional IT environment, as soon as applications need reliable connectivity. Easy to implement and with a short learning curve across different platforms, they are a good option to be considered when the need arises to simplify asynchronous communications between a number of heterogeneous applications using a messaging system.

On the other hand, MQ has Web services connectors that can use SOAP and XML to receive and transfer messages. Moreover, if an application that exposes MQ interface to the rest of the infrastructure needs to connect to a grid, it may be relevant to use MQ capabilities as an entry point to the grid, for example, by implementing a grid gateway that sits waiting for MQ messages on one side and sends grid messages on the other side. This would bring a substantial benefit to the architecture by making it unnecessary to modify the interfaces of the applications that submit jobs to the grid, while guaranteeing the reliability of the information transmitted as messages.

In a different context, a grid can have several entry points: standard Web services, a portal, and ad-hoc connectors, which may be implemented with MQ. For example, some platforms do not support Web services, or applications can be developed in a programming language that was not designed to easily create SOAP content. In this case, it is still possible to leverage the applications and to protect existing investments while enabling the building of a grid: instead of

Note: WebSphere MQ Family is part the WebSphere software solutions. More information can be found at:

http://www.ibm.com/software/integration/wmq/




creating several new pieces of code, one for each application that could submit a job to the grid, MQ and its connectors can be used to implement a gateway between the applications and the grid that accepts messages on the applications side and passes them to the grid.

2.3.5 WebSphere Messaging ResourcesWebSphere Application Server V6 includes new and additional capabilities related to messaging that can be used as a Java Messaging Service (JMS) provider by J2EE applications running on the application server. WebSphere Application Server V6 still supports other JMS providers, such as WebSphere MQ. Messaging capabilities, using standard interfaces such as those provided by JMS, enables an Enterprise Service Bus (ESB), to aid in integration among requesters and providers within a virtual organization. Requesters and providers are usually applications and systems that need to communicate to provide new services to the end users. Integration between requesters and providers usually requires “n-to-n” connections and messaging management. It implies building several interfaces. An ESB allows the creation of a single interface for every provider and requester to access the ESB itself. Then the ESB translates the message to the appropriate protocol for the requester or the provider that the message was sent to. In that way, the ESB reduces the number of interfaces, eases the administration and maintenance of the interfaces, and allows new requesters and providers to be plugged in with less effort.

These capabilities, in conjunction with WebSphere MQ, provide an ESB ready to use by J2EE applications running on WebSphere Application Server. Several connectors are available, such as HTML, HTTP, SOAP, Web Services, JMS, CICS®, SAP-BAPI, XLM, RMI, SQL, and so on, while WebSphere MQ provides the support for the JMS internal mechanism. These messaging resources can be accessed as Java beans.

The grid environment can access the ESB's services instead of accessing individual service providers with ad-hoc mechanisms. In fact, if an ESB already exists in an organization, and the services to be provided by the grid are accessible through the ESB, then the grid should access these services through the ESB.

Note: WebSphere Messaging Resources (WAS-MR) is a feature of WebSphere Application Server V6. More information can be found at:

http://www.ibm.com/developerworks/websphere/techjournal/0501_reinitz/0501_reinitz.html




Table 2-1 on page 39 shows a comparison overview between WebSphere Business Integration Messaging Broker (WBI-MQ) and WebSphere Messaging Resources (WAS-MR).

2.3.6 WebSphere Business Integration Messaging BrokerWebSphere Business Integration Messaging Broker (WBI-MQ), formerly MQ Integrator, is a hub of integration that consolidates and integrates the interfaces of messaging among several systems. It helps to build an Enterprise Service Bus (ESB). It is fully based on WebSphere MQ (WAS-MQ) and exports its services to applications through WAS-MQ connections and Java APIs. WAS-MQ only provides the connection or the link between requesters and providers of the services, while WBI-MQ adds powerful logic to transform the messages. It is common in a corporate environment that different applications have to communicate with others and several interfaces have to be developed. The number of interfaces is usually “n-to-n”, where n is the number of systems. Even with WAS-MQ, the applications have to implement all the messaging transformations in a “n-to-n” way, because WAS-MQ only gives the link between them. The messaging transformation is still the responsibility of the applications. On the other hand, WBI-MQ implements a hub in such way that the applications to be integrated only need to create one interface, to the hub, that works as an ESB. WBI-MQ provides tools to create the transformation rules and a repository to keep them alive. Then, messages are transformed on-the-fly.

WBI-MQ can be used in a grid environment as the ESB of a corporate grid. The grid environment can access ESB's services instead of accessing the service providers with ad-hoc mechanisms. In fact, if an ESB already exists in an organization, and the services to be provided by the grid are accessible through the ESB, then the grid should access these services through the ESB. A grid can also be a service provider of the ESB of an organization.

Comparing WBI-MQ and WAS-MRTable 2-1 on page 39 shows a comparison overview between WebSphere Business Integration Messaging Broker (WBI-MQ) and WebSphere Messaging Resources (WAS-MR).

Note: WebSphere Business Integration Messaging Broker is a hub of integration that consolidates and integrates the interfaces of messaging among several systems. More information can be found at:

http://www.ibm.com/software/integration/wbimessagebroker/


http://www.ibm.com/software/integration/wbimessagebroker/

Table 2-1 Comparing WBI-MQ and WAS-MR

2.4 ConclusionIn this chapter, we briefly described the standards and products associated with the building of basic grid computing infrastructures. Some basic products for the Web Services Core/Hosting discipline are part of the WebSphere family. Most of their capabilities are well known from the traditional IT perceptive and can be integrated with a variety of other products (for example, IBM Grid Toolbox includes WebSphere Application Server Express Edition).

WebSphere Business Integration Messaging Broker

WebSphere Messaging Resources

A whole product. A feature of WebSphere Application Server V6.

Implements a middleware outside the application server, but accessible by the application server and other applications.

Implements a JMS mechanism to the application server.

Uses WAS-MQ internally and externally. Uses WAS-MQ internally and supports MQ externally wrapped in JMS messages.

Publishes services through WAS-MQ queues. These services can be wrapped in Java classes.

Publishes services as JMS services and Java beans.

Provides basic connectors out of the box while others have to be added.

Provides several connectors out of the box.

Most of the connections with providers should be treated as MQ messages.

Usually connections with providers can be treated in the original protocols.

High availability is implemented with its own tools.

High availability is implemented on the application server.

Messaging transformation is provided by rules by using the tools provided. These rules are stored in an internal database that acts like a persistent repository.

Messaging transformation is provided by code, by using provided templates.



Chapter 3. Workload virtualization


� Concepts, technologies, and features related to workload virtualization

� Scheduler role on grid environment

� How to choose the appropriate grid scheduler

� Products that support grid scheduling

3


3.1 SchedulingScheduling is the core of workload virtualization in a grid solution, as it is the mediator or the middleman that manages the distribution of jobs for workload balance. This can be performed as simply as taking the next available resource, but often this task involves prioritizing job queues, managing the load, finding workarounds when encountering reserved resources, and monitoring progress.

Typically, a job scheduler is a program or application responsible for initiating and managing jobs automatically by processing job control statements prepared by the submitter. Some job schedulers also offer a graphical interface to monitor and analyze a job’s execution.

Some of the typical features of a job scheduler are:

� Receive a job submission for processing

� Monitor jobs, checking for completions or failures

� Advanced features

– Performance Monitoring

– Jobs backfilling

– Jobs status and report

– Jobs priority

Jobs schedulersJob schedulers are able to submit, control, and monitor the workload of jobs submitted in a network of computers. The jobs submitted are ran based on priorities and scheduling algorithms that are typically defined by the submitter. Some of the most popular job schedulers are IBM LoadLeveler®, Platform LSF, Portable Batch System (PBS), and Condor.

Figure 3-1 on page 43 points out the focus of the chapter and where job schedulers are typically used.


Figure 3-1 Focus of scheduling environment

3.2 Grid scheduling and policiesThere might be different levels of schedulers within a grid environment. For example, a cluster could be represented as a single resource and it may have its own scheduler to manage its nodes. A higher level scheduler (sometimes called a meta-scheduler) might be used to schedule work to be accomplished on multiple clusters, while each cluster's scheduler handles the actual scheduling of work on its individual nodes. The main goal is to optimize the allocation of jobs to resources and achieve better performance.

Here is a snapshot of some different types of schedulers:

� Gang scheduler

The gang scheduler permits a set of processes, or multiple threads from a single process, to be scheduled concurrently as a group.

Security

CSFIBM LoadlevelerPlatform's LSFOpenPBSetc

Job Submit MetaScheduler

Scheduler Scheduler

Application

Data access

User

Chapter 3. Workload virtualization 43

� Grid engine

A scheduling system that accepts job submissions and schedules them for execution on the appropriate nodes of a grid based on resource management policies.

� Meta-scheduler

It provides an interface and adapters to the lower level schedulers. May perform load balancing of workloads across multiple systems. Each system would then have its own local scheduler to determine how its job queue is processed.

3.3 How to choose the appropriate schedulerFor choosing the most suitable scheduler for a particular application, one should fully understand the nature of such an application.

� Is the application CPU bound? Can it be split into smaller units?

� Can it run on multiple platforms?

� Can the computations be split and carried out in any order?

� Does it mostly read data rather than write it and in which quantities?

� Is this a one-time execution or an application that is run over and over?

� Are portions of the results reusable in another runs of the application?

� Does the application involve large amounts of input data?

� Does the application involve receiving changing data?

� Does the application involve intensive communication?

Knowing the inner details of an application, and therefore knowing that this application can be “gridified”, lead us to choose a best fit for our scheduling requirements. Such requirements should also take into account operational issues like the ones addressed in the following questions.

� What are the user requirements?

User requirements are the most critical factors here; some users jobs may ran for days or maybe weeks before they complete. Users might want to have some checkpoints features in the scheduler so that it can resume without restarting the whole job. Status reporting and monitoring might be the area in which the user wants to focus on.

Some additional user requirements relies on workload scheduling activities, priority algorithms and policies, automatic rescheduling, load balancing, performance, and capacity.


� What are the hardware and operating systems requirements?

While this is not the most important question in defining the appropriate scheduler, we do know that some schedulers do not support multiple platforms.

� How do the users submit their jobs?

This question gives an insight into what kind of interfaces the schedulers support. Portals are very common and popular today. Most schedulers support portals access for submitting jobs.

� What is the data access used for the job run?

Most often when we submit a job, some amount of data needs to be transferred to the grid. Eventually, users use a mechanism such as “GridFTP” to transfer such data. In many cases, where data needs to be shared and accessed by multiple nodes, it is necessary to have some mechanism for managing multi-party access to data chunks.

3.4 Mapping user requirements into productsIn most cases, we need to directly map the users requirements into the appropriate product. There are two broad categorization of schedulers: the server-oriented grid and the desktop-oriented grid. Desktop grids are typically used in environments where CPU harvesting is deployed, trying to make use of idle workstations for processing. On the other hand, server-oriented grids are used in environments where specialized and high-performance machines are fully (or almost fully) available for the grid. In both cases, schedulers are responsible for managing the workload across the computing resources for efficient job execution.

3.4.1 Tivoli Workload SchedulerWhile workload management might be under the same umbrella as scheduling, it has some particularities that should be clearly noted. Workload management typically provides more functionality for managing the job that is being submitted, allowing it to be easily moved from one node to another. One such product is IBM Tivoli Workload Scheduler. It is calendar based, and has a choreographer scheduler that automates, monitors, and controls the flow of work through an enterprise's entire IT infrastructure on both local and remote systems. From a single point of control, the suite analyzes the status of the production work and drives the processing of the workload according to installation business policies. It supports a multiple end-user environment, enabling distributed processing and control across sites and departments within an enterprise.


Tivoli Workload Scheduler for Virtualized Data Centers V8.2 extends the capabilities of IBM Tivoli Workload Scheduler to workload optimization and grid computing. It provides an enterprise-level scheduling solution that helps you perform cross-platform, cross-domain, and cross-enterprise scheduling by integrating business applications across grid, mainframe, and traditional distributed environments. It also provides fault tolerance, scalability, and outstanding workload performance. Workload Scheduler for Virtualized Data Centers V8.2 provides the following features:

� Centralized control: All scheduling objects are defined and stored on a master scheduling controller.

� Delegation of work: Scheduling duties are sent from the master to other workstations in the network.

� Fault tolerance: In the event of a loss of network connectivity with the master, other workstations continue to run jobs.

� Remote administration: Manipulation of workload can be done remotely using the Job Scheduling Console.

Automatic driverWorkload Scheduler provides leading-edge solutions to problems in a production workload management. It can automate, plan, and control the processing of an enterprise’s entire production workload. It functions as an “automatic driver” for the production workload, maximizing the throughput of work and optimizing the resources usage. When Workload Scheduler interfaces with other system management products, it actually takes part in a more comprehensive integrated automation and systems management platform.

IBM Tivoli Workload Scheduler can manage the complete flow of work through an enterprise’s entire operation, on both local and remote systems.

Single point of controlFrom a single point of control, Workload Scheduler analyzes the status of the production work and drives the processing of the workload according to installation business policies. It supports a multiple end-user environment, enabling distributed processing and control across sites and departments within an enterprise.

Note: Tivoli Workload Scheduler is part of the IBM Tivoli family of products. For more information, refer to the following Web page:

http://www.ibm.com/software/sw-bycategory/tivoli/


http://www.ibm.com/software/sw-bycategory/tivoli/

3.4.2 LoadLevelerLoadLeveler is another popular scheduling software, typically deployed on IBM

pSeries hardware. It is also bundled with the Cluster 1600 solutions to provide job management. It has tight integration with other software from IBM, such as Engineering and Scientific Subroutine Library (ESSL), Parallel ESSL (PESSL), and Parallel Environment (PE).

LoadLeveler allows users to optimize workload execution and performance on a pool of pServer machines by matching the jobs’ processing needs with the available resources. LoadLeveler schedules jobs both on clusters and on a High Performance Computing (HPC) environment, and provides functions for submitting and processing jobs quickly and efficiently in a dynamic environment.

The scheduler manages both serial and parallel jobs over the grid nodes. In this particular case, this distributed environment consists of a pool of machines or servers, often referred to as a LoadLeveler cluster. Machines in the pool may be of three basic types: desktop workstations available for batch jobs (usually when not in use by their owner), dedicated servers, and parallel machines.

The allocation of jobs depends on the availability of resources within the cluster and on a set of rules defined by the LoadLeveler administrator. A user submits a job using a job command file, and the LoadLeveler scheduler attempts to find resources to satisfy the requirements of the job.

Here are the LoadLeveler scheduling features:

� Checkpoint/Restart: Saves the state of a running job so the job can be suspended and re-started from the same place later.

� Parallel Job Suspension: Suspends a parallel job in favor of a higher-priority job, then resumes the suspended job at the same place.

� Backfill Scheduling: Improves resource utilization by reducing idle time.

Note: LoadLeveler, from IBM, is a distributed network-wide job management facility designed to dynamically schedule work on IBM UNIX servers, such as the IBM pSeries and IBM RS/6000® systems. For more information, refer to the following Web site:

http://www.ibm.com/servers/eserver/clusters/software/

Important: Linux machines and AIX 5L™ machines can now exist in the same LoadLeveler cluster. For more information, refer to the following Web site:

http://publib.boulder.ibm.com/infocenter/clresctr/index.jsp?topic=/com.ibm.cluster.loadl.doc/doc_updates/ll3_2.1update.html





� Runs lower priority jobs in the queue on available resources.

� Runs the higher priority jobs as soon as the required resources become available; it schedules jobs to maximize resource utilization and minimize job completion time.

� Jobs are scheduled based on resource availability and user-defined rules to match processing needs with resources.

� Machines can be of different types: Dedicated servers, parallel machines, and idle desktops.

� Both serial and parallel jobs can be scheduled.

Figure 3-2 shows the different components in LoadLeveler.

Figure 3-2 LoadLeveler components

3.4.3 DataSynapse GridServerGridServer is a highly scalable software infrastructure that allows application services to operate in a virtualized fashion, unattached to specific hardware resources. Client applications submit requests to the grid environment and GridServer dynamically provisions services to respond to the request. Multiple client applications can submit multiple requests in parallel and GridServer can dynamically create multiple service instances to handle requests in parallel on different grid nodes. This architecture is therefore highly scalable in both speed and throughput. For example, a single client will see scalable performance gains

Scheduler• Receives jobs submitted by users• Manages each job to completion & maintains information• Asks Central Manager to find appropriate machine(s)

Central Manager

• Examines each job’s requirements & determines most appropriate machine(s) to run the job

• Central Manager is not a single point of failure

Executing Machines

• Run the jobs

Submitting Machines• Submit, query or cancel jobs• May be outside the LoadLeveler cluster


in the processing of multiple requests, and many applications and users will see scalable throughput of the aggregate load.

GridServer has four different components in its architecture:.

1. Grid Clients: The components that submit service requests to grid, also called Drivers.

2. Engines: The processes that host and run services on the grid nodes.

3. Brokers: The component that provides request queuing, scheduling, and load-balancing. Brokers are also responsible for managing Engines.

4. Directors: The component that assigns Grid Clients and engines to Brokers based on policy. They manage and load balance Engines across available Brokers.

Figure 3-3 presents a logical architecture that supports a N-Tier model of Brokers, Directors, and Engines. High-availability hot failover with checkpointing of method level workload processing is also available.

Figure 3-3 GridServer architecture

Engines and Grid Clients log in to the Director and are authenticated; the Director then routes Engines and Grid Clients to available Brokers. Grid Clients submit requests through the Broker. Engines receive work requests from the Broker, and in most cases, exchange data directly with the Engine processes. This allows the system to be highly scalable. Since the Brokers manage all work requests, load balancing is optimal and resilience is built into the system. Brokers manage Engines and Grid Clients, and schedule work via lightweight messages.

Client

GridServerManager

Director

Broker

Engine Engine Daemon

Engine Instance

Engine Instance


Grid Clients and Engines can exchange work data directly when using Direct Data Transfer. In addition to providing a programming model and runtime environment for Grid Services, GridServer also includes a highly manageable operating environment. There is a Web-based console, the GridServer Administration Tool, and a SOAP-based administration interface. Both allow you to deploy Services, manage the workloads running on the Grid, and configure the GridServer environment.

3.4.4 Platform LSF (Load Sharing Facility)Platform LSF is software for managing and accelerating batch workload processing for compute- and data-intensive applications. With Platform LSF, an organization can intelligently schedule and guarantee the completion of batch workload across a distributed, virtualized IT environment. Platform LSF fully utilizes all IT resources regardless of operating system, including desktops, servers, and mainframes, to ensure policy-driven, prioritized service levels for always-on access to resources. Underpinning Platform LSF is the production-proven, open, robust Virtual Execution Machine (VEM) architecture that sets the benchmark for performance and scalability across heterogeneous environments.

Platform LSF Features are:

� High performing, open scalable architecture � Comprehensive set of intelligent scheduling policies, including:

– Fairshare – Preemption – Advance Reservation – Resource Reservation – SLA Scheduling

� Advanced self-management� Heterogeneous platform support� Extensive application support� Comprehensive, extensible, and standards-based security

Note: GridServer is a software infrastructure that allows application services to operate in a virtualized fashion, unattached to specific hardware resources. For more information, refer to the following Web site:

http://www.datasynapse.com/

Note: Platform LSF is part of the Platform LSF family of products. For more information, refer to the following Web site:

http://www.platform.com/products/LSFfamily/




3.4.5 Platform LSF MultiClusterPlatform LSF MultiCluster extends an organization's reach to share virtualized resources beyond a single Platform LSF cluster to span geographical locations. With Platform LSF MultiCluster, local ownership and control is maintained, ensuring priority access to any local cluster while providing global access across an enterprise grid. Organizations using Platform LSF MultiCluster complete workload processing faster with increased computing power, enhancing productivity and speeding time to results.

3.4.6 Platform LSF HPC (High-Performance Computing)Platform LSF HPC is software for managing and accelerating High Performance Computing (HPC) mission-critical workloads. With Platform LSF HPC, you can intelligently schedule parallel and serial workloads, providing you with the capability of solving large, grand challenge problems while utilizing the available computing resources at maximum capacity. It enables you to take full advantage of high performance network interconnects available on clustered systems and supercomputers.

Platform LSF HPC features topology-based scheduling that enables maximum application performance for industry leading interconnects. Underpinning Platform LSF HPC is the robust Virtual Execution Machine (VEM). More than a distributed application technology, the VEM is a standards based, high performance, and system-level architecture. The VEM provides a true virtual environment that is ideal for critical workload management. This workload-centric approach offers many performance, reliability, and management advantages. Currently powering a variety of demanding industries and critical applications, the production-proven VEM is the ideal architecture to handle the ever-increasing demand for application performance.

Platform HPC Features:

� Optimized application, system, and hardware performance� Enhanced accounting, auditing, and control� Advanced self-management� Commercial grade system scalability and reliability� Extensive hardware support� Comprehensive, extensible, and standards-based security

Note: Platform LSF MultiCluster is part of the Platform LSF family of products. For more information, refer to the following Web site:




3.4.7 Platform SymphonyPlatform Symphony is software that distributes and virtualizes compute-intensive application services and processes across existing heterogeneous IT resources, creating a shared, scalable, and fault-tolerant infrastructure, delivering faster and reliable application performance.

Platform Symphony’ service-level grid infrastructure virtualizes compute intensive applications, improving their performance, scalability, and reliability. Platform Symphony dynamically allocates resources to satisfy the needs of single or multiple applications and their workloads across individual or multiple lines-of-businesses (LOB).

Platform Symphony Features:

� Support for the widest range of workloads found in financial services

� True service-level virtualization for greater application reliability

� Global, enterprise-class scalability for optimal performance and failover

� Guaranteed service execution of all application workloads

� A single common infrastructure (VEM) for rapid migration from the desktop to the data center

� The ability to deploy at any IT level from the application to the line-of-business (LOB) and throughout the enterprise

� A self-healing architecture lowering administrative costs

� Comprehensive, extensible, and standards-based security

3.4.8 United DevicesUnited Devices offers a software platform for deploying enterprise grids on heterogeneous and geographically distributed resources, such as clusters, servers, and desktops. The Smallpox Analyzer from http://www.grid.org was developed using United Devices software.

Note: Platform LSF HPC is part of the Platform LSF family of products. For more information, refer to the following Web site:


Note: For more informtion about Platform Symphony, refer to the following Web site:

http://www.platform.com/Products/Platform.Symphony/





http://www.grid.org

The United Devices Grid MP platform allows organizations to build a computation grid from a group of different resources. A GUI management console provides the ability to submit and monitor jobs as well as perform various administration functions. The product is useful for CPU intensive grid jobs that are self contained and that use small amounts of data, as well as for management of distributed resources traditionally controlled by DRMS or other tools. By adjusting polling times accordingly, the Grid MP platform demonstrates its scalability, handling a very large number of systems. The architecture is designed to scale from a few hundred devices to several hundred thousand devices to a million devices.

The Grid MP platform includes a workload scheduler that balances the computational needs of application users with the usage preferences of each device that is participating in the network. The system supports multiple hardware architectures and operating systems in the various desktops that serve as resources, while remaining transparent to the user or job submitter. It also includes a centralized scheduling engine that balances massive computational demands with a supply of compatible devices available in the grid. Selection of a job at dispatch time depends on the connecting device's capability, job priority, device availability, usage preferences, and other constraints associated with the job. In this way, Grid MP ensures that the job is sent to the devices that is best suited to run it. When a job is dispatched, its meta-data is also sent. The files containing actual data are uploaded or downloaded to/from the file service. The Dispatch Service also automatically reschedules jobs that have failed either due to device shutdown or job timeouts.

Jobs are automatically rescheduled when resources are overloaded or disconnected. Job priorities are automatically escalated based on the time since their last dispatch. In addition, users are able to manually suspend, migrate, or preempt jobs. Unlike traditional workload management systems that bind jobs to a specific resource at job submission time, Grid MP transparently schedules jobs to a variety of heterogeneous resources by binding to them at dispatch time. This dynamic binding balances workload more efficiently and vastly improves resource utilization.

Administrators can specify load thresholds to limit usage of device subcomponents, such as CPU, memory, and disk space. If a device exceeds its specified threshold, the grid application running on it is suspended until the load falls below the threshold.

Note: The grid solution from United Devices is part of the Grid MP platform family of products and services. For more information, refer to the following Web site:

http://www.ud.com/solutions/


http://www.ud.com/solutions/

In addition to traditional job scheduling built into the grid platform, United Devices offers a Unified Job Scheduling (UJS) product that allows organizations to manage isolated, dedicated Distributed Resource Managers (DRMs) from a single tool. This product preserves departmental control of resources while enabling the sharing of resource groups that previously operated in silos, with peaks and valleys in demand. Unified Job Scheduling provides virtualized access to all resources: systems, applications, and data. The architecture is based on open Web services standards that enable the creation of a true service-oriented architecture (SOA) for access to an enterprise-wide set of resources.

3.4.9 Altair PBS ProfessionalAltair PBS Professional is a workload management solution suitable for deployment in the most demanding production HPC environments. PBS Professional has the availability, redundancy, and resiliency features required for uninterrupted services.

PBS Professional is most often deployed across a set of server systems within an HPC datacenter environment. PBS Professional is a capable solution for large complex heterogeneous HPC datacenter deployments. PBS Professional can also be deployed across networks of workstations or personal computers. In cycle harvesting mode, PBS Professional provides significantly higher hardware utilization rates by placing batch jobs onto desktop systems when they are not being utilized for interactive work. PBS Professional is widely deployed throughout a variety of market segments.

PBS Professional provides three essential benefits:

1. All HPC assets (hardware and software licenses) are federated into a virtual pool of resources in an effort to maximize utilization of these assets.

2. The entire workload is then intelligently scheduled to run within this virtual pool based on predefined business policies, which determine how these HPC assets are to be shared.

3. The scientists and engineers are presented a consistent user interface for submitting and managing all types of workloads to all of the computing resources. This allows the users to concentrate on their work, not their computing environment.


3.4.10 OpenPBSOpenPBS is a simple workload management solution intended for deployment on small clusters comprised of dedicated homogeneous server nodes. OpenPBS is distributed in source code form and users will need to compile the software for each platform.

The end-user license agreement associated with OpenPBS is not a GPL license agreement. The OpenPBS license agreement allows any end user to utilize the software without any costs, but prohibits any entity from redistributing OpenPBS in conjunction with any commercial product (hardware, software, or service) that is not free of charge.

Altair does not offer professional software support for OpenPBS. However, there is an extremely large community of OpenPBS users worldwide and some level of support is facilitated through this open source community. The http://www.openpbs.org Web site presents a product comparison between OpenPBS and PBS Professional.

OpenPBS provides three essential benefits:

1. All HPC computer resources are federated into a virtual pool of resources in an effort to maximize utilization of these computers.

2. The entire workload is then scheduled to run within this virtual pool based on simple scheduling algorithms.

3. The scientists and engineers are presented a consistent user interface for submitting and managing all types of workloads to all of the computing resources. This allows the users to concentrate on their work, not their computing environment.

Note: PBS Professional is the commercial version of the original Portable Batch System (PBS) developed for NASA. PBS Professional is developed by Altair Grid Technologies (a subsidiary of Altair Engineering). Visit the following Web sites to learn more about PBS Professional:

http://www.altair.com/http://www.pbspro.com/


http://www.openpbs.org


http://www.altair.com/

http://www.pbspro.com/

3.4.11 CondorCondor, commonly used by universities and research and development companies, is a freely available, open source, software package that leverages the use of dedicated or nondedicated resources.

The Condor environment is suited to providing large amounts of computing power over significant periods of time. This has been termed High Throughput Computing (HTC) in contrast to High Performance Computing (HPC) which aims to provide peerless computing power over shorter periods through the use of special dedicated hardware or software infrastructures.

Condor converts collections of distributed workstations and dedicated clusters into a distributed high-throughput computing facility. Condor matches jobs with available machines using “Class Ads” for flexible resource matchmaking that provides resource management in a distributed environment with decentralized ownership of resources.

Figure 3-4 on page 57 is a very high level representation of how Condor works.

Note: OpenPBS is a freeware distribution based on the original Portable Batch System (PBS) developed for NASA. OpenPBS is owned and distributed by Altair Grid Technologies (a subsidiary of Altair Engineering). OpenPBS is downloadable at no cost to end-users in source code form. Visit the following Web site to learn more about OpenPBS and to download the software:


Note: Condor is a research project developed by the University of Wisconsin Madison. For more information, refer to the following Web site:

http://www.cs.wisc.edu/condor/


http://www.cs.wisc.edu/condor/


Figure 3-4 Simplistic view of Condor scheduling

3.4.12 Community Scheduler Framework (CSF)The Community Scheduler Framework (CSF) is an open-source implementation of a number of grid services, which together perform the functions of a grid metascheduler or community scheduler. CSF provides basic capabilities for scheduling and can be used as a development toolkit for implementing community schedulers.

Platform Computing has contributed the CSF open source metascheduler to the Globus Toolkit. CSF classes can be extended to provide more domain specific community schedulers and support many different kinds of grid deployment models. Examples of grid level scheduling algorithms include scheduling across multiple clusters within a virtual organization, co-scheduling across multiple resource managers, scheduling based on SLAs, and economic scheduling models. By making use of the open source CSF, grid scheduling implementations ensure that they interact with resource managers using standard interfaces without needing to know all of the underlying details of the specification or having to implement the protocol themselves.

C entra lM anager

S ubm itM achine

C heckpo in tD ata

C ondor C o llectorC ondor N egotia tor

E xecutionM achine

Note: For more informtion about CSF, please refer to the following Web sites:

http://www.platform.com/Products/Platform.Globus.Toolkit/Product.Information/Features.Benefits.htmhttp://sourceforge.net/projects/gcsf/


http://www.platform.com/Products/Platform.Globus.Toolkit/Product.Information/Features.Benefits.htm

http://sourceforge.net/projects/gcsf/

The CSF is composed of several features:

� Round-robin job scheduling � Advance reservation booking, query, and control � Reservation-based job scheduling � Job throttling support for increased reliability � File staging support

3.4.13 Maui SchedulerThe Maui Scheduler is a job scheduler that allows the administrator to have broad control over the different ways that jobs are submitted for scheduling. It also supports features such as advance reservation, Quality of Service (QoS) levels, backfill, and allocation management.

It has an advanced scheduling mechanism, with backfilling, and it allows jobs with a lower priority to execute prior to a higher priority only if it does not delay the start of a prioritized job.

Advance features that Maui adds to a scheduler include:

� Extensive job priority policies

� Job advance reservation support

� QoS Support

� Meta-scheduling Interface

� Different types of fair share policies

� Different types of back filling policies

Scheduling versus Advance ReservationThere are many types of scheduling algorithms, and we often need a way to reserve resources in advance. This is specially true when a single demanding job needs to be executed and the grid is out of available resources: this calls for starvation. To solve this problem, some schedulers have implemented a way to stop and hold smaller jobs so that the starving job can be started. Looking at this algorithm, it will affect all the jobs in the queue, not just the starving large jobs.

Note: Maui Scheduler is an open source software that is used to implement an advanced reservation HPC parallel batch scheduler. For more information, please refer to the following Web site:

http://sourceforge.net/projects/mauischeduler



To tackle this same problem, Maui implemented an algorithm called advance reservation, which is able to allocate resources in the future, just as though you were looking through a calendar and booking time.

Besides using Maui, starvation can also be resolved using a preemptive job scheduler, but this is not an easy task, as this implies a rather precise performance prediction schema.

3.5 Architecture examplesIn this section, we present some examples of the use of schedulers. As shown in the earlier sections, there are quite a few types of schedulers, with different applicability and functionality.

3.5.1 Desktop gridA research company “xyz” indicated that the computing power they have in the lab is insufficient for the massive jobs they run. Their research focuses on DNA data processing and the evolution of cancer. While there are many software packages able to handle this computational need, they lack the scalability to grow to millions of organisms that the researchers are looking forward to analyzing.

To meet these requirements, the proposed solution architecture must be able to scale well and the data should be split into multiple small pieces for processing before joining back together for analysis and results.

Current environmentThe researchers are currently running their application on a dedicated multi-processor machine, which is insufficient to achieve effective results on the jobs. Also, the jobs they are running often “hog” the computing resources in the lab, which is a shared facility. However, there are a number of student computer labs that are idle at night.

Proposed grid environmentIt is proposed to enable the application on the desktop computers of the company as well. The application will then use United Devices’ Grid MP product as the grid middleware to manage, submit, and collect the jobs.

The proposed Desktop grid environment, shown in Figure 3-5 on page 60, uses the United Devices grid framework for scheduling, monitoring, and jobs coordination across all the desktops.


Figure 3-5 Proposed Desktop Grid setup

3.5.2 Server gridThe “xyz” Corporation needs to leverage grid computing technology to maximize their computing resources on site to better perform research and analysis. They would like to poll all their servers together and expect to use a scheduler to performance load balancing and allocation of the available computing resources.

Current situationIn the current environment, the clients have the following infrastructure that they use as a file server, printer server, workstations, and others.

The current IT environment in this organization is as follows:

� Workstations running Linux all over the organization.

� NFS is used as a file sharing backbone.

� User programs are written in a mixture of C, C++, and FORTRAN, and mostly compiled under the GNU Compiler Collection (GCC).

Agent Agent

Agent

Agent Agent

Agent

Agent Agent

Agent Agent

Open MPServer United Devices

JobResults

LabLab

Lab

Job Split


� There are future plans to deploy applications such as Fluent, abacus, and so on.

In the current environment shown in Figure 3-6, the researchers in “xyz” Corporation need to access the system via telnet or ssh to submit a job. In case the server is busy or high in workload, the users log into another server. Once a suitable server is identified, the user will run the job with the corresponding data files available through the NFS.

Figure 3-6 Current environment

Pain of the current infrastructureAt a glance, we can identify that there are some areas of improvement in the current infrastructure. The researchers are spending much of their time identifying a suitable machine for their job, instead of focusing on their core competencies.

� Researchers are spending precious time finding a machine to execute a job.

� Security concerns, with so many users logging in all the servers.

� Management control, with the growing number of workstations.

� Users with urgent jobs do not get higher priority over the resources.

50. . . . . . . . .

Telnet/SSH

Researchers

Manual QueryJob Submission


Proposed solutionThe “xyz” Corporation’s IT team plans to use a grid-enabled scheduler to create a virtualized compute environment for the researchers to use to submit jobs so that they see the resources as a single pool.

Components of grid The grid consists of a number of different components that are suitable for a wide range of specific needs, but in this case, we first propose the addition of scheduling software, possibly OpenPBS.

In selecting a suitable scheduler for “xyz” Corporation, we need to evaluate some basic requirements for the solution. As we know, the basic feature of a scheduler is to be the “traffic” controller of all jobs.

The basic features that this scheduler must provide are:

� Queue a job that a user submits

� Evaluate the job requirements

� Select the appropriate host to run the jobs

� Dispatch the job to a suitable machine to run

� Monitor and inform the user when the job is successfully completed

Also, by placing all the various machines under the control of the scheduler, it automates the task of job dispatching, saving time and optimizing the use of hardware.

AdministrationBy placing the machines under the administration of the scheduler, the selection of servers/workstations to perform a certain task is now left to the scheduler. Users will also not be required to have access to all the servers to run the jobs. The scheduler will be running the job on the user’s behalf.

Hardware and softwareAs part of the solution, we propose an additional server be installed with the job scheduler software. OpenPBS is chosen, as it meets the “xyz” Corporation requirements and it also provides additional features, such as multiple queue support. This will enable the “xyz” Corporation to manage the jobs more efficiently, breaking their priority jobs from the lesser.

Also, OpenPBS has the capability for file staging, providing users with the ability to specify any files that need to be copied onto the execution host before the job runs, and any that need to be moved out after the job completes. The job will be scheduled to run only after the required files have been successfully transferred.


AdvantagesBased on the proposed solution for “xyz” Corporation, as shown in Figure 3-7, they could now harness and use all their computing power and resources:

� Researchers could now focus on their research and not look for servers to submit their job.

� Job scheduler will automatically run the jobs based the priority assigned.

� All the computing resources in the organization could be harnessed for the computing power.

� The solution is scalable, and additional nodes can be easily added to the schedulers.

� Better administration work, with less user access (such as telnet/ssh) into the systems.

Figure 3-7 Proposed Architecture for “xyz” Corporation

Tip: Altair PBS Professional, a workload management solution suitable for deployment in the most demanding production HPC environments, is also a good option to be used in this example.

ServerOpen PBS

Job Scheduler

Researchers/SSH


Future roadmapWith the current solution, “xyz” Corporation has effectively created virtualized computing power. This is a major step toward the creation of a virtualized environment. Some of the possible extensions to the current solution are as follows:

� Data Federation

� Web Portal for submitting jobs

A Web Portal will give the users the capability to submit jobs and manage the environment via Web services. Using this service, the users effectively have an interface to select the type of jobs, the platform, and the priority of the jobs. For more information about these products, see Chapter 2, “Web services core/hosting” on page 23.


Chapter 4. Information virtualization

The following topics are discussed in this chapter:

� Types of information infrastructure

� Overview of products available for this discipline

� Mapping user requirements to the type of information infrastructure

4


4.1 IntroductionInformation virtualization is the process by which data and information stored in distributed and unrelated devices is managed as a single and concise set of data. As such, virtualization can naturally be accomplished by data grids, as the distributed and heterogeneous nature of such grids make them suitable for unifying decentralized resources based on distinct technologies to provide the final user with a single vision of the data.

The virtualization process can take place in different levels of abstractions and, as such, data is treated in this context as a multiple-layer concept. The next section describes the data layers that we consider in this chapter and presents basic products that provide data virtualization functionality.

4.2 Information virtualization layersFigure 4-1 presents the three basic levels of virtualization typical found in an information infrastructure. The next sections briefly describe each one of them.

Figure 4-1 Virtualization layers

Attention: We use the term virtualization here in the sense of aggregating distinct entities to become one virtual entity. Other subdivision of layers can be defined depending on the architectural blueprint been used.

InformationInformation

File SystemFile System

Block DataBlock Data

Res

ourc

es


4.2.1 Block data virtualization layerBlock data infrastructure for grids provides unified access to chunks or blocks of data. As presented in Figure 4-2, this layer provides an abstraction over how raw data is stored on independent storage devices, offering operating systems with a unified vision of such data.

The main advantages of such independence between computers and storage devices are that it:

� Reduces the complexity and costs of managing storage

� Increases the utilization of existing capacity

� Centralizes the management of multiple storage controllers

The layer between computers and storage devices hide the complexity of storage and helps to increase the utilization. It also helps to reduce the costs of managing by centralizing the management of the multiple storage devices or subsystems.

Figure 4-2 Block data virtualization layer

4.2.2 File system virtualization layerThe file system infrastructure follows the main principle of the data block infrastructure but at a higher level, providing unified access to files, as presented in Figure 4-3 on page 68.

A number of solutions have been created to handle the problem of remote data access in homogeneous and heterogeneous environments. Network file systems, for example, are a convenient solution for sharing data across computer nodes and a typical example is the NFS protocol. Such file systems make possible the access to remote files in a networked environment as though they were locally available.

Storage Devices

Block Data Virtualization

Chapter 4. Information virtualization 67

Figure 4-3 File layer virtualization

4.2.3 Information virtualization layerInformation is usually defined as “meaningful data”, from the perspective of the end user. This layer provides an abstraction over disparate and distributed information sources, such as a Database Management System (DBMS), flat files (for example, comma-separated files), structured files (for example, XML documents), or a Content Management System.

The information virtualization layer also refers to the ability to federate or integrate data and information from heterogeneous resources into a unified repository. The whole idea is to present a single view of the information (see Figure 4-4).

Figure 4-4 Information virtualization layer

Some parallel databases are usually regarded as information virtualization tools, but in general, these products only support their own branded engines, and as such cannot be considered grid-enabled products.

File System Virtualization

File Systems

DBMS Flat File StructuredFile

ContentManagement

System

Information Virtualization


4.3 Requirements and productsIt is one of the IT architect’s role to choose the right products according to the solution’s requirements. In this section, we discuss how to depict the main requirements that a solution may require and which products are available for providing such requirements.

4.3.1 Defining requirementsIn the context presented above, answering the following questions is imperative:

Where is the data stored?Data to be accessed is stored on a single remote system or distributed across multiple networked systems.

How heterogeneous is the data source?The data to be accessed is stored on different types of source systems (for example, DBMS, Content Management Systems, and flat files). It also encompasses data coming from sources on different hardware bases and from the same "kind of" source, but provided by distinct products, and possibly from different vendors.

What are the data formats?Data types and formats used by the source systems differ in format and type.

What are the data semantics?Data that is owned by different domains (for example, business units) might differ in semantics (for example, homonym for entity customer).

How is the network performance?Bandwidth with a given latency.

How are the security policies?Ownership of data and procedures for authentication and authorization differs among source systems.

Attention: it is natural to think that information virtualization software provides virtualization in the two lower levels, as well as that file system virtualization software provides data-block virtualization. However, this is not always true, as we can see in most distributed database implementations.


Once a better understanding of the problem is acquired, the architect should define the layer of abstraction in which the solution is best fit. This is the first step when choosing the products to adopt in a particular implementation, as information, file system and data block products are usually part of unrelated families of products. In some cases, multi-layer solutions might be required. Some examples of solutions for each layer are:

� Information virtualization layer

Distinct servers could be the repository of data, for example databases, images, videos, or audio streams. Using IBM WebSphere Information Integration framework, we could federate the different data into a single database. A federated database creates references or links to the actual data. End users and applications will have a single view of all the different type of data.

� File system virtualization layer

A solution that has servers that see several storage devices as though they were one device. The servers could share out some of its folder to another department using a protocol such as NFS.

� Block data virtualization layer

A solution that has servers that see several storage devices as though they were one device. The servers are connected to storage managed by SAN Volume Controller, which manages file systems on multiple SAN storages.

The examples described above are not mutually exclusive and can be combined if the solution requires it.

Products overviewFigure 4-5 provides an overview of some Information virtualization products in each layer.

Figure 4-5 Example for a product overview

At the File System Layer GPFS, NFS Version 4, SAN File System, Network Attached Storage (NAS), and Andrew File System (AFS®) provide an abstraction

DB2 Universal Database

NFS V4

Tivoli Storage Resource Manager

SAN Volume ControllerBlock data

File System

Information

GPFS

SAN File System

IBM® WebSphere® Information Integration

DB2 Universal Database

NFS V4

Tivoli Storage Resource Manager

SAN Volume ControllerBlock data

File System

Information

GPFS

SAN File System

IBM® WebSphere® Information Integration


for the underlying files system. It shields user and application from the complexity that is associated with accessing remote data that might be stored in disparate file systems.

At the block data layer, Tivoli Storage Resource Manager and IBM SAN Volume Controller provide an abstraction of storage devices. It permits a level of indirection between storage devices, as seen by computers and storage devices exported by storage systems.

4.3.2 Block data layer productsAt the block data layer, as shown in Figure 4-6, Tivoli Storage Resource Manager and IBM SAN Volume Controller TotalStorage® provide an abstraction of storage devices.

Figure 4-6 Block data layer

Storage Area NetworkSANs originated to overcome the problems with network attached storage (NAS) devices, which - like ordinary servers - are difficult to manage and difficult to expand in capacity. SAN is made up of a number of fabric switches connected in a network. The most common form of SAN uses the Fibre Channel fabric protocol (with Fibre Channel switches). Connected to the SAN will be one or more disk array controllers and one or more servers. The SAN allows the storage space on the hard disks in the disk array controllers to be shared among the servers.

Clients can use SAN capabilities to reduce the complexity that is associated with managing large amounts of (probably) heterogeneous storage devices. Furthermore, it provides them with greater flexibility in allocating storage space.

Several products are available, with different virtualization capabilities.

Block Data LayerBlock Data LayerRes

ourc

es


SAN Volume ControllerThe SAN Volume Controller (“TotalStorage SAN Volume Controller” on page 193) (SAN VC) is a storage solution that enables SAN attached storage to be managed as one large pool rather than many individual islands of capacity, thus allowing clients to increase utilization of these storage resources.

With SAN VC, users can allocate their storage to applications based on application requirements instead of constraints in subsystem connectivity and functional capabilities. Besides this function, it also provides a centralized storage management capability.

This central point of control will increase storage administrator productivity by more flexible disk management, for example, dynamic creation and expansion of logical volumes. Also, the SAN VC provides a common platform for advanced copy services, such as FlashCopy® and Remote Mirroring that allow source and target volumes to be located on heterogeneous hardware platforms. With powerful data migration facilities as part of the base virtualization software, SAN Volume Controller can be added into existing SAN environments without requiring data movement from current physical locations. Once in place, transparent migration of data is supported (that is, data typically can be moved without interrupting host applications).

The SAN Volume Controller is a storage solution integrating specialized virtualization software with clustered hardware engines. The solution uses customized IBM xSeries® servers running an operating environment based on a Linux kernel. Industry standard host bus adapters (HBAs) will interface with the SAN fabric. Because the SAN VC is designed to work in an open environment with heterogeneous disk systems, clients will be able to deploy flexible storage solutions without concern for proprietary architectures. The flexible nature of the SAN Volume Controller enables clients to grow in performance and capacity according to their needs.

IBM Tivoli Storage ManagerIBM Tivoli Storage Manager protects an organization's data against hardware failures and other errors by storing backup and archive copies of data in offline storage.

Note: SAN Volume Controller is part of the IBM TotalStorage Open Software Family. The TotalStorage Open Software family is a set of comprehensive and flexible storage software solutions that can help enterprises address their storage management challenges today and is designed to help enterprises take steps toward evolving to an On Demand storage environment. For more information, please refer to the following Web site:

http://www.ibm.com/servers/storage/software/virtualization/svc/



It can scale to protect hundreds of computers, from laptops to mainframes, running a dozen different operating systems, connected via the Internet, WANs, or LANs. The centralized Web-based management, smart data move and store techniques, and comprehensive policy-based automation work together to minimize data protection administration costs and the impact on both computers and networks. Optional modules enable business-critical applications that must run 24x7x365 to utilize IBM Tivoli Storage Manager centralized data protection with no interruption to their service. The features of IBM Tivoli Storage Manager are as followed:

� Backup

Provides the ability to back up numerous versions of files to the hierarchy of storage. In case of a failure or someone accidentally deletes a file, the chosen version of the data can be easily restored. It supports also progressive backup.

� Tape resource sharing

Enables multiple Tivoli Storage Manager servers to use the same tape library and drives. This improves tape hardware asset utilization, recovery performance, and tape hardware asset utilization.

� Network-free rapid recovery

Supports high-speed client data recovery directly from a tape or CD-ROM. This minimizes recovery time by eliminating the use of network and central services resources.

� LAN-free data transfer

Effectively exploits SAN environments by moving back-end office and IT data transfers from the communication network to a dedicated data network or SAN. Internet Protocol (IP) communication bandwidth can then be used to improve service levels for end users and clients.

Note: IBM Tivoli Storage Manager is part of the IBM TotalStorage Open Software Family. The TotalStorage Open Software family is a set of comprehensive and flexible storage software solutions that can help enterprises address their storage management challenges today and is designed to help enterprises take steps toward evolving to an On Demand storage environment. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/storage-mgr/


http://www.ibm.com/software/tivoli/products/storage-mgr/

� Dynamic multi threaded transfer

Permits multiple data transfers, to and from the same Tivoli Storage Manager server. This new feature boosts performance backups to more than three times faster than the rate of a single-threaded session. This speed is achieved because the number of IBM Tivoli Storage Manager data transfer sessions is transparently optimized based on available system resources.

� Adaptive differencing technology

Changes the way data is transferred throughout the enterprise. Data is transferred by byte, block, or file level, based on data size and network characteristics. This newly patented technology supports a variety of connectivity strategies, including LANs, WANs, SANs, Internet, and dial-up connections. Adaptive Differencing technology is designed for mobile computer users and other users with a need to minimize data transmitted over the network.

� Enterprise administration

Simplifies centralized control across multiple Tivoli Storage Manager implementations without sacrificing network performance. This enables high-performance backups to locally attached storage devices using a minimum of network resources.

IBM Tivoli Storage Resource ManagerIBM Tivoli Storage Resource Manager is a comprehensive capacity management solution for heterogeneous storage environments. It includes enterprise wide reporting and monitoring, policy based management, and automated capacity provisioning for Directed Attached Storage (DAS), Network Attached Storage (NAS), and Storage Area Network (SAN) environments..

Note: IBM Tivoli Storage Resource Manager is part of the IBM TotalStorage Open Software Family. The TotalStorage Open Software family is a set of comprehensive and flexible storage software solutions that can help enterprises address their storage management challenges today and is designed to help enterprises take steps toward evolving to an On Demand storage environment. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/storage-resource-mgr/


http://www.ibm.com/software/tivoli/products/storage-resource-mgr/

And because it is policy-based, through its autonomic self-healing capabilities, it can detect potential problems and automatically make adjustments based on the policies and actions you have established. Tivoli Storage Resource Manager can help you:

� Maximize storage return on investment

With Tivoli Storage Resource Manager, you can leverage and optimize your existing storage resources and perform storage management with a high level of control.

� Maximize storage utilization

Tivoli Storage Resource Manager lets you find unused space and reduce wasted space by identifying outdated or nonessential data.

� Effectively manage more storage with the same staff

Because Tivoli Storage Resource Manager automatically collects and reports on storage assets according to the policies and thresholds you set, you spend less time diagnosing and fixing storage problems. Automatically taking corrective action based on policies can help administrators sustain the health of your storage infrastructure to maintain application availability through autonomic self-healing behavior

4.3.3 Files system layer productsAt the file system layer, as shown in Figure 4-7, GPFS, NFS Version 4, and SAN File System provide an abstraction of the underlying files system. It shields users and applications from the complexity that is associated with accessing remote data that might be stored in distinct file systems.

Figure 4-7 File system layer

File System LayerFile System Layer

Res

ourc

es


NFS Version 4The Network File System Version 4 is a distributed file system based on the previous releases with a more robust error recovery and a new transport protocol for heterogeneous environment.

It offers the following features:

� Strong security and a new protocol for file locking, supporting sophisticated security mechanisms. These mechanisms include Kerberos 5 and SPKM3, in addition to traditional AUTH_SYS security. A new API is provided to allow easy addition of new security mechanisms in the future.

� File delegation: The NFS Version 4 client can access and modify a file in its own cache without sending any network requests to the server, until the server indicates via a callback that another client wishes to access a file. This reduces the amount of traffic between clients and server considerably in cases where no other clients wish to access a set of files concurrently.

� NFS Version 4 introduces protocol support for file migration and replication.

IBM General Parallel File SystemIBM General Parallel File System (“GPFS” on page 174) provides global access to files.

Files created in GPFS can be accessed from every node that runs GPFS code. For those nodes not running GPFS, files will still be accessible by using NFS. A parallel file system not only offers performance advantages by eliminating the limitation of a single server for file services, but also offers a great deal of flexibility. With a parallel file system, since all nodes can “see” the file system, it is easier to move applications from one node to another. GPFS also has the ability to define data groups that are located under the same “single point of failure” unit. This allows the data access path to be automatically routed if any of the data

Note: The NFS version 4 protocol is a further revision of the NFS protocol version 2 (RFC1094) and 3 (RFC1813). For more information, please refer to the following Web sites:

http://www.ietf.org/rfc/rfc3530.txthttp://www.nfsv4.org/

Note: The IBM GPFS, originally developed for SP clusters, can today provide fast data access from all nodes in a cluster of IBM UNIX servers. For more information, please refer to the following Web sites:

http://www.ibm.com/servers/eserver/clusters/software/gpfs.htmlhttp://www.ibm.com/servers/eserver/pseries/software/sp/gpfs.html


http://www.ietf.org/rfc/rfc3530.txt


http://www.nfsv4.org/

http://www.ibm.com/servers/eserver/clusters/software/gpfs.html

http://www.ibm.com/servers/eserver/pseries/software/sp/gpfs.html

groups should fail. The file system is also able to bring itself down to read-only status to protect data integrity if required. This is especially valid for high availability solutions, if unrecoverable errors occur in the storage servers.

Sharing the same file system among several nodes has the benefit of increasing the maximum I/O bandwidth that otherwise would be limited by the maximum local I/O bandwidth of the single server.

SAN FSSAN FS (“Tivoli SAN FS” on page 189) provides file system virtualization that can help reduce complexity by treating storage resources as a single, common pool. The design of SAN FS is also meant to support a heterogeneous environment.

The meta-information about the files, such as file location, security, and the locking mechanism, are managed by the SAN FS. In many traditional file systems, the metadata resides within individual servers, which can create limitations in sharing and accessing data across servers or across file systems. By managing the metadata on the storage network using a metadata server, instead of within individual application servers, the design of the SAN File System helps move intelligence out of individual servers onto the storage network, so that it can be available to any application server in the network. Some of the relevant features are:

� Provide central point for data and storage management, with a single file system with consistent policies, management, and monitoring capabilities.

� Storage pools can be designed to represent differentiated classes of service for different business areas: a critical storage pool can have the data stored in the most expensive storage while the staging data can be kept in an older disk.

� SAN FS allows you to manage physical storage as a pool, with options to transparently add, remove, and change the application servers.

Note: SAN FS is part of the IBM TotalStorage Open Software Family. The TotalStorage Open Software family is a set of comprehensive and flexible storage software solutions that can help enterprises address their storage management challenges today and is designed to help enterprises take steps toward evolving to an on demand storage environment. For more information, please refer to the following Web site:

http://www.ibm.com/servers/storage/software/virtualization/sfs/


http://www.ibm.com/servers/storage/software/virtualization/sfs/

So how does it really work? There is a client code that needs to be running on the application servers, and interacts with the SAN File System metadata server to retrieve the metadata. Once the client code receives the metadata, it then accesses the file data directly through the SAN. No changes to the applications need to be made.

4.3.4 Information layer productsAt the information layer, as shown in Figure 4-8, IBM WebSphere Information Integration and DB2 UDB provide an abstraction over disparate and distributed data sources, such as a Database Management System (DBMS), flat files (for example, comma separated file), structured files (for example, XML documents), or a Content Management System.

Figure 4-8 Information layer

WebSphere Information Integration FamilyWebSphere Information Integration provides a strategic framework that gives companies real-time, integrated access to business information: structured and unstructured, public and private, and mainframe and distributed.

Federation of data in this sense means that end users and applications are able to access any data, in any format, on any heterogeneous platform. It also allows end users and applications to query and manipulate different types of data, organized in a structured database format, all at the same time

In addition to federation capabilities, WebSphere Information Integration family includes replication, enterprise searching, and publishing capabilities.

In short, the WebSphere Information Integration family provides:

� Data federation: The ability to provide unified access to different repositories of information running in a heterogeneous environment. It also allows the application to connect to other vendor relational databases, document databases like Domino, XML, flat files, digital media, and archiving databases, among others.

Information LayerInformation LayerR

esou

rces


� Replication: The ability to replicate content from foreign repositories to local repositories and vice-versa. A solution could require that functionality mostly in cases where there is WAN communication between the consumers and the repositories. A local replica of the content would provide better response time and less usage of the bandwidth.

The following WebSphere Information Integration products are fundamental when building an information layer in a grid solution:

� WebSphere Information Integrator

It provides the foundation for a strategic information integration framework, enabling the federation and virtualization of information in an organization (see “WebSphere Information Integrator” on page 199).

� WebSphere Information Integrator Content Edition

It enhances the ability to virtualize unstructured data, helping applications access and work with a broad range of unstructured information sources.

� WebSphere Information Integrator OmniFind™ Edition

It provides enterprise search middleware for powering intranets, extranets, and corporate public Web sites.

DB2 Product FamilyDB2 Universal Database™ is the database management system that delivers a flexible and cost-effective database platform to build robust on demand business applications. DB2 UDB further leverages your resources with broad support for open standards and popular development platforms, such as J2EE and Microsoft® .NET. The DB2 UDB family also includes solutions tailored for specific needs, such as business intelligence and advanced tooling. Whether your

Attention: Do not confuse WebSphere Information Integration with WebSphere Information Integrator.� WebSphere Information Integration is the family of products and the

solution portfolio.

� WebSphere Information Integrator is the product and the foundation for the information integration framework.

Note: The WebSphere Information Integration family provides a framework that includes data and content federation, replication, event publishing, and enterprise search. For more information, please refer to the following Web site:

http://www.ibm.com/software/data/integration/



business is large or small, DB2 UDB has a solution built and priced to meet your unique needs.

Database partitioning feature (DPF)DPF is a feature to enable scaling of a single database to more than one server. It offers the necessary scalability to distribute a large database over multiple partitions (logical and physical) using a shared-nothing architecture. DPF can be beneficial for environments consisting of more than one server.

DPF partition consists of its own data, indexes, configuration files, and transaction logs. A partitioned database is a database with two or more partitions. Tables can then be located in one or more database partitions. Processors associated with each database partition are used to satisfy table requests. Data retrieval and update requests are decomposed automatically into sub-requests, and executed in parallel among the applicable database partitions.

DB2 Content ManagerThe IBM DB2 Content Manager (“DB2 Content Manager” on page 173) is an enterprise content management engine for images, graphics, video, Web content, and compliance documents. It provides digital asset management and content integration.

Note: For more information about the DB2 Product Family, please refer to:


Note: A client who has purchased and installed DB2 Universal Database Enterprise Server Edition (ESE) can create a partitioned instance by using the database partitioning feature (DPF). More information about DPF can be found at:

http://www.ibm.com/developerworks/db2/library/techarticle/dm-0504mcarthur/http://publib.boulder.ibm.com/infocenter/db2help/index.jsp?topic=/com.ibm.db2.udb.concepts.doc/concepts/about_dpf.htm

Note: DB2 Content Manager is part of the DB2 Content Management family of products, which helps manage, share, integrate, and deliver critical business information on demand. For more information, please refer to the following Web site:

http://www.ibm.com/software/data/cm/


http://www-306.ibm.com/software/data/db2/

http://publib.boulder.ibm.com/infocenter/db2help/index.jsp?topic=/com.ibm.db2.udb.concepts.doc/concepts/about_dpf.htm


http://www.ibm.com/developerworks/db2/library/techarticle/dm-0504mcarthur/


DB2 Information Integrator for ContentIBM DB2 Information Integrator for Content is an information integration infrastructure that gives applications access to sources from inside and outside the enterprise.

From a Web browser or a portal, knowledge workers can concurrently access data and content from multiple sources, expanding the reach of their analysis and improving their productivity. DB2 Information Integrator for Content provides:

� A 'federated search', direct connectors, and simultaneous access across the multiple repositories.

� IBM Lotus Extended Search for scalable, distributed, parallel searches - across Domino and Microsoft mail, applications and indexes, LDAP, File system and Web content - returns results to a single Web browser interface.

� Advanced Web crawling and information mining.

� Workflow integration for items accessed by Information Integrator for Content.

� An application development toolkit with a consolidated set of APIs for rapid portal application.

� Broad platform support.

� Broad support for relational databases.

4.4 Case studyThe “xyz” Corporation has engaged their engineers to plan, design, and implement a solution to enable them to share data between the different departments. The “xyz” Corporation is a silicon design company that has an heterogeneous computing environment. Each department has its own data repository spread over several countries. The CIO has decided to have a common IT budget for the company and it should share information across departments by grid technologies. The core business today runs on top of the UNIX operating system and each department has its own server that runs the

Note: DB2 Information Integrator for Content is part of the DB2 family of products. DB2 Information Integrator for Content provides a strategic framework to help clients speed new application deployment and control IT costs. It provides a framework that includes data and content federation, replication, event publishing, and enterprise search. For more information, please refer to the following Web site:

http://www.ibm.com/software/data/eip/


http://www.ibm.com/software/data/eip/

Windows®-based operating system. Additionally, there are several Linux servers supporting the engineering and manufacturing department.

4.4.1 Problem overviewEach of the departments in the countries that the company operate in does its own research and stores their findings and information locally. Users from another country can only gain access to the data by asking their colleague in the other country for it, so there is redundancy of work at times.

At the management level, the CIO wants to evaluate the use of a virtualization technology to improve the research time and achieve higher resource utilization. Information grids support heterogeneous platforms and provide data sharing and large-scale collaboration. Figure 4-9 illustrates the current infrastructure that “xyz” Corporation has.

Figure 4-9 “xyz” Infrastructure

4.4.2 Grid solution planningBefore we put the data grid solution together, let us evaluate the findings that we got from the client and categorize them into “Current Environment” and “Challenges”.

F ac ility A

F ile F ile D BD B

F o lde rD BD B

F ile F ile D BD B

Fo lde r D BD B

F ile F ile D BD B

D BD B

Fac ility C

Fac ility B

N e tw ork


Current environment� Heterogeneous operating environment consisting of UNIX, Linux, and

Windows-based operating systems.

� Environment is distributed, with branches over multiple locations having their own data repositories.

� Each of the departments has their own IT budget and thus there are heterogeneous storage solutions: SAN, NAS, and directed attached storage.

Challenges� Engineers working on similar projects are located in distinct sites, but the

research repository data is only available to the main team.

� Access to data today is performed on a case-to-case basis, and data sharing between different research center is low.

� The file sharing protocols are different: The UNIX and LINUX machines access shared files using Network File System (NFS) and the Microsoft based machines access shared files using the Server Messages Block (SMB) protocol.

Having the above scenario in mind, IT engineers assigned by the CIO will work together to come up with a solution and implement it as a pilot data grid project to share information between two of “xyz” Corporation’s branches. An information grid in this case will enable engineers and other departments to share and view data regardless of where it is located or resides. It will also provides access to heterogeneous files, databases, and storage systems, and shares files through multiple protocols.

4.4.3 Federation of data Looking at the success of optimizing the infrastructure using a data grid, the CIO of “xyz” Corporation is now looking at building a knowledge portal for sharing information across the company. With this portal, the CIO also aims to give users the ability to search for data from multiple data sources regardless of the location and data type.

Current environment� There are multiple databases deployed throughout the corporation.

� There are also documents in text files and images.

� Users have to go through each database to search for files and information, and there is no universal data repository of the information.


IT engineers from “xyz” Corporation evaluated multiple products. The team is looking for a product that can federate data that they have without having to change the database platform. The existing infrastructure should remain and the portal should be present as an interface for the users to query the data.

After evaluation, the team decided to go for WebSphere Information Integration, as it allows them to federate data from structured and unstructured data into a single database. Therefore, for the pilot project, the team has decided to federate their databases.

The current final solution is shown in Figure 4-10.

Figure 4-10 Federated using WebSphere Information Integration

File File DB2DB2 MySQLMySQL

Facility B Facility CFacility A

Virtualized-NFS, SMB, SQL Access

WebSphere® Information Integration

OracleOracle DBDB


Chapter 5. Other disciplines


� Provisioning and orchestration

� Security

� Directory Services

� Licensing

� Billing and metering

5


5.1 IntroductionWhen building a grid infrastructure, there are other components that we should take into consideration at the design stage. The components will directly affect the future operation of the infrastructure. Some of the components are:

� Provisioning and orchestration

� Security

� Licensing

� Monitoring

� Billing and metering

5.2 Provisioning and orchestrationProvisioning is responsible for allocating new machines automatically while orchestration is responsible for provisioning resources in a time line (or along the time line). Orchestration is dynamic and on demand, and adapts to the real-time requirements of the enterprise. For example, provisioning can automate a task to install a server; orchestration can automate a task to put all the right provisioned resources in the right place in the right time.

It is quite usual that, on typical IT environments, workload varies broadly, with some peaks during the night, others during the day, and so on. You need to provision servers, storage, and infrastructure in order to support the highest workload. It means that these servers, storage, and IT infrastructure are going to be highly underutilized when those applications are not at their peak. Additionally, the processes for deploying these servers and the underlying software are mainly manual. This also creates another problem: The ability of a company to respond to changes is relatively slow because of the manual processes that are used to deploy and provision an application, or add capacity to a server farm, or to accommodate the increased utilization and peak workloads that you see.

ProvisioningProvisioning handles the coordination and allocation of managed resources, such as servers or storage within an on demand environment.

These resources are provisioned dynamically since they are required for satisfying the needs of a business application. Provisioning is done by allocating managed resources from a “resource pool". This gives the IT organization the flexibility to dynamically respond to the needs of the business. Resources can be dynamically allocated from the pool, then de-allocated and returned to the pool


after the resource is no longer needed for a specific business application. Once de-allocated and returned to the pool, the resource then becomes readily available to be allocated to another application, where resources might be needed to ensure that the application’s performance goals and commitments are met.

In addition to provisioning hardware resources, operating systems images, middleware (such as Web applications), server software, and DB software can be managed by the provisioning component of the framework. The provisioning component provides the "what to do" piece of automation to ensure that a newly provisioned resource is made fully usable to support the business goals. Provisioning and workload management work together to provide resources when needed. Given a pool of servers, the provisioning component will attempt to maximize the effectiveness of the pool by increasing or decreasing the number of servers assigned to a given cluster.

OrchestrationOrchestration extends the provisioning model by providing a sensing, capacity planning, and responsive software application to dynamically meet service level agreements. Because of this sensing and responsive software architecture, orchestration enables dynamic and automated allocation of server capacity on demand. Intelligent Orchestrator, by dynamically allocating capacity to applications that require it, improves utilization of underutilized computing systems. Orchestrator reallocates existing server capacity to support unpredictable application demands. It improves application service levels without investing in additional capacity. It increases the value of IT server investments by making server capacity available to all applications under its management. Using Intelligent Orchestrator, a data center can accurately predict impending capacity requirements, and facilitate dynamic infrastructure reallocation of server and network resources. Orchestration initiates the activities necessary to automatically maintain server availability and meet service level agreements. It provides the why, where, and when of a complete solution.

WhyBy monitoring the applications under its control, an orchestrator senses degrading performance and determines the actions that have to be taken.

WhereBecause solutions are monitored closely, an orchestrator determines where a server needs to be deployed and automatically instructs the provisioning function to deploy a server to the proper cluster and configure all the network attributes to keep the solution within adequate service levels.

Chapter 5. Other disciplines 87

WhenBy means of its capacity management capabilities, an orchestrator predicts when resources will become available or be needed. It starts the provisioning process just-in-time to meet an application’s growing or decreasing workload requirements.

5.2.1 Provisioning and orchestration productsIBM has a complete solution environment for provisioning and orchestration. The product family is composed IBM Provisioning Manager and IBM Tivoli Intelligent Orchestrator.

IBM Tivoli Provisioning ManagerThe systems provisioning capability introduces the concept of a pool of resources that are shared between different workloads. Tivoli Provisioning Manager provides the necessary tools for making machines available in a broad, grid-enabled infrastructure. Provisioning of systems is a sequence of activities that need to happen in a specific order. The activities have dependencies on previous activities. Typical activities include installing operating systems, remotely booting networks, configuring networks such as Virtual Private Networks (VPN) and storage environments like Storage Area Network (SAN) or Network Attached Storage (NAS), and "deprovisioning" resources that are no longer needed back into an available pool.

Automating the execution of these processes is a huge savings in terms of staff, enabling the leveraging of the highly skilled staff for other tasks more aligned to your business strategy, rather than continuously repeating traditional tasks.

IBM Tivoli Intelligent OrchestratorIBM Tivoli Intelligent Orchestrator (“Tivoli Intelligent Orchestrator” on page 180) enables staff to sense and respond to changes in the environment so that the infrastructure meets the service level commitments.

The IBM Tivoli Orchestration solution makes it possible for the staff to monitor the response times that clients are actually getting from the business applications. If the system senses that is about to break that minimum required service level, it will direct IBM Tivoli Provisioning Manager to provision more IT resources to support the application that those clients are using. IBM Tivoli Intelligent

Note: IBM Tivoli Provisioning Manager now enables on demand computing across the entire data center through server, storage, and network automation. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/prov-mgr/



Orchestrator does this automatically and is fully integrated with IBM Tivoli Provisioning Manager. When IBM Tivoli Intelligent Orchestrator perceives that the response time is back in the normal range, it can actually deprovision some resources, so that those excess or underutilized resources can then be redeployed and used somewhere else in the site as appropriate.

5.2.2 Provisioning and orchestration case studyThis case is based on a hypothetical Internet banking application. Grid environments sites are spread out across the multiple bank locations. in Figure 5-1, we have a typical IT infrastructure of an Internet bank system.

Figure 5-1 A typical Internet banking infrastructure

Note: IBM Tivoli Intelligent Orchestrator extends the benefits of the IBM Tivoli Provisioning Manager. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/intell-orch/



This three tier environment has a firewall that separates each layer, in order to protect them from intruders. The several site locations are interlinked on the same private network, creating a virtual private network (VPN), as seen in Figure 5-2.

Figure 5-2 A bank VPN infrastructure


Internet banking applicationInternet banking applications typically have a great number of clients and requests.

When the current Web server pool is not enough to respond to clients’ requests in an adequate amount of time, the orchestration mechanism takes over and searches for underutilized resources on other company’s sites. Under manual processing, these new machines will be added one by one, demanding many operators and much time. Using the Provisioning Manager, the deployment of new machines could be automated, demanding less time and no operator for the reallocation.

The operations described above are managed and made by IBM Tivoli Provisioning and IBM Tivoli Orchestration. Tivoli Provisioning is responsible for allocating new machines automatically while orchestration provides a dynamic and on demand environment for applying the company’s best practices.

5.2.3 Correlated productsThe IBM Cluster Systems Management (CSM) can be considered part of the provisioning discipline for grids. One of the most important features of CSM is to monitor both hardware and software events, and to trigger automated recovery actions when appropriate. These event monitoring and automated recovery features enhance the efficiency of the administrator and the reliability of the environment by enabling rapid problem detection and resolution.

IBM Cluster Systems Management The purpose of the IBM Cluster Systems Management (CSM) product is to provide broad management capabilities for a multiple nodes environment, such as clusters and grids. CSM provides management from a single point of control. These include resource monitoring, automated monitoring and operation, remote hardware control, remote command execution, security, configuration file management, parallel network installation, and diagnostics.

CSM provides a variety of other benefits:

� CSM helps administrators deploy their nodes rapidly by automating many configuration tasks and by leveraging existing Open Source products.

� CSM provides efficient monitoring of the environment resources without overwhelming network bandwidth.

� The automated error detection CSM provides helps catch problems before they impact the environment, and assists with rapid resolution and recovery of problems that occur.


� CSM has an architecture and modular construction that maximizes flexibility so your distributed solution can evolve and grow as your needs change.

� You can use CSM to manage a homogeneous or a heterogeneous environment.

5.3 SecurityThis section discuss security and its relationship with grid computing. In grid or in any distributed computing system, security plays a major role when ensuring the quality of service delivered. We can divide security into several subjects to analyze how a grid can satisfy basic security requirements:

Authentication The grid has to “be aware“ of the identity of the users that interact with it, either providing or using resources.

Authorization The grid has to restrict access to its resources to the users that are eligible to access it.

Integrity Data exchanged among grid nodes should not be subject to tampering.

Privacy The grid must provide facilities for assuring the privacy of the information exchanged between nodes when required.

In order to implement a comprehensive security infrastructure, with centralized definitions for resources, which usually are at different levels, some additional features may be recommended, for example, in a UNIX infrastructure, with operating systems that are part of the grid to run jobs, and, for example, for some specific nodes with WebSphere Application Server, we can have a LDAP Server with the definition of the users, passwords, and more resources. In that way, the UNIX operating systems and WebSphere nodes can request the LDAP Server when an user or application is trying to log into the grid. This centralized mechanism may be better than creating a set of users in WebSphere and another set of users on the UNIX boxes and maintaining two sets of users. This implementation may be part of a single sign-on strategy.

Other desirable mechanisms that a grid could implement are related to the trusted relationships that have to be established among different virtual organizations that belong to the same grid at a certain time. Globus Toolkit and

Note: IBM Cluster Systems Management provides a robust, powerful, and centralized way to manage a large number of nodes from a single point of control. For more information, please refer to the following Web site:




IBM Grid Toolbox require digital certificates to include a node into a grid. This requires a certification authority and a means of certificate validation. In a grid that has resources from different real organizations, some traditional IT mechanisms should be implemented to share resources, like a virtual private network (VPN). In more restricted grids, or in the communication between two independent grids, there should be mutually trusted mechanisms to ensure security, or, in other words, one grid should be able to rely on the policies of the other grid and vice-versa.

5.3.1 Firewalls, VPNs, and virtual organizationsGrid allows you to share resources between real organizations by creating a virtual organization. The way to create an effective virtual organization is often carried out through traditional IT mechanisms. This implies the use of firewalls, VPNs, portals, and others.

As an example, consider the scenario in which a corporate user submits a job to a computational grid that runs in a partner enterprise: in this case, a trusted relationship prior to the job’s submission exists, and this trusted relationship between two corporations includes the cross-execution of jobs in their grids. There are some traditional IT components that allow the joining of both IT infrastructures for specific services: leased lines and other hard communication links are examples of hardware components used to join different private networks.

On the software side, there are some alternatives: probably the most traditional is a SSL-based virtual private network (VPN). This consists of an encrypted, secure, and mutually trusted connection, which establishes a transparent tunnel of communication between two nodes that belong to different network contexts. In this sense, a VPN may be used to join two real organizations into one virtual organization for the implementation of a single grid.

Additional security requirements and interesting capabilities for grid are mutual authentication among different organizations and delegation of authentication so that a process becomes more efficient when a computational activity involves a number of spread resources.


5.3.2 Security productsIn this section, we present a set of security products for grid.

Tivoli Access ManagerTivoli Access Manager (“Tivoli Access Manager” on page 177) implements security mechanisms based on predefined policies. A policy is a higher level rule that details a set of behaviors for several systems in different situations; it means that the user can define policies rather than just permissions. Tivoli Access Manager provides a security infrastructure to an IT installation, and its security services can be invoked from Web servers, application servers, databases, operating systems, domain controllers, and others. Tivoli Access Manager gives you the opportunity to implement a real security layer that crosses most of the other software layers, instead of giving just simple rules for the different subsystems that belong to an infrastructure.

Tivoli Access Manager security mechanisms can be integrated in a grid to provide comprehensive security services to its resources. Tivoli Access Manager uses open standards to provide its services. As we can see, these features are very important for complex environments. These are examples of typical IT components that can be integrated into a distributed environment.

Tivoli Access Manager for e-businessProvides unified authentication and authorization to an IT infrastructure. Tivoli Access Manager for e-business supports single sign-on to mainframe, UNIX operating systems, Microsoft foundation, Web servers and Web application servers, and telnet. It exports a number of APIs to be used from .NET and J2EE applications. It offers flexibility in design time, as it can be used with plug-ins in Web servers and in highly scalable proxy architecture. All these services allow the developers to concentrate on the business logic, avoiding spending time on the design of security mechanisms. From the administrator’s point of view, Tivoli Access Manager provides an unified and centralized security mechanism to the whole infrastructure. It also gives benefits in time and cost to the IT management staff.

Note: Tivoli Access Manager for e-business is part of the IBM Tivoli Access Manager family of products. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/access-mgr-e-bus/

Note: More information about .NET can be found at:

http://www.microsoft.com/net/



http://www.microsoft.com/net/

Tivoli Access Manager for Business IntegrationTivoli Access Manager for Business Integration is a multi-platform security management solution. It was specifically designed for the WebSphere MQ family and it basically upgrades MQ’s native security services. The data-protection provided by Tivoli Access Manager for Business Integration is at the application level, so it is not necessary to modify or recompile the MQ applications that use Tivoli Access Manager for Business Integration. This level of protection covers the link and channel level, which are related to the confidentiality and integrity of the messages when they are in transit between two systems. Tivoli Access Manager for Business Integration also gives security mechanisms that are used when the messages are resident in a queue to the control of MQ.

Tivoli Access Manager for Operating SystemsTivoli Access Manager for Operating Systems is a powerful security system that locks resources and prevents unauthorized access. It is simple to use and it is intended to secure files, applications, and even other operating platforms. This product is useful for protecting the infrastructure even from internal unauthorized access, blocking and protecting the environment from both inside and outside suspicious activities. It provides audit mechanisms to midrange operating systems that are similar to those provided by the mainframe, like world-class protection, auditing, and tracking of the actions of every user. It also provides a tool for self-auditing, Personal Universal Auditing, that audits users, programs, files, ports, resources, and systems.

Grid Security Infrastructure (GSI)Globus Toolkit and IBM Grid Toolbox provide an implementation of Grid Security Infrastructure (GSI). GSI uses digital certificates to provide mutual authentication, single sign-on, and privacy of the information exchange between two grid nodes.

Note: Tivoli Access Manager for Business Integration is part of the IBM Tivoli Access Manager family of products. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/access-mgr-bus-integration/

Note: Tivoli Access Manager for Operating Systems is part of the IBM Tivoli Access Manager family of products. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/access-mgr-operating-sys/


http://www-unix.globus.org/toolkit/docs/3.2/security.html


http://www.ibm.com/software/tivoli/products/access-mgr-bus-integration/


The single sign-on mechanism across organizational boundaries includes delegation of credentials. That mechanism allows the secure execution of computational activities that involve resources scattered over multiple sites.

Mutual authentication occurs when two parties have their own certificates signed by their respective CAs (certification authorities), and those actors also trust the signature of the other’s CAs in the other’s certificates. This mechanism allows you to pass over the boundaries of an organization.

GSI also implements delegation of an authentication mechanism through a proxy. Typical situations where this mechanism is useful are, for example, when some computational activity in the grid requires several mutual authentications in several resources. That computational activity may be generated by a user or by local/remote agents on behalf of the user; this process might demand the reentering of the certificate’s passphrase every time a resource is accessed. A proxy can be useful in automating this process by delegation: in this context, it is just a new certificate with new public and private keys, and that certificate is signed by the owner instead of the CA. Then, the proxy’s certificate is used on behalf of the user’s certificate.

Further explanation of these processes and GSI can be found in the redbook Introduction to Grid Computing with Globus, SG24-6895 (at Redbooks Web site) and at the Globus Alliance Web site:

http://www.redbooks.ibm.com/http://www.globus.org

Enterprise Identity MappingEnterprise Identity Mapping (EIM) is part of the Autonomic Computing initiative for Server Groups IBM ^® security. It provides a mapping between the users of several systems and the EIM identity, by associating the user registered in every system with the identity. EIM implements a list or directory of users/passwords/systems and has connectors to access them. The benefit for

Note: GSI is a set of libraries and tools, provided with the Globus Toolkit, for accomplishing secure authentication over an open network. It is possible to use the authentication portion of the Globus Toolkit via two applications: GSI-enabled Secure Shell (SSH) and GSI-enabled FTP (gsiftp). For more information, please refer to the following Web sites:

http://www.globus.org/security/v1.1/standalone.htmlhttp://www-unix.globus.org/toolkit/docs/3.2/security.htmlhttp://publib.boulder.ibm.com/eserver/v1r1/en_US/index.htm?info/ogsainfo/secure.htmhttp://www-unix.globus.org/toolkit/docs/3.2/gsi/key/index.html



http://www.globus.org/security/v1.1/standalone.html


http://publib.boulder.ibm.com/eserver/v1r1/en_US/index.htm?info/ogsainfo/secure.htm

http://www-unix.globus.org/toolkit/docs/3.2/gsi/key/index.html


the user, the developer, and the infrastructure itself is that a given application that accesses different subsystems, all of them with the same human user but with different users to the systems' point of view, will not need to place, cache, or send a user and password for each subsystem. Instead, the application can access the EIM repository and then call EIM every time a new user/password is needed in the flow. EIM can be considered as a basic single-sign-on mechanism for IBM and non-IBM platforms.

The EIM Domain controller is based on a LDAP Server. In that server, EIM stores the identifiers, associations, and registries for the different systems. The registries contain a user and password for a given identity in a given system.

EIM can help in a grid environment by providing unified identities to users that have access with different user and passwords to heterogeneous systems.

5.3.3 ConclusionsIn summary, we can say that there are not yet a wide range of products specifically designed for security in grids. However, most traditional IT products may be integrated into a grid system. That is because a grid can be seen as a new way of using traditional IT components. Therefore, the ability to integrate traditional IT components with specifically designed components for grid is associated with those products capabilities.

A grid surely will require sophisticated security mechanisms due to its intrinsic complexity and distribution of resources. For that reason, we recommend having a security framework or a similar security infrastructure, based on open standards, to unify the security policies and ease the definition and management of complex rules. Toolkits like Globus Toolkit and IBM Grid Toolbox require digital certificates, but they usually provide only services to applications, and the grid has resources other than applications. At this point, a security framework and other mechanisms are vital for the health of the grid.

Note: Enterprise Identity Mapping is part of the Autonomic Computing initiative for Server Groups IBM security. For more information, please refer to the following Web site:

http://www.ibm.com/servers/eserver/security/eim/


http://www.ibm.com/servers/eserver/security/eim/

5.4 Directory ServicesWith the idea that a grid can be made up of many heterogeneous resources, it is vital to have a method of capturing what the grid looks like. One would want to be able to perform queries on both static and dynamic attributes of a grid. The static attributes may include the machines enrolled in a grid, the operation system version, CPU speed, physical memory, and so on. The dynamic attributes may include what machines are available, disk space available, current load, and so on. The larger the grid, the more important it is to be able to store and retrieve this kind of information. Some of grid products include this capability as part of the solution, such as the Globus Toolkit.

Most of the time, Directory Services are interrelated with security disciplines and products (please refer to 5.3, “Security” on page 92).

5.4.1 Directory Services productsIn this section, we present the Directory Services products.

Tivoli Directory Server Tivoli Directory Server (TDS) is a component of the IBM identity management solution. It is the foundation for deploying comprehensive identity management applications and advanced software architectures like grids. TDS implements the Internet Engineering Task Force (IETF) LDAP V3 specifications. It also includes enhancements added by IBM in functional and performance areas. TDS uses IBM DB2 Universal Database as backup storage to provide per-LDAP operation transaction integrity, high performance operations, and online backup and restore capability.

Tivoli Directory IntegratorTivoli Directory Integrator (TDI) is one of the products of the IBM identity management solution. It can integrate different directories of resources across the enterprise. These repositories can be placed on directories, databases, and applications.

TDI is a suitable solution for those organizations with heterogeneous repositories of resources that prefer keeping those repositories but providing a unified and integrated vision to the applications that use their services. TDI can be useful as well for these organizations when they decide to consolidate the heterogeneous

Note: Tivoli Directory Server is a component of the IBM identity management solution. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/directory-server/


http://www.ibm.com/software/tivoli/products/directory-server/

repositories because the consolidation activities can be done while TDI hides that complexity to the applications that use their capabilities. In other words, TDI can be used as a central hub of a decentralized directory infrastructure, while it can help to centralize it if the organization needs it for some reason.

TDI has built-in connectors to heterogeneous repositories of resources and provide Java APIs to modify or extend these connectors.

TDI can synchronize repositories, provide a consistent structure of directories, even though those repositories were different, and can implement rules to expose the right resource to right consumer.

It supports Web services, connectors to LDAP, JDBC™, JMS, XML, and JNDI, among others, and includes the IBM Tivoli Directory Server, which is the IBM directory of resources for mid-range platforms.

5.4.2 ConclusionsDirectory Services are used in a grid environment to make the search and discovery of resources placed in different directories or repositories easier by providing a unified vision of the directory to the grid environment or the user that asks for certain services from the grid.

5.5 LicensingIn most organizations, a main portion of their investment in information technology infrastructure is dedicated to software licensing. For this reason, it is not surprising that most companies are looking for a better way to manage their software licenses. This leads us to a new term: Software Asset Management. This is a process of managing software assets, which includes budget planning, software life cycle, such as upgrades, and others.

One of the concerns about grid implementations is related to the licensing of software components. This is especially true with Independent Software Vendor (ISV) applications, as most of them tie their licenses to processor count, host identification (ID), and processor type. When this happens, it limits the growth and the scalability of a grid.

Note: Tivoli Directory Integrator (TDI) is one of the products of the IBM identity management solution For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/directory-integrator



Some of the commonly heard terms about licensing are listed below:

� Site license

� Node locked license

� Floating license

Site licenseUnder site license, the company typically pays one single amount to have unlimited use of the software: the company is allowed to add additional servers without worrying about paying new fees or a nominal fee could be charged based on the amount of new servers. This is commonly used in data centers where the software is extensively used, and helps build up an agile environment, as users can run their applications anywhere. On the other hand, this is generally a very expensive solution.

Node locked licenseOne of the most commonly deployed licensing method is node locked license, which basically means that the application is constrained to certain hosts. Often, the ISV will generate a key for that applications based on MAC address, host ID, CPU ID, number of processors it has, type of processor, and others. In such a scenario, when a grid has to be deployed, the architect will require some additional planning and the grid platform has to keep track of which server has which applications.

Floating licenseIn the floating license method or model, a certain number of licenses is shared within a set of nodes. Most often, there is a license manager server controlling and keeping track of the licenses distributed. Applications request an available license from the license manager when it is started, and once completed, it will return it back. However, if the licenses are insufficient, additional applications instances are unable to execute. This model is very effective and very commonly used in grid applications (if the application supports it), allowing growth and scalability.


5.5.1 Licensing productsIn this section, we present a licensing product that may be integrated into a grid system.

IBM Tivoli License ManagerIBM Tivoli License Manager offers an effective way to manage multiple product licenses by means of company-tailored policies. IBM Tivoli License Manager (ITLM) helps with the following tasks:

� Information collected about the products installed, using an inventory scan technology

� Keeping track of the in-use licenses, available licenses, and procured licenses

� Monitoring the start and stop of a software on any machine

� Monitoring of license usage patterns and informing administrators

� Assigning licenses to systems

� Provides historical and real-time reports of the software usage

5.5.2 ConclusionAs more software vendors start to adopt grid into their software, the licensing strategy may expand. Major vendors have software licensing models catered towards grid environments. In extreme cases, a license can be managed by provisioning techniques. In this case, the software is installed on the server in which the execution is to be performed and, after the job is completed, the application is removed.

Note: License Manager is part of the IBM Tivoli software family to automate the software licensing and utilization, and provides advanced reporting and inventory capabilities. For more information, please refer to the following Web site:

http://www.ibm.com/software/tivoli/products/license-mgr/



5.6 Billing and meteringSome implementations may require the use of metering and billing techniques, such as utility-like services, so that the owner of the resources made available is somehow compensated by providing the resource.

Some of the commonly used methods to accomplish billing and metering today includes:

� Scheduler accounting logs

� Licensing management

Scheduler accounting logsWhen jobs are submitted, queued and pushed out for execution by job schedulers, many lot entries are generated. In this way, logs usually contain very useful information, such as when the job is started, when it ends, which machines are selected, and how long the execution took.

Licensing managementLicensing management is one of the easiest methods to monitor the usage of software, manually or through a software, such a IBM Tivoli License Manager. Usage can be tracked based on the application installation rather than the application’s executions. This may sound ineffective, but it provides the quickest method to deploy billing and metering. Products such as IBM Tivoli Orchestration and IBM Tivoli Provisioning Manager have some features for monitoring the usage, which can be used for billing and metering as well.

5.6.1 ConclusionAs utility-like services get more popular, there will be a greater demand for billing and metering software with better defined control. This is especially true in light of electronic utility computing, also known as “e-utility computing”, in which someone can use the computing power through simple and dynamic access, in an on-demand basis, and be billed at a later stage.


Part 3 Bundles

Part 3



Chapter 6. Virtualization Engine

This chapter discusses the following topic:

� Virtualization Engine™

6


6.1 IntroductionVirtualization Engine (VE) is an IBM suite, a set of systems, products, and technologies that work together in a comprehensive way to provide a seamless and unified side of the IT infrastructure. This reduces the complexity of allocating new resources to applications, as all of these resources are dealt within the same fashion. It also allows easier monitoring and management of the IT resources.

As shown on Figure 6-1, VE has three major groups of virtualization technologies and products:

� Systems services� Systems technologies� Operating systems

Figure 6-1 Groups of virtualization

System servicesSystem services’ products and technologies are higher level components in the stack of infrastructure layers. This set of components is intended to create the environment to run applications, manage the complexity of the technology, and create a unified repository of storage resources. There are two main groups of elements: those related to servers and those related to storage.

Note: The IBM Virtualization Engine utilizes key IBM virtualization technologies, making a comprehensive approach available to enterprise-wide virtualization that is consistent across heterogeneous environments.

Complete information about IBM Virtualization Solutions are presented at the following Web site:

http://www.ibm.com/servers/eserver/about/virtualization/index.html

S ystem sT ech n o lo g ies

O p era tin g S ystem s

S ystem s S ervice s

IB MV irtu a liza tio n

E n g in e ™



System technologiesSystem technologies of VE are the technologies available in IBM machines. These components provide virtualization intelligence in the processor, including micro partition abilities, memory, I/O, and network. These technologies are embedded in IBM hardware, and they are delivered in the IBM and TotalStorage brands. Examples of these technologies are Hypervisor, Virtual I/O, and Virtual LAN, available in Power5 machines.

Operating systemsVE features work on the operating system level, as they provide the ability to run multiple operating systems on a single box. These characteristics are available in pSeries, iSeries™ through LPAR and DLPAR technologies, and zSeries®, which allows LPAR and VM images to run different operating systems. xSeries operating systems can be virtualized through VMware software (http://www.vmware.com/) that can be integrated into a VE infrastructure. Examples of the operating systems that can use these features are z/OS®, i5/OS™, AIX 5L, Linux, and Windows.

6.2 Virtualization Engine suitesVirtualization Engine suite provides provisioning resources, workload management, monitoring, and a console for management and administration. This console is Web-based and has a sort of “dashboard” to manage the VE infrastructure (see “Virtualization Engine console” on page 108). Drill-down features and an intuitive user interface to identify different situations and better understand the infrastructure’s behavior are included. Additionally, VE also provides management capabilities for the grid and for its individual components.

There are two suites that make up Virtualization Engine systems services:

� IBM Virtualization Engine Suite for Servers� IBM Virtualization Engine Suite for Storage

The main differentiator of VE is that it provides services for the whole stack of infrastructure levels, creating an integrated environment to run and manage applications, and to provide, virtualize, and manage resources.

Chapter 6. Virtualization Engine 107

http://www.vmware.com/

Virtualization Engine consoleThe Virtualization Engine console is based on the IBM Integrated Solutions Console framework to provide a consolidated view for managing your virtualized enterprise resources. The Virtualization Engine console is used to manage your IT environment by looking at your systems beyond operating system boundaries and helping to better maximize resource sharing. It provides two powerful cross-platform applications: the health center and the launch-pad. The health center provides a single view point to perform key system and storage monitoring and management. The launch-pad provides a single view to launch key Virtualization Engine related systems and storage management consoles.

6.2.1 IBM Virtualization Engine Suite for ServersVirtualization Engine Suite for Servers, also known as Virtualization Engine Services Multiplatform, helps provide virtualization and management of resources across select systems, both in an individual server and across IBM and select non-IBM systems. This suite delivers tools for running grid applications as well as provisioning, management, and workload management. as Virtualization Engine Services Multiplatform consists of the following systems services:

� VE Console (see “Virtualization Engine console” on page 108� IBM Enterprise Workload Manager � IBM Director Multiplatform� IBM Tivoli Provisioning Manager� IBM Grid Toolbox V3 for Multiplatforms

IBM Enterprise Workload ManagerEnterprise Workload Manager (EWLM) enables you to automatically monitor and manage multi-tiered, distributed, heterogeneous, or homogeneous workloads across an IT infrastructure to better achieve defined business goals for end-user services. These capabilities allow you to identify work requests based on service class definitions, track performance of those requests across server and subsystem boundaries, and manage the underlying physical and network resources to set specified performance goals for each service class.

EWLM is intended to support application performance, making the right resources available to the applications that need them, in real time, usually during peak workloads. EWLM collects ARM messages sent by the resources; that way, EWLM can report metrics related to performance goals as business priority, response time, and others.

EWLM can also work jointly with Tivoli Intelligent Orchestrator (TIO). TIO can deploy the EWLM agent in the resources that it is allocating through provisioning and orchestration. Those resources are allocated in a EWLM domain, so EWLM


will recognize them and will optimize their utilization based on the designed policies.

EWLM can be integrated with WebSphere Application Server’s applications, as it uses open interfaces based on ARM. WebSphere and DB2 offer instrumentation data through ARM, so they can provide information to EWLM in order to be managed by it. The user defines the policies and transaction classes, which can be used in all the supported platforms. EWLM is able to monitor the behavior of WebSphere Application Server and DB2. Those are the input used to make workload balancing decisions based on business policies and infrastructure parameters, like server utilization, transaction response time, and topology. EWLM provides a Web based console with drill-down capabilities to inspect resources.

IBM Director MultiplatformDirector Multiplatform helps deliver a common, consistent, and cross-platform systems management solution for IBM servers, storage, and operating systems. It provides a single administrative console for management tasks (operating system, storage management, distributed systems management, and platform management), a common management infrastructure for upward integration with Tivoli, and a management foundation for the on demand architecture.

Using Director Multiplatform, many of the administrator's manual tasks can be automated to proactively and remotely manage systems. Tasks such as discovery, event logs and action plans, file transfer, inventory collection, process management, resource monitors and thresholds are among those tasks that fall into this category. Additionally, the predictive and proactive capabilities associated with alerting and real-time system diagnostics help maximize server uptime and reduce service downtime costs.

IBM Tivoli Provisioning ManagerSee “IBM Tivoli Provisioning Manager” on page 88.

IBM Grid Toolbox V3 for MultiplatformsGrid Toolbox V3 for Multiplatforms, based on the Globus Toolkit V3.0 and the Open Grid Services Architecture (OGSA), delivers the infrastructure to help rapidly develop, deploy, and manage distributed resources. Grid Toolbox V3 uses a service model approach for its architecture, which keeps the details of the implementation separate from the interface, and is based on the Open Grid Services Infrastructure (OGSI). For more information about Globus Toolkit and OGSA, see 2.2, “Grid standards and toolkits” on page 24.


6.2.2 IBM Virtualization Engine Suite for StorageVirtualization Engine Suite for Storage can complement the Virtualization Engine Suite for Servers, and consists of the following separately orderable products:

� IBM TotalStorage Productivity Center� IBM TotalStorage SAN Volume Controller� IBM TotalStorage SAN File System

IBM TotalStorage Productivity CenterTotalStorage Productivity Center is an open storage infrastructure management solution designed to help reduce the effort of managing complex storage infrastructures, to help improve storage capacity utilization, and to help increase administrative efficiency.

IBM TotalStorage SAN Volume ControllerSANs today create a fixed relationship between storage devices and servers. TotalStorage SAN Volume Controller (SVC) implements a virtualization layer between storage devices and servers. The servers can see virtual disks in the SAN, and SVC maps those virtual disks in the real physical devices. For more information about SVC, see 4.3.2, “Block data layer products” on page 71.

IBM TotalStorage SAN File SystemOnce the block data is virtualized in storage pools managed by SVC, servers can create their own file systems. However, it is common that several servers share file systems in an organization. TotalStorage SAN File System (SAN-FS) is a product that creates that virtual and shared file system. Instead of using other classic technologies to share file systems through the network and have several points of management, SAN-FS provides a shared file system for servers through the SAN itself, with a single point of management and name space. For more information about SAN-FS, see 4.3.2, “Block data layer products” on page 71.

6.3 ConclusionAs mentioned before in this redbook, there are several traditional IT components that can fit into a grid. Moreover, that grid can be improved and leveraged by these traditional IT components, although they were not specifically designed for grid. This is an example of the traditional IT technology tools, or non-grid tools, that are included in provisioning, Virtual I/O, Virtual LAN, micro-partitioning, monitoring, and so on. Several of these technologies provide services to the grid computing environment’s underlaying layers.


VE utilizes the key IBM virtualization technologies and products while grid computing is perhaps the most advanced degree of virtualization, as it enables multi-organization resource virtualization.

VE also provides sophisticated management tools, going further than just virtualization functions. It also creates a foundation for advanced automation within IBM and TotalStorage products, as well as non-IBM hardware in specific subjects.



Chapter 7. ISV suites

This chapter discusses the following bundles:

� Platform Computing

� DataSynapse

� United Devices

7


7.1 IntroductionBundles for grid computing are those sets of products and tools that allow for the development, deployment, execution, and management of a grid. Most of the bundles’ components enumerated in this chapter already have been described in this redbook, so we will not describe them here in detail.

7.2 Platform ComputingPlatform Computing has a complete set of tools to develop and run grid solutions. Platform’s products address the scalability, security, scheduling, and management attributes of a grid infrastructure. Mainly oriented to server grid, Platform also contributes to the Global Grid Forum (GGF), adheres to the Open Grid Services Architecture (OGSA), and supports several operating systems and hardware.

See 3.4.4, “Platform LSF (Load Sharing Facility)” on page 50, 3.4.5, “Platform LSF MultiCluster” on page 51, 3.4.6, “Platform LSF HPC (High-Performance Computing)” on page 51, and 3.4.7, “Platform Symphony” on page 52 for more informtion about this topic.

Platform offers two different types of bundles, including a full set of grid and distributed computing products and a family bundle of products.

The bundle of grid and distributing computing products include:

� Platform LSF

Platform LSF is one of Platform's distributed computing products and is one of the core products in the family bundle that manages and accelerates batch workload processing for compute- and data-intensive applications. With Platform LSF, an organization can intelligently schedule and guarantee the completion of batch workload across a distributed, virtualized IT environment. Platform LSF fully utilizes all IT resources regardless of operating system, including desktops, servers and mainframes, to ensure policy-driven, prioritized service levels for always-on access to resources. Underpinning Platform LSF is the production-proven, open, robust Virtual Execution Machine (VEM) architecture that sets the benchmark for performance and scalability across heterogeneous environments. Platform LSF has an

Note: Complete information about Platform Computing can be found at:

http://www.platform.com/




extensive set of scheduling capabilities, including Fairshare, Preemption, Advance Reservation, Resource Reservation, and SLA-based scheduling.

� Platform LSF HPC

Platform LSF HPC is one of Platform's distributed computing products and is one of the core products in the family bundle. Platform LSF HPC is software for high-performance computing needs. This software allows organizations to intelligently schedule parallel and serial workloads, providing the ability to solve large, grand challenging problems while utilizing the available computing resources at maximum capacity. It takes full advantage of high performance network interconnects available on clustered systems and supercomputers.

� Platform Symphony

Platform Symphony is one of Platform's grid and distributed computing products. Platform Symphony is a comprehensive grid solution that is enterprise-class software that distributes and virtualizes compute-intensive application services and processes across existing heterogeneous IT resources, creating a shared, scalable, and fault-tolerant infrastructure, delivering faster, more reliable application performance while reducing cost. Platform Symphony is engineered to provide business-critical applications compute power on demand. This is accomplished by decoupling application compute demands from dedicated hardware found in unreliable and underutilized in LOB IT silos. Underpinning Platform Symphony is the robust Virtual Execution Machine (VEM). More than simply a distributed application technology, the VEM is a standards-based, high performance, system-level architecture. The VEM resides below the application layer, providing a true virtual environment that is ideal for critical workload management. This workload-centric approach offers many performance, reliability and management advantages when compared to other approaches

The second family bundle of products includes:

� Platform LSF

� Platform LSF HPC

� Platform LSF MultiCluster

Platform LSF MultiCluster extends an organization's reach to share virtualized resources beyond a single Platform LSF cluster to span geographical locations. With Platform LSF MultiCluster, local ownership and control is maintained, ensuring priority access to any local cluster while providing global access across an enterprise grid. Organizations using Platform LSF MultiCluster complete workload processing faster with increased computing power, enhancing productivity and speeding time to results.

Chapter 7. ISV suites 115

� Platform LSF License Scheduler

Platform LSF License Scheduler optimizes the usage of all application licenses across Platform LSF clusters by allocating a virtualized pool of licenses to users based on an organization's established distribution policy. Using Platform LSF License Scheduler, organizations can realize dramatic reductions in additional application license purchases and increase overall access to license resources.

� Platform LSF Reports

Platform LSF Reports provides a number of key Platform LSF cluster performance metrics in a series of reports, which provide insight into cluster utilization. Using Platform LSF Reports, administrators are able to improve overall cluster performance and troubleshoot configuration problems.

� Platform LSF Analytics

Platform LSF Analytics uses workload and license management data to provide design centers with analytics support for project planning decisions. Platform LSF Analytics assists engineering managers to estimate project completion times and provision hardware and licenses more accurately, ensuring project priorities are being met. Platform LSF Analytics scales for large Platform LSF clusters and multiple sites around the world.

Components of these bundles are supported on several platforms. Platform also has its Globus Toolkit implementation. Platform Globus Toolkit supports OGSA, and it can be integrated within a traditional Platform architecture.

7.3 DataSynapseDataSynapse's GridServer software provides open standards compliant, pre-manufactured services across various grid computing disciplines: scheduling, workload management, data virtualization, and grid middleware. DataSynapse enables organizations to create "Service Grids" that support stateful, persistent, transactional processing of high concurrency and traditional parallel execution workloads.

GridServer enables the broadest set of enterprise applications to run on a shared infrastructure, unlocking performance bottlenecks across the enterprise for improved productivity and responsiveness. It creates a virtual application execution environment that dynamically fulfills service level requirements across virtual resources, delivering substantially higher service levels for radically lower costs.


DataSynapse GridServer provides a service scheduler that enables applications with transactional or HPC type workloads to be executed in a service oriented concurrent or parallel execution paradigm, including BPEL workflow support for event driven scheduling. Refer to 3.4.3, “DataSynapse GridServer” on page 48, which shows the GridServer components’ architecture.

GridServer also offers an essential role in the Information virtualization discipline. GridServer provides a standards based, multi-language distributed, synchronized in memory/disk cache that supports native cache loaders, transaction manager plug-ins, spring framework transaction manager support of POJOs, and EIS access via JBDC, EAI, Web Service, and Application calls. It also supports connection pooling and it can dynamically provision additional connections on-the-fly.

7.4 United DevicesUnited Devices offers a portfolio of products for developing, implementing, and managing diverse and geographically distributed grids of heterogeneous resources. In addition, its family of products operate the world's largest grid for massive-scale research projects, such as the grid.org project (see http://www.grid.org/) and the World Community Grid. For more information about World Community Grid, refer to Appendix B, “World Community Grid” on page 149.

The United Devices Grid MP platform family of products can operate grids of any size as well as aggregate resources on a network (including clusters, servers, desktops, and laptops) to create a high-performance compute grid for powering a wide range of applications. United Devices serves clients in a variety of industries, including life sciences, manufacturing, government, industrial engineering, and others.

Grid MP Enterprise from United Devices is able to manage groups of resources, from a single cluster to large-scale multi-resource grid. The package includes the MP services and agents supporting Windows, Linux, AIX, UNIX, Solaris™, and

Note: Complete information about DataSynapse's GridServer software can be found at:


Note: Complete information about United Devices can be found at:

http://www.ud.com/

Chapter 7. ISV suites 117


http://www.ud.com/

http://www.grid.org/


http://www.grid.org/

Mac OS. The product is suitable for highly parallelizable, CPU intensive computational activities that are “self contained.” This means that a given task can run without further communication with external resources, up to its completion. Grid MP is designed to scale up to hundreds of thousands of devices. Its scalability has been proven with its global grid projects on grid.org, a grid with over 3 million devices.

In addition, Grid MP also features the following:

� SDK for applications development that allows for rapid migration of applications to the grid.

� Provides an environment in an encrypted form. Furthermore, applications are run in a secure sandbox environment on all grid nodes.

� Web-based management interface for administrative users.

� Workload and scheduling capabilities. For example, it can balance the job assignation for different donors based on the amount of computing resources, such as memory or CPU power, made available to the grid. Thus, the unit of work that is sent to the donors may have different weights based on the donors’ capabilities. In short, Grid MP assigns jobs to the most appropriate resource available. Jobs can also be rescheduled if the donors have not returned the results after a certain period of time.

� Features a Unified Job Scheduling (UJS) product that allows organizations to manage isolated, dedicated Distributed Resource Managers (DRMs) from a single tool. This product preserves departmental control of resources while enabling the sharing of resource groups that previously operated in silos, with peaks and valleys in demand.

� Provides a data warehouse and reporting tool, MP Insight, which allows organizations to analyze current capacity and utilization information in order to make more informed purchasing and capacity planning decisions. MP Insight collects information related to applications, jobs, resources, and users in the grid and extracts this information into a data warehouse, where it is made available to users and administrators in the form of metrics-based reports.


Part 4 Services and solutions

Part 4



Chapter 8. Services and solutions


� Services portfolio for grid computing

� Solutions portfolio for grid computing

8


8.1 IntroductionIBM is committed to open standards and is working with The Globus Alliance open source development community, the Global Grid Forum, and the Interoperable Informatics Infrastructure Consortium (I3C) to promote the adoption of open standards and accelerate the availability of grid offerings that IBM and its business partners can provide in the grid computing business.

IBM Global ServicesIBM Global Services has experienced grid consultants and architects engaged in grid projects worldwide. With this experience, IBM can help clients in all industries enhance resiliency, scalability, stability, high availability, security, and evolve into On Demand Business with their grid deployments. IBM Global Services is widely recognized as the largest services company in the world and it is in 164 countries or regions. For more information, please refer to the following Web site:

http://www.ibm.com/igs

8.2 Services portfolio for grid computingIBM’s main focus is the creation, development, and manufacture of the industry’s most advanced information technologies. This includes computer systems, software, networking systems, storage devices, and microelectronics. IBM Global Services translates these advanced technologies into business value for clients and helps make information technology easy to acquire and manage. To learn more about the services portfolio, visit:

http://www.ibm.com/services/http://www.ibm.com/linux/grid/solutions/

Note: IBM and business partners work together with clients to leverage grid computing solutions in their existing IT infrastructure. It is important to mention that grid computing is not an out-of-the-box product: building a grid is the result of an assessment and architecture development carried out in a client-tailored fashion.

Important: The following Web site provides a search capability to access all IBM announcement letters, part numbers, sales manual, press releases, Redbooks, white papers and catalogs:

http://www.ibm.com/common/ssi



http://www.ibm.com/services/



http://www.ibm.com/linux/grid/solutions/


Table 8-1 provides a overview of the services IBM has specifically designed for grid computing.

Table 8-1 Grid computing services overview

The graphic in Figure 8-1 on page 124 shows an example roadmap for grid computing implementation services.

Services Description

Grid Innovation Workshop

Typically a 3~5 days workshop, its main goal is to provide a grid computing design overview that can be used as a basis for follow-up engagements.

Grid Strategy Engagement

Engagement service that defines the vision for a client to leverage innovative grid computing solutions. It also identifies how grid computing technologies can impact an organization’s existing enterprise architecture and business model.

Grid Design Services

Design service that precedes grid computing implementation services. Its main goal is to provide a detailed design to be implemented afterwards.

Grid Software Implementation Services

Service in which implementation, testing, and deployment activities of the grid computing environment, outlined and designed by the previous services, take place.

Grid Training and Education

Training and education offering available to complement and leverage the skills of the interveners in the grid computing implementation roadmap.

Chapter 8. Services and solutions 123

Figure 8-1 Grid computing services roadmap

8.2.1 Grid Innovation WorkshopGrid Innovation Workshop is a service designed to help companies and universities understand the innovation and potential of grid computing. Customized for each organization, these workshops are designed to provide tools that help in the identification of the benefits of grid computing, and define a roadmap for its implementation.

DescriptionThis service is designed with the objective of identifying high-level business goals, objectives, and critical success factors. Typically, a workshop that lasts from three to five days, is designed to define potential stakeholders and understand goals, needs, and expectations. This gives you a chance to identify and prioritize actionable pilot opportunities that are aligned with the business goals, objectives, and critical success factors. One of the objectives in this service is to develop a grid pilot plan that includes process identification and implementation timelines that provide awareness regarding the benefits and technology of grid.

This workshop also lets you define and provide an introduction to grid computing concepts, evolution, and adoption frameworks. It also provides a high-level introduction to grid architecture and technologies, and introduces potential

Grid Strategy Engagement

Grid Innovation Workshop

Grid DesignServicesProject 1

Grid ImplementationServicesProject 1

Grid DesignServicesProject 2

Grid DesignServicesProject n

Grid ImplementationServicesProject 2

Grid ImplementationServicesProject n

Grid Implementation Roadmap

Education and Training / Support services


opportunities (current processes, proposed processes, economic justification, business impact, and technological impact).

These workshops focus on designing a grid environment based on open standards, tailored for homogeneous and heterogeneous IT architectures, instead of being limited to certain software or hardware. This allows an optimization of the existing infrastructure, making the individual optimization of resources possible.

The main goal of this service is to have every stakeholder (IT representatives, product representatives, and decision makers) and IBM sitting at the same table, in order to be able to successfully outline an engagement for the steps of the grid implementation and to be able to successfully deploy the strategy that is outlined during this workshop.

For more information, refer to the following Web site:

http://www.ibm.com/grid/solutions/innovation_workshop.shtml

8.2.2 Grid Strategy EngagementGrid Strategy Engagement provides the tools for starting most typical grid engagements.

DescriptionThis service provides the initial steps necessary to evaluate a client's current IT environment and gives the client the information necessary to alter their overall IT strategy based on their business needs. This service is also key to typical follow-on engagements, where a more detailed grid design is developed and implemented.

The service addresses the following major areas of focus:

� Engagement and project management

� Understanding the client business environment and objectives

� Understanding the current IT environment and plans

� Future vision definition

� Development of grid applications, data, infrastructure, and management strategies

� Documentation of grid applications, data, infrastructure, and security and privacy policies and requirements

� Evaluation of identified applications, data stores, IT infrastructure, and plans

� Evaluation of the ability to manage a grid environment



� Review of security controls

� Assessment of application, host, and network security

� Evaluation of IT services, IT cost, and value, and the current strategy versus the grid strategy

� Evaluation of alternative solution approaches and organizational change readiness

� Initiative identification, justification, and phasing

� Transition planning

Using this service will allow the client to arrive at a logical and defensible plan for using grid technologies to support and enhance the client business plans. Positioned at the front of the full life cycle of grid services, it is followed by 8.2.3, “Grid Design Services” on page 126 and 8.2.4, “Grid Software Implementation Services” on page 127. It also provides a thorough review of all aspects of the client IT environment and is also a way of linking the client business strategy with a complete IT strategy, including the client IT and business organization: systems management, security, applications development, organizational change, testing, and business continuity.

8.2.3 Grid Design ServicesGrid Design Services provides the means for designing custom grid environments.

DescriptionThis service begins with the grid solution architecture overview and is narrowed to two progressively more detailed levels of design granularity:

1. First level design: The grid macro design provides a product free design that may also include the design of processes and a high level organizational structure.

2. Second level of design: The grid micro design is the physical or node level design that provides a more detailed product mapping overview of the grid architecture.

If processes are included in the scope of the design, the micro design will also include process workflow and a detailed design of the organization that will implement the processes. The intent of the grid micro design is to produce a design in enough detail to implement the grid solution.

These design phases provide the steps necessary to design a grid from the logical architecture level, down to a detailed design where vendor products are


selected. These steps will also provide tools to follow-on phases where the grid is implemented and operated.

Grid Design Services addresses the following major areas of focus for a successful grid design:


� Understanding the enterprise business environment and objectives as well as the current IT environment and plans

� Documenting grid application and infrastructure requirements

� Providing architectures describing grid applications, middleware, infrastructure, and security

� Designing macro level organization and technology solutions

� Development of physical models for applications, grid middleware, grid infrastructure, and grid management tools

� Conducting static testing

� Training and user support definition

� High level implementation planning

Positioned at the mid-point of the life cycle of grid services, following 8.2.2, “Grid Strategy Engagement” on page 125 and followed by 8.2.4, “Grid Software Implementation Services” on page 127, Grid Design Services provides a way of linking the grid infrastructure to other aspects of the enterprise IT and business organization at key design decision points, like systems management, security, applications development, organizational change, testing, and business continuity.

8.2.4 Grid Software Implementation ServicesGrid Software Implementation Services provide detailed planning and physical installation of a grid environment. It offers the ability to implement custom grid solutions using proven project planning principles.

DescriptionThe phases of the service provide the steps necessary to implement a grid environment, from the initial project planning, through piloting the environment, and completing with the migration into operations.


The grid implementation service addresses the following major areas of focus for a successful grid solution:


� Detailed implementation planning

� Non-production environment preparation

� Infrastructure management procedure preparation

� System components assembly

� Testing preparation

� Pilot deployment

� Transition planning

� Infrastructure and application customization

� Data migration

� Production deployment

� Engagement completion

� Delivery

Using this service will allow a client to implement previously planned applications modifications and install a grid infrastructure.

Positioned at the end of the life cycle of grid services following 8.2.2, “Grid Strategy Engagement” on page 125, and 8.2.3, “Grid Design Services” on page 126, Grid Software Implementation Services provides the tools to implement the grid environment achieved from the planning in previous services.

8.2.5 Grid Training and EducationGrid Training and Education is a service offered by IBM IT education services that provides clients with the know-how needed to successfully develop, implement, manage, and operate a grid environment. There are several grid related courses and lectures available. For more information, visit:

http://www.ibm.com/services/learning/us/catalog/

8.2.6 IBM support lineIBM offers enterprise-grade support for almost every component that can be used in a grid solution, via the Internet and voice support 24 hours a day, 7 days a week. This support ranges from answering usage questions to identifying problems, all revolving around the client’s needs. IBM offers world-class support as a standard offering, with several options, including the ability to customize




support according to the skills and experience of the client’s current IT staff. Support line is IBM’s support offering for e-infrastructure software. It is an established worldwide proven service that covers IBM and non-IBM software products. For more information about the IBM support line, refer to the following Web site:

http://www.ibm.com/services/its/us/supportline.html

8.3 Solutions portfolio for grid computingTable 8-2 provides an overview of the IBM grid solutions, pointing out the advantages for each offering.

Table 8-2 Grid solutions overview

Solution Description

IBM Grid Value at Work IBM's approach to grid value quantification

Analytics Acceleration Reduce time and effort required to analyze data; accelerate product development, client insight applications, and accelerate time-to-results

Design Collaboration Leverage idle hardware, speed data capture, and virtualize disparate systems

Engineering Design Reduce time to market, leverage idle hardware, speed data capture, and virtualize disparate systems

Geophysical Processing and Analysis

Improve efficiency, speed on seismic data analysis, and enhance competitiveness and agility

Information Accessibility Maximize exploitation of existing data resources and assets, achieve unified data access and simplified integration, secure and simplify data access, and optimize storage and integration of heterogeneous resources

IT Optimization Share large-scale raw data among researchers and across disciplines; simplify data access and integration in a flexible, extensible infrastructure

Research Collaboration Reduce overhead and increase capacity, flexibility, scalability, resiliency; exploit underutilized computing, storage, and human resources

Grid Clash analysis Optimize and accelerate clash analysis



8.3.1 IBM Grid Value at WorkBefore engaging a grid project, clients need to assess the value of their investment. IBM Business Consulting Services, using this offering, can provide quantifiable business value output to help businesses meet the challenges that face them.

DescriptionThis is an assessment offering. Its main goals are to predict the business value of the investment in a grid environment, validate the proposed grid model, determine the optimal grid blueprint, and monitor IT level indicators of financial value to efficiently manage the infrastructure.

IBM Grid Value at Work assists in developing a grid deployment roadmap. The roadmap includes components for total cost of ownership analysis, grid capacity planning, and business value estimation. It provides the flexibility to focus on one or more of these components to determine the appropriate investment levels for grid computing.

Additionally, the IBM Grid Value at Work service helps identify the grid’s most appropriate uses by analyzing the comprehensive business value of the implementation across all deployment stages and quantifying the following information:

� Business value of improving the speed, accuracy, frequency, or throughput of the analysis of results for business analytics

� IT infrastructure cost savings and potential benefits from faster time to market or improved design quality for research, engineering, and design

� Cost savings from virtualization of the IT infrastructure and value derived from exploiting existing, underutilized resources for IT infrastructure optimization

� Cost savings from integrated information access and shared usage of IT resources across departments and organizations

This offering provides a deep examination of the critical value components examined by an executive staff. In particular, it features four analysis modules that can be used collectively or individually:

1. Grid capacity planner: Helps to identify the applications that are best positioned to work in a grid

2. TCO estimator: Helps to identify the expected cost savings, as well as the expected expenses of a deployment

3. Business value estimator: Helps to identify opportunities for companies to extract the maximum value out of applications on a grid


4. Business case builder: Incorporates outputs from other modules within the Grid Value at Work tool to present a comprehensive, view of the proposed grid

With Grid Value at Work, companies can establish strong business cases to justify investment on a grid and help the clients to meet their business needs in a more competitive world.

8.3.2 Analytics AccelerationThe Analytics Acceleration offering allows businesses to run their existing analytics software, whether custom-built management systems, commercial applications, or a combination of both. It is an open and flexible architecture supporting a wide range of packaged and custom analytical applications. It can include SAS Credit Scoring Application software from SAS (http://www.sas.com/) and relies on grid middleware from DataSynapse to create distributed sets of virtualized resources.

DescriptionWorking with partners in its industry, IBM has built a grid offering that can deal with the most demanding applications today. Clients can now reach a new level of analysis with much more accurate results and yet with less investment in hardware resources.

This offering is particularly optimized for computing intensive tasks, mainly for the financial sector. Its main characteristic is that it intends to use every kind of hardware resources that the infrastructure provides: servers, clusters, mainframes, and desktops.

Figure 8-2 on page 132 provides an overview of a typical architecture for the Analytics Acceleration offering.


http://www.sas.com/

Figure 8-2 Analytics Acceleration architecture

8.3.3 Design CollaborationThe Design Collaboration offering aims to leverage idle hardware, speed data capture, and virtualize disparate systems.

DescriptionMost automobile and aerospace manufacturers use a leading edge design tool that has many complex features, but some partners may not want to purchase and maintain multiple design tools for different projects. Using another tool might require additional testing and analysis by design partners and the original equipment manufacturer, adding time and cost to the project. Also, most

ResourcesPools

SAS - Credit Scoring Application

ResourcesPools

SAS - Credit Scoring Application

Avaki / DataSynapse

Grid Servers

UnstructuredData

StructuredData

DB2 Information Integration


participants in the automotive and aerospace design process must work with restricted IT budgets, and some design partners may have limited access to highly trained IT experts.

This service provides tools to help clients in automotive and aerospace engineering benefit from a grid implementation, and it can help reduce the time required to capture, compile, and analyze research data and accelerate product development.

The solution also includes a Grid Innovation Workshop, a pilot implementation, and a complete portfolio of IBM Global Services solutions.

8.3.4 Engineering DesignIn an intensely competitive marketplace, automobile and aerospace companies must achieve faster time to market by decreasing the turnaround time for product design.

The Engineering Design offering aims to reduce time to market, leverage idle hardware, speed data capture, and virtualize disparate systems.

DescriptionEngineering Design uses the latest grid technologies, including the Open Grid Services Architecture (OGSA), which merges the open protocols used for grid computing with the protocols used for Web services. It also provides the latest software from leading grid software companies, such as Platform Computing, and the latest open source grid technologies from The Globus Alliance.

Figure 8-3 on page 134 provides an overview of a typical architecture for the Engineering Design offering.


Figure 8-3 Engineering Design architecture

8.3.5 Geophysical Processing and AnalysisAs processing requirements continue to grow, more and more petroleum companies are taking advantage of parallelization and clustering technologies. Grid computing can be used to facilitate application parallelization in which an application is divided into independent tasks that can then be executed concurrently, reducing the time to completion. Clusters link separate nodes to create a single, powerful, and scalable system, and grid technologies can help to increase the overall compute resource sharing and utilization in such clusters.

The Geophysical Processing and Analysis offering aims to improve efficiency and speed on seismic data analysis in order to enhance competitiveness and agility.


DescriptionIBM has a comprehensive approach to help determine the most appropriate combination of technologies for geophysical processing and analysis in the petroleum markets. WebSphere Web application servers, advanced computer cluster offerings, powerful database technologies, application integration approaches, and knowledge management techniques from IBM can help reduce the time and effort required to capture, compile, and analyze seismic and reservoir data, which can accelerate asset decision time.

Application parallelization can facilitate grid computing when used for geophysical processing and analysis. We use industry-standard messaging technologies, such as the Message Passing Interface, OpenMP, Parallel Virtual Machine, and pthreads, to parallelize applications.

IBM’s relationships with Platform Computing, DataSynapse, and United Devices allow clients to use a single product for application parallelization and distributed execution. These companies offer products that provide job and task-level scheduling for low-overhead job submission as well as for interactive computations. It also offers job and task-level schedulers, along with the Globus Toolkit, to support commercial application environments.

Figure 8-4 on page 136 provides an overview of a typical architecture for the Geophysical Processing and Analysis offering.


Figure 8-4 Geophysical Processing and Analysis architecture

8.3.6 Information AccessibilityThe Information Accessibility offering helps to reduce the time and effort required to capture, compile, and analyze research data, and do product development.

DescriptionIBM has scalable tools and systems to help life sciences organizations and governments benefit from a grid implementation. The offering can achieve a seamless integration of data and resources and leverage existing IT environments to provide centralized data access from multiple data sources.


WebSphere Information Integration family provides a virtualized data access for research, educational, and governmental entities by unifying and merging data from several heterogeneous data sources, and offering this data in a virtualized layer that applications can access without being concerned with where, how, and when this data is gathered and stored.

8.3.7 IT OptimizationIn a global economy, businesses increasingly rely on IT infrastructures that drive business strategy, evolution, and success. Streamlining these components and the supporting organizational processes is an essential yet complex undertaking, which can be accomplished through grid computing.

The IT Optimization offering helps to share large-scale raw data among researchers and across disciplines; it simplifies data access and integration.

DescriptionGrid computing can enable consolidation of applications, networks, and other data and can help reduce tension between existing systems and politics between departments. This new approach can help organizations share resources and improve resource management. Grid computing can and should be used to optimize existing traditional IT environments, leveraging management and monitoring automated provisioning and building a on demand computing environment. Using the tools and benefits provided by grid computing, clients can leverage their existing IT environment to a more productive level, increasing the return of the investment in these IT environments.

8.3.8 Research CollaborationIn response to the challenges of information access, research organizations and universities are redefining their information sharing methodologies, as well as redesigning their IT infrastructures to be more effective in the new research environment.

The Research Collaboration offering helps to reduce resource overhead and increase capacity, flexibility, scalability, and resiliency.

DescriptionGrid computing can help these organizations manage large-scale data sharing. This new approach helps maximize the use of existing data resources and makes both structured and unstructured data available across a department, organization, or consortium. In addition, grid computing can help researchers secure data access and optimize storage; plus, IBM has scalable tools and systems to help research organizations and universities benefit from a grid


implementation for research collaboration. To address researchers’ special security needs, this service also offers a complete line of security and systems management solutions to support a certifiable, multilevel security infrastructure using Tivoli Management products.

Figure 8-5 provides an overview of a typical architecture for the Research Collaboration offering.

Figure 8-5 Research Collaboration architecture


8.3.9 Grid Clash AnalysisWaiting until the product is complete to run clash tests creates delays in getting the product to market and limitations on the number of clash analysis jobs, which creates quality issues.

The Grid Clash Analysis offering helps to optimize and accelerate clash analysis.

DescriptionThis offering is designed to offer an increase of productivity to existing clash analysis environments. Typically, clash analysis is a very time consuming part of the design process for, for example, cars, aircraft, and other equipment. Using grid, IBM can provide a noticeable improvement to this process, providing increasing advantages to its clients.

Services for implementing the clash offering may include:

� Assessing the current usage of clash analysis in the client environment (Grid Strategy Assessment)

� Determining any software gaps (Grid Application Readiness Assessment)

� Planning the pilot implementation (see 8.2.1, “Grid Innovation Workshop” on page 124)

� Implementing the pilot using IBM Global Services custom code (Custom Application Development)

� Implementing performance enhancement steps in the grid scheduler environment (see 8.3.7, “IT Optimization” on page 137)

� Verifying performance results and documenting open issues (Testing Services)

� Transferring skills to client personnel (see 8.2.5, “Grid Training and Education” on page 128)

� Defining the next steps to implement a production grid (see 8.2.2, “Grid Strategy Engagement” on page 125)

8.3.10 SummaryFigure 8-3 on page 140 presents the availability of the above offerings’ availability by industry. Some of offerings are customized for specific industries and others are in fact cross-industry.


Table 8-3 Offering availability by industry

For complete list of the grid computing offerings, refer to the following Web site:

http://www.ibm.com/grid/solutions/index.shtml

8.4 Services approach to Grid ProjectsIBM applies the best methods and practices to access, plan, and implement the solutions and services designed.

These solutions and offering should be focused on the business need and should be addressed with a solution independent approach and oriented to offer the best business value throughout the project’s life.

Financial Services

Petroleumand Energy

AerospaceAutomotive Industry

Life Sciences,Research, and Government

Analytics Acceleration

Offering Available

Offering Available

Offering Available

Offering Available

Design Collaboration

Offering Available

Offering Available

Engineering Design

Offering Available

Geophysical Processing and Analysis

Offering Available

Offering Available

Information Accessibility

Offering Available

Offering Available

IT Optimization

Offering Available

Offering Available

Offering Available

Offering Available

Research Collaboration

Offering Available

Clash Analysis

Offering Available

IBM Grid Innovation Workshop

Offering Available

Offering Available

Offering Available

Offering Available



The evaluation of the grid middleware options should be focused on a “best tool for the job” basis and the use of this middleware should be balanced with the grid middleware ISV and IBM SWG Product Set to provide the best solution for client satisfaction.



Part 5 Appendixes

Part 5



Appendix A. Surrounding initiatives

This appendix provides the following:

� Initiatives that contribute to the grid community by sharing resources, programs, development tools, and open-source products.

A


World Community GridThe World Community Grid’s mission is to create the largest public computing grid benefiting humanity (see Appendix B, “World Community Grid” on page 149). The work is built on the belief that technological innovation combined with visionary scientific research and large-scale voluntaryism can change our world for the better. For more information, please refer to the following Web site:


Globus Alliance The Globus Alliance conducts research and development to create fundamental technologies for grid computing. The alliance is formed by a group of sponsors and collaborators from around the world. The core team is based at the Argonne National Laboratory and other worldwide institutions (see http://www.globus.org/about/team.html for more information).

The Globus Toolkit is being developed by the Globus Alliance and many others all over the world. For more information, please refer to the following Web site:


Global Grid ForumThe Global Grid Forum (GGF) is a community-initiated organization of thousands of active participants from industry and research. GGF's primary objectives in this organization is to promote development, deployment, and implementation of grid technologies through creation of technical specifications, user experiences, and implementation guidelines. For more information, please refer to the following Web site:

http://www.gridforum.org/




http://www.gridforum.org/

http://www.globus.org/about/team.html

OASIS and WSRF TCOrganization for the Advancement of Structured Information Standards (OASIS) is an international consortium that drives the development and adoption of e-business standards and technologies. For more information, please refer to the following Web site:

http://www.oasis-open.org/

The OASIS Web Services Resource Framework (WSRF) Technical Committee (TC) works on the definition of a generic, royalty-free, open framework for modeling and accessing stateful resources (required for grid computing) using Web services. For more information, please refer to the following Web site:

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsrf

W3CThe World Wide Web Consortium (W3C) is an international consortium that develops specifications and guidelines for Web technologies. For more information, please refer to the following Web site:

http://www.w3.org/

TeraGrid TeraGrid is a initiative to build and deploy the world's largest, distributed infrastructure for open scientific research. It combinations three programs: National Science Foundation (NSF) Terascale initiative: Terascale Computing System (TCS®), Distributed Terascale Facility (DTF) and Extensible Terascale Facility (ETF). For more information, please refer to the following Web site:

http://www.teragrid.org/http://www.nsf.gov/

Appendix A. Surrounding initiatives 147

http://www.oasis-open.org/


http://www.w3.org/

http://www.teragrid.org/

http://www.nsf.gov/


Appendix B. World Community Grid


� Informtion about the IBM World Community Grid initiative, whose purpose is to contribute to humanitarian projects by providing unused compute cycles through a world-scale grid.

� An overview of the grid architecture design and of the organization that would be the most efficient one to leverage the World Community Grid.

B


MotivationsOver the last few years, grid computing has established itself as an invaluable technology to speed up highly-demanding computing tasks. By joining together individual computing resources, it creates a large system with massive computational power. Despite its embryonic state, a number of sectors, such as life science, finance, geology, and industry, already benefit from this technology.

But beyond business concerns, grid computing also provides a way for volunteers to donate unused processing time on their computers, to solve large and complex problems for the benefit of humanity. Discovering treatments for killer viruses, delivering more accurate forecasts of looming natural disasters, and helping researchers identify cures for widespread diseases are only a few ways of how individual computers could be used in this scenario.

Being firmly convinced that grid computing is poised for philanthropic initiatives around the world, IBM is committed to providing the technical infrastructure required to support a philanthropic world-scale grid: the World Community Grid.The vision of this initiative is to create the world's largest public computing grid to tackle carefully selected humanitarian projects. The name was chosen to represent the endeavor's reach (worldwide), volunteer and scientific collaboration (community), and technical underpinnings (grid computing).

The motivation behind the World Community Grid is leadership. IBM and a group of leading philanthropic foundations will provide this leadership, crucial expertise, resources, and guidance, and the philanthropic missions of these foundations will be strengthened by the power of grid computing.

The official Web site of the World Community Grid can be found at:


Technical featuresTo match expectations and provide a high quality of service, the World Community Grid has the following properties:

� Desktop grid model� Easy to volunteer� High-performance� Reliable� Scalable� Secure� Entertaining



Desktop grid modelBasically, this translates into a grid architecture featuring:

� An Agent, that is, a program running on contributing computers to grid-enable them and take advantage of their unused processing cycles when idle

� A Server, which is in charge of dispatching the jobs and collecting the results

Easy to volunteerPrevious large-audience PC grids projects have established that most volunteers are driven, first, by the sense that they are contributing their resources to a good cause and, second, by the ease, both functional and financial, of contributing the PC's power.

High-performanceConsidering the complexity of the problems to be submitted by the research community, and the huge processing power their resolution implies, the World Community Grid must be designed to:

� Store very large amounts of data� Avoid bottlenecks and minimize latency time� Handle over one million of contributing computers from the launch

ReliableThe World Community Grid must be reliable in terms of:

� Availability of the platform

Arrangements must be made to avoid any interruption of service (server redundancy, 24x7 technical support).

� Dependability of the returned results

Computing tasks should be submitted a number of times, to several computers, and returned results compared with each other to make sure that faulty computations are discarded.

ScalableAs a large part of the 650 million PCs worldwide is expected to participate in the future, the World Community Grid must anticipate a far larger number of contributors. As a corollary, such a computing power increase implies that the World Community Grid will have the ability to handle several grid-enabled research projects, simultaneously deployed on the same platform.

Appendix B. World Community Grid 151

SecureSecurity aspects have to be taken into account for both the World Community Grid Agent and the servers:

� The Agent must not represent a security issue for contributing individuals in any way, and should contain a public key to enforce the validity of incoming data and computation code.

� On the other hand, the grid-enablement process of a research project should include an encryption phase of all data to be processed, as well as the computation code sent to the agents.

Of course, since the World Community Grid infrastructure is accessed from the Internet, drastic additional security measures must be considered to prevent any server from being compromised (firewall and other security devices).

EntertainingFinally, the World Community Grid should provide potential contributors with various collateral features to encourage participation, and to back up communication efforts aimed at advertising the World Community Grid:

� A few measures for the participants to evaluate their contribution:

– Total CPU time contributed

– Number of tasks completed

– An arbitrary system of points, based on the computing power of the device and on the time spent computing results

� A user-friendly interface for the agent, including a task execution progress bar and the measures described above; the Agent should also come with a screen saver displaying the computer’s World Community Grid-related activity.

� A Web site (http://www.worldcommunitygrid.org) to showcase the World Community Grid and to:

– Spread the word about the World Community Grid and about the research projects currently using it

– Register contributors and make the Agent available for download

– Provide additional entertaining features, some of them collaborative, such such as team creation, forums, global statistics, and so on


http://www.worldcommunitygrid.org


OrganizationAs presented in Figure B-1, four components can be identified that support the World Community Grid project: funding, technical resources, an advisory board of scientists and grid specialists, and a communication plan to promote the use of the World Community Grid by the widest possible population.

Figure B-1 World Community Grid - Basics

FundingThe World Community Grid will initially be sponsored by IBM and various foundations invited to join the effort, to cover the following activities:

� Infrastructure hosting with IBM Global Services, including all necessary hardware and software

� The development and ongoing maintenance of the World Community Grid Web site

� Application preparation and loading of the individual research projects onto the grid

� Ongoing support of the World Community Grid community through forums, online content, and e-mail

� Public relations and marketing activities to attract World Community Grid volunteers and research projects

The ultimate goal will be to establish an independent, self-sustaining, and not-for-profit organization.

Technical resourcesThese resources include hardware, software, and qualified people to operate them, as developed in the previous section.


World Community Grid Advisory BoardA board of advisors chaired by an IBM executive will search for research challenges that could improve the health and safety of human beings. This board will be composed of prominent scientists or representatives from research and philanthropic organizations, and of grid computing specialists and researchers who will provide their technical expertise. The advisory board will tackle difficult decisions: evaluating and prioritizing proposals to tap the grid, balancing pressing projects like the containment of new infectious agents against the need to make progress on longer-standing problems, and weighing the merits of “big science” proposals against requests of individual researchers. It will also develop specific evaluation criteria based on the overarching objective of improving human welfare.

Projects will be evaluated and prioritized based on several factors including:

Need Which proposed projects will help populations with the greatest needs?

Scope Which proposed projects might have the greatest impact, both in terms of breadth and depth?

Suitability Which proposals are best technologically suited for grid computing?

IBM's current guidelines for donations will serve as a starting point in determining the organizations that are eligible to apply for the computational power.

The list of members of the board can be found at:

http://www.worldcommunitygrid.org/about_us/advisory_board.html

Marketing and communicationIn order for the World Community Grid to become widely visible, a communication and media plan will be created and all supporting material will be provided by the different actors involved in the project.

High level designFrom the requirements above, it appears that the IT infrastructure to establish the World Community Grid is based upon three components:

� Clients� Servers� Administrators



From a global perspective, the World Community Grid architecture should resemble the one shown in Figure B-2.

Figure B-2 World Community Grid: High Level Solution Design

On the client sideLarge numbers of volunteers who are willing to download and install the grid desktop program, enabling them to donate the computing power of their idle PCs and servers.

On the server sideA large server infrastructure to act as the “brains” of the grid.

Grid Servers

World Community GridWeb Site Servers

Database ServersApplication Servers

Data

Data

Desktop PC

Engineering Workstation

World Community Grid Server Infrastructure

Fibre Attached Storage

Internet ConnectedWorld Community Grid

Volunteers

Laptop computer

Laptop computer


The Web serversThe infrastructure that hosts the http://www.worldgrid.org Web site, from where individuals and organizations can learn more about the projects, register and download the grid-participant program, and join Web-based forums to communicate with other volunteers.

In order to handle the heavy load that is expected once several millions of people are aware of the World Community Grid and start to use it frequently, it has been decided to split the treatment of static and dynamic pages between two groups of dedicated servers. The static pages, served by the front-end Web servers, contain informtion about the World Community Grid and on the research projects running on it, FAQs, troubleshooting tips, legal mentions, and so on. On the other hand, the dynamic pages will be served by the back-end application servers and deal mainly with user-oriented services, such as global, by project and individual statistics, member authentication, forums, and so on.

The grid serversThese servers will dispatch tasks to grid-enabled computers and will retrieve and compile the results. They will be in charge of making sure that a task has been completed and that the returned result is correct, for example, by submitting the same task to various computers running the World Community Grid Agent, and by comparing the returned results with one another.

For security reasons, the grid software running on these servers will also keep track of the submitted tasks so that only computing resources identified as valid devices registered through the agent interface are allowed to return a result.

The storage resourcesStorage is a crucial issue for the World Community Grid, and will have to handle two types of data.

� Desktop Grid data

This data can be categorized into application data and user data:

– Grid-enabled applications data

• Computation code

• Pre-processed projects input data

• Global statistics / project advancement

– Users data

• Project subscriptions

• Individual statistics


http://www.worldgrid.org

� Registered members information

These data originate from two sources:

– From World Community Grid contributors

• Through the World Community Grid registration form that every user has to fill in when installing the World Community Grid Agent (member name and password, e-mail address for password recovery and information updates, personal information, and address for statistic by geography)

• Through the device creation form (resource contribution schedule, type of internet connection, and so on)

• When using the World Community Grid collaborative features (team management, forums, and so on)

– From the grid database

• Consolidated individual statistics for integration into the World Community Grid Web site

In order to balance the load and minimize the risks of server failure, and because this data is accessed from distinct servers, the decision was made to distribute the data over two platforms, according to their type.

Figure B-3 on page 158 summarizes all the above considerations.


Figure B-3 World Community Grid: Architecture

The administration teamOf course, this platform cannot be installed, configured, hosted, and maintained without a team of qualified IT professionals, who will also take the applications identified by the World Community Grid advisory board and prepare them to run on the grid.

Products usedIn order to complete the design of the World Community Grid, hardware and software products have to be chosen for each architectural component.

World Community Grid Web ServersAccording to the architecture design, these servers are to be the front ends of the World Community Grid Web site; their purpose is to serve static pages only, and forward dynamic pages requests to the World Community Grid Application Servers.

For reasons of costs and maintenance, the selected servers were IBM xSeries, with Linux as the operating system.


As for the Web server software, it seemed a natural choice to go with WebSphere Application Server; this product may seem inappropriate, since it is to be used only for serving static pages, but on the other hand, it provides network deployment and administration features that are a definite value-add, especially in terms of implementation and hosting of the World Community Grid infrastructure. Moreover, it would not make much sense to choose different products for the front-end servers and the back-end servers.

World Community Grid Application ServersThe purpose of these servers is to relieve the front ends of all dynamic content, including database access and code execution.

For reasons listed above, these servers should be IBM xSeries running Linux as well.

The WebSphere Application Server is IBM's premier Web application server. It provides servlet and JSP™ processors, which includes support for JSP templates. It supports J2EE applications containing Enterprise Java Beans (EJBs) along with Bean and Container Managed Persistence (BMP/CMP). It also includes support for Web services through various protocols.

World Community Grid Storage ResourcesThese servers are to handle all data storage related activity for both the World Community Grid Web site and the grid-enabled applications.

Since a considerable amount of data is expected, the selected servers to host both databases were IBM Shark Enterprise Storage Servers running DB2 database software on AIX/pSeries technologies. In addition to their robustness and proven maturity, they fit perfectly in an all-IBM servers architecture, and can be easily managed by IBM Managed Storage Services (MSS).

MSS offers flexible and scalable storage networking solutions for midrange server environments. Storage capacity "on demand," backup/restore, and management services are available as fee-based services. Hosted storage resources receive the benefits of Storage Area Network (SAN) and Network Attached Storage (NAS) technologies as part of an integrated solution.


World Community Grid platformThese servers are the core of the World Community Grid. In order to make these servers homogenous with the rest of the infrastructure, they were also specified as xServers running Linux.

In terms of software, the grid infrastructure product that would be part of the World Community Grid had to demonstrate sizeable client counts and satisfy as many of the following criteria as possible:

� General considerations

– Good level of maturity, especially on open Internet usage

� Project-related considerations

– Ability to run multiple applications concurrently, depending on the contributor preferences

– Contributed resource evaluation system included (to keep track of the points)

– Collaborative features provided (teams)

� Agent features

– Non-intrusive technology

– Easy installation and configuration of software

– Low disk space, memory, and bandwidth required

– Ability to communicate through a firewall

– Screensaver

– Automatic update of the client software

� Security issues

– Protection against spoofing and man-in-the-middle attacks

– Communication secured by mutual handshake and encryption

After reviewing several products, and because previous collaborations had turned out to be definite success (in particular, the Smallpox Research Grid Project), IBM decided to trust the United Devices MP platform. For more information, please refer to the following Web site:

http://www.worldcommunitygrid.org/projects_showcase/projects_archive.html



Figure B-4 World Community Grid: Architecture with products

More informationTo conclude this presentation of the World Community Grid, we thought that it would be relevant to consider some aspects of the project from a contributor standpoint, in particular the visible part of the World Community Grid as it is today.

World Community Grid AgentThe Agent is the program that runs on every contributing resource. Its interface displays the progress of the task currently being processed, the estimated total points of the device, and its properties as a computing resource, among other information. Communications between the Agent and the World Community Grid


Servers are encrypted and authenticated using a public key system, which prevents falsification of data by third parties masquerading as legitimate users, as well as delivery of a computer virus. Additionally, the user can configure the Agent so that it does not use more hard disk than specified and can limit processing and communications to certain time windows.

Figure B-5 World Community Grid Agent Interface

Example of the World Community Grid Screen SaverWhen a contributing computer goes idle, the World Community Grid Screen Saver requisitions resources as permitted by the computer owner to resume its work, while providing graphical insights on the task being processed, as shown in Figure B-6 on page 163.


Figure B-6 World Community Grid Screen Saver

www.worldcommunitygrid.orgBelow are a few screen captures of the actual the World Community Grid Web site; this is solely intended as a glimpse of what kind of information should be provided to encourage potential participants, and to illustrate some of the previously discussed features.

The World Community Grid home page (Figure B-7 on page 164) gives quick access to the main sections of the Web site, and underlines the message the World Community Grid is conveying: “You can help change the world”.


Figure B-7 World Community Grid Web Site - Homepage

The World Community Grid Web site provides visitors with global in-depth statistics on the community contribution. These stats can be consolidated by various criteria, in particular by projects (see Figure B-8 on page 165).


Figure B-8 World Community Grid Web Site - Project Statistics

The first project is known as Human Proteome. Human Proteome, in short, is a study that includes a method of determining the final shape and function of proteins to ultimately combat a number of diseases. Human Proteome is not an end in itself, but rather it is a stepping stone to potentially provide a cure for a variety of diseases.

The World Community Grid Web site has a member area, for registered contributors to measure their individual contribution and manage their devices.


Statistics of the teams they belong to are also displayed in this area (see Figure B-9).

Figure B-9 World Community Grid: Member Area


ConclusionThe World Community Grid has been designed as an ambitious project from the start, and is likely to expand at a fast rate as more and more people around the world join the community of contributors.

In terms of measures, the goal is to enlist two million computers in the World Community Grid in the first year, running up to six applications, increasing that to eight million computers in the third year with more applications.

Successfully harnessing the value of grid computing on a grand, philanthropic scale requires three elements. Two of them – the technical infrastructure and the massive volunteer population – exist today. The third element, a guiding organization tapping these powerful forces for the greater good of humanity, is the World Community Grid.



Appendix C. Products reference


� A list of some products that can be part of any grid computing solution� A briefing of each product, including our analysis of its functionality� Links for more information about products

C


Big pictureTheoretically, any product can be used in a grid solution. The products presented here are the most popular ones or the grid-enabled ones. There is no product that by itself can provide a complete (or turn-key) grid solution; therefore, the product features must complement each other in order to build a grid environment. The product choice depends on the IT architect and the analysis of the functionality to match the architecture needs.

Products per functionalityThe following tables present the products distributed by functionality, but the products’ information is placed in alphabetic order.

Web services core/hostingWeb services core/hosting products presented in this appendix are listed in Table C-1 on page 171.

Note: There is no magic formula to consider if a product can be used in grid solutions or not. It depends on the functionality of the product, and the functionality will determine if the product is suitable for the environment.

Products covered in this redbook support (or not) the functions described in 1.2.2, “Functionality on grid computing” on page 16. Figures presented in this appendix illustrate our understanding of the functions each product can provide in a grid environment. It is not a prescription, but just a big picture about each product.

The width of each bar (0 to 3) reflects our perception of the importance of the product in a grid computing environment. Figures are to be read as follows:

� 0: The product has no features of this discipline.� 1: The product has some features of this discipline.� 2: The product has most of the features of this discipline.� 3: The product was developed and designed to completely cover the

discipline.

Important: The following Web site provides a search capability to access all IBM announcement letters, part numbers, sales manual, press releases, Redbooks, white papers, and catalogs:




Table C-1 Web Services Core/Hosting

Workload virtualizationWorkload virtualization products presented in this appendix are listed in Table C-2.

Table C-2 Workload virtualization

Information virtualizationInformation virtualization products presented in this appendix are listed in Table C-3.

Table C-3 Information virtualization

Products Distributor

WebSphere Application Server IBM

WebSphere Portal Server IBM

WebSphere MQ IBM

WebSphere Extended Deployment IBM


Tivoli WorkLoad Scheduler IBM

LoadLeveler IBM


WebSphere Information Integrator IBM

DB2 Content Manager IBM

IBM General Parallel File System (GPFS) IBM

TotalStorage SAN File System (SAN-FS) IBM

SAN Volume Controller (SVC) IBM

Appendix C. Products reference 171

Other disciplinesProducts for provisioning that are presented in this appendix are listed in Table C-4.

Table C-4 Provisioning

Products for security that are presented in this appendix are listed in Table C-5.

Table C-5 Security

Monitoring products for monitoring that are presented in this appendix are listed in Table C-6.

Table C-6 Monitoring

The products for licensing that are presented in this appendix are listed in Table C-7.

Table C-7 License management


Tivoli Provisioning Manager IBM

Tivoli Intelligent Orchestrator IBM


Tivoli Access Manager IBM


Tivoli NetView® IBM

Tivoli Monitoring IBM

Tivoli Management Framework IBM

Tivoli Enterprise™ Console IBM

Tivoli Service Level Advisor IBM


Tivoli License Manager IBM


DB2 Content ManagerDB2 Content Manager is a product for storing and managing digital media, such as images, video, sound, and non-structured data that has to be related in a structured way.

Figure C-1 and Table C-8 on page 174 present the importance of DB2 Content Manager in the grid computing environment.

Figure C-1 DB2 Content Manager

Other Disciplines






Table C-8 DB2 Content Manager

For more information, see:


GPFSIBM General Parallel File System (GPFS) is a file system designed to have high performance and high availability. Originally developed for RS/6000 SP clusters, it has been ported to Linux and is currently used as one of the first choices in grid solutions for distributed data storage.

Figure C-2 on page 175 and Table C-9 on page 175 present the importance of GPFS in the grid computing environment.

Discipline Description

Web Services Core/Hosting

No relevant features are available in this discipline.

Workload virtualization


Information virtualization

IBM DB2 Content Manager provides content management for images, graphics, video, Web content, and compliance documents. Apart from that, it also has the capability to provide imaging, digital asset management, Web content management, and content integration.

Other Disciplines No relevant features are available in this discipline.



Figure C-2 GPFS

Table C-9 GPFS


http://www.ibm.com/servers/eserver/clusters/software/gpfs.htmlhttp://www.ibm.com/servers/eserver/pseries/software/sp/gpfs.html







Allows parallel access to multiple different physical storage within the file system. It is designed as a high throughput file system with global name space.


Other Disciplines









LoadLevelerIBM LoadLeveler is a High Performance Computing job management system that allows users to optimize workload execution and performance on pool of IBM

pSeries machines by matching the jobs processing needs with the available resources. LoadLeveler schedules jobs in a HPC environment, and provides functions for submitting and processing jobs quickly and efficiently in a dynamic environment.

Figure C-3 and Table C-10 on page 177 present the importance of the IBM LoadLeveler in the grid computing environment.

Figure C-3 LoadLeveler

Other Disciplines






Table C-10 LoadLeveler


http://publib.boulder.ibm.com/clresctr/windows/public/llbooks.html

Tivoli Access ManagerTivoli Access Manager is a security product that provides a single point of user authentication and authorization administration. Access Manager provides a single point of authentication and authorization to Web-based resources, and provides standards-based APIs that allow Web application servers to access Tivoli Access Manager security services.

Figure C-4 on page 178 and Table C-11 on page 178 present the importance of Tivoli Access Manager in the grid computing environment.





Supports gang scheduling, which implements time sharing features that allows multiple jobs to execute simultaneously on a group of nodes, and also allows multiple jobs to timeshare on a single node.



Other Disciplines Provides GUI management for submitting jobs




Figure C-4 Tivoli Access Manager

Table C-11 Tivoli Access Manager





There are APIs and configuration tools.





Other Disciplines Implements a strong security framework that can be used in traditional IT and grid.

Other Disciplines







Tivoli Enterprise Console®Tivoli Enterprise Console is the focal point of events from all Tivoli products.

Figure C-5 and Table C-12 present the importance of Tivoli Enterprise Console in the grid computing environment.

Figure C-5 Tivoli Enterprise Console

Table C-12 Tivoli Enterprise Console








Other Disciplines Provides automated problem diagnosis and resolution to improve system performance and reduce support costs.

Other Disciplines







http://www.ibm.com/software/tivoli/products/enterprise-console/

Tivoli Intelligent OrchestratorTivoli Intelligent Orchestrator extends the benefits of the Tivoli Provisioning Manager. It intelligently and dynamically issues instructions to Tivoli Provisioning Manager, which then uses automation workflow to maintain server availability and meet required service levels. By constantly monitoring resources and requirements for anticipated peak workloads, it can then trigger the appropriate response.

Figure C-6 and Table C-13 on page 181 present the importance of Tivoli Intelligent Orchestrator in the grid computing environment.

Figure C-6 Tivoli Intelligent Orchestrator

Other Disciplines








Table C-13 Tivoli Intelligent Orchestrator



Tivoli License ManagerTivoli License Manager automates software licensing and utilization, and provides advanced reporting and inventory capabilities across an IT infrastructure.

Figure C-7 on page 182 and Table C-14 on page 182 present the importance of Tivoli License Manager in the grid computing environment.








Other Disciplines Combined with Provisioning Manager, Orchestrator enable grid environments with intelligent and dynamic provisioning, creating and managing an automatic provisioning infrastructure.




Figure C-7 Tivoli License Manager

Table C-14 Tivoli License Manager










Other Disciplines Provides automatic license management usage and also reporting and inventory.

Other Disciplines






http://www-306.ibm.com/software/tivoli/products/license-mgr/

Tivoli Management FrameworkTivoli Management Framework is the base component for the Tivoli product line. Using Tivoli Management Framework and a combination of Tivoli Enterprise applications, administrators can manage large distributed networks with multiple operating systems, various network services, diverse system tasks, and constantly changing nodes and users. Tivoli Management Framework provides a set of common services or features that are used by the Tivoli Enterprise applications installed on Tivoli Management Framework.

Figure C-8 and Table C-15 on page 184 present the importance of Tivoli Management Framework in the grid computing environment.

Figure C-8 Tivoli Management Framework

Other Disciplines






Table C-15 Tivoli Management Framework


http://www.ibm.com/software/tivoli/products/mgt-framework/

Tivoli Monitoring IBM Tivoli Monitoring provides the means to monitor several IT infrastructure resources. It detects bottlenecks and potential problems. It also provides automated best practices out of the box that proactively monitor critical hardware and software, including middleware, applications, and databases.

Figure C-9 on page 185 and Table C-16 on page 185 present the importance of Tivoli Monitoring in the grid computing environment.








Other Disciplines Builds a framework as a foundation for a suite of management applications that make systems and network management easy.




Figure C-9 Tivoli Monitoring

Table C-16 Tivoli Monitoring


http://www.ibm.com/software/tivoli/products/monitor/








Other Disciplines Provides monitoring for essential system resources.

Other Disciplines








Tivoli NetViewIBM Tivoli NetView extends traditional network management to ensure the availability of critical business systems and to provide rapid resolution of problems. It discovers TCP/IP networks, displays network topologies, correlates and manages events and SNMP traps, monitors network health, and gathers performance data.

Figure C-10 and Table C-16 on page 185 present the importance of Tivoli NetView in the grid computing environment.

Figure C-10 Tivoli NetView

Other Disciplines






Table C-17 Tivoli NetView


http://www.ibm.com/software/tivoli/products/netview/

Tivoli Provisioning ManagerIBM Tivoli Provisioning Manager is designed to help automate infrastructure deployment: It automates the manual provisioning and deployment process using pre-built "industry best practice" workflows to provide control and configuration of major vendor products. Tivoli Provisioning Manager automates manual tasks of provisioning and configuring servers, operating systems, middleware, applications, storage, and network devices.

Figure C-11 on page 188 and Table C-18 on page 188 present the importance of Tivoli Provisioning Manager in the grid computing environment.








Other Disciplines Extends traditional network management to ensure the availability of critical business systems and to provide rapid resolution of problems.




Figure C-11 Tivoli Provisioning Manager

Table C-18 Tivoli Provisioning Manager










Other Disciplines Developed to automate every provisioning task.

Other Disciplines








Tivoli SAN FSBased on the Storage Tank™ technology developed by IBM research, SAN FS is designed to provide a network-based heterogeneous file system for data sharing and centralized policy-based storage management in an open environment. The SAN File System provides a common file system for UNIX, Windows, and Linux servers, with a single global namespace to help provide data sharing across servers. It enables host systems to plug into a common SAN-wide file structure.

Figure C-12 and Table C-19 on page 190 present the importance of Tivoli SAN FS in the grid computing environment.

Figure C-12 Tivoli SAN FS

Other Disciplines






Table C-19 Tivoli SAN FS


http://www.storage.ibm.com/software/virtualization/sfs/index.html

Tivoli Service Level AdvisorTivoli Service Level Adviser is a service level management product that simplifies and automates the process of managing service level agreements. It provides the tools for proactively managing and reporting on service levels from across the management infrastructure.

Figure C-13 on page 191 and Table C-20 on page 191 present the importance of Tivoli Service Level Advisor in the grid computing environment.







SAN FS file system virtualization treats storage as a single, common pool with heterogeneous platform support. Storage space can be allocated and managed dynamically.





Figure C-13 Tivoli Service Level Advisor

Table C-20 Tivoli Service Level Advisor


http://www.ibm.com/software/tivoli/products/service-level-advisor/








Other Disciplines Designed to provide predictive service level management capabilities by enabling you to proactively predict when SLA violations are likely to occur and then take corrective actions to avoid an SLA violation.

Other Disciplines






http://www-306.ibm.com/software/tivoli/products/service-level-advisor/

Tivoli Workload SchedulerIBM Tivoli Workload Scheduler (ITWS) is a calendar based choreographer scheduler that automates, monitors, and controls the flow of work through an enterprise's entire IT infrastructure on both local and remote systems. From a single point of control, the suite analyzes the status of the production work and drives the processing of the workload according to installation business policies. It supports a multiple-end-user environment, enabling distributed processing and control across sites and departments within your enterprise.

Figure C-14 and Table C-21 on page 193 present the importance of Tivoli Workload Scheduler in the grid computing environment.

Figure C-14 Tivoli Workload Scheduler

Other Disciplines






Table C-21 Tivoli Workload Scheduler


http://www.ibm.com/software/tivoli/products/scheduler/

TotalStorage SAN Volume ControllerIBM TotalStorage SAN Volume Controller (SVC) is designed to reduce the complexity and costs of managing storage networks. It allows users to virtualize their storage and helps increase the utilization of existing capacity and centralizes the management of multiple controllers in an open-system SAN environment. It is designed to increase the flexibility of your storage infrastructure by enabling changes to the physical storage with minimal or no disruption to applications.

Figure C-15 on page 194 and Table C-22 on page 194 present the importance of TotalStorage SAN Volume Controller in the grid computing environment.





Provides an enterprise-level scheduling solution that enables clients to perform cross-platform, cross-domain, and cross-enterprise scheduling by integrating business applications across grid, mainframe, HPC, and traditional distributed environments.







Figure C-15 TotalStorage San Volume Controller

Table C-22 TotalStorage San Volume Controller









IBM SAN VC provides a virtualization layer between physical storages and manages them as a whole. Users can and might be transparently accessing data spread access different storages, giving better usage patterns.


Other Disciplines








Virtualization EngineVirtualization Engine (VE) provides a logical rather than physical view of data, computing power, storage capacity, and other resources, while automating the management of these resources based on business goals.

Figure C-16 and Table C-23 on page 196 present the importance of Virtualization Engine in the grid computing environment.

Figure C-16 Virtualization Engine

Other Disciplines






Table C-23 Virtualization Engine


http://www.ibm.com/servers/eserver/about/virtualization/?ca=vedemot&met=web&me=escallout

WebSphere Application ServerIBM has integrated grid computing capabilities into WebSphere Application Server. One of the key capabilities integrated is a “traffic-cop” load balancer, which routes WebSphere workload traffic across an application server cluster to deliver optimal performance. WebSphere Application Server provides a run-time environment that supports WSRF-compliant implementations of OGSA.

Figure C-17 on page 197 and Table C-24 on page 197 present the importance of WebSphere Application Server in the grid computing environment.



Has the IBM Grid Toolbox and is complemented by the WebSphere family of products.


Provides several scheduling components.


Implements some functionality through IBM Grid toolbox and can be complemented by WebSphere Information Integrator.

Other Disciplines Implements features like provisioning, security, monitoring, and resource management.


http://www-1.ibm.com/servers/eserver/about/virtualization/?ca=vedemot&met=web&me=escallout

Figure C-17 WebSphere Application Server

Table C-24 WebSphere Application Server


http://www.redbooks.ibm.com/redpapers/pdfs/redp3740.pdfhttp://www.ibm.com/websphere/



Designed to implement every feature in this discipline.





Other Disciplines Flexible management and security model with policies and authentication.

Other Disciplines






http://www.redbooks.ibm.com/redpapers/pdfs/redp3740.pdf

http://www.ibm.com/websphere/


WebSphere Extended Deployment WebSphere Extended Deployment is optional add-on software technology to WebSphere Application Server Network Deployment, delivering a new class of features for distributed platforms. This product allows an application to be split and run on different application servers, thereby enabling you to execute, in parallel, subsets of the application logic. As a result, the whole task can be executed faster than if it were executed in just a single box.

Figure C-17 on page 197 and Table C-24 on page 197 present the importance of WebSphere Extended Deployment in the grid computing environment.

Figure C-18 WebSphere Extended Deployment

Other Disciplines






Table C-25 WebSphere Extended Deployment


http://www.ibm.com/software/webservers/appserv/extend/features/http://www.ibm.com/websphere/

WebSphere Information IntegratorWebSphere Information Integrator is a product that provides the means to integrate information. It provides a SQL interface for applications that need to access distributed and heterogeneous data in virtually any data source.

Figure C-19 on page 200 and Table C-26 on page 200 present the importance of WebSphere Information Integrator in the grid computing environment.



Designed to implement every feature in this discipline.





Other Disciplines Flexible services distribution, management, and security model with policies and authentication.





Figure C-19 WebSphere Information Integrator

Table C-26 WebSphere Information Integrator









WebSphere Information Integrator is a data federation software that provides structured access to heterogeneous types of data.


Other Disciplines








WebSphere MQWebSphere MQ allows you to exchange information across different platforms, integrating new and existing business applications. WebSphere MQ Messaging provides any-to-any connectivity from desktop to mainframe, through business quality messaging, supporting over 35 platforms.

Figure C-20 and Table C-27 on page 202 present the importance of WebSphere MQ in the grid computing environment.

Figure C-20 WebSphere MQ

Other Disciplines






Table C-27 WebSphere MQ



WebSphere Portal ServerWebSphere Portal Server provides a single point of access to applications, application content, processes, and people in your network. It allows you to establish customized portals for your employees, Business Partners, and clients. The framework architecture implemented in this product provides a unified access point to internal and external Web applications, as well as portal access to other existing applications. In this way, users sign on to the portal and receive personalized Web pages.

Figure C-21 on page 203 and Table C-28 on page 203 present the importance of WebSphere Portal Server in the grid computing environment.





Can provide asynchronous message passing.



Other Disciplines Provides a messaging passing mechanism.




Figure C-21 WebSphere Portal Server

Table C-28 WebSphere Portal Server


http://www.ibm.com/software/genservers/portal/ http://www.ibm.com/websphere/



Portal capability on the top of an application server.





Other Disciplines Integration capability with other applications.

Other Disciplines






http://www-306.ibm.com/software/genservers/portal/



Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.

IBM RedbooksFor informtion about ordering these publications, see “How to get IBM Redbooks” on page 216. Note that some of the documents referenced here may be available in softcopy only.

� Fundamentals of Grid Computing, REDP-3613

� Globus Toolkit 3.0 Quick Start, REDP-3697

� Grid Computing with the IBM Grid Toolbox, SG24-6332

� Grid Computing on Research and Education, SG24-6649

� Grid Services Programming and Application Enablement, SG24-6100

� Introduction to Grid Computing with Globus, SG24-6895

Other publicationsThese publications are also relevant as further information sources:

� Abbas, Grid Computing: A Practical Guide to Technology and Applications, Charles River Media, 2003, ISBN 1584502762

� Berman, et al (editors), Grid Computing: Making The Global Infrastructure a Reality, Wiley, John & Sons, Incorporated, 2003, ISBN 0470853190

� Cockburn, Writing Effective Use Cases, Pearson Education, 2000, ISBN 0201702258

� Foster, et al (editors), The Grid 2: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers, 2003, ISBN 1558609334

� “GPFS: A Shared-Disk File System for Large Computing Clusters” by Frank Schmuck, Roger Haskin in Proceedings of the Conference on File and Storage Technologies, 2002

� Joseph, et al, Grid Computing, Prentice Hall Professional Technical Reference, 2003, ISBN 0131456601


Online resourcesThese Web sites and URLs are also relevant as further information sources:

� Accelrys home page

http://www.accelrys.com

� Altair Engineering

http://www.altair.com

� Apple: MacOS X

http://www.apple.com/macosx

� Cisco Systems

http://www.cisco.com

� Citrix home page

http://www.citrix.com

� Community Scheduler Framework

http://sourceforge.net/projects/gcsf

� Condor

http://www.cs.wisc.edu/condor

� The CrossGrid Project

http://www.eu-crossgrid.org

� The DataGrid Project

http://www.eu-datagrid.orghttp://www.eu-egee.org

� DataSynapse


� DataSynapse GridServer Technology

http://www.datasynapse.com/solutions/tech.html

� DB2 Content Management

http://www.ibm.com/software/data/cm

� WebSphere Information Integration

http://www.ibm.com/software/data/integration

� Database partitioning feature (DPF)



http://www.datasynapse.com/solutions/tech.html

http://www.cs.wisc.edu/condor

http://www.ibm.com/software/data/integration

http://www.ibm.com/software/data/cm

http://www.eu-crossgrid.org

http://www.eu-datagrid.org

http://www.eu-egee.org


http://www.accelrys.com

http://www.citrix.com

http://www.altair.com

http://sourceforge.net/projects/gcsf

http://www.cisco.com

http://www.apple.com/macosx



� DB2 Product Family

http://www.ibm.com/software/data/db2

� DCGrid Platform

http://www.entropia.com/dcgrid_platform.asp

� DCGrid Product Family

http://www.entropia.com/dcgrid_product_family.asp

� Design and implementation of an enterprise grid


� developerWorks: Analytics Acceleration Grid Environment, Part 1

http://www.ibm.com/developerworks/grid/library/gr-aage1/

� developerWorks: Analytics Acceleration Grid Environment, Part 2


� developerWorks: Build grid portals with Grid Portal Toolkit 3

http://www.ibm.com/developerworks/grid/library/gr-gridport/

� developerWorks: Community Scheduler Framework to implement grid meta-schedulers

http://www.ibm.com/developerworks/grid/library/gr-meta.html

� developerWorks: Community Scheduler Framework to create and submit grid jobs, Part 2

http://www.ibm.com/developerworks/grid/library/gr-meta2.html

� developerWorks: Enable existing applications for grid

http://www.ibm.com/developerworks/grid/library/gr-exist/

� developerWorks: Geographically dispersed grid, Part 1: Aligning computation and data

http://www.ibm.com/developerworks/grid/library/gr-gdg1/

� developerWorks: Globus Toolkit 4 Early Access: WSRF

http://www.ibm.com/developerworks/grid/library/gr-gt4early/

� developerWorks: Grid computing

http://www.ibm.com/developerworks/grid/

� developerWorks: Grid watch: Column summary

http://www.ibm.com/developerworks/grid/library/gr-watchcol.html

� developerWorks: The information grid: Article summary

http://www.ibm.com/developerworks/grid/library/gr-infocol.html

Related publications 207


http://www.entropia.com/dcgrid_product_family.asp

http://www.ibm.com/developerworks/grid/library/gr-meta.html

http://www.ibm.com/developerworks/grid/library/gr-meta2.html



http://www.ibm.com/developerworks/grid/library/gr-gt4early/

http://www.ibm.com/developerworks/grid/library/gr-gridport/

http://www.ibm.com/developerworks/grid/library/gr-gdg1/

http://www.ibm.com/developerworks/grid/library/gr-watchcol.html

http://www.ibm.com/developerworks/grid/library/gr-infocol.html

http://www.ibm.com/developerworks/grid/library/gr-exist/

http://www.ibm.com/developerworks/grid/

http://www.ibm.com/software/data/db2

http://www.entropia.com/dcgrid_platform.asp

� developerWorks: Orchestrating grid workloads -- neither feast nor famine

http://www.ibm.com/developerworks/grid/library/gr-feast/

� developerWorks: Six strategies for grid application enablement: Article summary

http://www.ibm.com/developerworks/grid/library/gr-sixcol.html

� developerWorks: zSeries and iSeries servers in the grid domain

http://www.ibm.com/developerworks/grid/library/gr-ziseries/

� Develop your grid service with the IBM Grid Toolbox

http://www.ibm.com/developerworks/grid/library/gr-develop

� Enterprise Identity Mapping

http://www.ibm.com/servers/eserver/security/eim

� Gartner home page

http://www.gartner.com

� General Parallel File System for AIX 5L


� General Parallel File System for Linux


� Global Grid Forum


� Globus Alliance


� Globus Alliance Team


� The Globus Toolkit

http://www-unix.globus.org/toolkithttp://www.globus.org/toolkit

� Globus Toolkit Security Documentation


� GRID.org

http://www.grid.org

� The Community Scheduler Framework

http://gcsf.sourceforge.net


http://www-unix.globus.org/toolkit


http://www.ibm.com/developerworks/grid/library/gr-develop

http://gcsf.sourceforge.net

http://www.ibm.com/developerworks/grid/library/gr-ziseries/

http://www.ibm.com/developerworks/grid/library/gr-feast/

http://www.ibm.com/developerworks/grid/library/gr-sixcol.html



http://www.grid.org

http://www.globus.org/toolkit



http://www.gartner.com


http://www.ibm.com/servers/eserver/security/eim

� Grid Engine

http://gridengine.sunsource.net

� GridFTP Protocol and Software

http://www-fp.globus.org/datagrid/gridftp.html

� GridFTP: Protocol Extensions to FTP for the Grid

http://www.globus.org/research/papers/GridftpSpec02.doc

� The Gridlab Project

http://www.gridlab.org/

� Grid MP Solutions Overview

http://www.ud.com/solutions

� Grid Security Infrastructure


� GUMS Certificate Management

http://www.atlasgrid.bnl.gov/testbed/gums/introduction.shtml

� GUMS Overview

http://grid.racf.bnl.gov/GUMS/guide_introduction.html

� Grid User Management Systems

http://www.atlasgrid.bnl.gov/testbed/gums

� HP-UX 11i home page

http://www.hp.com/products1/unix/operating

� IBM DB2 Information Integrator for Content: Overview

http://www.ibm.com/software/data/eip

� IBM Cluster software

http://www.ibm.com/servers/eserver/clusters/software

� IBM Software Information Center

http://publib.boulder.ibm.com/eserver/v1r1/en_US/index.htm?info/ogsainfo

� IBM Global Services

http://www.ibm.com/serviceshttp://www.ibm.com/igs

� IBM Grid computing: Business Partners


� IBM Grid computing: Solutions

http://www.ibm.com/linux/grid/solutions


http://www.gridlab.org/

http://www-fp.globus.org/datagrid/gridftp.html

http://www.atlasgrid.bnl.gov/testbed/gums


http://www.globus.org/research/papers/GridftpSpec02.doc

http://www.ud.com/solutions

http://www.ibm.com/servers/eserver/clusters/software

http://gridengine.sunsource.net

http://www.ibm.com/software/data/eip

http://www.atlasgrid.bnl.gov/testbed/gums/introduction.shtml

http://grid.racf.bnl.gov/GUMS/guide_introduction.html

http://publib.boulder.ibm.com/eserver/v1r1/en_US/index.htm?info/ogsainfo


http://www.hp.com/products1/unix/operating

http://www.ibm.com/services

http://www.ibm.com/linux/grid/solutions


� IBM Grid Innovation Workshop


� IBM Grid Solutions


� IBM Grid Toolbox


� IBM Informix

http://www.ibm.com/software/data/informix

� IBM IT Education Services

http://www.ibm.com/services/learning/us/catalog

� IBM LoadLeveler


� IBM Offering Information: User settings


� IBM Operational Support Services - Support Line


� IBM SAN File System: Overview

http://www.ibm.com/servers/storage/software/virtualization/sfs

� IBM SAN Volume Controller: Overview

http://www.ibm.com/servers/storage/software/virtualization/svc/index.html

� IBM System Journal - Design and implementation of an enterprise grid


� IBM Tivoli Access Manager for Business Integration

http://www.ibm.com/software/tivoli/products/access-mgr-bus-integration

� IBM Tivoli Access Manager for e-business

http://www.ibm.com/software/tivoli/products/access-mgr-e-bus

� IBM Tivoli Access Manager for Operating Systems

http://www.ibm.com/software/tivoli/products/access-mgr-operating-sys

� IBM Tivoli Directory Integrator


� IBM Tivoli Directory Server

http://www.ibm.com/software/tivoli/products/directory-server






http://www.ibm.com/servers/storage/software/virtualization/svc/index.html

http://www.ibm.com/servers/storage/software/virtualization/sfs

http://www.ibm.com/software/data/informix

http://www.ibm.com/software/tivoli/products/access-mgr-bus-integration

http://www.ibm.com/software/tivoli/products/access-mgr-operating-sys


http://www.ibm.com/services/learning/us/catalog


http://www.ibm.com/software/tivoli/products/access-mgr-e-bus


http://www.ibm.com/software/tivoli/products/directory-server


� IBM Tivoli Enterprise Console

http://www.ibm.com/software/tivoli/products/enterprise-console

� IBM Tivoli Intelligent Orchestrator

http://www.ibm.com/software/tivoli/products/intell-orch

� IBM Tivoli License Manager

http://www.ibm.com/software/tivoli/products/license-mgr

� IBM Tivoli Management Framework

http://www.ibm.com/software/tivoli/products/mgt-framework

� IBM Tivoli Monitoring

http://www.ibm.com/software/tivoli/products/monitor

� IBM Tivoli NetView

http://www.ibm.com/software/tivoli/products/netview

� IBM Tivoli Provisioning Manager

http://www.ibm.com/software/tivoli/products/prov-mgr

� IBM Tivoli Service Level Advisor

http://www.ibm.com/software/tivoli/products/service-level-advisor

� IBM Tivoli software

http://www.ibm.com/software/sw-bycategory/tivoli

� IBM Tivoli Storage Manager: Product Overview

http://www.ibm.com/software/tivoli/products/storage-mgr

� IBM Tivoli Storage Resource Manager

http://www.ibm.com/software/tivoli/products/storage-resource-mgr

� IBM Tivoli Workload Scheduler

http://www.ibm.com/software/tivoli/products/scheduler

� IBM Virtualization Solutions

http://www.ibm.com/servers/eserver/about/virtualization/?ca=vedemot&mhttp://www.ibm.com/servers/eserver/about/virtualization/index.htmlhttp://www.ibm.com/servers/eserver/about/virtualization/?ca=vedemot&met=web&me=escallout

� IBM WebSphere Developer Technical Journal: Building an Enterprise Service Bus with WebSphere Application Server V6 -- Part 1, found at:





http://www.ibm.com/software/tivoli/products/scheduler

http://www.ibm.com/software/tivoli/products/prov-mgr

http://www.ibm.com/software/tivoli/products/intell-orch

http://www.ibm.com/software/tivoli/products/monitor

http://www.ibm.com/software/tivoli/products/netview

http://www.ibm.com/software/tivoli/products/service-level-advisor

http://www.ibm.com/software/tivoli/products/license-mgr

http://www.ibm.com/servers/eserver/about/virtualization/?ca=vedemot&m

http://www.ibm.com/software/sw-bycategory/tivoli

http://www.ibm.com/software/tivoli/products/storage-resource-mgr

http://www.ibm.com/software/tivoli/products/storage-mgr


http://www-1.ibm.com/servers/eserver/about/virtualization/?ca=vedemot&met=web&me=escallout

http://www.ibm.com/software/tivoli/products/enterprise-console

http://www.ibm.com/software/tivoli/products/mgt-framework

� Install the DB2 UDB data partitioning feature on Linux


� InteGrade


� JDBC technology

http://java.sun.com/products/jdbc

� LoadLeveler 3.2.1 documentation updates


� Lotus Software

http://www.lotus.com

� Maui Scheduler


� Microsoft .NET

http://www.microsoft.com/net

� Microsoft Office Online

http://office.microsoft.com/en-us/default.aspx

� National Science Foundation

http://www.nsf.gov

� The National Science Foundation Middleware Initiative

http://www.nsf-middleware.org/

� Network File System (NFS) Version 4 Protocol


� NFS Version 4

http://www.nfsv4.org

� Nortel

http://www.nortel.com

� OASIS

http://www.oasis-open.orghttp://www.oasis-open.org/home/index.php

� OASIS Web Services Resource Framework (WSRF) TC




http://www.nsf-middleware.org/


http://office.microsoft.com/en-us/default.aspx

http://www.oasis-open.org

http://www.oasis-open.org/home/index.php

http://www.lotus.com


http://www.nfsv4.org

http://java.sun.com/products/jdbc

http://www.microsoft.com/net

http://www.nortel.com


http://www.nsf.gov



� OGSA-DAI project

http://www.ogsadai.org.uk

� OGSI::Lite and WSRF::Lite - Perl Grid Services


� OpenPBS

http://www.openpbs.org/about.html

� Oracle home page

http://www.oracle.com

� PBS Professional

http://www.pbspro.comhttp://www.openpbs.org/about_pbspro.html

� Platform Computing


� Platform Globus Toolkit


� Platform LSF

http://www.platform.com/products/LSF

� Platform LSF Family of Products

http://www.platform.com/products/LSFfamily

� Platform LSF License Scheduler

http://www.platform.com/products/LSF/addons.asp#license

� Platform Symphony


� Portable Batch System


� Python Globus

http://www-itg.lbl.gov/gtg/projects/pyGlobus

� SAN File Systems


� SAN Volume Controller

http://www.ibm.com/servers/storage/software/virtualization/svc

� SAS Institute, Inc.


http://www.platform.com/products/Globus

http://www.platform.com/products/LSF

http://www.openpbs.org/about.html

http://www.openpbs.org/about_pbspro.html


http://www.ogsadai.org.uk

http://www.ibm.com/servers/storage/software/virtualization/svc

http://www.platform.com/products/LSF/addons.asp#license



http://www-itg.lbl.gov/gtg/projects/pyGlobus

http://www.oracle.com


http://www.platform.com/products/LSFfamily

http://www.pbspro.com




http://www.sas.com/

� SETI@Home

http://setiathome.ssl.berkeley.edu

� The Smallpox Research Grid

http://www.grid.org/projects/smallpox

� Solaris Operating System

http://wwws.sun.com/software/solaris

� Sparsi


� SQL Server

http://www.microsoft.com/sql

� Stand-Alone Grid Security Infrastructure (GSI)


� Sun: Grid Computing Solutions

http://wwws.sun.com/software/grid

� Sybase home page

http://www.sybase.com

� Teragrid

http://www.teragrid.org

� Unicore

http://unicore.sourceforge.net

� United Devices

http://www.ud.com

� United Devices Grid MP

http://www.ud.com/solutions/fabs

� VMWare home page

http://www.vmware.com

� The Virtual Data Toolkit

http://www.cs.wisc.edu/vdt/

� WebSphere Application Server Performance Advisors Technology Preview

http://www.ibm.com/developerworks/websphere/downloads/performance_advisors.html




http://unicore.sourceforge.net

http://www.ud.com/solutions/fabs

http://www.sas.com/

http://www.cs.wisc.edu/vdt/

http://www.ud.com


http://setiathome.ssl.berkeley.edu

http://wwws.sun.com/software/grid

http://www.sybase.com

http://www.microsoft.com/sql


http://www.vmware.com

http://wwws.sun.com/software/solaris

http://www.teragrid.org

http://www.grid.org/projects/smallpox

� WebSphere Business Integration

http://www.ibm.com/developerworks/websphere/zones/businessintegration/bigpicture.html

� WebSphere Business Integration Message Broker

http://www.ibm.com/software/integration/wbimessagebroker

� WebSphere Business Integration Server

http://www.ibm.com/software/integration/wbiserver

� WebSphere Extended Deployment

http://www.ibm.com/software/webservers/appserv/extend/features

� WebSphere MQ

http://www.ibm.com/software/integration/wmq

� WebSphere Portal for Multiplatforms

http://www.ibm.com/software/genservers/portal

� WebSphere Product Overview


� WebSphere software

http://www.ibm.com/software/websphere/index.htmlhttp://www.ibm.com/websphere

� World Community Grid

http://www.worldcommunitygrid.orghttp://www.worldgrid.org

� World Community Grid: Advisory Board


� World Community Grid: Project Archive


� World Wide Web Consortium (W3C)

http://www.w3.org

� The WS-Resource Framework

http://www.globus.org/wsrf

� WSFRF.NET



http://www.ibm.com/websphere

http://www.ibm.com/software/genservers/portal

http://www.ibm.com/developerworks/websphere/zones/businessintegration/bigpicture.html

http://www.ibm.com/software/integration/wmq


http://www.ibm.com/software/integration/wbimessagebroker




http://www.ibm.com/software/integration/wbiserver

http://www.ibm.com/software/webservers/appserv/extend/features

http://www.globus.org/wsrf

http://www.w3.org


http://www.worldgrid.org


How to get IBM RedbooksYou can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site:

ibm.com/redbooks

Help from IBMIBM Support and downloads

ibm.com/support

IBM Global Services

ibm.com/services




http://www.ibm.com/support/

http://www.ibm.com/support/



Glossary

API. Application Programming Interface.

CIFS. Common Internet file system, the protocol formerly known as Server Message Block (SMB) and, before that, as the BAF protocol (after its original creator, Dr. Barry Feigenbaum). CIFS is a protocol for file and device sharing across a network.

CMM. Common Management Models.

DAS. Direct Attached Storage.

DHCP. Dynamic Host Configuration Protocol is a client-server networking protocol. A DHCP server provides configuration parameters specific to the DHCP client host requesting, generally, information required by the host to participate on the Internet network. DHCP also provides a mechanism for allocation of IP addresses to hosts.

DNS The Domain Name System is a system that stores information about host names and domain names on networks, such as the Internet.

EJB. Enterprise Java Beans, a set of JavaBeans™, currently being developed by Sun to support development of distributed applications.

FTP. File Transmission Protocol, a standard Internet protocol to exchange files on top of TCP/IP.

GCC. GNU Compiler Collection, which currently contains front ends for C, C++, Objective-C, Fortran, Java, and Ada, as well as libraries for these languages (libstdc++, libgcj, and so on).

GDC. Grid Domain Controller.

GGF. The Global Grid Forum was founded in 2001when the merger of regional grid organizations created a single worldwide one.

GIIS. Grid Index Information Services.

© Copyright IBM Corp. 2005. All rights reserved.

Globus MDS. Monitoring and Discovery System.

Globus MJS. Managed Job Service.

Globus RFT. Reliable File Transfer.

Globus UHE. User Hosting Environment.

Globus. A collaborative project centered at Argonne National Laboratory that is focused on enabling the application of grid concepts to computing.

GNU. GNU’s Not Unix.

GPFS. General Parallel File System is a type of mountable networked file systems.

GRAM. Globus Resource Allocation Manager, which requests and uses remote system resources for jobs.

GRIS. Grid Resource Information Service.

GSI. The Grid Security Infrastructure contains components to secure your grid network.

HPC High Performance Computing consists of computing applications on (parallel) supercomputers and computer clusters.

HTC. High Throughput Computing.

HTML. Hypertext Markup Language, which is a document type consisting of text and tags.

HTTP. Hypertext Transfer Protocol, which is an application protocol.

JDBC. Java Database Connectivity. Similar to ODBC, this set of application programming interfaces (APIs) provides a standard mechanism to allow Java applets access to a database.

217

JMS. Java Messaging System.

JSP. Java Server Pages, which is an extensible Web technology that uses static data, JSP elements, and server-side Java objects to generate dynamic content for a client.

LDAP. Lightweight Directory Access Protocol. IETF term for an TCP/IP based network protocol that enables access to a Directory System Agent.

MMJFS. Managed Job FactoryService, which is a job manager that provides an interface for requesting and using remote resources for the jobs.

MVC. Model View Controller.

NAS. Network Attached Storage. NAS is storage in a form that is readily accessible on a network, as opposed to direct-attached storage.

NFS. Network File System. A client/server application designed by Sun Microsystems™ that allows all network users to access shared files stored on computers of different types.

OASIS. Organization for the Advancement of Structured Information Standards.

ODBC. Open Database Connectivity. Open standard developed by Microsoft aimed at providing a consistent interface for relational databases, regardless of which database engine is used (for example, SQL Server, Oracle, or MySQL).

OGSA. The Open Grid Services Architecture is a standard that sets the basis for communication in grids across virtual organizations. OGSA marries open standards and grid computing protocols with Web Services, bringing together the ability to share computing resources with the ability to provide application interoperability over the Internet.

OGSA-DAI. Open Grid Services Architecture - Data Access Integration. A project developed by the UK Database Task Force whose objective is to provide a standard interface for a distributed query processing system to access data in different databases.

OGSI. Open Grid Services Infrastructure. A minimal set of grid services and supporting infrastructure on top of which OGSA systems and applications can be built or extended.

P2G. Peer to Group.

Peer to Peer .A network that does not rely on dedicated servers for communication but instead mostly uses direct connections between clients (peers). A pure peer-to-peer network does not have the notion of clients or servers, but only equal peer nodes that simultaneously function as both "clients" and "servers" to the other nodes on the network.

RLS. Replica Location Service.

SAN. A Storage Area Network is a high-speed special-purpose network (or subnetwork) that interconnects different kinds of data storage devices with associated data servers on behalf of a larger network of users.

SDK. Software Development Kit. A programming package that enables a programmer to develop applications for a specific platform.

SMB. Server Message Block. A file sharing protocol used in DOS, OS/2®, and early versions of Windows.

SOA. Service-Oriented Architecture is an architecture that represents software functionality as discoverable services on the network.

SOAP. Simple Object Access Protocol. A method invented by Microsoft to use RPC over the internet via HTTP calls.


SQL. Structured Query Language. A type of programming language used to construct database queries and perform updates and other maintenance of relational databases. SQL is not a full-fledged language that can create stand-alone applications, but it is strong enough to create interactive routines in other database programs.

SSL. Secure Socket Layer is a security protocol that provides communication privacy. SSL enables client/server applications to communicate in a way that is designed to prevent eavesdropping, tampering, and message forgery. SSL was developed by Netscape Communications Corp. and RSA Data Security.

VO. Virtual Organization. A virtual entity whose users and servers are geographically apart but share their resources collectively as a larger grid. The users of the grid can be organized dynamically into a number of virtual organizations, each with different policy requirements.

VPN. Virtual Private Network. A network that is constructed by using public wires to connect nodes, using encryption and other security mechanisms to ensure that only authorized users can access the network and that the data cannot be intercepted.

WAN. Wide Area Network. It can be made up of interconnected smaller networks spread throughout a building, a state, or the entire globe.

Web Services. A way of providing computationalcapabilities using standard Internet protocols andarchitectural elements

WSRF. Web Services Resource Framework. Defines a generic and open framework for modeling and accessing stateful resources using Web services.

XML. Extensible Markup Language. Modification of the SGML standard. In contrast to SGML documents, XML documents may exist without having their schema described in a document type definition.

Glossary 219


Index

AAccess Control Lists

see ACLACL 6AFS 70AIX 159Altair PBS Professional 54API 13, 18, 76, 177ARM 108authentication 6, 16, 92–94, 177authorization 6, 16, 92, 94, 177

Bbilling 15, 102

CCA 96certificates 93, 95, 97certification authorities

see CAcluster 11, 43, 47, 135, 196Condor 56CSF 57

DData grid 9Database 80Database partitioning feature (DPF) 80DataSynapse GridServer 48, 116, 135, 173DB2 Content Manager 80DB2 Information Integrator for Content 81DB2 Product Family

Database partitioning feature (DPF) 80DB2 Content Manager 80DB2 Information Integrator for Content 81DB2 UDB 78–79

DBMS 78deep computing

see HPCDesktop grid 45, 59, 150Directed Attached Storage

see DAS

© Copyright IBM Corp. 2005. All rights reserved.

distributed peer to peer grid see Network grid

EEnterprise Identity Mapping 96EWLM 108

Ffederated database 70File System Virtualization Layer 70, 75File systems

AFS 70NAS 70SAN 70, 75

firewall 152, 160FTP 10Functionality

Billing and metering 85Directory Services 85Information Virtualization 16–17, 19Information virtualization 65Licensing 85Other Disciplines 16, 20Provisioning and orchestration 85Scheduling 16, 19Security 85Web Services Core/Hosting 16, 18, 23Workload Virtualization 41

GGang Scheduler 43General Parallel File System

see IBM General Parallel File SystemGGF 114Global Grid Forum 122Globus Alliance 122, 133Globus Toolkit 92, 97, 116, 135, 174GPFS

see IBM General Parallel File SystemGrid computing models

Data grid 9Desktop grid 45, 59

221

Multipurpose grid 11Network grid 9Server grid 7, 45, 60, 114

grid delivery see Network grid

Grid Engine 44Grid stack product’s portfolio

Application 13Billing and metering 14Content management 14File and block data virtualization 16Grid middleware 16Job scheduling 15Orchestration and provisioning 14Software licensing 15Structured data virtualization 15System management 15Task scheduling 14Workload management 14

GridFTP 45GSI 95

HHigh Performance Computing

see HPCHigh Throughput Computing

see HTCHPC 11, 47, 56HTC 56

II3C 122i5/OS 107IBM AIX 107IBM Cluster Systems Management 91IBM DB2 109, 159IBM DB2 Content Manager 80IBM DB2 Information Integrator for Content 81IBM Director Multiplatform 109IBM downloadGrid 10IBM Enterprise Workload Manager 108IBM eServer pSeries 47, 176IBM eServer xSeries 72, 158IBM General Parallel File System 70, 75–76, 171, 174IBM Global Services 122, 133, 153IBM Grid Toolbox 93, 97, 109IBM LoadLeveler 47, 171, 176

IBM Managed Storage Services 159IBM Shark Enterprise Storage Servers 159IBM Tivoli Access Manager 94, 172, 177IBM Tivoli Directory Integrator 98IBM Tivoli Directory Server 98IBM Tivoli Enterprise Console 172, 179IBM Tivoli Intelligent Orchestrator 180IBM Tivoli Intelligent Think Dynamic Orchestrator 88, 172IBM Tivoli License Manager 101, 172, 181IBM Tivoli Management Framework 172, 183IBM Tivoli Monitoring 172, 184IBM Tivoli NetView 172, 186IBM Tivoli Provisioning Manager 88, 108–109, 172, 187IBM Tivoli Service Level Advisor 172, 190IBM Tivoli Storage Manager 72IBM Tivoli Storage Resource Manager 74IBM Tivoli Suite 15IBM Tivoli WorkLoad Scheduler 45, 171, 192IBM TotalStorage Productivity Center 110IBM TotalStorage SAN File System 9, 77, 110, 171, 189IBM TotalStorage SAN Volume Controller 72, 110, 171, 193IBM Virtualization Engine 106, 195IBM Virtualization Engine Suite for Servers 108IBM WebSphere Application Server 196IBM WebSphere MQ 201IBM WebSphere Portal Server 202Independent Software Vendor

see ISVInformation grid

see Data gridInteroperable Informatics Infrastructure Consortium

see I3CISV 20, 99

DataSynapse 48, 116, 131Platform Computing 50, 114, 133, 135United Devices 52, 59, 117, 135

JJava

EJB 159J2EE 94, 159JSP 159servlet 159


KKerberos 76

LLDAP 81, 92license management 15Linux 72, 83, 107, 158, 160, 189load balancing 44, 60

MMaui Scheduler 58Message Passing Interface 135metadata 77meta-scheduler 43–44metering 15, 102Multipurpose grid 11

NNAS 70–71, 74, 83, 159Network grid 9NFS 70, 75–76, 83

OOGSA 114, 116, 133, 196on demand 11–12, 72–74, 77, 122, 159open source 56open standards 4, 11, 94, 122OpenMP 135OpenPBS 55, 62orchestration 15, 87

PP2P 10Parallel Virtual Machine 135pay per usage 12peer-to-peer

see P2PPlatform Computing 133, 135Platform Load Sharing Facility

see Platform LSFPlatform LSF 50, 114Platform LSF Analytics 116Platform LSF HPC 51, 115Platform LSF License Scheduler 116Platform LSF MultiCluster 51, 115Platform LSF Reports 116

Platform Symphony 52, 115Products

Altair PBS Professional 54Condor 56CSF 57DataSynapse GridServer 48, 116, 135, 173Enterprise Identity Mapping 96Globus Toolkit 92, 116, 135, 174IBM Cluster Systems Management 91IBM DB2 109, 159IBM DB2 Content Manager 80, 171, 173IBM DB2 Information Integrator for Content 81IBM Director Multiplatform 109IBM Enterprise Workload Manager 108IBM General Parallel File System 76, 171, 174IBM Grid Toolbox 93, 109IBM LoadLeveler 47, 171, 176IBM Tivoli Access Manager 94, 172, 177IBM Tivoli Directory Integrator 98IBM Tivoli Directory Server 98IBM Tivoli Enterprise Console 172, 179IBM Tivoli Intelligent Orchestrator 180IBM Tivoli Intelligent Think Dynamic Orchestra-tor 88, 172IBM Tivoli License Manager 101, 172, 181IBM Tivoli Management Framework 172, 183IBM Tivoli Monitoring 172, 184IBM Tivoli NetView 172, 186IBM Tivoli Provisioning Manager 88, 108–109, 172, 187IBM Tivoli Service Level Advisor 172, 190IBM Tivoli Storage Manager 72IBM Tivoli Storage Resource Manager 74IBM Tivoli WorkLoad Scheduler 45, 171, 192IBM TotalStorage Productivity Center 110IBM TotalStorage SAN File System 9, 77, 110, 171, 189IBM TotalStorage SAN Volume Controller 72, 110, 171, 193IBM Virtualization Engine 106, 195IBM Virtualization Engine Suite for Servers 108Maui Scheduler 58NFS 60, 76OpenPBS 55, 62Platform LSF 50, 114Platform LSF Analytics 116Platform LSF HPC 51, 115Platform LSF License Scheduler 116Platform LSF MultiCluster 51, 115

Index 223

Platform LSF Reports 116Platform Symphony 52, 115United Devices 52United Devices MP 53, 117, 160Web Services Core/Hosting

IBM WebSphere Application Server 196IBM WebSphere MQ 201IBM WebSphere Portal Server 202

WebSphere Application Server 30, 92, 159, 171WebSphere Business Integration Messaging Broker 38WebSphere Extended Deployment 32, 171, 198WebSphere Information Integration 84, 137, 171, 199WebSphere Information Integrator 79WebSphere Information Integrator Content Edi-tion 79WebSphere Information Integrator OmniFind Edition 79WebSphere Messaging Resources 37WebSphere MQ 171WebSphere MQ Family 36WebSphere MQ Messaging 36WebSphere Portal Server 33, 171

provisioning 15, 86, 88, 107–108, 110, 187pthreads 135

QQoS 6, 58quality of service

see QoS

RRedbooks Web site 216

Contact us xxiresource pool 86

SSAN 71, 74, 83, 110, 159, 193SAN Volume Controller

see SVCSDK 13, 118Server grid 7, 45, 60, 114Service Level Agreement

see SLAServices

Grid Design Services 123, 126Grid Innovation Workshop 123–124Grid Software Implementation Services 123, 127Grid Strategy Engagement 123, 125Grid Training and Education 123, 128IBM Managed Storage Services 159IBM Support Line 128

Shared ResourcesCPU 5databases 5disks 5memory 5network 5storage devices 5

single sign-on 94–95SLA 14Smallpox Project 160SMB 83Software Development Kit

see SDKSolutions

Analytics Acceleration 129, 131, 140Clash Analisys 140Design Collaboration 129, 132, 140Engineering Design 129, 133, 140Geophysical Processing and Analysis 129, 134, 140Grid Clash Analysis 129, 139IBM Grid Innovation Workshop 140IBM Grid Value at Work 129–130Information Accessibility 129, 136, 140IT Optimization 129, 137, 140Research Collaboration 129, 137, 140

Storage Area Network 9 see SAN

Storage Virtualization Layer 71

TTCO 130

UUMI 12United Devices 52, 59, 135United Devices MP 53, 117, 160UNIX 83, 92, 189Utility Management Infrastructure

see UMI


Vvirtual computer 4virtual organization 7, 93virtual private network

see VPNVMWare 107VPN 90, 93

WWeb services 133WebSphere Application Server 30, 92, 159, 171WebSphere Business Integration Messaging Broker 38WebSphere Extended Deployment 32, 171WebSphere Information Integration 78WebSphere Information Integrator 79WebSphere Information Integrator Content Edition 79WebSphere Information Integrator OmniFind Edition 79WebSphere Messaging Resources 37WebSphere MQ 171WebSphere MQ Family 36WebSphere MQ Messaging 36WebSphere Portal Server 33, 171workload management 15, 45, 53, 87, 107–108World Community Grid 149, 151–154, 158, 167

Agent 161Screen Saver 162Servers 161

world-scale grid 150WSRF 196

XXML 15

Zz/OS 107

Index 225


(0.5” spine)0.475”<

->0.875”

250 <->

459 pages


®

SG24-6650-00 ISBN 0738491780

INTERNATIONAL TECHNICALSUPPORTORGANIZATION

BUILDING TECHNICALINFORMATION BASED ONPRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.

For more information:ibm.com/redbooks

Products and Services for Grid Computing

Grid computing functionality

Products for grid computing

Services and solutions

This IBM Redbook is part of a series of documents related to grid computing that IBM is presenting to the IT community to enrich it and all its players: clients, industry leaders, emerging enterprises, universities, and producers of technology. It is mainly oriented to IT architects or those who have the responsibility of analyzing the capabilities of the product used to build a grid solution.

We describe the functions of grid computing and the categorizations of the components within it. IBM, ISVs, and open source products will be explained. Some of these products are not specific for grid, but they can be used in a grid computing environment. We hope that this redbook helps you select functions and products and shows you how grid can fit into your IT picture as new products and services for grid are introduced.

Part 1 introduces the concept of grid computing and provides the terminology the readers will use in this redbook.

Part 2 explores the functionality associated with certain products and product families. It also presents some examples of the utilization of these products in the grid computing environment.

Part 3 presents grid computing product families that can be used to build a complete grid solution. These bundles, or suites, often implement most of the core grid disciplines.

Part 4 gives an overview of the grid computing offerings and services that IBM provides.

Back cover




Documents

Grid Computing Products and Services - IBM Redbooks · 8.3.9 Grid Clash Analysis ... B-3 World Community Grid: Architecture ... Joao Almeida Fabiano Lucchese Grid . Grid Computing