Upload
chastity-shaw
View
221
Download
1
Tags:
Embed Size (px)
Citation preview
Grid Computing Research @ Melbourne: Gridbus Perspective
Dr. Rajkumar Buyya
Grid Computing and Distributed Systems (GRIDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, Australiawww.gridbus.org
Gridbus Sponsors
2
Grids @ Various Schools in Melbourne:Technology Creation -> Applications
Faculty of Engineering – CS/EE/Geomatics GRIDS Lab and the Gridbus Project – technology focus SensorWeb (NICTA) Natural Language Processing Virtual organisation for tsunami research
Faculty of Science - Physics Belle and LHC Grid Australian Virtual Observatory
Faculty of Medicine – HFI, Anatomy MRI Grid Distributed Kidney Modelling
Social Sciences/Humanities The development of a collaborative distributed environment for humanities
research projects whose primary data takes the form of ethnographic audiovisual media.
A Grid-enabled technology for the sharing of graphics, audio, video and text between social science research groups working internationally.
Information Tech. Services Division MARC (Melbourne Advanced Research Computing)/HPC
3
Classes of Grid Services
Computational Services – CPU cycles Pooling computing power: SETI@Home, TeraGrid,
AusGrid, ChinaGrid, IndiaGrid, UK Grid,… Data Services
Collaborative data sharing generated by instruments, sensors, persons: LHC Grid, Napster
Application Services Access to remote software/libraries and license
management—NetSolve Interaction Services
eLearning, Virtual Tables, Group Communication (Access Grid), Gaming
Knowledge Services The way knowledge is acquired, processed and
managed—data mining. Utility Computing Services
Towards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.
Computational Grid
Data Grid
ASP Grid
Interaction Grid
Knowledge Grid
Utility Grid
infra
stru
ctu
re
Users
4
Grid Challenges
Security
Resource Allocation & Scheduling
Data locality
Network Management
System Management
Resource Discovery
Uniform Access
Computational Economy
Application Construction
5
GRIDS Lab @ Melbourne
Youngest and one of the rapidly growing research labs in our School/University:
Founded in 2002 Houses:
Research Fellows/PostDocs Research Programmers PhD candidates Honours/Masters students
Funding National and International organizations Australian Research Council & DEST Many industries (Sun, StorageTek, Microsoft,
IBM, Microsoft) University-wide collaboration:
Faculties of Science, Engineering, and Medicine
Many national and international collaborations.
Academics Industries
Software: Widely in academic and industrial users.
Publication: GRIDS Lab produces over 20% of our Dept’s
research output.
EducationR & D
+ Community Services: e.g., IEEE TC for Scalable Computing
6
Selected Funded Projects: 2004-2006
Australian Research Council: Discovery Projects:
"The Development of Data Grids for High Energy Physics" "Economic Scheduling for Efficient Management of Clusters and
their Cooperative Federation" Application Projects:
"Development of e-Research Tools for an MRI Grid Computing Facility"
"e-Research Grid Environment for Integration of Distributed Kidney Models and Resources”
7
The Gridbus Project @ Melbourne:Enable Leasing of ICT Services on Demand
WWG
Pushes Grid computing into mainstream
computing
Gridbus
8
The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying e-Research Applications on Utility Grids
The Gridbus Project @ GRIDS Lab, The University of Melbourne: The Gridbus Project @ GRIDS Lab, The University of Melbourne: Toolkit for Creating and Deploying eToolkit for Creating and Deploying e--Research Applications on Utility GridsResearch Applications on Utility Grids
Gridbus
Distributed Data
http://www.gridbus.org
• Gridbus is a “open source” Grid R&D project with focus on Grid Economy, Utility Grids and Service Oriented Computing.
• Gridbus Middleware components include:– Alchemi: .NET-based Enterprise Grid
– Grid Market Directory and Web Services
– Grid Bank: Accounting and Transaction Management
– Visual Tools for Creation of Distributed Applications
– Grid Service Broker and Scheduling
– Workflow Management Engine
– GridSim Toolkit
– Libra: SLA-based Resource Allocation
9
What does Grid players require?
They need tools and technologies that help them in value expression, value translation, and value enforcement.
Grid Service Consumers (GSCs): How do I express QoS requirements ? How do I trade between timeframe & cost ? How do I map jobs to resources to meet my QoS needs? How do I manage Grid dynamics and get my work done? …
Grid Service Providers (GSPs) How do I decide service pricing models ? How do I specify them ? How do I translate them into resource allocations ? How do I enforce them ? How do I advertise & attract consumers ? How do I do accounting and handle payments? …
10
Solution 1: Service Oriented Architecture (SOA)
A SOA is a contractual architecture for offering and consuming software as services.
There are four entities that make up an SOA service provider, service registry, and service consumer (also known as service requestor).
The functions or tasks that the service provider offers, along with other functional and technical information required for consumption, are defined in
the service definition or contract.
provider
registry
consumer
contract
11
Solution 2: Market-Oriented Grid Computing - (a) Sustained Resourced Sharing and (b)
Effective Management of Shared Resources
Grid Economy
12
Grid Node N
Service-Oriented Grid Architecture
Grid Service Consumer
Pro
gra
mm
ing
En
viro
nm
ents
Grid Resource Broker
Grid Service Providers
Grid Explorer
Schedule Advisor
Trade Manager
Job ControlAgent
Deployment Agent
Trade Server
Resource Allocation
ResourceReservation
R1
Misc. services
Information Service
R2 Rm…
Pricing Algorithms
Accounting
Grid Node1
…
Core Middleware Services
…
…
HealthMonitor
Grid Market Services
JobExec
Info ?
Secure
Trading
QoS
Storage
Sign-on
Grid Bank
Ap
pli
cati
on
s
Data Catalogue
13
Gridbus and Complementary Technologies – realizing Utility Grid
AIXSolarisWindows Linux
.NETGridFabricSoftware
GridApplications
Core GridMiddleware
User-LevelMiddleware
GridBank
Grid Exchange & Federation
JVM
Grid Brokers:
X-Parameter Sweep Lang.
Gridbus Data Broker
MPI
Condor SGE TomcatPBS
Alchemi
Workflow
IRIX OSF1 Mac
Libra
Globus Unicore ……Grid
MarketDirectory
PDB
CDB
Worldwide Grid
GridFabricHardware
……
PortalsScience Commerce Engineering ……Collaboratories
……
Workflow Engine
Grid Storage Economy
Gri
d E
con
om
y
NorduGrid XGrid
ExcellGrid
Nimrod-G
Gridscape
14
On Demand Assembly of Services: Putting Them All Together
ASP Catalogue
Grid Info Service
Grid Market Directory
GSP(Accounting Service)
GridbusGridBank
GSP(e.g., UofM)
PEGSP
(e.g., VPAC)
PE
GSP(e.g., IBM)
CPUorPE
Grid Service (GS)
(Globus)
Alchemi
GS
GTS
Cluster Scheduler
Job
8
GridResource Broker
2
Visual Application Composer
Application CodeExplore
data1
36
45
Resu
lts9 7
Results+
Cost Info
10
11
Bill
12Data Catalogue
1533
On Demand Assembly of Services: Putting Them All Together
ASP Catalogue
Grid Info Service
Grid Market Directory
GSP(Accounting Service)
GridbusGridBank
GSP(e.g., UofM)
PEGSP
(e.g., VPAC)
PE
GSP(e.g., IBM)
CPUorPE
Grid Service (GS)(Globus) Alchemi
GS
GTS
Cluster Scheduler
Job
8
GridResource Broker
2
Visual Application Composer
Application CodeExplore
data1
36
45
Res
ults
9 7
Results+
Cost Info
10
11
Bill
12Data Catalogue
16
Alchemi: .NET-based Enterprise Grid Platform & Web Services
InternetInternet
InternetInternet
Alchemi Worker Agents
Alchemi Manager
Alchemi Users
Web Services
Web Services
•SETI@Home like Model•General Purpose•Dedicated/Non-dedicate workers•Role-based Security•.NET and Web Services•C# Implementation•GridThread and Job Model Programming•Easy to setup and use• Widely in use!
17
Some Users of Alchemi
Tier Technologies, USALarge scale document processing using Alchemi framework
CSIRO, AustraliaNatural Resource Modeling
The Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes
Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework
The University of Sao Paulo, BrazilThe Alchemi Executor as a Windows Service
stochastix GmbH, GermanyServing clients in International Banking/Finance sector
Many users in Universities: See next for an example.
18
Agenda
Introduction Utility Networks and Grid Computing Application Drivers and Various Types of Grid Services
Global Grids and Challenges Security, resource management, pricing models, …
Service-Oriented Grid Architecture and Gridbus Solutions
Market-based Management, GMD, Grid Bank, Alchemi Grid Service Broker
Architecture, Design and Implementation Performance Evaluation: Experiments in Creation
and Deployment of Applications on Global Grids A Case Study in High Energy Physics
Summary and Conclusion
19
A resource broker for scheduling task farming data Grid applications with static or dynamic parameter sweeps on global Grids.
It uses computational economy paradigm for optimal selection of computational and data services depending on their quality, cost, and availability, and users’ QoS requirements (deadline, budget, & T/C optimisation)
Key Features A single window to manage & control experiment Programmable Task Farming Engine Resource Discovery and Resource Trading Optimal Data Source Discovery Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & sharing of results Accounting
Grid Service Broker (GSB)
20
Gridbus Broker Architecture
Grid Middleware
Gridbus Client Gridbus ClientGribus Client
Grid Info Server
Schedule Advisor
Trading Manager
Gridbus Farming Engine
RecordKeeper
Grid Explorer
GE GIS, NWSTM TS
RM & TS
Grid Dispatcher
RM: Local Resource Manager, TS: Trade Server
G
G
CU
Globus enabled node.A
L
Alchemi enabled node.
(Data Grid Scheduler)
DataCatalog
DataNode
Unicore enabled node.
$
$
$
App, T, $, Opt
(Bag of Tasks Applications)
21
Gridbus Broker: Separating “applications” from “different” remote service access
enablers and schedulers
Alchemi
Gateway
UnicoreData Store
Access Technology
Grid FTPSRB
-PBS-Condor-SGE
Globus
Job manager
fork() batch()
Gridbusagent
Data Catalog
-PBS-Condor-SGE-XGrid
SSH
fork()
batch()
Gridbusagent
Single-sign on securityHome Node/Portal
GridbusBroker
fork()
batch() -PBS-Condor-SGE-Alchemi-XGrid
Application Development Interface
Sch
ed
ulin
gIn
terfa
ces
Alogorithm1
AlogorithmN
Plugin Actuators
22
Gridbus Services for eScience applications
Application Development Environment: XML-based language for composition of task farming (legacy)
applications as parameter sweep applications. Task Farming APIs for new applications. Web APIs (e.g., Portlets) for Grid portal development. Threads-based Programming Interface Workflow interface and Gridbus-enabled workflow engine.
Resource Allocation and Scheduling Dynamic discovery of optional computational and data nodes
that meet user QoS requirements. Hide Low-Level Grid Middleware interfaces
Globus (v2, v4), SRB, Alchemi, Unicore, and ssh-based access to local/remote resources managed by XGrid, Condor, SGE.
23
Figure 3 : Logging into the portal.
Drug DesignMade Easy!
Click Here for Demo
24
Deadline (D) and Budget (B) Constrained Scheduling Algorithms
Algorithm Execution Time (D)
Execution Cost (B)
Compute Grid
Data Grid
Cost Opt Limited by D Minimize Yes Yes
Cost-Time Opt
Minimize if possible
Minimize Yes
Time Opt Minimize Limited by B Yes Yes
Conservative-Time Opt
Minimize Limited by B, jobs have guaranteed minimum budget
Yes
25
Gridbus Project: Some Applications and Users
Gridbus Project: Gridbus Project: Some Applications and UsersSome Applications and Users
http://www.gridbus.org
BioGrid: Molecular docking for Drug-discovery
BioGrid: Molecular docking for Drug-discovery
High Energy Physics: Particle Discovery
High Energy Physics: Particle Discovery
Melbourne University
NeuroScience: Brain Activity Analysis
NeuroScience: Brain Activity Analysis
Natural Resource ModelingNatural Resource Modeling
CSIRO Land and Water, Austraila.
Large Scale document processing
Large Scale document processing
Tier Technologies, USA.
Detection of patterns of transcription factors in mammalian genes
Detection of patterns of transcription factors in mammalian genes
26
GridSim: A Toolkit for Modeling and Simulation of Various Types of Grids
Basic Discrete Event Simulation Infrastructure
Virtual Machine (Java, cJVM, RMI)
PCs ClustersWorkstations
. . .
SMPs Distributed Resources
GridSim Toolkit
Application Modeling
InformationServices
Resource Allocation
Grid Resource Brokers or Schedulers’s Simulation
Statistics
Resource Modeling and Simulation (with Time and Space shared schedulers)
ClustersSingle CPU ReservationSMPs Load Pattern
Application Configuration
Resource Configuration
Visual Modeler
Grid Scenario
Network
SimJava Distributed SimJava
Resource Entities
Output
Application, User, Grid Scenario’s Input and Results
Add your own policy for resource allocation
AuctionFramework
27
Selected Funded Projects: 2006-2009
Australian Research Council: Discovery Project:
QoS-based Scheduling of e-Research Application Workflows on Global Grids
QoS-based Workflow Scheduling SLA-based Resource Allocation
Dept. of Science, Training and Education International Science Linkage
The Utility Grid Project: Autonomic and Utility-oriented Global Grids for Powering Emerging E-Research Applications
Links Gridbus Efforts to Many International Efforts
28
Grid Workflow Application Modeling & Definition Tools
Grid Workflow Specification and Verification
Grid Workflow Management System
Resource Info Service
Application Catalogue
Build Time
Run Time
Workflow Design & Monitoring
Workflow Execution Control & Monitoring
Interaction with Grid resources
Interaction with VO Info services
QoS-based Workflow Scheduling
Fault Management Data ManagementData Catalogue
Virtual Organization
R2 Rn……
SLA-based Resource Allocation System(Plug-in for Existing Local Resource Managers)
Negotiation Services
ExecutionMonitor
E-Researchers/Users
feedback
Core Grid Services
SynchrotronData source
Global Grid
29
SLA-based Allocation of Cluster Resources
Computation Node 1
Job Manager
Computation Node c
:::
Economy-based Admission
Control
Economy-based Resource Allocation
Computation Nodes
User u
:::
Manager Node
Node Status Monitor
Job Control
User Population
User 1job1
2 3
45
Utility-driven Job Scheduler
execution results
feedback
job submission
execution results
Pricing
Resource Manager
InterGrid: Internetworking of Islands of Grids
Rajkumar Buyya
Grid Computing and Distributed Systems(GRIDS) Laboratory
Dept. of Computer Science and Software EngineeringThe University of Melbourne, Australia
www.gridbus.org/gridsim
5th International Conference on Grid and Cooperative ComputingChangsha, China, October 2006
31
The Outcomes of Grid Computing thus Far: Islands of Grids
Organizational Gridor island of Grid
32
InterGrid: An architecture for Internetworking of “islands” of Grids
G rid
G rid G rid
IG G
IG G
IG G
RP
RP
RP
RP
RP
RP
RP
RP
RP
IRM
IRM
IRM
IntraG rid
G G
RP
RP IRM
IntraG rid
G G
G ridIG G
RP
RPIRM
IG G
RPRP
RP
IRM
G rid
IG G
RP
RP
IRM
G rid
IG G
RP RP
RP IRMPeering
InterGrid resourceallocationSites providing resources
Service deploym ent / usage
CL
G RB
33
The InterGrid: How to get there?
Organizational Gridor island of Grid
InterGrid G ateway
O rganizational Gridor island of G ridPeering arrangement
34
New Research Challenges
Decentralised “service” discovery from gateways-based to “leaf” nodes
Protocols and Business models for “Peering” between Grids
Application models that support growing and shrinking of application
Resource management and scheduling SLA based resource allocation.
35
Summary and Conclusion
Grids have emerged as enabler for Cyberinfrastructure that powers e-Science and e-Business applications.
Delivering ICT services as computing utilities. Australia is actively developing Grid technologies and
applications Recently Australian Govt. is pushing Grids heavily under
e-Research banner. Melbourne is leading a number of Grid projects nationally. SOA + Market-based Grid Management = Utility Grids Grids allow users to dynamically lease Grid services at
runtime based on their quality, cost, availability, and users QoS requirements.
Grids offer enormous opportunities for realizing e-Science and e-Business at global level.
36
Thanks for your attention!
We Welcome Cooperation in Research and Development!http:/www.gridbus.org