Upload
kylie-garza
View
35
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Grid Computing (2) (Special Topics in Computer Engineering). Veera Muangsin 30 January 200 4. Outline. High-Performance Computing Grid Computing Grid Applications Grid Architecture Parallel Computers Architectures Cluster Architecture Grid Architecture Grid Middleware Grid Services. - PowerPoint PPT Presentation
Citation preview
1
Grid Computing (2)(Special Topics in Computer Engineering)
Veera Muangsin
30 January 2004
2
Outline• High-Performance Computing • Grid Computing • Grid Applications• Grid Architecture
– Parallel Computers Architectures– Cluster Architecture– Grid Architecture
• Grid Middleware• Grid Services
3
Parallel Computer Architectures
4
Parallel Architecture Taxonomy
• Single Instruction Single Data (SISD )
• Multiple Instruction Single Data (MISD)
• Single Instruction Multiple Data (SIMD)
• Multiple Instruction Multiple Data (MIMD)– Shared Memory MIMD– Distributed Memory MIMD
5
SISD : A Conventional Computer
Speed is limited by the rate at which computer can transfer information internally.
ProcessorProcessorData Input Data Output
Instru
ctions
Ex: PC, Macintosh, Workstations
6
The MISD Architecture
More of an intellectual exercise than a practical configuration. Few built, but commercially not available
Data InputStream
Data OutputStream
Processor
A
Processor
B
Processor
C
InstructionStream A
InstructionStream B
Instruction Stream C
7
SIMD Architecture
Ex: CRAY machine vector processing
Ci<= Ai * Bi
InstructionStream
Processor
A
Processor
B
Processor
C
Data Inputstream A
Data Inputstream B
Data Inputstream C
Data Outputstream A
Data Outputstream B
Data Outputstream C
8
Unlike SISD, MISD, MIMD computer works asynchronously.
Shared memory (tightly coupled) MIMD
Distributed memory (loosely coupled) MIMD
MIMD Architecture
Processor
A
Processor
B
Processor
C
Data Inputstream A
Data Inputstream B
Data Inputstream C
Data Outputstream A
Data Outputstream B
Data Outputstream C
InstructionStream A
InstructionStream B
InstructionStream C
11
Clusters• Distributed Memory
MIMD• The most common
architecture in the TOP500
12
Top 2-5 Clusters• #2 LANL’s ASCI Q• 13.88 TFlops• 8192-node cluster
HP AlphaServer 1.25 GHz
• #3 Virginia Tech’s System X• 10.28 TFlops• 1,100-node cluster, Apple G5
13
• #4 NCSA’s Tungsten
• 9.81 TFlops
• 1,450-node cluster, dual-processor Dell PowerEdge 1750
• #5 PNNL’s MPP2• 8.63 TFlops• 980-node cluster, HP
Longs Peak, dual Intel Itanium-2 1.5 GHz
14Zeus and Athena
Apollo
Our Parallel Computers
15
Our Parallel ComputersApollo Cluster
•6-node cluster•Athlon XP 2000+ processor, 512 MB memory•Linux + MPI + PBS (batch scheduler system) + Globus (Grid middleware)
Zeus and Athena •Two 4-processor Sun Enterprise 420R multiprocessor computers •450 MHz UltraSPARC II processors, 1 GB memory•Solaris + Pthread + MPI
16
Cluster Architecture
17
Cluster Middleware
• Resides Between OS and Applications and offers in infrastructure for supporting:– Single System Image (SSI)
– System Availability (SA)
• SSI makes collection appear as single machine
• SA - Check pointing and process migration
18
Single System Image Components
• NFS (Network File System)
• NIS (Network Information System)
• NTP (Network Time Protocol)
server client
client
client
19
Programming Environments
• Threads (Cluster of SMPs)– POSIX Threads– Java Threads
• Message Passing– MPI– PVM
• Virtual Shared Memory• Batch Scheduling
– PBS, Condor, etc.
20
Batch Scheduling
• Process distribution
• Load balancing
• Job scheduling• PBS, Condor, Sun Grid Engine,
IBM Load Leveler, LSF, DQS, …
21
Cluster Applications• Sequential
• Parallel / Distributed (Cluster-aware app.)– Grand Challenging applications
• Weather Forecasting
• Quantum Chemistry
• Molecular Biology Modeling
• Engineering Analysis (CAD/CAM)
• ……………….
– Web servers, data-mining
22
Grid Architecture
23
What is Grid ?• An infrastructure that dynamically couples
– Computers (PCs, workstations, clusters, traditional supercomputers, and even laptops, notebooks, mobile computers, PDA, and so on)
– Software (e.g., renting special purpose applications on demand)
– Databases (e.g., transparent access to human genome database)
– Special Instruments (e.g., radio)
– People• across the local/wide-area networks (enterprise,
organisations, or Internet) and presents them as a unified resource or problem solving environment.
24
Grid Infrastructure
25
TeraGrid
26
Grid Applications• Old and new applications getting Grid-enabled via
coupling of computers, databases, instruments, people, etc:– (distributed) Supercomputing– Collaborative engineering– high-throughput computing
• large scale simulation & parameter studies
– Remote software access / Renting Software– Data-intensive computing– On-demand computing
27
Conceptual view of the Grid
28
How can the Grid help me?
• Provide access to a global distributed computing environment– via authentication, authorisation, negotiation, security
• Identify and allocate appropriate resources– interrogate information services -> resource discovery– enquire current status/loading via monitoring tools– decide strategy - eg move data or move application– (co-)allocate resources -> process flow
29
How can the Grid help me? (2)
• Schedule tasks and analyse results– ensure required application code is available on remote
machine– transfer or replicate data and update catalogues– monitor execution and resolve problems as they occur– retrieve and analyse results - eg using local
visualization
30
To make this happen you need …
• agreed protocols (Grid protocols)
• defined application programming interfaces (APIs)
• distributed data management
• availability of current status of resources
• monitoring tools
• accepted authentication procedures and policies
• network traffic management
Grid Components
GridFabricNetworked Resources across Organisations
Computers Clusters Data Sources Scientific InstrumentsStorage Systems
Local Resource Managers
Operating Systems Queuing Systems TCP/IP & UDP
…
Libraries & App Kernels …
Distributed Resources Coupling Services
Comm. Sign on & Security Information … QoSProcess Data Access
Development Environments and Tools
Languages Libraries Debuggers … Web toolsResource BrokersMonitoring
Applications and Portals
Prob. Solving Env.Scientific …CollaborationEngineering Web enabled Apps
GridApps.
GridMiddleware
GridTools
32
Before the GridUser
Application
Site A Site B
Network
The User is responsible for resolving the complexities
of the environment
• independent sites
• independent hardware and software
• independent user ids
• security policy requiring local connection to the machine.
33
First Step to the GridUser
Application
Site A Site B
Network
Centralized Scheduler and file staging
Metacenter • Two or more
resources connected in a controlled user environment
Constraints• common
architecture• single name
space• common
scheduler
A layer of abstraction is added that hides some of the complexities associated with running jobs in a distributed computing environment, however, limitations exist
34
Grid Middleware
UserApplication
Site A Site B
Network
Infrastructure
Common Middleware
- abstracts independent, hardware, software, user ids, into a service layer with defined APIs
- comprehensive security,
- allows for site autonomy
- provides a common infrastructure based on middleware
The Grid Today
1
Request info from the grid
1
2Get response2
3Make selection and submit job
3
The underlying infrastructure is abstracted into defined APIs thereby simplifying
developer and the user access to resources, however, this layer is not intelligent
35
The Near Future Grid
Grid Middleware - Infrastructure APIs (service oriented)
UserApplication
Intelligent, Customized Middleware
Site A Site B
Network
Infrastructure
Customizable Grid Services built on defined Infrastructure APIs
• automatic selection of resources
• information products tailored to users
• accountless processing
• flexible interface: web based, command line, APIs
Resources are accessed via various intelligent services that access
infrastructure APIs
The result: The Scientist and Application Developer can focus on
science and not on systems management
36Tom Hinke
How the User Sees a Grid• A set of grid functions that are available as
– Application programmer interfaces (APIs)– Command-line functions
• After authentication, functions can be used to– Spawn jobs on different processors with a single command– Access data on remote systems – Move data from one processor to another– Support the communication between programs executing on
different processors– Discover the properties of computational resources available
on the grid using the grid information service– Use a broker to select the best place for a job to run and then
negotiate the reservation and execution (coming soon).
37
Many GRID Projects and Initiatives
• PUBLIC FORUMS– Computing Portals– Grid Forum– European Grid Forum– IEEE TFCC!– GRID’2000 and more.
• Australia– Nimrod/G– EcoGrid and GRACE– DISCWorld
• Europe– UNICORE– MOL– METODIS– Globe– Poznan Metacomputing– CERN Data Grid– MetaMPI– DAS– JaWS– and many more...
• Public Grid Initiatives– Distributed.net– SETI@Home– Compute Power Grid
• USA– Globus– Legion– JAVELIN– AppLes– NASA IPG– Condor– Harness– NetSolve– NCSA Workbench– WebFlow– EveryWhere– and many more...
• Japan– Ninf– Bricks– and many more...
http://www.gridcomputing.com/
38
NetSolve Client/Server/Agent -- Based Computing
• Client-Server design
• Network-enabled solvers
• Seamless access to resources
• Non-hierarchical system
• Load Balancing
• Fault Tolerance
• Interfaces to Fortran, C, Java, Matlab, more
Easy-to-use tool to provide efficient and uniformaccess to a variety of scientific packages on UNIX platforms
NetSolve Client NetSolve Agent
Network ResourcesSoftware Repository
Software is availablewww.cs.utk.edu/netsolve/
request
choicereply
39
Nimrod - A Job Management System
http://www.dgs.monash.edu.au/~davida/nimrod.html
40
Job processing with Nimrod
41
Nimrod/G Architecture
Middleware Services
Nimrod/G Client Nimrod/G ClientNimrod/G Client
Grid Information Services
Schedule Advisor
Trading Manager
Nimrod Engine
GUSTO Test Bed
Persistent Store
Grid Explorer
GE GISTM TS
RM & TS
RM & TS
RM & TS
Dispatcher
RM: Local Resource Manager, TS: Trade Server
42
User
Application
Resource Broker
A Resource Domain
Grid Explorer
Schedule Advisor
Trade Manager
Job ControlAgent
Deployment Agent
Trade Server
Resource Allocation
ResourceReservation
R1
Other services
Trading
Grid Information Server
R2 Rn…
Charging Alg.
Accounting
Compute Power Market
43
Globus Toolkit
• Grid computing middleware– Software between the hardware and high-level
services– Basic libraries, services, command-line programs
• Most common middleware used in grids
• Integrated with Web Service
44
Globus Software Architecture
Grid SSH
Grid FTP
•login•execute commands•copy files
•get and put files•3rd party copy•interactive file management•parallel transfers
Monitoring and Discovery Service
(MDS)
information about resources and services
LDAP
distributed directory service
•single sign on•delegation of credentials•authorization
Grid Security Infrastructure (GSI)
SSL/TLSX.509 Certificates
•authentication•secure communication
credentials for users, services,
hosts
•execute remote applications
•stage executable, stdin, stdout, stderr
LSFPBS
Globus Resource Allocation Manager
(GRAM) fork/exec
job management
systems
45
Globus server system
PBS
GRAM Server
Grid FTP
Server
Grid SSH
ServerLSF
GRAM Server
Grid FTP
Server
Grid SSH
Server
Globus server system
Globus Deployment Architecture
MDS server system
MDS GRIS
MDS GIIS
MDS GRIS
Globus client
system
Clients are programs and
libraries
GRAM Client
Grid FTP
Client
MDS Client
Grid SSH
Client
User User application/tool
Web portal
46
For More Information
• Globus Project™– www.globus.org
• Grid Forum– www.gridforum.org
• Book (Morgan Kaufman)– www.mkp.com/grids