Grids, Clouds and HPC - The Munich Network Management Team

Grids, Clouds and HPC -The Munich Network Management Team

Dieter Kranzlmüllerkranzlmueller@ifi.lmu.de

Lehr- und Forschungseinheit für Kommunikationssysteme und Systemprogrammierung

D. Kranzlmüller Visit @ University of Virginia 2

About us - The MNM-Team.

D. Kranzlmüller Visit @ University of Virginia 3

Prof. Dr. Dieter Kranzlmüller, Prof. Dr. Heinz-Gerd Hegering

Dr. Victor ApostolescuDr. Michael Brenner

Dr. Ernst BötschDr. Wolfgang Hommel

Patricia MarcuDr. Helmut ReiserChristian Richter

Dr. Thomas SchaafDr. David Schmitz

Dr. Mark Yampolskiy

Prof.Dr. Gabi Dreo RodosekVolker Eiseler

Frank EyermannSebastian HanigkIris Hochstatter

Robert KochMichael Kretzschmar

Björn Stelte

Members of the MNM Team

Dr. Vitalian DanciuDr. Nils gentschen Felde

Christof KlauseckerSilvia Knittl

Annette KostelezkyThomas Köckerbauer

Ralf KönigTobias Lindinger

Feng LiuMartin Metzker

Dr. Michael SchiffersChristoph Spielman

Johannes Watzl

D. Kranzlmüller Visit @ University of Virginia 4

Topics

Service Management

Networks Grid computing

IT Security

HighPerformance

Computing

Virtualization

Cloud Computing

D. Kranzlmüller Visit @ University of Virginia 5

MNM-Team Lectures

D. Kranzlmüller Visit @ University of Virginia

Munich Region: High-Tech Cluster…

Information andCommunication

•Siemens•Infineon

•Sun•Intel•HP

•General Electric•…

Software•SoftLab

•Nemetschek•Oracle

•Microsoft•sd&m

Aerospace•EADS•ESG

•Galileo Industry•Astrium•IABG

•Eurocopter• …

Life Sciences, Media,

Telecommunication,Environmental Engineering…

Finance•Allianz

•Bay. Landesbank•HypoVereinsbank•MünchnerRück

• …

Venture Capitalists•3i

•Apax•Atlas Venture

•EarlyBird•TVM

•Wellington Partners• …

Research•DLR•FhG

•Max-Planck•LMU•FHM•TUM

•Univ. der Bundeswehr

• …

Patent•European Patent

Office•German Patent

Office

Automotive•BMW•Audi•MAN• …

6

Leibniz Supercomputing Centreof the Bavarian Academy

of Sciences and Humanities

© 2009 Leibniz Supercomputing Centre

By Ernst A. Graf

7

D. Kranzlmüller Visit @ University of Virginia

The Leibniz Supercomputing Centre•Computer Centre (~175 employees) for all Munich Universities with

-more than 80,000 students and-more than 26,000 employees-including 8,500 scientists

•Regional Computer Centre for all Bavarian Universities-Capacity computing-Special equipment-Backup and Archiving Centre (more than 7 petabyte, 5.5 billion files)

-Distributed File Systems-Competence centre (Networks, HPC, IT Management)

•National Supercomputing Centre-Gauss Centre for Supercomputing-Integrated in European HPC and Grid projects(DEISA, PRACE, EGI)

8

D. Kranzlmüller Visit @ University of Virginia

LRZ Supercomputer HLRB2

9

Foto Helmut Payer, produced by gsiCom

D. Kranzlmüller Visit @ University of Virginia

The LRZ: a Supercomputing Centre• National supercomputing system SGI Altix 4700

-9728 compute cores (Intel Itanium2 Montecito)-62.3 TFlop/s peak performance-56.5 TFlop/s Linpack benchmark-39 TByte Total Memory-660 TByte attached Disk space-Weighing 103 metric tons -Consuming ~1000 kVA-On 24 m x 12 m footprint

• 128 cpu SGI Altix 3700 Bx2 for scientists in the state of Bavaria

• 128 dual core cpu SGI Altix 4700 for scientists in the state of Bavaria

• Linux cluster with more than 3.500 cpus (resp. cores) mainly for the Munich universities and for scientists in the state of Bavaria

• More than 500 additional servers for general IT services

Foto Helmut Payer, produced by gsiCom

10

D. Kranzlmüller Visit @ University of Virginia

Examples of Applications @ LRZ•Computational Fluid Dynamics: Optimisation of turbines and wings,

noise reduction, air conditioning in trains•Fusion: Plasma in a future fusion reactor (ITER)•Astrophysics: Origin and evolution of stars and galaxies•Solid State Physics: Superconductivity, surface properties•Geophysics: Earth quake scenarios•Material Science: Semiconductors•Chemistry: Catalytic reactions•Medicine and Medical Engineering: Blood flow, aneurysms, air

conditioning of operating theatres•Biophysics: Properties of viruses, genome analysis•Climate research: Currents in oceans

11

D. Kranzlmüller Visit @ University of Virginia

Distribution of CPU-hours

12

D. Kranzlmüller Visit @ University of Virginia

Extension of the Buildings(Computer: March 2011, Staff: Autumn 2011)

Fotomontage

Südansicht

Nordansicht

13

D. Kranzlmüller Visit @ University of Virginia 14

Topics

Service Management

Grid computing

HighPerformance

ComputingCloud Computing

Networks

IT SecurityVirtualization

Grid computing

D. Kranzlmüller Visit @ University of Virginia25th NORDUnet

Conference, Copenhagen

15

290 Sites55 Countries144.000 Cores25 PetaBytes Disk>17.000 Users>200 VOs>330.000 Jobs/Day

ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…

Status July 2009: http://project.eu-egee.org/index.php?id=417

Jobs/month: +45% in a yearSites: +5% in a year

Countries: +10% in a yearVOs: +29% in a year

D. Kranzlmüller Visit @ University of Virginia

European Grid InitiativeObjectives:•Ensure the long-term sustainability of the European e-infrastructure

•Coordinate the integration and interaction between National Grid Infrastructures

•Operate the European level of the production Grid infrastructure for a wide range of scientific disciplines to link National Grid Infrastructures

EGI Grid Infrastructure should be•a large-scale, production Grid infrastructure •built on national grids that interoperate seamlessly at many levels,

•offering reliable and predictable services to a wide range of applications

D. Kranzlmüller Visit @ University of Virginia

EGI Operations TasksOperation of tools and services• Grid configuration repositories• Grid accounting repositories• Grid repositories for SLA

compliance and performance monitoring

• Grid operations portal• NGI Grid oversight

Security• Security policy development and

maintenance• Coordination of security and

incident response• Expert team for security

vulnerabilities

User support• Central ticket handling system• Gathering requirements for user

support tools

Other international tasks• MW deployment/roll-out and

support• Resource allocation & brokering

support• Interoperations between NGI’s and

with other grids• Network support• Definition of best practises,

procedures, requirements• Catch-all production grid core

services

EGI Blueprint Proposal (V3.0)http://www.eu-egi.eu/blueprint.pdf

• Functions of EGI• Financing of EGI• Transition to EGI

D. Kranzlmüller Visit @ University of Virginia

EGI Operations

EGI.eu EGI.eu global tasksNGI international tasksNGI local tasks

D. Kranzlmüller Visit @ University of Virginia

EGI Infrastructure

…

International Scientific and Research CollaborationInternational Scientific and Research Collaboration

NationalGrid

Initiative 1

NationalGrid

Initiative 1

NationalGrid

Initiative 2

NationalGrid

Initiative 2

NationalGrid

Initiative N

NationalGrid

Initiative N

Middlew

are A

Middlew

are B

Middlew

are B

Middlew

are X

gLite

UNICO

RE

ARC

EGI.euEGI.eu European-level Grid Services

Middleware… is essential for the grid… continues to evolve with

new functionalities… development is a

long-term process

D. Kranzlmüller Visit @ University of Virginia

D-Grid Project D-MON

ComputingService (DB)

VO-specific View (OGSA-DAI)

D. Kranzlmüller Visit @ University of Virginia 21

Topics

Service Management

HighPerformance

Computing

Networks Grid computing

IT SecurityVirtualization

Cloud Computing

Cloud Computing

D. Kranzlmüller Visit @ University of Virginia

Cloud Research @ MNM-Team

Questions:•What is the cloud?•What are the differencesbetween clouds and grids?

•Which technologies can beused within a cloud?

•How to use anonymousservices of cloudsefficiently?

•How to securely deploy and use clouds?

22

D. Kranzlmüller Visit @ University of Virginia 23

Cloud Testbed @ MNM

•Example: Zimory-T-Labs spin-off-Small company (~20 staff workers)-Interconnects 2 Telekom computing centers-Only enterprise-grade customers-Provides qualitatively high services-3 service-levels (Gold, Silver, Bronze)

D. Kranzlmüller Visit @ University of Virginia 24

Management solution to manage multiple clouds

Source: Zimory GmbH

Zimory Cloud (1/3)

D. Kranzlmüller Visit @ University of Virginia 25

•Gateway to enterpise cloud computing Resources•Support of SLAs

Source: Zimory GmbH

Zimory Cloud (2/3)

D. Kranzlmüller Visit @ University of Virginia 26Source: Zimory GmbH

Cloud GatewayDate Center manager, Orchestration layer, Security manager, Data policy manager, Resource quality managerData Center 1 Data Center 2

Cloud HostSecurity management, Resource

optimization, Interoperable access management, VM

migration, Virtualized Resources Virtualized Resources

VMware, Xen, KVM, Hyper-V VMware, Xen, KVM, Hyper-V

DB DB DB

Data Storage

DB DB DB

Data Storage

Database Hypervisor™Data management, Data

synchronization, Data migration, Storage reliability and security

Oracle, MySQL,… SAN Oracle, MySQL, … SAN

Cloud ManagerSLA manager, Resource planner,

Application profiler, Network optimizer, Load balancer, Migration manager, Accounting, Authentication, Policy

manager, Open APIInfrastructure ManagerInfrastructure Manager

vCenter, Platespin, etc. vCenter, Platespin, etc.

Zimory Public Cloud Solution

Wide Area Network

Zimory Cloud (3/3)

D. Kranzlmüller Visit @ University of Virginia 27

Topics

Service Management

Grid computing

HighPerformance

ComputingCloud Computing

Networks

IT SecurityVirtualization

HighPerformance

Computing

D. Kranzlmüller Visit @ University of Virginia

Scalable Parallel Debugging with

•Integrated workbench framework to access the power of existing computing infrastructures

•Built on top of the Eclipse framework, released under Eclipse Public License (EPL), continues as an Eclipse Technology Project

•Version 1.0 available for downloadhttp://www.eclipse.org/geclipse/

28

D. Kranzlmüller Visit @ University of Virginia 29

g-Eclipse Trace Viewer•Tool to visualize and analyze communication of parallel message passing programs

•Parallel debugging and development activities centered around Trace Viewer

•Integrated into g-Eclipse•Can be used asstandalone application

•Platform independent

D. Kranzlmüller Visit @ University of Virginia 30

g-Eclipse Trace Viewer

logical clocks

physical clocks

statistics

D. Kranzlmüller Visit @ University of Virginia

Scalability for Debugging•Abstraction techniques are necessary•Based on Actions and Markers extension points

-reorder, group or hide processes-mark interesting events

31

D. Kranzlmüller Visit @ University of Virginia

•By manually selecting the processes•Automatically by criteria

Reorder, Group or Hide Processes

32

D. Kranzlmüller Visit @ University of Virginia

•Search for patterns defined using a description language-serves program understanding

-helps to detect errors and inefficient communication

33

Communication Patterns

D. Kranzlmüller Visit @ University of Virginia 34

g-Eclipse Components Combined

D. Kranzlmüller Visit @ University of Virginia 35

Topics

Service Management

Networks Grid computing

IT Security

HighPerformance

Computing

Virtualization

Cloud Computing

D. Kranzlmüller Visit @ University of Virginia 36

Contact

www.nm.ifi.lmu.de

kranzlmueller@ifi.lmu.de