HPC over CloudHands-on: Cluster automation with Amazon Web Services

Cristian Mezzanotte

17.04.2012

© D

atal

ynx A

G

| S

lide

2| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Overview

• Datalynx AG

• Web Services and Cloud

• AWS

• Hands-on!

• Conclusions

© D

atal

ynx A

G

| S

lide

3| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Assumptions

• You have a basic knowledge of Cloud Computing concepts

• You know the difference between public, private and hybrid Clouds

• You are familiar with High Performance Computing concepts

© D

atal

ynx A

G

| S

lide

4| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Datalynx AG – www.datalynx.com

• Fast growing, customer-focusd, IT company based in Switzerland

• Competences in IT Architectures, Security, Project Management, Quality Management

• Advice. Create. Manage. Secure. Collaborate.

• Customers: Novartis, Roche, Credit Suisse, Swisscom, Adidas, Swiss Post, etc..

© D

atal

ynx A

G

| S

lide

5| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Web Services

• What are WSWS are distributed application components in aclient-server model

Do not confuse Web Services with Web Pages Web Pages are for human, Web Services are formachines

• Why use WSInteroperability, available on the WWW, open standards

• How do WS workHTTP, XML, SOAP, WSDL, UDDI

© D

atal

ynx A

G

| S

lide

6| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Amazon Web Services

• Public Cloud framework

• Infrastructure as a Service (IaaS)

• Management Console

• API

• Based on Web Services

© D

atal

ynx A

G

| S

lide

7| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Amazon Web Services

Elastic Computing Cloud (EC2)

• Create and select new or templeted images of any Operative System, libraries, data, etc..

• Configure network and security settings

• Start, terminate and monitor your instances

• Manage locations, IP endpoints and storage

• Pay-as-you-go

© D

atal

ynx A

G

| S

lide

8| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Amazon Web Services

Simple Storage Service (Amazon S3)

• Write, read and delete unlimited objects containing up to 5TB of data

• Data security tools

• Authentication

• Sharing

© D

atal

ynx A

G

| S

lide

9| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Amazon Web Services

Cloud Formation

• Template resources

• Create templates of machines and create instances in AWS

• Stack your infrastructure and replicate it into AWS

© D

atal

ynx A

G

| S

lide

10

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Amazon Web Services

EMR

VPC

© D

atal

ynx A

G

| S

lide

11

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Amazon Web Services trends

© D

atal

ynx A

G

| S

lide

12

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

The Simulation

• Collaboration with Milan University

• Replica Exchange Molecular Dynamics (REMD) simulation with Amber11

• Pellegrino, Contini et al, Chem. Eur. J 2012, DOI: 10.1002/chem.201104023

© D

atal

ynx A

G

| S

lide

13

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Cluster Instance

Img_remd

/vol

Img_remd

/vol

Img_remd

/vol

Img_remd

/vol

© D

atal

ynx A

G

| S

lide

14

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Big data file transfer service

• The amount of data to be processed and moved is growing fast

• Traditional file transfer tools (ftp, scp..) are inadequate

• Gridftp greatly improves performance up to 300 MB/s (LRZ), 200 MB/s (Teragrid)

© D

atal

ynx A

G

| S

lide

15

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

HPC over Cloud vs Traditional HPC

• Availability of resources

• No initial investment on infrastructure

• Pay what you use

• Scalability

• Versatility

• Automation

© D

atal

ynx A

G

| S

lide

16

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tion

Challenges

• Security

• Billing system integration

• Lower performance in some cases

• Licences in some cases

© D

atal

ynx A

G

| S

lide

17

| 1

9-A

pr-

12

| C

rist

ian

Mez

zan

ott

e | C

orp

ora

te P

rese

nta

tionThank you!

Contacts:

Cristian.Mezzanotte@datalynx.ch

Marcus.Kotschner@datalynx.ch