High Performance Computing @ Louisiana State University - LONI HPC Enablement Workshop – LaTech University,

High Performance Computing @ Louisiana State University - http://www.hpc.lsu.edu/High Performance Computing @ Louisiana State University - http://www.hpc.lsu.edu/

http://www.loni.org

LONI HPC Enablement Workshop – LaTech University, October 23, 2008

Introduction to Globus Procedures using LONI Resources

October 2009

Sirish TummalaLONI HPC Enablement Group

Email: [email protected]


http://www.loni.org


So Far in the HPC Workshop

10/06/09

•About LONI•Machines•Allocation Procedures•Software Environment•Linux, Vi Tutorials•Using the Batch System (PBS, Loadleveler)


http://www.loni.org


So Far Computing your Data

10/06/09

•Login to the Machine•Compile your Code using native compiler on the machine.•Write a submit file•Submit it to the Batch System (PBS, Loadleveler)


http://www.loni.org


So Far Data Management

10/06/09

•Copy your data to the machine using SCP, SFTP•Create directories manually•After Computation, archive it without exceeding the disk quotas.


http://www.loni.org


So Far: Workflows

10/06/09

•You write your own scripts to automate everything.•Stage in, Preprocessing, Processing, Post Processing, Stage out and Archival•Usually, without any fault tolerance (retrial from the point of failure, in case of one)


http://www.loni.org


AIX•Bluedawg•Ducky•Zeke•Neptune•LacumbaLINUX•Eric•Oliver•Poseidon•Louie•Painter•Queenbee•Unnamed

10/06/09

Too many Options?


http://www.loni.org


Constraint: Using one machine at a time

10/06/09

•You are more focused on your work than to use all the machines for your work.•You avoid playing with multiple machines and instead work on your code•Often, your code is not portable because you don’t use it on other machines.•So you end up working with one or very few machines (usually your favorite ones)


http://www.loni.org


What are you missing out?

10/06/09

•Unused computational cycles on other machines. Get your work done fast. No more long queues.•Spread your risk of machine failure while you need to get your results in time.•See how your code scales on different machines and use that for improvement.


http://www.loni.org


Use Globus Toolkit !

10/06/09

Using Globus you can:•There are 12 supercomputing machines on LONI. Use them all.•Submit jobs to all the machines, without going to each one.•When a machine goes down, throttle jobs to a different machine.•Get more computational throughput in a given amount of time, by spreading your allocation/computation over multiple machines.•Automate your data management. No more manual data peddling, disk quota exceeding = “no more annoying emails from us.”


http://www.loni.org


LONI Grid Service Nodes

qb1.loni.orgeric1.loni.org

BLUEDAWG.LONI.ORGPAINTER1.LONI.ORG

UNNAMED(NEW)LACUMBA.LONI.ORG

NEPTUNE.LONI.ORGPOSEIDON1.LONI.ORGOLIVER1.LONI.ORG

ZEKE.LONI.ORG

LOUIE1.LONI.ORGDUCKY.LONI.ORG

QB1.LONI.ORGERIC1.LONI.ORG


http://www.loni.org


Your Globus Grid Certificate• Globus uses the Grid Security Infrastructure

(GSI).

– Based on the Public-key infrastructure

– All Entities use certificates to estabilish trust and identity.

– A certificate contains:

• Entity's identity

• Public Key

• Certificate Authority Identity

– Delegation

• Services can act on behalf of users so that grid tasks can be automated and do not require entering a passphrase manually.

– GridMap File

• A list of Certificate subject entries and corresponding local unix accounts to correctly map certificate owners to local accounts.


http://www.loni.org


LONI MyProxy Service• “proxy01.hpc.lsu.edu”

• MyProxy is an online credential repository.

• Users having certificates can store their credentials in the MyProxy repository with a password of their choice.

• Used for delegating grid credentials to other resources.– You can have your certificates

in one place and obtain the delegated proxy on other resources.


http://www.loni.org


GridFTP Protocol

Basic TransferOne control channel, several parallel data channels

Basic TransferOne control channel, several parallel data channels

Third-partyTransferControl channels to each server, several parallel data channels between servers


http://www.loni.org


Reliable File Transfer Service• Reliable File Transfer: RFT provides a

management layer over GridFTP to provide reliability and fault tolerance.

– Uses a database to maintain state.– All file operations in the Web services

implementation of globus use RFT.


http://www.loni.org


Globus Pre-WSGRAM• DELEGATE, DELEGATE, DELEGATE !!!• A uniform service interface for remote

job submission and control– Includes file staging and I/O

management– Includes reliability features– Supports basic Grid security

mechanisms– Available in Pre-WS and WS forms


http://www.loni.org


Language of Pre-WS GRAM• GRAM uses the Resource Specification

Language (RSL) to describe execution tasks that need to be submitted to GRAM. The GRAM service endpoint is called the gatekeeper.


http://www.loni.org


10/06/09

Globus & PBS, Globus & Loadleveler, Globus & Workflows

•You submit Jobs and Data transfer tasks using Globus•It uses PBS/Loadleveler on the machine for Computation•It used GridFTP to manage your data•Gridway, DAGMan(Condor) to schedule your workflows.

Documents

High Performance Computing @ Louisiana State University - LONI HPC Enablement Workshop – LaTech University,