What is MCSR?

Who is MCSR?What Does MCSR Do?Who Does MCSR Serve?What Kinds of Accounts?Why Does Mississippi Need Supercomputers?What Kinds of Research?What Kinds of Instruction?What Kinds of Workshops?How Much Does it Cost?What Kinds of Software?What Supercomputers and Clusters?

What is MCSR?

Mississippi Center for Supercomputer Research

Established in 1987 by the Mississippi Legislature

Mission: Enhance Computational Research Climate at Mississippi’s 8 Public Universities

also: Support High Performance Computing (HPC) Education in Mississippi

What Does MCSR Do?

We make Mississippi scientists:- more competitive for federal grants- more productive in research

We provide extraordinary learning opportunities for Mississippi college students- instructional accounts- computing workshops- helpdesk support

Who Does MCSR Serve?

MCSR serves faculty, researchers, and students at all of Mississippi’s 8 public universities

• Alcorn State University• Jackson State University• Mississippi Sate University• Mississippi Valley State University• The University of Southern Mississippi• Delta State University• The University of Mississippi• Mississippi University for Women

Who Uses More MCSR Computing Time?

MSU; 38771; 4%

USM; 25279; 3%

UM; 172946; 19%

JSU; 663177; 73%

Other; 2145; 0%

CPU Hours Used by IHLAll MCSR SystemsFY 2009 (1st QTR)

2% 1% 10%

86%

SWEETGUM

1%28%

71%

REDWOOD

7%5%

16%

71%

MIMOSA

What Types of Computing Access is Available?

Research Accounts- provided for faculty and student “researchers”- good for the duration of employment or enrollment

Instructional Accounts- provided at an instructor’s request- for all students enrolled in a semester course- valid for the duration of the semester- can be “converted” to research account

Research vs. Class Accounts?

New Class Accounts; 266; 73%

New Research Accounts; 100;

27%

MCSR Accounts Created FY 08

New Class Accounts

New Research Accounts

Who Uses Research Accounts?

usm6%

msu5%

UM41%

jsu46%

other2%

MCSR Research Accounts by IHL

September 2008

usm

msu

um

jsu

other

Who Uses Instructional Accounts?

UM; 2

DSU; 8

USM; 1

Courses Supported with MCSR Class Accounts, FY 2008

UM

DSU

USM

UM; 5

DSU; 8

USM; 1

Courses Supported with MCSR Class Accounts, FY 2007

UM

DSU

USM

UM; 7

DSU; 5

USM; 1

Courses Supported with MCSR Class Accounts, FY 2006

UM

DSU

USM

MVSU; 2

UM; 13

DSU; 2 USM; 1

Courses Supported with MCSR Class Accounts, FY 2005

MVSU

UM

DSU

USM

What Types of Courses Use MCSR?

Over 82 University Courses Supported since 2000

C/C++, Fortran, MPI, OpenMP, MySQL, HTML,Javascript, Matlab, PHP, Perl, ….

http://www.mcsr.olemiss.edu/education.php

Why Do Mississippi Researchers Need Supercomputers?

Economics: researchers in a poor state like Mississippi can still make a big splash.

Computational simulations are faster, cheaper, and safer than laboratory experiments alone.

What Kinds of Research @ MCSR?90% of MCSR calculations are computational chemistry

Cleanup of high explosive materials.Design of high energy density rocket fuelsThe chemical underpinnings of high powered lasersMutation studies of enzyme activityDesigning weather-proofing coatings for machinery

Other AreasHurricane forecastingBlast resistant CoatingsBetter 3-D imaging for diagnosing brain tumors

What Types of Workshops by MCSR?

• MCSR consultants taught over 140 free seminars in FY08.

• Over 60 training topics available, and growing. • Fixed schedule or on-demand.• Unix/programming, Math Software, Stats Software,

Computational Chemistry Software

Do Researchers and Students Pay to Use MCSR?

No. MCSR services are provided at no cost to the individual, department, or institution.

Funded researchers may ask for priority access.

Mississippi researchers may claim the value of MCSR computing services received as an in-kind contribution from their institution when seeking federal grants.

How Much Does MCSR Cost Mississippi?

FY88

FY89

FY90

FY91

FY92

FY93

FY94

FY95

FY96

FY97

FY98

FY99

FY00

FY01

FY02

FY03

FY04

FY05

FY06

FY07

FY08

FY09

$-

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

MCSR Budget by Fiscal Year

Original

Revised

Special Funds

What Is the Value of MCSR to Mississippi?

RETURN ON INVESTMENT =

GRANTS SUPPORTED / MCSR BUDGET

For FY 2008:

$32,832,097 / $845,535

= $38.30 per dollar spent

What Software Environments @ MCSR

• Programming C/C++, FORTRAN, Java, Perl, PHP, MPI…

• Science/EngineeringPV-Wave, IMSL, GSL, Math Libraries, Abaqus

• Math/StatisticsSAS, SPSS, Matlab, Mathematica

• Chemistry Gaussian, Amber, NWChem, GAMESS, CPMD, MPQC, MolPro, GROMACS

What Supercomputers @ MCSR?

Supercomputers at MCSR: sweetgum

- SGI Origin 2800 128-CPU Supercomputer- 64 GB of shared memory

Supercomputers at MCSR: redwood

- 224 CPU SGI Altix 3700 Supercomputer- 224 GB of shared memory

Supercomputers at MCSR: mimosa

- 253 CPU Intel Linux Cluster – Pentium 4- Distributed memory – 500MB – 1GB per node- Gigabit Ethernet

Supercomputers at MCSR: mimosa

Supercomputers at MCSR: sequoia

- 22 nodes- 176 cores- 352 GB Memory- 20 TB Storage- InfiniBand Interconnect

Supercomputers at MCSR: sequoia

Introduction to Parallel Programming at MCSR

• Message Passing Computing– Processes communicate via calls to message passing library routines– Programmers “parallelize” algorithm and add message calls– At MCSR, this is via MPI programming with C or Fortran

• Sweetgum – Origin 2800 Supercomputer (128 CPUs)• Mimosa – Beowulf Cluster with 253 Nodes• Redwood – Altix 3700 Supercomputer (224 CPUs)• Sequoia – Altix XE 310 InfiniBand Cluster (176 cores)

• Shared Memory Computing– Threads coordinate/communicate results via shared memory variables– Care must be taken not to modify the wrong memory areas– At MCSR, this is via OpenMP programming with C or Fortran on

sweetgum, redwood, or sequoia

Speed-Up

http://www.mcsr.olemiss.edu/Engr692_TimingWorshkeet.xls

What MCSR Systems for USM Class Accounts

SweetgumMPI or OpenMP1 to 16 CPUsUp to 900mb per CPUPBS scripts preferred

#PBS –l ncpus=4Interactive computations will be killed after 30 minutesQueues: SM-4P, SM-8P, MM-8P, MM-16P Processors: Mix of 195 MHz and 300 MHzO/S: Irix (like Unix)Compilers: SGI’s Fortran, C/C++, GNU C/C++, w/ SGI MPT

What MCSR Systems for USM Class Accounts

MimosaMPI1 to 18 nodes400 GB Memory per nodePBS Scripts Only (no interactive jobs allowed)

#PBS –l nodes=4Queues: MCSR-CAProcessors: single 1.4 GHz P4/nodeO/S: SUSE Linux 10.3Compilers: Portland Group (PGI) Fortran, C/C++ w/mpichqstat –f (to find out what nodes your job is running on)

What MCSR Systems for USM Class AccountsSequoia

OpenMP (multiple processors on the same node)MPI (multiple processors on the same or different nodes)Hybrid (OpenMP within node, MPI across nodes)1 to 4 nodes, 1 to 8 CPUs per nodePBS Queues: SM-4P (for up to 4 CPUs on 1 node)

MCSR-Test (up to 8 CPUs on each of 4 nodes)PBS Scripts Only (no interactive jobs allowed)#PBS –l nodes=4:ppn=8 (to run on all 8 CPUs of all 4 nodes)#PBS –l ncpus=8:select= 16 GB Memory per node (2 GB per CPU)

qstat –f (to find out what nodes your job is running on)

Sequoia for USM Class AccountsTo run on 4 nodes, and 8 processors per node (32 processes)

#PBS –l nodes=4:ppn=8To run on 2 nodes, and 4 processors per node (8 processes)

#PBS –l nodes=2:ppn=4To run on 1 node, and up to 8 processors (OpenMP)

#PBS –l nodes=1:ppn=8To run on 8 processors, regardless of number of nodes

#PBS –l ncpus=8To run 8 processors, with preferences about node placement

#PBS –l ncpus=8#PBS –l place=scatter (distribute across as many nodes as can)#PBS –l place=pack (pack processes onto as few nodes as can)#PBS –l place=free (place processes on first available processors)

Parallel Efficiency