Common User Environments - Update

Shawn T. Brown, PSC(stbrown@psc.edu)

CUE Working Group Lead

TG Quartely

1

2

Team Members•Shawn Brown (PSC, Lead)

•Kevin Colby (Purdue)•Dan Lapine (NCSA)•David McWilliams (NICS)•Derek Simmel (PSC)•Rich Raymond (PSC, Managing Lead)• Jerry Greenberg (SDSC)•Roberto Gomez (PSC)• John Lockman (TACC)• Jim Lupo (LONI)•Diana Diehl (SDSC, TG Documentation, volunteer)

3

Philosophy

• Create commonality without destroying diversity.

• Focus on user requirements and experience.

• We are not developing a gateway.

• We are not catering to the hero users.

TeraGridResourcesTeraGrid

Resources

CUEDCUE - Documentation

A centrally located, clearly itemized area for documentation of

resources with both web and CLI based access.

CUEMSCUE Management

SystemA single common

command line system for managing one’s

environment, with a single entry to load the

CUE.

CUETPCUE – Testing PlatformSimple program or set of

programs that can be compiled and executed

through the CUE and will help to illustrate its use.

CUBECommon User Build

EnvironmentAttempting to make

common the tools needed for building usable

scientific code across resources

CUEVCCUE Variable

CollectionA set of environment variables that will be common across the TeraGrid, making job

submission and resource discovery easier.

TeraGridResourcesTeraGrid

Resources

CUEDCUE - Documentation

A centrally located, clearly itemized area for documentation of

resources with both web and CLI based access.

CUEMSCUE Management

SystemA single common

command line system for managing one’s

environment, with a single entry to load the

CUE.

CUETPCUE – Testing PlatformSimple program or set of

programs that can be compiled and executed

through the CUE and will help to illustrate its use.

CUBECommon User Build

EnvironmentAttempting to make

common the tools needed for building usable

scientific code across resources

CUEVCCUE Variable

CollectionA set of environment variables that will be common across the TeraGrid, making job

submission and resource discovery easier.

6

How did we proceed?

• Targeted RP “liaisons” to work on implementation.

• Developed implementation documents outlining the “rules” of the implementation.– Done in consultation with:• RP liaisons• SW Int working group• Campus Champions

• Worked to implement the CUEMS and CUEVC portions on current TG machines.

7

The Machines We are Working With

CUEMS – Environment Management

– Implementation of the Modules software environment manager on all systems

– Five basic modules:•cue-login-env Contains  the  CUEVC  definitions  for  environment  variables  

•cue-math     A  wrapper  for  the  modules  cue-mkl  cue-fftw

cue-lapack  cue-scalapack   •cue-build   A  wrapper  for  the  module  cue-compile  

•cue-comm   A  wrapper  for  the  default  mpi  stack    

•cue-tg     Contains  already  defined  TG  variables  for  the

site

– Application Modules•cue-namd, cue-gamess, cue-hdf5, etc..

8

9

CUEVC – Variable Collection

Proposed CUE Variable Collection Environment Variable Definition Example Values CUE_HOME Path to the current user's home directory visible on

login nodes and compute nodes /usr/users/0/janedoe /nics/j/home/janedoe /home/ncsa/janedoe /home/janedoe

CUE_DOCS URL for documentation specific to the current system

http://www.psc.edu/machines/cray/xt3/bigben.php

CUE_APPS Path to directory on the current system containing common software applications

/usr/local/apps /sw/xt5 /usr/local/packages/tg /software/linux-rhel4-ia64

CUE_COMMUNITY Path to directory containing subdirectories for specific user communities in which their applications are installed

/usr/projects /usr/local/packages/tg /soft/community

CUE_EXAMPLES Path to directory containing example files for user tools

/usr/local/packages/tg/examples /usr/local/examples /soft/community/examples

CUE_NODE_SCRATCH Path on a compute node to local scratch file space for that node (not necessarily visible to other compute nodes); node scratch filesystems local to the node may be deleted upon job completion.

/scr /lustre/scratch/johndoe /bessemer/johndoe

CUE_NODE_SCRATCH_TYPE

Filesystem type of the node local scratch filesystem. lustre ext3 gpfs posix

CUE_SCRATCH Path to the user's scratch directory on a shared filesystem visible to all compute nodes.

/gpfs_scratch1/janedoe /lustre/scratch/janedoe /scratcha/janedoe /scratch/gpfs/local/janedoe

CUE_SCRATCH_TYPE Filesystem type of the scratch filesystem visible to all compute nodes.

lustre ext3 gpfs posix

10

CUEMS – Environment Management

•Current Policy – Opt In approach– Provide users a clear and simple procedure for

implementing CUE as default.

.nosoft – tells the system that you want modules as your default environment management

.modules – Contains commented out cue modules that can be implemented at login.

11

CUED – Documentation

•Working with the documentation group to add modules documentation to TG Docs

•A getting started guide on how to activate modules

•https://www.teragrid.org/web/bof/cue

12

Rolling out

•Announce to the TG User Services group at next meeting.– Ask for feedback and testing.

•Ask Campus Champions to test out the implementation.

•Incorporate into the QA testing procedures– Already underway– Current implementation…. The Jerry Test

•Announcement and opening to public.

13

Not stopping…

•Discussion of common queue names.

•Continue work on CUED incorporation.

•Finish fitting this into the TG SW Integration Kits– Derek Simmel (PSC)