Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Steinbuch Centre for Computing (SCC)
Funding: www.bwhpc-c5.de
File Systems and Software Modules
Simon Raffeiner, Scientific Computing and Simulation Dept., SCC, KIT
File Systems and Software modules / S.Raffeiner05/04/2017 2
Reference: bwHPC-C5 Best Practices Repository
Most information given by this talk can be found at http://bwhpc-c5.de/wiki:
Category:Hardware_and_Architecture
Environment_Modules
Batch_Jobs
File Systems and Software modules / S.Raffeiner05/04/2017 3
Material: Slides & Scripts
https://indico.scc.kit.edu/indico/event/278/
@bwUniCluster/ForHLR I/ForHLR II:/pfs/data1/software_uc1/bwhpc/kit/workshop/2017-04-05
Abbreviation/Colour code Full meaning
$ command -option value $ = prompt of the interactive shellThe full prompt may look like: user@machine:path$The command has been entered in the interactive shell session
<integer><string>
<> = Placeholder for integer, string etc
foo, bar Metasyntactic variables
How to read the following slides
File Systems and Software modules / S.Raffeiner05/04/2017 4
File Systems
File Systems and Software modules / S.Raffeiner05/04/2017 5
File Systems
bwUniCluster, ForHLR I / ForHLR II, bwForCluster, …
Too many file systems to list them all, please see documentation!
Common „rules of thumb“
There is a small, global, permanent $HOME directory→ Usually backed up
There are one or more large, global temporary $WORK directories→ Old files might be deleted automatically→ Usually NOT backed up
There might be local, non-global, temporary storage on the computing nodes
→ Usually named $TMP, $SCRATCH or something similar→ Will be wiped when the job ends
File Systems and Software modules / S.Raffeiner05/04/2017 6
$HOME = Home directory
$HOME:
HowTo goto:
@ bwUniCluster/ForHLR: Current quota:
Diskusage:
@ KIT: $HOME directories of bwUniCluster, ForHLR I / II are the sameBut: different hardware, libraries, queueing etc.→ generalise your scripts to work on all systems using $CLUSTER
$ lfs quota -u $(whoami) $HOME
if [ ${CLUSTER} == “uc1“ ]; then <operations>fi
if [ ${CLUSTER} == “uc1“ ]; then <operations>fi
$ grep -E "$(whoami)|Account" ~/../../diskusage
$ cd $HOME
$ cd
File Systems and Software modules / S.Raffeiner05/04/2017 7
$HOME @ bwForClusters
$HOME:
@bwForCluster: Please check the documentation!
File Systems and Software modules / S.Raffeiner05/04/2017 8
$PROJECT = Project directory of ForHLR I/II
ONLY ForHLR I/II:All features of $HOME
Access granted based on approved projects
→ assigned „name/acronym“, e.g. /project/fh1-project-scs/mr2515
→ $PROJECT_GROUP, e.g. „fh1-project-scs“
Access project home directory:
Do not use: $HOME → it has very low quota for the project group!
Quota of Project:
$ cd $PROJECT
$ lfs quota -g ${PROJECT_GROUP} ${PROJECT}
File Systems and Software modules / S.Raffeiner05/04/2017 9
$WORK = Working directory
bwUniCluster/ForHLR I/II → addional parallel file system with limited lifetime, no redundancy, quotas
especially designed for parallel access and for a high throughput to large files
2 concepts of access via:(A) → $WORK(B) → workspaces
(A) $WORK:Change to it via:
Quota:
But: files no longer needed should be removed
→ any file inside your $WORK older than 28 days will be deleted
$ cd $WORK
$ lfs quota -u $(whoami) $WORK
File Systems and Software modules / S.Raffeiner05/04/2017 10
Workspaces: lifetime on allocated folder HowTo:
→ http://www.bwhpc-c5.de/wiki/index.php/BwUniCluster_File_System#Workspaces
$ ws_allocate foo 10 Allocate workspace foo for 10 days
$ ws_list -a List all your workspaces
$ ws_find foo Get absolute path of workspace foo
$ ws_extend foo 5 Extend lifetime of your workspace foo by 5 days from now. You can extend 3 times
→ max. lifetime of foo = 240 days (U+F) 90 days (J)
$ ws_release foo Manually erase your workspace foo
$ ws_allocate scratch 300WS: duration is longer than allowed for this filesystem!WS: setting duration to allowed maximum of 60 days!Workspace created. Duration is 1440 hours. Further extensions available: 3/work/workspace/scratch/mr2515-scratch-0
Example:
Workspaces = Working directory (1)
File Systems and Software modules / S.Raffeiner05/04/2017 12
Exercise 1
Allocate two workspaces
$ ws_allocate test 30Workspace created. Duration is 720 hours. Further extensions available: 3/work/workspace/scratch/mr2515-test-0
$ ws_allocate scratch 50Workspace created. Duration is 1200 hours. Further extensions available: 3/work/workspace/scratch/mr2515-scratch-0
File Systems and Software modules / S.Raffeiner05/04/2017 13
Exercise 2
List workspaces
$ ws_listFilesystem: defaultWorkspace ID Workspace location Creation date Remaining time------------------------------------------------------------------------------test Apr 5 09:35:42 29 days 23 hours /work/workspace/scratch/mr2515-test-0 acctcode:900000 available extensions:3scratch Apr 5 09:35:01 49 days 23 hours /work/workspace/scratch/mr2515-scratch-0 acctcode:900000 available extensions:3
File Systems and Software modules / S.Raffeiner05/04/2017 14
Exercise 3
Find workspace path and switch to it
$ ws_find scratch/work/workspace/scratch/mr2515-scratch-0
$ ws_find test/work/workspace/scratch/mr2515-test-0
$ cd $(ws_find test)$ pwd/work/workspace/scratch/mr2515-test-0
File Systems and Software modules / S.Raffeiner05/04/2017 15
Exercise 4
Extend the lifetime of a workspace
$ ws_extend test 60Duration of workspace is successfully changed!New duration is 1440 hours. Further extensions available: 2
$ ws_listFilesystem: defaultWorkspace ID Workspace location Creation date Remaining time--------------------------------------------------------------------------------test Apr 5 09:37:12 59 days 23 hours /work/workspace/scratch/mr2515-test-0 acctcode:900000 available extensions:2scratch Apr 5 09:35:01 49 days 23 hours /work/workspace/scratch/mr2515-scratch-0 acctcode:900000 available extensions:2
File Systems and Software modules / S.Raffeiner05/04/2017 16
Exercise 5
Release workspaces
$ ws_release test/work/workspace/scratch/mr2515-test-0Info: Workspace was deleted.
$ ws_release scratch/work/workspace/scratch/mr2515-scratch-0Info: Workspace was deleted.
File Systems and Software modules / S.Raffeiner05/04/2017 18
Software System
File Systems and Software modules / S.Raffeiner05/04/2017 19
Software (=Environment) modules
By default manual setup of $PATH, $LD_LIBRARY_PATH … forcompilers, libraries and software packages etc.
→ Getting complicated if multiple versions of same software installed
Solution:dynamic modification of the session environment by
→ instruction sets stored in modulefiles
HowTo?load and unload instruction sets (= modulefiles)
How to use modulefiles in general?
More information:http://www.bwhpc-c5.de/wiki/index.php/Environment_Modules
$ module help $ module help
File Systems and Software modules / S.Raffeiner05/04/2017 20
modulefiles: available / search
Display all modulefiles
Display all modulefiles with prefix „compiler“
$ module avail
$ module avail compiler
File Systems and Software modules / S.Raffeiner05/04/2017 21
modulefiles: help / whatis
Show help of modulefiles, e.g.
Show short info modulefile
$ module help compiler/intel
$ module whatis compiler/intel
compiler/intel : Intel(R) compiler suite (icc, icpc, ifort), debugger (gdb-ia), IPP and TBB
File Systems and Software modules / S.Raffeiner05/04/2017 22
modulefiles: show
Show all instructions of modulefile $ module show compiler/gnu/4.7
File Systems and Software modules / S.Raffeiner05/04/2017 23
modulefiles: load (1)
Modulefiles are sorted in categories, software name and versions:
Load a default software:
e.g. Intel compiler
→ loads currently Intel compiler suite 16
→ loads currently Intel-MPI 5.1.3 for Intel compiler 16.0
Display all loaded modules
$ module load <category>/<software_name>/<version>
$ module load <category>/<software_name>
$ module load compiler/intel mpi/impi
$ module list
Currently Loaded Modulefiles: 1) compiler/intel/16.0(default) 2) mpi/impi/5.1.3-intel-16.0(default)
File Systems and Software modules / S.Raffeiner05/04/2017 24
modulefiles: search
@ ForHLR
$ module spider openmpi
------------------------------------------------------------------------------ Core/mpi/openmpi:------------------------------------------------------------------------------ Versions: Core/mpi/openmpi/1.8_intel Core/mpi/openmpi/1.8 Core/mpi/openmpi/1.10_intel Core/mpi/openmpi/1.10 Core/mpi/openmpi/2.0
(..)
File Systems and Software modules / S.Raffeiner05/04/2017 25
modulefiles: categories & dependencies
Module names already implicate dependencies:
→ Category/softwarename/version_attributes-dependenciese.g numlib/fftw/3.3.5-impi-5.1.3-intel-16.0
→ fftw package version 3.3.5, compiled with Intel 16.0 and Intel-MPI 5.1.3
Categories:
compiler/ for compiler, e.g. intel, gnu, pgi, open64
devel/ for debugger, e.g. ddt, and development tools, e.g. cmake, itrac
mpi/ for MPI libraries, e.g. impi, openmpi, mvapich(2)
numlib/ for numerical libraries, e.g. Intel MKL, ACML, nag, gsl, fftw
lib/ for other libraries, e.g. netcdf, global array
bio/ for biology software, e.g. bowtie, abyss, mrbayes
cae/ for CAE software, e.g. ansys, abaqus, fluent
chem/ for chemistry software, e.g. gromacs, dacapo, turbomole
math/ for mathematics software, e.g. matlab, R
phys/ for physics software, e.g. geant4
vis/ for visualisation software, e.g. vmd, tigervnc
File Systems and Software modules / S.Raffeiner05/04/2017 26
modulefiles: load (2)
@ bw{Uni,For}Cluster - Conflicts:a) load different software version in the same session, e.g. Intel:
b) load module with dependencies on other modules
→ NOT an issue if the cluster uses Lmod (ForHLR I/II)
$ module load compiler/intel/14.0$ module load compiler/intel/15.0
compiler/intel/13.1(394):ERROR:150: Module 'compiler/intel/15.0'conflicts with the currently loaded module(s) 'compiler/intel/14.0'
$ module load mpi/openmpi/1.10-intel-16
Loading module dependency 'compiler/intel/16.0'.compiler/intel/16.0(394):ERROR:150: Module 'compiler/intel/16.0'Conflicts with the currently loaded module(s) 'compiler/intel/15.0'
File Systems and Software modules / S.Raffeiner05/04/2017 28
modulefiles: list
Depending on the cluster, default modules might be loaded or not
$ module list
Currently Loaded Modules: 1) dot 2) numlib/mkl/11.3 3) mpi/openmpi/1.8 4) compiler/gnu/5
File Systems and Software modules / S.Raffeiner05/04/2017 29
modulefiles: unload/swap (1)
To remove module foo:
or
@ bw{Uni,For}Cluster
be aware that you might create inconsistencies
e.g. you can remove
compiler/intel/16.0 while mpi/openmpi/1.10-intel-16.0 is still loaded
Swap = remove + load
e.g.:
$ module unload foo $ module remove foo
$ module swap compiler/intel/15.0 compiler/intel/16.0
File Systems and Software modules / S.Raffeiner05/04/2017 30
modulefiles: unload/swap (2)
@ ForHLR: automatic swap
$ module load compiler/gnu/5
Lmod is automatically replacing "compiler/intel/16.0" with "compiler/gnu/5".
File Systems and Software modules / S.Raffeiner05/04/2017 31
Private modulefiles
Each user can create own modulefiles:e.g. modulefiles that adds path of own programs, $HOME/special, to $PATH
→ content of this modulefile „mybin“
→ place „mybin“ under $HOME/privatemodules
→ to make all own modules visible to “module avail” command, enter:
or → former: own modules have lower priority than system ones if equally named
→ latter: own module have higher priority
Remove own modules:
or
#%Module1.0
Append-path PATH “$env(HOME)/special”
$ module load use.own $ module use $HOME/privatemodules
$ module unload use.own $ module unuse $HOME/privatemodules
File Systems and Software modules / S.Raffeiner05/04/2017 33
Exercise 1
Check loaded modules after login
Check default GNU Compiler version
$ module list
No Modulefiles Currently Loaded.
$ gcc --versiongcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)Copyright (C) 2015 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
File Systems and Software modules / S.Raffeiner05/04/2017 34
Exercise 2
Load a different GNU compiler version
$ module load compiler/gnu/5.2
$ echo $PATH/opt/bwhpc/common/compiler/gnu/5.2.0/bin:/software/all/bin:/usr/lib64/qt-3.3/bin:/opt/moab/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/kit/scc/mr2515/.local/bin:/home/kit/scc/mr2515/bin
$ gcc --versiongcc (GCC) 5.2.0Copyright (C) 2015 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
File Systems and Software modules / S.Raffeiner05/04/2017 35
Exercise 3
Switch GNU compiler with Intel Compiler
$ echo $CCgcc
$ module load compiler/intel
$ echo $CCicc
File Systems and Software modules / S.Raffeiner05/04/2017 36
Exercise 4
Try to compile an MPI application
Try to load OpenMPI
$ mpiccbash: mpicc: command not found…
$ module load mpi/openmpi
module load mpi/openmpiLoading module dependency 'compiler/intel/14.0'.compiler/intel/14.0(401):ERROR:150: Module 'compiler/intel/14.0' conflictswith the currently loaded module(s) 'compiler/intel/16.0'compiler/intel/14.0(401):ERROR:102: Tcl command execution failed: conflictcompiler/intel
ERROR: Failed to load module dependency 'compiler/intel/14.0'. Please checkconflicts with already loaded modules.
File Systems and Software modules / S.Raffeiner05/04/2017 37
Exercise 4
What is happening?
The default OpenMPI module was compiled against Intel Compiler version 14, but we loaded version 16 before
$ module show mpi/openmpi-------------------------------------------------------------------/opt/bwhpc/common/modulefiles/mpi/openmpi/1.8-intel-14.0:
(..)
File Systems and Software modules / S.Raffeiner05/04/2017 38
Exercise 4
Load the correct OpenMPI version
Now mpicc works
$ module load mpi/openmpi/1.10-intel-16.0
$ mpicc --versionicc (ICC) 16.0.4 20160811Copyright (C) 1985-2016 Intel Corporation. All rights reserved.