Embed Size (px)
Citation preview
GACRCUserQuickGuide:
MigratingfromZclustertoSapelo
8/4/17 1
TheGACRCStaffVersion1.0
DiscussionPoints
I. RequestSapeloUserAccount
II. Systems
III. TransferFiles
IV. ConfigureSoftwareEnvironment
V. JobSubmissionWorkflow
VI. JobSubmissionScript
VII. Appendices
8/4/17 2
Sapeloclusteruseraccount:[email protected]
Note:AvalidofficialUGAMyIDisaMUSTtocreateauseraccount!
38/4/17
RequestSapeloUserAccount
PIRequestUserAccount
PIVerification
NewUserTraining
UserAccount
Provisioning
WelcomeLetter
1. TheUGAPIusestheGACRConlineformhttp://help.gacrc.uga.edu/userAcct.php to
requestauseraccountforagroupmember.
2. Oncewereceivedtherequest,wewillverifyitwiththePI.
3. AfterverificationbythePI,thenewuserwillberequiredtoattendatrainingsession.
4. Aftertheuserattendedtraining,wewillprovisionaSapeloaccountfortheuser.
5. Awelcomeletterissenttotheuseronceuseraccountisready.
Systems– OverviewDiagram
TransferNodexfer.gacrc.uga.edu
LoginNodezcluster.rcc.uga.edu
LoginNodesapelo1.gacrc.uga.edu
InteractiveNode
InteractiveNode/home/abclab/userID /escratch4/userID
/home/MyID /lustre1/MyID
/project/abclab
ssh
ssh
ssh/scp
qlogin
qlogin
Zcluster
Sapelo
AccessingLine
8/4/17 4
Systems– Overview
LogontoZclusterLogin Node: ‘ssh [email protected]’(usingzclusterpassword)
OpenZclusterInteractiveNode: type‘qlogin’commandfromloginnode
ZclusterHomeDirectory: /home/abclab/userID
ZclusterGlobalScratch: /escratch4/userID (4TBquota)
Create WorkingDirectoryonZclusterGlobalScratch: type‘make_escratch’commandfromloginnode
Log onto SapeloLoginNode: ‘[email protected]’(usingUGAMyIDpassword)
OpenSapeloInteractiveNode: type‘qlogin’ commandfromloginnode
SapeloHomeDirectory: /home/MyID(100GBquota)
SapeloGlobalScratch: /lustre1/MyID
Create WorkingDirectoryonSapeloGlobalScratch: use‘mkdir’commandin/lustre1/MyIDfrominteractivenode
LogontoGACRCTransferNode: ‘ssh [email protected]’ (usingUGAMyIDpassword)
GroupProjectSpace: /project/abclab(“abc”areinitialsofthePI ofthegroup)
8/4/17 5
Systems– StorageEnvironmenthttps://wiki.gacrc.uga.edu/wiki/Disk_Storage
Name Role Purpose Zcluster Sapelo
Home sshlandingspot staticdata, e.g.,scripts,software /home/abclab/userID /home/MyID
(100GBquota)
GlobalScratch
Jobworkingspace
Current Jobdatabeingused byrunningjobs
/escratch4/userID(4TBquota)
/lustre1/MyID(noquota–limitedbycapacity)
Project Activedatastorage
Non-currentactivedataforfutureanalysis
/project/abclab(1TB initialquota)
8/4/17 6
Systems– ComputingResourcesOverviewhttps://wiki.gacrc.uga.edu/wiki/Systems
8/4/17 7
Zcluster
150x8-core16GBRAMnodeswithIntelXeon processors(total1200cores)45x12-core48GBRAMnodeswithIntelXeon processors(total540cores)
4x8-core192GBRAMnodeswithIntelXeon processors(total32cores)10x12-core256GBRAMnodeswithIntelXeon processors(total120cores)2x32-core512GBRAMnodeswithIntelXeon processors(total64cores)
6x32-core64GBRAMnodeswithAMDOpteronprocessors(total192cores)
4x12-core96GBRAMnodeswithIntelXeonprocessors, 8NVIDIAK20XmGPUcardseach
Sapelo
112x48-core128GBRAMnodeswithAMDOpteronprocessors(total5376cores)4x48-core256GBRAMnodeswithAMDOpteronprocessors(total192cores)
6x48-core512GBRAMnodeswithAMDOpteronprocessors(total288cores)1x48-core1TBRAMnodewithAMDOpteronprocessors (total48cores)3x 28-core1TBRAMnodewithIntelXeon processors(total84cores)
2x16-core128GBRAMnodeswithIntelXeonprocessors, 8NVIDIAK40mGPUcardseach
85+buy-innodeswithvariableconfigurations
TransferofFileshttps://wiki.gacrc.uga.edu/wiki/Transferring_Files
8/4/17 8
AllSapeloandzclusterusers,aswellasallGACRCPIs,canaccesstheGACRCtransfernodes(xfer.gacrc.uga.edu)andusingUGAMyID andMyIDpasswordtoauthenticate.
Notethatauser'shomedirectoryontransfernodeisthesameastheuser'sSapelohomedirectory,whichisnotthesameastheuser'sZclusterhomedirectory.
Transfernodescanaccessthefollowingfilesystemsusingtheirfullpaths:
SapeloHomeDirectory: cd/home/MyID
ZclusterHomeDirectory: cd/panfs/pstor.storage/home/abclab/userID
ZclusterGlobalScratch: cd/escratch4/userID
SapeloGlobalScratch: cd/lustre1/MyID
GroupProject Folder cd/project/abclab
Detailedinstructionscanbefoundatthelinkabove
local
FileTransferringTreehttps://wiki.gacrc.uga.edu/wiki/Transferring_Files
8/4/17 9
scp/ssh
TransferNode
/home/MyID
/lustre1/MyID
/project/abclab
/escratch4/userID
/panfs/pstor.storage/home/abclab/userID
ü scp:Remotefiletransferringbetweenyourlocalcomputerandyourfoldersonfilesystems
ü ssh:Loggingontotransfernode.Onceloggedon,youcanusecp ormv forfiletransferbetweenfilesystems.
ConfigureSoftwareEnvironmenthttps://wiki.gacrc.uga.edu/wiki/Software
8/4/17 10
SapeloUser issuesmodule commandstoconfigurevariouspathsfor softwarepackages
(inajobsubmissionscriptoronthecommandlineofaninteractivesession):
1. moduleavail: showallmodulesavailableoncluster2. moduleloadmoduleName/version:loadtheneededmodule“moduleName”withaversion3. modulelist:listallmodulesyoualreadyloaded
Forexample:
$moduleloadpython/2.7.8$moduleloadR/3.2.3$modulelistCurrentlyLoadedModules:1) moab/8.1.1 2)StdEnv 3)python/2.7.8 4) R/3.2.3
ZclusterUser “manually”exportsvariouspathsforsoftwarepackages(inajobsubmissionscriptoronthecommandlineofaninteractivesession),forexample:
export PATH=……:$PATH
JobSubmissionWorkflow(general)
8/4/17 11
Zcluster Sapelo
1. Logontozclusterloginnode:
2. Createaworkingdir onglobalscratch:
make_escratch
3. Changetoworkingdironglobalscratch:
cd/escratch4/userID/userID_mmm_dd
(mmm, dd:dateabbreviations)
4. Create azclusterjobsubmissionscript:
nano./sub.sh
5. Submitjob:
qsub-qrcc-30d./sub.sh
1. LogontoSapelologinnode:
2. GotoSapelo Interactivenode:
qlogin
3. ChangetoSapeloglobalscratch:
cd/lustre1/MyID
4. Createaworkingdironglobalscratch:
mkdir./workDir
5. Changetoworkingdironglobalscratch:
cd./workDir
6. CreateaSapelojobsubmissionscript:
nano./sub.sh
7. Submitjob:
qsub./sub.sh
JobSubmissionScript(NCBIBlastasexample)
8/4/17 12
sub.shonZcluster sub.shonSapelo
#!/bin/bash
cd`pwd`
exportPATH=/usr/local/ncbiblast/latest/bin:$PATH
blastall-pblastn-d/db/ncbiblast/latest/nt \-a4 -iQUERY\-oout.QUERY
#PBS-S/bin/bash#PBS-qbatch#PBS-NtestBlast#PBS-lnodes=1:ppn=4:AMD#PBS-lmem=20gb#PBS-lwalltime=480:00:00
cd$PBS_O_WORKDIR
moduleloadncbiblast+/2.2.26
blastall-pblastn-d/db/ncbiblast/nrte/latest/nt \-a4 -iQUERY\-oout.QUERY>outputfile2>&1
qsub-qrcc-30d-pethread4./sub.sh qsub./sub.sh
138/4/17
Appendix1:
148/4/17
Tasks Zcluster Sapelo
Commonfile/diroperations
ls,cd, pwd,file,cat, less,more,cpmv,mkdir,rm,rmdir,dos2unix,mac2unix,etc.
Transfer remotefiles scp, sftporSSHSecureClientFileTransfer,FileZilla,WinSCP
Openinteractivesession qlogin
Createworkingdirectoryonglobalscratch make_escratch mkdirsubdirectoryName
Edit scriptfiles nanosub.sh or vim sub.sh
Submitabatchjob qsub–qqueueName–l…–pe …sub.sh qsubsub.sh
checkjobstatus qstat
Checkdetailedjobinfo qsj jobID,qstat–jjobID,qacct–jjobID qstat–fjobID,showjobsjobID
Cancelajobfromqueue qdeljobID, qdel–uuserID qdeljobID
Appendix2:CommonCommandSummaryhttps://wiki.gacrc.uga.edu/wiki/Command_List
158/4/17
Appendix3:CurrentprocessforaUGAPItorequest(re)activationorcreationofaMyIDforanexternalcollaborator
Tokeepauser'sMyIDactiveafter he/sheleavesUGA,ortocreateaMyIDforanexternalresearchcollaborator:
1. ThePIaccessestheEITShelpdeskportalathttps://uga.teamdynamix.com/TDClient/Home/
2. Click on"MyIDAccountRequest”ontherightsideofthepagein“PopularServices”section.
3. UseMyID credentialstologontheservicepage.4. IntheMyIDRequestFrom,enterhis/herinformationandselect"GACRC
RemoteCollaborator"fortheAccountTypefield.5. AfterverificationbytheGACRCDirector,EITSwillgeneratecollaborator’sMyID6. IfcollaboratorisvisitingUGAcampus,MyID canbeobtainedthroughOVPR’s
VisitingResearcher&Scholarprogram– https://research.uga.edu/visiting/
• GACRCWebsite:http://gacrc.uga.edu
• GACRCWiki:https://wiki.gacrc.uga.edu/wiki/Main_Page
• Systems: https://wiki.gacrc.uga.edu/wiki/Systems
• HowtoTransferFiles:https://wiki.gacrc.uga.edu/wiki/Transferring_Files
• CurrentSoftwareList: https://wiki.gacrc.uga.edu/wiki/Software
• HowtoSubmitaJob:https://wiki.gacrc.uga.edu/wiki/Running_Jobs_on_Sapelo
• HowtoGetHelp: http://help.gacrc.uga.edu/
• GACRCTrainingOpportunities:https://wiki.gacrc.uga.edu/wiki/Training
• HowtoregisteranewGACRCNewLab:http://help.gacrc.uga.edu/labAcct.php
• HowtoregisteranewGACRCUser:http://help.gacrc.uga.edu/userAcct.php
8/4/17 16
Appendix4:UsefulLinks
