Upload
doanthuy
View
264
Download
3
Embed Size (px)
Citation preview
Tony Wildish!Dan Udwary
Advanced Git and Gitlab
-1-
May30,2017
Advanced Gitlab
• Prerequisites• BranchingandTagging• Buildingmul<plecontainers• Pushingimagestomul1plerepositories• Usingmetadataincontainers• DeployingrunnersonNERSChosts• Bestprac<ces&recommenda<ons
• =>Getthecodeforthistutorial:– Forkthetutorialrepository,thencloneyourforktoyourlaptop– h@ps://gitlab.com/TonyWildish/gitlab-advanced/
-2-
Prerequisites
• Familiaritywithgit,docker,gitlab– Git,version2.11orhigher– Docker,version1.12.3orhigher– Anaccountongitlab.com
• Earliertutorials:– h@ps://www.nersc.gov/assets/Uploads/Git+Docker-Tutorial-Dec01-2016.pdf
• Doexercises4and5– h@ps://www.nersc.gov/assets/Uploads/2017-02-06-Gitlab-CI.pdf
• Dothefirstexercise
-3-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Bonus gitlab tip: Notification emails
-4-
GotoyouraccountseXngs,No1fica1ons,Custom
Bonus gitlab tip: Notification emails
-5-
Chooseany/allfields,cansetper-projecttoo
Branching and tagging
• Branches– Allowparalleldevelopmentinasinglerepository– Createbranchesasneeded,deletewhenobsolete– Canmergebranchesifyoulike,orkeepforever
• Bugfixbranches:merge,deletethebranch• Featurebranches:keepforever.
– Canmergeback&forthtocontroldivergence
-6-
Master
Feature1
Bugfix1
“ProGit”,bySco@Chacon,Chapter3
Branching and tagging
• Tags– Sta1clabel,iden1fiesapar1cularcommit– Easilyrecoverpar1cularversionatany1meinfuture– Oncepushed,tagsshouldn’tbedeletedormoved!
-7-
Master
Feature1
Bugfix1
Tag1 Tag2
Tag3 Tag4
Branching and tagging
• Tagsandbranchesingitlab– Canbeusedtoiden1fybuildproducts,labelimagesetc
• Ifthere’satag,usethat• Ifnot,usethebranchname• ‘master’branch->‘latest’dockerversion(byconven1on)
– Let’sdoexercise01!
-8-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Working with forked repositories
• Howdoyoukeepaforkedrepositoryuptodate?– Addtheoriginalsourceasanother‘remote’repository
-9-
>[email protected]:TWildish/jgi-lapinpy.gitCloninginto'jgi-lapinpy'...[…]
>cdjgi-lapinpy/>[email protected]:berkeleylab/jgi-lapinpy.git>[email protected]:TWildish/jgi-lapinpy.git(fetch)[email protected]:TWildish/jgi-lapinpy.git(push)[email protected]:berkeleylab/jgi-lapinpy.git(fetch)[email protected]:berkeleylab/jgi-lapinpy.git(push)
>gitpullupstreammasterFrombitbucket.org:berkeleylab/jgi-lapinpy*branchmaster->FETCH_HEADUpda1nga3f5e1e..03943c8Fast-forward
“ProGit”,bySco@Chacon,Sec1on2.5
Working with forked repositories
• Howdoyoukeepaforkedrepouptodate?– Addtheoriginalsourceasanother‘remote’repository
-10-
>[email protected]:TWildish/jgi-lapinpy.gitCloninginto'jgi-lapinpy'...[…]
>cdjgi-lapinpy/>[email protected]:berkeleylab/jgi-lapinpy.git>[email protected]:TWildish/jgi-lapinpy.git(fetch)[email protected]:TWildish/jgi-lapinpy.git(push)[email protected]:berkeleylab/jgi-lapinpy.git(fetch)[email protected]:berkeleylab/jgi-lapinpy.git(push)
>gitpullupstreammasterFrombitbucket.org:berkeleylab/jgi-lapinpy*branchmaster->FETCH_HEADUpda1nga3f5e1e..03943c8Fast-forward
“ProGit”,bySco@Chacon,Sec1on2.5
Building multiple containers
• Supposeyouhaveapar<cularpackagewith:– Afewcoredependencies,verysmalltotal– Severalop1onalextrasthataddhundredsofMB
• Howdoyoubuildanop<malcontainer?– Includeeverything->baggagethatnotallusersneed– Leavestuffout->don’tsa1sfyallusers
• Solu<on:– Buildtwocontainers(ormore)inthesamerepository
-11-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Building multiple containers
• GitlabsupportsbuildingDockerimageswithnamesotherthantherepositoryname– DefaultDockernamestructure
• $REGISTRY_USER/$APPLICATION:$RELEASE_TAG– Extendedsyntax:
• $REGISTRY_USER/$APPLICATION/real-name:$RELEASE_TAG
– Useextendedsyntaxrepeatedlyin.gitlab-ci.yml,withdifferent‘real-name’s
– “myapp-lite”&“myapp”,or“myapp”&“myapp-full”
– Seeexercise02!
-12-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Pushing images to multiple repositories
-13-
GITLABandSHIFTERvariablespointto
differentregistryhosts
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Pushing images to multiple repositories
-14-
BuildandpushtoGITLAB
Re-tag,pushtoSHIFTER
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Pushing images to multiple repositories
• Caveat:Security!– Gitlabhandsyoualogin-tokenforeverybuild– Forshiser,onceyou’reinsidethefirewall,there’snoauthen1ca1onneeded,sonotoken
– Anywhereelse,youprobablyneedatokenorpassword,butwheredoyoustoreit?• Can’tbeintherepository,istoovisible• Hastobeintherunnerrun1meenvironmentsomehow• CandothisinSPIN,thoughnotverysecurelyatthemoment• Candoitonyourlaptops• Wanttodoitelsewhere?comeforachat
– Exercise03,inyourown1meJ
-15-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Using metadata in containers
• Passinforma<onfromthebuildenvironment– Totheimage,ortotheuseratrun1me
• Telltheuseranythingtheymightwanttoknow:– Whatrun1meenvironmentthesoswareneeds– Whatleveloftes1ng,cer1fica1onhasbeenperformed– Pointerstodocumenta1on,sourcecode,maintainers…– Run1medetails:
• wherethecontainerlooksforinput• whereitexpectstobeabletoputoutput…
-16-
h@p://docs.master.dockerproject.org/v1.5/userguide/labels-custom-metadata/h@ps://speakerdeck.com/garethr/shipping-manifests-bill-of-lading-and-docker-metadata-and-container
ThanksMichael,Alex
Using metadata in containers
-17-
Developmentenvironment
Buildcontext(dockerdaemon)(ARGXYZ=$XYZ)
Dockerimage(LABELXYZ=$XYZ)
Run1meenvironment(container)(ENVXYZ=$XYZ)
dockerinspect…|grepXYZ
dockerrun…echo$XYZ
dockerbuild…--build-argsXYZ=123
Howmetadatagoesfromthebuildenvironmenttotheimage,andtotherunningcontainer
SeeDockerfile.metadataintherepo
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Using metadata in containers
• Howcanweusemetadata?– E.g.definingaproperontology– Automa1ngpipelines,tes1ng,discovery…
• Workinggroup(?)toinves<gatethis– Probablylaterintheyearaserthemigra1on– Volunteers/sugges1onsgratefullyaccepted!
– Exercise04!
-18-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Deploying runners on NERSC hosts
• ArunneratNERSCwithwrite-accessto$HOMEetc?• Youcandothis,butthereareseriousrisksinvolved!
– Don’tsharetherunnerregistra1ontokenwithanyone• ~=givingthemyourNERSCpassword
– Don’tgiveotherusersmaster-levelaccesstoyourrepository– Consideralterna1ves:
• UseaDockerimage,withyourcustombuildenvironment,onSPIN• UseaVMsomewhere…
– Talktoaconsultantbeforea@emp1ngthis!
– Someoftheserisksaregitlab-specific– Someareinherentinrunninganyinternet-enabledservices
-19-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Deploying runners on NERSC hosts
• Basicrecipe– Downloadthebinaryforagitlabrunner– Registerit,giveitahost-specificconfigfile– Giveitspecifictagswhenregistering,toiden1fyit– Usethosetagsinyour.gitlab-ci.ymlfile– Yourpipelinecanroamovertheen1refilesystemifyouwant,butit’suptoyouthentoensurethedirectoriesyouuseareclean
– Seeexercise05fordetails–wewon’tdothistoday!
-20-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Other gitlab features • API,programmableinterfacetoGitlab
– h@ps://docs.gitlab.com/ee/api/• SeeJGI/gitlab-cli-toolsrepoforsomebasictools,contribu1onswelcome!
• Buildhooks– Triggerac1onsonexternalservicesotherthangitlab
• Similarcapabili1esongithub,bitbucket– Triggerac1onsingitlabfromexternalservice
• E.g.nightlybuild,regardlessofcommits
• Mirroringrepositories– Masterrepositoryinbitbucket/github?– Canmirrortogitlab,automa1cally,transparently
• Issue-tracking,wiki…– Othergoodiescomeforfreewithgitlab,aswithotherhos1ngservices
-21-
Best practices, recommendations
• Git:– Usethefork/pull-requestmodelinsteadofgran1ngpeopledirect-commitaccesstoyourrepository
– Usebranchestoexperiment,tryoutbugfixesetc• Mergelong-livedbranchesfrequentlytocontroldivergence
– Usetagstoiden1fystableversions,releasesetc
– Don’tdeleteormovetagsoncethey’repushedtothemasterrepository
-22-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Best practices, recommendations
• Gitlab:– Buildmul1pleDockerimagesifyouhavedifferentuse-casestoservefromthesamecode-base
– Pushingtomul1pleregistriesletsusersaccessyourimagesfrommanyplaces,easily
– Usemetadatainyourcontainers!• HelpusestablishstandardsforJGIcontainermetadata
– Controlaccesstoyourrepositories• Don’tgiveouttherunner-registra1ontoken• Avoidgivingothersadmin/developer-accesstotheproject• ThinktwicebeforedeployingrunnersonNERSCresources
-23-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
Finally…
• You’reallexpertsnow,soupdateyourresumes!
– “experiencebuildingandop1mizingDockerimagesforbioinforma1csosware”
– “experienceconfiguringandusingcon1nuous-integra1onpla~orms,suchasgitlab,toautomatebuildinganddeployingsosware”
– “in-depthunderstandingofbest-prac1cesforsoswaremanagement,suchasversioncontrolwithgitanduseofmetadatatodescribeDockerimages”
– “understandingofgitworkflowmodelsforteams,includingtheuseofbranches,tags,anddeveloperaccess-control”
-24-
h@ps://gitlab.com/TonyWildish/gitlab-advanced/
National Energy Research Scientific Computing Center
-25-