Upload
julien-pivotto
View
2.070
Download
27
Tags:
Embed Size (px)
DESCRIPTION
Talk I have given at T-Dose about git submodules and git slave and different worklow around them
Citation preview
...
Working with multiple git repositoriesWorking with multiple git repositories
Julien Pivotto
T-Dose, EindhovenOctober 26, 2014
..
whoamiwhoamiJulien PivottoJulien Pivotto
• System administrator at inuits.eu
• Git user for 5+ years• DevOps believer• Open-source defender since 2004• roidelapluie on twitter/github
..
Ops <3 DevOps <3 Dev
• Infrastructure as Code▶ SCM all the things▶ Monitoring▶ Configuration▶ Application deployment
• Taking part of software development▶ Understanding▶ Monitoring▶ High Availability▶ …
..
Me and gitMe and git
• I first used subversion (10 years ago)• Private forges and sourceforge• 5 years later a lot of projects moved to git• The Puppet community also uses a lot git• Git/Hg are the de-facto standards in OSprojects
..
Git is a free and open source distributed version control systemdesigned to handle everything from small to very large projects
with speed and efficiency.
Source: http://git-scm.com/
..
git loggit log
• Initiated in 2005 by Linus Torvalds• Replacement of closed-source BitKeeper• Created for the Linux Kernel development• Now used by thousands of projects
..
git is distributedgit is distributed
• Everything can be done in local (exeptpull&push)
• Work with several remotes• Share code with anyone• No unique central repository• A lot of workflows possible
..
Branching and merging is easyBranching and merging is easy
• You can branch locally• Merging is easy and simple• You can squash commits
..
How to access a repoHow to access a repo
• Locally (Local filesystems)• SSH• HTTP/HTTPs• Git protocol
..
git is open-sourcegit is open-source
• Published under GPL-2• Written in C• A lot of frontends/backends• A lot of libraries• A lot of hosting services• And a lot of users
..
Multiple git repos: why?Multiple git repos: why?
• Code isolation: plugins, libraries• Getting faster feedback from testing• Contributing back to upstream projects• Separate apps that are released together• Share code between projects
..
Multiple git repos usecasesMultiple git repos usecases
• Infrastructure as Code• Projects released together• Projects that embed external libraries• …
..
git submodules
..
git submodulesgit submodules
• Built-in with git• Embeds external repositories in asubdirectory or your repo
• Points these external repositories in aparticular commit
..
git submodulesgit submodules
• Submodules act like traditional gitrepositories
• Super repo contains a file with thesubmodules urls
• Super repo contains objects that can beupdated with commit hashes
..
Initiate super-repoInitiate super-repo.
.
$ git init .Initialized empty Git repository in /home/roidelapluie/super-repo/.git/$ > README$ git add README$ git commit -m "Initial commit"[master (root-commit) b526be8] Initial commit1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 README
..
Initiate subrepoInitiate subrepo.
.
$ git init .Initialized empty Git repository in /home/roidelapluie/subrepo/.git/$ > README$ git add README$ git commit -m "Initial commit"[master (root-commit) 41e64b6] Initial commit1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 README
..
Add first module in parent repoAdd first module in parent repo.
.
$ git submodule add ../subrepo/ subrepoCloning into 'subrepo'...done.$ git statusOn branch masterChanges to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: .gitmodulesnew file: subrepo
..
What just happened?What just happened?
• git created a .gitmodules file• It cloned the subrepo in a directory• It created a git object that points to the repo• That object contains the git revision
..
.gitmodules file.gitmodules file
.
.
$ cat .gitmodules[submodule "subrepo"]
path = subrepourl = ../subrepo/
• Unique file for all the submodules• Only contains the path and the url• In this case, we use relative URL
..
Commiting our changeCommiting our change
.
.
$ git commit -m "Add subrepo"[master 3251a63] Add subrepo2 files changed, 4 insertions(+)create mode 100644 .gitmodulescreate mode 160000 subrepo
..
Cloning that super repoCloning that super repo
.
.
$ git clone /home/roidelapluie/super-repo/ .Cloning into '.'...done.$ ls subrepo/$ git submodule initSubmodule 'subrepo' (/home/roidelapluie/subrepo)registered for path 'subrepo'$ git submodule updateCloning into 'subrepo'...done.Submodule path 'subrepo': checked out '41e64b6bc0c970521c296b9c5e6a0b9144778962'
..
Shorcut #1Shorcut #1
.
.
$ git submodule init$ git submodule update
.
.
$ git submodule update --initSubmodule 'subrepo' (/home/roidelapluie/subrepo)registered for path 'subrepo'Cloning into 'subrepo'...done.Submodule path 'subrepo': checked out '41e64b6bc0c970521c296b9c5e6a0b9144778962'
..
Shorcut #2Shorcut #2.
.
$ git clone /home/roidelapluie/super-repo/ .$ git submodule init$ git submodule update
.
.
$ git clone --recurse-submodules /home/roidelapluie/super-repo/ .Cloning into '.'...done.Submodule 'subrepo' (/home/roidelapluie/subrepo)registered for path 'subrepo'Cloning into 'subrepo'...done.Submodule path 'subrepo': checked out '41e64b6bc0c970521c296b9c5e6a0b9144778962'
..
Working in a submoduleWorking in a submodule.
.
$ cd subrepo$ > newfile$ git add newfile$ git commit -m "add newfile"[detached HEAD 6ee0e71] add newfile1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 newfile
$ git pushfatal: You are not currently on a branch.To push the history leading to the current (detached HEAD)state now, use
git push origin HEAD:<name-of-remote-branch>
..
Checking out a branchChecking out a branch.
.
$ git checkout masterWarning: you are leaving 1 commit behind, not connectedto any of your branches:
6ee0e71 add newfile
If you want to keep them by creating a new branch, thismay be a good time to do so with:
git branch new_branch_name 6ee0e71
Switched to branch 'master'Your branch is up-to-date with 'origin/master'.
..
Merging our commitMerging our commit
.
.
$ git merge 6ee0e71Updating 41e64b6..6ee0e71Fast-forwardnewfile | 01 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 newfile
..
Pushing our commitPushing our commit
.
.
$ git pushCounting objects: 2, done.Delta compression using up to 8 threads.Compressing objects: 100% (2/2), done.Writing objects: 100% (2/2), 247 bytes | 0 bytes/s, done.Total 2 (delta 0), reused 0 (delta 0)To /home/roidelapluie/subrepo
41e64b6..6ee0e71 master -> master
..
Pushing our commitPushing our commit
.
.
$ date > newfile$ git commit -am "update newfile"[master 8a56806] update newfile1 file changed, 1 insertion(+)
$ git pushCounting objects: 3, done.Delta compression using up to 8 threads.Compressing objects: 100% (2/2), done.Writing objects: 100% (3/3), 310 bytes | 0 bytes/s, done.Total 3 (delta 0), reused 0 (delta 0)To /home/roidelapluie/subrepo
6ee0e71..8a56806 master -> master
..
Let's rewind..Let's rewind..
• A submodule not attached to a branch▶ Because submodules are referenced by their commit hashonly
▶ It is called the DETACHED HEAD
• How to push then?▶ The easy/safe option is to checkout a branch
• You only need to do that once
..
git submodule foreachgit submodule foreach
• git submodule foreach allow you to run acommand on each repo
▶ git submodule foreach git checkout master▶ git submodule foreach ls▶ git submodule foreach git commit -am "foo"
..
Meanwhile, on the super-project…Meanwhile, on the super-project….
.
$ git statusOn branch masterChanges not staged for commit:
(use "git add <file>..." to update what will becommitted)
(use "git checkout -- <file>..." to discard changes inworking directory)
modified: subrepo (new commits)
no changes added to commit (use "git add" and/or "gitcommit -a")
..
Update a subrepoUpdate a subrepo
.
.
$ git commit -am "update subrepo"[master 731626b] update subrepo1 file changed, 1 insertion(+), 1 deletion(-)
..
LogsLogs
.
.
$ git log -p --submodule=log subrepo/commit 731626bc53d8c248647443b4c0523d96d316cd24Author: Julien Pivotto <[email protected]>Date: Sat Oct 25 22:26:56 2014 +0200
update subrepo
Submodule subrepo 41e64b6..01c1b9c:> work on newfile> update newfile> add newfile
..
Change a submodules URLChange a submodules URL
• Edit .gitmodules and commit the file• Run git submodule sync• Now the submodule will use the new URL
..
git submodule --recursivegit submodule --recursive
• Run commands recursively• You can get multiple levels of submodules• But you add complexity
▶ Each level is 1 more commit you need to write▶ Playing with remote urls becomes tricky of you work withremotes
..
Submodules integrationSubmodules integration
• Integrated in git, standard• Integrated in CI tools (Jenkins)• Nothing specific needed for pulling• Need to checkout the submodules afterpulling
..
Submodules in GerritSubmodules in Gerrit
• Gerrit integrates submodules• It can create commits in "super-repo" if acode change is merged
• Not a perfect way of doing if changes in tworepo's are depending of each other
..
Submodules in GerritSubmodules in Gerrit
• Gerrit is integrated in CI tools• Gerrit allows fine tuning ACL• Gerrit code reviews with submoduleschanges don't display changes in thesubmodules
..
Submodules pros and consSubmodules pros and cons
− Each commit in submodule is not insuper-repo
− It is easy to rollback by accident asubmodule
− Getting logs is not easy− You can forget to update super-repo− Submodules are not attached to a branch
..
Submodules pros and consSubmodules pros and cons
+ Built-in+ Can use relative urls+ You can control the version of the
submodules+ Supported by many tools+ Easy rollbacks of submodules+ git bissect works (on the super repo)
..
gitslave a.k.a. gits
..
gitslavegitslave
• First public release in 2010• Wrapper around git• "gits" can be used in place of git• Run each git commands in each subrepos
..
Initiate gitsInitiate gits
.
.
$ git init .Initialized empty Git repository in /home/roidelapluie/superrepo/.git/$ gits prepare[master (root-commit) 594c492] gits creating .gitslave1 file changed, 0 insertions(+), 0 deletions(-)create mode 100644 .gitslave
..
Add a git repoAdd a git repo
.
.
$ gits attach /home/roidelapluie/subrepo/ subrepoCloning into 'subrepo'...done.[master 926321f] gits adding "/home/roidelapluie/subrepo/" "subrepo"2 files changed, 2 insertions(+)create mode 100644 .gitignore
$ cat .gitslave"/home/roidelapluie/subrepo/" "subrepo"
..
Cloning a repoCloning a repo
.
.
$ gits clone ../superrepo/ .Cloning into '.'...done.Cloning into 'subrepo'...done.
..
Working in submodulesWorking in submodules
.
.
$ cd subrepo$ date > newfile$ git add newfile$ gits commit -m "Work on newfile"On: subrepo:
[master 1230de3] Work on newfile1 file changed, 1 insertion(+), 1 deletion(-)
On: (super-repo):On branch masterYour branch is up-to-date with 'origin/master'.nothing to commit, working directory clean
..
Pushing changesPushing changes
.
.
$ gits pushOn: subrepo:
To /home/roidelapluie/subrepo/fbcf73c..1230de3 master -> master
On: (super-repo):Everything up-to-date
..
Pulling changesPulling changes
.
.
$ gits pullOn: (super-repo), subrepo:
Already up-to-date.
..
git slave workflowgit slave workflow
• Subrepos are always attached to a branch• You can work within them "normally"• Run almost any git command with "gits"
..
PerformancePerformance
• gits --parallel 8: run 8 parallel git commands• No extra commit in the super repository• Close to traditional git workflow
..
A branching approachA branching approach
• Use gits branch and gits checkout to switchbetween branches
• All the repositories will switch branches (e.g1.0, 1.1)
• Easy merge with gits merge• Avoid upstream traditional branch names
..
gits outputgits output
• gits will simplify the output of gitcommands
• It will group the repos by output• Easier to see the status, the remotes, etc
..
Pros and cons of gitslavePros and cons of gitslave
− Not integrated in tools− Requires an additional binary− Not integrated in git− Hard to rollback on every repositories
..
Pros and cons of gitslavePros and cons of gitslave
+ Close to git workflow+ Less error-prone+ Use traditional git commands+ Every git push arrives in the repo
..
Mitigeate gitslaves' issuesMitigeate gitslaves' issues
• Use git hooks to trigger tests on CI▶ Polling SCM does not scale anyway▶ Use the "Quite period" feature
• Use CI to add the same tag in eachrepository
▶ Rollback to a tag, not a commit▶ Add a tag at each commit
..
Additional tips
..
Use SSH socketsUse SSH sockets
.
.
$ cat .ssh/configHost githost
controlmaster autocontrolpath /tmp/ssh-%r@%h:%pControlPersist 300
• Use only one SSH connection• Do not disconnect SSH
..
Find the tag of a commitFind the tag of a commit
.
.
$ git name-rev --name-only 106cb38dd0bc5ea2d2d0adef0b4006bd61884e42remotes/origin/8.5.2~1
..
git autocorrectgit autocorrect
.
.
$ git config help.autocorrect 10$ git puskWARNING: You called a Git command named 'pusk', which does not exist.Continuing under the assumption that you meant 'push'in 1.0 seconds automatically...
..
Use git hooksUse git hooks
• Multiple repositories = a lot of pull• Jenkins pulling X repos X times every 5minutes is a bad idea
• Git hooks to the resue• Reduce load on Jenkins and the git server• Reduce the time before Jenkins launchesbuilds
..
GPG-Sign your commitsGPG-Sign your commits
• git commit -S• Sign PR, merge commits• Do not sign every commit (then it becomesuseless)
..
Keeping multiple remotes in syncKeeping multiple remotes in sync
• git remote set-url --add• Pull & push will ose both urls
..
HomeworkHomework
• Git subtree (also built-in)• Repo (from Android)• gr• …
..
ConclusionsConclusions
• Git is an awesome, extandable tool• There are several ways to manage multiplegit repositories
• Working with multiple repos is easy• You have to learn tools anyway• Experiment them and pick one :-)
..
Thank youThank you
Any question?
..
ContactContact
Julien [email protected]@roidelapluie
INUITS bvbaBelgium+32 473 441 636https://inuits.eu