Embed Size (px)
DESCRIPTION
In November 2012 I helped the giant JBoss Tools repo containing 400+ eclipse plugins, 2+ millions line of code with 7+ year of history migrate from SVN into a multitude of Git repositories. Most existing large migrations I could find info about was about the size as one of our 30+ modules in JBoss tools and it turned out the advice in most of these just did not cope with a migration as large as this. During this tale I cover what tools I examined, rejected and in the end selected to do the migration. I will outline the migration process we did as a team, what things Git enabled immediately and which things we are moving towards now that Git have opened up for a more flexible build and development model and structure. Before this move a 6+ months long preparation and even longer thinking process was started to ensure all data would be migrated properly in an efficient and repeatable manner with as little possible impact on developer productivity as possible. After this talk you should be well prepared to do both small and large migrations to git for Eclipse based plugin projects and know what dangers to look out for and what tools to consider. If you want to take a sneakpeak you can see the scripts and instructions for the migration at https://github.com/maxandersen/jbosstools-gitmigration
Citation preview
A tale about a Big SVN to Git migration
Max rydahl Andersen@maxandersen
Senior Principal Engineer, red Hat
Wednesday 27 March 13
• Do not use svn2git
• Learn about fast-filter.py
• Scripts/Resources at:
• https://github.com/maxandersen/jbosstools-gitmigration
Wednesday 27 March 13
goals for git migration
• Split up Big Repo to smaller ones
• Keep the history
• Get to use GitHub
• Make it fast and fun again!
Wednesday 27 March 13
Keeping it simplecommon plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Wednesday 27 March 13
Keeping it simple
git svn clone -s \--authors-file=authors.txt \http://svn.example.com/svn-repository/ svn-mirror
common plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Svn
git svn
svn-mirror.git
Wednesday 27 March 13
...a real projectcommon plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Wednesday 27 March 13
...a real projectcommon plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
hibernate plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
JSF plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
CDI plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
seam plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Wednesday 27 March 13
How hard can it be ?
git svn clone --prefix=svn/ -s \--authors-file=authors.txt \http://svn.example.com/svn-repository/ svn-mirror
Svn
git svn
svn-mirror.git
Wednesday 27 March 13
Check list for migration1.Create svn-mirror.git from mirrored svn
2.Cleanup Mirror
1. rename master to trunk, Delete dead branches, Checkout all branches, tag branches to real tags
3.Create individual Repositories
1. Use fast-filter.py to process content, Remove empty tags/branches, Garbage Collect
4.Clean up Garbage
1. Prune big files, Fix line endings, Garbage Collect
5. ? (Publish to Github)
6.Profit!
Wednesday 27 March 13
SVN Mirroring
Svn svn sync SvnMirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) svnsync sync file://svn-mirror http://svn.example.com/repository
Wednesday 27 March 13
SVN Mirroring
Svn svn sync SvnMirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) svnsync sync file://svn-mirror http://svn.example.com/repository
git svn
svn-mirror.git
cd svn-mirror git svn rebase --fetch-all
Wednesday 27 March 13
SVN Mirroring
Svn svn sync SvnMirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) svnsync sync file://svn-mirror http://svn.example.com/repository
git svn
svn-mirror.git
cd svn-mirror git svn rebase --fetch-all
Svn and Git now isolated and always in sync !!!
Wednesday 27 March 13
svn mirroring• Atlassian blogs has excellent material, but...
• Use --fetch-all to get full history!
• Do not remove metadata - it will help you in case of failures
Wednesday 27 March 13
Cleanup Branches/Tagssvn-mirror.git
Wednesday 27 March 13
Cleanup Branches/Tags
• Branches:
• trunk
• deadbranch
• tags/jbt-4.0.1
• tags/jbt-4.0.2
• jbt-4.0.x
• jbt-3.03
svn-mirror.git
Wednesday 27 March 13
Cleanup Branches/Tags
• Branches:
• trunk
• deadbranch
• tags/jbt-4.0.1
• tags/jbt-4.0.2
• jbt-4.0.x
• jbt-3.03
svn-mirror.git
• Branches:
• master
• jbt-4.0.x
• jbt-3.03
• Tags:
• jbt-4.0.1
• jbt-4.0.2
Wednesday 27 March 13
Cleanup branches/tagssvn-mirror.git
# Rename trunk to master $ git branch -m trunk master $ git branch -rd origin/trunk
# Delete dead branches$ git branch -a -r | grep -E "dead|broken|weird|I_WAS_DRUNK" | xargs git branch -r -D
Wednesday 27 March 13
Cleanup branches/tagssvn-mirror.git
# Convert svn tag-branches into git-tags$ git branch | grep tags | sed -e "s/tags\///g" | xargs -n 1 -I {} \ git tag -m "svn branch tag" {} tags/{}
# delete svn tag-branches$ git branch | grep tags | xargs -n 1 git branch -d
Wednesday 27 March 13
To split or not to split ?
svn-mirror.git
Wednesday 27 March 13
To split or not to split ?
svn-mirror.git
Wednesday 27 March 13
Wednesday 27 March 13
split/filter process
• /modulex
• plugina
• pluginb
• /moduley
• plugina
svn-mirror.git
Wednesday 27 March 13
split/filter process
• /modulex
• plugina
• pluginb
• /moduley
• plugina
• /modulex
• plugina
• pluginb
svn-mirror.git fastfilter
Wednesday 27 March 13
split/filter process
• /modulex
• plugina
• pluginb
• /moduley
• plugina
• /modulex
• plugina
• pluginb
• plugina
• pluginb
svn-mirror.git fastfilter filter-branch
Wednesday 27 March 13
fast-filter+filter-branch
$ python filter_repo.py jbosstools-svn-mirror jbosstools-openshift "^openshift.*"
Wednesday 27 March 13
fast-filter+filter-branch
$ python filter_repo.py jbosstools-svn-mirror jbosstools-openshift "^openshift.*"
$ git filter-branch --tag-name-filter cat --prune-empty \ --subdirectory-filter openshift -f -- --all
Wednesday 27 March 13
Clean up branches (again)
• $ deleteemptybranches.sh openshift-repo
• # delete empty commits
• $ git filter-branch --tag-name-filter cat --prune-empty -- --all
Wednesday 27 March 13
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
cleanup
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
cleanup
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
cleanup
cleanup
“it’s simple”
Wednesday 27 March 13
Check list for migration1.Create svn-mirror.git from mirrored svn
2.Cleanup Mirror
1. rename master to trunk, Delete dead branches, Checkout all branches, tag branches to real tags
3.Create individual Repositories
1. Use fast-filter.py to process content, Remove empty tags/branches, Garbage Collect
4.Clean up Garbage
1. Prune big files, Fix line endings, add .gitignore Garbage Collect
5. ?
6.Profit!
Wednesday 27 March 13
Tips’n’Tricks
Wednesday 27 March 13
Train your team• http://try.github.com
• O'Reilly Videos
• Understand the internals: https://vimeo.com/49444883
• Learn the CLI before using the UI’s
• Ban git push --force
• Green Button Considered Harmful
Wednesday 27 March 13
guardiansvs
Kings
Wednesday 27 March 13
Git Hub api is your friend
Wednesday 27 March 13
Hub“github cli”
http://defunkt.io/hub/
• $ hub clone maxandersen/autolinkjboss
• $ hub fork
• $ hub pull-request \
• “Fixed this bad presentation”
Wednesday 27 March 13
Questions ?https://github.com/maxandersen/jbosstools-gitmigration
Wednesday 27 March 13
