A tale about a Big SVN to Git migration
Max rydahl Andersen@maxandersen
Senior Principal Engineer, red Hat
Wednesday 27 March 13
• Do not use svn2git
• Learn about fast-filter.py
• Scripts/Resources at:
• https://github.com/maxandersen/jbosstools-gitmigration
Wednesday 27 March 13
goals for git migration
• Split up Big Repo to smaller ones
• Keep the history
• Get to use GitHub
• Make it fast and fun again!
Wednesday 27 March 13
Keeping it simplecommon plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Wednesday 27 March 13
Keeping it simple
git svn clone -s \--authors-file=authors.txt \http://svn.example.com/svn-repository/ svn-mirror
common plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Svn
git svn
svn-mirror.git
Wednesday 27 March 13
...a real projectcommon plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Wednesday 27 March 13
...a real projectcommon plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
hibernate plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
JSF plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
CDI plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
... plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
seam plugins .core
.ui
...
features .feature
.source.feature
...
tests .tests
...
site
Wednesday 27 March 13
How hard can it be ?
git svn clone --prefix=svn/ -s \--authors-file=authors.txt \http://svn.example.com/svn-repository/ svn-mirror
Svn
git svn
svn-mirror.git
Wednesday 27 March 13
Check list for migration1.Create svn-mirror.git from mirrored svn
2.Cleanup Mirror
1. rename master to trunk, Delete dead branches, Checkout all branches, tag branches to real tags
3.Create individual Repositories
1. Use fast-filter.py to process content, Remove empty tags/branches, Garbage Collect
4.Clean up Garbage
1. Prune big files, Fix line endings, Garbage Collect
5. ? (Publish to Github)
6.Profit!
Wednesday 27 March 13
SVN Mirroring
Svn svn sync SvnMirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) svnsync sync file://svn-mirror http://svn.example.com/repository
Wednesday 27 March 13
SVN Mirroring
Svn svn sync SvnMirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) svnsync sync file://svn-mirror http://svn.example.com/repository
git svn
svn-mirror.git
cd svn-mirror git svn rebase --fetch-all
Wednesday 27 March 13
SVN Mirroring
Svn svn sync SvnMirror
svnsync init file://svn-mirror http://svn.example.com/repository
while(true) svnsync sync file://svn-mirror http://svn.example.com/repository
git svn
svn-mirror.git
cd svn-mirror git svn rebase --fetch-all
Svn and Git now isolated and always in sync !!!
Wednesday 27 March 13
svn mirroring• Atlassian blogs has excellent material, but...
• Use --fetch-all to get full history!
• Do not remove metadata - it will help you in case of failures
Wednesday 27 March 13
Cleanup Branches/Tagssvn-mirror.git
Wednesday 27 March 13
Cleanup Branches/Tags
• Branches:
• trunk
• deadbranch
• tags/jbt-4.0.1
• tags/jbt-4.0.2
• jbt-4.0.x
• jbt-3.03
svn-mirror.git
Wednesday 27 March 13
Cleanup Branches/Tags
• Branches:
• trunk
• deadbranch
• tags/jbt-4.0.1
• tags/jbt-4.0.2
• jbt-4.0.x
• jbt-3.03
svn-mirror.git
• Branches:
• master
• jbt-4.0.x
• jbt-3.03
• Tags:
• jbt-4.0.1
• jbt-4.0.2
Wednesday 27 March 13
Cleanup branches/tagssvn-mirror.git
# Rename trunk to master $ git branch -m trunk master $ git branch -rd origin/trunk
# Delete dead branches$ git branch -a -r | grep -E "dead|broken|weird|I_WAS_DRUNK" | xargs git branch -r -D
Wednesday 27 March 13
Cleanup branches/tagssvn-mirror.git
# Convert svn tag-branches into git-tags$ git branch | grep tags | sed -e "s/tags\///g" | xargs -n 1 -I {} \ git tag -m "svn branch tag" {} tags/{}
# delete svn tag-branches$ git branch | grep tags | xargs -n 1 git branch -d
Wednesday 27 March 13
To split or not to split ?
svn-mirror.git
Wednesday 27 March 13
To split or not to split ?
svn-mirror.git
Wednesday 27 March 13
Wednesday 27 March 13
split/filter process
• /modulex
• plugina
• pluginb
• /moduley
• plugina
svn-mirror.git
Wednesday 27 March 13
split/filter process
• /modulex
• plugina
• pluginb
• /moduley
• plugina
• /modulex
• plugina
• pluginb
svn-mirror.git fastfilter
Wednesday 27 March 13
split/filter process
• /modulex
• plugina
• pluginb
• /moduley
• plugina
• /modulex
• plugina
• pluginb
• plugina
• pluginb
svn-mirror.git fastfilter filter-branch
Wednesday 27 March 13
fast-filter+filter-branch
$ python filter_repo.py jbosstools-svn-mirror jbosstools-openshift "^openshift.*"
Wednesday 27 March 13
fast-filter+filter-branch
$ python filter_repo.py jbosstools-svn-mirror jbosstools-openshift "^openshift.*"
$ git filter-branch --tag-name-filter cat --prune-empty \ --subdirectory-filter openshift -f -- --all
Wednesday 27 March 13
Clean up branches (again)
• $ deleteemptybranches.sh openshift-repo
• # delete empty commits
• $ git filter-branch --tag-name-filter cat --prune-empty -- --all
Wednesday 27 March 13
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
cleanup
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
cleanup
“it’s simple”
Wednesday 27 March 13
Svn svn sync SvnMirror
git svn
svn-mirror.git
cleanup
cleanup
“it’s simple”
Wednesday 27 March 13
Check list for migration1.Create svn-mirror.git from mirrored svn
2.Cleanup Mirror
1. rename master to trunk, Delete dead branches, Checkout all branches, tag branches to real tags
3.Create individual Repositories
1. Use fast-filter.py to process content, Remove empty tags/branches, Garbage Collect
4.Clean up Garbage
1. Prune big files, Fix line endings, add .gitignore Garbage Collect
5. ?
6.Profit!
Wednesday 27 March 13
Tips’n’Tricks
Wednesday 27 March 13
Train your team• http://try.github.com
• O'Reilly Videos
• Understand the internals: https://vimeo.com/49444883
• Learn the CLI before using the UI’s
• Ban git push --force
• Green Button Considered Harmful
Wednesday 27 March 13
guardiansvs
Kings
Wednesday 27 March 13
Git Hub api is your friend
Wednesday 27 March 13
Hub“github cli”
http://defunkt.io/hub/
• $ hub clone maxandersen/autolinkjboss
• $ hub fork
• $ hub pull-request \
• “Fixed this bad presentation”
Wednesday 27 March 13
Questions ?https://github.com/maxandersen/jbosstools-gitmigration
Wednesday 27 March 13