C2SM SVN to Git migration · Provides output ﬁle migration.out for troubleshooting Will generate a git repository locally and push all the commits to your fork on Github Will also

C2SM SVN to Git migration

Katherine Osterried and Carlos Osuna, C2SM (ETH)

November 2, 2015

1 / 44

Git -> Which Git?

After evaluation phase: GitHub was selected for hosting the new C2SMrepositories

C2SMCenter for Climate Systems Modeling

github.com/C2SM-RCM/

is our central GitHub organization

All the code in the cosmo.cscs.ch and hpcforge SVN repositories willbe migrated to GitHubWe have designed a workflow for using GitHub for our C2SM-RCMusers

2 / 44

Part I: Structure of Git Repositories

3 / 44

Central repositories on Github

Central repositories are located at: github.com/C2SM-RCM

Users need to email [email protected] and sign the C2SMlicense agreement to gain access to the repositories

Most of the repositories are private and need to stay that way!

The git repositories will contain:

the full history of the development from the SVN repositories

the master branch and trunk tags only (no development branches!)

the vendor codes, which contain the code directly from the code owner

4 / 44

Mapping the SVN repos to git

The structure of the two SVN repositories has been analyzed andmapped to approximately 30 git repositories

The names of the git repositories are lowercase and correspond to thecode within

The full mapping of the SVN repositories to git repositories can befound at the end of the migration document

Example for the cosmo.cscs.ch repository:

Directory Sub-directories Sub-sub-directories Git repository

cosmo trunk cclm_c2sm cclmcosmo_c2sm cosmo

vendor cclm cclm-vendorcosmo cosmo-vendor

5 / 44

Permissions

There will be three different levels of permissions for accessing theC2SM-RCM repositories:

1 OwnersTwo or three people only ([email protected])Have complete control over the C2SM-RCM organizationCan create or delete repositories, add users, and write to everyrepository

2 AdminsOne or two people for each repository (admin-codename)Have write access to their assigned repositoryAdd new versions of code and incorporate new features and bug fixes

3 UsersEveryone who is not an owner or adminHave read access to all of the central repositoriesMust sign a license agreement before they are granted access

6 / 44

Part II: GitHub Workflow

7 / 44

Git Terminology

1 fork : personal copy of a repository on Github

2 branch : an independent line of development

3 master : the default branch

4 commit : a snapshot of your project at a certain time

5 remote : a repository linked to the local repository

6 pull request : request on web interface to merge code into a centrallocation

8 / 44

Start Using github.com/C2SM-RCM

github.com/C2SM-RCM/cosmo

Is the central COSMO repository, everyone in the organization canclone it!!

#git clone [email protected]:C2SM�RCM/cosmo.git

You candevelop localcommit (local action)

However the central github.com:C2SM-RCM

Will not contain branchesMaster does not allow you to push your commits.

So how do I develop my code?

9 / 44

Use Case I: User Develops NewFunctionality

1 Create a fork of the parent repository through github web.

2 Clone the repository into your workspace:

#git clone [email protected]:/cosunae/fieldextra.git

10 / 44

[email protected]:C2SM-

RCM/

fieldextra.git

[email protected]:cosuna/

fieldextra.git

Use Case I: User Develops NewFunctionality

3 Create a branch into your personal fork for development:#git branch �a <feature>

4 Develop:#git commit .

5 push repository into personal fork in order to save work#git push origin <feature>

C2SM-RCM/

cosmo.git

cosunae/cosmo.git

11 / 44

Use Case I: User develops newfunctionality

Your fork is your fork. You are free to choose your working flow.

Some Good Practices:

Develop local and perform (at least) daily push to save your work.Develop in branches.

Your code will be reviewed if you ever want to merge your developments inthe central repository, follow stricter policies:

1 Functionality <==> 1 Branch. Avoid building "monster" brancheslike "PhD_branch".Name of branch should resemble the functionality.Keep a clean history and meaningful commit messages. Avoidcommits like "apply fix".Git convention: first line of the commit message should be a summary.

12 / 44

Use Case II: Update your fork fromupstream repository

C2SM-RCM/

cosmo.git

cosunae/cosmo.git

1 Add the upstream repository as a remote#git remote add upstream https://github.com/C2SM�RCM/cosmo

2 Update the cloned code of the upstream#git fetch upstream

3 merge the upstream branch/commit/tag with your fork:#git merge <branch>

13 / 44

Use Case I/II

Demonstration of Use Case I/II

14 / 44

Use Case III: Collaborative Development

We will develop projects in different github organizations

Choose a workflow at the beginning of the project.Do not apply direct merges/push into master, always code reviewusing Pull Requests

15 / 44

Create/Manage Sub-Organizations

Project forks will be hosted in Sub-Organizations,

Your project manages this organization. You can decide

who has admin rightswho has write access (accept pull requests)Do not forget to protect your master

16 / 44

Communication Between Forks

How do we synchronize/communicate the different developments beingcarried out in the multiple forks??

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git

cosunae/cosmo.gitminusinf/

cosmo.git

IAC/cosmo.git

kosterreid/cosmo.git

17 / 44


Assume some development was done in cosunae/cosmo.git

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


18 / 44


Pull Request into parent fork...

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


P

u

l

l

R

e

q

u

e

s

t

19 / 44

Pull Request

Use for

Merge developments into a different branch or fork/branch.

Make sure the code passes the quality metrics of the repository (tests,documentation, etc.)Create a Pull Request from your branch into C2SM-RCM/<repo>Fix in your branches recommendations from reviewersGroup <repo>_admins group will do the merge when approved.

20 / 44


Other children forks from MCH-APN can merge from upstream repositoriesin order to update their repositories...

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


f

e

t

c

h

&

m

e

r

g

e

21 / 44


Technically possible, but not recommended

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


P

u

l

l

R

e

q

u

e

s

t

22 / 44


Instead... Pull Request to central repository will make the developmentsvisible to whole organization.

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


P

u

l

l

R

e

q

u

e

s

t

23 / 44


After any merge into master at C2SM-RCM, any fork can merge (updateits fork) from it.

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


f

e

t

c

h

&

m

e

r

g

e

f

e

t

c

h

&

m

e

r

g

e

24 / 44

Issues

Publisher Mode

You can use Issues as a tracking system.Easy way to organize your tasks and link them to the code.

25 / 44

Issues

Publisher Mode

But Issues are also very efficient way to communicate with children forks inthe organization.

Use labels for different categories:Report on open/close bugs: status, use cases affected...Report ongoing developments functionalities.Follow developers assigned to an issue allows direct contact for furtherinfo

bug new feature question

26 / 44

Issues

Leave a message in Issues of your parents

Issues and PR get an ID, for example ]158Reference Issue ID in git commits will be reflected in github Issuespagegit commit -m"This is a fix for Issue ]158" .https://help.github.com/articles/closing-issues-via-commit-messages/

27 / 44


Cherry Picking

git cherry-pick <commit>

A B C D E

F G D’

Head

git cherry-pick D

28 / 44

Cherry Picking Other Forks

We can cherry pick from any other fork.A commit is a unique hash, no need to specify the remote (fork)

C2SM-RCM/

cosmo.git

MCH-APN/cosmo.git


cosmo.git

IAC/cosmo.git


g

i

t

c

h

e

r

r

y

-

p

i

c

k

D

29 / 44

Policies

For central repository admins

Do not create branches in C2SM-RCM/<repo>

For project fork admins

Delete branches that are no longer usedAlways delete the fork once work is finished or merged into parentrepositoryAlways protect the master branch against direct push

For user forks

Do not invite external collaborators to your fork that did not sign theC2SM agreement.

30 / 44

Public Repositories

Public Repositories on C2SM-RCM have certain advantages

public repositories do not increase the costadditional tools offered for free for public repositories

Go Public!!

In C2SM-RCM organization if you can

31 / 44

Collaborative development

Demonstration

32 / 44

Part III: Migration Plan

33 / 44

Migration details

The git migration will take place in December

At the time of the migration, the SVN repositories will be frozen andshould not be used anymore

The central repositories will be created and placed atgithub.com/C2SM-RCM

As soon as the central repositories are ready, users will be notified byemail

The SVN repositories will be kept intact in a read-only state

Users must migrate their own branches

34 / 44

Migrating your personal branches

1 Sign up for an account on Github (github.com). Gain access to theC2SM-RCM organization by emailing [email protected] yourGithub user name.

35 / 44


2 Fork the desired code into your personal webspace. This is done bynavigating to the desired repository and clicking the Fork button.

36 / 44


3 Download the migration materials by going togithub.com/C2SM-RCM/migration and clicking the DOWNLOAD ZIPbutton. Save and unpack the zip file where you would like to developyour code.

!! Be sure to download the migration materials instead of cloning them asa git repository. This will ensure that you do not generate nested git

repositories !!37 / 44

About the migration script

4 Run the migration script, following the instructions in the READMEfile and the migration document.

c2sm_svn2git.sh: bash script to migrate user’s branchesScript requires Ruby gem svn2git- installed on all machines at IAC andon KeschAlso an authors file which should be placed in same location as thec2sm_svn2git fileREADME file with detailed instructionsUser must fill in 4 variables at beginning of script

38 / 44

Preparing the migration script

User must define 4 variables in the USER INPUT section of the script:

1 REPO:name of the git repository for which you would like to migrate branches.Example: for the cosmo repository, this line should read:REPO="cosmo"

2 BRANCHES:Name of the branches that should be migratedSame name as the folder in the SVN repository containing the branchFor one branch, use BRANCHES=(feature1), two branches useBRANCHES=(feature1 feature2)Example: To migrate the branch located athttps://cosmo.cscs.ch/cosmo/branch/iac/ksilver/calibrationBRANCHES=(calibration)

39 / 44

Preparing the migration script

User must define 4 variables in the USER INPUT section of the script:

3 USERYour user name for the SVN repositoryExample: USER="ksilver"

4 FORKPath to your fork of the code on GithubIf using https protocol, should include user name in addressExample (https protocol):FORK="https://[email protected]/kosterried/cosmo"Example (ssh protocol): FORK="[email protected]:kosterried/cosmo"

40 / 44

Running the migration script

Shell script should be run on the development computer where youwould like the git repository

Simply run as any other shell script: ./c2sm_svn2git.sh fromcommand line

May take between several minutes up to a couple of hours to run

Provides output file migration.out for troubleshooting

Will generate a git repository locally and push all the commits to yourfork on Github

Will also synchronize your local repository with the fork if there havebeen commits in the central repositories

May prompt you for your SVN password and Github username andpassword. Be careful which is which.

41 / 44

Part IV: Help and resources

42 / 44

Git resources

For help with:1 git

git help: displays general information about git and man pages forspecific commandshttp://git-scm.com/doc : Comprehensive git reference manualhttp://gitref.org: Quick reference of commonly used git commands

2 Githubhttps://help.github.com: Help pages for Githubhttps://guides.github.com: Step by step guides to Github features

3 C2SM-RCM repositories : For help with the C2SM-RCM repositoriesplease contact the organization owners at [email protected]

4 The migration : For help with the git migration contact either KatieOsterried ([email protected]) or Carlos Osuna([email protected])

43 / 44

When Things STILL Go Wrong With Git

44 / 44

Documents

C2SM SVN to Git migration · Provides output ﬁle migration.out for troubleshooting Will generate a git repository locally and push all the commits to your fork on Github Will also