Migrating repositories from Gitblit to Stash

While we loved using Gitblit the last few years, we wanted a more integrated toolset.

Wildcard decided to move a whole lot of repositories from Gitblit into Stash – almost 200. We wanted to do this in a single quick conversion, rather than trying to run two systems and slowly migrating repos.

A little bit of bash, curl, and the Stash REST API made the whole process take about 12 hours in all – including development, testing, execution, and then cleanup (e.g., making sure team members were up to speed and including documentation fixes).

First, I'd recommend taking a gander at the Stash API documentation; it has a lot of useful information and can manipulate most aspects of the system!

It can, for example, automatically create a project:

curl -u api_user:pass -X POST -H "Content-Type: application/json" -d @project.json "https://example.org/rest/api/1.0/projects"

where `@project.json` is an `@` followed by a path to a file that contains JSON formatted data for creating the project:

{
 "key": "EX",
 "name": "Example Project",
 "description": "just an example project"
 }

It can also automatically create a repo for a project:

curl -u api_user:pass -X POST -H "Content-Type: application/json" -d @repo.json "https://example.org/rest/api/1.0/projects/EX/repos"

Like the one for creating a project, here's the `repo.json` file:

{
 "name": "example repo",
 "scmId": "git"
}

Our overall process was to create a new project and repository in Stash (since the organization in our old system was largely incompatible with Stash), clone all branches and tags to a local copy, reset the remote 'origin' url on each repo, then push the repo with all of its tags and branches up to its new location in Stash.

The structure we decided to go with in Stash is fairly light on the "projects" Stash defines, so we just created those by hand. Our number of projects, however, is much larger and we needed some automation.

Combining a little bash with some API calls, we came up with a little script similar to the following:

#! /usr/env/bin bash
# user credentials for calling into the api
username=api_user
pass=pass
# loop through each line
while IFS='' read -r line || [[ -n "$line" ]]; do

    # each line is a list of values split by ';'
    IFS=';' read -ra parts <<< "$line"

    # store each value in a line in a more readable variable name
    projkey=${parts[0]}
    reponame=${parts[1]}

    # print some info about what's being done currently
    echo $projkey/$reposlug

    # create the repository
    curl -X POST -u $username:$pass -H "Content-Type: application/json" https://example.org/rest/keys/1.0/projects/$projkey/repos -d '{"name":"'$reponame'","scmId":"git"}'

    # pipe the file passed as the first parameter into the while loop's `read`
done < "$1"

You'd use the script something like this:

$ generate_repos.sh repolist.txt

And `repolist.txt` would look something like:

 EX;project1
 EX;project2
 EX;project3
 EX;project4

(it contains project key - project name mappings)

Then, we had a script run through the process of cloning our repos (including all branches and tags), and resetting the origin URLs:

#!/usr/bin/env bash
while IFS='' read -r line || [[ -n "$line" ]]; do
    IFS=';' read -ra parts <<< "$line"
    name=${parts[0]}
    oldurl=${parts[1]}
    newurl=${parts[2]}

    echo "changing ${oldurl}"
    cd /path/to/temporary/repository/storage/base/

    # get a clone of the repository, including branches and tags
    git clone $oldurl ./$name
    cd ./$name
    git branch -r | grep -v HEAD | grep –v master | awk -F'origin/' '{print $2 " " $1"origin/"$2}' | xargs -L 1 git branch -f --track 
    git fetch --all
    git pull --all

    # change the origin to the new url
    git remote set-url origin $newurl

    # push the repo into the new location (including branches and tags
    git push --all
    git push --tags
done < "$1"

The script would be called like:

$ clone_repos.sh repolist.txt

And `repolist.txt` would look something like:

project1;ssh://old.example.org/path/to/project1.git;ssh://new.example.org/EX/project1.git
project2;ssh://old.example.org/path/to/project2.git;ssh://new.example.org/EX/project2.git
project3;ssh://old.example.org/path/to/project3.git;ssh://new.example.org/EX/project3.git
project4;ssh://old.example.org/path/to/project4.git;ssh://new.example.org/EX/project4.git

And then it's a matter of going through the code base and deployment configuration to find any references to your old repositories and make sure they all work in the way you expect!

While we loved using Gitblit the last few years, we wanted a more integrated toolset.