SyncTab architecture. Part 1. Server.

This post is a continuation for my previous post with introduction to SyncTab. I'm going to describe SyncTab server application in this post.

Well, I had a desire to write a server application on python, but in result I wrote it on Grails. Why Grails? Because I was actively using it for more than a year to creating different web applications, including web API. And I knew that would be simple and fast to create such app with Grails and publish some sort of web service.

So server side is running on a single Amazon EC2 instance. Application is running under Tomcat. Nginx plays a role of proxy. MongoDB is used as a database, and mongodb grails plugin used to add MongoDB.

API is very simple. It's based on GET/POST requests and JSON result with a status flag and other values, like auth token, list of tags and links.

Some code sample:

/**
* Returns the list of last available tabs.
* Client can pass a "tabId" parameter with an id of the tab to get tags for.
*
* Returns:
*  - status: "true" if was removed, otherwise "false".
*  - tabs: an array of found tabs.
*/
def getLastTabs = {
if (request.method != 'GET') {
response.sendError 405
return
}

User user = session.user
Tag tag = getTagFromRequest()
RecentTabConditions conditions = new RecentTabConditions(user, tag, LAST_TABS_NUM)
List tabs = sharedTabService.getRecentSharedTabs(conditions)

render([status: true, tabs: prepareTabs(tabs)] as JSON)
}

An authentication token system is used in SyncTab. Each time user enters his username/password to login, a new authentication token is created and assigned to user. Auth token is invalidated after two weeks.

As both Android application and Chrome extension should support not only login but also registration, it is a part of API too. Actually, user related API contains only 4 functions: register, authenticate, logout and resetPassword.

There are more API functions related to sending/receiving links and view/edit the list of tags.

A short about tags. Tags is just a way to name the device. For example, user can create a tag for his home computer and another tag for work computer, and tags for phone and tablet. Each device/browser is subscribed only to one tag, and will receive and open only links sent with specified tag.

Server not only responsible for storing and fetching links by client applications, but also for preparing links for user. Usually, when user sends a link, information about it contains only name of the device, tag id and the link URL. Server is responsible to fetch the link title, description, favicon link and the final URL. The final URL may differ from sent URL. For example, user could send a link like http://bit.ly/xNPUeo, but user should see this link as http://blog.khmelyuk.com.

Well, looks like there is nothing interesting left that I would like to tell about SyncTab server application.

Ah, source code is not public, and I'm not planning to open source it soon. As VCS I'm using Git and BitBucket as hosting service.

Rename of GitHub account and Deploy Keys

Today I've fixed failed builds that were caused by renaming GitHub account. Bamboo couldn't pull the changes from GitHub master repository after rename.

So, I corrected the account name in the .git/config file for origin remote. But this didn't help - I had same the same problem to pull the changes.

That was a security problem: GitHub denied access to the repository, because I don't have access to the old repository (of course, that because account name was changed).

After some investigation, I decided to drop existing Deploy Key and create the same again.
And Voilà! This small manipulation fixed the problem with access to the GitHub repository and now builds are green again.

Get Total Commits Number in Git

In a few weeks there will be an anniversary as I've started working on current project. So I decided to find out the total number of made commits from the project begin. While it's obvious for Subversion, in Git you need something more that just look at the revision number.

First, I came out with a simple command that prints the total number of commits:

$ git log --pretty=format:'%h' | wc -l

prints out

2485

This is a bit primitive and that wouldn't be such a simple solution if need to find the total number of commits per author. I didn't want to believe there is not better way to get the total number of commits per author with standard Git tool.

After a few minutes of research I found there is a simple and powerful util git shortlog.

So next command

$ git shortlog -n -s

prints out

  2317  Ruslan Khmelyuk
   104  Author_1
    53  Author_2
     9  unknown
     2  Author_3

I've summed the commit numbers for each committer and found that the total commits number by git shortlog command is 2485 and that is equal to the result of git log | wc -l command. While git rev-list -all | wc -l command prints there are 2488 commits in the repository. Strange.

Short investigation helped me to understand that both commands working fine and show correct number. The difference between commands

$ git log --pretty=format:'%h' | wc -l
$ git shortlog -n -s

and

$ git rev-list --all | wc-l

is in the source they're using to calculate commits list.

git log and git shortlog use current branch commits log, while git rev-list uses all repository as source of commits list. I had 3 more commits in the other branch, so in my local repository I had in total 2488 commits.

Even more, If you need to calculate the total number of commits in the remote repository, you will need to use git rev-list command:

$ git fetch origin
$ git rev-list --remotes | wc -l

Thus I can say, that git rev-list command works best to find out the total number of made commits.

Migrate from SVN repository to Git

As I was saying in my previous post it is easy to migrate you project from Subversion to Git.

Really, you will need just a few commands to run.

First of all you need to make a clone of Subversion repository using git-svn tool:

$ git svn clone --authors-file=authors.txt https://example/svn/MyRepo/trunk/project project

On this command git will go through the history of the project and fetch it into local git repository. If your subversion repository is using standard layout (i.e. trunk, branches and tags paths are present in the SVN repository root tree) than you may be interesting in --stdlayout option. You may also specify custom path for trunk or branches or tags using -T or -t or -b option respectively.

Option --authors-file contains the path to the file with committers mapping.
This file contains SVN committer mapping to git user per line.
For example:

ruslan = Ruslan <ruslan@example.com>
john = John Smith <john@smith.com>

So now you have local working copy that is managed by Git and also you have all project history in your local repository. It's easy to check what branches and tags you have using next commands:

$ cd project
$ git branches
$ git tag -l

If you have remote Git repository (maybe in GitHub), than you may want to push source code with all history to that repository. And it is not a problem, as you have already everything you need.

1. So add remote repository:

$ git remote add origin git@github.com:myname/myproject.git

2. And push to the remote repository:

$ git push origin master

3. and be happy - now you have migrated to the git repository.

Remove directory with SVN repository clone and make another clone but from git repository and continue using it:

$ cd ..
$ rm -rf project
$ mkdir project
$ cd project
$ git init
$ git remote add origin git@github.com:myname/myproject.git
$ git pull origin master


Thats how it worked for me 2 months ago.

Git and GitHub

Completely moved to Git and it's awesome. Feeling like someone unleashed my hands. Git works much faster than Subversion in everyday use. Using branches is easy and NORMAL and I'm using it day by day. No more need to have many directories with strange names and different project versions. No more fear before merging. No more failures on merging.

My opinion is that everyone must try and use some of distributed version control system, like Git or Mercurial or Bazaar. I've started using Git intensively with projects under Subversion. It was possible thanks to the git-svn project and I was working on Windows in that times. Now I'm on Linux and using git is much easier and natural. git-svn also is very useful to migrate projects from svn repository to the git repository. On checking out project you need to wait some time, because it reads the all history of repository with all changes, that is pretty long in subversion for large projects. As soon you have clone of the project svn repository, you push your changes to central git repository if any. If you need you can do commiters mapping. If you need, you can continue work with both repositories.

And what about GitHub? It's another awesome software. A lot of open source projects are using GitHub. You easily can have and use private repositories and it's pretty cheap - only $7 per month for 5 private repositories.
Each repository goes with wiki, basic issues tracker, useful reports and many others. Especially, I like Network graph, because it helps to visualize commits and merge history.

Some interesting links:
1. ProGit online book
2. Git with Subversion
3. GitCasts
3. The way GitHub helped Erlang and the way Erlang helped Github

Other related posts:
1. Git and Subversion work together

Linus about Git

Перше серйозне знайомство з Git було в мене після перегляду відео презентації Лінусом Торвальдсом, на якій він розсказував наскільки Git крутий, а майже все інше близьке до лайна. Як на мене, занадто він впевнено про це заявляв, але і мені Git сподобався тоже.

Відео було на англійській мові, але не давно знайшов переклад цього відео. Точніше, переклад слів Торвальдса.
http://lib.custis.ru/index.php/Линус_Торвальдс_о_GIT_на_Google_Talks

Git and Subversion works together

I like the idea to use Subversion central repository with a Git local "stub" repository. With having local Git repository I have access to the all history of some branch or trunk, as I have a local copy of Subversion repository. Local commits and easy branching is a big plush for me too.

Most of us are working with Subversion repositories, as they are wide used and popular nowadays. These tool gives us advantages to work with branches, setup security, transaction commits and many more. While Git proposes easy and low cost branching, decentralized storing and low coupling developers interaction. Yes, I said low coupling, as 2 developers that are working on the same thing in parallel now are able to synchronize code locally, without central repository.

I don't know what is your experience with merging branches using Subversion, but for me it's a hell. With a few different branches, that are ready to be merged into trunk but is not, because "we don't need it in trunk now, but maybe in 2 weeks we will". Sometimes merging takes the same time as new feature development.

Now Git can help us with it, as has new operation git svn.
It's currently working fine on Linux, but you should be ready to get some unpredictable behavior on Windows. Expecting that with 1.6.3 and latest versions git-svn will work much better that with Git-1.6.2.2-preview20090408.

Git Advantages

Побавився сьогодні трохи з Git. Досі я в основному працював з SVN, і тільки із централізованими системами контролю версій.
В Git знайшов кілька цікавих речей, які мені ясно сподобалися, а саме:

1. Розприділені SCM дозволяють мати локальний репозиторій, з яким працювати є набагато зручніше. Адже він
   завжди зі мною, я можу виконувати кілька незалежних комітів, які просто потім відправляти в публічний репозиторій, коли є необхідність і можливість. Розприділені репозиторії дозволять мені також синхронізуватися з іншими програмістами без необхідності внесення змін в публічний репозиторій.
   
2. Можливість створювати декілька commit lists, для того щоб комітити різні зміни із різними коментарями. Дуже корисно для підтримки порядку.
   
3. Можливість вибудовувати ієрархію репозиторіїв. Розприділення відповідальності між програмістами тепер може бути на рівні окремого репозиторію. Річ цікава і корисна, і якщо коли-не-будь доводилося працювати із репозиторіями, що знаходяться на іншому кінці світу, заховані за тучою файерволів і взагалі частенько є недоступними, тоді мене можна буде зрозуміти.
   
4. Можливість вносити зміни в історію комітів, наприклад, видалити зайвий файл або підкорегувати коментар.
   
5. Зручна та швидка робота із бренчами.
   
6. Message при коміті є обов'язковим.
   
7. Зручна робота із git в консолі.
   
8. Нема необхідності виконувати бекапи публічного репозиторію, якщо є декілька програмістів, що мають його локальну копію.
   

Це тільки декілька можливостей. Мені вони здалися найкориснішими.
Звичайно, частину з них можна добитися і від SVN, але для цього треба докласти чимало часу, а інколи і грошей.

P.S.
Знайшов гарну статтю про переваги Git над SVN: http://git.or.cz/gitwiki/GitSvnComparsion і навпаки.