Update 2: This page incorrectly stated that this patch was included in git 1.7.0.1. It’s not! It’s in the git repository’s “next” branch – which will eventually become the git 1.7.2 series. I’ll update again when this is released.

Some people have noticed that using git svn (in my case msysgit) on Windows VM’s¹ can be incredibly slow. I’m one of those people.

Running a no-op rebase (i.e. there are no changes @ the svn end) looks like this.²

$ time perl git svn rebase
Current branch master is up to date.

real 2m56.750s
user 0m3.129s
sys 2m39.232s

Nearly 3 minutes to verify that my local git svn branch is up to date!

I used to (_many_ years ago) – do quite a bit of perl for for sysadmin stuff. I’ve already paid for those sins but this was slow enough that I wanted to see what was causing the problems.

Running perl -d:DProf and collecting a basic trace gave me some interesting info.³ SVN::Base::import (essentially a constructor) was importing all the svn dll entry points (using DynaLoader) every time git svn started up – which was extremely slow in my environment.

On a hunch⁴ I moved the code which was require’ing SVN::Base::import to run lazily (rather than at git-svn startup).

$ time perl /libexec/git-core/git-svn rebase
Current branch master is up to date.

real 0m33.407s
user 0m1.409s
sys 0m23.054s

6 times faster!

“git svn dcommit” sped up even more. Previously:

$ time perl /libexec/git-core/git-svn.orig dcommit -n
Committing to svn://XXXXX/trunk ...
diff-tree befd87ba1b65fa7d86779c3058faf1f4886d7020~1 befd87ba1b65fa7d86779c3058faf1f4886d7020

real 3m7.046s
user 0m3.190s
sys 2m53.897s

Afterwards:

$ time perl /libexec/git-core/git-svn dcommit -n
Committing to svn://XXXXX/trunk ...
diff-tree befd87ba1b65fa7d86779c3058faf1f4886d7020~1 befd87ba1b65fa7d86779c3058faf1f4886d7020

real 0m10.312s
user 0m1.109s
sys 0m4.632s

19 times faster!

For me – this was enough for now. I’d spent more time on the issue than I had – and git svn operations were now fast enough to not totally break my workflow.

I’ve uploaded the patch here: git-svn.patch

YMMV/No warranty expressed or implied/IANAL etc. If it breaks your repo/application/life – don’t come crying to me.

Thanks to Nic Wise for testing this for me and validating my timing data etc.

1. VMWare Fusion WinXP vm under OSX 10.5. This is probably important as people not using git svn on a VM don’t seem to sufffer such huge perf problems.
2. [1] My svn server is 350ms away – Auckland<->London and on a VPN – I’m using the SVN protocol to access it – SVN seems fastest of the SVN protocols.
3. Total Elapsed Time = 140.4030 Seconds
User+System Time = 3.919008 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
77.7 3.046 3.046 12 0.2538 0.2538 DynaLoader::dl_load_file
9.16 0.359 0.359 1 0.3590 0.3590 main::read_repo_config
6.38 0.250 0.250 18 0.0139 0.0139 Git::_command_common_pipe
1.22 0.048 3.063 40 0.0012 0.0766 SVN::Base::import
1.20 0.047 0.156 22 0.0021 0.0071 main::BEGIN
0.41 0.016 0.016 4 0.0040 0.0040 Exporter::as_heavy
0.41 0.016 0.032 7 0.0023 0.0046 IO::File::BEGIN
0.41 0.016 0.016 11 0.0015 0.0015 Git::SVN::BEGIN
0.41 0.016 0.016 7 0.0023 0.0023 Git::SVN::rev_map_max_norebuild
0.41 0.016 0.016 7 0.0023 0.0023 SVN::Git::Fetcher::BEGIN
0.38 0.015 0.015 1 0.0150 0.0150 warnings::BEGIN
0.38 0.015 0.015 7 0.0021 0.0021 Git::BEGIN
0.38 0.015 0.015 13 0.0012 0.0011 Error::subs::try
0.00 0.000 0.000 1 0.0000 0.0000 XSLoader::bootstrap_inherit
0.00 0.000 0.000 1 0.0000 0.0000 Exporter::Heavy::heavy_export_tags

   
4. git-svn shell’s out to itself a bunch so it seemed an obvious first step.

For the first time svn has built in support for repeated merges without spurious conflicts or manually managing the list of previously merged revisions.

This is _fantastic_ news. The basis of the merge tracking feature in 1.5 was svnmerge.py which means that many svn users will already have a reasonable idea how it works. (List of merged revisions are stored in svn properties on a branch). 

I’ve been watching the development of svn 1.5 for the last 18 months (give or take) crying everyday that I have to manually manage merging work as I oggle the feature list. 

TortoiseSVN has a significantly enhanced release as well. I’m particularly excited about this because it supports caching the change log which will make a huge difference for me given that my svn server it 2 continents and 300ms away.

It also looks like there may be some progress on cleaning up the working copy implementation.¹. Which would involve fixing the whole every folder with .svn folders problem (a la git or svk). 

1. http://subversion.tigris.org/roadmap.html

Svnmerge.py is a tool which tries to fix the _totally broken_ “svn merge” command.  Here’s an example scenario. My repository contains a number of branches of the same product:

/trunk – the main line of development. New features are added here.
/branches/Customer1 – We made a release to a customer – this is the version that was released.
/branches/Customer2 – Another release for a different customer.

/trunk, /Customer1 and /Customer2 all contain bug fixes fixing critical bugs which were discovered in production by different customers. In an ideal world we would be able to get our SCM to list all the changes to Customer1 and choose which ones to apply to /trunk and Customer2. Subversion lets you do this using the merge command.

svn merge /branches/Customer1 /trunk

What this does is take all the changes to Customer1 which have been made since the branching of Customer1 from trunk and apply them to trunk. The first time this works great. All the changes get made to my working copy and I can go through an commit them and every things groovy.

The next day I have (another) critical bug for Customer1. I fix it – and commit my change to the Customer1 branch. Then I need to merge again.

svn merge /branches/Customer1 /trunk

What this does is take all the changes to Customer1 which have been made since the branching of Customer1 from trunk and apply them to trunk. Can you see the problem here? If you can’t let me explain. It will attempt to apply the same changes it applied previously, this will generate conflicts, potentially _lots_ of conflicts. They will need to be manually resolved and then just the new (bug fix) changes committed.

Until next time when svn merge will – take all the changes to Customer1 which have been made since the branching of Customer1 from trunk and apply them to trunk. ARRG. See what I meant by brain dead?

It actually gets worse. I also want to merge bug fixes from /Customer2 -> /trunk and from Customer1 -> Customer2 and Customer2 -> Customer1. Not to mention the bugs you find while doing development and fix in the trunk that should be merged back to the customer branches. /trunk -> /Customer1 and /trunk -> /Customer2.  This is a boat load of manually fixing up conflicts that needs to happen. Every time. It’s almost easier to manually change the necessary files for each branch.

In all honnesty – I wouldn’t try to merge bug fixes from the trunk to branches – if I found a bug in dev – I’d chose a customer branch to fix it in and merge it into the trunk from there. Anything else would be too painful.

How does svnmerge.py help? What svnmerge.py does is store some metadata in your repository (actually – per branch – which is part of the problem) about which revisions you’ve previously merged. This way – when you use svnmerge – it will only apply changes to your working copy that have not been previously merged.

Yay – problem solved right? Unfortunately – no. There’s a problem. If you only ever have two branches – then this will work pretty well. If you have more than two then your stuffed. e.g. /trunk /branches/Customer1 /branches/Customer2 = 3 = bad news.

The reason is that  svnmerge tracks merges per branch – it doesn’t support what the svnmerge guys call graph based merges. A graph based merge looks like the following:

e.g. /Customer1 <->  /Customer2 <-> /trunk <-> /Customer1 (this scenario is described above when merging bugfixes from the trunk to branches).

You have to setup bidirectional merging on each branch.

/branches/Customer1 <-> /trunk
/branches/Customer2 <-> /trunk
/branches/Customer2 <-> /branches/Customer1

If I merge my fixes from /Customer1 -> trunk then /Customer2 knows nothing about it. When I try and merge all three branches I end up getting conflicts. The problem is that the history is lost when you merge a bugfix Customer1 -> trunk -> Customer2 – svnmerge only knows change in Customer2 came from the trunk – not from Customer1. If that change was previously merged Customer1->Customer2 (critical bug for example) then you’ll get conflicts again.

Git is a distributed SCM system – Linus wrote it in two weeks. It seems like it’s getting some traction among some of the Rails crowd. Mozilla seriously considered adopting it for Mozilla 2 work but the lack of a finished win32 client was a show stopper (although progress seems to have been made there).

There’s nothing like Tortoise for it (yet) – but it’s still pretty new – it’s been used by the kernel developers for the last two years – but it seems like it’s only in the last few months that it’s got easy enough for newbies to pickup.

Distributed SCM’s really take some brain twisting to get your head around – and Git in particular breaks with most conventions that people used to traditional centralised, file based systems are used to. Linus says that he thinks the only reason he’s been able to build git was because he didn’t have brain rot caused by exposure to traditional scm tools.