thinking in geek tagline’s are so web2.0

Speeding up git svn on Windows (win32)

Update: This fix is now included in git releases 1.7.0.1 and greater – for more info – click here.

Update 2: This page incorrectly stated that this patch was included in git 1.7.0.1. It’s not! It’s in the git repository’s “next” branch – which will eventually become the git 1.7.2 series. I’ll update again when this is released.

Some people have noticed that using git svn (in my case msysgit) on Windows VM’s¹ can be incredibly slow. I’m one of those people.

Running a no-op rebase (i.e. there are no changes @ the svn end) looks like this.²

$ time perl git svn rebase
Current branch master is up to date.

real 2m56.750s
user 0m3.129s
sys 2m39.232s

Nearly 3 minutes to verify that my local git svn branch is up to date!

I used to (_many_ years ago) – do quite a bit of perl for for sysadmin stuff. I’ve already paid for those sins but this was slow enough that I wanted to see what was causing the problems.

Running perl -d:DProf and collecting a basic trace gave me some interesting info.³ SVN::Base::import (essentially a constructor) was importing all the svn dll entry points (using DynaLoader) every time git svn started up – which was extremely slow in my environment.

On a hunch⁴ I moved the code which was require’ing SVN::Base::import to run lazily (rather than at git-svn startup).

$ time perl /libexec/git-core/git-svn rebase
Current branch master is up to date.

real 0m33.407s
user 0m1.409s
sys 0m23.054s

6 times faster!

“git svn dcommit” sped up even more. Previously:

$ time perl /libexec/git-core/git-svn.orig dcommit -n
Committing to svn://XXXXX/trunk ...
diff-tree befd87ba1b65fa7d86779c3058faf1f4886d7020~1 befd87ba1b65fa7d86779c3058faf1f4886d7020

real 3m7.046s
user 0m3.190s
sys 2m53.897s

Afterwards:

$ time perl /libexec/git-core/git-svn dcommit -n
Committing to svn://XXXXX/trunk ...
diff-tree befd87ba1b65fa7d86779c3058faf1f4886d7020~1 befd87ba1b65fa7d86779c3058faf1f4886d7020

real 0m10.312s
user 0m1.109s
sys 0m4.632s

19 times faster!

For me – this was enough for now. I’d spent more time on the issue than I had – and git svn operations were now fast enough to not totally break my workflow.

I’ve uploaded the patch here: git-svn.patch

YMMV/No warranty expressed or implied/IANAL etc. If it breaks your repo/application/life – don’t come crying to me.

Thanks to Nic Wise for testing this for me and validating my timing data etc.

1. VMWare Fusion WinXP vm under OSX 10.5. This is probably important as people not using git svn on a VM don’t seem to sufffer such huge perf problems. [↩]
2. [1] My svn server is 350ms away – Auckland<->London and on a VPN – I’m using the SVN protocol to access it – SVN seems fastest of the SVN protocols. [↩]
3. Total Elapsed Time = 140.4030 Seconds
User+System Time = 3.919008 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
77.7 3.046 3.046 12 0.2538 0.2538 DynaLoader::dl_load_file
9.16 0.359 0.359 1 0.3590 0.3590 main::read_repo_config
6.38 0.250 0.250 18 0.0139 0.0139 Git::_command_common_pipe
1.22 0.048 3.063 40 0.0012 0.0766 SVN::Base::import
1.20 0.047 0.156 22 0.0021 0.0071 main::BEGIN
0.41 0.016 0.016 4 0.0040 0.0040 Exporter::as_heavy
0.41 0.016 0.032 7 0.0023 0.0046 IO::File::BEGIN
0.41 0.016 0.016 11 0.0015 0.0015 Git::SVN::BEGIN
0.41 0.016 0.016 7 0.0023 0.0023 Git::SVN::rev_map_max_norebuild
0.41 0.016 0.016 7 0.0023 0.0023 SVN::Git::Fetcher::BEGIN
0.38 0.015 0.015 1 0.0150 0.0150 warnings::BEGIN
0.38 0.015 0.015 7 0.0021 0.0021 Git::BEGIN
0.38 0.015 0.015 13 0.0012 0.0011 Error::subs::try
0.00 0.000 0.000 1 0.0000 0.0000 XSLoader::bootstrap_inherit
0.00 0.000 0.000 1 0.0000 0.0000 Exporter::Heavy::heavy_export_tags

   [↩]
4. git-svn shell’s out to itself a bunch so it seemed an obvious first step. [↩]

Hi,

don’t you think it is a bit unclever to “publish” this on a blog next to nobody frequents who is interested in the issue?

I mean, you can just fire off a mail to git@vger.kernel.org and be done with it. (Of course, you should send a mail with a complete commit message, but that cannot take too much time.)

It would be a shame if that work would go to waste just because you chose a poor way to tell people about it.

Ciao,
Dscho

This is basically a thinking exercise for me to:

a) help me clearly specify what I’ve done
b) get some feedback from a couple of other people who’ve been working in similar situations.

Stand by for the patch.

Your post is totally appreciated. Git-svn is so slow on Windows + VM that it is practically unusable for me.

Hello.

Nice to see this patch, msysgit is horribly slow on my Windows XP box.

Though, I have installed msysgit 1.7.0.2:
$ git –version
git version 1.7.0.2.msysgit.0

But your patch is not in there:
$ grep -A 5 “sub fatal” /libexec/git-core/git-svn
sub fatal (@) { print STDERR “@_\n”; exit 1 }
require SVN::Core; # use()-ing this causes segfaults for me… *shrug*
require SVN::Ra;
require SVN::Delta;
if ($SVN::Core::VERSION lt ’1.1.0′) {
fatal “Need SVN::Core 1.1.0 or better (got $SVN::Core::VERSION)”;

Isn’t that strange?

I added the patch manually and it seems to work.

What are these guys at msysgit doing?

OK, well, from what I can see browsing the git-core history, this patch has not been incorporated before *1.7.1*!

So, not yet available with msysgit
(And we might wait for long being the extremely light roadmap of msysgit.)

Yep – sorry – don’t know what I was smoking when I said that. Looks like it’s scheduled for 1.7.2 (it’s in the “next” branch).