MustEat.org

Subversion Performance Tuning

Thu, 29 Oct 2009 23:41:56 -0000

This guide is my notes on keeping several subversion repositories happy that are currently growing by 1k-2k commits per month.

My current approach is based on Ubuntu 8.04 LTS, Apache 2.2 and Subversion 1.5.

I have considered upgrading to Subversion 1.6 for further speed improvements and features, but so far that hasn’t been needed. Additionally most clients accessing at the server are 1.6 clients where the features are more important.

No drastic measures have been taken about multiple subversion servers with a write master and read slaves, currently all activity is located locally on one machine with a 1TB raid.

These tips are mainly targeted at tweaking Apache.

Keep Alive

Due to way we’re using SVN, HTTP, the client will make lots of requests back to back to pull down a update or initial checkout. Essentially one request per file. To make this as efficient as possible, you can instruct Apache to allow keep-alive transactions, which allow for a single TCP session the client to continue to make requests.

Disable SVNPathAuthz

SVNPathAuthz is enabled by default in mod_dav_svn. For the software to determine if a “user” can see a path, it makes several gets (per folder/file) back to Apache to determine if they are actually visible for that user. If you do not perform per directory access controls (inside a single repository), you can safely turn this off. You will see a noticeable improvement on SVN Log information. In my tests our 15k commit repo went from 4 mins to do a svn log for the entire repo to 17 seconds (in an un-cached scenario with a 1.6 client). I picked up this tip from tty1.

Documentation on SVNPathAuthz

Enable LDAP Cache

We currently back Subversion to Active Directory over LDAP. Instead of each request round tripping to the Active Directory pool, we’ve enabled LDAP caching on the Apache side. By default mod_ldap has some caching enabled, normally 10 minutes with a certain memory size for the cache. We’ve found some small gains by drastically increasing the TTL’s from 10 min to 24 hours, and increasing the cache memory size to be up to 1gig. The idea is that, once a user accesses the site, we no longer have to round trip for the rest of that day’s operation.

Apache 2.2 mod_ldap documentation

As an additional note about Active Directory. Make sure that you have some DNS entry you can refer to that only contains Active Directory servers local to the Subversion server. For awhile had a few requests that would attempt to authenticate users to an Active Directory across the pond.

Awesomesauce

Fri, 12 Jun 2009 13:02:32 -0000

Deployment research for i want food

NGiNX
HAProxy
Munin
God
EBS

Mongrel
Rack

Memcached

Samsung Trance and Windows XP

Thu, 14 May 2009 00:27:44 -0000

If you are having issues getting the Samsung Trance SCH-u490 or any other MTP USB device to connect to Windows XP, install the latest Windows Media Player. MTP based device support was added to WinXP by Windows Media Player 10.

If you don’t do this, the phone will come up as a USB mass storage device that is unable to start.

Windows Media Player

Installing Subversion/mod_dav_svn 1.5.5 from source on Ubuntu 8.04

Thu, 19 Feb 2009 15:18:09 -0000

I assume you know how to setup and configure Apache2 and Subversion on Ubuntu already. If not go here.

You should already have apache2 and subversion installed through Ubuntu’s package management.

The following overwrites the existing Subversion install, but saves you the hassle of configuring the module system in Apache.

Warning

I’m still testing this approach before rolling it out for even my own general use, but initial testing appears positive.



# install tools and libraries to compile
sudo apt-get install make apache2-threaded-dev autoconf libtool libdb-dev

# get subversion release source and extract
wget http://subversion.tigris.org/downloads/subversion-1.5.5.tar.gz
tar xzvf subversion-1.5.5.tar.gz
cd subversion-1.5.5

# configure to use Apache apxs
./configure --with-apxs=/usr/bin/apxs2
# build and install
make clean && make && sudo make install

# have the system reload dynamic objects
sudo /sbin/ldconfig -n /usr/lib/apache2/modules


# At this stage you should update your repositories using
#  sudo svnadmin upgrade <repos path>

# restart apache
sudo /etc/init.d/apache2 restart

Safari and WWW-Authenticate: Negotiate

Fri, 23 Jan 2009 13:09:02 -0000

For my work on SPNEGO I’ve found that Safari does not play well with the Negotiate WWW-Authenticate message that needs to passed down to indicate a client can authorize with either NTLM or SPNEGO.

What appears to happen is that the Safari continuously requests the address location until some arbitrary timer is hit on the Safari side where it gives up.

I’m seeing this behavior with

Windows Safari 3.1.2(525.21)
Windows Safari 3.2.1(525.27.1)

It doesn’t matter if Negotiate is presented as the only authentication method, or if its presented along side Basic auth. Either way the browser behaves in the same way. I’d expect the client if unable to handle an authentication protocol, would simply display the 401 page provided from the request and stop. I’ve been verifying this behavior using Wireshark.

Haven’t yet found a explicit entry for this issue on Windows, but seen references to issues with NTLM on OSX

So far my only solution is to carefully examine the user-agent, but this solution isn’t elegant. I essentially disable Safari from working correctly if it ever does get fixed.

Update – New approach

So another approach has occurred to me, since I’m appending Tomcat session notes to the user, primarily to track the gss-api connection for Kerberos Auth in case it takes more then one packet to send the Kerberos key, I could also attach an additional value indicating that the user has been presented previously with a WWW-Negotiate and ignored it without sending up a valid response but the same session id. Now this falls on it’s face in some edge conditions, but most edge conditions can be resolved by simply restarting the browser or clearing the cookies for that website on the browser side.

Enabling debug output on Java PKCS #11

Fri, 16 Jan 2009 12:15:18 -0000

Pass a parameter to the VM


-Djava.security.debug="sunpkcs11"

This property is a comma separated list. For additional debug flags to turn on for the security systems in Java, see here

Google AppEngine: Data store timeout fun

Mon, 05 Jan 2009 11:55:43 -0000

My first availability issue I ran into on Google AppEngine, it appeared to resolve itself around 10 seconds later, but two sequential requests failed.
Nothing appears to be reported in the AppEngine Status

Application here

Update

It looks like there is some anomaly that Google is investigating. Graph Here

Error






      207.170.206.103 - - [05/Jan/2009:11:42:08 -0800] "GET / HTTP/1.1" 500 1244 - -

      E 01-05 11:42AM 08.253

      Traceback (most recent call last):
        File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 498, in __call__
          handler.get(*groups)
        File "/base/data/home/apps/blurry-lunch/6.330217185206397397/iwantlunch.py", line 28, in get
          for location in locations:
        File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1324, in __iter__
          return self.run()
        File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1657, in run
          query_run = self._proto_query.Run(*self._args, **self._kwds)
        File "/base/python_lib/versions/1/google/appengine/ext/gql/__init__.py", line 572, in Run
          it = bind_results.Run()
        File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 860, in Run
          return self._Run()
        File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 882, in _Run
          _ToDatastoreError(err)
        File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 1637, in _ToDatastoreError
          raise errors[err.application_error](err.error_detail)
      Timeout

Apache 2: mod_auth_ldap and Active Directory

Mon, 08 Dec 2008 15:59:42 -0000

At work we were looking to bind Subversion access to Active Directory. Of course with any large Active Directory installation, there is extensive use of nested groups to enable disable access to a wide variety of services around the company.

Thankfully mod_auth_ldap has a specific feature that can be enabled to support recursive group look ups.

Documentation Here

Combining mod_auth_ldap along with <LimitExcept> enables us to create read-only access groups to the server, while allowing developers commit access.

Notes on Compiling Tomcat 6.0.18

Wed, 03 Dec 2008 16:17:16 -0000

Why?

I needed to enable Log4j to make use of the SyslogAppender in a Linux environment. If the Juli logger was capable of this I wouldn’t have bothered.

What you need to know

You have to use JDK 1.5. The instructions presented here say 1.5.x or later, but they apparently mean 1.5 line ONLY. JDK 1.6 can not compile the DBCP dependency due to a backwards incompatible change in the Java core classes. For more information see the bug here.

Now if you are like me, you have the 1.6 JDK setup in a build environment to be used by default. Since my setup is localized to a batch script I launch a powershell environment from, all I need to do was modify the parts of the environment to point to the correct Java version. I need to put the 5 JDK Bin into my path above everything else, and update the JAVA_HOME environment variable.

This being powershell, this was a pretty simple task.


PS Z:\> $env:JAVA_HOME = "D:\dev\java\jdk1.5.0_15"
PS Z:\> $env:path = $env:JAVA_HOME + "\bin;" + $env:path
PS Z:\> java -version
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) Client VM (build 1.5.0_15-b04, mixed mode)
PS Z:\> $env:ANT_HOME
D:\dev\Java\ant\apache-ant-1.7.0

Then to compile


PS D:\dev\Java\EE\Tomcat\6.0.18\apache-tomcat-6.0.18-src> ant clean-depend
PS D:\dev\Java\EE\Tomcat\6.0.18\apache-tomcat-6.0.18-src> ant download
PS D:\dev\Java\EE\Tomcat\6.0.18\apache-tomcat-6.0.18-src> ant
PS D:\dev\Java\EE\Tomcat\6.0.18\apache-tomcat-6.0.18-src> ant -f extras.xml