March 10 2009

Reducing RAM resource usage on a VPS

Here are some miscellaneous notes as I tweak a new virtual private server (VPS). It's a time of fear and wonder as I'm leaving FreeBSD's ample and soft bosom for the as yet unknown pleasures to be found in the arms of my new Debian lover.

HTTP Daemon:

Problem? I think it's self evident. Yes, I know you can tune Apache and yes, I've been down that road before myself, but nothing I've done to tune Apache beats a simple lighttpd implementation, at least not with my application stack.

Solution: Uninstall Apache. Install lighttpd.

As a bonus, lighttpd supports SCGI (and FCGI) natively (to my knowledge, another popular lightweight httpd - nginx - does not support SCGI natively although contribs are available) which gives you deployment options for a variety of popular and not so popular Python web application frameworks.

Incidentally, QP deployed with SCGI (whether via lighttpd or Apache httpd) provides a nice combo that would be compatible with a multi-user shared hosting environment.

One of these days I'll get off my but and migrate my weblog over, and enable comments here in doing so, in order that I can satisfy a selfish desire to ask the lazy web if anyone has worked with a reasonable webmail application developed in Python. I'd really like to avoid supporting PHP in the future and at this point I only support PHP for a webmail application.

rsyslogd

This daemon appears to be the default system logging daemon on Debian; for some reason it uses many times more RAM than the default FreeBSD syslogd, even when it isn't actually doing any remote logging.

Solution: Add a call to ulimit -s 128 in do_start() in /etc/init.d/rsyslog before the daemon is started to reduce the stack size.

DNS

bind9 is a huge monster, something not often needed for a VPS or dedicated server that might need to also be an authoritative name server for a few domains. I'm sure there is some tweaking that can be done to reduce its footprint (no threads?) but for a clean break lets try another.

Solution: MaraDNS. Simple to configure (not that bind is all that hard, but some complain just the same), fast, and small. Config file uses Python syntax (n.b. use "somevalue" rather than single quotes 'some value').

Total savings so far: 80 - 120MB (more if one factors in Apache), which is pretty substantial given many low end VPS accounts start at 64 - 128MB of RAM allocation.

Time to Move, Random Observations

Looking Forward To Python 3

In my tech feed today I noted Thomas Guest sticking to his new year commitment to publish Python 3.x compatible code examples. I hope this becomes something of a meme and becomes more and more common in the pythonosphere.

On The Move

I'm decommissioning an older server I have co-located in New Jersey so over the next week or so I'll be moving my non-commercial sites and applications to a Virtual Private Server (VPS). I'm hoping to deploy all apps on the VPS using >= Python 3.1 (alpha, beta, released) alone.

Thanks to QP having already been released on Python 3 this won't be terribly difficult although as most of the applications are document centric, a workaround for docutils is inevitable for the time being as much of the content on the pro bono sites and applications I host lives in ReST format.

Markdown (the freewisdom.org version, not the ActiveState produced markdown2 code) on the other hand is already more or less serviceable under Python 3 after some 2to3 incantations.

Debian != FreeBSD but ~ Enough

After many years of running a FreeBSD-only shop at home and at work this move has more or less forced me to gain some deeper Linux knowledge. In this case I elected to have the VPS decked out with Debian 5.

While it is refreshing to explore a new OS environment just to see the differences, I do miss FreeBSD on the box, particularly the FreeBSD ports system, and I'll no doubt continue to trip on on the file system hier differences but I'll live I guess.

One aspect of running a VPS as opposed to a dedicated server is that you tend to need to be more circumspect about RAM usage. The default install of bind9 consumed a huge amount of ram which for what the VPS needs to deliver is just plain silly. So a query for "python bind alternative" drove me to choose another name daemon.

MaraDNS turns out to be a nicely lightweight authoritative and recursive DNS that uses far less machine resources than bind does. It seems to be a good alternative for a box which needs to serve up a few dozen or hundred records rather than tens of thousands, and it might even be a good choice for the latter too.