For the duration of the nuclear crisis in Japan I'm producing a XML (Atom) feed of the status reports published by the Japan Atomic Industrial Forum Inc. (JAIF) The JAIF site is checked every 10 minutes for updates; they seem to be updating a couple times per day.

News readers can subscribe to the feed at: http://mikewatkins.ca/japan_status.xml

Now included in the same feed are any post-earthquake updates from Tokyo Electric Power Company (TEPCO). In addition the feed includes earthquake reports from the U.S. Geological Survey, limited to earthquakes near Japan, Hawaii, Canada and the U.S. west coast - with a magnitude of 5.4 or greater.

The International Atomic Energy Agency is maintaining a daily update log.

Another compiled source of information on the disaster is available from http://nirs.org/ - organized chronologically, this PDF may be of interest.

Noted via #python on Twitter, a link to this discussion on creating a simple Movie and Actor database. Some suggestions for the original poster included using Shelve; myself I'd prefer to move up a rung or two and use a Python object database like ZODB or Durus, either of which can be used for anything from simple to complex solutions.

Attached is a dirt-simple module implementing a movie and actor database. Look ma, no SQL. The core of it is here, with the objects of interest subclassing Durus's Persistent class:

# Our "Models" -- it's just Python.

class Actor(Persistent):

    def __init__(self, name):
        self.name = name

    def __str__(self):
        return '<Actor: %s>' % self.name


class Movie(Persistent):

    def __init__(self, title):
        self.title = title
        self.actors = PersistentSet()

    def add_actor(self, actor):
        self.actors.add(actor)

    def __str__(self):
        return '<Movie: %s>' % self.title


class Database(Persistent):

    def __init__(self):
        self.movies = PersistentDict()
        self.actors = PersistentDict()

    def add_movie(self, movie):
        assert movie.title not in self.movies, 'Already exists!'
        self.movies[movie.title] = movie
        return movie

    def get_movie(self, title):
        return self.movies.get(title, None)

    def add_actor(self, actor):
        assert actor.name not in self.actors, 'Already exists!'
        self.actors[actor.name] = actor

    def get_actor(self, name):
        return self.actors.get(name, None)

    def appears_in(self, actor):
        """(actor : Actor) -> [Movie,]

        Returns a sequence of Movies, if any, the Actor played a role in.
        """
        return [m for m in self.movies.values() if actor in m.actors]

Objects which subclass the Persistent class become persistence-aware. Aside from having some special container classes for persistent dicts, lists, sets and BTrees for performance with large mappings, there isn't much of a mental burden to design objects that participate in a Durus or ZODB object database.

An example of retrieving some data from our film database:

if __name__ == '__main__':
    from examples.durus.flicks.movies import load_it, get_db
    from durus.file_storage import FileStorage
    from durus.connection import Connection

    connection = Connection(FileStorage("movies.durus"))
    # load some sample data
    if not connection.get_root().get('films', False):
        load_it(connection)
    db = get_db(connection)
    hopkins = db.get_actor('Anthony Hopkins')
    mel = db.get_actor('Mel Gibson')

    # Which films has an actor appeared in?
    for actor in (hopkins, mel):
        print()
        print('%s is in:' % actor.name)
        print([str(m) for m in db.appears_in(actor)])

    # What films are in our movie database and name known actors in each
    print("\nComplete Movie Listing")
    for m in db.movies.values():
        print(m.title, ':', ','.join([str(a) for a in m.actors]))

Running this script the first time you'll note some log activity to stdout as the object instances are committed to the database. The listing output will look like this:

Anthony Hopkins is in:
['<Movie: Mutiny on the Bounty>']

Mel Gibson is in:
['<Movie: Braveheart>', '<Movie: Lethal Weapon II>',
'<Movie: Mutiny on the Bounty>', '<Movie: Lethal Weapon III>',
'<Movie: Lethal Weapon>']

Complete Movie Listing
Braveheart : <Actor: Mel Gibson>
Lethal Weapon II : <Actor: Mel Gibson>,<Actor: Danny Glover>
Mutiny on the Bounty : <Actor: Mel Gibson>,<Actor: Anthony Hopkins>
Lethal Weapon III : <Actor: Mel Gibson>,<Actor: Danny Glover>
Lethal Weapon : <Actor: Mel Gibson>,<Actor: Danny Glover>

(No commentary on my taste for films, please)

In addition to the FileStorage class, Durus offers a client-server solution ClientStorage suitable for use by multi-process applications such as web apps. ZODB provides the same through its ZEO feature.

I'm sure I'm not the only one who suffers from "installation frustration" from time to time. At the outset I want to declare that I'm not complaining about easy_install at all here. In fact, I'm not complaining about anything at all but instead am pointing out what a bad installation experience does to the curious.

My story is cyclical in nature. I don't do any Zope work but do work with ZODB and Durus, a ZODB work alike, and a web app traversal mechanism that is Zope-similar. As a result I'm always inclined to check out major new releases of Zope or Grok or Plone and the ecosystem that surrounds same.

Sadly I've learned to expect frustration and failure in pursuit of my goal to get the new tool of the day installed and have a quick peek at a running demonstration app and examine what's under the covers.

Yet as time passes and the frustration fades, invariably I find myself willing to try again, which brings us to the present. Plone 4 is moving through beta releases and today seemed like a good day to install it and peek under the covers. The official documentation is simple enough, but installation failed and left me with a hankering to cure the problem with rm -rf plone.

Later in the day, inspired by the Jazkarta Blog entry on easy_install Plone, I thought I'd do just that give it another go. No joy there either, and the comments have since sprouted a warning that this should not be done, for well and good legitimate reasons.

But the bottom line remains that a buildout driven install plus official documentation failed to deliver a working instance, so what does one do? Roll out rm -rf plone and move on?

This sort of installation frustration isn't limited to major albeit often beta releases, and certainly isn't limited to Zope or Plone or Grok, although their very long chain of tools can make it daunting for a complete greenhorn to puzzle out a solution quickly. I've had the same experience same thing occur with official tutorials, or too often even basic installation documentation is simply wrong.

Remember, this isn't a complaint. I know how much effort goes into solution development and that many products are built largely or entirely on the back of volunteer labour. I'm merely pointing out that a bad installation experience can turn off someone who might otherwise be a relatively decent candidate to use the product and perhaps one day become part of the contributing community. Those who are on the outside of a product's community haven't yet drunk the koolaid and don't know what fantastic stuff is hidden within the bowels of your package, and they might never know, if the cost of admittance is too steep from the outset.

At the very least make sure the new release can install on a totally new system, or a clean virtualenv, by following only the installation instructions, followed verbatim, in your README or INSTALL files. Don't introduce jargon into these first documentation entry points that outsiders will not know.

Over the past nine months I've been leasing a FreeBSD VPS (Virtual Private Server) to supplement my own hardware and I've just realized that the virtual machine hasn't caused me any (more) grey hair or other anguish in all that time. In my books vendors that save hair and brain cells deserve at least a passing mention.

At present I'm using one FreeBSD VPS for staging Python web apps and hosting a couple of pro bono sites. I quite frequently refer people to more full-service shared hosting firms such as WebFaction (even my brother has an account there) but personally I prefer managing the entire box - virtual or not - myself.

As I'd like to turn off some aging servers of my own, I'm hoping that virtualization from firms like ARP Networks, or on grander scale like Google or Amazon's cloud services, will save me spending some funds on equipment leases. I've tried a few "virtual" server providers over the years but I do believe this is the first time I've passed on a recommendation here in these pages.

What I like:

• OS support: FreeBSD, OpenBSD, and most Linux distributions
• Decent multi-homed bandwidth with redundancy and plenty of peering agreements turned up; data center is in LA; quite good network access to Asia, Australia
• IPv6 support
• IP accessible / out-of-band console access
• Configure, build and install your own world and kernel if you like
• Great performance, terrific even if you measure performance per dollar. I can churn out > 1000 requests per second with a trivial test page with a single DB lookup, which is a darn sight better than some of my own ageing hardware is able to do.

What I wish they offered:

• a Canadian datacenter too!

The virtualization technology involves KVM (not Xen or OpenVZ) and I find that the resource allocation is very fair indeed. 768MB RAM / 20GB disk / 100GB bandwidth - all this and I get stability too for $20 USD a month.

There are plenty of really cheap virtual server offerings out there, but not very many stable yet inexpensive operations. ARP Networks is certainly one of the latter.

FreeBSDvps.com / OpenBSDvps.net will both lead you to ARP Networks which is the love child of Garry Dolley. Garry not only knows what he is doing, he actually appears to enjoy managing server and network hardware.

Everyone appreciates reliable hardware and networks, but I think the service offering will appeal most to those who can competently manage a server from the command line. ARP Networks' target market appears to be people who know what they are doing, which is good company to keep.

The folks at mems-exchange.org released a new version of the Python web application / site-management framework QP and supporting packages.

All of today's released packages support Python >= 2.4, which includes Python 3.1.

They also released an update to Durus, a compact and mature Python object database (which at its core operates like a minimal ZODB/ZEO work-a-like). The API to a Python object db is simply Python.

Key-value databases appear to be in vogue these days. Python developers with an interest in key-value databases may want to check out Durus (or ZODB): the key-value database you already know, and more.

More means more than key-value pairs and simple types. More means virtually any Python object / type. More means more than simple string or integer keys and simple values and offers not only persistent dictionaries but also persistent lists and sets, and persistent objects of most any design you may wish to implement.

Durus has no other package dependencies and is compact. Weighing in with less than 5000 lines of code it small enough to read in one sitting if you want to see how things tick. Or you can just dive in - start a server durus -s and a client durus -c and play.

Cold Fusion and some PHP sites are being hit hard over the last 24 hours by what appear to be Chinese cyber-criminals using an exploit found within the popular browser-based editor component FCKeditor. The vulnerability allows for remote code execution / uploading of files to arbitrary locations / installation of remote shells and it would appear that many sites are being attacked.

Versions <= the current shipping version (FCKeditor <= 2.6.4) are vulnerable. A patch does not yet exist; in the mean time disabling the file browser is one of the mitigation steps.

A number of Python projects utilize this editor component.

oCERT Advisory: http://www.ocert.org/advisories/ocert-2009-007.html

Updated to include a comparison to Python 2.5

Python 3.1 has closed much of the performance gap between 3.x and 2.x, as well as delivering some new features and enhancements to the language and stdlib. Enhancement that caught my eye first: re.sub | re.subn | re.split now accept a flags parameter, no more remembering how to pass that in as part of the regex. And I think I'm going to like auto numbered format() fields i.e. 'We are the {} who say {}'.format('Knights', 'Ni!').

On first blush it may not seem that the performance gap has been closed very much. A simplistic web app benchmark:

siege -b -t60s -c2  http://localhost:8000/

Old machine (light mail server load / FreeBSD 7.2)
Python 2.6     520.91 trans/sec
Python 3.1     388.38 trans/sec

Newer Virtual Machine (moderately loaded server / FreeBSD 7.2)
Python 2.6   1568.58 trans/sec
Python 3.1    1220.92 trans/sec

Faster Virtual Machine (quiet server / Debian 5 - OpenVZ)
Python 2.5    2304.78 trans/sec [1]
Python 3.1    1960.07 trans/sec

Looking deeper, I'm seeing little difference between 2.6 and 3.1, here benchmarking a more involved web app benchmark than "hello world" that is probably more representative of a typical yet simple web app:

siege -b -t60s -c2  http://localhost:8002/

Old machine (light mail server load / FreeBSD 7.2)
Python 2.6    197.40 trans/sec
Python 3.1    194.09 trans/sec

Newer Virtual Machine (moderately loaded server / FreeBSD 7.2)
Python 2.6    649.57 trans/sec
Python 3.1    629.50 trans/sec

Faster Virtual Machine (quiet server / Debian 5 - OpenVZ)
Python 2.5    765.88 trans/sec
Python 3.1    850.24 trans/sec

[1] Note Python 2.5 being compared in this test instance

From what I can tell, for the sorts of tasks I run into, Python 3.1 appears to perform as well or better than Python 2.5 and is approaching the current speed daemon, Python 2.6. Take that with a grain of salt, as I found 3.0 was fast enough for my needs.

Gosh, has it been three months since I posted last? House renos are coming along, but it only gets worse from here before it gets better. We'll be cooking on the patio soon enough.

Semi-random tech link of the day: FreeBSD ports now default to Python 2.6. Yay. You could always force FreeBSD to use a specific Python version -- edit /etc/make.conf and add the line PYTHON_DEFAULT_VERSION=python2.6 or whatever version you like.

I smell a skunk nearby. No, really, they like to walk down the sidewalk past my office window. Unpleasant, but not nearly as bad as having them spray the house. And this leads me to the skunk control tip of the day: do not try to dissuade two skunks from fighting at 2am in the morning below your kitchen window by throwing water at them.

Inspired by a series of slides Michael Schurter published on Tokyo Cabinet and PyTyrant, I thought I'd code up his examples using another database which can use a key-value approach, Durus.

Durus is a ZODB work-a-like which allows for easy persistence of Python objects, not just values. It's simple, fast, and useful.

Here's the baseline Tokyo Cabinet db example Michael published, using the pytc interface:

import pytc
db = pytc.HDB()
db.open('test.tch',  pytc.BDBOWRITER | pytc.BDBOREADER | pytc.BDBOCREAT)
for i in range(256):
    v = chr(i)
    for x in range(256):
        db.put(chr(x), v)
        db.get(chr(x))

Running it:

$ time python test.py
real    0m0.168s
user    0m0.157s
sys     0m0.010s

And here is a Durus example, accessing a local file-based storage:

# Durus example 1 - File-based persistent dictionary
from durus.file_storage import FileStorage
from durus.connection import Connection

conn = Connection(FileStorage('test.durus'))
db = conn.get_root()
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

Running it:

$ time python durus-test.py
real        0m0.197s
user        0m0.187s
sys         0m0.008s

Now lets change to client-server operation, delivering more or less the same abilities as PyTyrant/Tokyo cabinet. A minor change to durus-test.py gives us a client:

# Durus example 2 - Remote access to a File-based persistent dictionary
from durus.client_storage import ClientStorage
from durus.connection import Connection

conn = Connection(ClientStorage())
db = conn.get_root()
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

In between each run we'll remove the database file. We'll need a server running, so in another terminal lets fire one up:

$ rm test.durus
$ durus -s --file test.durus

Run the second example:

$ time python durus-remote-test.py
real        0m0.204s
user        0m0.189s
sys         0m0.013s

Lets use a more advanced container than a persistent dictionary, a BTree. First Tokyo Cabinet/pytc:

import pytc
db = pytc.BDB()
db.open('test.db',  pytc.BDBOWRITER | pytc.BDBOREADER | pytc.BDBOCREAT)
for i in range(256):
    v = chr(i)
    for x in range(256):
        db.put(chr(x), v)
        db.get(chr(x))

Running pytc with the BTree:

$ time python test.py

real    0m0.169s
user    0m0.157s
sys     0m0.011s

Nice and fast - its all C-based.

Now the Durus BTree code:

# Durus example 3 - File-based persistent BTree
from durus.file_storage import FileStorage
from durus.connection import Connection
from durus.btree import BTree

conn = Connection(FileStorage('test.durus'))
root = conn.get_root()
db = BTree()
root['db'] = db
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

Running this we see a significant performance delta compared to the C-based pytc/Tokyo Cabinet:

$ time python durus-btree.py
real        0m1.319s
user        0m1.308s
sys         0m0.011s

The delta will tip back into Durus's favour in the next two examples.

# Durus example 4 - client-server access to a persistent BTree
from durus.client_storage import ClientStorage
from durus.connection import Connection
from durus.btree import BTree

conn = Connection(ClientStorage())
root = conn.get_root()
db = BTree()
root['db'] = db
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

First, the access the BTree-based "db" via client-server:

$ time python durus-remote-btree-adding.py
real        0m1.691s
user        0m1.681s
sys         0m0.010s

Next we see that read only access, remote or local, remains fast, even with the BTree structure:

$ time python durus-remote-btree-ro.py
real        0m0.054s
user        0m0.040s
sys         0m0.012s

PyTyrant / TokyoCabinet has a nice simple API to accessing the remote server:

import pytyrant

t = pytyrant.PyTyrant.open('127.0.0.1', 1978)
for i in range(256):
    v = chr(i)
    for x in range(256):
        t[chr(x)] = v
        t[chr(x)]

PyTyrant client-server access to a BTree structure suggests future room for improvement:

$ time python pyt-test.py

real    0m11.151s
user    0m1.317s
sys     0m1.653s

Of course raw throughput isn't everything. Durus has persistent container types including Dictionary, BTree, Set and Lists. Keys in mappings can be any hashable object; values can be any pickleable object. Durus objects are Python objects, not merely strings or values.

Consider the following:

$ durus -c
Durus 127.0.0.1:2972
    connection -> the Connection
    root       -> the root instance
>>> from durus.persistent_dict import PersistentDict
>>> names = PersistentDict()
>>> root['names'] = names
>>> connection.commit()
>>> mike = 'Mike Watkins'
>>> fred = 'Fred Astaire'
>>> ringo = 'Ringo Starr'
>>> names[1] = mike
>>> names[2] = fred
>>> names[3] = ringo
>>> names[22] = fred
>>> id(names[2])
3082202976
>>> id(names[22])
3082202976
>>> connection.commit()

When we reconnect, we should expect the values within the mapping at keys 2 and 22 to be the same object:

$ durus -c
Durus 127.0.0.1:2972
    connection -> the Connection
    root       -> the root instance
>>> names = root['names']
>>> id(names[2])
3081790720
>>> id(names[22])
3081790720
>>> id(names[2]) == id(names[22])
True

Of late there seems to be plenty of interest in non-SQL database architectures -- CouchDB, Tokyo Cabinet among others getting attention, in part because they offer a language agnostic solution.

For those many other times when a project will benefit from a persistence layer tightly coupled with the language, object databases like Durus or ZODB are worthy of consideration.

Here are some miscellaneous notes as I tweak a new virtual private server (VPS). It's a time of fear and wonder as I'm leaving FreeBSD's ample and soft bosom for the as yet unknown pleasures to be found in the arms of my new Debian lover.

HTTP Daemon:

Problem? I think it's self evident. Yes, I know you can tune Apache and yes, I've been down that road before myself, but nothing I've done to tune Apache beats a simple lighttpd implementation, at least not with my application stack.

Solution: Uninstall Apache. Install lighttpd.

As a bonus, lighttpd supports SCGI (and FCGI) natively (to my knowledge, another popular lightweight httpd - nginx - does not support SCGI natively although contribs are available) which gives you deployment options for a variety of popular and not so popular Python web application frameworks.

Incidentally, QP deployed with SCGI (whether via lighttpd or Apache httpd) provides a nice combo that would be compatible with a multi-user shared hosting environment.

One of these days I'll get off my but and migrate my weblog over, and enable comments here in doing so, in order that I can satisfy a selfish desire to ask the lazy web if anyone has worked with a reasonable webmail application developed in Python. I'd really like to avoid supporting PHP in the future and at this point I only support PHP for a webmail application.

rsyslogd

This daemon appears to be the default system logging daemon on Debian; for some reason it uses many times more RAM than the default FreeBSD syslogd, even when it isn't actually doing any remote logging.

Solution: Add a call to ulimit -s 128 in do_start() in /etc/init.d/rsyslog before the daemon is started to reduce the stack size.

DNS

bind9 is a huge monster, something not often needed for a VPS or dedicated server that might need to also be an authoritative name server for a few domains. I'm sure there is some tweaking that can be done to reduce its footprint (no threads?) but for a clean break lets try another.

Solution: MaraDNS. Simple to configure (not that bind is all that hard, but some complain just the same), fast, and small. Config file uses Python syntax (n.b. use "somevalue" rather than single quotes 'some value').

Total savings so far: 80 - 120MB (more if one factors in Apache), which is pretty substantial given many low end VPS accounts start at 64 - 128MB of RAM allocation.