The folks at mems-exchange.org released a new version of the Python web application / site-management framework QP and supporting packages.

All of today's released packages support Python >= 2.4, which includes Python 3.1.

They also released an update to Durus, a compact and mature Python object database (which at its core operates like a minimal ZODB/ZEO work-a-like). The API to a Python object db is simply Python.

Key-value databases appear to be in vogue these days. Python developers with an interest in key-value databases may want to check out Durus (or ZODB): the key-value database you already know, and more.

More means more than key-value pairs and simple types. More means virtually any Python object / type. More means more than simple string or integer keys and simple values and offers not only persistent dictionaries but also persistent lists and sets, and persistent objects of most any design you may wish to implement.

Durus has no other package dependencies and is compact. Weighing in with less than 5000 lines of code it small enough to read in one sitting if you want to see how things tick. Or you can just dive in - start a server durus -s and a client durus -c and play.

Inspired by a series of slides Michael Schurter published on Tokyo Cabinet and PyTyrant, I thought I'd code up his examples using another database which can use a key-value approach, Durus.

Durus is a ZODB work-a-like which allows for easy persistence of Python objects, not just values. It's simple, fast, and useful.

Here's the baseline Tokyo Cabinet db example Michael published, using the pytc interface:

import pytc
db = pytc.HDB()
db.open('test.tch',  pytc.BDBOWRITER | pytc.BDBOREADER | pytc.BDBOCREAT)
for i in range(256):
    v = chr(i)
    for x in range(256):
        db.put(chr(x), v)
        db.get(chr(x))

Running it:

$ time python test.py
real    0m0.168s
user    0m0.157s
sys     0m0.010s

And here is a Durus example, accessing a local file-based storage:

# Durus example 1 - File-based persistent dictionary
from durus.file_storage import FileStorage
from durus.connection import Connection

conn = Connection(FileStorage('test.durus'))
db = conn.get_root()
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

Running it:

$ time python durus-test.py
real        0m0.197s
user        0m0.187s
sys         0m0.008s

Now lets change to client-server operation, delivering more or less the same abilities as PyTyrant/Tokyo cabinet. A minor change to durus-test.py gives us a client:

# Durus example 2 - Remote access to a File-based persistent dictionary
from durus.client_storage import ClientStorage
from durus.connection import Connection

conn = Connection(ClientStorage())
db = conn.get_root()
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

In between each run we'll remove the database file. We'll need a server running, so in another terminal lets fire one up:

$ rm test.durus
$ durus -s --file test.durus

Run the second example:

$ time python durus-remote-test.py
real        0m0.204s
user        0m0.189s
sys         0m0.013s

Lets use a more advanced container than a persistent dictionary, a BTree. First Tokyo Cabinet/pytc:

import pytc
db = pytc.BDB()
db.open('test.db',  pytc.BDBOWRITER | pytc.BDBOREADER | pytc.BDBOCREAT)
for i in range(256):
    v = chr(i)
    for x in range(256):
        db.put(chr(x), v)
        db.get(chr(x))

Running pytc with the BTree:

$ time python test.py

real    0m0.169s
user    0m0.157s
sys     0m0.011s

Nice and fast - its all C-based.

Now the Durus BTree code:

# Durus example 3 - File-based persistent BTree
from durus.file_storage import FileStorage
from durus.connection import Connection
from durus.btree import BTree

conn = Connection(FileStorage('test.durus'))
root = conn.get_root()
db = BTree()
root['db'] = db
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

Running this we see a significant performance delta compared to the C-based pytc/Tokyo Cabinet:

$ time python durus-btree.py
real        0m1.319s
user        0m1.308s
sys         0m0.011s

The delta will tip back into Durus's favour in the next two examples.

# Durus example 4 - client-server access to a persistent BTree
from durus.client_storage import ClientStorage
from durus.connection import Connection
from durus.btree import BTree

conn = Connection(ClientStorage())
root = conn.get_root()
db = BTree()
root['db'] = db
for i in range(256):
    v = chr(i)
    for x in range(256):
        db[chr(x)] = v
        db[chr(x)]
conn.commit()

First, the access the BTree-based "db" via client-server:

$ time python durus-remote-btree-adding.py
real        0m1.691s
user        0m1.681s
sys         0m0.010s

Next we see that read only access, remote or local, remains fast, even with the BTree structure:

$ time python durus-remote-btree-ro.py
real        0m0.054s
user        0m0.040s
sys         0m0.012s

PyTyrant / TokyoCabinet has a nice simple API to accessing the remote server:

import pytyrant

t = pytyrant.PyTyrant.open('127.0.0.1', 1978)
for i in range(256):
    v = chr(i)
    for x in range(256):
        t[chr(x)] = v
        t[chr(x)]

PyTyrant client-server access to a BTree structure suggests future room for improvement:

$ time python pyt-test.py

real    0m11.151s
user    0m1.317s
sys     0m1.653s

Of course raw throughput isn't everything. Durus has persistent container types including Dictionary, BTree, Set and Lists. Keys in mappings can be any hashable object; values can be any pickleable object. Durus objects are Python objects, not merely strings or values.

Consider the following:

$ durus -c
Durus 127.0.0.1:2972
    connection -> the Connection
    root       -> the root instance
>>> from durus.persistent_dict import PersistentDict
>>> names = PersistentDict()
>>> root['names'] = names
>>> connection.commit()
>>> mike = 'Mike Watkins'
>>> fred = 'Fred Astaire'
>>> ringo = 'Ringo Starr'
>>> names[1] = mike
>>> names[2] = fred
>>> names[3] = ringo
>>> names[22] = fred
>>> id(names[2])
3082202976
>>> id(names[22])
3082202976
>>> connection.commit()

When we reconnect, we should expect the values within the mapping at keys 2 and 22 to be the same object:

$ durus -c
Durus 127.0.0.1:2972
    connection -> the Connection
    root       -> the root instance
>>> names = root['names']
>>> id(names[2])
3081790720
>>> id(names[22])
3081790720
>>> id(names[2]) == id(names[22])
True

Of late there seems to be plenty of interest in non-SQL database architectures -- CouchDB, Tokyo Cabinet among others getting attention, in part because they offer a language agnostic solution.

For those many other times when a project will benefit from a persistence layer tightly coupled with the language, object databases like Durus or ZODB are worthy of consideration.

From the QP mailing list, Wednesday December 3 2008:

Today the MEMS Exchange released updates of 5 software packages: Durus, QP, Qpy, Sancho, and Dulcinea.

You can find details and downloads at the usual location: http://www.mems-exchange.org/software/

These packages require Python 2.4 or higher, and yes, they even work with Python 3.0.

It does seem that perhaps QP 2.1 and friends is among the first if not actually the first web and database development packages available on Python 3.0 which was released today.

(Previously I've written about QP's templating system, Qpy, and a performance increase moving from Python 2.5 to 3.0.)

The attached image is a screen shot of a web browser running on a Nokia N800, a Linux-based internet tablet. Weighing only ounces, the wireless device is a great platform for Python developers as the language has more or less become the default dynamic language for the device and, it would seem, for Nokia. GUI developers commonly employ pygtk/glade for Maemo applications; I'm not aware of much web development being done on the N series tablets as yet.

Pictured is the output of a highly Pythonic web framework and object database combination, QP and Durus - the app is merely a template provided by running mkqpapp.py.

I wanted to see what QP and this little tablet, underpowered by laptop or server standards, could do - here is a benchmark between a fairly fast Unix desktop, across a fairly slow wireless link, generating 10 concurrent request streams to the device

frog# /home/mw% siege -b -c10 -t10s http://n800:8000/
Transactions:                164 hits
Availability:             100.00 %
Elapsed time:               9.85 secs
Data transferred:           0.06 MB
Response time:              0.58 secs
Transaction rate:          16.65 trans/sec
Throughput:                 0.01 MB/sec
Concurrency:                9.71
Successful transactions:     164
Failed transactions:           0
Longest transaction:        3.66
Shortest transaction:       0.19

Not bad, considering its a full stack web framework and object database running on a little machine weighing only ounces that also is running what amounts to be a Gnome environment, browser, mail and other apps, all powered by a lowly TI 320MHz Armel architecture CPU.

Out of curiosity I added a hit counter to exercise the object database and the transaction rate was a respectable 12.36/second.

Worlds smallest portable web application demo machine!

Once I figure out how to make Debian packages for the armel architecture I'll post a deb link for a one click install of Durus, QP, QPY and Dulcinea.

Incidentally, while sqlite is in common use on Nokia tablets, there's clearly no reason why Durus could not be used. N-series Python developers might find that to be an ideal persistence pairing to go along with their web or GTK apps.

Python object databases need some love too

Flávio Coelho recently performed an examination of various Python database API and ORM interfaces to MySQL, Postgres, and SQLite, and included a benchmark for cPickle.

Here's an addition to Flávio's Fastest Python Database Interface article and script to include Durus: performance.py.

I also pointed out on Flávio's blog that his cPickle benchmark needed to include pickling 100,000 "Person" classes, in addition to 100,000 simple tuples - this to show the overhead of class instantiation and serialization / deserialization which all of the ORM's and object databases share in some form or another. An example of both can be found in performance.py.

In part five of this series we dove deep into QP and looked at the fundamentals of any QP application - SitePublisher and SiteDirectory - as well as explored the use of QPY templating. We also built a rudimentary UI for our Entry object.

In this installment of our web application diary we'll work more with the Durus object database by injecting some data into it; exploring the interactive interpreter (one of the cool features of Durus to be sure) and starting the basis for a conversion script to take weblog data in PyBlosxom format and insert it into our blog application database.

pythoni

% qp -i blog
Profile, connection, publisher, root, sessions, site, users
->>

->> pu

->> from durus.persistent_dict import PersistentDict
->> mydict = PersistentDict()
->> root['test'] = mydict
->> connection.commit()
->>
%

% qp -i blog
Profile, connection, publisher, root, sessions, site, test, users
->>

->> test
<PersistentDict 17020>
->> test.items()
[]

->> test[1] = 'My first persistent data'
->> connection.commit()
->>

% qp -i blog
Profile, connection, journals, publisher, root, sessions, site, test, users
->> test.items()
[(1, 'My first persistent data')]
->>

from dulcinea.base import DulcineaPersistent
from dulcinea.sort import attr_sort
from qp.lib.keep import Keep, Keyed, Stamped
from qp.lib.spec import add_getters_and_setters, boolean, both, datetime_with_tz
from qp.lib.spec import init, pattern, string, spec
from qp.pub.user import User


class Entry(DulcineaPersistent, Keyed, Stamped):
    """
    An entry in a journal.
    """
    title_is = spec(
        (string, None),
        "A string briefly describing the Entry")
    text_is = spec(
        (string, None),
        "The entry conten")
    published_is = spec(
        boolean,
        "Boolean indicating if Entry can be published")
    author_is = spec(
        User,
        "User responsible for creating entry")
    created_is = datetime_with_tz

    def __init__(self, author):
        Keyed.__init__(self)
        Stamped.__init__(self)
        init(self, author=author, created=self.stamp, published=False)

add_getters_and_setters(Entry)

from parlez.journal import Journal

class JournalTest(UTest):
    # we'll write this first, and then write Journal

    def _pre(self):
        # set up a journal which we'll use for most tests.
        self.j = Journal('science', User('einstein'))
        # it is automatically taken down following each individual test

    def init_test(self):
        # we want Journal to have a URL name and an owner, so force it
        Journal('musings', User('joe'))

    def create_entry_test(self):
        assert isinstance(self.j.create_entry(), Entry)

    def add_test(self):
        e = self.j.create_entry()
        self.j.add(e)
        assert e in self.j.get_all_entries()
        assert e == self.j.get_entry(1)

    def only_published_test(self):
        # nothing in
        assert self.j.get_all_entries() == []
        e = self.j.create_entry()
        self.j.add(e)
        e_published = self.j.create_entry()
        e_published.set_published(True)
        self.j.add(e_published)
        assert e not in self.j.get_entries()
        assert e_published in self.j.get_entries()
        # publish e now
        e.set_published(True)
        assert e in self.j.get_entries()
        # both should be in reverse sorted result, e last
        assert [e_published, e] == self.j.get_recent_entries()

if __name__ == '__main__':
    EntryTest()
    JournalTest()

../entries/categoryname/file1.txt
../entries/categoryname/someotherfile.rst
../entries/python/2007-06-08-08-44.rst

Some article title
#author Mike Watkins
The article content.

.h2 A subtitle

More content. Etc.

from qp.lib.site import Site
from parlez.journal import Entry, Journal

def bloxsom_to_mapping(entrypath):
    # here you'll deal with the specifics - see a future article
    data = {}
    # ...
    return data

def add_journal_entries(data, journal):
    for path, entry_data in data:
        # path I might store, or some component of it, in the Entry
        # object to facilitate mapping old to new URLs in the future.
        # for now, just ignoring it
        created, format, title, content = entry_data
        entry = journal.create_entry()
        entry.set_format(format)
        entry.set_title(title)
        entry.set_text(content)
        # normally we don't bypass getters/setters
        entry.created = created
        entry.stamp = created
        journal.add_entry(entry)

if __name__ == '__main__':
    BLOXSOM_ENTRY_PATH = '/home/mw/bloxsom/entries'
    APP_NAME = 'blog'
    JOURNAL_NAME = 'mw'
    USER_ID = 'mw'

    # the Site object gives us the ability to access
    # configuration information and live objects
    site = Site(APP_NAME)
    pub = site.get_publisher()
    root = pub.get_root()
    users = root['users']
    # make sure I exist in Users
    if USER_ID not in users:
       user = pub.create_user(USER_ID)
       users.add(user)
    if 'journal' not in root:
        journal = Journal(JOURNAL_NAME, user)
        root['journal'] = journal
    # move bloxsom data into Entry/Journal
    add_journal_entries(bloxsom_to_mapping(BLOXSOM_ENTRY_PATH),
                        journal)
    # made it here, commit everything to the database
    pub.get_connection().commit()
    # that's it!