mike watkins dot ca : Resty applications in QP

Resty applications in QP

There have of late been a number of articles posted to the Pythonosphere describing approaches to delivering a REST-like or RESTful API. In keeping with the meme Eric Florenzano started by posting his bare-metal no-framework WSGI approach, to the growing list I'll add an entry for the QP web framework and Durus object database tag-team.

Eric Florenzano's deliberately minimalist WSGI application illustrating a bare metal WSGI application with RESTful behaviour has drawn out from other Python web application framework communities a number of useful comparisons:

  1. Christian Wyglendowski's entry using cherrypy
  2. Tim Parkin's restish approach
  3. Grok showing real object persistence by Martijn Faasen.
  4. A Werkzeug solution also by Tim Parkin.
  5. Carlos de la Guardia provides a solution via repoze.bfg, which, like Grok, shows how Zope3 technologies can be used.

I particularly identify with the CherryPy solution because QP's default traversal mechanism feels similar - both use an object publishing metaphor that relies on traversal of classes rather than a routes or url regular expression approach.

QP inherited its traversal approach from its older cousin Quixote, which in turn borrowed the object publishing metaphor from Zope oh so many years ago. Quixote was always a minimalistic framework without strong opinions; QP is a little more opinionated. While you can quite easily break away from QP's intentional affinity for Durus for object persistence, out of the box QP provides user and session handling all backed by the Durus object database. Despite offering much more capability than Quixote by way of the defaults and conventions chosen, QP, QPY (the modern equivalent to Quixote's PTL templating system) and Durus are small enough to be read through in a single sitting with ease.

Martijn's solution (and any Zope-solution by inference) is also somewhat familiar to me because QP's default object persistence method is the Durus Python object database. Durus can best be thought of as a simpler ZODB (and ZEO) work-a-like.

Since Eric's goal was to see how many requests he could churn out of a bare-metal WSGI application, here are some relative numbers on my workstation, an old desktop running FreeBSD on a single CPU single-core. Even on this suboptimal box the QP songs app can respond to several hundred requests per second.

Relative Performance:

%       Application
---     -----------
100     Eric's bare metal WSGI dict-driven example as baseline
58      CherryPy read from dict-driven example, Spawning server
61      QP, read from dict-driven "database" example, QP native server
60      QP, read from Durus persistent object database, QP native server
42      QP, write to Durus persistent object database, QP native server
65      QP, same app/Durus, read, simple WSGI server
50      QP, same app/Durus, read, Spawning server
59      Repoze.bfd, Spawning server, based on numbers published by Carlos
??      Grok, identified by Martijn as being appox. 50% of Eric's app on
        his machine

Certainly in my own experience raw performance has never been an issue thus I doubt I would ever want to trade away a framework I was comfortable with to write bare metal WSGI apps. Following along with the "song" meme, I'll show in the next couple of posts one approach to structuring a QP application, and some background information on QP and Durus.

Song and Song Database Objects

Let's see if we can't kill two birds with one stone and show a slightly beefier version of Eric's simple dict-based "database" using nothing but plain Python. I've added a Counter object for use in counting "I heard it" events but also for use as a key generator for our "database", much like an identity column in a SQL database.

class Counter(object):

    def __init__(self):
        self.count = 0

    def next(self):
        self.count += 1
        return self.count

class Song(object):

    def __init__(self, title):
        self.key = None
        self.title = title
        self.counter = Counter()

class SongDatabase(dict):

    def __init__(self):
        self.counter = Counter()

    def add(self, song):
        song.key = self.counter.next()
        self[song.key] = song

Later in this article with barely any additional code we'll convert these to be full-fledged persistent Python objects. Peek ahead and you'll find: two import lines, class inheritance changes, and one trivial new line of actual code.

With this "database" in place lets write a class to expose our database as a web service. Later we'll add some other features like self-documentation and an HTML UI for humans.

I don't want to do my dispatching based on either the exported name or the request method alone, so with a few lines of code I implemented a more generalized Resource class, extending QP's default Directory class. QP's default traversal mechanism is being bent slightly to take into account the HTTP request method and accept header.

class SongDatabaseResource(Resource):

    def get_resources(self):
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    def dump(self):
        """
        curl  -X GET -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        data = {}
        for song in get_song_db().values():
            data[song.key] = dict(key=song.key,
                               title=song.title,
                               count=song.counter.count)
        return json.dumps(data)

    def clear(self):
        """
        curl  -X DELETE -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        songs = get_song_db()
        songs.clear()
        songs.counter = Counter()
        return json.dumps(True)

    def new(self):
        """
        Example:
            curl  -X POST -d '{"title":"Walking Contradiction"}' \
                  -H "Content-type: application/json" \
                  -H 'Accept: application/json' http://127.0.0.1:8023/api/
            Returns:
                {"count": 0, "id": 3, "title": "Walking Contradiction"}
        """
        data = get_request().read_body()
        try:
            data = json.loads(data)
        except ValueError:
            get_publisher().respond(
                "Bad Request", "Malformed json data %r" % data, status=400)
        song = Song(data.get('title'))
        try:
            get_song_db().add(song)
        except ValueError, e:
            get_publisher().respond("Bad Request", str(e), status=400)
        return json.dumps(dict(id=song.key,
                               title=song.title,
                               count=song.counter.count))

I've added a little more meat to the basic application meme but hopefully not so much that the plot is lost.

So far we have a web service or api exposed at the logial root of our web application. The "database" isn't persistent as yet, so lets fix that now with a couple of imports, one line of new code, and a change from the default Python object base class to persistent versions of same. With these minor changes our objects are now full partners in the Durus object database. Those familiar with ZODB will certainly recognize this pattern right away.

from durus.persistent import PersistentObject
from durus.persistent_dict import PersistentDict

# persistent Python object              # regular volatile Python objects
class Counter(PersistentObject):        #   class Counter(object):

    def __init__(self):                 #       def __init__(self):
        self.count = 0                  #           self.count = 0

    def next(self):                     #       def next(self):
        self.count += 1                 #           self.count += 1
        return self.count               #           return self.count

class Song(PersistentObject):           #   class Song(object):

    def __init__(self, title):          #       def __init__(self, title):
        self.key = None                 #           self.key = None
        self.title = title              #           self.title = title
        self.counter = Counter()        #           self.counter = Counter()

class SongDatabase(PersistentDict):     #   class SongDatabase(dict):

    def __init__(self):                 #       def __init__(self):
        PersistentDict.__init__(self)   #           self.counter = Counter()
        self.counter = Counter()
                                        #       def add(self, song):
    def add(self, song):                #           song.key = self.counter.next()
        song.key = self.counter.next()  #           self[song.key] = song
        self[song.key] = song

In my next post on this theme we'll implement the rest of the SongDatabaseResource / SongResource RESTful API and some HTML UI for humans, and I'll put all the code up and a live instance for a while too.