The chart, and numbers, say it all.

Harper and his Conservative government cronies have been piling on the debt for virtually all of fiscal year 2008/09, and, only after the fall election has come and gone, Harper and Flaherty now tell us its going to get much worse.

Cumulative financial source (surplus) and requirement (debt)

Figures are from the monthly Fiscal Monitor, published by the Ministry of Finance: All Months 2008 | November 2008 (PDF)

Even as Stephen Harper and Jimmy Flaherty were pounding the election trail telling us that Canada would not experience a recession, and would not entertain the notion of fiscal deficits, Stephen Harper's government was piling on the debt in Canada. That was back in October. In November it got even worse.

There is no other way to put this - the big lie they told allowed them to hang on to power.

During the election, Harper knew that his government was piling on the debt already. On the hustings both he and Flaherty promised that Canada would not entertain deficit spending, but even as he made that promise Canada was already funding its spending via debt. The government was, and clearly still is, spending more than it takes in.

While politicians may try to hind behind cute word games like "budgetary surplus", real Canadians know that if they spend more than they take in, they go deeper in debt, whether one had a budget for the spending or not.

As the financial source/(requirement) chart above shows, throughout this fiscal year the Harper government's financial performance has dramatically underperformed every single government this decade, and has done so for virtually the entire fiscal year. Even during the 911 and SARS crises Canada's finances handled the stormy weather with ease. Under Harper's watch, the bilge is filling, the pumps aren't working, and the boat threatens to sink into the sea.

Traversing the URI Path in QP Applications

In order to complete the Songs application we should understand how QP's default traversal mechanism makes it easy to provide a navigable UI for our objects we want to expose to the web.

In the last article in this series we saw through a hello, world example that a QP application has two key classes, a SitePublisher and a SiteDirectory, and that SiteDirectory exposes the root UI of a QP application. Let's flesh out a HTML interface to go along with the REST API offered by the Songs application, to include:

URI segment     Name    Returns to web browser
(export)
-------------------------------------------------------------
class SiteDirectory
/               index   HTML listing of songs
/new            new     HTML form / creation of new song
/delete         clear   HTML form allowing deletion of all songs

The functionality at these URI segments will be provided by SiteDirectory:

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('api', 'api', 'API', 'API documentation and entry point')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')

    def index:xml(self):
        header('Song Database')
        '<h1>Songs</h1>'
        '<ul>'
        for song in get_song_db().itervalues():
            '<li>%s: <a href="%s/">%s</a></li>' % (
                song.key, song.key, song.title)
        '</ul>'
        footer()

    # ... for the forms examples provided by new/delete
    # read the SongDirectory class at the end of  this article

What of the API? Thanks to content-negotiation capability provided by the Resource subclass of Directory, we could intermingle our API functionality at the same URI as our HTML UI. Emphasis is on the word could because from this point forward, purely for the sake of clarity in this and subsequent articles on this theme, I'm going to risk angering the RESTafarian gods and instead host the entire REST API under the /api/ url segment. To the SiteDirectory class above we'll merely add an attribute and an entry in get_exports():

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')
        yield ('api', 'api', 'API', 'API documentation and entry point')

    api = SongDatabaseResource()

    # ...

Therefore our API will be accessible via the SongDatabaseResource subclass (first discussed here) of Resource and will expose the following:

URI      method  accept              Return
-------------------------------------------------------------
class SongDatabaseResource
/api/    GET     application/json    json dict of all songs
/api/    DELETE  application/json    json {'result': 'true'} on success
/api/    POST    application/json    json dict of new song data

Architectural style purity aside, there are reasons in the practical world why one might wish to bury an API, even a REST API, behind a mount point as we've done. The busy developer with security (or marketing) concerns might want to only expose the API to authenticated users. Let's do that:

class SongDatabaseResource(Resource):

    def get_resources(self):
        get_publisher().ensure_signed_in()
        yield export('', '__api__') # docs for web browsers
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    # ...

OK, we sketched out the first / of the HTML UI. How do we traverse to individual songs?

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('api', 'api', 'API', 'API documentation and entry point')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')

    # ...

    def _q_lookup(self, component):
        try:
            component = int(component)
            song = get_song_db().get(component)
            if song:
                return SongDirectory(song)
        except ValueError:
            # None / an empty '' response being returned from _q_lookup
            # will result in 404 not_found being delivered as the response.
            return None

    api = SongDatabaseResource()

The important mechanism to note here is _q_lookup. The default QP traversal mechanism (which may easily be changed) traverses Directory subclasses looking for callables or objects. One can create arbitrarily complex URI schemes with levels nested many deep, if that turns your crank.

The object traversal approach is a flexible mechanism, quite different than mapping callables or objects to URIs via regular expressions. But one could do that in QP too, if you found a need to. If you need even more flexibility you can override the Directory class _q_traverse method or dive into the Publisher itself.

One benefit of this approach is others who may use these classes generally do not need to concern themselves at all about where in their URI namespace they deploy these UI components.

The attached file: song.qpy, is a QPY template file containing a quick draft of the HTML UI.

This RESTy Songs database example has morphed into something of a QP howto. For those who just want to see the end result, fear not, I see that coming around the bend soon. It is time to tie it all together... tomorrow.

QP doesn't use WSGI itself in the operation of its own built-in web and SCGI servers, but the framework does make it easy to drive QP applications via WSGI if you have a need to. Here's a quick how-to.

Exposing the QP application as a WSGI app is simple. Create a driver file (let's name it runwsgi.py) and place it in your Python path. Myself I prefer to put such files in my QP application's "site" directory.

# this is runwsgi.py
from qp.lib.site import Site
site = Site('songs')
application = site.get_publisher()

You can then run a WSGI server (such as Spawning) from the command line:

% spawn -t 0 -p 8000 qp.sites.songs.runwsgi.application

Or we can extend runwsgi.py to also provide a self-contained WSGI server:

#! /usr/bin/env python
# this is runwsgi.py
from qp.lib.site import Site
site = Site('songs')
application = site.get_publisher()

if __name__ == '__main__':
    from wsgiref.simple_server import make_server
    host, port = site.get_http_address()
    httpd = make_server(host, port, application)
    hp = httpd.socket.getsockname()
    print("Serving HTTP on %s port %d..." % (hp[0], hp[1]))
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        print("\nStopping server(s)")
        if site.is_durus_running():
            site.stop_durus()

We can execute this directly; the application will use the same host and port defined in our application's configuration.

% ./runwsgi.py

In this installment we'll take a break from the implementation started in the post previous of a QP and Durus based RESTy song database application loosely following the meme which has arisen from Eric Florenzano's example bare-metal WSGI implementation of same. Today I want to point out some QP features which are useful and perhaps a little unusual.

class SitePublisher(DurusPublisher):

    configuration = dict(
        durus_address=('localhost', 7023),
        http_address=('', 8023),
        )

class SiteDirectory(Directory):

    def get_exports(self):
        # URI segment exported, method name, crumb, title
        yield ('', 'index', 'Home', 'Home page of the site')

    def index(self):
        return 'hello, world.'

% qp -u songs

curl http://localhost:8023/?age=3&name=Mary

from qp.pub.common import get_request

class SiteDirectory(Directory):

    def index:xml(self):

        context = get_request().fields
        '''
        <html>
        <head>
            <title>Hello</title>
        </head>
        <body>
            <ol>
        '''
        for i in range(int(context.get('age', 1))):
            '<li>%(name)s, %(age)s years old</li>' % context
        '''
            </ol>
        </body>
        </html>
        '''

from qp.pub.common import get_publisher, get_user

class SiteDirectory(Directory):

    def get_exports(self):
        # URI segment exported, method name, crumb, title
        yield ('', 'index', 'Home', 'Home page of the site')
        get_publisher().ensure_signed_in()
        yield ('users', 'users', 'Users', 'Pithy title here')
        if get_user().is_admin():
            yield ('secrets', 'top_secret', 'Home', 'Secrets', 'Something intriguing')

    def index(self):
        return 'hello, world.'

    def users(self):
        return 'hello, real user %s' % get_user().id

    def top_secret(self):
        return 'the password is "orange". Sssh!'

There have of late been a number of articles posted to the Pythonosphere describing approaches to delivering a REST-like or RESTful API. In keeping with the meme Eric Florenzano started by posting his bare-metal no-framework WSGI approach, to the growing list I'll add an entry for the QP web framework and Durus object database tag-team.

Eric Florenzano's deliberately minimalist WSGI application illustrating a bare metal WSGI application with RESTful behaviour has drawn out from other Python web application framework communities a number of useful comparisons:

1. Christian Wyglendowski's entry using cherrypy
2. Tim Parkin's restish approach
3. Grok showing real object persistence by Martijn Faasen.
4. A Werkzeug solution also by Tim Parkin.
5. Carlos de la Guardia provides a solution via repoze.bfg, which, like Grok, shows how Zope3 technologies can be used.

I particularly identify with the CherryPy solution because QP's default traversal mechanism feels similar - both use an object publishing metaphor that relies on traversal of classes rather than a routes or url regular expression approach.

QP inherited its traversal approach from its older cousin Quixote, which in turn borrowed the object publishing metaphor from Zope oh so many years ago. Quixote was always a minimalistic framework without strong opinions; QP is a little more opinionated. While you can quite easily break away from QP's intentional affinity for Durus for object persistence, out of the box QP provides user and session handling all backed by the Durus object database. Despite offering much more capability than Quixote by way of the defaults and conventions chosen, QP, QPY (the modern equivalent to Quixote's PTL templating system) and Durus are small enough to be read through in a single sitting with ease.

Martijn's solution (and any Zope-solution by inference) is also somewhat familiar to me because QP's default object persistence method is the Durus Python object database. Durus can best be thought of as a simpler ZODB (and ZEO) work-a-like.

Since Eric's goal was to see how many requests he could churn out of a bare-metal WSGI application, here are some relative numbers on my workstation, an old desktop running FreeBSD on a single CPU single-core. Even on this suboptimal box the QP songs app can respond to several hundred requests per second.

Relative Performance:

%       Application
---     -----------
100     Eric's bare metal WSGI dict-driven example as baseline
58      CherryPy read from dict-driven example, Spawning server
61      QP, read from dict-driven "database" example, QP native server
60      QP, read from Durus persistent object database, QP native server
42      QP, write to Durus persistent object database, QP native server
65      QP, same app/Durus, read, simple WSGI server
50      QP, same app/Durus, read, Spawning server
59      Repoze.bfd, Spawning server, based on numbers published by Carlos
??      Grok, identified by Martijn as being appox. 50% of Eric's app on
        his machine

Certainly in my own experience raw performance has never been an issue thus I doubt I would ever want to trade away a framework I was comfortable with to write bare metal WSGI apps. Following along with the "song" meme, I'll show in the next couple of posts one approach to structuring a QP application, and some background information on QP and Durus.

class Counter(object):

    def __init__(self):
        self.count = 0

    def next(self):
        self.count += 1
        return self.count

class Song(object):

    def __init__(self, title):
        self.key = None
        self.title = title
        self.counter = Counter()

class SongDatabase(dict):

    def __init__(self):
        self.counter = Counter()

    def add(self, song):
        song.key = self.counter.next()
        self[song.key] = song

class SongDatabaseResource(Resource):

    def get_resources(self):
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    def dump(self):
        """
        curl  -X GET -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        data = {}
        for song in get_song_db().values():
            data[song.key] = dict(key=song.key,
                               title=song.title,
                               count=song.counter.count)
        return json.dumps(data)

    def clear(self):
        """
        curl  -X DELETE -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        songs = get_song_db()
        songs.clear()
        songs.counter = Counter()
        return json.dumps(True)

    def new(self):
        """
        Example:
            curl  -X POST -d '{"title":"Walking Contradiction"}' \
                  -H "Content-type: application/json" \
                  -H 'Accept: application/json' http://127.0.0.1:8023/api/
            Returns:
                {"count": 0, "id": 3, "title": "Walking Contradiction"}
        """
        data = get_request().read_body()
        try:
            data = json.loads(data)
        except ValueError:
            get_publisher().respond(
                "Bad Request", "Malformed json data %r" % data, status=400)
        song = Song(data.get('title'))
        try:
            get_song_db().add(song)
        except ValueError, e:
            get_publisher().respond("Bad Request", str(e), status=400)
        return json.dumps(dict(id=song.key,
                               title=song.title,
                               count=song.counter.count))

from durus.persistent import PersistentObject
from durus.persistent_dict import PersistentDict

# persistent Python object              # regular volatile Python objects
class Counter(PersistentObject):        #   class Counter(object):

    def __init__(self):                 #       def __init__(self):
        self.count = 0                  #           self.count = 0

    def next(self):                     #       def next(self):
        self.count += 1                 #           self.count += 1
        return self.count               #           return self.count

class Song(PersistentObject):           #   class Song(object):

    def __init__(self, title):          #       def __init__(self, title):
        self.key = None                 #           self.key = None
        self.title = title              #           self.title = title
        self.counter = Counter()        #           self.counter = Counter()

class SongDatabase(PersistentDict):     #   class SongDatabase(dict):

    def __init__(self):                 #       def __init__(self):
        PersistentDict.__init__(self)   #           self.counter = Counter()
        self.counter = Counter()
                                        #       def add(self, song):
    def add(self, song):                #           song.key = self.counter.next()
        song.key = self.counter.next()  #           self[song.key] = song
        self[song.key] = song

Ever since Google Chrome (a web browser which like Firefox is Open Source) first came out in beta I've been using it. I'd only intended to check it out but by the end of the day I had made it my default web browser. Firefox and Chrome are the only browsers I recommend for use on Windows.

This cartoon series on Google Chrome is a slightly technical presentation on what sets the Chrome web browser apart. Your time for a quick glance will be rewarded even if you aren't a geek, if only to admire the artistry.