mike watkins dot ca

Resty applications in QP, Part III

Thu, 22 Jan 2009 23:07:58 GMT

Traversing the URI Path in QP Applications

In order to complete the Songs application we should understand how QP's default traversal mechanism makes it easy to provide a navigable UI for our objects we want to expose to the web.

In the last article in this series we saw through a hello, world example that a QP application has two key classes, a SitePublisher and a SiteDirectory, and that SiteDirectory exposes the root UI of a QP application. Let's flesh out a HTML interface to go along with the REST API offered by the Songs application, to include:

URI segment     Name    Returns to web browser
(export)
-------------------------------------------------------------
class SiteDirectory
/               index   HTML listing of songs
/new            new     HTML form / creation of new song
/delete         clear   HTML form allowing deletion of all songs

The functionality at these URI segments will be provided by SiteDirectory:

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('api', 'api', 'API', 'API documentation and entry point')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')

    def index:xml(self):
        header('Song Database')
        'Songs
'
        ''
        for song in get_song_db().itervalues():
            '%s: %s/">%s
' % (
                song.key, song.key, song.title)
        '
'
        footer()

    # ... for the forms examples provided by new/delete
    # read the SongDirectory class at the end of  this article

What of the API? Thanks to content-negotiation capability provided by the Resource subclass of Directory, we could intermingle our API functionality at the same URI as our HTML UI. Emphasis is on the word could because from this point forward, purely for the sake of clarity in this and subsequent articles on this theme, I'm going to risk angering the RESTafarian gods and instead host the entire REST API under the /api/ url segment. To the SiteDirectory class above we'll merely add an attribute and an entry in get_exports():

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')
        yield ('api', 'api', 'API', 'API documentation and entry point')

    api = SongDatabaseResource()

    # ...

Therefore our API will be accessible via the SongDatabaseResource subclass (first discussed here) of Resource and will expose the following:

URI      method  accept              Return
-------------------------------------------------------------
class SongDatabaseResource
/api/    GET     application/json    json dict of all songs
/api/    DELETE  application/json    json {'result': 'true'} on success
/api/    POST    application/json    json dict of new song data

Architectural style purity aside, there are reasons in the practical world why one might wish to bury an API, even a REST API, behind a mount point as we've done. The busy developer with security (or marketing) concerns might want to only expose the API to authenticated users. Let's do that:

class SongDatabaseResource(Resource):

    def get_resources(self):
        get_publisher().ensure_signed_in()
        yield export('', '__api__') # docs for web browsers
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    # ...

OK, we sketched out the first / of the HTML UI. How do we traverse to individual songs?

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('api', 'api', 'API', 'API documentation and entry point')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')

    # ...

    def _q_lookup(self, component):
        try:
            component = int(component)
            song = get_song_db().get(component)
            if song:
                return SongDirectory(song)
        except ValueError:
            # None / an empty '' response being returned from _q_lookup
            # will result in 404 not_found being delivered as the response.
            return None

    api = SongDatabaseResource()

The important mechanism to note here is _q_lookup. The default QP traversal mechanism (which may easily be changed) traverses Directory subclasses looking for callables or objects. One can create arbitrarily complex URI schemes with levels nested many deep, if that turns your crank.

The object traversal approach is a flexible mechanism, quite different than mapping callables or objects to URIs via regular expressions. But one could do that in QP too, if you found a need to. If you need even more flexibility you can override the Directory class _q_traverse method or dive into the Publisher itself.

One benefit of this approach is others who may use these classes generally do not need to concern themselves at all about where in their URI namespace they deploy these UI components.

The attached file: song.qpy, is a QPY template file containing a quick draft of the HTML UI.

This RESTy Songs database example has morphed into something of a QP howto. For those who just want to see the end result, fear not, I see that coming around the bend soon. It is time to tie it all together... tomorrow.

Resty applications in QP, Part II

Wed, 21 Jan 2009 19:42:32 GMT

In this installment we'll take a break from the implementation started in the post previous of a QP and Durus based RESTy song database application loosely following the meme which has arisen from Eric Florenzano's example bare-metal WSGI implementation of same. Today I want to point out some QP features which are useful and perhaps a little unusual.

Inside

Basics
QP Hello World Prototype
QPY Templates
Authentication

Basics

While QP may not be a bare-metal framework, it also isn't an mega-framework encapsulating a ton of functionality. The designer's ethos might well be extracted from the last line in QP's README:

The abbreviation "qp" stands for "quantum placet", the Latin phrase meaning "as much as you please".

QP provides basic but solid building blocks for web applications out of the box, including via its partner package QPY high level protection against XSS (cross site scripting) attacks and a novel and very fast Python templating system. You'll also get a user authentication system including form, HTTP Basic and HTTP Digest authentication.

QP Hello World Prototype

QP application's typically minimally consist of a SitePublisher and SiteDirectory class. SitePublisher contains a configuration dictionary which can be used for both the framework and your applications. Since we are using the built-in Durus object database-backed User database for the Songs application, we'll employ a DurusPublisher and instruct the qp site management command line utility to start a web and Durus daemon:

class SitePublisher(DurusPublisher):

    configuration = dict(
        durus_address=('localhost', 7023),
        http_address=('', 8023),
        )

The "root directory" of a typical QP application is a class named SiteDirectory. A hello world application driven by the above publisher would look like this:

class SiteDirectory(Directory):

    def get_exports(self):
        # URI segment exported, method name, crumb, title
        yield ('', 'index', 'Home', 'Home page of the site')

    def index(self):
        return 'hello, world.'

That's it.

Starting a QP application using the built-in site management tool is simple:

% qp -u songs

QP includes all the facilities for starting and stopping web and other processes. It uses a multi-process rather than multi-thread model and spawns new worker processes (configurable) as needed. Multi-CPU / multi-core machines can eek out significant performance with nothing but the base QP servers; for a large site typically I'll put the QP application behind an Apache or lighttpd web daemon (using either a proxy or SCGI configuration), and serve the static content with those servers. QP can also play in the WSGI space - look for a quick how-to in the next post.

QPY Templates

Let us now demonstrate one of the cooler or unusual features of QP / QPY - Python centric templating. Whether this model works for you or not depends a lot on how you look at code and (x)HTML / text templates. Programming teams who are also responsible for the site layout may find refreshing the QPY approach of putting content-in-Python, as opposed to most templating packages which focus on enabling Python-like languages within text / HTML / XMl templates.

An example should make things clear. We'll write a method to say hello back to us n-times based on url parameters age and name, to respond to a query like:

curl http://localhost:8023/?age=3&name=Mary

Instead of using return to send a string back to the Publisher, we use a special notation on our method - :xml:

from qp.pub.common import get_request

class SiteDirectory(Directory):

    def index:xml(self):

        context = get_request().fields
        '''
        
        
            Hello
        
        
            
        '''
        for i in range(int(context.get('age', 1))):
            '%(name)s, %(age)s years old
' % context
        '''
            
        
        
        '''

When index is called the string literal will be completed and passed back as a return value. Its a much cleaner looking approach than building up a string.

The method descriptor :xml is enabled by QPY. There are two such descriptors provided by QPY, :xml and :str. String literals within :xml templates are cast as a quote no more type, while variable data passed to the template (such as the context dictionary in the example) will be escaped or quoted for safety once and only once as it is included in the template. :str templates provide no automatic escaping and quoting.

The QPY templating system provides four major benefits:

You already know the language because it is pure Python.
Automatic protection from cross site scripting (XSS) attacks and other malicious value injection.
Unicode sane and safe, like the rest of the QP stack.
QPY is uncomplicated and very fast.

Speaking of Unicode, QP applications treat all text data internally as Unicode. The default character set is utf-8. Writing QP / QPY applications to run on either Python 2.x or 3.x requires no special Unicode gymnastics.

In addition, all parts of the QP stack - Durus, QPY and QP itself - are Python 3 compatible, today. QPY is packaged separately from QP to make it accessible for other templating package authors to employ features such as its quote no more class and Python extensions.

Authentication

QP's default authentication method is HTTP Digest; this can be changed by trivially overriding the ensure_signed_in() method of Publisher. For the SongDatabase API we won't alter the default since Form isn't workable for a RESTful API, and using Digest over Basic gives us the ability to offer at least authentication security without using HTTPS.

If we extended our hello, world prototype to provide some more functionality if you are logged on user, and even more if you are an administrator, it would look like this:

from qp.pub.common import get_publisher, get_user

class SiteDirectory(Directory):

    def get_exports(self):
        # URI segment exported, method name, crumb, title
        yield ('', 'index', 'Home', 'Home page of the site')
        get_publisher().ensure_signed_in()
        yield ('users', 'users', 'Users', 'Pithy title here')
        if get_user().is_admin():
            yield ('secrets', 'top_secret', 'Home', 'Secrets', 'Something intriguing')

    def index(self):
        return 'hello, world.'

    def users(self):
        return 'hello, real user %s' % get_user().id

    def top_secret(self):
        return 'the password is "orange". Sssh!'

In the foregoing example an unauthenticated user will not even be able to navigate to /users or /secrets. An authenticated user who isn't also a site administrator will not be able to navigate to /secrets.

One could put the authentication checks within an exported method instead, but the nice thing about using the get_exports mechanism is menu and crumb generation routines also use the data yielded, so putting auth checks there is a benefit if you prefer not to show your site users menus they don't currently have authorization to access.

Coming up next, a quick note on running QP as a WSGI application, and we'll get back to the Song application.

Resty applications in QP

Sun, 18 Jan 2009 23:01:02 GMT

There have of late been a number of articles posted to the Pythonosphere describing approaches to delivering a REST-like or RESTful API. In keeping with the meme Eric Florenzano started by posting his bare-metal no-framework WSGI approach, to the growing list I'll add an entry for the QP web framework and Durus object database tag-team.

Eric Florenzano's deliberately minimalist WSGI application illustrating a bare metal WSGI application with RESTful behaviour has drawn out from other Python web application framework communities a number of useful comparisons:

Christian Wyglendowski's entry using cherrypy
Tim Parkin's restish approach
Grok showing real object persistence by Martijn Faasen.
A Werkzeug solution also by Tim Parkin.
Carlos de la Guardia provides a solution via repoze.bfg, which, like Grok, shows how Zope3 technologies can be used.

I particularly identify with the CherryPy solution because QP's default traversal mechanism feels similar - both use an object publishing metaphor that relies on traversal of classes rather than a routes or url regular expression approach.

QP inherited its traversal approach from its older cousin Quixote, which in turn borrowed the object publishing metaphor from Zope oh so many years ago. Quixote was always a minimalistic framework without strong opinions; QP is a little more opinionated. While you can quite easily break away from QP's intentional affinity for Durus for object persistence, out of the box QP provides user and session handling all backed by the Durus object database. Despite offering much more capability than Quixote by way of the defaults and conventions chosen, QP, QPY (the modern equivalent to Quixote's PTL templating system) and Durus are small enough to be read through in a single sitting with ease.

Martijn's solution (and any Zope-solution by inference) is also somewhat familiar to me because QP's default object persistence method is the Durus Python object database. Durus can best be thought of as a simpler ZODB (and ZEO) work-a-like.

Since Eric's goal was to see how many requests he could churn out of a bare-metal WSGI application, here are some relative numbers on my workstation, an old desktop running FreeBSD on a single CPU single-core. Even on this suboptimal box the QP songs app can respond to several hundred requests per second.

Relative Performance:

%       Application
---     -----------
100     Eric's bare metal WSGI dict-driven example as baseline
58      CherryPy read from dict-driven example, Spawning server
61      QP, read from dict-driven "database" example, QP native server
60      QP, read from Durus persistent object database, QP native server
42      QP, write to Durus persistent object database, QP native server
65      QP, same app/Durus, read, simple WSGI server
50      QP, same app/Durus, read, Spawning server
59      Repoze.bfd, Spawning server, based on numbers published by Carlos
??      Grok, identified by Martijn as being appox. 50% of Eric's app on
        his machine

Certainly in my own experience raw performance has never been an issue thus I doubt I would ever want to trade away a framework I was comfortable with to write bare metal WSGI apps. Following along with the "song" meme, I'll show in the next couple of posts one approach to structuring a QP application, and some background information on QP and Durus.

Song and Song Database Objects

Let's see if we can't kill two birds with one stone and show a slightly beefier version of Eric's simple dict-based "database" using nothing but plain Python. I've added a Counter object for use in counting "I heard it" events but also for use as a key generator for our "database", much like an identity column in a SQL database.

class Counter(object):

    def __init__(self):
        self.count = 0

    def next(self):
        self.count += 1
        return self.count

class Song(object):

    def __init__(self, title):
        self.key = None
        self.title = title
        self.counter = Counter()

class SongDatabase(dict):

    def __init__(self):
        self.counter = Counter()

    def add(self, song):
        song.key = self.counter.next()
        self[song.key] = song

Later in this article with barely any additional code we'll convert these to be full-fledged persistent Python objects. Peek ahead and you'll find: two import lines, class inheritance changes, and one trivial new line of actual code.

With this "database" in place lets write a class to expose our database as a web service. Later we'll add some other features like self-documentation and an HTML UI for humans.

I don't want to do my dispatching based on either the exported name or the request method alone, so with a few lines of code I implemented a more generalized Resource class, extending QP's default Directory class. QP's default traversal mechanism is being bent slightly to take into account the HTTP request method and accept header.

class SongDatabaseResource(Resource):

    def get_resources(self):
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    def dump(self):
        """
        curl  -X GET -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        data = {}
        for song in get_song_db().values():
            data[song.key] = dict(key=song.key,
                               title=song.title,
                               count=song.counter.count)
        return json.dumps(data)

    def clear(self):
        """
        curl  -X DELETE -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        songs = get_song_db()
        songs.clear()
        songs.counter = Counter()
        return json.dumps(True)

    def new(self):
        """
        Example:
            curl  -X POST -d '{"title":"Walking Contradiction"}' \
                  -H "Content-type: application/json" \
                  -H 'Accept: application/json' http://127.0.0.1:8023/api/
            Returns:
                {"count": 0, "id": 3, "title": "Walking Contradiction"}
        """
        data = get_request().read_body()
        try:
            data = json.loads(data)
        except ValueError:
            get_publisher().respond(
                "Bad Request", "Malformed json data %r" % data, status=400)
        song = Song(data.get('title'))
        try:
            get_song_db().add(song)
        except ValueError, e:
            get_publisher().respond("Bad Request", str(e), status=400)
        return json.dumps(dict(id=song.key,
                               title=song.title,
                               count=song.counter.count))

I've added a little more meat to the basic application meme but hopefully not so much that the plot is lost.

So far we have a web service or api exposed at the logial root of our web application. The "database" isn't persistent as yet, so lets fix that now with a couple of imports, one line of new code, and a change from the default Python object base class to persistent versions of same. With these minor changes our objects are now full partners in the Durus object database. Those familiar with ZODB will certainly recognize this pattern right away.

from durus.persistent import PersistentObject
from durus.persistent_dict import PersistentDict

# persistent Python object              # regular volatile Python objects
class Counter(PersistentObject):        #   class Counter(object):

    def __init__(self):                 #       def __init__(self):
        self.count = 0                  #           self.count = 0

    def next(self):                     #       def next(self):
        self.count += 1                 #           self.count += 1
        return self.count               #           return self.count

class Song(PersistentObject):           #   class Song(object):

    def __init__(self, title):          #       def __init__(self, title):
        self.key = None                 #           self.key = None
        self.title = title              #           self.title = title
        self.counter = Counter()        #           self.counter = Counter()

class SongDatabase(PersistentDict):     #   class SongDatabase(dict):

    def __init__(self):                 #       def __init__(self):
        PersistentDict.__init__(self)   #           self.counter = Counter()
        self.counter = Counter()
                                        #       def add(self, song):
    def add(self, song):                #           song.key = self.counter.next()
        song.key = self.counter.next()  #           self[song.key] = song
        self[song.key] = song

In my next post on this theme we'll implement the rest of the SongDatabaseResource / SongResource RESTful API and some HTML UI for humans, and I'll put all the code up and a live instance for a while too.