mike watkins dot ca

Resty applications in QP, Part III

Thu, 22 Jan 2009 23:07:58 GMT

Traversing the URI Path in QP Applications

In order to complete the Songs application we should understand how QP's default traversal mechanism makes it easy to provide a navigable UI for our objects we want to expose to the web.

In the last article in this series we saw through a hello, world example that a QP application has two key classes, a SitePublisher and a SiteDirectory, and that SiteDirectory exposes the root UI of a QP application. Let's flesh out a HTML interface to go along with the REST API offered by the Songs application, to include:

URI segment     Name    Returns to web browser
(export)
-------------------------------------------------------------
class SiteDirectory
/               index   HTML listing of songs
/new            new     HTML form / creation of new song
/delete         clear   HTML form allowing deletion of all songs

The functionality at these URI segments will be provided by SiteDirectory:

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('api', 'api', 'API', 'API documentation and entry point')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')

    def index:xml(self):
        header('Song Database')
        'Songs
'
        ''
        for song in get_song_db().itervalues():
            '%s: %s/">%s
' % (
                song.key, song.key, song.title)
        '
'
        footer()

    # ... for the forms examples provided by new/delete
    # read the SongDirectory class at the end of  this article

What of the API? Thanks to content-negotiation capability provided by the Resource subclass of Directory, we could intermingle our API functionality at the same URI as our HTML UI. Emphasis is on the word could because from this point forward, purely for the sake of clarity in this and subsequent articles on this theme, I'm going to risk angering the RESTafarian gods and instead host the entire REST API under the /api/ url segment. To the SiteDirectory class above we'll merely add an attribute and an entry in get_exports():

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')
        yield ('api', 'api', 'API', 'API documentation and entry point')

    api = SongDatabaseResource()

    # ...

Therefore our API will be accessible via the SongDatabaseResource subclass (first discussed here) of Resource and will expose the following:

URI      method  accept              Return
-------------------------------------------------------------
class SongDatabaseResource
/api/    GET     application/json    json dict of all songs
/api/    DELETE  application/json    json {'result': 'true'} on success
/api/    POST    application/json    json dict of new song data

Architectural style purity aside, there are reasons in the practical world why one might wish to bury an API, even a REST API, behind a mount point as we've done. The busy developer with security (or marketing) concerns might want to only expose the API to authenticated users. Let's do that:

class SongDatabaseResource(Resource):

    def get_resources(self):
        get_publisher().ensure_signed_in()
        yield export('', '__api__') # docs for web browsers
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    # ...

OK, we sketched out the first / of the HTML UI. How do we traverse to individual songs?

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Song List', 'This is the root / page of this site.')
        yield ('api', 'api', 'API', 'API documentation and entry point')
        yield ('new', 'new', 'New', 'Add a song to collection')
        yield ('delete', 'clear', 'Delete', 'Delete all songs from collection')

    # ...

    def _q_lookup(self, component):
        try:
            component = int(component)
            song = get_song_db().get(component)
            if song:
                return SongDirectory(song)
        except ValueError:
            # None / an empty '' response being returned from _q_lookup
            # will result in 404 not_found being delivered as the response.
            return None

    api = SongDatabaseResource()

The important mechanism to note here is _q_lookup. The default QP traversal mechanism (which may easily be changed) traverses Directory subclasses looking for callables or objects. One can create arbitrarily complex URI schemes with levels nested many deep, if that turns your crank.

The object traversal approach is a flexible mechanism, quite different than mapping callables or objects to URIs via regular expressions. But one could do that in QP too, if you found a need to. If you need even more flexibility you can override the Directory class _q_traverse method or dive into the Publisher itself.

One benefit of this approach is others who may use these classes generally do not need to concern themselves at all about where in their URI namespace they deploy these UI components.

The attached file: song.qpy, is a QPY template file containing a quick draft of the HTML UI.

This RESTy Songs database example has morphed into something of a QP howto. For those who just want to see the end result, fear not, I see that coming around the bend soon. It is time to tie it all together... tomorrow.

QP and WSGI

Wed, 21 Jan 2009 19:59:32 GMT

QP doesn't use WSGI itself in the operation of its own built-in web and SCGI servers, but the framework does make it easy to drive QP applications via WSGI if you have a need to. Here's a quick how-to.

Exposing the QP application as a WSGI app is simple. Create a driver file (let's name it runwsgi.py) and place it in your Python path. Myself I prefer to put such files in my QP application's "site" directory.

# this is runwsgi.py
from qp.lib.site import Site
site = Site('songs')
application = site.get_publisher()

You can then run a WSGI server (such as Spawning) from the command line:

% spawn -t 0 -p 8000 qp.sites.songs.runwsgi.application

Or we can extend runwsgi.py to also provide a self-contained WSGI server:

#! /usr/bin/env python
# this is runwsgi.py
from qp.lib.site import Site
site = Site('songs')
application = site.get_publisher()

if __name__ == '__main__':
    from wsgiref.simple_server import make_server
    host, port = site.get_http_address()
    httpd = make_server(host, port, application)
    hp = httpd.socket.getsockname()
    print("Serving HTTP on %s port %d..." % (hp[0], hp[1]))
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        print("\nStopping server(s)")
        if site.is_durus_running():
            site.stop_durus()

We can execute this directly; the application will use the same host and port defined in our application's configuration.

% ./runwsgi.py

Python 3 Compatibility

In keeping with my blog's objective of only publishing examples that are compatible with both Python 2.x and 3.x, I need to point out that the QP code presented certainly will run on Python 3.0 but the wsgi servers (spawning and the reference WSGI server) noted will not.

While QP and the rest of its stack is compatible with Python >= 2.4 (yes, that means Python 3.x too), the reference implementation of WSGI supplied with Python 3.0 is broken. As far as I know the only working WSGI server implementation for Python 3 at present is mod_wsgi, available for Apache here.

Resty applications in QP, Part II

Wed, 21 Jan 2009 19:42:32 GMT

In this installment we'll take a break from the implementation started in the post previous of a QP and Durus based RESTy song database application loosely following the meme which has arisen from Eric Florenzano's example bare-metal WSGI implementation of same. Today I want to point out some QP features which are useful and perhaps a little unusual.

Inside

Basics
QP Hello World Prototype
QPY Templates
Authentication

Basics

While QP may not be a bare-metal framework, it also isn't an mega-framework encapsulating a ton of functionality. The designer's ethos might well be extracted from the last line in QP's README:

The abbreviation "qp" stands for "quantum placet", the Latin phrase meaning "as much as you please".

QP provides basic but solid building blocks for web applications out of the box, including via its partner package QPY high level protection against XSS (cross site scripting) attacks and a novel and very fast Python templating system. You'll also get a user authentication system including form, HTTP Basic and HTTP Digest authentication.

QP Hello World Prototype

QP application's typically minimally consist of a SitePublisher and SiteDirectory class. SitePublisher contains a configuration dictionary which can be used for both the framework and your applications. Since we are using the built-in Durus object database-backed User database for the Songs application, we'll employ a DurusPublisher and instruct the qp site management command line utility to start a web and Durus daemon:

class SitePublisher(DurusPublisher):

    configuration = dict(
        durus_address=('localhost', 7023),
        http_address=('', 8023),
        )

The "root directory" of a typical QP application is a class named SiteDirectory. A hello world application driven by the above publisher would look like this:

class SiteDirectory(Directory):

    def get_exports(self):
        # URI segment exported, method name, crumb, title
        yield ('', 'index', 'Home', 'Home page of the site')

    def index(self):
        return 'hello, world.'

That's it.

Starting a QP application using the built-in site management tool is simple:

% qp -u songs

QP includes all the facilities for starting and stopping web and other processes. It uses a multi-process rather than multi-thread model and spawns new worker processes (configurable) as needed. Multi-CPU / multi-core machines can eek out significant performance with nothing but the base QP servers; for a large site typically I'll put the QP application behind an Apache or lighttpd web daemon (using either a proxy or SCGI configuration), and serve the static content with those servers. QP can also play in the WSGI space - look for a quick how-to in the next post.

QPY Templates

Let us now demonstrate one of the cooler or unusual features of QP / QPY - Python centric templating. Whether this model works for you or not depends a lot on how you look at code and (x)HTML / text templates. Programming teams who are also responsible for the site layout may find refreshing the QPY approach of putting content-in-Python, as opposed to most templating packages which focus on enabling Python-like languages within text / HTML / XMl templates.

An example should make things clear. We'll write a method to say hello back to us n-times based on url parameters age and name, to respond to a query like:

curl http://localhost:8023/?age=3&name=Mary

Instead of using return to send a string back to the Publisher, we use a special notation on our method - :xml:

from qp.pub.common import get_request

class SiteDirectory(Directory):

    def index:xml(self):

        context = get_request().fields
        '''
        
        
            Hello
        
        
            
        '''
        for i in range(int(context.get('age', 1))):
            '%(name)s, %(age)s years old
' % context
        '''
            
        
        
        '''

When index is called the string literal will be completed and passed back as a return value. Its a much cleaner looking approach than building up a string.

The method descriptor :xml is enabled by QPY. There are two such descriptors provided by QPY, :xml and :str. String literals within :xml templates are cast as a quote no more type, while variable data passed to the template (such as the context dictionary in the example) will be escaped or quoted for safety once and only once as it is included in the template. :str templates provide no automatic escaping and quoting.

The QPY templating system provides four major benefits:

You already know the language because it is pure Python.
Automatic protection from cross site scripting (XSS) attacks and other malicious value injection.
Unicode sane and safe, like the rest of the QP stack.
QPY is uncomplicated and very fast.

Speaking of Unicode, QP applications treat all text data internally as Unicode. The default character set is utf-8. Writing QP / QPY applications to run on either Python 2.x or 3.x requires no special Unicode gymnastics.

In addition, all parts of the QP stack - Durus, QPY and QP itself - are Python 3 compatible, today. QPY is packaged separately from QP to make it accessible for other templating package authors to employ features such as its quote no more class and Python extensions.

Authentication

QP's default authentication method is HTTP Digest; this can be changed by trivially overriding the ensure_signed_in() method of Publisher. For the SongDatabase API we won't alter the default since Form isn't workable for a RESTful API, and using Digest over Basic gives us the ability to offer at least authentication security without using HTTPS.

If we extended our hello, world prototype to provide some more functionality if you are logged on user, and even more if you are an administrator, it would look like this:

from qp.pub.common import get_publisher, get_user

class SiteDirectory(Directory):

    def get_exports(self):
        # URI segment exported, method name, crumb, title
        yield ('', 'index', 'Home', 'Home page of the site')
        get_publisher().ensure_signed_in()
        yield ('users', 'users', 'Users', 'Pithy title here')
        if get_user().is_admin():
            yield ('secrets', 'top_secret', 'Home', 'Secrets', 'Something intriguing')

    def index(self):
        return 'hello, world.'

    def users(self):
        return 'hello, real user %s' % get_user().id

    def top_secret(self):
        return 'the password is "orange". Sssh!'

In the foregoing example an unauthenticated user will not even be able to navigate to /users or /secrets. An authenticated user who isn't also a site administrator will not be able to navigate to /secrets.

One could put the authentication checks within an exported method instead, but the nice thing about using the get_exports mechanism is menu and crumb generation routines also use the data yielded, so putting auth checks there is a benefit if you prefer not to show your site users menus they don't currently have authorization to access.

Coming up next, a quick note on running QP as a WSGI application, and we'll get back to the Song application.

Resty applications in QP

Sun, 18 Jan 2009 23:01:02 GMT

There have of late been a number of articles posted to the Pythonosphere describing approaches to delivering a REST-like or RESTful API. In keeping with the meme Eric Florenzano started by posting his bare-metal no-framework WSGI approach, to the growing list I'll add an entry for the QP web framework and Durus object database tag-team.

Eric Florenzano's deliberately minimalist WSGI application illustrating a bare metal WSGI application with RESTful behaviour has drawn out from other Python web application framework communities a number of useful comparisons:

Christian Wyglendowski's entry using cherrypy
Tim Parkin's restish approach
Grok showing real object persistence by Martijn Faasen.
A Werkzeug solution also by Tim Parkin.
Carlos de la Guardia provides a solution via repoze.bfg, which, like Grok, shows how Zope3 technologies can be used.

I particularly identify with the CherryPy solution because QP's default traversal mechanism feels similar - both use an object publishing metaphor that relies on traversal of classes rather than a routes or url regular expression approach.

QP inherited its traversal approach from its older cousin Quixote, which in turn borrowed the object publishing metaphor from Zope oh so many years ago. Quixote was always a minimalistic framework without strong opinions; QP is a little more opinionated. While you can quite easily break away from QP's intentional affinity for Durus for object persistence, out of the box QP provides user and session handling all backed by the Durus object database. Despite offering much more capability than Quixote by way of the defaults and conventions chosen, QP, QPY (the modern equivalent to Quixote's PTL templating system) and Durus are small enough to be read through in a single sitting with ease.

Martijn's solution (and any Zope-solution by inference) is also somewhat familiar to me because QP's default object persistence method is the Durus Python object database. Durus can best be thought of as a simpler ZODB (and ZEO) work-a-like.

Since Eric's goal was to see how many requests he could churn out of a bare-metal WSGI application, here are some relative numbers on my workstation, an old desktop running FreeBSD on a single CPU single-core. Even on this suboptimal box the QP songs app can respond to several hundred requests per second.

Relative Performance:

%       Application
---     -----------
100     Eric's bare metal WSGI dict-driven example as baseline
58      CherryPy read from dict-driven example, Spawning server
61      QP, read from dict-driven "database" example, QP native server
60      QP, read from Durus persistent object database, QP native server
42      QP, write to Durus persistent object database, QP native server
65      QP, same app/Durus, read, simple WSGI server
50      QP, same app/Durus, read, Spawning server
59      Repoze.bfd, Spawning server, based on numbers published by Carlos
??      Grok, identified by Martijn as being appox. 50% of Eric's app on
        his machine

Certainly in my own experience raw performance has never been an issue thus I doubt I would ever want to trade away a framework I was comfortable with to write bare metal WSGI apps. Following along with the "song" meme, I'll show in the next couple of posts one approach to structuring a QP application, and some background information on QP and Durus.

Song and Song Database Objects

Let's see if we can't kill two birds with one stone and show a slightly beefier version of Eric's simple dict-based "database" using nothing but plain Python. I've added a Counter object for use in counting "I heard it" events but also for use as a key generator for our "database", much like an identity column in a SQL database.

class Counter(object):

    def __init__(self):
        self.count = 0

    def next(self):
        self.count += 1
        return self.count

class Song(object):

    def __init__(self, title):
        self.key = None
        self.title = title
        self.counter = Counter()

class SongDatabase(dict):

    def __init__(self):
        self.counter = Counter()

    def add(self, song):
        song.key = self.counter.next()
        self[song.key] = song

Later in this article with barely any additional code we'll convert these to be full-fledged persistent Python objects. Peek ahead and you'll find: two import lines, class inheritance changes, and one trivial new line of actual code.

With this "database" in place lets write a class to expose our database as a web service. Later we'll add some other features like self-documentation and an HTML UI for humans.

I don't want to do my dispatching based on either the exported name or the request method alone, so with a few lines of code I implemented a more generalized Resource class, extending QP's default Directory class. QP's default traversal mechanism is being bent slightly to take into account the HTTP request method and accept header.

class SongDatabaseResource(Resource):

    def get_resources(self):
        yield export('', 'dump', method='GET', accept='application/json')
        yield export('', 'new', method='POST', accept='application/json')
        yield export('', 'clear', method='DELETE', accept='application/json')

    def dump(self):
        """
        curl  -X GET -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        data = {}
        for song in get_song_db().values():
            data[song.key] = dict(key=song.key,
                               title=song.title,
                               count=song.counter.count)
        return json.dumps(data)

    def clear(self):
        """
        curl  -X DELETE -H 'Accept: application/json' http://127.0.0.1:8023/api/
        """
        songs = get_song_db()
        songs.clear()
        songs.counter = Counter()
        return json.dumps(True)

    def new(self):
        """
        Example:
            curl  -X POST -d '{"title":"Walking Contradiction"}' \
                  -H "Content-type: application/json" \
                  -H 'Accept: application/json' http://127.0.0.1:8023/api/
            Returns:
                {"count": 0, "id": 3, "title": "Walking Contradiction"}
        """
        data = get_request().read_body()
        try:
            data = json.loads(data)
        except ValueError:
            get_publisher().respond(
                "Bad Request", "Malformed json data %r" % data, status=400)
        song = Song(data.get('title'))
        try:
            get_song_db().add(song)
        except ValueError, e:
            get_publisher().respond("Bad Request", str(e), status=400)
        return json.dumps(dict(id=song.key,
                               title=song.title,
                               count=song.counter.count))

I've added a little more meat to the basic application meme but hopefully not so much that the plot is lost.

So far we have a web service or api exposed at the logial root of our web application. The "database" isn't persistent as yet, so lets fix that now with a couple of imports, one line of new code, and a change from the default Python object base class to persistent versions of same. With these minor changes our objects are now full partners in the Durus object database. Those familiar with ZODB will certainly recognize this pattern right away.

from durus.persistent import PersistentObject
from durus.persistent_dict import PersistentDict

# persistent Python object              # regular volatile Python objects
class Counter(PersistentObject):        #   class Counter(object):

    def __init__(self):                 #       def __init__(self):
        self.count = 0                  #           self.count = 0

    def next(self):                     #       def next(self):
        self.count += 1                 #           self.count += 1
        return self.count               #           return self.count

class Song(PersistentObject):           #   class Song(object):

    def __init__(self, title):          #       def __init__(self, title):
        self.key = None                 #           self.key = None
        self.title = title              #           self.title = title
        self.counter = Counter()        #           self.counter = Counter()

class SongDatabase(PersistentDict):     #   class SongDatabase(dict):

    def __init__(self):                 #       def __init__(self):
        PersistentDict.__init__(self)   #           self.counter = Counter()
        self.counter = Counter()
                                        #       def add(self, song):
    def add(self, song):                #           song.key = self.counter.next()
        song.key = self.counter.next()  #           self[song.key] = song
        self[song.key] = song

In my next post on this theme we'll implement the rest of the SongDatabaseResource / SongResource RESTful API and some HTML UI for humans, and I'll put all the code up and a live instance for a while too.

Python 2 and 3: Metaclasses

Sat, 29 Nov 2008 04:45:46 GMT

Carrying on with Python 2 to 3: Bridging the Gap, in this instalment lets cover one of the potentially thorny language changes in Python 3 - the new syntax for employing metaclasses.

Inside

Defining the metaclass
Using the metaclass in Python 2.x
Using the metaclass in Python 3.x
Using the metaclass in Python 2.x and 3.x
Ok, Prove it
Further reading on metaclasses
And now for something completely different

Syntax changes in the language are especially problematic as one or the other version simply won't know how to compile your code. Fortunately there is a clean solution for implementing metaclasses in both 2.x and 3.x, with the same source. If pressed for time, skip ahead for the solution; there is also a source code file attached to this post which encompasses all that is discussed.

I've wrapped this discussion around a working example - Monty Python inspired of course - of a basic metaclass which might be helpful for those who haven't had any exposure to the concept yet.

Defining the metaclass

The definition of a metaclass remains the same in both Python 2 and 3. Here is a possibly topical example (if you like old British humour):

class SillyWalksMetaClass(type):
    """
    A metaclass describing a class that acts as a singleton and provides a
    registry of funny walking styles, something no Monty Python fan should
    be without.
    """

    def __init__(self, class_name, bases, namespace):

        if not hasattr(self, 'walk_types'):
            self.walk_types = []
        else:
            self.walk_types.append(self)

    def __str__(self):
        return self.__name__

    def get_walks(self, *args, **kwargs):
        """Returns a list ordered by the class name"""
        return sorted(self.walk_types, key=lambda x: x.__name__)

All in all fairly straightforward. Due to what we are doing in the constructor method this class will act as a singleton - more on that later. First lets deal with using the metaclass in both 2.x and 3.x versions of Python and then look at the simple cross version solution.

Using the metaclass in Python 2.x

In Python 2.x you base your metaclass defined class by using a special attribute called __metaclass:

class SillyWalk(object):
    """
    Subclasses implement a silly walk of stunning sillyness
    """

    __metaclass__ = SillyWalksMetaClass

Creating a subclass of our new SillyWalk type is simple:

class LurchAndJump(SillyWalk):
    pass

Using the metaclass in Python 3.x

Here's a potential gotchya when migrating code to Python 3 - while the __metaclass__ attribute remains valid, it does absolutely nothing. One assumes your unit tests will have picked up on any future, ahem, oversight, but forewarned is forearmed. In Python 3 we dispense with the magic __metaclass__ attribute and instead the class definition syntax grows a keyword ability:

class SillyWalk(metaclass=SillyWalksMetaClass):
    """
    Subclasses implement a silly walk of stunning sillyness
    """

(Of course you create a subclass in exactly the same way as you would in Python 2.x)

I like this just fine, except for the fact that the Python 2.x compiler will spit up all over you if you have this syntax sprinkled in your hoped-to-be cross compatible application. Because Python 3 introduces a syntax change (one which Python 2 knows nothing about) you can't simply surround the class definition in a if sys.version < '3': block.

But never fear, there has always been an approach and its dead simple.

Using the metaclass in Python 2.x and 3.x

What we need is a method which works for both Python 2 and 3 and the solution is:

SillyWalk = SillyWalksMetaClass('SillyWalk', (object, ), {})

# That's it. Really.

Yes, that's all there is to it. Want to know more? I've included a bibliography - in particular have a look at Michele Simionato's recent articles. I wish I had as I was puzzling this out, as he provides the answer (without highlighting it as a cross-version solution mind you) in the first several paragraphs of part 1.

Digging a little deeper we see the answer was always hiding in front of us, as is frequently the case. Fire up your interpreters!

% python3.0
>>> help(type)
Help on class type in module builtins:

class type(object)
 |  type(object) -> the object's type
 |  type(name, bases, dict) -> a new type
 | ...

The answer was there all along: type(name, bases, dict) -> a new type which is exactly what we want out of our SillyWalksMetaClass which, by the way, has the following properties:

It acts as a singleton
It provides a 'registry' of sorts - you can imagine building a plugin registry or some such thing with a similar object.

Lets see SillyWalks in all its glory:

SillyWalk = SillyWalksMetaClass('SillyWalk', (object, ), {})

# As you see, it was only a flesh wound. We have a simple
# and clean approach for metaclasses for both Python 2 and
# 3.

# Subclassing remains the same for Python 2.x and Python 3+,
# lets create four favorite silly walks:

class Skip(SillyWalk):
    pass

class LurchAndSkip(SillyWalk):
    pass

class CleeseSpecial(SillyWalk):
    'The quintessential silly walk'

class HopWeaveLurchShudder(SillyWalk):
    pass

# got all four?
assert len(SillyWalk.get_walks()) == 4
# right then, lets have a look
print(', '.join([str(walk) for walk in SillyWalk.get_walks()]))
# Being a singleton, you'd expect the following to be identical:
print(', '.join([str(walk) for walk in LurchAndSkip.get_walks()]))
# and it is...
assert SillyWalk.get_walks() == HopWeaveLurchShudder.get_walks()

Ok, Prove it

$ python3.0 meta2and3.py
CleeseSpecial, HopWeaveLurchShudder, LurchAndSkip, Skip
$ python2.6 meta2and3.py
CleeseSpecial, HopWeaveLurchShudder, LurchAndSkip, Skip
$ python2.5 meta2and3.py
CleeseSpecial, HopWeaveLurchShudder, LurchAndSkip, Skip

And now for something completely different

Click for a classic, very silly, youtube video. Go on, you know you want to!

Python Web Application Diary, Part Seven

Wed, 20 Jun 2007 17:18:00 GMT

User interface handling in QP

In part six of Python Web Application Diary we looked at data persistence using Durus and wrote Sancho unit tests and a skeleton Pyblosxom migration tool. Today we'll extend the basic UI created in part five for our Entry object, and we'll create a base UI for the Journal object too. At that point we'll be able to wire our new JournalDirectory into SiteDirectory (also referenced in part five) - a QP application's "master controller" - which will allow us to start publishing previously created journal entries to the web.

JournalDirectory

I find it useful to map out the URL design for a set of related objects and then start to work on the UI. In part five we did this for EntryUI. Lets get to work on the UI 'container' for our Entries, JournalDirectory. In this first cut we'll implement enough to display a list of recent entries and put stubs in place for Atom and RSS feeds.

class JournalDirectory(Directory):
    """
    This class provides the functionality to display recent entries,
    to navigate to a specific entry, and to present RSS and Atom feeds.

    Our application urls for this UI component will be:

    ../             "index" or recent entries
    ../1234/        calls the EntryUI "index" method
    ../new          create a new Entry
    ../index.rss
    ../index.xml
    """

    def get_exports(self):
        yield ('', 'index', None, None)
        yield ('index.rss', 'rss', 'RSS', 'RSS feed for this journal')
        yield ('index.xml', 'atom', 'Atom', 'Atom feed for this journal')

    def __init__(self, journal):
        require(journal, Journal)
        self.journal = journal

    def index [html] (self):
        title = "%s's journal" % self.journal.get_name()
        header(title)
        'This is the journal of '
        self.journal.get_name()
        '
'
        for e in self.journal.get_recent_entries(count=10):
            '%s %s %s
' % (e.key, e.get_text().get_format(),
                                 href(str(e.key),e.get_title()))
        footer(title)

    def new [html] (self):
        ' not implemented '

    atom = rss = new

Looking at the code above it you can see that we've provided no way of navigating to individual entries. In index we've generated links which look like this:

Some title

What we've not done is provide a way of resolving that part of the URL path. Lets say for example we have a site:

http://mikewatkins.ca/blog

With entries:

http://mikewatkins.ca/blog/1234
http://mikewatkins.ca/blog/3456

How does the QP application resolve these paths?

_q_lookup

Unlike some web application frameworks which use regular expressions to map URL namespaces to objects/methods or functions, QP uses the concept of object publishing via object traversal. We've seen how a UI object that exposes methods can be called; _q_lookup provides us the ability to return "objects". Lets write a _q_lookup method for JournalDirectory that returns an EntryUI object for a specific journal Entry.

def _q_lookup(self, component):
    try:
        key = int(component)
    except ValueError:
        return not_found('%r is not a valid journal entry identifier.' % component)
    try:
        entry = self.journal.get_entry(key)
    except KeyError:
        return not_found('%r was not found.' % key)
    return EntryUI(entry)

At this point we have working JournalDirectory and EntryUI objects. Lets now replace our temporary demonstration code in the applications master controller with a connection to our Durus database and real data.

As you might imagine, with this object / UI pattern it would be easy to add support for multiple journals. Objects and their corresponding UI might look like:

Object              UI
--------------------------------------------
Entry               EntryUI
Journal             JournalDirectory
JournalDatabase     JournalDatabaseDirectory

Wiring up the application

For simplicities sake, we are going to first wire up our JournalDirectory to SiteDirectory which we last talked about in conjunction with the site's primary "driver", slash.qpy, last discussed in part five.

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Home', 'The Home page of this site.')
        yield ('blog', 'blog', 'Weblog', None)

    def index [html] (self):
        title = get_site().get_name()
        header(title)
        'It worked!
'
        'The %s application lives at 
' % title
        get_site().get_package_directory()
        '
'
        footer()

    mw_journal = get_publisher().get_root()['mw_journal']
    blog = JournalDirectory(mw_journal)

Or, we could wire up JournalDirectory as the 'root' of our application like this:

class SiteDirectory(JournalDirectory):

    def __init__(self):
        # I've implemented the JournalDatabase object and migrated some
        # data into a Journal.
        mw_journal = get_publisher().get_root()['journals'].get('mw')
        JournalDirectory.__init__(self, mw_journal)

Going with the latter example, at this point we can restart the qp application (qp blog restart) and visit http://localhost:8011/ and we should see a list of available posts displayed which we can navigate to. Clearly we need to do a bunch more work - implement a site look and feel; convert RST, Markdown, and Textile formatted posts into HTML, implement RSS and Atom feeds, but the basics of a journal or weblog application have come together. Fleshing out the display of existing data will be our next step.

We still can't create, edit, or delete journal entries - we'll discuss that in a soon to be forthcoming installment in this series.

Python Web Application Diary, Part Six

Fri, 08 Jun 2007 15:44:00 GMT

In part five of this series we dove deep into QP and looked at the fundamentals of any QP application - SitePublisher and SiteDirectory - as well as explored the use of QPY templating. We also built a rudimentary UI for our Entry object.

In this installment of our web application diary we'll work more with the Durus object database by injecting some data into it; exploring the interactive interpreter (one of the cool features of Durus to be sure) and starting the basis for a conversion script to take weblog data in PyBlosxom format and insert it into our blog application database.

Tip

Before going further, install Pyrepl - this is required to support QP / Durus interactive interpreter features, and adds significant functionality (optional) to Python's own interactive interpreter.

To see Pyrepl at work with regular Python launch:

pythoni

Durus, the database you already know

Now I know what you are thinking. I think. Well, that is my theory and it is mine and I own that theory. My theory is that you are thinking:

"Object database? what sort of weird and strange alchemy is that? Fear the unknown! Down with the unknown! Destroy the unknown with DELETE FROM queries!" -- you

While object databases are not exactly in commonplace use by the IT industry, within the Python community, there is a long history of kinship with object databases with ZODB, the Zope Object Database, arguably being the most well known example.

Durus is patterned after ZODB, and indeed was written by developers who had used ZODB extensively. Visit the Durus pages for more information on their rationale for reinventing this particular wheel; from my own experience I can only say that Durus is small and easy to read and understand.

What exactly is an object database? Put simply, Durus and ZODB allow you to persist your Python objects. Its more than pickle but not unlike pickle in some respects.

Tip

Launch a log viewer in another terminal window so you can watch what happens as we make changes to the Durus database. qp -l blog

Demonstrating Durus, Interactively

QP and Durus provide the facility to work directly with the Durus object database directly. Lets fire up an interactive session to show Durus basics.

% qp -i blog
Profile, connection, publisher, root, sessions, site, users
->>

Working within the interactive session: Pyrepl provides very useful search and command history capabilities. Control-P and Control-N step through previous lines entered. Control-R starts up reverse history search - start typing an entry you've made previously (searches substrings within) and Control-R again to step through the hits, if any.

Term expansion is perhaps my favorite Pyrepl enhancement - it certainly is the one that gets used enough. Try it now by entering in a couple letters:

->> pu

And press Tab - you'll be rewarded with either publisher or a list of terms in the namespace which match the letters entered so far. A real timesaver.

Access to objects: The interactive session provides us with access to QP objects (connection, site, publisher), application objects (sessions, users), a Profile testing class, but the most relevant to our discussion right now is root.

By convention our application data lives under root, which is itself a persistent object. Changes to root will persist from session to session provided a call to connection.commit() has been made to commit the changes to the database. Lets do some simple examples.

->> from durus.persistent_dict import PersistentDict
->> mydict = PersistentDict()
->> root['test'] = mydict
->> connection.commit()
->>
%

Control-D exits the interactive session, as it also exits a standard Python interpreter. Restart the interpreter to see if our object was 'saved' or persisted.

% qp -i blog
Profile, connection, publisher, root, sessions, site, test, users
->>

Very good, test, now shows up in our display -- objects living at the root level are conveniently displayed as a reminder when we fire up an interactive session. Lets put some data in test, but first, what was test?

->> test
<PersistentDict 17020>
->> test.items()
[]

Right, now I remember. Ok, add some data.

->> test[1] = 'My first persistent data'
->> connection.commit()
->>

Control-D to quit, and restart again to satisfy any fears that you may have about your important data.

% qp -i blog
Profile, connection, journals, publisher, root, sessions, site, test, users
->> test.items()
[(1, 'My first persistent data')]
->>

By now you can see that what we are doing is using Python to manage our data, and, by virtue of subclassing one of Durus persistent object classes, we can make our Python objects full partners in the Durus object database.

Durus is the database you already know. No object relational mappers to learn, no SQL to learn or work around.

Durus Mini FAQ

What about performance?: This is too difficult a question to answer simply, but its been my experience that I have been able to use Durus, instead of a SQL database (Postgres is my personal favorite among the open source databases), far more often than not. You won't put an on-line banking system processing millions of transactions a day on to Durus or ZODB; but you might base on Durus a complex company inventory system, even if there are hundreds of thousands of items and related history. Third party solutions marry Durus with relational databases as a back-end to Durus (transparent to the application) to extend Durus (ZODB has similar approaches I'm told) even further.
What about SQL / queries? How will I ever live?: One of the challenging things for a SQL-oriented developer (that was me, some time ago) is to start thinking in pure-Python again. Its not hard, but it does take some realignment of thought before it comes naturally - at least for me. Being able to dispense with relational thinking in the SQL sense brings a lot of design freedom.
What about sharing data with other systems?: My approach has been to export data as CSV or DIF for import into other systems SQL databases, or to provide APIs such as XML-RPC or REST / JSON approaches for other applications themselves, or to use RSS or Atom feeds when it makes sense.

The bottom line: Durus objects are Python objects. You've already invested in learning and knowing Python, so you already know Durus, so there is no time-to-learn downside to spending some time with Durus now. Lets press on.

Entries with no home

In part three of this series we turned a simple Entry object into a full partner of a Durus database merely by subclassing PersistentObject instead of the standard Python new-style class object. In part four we kicked things up a notch by fleshing out our Entry object with specifications provided by the QP module qp.lib.spec.

What we have not done, yet, is provide a place for our journal entries to 'live'. We need a container for Entry, and early on we decided to call that container Journal. We are really going to kick things up a notch by levering off of functionality provided by QP in qp.lib.keep. A Keep is a mapping of Keyed items using an integer as a key. Lets enhance Entry first, then we'll write some unit tests for Journal, and then write Journal itself.

All the code for the end-result objects will be available at the conclusion of this series, but for you folks following along at home, lets dive in and re-edit our journal.py and clean up our Entry object first. For brevity's sake I have included imports relevant to both Entry and the Journal object we will be writing.

from dulcinea.base import DulcineaPersistent
from dulcinea.sort import attr_sort
from qp.lib.keep import Keep, Keyed, Stamped
from qp.lib.spec import add_getters_and_setters, boolean, both, datetime_with_tz
from qp.lib.spec import init, pattern, string, spec
from qp.pub.user import User


class Entry(DulcineaPersistent, Keyed, Stamped):
    """
    An entry in a journal.
    """
    title_is = spec(
        (string, None),
        "A string briefly describing the Entry")
    text_is = spec(
        (string, None),
        "The entry conten")
    published_is = spec(
        boolean,
        "Boolean indicating if Entry can be published")
    author_is = spec(
        User,
        "User responsible for creating entry")
    created_is = datetime_with_tz

    def __init__(self, author):
        Keyed.__init__(self)
        Stamped.__init__(self)
        init(self, author=author, created=self.stamp, published=False)

add_getters_and_setters(Entry)

Lets now write Journal but before we write it, lets write the tests we want it to pass, first, and then write the object. Typically you might write only some of these tests, at least until you become familiar with the various features of the QP and Dulcinea libraries. In our ./test/utest_journal.py we'll add another test.

from parlez.journal import Journal

class JournalTest(UTest):
    # we'll write this first, and then write Journal

    def _pre(self):
        # set up a journal which we'll use for most tests.
        self.j = Journal('science', User('einstein'))
        # it is automatically taken down following each individual test

    def init_test(self):
        # we want Journal to have a URL name and an owner, so force it
        Journal('musings', User('joe'))

    def create_entry_test(self):
        assert isinstance(self.j.create_entry(), Entry)

    def add_test(self):
        e = self.j.create_entry()
        self.j.add(e)
        assert e in self.j.get_all_entries()
        assert e == self.j.get_entry(1)

    def only_published_test(self):
        # nothing in
        assert self.j.get_all_entries() == []
        e = self.j.create_entry()
        self.j.add(e)
        e_published = self.j.create_entry()
        e_published.set_published(True)
        self.j.add(e_published)
        assert e not in self.j.get_entries()
        assert e_published in self.j.get_entries()
        # publish e now
        e.set_published(True)
        assert e in self.j.get_entries()
        # both should be in reverse sorted result, e last
        assert [e_published, e] == self.j.get_recent_entries()

if __name__ == '__main__':
    EntryTest()
    JournalTest()

I've kept this briefer than I'd like it to be, as there are some other tests we need to write to completely cover our Journal object, but these tests of primary functionality - add, retrieve, retrieve all and sort - should give you the spirit of what we are trying to achieve here.

PyBlosxom to Journal Conversion

A common challenge: you've got data in one system and need to move it into a Durus database. A script to perform this task will be included in full at the end of this series. For now lets sketch out what we need to do, and look at how to access an application's Durus database from a script.

Pyblosxom maintains its files in a hierarchy that looks like something like this:

../entries/categoryname/file1.txt
../entries/categoryname/someotherfile.rst
../entries/python/2007-06-08-08-44.rst

And so on. My particular installation uses a plugin which parses the entry date from the file name if it is formatted as a datetime in the form of yyyy-mm-dd-hh-mm.ext, so for files formatted like that I can set Entry.created to a datetime parsed from the filename. Otherwise, I need to stat the file and get its creation date from the operating system, which isn't always reliable (in the case of edits and hapless administrators).

The file contents are simple for me to parse - content is either plain text, or in my instance, mostly Textile formatted with a sprinkling of reST and Markdown.:

Some article title
#author Mike Watkins
The article content.

.h2 A subtitle

More content. Etc.

I never used the #author directive; some files use the #parser directive to indicate which formatter should be used; most rely on file extensions (.rst, .txt, .mkd).

Ultimately my script needs to deliver to me:

Entry date
Format
Title
Content

And, if I intend to preserve the URLs (am debating this now... I really dislike the existing bloxsom / Pyblosxom URL design) I'll need to carry that information forward too. For now, lets assume we have a mapping containing file paths as keys and a list with the four above noted data elements to work with, and write a script to import that information into Durus.

Importing data to Durus

Working with a QP application's Durus database is easy - remember, its just Python.

from qp.lib.site import Site
from parlez.journal import Entry, Journal

def bloxsom_to_mapping(entrypath):
    # here you'll deal with the specifics - see a future article
    data = {}
    # ...
    return data

def add_journal_entries(data, journal):
    for path, entry_data in data:
        # path I might store, or some component of it, in the Entry
        # object to facilitate mapping old to new URLs in the future.
        # for now, just ignoring it
        created, format, title, content = entry_data
        entry = journal.create_entry()
        entry.set_format(format)
        entry.set_title(title)
        entry.set_text(content)
        # normally we don't bypass getters/setters
        entry.created = created
        entry.stamp = created
        journal.add_entry(entry)

if __name__ == '__main__':
    BLOXSOM_ENTRY_PATH = '/home/mw/bloxsom/entries'
    APP_NAME = 'blog'
    JOURNAL_NAME = 'mw'
    USER_ID = 'mw'

    # the Site object gives us the ability to access
    # configuration information and live objects
    site = Site(APP_NAME)
    pub = site.get_publisher()
    root = pub.get_root()
    users = root['users']
    # make sure I exist in Users
    if USER_ID not in users:
       user = pub.create_user(USER_ID)
       users.add(user)
    if 'journal' not in root:
        journal = Journal(JOURNAL_NAME, user)
        root['journal'] = journal
    # move bloxsom data into Entry/Journal
    add_journal_entries(bloxsom_to_mapping(BLOXSOM_ENTRY_PATH),
                        journal)
    # made it here, commit everything to the database
    pub.get_connection().commit()
    # that's it!

Next Installment

When we return in part seven of this series we will further flesh out our UI objects for Entry and Journal, adding methods for creating and editing objects. At that point we'll have a basic journal or weblog application ready to deploy to the world. Subsequent articles will add more functionality.

Python Web Application Diary, Part Five

Wed, 06 Jun 2007 18:51:00 GMT

In part four of this series we looked at the spec module from QP, and created some basic Sancho test cases. In this installment we'll write a few lines of code for the web UI of the application, our first real exposure to the web workings of QP.

QP Applications

In part two we created file system hierarchies for both a library and our project application. Since we plan on reusing our journal objects and related UI, those components will live in our library and our application itself will import them.

The fundamental requirements of any QP application are:

The application(s) must live somewhere in the QP search path. Issue qp -h at the command line for details.
Applications must define a module called either slash.py or slash.qpy. Both contain Python code, qpy files are QP template-aware Python modules which we'll talk about more shortly.
The slash module must expose two objects, a SitePublisher which defines the publishing environment and any application specific customizations, as well as a SiteDirectory which defines the "root" directory of the application.

We generated a basic application skeleton using a tool cooked up just for this article series, mkqpapp.py. This created in ~/qp_apps/blog the following hierarchy:

CHANGES
README
__init__.py
bin/
doc/
slash.qpy
test/
ui/
    test/
var/

Ignoring the obvious or already discussed, lets explore files of significance:

slash.qpy as discussed earlier can be thought of as the driver or root application directory of our new application.
__init__.py defines the directory as a package; however should there be any qpy modules in the given directory, __init__.py must include two lines of code:
```
from qpy.compile import compile_qpy_files
compile_qpy_files(__path__[0])
```
QPY, which was installed in part one of this series, provides unicode and quote-safe Python templating, in a manner quite different than all other Python templating approaches. Diligent use of qpy can reduce or eliminate your application's risk of being exploited by cross site scripting attacks, and SQL injection attacks, if your application uses SQL.

No magic or import hooks are required to integrate QPY with regular Python code; the two lines in __init__.py ensure that QPY files are compiled as regular Python code. Its a small price to pay for much flexibility as we'll see soon.

Exposing Objects, Methods or Functions on the Web

Lets now open the template slash.qpy module, explore, and make some changes. If you've not already started the template application created in part two, launch it now:

qp blog start

Visit the application at http://localhost:8011/ and you'll see:

It worked!

The blog application lives at /usr/home/yourhomedir/qp_sites/blog.

Lets edit slash.qpy and make some changes. First of all, since we've not discussed https secure http communications at all, yet, lets comment out some of the configuration:

class SitePublisher (DurusPublisher):

    configuration = dict(
        durus_address=('localhost', 7011),
        http_address=('', 8011),
        # as_https_address=('localhost', 9011),
        # https_address=('', 10011),
        )

The QP README contains a recipe for using stunnel for providing https SSL abilities to your QP applications.

SiteDirectory, Our Master Controller

Lets have a look at SiteDirectory, a required class which defines in essence the root of our web application. All paths lead from here, as we'll see now.

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Home', 'The Home page of this site.')

    def index [html] (self):
        title = get_site().get_name()
        header(title)
        'It worked!
'
        'The %s application lives at 
' % title
        get_site().get_package_directory()
        '
'
        footer()

Two items of note:

get_exports() explicitly defines what methods or functions we are willing to expose to the World Wide Web (or to other applications via REST or other RPC methods). Nothing is shared with the world by default, everything is explicitly enabled, if at all.

get_exports() returns a four element tuple containing:

The current directory 'pathname' to be exposed to the web, a null string if the path is to be '/'.

The name of the Python callable that the pathname refers to. The publisher 'translates' pathnames into callables, making it easy to expose deeply nested hierarchies of functionality.

A short name describing the callable; this is optionally used for menus.

A longer descriptive phrase describing the callable; this is optionally used for anchor titles / tooltips.

Thus get_exports currently defines only a single exposed callable, a null path component (''), which translates to the index of the class.

The SiteDirectory method def index [html] (self): introduces another significant component of the QP web application framework and family, QPY, which provides almost fully transparent source-code transformation of an "html template" into pure Python. QPY delivers other capabilities which deserve some looking at now.

HTML Template Basics - Hello, World

Lets add a new method to our SiteDirectory class and expose it to the web. First the method:

def hello [html] (self):
    'Hello, world
'

If you restart your application, and visit: http://localhost:8011/hello:

qp blog restart

You'll be rewarded with the default 404 response (which we can easily replace). Why? Because we forgot to expose hello() to the web:

'hello'?

Don't believe that was a 404 response? Lets also run the QP log watcher in another terminal window:

qp -l blog

You'll discover your last log entry looks something like:

2007-06-06 19:45:04 404 .007931 127.0.0.1 - - 3005 http GET /hello - Mozilla/5.0_

By now you know what must be done. Yes, that's right, we need to add an entry to get_exports():

def get_exports(self):
    yield ('', 'index', 'Home', 'The Home page of this site.')
    yield ('hello', 'hello', 'Greetings', 'A page for salutations')

Restart the application -- I map a key in vim to restart the current project I am editing -- and the revisit http://localhost:8011/hello and you'll now see returned:

Hello, world

That's not very exciting I admit. Lets alter hello() and supply the new method with some user-provided input to our hello method. For the purposes of this example, in order to keep things simple at this stage, lets pretend hello is asked to process the results of a form submission. We'll define, outside of our hello() method, a dictionary called form as such:

form = dict(
    title='User supplied title',
    text='Malicious user supplied ')

Lets update hello:

def hello [html] (self):
    'Hello, world, I'm a malicious user and posted this:p>'
    '
%(title)s
' % form
    '%(text)s
' % form

The result:

Hello, world, I am a malicious user and posted this:

User supplied title

Malicious user supplied

View source and you'll see that the content has been fully escaped and quoted:

[snip] Malicious user supplied <script>destroy_you_all()</script>

This demonstrates one of the core features of QPY - safe quoting of content not explicitly declared as safe beforehand.

With this lesson in mind, its time now to write a basic shell of a user interface for our Entry and Journal objects. The interface code won't go directly into our application but into our library of useful reusable routines - but of course you could simply create the objects and related UI in the blog application hierarchy.

Entry UI

Our last task for this installment is to flesh out the basic UI for the Entry object:

from qp.fill.directory import Directory
from qp.lib.spec import require
from qp.pub.common import header, footer
from parlez.journal import Entry, Journal, JournalDatabase

# this is for demonstration only.
from qp.pub.user import User
test_entry = Entry(User('foo'))
test_entry.set_title('An example journal entry')
test_entry.set_text("To QPY, or not to QPY,  is the question & what's for dinner?")

class EntryUI(Directory):
    """
    This class provides the functionality to display a single entry,
    edit a new or existing entry, delete an entry, and to represent
    an entry in different ways including RSS or Atom.

    Our application urls for this UI component will be:

        ../1234/
        ../1234/edit
        ../1234/delete
        ../1234/index.rss
        ../1234/index.xml
    """

    def __init__(self, component=test_entry):
        require(component, Entry)
        self.entry = component

    def get_exports(self):
        yield ('', 'index', None, None)
        yield ('edit', 'edit', 'Edit Entry', None)
        yield ('delete', 'delete', 'Delete Entry', None)
        yield ('index.rss', 'rss', 'RSS', 'RSS feed for this entry')
        yield ('index.xml', 'atom', 'Atom', 'Atom feed for this entry')

    def index [html] (self):
        # python docstrings can't be included in [html] templates
        # this template could also deliver the output of other
        # template engines such as Genshi, Cheetah, Mako, etc.
        # lets stick with QPY for now.
        title = self.entry.get_title()
        header(title)
        '%s
' % title
        self.entry.get_text()
        footer(title)

    def atom [html] (self):
        'Not Implemented'

    # lets just point all these methods to atom / not implemented for now until
    # we implement each
    edit = delete = rss = atom

If you want to race ahead and see what this looks like from the browser side, update slash.qpy by adding an import, changing get_exports and SiteDirectory as follows (don't forget to restart the application afterwards):

# import your UI object from wherever you put it.
from parlez.ui.journal import EntryUI

Modify get_exports as such:

def get_exports(self):
    yield ('', 'index', 'Home', 'The Home page of this site.')
    yield ('hello', 'hello', 'Greetings', 'A page for salutations')
    yield ('test_entry', 'test_entry', 'EntryUI Test', None)

Modify SiteDirectory by adding an attribute which points to an instance of EntryUI:

class SiteDirectory(Directory):
    # snip

    test_entry = EntryUI()

And then restart the app (qp blog restart) and visit the following URL's:

http://localhost:8011/test_entry (note what happens)

http://localhost:8011/test_entry/

http://localhost:8011/test_entry/edit

http://localhost:8011/test_entry/delete

http://localhost:8011/test_entry/index.rss

http://localhost:8011/test_entry/index.xml

Next Installment

When we return in part six of this series we will inject some real Entry data into our Durus database and explore how we will work with Durus objects interactively, from our application, and from scripts.

After that we'll turn our application shell into a functioning application by stripping out the quick and dirty demonstration data hacks and completing our "controller" logic for EntryUI and Journal user interface classes.

Python Web Application Diary, Part Four

Mon, 04 Jun 2007 21:18:00 GMT

In part three of this series we determined that Entry and Journal classes will be required, and we started to look at how straight Python classes could become full database participants in a Durus object database.

Our object model so far is very simplistic; lets add a healthy dose of constraints to aid both in testing and also prevent unintended (mis)use down the road.

I promise we'll get to webby things soon enough, we are just waiting for the chorus to come around again on the guitar. (All apologies to Arlo Guthrie)

Specifications: Contracts for Busy Developers

The QP package has a wonderful module, qp.lib.spec which deserves to get more attention whether QP does or not.

spec provides an easy way to declare or specify what various object attributes should contain, without littering your code with miles of asserts and other test. Examples will make things clear - lets take our too-simple Entry object and beef it up. First the original database-aware object:

from durus.persistent import PersistentObject

class Entry(PersistentObject):
    title = None
    text = None
    author = None
    created = None

Out of control: Clearly the object as its described presents a problem - its just an empty 'bag' with no constraints on what a user/developer might attempt to do with it. For example, whether intended or not, our dumb object allows for all sorts of questionable attribute assignments, as shown in this interactive session:

->> e = Entry()
->> e.title = 'my title'  # so far so good
->> e.title = None        # probably reasonable
->> e.title = 123.456     # not at all what we want
->> e.created = 3.1415
->> e.some_new_attribute = "foo"

The dynamic nature of Python is both blessing and curse at times; the above object requires lots of additional code to ensure that data it manages is what the developer intended, leading to significant code expansion via tests and assertions, reducing readability along the way.

In control: There is another way. Lets use spec and apply specifications and some helper methods:

from durus.persistent import PersistentObject
from qp.lib.spec import datetime_with_tz, spec, string
from qp.lib.spec import add_getters_and_setters

class Entry(PersistentObject):

    title_is = spec(
        (string, None),
        'A short description of the entry')
    text_is = spec(
        (string, None),
        'The full text of the entry')
    author_is = spec(
        User,
        'The individual responsible for the content of the entry')
    created_is = datetime_with_tz

add_getters_and_setters(Entry)

A spec can be a simple type assignment (see created_is above) or make use of the spec function which allows for a certain amount of self-documentation which can be very helpful. Arguments supplied in tuple imply the specification either, or you can spell it out: either(string, None).

Now lets run through the same interactive session as before:

->> e = Entry()
->> e.set_title('my title')
->> e.set_title(None)
->> e.set_title(123.456)
Traceback (most recent call last):
  File "", line 2, in <module>
  File "/usr/local/lib/python2.5/site-packages/qp/lib/spec.py", line 725, in f
    require(value, getattr(klass, name + '_is'))
  File "/usr/local/lib/python2.5/site-packages/qp/lib/spec.py", line 171, in require
    raise TypeError(error)
TypeError:
  Expected: (string, None)
  A short description of the entry
  Got: 123.456

Aha, for the cost of a pair of get and set methods (no moaning please, we didn't even have to write the getter and setter ourselves and they don't clutter the code), we've effectively constrained what type of data can be assigned to the title attribute of Entry, while also preserving the easy to read nature of the original, simple, code.

To demonstrate the utility of spec, I've pulled a number of examples from Dulcinea and my own code. As you can see, there is great flexibility provided:

date_is = datetime

approvals_is = spec(
    sequence(DulcineaUser, set),
    "The users who agree the issue is resolved.")

issues_is = spec(
    mapping({string:Issue}, PersistentDict),
    "Mapping of issue IDs to issues.")

id_is = spec(
    pattern('^[-A-Za-z0-9_@.]*$'),
    "unique among users here")

# some specs are referred to over and over again

datetime_without_tz = both(datetime, with_attribute(tzinfo=None))
datetime_with_tz = both(datetime, with_attribute(tzinfo=no(None)))
email_pattern = pattern("^.+@.+\..{2,4}$")
existing_user = both(User, with_attribute(id=no(None)))
hex_pattern = pattern('[a-fA-F0-9]*$')

# reuse them

email_is = email_pattern
id_is = spec(
    hex_pattern,
    'a lower case alphanumeric pattern')

# use the specifications in tests (more powerful and cleaner than
# testing for type and instance alone)

require(thing, either(list, tuple))
match(a_user, existing_user) # returns boolean

My experience is that you can create fairly complex specifications that remain very readable. Have a complex object that has to be "just so" before it is committed to a database in a transaction? Specify everything, and check it for sanity with a one line assertion: assert get_spec_problems(theobject_instance) == [] and you are done.

Testing, Testing, One Two Three

As you might imagine, it becomes easier to write unit tests when our objects are so highly specified. Sancho, a unit testing framework also from the same development shop from which QP originates, is designed for projects and teams who prefer to leave code in a working state, all or most of the time.

Tests live in ./test, one level down from our objects being tested, and there is no __init__.py. A utility, urun.py, will execute one test supplied on the command line, or all tests in the test subdirectories in the current directory and below. Lets write one for Entry:

# /www/lib/parlez/test/utest_journal.py
from parlez.journal import Entry
from sancho.utest import UTest, raises

entry_text = '''This is a blog entry.\n\n*We hope you like it*.'''

class EntryTest(UTest):

    def init_test(self):
        Entry()

    def entry_test(self):
        joe = User('joe')
        e = Entry()
        e.set_author(joe)
        # a string causes a TypeError, authors must be User instances
        raises(TypeError, e.set_author, 'Joe')
        assert e.get_author() == joe
        assert e.get_created() == e.get_stamp()
        e.set_text(entry_text)
        assert e.get_text() == entry_text
        e.set_stamp()
        assert e.get_created() != e.get_stamp()

class JournalTest(UTest):
    # we'll write this shortly, before Journal!
    pass

if __name__ == '__main__':
    EntryTest()
    JournalTest()

Run urun.py from the command line or from your editor and the result:

# /www/lib/parlez/test% urun.py
./utest_journal.py: EntryTest:

No tracebacks indicates successful test(s).

In part five of this series we'll start to write the HTML (remember, this article series is apparently about web development with QP) and other user interfaces for our Entry object.

Python Web Application Diary, Part Three

Mon, 04 Jun 2007 18:33:00 GMT

In part two of this series we created a location and file system hierarchy for application library objects, UI and other components, and did the same for an actual application by using a script mkqpapp.py that automates those tasks.

Today lets start writing code -- we'll begin by defining basic objects for managing weblog or journal entries, and then we'll move on to showing how QP and Durus make defining and publishing your Python objects as easy as, well, py.

Basic Data Elements

As discussed in part one, this tutorial / web application project will result in a basic weblog or on-line journal application. Lets break down a weblog into its most basic data elements:

A weblog is a collection of writing, generally presented in chronological fashion. A weblog could be considered a diary or journal, so lets use the term Journal to describe its function.
A journal usually, but not always, represents the thoughts and opinions of a single author.
Each item in a journal can be considered an article or a post - lets use a more generic term and call each item in the journal an Entry. Entries are typically short bits of text so lets enter and store them as such. Each entry may have a title, and may include other information including dates relating to when the Entry was created, made available to readers, or changed -- but the principal information is the entry itself.

Our first classes

Turning to Python then, we could easily represent Entry as:

class Entry(object):
    title = None
    text = None
    created = None

We could then use the class:

e = Entry()
e.title = 'Python Web Application Diary, Part Three'
e.text = 'Hello, Bruce, my name is Bruce.'
created = datetime.datetime.now()

That was pretty simple, no? Simplicity can be both a boon and a pain in the butt, and experienced developers will recognize at least two significant problems with our still too-simple Entry object:

There is no way of easily persisting this data (saving it so that its available later when we need it)
The current design doesn't warn or otherwise prevent someone from intentionally or accidentally storing data we don't expect, such as:

e = Entry()
e.title = datetime.datetime.now()
e.created = 'Python Web Application Humour'

Kicking Entry Up a Notch - Persistence

Lets first look at the issue of persistence. Keeping our journal entries around for future display (or edits) could be done by:

Saving the data into individual files
Saving the data in a relational (SQL) database such as Postgres, Oracle, MySQL or MS SQL Server

To SQL or not to SQL, that is the question

Most often these days by default a developer will turn to a SQL database to store and manage persistent data. While there is nothing wrong with this, introducing SQL into the mix does complicate matters some what. SQL types are not exactly analogous to Python data types, and accessing and updating data held in a SQL repository can often require lots of tedious SQL code, in addition to your Python code and objects.

To ease the friction or so-called impedance mismatch between Python and SQL, various Object Relational Mappers (ORMs) have appeared on the Python scene. While ORMs like SQLObject and SQL Alchemy do make using SQL-based data within a Python application somewhat more convenient, its equally true that not all applications need the added complexity and there are other alternatives which can be useful to Python programmers regardless of complexity.

QP doesn't enforce a particular data persistence approach upon a developer, but it does make a choice for you which you can then consciously choose to ignore.

Durus, a Python Object Database

Rather than deal with the impedance mismatch between Python and SQL, QP by default uses Durus, a Python object database, to persist application data.

The truly neat thing about Durus, for Python users, is that you almost know how to use it now, sight unseen.

Lets take our dirt-simple Entry object and make it database aware. You'll recall the basic object looked like this:

class Entry(object):
    title = None
    text = None
    created = None

An Entry object able to participate in the Durus object database looks like this:

class Entry(PersistentObject):
    title = None
    text = None
    created = None

As you can see, other than subclassing a special Durus type, PersistentObject, there are no outward differences. We can therefore make a straightforward claim: Durus is the database you already know.

We'll revisit Durus and object persistence in a future installment. In Part Four of this series we shall take our too-simple, but persistent, object and show how specifications can add useful constraints. We'll also take our first look at Sancho, a unit testing framework.