QP Web Framework, One Week In

So far its been a delight to use QP, the new web framework, put out by the Mems-Exchange folks. I bow in the general direction of CNRI / Mems-exchange and all those from there, past and present, that have made available to Python web developers Quixote, Durus, QP and other quality tools.

I mentioned QP the other day and since then I’ve been diving right in. After putting up a (live) demo wiki application, written in a single sitting, I’ve had chance to do some real work with QP. Unicode handling, site management, integration with Durus, and QPY make for a truly productive work environment.

Bonus feature of QP/QPY discovered today: pydoc works on QPY templates, where pydoc choked on PTL. Since my PTL templates more or less just moved over by renaming the file extension, that alone is almost reason enough to switch!

I’m migrating a moderately sized application over from Quixote to QP at present, partly because it was good timing and partly because unicode just works in QP. I also like the authentication mechanisms implemented in QP.

SCGI - Python-Ruby Pollination

Python roots help Ruby on Rails: Neil Schemenauer was behind the creation of SCGI, a simple, high performance replacement for CGI, first implemented for the Quixote web framework. I wonder if Neil expected it to become so popular for other languages —Ruby on Rails for example now has SCGI Ruby on Rails Runner and some 25,000 other hits on Google and growing.

SCGI is supported on Apache via Neil’s mod_scgi work; also native support is available in the now very popular lighttpd; maybe these Cherokee folks will get with the program one day too:

> you might also want to take a look at scgi
> http://www.mems-exchange.org/software/scgi/
>
> seems to have a lot of the advantages of fast CGI
> but it’s simpler.

Yeah, I know it. It isn’t supported in the main stream just because it doesn’t seem to be very useful. I mean, at this moment, only a few Python guys are supporting that approach. Anyway, as you said, it is really easy to implement, so if it gets more popular we can write it down quickly.

Probably time now, as interest has moved way beyond just a few Python guys now, even within the Python community alone; and now Rails has pretty much guaranteed popularity for SCGI. Support for SCGI is on my web server check-list now, I wouldn’t consider adopting a general purpose web server unless it offered SCGI.

PS: QP, a Durus-centric, Quixote-inspired, web framework, made SCGI as one of its design decisions (at this point that means Windows users need not apply).

And then there was one... more.

Yes, another web ‘framework’ for Python. Get over it!

The folks over at mems-exchange released yesterday two new packages, QP and Qpy. Given the on-going discussion regarding Python web frameworks, the release of QP is bound to re-ignite the too many frameworks debate once again. I happen to disagree with that notion myself – diversity is good and good work done by different people is bound to move the whole state of the art forward. It may be that Quixote’s installed base was already big enough to slow innovation; I wonder what those behind QP have in mind for future iterations?

From the QP package home page

This is QP, a package for defining and running multiple web applications based on Durus for persistence, standard persistent Session and User classes, easy interactive database sessions, qpy for assembling html, and Quixote2-style forms and path traversal. QP makes it easier than ever to use these tools together.

From the Qpy package home page and README

Qpy provides a convenient mechanism for generating safely-quoted html text from python code. It does this by implementing a quoted-string data type and a modification of the python compiler. (This main idea comes from Quixote’s htmltext/PTL.)

The quoted-string class is named “h8”. The h8 class is a subclass of unicode, and similarly represents sequences of characters. The main distinction is that h8 instances represent strings that need no further quoting for use in html documents. When an h8 instance is combined with an ordinary string with an operator like ”+” or ”%”, an html-quoting function is applied to make sure that the result is an h8 instance for which no more quoting is needed.

Now that’s exciting. I’ve been converting a Quixote application over to use nothing but unicode internally and I must say its been more work than I expected. Anything that aids in writing applications that can be expected to behave safely with unicode is a plus for me.

First blush: once installed, getting a new application outline up and running, with its persistent data store (including sessions and user data stores), is dirt simple. After reading the QP source I can see that any Quixote-familiar developer will have an easy time adopting the framework, although imports and module names are quite different so some period of familiarization will be required.

QP’s focus on Durus as a data store suggests developers who have to write for different storage back ends may prefer to continue down the straight Quixote path, although for the additional Unicode support alone, I am tempted myself to look at a mixed approach – take QP’s Durus-centric view for users and sessions and anything else that suits Durus well, and use SQL stores only when it makes sense or is required, much like Zope does.

SimpleTemplate... Yet Another Python Template

SimpleTemplate is not a package or product of any sort, just a simple little recipe for doing string substitution in simple text templates, while still taking advantage of Quixote’s PTL: Python Template Language, the un-templating template language.

`_escape_namespace` returns a copy of the mapping supplied to format_text, automatically escaping all values supplied to the template, unless they’ve already been explicitly declared safe using Quixote’s ‘htmltext’ class, or have already been ‘htmlescaped’.

Its simple, perhaps too simple, but seems useful for injecting ‘chunks’ of text or html into placeholder-style templates.

Memories of Visi/SuperCalc

Dave’s drawing analogies between Google News Reader and ? (one supposes Radio) being compariable as to Lotus Jazz (yuk) compared to the forerunner – VisiCalc. Boy do these names draw some memories back from the dungeon.

Anyone remember Framework? Text based ‘windows’ multifunction software, and surprisingly useful as I recall.

Excel (and Word) certainly did pave the way for Windows use, but even Wordstar or MS Word for DOS were very functional. I was never much of a 1–2-3 fan – my personal favorite was SuperCalc, which I first discovered on CPM. That OS was a brief fling, not a love affair, terminated once I had a smokin’ 5 megabyte hard drive (yes 5 megabyte!) in an early IBM PC running MS DOS. SuperCalc for many years was a better spreadsheet, building on VisiCalc and taking it a step further. Unfortunately, Lotus were better marketers. Fortunately, SuperCalc was available on DOS at the time too.

Noo much later I ended up working for Computer Associates, who completed the circle by buying Sorcim (VisiCalc’s maker), along with Basic Software Group (ACCPAC) and in later days Nantucket (Clipper) and all sorts of other tools that I had first used as a user (internally within CA), later as functioning cog of the massive acquiring machine of CA.

Now its 25+ years later… where did the time go? My fingers ache, thinking about how many pc (and only briefly a mac), mainframe, vaxen and unix driven keyboards I’ve typed on over the years.

Google

Google released a news reader that supports both RSS and Atom. Reaction is biased in some quarters:

[Scripting News] Google’s news reader is an awkward slow, hard to use piece of software, like all news readers. Nicely done though, as if that matters. I’m afraid most people will think This is RSS? and give up on it. They really need to check out River of News. Download a copy of Radio, the 30-day trial is free (as if Google couldn’t afford the $39) and just try it out already. No patents. Steal from the best, it’s respectful.Dave Winer

I agree with Dave that a long stream of articles, either in full form or snippets, is a very efficient way of keeping up with news. There are plenty of examples out there of this, not just Radio though… Rawdog is what I use; the various ‘planet’ feeds (planet python) follow this style as well.

Its likely Google knows how to add this functionality… after all, just spitting out everything is the simplest functionality to deliver:

[for newsitem in newsitems output_html(newsitem)]

…so one can only assume they either a) don’t use it themselves and thus aren’t clued in (unlikely), or b) have business reasons for not spitting out everything in a users feed stream (more likely).

The simplest reason I can think of is that an unpaged, full feed, stream of RSS-aggregator-driven HTML will be… big! Mine is 287k at present, varies up and down from there.

Contrast that to what Google serves up as their core business offering, a search result page—these tend to run around 4 – 6K per page.

Did Google pass on the River of News functionality because they are inept programmers? Clearly not. Bad designers? Possible, but not likely.

To get to the answer, we have to first look at what motivates them to offer the functionality, for free, in the first place. Clearly its the value of the data lying within user subscriptions to RSS; this rich web gives them more insight as to the interconnectedness of the web – what and who is hot and where – and thus can allow them to refine their core product, the index, further. That’s the prime mission—delivering the best index in the world.

They don’t need to offer the worlds best feed aggregator to gain that benefit from their users, just a useful one, and one that doesn’t break the bank in the process.

Could there be a valid business reason for not dishing out 200 – 500K per page refresh, to millions if not hundreds of millions of potential users, for free?

I think so.

Dumbed Down, Tarted Up

It was Dan Rather who stated recently that television news has been “dumbed down and tarted up”. I could not agree more. Its one reason why, in Canada, the few bright lights on television, which includes the CBC news gathering and reporting crew, needs to be preserved.

Clearly, the purpose of television news is no longer to inform the American people or serve the public interest. It is to “glue eyeballs to the screen” in order to build ratings and sell advertising. If you have any doubt, just look at what’s on: The Robert Blake trial. The Laci Peterson tragedy. The Michael Jackson trial. The Runaway Bride. The search in Aruba. The latest twist in various celebrity couplings, and on and on and on. Al Gore Full text of speech >

Web Framework Popularity

Benji asked in Web Framework Popularity Contest

Can anyone tell me why they don’t like/don’t know about/are afraid of Zope 3?

I was one of the commenters on his blog and said: “As a former systems integrator working with high end document, workflow, records and content management systems, I can appreciate the careful thought that went into Zope3.

As a developer who now needs to do things quicker, cheaper and on less expensive hardware (often), I find Zope still has too much overhead, both from a learning curve perspective as well as from a performance pespective.

For example, your Hello World quick start:
http://www.benjiyork.com/quick_start.txt

Contrast that with a single file hello world example in any other Python web framework, I’ll use Quixote but they can all be made to look similar I’d expect – compare and contrast the Zope quickstart to the Quixote quick finish: Quixote mini demo

Note: I am NOT saying that other web frameworks are offering anything near functionally equivalent to Zope, not at all, but all that functionality has some costs.

I keep wandering back to have a look at Zope with an open mind, but still have been unable to justify adopting Z3, although at least its much cleaner now (but still obtuse enough to defy casual understanding).

When I weigh the trade-offs, I keep sticking with ‘simpler’. Clearly there are benefits to having internationalization ‘for free’ in your web framework; in Quixote I have to sort this out on my own. But that effort was less than what was required to adopt Z3, so…

Perhaps the lack of a fairly comprehensive, extensible, pre-built, CMS in Zope3 (correct me if my impression is wrong) is hindering broader adoption.

I’d like to think that if I had something like CMF only better (please dont ask me to define this right now) there for the taking and simple extending, then I might be willing to overlook Zope performance per page view down in the single or low double digits per second.

A quick comment on performance – pages per second is not the only yardstick I use, far from it, but the higher it is, the more leeway a system designer has with the tools at hand. Clearly some intended application usage would find even 1 page per second more than adequate; but for many public facing sites, Z2/3 performance means taking other steps – caches, static site renderers, whatever, that only increases complexity.

Back in the old days we made a lot of $ selling customizations on top of a monolithic ‘black box’ Windows (and later web) client-server enterprise DM system; our success with the software and our client’s success with the solutions was due to one factor – the CMS designers, by accident or by purpose, had made it possible to simply and quickly extend what they provided, often in ways completely unintended. That the CMS used bog-standard SQL backends certainly made that task easier too.

I started looking at Zope a number of years ago, thinking that an open source CMS/app server with a lot of thought behind it could be very useful in the enterprise DM (doc management as opposed to Content Management) space – but by and large I think I’d be fighting performance as a critical issue. Many of my old clients had managed document collections measuring in the millions of documents, with tens of thousands of active workflow instances live at any given time.

Integration with legacy systems… even to this day… also tends to be more SQL than XML although now some of the web/xml solutions we and others built are now old enough to be called “legacy” systems ;-). Zope offered a path here; but I mention it because integration [with legacy systems] was always, always, a big issue.

Leaving off with something positive, I see no real downside to all the interest in multiple web frameworks… clearly there is plenty of python activity out there, which can only be good for the community as a whole.”

Object Publishing on the Web - Quixote

From a user's perspective This weekend it seems to be de rigueur to comment on web object publishing, inspired by Ben Bangert's article on 'best of breed' controllers for MVC web frameworks.

A long discussion about Zope and its ancestors ensued in the comments - Quixote borrows the key idea from Zope - simple object publishing via the web, and, in my opinion, succeeds because it kept it simple.

I'm taking a very quick stab at explaining how the Quixote web application development framework hangs together, from a mere users perspective (me) rather than a web framework developers perspective.

There are a number of other Quixote tutorials and documentation out there; in this effort I'm aiming to (but may miss completely) explain some Quixote basics, and perhaps impact enough information such that a prospective web framework shopper will themselves be able to put Quixote in context with some of the other web frameworks.

Knowing me, very quick will soon turn into long and convoluted, although as kids soccer is fast approaching on the clock, perhaps the time constraint will work to good effect. Nevertheless, expect revisions!

On Groovie, Ben takes a look at Best of breed Controllers for MVC web frameworks - its a worthwhile read that dives into some of the differences between the currently hot names, although I am surprised not to at least see a mention of Quixote.

I don't believe the good folks that wrote Quixote and its related tools are much into the self-promotion (some articles), but they certainly have written some fine web development software and it deserves a little talking up now and then. Perhaps other Quixote users like myself will jump into the fray.

Now at version 2.2, Quixote continues to evolve and progress, off a very fine base. Its in production use across a wide array of web platforms ranging from simple Python servers, to python Medusa, mod_python, AOL server, Apache, lighttpd, twisted and no doubt more. I use Apache (but perhaps soon lighttpd) in conjunction with scgi, which is a replacement for FastCGI that works particularly well. lighttpd supports scgi natively now too.

Carrying on from Ben's article, I'm at a loss to plug Quixote's object publishing method into his categories. In Quixote, objects are accessible according to URL (Ben calls this implicit mapping) but the application developer must take this into account when designing the UI (Ben calls this programmatic mapping). What app doesn't, I wonder?

I suspect most frameworks that allow something like http://someserver.net/calendar/new resolve to calling the new() method of a class called calendar use some sort of rule set (whether its by convention or explicit) to determine what gets called where.

Quixote's design allows your UI to naturally follow the url from root to end:

/ (root) will call
    calendar which will have one or more methods such as
        _index
        new
        edit
        delete

In reality, your typical Quixote application will have object classes and separate, but related, object UI classes, looking rather like this:

 / (root UI class) will call
    CalendarUI which will have one or more methods such as
        _index
        new         ... each of which will act upon a Calendar object)
        edit        ... and perhaps this also dives deeper and... deeper...
            CalendarEditUI
                add_resource
                book_resource
        delete      ...

We'll look later at how the hierarchy of UI classes resolves as you drive down, but lets first start off with a basic project skeleton - there is nothing written in stone in this regard; perhaps it might be useful if the Quixote package included a project skeleton builder.

Quixote encourages, but does not enforce, separation between objects and UI. A typical project might look like this:

myapp/
    bin/
        runapp.py      (this is your executable that runs the app)
    doc/
    obj/
        calendar.py
        contact.py
        resource.py
    ui/
        calendar.ptl*  (UI classes for object)
        contact.ptl    (UI classes for object)
        home.ptl       (perhaps an index class or function for home page
                        content)
        publisher.py   (typically a subclass of Publisher for your app)
        resource.ptl   (UI classes for object)
        root.ptl       (or call it driver.ptl or what turns your crank:
                        this defines the root web interface from which
                        all things flow. Q-fans might call theirs qslash.ptl)
        util.ptl       (UI helper classes and functions used elsewhere)
    __init__.py
    date.py            (various helper bits used by both obj and ui)
    utils.py

*don't worry about these ptl extensions for now. They more or less contain straight python code, no need to learn anything new here... move along.

We shall have a look at a Quixote application from the ground (from the driver or 'root' interface) on up (or down depending on how you view these things. Actually we'll start one level below the driver - every driver needs a car (or better yet, a bicycle!) - and in this case our car is a Publisher, which is the interface between your web server and your web application.

I'll use a simple python-based HTTP server for my example, but this is principally the same whether one uses Apache, Medusa, lighttpd, or AOL server, only the app to server interfaces change. In this case my app script is running both the web server (simple_server) and the application itself (create_publisher). bin/runapp.py:

from typicalapp.publisher import create_publisher
if __name__ == '__main__':
    from quixote.server.simple_server import run
    print 'creating demo listening on http://localhost:8080/'
    run(create_publisher, host='localhost', port=8080)

Ok, that's pretty straight forward. Now on to the heart of the matter, publisher.py:

from quixote.publish import Publisher
from typicalapp.ui.root import RootDirectory

def create_publisher():
    return Publisher(RootDirectory(),
                     display_exceptions='plain')

Of course a real application might subclass Publisher to do all sorts of other things, including registering a session handler, filtering output through HTMLTidy, custom logging, whatever...

Next, what are we publishing? Why, RootDirectory of course - from there all things flow. First, here's prototypical the web hello, world example. We are going to look at two different examples of this and explain why there are two different ways of pumping HTML out from your Quixote application. First - the base case:

class RootDirectory(Directory):

    _q_exports = ['', 'hello']

    def _q_index(self):
        return '''<html>
                    <body>Welcome to the Quixote demo.  Here is a
                    <a href="hello">link</a>.
                    </body>
                  </html>
                '''

    def hello(self):
        return '<html><body>Hello world!</body></html>'

    def goodbye(self):
        return '<html><body>Cheers! Salut!</body></html>'

Pretty straightforward, no?

Hey, guess what, there really is no step one. You don't have to design some complicated regex just to get to the root, or to any object no matter how complex your web applications url hierarchy may get to. To the regex challenged, this might be reason enough to consider Quixote!

The URL your client/user is resolved from start towards its end, from 'root' on down. This a call ro http://sometypicalserver.net/ will have Quixote dish up whatever has been defined as its root UI. In most cases this will be a class with one or more methods.

Ok, there are two things happening here that are not immediately obvious - what are _q_index and _q_exports? Before going there, let me interject: If there is one basic principle in Quixote its *there shall be no magic*. To a large extent this is true, there is no magic, however it wouldn't be a framework without a few basic rules and conventions.

_q_exports is a list containing all methods exposed by the class to the web. A '' name in exports allows the _q_index function to be exposed as a callable. Extra work? Perhaps, but the rule is be explicit in Quixote-land. This is a basic defense against exposing things that you don't want to, forcing the developer to think about what will be shown the world. This list can be dynamic of course, but more on that at another time.

When Quixote gets to the requested object along the url requested, it expects to find a callable which returns a result; in our example about the callable _q_index() is returning some simple html.

If the client had requested http://sometypicalserver.net/hello/, Quixote would resolve or dispatch the request to the hello method of RootDirectory. Had the client requested http://sometypicalserver.net/goodbye/, Quixote will return a 404 error, as the goodbye method was not listed in _q_exports. Clear as mud? Excellent, lets press on.

Wait! Before going on, lets look at one minor tweak to RootDirectory and introduce the concept of Quixote's Python Template Language (PTL), which is to traditional templating languages as the un-Cola is to regular colas. It spits out html, but it doesn't look like the other stuff you drink.

PTL is a hook to Python that allows you to write Python functions that integrate textual output more easily. An example will help. First, we have to enable PTL in our publisher script before PTL modules or functions can be called. That's easy, just add to runnapp.py:

from quixote import enable_ptl
enable_ptl()

enable_ptl() is called before create_pubilsher is called. Now we can re-write our prototypical hello, world RootDirectory class as such:

class RootDirectory(Directory):

    _q_exports = ['', 'hello']

    def _q_index [html](self):
        '''
        <html>
            <body>Welcome to the Quixote demo.  Here is a
            <a href="hello">link</a>.
            </body>
        </html>
        '''

    def hello [html](self):
        '''
        <html>
            <body>
            <ol>
        '''
        for n in range(1,101):
            '<li>Hello world, %d times!</li>\n' % n
        '''</ol>
            </body>
        </html>
        '''
# *Note*: you don't HAVE to use ptl extensions for all UI functions...
    def goodbye(self):
        return '<html><body>Cheers! Salut!</body></html>'

Look closely or you'll miss the difference, its pretty slight.

Doesn't look like much of a change, does it? On the surface, not much has changed other than the introduction of a [html] decorator-like extension to python, and the ability to intersperse text in among python code. This latter ability becomes very handy when your "template" is computationally complex, and allows you to lever what you already know - Python - rather than struggle with learning a new templating language and possibly force-fit your needs into whatever constraints the template language demands.

You do not need to use PTL! Many Quixote developers use other templating packages including Cheetah, Kid, or their own. Andrew Kuchling once described PTL in a whitepaper:

Without us ever realizing this while designing it, PTL turns the idea of HTML templating on its head: the default is program code, and there's an escape sequence into HTML. The escape sequence is just Python's notation for string literals, which is a compact and easily readable notation. Functions that actually contain HTML therefore can look a little messy, but most HTML only appears e lowest-level functions; the bulk of our PTL code simply calls other PTL templates and inserts the odd <br> or <table> here and there.

Greg Ward also discussed PTL in a comprehensive article on Quixote in Linux Journal. Other links to articles on Quixote can be found on the project web site.

PTL is not web designer friendly, but it is very programmer friendly. I'm not a web designer but have created some half decent xhtml/css based designs - my workflow is to prototype the design in straight xthml and css, and then decide how I want to chunk same up within my application. I don't find this to be problematic for the overall design of a site or web application.

Bits and pieces within the application I rarely find the need to prototype in xhtml first; generally I find I spend more time refactoring ptl code into smaller reusable widgets. Your mileage may vary, but for myself, I find it now hurts my eyes to look at html peppered with %this%, %some for loop that%. I'd rather look at plain python. Oh, by the way, vim has a ptl syntax add-on too.

PTL also delivers other benefits, chiefly automatic escaping of strings that are not explicitly safe. Read the project documentation on PTL for further information.

In installment two we'll look at how Quixote traverses a hierarchy of UI objects.