mike watkins dot ca : Ryan's Questions

Ryan's Questions

Ryan asks a question about the test situation used to come up with the numbers I published this week. I’m glad someone poked at this issue to give me the motivation to dig deeper, as I am also keen to avoid a flame-fest.

(Update: Do also check out Taking Textile Out for more on the comparison)

So I dug deeper, and to make a long story short, here is what I’ve found: performance is much closer on an apples to apples comparison basis than my first blush at midnight look suggested, however, there are still some outstanding questions which suggest looking at performance more closely would be time well spent if someone is making a toolkit and language decision.

Quixote still appears to edge ahead, and here’s the scoop:

What was wrong

I took a quick look into Instiki’s code and from my unpracticed in Ruby eye, it appears that content is stored in the native markup format and rendered at display time – in this case, the default format is Textile.

My application does the same, but stores a pre-rendered (optionally) version of the content which is served up. I’d left this optional feature on, by default, and did not factor the performance of rendering Textile or Markdown formatted content into HTML in my last look at this.

Naturally this makes a big difference in performance as you might expect, dropping from 240 some odd requests per second to 25 to 175 requests per second in my application depending on the scenario – more on this later.

The original intent for my ab exploits was to perform a quick benchmark on my solution, not throw any stones elsewhere. Mea culpa for drawing inappropriate conclusions, and thanks Ryan for giving me the motivation to dig deeper, although as you’ll find out, there are still unanswered questions even after having dug deeper.

What else have I learned?

Instiki isn’t using Rails per se. At the time I did the test I was not aware that the current version of Instiki does not use Rails per se, rather, it seems that in writing instiki, the Rails web application framework was inspired and undoubtedly refactored into a general purpose application framework.

FYI the group supporting Instiki seems to be moving current parts of Rails back into the code with a goal to reaching a version 2 where Instiki is a model web app for Rails itself.

Updated performance comparison. Here’s the environment and application scenario:

  • Ruby/ Instiki versions: Ruby 1.8.2, recently built; Instiki 0.9.2
  • Python / Quixote versions: 2.4 / Quixote 2.0a4

Application – Instiki:

  • Created a new Instiki instance
  • Backend is a file system; server is Ruby’s WEBrick
  • Ran Instiki as a daemon, the default; checking the command line options using—help I did not see any option available which speaks to performanc tweaks
  • Loaded content into two wiki pages. The first contains Textile formatted content; the second contains only the html entity for non-breakable space.
  • The path to the page, being a wiki, is simple and flat; url looks like http://localhost:2500/wiki/show/SomePage
  • Test – 100 retrievals: ab -n100 http://localhost:2500/wiki/show/SomePage

Application – Quixote-based content management solution with a Wiki component

  • Backend is a Durus object database accessed over the network but app and Durus are on the same machine for this test.
  • Web server is a pure Python server
  • Same content and scenario as the corresponding Instiki pages
  • Disabled pre-rendered storage
  • The url path is somewhat more complex, having to do more than one lookup to get to the end point object, as my solution provides for multiple wikis. I could map mywiki such that it got called directly off the root of the application but for comparisons sake it would not make a big difference. URL http://localhost:8080/wikis/mywiki/SomePage

Result, averaged over three runs:

  • Instiki/Ruby/Rails pre-cursor:
    • With content:9.5 pages / second;
    • Without content: 18 pages / second
  • Quixote/Python/Dulcinea + Durus based app:
    • With content:54 pages / second;
    • Without content: 147 pages / second

Still a substantial difference in performance. Perhaps I’ll dig through Instiki’s code some more and cut out the Textile rendering to get a better comparison without that variable in the mix. And I will speak to Durus vs SQL at a later date. I’m actually an old (older than I wished!) SQL hack and have no misconceptions about either.