Tuesday, February 28, 2006

Python: Django Custom Tags

In my last post, I complained that code like the following is redundant:
<tr class="fieldrow">
<th><label for="id_subject">Subject:</label></th>
{{ form.subject }}<br>
{% if form.subject.errors %}
<div class="error">{{ form.subject.errors|join:", " }}</div>
{% endif %}

<tr class="fieldrow">
<th><label for="id_name">
Poster's Name:
{{ form.name }}<br>
{% if form.name.errors %}
<div class="error">{{ form.name.errors|join:", " }}</div>
{% endif %}

<tr class="fieldrow">
<th><label for="id_email">
Poster's Email:
{{ form.email }}<br>
{% if form.email.errors %}
<div class="error">{{ form.email.errors|join:", " }}</div>
{% endif %}

{{ form.body }}<br>
{% if form.body.errors %}
<div class="error">{{ form.body.errors|join:", " }}</div>
{% endif %}
What I really want is:
{% load formutils %}
{% fieldrow form "subject" %}Subject:{% endfieldrow %}
{% fieldrow form "name" %}Poster's Name:{% endfieldrow %}
{% fieldrow form "email" %}Poster's Email:{% endfieldrow %}
{% fieldrow form "body" %}{% endfieldrow %}
This would allow me to create new forms and form fields quickly. Naturally, there could be other custom tags besides just "fieldrow" for different style layouts. This is just the beginning. Well, I started by creating a custom tag:
"""These are custom tags for creating forms more easily."""

from django.core import template
from django.core.template import Context, loader

register = template.Library()

def fieldrow(parser, token):
"""This tag creates a table row with a form field.

Example: {% fieldrow form "name" %}Name:{% endfieldrow %}

form -- This is a formfields.FormWrapper object.
name -- This is the name of the field. It may be a variable or a quoted
string. If it is a quoted string, it may not contain spaces.
Name: -- This is the label for the field.

Note, I'll use <tr class="fieldrow">.

tag_name, form, fieldname = token.contents.split()
except ValueError:
raise template.TemplateSyntaxError(
"%s tag requires two arguments" % token.contents[0])
nodelist = parser.parse(('endfieldrow',))
return FieldRow(form, fieldname, nodelist)

class FieldRow(template.Node):

def __init__(self, form, fieldname, nodelist):
self.form = form
self.fieldname = fieldname
self.nodelist = nodelist

def render(self, context):
form = template.resolve_variable(self.form, context)
fieldname = resolve_variable_or_string(self.fieldname, context)
field = form[fieldname]
label = self.nodelist.render(context)
t = loader.get_template('bulletin/fieldrow')
c = Context({"form": form, "fieldname": fieldname,
"field": field, "label": label})
return t.render(c)

def resolve_variable_or_string(s, context):
"""s may be a variable or a quoted string.

If s is a quoted string, unquote it and return it. If s is a variable,
resolve it and return it.

if not s[0] in ("'", '"'):
return template.resolve_variable(s, context)
if s[-1] == s[0]:
s = s[:-1] # Strip trailing quote, if any.
s = s[1:] # Strip starting quote.
return s
This made use of a template:
{% comment %}
This template is used by the fieldrow custom tag to create a table row with
a form field.
{% endcomment %}

<tr class="fieldrow">
<th><label for="id_{{ fieldname }}">
{{ label }}
{{ field }}<br>
{% if field.errors %}
<div class="error">{{ field.errors|join:", " }}</div>
{% endif %}
Viola! I'm done. It works.

Now, without a doubt, writing that custom tag is a pain. The hard part is that I have to parse the contents of the custom tag, resolve variables, etc. It sure would be nice if the templating system did that stuff for me. I personally think it would be much nicer if I could simply create a block that takes arguments. Consider something like:
{% block fieldrow(form, fieldname, field) %}
<tr class="fieldrow">
<th><label for="id_{{ fieldname }}">
{{ yield }}
{{ field }}<br>
{% if field.errors %}
<div class="error">{{ field.errors|join:", " }}</div>
{% endif %}
{% endblock %}

{% fieldrow(form, "name", form.name) %}Subject's Name:{% endfieldrow %}
Sure, a template author might not be able to write that "fieldrow" block. The template authors I work with do understand code like this. If nothing else, a programmer could create a separate template containing that block. He'd be able to avoid duplicating code with very little work. Having the templating system handle parsing arguments and resolving variables would make this task much easier than creating a custom tag.

Monday, February 27, 2006

Python: Django-palooza

I think the Django guys have written some really nice software. It's good stuff. No, it won't cure cancer; it's not a CMS; it's not a shopping cart; but it does what it does well.

In fact, I must admit that I felt really at home using Django. Django + WSGI reminds me a lot of how Aquarium works. In many subtle places, the designs are very similar. There are some things I feel Aquarium does better, but I can readily admit there are some things that Django does better. Despite the fact that I too am a perfectionist, I think Django may be more polished--apparently, one perfectionist just can't code as fast as two perfectionists ;) Actually, I wish it was closer to meeting my needs (stackless Python, coroutines, a custom data store instead of an RDBMS, etc.), but I can appreciate it for what it is.

Since Django can't possibly meet everyone's needs--for instance, it doesn't make a good replacement for Linux or Vim (yes, that's a joke)--let me state some of the things that bothered me most about Django while I was using it to write a small application like this one.
  1. In my application, end users have a form that allows them to create new messages. I created a new message that had <script>alert('hi!')</script> for its subject. My code correctly escaped things. However, in the admin interface, I got an alert "hi" when I went to update the message. That means that you cannot safely use the admin interface to look at end user generated records. Otherwise, end users might inject JavaScript that causes the admin user's browser to "do bad things."

  2. In the templating language, if you use a variable that doesn't exist, it will be silently replaced by "". This is like Perl, and I really dislike it. I'd much rather have an appfault so that I can fix the problem.

  3. Certain filters take arguments. A filter argument looks like this: {{ bio|truncatewords:"30" }}. I personally don't like this syntax, but I understand that that's just a matter of taste.

  4. You can't treat templates like functions. "include" is not a valid replacement for functions because you can't pass arguments. That means that you can't call it twice passing different arguments (don't even think about trying recursion). I think that functions are the most basic tool for applying DRY (don't repeat yourself), and I think that DRY applies to HTML too! Redundant code is bad, period. Even if the guy getting paid to update it is getting paid less than a programmer, it still takes a long time to make changes to redundant code, and it's an error-prone process.

  5. Nor are custom tags a good replacement for functions. First of all, template authors don't have access to write them. Secondly, consider the following custom tag call: {{ create_form_field email "Please enter your email:" }} Did you know that it's up to the coder to parse that? The custom tag infrastructure can't even break that call into two arguments, let alone take care of unquoting the string. It doesn't even resolve the variables within the caller's context--you have to do that explicitly. Consider an exmaple straight out of the Django docs:
    <form method="post" action=".">
    <label for="id_name">Name:</label> {{ form.name }}
    {% if form.name.errors %}*** {{ form.name.errors|join:", " }}{% endif %}
    <label for="id_address">Address:</label> {{ form.address }}
    {% if form.address.errors %}*** {{ form.address.errors|join:", " }}{% endif %}
    <label for="id_city">City:</label> {{ form.city }}
    {% if form.city.errors %}*** {{ form.city.errors|join:", " }}{% endif %}
    <label for="id_state">State:</label> {{ form.state }}
    {% if form.state.errors %}*** {{ form.state.errors|join:", " }}{% endif %}
    <label for="id_zip_code">Zip:</label> {{ form.zip_code }}
    {% if form.zip_code.errors %}*** {{ form.zip_code.errors|join:", " }}{% endif %}
    <label for="id_place_type">Place type:</label> {{ form.place_type }}
    {% if form.place_type.errors %}*** {{ form.place_type.errors|join:", " }}{% endif %}
    <input type="submit" />
    Perhaps you'll disagree with me, but I think that code is verbose and redundant. I think the template author should have a way of factoring out the common bits (what a row looks like), but neither includes nor custom tags are suitable replacements. I'm going to make due with a custom tag, but I'm not happy about it.

  6. I like the urls.py stuff, but when you generate URLs, the structure of those URLs is hardcoded. That is, urls.py is used to parse URLs, but not to create them. If I suddenly want to change how my URLs work, I can't just go to urls.py and change it. I also have to change all the places where the URLs are generated.

  7. Blocks are not suitable replacements for functions either. For instance, you can't just call a block. You can only override a parent template's block. My parent template had a "title" block that the child templates were overriding. I wanted to call the "title" block somewhere else in the child template, but I couldn't. I felt like my hands were chained together.

  8. The builtin media server is only meant for development. I can respect that. It's not supposed to be efficient. However, it doesn't understand the IfModifiedSince header. Hence, media is re-downloaded on every page hit. (By the way, I'd be happy to donate this from Aquarium, but I fear the authors aren't interested in my contributions.)

  9. I see form = formfields.FormWrapper(manipulator, new_data, errors). This is similar to how FormUtil works in Aquarium, however, it would be better if the new_data argument could be a list of dicts, i.e. "Use the user's previous data if present, otherwise use the current value of the data from the database, otherwise use..." Using a list of dicts is a nice bit of flexibility.

  10. Why are the Django guys so rude to other programmers? (You'll note that this blog contains compliments as well as critiques.) Here are some reasons it's bad to be rude to other programmers:

    • Those other programmers are human beings. They have feelings too!

    • I can guarantee Django will one day be obsolete, and younger programmers will say rude things about the authors. It's best if we all just treat each other with respect all the time.

    • Being rude to other programmers turns valuable bug and patch submitters into competitors. That makes it a double no-no.

  11. I wonder why they picked Dojo Toolkit instead of MochiKit. Afterall, the MochiKit guy is definitely a fellow perfectionist, and the Dojo guy most definitely isn't (even though he's a really nice guy). Personally, despite the fact that I'm a perfectionist, I like Dojo because it's very complete (it has stuff that I need that MochiKit doesn't). I can only hope they picked Dojo for this reason. It'd be awful if they picked it simply because MochiKit is used by TurboGears.

  12. If the Django guys are perfectionists, why don't they follow Guido's style guide more closely? That kind of stuff will drive obsessive compulsive perfectionists like me crazy!

  13. Did I mention that it's nearly impossible to apply DRY to templates in Django?
The following are some small points that are fixed in Aquarium, but not in Django. Again, I'd be happy to offer code.
  1. The login cookie isn't being set correctly, because the domain of the cookie sent out by Django doesn't match the domain in your browser.
    Aquarium automatically respects the X_FORWARDED_HOST header.

  2. You have to restart the Web server everytime you make a change under mod_python.

  3. The broken pipe exceptions are irritating. They're easy enough to handle more gracefully.
So anyway, Django guys, if you're reading this, remember that I continue to have a lot of respect for what you've done. In general, it's good stuff.

Wednesday, February 22, 2006

Linux: Ubuntu 5.10 on Dell Inspiron B130

If you read my last post, you may be relieved to hear that I did not set my laptop on fire. Ubuntu is working great! It even supported suspend to disk out of the box! I haven't been able to do this in a good five years. The only mildly tricky thing was that I needed to follow the instructions on this forum in order to get it to support WXGA resolutions (i.e. 1280x800). I'll probably need to use NDIS to get the wireless card to work, but I haven't gotten to that yet.

I guess I should go write a Linux on Laptops page now.

Sunday, February 19, 2006

Linux: Fedora Core 4 on Dell Inspiron B130

Rant warning:

If you're thinking about buying a Dell Inspiron B130 for use with Fedora Core 4, my initial assessment is don't:
  1. If you use the automatic partitioning, you'll get an exception during install. This may be Fedora's fault, though, because I've seen this problem before.

  2. After the installation actually started installing RPMSs, it just froze and died. It may have been display related, but I'm not sure.

  3. I installed in text mode (HTTP install) with the bare minimal number of packages. But then, when it tried to boot, it froze on the line, "Initializing hardware... storage network".

  4. I know that the display is hard to get working, but I've heard it just comes down to getting the right mode line, "Modeline "1280x800" 80.58 1280 1344 1480 1680 800 801 804 827".
Well, I will continue the battle tomorrow. It's too late for me to return this laptop. I'm not a happy camper at this moment. Apparently, very few people have installed Linux on this laptop so far. It doesn't even have its own Linux on Laptops page yet. I might also try Ubuntu because some people have reported success.

By the way, remember how I said Dell agreed to pay me back a small amount of money to offset the Microsoft tax? The money never showed up in my account. If I am stuck running Windows on this thing, I may just set fire to it and let it burn--not that I'm bitter or anything ;)

Tuesday, February 14, 2006

Haskell: Haskell for C Programmers

I'm just starting to read Haskell for C Programmers, and I can already tell that it is very helpful. If only I had seen this four years ago!

Haskell: Tour of the Haskell Syntax

This Tour of the Haskell Syntax is frickin' awesome! It's not enough to learn Haskell, but it sure is helpful to step back and see so much of Haskell explained succinctly.

Thursday, February 09, 2006

Python: Recursion in Django Templates

Whenever learning a new Web technology, I like to write a bulletin board application. It's easy, I have it memorized, and it gives me a chance to compare and contrast the new technology with what I already know. See here for the application written in Aquarium.

The thing that was hardest about using Django templates to write this application was the lack of functions and recursion in the templating engine. Recursion is really suitable for generating a hierarchy of messages. This post is about how I had to work around the problem. I got it to work, but it wasn't what I'd call elegant.

To use Django to generate a page that looks like this, you break it up into a few steps. First, you create a template that has everything other than the hierarchy of messages. I called it "index.html":
{% comment %}
Output the messages, and output a message input form.
{% endcomment %}

{% extends "bulletinboard/vertical" %}

{% block vertical_body %}
<div class="container">
<div style="float: right">[<a href="">Start a New Thread</a>]</div>
<p><b>List of Messages:</b></p>
{% if not messages %}
<p>There are no messages yet.</p>
{% else %}
{{ messages }}
{% endif %}
</div><br />
{% endblock %}
Secondly, you pass the pre-generated hierarchy of messages as a chunk of HTML to the first template. Hence, in my view, I have code:
    return render_to_response('bulletinboard/index', 
{'messages': show_message_hierarchy()})
where "show_message_hierarchy()" generates the big chunk of HTML. Next, I created a template, "showmessagehierarchy.html", that gets called recursively (although, naturally, it can't call itself recursively in Django):
{% comment %}
This template is called recursively to output the bulletin board messages in
a nice hierarchy.
{% endcomment %}

{% for child in children %}
<a href="/messages/{{ child.id }}/">{{ child.subject|escape }}</a>
{{ child.recurse }}
{% endfor %}
Note, the recursion happens with "child.recurse", which is a callback that the view sets up. Each child has a different callback--it's not possible to say "recurse(child)" because there are no function arguments in Django (unless you use a custom tag). "child.recurse" is not a method of "child"; it's just a function that I've attached to "child". Now, let's look at the view that pulls everything together:
def index(request):

"""Put the list of messages in a nice hierarchical format."""

def show_message_hierarchy(parent=None):
"""Show the bulletin board messages in a nice hierarchy."""
children = message_dict.get(parent, [])
if not children:
return ""
c = Context({"children": children})
return template.render(c)

def create_recurse_callback(id):

"""This is a closure.

It curries the id for show_message_hierarchy. The "partial" function
doesn't exist yet in Python 2.4.


def inner():
return show_message_hierarchy(id)

return inner

message_list = messages.get_list()
message_dict = {}
for message in message_list:
message.recurse = create_recurse_callback(message.id)
message_dict[message.parent_id] = message_dict.get(message.parent_id, []) + [message]
template = loader.get_template('bulletinboard/showmessagehierarchy')
return render_to_response('bulletinboard/index',
{'messages': show_message_hierarchy()})
Here you can see "child.recurse", i.e. "message.recurse", getting setup: "message.recurse = create_recurse_callback(message.id)". "create_recurse_callback" is a closure that is used to curry the "message.id" so that you can call "show_message_hierarchy" without needing to pass an argument. I got that trick from function programming ;) Notice that, basically, the "show_message_hierarchy" function and the "showmessagehierarchy" template recursively call each other.

When I did this in Cheetah, I just had the main template, and another function in the template that could call itself recursively. Nonetheless, I was able to get it to work in Django, albeit in a non-elegant fashion.

Python: Django Templates

"As simple as possible, but no simpler."

I have a lot of respect for what the Django guys have done. I actually really like a lot of Django. However, having tried it, I find the templating engine to be elegant, but too simple. What frustrates me most is the lack of functions. In my first programming class, I learned how to use functions for code reuse. DRY applies even to HTML. At my company, even the "template authors" know how to create a function to avoid code duplication. Nor is the "include" or "block" statement a suitable replacement--functions have parameters. Different arguments lead to different results. This is necessary functionality. Also, functions should support recursion. I learned in my second programming class how to think recursively, and every once in a while it really comes in handy (for instance, when you're writing a bulletin board application with a hierarchy of messages).

When I was using Django templates, I really felt like my hands were tied. I don't think lack of functions improves security. I don't think custom tags are always a suitable replacement--I'm talking about simple HTML reuse. Please forgive my boldness, but I don't think I should have to live without functions just because a template writer working at some other company doesn't understand them.

Next, I think I'll try out using Django with Cheetah. So far, Cheetah has always met my needs exceedingly well. Hopefully, I can get it to fit in smoothly, maintaining Django's overall elegant feel.

Wednesday, February 08, 2006

Python: Clearing sys.modules of Stale Modules

I'm posting this here in case someone might find it useful. I also submitted it to the Cookbook.
"""Clear ``sys.modules`` of specific types of modules if one is stale.

BASIC IDEA: Clear ``sys.modules`` of stale code without having to restart your
server. It's a hell of a lot harder to do right then it sounds.


__docformat__ = "restructuredtext"

_lastModuleUpdate = time.time()
def clearModules():
"""Clear ``sys.modules`` of specific types of modules if one is stale.

See ``properties.CLEAR_MODULES``.

I took this method out of the ``InternalLibrary`` class so that you can
call it *really* early, even before you create a ``Context`` to pass to


The problem that this method solves is simple: if I change a file, I don't
want to have to restart the server. It's a simple problem, but it's tough
to implement right. To prevent repeating mistakes, here's what has failed
in the past:

* Remove all modules from ``sys.modules`` on every page load.

- Some modules have state.

* Delete only those modules that don't have state.

- There's no convenient way to know which ones have state.

* Use module attributes.

- It's not convenient.

* Delete only those modules that have grown stale.

- If a parent class gets reloaded, child classes in other modules will
need to get reloaded, but we don't know which modules those classes are

* Look through all the modules for module references to the modules
that'll get deleted and delete those too.

- Its very common to only import the class, not the whole module. Hence,
we still don't know which modules have child classes that need to get

* Just clear out ``sys.modules`` of all modules of certain types on every
page load.

- Even a moderate amount of kiddie clicking will result in exceptions.
I think the browsers hide the problem, but you'll see the exceptions
in the logs.

* Clear out ``sys.modules`` of all modules of certain types on every page
load, but only if at least one of the modules is stale.

- This is good because it handles the kiddie clicking case, and it also
handles the super class case.

global _lastModuleUpdate
if not properties.CLEAR_MODULES:
deleteTheseTypes = properties.CLEAR_MODULES
if not isinstance(deleteTheseTypes, list):
# Update Seamstress Exchange's properties file if you change this.
deleteTheseTypes = ["aquarium.layout","aquarium.navigation",
"aquarium.screen", "aquarium.widget"]
deleteThese = [
for moduleType in deleteTheseTypes
for moduleName in sys.modules.keys()
if (moduleName == moduleType or
moduleName.startswith(moduleType + "."))
for i in deleteThese:
file = sys.modules[i].__file__
except AttributeError:
if file.endswith(".pyc") and os.path.exists(file[:-1]):
file = file[:-1]
if (_lastModuleUpdate < os.stat(file)[ST_MTIME]):
staleModules = True
staleModules = False
if staleModules:
for i in deleteThese: # You can't modify a dictionary
del sys.modules[i] # during an iteration.
_lastModuleUpdate = time.time()

Wednesday, February 01, 2006

Microsoft: A [Very] Small Victory

I recently bought three laptops from Dell. They came with Windows XP Home edition. I'm a passionate Linux guy. Microsoft and Dell have a deal where Dell can only sell computers (perhaps it's just their consumer stuff?) with Windows on them. Furthermore, as part of this deal, Dell cannot reveal how much each copy costs. Dell refused to sell me the laptops with no software. After getting the laptops, I called up customer support to try to arrange a refund on Microsoft licenses I didn't want. I had to insist on talking with a supervisor. He called me back hours later. I pleaded my case. He argued strongly that the software didn't cost me anything, it was part of the package. I argued strongly that somehow Microsoft was getting my money, and I wanted no part in that. I wanted $100 back per laptop. He argued that since the cost of my laptops was so low (they were the cheapest models), he didn't feel that Dell could possibly be spending $50 a piece. After some haggling, I settled for $25 a piece. Other people have gotten more back. Dell paid me money to go away, and I'm sure this had no impact on Microsoft, but I'm at least somewhat content to have caused a stir. I am not happy with Dell's policy.

See also: here and here.