Skip to main content

Python: Django: SQL Queries in the Templates

I think Django is a great framework for Python Web newbies. Those guys have done a great job providing a cohesive framework that's easy to get started with. However, one thing that has always bothered me is the templating system.

I was reading a bit of someone else's Django code, and I came across this in a template:
<div>
<div>Services:</div>
<ul>
{% for service in server.services.all %}
<li>{{ service.name }}</li>
{% endfor %}
</ul>
</div>
There are two problems. First of all, "service.name" is not being HTML escaped. That means you have to worry about XSS vulnerabilities. Thankfully, I no longer need to worry so much about things like that because Genshi can auto-escape things smartly.

However, there's a much graver problem. "server.services.all" is really a SQL query.

Django tries really hard to provide a restricted templating environment because it assumes template writers aren't programmers. This assumption drives me nuts because I do both the back end and the front end. A naive template writer could easily write things like "server.services.all" in an inner loop, such as listing all the services of all the servers. The multitude of small queries could wreck havoc on the database if there were a ton of servers. A smart programmer could replace all those queries with a single query done ahead of time.

My point is that Django's restricted templating system handcuffs me and prevents me from easily writing functions to avoid duplicating markup while at the same time leaving enough rope for naive template writers to hang themselves.

I'm so much happier with Genshi. A smart programmer + a smart templating system = less code duplication. I might be a perfectionist with a deadline, but I also have to maintain my code, so DRY still applies.

Comments

Yes, I know you can use different templating systems with Django. However, you sacrifice a lot when you start doing things the non-Django way.

Yes, I know Genshi is slower. I personally couldn't care less because it makes *me* a lot faster.

Adrian, Jacob, Simon, please don't be too offended. I still think you guys are talented engineers even if I disagree with you on simple matters of taste such as this ;)
Anonymous said…
Two quickies,

1) Django (trunk, and future releases) now auto-escapes everything by default. So you actually have to go out of your way to write out un-escaped data.

2) Everything is very loosely coupled, if you like the URL matching, ORM, or other parts you can easily import your favorite template engine and use it as you normally would inside of a Django view.
Anonymous said…
Ah, you beat me. :)

Do you really "sacrifice a lot"?
Anonymous said…
Shannon:

After reading this post it is clear that you have no idea how Django templates work.

I think you read some template code and assumed you knew what it did.

You were wrong.
Adrian Holovaty said…
Hi Shannon,

As an anonymous commenter noted, the template system in Django's development version ("trunk") auto-escapes everything. This is a *huge* protection against XSS vulnerabilities, as you know from using Genshi.

Also, regarding your second point, I'm not sure what you're trying to argue. Yes, the "server.services.all" variable is a database query, but wouldn't this same construct be possible using any other template system that allows attribute access, such as Genshi?

It seems that the main thing you're arguing is: "Designers shouldn't be given this much control in templates, because they might put an SQL query in an inner loop." That's not an anti-Django argument, that's an anti-designer-control argument, and it applies to *any* template system.
> After reading this post it is clear that you have no idea how Django templates work.

> I think you read some template code and assumed you knew what it did.

> You were wrong.

Boy, you can be condescending! I've actually written applications in Django. Check out some of my other posts on Django.
> As an anonymous commenter noted, the template system in Django's development version ("trunk") auto-escapes everything. This is a *huge* protection against XSS vulnerabilities, as you know from using Genshi.

Very good! I stand corrected.

> Also, regarding your second point, I'm not sure what you're trying to argue. Yes, the "server.services.all" variable is a database query, but wouldn't this same construct be possible using any other template system that allows attribute access, such as Genshi?

I'm saying that for all the restrictions Django enforces, it still doesn't prevent the template writer from doing things he really ought not to. In Genshi, it's different. The templating engine doesn't enforce very many restrictions. In fact, it allows the developer to write functions, etc. The template writer is assumed to be of a higher caliber. I'm arguing against the idea that you can have dumb template authors without paying heavy consequences.

> It seems that the main thing you're arguing is: "Designers shouldn't be given this much control in templates, because they might put an SQL query in an inner loop." That's not an anti-Django argument, that's an anti-designer-control argument, and it applies to *any* template system.

I'm saying that Django doesn't prevent dumb template authors from hanging themselves, so why bother with the restrictions? Let's stick with a templating system that does let good programmers get creative ;)

These include functions and clever match templates.
Adrian Holovaty said…
Shannon --

Ah, so your argument is, "Hire designers who know how to program." :-)

In any case, I don't see how arguing about template systems is productive. Like you said in an above comment, it's a matter of taste.

Happy new year, and may our years be filled with spirited discussion about the Web sites we *make* with Python, rather than the specific Python tools we use. :-)
> Ah, so your argument is, "Hire designers who know how to program." :-)

Yes. Also:

* Remember to apply DRY to templates too.

* Templating engines should come with something like functions in order to respect the above point.

* Making up tag names and then using Genshi match templates (perhaps in the same template) to "interpret" them is a really useful technique ;)

* Genshi and Django templates are both a cut above many other templating engines in that they don't require the engineer to remember to escape everything.

> In any case, I don't see how arguing about template systems is productive. Like you said in an above comment, it's a matter of taste.

Sorry ;) I couldn't resist when I saw that SQL query being executed by the template. It seems to run contrary to the "Django template philosophy".

> Happy new year,

Happy New Year!

> and may our years be filled with spirited discussion about the Web sites we *make* with Python, rather than the specific Python tools we use. :-)

Ah, now you're talking! I'm working on http://www.freebase.com/ which I'll argue is the ultimate data mashup ;)

Thanks for your friendly, level-headed comments :)
Anonymous said…
I have experienced a very similar issue about the restrictive nature of Django templates.
I had a sql query that returned a list a thumbnails of images.
In order to better use the available screen space, I wanted to put 5 thumbnails in a single row.
With the templating system of django this is impossible to do, because it does not have any counter functionality. The only thing I found I could do, is to develop custom tags.
For any templating system with python capabilities (e.g. Mako) this is a no brainer.
Brian Beck said…
Template tags and filters *are* Django's template functions/macros. See especially inclusion tags. It makes so much more sense to me to write functions in Python than in some half-Python HTML file which is then included in all templates that happen to need that function. People act like custom template tags are equivalent to having to monkeypatch Django or something.

And to the anonymous commenter wanting to put 5 thumbnails per row: put this in your for loop...

{% if forloop.counter|divisibleby:5 %} <br/> {% endif %}
James Chua said…
I feel the same way also the first time I use django's template, but that changes as I use Django more and more. You can be creative and productive by creating your own custom template tag and filter.
Chris Pratt said…
Isn't it the mark of creativity to be able to do more with less? If the templating engine limits you, then that forces you to get very creative about your implementation.

Nevertheless, I see the argument. If there's even a little room to wreak havoc, then restricting it at all is rather pointless. However, it should be noted that another purpose of Django's templating system's restrictions to discourage the PHP-style code/html mashups, so whether or not it's designer-capable, it still serves that purpose.
> It makes so much more sense to me to write functions in Python than in some half-Python HTML file which is then included in all templates that happen to need that function.

My main aversion to Django's template tags is that I have to parse the arguments myself. Maybe this has changed. I haven't tried them in a year or so. I really like Python's syntax where I can call a function like "f('a', b=2)".

Furthermore, I'm not against writing a function in Genshi for outputting markup in Genshi. I wouldn't want to embed all that markup in Python.
> Isn't it the mark of creativity to be able to do more with less? If the templating engine limits you, then that forces you to get very creative about your implementation.

Ah, yes, but it can get painful. For instance, I blogged about the time I needed recursion in a template:

http://jjinux.blogspot.com/2006/02/python-recursion-in-django-templates.html

> Nevertheless, I see the argument. If there's even a little room to wreak havoc, then restricting it at all is rather pointless.

Thank you for your understanding ;)

> However, it should be noted that another purpose of Django's templating system's restrictions to discourage the PHP-style code/html mashups, so whether or not it's designer-capable, it still serves that purpose.

Point granted.
I just can't believe that after all these years, people are still discussing how to do templates properly. Its like history just keeps repeating itself every time a newbie figures out that the internets can be programmed (ie: Rails).
By the way, I'm sure you have heard of this one, but it has been updated. Pretty funny.

http://www.zedshaw.com/rants/rails_is_a_ghetto.html
> I just can't believe that after all these years, people are still discussing how to do templates properly.

The problem is that you can successfully use several different approaches. It boils down to engineering taste.

> http://www.zedshaw.com/rants/rails_is_a_ghetto.html

Yep.
Noah Gift said…
Shannon,
I am going to have to side with you on this. I have really big Django project I am finishing up and the last view I need to write involves nested regions of the world.

I had a pretty good time up until I got here. Now in order to do something as simple as recursion in the template, I am going to have to resort to the technique you suggest in the earlier post. Yes, I have to write the front and the back so this sucks for me.

As to Adrian's comments, "and may our years be filled with spirited discussion about the Web sites we *make* with Python, rather than the specific Python tools we use. :-)" I guess I don't agree. If you say your framework is for "perfectionists on a deadline", you are in fact creating a spirited discussion about Python tools in advance. No offense, but that is faulty logic :)
analoguemike said…
"server.services.all" is really a SQL query.

True. However, you don't have to pass the 'server' object to the template. If you're worried about the designer hanging themselves with a query like server.services.all then you shouldn't pass the server object in the first place. You could instead pass a simple list named 'all_services' where you have taken the responsibility of calculating the list of services in the most efficient way. Also, {% for service in all_services %} would also be much easier for the designer to understand.

There should be comments at the top of the template explaining what variables are available in the template. In this case the only variable available would be 'all_services'.

It's also more flexible. What the architecture changes and all_services needs to get it's data from a web service rather than a database call. In this case you wouldn't need to change the template code.
Shannon Behrens said…
analoguemike, what you say is true. However, I was trying to point out that Django templates are not like some templating systems where it is literally impossible for the template to do things like query the database. In some templating systems, the template renderer is given a template and plain, serialized data and it combines the two. In such systems, the template is not able to do anything other than use the pure data provided.

To be fair, I don't prefer such templating systems. I prefer templating systems that allow me to use normal programming language expressions. For instance, I'm perfectly happy with ERB in Ruby on Rails.

Naturally, templating engines are a matter of engineering taste. Smart people can still disagree.
Jon Stevens said…
I'm doing less and less templating on the server side and more and more on the client with HandlebarsJS. Processing streams of json allows for more dynamic websites where you don't need full page refreshes to load new data.