Tuesday, December 26, 2006

Linux: Netgear Open Source Wireless-G Router


I just bought Netgear's Open Source Wireless-G Router, model KWGR614. The setup wizard doesn't work with Firefox. I spent a lot of time fighting with tech support. They told me to use Internet Explorer even though the box says, "Open source code enables router customization for Linux developers and hobbyists". I tried to fix the source myself, but the C code for the admin interface is specifically missing from the source download. I'm quite frustrated.
In the setup wizard, I tried to click the "Next" button, and it didn't do anything. I got the error, '"document.forms.MatchSubnet" has no properties'. I'm using the Firebug Firefox extension (which I love) to see such error messages.

I decided to call tech support. It was some woman in India, of course. Here were her suggestions over the course of a 45 minute conversation with her:
  • Use XP
  • Pay for premium support (Yeah right! Pay $35 for a problem in their code!?!)
  • Use XP
  • Pay for premium support
  • She finally acknowledged the problem
  • Switch laptops
  • Reset the box
  • Use Safari
  • Use IE
  • Firewall problem?
  • Contact Linux vendor
  • Contact Linux vendor
  • Contact Linux vendor
After 45 minutes, she finally let me talk to her supervisor. He was friendly, humble, and knowledgeable. He was smart enough to ask which distro I was using--Ubuntu. He said that he had tried it with Firefox. However, it turned out that he had tried the WGR614 with Firefox, not the KWGR614. I'm guessing that the K means "the open source version".

He was able to tell me that I can go to the "basicsetting.htm" page in the admin utility in order to bypass the setup wizard. Still, there were silly errors all over the place. For instance, on one screen, I clicked "Next" and it tried to actually download the results of the CGI. It turned out to be a 0 length file. I'm guessing the headers were just really messed up. On another page, there was a parsing error. It's clear that the admin utility wasn't written cleanly by anyone who knew anything about the Web.

I tried looking for help in the forums and on Google, but I can't find many other Linux users who have tried this. I'm guessing that it's brand new, and all the other Linux users already had an AP. That would have included me, except my AP recently broke. (It was a D-Link. It would freeze up every couple minutes, and hard resetting it didn't help.)

In trying to fix the Netgear, I checked to see if there were any firmware updates. I was already using the most recent firmware ( or something like that).

Next, I tried to download the source and fix it myself. It's open source--the road is often a little bumpy, right? I found the HTML containing the error, but the CGI (written in C) was specifically missing. In it's place was a ".a" file:
$ cd ~/Desktop/KWGR614_V1.0.0_09.25NA_src
$ grep -ril MatchSubnet .
grep: ./linux-2.4.x/drivers/net/re865x/cle/cle: No such file or directory
The earlier support guy said he was going to submit a bug and inform the developers, which was really my goal in the first place. However, I just got another email:
My name is Patty and I am a Netgear level 2 technician. Could you please state the problem that you are trying to report? I have read the notes and the notes are not clear to the problem you are having with the router. As soon as the information is received I can then have it looked into.
Ugh! It doesn't work in Firefox. Is that really that hard to understand?

My buddy said he has a Linksys WRT54GL. He said the source is available for that as well, and there are already open source hacks for it. My guess is that Netgear is throwing out the KWGR614 as a lame attempt to try to capture some of the open source market. Ok, they got me, but not for long! I think I'll just return it and buy the Linksys!

Friday, December 08, 2006

Linux: Coping With Settings on Remote Systems

If you're like me, you probably end up working a lot on remote systems. It can be a pain to manage your settings on them. I think we all have a way of dealing with this. Ideally, we'd all manage our dot files in some publicly accessible source control. If you're a mere mortal like me, though, you just want a way to copy your dot files to the remote system and install your ssh key.

If so, then the following snippit from my .zshrc may be helpful to you. It's been tested with zsh, but probably works just fine in .bashrc. If you add to your .zshrc, remember to re-source that file or re-login. Edit to suit your tastes:
# Install my ssh key on a remote system.
ssh-installkey() {
[ -n "$1" ] || {
echo "usage: ssh-installkey username@host" >&2
return 1
ssh $1 "mkdir -p -m 700 .ssh"
ssh $1 "cat >> ~/.ssh/authorized_keys2" < ~/.ssh/id_dsa.pub

# Install some basic settings on the remote system for things like zsh, vim,
# and screen. Then, try to change shells.
# This does not call ssh-installkey because that command should not be executed
# multiple times. Call it first.
ssh-installsettings() {
[ -n "$1" ] || {
echo "usage: ssh-installsettings username@host" >&2
return 1
scp -r \
.zlogin .zshenv .zshrc \
.vim .vimrc \
.screenrc \
echo "Attempting to set login shell." >&2
ssh $1 "chsh -s /usr/bin/zsh"

Wednesday, December 06, 2006

Python: Trick of the Day

Let's say you have a function that makes use of a global. (Pretend for a moment that global isn't a dirty word.) Let's say you want to define that global within that function, but only if it doesn't already exist. You can use "globals()" and "setdefault" together in the following, fun one-liner:
x = globals().setdefault("x", 0)
This will get the value of the global x. If x didn't already exist, it'll be initialized to 0.

Wednesday, November 29, 2006

Python: Yahoo Looking for Aquarium Developers

I saw an email from a recruiter at Yahoo on the BayPiggies list today. He's looking for a Web developer, "We have deployed several applications using Cheetah templates and the Aquarium framework." I wrote Aquarium, so you can imagine how cool this is for me to see!

Blogger: New Template

I just picked a new template for this blog. Reading Ian Bicking's blog, I learned that fixed width columns are like totally uncool. Hence, I picked something that flowed to the page width.

If this new look pisses you off, I can change it back. If there's anything you see missing, please tell me. Otherwise, tell me if you like the new template. It's the one that irritated me the least.

Linux: Internet Explorer on Linux Using a Separate User Account

My library provides free access to Rosetta Stone which I wanted my kids to be able to use to learn Spanish. Rosetta Stone requires Shockwave which doesn't run under Linux like Flash does. The Ubuntu wiki offers some instructions on how to get it to work under Wine using a Windows version of Firefox. Alternatively, CrossOver Linux specifically supports Shockwave.

Unfortunately, neither of these options worked out for me on either of my laptops or in either Ubuntu 6.10 or Ubuntu 6.06. I was having weird X issues with both Wine and CrossOver Linux on my home machine. A bit of Googling suggested that the problem had not been fixed. My work machine wasn't being very cooperative either. In trying all the possible combinations of ways to get this to work, I wasted two days of my Thanksgiving vacation. I was really hoping that patience would lead to victory. Oh well, at least I upgraded my home laptop to Ubuntu 6.10 in the process.

There were some really nice instructions here about how to get IE to run under Linux. Since this would be useful to me as a Web developer, I decided to try it out. It worked really well. I even managed to get the Shockwave plugin working in IE--more on that in a bit.

It was late at night, so I went to bed. Having just installed IE, and since I'm pretty obsessive compulsive, I felt really dirty. In fact, I had nightmares all night. The next morning, I checked, and sure enough, IE had full access to all my files. I was hoping it was in some sort of Wine jail. I decided to wipe my entire home directory and restore it from backup. I felt better already! However, I still wanted access to IE. I decided to dedicate a user account just for IE. In order to access IE, I setup ssh with X11 forwarding so that I could run it while logged in as "jj" even though it's actually running as the "ie" user. Here's how:

First, I created a user using the "Users and Groups" system administration tool. I named it "ie". I saved the password. I used an unprivileged profile, although I did grant the user the "Use audio devices" privilege. I made sure that it had its own group named "ie".

Next, I did:
apt-get install wine
apt-get install cabextract
Then, as "jj", I installed my public ssh key into the "ie" account. If you don't know what I'm talking about, you can skip this step.

Then, per the instructions in the link above, as the "ie" user, I did:
    mkdir Desktop
wget http://www.tatanka.com.br/ies4linux/downloads/ies4linux-latest.tar.gz
tar zxvf ies4linux-latest.tar.gz
cd ies4linux-*
This installs into ~/.ies4linux. To run it, you use "/home/ie/.ies4linux/bin/ie6".

Then, as "jj", I did:
    cp /home/ie/Desktop/IE6.0.desktop ~/Desktop
I modified it so that the Exec line says:
Exec=ssh -Y ie@localhost /home/ie/.ies4linux/bin/ie6
This gave me a nice little icon to click on to launch IE without even thinking about the fact that it's running as a separate user.

Next, in IE, I installed the Shockwave Player. I had to play with the security settings in
IE's "Tools >> Internet Options >> Security" to get permission to do it. Not being a Windows user for almost a decade, this was all new to me.

After all that, it turns out that Rosetta Stone still wouldn't work. Sound didn't work in Shockwave, although it did work for the Flash plugin in IE. How frustrating! However, IE works now, and best of all, I don't feel dirty anymore!

Tuesday, November 21, 2006

Python, Pylons, Genshi, SQLAlchemy, and...Erlang!

Bet you don't think I'm crazy now!

MochiAds - Flash Game Ad Network

(Realistically, you probably still think I'm crazy.)

Wednesday, November 15, 2006

JavaScript: Douglas Crockford Video

Douglas Crockford is Yahoo!’s leading JavaScript Architect. He has written extensively on JavaScript and has been among the protagonists of the JavaScript developer community for more than a decade. Douglas is the discoverer of the JSON data format and a frequent contributor to YUIBlog.
I saw him give a talk at Yahoo's Hack Day, and it was amazing. The video is now available. Douglas provides an amazing historical perspective on JavaScript.

Tuesday, November 14, 2006

MySQL: Deleting Orphans

Referential integrity is a beautiful thing. Foreign key constraints are a godsend. However, sometimes, in certain situations you might have to get dirty and deal with orphans. Orphans are records that contain foreign key references to other records that don't exist anymore.

Suppose we have two tables, tags and article. Now suppose they have a many to many relationship via the tag_article_map table. Now, suppose for some unknown reason that someone deleted some tags from the tags table. We may now have orphans in the tag_article_map table. How do we delete them? Some special MySQL syntax makes it quite easy:
DELETE tag_article_map FROM tag_article_map
ON tag_article_map.tag_id = tags.tag_id
WHERE tags.tag_id IS NULL;

Books: Building Scalable Web Sites

Imagine if Forrest Gump developed a famous Web site. Imagine if one day he sat down on a park bench to tell you how he did it. Now, imagine it with a strong English accent ;)

"Building Scalable Web Sites" is a virtual brain dump of everything technical that you need to know in order to build something like Flickr. (The author, Cal Henderson is the engineering manager for Flickr.) If you're a new engineer, it might be a godsend. If you're a more senior engineer with little time and less patience, it can be incredibly frustrating.

I found myself alternating between these two extremes. I was tearing my hair out when Cal spent five pages explaining what source control is and listing its basic features. I questioned relevance when Cal explained how to write an HTTP client from scratch in PHP (p. 143), and I was baffled when the very next section randomly jumped to the topic of "Remote Services Redundancy" (p. 145). On the other hand, I was grateful when Cal showed me how to use Ganglia as a wrapper for RRDTool to collect and graph application-specific metrics.

This book lacked O'Reilly's standard polish. For instance, in one code sample, the indentation didn't match the braces (p. 148). There's a simple typo, "abilityto" (p. 108), that shouldn't have made it past a spell checker, let alone an editor. It makes me wonder if O'Reilly was rushing this book out the door because Flickr is hot right now, or perhaps "translating" Cal's British English was burdensome on the editors.

In summary, about 30% of the material was new to me. I patiently marched through what I already knew, and I managed to learn a few new things, for which I'm thankful.

Wednesday, November 08, 2006

Vim: Email from Bram Moolenaar

Whee! I got email from Bram Moolenaar, the author of Vim, because of a trivial doc bug I submitted last night! I feel famous!

(Gees, I really should help him help AIDS victims like Vim always asks me to do.)

Vim: More Vim Tidbits

I reread "vimtutor" for the first time in years. I learned a few things:
"e" is a motion that means "to the end of the word". "ce" is useful.

When searching, "n" to search forward, "N" to go backwards. Start with "?" to aways go backwards. I had forgotten about "N".

Do a search and replace only between two line numbers.

Do a search and replace with prompts.

:r !shellcommand
Execute a shell command and insert the results at the cursor.

This is over-write mode.

Show possible command completions, then tab to complete.

gvim in "easy" mode.
In the Vim tutorial, there's a mention of "$VIMRUNTIME/vimrc_example.vim" which I had never seen before. This helped me to improve my own .vimrc:
" These are general settings.
colorscheme torte
set autoindent
set guifont=Monospace\ 9
set hlsearch
set incsearch
set ruler
set showcmd

" Allow backspacing over everything in insert mode.
set backspace=indent,eol,start

" "T" toggles the taglist for ctags.
map T :TlistToggle<CR>

" Enable file type detection. Use the default filetype settings, so that
" mail gets 'tw' set to 72, 'cindent' is on in C files, etc. Also load
" indent files, to automatically do language-dependent indenting.
syntax on
filetype plugin indent on

" Put these in an autocmd group, so that we can delete them easily.
augroup vimrc
autocmd FileType css setlocal sw=4 sts=4 et
autocmd FileType html setlocal sw=2 sts=2 et
autocmd FileType java setlocal sw=4 sts=4 et
autocmd FileType javascript setlocal sw=4 sts=4 et
autocmd FileType mason setlocal sw=2 sts=2 et
autocmd FileType ocaml setlocal sw=2 sts=2 et
autocmd FileType php setlocal sw=2 sts=2 et
autocmd FileType perl setlocal sw=4 sts=4 et
autocmd FileType python setlocal sw=4 sts=4 et tw=72
autocmd FileType scheme setlocal sw=2 sts=2 et
autocmd FileType sql setlocal et
autocmd FileType htmlcheetah setlocal sw=2 sts=2 et
autocmd FileType text setlocal sw=2 sts=2 et tw=79
augroup END
Now that I know how to turn on intelligent indentation *and* still control it on a per-language basis, I'm even happier :)

Friday, November 03, 2006

Scalability: The Scaling Myth

Scalability is not about using Java...We can create a scalable system in PHP that contains only one tier, no XML, no Java, and isn't even very fast:
echo "Hello world!";
...It meets our three scalability criteria nicely. We can accommodate traffic growth by adding more web servers; nothing in the code needs to be changed...Our code is also very maintainable; there's not a trained programmer out there who couldn't figure out how to maintain it--for instance, if we needed to change it to say "Hello there". ["Building Scalable Web Sites" p. 203]

Tuesday, October 24, 2006

Python: Myghty vs. Django templates

It came out today on the Genshi mailing list that Myghty performs far better than Django templates.

I personally think that the productivity improvements afforded by Genshi by things such as its match mechanism and the fact that you don't have to worry about XSS attacks fully compensate for any loss in performance.

However, I'd like to say something specifically about Myghty vs. Django templates. Both are good software. Both have friendly authors. I personally think in this case the performance issue is a red herring. If you can't trust your template authors to run with scissors, you should not use Myghty. If your template authors know the difference between a function and a dict, you shouldn't use Django templates.

I think that what the Django guys did makes a lot of sense for their situation, but using Django templates drives someone like me crazy. I like to create functions (containing re-useable bits of HTML) within my templates. I don't like repeating myself, even in HTML. Nor are custom tags a simple alternative. In stark contrast, Myghty's Component-Call-With-Content Arguments is frickin' awesome!

Ok, no religious wars. I really think you should just use the right tool for the right job!

Monday, October 23, 2006

Web Design: Faking Alpha Blending with gifs

pngs are nicer than gifs because they have true alpha blending, not just transparency. However, they don't work in IE. I'm using the "pngfix.js" script to make alpha blending work in IE, but it's just not perfect. The divs that the script adds mess up my CSS. True alpha blending becomes important if you're rendering your image on a complex background such as a gradient.

I found a suitable workaround. If I start with a png and get it to look right in the browser, I can grab a screen shot. Then I trim the outside of the image to make it transparent. Hence, the actual background is "hard coded" into the gif along the edges of where it becomes transparent so that true alpha blending is not necessary, but most of the image is still transparent so that the complex background shows through. Best of all, the gifs are a lot smaller than the original pngs.

I'm sure this trick is well known, but it was new to me. Oh the depths I go through to make IE users happy!

Friday, October 20, 2006

Vim: Getting Started With Vim

A couple of readers asked me to do an overview of how to get started with Vim. I'm sure there are far better overviews than mine, but here are some personal notes:

Be sure you take the Vim tutorial. You can use "vimtutor" from the shell command line.

When you're taking the tutorial, learn how to use "hjkl" instead of the arrow keys. Not having to use the mouse or the arrow keys and rarely needing to use the control key is one of the great benefits of being a Vim power user.

It's best to get Vim out of compatibility mode. You can do this by creating a .vimrc file (even an empty one) in your $HOME directory.

Next, I personally prefer to use gvim. Using the gvim command starts Vim in GUI mode. I like the GUI menus, etc. Of course, you'll need a version of Vim with GUI support. There are GUI versions of Vim for Windows, Linux, Mac, etc.

Next, if you can get Vim 7, do that. It has nice, native tabs and a bunch of other useful features.

In response to a reader's question, I think the auto folding comes builtin. I've never used it, so I can't say more. I have seen it though.

There's a Vim cheat sheet.

Here's a list of my favorite Vim commands copied from my notes:
  Format something: <highlight>gq<or motion>
Set a map:
:map <key> <several keystrokes>
Don't be afraid to use C-V to type special characters.
Visual mode (so you can visually highlight text):
v: highlight by characters
V: highlight by lines
C-v: highlight vertical blocks
<highlight><cmd><or motion>
cmd may be either "<" or ">"
You often follow this up with one or more .'s
Named buffers: "<letter><some cut or paste like command>
Create: q<letter><go about your business>q
Use: @<letter>
If you're on a search and replace mission, end your macro with a search for
the next thing you need to search for. Otherwise, your "search next"
will probably have been blown away during the macro.
@@ will repeat the last macro you executed.
Where was I?
^o: go backwards
^i: go fowards
This will even change your file!
m<letter>: Set a marker named <letter>
'<letter>: Jump back to that marker
cmdline mode:
"q:": Enter into cmdline mode.
^f: Enter into cmdline mode in the middle of typing a command.
omni complete:
^x^o: autocomplete code
I have a pretty simple, self-explanatory .vimrc file. About the only thing complex is that I specify how to indent various programming languages based on my psychotic understanding of the various style guides for those languages. To learn more about any of these settings, use "help somesetting" withing Vim:
" These are general settings.
set guifont=Monospace\ 9
syntax on
colorscheme torte

" autoindent works better if you fix the backspace key.
set backspace=indent,eol,start

" "T" toggles the taglist for ctags.
map T :TlistToggle

" These are settings for various file types.
au BufRead,BufNewFile *.c,*.h,*.cpp set ai sw=4
au BufRead,BufNewFile *.css set ai sw=4 sts=4 et
au BufRead,BufNewFile *.html set ai sw=2 sts=2 et
au BufRead,BufNewFile *.java set ai sw=4 sts=4 et
au BufRead,BufNewFile *.js set ai sw=4 sts=4 et
au BufRead,BufNewFile *.mhtml,*.mas set ai sw=2 sts=2 et ft=mason
au BufRead,BufNewFile *.ml,*.mll,*.mly set ai sw=2 sts=2 et
au BufRead,BufNewFile *.php,*.php3,*.php4 set ai sw=2 sts=2 et
au BufRead,BufNewFile *.pl,*.pm set ai sw=4 sts=4 et
au BufRead,BufNewFile *.py set ai sw=4 sts=4 et tw=72 " Doc strs
au BufRead,BufNewFile *.scm set ai sw=2 sts=2 et
au BufRead,BufNewFile *.sh set ai sw=4 sts=4
au BufRead,BufNewFile *.sql set ai et
au BufRead,BufNewFile *.tmpl set ai sw=2 sts=2 et
au BufRead,BufNewFile *.txt set ai sw=2 sts=2 et tw=79
You can download plugins for Vim from the Vim Web site. The taglist plugin is a must have for using ctags. I use this to see all of the functions and classes in the Python module I'm currently editing. The minibufexpl plugin is useful if you're not using Vim 7. It shows all of your open buffers in fake tabs. Be sure to add "set hidden" to your .vimrc if you use it so that you don't have to save the current buffer before switching to another one.

Vim 7 has some neat features such as an improved ability to edit and manage files over scp. I'm just getting started with it, but "gvim scp://foxmarks.com/" is already useful.

Also, I should mention something about workflow. IDE users and Emacs users rarely have more than one copy of their editor running at any one time. Since startup in Vim is so fast, I must admit to opening up multiple gvim windows all the time. I might have 5 of them spread across my screen at the same time. However, it's now possible in Vim 7 to open up a document from the command line in a new tab within the currently running editor: "gvim --remote-tab file.txt".

Furthermore, I should state that I don't use Vim as an IDE replacement. If you want a full IDE for Python, I can recommend Wing IDE. I don't use an IDE. Rather, I use Vim as an important part of my "development environment". I'm still very shell oriented. I use the shell for all sorts of things such as file management, Subversion, grep, find, wc, etc. Recently, I switched to zsh, but that's a topic for another post ;)

To finish this post, as well as to give a flavor for how I use the shell and Vim together, I'll show you how I do global search and replace across my entire project. Start at the top of the project in your shell. Then do: "gvim $(grep -rl sometext .)". This will open up all the files of interest. Then, "/sometext". This will jump to the text you're looking for. Use the "c" command to change the text in one shot. Now, "n" to jump to the next instance and "." to change that instance too. At the end of the file ":wn" to write and jump to the next file. Following this "work idiom" you can make each change by hand, but still do so really fast: "n.n.n.n.:wnn.n.n.n:wn". To make more complicated edits, I follow the same pattern, but I use a macro to change the text.

Happy Hacking!

Tuesday, October 17, 2006

Foxmarks: We Just Launched!

We just launched the new version of Foxmarks!
Foxmarks keeps your bookmarks automatically synchronized between two or more computers. A simple wizard guides you through the startup process. Foxmarks works silently in the background. As a bonus feature, you can access your bookmarks from anywhere via my.foxmarks.com. It's simple and solid.
We had to hustle in order to get it out the door before Firefox 2 comes out. The dust hasn't yet settled, but at least I got some sleep last night ;) Word to the wise--vainly attempting an all-nighter every night for a week just doesn't work. No, it doesn't matter how much coffee you drink!

Anyway, I'm pretty excited. Despite the short timeline, our team was amazingly well organized, and the code is really clean. My co-worker wrote the synchronization server using Twisted, and it's performing amazingly well. My boss is still primarily responsible for the extension itself, although he was very helpful with all the other code. I wrote the code for the new versions of www.foxmarks.com, my.foxmarks.com, and login.foxmarks.com (the common login server), but of course I borrowed from the previous code base.

If you're wondering, I used Pylons, Genshi, and SQLAlchemy, and I've been really pleased with the results. I also decided to go the whole "semantic markup" route. I used CSS for layout (although I'm still using tables for forms--old habits die hard). My knowledge of CSS has come a long way in the last week, but the markup looks really clean!

Saturday, October 14, 2006

Vim 7: Lovin' It

I finally installed Vim 7. I also took the time to install various plugins. Here are some things that are making me happy:
  • Honest to goodness, real tabs.
  • Auto-completion and built-in documentation lookup for many different programming languages.
  • Editing files and browsing directories remotely over scp works.
  • Subversion integration via the vcscommand plugin is helpful.
  • Vim is still charityware which makes me proud to use it.
Vim is still my favorite editor because I truly believe that its style of keybindings is faster for experts. Furthermore, it has one of the strongest and most flexible syntax highlighting systems:
  • It supports a ton of languages by default.
  • It does well with files that mix languages like HTML, CSS, and JavaScript.
  • It doesn't get confused by Python strings such as """He said, "hi"!""" ;)
I code in a lot of languages, and I love that Vim is consistent and helpful.

Friday, October 13, 2006

Books: Thank God for Editors!

Thank God for editors in general, and O'Reilly editors especially!

As I was reading an O'Reilly book on the train today, I realized how grateful I am for all of the hard work editors put into improving the quality of technical books. Let's face it, technical topics are tough, and programmers aren't always the best writers. As programmers, there's a monumental amount of learning we need to do in very little time using books written by other programmers who also have very little time. Editors help by making the text more uniform. Uniform text can be ingested more quickly. Without the painstaking effort that editors put in, I would have a much harder time reading what I need to read. In fact, every time I come across glaring typos or grammar mistakes, it distracts me from the technical material I'm trying to learn. Editors enable me to focus on the topic at hand.

I know that authors always thank their editors, and that this post might seem a little silly. However, I'd like to tell all the editors out there how thankful I am for the work they do!

Python: Modifying a Counter in a Closure

I helped a fellow Pythonista with a problem today, and it reminded me that a lot of people might not know about the trick of using a list to box variables within a closure. If that doesn't make sense, read the code below:
def counter(start):

"""Create a counter starting at ``start``."""

# The value "curr" needs to be wrapped in a list. Otherwise, when
# "+=" is used in "inc", Python binds the variable at that scope
# instead of at this scope. We don't want to redefine a variable at
# the inner scope. We want to modify the variable at this outer
# scope. Java programmers would call this "boxing", but they would
# use an Integer instance instead of a list.

curr = [start]

def inc():
"""Increment the counter and return the new value."""
curr[0] += 1
return curr[0]

return inc

if __name__ == '__main__':
c = counter(5)
for i in range(3):
print c() # Prints 6 7 8.

Wednesday, October 04, 2006

Python: Greenlets

Someone pointed me at Greenlets the other day, and I'm really quite excited about them. This fully addresses the Limitations of Coroutines via Enhanced Generators I spoke of earlier. I see that "some very smart people" have gone down the path of tying it together with kqueue and select.poll along with an IO-based scheduler.

Greenlets look polished enough, but I sure wish the other stuff was polished and documented so that I could use it in production. The pragmatist in me (which doesn't come out very often) is still insisting that Twisted is still the safer route in the near term even if it is more irritating to code because of its asynchronous nature.

Monday, October 02, 2006

Hacking: Yahoo Hack Day

I attended Yahoo Hack Day. It was a lot of fun. They put on a concert by Beck, which I unfortunately didn't get to see. They also loaded us up with plenty of swag, including a T-shirt, a copy of Yahoo Hacks, and a 256 mb non-branded USB flash drive, which was one of the weirdest but most useful bits of swag ever. I attended talks all day long. Here are some of the takeaways:
  • Yahoo! UI Library: Fonts CSS is a CSS library that completely levels the playing field concerning fonts across all of the browsers. I'm not that great a CSS developer, so I wasn't all that excited about this, but my friends explained to me how hard cross-browser uniformity is to achieve.

  • Yahoo! UI Library: Grids CSS is a CSS library that lets you do standardized layout in a cross-browser way that works now. I wasn't so thrilled about this either since I knew that even though CSS layout isn't ready for prime time, tables still do the trick. However, once again, my friends set me straight by explaining how powerful it was to simply flip a few class attributes and have the layout change drastically, even within the user's browser. That at least is a bit harder to do with tables.

  • There was a fantastic talk given concerning the history of JavaScript and the Web. The speaker made a good point that it's amazing the stuff even works if you consider that the Web is really just a series of hack upon hack. He gave a plug for JavaScript: The Definitive Guide, specifically the 5th edition. He said that this book sucks the least in terms of really understanding the functional nature of JavaScript. He also confirmed what I had long believed that JavaScript is the most misunderstood language ever. He also made a good point about how bad the standards were for JavaScript:

    • The ECMAScript standard is unreadable. Furthermore, Microsoft insisted on encoding various bugs in the standard itself.
    • Neither was the W3C blameless. Consider how hard it is to use the DOM API in contrast to innerHTML. However, the DOM API is in the standard, and innerHTML isn't. Next consider how "un-JavaScript" code such as .setAttribute('foo', 'bar') is when you consider it next to code such as img.src = 'foo.jpg'!

    I asked him if there was anything substantial missing from Dojo Toolkit that was present in the Yahoo! UI Library (YUI). He said no, but that there were probably too many JavaScript frameworks right now, and the market would probably only support Microsoft Atlas and YUI in the long term. For being such a well-educated guy, this really surprised me. Perhaps he was just taking the party line, or perhaps he had a case of "not invented here".

  • "Semantic markup" is the idea that your markup should say what it means instead of how it should appear. "Progressive enhancement" is the idea that when you build a Web application, you should start with semantic markup that actually works for any browser. Then, on top of that, you use deep CSS and JavaScript tricks to enhance the experience for "A-grade" browsers. Notice that Yahoo's graded browser support is pretty inclusive. Also note that Yahoo is still using HTML 4.01 Strict.

  • Yahoo! UI Library: Event is exactly what you would expect from a JavaScript event library. There was nothing astonishing or surprising.

  • FireBug, Venkman, Web Developer toolbar, and the JavaScript debugger found on the Microsoft Office CD are great tools to debug JavaScript, but JSLint got a couple of plugs as well. Fiddler is an HTTP debugging proxy which logs all HTTP traffic between your computer and the Internet, which can be helpful to debug AJAX requests. Drip is a memory leak detector for Internet Explorer.

  • Yahoo's Design Pattern Library got a plug as well. I saw a talk on this a few weeks ago, and it looked pretty useful.

  • And of course, one of the speakers gave a plug for "working is better than elegant", which is always a challenge to perfectionists like me.

Wednesday, September 27, 2006

Books: Scalable Internet Architectures

Scalable Internet Architectures

I can summarize this book:
  1. High availabililty and load balancing are as completely different as peanut butter and jelly.
  2. Spread is cool.
  3. Look, ma! I can write complex code in multiple languages!
Overall, however, I liked the book. It was all new material to me, and I'm glad I read it.

Friday, September 22, 2006

Databases: More Atomic Cluster Commits

I'm currently reading the book Scalable Internet Architectures, and I'm enjoying it.

On page 141, he describes two-phase commits:
The basic idea is that the node attempting the transaction will notify its peers that it is about to commit, and they will react by preparing the transaction and notifying the originating node that they are ready to commit. The second phase is noticing any aborts in the process and then possibly following through on the commit (hopefully everywhere)...The two-phase commit is not perfect, but it is considered sufficient for applications such as stock trading and banking...Despite its shortcomings, the only common "correct" multimaster replication technique used today is the 2PC, and, due to its overhead costs, it is not widely used for geographic replications.
I didn't catch the phrase "not perfect" the first time I read that section, so I spent the day wondering how the hell they implemented fully atomic transactions across a cluster of machines. What happens if all the machines agree that they're right about to finish the commit and then one of the machines goes down right before writing the final bit to the transaction log and the others don't? I'll readily admit I'm no database expert, so I figured I must be missing something. I fought with it for a while, and I was very pleased to re-read that section and see the phrase "not perfect".

Well, how could you do a perfect cluster-wide commit with absolutely no race conditions? Let's assume the machines are all near each other since 2PC doesn't work across the country anyway without a ton of lag. I have a silly brute force solution that may not be perfect, but is a bit closer to atomic (i.e. it relies on the hardware in the same way mutexes do):

Start with battery-backed RAID cards. However, design the RAID cards so that they have one input wire and one output wire specifically to address this problem. Now, put all the machines in a circuit using this wire:
| |
On each of the RAID cards is a switch. If all the machines open their switches, current can flow. If any one of them closes their switch, current doesn't flow. It's the hardware equivalent of an AND operation; did I mention I'm not a hardware guy? Anyway, my commit system proceeds pretty much the same as the 2PC. When the machines have agreed that they're ready to do the final commit, the kernel setups up the sector to be written to disk and lets the RAID card take over. Then, they each open their switches. When the current starts flowing, the RAID card recognizes this signal to write the sector containing the data for the transaction log saying that the commit took place. If the RAID card is, for some reason, unable to perform its duties, it shuts down and declares that it's broken. I.e. the machine will die, but there won't be inconsistency.

Well, there may be holes in this scheme. Perhaps it was written about 30 years ago just like all the other interesting CS problems. It does require specialized hardware. All those reservations aside, I think this leads to a more atomic cluster commit.

Python: I Love Genshi!

I’ve totally fallen in love with Genshi! It's an XML templating engine for Python. I learned it between midnight and 2AM one night. By the next day, I was totally productive and totally loving it! I like the fact that template inheritance works so easily, and I love the XPath stuff. It's nice to be free of XSS vulnerabilities to some extent. I really didn't like Tal, so I was surprised to find that Genshi was so nice. It’s weird--Genshi is like a superset of all the templating engines, but in a way that is conceptually simple and elegant.

More about Genshi

Monday, September 18, 2006

Python: Stacked Thread-local Object

This is a cute trick: Registry for handling request-local module globals sanely.

It has all the ease of use of globals but with the thread-safety of shoving everything in a per-request object called ctx.

Friday, September 15, 2006

A Summary of Talks I've Attended Recently

I've been to a lot of talks and a conference recently. These included:

I thought I would blog about ideas that were either:

  • New and interesting.
  • Not new, but came up a lot.


  • Be passionate or go home.
  • Be a pain killer, not a vitamin.
  • Be pragmatic; working code is better than beautiful code.
  • What's in it for the user?

Startups Should Keep in Mind

  • Big, bloated companies are good targets.
  • There is lots of room for passion-centric communities.
  • We consistently see the pattern of boom, correction, lasting change.
  • Amateurs are becoming really powerful.

Getting Started

  • Keep talking to real users, not just your techie friends.
  • Don't go beta with something that sucks.
  • Raise less money, spend less money, hire slowly, fire quickly.
  • Developers should be in the same timezone.
  • Use specialists; for instance, don't waste a good engineer fighting XHTML/CSS browser issues.
  • Don't cut corners; take care of the details.

Attracting and Keeping an Audience

  • Make it dead simple to use.
  • Do your own support.
  • Make contacting you really easy; an email address is not enough.
  • Declaw your customers by being polite and apologetic when they are rude.
  • Groups begin to fail when there are more than 150 people.
  • Don't break APIs.
  • Support CSV.

User Experience and Design

  • Put off forcing people to sign in as long as possible.
  • It's all about discoverability, recoverability, context, and feedback.
  • Make it pretty.

Production Environment

  • Plan for maintenance.
  • Graph and measure like crazy; create a dashboard.
  • Do one touch deploy; automate everything.
  • It's all about process.
  • Don't be special.
  • Design for debugability.
  • Embrace cheap hardware; expect hardware failure.

User Contributed Content

  • Most people won't contribute--that's okay.
  • Spam is a deep problem.
  • A self policing community is necessary, but not sufficient.

What's Cool

  • flickr
  • digg
  • Web APIs
  • Second Life
  • Creative Commons
  • Agile software development
  • Interoperability between Web and desktop
  • Adobe Apollo
  • memcached
  • Newsvine
  • Techcrunch

The most popular and impactful talks did not use traditional bullet point slides. Rather, they were mostly verbal, without reference to notes. Nonetheless, they were extremely well focused and structured. Slides often consisted of images licensed under Creative Commons found on flickr. The images set the mood. Also popular were slides that showed a single word or phrase. These slides were synchronized perfectly to the speech.

Tuesday, September 12, 2006

Free Culture: Lawrence Lessig Talk Now Available

Lawrence Lessig's keynote Free Culture: What we need from you is now available. It was the best keynote I've ever seen at LinuxWorld.

Take the time--it'll blow your mind! ;)

Erlang: Toying with Erlang

I'm really interested in distributed computing and concurrency right now. Given that I've been playing with Haskell so much, I thought I'd give Erlang a try since it's all about distributed computing, concurrency, and scalability.

I must admit that it already seems easier for me to read than Haskell. I'm not sure why. As I mentioned before, I do like Erlang-style concurrency, since strangely enough, that's how I had always thought things should work.

I'm fascinated by Mnesia, Erlang's distributed database. Unfortunately, according to this:
Mnesia is not perfect, of course, and its biggest downside at the moment is that its disc storage engine isn't suited for storing large volumes of data (Mnesia was designed for soft real-time applications where the data is stored mostly in RAM), but I hope this will be resolved in the not-too-distant future.
This is a major downer for me because I'm currently interested in terabyte-sized data sets. Nor is there a suitable MySQL driver.

Yaws is the Web application framework for Erlang. I can't say whether it's good or bad, but as a matter of taste, ever since I used Apache's Element Construction Set back in the day, I've had a particular dislike for generating HTML using programming language syntax as is done in Yaws:
%% the little status field in the upper left corner
head_status(User) ->
T =
{table, [],
{tr, [],
[{td, [{width, "30%"}],
{table, [ {border, "1"}, {bgcolor, beige},{bordercolor, black}],
[{tr, [], {td, [], pb("User: ~s", [User])}}
{td, [{align, right}], {img, [{src, "junk.jpg"}
Yuck :(

Nonetheless, Erlang has already taught me new ways of thinking about distributed computing. I guess I'm wondering if there's anything I can't do the Erlang way in Python. Afterall, consider Candigram, which is an implementation of Erlang concurrency primitives in Python; although Candigram itself doesn't provide microthreads like Erlang. That reminds me, I'm still looking for an answer to my post Limitations of Coroutines via Enhanced Generators.

And if you're wondering--no, I don't know what the hell I'm talking about! I've only been looking at this stuff for like two days! I guess I should just shut up and go back to reading Erlang for C, C++ and Java Programmers. Hmm, I wonder where the tutorial for Python programmers is ;)

Wednesday, September 06, 2006

Erlang: Erlang Style Concurrency

I'm lovin' this article: http://www.defmacro.org/ramblings/concurrency.html.

It meshes nicely with my earlier post here. Erlang is like stackless Python with coroutines in that it's a lightweight threading system built on top of asynchronous IO. However, it's different in that there is no data shared between threads (i.e. a shared heap protected by locks). Instead, to share data, you must use message passing. This part matches my earlier post. Of course, the benefit is that it's trivial to do distributed computing with such systems.

Anyway, Erlang has been around a long time and is incredibly robust. I'd love to get a chance to use it.

Thursday, August 31, 2006

Linux: On-the-fly BASH Aliases with Cntl-R

If you use BASH, I hope you make use of Cntl-R to search for commands you've already executed. Even though I know about and use Cntl-R, even I underuse it. For instance, you can use Cntl-R to create on-the-fly, short-lived aliases. Start with:
cd /some/long/path # cdlongpath
Then later:
It will autocomplete to your earlier command. If you happen to forget an alias, no big deal :)

Wednesday, August 30, 2006

IETF: RFC 4193

At long last, an Internet draft I contributed to has become RFC 4193, and I am acknowledged in it :-D

Thursday, August 24, 2006

Python: Limitations of Coroutines via Enhanced Generators

I've been thinking a lot about tricks such as the coroutine system in this article. PEP 342 improves the situation for Python 2.5. In fact, Example 3 in the PEP is exactly what was on my mind. I came to the same conclusion as the PEP:
In effect, this example emulates simple tasklets as are used in Stackless Python, as long as you use a yield expression to invoke routines that would otherwise "block".
However, what happens if you can't use a yield expression to invoke routines that would otherwise block?

For instance, say coro() is a coroutine that calls third_party_1() that calls third_party_2() that calls blocker() where blocker() blocks. If third_party_1() and third_party_2() are intermediary functions that exist in third-party code, they won't know how to "do the right" thing using yield. If third_party_1() and third_party_2() don't do the right thing with yield, there's no way for blocker() to yield back to the scheduler.

Ideally, I'd like for coro() to be a coroutine that blocks within blocker(), leaving third_party_1() and third_party_2() ignorant of the whole process. If you know how, please tell me!

Monday, August 21, 2006

Python: Stunning News: Django isn't Perfect!

Concerning http://www.cmlenz.net/blog/2006/08/the_python_web_.html:

Although I have a ton of respect for what the Django guys have accomplished (I really do!), I couldn't agree more with this post based on my own experience and the experience of many of my co-workers.

By the way, when he mentions, "There was a comment by someone in the audience at the Google TechTalk on Django along the lines of...or scale down to teams where the programmers are also those who write the templates," that was me! :)

It's not that our team wasn't large enough for separate template authors. Quite the opposite; we had 169,000 lines of code. It's that I generally don't trust template authors because I fear they write redundant code. Compared to Cheetah or Myghty, Django makes it hard to write quick little template functions that take keyword arguments, thus I suspect that a lot of people put up with redundant templates because, after all, (tongue in cheek) it's HTML, not code, riiight???

Thursday, August 17, 2006

Linux: Ubuntu 6.06 on Dell Inspiron B130

Pretty much everything "just works", including hibernation and sound.

To get my resolution up to 1280x800, I had to install the 915resolution package from the Universe repository as mentioned here.

I bought a mini PCI wireless card from newegg, and it just worked out of the box.

I'm happy :)

Thursday, August 03, 2006

Haskell: Translating C to English Using Haskell

I've written a two-part series for "Linux Journal" called "Translating C to English Using Haskell". Here are links to the two parts:


The catchline is:

Write a program in Haskell that translates C type declarations into English. Manually translate the Haskell into English.

I've tried very hard to make it an entertaining article for Haskell newbies, so if your head exploded when you tried to read "A Gentle Introduction to Haskell", give my article a shot!

Friday, July 28, 2006

Python: Don't Mix Wing IDE and sshfs

Wing IDE Professional is awesome. sshfs is awesome. Massively large codes bases are, well, sometimes you're stuck with them.

But note, Wing IDE, sshfs, and massively large code bases don't mix; it's slooooowwwww. Use NFS instead.

By the way, the ability to run Wing IDE on my Linux laptop in order to debug a process running on a remote FreeBSD server (using stackless Python no less!) is fickin' awesome!

Thursday, July 06, 2006

Linux: Twin

Twin is a text-mode window environment. It turns a text terminal into a X11-style display with window manager, terminal windows, and can also serve as display for remote applications. Each terminal window provides the functions of a text-mode Linux console. Twin runs on X11, libggi, itself, the Linux console, and any termcap/ncurses-compatible tty. It supports multiple simultaneous displays, and can attach/detach each display on the fly. [from freshmeat.net]

I do most of my development on a remote server. Since I'm a bash + vim type guy, that suits me just fine. I usually use screen. This allows me to create multiple virtual terminals as well as detach from the session and reattach later, perhaps from a different computer. Twin can do all this and more, but in a user friendly way! I'm just getting started, but it sure does look neat!

Wednesday, June 14, 2006

Haskell: Microsoft Haskell PDF

I'm not sure where I got this or how it ended up on my "to read" list, but I'm really quite blown away by it. It amazes me to know that some hardcore Haskell guy somewhere is busy shoving ideas like monads into commonplace languages like Visual Basic. I had also never stopped to consider that Visual Basic is indeed interesting because it embraces the idea "static typing where possible, dynamic typing where needed."

Monday, June 05, 2006

The Career Programmer: Guerilla Tactics for an Imperfect World

I'm currently reading The Career Programmer: Guerilla Tactics for an Imperfect World. It's fantastic! It's as entertaining as Expert C Programming: Deep C Secrets but as important to read as The Mythical Man-Month.

Simply put, this is a must read, especially if you've been coding professionally for a while, and you're tired of getting kicked around by unreasonable deadlines, politics, etc. This book has taught me that ignoring politics is like thinking you can walk across a freeway just because you have your eyes closed. In between the non-stop entertainment is a fully-developed, practical plan for dealing with work when things don't go according to plan.

I'm not quite done with it yet, but so far, it's a godsend.

Linux: Ubuntu 6.06 on Dell Latitude D610

I just installed Ubuntu 6.06 on my Dell Latitude D610. I love it! You start with a live CD. Hence, you can "try before you buy". Then, when you're ready, you double-click the "Install" icon; you don't even have to burn another CD.

Compared to Ubuntu 5.10, a lot of polish has been added. Since I came from a Fedora background, I was impressed to see that Ubuntu has caught up with all the polish that Fedora has (graphical installer, graphical boot, etc.). Furthermore, despite the fact that I installed it the day after it came out, the Wiki had already been updated with instructions on how to get deal with proprietary formats such as MP3s, Windows video, Flash, etc.

As far as hardware goes, everything pretty much just worked--even hibernate and suspend! To support the builtin wireless, I had to use NDIS. Here are my abbreviated instructions:
Add the ndiswrapper-utils package.
Download the driver from ftp://ftp.us.dell.com/network/R102320.EXE.
mkdir driver
cd driver
unzip ../R102320.EXE
ndiswrapper -i bcmwl5.inf
ndiswrapper -l
ndiswrapper -m
Reboot. I'm not sure if this is necessary.
Setup the network settings as usual.
If the card doesn't work, make sure you didn't hit some function key that turns off the wireless in the BIOS.
All in all, I'm stoked. :)

Tuesday, May 30, 2006

Computer Science: More Threads = Less Contention?

Have you read The Next Mainstream Programming Language? The original PDF was written by Tim Sweeney, the founder of Epic, the video game company that created Unreal. He really got me thinking about the problems he was facing.

Let's suppose you have 10,000-100,000 objects. Let's suppose that at any moment any one of them might need to "touch" 50 other objects. That's the setup.

Now, let's suppose that you also have a super-lightweight threading system that can handle 100,000 threads. Yeah, sounds far-fetched, but you'd be surprised. Consider allocating a thread for each object. Along with its own thread, each object should have an input queue into which you can post new events. Naturally, this input queue should be protected by a mutex.

The interesting result is that because there are so many threads trying to access so many mutexes, the chances of any two threads trying to access the same mutex to push a new event is relatively small is you assume a random distribution of events. Practically speaking, more fine grained locking can result in reduced thread contention.

Of course, life isn't random, but it's an interesting thought.

Wednesday, May 03, 2006

Python: Django Meeting at Google now on Google Video

The talk that I mentioned earlier is now up on Google video. Hence, my new claim to fame is that I'm in a Google TechTalk :)

Friday, April 28, 2006

Python: Protecting UTF-8 Strings from Naive Code

"""Temporarily convert a UTF-8 string to Unicode to prevent breakage.

BASIC IDEA: protect_utf8 is a function decorator that can prevent naive
functions from breaking UTF-8.


def protect_utf8(wrapped_function, encoding='UTF-8'):

"""Temporarily convert a UTF-8 string to Unicode to prevent breakage.

protect_utf8 is a function decorator that can prevent naive
functions from breaking UTF-8.

If the wrapped function takes a string, and that string happens to be valid
UTF-8, convert it to a unicode object and call the wrapped function. If a
conversion was done and if a unicode object was returned, convert it back
to a UTF-8 string.

The wrapped function should take a string as its first parameter and it may
return an object of the same type. Anything else is optional. For

def truncate(s):
return s[:1]

Pass "encoding" if you want to protect something other than UTF-8.

Ideally, we'd have unicode objects everywhere, but sometimes life is not
ideal. :)


def proxy_function(s, *args, **kargs):
unconvert = False
if isinstance(s, str):
s = s.decode(encoding)
unconvert = True
except UnicodeDecodeError:
ret = wrapped_function(s, *args, **kargs)
if unconvert and isinstance(ret, unicode):
ret = ret.encode(encoding)
return ret

return proxy_function

def truncate(s, length=1, etc="..."):
"""Truncate a string to the given length.

If truncation is necessary, append the value of "etc".

This is really just a silly test.

if len(s) < length:
return s
return s[:length] + etc
truncate = protect_utf8(truncate) # I'm stuck on Python 2.3.

if __name__ == '__main__':
assert (truncate('\xe3\x82\xa6\xe3\x82\xb6\xe3\x83\x86', etc="") ==
assert truncate('abc') == 'a...'
assert truncate(u'\u30a0\u30b1\u30c3', etc="") == u'\u30a0'

Humor: bash-3.00$ jj < coffee.cup > code.py

This is my second all-nighter this week (well, technically, I slept for a couple hours the other day). I had two double espressos yesterday, a cup of coffee last night at 10:30 PM, and I got one this morning at 5:00 AM. I think I realized I had a problem when I discovered that I was irritated that Star Bucks wasn't open between 11:00 PM and 4:30 AM.

Wednesday, April 26, 2006

Python: Django Meeting at Google

I've organized a BayPIGies meeting to take place at Google tonight at 7:30PM. Jacob Kaplan-Moss, one of the lead developers of Django, will be giving a talk on Django. There's more information on the BayPIGies Web site.

Wednesday, April 19, 2006

UNIX: ssh + tar + gzip -q = goodness

To retrieve a hierarchy of files from a remote server (or to copy it back to a remote server), I often do something like:
ssh servername "tar cvzf - dirname" | tar xvfz -
However, I usually get the following error message:
gzip: stdin: decompression OK, trailing garbage ignored
tar: Child returned status 2
tar: Error exit delayed from previous errors
Strangely enough, as I write this, I get the error message copying something from one FreeBSD system to another FreeBSD system, but I don't get it when copying something from one FreeBSD system to my Ubuntu system. Weird.

I put up with this problem for years. However, I recently needed to use it in a Makefile. Having an error like that is fine when you're a human, but a non-zero return code is a deal-breaker in a Makefile. I needed to clean up my act.

One easy way to make the problem go away is to not use the "z" flag for both instances of tar. This is somewhat icky, because it really would be nice to have the content gzipped. Otherwise, it could take too long to transfer.

Finally, I found the real solution on the gzip Web site. Instead of passing the "z" flag to tar when untarring, use gunzip separately and pass the "q" flag to tell it to be quiet:
ssh server "tar cvzf - dirname" | gunzip -q | tar xvf -
By the way, as is standard in UNIX, there are plenty of other variations of this tar + ssh idiom. For instance, consider:
ssh server "cd myapplication/share/locale && 
find . -name '*.po' -o -name '*.mo' |
xargs tar cvzf -" |
gunzip -q | tar xvf -

Wednesday, April 05, 2006

Software Engineering: Professional Software Development

I've written a comprehensive summary of the book "Professional Software Development". You can find the slides here. I highly encourage everyone to take the time to read the slides as the signal to noise ratio is extremely high.

Sunday, April 02, 2006

Hardware: Smarter Memory

I'm not a hardware guy, but it seems to me that it would be really nice if RAM could be a little smarter and implement a few simple commands:
  • memcpy - Copy one area of memory to another.
  • memcmp - Are the given two strings of memory equal?
  • memzero - Zero out an area of memory.
Now naturally, these commands can't directly be used by applications because of the difficulties of virtual memory. I also wouldn't expect them to be as smart as their C counterparts. However, given some support in the standard library and the kernel, these commands could be very useful optimizations.

Emacs: Syntax Highlighting

It's that time again! Whether it's because I'm drinking coffee and that's causing my compulsive obsessive nature to do crazy things, or because I'm inspired by other smart programmers who use Emacs, I'm getting "must use Emacs" cravings again. However, as soon as I started it up, the syntax highlighting irritations hit me like a stop sign over the head. The time, it's just a normal Python file. Notice how Emacs doesn't understand that double quotes can be embedded in triple double quotes. By the way, this isn't my code, so don't send me hate mail because there's HTML embedded in Python ;)

Monday, March 20, 2006

Linux: Verizon Yahoo DSL

I got Verizon Yahoo DSL working on my Dad's Fedora Core 4 box. His modem is a Westell 6100. These directions were very help (which is half the point of this post). Here are some notes:
  • The default username and password for the modem are admin and password. It prompts you to change these once you login.
  • When I called Verizon, they gave me some userid and password, but I don't remember ever needing to use them.
  • The tech support at Verizon was from Alabama and spoke English well.
  • I had problems with the DSL not syncing. The DSL light was just blinking. It turns out my dad plugged the modem into a wall jack that wasn't actually connected to anything. It's strange that they can't work around problems like that in software ;)
  • Since I was helping my dad over the phone, I was really pleased to find out he could give me remote access to the modem. He logged into the modem at He navigated to Maintenance > Remote Access. Once there, he picked a password and turned remote access on. It told him the URL I could use from "outside". Best of all, remote access turns itself off after 20 minutes.
  • PPPoE is running on the modem itself.
  • I'm somewhat confused because I really don't remember configuring much of anything. :-/

Wednesday, March 15, 2006

HTML: Escaping &'s in URLs in HTML

Warning: Failure to ignore the following validation warning may result in lost productivity!
Concerning Ampersands (&'s) in URLs, the following is what I wrote in the Aquarium documentation:

The short answer is, if you have a URL with more than one parameter, you should wrap it with $htmlent when you embed it in HTML if you want to pass DTD validation. If you don't care, then it really won't matter. What follows is an explanation of why I can't make it any easier on you.
  1. You must escape &'s in URLs in order to pass DTD validation. Per the spec, a browser could look at http://a.com/?a=b&copy;=2 and interpret the ©= as part of value of the a variable instead of a new variable named copy; because &copy; is an HTML entity.

  2. To handle #1, Aquarium use to escape the &'s in every URL automatically.

  3. However, #2 broke redirects if you redirected to a URL with more than one parameter. I so rarely did this, that I didn't know about the bug for a good year.

  4. To fix the bug in #3, we came up with a scheme to always escape &'s but deal with the redirect case specially. Now imagine if you create a HTML link whose URL has a parameter named referrer that is set to your current URL, which itself happens to have two parameters. When the user clicks on the link, Aquarium now has a GET parameter named referrer that is a URL. The programmer can use that URL directly in HTML (in which case the &'s must be escaped) or he might redirect to it (in which case the &'s must not be escaped). The programmer is never going to remember whether the URL is already escaped (per #3, it already is) and whether he needs to escape it for a link or unescape it for a redirect. His brain would core dump.
    When it comes to escaping things, a good general rule is to escape things at the last possible moment. By violating this rule, bad things were happening.
  5. We could force engineers to wrap every URL in HTML with htmlent, but that would suck. Too much existing code doesn't.

  6. Browsers are smart, and most of the time, if you don't escape the &'s, the browser won't get confused. In fact, you can't generate a URL like http://a.com/?a=b&amp;copy;=2 with Aquarium anyway, because the ; will get urlencoded to %3B. Hence, it's not possible to get Aquarium to generate a URL that would confuse the browser. Programmers who are worried about passing DTD validation and have a URL with more than one parameter will just have to use $htmlent. That's better than forcing every programmer to think about the problem every time he generates a URL since, practically speaking, the warning is pedantic.

Sunday, March 05, 2006

Python: Django: Now with 73% More Friendliness!

I completely withdraw my earlier complaint:
Why are the Django guys so rude to other programmers?
You can read the comments to my earlier post to see what I'm talking about.

It reminds me of the fact that my mom use to say to me, "If you can't eavesdrop all the way, don't eavesdrop at all!"

Friday, March 03, 2006

Python: Fun with Classes


# Author: Shannon -jj Behrens
# Date: Fri Mar 3 16:43:43 PST 2006
# In response to: http://secretartofscience.com/blog/?p=8
# Although Python isn't a prototype-based language, it's possible to do a lot
# of these same wacky ideas in Python.
# By the way, Io is a cool, new prototype based language. I've blogged about
# it before. There was also a knockoff of Python that was prototype based, but
# I don't remember the name--email me if you must know.

# Ok, let's start by changing our class on the fly.

class Draws(object):

def draw(self):
print "draw"

class DrawsSmall(Draws):

def draw(self):
print "small:",

obj = Draws()
obj.__class__ = DrawsSmall

# This time, let's mixin something on the fly.

class DrawsBig(Draws):

def draw(self):
print "big:",

obj = Draws()
class MixedIn(DrawsBig, Draws): pass
obj.__class__ = MixedIn

# This time, do it dynamically. I'll leave it as an exercise to the reader to
# implement "replace_parent_class".

import random

def inject_parent_class(obj, Class):
NewClass = type("DynamicGeneratedClass",
(Class,) + obj.__class__.__bases__, {})
obj.__class__ = NewClass

obj = Draws()
Class = random.choice([DrawsBig, DrawsSmall])
inject_parent_class(obj, Class)

Tuesday, February 28, 2006

Python: Django Custom Tags

In my last post, I complained that code like the following is redundant:
<tr class="fieldrow">
<th><label for="id_subject">Subject:</label></th>
{{ form.subject }}<br>
{% if form.subject.errors %}
<div class="error">{{ form.subject.errors|join:", " }}</div>
{% endif %}

<tr class="fieldrow">
<th><label for="id_name">
Poster's Name:
{{ form.name }}<br>
{% if form.name.errors %}
<div class="error">{{ form.name.errors|join:", " }}</div>
{% endif %}

<tr class="fieldrow">
<th><label for="id_email">
Poster's Email:
{{ form.email }}<br>
{% if form.email.errors %}
<div class="error">{{ form.email.errors|join:", " }}</div>
{% endif %}

{{ form.body }}<br>
{% if form.body.errors %}
<div class="error">{{ form.body.errors|join:", " }}</div>
{% endif %}
What I really want is:
{% load formutils %}
{% fieldrow form "subject" %}Subject:{% endfieldrow %}
{% fieldrow form "name" %}Poster's Name:{% endfieldrow %}
{% fieldrow form "email" %}Poster's Email:{% endfieldrow %}
{% fieldrow form "body" %}{% endfieldrow %}
This would allow me to create new forms and form fields quickly. Naturally, there could be other custom tags besides just "fieldrow" for different style layouts. This is just the beginning. Well, I started by creating a custom tag:
"""These are custom tags for creating forms more easily."""

from django.core import template
from django.core.template import Context, loader

register = template.Library()

def fieldrow(parser, token):
"""This tag creates a table row with a form field.

Example: {% fieldrow form "name" %}Name:{% endfieldrow %}

form -- This is a formfields.FormWrapper object.
name -- This is the name of the field. It may be a variable or a quoted
string. If it is a quoted string, it may not contain spaces.
Name: -- This is the label for the field.

Note, I'll use <tr class="fieldrow">.

tag_name, form, fieldname = token.contents.split()
except ValueError:
raise template.TemplateSyntaxError(
"%s tag requires two arguments" % token.contents[0])
nodelist = parser.parse(('endfieldrow',))
return FieldRow(form, fieldname, nodelist)

class FieldRow(template.Node):

def __init__(self, form, fieldname, nodelist):
self.form = form
self.fieldname = fieldname
self.nodelist = nodelist

def render(self, context):
form = template.resolve_variable(self.form, context)
fieldname = resolve_variable_or_string(self.fieldname, context)
field = form[fieldname]
label = self.nodelist.render(context)
t = loader.get_template('bulletin/fieldrow')
c = Context({"form": form, "fieldname": fieldname,
"field": field, "label": label})
return t.render(c)

def resolve_variable_or_string(s, context):
"""s may be a variable or a quoted string.

If s is a quoted string, unquote it and return it. If s is a variable,
resolve it and return it.

if not s[0] in ("'", '"'):
return template.resolve_variable(s, context)
if s[-1] == s[0]:
s = s[:-1] # Strip trailing quote, if any.
s = s[1:] # Strip starting quote.
return s
This made use of a template:
{% comment %}
This template is used by the fieldrow custom tag to create a table row with
a form field.
{% endcomment %}

<tr class="fieldrow">
<th><label for="id_{{ fieldname }}">
{{ label }}
{{ field }}<br>
{% if field.errors %}
<div class="error">{{ field.errors|join:", " }}</div>
{% endif %}
Viola! I'm done. It works.

Now, without a doubt, writing that custom tag is a pain. The hard part is that I have to parse the contents of the custom tag, resolve variables, etc. It sure would be nice if the templating system did that stuff for me. I personally think it would be much nicer if I could simply create a block that takes arguments. Consider something like:
{% block fieldrow(form, fieldname, field) %}
<tr class="fieldrow">
<th><label for="id_{{ fieldname }}">
{{ yield }}
{{ field }}<br>
{% if field.errors %}
<div class="error">{{ field.errors|join:", " }}</div>
{% endif %}
{% endblock %}

{% fieldrow(form, "name", form.name) %}Subject's Name:{% endfieldrow %}
Sure, a template author might not be able to write that "fieldrow" block. The template authors I work with do understand code like this. If nothing else, a programmer could create a separate template containing that block. He'd be able to avoid duplicating code with very little work. Having the templating system handle parsing arguments and resolving variables would make this task much easier than creating a custom tag.

Monday, February 27, 2006

Python: Django-palooza

I think the Django guys have written some really nice software. It's good stuff. No, it won't cure cancer; it's not a CMS; it's not a shopping cart; but it does what it does well.

In fact, I must admit that I felt really at home using Django. Django + WSGI reminds me a lot of how Aquarium works. In many subtle places, the designs are very similar. There are some things I feel Aquarium does better, but I can readily admit there are some things that Django does better. Despite the fact that I too am a perfectionist, I think Django may be more polished--apparently, one perfectionist just can't code as fast as two perfectionists ;) Actually, I wish it was closer to meeting my needs (stackless Python, coroutines, a custom data store instead of an RDBMS, etc.), but I can appreciate it for what it is.

Since Django can't possibly meet everyone's needs--for instance, it doesn't make a good replacement for Linux or Vim (yes, that's a joke)--let me state some of the things that bothered me most about Django while I was using it to write a small application like this one.
  1. In my application, end users have a form that allows them to create new messages. I created a new message that had <script>alert('hi!')</script> for its subject. My code correctly escaped things. However, in the admin interface, I got an alert "hi" when I went to update the message. That means that you cannot safely use the admin interface to look at end user generated records. Otherwise, end users might inject JavaScript that causes the admin user's browser to "do bad things."

  2. In the templating language, if you use a variable that doesn't exist, it will be silently replaced by "". This is like Perl, and I really dislike it. I'd much rather have an appfault so that I can fix the problem.

  3. Certain filters take arguments. A filter argument looks like this: {{ bio|truncatewords:"30" }}. I personally don't like this syntax, but I understand that that's just a matter of taste.

  4. You can't treat templates like functions. "include" is not a valid replacement for functions because you can't pass arguments. That means that you can't call it twice passing different arguments (don't even think about trying recursion). I think that functions are the most basic tool for applying DRY (don't repeat yourself), and I think that DRY applies to HTML too! Redundant code is bad, period. Even if the guy getting paid to update it is getting paid less than a programmer, it still takes a long time to make changes to redundant code, and it's an error-prone process.

  5. Nor are custom tags a good replacement for functions. First of all, template authors don't have access to write them. Secondly, consider the following custom tag call: {{ create_form_field email "Please enter your email:" }} Did you know that it's up to the coder to parse that? The custom tag infrastructure can't even break that call into two arguments, let alone take care of unquoting the string. It doesn't even resolve the variables within the caller's context--you have to do that explicitly. Consider an exmaple straight out of the Django docs:
    <form method="post" action=".">
    <label for="id_name">Name:</label> {{ form.name }}
    {% if form.name.errors %}*** {{ form.name.errors|join:", " }}{% endif %}
    <label for="id_address">Address:</label> {{ form.address }}
    {% if form.address.errors %}*** {{ form.address.errors|join:", " }}{% endif %}
    <label for="id_city">City:</label> {{ form.city }}
    {% if form.city.errors %}*** {{ form.city.errors|join:", " }}{% endif %}
    <label for="id_state">State:</label> {{ form.state }}
    {% if form.state.errors %}*** {{ form.state.errors|join:", " }}{% endif %}
    <label for="id_zip_code">Zip:</label> {{ form.zip_code }}
    {% if form.zip_code.errors %}*** {{ form.zip_code.errors|join:", " }}{% endif %}
    <label for="id_place_type">Place type:</label> {{ form.place_type }}
    {% if form.place_type.errors %}*** {{ form.place_type.errors|join:", " }}{% endif %}
    <input type="submit" />
    Perhaps you'll disagree with me, but I think that code is verbose and redundant. I think the template author should have a way of factoring out the common bits (what a row looks like), but neither includes nor custom tags are suitable replacements. I'm going to make due with a custom tag, but I'm not happy about it.

  6. I like the urls.py stuff, but when you generate URLs, the structure of those URLs is hardcoded. That is, urls.py is used to parse URLs, but not to create them. If I suddenly want to change how my URLs work, I can't just go to urls.py and change it. I also have to change all the places where the URLs are generated.

  7. Blocks are not suitable replacements for functions either. For instance, you can't just call a block. You can only override a parent template's block. My parent template had a "title" block that the child templates were overriding. I wanted to call the "title" block somewhere else in the child template, but I couldn't. I felt like my hands were chained together.

  8. The builtin media server is only meant for development. I can respect that. It's not supposed to be efficient. However, it doesn't understand the IfModifiedSince header. Hence, media is re-downloaded on every page hit. (By the way, I'd be happy to donate this from Aquarium, but I fear the authors aren't interested in my contributions.)

  9. I see form = formfields.FormWrapper(manipulator, new_data, errors). This is similar to how FormUtil works in Aquarium, however, it would be better if the new_data argument could be a list of dicts, i.e. "Use the user's previous data if present, otherwise use the current value of the data from the database, otherwise use..." Using a list of dicts is a nice bit of flexibility.

  10. Why are the Django guys so rude to other programmers? (You'll note that this blog contains compliments as well as critiques.) Here are some reasons it's bad to be rude to other programmers:

    • Those other programmers are human beings. They have feelings too!

    • I can guarantee Django will one day be obsolete, and younger programmers will say rude things about the authors. It's best if we all just treat each other with respect all the time.

    • Being rude to other programmers turns valuable bug and patch submitters into competitors. That makes it a double no-no.

  11. I wonder why they picked Dojo Toolkit instead of MochiKit. Afterall, the MochiKit guy is definitely a fellow perfectionist, and the Dojo guy most definitely isn't (even though he's a really nice guy). Personally, despite the fact that I'm a perfectionist, I like Dojo because it's very complete (it has stuff that I need that MochiKit doesn't). I can only hope they picked Dojo for this reason. It'd be awful if they picked it simply because MochiKit is used by TurboGears.

  12. If the Django guys are perfectionists, why don't they follow Guido's style guide more closely? That kind of stuff will drive obsessive compulsive perfectionists like me crazy!

  13. Did I mention that it's nearly impossible to apply DRY to templates in Django?
The following are some small points that are fixed in Aquarium, but not in Django. Again, I'd be happy to offer code.
  1. The login cookie isn't being set correctly, because the domain of the cookie sent out by Django doesn't match the domain in your browser.
    Aquarium automatically respects the X_FORWARDED_HOST header.

  2. You have to restart the Web server everytime you make a change under mod_python.

  3. The broken pipe exceptions are irritating. They're easy enough to handle more gracefully.
So anyway, Django guys, if you're reading this, remember that I continue to have a lot of respect for what you've done. In general, it's good stuff.

Wednesday, February 22, 2006

Linux: Ubuntu 5.10 on Dell Inspiron B130

If you read my last post, you may be relieved to hear that I did not set my laptop on fire. Ubuntu is working great! It even supported suspend to disk out of the box! I haven't been able to do this in a good five years. The only mildly tricky thing was that I needed to follow the instructions on this forum in order to get it to support WXGA resolutions (i.e. 1280x800). I'll probably need to use NDIS to get the wireless card to work, but I haven't gotten to that yet.

I guess I should go write a Linux on Laptops page now.

Sunday, February 19, 2006

Linux: Fedora Core 4 on Dell Inspiron B130

Rant warning:

If you're thinking about buying a Dell Inspiron B130 for use with Fedora Core 4, my initial assessment is don't:
  1. If you use the automatic partitioning, you'll get an exception during install. This may be Fedora's fault, though, because I've seen this problem before.

  2. After the installation actually started installing RPMSs, it just froze and died. It may have been display related, but I'm not sure.

  3. I installed in text mode (HTTP install) with the bare minimal number of packages. But then, when it tried to boot, it froze on the line, "Initializing hardware... storage network".

  4. I know that the display is hard to get working, but I've heard it just comes down to getting the right mode line, "Modeline "1280x800" 80.58 1280 1344 1480 1680 800 801 804 827".
Well, I will continue the battle tomorrow. It's too late for me to return this laptop. I'm not a happy camper at this moment. Apparently, very few people have installed Linux on this laptop so far. It doesn't even have its own Linux on Laptops page yet. I might also try Ubuntu because some people have reported success.

By the way, remember how I said Dell agreed to pay me back a small amount of money to offset the Microsoft tax? The money never showed up in my account. If I am stuck running Windows on this thing, I may just set fire to it and let it burn--not that I'm bitter or anything ;)

Tuesday, February 14, 2006

Haskell: Haskell for C Programmers

I'm just starting to read Haskell for C Programmers, and I can already tell that it is very helpful. If only I had seen this four years ago!

Haskell: Tour of the Haskell Syntax

This Tour of the Haskell Syntax is frickin' awesome! It's not enough to learn Haskell, but it sure is helpful to step back and see so much of Haskell explained succinctly.

Thursday, February 09, 2006

Python: Recursion in Django Templates

Whenever learning a new Web technology, I like to write a bulletin board application. It's easy, I have it memorized, and it gives me a chance to compare and contrast the new technology with what I already know. See here for the application written in Aquarium.

The thing that was hardest about using Django templates to write this application was the lack of functions and recursion in the templating engine. Recursion is really suitable for generating a hierarchy of messages. This post is about how I had to work around the problem. I got it to work, but it wasn't what I'd call elegant.

To use Django to generate a page that looks like this, you break it up into a few steps. First, you create a template that has everything other than the hierarchy of messages. I called it "index.html":
{% comment %}
Output the messages, and output a message input form.
{% endcomment %}

{% extends "bulletinboard/vertical" %}

{% block vertical_body %}
<div class="container">
<div style="float: right">[<a href="">Start a New Thread</a>]</div>
<p><b>List of Messages:</b></p>
{% if not messages %}
<p>There are no messages yet.</p>
{% else %}
{{ messages }}
{% endif %}
</div><br />
{% endblock %}
Secondly, you pass the pre-generated hierarchy of messages as a chunk of HTML to the first template. Hence, in my view, I have code:
    return render_to_response('bulletinboard/index', 
{'messages': show_message_hierarchy()})
where "show_message_hierarchy()" generates the big chunk of HTML. Next, I created a template, "showmessagehierarchy.html", that gets called recursively (although, naturally, it can't call itself recursively in Django):
{% comment %}
This template is called recursively to output the bulletin board messages in
a nice hierarchy.
{% endcomment %}

{% for child in children %}
<a href="/messages/{{ child.id }}/">{{ child.subject|escape }}</a>
{{ child.recurse }}
{% endfor %}
Note, the recursion happens with "child.recurse", which is a callback that the view sets up. Each child has a different callback--it's not possible to say "recurse(child)" because there are no function arguments in Django (unless you use a custom tag). "child.recurse" is not a method of "child"; it's just a function that I've attached to "child". Now, let's look at the view that pulls everything together:
def index(request):

"""Put the list of messages in a nice hierarchical format."""

def show_message_hierarchy(parent=None):
"""Show the bulletin board messages in a nice hierarchy."""
children = message_dict.get(parent, [])
if not children:
return ""
c = Context({"children": children})
return template.render(c)

def create_recurse_callback(id):

"""This is a closure.

It curries the id for show_message_hierarchy. The "partial" function
doesn't exist yet in Python 2.4.


def inner():
return show_message_hierarchy(id)

return inner

message_list = messages.get_list()
message_dict = {}
for message in message_list:
message.recurse = create_recurse_callback(message.id)
message_dict[message.parent_id] = message_dict.get(message.parent_id, []) + [message]
template = loader.get_template('bulletinboard/showmessagehierarchy')
return render_to_response('bulletinboard/index',
{'messages': show_message_hierarchy()})
Here you can see "child.recurse", i.e. "message.recurse", getting setup: "message.recurse = create_recurse_callback(message.id)". "create_recurse_callback" is a closure that is used to curry the "message.id" so that you can call "show_message_hierarchy" without needing to pass an argument. I got that trick from function programming ;) Notice that, basically, the "show_message_hierarchy" function and the "showmessagehierarchy" template recursively call each other.

When I did this in Cheetah, I just had the main template, and another function in the template that could call itself recursively. Nonetheless, I was able to get it to work in Django, albeit in a non-elegant fashion.

Python: Django Templates

"As simple as possible, but no simpler."

I have a lot of respect for what the Django guys have done. I actually really like a lot of Django. However, having tried it, I find the templating engine to be elegant, but too simple. What frustrates me most is the lack of functions. In my first programming class, I learned how to use functions for code reuse. DRY applies even to HTML. At my company, even the "template authors" know how to create a function to avoid code duplication. Nor is the "include" or "block" statement a suitable replacement--functions have parameters. Different arguments lead to different results. This is necessary functionality. Also, functions should support recursion. I learned in my second programming class how to think recursively, and every once in a while it really comes in handy (for instance, when you're writing a bulletin board application with a hierarchy of messages).

When I was using Django templates, I really felt like my hands were tied. I don't think lack of functions improves security. I don't think custom tags are always a suitable replacement--I'm talking about simple HTML reuse. Please forgive my boldness, but I don't think I should have to live without functions just because a template writer working at some other company doesn't understand them.

Next, I think I'll try out using Django with Cheetah. So far, Cheetah has always met my needs exceedingly well. Hopefully, I can get it to fit in smoothly, maintaining Django's overall elegant feel.

Wednesday, February 08, 2006

Python: Clearing sys.modules of Stale Modules

I'm posting this here in case someone might find it useful. I also submitted it to the Cookbook.
"""Clear ``sys.modules`` of specific types of modules if one is stale.

BASIC IDEA: Clear ``sys.modules`` of stale code without having to restart your
server. It's a hell of a lot harder to do right then it sounds.


__docformat__ = "restructuredtext"

_lastModuleUpdate = time.time()
def clearModules():
"""Clear ``sys.modules`` of specific types of modules if one is stale.

See ``properties.CLEAR_MODULES``.

I took this method out of the ``InternalLibrary`` class so that you can
call it *really* early, even before you create a ``Context`` to pass to


The problem that this method solves is simple: if I change a file, I don't
want to have to restart the server. It's a simple problem, but it's tough
to implement right. To prevent repeating mistakes, here's what has failed
in the past:

* Remove all modules from ``sys.modules`` on every page load.

- Some modules have state.

* Delete only those modules that don't have state.

- There's no convenient way to know which ones have state.

* Use module attributes.

- It's not convenient.

* Delete only those modules that have grown stale.

- If a parent class gets reloaded, child classes in other modules will
need to get reloaded, but we don't know which modules those classes are

* Look through all the modules for module references to the modules
that'll get deleted and delete those too.

- Its very common to only import the class, not the whole module. Hence,
we still don't know which modules have child classes that need to get

* Just clear out ``sys.modules`` of all modules of certain types on every
page load.

- Even a moderate amount of kiddie clicking will result in exceptions.
I think the browsers hide the problem, but you'll see the exceptions
in the logs.

* Clear out ``sys.modules`` of all modules of certain types on every page
load, but only if at least one of the modules is stale.

- This is good because it handles the kiddie clicking case, and it also
handles the super class case.

global _lastModuleUpdate
if not properties.CLEAR_MODULES:
deleteTheseTypes = properties.CLEAR_MODULES
if not isinstance(deleteTheseTypes, list):
# Update Seamstress Exchange's properties file if you change this.
deleteTheseTypes = ["aquarium.layout","aquarium.navigation",
"aquarium.screen", "aquarium.widget"]
deleteThese = [
for moduleType in deleteTheseTypes
for moduleName in sys.modules.keys()
if (moduleName == moduleType or
moduleName.startswith(moduleType + "."))
for i in deleteThese:
file = sys.modules[i].__file__
except AttributeError:
if file.endswith(".pyc") and os.path.exists(file[:-1]):
file = file[:-1]
if (_lastModuleUpdate < os.stat(file)[ST_MTIME]):
staleModules = True
staleModules = False
if staleModules:
for i in deleteThese: # You can't modify a dictionary
del sys.modules[i] # during an iteration.
_lastModuleUpdate = time.time()