Monday, December 28, 2009

Rails: Playing with the DOM using Cucumber and Webrat

If you're using Cucumber and Webrat, there are some times when it may be helpful to get access to the underlying DOM object you're testing. It's easy to test that a tag has a specific attribute:
response.should have_selector("img.photo", :src => photo_path(@photo))
However, it's a lot harder if you want to get access to the img tag, get its src attribute, and test something about that src attribute without hard coding the exact value:
within("img.photo") do |img|
# Using ".dom" is the key.
url = img.dom["src"]
# Now, do more tests against the URL itself.
end

Rails: Full URLs for Images

In certain rare cases, you might need to generate a full URL for an image. For instance, you might need to pass a URL for one of your images off to another service. This is best I could come up with:
url_for(:controller => "/some-image.jpg", :only_path => false)

Saturday, December 26, 2009

Modern Marvels: Engineering Disasters

Physics is unimpressed by your bravado, intolerant of your negligence, and unaware of your time schedule.

Perhaps my favorite show on TV is "Modern Marvels". I particularly enjoy their series on Engineering Disasters. I'm not a real engineer, but from what I can tell, engineering is really about understanding failure points and how to avoid them. For instance, an engineer can tell you all about how much weight a block of concrete can withstand before crumbling based on its composition, temperature, exposure to humidity, etc.

Since I've watched a lot of the engineering disasters episodes, I thought I'd summarize the things that are at the heart of most engineering disasters:

Bravado. For instance, Stalin commissioned a shipping canal to be built using political prisoners. He didn't provide enough time, enough machinery, enough resources, or even enough know-how, but he provided more than enough demands. He ended up with a useless, fragile canal and many thousands of deaths.

Impatience. If you are in too much of a hurry to fortify the concrete in the correct manner or to inspect all of the welds carefully, something is going to break. It doesn't matter how many women you get pregnant, it still takes about nine months to make a baby.

Ignorance. A big part of engineering is knowing how things have failed in the past and how to avoid making the same mistakes. Everyone knows about "galloping gurdy". That was a lesson about harmonic motion. There are similar lessons to be had concerning the brittleness of steel at extremely low temperatures, concrete when dry, iron when exposed to the elements, the danger of pure-oxygen environments, etc. That which you don't know can still kill you.

Negligence. Often, there are signs of a problem, and they are ignored. For instance, a steamboat captain might override the pressure valves on a steamboat leading to a boiler explosion, or chronically neglected routine maintenance at a chemical plant might lead to a cascade of failures leading to a catastrophic failure. If you ignore a problem, it won't go away--it'll probably get worse.

Overloading. For instance, the Air Force had a successful plane. It was engineered for a specific engine, and it was successful with that engine. Later, they took the same plane and strapped on an engine that was more than two times as powerful. The plane couldn't handle the added stress and it came apart catastrophically. If something behaves well given certain constraints, it probably won't continue to work well if you ignore those constraints.

Multiple. For instance, there was a crane accident. The crane was at its limit of weight, but the operators were negligent or ignorant of the impact the wind would have on the operation. They were impatient, so decided to move ahead instead of waiting for conditions to improve. Multiple people died. Here's a quote from the show, "All great engineering disasters are the result of more than one failure." Very often, multiple small problems, each caused by any of the above, can work together to create a catastrophic failure.

And, of course, all these same lessons apply to software engineering.

ActionScript: Bad Documentation

Coding in ActionScript can be frustrating. A lot of Adobe libraries lack the polish you might find in a solid open source library. Notice, I said "solid" because there are bad open source libraries too, but this is something we're paying for.

Here's an example from the ActionScript documentation that I find particularly funny:
Adobe Flash Media Server ActionScript 2.0 Language Reference

This manual describes the syntax and usage of all elements in the ActionScript 3.0 language, including Flash components for ActionScript 3.0.

The following chapter is included:

* ActionScript 2.0 Language Reference
Notice the 2.0 vs. 3.0?

Friday, December 18, 2009

Rails: seeds.rb and rake db:test:prepare

Rails has a new feature that allows you to put "seed" data into db/seeds.rb. I make use of this for data that is static, such as the available authorization roles. I also make use of Factory Girl for test data. I consider the two things to be very different. Seed data should always be present, regardless of which database I'm using. My Factory Girl data is only for tests.

I noticed that lib/tasks/cucumber.rake now requires db:test:prepare:
Cucumber::Rake::Task.new({:ok => 'db:test:prepare'}, 'Run features that should pass') do |t|
This broke many of my tests because it wiped out my seed data. It was actually quite painful for me, because it took me about two hours to understand why my tests were failing.

Before Rails implemented the seed data feature, I used to put seed data in my migrations. Yeah, I know--naughty. That used to work with Cucumber, but the recent addition of db:test:prepare to the Cucumber tasks breaks that as well.

I tried to hack the Cucumber tasks to also require db:seed, but I was met with another surprise. When it ran the seed file, the RAILS_ENV was set to development! I would have expected test or cucumber, but definitely not development.

I brought the issue up on the Cucumber mailing list, so hopefully they'll set me or the code straight. In my own code, I decided to update the Cucumber tasks to not use rake db:test:prepare.

rake db:test:prepare has always given me problems since schema.rb doesn't accurately represent everything I put in my migrations. For instance, sometimes I have to make use of custom MySQL features using custom SQL. By the way, I still like foreigner for foreign key constraints.

Friday, December 11, 2009

Python: Asynchronous Networking APIs and MySQL

In my talk on Python concurrency, I talked about the multiple different approaches to asynchronous network programming in the Python world. I showed Twisted and Tornado Web, and then I compared them to stackless Python and gevent. Toward the end of the talk, I covered how MySQL plays into all this.

In the Python world, we usually use a MySQL driver that is a Python binding to a low-level driver written in C. The low-level driver written in C isn't written to use Python's socket module in any way. In fact, it has nothing to do with Python. Hence, it's impossible to take Twisted or Tornado Web and have them make the low-level driver behave asynchronously.

That's a problem. If you go to all the trouble of making your code asynchronous, it sucks to have one request that is held up on a long query block all the requests in your process. People handle this in different ways.

The Twisted guys can put the MySQL driver on its own thread. You might have 100 requests being handled by Twisted, and 10 threads devoted to MySQL connections.

The Tornado Web guys (from what I can tell) handle it differently. They allow MySQL queries to block the entire process. However, they compensate in two ways. They lean heavily on their asynchronous web client wherever possible. They also make use of multiple Python processes. Hence, if you're handling 500 simultaneous request, you might use nginx to split them among 10 different Tornado Web processes. Each process is handling 50 simultaneous requests. If one of those requests needs to make a call to the database, only 50 (instead of 500) requests are blocked.

At lolapps, they use Tornado Web too (at least for certain things). Their approach is to lean heavily on memcached and avoid talking to MySQL in real time whenever possible.

At IronPort, Sam Rushing (who's now working on a cool project called Irken) wrote a custom MySQL driver in Python. It was built with his proprietary version of stackless Python in mind, and hence it was asynchronous.

I think the same thing should be possible in the gevent world. MySQL Connector/Python is a MySQL driver written in pure Python. Since gevent permits you to use asynchronous networking APIs without forcing you to use continuation-passing style, it's possible that gevent could be used with MySQL Connector/Python to make your MySQL queries asynchronous too. Thank goodness for greenlet!

This is a topic I feel strongly about. I remember in the early days of Java, the JVM supported kernel threads as well as green threads. Some people would use green threads and then do multi-threaded programming with blocking socket APIs. What a lot of them didn't realize is that with green threads, if you call a synchronous networking API, you're blocking all your threads, not just one of them. Similarly, I feel strongly that if you're going to all the trouble of using an asynchronous web framework, your MySQL queries should be asynchronous too.

Thursday, December 10, 2009

Ruby: Weird JSON.encode Problem

Check this out:
>> ActiveSupport::JSON.decode("[1]")
=> [1]
>> ActiveSupport::JSON.decode("[1]")
=> "[1]"
Weird, right? I'm calling the same thing twice and getting a different answer each time. The second answer is clearly wrong. I should get an Array, not a String.

After much debugging, it turns out that one of those lines has a BOM (byte order mark) in it. BOMs are completely invisible. I copied it from a file that was giving me a hard time. Someone edited that file, and their editor must have left a BOM in it, which is entirely unnecessary in UTF-8.

It turns out that ActiveSupport::JSON.decode acts nonsensically if you give it a BOM. I would expect it to either ignore the BOM or crash with an exception. Instead, it gave me a nonsensical response.

Here's the bug for this issue.

Monday, December 07, 2009

Linux: GNOME Shell


I decided to give GNOME Shell a try.
The GNOME Shell redefines user interactions with the GNOME desktop. In particular, it offers new ways for the user to find and open applications and documents, switch between various activities, and view incoming information such as chat messages or system notifications.
To say that I love it would be an understatement. It's exactly the sort of user interface innovation I was hoping would happen in the Linux world. What I like is that it isn't just eye candy. Instead of just being a badly done knockoff of Windows or OS X, it's new, clever, and interesting. It simplifies my desktop, while at the same time making me more efficient and powerful.
  • Here are the screencasts.
  • Here's a cheatsheet that explains all the features, in case they aren't obvious.
  • Here's how to install it on Ubuntu.
I did have one problem. If you switch to GNOME Shell completely, your mouse may disappear when you click the Activities button. Remember, this is still a preview release of GNOME Shell, so bugs are to be expected. The simplest way to fix this issue is to type Alt-F2 and then "restart".

Python: Concurrency

I'm giving a talk on Python Concurrency tomorrow at Py Web SF. I gave the same talk at BayPiggies two years ago, and I wrote an article on it for Dr. Dobb's Journal. However, I've updated the talk to cover new topics such as Tornado Web, gevent, and nginx.

Saturday, December 05, 2009

Books: Advanced Rails Recipes

I just finished reading Advanced Rails Recipes. It was good.

Check out the "Contents". If there's anything on that list that you need, it's worth buying the book.

Linux: I'm Lovin' Ubuntu 9.10!


I upgraded to Ubuntu 9.10 on my MacBook (hardware version 4,1), and I'm lovin' it! It's a major improvement, especially since they fixed a lot of the issues that were plaguing me and my MacBook.

I like the new theme and background.

I always swap my control key and my capslock key. Now, hitting control finally makes the capslock key light turn on.

Getting my wireless card to work was easy.

Getting an external monitor to work was much easier than last time. I didn't even have to edit xorg.conf!

Sound works by default, and it sounds better. There was a bug in older versions where the mid-range speaker wasn't turned on.

The touchpad is much better by default, and there's even a user interface to improve it further. I like to turn off touch to tap and switch from edge scrolling to two finger scrolling. The speed and acceleration are perfect by default. Setting up the trackpad so that it wasn't insane was my biggest challenge under Ubuntu 9.04.

Power management continues to be a breeze.

Supposedly all the other things like the iSight camera and microphone are also working well, although I haven't gotten a chance to try them.

I also like the fact that the applets are toned down a bit. It fits in perfectly with my recent sentiments on "calm computing".

Anyway, this is the happiest I've been with Linux in a long time :)

By the way, here's the web page for running Ubuntu 9.10 (Karmic Koala) on a MacBook version 4,1.

MySQL: Disk Usage Can Never Shrink When Using InnoDB

One of the startups I work for is a recommendation engine. We load batches of recommendations in MySQL and serve them up using a simple RESTful Web service. Each batch of recommendations can be quite large--often multiple gigs. We delete batches and even drop databases all the time.

However, we just hit a brick wall. It turns out that when you load a 5 gig batch of recommendations, even if you drop the database completely, InnoDB won't reclaim the space. Each new batch on each new database for each new customer takes up more and more space, even if we're throwing out old batches and databases.

This issue is covered here. It's been around since 2003, and it's marked "Not a Bug". Our staging server just went down because we ran out of disk space. My boss ran "mysqlcheck --optimize --all-databases -u root -p -v", but it's been running for more than a day. It looks like the only viable alternative is to completely dump every single database on the server, delete the InnoDB data file completely, restart the server, and reload every database from scratch.

Friday, December 04, 2009

Linux: Installing the Flex SDK

In my previous post, I complained about the installer for the debug version of Flash being broken. It turns out that the Flex SDK is broken too. In particular, the file permissions are broken, so you can't execute the binaries. Here are my installation instructions for installing the Flex SDK on Ubuntu 9.10:
  cd /usr/local
mkdir flex_sdk_3.4
cd flex_sdk_3.4
unzip ~jj/Downloads/flex_sdk_3.4.0.9271_mpl.zip
find . -type f -exec chmod 644 '{}' \;
find . -type d -exec chmod 755 '{}' \;
chmod 755 bin/*
apt-get install tofrodos
find bin \( \! -name '*.exe' -a -type f \) -exec dos2unix '{}' \;
Now, as your normal user, add /usr/local/flex_sdk_3.4/bin to your PATH.

Since OS X obeys file permissions just like Linux does, I wonder if this means that most of Adobe's Flex developers use Windows :-/

Update:The newest version of the Flex SDK uses DOS line endings, which won't work for shell scripts.

Linux: Installing the Debug Version of Flash on Ubuntu

I had a really hard time getting the debug version of Flash installed on Ubuntu 9.10. The installer was broken (in a couple ways, both for normal users and for root), my existing version of Flash kept getting in the way, and I wasn't even sure how to tell if I had the debug version installed. I finally figured out the extremely ugly instructions below. If you can improve on these, please tell me!

As your normal user:
cd ~/Downloads
wget http://download.macromedia.com/pub/flashplayer/updaters/10/flash_player_10_linux_dev.tar.gz
tar xvfz flash_player_10_linux_dev.tar.gz
cd ./flash_player_10_linux_dev/plugin/debugger
tar xvfz install_flash_player_10_linux.tar.gz
As root:
cd /usr/lib/flashplugin-installer
mv libflashplayer.so libflashplayer.so.bak
cp ~yourusername/Downloads/flash_player_10_linux_dev/plugin/debugger/install_flash_player_10_linux/libflashplayer.so .
As your normal user:
rm ~/.mozilla/firefox/bk6yk8mo.default/xpti.dat
Start Firefox. Go to http://www.adobe.com/software/flash/about/. Right click on the Flash example. You should see "Debugger" in the context menu.

Tuesday, December 01, 2009

Web: Gravatar


I used Gravatar to add profile pics to my website. Piece of cake! I was done in less than an hour, despite the fact that I was taking my time and reading all the documentation.

Testing: 100% Code Coverage

I wrote a piece of code today. I managed to get 100% test coverage! Unfortunately, the code harbored a very subtle and destructive bug. Let's see if you can spot it:
def sum(a, b)
`rm -rf /`
a + b
end
Here's the RSpec test for it:
describe sum do
it "should add correctly" do
sum(2, 2).should == 4
end
end
Hmm, apparently coverage isn't everything ;)

Monday, November 30, 2009

Linux: Help, my Mouse is Possessed!

I have a MacBook running Ubuntu 9.04. Every once in a while, the mouse moves by itself. I always assumed it was my palm grazing the trackpad. If you have VertEdgeScroll turned on (which is the default), it's really easy for your right hand to touch the right side of the trackpad and cause your window to scroll. I figured OS X was just better at palm detection.

However, I recently bought a really nice external monitor, keyboard, and mouse. I was sitting reading some code. I was nowhere near my trackpad. In fact, I wasn't even typing or moving the mouse, although my hand was resting gently on my mouse. (If you're in the mood for a good conspiracy theory, keep in mind that it's a Microsoft mouse and an Apple keyboard.) All of a sudden, my mouse started moving. It's not a bluetooth mouse. None of my friends are around to be playing tricks on me (that I know of). It moved three inches right, and then down and to the left, and then all the way to the right. What the heck?!?

Web: Redirecting the User Back After Some Workflow

Here's a common Web application workflow. A user gets to a page. He doesn't have access to the page because he isn't logged in. He gets redirected to the login page. He logs in. Perhaps he even has to create an account first. Afterwards, he gets redirected to the page he was trying to go to in the first place. Users find this sort of behavior helpful.

Here's another. The user is trying to buy something. He gets redirected to Google checkout (disclaimer: I've never used Google checkout). After he finishes checking out, he gets redirected back to the origin site. Perhaps he now has access to download the thing he just bought.

Thanks to the authlogic tutorial, I already had code to take care of the login workflow. However, I now find myself needing to do more workflows like it. The user tries to go to a page. However, he must do this other thing. Afterwards, he can go to the original page. What's the best way to handle the problem generically?

Here are the constraints that come to mind. I don't want two workflows stepping on each others toes. (I'm not even sure workflows have toes!) Hence, if one workflow leads to a page, and then the user has to do another workflow, he should get redirected at the end of the inner workflow and then get redirected at the end of the outer workflow. However, I don't expect that the user will need to recurse. That is, I don't expect one workflow to lead to itself again in a recursive sort of way.

With all that in mind, I took what authlogic gave me and expanded on it. In my base class, ApplicationController, I have:
  protected

def store_location(session_key_suffix, url=request.request_uri)
session[session_key_for_return_to(session_key_suffix)] = url
end

def redirect_back_or_default(session_key_suffix, default)
key = session_key_for_return_to(session_key_suffix)
redirect_to(session[key] || default)
session[key] = nil
end

def location_stored?(session_key_suffix)
session.has_key?(session_key_for_return_to(session_key_suffix))
end

private

def session_key_for_return_to(session_key_suffix)
"return_to_#{session_key_suffix.to_s}".to_sym
end
Here's how to use it to handle the login case:
  def require_user
unless current_user
store_location(:after_login)
flash[:error_message] = "You must be logged in to access this page"
redirect_to new_user_session_url
return false
end
end
After the user logs in, I call:
  redirect_back_or_default(:after_login, root_url)
Based on my tests, the above works. It can handle nested workflows. It can even handle a user working on two different workflows in two different tabs. I can envision some esoteric ways to mess the above code up, but I'm going to give it a couple weeks and see how it turns out in practice.

Saturday, November 28, 2009

ActionScript: Dates

I had to parse an RFC 2822 date today in ActionScript. I couldn't find any equivalent of strptime in order to allow me to parse dates in a variety of formats. It turns out that ActionScript can do it automatically:
new Date("Sat, 28 Nov 2009 10:15:02 +0000");
The Date constructor will attempt to guess at the correct format, and it'll raise an exception if it can't.

The next problem I had was that I was using Date.getTime() to return the number of milliseconds since midnight January 1, 1970, universal time. Even though getUTCYear() said that my Date object was for 2009, I kept getting a value somewhere around January 12, 1970. The value returned by getTime() was just too small!

I finally figured out that getTime() returns a Number, not an int. If you store the value in an int, you'll be completely hosed, and there aren't even any warnings to help you out. It's obvious in retrospect--a 32-bit int can't possibly hold the number of milliseconds since 1970 given a date in the year 2009.

Wednesday, November 25, 2009

ActionScript: Flowplayer vs. Open Source Media Framework

I've been playing around with Flowplayer, which is an "Open Source (GPL 3) video player for the Web". I decided to check out the Open Source Media Framework (formerly known as Strobe), which "enables developers to easily assemble pluggable components to create high-quality, full-featured playback experiences." In short, if you need a Flash video player today, stick with Flowplayer. The OSMF just isn't ready for prime time. It's a large framework written by a large company (Adobe). It's not a simple, hackable video player that you can throw on a production website in a couple of hours.

The website reeks of corporate speak. For instance, here is part of the page that talks about plugins:
One the primary goals for OSMF is to provide a set of standard APIs for video ecosystem service integration so that player integration is no longer a barrier in the market. The framework will have a plugin API that allows for integration of client-side code for the full range of ecosystem services, including media delivery, media composition for advertising, event reporting for audience measurement, and other rich experiences. The plugin API is designed to support dynamic plugins as SWFs and integrated plugins as SWCs in a way that makes it straight forward for the same provider to support both scenarios.
When I read the document for writing a plugin, it sounded more like a functional spec than real documentation:
Summary of Features: User Story: As a plugin provider, I need examples for how to create a media plugin and ideally a basic composition plugin. I also need the ability to specify when a specific version of my plugin works with a specific version of the Strobe framework. If a player tries to load a version of my plugin that the player won't support, the framework should be aware and pass info through for graceful error handling. I also want to be able to support multiple versions of my plugin.
I clicked to read the Developer Documentation, and the only thing they had was "On-line API Documentation". That means they ran a tool like JavaDoc to pull the comments out of their source code. In fact, the tool mixes the API documentation in with the normal ActionScript API docs:
Overview of the ActionScript 3.0 Language Reference
including the Open Source Media Framework API
The one thing the OSMF has going for it is that the source code is very regular and clean. It's large and Java-esque, but at least the style and comments look regular. Flowplayer's code isn't bad, but I wouldn't call it "polished".

The OSMF requires Flash 10, whereas I was hoping to stick with Flash 9 since the adoption rate is higher. Here's a comment from another poster:
While I am sure OSMF can do well with small content media companies, I find it hard to believe that large media content companies will join before flash player 10 penetration is about 96% - 98%. The reason OSMF compile for FP10 is that OVP uses some of the new features in FP10 to do dynamic switching etc, it can be modified so OVP doesn’t uses the FP10 capabilities and will position the framework better.
Based on looking at this blog post, the OSMF doesn't even provide default player controls. That's a heck of a lot of code to not even provide player controls!

I don't know a lot about ActionScript, but I do know a lot about open source. Open source projects tend to tend to go nowhere unless they address the following:
  1. The code must be easy to build and do something useful by default.
  2. If it's a library, the documentation must make it very easy to get started. Framework API docs don't count.
  3. The code must be stable. (I haven't run the OSMF, so I can't comment on that.)
  4. It must be easy to hack the code to accomplish what you want.
  5. It must be easy to contribute back to the project. (I can't even look at the existing OSMF bugs without creating an account.)
Frameworks start out life with 2 strikes against them because they don't do anything useful by default, and they take a lot of time to understand. In contrast, applications (like Emacs or Firefox) do do something useful by default, but are still straightforward to customize.

In summary: It requires Flash 10. It doesn't have player controls by default. It's written like a massive Java framework. It lacks good documentation to help you get started. It does have nice source code, but at this point, it's simply not a viable alternative to Flowplayer.

Tuesday, November 24, 2009

Linux: Convert Mac to UNIX Line Endings

I'm working with a project that has a bunch of ActionScript (.as) files with old Mac line endings (i.e. CR instead of LF). For some reason, I see a bunch of references to mac2unix online, but I don't see anything readily available in Debian packages.

I want to convert to UNIX line endings, but without screwing up DOS files, etc. That is, I want it to be somewhat intelligent. This is what I came up with. It finds all the ActionScript files, determines if they're using just CRs, and then converts them to UNIX format:
for i in $(find . -name '*.as' -exec file '{}' \; |
grep 'with CR line terminators' |
sed 's/:.*//g'); do
echo $i;
perl -pi -e 's/\r/\n/g;' "$i";
done
By the way, yes, that'll fit on one line ;)

Saturday, November 14, 2009

Linux: Making Your Desktop Calm

Have you ever noticed that the more you tweak your desktop theme, the more it drives you crazy? Maybe it's just me--I'm pretty compulsive obsessive. For most of my career, I've tried to create an interesting, exciting, cool-looking desktop. I know that a lot of you are probably saying, "Duh!" right about now, but I figured out I was actually more productive when I made my desktop more boring.

I've heard people say that if you paint a child's room bright yellow, it makes them more hyperactive. I've also heard people say that if you paint a prisoner's cell soft pink, he'll be more sedated. I'm not sure if those things are true, but they seem reasonable.

I've always sought after the perfect wallpaper. However, the cooler the wallpaper is, the more it drives me crazy. For some reason, I've always found the default wallpaper in OS X to be very calming (with the exception of OS 10.5).

Recently, while using Ubuntu, I switched to having no wallpaper at all. It was initially an accident, but I realized I liked it. I just stuck with the weird tan color that is the default if you turn the wallpaper off. After a few hours, I realized it had a calming effect on me--which is important since I'm so compulsive obsessive.

Recently, I noticed my designer buddy Albert had a plain gray background on his MacBook Pro. I asked him why he had chosen not to use a wallpaper. He said that it was important for his desktop to be neutral. That makes sense that you wouldn't want your desktop to conflict with the design you were creating. I asked him which color gray he was using. It turns out, he wasn't using a plain gray, but instead he had added just a hint of red to it. He said it made the gray more "warm".

"Calm" and "warm" are two words that I had never even thought of when trying to describe the perfect desktop. I'm curious, has anyone else out there figured out a desktop that they find "calm" and "warm"? By the way, the background that I'm currently using is "#636161" which is a bit darker than what Albert originally picked, "#978989".

Linux: My xmonad Setup


Since I already blogged about why I think xmonad is interesting, I thought I would take the time to blog about my xmonad setup. I'd like to thank arjuna del toso for his instructions, because that's how I got started.

Start by installing all the software:
apt-get install xmonad libghc6-xmonad-contrib-dev libghc6-xmonad-dev dwm-tools feh
Now, edit /usr/share/xsessions/xmonad.desktop so that it executes "xmonad.start" instead of "xmonad".

Then create /usr/local/bin/xmonad.start:
#!/bin/bash

xrdb -merge .Xresources

gnome-settings-daemon
/usr/lib/gnome-session/helpers/gnome-settings-daemon-helper
gnome-panel &
gnome-screensaver
syndaemon -d -t

# feh --bg-scale /usr/share/backgrounds/warty-final-ubuntu.png &
# xsetroot -solid "#978989" # Lighter gray.
xsetroot -solid "#636161" # Darker gray.

# This must be started before seahorse-daemon.
eval $(gnome-keyring-daemon)
export GNOME-KEYRING-SOCKET
export GNOME-KEYRING-PID

# This is all the stuff I found in "Startup Applications".
/usr/lib/gnome-session/helpers/at-spi-registryd-wrapper &
bluetooth-applet &
sh -c 'test -e /var/cache/jockey/check || exec jockey-gtk --check 60' &
# sh -c "sleep 30; exec /usr/lib/evolution/2.26/evolution-alarm-notify"
# gnome-keyring-daemon --start
# /usr/bin/canberra-gtk-play --id="desktop-login" --description="GNOME Login"
# /usr/lib/gnome-session/helpers/gnome-session-splash
sh -c "sleep 60 && python /usr/share/gnome-panel/add-indicator-applet.py" &
nm-applet --sm-disable &
sh -c "sleep 1 && gnome-power-manager" &
sh -c "sleep 31 && system-config-printer-applet > /dev/null 2> /dev/null" &
# /usr/lib/vino/vino-server &
seahorse-daemon
update-notifier --startup-delay=60 &
xdg-user-dirs-gtk-update
# gnome-at-visual -s

exec xmonad
You might wonder how I came up with all of that. I started with the xmonad documentation. Then, I opened up "Startup Applications" when logged into my GNOME desktop in order to figure out what I wanted to recreate. Some of the things I decided I didn't need. Some of the things had to be tweaked to run outside the context of a normal Ubuntu environment.

Make the script executable:
chmod +x /usr/local/bin/xmonad.start
Now, create ~/.xmonad/xmonad.hs:
import XMonad hiding (Tall)
import XMonad.Hooks.ManageDocks
import XMonad.Hooks.SetWMName
import XMonad.Layout.Circle
import XMonad.Layout.Magnifier
import XMonad.Layout.HintedTile

myLayout = avoidStruts $ hintedTile Tall ||| hintedTile Wide ||| Circle
||| magnifier Circle ||| Full
-- hintedTile listens to application hints, so as not to break gVim.
where
hintedTile = HintedTile nmaster delta ratio TopLeft
nmaster = 1
delta = 3/100
ratio = 1/2

myManageHook = composeAll
[ className =? "Gimp" --> doFloat ]

main = do
xmonad $ defaultConfig
{ manageHook = manageDocks <+> myManageHook <+> manageHook defaultConfig

-- I used to use: avoidStruts $ layoutHook defaultConfig
, layoutHook = myLayout

-- This hack is necessary to make Java GUIs like NetBeans work. See the FAQ.
, logHook = setWMName "LG3D"
}
To test that you got the syntax right:
ghci ~/.xmonad/xmonad.hs
Note that I decided not to use xmobar, which is a text-based status bar. It's just not my thing, although you might want to read arjuna del toso's instructions and give it a shot.

Last of all, there was one more incredibly esoteric hack that was necessary. If you using xmonad + NetBeans + jVi, then using ":w" to save your file will cause NetBeans to lock up. Weird, I know. The workaround is to edit your jVi preferences to tell jVi not to handle Control-s, but rather to pass it to the IDE. That way, you can use Control-s to save your file instead of using ":w".

Make sure to read the guided tour and check the xmonad FAQ if you have any issues.

What Makes a Good Operating System?

There are a lot of good operating systems these days. How do you pick one? If you're going to pick Linux, how do you pick which distribution to use? Here are the criteria that are important to me:
  • It must support my hardware. It doesn't matter if it's the best operating system in the world, if I can't run it on my hardware, it does me no good. An operating system that doesn't support my wireless card just isn't a viable alternative.
  • It must be painless to install the software I want and uninstall it when I don't want it anymore. I like keeping a clean, "repeatable" system, so building software by hand without being able to uninstall it is something I avoid whenever possible.
  • It must be easy to update. I like to apply security patches as soon as they come out, and I like running modern versions of all my favorite software. Installing a new kernel in production under Ubuntu is trivial. Running "make world" in production isn't.
  • It must be FOSS, because that's the future I want to see. Being "mostly" open source like OpenSolaris just doesn't cut it for me.
Criteria involve trade offs. You probably have your own criteria, but these are the criteria that have lead me to be such a fan of Ubuntu.

Linux: xmonad


xmonad is a tiling window manager. If you don't know what I'm talking about, take a peek at one of the screencasts. I've been using xmonad for the last couple weeks. It's been a couple years since I tried it last, and it's really improved:
  • It's now a lot easier to install on Ubuntu.
  • It's now a lot easier to integrate with panels such as gnome-panel or xmobar.
  • It's now a lot easier to try out various layouts, and there are more layouts to choose from.
Every time I try out a tiling window manager, I am reminded of the fact that I fundamentally disagree with the premise:
  • They think that maximizing a window as much as possible whenever possible is useful. I think that's true with terminals and chat windows, but less so for many other windows. For instance, I always want GVim to be 80 columns wide.
  • They think that minimizing windows to very small sizes is more acceptable than allowing windows to overlap. I disagree.
  • They think that forcibly resizing windows won't break them. Unfortunately, forcibly resizing windows makes the Gimp look terrible, and it often cuts off the last line of text in GVim.
  • They think that I don't care about having an integrated desktop environment such as the one GNOME provides. In reality, I like the idea of using weird window managers as part of my GNOME desktop. I really like all those things that Ubuntu puts on my gnome-panel such as the network manager, update manager, mixer, etc.
Fortunately, of all the tiling window managers, xmonad is the most understanding of my needs:
  • It works with the gnome-panel.
  • It has layout managers that can take hints from applications so as not to chop off the last line of text in GVim.
  • It can create exceptions for certain applications, such as the Gimp, placing them in a "floating" layer that is more suitable for such applications.
  • It has layouts that allow windows to overlap in useful ways.
Hence, xmonad is my favorite among all the tiling window managers that I've tried. There are a couple premises that I fundamentally agree with:
  • Having the computer help you manage your windows using smart algorithms is a good idea.
  • Trying bold, new ideas in user interface design is a great idea.
That last point is important. There's a great saying in The Myths of Innovation: "Don't worry about people stealing your ideas. If they're any good, you'll have to cram them down people's throats!" Not every new idea is good, but trying out lots of new ideas is very good.

xmonad exemplifies what I'd like to see more of in the Linux world. Linux has always been a little behind in terms of user interface design. Instead of cloning Windows or OS X, I really like the idea of creating our own new, interesting, and clever approaches to user interface design. Do you remember the time when neither Windows nor OS X had virtual desktops? I like having things that those guys don't have, which is why I'm happy to see the innovation that xmonad is providing.

Linux: Minimizing Memory Usage

It's still fashionable in certain Linux circles these days to pick a leaner distro in order to minimize resource utilization. Advocates of Arch Linux and Gentoo like to point out that Ubuntu is a bit heavy on RAM usage. As a Linux old timer, I'm sympathetic to the yearnings for a leaner, meaner past. Even Linus Torvalds admits that Linux is getting a bit bloated these days.

However, does switching to something like Arch Linux or Xubuntu really help? Consider the fact that I have a dual core MacBook with two gigs of RAM. It'd be one thing if I had an ancient machine, but I actually have relatively beefy hardware. Also, consider what top says when I sort by RAM usage:
top - 03:32:53 up 10 days,  4:03,  3 users,  load average: 0.37, 0.37, 0.41
Tasks: 154 total, 2 running, 149 sleeping, 0 stopped, 3 zombie
Cpu(s): 12.9%us, 5.4%sy, 0.0%ni, 81.5%id, 0.0%wa, 0.2%hi, 0.0%si, 0.0%st
Mem: 2026720k total, 1834468k used, 192252k free, 189524k buffers
Swap: 3559984k total, 36252k used, 3523732k free, 710924k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29545 jj 20 0 591m 310m 31m S 21 15.7 55:11.01 firefox
31458 root 20 0 972m 216m 58m S 8 10.9 86:57.34 Xorg
9677 jj 20 0 762m 118m 19m S 0 6.0 0:17.02 java
32600 jj 20 0 248m 55m 18m S 4 2.8 10:54.55 skype
32602 jj 20 0 109m 44m 20m S 0 2.3 7:03.57 pidgin
31311 jj 20 0 49784 36m 3468 S 1 1.8 1:48.45 ruby
1129 jj 20 0 54408 27m 13m S 1 1.4 4:27.88 gvim
31600 jj 20 0 34152 22m 8100 S 0 1.2 0:47.06 notify-osd
31563 jj 20 0 37368 20m 13m S 0 1.0 0:30.85 gnome-panel
1119 jj 20 0 55848 19m 10m R 2 1.0 1:13.64 gnome-terminal
31644 jj 20 0 73664 16m 10m S 0 0.8 0:08.64 gnome-power-man
31627 jj 20 0 46588 16m 11m S 0 0.8 0:00.74 mixer_applet2
31632 jj 20 0 29372 16m 8472 S 0 0.8 2:03.24 indicator-apple
31588 jj 20 0 26880 15m 9.8m S 0 0.8 0:09.21 nm-applet
31602 jj 20 0 28128 14m 10m S 0 0.7 0:03.17 update-notifier
31624 jj 20 0 27528 14m 10m S 0 0.7 0:07.55 fast-user-switcher
...
Firefox is eating up 310MB of resident memory. Oops! Time to restart it! NetBeans is eating up a whopping 762MB of virtual memory, although only 118MB of it are resident. Even GVim is using 27MB of resident memory. I have no clue why X is taking up so much memory. Despite all that, I still have massive amounts of memory free for the OS to dedicate to buffers and cache.

The fact of the matter is I need Firefox with Firebug, etc. in order to do my job, and using NetBeans (with the jVi plugin) makes me more productive than using GVim or Emacs alone. Remember the days when people joked that Emacs stood for "Eight Megs And Constantly Swapping"? It's hard to imagine an integrated development environment taking up only 8MB. Even my terminal eats more than that these days!

Hence, my point is that I could switch to Arch Linux, but that's optimizing the wrong thing. NetBeans and Firefox are the ones that eat the most RAM.

Tuesday, November 10, 2009

Python: Concurrency

With all the excitement surrounding Tornado Web Server, I'd like to mention that I wrote a great article last year on Python Concurrency, with an emphasis on the various approaches to writing Web servers.

By the way, has anyone played with gevent or Eventlet? Like the proprietary version of stackless Python we used at IronPort, they offer asynchronous networking without the need to use continuation passing style.

Happy Hacking!

Friday, November 06, 2009

My Friend Committed Suicide

He was a brilliant programmer, and he had everything going for him. He was very successful.

I'm crushed because I know I could have helped if only he had given me a chance. He never did.

We in the programming world aren't always the most emotionally balanced. I know of three others who took their lives in the programming world. I've hinted at this before on my Bipolar Lisp Programmer post. To compound matters, our society has been moving away from personal interaction and responsibility for decades, leading to a culture that is toxic.

Mother Theresa said that the greatest poverty that she ever saw was to see people who felt unloved. If your friends are feeling unloved, please reach out to them. We are each far more loved than we think. In the programming world, it's so easy to get caught up in petty struggles, like Pylons vs. Django, Ruby vs. Python, free software vs. open source, Linux vs. pretty much everything else ;) What we forget is that we're all people with hopes and dreams, fears and insecurities. We're all trying to change the world, but as Mother Theresa said, "In this life we cannot do great things. We can only do small things with great love."

Look out for each other, but if you have no where to turn, contact me! Call me directly at (925) 209-6439. Don't consider the middle of the night an inconvenience. When else am I going to work with five kids around ;) I will do everything I can to help. I don't want to wake up on another day to find another brilliant mind missing from this great community of thinkers. We are a community and we are only as great as the individuals that thrive and share with one another in that community. Everyone contributes, whether in small ways or large, and every loss is felt by more people than we consider in our moments of darkness.

Friday, October 30, 2009

Vim: Editing ActionScript

It took me an embarrassingly long time to figure out how to install the ActionScript plugin for Vim. First, download it from here. Put it in ~/.vim/syntax. Now, update your ~/.vimrc file to look something like this:
set autoindent
syntax on
filetype plugin on

" Use .as for ActionScript files, not Atlas files.
au BufNewFile,BufRead *.as set filetype=actionscript

" Put these in an autocmd group, so that we can delete them easily.
augroup vimrc
au!
autocmd FileType actionscript setlocal sw=4 sts=4 et
autocmd FileType python setlocal sw=4 sts=4 et tw=72
autocmd FileType ruby setlocal sw=2 sts=2 et
augroup END

Thursday, October 29, 2009

Forking a Subversion Repository Using Git the Brutal Way

I work at a company. We use GitHub. There's a project called CoolProject. CoolProject uses Subversion. I want to fork CoolProject for internal use. There's no use submitting our patches back--they wouldn't want them. I want to maintain our own fork, but I also want to keep updated with changes that they make. Since Subversion isn't a distributed revision control system, forking like this and getting updates is hard. If you know of a better solution than the one I've proposed below, please tell me!

Git has support for Subversion. A great video on how to use it is here. The official documentation for git-svn is here. Here's the gotcha:
For the sake of simplicity and interoperating with a less-capable system (SVN), it is recommended that all git svn users clone, fetch and dcommit directly from the SVN server, and avoid all git clone/pull/merge/push operations between git repositories and branches. The recommended method of exchanging code between git branches and users is git format-patch and git am, or just 'dcommit'ing to the SVN repository.
Having each developer pull and manage their own repository and then share patches manually sounds painful. What I really want is to just check the source code into GitHub and have things just work. I asked on IRC what the best approach was, and I didn't get a response. I searched on the Web, and it seems like everyone using git-svn is working alone--i.e. using Git to work with an existing Subversion repository, but not trying to manage their own running fork with a group of people. I asked vark.com, and me and another guy came up with the following solution.

I checked out all of CoolProject's code from Subversion. I checked it into Git. I created a branch called coolproject_subversion that will be updated with updated with CoolProject's Subversion repository per the method below. My company will commit our changes to master. Once in a while, I'll merge changes from coolproject_subversion to master.

This is a brutally blunt method. We're losing all of Subversion's change history, all their commit messages, etc. However, it lets us keep up to date with their changes, and it lets us keep our own fork--as a group. It's the best thing I could think of. If I need to, I can always look at CoolProject's Subversion repository for log messages, etc.

I think it's a good idea to avoid munging with CoolProject's code as much as possible in order to avoid merge problems. The goal is to keep our code as isolated as possible, and only hack in some integration points. Okay, here's how I got started:
cd somewhere_in_my_git_repo
svn export http://coolproject.googlecode.com/svn/coolproject/trunk/ coolproject
git add *
git commit -a
git checkout -b coolproject_subversion
git checkout master
git push origin HEAD
git push origin coolproject_subversion
Here's how to grab changes from CoolProject's Subversion repository:
git checkout coolproject_subversion
rm -rf * # Double check where you are!
svn export http://coolproject.googlecode.com/svn/coolproject/trunk/ coolproject
git commit -a
git checkout master
git merge coolproject_subversion
git push origin HEAD
git push origin coolproject_subversion
Updated: Use "svn export" instead of using "svn checkout" and then deleting the .svn files.

Saturday, October 17, 2009

Ruby: An Introduction to Behavioral Driven Development with RSpec and Cucumber

On October 20, 2009 at 6:30, I'm going to be giving a talk for the East Bay Ruby Meetup Group called "An Introduction to Behavioral Driven Development with RSpec and Cucumber". This is an introduction to behavioral driven development in Rails using Cucumber, RSpec, Webrat, and factory_girl.

For the second time in my life, I finished preparing several days before the actual talk. If you're interested, here are the slides.

Happy testing! :-D

Thursday, October 15, 2009

Exploding Software-Engineering Myths

If you haven't read Exploding Software-Engineering Myths from Microsoft Research, I highly recommend it.

Tuesday, October 13, 2009

Ruby: Escaping URL Parameters

If you are trying to escape some URL parameters completely separate of Rails' routing infrastructure, URI.escape (AKA URI.encode) is not what you're looking for. In fact, I can't even figure out what problem URI.escape is trying to solve. Consider:
irb(main):003:0> require "uri"
=> true
irb(main):004:0> url = "http://foo.com?a=" + URI.escape("a&b+c")
=> "http://foo.com?a=a&b+c"
What you're looking for is CGI::escape:
irb(main):005:0> require "cgi"
=> true
irb(main):006:0> url = "http://foo.com?a=" + CGI::escape("a&b+c")
=> "http://foo.com?a=a%26b%2Bc"
The same thing goes for URI.decode vs. CGI::unescape.

However, if you're trying parse the query string from a complete URL, try this:
irb(main):008:0> require "uri"
=> false
irb(main):009:0> require "cgi"
=> false
irb(main):010:0> url = "http://foo.com?a=b+c&d=a&d=b"
=> "http://foo.com?a=b+c&d=a&d=b"
irb(main):011:0> uri = URI.parse(url)
=> #<URI::HTTP:0xb7bab258 URL:http://foo.com?a=b+c&d=a&d=b>
irb(main):012:0> uri.query
=> "a=b+c&d=a&d=b"
irb(main):013:0> params = CGI.parse(uri.query)
=> {"a"=>["b c"], "d"=>["a", "b"]}

Friday, October 09, 2009

Ruby: Testing Rake Tasks

I wrote a rake task for my Rails project, and I figured I should write some tests for it. The rake task just creates a certain file, hence it's easy to test. I looked for some entries on testing rake tasks using RSpec.

This blog post shows how to launch rake from within your test--in process. I couldn't get it to work because "Rake" wasn't loaded. This blog post shows how to refactor your rake code into a library that can be more easily tested. I decided to shell out to rake instead.

After reading about the differences between kernel.exec, kernel.system, %x{}, and backticks, I decided to just use backticks. The code ended up being fairly simple since it doesn't require much setup:
require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')

describe "rake some:task" do
context "subtask" do
it "should create some file containing some stuff" do
begin
`rake some:task:subtask SOME=args`
$?.exitstatus.should == 0
created_file = File.expand_path(File.dirname(__FILE__) + '/../../config/some_subdir/this_got_created.json')
file_contents = File.new(created_file).read()
# Verify the file contents...
ensure
File.unlink(created_file) if File.exist?(created_file)
end
end
end
end
This definitely adds a second or two to my tests, but I'm okay with that.

Saturday, October 03, 2009

Python: How to Blow Up Helicopters Using Pygame

This Thursday, I'm going to be giving a talk for PyGameSF called "How to Blow Up Helicopters Using Pygame". It's a summary of some of the libraries and tricks I used for my two PyWeek entries. I'll be covering topics such as PGU, generator-based animations, and state machine based levels.

For once in my life, I finished preparing several days before the actual talk. If you're interested, here are the slides.

By the way, I used the online service 280 Slides to build my presentation. It was pleasantly simple to use. It felt just a little sluggish compared to desktop software, but for a piece of web-based software, the UI was pretty amazing.

Saturday, September 26, 2009

Linux: The Least Bad Synaptics Configuration for a MacBook Running Ubuntu

In my last post, Linux: My Mouse Ate My Homework!, I explained how dangerous it is to have a touchpad configuration that isn't finely tuned when you're running Ubuntu on a MacBook. For some reason, I've always been happy with the touchpad configuration under OS X, but I've never managed to set it up just right under Ubuntu. I spent three hours yesterday reading the synaptics man page (which is very well written, but not always helpful) and tweaking the settings.

In this post, I'm going to document the settings I came up with. I invite you to submit comments if you've found settings that feel more natural and are less likely to cause inadvertent mayhem. I'd be especially interested if you managed to get palm detection working.

First, start by creating /etc/hal/fdi/policy/appletouch.fdi as root:
<?xml version="1.0" encoding="ISO-8859-1"?>

<!--
To learn more about this file: man 4 synaptics

Check for updates to this configuration here:
http://jjinux.blogspot.com/2009/09/linux-least-bad-synaptics-configuration.html

References:
https://help.ubuntu.com/community/MacBook4-1/Jaunty#Touchpad%20(appletouch)
https://help.ubuntu.com/community/MacBook_Santa_Rosa#Disable%20Touchpad%20While%20Typing
http://linuxwisdom.blogspot.com/2007/07/macbooksynaptics-trackpad-configuration.html
https://help.ubuntu.com/community/MacBook%20Aluminum#Trackpad
-->

<deviceinfo version="0.2">
<device>
<match key="input.x11_driver" string="synaptics">

<!--
You need this turned on in order to run syndaemon. Once you have it turned on, you can run
"synclient -l" to look at your existing settings, and "synclient VAR=VAL" to change settings at runtime.
-->
<merge key="input.x11_options.SHMConfig" type="string">true</merge>

<!-- Disable tapping. My palm always causes the context menu to pop up, and I hate that. -->
<merge key="input.x11_options.TapButton1" type="string">0</merge>
<merge key="input.x11_options.TapButton2" type="string">0</merge>
<merge key="input.x11_options.TapButton3" type="string">0</merge>

<!-- Set this to 30 and 40 if you want to have to press harder. -->
<merge key="input.x11_options.FingerLow" type="string">10</merge>
<merge key="input.x11_options.FingerHigh" type="string">30</merge>

<!-- This has to do with acceleration. It's a matter of taste. -->
<merge key="input.x11_options.AccelFactor" type="string">0.10</merge>
<merge key="input.x11_options.MaxSpeed" type="string">1.2</merge>
<merge key="input.x11_options.MinSpeed" type="string">0.5</merge>

<!--
2 fingers + click = right mouse button
3 fingers + click = middle mouse button
-->
<merge key="input.x11_options.ClickFinger2" type="string">3</merge>
<merge key="input.x11_options.ClickFinger3" type="string">2</merge>

<!-- Set the speed of scrolling. -->
<merge key="input.x11_options.VertScrollDelta" type="string">20</merge>
<merge key="input.x11_options.HorizScrollDelta" type="string">20</merge>

<!--
You might want to disable one or the other of these. Of course, you'll need a multi-touch compatible
touchpad to use TwoFingerScroll.

<merge key="input.x11_options.VertTwoFingerScroll" type="string">false</merge>
<merge key="input.x11_options.HorizTwoFingerScroll" type="string">false</merge>
<merge key="input.x11_options.HorizEdgeScroll" type="string">false</merge>
<merge key="input.x11_options.VertEdgeScroll" type="string">false</merge>
-->
<merge key="input.x11_options.HorizEdgeScroll" type="string">false</merge>
<merge key="input.x11_options.VertEdgeScroll" type="string">false</merge>

<!-- This never seems to work for me, but hopefully one day it might. -->
<merge key="input.x11_options.PalmDetect" type="string">1</merge>
</match>
</device>
</deviceinfo>
Now, use System :: Preferences :: Startup Applications, and create a new startup item "syndaemon -d -t" and then reboot. Read the file above to learn how to tweak these settings at runtime once you've rebooted.

Linux: My Mouse Ate My Homework!

One of the biggest challenges with using Ubuntu on a MacBook is getting used to all the "fun features" offered by the Synaptics touchpad, especially if you have big, clumsy hands like I do.

For instance, by tapping two fingers on my touchpad with a blank desktop, I can create a folder named "Untitled Folder". I'm not sure why I would want that, but it's definitely convenient! The problem is that a two finger tap opens up the context menu, and another tap selects the first item in the menu--which is to create a new folder named "Untitled Folder".

Another fun feature is that when I accidentally brush the touchpad with my palm, it'll find the virtual desktop where my browser is running and start scrolling the window. If I try to brush the touchpad in the opposite direction to get back to where I was, it just scrolls the browser in the opposite direction. That's because a two-finger drag on a blank desktop starts flipping through all the virtual desktops. Once you stumble upon a virtual desktop that has a fully maximized application, such as a browser, the two-finger drag just scrolls that window. You can't use the touchpad to get back to the original virtual desktop if there's a maximized window in the way.

As a Vim user, I was pleasantly surprised to find that my touchpad was helpful when using Vim. It's called the "I'm feeling lucky!" feature. Vim has two modes, insert mode and command mode. If you're in command mode and accidentally hit your palm on the touchpad, it'll paste whatever you have in X11's copy-paste buffer into Vim which treats it as random commands. It's even more fun in Vi since Vi doesn't have unlimited undo.

Last of all, my favorite feature is that "I'm feeling lucky!" works with the shell too. If you've highlighted a big block of text, and then accidentally hit your palm on the touchpad while typing in a terminal window, it'll paste all the things you've highlighted and treat them as shell commands. I think that feature is a useful reminder that you should never highlight anything with "rm" in it!

Wednesday, September 23, 2009

JavaScript: jQuery Event Handling in FriendFeed

My buddy Andreas Schobel and I both use jQuery. He was mentioning to me that his page was getting too big, and setting up all the event handlers was taking too long. He noticed that FriendFeed also uses jQuery but doesn't have this problem. In fact, he couldn't figure out how FriendFeed's click handlers even worked. We decided to investigate.

We started investigating this file. It was minimized, so we ran it through the Javascript Beautifier. This reindents the code. However, it can't deduce the original variable names for local variables. That makes understanding the code a little harder, but it's still possible.

We started looking for instances of the word "click". It looked like the file contained jQuery plugins that made use of the click function, but the main app didn't. We saw this line:
var clickHandlers = {};
And a bunch of functions like:
clickHandlers.expandcomments = function (A) {...};
However, we couldn't find any place that was actually using clickHandlers. Finally, I started looking for just the word "Handlers", and I came across this:
function bindEvents(A) {
var B = window[A + "Handlers"];
$("body")[A](function (F) {
if (A == "click" && handleFBClick(F)) {
return false
}
for (var E = F.target; E; E = E.parentNode) {
if (!E.className) {
continue
}
var C = E.className.match(/\bl_([\w]+)\b/);
if (!C) {
continue
}
var D = B[C[1]];
if (!D) {
continue
}
if (A == "click") {
$(E).blur();
return D($(E), F) ? undefined : false
} else {
D($(E), F);
return
}
}
if (!window.gIphoneMode && A == "click") {
popup.hide();
$.closePopupMenu()
}
})
}
It took a little longer to figure out what it meant.

Events are allowed to propagate all the way up to the body. The body has handlers for many types of events such as "click", but they all point to the same event handler function, which is nested anonymously inside bindEvents.

Let's suppose a JavaScript link is clicked. The event will propagate up to the body tag which will try to handle it. The class name for the link is inspected. If it is something like "l_expandcomments", and clickHandlers has a member called expandcomments, then that function is called. There is special code to a) stop further event propagation unless appropriate b) stop the user from double clicking c) do something special for iPhone users.

Hence, they can connect an HTML element with a JavaScript handler just by adding a class such as "l_expandcomments" and adding an "expandcomments" function to clickHandlers. That's convenient and clever. JavaScript's malleable (almost chaotic) nature reminds me more and more of Lisp (which I've been saying for years). Going back to my buddy's original problem, this approach seems like a viable way to reduce the event handler setup time. I wonder what initially lead them to take this approach.

Tuesday, September 22, 2009

Web: 960 Grid System

I just watched the video for 960 Grid System. What a weird idea: constrain the page to 960 pixels and cut it up into 12 columns. When laying out your page, each piece fits into one or more contiguous columns. It might make more sense if you look at their examples.

Naturally, I knew about YUI Grid, but for some reason, watching the video for 960 Grid System helped me envision how I might actually use it. I've never really been good at super complicated layouts without using tables, but this seems easy enough to use. There's even a version of the CSS that isn't limited to 960 pixels.

Anyway, I'm not saying I'm going to convert all my sites, but I do think it's weird that somehow constraining yourself to fixed numbers like 960 pixels and 12 columns leads to greater flexbiliity. Since I'm a fan of haiku, that kind of makes sense to me.

Thursday, September 17, 2009

Apple: Snow Leopard

I upgraded to Snow Leopard today. The upgrader refused to run until I deleted my Linux partition:
When trying to install Snow Leopard, some people are having a problem where the installer will not recognize the current boot drive as a valid destination for Snow Leopard. Instead, it will display the drive with a yellow triangle on it, indicating something is wrong with that drive. When the drive is selected, the installer claims the system cannot boot from the drive. From cnet.
There's more about the issue here. Once I upgraded, I found out that when you upgrade OS X, you have to install all your MacPorts from scratch. Hence, I didn't get much work done today. On the bright side, I really like one of the new backgrounds! ;)

I'm thinking of trading my MacBook on Craigslist for an IBM Thinkpad T61p.

Wednesday, September 09, 2009

Rails: Handling Duplicate Entries with Unique Indexes

If you add a unique index to a column in a database table, the database will prevent you from having two records with the same value for the given column. Rails has similar functionality at the application level. However, just like with foreign keys, the application isn't in a good position to enforce such a constraint. As soon as you have multiple application servers hitting the same database, it becomes obvious that only the database is in a position to enforce the constraint without suffering from ugly race conditions.

Ok, so how do you let the database enforce the constraint, but still handle duplicate records gracefully? Start with adding the unique index to the database in a migration:
add_index :wishlist_books, [:user_id, :book_id], :unique => true
Now, instead of creating the record directly in the controller, call a method in the model:
current_user.add_book_to_wishlist(book)
In the user model, you might have something like this:
# Add a book to the wish list.
#
# This is idempotent. It will just do the right thing if the user tries to
# insert a duplicate record.
#
# Important: if you call this method, then the next time you load the wish
# list, you must reload to clear the cache. For instance:
#
# current_user.wishlist_books(true)
def add_book_to_wishlist(book)
wishlist_books.create! do |list_item|
list_item.book = book
end
rescue ActiveRecord::StatementInvalid => e
raise e unless /Mysql::Error: Duplicate entry/.match(e)
end
Don't worry too much about the associations. I'm just trying to show the code somewhat in context. The important part is the exception that is caught.

Naturally, this is MySQL specific. I don't know the exact thing to look for with PostgreSQL. However, that's okay. I don't mind using MySQL specific features in my models where it makes sense.

Tuesday, September 01, 2009

My Take on Writing Code that Doesn't Suck

I just finished watching Writing Code That Doesn't Suck by Yehuda Katz. I think he also could have called it "Writing Tests That Don't Suck". I'm going to write a summary of some of his points (along with some of my own notes). He covers a lot of the things I've been thinking about over the last couple months as a Python coder coding in Rails using behavioral driven development (BDD).

Most Rails programmers agree that tests are good. However, we don't spend enough time considering which tests are the most valuable. In fact, Yehuda points out that a lot of tests are quite useless.

Unit tested code doesn't mean bug free code. In my own experience, I'm often so tired after writing automated tests that I don't have enough mental energy to do solid exploratory testing. Having someone other than the programmer do some serious exploratory testing is necessary.

alias_method_chain is bad. Having a chain of modules monkeypatching the same method leads to a brittle modularity nightmare that is unmaintainable and unreausable. It's also a sign of a poorly designed API. Just because Ruby lets you do things like that doesn't mean we should be content with poorly designed APIs.

Matz specifically doesn't want to add interfaces to Ruby. However, Yehuda argues that interface-oriented design is a very good thing. Yehuda reminds us that everywhere other than Ruby, coders know that good design means coding to an interface, not to an implementation.

You might have code and unit tests that test that code. You might refactor them both together, and all your tests might pass. However, if you break the API to your code, you've broken everyone else's code that is reliant on that interface. That's why regression testing is important. It doesn't matter how the implementation works--the only thing that matters is that your interface continues to behave as promised.

That's why testing the external interface is so much more important than unit testing the internal implementation. Heavily mocked and stubbed unit tests just aren't as useful as full-stack testing (with, say, Cucumber and Webrat). I personally think there's nothing worse than a controller, model, and a view that are tested individually using mocks and stubs, but which blow up when combined. That's not to say that unit testing isn't useful--it's just that what I would call integration testing or what Yehuda would call regression testing is even more useful. Test what you actually care about--i.e. the external behavior.

"some_object.should respond_to(:some_method)" is not a useful test. If you're not testing what some_method actually does, then why bother, and if you are testing what some_method actually does, then you don't need to test for its existence. I think that "some_object.should respond_to(:some_method)" is sort of like sunbathing--it makes you feel better about yourself and (arguably) makes you look good, but it's not helpful for your overall health and well being.

At the risk of beating a dead horse, when I see tests like "some_object.should respond_to(:some_method)", it really makes me wish I were coding in Haskell. Haskell's type system can enforce interfaces like this without writing boring, do nothing tests. Perhaps that's why so many Ruby coders have switched to Scala.

Yehuta does think that TDD with unit tests is helpful for designing your code, but those tests aren't good as regression tests. After your code is working, then you should write regression tests that cover the things that you actually care about, not the irrelevant implementation details.

Yehuda doesn't have a computer science background. Hence, he is relearning a lot of the lessons that MIT's Structure and Interpretation of Computer Programs has been teaching for decades, and I think that's good. It seems like much of the Rails world has forgotten a lot of the things we've learned about modularity, encapsulation, and good design over the last 30 years. Hopefully, thanks to Yehuda and Merb, that will improve.

It will be interesting to watch the synthesis of good design as taught by SICP with a strong commitment to testing that is popular in the Rails world.

Monday, August 31, 2009

Rails: Ratings


I had to add a ratings widget to my app, sort of like Amazon has. I decided to use the jQuery Star Rating Plugin on the front end. That worked out well. I decided to code the back end from scratch. That took longer than I would have expected, but the code is super tight.

Only logged in users can vote. If a user votes again, it should update his existing vote rather than letting him stuff the ballot box. Aside from keeping track of the ratings for each user, I wanted the item itself, i.e. the book, to have a rating_average field. Furthermore, I didn't want rating_average to have to calculate the average rating every time I loaded the page. It should be cached in the same way that Rails can cache the number of children a parent has.

Here's what my schema looks like:
class CreateBookRatings < ActiveRecord::Migration
def self.up
execute %{
CREATE TABLE book_ratings (
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
user_id INT NOT NULL,
book_id INT NOT NULL,
rating FLOAT NOT NULL,
created_at DATETIME,
updated_at DATETIME,

FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE,
FOREIGN KEY (book_id) REFERENCES books(id) ON DELETE CASCADE,

UNIQUE INDEX (book_id, user_id),
INDEX (user_id)
) ENGINE = INNODB
}

add_column :books, :rating_average, :float
add_column :books, :rating_count, :int, :null => false, :default => 0
end

def self.down
remove_column :books, :rating_count
remove_column :books, :rating_average

drop_table :book_ratings
end
end
I'll skip the controller, routing, and view. They're relatively straightforward once you understand how the jQuery plugin works. The hardest part was the model:
class BookRating < ActiveRecord::Base
MAX_STARS = 5
SPLIT = 2 # You can have half of a star.

belongs_to :user
belongs_to :book
validates_numericality_of :rating, :greater_than_or_equal_to => 0,
:less_than_or_equal_to => BookRating::MAX_STARS
attr_accessible :rating

# Save a book rating.
#
# This may raise ActiveRecord::RecordInvalid if the rating is invalid.
#
# This automatically calls recalculate_average_and_count!.
def self.rate_book!(user, book, rating)
if rating.nil?
BookRating.delete_all(["user_id = ? AND book_id = ?", user.id, book.id])
else
book_rating = BookRating.new(:rating => rating)
book_rating.user = user
book_rating.book = book

# Validate manually so that I can use custom SQL.
if book_rating.invalid?
raise ActiveRecord::RecordInvalid.new(book_rating)
end

# This lets users create a new rating or update their existing rating.
# Unfortunately, insert_sql can't take an array, so I have to use
# connection.quote manually. I'm using book_rating.rating so that
# ActiveRecord can take care of the casting.
connection.insert_sql(%{
INSERT INTO book_ratings (user_id, book_id, rating, created_at, updated_at)
VALUES (#{connection.quote(user.id)},
#{connection.quote(book.id)},
#{connection.quote(book_rating.rating)},
NOW(), NOW())
ON DUPLICATE KEY UPDATE rating = #{connection.quote(book_rating.rating)},
updated_at = NOW()
})
end

recalculate_average_and_count!(book)
end

# Update book.rating_average and book.rating_count.
#
# I can calculate the average without having to scan the table when the
# user creates a new book rating, but that falls apart if he updates his
# existing rating. Hence, in the name of simplicitly, I'll just let the
# database calculate the average.
#
# I'm not going to put rate_book! and recalculate_average_and_count! into a
# single transaction. Transactions break my tests when I
# use_transactional_features, and in this case, it just isn't that crucial.
def self.recalculate_average_and_count!(book)
options = {:conditions => ["book_id = ?", book.id]}
book.rating_average = BookRating.average(:rating, options)
book.rating_count = BookRating.count(:rating, options)
book.save!
end
end
The most interesting bits are the use of "ON DUPLICATE KEY UPDATE" and the fact that the rating average is updated every time the user rates a book.

I haven't yet added Ajax to the mix. The user actually has to click a button to submit the form. However, the widget still works if JavaScript is disabled. The guys who wrote the jQuery plugin did a good job making use of semantic HTML. Because it works even without JavaScript, I was able to write Cucumber and RSpec tests for everything :)

Tuesday, August 25, 2009

JavaScript: Ajax Notifications with jQuery

If you're using Ajax, you should give hints to the user when a request to the server is being made, and you should tell him if something goes wrong. Gmail does this really well. Here's how I coded it:

First, add this to your layout:
<div id="ajaxNotifications"></div>
Add some CSS to make it purty:
#ajaxNotifications {
position: fixed;
top: 0px;
right: 0px;
background: yellow;
text-color: black;
padding: 5px;
display: none;
}
Setup jQuery, jQuery UI, and your own application.js file. This syntax is for Rails, but it's similar in Python:
<script src="http://www.google.com/jsapi" type="text/javascript"></script>
<script type="text/javascript">
<% extras = RAILS_ENV == "development" ? "{ uncompressed: true }" : "{}" %>
google.load("jquery", "1.3.2", <%= extras %>);
google.load("jqueryui", "1.7.2", <%= extras %>);
</script>
<%= javascript_include_tag "application" %>
Finally, here's application.js to pull it all together:
// This is the main application object.  It follows the module pattern.
var application = function () {
// Private functions and data go here.

return {
// Public functions and data go here.
HTTP_CONFLICT: 409,

// Show the user a quick little message.
showAjaxNotification: function(text) {
$("#ajaxNotifications").text(text).show();
},

// Hide the message. If possible, use hideAjaxNotificationIfMatches
// instead.
hideAjaxNotification: function() {
$("#ajaxNotifications").hide().text("");
},

// Hide the message, but only if it matches the given text.
hideAjaxNotificationIfMatches: function(text) {
if ($("#ajaxNotifications").text() == text) {
application.hideAjaxNotification();
}
}
};
}();

$(document).ready(function() {
var loadingMessage = "Loading...";

// When an Ajax request is being made, tell the user "Loading...".
$(this).ajaxStart(function() {
application.showAjaxNotification(loadingMessage);
});

// Only if it completes normally do we hide the message.
$(this).ajaxSuccess(function() {
application.hideAjaxNotificationIfMatches(loadingMessage);
});

// Otherwise, we give the user an error message.
$(this).ajaxError(function(event, xhr, ajaxOptions, thrownError) {

// If we get an HTTP_CONFLICT, it means our data is stale. Reload the
// page. To test this functionality, use two tabs in your browser.
if (xhr.status == application.HTTP_CONFLICT) {
location.reload();
}

// Otherwise, just tell the user something went wrong.
else {
application.showAjaxNotification("Request failed: could not contact server");
}
});
});

Monday, August 24, 2009

Rails: Engine Yard Flex


I'm reading the Engine Yard Flex documentation. It's pretty interesting. Here's a snippet:
Each Application or Application Master server is setup to run haproxy on port 80 and then nginx or apache on port 81. Each App server has its own haproxy setup to balance load to all the other App servers so any one App server could become master at any point if the main master fails for any reason. We have an 'ey-monitor' daemon that runs on all the application slave servers and periodically does health checks on the current Application Master server to see if it is still running properly or not. If the App Master fails for any reason then the App slaves will try to take over as master by using a technique called STONITH(shoot the other node in the head). This means that once the master fails, the slaves will wait for a few bad health checks and then the slaves will all race to grab a distributed lock. Whichever slave gets the lock will steal the IP address of the failing master server, then it will notify our control tier which in turn will violently terminate the failed app master. Then the system will boot a new server to replace the failed node and will use the same volumes that old master had so it has the full current state of the world.

This all happens transparently to you as a user and needs no input. The system will try its best to keep itself running and at the capacity that you have stated. There can be a very short downtime when slaves take over for masters, but generally it happens in 60 seconds or less.
Boy I'm glad I don't have to set all that stuff up myself!

JavaScript: Coping with Client Server Conflicts

One of the things that worries me about writing Ajax apps is how easy it is for the client and the server to get out of sync. Open up the app in two different browsers, and it's easy to play havoc. Even if you implement Comet, all you have to do is temporarily lose Internet connectivity to go out of sync.

One way to cope with this problem is by using the "409 Conflict" HTTP status code. If the client makes a request, and the server knows that the client is working with stale state, it can return a "409 Conflict". In my own case, if the client makes an Ajax request, and the server returns a "409 Conflict", the client will reload the current page. It's not perfect, but it's a simple, effect, and easily-implemented strategy.

Here's how to return a "409 Conflict" in Rails:

render :text => "Your browser is out of sync with the server. Please reload the page.",
:status => "409 Conflict"
Here's how you can handle a "409 Conflict" if you are making an Ajax request with jQuery:
var HTTP_CONFLICT = 409;

$.ajax({
type: "POST",
url: "/some/url",
data: data,

// If we get an HTTP_CONFLICT, it means our data is stale. Reload the
// page. To test this functionality, use two tabs in your browser.
error: function(xhr, textStatus, errorThrown) {
if (xhr.status == HTTP_CONFLICT) {
location.reload();
}
}
});

Rails: Using jQuery via Google's AJAX Libraries API

Adding jQuery to your project via Google's AJAX Libraries API is a piece of cake! I just updated my application layout to include:
<script src="http://www.google.com/jsapi" type="text/javascript"></script>
<script type="text/javascript">
<% extras = RAILS_ENV == "development" ? "{ uncompressed: true }": "{}" %>
google.load("jquery", "1.3.2", <%= extras %>);
google.load("jqueryui", "1.7.2", <%= extras %>);
</script>
That code loads jquery and jqueryui. I don't have to keep them on my server. I don't have to cache them. It uses the compressed version for production and the uncompressed version in development. Best of all, the user very well might already have that file cached in their browser.

Thursday, August 20, 2009

JavaScript: DOM vs. innerHTML, Server-driven vs. Client-driven

What's the best approach to architecting JavaScript, and which frameworks best support that approach? Is it best to build the app mostly on the client like Gmail and Google Maps, or is it better to provide a normal HTML page, but with lots of Ajax mixed in like YouTube? Which approach leads to the fewest bugs when the client and server get out of sync? How does your server respond to Ajax requests? Does it serve up JavaScript code to run, JSON or XML data to digest, or pre-rendered HTML?

In the Rails world, there are all these helper functions that generate JavaScript in your HTML pages. The JavaScript might result in Ajax requests that themselves serve up more JavaScript (via .js.erb or .rjs files). There is also heavy use of innerHTML. The server is in control of the application flow.

In the jQuery world, it's standard to keep the JavaScript separate of the HTML. I think innerHTML use is still very common via the append() method.

It seems like having the client be in control is more common in the YUI world. That is, I think rich internet applications that talk to a server that just serves up data is more common in YUI than in jQuery. I think that's true of Dojo too.

I've seen some applications that only request data from the server and build the entire UI using DOM functionality. I've heard that this approach is painful, heavy, and occasionally very frustrating.

You could also build an application that only requests data from the server and builds the entire UI using mostly innerHTML. Building up a lot of HTML using JavaScript strings doesn't seem particularly pleasant either.

In GWT and Pyjamas, you write your JavaScript application in Java or Python respectively and then compile the app down to JavaScript. I'm guessing that the JavaScript builds the UI using DOM calls, but I'm not 100% sure. Has anyone out there tried Pyjamas and liked it better than, say, jQuery or YUI?

I've read the documentation for MochiKit, Dojo, YUI, and jQuery at various times over the years, and I've even read a couple books on Ajax. However, I've never read anything that gave a comprehensive break down of the pluses and minuses of each of these approaches.

At Metaweb, I do believe they started with the "build everything from scratch on the client using DOM calls" approach, and eventually the browser keeled over because there was just too much data. (Freebase produces a lot of data.) They switched to generating HTML on the server, and using Ajax to ask for even more HTML from the server when it was necessary. They liked that better. That approach is also recommended in JavaScript Best Practices on Dev.Opera.

I think most people pick an approach without even really thinking about it and never think about alternatives. Have you ever taken one approach and switched to another?

Wednesday, August 19, 2009

Web: More Lost than Ever

I've built Web apps in ASP, PHP, Perl (with mod_perl and Mason), Python, and Ruby. Just in the Python world, I've used Webware, Aquarium (which I wrote), Zope, Plone, Web.py, Pylons, and Django. I've also used ZPT, Cheetah, Myghty, Mako, Genshi, and Django templates. On the JavaScript side, I've used MochiKit, Dojo, and jQuery, and I've read the docs for YUI, Prototype, and Script.aculo.us. I've written Greasemonkey scripts, and read all the XUL documentation. I've used Apache, Nginx, IIS, and I've even written my own Web server. I've done ecommerce sites, configuration dashboards, XML-RPC services, RESTful Web services, and Ajax apps. I do everything from CSS to scalability. I'm even up to date with the latest books: I've read "JavaScript the Good Parts", "Building Scalable Web Sites", "RESTful Web Services", etc.

After a decade of working with the Web, you might think I'd know it like the back of my hand, but to tell you the truth, I feel more lost than ever. I still haven't found any Web technology that makes building Web apps anything other than a messy, complicated pain in the butt!

Thursday, August 13, 2009

Rails: Dynamic 404s, authlogic, Cucumber, and rescue_from

I'm using Rails, Cucumber, and authlogic. I want my 404 pages to be rendered dynamically; that way, I can use my application-wide layout, which is a lot friendlier for lost users. I want ActiveRecord::RecordNotFound exceptions to be handled by the dynamic 404 page. I want Cucumber tests to verify that everything is working correctly.

Getting everything working together at the same time turned out to be extremely challenging. Cucumber made it hard for me to test my ActiveRecord::RecordNotFound handling. authlogic made it impossible for me to catch ActionController::RoutingError exceptions. Defining ApplicationController#render_optional_error_file as described here conflicted with authlogic and/or Cucumber. After several hours, I finally got it all working. Here's how:

First, I added the following to ApplicationController:
# Note, I can't use rescue_from to catch ActionController::RoutingError,
# otherwise authlogic breaks. Hence, I set a default route instead.
rescue_from ActiveRecord::RecordNotFound, :with => :render_404
...
# I'm handling 404s manually so that I can use the application-wide layout.
def render_404
render :template => "errors/error_404", :status => "404 Not Found"
end
Note, render_404 must be a public method.

At the bottom of routes.rb, I have:
# If all else fails, render a 404.
map.connect '*path', :controller => :application, :action => :render_404
In app/views/errors/error_404.html.erb, I have:
Sorry, the page you were looking for does not exist.
That takes care of rendering 404s if no routes match or if there is an ActiveRecord::RecordNotFound exception.

My Cucumber tests look something like:
Scenario: there should be a custom 404 page for routing errors
Given I am on the homepage
And I am simulating a remote request
When I am on an invalid URL
Then I should get a "404 Not Found" response
And I should see "Sorry, the page you were looking for does not exist."
And I should see "Log In"

Scenario: there should be a custom 404 page for record not found errors
Given I am logged in as admin
And I am simulating a remote request
When I am on an invalid user
Then I should get a "404 Not Found" response
And I should see "Sorry, the page you were looking for does not exist.
In features/step_definitions/helper_steps.rb, I have:
def assert_response_status(http_status, message)
response.status.should == "#{http_status} #{message}"
end

Then /^I should get a "(\d+) ([^"]+)" response$/ do |http_status, message|
assert_response_status(http_status, message)
end

Given /^I am simulating a remote request$/ do
header "REMOTE-ADDR", "10.0.1.1"
end
Note, the REMOTE-ADDR header is necessary or else Rails will render a normal exception page when you're trying to test the 404 page.

In features/support/env.rb b/features/support/env.rb I have:
# Comment out the next line if you want Rails' own error handling
# (e.g. rescue_action_in_public / rescue_responses / rescue_from)
#
# I'm commenting it out in order to test that my 404 handling (using
# rescue_from) works properly.
# Cucumber::Rails.bypass_rescue
Update: Please see the comments on bypass_rescue and @allow-rescue below.

Figuring that out took the longest. Cucumber was monkey patching Rails behind my back. I spent a few hours trying to figure out why my Rails rescue_from clause wasn't running when it turned out that Cucumber was to blame.

In features/support/paths.rb, I have these paths defined:
when /an invalid URL/
"/totally_bogus_dude"
when /an invalid user/
user_path(-1)
Finally, I added the following to public/404.html:
<!--
In theory, this file should no longer be necessary since I'm handling 404s
dynamically. I'm going to leave it here in case something breaks behind my
back.
-->
Whew! That was rough!

Wednesday, August 12, 2009

Rails: authlogic Code Review

I just spent a couple days code reviewing authlogic. First of all, let me say it's good code. Usually, I have a lot to say when I code review someone's code, but this time, I was pretty happy. Here are the corrections I submitted.

Clearly, the author of authlogic knows Ruby a heck of a lot better than I do; I'm still relatively new to Ruby. There were a bunch of idioms, language features, and design decisions that caught my attention as a Python programmer. That's what this blog post is about.

The first thing I noticed is that the code pieces together huge classes by mixing in tons of modules. For instance, in authlogic/session/base.rb:
module Authlogic
module Session
# This is the base class Authlogic, where all modules are included. For
# information on functionality see the various sub modules.
class Base
include Foundation
include Callbacks
include Timeout
include Params
include Cookies
include Session
include HttpAuth
...
There are 22 included modules in all.

Here is one of the included modules, authlogic/session/foundation.rb:
module Authlogic
module Session
module Foundation
def self.included(klass)
klass.class_eval do
extend ClassMethods
include InstanceMethods
end
end

module ClassMethods
...
end

module InstanceMethods
...
end
end
end
end

The "included" method gets called when the module is included. It uses "class_eval" to explicitly mix in some methods. Notice that the "ClassMethods" module is explicitly mixed in using "extend", and the "InstanceMethods" module is explicitly mixed in using "include".

I had to refer to my Ruby book to figure out the difference. A class includes a module when it wants to mix in the module's functions as instance methods. It affects every instance of the class. An object extends a module when it wants to mix in the module's functions just into itself. In this case, the object in question is a class, so it's in essence mixing in functions as class methods. This is a vivid reminder that unlike languages like C++, classes in Ruby are objects too.

By the way, I do worry that this excessive use of mixins will lead to namespace conflicts. I ran self.methods.size in my UserSession class, which inherits from Authlogic::Session::Base. It reported 271 methods. I did the same thing in my User model, which inherits from ActiveRecord::Base. It reported 550 methods!

The next thing I noticed was methods like:
def session_ids
self.class.session_ids
end

def session_class
self.class.session_class
end
In Python, if you call a method on an instance, if it can't find the method among the instance methods, it'll also look at the class methods. Java does this too, although it's frowned upon. Ruby doesn't look among the class methods when you call an instance method. Hence, the code is explicitly delegating to the class in this code. Code like the above happens a surprisingly large number of times across the code base. I'm surprised there isn't a helper like "delegate_to_class :only => [:session_ids, :session_class]".

Another thing I noticed is code like this:
module Callbacks
METHODS = [
"before_password_set", "after_password_set",
"before_password_verification", "after_password_verification"
]
...
private
METHODS.each do |method|
class_eval <<-"end_eval", __FILE__, __LINE__
def #{method}
run_callbacks(:#{method}) { |result, object| result == false }
end
end_eval
end
end
Look closely at '<<-"end_eval", __FILE__, __LINE__...end_eval'. That's actually evaluating a "heredoc" (or at least that's what they call it in other languages) in order to define a method. Using some flavor of "eval" to add methods on the fly is fairly common in Ruby.

Another thing I noticed is code like:
def find_using_perishable_token(token, age = self.perishable_token_valid_for)
In Python, the defaults to a function are evaluated once, as the function is defined. They are not evaluated every time the function is called. If you forget that Python has "static defaults", you'll eventually get bitten by a bug. Apparently, that's not the case in Ruby:
irb(main):001:0> def f(default = [])
irb(main):002:1> default << "hi"
irb(main):003:1> end
=> nil
irb(main):004:0> f
=> ["hi"]
irb(main):005:0> f
=> ["hi"]
authlogic has impressively good docstrings. Ruby uses comments for docstrings. Python uses specially-placed strings. Hence, in Python, you can piece together docstrings at runtime using string interpolation, etc. I often make good use of this to keep my docstrings DRY and to prevent them from going stale. For instance, I might put a comment in a variable, and reuse that same comment in multiple docstrings. The code for authlogic occasionally has to duplicate the same comment.

Ruby programmers tend to use longer function names and they don't often try to limit their code to 80 columns. Here is a mildly comical case:
# A convenience function to merge options into the validates_length_of_login_field_options. So intead of:
#
# self.validates_length_of_password_field_options = validates_length_of_password_field_options.merge(:my_option => my_value)
#
# You can do this:
#
# merge_validates_length_of_password_field_options :my_option => my_value
def merge_validates_length_of_password_field_options(options = {})
self.validates_length_of_password_field_options = validates_length_of_password_field_options.merge(options)
end
I enjoyed the metaprogramming in authlogic. For instance, code like the following is fairly common across the codebase:
self.class.send(:attr_writer, login_field) if !respond_to?("#{login_field}=")
The last thing that caught me off guard was that Ruby supports:
1/0 rescue 'hi'
Apparently, that's an expression-level form of begin/rescue (aka try/finally). The above evaluates to "hi".

Anyway, as I said, authlogic is good code, and I learned a lot :)