Tuesday, September 30, 2008

Web: REST Verbs

I find it curious that REST enthusiasts insist on viewing the world through the five verbs GET, HEAD, PUT, POST, and DELETE. It reminds me of a story:

Back in the early '80s, I worked for DARPA. During the height of the Cold War, we were really worried about being attacked by Russia. My team was charged with designing a RESTful interface to a nuclear launch site; as far as technology goes, we were way ahead of our time.

Anyway, I wanted the interface to be "PUT /bomb". However, my co-worker insisted that it should be "DELETE /russia". One of my other buddies suggested that we compromise on something more mainstream like "POST /russia/bomb".

Finally, my boss put an end to the whole fiasco. He argued that any strike against the USSR would necessarily be in retaliation to an attack from them. Hence, he suggested that it be "GET /even", so that's what we went with.

You have to understand, back then, GETs with side effects weren't yet considered harmful.

IPv6 T-shirt

Here's a shout out to all my homies in the IPv6 world! If you can't read it, it says "There is no place like (except maybe ::1)". Thanks go to Tarek Ziade (ziade.tarek at gmail.com) for the custom T-shirt design.

Books: Expert Python Programming

I just received my copy of Expert Python Programming. I was the technical editor, and I also wrote the foreword. This is the first time I've ever been mentioned on the front cover of a book, so I'm very excited!

I really enjoyed editing this book. It's the first expert-level book on Python I've read. For a long time, I considered writing one. Tarek beat me to the punch, and I think he did a fantastic job!

Thursday, September 25, 2008

A Python Programmmer's Perspective on C#

Being a language fanatic, I was really excited when I met a really smart guy named Corey Kosak who gave me a tour of C#'s newest features. I had heard a lot of good things about C# lately, including that it had been strongly influenced by Haskell, which makes sense since Microsoft actually funds research on Haskell. Anyway, a lot of C#'s newest features are a lot more like Python than Java. Let me show you some examples.
Here is a sample C# iterator:
foreach(var x in CountForeverFrom(123).Take(5)) {
In Python, I'd write:
for i in itertools.islice(itertools.count(123), 5):
print i

C# also iterators that are similar to Python's generators. Here is the C#:
public static IEnumerable<int> CountForeverFrom(int start) {
while(true) {
yield return start;
In Python, I'd write:
def count_forever_from(start):
while True:
yield start
start += 1

C#'s LINQ syntax is similar to Python's generator expressions. Here's the C#:
var names=new[] { "bill", "bob", "tim", "tom", "corey",
"carl", "jj", "sophie" };
foreach(var x in (from name in names where name.Length>5 select name)) {
In Python, I'd write:
names = ["bill", "bob", "tim", "tom", "corey", "carl", "jj", "sophie"]
for x in (name for name in names if len(name) > 5):
print x

Here's a pretty amazing example that ties a lot of things together. It shows LINQ, a "group by" clause, an anonymous but strongly-typed class ("new {...}"), and even some type inferencing ("var item" and "item.FirstChar")
var crap=from n in names
group n by n[0]
into g
select new { FirstChar=g.Key,
Data=(from x in g select x).ToArray() };

foreach(var item in crap) {
"First group is {0} which has length {1}. The contents are:",
item.FirstChar, item.Data.Length);
foreach(var x in item.Data) {
Corey said that C#'s type inferencing is still pretty basic. It can figure out the type of a local variable, but it's definitely not as sophisticated as ML's type system. Also note that the anonymous class is more impressive that an inner class in Java because it didn't require you to use a name or an interface.

"Loosely translated", in Python I'd write:
crap = itertools.groupby(names, lambda n: n[0])
for first_char, subiter in crap:
group = list(subiter)
print "Group is %s which has length %s. The contents are:\n%s" % (
first_char, len(group), "\n".join(group))

C#'s Select method can be used like map in Python. Notice the use of an anonymous function!
var newInts=ints.Select(x => x*x);
In Python, I'd write:
new_ints = map(lambda x: x * x, ints)
The C# version runs lazily (i.e. "on the fly"), which means it only computes as much as requested. Python's map function isn't lazy. However, itertools.imap is.
The above example can also be written in LINQ style:
var newInts2=(from temp in ints select temp*temp);
In Python I'd write:
new_ints2 = (temp * temp for temp in ints)
Both the C# and the Python are lazy in this case.
If you don't want newInts to be lazy, you can do:
var intArray=newInts.ToArray();
var intList=new List<int>(newInts);
In Python, I'd write:

Since C# has anonymous functions, it should come as no surprise that it also has nested scopes and first-class functions (i.e. you can return a function). Although you can't nest named functions, it's easy enough to fake with anonymous functions:
private static Action<int> NestedFunctions() {
int x=5;

Action<int> addToX=newValue => {


return addToX;
In Python, I'd write:
def nested_functions():

def add_to_x(new_value):
add_to_x.x += new_value

add_to_x.x = 5
print add_to_x.x
return add_to_x

C# also has closures:
private static void BetterExampleOfClosures() {
var a=MakeAction(5);

private static Action MakeAction(int x) {
return () => Console.WriteLine(x++);
Python has closures too. (There's a small caveat here. You can modify a variable that's in an outer scope, but there's no syntax for rebinding that variable. Python 3000 fixes this with the introduction of a nonlocal keyword. In the meantime, it's trivial to work around this problem.):
def better_example_of_closures():
a = make_action(5)

def make_action(x):

def action():
print action.x
action.x += 1

action.x = x
return action

C#'s generics are a bit more powerful than Java's generics since they don't suffer from erasure. I can't say I'm an expert on the subject. Nonetheless, I'm pretty sure you can't easily translate this example into Java. It creates a new instance of the same class as the instance that was passed as a parameter:
public abstract class Animal {
public abstract void Eat();

public class Cow : Animal {
public override void Eat() {

public class Horse : Animal {
public override void Eat() {

public static T Func<T>(T a, List<T> list) where T : Animal, new() {
return new T();
Corey told me that while C#'s generics are stronger than Java's generics, they still weren't as strong as C++'s generics since C++ generics act in an almost macro-like way.

Python has duck typing, so it doesn't have or need generics. Here's what I would write in Python:
class Animal():
def eat(self):
raise NotImplementedError

class Cow():
def eat(self):

class Horse():
def eat(self):

def func(a, list_of_a):
return a.__class__()

Unfortunately, those are all the examples I have, but let me mention a few other things he showed me.

C# has a method called Aggregate that is the same as what other languages called inject or reduce.

C# has Lisp-like macros! You can pass an AST (abstract syntax tree) around, play with it, and then compile it at runtime.

C# has an interesting feature called "extension methods". They're somewhat like a mixin or reopening a class in Ruby. Using an extension method, you can set things up so that you can write "5.Minutes()". Unlike a mixin or reopening a class, they're pure syntax and do not actually affect the class. Hence, the above translates to something like "SomeClass.Minutes(5)". Although "5" looks like the object being acted upon, it's really just a parameter to a static method.

Another thing that impressed me was just how hard Visual Studio works to keep your whitespace neat. It doesn't just indent your code. It also adds whitespace within your expressions.

Ok, that's it. As usual, I hope you've enjoyed a look at another language. I'd like to thank Corey Kosak for sending me the C# code. If I've gotten anything wrong, please do not be offended, just post a correction in the comments.

Tuesday, September 23, 2008

Python: Debugging Memory Leaks

I wrote a simple tool that could take Web logs and replay them against a server in "real time". I was performance testing my Web app over the course of a day by hitting it with many days worth of Web logs at the same time.

By monitoring top, I found out that it was leaking memory. I was excited to try out Guppy, but it didn't help. Neither did playing around with the gc module. I had too many objects coming and going to make sense of it all.

Hence, I fell back to a simple process of elimination. Divide-and-conquer! I would make a change to the code, then I would exercise the code in a loop and monitor the output from top for ever-increasing memory usage.

Several hours later, I was able to nail it down to this simple repro:
# This program leaks memory rather quickly.  Removing the charset
# parameter fixes it.

import MySQLdb
import sys

while True:
connection = MySQLdb.connect(user='user', passwd='password',
host='localhost', db='development',
cursor = connection.cursor()
cursor.execute('select * from mytable where false')
It makes sense that if the memory leak is at the C level, I might not be able to find it with Python-level tools. I'll go hunting tomorrow to see if the MySQLdb team has already fixed it, and if not, I'll submit a bug.

Friday, September 12, 2008

Software Engineering: Reuse Has Finally Arrived

Have you noticed that code reuse works these days? For a long time, software engineers struggled with the difficulty of reusing existing software, but it's now common place

Let me give you some examples. I use Linux, Nginx, MySQL, and Python, not to mention a Web browser. These days, very few people need to write a custom kernel, Web server, database, or programming language to solve their particular problem. Sure it happens, but it's far more common to reuse something existing.

I even make use of an existing Web framework, Pylons, and an existing templating engine, Mako. Those things are often written from scratch, but I didn't need to. They were fine.

Even within my own code, I find plenty of places for reuse. Each of my clients has a pretty different setup. Their input formats and output formats are often pretty different, but by using a UNIXy "small tools that can be pieced together" approach, I usually write only a small amount of code when I get a new customer.

What has changed? Why is it suddenly so easy to reuse code? Has object-oriented programming finally paid off? Maybe. However, I think the more likely culprit is open source. Small companies are now viable because they have access to a huge corpus of freely available source code. They don't have to pay for it. They can look at the source if the documentation is inadequate. They can contribute bug fixes if they encounter bugs. They can even hack it in deep ways to accomplish special tasks. This is particularly common in the BSD world.

Last of all, testing and a strong dedication to docstrings help me with reusing my own code. Per agile thinking, I don't try to get it right the first time. If I need to add a feature to make use of code in an unexpected way, I can. The docstrings help me understand what's already there, and the tests help make sure I don't break it.

Thursday, September 11, 2008

Free Software: Stallman and Births

Since I have four children, I found the following quote from Stallman to be very disturbing:
Hundreds of thousands of babies are born every day. While the whole phenomenon is menacing, one of them by itself is not newsworthy. Nor is it a difficult achievement—even some fish can do it.
When a fellow Emacs developer said that he had just become a father, Stallman replied, "I am sorry to hear it."

Perhaps he was just trolling. Well, Stallman's right. Even fish can reproduce. However, even a dog knows not to piss on his friend's leg.

Python: Bambi Meets Godzilla

I just re-read a blog post that I read a couple years ago called Bambi Meets Godzilla, and I enjoyed it just as much the second time around. It's a brief history of Smalltalk, Java, Perl, Python, and Ruby, and it talks about why hype is vitally important. It also spends a fair amount of time critiquing Python's culture. If you haven't read it yet, stop reading my post, and go read it instead ;)

It reminds me of The UNIX-HATERS Handbook, which I also love. The funny thing is that to some degree, he's right about Python's culture. I've seen it with my own eyes.

Don't believe me? If I were to admit that I preferred Ruby on Rails over Django, how long do you think it would take for someone to flame me in a comment calling me either an idiot, a troll, a loser, or a heretic, or to say something like "You can recognize good design by the inanity of its detractors"?

Tuesday, September 09, 2008

Web: SilverStripe

A couple years ago, I built my church's website using Plone. I had to read most of "The Definitive Guide to Plone", but I did it and it worked.

Recently, I realized it was time to overhaul the website. My buddy is a Plone expert, and he told me I would have an easier time rebuilding the website than trying to migrate it since my version of Plone is so old. After two years, I had forgotten much of what I knew about Plone, and I knew that my book was out of date.

I went looking for something that didn't have quite the same learning curve. Plone is fantastic if you're a Plone expert, but I'm not. I just needed "an overly simplistic content management system." I tried out Drupal and Joomla, but for long and complicated reasons, some of which involved my ISP, I decided against them; I'm sure they're quite nice.

My buddy Leon Atkinson told me that he had seen a cool demo for SilverStripe. SilverStripe is PHP, but I decided to watch the video anyway. I was amazed. It's worth the five minutes it takes to watch the screencast.

I decided to actually try it out. Within three hours, I had installed it, read one page of the tutorial, and actually built out a decent portion of the website.

What I like about SilverStripe is that it's super simple. It uses TinyMCE, so you can get a lot done with just a WYSIWYG editor. However, it also encourages you to dip into flat files in the filesystem to edit templates. It's like the best of both worlds for me. I'm almost done with the website, and I still haven't actually had to code any PHP yet. TinyMCE is occasionally a bit fickle and I've seen weird caching problems, but overall, I'm really happy.

Permit me to wax philosophic. For a hundred different reasons, I prefer Python over PHP. However, there's no denying that there's a ton of really good projects written in PHP. Consider php forum, MediaWiki, WordPress, Flickr, etc.

I have a pet theory about why this is so. I care an awful lot about how I build something, but I don't care much at all about what I build. Hence, I can use Python to build whatever, and I'm happy. Seriously, I know a ton of stuff, and I write beautiful code, but I never have any interesting ideas about what to code ;)

Most people aren't like me. For them, a programming language is just a tool to build something they want. They don't care how it gets coded as long as it does get coded. Product people often build beautiful things using not-so-beautiful code. I'm not saying that PHP can't be beautiful. I'm just saying that sometimes it doesn't matter.

Perhaps I'm just feeling a bit bipolar.

Wednesday, September 03, 2008

Python for Unix and Linux System Administration

The good news is that I was a lead technical editor of Python for Unix and Linux System Administration which just came out.

The bad news is that as my wife called me to tell me that my copy of the book had arrived, I noticed that someone had clipped my car in the parking lot and tore off part of the bumper. It looks like I'll have to replace the whole bumper.

C'est la vie.

Anyway, about the book, it's exactly what the title says it is. If you have a computer science background, this book is not for you. However, if you're a sysadmin trying to learn Python, it's perfect. In fact, when I think of all the sysadmins I've met who do a bit of scripting, this book matches them perfectly.