Posts

Showing posts from 2011

Python: Shrapnel

After many years of hopeful expectation, IronPort (now part of Cisco) finally open sourced a bunch of stuff, most importantly Shrapnel, our proprietary version of stackless Python. Back when I worked at IronPort (2004-2006), I was dying for them to open source this stuff. The existing open source version of stackless Python at the time never had quite the same flavor or emphasis as Shrapnel, but these days gevent is very similar.

Anyway, congratulations to Sam Rushing, Mark Peek, etc., and thank you Cisco! By the way, there's a reference to me in the code ;)

Dart at g|egypt 2.0

Image
I gave a talk on Dart at g|egypt 2.0. All I can say is, wow!

First of all, I wasn't even on the agenda. In fact, the room I was using wasn't really even labelled on the map--it took me several minutes to find it. It was a small room. 2 minutes before my talk was supposed to start, I only had 3 people in the audience. However, 10 minutes later, every seat was taken, and people were standing at the back and along the sides. The room was packed!

Since Dart is so new, we haven't spoken about it at very many places. I told the Egyptians that I was giving them a chance to become the best Dart programmers in the world because they were seeing the talk before most of the rest of the world had seen it. I almost got a standing ovation ;)

People were very excited about Dart and asked a ton of excellent questions. Even after my talk, I had to stand around for 2.5 hours answering more questions.

The comments on Google+ were very supportive:

Hady Allam said, "your DART session…

YouTube for Your Business at g|egypt 2.0

Image
I gave a talk called YouTube for Your Business at g|egypt 2.0.

The talk went very well. There were about 200 people in the audience (and almost 900 in total at the event), which is a great turnout for a YouTube API talk. People asked me questions for about an hour afterwards. I even created a video for the talk:



I was really worried that I would lose my voice during the talk. It's very crackly right now.

Since this is the first time I've visited the middle east, I'm suffering a little from culture shock. I forgot to remove a beer joke from one of my slides. It took me a few seconds to realize why people weren't laughing. What's really funny is that I don't even drink!

I was very amazed by a few things. There were a lot of women at the event. There are far more female programmers in Egypt then there are in the US, as far as I can tell. The next thing that surprised me was that almost everyone here uses Windows--even on the server side. That's certain…

Computer Science: The Travelling Salesman Problem

I was thinking about the Travelling Salesman problem this morning. I came up with an algorithm that permits a few nice optimizations. My guess is that Knuth probably already came up with this algorithm, formally analyzed it, and then came up with 15 others that were much better. Nonetheless, I figured I'd jot it down.

Warning: This is a rough draft of an algorithm. You shouldn't bother reading this if you want real rigor. I'm just writing it down to get it off my chest.

Rather than picking the first segment of the journey and working my way to the end, I want to pick a middle segment of the journey and work my way outwards recursively.

Given a list of distances (or times, or costs, or whatever) between cities, sort the list from shortest distance to longest distance regardless of which cities are involved. Now, loop over this list and use each element of the list as a middle segment of your journey. This gives us a list of "first steps" (or rather, first midd…

Math: Factoring Numbers

I was thinking this morning about factoring numbers. I wonder if it might sometimes be helpful to use real numbers to gain an interesting perspective in order to solve certain problems involving integer numbers (i.e. number theory problems). For instance, I was thinking about factoring large numbers.

For every natural number, C, (that isn't equal to 0 or 1), there are an infinite number of pairs of positive, real numbers A and B for which A * B = C. For instance, 6 = 1.0 * 6.0 = 2.4 * 2.5 = 2.0 * 3.0 = ... I wonder if playing around with pairs of real numbers like (2.4, 2.5) can lead you to pairs of integer numbers like (2, 3).

Imagine all the pairs (A, B) for which A * B = C. Let's create a way to graph all such pairs in a funny sort of way. Let's pick a bunch of A's going from 0 to C. For each different A, we can calculate B via C / A. Let's consider the parts of A and B that are to the right of the decimal point to see if one pair, (A, B) can lead us to an…

Computer Science: NP-complete Problems are Really NP-complete

First of all, let me apologize for my sloppy terminology. I've always been a better software engineer than a computer scientist.

This is how Wikipedia describes NP-complete problems:In computational complexity theory, the complexity class NP-complete (abbreviated NP-C or NPC) is a class of decision problems. A decision problem L is NP-complete if it is in the set of NP problems so that any given solution to the decision problem can be verified in polynomial-time, and also in the set of NP-hard problems so that any NP problem can be converted into L by a transformation of the inputs in polynomial-time.
Although any given solution to such a problem can be verified quickly, there is no known efficient way to locate a solution in the first place; indeed, the most notable characteristic of NP-complete problems is that no fast solution to them is known. That is, the time required to solve the problem using any currently known algorithm increases very quickly as the size of the problem gr…

Tetris in Dart

Image
For my first Dart program, I decided to implement Tetris. I was inspired by Alexei Kourbatov at javascripter.net which contains an implementation of Tetris he wrote way back in 1999!

You can play my version here. The source code is here.

Overall, I like Dart. The Dart version of my code is about the same length as the JavaScript version of my code (the Dart version is slightly longer because I switched from using innerHTML to using the DOM). The combination of the optional static typing and Dart Editor helped me avoid many, many bugs. I'm still a Python guy at heart, but my experience with Dart was a very pleasant one.

I jotted down some lessons I learned. Warning: This is my first Dart program, and I haven't even read the tutorial yet, so I could be missing the boat on some of these points!

Don't try to play with DOM elements before the DOM is loaded. The easiest way to achieve this is to import your Dart-compiled JavaScript at the bottom of the page.

Dart has gene…

Eclipse vs. Intellij for Android, PlayN, Google App Engine, and Python Development

I can't figure out whether I should use Eclipse or IntelliJ.

Eclipse is good in that it's open source and free. Furthermore, Google has released several plugins for it such as the ADT (Android Developer Tools) plugin and the Google Plugin for Eclipse. However, Eclipse generally leaves me feeling confused, overwhelmed, and out of control. I spent two days reading a bunch of tutorials, but I still feel like I can't do what I want. I installed Aptana Studio 3 (which includes PyDev) in order to play around with my Python Google App Engine project. However, I couldn't figure out how to do two things: 1) update the PYTHONPATH for my project to include a few project-specific directories 2) use two space indents in Python code just for this project (which is the standard at Google).

On the other hand, there's IntelliJ. I've never used IntelliJ for Java development, but I've used PyCharm and RubyMine for Python and Ruby development. The downside is that it'…

YouTube: Magisto

Image
It's tough work being a developer advocate at YouTube. I often have to do things like blog, talk tech with other programmers, and invite people to lunch. It even requires a higher standard of personal hygiene. For instance, the other day, I had to buy a new Google T-shirt because I had a large toothpaste stain on the shirt I was wearing, and I had a business lunch with a potential partner.

Well after months of effort, my work has finally come to fruition. TechCrunch just announced that Magisto is now part of YouTube.com/create.



Hmm, probably a good time to go to the gym. All those business lunches are starting to show ;)

Humor: Millions of Microprocessors

Today I ran "fortune", and I got:I went to my first computer conference at the New York Hilton about 20 years ago. When somebody there predicted the market for microprocessors would eventually be in the millions, someone else said, "Where are they all going to go? It's not like you need a computer in every doorknob!"

Years later, I went back to the same hotel. I noticed the room keys had been replaced by electronic cards you slide into slots in the doors.

There was a computer in every doorknob.
-- Danny HillisAside from the fact that that's an awesome quote, I'm pretty excited about this because Dannis Hillis was one of the founders at a company I worked for (Metaweb).

Linux: Vim Freezing

I had a problem with Vim freezing, and I finally figured out why. If you log into a Linux machine with X11 forwarding turned on, but without an X server, vim hangs when you try to start it. So do various other commands.

I recently switched from an Ubuntu laptop to a MacBook Pro running OS X. In my .ssh/config, I had:Host my-server-name
ForwardX11 yesWhen I sshed in from my Linux box, things were fine. However, when I sshed in from my Mac, things were broken. There are three workarounds:Don't enabled X11 forwarding. X11 forwarding is enabled for ssh by either the -Y or -X flags or by the ForwardX11 configuration variable.If you do enable X11 forwarding, make sure you start an X11 server first and then log in with a fresh xterm or terminal window so that the DISPLAY environmental variable is properly set.Pass -X to vim so that it doesn't try to connect to the X server. Vim will not be able to alter the window title or the clipboard. It was surprising to me that vim will…

CSS: Fading Clipped Text

Image
CSS has a feature, "text-overflow: ellipsis;", that works really well when you want to clip a piece of text and replace it with "...". However, it only works for a single line of text. It doesn't work when you have a paragraph of text, and you want to clip all the text below a certain point.

This is a problem I've struggled with a couple times. I've always wanted to figure out how to have the text fade out at the bottom of the box. Before now, I've aways just clipped the text at the bottom of the box; that looks ugly since most of the time, the text is vertically clipped in the middle of a sentence.

During a recent HTLM5 Hackathon at Google, I learned about gradients and masks. After about 45 minutes worth of futzing around, I finally achieved the effect I was looking for. Here's the code. It basically puts a mask over the text with a gradient that makes the text fade out.<!DOCTYPE html>

<html>
<head>
<meta ht…

Python: Newbie Nugget: Lambda Expressions

I gave a talk for the SF Python Meetup on lambda expressions:



(If you can't see the slides above, you can see them here.)

Linux: VNC Stopped Working

I have been using VNC to connect from my Ubuntu box to an OS X box. On the Ubuntu side, I was using vinagre, which is the standard "Remote Desktop Viewer" application. On the Mac side, I was using standard desktop sharing.

Things were working for months, but they stopped working for me the other day. I could connect just fine, but I could only see a black screen. Apparently, this is a known issue.

Switching to Remmina (sudo apt-get install remmina) fixed the problem.

Go: A Second Impression

My initial impressions of Go have always been somewhat negative because I was comparing it to things like Scala, Haskell, and Python. However, having read a good part of the tutorial, I've changed my opinion. I really like Go!

Go doesn't try to be an extension of C to make it more like Smalltalk--that's Objective C. Go doesn't try to be the end-all-be-all of object-oriented languages derived from C--that's Scala.

Since it neither maintains backwards compatibility with C nor adds a ton of features, I originally had a hard time getting excited about Go. Now I see that Go is a modern language that tries to follow what I think of as C's philosophy. It's simple, elegant, small, and native.

There's an old saying:Do not seek to follow in the footsteps of the wise men of old. Seek what they sought.

- Matsuo Munefusa (”Basho”)I think that perfectly describes Go.

Humor: Proving Programs

I've always been weary of programming proofs.

For instance, I can mathematically prove that 4195835 / 3145727 > 1.3338. However, I know of certain Pentium processors that would disagree with me. If I try to prove that the following bit of C code prints out "Hello World":if (4195835.0 / 3145727.0 > 1.3338)
printf("Hello World\n");
else
system("rm -rf /");I might be a bit surprised when it deletes my hard drive instead ;)

1 bunny + 1 bunny = 2 bunnies, right? Well it depends on their sexes. It's possible that in a given time period, 1 bunny + 1 bunny might equal 5 bunnies. As I joked in a previous blog post, "All models are wrong. Some models are useful."

I really think this same thing applies to proving programs. Donald Knuth famously said, "Beware of bugs in the above code; I have only proved it correct, not tried it."

Books: Masterminds of Programming

I just finished reading Masterminds of Programming: Conversations with the Creators of Major Programming LanguagesMasterminds of Programming features exclusive interviews with the creators of several historic and highly influential programming languages. In this unique collection, you'll learn about the processes that led to specific design decisions, including the goals they had in mind, the trade-offs they had to make, and how their experiences have left an impact on programming today.In short, I really enjoyed it. Here's an extremely abbreviated and opinionated summary:

Adin D. Falkoff (APL) made programming as mathematical as possible.

Thomas E. Kurtz (BASIC) was generally a nice guy who wanted to bring programming to the masses.

Charles H. Moore (FORTH) frustrated the heck out of me. He stated that operating systems are the software industry's biggest con job. I disagree. Operating systems protect me to some degree from bad and malicious code. They also let me run m…

Books: Python 3 Web Development Beginner’s Guide

Image
Packt Publishing asked me to review Python 3 Web Development Beginner's Guide. I'll have to admit, it's a bit of an odd duck. A better (albeit overly verbose) title might have been "An Introduction to Rich Internet Application Development Using jQuery UI, a Very Modern Version of Python, a Relatively Old Python Web Application Framework Named CherryPy, and an Ancient Version of HTML Written by a Guy Who Uses Windows".

The first tipoff that this book was a bit strange was that the author uses Windows and some combination of Firefox and IE. It seems like most web developers use OS X (or occasionally Linux), and they prefer Chrome over IE.

The next tipoff was the use of jQuery UI. jQuery UI is a very modern technology which is often used to build rich internet applications. RIAs really aren't the sort of thing that I would expect to see in a book for beginners. What happened to the old days when beginning web applications focused on the server dynamically ge…

Google I/O 2011

I went to Google I/O May 10-12, 2011. Note that the first day of Google I/O was my second day as a Google employee. These are my personal notes. If you're interested in any of these talks, you can find them on YouTube.KeynoteOne of the slides during the keynote showed a picture of an android eating an apple.

There were 5000 people in the room for the keynote.

There were 110 cities watching at viewer parties.

Android has come a long way: 100 million activations, 450,000 developers, 215 carriers, 310 devices, 112 countries, 400,000 activations per day, 200,000 apps in the Android market, 4.5 billion apps installed.

Android 3.1 is Honeycomb.

Android can now act as a USB host. Hence, you can now get a keyboard and a mouse for your tablet.

You can use an X-Box controller with it.

Android 3.1 will be used for Google TV.

Android market will be available on Google TV.

Ice Cream Sandwich will be the new Android release.

The hope is for one Android OS everywhere.

It will be open source.

They have…

Software Engineering: Code Reviews vs. Peer Programming

I've been thinking lately about the benefits of code reviews vs. peer programming. In general, I think most companies that really care about code quality use either code reviews or peer programming. Various companies are famous for using either one approach or the other, which leads me to wonder which approach is better under which circumstances?

Pivotal Labs is famous for doing full-time peer programming. It's well known that they do a good job writing software. However, I wonder if full-time peer programming might be too expensive for mundane code. I also wonder if it makes sense to work as a pair when someone needs to spend a few days reading, learning, and researching.

In contrast, Google is famous for code reviewing all commits before checkin. Certainly this frees up engineers to get more work done since they can spend a high percentage of their time working in parallel on separate tasks. However, I wonder if a code reviewer really has the ability to make the same le…

Amazon Web Services Summit 2011: Navigate the Cloud

On June 21, 2011 I went to an Amazon Web Services conference in San Francisco. These are my notes.

In general, the conference was informative, but a bit subdued. It was about a quarter of the size of Google IO, and they weren't giving away anything huge for free. Furthermore, it was only one day, and the size of the crowd was much smaller. However, that makes sense since they have multiple of these conferences per year. Nonetheless, it was interesting, and I learned some stuff.KeynoteThe part of the keynote was given by Dr. Werner Vogels, the CTO.

AWS doesn't lock you into an OS or language.

There are 339 billion objects in S3, and S3 handles 200,000+ storage transactions per second.

On a daily basis, they add as much capacity as they had in total in
2000.

Spot pricing allows you to spin up an instance when the price is right.

AWS CloudFormation deploys a stack of software.

AWS Elastic Beanstalk is easy to begin and impossible to outgrow.

CloudFormation and Elastic Beanstalk don&#…

Introducing YouTube.com/create

Image
I just wrote my first blog post for the YouTube API blog: Introducing YouTube.com/create:YouTube.com/create is a platform for third-party applications that enable users to create videos. The idea is simple. The third-party application runs in an HTML iframe on YouTube. The user creates a video with the application, and then the application uploads the video to YouTube for the user to watch and share.

To read more, check out the story on the YouTube API blog.

Books: Python 3 Web Development Beginner’s Guide

Image
I've been asked by Packt to review a new book called Python 3 Web Development Beginner’s Guide. Here's the overview:Build your own Python web applications from scratchFollow the examples to create a number of different Python-based web applications, including a task list, book database, and wiki applicationHave the freedom to make your site your own without having to learn another frameworkI looked through the table of contents, and I scanned the first 30 page. It looks like a fun book! In fact, it's exactly the sort of book I was hoping someone would write.

Python: Increasing the Timeouts for urlfetch in Google App Engine

Google App Engine provides a function, google.appengine.api.urlfetch.fetch, for fetching URLs. I do believe all the other HTTP client libraries are monkey patched to make use of that function, which is written to take advantage of various Google infrastructure. The fetch function has a default timeout of 5 seconds. You can set a higher timeout by passing a deadline parameter, but the maximum is 10 seconds. Unfortunately, passing a deadline keyword parameter is often difficult if it's a third-party library that is making the call to fetch, for instance if you're using the GData client library.

I looked for a way to set the deadline parameter in a more global way, but I couldn't fine one by mere inspection of the code. I came up with the following HACK in order to work around this problem:# HACK: Monkeypatch google.appengine.api.urlfetch.fetch to increase the
# deadline. This is used by the various client libraries.
def _fetch(*args, **kargs):
from google.appengine.api.u…

Google I/O 2011

I went to Google I/O a few weeks ago. Although I took notes, rather than blogging all my notes, I thought I'd just link to my favorite two talks:

Programming Well with Others: Social Skills for GeeksAre languages, compilers, debuggers, and algorithms all you need to be a successful software engineer? In a perfect world, those who produce the best code should be the most successful. Unfortunately, we live in a world of imperfect people, and collaborating with others is at least as important as having great technical skills.HTML5 Showcase for Web Developers: The Wow and the HowWe'll share the strengths and extents of HTML5, showing magnificent demos of bleeding-edge features in Google Chrome. Digging into high-fidelity graphics, performance, and system integration, we'll break each demo down on the big screen to show how it was constructed. Then we'll show you how to use Chrome to its full potential in your own projects.

Python: Getting a Fair Flip out of an Unfair Coin

If you have an unfair coin (i.e. one that favors heads or tails), how do generate a fair flip (i.e. one that doesn't favor heads or tails)? My buddy Hy Carrinski and I came up with the following algorithm:"""Get a fair flip out of an unfair coin."""

from collections import defaultdict
from random import random

FLIP_RATIO = 0.7


def flip_unfair_coin():
"""Flip an unfair coin. Return a boolean."""
return random() > FLIP_RATIO


def create_fair_flip():
"""Generate a fair flip. Return a boolean."""
while True:
flip = flip_unfair_coin()
if flip != flip_unfair_coin():
return flip


# Demonstrate that it works.

if __name__ == '__main__':
results = defaultdict(int)
for i in xrange(1000000):
results[create_fair_flip()] += 1
percentage = (results[False] / float(results[False] + results[True])) * 100
print "Percentage heads:", percentag…

Personal: I Work at YouTube!

I got a job at Google working as a developer advocate for YouTube APIs. It's nice because it's a bit more of a social position than I usually have, but it still involves some work on open source code. Unfortunately, my desk is in Mountain View, so my commute is three hours a day; I probably won't be able to keep that up indefinitely. Anyway, I'm still very excited :)

I started three weeks ago, and I must admit that Google is about twice as awesome as I was even hoping for :)

Personal: Name Dropping

Ken Thompson (initial author of Unix) was awarded the Japan Prize today at Google. I just got to meet and talk to him, Vint Cerf (one of the "fathers of the Internet"), and Rob Pike (legendary Unix hacker, famous author, and co-author of Go). Since I shook each of their hands, I've decided to never wash my hand again ;)

Call Me Crazy: Calling Conventions

It's been said that the most significant contribution to computer science was the invention of the subroutine. Every major programming language has its own variation on how subroutines work. For instance, there are procedures, functions, methods, coroutines, etc. When implementing a compiled language or a virtual machine, you must decide how to implement subroutines at the assembly language level. For instance, you must decide how to pass arguments, what goes on the stack, what goes on the heap, what goes in registers, what goes in statically allocated memory, etc. These conventions are called calling conventions. You might not think of calling conventions very often, but they're an interesting topic.

First of all, what does a subroutine provide over a simple goto? A subroutine provides a way to pass arguments, execute a chunk of code, and return to where you were before you called the subroutine. You can implement each of these yourself in assembly, but standardizing c…

Perl: Text-based Address Books

Over the years, I've stored my addresses in a variety of formats. I used to have a Palm Pilot. These days, I have an Android phone, and Google keeps track of my addresses. However, I've always had a second copy of my addresses in a text file. The format looks something like this:A & R Stock to Performance:
Address: 2849 Willow Pass Rd. #A, Concord, CA 94519
Work Phone: (925) 689-1846There are many advantages to using a text-based format. For instance, since I'm an expert Vim users, I can search and edit the file extremely quickly. Best of all, the format works on every operating system and never goes obsolete. The only downside is that I have to input the address twice: once for Google and once for my own notes.

Long ago, I wrote a Perl script to convert my notes file into a Palm Pilot database. Even though I don't have a Palm Pilot anymore, I keep the script around. I can easily alter it to output the addresses in different formats.

If you like the form…

Of Neuroscience and Jazz

Image
I am neither a neuroscientist nor a very accomplished musician, but I'd like to talk about the intersection of neuroscience and music. I have a theory that there is a neuroscience basis for why it takes a mature musical palate to enjoy jazz.

First, let me say a little something about neuroscience (based on the limited understanding I've gained by watching a bunch of talks). One of the things your brain is particularly good at is recognizing patterns and predicting patterns. At the lowest level, if two nerves are close to each other, and they both fire, it's counted as a pattern--i.e. those two things are connected. Similarly, if a nerve fires and then a short while later it fires again, that's a pattern as well. Hence, if both of my fingers feel something, there's a pattern, or if I feel a tapping on a single finger, that's a pattern as well.

However, the brain is not limited to low level patterns. Rather, it can respond to a hierarchy of patterns. Paraphr…