Python: Debugging Memory Leaks

I wrote a simple tool that could take Web logs and replay them against a server in "real time". I was performance testing my Web app over the course of a day by hitting it with many days worth of Web logs at the same time.

By monitoring top, I found out that it was leaking memory. I was excited to try out Guppy, but it didn't help. Neither did playing around with the gc module. I had too many objects coming and going to make sense of it all.

Hence, I fell back to a simple process of elimination. Divide-and-conquer! I would make a change to the code, then I would exercise the code in a loop and monitor the output from top for ever-increasing memory usage.

Several hours later, I was able to nail it down to this simple repro:

# This program leaks memory rather quickly.  Removing the charset
# parameter fixes it.

import MySQLdb
import sys


while True:
    connection = MySQLdb.connect(user='user', passwd='password',
                                 host='localhost', db='development',
                                 charset='utf8')
    try:
        cursor = connection.cursor()
        cursor.execute('select * from mytable where false')
        sys.stdout.write('.')
        sys.stdout.flush()
    finally:
        connection.close()

It makes sense that if the memory leak is at the C level, I might not be able to find it with Python-level tools. I'll go hunting tomorrow to see if the MySQLdb team has already fixed it, and if not, I'll submit a bug.