PyCon: Scaling Python for High-load Web Sites

  • The speakers were smart, but their content was pretty standard.
  • To avoid the global interpreter lock, run two instances of CherryPy.
  • They were using Nginx for load balancing. It can "spoon feed" clients. It can talk to memcached.
  • Of course, a missing index in the database can destroy everything.
  • They automated deployment to new servers.
  • Their "dirty little secret" was that they used ZXTM Zeus server for caching, load balancing, and SSL accelaration.
  • If users repeated hit the reload button because of a slow page load, they will compound the situation until they eventually bring down the server. You may want time limits. You may need maximum client connection limits.
  • You may want to disable swap to avoid thrashing.
The funny thing was that they used a demo application called "Flikr Killer" as the basis for their talk. Apparently, they hadn't read Cal Henderson's book "Building Scalable Web Sites." The great irony is that Cal is an engineering manager for Flikr, and in his book, he covers exactly the same things they did, and more. We all had a good chuckle, and one of them promised to read the book ;)