Tuesday, November 10, 2009

Python: Concurrency

With all the excitement surrounding Tornado Web Server, I'd like to mention that I wrote a great article last year on Python Concurrency, with an emphasis on the various approaches to writing Web servers.

By the way, has anyone played with gevent or Eventlet? Like the proprietary version of stackless Python we used at IronPort, they offer asynchronous networking without the need to use continuation passing style.

Happy Hacking!

4 comments:

Marcin said...

Hi!

I have an eventlet app running in production and I'm in progress of writing next. It's definitely worth giving a shot! You may be interested in eventlet's twisted hub, although I didn't try it myself.

Shannon -jj Behrens said...

Cool! Thanks, Marcin!

The Ubiquitous Mr. Lovegroove said...

In Your article, You forget to mention that Python threads are still constrained by GIL.
Over Twisted Python world, the general idea is to have as much processes as You have CPUs and then somehow load balance across.

Shannon -jj Behrens said...

Perhaps you missed the second page of the article: http://www.ddj.com/linux-open-source/206103078;jsessionid=QNNGXN5JC2NVBQE1GHOSKH4ATMY32JVN?pgno=2.

The GIL is a subtle thing. As long as the threads don't require use of the GIL, you're fine. For instance, they can each be blocked on IO or doing something a C library that releases the GIL.

I agree with using processes over threads in a lot of situations. However, I like Erlang's approach even more, and I've always been a fan of the tricks used by gEvent.

Thanks for reading!