Skip to main content

PyCon: Distributed Tasks with Celery

Distributed Tasks with Celery

Celery is an asynchronous job queue based on distributed message passing.

ShootQ looks like a useful site for professional photographers.

Celery takes care of scheduling. It also handles errors with retries.

It works with Web hooks too, although normally you can stick to just plain Python.

It can control nodes via remote control by broadcasting messages.

It can run on top of Reddis, etc. as well.

Use CELERY_ALWAYS_EAGER when unit testing.

RabbitMQ does NOT like running out of memory! Really bad things happen.

RabbitMQ doesn't really have replication.

Side note: Convore is really popular. I had never even heard of it, and it seemed like about half the people present were using it. It's a web-based chat system. It's a mix of Twitter and IRC. It's very threaded.

Celery doesn't yet have priority queues.