Download Celery Documentation

Celery Documentation, Release 3.0.24
Rate Limits
The system responsible for enforcing rate limits introduces some overhead, so if you’re not using rate limits it may be
a good idea to disable them completely. This will disable one thread, and it won’t spend as many CPU cycles when
the queue is inactive.
Set the CELERY_DISABLE_RATE_LIMITS setting to disable the rate limit subsystem:
CELERY_DISABLE_RATE_LIMITS = True
2.3.12 Concurrency
Release 3.0
Date October 28, 2013
Concurrency with Eventlet
Introduction
The Eventlet homepage describes it as; A concurrent networking library for Python that allows you to change how you
run your code, not how you write it.
• It uses epoll(4) or libevent for highly scalable non-blocking I/O.
• Coroutines ensure that the developer uses a blocking style of programming that is similar to threading, but
provide the benefits of non-blocking I/O.
• The event dispatch is implicit, which means you can easily use Eventlet from the Python interpreter, or as a
small part of a larger application.
Celery supports Eventlet as an alternative execution pool implementation. It is in some cases superior to multiprocessing, but you need to ensure your tasks do not perform blocking calls, as this will halt all other operations in the worker
until the blocking call returns.
The multiprocessing pool can take use of multiple processes, but how many is often limited to a few processes per
CPU. With Eventlet you can efficiently spawn hundreds, or thousands of green threads. In an informal test with a
feed hub system the Eventlet pool could fetch and process hundreds of feeds every second, while the multiprocessing
pool spent 14 seconds processing 100 feeds. Note that is one of the applications evented I/O is especially good at
(asynchronous HTTP requests). You may want a mix of both Eventlet and multiprocessing workers, and route tasks
according to compatibility or what works best.
Enabling Eventlet
You can enable the Eventlet pool by using the -P option to celery worker:
$ celery worker -P eventlet -c 1000
Examples
See the Eventlet examples directory in the Celery distribution for some examples taking use of Eventlet support.
120
Chapter 2. Contents