Celery Does Not Release Memory

It looks like celery does not release memory after task finished. Every time a task finishes, there would be 5m-10m memory leak. So with thousands of tasks, soon it will use up all

Solution 1:

It was this config option that made my worker does not release memory.

CELERYD_TASK_TIME_LIMIT = 600

refer to: https://github.com/celery/celery/issues/1427

Solution 2:

There are two settings which can help you mitigate growing memory consumption of celery workers:

• Max tasks per child setting (v2.0+):

  With this option you can configure the maximum number of tasks a worker can execute before it’s replaced by a new process. This is useful if you have memory leaks you have no control over for example from closed source C extensions.

• Max memory per child setting (v4.0+):

  With this option you can configure the maximum amount of resident memory a worker can execute before it’s replaced by a new process. This is useful if you have memory leaks you have no control over for example from closed source C extensions.

Solution 3:

When you start your worker just set the max-tasks-per-child option like this to restart worker processes after every task:

celery -A app worker --loglevel=info --max-tasks-per-child=1

Here's the documentation:

http://docs.celeryproject.org/en/latest/userguide/workers.html#max-memory-per-child-setting

Solution 4:

This was an issue in celery which I think is fixed.

Please refer: https://github.com/celery/celery/issues/2927

Solution 5:

set worker_max_tasks_per_child in your settings