Skip to main content

Browse the archive

https://forge.softwareheritage.org/source/swh-scheduler.git
13 March 2024, 03:48:39 UTC
Revision 14feab9523804dd8b18acab29632a38cabe3e2d9 authored by Nicolas Dandrimont on 03 February 2021, 18:51:30 UTC, committed by Nicolas Dandrimont on 03 February 2021, 19:10:26 UTC 
celery: acknowledge tasks as soon as they're received
 
With late acknowledgements, RabbitMQ will re-send tasks to clients even
if they can't ever complete the task (e.g. when the task gets killed
because the machine is out of memory).

This problem only increases over time, leading to complete starvation of
the ingestion system.

Now that we have multiple mechanisms to issue retries of tasks, we can
use early acknowledgements for tasks instead, which should mitigate the
ongoing starvation, at the expense of having to retry tasks externally.
1 parent aaffff2
History
Tip revision: 14feab9523804dd8b18acab29632a38cabe3e2d9 authored by Nicolas Dandrimont on 03 February 2021, 18:51:30 UTC
celery: acknowledge tasks as soon as they're received
Tip revision: 14feab9
File Mode Size
data
docs
sql
swh
.gitignore -rw-r--r-- 129 bytes
.pre-commit-config.yaml -rw-r--r-- 1.0 KB
AUTHORS -rw-r--r-- 112 bytes
CODE_OF_CONDUCT.md -rw-r--r-- 3.3 KB
CONTRIBUTORS -rw-r--r-- 14 bytes
LICENSE -rw-r--r-- 34.3 KB
LICENSE.Celery -rw-r--r-- 2.6 KB
MANIFEST.in -rw-r--r-- 154 bytes
Makefile -rw-r--r-- 163 bytes
README.md -rw-r--r-- 292 bytes
conftest.py -rw-r--r-- 620 bytes
mypy.ini -rw-r--r-- 750 bytes
pyproject.toml -rw-r--r-- 237 bytes
pytest.ini -rw-r--r-- 33 bytes
requirements-journal.txt -rw-r--r-- 12 bytes
requirements-simulator.txt -rw-r--r-- 21 bytes
requirements-swh.txt -rw-r--r-- 47 bytes
requirements-test.txt -rw-r--r-- 65 bytes
requirements.txt -rw-r--r-- 403 bytes
setup.cfg -rw-r--r-- 247 bytes
setup.py -rwxr-xr-x 2.5 KB
tox.ini -rw-r--r-- 673 bytes

README.md

back to top