Skip to main content

Browse the archive

https://forge.softwareheritage.org/source/swh-scheduler.git
13 March 2024, 03:48:39 UTC
Revision 50d7fd7ff49b02654c1416f80e786acac3a980d5 authored by Nicolas Dandrimont on 27 October 2021, 10:09:42 UTC, committed by Antoine R. Dumont (@ardumont) on 28 October 2021, 11:06:56 UTC 
Add a new cli endpoint to schedule recurrent visits in Celery
 
For each known visit type, we run a loop which:
 - monitors the size of the relevant celery queue
 - schedules more visits of the relevant type once the number of
 available slots goes over a given threshold (currently set to 5% of the
 max queue size).

The scheduling of visits combines multiple scheduling policies, for now
using static ratios set in the `POLICY_RATIOS` dict. We emit a warning
if the ratio of origins fetched for each policy is skewed with respect
to the original request (allowing, for now, manual adjustement of the
ratios).

The CLI endpoint spawns one thread for each visit type, which all handle
connections to RabbitMQ and the scheduler backend separately. For now,
we handle exceptions in the visit scheduling threads by (stupidly)
respawning the relevant thread directly. We should probably improve this
to give up after a specific number of tries.

Co-authored-by: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
1 parent 0c7ef27
History
Tip revision: 50d7fd7ff49b02654c1416f80e786acac3a980d5 authored by Nicolas Dandrimont on 27 October 2021, 10:09:42 UTC
Add a new cli endpoint to schedule recurrent visits in Celery
Tip revision: 50d7fd7
File Mode Size
data
docs
sql
swh
.gitignore -rw-r--r-- 129 bytes
.pre-commit-config.yaml -rw-r--r-- 1.0 KB
AUTHORS -rw-r--r-- 112 bytes
CODE_OF_CONDUCT.md -rw-r--r-- 3.3 KB
CONTRIBUTORS -rw-r--r-- 14 bytes
LICENSE -rw-r--r-- 34.3 KB
LICENSE.Celery -rw-r--r-- 2.6 KB
MANIFEST.in -rw-r--r-- 154 bytes
Makefile -rw-r--r-- 163 bytes
README.md -rw-r--r-- 292 bytes
conftest.py -rw-r--r-- 664 bytes
mypy.ini -rw-r--r-- 750 bytes
pyproject.toml -rw-r--r-- 237 bytes
pytest.ini -rw-r--r-- 33 bytes
requirements-journal.txt -rw-r--r-- 12 bytes
requirements-simulator.txt -rw-r--r-- 21 bytes
requirements-swh.txt -rw-r--r-- 50 bytes
requirements-test.txt -rw-r--r-- 155 bytes
requirements.txt -rw-r--r-- 384 bytes
setup.cfg -rw-r--r-- 247 bytes
setup.py -rwxr-xr-x 2.5 KB
tox.ini -rw-r--r-- 1.5 KB

README.md

back to top