https://forge.softwareheritage.org/source/swh-scheduler.git
Revision 8c4ae9f14d6abdca41a4f01b438310501ecb6259 authored by Antoine R. Dumont (@ardumont) on 29 June 2021, 14:00:01 UTC, committed by Antoine R. Dumont (@ardumont) on 06 July 2021, 12:35:13 UTC
For origin without any last_update information [1], the journal client is now also in
charge of moving their next position in the queue for rescheduling. Depending on their
status, the next position offset and next_visit_queue_position are updated after each
visit completes:

- if the visit has failed, increase the next visit target by the minimal visit
  interval (to take into account transient loading issues)
- if the visit is successful, and records some changes, decrease the visit interval
  index by 2 (visit the origin *way* more often).
- if the visit is successful, and records no changes, increase the visit interval index
  by 1 (visit the origin less often).

We then set the next visit target to its current value + the new visit interval
multiplied by a random fudge factor (picked in the -/+ 10% range).

The fudge factor allows the visits to spread out, avoiding "bursts" of loaded origins
e.g. when a number of origins from a single hoster are processed at once.

Note that the computations happen for all origins for simplicity and code maintenance
but it will only be used by a new soon-to-be scheduling policy.

[1] Lister cannot provide it for some reason.
1 parent cb1edf1
History
Tip revision: 8c4ae9f14d6abdca41a4f01b438310501ecb6259 authored by Antoine R. Dumont (@ardumont) on 29 June 2021, 14:00:01 UTC
journal_client: Compute next position for origin visit
Tip revision: 8c4ae9f
File Mode Size
data
docs
sql
swh
.gitignore -rw-r--r-- 129 bytes
.pre-commit-config.yaml -rw-r--r-- 1.0 KB
AUTHORS -rw-r--r-- 112 bytes
CODE_OF_CONDUCT.md -rw-r--r-- 3.3 KB
CONTRIBUTORS -rw-r--r-- 14 bytes
LICENSE -rw-r--r-- 34.3 KB
LICENSE.Celery -rw-r--r-- 2.6 KB
MANIFEST.in -rw-r--r-- 154 bytes
Makefile -rw-r--r-- 163 bytes
README.md -rw-r--r-- 292 bytes
conftest.py -rw-r--r-- 620 bytes
mypy.ini -rw-r--r-- 750 bytes
pyproject.toml -rw-r--r-- 237 bytes
pytest.ini -rw-r--r-- 33 bytes
requirements-journal.txt -rw-r--r-- 12 bytes
requirements-simulator.txt -rw-r--r-- 21 bytes
requirements-swh.txt -rw-r--r-- 50 bytes
requirements-test.txt -rw-r--r-- 117 bytes
requirements.txt -rw-r--r-- 378 bytes
setup.cfg -rw-r--r-- 247 bytes
setup.py -rwxr-xr-x 2.5 KB
tox.ini -rw-r--r-- 1.5 KB

README.md

back to top