https://forge.softwareheritage.org/source/swh-scheduler.git
Revision 8c4ae9f14d6abdca41a4f01b438310501ecb6259 authored by Antoine R. Dumont (@ardumont) on 29 June 2021, 14:00:01 UTC, committed by Antoine R. Dumont (@ardumont) on 06 July 2021, 12:35:13 UTC
For origin without any last_update information [1], the journal client is now also in charge of moving their next position in the queue for rescheduling. Depending on their status, the next position offset and next_visit_queue_position are updated after each visit completes: - if the visit has failed, increase the next visit target by the minimal visit interval (to take into account transient loading issues) - if the visit is successful, and records some changes, decrease the visit interval index by 2 (visit the origin *way* more often). - if the visit is successful, and records no changes, increase the visit interval index by 1 (visit the origin less often). We then set the next visit target to its current value + the new visit interval multiplied by a random fudge factor (picked in the -/+ 10% range). The fudge factor allows the visits to spread out, avoiding "bursts" of loaded origins e.g. when a number of origins from a single hoster are processed at once. Note that the computations happen for all origins for simplicity and code maintenance but it will only be used by a new soon-to-be scheduling policy. [1] Lister cannot provide it for some reason.
1 parent cb1edf1
Tip revision: 8c4ae9f14d6abdca41a4f01b438310501ecb6259 authored by Antoine R. Dumont (@ardumont) on 29 June 2021, 14:00:01 UTC
journal_client: Compute next position for origin visit
journal_client: Compute next position for origin visit
Tip revision: 8c4ae9f
File | Mode | Size |
---|---|---|
data | ||
docs | ||
sql | ||
swh | ||
.gitignore | -rw-r--r-- | 129 bytes |
.pre-commit-config.yaml | -rw-r--r-- | 1.0 KB |
AUTHORS | -rw-r--r-- | 112 bytes |
CODE_OF_CONDUCT.md | -rw-r--r-- | 3.3 KB |
CONTRIBUTORS | -rw-r--r-- | 14 bytes |
LICENSE | -rw-r--r-- | 34.3 KB |
LICENSE.Celery | -rw-r--r-- | 2.6 KB |
MANIFEST.in | -rw-r--r-- | 154 bytes |
Makefile | -rw-r--r-- | 163 bytes |
README.md | -rw-r--r-- | 292 bytes |
conftest.py | -rw-r--r-- | 620 bytes |
mypy.ini | -rw-r--r-- | 750 bytes |
pyproject.toml | -rw-r--r-- | 237 bytes |
pytest.ini | -rw-r--r-- | 33 bytes |
requirements-journal.txt | -rw-r--r-- | 12 bytes |
requirements-simulator.txt | -rw-r--r-- | 21 bytes |
requirements-swh.txt | -rw-r--r-- | 50 bytes |
requirements-test.txt | -rw-r--r-- | 117 bytes |
requirements.txt | -rw-r--r-- | 378 bytes |
setup.cfg | -rw-r--r-- | 247 bytes |
setup.py | -rwxr-xr-x | 2.5 KB |
tox.ini | -rw-r--r-- | 1.5 KB |
Computing file changes ...