Skip to main content

Browse the archive

https://forge.softwareheritage.org/source/swh-scheduler.git
13 March 2024, 03:48:39 UTC
Revision 87e66faa300df8e1e3cb8c82660d8155e877564f authored by Nicolas Dandrimont on 23 July 2021, 09:48:23 UTC, committed by Nicolas Dandrimont on 23 July 2021, 09:56:32 UTC 
Only record last_visited and last_successful in origin_visit_stats
 
After using this schema for a while, all queries can be implemented in
terms of these two timestamps, instead of the four original
last_eventful, last_uneventful, last_failed and last_notfound
timestamps.

This ends up simplifying the logic within the journal client, as well as
that of the grab_next_visits query builder.

To make this change work, we also stop considering out of order messages
altogether in journal_client. This welcome simplification is an accuracy
tradeoff that is explained in the updated documentation of the journal
client:

.. [1] Ignoring out of order messages makes the initialization of the
      origin_visit_status table (from a full journal) less deterministic: only the
      `last_visit`, `last_visit_state` and `last_successful` fields are guaranteed
      to be exact, the `next_position_offset` field is a best effort estimate
      (which should converge once the client has run for a while on in-order
      messages).
1 parent 3ca0d65
History
Tip revision: 87e66faa300df8e1e3cb8c82660d8155e877564f authored by Nicolas Dandrimont on 23 July 2021, 09:48:23 UTC
Only record last_visited and last_successful in origin_visit_stats
Tip revision: 87e66fa
File Mode Size
data
docs
sql
swh
.gitignore -rw-r--r-- 129 bytes
.pre-commit-config.yaml -rw-r--r-- 1.0 KB
AUTHORS -rw-r--r-- 112 bytes
CODE_OF_CONDUCT.md -rw-r--r-- 3.3 KB
CONTRIBUTORS -rw-r--r-- 14 bytes
LICENSE -rw-r--r-- 34.3 KB
LICENSE.Celery -rw-r--r-- 2.6 KB
MANIFEST.in -rw-r--r-- 154 bytes
Makefile -rw-r--r-- 163 bytes
README.md -rw-r--r-- 292 bytes
conftest.py -rw-r--r-- 620 bytes
mypy.ini -rw-r--r-- 750 bytes
pyproject.toml -rw-r--r-- 237 bytes
pytest.ini -rw-r--r-- 33 bytes
requirements-journal.txt -rw-r--r-- 12 bytes
requirements-simulator.txt -rw-r--r-- 21 bytes
requirements-swh.txt -rw-r--r-- 50 bytes
requirements-test.txt -rw-r--r-- 117 bytes
requirements.txt -rw-r--r-- 378 bytes
setup.cfg -rw-r--r-- 247 bytes
setup.py -rwxr-xr-x 2.5 KB
tox.ini -rw-r--r-- 1.5 KB

README.md

back to top