Skip to main content

Browse the archive

https://forge.softwareheritage.org/source/swh-scheduler.git
13 March 2024, 03:48:39 UTC
Revision 2abb39368405ce684e2fb54dda03d4504328db6f authored by Valentin Lorentz on 22 November 2021, 12:32:20 UTC, committed by Valentin Lorentz on 22 November 2021, 12:36:24 UTC 
Fix CardinalityViolation in grab_next_visits on duplicate origins
 
grab_next_visits grabs from `listed_origins`, whose primary key is
`(lister_id, url, visit_type)` and uses it to upsert in origin_visit_stats,
whose primary key is `(url, visit_type)`.
This causes the error `ON CONFLICT DO UPDATE command cannot affect row a
second time` when the same (origin, type) pair is grabbed twice.

This commit deduplicates the (origin, type) pairs before upserting.
1 parent 00ff02e
History
Tip revision: 2abb39368405ce684e2fb54dda03d4504328db6f authored by Valentin Lorentz on 22 November 2021, 12:32:20 UTC
Fix CardinalityViolation in grab_next_visits on duplicate origins
Tip revision: 2abb393
File Mode Size
data
docs
sql
swh
.gitignore -rw-r--r-- 129 bytes
.pre-commit-config.yaml -rw-r--r-- 1.0 KB
AUTHORS -rw-r--r-- 112 bytes
CODE_OF_CONDUCT.md -rw-r--r-- 3.3 KB
CONTRIBUTORS -rw-r--r-- 14 bytes
LICENSE -rw-r--r-- 34.3 KB
LICENSE.Celery -rw-r--r-- 2.6 KB
MANIFEST.in -rw-r--r-- 154 bytes
Makefile -rw-r--r-- 163 bytes
README.md -rw-r--r-- 292 bytes
conftest.py -rw-r--r-- 664 bytes
mypy.ini -rw-r--r-- 750 bytes
pyproject.toml -rw-r--r-- 237 bytes
pytest.ini -rw-r--r-- 33 bytes
requirements-journal.txt -rw-r--r-- 12 bytes
requirements-simulator.txt -rw-r--r-- 21 bytes
requirements-swh.txt -rw-r--r-- 50 bytes
requirements-test.txt -rw-r--r-- 155 bytes
requirements.txt -rw-r--r-- 384 bytes
setup.cfg -rw-r--r-- 247 bytes
setup.py -rwxr-xr-x 2.5 KB
tox.ini -rw-r--r-- 1.5 KB

README.md

back to top