Skip to main content

Browse the archive

https://forge.softwareheritage.org/source/swh-scheduler.git
13 March 2024, 03:48:39 UTC
Revision 68c42fb3ca75a9061b506ce65f3a900e22c76adc authored by Antoine R. Dumont (@ardumont) on 03 February 2020, 08:20:57 UTC, committed by Antoine R. Dumont (@ardumont) on 26 February 2020, 09:34:09 UTC 
scheduler.backend_es: Leave index opened when streaming bulk
 
Prior to this commit, we had the proper behavior of closing index when done
streaming. Unfortunately, this created too much gc on es nodes down the line.
So for now, we remove that behavior.

Note that this implies we need another cog that makes a pass once in a while on
indices to close.

Also, this has been running on production for 2 weeks now and no more gc issues
arose since then.
1 parent af58466
History
Tip revision: 68c42fb3ca75a9061b506ce65f3a900e22c76adc authored by Antoine R. Dumont (@ardumont) on 03 February 2020, 08:20:57 UTC
scheduler.backend_es: Leave index opened when streaming bulk
Tip revision: 68c42fb
File Mode Size
bin
data
docs
sql
swh
.gitignore -rw-r--r-- 129 bytes
.pre-commit-config.yaml -rw-r--r-- 1.1 KB
AUTHORS -rw-r--r-- 112 bytes
CODE_OF_CONDUCT.md -rw-r--r-- 3.3 KB
CONTRIBUTORS -rw-r--r-- 14 bytes
LICENSE -rw-r--r-- 34.3 KB
LICENSE.Celery -rw-r--r-- 2.6 KB
MANIFEST.in -rw-r--r-- 154 bytes
Makefile -rw-r--r-- 163 bytes
README.md -rw-r--r-- 292 bytes
conftest.py -rw-r--r-- 292 bytes
mypy.ini -rw-r--r-- 505 bytes
pytest.ini -rw-r--r-- 54 bytes
requirements-swh.txt -rw-r--r-- 51 bytes
requirements-test.txt -rw-r--r-- 92 bytes
requirements.txt -rw-r--r-- 349 bytes
setup.py -rwxr-xr-x 2.2 KB
tox.ini -rw-r--r-- 600 bytes

README.md

back to top