https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
218955b Updated backport on buster-swh from debian/0.5.2-1_swh1 (unstable-swh) 10 July 2020, 11:10:43 UTC
4aabbf8 Merge tag 'debian/0.5.2-1_swh1' into debian/buster-swh 10 July 2020, 11:10:42 UTC
d9864de Updated debian changelog for version 0.5.2 10 July 2020, 11:08:30 UTC
8a7128e Update upstream source from tag 'debian/upstream/0.5.2' Update to upstream version '0.5.2' with Debian dir 8499264f1f48674bf11b291218be39c112ee4345 10 July 2020, 11:08:30 UTC
c26569f New upstream version 0.5.2 10 July 2020, 11:08:29 UTC
f842383 d/rules: Avoid double pytest plugin loading clash 10 July 2020, 11:04:02 UTC
254e24a Do no expose pytest-plugin through setuptools, let modules require it when needed Defining the pytest-plugin though the pytest-plugin [1] makes it loaded by default. This creates loading issues on modules depending on scheduler but not on the pytest plugin scheduler exposes as explained in the doc [2] Instead we'll explicitely define to modules depending on the pytest plugins in their root conftest [3]: pytest_plugins = [ "swh.scheduler.pytest_plugin" ] [1] https://docs.pytest.org/en/stable/writing_plugins.html#setuptools-entry-points [2] https://docs.pytest.org/en/stable/writing_plugins.html#plugin-discovery-order-at-tool-startup [3] https://docs.pytest.org/en/stable/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file Related to D3475 Related to T2484 10 July 2020, 10:27:42 UTC
27c6511 Updated backport on buster-swh from debian/0.5.1-1_swh1 (unstable-swh) 09 July 2020, 09:53:53 UTC
abdd3c2 Merge tag 'debian/0.5.1-1_swh1' into debian/buster-swh 09 July 2020, 09:53:52 UTC
2107e14 Updated debian changelog for version 0.5.1 09 July 2020, 09:51:38 UTC
80d822f Update upstream source from tag 'debian/upstream/0.5.1' Update to upstream version '0.5.1' with Debian dir 8f091252a383122d11cea61b9032ee46a6ae5f33 09 July 2020, 09:51:37 UTC
7a6149f New upstream version 0.5.1 09 July 2020, 09:51:36 UTC
02fe44a Updated backport on buster-swh from debian/0.5.0-1_swh1 (unstable-swh) 09 July 2020, 08:22:58 UTC
d3c0e74 Merge tag 'debian/0.5.0-1_swh1' into debian/buster-swh 09 July 2020, 08:22:57 UTC
18e8170 Updated debian changelog for version 0.5.0 09 July 2020, 08:20:42 UTC
0bc33b2 New upstream version 0.5.0 09 July 2020, 08:20:41 UTC
ad2f47b Update upstream source from tag 'debian/upstream/0.5.0' Update to upstream version '0.5.0' with Debian dir d21727b06ae0dff2ba04108d3e7ec6c3d9013eef 09 July 2020, 08:20:41 UTC
ece598c requirements.txt: Remove future dependency This was needed for celery 4.4.4 but that version is not used anymore. 08 July 2020, 16:33:25 UTC
9322643 debian/pybuild.testfiles: Add missing new line at end of file 08 July 2020, 16:26:56 UTC
024c981 debian: Add pybuild.testfiles fixing tests execution in package build This is needed for the scheduler pytest plugin to be loaded. 08 July 2020, 16:08:19 UTC
7009c3b Move all celery-related fixtures to the swh.scheduler pytest plugin This allows us to reuse these fixtures in other modules without brittle swh.scheduler.tests.conftest star imports. Unfortunately, we can't really override pytest fixtures from one plugin to another. We therefore reimplement the fixtures provided by celery, inlining the static configuration and renaming them to our names in the process. This also adds a backwards-compatibility import from pytest_plugin to conftest, to allow old users of the conftest fixtures to keep working. 08 July 2020, 15:59:15 UTC
ce63e6a pytest.ini: Drop filterwarnings which never worked 07 July 2020, 10:18:50 UTC
a05ae5a Updated debian changelog for version 0.4.0 06 July 2020, 14:52:42 UTC
b99646b Update upstream source from tag 'debian/upstream/0.4.0' Update to upstream version '0.4.0' with Debian dir 19b359f62e15b440019ca58e670b97fb2a8659ad 06 July 2020, 14:52:42 UTC
7dadc14 New upstream version 0.4.0 06 July 2020, 14:52:41 UTC
b2cbb9b Move shareable fixtures out of conftest into a dedicated pytest plugin This avoids having to run `from swh.scheduler.tests.conftest import *` in other modules, e.g. swh.lister, to import and use the swh_scheduler pytest fixture. 06 July 2020, 14:42:04 UTC
7064182 Updated backport on buster-swh from debian/0.3.0-1_swh1 (unstable-swh) 06 July 2020, 10:25:40 UTC
cb8ba4f Merge tag 'debian/0.3.0-1_swh1' into debian/buster-swh 06 July 2020, 10:25:40 UTC
93dc413 Updated debian changelog for version 0.3.0 06 July 2020, 10:23:31 UTC
f118f3b Update upstream source from tag 'debian/upstream/0.3.0' Update to upstream version '0.3.0' with Debian dir 81527c8249f1b2d83bcfb358b3e0d0886f1a1db2 06 July 2020, 10:23:31 UTC
189d845 New upstream version 0.3.0 06 July 2020, 10:23:30 UTC
5b373ce Introduce a get_listed_origins endpoint This paginated endpoint allows retrieving information about the origins recorded by listers. 06 July 2020, 09:51:10 UTC
aefc5c9 Don't recurse into attrs objects when serializing We need to use our serialization hook recursively to make sure that we can deserialize nested data structures. 06 July 2020, 07:48:29 UTC
659dfbc Updated backport on buster-swh from debian/0.2.2-1_swh1 (unstable-swh) 22 June 2020, 12:09:11 UTC
b73301e Merge tag 'debian/0.2.2-1_swh1' into debian/buster-swh 22 June 2020, 12:09:10 UTC
9884b29 Updated debian changelog for version 0.2.2 22 June 2020, 12:07:05 UTC
6fcf8c9 Update upstream source from tag 'debian/upstream/0.2.2' Update to upstream version '0.2.2' with Debian dir e7d54983c07ac4f799723e397ca22c85dd6deab2 22 June 2020, 12:07:04 UTC
39d886b New upstream version 0.2.2 22 June 2020, 12:07:03 UTC
cc8fa7f Re-introduce the root endpoint for the rpc server 22 June 2020, 10:55:11 UTC
f4c4b82 Updated backport on buster-swh from debian/0.2.1-1_swh1 (unstable-swh) 22 June 2020, 10:15:06 UTC
eb5988d Merge tag 'debian/0.2.1-1_swh1' into debian/buster-swh 22 June 2020, 10:15:06 UTC
c871914 Updated debian changelog for version 0.2.1 22 June 2020, 10:12:50 UTC
9ace381 Update upstream source from tag 'debian/upstream/0.2.1' Update to upstream version '0.2.1' with Debian dir 5e8e4b1125e2f5ba7e48c23b28aedf79532862ac 22 June 2020, 10:12:49 UTC
fa7357b New upstream version 0.2.1 22 June 2020, 10:12:49 UTC
3b4084a Force celery >= 4.3 22 June 2020, 10:09:22 UTC
265bc8b The celery-monitor subcommand glob filtering needs celery >= 4.3 22 June 2020, 08:58:09 UTC
7478158 Updated backport on buster-swh from debian/0.2.0-1_swh1 (unstable-swh) 22 June 2020, 08:38:55 UTC
d25588c Merge tag 'debian/0.2.0-1_swh1' into debian/buster-swh 22 June 2020, 08:38:55 UTC
4177066 Updated debian changelog for version 0.2.0 22 June 2020, 08:36:49 UTC
434ac56 Update upstream source from tag 'debian/upstream/0.2.0' Update to upstream version '0.2.0' with Debian dir 6bcb470a30a47c5778da29817f19dfb1f95df145 22 June 2020, 08:36:48 UTC
da69466 New upstream version 0.2.0 22 June 2020, 08:36:48 UTC
8a1724a Add SQL for version 16 of the schema 22 June 2020, 08:26:40 UTC
d107a55 Implement storage of listed origins This new API endpoint allows listers to record the origins they have seen during their current run. Origins are identified by the lister instance, the url of the origin, and the type of loader that should be used to load this origin. The implementation allows listers just send the list of origins they've seen (with some lightweight extra information), leaving the backend to handle whether to do an insertion or an update to an existing origin. The current implementation doesn't disable origins that have disappeared when doing a full listing run. This step will be done by a separate "origin garbage collection" endpoint, which will peruse the `last_seen` field. 16 June 2020, 08:25:08 UTC
e0fa5c5 Move lister addition in scheduler tests to a pytest fixture This lets us keep the tests a little DRYer. 16 June 2020, 08:24:03 UTC
04894bd Lister.instance_name doesn't need a factory/default value 16 June 2020, 08:22:23 UTC
f520108 Improve support of primary keys This splits primary keys across "automatic" primary keys (handled by the database) and manual primary keys (managed by the user). Use the opportunity to improve/clarify the documentation of field metadata attributes. 16 June 2020, 08:22:12 UTC
1c93e55 Implement basic storage and retrieval of lister information This adds a pair a functions to the backend: - `get_or_create_lister` pulls the record for a given lister from the database - `update_lister` updates the record for a given lister in the database This is one of the basic building blocks for the integration of lister information directly in the scheduler database. Related to T2442. 15 June 2020, 13:41:02 UTC
466ac59 Introduce a SchedulerException base class This allows us to automatically serialize/deserialize exceptions under this base class within our RPC framework. 15 June 2020, 12:53:30 UTC
c509a12 Introduce some scaffolding for an attrs-based BaseSchedulerModel Alongside swh.model.model, this allows us to define data models for the objects the scheduler is working with, and to serialize/deserialize these objects transparently at the RPC layer. This also introduces some mild ORM-like logic so we can keep the actual SQL a little DRYer. 15 June 2020, 10:49:25 UTC
4c0c37b Use the automatic RPC client/server generation 11 June 2020, 09:42:37 UTC
aedd323 Replace swh-worker-control with a swh scheduler celery-monitor subcommand This new subcommand has two commands: - ping: checks whether the given worker instance answers within a given timeout - list-running: lists running tasks on the given worker instance 10 June 2020, 10:15:54 UTC
8411335 Remove double logging setup in cli The logging module is already initialized by the main swh.core cli; This only creates double logging with no advantages whatsoever. 10 June 2020, 09:30:31 UTC
873cdac Handle psycopg2 OperationalError in cli initialization When running the cli with default settings (i.e. pointing to a softwareheritage-scheduler-dev database), and the database doesn't exist, an OperationalError is raised. This shouldn't prevent (some of the) cli subcommands from working, so catch this error and ignore it as one of the scheduler backend setup failure modes. 10 June 2020, 09:28:19 UTC
daa4ad9 wrap-and-sort 09 June 2020, 15:32:58 UTC
37a1c27 Switch from vcversioner to setuptools-scm 09 June 2020, 15:32:20 UTC
28c5b8d Replace vcversioner with setuptools-scm 09 June 2020, 13:49:00 UTC
14cd5bb Blacken for python3.7+ 03 June 2020, 15:19:00 UTC
6ac3d56 Drop use of pifpaf and the "db" pytest mark We've been using pytest-postgresql for... a year (4117d5a). 03 June 2020, 10:34:11 UTC
8e44d5d Updated backport on buster-swh from debian/0.1.1-1_swh1 (unstable-swh) 03 June 2020, 09:41:08 UTC
8a33ce6 Merge tag 'debian/0.1.1-1_swh1' into debian/buster-swh 03 June 2020, 09:41:08 UTC
2aee335 Updated debian changelog for version 0.1.1 03 June 2020, 09:39:25 UTC
31d16ad Update upstream source from tag 'debian/upstream/0.1.1' Update to upstream version '0.1.1' with Debian dir e420c8ab5f0c960bdbefb9622b4836b16233ffa6 03 June 2020, 09:39:25 UTC
db7f167 New upstream version 0.1.1 03 June 2020, 09:39:24 UTC
3f42423 Add future dependency, missing from celery 4.4.4 Without future, the tests involving celery hang indefinitely. Upstream issue: https://github.com/celery/celery/issues/6145 03 June 2020, 09:29:58 UTC
2ad8436 Updated backport on buster-swh from debian/0.1.0-1_swh1 (unstable-swh) 19 May 2020, 09:54:10 UTC
8a35c30 Merge tag 'debian/0.1.0-1_swh1' into debian/buster-swh 19 May 2020, 09:54:10 UTC
4fb0a30 Updated debian changelog for version 0.1.0 19 May 2020, 09:52:31 UTC
262978b Update upstream source from tag 'debian/upstream/0.1.0' Update to upstream version '0.1.0' with Debian dir d8dcc4b3142822a076dc58cee3e1bcd627db8b8c 19 May 2020, 09:52:31 UTC
e06c756 New upstream version 0.1.0 19 May 2020, 09:52:30 UTC
92c0869 Celery runner: only schedule tasks when the buffer is less than 80% full The queries to pick up tasks from the scheduler sometimes degenerate when the number of tasks fetched is too low, which hangs the runner for all other tasks. Adding this lower bound helps postgresql use proper optimizations to pull tasks. 19 May 2020, 09:34:52 UTC
b839906 Disable the azure http logger in the celery worker base config This is suboptimal (we should move all of this to a logconfig where we can set this stuff), but this is consistent with how we do things currently. 19 May 2020, 09:14:25 UTC
2ea919c Fix black for py37 19 May 2020, 09:12:26 UTC
3a74069 test_scheduler: Fix pep8 violation This fixes ci build [1] [1] https://jenkins.softwareheritage.org/job/DSCH/job/tests/859/console 12 May 2020, 09:55:09 UTC
2cc8aa0 setup.py: add documentation link 29 April 2020, 16:33:16 UTC
1abff22 setup: Update the minimum required runtime python3 version Related to T2367 20 April 2020, 15:29:49 UTC
551ceac Add a pyproject.toml file to target py37 for black 08 April 2020, 20:16:58 UTC
cc0ef04 Enable black - blackify all the python files, - enable black in pre-commit, - add a black tox environment. 08 April 2020, 14:58:01 UTC
77b2d0b tests: Adapt model according to latest change origin model no longer allows to have type. Related to f533f62bbf114cfcc29f7c72307c4dfbe99cf048 27 March 2020, 06:43:03 UTC
18a8430 Updated backport on buster-swh from debian/0.0.72-1_swh1 (unstable-swh) 23 March 2020, 12:14:05 UTC
95dfa2a Merge tag 'debian/0.0.72-1_swh1' into debian/buster-swh 23 March 2020, 12:14:05 UTC
2747e34 Updated debian changelog for version 0.0.72 23 March 2020, 12:12:00 UTC
75bf007 New upstream version 0.0.72 23 March 2020, 12:11:59 UTC
80dc563 Update upstream source from tag 'debian/upstream/0.0.72' Update to upstream version '0.0.72' with Debian dir f1a9e9f451bda3277c75724e6975cd3ed9f2af13 23 March 2020, 12:11:59 UTC
447b937 Add build-dependency on python3-pika 23 March 2020, 12:06:58 UTC
e6c2a86 Implement listener on top of pika instead of celery 23 March 2020, 11:52:06 UTC
68c42fb scheduler.backend_es: Leave index opened when streaming bulk Prior to this commit, we had the proper behavior of closing index when done streaming. Unfortunately, this created too much gc on es nodes down the line. So for now, we remove that behavior. Note that this implies we need another cog that makes a pass once in a while on indices to close. Also, this has been running on production for 2 weeks now and no more gc issues arose since then. 26 February 2020, 09:34:09 UTC
af58466 backend: Make create_task_type idempotent There is no reason to raise an error when a task type has already been created and it enables to stop leaking psycopg2 IntegrityError exception as part of the scheduler interface. 18 February 2020, 14:17:02 UTC
b92e3fd Use swh-storage validation proxy. Required by swh-storage >= v0.0.172. 12 February 2020, 12:48:52 UTC
73d1e5e cli.task: Change `get_storage` according to latest change 31 January 2020, 08:18:25 UTC
1c923aa test_cli: Fix storage instantiation following api change Using the `swh.storage.get_storage` function instead of calling directly the class name. This actually fixes the master ci build [1] [1] https://jenkins.softwareheritage.org/job/DSCH/job/tests/743/console 31 January 2020, 08:16:20 UTC
back to top