7478158 | Jenkins for Software Heritage | 22 June 2020, 08:38:55 UTC | Updated backport on buster-swh from debian/0.2.0-1_swh1 (unstable-swh) | 22 June 2020, 08:38:55 UTC |
d25588c | Jenkins for Software Heritage | 22 June 2020, 08:38:55 UTC | Merge tag 'debian/0.2.0-1_swh1' into debian/buster-swh | 22 June 2020, 08:38:55 UTC |
4177066 | Jenkins for Software Heritage | 22 June 2020, 08:36:49 UTC | Updated debian changelog for version 0.2.0 | 22 June 2020, 08:36:49 UTC |
434ac56 | Jenkins for Software Heritage | 22 June 2020, 08:36:48 UTC | Update upstream source from tag 'debian/upstream/0.2.0' Update to upstream version '0.2.0' with Debian dir 6bcb470a30a47c5778da29817f19dfb1f95df145 | 22 June 2020, 08:36:48 UTC |
da69466 | Jenkins for Software Heritage | 22 June 2020, 08:36:48 UTC | New upstream version 0.2.0 | 22 June 2020, 08:36:48 UTC |
8a1724a | Nicolas Dandrimont | 22 June 2020, 08:26:40 UTC | Add SQL for version 16 of the schema | 22 June 2020, 08:26:40 UTC |
d107a55 | Nicolas Dandrimont | 16 June 2020, 08:25:08 UTC | Implement storage of listed origins This new API endpoint allows listers to record the origins they have seen during their current run. Origins are identified by the lister instance, the url of the origin, and the type of loader that should be used to load this origin. The implementation allows listers just send the list of origins they've seen (with some lightweight extra information), leaving the backend to handle whether to do an insertion or an update to an existing origin. The current implementation doesn't disable origins that have disappeared when doing a full listing run. This step will be done by a separate "origin garbage collection" endpoint, which will peruse the `last_seen` field. | 16 June 2020, 08:25:08 UTC |
e0fa5c5 | Nicolas Dandrimont | 16 June 2020, 08:24:03 UTC | Move lister addition in scheduler tests to a pytest fixture This lets us keep the tests a little DRYer. | 16 June 2020, 08:24:03 UTC |
04894bd | Nicolas Dandrimont | 16 June 2020, 08:22:23 UTC | Lister.instance_name doesn't need a factory/default value | 16 June 2020, 08:22:23 UTC |
f520108 | Nicolas Dandrimont | 16 June 2020, 08:08:59 UTC | Improve support of primary keys This splits primary keys across "automatic" primary keys (handled by the database) and manual primary keys (managed by the user). Use the opportunity to improve/clarify the documentation of field metadata attributes. | 16 June 2020, 08:22:12 UTC |
1c93e55 | Nicolas Dandrimont | 12 June 2020, 10:24:20 UTC | Implement basic storage and retrieval of lister information This adds a pair a functions to the backend: - `get_or_create_lister` pulls the record for a given lister from the database - `update_lister` updates the record for a given lister in the database This is one of the basic building blocks for the integration of lister information directly in the scheduler database. Related to T2442. | 15 June 2020, 13:41:02 UTC |
466ac59 | Nicolas Dandrimont | 15 June 2020, 12:46:28 UTC | Introduce a SchedulerException base class This allows us to automatically serialize/deserialize exceptions under this base class within our RPC framework. | 15 June 2020, 12:53:30 UTC |
c509a12 | Nicolas Dandrimont | 12 June 2020, 09:03:26 UTC | Introduce some scaffolding for an attrs-based BaseSchedulerModel Alongside swh.model.model, this allows us to define data models for the objects the scheduler is working with, and to serialize/deserialize these objects transparently at the RPC layer. This also introduces some mild ORM-like logic so we can keep the actual SQL a little DRYer. | 15 June 2020, 10:49:25 UTC |
4c0c37b | Nicolas Dandrimont | 10 June 2020, 14:09:53 UTC | Use the automatic RPC client/server generation | 11 June 2020, 09:42:37 UTC |
aedd323 | Nicolas Dandrimont | 10 June 2020, 09:31:45 UTC | Replace swh-worker-control with a swh scheduler celery-monitor subcommand This new subcommand has two commands: - ping: checks whether the given worker instance answers within a given timeout - list-running: lists running tasks on the given worker instance | 10 June 2020, 10:15:54 UTC |
8411335 | Nicolas Dandrimont | 10 June 2020, 09:30:31 UTC | Remove double logging setup in cli The logging module is already initialized by the main swh.core cli; This only creates double logging with no advantages whatsoever. | 10 June 2020, 09:30:31 UTC |
873cdac | Nicolas Dandrimont | 10 June 2020, 09:28:19 UTC | Handle psycopg2 OperationalError in cli initialization When running the cli with default settings (i.e. pointing to a softwareheritage-scheduler-dev database), and the database doesn't exist, an OperationalError is raised. This shouldn't prevent (some of the) cli subcommands from working, so catch this error and ignore it as one of the scheduler backend setup failure modes. | 10 June 2020, 09:28:19 UTC |
daa4ad9 | Nicolas Dandrimont | 09 June 2020, 15:32:58 UTC | wrap-and-sort | 09 June 2020, 15:32:58 UTC |
37a1c27 | Nicolas Dandrimont | 09 June 2020, 15:32:20 UTC | Switch from vcversioner to setuptools-scm | 09 June 2020, 15:32:20 UTC |
28c5b8d | Nicolas Dandrimont | 09 June 2020, 13:47:26 UTC | Replace vcversioner with setuptools-scm | 09 June 2020, 13:49:00 UTC |
14cd5bb | Nicolas Dandrimont | 03 June 2020, 15:17:50 UTC | Blacken for python3.7+ | 03 June 2020, 15:19:00 UTC |
6ac3d56 | Nicolas Dandrimont | 03 June 2020, 10:34:11 UTC | Drop use of pifpaf and the "db" pytest mark We've been using pytest-postgresql for... a year (4117d5a). | 03 June 2020, 10:34:11 UTC |
8e44d5d | Jenkins for Software Heritage | 03 June 2020, 09:41:08 UTC | Updated backport on buster-swh from debian/0.1.1-1_swh1 (unstable-swh) | 03 June 2020, 09:41:08 UTC |
8a33ce6 | Jenkins for Software Heritage | 03 June 2020, 09:41:08 UTC | Merge tag 'debian/0.1.1-1_swh1' into debian/buster-swh | 03 June 2020, 09:41:08 UTC |
2aee335 | Jenkins for Software Heritage | 03 June 2020, 09:39:25 UTC | Updated debian changelog for version 0.1.1 | 03 June 2020, 09:39:25 UTC |
31d16ad | Jenkins for Software Heritage | 03 June 2020, 09:39:25 UTC | Update upstream source from tag 'debian/upstream/0.1.1' Update to upstream version '0.1.1' with Debian dir e420c8ab5f0c960bdbefb9622b4836b16233ffa6 | 03 June 2020, 09:39:25 UTC |
db7f167 | Jenkins for Software Heritage | 03 June 2020, 09:39:24 UTC | New upstream version 0.1.1 | 03 June 2020, 09:39:24 UTC |
3f42423 | Nicolas Dandrimont | 03 June 2020, 09:29:58 UTC | Add future dependency, missing from celery 4.4.4 Without future, the tests involving celery hang indefinitely. Upstream issue: https://github.com/celery/celery/issues/6145 | 03 June 2020, 09:29:58 UTC |
2ad8436 | Jenkins for Software Heritage | 19 May 2020, 09:54:10 UTC | Updated backport on buster-swh from debian/0.1.0-1_swh1 (unstable-swh) | 19 May 2020, 09:54:10 UTC |
8a35c30 | Jenkins for Software Heritage | 19 May 2020, 09:54:10 UTC | Merge tag 'debian/0.1.0-1_swh1' into debian/buster-swh | 19 May 2020, 09:54:10 UTC |
4fb0a30 | Jenkins for Software Heritage | 19 May 2020, 09:52:31 UTC | Updated debian changelog for version 0.1.0 | 19 May 2020, 09:52:31 UTC |
262978b | Jenkins for Software Heritage | 19 May 2020, 09:52:31 UTC | Update upstream source from tag 'debian/upstream/0.1.0' Update to upstream version '0.1.0' with Debian dir d8dcc4b3142822a076dc58cee3e1bcd627db8b8c | 19 May 2020, 09:52:31 UTC |
e06c756 | Jenkins for Software Heritage | 19 May 2020, 09:52:30 UTC | New upstream version 0.1.0 | 19 May 2020, 09:52:30 UTC |
92c0869 | Nicolas Dandrimont | 19 May 2020, 09:30:13 UTC | Celery runner: only schedule tasks when the buffer is less than 80% full The queries to pick up tasks from the scheduler sometimes degenerate when the number of tasks fetched is too low, which hangs the runner for all other tasks. Adding this lower bound helps postgresql use proper optimizations to pull tasks. | 19 May 2020, 09:34:52 UTC |
b839906 | Nicolas Dandrimont | 19 May 2020, 09:12:55 UTC | Disable the azure http logger in the celery worker base config This is suboptimal (we should move all of this to a logconfig where we can set this stuff), but this is consistent with how we do things currently. | 19 May 2020, 09:14:25 UTC |
2ea919c | Nicolas Dandrimont | 19 May 2020, 09:12:26 UTC | Fix black for py37 | 19 May 2020, 09:12:26 UTC |
3a74069 | Antoine R. Dumont (@ardumont) | 12 May 2020, 09:55:09 UTC | test_scheduler: Fix pep8 violation This fixes ci build [1] [1] https://jenkins.softwareheritage.org/job/DSCH/job/tests/859/console | 12 May 2020, 09:55:09 UTC |
2cc8aa0 | Stefano Zacchiroli | 29 April 2020, 16:33:16 UTC | setup.py: add documentation link | 29 April 2020, 16:33:16 UTC |
1abff22 | Antoine R. Dumont (@ardumont) | 20 April 2020, 15:29:49 UTC | setup: Update the minimum required runtime python3 version Related to T2367 | 20 April 2020, 15:29:49 UTC |
551ceac | David Douard | 08 April 2020, 20:16:58 UTC | Add a pyproject.toml file to target py37 for black | 08 April 2020, 20:16:58 UTC |
cc0ef04 | David Douard | 08 April 2020, 14:58:01 UTC | Enable black - blackify all the python files, - enable black in pre-commit, - add a black tox environment. | 08 April 2020, 14:58:01 UTC |
77b2d0b | Antoine R. Dumont (@ardumont) | 27 March 2020, 06:43:03 UTC | tests: Adapt model according to latest change origin model no longer allows to have type. Related to f533f62bbf114cfcc29f7c72307c4dfbe99cf048 | 27 March 2020, 06:43:03 UTC |
18a8430 | Jenkins for Software Heritage | 23 March 2020, 12:14:05 UTC | Updated backport on buster-swh from debian/0.0.72-1_swh1 (unstable-swh) | 23 March 2020, 12:14:05 UTC |
95dfa2a | Jenkins for Software Heritage | 23 March 2020, 12:14:05 UTC | Merge tag 'debian/0.0.72-1_swh1' into debian/buster-swh | 23 March 2020, 12:14:05 UTC |
2747e34 | Jenkins for Software Heritage | 23 March 2020, 12:12:00 UTC | Updated debian changelog for version 0.0.72 | 23 March 2020, 12:12:00 UTC |
75bf007 | Jenkins for Software Heritage | 23 March 2020, 12:11:59 UTC | New upstream version 0.0.72 | 23 March 2020, 12:11:59 UTC |
80dc563 | Jenkins for Software Heritage | 23 March 2020, 12:11:59 UTC | Update upstream source from tag 'debian/upstream/0.0.72' Update to upstream version '0.0.72' with Debian dir f1a9e9f451bda3277c75724e6975cd3ed9f2af13 | 23 March 2020, 12:11:59 UTC |
447b937 | Nicolas Dandrimont | 23 March 2020, 12:06:58 UTC | Add build-dependency on python3-pika | 23 March 2020, 12:06:58 UTC |
e6c2a86 | Nicolas Dandrimont | 23 March 2020, 09:45:30 UTC | Implement listener on top of pika instead of celery | 23 March 2020, 11:52:06 UTC |
68c42fb | Antoine R. Dumont (@ardumont) | 03 February 2020, 08:20:57 UTC | scheduler.backend_es: Leave index opened when streaming bulk Prior to this commit, we had the proper behavior of closing index when done streaming. Unfortunately, this created too much gc on es nodes down the line. So for now, we remove that behavior. Note that this implies we need another cog that makes a pass once in a while on indices to close. Also, this has been running on production for 2 weeks now and no more gc issues arose since then. | 26 February 2020, 09:34:09 UTC |
af58466 | Antoine Lambert | 17 February 2020, 15:55:20 UTC | backend: Make create_task_type idempotent There is no reason to raise an error when a task type has already been created and it enables to stop leaking psycopg2 IntegrityError exception as part of the scheduler interface. | 18 February 2020, 14:17:02 UTC |
b92e3fd | Valentin Lorentz | 12 February 2020, 12:48:52 UTC | Use swh-storage validation proxy. Required by swh-storage >= v0.0.172. | 12 February 2020, 12:48:52 UTC |
73d1e5e | Antoine R. Dumont (@ardumont) | 31 January 2020, 08:18:25 UTC | cli.task: Change `get_storage` according to latest change | 31 January 2020, 08:18:25 UTC |
1c923aa | Antoine R. Dumont (@ardumont) | 31 January 2020, 08:16:20 UTC | test_cli: Fix storage instantiation following api change Using the `swh.storage.get_storage` function instead of calling directly the class name. This actually fixes the master ci build [1] [1] https://jenkins.softwareheritage.org/job/DSCH/job/tests/743/console | 31 January 2020, 08:16:20 UTC |
673d473 | Jenkins for Software Heritage | 23 January 2020, 13:31:30 UTC | Updated backport on buster-swh from debian/0.0.71-1_swh1 (unstable-swh) | 23 January 2020, 13:31:30 UTC |
2eb0abb | Jenkins for Software Heritage | 23 January 2020, 13:31:30 UTC | Merge tag 'debian/0.0.71-1_swh1' into debian/buster-swh | 23 January 2020, 13:31:30 UTC |
900330a | Jenkins for Software Heritage | 23 January 2020, 13:29:33 UTC | Updated debian changelog for version 0.0.71 | 23 January 2020, 13:29:33 UTC |
075711c | Jenkins for Software Heritage | 23 January 2020, 13:29:33 UTC | Update upstream source from tag 'debian/upstream/0.0.71' Update to upstream version '0.0.71' with Debian dir 28ff51b420669492d8b274560eb126629fd1ef31 | 23 January 2020, 13:29:33 UTC |
cfaa584 | Jenkins for Software Heritage | 23 January 2020, 13:29:32 UTC | New upstream version 0.0.71 | 23 January 2020, 13:29:32 UTC |
f6cc231 | Antoine R. Dumont (@ardumont) | 23 January 2020, 13:21:21 UTC | sentry: Fix initialization init_sentry call Api wise, the `sentry_dsn` is expected to be passed as first parameter. Which in the scheduler's case is not set yet. Forcing it to None for now. | 23 January 2020, 13:21:21 UTC |
2a0e4c1 | Jenkins for Software Heritage | 23 January 2020, 12:49:38 UTC | Updated backport on buster-swh from debian/0.0.70-1_swh1 (unstable-swh) | 23 January 2020, 12:49:38 UTC |
a43967c | Jenkins for Software Heritage | 23 January 2020, 12:49:38 UTC | Merge tag 'debian/0.0.70-1_swh1' into debian/buster-swh | 23 January 2020, 12:49:38 UTC |
b1ab20f | Jenkins for Software Heritage | 23 January 2020, 12:47:43 UTC | Updated debian changelog for version 0.0.70 | 23 January 2020, 12:47:43 UTC |
6817890 | Jenkins for Software Heritage | 23 January 2020, 12:47:42 UTC | New upstream version 0.0.70 | 23 January 2020, 12:47:42 UTC |
757092a | Jenkins for Software Heritage | 23 January 2020, 12:47:42 UTC | Update upstream source from tag 'debian/upstream/0.0.70' Update to upstream version '0.0.70' with Debian dir 68b937656bf2a5b3699c43a35020d652a0beb744 | 23 January 2020, 12:47:42 UTC |
0712207 | Valentin Lorentz | 10 January 2020, 14:13:07 UTC | Use swh.core.sentry instead of calling sentry_sdk.init directly. This adds support for SWH_MAIN_PACKAGE to initialize sentry_sdk with a release. | 10 January 2020, 14:13:07 UTC |
b488d69 | Antoine R. Dumont (@ardumont) | 17 December 2019, 22:23:35 UTC | backend_es: Fix configuration mapping | 17 December 2019, 22:23:35 UTC |
558515f | Jenkins for Software Heritage | 17 December 2019, 15:06:40 UTC | Updated backport on buster-swh from debian/0.0.69-1_swh1 (unstable-swh) | 17 December 2019, 15:06:40 UTC |
4c6883b | Jenkins for Software Heritage | 17 December 2019, 15:06:39 UTC | Merge tag 'debian/0.0.69-1_swh1' into debian/buster-swh swh-scheduler Debian release 0.0.69-1~swh1 # Conflicts: # debian/changelog | 17 December 2019, 15:06:39 UTC |
72f91d4 | Jenkins for Software Heritage | 17 December 2019, 15:04:49 UTC | Updated debian changelog for version 0.0.69 | 17 December 2019, 15:04:49 UTC |
c6b2162 | Jenkins for Software Heritage | 17 December 2019, 15:04:48 UTC | Update upstream source from tag 'debian/upstream/0.0.69' Update to upstream version '0.0.69' with Debian dir 5678b3095482c6b7bc27075529d8602027586837 | 17 December 2019, 15:04:48 UTC |
9896f0f | Jenkins for Software Heritage | 17 December 2019, 15:04:47 UTC | New upstream version 0.0.69 | 17 December 2019, 15:04:47 UTC |
cc2de16 | Antoine R. Dumont (@ardumont) | 17 December 2019, 14:57:33 UTC | tests: Try to avoid fixture redefinition Somehow, that messes other tests in the debian build. | 17 December 2019, 14:57:33 UTC |
1eeac4c | Jenkins for Software Heritage | 17 December 2019, 14:33:33 UTC | Updated debian changelog for version 0.0.68 | 17 December 2019, 14:33:33 UTC |
4fa31c7 | Jenkins for Software Heritage | 17 December 2019, 14:33:32 UTC | Update upstream source from tag 'debian/upstream/0.0.68' Update to upstream version '0.0.68' with Debian dir 274be693d455a6978cebb7c051d89bbc0d3ff145 | 17 December 2019, 14:33:32 UTC |
a901970 | Jenkins for Software Heritage | 17 December 2019, 14:33:32 UTC | New upstream version 0.0.68 | 17 December 2019, 14:33:32 UTC |
73ade78 | Antoine R. Dumont (@ardumont) | 17 December 2019, 14:27:15 UTC | tests: Avoid fixture clash in different purposes fixture Somehow, that fails in the debian build | 17 December 2019, 14:27:50 UTC |
e096318 | Jenkins for Software Heritage | 17 December 2019, 13:38:03 UTC | Updated debian changelog for version 0.0.67 | 17 December 2019, 13:38:03 UTC |
1d57187 | Jenkins for Software Heritage | 17 December 2019, 13:38:02 UTC | Update upstream source from tag 'debian/upstream/0.0.67' Update to upstream version '0.0.67' with Debian dir 59ecb35827297ab96861353827234f0b63f4d31b | 17 December 2019, 13:38:02 UTC |
652b583 | Jenkins for Software Heritage | 17 December 2019, 13:38:02 UTC | New upstream version 0.0.67 | 17 December 2019, 13:38:02 UTC |
e9d8a5f | Antoine R. Dumont (@ardumont) | 17 December 2019, 12:28:42 UTC | scheduler.backend: Rename appropriately module elasticsearch_memory | 17 December 2019, 12:33:43 UTC |
2cbfb78 | Antoine R. Dumont (@ardumont) | 17 December 2019, 11:51:28 UTC | Add tests to in memory elasticsearch implementation | 17 December 2019, 12:33:43 UTC |
ba5920d | Antoine R. Dumont (@ardumont) | 17 December 2019, 11:51:13 UTC | backend_es: Add tests around elasticsearch client instantiation | 17 December 2019, 12:33:43 UTC |
38d17de | Antoine R. Dumont (@ardumont) | 17 December 2019, 11:50:13 UTC | tests/common: Remove uneeded behavior | 17 December 2019, 12:33:43 UTC |
ac32b5e | Antoine R. Dumont (@ardumont) | 17 December 2019, 09:59:19 UTC | backend: Add alternate memory elasticsearch implem to allow testing | 17 December 2019, 12:33:43 UTC |
7b1c2d5 | Antoine R. Dumont (@ardumont) | 17 December 2019, 09:57:31 UTC | scheduler.backend_es: Allow using different elasticsearch clients For the moment, only 1 official es client exists | 17 December 2019, 12:33:43 UTC |
ec207fb | Antoine R. Dumont (@ardumont) | 17 December 2019, 09:51:20 UTC | scheduler.backend: Make the returned result a dict | 17 December 2019, 12:33:42 UTC |
f97bff6 | Antoine R. Dumont (@ardumont) | 17 December 2019, 09:50:27 UTC | cli.task: Make page_token actually a string even from the cli That actually make it consistent with the api | 17 December 2019, 12:33:42 UTC |
d8859d7 | Antoine R. Dumont (@ardumont) | 16 December 2019, 16:15:42 UTC | backend_es: Add initialization endpoint | 17 December 2019, 12:33:42 UTC |
d5cea20 | Antoine R. Dumont (@ardumont) | 16 December 2019, 16:15:24 UTC | backend_es: Remove unused endpoint | 17 December 2019, 12:33:42 UTC |
18df124 | Antoine R. Dumont (@ardumont) | 16 December 2019, 16:14:54 UTC | cli.tasks: Unify logging instruction | 17 December 2019, 12:33:42 UTC |
c5e189b | Antoine R. Dumont (@ardumont) | 16 December 2019, 16:14:08 UTC | test: Allow status definition during task template generation | 17 December 2019, 12:33:42 UTC |
844f3e0 | Antoine R. Dumont (@ardumont) | 16 December 2019, 10:07:27 UTC | tests.scheduler: Extract common utility function and test it | 17 December 2019, 12:33:42 UTC |
2d56669 | Antoine R. Dumont (@ardumont) | 16 December 2019, 09:07:01 UTC | scheduler.cli.task: Rename appropriately backend variable | 17 December 2019, 12:33:42 UTC |
793c233 | Antoine R. Dumont (@ardumont) | 16 December 2019, 09:06:10 UTC | scheduler.backend_es: Rename backend class appropriately | 17 December 2019, 12:33:42 UTC |
d5bf6b1 | Antoine R. Dumont (@ardumont) | 14 December 2019, 17:44:57 UTC | cli.task: Rename internal method appropriately | 17 December 2019, 12:33:42 UTC |
eb1c3d3 | Antoine R. Dumont (@ardumont) | 14 December 2019, 17:43:21 UTC | backend_es: Use consistent logging instruction | 17 December 2019, 12:33:42 UTC |
b376eb9 | Antoine R. Dumont (@ardumont) | 14 December 2019, 17:42:14 UTC | backend_es: Enclose close instruction within finally | 17 December 2019, 12:33:42 UTC |
f6726e9 | Antoine R. Dumont (@ardumont) | 14 December 2019, 10:10:49 UTC | backend_es: Create index when it does not exist | 17 December 2019, 12:33:41 UTC |
ad54c6b | Antoine R. Dumont (@ardumont) | 14 December 2019, 09:49:59 UTC | backend_es: Open indices prior to indexing method calls | 17 December 2019, 12:33:41 UTC |