| af37898 | Valentin Lorentz | 21 January 2021, 11:04:55 UTC | Run Black. It wasn't ran on d464b4cc1f9ae6a5c5c94a534826eff5cc27f12f. | 21 January 2021, 11:04:55 UTC |
| b641ac8 | Nicolas Dandrimont | 20 January 2021, 16:17:17 UTC | Make the grab_next_visits sql query modular This will allow us to easily plug new scheduling policies in that function. | 21 January 2021, 10:32:33 UTC |
| 9fb0dd6 | Antoine R. Dumont (@ardumont) | 19 January 2021, 17:45:36 UTC | journal_client: Read visit_stats entries by batch out of the loop Related to T2967 | 21 January 2021, 09:53:48 UTC |
| d464b4c | Antoine R. Dumont (@ardumont) | 19 January 2021, 17:12:33 UTC | scheduler: Make origin_visit_stats_get read multiple entries Related to T2967 | 21 January 2021, 09:53:46 UTC |
| ffe2aed | David Douard | 20 January 2021, 11:32:40 UTC | Simplify journal client tests - sort visits by default (there is a test dedicated to dealing with unsorted messagaes from the journal), - remove "intermediate checks" in several tests: these do not help much but make the code more difficult to read and maintain, - rename VISIT_STATUSES1 as VISIT_STATUSES_1 to make less prone to being confused with VISIT_STATUSES (which also exists). | 20 January 2021, 17:02:57 UTC |
| c7b740c | David Douard | 20 January 2021, 16:39:37 UTC | Revert "Make sure swh.scheduler.cli.journal is loaded in test_cli_journal.py" This reverts commit b03d978241a67e741e0f62696a0bbca17d768271. It's actually not needed, after all... | 20 January 2021, 17:01:51 UTC |
| 898820f | Nicolas Dandrimont | 20 January 2021, 11:11:05 UTC | simulator: collect and plot scheduler metrics over time For now, only plot the known_origins and origins_never_visited metrics. | 20 January 2021, 16:37:44 UTC |
| 9ce68f8 | Valentin Lorentz | 19 January 2021, 17:36:53 UTC | simulator: stop using get_scheduler directly This reuses the scheduler instantiated by the cli instead of hardcoding our own using the PG* variables. | 20 January 2021, 16:37:44 UTC |
| 88e0b42 | Valentin Lorentz | 19 January 2021, 15:32:27 UTC | simulator: Add documentation. | 20 January 2021, 16:37:44 UTC |
| 62c6d90 | Valentin Lorentz | 19 January 2021, 15:17:24 UTC | simulator: Make min_batch_size a parameter defined in the setup. | 20 January 2021, 16:37:44 UTC |
| 9468bb9 | Nicolas Dandrimont | 18 January 2021, 12:51:35 UTC | simulator: add basic tests for fill_test_data and run | 20 January 2021, 16:37:44 UTC |
| ead7b34 | Nicolas Dandrimont | 15 January 2021, 15:33:43 UTC | simulator: implement a simulator for the "old" task-based scheduler We extend the Task object with an autogenerated uuid allowing us to track the task lifetime between its creation and the generation of visit statuses, as the task-based scheduler does. | 20 January 2021, 16:37:44 UTC |
| aecd27e | Nicolas Dandrimont | 15 January 2021, 15:31:42 UTC | Move the simulator cli to the main cli module | 20 January 2021, 16:37:44 UTC |
| 05067e3 | Nicolas Dandrimont | 15 January 2021, 14:37:59 UTC | simulator: Replace attrs with dataclasses for consistency | 20 January 2021, 16:37:44 UTC |
| 24922fe | Nicolas Dandrimont | 15 January 2021, 14:31:41 UTC | simulator: wrap tasks and task events in typechecked objects This allows us to extend these objects without redefining a bunch of type annotations. | 20 January 2021, 16:37:44 UTC |
| d5318ae | Nicolas Dandrimont | 15 January 2021, 13:47:33 UTC | simulator: also fill data for the task-based scheduler | 20 January 2021, 16:37:44 UTC |
| 22ebb7a | Valentin Lorentz | 15 January 2021, 13:41:05 UTC | simulator: Split into smaller files in the same package | 20 January 2021, 16:37:44 UTC |
| ad7bfbe | Nicolas Dandrimont | 15 January 2021, 11:50:00 UTC | simulator: Make the run time a CLI argument | 20 January 2021, 16:37:44 UTC |
| df34db0 | Nicolas Dandrimont | 15 January 2021, 11:40:16 UTC | simulator: tweak simulation environment constants | 20 January 2021, 16:37:44 UTC |
| 21ce2c8 | Nicolas Dandrimont | 15 January 2021, 11:37:00 UTC | simulator: generate more origins in fill_data | 20 January 2021, 16:37:44 UTC |
| 2920419 | Nicolas Dandrimont | 15 January 2021, 11:35:01 UTC | simulator: add typing for Environment.scheduler | 20 January 2021, 16:37:44 UTC |
| 6433266 | Nicolas Dandrimont | 15 January 2021, 11:00:21 UTC | simulator: add support for a basic SimulationReport For now, this collects the runtime of tasks that have run, and gets printed at the end of the simulation. | 20 January 2021, 16:37:44 UTC |
| c474a82 | Nicolas Dandrimont | 15 January 2021, 10:45:23 UTC | simulator: refine origin model to follow an exponential distribution This models origins using a consistent characteristic "time between commits" that follows an exponential distribution between 1 second and 10 years. From this characteristic time, and feedback from the OriginVisitStats, we can generate the expected run time and output status of the next visit of that origin. | 20 January 2021, 16:37:44 UTC |
| 2459bad | Nicolas Dandrimont | 15 January 2021, 10:43:20 UTC | simulator: Remove some debug statements and lower log level | 20 January 2021, 16:37:44 UTC |
| cb12449 | Valentin Lorentz | 14 January 2021, 14:17:11 UTC | simulator: simulate the scheduler journal client | 20 January 2021, 16:37:44 UTC |
| 20b7f9c | Valentin Lorentz | 14 January 2021, 14:12:38 UTC | simulator: generate OriginVisitStatus objects in modeled visits To be able to generate uneventful visits, we would need to store the last snapshot seen for a given origin. Instead of storing this within the simulator, which would be a concern for large scale simulations, we use the scheduler visit cache directly. | 20 January 2021, 16:37:44 UTC |
| 39ad47d | Valentin Lorentz | 14 January 2021, 14:09:58 UTC | simulator: Move scheduler into the simulation environment object The scheduler is used by a lot of the simulated actors, it makes sense to share it all the time. | 20 January 2021, 16:37:44 UTC |
| 31967fa | Valentin Lorentz | 14 January 2021, 14:07:56 UTC | simulator: Use datetimes instead of a floating point simulated time | 20 January 2021, 16:37:44 UTC |
| fc3f06b | Nicolas Dandrimont | 13 January 2021, 15:13:01 UTC | Introduce scaffolding for a scheduler simulator This simulator will allow us to compare the behavior of the old and new schedulers, as well as to test the impact of scheduler policies and their parameters on the performance of the Software Heritage archival infrastructure as a whole. | 20 January 2021, 16:37:44 UTC |
| 7905a6b | Valentin Lorentz | 19 January 2021, 17:39:21 UTC | Add a cli for the scheduler metrics update endpoint | 20 January 2021, 16:35:05 UTC |
| c386fdf | David Douard | 20 January 2021, 11:24:10 UTC | Make the max_date() helper function accept *dates as argument so it can be called with more than 2 dates. | 20 January 2021, 11:28:02 UTC |
| b03d978 | David Douard | 20 January 2021, 11:14:23 UTC | Make sure swh.scheduler.cli.journal is loaded in test_cli_journal.py needed to make pytest able to run directly (without tox). | 20 January 2021, 11:18:25 UTC |
| 737d12e | Nicolas Dandrimont | 19 January 2021, 16:48:31 UTC | Introduce a new lister_get endpoint | 20 January 2021, 10:02:21 UTC |
| 114ed95 | Nicolas Dandrimont | 19 January 2021, 13:23:32 UTC | Implement some basic aggregated metrics on listed origins Metrics are computed and cached database-side by the `update_metrics` function. The `get_metrics` function only retrieves the cached data. The metrics are aggregated for each lister instance and visit type (allowing complete reaggregation by visit type for cross-cutting statistics). The following metrics have been implemented: - number of known origins overall - number of enabled origins (origins seen in the last listing) - number of enabled origins that have never been successfully visited - number of enabled origins with known activity since our last successful visit | 20 January 2021, 09:54:27 UTC |
| 9852653 | Nicolas Dandrimont | 19 January 2021, 16:56:44 UTC | Import the journal subcommand in the main swh.scheduler cli This issue was masked by tox.ini using pytest with --doctest-modules, which imports all modules during test collection, and therefore executing the side-effects of swh.scheduler.cli.journal. | 20 January 2021, 09:35:09 UTC |
| f8627a9 | David Douard | 19 January 2021, 13:45:09 UTC | Move the `last_scheduled` ts from ListedOrigin to OriginVisitStatus this timestamp being actually a loading-related value, it makes more sense to keep it in the OriginVisitStatus table. Related to T2444. | 19 January 2021, 16:48:51 UTC |
| 0a32a31 | David Douard | 19 January 2021, 15:16:30 UTC | Make the journal-client cli subcommand automagically loaded otherwise it won't be advertized as a `swh scehduler` subcommand by default. Also add a short dosctring for better --help. | 19 January 2021, 15:18:49 UTC |
| 5e609d5 | Antoine R. Dumont (@ardumont) | 19 January 2021, 11:04:37 UTC | requirements: Make swh.journal and optional dependency This avoids pulling journal dependencies when modules only needs the swh-scheduler dependency. | 19 January 2021, 11:04:37 UTC |
| 9395aa0 | Antoine R. Dumont (@ardumont) | 18 January 2021, 13:46:57 UTC | scheduler.cli.journal: Add `swh scheduler journal-client` cli This adds the cli entrypoint to actually process origin_visit_status topics and write to the origin_visit_stats db table. Related to T2967 | 19 January 2021, 10:10:41 UTC |
| 58ca796 | Antoine R. Dumont (@ardumont) | 15 January 2021, 14:49:41 UTC | journal_client: Improve stats detection This adds an integration test which permutes input to ensure out of order renders the same result. This also improves the current algorithm which revealed some hit-and-miss cases: - Initialization of the first visit detection (through the "last_snapshot" absence field, the previous implementation check could fail otherwise). - out of order policy (ignore old event) in case of supposedly "eventful" event was done too early which ignored too much messages (those new test cases failed in some permutations). This is now specifically checked in case of referenced snapshots which led to cases of possibly changing eventful event into uneventful one. For example, the case of an anterior eventful event is caught which means that the current most-up-to-date eventful event is actually an uneventful one). ... Related to T2967 | 19 January 2021, 09:17:05 UTC |
| d3afd14 | Nicolas Dandrimont | 15 January 2021, 14:10:44 UTC | Use the recorded task end time for the task scheduler feedback loop This allows us to run "time-warping" simulations without interference from the real wall clock time. | 15 January 2021, 16:04:30 UTC |
| a5fb291 | Antoine R. Dumont (@ardumont) | 14 January 2021, 17:38:05 UTC | backend: Make origin_visit_stats_upsert a batch api Related to T2967 | 15 January 2021, 13:34:06 UTC |
| 608aa20 | Antoine R. Dumont (@ardumont) | 13 January 2021, 12:03:28 UTC | Populate origin_visit_stats table out of the origin_visit_status topic The snapshot is used to determine the "eventful/uneventful" nature of the origin visit status. When no snapshot is provided, the visit is considered as failed so the last_failed column is updated. As there is no time guarantee when reading message from the topic, the code tries to keep the data in the most timely ordered as possible. Only most recent information is kept. Related to T2967 | 15 January 2021, 13:34:05 UTC |
| ca45d40 | Nicolas Dandrimont | 13 January 2021, 14:31:55 UTC | Filter origins by visit type when scheduling the next visits We have separate task queues and workers for each visit type, so it makes sense to split this endpoint along these lines too, at least for now. | 14 January 2021, 12:53:31 UTC |
| 59b4cb3 | Nicolas Dandrimont | 13 January 2021, 14:25:56 UTC | Reorganize ListedOrigin fixtures to generate multiple visit_types | 14 January 2021, 12:53:31 UTC |
| 4f5338f | Nicolas Dandrimont | 12 January 2021, 16:10:39 UTC | Introduce a `swh scheduler origin schedule-next` cli This creates one-shot tasks in the classic scheduler for the next visits to run according to the visit scheduling policy. | 14 January 2021, 12:53:31 UTC |
| 3dd1d5f | Nicolas Dandrimont | 12 January 2021, 16:28:33 UTC | Rename test task types to names that match real tasks The success of tests using these task types would depend on the test run order, because these task types are (currently) being created by swh/scheduler/sql/50-data.sql, but the table is truncated after the first test completes. | 14 January 2021, 12:53:31 UTC |
| 5d7b002 | Nicolas Dandrimont | 12 January 2021, 15:16:31 UTC | Introduce a `swh scheduler origin grab-next` cli This returns, as CSV, the next origins to be visited according to the passed scheduling policy. | 14 January 2021, 12:53:31 UTC |
| a620033 | Vincent SELLIER | 12 January 2021, 11:13:07 UTC | Add an new origin visit info model object and related backend api Upsert and Read methods Related to T2443 | 12 January 2021, 13:47:49 UTC |
| b13cb1f | Nicolas Dandrimont | 11 January 2021, 14:29:09 UTC | Implement a basic endpoint for getting the next origins to visit The basic policy implemented is a FIFO, to get things going. | 11 January 2021, 14:40:17 UTC |
| 619100e | David Douard | 18 December 2020, 14:57:00 UTC | Add a cli section to the doc | 18 December 2020, 14:57:00 UTC |
| ebff12b | Antoine R. Dumont (@ardumont) | 07 December 2020, 08:10:34 UTC | requirement: Adapt celery requirements This adapts the celery requirements to the last known where our builds are fine. Currently, 5.0.3 got released and this ends up making all the swh modules relying on tasks timeout. A bug upstream is opened [1]. In the mean time, this workaround fixes [2] and most probably the remaining swh builds. [1] https://github.com/celery/celery/issues/6521 [2] https://jenkins.softwareheritage.org/job/DSCH/job/tests/1132/console | 07 December 2020, 08:28:00 UTC |
| 3c87075 | David Douard | 02 December 2020, 08:49:33 UTC | Replace usage of arrow datetime objects in favor of pure datetime ones Note that the humanize library is now used in the cli pretty printing function (in place of the arrow hiumanize feature). As a result, displayed output from some cli commands may slightly differ. Closes T2835. | 03 December 2020, 09:25:36 UTC |
| 1b390a7 | Antoine R. Dumont (@ardumont) | 25 November 2020, 14:33:29 UTC | Stop using the deprecated configuration scheme | 25 November 2020, 14:56:18 UTC |
| 1f68031 | Antoine R. Dumont (@ardumont) | 25 November 2020, 14:30:10 UTC | cli.task_type: All task_type clis without a scheduler should raise As the code will plainly fail on calling methods on None instance if not caught early. | 25 November 2020, 14:56:18 UTC |
| 9e5b17f | Antoine R. Dumont (@ardumont) | 24 November 2020, 13:03:17 UTC | conftest: Reference swh.core.db.pytest_plugin As it's exposed through the swh.scheduler.pytest_plugin itself used by other swh modules, this needs to be declared to avoid other swh module build failures. Related to T2746 | 24 November 2020, 13:10:31 UTC |
| 49ed819 | Antoine R. Dumont (@ardumont) | 20 November 2020, 16:17:14 UTC | requirements-test.txt: Drop no longer needed pytest-postgresql requirement requirements-swh.txt already declares the swh.core[db] dependency which transitively pulls it. Related to T2746 | 23 November 2020, 12:11:02 UTC |
| 2f9e8ec | Antoine R. Dumont (@ardumont) | 21 October 2020, 14:52:45 UTC | scheduler.pytest_plugin: Make scheduler tests faster Reuse the swh.core.db.pytest_plugin factory | 22 October 2020, 10:09:06 UTC |
| 6a4455c | Antoine R. Dumont (@ardumont) | 19 October 2020, 07:25:04 UTC | pytest_plugin: Explicitely name the scheduler test db differently When using tests on modules with different lower level modules (e.g storage, scheduler, ...) this avoids clashes. | 19 October 2020, 07:25:04 UTC |
| 13dcadd | Antoine R. Dumont (@ardumont) | 16 October 2020, 11:12:27 UTC | scheduler: Type and unify get_scheduler factory with other factories Related to T1410 | 16 October 2020, 16:24:03 UTC |
| dd33cdc | Antoine R. Dumont (@ardumont) | 16 October 2020, 11:35:54 UTC | test_server: Simplify exception manipulations | 16 October 2020, 11:43:54 UTC |
| 315a2c9 | Stefano Zacchiroli | 02 October 2020, 14:24:01 UTC | tox.ini: pin black to the pre-commit version (19.10b0) to avoid flip-flops | 02 October 2020, 14:24:01 UTC |
| b7e5358 | Nicolas Dandrimont | 25 September 2020, 15:19:10 UTC | Drop vcversioner from requirements We stopped using it months ago. | 25 September 2020, 15:19:17 UTC |
| 4951a23 | Nicolas Dandrimont | 25 September 2020, 12:19:21 UTC | Run isort after the CLI import changes | 25 September 2020, 12:19:21 UTC |
| be7a5ae | David Douard | 22 September 2020, 08:39:15 UTC | Rename sql files according to swh.core 0.3 | 25 September 2020, 07:53:53 UTC |
| 5cc573d | David Douard | 22 September 2020, 08:36:11 UTC | Adapt cli declaration entrypoint to swh.core 0.3 | 25 September 2020, 07:48:38 UTC |
| 99e5af8 | Nicolas Dandrimont | 24 September 2020, 15:44:00 UTC | Move from kombu.five.monotonic to time.monotonic Looks like kombu finally axed python2 support. | 24 September 2020, 15:44:00 UTC |
| 7b0d48f | Antoine Lambert | 17 September 2020, 16:01:30 UTC | python: Reorder imports with isort Related to T2610 | 17 September 2020, 16:03:39 UTC |
| 8d8b58f | Antoine Lambert | 17 September 2020, 15:26:12 UTC | pre-commit: Add isort hook and configuration Related to T2610 | 17 September 2020, 16:03:39 UTC |
| 4bec5c8 | Antoine Lambert | 17 September 2020, 16:03:24 UTC | pre-commit: Update flake8 hook configuration flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks so now use the one from https://gitlab.com/pycqa/flake8 | 17 September 2020, 16:03:39 UTC |
| f5c8154 | David Douard | 10 September 2020, 09:25:08 UTC | cli: speedup the `swh` cli command startup time by moving import statements in functions and using conditional import of typechecking modules (especially StorageInterface which triggers the loading of 300+ modules). Related to T2575. | 10 September 2020, 15:46:08 UTC |
| b24be0c | Valentin Lorentz | 25 August 2020, 08:41:38 UTC | Tell pytest not to recurse in dotdirs. pytest wastes a lot of time in .hypothesis and .git; this commit excludes them. | 25 August 2020, 08:41:38 UTC |
| 6426208 | Antoine R. Dumont (@ardumont) | 01 August 2020, 08:03:23 UTC | cli.task: Migrate scheduler cli to latest storage change on iter_origins Related to T645 | 03 August 2020, 10:18:23 UTC |
| 849d063 | Antoine R. Dumont (@ardumont) | 24 July 2020, 08:16:09 UTC | test_cli: Adapt tests data and drop unsupported "validate" proxy | 24 July 2020, 08:22:07 UTC |
| 9f52d95 | Antoine R. Dumont (@ardumont) | 21 July 2020, 08:34:34 UTC | cli.task: Fix iter_origin returned types Related to T2494 | 21 July 2020, 08:36:03 UTC |
| 254e24a | Antoine R. Dumont (@ardumont) | 10 July 2020, 10:11:23 UTC | Do no expose pytest-plugin through setuptools, let modules require it when needed Defining the pytest-plugin though the pytest-plugin [1] makes it loaded by default. This creates loading issues on modules depending on scheduler but not on the pytest plugin scheduler exposes as explained in the doc [2] Instead we'll explicitely define to modules depending on the pytest plugins in their root conftest [3]: pytest_plugins = [ "swh.scheduler.pytest_plugin" ] [1] https://docs.pytest.org/en/stable/writing_plugins.html#setuptools-entry-points [2] https://docs.pytest.org/en/stable/writing_plugins.html#plugin-discovery-order-at-tool-startup [3] https://docs.pytest.org/en/stable/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file Related to D3475 Related to T2484 | 10 July 2020, 10:27:42 UTC |
| ece598c | Antoine Lambert | 08 July 2020, 16:12:15 UTC | requirements.txt: Remove future dependency This was needed for celery 4.4.4 but that version is not used anymore. | 08 July 2020, 16:33:25 UTC |
| 7009c3b | Nicolas Dandrimont | 08 July 2020, 15:55:07 UTC | Move all celery-related fixtures to the swh.scheduler pytest plugin This allows us to reuse these fixtures in other modules without brittle swh.scheduler.tests.conftest star imports. Unfortunately, we can't really override pytest fixtures from one plugin to another. We therefore reimplement the fixtures provided by celery, inlining the static configuration and renaming them to our names in the process. This also adds a backwards-compatibility import from pytest_plugin to conftest, to allow old users of the conftest fixtures to keep working. | 08 July 2020, 15:59:15 UTC |
| ce63e6a | Antoine R. Dumont (@ardumont) | 07 July 2020, 10:17:55 UTC | pytest.ini: Drop filterwarnings which never worked | 07 July 2020, 10:18:50 UTC |
| b2cbb9b | Nicolas Dandrimont | 06 July 2020, 12:51:41 UTC | Move shareable fixtures out of conftest into a dedicated pytest plugin This avoids having to run `from swh.scheduler.tests.conftest import *` in other modules, e.g. swh.lister, to import and use the swh_scheduler pytest fixture. | 06 July 2020, 14:42:04 UTC |
| 5b373ce | Nicolas Dandrimont | 06 July 2020, 07:49:44 UTC | Introduce a get_listed_origins endpoint This paginated endpoint allows retrieving information about the origins recorded by listers. | 06 July 2020, 09:51:10 UTC |
| aefc5c9 | Nicolas Dandrimont | 06 July 2020, 07:48:29 UTC | Don't recurse into attrs objects when serializing We need to use our serialization hook recursively to make sure that we can deserialize nested data structures. | 06 July 2020, 07:48:29 UTC |
| cc8fa7f | Nicolas Dandrimont | 22 June 2020, 10:46:09 UTC | Re-introduce the root endpoint for the rpc server | 22 June 2020, 10:55:11 UTC |
| 265bc8b | Nicolas Dandrimont | 22 June 2020, 08:58:09 UTC | The celery-monitor subcommand glob filtering needs celery >= 4.3 | 22 June 2020, 08:58:09 UTC |
| 8a1724a | Nicolas Dandrimont | 22 June 2020, 08:26:40 UTC | Add SQL for version 16 of the schema | 22 June 2020, 08:26:40 UTC |
| d107a55 | Nicolas Dandrimont | 16 June 2020, 08:25:08 UTC | Implement storage of listed origins This new API endpoint allows listers to record the origins they have seen during their current run. Origins are identified by the lister instance, the url of the origin, and the type of loader that should be used to load this origin. The implementation allows listers just send the list of origins they've seen (with some lightweight extra information), leaving the backend to handle whether to do an insertion or an update to an existing origin. The current implementation doesn't disable origins that have disappeared when doing a full listing run. This step will be done by a separate "origin garbage collection" endpoint, which will peruse the `last_seen` field. | 16 June 2020, 08:25:08 UTC |
| e0fa5c5 | Nicolas Dandrimont | 16 June 2020, 08:24:03 UTC | Move lister addition in scheduler tests to a pytest fixture This lets us keep the tests a little DRYer. | 16 June 2020, 08:24:03 UTC |
| 04894bd | Nicolas Dandrimont | 16 June 2020, 08:22:23 UTC | Lister.instance_name doesn't need a factory/default value | 16 June 2020, 08:22:23 UTC |
| f520108 | Nicolas Dandrimont | 16 June 2020, 08:08:59 UTC | Improve support of primary keys This splits primary keys across "automatic" primary keys (handled by the database) and manual primary keys (managed by the user). Use the opportunity to improve/clarify the documentation of field metadata attributes. | 16 June 2020, 08:22:12 UTC |
| 1c93e55 | Nicolas Dandrimont | 12 June 2020, 10:24:20 UTC | Implement basic storage and retrieval of lister information This adds a pair a functions to the backend: - `get_or_create_lister` pulls the record for a given lister from the database - `update_lister` updates the record for a given lister in the database This is one of the basic building blocks for the integration of lister information directly in the scheduler database. Related to T2442. | 15 June 2020, 13:41:02 UTC |
| 466ac59 | Nicolas Dandrimont | 15 June 2020, 12:46:28 UTC | Introduce a SchedulerException base class This allows us to automatically serialize/deserialize exceptions under this base class within our RPC framework. | 15 June 2020, 12:53:30 UTC |
| c509a12 | Nicolas Dandrimont | 12 June 2020, 09:03:26 UTC | Introduce some scaffolding for an attrs-based BaseSchedulerModel Alongside swh.model.model, this allows us to define data models for the objects the scheduler is working with, and to serialize/deserialize these objects transparently at the RPC layer. This also introduces some mild ORM-like logic so we can keep the actual SQL a little DRYer. | 15 June 2020, 10:49:25 UTC |
| 4c0c37b | Nicolas Dandrimont | 10 June 2020, 14:09:53 UTC | Use the automatic RPC client/server generation | 11 June 2020, 09:42:37 UTC |
| aedd323 | Nicolas Dandrimont | 10 June 2020, 09:31:45 UTC | Replace swh-worker-control with a swh scheduler celery-monitor subcommand This new subcommand has two commands: - ping: checks whether the given worker instance answers within a given timeout - list-running: lists running tasks on the given worker instance | 10 June 2020, 10:15:54 UTC |
| 8411335 | Nicolas Dandrimont | 10 June 2020, 09:30:31 UTC | Remove double logging setup in cli The logging module is already initialized by the main swh.core cli; This only creates double logging with no advantages whatsoever. | 10 June 2020, 09:30:31 UTC |
| 873cdac | Nicolas Dandrimont | 10 June 2020, 09:28:19 UTC | Handle psycopg2 OperationalError in cli initialization When running the cli with default settings (i.e. pointing to a softwareheritage-scheduler-dev database), and the database doesn't exist, an OperationalError is raised. This shouldn't prevent (some of the) cli subcommands from working, so catch this error and ignore it as one of the scheduler backend setup failure modes. | 10 June 2020, 09:28:19 UTC |
| 28c5b8d | Nicolas Dandrimont | 09 June 2020, 13:47:26 UTC | Replace vcversioner with setuptools-scm | 09 June 2020, 13:49:00 UTC |
| 14cd5bb | Nicolas Dandrimont | 03 June 2020, 15:17:50 UTC | Blacken for python3.7+ | 03 June 2020, 15:19:00 UTC |
| 6ac3d56 | Nicolas Dandrimont | 03 June 2020, 10:34:11 UTC | Drop use of pifpaf and the "db" pytest mark We've been using pytest-postgresql for... a year (4117d5a). | 03 June 2020, 10:34:11 UTC |
| 3f42423 | Nicolas Dandrimont | 03 June 2020, 09:29:58 UTC | Add future dependency, missing from celery 4.4.4 Without future, the tests involving celery hang indefinitely. Upstream issue: https://github.com/celery/celery/issues/6145 | 03 June 2020, 09:29:58 UTC |
