| 828d288 | Jenkins for Software Heritage | 20 April 2021, 09:51:00 UTC | Updated debian changelog for version 0.13.0 | 20 April 2021, 09:51:00 UTC |
| f2122c2 | Jenkins for Software Heritage | 20 April 2021, 09:50:59 UTC | Update upstream source from tag 'debian/upstream/0.13.0' Update to upstream version '0.13.0' with Debian dir 753ba2cc0646d7be73acfff0f55b36cc7fe12e65 | 20 April 2021, 09:50:59 UTC |
| 15684cb | Jenkins for Software Heritage | 20 April 2021, 09:50:59 UTC | New upstream version 0.13.0 | 20 April 2021, 09:50:59 UTC |
| befccb9 | Antoine R. Dumont (@ardumont) | 19 April 2021, 10:26:58 UTC | scheduler: Clean up priority/ratio task dead code Since [1], tasks with priority are routed to dedicated queues (see tasks for more details). The tasks with priority to be scheduled have their own dedicated endpoints to be called. [1] Related to T3084 Related to T3271 | 20 April 2021, 09:27:18 UTC |
| 4e06bcd | Valentin Lorentz | 20 April 2021, 08:59:53 UTC | Parse task_ids before calling set_status_tasks. So errors on the CLI side do not trigger an exception on the server | 20 April 2021, 09:19:52 UTC |
| 974c0c2 | Antoine R. Dumont (@ardumont) | 15 April 2021, 11:22:11 UTC | tests: Complete checks on message with priority consumption Related to T3084 | 15 April 2021, 12:57:25 UTC |
| 38ef60a | Jenkins for Software Heritage | 15 April 2021, 11:36:14 UTC | Updated debian changelog for version 0.12.0 | 15 April 2021, 11:36:14 UTC |
| 9db86c1 | Jenkins for Software Heritage | 15 April 2021, 11:36:13 UTC | Update upstream source from tag 'debian/upstream/0.12.0' Update to upstream version '0.12.0' with Debian dir 788879e9b7caf6d54d8f4645d720dc7b7859e069 | 15 April 2021, 11:36:13 UTC |
| a8b8fde | Jenkins for Software Heritage | 15 April 2021, 11:36:12 UTC | New upstream version 0.12.0 | 15 April 2021, 11:36:12 UTC |
| 17052c4 | Antoine R. Dumont (@ardumont) | 14 April 2021, 10:49:47 UTC | Route priority tasks to dedicated save code now queues This splits the calls to read tasks into 2 calls, one for tasks with no priority (standard), another call for tasks with priority. If any tasks with priority are detected, they are routed to dedicated `save_code_now:` prefixed named queues (per task type). Related to T3084 | 15 April 2021, 11:24:13 UTC |
| bfc1a87 | Valentin Lorentz | 13 April 2021, 19:51:42 UTC | Fix various Sphinx warnings | 15 April 2021, 08:19:50 UTC |
| 3bf2f9a | Jenkins for Software Heritage | 14 April 2021, 16:19:31 UTC | Updated debian changelog for version 0.11.0 | 14 April 2021, 16:19:31 UTC |
| 77ea590 | Jenkins for Software Heritage | 14 April 2021, 16:19:30 UTC | New upstream version 0.11.0 | 14 April 2021, 16:19:30 UTC |
| c32aff0 | Jenkins for Software Heritage | 14 April 2021, 16:19:30 UTC | Update upstream source from tag 'debian/upstream/0.11.0' Update to upstream version '0.11.0' with Debian dir aac6bb6258a43d962787d39b4006589985c90bcb | 14 April 2021, 16:19:30 UTC |
| 3e2ae3d | Antoine R. Dumont (@ardumont) | 13 April 2021, 15:16:37 UTC | backend: Open endpoints to peek/grab tasks with any priority The priority notion becomes a blur. Any tasks with a non null priority is considered for reading or grabbing. In a future commit, this should allow to make the runner evolve to reroute tasks with priority to other queues. Related to T3084 | 13 April 2021, 16:05:29 UTC |
| ecab745 | Nicolas Dandrimont | 11 February 2021, 18:16:19 UTC | Make origin_visit_stats_get return results from all pages psycopg2.extras.execute_values executes queries in batches of 100 by default. At the end of execute_values, only the last batch of results is available in the cursor; To fetch all results, one needs to set fetch=True instead of using the cursor. | 11 February 2021, 18:39:29 UTC |
| 86ada44 | Nicolas Dandrimont | 09 February 2021, 09:36:23 UTC | journal client: Filter out status messages without type This allows us to support reading the journal from the beginning, ignoring messages with the old schema. | 11 February 2021, 18:38:44 UTC |
| cdb1775 | Nicolas Dandrimont | 09 February 2021, 09:33:19 UTC | Simplify max_date() The built-in `max` function can take an iterable directly, no need to reimplement it. | 11 February 2021, 18:24:01 UTC |
| cf32e37 | Vincent SELLIER | 09 February 2021, 13:56:56 UTC | journal_client: Fix date computations for (un)eventful visits Fix a wrong computation when several messages (>=3) for the same snapshot are received in the wrong order For example, before the fix, the following occurs: ``` | date | snapshot | | last_ev | last_unev | Snap | | ---- | -------- | --- | -------- | --------- | ---- | | 2022 | S2 | | 2022 | | S2 | | 2020 | S2 | | 2020 | 2022 | S2 | | 2021 | S2 | | **2021** | **2020** | S2 | ``` as it should be: ``` | date | snapshot | | last_ev | last_unev | Snap | | ---- | -------- | --- | -------- | --------- | ---- | | 2022 | S2 | | 2022 | | S2 | | 2020 | S2 | | 2020 | 2022 | S2 | | 2021 | S2 | | **2020** | **2022** | S2 | ``` Related to T3000 | 09 February 2021, 17:10:46 UTC |
| aa507ac | Antoine R. Dumont (@ardumont) | 05 February 2021, 13:31:26 UTC | journal_client: Deal with failed status message As loader will start to create failed status message, deal with them if any. Related to T3030 | 05 February 2021, 14:06:48 UTC |
| a78d5b2 | Jenkins for Software Heritage | 03 February 2021, 22:10:13 UTC | Updated debian changelog for version 0.10.0 | 03 February 2021, 22:10:13 UTC |
| 2cf46e3 | Jenkins for Software Heritage | 03 February 2021, 22:10:12 UTC | New upstream version 0.10.0 | 03 February 2021, 22:10:12 UTC |
| 97ca32a | Jenkins for Software Heritage | 03 February 2021, 22:10:12 UTC | Update upstream source from tag 'debian/upstream/0.10.0' Update to upstream version '0.10.0' with Debian dir 17cb8a0e3def3b15efe5ce2e2ae36c621314e1f3 | 03 February 2021, 22:10:12 UTC |
| 14feab9 | Nicolas Dandrimont | 03 February 2021, 18:51:30 UTC | celery: acknowledge tasks as soon as they're received With late acknowledgements, RabbitMQ will re-send tasks to clients even if they can't ever complete the task (e.g. when the task gets killed because the machine is out of memory). This problem only increases over time, leading to complete starvation of the ingestion system. Now that we have multiple mechanisms to issue retries of tasks, we can use early acknowledgements for tasks instead, which should mitigate the ongoing starvation, at the expense of having to retry tasks externally. | 03 February 2021, 19:10:26 UTC |
| aaffff2 | David Douard | 22 January 2021, 11:17:00 UTC | Simulator: allow to export results in a csv file | 01 February 2021, 14:37:31 UTC |
| 9fce3f6 | David Douard | 01 February 2021, 14:36:16 UTC | Add minimal tests for the SimulationReport.format() method | 01 February 2021, 14:37:31 UTC |
| aaf7dd6 | David Douard | 22 January 2021, 11:15:47 UTC | Make plottings optional in simulator cli output | 29 January 2021, 15:00:36 UTC |
| cf0583b | Nicolas Dandrimont | 21 January 2021, 16:55:43 UTC | simulator: stop validating the scheduling policy in the CLI We already do that in the scheduler backend function | 26 January 2021, 12:33:16 UTC |
| ebb5847 | Nicolas Dandrimont | 21 January 2021, 16:55:16 UTC | Run simulator tests on all known scheduling policies | 26 January 2021, 12:33:05 UTC |
| 1f77521 | Nicolas Dandrimont | 21 January 2021, 16:48:38 UTC | simulator: record visit metrics alongside scheduler metrics This allows us to check the behavior of the archive over time in terms of number of visits. | 26 January 2021, 12:32:54 UTC |
| 8898394 | Nicolas Dandrimont | 21 January 2021, 16:45:23 UTC | simulator: stop using the database as a cache for origin data This was a significant bottleneck of the simulator. To work around this, we: - Generate snapshot ids consistently in the OriginModel - Cache the origin data locally in the simulator, to compute the eventfulness of visits - Cache the last visit time for all origins to compute the estimated run time of visit tasks. | 26 January 2021, 12:31:57 UTC |
| c92ead5 | Nicolas Dandrimont | 21 January 2021, 16:31:43 UTC | grab_next_visits: don't re-schedule visits too fast The earlier implementation would just schedule new visits for origins forever, regardless of whether they were already scheduled or not. | 26 January 2021, 12:20:39 UTC |
| 2b39cbc | Nicolas Dandrimont | 21 January 2021, 16:29:45 UTC | Allow overriding the timestamp of grab_next_visits This makes the simulator behavior more consistent with reality. | 26 January 2021, 12:20:39 UTC |
| 7ffbdd1 | Nicolas Dandrimont | 21 January 2021, 16:27:40 UTC | Construct grab_next_visits query arguments incrementally | 26 January 2021, 12:20:39 UTC |
| ea068b4 | Valentin Lorentz | 21 January 2021, 13:57:42 UTC | simulator: add simple lister simulation | 26 January 2021, 12:20:39 UTC |
| cfafc72 | Jenkins for Software Heritage | 25 January 2021, 15:31:22 UTC | Updated debian changelog for version 0.9.2 | 25 January 2021, 15:31:22 UTC |
| ac65074 | Jenkins for Software Heritage | 25 January 2021, 15:31:21 UTC | Update upstream source from tag 'debian/upstream/0.9.2' Update to upstream version '0.9.2' with Debian dir 0835a7d748607669851821974686036d543f5e79 | 25 January 2021, 15:31:21 UTC |
| db8fa8e | Jenkins for Software Heritage | 25 January 2021, 15:31:20 UTC | New upstream version 0.9.2 | 25 January 2021, 15:31:20 UTC |
| 7af98e2 | Valentin Lorentz | 21 January 2021, 13:54:53 UTC | Factor out ListedOrigin generation to use the OriginModel This generates consistent last_update values according to the model and simulated time. | 25 January 2021, 13:39:30 UTC |
| 2906b4e | Antoine Lambert | 22 January 2021, 10:24:49 UTC | model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any] Some loaders, for instance the debian one, can have non string arguments so change the extra_loader_arguments type of the ListedOrigin model to something more generic. Related to T2979 | 25 January 2021, 13:10:25 UTC |
| 3d13cda | Vincent SELLIER | 21 January 2021, 17:49:38 UTC | Solve uneventful/eventful with unordered messages with snapshots Fix the case: m1: date2/snapshot1 m2: date1/snaptshot1 which results to: last_eventful = date2 last_uneventful = date2 The upsert was always keeping the most recent date when the eventful/uneventful dates were switched Related to T2978 | 23 January 2021, 18:57:17 UTC |
| d528998 | Vincent SELLIER | 21 January 2021, 15:15:26 UTC | Do not consider duplicated messages as uneventful event Avoid to copy the eventful date to the uneventful date when a duplicated message (same date/same snapshot) is received, related to T2978 | 23 January 2021, 18:57:17 UTC |
| 86b2555 | David Douard | 21 January 2021, 10:30:21 UTC | Add a --num-origins option to the fill-test-data cli command | 22 January 2021, 13:10:59 UTC |
| abb513c | David Douard | 22 January 2021, 09:47:18 UTC | Simulation: log at info level recorded metrics this allows to follows what the simulation is doing. | 22 January 2021, 13:08:30 UTC |
| 835296a | Jenkins for Software Heritage | 21 January 2021, 18:28:00 UTC | Updated debian changelog for version 0.9.1 | 21 January 2021, 18:28:00 UTC |
| f266da3 | Jenkins for Software Heritage | 21 January 2021, 18:28:00 UTC | Update upstream source from tag 'debian/upstream/0.9.1' Update to upstream version '0.9.1' with Debian dir 83c90829de6abb48a74e477387ee087a4de998ee | 21 January 2021, 18:28:00 UTC |
| 70532dc | Jenkins for Software Heritage | 21 January 2021, 18:27:59 UTC | New upstream version 0.9.1 | 21 January 2021, 18:27:59 UTC |
| 82b7a8a | Vincent SELLIER | 21 January 2021, 17:49:38 UTC | Solve uneventful/eventful with unordered messages with snapshots Fix the case: m1: date2/snapshot1 m2: date1/snaptshot1 which results to: last_eventful = date2 last_uneventful = date2 The upsert was always keeping the most recent date when the eventful/uneventful dates were switched Related to T2978 | 21 January 2021, 18:15:05 UTC |
| 25d036e | Vincent SELLIER | 21 January 2021, 15:15:26 UTC | Do not consider duplicated messages as uneventful event Avoid to copy the eventful date to the uneventful date when a duplicated message (same date/same snapshot) is received, related to T2978 | 21 January 2021, 18:15:04 UTC |
| b93aa5b | Valentin Lorentz | 21 January 2021, 12:01:53 UTC | Make PaginatedListedOriginList a concretization of PagedResult 1. consistent with swh-storage and swh-indexer-storage 2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins. | 21 January 2021, 13:26:39 UTC |
| 58ec03f | Antoine R. Dumont (@ardumont) | 21 January 2021, 13:20:54 UTC | d/changelog: Bump new release Related to T2978 | 21 January 2021, 13:21:41 UTC |
| b0e941d | Antoine R. Dumont (@ardumont) | 21 January 2021, 11:17:09 UTC | d/control: Update dependencies This builds the debian package without the swh.scheduler.simulator module though. It is currently missing the plotille debian package. It will be dealt with later. Related to T2978 | 21 January 2021, 13:17:25 UTC |
| 0346020 | Nicolas Dandrimont | 20 January 2021, 16:23:03 UTC | Reorganize grab_next_visits tests to better check sorting behavior - factor out test setup and results checking - properly exercize corner cases of the oldest_scheduled_first policy | 21 January 2021, 12:02:39 UTC |
| 2f47936 | Nicolas Dandrimont | 20 January 2021, 16:29:16 UTC | Add scheduling policy for already visited origins with known last update This policy schedules origins by decreasing order of "visit lag" (that is, origins with the most lag are scheduled first). | 21 January 2021, 12:02:39 UTC |
| acad712 | Nicolas Dandrimont | 20 January 2021, 16:25:46 UTC | Add scheduling policy for never visited origins This policy orders never visited origins by increasing date of last update (scheduling the "oldest" never visited origins first). | 21 January 2021, 12:02:39 UTC |
| af37898 | Valentin Lorentz | 21 January 2021, 11:04:55 UTC | Run Black. It wasn't ran on d464b4cc1f9ae6a5c5c94a534826eff5cc27f12f. | 21 January 2021, 11:04:55 UTC |
| d2c4725 | Jenkins for Software Heritage | 21 January 2021, 11:00:29 UTC | Updated debian changelog for version 0.9.0 | 21 January 2021, 11:00:29 UTC |
| 5a9c2e7 | Jenkins for Software Heritage | 21 January 2021, 11:00:29 UTC | Update upstream source from tag 'debian/upstream/0.9.0' Update to upstream version '0.9.0' with Debian dir 77a48e68dc2ef078b65dde6e263407b0ad75c59d | 21 January 2021, 11:00:29 UTC |
| 2fd414f | Jenkins for Software Heritage | 21 January 2021, 11:00:28 UTC | New upstream version 0.9.0 | 21 January 2021, 11:00:28 UTC |
| b641ac8 | Nicolas Dandrimont | 20 January 2021, 16:17:17 UTC | Make the grab_next_visits sql query modular This will allow us to easily plug new scheduling policies in that function. | 21 January 2021, 10:32:33 UTC |
| 9fb0dd6 | Antoine R. Dumont (@ardumont) | 19 January 2021, 17:45:36 UTC | journal_client: Read visit_stats entries by batch out of the loop Related to T2967 | 21 January 2021, 09:53:48 UTC |
| d464b4c | Antoine R. Dumont (@ardumont) | 19 January 2021, 17:12:33 UTC | scheduler: Make origin_visit_stats_get read multiple entries Related to T2967 | 21 January 2021, 09:53:46 UTC |
| ffe2aed | David Douard | 20 January 2021, 11:32:40 UTC | Simplify journal client tests - sort visits by default (there is a test dedicated to dealing with unsorted messagaes from the journal), - remove "intermediate checks" in several tests: these do not help much but make the code more difficult to read and maintain, - rename VISIT_STATUSES1 as VISIT_STATUSES_1 to make less prone to being confused with VISIT_STATUSES (which also exists). | 20 January 2021, 17:02:57 UTC |
| c7b740c | David Douard | 20 January 2021, 16:39:37 UTC | Revert "Make sure swh.scheduler.cli.journal is loaded in test_cli_journal.py" This reverts commit b03d978241a67e741e0f62696a0bbca17d768271. It's actually not needed, after all... | 20 January 2021, 17:01:51 UTC |
| 898820f | Nicolas Dandrimont | 20 January 2021, 11:11:05 UTC | simulator: collect and plot scheduler metrics over time For now, only plot the known_origins and origins_never_visited metrics. | 20 January 2021, 16:37:44 UTC |
| 9ce68f8 | Valentin Lorentz | 19 January 2021, 17:36:53 UTC | simulator: stop using get_scheduler directly This reuses the scheduler instantiated by the cli instead of hardcoding our own using the PG* variables. | 20 January 2021, 16:37:44 UTC |
| 88e0b42 | Valentin Lorentz | 19 January 2021, 15:32:27 UTC | simulator: Add documentation. | 20 January 2021, 16:37:44 UTC |
| 62c6d90 | Valentin Lorentz | 19 January 2021, 15:17:24 UTC | simulator: Make min_batch_size a parameter defined in the setup. | 20 January 2021, 16:37:44 UTC |
| 9468bb9 | Nicolas Dandrimont | 18 January 2021, 12:51:35 UTC | simulator: add basic tests for fill_test_data and run | 20 January 2021, 16:37:44 UTC |
| ead7b34 | Nicolas Dandrimont | 15 January 2021, 15:33:43 UTC | simulator: implement a simulator for the "old" task-based scheduler We extend the Task object with an autogenerated uuid allowing us to track the task lifetime between its creation and the generation of visit statuses, as the task-based scheduler does. | 20 January 2021, 16:37:44 UTC |
| aecd27e | Nicolas Dandrimont | 15 January 2021, 15:31:42 UTC | Move the simulator cli to the main cli module | 20 January 2021, 16:37:44 UTC |
| 05067e3 | Nicolas Dandrimont | 15 January 2021, 14:37:59 UTC | simulator: Replace attrs with dataclasses for consistency | 20 January 2021, 16:37:44 UTC |
| 24922fe | Nicolas Dandrimont | 15 January 2021, 14:31:41 UTC | simulator: wrap tasks and task events in typechecked objects This allows us to extend these objects without redefining a bunch of type annotations. | 20 January 2021, 16:37:44 UTC |
| d5318ae | Nicolas Dandrimont | 15 January 2021, 13:47:33 UTC | simulator: also fill data for the task-based scheduler | 20 January 2021, 16:37:44 UTC |
| 22ebb7a | Valentin Lorentz | 15 January 2021, 13:41:05 UTC | simulator: Split into smaller files in the same package | 20 January 2021, 16:37:44 UTC |
| ad7bfbe | Nicolas Dandrimont | 15 January 2021, 11:50:00 UTC | simulator: Make the run time a CLI argument | 20 January 2021, 16:37:44 UTC |
| df34db0 | Nicolas Dandrimont | 15 January 2021, 11:40:16 UTC | simulator: tweak simulation environment constants | 20 January 2021, 16:37:44 UTC |
| 21ce2c8 | Nicolas Dandrimont | 15 January 2021, 11:37:00 UTC | simulator: generate more origins in fill_data | 20 January 2021, 16:37:44 UTC |
| 2920419 | Nicolas Dandrimont | 15 January 2021, 11:35:01 UTC | simulator: add typing for Environment.scheduler | 20 January 2021, 16:37:44 UTC |
| 6433266 | Nicolas Dandrimont | 15 January 2021, 11:00:21 UTC | simulator: add support for a basic SimulationReport For now, this collects the runtime of tasks that have run, and gets printed at the end of the simulation. | 20 January 2021, 16:37:44 UTC |
| c474a82 | Nicolas Dandrimont | 15 January 2021, 10:45:23 UTC | simulator: refine origin model to follow an exponential distribution This models origins using a consistent characteristic "time between commits" that follows an exponential distribution between 1 second and 10 years. From this characteristic time, and feedback from the OriginVisitStats, we can generate the expected run time and output status of the next visit of that origin. | 20 January 2021, 16:37:44 UTC |
| 2459bad | Nicolas Dandrimont | 15 January 2021, 10:43:20 UTC | simulator: Remove some debug statements and lower log level | 20 January 2021, 16:37:44 UTC |
| cb12449 | Valentin Lorentz | 14 January 2021, 14:17:11 UTC | simulator: simulate the scheduler journal client | 20 January 2021, 16:37:44 UTC |
| 20b7f9c | Valentin Lorentz | 14 January 2021, 14:12:38 UTC | simulator: generate OriginVisitStatus objects in modeled visits To be able to generate uneventful visits, we would need to store the last snapshot seen for a given origin. Instead of storing this within the simulator, which would be a concern for large scale simulations, we use the scheduler visit cache directly. | 20 January 2021, 16:37:44 UTC |
| 39ad47d | Valentin Lorentz | 14 January 2021, 14:09:58 UTC | simulator: Move scheduler into the simulation environment object The scheduler is used by a lot of the simulated actors, it makes sense to share it all the time. | 20 January 2021, 16:37:44 UTC |
| 31967fa | Valentin Lorentz | 14 January 2021, 14:07:56 UTC | simulator: Use datetimes instead of a floating point simulated time | 20 January 2021, 16:37:44 UTC |
| fc3f06b | Nicolas Dandrimont | 13 January 2021, 15:13:01 UTC | Introduce scaffolding for a scheduler simulator This simulator will allow us to compare the behavior of the old and new schedulers, as well as to test the impact of scheduler policies and their parameters on the performance of the Software Heritage archival infrastructure as a whole. | 20 January 2021, 16:37:44 UTC |
| 7905a6b | Valentin Lorentz | 19 January 2021, 17:39:21 UTC | Add a cli for the scheduler metrics update endpoint | 20 January 2021, 16:35:05 UTC |
| c386fdf | David Douard | 20 January 2021, 11:24:10 UTC | Make the max_date() helper function accept *dates as argument so it can be called with more than 2 dates. | 20 January 2021, 11:28:02 UTC |
| b03d978 | David Douard | 20 January 2021, 11:14:23 UTC | Make sure swh.scheduler.cli.journal is loaded in test_cli_journal.py needed to make pytest able to run directly (without tox). | 20 January 2021, 11:18:25 UTC |
| 737d12e | Nicolas Dandrimont | 19 January 2021, 16:48:31 UTC | Introduce a new lister_get endpoint | 20 January 2021, 10:02:21 UTC |
| 114ed95 | Nicolas Dandrimont | 19 January 2021, 13:23:32 UTC | Implement some basic aggregated metrics on listed origins Metrics are computed and cached database-side by the `update_metrics` function. The `get_metrics` function only retrieves the cached data. The metrics are aggregated for each lister instance and visit type (allowing complete reaggregation by visit type for cross-cutting statistics). The following metrics have been implemented: - number of known origins overall - number of enabled origins (origins seen in the last listing) - number of enabled origins that have never been successfully visited - number of enabled origins with known activity since our last successful visit | 20 January 2021, 09:54:27 UTC |
| 9852653 | Nicolas Dandrimont | 19 January 2021, 16:56:44 UTC | Import the journal subcommand in the main swh.scheduler cli This issue was masked by tox.ini using pytest with --doctest-modules, which imports all modules during test collection, and therefore executing the side-effects of swh.scheduler.cli.journal. | 20 January 2021, 09:35:09 UTC |
| f8627a9 | David Douard | 19 January 2021, 13:45:09 UTC | Move the `last_scheduled` ts from ListedOrigin to OriginVisitStatus this timestamp being actually a loading-related value, it makes more sense to keep it in the OriginVisitStatus table. Related to T2444. | 19 January 2021, 16:48:51 UTC |
| 0a32a31 | David Douard | 19 January 2021, 15:16:30 UTC | Make the journal-client cli subcommand automagically loaded otherwise it won't be advertized as a `swh scehduler` subcommand by default. Also add a short dosctring for better --help. | 19 January 2021, 15:18:49 UTC |
| 5e609d5 | Antoine R. Dumont (@ardumont) | 19 January 2021, 11:04:37 UTC | requirements: Make swh.journal and optional dependency This avoids pulling journal dependencies when modules only needs the swh-scheduler dependency. | 19 January 2021, 11:04:37 UTC |
| 9395aa0 | Antoine R. Dumont (@ardumont) | 18 January 2021, 13:46:57 UTC | scheduler.cli.journal: Add `swh scheduler journal-client` cli This adds the cli entrypoint to actually process origin_visit_status topics and write to the origin_visit_stats db table. Related to T2967 | 19 January 2021, 10:10:41 UTC |
| 58ca796 | Antoine R. Dumont (@ardumont) | 15 January 2021, 14:49:41 UTC | journal_client: Improve stats detection This adds an integration test which permutes input to ensure out of order renders the same result. This also improves the current algorithm which revealed some hit-and-miss cases: - Initialization of the first visit detection (through the "last_snapshot" absence field, the previous implementation check could fail otherwise). - out of order policy (ignore old event) in case of supposedly "eventful" event was done too early which ignored too much messages (those new test cases failed in some permutations). This is now specifically checked in case of referenced snapshots which led to cases of possibly changing eventful event into uneventful one. For example, the case of an anterior eventful event is caught which means that the current most-up-to-date eventful event is actually an uneventful one). ... Related to T2967 | 19 January 2021, 09:17:05 UTC |
| d3afd14 | Nicolas Dandrimont | 15 January 2021, 14:10:44 UTC | Use the recorded task end time for the task scheduler feedback loop This allows us to run "time-warping" simulations without interference from the real wall clock time. | 15 January 2021, 16:04:30 UTC |
| a5fb291 | Antoine R. Dumont (@ardumont) | 14 January 2021, 17:38:05 UTC | backend: Make origin_visit_stats_upsert a batch api Related to T2967 | 15 January 2021, 13:34:06 UTC |
