https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
db8fa8e New upstream version 0.9.2 25 January 2021, 15:31:20 UTC
2906b4e model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any] Some loaders, for instance the debian one, can have non string arguments so change the extra_loader_arguments type of the ListedOrigin model to something more generic. Related to T2979 25 January 2021, 13:10:25 UTC
3d13cda Solve uneventful/eventful with unordered messages with snapshots Fix the case: m1: date2/snapshot1 m2: date1/snaptshot1 which results to: last_eventful = date2 last_uneventful = date2 The upsert was always keeping the most recent date when the eventful/uneventful dates were switched Related to T2978 23 January 2021, 18:57:17 UTC
d528998 Do not consider duplicated messages as uneventful event Avoid to copy the eventful date to the uneventful date when a duplicated message (same date/same snapshot) is received, related to T2978 23 January 2021, 18:57:17 UTC
86b2555 Add a --num-origins option to the fill-test-data cli command 22 January 2021, 13:10:59 UTC
abb513c Simulation: log at info level recorded metrics this allows to follows what the simulation is doing. 22 January 2021, 13:08:30 UTC
70532dc New upstream version 0.9.1 21 January 2021, 18:27:59 UTC
82b7a8a Solve uneventful/eventful with unordered messages with snapshots Fix the case: m1: date2/snapshot1 m2: date1/snaptshot1 which results to: last_eventful = date2 last_uneventful = date2 The upsert was always keeping the most recent date when the eventful/uneventful dates were switched Related to T2978 21 January 2021, 18:15:05 UTC
25d036e Do not consider duplicated messages as uneventful event Avoid to copy the eventful date to the uneventful date when a duplicated message (same date/same snapshot) is received, related to T2978 21 January 2021, 18:15:04 UTC
b93aa5b Make PaginatedListedOriginList a concretization of PagedResult 1. consistent with swh-storage and swh-indexer-storage 2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins. 21 January 2021, 13:26:39 UTC
0346020 Reorganize grab_next_visits tests to better check sorting behavior - factor out test setup and results checking - properly exercize corner cases of the oldest_scheduled_first policy 21 January 2021, 12:02:39 UTC
2f47936 Add scheduling policy for already visited origins with known last update This policy schedules origins by decreasing order of "visit lag" (that is, origins with the most lag are scheduled first). 21 January 2021, 12:02:39 UTC
acad712 Add scheduling policy for never visited origins This policy orders never visited origins by increasing date of last update (scheduling the "oldest" never visited origins first). 21 January 2021, 12:02:39 UTC
af37898 Run Black. It wasn't ran on d464b4cc1f9ae6a5c5c94a534826eff5cc27f12f. 21 January 2021, 11:04:55 UTC
2fd414f New upstream version 0.9.0 21 January 2021, 11:00:28 UTC
b641ac8 Make the grab_next_visits sql query modular This will allow us to easily plug new scheduling policies in that function. 21 January 2021, 10:32:33 UTC
9fb0dd6 journal_client: Read visit_stats entries by batch out of the loop Related to T2967 21 January 2021, 09:53:48 UTC
d464b4c scheduler: Make origin_visit_stats_get read multiple entries Related to T2967 21 January 2021, 09:53:46 UTC
ffe2aed Simplify journal client tests - sort visits by default (there is a test dedicated to dealing with unsorted messagaes from the journal), - remove "intermediate checks" in several tests: these do not help much but make the code more difficult to read and maintain, - rename VISIT_STATUSES1 as VISIT_STATUSES_1 to make less prone to being confused with VISIT_STATUSES (which also exists). 20 January 2021, 17:02:57 UTC
c7b740c Revert "Make sure swh.scheduler.cli.journal is loaded in test_cli_journal.py" This reverts commit b03d978241a67e741e0f62696a0bbca17d768271. It's actually not needed, after all... 20 January 2021, 17:01:51 UTC
898820f simulator: collect and plot scheduler metrics over time For now, only plot the known_origins and origins_never_visited metrics. 20 January 2021, 16:37:44 UTC
9ce68f8 simulator: stop using get_scheduler directly This reuses the scheduler instantiated by the cli instead of hardcoding our own using the PG* variables. 20 January 2021, 16:37:44 UTC
88e0b42 simulator: Add documentation. 20 January 2021, 16:37:44 UTC
62c6d90 simulator: Make min_batch_size a parameter defined in the setup. 20 January 2021, 16:37:44 UTC
9468bb9 simulator: add basic tests for fill_test_data and run 20 January 2021, 16:37:44 UTC
ead7b34 simulator: implement a simulator for the "old" task-based scheduler We extend the Task object with an autogenerated uuid allowing us to track the task lifetime between its creation and the generation of visit statuses, as the task-based scheduler does. 20 January 2021, 16:37:44 UTC
aecd27e Move the simulator cli to the main cli module 20 January 2021, 16:37:44 UTC
05067e3 simulator: Replace attrs with dataclasses for consistency 20 January 2021, 16:37:44 UTC
24922fe simulator: wrap tasks and task events in typechecked objects This allows us to extend these objects without redefining a bunch of type annotations. 20 January 2021, 16:37:44 UTC
d5318ae simulator: also fill data for the task-based scheduler 20 January 2021, 16:37:44 UTC
22ebb7a simulator: Split into smaller files in the same package 20 January 2021, 16:37:44 UTC
ad7bfbe simulator: Make the run time a CLI argument 20 January 2021, 16:37:44 UTC
df34db0 simulator: tweak simulation environment constants 20 January 2021, 16:37:44 UTC
21ce2c8 simulator: generate more origins in fill_data 20 January 2021, 16:37:44 UTC
2920419 simulator: add typing for Environment.scheduler 20 January 2021, 16:37:44 UTC
6433266 simulator: add support for a basic SimulationReport For now, this collects the runtime of tasks that have run, and gets printed at the end of the simulation. 20 January 2021, 16:37:44 UTC
c474a82 simulator: refine origin model to follow an exponential distribution This models origins using a consistent characteristic "time between commits" that follows an exponential distribution between 1 second and 10 years. From this characteristic time, and feedback from the OriginVisitStats, we can generate the expected run time and output status of the next visit of that origin. 20 January 2021, 16:37:44 UTC
2459bad simulator: Remove some debug statements and lower log level 20 January 2021, 16:37:44 UTC
cb12449 simulator: simulate the scheduler journal client 20 January 2021, 16:37:44 UTC
20b7f9c simulator: generate OriginVisitStatus objects in modeled visits To be able to generate uneventful visits, we would need to store the last snapshot seen for a given origin. Instead of storing this within the simulator, which would be a concern for large scale simulations, we use the scheduler visit cache directly. 20 January 2021, 16:37:44 UTC
39ad47d simulator: Move scheduler into the simulation environment object The scheduler is used by a lot of the simulated actors, it makes sense to share it all the time. 20 January 2021, 16:37:44 UTC
31967fa simulator: Use datetimes instead of a floating point simulated time 20 January 2021, 16:37:44 UTC
fc3f06b Introduce scaffolding for a scheduler simulator This simulator will allow us to compare the behavior of the old and new schedulers, as well as to test the impact of scheduler policies and their parameters on the performance of the Software Heritage archival infrastructure as a whole. 20 January 2021, 16:37:44 UTC
7905a6b Add a cli for the scheduler metrics update endpoint 20 January 2021, 16:35:05 UTC
c386fdf Make the max_date() helper function accept *dates as argument so it can be called with more than 2 dates. 20 January 2021, 11:28:02 UTC
b03d978 Make sure swh.scheduler.cli.journal is loaded in test_cli_journal.py needed to make pytest able to run directly (without tox). 20 January 2021, 11:18:25 UTC
737d12e Introduce a new lister_get endpoint 20 January 2021, 10:02:21 UTC
114ed95 Implement some basic aggregated metrics on listed origins Metrics are computed and cached database-side by the `update_metrics` function. The `get_metrics` function only retrieves the cached data. The metrics are aggregated for each lister instance and visit type (allowing complete reaggregation by visit type for cross-cutting statistics). The following metrics have been implemented: - number of known origins overall - number of enabled origins (origins seen in the last listing) - number of enabled origins that have never been successfully visited - number of enabled origins with known activity since our last successful visit 20 January 2021, 09:54:27 UTC
9852653 Import the journal subcommand in the main swh.scheduler cli This issue was masked by tox.ini using pytest with --doctest-modules, which imports all modules during test collection, and therefore executing the side-effects of swh.scheduler.cli.journal. 20 January 2021, 09:35:09 UTC
f8627a9 Move the `last_scheduled` ts from ListedOrigin to OriginVisitStatus this timestamp being actually a loading-related value, it makes more sense to keep it in the OriginVisitStatus table. Related to T2444. 19 January 2021, 16:48:51 UTC
0a32a31 Make the journal-client cli subcommand automagically loaded otherwise it won't be advertized as a `swh scehduler` subcommand by default. Also add a short dosctring for better --help. 19 January 2021, 15:18:49 UTC
5e609d5 requirements: Make swh.journal and optional dependency This avoids pulling journal dependencies when modules only needs the swh-scheduler dependency. 19 January 2021, 11:04:37 UTC
9395aa0 scheduler.cli.journal: Add `swh scheduler journal-client` cli This adds the cli entrypoint to actually process origin_visit_status topics and write to the origin_visit_stats db table. Related to T2967 19 January 2021, 10:10:41 UTC
58ca796 journal_client: Improve stats detection This adds an integration test which permutes input to ensure out of order renders the same result. This also improves the current algorithm which revealed some hit-and-miss cases: - Initialization of the first visit detection (through the "last_snapshot" absence field, the previous implementation check could fail otherwise). - out of order policy (ignore old event) in case of supposedly "eventful" event was done too early which ignored too much messages (those new test cases failed in some permutations). This is now specifically checked in case of referenced snapshots which led to cases of possibly changing eventful event into uneventful one. For example, the case of an anterior eventful event is caught which means that the current most-up-to-date eventful event is actually an uneventful one). ... Related to T2967 19 January 2021, 09:17:05 UTC
d3afd14 Use the recorded task end time for the task scheduler feedback loop This allows us to run "time-warping" simulations without interference from the real wall clock time. 15 January 2021, 16:04:30 UTC
a5fb291 backend: Make origin_visit_stats_upsert a batch api Related to T2967 15 January 2021, 13:34:06 UTC
608aa20 Populate origin_visit_stats table out of the origin_visit_status topic The snapshot is used to determine the "eventful/uneventful" nature of the origin visit status. When no snapshot is provided, the visit is considered as failed so the last_failed column is updated. As there is no time guarantee when reading message from the topic, the code tries to keep the data in the most timely ordered as possible. Only most recent information is kept. Related to T2967 15 January 2021, 13:34:05 UTC
ca45d40 Filter origins by visit type when scheduling the next visits We have separate task queues and workers for each visit type, so it makes sense to split this endpoint along these lines too, at least for now. 14 January 2021, 12:53:31 UTC
59b4cb3 Reorganize ListedOrigin fixtures to generate multiple visit_types 14 January 2021, 12:53:31 UTC
4f5338f Introduce a `swh scheduler origin schedule-next` cli This creates one-shot tasks in the classic scheduler for the next visits to run according to the visit scheduling policy. 14 January 2021, 12:53:31 UTC
3dd1d5f Rename test task types to names that match real tasks The success of tests using these task types would depend on the test run order, because these task types are (currently) being created by swh/scheduler/sql/50-data.sql, but the table is truncated after the first test completes. 14 January 2021, 12:53:31 UTC
5d7b002 Introduce a `swh scheduler origin grab-next` cli This returns, as CSV, the next origins to be visited according to the passed scheduling policy. 14 January 2021, 12:53:31 UTC
a620033 Add an new origin visit info model object and related backend api Upsert and Read methods Related to T2443 12 January 2021, 13:47:49 UTC
b13cb1f Implement a basic endpoint for getting the next origins to visit The basic policy implemented is a FIFO, to get things going. 11 January 2021, 14:40:17 UTC
619100e Add a cli section to the doc 18 December 2020, 14:57:00 UTC
1d55616 New upstream version 0.8.2 07 December 2020, 08:55:37 UTC
ebff12b requirement: Adapt celery requirements This adapts the celery requirements to the last known where our builds are fine. Currently, 5.0.3 got released and this ends up making all the swh modules relying on tasks timeout. A bug upstream is opened [1]. In the mean time, this workaround fixes [2] and most probably the remaining swh builds. [1] https://github.com/celery/celery/issues/6521 [2] https://jenkins.softwareheritage.org/job/DSCH/job/tests/1132/console 07 December 2020, 08:28:00 UTC
3c87075 Replace usage of arrow datetime objects in favor of pure datetime ones Note that the humanize library is now used in the cli pretty printing function (in place of the arrow hiumanize feature). As a result, displayed output from some cli commands may slightly differ. Closes T2835. 03 December 2020, 09:25:36 UTC
1b390a7 Stop using the deprecated configuration scheme 25 November 2020, 14:56:18 UTC
1f68031 cli.task_type: All task_type clis without a scheduler should raise As the code will plainly fail on calling methods on None instance if not caught early. 25 November 2020, 14:56:18 UTC
f37a1bf New upstream version 0.8.1 24 November 2020, 13:16:07 UTC
9e5b17f conftest: Reference swh.core.db.pytest_plugin As it's exposed through the swh.scheduler.pytest_plugin itself used by other swh modules, this needs to be declared to avoid other swh module build failures. Related to T2746 24 November 2020, 13:10:31 UTC
c17a34f New upstream version 0.8.0 23 November 2020, 12:44:39 UTC
49ed819 requirements-test.txt: Drop no longer needed pytest-postgresql requirement requirements-swh.txt already declares the swh.core[db] dependency which transitively pulls it. Related to T2746 23 November 2020, 12:11:02 UTC
2f9e8ec scheduler.pytest_plugin: Make scheduler tests faster Reuse the swh.core.db.pytest_plugin factory 22 October 2020, 10:09:06 UTC
f385291 New upstream version 0.7.0 19 October 2020, 07:33:53 UTC
6a4455c pytest_plugin: Explicitely name the scheduler test db differently When using tests on modules with different lower level modules (e.g storage, scheduler, ...) this avoids clashes. 19 October 2020, 07:25:04 UTC
13dcadd scheduler: Type and unify get_scheduler factory with other factories Related to T1410 16 October 2020, 16:24:03 UTC
dd33cdc test_server: Simplify exception manipulations 16 October 2020, 11:43:54 UTC
315a2c9 tox.ini: pin black to the pre-commit version (19.10b0) to avoid flip-flops 02 October 2020, 14:24:01 UTC
b7e5358 Drop vcversioner from requirements We stopped using it months ago. 25 September 2020, 15:19:17 UTC
4951a23 Run isort after the CLI import changes 25 September 2020, 12:19:21 UTC
ba781a5 New upstream version 0.6.0 25 September 2020, 10:06:31 UTC
be7a5ae Rename sql files according to swh.core 0.3 25 September 2020, 07:53:53 UTC
5cc573d Adapt cli declaration entrypoint to swh.core 0.3 25 September 2020, 07:48:38 UTC
1d40f20 New upstream version 0.5.3 24 September 2020, 15:53:24 UTC
99e5af8 Move from kombu.five.monotonic to time.monotonic Looks like kombu finally axed python2 support. 24 September 2020, 15:44:00 UTC
7b0d48f python: Reorder imports with isort Related to T2610 17 September 2020, 16:03:39 UTC
8d8b58f pre-commit: Add isort hook and configuration Related to T2610 17 September 2020, 16:03:39 UTC
4bec5c8 pre-commit: Update flake8 hook configuration flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks so now use the one from https://gitlab.com/pycqa/flake8 17 September 2020, 16:03:39 UTC
f5c8154 cli: speedup the `swh` cli command startup time by moving import statements in functions and using conditional import of typechecking modules (especially StorageInterface which triggers the loading of 300+ modules). Related to T2575. 10 September 2020, 15:46:08 UTC
b24be0c Tell pytest not to recurse in dotdirs. pytest wastes a lot of time in .hypothesis and .git; this commit excludes them. 25 August 2020, 08:41:38 UTC
6426208 cli.task: Migrate scheduler cli to latest storage change on iter_origins Related to T645 03 August 2020, 10:18:23 UTC
849d063 test_cli: Adapt tests data and drop unsupported "validate" proxy 24 July 2020, 08:22:07 UTC
9f52d95 cli.task: Fix iter_origin returned types Related to T2494 21 July 2020, 08:36:03 UTC
c26569f New upstream version 0.5.2 10 July 2020, 11:08:29 UTC
254e24a Do no expose pytest-plugin through setuptools, let modules require it when needed Defining the pytest-plugin though the pytest-plugin [1] makes it loaded by default. This creates loading issues on modules depending on scheduler but not on the pytest plugin scheduler exposes as explained in the doc [2] Instead we'll explicitely define to modules depending on the pytest plugins in their root conftest [3]: pytest_plugins = [ "swh.scheduler.pytest_plugin" ] [1] https://docs.pytest.org/en/stable/writing_plugins.html#setuptools-entry-points [2] https://docs.pytest.org/en/stable/writing_plugins.html#plugin-discovery-order-at-tool-startup [3] https://docs.pytest.org/en/stable/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file Related to D3475 Related to T2484 10 July 2020, 10:27:42 UTC
7a6149f New upstream version 0.5.1 09 July 2020, 09:51:36 UTC
0bc33b2 New upstream version 0.5.0 09 July 2020, 08:20:41 UTC
ece598c requirements.txt: Remove future dependency This was needed for celery 4.4.4 but that version is not used anymore. 08 July 2020, 16:33:25 UTC
back to top