https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
9e68b73 Updated backport on buster-swh from debian/1.2.2-1_swh1 (unstable-swh) 15 September 2022, 12:02:44 UTC
d621c90 Merge tag 'debian/1.2.2-1_swh1' into debian/buster-swh 15 September 2022, 12:02:44 UTC
4604ff8 Updated debian changelog for version 1.2.2 15 September 2022, 12:00:17 UTC
889cf7b Update upstream source from tag 'debian/upstream/1.2.2' Update to upstream version '1.2.2' with Debian dir fb5e868766d116ab788888d47d0f6258440a68b2 15 September 2022, 12:00:16 UTC
cfa5a6f New upstream version 1.2.2 15 September 2022, 12:00:15 UTC
b1afdab recurrent_visits: Allow to set no origins scheduled backoff in config The send_visits_for_visit_type function uses a default schedule backoff of 20 minutes where there is no origins to schedule for a given visit type. It exists use cases when we would like that schedule backoff to be shorter in order to schedule listed origins for loading into the archive more rapidly, typically in the docker environment. So allow to set that backoff value through configuration. 15 September 2022, 08:41:20 UTC
7cfaa98 sql/Makefile: Fix swh-scheduler SQL file paths Those files have been renamed so the database could not be filled. 22 August 2022, 13:19:50 UTC
fd6df6a api/server: Clarify load and check configuration backend This adds type to the function, update its docstring and clarify its associated tests as well. 29 July 2022, 08:12:23 UTC
07bd0b2 Updated backport on buster-swh from debian/1.2.1-1_swh1 (unstable-swh) 08 July 2022, 14:57:04 UTC
f5a9137 Merge tag 'debian/1.2.1-1_swh1' into debian/buster-swh 08 July 2022, 14:57:03 UTC
4b6972c Updated debian changelog for version 1.2.1 08 July 2022, 14:53:11 UTC
b462eab Update upstream source from tag 'debian/upstream/1.2.1' Update to upstream version '1.2.1' with Debian dir 8876e6480e8b880ad9f16a682166f4ad7951a25b 08 July 2022, 14:53:11 UTC
58b365e New upstream version 1.2.1 08 July 2022, 14:53:10 UTC
d847448 Fix the load_and_check_config() function to support the "postgresql" cls value and replace usage of the "local" scheduler cls with "postgresql" everywhere. 08 July 2022, 12:23:46 UTC
7f60f11 Updated backport on buster-swh from debian/1.2.0-1_swh1 (unstable-swh) 03 June 2022, 13:52:19 UTC
d7d7aeb Merge tag 'debian/1.2.0-1_swh1' into debian/buster-swh 03 June 2022, 13:52:14 UTC
d8bc426 Updated debian changelog for version 1.2.0 03 June 2022, 13:47:47 UTC
8ebdc1a Update upstream source from tag 'debian/upstream/1.2.0' Update to upstream version '1.2.0' with Debian dir 5308e2d13ddbbe929fe3a61aa2a483e605787857 03 June 2022, 13:47:45 UTC
ad7ca47 New upstream version 1.2.0 03 June 2022, 13:47:44 UTC
0496c39 Remove unused get_current_version method Attribute current_version is already set and directly used by swh db [version|init|upgrade] clis. Related to T4305 03 June 2022, 12:44:56 UTC
ef15385 tests: use stock pytest_postgresql factory function instead of (soon-to-be-deprecated) swh-core's postgresql_fact one. 31 May 2022, 14:46:05 UTC
1e34659 Updated backport on buster-swh from debian/1.1.2-1_swh1 (unstable-swh) 12 May 2022, 11:58:56 UTC
bc1a6cb Merge tag 'debian/1.1.2-1_swh1' into debian/buster-swh 12 May 2022, 11:58:55 UTC
4e04ccf Updated debian changelog for version 1.1.2 12 May 2022, 11:55:14 UTC
b2e342f Update upstream source from tag 'debian/upstream/1.1.2' Update to upstream version '1.1.2' with Debian dir 77f8815b707fded0e03cae23a4909d7f281a2e97 12 May 2022, 11:55:13 UTC
407dd3d New upstream version 1.1.2 12 May 2022, 11:55:12 UTC
e56fc4d interface: Return enabled origins only by default in get_listed_origins Add a new enabled_only parameter set to True by default in get_listed_origins scheduler method. It enables to filter out by default disabled listed origins when requesting the result of a listing and avoid possible errors in listers implementation. 12 May 2022, 10:07:17 UTC
c7c53ea add strict asyncio_mode in pytest.ini 09 May 2022, 10:13:54 UTC
1d50b2e cli/task: Fix sphinx >= 4.4 warning Fix "more than one target found for cross-reference 'Origin'" sphinx warning. 06 May 2022, 15:06:23 UTC
881b521 Add missing sentry captures 28 April 2022, 13:59:44 UTC
5c68ac1 Updated backport on buster-swh from debian/1.1.1-1_swh1 (unstable-swh) 28 April 2022, 09:40:07 UTC
3b23f58 Merge tag 'debian/1.1.1-1_swh1' into debian/buster-swh 28 April 2022, 09:40:06 UTC
f092ed3 Updated debian changelog for version 1.1.1 28 April 2022, 09:36:24 UTC
23ce0d9 Update upstream source from tag 'debian/upstream/1.1.1' Update to upstream version '1.1.1' with Debian dir 51b9198d0925a58c5f477ee300095bd0c9e8f9b6 28 April 2022, 09:36:23 UTC
d9e982e New upstream version 1.1.1 28 April 2022, 09:36:23 UTC
82274c1 cli/utils: Fix parsing of empty strings 27 April 2022, 13:15:28 UTC
353cf2a Bump mypy to v0.942 26 April 2022, 11:05:15 UTC
a79109d Updated backport on buster-swh from debian/1.1.0-1_swh1 (unstable-swh) 26 April 2022, 10:39:25 UTC
053d425 Merge tag 'debian/1.1.0-1_swh1' into debian/buster-swh 26 April 2022, 10:39:24 UTC
f642da4 Updated debian changelog for version 1.1.0 26 April 2022, 10:35:52 UTC
d912c65 Update upstream source from tag 'debian/upstream/1.1.0' Update to upstream version '1.1.0' with Debian dir 728c35186bf7d46bb2e39efbe69cf3e4981c7311 26 April 2022, 10:35:51 UTC
442fcdb New upstream version 1.1.0 26 April 2022, 10:35:50 UTC
0365b85 Add a 'lister_instance_name' argument to all tasks created from ListedOrigin This will allow loaders to use the right API credentials to fetch extrinsic metadata for the origin from the forge. 26 April 2022, 10:28:37 UTC
42e362d Add a 'lister_name' argument to all tasks created from ListedOrigin This will allow loaders to guess the forge type, and use the right API to fetch extrinsic metadata for the origin from the forge. 26 April 2022, 10:28:33 UTC
3687931 Update a bit the documentation for the new origin visit scheduler 26 April 2022, 08:38:05 UTC
9483493 Make create_origin_task_dict a standalone function It feels off as an object method; and I am going to make it use joins in a future commit, so it makes more sense this way. 21 April 2022, 15:15:06 UTC
5e9ee60 test_utils.py: Convert to pytest-style tests 21 April 2022, 11:47:58 UTC
9627e6d pre-commit: Remove codespell commit-msg hook That hook can be frustrating as it can discard a long commit message if it finds a typo in it so better removing it. 21 April 2022, 11:39:49 UTC
a76bb02 Make scheduling policy used in schedule_recurrent configurable Add support for a configuration option "scheduling_policy" in the config file loaded by the 'swh scheduler schedule-recurrent' command. This config entry allows to specify the scheduling policies used by the schedule-recurrent tool, instead of having them hardcoded in the source code. A visit type policy config entry should have at least a 'weight' value for each policy. Default values are unchanged. Eg.: scheduling_policy: git: - policy: already_visited_order_by_lag weight: 55 tablesample: 0.5 - policy: never_visited_oldest_update_first weight: 45 tablesample: 0.5 Note: there may not be configuration entries for all visit types, but if a visit type policy is configured, the config entry should be complete (in other words, the merging of the configuration with the default values is only done at first config level). 20 April 2022, 14:34:23 UTC
5302efd Add .git-blame-ignore-revs file with automatic reformatting commits 08 April 2022, 13:15:35 UTC
3f0843b python: Reformat code with black 22.3.0 Related to T3922 08 April 2022, 13:15:09 UTC
d9a2512 pre-commit, tox: Bump black from 19.10b0 to 22.3.0 black is considered stable since release 22.1.0 and the version we are currently using is quite outdated and not compatible with click 8.1.0, so it is time to bump it to its latest stable release. Please note that E501 pycodestyle warning related to line length is replaced by B950 one from flake8-bugbear as recommended by black. https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length Related to T3922 08 April 2022, 13:13:50 UTC
bafe03f requirements-test: Remove pytest pinning to < 7 pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for pytest >= 7 so we can now drop the pytest pinning. 06 April 2022, 15:14:52 UTC
78f5579 pytest: Exclude build directory for tests discovery Due to test modules being copied in subdirectories of the build directory by setuptools, it makes pytest fail by raising ImportPathMismatchError exceptions when invoked from root directory of the module. So ignore the build folder to discover tests. 22 March 2022, 10:58:10 UTC
488e97d Updated backport on buster-swh from debian/1.0.0-1_swh1 (unstable-swh) 24 February 2022, 16:07:31 UTC
ca4501c Merge tag 'debian/1.0.0-1_swh1' into debian/buster-swh 24 February 2022, 16:07:31 UTC
fded717 Updated debian changelog for version 1.0.0 24 February 2022, 16:03:55 UTC
87e54e3 Update upstream source from tag 'debian/upstream/1.0.0' Update to upstream version '1.0.0' with Debian dir 7e7d67a960f191f55f41140f0b00c7a1fe6e30fc 24 February 2022, 16:03:54 UTC
a63dbac New upstream version 1.0.0 24 February 2022, 16:03:53 UTC
43794aa Prepare v1: bump dependency to swh.core 2 also match dependency on swh.storage with requirements-swh.txt 24 February 2022, 15:52:44 UTC
5cc62be Adapt to swh.core 2.0.0 - add the `get_datastore` function in `swh.scheduler` - add the `get_current_version` method in `SchedulerBackend`, - remove dbversion management from sql init script - update tests accordingly 24 February 2022, 14:51:44 UTC
234e165 pre-commit: Bump hooks and add new one to check commit message spelling To install the new hook: $ pre-commit install -t commit-msg 10 February 2022, 16:23:34 UTC
fddec02 requirements: Remove click version pin Latest versions of celery and flask now support click >= 8.0 so we can remove the version pin. 09 February 2022, 13:22:46 UTC
c46ffad Prefix task types used in tests with 'test-' so that tests do not depend on a lucky guess on what the scheduler db state actually is. DB initialization scripts do create task types for git, hg and svn (used in tests) but these tests depends on the fact the db fixture has been called already once before, so tables are truncated (especially the task and task_type ones). For example running a single test involved in task-type creation was failing (eg. 'pytest swh -k test_create_task_type_idempotence'). This commit does make tests not collide with any existing task or task type initialization scripts may create. Note that this also means that there is actually no test dealing with the scheduler db state after initialization, which is not grat and should be addressed. 08 February 2022, 16:34:10 UTC
9f601f5 requirements-test: Pin pytest to < 7.0.0 Related to T3916 07 February 2022, 15:47:00 UTC
ce11283 Fix ReST syntax 21 January 2022, 10:14:59 UTC
b5477ea sql: Clean up task/task_run data model This archives current task and task_run tables, creating new ones filtering only necessary tasks (last 2 months' oneshot tasks plus some recurring tasks; lister, indexer, ...). Those filtered tasks are the ones scheduled by the runner and runner priority services. This archiving will allow those services to be faster (corresponding query execution time will outputs results faster without the archived data). Related to T3837 12 January 2022, 10:30:36 UTC
2816054 Updated backport on buster-swh from debian/0.23.0-1_swh1 (unstable-swh) 06 January 2022, 08:42:54 UTC
3001d3a Merge tag 'debian/0.23.0-1_swh1' into debian/buster-swh 06 January 2022, 08:42:54 UTC
3b6e1d4 Updated debian changelog for version 0.23.0 06 January 2022, 08:39:47 UTC
67e1896 Update upstream source from tag 'debian/upstream/0.23.0' Update to upstream version '0.23.0' with Debian dir f7e1a8a1f5f6dc07dc335a3ea905631cb4f80385 06 January 2022, 08:39:46 UTC
4c9e164 New upstream version 0.23.0 06 January 2022, 08:39:45 UTC
5c836d6 Allow to specify the visit grab parameters per visit type and policy Related to T3827 05 January 2022, 17:18:32 UTC
559f345 Pin mypy and drop type annotations which makes mypy unhappy This also drops spurious copyright headers to those files if present. Related to T3812 16 December 2021, 14:47:56 UTC
e051b32 Use a temporary table to update scheduler metrics When using ``insert into <...> select <...>``, PostgreSQL disables parallel querying. Under some circumstances (in our large production database), this makes updating the scheduler metrics take a (very) long time. Parallel querying is allowed for ``create table <...> as select <...>``, and doing so restores the small(er) runtimes for this query (15 minutes instead of multiple hours). To use that, we have to turn the function into plpgsql instead of plain sql. 09 December 2021, 14:16:06 UTC
a8edbdb Clean up disabled scheduler archival task related services This is dead code now as this has long been stopped and disabled in production. Related to T3777 08 December 2021, 10:12:53 UTC
7b39742 Updated backport on buster-swh from debian/0.22.0-1_swh1 (unstable-swh) 08 December 2021, 09:13:32 UTC
623fdb7 Merge tag 'debian/0.22.0-1_swh1' into debian/buster-swh 08 December 2021, 09:13:30 UTC
0086f5a Updated debian changelog for version 0.22.0 08 December 2021, 09:06:02 UTC
bced01c Update upstream source from tag 'debian/upstream/0.22.0' Update to upstream version '0.22.0' with Debian dir 6ee09dd6732003e781781fea731b4a981ee1d0f1 08 December 2021, 09:05:45 UTC
10d495b New upstream version 0.22.0 08 December 2021, 09:05:44 UTC
5de8ba4 Make next_visit_queue_position an integer In visit types with small amounts of origins having no last_update field, we would end up overflowing Python datetimes (which only go up to 31 December 9999) pretty quickly. Making the queue position a 64-bit integer should give us some more leeway. The queue position now defaults to zero instead of an arbitrary point in time. Queue offsets are still commensurate with seconds, but that's mostly to give them some space to be splayed by the fudge factors. 07 December 2021, 16:39:48 UTC
0918b3a Updated backport on buster-swh from debian/0.21.0-1_swh1 (unstable-swh) 07 December 2021, 07:48:59 UTC
3f674a1 Merge tag 'debian/0.21.0-1_swh1' into debian/buster-swh 07 December 2021, 07:48:59 UTC
c5e514f Updated debian changelog for version 0.21.0 07 December 2021, 07:45:41 UTC
bedb322 Update upstream source from tag 'debian/upstream/0.21.0' Update to upstream version '0.21.0' with Debian dir 25bb4a20c7da58a06e1f9b407b51f2101d951472 07 December 2021, 07:45:40 UTC
a7851ad New upstream version 0.21.0 07 December 2021, 07:45:39 UTC
0a6aac5 Ensure there is no duplicated origins in the insertion batches when a lister try to insert duplicate origins in the same batch, the insertion is failing because the "on cascade do update" instruction cannot manage duplicates in the same transaction Related to T3769 06 December 2021, 20:11:40 UTC
3d366b1 Updated backport on buster-swh from debian/0.20.0-1_swh1 (unstable-swh) 22 November 2021, 15:19:15 UTC
26d32c9 Merge tag 'debian/0.20.0-1_swh1' into debian/buster-swh 22 November 2021, 15:19:14 UTC
377716e Updated debian changelog for version 0.20.0 22 November 2021, 15:14:57 UTC
3a9bdba Update upstream source from tag 'debian/upstream/0.20.0' Update to upstream version '0.20.0' with Debian dir 648a27f772aa18cbaeb88421ad44cfbf18517068 22 November 2021, 15:14:56 UTC
a42dbf8 New upstream version 0.20.0 22 November 2021, 15:14:55 UTC
2abb393 Fix CardinalityViolation in grab_next_visits on duplicate origins grab_next_visits grabs from `listed_origins`, whose primary key is `(lister_id, url, visit_type)` and uses it to upsert in origin_visit_stats, whose primary key is `(url, visit_type)`. This causes the error `ON CONFLICT DO UPDATE command cannot affect row a second time` when the same (origin, type) pair is grabbed twice. This commit deduplicates the (origin, type) pairs before upserting. 22 November 2021, 12:36:24 UTC
00ff02e recurrent visits: use policy weights instead of ratios The ratios weren't checked for normalization; using relative weights explicitly ensures that the settings won't be misinterpreted. 29 October 2021, 13:58:31 UTC
7f434c3 Improve docs rendering for recurrent visits scheduler 29 October 2021, 13:44:56 UTC
add752a Updated backport on buster-swh from debian/0.19.0-1_swh1 (unstable-swh) 28 October 2021, 11:17:22 UTC
0722eba Merge tag 'debian/0.19.0-1_swh1' into debian/buster-swh 28 October 2021, 11:17:21 UTC
83d25c5 Updated debian changelog for version 0.19.0 28 October 2021, 11:15:10 UTC
e78e56d Update upstream source from tag 'debian/upstream/0.19.0' Update to upstream version '0.19.0' with Debian dir 25276af0f39eed8f7341a27f86fd12f2168c0212 28 October 2021, 11:15:10 UTC
back to top