https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
241bd25 Updated debian changelog for version 1.8.0 31 March 2023, 10:40:07 UTC
2bb9e87 Update upstream source from tag 'debian/upstream/1.8.0' Update to upstream version '1.8.0' with Debian dir 0106931d01df74e904ec9d14e6bae1e8d447ad50 31 March 2023, 10:40:06 UTC
5f13852 New upstream version 1.8.0 31 March 2023, 10:40:05 UTC
ddcd7c8 celery_backend/config: Enable to set Sentry DSN per task type Add a task_prerun celery signal handler in order to set Sentry DSN based on task name or package name. The mapping between a task/package name and its DSN must be stored in configuration under a "sentry_settings_for_celery_tasks" key. For this feature to work, no SWH_SENTRY_DSN and SWH_MAIN_PACKAGE environment variables should be defined as they override the sentry_dsn and main_package values passed to init_sentry function. Related to swh/meta#4949. 28 March 2023, 15:52:15 UTC
9e790d4 Updated debian changelog for version 1.7.0 21 March 2023, 13:44:56 UTC
e6629da Update upstream source from tag 'debian/upstream/1.7.0' Update to upstream version '1.7.0' with Debian dir 5ffe6676732b6589e3350b3f65ec83e5b32720ea 21 March 2023, 13:44:55 UTC
707698a New upstream version 1.7.0 21 March 2023, 13:44:54 UTC
5936ae1 add-forge-now: Allow scheduling of cgit task type Refs. swh/infra/sysadm-environment#4813 21 March 2023, 12:00:04 UTC
c24b0c8 mypy: Bump to 1.0.1 and fix new typing errors Related to swh/meta#4960 17 February 2023, 16:59:03 UTC
4cb605e Update and clean tox configuration for version 4 Related to swh/meta#4959 16 February 2023, 16:10:00 UTC
e33d0ad pre-commit: Bump isort from 5.10.1 to 5.11.5 This fixes python 3.7 support due to poetry, a dependency of isort, that removed support for that Python version in a recent release. 02 February 2023, 10:07:36 UTC
195b832 Updated debian changelog for version 1.6.0 31 January 2023, 17:19:09 UTC
c57e758 Update upstream source from tag 'debian/upstream/1.6.0' Update to upstream version '1.6.0' with Debian dir 0eaa4da2fc97d458abc4a235f88c53ad26c969b8 31 January 2023, 17:19:08 UTC
cb428ae New upstream version 1.6.0 31 January 2023, 17:19:07 UTC
9beef90 Configure logging from environment variable SWH_LOG_CONFIG When not provided, this uses the logging configuration coded in the scheduler (as before). Refs. swh/infra/sysadm-environment#4524 31 January 2023, 16:59:14 UTC
bebf298 swh.scheduler.cli: Pass initialization exceptions to subcommands 30 January 2023, 15:27:11 UTC
3546c1c Updated debian changelog for version 1.5.1 27 January 2023, 11:26:14 UTC
9cf76b6 Update upstream source from tag 'debian/upstream/1.5.1' Update to upstream version '1.5.1' with Debian dir c4cdd8eb7810cfde921d809d5bf37bddfc587c53 27 January 2023, 11:26:13 UTC
f7947dd New upstream version 1.5.1 27 January 2023, 11:26:12 UTC
a65c4ed celery_backend/config: Fix missing comma in setup_log_handler Because of that missing comma, an exception was raised (tuple object is not callable) but it was caught and displayed by the _print_errors decorator so tests could not detect it. As a consequence, the logging configuration of celery workers was broken. Add a test to check if an exception was raised by the setup_log_handler function to avoid bad surprises when deploying to production or in docker. 26 January 2023, 15:11:11 UTC
7d3e9ae require pytest-postgresql < 4.0.0 25 January 2023, 13:48:56 UTC
037946a Add missing dependency on pytest-postgresql It is used by the pytest plugin 25 January 2023, 13:37:42 UTC
d68d03c Updated debian changelog for version 1.5.0 24 January 2023, 13:26:22 UTC
00fd130 Update upstream source from tag 'debian/upstream/1.5.0' Update to upstream version '1.5.0' with Debian dir 5c5abea93e5496c1e4ce76325e777e811f41bb4a 24 January 2023, 13:26:21 UTC
5a97137 New upstream version 1.5.0 24 January 2023, 13:26:20 UTC
8f0849a Allow logging configuration from configuration yaml file This will allow proper logging configuration for the services which are currently running in the dynamic infrastructure. Their logs are current written in the wrong elasticsearch indices. Ref. swh/infra/sysadm-environment#4524 23 January 2023, 17:03:12 UTC
fccf944 Add missing __init__.py so find_packages keep finding sql modules Otherwise, at some point, this will get discarded as per the debian build warning [1] [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DSCH/job/gbp-buildpackage/182/console 02 January 2023, 09:21:57 UTC
d521ab7 docs: Include module indices only when building standalone package doc In order to remove warnings about /apidoc/*.rst files being included multiple times in toc when building full swh documentation, prefer to include module indices only when building standalone package documentation. Also include them the proper sphinx way. Related to T4496 19 December 2022, 14:10:54 UTC
3ca9293 Updated debian changelog for version 1.4.0 12 December 2022, 10:51:31 UTC
0f46f3a Update upstream source from tag 'debian/upstream/1.4.0' Update to upstream version '1.4.0' with Debian dir 0fc297ff329f9f004f363b713e958febf0acc324 12 December 2022, 10:51:30 UTC
76030a1 New upstream version 1.4.0 12 December 2022, 10:51:30 UTC
8e125f1 cli.add_forge_now: Open `register-lister` with sensible defaults This will ease scheduling of new add-forge-now requests, on: - staging: this will list a subset of disabled origins once - production: this will register recurring tasks (full, incremental if any) to list that new forge This also unifies the previous subcommand schedule-first-visits with the --preset flag. So, the following would be enough to list appropriately in staging/production: ``` swh scheduler add-forge-now \ ( --preset [production|staging] \ # to enable a pre-defined set of rules ) register-lister \ gitea \ url=https://git.afpy.org/api/v1/ ``` Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 08 December 2022, 17:51:45 UTC
1c34e98 cli.add_forge_now: Open `schedule-first-visits` with sensible defaults This should ease scheduling the first visits for add-forge-now request. The following would be enough to fetch and schedule the forge just listed (be it in production or staging): ``` swh scheduler add-forge-now \ schedule-first-visits \ --visit-type git \ (--visit-type svn \ # if a lister lists multiple kinds of visit, we can mention it ) --lister-name gitea \ --lister-instance-name git.afpy.org \ ( --production | --staging ) # to list only enabled | disabled origins ``` Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 07 December 2022, 15:44:28 UTC
03c0d1b Updated debian changelog for version 1.3.0 07 December 2022, 12:50:33 UTC
80c1df4 Update upstream source from tag 'debian/upstream/1.3.0' Update to upstream version '1.3.0' with Debian dir 4b8c2ba3ff41fa515c60acf0dd8a33c9e97e7600 07 December 2022, 12:50:32 UTC
354f2d4 New upstream version 1.3.0 07 December 2022, 12:50:31 UTC
e2878b5 task add: Ensure task type provided exist and raise otherwise Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 07 December 2022, 11:57:04 UTC
cd16fce grab_next_visits: Open lister name and instance name filtering Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 06 December 2022, 16:03:32 UTC
a776963 send-to-celery: Adapt to schedule from lister name & instance_name This allows to bypass the lister id retrieval step using directly the name and instance name of the lister to discover the uuid. This also drops the --lister-uuid flag which is somewhat difficult to use. Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 06 December 2022, 15:54:02 UTC
ff75e74 Ensure origins are not visited faster than twice a day The scheduled_cooldown only applies to tasks that have not been executed yet. absolute_cooldown avoids archiving objects faster than that. 25 October 2022, 14:48:51 UTC
1f9109f Refresh task type data from the database every time recurrent tasks are run Avoids inconsistencies between the database state and an ongoing recurrent task scheduler. 25 October 2022, 14:48:51 UTC
bde27a9 Use json instead of msgpack for serializers Recent celery versions generate serialized messages with mime types incompatible with older versions when using msgpack 25 October 2022, 13:51:01 UTC
aeb870a pre-commit, tox: Bump pre-commit, codespell, black and flake8 - pre-commit from 4.1.0 to 4.3.0, - codespell from 2.2.1 to 2.2.2, - black from 22.3.0 to 22.10.0 and - flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies. Also change flake8's repo config to github (the gitlab mirror being outdated). 18 October 2022, 16:53:38 UTC
87ff3db Updated debian changelog for version 1.2.3 03 October 2022, 12:07:44 UTC
edc1a2e Update upstream source from tag 'debian/upstream/1.2.3' Update to upstream version '1.2.3' with Debian dir 692868b76356cffe70463fb90e278de55a0e035c 03 October 2022, 12:07:43 UTC
0929c07 New upstream version 1.2.3 03 October 2022, 12:07:42 UTC
17c6d48 Fix compatibility issue with latest dependency version This currently fails all swh related builds which depend on the celery/kombu stack due to that dependency's latest version release. 03 October 2022, 11:58:46 UTC
6d0b1d1 backend: Prevent query exception when lister ids is empty Related to T4545 23 September 2022, 07:49:04 UTC
4604ff8 Updated debian changelog for version 1.2.2 15 September 2022, 12:00:17 UTC
889cf7b Update upstream source from tag 'debian/upstream/1.2.2' Update to upstream version '1.2.2' with Debian dir fb5e868766d116ab788888d47d0f6258440a68b2 15 September 2022, 12:00:16 UTC
cfa5a6f New upstream version 1.2.2 15 September 2022, 12:00:15 UTC
b1afdab recurrent_visits: Allow to set no origins scheduled backoff in config The send_visits_for_visit_type function uses a default schedule backoff of 20 minutes where there is no origins to schedule for a given visit type. It exists use cases when we would like that schedule backoff to be shorter in order to schedule listed origins for loading into the archive more rapidly, typically in the docker environment. So allow to set that backoff value through configuration. 15 September 2022, 08:41:20 UTC
7cfaa98 sql/Makefile: Fix swh-scheduler SQL file paths Those files have been renamed so the database could not be filled. 22 August 2022, 13:19:50 UTC
fd6df6a api/server: Clarify load and check configuration backend This adds type to the function, update its docstring and clarify its associated tests as well. 29 July 2022, 08:12:23 UTC
4b6972c Updated debian changelog for version 1.2.1 08 July 2022, 14:53:11 UTC
b462eab Update upstream source from tag 'debian/upstream/1.2.1' Update to upstream version '1.2.1' with Debian dir 8876e6480e8b880ad9f16a682166f4ad7951a25b 08 July 2022, 14:53:11 UTC
58b365e New upstream version 1.2.1 08 July 2022, 14:53:10 UTC
d847448 Fix the load_and_check_config() function to support the "postgresql" cls value and replace usage of the "local" scheduler cls with "postgresql" everywhere. 08 July 2022, 12:23:46 UTC
d8bc426 Updated debian changelog for version 1.2.0 03 June 2022, 13:47:47 UTC
8ebdc1a Update upstream source from tag 'debian/upstream/1.2.0' Update to upstream version '1.2.0' with Debian dir 5308e2d13ddbbe929fe3a61aa2a483e605787857 03 June 2022, 13:47:45 UTC
ad7ca47 New upstream version 1.2.0 03 June 2022, 13:47:44 UTC
0496c39 Remove unused get_current_version method Attribute current_version is already set and directly used by swh db [version|init|upgrade] clis. Related to T4305 03 June 2022, 12:44:56 UTC
ef15385 tests: use stock pytest_postgresql factory function instead of (soon-to-be-deprecated) swh-core's postgresql_fact one. 31 May 2022, 14:46:05 UTC
4e04ccf Updated debian changelog for version 1.1.2 12 May 2022, 11:55:14 UTC
b2e342f Update upstream source from tag 'debian/upstream/1.1.2' Update to upstream version '1.1.2' with Debian dir 77f8815b707fded0e03cae23a4909d7f281a2e97 12 May 2022, 11:55:13 UTC
407dd3d New upstream version 1.1.2 12 May 2022, 11:55:12 UTC
e56fc4d interface: Return enabled origins only by default in get_listed_origins Add a new enabled_only parameter set to True by default in get_listed_origins scheduler method. It enables to filter out by default disabled listed origins when requesting the result of a listing and avoid possible errors in listers implementation. 12 May 2022, 10:07:17 UTC
c7c53ea add strict asyncio_mode in pytest.ini 09 May 2022, 10:13:54 UTC
1d50b2e cli/task: Fix sphinx >= 4.4 warning Fix "more than one target found for cross-reference 'Origin'" sphinx warning. 06 May 2022, 15:06:23 UTC
881b521 Add missing sentry captures 28 April 2022, 13:59:44 UTC
f092ed3 Updated debian changelog for version 1.1.1 28 April 2022, 09:36:24 UTC
23ce0d9 Update upstream source from tag 'debian/upstream/1.1.1' Update to upstream version '1.1.1' with Debian dir 51b9198d0925a58c5f477ee300095bd0c9e8f9b6 28 April 2022, 09:36:23 UTC
d9e982e New upstream version 1.1.1 28 April 2022, 09:36:23 UTC
82274c1 cli/utils: Fix parsing of empty strings 27 April 2022, 13:15:28 UTC
353cf2a Bump mypy to v0.942 26 April 2022, 11:05:15 UTC
f642da4 Updated debian changelog for version 1.1.0 26 April 2022, 10:35:52 UTC
d912c65 Update upstream source from tag 'debian/upstream/1.1.0' Update to upstream version '1.1.0' with Debian dir 728c35186bf7d46bb2e39efbe69cf3e4981c7311 26 April 2022, 10:35:51 UTC
442fcdb New upstream version 1.1.0 26 April 2022, 10:35:50 UTC
0365b85 Add a 'lister_instance_name' argument to all tasks created from ListedOrigin This will allow loaders to use the right API credentials to fetch extrinsic metadata for the origin from the forge. 26 April 2022, 10:28:37 UTC
42e362d Add a 'lister_name' argument to all tasks created from ListedOrigin This will allow loaders to guess the forge type, and use the right API to fetch extrinsic metadata for the origin from the forge. 26 April 2022, 10:28:33 UTC
3687931 Update a bit the documentation for the new origin visit scheduler 26 April 2022, 08:38:05 UTC
9483493 Make create_origin_task_dict a standalone function It feels off as an object method; and I am going to make it use joins in a future commit, so it makes more sense this way. 21 April 2022, 15:15:06 UTC
5e9ee60 test_utils.py: Convert to pytest-style tests 21 April 2022, 11:47:58 UTC
9627e6d pre-commit: Remove codespell commit-msg hook That hook can be frustrating as it can discard a long commit message if it finds a typo in it so better removing it. 21 April 2022, 11:39:49 UTC
a76bb02 Make scheduling policy used in schedule_recurrent configurable Add support for a configuration option "scheduling_policy" in the config file loaded by the 'swh scheduler schedule-recurrent' command. This config entry allows to specify the scheduling policies used by the schedule-recurrent tool, instead of having them hardcoded in the source code. A visit type policy config entry should have at least a 'weight' value for each policy. Default values are unchanged. Eg.: scheduling_policy: git: - policy: already_visited_order_by_lag weight: 55 tablesample: 0.5 - policy: never_visited_oldest_update_first weight: 45 tablesample: 0.5 Note: there may not be configuration entries for all visit types, but if a visit type policy is configured, the config entry should be complete (in other words, the merging of the configuration with the default values is only done at first config level). 20 April 2022, 14:34:23 UTC
5302efd Add .git-blame-ignore-revs file with automatic reformatting commits 08 April 2022, 13:15:35 UTC
3f0843b python: Reformat code with black 22.3.0 Related to T3922 08 April 2022, 13:15:09 UTC
d9a2512 pre-commit, tox: Bump black from 19.10b0 to 22.3.0 black is considered stable since release 22.1.0 and the version we are currently using is quite outdated and not compatible with click 8.1.0, so it is time to bump it to its latest stable release. Please note that E501 pycodestyle warning related to line length is replaced by B950 one from flake8-bugbear as recommended by black. https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length Related to T3922 08 April 2022, 13:13:50 UTC
bafe03f requirements-test: Remove pytest pinning to < 7 pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for pytest >= 7 so we can now drop the pytest pinning. 06 April 2022, 15:14:52 UTC
78f5579 pytest: Exclude build directory for tests discovery Due to test modules being copied in subdirectories of the build directory by setuptools, it makes pytest fail by raising ImportPathMismatchError exceptions when invoked from root directory of the module. So ignore the build folder to discover tests. 22 March 2022, 10:58:10 UTC
fded717 Updated debian changelog for version 1.0.0 24 February 2022, 16:03:55 UTC
87e54e3 Update upstream source from tag 'debian/upstream/1.0.0' Update to upstream version '1.0.0' with Debian dir 7e7d67a960f191f55f41140f0b00c7a1fe6e30fc 24 February 2022, 16:03:54 UTC
a63dbac New upstream version 1.0.0 24 February 2022, 16:03:53 UTC
43794aa Prepare v1: bump dependency to swh.core 2 also match dependency on swh.storage with requirements-swh.txt 24 February 2022, 15:52:44 UTC
5cc62be Adapt to swh.core 2.0.0 - add the `get_datastore` function in `swh.scheduler` - add the `get_current_version` method in `SchedulerBackend`, - remove dbversion management from sql init script - update tests accordingly 24 February 2022, 14:51:44 UTC
234e165 pre-commit: Bump hooks and add new one to check commit message spelling To install the new hook: $ pre-commit install -t commit-msg 10 February 2022, 16:23:34 UTC
fddec02 requirements: Remove click version pin Latest versions of celery and flask now support click >= 8.0 so we can remove the version pin. 09 February 2022, 13:22:46 UTC
c46ffad Prefix task types used in tests with 'test-' so that tests do not depend on a lucky guess on what the scheduler db state actually is. DB initialization scripts do create task types for git, hg and svn (used in tests) but these tests depends on the fact the db fixture has been called already once before, so tables are truncated (especially the task and task_type ones). For example running a single test involved in task-type creation was failing (eg. 'pytest swh -k test_create_task_type_idempotence'). This commit does make tests not collide with any existing task or task type initialization scripts may create. Note that this also means that there is actually no test dealing with the scheduler db state after initialization, which is not grat and should be addressed. 08 February 2022, 16:34:10 UTC
9f601f5 requirements-test: Pin pytest to < 7.0.0 Related to T3916 07 February 2022, 15:47:00 UTC
ce11283 Fix ReST syntax 21 January 2022, 10:14:59 UTC
back to top