https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
76c3420 Updated debian changelog for version 1.13.0 04 December 2023, 16:29:11 UTC
6e8e7aa Update upstream source from tag 'debian/upstream/1.13.0' Update to upstream version '1.13.0' with Debian dir 99b8a698b6863771f6a9301ba82075e96ec69596 04 December 2023, 16:29:10 UTC
72762ce New upstream version 1.13.0 04 December 2023, 16:29:09 UTC
23dc6fb Apply swh-py-template 0.1.6 03 December 2023, 17:36:44 UTC
88d6854 Migrate to copier-based swh-py-template 29 November 2023, 15:34:21 UTC
a38d793 Updated debian changelog for version 1.12.1 20 November 2023, 15:11:40 UTC
cb27c18 Update upstream source from tag 'debian/upstream/1.12.1' Update to upstream version '1.12.1' with Debian dir d7a9719759b45411ae1dc31308b7a5a121ae089a 20 November 2023, 15:11:39 UTC
30b59c9 New upstream version 1.12.1 20 November 2023, 15:11:38 UTC
7c775bb docs: include the README file in the main index page Convert README from markdown to ReST to make it embeddable in docs/index.rst 16 November 2023, 17:01:25 UTC
70687af Add b-d on python3-testing.postgresql 09 November 2023, 15:42:00 UTC
aae9583 sql: remove task type creation from the sql init scripts This is now handled by the worker plugin system. 08 November 2023, 09:13:46 UTC
daa6398 Updated debian changelog for version 1.12.0 07 November 2023, 18:02:32 UTC
58c9cd0 New upstream version 1.12.0 07 November 2023, 18:02:31 UTC
f744d09 Update upstream source from tag 'debian/upstream/1.12.0' Update to upstream version '1.12.0' with Debian dir f8ccc6ab3d6768fb0c120f29fece22e0507848a3 07 November 2023, 18:02:31 UTC
6247685 backend: Add a temporary postgresql scheduler backend When using that backend, a temporary scheduler database is spawned then removed when the backend gets destroyed. It can be used for testing SWH components that require a scheduler instance (listers for instance). 07 November 2023, 16:21:29 UTC
a726552 Remove version restriction on importlib_metadata for python > 3.7 The issue has been solved for some time now. 20 October 2023, 09:50:11 UTC
df95a28 Updated debian changelog for version 1.11.0 06 October 2023, 14:17:42 UTC
38a32b0 Update upstream source from tag 'debian/upstream/1.11.0' Update to upstream version '1.11.0' with Debian dir ec936f438ce3fae0e8d8fc8124f5da20ffea370c 06 October 2023, 14:17:42 UTC
2e35dd9 New upstream version 1.11.0 06 October 2023, 14:17:41 UTC
2a64df0 parse_time_interval: Improve parsing to deal down to seconds This also improves: - the spacing between time period, making it optional. - the units can be simple up to the full units (e.g h, hr, hrs, hour, hours...) 06 October 2023, 14:04:30 UTC
c175460 cli.origin.send-to-celery: Allow providing cooldown flags This should ease rescheduling origins manually in staging for testing purposes. 06 October 2023, 13:36:20 UTC
5b13c1d Drop duplicated, desynchonized and misplaced test 06 October 2023, 13:14:32 UTC
869c499 Updated debian changelog for version 1.10.2 03 October 2023, 07:55:07 UTC
774ca72 Update upstream source from tag 'debian/upstream/1.10.2' Update to upstream version '1.10.2' with Debian dir 77d2779d1d005db3d2bef5141aeaaa12ad1621eb 03 October 2023, 07:55:06 UTC
6556394 New upstream version 1.10.2 03 October 2023, 07:55:05 UTC
0dba862 Provide a default max_queue_length value to task_type Because the register task type routine does not provide the value, it's left unchecked. Once a new lister starts listing origins, the scheduler keeps on scheduling new origins in the queue without limits. As the consumption may be slower than the production, that tends towards too much resources usage in rabbitmq. This should limit the issue for new deployments. 02 October 2023, 15:34:12 UTC
70ec45d Updated debian changelog for version 1.10.1 30 August 2023, 12:07:35 UTC
1e20054 Update upstream source from tag 'debian/upstream/1.10.1' Update to upstream version '1.10.1' with Debian dir 9629a3591550c3e377c3cc8849bcb523405fbbff 30 August 2023, 12:07:34 UTC
1bdaf91 New upstream version 1.10.1 30 August 2023, 12:07:33 UTC
5b4029f debian/control: Remove python3-swh.lister build dependency 30 August 2023, 11:59:57 UTC
9a91b8d MANIFEST.in: Include missing tests datadir 30 August 2023, 11:51:28 UTC
c99be45 requirements-test: Remove swh.lister dependency The swh.lister package was required as testing dependency to check registration of celery tasks for listers through plugins declared in swh.workers entrypoints. However, it is easy to create a fake lister in tests data to check its correct registration through scheduler CLI so that swh.lister dependency is not really needed. As a consequence, worker plugins are now discovered in the function register_task_types from the swh.scheduler.cli.task_type module, previoulsy it was done when importing the module. 30 August 2023, 11:51:28 UTC
f052c9b cli: Document configuration expected by every endpoint 30 August 2023, 11:39:36 UTC
45c8083 Updated debian changelog for version 1.10.0 17 August 2023, 13:30:36 UTC
748513f Update upstream source from tag 'debian/upstream/1.10.0' Update to upstream version '1.10.0' with Debian dir 295b25587e1858468b53521aaf6889672c5be82c 17 August 2023, 13:30:35 UTC
24e6746 New upstream version 1.10.0 17 August 2023, 13:30:34 UTC
d379ab6 add-forge-now/cli: add check-ingested-origins command Related to swh/devel/swh-scheduler#4684 07 August 2023, 11:12:48 UTC
8d543cf add-forge-now/cli: add check-listed-origins command Related to swh/devel/swh-scheduler#4683 07 August 2023, 11:12:48 UTC
a79332f Updated debian changelog for version 1.9.2 10 July 2023, 15:12:00 UTC
61bacb2 Update upstream source from tag 'debian/upstream/1.9.2' Update to upstream version '1.9.2' with Debian dir e263213862aaf44891039eeeebd73c711c7f6579 10 July 2023, 15:11:59 UTC
ecb604e New upstream version 1.9.2 10 July 2023, 15:11:58 UTC
4b316fa Fix mypy/click: add swh.core[testing] in requirements-test.txt It now needs types-click which is indeed a dependency of swh.core[testing]. 07 July 2023, 14:51:48 UTC
9f849c2 scheduler: Update default policy to schedule origins without last update Prior to this, we considered not to do it. However, we do have some listers which are not able to list origins with a last update. And we still need to be able to schedule those origins nonetheless hence this change. Refs. swh/infra/sysadm-environment#4971 04 July 2023, 15:58:34 UTC
836d9e2 cli: Fix docstring format 20 June 2023, 12:51:41 UTC
f526038 Updated debian changelog for version 1.9.1 13 June 2023, 11:14:15 UTC
18f2cd6 New upstream version 1.9.1 13 June 2023, 11:14:14 UTC
ef6ec9f Update upstream source from tag 'debian/upstream/1.9.1' Update to upstream version '1.9.1' with Debian dir d5f40620c0d43877f9972f63c92be6a85be2aa81 13 June 2023, 11:14:14 UTC
006d60c sql: Fix task creation when providing a custom next_run value Previously, it was not possible to create sequentially two oneshot tasks whose only differ by their next_run value. Related to swh/devel/swh-web#4548 13 June 2023, 11:07:58 UTC
04a2207 cli: Use ctx.fail instead of raising an exception 07 June 2023, 14:43:47 UTC
b36c577 Updated debian changelog for version 1.9.0 31 May 2023, 08:19:51 UTC
eff1241 Update upstream source from tag 'debian/upstream/1.9.0' Update to upstream version '1.9.0' with Debian dir 856ce9f56b53c9cebd36d4080a5e148fcf8d8db9 31 May 2023, 08:19:50 UTC
19633d9 New upstream version 1.9.0 31 May 2023, 08:19:49 UTC
cf2ca93 cli.add_forge_now: Allow queue name prefix override This will allow to schedule add-forge-now requests to different queues. For example [1] will send git tasks to the add_forge_now_slow:swh.loader.git... queue. [1] ``` swh ... add-forge-now ... \ --preset $ENVIRONMENT \ schedule-first-visits \ --type-name git \ --prefix-queue add_forge_now_slow ``` 30 May 2023, 15:14:22 UTC
6299df4 add-forge-now: Improve conditional so incremental listing is delayed Otherwise, listing types without a 'list-%-full', 'list-%-incremental' pattern (e.g list-cgit, ...) are systematically delayed 1 day the first time the add-forge-now schedules them. Refs. swh/infra/sysadm-environment#4845 12 April 2023, 09:28:17 UTC
241bd25 Updated debian changelog for version 1.8.0 31 March 2023, 10:40:07 UTC
2bb9e87 Update upstream source from tag 'debian/upstream/1.8.0' Update to upstream version '1.8.0' with Debian dir 0106931d01df74e904ec9d14e6bae1e8d447ad50 31 March 2023, 10:40:06 UTC
5f13852 New upstream version 1.8.0 31 March 2023, 10:40:05 UTC
ddcd7c8 celery_backend/config: Enable to set Sentry DSN per task type Add a task_prerun celery signal handler in order to set Sentry DSN based on task name or package name. The mapping between a task/package name and its DSN must be stored in configuration under a "sentry_settings_for_celery_tasks" key. For this feature to work, no SWH_SENTRY_DSN and SWH_MAIN_PACKAGE environment variables should be defined as they override the sentry_dsn and main_package values passed to init_sentry function. Related to swh/meta#4949. 28 March 2023, 15:52:15 UTC
9e790d4 Updated debian changelog for version 1.7.0 21 March 2023, 13:44:56 UTC
e6629da Update upstream source from tag 'debian/upstream/1.7.0' Update to upstream version '1.7.0' with Debian dir 5ffe6676732b6589e3350b3f65ec83e5b32720ea 21 March 2023, 13:44:55 UTC
707698a New upstream version 1.7.0 21 March 2023, 13:44:54 UTC
5936ae1 add-forge-now: Allow scheduling of cgit task type Refs. swh/infra/sysadm-environment#4813 21 March 2023, 12:00:04 UTC
c24b0c8 mypy: Bump to 1.0.1 and fix new typing errors Related to swh/meta#4960 17 February 2023, 16:59:03 UTC
4cb605e Update and clean tox configuration for version 4 Related to swh/meta#4959 16 February 2023, 16:10:00 UTC
e33d0ad pre-commit: Bump isort from 5.10.1 to 5.11.5 This fixes python 3.7 support due to poetry, a dependency of isort, that removed support for that Python version in a recent release. 02 February 2023, 10:07:36 UTC
195b832 Updated debian changelog for version 1.6.0 31 January 2023, 17:19:09 UTC
c57e758 Update upstream source from tag 'debian/upstream/1.6.0' Update to upstream version '1.6.0' with Debian dir 0eaa4da2fc97d458abc4a235f88c53ad26c969b8 31 January 2023, 17:19:08 UTC
cb428ae New upstream version 1.6.0 31 January 2023, 17:19:07 UTC
9beef90 Configure logging from environment variable SWH_LOG_CONFIG When not provided, this uses the logging configuration coded in the scheduler (as before). Refs. swh/infra/sysadm-environment#4524 31 January 2023, 16:59:14 UTC
bebf298 swh.scheduler.cli: Pass initialization exceptions to subcommands 30 January 2023, 15:27:11 UTC
3546c1c Updated debian changelog for version 1.5.1 27 January 2023, 11:26:14 UTC
9cf76b6 Update upstream source from tag 'debian/upstream/1.5.1' Update to upstream version '1.5.1' with Debian dir c4cdd8eb7810cfde921d809d5bf37bddfc587c53 27 January 2023, 11:26:13 UTC
f7947dd New upstream version 1.5.1 27 January 2023, 11:26:12 UTC
a65c4ed celery_backend/config: Fix missing comma in setup_log_handler Because of that missing comma, an exception was raised (tuple object is not callable) but it was caught and displayed by the _print_errors decorator so tests could not detect it. As a consequence, the logging configuration of celery workers was broken. Add a test to check if an exception was raised by the setup_log_handler function to avoid bad surprises when deploying to production or in docker. 26 January 2023, 15:11:11 UTC
7d3e9ae require pytest-postgresql < 4.0.0 25 January 2023, 13:48:56 UTC
037946a Add missing dependency on pytest-postgresql It is used by the pytest plugin 25 January 2023, 13:37:42 UTC
d68d03c Updated debian changelog for version 1.5.0 24 January 2023, 13:26:22 UTC
00fd130 Update upstream source from tag 'debian/upstream/1.5.0' Update to upstream version '1.5.0' with Debian dir 5c5abea93e5496c1e4ce76325e777e811f41bb4a 24 January 2023, 13:26:21 UTC
5a97137 New upstream version 1.5.0 24 January 2023, 13:26:20 UTC
8f0849a Allow logging configuration from configuration yaml file This will allow proper logging configuration for the services which are currently running in the dynamic infrastructure. Their logs are current written in the wrong elasticsearch indices. Ref. swh/infra/sysadm-environment#4524 23 January 2023, 17:03:12 UTC
fccf944 Add missing __init__.py so find_packages keep finding sql modules Otherwise, at some point, this will get discarded as per the debian build warning [1] [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DSCH/job/gbp-buildpackage/182/console 02 January 2023, 09:21:57 UTC
d521ab7 docs: Include module indices only when building standalone package doc In order to remove warnings about /apidoc/*.rst files being included multiple times in toc when building full swh documentation, prefer to include module indices only when building standalone package documentation. Also include them the proper sphinx way. Related to T4496 19 December 2022, 14:10:54 UTC
3ca9293 Updated debian changelog for version 1.4.0 12 December 2022, 10:51:31 UTC
0f46f3a Update upstream source from tag 'debian/upstream/1.4.0' Update to upstream version '1.4.0' with Debian dir 0fc297ff329f9f004f363b713e958febf0acc324 12 December 2022, 10:51:30 UTC
76030a1 New upstream version 1.4.0 12 December 2022, 10:51:30 UTC
8e125f1 cli.add_forge_now: Open `register-lister` with sensible defaults This will ease scheduling of new add-forge-now requests, on: - staging: this will list a subset of disabled origins once - production: this will register recurring tasks (full, incremental if any) to list that new forge This also unifies the previous subcommand schedule-first-visits with the --preset flag. So, the following would be enough to list appropriately in staging/production: ``` swh scheduler add-forge-now \ ( --preset [production|staging] \ # to enable a pre-defined set of rules ) register-lister \ gitea \ url=https://git.afpy.org/api/v1/ ``` Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 08 December 2022, 17:51:45 UTC
1c34e98 cli.add_forge_now: Open `schedule-first-visits` with sensible defaults This should ease scheduling the first visits for add-forge-now request. The following would be enough to fetch and schedule the forge just listed (be it in production or staging): ``` swh scheduler add-forge-now \ schedule-first-visits \ --visit-type git \ (--visit-type svn \ # if a lister lists multiple kinds of visit, we can mention it ) --lister-name gitea \ --lister-instance-name git.afpy.org \ ( --production | --staging ) # to list only enabled | disabled origins ``` Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 07 December 2022, 15:44:28 UTC
03c0d1b Updated debian changelog for version 1.3.0 07 December 2022, 12:50:33 UTC
80c1df4 Update upstream source from tag 'debian/upstream/1.3.0' Update to upstream version '1.3.0' with Debian dir 4b8c2ba3ff41fa515c60acf0dd8a33c9e97e7600 07 December 2022, 12:50:32 UTC
354f2d4 New upstream version 1.3.0 07 December 2022, 12:50:31 UTC
e2878b5 task add: Ensure task type provided exist and raise otherwise Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 07 December 2022, 11:57:04 UTC
cd16fce grab_next_visits: Open lister name and instance name filtering Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 06 December 2022, 16:03:32 UTC
a776963 send-to-celery: Adapt to schedule from lister name & instance_name This allows to bypass the lister id retrieval step using directly the name and instance name of the lister to discover the uuid. This also drops the --lister-uuid flag which is somewhat difficult to use. Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 06 December 2022, 15:54:02 UTC
ff75e74 Ensure origins are not visited faster than twice a day The scheduled_cooldown only applies to tasks that have not been executed yet. absolute_cooldown avoids archiving objects faster than that. 25 October 2022, 14:48:51 UTC
1f9109f Refresh task type data from the database every time recurrent tasks are run Avoids inconsistencies between the database state and an ongoing recurrent task scheduler. 25 October 2022, 14:48:51 UTC
bde27a9 Use json instead of msgpack for serializers Recent celery versions generate serialized messages with mime types incompatible with older versions when using msgpack 25 October 2022, 13:51:01 UTC
aeb870a pre-commit, tox: Bump pre-commit, codespell, black and flake8 - pre-commit from 4.1.0 to 4.3.0, - codespell from 2.2.1 to 2.2.2, - black from 22.3.0 to 22.10.0 and - flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies. Also change flake8's repo config to github (the gitlab mirror being outdated). 18 October 2022, 16:53:38 UTC
87ff3db Updated debian changelog for version 1.2.3 03 October 2022, 12:07:44 UTC
edc1a2e Update upstream source from tag 'debian/upstream/1.2.3' Update to upstream version '1.2.3' with Debian dir 692868b76356cffe70463fb90e278de55a0e035c 03 October 2022, 12:07:43 UTC
0929c07 New upstream version 1.2.3 03 October 2022, 12:07:42 UTC
back to top