| e2878b5 | Antoine R. Dumont (@ardumont) | 07 December 2022, 11:38:58 UTC | task add: Ensure task type provided exist and raise otherwise Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 | 07 December 2022, 11:57:04 UTC |
| cd16fce | Antoine R. Dumont (@ardumont) | 06 December 2022, 16:01:41 UTC | grab_next_visits: Open lister name and instance name filtering Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 | 06 December 2022, 16:03:32 UTC |
| a776963 | Antoine R. Dumont (@ardumont) | 06 December 2022, 11:24:33 UTC | send-to-celery: Adapt to schedule from lister name & instance_name This allows to bypass the lister id retrieval step using directly the name and instance name of the lister to discover the uuid. This also drops the --lister-uuid flag which is somewhat difficult to use. Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4674 | 06 December 2022, 15:54:02 UTC |
| ff75e74 | Nicolas Dandrimont | 25 October 2022, 13:48:55 UTC | Ensure origins are not visited faster than twice a day The scheduled_cooldown only applies to tasks that have not been executed yet. absolute_cooldown avoids archiving objects faster than that. | 25 October 2022, 14:48:51 UTC |
| 1f9109f | Nicolas Dandrimont | 25 October 2022, 13:47:37 UTC | Refresh task type data from the database every time recurrent tasks are run Avoids inconsistencies between the database state and an ongoing recurrent task scheduler. | 25 October 2022, 14:48:51 UTC |
| bde27a9 | Nicolas Dandrimont | 25 October 2022, 13:46:26 UTC | Use json instead of msgpack for serializers Recent celery versions generate serialized messages with mime types incompatible with older versions when using msgpack | 25 October 2022, 13:51:01 UTC |
| aeb870a | David Douard | 18 October 2022, 16:21:00 UTC | pre-commit, tox: Bump pre-commit, codespell, black and flake8 - pre-commit from 4.1.0 to 4.3.0, - codespell from 2.2.1 to 2.2.2, - black from 22.3.0 to 22.10.0 and - flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies. Also change flake8's repo config to github (the gitlab mirror being outdated). | 18 October 2022, 16:53:38 UTC |
| 17c6d48 | Antoine R. Dumont (@ardumont) | 03 October 2022, 11:43:32 UTC | Fix compatibility issue with latest dependency version This currently fails all swh related builds which depend on the celery/kombu stack due to that dependency's latest version release. | 03 October 2022, 11:58:46 UTC |
| 6d0b1d1 | Antoine R. Dumont (@ardumont) | 23 September 2022, 07:48:30 UTC | backend: Prevent query exception when lister ids is empty Related to T4545 | 23 September 2022, 07:49:04 UTC |
| b1afdab | Antoine Lambert | 14 September 2022, 14:18:51 UTC | recurrent_visits: Allow to set no origins scheduled backoff in config The send_visits_for_visit_type function uses a default schedule backoff of 20 minutes where there is no origins to schedule for a given visit type. It exists use cases when we would like that schedule backoff to be shorter in order to schedule listed origins for loading into the archive more rapidly, typically in the docker environment. So allow to set that backoff value through configuration. | 15 September 2022, 08:41:20 UTC |
| 7cfaa98 | Antoine Lambert | 22 August 2022, 13:19:50 UTC | sql/Makefile: Fix swh-scheduler SQL file paths Those files have been renamed so the database could not be filled. | 22 August 2022, 13:19:50 UTC |
| fd6df6a | Antoine R. Dumont (@ardumont) | 29 July 2022, 08:12:23 UTC | api/server: Clarify load and check configuration backend This adds type to the function, update its docstring and clarify its associated tests as well. | 29 July 2022, 08:12:23 UTC |
| d847448 | David Douard | 08 July 2022, 12:00:33 UTC | Fix the load_and_check_config() function to support the "postgresql" cls value and replace usage of the "local" scheduler cls with "postgresql" everywhere. | 08 July 2022, 12:23:46 UTC |
| 0496c39 | Antoine R. Dumont (@ardumont) | 03 June 2022, 12:41:51 UTC | Remove unused get_current_version method Attribute current_version is already set and directly used by swh db [version|init|upgrade] clis. Related to T4305 | 03 June 2022, 12:44:56 UTC |
| ef15385 | David Douard | 31 May 2022, 12:21:31 UTC | tests: use stock pytest_postgresql factory function instead of (soon-to-be-deprecated) swh-core's postgresql_fact one. | 31 May 2022, 14:46:05 UTC |
| e56fc4d | Antoine Lambert | 12 May 2022, 09:08:09 UTC | interface: Return enabled origins only by default in get_listed_origins Add a new enabled_only parameter set to True by default in get_listed_origins scheduler method. It enables to filter out by default disabled listed origins when requesting the result of a listing and avoid possible errors in listers implementation. | 12 May 2022, 10:07:17 UTC |
| c7c53ea | Pratyush Desai | 09 May 2022, 10:13:54 UTC | add strict asyncio_mode in pytest.ini | 09 May 2022, 10:13:54 UTC |
| 1d50b2e | Antoine Lambert | 06 May 2022, 15:05:20 UTC | cli/task: Fix sphinx >= 4.4 warning Fix "more than one target found for cross-reference 'Origin'" sphinx warning. | 06 May 2022, 15:06:23 UTC |
| 881b521 | Benoit Chauvet | 28 April 2022, 13:56:01 UTC | Add missing sentry captures | 28 April 2022, 13:59:44 UTC |
| 82274c1 | Valentin Lorentz | 27 April 2022, 13:15:28 UTC | cli/utils: Fix parsing of empty strings | 27 April 2022, 13:15:28 UTC |
| 353cf2a | Valentin Lorentz | 26 April 2022, 11:05:15 UTC | Bump mypy to v0.942 | 26 April 2022, 11:05:15 UTC |
| 0365b85 | Valentin Lorentz | 21 April 2022, 16:40:55 UTC | Add a 'lister_instance_name' argument to all tasks created from ListedOrigin This will allow loaders to use the right API credentials to fetch extrinsic metadata for the origin from the forge. | 26 April 2022, 10:28:37 UTC |
| 42e362d | Valentin Lorentz | 21 April 2022, 10:22:03 UTC | Add a 'lister_name' argument to all tasks created from ListedOrigin This will allow loaders to guess the forge type, and use the right API to fetch extrinsic metadata for the origin from the forge. | 26 April 2022, 10:28:33 UTC |
| 3687931 | David Douard | 25 April 2022, 16:14:29 UTC | Update a bit the documentation for the new origin visit scheduler | 26 April 2022, 08:38:05 UTC |
| 9483493 | Valentin Lorentz | 21 April 2022, 09:22:48 UTC | Make create_origin_task_dict a standalone function It feels off as an object method; and I am going to make it use joins in a future commit, so it makes more sense this way. | 21 April 2022, 15:15:06 UTC |
| 5e9ee60 | Valentin Lorentz | 21 April 2022, 09:21:05 UTC | test_utils.py: Convert to pytest-style tests | 21 April 2022, 11:47:58 UTC |
| 9627e6d | Antoine Lambert | 21 April 2022, 11:39:49 UTC | pre-commit: Remove codespell commit-msg hook That hook can be frustrating as it can discard a long commit message if it finds a typo in it so better removing it. | 21 April 2022, 11:39:49 UTC |
| a76bb02 | David Douard | 15 April 2022, 16:08:49 UTC | Make scheduling policy used in schedule_recurrent configurable Add support for a configuration option "scheduling_policy" in the config file loaded by the 'swh scheduler schedule-recurrent' command. This config entry allows to specify the scheduling policies used by the schedule-recurrent tool, instead of having them hardcoded in the source code. A visit type policy config entry should have at least a 'weight' value for each policy. Default values are unchanged. Eg.: scheduling_policy: git: - policy: already_visited_order_by_lag weight: 55 tablesample: 0.5 - policy: never_visited_oldest_update_first weight: 45 tablesample: 0.5 Note: there may not be configuration entries for all visit types, but if a visit type policy is configured, the config entry should be complete (in other words, the merging of the configuration with the default values is only done at first config level). | 20 April 2022, 14:34:23 UTC |
| 5302efd | Antoine Lambert | 08 April 2022, 13:15:35 UTC | Add .git-blame-ignore-revs file with automatic reformatting commits | 08 April 2022, 13:15:35 UTC |
| 3f0843b | Antoine Lambert | 08 April 2022, 13:15:09 UTC | python: Reformat code with black 22.3.0 Related to T3922 | 08 April 2022, 13:15:09 UTC |
| d9a2512 | Antoine Lambert | 08 April 2022, 13:13:50 UTC | pre-commit, tox: Bump black from 19.10b0 to 22.3.0 black is considered stable since release 22.1.0 and the version we are currently using is quite outdated and not compatible with click 8.1.0, so it is time to bump it to its latest stable release. Please note that E501 pycodestyle warning related to line length is replaced by B950 one from flake8-bugbear as recommended by black. https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length Related to T3922 | 08 April 2022, 13:13:50 UTC |
| bafe03f | Antoine Lambert | 06 April 2022, 15:14:52 UTC | requirements-test: Remove pytest pinning to < 7 pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for pytest >= 7 so we can now drop the pytest pinning. | 06 April 2022, 15:14:52 UTC |
| 78f5579 | Antoine Lambert | 22 March 2022, 10:58:10 UTC | pytest: Exclude build directory for tests discovery Due to test modules being copied in subdirectories of the build directory by setuptools, it makes pytest fail by raising ImportPathMismatchError exceptions when invoked from root directory of the module. So ignore the build folder to discover tests. | 22 March 2022, 10:58:10 UTC |
| 5cc62be | David Douard | 08 February 2022, 13:59:29 UTC | Adapt to swh.core 2.0.0 - add the `get_datastore` function in `swh.scheduler` - add the `get_current_version` method in `SchedulerBackend`, - remove dbversion management from sql init script - update tests accordingly | 24 February 2022, 14:51:44 UTC |
| 234e165 | Antoine Lambert | 10 February 2022, 16:23:34 UTC | pre-commit: Bump hooks and add new one to check commit message spelling To install the new hook: $ pre-commit install -t commit-msg | 10 February 2022, 16:23:34 UTC |
| fddec02 | Antoine Lambert | 09 February 2022, 13:22:06 UTC | requirements: Remove click version pin Latest versions of celery and flask now support click >= 8.0 so we can remove the version pin. | 09 February 2022, 13:22:46 UTC |
| c46ffad | David Douard | 08 February 2022, 16:26:17 UTC | Prefix task types used in tests with 'test-' so that tests do not depend on a lucky guess on what the scheduler db state actually is. DB initialization scripts do create task types for git, hg and svn (used in tests) but these tests depends on the fact the db fixture has been called already once before, so tables are truncated (especially the task and task_type ones). For example running a single test involved in task-type creation was failing (eg. 'pytest swh -k test_create_task_type_idempotence'). This commit does make tests not collide with any existing task or task type initialization scripts may create. Note that this also means that there is actually no test dealing with the scheduler db state after initialization, which is not grat and should be addressed. | 08 February 2022, 16:34:10 UTC |
| 9f601f5 | Antoine R. Dumont (@ardumont) | 07 February 2022, 15:46:47 UTC | requirements-test: Pin pytest to < 7.0.0 Related to T3916 | 07 February 2022, 15:47:00 UTC |
| ce11283 | Valentin Lorentz | 21 January 2022, 10:10:48 UTC | Fix ReST syntax | 21 January 2022, 10:14:59 UTC |
| b5477ea | Antoine R. Dumont (@ardumont) | 12 January 2022, 09:58:58 UTC | sql: Clean up task/task_run data model This archives current task and task_run tables, creating new ones filtering only necessary tasks (last 2 months' oneshot tasks plus some recurring tasks; lister, indexer, ...). Those filtered tasks are the ones scheduled by the runner and runner priority services. This archiving will allow those services to be faster (corresponding query execution time will outputs results faster without the archived data). Related to T3837 | 12 January 2022, 10:30:36 UTC |
| 5c836d6 | Vincent SELLIER | 04 January 2022, 23:08:50 UTC | Allow to specify the visit grab parameters per visit type and policy Related to T3827 | 05 January 2022, 17:18:32 UTC |
| 559f345 | Antoine R. Dumont (@ardumont) | 16 December 2021, 14:47:56 UTC | Pin mypy and drop type annotations which makes mypy unhappy This also drops spurious copyright headers to those files if present. Related to T3812 | 16 December 2021, 14:47:56 UTC |
| e051b32 | Nicolas Dandrimont | 09 December 2021, 13:54:09 UTC | Use a temporary table to update scheduler metrics When using ``insert into <...> select <...>``, PostgreSQL disables parallel querying. Under some circumstances (in our large production database), this makes updating the scheduler metrics take a (very) long time. Parallel querying is allowed for ``create table <...> as select <...>``, and doing so restores the small(er) runtimes for this query (15 minutes instead of multiple hours). To use that, we have to turn the function into plpgsql instead of plain sql. | 09 December 2021, 14:16:06 UTC |
| a8edbdb | Antoine R. Dumont (@ardumont) | 07 December 2021, 13:31:34 UTC | Clean up disabled scheduler archival task related services This is dead code now as this has long been stopped and disabled in production. Related to T3777 | 08 December 2021, 10:12:53 UTC |
| 5de8ba4 | Nicolas Dandrimont | 07 December 2021, 12:57:51 UTC | Make next_visit_queue_position an integer In visit types with small amounts of origins having no last_update field, we would end up overflowing Python datetimes (which only go up to 31 December 9999) pretty quickly. Making the queue position a 64-bit integer should give us some more leeway. The queue position now defaults to zero instead of an arbitrary point in time. Queue offsets are still commensurate with seconds, but that's mostly to give them some space to be splayed by the fudge factors. | 07 December 2021, 16:39:48 UTC |
| 0a6aac5 | Vincent SELLIER | 06 December 2021, 15:23:49 UTC | Ensure there is no duplicated origins in the insertion batches when a lister try to insert duplicate origins in the same batch, the insertion is failing because the "on cascade do update" instruction cannot manage duplicates in the same transaction Related to T3769 | 06 December 2021, 20:11:40 UTC |
| 2abb393 | Valentin Lorentz | 22 November 2021, 12:32:20 UTC | Fix CardinalityViolation in grab_next_visits on duplicate origins grab_next_visits grabs from `listed_origins`, whose primary key is `(lister_id, url, visit_type)` and uses it to upsert in origin_visit_stats, whose primary key is `(url, visit_type)`. This causes the error `ON CONFLICT DO UPDATE command cannot affect row a second time` when the same (origin, type) pair is grabbed twice. This commit deduplicates the (origin, type) pairs before upserting. | 22 November 2021, 12:36:24 UTC |
| 00ff02e | Nicolas Dandrimont | 29 October 2021, 13:58:31 UTC | recurrent visits: use policy weights instead of ratios The ratios weren't checked for normalization; using relative weights explicitly ensures that the settings won't be misinterpreted. | 29 October 2021, 13:58:31 UTC |
| 7f434c3 | Nicolas Dandrimont | 29 October 2021, 13:44:56 UTC | Improve docs rendering for recurrent visits scheduler | 29 October 2021, 13:44:56 UTC |
| 50d7fd7 | Nicolas Dandrimont | 27 October 2021, 10:09:42 UTC | Add a new cli endpoint to schedule recurrent visits in Celery For each known visit type, we run a loop which: - monitors the size of the relevant celery queue - schedules more visits of the relevant type once the number of available slots goes over a given threshold (currently set to 5% of the max queue size). The scheduling of visits combines multiple scheduling policies, for now using static ratios set in the `POLICY_RATIOS` dict. We emit a warning if the ratio of origins fetched for each policy is skewed with respect to the original request (allowing, for now, manual adjustement of the ratios). The CLI endpoint spawns one thread for each visit type, which all handle connections to RabbitMQ and the scheduler backend separately. For now, we handle exceptions in the visit scheduling threads by (stupidly) respawning the relevant thread directly. We should probably improve this to give up after a specific number of tries. Co-authored-by: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> | 28 October 2021, 11:06:56 UTC |
| 0c7ef27 | Nicolas Dandrimont | 27 October 2021, 13:45:09 UTC | grab_next_visits: avoid time interval calculations in PostgreSQL When the database is in a non-UTC timezone with DST, and a `timestamptz - interval` calculation crosses a DST change, the result of the calculation can be one hour off from the expected value: PostgreSQL will vary the timestamp by the amount of days in the interval, and will keep the same (local) time, which will be offset by an hour because of the DST change. Doing the datetime +- timedelta calculations in Python instead of PostgreSQL avoids this caveat altogether. | 27 October 2021, 13:45:09 UTC |
| ecc0e28 | Antoine R. Dumont (@ardumont) | 22 October 2021, 08:44:08 UTC | Restrict the click version to avoid conflict version with celery's Otherwise, in some edge case, like run in docker, the install fails on conflict. Related to P1205#8092 | 22 October 2021, 09:21:36 UTC |
| 243a69f | Antoine R. Dumont (@ardumont) | 20 October 2021, 08:42:34 UTC | Add docstring to runner and listener modules Related to T3667 | 20 October 2021, 09:25:38 UTC |
| 5b53196 | Antoine R. Dumont (@ardumont) | 20 October 2021, 08:44:34 UTC | Drop deprecated listener module It's been deprecated for enough time. Related to T3667 | 20 October 2021, 09:02:11 UTC |
| f15c510 | Antoine R. Dumont (@ardumont) | 20 October 2021, 07:36:27 UTC | scheduler: Deprecate unused main celery runner | 20 October 2021, 08:31:28 UTC |
| 3aed688 | Antoine R. Dumont (@ardumont) | 18 October 2021, 12:14:02 UTC | Use swh_storage fixture for cli tests This actually fixes the debian build failure. Related to T3666 | 18 October 2021, 12:16:38 UTC |
| 3aed7bf | Antoine R. Dumont (@ardumont) | 15 October 2021, 13:14:12 UTC | Return 0 slot if no more slots available in the queues This scenario happens with the loader oneshot for example. This loader deals with more than 1 type of origins to ingest in the same queue. So the computation of that function returned negative value [1]. Which is ultimately not possible to execute in sql [1]. This commits fixes that behavior. This also explicits that the function must return positive values in its docstring. [1] ``` ... psycopg2.errors.InvalidRowCountInLimitClause: LIMIT must not be negative ``` | 15 October 2021, 13:22:52 UTC |
| ecc1400 | Antoine R. Dumont (@ardumont) | 08 June 2021, 15:36:28 UTC | runner: Improve help message on the task types flag. | 02 September 2021, 09:15:36 UTC |
| 63fdda0 | Antoine R. Dumont (@ardumont) | 03 June 2021, 14:03:26 UTC | send-to-celery: Add more options to allow scheduling of edge cases In the non optimal case, we may want to trigger specific case (not-yet enabled origins, origin from specific lister...). Related to T3350 | 27 August 2021, 11:26:38 UTC |
| 7cc37fa | Nicolas Dandrimont | 01 June 2021, 17:17:16 UTC | Refine scheduling policy for origins with no known last update For origins that have never been visited, and for which we don't have a queue position yet, we want to visit them in the order they've been added. | 26 August 2021, 14:49:37 UTC |
| 2efad28 | Nicolas Dandrimont | 01 June 2021, 18:04:11 UTC | Add a swh scheduler origin send-to-celery subcommand The subcommand bypasses the legacy task-based mechanism to directly send new origin visits to celery | 26 August 2021, 14:48:46 UTC |
| 5e8007f | Nicolas Dandrimont | 01 June 2021, 13:48:05 UTC | Add table sampling option to grab_next_visits Running common operations on all git origins is pretty intense. Using table sampling gives us the opportunity to at least schedule some jobs in (decently small) time. | 26 August 2021, 14:47:52 UTC |
| cc76a57 | Antoine R. Dumont (@ardumont) | 26 August 2021, 09:44:14 UTC | journal_client: Only upsert if we have something to upsert | 26 August 2021, 09:44:14 UTC |
| 506f78c | Antoine R. Dumont (@ardumont) | 25 August 2021, 16:15:06 UTC | journal_client: Ensure queue position does not overflow Queue positions are date and the current next_position_offset used to compute the new queue position was not bounded. This has the side-effect of making overflow error. This commit adapts the journal client computations to limit such next_position_offset to 10. This value was chosen because above that exponent the dates overflow (and we are way in the future already). Related to T3502 | 26 August 2021, 08:24:11 UTC |
| 28ae1d8 | Valentin Lorentz | 18 August 2021, 09:20:25 UTC | Replace index-fossology-license-for-range with index-fossology-license-for-partition We changed the task name/interface a while ago | 18 August 2021, 09:20:25 UTC |
| 8281e35 | Antoine R. Dumont (@ardumont) | 08 July 2021, 09:24:42 UTC | journal_client: Disable origins when too many visited attempts failed This disable origins for either failed or not found attempts 3 times in a row. It's not definitive though as it's the lister's responsibility to activate back origins if they get listed again. Related to T2345 | 03 August 2021, 11:56:32 UTC |
| 1bcf84d | David Douard | 07 July 2021, 14:55:57 UTC | Add a successive_visits counter to origin visit stats This maintains the number of successive visits resulting in the same status. This will help implementing disabling of too many successive failed or not_found visits for a given origin. Related to T2345 | 03 August 2021, 10:49:45 UTC |
| 4fa29fe | Antoine R. Dumont (@ardumont) | 30 July 2021, 13:35:17 UTC | journal_client: Update get_last_status docstring Related to T2345 | 30 July 2021, 13:35:17 UTC |
| 3b929d0 | Antoine R. Dumont (@ardumont) | 30 July 2021, 13:23:14 UTC | journal_client: Refactor by inlining the update_position_offset This is no longer required as it's called once. Related to T2345 | 30 July 2021, 13:23:14 UTC |
| 87e66fa | Nicolas Dandrimont | 23 July 2021, 09:48:23 UTC | Only record last_visited and last_successful in origin_visit_stats After using this schema for a while, all queries can be implemented in terms of these two timestamps, instead of the four original last_eventful, last_uneventful, last_failed and last_notfound timestamps. This ends up simplifying the logic within the journal client, as well as that of the grab_next_visits query builder. To make this change work, we also stop considering out of order messages altogether in journal_client. This welcome simplification is an accuracy tradeoff that is explained in the updated documentation of the journal client: .. [1] Ignoring out of order messages makes the initialization of the origin_visit_status table (from a full journal) less deterministic: only the `last_visit`, `last_visit_state` and `last_successful` fields are guaranteed to be exact, the `next_position_offset` field is a best effort estimate (which should converge once the client has run for a while on in-order messages). | 23 July 2021, 09:56:32 UTC |
| 3ca0d65 | Antoine R. Dumont (@ardumont) | 23 July 2021, 07:22:46 UTC | test_journal_client: Unify test assertion like the rest Related to D5917 | 23 July 2021, 07:22:46 UTC |
| 8cf2238 | Antoine R. Dumont (@ardumont) | 22 July 2021, 09:42:24 UTC | test: Refactor assert_visit_stats_ok to ignore_fields This simplifies and unifies properly the utility test function to compare visit stats. | 23 July 2021, 07:18:20 UTC |
| d58776a | Antoine R. Dumont (@ardumont) | 22 July 2021, 10:22:24 UTC | Introduce new scheduling policy to grab origins without last update This is in charge of scheduling origins without last update. This also updates the global queue position so the journal client can initialize correctly the next position per origin and visit type. Related to T2345 | 22 July 2021, 10:23:44 UTC |
| 825e8cf | Nicolas Dandrimont | 22 July 2021, 10:19:42 UTC | grab_next_visits: make the handling of CTEs more modular This allows us to insert extra CTEs if a scheduling policy needs it. | 22 July 2021, 10:19:42 UTC |
| 8c4ae9f | Antoine R. Dumont (@ardumont) | 29 June 2021, 14:00:01 UTC | journal_client: Compute next position for origin visit For origin without any last_update information [1], the journal client is now also in charge of moving their next position in the queue for rescheduling. Depending on their status, the next position offset and next_visit_queue_position are updated after each visit completes: - if the visit has failed, increase the next visit target by the minimal visit interval (to take into account transient loading issues) - if the visit is successful, and records some changes, decrease the visit interval index by 2 (visit the origin *way* more often). - if the visit is successful, and records no changes, increase the visit interval index by 1 (visit the origin less often). We then set the next visit target to its current value + the new visit interval multiplied by a random fudge factor (picked in the -/+ 10% range). The fudge factor allows the visits to spread out, avoiding "bursts" of loaded origins e.g. when a number of origins from a single hoster are processed at once. Note that the computations happen for all origins for simplicity and code maintenance but it will only be used by a new soon-to-be scheduling policy. [1] Lister cannot provide it for some reason. | 06 July 2021, 12:35:13 UTC |
| cb1edf1 | Antoine R. Dumont (@ardumont) | 23 June 2021, 16:07:59 UTC | Introduce storage for the recurrent visit scheduler queue position | 01 July 2021, 08:36:44 UTC |
| ec6e69f | Antoine R. Dumont (@ardumont) | 23 June 2021, 14:42:26 UTC | Start handling of recurrent loading tasks in scheduler This deals first and foremost with the next_position_offset update done by the scheduler journal client. | 01 July 2021, 08:36:44 UTC |
| c486b28 | Antoine R. Dumont (@ardumont) | 29 June 2021, 12:41:07 UTC | journal_client: Explicit docstring | 29 June 2021, 13:16:15 UTC |
| 98f99b9 | Antoine R. Dumont (@ardumont) | 23 June 2021, 14:39:40 UTC | journal_client: Only check last_* fields for some permutation tests In a future commit, we will add new fields whose values will be permutation dependent. | 23 June 2021, 15:02:34 UTC |
| 1006f0a | Antoine R. Dumont (@ardumont) | 23 June 2021, 13:37:40 UTC | journal_client: Auto-generate the empty object from model fields This will help us when adding new fields to the table. | 23 June 2021, 14:54:34 UTC |
| 6400cc2 | Antoine R. Dumont (@ardumont) | 23 June 2021, 13:28:55 UTC | backend: Auto-generate origin visit stats upsert query This will help us when adding new fields to the table. | 23 June 2021, 14:54:34 UTC |
| 3762c34 | Antoine R. Dumont (@ardumont) | 23 June 2021, 14:49:12 UTC | cli/task: Ensure cli output is always in the same order | 23 June 2021, 14:54:34 UTC |
| ed81870 | Nicolas Dandrimont | 01 June 2021, 13:47:19 UTC | Add a specific cooldown for notfound origins This allows us to avoid repeating visits on them, until a next pass of the lister can mark them as disabled. | 23 June 2021, 09:13:00 UTC |
| 651ddcc | Nicolas Dandrimont | 21 June 2021, 15:34:00 UTC | Add a (longer) specific cooldown for failed origin visits | 23 June 2021, 09:13:00 UTC |
| ce8608d | Nicolas Dandrimont | 21 June 2021, 15:36:00 UTC | Make the origin visit scheduling cooldown configurable | 23 June 2021, 09:13:00 UTC |
| 7f51f27 | Antoine Lambert | 22 June 2021, 12:35:50 UTC | interface: Add get_listers method Add new method to scheduler interface returning the full list of listers registered in the database. Related to T3127 | 22 June 2021, 12:36:08 UTC |
| 9e1b414 | Nicolas Dandrimont | 01 June 2021, 13:43:32 UTC | Drop duplicate docstring from backend | 21 June 2021, 13:46:12 UTC |
| c7707b5 | Antoine R. Dumont (@ardumont) | 08 June 2021, 15:36:28 UTC | runner: Separate scheduling tasks with and without priority concern In effect, this will allow to run 2 runners: - one for recurring tasks - one for the save code now This should decrease the probability of the scheduling tasks for the save code now to be stuck behind the main scheduler runner. Related to T3367 | 10 June 2021, 12:55:04 UTC |
| 21c4279 | Antoine R. Dumont (@ardumont) | 10 June 2021, 10:15:22 UTC | Refactor and extract a get_available_slots utility This adds coverage as well. This will be needed for subsidiary diffs. Related to T3367 | 10 June 2021, 10:15:22 UTC |
| 9d2618d | Antoine R. Dumont (@ardumont) | 09 June 2021, 10:29:38 UTC | Add typing stubs dependencies for mypy>0.900 This also explicits missing dependencies | 09 June 2021, 12:13:36 UTC |
| 9f7ab8f | Antoine Lambert | 25 May 2021, 10:15:04 UTC | pytest_plugin: Explicitly set hostname in broker_url for celery TestApp Since the release of kombu 5.1.0, a warning is now issued when a hostname is not set in the broker_url config value of a celery app. That change makes the test_celery_monitor_ping test fails due to that new unexpected warning. So explicitly add localhost hostname in the broker_url value of the celery TestApp config. | 25 May 2021, 11:43:03 UTC |
| fe9d949 | Valentin Lorentz | 06 May 2021, 14:16:50 UTC | Fix flaky test_grab_ready_* tests | 06 May 2021, 14:20:57 UTC |
| 8a892e2 | Valentin Lorentz | 06 May 2021, 12:24:47 UTC | Use swh.core 0.14 It renamed db_name to dbname, which is a breaking change. | 06 May 2021, 13:49:47 UTC |
| bab557e | Nicolas Dandrimont | 26 April 2021, 10:55:00 UTC | Remove row locking from SQL queries This would only be useful if we had multiple runners running concurrently, but that's not the case. | 30 April 2021, 18:13:38 UTC |
| feff179 | Antoine Lambert | 26 April 2021, 16:01:59 UTC | tox: Add sphinx environments to check sane doc build Enable to check package documentation can be built without producing sphinx warnings. The sphinx environment is designed to be used in continuous integration in order to prevent breaking documentation build when committing changes. The sphinx-dev environment is designed to be used inside a full swh development environment. Related to T3258 | 26 April 2021, 16:01:59 UTC |
| f186910 | Antoine R. Dumont (@ardumont) | 20 April 2021, 13:47:08 UTC | Add default index to task(type, next_run) in schema The staging scheduler runner was slow when fetching task due to that missing index. Related to T3271#63831 | 20 April 2021, 13:50:19 UTC |
| f33f743 | Valentin Lorentz | 20 April 2021, 09:46:53 UTC | Simplify priority computation in tests + improve exhaustivity We no longer need to deal with ratios, so let's count the objects directly instead. Plus, the existing tests did not check tasks with None priority (because they did not have access to it when ratios were given by the backend), so they do now. | 20 April 2021, 11:01:33 UTC |
| f4e6292 | Antoine R. Dumont (@ardumont) | 20 April 2021, 10:18:23 UTC | sql/updates/27: Fix sql upgrade script Related to TT3271 | 20 April 2021, 10:18:23 UTC |
| befccb9 | Antoine R. Dumont (@ardumont) | 19 April 2021, 10:26:58 UTC | scheduler: Clean up priority/ratio task dead code Since [1], tasks with priority are routed to dedicated queues (see tasks for more details). The tasks with priority to be scheduled have their own dedicated endpoints to be called. [1] Related to T3084 Related to T3271 | 20 April 2021, 09:27:18 UTC |
| 4e06bcd | Valentin Lorentz | 20 April 2021, 08:59:53 UTC | Parse task_ids before calling set_status_tasks. So errors on the CLI side do not trigger an exception on the server | 20 April 2021, 09:19:52 UTC |
