https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
0918b3a Updated backport on buster-swh from debian/0.21.0-1_swh1 (unstable-swh) 07 December 2021, 07:48:59 UTC
3f674a1 Merge tag 'debian/0.21.0-1_swh1' into debian/buster-swh 07 December 2021, 07:48:59 UTC
c5e514f Updated debian changelog for version 0.21.0 07 December 2021, 07:45:41 UTC
bedb322 Update upstream source from tag 'debian/upstream/0.21.0' Update to upstream version '0.21.0' with Debian dir 25bb4a20c7da58a06e1f9b407b51f2101d951472 07 December 2021, 07:45:40 UTC
a7851ad New upstream version 0.21.0 07 December 2021, 07:45:39 UTC
0a6aac5 Ensure there is no duplicated origins in the insertion batches when a lister try to insert duplicate origins in the same batch, the insertion is failing because the "on cascade do update" instruction cannot manage duplicates in the same transaction Related to T3769 06 December 2021, 20:11:40 UTC
3d366b1 Updated backport on buster-swh from debian/0.20.0-1_swh1 (unstable-swh) 22 November 2021, 15:19:15 UTC
26d32c9 Merge tag 'debian/0.20.0-1_swh1' into debian/buster-swh 22 November 2021, 15:19:14 UTC
377716e Updated debian changelog for version 0.20.0 22 November 2021, 15:14:57 UTC
3a9bdba Update upstream source from tag 'debian/upstream/0.20.0' Update to upstream version '0.20.0' with Debian dir 648a27f772aa18cbaeb88421ad44cfbf18517068 22 November 2021, 15:14:56 UTC
a42dbf8 New upstream version 0.20.0 22 November 2021, 15:14:55 UTC
2abb393 Fix CardinalityViolation in grab_next_visits on duplicate origins grab_next_visits grabs from `listed_origins`, whose primary key is `(lister_id, url, visit_type)` and uses it to upsert in origin_visit_stats, whose primary key is `(url, visit_type)`. This causes the error `ON CONFLICT DO UPDATE command cannot affect row a second time` when the same (origin, type) pair is grabbed twice. This commit deduplicates the (origin, type) pairs before upserting. 22 November 2021, 12:36:24 UTC
00ff02e recurrent visits: use policy weights instead of ratios The ratios weren't checked for normalization; using relative weights explicitly ensures that the settings won't be misinterpreted. 29 October 2021, 13:58:31 UTC
7f434c3 Improve docs rendering for recurrent visits scheduler 29 October 2021, 13:44:56 UTC
add752a Updated backport on buster-swh from debian/0.19.0-1_swh1 (unstable-swh) 28 October 2021, 11:17:22 UTC
0722eba Merge tag 'debian/0.19.0-1_swh1' into debian/buster-swh 28 October 2021, 11:17:21 UTC
83d25c5 Updated debian changelog for version 0.19.0 28 October 2021, 11:15:10 UTC
e78e56d Update upstream source from tag 'debian/upstream/0.19.0' Update to upstream version '0.19.0' with Debian dir 25276af0f39eed8f7341a27f86fd12f2168c0212 28 October 2021, 11:15:10 UTC
ae3c3c6 New upstream version 0.19.0 28 October 2021, 11:15:09 UTC
50d7fd7 Add a new cli endpoint to schedule recurrent visits in Celery For each known visit type, we run a loop which: - monitors the size of the relevant celery queue - schedules more visits of the relevant type once the number of available slots goes over a given threshold (currently set to 5% of the max queue size). The scheduling of visits combines multiple scheduling policies, for now using static ratios set in the `POLICY_RATIOS` dict. We emit a warning if the ratio of origins fetched for each policy is skewed with respect to the original request (allowing, for now, manual adjustement of the ratios). The CLI endpoint spawns one thread for each visit type, which all handle connections to RabbitMQ and the scheduler backend separately. For now, we handle exceptions in the visit scheduling threads by (stupidly) respawning the relevant thread directly. We should probably improve this to give up after a specific number of tries. Co-authored-by: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> 28 October 2021, 11:06:56 UTC
0c7ef27 grab_next_visits: avoid time interval calculations in PostgreSQL When the database is in a non-UTC timezone with DST, and a `timestamptz - interval` calculation crosses a DST change, the result of the calculation can be one hour off from the expected value: PostgreSQL will vary the timestamp by the amount of days in the interval, and will keep the same (local) time, which will be offset by an hour because of the DST change. Doing the datetime +- timedelta calculations in Python instead of PostgreSQL avoids this caveat altogether. 27 October 2021, 13:45:09 UTC
ecc0e28 Restrict the click version to avoid conflict version with celery's Otherwise, in some edge case, like run in docker, the install fails on conflict. Related to P1205#8092 22 October 2021, 09:21:36 UTC
243a69f Add docstring to runner and listener modules Related to T3667 20 October 2021, 09:25:38 UTC
5b53196 Drop deprecated listener module It's been deprecated for enough time. Related to T3667 20 October 2021, 09:02:11 UTC
f15c510 scheduler: Deprecate unused main celery runner 20 October 2021, 08:31:28 UTC
7a60322 Updated backport on buster-swh from debian/0.18.2-1_swh1 (unstable-swh) 18 October 2021, 13:21:35 UTC
1502e1b Merge tag 'debian/0.18.2-1_swh1' into debian/buster-swh 18 October 2021, 13:21:34 UTC
628c955 Updated debian changelog for version 0.18.2 18 October 2021, 13:18:57 UTC
9b989a8 Update upstream source from tag 'debian/upstream/0.18.2' Update to upstream version '0.18.2' with Debian dir 064dff3f140dad4fe33f0893b48748460076b59a 18 October 2021, 13:18:54 UTC
4241b21 New upstream version 0.18.2 18 October 2021, 13:18:53 UTC
3aed688 Use swh_storage fixture for cli tests This actually fixes the debian build failure. Related to T3666 18 October 2021, 12:16:38 UTC
ea19f6e Updated debian changelog for version 0.18.1 15 October 2021, 13:53:38 UTC
22cdd27 New upstream version 0.18.1 15 October 2021, 13:53:35 UTC
34b60f4 Update upstream source from tag 'debian/upstream/0.18.1' Update to upstream version '0.18.1' with Debian dir 33b5f1325f7a752172f38a90c9be255f3bac2402 15 October 2021, 13:53:35 UTC
3aed7bf Return 0 slot if no more slots available in the queues This scenario happens with the loader oneshot for example. This loader deals with more than 1 type of origins to ingest in the same queue. So the computation of that function returned negative value [1]. Which is ultimately not possible to execute in sql [1]. This commits fixes that behavior. This also explicits that the function must return positive values in its docstring. [1] ``` ... psycopg2.errors.InvalidRowCountInLimitClause: LIMIT must not be negative ``` 15 October 2021, 13:22:52 UTC
fa9d328 Updated backport on buster-swh from debian/0.18.0-1_swh1 (unstable-swh) 02 September 2021, 09:37:34 UTC
db6bff1 Merge tag 'debian/0.18.0-1_swh1' into debian/buster-swh 02 September 2021, 09:37:34 UTC
1a70fd6 Updated debian changelog for version 0.18.0 02 September 2021, 09:35:32 UTC
22baf3f Update upstream source from tag 'debian/upstream/0.18.0' Update to upstream version '0.18.0' with Debian dir e98b6b91a4c915d0ad9f6ee1286273ed99ee3b5b 02 September 2021, 09:35:31 UTC
66bf492 New upstream version 0.18.0 02 September 2021, 09:35:31 UTC
ecc1400 runner: Improve help message on the task types flag. 02 September 2021, 09:15:36 UTC
63fdda0 send-to-celery: Add more options to allow scheduling of edge cases In the non optimal case, we may want to trigger specific case (not-yet enabled origins, origin from specific lister...). Related to T3350 27 August 2021, 11:26:38 UTC
7cc37fa Refine scheduling policy for origins with no known last update For origins that have never been visited, and for which we don't have a queue position yet, we want to visit them in the order they've been added. 26 August 2021, 14:49:37 UTC
2efad28 Add a swh scheduler origin send-to-celery subcommand The subcommand bypasses the legacy task-based mechanism to directly send new origin visits to celery 26 August 2021, 14:48:46 UTC
5e8007f Add table sampling option to grab_next_visits Running common operations on all git origins is pretty intense. Using table sampling gives us the opportunity to at least schedule some jobs in (decently small) time. 26 August 2021, 14:47:52 UTC
cc76a57 journal_client: Only upsert if we have something to upsert 26 August 2021, 09:44:14 UTC
ab15591 Updated backport on buster-swh from debian/0.17.1-1_swh1 (unstable-swh) 26 August 2021, 08:43:42 UTC
395f14f Merge tag 'debian/0.17.1-1_swh1' into debian/buster-swh 26 August 2021, 08:43:41 UTC
4053937 Updated debian changelog for version 0.17.1 26 August 2021, 08:41:41 UTC
c36a724 Update upstream source from tag 'debian/upstream/0.17.1' Update to upstream version '0.17.1' with Debian dir e98b6f8fc3c8547ef7148fd0b9915f432584ab81 26 August 2021, 08:41:40 UTC
d04dbb3 New upstream version 0.17.1 26 August 2021, 08:41:40 UTC
506f78c journal_client: Ensure queue position does not overflow Queue positions are date and the current next_position_offset used to compute the new queue position was not bounded. This has the side-effect of making overflow error. This commit adapts the journal client computations to limit such next_position_offset to 10. This value was chosen because above that exponent the dates overflow (and we are way in the future already). Related to T3502 26 August 2021, 08:24:11 UTC
28ae1d8 Replace index-fossology-license-for-range with index-fossology-license-for-partition We changed the task name/interface a while ago 18 August 2021, 09:20:25 UTC
ec56447 Updated backport on buster-swh from debian/0.17.0-1_swh1 (unstable-swh) 06 August 2021, 09:14:02 UTC
ef76af4 Merge tag 'debian/0.17.0-1_swh1' into debian/buster-swh 06 August 2021, 09:14:01 UTC
fa762c5 Updated debian changelog for version 0.17.0 06 August 2021, 09:11:54 UTC
416b2c5 Update upstream source from tag 'debian/upstream/0.17.0' Update to upstream version '0.17.0' with Debian dir c55959218dbe52af5f47cb3541419dfb69d77945 06 August 2021, 09:11:54 UTC
3c61059 New upstream version 0.17.0 06 August 2021, 09:11:53 UTC
8281e35 journal_client: Disable origins when too many visited attempts failed This disable origins for either failed or not found attempts 3 times in a row. It's not definitive though as it's the lister's responsibility to activate back origins if they get listed again. Related to T2345 03 August 2021, 11:56:32 UTC
1bcf84d Add a successive_visits counter to origin visit stats This maintains the number of successive visits resulting in the same status. This will help implementing disabling of too many successive failed or not_found visits for a given origin. Related to T2345 03 August 2021, 10:49:45 UTC
4fa29fe journal_client: Update get_last_status docstring Related to T2345 30 July 2021, 13:35:17 UTC
3b929d0 journal_client: Refactor by inlining the update_position_offset This is no longer required as it's called once. Related to T2345 30 July 2021, 13:23:14 UTC
87e66fa Only record last_visited and last_successful in origin_visit_stats After using this schema for a while, all queries can be implemented in terms of these two timestamps, instead of the four original last_eventful, last_uneventful, last_failed and last_notfound timestamps. This ends up simplifying the logic within the journal client, as well as that of the grab_next_visits query builder. To make this change work, we also stop considering out of order messages altogether in journal_client. This welcome simplification is an accuracy tradeoff that is explained in the updated documentation of the journal client: .. [1] Ignoring out of order messages makes the initialization of the origin_visit_status table (from a full journal) less deterministic: only the `last_visit`, `last_visit_state` and `last_successful` fields are guaranteed to be exact, the `next_position_offset` field is a best effort estimate (which should converge once the client has run for a while on in-order messages). 23 July 2021, 09:56:32 UTC
3ca0d65 test_journal_client: Unify test assertion like the rest Related to D5917 23 July 2021, 07:22:46 UTC
8cf2238 test: Refactor assert_visit_stats_ok to ignore_fields This simplifies and unifies properly the utility test function to compare visit stats. 23 July 2021, 07:18:20 UTC
d58776a Introduce new scheduling policy to grab origins without last update This is in charge of scheduling origins without last update. This also updates the global queue position so the journal client can initialize correctly the next position per origin and visit type. Related to T2345 22 July 2021, 10:23:44 UTC
825e8cf grab_next_visits: make the handling of CTEs more modular This allows us to insert extra CTEs if a scheduling policy needs it. 22 July 2021, 10:19:42 UTC
8c4ae9f journal_client: Compute next position for origin visit For origin without any last_update information [1], the journal client is now also in charge of moving their next position in the queue for rescheduling. Depending on their status, the next position offset and next_visit_queue_position are updated after each visit completes: - if the visit has failed, increase the next visit target by the minimal visit interval (to take into account transient loading issues) - if the visit is successful, and records some changes, decrease the visit interval index by 2 (visit the origin *way* more often). - if the visit is successful, and records no changes, increase the visit interval index by 1 (visit the origin less often). We then set the next visit target to its current value + the new visit interval multiplied by a random fudge factor (picked in the -/+ 10% range). The fudge factor allows the visits to spread out, avoiding "bursts" of loaded origins e.g. when a number of origins from a single hoster are processed at once. Note that the computations happen for all origins for simplicity and code maintenance but it will only be used by a new soon-to-be scheduling policy. [1] Lister cannot provide it for some reason. 06 July 2021, 12:35:13 UTC
cb1edf1 Introduce storage for the recurrent visit scheduler queue position 01 July 2021, 08:36:44 UTC
ec6e69f Start handling of recurrent loading tasks in scheduler This deals first and foremost with the next_position_offset update done by the scheduler journal client. 01 July 2021, 08:36:44 UTC
c486b28 journal_client: Explicit docstring 29 June 2021, 13:16:15 UTC
98f99b9 journal_client: Only check last_* fields for some permutation tests In a future commit, we will add new fields whose values will be permutation dependent. 23 June 2021, 15:02:34 UTC
1006f0a journal_client: Auto-generate the empty object from model fields This will help us when adding new fields to the table. 23 June 2021, 14:54:34 UTC
6400cc2 backend: Auto-generate origin visit stats upsert query This will help us when adding new fields to the table. 23 June 2021, 14:54:34 UTC
3762c34 cli/task: Ensure cli output is always in the same order 23 June 2021, 14:54:34 UTC
ed81870 Add a specific cooldown for notfound origins This allows us to avoid repeating visits on them, until a next pass of the lister can mark them as disabled. 23 June 2021, 09:13:00 UTC
651ddcc Add a (longer) specific cooldown for failed origin visits 23 June 2021, 09:13:00 UTC
ce8608d Make the origin visit scheduling cooldown configurable 23 June 2021, 09:13:00 UTC
3b5a8d7 Updated backport on buster-swh from debian/0.16.0-1_swh1 (unstable-swh) 22 June 2021, 15:41:47 UTC
18bb587 Merge tag 'debian/0.16.0-1_swh1' into debian/buster-swh 22 June 2021, 15:41:46 UTC
61259ee Updated debian changelog for version 0.16.0 22 June 2021, 15:39:45 UTC
4ed7ce8 Update upstream source from tag 'debian/upstream/0.16.0' Update to upstream version '0.16.0' with Debian dir 76fdfbde021aeb32390560c3684c46131a969de1 22 June 2021, 15:39:44 UTC
e63d38c New upstream version 0.16.0 22 June 2021, 15:39:43 UTC
7f51f27 interface: Add get_listers method Add new method to scheduler interface returning the full list of listers registered in the database. Related to T3127 22 June 2021, 12:36:08 UTC
9e1b414 Drop duplicate docstring from backend 21 June 2021, 13:46:12 UTC
919334e Updated backport on buster-swh from debian/0.15.0-1_swh1 (unstable-swh) 10 June 2021, 14:51:04 UTC
ab127f6 Merge tag 'debian/0.15.0-1_swh1' into debian/buster-swh 10 June 2021, 14:51:04 UTC
4e410ba Updated debian changelog for version 0.15.0 10 June 2021, 14:48:52 UTC
79e2023 New upstream version 0.15.0 10 June 2021, 14:48:51 UTC
d93dbf7 Update upstream source from tag 'debian/upstream/0.15.0' Update to upstream version '0.15.0' with Debian dir d49b2d7bf8b3e4b7635110179f0d613aeda1d740 10 June 2021, 14:48:51 UTC
c7707b5 runner: Separate scheduling tasks with and without priority concern In effect, this will allow to run 2 runners: - one for recurring tasks - one for the save code now This should decrease the probability of the scheduling tasks for the save code now to be stuck behind the main scheduler runner. Related to T3367 10 June 2021, 12:55:04 UTC
21c4279 Refactor and extract a get_available_slots utility This adds coverage as well. This will be needed for subsidiary diffs. Related to T3367 10 June 2021, 10:15:22 UTC
9d2618d Add typing stubs dependencies for mypy>0.900 This also explicits missing dependencies 09 June 2021, 12:13:36 UTC
9f7ab8f pytest_plugin: Explicitly set hostname in broker_url for celery TestApp Since the release of kombu 5.1.0, a warning is now issued when a hostname is not set in the broker_url config value of a celery app. That change makes the test_celery_monitor_ping test fails due to that new unexpected warning. So explicitly add localhost hostname in the broker_url value of the celery TestApp config. 25 May 2021, 11:43:03 UTC
479b38d Updated backport on buster-swh from debian/0.14.2-1_swh1 (unstable-swh) 06 May 2021, 15:15:12 UTC
0035687 Merge tag 'debian/0.14.2-1_swh1' into debian/buster-swh 06 May 2021, 15:15:12 UTC
04db822 Updated debian changelog for version 0.14.2 06 May 2021, 15:13:11 UTC
ebe1d6d Update upstream source from tag 'debian/upstream/0.14.2' Update to upstream version '0.14.2' with Debian dir 50bf236b540cecee0d42d1299ec4f4c9131e450a 06 May 2021, 15:13:10 UTC
628a203 New upstream version 0.14.2 06 May 2021, 15:13:09 UTC
fe9d949 Fix flaky test_grab_ready_* tests 06 May 2021, 14:20:57 UTC
back to top