| 25f03c2 | Jenkins for Software Heritage | 13 December 2019, 10:54:30 UTC | New upstream version 0.0.65 | 13 December 2019, 10:54:30 UTC |
| ee162fe | Nicolas Dandrimont | 13 December 2019, 10:29:22 UTC | Use a btree of (task_type, md5(arguments)) to match task arguments The former index on hash(arguments->'args') has lost relevance as about half the tasks (the ones for the loader) have the same value (an empty list) for this field. This index is more universal, faster, and also easier to convince the planner of using. If we want more specific indexes (e.g. on specific keyword arguments) we'll be able to add that separately. | 13 December 2019, 10:32:33 UTC |
| 0b04220 | David Douard | 12 December 2019, 16:37:39 UTC | Remove the creation of the 'load-deposit' task type it's now managed by swh-loader-core directly. | 12 December 2019, 16:37:39 UTC |
| 4071d71 | David Douard | 12 December 2019, 11:27:13 UTC | Make --status option of 'swh scheduler task list' a click.Choice | 12 December 2019, 11:27:13 UTC |
| a18f562 | David Douard | 04 December 2019, 09:55:25 UTC | celery: add 2 statsd probes for the runner and listener - runner: counting the number of scheduled tasks, - listener: counting the number of processed events. | 04 December 2019, 15:36:19 UTC |
| f206076 | David Douard | 04 December 2019, 09:25:37 UTC | celery: make SWHTask send start/end of execution statsd gauges with timestamps Closes T2119. | 04 December 2019, 09:28:26 UTC |
| 08243bb | David Douard | 04 December 2019, 09:23:15 UTC | tests: fix celery_task's test_multiping kwargs were not passed correctly. Also add a test_ping_with_kw test. | 04 December 2019, 09:25:05 UTC |
| 8c1e051 | Antoine R. Dumont (@ardumont) | 26 November 2019, 11:29:47 UTC | scheduler.updater: Remove dead code | 26 November 2019, 11:29:47 UTC |
| 95940a8 | Nicolas Dandrimont | 21 November 2019, 12:57:25 UTC | Migrate tox.ini to extras = xxx instead of deps = .[testing] | 21 November 2019, 12:57:25 UTC |
| 7c40132 | Nicolas Dandrimont | 21 November 2019, 12:50:03 UTC | Merge tox test environment configurations | 21 November 2019, 12:54:47 UTC |
| 101a131 | David Douard | 21 November 2019, 12:50:21 UTC | Add a pre-commit config file | 21 November 2019, 12:50:21 UTC |
| 104fee0 | Nicolas Dandrimont | 21 November 2019, 11:04:54 UTC | Drop version constraint on pytest < 4 | 21 November 2019, 11:04:54 UTC |
| 56e4a12 | Nicolas Dandrimont | 20 November 2019, 18:56:38 UTC | Include all requirements in MANIFEST.in | 20 November 2019, 18:56:38 UTC |
| 64b771c | Jenkins for Software Heritage | 20 November 2019, 13:29:36 UTC | New upstream version 0.0.64 | 20 November 2019, 13:29:36 UTC |
| c973ec0 | Antoine R. Dumont (@ardumont) | 19 November 2019, 14:12:27 UTC | req-swh*: Remove old package loader backend names Related to T1389 T2098 Related to D2306 D2305 D2304 | 19 November 2019, 14:12:27 UTC |
| 4cb289c | Jenkins for Software Heritage | 19 November 2019, 13:14:58 UTC | New upstream version 0.0.63 | 19 November 2019, 13:14:58 UTC |
| 9358572 | Antoine R. Dumont (@ardumont) | 15 November 2019, 15:02:12 UTC | swh.scheduler.cli: Add `swh scheduler task-type register` cli This allos registering of worker's task types to the scheduler through setuptools' mechanism. | 19 November 2019, 11:11:33 UTC |
| 8ec34fe | Nicolas Dandrimont | 23 October 2019, 08:40:37 UTC | Remove collect_ignore from conftest.py This got solved when we started using the shared_task decorator instead of instantiating our own app. | 23 October 2019, 08:40:42 UTC |
| 4df2406 | Nicolas Dandrimont | 18 October 2019, 15:54:04 UTC | Use the shared_task decorator instead of binding to a specific celery app | 23 October 2019, 08:32:26 UTC |
| ecf38eb | David Douard | 18 October 2019, 14:37:28 UTC | celery/tests: mostly revert e770eb30 to fix celery app initialization in tests This revision did fix tests for the scheduler itself, but broke all other tests of scheduler dependent swh packages. In this fix, we ensure we override the `app` in swh.scheduler.celery_backend.config, since it is used by all celery task declarations (via the @app.task() decorator). | 18 October 2019, 14:37:28 UTC |
| 9485932 | Jenkins for Software Heritage | 18 October 2019, 11:46:25 UTC | New upstream version 0.0.62 | 18 October 2019, 11:46:25 UTC |
| 787c7a9 | Antoine R. Dumont (@ardumont) | 18 October 2019, 09:49:39 UTC | celery_backend.config: Make JournalHandler import optional swh-core no longer comes with JournalHandler by default. | 18 October 2019, 11:33:25 UTC |
| c2a020d | David Douard | 16 October 2019, 08:50:06 UTC | tests: rewrite tests using pytest and the new rpc fixtures from swh.core | 16 October 2019, 11:20:42 UTC |
| a7e15bf | David Douard | 16 October 2019, 08:46:11 UTC | add a new get_priority_ratios endpoint to the scheduler this is necessary to make it much easier to write tests so they do not need to execute SQL statements, which makes possible to run exactly the same tests with the SchedulerBackend as the RemoteScheduler one (see the following revision). | 16 October 2019, 11:20:42 UTC |
| c2ccf46 | David Douard | 16 October 2019, 08:43:03 UTC | updater/tests: rewrite updater's tests as pytest functions The way the scheduler_db and updater_db fixtures are built is not very straighforward nor satisfying, but it works. | 16 October 2019, 11:20:42 UTC |
| 37b909e | David Douard | 16 October 2019, 08:40:28 UTC | conftest: simplify the swh_scheduler() fixture simply use the postgresql.dsn as connection string. | 16 October 2019, 08:42:04 UTC |
| e770eb3 | Antoine R. Dumont (@ardumont) | 10 October 2019, 13:14:25 UTC | tests: Explicit registering test tasks step for the swh_app Prior to this commit, the celery "app" import changed. Making the runtime application load prior to the tests "swh_app". In effect, making the tasks not being consumed by workers. This explicitely forces the tests tasks registering to "swh_app". In effect clarifying code and fixing the current tests. Related D2082 Related 8eafc70 | 10 October 2019, 13:26:54 UTC |
| 349d23e | Antoine R. Dumont (@ardumont) | 10 October 2019, 10:05:38 UTC | scheduler: Use directly the package's server module to start server Related D2109 Related D2110 | 10 October 2019, 10:05:38 UTC |
| 8eafc70 | Antoine R. Dumont (@ardumont) | 10 October 2019, 09:51:27 UTC | tox.ini: Use tests installed files instead of working directory Related D2082 | 10 October 2019, 09:51:27 UTC |
| 7f9b26f | Jenkins for Software Heritage | 07 October 2019, 14:37:36 UTC | New upstream version 0.0.61 | 07 October 2019, 14:37:36 UTC |
| 5955c8d | Antoine R. Dumont (@ardumont) | 02 October 2019, 04:53:21 UTC | celery_backend/config: Fix wrong statement Dict's get method does not take keyword argument. Related D2033#47672 | 02 October 2019, 04:56:13 UTC |
| 2984a95 | Jenkins for Software Heritage | 01 October 2019, 11:22:42 UTC | New upstream version 0.0.60 | 01 October 2019, 11:22:42 UTC |
| 06137f0 | Stefano Zacchiroli | 01 October 2019, 11:08:23 UTC | tox: anticipate mypy run to just after flake8 | 01 October 2019, 11:08:23 UTC |
| c78b846 | Stefano Zacchiroli | 27 September 2019, 08:38:31 UTC | init.py: switch to documented way of extending path make mypy 0.730 pass cleanly again | 27 September 2019, 08:38:31 UTC |
| 1a691b5 | Stefano Zacchiroli | 24 September 2019, 11:55:04 UTC | tox.ini: add mypy section | 24 September 2019, 11:55:04 UTC |
| c4fa353 | Stefano Zacchiroli | 24 September 2019, 11:43:34 UTC | typing: minimal changes to make a no-op mypy run pass | 24 September 2019, 11:44:50 UTC |
| 3cd5697 | Stefano Zacchiroli | 24 September 2019, 11:42:23 UTC | fix typo in docstring and sample file name courtesy of codespell | 24 September 2019, 11:42:23 UTC |
| 0a4fd7a | Stefano Zacchiroli | 17 September 2019, 15:49:06 UTC | admin CLI: drop obsolete backward compatibility aliases These have been around for a while and are no longer advertised in the --help output. | 17 September 2019, 15:51:03 UTC |
| 6a054b9 | Stefano Zacchiroli | 16 September 2019, 15:13:55 UTC | click "required" param wants bool, not int courtesy of mypy | 16 September 2019, 15:13:55 UTC |
| fcf8e5f | Jenkins for Software Heritage | 04 September 2019, 14:11:47 UTC | New upstream version 0.0.59 | 04 September 2019, 14:11:47 UTC |
| a64ac3f | David Douard | 04 September 2019, 13:53:19 UTC | sql: stop inserting lister tasks at scheduler's creation time This is now the job of the swh-lister to ensure task-type entries are created in the scheduler (in swh-lister >= 0.0.34). | 04 September 2019, 13:53:19 UTC |
| bf51443 | Jenkins for Software Heritage | 03 September 2019, 08:28:18 UTC | New upstream version 0.0.58 | 03 September 2019, 08:28:18 UTC |
| 9c2b329 | Antoine R. Dumont (@ardumont) | 03 September 2019, 08:18:30 UTC | 50-swh-data: Add load-tar task-type This is needed to be in sync with the docker-dev environment. | 03 September 2019, 08:18:30 UTC |
| c3fae15 | David Douard | 13 May 2019, 11:57:12 UTC | celery: auto add tasks declared in the swh.workers entry point in task_modules allows to declare worker tasks in a 'swh.workers' entry point. This later is expected to be a callable which returns a dict which 'task_modules' key is a list of python module names where celery tasks are defined. | 02 September 2019, 11:33:28 UTC |
| e1b216d | Antoine Lambert | 23 July 2019, 12:38:29 UTC | api/client: use RPCClient instead of deprecated SWHRemoteAPI | 23 July 2019, 12:38:29 UTC |
| a43b3c7 | Valentin Lorentz | 12 July 2019, 10:18:15 UTC | Make schedule_origins use origin urls instead of ids in task arguments. | 19 July 2019, 14:35:25 UTC |
| cc1b590 | Stefano Zacchiroli | 11 July 2019, 14:29:50 UTC | add code of conduct document | 11 July 2019, 14:29:50 UTC |
| f63274a | David Douard | 10 July 2019, 13:23:18 UTC | docs: very beginning of a practical documentation on the scheduler | 10 July 2019, 14:01:37 UTC |
| 9b7407f | David Douard | 10 July 2019, 13:58:47 UTC | updater: small refactoring to please codespell | 10 July 2019, 13:58:47 UTC |
| 92b81f3 | David Douard | 10 July 2019, 13:52:26 UTC | Add a pre-commit config file | 10 July 2019, 13:52:26 UTC |
| 2044c07 | Antoine R. Dumont (@ardumont) | 28 June 2019, 15:52:57 UTC | data: Insert new cgit instance lister task Related T1835 | 28 June 2019, 15:52:57 UTC |
| 09d9c84 | Stefano Zacchiroli | 28 June 2019, 07:42:00 UTC | CONTRIBUTORS: add Ishan Bhanuka | 28 June 2019, 07:42:00 UTC |
| 0ee2895 | Jenkins for Software Heritage | 26 June 2019, 13:05:18 UTC | New upstream version 0.0.57 | 26 June 2019, 13:05:18 UTC |
| 09e7245 | Ishan Bhanuka | 15 June 2019, 11:56:53 UTC | Added comments to few columns in dbversion, task and task_run Change double quoted comments to single quoted comments To keep consistent comment format and to prevent syntax exception on jenkins build Make changes as per review Made changes as per review | 19 June 2019, 16:13:09 UTC |
| 237e8c5 | David Douard | 17 May 2019, 15:06:34 UTC | cli: rename a admin cli commands use more imperative-like command names. Aliases are used to keep old commands working. Also add an explicit deprecation message in swh-scheduler's help message. | 13 June 2019, 07:29:09 UTC |
| da77f89 | Antoine R. Dumont (@ardumont) | 12 June 2019, 13:22:32 UTC | 50-swh-data: Update new lister task types Related T1797 | 12 June 2019, 13:22:32 UTC |
| 46223d8 | Antoine R. Dumont (@ardumont) | 24 May 2019, 10:21:43 UTC | scheduler.cli.task: Fix wrong import | 24 May 2019, 10:21:43 UTC |
| aa70df4 | Antoine R. Dumont (@ardumont) | 24 May 2019, 08:11:37 UTC | Drop backward compatible names The migration is now done. No longer needs this adaptation code. Related T1508 | 24 May 2019, 08:11:37 UTC |
| 63eeabf | Antoine R. Dumont (@ardumont) | 23 May 2019, 11:27:41 UTC | data: Add missing swh-vault-batch-cooking/cook-vault-bundle-type Related T1508 | 23 May 2019, 11:27:41 UTC |
| 766dae9 | Antoine R. Dumont (@ardumont) | 23 May 2019, 11:22:59 UTC | 50-swh-data: Fix inverted select/insert in backward sql function Prior to this, the select wanted to retrieve the new name to insert the old one Ending up doing a noop. Related T1508 | 23 May 2019, 11:23:28 UTC |
| d6fce0d | Antoine Lambert | 14 May 2019, 15:53:58 UTC | swh-scheduler: Use new task names Related T1508 | 21 May 2019, 11:34:25 UTC |
| d6dc50f | David Douard | 13 May 2019, 11:55:31 UTC | tests: update to swh.core 0.0.61 in which (db|api)_testing.py files moved. | 17 May 2019, 15:09:41 UTC |
| 52f8c06 | Valentin Lorentz | 17 May 2019, 11:32:20 UTC | Update import of swh.core.api.tests.server_testing. | 17 May 2019, 11:32:20 UTC |
| 45aa02c | Antoine Lambert | 14 May 2019, 14:19:00 UTC | cli: Fix swh-scheduler command (missing log_level in context) | 15 May 2019, 11:57:22 UTC |
| c20005a | Jenkins for Software Heritage | 14 May 2019, 12:40:08 UTC | New upstream version 0.0.56 | 14 May 2019, 12:40:08 UTC |
| ac1b8db | Nicolas Dandrimont | 07 May 2019, 16:16:20 UTC | listener: Release the db object after using it This is the contract that get_db/put_db is supposed to conform to. | 10 May 2019, 13:52:36 UTC |
| edaf202 | David Douard | 07 May 2019, 08:52:06 UTC | cli: split the main cli/__init__.py file in 4 parts extract: - the 'task' group (and its related commands) in cli/task.py - the 'task_type' group in cli/task_type.py - remaining commands in cli/admin.py | 07 May 2019, 08:52:06 UTC |
| 3f88069 | David Douard | 06 May 2019, 13:52:01 UTC | README: remove the deprecated tests section | 06 May 2019, 13:52:01 UTC |
| 84c0bd7 | David Douard | 02 May 2019, 10:07:31 UTC | cli: make swh-scheduler a subcommand of swh.core's main "swh" command the old "swh-scheduler" still exists for now, and is mostly backward compatible but the --log-level option (which is defined in the "swh" main command group). Unit tests have been adapted and should pass ok. | 06 May 2019, 13:48:31 UTC |
| eb42639 | David Douard | 19 April 2019, 14:10:55 UTC | cli: make it a subpackage | 06 May 2019, 13:47:40 UTC |
| fd6c7c1 | Jenkins for Software Heritage | 06 May 2019, 09:54:50 UTC | New upstream version 0.0.55 | 06 May 2019, 09:54:50 UTC |
| d338c76 | Antoine Lambert | 02 May 2019, 16:00:55 UTC | sql/swh-data: Update scheduler task names but keep backward compatibility Related T1508 | 03 May 2019, 08:50:30 UTC |
| 72eaa42 | Nicolas Dandrimont | 12 April 2019, 10:03:09 UTC | Update coverage gitignore | 12 April 2019, 10:03:09 UTC |
| 7bb55c9 | Jenkins for Software Heritage | 11 April 2019, 09:40:13 UTC | New upstream version 0.0.54 | 11 April 2019, 09:40:13 UTC |
| 24e6a0a | Nicolas Dandrimont | 10 April 2019, 14:15:11 UTC | cli_utils: Use yaml.safe_load instead of yaml.load | 10 April 2019, 18:36:47 UTC |
| b644006 | Valentin Lorentz | 10 April 2019, 13:39:49 UTC | Fix support of latest versions of swh-core and psycopg2. | 10 April 2019, 16:55:20 UTC |
| 4b0e952 | Antoine Lambert | 10 April 2019, 14:12:25 UTC | sql/data: Add npm related task types Related T1629 Related T1508 | 10 April 2019, 16:25:53 UTC |
| 356b780 | Jenkins for Software Heritage | 04 April 2019, 14:55:19 UTC | New upstream version 0.0.53 | 04 April 2019, 14:55:19 UTC |
| aaa9ee4 | Antoine Lambert | 04 April 2019, 14:45:14 UTC | SchedulerBackend.create_tasks: Remove duplicated task key This issue was spotted after the upgrade of psycopg2 to version 2.8 | 04 April 2019, 14:45:14 UTC |
| 030fe02 | Jenkins for Software Heritage | 03 April 2019, 08:59:00 UTC | New upstream version 0.0.52 | 03 April 2019, 08:59:00 UTC |
| 38cafb9 | Nicolas Dandrimont | 02 April 2019, 16:04:19 UTC | Make result_serializer json We kind-of-need that to test that we'll be able to understand results in production (even though we're sending results by hand, not through this event) and it works around https://github.com/celery/celery/issues/5366 in Celery 4.3. | 02 April 2019, 16:05:33 UTC |
| b9903d7 | Antoine Lambert | 01 April 2019, 14:22:18 UTC | sql/Makefile: Fix scheduler db creation | 01 April 2019, 14:22:18 UTC |
| 6e69026 | Jenkins for Software Heritage | 22 March 2019, 11:16:06 UTC | New upstream version 0.0.51 | 22 March 2019, 11:16:06 UTC |
| c2b1793 | Antoine R. Dumont (@ardumont) | 21 March 2019, 10:43:30 UTC | requirements.txt: Remove kombu dependency This actually conflicts with the latest celery version. We should not depend on it as this is drawn by celery already. | 21 March 2019, 10:46:17 UTC |
| 4ee93d3 | David Douard | 06 March 2019, 11:32:55 UTC | API: use default's APIError exception instead of the SchedulerAPIError this later does not brings anything over the default's one. This requires swh.core >= 0.0.56 | 18 March 2019, 16:02:14 UTC |
| a49761d | Jenkins for Software Heritage | 15 March 2019, 17:16:02 UTC | New upstream version 0.0.50 | 15 March 2019, 17:16:02 UTC |
| 693a147 | Nicolas Dandrimont | 15 March 2019, 13:25:15 UTC | Remove info-level logs about already acknowledged messages Celery voluntarily pools messages to reduce traffic, so it's expected that some messages end up here several times. No need to spam logs about it. | 15 March 2019, 13:25:15 UTC |
| 763dbda | Valentin Lorentz | 14 March 2019, 12:15:29 UTC | Ignore INFO messages when testing CLI output. | 14 March 2019, 15:36:08 UTC |
| 1122ee1 | Valentin Lorentz | 13 March 2019, 14:20:22 UTC | Add a 'task schedule_origins' command to run a task on all origins. | 14 March 2019, 15:36:08 UTC |
| 6b795e3 | Nicolas Dandrimont | 08 March 2019, 16:37:01 UTC | Introduce SWH_LOG_TARGET to allow switching between console and journal The current state of the celery worker/utility logging setup makes us log twice: the default value of log_console can't be overridden by the cli flag in celery workers. The environment variable can be set to journal when running as a systemd service; this shuts down the console logger, and allows the metadata added to log messages to pass through. | 13 March 2019, 14:05:24 UTC |
| 6dbfd06 | Valentin Lorentz | 13 March 2019, 09:17:27 UTC | Remove ghtorrent/fake.py, it's not used anymore. | 13 March 2019, 09:17:27 UTC |
| e6ee2a7 | Valentin Lorentz | 01 March 2019, 13:27:34 UTC | Add tests for 'swh-scheduler task list-pending' and 'swh-scheduler task list'. | 13 March 2019, 08:58:19 UTC |
| da290c2 | Valentin Lorentz | 13 March 2019, 08:57:01 UTC | Add support for non-string options when scheduling tasks. This also fixes the pretty-printing of tasks, which was ambiguous (`42` and `"42"` where both printed as `42`). | 13 March 2019, 08:57:01 UTC |
| b0aea8c | David Douard | 06 March 2019, 11:44:51 UTC | tests: enforce LC_ALL since test_cli depends on it | 06 March 2019, 11:45:38 UTC |
| 85b858b | Jenkins for Software Heritage | 03 March 2019, 07:55:40 UTC | New upstream version 0.0.49 | 03 March 2019, 07:55:40 UTC |
| dc6afe4 | Valentin Lorentz | 01 March 2019, 10:16:24 UTC | Revert "Hack to fix the ImportError in updater/ghtorrent/fake.py." This reverts commit 3939a41d73bdef94913a6e8c3aa212532d281463. Wasn't supposed to be committed yet. | 01 March 2019, 10:16:24 UTC |
| 5499400 | Valentin Lorentz | 26 February 2019, 12:51:52 UTC | Move option parsing to its own (tested) function. | 01 March 2019, 10:14:06 UTC |
| 3939a41 | Valentin Lorentz | 28 February 2019, 17:09:56 UTC | Hack to fix the ImportError in updater/ghtorrent/fake.py. Sphinx complains about it. | 28 February 2019, 17:09:56 UTC |
| f698b01 | Valentin Lorentz | 27 February 2019, 16:22:45 UTC | Move parts of 30-swh-schema.sql to 40-swh-func.sql and 60-swh-indexes.sql. For uniformity with other SWH packages. | 28 February 2019, 13:42:06 UTC |
| 7ddfdf4 | Antoine Lambert | 27 February 2019, 16:51:40 UTC | celery_backend.listener: Fix debug output format string | 27 February 2019, 16:51:40 UTC |
