| 3a4f062 | Jenkins for Software Heritage | 16 January 2019, 12:42:36 UTC | New upstream version 0.0.39 | 16 January 2019, 12:42:36 UTC |
| caaa44b | David Douard | 16 January 2019, 08:56:32 UTC | Revert "tox: pifpaf is not needed any more" This reverts commit f267f454fb91b421994f4ceabb50baa70982b76d. Pushed by mistake. | 16 January 2019, 08:56:32 UTC |
| f267f45 | David Douard | 15 January 2019, 17:11:48 UTC | tox: pifpaf is not needed any more | 15 January 2019, 17:11:48 UTC |
| 99bf79e | David Douard | 15 January 2019, 17:06:41 UTC | enforce dep on pytest<4 see https://github.com/pytest-dev/pytest/issues/4641 It should be possible to let this constraint go when celery 4.3 is out | 15 January 2019, 17:11:37 UTC |
| e1c25d2 | David Douard | 15 January 2019, 09:17:17 UTC | Replace task tests (in test_task.py) with pytest-based ones also remove now useless celery_testing.py and scheduler_testing.py files. | 15 January 2019, 16:39:32 UTC |
| e741c93 | David Douard | 15 January 2019, 08:59:30 UTC | Add a 'swh_scheduler' pytest fixture that creates a swh scheduler usable for tests this uses the pytest-postgresql package to manage the database life cycle. | 15 January 2019, 16:39:10 UTC |
| 9725365 | David Douard | 14 January 2019, 09:24:41 UTC | Configure the celery result backend (as 'rpc://') by default the result backend must be configured to be able to save (celery) group results (as it is now the case for lister tasks for example). | 15 January 2019, 14:02:06 UTC |
| 6cb1813 | David Douard | 14 January 2019, 09:23:07 UTC | Add a few debug statements in cli | 15 January 2019, 14:02:06 UTC |
| c0f9320 | David Douard | 14 January 2019, 09:23:41 UTC | Add a new 'task respawn' command that allows to respawn any task immediately (or any later date). | 15 January 2019, 14:02:01 UTC |
| cafcb46 | David Douard | 14 January 2019, 09:03:12 UTC | Add a 'next_run' argument to the SchedulerBackend.set_status_tasks method so this method can be used to respawn a task immediately. | 15 January 2019, 13:57:44 UTC |
| 1aadc10 | David Douard | 14 January 2019, 09:12:21 UTC | Add a 'task list' cli command to list tasks with search criterions and not only pending tasks. | 15 January 2019, 13:57:39 UTC |
| b3fd48e | David Douard | 14 January 2019, 09:07:12 UTC | Add a 'full' flag argument to the pretty_print_task function to display also the status and priority fields. | 15 January 2019, 13:49:49 UTC |
| e0f224e | David Douard | 14 January 2019, 09:05:14 UTC | Add a SchedulerBackend.search_tasks method | 15 January 2019, 09:20:22 UTC |
| 2dbdb6c | David Douard | 14 January 2019, 09:21:00 UTC | Add a new 'task-type add' cli command | 15 January 2019, 09:20:22 UTC |
| c3bb48d | David Douard | 14 January 2019, 08:59:38 UTC | Make SchedulerBackend.create_task_type work with only a subset of keys Not all keys are mandatory, so do not expect all the possible keys for the task_type table to be given when calling create_task_type(). Note: no validation is made whether the given set of keys fullfill the table constraints. | 15 January 2019, 09:20:22 UTC |
| 4c75330 | David Douard | 14 January 2019, 09:10:11 UTC | Add a new task-type cli command group and move the task_type listing in there | 14 January 2019, 10:38:19 UTC |
| d658f93 | David Douard | 14 January 2019, 08:56:34 UTC | Add a few debug statements in swh.sceduler.backend | 14 January 2019, 10:31:14 UTC |
| 6bf1110 | David Douard | 14 January 2019, 08:50:43 UTC | Fix swh.scheduler.compute_nb_tasks_from function to return integers instead of float numbers. | 14 January 2019, 10:31:14 UTC |
| 996e905 | David Douard | 10 January 2019, 14:41:08 UTC | Add a --priority cmdline option to the 'swh-scheduler task add' tool | 14 January 2019, 10:31:09 UTC |
| 9f33c05 | David Douard | 09 January 2019, 17:55:48 UTC | Make the SWHTask class the default base class for tasks and add some kind of unit tests. | 10 January 2019, 14:43:40 UTC |
| 20be147 | David Douard | 09 January 2019, 16:37:19 UTC | Kill swh.scheduler.api.server.launch() in favor of the 'swh-scheduler api-server' command. | 10 January 2019, 14:43:40 UTC |
| 5c02d51 | David Douard | 09 January 2019, 15:41:32 UTC | Add a logging statement in the runner cli command also log an eventual exception instead of crashing the runner service. | 10 January 2019, 14:43:39 UTC |
| 292e5dc | David Douard | 07 January 2019, 15:53:15 UTC | Kill the CustomCelery class use functions instead of methods. This is required to be able to use celery pytest fixtures so one can really test celery tasks (especially when a task spawns sub tasks). one (get_queue_lenth) of the 3 methods has been added as (monkeypatched) method on the Celery class for the sake of bw compat, but this should really be removed as well as soon as possible (seems only used in swh-archiver). | 10 January 2019, 14:43:39 UTC |
| 96ad58b | David Douard | 07 January 2019, 13:26:00 UTC | Move logging configuration into the cli group function so that logging level config can be consistently set for all swh-scheduler commands. | 10 January 2019, 14:43:39 UTC |
| 69f1759 | David Douard | 07 January 2019, 13:19:54 UTC | Move the scheduler verification from the main cli group definition to subcommands since it's actually the responsibility of each subcommand to decide whether it can run without a properly configured scheduler instance. This is also required so the user can run: swh-scheduler subcommand --help even with a non-properly configured scheduler. | 10 January 2019, 14:43:39 UTC |
| df075ba | David Douard | 07 January 2019, 10:34:49 UTC | Add 3 cli commands: runner, listener and api-server These commands do what they say, ie. start a runner, listener or API server process. Note that processes are not daemonized and run in front. Typically used as: swh-scheduler --cls local --database postgresql:///?service=swh api-server --host 127.0.0.1 --port 5008 swh-scheduler --cls remote --url http://127.0.0.1:5008 runner --period 10 swh-scheduler --cls remote --url http://127.0.0.1:5008 listener | 10 January 2019, 14:43:39 UTC |
| 6eb80f5 | David Douard | 07 January 2019, 10:32:41 UTC | Use a dict as click context obj in cli This is needed to be able to add more context objects (see following revisions). | 10 January 2019, 14:43:39 UTC |
| 584724d | David Douard | 21 December 2018, 15:52:57 UTC | Add a new SWHTask class to be used as base class for swh celery tasks It is meant to be used to declare swh tasks via the task decorator instead of subclassing the (now deprecated) Task class. It is typically used like: from swh.scheduler.celery_backend.config import app from swh.scheduler.tasks import SWHTask @app.task(base=SWHTask) def ping(): return 'pong' | 10 January 2019, 14:43:30 UTC |
| f2faafc | Jenkins for Software Heritage | 09 January 2019, 18:32:12 UTC | New upstream version 0.0.38 | 09 January 2019, 18:32:12 UTC |
| b812434 | David Douard | 21 December 2018, 15:48:02 UTC | Replace the TaskRouter class by a simple function class-based router is a remainder of celery 3. | 08 January 2019, 09:39:49 UTC |
| 8559cec | David Douard | 21 December 2018, 15:49:59 UTC | Improve logging of the listener - do use a dedicated logger instead of the root logger, - add a couple of logging statements (in perform_action methods), - replace the --verbose cli option by --log-level | 08 January 2019, 09:39:49 UTC |
| be0e938 | David Douard | 21 December 2018, 15:44:56 UTC | Improve logging configuration in celery's setup_log_handler function - allow to pass the loglevel as a string, - kill useless tmp variables, - add a filter to prevent amqp's heartbeat_tick debug messages. | 08 January 2019, 09:39:49 UTC |
| b6bc2f2 | Nicolas Dandrimont | 18 December 2018, 16:08:45 UTC | Explicitly register class-based tasks inheriting from our own class This lets go of the metaclass madness imported from Celery 3, and goes for an explicit task registration mechanism as advised in Celery 4. We do explicit registration of all our tasks in the 'worker ready' signal, just before they automatically subscribe to the task queues. Unfortunately that signal is not emitted by the "test fixture" worker, so we need to explicitly register class-based tasks that are being used in tests. This doesn't show up for functions as the @task decorator handles registration. | 19 December 2018, 14:39:10 UTC |
| 05c641b | David Douard | 18 December 2018, 17:02:34 UTC | Refactor a bit the 'swh-scheduler task list-pending' cli tool since the task-type is mandatory, make it a command argument instead of an option, and make it variadic, so one can list several pending task types at once. Also do not use the pager. Let the user decide whether she wants paginated results. | 18 December 2018, 17:38:59 UTC |
| 2582293 | David Douard | 18 December 2018, 17:00:06 UTC | Add a new 'swh-schedule task add' command that allows to easily insert a new task in the scheduler's database, eg. swh-scheduler --database 'service=swh-scheduler' \ task add swh-lister-debian distribution=stretch policy=oneshot | 18 December 2018, 17:38:59 UTC |
| 4317681 | David Douard | 18 December 2018, 17:01:31 UTC | Add a new policy argument to SchedulerBackend.create_tasks so that we can expose it to cli tools. | 18 December 2018, 17:38:59 UTC |
| d2ea7b1 | David Douard | 18 December 2018, 16:58:42 UTC | Add a --list-types option to the 'swh-scheduler task' command that lists available task types. | 18 December 2018, 17:38:47 UTC |
| 4335cfc | Nicolas Dandrimont | 18 December 2018, 16:04:24 UTC | Automatically subscribe workers to the per-task queues That only works for actual workers (the celeryd_after_setup signal isn't sent by the test fixture), so keep the workaround in the fixture | 18 December 2018, 16:16:25 UTC |
| bbe9e55 | Nicolas Dandrimont | 18 December 2018, 15:40:03 UTC | Fix typo in celery_testing | 18 December 2018, 16:16:25 UTC |
| 4cacd65 | Nicolas Dandrimont | 18 December 2018, 14:38:39 UTC | Route tasks using their task name rather than a task_queue attribute | 18 December 2018, 16:16:25 UTC |
| f984974 | Nicolas Dandrimont | 18 December 2018, 14:32:48 UTC | celery_backend: more robust queue length management - Ignore the absence of the rabbitmq management interface - Handle inexistent queues as if they were empty | 18 December 2018, 16:16:25 UTC |
| af53734 | Nicolas Dandrimont | 18 December 2018, 14:30:01 UTC | Don't run SQL query with an empty tuple Noticed the issue when repeatedly running tests without really letting the database tear down properly... | 18 December 2018, 16:16:25 UTC |
| 15262bb | Nicolas Dandrimont | 18 December 2018, 14:28:59 UTC | Switch celery settings to lowercase names | 18 December 2018, 16:16:25 UTC |
| 7daad6f | Nicolas Dandrimont | 18 December 2018, 14:25:41 UTC | Record requirement on celery 4 The tests scaffolding depends on fixtures introduced in Celery 4 | 18 December 2018, 16:15:56 UTC |
| 9966e55 | David Douard | 18 December 2018, 09:36:47 UTC | Remove pickle from Celery's accepted content type | 18 December 2018, 09:36:47 UTC |
| 041f103 | David Douard | 18 December 2018, 09:33:56 UTC | Fix Task's docstring | 18 December 2018, 09:33:56 UTC |
| 97396ed | Antoine R. Dumont (@ardumont) | 14 December 2018, 14:50:12 UTC | Update requirements to latest swh.core Related T1444 | 14 December 2018, 14:50:12 UTC |
| 46b2ba6 | Valentin Lorentz | 03 December 2018, 13:44:54 UTC | Fallback for get_queue_stats() when using memory:// broker. Summary: When running tests, RabbitMQ is not used, so get_queue_stats() does not work. We did not notice this issue before because pytest skipped importing modules that define tasks with a queue, so this function was never actually called while running tests. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D763 | 03 December 2018, 14:53:48 UTC |
| 0f77286 | Valentin Lorentz | 03 December 2018, 13:44:54 UTC | Fallback for get_queue_stats() when using memory:// broker. When running tests, RabbitMQ is not used, so get_queue_stats() does not work. We did not notice this issue before because pytest skipped importing modules that define tasks with a queue, so this function was never actually called while running tests. | 03 December 2018, 13:44:54 UTC |
| 042edee | David Douard | 28 November 2018, 10:54:37 UTC | README: fix a typo | 28 November 2018, 10:54:37 UTC |
| cea1aed | David Douard | 22 November 2018, 11:00:15 UTC | doc: update index to match new swh-doc format related to T1330 | 23 November 2018, 12:52:54 UTC |
| 5e67d28 | Antoine R. Dumont (@ardumont) | 21 November 2018, 15:35:21 UTC | data/sql: Insert new task type for origin metadata indexer Related T1376 | 21 November 2018, 15:35:21 UTC |
| b7b596f | Antoine R. Dumont (@ardumont) | 21 November 2018, 15:33:23 UTC | data/sql: Insert new task type for revision metadata indexer Related T1375 | 21 November 2018, 15:33:23 UTC |
| 5c636ef | Antoine R. Dumont (@ardumont) | 21 November 2018, 12:00:13 UTC | data/sql: Insert new task type for origin indexer Related T1326 | 21 November 2018, 12:00:13 UTC |
| 26efaad | Antoine R. Dumont (@ardumont) | 20 November 2018, 11:23:35 UTC | sql/data: Create new range indexer fossology license task type | 20 November 2018, 11:23:35 UTC |
| 85526e1 | Antoine R. Dumont (@ardumont) | 19 November 2018, 13:31:46 UTC | sql/data: Remove orchestrator Related T1311 | 19 November 2018, 13:31:46 UTC |
| fc38be1 | Antoine R. Dumont (@ardumont) | 19 November 2018, 13:07:25 UTC | sql/data: Create new range indexer mimetype task type Related T911 | 19 November 2018, 13:08:29 UTC |
| e2c8e9b | David Douard | 09 November 2018, 11:04:33 UTC | Fix rst syntax in docstrings | 09 November 2018, 11:06:14 UTC |
| 3bdb28f | Antoine Lambert | 08 November 2018, 13:51:51 UTC | sql/updater/Makefile: Fix pifpaf test as it was always considered used | 08 November 2018, 13:51:51 UTC |
| 816f0b9 | Antoine Lambert | 08 November 2018, 13:46:07 UTC | sql/Makefile: Fix pifpaf test as it was always considered used | 08 November 2018, 13:46:07 UTC |
| be65cdb | Antoine R. Dumont (@ardumont) | 29 October 2018, 09:07:08 UTC | New upstream version 0.0.35 | 29 October 2018, 09:07:08 UTC |
| 762b528 | Valentin Lorentz | 29 October 2018, 08:45:23 UTC | Add SchedulerTestFixture. Summary: Will be used by the indexers. Test Plan: There's a test included Reviewers: #reviewers, douardda, ardumont Reviewed By: #reviewers, douardda, ardumont Subscribers: ardumont, douardda, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D605 | 29 October 2018, 08:54:13 UTC |
| 4b05032 | Antoine R. Dumont (@ardumont) | 26 October 2018, 19:31:21 UTC | sql/40-swh-data: Update new indexer task types for local db Those new types need to be deployed in production when the new indexer will hit production. Related T1290 | 29 October 2018, 08:40:41 UTC |
| e51e9ff | Antoine R. Dumont (@ardumont) | 29 October 2018, 08:33:06 UTC | swh.scheduler.utils: Improve implementation to clarify intent | 29 October 2018, 08:33:06 UTC |
| eefe310 | Antoine R. Dumont (@ardumont) | 26 October 2018, 18:32:32 UTC | swh.scheduler.utils: Allow to add more task information Also add test on that function. | 26 October 2018, 18:32:32 UTC |
| 3d23ded | Antoine R. Dumont (@ardumont) | 25 October 2018, 15:52:03 UTC | New upstream version 0.0.34 | 25 October 2018, 15:52:03 UTC |
| f326cef | Antoine R. Dumont (@ardumont) | 25 October 2018, 15:47:40 UTC | d/rules: Update build with pytest flags Related T1261 | 25 October 2018, 15:48:09 UTC |
| e25fdf7 | Antoine R. Dumont (@ardumont) | 25 October 2018, 14:26:28 UTC | d/control: Update pytest build dependency Related T1261 | 25 October 2018, 15:47:56 UTC |
| 6362b21 | David Douard | 25 October 2018, 14:03:16 UTC | New upstream version 0.0.33 | 25 October 2018, 14:03:16 UTC |
| 923e0b5 | David Douard | 25 October 2018, 13:10:23 UTC | celery: configure result_serializer to msgpack | 25 October 2018, 13:10:23 UTC |
| 1203083 | Nicolas Dandrimont | 23 October 2018, 14:48:15 UTC | Don't run pytest in the docs directory | 23 October 2018, 14:48:15 UTC |
| 73a4e43 | David Douard | 23 October 2018, 08:18:18 UTC | tests: add a too_slow setting on UpdaterConsumerWithEventTest.test_running so that hypothesis does not complains when executed on a slower (aka CI) nachine. | 23 October 2018, 08:18:18 UTC |
| 3d23931 | David Douard | 18 October 2018, 14:55:07 UTC | tests: kill deprecation warnings - assertEquals -> assertEqual - warn -> warning | 23 October 2018, 07:57:17 UTC |
| 80041ad | David Douard | 18 October 2018, 14:42:10 UTC | tests: replace nose's attr() by pytest's mark() for tests tagging related to T1261 | 23 October 2018, 07:57:17 UTC |
| 3e64657 | David Douard | 18 October 2018, 14:41:10 UTC | replace nose by pytest in test requirements and add a simple tox file related to T1261 | 23 October 2018, 07:57:09 UTC |
| e45eb14 | Antoine R. Dumont (@ardumont) | 22 October 2018, 13:37:51 UTC | New upstream version 0.0.32 | 22 October 2018, 13:37:51 UTC |
| 8df816a | David Douard | 17 October 2018, 07:47:10 UTC | tests: properly implement Celery broker and result configuration for tests purpose ie. use memory:// based urls. Note that these configs can be overloaded via CELERY_BROKER_URL and CELERY_RESULT_BACKEND env vars. | 18 October 2018, 07:29:58 UTC |
| 3c9fcd8 | David Douard | 17 October 2018, 14:07:49 UTC | packaging: bump dep to swh-core v0.0.44 | 17 October 2018, 14:07:49 UTC |
| f15eeb4 | David Douard | 16 October 2018, 13:24:53 UTC | tests: make tests use sql files from the package instead of dump files in test/data, and get rid of them. Related to T1265. | 16 October 2018, 13:27:29 UTC |
| ff4df2c | David Douard | 16 October 2018, 12:27:45 UTC | sql: move sql files in the python package also delete unused file swh-scheduler-testdata.sql and fix sql/Makefile and sql/updater/sql/Makefile to make them properly generate svg/pdf images and make. Related to T1265. | 16 October 2018, 13:23:34 UTC |
| 3976f99 | Antoine Lambert | 15 October 2018, 09:34:20 UTC | sql/swh-scheduler-data: Fix some task names and add svn update task | 15 October 2018, 11:28:43 UTC |
| 1e371db | Nicolas Dandrimont | 15 October 2018, 08:27:28 UTC | Rename test methods to test_* to allow collection by pytest Summary: Part of T1261 Test Plan: 37 tests on both sides of the jump Reviewers: ardumont Differential Revision: https://forge.softwareheritage.org/D530 | 15 October 2018, 08:36:58 UTC |
| 2f0f543 | Nicolas Dandrimont | 15 October 2018, 08:25:36 UTC | Use raw strings for regexps that need to escape parentheses | 15 October 2018, 08:26:20 UTC |
| a205968 | David Douard | 11 October 2018, 14:53:13 UTC | README: document how to get test data | 12 October 2018, 11:12:51 UTC |
| 8ed3f86 | David Douard | 11 October 2018, 14:10:59 UTC | tests: add celery to test requirements | 12 October 2018, 11:12:51 UTC |
| 15e895e | David Douard | 11 October 2018, 14:02:28 UTC | tests: add a CeleryTestFixture that allows to specify a celery broker URL this especially aims at allowing to run tests that do actually use non mocked celery broker using pifpaf. | 12 October 2018, 11:12:51 UTC |
| 50966d5 | David Douard | 12 October 2018, 11:07:20 UTC | tests: move the test db dump back here and ensure it is included in the source package. Related to T1265. Also add README.md to the MANIFEST.in file so it does not require a too recent setuptool. | 12 October 2018, 11:07:20 UTC |
| d5473d6 | Antoine R. Dumont (@ardumont) | 10 October 2018, 13:44:11 UTC | listener: Deal with messages received as list Prior to celery 4, the body message were sent 1 by 1. Now (celery 4), there are sent as list. This takes care of the change but can deal with prior version as well (think debian stable). Related D312#inline-2518 | 11 October 2018, 12:54:41 UTC |
| 339028c | Antoine R. Dumont (@ardumont) | 10 October 2018, 13:32:16 UTC | listener: Drop unused queue_arguments settings Related D312#inline-2519 | 11 October 2018, 12:54:41 UTC |
| cce9505 | Antoine R. Dumont (@ardumont) | 09 October 2018, 12:56:07 UTC | celery_backend.listener: Use debugging logging statements | 11 October 2018, 12:54:41 UTC |
| 52d041e | Antoine R. Dumont (@ardumont) | 09 October 2018, 12:51:26 UTC | celery_backend.listener: Access directly existing queue_arguments | 11 October 2018, 12:54:40 UTC |
| bb29ff3 | Antoine R. Dumont (@ardumont) | 27 April 2018, 13:33:08 UTC | Fix scheduler listener on buster's celery version (4.1.0-4) | 11 October 2018, 12:54:40 UTC |
| 4e52e40 | Nicolas Dandrimont | 20 September 2018, 13:25:51 UTC | Make swh_scheduler_create_tasks_from_temp use indexes We don't have an index on task.arguments, but we have one on each of its args and kwargs elements. We therefore make sure we use these indexes. | 10 October 2018, 15:10:55 UTC |
| 5f256e5 | David Douard | 04 October 2018, 14:46:47 UTC | add build/ and dist/ to gitignore | 08 October 2018, 10:18:46 UTC |
| 226c398 | David Douard | 04 October 2018, 14:45:17 UTC | setup: prepare for pypi upload related to T1242 | 08 October 2018, 10:18:46 UTC |
| 99d5e7f | David Douard | 04 October 2018, 14:44:51 UTC | docs: add a simple README file | 08 October 2018, 10:16:47 UTC |
| d070070 | Antoine Pietri | 05 October 2018, 10:55:39 UTC | Add requirements-test.txt | 05 October 2018, 10:55:56 UTC |
| 0f4da9a | Valentin Lorentz | 02 October 2018, 14:57:51 UTC | Add test requirement. | 02 October 2018, 14:58:17 UTC |
| 239bdec | Stefano Zacchiroli | 01 October 2018, 14:01:14 UTC | docs: add title and brief module description | 01 October 2018, 14:01:14 UTC |
| da09ad0 | Antoine R. Dumont (@ardumont) | 28 September 2018, 10:22:47 UTC | sql/data: Update svn task name according to latest development | 28 September 2018, 10:28:43 UTC |
