| 06137f0 | Stefano Zacchiroli | 01 October 2019, 11:08:23 UTC | tox: anticipate mypy run to just after flake8 | 01 October 2019, 11:08:23 UTC |
| c78b846 | Stefano Zacchiroli | 27 September 2019, 08:38:31 UTC | init.py: switch to documented way of extending path make mypy 0.730 pass cleanly again | 27 September 2019, 08:38:31 UTC |
| 1a691b5 | Stefano Zacchiroli | 24 September 2019, 11:55:04 UTC | tox.ini: add mypy section | 24 September 2019, 11:55:04 UTC |
| c4fa353 | Stefano Zacchiroli | 24 September 2019, 11:43:34 UTC | typing: minimal changes to make a no-op mypy run pass | 24 September 2019, 11:44:50 UTC |
| 3cd5697 | Stefano Zacchiroli | 24 September 2019, 11:42:23 UTC | fix typo in docstring and sample file name courtesy of codespell | 24 September 2019, 11:42:23 UTC |
| 0a4fd7a | Stefano Zacchiroli | 17 September 2019, 15:49:06 UTC | admin CLI: drop obsolete backward compatibility aliases These have been around for a while and are no longer advertised in the --help output. | 17 September 2019, 15:51:03 UTC |
| 6a054b9 | Stefano Zacchiroli | 16 September 2019, 15:13:55 UTC | click "required" param wants bool, not int courtesy of mypy | 16 September 2019, 15:13:55 UTC |
| a64ac3f | David Douard | 04 September 2019, 13:53:19 UTC | sql: stop inserting lister tasks at scheduler's creation time This is now the job of the swh-lister to ensure task-type entries are created in the scheduler (in swh-lister >= 0.0.34). | 04 September 2019, 13:53:19 UTC |
| 9c2b329 | Antoine R. Dumont (@ardumont) | 03 September 2019, 08:18:30 UTC | 50-swh-data: Add load-tar task-type This is needed to be in sync with the docker-dev environment. | 03 September 2019, 08:18:30 UTC |
| c3fae15 | David Douard | 13 May 2019, 11:57:12 UTC | celery: auto add tasks declared in the swh.workers entry point in task_modules allows to declare worker tasks in a 'swh.workers' entry point. This later is expected to be a callable which returns a dict which 'task_modules' key is a list of python module names where celery tasks are defined. | 02 September 2019, 11:33:28 UTC |
| e1b216d | Antoine Lambert | 23 July 2019, 12:38:29 UTC | api/client: use RPCClient instead of deprecated SWHRemoteAPI | 23 July 2019, 12:38:29 UTC |
| a43b3c7 | Valentin Lorentz | 12 July 2019, 10:18:15 UTC | Make schedule_origins use origin urls instead of ids in task arguments. | 19 July 2019, 14:35:25 UTC |
| cc1b590 | Stefano Zacchiroli | 11 July 2019, 14:29:50 UTC | add code of conduct document | 11 July 2019, 14:29:50 UTC |
| f63274a | David Douard | 10 July 2019, 13:23:18 UTC | docs: very beginning of a practical documentation on the scheduler | 10 July 2019, 14:01:37 UTC |
| 9b7407f | David Douard | 10 July 2019, 13:58:47 UTC | updater: small refactoring to please codespell | 10 July 2019, 13:58:47 UTC |
| 92b81f3 | David Douard | 10 July 2019, 13:52:26 UTC | Add a pre-commit config file | 10 July 2019, 13:52:26 UTC |
| 2044c07 | Antoine R. Dumont (@ardumont) | 28 June 2019, 15:52:57 UTC | data: Insert new cgit instance lister task Related T1835 | 28 June 2019, 15:52:57 UTC |
| 09d9c84 | Stefano Zacchiroli | 28 June 2019, 07:42:00 UTC | CONTRIBUTORS: add Ishan Bhanuka | 28 June 2019, 07:42:00 UTC |
| 09e7245 | Ishan Bhanuka | 15 June 2019, 11:56:53 UTC | Added comments to few columns in dbversion, task and task_run Change double quoted comments to single quoted comments To keep consistent comment format and to prevent syntax exception on jenkins build Make changes as per review Made changes as per review | 19 June 2019, 16:13:09 UTC |
| 237e8c5 | David Douard | 17 May 2019, 15:06:34 UTC | cli: rename a admin cli commands use more imperative-like command names. Aliases are used to keep old commands working. Also add an explicit deprecation message in swh-scheduler's help message. | 13 June 2019, 07:29:09 UTC |
| da77f89 | Antoine R. Dumont (@ardumont) | 12 June 2019, 13:22:32 UTC | 50-swh-data: Update new lister task types Related T1797 | 12 June 2019, 13:22:32 UTC |
| 46223d8 | Antoine R. Dumont (@ardumont) | 24 May 2019, 10:21:43 UTC | scheduler.cli.task: Fix wrong import | 24 May 2019, 10:21:43 UTC |
| aa70df4 | Antoine R. Dumont (@ardumont) | 24 May 2019, 08:11:37 UTC | Drop backward compatible names The migration is now done. No longer needs this adaptation code. Related T1508 | 24 May 2019, 08:11:37 UTC |
| 63eeabf | Antoine R. Dumont (@ardumont) | 23 May 2019, 11:27:41 UTC | data: Add missing swh-vault-batch-cooking/cook-vault-bundle-type Related T1508 | 23 May 2019, 11:27:41 UTC |
| 766dae9 | Antoine R. Dumont (@ardumont) | 23 May 2019, 11:22:59 UTC | 50-swh-data: Fix inverted select/insert in backward sql function Prior to this, the select wanted to retrieve the new name to insert the old one Ending up doing a noop. Related T1508 | 23 May 2019, 11:23:28 UTC |
| d6fce0d | Antoine Lambert | 14 May 2019, 15:53:58 UTC | swh-scheduler: Use new task names Related T1508 | 21 May 2019, 11:34:25 UTC |
| d6dc50f | David Douard | 13 May 2019, 11:55:31 UTC | tests: update to swh.core 0.0.61 in which (db|api)_testing.py files moved. | 17 May 2019, 15:09:41 UTC |
| 52f8c06 | Valentin Lorentz | 17 May 2019, 11:32:20 UTC | Update import of swh.core.api.tests.server_testing. | 17 May 2019, 11:32:20 UTC |
| 45aa02c | Antoine Lambert | 14 May 2019, 14:19:00 UTC | cli: Fix swh-scheduler command (missing log_level in context) | 15 May 2019, 11:57:22 UTC |
| ac1b8db | Nicolas Dandrimont | 07 May 2019, 16:16:20 UTC | listener: Release the db object after using it This is the contract that get_db/put_db is supposed to conform to. | 10 May 2019, 13:52:36 UTC |
| edaf202 | David Douard | 07 May 2019, 08:52:06 UTC | cli: split the main cli/__init__.py file in 4 parts extract: - the 'task' group (and its related commands) in cli/task.py - the 'task_type' group in cli/task_type.py - remaining commands in cli/admin.py | 07 May 2019, 08:52:06 UTC |
| 3f88069 | David Douard | 06 May 2019, 13:52:01 UTC | README: remove the deprecated tests section | 06 May 2019, 13:52:01 UTC |
| 84c0bd7 | David Douard | 02 May 2019, 10:07:31 UTC | cli: make swh-scheduler a subcommand of swh.core's main "swh" command the old "swh-scheduler" still exists for now, and is mostly backward compatible but the --log-level option (which is defined in the "swh" main command group). Unit tests have been adapted and should pass ok. | 06 May 2019, 13:48:31 UTC |
| eb42639 | David Douard | 19 April 2019, 14:10:55 UTC | cli: make it a subpackage | 06 May 2019, 13:47:40 UTC |
| d338c76 | Antoine Lambert | 02 May 2019, 16:00:55 UTC | sql/swh-data: Update scheduler task names but keep backward compatibility Related T1508 | 03 May 2019, 08:50:30 UTC |
| 72eaa42 | Nicolas Dandrimont | 12 April 2019, 10:03:09 UTC | Update coverage gitignore | 12 April 2019, 10:03:09 UTC |
| 24e6a0a | Nicolas Dandrimont | 10 April 2019, 14:15:11 UTC | cli_utils: Use yaml.safe_load instead of yaml.load | 10 April 2019, 18:36:47 UTC |
| b644006 | Valentin Lorentz | 10 April 2019, 13:39:49 UTC | Fix support of latest versions of swh-core and psycopg2. | 10 April 2019, 16:55:20 UTC |
| 4b0e952 | Antoine Lambert | 10 April 2019, 14:12:25 UTC | sql/data: Add npm related task types Related T1629 Related T1508 | 10 April 2019, 16:25:53 UTC |
| aaa9ee4 | Antoine Lambert | 04 April 2019, 14:45:14 UTC | SchedulerBackend.create_tasks: Remove duplicated task key This issue was spotted after the upgrade of psycopg2 to version 2.8 | 04 April 2019, 14:45:14 UTC |
| 38cafb9 | Nicolas Dandrimont | 02 April 2019, 16:04:19 UTC | Make result_serializer json We kind-of-need that to test that we'll be able to understand results in production (even though we're sending results by hand, not through this event) and it works around https://github.com/celery/celery/issues/5366 in Celery 4.3. | 02 April 2019, 16:05:33 UTC |
| b9903d7 | Antoine Lambert | 01 April 2019, 14:22:18 UTC | sql/Makefile: Fix scheduler db creation | 01 April 2019, 14:22:18 UTC |
| c2b1793 | Antoine R. Dumont (@ardumont) | 21 March 2019, 10:43:30 UTC | requirements.txt: Remove kombu dependency This actually conflicts with the latest celery version. We should not depend on it as this is drawn by celery already. | 21 March 2019, 10:46:17 UTC |
| 4ee93d3 | David Douard | 06 March 2019, 11:32:55 UTC | API: use default's APIError exception instead of the SchedulerAPIError this later does not brings anything over the default's one. This requires swh.core >= 0.0.56 | 18 March 2019, 16:02:14 UTC |
| 693a147 | Nicolas Dandrimont | 15 March 2019, 13:25:15 UTC | Remove info-level logs about already acknowledged messages Celery voluntarily pools messages to reduce traffic, so it's expected that some messages end up here several times. No need to spam logs about it. | 15 March 2019, 13:25:15 UTC |
| 763dbda | Valentin Lorentz | 14 March 2019, 12:15:29 UTC | Ignore INFO messages when testing CLI output. | 14 March 2019, 15:36:08 UTC |
| 1122ee1 | Valentin Lorentz | 13 March 2019, 14:20:22 UTC | Add a 'task schedule_origins' command to run a task on all origins. | 14 March 2019, 15:36:08 UTC |
| 6b795e3 | Nicolas Dandrimont | 08 March 2019, 16:37:01 UTC | Introduce SWH_LOG_TARGET to allow switching between console and journal The current state of the celery worker/utility logging setup makes us log twice: the default value of log_console can't be overridden by the cli flag in celery workers. The environment variable can be set to journal when running as a systemd service; this shuts down the console logger, and allows the metadata added to log messages to pass through. | 13 March 2019, 14:05:24 UTC |
| 6dbfd06 | Valentin Lorentz | 13 March 2019, 09:17:27 UTC | Remove ghtorrent/fake.py, it's not used anymore. | 13 March 2019, 09:17:27 UTC |
| e6ee2a7 | Valentin Lorentz | 01 March 2019, 13:27:34 UTC | Add tests for 'swh-scheduler task list-pending' and 'swh-scheduler task list'. | 13 March 2019, 08:58:19 UTC |
| da290c2 | Valentin Lorentz | 13 March 2019, 08:57:01 UTC | Add support for non-string options when scheduling tasks. This also fixes the pretty-printing of tasks, which was ambiguous (`42` and `"42"` where both printed as `42`). | 13 March 2019, 08:57:01 UTC |
| b0aea8c | David Douard | 06 March 2019, 11:44:51 UTC | tests: enforce LC_ALL since test_cli depends on it | 06 March 2019, 11:45:38 UTC |
| dc6afe4 | Valentin Lorentz | 01 March 2019, 10:16:24 UTC | Revert "Hack to fix the ImportError in updater/ghtorrent/fake.py." This reverts commit 3939a41d73bdef94913a6e8c3aa212532d281463. Wasn't supposed to be committed yet. | 01 March 2019, 10:16:24 UTC |
| 5499400 | Valentin Lorentz | 26 February 2019, 12:51:52 UTC | Move option parsing to its own (tested) function. | 01 March 2019, 10:14:06 UTC |
| 3939a41 | Valentin Lorentz | 28 February 2019, 17:09:56 UTC | Hack to fix the ImportError in updater/ghtorrent/fake.py. Sphinx complains about it. | 28 February 2019, 17:09:56 UTC |
| f698b01 | Valentin Lorentz | 27 February 2019, 16:22:45 UTC | Move parts of 30-swh-schema.sql to 40-swh-func.sql and 60-swh-indexes.sql. For uniformity with other SWH packages. | 28 February 2019, 13:42:06 UTC |
| 7ddfdf4 | Antoine Lambert | 27 February 2019, 16:51:40 UTC | celery_backend.listener: Fix debug output format string | 27 February 2019, 16:51:40 UTC |
| 099effd | Valentin Lorentz | 21 February 2019, 14:08:51 UTC | Start writing tests for the CLI. | 27 February 2019, 11:16:52 UTC |
| 40d32b4 | David Douard | 25 February 2019, 12:57:46 UTC | celery: fix the celery config to ensure the default 'celery' queue exists as it used to be before the recent refactorings. | 25 February 2019, 13:10:00 UTC |
| 55f7e1f | David Douard | 25 February 2019, 12:56:02 UTC | celery: do not set the default value for celery's result_backend in production, we do not want a result backend to be used, and the pytest fixture handle this configuration properly. Closes T1539. | 25 February 2019, 13:10:00 UTC |
| 5b08922 | David Douard | 25 February 2019, 12:59:23 UTC | cli: kill a useless statement the Flask app object does not need a scheduler attribute. | 25 February 2019, 13:02:59 UTC |
| cfaf833 | Antoine R. Dumont (@ardumont) | 22 February 2019, 23:35:13 UTC | Remove no longer used default config path Related T1533 | 22 February 2019, 23:35:13 UTC |
| ce90908 | Antoine R. Dumont (@ardumont) | 22 February 2019, 23:23:43 UTC | swh.scheduler.api: Make api use explicit configuration (+ checks) Related T1533 | 22 February 2019, 23:23:43 UTC |
| 1d4982a | Antoine R. Dumont (@ardumont) | 22 February 2019, 15:09:46 UTC | sql/30-swh-schema.sql: Fix comment instruction To help ci build: https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DCIDX/job/gbp-buildpackage/30/console | 22 February 2019, 15:09:46 UTC |
| 981d3f9 | Valentin Lorentz | 20 February 2019, 15:53:20 UTC | Fix crash of SchedulerBackend.search_tasks when no argument is given. | 20 February 2019, 16:16:47 UTC |
| 058f120 | David Douard | 15 February 2019, 16:03:11 UTC | cli: add a --runs option to 'swh-scheduler task list' tool this options allws to list task runs for selected tasks. | 20 February 2019, 09:23:20 UTC |
| 7f951d3 | David Douard | 15 February 2019, 15:56:48 UTC | Keep the main loaded configuration as swh.scheduler.CONFIG so that it can be used in tasks instead of reloading config files on each task execution. | 20 February 2019, 09:23:20 UTC |
| 73bca41 | David Douard | 15 February 2019, 15:58:36 UTC | celery_backend: remove debug statements and reduce info level verbosity by lowering the logging statement in catchall_event to DEBUG. | 20 February 2019, 08:17:36 UTC |
| 68edb6e | Antoine R. Dumont (@ardumont) | 19 February 2019, 12:36:38 UTC | Remove debian packaging from master branch Related T1532 | 19 February 2019, 12:36:38 UTC |
| 9534859 | Valentin Lorentz | 15 February 2019, 10:24:44 UTC | Fix documentation of CLI option --verbose. | 18 February 2019, 13:38:16 UTC |
| 3c41cef | Antoine R. Dumont (@ardumont) | 15 February 2019, 13:56:04 UTC | scheduler.task: Remove no longer used Task class All scheduler tasks have been rewritten to avoid using the inheritance paradigm. The post worker startup initialization no longer creates automatically queues for registered tasks. Queues creation is managed through explicit configuration entries: celery: task_queues: ... task_modules: ... | 15 February 2019, 13:59:14 UTC |
| 65d0f73 | Antoine R. Dumont (@ardumont) | 15 February 2019, 09:41:00 UTC | celery_backend/config: Fix loglevel for amqp module | 15 February 2019, 09:43:00 UTC |
| 283cb7c | Antoine R. Dumont (@ardumont) | 14 February 2019, 08:42:26 UTC | tests: Use hypothesis profile to configure sample generation sizes | 14 February 2019, 08:42:26 UTC |
| c701c88 | Antoine R. Dumont (@ardumont) | 13 February 2019, 14:57:08 UTC | api/server: Do not read configuration at each request | 13 February 2019, 15:19:59 UTC |
| f0a8c43 | David Douard | 13 February 2019, 13:37:18 UTC | listener: make the listener's queue name independent from the hostname the queue being durable and not auto deleted (auto_delete=False), we do not want a new queue to be spawned for each listener instance (eg. in a docker environment). | 13 February 2019, 13:37:18 UTC |
| b423c0b | David Douard | 13 February 2019, 13:28:37 UTC | runner: fix task_run configuration bootstrap ensure the task_run is created before sending the celery task; this later task could be executed before the db commit. As a result, the task_run may not have its 'started' field properly set; even the 'ended' and 'status'. | 13 February 2019, 13:28:37 UTC |
| 3d761f1 | David Douard | 13 February 2019, 13:25:58 UTC | sql: add the swh-lister-bitbucket-* task types | 13 February 2019, 13:25:58 UTC |
| 7e3f2fc | David Douard | 13 February 2019, 13:23:19 UTC | task: do not send the task-result-exception event in Task.on_failure() since it is unused; the task-failed event being already sent by celery and handled by the listener. | 13 February 2019, 13:23:19 UTC |
| 3488c26 | Antoine R. Dumont (@ardumont) | 11 February 2019, 10:44:40 UTC | requirements-test: Enforce a version for hypothesis Same as other swh modules (swh-web, swh-storage, swh-indexer) Inferior version do not follow the specifications Related P356 | 11 February 2019, 11:26:24 UTC |
| db25694 | David Douard | 06 February 2019, 13:18:18 UTC | Fix a bug in the listener: commit() is not defined in the backend but on its connection. | 06 February 2019, 13:18:18 UTC |
| c29d383 | Nicolas Dandrimont | 19 December 2018, 14:59:38 UTC | Add basic stats to tasks This just increments a counter for started tasks, ended tasks, tasks that failed with an exception. It also registers a timer for every task run. Close T1460. | 06 February 2019, 12:06:39 UTC |
| abfe3db | David Douard | 01 February 2019, 15:57:29 UTC | Allow to override celery config file name via the SWH_CONFIG_FILENAME env var this will take precedence over the implicit config file scheme. The expected config file given via the environment variable is expected to have a [celery] section which will be used as config for the Celery app created in swh.scheduler.celery_backend.config. Related to T1410 and T826. | 06 February 2019, 09:25:07 UTC |
| 0f2f3ff | David Douard | 01 February 2019, 14:32:21 UTC | Make (celery) tests immune to environment variables especially CELERY_BROKER_URL... | 01 February 2019, 14:32:21 UTC |
| e2a91fb | Valentin Lorentz | 01 February 2019, 12:47:47 UTC | Remove call to tobytes(), BaseDB now handles conversion. | 01 February 2019, 14:03:48 UTC |
| 0c3306b | David Douard | 31 January 2019, 14:27:52 UTC | Fix the task_queues Celery config setting in build_app() ensure the config entry contains Queue objects. | 31 January 2019, 14:52:41 UTC |
| f188343 | David Douard | 31 January 2019, 14:26:57 UTC | cli: display a sorted list of task-types | 31 January 2019, 14:52:41 UTC |
| b0e6dd8 | David Douard | 31 January 2019, 12:37:38 UTC | Fix the listener: accessing the db cnx from the backend has changed | 31 January 2019, 14:52:41 UTC |
| 5d40529 | David Douard | 31 January 2019, 12:35:56 UTC | Make cli tools output logs on the console by default for other log levels than DEBUG, and add a --no-stdout option flag to disable this. | 31 January 2019, 14:52:41 UTC |
| a8bc684 | David Douard | 31 January 2019, 11:14:54 UTC | cli: build the celery app from a celery section of the given configuration file for runner and listener commands. related to T1410 | 31 January 2019, 14:52:32 UTC |
| 4246286 | David Douard | 31 January 2019, 11:13:44 UTC | Add a build_app() function to instantiate a Celery app with controlled config ie. being able to give the celery config dict as parameter. related to T1410 | 31 January 2019, 14:45:37 UTC |
| 70581b6 | David Douard | 30 January 2019, 09:43:40 UTC | Add a /site-map endpoint that lists published routes for this server | 31 January 2019, 10:15:50 UTC |
| 5880c52 | David Douard | 31 January 2019, 09:22:41 UTC | Activate the support for options from environment variables for swh-scheduler tool so that one can type (typically in a venv, with services running in dockers): (venv) swh-environment$ export SWH_SCHEDULER_URL=http://127.0.0.1:5008 (venv) swh-environment$ swh-scheduler task-type list | 31 January 2019, 09:29:43 UTC |
| 9bc5640 | David Douard | 31 January 2019, 09:21:13 UTC | Fix 'swh-scheduler runner' command: rollback() has beed removed from the SchedulerBackend and is not needed any more there. | 31 January 2019, 09:29:43 UTC |
| 174d89b | David Douard | 31 January 2019, 09:18:33 UTC | Fix get_scheduler's cls value when using 'swh-scheduler --url' cli option also ensure args dict does not have default db settings (unsupported by the RemoteScheduler class). | 31 January 2019, 09:29:43 UTC |
| 6da09a1 | David Douard | 30 January 2019, 15:55:31 UTC | Fix a regression introduced in 61c91b82 when deleting a click.option, one would better delete the function argument as well... | 30 January 2019, 15:58:11 UTC |
| b25c7cd | Valentin Lorentz | 30 January 2019, 10:27:00 UTC | Drop 'except Exception', it catches too many errors. eg. ImportErrors when negotiate is not installed. | 30 January 2019, 12:55:58 UTC |
| 84cded2 | David Douard | 30 January 2019, 11:21:28 UTC | BaseDb.copy_to's default_columns has been renamed as default_values | 30 January 2019, 11:22:51 UTC |
| 4fc7a89 | David Douard | 30 January 2019, 09:19:11 UTC | Make the prepare_event helper function pre-aggregate the events with same url and strip these urls as well. It makes no sense to register 2 different URLs when they are equal to a trailing ws detail. As a result, we must preaggregate them because the swh_cache_put() sql function won't allow several 'on conflict' for the same id. | 30 January 2019, 10:51:22 UTC |
| 4117d5a | David Douard | 29 January 2019, 14:34:04 UTC | Rewrite updater/test_backend.py with pytest and use the postgresql fixture also implement the test with more precise expected behavior, especially the content pre-aggregation of events and the url stripping. As is, this test will fail, the pre-aggregation and url stripping being implemented in the following revision. | 30 January 2019, 09:57:11 UTC |
| ebee014 | David Douard | 30 January 2019, 09:14:19 UTC | Kill DbBackend class we can now use directly the implementation of the copy_to() method from swh.core's BaseDb, so we just have to extract the format_query as a simple function (which it should have been since the beginning). Adapt updater/backend.py acordingly. | 30 January 2019, 09:57:11 UTC |
