| 6e69026 | Jenkins for Software Heritage | 22 March 2019, 11:16:06 UTC | New upstream version 0.0.51 | 22 March 2019, 11:16:06 UTC |
| c2b1793 | Antoine R. Dumont (@ardumont) | 21 March 2019, 10:43:30 UTC | requirements.txt: Remove kombu dependency This actually conflicts with the latest celery version. We should not depend on it as this is drawn by celery already. | 21 March 2019, 10:46:17 UTC |
| 4ee93d3 | David Douard | 06 March 2019, 11:32:55 UTC | API: use default's APIError exception instead of the SchedulerAPIError this later does not brings anything over the default's one. This requires swh.core >= 0.0.56 | 18 March 2019, 16:02:14 UTC |
| 455ce86 | Nicolas Dandrimont | 15 March 2019, 17:24:59 UTC | Update build- and runtime dependencies | 15 March 2019, 17:24:59 UTC |
| 753e3da | Jenkins for Software Heritage | 15 March 2019, 17:16:04 UTC | Updated debian changelog for version 0.0.50 | 15 March 2019, 17:16:04 UTC |
| ba40c3e | Jenkins for Software Heritage | 15 March 2019, 17:16:03 UTC | Updated version 0.0.50 from 'debian/upstream/0.0.50' with Debian dir e821555864dcd41121b3efa31d516cfcb045ea1f | 15 March 2019, 17:16:03 UTC |
| a49761d | Jenkins for Software Heritage | 15 March 2019, 17:16:02 UTC | New upstream version 0.0.50 | 15 March 2019, 17:16:02 UTC |
| 693a147 | Nicolas Dandrimont | 15 March 2019, 13:25:15 UTC | Remove info-level logs about already acknowledged messages Celery voluntarily pools messages to reduce traffic, so it's expected that some messages end up here several times. No need to spam logs about it. | 15 March 2019, 13:25:15 UTC |
| 763dbda | Valentin Lorentz | 14 March 2019, 12:15:29 UTC | Ignore INFO messages when testing CLI output. | 14 March 2019, 15:36:08 UTC |
| 1122ee1 | Valentin Lorentz | 13 March 2019, 14:20:22 UTC | Add a 'task schedule_origins' command to run a task on all origins. | 14 March 2019, 15:36:08 UTC |
| 3c9497d | Nicolas Dandrimont | 14 March 2019, 12:42:55 UTC | Export LC_ALL=C.UTF-8 | 14 March 2019, 12:42:55 UTC |
| 6b795e3 | Nicolas Dandrimont | 08 March 2019, 16:37:01 UTC | Introduce SWH_LOG_TARGET to allow switching between console and journal The current state of the celery worker/utility logging setup makes us log twice: the default value of log_console can't be overridden by the cli flag in celery workers. The environment variable can be set to journal when running as a systemd service; this shuts down the console logger, and allows the metadata added to log messages to pass through. | 13 March 2019, 14:05:24 UTC |
| 6dbfd06 | Valentin Lorentz | 13 March 2019, 09:17:27 UTC | Remove ghtorrent/fake.py, it's not used anymore. | 13 March 2019, 09:17:27 UTC |
| e6ee2a7 | Valentin Lorentz | 01 March 2019, 13:27:34 UTC | Add tests for 'swh-scheduler task list-pending' and 'swh-scheduler task list'. | 13 March 2019, 08:58:19 UTC |
| da290c2 | Valentin Lorentz | 13 March 2019, 08:57:01 UTC | Add support for non-string options when scheduling tasks. This also fixes the pretty-printing of tasks, which was ambiguous (`42` and `"42"` where both printed as `42`). | 13 March 2019, 08:57:01 UTC |
| b0aea8c | David Douard | 06 March 2019, 11:44:51 UTC | tests: enforce LC_ALL since test_cli depends on it | 06 March 2019, 11:45:38 UTC |
| 40bb4cd | Jenkins for Software Heritage | 03 March 2019, 07:55:41 UTC | Updated debian changelog for version 0.0.49 | 03 March 2019, 07:55:41 UTC |
| 346e4de | Jenkins for Software Heritage | 03 March 2019, 07:55:40 UTC | Updated version 0.0.49 from 'debian/upstream/0.0.49' with Debian dir 81d10b10f83296d1687c91f71faf406342d63e1b | 03 March 2019, 07:55:40 UTC |
| 85b858b | Jenkins for Software Heritage | 03 March 2019, 07:55:40 UTC | New upstream version 0.0.49 | 03 March 2019, 07:55:40 UTC |
| dc6afe4 | Valentin Lorentz | 01 March 2019, 10:16:24 UTC | Revert "Hack to fix the ImportError in updater/ghtorrent/fake.py." This reverts commit 3939a41d73bdef94913a6e8c3aa212532d281463. Wasn't supposed to be committed yet. | 01 March 2019, 10:16:24 UTC |
| 5499400 | Valentin Lorentz | 26 February 2019, 12:51:52 UTC | Move option parsing to its own (tested) function. | 01 March 2019, 10:14:06 UTC |
| 3939a41 | Valentin Lorentz | 28 February 2019, 17:09:56 UTC | Hack to fix the ImportError in updater/ghtorrent/fake.py. Sphinx complains about it. | 28 February 2019, 17:09:56 UTC |
| f698b01 | Valentin Lorentz | 27 February 2019, 16:22:45 UTC | Move parts of 30-swh-schema.sql to 40-swh-func.sql and 60-swh-indexes.sql. For uniformity with other SWH packages. | 28 February 2019, 13:42:06 UTC |
| 7ddfdf4 | Antoine Lambert | 27 February 2019, 16:51:40 UTC | celery_backend.listener: Fix debug output format string | 27 February 2019, 16:51:40 UTC |
| 099effd | Valentin Lorentz | 21 February 2019, 14:08:51 UTC | Start writing tests for the CLI. | 27 February 2019, 11:16:52 UTC |
| 40d32b4 | David Douard | 25 February 2019, 12:57:46 UTC | celery: fix the celery config to ensure the default 'celery' queue exists as it used to be before the recent refactorings. | 25 February 2019, 13:10:00 UTC |
| 55f7e1f | David Douard | 25 February 2019, 12:56:02 UTC | celery: do not set the default value for celery's result_backend in production, we do not want a result backend to be used, and the pytest fixture handle this configuration properly. Closes T1539. | 25 February 2019, 13:10:00 UTC |
| 5b08922 | David Douard | 25 February 2019, 12:59:23 UTC | cli: kill a useless statement the Flask app object does not need a scheduler attribute. | 25 February 2019, 13:02:59 UTC |
| cfaf833 | Antoine R. Dumont (@ardumont) | 22 February 2019, 23:35:13 UTC | Remove no longer used default config path Related T1533 | 22 February 2019, 23:35:13 UTC |
| ce90908 | Antoine R. Dumont (@ardumont) | 22 February 2019, 23:23:43 UTC | swh.scheduler.api: Make api use explicit configuration (+ checks) Related T1533 | 22 February 2019, 23:23:43 UTC |
| ed616a6 | Jenkins for Software Heritage | 22 February 2019, 15:17:20 UTC | Updated debian changelog for version 0.0.48 | 22 February 2019, 15:17:20 UTC |
| 46ee44e | Jenkins for Software Heritage | 22 February 2019, 15:17:19 UTC | Updated version 0.0.48 from 'debian/upstream/0.0.48' with Debian dir e4a452346bacf23dca5311f5e562df5622729d19 | 22 February 2019, 15:17:19 UTC |
| 9f76738 | Jenkins for Software Heritage | 22 February 2019, 15:17:19 UTC | New upstream version 0.0.48 | 22 February 2019, 15:17:19 UTC |
| 1d4982a | Antoine R. Dumont (@ardumont) | 22 February 2019, 15:09:46 UTC | sql/30-swh-schema.sql: Fix comment instruction To help ci build: https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DCIDX/job/gbp-buildpackage/30/console | 22 February 2019, 15:09:46 UTC |
| 3f99680 | Antoine R. Dumont (@ardumont) | 21 February 2019, 14:41:31 UTC | d/changelog: Fix build dependency release | 21 February 2019, 14:41:31 UTC |
| f950116 | Antoine R. Dumont (@ardumont) | 21 February 2019, 14:39:23 UTC | d/control: Add postgresql-contrib dependency Not really needed for unstable, needed for stretch-swh | 21 February 2019, 14:39:23 UTC |
| 46248bd | Jenkins for Software Heritage | 21 February 2019, 09:13:07 UTC | Updated debian changelog for version 0.0.47 | 21 February 2019, 09:13:07 UTC |
| 0aa9d50 | Jenkins for Software Heritage | 21 February 2019, 09:13:06 UTC | Updated version 0.0.47 from 'debian/upstream/0.0.47' with Debian dir 1997c93daa313854868e0f4d2efd3d2dd3f808fb | 21 February 2019, 09:13:06 UTC |
| 759824f | Jenkins for Software Heritage | 21 February 2019, 09:13:06 UTC | New upstream version 0.0.47 | 21 February 2019, 09:13:06 UTC |
| 981d3f9 | Valentin Lorentz | 20 February 2019, 15:53:20 UTC | Fix crash of SchedulerBackend.search_tasks when no argument is given. | 20 February 2019, 16:16:47 UTC |
| 058f120 | David Douard | 15 February 2019, 16:03:11 UTC | cli: add a --runs option to 'swh-scheduler task list' tool this options allws to list task runs for selected tasks. | 20 February 2019, 09:23:20 UTC |
| 7f951d3 | David Douard | 15 February 2019, 15:56:48 UTC | Keep the main loaded configuration as swh.scheduler.CONFIG so that it can be used in tasks instead of reloading config files on each task execution. | 20 February 2019, 09:23:20 UTC |
| 73bca41 | David Douard | 15 February 2019, 15:58:36 UTC | celery_backend: remove debug statements and reduce info level verbosity by lowering the logging statement in catchall_event to DEBUG. | 20 February 2019, 08:17:36 UTC |
| 68edb6e | Antoine R. Dumont (@ardumont) | 19 February 2019, 12:36:38 UTC | Remove debian packaging from master branch Related T1532 | 19 February 2019, 12:36:38 UTC |
| 9534859 | Valentin Lorentz | 15 February 2019, 10:24:44 UTC | Fix documentation of CLI option --verbose. | 18 February 2019, 13:38:16 UTC |
| 4f968cb | Jenkins for Software Heritage | 15 February 2019, 14:15:26 UTC | Updated debian changelog for version 0.0.46 | 15 February 2019, 14:15:26 UTC |
| 240ac74 | Jenkins for Software Heritage | 15 February 2019, 14:15:26 UTC | Updated version 0.0.46 from 'debian/upstream/0.0.46' with Debian dir 303d76a0a55d623e545578a3c87e6cc7b259685f | 15 February 2019, 14:15:26 UTC |
| 349fc1c | Jenkins for Software Heritage | 15 February 2019, 14:15:25 UTC | New upstream version 0.0.46 | 15 February 2019, 14:15:25 UTC |
| 3c41cef | Antoine R. Dumont (@ardumont) | 15 February 2019, 13:56:04 UTC | scheduler.task: Remove no longer used Task class All scheduler tasks have been rewritten to avoid using the inheritance paradigm. The post worker startup initialization no longer creates automatically queues for registered tasks. Queues creation is managed through explicit configuration entries: celery: task_queues: ... task_modules: ... | 15 February 2019, 13:59:14 UTC |
| d611b5b | Jenkins for Software Heritage | 15 February 2019, 09:48:25 UTC | Updated debian changelog for version 0.0.45 | 15 February 2019, 09:48:25 UTC |
| 2d02485 | Jenkins for Software Heritage | 15 February 2019, 09:48:24 UTC | New upstream version 0.0.45 | 15 February 2019, 09:48:24 UTC |
| f42405a | Jenkins for Software Heritage | 15 February 2019, 09:48:24 UTC | Updated version 0.0.45 from 'debian/upstream/0.0.45' with Debian dir 4a6c99e168ef4182e4433ce3f2e38459be3cbae9 | 15 February 2019, 09:48:24 UTC |
| 65d0f73 | Antoine R. Dumont (@ardumont) | 15 February 2019, 09:41:00 UTC | celery_backend/config: Fix loglevel for amqp module | 15 February 2019, 09:43:00 UTC |
| 283cb7c | Antoine R. Dumont (@ardumont) | 14 February 2019, 08:42:26 UTC | tests: Use hypothesis profile to configure sample generation sizes | 14 February 2019, 08:42:26 UTC |
| 35b9979 | Jenkins for Software Heritage | 13 February 2019, 15:34:34 UTC | Updated debian changelog for version 0.0.44 | 13 February 2019, 15:34:34 UTC |
| cb7e2a4 | Jenkins for Software Heritage | 13 February 2019, 15:34:33 UTC | Updated version 0.0.44 from 'debian/upstream/0.0.44' with Debian dir 2ab2c749236a6007df63d225909cc2d4d92f1521 | 13 February 2019, 15:34:33 UTC |
| e152042 | Jenkins for Software Heritage | 13 February 2019, 15:34:33 UTC | New upstream version 0.0.44 | 13 February 2019, 15:34:33 UTC |
| c701c88 | Antoine R. Dumont (@ardumont) | 13 February 2019, 14:57:08 UTC | api/server: Do not read configuration at each request | 13 February 2019, 15:19:59 UTC |
| cf550ce | Jenkins for Software Heritage | 13 February 2019, 14:47:00 UTC | Updated debian changelog for version 0.0.43 | 13 February 2019, 14:47:00 UTC |
| a158f4c | Jenkins for Software Heritage | 13 February 2019, 14:46:59 UTC | Updated version 0.0.43 from 'debian/upstream/0.0.43' with Debian dir c95521bb0b3f7b9ebaa20f79ce5a175de0bb10b6 | 13 February 2019, 14:46:59 UTC |
| ac3f36a | Jenkins for Software Heritage | 13 February 2019, 14:46:58 UTC | New upstream version 0.0.43 | 13 February 2019, 14:46:58 UTC |
| f0a8c43 | David Douard | 13 February 2019, 13:37:18 UTC | listener: make the listener's queue name independent from the hostname the queue being durable and not auto deleted (auto_delete=False), we do not want a new queue to be spawned for each listener instance (eg. in a docker environment). | 13 February 2019, 13:37:18 UTC |
| b423c0b | David Douard | 13 February 2019, 13:28:37 UTC | runner: fix task_run configuration bootstrap ensure the task_run is created before sending the celery task; this later task could be executed before the db commit. As a result, the task_run may not have its 'started' field properly set; even the 'ended' and 'status'. | 13 February 2019, 13:28:37 UTC |
| 3d761f1 | David Douard | 13 February 2019, 13:25:58 UTC | sql: add the swh-lister-bitbucket-* task types | 13 February 2019, 13:25:58 UTC |
| 7e3f2fc | David Douard | 13 February 2019, 13:23:19 UTC | task: do not send the task-result-exception event in Task.on_failure() since it is unused; the task-failed event being already sent by celery and handled by the listener. | 13 February 2019, 13:23:19 UTC |
| 66518be | Jenkins for Software Heritage | 11 February 2019, 13:33:48 UTC | Updated debian changelog for version 0.0.42 | 11 February 2019, 13:33:48 UTC |
| 35898cb | Jenkins for Software Heritage | 11 February 2019, 13:33:47 UTC | New upstream version 0.0.42 | 11 February 2019, 13:33:47 UTC |
| bf3c8df | Jenkins for Software Heritage | 11 February 2019, 13:33:47 UTC | Updated version 0.0.42 from 'debian/upstream/0.0.42' with Debian dir c9c59b2c60063a71531e63afa4712a04985920de | 11 February 2019, 13:33:47 UTC |
| d870001 | Antoine R. Dumont (@ardumont) | 11 February 2019, 11:34:21 UTC | d/control: Enforce superior version for hypothesis Same as other swh modules (swh-web, swh-storage, swh-indexer). Inferior version do not follow test specifications. Related P356 | 11 February 2019, 11:34:21 UTC |
| 3488c26 | Antoine R. Dumont (@ardumont) | 11 February 2019, 10:44:40 UTC | requirements-test: Enforce a version for hypothesis Same as other swh modules (swh-web, swh-storage, swh-indexer) Inferior version do not follow the specifications Related P356 | 11 February 2019, 11:26:24 UTC |
| 2b00089 | Antoine R. Dumont (@ardumont) | 09 February 2019, 09:01:38 UTC | d/control: Add build dependency | 09 February 2019, 19:07:45 UTC |
| a66897a | Jenkins for Software Heritage | 06 February 2019, 15:33:04 UTC | Updated debian changelog for version 0.0.41 | 06 February 2019, 15:33:04 UTC |
| 59f50d0 | Jenkins for Software Heritage | 06 February 2019, 15:33:03 UTC | New upstream version 0.0.41 | 06 February 2019, 15:33:03 UTC |
| e6533ed | Jenkins for Software Heritage | 06 February 2019, 15:33:03 UTC | Updated version 0.0.41 from 'debian/upstream/0.0.41' with Debian dir db3eefcae33791c5d593a0e3ffba4c0359b678f9 | 06 February 2019, 15:33:03 UTC |
| db25694 | David Douard | 06 February 2019, 13:18:18 UTC | Fix a bug in the listener: commit() is not defined in the backend but on its connection. | 06 February 2019, 13:18:18 UTC |
| c29d383 | Nicolas Dandrimont | 19 December 2018, 14:59:38 UTC | Add basic stats to tasks This just increments a counter for started tasks, ended tasks, tasks that failed with an exception. It also registers a timer for every task run. Close T1460. | 06 February 2019, 12:06:39 UTC |
| abfe3db | David Douard | 01 February 2019, 15:57:29 UTC | Allow to override celery config file name via the SWH_CONFIG_FILENAME env var this will take precedence over the implicit config file scheme. The expected config file given via the environment variable is expected to have a [celery] section which will be used as config for the Celery app created in swh.scheduler.celery_backend.config. Related to T1410 and T826. | 06 February 2019, 09:25:07 UTC |
| 0f2f3ff | David Douard | 01 February 2019, 14:32:21 UTC | Make (celery) tests immune to environment variables especially CELERY_BROKER_URL... | 01 February 2019, 14:32:21 UTC |
| e2a91fb | Valentin Lorentz | 01 February 2019, 12:47:47 UTC | Remove call to tobytes(), BaseDB now handles conversion. | 01 February 2019, 14:03:48 UTC |
| 0c3306b | David Douard | 31 January 2019, 14:27:52 UTC | Fix the task_queues Celery config setting in build_app() ensure the config entry contains Queue objects. | 31 January 2019, 14:52:41 UTC |
| f188343 | David Douard | 31 January 2019, 14:26:57 UTC | cli: display a sorted list of task-types | 31 January 2019, 14:52:41 UTC |
| b0e6dd8 | David Douard | 31 January 2019, 12:37:38 UTC | Fix the listener: accessing the db cnx from the backend has changed | 31 January 2019, 14:52:41 UTC |
| 5d40529 | David Douard | 31 January 2019, 12:35:56 UTC | Make cli tools output logs on the console by default for other log levels than DEBUG, and add a --no-stdout option flag to disable this. | 31 January 2019, 14:52:41 UTC |
| a8bc684 | David Douard | 31 January 2019, 11:14:54 UTC | cli: build the celery app from a celery section of the given configuration file for runner and listener commands. related to T1410 | 31 January 2019, 14:52:32 UTC |
| 4246286 | David Douard | 31 January 2019, 11:13:44 UTC | Add a build_app() function to instantiate a Celery app with controlled config ie. being able to give the celery config dict as parameter. related to T1410 | 31 January 2019, 14:45:37 UTC |
| 70581b6 | David Douard | 30 January 2019, 09:43:40 UTC | Add a /site-map endpoint that lists published routes for this server | 31 January 2019, 10:15:50 UTC |
| 5880c52 | David Douard | 31 January 2019, 09:22:41 UTC | Activate the support for options from environment variables for swh-scheduler tool so that one can type (typically in a venv, with services running in dockers): (venv) swh-environment$ export SWH_SCHEDULER_URL=http://127.0.0.1:5008 (venv) swh-environment$ swh-scheduler task-type list | 31 January 2019, 09:29:43 UTC |
| 9bc5640 | David Douard | 31 January 2019, 09:21:13 UTC | Fix 'swh-scheduler runner' command: rollback() has beed removed from the SchedulerBackend and is not needed any more there. | 31 January 2019, 09:29:43 UTC |
| 174d89b | David Douard | 31 January 2019, 09:18:33 UTC | Fix get_scheduler's cls value when using 'swh-scheduler --url' cli option also ensure args dict does not have default db settings (unsupported by the RemoteScheduler class). | 31 January 2019, 09:29:43 UTC |
| 6da09a1 | David Douard | 30 January 2019, 15:55:31 UTC | Fix a regression introduced in 61c91b82 when deleting a click.option, one would better delete the function argument as well... | 30 January 2019, 15:58:11 UTC |
| b25c7cd | Valentin Lorentz | 30 January 2019, 10:27:00 UTC | Drop 'except Exception', it catches too many errors. eg. ImportErrors when negotiate is not installed. | 30 January 2019, 12:55:58 UTC |
| 84cded2 | David Douard | 30 January 2019, 11:21:28 UTC | BaseDb.copy_to's default_columns has been renamed as default_values | 30 January 2019, 11:22:51 UTC |
| 4fc7a89 | David Douard | 30 January 2019, 09:19:11 UTC | Make the prepare_event helper function pre-aggregate the events with same url and strip these urls as well. It makes no sense to register 2 different URLs when they are equal to a trailing ws detail. As a result, we must preaggregate them because the swh_cache_put() sql function won't allow several 'on conflict' for the same id. | 30 January 2019, 10:51:22 UTC |
| 4117d5a | David Douard | 29 January 2019, 14:34:04 UTC | Rewrite updater/test_backend.py with pytest and use the postgresql fixture also implement the test with more precise expected behavior, especially the content pre-aggregation of events and the url stripping. As is, this test will fail, the pre-aggregation and url stripping being implemented in the following revision. | 30 January 2019, 09:57:11 UTC |
| ebee014 | David Douard | 30 January 2019, 09:14:19 UTC | Kill DbBackend class we can now use directly the implementation of the copy_to() method from swh.core's BaseDb, so we just have to extract the format_query as a simple function (which it should have been since the beginning). Adapt updater/backend.py acordingly. | 30 January 2019, 09:57:11 UTC |
| f338b75 | David Douard | 29 January 2019, 13:29:22 UTC | Refactor swh/scheduler/updater as well - use the same config system as the main backend, with the same conventions, - do not make SchedulerUpdaterBackend inherit from DbBackend, use a simple association, similar to the SchedulerBackend class, - refactor the ghtorrent the same (explicit config), - same for the updater writer - move their main functions in cli.py - adapt tests accordingly | 30 January 2019, 09:49:59 UTC |
| 48e5372 | David Douard | 29 January 2019, 13:24:17 UTC | Fix tests for the scheduler and the API adapt the configuration to latest refactorings (especially the 'scheduling_db' config entry which is now just 'db'). Also refactor the scheduler tests to simplify the test class a bit: constants used in the test do not need to be in the setUp of the test class, making this setUp method useless. Doing so, ensure to use simple dicts as test constants instead of copy-the-modify them. This is (unit)test file, the dumbest, the better. | 30 January 2019, 09:49:59 UTC |
| 61c91b8 | David Douard | 29 January 2019, 13:11:07 UTC | Refactor config handling in cli.py Move the config file loading in the main cli group so that every command in this group have a consistent config loading behavior. This means that some cli commands "signatures" have changed: - every command now accepts a -C/--config-file option - the --cls has been dropped: either you give a config file, or passing a --database or --url option determine the 'class' of backend to use, - the api-server command 'config-path' argument has been dropped (use --config-file instead), | 30 January 2019, 09:49:59 UTC |
| 63af874 | David Douard | 29 January 2019, 12:50:14 UTC | Move sheduler's default conf in swh.scheduler | 30 January 2019, 09:49:59 UTC |
| a840ec4 | David Douard | 29 January 2019, 13:07:35 UTC | Make configuration of SWHElasticSearchClient use an explicit and consistent behavior instead of looking the config file for this entity in a hard to find dedicated file, expect a config object (dict) as constructor kwargs and look for its config under the 'elasticsearch' section. | 30 January 2019, 09:49:59 UTC |
