https://forge.softwareheritage.org/source/swh-scheduler.git

sort by:
Revision Author Date Message Commit Date
c2b1793 requirements.txt: Remove kombu dependency This actually conflicts with the latest celery version. We should not depend on it as this is drawn by celery already. 21 March 2019, 10:46:17 UTC
4ee93d3 API: use default's APIError exception instead of the SchedulerAPIError this later does not brings anything over the default's one. This requires swh.core >= 0.0.56 18 March 2019, 16:02:14 UTC
693a147 Remove info-level logs about already acknowledged messages Celery voluntarily pools messages to reduce traffic, so it's expected that some messages end up here several times. No need to spam logs about it. 15 March 2019, 13:25:15 UTC
763dbda Ignore INFO messages when testing CLI output. 14 March 2019, 15:36:08 UTC
1122ee1 Add a 'task schedule_origins' command to run a task on all origins. 14 March 2019, 15:36:08 UTC
6b795e3 Introduce SWH_LOG_TARGET to allow switching between console and journal The current state of the celery worker/utility logging setup makes us log twice: the default value of log_console can't be overridden by the cli flag in celery workers. The environment variable can be set to journal when running as a systemd service; this shuts down the console logger, and allows the metadata added to log messages to pass through. 13 March 2019, 14:05:24 UTC
6dbfd06 Remove ghtorrent/fake.py, it's not used anymore. 13 March 2019, 09:17:27 UTC
e6ee2a7 Add tests for 'swh-scheduler task list-pending' and 'swh-scheduler task list'. 13 March 2019, 08:58:19 UTC
da290c2 Add support for non-string options when scheduling tasks. This also fixes the pretty-printing of tasks, which was ambiguous (`42` and `"42"` where both printed as `42`). 13 March 2019, 08:57:01 UTC
b0aea8c tests: enforce LC_ALL since test_cli depends on it 06 March 2019, 11:45:38 UTC
dc6afe4 Revert "Hack to fix the ImportError in updater/ghtorrent/fake.py." This reverts commit 3939a41d73bdef94913a6e8c3aa212532d281463. Wasn't supposed to be committed yet. 01 March 2019, 10:16:24 UTC
5499400 Move option parsing to its own (tested) function. 01 March 2019, 10:14:06 UTC
3939a41 Hack to fix the ImportError in updater/ghtorrent/fake.py. Sphinx complains about it. 28 February 2019, 17:09:56 UTC
f698b01 Move parts of 30-swh-schema.sql to 40-swh-func.sql and 60-swh-indexes.sql. For uniformity with other SWH packages. 28 February 2019, 13:42:06 UTC
7ddfdf4 celery_backend.listener: Fix debug output format string 27 February 2019, 16:51:40 UTC
099effd Start writing tests for the CLI. 27 February 2019, 11:16:52 UTC
40d32b4 celery: fix the celery config to ensure the default 'celery' queue exists as it used to be before the recent refactorings. 25 February 2019, 13:10:00 UTC
55f7e1f celery: do not set the default value for celery's result_backend in production, we do not want a result backend to be used, and the pytest fixture handle this configuration properly. Closes T1539. 25 February 2019, 13:10:00 UTC
5b08922 cli: kill a useless statement the Flask app object does not need a scheduler attribute. 25 February 2019, 13:02:59 UTC
cfaf833 Remove no longer used default config path Related T1533 22 February 2019, 23:35:13 UTC
ce90908 swh.scheduler.api: Make api use explicit configuration (+ checks) Related T1533 22 February 2019, 23:23:43 UTC
1d4982a sql/30-swh-schema.sql: Fix comment instruction To help ci build: https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DCIDX/job/gbp-buildpackage/30/console 22 February 2019, 15:09:46 UTC
981d3f9 Fix crash of SchedulerBackend.search_tasks when no argument is given. 20 February 2019, 16:16:47 UTC
058f120 cli: add a --runs option to 'swh-scheduler task list' tool this options allws to list task runs for selected tasks. 20 February 2019, 09:23:20 UTC
7f951d3 Keep the main loaded configuration as swh.scheduler.CONFIG so that it can be used in tasks instead of reloading config files on each task execution. 20 February 2019, 09:23:20 UTC
73bca41 celery_backend: remove debug statements and reduce info level verbosity by lowering the logging statement in catchall_event to DEBUG. 20 February 2019, 08:17:36 UTC
68edb6e Remove debian packaging from master branch Related T1532 19 February 2019, 12:36:38 UTC
9534859 Fix documentation of CLI option --verbose. 18 February 2019, 13:38:16 UTC
3c41cef scheduler.task: Remove no longer used Task class All scheduler tasks have been rewritten to avoid using the inheritance paradigm. The post worker startup initialization no longer creates automatically queues for registered tasks. Queues creation is managed through explicit configuration entries: celery: task_queues: ... task_modules: ... 15 February 2019, 13:59:14 UTC
65d0f73 celery_backend/config: Fix loglevel for amqp module 15 February 2019, 09:43:00 UTC
283cb7c tests: Use hypothesis profile to configure sample generation sizes 14 February 2019, 08:42:26 UTC
c701c88 api/server: Do not read configuration at each request 13 February 2019, 15:19:59 UTC
f0a8c43 listener: make the listener's queue name independent from the hostname the queue being durable and not auto deleted (auto_delete=False), we do not want a new queue to be spawned for each listener instance (eg. in a docker environment). 13 February 2019, 13:37:18 UTC
b423c0b runner: fix task_run configuration bootstrap ensure the task_run is created before sending the celery task; this later task could be executed before the db commit. As a result, the task_run may not have its 'started' field properly set; even the 'ended' and 'status'. 13 February 2019, 13:28:37 UTC
3d761f1 sql: add the swh-lister-bitbucket-* task types 13 February 2019, 13:25:58 UTC
7e3f2fc task: do not send the task-result-exception event in Task.on_failure() since it is unused; the task-failed event being already sent by celery and handled by the listener. 13 February 2019, 13:23:19 UTC
3488c26 requirements-test: Enforce a version for hypothesis Same as other swh modules (swh-web, swh-storage, swh-indexer) Inferior version do not follow the specifications Related P356 11 February 2019, 11:26:24 UTC
db25694 Fix a bug in the listener: commit() is not defined in the backend but on its connection. 06 February 2019, 13:18:18 UTC
c29d383 Add basic stats to tasks This just increments a counter for started tasks, ended tasks, tasks that failed with an exception. It also registers a timer for every task run. Close T1460. 06 February 2019, 12:06:39 UTC
abfe3db Allow to override celery config file name via the SWH_CONFIG_FILENAME env var this will take precedence over the implicit config file scheme. The expected config file given via the environment variable is expected to have a [celery] section which will be used as config for the Celery app created in swh.scheduler.celery_backend.config. Related to T1410 and T826. 06 February 2019, 09:25:07 UTC
0f2f3ff Make (celery) tests immune to environment variables especially CELERY_BROKER_URL... 01 February 2019, 14:32:21 UTC
e2a91fb Remove call to tobytes(), BaseDB now handles conversion. 01 February 2019, 14:03:48 UTC
0c3306b Fix the task_queues Celery config setting in build_app() ensure the config entry contains Queue objects. 31 January 2019, 14:52:41 UTC
f188343 cli: display a sorted list of task-types 31 January 2019, 14:52:41 UTC
b0e6dd8 Fix the listener: accessing the db cnx from the backend has changed 31 January 2019, 14:52:41 UTC
5d40529 Make cli tools output logs on the console by default for other log levels than DEBUG, and add a --no-stdout option flag to disable this. 31 January 2019, 14:52:41 UTC
a8bc684 cli: build the celery app from a celery section of the given configuration file for runner and listener commands. related to T1410 31 January 2019, 14:52:32 UTC
4246286 Add a build_app() function to instantiate a Celery app with controlled config ie. being able to give the celery config dict as parameter. related to T1410 31 January 2019, 14:45:37 UTC
70581b6 Add a /site-map endpoint that lists published routes for this server 31 January 2019, 10:15:50 UTC
5880c52 Activate the support for options from environment variables for swh-scheduler tool so that one can type (typically in a venv, with services running in dockers): (venv) swh-environment$ export SWH_SCHEDULER_URL=http://127.0.0.1:5008 (venv) swh-environment$ swh-scheduler task-type list 31 January 2019, 09:29:43 UTC
9bc5640 Fix 'swh-scheduler runner' command: rollback() has beed removed from the SchedulerBackend and is not needed any more there. 31 January 2019, 09:29:43 UTC
174d89b Fix get_scheduler's cls value when using 'swh-scheduler --url' cli option also ensure args dict does not have default db settings (unsupported by the RemoteScheduler class). 31 January 2019, 09:29:43 UTC
6da09a1 Fix a regression introduced in 61c91b82 when deleting a click.option, one would better delete the function argument as well... 30 January 2019, 15:58:11 UTC
b25c7cd Drop 'except Exception', it catches too many errors. eg. ImportErrors when negotiate is not installed. 30 January 2019, 12:55:58 UTC
84cded2 BaseDb.copy_to's default_columns has been renamed as default_values 30 January 2019, 11:22:51 UTC
4fc7a89 Make the prepare_event helper function pre-aggregate the events with same url and strip these urls as well. It makes no sense to register 2 different URLs when they are equal to a trailing ws detail. As a result, we must preaggregate them because the swh_cache_put() sql function won't allow several 'on conflict' for the same id. 30 January 2019, 10:51:22 UTC
4117d5a Rewrite updater/test_backend.py with pytest and use the postgresql fixture also implement the test with more precise expected behavior, especially the content pre-aggregation of events and the url stripping. As is, this test will fail, the pre-aggregation and url stripping being implemented in the following revision. 30 January 2019, 09:57:11 UTC
ebee014 Kill DbBackend class we can now use directly the implementation of the copy_to() method from swh.core's BaseDb, so we just have to extract the format_query as a simple function (which it should have been since the beginning). Adapt updater/backend.py acordingly. 30 January 2019, 09:57:11 UTC
f338b75 Refactor swh/scheduler/updater as well - use the same config system as the main backend, with the same conventions, - do not make SchedulerUpdaterBackend inherit from DbBackend, use a simple association, similar to the SchedulerBackend class, - refactor the ghtorrent the same (explicit config), - same for the updater writer - move their main functions in cli.py - adapt tests accordingly 30 January 2019, 09:49:59 UTC
48e5372 Fix tests for the scheduler and the API adapt the configuration to latest refactorings (especially the 'scheduling_db' config entry which is now just 'db'). Also refactor the scheduler tests to simplify the test class a bit: constants used in the test do not need to be in the setUp of the test class, making this setUp method useless. Doing so, ensure to use simple dicts as test constants instead of copy-the-modify them. This is (unit)test file, the dumbest, the better. 30 January 2019, 09:49:59 UTC
61c91b8 Refactor config handling in cli.py Move the config file loading in the main cli group so that every command in this group have a consistent config loading behavior. This means that some cli commands "signatures" have changed: - every command now accepts a -C/--config-file option - the --cls has been dropped: either you give a config file, or passing a --database or --url option determine the 'class' of backend to use, - the api-server command 'config-path' argument has been dropped (use --config-file instead), 30 January 2019, 09:49:59 UTC
63af874 Move sheduler's default conf in swh.scheduler 30 January 2019, 09:49:59 UTC
a840ec4 Make configuration of SWHElasticSearchClient use an explicit and consistent behavior instead of looking the config file for this entity in a hard to find dedicated file, expect a config object (dict) as constructor kwargs and look for its config under the 'elasticsearch' section. 30 January 2019, 09:49:59 UTC
488b154 Remove the main function from listener.py it's now in cli.py 30 January 2019, 09:49:59 UTC
cebc11d Refactor the scheduler's backend to use swh.core.db.BaseDb (part 2) so it uses a proper connection pool to access the database. 30 January 2019, 09:46:29 UTC
94258bd Refactor the scheduler's backend to use swh.core.db.BaseDb so it uses a proper connection pool to access the database. 30 January 2019, 09:33:54 UTC
35c285f Add debug statements in the runner 30 January 2019, 09:33:54 UTC
7f51159 Make the 'runner' cli command a bit less verbose do only report the number of scheduled tasks at info level, if any, and lower the 'Run ready tasks' log message at debug level. 30 January 2019, 09:33:54 UTC
63e750a Small fix in the 'task respawn' cli command when respawning several tasks at once. 30 January 2019, 09:33:54 UTC
1203544 Do not crash the listener if a message has already been ack'ed Note sure whether we should be concerned by the fact this sometimes occurs (under heavy load in the docker env). 30 January 2019, 09:33:54 UTC
0252051 prevent pytest from displaying gazillions of warnings especially the infamous psycopg2 one. 30 January 2019, 09:33:54 UTC
7b81304 Increase the max queue size of the origin_metadata indexer. It's going very fast now, enough to empty its queue between two scheduler-runner runs. 29 January 2019, 16:12:25 UTC
53136b7 swh.scheduler.tests: Mark db tests as such This will work around the current debian package build failure Related T1498 28 January 2019, 15:22:53 UTC
e26aec7 Force tox environment to C.UTF-8 locale Works around https://github.com/ClearcodeHQ/pytest-postgresql/issues/16 18 January 2019, 16:45:02 UTC
df1cca3 Add debug logging in the SWHTask class instead of replicating the same logging code everywhere. 17 January 2019, 10:24:47 UTC
caaa44b Revert "tox: pifpaf is not needed any more" This reverts commit f267f454fb91b421994f4ceabb50baa70982b76d. Pushed by mistake. 16 January 2019, 08:56:32 UTC
f267f45 tox: pifpaf is not needed any more 15 January 2019, 17:11:48 UTC
99bf79e enforce dep on pytest<4 see https://github.com/pytest-dev/pytest/issues/4641 It should be possible to let this constraint go when celery 4.3 is out 15 January 2019, 17:11:37 UTC
e1c25d2 Replace task tests (in test_task.py) with pytest-based ones also remove now useless celery_testing.py and scheduler_testing.py files. 15 January 2019, 16:39:32 UTC
e741c93 Add a 'swh_scheduler' pytest fixture that creates a swh scheduler usable for tests this uses the pytest-postgresql package to manage the database life cycle. 15 January 2019, 16:39:10 UTC
9725365 Configure the celery result backend (as 'rpc://') by default the result backend must be configured to be able to save (celery) group results (as it is now the case for lister tasks for example). 15 January 2019, 14:02:06 UTC
6cb1813 Add a few debug statements in cli 15 January 2019, 14:02:06 UTC
c0f9320 Add a new 'task respawn' command that allows to respawn any task immediately (or any later date). 15 January 2019, 14:02:01 UTC
cafcb46 Add a 'next_run' argument to the SchedulerBackend.set_status_tasks method so this method can be used to respawn a task immediately. 15 January 2019, 13:57:44 UTC
1aadc10 Add a 'task list' cli command to list tasks with search criterions and not only pending tasks. 15 January 2019, 13:57:39 UTC
b3fd48e Add a 'full' flag argument to the pretty_print_task function to display also the status and priority fields. 15 January 2019, 13:49:49 UTC
e0f224e Add a SchedulerBackend.search_tasks method 15 January 2019, 09:20:22 UTC
2dbdb6c Add a new 'task-type add' cli command 15 January 2019, 09:20:22 UTC
c3bb48d Make SchedulerBackend.create_task_type work with only a subset of keys Not all keys are mandatory, so do not expect all the possible keys for the task_type table to be given when calling create_task_type(). Note: no validation is made whether the given set of keys fullfill the table constraints. 15 January 2019, 09:20:22 UTC
4c75330 Add a new task-type cli command group and move the task_type listing in there 14 January 2019, 10:38:19 UTC
d658f93 Add a few debug statements in swh.sceduler.backend 14 January 2019, 10:31:14 UTC
6bf1110 Fix swh.scheduler.compute_nb_tasks_from function to return integers instead of float numbers. 14 January 2019, 10:31:14 UTC
996e905 Add a --priority cmdline option to the 'swh-scheduler task add' tool 14 January 2019, 10:31:09 UTC
9f33c05 Make the SWHTask class the default base class for tasks and add some kind of unit tests. 10 January 2019, 14:43:40 UTC
20be147 Kill swh.scheduler.api.server.launch() in favor of the 'swh-scheduler api-server' command. 10 January 2019, 14:43:40 UTC
5c02d51 Add a logging statement in the runner cli command also log an eventual exception instead of crashing the runner service. 10 January 2019, 14:43:39 UTC
292e5dc Kill the CustomCelery class use functions instead of methods. This is required to be able to use celery pytest fixtures so one can really test celery tasks (especially when a task spawns sub tasks). one (get_queue_lenth) of the 3 methods has been added as (monkeypatched) method on the Celery class for the sake of bw compat, but this should really be removed as well as soon as possible (seems only used in swh-archiver). 10 January 2019, 14:43:39 UTC
96ad58b Move logging configuration into the cli group function so that logging level config can be consistently set for all swh-scheduler commands. 10 January 2019, 14:43:39 UTC
69f1759 Move the scheduler verification from the main cli group definition to subcommands since it's actually the responsibility of each subcommand to decide whether it can run without a properly configured scheduler instance. This is also required so the user can run: swh-scheduler subcommand --help even with a non-properly configured scheduler. 10 January 2019, 14:43:39 UTC
df075ba Add 3 cli commands: runner, listener and api-server These commands do what they say, ie. start a runner, listener or API server process. Note that processes are not daemonized and run in front. Typically used as: swh-scheduler --cls local --database postgresql:///?service=swh api-server --host 127.0.0.1 --port 5008 swh-scheduler --cls remote --url http://127.0.0.1:5008 runner --period 10 swh-scheduler --cls remote --url http://127.0.0.1:5008 listener 10 January 2019, 14:43:39 UTC
back to top