ffd2dda | Antoine R. Dumont (@ardumont) | 26 April 2018, 15:33:52 UTC | d/rules: Fix package build | 26 April 2018, 15:33:52 UTC |
950a7d5 | Antoine R. Dumont (@ardumont) | 25 April 2018, 14:10:26 UTC | swh.scheduler.tests: Test remote scheduler api as well Related T1036 | 25 April 2018, 16:37:43 UTC |
6ef0a88 | Antoine R. Dumont (@ardumont) | 25 April 2018, 11:03:44 UTC | swh.scheduler: Add tests around removing archivable tasks Related T986 Related T1034 | 25 April 2018, 16:36:34 UTC |
7afd050 | Antoine R. Dumont (@ardumont) | 25 April 2018, 09:57:17 UTC | swh.scheduler: Add tests around filtering archivable tasks Related T986 Related T1034 | 25 April 2018, 16:36:34 UTC |
5ccfa8b | Antoine R. Dumont (@ardumont) | 25 April 2018, 09:02:02 UTC | swh-scheduler-schema: Fix unneeded drop instructions Should not have been committed in that file, only in the migration part. | 25 April 2018, 16:36:34 UTC |
b7490ee | Antoine R. Dumont (@ardumont) | 24 April 2018, 14:48:05 UTC | swh.scheduler.cli: Improve docstring | 25 April 2018, 16:36:34 UTC |
8c3910a | Antoine R. Dumont (@ardumont) | 24 April 2018, 14:26:13 UTC | swh.scheduler.cli: Permit to specify the backend to use in cli Related T1034 | 25 April 2018, 16:36:34 UTC |
9e13fd2 | Antoine R. Dumont (@ardumont) | 24 April 2018, 13:40:33 UTC | swh.scheduler.api: Bootstrap scheduler's remote api Related T1034 | 25 April 2018, 16:36:34 UTC |
5349689 | Antoine R. Dumont (@ardumont) | 24 April 2018, 12:08:28 UTC | swh.scheduler: Use `get_scheduler` api to instantiate a scheduler Related T1034 | 24 April 2018, 14:55:43 UTC |
ef4eb14 | Antoine R. Dumont (@ardumont) | 24 April 2018, 12:07:48 UTC | swh.scheduler.backend: Fix docstring | 24 April 2018, 12:07:48 UTC |
8124229 | Antoine R. Dumont (@ardumont) | 18 April 2018, 09:33:45 UTC | swh.scheduler.cli.archive: Index arguments.kwargs as text Related T1023 | 18 April 2018, 09:33:45 UTC |
f4587a3 | Antoine R. Dumont (@ardumont) | 13 April 2018, 09:45:32 UTC | data/template: Do not index the arguments field (it's in _source) As this field is randomly large depending on the task at end, this triggers a limit (index.mapping.total_fields.limit to 1000). We do not really need this in the index as the data will still be in the _source. [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings Related T1023 | 13 April 2018, 09:59:37 UTC |
915db34 | Antoine R. Dumont (@ardumont) | 13 April 2018, 09:10:30 UTC | data/README: Add a small readme to explain es install step | 13 April 2018, 09:59:36 UTC |
547eb89 | Antoine R. Dumont (@ardumont) | 13 April 2018, 09:07:51 UTC | swh.scheduler.cli: Add a bulk index flag to separate read from index Try to reduce the number of connection timeout error. There are 3 solutions: - either increase the default timeout of 10 per request - same on server configuration basis (no) - either reduce the number of data to bulk index, this is the chosen solution Related T1023 | 13 April 2018, 09:59:27 UTC |
e972b6a | Antoine R. Dumont (@ardumont) | 10 April 2018, 14:21:57 UTC | swh.scheduler.cli.archive: Simplify task and task_run ids extraction Related T986 | 10 April 2018, 15:43:03 UTC |
6c11eb6 | Antoine R. Dumont (@ardumont) | 10 April 2018, 14:06:38 UTC | swh.sched.cli.archive: Improve logging Related T986 | 10 April 2018, 15:43:03 UTC |
1da2d71 | Antoine R. Dumont (@ardumont) | 10 April 2018, 13:53:05 UTC | swh.scheduler.cli.archive: Delete only completely indexed tasks Prior to this commit, it could happen that we removed tasks even though we did not yet index associated task_run. Related T986 | 10 April 2018, 15:42:57 UTC |
04ccc2d | Antoine R. Dumont (@ardumont) | 10 April 2018, 12:44:15 UTC | swh.sched.cli.archive: Use interval period to filter archival tasks Related T986 | 10 April 2018, 14:00:12 UTC |
962fd8b | Antoine R. Dumont (@ardumont) | 10 April 2018, 12:42:35 UTC | swh.scheduler.backend_es: Return operation failure instead of raising Prior to this commit, an error would raise and stop all indexation. As the code is already waiting for a tuple (operation-status, item), we instead leverage this and continue working on indexation. All items whose `operation-status` is False (meaning failure to index, whatever the reason) is not indexed. Another run would then pickup the leftover and index it. Related T986 | 10 April 2018, 13:55:38 UTC |
9b9b88c | Antoine R. Dumont (@ardumont) | 09 April 2018, 13:59:16 UTC | d/control: Update to recent python3-elasticsearch client | 09 April 2018, 14:09:12 UTC |
ffd00cb | Antoine R. Dumont (@ardumont) | 30 March 2018, 12:54:26 UTC | swh.scheduler.backend_es: Fix config base filename variable name | 30 March 2018, 12:55:25 UTC |
ad0f0e2 | Antoine R. Dumont (@ardumont) | 30 March 2018, 10:22:28 UTC | data/elastic-template.json: Use elasticsearch's default conf Defaults to 5 shards and 1 replica | 30 March 2018, 12:54:54 UTC |
d6b393d | Antoine R. Dumont (@ardumont) | 29 March 2018, 15:11:37 UTC | swh.scheduler.cli.archive: By default, archive last month's data Related T986 | 29 March 2018, 15:32:25 UTC |
e785252 | Antoine R. Dumont (@ardumont) | 29 March 2018, 08:47:08 UTC | swh.scheduler.cli: Use bulk api to index tasks Unfortunately, the bulk api does not return the full indexed data's original source [1] (which we need to identify the data clean up in the db). So we leverage elasticsearch's multi-get api to read back the original source. Related T986 | 29 March 2018, 10:32:37 UTC |
4d13f5d | Antoine R. Dumont (@ardumont) | 26 March 2018, 15:51:33 UTC | swh.scheduler.cli.archive: Improve dry-run behavior Related T986 | 26 March 2018, 15:56:00 UTC |
af4c9b8 | Antoine R. Dumont (@ardumont) | 26 March 2018, 15:32:57 UTC | swh.scheduler.cli.archive: Optimize reading task lookup query Related T986 | 26 March 2018, 15:32:57 UTC |
d83da86 | Antoine R. Dumont (@ardumont) | 22 March 2018, 16:16:12 UTC | swh.scheduler.cli.archive: Remove unnecessary order by condition Related T986 | 22 March 2018, 16:39:13 UTC |
bcb4f74 | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:57:35 UTC | swh.scheduler.cli.archive: Open batch size option for index/cleanup Related T986 | 22 March 2018, 16:39:13 UTC |
9e9f7ef | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:51:03 UTC | swh.scheduler.cli.archive: Stream indexed tasks removal Related T986 | 22 March 2018, 15:51:03 UTC |
e3867ed | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:50:07 UTC | swh.scheduler.cli.archive: Fix edge case when reading tasks Related T986 | 22 March 2018, 15:50:07 UTC |
b76384b | Antoine R. Dumont (@ardumont) | 22 March 2018, 12:51:32 UTC | swh.scheduler.cli: Open endpoint to archive tasks in elasticsearch Related T986 | 22 March 2018, 12:51:48 UTC |
8435efd | Antoine R. Dumont (@ardumont) | 21 March 2018, 18:58:45 UTC | swh.scheduler.cli: Add sql function to list archivable tasks Related T986 Related P240 | 21 March 2018, 18:58:45 UTC |
0324fb2 | Antoine R. Dumont (@ardumont) | 15 March 2018, 10:21:06 UTC | sql/data: Fix missing num_retries in deposit task types | 15 March 2018, 10:21:06 UTC |
367ab15 | Antoine R. Dumont (@ardumont) | 14 March 2018, 12:31:59 UTC | swh.core.serializer: Move to msgpack serialization format This permits to serialize bytes which was an existing behavior in celery 3.1 (when pickle was the default format). Related T782 Related D300 | 14 March 2018, 12:31:59 UTC |
a474ed5 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:56:03 UTC | swh.scheduler.cli: Unify pretty print output | 09 March 2018, 15:56:03 UTC |
0e8e237 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:51:04 UTC | sql/data: Add new task type for loading mercurial dump | 09 March 2018, 15:54:32 UTC |
5a70627 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:50:41 UTC | swh.scheduler.cli: Add sample use case for the scheduling cli | 09 March 2018, 15:54:32 UTC |
3d68f20 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:50:11 UTC | swh.scheduler.cli: Open policy column to the scheduling cli | 09 March 2018, 15:54:31 UTC |
185f4b3 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:44:19 UTC | swh.scheduler.cli: Open the delimiter option as cli argument When providing json values (args|kwargs column), csv's default comma separator and json's default separator values clash. So the cli fails to schedule task. This commit permits to overcome that issue by providing the delimiter ';' to the cli call when json is required. | 09 March 2018, 15:52:49 UTC |
4b918af | Antoine R. Dumont (@ardumont) | 08 March 2018, 10:27:47 UTC | Fix issue when updating task-type without any retry delay defined Related P231 Related T902 | 08 March 2018, 10:27:47 UTC |
f8ddae1 | Antoine R. Dumont (@ardumont) | 08 March 2018, 10:27:26 UTC | Fix pep8 violation regarding bare except | 08 March 2018, 10:27:26 UTC |
46329da | Antoine R. Dumont (@ardumont) | 19 December 2017, 13:35:40 UTC | swh-scheduler/data: Add new oneshot scheduling load-mercurial task Related T908 | 19 December 2017, 13:37:59 UTC |
b74c89f | Antoine Pietri | 14 December 2017, 17:26:59 UTC | backend: fix default scheduling_db value for consistency | 14 December 2017, 17:26:59 UTC |
5b17566 | Antoine Pietri | 14 December 2017, 16:53:30 UTC | backend: doc: fix return value of create_tasks | 14 December 2017, 16:53:30 UTC |
f08d9e7 | Antoine R. Dumont (@ardumont) | 07 December 2017, 14:18:31 UTC | Reference new task_type in scheduler | 07 December 2017, 14:18:31 UTC |
255d851 | Antoine R. Dumont (@ardumont) | 29 November 2017, 11:09:32 UTC | swh.scheduler.utils: Open utility function to create oneshot task | 29 November 2017, 11:30:35 UTC |
723221e | Nicolas Dandrimont | 08 November 2017, 15:38:52 UTC | Update for celery 4 | 08 November 2017, 16:00:50 UTC |
09b82c9 | Stefano Zacchiroli | 02 November 2017, 10:09:25 UTC | docs: add absolute anchor to documentation index | 02 November 2017, 10:09:25 UTC |
cd827b6 | Nicolas Dandrimont | 12 October 2017, 16:47:31 UTC | Add missing __init__.py files | 12 October 2017, 16:48:15 UTC |
0a3d8ec | Nicolas Dandrimont | 12 October 2017, 15:16:58 UTC | Cleanup packaging | 12 October 2017, 15:17:55 UTC |
79b0ccd | Antoine R. Dumont (@ardumont) | 11 October 2017, 08:43:27 UTC | swh.scheduler.cli: Fix missing task type option | 11 October 2017, 08:57:49 UTC |
d5f2d51 | Nicolas Dandrimont | 11 September 2017, 12:10:41 UTC | sql/updates/06: add name to check constraint | 11 September 2017, 12:10:41 UTC |
73f4aee | Nicolas Dandrimont | 11 September 2017, 12:01:31 UTC | sql: one-shot tasks don't really have an interval | 11 September 2017, 12:06:56 UTC |
b4bbc05 | Stefano Zacchiroli | 06 September 2017, 18:45:30 UTC | sanitize docstrings for sphinx | 06 September 2017, 18:45:30 UTC |
ca0230d | Nicolas Dandrimont | 01 September 2017, 18:20:42 UTC | sql: clean up schema syntax issues | 01 September 2017, 18:21:29 UTC |
f785a81 | Stefano Zacchiroli | 30 August 2017, 10:26:02 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:02 UTC |
89625c1 | Nicolas Dandrimont | 20 August 2017, 02:58:55 UTC | updates/05: fix SQL syntax to let the update run | 20 August 2017, 02:58:55 UTC |
ee545d4 | Nicolas Dandrimont | 07 August 2017, 16:43:57 UTC | test_scheduler: add missing db attribute to database tests | 07 August 2017, 16:43:57 UTC |
deb40f2 | Nicolas Dandrimont | 07 August 2017, 16:33:40 UTC | requirements: bump swh.core to 0.0.34 | 07 August 2017, 16:33:54 UTC |
8348905 | Nicolas Dandrimont | 07 August 2017, 16:07:30 UTC | sql: add update script from 04 to 05 | 07 August 2017, 16:07:30 UTC |
3194c2b | Nicolas Dandrimont | 25 July 2017, 16:49:17 UTC | sql: new logic for recurrence of one-shot tasks | 25 July 2017, 16:49:17 UTC |
030a51f | Nicolas Dandrimont | 25 July 2017, 15:42:31 UTC | sql: add the recurrence policy to tasks and related fields to task_type This is the groundwork for the implementation of T720: - it adds the minimal set of fields required to allow for a task to be retried on transient failures; - it adds knowledge of whether a task is one-shot or recurring | 25 July 2017, 15:44:40 UTC |
adb5afe | Nicolas Dandrimont | 25 July 2017, 13:30:48 UTC | test_scheduler: more generic tests wrt task types | 25 July 2017, 13:30:48 UTC |
31a6697 | Nicolas Dandrimont | 25 July 2017, 12:39:26 UTC | celery_backend.runner: respect the task's maximum queue length Close T726 | 25 July 2017, 12:39:26 UTC |
a0420b0 | Nicolas Dandrimont | 21 July 2017, 16:57:18 UTC | backend: Add max queue length to task types One more stepping stone towards T726 | 21 July 2017, 16:57:18 UTC |
4fe37f7 | Nicolas Dandrimont | 21 July 2017, 14:09:55 UTC | backend: add get_tasks method | 21 July 2017, 14:09:55 UTC |
5ccc579 | Nicolas Dandrimont | 21 July 2017, 13:58:35 UTC | backend: add task_type argument to peek/grab_ready_tasks | 21 July 2017, 13:58:35 UTC |
af220e2 | Nicolas Dandrimont | 19 July 2017, 18:02:11 UTC | backend: add tests for grab_ready_tasks | 19 July 2017, 18:02:11 UTC |
d8945ae | Nicolas Dandrimont | 19 July 2017, 14:39:42 UTC | backend: add peek_ready_tasks tests | 19 July 2017, 14:39:42 UTC |
d74d79f | Nicolas Dandrimont | 19 July 2017, 12:04:54 UTC | tests: add create_tasks test | 19 July 2017, 12:10:21 UTC |
0b88b14 | Nicolas Dandrimont | 19 July 2017, 10:08:30 UTC | backend: add get_task_types | 19 July 2017, 10:08:30 UTC |
2e2205c | Nicolas Dandrimont | 19 July 2017, 10:06:03 UTC | tests: refactor and extend | 19 July 2017, 10:07:01 UTC |
d4eb07e | Nicolas Dandrimont | 17 July 2017, 17:28:43 UTC | Add test infrastructure for the scheduler backend | 17 July 2017, 17:31:19 UTC |
285e7b5 | Nicolas Dandrimont | 17 July 2017, 17:04:32 UTC | Add sql Makefile | 17 July 2017, 17:04:32 UTC |
0e14eff | Nicolas Dandrimont | 19 June 2017, 17:00:18 UTC | celery_backend.config: add a shortcut to get a queue length | 19 June 2017, 17:00:18 UTC |
7caf543 | Nicolas Dandrimont | 19 June 2017, 14:03:19 UTC | celery_backend.config: add a method to get queue statistics from RabbitMQ First part of implementing T726 | 19 June 2017, 14:03:19 UTC |
54bcdda | Nicolas Dandrimont | 19 June 2017, 09:51:39 UTC | swh-worker-control: remove spurious print | 19 June 2017, 09:51:39 UTC |
80851f5 | Antoine Pietri | 15 June 2017, 15:13:22 UTC | celery: enable console logging on debug loglevel | 15 June 2017, 15:13:22 UTC |
7c1c041 | Nicolas Dandrimont | 12 June 2017, 15:49:58 UTC | listener: update for the new task-result event | 12 June 2017, 15:49:58 UTC |
d3abfa4 | Nicolas Dandrimont | 12 June 2017, 15:49:12 UTC | listener: derive the reliable events receiver from the base celery class | 12 June 2017, 15:49:12 UTC |
671a2c0 | Nicolas Dandrimont | 12 June 2017, 13:30:02 UTC | task: move run to run_task to allow overriding the default behavior of tasks This is an API break, but it's just a matter of renaming all existing tasks to run_task instead of run. Close T536 | 12 June 2017, 13:30:02 UTC |
07ea766 | Nicolas Dandrimont | 31 May 2017, 16:27:53 UTC | swh-worker-control: read the remote monotonic time to get task start times | 31 May 2017, 16:27:53 UTC |
c02757c | Nicolas Dandrimont | 31 May 2017, 16:27:00 UTC | celery_backend: add a remote-control command to get the monotonic clock | 31 May 2017, 16:27:00 UTC |
324a3cc | Nicolas Dandrimont | 31 May 2017, 15:49:06 UTC | bin/swh-worker-control: add command-line interface for worker remote control | 31 May 2017, 15:51:04 UTC |
f758291 | Nicolas Dandrimont | 07 April 2017, 09:57:36 UTC | requirements-swh.txt: use proper syntax | 07 April 2017, 09:57:36 UTC |
a760866 | Nicolas Dandrimont | 07 April 2017, 09:53:08 UTC | d/copyright: cleanup | 07 April 2017, 09:53:08 UTC |
35997f8 | Nicolas Dandrimont | 07 April 2017, 09:52:27 UTC | celery_backend.config: use the JournalHandler for logging | 07 April 2017, 09:52:27 UTC |
aaa5dfd | Nicolas Dandrimont | 09 March 2017, 10:12:09 UTC | celery_backend.config: only log to database if log_db is present | 09 March 2017, 10:12:09 UTC |
0767b4c | Antoine Pietri | 15 February 2017, 14:23:52 UTC | requirements.txt: ensure we use celery<4 (we don't support other versions for now) | 15 February 2017, 14:23:52 UTC |
6192ac1 | Antoine R. Dumont (@ardumont) | 10 February 2017, 14:03:02 UTC | Add utils.get_task fn to simplify contextual task instance retrieval Permits to unify the task instance retrieval, which then permits to send messages to the associated task's queue. | 10 February 2017, 14:11:54 UTC |
1aedc67 | Antoine Pietri | 09 February 2017, 11:21:06 UTC | requirements: split internal and external requirements in two separate files | 09 February 2017, 13:32:06 UTC |
d65742e | Antoine Pietri | 09 February 2017, 11:19:06 UTC | celery_backend: pep8: fix missing blank lines | 09 February 2017, 11:21:23 UTC |
05f41c2 | Nicolas Dandrimont | 21 October 2016, 14:09:11 UTC | sql: fix default_interval selection for new tasks | 21 October 2016, 14:09:11 UTC |
b048926 | Nicolas Dandrimont | 20 October 2016, 14:02:57 UTC | backend: the status of a task is task.status, not task.task_status... | 20 October 2016, 14:02:57 UTC |
da7b1b7 | Nicolas Dandrimont | 19 October 2016, 13:39:05 UTC | backend: allow disabling tasks | 19 October 2016, 13:39:05 UTC |
0fb8eef | Antoine R. Dumont (@ardumont) | 23 August 2016, 16:44:15 UTC | Fix docstring typo | 05 September 2016, 12:27:28 UTC |
51cb666 | Nicolas Dandrimont | 01 September 2016, 15:23:15 UTC | celery_backend.config: add SWH_WORKER_INSTANCE environment variable This allows to run several workers with different configuration files. | 01 September 2016, 15:23:15 UTC |
dced5d5 | Nicolas Dandrimont | 24 February 2016, 15:36:52 UTC | celery_backend.config: Put pickle last in the accepted encodings | 24 February 2016, 15:36:52 UTC |
7d6e331 | Nicolas Dandrimont | 24 February 2016, 15:36:02 UTC | backend: allow for mass-scheduling of task runs This allows the runner to be (~10 times) more efficient when running tasks | 24 February 2016, 15:36:02 UTC |
15c3d8f | Nicolas Dandrimont | 24 February 2016, 11:10:51 UTC | backend: use copy for mass-scheduling of tasks | 24 February 2016, 11:10:51 UTC |