d6b393d | Antoine R. Dumont (@ardumont) | 29 March 2018, 15:11:37 UTC | swh.scheduler.cli.archive: By default, archive last month's data Related T986 | 29 March 2018, 15:32:25 UTC |
e785252 | Antoine R. Dumont (@ardumont) | 29 March 2018, 08:47:08 UTC | swh.scheduler.cli: Use bulk api to index tasks Unfortunately, the bulk api does not return the full indexed data's original source [1] (which we need to identify the data clean up in the db). So we leverage elasticsearch's multi-get api to read back the original source. Related T986 | 29 March 2018, 10:32:37 UTC |
4d13f5d | Antoine R. Dumont (@ardumont) | 26 March 2018, 15:51:33 UTC | swh.scheduler.cli.archive: Improve dry-run behavior Related T986 | 26 March 2018, 15:56:00 UTC |
af4c9b8 | Antoine R. Dumont (@ardumont) | 26 March 2018, 15:32:57 UTC | swh.scheduler.cli.archive: Optimize reading task lookup query Related T986 | 26 March 2018, 15:32:57 UTC |
d83da86 | Antoine R. Dumont (@ardumont) | 22 March 2018, 16:16:12 UTC | swh.scheduler.cli.archive: Remove unnecessary order by condition Related T986 | 22 March 2018, 16:39:13 UTC |
bcb4f74 | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:57:35 UTC | swh.scheduler.cli.archive: Open batch size option for index/cleanup Related T986 | 22 March 2018, 16:39:13 UTC |
9e9f7ef | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:51:03 UTC | swh.scheduler.cli.archive: Stream indexed tasks removal Related T986 | 22 March 2018, 15:51:03 UTC |
e3867ed | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:50:07 UTC | swh.scheduler.cli.archive: Fix edge case when reading tasks Related T986 | 22 March 2018, 15:50:07 UTC |
b76384b | Antoine R. Dumont (@ardumont) | 22 March 2018, 12:51:32 UTC | swh.scheduler.cli: Open endpoint to archive tasks in elasticsearch Related T986 | 22 March 2018, 12:51:48 UTC |
8435efd | Antoine R. Dumont (@ardumont) | 21 March 2018, 18:58:45 UTC | swh.scheduler.cli: Add sql function to list archivable tasks Related T986 Related P240 | 21 March 2018, 18:58:45 UTC |
0324fb2 | Antoine R. Dumont (@ardumont) | 15 March 2018, 10:21:06 UTC | sql/data: Fix missing num_retries in deposit task types | 15 March 2018, 10:21:06 UTC |
367ab15 | Antoine R. Dumont (@ardumont) | 14 March 2018, 12:31:59 UTC | swh.core.serializer: Move to msgpack serialization format This permits to serialize bytes which was an existing behavior in celery 3.1 (when pickle was the default format). Related T782 Related D300 | 14 March 2018, 12:31:59 UTC |
a474ed5 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:56:03 UTC | swh.scheduler.cli: Unify pretty print output | 09 March 2018, 15:56:03 UTC |
0e8e237 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:51:04 UTC | sql/data: Add new task type for loading mercurial dump | 09 March 2018, 15:54:32 UTC |
5a70627 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:50:41 UTC | swh.scheduler.cli: Add sample use case for the scheduling cli | 09 March 2018, 15:54:32 UTC |
3d68f20 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:50:11 UTC | swh.scheduler.cli: Open policy column to the scheduling cli | 09 March 2018, 15:54:31 UTC |
185f4b3 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:44:19 UTC | swh.scheduler.cli: Open the delimiter option as cli argument When providing json values (args|kwargs column), csv's default comma separator and json's default separator values clash. So the cli fails to schedule task. This commit permits to overcome that issue by providing the delimiter ';' to the cli call when json is required. | 09 March 2018, 15:52:49 UTC |
4b918af | Antoine R. Dumont (@ardumont) | 08 March 2018, 10:27:47 UTC | Fix issue when updating task-type without any retry delay defined Related P231 Related T902 | 08 March 2018, 10:27:47 UTC |
f8ddae1 | Antoine R. Dumont (@ardumont) | 08 March 2018, 10:27:26 UTC | Fix pep8 violation regarding bare except | 08 March 2018, 10:27:26 UTC |
46329da | Antoine R. Dumont (@ardumont) | 19 December 2017, 13:35:40 UTC | swh-scheduler/data: Add new oneshot scheduling load-mercurial task Related T908 | 19 December 2017, 13:37:59 UTC |
b74c89f | Antoine Pietri | 14 December 2017, 17:26:59 UTC | backend: fix default scheduling_db value for consistency | 14 December 2017, 17:26:59 UTC |
5b17566 | Antoine Pietri | 14 December 2017, 16:53:30 UTC | backend: doc: fix return value of create_tasks | 14 December 2017, 16:53:30 UTC |
f08d9e7 | Antoine R. Dumont (@ardumont) | 07 December 2017, 14:18:31 UTC | Reference new task_type in scheduler | 07 December 2017, 14:18:31 UTC |
255d851 | Antoine R. Dumont (@ardumont) | 29 November 2017, 11:09:32 UTC | swh.scheduler.utils: Open utility function to create oneshot task | 29 November 2017, 11:30:35 UTC |
723221e | Nicolas Dandrimont | 08 November 2017, 15:38:52 UTC | Update for celery 4 | 08 November 2017, 16:00:50 UTC |
09b82c9 | Stefano Zacchiroli | 02 November 2017, 10:09:25 UTC | docs: add absolute anchor to documentation index | 02 November 2017, 10:09:25 UTC |
cd827b6 | Nicolas Dandrimont | 12 October 2017, 16:47:31 UTC | Add missing __init__.py files | 12 October 2017, 16:48:15 UTC |
0a3d8ec | Nicolas Dandrimont | 12 October 2017, 15:16:58 UTC | Cleanup packaging | 12 October 2017, 15:17:55 UTC |
79b0ccd | Antoine R. Dumont (@ardumont) | 11 October 2017, 08:43:27 UTC | swh.scheduler.cli: Fix missing task type option | 11 October 2017, 08:57:49 UTC |
d5f2d51 | Nicolas Dandrimont | 11 September 2017, 12:10:41 UTC | sql/updates/06: add name to check constraint | 11 September 2017, 12:10:41 UTC |
73f4aee | Nicolas Dandrimont | 11 September 2017, 12:01:31 UTC | sql: one-shot tasks don't really have an interval | 11 September 2017, 12:06:56 UTC |
b4bbc05 | Stefano Zacchiroli | 06 September 2017, 18:45:30 UTC | sanitize docstrings for sphinx | 06 September 2017, 18:45:30 UTC |
ca0230d | Nicolas Dandrimont | 01 September 2017, 18:20:42 UTC | sql: clean up schema syntax issues | 01 September 2017, 18:21:29 UTC |
f785a81 | Stefano Zacchiroli | 30 August 2017, 10:26:02 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:02 UTC |
89625c1 | Nicolas Dandrimont | 20 August 2017, 02:58:55 UTC | updates/05: fix SQL syntax to let the update run | 20 August 2017, 02:58:55 UTC |
ee545d4 | Nicolas Dandrimont | 07 August 2017, 16:43:57 UTC | test_scheduler: add missing db attribute to database tests | 07 August 2017, 16:43:57 UTC |
deb40f2 | Nicolas Dandrimont | 07 August 2017, 16:33:40 UTC | requirements: bump swh.core to 0.0.34 | 07 August 2017, 16:33:54 UTC |
8348905 | Nicolas Dandrimont | 07 August 2017, 16:07:30 UTC | sql: add update script from 04 to 05 | 07 August 2017, 16:07:30 UTC |
3194c2b | Nicolas Dandrimont | 25 July 2017, 16:49:17 UTC | sql: new logic for recurrence of one-shot tasks | 25 July 2017, 16:49:17 UTC |
030a51f | Nicolas Dandrimont | 25 July 2017, 15:42:31 UTC | sql: add the recurrence policy to tasks and related fields to task_type This is the groundwork for the implementation of T720: - it adds the minimal set of fields required to allow for a task to be retried on transient failures; - it adds knowledge of whether a task is one-shot or recurring | 25 July 2017, 15:44:40 UTC |
adb5afe | Nicolas Dandrimont | 25 July 2017, 13:30:48 UTC | test_scheduler: more generic tests wrt task types | 25 July 2017, 13:30:48 UTC |
31a6697 | Nicolas Dandrimont | 25 July 2017, 12:39:26 UTC | celery_backend.runner: respect the task's maximum queue length Close T726 | 25 July 2017, 12:39:26 UTC |
a0420b0 | Nicolas Dandrimont | 21 July 2017, 16:57:18 UTC | backend: Add max queue length to task types One more stepping stone towards T726 | 21 July 2017, 16:57:18 UTC |
4fe37f7 | Nicolas Dandrimont | 21 July 2017, 14:09:55 UTC | backend: add get_tasks method | 21 July 2017, 14:09:55 UTC |
5ccc579 | Nicolas Dandrimont | 21 July 2017, 13:58:35 UTC | backend: add task_type argument to peek/grab_ready_tasks | 21 July 2017, 13:58:35 UTC |
af220e2 | Nicolas Dandrimont | 19 July 2017, 18:02:11 UTC | backend: add tests for grab_ready_tasks | 19 July 2017, 18:02:11 UTC |
d8945ae | Nicolas Dandrimont | 19 July 2017, 14:39:42 UTC | backend: add peek_ready_tasks tests | 19 July 2017, 14:39:42 UTC |
d74d79f | Nicolas Dandrimont | 19 July 2017, 12:04:54 UTC | tests: add create_tasks test | 19 July 2017, 12:10:21 UTC |
0b88b14 | Nicolas Dandrimont | 19 July 2017, 10:08:30 UTC | backend: add get_task_types | 19 July 2017, 10:08:30 UTC |
2e2205c | Nicolas Dandrimont | 19 July 2017, 10:06:03 UTC | tests: refactor and extend | 19 July 2017, 10:07:01 UTC |
d4eb07e | Nicolas Dandrimont | 17 July 2017, 17:28:43 UTC | Add test infrastructure for the scheduler backend | 17 July 2017, 17:31:19 UTC |
285e7b5 | Nicolas Dandrimont | 17 July 2017, 17:04:32 UTC | Add sql Makefile | 17 July 2017, 17:04:32 UTC |
0e14eff | Nicolas Dandrimont | 19 June 2017, 17:00:18 UTC | celery_backend.config: add a shortcut to get a queue length | 19 June 2017, 17:00:18 UTC |
7caf543 | Nicolas Dandrimont | 19 June 2017, 14:03:19 UTC | celery_backend.config: add a method to get queue statistics from RabbitMQ First part of implementing T726 | 19 June 2017, 14:03:19 UTC |
54bcdda | Nicolas Dandrimont | 19 June 2017, 09:51:39 UTC | swh-worker-control: remove spurious print | 19 June 2017, 09:51:39 UTC |
80851f5 | Antoine Pietri | 15 June 2017, 15:13:22 UTC | celery: enable console logging on debug loglevel | 15 June 2017, 15:13:22 UTC |
7c1c041 | Nicolas Dandrimont | 12 June 2017, 15:49:58 UTC | listener: update for the new task-result event | 12 June 2017, 15:49:58 UTC |
d3abfa4 | Nicolas Dandrimont | 12 June 2017, 15:49:12 UTC | listener: derive the reliable events receiver from the base celery class | 12 June 2017, 15:49:12 UTC |
671a2c0 | Nicolas Dandrimont | 12 June 2017, 13:30:02 UTC | task: move run to run_task to allow overriding the default behavior of tasks This is an API break, but it's just a matter of renaming all existing tasks to run_task instead of run. Close T536 | 12 June 2017, 13:30:02 UTC |
07ea766 | Nicolas Dandrimont | 31 May 2017, 16:27:53 UTC | swh-worker-control: read the remote monotonic time to get task start times | 31 May 2017, 16:27:53 UTC |
c02757c | Nicolas Dandrimont | 31 May 2017, 16:27:00 UTC | celery_backend: add a remote-control command to get the monotonic clock | 31 May 2017, 16:27:00 UTC |
324a3cc | Nicolas Dandrimont | 31 May 2017, 15:49:06 UTC | bin/swh-worker-control: add command-line interface for worker remote control | 31 May 2017, 15:51:04 UTC |
f758291 | Nicolas Dandrimont | 07 April 2017, 09:57:36 UTC | requirements-swh.txt: use proper syntax | 07 April 2017, 09:57:36 UTC |
a760866 | Nicolas Dandrimont | 07 April 2017, 09:53:08 UTC | d/copyright: cleanup | 07 April 2017, 09:53:08 UTC |
35997f8 | Nicolas Dandrimont | 07 April 2017, 09:52:27 UTC | celery_backend.config: use the JournalHandler for logging | 07 April 2017, 09:52:27 UTC |
aaa5dfd | Nicolas Dandrimont | 09 March 2017, 10:12:09 UTC | celery_backend.config: only log to database if log_db is present | 09 March 2017, 10:12:09 UTC |
0767b4c | Antoine Pietri | 15 February 2017, 14:23:52 UTC | requirements.txt: ensure we use celery<4 (we don't support other versions for now) | 15 February 2017, 14:23:52 UTC |
6192ac1 | Antoine R. Dumont (@ardumont) | 10 February 2017, 14:03:02 UTC | Add utils.get_task fn to simplify contextual task instance retrieval Permits to unify the task instance retrieval, which then permits to send messages to the associated task's queue. | 10 February 2017, 14:11:54 UTC |
1aedc67 | Antoine Pietri | 09 February 2017, 11:21:06 UTC | requirements: split internal and external requirements in two separate files | 09 February 2017, 13:32:06 UTC |
d65742e | Antoine Pietri | 09 February 2017, 11:19:06 UTC | celery_backend: pep8: fix missing blank lines | 09 February 2017, 11:21:23 UTC |
05f41c2 | Nicolas Dandrimont | 21 October 2016, 14:09:11 UTC | sql: fix default_interval selection for new tasks | 21 October 2016, 14:09:11 UTC |
b048926 | Nicolas Dandrimont | 20 October 2016, 14:02:57 UTC | backend: the status of a task is task.status, not task.task_status... | 20 October 2016, 14:02:57 UTC |
da7b1b7 | Nicolas Dandrimont | 19 October 2016, 13:39:05 UTC | backend: allow disabling tasks | 19 October 2016, 13:39:05 UTC |
0fb8eef | Antoine R. Dumont (@ardumont) | 23 August 2016, 16:44:15 UTC | Fix docstring typo | 05 September 2016, 12:27:28 UTC |
51cb666 | Nicolas Dandrimont | 01 September 2016, 15:23:15 UTC | celery_backend.config: add SWH_WORKER_INSTANCE environment variable This allows to run several workers with different configuration files. | 01 September 2016, 15:23:15 UTC |
dced5d5 | Nicolas Dandrimont | 24 February 2016, 15:36:52 UTC | celery_backend.config: Put pickle last in the accepted encodings | 24 February 2016, 15:36:52 UTC |
7d6e331 | Nicolas Dandrimont | 24 February 2016, 15:36:02 UTC | backend: allow for mass-scheduling of task runs This allows the runner to be (~10 times) more efficient when running tasks | 24 February 2016, 15:36:02 UTC |
15c3d8f | Nicolas Dandrimont | 24 February 2016, 11:10:51 UTC | backend: use copy for mass-scheduling of tasks | 24 February 2016, 11:10:51 UTC |
4886edf | Nicolas Dandrimont | 23 February 2016, 16:08:46 UTC | Add cli to setup.py | 23 February 2016, 16:08:46 UTC |
f390dec | Nicolas Dandrimont | 23 February 2016, 16:05:27 UTC | cli: New command-line interface based on Click | 23 February 2016, 16:06:13 UTC |
77bcb2a | Nicolas Dandrimont | 23 February 2016, 16:05:44 UTC | backend: allow overriding the config at instantiation | 23 February 2016, 16:06:13 UTC |
35edc1a | Nicolas Dandrimont | 23 February 2016, 16:04:33 UTC | backend: use arrow instead of dateutil | 23 February 2016, 16:04:33 UTC |
e9c4442 | Nicolas Dandrimont | 23 February 2016, 11:07:03 UTC | d/copyright: update for Celery license | 23 February 2016, 11:07:36 UTC |
d89860f | Nicolas Dandrimont | 23 February 2016, 10:58:01 UTC | Add missing Celery license file | 23 February 2016, 10:58:01 UTC |
314e4dd | Nicolas Dandrimont | 23 February 2016, 10:56:41 UTC | celery_backend.listener: Remove useless arguments to event functions | 23 February 2016, 10:56:41 UTC |
606fd39 | Nicolas Dandrimont | 23 February 2016, 10:56:06 UTC | celery_backend.listener: force sending the events at startup | 23 February 2016, 10:56:06 UTC |
e88c731 | Nicolas Dandrimont | 22 February 2016, 14:07:52 UTC | celery_backend.listener: make the event listener reliable We reimplement the event listener using kombu directly instead of Celery, which allows us to create persistent queues and manually ack the messages once we have received them. We bundle messages by packets of 1000 or 1/second when sending them to the database. We only ack the messages once the database transcation has ended. | 22 February 2016, 14:07:52 UTC |
c4e1b63 | Nicolas Dandrimont | 22 February 2016, 14:06:55 UTC | backend: implement the timestamp argument for the task_run functions | 22 February 2016, 14:07:22 UTC |
911df24 | Nicolas Dandrimont | 22 February 2016, 14:06:03 UTC | celery_backend.runner: only commit once per packet | 22 February 2016, 14:06:03 UTC |
f3c8cad | Nicolas Dandrimont | 19 February 2016, 17:49:45 UTC | celery: move to celery_backend to avoid clashing with system celery | 19 February 2016, 17:50:40 UTC |
0df7211 | Nicolas Dandrimont | 19 February 2016, 17:46:40 UTC | d/control: add missing B-D on Celery | 19 February 2016, 17:46:40 UTC |
d389ce2 | Nicolas Dandrimont | 19 February 2016, 17:35:27 UTC | setup.py: install all packages | 19 February 2016, 17:43:53 UTC |
06d7095 | Nicolas Dandrimont | 19 February 2016, 17:34:53 UTC | tests: Add basic tests for task | 19 February 2016, 17:34:53 UTC |
482d414 | Nicolas Dandrimont | 19 February 2016, 17:27:03 UTC | schema: replace the eventful boolean with a task run status | 19 February 2016, 17:27:03 UTC |
655a9d0 | Nicolas Dandrimont | 19 February 2016, 11:47:49 UTC | scheduler.celery: add listener for events and runner to generate tasks | 19 February 2016, 11:47:49 UTC |
8d512e3 | Nicolas Dandrimont | 18 February 2016, 16:47:49 UTC | backend: wrap all the DB methods | 18 February 2016, 16:47:49 UTC |
c70627e | Nicolas Dandrimont | 16 February 2016, 18:24:55 UTC | backend: initialize a simple backend for swh.scheduler | 16 February 2016, 18:24:55 UTC |
196e506 | Nicolas Dandrimont | 16 February 2016, 18:24:25 UTC | celery: move worker to a "celery.config" class, add listener | 16 February 2016, 18:24:25 UTC |
efc7ff8 | Nicolas Dandrimont | 16 February 2016, 10:27:40 UTC | sql: update testdata and functions for new schema | 16 February 2016, 10:27:40 UTC |
86f5e28 | Nicolas Dandrimont | 16 February 2016, 09:51:17 UTC | sql/schema: Clean up formatting | 16 February 2016, 09:51:17 UTC |