| 52614b9 | Antoine R. Dumont (@ardumont) | 17 May 2018, 11:32:06 UTC | swh.scheduler.updater.ghtorrent: Move implem inside its own module | 17 May 2018, 11:35:08 UTC |
| e2be888 | Antoine R. Dumont (@ardumont) | 17 May 2018, 10:37:46 UTC | swh.scheduler.updater.ghtorrent: Simplify consumer interface | 17 May 2018, 11:26:17 UTC |
| 0c1b67e | Antoine R. Dumont (@ardumont) | 17 May 2018, 10:24:48 UTC | swh.scheduler.updater.ghtorrent: Improve reading events | 17 May 2018, 10:29:50 UTC |
| 441aa92 | Antoine R. Dumont (@ardumont) | 16 May 2018, 16:37:56 UTC | swh.scheduler.updater.consumer: Flush memory cache | 16 May 2018, 16:39:39 UTC |
| d4dbf0a | Antoine R. Dumont (@ardumont) | 16 May 2018, 16:37:30 UTC | swh.scheduler.updater.consumer: Check for data to send in final step | 16 May 2018, 16:37:30 UTC |
| 333afe9 | Antoine R. Dumont (@ardumont) | 16 May 2018, 16:26:08 UTC | swh.scheduler.updater: Design the UpdaterConsumer interface | 16 May 2018, 16:26:08 UTC |
| f16fa5f | Antoine R. Dumont (@ardumont) | 16 May 2018, 15:38:21 UTC | swh.scheduler.updater: Don't subscribe to create event Empty repository are not that interesting, a push event on it will be much better later. | 16 May 2018, 15:38:21 UTC |
| 5bda011 | Antoine R. Dumont (@ardumont) | 16 May 2018, 13:21:24 UTC | swh.scheduler.updater: Actually use the cache_put method with list | 16 May 2018, 13:21:24 UTC |
| 3a39baa | Antoine R. Dumont (@ardumont) | 16 May 2018, 12:19:50 UTC | updater/ghtorrent: Open ghtorrent consumer as cli script | 16 May 2018, 12:27:34 UTC |
| 8d5bb5b | Antoine R. Dumont (@ardumont) | 16 May 2018, 11:44:12 UTC | updater: Actually consume ghtorrent event Also make GHTorrent and FakeGHTorrent publisher converge. Related T1051 | 16 May 2018, 11:44:12 UTC |
| b9b3c04 | Antoine R. Dumont (@ardumont) | 16 May 2018, 11:43:18 UTC | tests/updater: Simplify events tests | 16 May 2018, 11:43:18 UTC |
| 0f28201 | Antoine R. Dumont (@ardumont) | 16 May 2018, 08:18:04 UTC | swh.scheduler.updater: Move updater tests to its own arborescence | 16 May 2018, 11:42:51 UTC |
| 05cc4c3 | Antoine R. Dumont (@ardumont) | 14 May 2018, 13:17:43 UTC | updater.ghtorrent: Write events to scheduler updater backend | 14 May 2018, 13:19:18 UTC |
| 1f6b4d8 | Antoine R. Dumont (@ardumont) | 14 May 2018, 13:11:16 UTC | events: Rename event key to type | 14 May 2018, 13:11:16 UTC |
| fd37df2 | Antoine R. Dumont (@ardumont) | 14 May 2018, 13:09:56 UTC | updater.scratch: Update tryout code | 14 May 2018, 13:09:56 UTC |
| 7e88ec2 | Antoine R. Dumont (@ardumont) | 14 May 2018, 09:05:33 UTC | scheduler.api.server: Instantiate scheduler backend once per import Related 18c9dad986a1f6f19d57dd97079dc22ad10b04df | 14 May 2018, 09:05:33 UTC |
| 97f03a8 | Antoine R. Dumont (@ardumont) | 14 May 2018, 07:00:30 UTC | Fix pep8 violation, remove unused import, fix typo | 14 May 2018, 07:00:30 UTC |
| 9dd3cd3 | Antoine R. Dumont (@ardumont) | 09 May 2018, 16:06:38 UTC | swh.scheduler.updater.backend: Bootstrap backend api Related T1051 | 09 May 2018, 16:06:38 UTC |
| 464d759 | Antoine R. Dumont (@ardumont) | 09 May 2018, 09:11:40 UTC | Reference tryout code work to assert we can work with ghtorrent We cannot so far Related T1051 | 09 May 2018, 09:13:17 UTC |
| ccdc134 | Antoine R. Dumont (@ardumont) | 09 May 2018, 08:31:24 UTC | swh.scheduler.updater: Add SWHEvent class and tests around it Related T1051 | 09 May 2018, 08:33:47 UTC |
| 5de5eb9 | Antoine R. Dumont (@ardumont) | 09 May 2018, 08:29:28 UTC | scheduler.updater: Add publish/subscribe fake ghtorrent class As there remain issue with ghtorrent's infra [1], i'm using fake random generator event for now. [1] https://github.com/ghtorrent/ghtorrent.org/issues/397#issuecomment-387052462 Related T1051 | 09 May 2018, 08:33:47 UTC |
| 65e4b21 | Antoine R. Dumont (@ardumont) | 02 May 2018, 15:26:34 UTC | swh.scheduler.backend: Permit to create tasks with priority Related T1035 | 03 May 2018, 10:15:03 UTC |
| 5e92ae1 | Antoine R. Dumont (@ardumont) | 27 April 2018, 17:29:34 UTC | swh.scheduler: Schedule tasks with/without priority Related T1035 | 03 May 2018, 10:15:03 UTC |
| 9bf1a79 | Nicolas Dandrimont | 02 May 2018, 11:15:53 UTC | Don't override scheduler configuration by default | 02 May 2018, 11:15:53 UTC |
| bd3da9b | Antoine R. Dumont (@ardumont) | 26 April 2018, 15:34:07 UTC | New upstream version 0.0.26 | 26 April 2018, 15:34:07 UTC |
| ffd2dda | Antoine R. Dumont (@ardumont) | 26 April 2018, 15:33:52 UTC | d/rules: Fix package build | 26 April 2018, 15:33:52 UTC |
| 950a7d5 | Antoine R. Dumont (@ardumont) | 25 April 2018, 14:10:26 UTC | swh.scheduler.tests: Test remote scheduler api as well Related T1036 | 25 April 2018, 16:37:43 UTC |
| 6ef0a88 | Antoine R. Dumont (@ardumont) | 25 April 2018, 11:03:44 UTC | swh.scheduler: Add tests around removing archivable tasks Related T986 Related T1034 | 25 April 2018, 16:36:34 UTC |
| 7afd050 | Antoine R. Dumont (@ardumont) | 25 April 2018, 09:57:17 UTC | swh.scheduler: Add tests around filtering archivable tasks Related T986 Related T1034 | 25 April 2018, 16:36:34 UTC |
| 5ccfa8b | Antoine R. Dumont (@ardumont) | 25 April 2018, 09:02:02 UTC | swh-scheduler-schema: Fix unneeded drop instructions Should not have been committed in that file, only in the migration part. | 25 April 2018, 16:36:34 UTC |
| b7490ee | Antoine R. Dumont (@ardumont) | 24 April 2018, 14:48:05 UTC | swh.scheduler.cli: Improve docstring | 25 April 2018, 16:36:34 UTC |
| 8c3910a | Antoine R. Dumont (@ardumont) | 24 April 2018, 14:26:13 UTC | swh.scheduler.cli: Permit to specify the backend to use in cli Related T1034 | 25 April 2018, 16:36:34 UTC |
| 9e13fd2 | Antoine R. Dumont (@ardumont) | 24 April 2018, 13:40:33 UTC | swh.scheduler.api: Bootstrap scheduler's remote api Related T1034 | 25 April 2018, 16:36:34 UTC |
| 5349689 | Antoine R. Dumont (@ardumont) | 24 April 2018, 12:08:28 UTC | swh.scheduler: Use `get_scheduler` api to instantiate a scheduler Related T1034 | 24 April 2018, 14:55:43 UTC |
| ef4eb14 | Antoine R. Dumont (@ardumont) | 24 April 2018, 12:07:48 UTC | swh.scheduler.backend: Fix docstring | 24 April 2018, 12:07:48 UTC |
| 7d95da2 | Antoine R. Dumont (@ardumont) | 18 April 2018, 10:34:43 UTC | New upstream version 0.0.25 | 18 April 2018, 10:34:43 UTC |
| 8124229 | Antoine R. Dumont (@ardumont) | 18 April 2018, 09:33:45 UTC | swh.scheduler.cli.archive: Index arguments.kwargs as text Related T1023 | 18 April 2018, 09:33:45 UTC |
| b67f570 | Antoine R. Dumont (@ardumont) | 13 April 2018, 12:55:32 UTC | New upstream version 0.0.24 | 13 April 2018, 12:55:32 UTC |
| f4587a3 | Antoine R. Dumont (@ardumont) | 13 April 2018, 09:45:32 UTC | data/template: Do not index the arguments field (it's in _source) As this field is randomly large depending on the task at end, this triggers a limit (index.mapping.total_fields.limit to 1000). We do not really need this in the index as the data will still be in the _source. [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings Related T1023 | 13 April 2018, 09:59:37 UTC |
| 915db34 | Antoine R. Dumont (@ardumont) | 13 April 2018, 09:10:30 UTC | data/README: Add a small readme to explain es install step | 13 April 2018, 09:59:36 UTC |
| 547eb89 | Antoine R. Dumont (@ardumont) | 13 April 2018, 09:07:51 UTC | swh.scheduler.cli: Add a bulk index flag to separate read from index Try to reduce the number of connection timeout error. There are 3 solutions: - either increase the default timeout of 10 per request - same on server configuration basis (no) - either reduce the number of data to bulk index, this is the chosen solution Related T1023 | 13 April 2018, 09:59:27 UTC |
| 532a340 | Antoine R. Dumont (@ardumont) | 10 April 2018, 15:43:07 UTC | New upstream version 0.0.23 | 10 April 2018, 15:43:07 UTC |
| e972b6a | Antoine R. Dumont (@ardumont) | 10 April 2018, 14:21:57 UTC | swh.scheduler.cli.archive: Simplify task and task_run ids extraction Related T986 | 10 April 2018, 15:43:03 UTC |
| 6c11eb6 | Antoine R. Dumont (@ardumont) | 10 April 2018, 14:06:38 UTC | swh.sched.cli.archive: Improve logging Related T986 | 10 April 2018, 15:43:03 UTC |
| 1da2d71 | Antoine R. Dumont (@ardumont) | 10 April 2018, 13:53:05 UTC | swh.scheduler.cli.archive: Delete only completely indexed tasks Prior to this commit, it could happen that we removed tasks even though we did not yet index associated task_run. Related T986 | 10 April 2018, 15:42:57 UTC |
| 04ccc2d | Antoine R. Dumont (@ardumont) | 10 April 2018, 12:44:15 UTC | swh.sched.cli.archive: Use interval period to filter archival tasks Related T986 | 10 April 2018, 14:00:12 UTC |
| 962fd8b | Antoine R. Dumont (@ardumont) | 10 April 2018, 12:42:35 UTC | swh.scheduler.backend_es: Return operation failure instead of raising Prior to this commit, an error would raise and stop all indexation. As the code is already waiting for a tuple (operation-status, item), we instead leverage this and continue working on indexation. All items whose `operation-status` is False (meaning failure to index, whatever the reason) is not indexed. Another run would then pickup the leftover and index it. Related T986 | 10 April 2018, 13:55:38 UTC |
| a04fb85 | Antoine R. Dumont (@ardumont) | 09 April 2018, 14:09:16 UTC | New upstream version 0.0.22 | 09 April 2018, 14:09:16 UTC |
| 9b9b88c | Antoine R. Dumont (@ardumont) | 09 April 2018, 13:59:16 UTC | d/control: Update to recent python3-elasticsearch client | 09 April 2018, 14:09:12 UTC |
| 9e5bf35 | Antoine R. Dumont (@ardumont) | 30 March 2018, 13:02:55 UTC | New upstream version 0.0.21 | 30 March 2018, 13:02:55 UTC |
| ffd00cb | Antoine R. Dumont (@ardumont) | 30 March 2018, 12:54:26 UTC | swh.scheduler.backend_es: Fix config base filename variable name | 30 March 2018, 12:55:25 UTC |
| ad0f0e2 | Antoine R. Dumont (@ardumont) | 30 March 2018, 10:22:28 UTC | data/elastic-template.json: Use elasticsearch's default conf Defaults to 5 shards and 1 replica | 30 March 2018, 12:54:54 UTC |
| 6cc6cb7 | Antoine R. Dumont (@ardumont) | 30 March 2018, 09:44:18 UTC | New upstream version 0.0.20 | 30 March 2018, 09:44:18 UTC |
| d6b393d | Antoine R. Dumont (@ardumont) | 29 March 2018, 15:11:37 UTC | swh.scheduler.cli.archive: By default, archive last month's data Related T986 | 29 March 2018, 15:32:25 UTC |
| e785252 | Antoine R. Dumont (@ardumont) | 29 March 2018, 08:47:08 UTC | swh.scheduler.cli: Use bulk api to index tasks Unfortunately, the bulk api does not return the full indexed data's original source [1] (which we need to identify the data clean up in the db). So we leverage elasticsearch's multi-get api to read back the original source. Related T986 | 29 March 2018, 10:32:37 UTC |
| 4d13f5d | Antoine R. Dumont (@ardumont) | 26 March 2018, 15:51:33 UTC | swh.scheduler.cli.archive: Improve dry-run behavior Related T986 | 26 March 2018, 15:56:00 UTC |
| af4c9b8 | Antoine R. Dumont (@ardumont) | 26 March 2018, 15:32:57 UTC | swh.scheduler.cli.archive: Optimize reading task lookup query Related T986 | 26 March 2018, 15:32:57 UTC |
| d83da86 | Antoine R. Dumont (@ardumont) | 22 March 2018, 16:16:12 UTC | swh.scheduler.cli.archive: Remove unnecessary order by condition Related T986 | 22 March 2018, 16:39:13 UTC |
| bcb4f74 | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:57:35 UTC | swh.scheduler.cli.archive: Open batch size option for index/cleanup Related T986 | 22 March 2018, 16:39:13 UTC |
| 9e9f7ef | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:51:03 UTC | swh.scheduler.cli.archive: Stream indexed tasks removal Related T986 | 22 March 2018, 15:51:03 UTC |
| e3867ed | Antoine R. Dumont (@ardumont) | 22 March 2018, 15:50:07 UTC | swh.scheduler.cli.archive: Fix edge case when reading tasks Related T986 | 22 March 2018, 15:50:07 UTC |
| b76384b | Antoine R. Dumont (@ardumont) | 22 March 2018, 12:51:32 UTC | swh.scheduler.cli: Open endpoint to archive tasks in elasticsearch Related T986 | 22 March 2018, 12:51:48 UTC |
| 8435efd | Antoine R. Dumont (@ardumont) | 21 March 2018, 18:58:45 UTC | swh.scheduler.cli: Add sql function to list archivable tasks Related T986 Related P240 | 21 March 2018, 18:58:45 UTC |
| 0324fb2 | Antoine R. Dumont (@ardumont) | 15 March 2018, 10:21:06 UTC | sql/data: Fix missing num_retries in deposit task types | 15 March 2018, 10:21:06 UTC |
| 367ab15 | Antoine R. Dumont (@ardumont) | 14 March 2018, 12:31:59 UTC | swh.core.serializer: Move to msgpack serialization format This permits to serialize bytes which was an existing behavior in celery 3.1 (when pickle was the default format). Related T782 Related D300 | 14 March 2018, 12:31:59 UTC |
| a474ed5 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:56:03 UTC | swh.scheduler.cli: Unify pretty print output | 09 March 2018, 15:56:03 UTC |
| 0e8e237 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:51:04 UTC | sql/data: Add new task type for loading mercurial dump | 09 March 2018, 15:54:32 UTC |
| 5a70627 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:50:41 UTC | swh.scheduler.cli: Add sample use case for the scheduling cli | 09 March 2018, 15:54:32 UTC |
| 3d68f20 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:50:11 UTC | swh.scheduler.cli: Open policy column to the scheduling cli | 09 March 2018, 15:54:31 UTC |
| 185f4b3 | Antoine R. Dumont (@ardumont) | 09 March 2018, 15:44:19 UTC | swh.scheduler.cli: Open the delimiter option as cli argument When providing json values (args|kwargs column), csv's default comma separator and json's default separator values clash. So the cli fails to schedule task. This commit permits to overcome that issue by providing the delimiter ';' to the cli call when json is required. | 09 March 2018, 15:52:49 UTC |
| 4b918af | Antoine R. Dumont (@ardumont) | 08 March 2018, 10:27:47 UTC | Fix issue when updating task-type without any retry delay defined Related P231 Related T902 | 08 March 2018, 10:27:47 UTC |
| f8ddae1 | Antoine R. Dumont (@ardumont) | 08 March 2018, 10:27:26 UTC | Fix pep8 violation regarding bare except | 08 March 2018, 10:27:26 UTC |
| 46329da | Antoine R. Dumont (@ardumont) | 19 December 2017, 13:35:40 UTC | swh-scheduler/data: Add new oneshot scheduling load-mercurial task Related T908 | 19 December 2017, 13:37:59 UTC |
| b74c89f | Antoine Pietri | 14 December 2017, 17:26:59 UTC | backend: fix default scheduling_db value for consistency | 14 December 2017, 17:26:59 UTC |
| 5b17566 | Antoine Pietri | 14 December 2017, 16:53:30 UTC | backend: doc: fix return value of create_tasks | 14 December 2017, 16:53:30 UTC |
| f08d9e7 | Antoine R. Dumont (@ardumont) | 07 December 2017, 14:18:31 UTC | Reference new task_type in scheduler | 07 December 2017, 14:18:31 UTC |
| 11ec4ba | Antoine R. Dumont (@ardumont) | 29 November 2017, 11:51:15 UTC | New upstream version 0.0.19 | 29 November 2017, 11:51:15 UTC |
| 255d851 | Antoine R. Dumont (@ardumont) | 29 November 2017, 11:09:32 UTC | swh.scheduler.utils: Open utility function to create oneshot task | 29 November 2017, 11:30:35 UTC |
| cb7d5fa | Nicolas Dandrimont | 08 November 2017, 16:06:22 UTC | New upstream version 0.0.18 | 08 November 2017, 16:06:22 UTC |
| 723221e | Nicolas Dandrimont | 08 November 2017, 15:38:52 UTC | Update for celery 4 | 08 November 2017, 16:00:50 UTC |
| 09b82c9 | Stefano Zacchiroli | 02 November 2017, 10:09:25 UTC | docs: add absolute anchor to documentation index | 02 November 2017, 10:09:25 UTC |
| 926cb2e | Nicolas Dandrimont | 12 October 2017, 16:49:02 UTC | New upstream version 0.0.17 | 12 October 2017, 16:49:02 UTC |
| cd827b6 | Nicolas Dandrimont | 12 October 2017, 16:47:31 UTC | Add missing __init__.py files | 12 October 2017, 16:48:15 UTC |
| 0a3d8ec | Nicolas Dandrimont | 12 October 2017, 15:16:58 UTC | Cleanup packaging | 12 October 2017, 15:17:55 UTC |
| 79b0ccd | Antoine R. Dumont (@ardumont) | 11 October 2017, 08:43:27 UTC | swh.scheduler.cli: Fix missing task type option | 11 October 2017, 08:57:49 UTC |
| d5f2d51 | Nicolas Dandrimont | 11 September 2017, 12:10:41 UTC | sql/updates/06: add name to check constraint | 11 September 2017, 12:10:41 UTC |
| 73f4aee | Nicolas Dandrimont | 11 September 2017, 12:01:31 UTC | sql: one-shot tasks don't really have an interval | 11 September 2017, 12:06:56 UTC |
| b4bbc05 | Stefano Zacchiroli | 06 September 2017, 18:45:30 UTC | sanitize docstrings for sphinx | 06 September 2017, 18:45:30 UTC |
| ca0230d | Nicolas Dandrimont | 01 September 2017, 18:20:42 UTC | sql: clean up schema syntax issues | 01 September 2017, 18:21:29 UTC |
| f785a81 | Stefano Zacchiroli | 30 August 2017, 10:26:02 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:02 UTC |
| 89625c1 | Nicolas Dandrimont | 20 August 2017, 02:58:55 UTC | updates/05: fix SQL syntax to let the update run | 20 August 2017, 02:58:55 UTC |
| 017c885 | Nicolas Dandrimont | 07 August 2017, 16:44:03 UTC | New upstream version 0.0.16 | 07 August 2017, 16:44:03 UTC |
| ee545d4 | Nicolas Dandrimont | 07 August 2017, 16:43:57 UTC | test_scheduler: add missing db attribute to database tests | 07 August 2017, 16:43:57 UTC |
| deb40f2 | Nicolas Dandrimont | 07 August 2017, 16:33:40 UTC | requirements: bump swh.core to 0.0.34 | 07 August 2017, 16:33:54 UTC |
| 8348905 | Nicolas Dandrimont | 07 August 2017, 16:07:30 UTC | sql: add update script from 04 to 05 | 07 August 2017, 16:07:30 UTC |
| 3194c2b | Nicolas Dandrimont | 25 July 2017, 16:49:17 UTC | sql: new logic for recurrence of one-shot tasks | 25 July 2017, 16:49:17 UTC |
| 030a51f | Nicolas Dandrimont | 25 July 2017, 15:42:31 UTC | sql: add the recurrence policy to tasks and related fields to task_type This is the groundwork for the implementation of T720: - it adds the minimal set of fields required to allow for a task to be retried on transient failures; - it adds knowledge of whether a task is one-shot or recurring | 25 July 2017, 15:44:40 UTC |
| adb5afe | Nicolas Dandrimont | 25 July 2017, 13:30:48 UTC | test_scheduler: more generic tests wrt task types | 25 July 2017, 13:30:48 UTC |
| 31a6697 | Nicolas Dandrimont | 25 July 2017, 12:39:26 UTC | celery_backend.runner: respect the task's maximum queue length Close T726 | 25 July 2017, 12:39:26 UTC |
| a0420b0 | Nicolas Dandrimont | 21 July 2017, 16:57:18 UTC | backend: Add max queue length to task types One more stepping stone towards T726 | 21 July 2017, 16:57:18 UTC |
