7245ee9 | Jenkins for Software Heritage | 24 January 2020, 14:01:58 UTC | Updated debian changelog for version 0.0.167 | 24 January 2020, 14:01:58 UTC |
e180f2a | Jenkins for Software Heritage | 24 January 2020, 14:01:57 UTC | Update upstream source from tag 'debian/upstream/0.0.167' Update to upstream version '0.0.167' with Debian dir e797a96b300338124b09f8b75bef2996fc8220d9 | 24 January 2020, 14:01:57 UTC |
e62d6e4 | Jenkins for Software Heritage | 24 January 2020, 14:01:56 UTC | New upstream version 0.0.167 | 24 January 2020, 14:01:56 UTC |
c8389c2 | Antoine R. Dumont (@ardumont) | 24 January 2020, 13:54:26 UTC | 146: Fix typo | 24 January 2020, 13:54:26 UTC |
2ebcdf3 | Antoine R. Dumont (@ardumont) | 24 January 2020, 10:57:46 UTC | pgstorage: Empty temp tables instead of dropping them Due to our pattern of adding objects [1], vacuum is triggered regularly on pg_catalog.*, having an heavy impact on performance. This commit tries to avoid the dropping the temporary tables part, emptying them instead (they still are dropped at the end of the session but less often). This should decrease the bloat on pg_catalog.* tables. [1] - create temporary table - insert data from temporary table to production table with filtering - drop temporary table | 24 January 2020, 11:14:00 UTC |
7db8312 | Jenkins for Software Heritage | 24 January 2020, 09:00:12 UTC | Updated debian changelog for version 0.0.166 | 24 January 2020, 09:00:12 UTC |
9da9a85 | Jenkins for Software Heritage | 24 January 2020, 09:00:11 UTC | Update upstream source from tag 'debian/upstream/0.0.166' Update to upstream version '0.0.166' with Debian dir ee457449f805ebd9a5e9e10c911c8939bc4d5b4c | 24 January 2020, 09:00:11 UTC |
74bb123 | Jenkins for Software Heritage | 24 January 2020, 09:00:10 UTC | New upstream version 0.0.166 | 24 January 2020, 09:00:10 UTC |
cc25810 | Daniele Serafini | 22 January 2020, 14:03:41 UTC | assert list doesn't have too many values | 22 January 2020, 14:24:08 UTC |
2ebce62 | Daniele Serafini | 22 January 2020, 13:28:38 UTC | test endpoint: content_missing (sha1_git), snapshot_missing | 22 January 2020, 14:24:08 UTC |
c40d327 | Daniele Serafini | 22 January 2020, 13:27:36 UTC | in memory changes | 22 January 2020, 14:24:08 UTC |
55ebd23 | Daniele Serafini | 22 January 2020, 11:29:07 UTC | storage: Add endpoint to get missing content (by sha1_git) and missing snapshot | 22 January 2020, 14:24:08 UTC |
cfee7b5 | Valentin Lorentz | 22 January 2020, 11:25:24 UTC | Remove redundant config checks in load_and_check_config. 1. There is no reason to force the server to serve only the 'local' backend anymore 2. Missing arguments will error when instantiating the backend class. | 22 January 2020, 11:25:24 UTC |
2454a78 | Antoine Lambert | 17 January 2020, 16:02:35 UTC | docs: Fix sphinx warnings Related to T2188 | 17 January 2020, 16:02:35 UTC |
2dc17cd | Valentin Lorentz | 17 January 2020, 13:49:26 UTC | Remove 'id' and 'object_id' from the output of object_find_by_sha1_git. 'id' is not used anymore, and 'object_id' never was. This commit slightly simplifies existing code, and will allow some deduplication in the upcoming Cassandra backend. | 17 January 2020, 14:10:56 UTC |
b5a5084 | Valentin Lorentz | 16 January 2020, 15:17:44 UTC | Make origin_visit_get_random return None instead of {} if there are no results. This is more consistent with other endpoints. | 17 January 2020, 14:10:36 UTC |
dba9e04 | Valentin Lorentz | 16 January 2020, 13:16:11 UTC | Rewrite test_content_get_partition_empty to not assume partitions are based on sha1. This is not true of the upcoming Cassandra backend. | 17 January 2020, 14:10:11 UTC |
e584655 | Valentin Lorentz | 16 January 2020, 13:13:43 UTC | Remove test_content_*_same_input, which check for behavior we do not want to guarantee. They check that content_add deduplicates with existing content/duplicated input. This is unneeded (the loaders don't send such data), so providing these guarantees unnecessarily complicates swh-storage code, especially the upcoming Cassandra backend. | 17 January 2020, 14:09:38 UTC |
92af2cd | Jenkins for Software Heritage | 17 January 2020, 13:09:40 UTC | Updated debian changelog for version 0.0.165 | 17 January 2020, 13:09:40 UTC |
ac79988 | Jenkins for Software Heritage | 17 January 2020, 13:09:39 UTC | Update upstream source from tag 'debian/upstream/0.0.165' Update to upstream version '0.0.165' with Debian dir c24845a1a2ad014144923b527940b11c269f8081 | 17 January 2020, 13:09:39 UTC |
079fa61 | Jenkins for Software Heritage | 17 January 2020, 13:09:38 UTC | New upstream version 0.0.165 | 17 January 2020, 13:09:38 UTC |
bf77f14 | Antoine R. Dumont (@ardumont) | 17 January 2020, 12:55:36 UTC | storage.retry: Fix objects loading when using generator parameters This will fix related retry error [1] [1] https://sentry.softwareheritage.org/share/issue/ddbbdd3c235b40ca826bf2c820989f14/ Related to cc29708564c35575f569e863f028a480a9905cf4 Related to D2543 | 17 January 2020, 12:55:36 UTC |
a7cbaf1 | Jenkins for Software Heritage | 16 January 2020, 17:05:03 UTC | Updated debian changelog for version 0.0.164 | 16 January 2020, 17:05:03 UTC |
4578987 | Jenkins for Software Heritage | 16 January 2020, 17:05:02 UTC | Update upstream source from tag 'debian/upstream/0.0.164' Update to upstream version '0.0.164' with Debian dir 23094cd1a4429ea115d7ffdaac67f3aea22ae353 | 16 January 2020, 17:05:02 UTC |
4738b2e | Jenkins for Software Heritage | 16 January 2020, 17:05:01 UTC | New upstream version 0.0.164 | 16 January 2020, 17:05:01 UTC |
cc29708 | Antoine Lambert | 16 January 2020, 16:16:09 UTC | storage: Fix objects loading when using generator parameters Some objects (directories, origins, releases, revisions) will not be added into the storage if they are provided as generator parameters to the *_add methods instead of lists. So ensure to transform generators into lists before processing the objects. | 16 January 2020, 16:53:22 UTC |
0b3610e | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:26:54 UTC | d/changelog: Bump new release | 14 January 2020, 16:27:58 UTC |
8c3ef4b | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:21:39 UTC | d/control: Add missing test dependency | 14 January 2020, 16:22:48 UTC |
6690419 | Jenkins for Software Heritage | 14 January 2020, 16:17:45 UTC | Updated debian changelog for version 0.0.163 | 14 January 2020, 16:17:45 UTC |
0481c44 | Jenkins for Software Heritage | 14 January 2020, 16:17:45 UTC | Update upstream source from tag 'debian/upstream/0.0.163' Update to upstream version '0.0.163' with Debian dir 4ed2720d31becfe82ffadf4a9175de855e6ee5da | 14 January 2020, 16:17:45 UTC |
e362b9d | Jenkins for Software Heritage | 14 January 2020, 16:17:43 UTC | New upstream version 0.0.163 | 14 January 2020, 16:17:43 UTC |
b641e53 | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:10:13 UTC | d/control: Align dependency versions with requirements.txt | 14 January 2020, 16:10:13 UTC |
e70f427 | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:09:54 UTC | d/control: Add new runtime dependency | 14 January 2020, 16:09:54 UTC |
8dcac2b | Antoine R. Dumont (@ardumont) | 14 January 2020, 12:40:56 UTC | retry: Implement content_add_metadata endpoint with retry policy | 14 January 2020, 12:45:11 UTC |
aa588c9 | Antoine R. Dumont (@ardumont) | 14 January 2020, 12:40:39 UTC | retry: Migrate to tenacity Which is a maintained fork of retry | 14 January 2020, 12:40:39 UTC |
4aa4d79 | Antoine R. Dumont (@ardumont) | 11 January 2020, 11:05:57 UTC | test_retry: Improve and align consistently assertion checks | 14 January 2020, 10:41:35 UTC |
2b7d770 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:56:04 UTC | storage.retry: Implement snapshot_add with retry policy | 14 January 2020, 10:41:35 UTC |
df3f33f | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:47:57 UTC | storage.retry: Implement release_add with retry policy | 14 January 2020, 10:41:35 UTC |
54890f7 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:44:20 UTC | storage.retry: Implement revision_add with retry policy | 14 January 2020, 10:41:35 UTC |
a8efa95 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:38:48 UTC | storage.retry: Implement directory_add with retry policy | 14 January 2020, 10:41:34 UTC |
dddb6d9 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:37:41 UTC | in_memory: Make directory_get_random return None when storage empty | 14 January 2020, 10:41:34 UTC |
2dd578c | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:25:31 UTC | storage.retry: Implement origin_visit_update with retry policy | 14 January 2020, 10:41:34 UTC |
32c460c | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:05:13 UTC | storage.retry: Implement origin_metadata_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
3cf7adb | Antoine R. Dumont (@ardumont) | 10 January 2020, 17:10:03 UTC | storage.retry: Implement metadata_provider_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
08f2f38 | Antoine R. Dumont (@ardumont) | 10 January 2020, 15:48:11 UTC | storage.retry: Implement tool_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
fe6440e | Antoine R. Dumont (@ardumont) | 10 January 2020, 15:37:24 UTC | storage.retry: Implement origin_visit_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
351b977 | Antoine R. Dumont (@ardumont) | 10 January 2020, 14:59:52 UTC | storage.retry: Implement origin_add_one endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
024eaea | Antoine R. Dumont (@ardumont) | 13 January 2020, 15:14:50 UTC | content_get_metadata: Change api to return Dict[bytes, List[Dict]] Clients will be able to introspect directly from the result whether a content is known or not. | 14 January 2020, 10:38:20 UTC |
07b6dc3 | Antoine R. Dumont (@ardumont) | 13 January 2020, 14:18:55 UTC | storage.content_get_metadata: Adapt to return nothing if inexistent id is passed as input | 13 January 2020, 14:18:55 UTC |
4837f46 | Antoine R. Dumont (@ardumont) | 09 January 2020, 16:17:30 UTC | storage: Add basic proxy storage with retry policy | 10 January 2020, 13:40:01 UTC |
fdf2a3c | Valentin Lorentz | 22 November 2019, 15:09:55 UTC | Add Storage.content_get_partition endpoint, to replace content_get_range. With no guarantees on the order or how partitioning is done, and with the new-style pagination. | 17 December 2019, 12:59:09 UTC |
0f94312 | Valentin Lorentz | 21 November 2019, 13:28:43 UTC | Add endpoint 'origin_list', that will replace 'origin_get_range'. And uses the new pagination scheme, instead of origin ids. | 16 December 2019, 14:17:21 UTC |
31a8dcf | Jenkins for Software Heritage | 16 December 2019, 13:41:40 UTC | Updated debian changelog for version 0.0.162 | 16 December 2019, 13:41:40 UTC |
931d588 | Jenkins for Software Heritage | 16 December 2019, 13:41:39 UTC | Update upstream source from tag 'debian/upstream/0.0.162' Update to upstream version '0.0.162' with Debian dir b1e0a120fcce8724776bef937a3e5437289ba707 | 16 December 2019, 13:41:39 UTC |
33925f7 | Jenkins for Software Heritage | 16 December 2019, 13:41:38 UTC | New upstream version 0.0.162 | 16 December 2019, 13:41:38 UTC |
295144f | Valentin Lorentz | 13 December 2019, 13:06:12 UTC | Add {content,directory,revision,release,snapshot}_get_random. Will be used to pick random objects to use them in Icinga checks. | 16 December 2019, 12:38:13 UTC |
fe6ac8f | Valentin Lorentz | 12 December 2019, 18:09:01 UTC | Move origin_visit_get_random to the right place in the code and fix its docstring. | 16 December 2019, 12:37:10 UTC |
31b2fc5 | Valentin Lorentz | 12 December 2019, 18:01:38 UTC | Deduplicate server code and move metric handling to storage.py | 16 December 2019, 12:33:31 UTC |
869100b | Valentin Lorentz | 12 December 2019, 17:16:24 UTC | Deduplicate client code. | 12 December 2019, 17:26:14 UTC |
21e144f | Jenkins for Software Heritage | 10 December 2019, 14:08:15 UTC | Updated debian changelog for version 0.0.161 | 10 December 2019, 14:08:15 UTC |
69c7acd | Jenkins for Software Heritage | 10 December 2019, 14:08:15 UTC | Update upstream source from tag 'debian/upstream/0.0.161' Update to upstream version '0.0.161' with Debian dir 778c3c5bb55b9131271bf8de1856a3f039eba268 | 10 December 2019, 14:08:15 UTC |
6e74f85 | Jenkins for Software Heritage | 10 December 2019, 14:08:13 UTC | New upstream version 0.0.161 | 10 December 2019, 14:08:13 UTC |
bd2a196 | Antoine R. Dumont (@ardumont) | 10 December 2019, 13:34:11 UTC | storage: Make origin_get_random simpler and faster | 10 December 2019, 13:34:11 UTC |
b440d3a | Antoine R. Dumont (@ardumont) | 09 December 2019, 12:43:53 UTC | storage: Prefer sample query on origin_visit The counter table is faster but may contain holes and be less up-to-date. | 09 December 2019, 12:51:08 UTC |
a2401b5 | Antoine R. Dumont (@ardumont) | 06 December 2019, 11:48:55 UTC | storage: Add endpoint to randomly pick an origin Related to T2120 | 09 December 2019, 12:46:13 UTC |
3855b6e | Antoine R. Dumont (@ardumont) | 09 December 2019, 12:40:57 UTC | tox.ini: Add a py3-dev environment | 09 December 2019, 12:46:13 UTC |
acea3fe | Jenkins for Software Heritage | 06 December 2019, 10:23:44 UTC | Updated debian changelog for version 0.0.160 | 06 December 2019, 10:23:44 UTC |
4069ebc | Jenkins for Software Heritage | 06 December 2019, 10:23:43 UTC | Update upstream source from tag 'debian/upstream/0.0.160' Update to upstream version '0.0.160' with Debian dir e9e59c3afba4f2a3101a98a32398ba5402f4d4fc | 06 December 2019, 10:23:43 UTC |
0599649 | Jenkins for Software Heritage | 06 December 2019, 10:23:42 UTC | New upstream version 0.0.160 | 06 December 2019, 10:23:42 UTC |
bee73be | Antoine R. Dumont (@ardumont) | 06 December 2019, 10:11:52 UTC | storage.buffer: Buffer release objects as well | 06 December 2019, 10:11:52 UTC |
382e500 | Antoine R. Dumont (@ardumont) | 06 December 2019, 10:11:25 UTC | storage.tests: Unify tests sample data | 06 December 2019, 10:11:25 UTC |
0e23408 | Antoine R. Dumont (@ardumont) | 26 November 2019, 16:09:38 UTC | d/changelog: Bump version | 26 November 2019, 16:09:38 UTC |
d233974 | Antoine Lambert | 26 November 2019, 14:30:05 UTC | debian/rules: Force fast hypothesis profile when running tests | 26 November 2019, 14:30:05 UTC |
27281e8 | Antoine Lambert | 26 November 2019, 14:21:02 UTC | Makefile.local: Fix test target execution Explicitly pass tests folder as paramter when invoking pytest in order for the hypothesis profiles to be found. | 26 November 2019, 14:22:36 UTC |
2cac339 | Nicolas Dandrimont | 22 November 2019, 17:27:52 UTC | Implement origin lookup by sha1 Close T2045. | 25 November 2019, 14:18:44 UTC |
0fcb8bc | Valentin Lorentz | 22 November 2019, 15:08:37 UTC | Get rid of warnings about the 'args' argument to get_storage. | 22 November 2019, 15:08:37 UTC |
c294b73 | Valentin Lorentz | 22 November 2019, 12:23:54 UTC | Remove/fix wrong comments. | 22 November 2019, 12:23:54 UTC |
68482bb | Jenkins for Software Heritage | 22 November 2019, 10:10:31 UTC | Updated debian changelog for version 0.0.159 | 22 November 2019, 10:10:31 UTC |
1c9119c | Jenkins for Software Heritage | 22 November 2019, 10:10:30 UTC | Update upstream source from tag 'debian/upstream/0.0.159' Update to upstream version '0.0.159' with Debian dir 248b0a0d38fcfc84e3f5b2af06e628a0e81ac769 | 22 November 2019, 10:10:30 UTC |
6c99bca | Jenkins for Software Heritage | 22 November 2019, 10:10:29 UTC | New upstream version 0.0.159 | 22 November 2019, 10:10:29 UTC |
a3fd826 | Nicolas Dandrimont | 21 November 2019, 13:10:29 UTC | Migrate tox.ini to extras = xxx instead of deps = .[testing] | 21 November 2019, 13:10:29 UTC |
1594e25 | Nicolas Dandrimont | 21 November 2019, 13:07:08 UTC | Drop unused listener extra | 21 November 2019, 13:07:08 UTC |
df4df8b | Nicolas Dandrimont | 21 November 2019, 13:06:55 UTC | Merge tox test environment configurations | 21 November 2019, 13:06:55 UTC |
06bd050 | Valentin Lorentz | 21 November 2019, 12:38:03 UTC | Deduplicate code of test_origin_get_range. | 21 November 2019, 12:38:03 UTC |
29eb548 | David Douard | 20 November 2019, 10:18:57 UTC | Fix a few typos reported by codespell | 21 November 2019, 12:16:51 UTC |
bc0e81c | David Douard | 20 November 2019, 10:17:40 UTC | pre-commit: explicitely whitelist 'iff' when running codespell | 21 November 2019, 12:16:51 UTC |
1472c8e | David Douard | 20 November 2019, 10:00:57 UTC | fix trailing ws reported by pre-commit | 21 November 2019, 12:16:51 UTC |
264cd33 | David Douard | 20 November 2019, 09:27:42 UTC | Add a pre-commit-hooks.yaml config file | 21 November 2019, 12:16:51 UTC |
a97db93 | David Douard | 20 November 2019, 10:25:51 UTC | Fix swh-storage-add-dir to please mypy, at least. | 21 November 2019, 10:19:43 UTC |
c787808 | David Douard | 20 November 2019, 10:19:26 UTC | Remove utils/(dump|fix)_revisions scripts these are now deprecated. | 21 November 2019, 10:19:43 UTC |
b337b4a | Valentin Lorentz | 14 November 2019, 13:16:49 UTC | Add 'pipeline' storage "class" for more readable configurations. This would allow writing configurations like: ``` storage: cls: pipeline steps: - cls: filter - cls: buffer - cls: remote url: http://swh-storage:5002/ ``` or ``` storage: cls: filter storage: cls: buffer storage: cls: remote url: http://swh-storage:5002/ ``` instead of: ``` storage: cls: filter args: storage: cls: buffer args: storage: cls: remote args: url: http://swh-storage:5002/ ``` | 19 November 2019, 03:16:39 UTC |
ea9aa47 | David Douard | 18 November 2019, 10:16:18 UTC | and not only for an existing origin visit. This is needed in situations where the snapshot table is not in sync with the origin_visit one; typically occurs on mirrors. Also add tests for with_visit argument. | 18 November 2019, 12:28:45 UTC |
0671131 | Jenkins for Software Heritage | 14 November 2019, 12:37:18 UTC | Updated debian changelog for version 0.0.158 | 14 November 2019, 12:37:18 UTC |
be50ce8 | Jenkins for Software Heritage | 14 November 2019, 12:37:18 UTC | Update upstream source from tag 'debian/upstream/0.0.158' Update to upstream version '0.0.158' with Debian dir 9562b6097a9ae7290bfe958ec1acbf6ba4342db4 | 14 November 2019, 12:37:18 UTC |
2a46802 | Jenkins for Software Heritage | 14 November 2019, 12:37:16 UTC | New upstream version 0.0.158 | 14 November 2019, 12:37:16 UTC |
9b4d339 | Antoine R. Dumont (@ardumont) | 14 November 2019, 09:54:02 UTC | d/*: Remove python3-swh.storage.schemata package The modules no longer exists Related D2272 | 14 November 2019, 09:54:02 UTC |
e296dfb | Antoine R. Dumont (@ardumont) | 14 November 2019, 09:26:41 UTC | swh.storage.schemata: Drop schemata from storage As this got migrated back to the swh.lister module | 14 November 2019, 09:26:41 UTC |
f2193c3 | Jenkins for Software Heritage | 13 November 2019, 12:27:07 UTC | Updated debian changelog for version 0.0.157 | 13 November 2019, 12:27:07 UTC |
07c7ce9 | Jenkins for Software Heritage | 13 November 2019, 12:27:06 UTC | Update upstream source from tag 'debian/upstream/0.0.157' Update to upstream version '0.0.157' with Debian dir ca1b4e56d7d3820b3368621c3d2d187e996b6892 | 13 November 2019, 12:27:06 UTC |
05d648e | Jenkins for Software Heritage | 13 November 2019, 12:27:05 UTC | New upstream version 0.0.157 | 13 November 2019, 12:27:05 UTC |