9f166f3 | Jenkins for Software Heritage | 19 February 2020, 15:00:32 UTC | Updated debian changelog for version 0.0.174 | 19 February 2020, 15:00:32 UTC |
4a3c8f3 | Jenkins for Software Heritage | 19 February 2020, 15:00:31 UTC | Update upstream source from tag 'debian/upstream/0.0.174' Update to upstream version '0.0.174' with Debian dir 8d31e83ca5c9c40500801331887fb0e85a0ad5e2 | 19 February 2020, 15:00:31 UTC |
a20779f | Jenkins for Software Heritage | 19 February 2020, 15:00:30 UTC | New upstream version 0.0.174 | 19 February 2020, 15:00:30 UTC |
7cf0864 | Valentin Lorentz | 14 February 2020, 16:26:09 UTC | Add support for (de)serializing swh-model in RPC calls. This allows running the validating proxy on the client side instead of the server side. | 18 February 2020, 13:22:33 UTC |
80befa5 | Valentin Lorentz | 12 February 2020, 15:49:01 UTC | Make storage proxies use swh-model objects instead of dicts. This means that instead of having the validation proxy right before the backend class, it must now be at the beginning of pipelines. | 18 February 2020, 12:45:46 UTC |
29b0948 | Valentin Lorentz | 12 February 2020, 15:18:29 UTC | Fix FilteringProxy to not drop skipped-contents with a missing sha1_git. Passes them all to the backend instead of silently dropping them all if any of them is not missing. | 18 February 2020, 12:45:22 UTC |
51b2016 | Valentin Lorentz | 12 February 2020, 15:08:10 UTC | Fix inconsistent behavior of skipped_content_missing across backends. Two fixes: * in-mem ignored None keys * cassandra yielded input dicts as-is instead of a dict with just the hashes | 14 February 2020, 16:49:05 UTC |
03c2a02 | Valentin Lorentz | 06 February 2020, 13:22:29 UTC | Re-raise StorageArgumentException through API calls. So clients will get a nice exception looking like the original one, instead of generic RemoteApiError. | 12 February 2020, 15:51:26 UTC |
fa4dac4 | Jenkins for Software Heritage | 12 February 2020, 13:13:47 UTC | Updated debian changelog for version 0.0.172 | 12 February 2020, 13:13:47 UTC |
65f1c6e | Jenkins for Software Heritage | 12 February 2020, 13:13:47 UTC | Update upstream source from tag 'debian/upstream/0.0.172' Update to upstream version '0.0.172' with Debian dir 355e8d105e8ea7d1af1c1238c772d8e57697fe84 | 12 February 2020, 13:13:47 UTC |
b668651 | Jenkins for Software Heritage | 12 February 2020, 13:13:45 UTC | New upstream version 0.0.172 | 12 February 2020, 13:13:45 UTC |
652ecf0 | Antoine R. Dumont (@ardumont) | 07 February 2020, 01:44:43 UTC | storages: Refactor journal operations with a dedicated writer collab Prior to this commit, the code was triplicated across the storage backends. Now all storages use the same collaborator whose concern is writing to the storage. Could be a stepping stone to make that a proxy storage. | 11 February 2020, 14:40:36 UTC |
adcbf95 | Valentin Lorentz | 11 February 2020, 13:12:12 UTC | Fix RecursionError when storage proxies are deepcopied or unpickled. They both get attributes (eg. __setattr__) before setting any attribute, and don't call the constructor; so self.storage is not set when __getattr__ is called for the first times. | 11 February 2020, 13:12:12 UTC |
a2e565d | Antoine R. Dumont (@ardumont) | 10 February 2020, 17:43:05 UTC | d/control: Update runtime dependency | 10 February 2020, 17:43:05 UTC |
de7797d | Antoine R. Dumont (@ardumont) | 10 February 2020, 16:17:59 UTC | tests: Remove print statement | 10 February 2020, 16:18:33 UTC |
5b3c940 | Antoine R. Dumont (@ardumont) | 10 February 2020, 12:18:21 UTC | storages: Refactor objstorage operations with a dedicated collaborator Prior to this commit, the code was triplicated across the storage backends. Now all storages use the same collaborator whose concern is writing to the objstorage. | 10 February 2020, 12:38:29 UTC |
68ff23c | Valentin Lorentz | 07 February 2020, 15:35:20 UTC | Add a validation proxy for _add() methods. It converts input dictionaries into swh-model objects, which validates them and raises an appropriate error. This removes duplicated validation code that is currently present in all three storage backends. It also uses well-defined object types instead of loose dicts, which gives these _add() methods a more strict type. | 10 February 2020, 11:03:49 UTC |
6383637 | Valentin Lorentz | 06 February 2020, 12:26:47 UTC | Unify exception raised by invalid input to API endpoints. This is a first step toward not pickling exceptions. | 07 February 2020, 10:40:23 UTC |
5a6b025 | Jenkins for Software Heritage | 06 February 2020, 14:07:38 UTC | Updated debian changelog for version 0.0.171 | 06 February 2020, 14:07:38 UTC |
ae6befc | Jenkins for Software Heritage | 06 February 2020, 14:07:37 UTC | Update upstream source from tag 'debian/upstream/0.0.171' Update to upstream version '0.0.171' with Debian dir 55757ca0c14457bc364a00b1f333b4c38160f5b2 | 06 February 2020, 14:07:37 UTC |
be225df | Jenkins for Software Heritage | 06 February 2020, 14:07:35 UTC | New upstream version 0.0.171 | 06 February 2020, 14:07:35 UTC |
2b029b7 | Valentin Lorentz | 05 February 2020, 11:17:59 UTC | Split 'content_add' method into 'content_add' and 'skipped_content_add'. Respectively to add present content and skipped content. This simplifies the logic of both methods, and is a necessary step to typing / using swh-model objects everywhere, as contents have quite different attributes depending on whether they are present or missing. | 06 February 2020, 13:29:31 UTC |
93ea487 | Valentin Lorentz | 04 February 2020, 12:38:32 UTC | Increase Cassandra requests timeout to 1 second. 100ms worked fine so far, but we're starting to get some timeouts on the Azure test cluster. Multiplying the timeout by 10 should give us ample room to work with. | 04 February 2020, 12:38:32 UTC |
4a0a055 | Antoine R. Dumont (@ardumont) | 03 February 2020, 16:31:34 UTC | d/changelog: Bump new release | 03 February 2020, 16:33:29 UTC |
1d233df | Antoine R. Dumont (@ardumont) | 03 February 2020, 16:04:49 UTC | d/control: Update build dependencies Forcing use of openjdk-11-jre as there are issues to make cassandra run with the default dependency (openjdk-8-jre). | 03 February 2020, 16:33:29 UTC |
a119a7a | Antoine R. Dumont (@ardumont) | 03 February 2020, 15:01:20 UTC | d/changelog: Update dependencies Build will still fail but we will have more information to debug | 03 February 2020, 15:01:20 UTC |
41ca2bf | Antoine R. Dumont (@ardumont) | 03 February 2020, 13:36:08 UTC | d/control: Add build dependencies | 03 February 2020, 13:36:08 UTC |
bfe4586 | Jenkins for Software Heritage | 03 February 2020, 13:23:48 UTC | Updated debian changelog for version 0.0.170 | 03 February 2020, 13:23:48 UTC |
2812822 | Jenkins for Software Heritage | 03 February 2020, 13:23:47 UTC | Update upstream source from tag 'debian/upstream/0.0.170' Update to upstream version '0.0.170' with Debian dir 08005486c202dfaa978bf8aadb9469768d769e1e | 03 February 2020, 13:23:47 UTC |
a66e16c | Jenkins for Software Heritage | 03 February 2020, 13:23:46 UTC | New upstream version 0.0.170 | 03 February 2020, 13:23:46 UTC |
b315f9d | Valentin Lorentz | 31 January 2020, 16:09:31 UTC | Tune Cassandra test config for lower test latency. | 03 February 2020, 12:31:28 UTC |
25941d5 | Valentin Lorentz | 03 February 2020, 11:25:49 UTC | Make tests reuse the same keyspace/schema instead of recreating it for each test. This makes tests run 16 times faster than https://forge.softwareheritage.org/D2612 (which is itself 3 times faster than this commit's parent) | 03 February 2020, 11:26:24 UTC |
eb155ad | Valentin Lorentz | 21 January 2020, 13:59:26 UTC | Add Cassandra backend. | 31 January 2020, 15:05:53 UTC |
ba373c9 | Jenkins for Software Heritage | 30 January 2020, 13:26:23 UTC | Updated debian changelog for version 0.0.169 | 30 January 2020, 13:26:23 UTC |
7dc7029 | Jenkins for Software Heritage | 30 January 2020, 13:26:22 UTC | Update upstream source from tag 'debian/upstream/0.0.169' Update to upstream version '0.0.169' with Debian dir aae69d992484644229881c13c543f8489e58a53c | 30 January 2020, 13:26:22 UTC |
523f2eb | Jenkins for Software Heritage | 30 January 2020, 13:26:21 UTC | New upstream version 0.0.169 | 30 January 2020, 13:26:21 UTC |
cf45ec6 | Antoine R. Dumont (@ardumont) | 30 January 2020, 11:22:16 UTC | retry: Add retry behavior on pipeline storage with flushing failure Currently, wrong "hash collisions" are happening a lot on ingestion [1] [2] [3] The last loading step (flush) is failing on most loaders (git, npm, etc...). This commits adds the retry behavior to the current pipeline storage deployed. Which should decrease the frequency of that error. The remaining hash collision which won't subside should be then real hash collisions. [1] https://sentry.softwareheritage.org/share/issue/102aace238fe4ba6b49bcc5531f7c2bf/ [2] https://sentry.softwareheritage.org/share/issue/8e8b48a1d94c465b8109e76311ecdbe7/ [3] https://sentry.softwareheritage.org/share/issue/d4f1208b7eec4b43b11e38494ff039cc/ | 30 January 2020, 11:22:16 UTC |
0dc1626 | Jenkins for Software Heritage | 30 January 2020, 10:25:31 UTC | Updated debian changelog for version 0.0.168 | 30 January 2020, 10:25:31 UTC |
0c6e828 | Jenkins for Software Heritage | 30 January 2020, 10:25:30 UTC | Update upstream source from tag 'debian/upstream/0.0.168' Update to upstream version '0.0.168' with Debian dir 83be097d86cd3d927e896c561f682dfeae685245 | 30 January 2020, 10:25:30 UTC |
3e6d2bf | Jenkins for Software Heritage | 30 January 2020, 10:25:29 UTC | New upstream version 0.0.168 | 30 January 2020, 10:25:29 UTC |
1608fcd | Valentin Lorentz | 29 January 2020, 15:15:40 UTC | Allow deprecated endpoints to be missing from a backend class. | 29 January 2020, 15:50:10 UTC |
68702b5 | Stefano Zacchiroli | 29 January 2020, 13:24:09 UTC | CONTRIBUTORS: add Daniele Serafini | 29 January 2020, 13:24:09 UTC |
32d455b | Valentin Lorentz | 24 January 2020, 17:10:40 UTC | Rename in_memory.Storage to in_memory.InMemoryStorage. For consistency with the other class names. | 29 January 2020, 12:34:38 UTC |
d4fb270 | Valentin Lorentz | 24 January 2020, 16:49:42 UTC | Move Storage documentation and endpoint paths to a new StorageInterface class Documentation was duplicated between the in-mem and postgresql storage, and one of them regularly goes out of date. This deduplicates them both to a new class. This new class is also the one declaring the API paths, as it did not make sense to have this declaration on the postgresql storage. Last but not least, this commit adds a test that checks backend classes have all the functions, and they have exactly the same signature as the interface. This will catch stupid bugs before production, eg. if an argument does not have the same name in all classes. | 29 January 2020, 11:16:55 UTC |
1775edd | Valentin Lorentz | 27 January 2020, 16:37:50 UTC | in_memory: Fix content_get_metadata when there is no 'data' key. | 27 January 2020, 16:37:57 UTC |
0f51e8a | Valentin Lorentz | 24 January 2020, 13:41:08 UTC | Remove cur/db arguments from the in-mem storage. They shouldn't be there; bad copy-pasting. | 24 January 2020, 15:53:27 UTC |
1cd53c1 | Valentin Lorentz | 24 January 2020, 15:46:47 UTC | Implement content_update for the in-mem storage. | 24 January 2020, 15:47:10 UTC |
7245ee9 | Jenkins for Software Heritage | 24 January 2020, 14:01:58 UTC | Updated debian changelog for version 0.0.167 | 24 January 2020, 14:01:58 UTC |
e180f2a | Jenkins for Software Heritage | 24 January 2020, 14:01:57 UTC | Update upstream source from tag 'debian/upstream/0.0.167' Update to upstream version '0.0.167' with Debian dir e797a96b300338124b09f8b75bef2996fc8220d9 | 24 January 2020, 14:01:57 UTC |
e62d6e4 | Jenkins for Software Heritage | 24 January 2020, 14:01:56 UTC | New upstream version 0.0.167 | 24 January 2020, 14:01:56 UTC |
c8389c2 | Antoine R. Dumont (@ardumont) | 24 January 2020, 13:54:26 UTC | 146: Fix typo | 24 January 2020, 13:54:26 UTC |
2ebcdf3 | Antoine R. Dumont (@ardumont) | 24 January 2020, 10:57:46 UTC | pgstorage: Empty temp tables instead of dropping them Due to our pattern of adding objects [1], vacuum is triggered regularly on pg_catalog.*, having an heavy impact on performance. This commit tries to avoid the dropping the temporary tables part, emptying them instead (they still are dropped at the end of the session but less often). This should decrease the bloat on pg_catalog.* tables. [1] - create temporary table - insert data from temporary table to production table with filtering - drop temporary table | 24 January 2020, 11:14:00 UTC |
7db8312 | Jenkins for Software Heritage | 24 January 2020, 09:00:12 UTC | Updated debian changelog for version 0.0.166 | 24 January 2020, 09:00:12 UTC |
9da9a85 | Jenkins for Software Heritage | 24 January 2020, 09:00:11 UTC | Update upstream source from tag 'debian/upstream/0.0.166' Update to upstream version '0.0.166' with Debian dir ee457449f805ebd9a5e9e10c911c8939bc4d5b4c | 24 January 2020, 09:00:11 UTC |
74bb123 | Jenkins for Software Heritage | 24 January 2020, 09:00:10 UTC | New upstream version 0.0.166 | 24 January 2020, 09:00:10 UTC |
cc25810 | Daniele Serafini | 22 January 2020, 14:03:41 UTC | assert list doesn't have too many values | 22 January 2020, 14:24:08 UTC |
2ebce62 | Daniele Serafini | 22 January 2020, 13:28:38 UTC | test endpoint: content_missing (sha1_git), snapshot_missing | 22 January 2020, 14:24:08 UTC |
c40d327 | Daniele Serafini | 22 January 2020, 13:27:36 UTC | in memory changes | 22 January 2020, 14:24:08 UTC |
55ebd23 | Daniele Serafini | 22 January 2020, 11:29:07 UTC | storage: Add endpoint to get missing content (by sha1_git) and missing snapshot | 22 January 2020, 14:24:08 UTC |
cfee7b5 | Valentin Lorentz | 22 January 2020, 11:25:24 UTC | Remove redundant config checks in load_and_check_config. 1. There is no reason to force the server to serve only the 'local' backend anymore 2. Missing arguments will error when instantiating the backend class. | 22 January 2020, 11:25:24 UTC |
2454a78 | Antoine Lambert | 17 January 2020, 16:02:35 UTC | docs: Fix sphinx warnings Related to T2188 | 17 January 2020, 16:02:35 UTC |
2dc17cd | Valentin Lorentz | 17 January 2020, 13:49:26 UTC | Remove 'id' and 'object_id' from the output of object_find_by_sha1_git. 'id' is not used anymore, and 'object_id' never was. This commit slightly simplifies existing code, and will allow some deduplication in the upcoming Cassandra backend. | 17 January 2020, 14:10:56 UTC |
b5a5084 | Valentin Lorentz | 16 January 2020, 15:17:44 UTC | Make origin_visit_get_random return None instead of {} if there are no results. This is more consistent with other endpoints. | 17 January 2020, 14:10:36 UTC |
dba9e04 | Valentin Lorentz | 16 January 2020, 13:16:11 UTC | Rewrite test_content_get_partition_empty to not assume partitions are based on sha1. This is not true of the upcoming Cassandra backend. | 17 January 2020, 14:10:11 UTC |
e584655 | Valentin Lorentz | 16 January 2020, 13:13:43 UTC | Remove test_content_*_same_input, which check for behavior we do not want to guarantee. They check that content_add deduplicates with existing content/duplicated input. This is unneeded (the loaders don't send such data), so providing these guarantees unnecessarily complicates swh-storage code, especially the upcoming Cassandra backend. | 17 January 2020, 14:09:38 UTC |
92af2cd | Jenkins for Software Heritage | 17 January 2020, 13:09:40 UTC | Updated debian changelog for version 0.0.165 | 17 January 2020, 13:09:40 UTC |
ac79988 | Jenkins for Software Heritage | 17 January 2020, 13:09:39 UTC | Update upstream source from tag 'debian/upstream/0.0.165' Update to upstream version '0.0.165' with Debian dir c24845a1a2ad014144923b527940b11c269f8081 | 17 January 2020, 13:09:39 UTC |
079fa61 | Jenkins for Software Heritage | 17 January 2020, 13:09:38 UTC | New upstream version 0.0.165 | 17 January 2020, 13:09:38 UTC |
bf77f14 | Antoine R. Dumont (@ardumont) | 17 January 2020, 12:55:36 UTC | storage.retry: Fix objects loading when using generator parameters This will fix related retry error [1] [1] https://sentry.softwareheritage.org/share/issue/ddbbdd3c235b40ca826bf2c820989f14/ Related to cc29708564c35575f569e863f028a480a9905cf4 Related to D2543 | 17 January 2020, 12:55:36 UTC |
a7cbaf1 | Jenkins for Software Heritage | 16 January 2020, 17:05:03 UTC | Updated debian changelog for version 0.0.164 | 16 January 2020, 17:05:03 UTC |
4578987 | Jenkins for Software Heritage | 16 January 2020, 17:05:02 UTC | Update upstream source from tag 'debian/upstream/0.0.164' Update to upstream version '0.0.164' with Debian dir 23094cd1a4429ea115d7ffdaac67f3aea22ae353 | 16 January 2020, 17:05:02 UTC |
4738b2e | Jenkins for Software Heritage | 16 January 2020, 17:05:01 UTC | New upstream version 0.0.164 | 16 January 2020, 17:05:01 UTC |
cc29708 | Antoine Lambert | 16 January 2020, 16:16:09 UTC | storage: Fix objects loading when using generator parameters Some objects (directories, origins, releases, revisions) will not be added into the storage if they are provided as generator parameters to the *_add methods instead of lists. So ensure to transform generators into lists before processing the objects. | 16 January 2020, 16:53:22 UTC |
0b3610e | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:26:54 UTC | d/changelog: Bump new release | 14 January 2020, 16:27:58 UTC |
8c3ef4b | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:21:39 UTC | d/control: Add missing test dependency | 14 January 2020, 16:22:48 UTC |
6690419 | Jenkins for Software Heritage | 14 January 2020, 16:17:45 UTC | Updated debian changelog for version 0.0.163 | 14 January 2020, 16:17:45 UTC |
0481c44 | Jenkins for Software Heritage | 14 January 2020, 16:17:45 UTC | Update upstream source from tag 'debian/upstream/0.0.163' Update to upstream version '0.0.163' with Debian dir 4ed2720d31becfe82ffadf4a9175de855e6ee5da | 14 January 2020, 16:17:45 UTC |
e362b9d | Jenkins for Software Heritage | 14 January 2020, 16:17:43 UTC | New upstream version 0.0.163 | 14 January 2020, 16:17:43 UTC |
b641e53 | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:10:13 UTC | d/control: Align dependency versions with requirements.txt | 14 January 2020, 16:10:13 UTC |
e70f427 | Antoine R. Dumont (@ardumont) | 14 January 2020, 16:09:54 UTC | d/control: Add new runtime dependency | 14 January 2020, 16:09:54 UTC |
8dcac2b | Antoine R. Dumont (@ardumont) | 14 January 2020, 12:40:56 UTC | retry: Implement content_add_metadata endpoint with retry policy | 14 January 2020, 12:45:11 UTC |
aa588c9 | Antoine R. Dumont (@ardumont) | 14 January 2020, 12:40:39 UTC | retry: Migrate to tenacity Which is a maintained fork of retry | 14 January 2020, 12:40:39 UTC |
4aa4d79 | Antoine R. Dumont (@ardumont) | 11 January 2020, 11:05:57 UTC | test_retry: Improve and align consistently assertion checks | 14 January 2020, 10:41:35 UTC |
2b7d770 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:56:04 UTC | storage.retry: Implement snapshot_add with retry policy | 14 January 2020, 10:41:35 UTC |
df3f33f | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:47:57 UTC | storage.retry: Implement release_add with retry policy | 14 January 2020, 10:41:35 UTC |
54890f7 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:44:20 UTC | storage.retry: Implement revision_add with retry policy | 14 January 2020, 10:41:35 UTC |
a8efa95 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:38:48 UTC | storage.retry: Implement directory_add with retry policy | 14 January 2020, 10:41:34 UTC |
dddb6d9 | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:37:41 UTC | in_memory: Make directory_get_random return None when storage empty | 14 January 2020, 10:41:34 UTC |
2dd578c | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:25:31 UTC | storage.retry: Implement origin_visit_update with retry policy | 14 January 2020, 10:41:34 UTC |
32c460c | Antoine R. Dumont (@ardumont) | 11 January 2020, 10:05:13 UTC | storage.retry: Implement origin_metadata_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
3cf7adb | Antoine R. Dumont (@ardumont) | 10 January 2020, 17:10:03 UTC | storage.retry: Implement metadata_provider_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
08f2f38 | Antoine R. Dumont (@ardumont) | 10 January 2020, 15:48:11 UTC | storage.retry: Implement tool_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
fe6440e | Antoine R. Dumont (@ardumont) | 10 January 2020, 15:37:24 UTC | storage.retry: Implement origin_visit_add endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
351b977 | Antoine R. Dumont (@ardumont) | 10 January 2020, 14:59:52 UTC | storage.retry: Implement origin_add_one endpoint with retry policy | 14 January 2020, 10:41:34 UTC |
024eaea | Antoine R. Dumont (@ardumont) | 13 January 2020, 15:14:50 UTC | content_get_metadata: Change api to return Dict[bytes, List[Dict]] Clients will be able to introspect directly from the result whether a content is known or not. | 14 January 2020, 10:38:20 UTC |
07b6dc3 | Antoine R. Dumont (@ardumont) | 13 January 2020, 14:18:55 UTC | storage.content_get_metadata: Adapt to return nothing if inexistent id is passed as input | 13 January 2020, 14:18:55 UTC |
4837f46 | Antoine R. Dumont (@ardumont) | 09 January 2020, 16:17:30 UTC | storage: Add basic proxy storage with retry policy | 10 January 2020, 13:40:01 UTC |
fdf2a3c | Valentin Lorentz | 22 November 2019, 15:09:55 UTC | Add Storage.content_get_partition endpoint, to replace content_get_range. With no guarantees on the order or how partitioning is done, and with the new-style pagination. | 17 December 2019, 12:59:09 UTC |
0f94312 | Valentin Lorentz | 21 November 2019, 13:28:43 UTC | Add endpoint 'origin_list', that will replace 'origin_get_range'. And uses the new pagination scheme, instead of origin ids. | 16 December 2019, 14:17:21 UTC |
31a8dcf | Jenkins for Software Heritage | 16 December 2019, 13:41:40 UTC | Updated debian changelog for version 0.0.162 | 16 December 2019, 13:41:40 UTC |