https://github.com/SoftwareHeritage/swh-storage

sort by:
Revision Author Date Message Commit Date
82a376c Updated backport on buster-swh from debian/0.24.1-1_swh1 (unstable-swh) 04 March 2021, 22:43:55 UTC
01edfb6 Merge tag 'debian/0.24.1-1_swh1' into debian/buster-swh 04 March 2021, 22:43:54 UTC
a905850 Updated debian changelog for version 0.24.1 04 March 2021, 22:39:03 UTC
2ce60b4 Update upstream source from tag 'debian/upstream/0.24.1' Update to upstream version '0.24.1' with Debian dir 5ee23ceab866b0a4b34fcececd972236d18799e9 04 March 2021, 22:39:03 UTC
b4574cb New upstream version 0.24.1 04 March 2021, 22:39:01 UTC
88ff2c2 postgresql: Ensure a minimum limit for the snapshot branches query With small limits (< 10), the snapshot branches query can degenerate into using the deduplication index on snapshot_branch (name, target, target_type), and the postgresql planner happily scans several hundred million rows. So ensure a minimum limit value of 10 before executing the query for optimal performances when a small branches_count value is provided to the snapshot_get_branches method of the Storage interface. Related to P966 03 March 2021, 16:49:20 UTC
ce8335d Remove the remaining references to the deprecated SWHID class 03 March 2021, 16:46:50 UTC
f46244b tests: Drop hypothesis < 6 requirement Ensure tests can be executed using hypothesis >= 6 by suppressing the function_scoped_fixture health check on tests that use a function scope fixture in combination with @given that does not need to be reset between individual hypothesis examples. 03 March 2021, 10:53:08 UTC
6f244c7 Updated backport on buster-swh from debian/0.24.0-1_swh1 (unstable-swh) 02 March 2021, 09:16:10 UTC
e66491b Merge tag 'debian/0.24.0-1_swh1' into debian/buster-swh 02 March 2021, 09:16:09 UTC
de1ffbe Updated debian changelog for version 0.24.0 02 March 2021, 09:11:16 UTC
19e6fbb Update upstream source from tag 'debian/upstream/0.24.0' Update to upstream version '0.24.0' with Debian dir 71fb0d472040239539f3beb6144a6a2e65d799eb 02 March 2021, 09:11:14 UTC
fd0efad New upstream version 0.24.0 02 March 2021, 09:11:13 UTC
14739c5 RawExtrinsicMetadata: update to use the API in swh-model 1.0.0 01 March 2021, 16:38:44 UTC
2388748 storage_tests: recompute ids when evolving RawExtrinsicMetadata objects. For now this does nothing as RawExtrinsicMetadata has no 'id' field, but the equality assertions will become errors when the next version of swh.model is released. 25 February 2021, 15:33:40 UTC
4197263 Updated backport on buster-swh from debian/0.23.2-1_swh1 (unstable-swh) 19 February 2021, 11:03:48 UTC
b35fefc Merge tag 'debian/0.23.2-1_swh1' into debian/buster-swh 19 February 2021, 11:03:47 UTC
3d9685b Updated debian changelog for version 0.23.2 19 February 2021, 10:58:51 UTC
53285b9 Update upstream source from tag 'debian/upstream/0.23.2' Update to upstream version '0.23.2' with Debian dir 39796b0edac0471541994c14a7f6c71c2c459858 19 February 2021, 10:58:50 UTC
f56267f New upstream version 0.23.2 19 February 2021, 10:58:48 UTC
f3ef6e6 storage: Implement visit types filtering in origin_search method Enable to filter searched origins by visit types. Add a new optional visit_types parameter to origin_search method in StorageInterface. Implement visit types filtering in storage backends, an origin wil be returned if it has any of the requested visit types. This is clearly not designed to be used in production due to performance issues but rather in testing environments with small archive dataset. Related to T2869 19 February 2021, 10:36:29 UTC
7b4c124 167: Make the migration script unblocking 17 February 2021, 09:18:26 UTC
e0cba3c Updated backport on buster-swh from debian/0.23.1-1_swh1 (unstable-swh) 16 February 2021, 16:33:17 UTC
13ae656 Merge tag 'debian/0.23.1-1_swh1' into debian/buster-swh 16 February 2021, 16:33:16 UTC
b5eccae Updated debian changelog for version 0.23.1 16 February 2021, 16:28:25 UTC
014a193 Update upstream source from tag 'debian/upstream/0.23.1' Update to upstream version '0.23.1' with Debian dir fe827c9c7c8bf012f75037e6da85c36511eb7361 16 February 2021, 16:28:24 UTC
f7f161d New upstream version 0.23.1 16 February 2021, 16:28:23 UTC
cc3eb4b Switch anonymized replayer test to use pytest parametrization This allows us to only read the kafka topics once instead of twice in the same tests, which is apparently a hard thing to do in a way compatible with both confluent-kafka 1.5 and 1.6. 16 February 2021, 16:09:03 UTC
724a619 Updated backport on buster-swh from debian/0.23.0-1_swh2 (unstable-swh) 16 February 2021, 13:52:08 UTC
5bd35c3 Merge tag 'debian/0.23.0-1_swh2' into debian/buster-swh 16 February 2021, 13:52:08 UTC
707d6ed d/changelog: Bump new release to fix build Related to T3053 16 February 2021, 13:37:48 UTC
afacfa0 d/changelog: Fix typos so pre-commit stop failing 16 February 2021, 13:37:35 UTC
3f9b52a d/rules: Use Build-Conflicts to exclude openjdk-{17,16,15}-jre deps So the correct java version is pulled for the tests (cassandr) to run ok. Related to T3053#58816 16 February 2021, 13:31:06 UTC
253aa41 d/rules: Position the jre dependency prior to the cassandra one That way, the openjdk-17-jre is not pulled within the chroot. It it's pulled, it becomes the default java to start cassandra and that does not work. Related to T3053 16 February 2021, 12:10:12 UTC
78c70c7 d/rules: Overwrite the JAVA_HOME so tests are running fine Related to T3053 16 February 2021, 11:05:21 UTC
c45a660 d/rules: Make test fail immediately on error Related to T3053 16 February 2021, 11:05:21 UTC
4e2a041 d/rules: Activate back cassandra tests with debug logs Related to T3053 16 February 2021, 11:05:21 UTC
ce15713 d/rules: Deactivate cassandra tests They fail as the cassandra backend somehow does not run: ``` > if not running: > raise Exception("cassandra process stopped unexpectedly.") E Exception: cassandra process stopped unexpectedly. ``` Related to T3053 16 February 2021, 11:05:17 UTC
33badfc Updated debian changelog for version 0.23.0 15 February 2021, 14:39:04 UTC
af06625 Update upstream source from tag 'debian/upstream/0.23.0' Update to upstream version '0.23.0' with Debian dir 8f59db23317b3afa3f294e66416d6faf260447ae 15 February 2021, 14:39:03 UTC
5c6b53c New upstream version 0.23.0 15 February 2021, 14:39:02 UTC
e0e88b2 storage: Refactor OriginVisitStatus instantiation 09 February 2021, 16:01:26 UTC
d30ca93 db: Unify sql joins on origin_visit_status using "USING" 09 February 2021, 16:01:26 UTC
046fe57 storage.postgresql: Use origin_visit_status.type value as source This stops using the origin_visit.type as fallback values as now, the database has been migrated. So this makes the origin_visit_status.type a not nullable column. This also drops now redundant join instructions on origin_visit table when reading. Related to T2968 09 February 2021, 16:01:25 UTC
51df58e test_replay: Fix hang since confluent-kafka 1.6 release Side effect of the following commit in librdkafka 1.6: https://github.com/edenhill/librdkafka/commit/f418e0f721518d71ff533759698b647cb2e89b80 Tests was relying on a buggy behavior of the mocked kafka cluster: two subsequent consumers setup with the same group id should receive a different set of messages, rather than the same set of messages. Also explicitly commit messages once consumed. 09 February 2021, 14:56:15 UTC
b038383 postgresql: Fix dbversion() to return the max version instead of a random one. 08 February 2021, 11:13:03 UTC
efd8815 buffer: ensure objects are flushed in topological order This new integration test checks that, when flushing the buffer storage, the addition functions of the underlying storage backend are called in topological order (content, directory, revision, release then snapshot). This reduces the probability of "data consistency" regressions caused by the use of the buffering storage proxy alone. 04 February 2021, 18:17:11 UTC
1526107 Return an accurate summary from buffer's flush() method The earlier implementation would only return summary data from keys that existed in the last `_add` backend method run, rather than collating all the results. 04 February 2021, 18:14:03 UTC
5b3e6c9 buffer: add support for snapshots This is mostly a consistency addition, considering that most (if not all) loaders will only add a single snapshot. The common pattern of loading objects in topological order (content > directory > revision > release > snapshot), then flushing the storage, is now fully consistent; Without this addition, the snapshot addition would reach the backend storage before all other objects are added, leading to potential inconsistencies if the flush of other object types fails. 04 February 2021, 13:37:12 UTC
18967ed buffer: add type annotations for tests 04 February 2021, 09:19:34 UTC
f099f4c Updated backport on buster-swh from debian/0.22.0-1_swh1 (unstable-swh) 03 February 2021, 11:20:05 UTC
bc076da Merge tag 'debian/0.22.0-1_swh1' into debian/buster-swh 03 February 2021, 11:20:04 UTC
41b5e73 Updated debian changelog for version 0.22.0 03 February 2021, 11:15:28 UTC
ecfc90c Update upstream source from tag 'debian/upstream/0.22.0' Update to upstream version '0.22.0' with Debian dir ac16de6be52bc9e55ffbc4ea50a81eb757f0833e 03 February 2021, 11:15:27 UTC
f1e523e New upstream version 0.22.0 03 February 2021, 11:15:26 UTC
9a9f234 storage: Make origin_get_latest_visit_status return OriginVisitStatus This returned a Tuple[OriginVisit, OriginVisitStatus]. This was required to have the missing information "type" for visit-status. This is no longer needed as now OriginVisitStatus holds the type information. 01 February 2021, 11:06:35 UTC
626b0bf Change origin_visit_status_get_random interface to return visit_status This returned a Tuple[OriginVisit, OriginVisitStatus] which is no longer needed as now OriginVisitStatus held the type information now. 01 February 2021, 11:06:34 UTC
f6ae8a0 Write introduction to swh-storage. Explains: * when to use swh-web instead * that `get_storage` should always be used to instantiate the storage * `StorageInterface` * model objects * pagination * backends 01 February 2021, 11:03:02 UTC
68331b5 Updated backport on buster-swh from debian/0.21.1-1_swh1 (unstable-swh) 28 January 2021, 13:24:27 UTC
3f1dc5c Merge tag 'debian/0.21.1-1_swh1' into debian/buster-swh 28 January 2021, 13:24:26 UTC
3d71ec6 Updated debian changelog for version 0.21.1 28 January 2021, 13:19:24 UTC
9580e45 Update upstream source from tag 'debian/upstream/0.21.1' Update to upstream version '0.21.1' with Debian dir 086f067ed910d482b6d861ee9c917a169e5d82de 28 January 2021, 13:19:23 UTC
57d3066 New upstream version 0.21.1 28 January 2021, 13:19:21 UTC
76de53c Correctly return origin_visit_status.type value everywhere If the type is not present on an origin_visit_status, it should be computed from the origin_visit. There were some methods which only return the origin_visit_status value. It breaks the webapp mangling the type to empty value on the search result page. Related to T3001 28 January 2021, 11:15:11 UTC
55d7898 Updated backport on buster-swh from debian/0.21.0-1_swh1 (unstable-swh) 20 January 2021, 14:56:52 UTC
7f014af Merge tag 'debian/0.21.0-1_swh1' into debian/buster-swh 20 January 2021, 14:56:51 UTC
5bd8ed5 Updated debian changelog for version 0.21.0 20 January 2021, 14:52:21 UTC
ef7b545 Update upstream source from tag 'debian/upstream/0.21.0' Update to upstream version '0.21.0' with Debian dir 12ba6aba188abf4a213a5074e30832a0a4676790 20 January 2021, 14:52:20 UTC
47e0a4c New upstream version 0.21.0 20 January 2021, 14:52:18 UTC
e433255 db: Allow new status values not_found, failed to OriginVisitStatus Related to T2961 20 January 2021, 14:36:12 UTC
47f78d7 Updated backport on buster-swh from debian/0.20.0-1_swh1 (unstable-swh) 20 January 2021, 09:34:29 UTC
fc66b25 Merge tag 'debian/0.20.0-1_swh1' into debian/buster-swh 20 January 2021, 09:34:28 UTC
7c47e51 Updated debian changelog for version 0.20.0 20 January 2021, 09:29:54 UTC
37a1ac6 Update upstream source from tag 'debian/upstream/0.20.0' Update to upstream version '0.20.0' with Debian dir 8a95cef4e85584d63f5645c29998daf165907e77 20 January 2021, 09:29:54 UTC
45803cf New upstream version 0.20.0 20 January 2021, 09:29:52 UTC
d04165f Add type to the origin_visit_status topic useful when the type is not yet populated in the database Related to T2966 18 January 2021, 10:49:34 UTC
c24d35f Add persistence of the field OriginVisitStatus.type (!) A new database upgrade is needed (165.sql) for postgresql backend Related to T2964 15 January 2021, 11:38:38 UTC
da55308 Make test_content_add_race fail for the right reason. Since 209de5dbaa127dacd114fbbd084f22632982eb77, it was failing because of: TypeError("content_add() got an unexpected keyword argument 'db'") 15 January 2021, 10:30:36 UTC
530d11f Updated backport on buster-swh from debian/0.19.0-1_swh1 (unstable-swh) 14 January 2021, 10:23:13 UTC
b6ee9fe Merge tag 'debian/0.19.0-1_swh1' into debian/buster-swh 14 January 2021, 10:23:13 UTC
753936e Updated debian changelog for version 0.19.0 14 January 2021, 10:18:32 UTC
cee711b Update upstream source from tag 'debian/upstream/0.19.0' Update to upstream version '0.19.0' with Debian dir f25b1bc13ecf51db73cfd3eb91a6550d40da724b 14 January 2021, 10:18:31 UTC
2204346 New upstream version 0.19.0 14 January 2021, 10:18:30 UTC
0b44b37 Adapt cassandra storage to ignore the new OriginVisitStatus.type field Depends on D4848 Related to T2443 13 January 2021, 10:06:12 UTC
728c3ee Allow to use the JAVA_HOME environment for cassandra tests This allows to enforce a specific version of java to be used. For example, since cassandra seems not to support java 14 yet, this allows to run tests on bullseye: JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/ pytest swh 13 January 2021, 09:56:07 UTC
30945a5 Enforce hypothesis <6 to prevent test breakage hypothesis 6 upgraded a warning into an error: now raises a FailedHealthCheck when using a pytest fixture with a @given generative test set. See https://hypothesis.readthedocs.io/en/latest/healthchecks.html 13 January 2021, 09:42:42 UTC
74e6f58 Make the CREATE_TABLES_QUERIES in cassandra/schema.py an explicit list prevent being fooled by a missing '\n'. 08 January 2021, 13:20:08 UTC
2b35198 Add a cli section in the doc 18 December 2020, 12:41:23 UTC
04ae89f storage.backfill: Allow cli run for origin_visit_status as well 24 November 2020, 17:21:21 UTC
64ee845 conftest: Reference swh.core.db.pytest_plugin As it's exposed through the swh.storage.pytest_plugin itself used by other swh modules, this needs to be declared to avoid other swh module build failures. Related to T2746 24 November 2020, 13:08:12 UTC
ec2f392 Updated backport on buster-swh from debian/0.18.0-1_swh1 (unstable-swh) 23 November 2020, 14:00:20 UTC
1fc26df Merge tag 'debian/0.18.0-1_swh1' into debian/buster-swh 23 November 2020, 14:00:19 UTC
cbbaeea Updated debian changelog for version 0.18.0 23 November 2020, 13:52:33 UTC
e5bce82 Update upstream source from tag 'debian/upstream/0.18.0' Update to upstream version '0.18.0' with Debian dir 6ef3ab996694ae4582886e07c25624fa073dc956 23 November 2020, 13:52:32 UTC
4c46835 New upstream version 0.18.0 23 November 2020, 13:52:31 UTC
e289593 requirements-test.txt: Drop no longer needed pytest-postgresql requirement requirements-swh.txt already declares the swh.core[db] dependency which transitively pulls it. Related to T2746 23 November 2020, 12:07:45 UTC
b1375bc d/control: Rework test dependency Related to T2746 23 November 2020, 10:54:40 UTC
0065d4d backfill: Reverse flawed logic in SnapshotBranch generation The previous code would nullify all non-null branches, and try to create a SnapshotBranch out of null branches. 13 November 2020, 15:51:29 UTC
f501136 migrate_extrinsic_metadata: don't crash when deb revisions aren't referenced by any snapshot As this happens for about 50 revisions in the archive. 13 November 2020, 14:26:15 UTC
dc65326 Updated backport on buster-swh from debian/0.17.2-1_swh1 (unstable-swh) 13 November 2020, 11:13:32 UTC
back to top