https://github.com/SoftwareHeritage/swh-storage

sort by:
Revision Author Date Message Commit Date
1b96ace Updated debian changelog for version 0.11.10 04 August 2020, 12:15:22 UTC
9d11a4e Update upstream source from tag 'debian/upstream/0.11.10' Update to upstream version '0.11.10' with Debian dir 0682ec4aedae9d395e591d7baaf54449a576de9b 04 August 2020, 12:15:20 UTC
6ca837c New upstream version 0.11.10 04 August 2020, 12:15:16 UTC
a5232b7 tests: Improve coverage on directory_ls endpoints This fixes the current directory listing tests coverage to check down to the contents. This also fixes one inconsistent test data and the tests impacted by this change. 04 August 2020, 11:34:48 UTC
15e8c99 storage*: Type content_find(...) -> List[Content] Related to T645 04 August 2020, 09:22:45 UTC
3c2e5a3 storage*: Type {cnt,dir,rev,rel,snp}_get_random(...) -> Sha1Git Related to T645 03 August 2020, 14:23:44 UTC
44807d8 Updated debian changelog for version 0.11.9 03 August 2020, 10:02:56 UTC
37689c9 Update upstream source from tag 'debian/upstream/0.11.9' Update to upstream version '0.11.9' with Debian dir 01423f66eb2af59524c287338dc4235b9f95cf3c 03 August 2020, 10:02:55 UTC
e420486 New upstream version 0.11.9 03 August 2020, 10:02:51 UTC
aa58e10 storage*: Drop origin-get-range in favor of origin-list Related to T645 03 August 2020, 09:39:58 UTC
87c5ba2 storage*: Do not allow unknown visit status in origin_visit*_get_latest That makes some storage (pg-storage) fail without that filtering. 01 August 2020, 07:07:12 UTC
9c18381 Updated debian changelog for version 0.11.8 31 July 2020, 13:17:40 UTC
de9a56a Update upstream source from tag 'debian/upstream/0.11.8' Update to upstream version '0.11.8' with Debian dir 04155cd323a18583d6dae27f6620810e55c1e0d4 31 July 2020, 13:17:38 UTC
cce71b8 New upstream version 0.11.8 31 July 2020, 13:17:34 UTC
92f1183 storage*: Add type annotation to origin_count Related to T645 31 July 2020, 12:51:14 UTC
3466e48 Reuse swh.core stream_results function Related to T645 31 July 2020, 12:15:51 UTC
0eb309e Rename argument 'object_type' of raw_extrinsic_metadata_get to 'type'. For consistency with RawExtrinsicMetadata. 31 July 2020, 11:00:59 UTC
b385c79 Make raw_extrinsic_metadata_get return PagedResult instead of Dict. 31 July 2020, 10:56:34 UTC
df943ec pg: Rewrite _origin_query to force the query planner to filter on URLs before filtering on visits. URL filters usually have a few matches and use the index; whereas filtering on visits requires to scan the entire origin table first. This makes the query considerably faster. Credit for the idea goes to @olasd. 31 July 2020, 08:12:41 UTC
0c5a8e2 Add support for metadata-related object types to the backfiller and replayer. Existing tests automatically test them, using data from swh.journal.tests. 30 July 2020, 17:39:41 UTC
24bc51d test_replay: update for swh.journal 0.4.1. DUPLICATE_CONTENTS now contains BaseModel objects. 30 July 2020, 17:36:49 UTC
221e40c Updated debian changelog for version 0.11.6 30 July 2020, 14:29:11 UTC
c8d6673 Update upstream source from tag 'debian/upstream/0.11.6' Update to upstream version '0.11.6' with Debian dir 71a5fa5147e4c905fe8adc91fbd3ee792f62aae1 30 July 2020, 14:29:09 UTC
8608172 New upstream version 0.11.6 30 July 2020, 14:29:06 UTC
cf9f44e storage*: Type origin_search(...) -> PagedResult[Origin] Related to T645 30 July 2020, 14:24:55 UTC
4d52fc1 storage*: Adapt origin_list(...) -> PagedResult[Origin] Related to T645 30 July 2020, 12:32:20 UTC
7beba93 algos.snapshot: Open snapshot_id_get_from_revision This leverages the latest change in origin_visit_get and origin_visit_status_get to iterate over visit and visit status to detect a snapshot targetting a revision. This algo got used recently in the deposit. It may serve again. Related to T645 30 July 2020, 12:10:12 UTC
b81f928 storage*: add origin_visit_status_get(...) -> PagedResult[OriginVisitStatus] Related to T645 30 July 2020, 12:05:39 UTC
8cf6efa Add type annotations on get_storage. The main goal of this commit is to add '-> StorageInterface', so all storage users automatically get type-checked. But mypy was whining about the multiple imports, so this commit also deduplicates the import statements as a side-effect. 30 July 2020, 11:53:38 UTC
e63b78c buffer: Pass lists to backend functions, not iterables. Unbreaks swh-loader-core's tests, which are all red when run with the current swh-storage version. Unfortunately, swh-storage's tests did not catch it, because the in-memory backend works fine with iterables, but the pg backend doesn't. 30 July 2020, 09:30:28 UTC
7667f7e test_origin_visit_get_all: Use tokens as opaque values. 29 July 2020, 15:23:33 UTC
708522d test_origin_visit_get_all: Make assertions consistent. 29 July 2020, 15:23:24 UTC
2bfd2f7 storage*: Simplify next-page-token computation Related to T645 29 July 2020, 15:12:00 UTC
874646e filter: Fix types passed to the proxied storage. 29 July 2020, 14:51:51 UTC
24559bb Fix upcoming type warning with swh.core > v0.1.2. origin_visit_status_get_latest expects an int, not Optional[int]; but this error wasn't detected so far because mypy couldn't detect the type of the function. The next release of swh.core fixes that issue, so this fix is now needed to not trigger a mypy error. 29 July 2020, 14:24:11 UTC
b6202a0 Make API endpoints take Lists instead of Iterables as arguments 1. clients crash when they call the API with an iterator 2. some backend implementations violate the contract by assuming the argument is a sequence (eg. by iterating twice on it, and assuming the elements are the same) 3. no matter what we do, the endpoints will always get a list as argument in practice, because they go through the RPC framework 4. there is no concrete (ha!) advantage in taking an Iterable rather than a List. 29 July 2020, 14:24:11 UTC
21b7730 storage*: use an enum to explicit the order in origin_visit_get Related to T645 29 July 2020, 12:43:36 UTC
643ebc6 storage*: origin_visit_get(...) -> PagedResult[OriginVisit] Related to T645 29 July 2020, 12:43:36 UTC
f543bd5 Write metadata + metadata authorities/fetchers to the journal. 28 July 2020, 13:59:35 UTC
9167dc7 Updated debian changelog for version 0.11.5 28 July 2020, 08:10:21 UTC
d48adf4 Update upstream source from tag 'debian/upstream/0.11.5' Update to upstream version '0.11.5' with Debian dir a25d13a30234e582d4f2f5c2700eab3db2c2a492 28 July 2020, 08:10:20 UTC
b4cd42d New upstream version 0.11.5 28 July 2020, 08:10:17 UTC
77960ca in_memory: fix tie-breaking when two visits have the same date. swh-loader-core's tests depend on this behavior. 28 July 2020, 07:54:55 UTC
7543dc8 Updated debian changelog for version 0.11.4 27 July 2020, 14:16:18 UTC
548f018 Update upstream source from tag 'debian/upstream/0.11.4' Update to upstream version '0.11.4' with Debian dir a95acf15bcf503742e7f5ce212010932a5cc49a5 27 July 2020, 14:16:17 UTC
b64c58d New upstream version 0.11.4 27 July 2020, 14:16:13 UTC
119d01e storage*: origin_visit_get_by -> Optional[OriginVisit] Related to T645 27 July 2020, 12:44:16 UTC
2d51be9 Rename object_metadata to raw_extrinsic_metadata. For consistency with the name in swh-model. 27 July 2020, 12:18:49 UTC
57e305e storage*: origin_visit_find_by_date -> Optional[OriginVisit] Related to T645 27 July 2020, 10:46:35 UTC
b31c304 algos.origin: Simplify origin_get_latest_visit_status function 27 July 2020, 09:56:33 UTC
5344a6f storage*: type origin_visit_get_latest endpoint result The endpoint returns an optional OriginVisit object instead of a dict: ``` def origin_visit_get_latest(...) -> Optional[OriginVisit] ``` It also fixes the in-memory storage implementation which filtered data too early. It only filtered on the latest origin visit status associated to the origin visit. So depending on filters, this could have been wrong. It was not much of a problem as there is no longer any direct clients of this api (they are using [1] now). [1] swh.storage.algos.origin.origin_get_latest_origin_visit_status function Related to T645 27 July 2020, 09:56:33 UTC
6f0946d Updated debian changelog for version 0.11.3 27 July 2020, 06:08:56 UTC
2324094 Update upstream source from tag 'debian/upstream/0.11.3' Update to upstream version '0.11.3' with Debian dir b2477219c2018c32c320df9ddcf9f011f7185c12 27 July 2020, 06:08:54 UTC
65ea9e8 New upstream version 0.11.3 27 July 2020, 06:08:50 UTC
789972f metadata_{authority,fetcher}_add: Fix crash when the iterable argument is empty. 24 July 2020, 07:28:33 UTC
7e94767 storage*: origin_get(Iterable[str]) -> Iterable[Optional[Origin]] This: - drops the legacy behavior (no more input as list of dicts or even one dict). - aligns with other _get endpoints (only 1 iterable of identifiers as input, here the origin urls). - migrates towards returning an iterable of optional origin model objects (again the optional part is alignment with existing get endpoint) Related to T645 23 July 2020, 16:16:38 UTC
d8583eb storage*.origin_visit_get_random: Read model objects Related to T645 23 July 2020, 12:10:07 UTC
ccbcee9 Updated debian changelog for version 0.11.2 23 July 2020, 10:18:15 UTC
1bd6fa1 Update upstream source from tag 'debian/upstream/0.11.2' Update to upstream version '0.11.2' with Debian dir d959a8c4b061404f92da4b96189a773245484f35 23 July 2020, 10:18:14 UTC
8052056 New upstream version 0.11.2 23 July 2020, 10:18:11 UTC
b2055f4 pgstorage: Drop unnecessary indirection from reading origin_visit It's a missing left-over from the migration to making the origin-visit immutable. 23 July 2020, 08:28:09 UTC
ccbd2e9 pytest_plugin: Make sample_data an object Note that this: - drops the no longer needed copy done by the StorageData instance (used by sample_data) since now it returned immutable BaseModel objects. - centralizes some left-over tests to use sample_data as well 22 July 2020, 10:41:08 UTC
67a909e pytest_plugin: Rename sample_data_model to sample_data Related to T2494 22 July 2020, 09:37:40 UTC
e005900 pytest_plugin: Drop sample_data in favor of sample_data_model Related to T2494 22 July 2020, 09:30:33 UTC
bbe840e storage_data: Expose snapshots as model objects Related to T2494 21 July 2020, 15:43:35 UTC
d0cf317 storage_data: Expose release as model objects Related to T2494 21 July 2020, 15:32:03 UTC
3be5327 storage_data: Expose origin_visits as model objects Related to T2494 21 July 2020, 15:21:20 UTC
bcc0aee storage_data: Expose origins as model objects Related to T2494 21 July 2020, 15:03:26 UTC
d4cd33c storage_data: Expose revisions as model objects Related to T2494 21 July 2020, 14:24:16 UTC
955b6e2 storage_data: Expose directories as directory model objects Related to T2494 21 July 2020, 13:53:49 UTC
95dbdf7 storage_data: Remove unused fixture data Less to maintain Related to T2494 21 July 2020, 13:53:49 UTC
98a87fe storage_data: Expose contents as content model object Related to T2494 21 July 2020, 13:53:49 UTC
a23b748 pytest_plugin: Drop unnecessary back and forth conversion This is preparatory work to incrementally migrate the sample_data fixture to use model objects directly. Related to T2494 21 July 2020, 13:53:49 UTC
6338ad2 Drop validate proxy The validate proxy was initially an helper to ease the transition from the use of dicts towards model objects in "*_add" production endpoints. It was not removed immediately and grew some behavior it should not have (notably revision conversion so the comparison within those related tests work). After finally migrated away from dicts within the tests, we can now drop it [1]. Note that this moves the extra revision conversion behavior from the validate proxy to those related tests. This extra step will also disappear when we finally move the "*_get" endpoints to return model objects as well. Note: - This drops fixture redefinitions in the process (introduced so we could have that validate proxy at the time). - Remove the "validate" keyword from the get_storage function (so no longer possible to instantiate one [2]) [1] T2994 [2] which, practically, is the case today, nothing runs on production with it. Related to T2499 21 July 2020, 11:25:24 UTC
e0152b0 157: Fix migration script a posteriori Data has been fixed in production 21 July 2020, 10:35:40 UTC
96b2636 tests: Convert left-over dicts to model objects Related to T2494 21 July 2020, 08:41:22 UTC
42ae56d test_storage: Migrate last storage to use model objects Related to T2494 20 July 2020, 20:23:09 UTC
d4f896e test_storage: test_origin: Use data model object Related to T2494 20 July 2020, 20:21:56 UTC
6bdfd85 test_storage: origin_metadata: Centralize objects within sample_data_model Related to T2494 20 July 2020, 19:44:09 UTC
a6f70c3 test_storage: content_metadata: Centralize objects within sample_data_model Related to T2494 20 July 2020, 19:43:25 UTC
c9e921e test_storage: test_object_find_by_sha1: Use data model object Related to T2494 20 July 2020, 15:59:00 UTC
caa7f79 test_storage: content_find: Use data model object Related to T2494 20 July 2020, 15:55:35 UTC
6453504 test_storage: stat_counters: Use data model object Related to T2494 20 July 2020, 15:55:35 UTC
cf80d3c test_storage: snapshot: Use data model object Related to T2494 20 July 2020, 14:22:35 UTC
4f10171 test_storage: origin_visit/origin_visit_status: Use data model object Related to T2494 20 July 2020, 13:36:05 UTC
1c1bef9 test_storage: revision/release: Drop no longer needed conditionals Related to T2494 20 July 2020, 12:52:37 UTC
cdf6f58 test_storage: origin: Use data model object Related to T2494 20 July 2020, 12:51:22 UTC
bbdd7ed tests: Drop deprecated storage.origin_add_one use This is no longer used anywhere. Related to T2494 20 July 2020, 11:44:08 UTC
4971c25 test_storage: release: Use data model object Related to T2494 20 July 2020, 11:20:59 UTC
927b95d Updated debian changelog for version 0.11.1 20 July 2020, 11:14:39 UTC
4d5d3c8 Update upstream source from tag 'debian/upstream/0.11.1' Update to upstream version '0.11.1' with Debian dir 28028312f22d69d4ca9d0b7c2a432af895a0fbc9 20 July 2020, 11:14:38 UTC
f600de9 New upstream version 0.11.1 20 July 2020, 11:14:37 UTC
a17c412 test_storage: revision: Use data model object Related to T2494 20 July 2020, 11:03:18 UTC
03c6e15 Rename 'deposit' authority type to 'deposit_client'. It makes more sense semantically; as the client is the authority not the deposit server. 20 July 2020, 11:00:22 UTC
87b1070 test_storage: directory: Use data model object Related to T2494 20 July 2020, 09:58:34 UTC
4b86a12 Updated debian changelog for version 0.11.0 20 July 2020, 09:12:25 UTC
b616ec3 Update upstream source from tag 'debian/upstream/0.11.0' Update to upstream version '0.11.0' with Debian dir a87b48762949ad167037acd21d8ba78e4b06fca9 20 July 2020, 09:12:25 UTC
0a24452 New upstream version 0.11.0 20 July 2020, 09:12:23 UTC
1a2b85f test_storage: Make swh_contents fixture generate content model objects Note that this will be improved upon after the storage migration. We can take a look at the swh.model.tests.generate_testdata.gen_contents and make it generate BaseContent objects directly if it's possible. This was not done here so the impacts is limited to storage. Related to T2494 20 July 2020, 09:09:08 UTC
99a28ad tests.generate_data_test: Remove dead code The storage fixtures now uses the swh.model.tests.generate_data_test instead Related to T2494 20 July 2020, 09:01:26 UTC
back to top