0eb309e | Valentin Lorentz | 31 July 2020, 11:00:42 UTC | Rename argument 'object_type' of raw_extrinsic_metadata_get to 'type'. For consistency with RawExtrinsicMetadata. | 31 July 2020, 11:00:59 UTC |
b385c79 | Valentin Lorentz | 31 July 2020, 10:30:31 UTC | Make raw_extrinsic_metadata_get return PagedResult instead of Dict. | 31 July 2020, 10:56:34 UTC |
df943ec | Valentin Lorentz | 31 July 2020, 08:11:57 UTC | pg: Rewrite _origin_query to force the query planner to filter on URLs before filtering on visits. URL filters usually have a few matches and use the index; whereas filtering on visits requires to scan the entire origin table first. This makes the query considerably faster. Credit for the idea goes to @olasd. | 31 July 2020, 08:12:41 UTC |
0c5a8e2 | Valentin Lorentz | 30 July 2020, 17:39:41 UTC | Add support for metadata-related object types to the backfiller and replayer. Existing tests automatically test them, using data from swh.journal.tests. | 30 July 2020, 17:39:41 UTC |
24bc51d | Valentin Lorentz | 30 July 2020, 17:33:14 UTC | test_replay: update for swh.journal 0.4.1. DUPLICATE_CONTENTS now contains BaseModel objects. | 30 July 2020, 17:36:49 UTC |
cf9f44e | Antoine R. Dumont (@ardumont) | 30 July 2020, 14:07:53 UTC | storage*: Type origin_search(...) -> PagedResult[Origin] Related to T645 | 30 July 2020, 14:24:55 UTC |
4d52fc1 | Antoine R. Dumont (@ardumont) | 30 July 2020, 12:32:20 UTC | storage*: Adapt origin_list(...) -> PagedResult[Origin] Related to T645 | 30 July 2020, 12:32:20 UTC |
7beba93 | Antoine R. Dumont (@ardumont) | 30 July 2020, 07:58:40 UTC | algos.snapshot: Open snapshot_id_get_from_revision This leverages the latest change in origin_visit_get and origin_visit_status_get to iterate over visit and visit status to detect a snapshot targetting a revision. This algo got used recently in the deposit. It may serve again. Related to T645 | 30 July 2020, 12:10:12 UTC |
b81f928 | Antoine R. Dumont (@ardumont) | 29 July 2020, 14:33:31 UTC | storage*: add origin_visit_status_get(...) -> PagedResult[OriginVisitStatus] Related to T645 | 30 July 2020, 12:05:39 UTC |
8cf6efa | Valentin Lorentz | 29 July 2020, 13:13:04 UTC | Add type annotations on get_storage. The main goal of this commit is to add '-> StorageInterface', so all storage users automatically get type-checked. But mypy was whining about the multiple imports, so this commit also deduplicates the import statements as a side-effect. | 30 July 2020, 11:53:38 UTC |
e63b78c | Valentin Lorentz | 30 July 2020, 09:30:28 UTC | buffer: Pass lists to backend functions, not iterables. Unbreaks swh-loader-core's tests, which are all red when run with the current swh-storage version. Unfortunately, swh-storage's tests did not catch it, because the in-memory backend works fine with iterables, but the pg backend doesn't. | 30 July 2020, 09:30:28 UTC |
7667f7e | Valentin Lorentz | 29 July 2020, 14:59:59 UTC | test_origin_visit_get_all: Use tokens as opaque values. | 29 July 2020, 15:23:33 UTC |
708522d | Valentin Lorentz | 29 July 2020, 14:52:10 UTC | test_origin_visit_get_all: Make assertions consistent. | 29 July 2020, 15:23:24 UTC |
2bfd2f7 | Antoine R. Dumont (@ardumont) | 29 July 2020, 14:55:16 UTC | storage*: Simplify next-page-token computation Related to T645 | 29 July 2020, 15:12:00 UTC |
874646e | Valentin Lorentz | 29 July 2020, 14:51:51 UTC | filter: Fix types passed to the proxied storage. | 29 July 2020, 14:51:51 UTC |
24559bb | Valentin Lorentz | 29 July 2020, 08:30:45 UTC | Fix upcoming type warning with swh.core > v0.1.2. origin_visit_status_get_latest expects an int, not Optional[int]; but this error wasn't detected so far because mypy couldn't detect the type of the function. The next release of swh.core fixes that issue, so this fix is now needed to not trigger a mypy error. | 29 July 2020, 14:24:11 UTC |
b6202a0 | Valentin Lorentz | 29 July 2020, 08:26:32 UTC | Make API endpoints take Lists instead of Iterables as arguments 1. clients crash when they call the API with an iterator 2. some backend implementations violate the contract by assuming the argument is a sequence (eg. by iterating twice on it, and assuming the elements are the same) 3. no matter what we do, the endpoints will always get a list as argument in practice, because they go through the RPC framework 4. there is no concrete (ha!) advantage in taking an Iterable rather than a List. | 29 July 2020, 14:24:11 UTC |
21b7730 | Antoine R. Dumont (@ardumont) | 27 July 2020, 20:17:59 UTC | storage*: use an enum to explicit the order in origin_visit_get Related to T645 | 29 July 2020, 12:43:36 UTC |
643ebc6 | Antoine R. Dumont (@ardumont) | 27 July 2020, 20:17:59 UTC | storage*: origin_visit_get(...) -> PagedResult[OriginVisit] Related to T645 | 29 July 2020, 12:43:36 UTC |
f543bd5 | Valentin Lorentz | 28 July 2020, 13:59:35 UTC | Write metadata + metadata authorities/fetchers to the journal. | 28 July 2020, 13:59:35 UTC |
77960ca | Valentin Lorentz | 28 July 2020, 07:40:57 UTC | in_memory: fix tie-breaking when two visits have the same date. swh-loader-core's tests depend on this behavior. | 28 July 2020, 07:54:55 UTC |
119d01e | Antoine R. Dumont (@ardumont) | 27 July 2020, 12:18:32 UTC | storage*: origin_visit_get_by -> Optional[OriginVisit] Related to T645 | 27 July 2020, 12:44:16 UTC |
2d51be9 | Valentin Lorentz | 27 July 2020, 11:23:18 UTC | Rename object_metadata to raw_extrinsic_metadata. For consistency with the name in swh-model. | 27 July 2020, 12:18:49 UTC |
57e305e | Antoine R. Dumont (@ardumont) | 27 July 2020, 10:44:37 UTC | storage*: origin_visit_find_by_date -> Optional[OriginVisit] Related to T645 | 27 July 2020, 10:46:35 UTC |
b31c304 | Antoine R. Dumont (@ardumont) | 26 July 2020, 08:22:40 UTC | algos.origin: Simplify origin_get_latest_visit_status function | 27 July 2020, 09:56:33 UTC |
5344a6f | Antoine R. Dumont (@ardumont) | 26 July 2020, 06:05:10 UTC | storage*: type origin_visit_get_latest endpoint result The endpoint returns an optional OriginVisit object instead of a dict: ``` def origin_visit_get_latest(...) -> Optional[OriginVisit] ``` It also fixes the in-memory storage implementation which filtered data too early. It only filtered on the latest origin visit status associated to the origin visit. So depending on filters, this could have been wrong. It was not much of a problem as there is no longer any direct clients of this api (they are using [1] now). [1] swh.storage.algos.origin.origin_get_latest_origin_visit_status function Related to T645 | 27 July 2020, 09:56:33 UTC |
789972f | Valentin Lorentz | 24 July 2020, 07:28:33 UTC | metadata_{authority,fetcher}_add: Fix crash when the iterable argument is empty. | 24 July 2020, 07:28:33 UTC |
7e94767 | Antoine R. Dumont (@ardumont) | 23 July 2020, 15:47:04 UTC | storage*: origin_get(Iterable[str]) -> Iterable[Optional[Origin]] This: - drops the legacy behavior (no more input as list of dicts or even one dict). - aligns with other _get endpoints (only 1 iterable of identifiers as input, here the origin urls). - migrates towards returning an iterable of optional origin model objects (again the optional part is alignment with existing get endpoint) Related to T645 | 23 July 2020, 16:16:38 UTC |
d8583eb | Antoine R. Dumont (@ardumont) | 22 July 2020, 12:22:45 UTC | storage*.origin_visit_get_random: Read model objects Related to T645 | 23 July 2020, 12:10:07 UTC |
b2055f4 | Antoine R. Dumont (@ardumont) | 23 July 2020, 07:16:19 UTC | pgstorage: Drop unnecessary indirection from reading origin_visit It's a missing left-over from the migration to making the origin-visit immutable. | 23 July 2020, 08:28:09 UTC |
ccbd2e9 | Antoine R. Dumont (@ardumont) | 22 July 2020, 06:46:10 UTC | pytest_plugin: Make sample_data an object Note that this: - drops the no longer needed copy done by the StorageData instance (used by sample_data) since now it returned immutable BaseModel objects. - centralizes some left-over tests to use sample_data as well | 22 July 2020, 10:41:08 UTC |
67a909e | Antoine R. Dumont (@ardumont) | 21 July 2020, 16:15:41 UTC | pytest_plugin: Rename sample_data_model to sample_data Related to T2494 | 22 July 2020, 09:37:40 UTC |
e005900 | Antoine R. Dumont (@ardumont) | 21 July 2020, 16:07:44 UTC | pytest_plugin: Drop sample_data in favor of sample_data_model Related to T2494 | 22 July 2020, 09:30:33 UTC |
bbe840e | Antoine R. Dumont (@ardumont) | 21 July 2020, 15:43:22 UTC | storage_data: Expose snapshots as model objects Related to T2494 | 21 July 2020, 15:43:35 UTC |
d0cf317 | Antoine R. Dumont (@ardumont) | 21 July 2020, 15:32:03 UTC | storage_data: Expose release as model objects Related to T2494 | 21 July 2020, 15:32:03 UTC |
3be5327 | Antoine R. Dumont (@ardumont) | 21 July 2020, 15:21:20 UTC | storage_data: Expose origin_visits as model objects Related to T2494 | 21 July 2020, 15:21:20 UTC |
bcc0aee | Antoine R. Dumont (@ardumont) | 21 July 2020, 14:52:22 UTC | storage_data: Expose origins as model objects Related to T2494 | 21 July 2020, 15:03:26 UTC |
d4cd33c | Antoine R. Dumont (@ardumont) | 21 July 2020, 14:24:16 UTC | storage_data: Expose revisions as model objects Related to T2494 | 21 July 2020, 14:24:16 UTC |
955b6e2 | Antoine R. Dumont (@ardumont) | 21 July 2020, 13:35:23 UTC | storage_data: Expose directories as directory model objects Related to T2494 | 21 July 2020, 13:53:49 UTC |
95dbdf7 | Antoine R. Dumont (@ardumont) | 21 July 2020, 13:04:04 UTC | storage_data: Remove unused fixture data Less to maintain Related to T2494 | 21 July 2020, 13:53:49 UTC |
98a87fe | Antoine R. Dumont (@ardumont) | 21 July 2020, 12:59:09 UTC | storage_data: Expose contents as content model object Related to T2494 | 21 July 2020, 13:53:49 UTC |
a23b748 | Antoine R. Dumont (@ardumont) | 21 July 2020, 12:43:21 UTC | pytest_plugin: Drop unnecessary back and forth conversion This is preparatory work to incrementally migrate the sample_data fixture to use model objects directly. Related to T2494 | 21 July 2020, 13:53:49 UTC |
6338ad2 | Antoine R. Dumont (@ardumont) | 21 July 2020, 07:42:14 UTC | Drop validate proxy The validate proxy was initially an helper to ease the transition from the use of dicts towards model objects in "*_add" production endpoints. It was not removed immediately and grew some behavior it should not have (notably revision conversion so the comparison within those related tests work). After finally migrated away from dicts within the tests, we can now drop it [1]. Note that this moves the extra revision conversion behavior from the validate proxy to those related tests. This extra step will also disappear when we finally move the "*_get" endpoints to return model objects as well. Note: - This drops fixture redefinitions in the process (introduced so we could have that validate proxy at the time). - Remove the "validate" keyword from the get_storage function (so no longer possible to instantiate one [2]) [1] T2994 [2] which, practically, is the case today, nothing runs on production with it. Related to T2499 | 21 July 2020, 11:25:24 UTC |
e0152b0 | Antoine R. Dumont (@ardumont) | 21 July 2020, 10:34:43 UTC | 157: Fix migration script a posteriori Data has been fixed in production | 21 July 2020, 10:35:40 UTC |
96b2636 | Antoine R. Dumont (@ardumont) | 21 July 2020, 08:12:37 UTC | tests: Convert left-over dicts to model objects Related to T2494 | 21 July 2020, 08:41:22 UTC |
42ae56d | Antoine R. Dumont (@ardumont) | 20 July 2020, 20:23:09 UTC | test_storage: Migrate last storage to use model objects Related to T2494 | 20 July 2020, 20:23:09 UTC |
d4f896e | Antoine R. Dumont (@ardumont) | 20 July 2020, 20:21:56 UTC | test_storage: test_origin: Use data model object Related to T2494 | 20 July 2020, 20:21:56 UTC |
6bdfd85 | Antoine R. Dumont (@ardumont) | 20 July 2020, 19:44:09 UTC | test_storage: origin_metadata: Centralize objects within sample_data_model Related to T2494 | 20 July 2020, 19:44:09 UTC |
a6f70c3 | Antoine R. Dumont (@ardumont) | 20 July 2020, 19:12:55 UTC | test_storage: content_metadata: Centralize objects within sample_data_model Related to T2494 | 20 July 2020, 19:43:25 UTC |
c9e921e | Antoine R. Dumont (@ardumont) | 20 July 2020, 15:59:00 UTC | test_storage: test_object_find_by_sha1: Use data model object Related to T2494 | 20 July 2020, 15:59:00 UTC |
caa7f79 | Antoine R. Dumont (@ardumont) | 20 July 2020, 15:39:02 UTC | test_storage: content_find: Use data model object Related to T2494 | 20 July 2020, 15:55:35 UTC |
6453504 | Antoine R. Dumont (@ardumont) | 20 July 2020, 14:35:46 UTC | test_storage: stat_counters: Use data model object Related to T2494 | 20 July 2020, 15:55:35 UTC |
cf80d3c | Antoine R. Dumont (@ardumont) | 20 July 2020, 13:58:35 UTC | test_storage: snapshot: Use data model object Related to T2494 | 20 July 2020, 14:22:35 UTC |
4f10171 | Antoine R. Dumont (@ardumont) | 20 July 2020, 13:36:05 UTC | test_storage: origin_visit/origin_visit_status: Use data model object Related to T2494 | 20 July 2020, 13:36:05 UTC |
1c1bef9 | Antoine R. Dumont (@ardumont) | 20 July 2020, 12:52:37 UTC | test_storage: revision/release: Drop no longer needed conditionals Related to T2494 | 20 July 2020, 12:52:37 UTC |
cdf6f58 | Antoine R. Dumont (@ardumont) | 20 July 2020, 12:34:24 UTC | test_storage: origin: Use data model object Related to T2494 | 20 July 2020, 12:51:22 UTC |
bbdd7ed | Antoine R. Dumont (@ardumont) | 20 July 2020, 11:42:45 UTC | tests: Drop deprecated storage.origin_add_one use This is no longer used anywhere. Related to T2494 | 20 July 2020, 11:44:08 UTC |
4971c25 | Antoine R. Dumont (@ardumont) | 20 July 2020, 11:20:59 UTC | test_storage: release: Use data model object Related to T2494 | 20 July 2020, 11:20:59 UTC |
a17c412 | Antoine R. Dumont (@ardumont) | 20 July 2020, 10:55:46 UTC | test_storage: revision: Use data model object Related to T2494 | 20 July 2020, 11:03:18 UTC |
03c6e15 | Valentin Lorentz | 20 July 2020, 09:31:30 UTC | Rename 'deposit' authority type to 'deposit_client'. It makes more sense semantically; as the client is the authority not the deposit server. | 20 July 2020, 11:00:22 UTC |
87b1070 | Antoine R. Dumont (@ardumont) | 20 July 2020, 09:58:34 UTC | test_storage: directory: Use data model object Related to T2494 | 20 July 2020, 09:58:34 UTC |
1a2b85f | Antoine R. Dumont (@ardumont) | 20 July 2020, 09:09:08 UTC | test_storage: Make swh_contents fixture generate content model objects Note that this will be improved upon after the storage migration. We can take a look at the swh.model.tests.generate_testdata.gen_contents and make it generate BaseContent objects directly if it's possible. This was not done here so the impacts is limited to storage. Related to T2494 | 20 July 2020, 09:09:08 UTC |
99a28ad | Antoine R. Dumont (@ardumont) | 20 July 2020, 08:41:02 UTC | tests.generate_data_test: Remove dead code The storage fixtures now uses the swh.model.tests.generate_data_test instead Related to T2494 | 20 July 2020, 09:01:26 UTC |
03e17c3 | Antoine R. Dumont (@ardumont) | 20 July 2020, 08:12:13 UTC | test_storage: skipped_content/content_missing: Use data model object Related to T2494 | 20 July 2020, 09:01:26 UTC |
7131dcb | Valentin Lorentz | 07 July 2020, 11:07:33 UTC | Make metadata-related endpoints consistent with other endpoints by using Iterables of swh-model objects instead of a dict. | 20 July 2020, 08:48:35 UTC |
997ec1d | Antoine R. Dumont (@ardumont) | 17 July 2020, 15:51:08 UTC | test.storage: content_add_metadata: Use data model object Related to T2494 | 17 July 2020, 15:51:08 UTC |
8c2d635 | Antoine R. Dumont (@ardumont) | 17 July 2020, 15:00:29 UTC | test.storage: content_add: Use data model object Related to T2494 | 17 July 2020, 15:16:20 UTC |
2b239f0 | Antoine R. Dumont (@ardumont) | 17 July 2020, 14:37:31 UTC | test_cassandra: Use data model object Related to T2494 | 17 July 2020, 14:37:31 UTC |
eb2bf8c | Antoine R. Dumont (@ardumont) | 17 July 2020, 12:50:34 UTC | test_db: Drop redundant test This is already tested through the test_storage scenario | 17 July 2020, 12:50:34 UTC |
04d25df | Antoine R. Dumont (@ardumont) | 16 July 2020, 15:31:28 UTC | test_cli: Use snapshot model object within test That commit is not so interesting. But at least we validate the snapshot is correct prior to sending it. Also that removes a bit duplicated storage configuration. Related to T2494 | 16 July 2020, 15:52:50 UTC |
2d4f727 | Antoine R. Dumont (@ardumont) | 16 July 2020, 13:33:36 UTC | algos.test_origin: Use data model object and drop validate proxy use Related to T2494 | 16 July 2020, 15:16:36 UTC |
97a0721 | Antoine R. Dumont (@ardumont) | 16 July 2020, 13:17:33 UTC | algos.test_snapshot: Use model objects from sample_data_model This opens up origin_visits and add new snapshots to the fixture. So we can reuse those. Related to T2494 | 16 July 2020, 13:34:23 UTC |
b6971b5 | Antoine R. Dumont (@ardumont) | 16 July 2020, 13:06:55 UTC | pytest_plugin: Ensure fixture instantiates correctly Related to T2484 Should fix [1] [1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/154/console | 16 July 2020, 13:13:05 UTC |
3abf6b3 | Antoine R. Dumont (@ardumont) | 16 July 2020, 10:30:55 UTC | pytest_plugin: Do not expose the validate proxy storage Only the storage module needs it. This should fix the debian jenkins build [1] [1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/153/console | 16 July 2020, 12:00:46 UTC |
a688e82 | Antoine R. Dumont (@ardumont) | 16 July 2020, 08:26:03 UTC | test_revision_bw_compat: Use revision model object Related to T2494 | 16 July 2020, 10:08:33 UTC |
21efe2a | Antoine R. Dumont (@ardumont) | 15 July 2020, 18:05:46 UTC | test_filter: Use model objects in tests and drop validate proxy | 16 July 2020, 09:28:21 UTC |
2ff4c6f | Antoine R. Dumont (@ardumont) | 15 July 2020, 17:29:45 UTC | test_buffer: Use model objects in tests and drop validate proxy | 16 July 2020, 09:28:21 UTC |
df45641 | Antoine R. Dumont (@ardumont) | 15 July 2020, 17:07:32 UTC | test_retry: Drop validate proxy when we can When we use the sample_data_model (almost all object types except the metadata ones), we can use a storage with no validate proxy. Depends on D3510 | 16 July 2020, 09:28:21 UTC |
14b1648 | Antoine R. Dumont (@ardumont) | 15 July 2020, 15:40:04 UTC | test_retry: Use sample_data_model fixture to manipulate model objects | 16 July 2020, 09:28:20 UTC |
df3f46d | Antoine R. Dumont (@ardumont) | 15 July 2020, 15:03:33 UTC | pytest-plugin: Expose a sample_data_model fixture This is almost the same fixture as sample_data except: - it's BaseModel object instance within - not complete as we cannot convert yet the metadata objects (there is a diff pending which will allow it but right now we cannot). The next commits will use this fixture to allow the switch from dict to model objects. | 16 July 2020, 09:28:20 UTC |
8bc7944 | Antoine R. Dumont (@ardumont) | 16 July 2020, 07:33:00 UTC | pytest_plugin: Avoid fixture client to declare optional dependency Prior to this commit, this would make swh_storage_backend_config fixture clients need to declare an optional dependency on swh.journal. Otherwise, it would not work [1]. This commit fixes it by dropping this configuration in the main pytest plugin. It keeps the storage tests testing with that journal_writer collaborator though by declaring an override which still provides it. This fixes the debian build [1] [1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/152/console | 16 July 2020, 07:33:00 UTC |
f5811da | Antoine R. Dumont (@ardumont) | 12 July 2020, 13:57:47 UTC | Allow cassandra binary path to be configured through env variable The current hard-coded value won't work for other distributions not relying on standard conventions (e.g. nixos...). This keeps the original behavior and only allow to diverge based on the environment variable SWH_CASSANDRA_BIN. This also: - fixes an issue on log path inexistence which raises. - renames the other env variable LOG_CASSANDRA to SWH_CASSANDRA_LOG (for consistency) | 15 July 2020, 10:09:55 UTC |
1a8924b | Antoine R. Dumont (@ardumont) | 11 July 2020, 06:42:17 UTC | 158: Make schema and migration converge so the migration works In the end, the order of the revision entry matters whether we select * or not. So the select must match the order defined in the revision_entry type type. Otherwise, a mismatch type error occurs [1] [1] psql:sql/upgrades/158.sql:74: ERROR: return type mismatch in function declared to return revision_entry | 11 July 2020, 06:52:26 UTC |
9219a23 | Antoine Lambert | 10 July 2020, 14:02:30 UTC | in_memory: Fix snapshot_get_branches regression with target_types When providing target_types parameter, snapshot branches must be sorted when iterating otherwise wrong branches can be returned. | 10 July 2020, 14:15:54 UTC |
23318c2 | Antoine R. Dumont (@ardumont) | 09 July 2020, 08:11:01 UTC | setup: Do no expose the pytest-plugin any longer Defining the pytest-plugin though the pytest-plugin [1] makes it loaded by default. This creates loading issues on modules depending on storage but not on the pytest plugin storage exposes. It was explained in the doc and I did not realize [2] Instead we'll explicitely define to modules depending on the pytest plugins in their root conftest [3]: ``` pytest_plugins = [ "swh.storage.pytest_plugin" ] ``` [1] https://docs.pytest.org/en/stable/writing_plugins.html#setuptools-entry-points [2] https://docs.pytest.org/en/stable/writing_plugins.html#plugin-discovery-order-at-tool-startup [3] https://docs.pytest.org/en/stable/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file Related to T2484 | 10 July 2020, 06:19:39 UTC |
124e76d | Nicolas Dandrimont | 09 July 2020, 17:38:29 UTC | Rework dia -> pdf pipeline for inkscape 1.0 - Use dia directly to convert from .dia to .svg (inkscape would use dia via a plugin anyway) - Add proper runes to detect inkscape >= 1 and use the export options for that. | 09 July 2020, 17:38:29 UTC |
de38cd1 | Valentin Lorentz | 09 July 2020, 16:02:10 UTC | Remove overhead of to_dict/from_dict in test_snapshot_large. This should make it fast enough not to exceed the deadline. | 09 July 2020, 16:02:10 UTC |
e415488 | Valentin Lorentz | 09 July 2020, 15:58:12 UTC | in_memory: Fix quadratic run time in snapshot_get_branches. snapshot.branches is now an ImmutableDict, which is backed by a tuple of tuples; so random accesses now take a linear time instead of a constant time. This commit replaces random accesses with a single scan of all the items, and does existence checks in a set instead. | 09 July 2020, 15:59:48 UTC |
c3803ef | David Douard | 09 July 2020, 09:54:07 UTC | Fix a typo I introduced in previous revision dict(x if x is not None else None) != dict(x) if x is not None else None... | 09 July 2020, 09:56:35 UTC |
8bf3794 | Valentin Lorentz | 09 July 2020, 08:27:12 UTC | Convert ImmutableDict to dict before passing it to json.dumps. To work with the new swh-model version, which uses ImmutableDict in model objects. | 09 July 2020, 09:31:50 UTC |
c21d0e3 | Antoine R. Dumont (@ardumont) | 07 July 2020, 09:09:25 UTC | Move sharable fixtures out of conftest into a dedicated pytest plugin This will allow loaders to reuse those dedicated fixtures within their code base without having to import the swh.storage.tests.conftest module. Related to T2484 | 08 July 2020, 09:50:21 UTC |
e45ca76 | Antoine R. Dumont (@ardumont) | 07 July 2020, 10:07:18 UTC | Migrate from vcversioner to setuptools-scm Related to T2105 | 07 July 2020, 15:42:30 UTC |
5ab7023 | David Douard | 03 July 2020, 09:45:06 UTC | Extract revision's extra_header as a top level attribute Follows swh.model's evolution for the Revision model class. In Postgresql, store the extra_headers as a bytea[][]. Ensure data present in postgres with extra_headers in the metadata field are properly supported by the pg-backed storage. Get rid of the (now useless) git_headers_to_db() converter function. In Cassandra, store them as frozen<list<list<blob>>>. | 07 July 2020, 14:49:14 UTC |
8010848 | Antoine R. Dumont (@ardumont) | 06 July 2020, 07:45:40 UTC | storage: Send metrics from the origin_add endpoint Prior to this commit, since the loaders got migrated to use the main endpoint, no metrics were sent for the origin any longer. This commit fixes it. It also drops the send_metrics call from the deprecated endpoint origin_add_one (which, as an implementation details calls the other one). | 06 July 2020, 07:45:40 UTC |
95fd660 | Antoine R. Dumont (@ardumont) | 03 July 2020, 15:54:04 UTC | pg-storage: Add missing cur parameter passing Although, this also pulled a refactoring on the insertion query as the default naive approach ended up with issues on cur already being closed [1] [1] Related to P715 Related to D3416 | 03 July 2020, 15:54:04 UTC |
348bc7b | Antoine R. Dumont (@ardumont) | 03 July 2020, 14:31:22 UTC | storage.db: Drop db.origin_visit_upsert behavior The initial desired behavior was to allow creation of origin-visit if they already had their id set. This is the what's needed for the replayer to actually work. But somehow, this left the possibility to update the origin-visit... This commit fixes it by dropping conflictual origin-visits if any. In effect, we can no longer overwrite origin-visits (pg-storage wise). Related to T2310 | 03 July 2020, 14:42:20 UTC |
248c277 | Valentin Lorentz | 30 June 2020, 14:35:59 UTC | Move tests of content_metadata_* next to origin_metadata_* For consistency with the main code. | 02 July 2020, 09:04:43 UTC |
f2619b6 | Antoine R. Dumont (@ardumont) | 01 July 2020, 13:39:59 UTC | Rework 157 migration to ease replication setup Past experience showed that altering tables is more stressful than plain creation. As in here. Related to T2306 Related P707 | 01 July 2020, 13:41:16 UTC |
312127a | Antoine R. Dumont (@ardumont) | 30 June 2020, 13:22:59 UTC | storage*: Drop intermediary conversion step into OriginVisit This is no longer possible as OriginVisit no longer hold the same information as OriginVisitStatus. This will allow to drop entirely those fields in the model. Related to T2310 | 30 June 2020, 13:54:01 UTC |
953bd29 | Valentin Lorentz | 30 June 2020, 13:11:49 UTC | pg: use 'on conflict do nothing' strategy for duplicate metadata rows. "updates are a problem for postgresql logical replication" | 30 June 2020, 13:25:53 UTC |