swh:1:snp:eb70f1f85391e4b077c211bec36af0061c4bf937

sort by:
Revision Author Date Message Commit Date
57aaf30 New upstream version 0.8.0 29 June 2020, 07:43:58 UTC
10443b8 Iterate over paginated visits in batches to retrieve latest visit/snapshot This should stops the current timeouts on origin with a high number of visits. Related to T2310 26 June 2020, 15:38:22 UTC
182ee49 storage*: Open order parameter to origin-visit-get endpoint This allows clients to search from most recent to oldest visit when calling the endpoint with the "order" parameter set to "desc" (visit id desc). This keeps and explicits the existing sorting order as visit id "asc". Related to T2310 26 June 2020, 11:22:40 UTC
f75cd41 tests*: Drop obsolete origin visit fields Related to T2310 26 June 2020, 10:28:06 UTC
8620519 replayer: Drop obsolete fields from origin-visit Otherwise, we won't be able to replay them. Related T2310 26 June 2020, 07:50:38 UTC
b991e69 test_storage: Add missing tests on origin_visit_get method 25 June 2020, 12:47:11 UTC
89e9dae storage: Given origin-visit index a name to avoid future dev/prod divergence Related to D3342#inline-23217 25 June 2020, 12:37:39 UTC
12d729b Relax checks on journal writes regarding origin-visit* 25 June 2020, 12:35:38 UTC
c6e6f33 replayer: Fix isoformat datetime string for origin-visit We no longer write datetime as strings in the journal. Still, the current journal must have those old values within. Related to D3336 Related to D3345 25 June 2020, 09:19:55 UTC
e5e80ef storage*: Drop obsolete fields from origin_visit Related to T2310 25 June 2020, 08:35:18 UTC
621fc8d Deprecate the origin_add_one() endpoint This endpoint is not really useful since the origin_add() can be used instead. Using a single API endpoint would also make the API a bit more consistant (most other endpoints only provide a xxx_add endpoint) ; having a single endpoint per object_type make is enough and make the whole API simpler. 23 June 2020, 14:07:09 UTC
fb603e1 Make Storage.add_origin() return a sumary dict make it consistent with other add_xxx methods by making it return a summary dict `{"origin:add": int}`. 23 June 2020, 13:58:54 UTC
c1f0e27 New upstream version 0.7.0 22 June 2020, 14:09:31 UTC
2d497ff test_origin: Rename appropriately tests So one can trigger tests separately by name tagging. 22 June 2020, 12:39:32 UTC
e9f4554 algos: Improve origin visit get latest visit status algorithm Prior to this commit, this looked up only the latest visit information. This now looks up across multiple visits up (from most recent visit to the oldest) until one visit which match the criteria is elected. 22 June 2020, 12:39:32 UTC
041543d test_snapshot: Do not use origin_visit_add returned result This api will be realigned with other add endpoints. 22 June 2020, 09:33:15 UTC
32fded1 algos.snapshot: Fix edge case when snapshot is not resolved Fixes [1] [1] https://sentry.softwareheritage.org/share/issue/9848d9ea23d94d6ba8855bc7a7d7d297/ 22 June 2020, 09:19:38 UTC
53c4392 Ensure ids are correct in tests' storage_data Also add an "objects" dict to easily retrieve available objects from their object_type. 22 June 2020, 08:57:47 UTC
46ac997 Fix tests' storage_data revisions one of them was actually invalid (extra_header metadata being used in hash computation) 22 June 2020, 08:57:39 UTC
19354bc SQL: replace the hash(url) index by a unique btree(url) on the origin table This ensures unicity of url in the origin table. 22 June 2020, 08:09:23 UTC
9514a1d Make sure the pagination in swh_snapshot_get_by_id uses the proper indexes 19 June 2020, 15:14:59 UTC
f6936c8 New upstream version 0.6.0 19 June 2020, 12:45:30 UTC
1600907 Move deprecated endpoint snapshot_get_latest from api endpoint to algos This allows to avoid repeating the same pattern of retrieving the last snapshot for a given origin. Note that this also makes the new function return a Snapshot model object as well. Related to T2310 19 June 2020, 09:19:58 UTC
5480b7b algos.origin: Open origin-get-latest-visit-status function This will allow to avoid repeating the same pattern of retrieving the last visit status for a given origin. Related to T2310 18 June 2020, 11:40:16 UTC
c498901 storage*: Allow origin-visit-get-latest to filter on type 18 June 2020, 10:25:12 UTC
822d96b test_origin: Align storage initialization within tests This aligns consistently the storage initialization with other tests. 18 June 2020, 06:55:34 UTC
7406df2 New upstream version 0.5.0 17 June 2020, 14:19:35 UTC
c3d177b test_storage: Fix flakiness in round to milliseconds test util method Prior to this commit, the tests would fail [1] for no good reason [2]. This fixes it. [1] https://jenkins.softwareheritage.org/job/DSTO/job/tests/1264/console [2] microseconds would exceed a limit of 999999 from time to time 17 June 2020, 13:13:41 UTC
7319495 storage*: Add origin-visit-status-get-latest endpoint So we can read the latest origin-visit-status out of a storage Related to T2310 17 June 2020, 10:20:48 UTC
692bfa3 Fix/update the backfiller The backfiller has not been updated to match recent changes in several places. This has not been detected because there was no proper test of the backfiller function as a whole. This is now done. 17 June 2020, 09:35:35 UTC
057c6fd validate: accept model objects as well as dicts on all add endpoints This generalizes work by Antoine Dumont to all object addition endpoints, as a further step towards completely dropping the validate proxy in tests. 17 June 2020, 09:22:52 UTC
d153a80 cql: Fix blackified strings 16 June 2020, 18:11:31 UTC
5e053f8 storage: Add missing cur parameter 16 June 2020, 16:11:35 UTC
c2b673b Fix db_to_author() converter to return None is all fields are None Fix T2455. 16 June 2020, 10:32:20 UTC
ae131a9 New upstream version 0.4.0 16 June 2020, 07:58:21 UTC
8f1ac4c storage*: Drop leftover code This is no longer used, it should have been dropped with previous commits. Related to T2310 15 June 2020, 13:28:38 UTC
d6144d2 storage*: Drop origin_visit_upsert endpoint Related to T2310 15 June 2020, 12:26:42 UTC
c7f3060 storage*: Remove origin-visit-update endpoint Related to T2310 15 June 2020, 12:11:52 UTC
2bcbc82 replay: Replay origin-visit and origin-visit-status This now uses the respective origin-visit-add and origin-visit-status-add endpoints. Related to T2310 15 June 2020, 12:06:10 UTC
0183fec in_memory: Make origin-visit-status-add respect "on conflict ignore" policy Prior to this commit, that behavior was not properly tested and inconsistent between backends. All backends except in-memory were respecting it. This commit aligns the in-memory backend implementation and test it. Related to T2310 15 June 2020, 11:44:06 UTC
46a7839 test_storage: Add journal behavior coverage for origin-visit-*add This was missing some coverage on origin-visit-add and origin-visit-status-add for the journal part. Related to T2310 15 June 2020, 09:50:33 UTC
874da2d Start migrating the validate proxy toward using BaseModel objects This will allow to progress incrementally towards removing it. When it allows to use BaseModel objects everywhere (and tests in test_storage are adapted to use this property), it will be time to remove it entirely (as it's only used in test). It's preparatory work for future diffs. 13 June 2020, 06:37:57 UTC
33efdb0 storage*: Do not write twice origin-visit-status in journal Related to T2310 12 June 2020, 16:52:19 UTC
1661ef9 New upstream version 0.3.0 12 June 2020, 07:22:01 UTC
37c4530 storage*: Align origin-visit-add to take iterable of OriginVisit objects This makes its api consistent with other add endpoints. This is preparatory work towards removing origin-visit-upsert. Related to T2310 11 June 2020, 16:55:33 UTC
5d61633 test: Remove dead code 1. obj_type is now origin_visit_status. So this means, we actually never pass here. 2. Those objects have now a storage endpoint anyway. So it's dead code alright. Related to T2310 11 June 2020, 12:33:53 UTC
3a0e49f New upstream version 0.2.0 10 June 2020, 10:02:43 UTC
d68c7ec origin-visit-upsert: Write visit status objects to the journal Related to T2310 10 June 2020, 08:51:51 UTC
86d05fb origin-visit-update: Write visit status objects to the journal Related to T2310 09 June 2020, 12:33:58 UTC
0860920 origin-visit-add: Write visit status to the journal This also makes the instruction order consistent across the different storage implementations. First, write objects to the journal, then write objects to the storage backend. Related to T2310 09 June 2020, 12:29:57 UTC
7eb44d4 Add pagination to origin_metadata_get. 08 June 2020, 14:02:15 UTC
26a8d4f Add SortedList.iter_after. Strict version of iter_from. I'll need it for pagination. 08 June 2020, 14:01:48 UTC
6ebdc2f Deduplicate origin-metadata when they have the same authority + discovery_date + fetcher. By replacing the old value with the new one. This will allow an easy implementation of pagination, using the fetcher id as an opaque page_token. Plus, it did not make sense logically to have different metadata from the same authority at the same time (especially with the same fetcher). 08 June 2020, 14:01:30 UTC
dcef916 Open `origin_visit_status_add` endpoint to add origin visit statuses Related to T2310 05 June 2020, 16:32:12 UTC
88271f8 Add a replayer test for anonymized journal topics This new test check the behavior of the storage replayer mechanism when replaying a journal with privileged topics. 05 June 2020, 16:10:42 UTC
c75da7a Small refactoring of the InMemoryStorage to make it more consistent - make self._persons a dict - make self._snapshots value Snapshot only instead of the couple (Snapshot, sorted_branch_names) 05 June 2020, 10:28:04 UTC
91c3d65 New upstream version 0.1.1 04 June 2020, 14:56:52 UTC
25f584f Use explicit configuration (without journal writer) for algos tests Using the in-memory journal writer sometimes makes the tests hang when (very) large objects are used. This works around the issue. 04 June 2020, 14:14:07 UTC
525cf13 New upstream version 0.1.0 04 June 2020, 10:28:41 UTC
f9b2ca3 Replace MockedJournalClient and MockedKafkaWriter by proper kafka test scaffolding This also kills test_write_replay.py file since it does not test anything more than what is currently tested in test_replay.py. 04 June 2020, 09:46:12 UTC
ad9c9bb Adapt to swh.model 0.3 in which List attributes have replaced by Tuple ones. This requires a bit of adaptation in the code of the ValidatingProxyStorage to ensure dict representation of revision objects are properly typed. The test_api_client_dicts.py has been removed since it's not really useful any more and would require a fair amount of work to fix it. 04 June 2020, 09:46:12 UTC
eef4900 Fix InMemoryStorage.origin_visit_upsert() method the self._origin_visits[origin_url] list was built one element too big (since visit ids starts from 1 and not 0). This is needed to ease writing replayer tests (by comparing these lists). 29 May 2020, 13:23:28 UTC
6c6080b Fix type annotation. 28 May 2020, 15:03:13 UTC
9332547 Remove function drops from the migration. I committed these two lines by mistake 28 May 2020, 14:34:53 UTC
14b3019 New upstream version 0.0.193 28 May 2020, 12:37:55 UTC
2209d31 README: Update necessary dependencies for test purposes This also adds a mention on how to avoid running the cassandra tests. 28 May 2020, 12:25:31 UTC
7cb3694 152: Fix typo 28 May 2020, 12:07:04 UTC
738d648 db: Use query_params instead of mogrify Related to D3180 28 May 2020, 11:32:16 UTC
cada7fc pg: Write origin visit updates & status, read from origin_visit_status That still writes new origin visit and origin visit status but reads from origin visit status. This partially reverts commit b0b767b91ca077a14368eaac1f98120261d7460c [1] [1] Related to D3101 28 May 2020, 11:08:10 UTC
fe877ce Make content.blake2s256 not null. There is no reason to allow null anymore; and the production DB doesn't have any. 28 May 2020, 09:14:10 UTC
b0bff97 Remove unused SQL functions. 28 May 2020, 08:50:15 UTC
1aff3c6 Add a pre-commit hook to check there are version bumps in sql/upgrades/*.sql 28 May 2020, 08:21:50 UTC
3613438 Add missing dbversion bump in 150.sql. 28 May 2020, 07:54:12 UTC
213f1b1 Add artifact metadata to the extrinsic metadata storage specification. 26 May 2020, 11:01:52 UTC
f1b51a7 Add not null constraints to metadata_authority/origin_metadata Out of 149, I understood we wanted to make not null constraints on the following tables, so here it goes. This commits fixes that, migrating first the data not respecting those constraints. Related to D2988 Related to T2075 20 May 2020, 10:08:45 UTC
c69d100 Realign schema with latest 149 migration script If we really want to make those not null, please let's make another migration script. 19 May 2020, 19:59:24 UTC
1523dbc New upstream version 0.0.192 19 May 2020, 16:53:57 UTC
8c2ee70 origin_metadata_add: Reject non-bytes types for 'metadata'. The left-over jsonize() allowed passing a dict under some circumstances, which allowed tests to pass but failed in production. 19 May 2020, 16:29:24 UTC
0671780 New upstream version 0.0.191 19 May 2020, 11:51:58 UTC
e645e63 Implement extrinsic origin metadata specification. https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html 18 May 2020, 16:26:06 UTC
08f99ed New upstream version 0.0.190 18 May 2020, 12:18:07 UTC
6d24ed7 storage: metadata_provider: Ensure idempotency when creating provider Currently, in production, we have duplicated entries in origin_metadata and metadata_provider. This is due to a bad primary key on metadata_provider. This commit ensures uniqueness when adding new metadata_provider by adding a new unique primary key on (provider_type, provider_url). This is preparatory work to allow a smooth transition for D2988 (which will need to be rebased). Related to T2075 18 May 2020, 11:41:05 UTC
87f7bee journal: add a skipped_content topic dedicated to SkippedContent objects instead of mixing them with Content in the content topic. 13 May 2020, 10:12:41 UTC
306aa69 Add missing return annotations on JournalWriter methods 11 May 2020, 14:25:22 UTC
b04cb8f Improve a bit the exception message of JournalWriter.content_update 11 May 2020, 14:25:22 UTC
46f9a7a Refactor the JournalWriter class to normalize its methods make all the add_xxx methods take an Iterable as argument. Also extract the conditional execution of the underlying journal method calls in a dedicated methof to prevent repeating the ``if self.journal`` all over the place. Ensure all the storage code that uses this JournalWriter class are updated accordingly. 11 May 2020, 14:25:20 UTC
e64944b tests: fix test_replay; do only use aware datetime objects naive dates being now properly forbidden. 11 May 2020, 14:21:29 UTC
4edf9e1 test_kafka_writer: Add missing object type skipped_content 07 May 2020, 14:56:08 UTC
8412d49 swh-schema.sql: improve comments on revision columns Reviewers: #reviewers, vlorentz, ardumont Reviewed By: #reviewers, vlorentz, ardumont Subscribers: ardumont, vlorentz Differential Revision: https://forge.softwareheritage.org/D2825 07 May 2020, 14:54:24 UTC
74cffb7 swh-schema.sql: improve comments on revision columns Reviewers: #reviewers, vlorentz, ardumont Reviewed By: #reviewers, vlorentz, ardumont Subscribers: ardumont, vlorentz Differential Revision: https://forge.softwareheritage.org/D2825 07 May 2020, 14:53:54 UTC
68eb6b3 Update test_kafka_writer for swh.journal 0.1.0 07 May 2020, 08:26:24 UTC
b579670 New upstream version 0.0.189 30 April 2020, 12:58:55 UTC
b0b767b pg: Write both origin visit updates & status, read from origin_visit This partially reverts commit [1]. That now (still) writes new origin visit status... But, as before [1]: - update origin visit (with same values as origin visit status) - read from origin visit That does not revert the new in-memory (D2937) nor the cassandra (D2939) storage implementations. [1] a720caed6eebbb68a9f9b5be554a52859aa052d6 D2938 Related to D2938 Related to T2310#44043 30 April 2020, 11:52:13 UTC
0e8234f pg-storage: Add new created state Related to T2310 30 April 2020, 11:43:05 UTC
2b95dd3 setup.py: add documentation link 29 April 2020, 16:33:40 UTC
4dc2eb6 metadata spec: Fix title hierarchy 29 April 2020, 15:02:50 UTC
707f647 tests: Use aware datetimes instead of naive ones. Production should only use aware datetimes. 29 April 2020, 11:03:15 UTC
e3e76c4 cassandra: Adapt internal implementations to use origin visit update Related to T2310 28 April 2020, 14:46:47 UTC
a720cae pg-storage: Adapt internal implementations to use origin visit update Related to T2310 28 April 2020, 14:46:46 UTC
ead8088 in_memory: Adapt internal implementations to use origin visit update (pairing with @vlorentz) Related to T2310 28 April 2020, 14:46:43 UTC
back to top