swh:1:snp:eb70f1f85391e4b077c211bec36af0061c4bf937

sort by:
Revision Author Date Message Commit Date
37c4530 storage*: Align origin-visit-add to take iterable of OriginVisit objects This makes its api consistent with other add endpoints. This is preparatory work towards removing origin-visit-upsert. Related to T2310 11 June 2020, 16:55:33 UTC
5d61633 test: Remove dead code 1. obj_type is now origin_visit_status. So this means, we actually never pass here. 2. Those objects have now a storage endpoint anyway. So it's dead code alright. Related to T2310 11 June 2020, 12:33:53 UTC
d68c7ec origin-visit-upsert: Write visit status objects to the journal Related to T2310 10 June 2020, 08:51:51 UTC
86d05fb origin-visit-update: Write visit status objects to the journal Related to T2310 09 June 2020, 12:33:58 UTC
0860920 origin-visit-add: Write visit status to the journal This also makes the instruction order consistent across the different storage implementations. First, write objects to the journal, then write objects to the storage backend. Related to T2310 09 June 2020, 12:29:57 UTC
7eb44d4 Add pagination to origin_metadata_get. 08 June 2020, 14:02:15 UTC
26a8d4f Add SortedList.iter_after. Strict version of iter_from. I'll need it for pagination. 08 June 2020, 14:01:48 UTC
6ebdc2f Deduplicate origin-metadata when they have the same authority + discovery_date + fetcher. By replacing the old value with the new one. This will allow an easy implementation of pagination, using the fetcher id as an opaque page_token. Plus, it did not make sense logically to have different metadata from the same authority at the same time (especially with the same fetcher). 08 June 2020, 14:01:30 UTC
dcef916 Open `origin_visit_status_add` endpoint to add origin visit statuses Related to T2310 05 June 2020, 16:32:12 UTC
88271f8 Add a replayer test for anonymized journal topics This new test check the behavior of the storage replayer mechanism when replaying a journal with privileged topics. 05 June 2020, 16:10:42 UTC
c75da7a Small refactoring of the InMemoryStorage to make it more consistent - make self._persons a dict - make self._snapshots value Snapshot only instead of the couple (Snapshot, sorted_branch_names) 05 June 2020, 10:28:04 UTC
25f584f Use explicit configuration (without journal writer) for algos tests Using the in-memory journal writer sometimes makes the tests hang when (very) large objects are used. This works around the issue. 04 June 2020, 14:14:07 UTC
f9b2ca3 Replace MockedJournalClient and MockedKafkaWriter by proper kafka test scaffolding This also kills test_write_replay.py file since it does not test anything more than what is currently tested in test_replay.py. 04 June 2020, 09:46:12 UTC
ad9c9bb Adapt to swh.model 0.3 in which List attributes have replaced by Tuple ones. This requires a bit of adaptation in the code of the ValidatingProxyStorage to ensure dict representation of revision objects are properly typed. The test_api_client_dicts.py has been removed since it's not really useful any more and would require a fair amount of work to fix it. 04 June 2020, 09:46:12 UTC
eef4900 Fix InMemoryStorage.origin_visit_upsert() method the self._origin_visits[origin_url] list was built one element too big (since visit ids starts from 1 and not 0). This is needed to ease writing replayer tests (by comparing these lists). 29 May 2020, 13:23:28 UTC
6c6080b Fix type annotation. 28 May 2020, 15:03:13 UTC
9332547 Remove function drops from the migration. I committed these two lines by mistake 28 May 2020, 14:34:53 UTC
2209d31 README: Update necessary dependencies for test purposes This also adds a mention on how to avoid running the cassandra tests. 28 May 2020, 12:25:31 UTC
7cb3694 152: Fix typo 28 May 2020, 12:07:04 UTC
738d648 db: Use query_params instead of mogrify Related to D3180 28 May 2020, 11:32:16 UTC
cada7fc pg: Write origin visit updates & status, read from origin_visit_status That still writes new origin visit and origin visit status but reads from origin visit status. This partially reverts commit b0b767b91ca077a14368eaac1f98120261d7460c [1] [1] Related to D3101 28 May 2020, 11:08:10 UTC
fe877ce Make content.blake2s256 not null. There is no reason to allow null anymore; and the production DB doesn't have any. 28 May 2020, 09:14:10 UTC
b0bff97 Remove unused SQL functions. 28 May 2020, 08:50:15 UTC
1aff3c6 Add a pre-commit hook to check there are version bumps in sql/upgrades/*.sql 28 May 2020, 08:21:50 UTC
3613438 Add missing dbversion bump in 150.sql. 28 May 2020, 07:54:12 UTC
213f1b1 Add artifact metadata to the extrinsic metadata storage specification. 26 May 2020, 11:01:52 UTC
f1b51a7 Add not null constraints to metadata_authority/origin_metadata Out of 149, I understood we wanted to make not null constraints on the following tables, so here it goes. This commits fixes that, migrating first the data not respecting those constraints. Related to D2988 Related to T2075 20 May 2020, 10:08:45 UTC
c69d100 Realign schema with latest 149 migration script If we really want to make those not null, please let's make another migration script. 19 May 2020, 19:59:24 UTC
8c2ee70 origin_metadata_add: Reject non-bytes types for 'metadata'. The left-over jsonize() allowed passing a dict under some circumstances, which allowed tests to pass but failed in production. 19 May 2020, 16:29:24 UTC
e645e63 Implement extrinsic origin metadata specification. https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html 18 May 2020, 16:26:06 UTC
6d24ed7 storage: metadata_provider: Ensure idempotency when creating provider Currently, in production, we have duplicated entries in origin_metadata and metadata_provider. This is due to a bad primary key on metadata_provider. This commit ensures uniqueness when adding new metadata_provider by adding a new unique primary key on (provider_type, provider_url). This is preparatory work to allow a smooth transition for D2988 (which will need to be rebased). Related to T2075 18 May 2020, 11:41:05 UTC
87f7bee journal: add a skipped_content topic dedicated to SkippedContent objects instead of mixing them with Content in the content topic. 13 May 2020, 10:12:41 UTC
306aa69 Add missing return annotations on JournalWriter methods 11 May 2020, 14:25:22 UTC
b04cb8f Improve a bit the exception message of JournalWriter.content_update 11 May 2020, 14:25:22 UTC
46f9a7a Refactor the JournalWriter class to normalize its methods make all the add_xxx methods take an Iterable as argument. Also extract the conditional execution of the underlying journal method calls in a dedicated methof to prevent repeating the ``if self.journal`` all over the place. Ensure all the storage code that uses this JournalWriter class are updated accordingly. 11 May 2020, 14:25:20 UTC
e64944b tests: fix test_replay; do only use aware datetime objects naive dates being now properly forbidden. 11 May 2020, 14:21:29 UTC
4edf9e1 test_kafka_writer: Add missing object type skipped_content 07 May 2020, 14:56:08 UTC
8412d49 swh-schema.sql: improve comments on revision columns Reviewers: #reviewers, vlorentz, ardumont Reviewed By: #reviewers, vlorentz, ardumont Subscribers: ardumont, vlorentz Differential Revision: https://forge.softwareheritage.org/D2825 07 May 2020, 14:54:24 UTC
74cffb7 swh-schema.sql: improve comments on revision columns Reviewers: #reviewers, vlorentz, ardumont Reviewed By: #reviewers, vlorentz, ardumont Subscribers: ardumont, vlorentz Differential Revision: https://forge.softwareheritage.org/D2825 07 May 2020, 14:53:54 UTC
68eb6b3 Update test_kafka_writer for swh.journal 0.1.0 07 May 2020, 08:26:24 UTC
b0b767b pg: Write both origin visit updates & status, read from origin_visit This partially reverts commit [1]. That now (still) writes new origin visit status... But, as before [1]: - update origin visit (with same values as origin visit status) - read from origin visit That does not revert the new in-memory (D2937) nor the cassandra (D2939) storage implementations. [1] a720caed6eebbb68a9f9b5be554a52859aa052d6 D2938 Related to D2938 Related to T2310#44043 30 April 2020, 11:52:13 UTC
0e8234f pg-storage: Add new created state Related to T2310 30 April 2020, 11:43:05 UTC
2b95dd3 setup.py: add documentation link 29 April 2020, 16:33:40 UTC
4dc2eb6 metadata spec: Fix title hierarchy 29 April 2020, 15:02:50 UTC
707f647 tests: Use aware datetimes instead of naive ones. Production should only use aware datetimes. 29 April 2020, 11:03:15 UTC
e3e76c4 cassandra: Adapt internal implementations to use origin visit update Related to T2310 28 April 2020, 14:46:47 UTC
a720cae pg-storage: Adapt internal implementations to use origin visit update Related to T2310 28 April 2020, 14:46:46 UTC
ead8088 in_memory: Adapt internal implementations to use origin visit update (pairing with @vlorentz) Related to T2310 28 April 2020, 14:46:43 UTC
49109d1 test_retry: Centralize time.sleep setup within a fixture This monkeypatches the internal sleep function used to not wait. This kept the previous behavior. It changes the implementation to monkeypatch though. This also centralizes within a fixture. This avoids repeating setup. The previous implementation cluttered the tests body instruction with internal implementation details. 27 April 2020, 13:10:05 UTC
ecadd53 Remove recently added test_cli.test_rpc_serve* tests these are unreliable and werkzeug's tests WSGI server does not handle being executed from pytest very well (means it executes pytest all over again). 24 April 2020, 13:16:06 UTC
038c30b Adapt journal client loading to swh.journal 0.0.31 24 April 2020, 09:58:02 UTC
4b7ba1f Copy the graph replayer component from swh-journal The CLI command is included as well as `swh storage replay`. Copied test test_replay, in test_cli.py, should be identical. 24 April 2020, 09:58:02 UTC
5fd9b56 Deprecate the `config-path` argument of the `swh storage rpc-serve` command in favor of the standard `--config-file` option of `swh storage`. Attempt to write a couple of tests for the rpc-serve command. 24 April 2020, 09:27:07 UTC
cd32cf4 Normalize the configuration file handling in the `swh storage` CLI command Almost every other swh package handles the loading of the config file from the main click command group of the package. So we make storage behaves the same. 23 April 2020, 11:31:25 UTC
698af8c cli: rename the command 'backfiller' as 'backfill' for the sake of consistency. 23 April 2020, 11:31:25 UTC
b2bba45 Copy the backfiller component from swh-journal This componant makes more sense in the swh-storage package. 23 April 2020, 11:31:25 UTC
fe56005 Add a wrapper to manage a sorted list. For now this is only used by sorted_sha1, but we'll need it for origin_metadata soon. 23 April 2020, 11:03:28 UTC
bca643a setup: Update the minimum required runtime python3 version Related to T2367 20 April 2020, 15:22:45 UTC
32b3e93 doc: reference SWHID using explicit anchors 17 April 2020, 14:28:18 UTC
f66184d storage.interface: Actually define the remote flush operation As all storage are chained together, we need to define it. We really need the actual backends, which should be the last storage chained to do noop when that endpoint is called (and they do). 14 April 2020, 16:05:03 UTC
4359874 Drop BWCompatInMemoryJournalWriter (released with swh.journal 0.0.30) 14 April 2020, 14:26:18 UTC
2cc263d test: update storage tests to (future) swh.journal 0.0.30 which will handle swh.model objects everywhere instead of dicts. Also add a BW compat version os the InMemoryJournalWriter so tests will pass with current version of swh.journal (0.0.29). 14 April 2020, 09:37:31 UTC
e5e5943 storage.filter: Remove internal state 14 April 2020, 08:54:14 UTC
ddac3d2 test_retry: Add missing skipped_content_add tests 10 April 2020, 11:53:34 UTC
54b2907 storage*: Add flush endpoints to storage implems (backend, proxy) All storage defines one endpoint even if it's mostly noop. This avoids introspection surprises. Related to D2966 (to be consistent) 10 April 2020, 09:08:05 UTC
b0b0313 test_filter: Extract the filter storage into a fixture 09 April 2020, 07:37:20 UTC
566c325 storage*: Add `clear_buffers` operation for proxy storages This also adds the endpoint as noop for the main backend implementations. Related to T2352 09 April 2020, 07:37:20 UTC
ed4097c Add a pyproject.toml file to target py37 for black 08 April 2020, 20:11:21 UTC
cd52a03 Enable black - blackify all the python files, - enable black in pre-commit, - add a black tox environment. 08 April 2020, 13:16:34 UTC
0fe4665 Fix Storage.origin_visit_update(); ensure it raises a StorageArgumentException 08 April 2020, 13:16:34 UTC
be954f2 buffer: filter out duplicate objects. 08 April 2020, 11:01:00 UTC
c51139e Make Storage.origin_visit_update() add an OriginVisit model entity in the journal instead of a dict, to comply with next version of swh.journal (which will require swh.model objects). 08 April 2020, 10:03:38 UTC
bf48cfe Make swh/storage/storage.py flake8 compliant 08 April 2020, 10:03:38 UTC
fbb51aa Add a setup.cfg file to configure flake8 for black compatibility 08 April 2020, 10:03:38 UTC
8e8577e Prevent erroneous HashCollisions by using the same ctime for all rows. 'swh_content_add' tries to avoid this issue with a DISTINCT clause on the entire row; but it is useless because 'ctime' cells differ by a few microseconds. This commit ensures all ctime values are exactly the same, so they are filtered out. An alternative would be to change 'swh_content_add' to do: ``` select distinct on (sha1, sha1_git, sha256, blake2s256, length, status) sha1, sha1_git, sha256, blake2s256, length, status, ctime from tmp_content ``` instead of: ``` select distinct sha1, sha1_git, sha256, blake2s256, length, status, ctime from tmp_content ``` but this is more verbose and there's no good reason to call 'now()' for every row. 08 April 2020, 08:30:41 UTC
82b41ba Remove magic CassObject class, use dicts instead. 02 April 2020, 14:40:00 UTC
df3207a Adapt cassandra backend to validating model types This is required to be able to activate type validation in the model (in swh.model.model). It requires to replace the "distorded" usage that was done of model entities to build objects compatible with CqlRunner's object addition logic. Since we cannot create invalid model entities any more in this context, we add a new CassObject type (just a dict with __getattr__=__getitem__) and use it as object passed to the CqlRunner for entity types that need special care (namely Revision and Release). This should still work with swh.model v0.0.62 (without type validation) as well as the (next) v0.0.63 which will come with type valdation. 01 April 2020, 12:19:41 UTC
20baa1b test: convert test_converts.py to pytest style 01 April 2020, 12:19:41 UTC
fa4a043 test: get rid of normalized_xxx in tests This is not needed any more with properly typed test data. 01 April 2020, 12:19:41 UTC
fcca905 test: ensure timestamp in test data are properly typed according to model declaration, a timestamp must be a dict with 2 keys, 'seconds' and 'microseconds'. Also add a few more tests for the date_to_db helper function so that the test coverage of this later remains. 01 April 2020, 12:19:41 UTC
377e6a8 tests: Prepare tests for origin_visit_update objects Related to T2310 01 April 2020, 09:56:05 UTC
9c22156 storage*: Add missing type annotations on origin_visit_get* endpoints 01 April 2020, 09:02:03 UTC
ff0a538 tests: Skip internal origin_visit_update model object generation Beside making the tests fail, they are not helpful right now. This commit will avoid the current master build from breaking. Also, the model bump is a mandatory preparatory work for making the origin visit immutable. Related to T2310 01 April 2020, 08:52:54 UTC
2856004 Ensure visit id is set in origin_visit_upsert before journal writes 31 March 2020, 14:28:25 UTC
acf057e storage*: Unify validation exception capture across storages 31 March 2020, 13:56:35 UTC
81e7575 storage*: Stop duplication and use storage.utils.now function 31 March 2020, 13:43:45 UTC
eb82792 cassandra/cql: Simplify type using Iterator 31 March 2020, 13:02:30 UTC
46fa27e storage*: Add types to origin_visit_get 31 March 2020, 12:58:55 UTC
4bdde50 storage*: Align origin_visit_update interface and implementations This also adds an unused (yet) optional parameter date. It will soon be used in the context of origin_visit_update use. Related to T2310 31 March 2020, 12:42:08 UTC
69862b0 storage: Define a now() function 31 March 2020, 12:35:00 UTC
8e8e3a9 test_retry: Remove unused import 31 March 2020, 12:34:22 UTC
c53433d test_retry: Use datetime instead of string Reuse a date_visit from the sample storage 31 March 2020, 12:24:36 UTC
623a1b7 test: add a small test to check for type validation (using release_add) 27 March 2020, 14:56:01 UTC
90c4112 validate: fix type annotation for origin_visit_add date argument is expected to be a datetime. 27 March 2020, 08:50:12 UTC
1916fd7 validate: ensure StorageArgumentException is always encodable by embeding a string representation of the original Exception as StorageArgumentException args instead of the original exc.args since this can contain any python (possibli non-serializable) object. This is needed e.g. when swh.model has runtime type validation. 27 March 2020, 08:50:12 UTC
c67fe21 writer: fix skipped_content_add type declaration to use SkippedContent instead of plain Content. 27 March 2020, 08:50:12 UTC
19be96f tests: fix types of several test data sets these are currently accepted by swh.model, but won't be any more as soon as we activate type validation in swh.model. 27 March 2020, 08:50:12 UTC
570dce2 Shut down cassandra connection before closing the fixture down 26 March 2020, 19:43:30 UTC
ce5d2bf storage*: Update origin_visit_update to make status parameter mandatory This actually aligns with the origin_visit model whose status is already mandatory. 26 March 2020, 15:13:45 UTC
40a7569 test: Adapt origin validation test according to latest model changes Origin model no longer allows to have a type. Related to f533f62bbf114cfcc29f7c72307c4dfbe99cf048 26 March 2020, 14:44:24 UTC
back to top