https://github.com/SoftwareHeritage/swh-storage

sort by:
Revision Author Date Message Commit Date
96d7d7d Updated backport on buster-swh from debian/0.40.0-2_swh1 (unstable-swh) 17 November 2021, 09:20:41 UTC
4e093d6 Merge tag 'debian/0.40.0-2_swh1' into debian/buster-swh 17 November 2021, 09:20:40 UTC
ba105df d/changelog: version 0.40.0-2~swh1 17 November 2021, 08:46:19 UTC
9af9b26 Update dependencies in d/control 17 November 2021, 08:44:58 UTC
0b1b158 Updated debian changelog for version 0.40.0 16 November 2021, 09:45:39 UTC
1b6c7fc Update upstream source from tag 'debian/upstream/0.40.0' Update to upstream version '0.40.0' with Debian dir 0decf5aa20880a80efe3b342417439b86249cc5f 16 November 2021, 09:45:38 UTC
d8994d5 New upstream version 0.40.0 16 November 2021, 09:45:37 UTC
850a755 Add support for a redis-based reporting for invalid mirrorred objects The idea is that we check the BaseModel validity at journal deserialization time so that we still have access to the raw object from kafka for complete reporting (object id plus raw message from kafka). This uses a new ModelObjectDeserializer class that is responsible for deserializing the kafka message (still using kafka_to_value) then immediately create the BaseModel object from that dict. Its `convert` method is then passed as `value_deserializer` argument of the `JournalClient`. Then, for each deserialized object from kafka, if it's a HashableObject, check its validity by comparing the computed hash with its id. If it's invalid, report the error in logs, and if configured, register the invalid object in via the `reporter` callback. In the cli code, a `Redis.set()` is used a such a callback (if configured). So it simply stores invalid objects using the object id a key (typically its swhid), and the raw kafka message value as value. Related to T3693. 09 November 2021, 15:36:34 UTC
04bd15a Refactor fixer.fix_objects() to extract the inner object_fixers dict allowing to use this dict independently of the fix_objects() function. 09 November 2021, 15:36:34 UTC
d655c85 Remove now useless fixers keep the the fix_objects() function for bw compat for now. 09 November 2021, 15:36:34 UTC
55eed77 Add a --type option to 'swh storage replay' allows to choose replayed object types from the cli. 09 November 2021, 15:36:34 UTC
0262f1c Update extrinsic-metadata-specification.rst to match the current implementation * merged origin and artifact metadata * added metametadata * uses structures instead of dict * removed raw_extrinsic_metadata_get_latest 09 November 2021, 14:47:31 UTC
f6af4b4 Updated backport on buster-swh from debian/0.39.0-1_swh1 (unstable-swh) 29 October 2021, 09:23:51 UTC
0ced1ee Merge tag 'debian/0.39.0-1_swh1' into debian/buster-swh 29 October 2021, 09:23:50 UTC
467aa27 Updated debian changelog for version 0.39.0 29 October 2021, 09:17:36 UTC
8c63421 Update upstream source from tag 'debian/upstream/0.39.0' Update to upstream version '0.39.0' with Debian dir fc10c75ae5be9e40348d539c00fc26603ce1ecd6 29 October 2021, 09:17:35 UTC
3ce052a New upstream version 0.39.0 29 October 2021, 09:17:33 UTC
a5bfe5b interface: Add origin_snapshot_get_all method It enables to return in an efficient way the list of unique snapshot identifiers resulting from the visits of an origin. Previously it was required to query all visits of an origin then query all visit statuses for each visit to extract such information. Introduced method enables to extract origin snaphots information in a single datase query. Related to T3631 28 October 2021, 12:23:14 UTC
49a932c algos/revisions_walker: Handle case of revision without committer date Some revisions in the archive do not have committer date so workaround it to avoid errors when walking on such revisions when using the class CommitterDateRevisionsWalker. 22 October 2021, 09:32:28 UTC
c02be8e test_revisions_walker: Migrate from unittest to pytest 21 October 2021, 14:50:57 UTC
a986710 cassandra: Fix incomplete check of content existence in object_find_by_sha1_git content_missing_by_sha1_git only checks the index and not the main table. This is incorrect, because contents should not be considered written before an entry is written to the main table, even if an entry exists in one of the indexes. 18 October 2021, 11:10:54 UTC
4e1c0da Updated backport on buster-swh from debian/0.38.0-1_swh1 (unstable-swh) 11 October 2021, 15:04:39 UTC
497ba10 Merge tag 'debian/0.38.0-1_swh1' into debian/buster-swh 11 October 2021, 15:04:38 UTC
b639a49 Updated debian changelog for version 0.38.0 11 October 2021, 14:58:23 UTC
e86eafe Update upstream source from tag 'debian/upstream/0.38.0' Update to upstream version '0.38.0' with Debian dir fb3ad9c54e04b857bee1ba72970de66ed96cae17 11 October 2021, 14:58:22 UTC
8567342 New upstream version 0.38.0 11 October 2021, 14:58:20 UTC
e9fd74d serializers: Prepare rename of 'identifiers_enum' to 'swhids_enum'. This will be done in three steps to avoid any disruption: 1. (this step) add support for decoding the new name, but keep encoding as the old one 2. start encoding as the new name 3. remove support for decoding the old name 11 October 2021, 11:58:23 UTC
ea86a86 Rename imports of swh.model.identifiers to fix deprecation warnings. 11 October 2021, 11:58:23 UTC
3441f68 buffer: add some debug logging for number of objects sent 08 October 2021, 14:53:27 UTC
b604014 buffer: add a threshold for the estimated size of revision and release batches The size of individual revisions and releases is essentially unbounded. This means that, when the buffer storage is used as a way of limiting memory use for an ingestion process, it is still possible to go beyond the expected memory use when adding a batch of revisions or releases with large messages or other metadata. The duration of the database operations for revision_add or release_add is also commensurate to the size of the objects added in a batch, so using the buffer proxy to limit the time individual database operations takes was not effective. Adding a threshold on estimated sizes for batches of revision and release objects makes this overuse of memory and of database transaction time much less likely. 08 October 2021, 14:53:27 UTC
7c5b0ec buffer: add a threshold for the number of revision parents in one batch The size of individual revisions is essentially unbounded. This means that, when the buffer storage is used as a way of limiting memory use for an ingestion process, it is still possible to go beyond the expected memory use when adding a batch of revisions with extensive histories. The duration of the database operation for revision_add is also commensurate to the number of revision parents added in a batch, so using the buffer proxy to limit the time individual database operations takes was not effective. Adding a threshold on cumulated number of revision parents per batch makes this overuse of memory and of database transaction time much less likely. 08 October 2021, 13:42:35 UTC
5edc0ba buffer: add a threshold for the number of directory entries in one batch The size of individual directories is essentially unbounded. This means that, when the buffer storage is used as a way of limiting memory use for an ingestion process, it is still possible to go beyond the expected memory use when adding a batch of (very) large directories. The duration of the database operation for directory_add is also commensurate to the number of directory entries added in a batch, so using the buffer proxy to limit the time individual database operations takes was not effective. Adding a threshold on cumulated number of directory entries per batch makes this overuse of memory and of database transaction time much less likely. 08 October 2021, 13:13:05 UTC
abe95b3 filter: add filtering for release_add 08 October 2021, 12:11:32 UTC
c52b7b6 filter: do not call the underlying functions if there's nothing to add 08 October 2021, 10:20:22 UTC
5d5d4c9 buffer: Ensure that we don't send data from empty buffers This was already the case (as grouper called on an empty iterator just returns no batches), but add a test to enforce it. 08 October 2021, 09:45:25 UTC
c7bc213 Updated backport on buster-swh from debian/0.37.1-1_swh1 (unstable-swh) 29 September 2021, 10:19:01 UTC
1044918 Merge tag 'debian/0.37.1-1_swh1' into debian/buster-swh 29 September 2021, 10:19:01 UTC
91091fe Updated debian changelog for version 0.37.1 29 September 2021, 10:12:31 UTC
4ec651f Update upstream source from tag 'debian/upstream/0.37.1' Update to upstream version '0.37.1' with Debian dir 1d9eff5555cc4c457089da74ffcf0997ef88b7c2 29 September 2021, 10:12:30 UTC
4c0ea11 New upstream version 0.37.1 29 September 2021, 10:12:28 UTC
113088a replay: add type annotation for process_replay_objects() 29 September 2021, 09:01:57 UTC
9a3589f replay: fix raw_extrinsic_metadata insertion and type annotation due to missing type annotation of the storage argument of _insert_objects(), we missed a bug in the processing of raw_extrinsic_metadata objects, passing set() as arguments of storage add methods. 29 September 2021, 08:48:13 UTC
21aff2d replay: fix annotation of collision_aware_content_add() now the callable is expected to return a dict. 29 September 2021, 08:48:13 UTC
8ba232c Fix support of swh-model 3.0.0 28 September 2021, 15:59:20 UTC
42bad90 postgresql: Don't raise StorageArgumentException in case of write conflicts 27 September 2021, 13:30:14 UTC
ec548ee Add bazaar as supported revision type This has a corresponding change in swh.model 23 September 2021, 08:55:35 UTC
61e9e4a cassandra: Make _content_get_from_hashes run concurrently This is used by directory_ls and content_get. 21 September 2021, 16:14:28 UTC
59e63db postgresql: Fix regression introduced in previous commit Methods snapshot_get and snapshot_get_branches should return None if the snapshot does not exist in the archive. Add missing tests to cover that case. 21 September 2021, 10:25:42 UTC
9465054 postgresql: Fix get_snapshot_branches return value for empty search When searching for branches in an existing snapshot, a PartialBranches object must be returned regardless the number of found branches. None should only be returned when a snapshot does not exist. This fixes an inconsistency between the postgresql and cassandra backends. Related to T3413 16 September 2021, 12:09:36 UTC
f52f62e Updated backport on buster-swh from debian/0.37.0-1_swh2 (unstable-swh) 16 September 2021, 08:00:40 UTC
9b8c04c Merge tag 'debian/0.37.0-1_swh2' into debian/buster-swh 16 September 2021, 08:00:39 UTC
3da47c9 d/changelog: Bump new release Related to T3578 16 September 2021, 07:53:26 UTC
0db1105 d/control: Update missing deps Related to T3578 16 September 2021, 07:40:00 UTC
5abd216 Updated debian changelog for version 0.37.0 16 September 2021, 06:42:23 UTC
9263b69 Update upstream source from tag 'debian/upstream/0.37.0' Update to upstream version '0.37.0' with Debian dir 304ad56e032c6f722cf702178ece7d62e3dcef12 16 September 2021, 06:42:21 UTC
cb10244 New upstream version 0.37.0 16 September 2021, 06:42:19 UTC
a9fde72 Allow filtering extids per extid_version/extid_type when reading This impacts both the `extid_get_from_extid` and `extid_get_from_target` endpoints. Whe extid_version/extid_type are not provided, this keeps the existing behavior of returning all extids matching. Related to T3567 15 September 2021, 16:34:59 UTC
589d20e migrate_extrinsic_metadata: Fix missing f-stringification 14 September 2021, 09:15:41 UTC
1c8337f migrate_extrinsic_metadata: Fix crash on deposit hal-02355563 10 September 2021, 17:25:51 UTC
3315738 migrate_extrinsic_metadata: Fix remaining pypi issues All packages now pass 10 September 2021, 17:25:03 UTC
8e94afa migrate_extrinsic_metadata: Fix off-by-one error, causing the first_id to be skipped 10 September 2021, 16:17:56 UTC
5facf66 cassandra: Make directory_ls fetch contents in batch instead of one-by-one This should make it run up to 100 times faster, even on average directories. 09 September 2021, 13:45:30 UTC
0570a42 content_get: Fetch rows concurrently Instead of fetching them one-by-one, with the very high latency this entails. This is preliminary work to make `directory_ls` less painfully slow. 09 September 2021, 13:43:42 UTC
50fb54f directory_entry_add_batch: Remove the temporary prepared statement entirely And fall back to concurrent insertion. 09 September 2021, 09:35:22 UTC
da7e63e directory_entry_add_batch: Reduce churn of prepared statements By reusing the 'steady state' main statement (which is quite large) across calls. 08 September 2021, 09:56:49 UTC
fc950de cassandra: Add option to select (hopefully) more efficient batch insertion algos This adds a new config option for the cassandra backend, 'directory_entries_insert_algo', with three possible values: * 'one-per-one' is the default, and preserves the current naive behavior * 'concurrent' and 'batch' are attempts at being more efficient 08 September 2021, 09:54:57 UTC
7dc2863 migrate_extrinsic_metadata: Add an option to limit the number of revisions This will be used as a second pass on objects that failed with older versions of the script. 06 September 2021, 12:45:40 UTC
834a49d test_directory_get_entries_pagination: don't depend on result order 03 September 2021, 12:56:15 UTC
e8aad0f cassandra: Remove stat_counters. They were inaccurate and a performance bottleneck. We can/should use swh-counters instead, now. 31 August 2021, 08:41:48 UTC
3ad1bec postgresql: Fix a column order mismatch between the query and object builder resulting in OriginVisitStatus trying to put a snapshot id in the metadata field Related to T3539 30 August 2021, 15:39:42 UTC
999ea6b cassandra: generate statsd metrics on method calls Related to T3517 30 August 2021, 15:25:59 UTC
47a6919 Add counting storage proxy It will be used in the Cassandra experiment. Currently we use the built-in counters of the Cassandra backend; but in addition to being inaccurate, they seem to be a bottleneck. This proxy will be a lightweight solution for counting object insertion, without needing to run Kafka on the test cluster. 27 August 2021, 11:31:37 UTC
89d28d5 Updated backport on buster-swh from debian/0.36.0-1_swh1 (unstable-swh) 24 August 2021, 15:07:34 UTC
dac2450 Merge tag 'debian/0.36.0-1_swh1' into debian/buster-swh 24 August 2021, 15:07:33 UTC
ffe636f Updated debian changelog for version 0.36.0 24 August 2021, 15:01:32 UTC
6924a7e Update upstream source from tag 'debian/upstream/0.36.0' Update to upstream version '0.36.0' with Debian dir 179c1ad6d3ce02e0f64d5944d38e3e3d48e86d89 24 August 2021, 15:01:30 UTC
3a224a9 New upstream version 0.36.0 24 August 2021, 15:01:29 UTC
b110d1b Add cvs as supported revision_type 24 August 2021, 14:39:03 UTC
8f1cdf6 Add test for origin_visit_get_latest in presence of mismatched id and date orders It was unclear this actually worked; I had to write this test to realize the code wasn't buggy. Also replaced a conditional that is always False (because Cassandra always returns results in the order of the clustering key) with an assertion, so the code is less confusing. 24 August 2021, 13:14:39 UTC
cf880db cassandra: Bump next_visit_id when origin_visit_add is called by a replayer When called by a replayer, the visit.visit field is set; but origin.next_visit_id was never incremented, so on the next loader run, the visit id would be 1 even if there is already a visit with that id. 24 August 2021, 13:14:39 UTC
54b5abf cassandra: Make content_missing query in batches Instead of calling content_find() for each object, which needs to make two queries for each. Given the latency of Cassandra queries, this should be a significant speed-up (possibly up to 100 times faster, as this is the value of PARTITION_KEY_RESTRICTION_MAX_SIZE). This also changes the schema, because CQL does not allow doing `IN` queries on compound partition keys. 24 August 2021, 13:14:39 UTC
7113198 backfill: add extra where clause to use the right index for extid requests Related to T3485 24 August 2021, 11:52:32 UTC
55c3f0c Updated backport on buster-swh from debian/0.35.1-1_swh1 (unstable-swh) 20 August 2021, 10:07:02 UTC
8f1e8cd Merge tag 'debian/0.35.1-1_swh1' into debian/buster-swh 20 August 2021, 10:07:02 UTC
ae70564 Updated debian changelog for version 0.35.1 20 August 2021, 10:01:16 UTC
ca5ee3d Update upstream source from tag 'debian/upstream/0.35.1' Update to upstream version '0.35.1' with Debian dir 313295a88c0c4c4f7c924c7c36bea4c310f2cbca 20 August 2021, 10:01:15 UTC
1c038f0 New upstream version 0.35.1 20 August 2021, 10:01:14 UTC
9f00eb9 cassandra: Fix crash when using _missing() functions with more than 100 ids with ScyllaDB. 06 August 2021, 12:59:29 UTC
18418b2 Updated backport on buster-swh from debian/0.35.0-1_swh1 (unstable-swh) 28 July 2021, 13:11:03 UTC
1ac2093 Merge tag 'debian/0.35.0-1_swh1' into debian/buster-swh 28 July 2021, 13:11:03 UTC
5269c40 Updated debian changelog for version 0.35.0 28 July 2021, 08:43:06 UTC
96b0ed6 Update upstream source from tag 'debian/upstream/0.35.0' Update to upstream version '0.35.0' with Debian dir 6ad84ab1c5fb66c4ead922e0b750291db31def2f 28 July 2021, 08:43:05 UTC
3be3f63 New upstream version 0.35.0 28 July 2021, 08:43:04 UTC
912d04e sql: Adapt extid.extid_version comment 27 July 2021, 14:58:02 UTC
7a38045 Implement storage of the ExtID.extid_version field This fields allows having multiple version of the ExtID -> SWHID mapping, for instance when the implementation of a loader changes in a backwards-incompatible way. For now, we don't change the API used to query or store ExtIDs. When querying for the SWHIDs corresponding to a given external objects, all versions are returned, and the client is expected to do the filtering. 23 July 2021, 15:37:12 UTC
f224765 Updated backport on buster-swh from debian/0.34.0-1_swh1 (unstable-swh) 07 July 2021, 17:04:51 UTC
3408528 Merge tag 'debian/0.34.0-1_swh1' into debian/buster-swh 07 July 2021, 17:04:50 UTC
f57de45 Updated debian changelog for version 0.34.0 07 July 2021, 16:58:42 UTC
d051810 Update upstream source from tag 'debian/upstream/0.34.0' Update to upstream version '0.34.0' with Debian dir d1489fbcf94a7c6a63233ae05f74d2de68f382bc 07 July 2021, 16:58:39 UTC
b07c48a New upstream version 0.34.0 07 July 2021, 16:58:37 UTC
back to top