| cbbaeea | Jenkins for Software Heritage | 23 November 2020, 13:52:33 UTC | Updated debian changelog for version 0.18.0 | 23 November 2020, 13:52:33 UTC |
| e5bce82 | Jenkins for Software Heritage | 23 November 2020, 13:52:32 UTC | Update upstream source from tag 'debian/upstream/0.18.0' Update to upstream version '0.18.0' with Debian dir 6ef3ab996694ae4582886e07c25624fa073dc956 | 23 November 2020, 13:52:32 UTC |
| 4c46835 | Jenkins for Software Heritage | 23 November 2020, 13:52:31 UTC | New upstream version 0.18.0 | 23 November 2020, 13:52:31 UTC |
| e289593 | Antoine R. Dumont (@ardumont) | 20 November 2020, 16:25:27 UTC | requirements-test.txt: Drop no longer needed pytest-postgresql requirement requirements-swh.txt already declares the swh.core[db] dependency which transitively pulls it. Related to T2746 | 23 November 2020, 12:07:45 UTC |
| b1375bc | Antoine R. Dumont (@ardumont) | 23 November 2020, 10:54:40 UTC | d/control: Rework test dependency Related to T2746 | 23 November 2020, 10:54:40 UTC |
| 0065d4d | Nicolas Dandrimont | 13 November 2020, 15:50:18 UTC | backfill: Reverse flawed logic in SnapshotBranch generation The previous code would nullify all non-null branches, and try to create a SnapshotBranch out of null branches. | 13 November 2020, 15:51:29 UTC |
| f501136 | Valentin Lorentz | 06 November 2020, 11:59:15 UTC | migrate_extrinsic_metadata: don't crash when deb revisions aren't referenced by any snapshot As this happens for about 50 revisions in the archive. | 13 November 2020, 14:26:15 UTC |
| 4bec746 | Jenkins for Software Heritage | 13 November 2020, 11:05:35 UTC | Updated debian changelog for version 0.17.2 | 13 November 2020, 11:05:35 UTC |
| bf7d9c7 | Jenkins for Software Heritage | 13 November 2020, 11:05:34 UTC | Update upstream source from tag 'debian/upstream/0.17.2' Update to upstream version '0.17.2' with Debian dir cb682d738260a73e1a5c714e32fe464181268a21 | 13 November 2020, 11:05:34 UTC |
| 6089094 | Jenkins for Software Heritage | 13 November 2020, 11:05:33 UTC | New upstream version 0.17.2 | 13 November 2020, 11:05:33 UTC |
| 20d3f8e | Nicolas Dandrimont | 12 November 2020, 17:04:44 UTC | backfill: only flush the journal writer on every batch This module's use of write_addition predated the introduction of reliable writing in swh.journal; Since this introduction, the backfiller has been flushing the kafka writer after writing each single object, leading to a 3x measured slowdown on backfilling contents. | 13 November 2020, 10:17:31 UTC |
| 248a04b | Nicolas Dandrimont | 12 November 2020, 17:04:17 UTC | Don't use string expansions in debug logging | 12 November 2020, 17:07:14 UTC |
| 3eba73d | Valentin Lorentz | 06 November 2020, 11:57:02 UTC | migrate_extrinsic_metadata: Remove log output when a CRAN origin is missing as this happens quite often and isn't an error. | 09 November 2020, 15:32:20 UTC |
| f3652a9 | Valentin Lorentz | 06 November 2020, 11:55:58 UTC | migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames. | 09 November 2020, 15:32:20 UTC |
| c0a3d96 | Valentin Lorentz | 06 November 2020, 11:54:14 UTC | migrate_extrinsic_metadata: use the retry proxy Because it makes a lot of get requests and doesn't handle failures, it crashed often. | 09 November 2020, 15:32:20 UTC |
| aded45b | Valentin Lorentz | 06 November 2020, 11:52:50 UTC | Make the retry proxy work on all functions. The metadata migration script kept crashing otherwise. | 09 November 2020, 15:32:20 UTC |
| 2e7d489 | Valentin Lorentz | 05 November 2020, 14:28:49 UTC | Set the value_sanitizer argument of get_journal_writer. The next version of swh-journal will remove the default value. | 09 November 2020, 15:32:20 UTC |
| 24cdc85 | Antoine Lambert | 09 November 2020, 13:13:10 UTC | cassandra: Fix content_missing_per_sha1_git implementation | 09 November 2020, 13:13:10 UTC |
| 886999c | Jenkins for Software Heritage | 05 November 2020, 12:56:53 UTC | Updated debian changelog for version 0.17.1 | 05 November 2020, 12:56:53 UTC |
| a0b1792 | Jenkins for Software Heritage | 05 November 2020, 12:56:53 UTC | Update upstream source from tag 'debian/upstream/0.17.1' Update to upstream version '0.17.1' with Debian dir 1a06182f324d3f658eec3faac9d724b9c9776f3a | 05 November 2020, 12:56:53 UTC |
| d8a6720 | Jenkins for Software Heritage | 05 November 2020, 12:56:51 UTC | New upstream version 0.17.1 | 05 November 2020, 12:56:51 UTC |
| 84984a6 | Valentin Lorentz | 05 November 2020, 11:08:13 UTC | algos.snapshot.snapshot_resolve_alias: Don't return the branch list. It complicates the signature and the code, and we don't have any use for it currently. | 05 November 2020, 11:08:13 UTC |
| fa86834 | Valentin Lorentz | 05 November 2020, 10:28:10 UTC | Add test for snapshot_resolve_alias with a missing branch. | 05 November 2020, 10:57:08 UTC |
| 1826b2b | Valentin Lorentz | 05 November 2020, 10:00:03 UTC | Simplify algos.snapshot.snapshot_resolve_alias. 1. rename branch_info to last_branch 2. exclude the last_branch from 'branches', so that: a) it never needs to contain a None value, so we don't need a cast b) no need for slicing | 05 November 2020, 10:57:08 UTC |
| 328907a | Jenkins for Software Heritage | 03 November 2020, 17:20:46 UTC | Updated debian changelog for version 0.17.0 | 03 November 2020, 17:20:46 UTC |
| 9128c5b | Jenkins for Software Heritage | 03 November 2020, 17:20:45 UTC | Update upstream source from tag 'debian/upstream/0.17.0' Update to upstream version '0.17.0' with Debian dir 170e6e6bb959e47c04b6d85458d4b4076ed71012 | 03 November 2020, 17:20:45 UTC |
| 57956ce | Jenkins for Software Heritage | 03 November 2020, 17:20:44 UTC | New upstream version 0.17.0 | 03 November 2020, 17:20:44 UTC |
| 943e440 | Nicolas Dandrimont | 03 November 2020, 14:59:52 UTC | Rename the `id` argument of raw_extrinsic_metadata_get to `target` Consistently with the new name for this attribute in swh.model 0.7.2. | 03 November 2020, 14:59:52 UTC |
| 48b6dbe | Nicolas Dandrimont | 03 November 2020, 11:46:23 UTC | cassandra/in_memory: rename raw_extrinsic_metadata.id to target For consistency with swh.model v0.7.2, to prepare for the addition of an (intrinsic) id field to RawExtrinsicMetadata objects. | 03 November 2020, 13:56:22 UTC |
| 4fbf481 | Nicolas Dandrimont | 03 November 2020, 10:40:01 UTC | PostgreSQL: rename raw_extrinsic_metadata.id to target For consistency with swh.model v0.7.2, to prepare for the addition of an (intrinsic) id field to RawExtrinsicMetadata objects. | 03 November 2020, 13:56:16 UTC |
| 8b18155 | Antoine Lambert | 27 October 2020, 15:45:11 UTC | algos/snapshot: Add function to resolve branch alias to real target Related to T2734 | 03 November 2020, 11:49:07 UTC |
| 6e3e350 | Valentin Lorentz | 29 October 2020, 10:14:48 UTC | migrate_extrinsic_metadata: Write metadata on directories instead of revisions. To match the new behavior of package loaders. | 29 October 2020, 10:14:48 UTC |
| 97d0b05 | Antoine Lambert | 27 October 2020, 15:39:26 UTC | pre-commit: Fix codespell regexp related error | 27 October 2020, 15:43:36 UTC |
| 9645aef | Nicolas Dandrimont | 27 October 2020, 12:52:22 UTC | Replace RawExtrinsicMetadata `id` attribute with `target`. The old attribute was deprecated in swh.model 0.7.2 | 27 October 2020, 14:37:13 UTC |
| 5819683 | Nicolas Dandrimont | 27 October 2020, 13:09:23 UTC | Update swh.storage.validate for swh.model 0.7.2 swh.model.model.ModelObject.compute_hash was changed to a method instead of a staticmethod. | 27 October 2020, 13:11:34 UTC |
| 4f35f7f | Nicolas Dandrimont | 27 October 2020, 12:47:33 UTC | Add black change on swh.storage.backfill | 27 October 2020, 12:47:33 UTC |
| 474ee72 | Valentin Lorentz | 22 October 2020, 20:35:32 UTC | --amend | 22 October 2020, 20:35:32 UTC |
| eb3952f | Valentin Lorentz | 22 October 2020, 14:19:44 UTC | migrate_extrinsic_metadata: Make pypi_origin_from_filename fix project names when possible using PyPI's API. | 22 October 2020, 14:23:30 UTC |
| b1a3b80 | Valentin Lorentz | 22 October 2020, 13:19:37 UTC | migrate_extrinsic_metadata: move pypi_origin_from_filename to its own function. Instead of bloating handle_row, which is already way too long. | 22 October 2020, 14:22:46 UTC |
| aeb72c7 | Valentin Lorentz | 22 October 2020, 10:32:48 UTC | migrate_extrinsic_metadata: add support for guix revisions | 22 October 2020, 10:32:48 UTC |
| 73dc5e3 | Valentin Lorentz | 22 October 2020, 10:28:18 UTC | migrate_extrinsic_metadata: allow deposits with 'id' missing from their metadata. | 22 October 2020, 10:28:18 UTC |
| c483066 | Valentin Lorentz | 22 October 2020, 10:26:33 UTC | migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames. It now supports all pypi revisions with an id starting with an hex digit from 0 to 5. | 22 October 2020, 10:27:16 UTC |
| 2bfd9fe | Antoine R. Dumont (@ardumont) | 21 October 2020, 11:28:08 UTC | storage.pytest_plugin: Reuse swh.core.db.db_utils postgresql_fact | 22 October 2020, 09:58:03 UTC |
| d93429f | Antoine R. Dumont (@ardumont) | 19 October 2020, 13:06:36 UTC | api.server: Add missing coverage on make_app_from_configfile factory This is actually what starts the server, so it sounds more reasonable to test that part. | 19 October 2020, 13:08:01 UTC |
| ca8e6aa | Antoine R. Dumont (@ardumont) | 19 October 2020, 13:07:39 UTC | api.server: Drop the % in the error message | 19 October 2020, 13:08:01 UTC |
| 49d787c | Antoine R. Dumont (@ardumont) | 16 October 2020, 13:03:38 UTC | storage.api.server: Add type to load_and_check_config then refactor tests This also drops the type parameter from load_and_check_config which is never used. | 16 October 2020, 13:57:11 UTC |
| 1a9687f | Valentin Lorentz | 29 September 2020, 13:54:55 UTC | backfill: use get_journal_writer instead if instantiating JournalWriter directly. A future version of swh-journal will add a mandatory argument to JournalWriter, whic get_journal_writer sets by default. | 12 October 2020, 17:21:09 UTC |
| b425b5c | Valentin Lorentz | 12 October 2020, 12:57:25 UTC | migrate_extrinsic_metadata: add support for the new deposit metadata formats introduced in late september. * https://forge.softwareheritage.org/D4065 * https://forge.softwareheritage.org/D4105 | 12 October 2020, 13:07:27 UTC |
| cd07c36 | Jenkins for Software Heritage | 09 October 2020, 16:33:11 UTC | Updated debian changelog for version 0.16.0 | 09 October 2020, 16:33:11 UTC |
| d97d189 | Jenkins for Software Heritage | 09 October 2020, 16:33:10 UTC | Update upstream source from tag 'debian/upstream/0.16.0' Update to upstream version '0.16.0' with Debian dir fe2380c191d3dfe05572b51a4bc92614491ed390 | 09 October 2020, 16:33:10 UTC |
| aade84f | Jenkins for Software Heritage | 09 October 2020, 16:33:09 UTC | New upstream version 0.16.0 | 09 October 2020, 16:33:09 UTC |
| a11d58a | Nicolas Dandrimont | 09 October 2020, 15:29:10 UTC | Remove a bunch of deprecated instances of `args` in configurations Notably, `get_objstorage`'s `args` has been deprecated as of swh.objstorage 0.2.2. | 09 October 2020, 15:29:10 UTC |
| a085b7e | Nicolas Dandrimont | 08 October 2020, 18:34:35 UTC | backfill: use the common `storage` top-level config key This makes the backfiller configuration compatible with all other modules. | 08 October 2020, 18:35:49 UTC |
| dceeb74 | Nicolas Dandrimont | 08 October 2020, 16:32:35 UTC | backfill: support arbitrary journal writer configuration This allows more settings than the previous hardcoded three, e.g. the `privileged` flag to backfill a journal containing anonymous topics. | 08 October 2020, 18:35:49 UTC |
| a6af589 | Valentin Lorentz | 01 October 2020, 18:03:44 UTC | retry: don't retry on keyboardinterrupt. Otherwise, Ctrl-C is ignored if pressed while sending a request. | 02 October 2020, 09:07:09 UTC |
| 889bd87 | Valentin Lorentz | 01 October 2020, 17:57:29 UTC | migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames. It now supports all pypi revisions with an id starting with 0, 1, or 2. | 02 October 2020, 09:07:09 UTC |
| 9ddbb69 | Valentin Lorentz | 01 October 2020, 17:54:57 UTC | migrate_extrinsic_metadata: allow dash in deposit client and collection names. | 02 October 2020, 09:07:09 UTC |
| 59e7e68 | Valentin Lorentz | 01 October 2020, 17:53:53 UTC | migrate_extrinsic_metadata: update name of column deposit.swhid_context. It was renamed in 4d72d1be529a568784842f5c0864e862a4b4705c. | 02 October 2020, 09:07:09 UTC |
| 07df3f6 | Valentin Lorentz | 30 September 2020, 11:07:49 UTC | migrate_extrinsic_metadata: Add support for the current format of original_artifacts written by the CRAN loader. | 02 October 2020, 09:07:09 UTC |
| bef08d6 | tenma | 28 September 2020, 13:59:39 UTC | Fix object_types default in buffer interface protocol and impls Default argument object_types was not properly declared in StorageInterface and concrete implmentations PostgreSQL and Cassandra. Reverted unnecessary fix in storage tests. | 30 September 2020, 09:26:12 UTC |
| 40997c0 | Valentin Lorentz | 26 September 2020, 06:10:09 UTC | migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames. It now supports all pypi revisions with an id starting with 0 or 1. | 29 September 2020, 12:36:24 UTC |
| e37c8f7 | Nicolas Dandrimont | 28 September 2020, 13:37:43 UTC | Pin black in tox to the same version as .pre-commit-config.yaml | 28 September 2020, 13:37:43 UTC |
| c812c79 | Valentin Lorentz | 18 September 2020, 15:50:26 UTC | migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames. | 26 September 2020, 06:04:52 UTC |
| 0adb8fc | Nicolas Dandrimont | 24 September 2020, 17:50:42 UTC | Add a regression test for the buffer proxy default settings This is used by swh.loader.core, regressed in v0.15.0 but wasn't caught by local tests. | 25 September 2020, 15:14:16 UTC |
| dd5fb8d | Nicolas Dandrimont | 25 September 2020, 15:12:16 UTC | Drop vcversioner from requirements We stopped using it months ago. | 25 September 2020, 15:14:16 UTC |
| 632e99e | tenma | 24 September 2020, 13:59:55 UTC | Add static check to object_type literals in buffers | 25 September 2020, 12:23:30 UTC |
| a75c5ca | tenma | 24 September 2020, 13:48:41 UTC | Improve typing of the buffering interface - use more generic collection types, so that parametrized types can be made stricter (e.g. str, in the next revision) - remove Optionals that are not needed and provide better defaults | 25 September 2020, 12:23:30 UTC |
| e8f1136 | Nicolas Dandrimont | 25 September 2020, 12:19:21 UTC | Run isort after the CLI import changes | 25 September 2020, 12:19:21 UTC |
| 19167e1 | Jenkins for Software Heritage | 24 September 2020, 18:24:14 UTC | Updated debian changelog for version 0.15.3 | 24 September 2020, 18:24:14 UTC |
| 55d1e74 | Jenkins for Software Heritage | 24 September 2020, 18:24:13 UTC | Update upstream source from tag 'debian/upstream/0.15.3' Update to upstream version '0.15.3' with Debian dir 44405d462345831220c37a2a71fff14720013476 | 24 September 2020, 18:24:13 UTC |
| c0440b0 | Jenkins for Software Heritage | 24 September 2020, 18:24:12 UTC | New upstream version 0.15.3 | 24 September 2020, 18:24:12 UTC |
| ac3c537 | Nicolas Dandrimont | 24 September 2020, 17:50:12 UTC | Update sql paths for the moved SQL files This should fix the currently failing documentation build. | 24 September 2020, 18:08:56 UTC |
| 68b6733 | Jenkins for Software Heritage | 24 September 2020, 17:28:22 UTC | Updated debian changelog for version 0.15.2 | 24 September 2020, 17:28:22 UTC |
| 370ad96 | Jenkins for Software Heritage | 24 September 2020, 17:28:22 UTC | Update upstream source from tag 'debian/upstream/0.15.2' Update to upstream version '0.15.2' with Debian dir 7efa6a39fe821feba8691b02fed4b6355ed4d6ea | 24 September 2020, 17:28:22 UTC |
| 20808a4 | Jenkins for Software Heritage | 24 September 2020, 17:28:20 UTC | New upstream version 0.15.2 | 24 September 2020, 17:28:20 UTC |
| f677b84 | Jenkins for Software Heritage | 24 September 2020, 16:44:22 UTC | Updated debian changelog for version 0.15.1 | 24 September 2020, 16:44:22 UTC |
| cdf972d | Jenkins for Software Heritage | 24 September 2020, 16:44:21 UTC | Update upstream source from tag 'debian/upstream/0.15.1' Update to upstream version '0.15.1' with Debian dir 7a4e6e501403f48373a18b995786d9b017dcd3c6 | 24 September 2020, 16:44:21 UTC |
| ee6041c | Jenkins for Software Heritage | 24 September 2020, 16:44:20 UTC | New upstream version 0.15.1 | 24 September 2020, 16:44:20 UTC |
| 96be9bd | tenma | 24 September 2020, 12:29:15 UTC | Fix default value handling in constructor Use a more simple default value and do not identity check against it. | 24 September 2020, 16:33:09 UTC |
| 02d9d0d | Jenkins for Software Heritage | 24 September 2020, 15:03:58 UTC | Updated debian changelog for version 0.15.0 | 24 September 2020, 15:03:58 UTC |
| 7aea535 | Jenkins for Software Heritage | 24 September 2020, 15:03:57 UTC | Update upstream source from tag 'debian/upstream/0.15.0' Update to upstream version '0.15.0' with Debian dir d2dad0b2426ce586558c43b33f25576ab6825d82 | 24 September 2020, 15:03:57 UTC |
| e6808fa | Jenkins for Software Heritage | 24 September 2020, 15:03:56 UTC | New upstream version 0.15.0 | 24 September 2020, 15:03:56 UTC |
| 829118a | Nicolas Dandrimont | 24 September 2020, 10:44:17 UTC | Add the SQL commands used to set up the logical replication publication | 24 September 2020, 11:57:18 UTC |
| 5d3de06 | Nicolas Dandrimont | 17 September 2020, 17:55:58 UTC | Support different database flavors in the SQL scripts This uses a new database table and some psql conditionals to introduce three different flavors for the swh.storage Postgres database: - the 'default' flavor has all the deduplication features, foreign keys and read indexes - the 'mirror' flavor has all the deduplication features and read indexes; it drops some foreign keys to allow for out of order addition of some object types - the 'read_replica' flavor has the minimal set of indexes to support read queries, and replication using the PostgreSQL logical replication feature Related to T2604. | 24 September 2020, 11:57:14 UTC |
| 63426e6 | Nicolas Dandrimont | 17 September 2020, 17:53:45 UTC | pytest_plugin: Use psql to load SQL files instead of connecting with psycopg2 This avoids running into issues when the SQL files contain psql-specific features like backslash-escapes. | 24 September 2020, 11:54:38 UTC |
| 38b1dbf | Nicolas Dandrimont | 24 September 2020, 11:53:55 UTC | Output a warning when the version of the database is different than expected | 24 September 2020, 11:54:38 UTC |
| e37f639 | tenma | 22 September 2020, 15:57:24 UTC | Improve code quality and doc in BufferedProxyStorage - better names related to the object buffers - extracted parameter dicts from the constructor - used more generic typing in function parameters and more specific in other contexts in order to apply the principle of robustness | 23 September 2020, 22:21:54 UTC |
| c97b23b | David Douard | 16 September 2020, 07:03:13 UTC | Adapt cli declaration entrypoint to swh.core 0.3 | 23 September 2020, 14:13:01 UTC |
| 924621f | Antoine R. Dumont (@ardumont) | 23 September 2020, 09:56:34 UTC | pytest_plugin: Order the fixture definitions in dependency order | 23 September 2020, 10:30:55 UTC |
| 6286e18 | Antoine R. Dumont (@ardumont) | 23 September 2020, 09:04:43 UTC | pytest_plugin: Change dbname to storage to avoid clash in tests Other similar fixtures in other modules which use the same "tests" db already. Clash can then happen when table names exists in different modules (e.g. dbversion exist both in scheduler and storage dbs). | 23 September 2020, 10:28:52 UTC |
| 8c44a29 | Antoine R. Dumont (@ardumont) | 23 September 2020, 10:24:04 UTC | pytest_plugin: Reuse swh_storage_postgresql connection string The `swh_storage_postgresql.dsn` string already contains the connection information necessary for the tests to run. | 23 September 2020, 10:27:02 UTC |
| 30cdb78 | David Douard | 22 September 2020, 08:18:07 UTC | Drop the -swh- part of sql files it does not bring any meaningful info and makes it somewhat inconsistent with the new -superuser- "tag". | 22 September 2020, 08:18:07 UTC |
| 915575d | David Douard | 22 September 2020, 08:14:15 UTC | Rename 10-swh-init.sql as 10-superuser-init.sql so the db initialization from swh.core (>= 0.3) executes this during the database creation step (i.e. while having a superuser level connection to the database). | 22 September 2020, 08:14:15 UTC |
| 67ee86b | David Douard | 18 September 2020, 14:31:35 UTC | Warn about skipped_content sneaking the 'content' topics | 21 September 2020, 08:43:00 UTC |
| 8de6564 | David Douard | 18 September 2020, 14:30:37 UTC | Small fix in the graph replayer to prevent a wrong warning | 18 September 2020, 14:30:37 UTC |
| 61e880d | Jenkins for Software Heritage | 17 September 2020, 16:53:57 UTC | Updated debian changelog for version 0.14.3 | 17 September 2020, 16:53:57 UTC |
| 7045a62 | Jenkins for Software Heritage | 17 September 2020, 16:53:55 UTC | Update upstream source from tag 'debian/upstream/0.14.3' Update to upstream version '0.14.3' with Debian dir 66526310514244ae6c614cfdc030860320fde0eb | 17 September 2020, 16:53:55 UTC |
| 8387ef5 | Jenkins for Software Heritage | 17 September 2020, 16:53:52 UTC | New upstream version 0.14.3 | 17 September 2020, 16:53:52 UTC |
| b0027ab | Antoine Lambert | 17 September 2020, 16:05:32 UTC | python: Reorder imports with isort Related to T2610 | 17 September 2020, 16:06:07 UTC |
| d27a046 | Antoine Lambert | 17 September 2020, 15:06:53 UTC | pre-commit: Add isort hook and configuration Related to T2610 | 17 September 2020, 16:06:06 UTC |
