| ea2f9f4 | Jenkins for Software Heritage | 05 August 2022, 08:44:39 UTC | New upstream version 1.5.0 | 05 August 2022, 08:44:39 UTC |
| 4825f40 | Valentin Lorentz | 05 August 2022, 08:08:01 UTC | Fix flakiness in test_directory_add_get_arbitrary By ignoring other attributes when raw_manifest is not None; just like we already do in test_revision_add_get_arbitrary and test_release_add_get_arbitrary. | 05 August 2022, 08:08:01 UTC |
| fc89059 | Valentin Lorentz | 04 August 2022, 14:02:47 UTC | Stop logging and sending postgresql timeouts to Sentry They are very noisy, and clients are expected to retry a few times before re-raising the exception on their side. | 04 August 2022, 14:02:47 UTC |
| 1e7ede1 | Valentin Lorentz | 04 August 2022, 11:53:29 UTC | Stop using `USE <keyspace>` with prepared statements This caused the following warning: ``` WARNING cassandra.protocol:libevreactor.py:361 Server warning: `USE <keyspace>` with prepared statements is considered to be an anti-pattern due to ambiguity in non-qualified table names. Please consider removing instances of `Session#setKeyspace(<keyspace>)`, `Session#execute("USE <keyspace>")` and `cluster.newSession(<keyspace>)` from your code, and always use fully qualified table names (e.g. <keyspace>.<table>). ``` This also prepends 'test' to the name of keyspaces used in tests, so they are guaranteed to start with an letter (starting with digits cause syntax errors in most statements). | 04 August 2022, 11:53:29 UTC |
| 0aff461 | Valentin Lorentz | 04 August 2022, 11:51:16 UTC | cassandra: Simplify SELECT statement formatting | 04 August 2022, 11:51:16 UTC |
| 2205fa6 | Valentin Lorentz | 04 August 2022, 09:55:28 UTC | Add test_directory_add_raw_manifest__different_entries This reproduces what I think is the issue found in https://jenkins.softwareheritage.org/job/debian/job/packages/job/DSTO/job/gbp-buildpackage/423/consoleFull This does not fix the issue as it is a consequence of the design, but documents this problematic behavior. | 04 August 2022, 09:55:40 UTC |
| fad99cc | Jenkins for Software Heritage | 04 August 2022, 08:13:22 UTC | New upstream version 1.4.2 | 04 August 2022, 08:13:22 UTC |
| fbe3803 | Antoine Lambert | 13 July 2022, 14:44:13 UTC | postgresql: Increase some timeouts to get origin visits Even if missing index to speedup origin visit queries has been added to replica database, the configured timeouts for origin_visit_get_with_statuses and origin_visit_find_by_date were still too low to avoid query timeouts in production. After performing some tests locally, bumping them to 2000ms makes the timeouts go away. Related to T4386 | 13 July 2022, 14:44:35 UTC |
| cfc8679 | Valentin Lorentz | 06 July 2022, 11:37:00 UTC | backfill: Add support for directories with duplicated entries This uses Directory.from_possibly_duplicated_entries() to mangle entry names instead of crashing. | 12 July 2022, 12:26:13 UTC |
| d6db4e4 | David Douard | 08 July 2022, 11:57:44 UTC | cli: move an import statement in the cli command | 08 July 2022, 11:57:44 UTC |
| e0825ac | David Douard | 01 July 2022, 15:46:48 UTC | do not always auto-create an OriginVisitStatus object in origin_visit_add() when the OriginVisit object given as argument to be inserted already have its visit id set (which is usually the case in a replayer-like session), it makes no sense to auto-add the first OriginVisitStatus objects related to this visit; this behavior is expected only when the origin_visit_add() is called from a loading session. Adapt tests accordingly -- several tests did depend on the auto-add behavior of the origin_visit_add method for OriginVisit objects which visit_id is given in the test dataset. | 06 July 2022, 15:28:47 UTC |
| 47caf04 | David Douard | 01 July 2022, 12:56:47 UTC | Add a Storage.flavor property to the postgresql backend and add tests for 'mirror' and 'read_replica' flavors. | 01 July 2022, 13:03:43 UTC |
| a00650e | David Douard | 01 July 2022, 12:50:28 UTC | Update pytest_plugin for swh.core 2.10 | 01 July 2022, 13:03:41 UTC |
| 5b366ae | Jenkins for Software Heritage | 03 June 2022, 15:25:17 UTC | New upstream version 1.4.1 | 03 June 2022, 15:25:17 UTC |
| c19f53f | Antoine R. Dumont (@ardumont) | 03 June 2022, 09:35:32 UTC | Set current_version attribute to postgresql datastore This also simplifies the db collaborator code reusing core.db functions to check the code version and the actual db version matches. Related to T4305 | 03 June 2022, 13:40:56 UTC |
| e64d64e | David Douard | 31 May 2022, 10:37:04 UTC | pytest_plugin: use the stock pytest_postgresql postgresql factory instead of swh-core's postgresql_fact one, since we actually do not use its custom features any more in swh-storage. | 31 May 2022, 16:41:53 UTC |
| a936cfd | David Douard | 31 May 2022, 11:35:36 UTC | Add missing __init__.py in proxies/ | 31 May 2022, 11:35:36 UTC |
| cb12394 | Valentin Lorentz | 10 May 2022, 12:05:02 UTC | docs: Describe metadata formats more precisely, and mention github and gitlab's | 10 May 2022, 12:05:12 UTC |
| 27d3c8a | Pratyush Desai | 09 May 2022, 10:14:18 UTC | add strict asyncio_mode in pytest.ini | 09 May 2022, 10:14:18 UTC |
| d0af35e | Jenkins for Software Heritage | 03 May 2022, 09:55:04 UTC | New upstream version 1.4.0 | 03 May 2022, 09:55:04 UTC |
| 9562953 | Valentin Lorentz | 02 May 2022, 10:18:27 UTC | Add function storage.algos.directory.directory_get It will be used i swh-storage to fetch a complete directory object, ie. with the raw_manifest and all branches. | 02 May 2022, 10:26:23 UTC |
| fb55141 | Antoine R. Dumont (@ardumont) | 28 April 2022, 10:23:22 UTC | client: Migrate to _post method call to stop deprecation warnings | 28 April 2022, 10:23:22 UTC |
| a942679 | Valentin Lorentz | 26 April 2022, 11:06:09 UTC | Bump mypy to v0.942 | 26 April 2022, 11:06:14 UTC |
| 2c45e26 | Jenkins for Software Heritage | 25 April 2022, 12:33:50 UTC | New upstream version 1.3.2 | 25 April 2022, 12:33:50 UTC |
| e6e658e | Antoine Lambert | 21 April 2022, 11:39:50 UTC | pre-commit: Remove codespell commit-msg hook That hook can be frustrating as it can discard a long commit message if it finds a typo in it so better removing it. | 21 April 2022, 11:39:50 UTC |
| f136559 | David Douard | 14 April 2022, 10:13:00 UTC | User logger everywhere in tenacious.py | 14 April 2022, 10:13:58 UTC |
| bbb4fc1 | Nicolas Dandrimont | 13 April 2022, 13:37:57 UTC | retry: re-raise original exception instead of a RetryError This will make the sentry reports more usable. If the exception changes across calls, the earlier exceptions are still logged and available as breadcrumbs. | 13 April 2022, 13:37:57 UTC |
| fa5bc0f | Jenkins for Software Heritage | 12 April 2022, 12:48:01 UTC | New upstream version 1.3.1 | 12 April 2022, 12:48:01 UTC |
| 75aa073 | Nicolas Dandrimont | 12 April 2022, 11:53:43 UTC | postgresql: ensure origin_visit(_status) queries use index When using an inner join for the single origin value, instead of a subquery, the query fails to use the (origin, visit) indexes and falls back to fetching all the visits (or all the statuses) for the origin and sorting them. This breaks down for origins with a lot of visits, such as the ones that are being used for end to end monitoring. Using a subselect to generate a single origin id value ensures that the queries can use the proper indexes. | 12 April 2022, 12:21:50 UTC |
| 035d4c1 | Antoine Lambert | 12 April 2022, 10:15:41 UTC | origin_get_with_statuses: Rename RPC endpoint path Align RPC endpoint path with method name. | 12 April 2022, 10:15:52 UTC |
| d255fb3 | Antoine Lambert | 12 April 2022, 10:06:14 UTC | origin_get_with_statuses: Fix case when fetched visits list is empty Ensure to return an empty list of results when fetched visits list is empty, for instance when the provided page_token is greater or equal to latest origin visit. Related to T4090 | 12 April 2022, 10:06:28 UTC |
| b655efc | Jenkins for Software Heritage | 11 April 2022, 12:36:59 UTC | New upstream version 1.3.0 | 11 April 2022, 12:36:59 UTC |
| 7c9586a | Antoine Lambert | 08 April 2022, 13:15:36 UTC | Add .git-blame-ignore-revs file with automatic reformatting commits | 08 April 2022, 13:15:36 UTC |
| b146bb7 | Antoine Lambert | 08 April 2022, 13:15:09 UTC | python: Reformat code with black 22.3.0 Related to T3922 | 08 April 2022, 13:15:09 UTC |
| 93aecb6 | Antoine Lambert | 08 April 2022, 13:13:51 UTC | pre-commit, tox: Bump black from 19.10b0 to 22.3.0 black is considered stable since release 22.1.0 and the version we are currently using is quite outdated and not compatible with click 8.1.0, so it is time to bump it to its latest stable release. Please note that E501 pycodestyle warning related to line length is replaced by B950 one from flake8-bugbear as recommended by black. https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length Related to T3922 | 08 April 2022, 13:13:51 UTC |
| 567c8e4 | Antoine Lambert | 28 March 2022, 12:33:53 UTC | interface: Add new method origin_visit_get_with_statuses It enables to retrieve in an efficient and paginated way the list of visits and all their statuses for a given origin. Previously, it was required to call origin_visit_status_get on each visit of the origin to get such list. Related to T4090 | 07 April 2022, 14:47:06 UTC |
| 812590c | Antoine Lambert | 06 April 2022, 15:14:53 UTC | requirements-test: Remove pytest pinning to < 7 pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for pytest >= 7 so we can now drop the pytest pinning. | 06 April 2022, 15:14:53 UTC |
| c6dc5cd | David Douard | 30 March 2022, 12:39:28 UTC | Make postgresql's Storage client options configurable from config Adding a `query_options` member to postgresql's Storage, in conjunction with swh.core >= 2.5, allows to set/overwrite SQL client options from the storage configuration file. Default values are set, as they used to be, from the decorator arguments. But in addition to this, one can overload these value at run time from the storage configuration file. For example: .. code-block:: yaml storage: cls: postgresql db: testdb objstorage: cls: memory query_options: directory_ls: statement_timeout: 180000 will provide a Storage instance for which the timeout value for the `directory_ls` endpoint is 3mn (instead of the default 20s). | 04 April 2022, 14:43:17 UTC |
| 5029870 | Jenkins for Software Heritage | 23 March 2022, 16:00:30 UTC | New upstream version 1.2.0 | 23 March 2022, 16:00:30 UTC |
| 835feb6 | Nicolas Dandrimont | 23 March 2022, 15:29:59 UTC | Fix tenacious storage tests for swh.model 6 The logic for testing the tenacious storage proxy by interspersing bad objects with good ones triggers when the object list is larger than 3 items. However, the allowed error rate of 1 failure for a window of 3 objects would only work for lists larger than six objects (putting at least 2 good objects between each bad object). swh.model 6 made the directory, revision and release test object lists 3 object long, triggering the buggy code. Reducing the window size to 2 objects makes the test logic works for lists of 3 or more objects. | 23 March 2022, 15:34:50 UTC |
| 6057aa8 | Jenkins for Software Heritage | 23 March 2022, 15:01:45 UTC | New upstream version 1.1.0 | 23 March 2022, 15:01:45 UTC |
| 6fdaf8a | Valentin Lorentz | 21 March 2022, 14:46:43 UTC | Remove typing workarounds for Revision.author or Revision.committer being None swh-model 6.0.0 adds proper support for them. + fix issue found by mypy | 23 March 2022, 10:01:34 UTC |
| 3eff720 | Valentin Lorentz | 21 March 2022, 14:18:20 UTC | Add support for author=None and committer=None committer=None happens on some malformed commits generated by old dgit version; and it is possible for author=None to happen for the same reason. For now, this is not supported by swh-model, so tests temporarily disable attrs checks that swh-model relies on. | 23 March 2022, 10:00:22 UTC |
| 92c78ab | Antoine Lambert | 22 March 2022, 10:58:26 UTC | pytest: Exclude build directory for tests discovery Due to test modules being copied in subdirectories of the build directory by setuptools, it makes pytest fail by raising ImportPathMismatchError exceptions when invoked from root directory of the module. So ignore the build folder to discover tests. | 22 March 2022, 10:58:26 UTC |
| 98b41c8 | Valentin Lorentz | 16 March 2022, 11:24:33 UTC | backfill: Add missing raw_manifest to directories This was not covered by tests so far, because swh.model.tests.swh_model_data.TEST_OBJECTS did not contain any object with a raw_manifest. But it will in swh-model > 5.0.0 | 16 March 2022, 11:24:46 UTC |
| 8b65e42 | Valentin Lorentz | 15 March 2022, 15:56:40 UTC | backfill: Make integer_ranges() work on str args + add typing to RANGE_GENERATORS Without the type annotation, mypy errors with 'Cannot call function of unknown type' when called from a type-checked function. | 15 March 2022, 16:19:22 UTC |
| 77f7e6d | Valentin Lorentz | 15 March 2022, 12:36:33 UTC | postgresql: Remove unused listener code from db.py | 15 March 2022, 12:36:42 UTC |
| ccde097 | Valentin Lorentz | 11 March 2022, 12:38:51 UTC | origin_visit_get_latest: Order by visit id instead of date This allows both the postgresql and cassandra backends to make efficient queries by using an index (resp. clustering key) instead of scanning all visits of the given origin then sorting by date. This does not affect the results for the last majority of cases, as ids are always in increasing chronological, unless an origin was re-loaded from an old archive. | 11 March 2022, 12:43:36 UTC |
| 600e87f | Valentin Lorentz | 11 March 2022, 12:36:07 UTC | origin_visit_get_latest: Materialize subquery on 'origin' table. postgresql's query planner does not understand the origin is unique, so it performs a partial index scan on origin_visit_pkey, which is inefficient on origins with many visits. This commit itself is not enough to make it use the proper index, but provides this necessary change that will be used by a future commit. | 11 March 2022, 12:36:07 UTC |
| b0cdab5 | Valentin Lorentz | 11 March 2022, 10:36:01 UTC | postgresql: Increase timeouts that often fail According to Sentry, in the last 30 days: * directory_entry_get_by_path: 958 events, https://sentry.softwareheritage.org/share/issue/c4c2124953a145b2bd325f6f6b7df5a6/ * revision_get: 841 events, https://sentry.softwareheritage.org/share/issue/55fbe01c6f4d4c9bbf684c7608a62ad9/ * release_get: 14 events, https://sentry.softwareheritage.org/share/issue/37c53354541b4c4eaa1faf4e20a68418/ * origin_visit_find_by_date: 114 events, https://sentry.softwareheritage.org/share/issue/a674c12049a941968a717661a0226559/ * origin_get: 79 events, https://sentry.softwareheritage.org/share/issue/bf21d6bc7b24442eb18643d80d936d27/ ; 67 events, https://sentry.softwareheritage.org/share/issue/010a4b1e085a4e2089ba4897c6de6038/ | 11 March 2022, 12:19:28 UTC |
| 4e78014 | David Douard | 08 March 2022, 15:22:49 UTC | Remove aiohttp from requirements.txt it's not used by swh.storage. | 08 March 2022, 15:29:15 UTC |
| 284a4ab | Valentin Lorentz | 02 March 2022, 11:17:07 UTC | Move metrics handling from backends to RPC server Motivation: replaces code duplication in the backends with a single one, to be consistent with the objstorage (which has many more backends) This also fixes the issue of metrics from 'extid_add' to be missing when using the postgresql storage. | 02 March 2022, 11:32:18 UTC |
| 35e7f0c | Jenkins for Software Heritage | 24 February 2022, 11:13:55 UTC | New upstream version 1.0.0 | 24 February 2022, 11:13:55 UTC |
| 215162b | David Douard | 02 February 2022, 14:56:18 UTC | Update for swh.core 2.0.0 - Add expected entry points for swh.core 2 db handling new features: - add a ``swh.storge.get_datastore()`` function - add ``swh.storage.postgreql.storage.Storage.get_current_version()`` method - move sql migration scripts in ``swh/storage/sql/upgrades`` - modify sql initialization scripts to match swh.core 2 (remove dbversion management code). - Update tests to use the new template-based database handling; this should have only minimal impact on test execution performances. | 24 February 2022, 10:05:01 UTC |
| 386fb4d | David Douard | 24 February 2022, 10:00:49 UTC | Add types-toml to requirements-test.txt | 24 February 2022, 10:05:01 UTC |
| f578377 | Antoine Lambert | 10 February 2022, 16:28:19 UTC | pre-commit: Bump hooks and add new one to check commit message spelling To install the new hook: $ pre-commit install -t commit-msg | 10 February 2022, 16:29:28 UTC |
| 10d367f | Jenkins for Software Heritage | 08 February 2022, 09:31:31 UTC | New upstream version 0.43.1 | 08 February 2022, 09:31:31 UTC |
| 8bf07c3 | Valentin Lorentz | 04 February 2022, 15:45:53 UTC | revision_walker: Actually pass ignore_displaynam to revision_log I somehow forgot to stage this change in the previous commit. | 04 February 2022, 15:46:07 UTC |
| 0f9f54b | Valentin Lorentz | 03 February 2022, 10:51:24 UTC | revision_walker: Add support for ignore_displayname. This is needed by the vault. | 04 February 2022, 15:14:59 UTC |
| 75a7f09 | Valentin Lorentz | 03 February 2022, 10:50:25 UTC | Add typing to revision_walker.py and make the state a dataclass | 04 February 2022, 15:14:53 UTC |
| a3a63d8 | Valentin Lorentz | 04 February 2022, 14:01:47 UTC | Require pytest to be <7.0.0 | 04 February 2022, 14:02:23 UTC |
| ec42d59 | Jenkins for Software Heritage | 02 February 2022, 17:56:31 UTC | New upstream version 0.43.0 | 02 February 2022, 17:56:31 UTC |
| 4544d7c | Nicolas Dandrimont | 31 January 2022, 17:46:12 UTC | Introduce a new displayname field for persons in the PostgreSQL storage Extend the APIs for Revisions and Releases to honor the field by default, unless the new `ignore_displayname` argument is set. | 01 February 2022, 12:03:52 UTC |
| 97caa93 | Nicolas Dandrimont | 01 February 2022, 12:01:15 UTC | Make test_release_add_get_arbitrary non-flaky It was made flaky by d4ddd41535d0ce1cd50d51d297e154bf0ab6e649. | 01 February 2022, 12:03:52 UTC |
| f868f3c | Nicolas Dandrimont | 31 January 2022, 19:09:43 UTC | Mostly use normalized Person objects in tests This opens up the possibility of eventually ignoring the `name` and `email` fields stored in database in favor of parsing them again from the fullname field (and therefore to update our parsing logic without having to affect stored data). | 31 January 2022, 20:56:47 UTC |
| d4ddd41 | Nicolas Dandrimont | 31 January 2022, 19:08:04 UTC | postgresql: Use Person.from_fullname if name and email are None This allows us to populate sensible name and email values out of the new displayname field, without having to store them. | 31 January 2022, 19:12:09 UTC |
| bedc372 | Jenkins for Software Heritage | 25 January 2022, 15:54:44 UTC | New upstream version 0.42.0 | 25 January 2022, 15:54:44 UTC |
| 6f02524 | Valentin Lorentz | 24 January 2022, 11:52:04 UTC | Fix directory_add to actually insert the manifest + add directory_get_raw_manifest I don't expect directory_get_raw_manifest to be used, but it is needed for tests, so why not. | 25 January 2022, 09:46:27 UTC |
| 5874905 | Valentin Lorentz | 21 January 2022, 12:01:42 UTC | Stop using the deprecated 'TimestampWithTimezone.offset' attribute It will be replaced by what is currently called 'offset_bytes' | 21 January 2022, 14:01:31 UTC |
| 2e74138 | Valentin Lorentz | 12 January 2022, 15:36:24 UTC | Remove 'offset' and 'negative_utc' This only keeps 'offset_bytes' to store the timezone, to support swh-model v5.0.0. However, this keeps writing 'offset' and 'negative_utc' to the postgresql database, just in case we need to roll back this change. But they are not read anymore. | 21 January 2022, 13:07:02 UTC |
| c68a4fd | Valentin Lorentz | 17 January 2022, 15:59:37 UTC | postgres: Add indices to keep track of objects with a raw_manifest They should be a rare occurence, so adding these indices allows us to count and enumerate them without expensive full table scans. | 18 January 2022, 10:04:28 UTC |
| 228de33 | Valentin Lorentz | 18 January 2022, 09:16:01 UTC | Fix sphinx error [2022-01-17T16:03:27.448Z] /var/lib/jenkins/workspace/DSTO/tests-on-diff@2/docs/index.rst:25:hardcoded link 'https://archive.softwareheritage.org/api/' could be replaced by an extlink (try using ':swh_web:`api/`' instead) | 18 January 2022, 09:16:05 UTC |
| 40a57d4 | Valentin Lorentz | 07 January 2022, 12:04:25 UTC | cassandra: Make content_missing run in linear time instead of quadratic Assuming all contents passed to content_missing() have (at least) a missing algo, the function used to iterate over the size of the arg squared in the worst case (when all contents are found). With this commit, it starts with bucketing them by hash, so it does not need to iterate over *all* found contents for each content passed as arg. | 12 January 2022, 10:38:20 UTC |
| d5f1f0e | Valentin Lorentz | 18 October 2021, 11:25:20 UTC | cassandra: Rewrite content_missing to run queries concurrently. This is twice as fast, according to https://forge.softwareheritage.org/T3577#72791 | 12 January 2022, 10:37:57 UTC |
| 4a24505 | Valentin Lorentz | 06 January 2022, 11:41:45 UTC | cassandra: Use concurrent queries in *_missing() instead of naive grouping Instead of grouping ids in queries in arbitrary batches (which forces the server node to coordinate with other nodes to complete the query), this sends queries with one id each, directly to the right node. This is the 'concurrent' algorithm from https://forge.softwareheritage.org/T3577#72791 which gives a >=2x speed-up on directories, and a >=8x speed-up on revisions. | 06 January 2022, 11:43:09 UTC |
| c7cbc9a | Jenkins for Software Heritage | 04 January 2022, 14:55:31 UTC | New upstream version 0.41.1 | 04 January 2022, 14:55:31 UTC |
| 259bf6f | David Douard | 09 December 2021, 12:11:46 UTC | Improve documentation of the replay command | 04 January 2022, 11:02:44 UTC |
| 1071781 | David Douard | 09 December 2021, 12:05:16 UTC | Move the 'error_reporter' config entry in a dedicated 'replayer' section | 04 January 2022, 11:02:44 UTC |
| aa3ce30 | Jenkins for Software Heritage | 22 December 2021, 15:22:15 UTC | New upstream version 0.41.0 | 22 December 2021, 15:22:15 UTC |
| f3232e6 | Valentin Lorentz | 15 December 2021, 18:01:04 UTC | Add columns {,committer_}date_offset to rev/rel and raw_manifest to dir/rev/rel | 22 December 2021, 12:29:11 UTC |
| f09a54d | Antoine R. Dumont (@ardumont) | 16 December 2021, 15:18:36 UTC | Pin mypy and drop type annotations which makes mypy unhappy This also drops: - spurious copyright headers to those files if present. - fix a type issue revealed by the new mypy Related to T3812 | 16 December 2021, 15:18:36 UTC |
| c40ceb3 | Valentin Lorentz | 09 December 2021, 16:27:14 UTC | Add tests checking round-tripping of dir/rev/rel/snp objects generated by Hypothesis | 15 December 2021, 12:11:56 UTC |
| 45687d8 | Valentin Lorentz | 09 December 2021, 16:04:40 UTC | Add test_revision_add_fractional_timezone | 15 December 2021, 12:11:51 UTC |
| fb1b3a0 | Valentin Lorentz | 09 December 2021, 15:25:20 UTC | postgresql: Fix one-by-one error in db_to_date on negative dates Using `int()` on `date.timestamp()` rounded it up (toward zero), but the semantics of `model.Timestamp` is that the actual time is `ts.seconds + ts.microseconds/1000000`, so all negative dates were shifted one second up. In particular, this causes dates from `1969-12-31T23:59:59.000001` to `1969-12-31T23:59:59.999999` (inclusive) to smash into dates from `1970-01-01T00:00:00.000001` to `1970-01-01T00:00:00.999999`, which is how I discovered the issue. | 13 December 2021, 11:54:35 UTC |
| 34ca67e | Valentin Lorentz | 09 December 2021, 15:23:44 UTC | postgresql: Add tests for db_to_date. | 09 December 2021, 15:23:44 UTC |
| 7cb4128 | Antoine Lambert | 08 December 2021, 11:09:04 UTC | proxies/retry: Remove no longer needed tenacity workarounds Now that we have packaged tenacity 6.2 for debian buster and use it in production, we can remove the workarounds to support tenacity < 5. | 08 December 2021, 11:09:15 UTC |
| 615fb99 | Antoine Lambert | 07 December 2021, 12:36:28 UTC | test_cassandra: Fix failing tests since swh-model update Directory entries are now checked for name duplicates in swh-model so we must ensure the CrashyEntry class is properly initialized. Closes T3776 | 07 December 2021, 12:36:43 UTC |
| d8994d5 | Jenkins for Software Heritage | 16 November 2021, 09:45:37 UTC | New upstream version 0.40.0 | 16 November 2021, 09:45:37 UTC |
| 850a755 | David Douard | 27 October 2021, 15:31:18 UTC | Add support for a redis-based reporting for invalid mirrorred objects The idea is that we check the BaseModel validity at journal deserialization time so that we still have access to the raw object from kafka for complete reporting (object id plus raw message from kafka). This uses a new ModelObjectDeserializer class that is responsible for deserializing the kafka message (still using kafka_to_value) then immediately create the BaseModel object from that dict. Its `convert` method is then passed as `value_deserializer` argument of the `JournalClient`. Then, for each deserialized object from kafka, if it's a HashableObject, check its validity by comparing the computed hash with its id. If it's invalid, report the error in logs, and if configured, register the invalid object in via the `reporter` callback. In the cli code, a `Redis.set()` is used a such a callback (if configured). So it simply stores invalid objects using the object id a key (typically its swhid), and the raw kafka message value as value. Related to T3693. | 09 November 2021, 15:36:34 UTC |
| 04bd15a | David Douard | 27 October 2021, 14:56:53 UTC | Refactor fixer.fix_objects() to extract the inner object_fixers dict allowing to use this dict independently of the fix_objects() function. | 09 November 2021, 15:36:34 UTC |
| d655c85 | David Douard | 27 October 2021, 14:55:03 UTC | Remove now useless fixers keep the the fix_objects() function for bw compat for now. | 09 November 2021, 15:36:34 UTC |
| 55eed77 | David Douard | 26 October 2021, 14:38:26 UTC | Add a --type option to 'swh storage replay' allows to choose replayed object types from the cli. | 09 November 2021, 15:36:34 UTC |
| 0262f1c | Valentin Lorentz | 21 October 2021, 11:09:32 UTC | Update extrinsic-metadata-specification.rst to match the current implementation * merged origin and artifact metadata * added metametadata * uses structures instead of dict * removed raw_extrinsic_metadata_get_latest | 09 November 2021, 14:47:31 UTC |
| 3ce052a | Jenkins for Software Heritage | 29 October 2021, 09:17:33 UTC | New upstream version 0.39.0 | 29 October 2021, 09:17:33 UTC |
| a5bfe5b | Antoine Lambert | 27 October 2021, 16:26:20 UTC | interface: Add origin_snapshot_get_all method It enables to return in an efficient way the list of unique snapshot identifiers resulting from the visits of an origin. Previously it was required to query all visits of an origin then query all visit statuses for each visit to extract such information. Introduced method enables to extract origin snaphots information in a single datase query. Related to T3631 | 28 October 2021, 12:23:14 UTC |
| 49a932c | Antoine Lambert | 21 October 2021, 15:12:32 UTC | algos/revisions_walker: Handle case of revision without committer date Some revisions in the archive do not have committer date so workaround it to avoid errors when walking on such revisions when using the class CommitterDateRevisionsWalker. | 22 October 2021, 09:32:28 UTC |
| c02be8e | Antoine Lambert | 21 October 2021, 14:50:57 UTC | test_revisions_walker: Migrate from unittest to pytest | 21 October 2021, 14:50:57 UTC |
| a986710 | Valentin Lorentz | 18 October 2021, 11:00:00 UTC | cassandra: Fix incomplete check of content existence in object_find_by_sha1_git content_missing_by_sha1_git only checks the index and not the main table. This is incorrect, because contents should not be considered written before an entry is written to the main table, even if an entry exists in one of the indexes. | 18 October 2021, 11:10:54 UTC |
| 8567342 | Jenkins for Software Heritage | 11 October 2021, 14:58:20 UTC | New upstream version 0.38.0 | 11 October 2021, 14:58:20 UTC |
| e9fd74d | Valentin Lorentz | 07 October 2021, 09:22:43 UTC | serializers: Prepare rename of 'identifiers_enum' to 'swhids_enum'. This will be done in three steps to avoid any disruption: 1. (this step) add support for decoding the new name, but keep encoding as the old one 2. start encoding as the new name 3. remove support for decoding the old name | 11 October 2021, 11:58:23 UTC |
