https://github.com/SoftwareHeritage/swh-storage

sort by:
Revision Author Date Message Commit Date
de4a842 Updated backport on buster-swh from debian/1.5.1-1_swh1 (unstable-swh) 05 August 2022, 12:27:52 UTC
08cc3cd Merge tag 'debian/1.5.1-1_swh1' into debian/buster-swh 05 August 2022, 12:27:51 UTC
4e53af0 Updated debian changelog for version 1.5.1 05 August 2022, 12:17:23 UTC
5b0437e Update upstream source from tag 'debian/upstream/1.5.1' Update to upstream version '1.5.1' with Debian dir aecb33b9607d268dc35fe135a46a97eab940b3e5 05 August 2022, 12:17:22 UTC
6dba4b5 New upstream version 1.5.1 05 August 2022, 12:17:20 UTC
280ecc3 cassandra: Fix flakiness of test_directory_add_atomic[concurrent] 05 August 2022, 10:24:10 UTC
68b93a6 Fix crash of test_*_arbitrary when given objects with the same id 05 August 2022, 09:39:42 UTC
9b9eb28 cassandra: Make origin_visit_status_get_random's interval consistent with postgresql The postgresql implementation uses '3 months', which is closer to 13 weeks than to 12 weeks. 05 August 2022, 09:20:33 UTC
56f69e5 Fix flakiness of test_origin_visit_status_get_random_nothing_found start is increased from 13 to 14, because 13 weeks is 91 days, ie. 30+31+30; so it is sometimes smaller than 3 months. This was only hit rarely because the number of visits was small, so this commit also increases the number of visits to make the test more likely to fail if it should actually fail. 05 August 2022, 09:18:46 UTC
cb92e90 Updated backport on buster-swh from debian/1.5.0-1_swh1 (unstable-swh) 05 August 2022, 08:55:08 UTC
48b6668 Merge tag 'debian/1.5.0-1_swh1' into debian/buster-swh 05 August 2022, 08:55:07 UTC
7d18537 Updated debian changelog for version 1.5.0 05 August 2022, 08:44:42 UTC
eae94e6 Update upstream source from tag 'debian/upstream/1.5.0' Update to upstream version '1.5.0' with Debian dir 34ae119254de09bf2789c8e454ecae8cb67a614d 05 August 2022, 08:44:41 UTC
ea2f9f4 New upstream version 1.5.0 05 August 2022, 08:44:39 UTC
4825f40 Fix flakiness in test_directory_add_get_arbitrary By ignoring other attributes when raw_manifest is not None; just like we already do in test_revision_add_get_arbitrary and test_release_add_get_arbitrary. 05 August 2022, 08:08:01 UTC
fc89059 Stop logging and sending postgresql timeouts to Sentry They are very noisy, and clients are expected to retry a few times before re-raising the exception on their side. 04 August 2022, 14:02:47 UTC
1e7ede1 Stop using `USE <keyspace>` with prepared statements This caused the following warning: ``` WARNING cassandra.protocol:libevreactor.py:361 Server warning: `USE <keyspace>` with prepared statements is considered to be an anti-pattern due to ambiguity in non-qualified table names. Please consider removing instances of `Session#setKeyspace(<keyspace>)`, `Session#execute("USE <keyspace>")` and `cluster.newSession(<keyspace>)` from your code, and always use fully qualified table names (e.g. <keyspace>.<table>). ``` This also prepends 'test' to the name of keyspaces used in tests, so they are guaranteed to start with an letter (starting with digits cause syntax errors in most statements). 04 August 2022, 11:53:29 UTC
0aff461 cassandra: Simplify SELECT statement formatting 04 August 2022, 11:51:16 UTC
2205fa6 Add test_directory_add_raw_manifest__different_entries This reproduces what I think is the issue found in https://jenkins.softwareheritage.org/job/debian/job/packages/job/DSTO/job/gbp-buildpackage/423/consoleFull This does not fix the issue as it is a consequence of the design, but documents this problematic behavior. 04 August 2022, 09:55:40 UTC
633c08e Updated backport on buster-swh from debian/1.4.2-1_swh1 (unstable-swh) 04 August 2022, 08:23:22 UTC
c1ef03a Merge tag 'debian/1.4.2-1_swh1' into debian/buster-swh 04 August 2022, 08:23:21 UTC
11fde18 Updated debian changelog for version 1.4.2 04 August 2022, 08:13:26 UTC
95fc846 Update upstream source from tag 'debian/upstream/1.4.2' Update to upstream version '1.4.2' with Debian dir 9e4f8fb309e3513a2cc36f0dd6a80fb264910e05 04 August 2022, 08:13:24 UTC
fad99cc New upstream version 1.4.2 04 August 2022, 08:13:22 UTC
fbe3803 postgresql: Increase some timeouts to get origin visits Even if missing index to speedup origin visit queries has been added to replica database, the configured timeouts for origin_visit_get_with_statuses and origin_visit_find_by_date were still too low to avoid query timeouts in production. After performing some tests locally, bumping them to 2000ms makes the timeouts go away. Related to T4386 13 July 2022, 14:44:35 UTC
cfc8679 backfill: Add support for directories with duplicated entries This uses Directory.from_possibly_duplicated_entries() to mangle entry names instead of crashing. 12 July 2022, 12:26:13 UTC
d6db4e4 cli: move an import statement in the cli command 08 July 2022, 11:57:44 UTC
e0825ac do not always auto-create an OriginVisitStatus object in origin_visit_add() when the OriginVisit object given as argument to be inserted already have its visit id set (which is usually the case in a replayer-like session), it makes no sense to auto-add the first OriginVisitStatus objects related to this visit; this behavior is expected only when the origin_visit_add() is called from a loading session. Adapt tests accordingly -- several tests did depend on the auto-add behavior of the origin_visit_add method for OriginVisit objects which visit_id is given in the test dataset. 06 July 2022, 15:28:47 UTC
47caf04 Add a Storage.flavor property to the postgresql backend and add tests for 'mirror' and 'read_replica' flavors. 01 July 2022, 13:03:43 UTC
a00650e Update pytest_plugin for swh.core 2.10 01 July 2022, 13:03:41 UTC
3567976 Updated backport on buster-swh from debian/1.4.1-1_swh1 (unstable-swh) 03 June 2022, 15:40:41 UTC
6c840c8 Merge tag 'debian/1.4.1-1_swh1' into debian/buster-swh 03 June 2022, 15:40:40 UTC
9d8f0d8 Updated debian changelog for version 1.4.1 03 June 2022, 15:25:20 UTC
cf5448f Update upstream source from tag 'debian/upstream/1.4.1' Update to upstream version '1.4.1' with Debian dir dbb991421736a20fe750987037213b265bfc0620 03 June 2022, 15:25:19 UTC
5b366ae New upstream version 1.4.1 03 June 2022, 15:25:17 UTC
c19f53f Set current_version attribute to postgresql datastore This also simplifies the db collaborator code reusing core.db functions to check the code version and the actual db version matches. Related to T4305 03 June 2022, 13:40:56 UTC
e64d64e pytest_plugin: use the stock pytest_postgresql postgresql factory instead of swh-core's postgresql_fact one, since we actually do not use its custom features any more in swh-storage. 31 May 2022, 16:41:53 UTC
a936cfd Add missing __init__.py in proxies/ 31 May 2022, 11:35:36 UTC
cb12394 docs: Describe metadata formats more precisely, and mention github and gitlab's 10 May 2022, 12:05:12 UTC
27d3c8a add strict asyncio_mode in pytest.ini 09 May 2022, 10:14:18 UTC
273538c Updated backport on buster-swh from debian/1.4.0-1_swh1 (unstable-swh) 03 May 2022, 10:09:00 UTC
46332af Merge tag 'debian/1.4.0-1_swh1' into debian/buster-swh 03 May 2022, 10:08:59 UTC
002897e Updated debian changelog for version 1.4.0 03 May 2022, 09:55:07 UTC
2c4770f Update upstream source from tag 'debian/upstream/1.4.0' Update to upstream version '1.4.0' with Debian dir 73644a39fe481a5df44d85383ea7932ad94b8b0e 03 May 2022, 09:55:06 UTC
d0af35e New upstream version 1.4.0 03 May 2022, 09:55:04 UTC
9562953 Add function storage.algos.directory.directory_get It will be used i swh-storage to fetch a complete directory object, ie. with the raw_manifest and all branches. 02 May 2022, 10:26:23 UTC
fb55141 client: Migrate to _post method call to stop deprecation warnings 28 April 2022, 10:23:22 UTC
a942679 Bump mypy to v0.942 26 April 2022, 11:06:14 UTC
62cf989 Updated backport on buster-swh from debian/1.3.2-1_swh1 (unstable-swh) 25 April 2022, 12:48:49 UTC
5ad117d Merge tag 'debian/1.3.2-1_swh1' into debian/buster-swh 25 April 2022, 12:48:48 UTC
d4183fb Updated debian changelog for version 1.3.2 25 April 2022, 12:33:52 UTC
9daf7bc Update upstream source from tag 'debian/upstream/1.3.2' Update to upstream version '1.3.2' with Debian dir 4800aced1470880655cac85fcaa18ec010184687 25 April 2022, 12:33:51 UTC
2c45e26 New upstream version 1.3.2 25 April 2022, 12:33:50 UTC
e6e658e pre-commit: Remove codespell commit-msg hook That hook can be frustrating as it can discard a long commit message if it finds a typo in it so better removing it. 21 April 2022, 11:39:50 UTC
f136559 User logger everywhere in tenacious.py 14 April 2022, 10:13:58 UTC
bbb4fc1 retry: re-raise original exception instead of a RetryError This will make the sentry reports more usable. If the exception changes across calls, the earlier exceptions are still logged and available as breadcrumbs. 13 April 2022, 13:37:57 UTC
72c2b91 Updated backport on buster-swh from debian/1.3.1-1_swh1 (unstable-swh) 12 April 2022, 13:02:20 UTC
20cbf17 Merge tag 'debian/1.3.1-1_swh1' into debian/buster-swh 12 April 2022, 13:02:19 UTC
f66d1e5 Updated debian changelog for version 1.3.1 12 April 2022, 12:48:04 UTC
ac7bd15 Update upstream source from tag 'debian/upstream/1.3.1' Update to upstream version '1.3.1' with Debian dir 396826333823bf05c82e05c3784d0cea79d93835 12 April 2022, 12:48:03 UTC
fa5bc0f New upstream version 1.3.1 12 April 2022, 12:48:01 UTC
75aa073 postgresql: ensure origin_visit(_status) queries use index When using an inner join for the single origin value, instead of a subquery, the query fails to use the (origin, visit) indexes and falls back to fetching all the visits (or all the statuses) for the origin and sorting them. This breaks down for origins with a lot of visits, such as the ones that are being used for end to end monitoring. Using a subselect to generate a single origin id value ensures that the queries can use the proper indexes. 12 April 2022, 12:21:50 UTC
035d4c1 origin_get_with_statuses: Rename RPC endpoint path Align RPC endpoint path with method name. 12 April 2022, 10:15:52 UTC
d255fb3 origin_get_with_statuses: Fix case when fetched visits list is empty Ensure to return an empty list of results when fetched visits list is empty, for instance when the provided page_token is greater or equal to latest origin visit. Related to T4090 12 April 2022, 10:06:28 UTC
303fa7a Updated backport on buster-swh from debian/1.3.0-1_swh1 (unstable-swh) 11 April 2022, 12:51:42 UTC
1792143 Merge tag 'debian/1.3.0-1_swh1' into debian/buster-swh 11 April 2022, 12:51:41 UTC
034c0d2 Updated debian changelog for version 1.3.0 11 April 2022, 12:37:01 UTC
96b96f0 Update upstream source from tag 'debian/upstream/1.3.0' Update to upstream version '1.3.0' with Debian dir 6ba548a311bba2851a07d5071aa45cd4ed4f4e10 11 April 2022, 12:37:00 UTC
b655efc New upstream version 1.3.0 11 April 2022, 12:36:59 UTC
7c9586a Add .git-blame-ignore-revs file with automatic reformatting commits 08 April 2022, 13:15:36 UTC
b146bb7 python: Reformat code with black 22.3.0 Related to T3922 08 April 2022, 13:15:09 UTC
93aecb6 pre-commit, tox: Bump black from 19.10b0 to 22.3.0 black is considered stable since release 22.1.0 and the version we are currently using is quite outdated and not compatible with click 8.1.0, so it is time to bump it to its latest stable release. Please note that E501 pycodestyle warning related to line length is replaced by B950 one from flake8-bugbear as recommended by black. https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length Related to T3922 08 April 2022, 13:13:51 UTC
567c8e4 interface: Add new method origin_visit_get_with_statuses It enables to retrieve in an efficient and paginated way the list of visits and all their statuses for a given origin. Previously, it was required to call origin_visit_status_get on each visit of the origin to get such list. Related to T4090 07 April 2022, 14:47:06 UTC
812590c requirements-test: Remove pytest pinning to < 7 pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for pytest >= 7 so we can now drop the pytest pinning. 06 April 2022, 15:14:53 UTC
c6dc5cd Make postgresql's Storage client options configurable from config Adding a `query_options` member to postgresql's Storage, in conjunction with swh.core >= 2.5, allows to set/overwrite SQL client options from the storage configuration file. Default values are set, as they used to be, from the decorator arguments. But in addition to this, one can overload these value at run time from the storage configuration file. For example: .. code-block:: yaml storage: cls: postgresql db: testdb objstorage: cls: memory query_options: directory_ls: statement_timeout: 180000 will provide a Storage instance for which the timeout value for the `directory_ls` endpoint is 3mn (instead of the default 20s). 04 April 2022, 14:43:17 UTC
fa6c2df Updated backport on buster-swh from debian/1.2.0-1_swh1 (unstable-swh) 23 March 2022, 16:14:20 UTC
d4472a4 Merge tag 'debian/1.2.0-1_swh1' into debian/buster-swh 23 March 2022, 16:14:20 UTC
16a98b9 Updated debian changelog for version 1.2.0 23 March 2022, 16:00:33 UTC
8e2b1a4 Update upstream source from tag 'debian/upstream/1.2.0' Update to upstream version '1.2.0' with Debian dir 3e9154aa3229b4f3f0f8aa34d6b653a3bd67ae99 23 March 2022, 16:00:32 UTC
5029870 New upstream version 1.2.0 23 March 2022, 16:00:30 UTC
835feb6 Fix tenacious storage tests for swh.model 6 The logic for testing the tenacious storage proxy by interspersing bad objects with good ones triggers when the object list is larger than 3 items. However, the allowed error rate of 1 failure for a window of 3 objects would only work for lists larger than six objects (putting at least 2 good objects between each bad object). swh.model 6 made the directory, revision and release test object lists 3 object long, triggering the buggy code. Reducing the window size to 2 objects makes the test logic works for lists of 3 or more objects. 23 March 2022, 15:34:50 UTC
1d7df1e Updated debian changelog for version 1.1.0 23 March 2022, 15:01:48 UTC
3c105d7 Update upstream source from tag 'debian/upstream/1.1.0' Update to upstream version '1.1.0' with Debian dir 03abd5d07629cc28195bb55df5963468fc443ebe 23 March 2022, 15:01:47 UTC
6057aa8 New upstream version 1.1.0 23 March 2022, 15:01:45 UTC
6fdaf8a Remove typing workarounds for Revision.author or Revision.committer being None swh-model 6.0.0 adds proper support for them. + fix issue found by mypy 23 March 2022, 10:01:34 UTC
3eff720 Add support for author=None and committer=None committer=None happens on some malformed commits generated by old dgit version; and it is possible for author=None to happen for the same reason. For now, this is not supported by swh-model, so tests temporarily disable attrs checks that swh-model relies on. 23 March 2022, 10:00:22 UTC
92c78ab pytest: Exclude build directory for tests discovery Due to test modules being copied in subdirectories of the build directory by setuptools, it makes pytest fail by raising ImportPathMismatchError exceptions when invoked from root directory of the module. So ignore the build folder to discover tests. 22 March 2022, 10:58:26 UTC
98b41c8 backfill: Add missing raw_manifest to directories This was not covered by tests so far, because swh.model.tests.swh_model_data.TEST_OBJECTS did not contain any object with a raw_manifest. But it will in swh-model > 5.0.0 16 March 2022, 11:24:46 UTC
8b65e42 backfill: Make integer_ranges() work on str args + add typing to RANGE_GENERATORS Without the type annotation, mypy errors with 'Cannot call function of unknown type' when called from a type-checked function. 15 March 2022, 16:19:22 UTC
77f7e6d postgresql: Remove unused listener code from db.py 15 March 2022, 12:36:42 UTC
ccde097 origin_visit_get_latest: Order by visit id instead of date This allows both the postgresql and cassandra backends to make efficient queries by using an index (resp. clustering key) instead of scanning all visits of the given origin then sorting by date. This does not affect the results for the last majority of cases, as ids are always in increasing chronological, unless an origin was re-loaded from an old archive. 11 March 2022, 12:43:36 UTC
600e87f origin_visit_get_latest: Materialize subquery on 'origin' table. postgresql's query planner does not understand the origin is unique, so it performs a partial index scan on origin_visit_pkey, which is inefficient on origins with many visits. This commit itself is not enough to make it use the proper index, but provides this necessary change that will be used by a future commit. 11 March 2022, 12:36:07 UTC
b0cdab5 postgresql: Increase timeouts that often fail According to Sentry, in the last 30 days: * directory_entry_get_by_path: 958 events, https://sentry.softwareheritage.org/share/issue/c4c2124953a145b2bd325f6f6b7df5a6/ * revision_get: 841 events, https://sentry.softwareheritage.org/share/issue/55fbe01c6f4d4c9bbf684c7608a62ad9/ * release_get: 14 events, https://sentry.softwareheritage.org/share/issue/37c53354541b4c4eaa1faf4e20a68418/ * origin_visit_find_by_date: 114 events, https://sentry.softwareheritage.org/share/issue/a674c12049a941968a717661a0226559/ * origin_get: 79 events, https://sentry.softwareheritage.org/share/issue/bf21d6bc7b24442eb18643d80d936d27/ ; 67 events, https://sentry.softwareheritage.org/share/issue/010a4b1e085a4e2089ba4897c6de6038/ 11 March 2022, 12:19:28 UTC
4e78014 Remove aiohttp from requirements.txt it's not used by swh.storage. 08 March 2022, 15:29:15 UTC
284a4ab Move metrics handling from backends to RPC server Motivation: replaces code duplication in the backends with a single one, to be consistent with the objstorage (which has many more backends) This also fixes the issue of metrics from 'extid_add' to be missing when using the postgresql storage. 02 March 2022, 11:32:18 UTC
bdfe9d9 Updated backport on buster-swh from debian/1.0.0-1_swh1 (unstable-swh) 24 February 2022, 11:27:28 UTC
730eeed Merge tag 'debian/1.0.0-1_swh1' into debian/buster-swh 24 February 2022, 11:27:28 UTC
3612cc6 Updated debian changelog for version 1.0.0 24 February 2022, 11:13:57 UTC
eae87d8 Update upstream source from tag 'debian/upstream/1.0.0' Update to upstream version '1.0.0' with Debian dir 8ae626ab798d36848604eb33d6358be5da4d7885 24 February 2022, 11:13:57 UTC
35e7f0c New upstream version 1.0.0 24 February 2022, 11:13:55 UTC
back to top