| 63858e5 | Antoine Lambert | 14 November 2017, 16:41:48 UTC | Add method to storage for searching origins This adds method 'origin_search' to storage enabling to search for origins whose urls contain a string pattern or match a regular expression Related T848 | 05 December 2017, 10:47:54 UTC |
| a4abad0 | Antoine R. Dumont (@ardumont) | 24 November 2017, 10:13:48 UTC | schemata: Add missing __init__.py for build package purposes | 24 November 2017, 10:13:48 UTC |
| c33ad51 | Antoine R. Dumont (@ardumont) | 17 November 2017, 14:05:53 UTC | sql/upgrades: add 112 -> 113 Related T851 | 17 November 2017, 14:05:53 UTC |
| 8311a4f | Antoine R. Dumont (@ardumont) | 17 November 2017, 10:20:44 UTC | swh.storage: Open indexer_configuration_add endpoint Related T851 | 17 November 2017, 13:46:39 UTC |
| 5a0ddd6 | Antoine R. Dumont (@ardumont) | 17 November 2017, 10:27:36 UTC | swh.storage.tests: Fix broken content_mimetype tests Due to a new value in db for the same tool. Related 35253443fe0bd792e84f3ce939ee20e7eed52f9b Related T849 | 17 November 2017, 10:28:49 UTC |
| 3525344 | Antoine R. Dumont (@ardumont) | 15 November 2017, 14:34:46 UTC | swh-data: Add new content mimetype's indexer configuration Related T849 | 15 November 2017, 14:36:19 UTC |
| 775a1b8 | Nicolas Dandrimont | 10 November 2017, 13:15:28 UTC | add __init__.py for tests | 10 November 2017, 13:16:40 UTC |
| c5740fd | Nicolas Dandrimont | 09 November 2017, 20:51:34 UTC | origin_visit_get: make order repeatable | 10 November 2017, 13:16:40 UTC |
| c4ec821 | Nicolas Dandrimont | 09 November 2017, 17:33:37 UTC | Add missing visit key for occurrence_add documentation | 10 November 2017, 13:16:40 UTC |
| 9b29324 | Nicolas Dandrimont | 07 November 2017, 12:43:52 UTC | Make unique indices actually unique and vice versa | 09 November 2017, 17:11:11 UTC |
| f87a530 | Nicolas Dandrimont | 06 November 2017, 12:46:56 UTC | Add 110->111 SQL upgrade script | 06 November 2017, 12:46:56 UTC |
| e7a7b56 | Nicolas Dandrimont | 03 November 2017, 15:29:34 UTC | Remove unused content provenance cache tables | 03 November 2017, 15:31:36 UTC |
| 75f4edd | Nicolas Dandrimont | 03 November 2017, 11:41:28 UTC | sql/upgrades/110: add metadata tables to SQL schema | 03 November 2017, 11:41:28 UTC |
| 477bd44 | Stefano Zacchiroli | 02 November 2017, 10:09:27 UTC | docs: add absolute anchor to documentation index | 02 November 2017, 10:09:27 UTC |
| d13751a | moranegg | 23 October 2017, 14:42:34 UTC | Refactor entry points to origin_metadata table with get_by function deleting entry points get_all and get_by_provider creating one unique entry point origin_metadata_get_by adding entry point provider_get_by name+url | 24 October 2017, 12:30:01 UTC |
| 48fc6f7 | moranegg | 13 October 2017, 14:37:09 UTC | Refactor origin_metadata and adding provider table and logic added documentation and new version for schema | 23 October 2017, 14:33:26 UTC |
| 76c5326 | moranegg | 06 October 2017, 13:31:52 UTC | Create origin_metadata tables and logic Summary: adding add and get entry points and tests for origin_metadata test for origin_metadata add and get functions pass Added entry points get_all and get_by_provenance for origin_metadata Closes T737 References P168 Test Plan: tests for add, get, get_all and get_by_provenance Reviewers: ardumont, #reviewers! Maniphest Tasks: T737 Differential Revision: https://forge.softwareheritage.org/D254 | 23 October 2017, 14:33:26 UTC |
| 17882a9 | moranegg | 06 October 2017, 13:30:00 UTC | test for origin_metadata add and get functions pass | 23 October 2017, 14:33:26 UTC |
| 3b425af | moranegg | 04 October 2017, 10:14:39 UTC | Create origin_metadata tables and logic adding add and get entry points and tests for origin_metadata | 23 October 2017, 14:33:26 UTC |
| 7e2e9a9 | Stefano Zacchiroli | 21 October 2017, 15:19:40 UTC | docs: integrate postgres DB schema in dev docs | 21 October 2017, 15:19:40 UTC |
| cc8a618 | Stefano Zacchiroli | 21 October 2017, 15:04:23 UTC | sql doc: also generate SVG version of the schema chart | 21 October 2017, 15:04:23 UTC |
| 6ecf3d6 | Stefano Zacchiroli | 21 October 2017, 14:47:03 UTC | doc: convert archiver blueprint to rst and link it from doc index | 21 October 2017, 14:47:03 UTC |
| 8915caf | Stefano Zacchiroli | 21 October 2017, 13:54:23 UTC | db schema chart: add metadata and statistic clusters to place stray tables where they belong | 21 October 2017, 13:54:23 UTC |
| 89b9ce4 | Stefano Zacchiroli | 21 October 2017, 13:53:30 UTC | db schema chart: use more current titles for some clusters | 21 October 2017, 13:54:13 UTC |
| d35e739 | Nicolas Dandrimont | 12 October 2017, 17:51:00 UTC | Make swh.storage.schemata work on SQLalchemy 1.0 | 12 October 2017, 17:51:00 UTC |
| 91e1e5e | Nicolas Dandrimont | 12 October 2017, 16:41:33 UTC | Drop doctests from build as they mess up flask | 12 October 2017, 16:41:33 UTC |
| ecdb993 | Nicolas Dandrimont | 12 October 2017, 15:16:59 UTC | Cleanup packaging | 12 October 2017, 15:16:59 UTC |
| 92e46df | Nicolas Dandrimont | 11 October 2017, 16:51:04 UTC | Move kafka_python to extra requirements | 11 October 2017, 16:51:37 UTC |
| f56e812 | Antoine R. Dumont (@ardumont) | 11 October 2017, 15:41:45 UTC | swh.storage.converters: Fix typo in docstring | 11 October 2017, 15:42:53 UTC |
| b8f5018 | Nicolas Dandrimont | 11 October 2017, 15:39:03 UTC | Flask and doctest are unhappy with each other See also https://github.com/pallets/flask/issues/1680 | 11 October 2017, 15:39:11 UTC |
| 1964eaf | Nicolas Dandrimont | 11 October 2017, 15:33:01 UTC | Cleanup kafka-related requirements | 11 October 2017, 15:33:01 UTC |
| 13a62b1 | Nicolas Dandrimont | 11 October 2017, 15:30:25 UTC | swh.storage.listener: drop cyclic dependency on swh.journal | 11 October 2017, 15:30:25 UTC |
| d910b67 | Nicolas Dandrimont | 11 October 2017, 15:25:06 UTC | add python3-kafka to build-depends | 11 October 2017, 15:25:06 UTC |
| 6cf4fc9 | Nicolas Dandrimont | 11 October 2017, 15:22:09 UTC | Cleanup tests during debian package build | 11 October 2017, 15:22:46 UTC |
| e36a36b | Nicolas Dandrimont | 11 October 2017, 15:18:34 UTC | Bump dependency on swh.model | 11 October 2017, 15:18:34 UTC |
| 2022da2 | Nicolas Dandrimont | 10 October 2017, 13:59:31 UTC | schemata.distribution: update for reuse by the Debian loader | 10 October 2017, 13:59:31 UTC |
| 6a97d92 | Nicolas Dandrimont | 09 October 2017, 10:22:57 UTC | test_storage: update tests to use DentryPerms instead of raw values | 09 October 2017, 10:23:40 UTC |
| c1bdce3 | Nicolas Dandrimont | 09 October 2017, 10:15:05 UTC | db: properly handle IntEnums (e.g. DentryPerms) | 09 October 2017, 10:15:05 UTC |
| e67765b | Nicolas Dandrimont | 14 September 2017, 15:30:39 UTC | schemata: add a new package for ancillary schemata This package is inaugurated by the distribution schemata | 14 September 2017, 15:30:39 UTC |
| 82ca9bb | Nicolas Dandrimont | 14 September 2017, 15:29:18 UTC | debian/control: wrap-and-sort | 14 September 2017, 15:29:18 UTC |
| 118a962 | Stefano Zacchiroli | 07 September 2017, 08:21:34 UTC | sanitize docstrings for sphinx | 07 September 2017, 08:21:34 UTC |
| e14e72d | Nicolas Dandrimont | 04 September 2017, 18:17:03 UTC | sql: add origin_visit to swh_stat_counters() | 04 September 2017, 18:17:03 UTC |
| 4fbc237 | Nicolas Dandrimont | 01 September 2017, 12:14:21 UTC | sql/upgrades: add 107 to 108 script | 01 September 2017, 12:14:21 UTC |
| 20f47dd | Nicolas Dandrimont | 01 September 2017, 08:15:47 UTC | sql/swh-func: keep a cache of exact object counts as a table Close T719 (cc @rdicosmo) | 01 September 2017, 08:18:42 UTC |
| e1ec2d8 | Nicolas Dandrimont | 01 September 2017, 08:14:35 UTC | test_storage: move tests that were inadvertently "local-only" to the base class | 01 September 2017, 08:18:42 UTC |
| 45b3426 | Stefano Zacchiroli | 30 August 2017, 10:26:04 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:04 UTC |
| 464d5bc | Antoine Pietri | 03 August 2017, 15:42:30 UTC | storage_testing: leverage reset_db_tables from db_testing | 03 August 2017, 16:16:16 UTC |
| 9d416d3 | moranegg | 27 July 2017, 13:34:53 UTC | Added revision_metadata table and methods into storage Summary: - testing missing, add and get methods on revision_metadata Reviewers: ardumont, #reviewers! Differential Revision: https://forge.softwareheritage.org/D235 | 28 July 2017, 10:08:27 UTC |
| 806f511 | Antoine Pietri | 19 July 2017, 14:56:27 UTC | tests: move teardown reset_tables logic to storage_testing | 19 July 2017, 14:56:27 UTC |
| db4bee4 | Antoine Pietri | 17 July 2017, 15:02:59 UTC | storage_testing: allow reuse of storage test config in subclasses | 17 July 2017, 15:03:11 UTC |
| 6d063ac | Antoine Pietri | 11 July 2017, 16:01:39 UTC | tests: use new DbTestFixture, add new storage test fixture | 17 July 2017, 12:18:12 UTC |
| cd9513a | Antoine Pietri | 13 July 2017, 14:00:58 UTC | tests: server: don't print run_app line on startup | 17 July 2017, 12:18:01 UTC |
| af4d3da | moranegg | 13 July 2017, 09:18:38 UTC | Merge branch 'content_metadata_logic' | 13 July 2017, 09:18:38 UTC |
| bb17076 | moranegg | 11 July 2017, 09:29:30 UTC | Refactor to keep only content_metadata logic using the metadata_indexer with NPM tool added tests for add with duplicate or without and tests for missing content_metadata deleted redundant conversion for cdb_copy | 12 July 2017, 08:32:02 UTC |
| 8465e7b | Morane Otilia Gruenpeter | 27 June 2017, 14:16:56 UTC | Added content_metadata logic to the storage Summary: Keeping the indexed content_metadata in the storage with a content_metadata table. Possibility to add with and without duplications and getting the content_metadata with tool information. works only for NPM context also because tool configuration. TODO: - add json schema in json/ (should we keep CodeMeta schema as is or put PURl to it for the translated_metadata_property?) Test Plan: - content_metadata_missing - content_metadata_add skipping duplicate - content_metadata_add dropping duplicate - content_metadata_get - db_to_metadata conveter Reviewers: ardumont Differential Revision: https://forge.softwareheritage.org/D219 | 07 July 2017, 13:27:14 UTC |
| b8c480f | Nicolas Dandrimont | 30 June 2017, 12:44:01 UTC | Remove more references to the old swh.storage.archiver module | 30 June 2017, 12:44:01 UTC |
| fb39e1c | Nicolas Dandrimont | 26 June 2017, 12:33:29 UTC | Move archiver to its own package | 26 June 2017, 12:33:29 UTC |
| 0a86965 | Nicolas Dandrimont | 22 June 2017, 15:24:39 UTC | archiver: remove useless test method | 22 June 2017, 15:24:39 UTC |
| 4a40af7 | Nicolas Dandrimont | 22 June 2017, 15:23:13 UTC | archiver: update tests | 22 June 2017, 15:23:13 UTC |
| 5e34b84 | Nicolas Dandrimont | 22 June 2017, 15:17:22 UTC | archiver.director: properly initialize start_id | 22 June 2017, 15:17:22 UTC |
| afce552 | Nicolas Dandrimont | 19 June 2017, 16:59:26 UTC | archiver.director: add throttling for delayed workers | 19 June 2017, 16:59:26 UTC |
| 93283d3 | Nicolas Dandrimont | 19 June 2017, 14:22:19 UTC | archiver.worker: add filtering for sources This allows us to avoid copying data from sources where we would be charged for accesses. | 19 June 2017, 14:22:19 UTC |
| 96697de | Nicolas Dandrimont | 19 June 2017, 14:07:36 UTC | archiver.worker: adjust ongoing behavior for new backend API The backend API now returns proper datetime objects rather than plain timestamps. | 19 June 2017, 14:07:36 UTC |
| a2ebef4 | Nicolas Dandrimont | 19 June 2017, 14:07:07 UTC | archiver.director: add a start-id option to allow parallelism and restarts | 19 June 2017, 14:07:07 UTC |
| 55d7b73 | Nicolas Dandrimont | 19 June 2017, 14:06:29 UTC | archiver.copier: don't eat exceptions when logging them | 19 June 2017, 14:06:29 UTC |
| 9b2d191 | Antoine Pietri | 14 June 2017, 13:49:48 UTC | requirements: remove python-fastimport (migrated from the vault) | 14 June 2017, 13:51:55 UTC |
| 9c8455e | Nicolas Dandrimont | 12 June 2017, 13:38:13 UTC | bump requirements for new version of scheduler | 12 June 2017, 13:38:13 UTC |
| c5be4dc | Nicolas Dandrimont | 12 June 2017, 13:35:07 UTC | tasks: update to new swh.scheduler.task api | 12 June 2017, 13:35:07 UTC |
| cb855e0 | Nicolas Dandrimont | 06 June 2017, 16:42:01 UTC | Merge tag 'v0.0.85' v0.0.85 - Improve license endpoint's unknown license policy | 06 June 2017, 16:42:01 UTC |
| 0ae9592 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 16:36:38 UTC |
| cbbce73 | Nicolas Dandrimont | 06 June 2017, 16:14:43 UTC | archiver: minimal update for new schema | 06 June 2017, 16:14:58 UTC |
| 0965a85 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 15:52:02 UTC |
| 0a7fdb0 | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:33:32 UTC | sql/upgrades: create db upgrade 106->107 Related T721 | 06 June 2017, 13:33:32 UTC |
| 80944cb | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:19:24 UTC | archiver: Make package work for actual objstorage dependency version The head version of the objstorage migrated. Thus the archiver tests broke locally. The test code was fixed to make the local tests pass with the actual head version of the objstorage. But we did not deploy the new objstorage yet, thus the archiver tests break during the packaging phase (of any swh.storage.* package). This commit is a temporary fix to make everyone happy (since we do need to package other storage version for example). This needs to be reverted as soon as we bump the dependency to the new objstorage (> 0.17). Related f681f83 Related b4dfe06 Related 08827dc Related D205 | 06 June 2017, 13:29:23 UTC |
| 0cf935e | Antoine R. Dumont (@ardumont) | 06 June 2017, 12:24:48 UTC | swh.storage: Improve license endpoint's unknown license policy Prior to this commit, the content_fossology_license_add endpoint was not symmetric with other endpoints since it returned the list of licenses it was unable to deal with. Now, it simply adds those unknown licenses in the fossology_license table. And it no longer returns anything (as other endpoints). Related T721 | 06 June 2017, 13:28:32 UTC |
| f681f83 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 02 June 2017, 14:20:45 UTC |
| 08827dc | Antoine R. Dumont (@ardumont) | 02 June 2017, 14:13:28 UTC | Revert "swh.storage.archiver: Fix tests" for packaging reason This reverts commit b4dfe06f9f06bab2ce4b6f44ca9c20dec2e7f28f. | 02 June 2017, 14:16:05 UTC |
| 4bcd830 | Antoine R. Dumont (@ardumont) | 02 June 2017, 12:43:42 UTC | sql/upgrades: create db upgrade 105->106 Related T722 | 02 June 2017, 12:43:42 UTC |
| 5ff3979 | Antoine R. Dumont (@ardumont) | 01 June 2017, 14:55:59 UTC | swh.storage: Update db schema to new version Related T722 | 01 June 2017, 14:55:59 UTC |
| a95a3c4 | Antoine R. Dumont (@ardumont) | 01 June 2017, 14:29:57 UTC | Add new entry for language indexer tool Related T722 | 01 June 2017, 14:29:57 UTC |
| 8f7a5c5 | Antoine R. Dumont (@ardumont) | 01 June 2017, 13:31:03 UTC | swh.storage: fossology license endpoints: use idx_configuration_id Related T722 | 01 June 2017, 13:31:03 UTC |
| 7738a76 | Antoine R. Dumont (@ardumont) | 01 June 2017, 12:21:55 UTC | swh.storage: indexer endpoints: Fix filtering missing data issue The filtering on missing data was only using the content's sha1 and not the indexer configuration id. Nothing to fix in production though. We do not use yet the filtering nor have multiple tools for the same indexer though. Related T722 | 01 June 2017, 12:22:35 UTC |
| f18e2df | Antoine R. Dumont (@ardumont) | 01 June 2017, 12:12:08 UTC | swh.storage: ctags endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 12:12:08 UTC |
| a8ce0d9 | Antoine R. Dumont (@ardumont) | 01 June 2017, 11:22:03 UTC | swh.storage.tests: Refactor reading the indexer tools Related T722 | 01 June 2017, 11:22:03 UTC |
| f3600de | Antoine R. Dumont (@ardumont) | 01 June 2017, 10:31:50 UTC | swh.storage: language endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 10:32:22 UTC |
| 45a923b | Antoine R. Dumont (@ardumont) | 01 June 2017, 10:12:36 UTC | swh.storage: mimetype endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 10:16:10 UTC |
| b4dfe06 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 01 June 2017, 08:55:48 UTC |
| 85fdb1c | Antoine R. Dumont (@ardumont) | 31 May 2017, 16:26:01 UTC | swh-storage: Add indexer_configuration_get endpoint | 31 May 2017, 16:26:49 UTC |
| 8e97754 | Nicolas Dandrimont | 30 May 2017, 11:08:16 UTC | sql/swh-func: use pg_stat_user_tables for more accurate tuple statistics Close T719 | 30 May 2017, 11:09:33 UTC |
| 55495ec | Antoine Pietri | 09 May 2017, 12:31:56 UTC | vault: remove from storage repository | 09 May 2017, 13:17:37 UTC |
| 76ed458 | Antoine R. Dumont (@ardumont) | 27 March 2017, 13:58:02 UTC | swh.storage: Use aggregate key to filter on missing skipped contents Related T703 Closes D200 | 31 March 2017, 10:26:44 UTC |
| 0b5e938 | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:43:35 UTC | sql/upgrades: add 103 -> 104 Adapt according to review on D200. Use sql/bin/db-upgrade to generate the upgrade script. Related T703 | 30 March 2017, 12:13:00 UTC |
| abeb873 | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:21:57 UTC | Revert "swh.storage: Use upsert scheme on (skipped_)content_add function" This reverts commit 8dddc91a7860f9db04026132c874d809a581f5db. | 27 March 2017, 12:21:57 UTC |
| 8dddc91 | Antoine R. Dumont (@ardumont) | 24 March 2017, 23:36:30 UTC | swh.storage: Use upsert scheme on (skipped_)content_add function Related T703 | 27 March 2017, 12:21:23 UTC |
| ada557c | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:18:16 UTC | swh.storage: Extract key variable for insertion Related T703 | 27 March 2017, 12:21:23 UTC |
| 4ff4307 | Antoine R. Dumont (@ardumont) | 24 March 2017, 23:07:51 UTC | swh.storage: Add checksum blake2s256 Related T703 | 24 March 2017, 23:07:51 UTC |
| c94ba89 | Antoine R. Dumont (@ardumont) | 24 March 2017, 11:57:18 UTC | d/control: Add python3-swh.journal dependency with version | 24 March 2017, 11:57:18 UTC |
| 3fe8924 | Antoine R. Dumont (@ardumont) | 23 March 2017, 14:48:15 UTC | swh.storage.listener: Use swh.journal.serializers.key_to_kafka function Related T529 | 23 March 2017, 14:49:40 UTC |
| 47cb71b | Antoine R. Dumont (@ardumont) | 23 March 2017, 13:47:28 UTC | swh.storage.listener: Send notify data as dict of composite primary key All data are sent as dictionary of composite primary key for each object type. This also aligns the content type data with the skipped_content type data (sending sha1, sha1_git, sha256). Related T529 | 23 March 2017, 14:06:05 UTC |
| 9be297b | Antoine Pietri | 21 March 2017, 17:43:42 UTC | vault: gitfast: add documentation, remove more useless directory explorations | 22 March 2017, 13:02:02 UTC |
