| 91e1e5e | Nicolas Dandrimont | 12 October 2017, 16:41:33 UTC | Drop doctests from build as they mess up flask | 12 October 2017, 16:41:33 UTC |
| ecdb993 | Nicolas Dandrimont | 12 October 2017, 15:16:59 UTC | Cleanup packaging | 12 October 2017, 15:16:59 UTC |
| 92e46df | Nicolas Dandrimont | 11 October 2017, 16:51:04 UTC | Move kafka_python to extra requirements | 11 October 2017, 16:51:37 UTC |
| f56e812 | Antoine R. Dumont (@ardumont) | 11 October 2017, 15:41:45 UTC | swh.storage.converters: Fix typo in docstring | 11 October 2017, 15:42:53 UTC |
| b8f5018 | Nicolas Dandrimont | 11 October 2017, 15:39:03 UTC | Flask and doctest are unhappy with each other See also https://github.com/pallets/flask/issues/1680 | 11 October 2017, 15:39:11 UTC |
| 1964eaf | Nicolas Dandrimont | 11 October 2017, 15:33:01 UTC | Cleanup kafka-related requirements | 11 October 2017, 15:33:01 UTC |
| 13a62b1 | Nicolas Dandrimont | 11 October 2017, 15:30:25 UTC | swh.storage.listener: drop cyclic dependency on swh.journal | 11 October 2017, 15:30:25 UTC |
| d910b67 | Nicolas Dandrimont | 11 October 2017, 15:25:06 UTC | add python3-kafka to build-depends | 11 October 2017, 15:25:06 UTC |
| 6cf4fc9 | Nicolas Dandrimont | 11 October 2017, 15:22:09 UTC | Cleanup tests during debian package build | 11 October 2017, 15:22:46 UTC |
| e36a36b | Nicolas Dandrimont | 11 October 2017, 15:18:34 UTC | Bump dependency on swh.model | 11 October 2017, 15:18:34 UTC |
| 2022da2 | Nicolas Dandrimont | 10 October 2017, 13:59:31 UTC | schemata.distribution: update for reuse by the Debian loader | 10 October 2017, 13:59:31 UTC |
| 6a97d92 | Nicolas Dandrimont | 09 October 2017, 10:22:57 UTC | test_storage: update tests to use DentryPerms instead of raw values | 09 October 2017, 10:23:40 UTC |
| c1bdce3 | Nicolas Dandrimont | 09 October 2017, 10:15:05 UTC | db: properly handle IntEnums (e.g. DentryPerms) | 09 October 2017, 10:15:05 UTC |
| e67765b | Nicolas Dandrimont | 14 September 2017, 15:30:39 UTC | schemata: add a new package for ancillary schemata This package is inaugurated by the distribution schemata | 14 September 2017, 15:30:39 UTC |
| 82ca9bb | Nicolas Dandrimont | 14 September 2017, 15:29:18 UTC | debian/control: wrap-and-sort | 14 September 2017, 15:29:18 UTC |
| 118a962 | Stefano Zacchiroli | 07 September 2017, 08:21:34 UTC | sanitize docstrings for sphinx | 07 September 2017, 08:21:34 UTC |
| e14e72d | Nicolas Dandrimont | 04 September 2017, 18:17:03 UTC | sql: add origin_visit to swh_stat_counters() | 04 September 2017, 18:17:03 UTC |
| 4fbc237 | Nicolas Dandrimont | 01 September 2017, 12:14:21 UTC | sql/upgrades: add 107 to 108 script | 01 September 2017, 12:14:21 UTC |
| 20f47dd | Nicolas Dandrimont | 01 September 2017, 08:15:47 UTC | sql/swh-func: keep a cache of exact object counts as a table Close T719 (cc @rdicosmo) | 01 September 2017, 08:18:42 UTC |
| e1ec2d8 | Nicolas Dandrimont | 01 September 2017, 08:14:35 UTC | test_storage: move tests that were inadvertently "local-only" to the base class | 01 September 2017, 08:18:42 UTC |
| 45b3426 | Stefano Zacchiroli | 30 August 2017, 10:26:04 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:04 UTC |
| 464d5bc | Antoine Pietri | 03 August 2017, 15:42:30 UTC | storage_testing: leverage reset_db_tables from db_testing | 03 August 2017, 16:16:16 UTC |
| 9d416d3 | moranegg | 27 July 2017, 13:34:53 UTC | Added revision_metadata table and methods into storage Summary: - testing missing, add and get methods on revision_metadata Reviewers: ardumont, #reviewers! Differential Revision: https://forge.softwareheritage.org/D235 | 28 July 2017, 10:08:27 UTC |
| 806f511 | Antoine Pietri | 19 July 2017, 14:56:27 UTC | tests: move teardown reset_tables logic to storage_testing | 19 July 2017, 14:56:27 UTC |
| db4bee4 | Antoine Pietri | 17 July 2017, 15:02:59 UTC | storage_testing: allow reuse of storage test config in subclasses | 17 July 2017, 15:03:11 UTC |
| 6d063ac | Antoine Pietri | 11 July 2017, 16:01:39 UTC | tests: use new DbTestFixture, add new storage test fixture | 17 July 2017, 12:18:12 UTC |
| cd9513a | Antoine Pietri | 13 July 2017, 14:00:58 UTC | tests: server: don't print run_app line on startup | 17 July 2017, 12:18:01 UTC |
| af4d3da | moranegg | 13 July 2017, 09:18:38 UTC | Merge branch 'content_metadata_logic' | 13 July 2017, 09:18:38 UTC |
| bb17076 | moranegg | 11 July 2017, 09:29:30 UTC | Refactor to keep only content_metadata logic using the metadata_indexer with NPM tool added tests for add with duplicate or without and tests for missing content_metadata deleted redundant conversion for cdb_copy | 12 July 2017, 08:32:02 UTC |
| 8465e7b | Morane Otilia Gruenpeter | 27 June 2017, 14:16:56 UTC | Added content_metadata logic to the storage Summary: Keeping the indexed content_metadata in the storage with a content_metadata table. Possibility to add with and without duplications and getting the content_metadata with tool information. works only for NPM context also because tool configuration. TODO: - add json schema in json/ (should we keep CodeMeta schema as is or put PURl to it for the translated_metadata_property?) Test Plan: - content_metadata_missing - content_metadata_add skipping duplicate - content_metadata_add dropping duplicate - content_metadata_get - db_to_metadata conveter Reviewers: ardumont Differential Revision: https://forge.softwareheritage.org/D219 | 07 July 2017, 13:27:14 UTC |
| b8c480f | Nicolas Dandrimont | 30 June 2017, 12:44:01 UTC | Remove more references to the old swh.storage.archiver module | 30 June 2017, 12:44:01 UTC |
| fb39e1c | Nicolas Dandrimont | 26 June 2017, 12:33:29 UTC | Move archiver to its own package | 26 June 2017, 12:33:29 UTC |
| 0a86965 | Nicolas Dandrimont | 22 June 2017, 15:24:39 UTC | archiver: remove useless test method | 22 June 2017, 15:24:39 UTC |
| 4a40af7 | Nicolas Dandrimont | 22 June 2017, 15:23:13 UTC | archiver: update tests | 22 June 2017, 15:23:13 UTC |
| 5e34b84 | Nicolas Dandrimont | 22 June 2017, 15:17:22 UTC | archiver.director: properly initialize start_id | 22 June 2017, 15:17:22 UTC |
| afce552 | Nicolas Dandrimont | 19 June 2017, 16:59:26 UTC | archiver.director: add throttling for delayed workers | 19 June 2017, 16:59:26 UTC |
| 93283d3 | Nicolas Dandrimont | 19 June 2017, 14:22:19 UTC | archiver.worker: add filtering for sources This allows us to avoid copying data from sources where we would be charged for accesses. | 19 June 2017, 14:22:19 UTC |
| 96697de | Nicolas Dandrimont | 19 June 2017, 14:07:36 UTC | archiver.worker: adjust ongoing behavior for new backend API The backend API now returns proper datetime objects rather than plain timestamps. | 19 June 2017, 14:07:36 UTC |
| a2ebef4 | Nicolas Dandrimont | 19 June 2017, 14:07:07 UTC | archiver.director: add a start-id option to allow parallelism and restarts | 19 June 2017, 14:07:07 UTC |
| 55d7b73 | Nicolas Dandrimont | 19 June 2017, 14:06:29 UTC | archiver.copier: don't eat exceptions when logging them | 19 June 2017, 14:06:29 UTC |
| 9b2d191 | Antoine Pietri | 14 June 2017, 13:49:48 UTC | requirements: remove python-fastimport (migrated from the vault) | 14 June 2017, 13:51:55 UTC |
| 9c8455e | Nicolas Dandrimont | 12 June 2017, 13:38:13 UTC | bump requirements for new version of scheduler | 12 June 2017, 13:38:13 UTC |
| c5be4dc | Nicolas Dandrimont | 12 June 2017, 13:35:07 UTC | tasks: update to new swh.scheduler.task api | 12 June 2017, 13:35:07 UTC |
| cb855e0 | Nicolas Dandrimont | 06 June 2017, 16:42:01 UTC | Merge tag 'v0.0.85' v0.0.85 - Improve license endpoint's unknown license policy | 06 June 2017, 16:42:01 UTC |
| 0ae9592 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 16:36:38 UTC |
| cbbce73 | Nicolas Dandrimont | 06 June 2017, 16:14:43 UTC | archiver: minimal update for new schema | 06 June 2017, 16:14:58 UTC |
| 0965a85 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 15:52:02 UTC |
| 0a7fdb0 | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:33:32 UTC | sql/upgrades: create db upgrade 106->107 Related T721 | 06 June 2017, 13:33:32 UTC |
| 80944cb | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:19:24 UTC | archiver: Make package work for actual objstorage dependency version The head version of the objstorage migrated. Thus the archiver tests broke locally. The test code was fixed to make the local tests pass with the actual head version of the objstorage. But we did not deploy the new objstorage yet, thus the archiver tests break during the packaging phase (of any swh.storage.* package). This commit is a temporary fix to make everyone happy (since we do need to package other storage version for example). This needs to be reverted as soon as we bump the dependency to the new objstorage (> 0.17). Related f681f83 Related b4dfe06 Related 08827dc Related D205 | 06 June 2017, 13:29:23 UTC |
| 0cf935e | Antoine R. Dumont (@ardumont) | 06 June 2017, 12:24:48 UTC | swh.storage: Improve license endpoint's unknown license policy Prior to this commit, the content_fossology_license_add endpoint was not symmetric with other endpoints since it returned the list of licenses it was unable to deal with. Now, it simply adds those unknown licenses in the fossology_license table. And it no longer returns anything (as other endpoints). Related T721 | 06 June 2017, 13:28:32 UTC |
| f681f83 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 02 June 2017, 14:20:45 UTC |
| 08827dc | Antoine R. Dumont (@ardumont) | 02 June 2017, 14:13:28 UTC | Revert "swh.storage.archiver: Fix tests" for packaging reason This reverts commit b4dfe06f9f06bab2ce4b6f44ca9c20dec2e7f28f. | 02 June 2017, 14:16:05 UTC |
| 4bcd830 | Antoine R. Dumont (@ardumont) | 02 June 2017, 12:43:42 UTC | sql/upgrades: create db upgrade 105->106 Related T722 | 02 June 2017, 12:43:42 UTC |
| 5ff3979 | Antoine R. Dumont (@ardumont) | 01 June 2017, 14:55:59 UTC | swh.storage: Update db schema to new version Related T722 | 01 June 2017, 14:55:59 UTC |
| a95a3c4 | Antoine R. Dumont (@ardumont) | 01 June 2017, 14:29:57 UTC | Add new entry for language indexer tool Related T722 | 01 June 2017, 14:29:57 UTC |
| 8f7a5c5 | Antoine R. Dumont (@ardumont) | 01 June 2017, 13:31:03 UTC | swh.storage: fossology license endpoints: use idx_configuration_id Related T722 | 01 June 2017, 13:31:03 UTC |
| 7738a76 | Antoine R. Dumont (@ardumont) | 01 June 2017, 12:21:55 UTC | swh.storage: indexer endpoints: Fix filtering missing data issue The filtering on missing data was only using the content's sha1 and not the indexer configuration id. Nothing to fix in production though. We do not use yet the filtering nor have multiple tools for the same indexer though. Related T722 | 01 June 2017, 12:22:35 UTC |
| f18e2df | Antoine R. Dumont (@ardumont) | 01 June 2017, 12:12:08 UTC | swh.storage: ctags endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 12:12:08 UTC |
| a8ce0d9 | Antoine R. Dumont (@ardumont) | 01 June 2017, 11:22:03 UTC | swh.storage.tests: Refactor reading the indexer tools Related T722 | 01 June 2017, 11:22:03 UTC |
| f3600de | Antoine R. Dumont (@ardumont) | 01 June 2017, 10:31:50 UTC | swh.storage: language endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 10:32:22 UTC |
| 45a923b | Antoine R. Dumont (@ardumont) | 01 June 2017, 10:12:36 UTC | swh.storage: mimetype endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 10:16:10 UTC |
| b4dfe06 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 01 June 2017, 08:55:48 UTC |
| 85fdb1c | Antoine R. Dumont (@ardumont) | 31 May 2017, 16:26:01 UTC | swh-storage: Add indexer_configuration_get endpoint | 31 May 2017, 16:26:49 UTC |
| 8e97754 | Nicolas Dandrimont | 30 May 2017, 11:08:16 UTC | sql/swh-func: use pg_stat_user_tables for more accurate tuple statistics Close T719 | 30 May 2017, 11:09:33 UTC |
| 55495ec | Antoine Pietri | 09 May 2017, 12:31:56 UTC | vault: remove from storage repository | 09 May 2017, 13:17:37 UTC |
| 76ed458 | Antoine R. Dumont (@ardumont) | 27 March 2017, 13:58:02 UTC | swh.storage: Use aggregate key to filter on missing skipped contents Related T703 Closes D200 | 31 March 2017, 10:26:44 UTC |
| 0b5e938 | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:43:35 UTC | sql/upgrades: add 103 -> 104 Adapt according to review on D200. Use sql/bin/db-upgrade to generate the upgrade script. Related T703 | 30 March 2017, 12:13:00 UTC |
| abeb873 | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:21:57 UTC | Revert "swh.storage: Use upsert scheme on (skipped_)content_add function" This reverts commit 8dddc91a7860f9db04026132c874d809a581f5db. | 27 March 2017, 12:21:57 UTC |
| 8dddc91 | Antoine R. Dumont (@ardumont) | 24 March 2017, 23:36:30 UTC | swh.storage: Use upsert scheme on (skipped_)content_add function Related T703 | 27 March 2017, 12:21:23 UTC |
| ada557c | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:18:16 UTC | swh.storage: Extract key variable for insertion Related T703 | 27 March 2017, 12:21:23 UTC |
| 4ff4307 | Antoine R. Dumont (@ardumont) | 24 March 2017, 23:07:51 UTC | swh.storage: Add checksum blake2s256 Related T703 | 24 March 2017, 23:07:51 UTC |
| c94ba89 | Antoine R. Dumont (@ardumont) | 24 March 2017, 11:57:18 UTC | d/control: Add python3-swh.journal dependency with version | 24 March 2017, 11:57:18 UTC |
| 3fe8924 | Antoine R. Dumont (@ardumont) | 23 March 2017, 14:48:15 UTC | swh.storage.listener: Use swh.journal.serializers.key_to_kafka function Related T529 | 23 March 2017, 14:49:40 UTC |
| 47cb71b | Antoine R. Dumont (@ardumont) | 23 March 2017, 13:47:28 UTC | swh.storage.listener: Send notify data as dict of composite primary key All data are sent as dictionary of composite primary key for each object type. This also aligns the content type data with the skipped_content type data (sending sha1, sha1_git, sha256). Related T529 | 23 March 2017, 14:06:05 UTC |
| 9be297b | Antoine Pietri | 21 March 2017, 17:43:42 UTC | vault: gitfast: add documentation, remove more useless directory explorations | 22 March 2017, 13:02:02 UTC |
| 39db44a | Antoine Pietri | 21 March 2017, 16:54:19 UTC | vault: gitfast: use a single db transaction for all the export | 22 March 2017, 13:02:02 UTC |
| 2f9860c | Antoine Pietri | 21 March 2017, 16:49:15 UTC | vault: gifast: add an lru_cache around get_dir_ents | 22 March 2017, 13:02:02 UTC |
| c0e66af | Antoine Pietri | 21 March 2017, 16:47:33 UTC | vault: gifast: add a reset command when starting a new root | 22 March 2017, 13:02:02 UTC |
| b399957 | Antoine Pietri | 21 March 2017, 14:48:54 UTC | vault: make gitfast explore only modified directories | 22 March 2017, 13:02:02 UTC |
| e50f990 | Antoine Pietri | 22 March 2017, 12:04:55 UTC | storage: fix unused cur= argument in directory_ls | 22 March 2017, 12:04:55 UTC |
| 7eabf17 | Nicolas Dandrimont | 21 March 2017, 14:47:10 UTC | db.py: fix flake8 mistake | 21 March 2017, 14:47:10 UTC |
| 627bde4 | Antoine R. Dumont (@ardumont) | 21 March 2017, 13:22:50 UTC | swh.storage.listener: Subscribe to new origin notifications | 21 March 2017, 13:25:55 UTC |
| 9ed7fbb | Antoine Pietri | 17 March 2017, 11:59:15 UTC | vault: pass the obj_id as a class parameter | 17 March 2017, 14:11:18 UTC |
| 90e4e54 | Antoine Pietri | 17 March 2017, 11:55:58 UTC | vault: coherent API view names | 17 March 2017, 14:11:18 UTC |
| 0f3006f | Nicolas Dandrimont | 16 March 2017, 14:57:51 UTC | sql/upgrades: create db upgrade 101->102 | 16 March 2017, 15:01:40 UTC |
| 795d681 | Nicolas Dandrimont | 16 March 2017, 14:40:10 UTC | sql/swh-func: check for equality on the three columns for swh_content_missing It turns out that PostgreSQL is less naive than the comment implies. It will know how to use the most specific index for the equality query. That should speed us up quite a bit. | 16 March 2017, 14:52:30 UTC |
| 64a740b | Nicolas Dandrimont | 16 March 2017, 14:37:29 UTC | storage: add length to directory listing primitives | 16 March 2017, 14:37:29 UTC |
| 39ec770 | Nicolas Dandrimont | 16 March 2017, 14:17:40 UTC | test_storage: make AlteringSchemaTest depend on a new base class This avoids running all tests three times... | 16 March 2017, 14:20:43 UTC |
| f349d4c | Antoine Pietri | 15 March 2017, 16:38:00 UTC | vault: revision_git: faster toposort without useless copies | 16 March 2017, 13:03:22 UTC |
| ac9a2d2 | Antoine Pietri | 15 March 2017, 16:32:32 UTC | vault: directory builder: actually default invalid permissions to 644 | 16 March 2017, 13:03:22 UTC |
| cb03a7d | Antoine R. Dumont (@ardumont) | 15 March 2017, 13:43:34 UTC | Migrate from swh.core.hashutil to swh.model.hashutil Related T700 | 15 March 2017, 15:12:31 UTC |
| 39280bc | Antoine Pietri | 09 March 2017, 12:53:10 UTC | vault: add a git fast-import cooker | 13 March 2017, 18:09:45 UTC |
| 20556dc | Antoine Pietri | 13 March 2017, 14:40:58 UTC | vault cookers: move repeated functions to base class | 13 March 2017, 15:36:35 UTC |
| 633b1d3 | Antoine R. Dumont (@ardumont) | 13 March 2017, 14:39:32 UTC | archiver_content_updater: Use standard archiver_storage instanciation Related 5cad6d37a915474e34fc252e314e0142fa47a079 | 13 March 2017, 14:41:01 UTC |
| d2d45b2 | Antoine R. Dumont (@ardumont) | 03 March 2017, 09:39:37 UTC | swh.storage.archiver.updater: Add specific configuration filename | 13 March 2017, 13:22:58 UTC |
| ff48a42 | Antoine R. Dumont (@ardumont) | 03 March 2017, 09:36:38 UTC | swh.storage.archiver.updater: Add logging level to INFO | 13 March 2017, 13:22:58 UTC |
| 1dc2069 | Antoine R. Dumont (@ardumont) | 27 February 2017, 12:12:30 UTC | swh.storage.archiver.updater: Call directly content_archive_add This function will drop any existing contents if encountered Related T494 Related T569 | 13 March 2017, 13:22:58 UTC |
| 6136af8 | Antoine R. Dumont (@ardumont) | 25 February 2017, 00:12:02 UTC | d/control: Add swh-journal dependency to swh.storage.archiver Related T494 | 13 March 2017, 13:22:58 UTC |
| 4f1d48c | Antoine R. Dumont (@ardumont) | 25 February 2017, 00:04:51 UTC | Add journal client to update content archiver with new content Related T494 | 13 March 2017, 13:22:58 UTC |
| f711c20 | Antoine Pietri | 09 March 2017, 14:42:23 UTC | vault: more explicit DirectoryBuilder variable names | 09 March 2017, 14:42:33 UTC |
