| d622f3a | Nicolas Dandrimont | 19 December 2017, 15:25:57 UTC | New upstream version 0.0.96 | 19 December 2017, 15:25:57 UTC |
| 76e3670 | Nicolas Dandrimont | 19 December 2017, 15:22:22 UTC | db: remove bare except: | 19 December 2017, 15:22:34 UTC |
| afcbe94 | Antoine R. Dumont (@ardumont) | 19 December 2017, 13:24:34 UTC | sql/upgrades: add 115 -> 116 swh-enums: Reference new mercurial revision type Related T329 | 19 December 2017, 13:27:40 UTC |
| a455fc0 | Stefano Zacchiroli | 19 December 2017, 11:06:59 UTC | docs: reference original db-schema.svg without symlink indirection the new sphinx seems to be confused by relative symlinks when we merge all docs together | 19 December 2017, 11:06:59 UTC |
| 655bcb6 | Stefano Zacchiroli | 19 December 2017, 11:06:04 UTC | doc generation: use more specific db-schema basename for images was: "swh" this way we can ship the images as is in the global doc, with minimal clash risks | 19 December 2017, 11:06:04 UTC |
| 73256e6 | Nicolas Dandrimont | 18 December 2017, 15:12:01 UTC | Add 114->115 SQL upgrade script | 18 December 2017, 15:12:01 UTC |
| 36aee70 | Nicolas Dandrimont | 15 December 2017, 14:48:16 UTC | Fix documentation formatting for tool_add/tool_get | 15 December 2017, 14:48:16 UTC |
| f4ea97c | Nicolas Dandrimont | 09 November 2017, 17:11:40 UTC | Add snapshot models Summary: Snapshots are the new, improved occurrences; They're the topmost object in the Software Heritage Merkle Tree, and represent a full picture of an origin at a given time. Snapshots contain a list of named pointers to objects in the Software Heritage archive, as well as an intrinsic identifier. The full specification is supported: pointers to all types of objects, dangling pointers, as well as alias branches. They're implemented with a somewhat classic fully normalised model; Foreign keys use a sha1_git, which makes more sense regarding pointing at non-existent objects, at the expense of some economies of size. Backwards compatibility both ways with occurrences is ensured: when adding a snapshot linked to an origin visit, the corresponding occurrences are created in occurrence_history; when querying the snapshot for an origin visit where we haven't generated the snapshot yet, a virtual snapshot with id None is returned. This lets us migrate to the new tables gently. Close T567. Test Plan: Integration tests are included. Reviewers: zack, #reviewers! Maniphest Tasks: T567 Differential Revision: https://forge.softwareheritage.org/D268 | 15 December 2017, 14:36:58 UTC |
| 0bfd709 | Antoine R. Dumont (@ardumont) | 07 December 2017, 16:38:32 UTC | sql/upgrades: add 113 -> 114 Finalize it really. Related T873 | 07 December 2017, 16:58:55 UTC |
| 20d59c3 | Antoine R. Dumont (@ardumont) | 07 December 2017, 13:14:28 UTC | swh.storage.api.server: Do not hardcode the config file Related T871 | 07 December 2017, 13:14:28 UTC |
| 62e8472 | Antoine R. Dumont (@ardumont) | 07 December 2017, 12:18:56 UTC | d/rules: Fix double export instruction | 07 December 2017, 12:18:56 UTC |
| 49ad68c | Antoine R. Dumont (@ardumont) | 07 December 2017, 08:56:31 UTC | New upstream version 0.0.95 | 07 December 2017, 08:56:31 UTC |
| 08121f7 | Antoine R. Dumont (@ardumont) | 06 December 2017, 17:39:34 UTC | swh-storage: Rename indexer_configuration to tool Related T871 | 07 December 2017, 08:45:24 UTC |
| 021512b | Antoine R. Dumont (@ardumont) | 06 December 2017, 15:38:37 UTC | swh-storage: Migrate indexer model to its own model Related T871 | 07 December 2017, 08:44:26 UTC |
| ecc22ad | Antoine R. Dumont (@ardumont) | 05 December 2017, 11:32:57 UTC | New upstream version 0.0.94 | 05 December 2017, 11:32:57 UTC |
| 63858e5 | Antoine Lambert | 14 November 2017, 16:41:48 UTC | Add method to storage for searching origins This adds method 'origin_search' to storage enabling to search for origins whose urls contain a string pattern or match a regular expression Related T848 | 05 December 2017, 10:47:54 UTC |
| bfcc50d | Antoine R. Dumont (@ardumont) | 24 November 2017, 10:14:11 UTC | New upstream version 0.0.93 | 24 November 2017, 10:14:11 UTC |
| a4abad0 | Antoine R. Dumont (@ardumont) | 24 November 2017, 10:13:48 UTC | schemata: Add missing __init__.py for build package purposes | 24 November 2017, 10:13:48 UTC |
| c33ad51 | Antoine R. Dumont (@ardumont) | 17 November 2017, 14:05:53 UTC | sql/upgrades: add 112 -> 113 Related T851 | 17 November 2017, 14:05:53 UTC |
| 8311a4f | Antoine R. Dumont (@ardumont) | 17 November 2017, 10:20:44 UTC | swh.storage: Open indexer_configuration_add endpoint Related T851 | 17 November 2017, 13:46:39 UTC |
| 5a0ddd6 | Antoine R. Dumont (@ardumont) | 17 November 2017, 10:27:36 UTC | swh.storage.tests: Fix broken content_mimetype tests Due to a new value in db for the same tool. Related 35253443fe0bd792e84f3ce939ee20e7eed52f9b Related T849 | 17 November 2017, 10:28:49 UTC |
| 3525344 | Antoine R. Dumont (@ardumont) | 15 November 2017, 14:34:46 UTC | swh-data: Add new content mimetype's indexer configuration Related T849 | 15 November 2017, 14:36:19 UTC |
| 775a1b8 | Nicolas Dandrimont | 10 November 2017, 13:15:28 UTC | add __init__.py for tests | 10 November 2017, 13:16:40 UTC |
| c5740fd | Nicolas Dandrimont | 09 November 2017, 20:51:34 UTC | origin_visit_get: make order repeatable | 10 November 2017, 13:16:40 UTC |
| c4ec821 | Nicolas Dandrimont | 09 November 2017, 17:33:37 UTC | Add missing visit key for occurrence_add documentation | 10 November 2017, 13:16:40 UTC |
| 9b29324 | Nicolas Dandrimont | 07 November 2017, 12:43:52 UTC | Make unique indices actually unique and vice versa | 09 November 2017, 17:11:11 UTC |
| f87a530 | Nicolas Dandrimont | 06 November 2017, 12:46:56 UTC | Add 110->111 SQL upgrade script | 06 November 2017, 12:46:56 UTC |
| e7a7b56 | Nicolas Dandrimont | 03 November 2017, 15:29:34 UTC | Remove unused content provenance cache tables | 03 November 2017, 15:31:36 UTC |
| 75f4edd | Nicolas Dandrimont | 03 November 2017, 11:41:28 UTC | sql/upgrades/110: add metadata tables to SQL schema | 03 November 2017, 11:41:28 UTC |
| 477bd44 | Stefano Zacchiroli | 02 November 2017, 10:09:27 UTC | docs: add absolute anchor to documentation index | 02 November 2017, 10:09:27 UTC |
| d13751a | moranegg | 23 October 2017, 14:42:34 UTC | Refactor entry points to origin_metadata table with get_by function deleting entry points get_all and get_by_provider creating one unique entry point origin_metadata_get_by adding entry point provider_get_by name+url | 24 October 2017, 12:30:01 UTC |
| 48fc6f7 | moranegg | 13 October 2017, 14:37:09 UTC | Refactor origin_metadata and adding provider table and logic added documentation and new version for schema | 23 October 2017, 14:33:26 UTC |
| 76c5326 | moranegg | 06 October 2017, 13:31:52 UTC | Create origin_metadata tables and logic Summary: adding add and get entry points and tests for origin_metadata test for origin_metadata add and get functions pass Added entry points get_all and get_by_provenance for origin_metadata Closes T737 References P168 Test Plan: tests for add, get, get_all and get_by_provenance Reviewers: ardumont, #reviewers! Maniphest Tasks: T737 Differential Revision: https://forge.softwareheritage.org/D254 | 23 October 2017, 14:33:26 UTC |
| 17882a9 | moranegg | 06 October 2017, 13:30:00 UTC | test for origin_metadata add and get functions pass | 23 October 2017, 14:33:26 UTC |
| 3b425af | moranegg | 04 October 2017, 10:14:39 UTC | Create origin_metadata tables and logic adding add and get entry points and tests for origin_metadata | 23 October 2017, 14:33:26 UTC |
| 7e2e9a9 | Stefano Zacchiroli | 21 October 2017, 15:19:40 UTC | docs: integrate postgres DB schema in dev docs | 21 October 2017, 15:19:40 UTC |
| cc8a618 | Stefano Zacchiroli | 21 October 2017, 15:04:23 UTC | sql doc: also generate SVG version of the schema chart | 21 October 2017, 15:04:23 UTC |
| 6ecf3d6 | Stefano Zacchiroli | 21 October 2017, 14:47:03 UTC | doc: convert archiver blueprint to rst and link it from doc index | 21 October 2017, 14:47:03 UTC |
| 8915caf | Stefano Zacchiroli | 21 October 2017, 13:54:23 UTC | db schema chart: add metadata and statistic clusters to place stray tables where they belong | 21 October 2017, 13:54:23 UTC |
| 89b9ce4 | Stefano Zacchiroli | 21 October 2017, 13:53:30 UTC | db schema chart: use more current titles for some clusters | 21 October 2017, 13:54:13 UTC |
| 62b086d | Nicolas Dandrimont | 12 October 2017, 17:51:24 UTC | New upstream version 0.0.92 | 12 October 2017, 17:51:24 UTC |
| d35e739 | Nicolas Dandrimont | 12 October 2017, 17:51:00 UTC | Make swh.storage.schemata work on SQLalchemy 1.0 | 12 October 2017, 17:51:00 UTC |
| e4ee578 | Nicolas Dandrimont | 12 October 2017, 16:41:46 UTC | New upstream version 0.0.91 | 12 October 2017, 16:41:46 UTC |
| 91e1e5e | Nicolas Dandrimont | 12 October 2017, 16:41:33 UTC | Drop doctests from build as they mess up flask | 12 October 2017, 16:41:33 UTC |
| ecdb993 | Nicolas Dandrimont | 12 October 2017, 15:16:59 UTC | Cleanup packaging | 12 October 2017, 15:16:59 UTC |
| a46d492 | Nicolas Dandrimont | 11 October 2017, 16:53:22 UTC | New upstream version 0.0.90 | 11 October 2017, 16:53:22 UTC |
| 92e46df | Nicolas Dandrimont | 11 October 2017, 16:51:04 UTC | Move kafka_python to extra requirements | 11 October 2017, 16:51:37 UTC |
| f56e812 | Antoine R. Dumont (@ardumont) | 11 October 2017, 15:41:45 UTC | swh.storage.converters: Fix typo in docstring | 11 October 2017, 15:42:53 UTC |
| 160f109 | Nicolas Dandrimont | 11 October 2017, 15:39:29 UTC | New upstream version 0.0.89 | 11 October 2017, 15:39:29 UTC |
| b8f5018 | Nicolas Dandrimont | 11 October 2017, 15:39:03 UTC | Flask and doctest are unhappy with each other See also https://github.com/pallets/flask/issues/1680 | 11 October 2017, 15:39:11 UTC |
| 1964eaf | Nicolas Dandrimont | 11 October 2017, 15:33:01 UTC | Cleanup kafka-related requirements | 11 October 2017, 15:33:01 UTC |
| 13a62b1 | Nicolas Dandrimont | 11 October 2017, 15:30:25 UTC | swh.storage.listener: drop cyclic dependency on swh.journal | 11 October 2017, 15:30:25 UTC |
| d910b67 | Nicolas Dandrimont | 11 October 2017, 15:25:06 UTC | add python3-kafka to build-depends | 11 October 2017, 15:25:06 UTC |
| 6cf4fc9 | Nicolas Dandrimont | 11 October 2017, 15:22:09 UTC | Cleanup tests during debian package build | 11 October 2017, 15:22:46 UTC |
| e36a36b | Nicolas Dandrimont | 11 October 2017, 15:18:34 UTC | Bump dependency on swh.model | 11 October 2017, 15:18:34 UTC |
| 2022da2 | Nicolas Dandrimont | 10 October 2017, 13:59:31 UTC | schemata.distribution: update for reuse by the Debian loader | 10 October 2017, 13:59:31 UTC |
| 6a97d92 | Nicolas Dandrimont | 09 October 2017, 10:22:57 UTC | test_storage: update tests to use DentryPerms instead of raw values | 09 October 2017, 10:23:40 UTC |
| c1bdce3 | Nicolas Dandrimont | 09 October 2017, 10:15:05 UTC | db: properly handle IntEnums (e.g. DentryPerms) | 09 October 2017, 10:15:05 UTC |
| e67765b | Nicolas Dandrimont | 14 September 2017, 15:30:39 UTC | schemata: add a new package for ancillary schemata This package is inaugurated by the distribution schemata | 14 September 2017, 15:30:39 UTC |
| 82ca9bb | Nicolas Dandrimont | 14 September 2017, 15:29:18 UTC | debian/control: wrap-and-sort | 14 September 2017, 15:29:18 UTC |
| 118a962 | Stefano Zacchiroli | 07 September 2017, 08:21:34 UTC | sanitize docstrings for sphinx | 07 September 2017, 08:21:34 UTC |
| e14e72d | Nicolas Dandrimont | 04 September 2017, 18:17:03 UTC | sql: add origin_visit to swh_stat_counters() | 04 September 2017, 18:17:03 UTC |
| 4fbc237 | Nicolas Dandrimont | 01 September 2017, 12:14:21 UTC | sql/upgrades: add 107 to 108 script | 01 September 2017, 12:14:21 UTC |
| 20f47dd | Nicolas Dandrimont | 01 September 2017, 08:15:47 UTC | sql/swh-func: keep a cache of exact object counts as a table Close T719 (cc @rdicosmo) | 01 September 2017, 08:18:42 UTC |
| e1ec2d8 | Nicolas Dandrimont | 01 September 2017, 08:14:35 UTC | test_storage: move tests that were inadvertently "local-only" to the base class | 01 September 2017, 08:18:42 UTC |
| 45b3426 | Stefano Zacchiroli | 30 August 2017, 10:26:04 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:04 UTC |
| 464d5bc | Antoine Pietri | 03 August 2017, 15:42:30 UTC | storage_testing: leverage reset_db_tables from db_testing | 03 August 2017, 16:16:16 UTC |
| 9d416d3 | moranegg | 27 July 2017, 13:34:53 UTC | Added revision_metadata table and methods into storage Summary: - testing missing, add and get methods on revision_metadata Reviewers: ardumont, #reviewers! Differential Revision: https://forge.softwareheritage.org/D235 | 28 July 2017, 10:08:27 UTC |
| 806f511 | Antoine Pietri | 19 July 2017, 14:56:27 UTC | tests: move teardown reset_tables logic to storage_testing | 19 July 2017, 14:56:27 UTC |
| db4bee4 | Antoine Pietri | 17 July 2017, 15:02:59 UTC | storage_testing: allow reuse of storage test config in subclasses | 17 July 2017, 15:03:11 UTC |
| 6d063ac | Antoine Pietri | 11 July 2017, 16:01:39 UTC | tests: use new DbTestFixture, add new storage test fixture | 17 July 2017, 12:18:12 UTC |
| cd9513a | Antoine Pietri | 13 July 2017, 14:00:58 UTC | tests: server: don't print run_app line on startup | 17 July 2017, 12:18:01 UTC |
| af4d3da | moranegg | 13 July 2017, 09:18:38 UTC | Merge branch 'content_metadata_logic' | 13 July 2017, 09:18:38 UTC |
| bb17076 | moranegg | 11 July 2017, 09:29:30 UTC | Refactor to keep only content_metadata logic using the metadata_indexer with NPM tool added tests for add with duplicate or without and tests for missing content_metadata deleted redundant conversion for cdb_copy | 12 July 2017, 08:32:02 UTC |
| 8465e7b | Morane Otilia Gruenpeter | 27 June 2017, 14:16:56 UTC | Added content_metadata logic to the storage Summary: Keeping the indexed content_metadata in the storage with a content_metadata table. Possibility to add with and without duplications and getting the content_metadata with tool information. works only for NPM context also because tool configuration. TODO: - add json schema in json/ (should we keep CodeMeta schema as is or put PURl to it for the translated_metadata_property?) Test Plan: - content_metadata_missing - content_metadata_add skipping duplicate - content_metadata_add dropping duplicate - content_metadata_get - db_to_metadata conveter Reviewers: ardumont Differential Revision: https://forge.softwareheritage.org/D219 | 07 July 2017, 13:27:14 UTC |
| 1089069 | Nicolas Dandrimont | 30 June 2017, 12:52:12 UTC | New upstream version 0.0.88 | 30 June 2017, 12:52:12 UTC |
| b8c480f | Nicolas Dandrimont | 30 June 2017, 12:44:01 UTC | Remove more references to the old swh.storage.archiver module | 30 June 2017, 12:44:01 UTC |
| fb39e1c | Nicolas Dandrimont | 26 June 2017, 12:33:29 UTC | Move archiver to its own package | 26 June 2017, 12:33:29 UTC |
| 0a86965 | Nicolas Dandrimont | 22 June 2017, 15:24:39 UTC | archiver: remove useless test method | 22 June 2017, 15:24:39 UTC |
| 4a40af7 | Nicolas Dandrimont | 22 June 2017, 15:23:13 UTC | archiver: update tests | 22 June 2017, 15:23:13 UTC |
| 5e34b84 | Nicolas Dandrimont | 22 June 2017, 15:17:22 UTC | archiver.director: properly initialize start_id | 22 June 2017, 15:17:22 UTC |
| afce552 | Nicolas Dandrimont | 19 June 2017, 16:59:26 UTC | archiver.director: add throttling for delayed workers | 19 June 2017, 16:59:26 UTC |
| 93283d3 | Nicolas Dandrimont | 19 June 2017, 14:22:19 UTC | archiver.worker: add filtering for sources This allows us to avoid copying data from sources where we would be charged for accesses. | 19 June 2017, 14:22:19 UTC |
| 96697de | Nicolas Dandrimont | 19 June 2017, 14:07:36 UTC | archiver.worker: adjust ongoing behavior for new backend API The backend API now returns proper datetime objects rather than plain timestamps. | 19 June 2017, 14:07:36 UTC |
| a2ebef4 | Nicolas Dandrimont | 19 June 2017, 14:07:07 UTC | archiver.director: add a start-id option to allow parallelism and restarts | 19 June 2017, 14:07:07 UTC |
| 55d7b73 | Nicolas Dandrimont | 19 June 2017, 14:06:29 UTC | archiver.copier: don't eat exceptions when logging them | 19 June 2017, 14:06:29 UTC |
| 9b2d191 | Antoine Pietri | 14 June 2017, 13:49:48 UTC | requirements: remove python-fastimport (migrated from the vault) | 14 June 2017, 13:51:55 UTC |
| 2ab7f19 | Nicolas Dandrimont | 12 June 2017, 15:54:11 UTC | New upstream version 0.0.87 | 12 June 2017, 15:54:11 UTC |
| 9c8455e | Nicolas Dandrimont | 12 June 2017, 13:38:13 UTC | bump requirements for new version of scheduler | 12 June 2017, 13:38:13 UTC |
| c5be4dc | Nicolas Dandrimont | 12 June 2017, 13:35:07 UTC | tasks: update to new swh.scheduler.task api | 12 June 2017, 13:35:07 UTC |
| 34f7638 | Nicolas Dandrimont | 06 June 2017, 16:43:43 UTC | New upstream version 0.0.86 | 06 June 2017, 16:43:43 UTC |
| cb855e0 | Nicolas Dandrimont | 06 June 2017, 16:42:01 UTC | Merge tag 'v0.0.85' v0.0.85 - Improve license endpoint's unknown license policy | 06 June 2017, 16:42:01 UTC |
| 0ae9592 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 16:36:38 UTC |
| cbbce73 | Nicolas Dandrimont | 06 June 2017, 16:14:43 UTC | archiver: minimal update for new schema | 06 June 2017, 16:14:58 UTC |
| 5d0d03d | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:55:39 UTC | New upstream version 0.0.85 | 06 June 2017, 15:55:39 UTC |
| 0965a85 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 15:52:02 UTC |
| 0a7fdb0 | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:33:32 UTC | sql/upgrades: create db upgrade 106->107 Related T721 | 06 June 2017, 13:33:32 UTC |
| 80944cb | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:19:24 UTC | archiver: Make package work for actual objstorage dependency version The head version of the objstorage migrated. Thus the archiver tests broke locally. The test code was fixed to make the local tests pass with the actual head version of the objstorage. But we did not deploy the new objstorage yet, thus the archiver tests break during the packaging phase (of any swh.storage.* package). This commit is a temporary fix to make everyone happy (since we do need to package other storage version for example). This needs to be reverted as soon as we bump the dependency to the new objstorage (> 0.17). Related f681f83 Related b4dfe06 Related 08827dc Related D205 | 06 June 2017, 13:29:23 UTC |
| 0cf935e | Antoine R. Dumont (@ardumont) | 06 June 2017, 12:24:48 UTC | swh.storage: Improve license endpoint's unknown license policy Prior to this commit, the content_fossology_license_add endpoint was not symmetric with other endpoints since it returned the list of licenses it was unable to deal with. Now, it simply adds those unknown licenses in the fossology_license table. And it no longer returns anything (as other endpoints). Related T721 | 06 June 2017, 13:28:32 UTC |
| f681f83 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 02 June 2017, 14:20:45 UTC |
