| 9082524 | Nicolas Dandrimont | 06 February 2018, 14:27:10 UTC | api.client: switch back_compat for snapshot_add as well | 06 February 2018, 14:27:10 UTC |
| ada0d38 | Nicolas Dandrimont | 06 February 2018, 14:25:36 UTC | storage: swap back_compat switch for snapshot_add | 06 February 2018, 14:25:36 UTC |
| 9f84cec | Nicolas Dandrimont | 31 January 2018, 14:19:47 UTC | Add function to retrieve latest snapshot based on origin filtering This function does _not_ have any backwards compatibility provisions. | 31 January 2018, 14:19:47 UTC |
| 32e89b0 | Nicolas Dandrimont | 26 January 2018, 13:54:36 UTC | Add snapshot information when fetching origin_visits from the database | 31 January 2018, 10:49:14 UTC |
| d6c3f65 | Stefano Zacchiroli | 19 January 2018, 13:21:28 UTC | sql doc Makefile: (try to) dropdb before creating it this way doc (re)building is more robust | 19 January 2018, 13:22:11 UTC |
| b32d2c9 | Nicolas Dandrimont | 19 January 2018, 10:53:05 UTC | Add a way to remove the backwards-compatibility feature of snapshots | 19 January 2018, 10:54:00 UTC |
| ab0e90c | Stefano Zacchiroli | 19 January 2018, 10:30:42 UTC | doc Makefile: clean SQL autodoc upon "clean" | 19 January 2018, 10:30:42 UTC |
| d163501 | Stefano Zacchiroli | 19 January 2018, 10:28:35 UTC | SQL doc Makefile: ensure destdir gets created wherever needed | 19 January 2018, 10:28:35 UTC |
| 985a912 | Stefano Zacchiroli | 16 January 2018, 15:48:53 UTC | sql/Makefile: add missing deps on pdf/svg DB schema generation | 16 January 2018, 15:48:53 UTC |
| 2e033d0 | Nicolas Dandrimont | 16 January 2018, 14:35:23 UTC | sql: don't depend on plpython3u, we don't use it | 16 January 2018, 14:36:32 UTC |
| 56e0092 | Stefano Zacchiroli | 10 January 2018, 11:16:47 UTC | db-init: enforce UTF8 locale for jsonb fanciness Related T918 | 10 January 2018, 11:16:47 UTC |
| f5acfeb | Antoine R. Dumont (@ardumont) | 09 January 2018, 10:46:46 UTC | sql: Fix missing migration instructions Related 08121f7 | 09 January 2018, 10:46:46 UTC |
| 7b5c111 | Stefano Zacchiroli | 06 January 2018, 14:03:33 UTC | db-init: actually grant privileges on tables and sequences | 06 January 2018, 14:03:33 UTC |
| ca308c7 | Stefano Zacchiroli | 06 January 2018, 13:52:20 UTC | sql/bin/db-init: new helper script to setup development DB | 06 January 2018, 13:52:20 UTC |
| cf6cac7 | Stefano Zacchiroli | 06 January 2018, 09:21:51 UTC | SQL: move definition of hash_sha1() function to swh-func.sql rationale: swh-init.sql should only contain stuff that requires being Postgres superuser | 06 January 2018, 09:21:51 UTC |
| a0ac20a | Nicolas Dandrimont | 21 December 2017, 18:45:47 UTC | db: use a thread instead of a tmpfile to COPY data to postgresql Summary: Avoids writing a bunch of temporary files for no good reason. T866 was a symptom of this. Test Plan: No regressions in the unit tests Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D282 | 21 December 2017, 18:48:16 UTC |
| 419d135 | Nicolas Dandrimont | 21 December 2017, 18:45:19 UTC | db: move external imports further down | 21 December 2017, 18:45:19 UTC |
| 221f8b1 | Nicolas Dandrimont | 21 December 2017, 18:40:57 UTC | Add 116->117 SQL upgrade script | 21 December 2017, 18:40:57 UTC |
| 11fcb82 | Nicolas Dandrimont | 21 December 2017, 18:38:50 UTC | Refactor swh_snapshot_add to hit more indexes | 21 December 2017, 18:38:50 UTC |
| 76e3670 | Nicolas Dandrimont | 19 December 2017, 15:22:22 UTC | db: remove bare except: | 19 December 2017, 15:22:34 UTC |
| afcbe94 | Antoine R. Dumont (@ardumont) | 19 December 2017, 13:24:34 UTC | sql/upgrades: add 115 -> 116 swh-enums: Reference new mercurial revision type Related T329 | 19 December 2017, 13:27:40 UTC |
| a455fc0 | Stefano Zacchiroli | 19 December 2017, 11:06:59 UTC | docs: reference original db-schema.svg without symlink indirection the new sphinx seems to be confused by relative symlinks when we merge all docs together | 19 December 2017, 11:06:59 UTC |
| 655bcb6 | Stefano Zacchiroli | 19 December 2017, 11:06:04 UTC | doc generation: use more specific db-schema basename for images was: "swh" this way we can ship the images as is in the global doc, with minimal clash risks | 19 December 2017, 11:06:04 UTC |
| 73256e6 | Nicolas Dandrimont | 18 December 2017, 15:12:01 UTC | Add 114->115 SQL upgrade script | 18 December 2017, 15:12:01 UTC |
| 36aee70 | Nicolas Dandrimont | 15 December 2017, 14:48:16 UTC | Fix documentation formatting for tool_add/tool_get | 15 December 2017, 14:48:16 UTC |
| f4ea97c | Nicolas Dandrimont | 09 November 2017, 17:11:40 UTC | Add snapshot models Summary: Snapshots are the new, improved occurrences; They're the topmost object in the Software Heritage Merkle Tree, and represent a full picture of an origin at a given time. Snapshots contain a list of named pointers to objects in the Software Heritage archive, as well as an intrinsic identifier. The full specification is supported: pointers to all types of objects, dangling pointers, as well as alias branches. They're implemented with a somewhat classic fully normalised model; Foreign keys use a sha1_git, which makes more sense regarding pointing at non-existent objects, at the expense of some economies of size. Backwards compatibility both ways with occurrences is ensured: when adding a snapshot linked to an origin visit, the corresponding occurrences are created in occurrence_history; when querying the snapshot for an origin visit where we haven't generated the snapshot yet, a virtual snapshot with id None is returned. This lets us migrate to the new tables gently. Close T567. Test Plan: Integration tests are included. Reviewers: zack, #reviewers! Maniphest Tasks: T567 Differential Revision: https://forge.softwareheritage.org/D268 | 15 December 2017, 14:36:58 UTC |
| 0bfd709 | Antoine R. Dumont (@ardumont) | 07 December 2017, 16:38:32 UTC | sql/upgrades: add 113 -> 114 Finalize it really. Related T873 | 07 December 2017, 16:58:55 UTC |
| 20d59c3 | Antoine R. Dumont (@ardumont) | 07 December 2017, 13:14:28 UTC | swh.storage.api.server: Do not hardcode the config file Related T871 | 07 December 2017, 13:14:28 UTC |
| 62e8472 | Antoine R. Dumont (@ardumont) | 07 December 2017, 12:18:56 UTC | d/rules: Fix double export instruction | 07 December 2017, 12:18:56 UTC |
| 08121f7 | Antoine R. Dumont (@ardumont) | 06 December 2017, 17:39:34 UTC | swh-storage: Rename indexer_configuration to tool Related T871 | 07 December 2017, 08:45:24 UTC |
| 021512b | Antoine R. Dumont (@ardumont) | 06 December 2017, 15:38:37 UTC | swh-storage: Migrate indexer model to its own model Related T871 | 07 December 2017, 08:44:26 UTC |
| 63858e5 | Antoine Lambert | 14 November 2017, 16:41:48 UTC | Add method to storage for searching origins This adds method 'origin_search' to storage enabling to search for origins whose urls contain a string pattern or match a regular expression Related T848 | 05 December 2017, 10:47:54 UTC |
| a4abad0 | Antoine R. Dumont (@ardumont) | 24 November 2017, 10:13:48 UTC | schemata: Add missing __init__.py for build package purposes | 24 November 2017, 10:13:48 UTC |
| c33ad51 | Antoine R. Dumont (@ardumont) | 17 November 2017, 14:05:53 UTC | sql/upgrades: add 112 -> 113 Related T851 | 17 November 2017, 14:05:53 UTC |
| 8311a4f | Antoine R. Dumont (@ardumont) | 17 November 2017, 10:20:44 UTC | swh.storage: Open indexer_configuration_add endpoint Related T851 | 17 November 2017, 13:46:39 UTC |
| 5a0ddd6 | Antoine R. Dumont (@ardumont) | 17 November 2017, 10:27:36 UTC | swh.storage.tests: Fix broken content_mimetype tests Due to a new value in db for the same tool. Related 35253443fe0bd792e84f3ce939ee20e7eed52f9b Related T849 | 17 November 2017, 10:28:49 UTC |
| 3525344 | Antoine R. Dumont (@ardumont) | 15 November 2017, 14:34:46 UTC | swh-data: Add new content mimetype's indexer configuration Related T849 | 15 November 2017, 14:36:19 UTC |
| 775a1b8 | Nicolas Dandrimont | 10 November 2017, 13:15:28 UTC | add __init__.py for tests | 10 November 2017, 13:16:40 UTC |
| c5740fd | Nicolas Dandrimont | 09 November 2017, 20:51:34 UTC | origin_visit_get: make order repeatable | 10 November 2017, 13:16:40 UTC |
| c4ec821 | Nicolas Dandrimont | 09 November 2017, 17:33:37 UTC | Add missing visit key for occurrence_add documentation | 10 November 2017, 13:16:40 UTC |
| 9b29324 | Nicolas Dandrimont | 07 November 2017, 12:43:52 UTC | Make unique indices actually unique and vice versa | 09 November 2017, 17:11:11 UTC |
| f87a530 | Nicolas Dandrimont | 06 November 2017, 12:46:56 UTC | Add 110->111 SQL upgrade script | 06 November 2017, 12:46:56 UTC |
| e7a7b56 | Nicolas Dandrimont | 03 November 2017, 15:29:34 UTC | Remove unused content provenance cache tables | 03 November 2017, 15:31:36 UTC |
| 75f4edd | Nicolas Dandrimont | 03 November 2017, 11:41:28 UTC | sql/upgrades/110: add metadata tables to SQL schema | 03 November 2017, 11:41:28 UTC |
| 477bd44 | Stefano Zacchiroli | 02 November 2017, 10:09:27 UTC | docs: add absolute anchor to documentation index | 02 November 2017, 10:09:27 UTC |
| d13751a | moranegg | 23 October 2017, 14:42:34 UTC | Refactor entry points to origin_metadata table with get_by function deleting entry points get_all and get_by_provider creating one unique entry point origin_metadata_get_by adding entry point provider_get_by name+url | 24 October 2017, 12:30:01 UTC |
| 48fc6f7 | moranegg | 13 October 2017, 14:37:09 UTC | Refactor origin_metadata and adding provider table and logic added documentation and new version for schema | 23 October 2017, 14:33:26 UTC |
| 76c5326 | moranegg | 06 October 2017, 13:31:52 UTC | Create origin_metadata tables and logic Summary: adding add and get entry points and tests for origin_metadata test for origin_metadata add and get functions pass Added entry points get_all and get_by_provenance for origin_metadata Closes T737 References P168 Test Plan: tests for add, get, get_all and get_by_provenance Reviewers: ardumont, #reviewers! Maniphest Tasks: T737 Differential Revision: https://forge.softwareheritage.org/D254 | 23 October 2017, 14:33:26 UTC |
| 17882a9 | moranegg | 06 October 2017, 13:30:00 UTC | test for origin_metadata add and get functions pass | 23 October 2017, 14:33:26 UTC |
| 3b425af | moranegg | 04 October 2017, 10:14:39 UTC | Create origin_metadata tables and logic adding add and get entry points and tests for origin_metadata | 23 October 2017, 14:33:26 UTC |
| 7e2e9a9 | Stefano Zacchiroli | 21 October 2017, 15:19:40 UTC | docs: integrate postgres DB schema in dev docs | 21 October 2017, 15:19:40 UTC |
| cc8a618 | Stefano Zacchiroli | 21 October 2017, 15:04:23 UTC | sql doc: also generate SVG version of the schema chart | 21 October 2017, 15:04:23 UTC |
| 6ecf3d6 | Stefano Zacchiroli | 21 October 2017, 14:47:03 UTC | doc: convert archiver blueprint to rst and link it from doc index | 21 October 2017, 14:47:03 UTC |
| 8915caf | Stefano Zacchiroli | 21 October 2017, 13:54:23 UTC | db schema chart: add metadata and statistic clusters to place stray tables where they belong | 21 October 2017, 13:54:23 UTC |
| 89b9ce4 | Stefano Zacchiroli | 21 October 2017, 13:53:30 UTC | db schema chart: use more current titles for some clusters | 21 October 2017, 13:54:13 UTC |
| d35e739 | Nicolas Dandrimont | 12 October 2017, 17:51:00 UTC | Make swh.storage.schemata work on SQLalchemy 1.0 | 12 October 2017, 17:51:00 UTC |
| 91e1e5e | Nicolas Dandrimont | 12 October 2017, 16:41:33 UTC | Drop doctests from build as they mess up flask | 12 October 2017, 16:41:33 UTC |
| ecdb993 | Nicolas Dandrimont | 12 October 2017, 15:16:59 UTC | Cleanup packaging | 12 October 2017, 15:16:59 UTC |
| 92e46df | Nicolas Dandrimont | 11 October 2017, 16:51:04 UTC | Move kafka_python to extra requirements | 11 October 2017, 16:51:37 UTC |
| f56e812 | Antoine R. Dumont (@ardumont) | 11 October 2017, 15:41:45 UTC | swh.storage.converters: Fix typo in docstring | 11 October 2017, 15:42:53 UTC |
| b8f5018 | Nicolas Dandrimont | 11 October 2017, 15:39:03 UTC | Flask and doctest are unhappy with each other See also https://github.com/pallets/flask/issues/1680 | 11 October 2017, 15:39:11 UTC |
| 1964eaf | Nicolas Dandrimont | 11 October 2017, 15:33:01 UTC | Cleanup kafka-related requirements | 11 October 2017, 15:33:01 UTC |
| 13a62b1 | Nicolas Dandrimont | 11 October 2017, 15:30:25 UTC | swh.storage.listener: drop cyclic dependency on swh.journal | 11 October 2017, 15:30:25 UTC |
| d910b67 | Nicolas Dandrimont | 11 October 2017, 15:25:06 UTC | add python3-kafka to build-depends | 11 October 2017, 15:25:06 UTC |
| 6cf4fc9 | Nicolas Dandrimont | 11 October 2017, 15:22:09 UTC | Cleanup tests during debian package build | 11 October 2017, 15:22:46 UTC |
| e36a36b | Nicolas Dandrimont | 11 October 2017, 15:18:34 UTC | Bump dependency on swh.model | 11 October 2017, 15:18:34 UTC |
| 2022da2 | Nicolas Dandrimont | 10 October 2017, 13:59:31 UTC | schemata.distribution: update for reuse by the Debian loader | 10 October 2017, 13:59:31 UTC |
| 6a97d92 | Nicolas Dandrimont | 09 October 2017, 10:22:57 UTC | test_storage: update tests to use DentryPerms instead of raw values | 09 October 2017, 10:23:40 UTC |
| c1bdce3 | Nicolas Dandrimont | 09 October 2017, 10:15:05 UTC | db: properly handle IntEnums (e.g. DentryPerms) | 09 October 2017, 10:15:05 UTC |
| e67765b | Nicolas Dandrimont | 14 September 2017, 15:30:39 UTC | schemata: add a new package for ancillary schemata This package is inaugurated by the distribution schemata | 14 September 2017, 15:30:39 UTC |
| 82ca9bb | Nicolas Dandrimont | 14 September 2017, 15:29:18 UTC | debian/control: wrap-and-sort | 14 September 2017, 15:29:18 UTC |
| 118a962 | Stefano Zacchiroli | 07 September 2017, 08:21:34 UTC | sanitize docstrings for sphinx | 07 September 2017, 08:21:34 UTC |
| e14e72d | Nicolas Dandrimont | 04 September 2017, 18:17:03 UTC | sql: add origin_visit to swh_stat_counters() | 04 September 2017, 18:17:03 UTC |
| 4fbc237 | Nicolas Dandrimont | 01 September 2017, 12:14:21 UTC | sql/upgrades: add 107 to 108 script | 01 September 2017, 12:14:21 UTC |
| 20f47dd | Nicolas Dandrimont | 01 September 2017, 08:15:47 UTC | sql/swh-func: keep a cache of exact object counts as a table Close T719 (cc @rdicosmo) | 01 September 2017, 08:18:42 UTC |
| e1ec2d8 | Nicolas Dandrimont | 01 September 2017, 08:14:35 UTC | test_storage: move tests that were inadvertently "local-only" to the base class | 01 September 2017, 08:18:42 UTC |
| 45b3426 | Stefano Zacchiroli | 30 August 2017, 10:26:04 UTC | docs/: add sphinx apidoc generation skeleton change cherry picked from python module template commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d | 30 August 2017, 10:26:04 UTC |
| 464d5bc | Antoine Pietri | 03 August 2017, 15:42:30 UTC | storage_testing: leverage reset_db_tables from db_testing | 03 August 2017, 16:16:16 UTC |
| 9d416d3 | moranegg | 27 July 2017, 13:34:53 UTC | Added revision_metadata table and methods into storage Summary: - testing missing, add and get methods on revision_metadata Reviewers: ardumont, #reviewers! Differential Revision: https://forge.softwareheritage.org/D235 | 28 July 2017, 10:08:27 UTC |
| 806f511 | Antoine Pietri | 19 July 2017, 14:56:27 UTC | tests: move teardown reset_tables logic to storage_testing | 19 July 2017, 14:56:27 UTC |
| db4bee4 | Antoine Pietri | 17 July 2017, 15:02:59 UTC | storage_testing: allow reuse of storage test config in subclasses | 17 July 2017, 15:03:11 UTC |
| 6d063ac | Antoine Pietri | 11 July 2017, 16:01:39 UTC | tests: use new DbTestFixture, add new storage test fixture | 17 July 2017, 12:18:12 UTC |
| cd9513a | Antoine Pietri | 13 July 2017, 14:00:58 UTC | tests: server: don't print run_app line on startup | 17 July 2017, 12:18:01 UTC |
| af4d3da | moranegg | 13 July 2017, 09:18:38 UTC | Merge branch 'content_metadata_logic' | 13 July 2017, 09:18:38 UTC |
| bb17076 | moranegg | 11 July 2017, 09:29:30 UTC | Refactor to keep only content_metadata logic using the metadata_indexer with NPM tool added tests for add with duplicate or without and tests for missing content_metadata deleted redundant conversion for cdb_copy | 12 July 2017, 08:32:02 UTC |
| 8465e7b | Morane Otilia Gruenpeter | 27 June 2017, 14:16:56 UTC | Added content_metadata logic to the storage Summary: Keeping the indexed content_metadata in the storage with a content_metadata table. Possibility to add with and without duplications and getting the content_metadata with tool information. works only for NPM context also because tool configuration. TODO: - add json schema in json/ (should we keep CodeMeta schema as is or put PURl to it for the translated_metadata_property?) Test Plan: - content_metadata_missing - content_metadata_add skipping duplicate - content_metadata_add dropping duplicate - content_metadata_get - db_to_metadata conveter Reviewers: ardumont Differential Revision: https://forge.softwareheritage.org/D219 | 07 July 2017, 13:27:14 UTC |
| b8c480f | Nicolas Dandrimont | 30 June 2017, 12:44:01 UTC | Remove more references to the old swh.storage.archiver module | 30 June 2017, 12:44:01 UTC |
| fb39e1c | Nicolas Dandrimont | 26 June 2017, 12:33:29 UTC | Move archiver to its own package | 26 June 2017, 12:33:29 UTC |
| 0a86965 | Nicolas Dandrimont | 22 June 2017, 15:24:39 UTC | archiver: remove useless test method | 22 June 2017, 15:24:39 UTC |
| 4a40af7 | Nicolas Dandrimont | 22 June 2017, 15:23:13 UTC | archiver: update tests | 22 June 2017, 15:23:13 UTC |
| 5e34b84 | Nicolas Dandrimont | 22 June 2017, 15:17:22 UTC | archiver.director: properly initialize start_id | 22 June 2017, 15:17:22 UTC |
| afce552 | Nicolas Dandrimont | 19 June 2017, 16:59:26 UTC | archiver.director: add throttling for delayed workers | 19 June 2017, 16:59:26 UTC |
| 93283d3 | Nicolas Dandrimont | 19 June 2017, 14:22:19 UTC | archiver.worker: add filtering for sources This allows us to avoid copying data from sources where we would be charged for accesses. | 19 June 2017, 14:22:19 UTC |
| 96697de | Nicolas Dandrimont | 19 June 2017, 14:07:36 UTC | archiver.worker: adjust ongoing behavior for new backend API The backend API now returns proper datetime objects rather than plain timestamps. | 19 June 2017, 14:07:36 UTC |
| a2ebef4 | Nicolas Dandrimont | 19 June 2017, 14:07:07 UTC | archiver.director: add a start-id option to allow parallelism and restarts | 19 June 2017, 14:07:07 UTC |
| 55d7b73 | Nicolas Dandrimont | 19 June 2017, 14:06:29 UTC | archiver.copier: don't eat exceptions when logging them | 19 June 2017, 14:06:29 UTC |
| 9b2d191 | Antoine Pietri | 14 June 2017, 13:49:48 UTC | requirements: remove python-fastimport (migrated from the vault) | 14 June 2017, 13:51:55 UTC |
| 9c8455e | Nicolas Dandrimont | 12 June 2017, 13:38:13 UTC | bump requirements for new version of scheduler | 12 June 2017, 13:38:13 UTC |
| c5be4dc | Nicolas Dandrimont | 12 June 2017, 13:35:07 UTC | tasks: update to new swh.scheduler.task api | 12 June 2017, 13:35:07 UTC |
| cb855e0 | Nicolas Dandrimont | 06 June 2017, 16:42:01 UTC | Merge tag 'v0.0.85' v0.0.85 - Improve license endpoint's unknown license policy | 06 June 2017, 16:42:01 UTC |
