| cf7ff9e | Antoine R. Dumont (@ardumont) | 05 September 2016, 09:10:40 UTC | origin_visit_get_by: Update to retrieve associated occurrence info Closed T559 | 05 September 2016, 09:12:41 UTC |
| 157c129 | Antoine R. Dumont (@ardumont) | 02 September 2016, 13:09:11 UTC | Open origin_visit_get_by api to retrieve information on origin, visit Related T559 | 02 September 2016, 13:09:11 UTC |
| fc0f589 | Antoine R. Dumont (@ardumont) | 01 September 2016, 15:40:11 UTC | Simplify content provenance information retrieval After further discussion, we only retrieve partial information (revision, origin, visit, path) for now. Related T553 | 01 September 2016, 16:41:19 UTC |
| 05548a8 | Antoine R. Dumont (@ardumont) | 01 September 2016, 13:16:33 UTC | Fix: drop a function and not a type Related T553 | 01 September 2016, 13:24:27 UTC |
| 963634d | Nicolas Dandrimont | 01 September 2016, 12:26:13 UTC | provenance.tasks: add the revision -> origin cache task | 01 September 2016, 12:26:13 UTC |
| fc3a8b5 | Nicolas Dandrimont | 01 September 2016, 10:02:02 UTC | test_storage: properly pipeline origin and content provenance caches | 01 September 2016, 10:02:02 UTC |
| 2c70c08 | Nicolas Dandrimont | 01 September 2016, 10:01:40 UTC | storage: make the origin cache properly return revisions | 01 September 2016, 10:01:40 UTC |
| 346ccd2 | Nicolas Dandrimont | 01 September 2016, 09:28:38 UTC | tests.server_testing: sleep for shorter amounts of time to speed up tests | 01 September 2016, 09:28:38 UTC |
| 8c47b78 | Nicolas Dandrimont | 01 September 2016, 09:28:13 UTC | db: pull the data from the swh_cache_revision_origin_add operation | 01 September 2016, 09:28:13 UTC |
| 72d27a8 | Antoine R. Dumont (@ardumont) | 31 August 2016, 18:20:29 UTC | Open api endpoint to read content provenance information Related T553 | 31 August 2016, 18:20:29 UTC |
| 1a237ae | Antoine R. Dumont (@ardumont) | 31 August 2016, 18:19:03 UTC | Open api entry point to populate cache_revision_origin Related T554 | 31 August 2016, 18:19:03 UTC |
| 088e4f0 | Antoine R. Dumont (@ardumont) | 31 August 2016, 17:03:51 UTC | sql/upgrades/079: Add stored procedure to read content provenance Drop obsolete content_find_occurrence stored procedure Related T554 Related T553 | 31 August 2016, 17:14:37 UTC |
| 0885000 | Antoine R. Dumont (@ardumont) | 31 August 2016, 14:43:53 UTC | Intermediary - Refactor - Add content provenance information reading ability Remove unneeded stored procedure and use CTE instead. Related T553 Note: Common Table Expression https://www.postgresql.org/docs/current/static/queries-with.html | 31 August 2016, 16:54:08 UTC |
| 5d0ac29 | Antoine R. Dumont (@ardumont) | 31 August 2016, 14:35:12 UTC | Intermediary - Add content provenance information reading ability This is only stored procedure for now. They will be simplified soon. Related T553 Note: docs on 7.2.1.5. LATERAL Subqueries: https://www.postgresql.org/docs/current/static/queries-table-expressions.html This permits to let the `from table` data be accessible to ulterior function call. | 31 August 2016, 16:54:08 UTC |
| 74c80b7 | Nicolas Dandrimont | 31 August 2016, 14:46:38 UTC | sql/swh-func: Add population functions for cache_revision_origin Refs T554 | 31 August 2016, 14:59:47 UTC |
| 6915737 | Antoine R. Dumont (@ardumont) | 31 August 2016, 09:48:10 UTC | d/control: Add package python3-swh.storage.provenance As for python3-swh.storage.archiver, the runtime dependency on python3-swh.core is the implicit one from python3-swh.storage. The same note about splitting repository and adding explicit dependency applies. Related T557 | 31 August 2016, 09:50:14 UTC |
| 183e60a | Antoine R. Dumont (@ardumont) | 31 August 2016, 09:34:58 UTC | Add a celery task to permit to populate the cache_content_revision table Closes T557 Related b2be2336e98b6cb69b624e3fd3b071bcf2b99b6f | 31 August 2016, 09:38:45 UTC |
| f2b66e2 | Antoine R. Dumont (@ardumont) | 31 August 2016, 09:30:09 UTC | d/control: archiver - Remove transitive dependency on swh-objstorage After discussion, as swh-storage already depends on swh-objstorage, we can already use it without explicit declaration on it. This will simplify maintenance in regards to version. Note: We may have to revisit this if someday, we split the git repository (as the dependency won't be seen immediately). | 31 August 2016, 09:32:50 UTC |
| 50066d8 | Antoine R. Dumont (@ardumont) | 31 August 2016, 08:57:41 UTC | d/control: Update swh.storage.archiver's runtime dependency on swh.scheduler The other packages do not depend on swh.scheduler | 31 August 2016, 08:58:49 UTC |
| 490d82d | Antoine R. Dumont (@ardumont) | 30 August 2016, 13:22:45 UTC | Open cache_content_revision_add(revision_id) api endpoint This permits to trigger the population of the cache_content_revision table on a per revision basis. If that revision has already been cached, this does nothing. Closes T552 | 30 August 2016, 14:56:30 UTC |
| fdfa56b | Antoine R. Dumont (@ardumont) | 30 August 2016, 13:21:13 UTC | Refactor - Align listing columns for directory_ls function | 30 August 2016, 13:21:13 UTC |
| 4ea7845 | Nicolas Dandrimont | 30 August 2016, 12:50:45 UTC | Add content provenance information cache tables Close T550 | 30 August 2016, 12:52:55 UTC |
| f1141a8 | Antoine R. Dumont (@ardumont) | 29 August 2016, 14:14:22 UTC | Fix typos and sentence phrasing in archiver/vault blueprint | 29 August 2016, 14:20:57 UTC |
| a242eec | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:43:49 UTC | d/control: Package python3-swh.storage.archiver Related T545 | 25 August 2016, 12:55:12 UTC |
| c160101 | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:54:36 UTC | Fix pep8 warning | 25 August 2016, 12:55:11 UTC |
| ac089d8 | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:25:34 UTC | Separate swh.storage.db from swh.storage.archiver.db | 25 August 2016, 12:48:07 UTC |
| c0c9f16 | Nicolas Dandrimont | 24 August 2016, 12:22:29 UTC | sql/swh-schema: add a `metadata` column to origin_visit This is a preparation for the removal of fetch_history. | 24 August 2016, 12:29:33 UTC |
| a1d61bb | Quentin Campos | 24 August 2016, 10:53:08 UTC | Update a script to match Storage API Summary: An unused script was left without being updated. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D107 | 24 August 2016, 11:28:40 UTC |
| a184e94 | Antoine R. Dumont (@ardumont) | 23 August 2016, 13:32:24 UTC | d/control: Update dependency version on swh.core | 23 August 2016, 14:29:19 UTC |
| 490f461 | Nicolas Dandrimont | 22 August 2016, 16:51:48 UTC | storage: add a function to pull (only) metadata for a list of contents | 22 August 2016, 16:51:48 UTC |
| f14c4be | Antoine R. Dumont (@ardumont) | 19 August 2016, 15:18:56 UTC | Update occurrence_add api entry point to properly deal with origin_visit The 'date' column in occurrence data is no longer relevant. This was used to join information on origin_visit. Now, occurrences entries must be enriched with the key 'visit'. This key, with its dual 'origin' key pair defines clearly which visit is currently being done. This closely matches our origin_visit model. Closes T540 Closes D101 | 22 August 2016, 13:47:47 UTC |
| 6b33471 | Antoine R. Dumont (@ardumont) | 19 August 2016, 10:14:21 UTC | Add origin_visit api entry points to create/update origin_visit Closes T538 | 22 August 2016, 13:47:47 UTC |
| 7a16279 | Antoine R. Dumont (@ardumont) | 19 August 2016, 10:09:06 UTC | sql/upgrades/075: 074→075 - Add status info to origin_visit Closes T534 | 22 August 2016, 13:47:47 UTC |
| febdc3a | Nicolas Dandrimont | 19 August 2016, 11:45:41 UTC | Proper versioned dependency on python-kafka | 19 August 2016, 11:45:41 UTC |
| 7d7f403 | Nicolas Dandrimont | 18 August 2016, 16:52:21 UTC | d/rules: move listener to specific package | 19 August 2016, 10:13:17 UTC |
| 87f7f2d | Nicolas Dandrimont | 19 August 2016, 09:33:43 UTC | Add versioned requirement on swh.objstorage | 19 August 2016, 10:13:17 UTC |
| d74b3e5 | Nicolas Dandrimont | 18 August 2016, 16:12:12 UTC | listener: Add a new listener publishing to kafka | 18 August 2016, 16:26:02 UTC |
| 0d29365 | Nicolas Dandrimont | 16 August 2016, 15:33:55 UTC | swh-schema: add PostgreSQL notification support for new objects This allows a PostgreSQL client to listen to messages informing it of new objects being added to the database. Implements the SQL side of T526 | 16 August 2016, 16:42:31 UTC |
| 48f4c05 | Quentin Campos | 05 August 2016, 14:22:57 UTC | Make the archiver use content_archive.num_present content_archive.num_present act like a cache for the number of copies where the content is present. This allow to make a much faster request when retrieving the contents that have fewer copies than expected. | 16 August 2016, 14:53:53 UTC |
| 1e97458 | Stefano Zacchiroli | 16 August 2016, 12:15:40 UTC | check-in draft Vault API | 16 August 2016, 12:15:40 UTC |
| b5780ee | Nicolas Dandrimont | 11 August 2016, 17:33:09 UTC | swh-func: add functions to retrieve objects by object_id References: T525 | 11 August 2016, 17:35:17 UTC |
| ddaf1a0 | Nicolas Dandrimont | 11 August 2016, 17:28:13 UTC | swh-func: add object_id to revision and release return values | 11 August 2016, 17:32:48 UTC |
| 65f1b4f | Nicolas Dandrimont | 11 August 2016, 14:12:27 UTC | converters: Pass through the object_id from the database | 11 August 2016, 14:12:27 UTC |
| 9fdbadd | Nicolas Dandrimont | 03 August 2016, 16:34:26 UTC | sql/archiver: add a cache column for the number of present copies | 03 August 2016, 16:34:26 UTC |
| c73941c | Quentin Campos | 02 August 2016, 13:38:26 UTC | Remove cli arguments for the archiver launching | 03 August 2016, 09:00:59 UTC |
| f79a679 | Quentin Campos | 02 August 2016, 13:37:41 UTC | Correct config override in the archiver's test | 03 August 2016, 09:00:59 UTC |
| ed9af69 | Quentin Campos | 02 August 2016, 11:12:54 UTC | Remove outdated manual test | 03 August 2016, 09:00:59 UTC |
| 287da80 | Quentin Campos | 02 August 2016, 10:59:18 UTC | archiver.worker: Remove additional config Additional configuration was passed from director to the worker at initialization. We remove it so the queued messages in celery are only lists of sha1 with no dict to pack/unpack. | 03 August 2016, 09:00:59 UTC |
| 801ff72 | Quentin Campos | 01 August 2016, 15:09:29 UTC | archiver: correct the worker when handling ongoing status | 03 August 2016, 09:00:59 UTC |
| 212d903 | Quentin Campos | 01 August 2016, 14:06:51 UTC | archiver: Director & Worker arguments are now parsed from conf file The arguments for these objects are now parsed from a configuration file instead of being passed as rguments | 03 August 2016, 09:00:59 UTC |
| dd49151 | Quentin Campos | 29 July 2016, 14:42:01 UTC | Make the worker correclty catch corrupted contents | 03 August 2016, 09:00:59 UTC |
| f76cd4d | Quentin Campos | 29 July 2016, 12:57:05 UTC | Perform a complete rewriting of the archiver Those changes allow the archiver to use any storage as a source or a destination making it the central point of all archival. Also, the archiver's mecanisms and code are simplified. See T512 for new architecture. Closes T512 | 03 August 2016, 09:00:59 UTC |
| 148d157 | Nicolas Dandrimont | 01 August 2016, 15:14:46 UTC | sql/swh-schema: Add indexes on object_ids | 01 August 2016, 15:27:13 UTC |
| 591e886 | Quentin Campos | 29 July 2016, 13:32:58 UTC | Update the archiver specification Reviewers: #reviewers!, olasd, zack Maniphest Tasks: T512 Differential Revision: https://forge.softwareheritage.org/D88 | 01 August 2016, 15:23:25 UTC |
| 87386de | Quentin Campos | 29 July 2016, 14:01:25 UTC | Add the 'corrupted' state in the archive_status enum Before the archiver copy a content, it have to check if it is valid. If it's not the case, the content status is updated to reflect his invalid state. | 01 August 2016, 10:37:39 UTC |
| 65bae82 | Quentin Campos | 27 July 2016, 14:02:50 UTC | Update tests to follow RemoteObjStorage api fixes Summary: As the api of RemoteObjStorage have been fixed (See D83) and is now aligned to the base ObjStorage one, places where the remote API were used directly must be updated. Related D83 Reviewers: #reviewers!, olasd Maniphest Tasks: T513 Differential Revision: https://forge.softwareheritage.org/D84 | 27 July 2016, 14:08:52 UTC |
| edf12b9 | Nicolas Dandrimont | 25 July 2016, 11:09:52 UTC | archiver.director: only schedule missing objects | 25 July 2016, 11:09:52 UTC |
| d0a5097 | Nicolas Dandrimont | 25 July 2016, 11:08:47 UTC | archiver.worker: don't explode if content status is partial | 25 July 2016, 11:08:47 UTC |
| b9b5bc4 | Nicolas Dandrimont | 22 July 2016, 14:48:02 UTC | archiver: Update get_unarchived_content to do proper batch processing | 22 July 2016, 14:48:02 UTC |
| f4ef5a5 | Quentin Campos | 22 July 2016, 10:14:25 UTC | Archiver db's get of content_archive is now ordered by sha1 In db.content_archive_get, when no content_id is specified, the list of contents is now sorted by sha1. Related T412 | 22 July 2016, 10:18:34 UTC |
| 9451c5f | Quentin Campos | 21 July 2016, 14:52:00 UTC | Use postgres 9.5's functions for jsonb manipulation | 21 July 2016, 16:12:43 UTC |
| 5c8085d | Quentin Campos | 21 July 2016, 12:35:09 UTC | Update the archiver for the new db shema Make the required changes into the db and keep up-to-date the archiver's tests. | 21 July 2016, 16:12:43 UTC |
| 134ac77 | Antoine R. Dumont (@ardumont) | 21 July 2016, 12:08:56 UTC | Update archiver's schema Related T412 | 21 July 2016, 12:08:56 UTC |
| df1fe44 | Antoine R. Dumont (@ardumont) | 21 July 2016, 09:06:24 UTC | sql/upgrades/070: Fix 069→070 migration script | 21 July 2016, 09:06:24 UTC |
| c779f05 | Antoine R. Dumont (@ardumont) | 20 July 2016, 17:02:54 UTC | Fix - Remove unused parameter Related D78 Closes T499 | 20 July 2016, 17:02:54 UTC |
| f500ca2 | Quentin Campos | 20 July 2016, 16:56:28 UTC | archiver: change the master's storage to an object storage Summary: Related T499 test.archiver: update the tests to follow changes in the archiver Reviewers: #reviewers, qcampos Reviewed By: #reviewers Maniphest Tasks: T499 Differential Revision: https://forge.softwareheritage.org/D78 Closes D78 | 20 July 2016, 16:58:44 UTC |
| e0ca7c5 | Antoine R. Dumont (@ardumont) | 20 July 2016, 13:52:04 UTC | Fix broken tests after migrating archive_id to enum Also, fix typos in sql schema and data definition | 20 July 2016, 14:02:03 UTC |
| 78a68f8 | Antoine R. Dumont (@ardumont) | 20 July 2016, 13:03:35 UTC | Update schema to use enum for archive identifier | 20 July 2016, 13:03:35 UTC |
| f4341d0 | Antoine R. Dumont (@ardumont) | 19 July 2016, 13:03:33 UTC | d/control: Fix potential issue on debian package Related P98 Related rDWUI2fff31f14724d89a0316fdf519c09b0aef09feb1 | 19 July 2016, 13:03:33 UTC |
| beef951 | Antoine R. Dumont (@ardumont) | 19 July 2016, 13:03:17 UTC | d/control: Bump dependency | 19 July 2016, 13:03:17 UTC |
| 9e34ae9 | Antoine R. Dumont (@ardumont) | 19 July 2016, 12:41:30 UTC | Make the archiver's content_archive constraints deferrable Related: T412#8084 | 19 July 2016, 12:41:30 UTC |
| 73139f9 | Antoine R. Dumont (@ardumont) | 18 July 2016, 18:19:07 UTC | Make swh.storage.archiver able to run in a separated db In effect, this separates concerns between storing/reading contents and archiving contents. This updates: - main schema to drop existing archiver table (070.sql) - integration/unit tests - Separate sql files to deal with install/upgrades in sql/archiver/ folder - Add makefile to deal with local dump and restore actions Remains to deal with updating when receiving new contents (as in the original archiver's code) | 18 July 2016, 18:19:07 UTC |
| 50ae16f | Antoine R. Dumont (@ardumont) | 08 July 2016, 09:54:31 UTC | Revert unused swh_visit_get sql function swh_visit_get was installed to initially use it in the new api. It was finally dismissed but got commited by error. | 08 July 2016, 09:54:31 UTC |
| ee4a1bd | Antoine R. Dumont (@ardumont) | 05 July 2016, 09:18:44 UTC | Open listing visits per origin api Related T286 | 05 July 2016, 09:18:44 UTC |
| 92e6cd8 | Quentin Campos | 27 June 2016, 11:26:08 UTC | Update debian packaging configuration files Closes T460 | 27 June 2016, 12:06:03 UTC |
| 3ab3d3d | Quentin Campos | 23 June 2016, 12:59:45 UTC | Remove objstorage scripts | 27 June 2016, 11:27:22 UTC |
| 2cf3a5a | Quentin Campos | 27 June 2016, 11:24:59 UTC | Extract http servers common code to swh.objstorage for dependencies | 27 June 2016, 11:27:22 UTC |
| 7a0369f | Quentin Campos | 23 June 2016, 12:29:10 UTC | Extract the object storage to its own package and do the required modifications on the imports Summary: Also, remove the objstorage-related module swh.storage.checker to put in into swh.objstorage.checker. Reviewers: olasd, zack, #reviewers! Differential Revision: https://forge.softwareheritage.org/D64 | 23 June 2016, 12:29:42 UTC |
| a623ce8 | Quentin Campos | 21 June 2016, 13:48:56 UTC | objstorage.multiplexer: add some use case example to the MultiplexerObjStorage main docstring Closes T459 | 21 June 2016, 14:01:00 UTC |
| d5289ab | Quentin Campos | 21 June 2016, 13:03:19 UTC | debian packaging: update package list | 21 June 2016, 13:03:19 UTC |
| bed4d47 | Quentin Campos | 20 June 2016, 13:28:49 UTC | objstorage.multiplexer.filter: add tests for the filters implementations | 21 June 2016, 12:11:42 UTC |
| fc4791a | Quentin Campos | 15 June 2016, 16:00:17 UTC | objstorage.multiplexer.filter: add a way to easily create filter chaining | 21 June 2016, 12:11:42 UTC |
| 17bd1a3 | Quentin Campos | 15 June 2016, 14:56:42 UTC | objstorage.multiplexer.filter: Implement the multiplexer filter mecanism Also, add a few basic filters as read-only, filter by hash prefix/hash regex matching | 21 June 2016, 12:11:42 UTC |
| ac06326 | Quentin Campos | 16 June 2016, 09:10:47 UTC | objstorage.multiplexer: add tests for the multiplexer object storage | 21 June 2016, 12:11:42 UTC |
| 1ee994b | Quentin Campos | 15 June 2016, 14:00:40 UTC | objstorage.multiplexer: Implement the object storage multiplexer | 21 June 2016, 12:11:42 UTC |
| cbaed54 | Antoine R. Dumont (@ardumont) | 21 June 2016, 08:42:06 UTC | Open (and fix) swh.storage.storage.revision_log_by Closes T432 Closes T457 Related D54 | 21 June 2016, 09:22:15 UTC |
| 68abde3 | Quentin Campos | 14 June 2016, 15:33:24 UTC | archiver: change objstorage initialization into the archiver Closes T433 | 16 June 2016, 13:10:31 UTC |
| d048051 | Quentin Campos | 14 June 2016, 15:31:22 UTC | checker: change the objstorage initialization of the checker Also keep the test up-to-date with the changes | 16 June 2016, 12:47:50 UTC |
| 029a951 | Quentin Campos | 14 June 2016, 15:10:48 UTC | storage: change objstorage initialization arguments Change those arguments in order to keep up with objstorage new version. | 16 June 2016, 12:47:50 UTC |
| c7e1ed3 | Quentin Campos | 14 June 2016, 15:03:14 UTC | objstorage.api: Change instantiation of the object storage Keep up with the changes of pathslicing objstorage. | 16 June 2016, 12:47:50 UTC |
| 746e049 | Quentin Campos | 14 June 2016, 15:01:40 UTC | objstorage: PathslicingObjStorage now allow arbitrary slicing Example, for "abcdef1234567890" with slicing "0:2/0:5", path will be "root/ab/abcde/". Also, update pathslicing objstorage tests to follow those changes | 16 June 2016, 12:47:50 UTC |
| 252e49b | Quentin Campos | 16 June 2016, 09:35:19 UTC | objstorage: move the constant ID_HASH_ALGO to ObjStorage | 16 June 2016, 10:51:47 UTC |
| 8c6e8d7 | Nicolas Dandrimont | 14 June 2016, 15:23:09 UTC | d/control; requirements: Bump swh.core dependency to 0.0.20 | 14 June 2016, 15:23:09 UTC |
| faa35f7 | Nicolas Dandrimont | 14 June 2016, 15:22:11 UTC | d/rules: move to build_dir before testing | 14 June 2016, 15:22:16 UTC |
| cd965e1 | Nicolas Dandrimont | 14 June 2016, 15:21:11 UTC | test_archiver: remove debug prints | 14 June 2016, 15:22:16 UTC |
| d5f02d4 | Quentin Campos | 13 June 2016, 12:49:24 UTC | Correct an error in the ArchiveWorker initialization Summary: This bug is in the asynchronous code and is **not** related to T436 Ref T401 Reviewers: #reviewers Maniphest Tasks: T401 Differential Revision: https://forge.softwareheritage.org/D46 | 13 June 2016, 14:00:03 UTC |
| 45eab09 | Quentin Campos | 06 June 2016, 14:34:07 UTC | Update an administration script to make it follow the changes of objstorage api | 13 June 2016, 09:35:06 UTC |
| 785592a | Quentin Campos | 06 June 2016, 14:31:32 UTC | Change the objstorage http server in order to follow the changes of the local version | 13 June 2016, 09:35:06 UTC |
| 96ff4ef | Quentin Campos | 06 June 2016, 14:30:32 UTC | Update the content integrity checker in order to follow objstorage changes | 13 June 2016, 09:35:06 UTC |
| 7c72d90 | Quentin Campos | 06 June 2016, 12:55:30 UTC | Update the storage module in order to follow objstorage refactoring Ref T433 | 13 June 2016, 09:35:06 UTC |
