| 29eefbe | Nicolas Dandrimont | 22 September 2016, 16:54:52 UTC | New upstream version 0.0.62 | 22 September 2016, 16:54:52 UTC |
| ff87ac5 | Nicolas Dandrimont | 22 September 2016, 16:49:20 UTC | swh-func: content-revision cache population now takes a list of revs | 22 September 2016, 16:51:14 UTC |
| 30f5645 | Nicolas Dandrimont | 22 September 2016, 12:35:50 UTC | swh-func: less churn in the cache_content_revision table | 22 September 2016, 12:42:43 UTC |
| 5c9b562 | Antoine R. Dumont (@ardumont) | 22 September 2016, 11:44:05 UTC | New upstream version 0.0.61 | 22 September 2016, 11:44:05 UTC |
| 4c3623c | Antoine R. Dumont (@ardumont) | 22 September 2016, 11:42:19 UTC | Archiver: Fix to copy only to targeted destination Before that, it could for example pushed copies to other mirrors where the content was missing. | 22 September 2016, 11:43:45 UTC |
| cdf11d5 | Antoine R. Dumont (@ardumont) | 22 September 2016, 10:25:06 UTC | d/control: Bump dependency version to latest python3-swh.core | 22 September 2016, 10:37:43 UTC |
| 57053fe | Antoine R. Dumont (@ardumont) | 22 September 2016, 10:31:46 UTC | Refactor: Align source/destination configuration property names | 22 September 2016, 10:37:43 UTC |
| f163c2a | Antoine R. Dumont (@ardumont) | 22 September 2016, 10:24:15 UTC | Handle copies of not registered contents in archiver db Closes T569 | 22 September 2016, 10:37:42 UTC |
| df2e00a | Antoine R. Dumont (@ardumont) | 21 September 2016, 07:59:18 UTC | Refactor logging warning/critical message | 21 September 2016, 16:18:11 UTC |
| 88e0a05 | Antoine R. Dumont (@ardumont) | 21 September 2016, 14:42:02 UTC | Improve on cooking code and docstrings - Fix docstring typos - Some function calls were not renamed. | 21 September 2016, 16:14:44 UTC |
| 0879adf | Quentin Campos | 21 September 2016, 14:20:18 UTC | Refactor the vault cooker to add new bundle types Summary: Make some updates to the vault in order to prepare the next arrival of the revision cooker. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont Maniphest Tasks: T531 Differential Revision: https://forge.softwareheritage.org/D115 Closes D115 | 21 September 2016, 16:14:44 UTC |
| 6900685 | Antoine R. Dumont (@ardumont) | 20 September 2016, 18:10:16 UTC | Be defensive against potential not found content | 20 September 2016, 18:10:16 UTC |
| 0b72673 | Antoine R. Dumont (@ardumont) | 20 September 2016, 14:46:48 UTC | New upstream version 0.0.60 | 20 September 2016, 14:46:48 UTC |
| 189e9c1 | Antoine R. Dumont (@ardumont) | 20 September 2016, 14:37:38 UTC | d/control: Fix typo in version | 20 September 2016, 14:37:38 UTC |
| 9ad3aef | Antoine R. Dumont (@ardumont) | 20 September 2016, 14:35:34 UTC | d/control: Bump dependency version to latest python3-swh.objstorage | 20 September 2016, 14:35:34 UTC |
| 0476b9c | Antoine R. Dumont (@ardumont) | 20 September 2016, 14:23:45 UTC | Fix objstorage instanciation in tests | 20 September 2016, 14:31:23 UTC |
| 2b18a7a | Antoine R. Dumont (@ardumont) | 20 September 2016, 10:35:10 UTC | Remove optional dependency on swh.objstorage.cloud If you want the archiver to have cloud abilities, install the package python3-swh.objstorage.cloud on the server as well. | 20 September 2016, 14:31:23 UTC |
| f7665b3 | Nicolas Dandrimont | 19 September 2016, 12:21:38 UTC | sql/swh-schema: content->revision cache only has one line per content | 19 September 2016, 12:31:03 UTC |
| 42cd25c | Antoine R. Dumont (@ardumont) | 17 September 2016, 10:50:14 UTC | New upstream version 0.0.59 | 17 September 2016, 10:50:14 UTC |
| 941c52c | Antoine R. Dumont (@ardumont) | 17 September 2016, 09:58:55 UTC | Unify configuration property between director/worker | 17 September 2016, 10:47:53 UTC |
| 76d13b0 | Antoine R. Dumont (@ardumont) | 17 September 2016, 08:38:02 UTC | Deal with potential missing contents in the archiver db Logging an entry about it | 17 September 2016, 10:47:50 UTC |
| 5281ad1 | Antoine R. Dumont (@ardumont) | 17 September 2016, 08:14:07 UTC | Improve get_contents_error implementation - Only read the storage key once. - Improve the logging error. | 17 September 2016, 10:47:33 UTC |
| d26477a | Antoine R. Dumont (@ardumont) | 17 September 2016, 10:28:47 UTC | Remove dead code already moved in archiver/db | 17 September 2016, 10:47:32 UTC |
| 64c7441 | Antoine R. Dumont (@ardumont) | 16 September 2016, 20:17:14 UTC | New upstream version 0.0.58 | 16 September 2016, 20:17:14 UTC |
| 80b21aa | Antoine R. Dumont (@ardumont) | 16 September 2016, 20:10:46 UTC | Adapt archiver director to read sha1 from stdin Also, adds a force_copy flag in the configuration file to avoid checking preexistence of sha1. This is to be efficient for the first time copy in a new backend. | 16 September 2016, 20:14:38 UTC |
| ee4ecd7 | Antoine R. Dumont (@ardumont) | 15 September 2016, 15:01:52 UTC | archiver: Unify configuration file between director/worker The initial director and worker had split configuration files. Now it's unified to be both archiver/worker.yml file | 16 September 2016, 20:12:53 UTC |
| c636e55 | Antoine R. Dumont (@ardumont) | 15 September 2016, 14:30:11 UTC | New upstream version 0.0.57 | 15 September 2016, 14:30:11 UTC |
| 5db327a | Antoine R. Dumont (@ardumont) | 15 September 2016, 14:24:53 UTC | Archiver: Adapt ArchiverToBackendDirector to latest storage api | 15 September 2016, 14:30:03 UTC |
| 57ee3b6 | Antoine R. Dumont (@ardumont) | 15 September 2016, 14:10:03 UTC | content_archive_get: api entry point to list cache contents | 15 September 2016, 14:12:59 UTC |
| 7d0b963 | Antoine R. Dumont (@ardumont) | 15 September 2016, 11:24:36 UTC | archiver: Add missing instruction about 003 upgrade | 15 September 2016, 13:07:10 UTC |
| 1228c3f | Antoine R. Dumont (@ardumont) | 15 September 2016, 09:06:04 UTC | Remove print statement | 15 September 2016, 09:06:04 UTC |
| 53a366d | Antoine R. Dumont (@ardumont) | 15 September 2016, 08:56:35 UTC | New upstream version 0.0.56 | 15 September 2016, 08:56:35 UTC |
| 315a8eb | Antoine R. Dumont (@ardumont) | 15 September 2016, 08:54:42 UTC | cache_content_get: Fix broken test Related ae99623a1eb7e959944f420c9418ad519ce5bc6e | 15 September 2016, 08:55:50 UTC |
| ce2f4e6 | Antoine R. Dumont (@ardumont) | 15 September 2016, 08:21:40 UTC | Improve choose backup contents to use multiple sources | 15 September 2016, 08:21:40 UTC |
| 02e1f1e | Antoine R. Dumont (@ardumont) | 14 September 2016, 18:18:26 UTC | Add logging ability to copier | 14 September 2016, 18:18:26 UTC |
| ae99623 | Antoine R. Dumont (@ardumont) | 14 September 2016, 16:55:56 UTC | Archiver: Filter missing contents before archival | 14 September 2016, 17:48:25 UTC |
| 791444a | Antoine R. Dumont (@ardumont) | 14 September 2016, 10:01:06 UTC | archive - sql/upgrades/004: Insert new archive id + clean up Drop unused archive.url column. Simplify associated tests setup on the archiver with retention policy. | 14 September 2016, 10:06:21 UTC |
| d0cbf0a | Antoine R. Dumont (@ardumont) | 12 September 2016, 16:26:49 UTC | Archiver Director/Worker: Add copy to backend worker implementation Actions storage: - Open cache_content_get to retrieve contents in cache - sql/upgrades/080: Add stored procedure to read contents from cache Actions archiver: - d/control: Add dependency to archiver on python3-swh.objstorage.cloud - Renamed Archiver(Director|Worker) to ArchiverWithRetentionPolicy(Director|Worker) - Add ArchiverToBackend(Director|Worker) - Add new celery task dedicated for new workers - Update docstring details Related T555 | 14 September 2016, 09:31:56 UTC |
| a71109b | Quentin Campos | 24 August 2016, 10:50:24 UTC | Http API to access the SWH vault Summary: This API currently only concern the directories as it uses the first draft of the cooker. Ref T532 Depends on D102 Reviewers: #reviewers! Maniphest Tasks: T532 Differential Revision: https://forge.softwareheritage.org/D108 | 12 September 2016, 12:23:38 UTC |
| b11cfe3 | Quentin Campos | 19 August 2016, 13:05:20 UTC | First version of the directory cooker & cache Summary: This first version does create a compressed folder of an archive directory but is not linked to any API or notification system. This diff is submitted for architecture and code review and will evolve. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D102 | 12 September 2016, 10:51:20 UTC |
| de46caf | Antoine R. Dumont (@ardumont) | 08 September 2016, 13:21:28 UTC | New upstream version 0.0.55 | 08 September 2016, 13:21:28 UTC |
| 1c90cd4 | Antoine R. Dumont (@ardumont) | 08 September 2016, 12:13:08 UTC | Refactor: Rename adequately swh.storage.db.Db.origin_visit functions In swh.storage.db: - origin_visit_get -> origin_visit_get_all - origin_visit_get_by -> occurrence_by_origin_visit - origin_visit_info -> origin_visit_get | 08 September 2016, 12:13:08 UTC |
| 598f38c | Antoine R. Dumont (@ardumont) | 08 September 2016, 11:43:37 UTC | origin_visit_get_by: Fix origin_visit output data + format change This endpoint was wrongly returning only 1 result. | 08 September 2016, 11:49:04 UTC |
| 8c10704 | Antoine R. Dumont (@ardumont) | 06 September 2016, 12:25:18 UTC | Fix typo and remove unused parameter | 06 September 2016, 12:27:27 UTC |
| 6feddb3 | Antoine R. Dumont (@ardumont) | 06 September 2016, 12:23:31 UTC | Fix and explicit docstring's meaning | 06 September 2016, 12:27:27 UTC |
| 65757a7 | Antoine R. Dumont (@ardumont) | 05 September 2016, 10:36:34 UTC | New upstream version 0.0.54 | 05 September 2016, 10:36:34 UTC |
| cf7ff9e | Antoine R. Dumont (@ardumont) | 05 September 2016, 09:10:40 UTC | origin_visit_get_by: Update to retrieve associated occurrence info Closed T559 | 05 September 2016, 09:12:41 UTC |
| 157c129 | Antoine R. Dumont (@ardumont) | 02 September 2016, 13:09:11 UTC | Open origin_visit_get_by api to retrieve information on origin, visit Related T559 | 02 September 2016, 13:09:11 UTC |
| d0400dc | Antoine R. Dumont (@ardumont) | 02 September 2016, 09:14:09 UTC | New upstream version 0.0.53 | 02 September 2016, 09:14:09 UTC |
| fc0f589 | Antoine R. Dumont (@ardumont) | 01 September 2016, 15:40:11 UTC | Simplify content provenance information retrieval After further discussion, we only retrieve partial information (revision, origin, visit, path) for now. Related T553 | 01 September 2016, 16:41:19 UTC |
| 05548a8 | Antoine R. Dumont (@ardumont) | 01 September 2016, 13:16:33 UTC | Fix: drop a function and not a type Related T553 | 01 September 2016, 13:24:27 UTC |
| 963634d | Nicolas Dandrimont | 01 September 2016, 12:26:13 UTC | provenance.tasks: add the revision -> origin cache task | 01 September 2016, 12:26:13 UTC |
| fc3a8b5 | Nicolas Dandrimont | 01 September 2016, 10:02:02 UTC | test_storage: properly pipeline origin and content provenance caches | 01 September 2016, 10:02:02 UTC |
| 2c70c08 | Nicolas Dandrimont | 01 September 2016, 10:01:40 UTC | storage: make the origin cache properly return revisions | 01 September 2016, 10:01:40 UTC |
| 346ccd2 | Nicolas Dandrimont | 01 September 2016, 09:28:38 UTC | tests.server_testing: sleep for shorter amounts of time to speed up tests | 01 September 2016, 09:28:38 UTC |
| 8c47b78 | Nicolas Dandrimont | 01 September 2016, 09:28:13 UTC | db: pull the data from the swh_cache_revision_origin_add operation | 01 September 2016, 09:28:13 UTC |
| 72d27a8 | Antoine R. Dumont (@ardumont) | 31 August 2016, 18:20:29 UTC | Open api endpoint to read content provenance information Related T553 | 31 August 2016, 18:20:29 UTC |
| 1a237ae | Antoine R. Dumont (@ardumont) | 31 August 2016, 18:19:03 UTC | Open api entry point to populate cache_revision_origin Related T554 | 31 August 2016, 18:19:03 UTC |
| 088e4f0 | Antoine R. Dumont (@ardumont) | 31 August 2016, 17:03:51 UTC | sql/upgrades/079: Add stored procedure to read content provenance Drop obsolete content_find_occurrence stored procedure Related T554 Related T553 | 31 August 2016, 17:14:37 UTC |
| 0885000 | Antoine R. Dumont (@ardumont) | 31 August 2016, 14:43:53 UTC | Intermediary - Refactor - Add content provenance information reading ability Remove unneeded stored procedure and use CTE instead. Related T553 Note: Common Table Expression https://www.postgresql.org/docs/current/static/queries-with.html | 31 August 2016, 16:54:08 UTC |
| 5d0ac29 | Antoine R. Dumont (@ardumont) | 31 August 2016, 14:35:12 UTC | Intermediary - Add content provenance information reading ability This is only stored procedure for now. They will be simplified soon. Related T553 Note: docs on 7.2.1.5. LATERAL Subqueries: https://www.postgresql.org/docs/current/static/queries-table-expressions.html This permits to let the `from table` data be accessible to ulterior function call. | 31 August 2016, 16:54:08 UTC |
| 74c80b7 | Nicolas Dandrimont | 31 August 2016, 14:46:38 UTC | sql/swh-func: Add population functions for cache_revision_origin Refs T554 | 31 August 2016, 14:59:47 UTC |
| 6915737 | Antoine R. Dumont (@ardumont) | 31 August 2016, 09:48:10 UTC | d/control: Add package python3-swh.storage.provenance As for python3-swh.storage.archiver, the runtime dependency on python3-swh.core is the implicit one from python3-swh.storage. The same note about splitting repository and adding explicit dependency applies. Related T557 | 31 August 2016, 09:50:14 UTC |
| 183e60a | Antoine R. Dumont (@ardumont) | 31 August 2016, 09:34:58 UTC | Add a celery task to permit to populate the cache_content_revision table Closes T557 Related b2be2336e98b6cb69b624e3fd3b071bcf2b99b6f | 31 August 2016, 09:38:45 UTC |
| f2b66e2 | Antoine R. Dumont (@ardumont) | 31 August 2016, 09:30:09 UTC | d/control: archiver - Remove transitive dependency on swh-objstorage After discussion, as swh-storage already depends on swh-objstorage, we can already use it without explicit declaration on it. This will simplify maintenance in regards to version. Note: We may have to revisit this if someday, we split the git repository (as the dependency won't be seen immediately). | 31 August 2016, 09:32:50 UTC |
| 50066d8 | Antoine R. Dumont (@ardumont) | 31 August 2016, 08:57:41 UTC | d/control: Update swh.storage.archiver's runtime dependency on swh.scheduler The other packages do not depend on swh.scheduler | 31 August 2016, 08:58:49 UTC |
| 490d82d | Antoine R. Dumont (@ardumont) | 30 August 2016, 13:22:45 UTC | Open cache_content_revision_add(revision_id) api endpoint This permits to trigger the population of the cache_content_revision table on a per revision basis. If that revision has already been cached, this does nothing. Closes T552 | 30 August 2016, 14:56:30 UTC |
| fdfa56b | Antoine R. Dumont (@ardumont) | 30 August 2016, 13:21:13 UTC | Refactor - Align listing columns for directory_ls function | 30 August 2016, 13:21:13 UTC |
| 4ea7845 | Nicolas Dandrimont | 30 August 2016, 12:50:45 UTC | Add content provenance information cache tables Close T550 | 30 August 2016, 12:52:55 UTC |
| f1141a8 | Antoine R. Dumont (@ardumont) | 29 August 2016, 14:14:22 UTC | Fix typos and sentence phrasing in archiver/vault blueprint | 29 August 2016, 14:20:57 UTC |
| b9d0a82 | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:55:23 UTC | New upstream version 0.0.52 | 25 August 2016, 12:55:23 UTC |
| a242eec | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:43:49 UTC | d/control: Package python3-swh.storage.archiver Related T545 | 25 August 2016, 12:55:12 UTC |
| c160101 | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:54:36 UTC | Fix pep8 warning | 25 August 2016, 12:55:11 UTC |
| ac089d8 | Antoine R. Dumont (@ardumont) | 25 August 2016, 12:25:34 UTC | Separate swh.storage.db from swh.storage.archiver.db | 25 August 2016, 12:48:07 UTC |
| 0d9a844 | Nicolas Dandrimont | 24 August 2016, 12:36:03 UTC | New upstream version 0.0.51 | 24 August 2016, 12:36:03 UTC |
| c0c9f16 | Nicolas Dandrimont | 24 August 2016, 12:22:29 UTC | sql/swh-schema: add a `metadata` column to origin_visit This is a preparation for the removal of fetch_history. | 24 August 2016, 12:29:33 UTC |
| a1d61bb | Quentin Campos | 24 August 2016, 10:53:08 UTC | Update a script to match Storage API Summary: An unused script was left without being updated. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D107 | 24 August 2016, 11:28:40 UTC |
| c83c1d2 | Antoine R. Dumont (@ardumont) | 23 August 2016, 14:29:26 UTC | New upstream version 0.0.50 | 23 August 2016, 14:29:26 UTC |
| a184e94 | Antoine R. Dumont (@ardumont) | 23 August 2016, 13:32:24 UTC | d/control: Update dependency version on swh.core | 23 August 2016, 14:29:19 UTC |
| 490f461 | Nicolas Dandrimont | 22 August 2016, 16:51:48 UTC | storage: add a function to pull (only) metadata for a list of contents | 22 August 2016, 16:51:48 UTC |
| f14c4be | Antoine R. Dumont (@ardumont) | 19 August 2016, 15:18:56 UTC | Update occurrence_add api entry point to properly deal with origin_visit The 'date' column in occurrence data is no longer relevant. This was used to join information on origin_visit. Now, occurrences entries must be enriched with the key 'visit'. This key, with its dual 'origin' key pair defines clearly which visit is currently being done. This closely matches our origin_visit model. Closes T540 Closes D101 | 22 August 2016, 13:47:47 UTC |
| 6b33471 | Antoine R. Dumont (@ardumont) | 19 August 2016, 10:14:21 UTC | Add origin_visit api entry points to create/update origin_visit Closes T538 | 22 August 2016, 13:47:47 UTC |
| 7a16279 | Antoine R. Dumont (@ardumont) | 19 August 2016, 10:09:06 UTC | sql/upgrades/075: 074→075 - Add status info to origin_visit Closes T534 | 22 August 2016, 13:47:47 UTC |
| 59dfd02 | Nicolas Dandrimont | 19 August 2016, 11:45:52 UTC | New upstream version 0.0.49 | 19 August 2016, 11:45:52 UTC |
| febdc3a | Nicolas Dandrimont | 19 August 2016, 11:45:41 UTC | Proper versioned dependency on python-kafka | 19 August 2016, 11:45:41 UTC |
| 772b42f | Nicolas Dandrimont | 19 August 2016, 10:13:50 UTC | New upstream version 0.0.48 | 19 August 2016, 10:13:50 UTC |
| 7d7f403 | Nicolas Dandrimont | 18 August 2016, 16:52:21 UTC | d/rules: move listener to specific package | 19 August 2016, 10:13:17 UTC |
| 87f7f2d | Nicolas Dandrimont | 19 August 2016, 09:33:43 UTC | Add versioned requirement on swh.objstorage | 19 August 2016, 10:13:17 UTC |
| d74b3e5 | Nicolas Dandrimont | 18 August 2016, 16:12:12 UTC | listener: Add a new listener publishing to kafka | 18 August 2016, 16:26:02 UTC |
| 0d29365 | Nicolas Dandrimont | 16 August 2016, 15:33:55 UTC | swh-schema: add PostgreSQL notification support for new objects This allows a PostgreSQL client to listen to messages informing it of new objects being added to the database. Implements the SQL side of T526 | 16 August 2016, 16:42:31 UTC |
| 48f4c05 | Quentin Campos | 05 August 2016, 14:22:57 UTC | Make the archiver use content_archive.num_present content_archive.num_present act like a cache for the number of copies where the content is present. This allow to make a much faster request when retrieving the contents that have fewer copies than expected. | 16 August 2016, 14:53:53 UTC |
| 1e97458 | Stefano Zacchiroli | 16 August 2016, 12:15:40 UTC | check-in draft Vault API | 16 August 2016, 12:15:40 UTC |
| b5780ee | Nicolas Dandrimont | 11 August 2016, 17:33:09 UTC | swh-func: add functions to retrieve objects by object_id References: T525 | 11 August 2016, 17:35:17 UTC |
| ddaf1a0 | Nicolas Dandrimont | 11 August 2016, 17:28:13 UTC | swh-func: add object_id to revision and release return values | 11 August 2016, 17:32:48 UTC |
| 65f1b4f | Nicolas Dandrimont | 11 August 2016, 14:12:27 UTC | converters: Pass through the object_id from the database | 11 August 2016, 14:12:27 UTC |
| 9fdbadd | Nicolas Dandrimont | 03 August 2016, 16:34:26 UTC | sql/archiver: add a cache column for the number of present copies | 03 August 2016, 16:34:26 UTC |
| c73941c | Quentin Campos | 02 August 2016, 13:38:26 UTC | Remove cli arguments for the archiver launching | 03 August 2016, 09:00:59 UTC |
| f79a679 | Quentin Campos | 02 August 2016, 13:37:41 UTC | Correct config override in the archiver's test | 03 August 2016, 09:00:59 UTC |
| ed9af69 | Quentin Campos | 02 August 2016, 11:12:54 UTC | Remove outdated manual test | 03 August 2016, 09:00:59 UTC |
| 287da80 | Quentin Campos | 02 August 2016, 10:59:18 UTC | archiver.worker: Remove additional config Additional configuration was passed from director to the worker at initialization. We remove it so the queued messages in celery are only lists of sha1 with no dict to pack/unpack. | 03 August 2016, 09:00:59 UTC |
