https://github.com/SoftwareHeritage/swh-storage

sort by:
Revision Author Date Message Commit Date
941c52c Unify configuration property between director/worker 17 September 2016, 10:47:53 UTC
76d13b0 Deal with potential missing contents in the archiver db Logging an entry about it 17 September 2016, 10:47:50 UTC
5281ad1 Improve get_contents_error implementation - Only read the storage key once. - Improve the logging error. 17 September 2016, 10:47:33 UTC
d26477a Remove dead code already moved in archiver/db 17 September 2016, 10:47:32 UTC
80b21aa Adapt archiver director to read sha1 from stdin Also, adds a force_copy flag in the configuration file to avoid checking preexistence of sha1. This is to be efficient for the first time copy in a new backend. 16 September 2016, 20:14:38 UTC
ee4ecd7 archiver: Unify configuration file between director/worker The initial director and worker had split configuration files. Now it's unified to be both archiver/worker.yml file 16 September 2016, 20:12:53 UTC
5db327a Archiver: Adapt ArchiverToBackendDirector to latest storage api 15 September 2016, 14:30:03 UTC
57ee3b6 content_archive_get: api entry point to list cache contents 15 September 2016, 14:12:59 UTC
7d0b963 archiver: Add missing instruction about 003 upgrade 15 September 2016, 13:07:10 UTC
1228c3f Remove print statement 15 September 2016, 09:06:04 UTC
315a8eb cache_content_get: Fix broken test Related ae99623a1eb7e959944f420c9418ad519ce5bc6e 15 September 2016, 08:55:50 UTC
ce2f4e6 Improve choose backup contents to use multiple sources 15 September 2016, 08:21:40 UTC
02e1f1e Add logging ability to copier 14 September 2016, 18:18:26 UTC
ae99623 Archiver: Filter missing contents before archival 14 September 2016, 17:48:25 UTC
791444a archive - sql/upgrades/004: Insert new archive id + clean up Drop unused archive.url column. Simplify associated tests setup on the archiver with retention policy. 14 September 2016, 10:06:21 UTC
d0cbf0a Archiver Director/Worker: Add copy to backend worker implementation Actions storage: - Open cache_content_get to retrieve contents in cache - sql/upgrades/080: Add stored procedure to read contents from cache Actions archiver: - d/control: Add dependency to archiver on python3-swh.objstorage.cloud - Renamed Archiver(Director|Worker) to ArchiverWithRetentionPolicy(Director|Worker) - Add ArchiverToBackend(Director|Worker) - Add new celery task dedicated for new workers - Update docstring details Related T555 14 September 2016, 09:31:56 UTC
a71109b Http API to access the SWH vault Summary: This API currently only concern the directories as it uses the first draft of the cooker. Ref T532 Depends on D102 Reviewers: #reviewers! Maniphest Tasks: T532 Differential Revision: https://forge.softwareheritage.org/D108 12 September 2016, 12:23:38 UTC
b11cfe3 First version of the directory cooker & cache Summary: This first version does create a compressed folder of an archive directory but is not linked to any API or notification system. This diff is submitted for architecture and code review and will evolve. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D102 12 September 2016, 10:51:20 UTC
1c90cd4 Refactor: Rename adequately swh.storage.db.Db.origin_visit functions In swh.storage.db: - origin_visit_get -> origin_visit_get_all - origin_visit_get_by -> occurrence_by_origin_visit - origin_visit_info -> origin_visit_get 08 September 2016, 12:13:08 UTC
598f38c origin_visit_get_by: Fix origin_visit output data + format change This endpoint was wrongly returning only 1 result. 08 September 2016, 11:49:04 UTC
8c10704 Fix typo and remove unused parameter 06 September 2016, 12:27:27 UTC
6feddb3 Fix and explicit docstring's meaning 06 September 2016, 12:27:27 UTC
cf7ff9e origin_visit_get_by: Update to retrieve associated occurrence info Closed T559 05 September 2016, 09:12:41 UTC
157c129 Open origin_visit_get_by api to retrieve information on origin, visit Related T559 02 September 2016, 13:09:11 UTC
fc0f589 Simplify content provenance information retrieval After further discussion, we only retrieve partial information (revision, origin, visit, path) for now. Related T553 01 September 2016, 16:41:19 UTC
05548a8 Fix: drop a function and not a type Related T553 01 September 2016, 13:24:27 UTC
963634d provenance.tasks: add the revision -> origin cache task 01 September 2016, 12:26:13 UTC
fc3a8b5 test_storage: properly pipeline origin and content provenance caches 01 September 2016, 10:02:02 UTC
2c70c08 storage: make the origin cache properly return revisions 01 September 2016, 10:01:40 UTC
346ccd2 tests.server_testing: sleep for shorter amounts of time to speed up tests 01 September 2016, 09:28:38 UTC
8c47b78 db: pull the data from the swh_cache_revision_origin_add operation 01 September 2016, 09:28:13 UTC
72d27a8 Open api endpoint to read content provenance information Related T553 31 August 2016, 18:20:29 UTC
1a237ae Open api entry point to populate cache_revision_origin Related T554 31 August 2016, 18:19:03 UTC
088e4f0 sql/upgrades/079: Add stored procedure to read content provenance Drop obsolete content_find_occurrence stored procedure Related T554 Related T553 31 August 2016, 17:14:37 UTC
0885000 Intermediary - Refactor - Add content provenance information reading ability Remove unneeded stored procedure and use CTE instead. Related T553 Note: Common Table Expression https://www.postgresql.org/docs/current/static/queries-with.html 31 August 2016, 16:54:08 UTC
5d0ac29 Intermediary - Add content provenance information reading ability This is only stored procedure for now. They will be simplified soon. Related T553 Note: docs on 7.2.1.5. LATERAL Subqueries: https://www.postgresql.org/docs/current/static/queries-table-expressions.html This permits to let the `from table` data be accessible to ulterior function call. 31 August 2016, 16:54:08 UTC
74c80b7 sql/swh-func: Add population functions for cache_revision_origin Refs T554 31 August 2016, 14:59:47 UTC
6915737 d/control: Add package python3-swh.storage.provenance As for python3-swh.storage.archiver, the runtime dependency on python3-swh.core is the implicit one from python3-swh.storage. The same note about splitting repository and adding explicit dependency applies. Related T557 31 August 2016, 09:50:14 UTC
183e60a Add a celery task to permit to populate the cache_content_revision table Closes T557 Related b2be2336e98b6cb69b624e3fd3b071bcf2b99b6f 31 August 2016, 09:38:45 UTC
f2b66e2 d/control: archiver - Remove transitive dependency on swh-objstorage After discussion, as swh-storage already depends on swh-objstorage, we can already use it without explicit declaration on it. This will simplify maintenance in regards to version. Note: We may have to revisit this if someday, we split the git repository (as the dependency won't be seen immediately). 31 August 2016, 09:32:50 UTC
50066d8 d/control: Update swh.storage.archiver's runtime dependency on swh.scheduler The other packages do not depend on swh.scheduler 31 August 2016, 08:58:49 UTC
490d82d Open cache_content_revision_add(revision_id) api endpoint This permits to trigger the population of the cache_content_revision table on a per revision basis. If that revision has already been cached, this does nothing. Closes T552 30 August 2016, 14:56:30 UTC
fdfa56b Refactor - Align listing columns for directory_ls function 30 August 2016, 13:21:13 UTC
4ea7845 Add content provenance information cache tables Close T550 30 August 2016, 12:52:55 UTC
f1141a8 Fix typos and sentence phrasing in archiver/vault blueprint 29 August 2016, 14:20:57 UTC
a242eec d/control: Package python3-swh.storage.archiver Related T545 25 August 2016, 12:55:12 UTC
c160101 Fix pep8 warning 25 August 2016, 12:55:11 UTC
ac089d8 Separate swh.storage.db from swh.storage.archiver.db 25 August 2016, 12:48:07 UTC
c0c9f16 sql/swh-schema: add a `metadata` column to origin_visit This is a preparation for the removal of fetch_history. 24 August 2016, 12:29:33 UTC
a1d61bb Update a script to match Storage API Summary: An unused script was left without being updated. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D107 24 August 2016, 11:28:40 UTC
a184e94 d/control: Update dependency version on swh.core 23 August 2016, 14:29:19 UTC
490f461 storage: add a function to pull (only) metadata for a list of contents 22 August 2016, 16:51:48 UTC
f14c4be Update occurrence_add api entry point to properly deal with origin_visit The 'date' column in occurrence data is no longer relevant. This was used to join information on origin_visit. Now, occurrences entries must be enriched with the key 'visit'. This key, with its dual 'origin' key pair defines clearly which visit is currently being done. This closely matches our origin_visit model. Closes T540 Closes D101 22 August 2016, 13:47:47 UTC
6b33471 Add origin_visit api entry points to create/update origin_visit Closes T538 22 August 2016, 13:47:47 UTC
7a16279 sql/upgrades/075: 074→075 - Add status info to origin_visit Closes T534 22 August 2016, 13:47:47 UTC
febdc3a Proper versioned dependency on python-kafka 19 August 2016, 11:45:41 UTC
7d7f403 d/rules: move listener to specific package 19 August 2016, 10:13:17 UTC
87f7f2d Add versioned requirement on swh.objstorage 19 August 2016, 10:13:17 UTC
d74b3e5 listener: Add a new listener publishing to kafka 18 August 2016, 16:26:02 UTC
0d29365 swh-schema: add PostgreSQL notification support for new objects This allows a PostgreSQL client to listen to messages informing it of new objects being added to the database. Implements the SQL side of T526 16 August 2016, 16:42:31 UTC
48f4c05 Make the archiver use content_archive.num_present content_archive.num_present act like a cache for the number of copies where the content is present. This allow to make a much faster request when retrieving the contents that have fewer copies than expected. 16 August 2016, 14:53:53 UTC
1e97458 check-in draft Vault API 16 August 2016, 12:15:40 UTC
b5780ee swh-func: add functions to retrieve objects by object_id References: T525 11 August 2016, 17:35:17 UTC
ddaf1a0 swh-func: add object_id to revision and release return values 11 August 2016, 17:32:48 UTC
65f1b4f converters: Pass through the object_id from the database 11 August 2016, 14:12:27 UTC
9fdbadd sql/archiver: add a cache column for the number of present copies 03 August 2016, 16:34:26 UTC
c73941c Remove cli arguments for the archiver launching 03 August 2016, 09:00:59 UTC
f79a679 Correct config override in the archiver's test 03 August 2016, 09:00:59 UTC
ed9af69 Remove outdated manual test 03 August 2016, 09:00:59 UTC
287da80 archiver.worker: Remove additional config Additional configuration was passed from director to the worker at initialization. We remove it so the queued messages in celery are only lists of sha1 with no dict to pack/unpack. 03 August 2016, 09:00:59 UTC
801ff72 archiver: correct the worker when handling ongoing status 03 August 2016, 09:00:59 UTC
212d903 archiver: Director & Worker arguments are now parsed from conf file The arguments for these objects are now parsed from a configuration file instead of being passed as rguments 03 August 2016, 09:00:59 UTC
dd49151 Make the worker correclty catch corrupted contents 03 August 2016, 09:00:59 UTC
f76cd4d Perform a complete rewriting of the archiver Those changes allow the archiver to use any storage as a source or a destination making it the central point of all archival. Also, the archiver's mecanisms and code are simplified. See T512 for new architecture. Closes T512 03 August 2016, 09:00:59 UTC
148d157 sql/swh-schema: Add indexes on object_ids 01 August 2016, 15:27:13 UTC
591e886 Update the archiver specification Reviewers: #reviewers!, olasd, zack Maniphest Tasks: T512 Differential Revision: https://forge.softwareheritage.org/D88 01 August 2016, 15:23:25 UTC
87386de Add the 'corrupted' state in the archive_status enum Before the archiver copy a content, it have to check if it is valid. If it's not the case, the content status is updated to reflect his invalid state. 01 August 2016, 10:37:39 UTC
65bae82 Update tests to follow RemoteObjStorage api fixes Summary: As the api of RemoteObjStorage have been fixed (See D83) and is now aligned to the base ObjStorage one, places where the remote API were used directly must be updated. Related D83 Reviewers: #reviewers!, olasd Maniphest Tasks: T513 Differential Revision: https://forge.softwareheritage.org/D84 27 July 2016, 14:08:52 UTC
edf12b9 archiver.director: only schedule missing objects 25 July 2016, 11:09:52 UTC
d0a5097 archiver.worker: don't explode if content status is partial 25 July 2016, 11:08:47 UTC
b9b5bc4 archiver: Update get_unarchived_content to do proper batch processing 22 July 2016, 14:48:02 UTC
f4ef5a5 Archiver db's get of content_archive is now ordered by sha1 In db.content_archive_get, when no content_id is specified, the list of contents is now sorted by sha1. Related T412 22 July 2016, 10:18:34 UTC
9451c5f Use postgres 9.5's functions for jsonb manipulation 21 July 2016, 16:12:43 UTC
5c8085d Update the archiver for the new db shema Make the required changes into the db and keep up-to-date the archiver's tests. 21 July 2016, 16:12:43 UTC
134ac77 Update archiver's schema Related T412 21 July 2016, 12:08:56 UTC
df1fe44 sql/upgrades/070: Fix 069→070 migration script 21 July 2016, 09:06:24 UTC
c779f05 Fix - Remove unused parameter Related D78 Closes T499 20 July 2016, 17:02:54 UTC
f500ca2 archiver: change the master's storage to an object storage Summary: Related T499 test.archiver: update the tests to follow changes in the archiver Reviewers: #reviewers, qcampos Reviewed By: #reviewers Maniphest Tasks: T499 Differential Revision: https://forge.softwareheritage.org/D78 Closes D78 20 July 2016, 16:58:44 UTC
e0ca7c5 Fix broken tests after migrating archive_id to enum Also, fix typos in sql schema and data definition 20 July 2016, 14:02:03 UTC
78a68f8 Update schema to use enum for archive identifier 20 July 2016, 13:03:35 UTC
f4341d0 d/control: Fix potential issue on debian package Related P98 Related rDWUI2fff31f14724d89a0316fdf519c09b0aef09feb1 19 July 2016, 13:03:33 UTC
beef951 d/control: Bump dependency 19 July 2016, 13:03:17 UTC
9e34ae9 Make the archiver's content_archive constraints deferrable Related: T412#8084 19 July 2016, 12:41:30 UTC
73139f9 Make swh.storage.archiver able to run in a separated db In effect, this separates concerns between storing/reading contents and archiving contents. This updates: - main schema to drop existing archiver table (070.sql) - integration/unit tests - Separate sql files to deal with install/upgrades in sql/archiver/ folder - Add makefile to deal with local dump and restore actions Remains to deal with updating when receiving new contents (as in the original archiver's code) 18 July 2016, 18:19:07 UTC
50ae16f Revert unused swh_visit_get sql function swh_visit_get was installed to initially use it in the new api. It was finally dismissed but got commited by error. 08 July 2016, 09:54:31 UTC
ee4a1bd Open listing visits per origin api Related T286 05 July 2016, 09:18:44 UTC
92e6cd8 Update debian packaging configuration files Closes T460 27 June 2016, 12:06:03 UTC
3ab3d3d Remove objstorage scripts 27 June 2016, 11:27:22 UTC
2cf3a5a Extract http servers common code to swh.objstorage for dependencies 27 June 2016, 11:27:22 UTC
7a0369f Extract the object storage to its own package and do the required modifications on the imports Summary: Also, remove the objstorage-related module swh.storage.checker to put in into swh.objstorage.checker. Reviewers: olasd, zack, #reviewers! Differential Revision: https://forge.softwareheritage.org/D64 23 June 2016, 12:29:42 UTC
back to top