| ee4a1bd | Antoine R. Dumont (@ardumont) | 05 July 2016, 09:18:44 UTC | Open listing visits per origin api Related T286 | 05 July 2016, 09:18:44 UTC |
| 92e6cd8 | Quentin Campos | 27 June 2016, 11:26:08 UTC | Update debian packaging configuration files Closes T460 | 27 June 2016, 12:06:03 UTC |
| 3ab3d3d | Quentin Campos | 23 June 2016, 12:59:45 UTC | Remove objstorage scripts | 27 June 2016, 11:27:22 UTC |
| 2cf3a5a | Quentin Campos | 27 June 2016, 11:24:59 UTC | Extract http servers common code to swh.objstorage for dependencies | 27 June 2016, 11:27:22 UTC |
| 7a0369f | Quentin Campos | 23 June 2016, 12:29:10 UTC | Extract the object storage to its own package and do the required modifications on the imports Summary: Also, remove the objstorage-related module swh.storage.checker to put in into swh.objstorage.checker. Reviewers: olasd, zack, #reviewers! Differential Revision: https://forge.softwareheritage.org/D64 | 23 June 2016, 12:29:42 UTC |
| a623ce8 | Quentin Campos | 21 June 2016, 13:48:56 UTC | objstorage.multiplexer: add some use case example to the MultiplexerObjStorage main docstring Closes T459 | 21 June 2016, 14:01:00 UTC |
| d5289ab | Quentin Campos | 21 June 2016, 13:03:19 UTC | debian packaging: update package list | 21 June 2016, 13:03:19 UTC |
| bed4d47 | Quentin Campos | 20 June 2016, 13:28:49 UTC | objstorage.multiplexer.filter: add tests for the filters implementations | 21 June 2016, 12:11:42 UTC |
| fc4791a | Quentin Campos | 15 June 2016, 16:00:17 UTC | objstorage.multiplexer.filter: add a way to easily create filter chaining | 21 June 2016, 12:11:42 UTC |
| 17bd1a3 | Quentin Campos | 15 June 2016, 14:56:42 UTC | objstorage.multiplexer.filter: Implement the multiplexer filter mecanism Also, add a few basic filters as read-only, filter by hash prefix/hash regex matching | 21 June 2016, 12:11:42 UTC |
| ac06326 | Quentin Campos | 16 June 2016, 09:10:47 UTC | objstorage.multiplexer: add tests for the multiplexer object storage | 21 June 2016, 12:11:42 UTC |
| 1ee994b | Quentin Campos | 15 June 2016, 14:00:40 UTC | objstorage.multiplexer: Implement the object storage multiplexer | 21 June 2016, 12:11:42 UTC |
| cbaed54 | Antoine R. Dumont (@ardumont) | 21 June 2016, 08:42:06 UTC | Open (and fix) swh.storage.storage.revision_log_by Closes T432 Closes T457 Related D54 | 21 June 2016, 09:22:15 UTC |
| 68abde3 | Quentin Campos | 14 June 2016, 15:33:24 UTC | archiver: change objstorage initialization into the archiver Closes T433 | 16 June 2016, 13:10:31 UTC |
| d048051 | Quentin Campos | 14 June 2016, 15:31:22 UTC | checker: change the objstorage initialization of the checker Also keep the test up-to-date with the changes | 16 June 2016, 12:47:50 UTC |
| 029a951 | Quentin Campos | 14 June 2016, 15:10:48 UTC | storage: change objstorage initialization arguments Change those arguments in order to keep up with objstorage new version. | 16 June 2016, 12:47:50 UTC |
| c7e1ed3 | Quentin Campos | 14 June 2016, 15:03:14 UTC | objstorage.api: Change instantiation of the object storage Keep up with the changes of pathslicing objstorage. | 16 June 2016, 12:47:50 UTC |
| 746e049 | Quentin Campos | 14 June 2016, 15:01:40 UTC | objstorage: PathslicingObjStorage now allow arbitrary slicing Example, for "abcdef1234567890" with slicing "0:2/0:5", path will be "root/ab/abcde/". Also, update pathslicing objstorage tests to follow those changes | 16 June 2016, 12:47:50 UTC |
| 252e49b | Quentin Campos | 16 June 2016, 09:35:19 UTC | objstorage: move the constant ID_HASH_ALGO to ObjStorage | 16 June 2016, 10:51:47 UTC |
| 8c6e8d7 | Nicolas Dandrimont | 14 June 2016, 15:23:09 UTC | d/control; requirements: Bump swh.core dependency to 0.0.20 | 14 June 2016, 15:23:09 UTC |
| faa35f7 | Nicolas Dandrimont | 14 June 2016, 15:22:11 UTC | d/rules: move to build_dir before testing | 14 June 2016, 15:22:16 UTC |
| cd965e1 | Nicolas Dandrimont | 14 June 2016, 15:21:11 UTC | test_archiver: remove debug prints | 14 June 2016, 15:22:16 UTC |
| d5f02d4 | Quentin Campos | 13 June 2016, 12:49:24 UTC | Correct an error in the ArchiveWorker initialization Summary: This bug is in the asynchronous code and is **not** related to T436 Ref T401 Reviewers: #reviewers Maniphest Tasks: T401 Differential Revision: https://forge.softwareheritage.org/D46 | 13 June 2016, 14:00:03 UTC |
| 45eab09 | Quentin Campos | 06 June 2016, 14:34:07 UTC | Update an administration script to make it follow the changes of objstorage api | 13 June 2016, 09:35:06 UTC |
| 785592a | Quentin Campos | 06 June 2016, 14:31:32 UTC | Change the objstorage http server in order to follow the changes of the local version | 13 June 2016, 09:35:06 UTC |
| 96ff4ef | Quentin Campos | 06 June 2016, 14:30:32 UTC | Update the content integrity checker in order to follow objstorage changes | 13 June 2016, 09:35:06 UTC |
| 7c72d90 | Quentin Campos | 06 June 2016, 12:55:30 UTC | Update the storage module in order to follow objstorage refactoring Ref T433 | 13 June 2016, 09:35:06 UTC |
| c664995 | Quentin Campos | 06 June 2016, 12:33:06 UTC | Add an implementation of the object storage api and a test class | 13 June 2016, 09:35:06 UTC |
| 90c716d | Quentin Campos | 01 June 2016, 13:23:23 UTC | Create a base API that define the objstorage behavior Also add a mixin test class for the contract verification. | 13 June 2016, 09:35:06 UTC |
| 363f3e6 | Jordi BERTRAN DE BALANDA (swhintern) | 10 June 2016, 12:59:38 UTC | converters: update to reflect decoding logic moved to swh.core.utils | 10 June 2016, 12:59:38 UTC |
| bca87a1 | Quentin Campos | 01 June 2016, 12:15:34 UTC | Make the checker able to run as daemon or one-shot When ran as daemon, the checker run forever in background. When not as daemon, it only check for a single random batch of content. That could allow the checker to run as multiple one-shot with a cron. | 01 June 2016, 14:38:21 UTC |
| 4a18a04 | Quentin Campos | 01 June 2016, 10:13:07 UTC | Add the checker to the debian packaging paths Update the setup.py file in order to add the new created directory 'swh/storage/checker' | 01 June 2016, 11:18:52 UTC |
| ba83478 | Quentin Campos | 31 May 2016, 13:20:10 UTC | Improve content integrity checker's test coverage Add a test that verify if the content integrity checker can repair the files if the content is exclusively distributed among multiple servers. | 01 June 2016, 11:18:52 UTC |
| 413115c | Quentin Campos | 31 May 2016, 13:14:11 UTC | The checker now have the possibility to have multiple backup servers The config file now allows the syntax backup_url=url1, url2, url3, ... This allows to have more than one server to contact in case of corrupted content. | 01 June 2016, 11:18:52 UTC |
| 59ca1a3 | Quentin Campos | 31 May 2016, 14:43:35 UTC | Fix an object id encoding into a test of objstorage api This bug wasn't caught by the phabricator's test runs. | 31 May 2016, 15:08:26 UTC |
| d0ce099 | Quentin Campos | 27 May 2016, 14:30:24 UTC | Forgot to change the encoding into the tests. Reviewers: #reviewers Differential Revision: https://forge.softwareheritage.org/D34 | 27 May 2016, 14:36:26 UTC |
| eb8ada4 | Quentin Campos | 24 May 2016, 13:50:31 UTC | Add some tests for the content integrity checker Closes T423 | 27 May 2016, 12:24:29 UTC |
| e4881f0 | Quentin Campos | 23 May 2016, 13:26:30 UTC | Create a content integrity checker that runs in local to verify objects | 27 May 2016, 12:24:29 UTC |
| b216139 | Quentin Campos | 27 May 2016, 10:55:16 UTC | Also, add a get_random_contents access to the remote API | 27 May 2016, 12:24:29 UTC |
| 26e933b | Quentin Campos | 26 May 2016, 10:11:58 UTC | Add some methods to the object storage in order to allow a content integrity checker to run - restore_bytes method is needed to repair a content as the add_bytes methods does nothing to the content if the id is already in the file system - get_random_contents is used for the content integrity checker in order to have a stateless probabilistic method for the cerification scheduling | 27 May 2016, 11:32:03 UTC |
| 60a0be9 | Antoine R. Dumont (@ardumont) | 20 May 2016, 13:26:43 UTC | Add run_from_webserver function for objstorage api server This is the main function used when deployed | 20 May 2016, 13:26:43 UTC |
| b78a9fa | Antoine R. Dumont (@ardumont) | 20 May 2016, 13:24:44 UTC | Add unique identifier message on default api server route endpoints | 20 May 2016, 13:24:44 UTC |
| e522d1e | Quentin Campos | 20 May 2016, 10:00:49 UTC | Command line launch improvements Object storage and storage http servers now have the same --help description. Also, add a security check for argument type of the port that must be an int. Ref T402 | 20 May 2016, 11:40:15 UTC |
| 6941140 | Quentin Campos | 20 May 2016, 09:52:57 UTC | Add a way to launch archiver director from cl With click module, the archiver can now be launched in cl, e.g. by cron. Ref T240 | 20 May 2016, 11:40:14 UTC |
| 5c0979e | Quentin Campos | 19 May 2016, 22:05:51 UTC | Change test attribute in order to skip some tests | 19 May 2016, 22:06:34 UTC |
| 53fbe53 | Quentin Campos | 17 May 2016, 13:53:33 UTC | Update debian package dependencies Closes T404 | 19 May 2016, 22:06:19 UTC |
| 5dd32b1 | Quentin Campos | 17 May 2016, 14:42:16 UTC | Fix tests according to 37c8b88 (align extra_git_headers to extra_headers) Reviewers: #reviewers, ardumont Differential Revision: https://forge.softwareheritage.org/D28 | 17 May 2016, 14:42:54 UTC |
| 4e97e4c | Quentin Campos | 13 May 2016, 14:40:24 UTC | Improve test coverage of the archiver Closes T403 | 17 May 2016, 10:38:37 UTC |
| f92426d | Quentin Campos | 13 May 2016, 09:36:52 UTC | Fixes and improvements for T401 Summary: Ref T401 - Fix : The 'ongoing' status wasn't set to the content that were supposed to be in que archival queue. - Refactor : - Extract sql query from archive worker and place it into db module - Refactor in order to improve code quality & separation and to simplify unit testing. - Change the way the director's configuration is stored to make it easy to send the whole config to ArchiverWorker Reviewers: #reviewers Maniphest Tasks: T401 Differential Revision: https://forge.softwareheritage.org/D27 | 17 May 2016, 10:38:37 UTC |
| fc5ed93 | Quentin Campos | 13 May 2016, 09:29:16 UTC | Extract sql query from archive worker and place it into db module | 13 May 2016, 12:08:08 UTC |
| f74f923 | Quentin Campos | 13 May 2016, 08:43:37 UTC | Fix the status update of content during archival process The content status wasn't correclty set to 'ongoing'. The 'ongoing' status wasn't set to the content that were supposed to be in que archival queue. | 13 May 2016, 12:07:28 UTC |
| b1d99e1 | Quentin Campos | 12 May 2016, 13:08:50 UTC | Improve starting routines of storage & objstorage servers Summary: Use click in order to improve the routine. Closes T402 Reviewers: #reviewers, ardumont Maniphest Tasks: T402 Differential Revision: https://forge.softwareheritage.org/D26 | 12 May 2016, 13:10:03 UTC |
| 7afda21 | Quentin Campos | 12 May 2016, 10:43:55 UTC | Make the worker asynchronous optional Closes T406 | 12 May 2016, 11:26:57 UTC |
| 0e84344 | Quentin Campos | 11 May 2016, 14:10:47 UTC | Add asynchronous mode Closes T401 | 12 May 2016, 11:26:57 UTC |
| f0388f1 | Quentin Campos | 04 May 2016, 11:24:29 UTC | Add synchronous first implementation of the archiver Closes T400 | 12 May 2016, 11:26:57 UTC |
| e32124d | Quentin Campos | 29 April 2016, 13:51:36 UTC | Change database model and update swh.storage.db api in order to follow those changes | 12 May 2016, 11:08:49 UTC |
| 09c0f2f | Quentin Campos | 11 May 2016, 15:30:16 UTC | Change storage & objstorage server mains in order to change default port Related T402 Reviewers: #reviewers Differential Revision: https://forge.softwareheritage.org/D25 | 12 May 2016, 11:07:58 UTC |
| 37c8b88 | Nicolas Dandrimont | 11 May 2016, 12:33:00 UTC | converters: rename extra_git_headers in revision to extra_headers | 11 May 2016, 12:34:38 UTC |
| 2c719ee | Quentin Campos | 10 May 2016, 15:10:43 UTC | Change the server's imports to absolute ones Summary: As the servers are supposed to be launched, their imports cannot be relatives. Reviewers: #reviewers Differential Revision: https://forge.softwareheritage.org/D24 | 10 May 2016, 15:16:36 UTC |
| f0bda08 | Quentin Campos | 29 April 2016, 13:52:19 UTC | Add tests for the remote objstorage API and share some code with storage.api tests Add some nosetest unit-tests for the remote object storage client/server API, and perform some refactoring in the test of swh.storage.api in order to share code, due to very similar architecture. | 04 May 2016, 09:10:12 UTC |
| a3a7bd4 | Quentin Campos | 02 May 2016, 09:56:08 UTC | Add an http API for objstorage and refactor the shared code with storage.api Create an http API for swh.storage.objstorage and perform some refactoring in order to avoid code duplication with swh.storage.api | 04 May 2016, 09:10:12 UTC |
| 82b0e8a | Quentin Campos | 02 May 2016, 09:52:39 UTC | Change the objstorage file to a module | 04 May 2016, 09:10:12 UTC |
| ede4392 | Antoine R. Dumont (@ardumont) | 15 April 2016, 10:11:37 UTC | d/control: Fix homepage | 15 April 2016, 10:11:37 UTC |
| 4f2792d | Nicolas Dandrimont | 12 April 2016, 13:50:31 UTC | Move docs to the docs directory | 12 April 2016, 13:50:31 UTC |
| 24979b7 | Nicolas Dandrimont | 07 April 2016, 16:44:56 UTC | sql/upgrades/068: Add 067→068 migration script | 07 April 2016, 16:44:56 UTC |
| 740bef4 | Nicolas Dandrimont | 07 April 2016, 16:28:29 UTC | sql/swh-schema: add a fullname field to person The fullname field is the new canonical field for persons. name and email are only populated in an advisory capacity if they are parsed. We use this opportunity to cleanup the mess we did of SQL queries for releases and revisions, and stop using SELECT *. Closes T366 | 07 April 2016, 16:28:29 UTC |
| c5b259b | Nicolas Dandrimont | 07 April 2016, 14:03:52 UTC | sql: use tmp_bytea to retrieve information about revisions | 07 April 2016, 14:24:47 UTC |
| e06dfcf | Nicolas Dandrimont | 07 April 2016, 12:59:11 UTC | utils: latest fix_revisions_from_dump run | 07 April 2016, 12:59:11 UTC |
| 848fdd5 | Nicolas Dandrimont | 05 April 2016, 13:11:17 UTC | utils: add some tools to dump revisions and/or try to fix them | 05 April 2016, 13:13:06 UTC |
| 1cade4f | Nicolas Dandrimont | 05 April 2016, 13:10:44 UTC | db: return properly decoded lines even when using dict cursors | 05 April 2016, 13:13:06 UTC |
| e3a4bb7 | Antoine R. Dumont (@ardumont) | 02 April 2016, 16:26:48 UTC | Permit to filter other kinds of revision than git | 02 April 2016, 16:28:58 UTC |
| 1201a2b | Antoine R. Dumont (@ardumont) | 18 March 2016, 18:10:02 UTC | sql/upgrades/066: add 065-066 upgrade script - Add 'svn' revision type | 18 March 2016, 18:14:49 UTC |
| de2a586 | Nicolas Dandrimont | 09 March 2016, 15:49:45 UTC | sql/swh-schema: rename entity{_history}.doap to metadata | 09 March 2016, 15:58:24 UTC |
| 92e1f19 | Nicolas Dandrimont | 09 March 2016, 10:31:30 UTC | sql/swh-schema: convert all json fields to jsonb | 09 March 2016, 10:34:56 UTC |
| f962f70 | Nicolas Dandrimont | 08 March 2016, 16:27:48 UTC | sql/upgrades/063: add 062-063 upgrade script | 08 March 2016, 16:27:48 UTC |
| 438263f | Nicolas Dandrimont | 08 March 2016, 16:14:45 UTC | sql/json: update entity{,_history}.lister_metadata schemata | 08 March 2016, 16:14:45 UTC |
| 6711726 | Nicolas Dandrimont | 08 March 2016, 16:02:57 UTC | swh-schema: fold the entity.lister column in entity.lister_metadata | 08 March 2016, 16:03:39 UTC |
| bfc8984 | Stefano Zacchiroli | 04 March 2016, 17:07:46 UTC | revision.metadata JSON Schema: set allowed values to: tar, zip, dsc | 04 March 2016, 17:07:46 UTC |
| 0d0df37 | Stefano Zacchiroli | 04 March 2016, 16:48:18 UTC | revision.metadata JSON schema: origin_artifact is a list... ... of objects with the previously declared properties | 04 March 2016, 16:48:18 UTC |
| 6f3cb2e | Stefano Zacchiroli | 04 March 2016, 16:41:12 UTC | json schemas: fix whitespace issues (and a few typos) | 04 March 2016, 16:41:12 UTC |
| e2195ca | Stefano Zacchiroli | 04 March 2016, 16:28:57 UTC | sql/json/: first stab at documenting JSON(B) values using JSON Schema This is still very preliminar. I've merely extracted all examples from the current real DB, looking at all columns with type JSON or JSONB. Based on the examples I've documented all properties, applying the most reasonable types. The largest example is the revision.metadata column. A lot of documentation work is still missing; - required properties are not declared (but should, most notably in revision.metadata, given that a large part of that schema is valid only for Debian packages). Note that by default in JSON Schema all properties are *optional* so this aspect shoult not affect validity - title/description generic properties are seldomly used and should be used more - default generic property is never used, but probably should On the validation front we should: - validate all JSON(B) values in the DB against these schemas. Some upcoming validation errors are already evident. In particular: tarball artifacts do not have a "length" property, and symmetrically debian source package artifacts lack an "archive_type" property - validate the JSON Schema itself, as I might have gotten the spec wrong | 04 March 2016, 16:28:57 UTC |
| 9ec836e | Nicolas Dandrimont | 04 March 2016, 15:30:05 UTC | storage: add entity_get_one to retrieve a single entity | 04 March 2016, 15:30:05 UTC |
| 911086f | Nicolas Dandrimont | 04 March 2016, 13:46:08 UTC | sql/swh-func: make the entity update trigger lighter | 04 March 2016, 13:47:38 UTC |
| 6daf5fc | Nicolas Dandrimont | 04 March 2016, 11:28:59 UTC | sql/swh-func: drop temporary entity-related tables on commit | 04 March 2016, 13:44:25 UTC |
| c5374c3 | Nicolas Dandrimont | 04 March 2016, 11:05:24 UTC | db: allow passing UUIDs back and forth with the database | 04 March 2016, 11:05:24 UTC |
| e6c0959 | Nicolas Dandrimont | 25 February 2016, 14:19:15 UTC | storage: add new method object_find_by_sha1_git The new method looks for objects with the given sha1_gits and returns some information about them. | 25 February 2016, 14:21:11 UTC |
| 48ffd2a | Nicolas Dandrimont | 25 February 2016, 13:52:13 UTC | sql: factor out tmp_release_get and tmp_content_get into a single table | 25 February 2016, 13:57:12 UTC |
| 99e83ca | Antoine R. Dumont (@ardumont) | 11 February 2016, 11:34:33 UTC | Fix pep8 compliance | 11 February 2016, 11:34:33 UTC |
| 3172337 | Antoine R. Dumont (@ardumont) | 11 February 2016, 11:19:10 UTC | sql/upgrades/058: add 057 → 058 upgrade script | 11 February 2016, 11:33:36 UTC |
| 456f44d | Antoine R. Dumont (@ardumont) | 11 February 2016, 11:18:57 UTC | Open content_missing_per_sha1 | 11 February 2016, 11:33:36 UTC |
| a401d86 | Nicolas Dandrimont | 10 February 2016, 11:22:10 UTC | swh.storage: Allow storage of extra git headers in revisions We store the extra git headers (bytes) in the metadata->'extra_git_headers' field of the revision. The values are decoded as if they were utf-8 strings. When an invalid sequence is encountered, the bytes are encoded as a \x escape. We also encode NUL bytes as such, as U+0000 is invalid in a JSON string and can't be stored in PostgreSQL. | 10 February 2016, 11:22:10 UTC |
| 26ce6d8 | Nicolas Dandrimont | 08 February 2016, 14:31:10 UTC | sql/swh-schema: add some constraints to occurrence{_history,} | 08 February 2016, 14:37:56 UTC |
| 2e29497 | Nicolas Dandrimont | 08 February 2016, 14:24:30 UTC | sql/swh-func: Properly deduplicate occurrence_history | 08 February 2016, 14:26:06 UTC |
| c64b05e | Nicolas Dandrimont | 08 February 2016, 14:15:24 UTC | sql/upgrades/055: add 054 → 055 upgrade script | 08 February 2016, 14:15:24 UTC |
| f6041b6 | Nicolas Dandrimont | 08 February 2016, 14:13:46 UTC | sql/swh-func: Properly increment the visit numbers for origin_visit | 08 February 2016, 14:13:46 UTC |
| 2a90566 | Nicolas Dandrimont | 08 February 2016, 14:13:01 UTC | sql/upgrades/054: Add 053 → 054 update script | 08 February 2016, 14:13:01 UTC |
| 12556ab | Nicolas Dandrimont | 08 February 2016, 12:15:25 UTC | swh-func: add prefixes to the occurrence update function names | 08 February 2016, 12:15:25 UTC |
| da91787 | Nicolas Dandrimont | 05 February 2016, 14:58:25 UTC | sql/upgrades/053: add 52 → 53 upgrade script | 05 February 2016, 14:58:25 UTC |
| 9afcea5 | Nicolas Dandrimont | 05 February 2016, 14:54:31 UTC | sql/swh-func: uniformize the tmp_release_get and tmp_release tables | 05 February 2016, 14:54:31 UTC |
| 137f82c | Nicolas Dandrimont | 05 February 2016, 10:12:55 UTC | sql/upgrades/052: add 051 → 052 upgrade script | 05 February 2016, 10:12:55 UTC |
