https://github.com/SoftwareHeritage/swh-storage

sort by:
Revision Author Date Message Commit Date
80b4644 Update version requirements for swh.model 08 October 2018, 15:03:46 UTC
7ae19db add build/ to gitignore 08 October 2018, 10:22:39 UTC
63c924d setup: prepare for pypi upload related to T1242 08 October 2018, 10:22:32 UTC
b1eaa9a docs: merge both README files in a single README.md also documement the usage of pifpaf to run pg tests 05 October 2018, 14:38:22 UTC
fb794bf Add requirements-test.txt 05 October 2018, 10:56:16 UTC
3d1b95e Remove the occurrence table Summary: This table is obsolete, and was never really used in the first place. This solves the first half of T839. Depends on D471. Test Plan: rebuild-testdata and make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D472 03 October 2018, 16:55:46 UTC
842b7a3 Remove some unused SQL functions Summary: These functions aren't used by any known clients Test Plan: rebuild-testdata + make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D471 03 October 2018, 16:34:31 UTC
5193e6d Drop backwards-compatibility features between snapshots and occurrences Summary: Old occurrences have been fully migrated to snapshots now, so backwards compatibility is not needed anymore. That's one more step towards the removal of occurrences (T830). This changes the origin_visit_get_by method to stop returning a list of occurrences. The only user of this method is the api/1/origin/{x}/visit/{y} view of the web API, which will need to be updated for T1207 anyway. Test Plan: integration tests updated for origin_visit_get_by; Running the web app with the updated storage only yields the expected change in behavior for the origin/visit API view, navigation is unaffected. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D466 03 October 2018, 16:18:42 UTC
4b1db40 doc: use inkscape instead of dia to generate pdf and svg image files 03 October 2018, 13:44:09 UTC
8e545c7 SQL schema: uniformize comments Summary: Partial cherry-pick of ddb37a5896d04a95cafee413d048bea8df265379 Test Plan: rebuild-testdata and make test report no changes Reviewers: #reviewers! Subscribers: zack Differential Revision: https://forge.softwareheritage.org/D463 03 October 2018, 13:14:04 UTC
1751385 Remove unused function release_get_by Summary: release_get_by has never been used except by its own tests, and it still uses the legacy occurrence_* tables that we want to remove (T830). Test Plan: - make test in swh-environment passes - grep -r release_get_by swh-* yields no result Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D460 03 October 2018, 12:32:27 UTC
16ffb6e docs: make sure images are built/cleaned on sphinx targets 03 October 2018, 09:11:19 UTC
7fa1a34 images/Makefile: fix filenames they have been broken since the recent rename 03 October 2018, 09:10:46 UTC
f5dd8ac nicer landing page, pointing to homepage and API doc Closes T1194 02 October 2018, 08:53:49 UTC
003e1a7 docs: add title and brief module description 01 October 2018, 13:47:54 UTC
2eced78 algos.dir_iterators: Implement iterator protocol in DirectoryIterator - add Python iterator protocol support in the DirectoryIterator class in order to easily visit in a recursive way any directory stored in the archive. - add convenient function dir_iterator wrapping the instantiation of the DirectoryIterator class - add tests Related T1177 28 September 2018, 12:32:17 UTC
9adaed7 algos.dir_iterators: Fix handling of empty directory Two issues found regarding the way empty directories were handled: - _empty_dir_hash variable did not have correct type (str instead of bytes) so empty directory test based on hash comparison was always failing - in the step method of DirectoryIterator, no need to push a new frame for an empty directory as this will stop the iteration 28 September 2018, 12:32:17 UTC
3a86982 sql/bin/db-init: remove createuser and privileges (we now assume a peer superuser) 25 September 2018, 11:05:21 UTC
1286dfa sql/bin/db-init: simpler usage with sensible defaults 25 September 2018, 10:52:10 UTC
332a1e1 use /usr/bin/env python3 as shebang, to be nice to virtualenv 19 September 2018, 15:19:44 UTC
e01f26c swh.storage.db: Fix origin_visit_get_latest_snapshot query Prior to this fix, storage.snapshot_get_latest(origin) was returning the first snapshot instead of the last one. 19 September 2018, 09:32:04 UTC
384b120 docs: add description of the archive copies layout diagram 07 September 2018, 11:28:56 UTC
a66745a docs: add layout diagram describing existing archive copies 07 September 2018, 09:39:44 UTC
46822cb Add snapshot to the statistics endpoint counter Related T1180 06 September 2018, 18:41:18 UTC
8ccac07 docs: add general intro, drop bogus reference to archiver blueprint 06 September 2018, 10:19:41 UTC
19c0344 algos/diff.py: fix typo in comment brought to you by codespell 06 September 2018, 10:19:20 UTC
329d0f9 storage: set timeout to 20s for directory_ls 05 September 2018, 12:30:46 UTC
9c39449 origin_search: Add option to filter out origins with no visit 02 August 2018, 13:01:24 UTC
9e5a8b7 swh.storage.storage: origin_add returns updated list of dict with id Prior to this commit, this returned only the list of new ids. This is currently not used anywhere in our stack. Would it have been, this would have force the client to try and be smart for dealing with ids. 27 July 2018, 13:57:13 UTC
a167960 Migrate away from temporary tables for read queries Summary: To do so, we import a function from a recent version of psycopg2, execute_values, which can execute queries efficiently with a list of values. We also scale the cursor back from having things in SQL functions towards having things inside the db.py database "backend". This will make it easier to iterate, as we won't have to deploy function changes to twenty different databases. After these changes, testing the web UI on a physical replica works. Close T1073 Test Plan: Local integration tests are happy; Navigating the frontend backed by a physical replica seems to be okay now. Reviewers: #reviewers! Maniphest Tasks: T1073 Differential Revision: https://forge.softwareheritage.org/D340 05 June 2018, 11:56:09 UTC
7c85120 Remove now unused function directory_get Summary: As of D337, directory_get has no users left; Time to remove it. Test Plan: make test in the toplevel environment still works Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D339 04 June 2018, 15:36:44 UTC
9f2b266 Make sure there are no pagination issues for origin_visit_get Although most visits should be sequential, and the date should be monotonic, re-importing an old snapshot as a new visit would break the pagination. 30 May 2018, 12:31:36 UTC
c4494aa Fix codespell issues 29 May 2018, 15:30:18 UTC
779cfb5 Don't assign the objstorage result to a variable 29 May 2018, 15:26:24 UTC
2c17350 Rewrite concurrent.futures import to shorten lines 29 May 2018, 15:22:31 UTC
79cc3f6 Better test coverage for content_missing 29 May 2018, 14:38:50 UTC
70574c5 Don't use a temporary table to fetch info about contents 28 May 2018, 16:04:44 UTC
6d5d999 Make the db_transaction{,_generator} decorators support client options Summary: Allow adding server-side statement timeouts for database operations Test Plan: make test still works Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D334 28 May 2018, 13:18:26 UTC
d826f75 swh.storage.api.client: Permit to specify the query timeout option Related T1061 24 May 2018, 10:02:11 UTC
77e69ec Use a concurrent.future to parallelize objstorage and storage addition 12 May 2018, 15:49:44 UTC
4a9b623 test_api_client: stop leaking directories Summary: When mkdtemp is called, shutil.rmtree must be called as well Test Plan: Look at /tmp before and after running tests, notice no new directories instead of 60. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D331 11 May 2018, 13:42:09 UTC
721d088 Add test to ensure storage/objstorage consistency is kept at all times Summary: The behavior of storage when the underlying objstorage had an exception was never actually tested. This new test weeded out a bug in the threaded implementation for copy_to. Test Plan: the new test passes Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D330 11 May 2018, 10:13:44 UTC
18c9dad Only instantiate the storage backend once per import This allows connection reuse for postgresql and potential remote backends such as for the object storage, rather than reinitiating all connections on every request. 09 May 2018, 14:43:32 UTC
5a2de1c Use thread-aware psycopg2 connection pooling for database access Summary: This allows to use swh.storage with a modicum of concurrency Test Plan: clearly, make test should still pass Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D325 07 May 2018, 13:52:35 UTC
6c93693 Stop using the storage.db attribute directly Summary: Add a level of indirection to allow swapping out the implementation of the db attribute Test Plan: once again, make test keeps on working Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D324 07 May 2018, 13:46:21 UTC
e1e5025 Make the storage test fixture connect to postgres itself Summary: This avoids reusing a potentially stale connection handle. Also allows testing potential connection pooling behavior. This forces us to do proper cursor sanitation as well, a bunch of "transactional" operations weren't actually transactional. Test Plan: another round of make test still working Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D323 07 May 2018, 13:43:27 UTC
ff2d37d Deallocate storage when the tests teardown Summary: Helps avoid lingering postgresql connections when a test fails Test Plan: make test still works ;) Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D322 07 May 2018, 13:35:03 UTC
9ca8f7a Make sure schema changes get committed when doing the AlteringSchemaTests 07 May 2018, 13:31:21 UTC
03eea50 tests: Move test fixture to swh.core.tests.server_testing Related T1036 25 April 2018, 13:00:25 UTC
b4147ec fix typo in docstrings/comments (tnx codespell) 12 March 2018, 11:01:52 UTC
916d9cc Remove file that should not have been committed and update gitignore 27 February 2018, 09:40:18 UTC
d8ad992 storage: Add methods to compute directories/revisions diff This commit adds the implementation of an efficient algorithm for comparing two directory trees in order to compute the list of introduced file changes in terms of addition / deletion / modification/ renaming. It can be found in the diff module located in the new namespace swh.storage.algos That algorithm is used to extend the storage API with the following methods: - diff_directories: compute diff between two arbitrary directories - diff_revisions: compute diff between two arbitrary revisions - diff_revision: compute diff between a revision and its first parent Related T921 Closes D295 20 February 2018, 12:19:09 UTC
6b80cff Add a new table for "bucketed" object counts This table allows counting objects by bucket, keeping the transactions for counting objects short (a few dozen seconds at most). Also add a "single_update" boolean field to the main object_counts table to be able to discriminate tables that are counted via buckets and tables counted on one go. The main table is updated every 256 counted buckets to avoid too much churn on the table. Close T962. 19 February 2018, 18:30:41 UTC
86d68a6 doc: update table clusters in SQL diagram 09 February 2018, 11:48:40 UTC
22ee4c5 swh.storage.content_missing: Improve docstring 08 February 2018, 18:27:29 UTC
9082524 api.client: switch back_compat for snapshot_add as well 06 February 2018, 14:27:10 UTC
ada0d38 storage: swap back_compat switch for snapshot_add 06 February 2018, 14:25:36 UTC
9f84cec Add function to retrieve latest snapshot based on origin filtering This function does _not_ have any backwards compatibility provisions. 31 January 2018, 14:19:47 UTC
32e89b0 Add snapshot information when fetching origin_visits from the database 31 January 2018, 10:49:14 UTC
d6c3f65 sql doc Makefile: (try to) dropdb before creating it this way doc (re)building is more robust 19 January 2018, 13:22:11 UTC
b32d2c9 Add a way to remove the backwards-compatibility feature of snapshots 19 January 2018, 10:54:00 UTC
ab0e90c doc Makefile: clean SQL autodoc upon "clean" 19 January 2018, 10:30:42 UTC
d163501 SQL doc Makefile: ensure destdir gets created wherever needed 19 January 2018, 10:28:35 UTC
985a912 sql/Makefile: add missing deps on pdf/svg DB schema generation 16 January 2018, 15:48:53 UTC
2e033d0 sql: don't depend on plpython3u, we don't use it 16 January 2018, 14:36:32 UTC
56e0092 db-init: enforce UTF8 locale for jsonb fanciness Related T918 10 January 2018, 11:16:47 UTC
f5acfeb sql: Fix missing migration instructions Related 08121f7 09 January 2018, 10:46:46 UTC
7b5c111 db-init: actually grant privileges on tables and sequences 06 January 2018, 14:03:33 UTC
ca308c7 sql/bin/db-init: new helper script to setup development DB 06 January 2018, 13:52:20 UTC
cf6cac7 SQL: move definition of hash_sha1() function to swh-func.sql rationale: swh-init.sql should only contain stuff that requires being Postgres superuser 06 January 2018, 09:21:51 UTC
a0ac20a db: use a thread instead of a tmpfile to COPY data to postgresql Summary: Avoids writing a bunch of temporary files for no good reason. T866 was a symptom of this. Test Plan: No regressions in the unit tests Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D282 21 December 2017, 18:48:16 UTC
419d135 db: move external imports further down 21 December 2017, 18:45:19 UTC
221f8b1 Add 116->117 SQL upgrade script 21 December 2017, 18:40:57 UTC
11fcb82 Refactor swh_snapshot_add to hit more indexes 21 December 2017, 18:38:50 UTC
76e3670 db: remove bare except: 19 December 2017, 15:22:34 UTC
afcbe94 sql/upgrades: add 115 -> 116 swh-enums: Reference new mercurial revision type Related T329 19 December 2017, 13:27:40 UTC
a455fc0 docs: reference original db-schema.svg without symlink indirection the new sphinx seems to be confused by relative symlinks when we merge all docs together 19 December 2017, 11:06:59 UTC
655bcb6 doc generation: use more specific db-schema basename for images was: "swh" this way we can ship the images as is in the global doc, with minimal clash risks 19 December 2017, 11:06:04 UTC
73256e6 Add 114->115 SQL upgrade script 18 December 2017, 15:12:01 UTC
36aee70 Fix documentation formatting for tool_add/tool_get 15 December 2017, 14:48:16 UTC
f4ea97c Add snapshot models Summary: Snapshots are the new, improved occurrences; They're the topmost object in the Software Heritage Merkle Tree, and represent a full picture of an origin at a given time. Snapshots contain a list of named pointers to objects in the Software Heritage archive, as well as an intrinsic identifier. The full specification is supported: pointers to all types of objects, dangling pointers, as well as alias branches. They're implemented with a somewhat classic fully normalised model; Foreign keys use a sha1_git, which makes more sense regarding pointing at non-existent objects, at the expense of some economies of size. Backwards compatibility both ways with occurrences is ensured: when adding a snapshot linked to an origin visit, the corresponding occurrences are created in occurrence_history; when querying the snapshot for an origin visit where we haven't generated the snapshot yet, a virtual snapshot with id None is returned. This lets us migrate to the new tables gently. Close T567. Test Plan: Integration tests are included. Reviewers: zack, #reviewers! Maniphest Tasks: T567 Differential Revision: https://forge.softwareheritage.org/D268 15 December 2017, 14:36:58 UTC
0bfd709 sql/upgrades: add 113 -> 114 Finalize it really. Related T873 07 December 2017, 16:58:55 UTC
20d59c3 swh.storage.api.server: Do not hardcode the config file Related T871 07 December 2017, 13:14:28 UTC
62e8472 d/rules: Fix double export instruction 07 December 2017, 12:18:56 UTC
08121f7 swh-storage: Rename indexer_configuration to tool Related T871 07 December 2017, 08:45:24 UTC
021512b swh-storage: Migrate indexer model to its own model Related T871 07 December 2017, 08:44:26 UTC
63858e5 Add method to storage for searching origins This adds method 'origin_search' to storage enabling to search for origins whose urls contain a string pattern or match a regular expression Related T848 05 December 2017, 10:47:54 UTC
a4abad0 schemata: Add missing __init__.py for build package purposes 24 November 2017, 10:13:48 UTC
c33ad51 sql/upgrades: add 112 -> 113 Related T851 17 November 2017, 14:05:53 UTC
8311a4f swh.storage: Open indexer_configuration_add endpoint Related T851 17 November 2017, 13:46:39 UTC
5a0ddd6 swh.storage.tests: Fix broken content_mimetype tests Due to a new value in db for the same tool. Related 35253443fe0bd792e84f3ce939ee20e7eed52f9b Related T849 17 November 2017, 10:28:49 UTC
3525344 swh-data: Add new content mimetype's indexer configuration Related T849 15 November 2017, 14:36:19 UTC
775a1b8 add __init__.py for tests 10 November 2017, 13:16:40 UTC
c5740fd origin_visit_get: make order repeatable 10 November 2017, 13:16:40 UTC
c4ec821 Add missing visit key for occurrence_add documentation 10 November 2017, 13:16:40 UTC
9b29324 Make unique indices actually unique and vice versa 09 November 2017, 17:11:11 UTC
f87a530 Add 110->111 SQL upgrade script 06 November 2017, 12:46:56 UTC
e7a7b56 Remove unused content provenance cache tables 03 November 2017, 15:31:36 UTC
75f4edd sql/upgrades/110: add metadata tables to SQL schema 03 November 2017, 11:41:28 UTC
477bd44 docs: add absolute anchor to documentation index 02 November 2017, 10:09:27 UTC
back to top