sort by:
Revision Author Date Message Commit Date
1c74f6d tox.ini: Align configuration file with other Also, without this, this failed because somehow, something is not finding requirements-test.txt. ``` Could not open requirements file: [Errno 2] No such file or directory: ' requirements-test.txt' ``` 24 October 2018, 13:42:12 UTC
7628710 SQL schema: update to reflect recent changes recent changes include most notably the removal of occurrence-related tables, but also the addition of table columns accumulated over time Closes T1283 24 October 2018, 12:35:02 UTC
a6742b7 DB schema doc: error out if SQL schema cannot be found 24 October 2018, 12:34:52 UTC
582d72d docs Makefile: only echo SQL queries on error 24 October 2018, 12:22:48 UTC
5fe9221 docs Makefile: forward "distclean" to sql dir 24 October 2018, 12:20:01 UTC
b91482a gitignore and remove generate archive copies diagram Closes T1269 24 October 2018, 12:00:01 UTC
242f985 Don't run pytest in the docs directory 23 October 2018, 14:48:15 UTC
e77d809 tests: increase deadline for CI seems to run slower on there. 22 October 2018, 16:06:59 UTC
076144e tests: assertEquals -> assertEqual 22 October 2018, 14:46:34 UTC
32ef182 tests: use pytest and add a simple tox file related to T1261 22 October 2018, 14:45:08 UTC
5b38656 flake8 22 October 2018, 14:32:18 UTC
a8fbce5 Warn when using db_transaction/db_transaction_generator where you shouldn't. Summary: I did it, it caused closed cursors that hang the process, that's hard to debug. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont Differential Revision: https://forge.softwareheritage.org/D556 22 October 2018, 09:03:36 UTC
d211615 Mark snapshot tests as needing the database 18 October 2018, 16:30:44 UTC
b2d4921 Mark minimum hypothesis version for datetimes 18 October 2018, 16:27:56 UTC
ec7a119 Add hypothesis to debian build-dependencies 18 October 2018, 16:25:07 UTC
ad92248 Add a function to retrieve a full snapshot Summary: This function wraps the paginating branch listing API to do its work. It's in swh.storage.algos because it should be done client-side rather than server-side. The tests are fairly simple, and mostly add a bunch of tooling around hypothesis to generate data that conforms to the swh data schema. Those things could probably be adequately moved to a new submodule of swh.model. Test Plan: new tests added Reviewers: anlambert Differential Revision: https://forge.softwareheritage.org/D551 18 October 2018, 14:57:55 UTC
7cd7597 Rewrite snapshot_get in terms of snapshot_get_branches Summary: This increases uniformity and reduces code duplication between the two functions. The addition of the 'next_branch' return value in snapshot_get_branches allows callers to get branches incrementally in a loop which is friendlier. Depends on D545 Test Plan: tests have been updated to add the new next_branch field Reviewers: anlambert Differential Revision: https://forge.softwareheritage.org/D546 17 October 2018, 14:56:42 UTC
0d52bf4 Rewrite snapshot pagination tests declaratively Summary: This looks less confusing than removing a bunch of values from the test data. Test Plan: make test still passes Reviewers: anlambert Differential Revision: https://forge.softwareheritage.org/D545 17 October 2018, 14:56:07 UTC
80cd420 packaging: bump swh-core dependency to 0.0.44 17 October 2018, 13:56:52 UTC
9ce5f78 README: swh-storage-testdata is not required any more 17 October 2018, 13:56:13 UTC
a9c32d2 sql: adapt Makefile for sql file renames and make it pifpaf compliant ie. delete stamps when executed via pifpaf for the db setup. 17 October 2018, 13:46:36 UTC
ee27736 kill sql/db-init in favor of swh-core's new swh-db-init tool 17 October 2018, 13:46:36 UTC
c4cdb64 tests: refactor the StorageTestFixture class and move sql files in the package so we can build the db setup sql file on the fly: - make it inherit from SingleDbTestFixture, so we can - move sql files in swh/storage/sql/ - get rid og the setUpClass static method to reuse the one from SingleDbTestFixture, - apply the required modifications to tests based on this StorageTestFixture. Warnings: - this requires D538, - this will break tests in swh-vault. Also add README.md to the MANIFEST.in file so it does not require a too recent setuptool. 17 October 2018, 13:46:36 UTC
8f9af9b Remove some leftover compatibility code 17 October 2018, 13:16:31 UTC
2d5efac Reorder upgrade statements so that they actually pass 17 October 2018, 12:44:42 UTC
4f05900 Remove redundant removals from the 122->123 upgrade 17 October 2018, 12:22:08 UTC
435ebcd Drop table occurrence_history Summary: The migration to snapshots has been completed, and all users have been migrated. Depends on D534. Close T830. Test Plan: make test after a testdata rebuild is ok Reviewers: zack Maniphest Tasks: T830 Differential Revision: https://forge.softwareheritage.org/D535 15 October 2018, 14:22:17 UTC
b486063 Drop confusing functions revision_get_by/revision_log_by Summary: The combination of timestamp search for an occurrence with branch name filtering never got ported to the snapshot model. As this function only has a single user, which is a single endpoint deep in the public-facing API, let's just drop it. revision_log_by uses the same filter but gives a log of parent revisions, and has never been used by any client. This will break this API endpoint, which is tracked in T1221, but iff we want to keep it, I think it can be easily reimplemented by composing origin_visit_get_by and the new snapshot_get_branches function. Test Plan: It's dead, Jim. Reviewers: zack Differential Revision: https://forge.softwareheritage.org/D534 15 October 2018, 13:58:52 UTC
3a59229 Rename test methods to test_* to allow py.test collection Summary: Part of T1261 Test Plan: Make test runs the same number of tests before and after Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D522 12 October 2018, 09:57:14 UTC
65e6b69 Drop unused entity tables Summary: Close T838 Test Plan: Remove all the tests! Reviewers: #reviewers! Maniphest Tasks: T838 Differential Revision: https://forge.softwareheritage.org/D509 10 October 2018, 14:25:54 UTC
ee40117 Merge remote-tracking branch 'origin/master' 10 October 2018, 13:55:36 UTC
e9e5e73 storage: Enable to paginate, filter and count snapshot content Related T1207 09 October 2018, 13:28:34 UTC
80b4644 Update version requirements for swh.model 08 October 2018, 15:03:46 UTC
7ae19db add build/ to gitignore 08 October 2018, 10:22:39 UTC
63c924d setup: prepare for pypi upload related to T1242 08 October 2018, 10:22:32 UTC
b1eaa9a docs: merge both README files in a single README.md also documement the usage of pifpaf to run pg tests 05 October 2018, 14:38:22 UTC
fb794bf Add requirements-test.txt 05 October 2018, 10:56:16 UTC
3d1b95e Remove the occurrence table Summary: This table is obsolete, and was never really used in the first place. This solves the first half of T839. Depends on D471. Test Plan: rebuild-testdata and make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D472 03 October 2018, 16:55:46 UTC
842b7a3 Remove some unused SQL functions Summary: These functions aren't used by any known clients Test Plan: rebuild-testdata + make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D471 03 October 2018, 16:34:31 UTC
5193e6d Drop backwards-compatibility features between snapshots and occurrences Summary: Old occurrences have been fully migrated to snapshots now, so backwards compatibility is not needed anymore. That's one more step towards the removal of occurrences (T830). This changes the origin_visit_get_by method to stop returning a list of occurrences. The only user of this method is the api/1/origin/{x}/visit/{y} view of the web API, which will need to be updated for T1207 anyway. Test Plan: integration tests updated for origin_visit_get_by; Running the web app with the updated storage only yields the expected change in behavior for the origin/visit API view, navigation is unaffected. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D466 03 October 2018, 16:18:42 UTC
4b1db40 doc: use inkscape instead of dia to generate pdf and svg image files 03 October 2018, 13:44:09 UTC
8e545c7 SQL schema: uniformize comments Summary: Partial cherry-pick of ddb37a5896d04a95cafee413d048bea8df265379 Test Plan: rebuild-testdata and make test report no changes Reviewers: #reviewers! Subscribers: zack Differential Revision: https://forge.softwareheritage.org/D463 03 October 2018, 13:14:04 UTC
1751385 Remove unused function release_get_by Summary: release_get_by has never been used except by its own tests, and it still uses the legacy occurrence_* tables that we want to remove (T830). Test Plan: - make test in swh-environment passes - grep -r release_get_by swh-* yields no result Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D460 03 October 2018, 12:32:27 UTC
16ffb6e docs: make sure images are built/cleaned on sphinx targets 03 October 2018, 09:11:19 UTC
7fa1a34 images/Makefile: fix filenames they have been broken since the recent rename 03 October 2018, 09:10:46 UTC
f5dd8ac nicer landing page, pointing to homepage and API doc Closes T1194 02 October 2018, 08:53:49 UTC
003e1a7 docs: add title and brief module description 01 October 2018, 13:47:54 UTC
2eced78 algos.dir_iterators: Implement iterator protocol in DirectoryIterator - add Python iterator protocol support in the DirectoryIterator class in order to easily visit in a recursive way any directory stored in the archive. - add convenient function dir_iterator wrapping the instantiation of the DirectoryIterator class - add tests Related T1177 28 September 2018, 12:32:17 UTC
9adaed7 algos.dir_iterators: Fix handling of empty directory Two issues found regarding the way empty directories were handled: - _empty_dir_hash variable did not have correct type (str instead of bytes) so empty directory test based on hash comparison was always failing - in the step method of DirectoryIterator, no need to push a new frame for an empty directory as this will stop the iteration 28 September 2018, 12:32:17 UTC
3a86982 sql/bin/db-init: remove createuser and privileges (we now assume a peer superuser) 25 September 2018, 11:05:21 UTC
1286dfa sql/bin/db-init: simpler usage with sensible defaults 25 September 2018, 10:52:10 UTC
332a1e1 use /usr/bin/env python3 as shebang, to be nice to virtualenv 19 September 2018, 15:19:44 UTC
e01f26c swh.storage.db: Fix origin_visit_get_latest_snapshot query Prior to this fix, storage.snapshot_get_latest(origin) was returning the first snapshot instead of the last one. 19 September 2018, 09:32:04 UTC
384b120 docs: add description of the archive copies layout diagram 07 September 2018, 11:28:56 UTC
a66745a docs: add layout diagram describing existing archive copies 07 September 2018, 09:39:44 UTC
46822cb Add snapshot to the statistics endpoint counter Related T1180 06 September 2018, 18:41:18 UTC
8ccac07 docs: add general intro, drop bogus reference to archiver blueprint 06 September 2018, 10:19:41 UTC
19c0344 algos/diff.py: fix typo in comment brought to you by codespell 06 September 2018, 10:19:20 UTC
329d0f9 storage: set timeout to 20s for directory_ls 05 September 2018, 12:30:46 UTC
9c39449 origin_search: Add option to filter out origins with no visit 02 August 2018, 13:01:24 UTC
9e5a8b7 swh.storage.storage: origin_add returns updated list of dict with id Prior to this commit, this returned only the list of new ids. This is currently not used anywhere in our stack. Would it have been, this would have force the client to try and be smart for dealing with ids. 27 July 2018, 13:57:13 UTC
a167960 Migrate away from temporary tables for read queries Summary: To do so, we import a function from a recent version of psycopg2, execute_values, which can execute queries efficiently with a list of values. We also scale the cursor back from having things in SQL functions towards having things inside the db.py database "backend". This will make it easier to iterate, as we won't have to deploy function changes to twenty different databases. After these changes, testing the web UI on a physical replica works. Close T1073 Test Plan: Local integration tests are happy; Navigating the frontend backed by a physical replica seems to be okay now. Reviewers: #reviewers! Maniphest Tasks: T1073 Differential Revision: https://forge.softwareheritage.org/D340 05 June 2018, 11:56:09 UTC
7c85120 Remove now unused function directory_get Summary: As of D337, directory_get has no users left; Time to remove it. Test Plan: make test in the toplevel environment still works Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D339 04 June 2018, 15:36:44 UTC
9f2b266 Make sure there are no pagination issues for origin_visit_get Although most visits should be sequential, and the date should be monotonic, re-importing an old snapshot as a new visit would break the pagination. 30 May 2018, 12:31:36 UTC
c4494aa Fix codespell issues 29 May 2018, 15:30:18 UTC
779cfb5 Don't assign the objstorage result to a variable 29 May 2018, 15:26:24 UTC
2c17350 Rewrite concurrent.futures import to shorten lines 29 May 2018, 15:22:31 UTC
79cc3f6 Better test coverage for content_missing 29 May 2018, 14:38:50 UTC
70574c5 Don't use a temporary table to fetch info about contents 28 May 2018, 16:04:44 UTC
6d5d999 Make the db_transaction{,_generator} decorators support client options Summary: Allow adding server-side statement timeouts for database operations Test Plan: make test still works Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D334 28 May 2018, 13:18:26 UTC
d826f75 swh.storage.api.client: Permit to specify the query timeout option Related T1061 24 May 2018, 10:02:11 UTC
77e69ec Use a concurrent.future to parallelize objstorage and storage addition 12 May 2018, 15:49:44 UTC
4a9b623 test_api_client: stop leaking directories Summary: When mkdtemp is called, shutil.rmtree must be called as well Test Plan: Look at /tmp before and after running tests, notice no new directories instead of 60. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D331 11 May 2018, 13:42:09 UTC
721d088 Add test to ensure storage/objstorage consistency is kept at all times Summary: The behavior of storage when the underlying objstorage had an exception was never actually tested. This new test weeded out a bug in the threaded implementation for copy_to. Test Plan: the new test passes Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D330 11 May 2018, 10:13:44 UTC
18c9dad Only instantiate the storage backend once per import This allows connection reuse for postgresql and potential remote backends such as for the object storage, rather than reinitiating all connections on every request. 09 May 2018, 14:43:32 UTC
5a2de1c Use thread-aware psycopg2 connection pooling for database access Summary: This allows to use swh.storage with a modicum of concurrency Test Plan: clearly, make test should still pass Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D325 07 May 2018, 13:52:35 UTC
6c93693 Stop using the storage.db attribute directly Summary: Add a level of indirection to allow swapping out the implementation of the db attribute Test Plan: once again, make test keeps on working Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D324 07 May 2018, 13:46:21 UTC
e1e5025 Make the storage test fixture connect to postgres itself Summary: This avoids reusing a potentially stale connection handle. Also allows testing potential connection pooling behavior. This forces us to do proper cursor sanitation as well, a bunch of "transactional" operations weren't actually transactional. Test Plan: another round of make test still working Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D323 07 May 2018, 13:43:27 UTC
ff2d37d Deallocate storage when the tests teardown Summary: Helps avoid lingering postgresql connections when a test fails Test Plan: make test still works ;) Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D322 07 May 2018, 13:35:03 UTC
9ca8f7a Make sure schema changes get committed when doing the AlteringSchemaTests 07 May 2018, 13:31:21 UTC
03eea50 tests: Move test fixture to swh.core.tests.server_testing Related T1036 25 April 2018, 13:00:25 UTC
b4147ec fix typo in docstrings/comments (tnx codespell) 12 March 2018, 11:01:52 UTC
916d9cc Remove file that should not have been committed and update gitignore 27 February 2018, 09:40:18 UTC
d8ad992 storage: Add methods to compute directories/revisions diff This commit adds the implementation of an efficient algorithm for comparing two directory trees in order to compute the list of introduced file changes in terms of addition / deletion / modification/ renaming. It can be found in the diff module located in the new namespace swh.storage.algos That algorithm is used to extend the storage API with the following methods: - diff_directories: compute diff between two arbitrary directories - diff_revisions: compute diff between two arbitrary revisions - diff_revision: compute diff between a revision and its first parent Related T921 Closes D295 20 February 2018, 12:19:09 UTC
6b80cff Add a new table for "bucketed" object counts This table allows counting objects by bucket, keeping the transactions for counting objects short (a few dozen seconds at most). Also add a "single_update" boolean field to the main object_counts table to be able to discriminate tables that are counted via buckets and tables counted on one go. The main table is updated every 256 counted buckets to avoid too much churn on the table. Close T962. 19 February 2018, 18:30:41 UTC
86d68a6 doc: update table clusters in SQL diagram 09 February 2018, 11:48:40 UTC
22ee4c5 swh.storage.content_missing: Improve docstring 08 February 2018, 18:27:29 UTC
9082524 api.client: switch back_compat for snapshot_add as well 06 February 2018, 14:27:10 UTC
ada0d38 storage: swap back_compat switch for snapshot_add 06 February 2018, 14:25:36 UTC
9f84cec Add function to retrieve latest snapshot based on origin filtering This function does _not_ have any backwards compatibility provisions. 31 January 2018, 14:19:47 UTC
32e89b0 Add snapshot information when fetching origin_visits from the database 31 January 2018, 10:49:14 UTC
d6c3f65 sql doc Makefile: (try to) dropdb before creating it this way doc (re)building is more robust 19 January 2018, 13:22:11 UTC
b32d2c9 Add a way to remove the backwards-compatibility feature of snapshots 19 January 2018, 10:54:00 UTC
ab0e90c doc Makefile: clean SQL autodoc upon "clean" 19 January 2018, 10:30:42 UTC
d163501 SQL doc Makefile: ensure destdir gets created wherever needed 19 January 2018, 10:28:35 UTC
985a912 sql/Makefile: add missing deps on pdf/svg DB schema generation 16 January 2018, 15:48:53 UTC
2e033d0 sql: don't depend on plpython3u, we don't use it 16 January 2018, 14:36:32 UTC
56e0092 db-init: enforce UTF8 locale for jsonb fanciness Related T918 10 January 2018, 11:16:47 UTC
f5acfeb sql: Fix missing migration instructions Related 08121f7 09 January 2018, 10:46:46 UTC
7b5c111 db-init: actually grant privileges on tables and sequences 06 January 2018, 14:03:33 UTC
back to top