sort by:
Revision Author Date Message Commit Date
a985ba9 New upstream version 0.0.107 17 October 2018, 13:06:07 UTC
2d5efac Reorder upgrade statements so that they actually pass 17 October 2018, 12:44:42 UTC
4f05900 Remove redundant removals from the 122->123 upgrade 17 October 2018, 12:22:08 UTC
435ebcd Drop table occurrence_history Summary: The migration to snapshots has been completed, and all users have been migrated. Depends on D534. Close T830. Test Plan: make test after a testdata rebuild is ok Reviewers: zack Maniphest Tasks: T830 Differential Revision: https://forge.softwareheritage.org/D535 15 October 2018, 14:22:17 UTC
b486063 Drop confusing functions revision_get_by/revision_log_by Summary: The combination of timestamp search for an occurrence with branch name filtering never got ported to the snapshot model. As this function only has a single user, which is a single endpoint deep in the public-facing API, let's just drop it. revision_log_by uses the same filter but gives a log of parent revisions, and has never been used by any client. This will break this API endpoint, which is tracked in T1221, but iff we want to keep it, I think it can be easily reimplemented by composing origin_visit_get_by and the new snapshot_get_branches function. Test Plan: It's dead, Jim. Reviewers: zack Differential Revision: https://forge.softwareheritage.org/D534 15 October 2018, 13:58:52 UTC
3a59229 Rename test methods to test_* to allow py.test collection Summary: Part of T1261 Test Plan: Make test runs the same number of tests before and after Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D522 12 October 2018, 09:57:14 UTC
65e6b69 Drop unused entity tables Summary: Close T838 Test Plan: Remove all the tests! Reviewers: #reviewers! Maniphest Tasks: T838 Differential Revision: https://forge.softwareheritage.org/D509 10 October 2018, 14:25:54 UTC
ee40117 Merge remote-tracking branch 'origin/master' 10 October 2018, 13:55:36 UTC
e9e5e73 storage: Enable to paginate, filter and count snapshot content Related T1207 09 October 2018, 13:28:34 UTC
a4bd5d4 New upstream version 0.0.106 08 October 2018, 15:03:54 UTC
80b4644 Update version requirements for swh.model 08 October 2018, 15:03:46 UTC
7ae19db add build/ to gitignore 08 October 2018, 10:22:39 UTC
63c924d setup: prepare for pypi upload related to T1242 08 October 2018, 10:22:32 UTC
b1eaa9a docs: merge both README files in a single README.md also documement the usage of pifpaf to run pg tests 05 October 2018, 14:38:22 UTC
fb794bf Add requirements-test.txt 05 October 2018, 10:56:16 UTC
3d1b95e Remove the occurrence table Summary: This table is obsolete, and was never really used in the first place. This solves the first half of T839. Depends on D471. Test Plan: rebuild-testdata and make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D472 03 October 2018, 16:55:46 UTC
842b7a3 Remove some unused SQL functions Summary: These functions aren't used by any known clients Test Plan: rebuild-testdata + make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D471 03 October 2018, 16:34:31 UTC
5193e6d Drop backwards-compatibility features between snapshots and occurrences Summary: Old occurrences have been fully migrated to snapshots now, so backwards compatibility is not needed anymore. That's one more step towards the removal of occurrences (T830). This changes the origin_visit_get_by method to stop returning a list of occurrences. The only user of this method is the api/1/origin/{x}/visit/{y} view of the web API, which will need to be updated for T1207 anyway. Test Plan: integration tests updated for origin_visit_get_by; Running the web app with the updated storage only yields the expected change in behavior for the origin/visit API view, navigation is unaffected. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D466 03 October 2018, 16:18:42 UTC
4b1db40 doc: use inkscape instead of dia to generate pdf and svg image files 03 October 2018, 13:44:09 UTC
8e545c7 SQL schema: uniformize comments Summary: Partial cherry-pick of ddb37a5896d04a95cafee413d048bea8df265379 Test Plan: rebuild-testdata and make test report no changes Reviewers: #reviewers! Subscribers: zack Differential Revision: https://forge.softwareheritage.org/D463 03 October 2018, 13:14:04 UTC
1751385 Remove unused function release_get_by Summary: release_get_by has never been used except by its own tests, and it still uses the legacy occurrence_* tables that we want to remove (T830). Test Plan: - make test in swh-environment passes - grep -r release_get_by swh-* yields no result Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D460 03 October 2018, 12:32:27 UTC
16ffb6e docs: make sure images are built/cleaned on sphinx targets 03 October 2018, 09:11:19 UTC
7fa1a34 images/Makefile: fix filenames they have been broken since the recent rename 03 October 2018, 09:10:46 UTC
f5dd8ac nicer landing page, pointing to homepage and API doc Closes T1194 02 October 2018, 08:53:49 UTC
003e1a7 docs: add title and brief module description 01 October 2018, 13:47:54 UTC
2eced78 algos.dir_iterators: Implement iterator protocol in DirectoryIterator - add Python iterator protocol support in the DirectoryIterator class in order to easily visit in a recursive way any directory stored in the archive. - add convenient function dir_iterator wrapping the instantiation of the DirectoryIterator class - add tests Related T1177 28 September 2018, 12:32:17 UTC
9adaed7 algos.dir_iterators: Fix handling of empty directory Two issues found regarding the way empty directories were handled: - _empty_dir_hash variable did not have correct type (str instead of bytes) so empty directory test based on hash comparison was always failing - in the step method of DirectoryIterator, no need to push a new frame for an empty directory as this will stop the iteration 28 September 2018, 12:32:17 UTC
3a86982 sql/bin/db-init: remove createuser and privileges (we now assume a peer superuser) 25 September 2018, 11:05:21 UTC
1286dfa sql/bin/db-init: simpler usage with sensible defaults 25 September 2018, 10:52:10 UTC
332a1e1 use /usr/bin/env python3 as shebang, to be nice to virtualenv 19 September 2018, 15:19:44 UTC
e01f26c swh.storage.db: Fix origin_visit_get_latest_snapshot query Prior to this fix, storage.snapshot_get_latest(origin) was returning the first snapshot instead of the last one. 19 September 2018, 09:32:04 UTC
eed11d4 New upstream version 0.0.105 10 September 2018, 09:36:27 UTC
384b120 docs: add description of the archive copies layout diagram 07 September 2018, 11:28:56 UTC
a66745a docs: add layout diagram describing existing archive copies 07 September 2018, 09:39:44 UTC
46822cb Add snapshot to the statistics endpoint counter Related T1180 06 September 2018, 18:41:18 UTC
8ccac07 docs: add general intro, drop bogus reference to archiver blueprint 06 September 2018, 10:19:41 UTC
19c0344 algos/diff.py: fix typo in comment brought to you by codespell 06 September 2018, 10:19:20 UTC
329d0f9 storage: set timeout to 20s for directory_ls 05 September 2018, 12:30:46 UTC
8455482 New upstream version 0.0.104 29 August 2018, 13:55:37 UTC
9c39449 origin_search: Add option to filter out origins with no visit 02 August 2018, 13:01:24 UTC
cefaf9c New upstream version 0.0.103 30 July 2018, 09:47:53 UTC
9e5a8b7 swh.storage.storage: origin_add returns updated list of dict with id Prior to this commit, this returned only the list of new ids. This is currently not used anywhere in our stack. Would it have been, this would have force the client to try and be smart for dealing with ids. 27 July 2018, 13:57:13 UTC
67fe525 New upstream version 0.0.102 05 June 2018, 12:06:54 UTC
a167960 Migrate away from temporary tables for read queries Summary: To do so, we import a function from a recent version of psycopg2, execute_values, which can execute queries efficiently with a list of values. We also scale the cursor back from having things in SQL functions towards having things inside the db.py database "backend". This will make it easier to iterate, as we won't have to deploy function changes to twenty different databases. After these changes, testing the web UI on a physical replica works. Close T1073 Test Plan: Local integration tests are happy; Navigating the frontend backed by a physical replica seems to be okay now. Reviewers: #reviewers! Maniphest Tasks: T1073 Differential Revision: https://forge.softwareheritage.org/D340 05 June 2018, 11:56:09 UTC
7c85120 Remove now unused function directory_get Summary: As of D337, directory_get has no users left; Time to remove it. Test Plan: make test in the toplevel environment still works Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D339 04 June 2018, 15:36:44 UTC
9f2b266 Make sure there are no pagination issues for origin_visit_get Although most visits should be sequential, and the date should be monotonic, re-importing an old snapshot as a new visit would break the pagination. 30 May 2018, 12:31:36 UTC
c4494aa Fix codespell issues 29 May 2018, 15:30:18 UTC
779cfb5 Don't assign the objstorage result to a variable 29 May 2018, 15:26:24 UTC
2c17350 Rewrite concurrent.futures import to shorten lines 29 May 2018, 15:22:31 UTC
79cc3f6 Better test coverage for content_missing 29 May 2018, 14:38:50 UTC
70574c5 Don't use a temporary table to fetch info about contents 28 May 2018, 16:04:44 UTC
6d5d999 Make the db_transaction{,_generator} decorators support client options Summary: Allow adding server-side statement timeouts for database operations Test Plan: make test still works Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D334 28 May 2018, 13:18:26 UTC
27c942c New upstream version 0.0.101 24 May 2018, 10:13:51 UTC
d826f75 swh.storage.api.client: Permit to specify the query timeout option Related T1061 24 May 2018, 10:02:11 UTC
a1aa230 New upstream version 0.0.100 12 May 2018, 16:12:40 UTC
77e69ec Use a concurrent.future to parallelize objstorage and storage addition 12 May 2018, 15:49:44 UTC
4a9b623 test_api_client: stop leaking directories Summary: When mkdtemp is called, shutil.rmtree must be called as well Test Plan: Look at /tmp before and after running tests, notice no new directories instead of 60. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D331 11 May 2018, 13:42:09 UTC
721d088 Add test to ensure storage/objstorage consistency is kept at all times Summary: The behavior of storage when the underlying objstorage had an exception was never actually tested. This new test weeded out a bug in the threaded implementation for copy_to. Test Plan: the new test passes Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D330 11 May 2018, 10:13:44 UTC
18c9dad Only instantiate the storage backend once per import This allows connection reuse for postgresql and potential remote backends such as for the object storage, rather than reinitiating all connections on every request. 09 May 2018, 14:43:32 UTC
5a2de1c Use thread-aware psycopg2 connection pooling for database access Summary: This allows to use swh.storage with a modicum of concurrency Test Plan: clearly, make test should still pass Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D325 07 May 2018, 13:52:35 UTC
6c93693 Stop using the storage.db attribute directly Summary: Add a level of indirection to allow swapping out the implementation of the db attribute Test Plan: once again, make test keeps on working Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D324 07 May 2018, 13:46:21 UTC
e1e5025 Make the storage test fixture connect to postgres itself Summary: This avoids reusing a potentially stale connection handle. Also allows testing potential connection pooling behavior. This forces us to do proper cursor sanitation as well, a bunch of "transactional" operations weren't actually transactional. Test Plan: another round of make test still working Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D323 07 May 2018, 13:43:27 UTC
ff2d37d Deallocate storage when the tests teardown Summary: Helps avoid lingering postgresql connections when a test fails Test Plan: make test still works ;) Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D322 07 May 2018, 13:35:03 UTC
9ca8f7a Make sure schema changes get committed when doing the AlteringSchemaTests 07 May 2018, 13:31:21 UTC
03eea50 tests: Move test fixture to swh.core.tests.server_testing Related T1036 25 April 2018, 13:00:25 UTC
b4147ec fix typo in docstrings/comments (tnx codespell) 12 March 2018, 11:01:52 UTC
916d9cc Remove file that should not have been committed and update gitignore 27 February 2018, 09:40:18 UTC
49a81bf New upstream version 0.0.99 20 February 2018, 12:32:25 UTC
d8ad992 storage: Add methods to compute directories/revisions diff This commit adds the implementation of an efficient algorithm for comparing two directory trees in order to compute the list of introduced file changes in terms of addition / deletion / modification/ renaming. It can be found in the diff module located in the new namespace swh.storage.algos That algorithm is used to extend the storage API with the following methods: - diff_directories: compute diff between two arbitrary directories - diff_revisions: compute diff between two arbitrary revisions - diff_revision: compute diff between a revision and its first parent Related T921 Closes D295 20 February 2018, 12:19:09 UTC
6b80cff Add a new table for "bucketed" object counts This table allows counting objects by bucket, keeping the transactions for counting objects short (a few dozen seconds at most). Also add a "single_update" boolean field to the main object_counts table to be able to discriminate tables that are counted via buckets and tables counted on one go. The main table is updated every 256 counted buckets to avoid too much churn on the table. Close T962. 19 February 2018, 18:30:41 UTC
86d68a6 doc: update table clusters in SQL diagram 09 February 2018, 11:48:40 UTC
22ee4c5 swh.storage.content_missing: Improve docstring 08 February 2018, 18:27:29 UTC
4fdbb62 New upstream version 0.0.98 06 February 2018, 14:27:15 UTC
9082524 api.client: switch back_compat for snapshot_add as well 06 February 2018, 14:27:10 UTC
ada0d38 storage: swap back_compat switch for snapshot_add 06 February 2018, 14:25:36 UTC
187343f New upstream version 0.0.97 06 February 2018, 13:07:07 UTC
9f84cec Add function to retrieve latest snapshot based on origin filtering This function does _not_ have any backwards compatibility provisions. 31 January 2018, 14:19:47 UTC
32e89b0 Add snapshot information when fetching origin_visits from the database 31 January 2018, 10:49:14 UTC
d6c3f65 sql doc Makefile: (try to) dropdb before creating it this way doc (re)building is more robust 19 January 2018, 13:22:11 UTC
b32d2c9 Add a way to remove the backwards-compatibility feature of snapshots 19 January 2018, 10:54:00 UTC
ab0e90c doc Makefile: clean SQL autodoc upon "clean" 19 January 2018, 10:30:42 UTC
d163501 SQL doc Makefile: ensure destdir gets created wherever needed 19 January 2018, 10:28:35 UTC
985a912 sql/Makefile: add missing deps on pdf/svg DB schema generation 16 January 2018, 15:48:53 UTC
2e033d0 sql: don't depend on plpython3u, we don't use it 16 January 2018, 14:36:32 UTC
56e0092 db-init: enforce UTF8 locale for jsonb fanciness Related T918 10 January 2018, 11:16:47 UTC
f5acfeb sql: Fix missing migration instructions Related 08121f7 09 January 2018, 10:46:46 UTC
7b5c111 db-init: actually grant privileges on tables and sequences 06 January 2018, 14:03:33 UTC
ca308c7 sql/bin/db-init: new helper script to setup development DB 06 January 2018, 13:52:20 UTC
cf6cac7 SQL: move definition of hash_sha1() function to swh-func.sql rationale: swh-init.sql should only contain stuff that requires being Postgres superuser 06 January 2018, 09:21:51 UTC
a0ac20a db: use a thread instead of a tmpfile to COPY data to postgresql Summary: Avoids writing a bunch of temporary files for no good reason. T866 was a symptom of this. Test Plan: No regressions in the unit tests Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D282 21 December 2017, 18:48:16 UTC
419d135 db: move external imports further down 21 December 2017, 18:45:19 UTC
221f8b1 Add 116->117 SQL upgrade script 21 December 2017, 18:40:57 UTC
11fcb82 Refactor swh_snapshot_add to hit more indexes 21 December 2017, 18:38:50 UTC
d622f3a New upstream version 0.0.96 19 December 2017, 15:25:57 UTC
76e3670 db: remove bare except: 19 December 2017, 15:22:34 UTC
afcbe94 sql/upgrades: add 115 -> 116 swh-enums: Reference new mercurial revision type Related T329 19 December 2017, 13:27:40 UTC
a455fc0 docs: reference original db-schema.svg without symlink indirection the new sphinx seems to be confused by relative symlinks when we merge all docs together 19 December 2017, 11:06:59 UTC
655bcb6 doc generation: use more specific db-schema basename for images was: "swh" this way we can ship the images as is in the global doc, with minimal clash risks 19 December 2017, 11:06:04 UTC
73256e6 Add 114->115 SQL upgrade script 18 December 2017, 15:12:01 UTC
36aee70 Fix documentation formatting for tool_add/tool_get 15 December 2017, 14:48:16 UTC
back to top