swh:1:snp:eb70f1f85391e4b077c211bec36af0061c4bf937

sort by:
Revision Author Date Message Commit Date
d5e8229 storage.in_memory: Use integer ids for origin and visit instead of tuples Related T1421 04 December 2018, 14:58:15 UTC
bda7cf1 snapshot_add: Check the origin visit exists. Summary: Instead of failing silently (PG storage) or raising an un-documented exception (in-mem storage). Reviewers: #reviewers, anlambert Reviewed By: #reviewers, anlambert Subscribers: swh-public-ci Differential Revision: https://forge.softwareheritage.org/D769 04 December 2018, 14:56:12 UTC
b6b004b snapshot_add: Check the origin visit exists. Instead of failing silently (PG storage) or raising an un-documented exception (in-mem storage). 04 December 2018, 13:00:25 UTC
12f5c5a Add 'recursive' argument to the in-mem storage. Summary: Also add test coverage for that feature and improve doc. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: swh-public-ci Differential Revision: https://forge.softwareheritage.org/D737 30 November 2018, 10:28:16 UTC
df27d11 Add 'recursive' argument to the in-mem storage. Summary: Also add test coverage for that feature and improve doc. Reviewers: #reviewers Differential Revision: https://forge.softwareheritage.org/D737 29 November 2018, 14:59:03 UTC
d8463b1 Use the new doc style for {revision,release}_add. Summary: Also fix the 'committer_date' on the PG Storage's revision_add, that slipped through D676. Reviewers: #reviewers, douardda Reviewed By: #reviewers, douardda Subscribers: douardda, ardumont, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D698 29 November 2018, 13:40:49 UTC
f203821 Fix column name in metadata_provider_get. Test Plan: Test coverage will come in the next diff Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D716 29 November 2018, 13:40:40 UTC
7af40f6 Fix documented return type of metadata_provider_add. Summary: As the in-mem storage was written based on this documentation, fix it as well. Add a test for this behavior. Depends on D716. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D717 29 November 2018, 09:21:43 UTC
abee27c Storage.release_get: Yield None for non-existent release Similar to Storage.revision_get, yields None when querying a non-existent release. 27 November 2018, 16:31:51 UTC
cd2db9b Document edge cases of content_get_range. Reviewers: #reviewers, douardda, ardumont Reviewed By: #reviewers, douardda, ardumont Subscribers: ardumont, douardda, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D694 27 November 2018, 10:44:12 UTC
479ad67 sql: Update origin trigger to provide full origin information 26 November 2018, 14:28:28 UTC
d190fda Implement content_get and content_get_range for the in-mem storage. Summary: Needed to work on T1307 for the indexers. Reviewers: #reviewers, ardumont, douardda Reviewed By: #reviewers, ardumont, douardda Subscribers: douardda, ardumont, zack, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D695 26 November 2018, 14:24:36 UTC
e894064 doc: update index to match new swh-doc format related to T1330 23 November 2018, 13:48:09 UTC
90da2a9 Fix docstrings for Storage.add_{release,revision}. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D676 22 November 2018, 12:10:31 UTC
dd62eae Increase test coverage for stat_counters + fix its bugs. Reviewers: #reviewers, anlambert Reviewed By: #reviewers, anlambert Subscribers: anlambert, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D679 21 November 2018, 16:09:34 UTC
5d822c9 Add 'inmemory' storage class in get_storage. Summary: https://forge.softwareheritage.org/D684#inline-3666 Reviewers: #reviewers, anlambert Reviewed By: #reviewers, anlambert Subscribers: swh-public-ci, ardumont, anlambert Differential Revision: https://forge.softwareheritage.org/D689 21 November 2018, 14:50:48 UTC
d1ce244 Run property-based tests on RemoteStorage and in_memory.Storage. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D678 21 November 2018, 14:33:19 UTC
37d70d6 Make it official that content_missing accepts missing keys. Summary: The PG Storage provides that feature, and the Git loader uses it. This commit documents that feature, adds it to the Mem Storage, and changes test_content_missing to test that behavior. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: swh-public-ci Differential Revision: https://forge.softwareheritage.org/D680 21 November 2018, 14:32:35 UTC
97b356a Add empty fetch_history_{start,end} endpoints to the in-mem storage. Summary: The Git loader calls them; but we're going to remove them (T829). Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: swh-public-ci Differential Revision: https://forge.softwareheritage.org/D681 21 November 2018, 14:32:21 UTC
8ea06fa tests: Move generation tests fixture in a dedicated module So that tests that requires tests fixture without hypothesis does not break. Related T1354 19 November 2018, 09:28:26 UTC
b61cf3d Add in-memory storage. Test Plan: Land the two parent diffs, then rebase this one on top of them. Reviewers: #reviewers, douardda Reviewed By: #reviewers, douardda Subscribers: olasd, douardda, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D645 19 November 2018, 09:03:52 UTC
fc3b9dc Document metadata providers. Reviewers: #reviewers, douardda, moranegg Reviewed By: #reviewers, douardda, moranegg Subscribers: douardda, moranegg, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D637 15 November 2018, 09:02:57 UTC
12edfac tox: Define a py3-slow environment Use: tox -e py3-slow 14 November 2018, 15:07:33 UTC
880aa9c tests: Use fast/slow profile names 14 November 2018, 15:07:33 UTC
3f05ada Makefile: Adapt makefile to use the dev profile 14 November 2018, 15:07:33 UTC
076a392 tests: Use hypothesis profile mechanism to configure samples size 14 November 2018, 15:07:33 UTC
04323a6 tests: Decrease generation sample to make tests faster Currently investigating hypothesis settings [1] [1] https://hypothesis.readthedocs.io/en/latest/settings.html#settings-profiles 14 November 2018, 15:07:33 UTC
8aa990f tests: Add docstring to test to detail what's tested 14 November 2018, 15:07:33 UTC
e1a460c tests: Move property_based marker to toplevel test class 14 November 2018, 15:07:33 UTC
3829885 tests: Use sorted 14 November 2018, 15:07:33 UTC
c876d86 storage: Open content_get_range endpoint Related T991 14 November 2018, 15:07:33 UTC
a429574 tests: Use sorted on list directly 14 November 2018, 15:07:33 UTC
c7a3266 tests: Build content generated in less steps 14 November 2018, 15:07:33 UTC
f87808c tests: Clarify gen_content docstring 14 November 2018, 15:07:33 UTC
0c51e4d tests: Add test content generation util function and use it Test back `content_get` and `content_get_metadata` as a first step. Plan to use those for the new and upcoming `content_get_range` function. 14 November 2018, 15:07:32 UTC
823465f Remove SQLisms from the tests and API. Reviewers: #reviewers, olasd Reviewed By: #reviewers, olasd Subscribers: olasd, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D642 14 November 2018, 14:58:30 UTC
64e67aa make-doc: add an 'assets' make target for the doc used by the swh-docs to generate images and schemas when building the whole documentation. 12 November 2018, 15:19:22 UTC
1031dc5 debian: Remove nose and migrate to pytest 12 November 2018, 13:10:07 UTC
1a1d214 algos.revisions_walker: Fix get_revisions_walker doc 12 November 2018, 10:12:36 UTC
14e17d1 Fix storage doc. Reviewers: #reviewers, olasd Reviewed By: #reviewers, olasd Subscribers: olasd, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D643 09 November 2018, 17:38:49 UTC
b944f00 Restructure tests to prepare for tests with no DB. Reviewers: #reviewers, olasd Reviewed By: #reviewers, olasd Subscribers: swh-public-ci Differential Revision: https://forge.softwareheritage.org/D644 09 November 2018, 17:38:38 UTC
7934ded Fix rst syntax in docstrings 09 November 2018, 14:15:17 UTC
81b8c9b Add :orphan: metadata to archive-copies and sql-storage doc files to prevent sphinx from complaining about these not being included. 09 November 2018, 14:15:17 UTC
88c33cc algos: Add iterators to walk across revisions history This commit adds iterators to walk across the history of revisions heading to a given one. The following types of iteration are offered: - committer_date: revisions are returned in reverse chronological order of their commit date (same as git log) - dfs: revisions are returned in the same order they are visited when performing a depth-first search in pre order on the revisions DAG - dfs_post: revisions are returned in the same order they are visited when performing a depth-first search in post order on the revisions DAG - bfs: revisions are returned in the same order they are visited when performing a breadth-first search on the revisions DAG Another iterator of type path, returning only revisions that modify a specific path in reverse chronological order of their commit date, is also introduced. Nevertheless, due to client-side implementation, its performances are far from optimal when walking across a really large history. Related T1026 Related T1284 09 November 2018, 13:04:04 UTC
19d4019 Rename the 'ts' argument of 'origin_visit_add' to 'date'. Summary: To be consistent with the name used in the DB and the dict key returned by 'origin_visit_get'. Note that this makes the client incompatible with older versions of the server. Reviewers: #reviewers, olasd Reviewed By: #reviewers, olasd Subscribers: olasd, swh-public-ci Differential Revision: https://forge.softwareheritage.org/D636 09 November 2018, 10:45:41 UTC
884d9e7 sql/Makefile: Fix pifpaf test as it was always considered used 08 November 2018, 13:43:03 UTC
465192e storage.listener: Subscribe listener to snapshot insertion Related T1275 26 October 2018, 10:21:52 UTC
1ad38af sql: Add snapshot trigger event on insertion Related T1275 26 October 2018, 10:21:52 UTC
fba230d DB schema: manual fix for some arrow source/dest to avoid degenerate angles in rendering 26 October 2018, 07:57:04 UTC
a579a1a DB schema: add light foreign key arrows from snapshot to * 26 October 2018, 07:51:57 UTC
b032d2a DB schema: remove left-over occurrence stuff from snapshot cluster 26 October 2018, 07:51:25 UTC
1c74f6d tox.ini: Align configuration file with other Also, without this, this failed because somehow, something is not finding requirements-test.txt. ``` Could not open requirements file: [Errno 2] No such file or directory: ' requirements-test.txt' ``` 24 October 2018, 13:42:12 UTC
7628710 SQL schema: update to reflect recent changes recent changes include most notably the removal of occurrence-related tables, but also the addition of table columns accumulated over time Closes T1283 24 October 2018, 12:35:02 UTC
a6742b7 DB schema doc: error out if SQL schema cannot be found 24 October 2018, 12:34:52 UTC
582d72d docs Makefile: only echo SQL queries on error 24 October 2018, 12:22:48 UTC
5fe9221 docs Makefile: forward "distclean" to sql dir 24 October 2018, 12:20:01 UTC
b91482a gitignore and remove generate archive copies diagram Closes T1269 24 October 2018, 12:00:01 UTC
242f985 Don't run pytest in the docs directory 23 October 2018, 14:48:15 UTC
e77d809 tests: increase deadline for CI seems to run slower on there. 22 October 2018, 16:06:59 UTC
076144e tests: assertEquals -> assertEqual 22 October 2018, 14:46:34 UTC
32ef182 tests: use pytest and add a simple tox file related to T1261 22 October 2018, 14:45:08 UTC
5b38656 flake8 22 October 2018, 14:32:18 UTC
a8fbce5 Warn when using db_transaction/db_transaction_generator where you shouldn't. Summary: I did it, it caused closed cursors that hang the process, that's hard to debug. Reviewers: #reviewers, ardumont Reviewed By: #reviewers, ardumont Subscribers: ardumont Differential Revision: https://forge.softwareheritage.org/D556 22 October 2018, 09:03:36 UTC
d211615 Mark snapshot tests as needing the database 18 October 2018, 16:30:44 UTC
b2d4921 Mark minimum hypothesis version for datetimes 18 October 2018, 16:27:56 UTC
ec7a119 Add hypothesis to debian build-dependencies 18 October 2018, 16:25:07 UTC
ad92248 Add a function to retrieve a full snapshot Summary: This function wraps the paginating branch listing API to do its work. It's in swh.storage.algos because it should be done client-side rather than server-side. The tests are fairly simple, and mostly add a bunch of tooling around hypothesis to generate data that conforms to the swh data schema. Those things could probably be adequately moved to a new submodule of swh.model. Test Plan: new tests added Reviewers: anlambert Differential Revision: https://forge.softwareheritage.org/D551 18 October 2018, 14:57:55 UTC
7cd7597 Rewrite snapshot_get in terms of snapshot_get_branches Summary: This increases uniformity and reduces code duplication between the two functions. The addition of the 'next_branch' return value in snapshot_get_branches allows callers to get branches incrementally in a loop which is friendlier. Depends on D545 Test Plan: tests have been updated to add the new next_branch field Reviewers: anlambert Differential Revision: https://forge.softwareheritage.org/D546 17 October 2018, 14:56:42 UTC
0d52bf4 Rewrite snapshot pagination tests declaratively Summary: This looks less confusing than removing a bunch of values from the test data. Test Plan: make test still passes Reviewers: anlambert Differential Revision: https://forge.softwareheritage.org/D545 17 October 2018, 14:56:07 UTC
80cd420 packaging: bump swh-core dependency to 0.0.44 17 October 2018, 13:56:52 UTC
9ce5f78 README: swh-storage-testdata is not required any more 17 October 2018, 13:56:13 UTC
a9c32d2 sql: adapt Makefile for sql file renames and make it pifpaf compliant ie. delete stamps when executed via pifpaf for the db setup. 17 October 2018, 13:46:36 UTC
ee27736 kill sql/db-init in favor of swh-core's new swh-db-init tool 17 October 2018, 13:46:36 UTC
c4cdb64 tests: refactor the StorageTestFixture class and move sql files in the package so we can build the db setup sql file on the fly: - make it inherit from SingleDbTestFixture, so we can - move sql files in swh/storage/sql/ - get rid og the setUpClass static method to reuse the one from SingleDbTestFixture, - apply the required modifications to tests based on this StorageTestFixture. Warnings: - this requires D538, - this will break tests in swh-vault. Also add README.md to the MANIFEST.in file so it does not require a too recent setuptool. 17 October 2018, 13:46:36 UTC
8f9af9b Remove some leftover compatibility code 17 October 2018, 13:16:31 UTC
2d5efac Reorder upgrade statements so that they actually pass 17 October 2018, 12:44:42 UTC
4f05900 Remove redundant removals from the 122->123 upgrade 17 October 2018, 12:22:08 UTC
435ebcd Drop table occurrence_history Summary: The migration to snapshots has been completed, and all users have been migrated. Depends on D534. Close T830. Test Plan: make test after a testdata rebuild is ok Reviewers: zack Maniphest Tasks: T830 Differential Revision: https://forge.softwareheritage.org/D535 15 October 2018, 14:22:17 UTC
b486063 Drop confusing functions revision_get_by/revision_log_by Summary: The combination of timestamp search for an occurrence with branch name filtering never got ported to the snapshot model. As this function only has a single user, which is a single endpoint deep in the public-facing API, let's just drop it. revision_log_by uses the same filter but gives a log of parent revisions, and has never been used by any client. This will break this API endpoint, which is tracked in T1221, but iff we want to keep it, I think it can be easily reimplemented by composing origin_visit_get_by and the new snapshot_get_branches function. Test Plan: It's dead, Jim. Reviewers: zack Differential Revision: https://forge.softwareheritage.org/D534 15 October 2018, 13:58:52 UTC
3a59229 Rename test methods to test_* to allow py.test collection Summary: Part of T1261 Test Plan: Make test runs the same number of tests before and after Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D522 12 October 2018, 09:57:14 UTC
65e6b69 Drop unused entity tables Summary: Close T838 Test Plan: Remove all the tests! Reviewers: #reviewers! Maniphest Tasks: T838 Differential Revision: https://forge.softwareheritage.org/D509 10 October 2018, 14:25:54 UTC
ee40117 Merge remote-tracking branch 'origin/master' 10 October 2018, 13:55:36 UTC
e9e5e73 storage: Enable to paginate, filter and count snapshot content Related T1207 09 October 2018, 13:28:34 UTC
80b4644 Update version requirements for swh.model 08 October 2018, 15:03:46 UTC
7ae19db add build/ to gitignore 08 October 2018, 10:22:39 UTC
63c924d setup: prepare for pypi upload related to T1242 08 October 2018, 10:22:32 UTC
b1eaa9a docs: merge both README files in a single README.md also documement the usage of pifpaf to run pg tests 05 October 2018, 14:38:22 UTC
fb794bf Add requirements-test.txt 05 October 2018, 10:56:16 UTC
3d1b95e Remove the occurrence table Summary: This table is obsolete, and was never really used in the first place. This solves the first half of T839. Depends on D471. Test Plan: rebuild-testdata and make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D472 03 October 2018, 16:55:46 UTC
842b7a3 Remove some unused SQL functions Summary: These functions aren't used by any known clients Test Plan: rebuild-testdata + make test Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D471 03 October 2018, 16:34:31 UTC
5193e6d Drop backwards-compatibility features between snapshots and occurrences Summary: Old occurrences have been fully migrated to snapshots now, so backwards compatibility is not needed anymore. That's one more step towards the removal of occurrences (T830). This changes the origin_visit_get_by method to stop returning a list of occurrences. The only user of this method is the api/1/origin/{x}/visit/{y} view of the web API, which will need to be updated for T1207 anyway. Test Plan: integration tests updated for origin_visit_get_by; Running the web app with the updated storage only yields the expected change in behavior for the origin/visit API view, navigation is unaffected. Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D466 03 October 2018, 16:18:42 UTC
4b1db40 doc: use inkscape instead of dia to generate pdf and svg image files 03 October 2018, 13:44:09 UTC
8e545c7 SQL schema: uniformize comments Summary: Partial cherry-pick of ddb37a5896d04a95cafee413d048bea8df265379 Test Plan: rebuild-testdata and make test report no changes Reviewers: #reviewers! Subscribers: zack Differential Revision: https://forge.softwareheritage.org/D463 03 October 2018, 13:14:04 UTC
1751385 Remove unused function release_get_by Summary: release_get_by has never been used except by its own tests, and it still uses the legacy occurrence_* tables that we want to remove (T830). Test Plan: - make test in swh-environment passes - grep -r release_get_by swh-* yields no result Reviewers: #reviewers! Differential Revision: https://forge.softwareheritage.org/D460 03 October 2018, 12:32:27 UTC
16ffb6e docs: make sure images are built/cleaned on sphinx targets 03 October 2018, 09:11:19 UTC
7fa1a34 images/Makefile: fix filenames they have been broken since the recent rename 03 October 2018, 09:10:46 UTC
f5dd8ac nicer landing page, pointing to homepage and API doc Closes T1194 02 October 2018, 08:53:49 UTC
003e1a7 docs: add title and brief module description 01 October 2018, 13:47:54 UTC
2eced78 algos.dir_iterators: Implement iterator protocol in DirectoryIterator - add Python iterator protocol support in the DirectoryIterator class in order to easily visit in a recursive way any directory stored in the archive. - add convenient function dir_iterator wrapping the instantiation of the DirectoryIterator class - add tests Related T1177 28 September 2018, 12:32:17 UTC
9adaed7 algos.dir_iterators: Fix handling of empty directory Two issues found regarding the way empty directories were handled: - _empty_dir_hash variable did not have correct type (str instead of bytes) so empty directory test based on hash comparison was always failing - in the step method of DirectoryIterator, no need to push a new frame for an empty directory as this will stop the iteration 28 September 2018, 12:32:17 UTC
back to top