43f5bf9 | Nicolas Dandrimont | 01 October 2015, 09:32:07 UTC | Bump swh.core dependency version | 01 October 2015, 09:32:07 UTC |
a718e9d | Stefano Zacchiroli | 30 September 2015, 14:17:10 UTC | db.py: drop raises psycopg2.internalerror it is no longer the case | 30 September 2015, 14:17:10 UTC |
64a7832 | Stefano Zacchiroli | 30 September 2015, 12:51:01 UTC | objstorage: fix predictable tempfile race when adding objects Before this change, two workers adding the same object will end up racing to write <SHA1>.tmp. That should not lead to corruption, because they are (in principle) going to write the exact same content, but it made the one of the two that arrives last fail the final rename. With this fix they will write in different <SHA1>.RANDOM_STUFF.tmp, and then rename their own file into place. Note that this change slightly increases the chances of having garbage in the object store, so we will need to look for this during GC passes. | 30 September 2015, 12:51:01 UTC |
e4feb05 | Stefano Zacchiroli | 30 September 2015, 10:31:16 UTC | upgrades/17.sql: add recent changes | 30 September 2015, 10:33:00 UTC |
25b0544 | Stefano Zacchiroli | 30 September 2015, 09:56:25 UTC | swh-func.sql: use LANGUAGE sql wherever possible Overall, we can use LANGUAGE sql only in 6 cases (vs 17 functions that remains plpgsql). This is in large part due to the fact that LANGUAGE sql stored procedures cannot non-existent tables, and most notably temporary tables. Closes T55. | 30 September 2015, 10:33:00 UTC |
cd507ba | Stefano Zacchiroli | 30 September 2015, 10:11:55 UTC | SQL: nicer API for swh_content_find_* functions the functions no longer fail with internal errors (due to "insert into ... strict"), but rather return NULL in those cases, because no match has been found. | 30 September 2015, 10:33:00 UTC |
d4131b3 | Antoine R. Dumont | 30 September 2015, 09:48:37 UTC | Update docstring | 30 September 2015, 09:48:46 UTC |
7fe460f | Stefano Zacchiroli | 30 September 2015, 09:01:43 UTC | bump SQL schema version to 17 include upgrade script from 16 | 30 September 2015, 09:04:30 UTC |
be72b19 | Stefano Zacchiroli | 30 September 2015, 08:57:00 UTC | sql-func refactoring: single dir*_entry_add function The function is parametric in the type of entries to be added (file/dir/rev). Get rid of the previous 3 (99% overlapping) functions swh_directory_{file,rev,dir}_entry_add functions. Make storage.py use the new function. | 30 September 2015, 09:04:30 UTC |
4612bc6 | Antoine R. Dumont | 30 September 2015, 08:57:31 UTC | Refactor - pep8 compliance - Remove unused import | 30 September 2015, 08:57:57 UTC |
c956386 | Antoine R. Dumont | 30 September 2015, 08:39:42 UTC | Improve output response for find occurrence api | 30 September 2015, 08:56:54 UTC |
141d392 | Antoine R. Dumont | 29 September 2015, 15:32:07 UTC | Implement content_find_occurrence up to the db | 30 September 2015, 08:43:30 UTC |
3e2fe23 | Antoine R. Dumont | 29 September 2015, 14:54:34 UTC | Refactor - db.py returns None or the content found The triplet of None should be sandboxed to db.py | 30 September 2015, 08:43:30 UTC |
91debb1 | Antoine R. Dumont | 29 September 2015, 13:52:57 UTC | Refactor - Permit reuse of the content_find api | 30 September 2015, 08:43:30 UTC |
6944ab8 | Antoine R. Dumont | 29 September 2015, 13:18:15 UTC | Refactor - Return result directory Remove the needless yield | 30 September 2015, 08:42:51 UTC |
311de1b | Antoine R. Dumont | 29 September 2015, 13:08:20 UTC | Refactor - Reuse | 30 September 2015, 08:42:51 UTC |
df4bdfa | Antoine R. Dumont | 29 September 2015, 12:16:27 UTC | Open content-find-occurrence | 30 September 2015, 08:42:51 UTC |
bdfc021 | Stefano Zacchiroli | 30 September 2015, 07:33:27 UTC | test_storage: add collision detection test | 30 September 2015, 07:40:40 UTC |
9aa18e4 | Stefano Zacchiroli | 30 September 2015, 07:25:06 UTC | test_api_client: use absolute import to allow rm __init__.py | 30 September 2015, 07:25:06 UTC |
b4e4b20 | Stefano Zacchiroli | 30 September 2015, 07:18:22 UTC | SQL swh_content_missing: build result incrementally by uniting on contents that appear to be missing when looking at a single checksum algorithm. This is semantically equivalent to the previous approach, but allows to exploit single-column indexes; which are non-redundant, smaller, and hence more likely to be used. This is the "metro ligne 6" approach. | 30 September 2015, 07:18:22 UTC |
7a33494 | Stefano Zacchiroli | 29 September 2015, 16:15:56 UTC | db_testing.py: allow to customize DB restore | 29 September 2015, 16:15:56 UTC |
f88331c | Nicolas Dandrimont | 29 September 2015, 14:51:45 UTC | Bump dependencies on swh.core to >= 0.0.6 | 29 September 2015, 14:51:45 UTC |
bfe9016 | Nicolas Dandrimont | 29 September 2015, 14:51:25 UTC | debian/control: indent with spaces | 29 September 2015, 14:51:25 UTC |
e424885 | Nicolas Dandrimont | 29 September 2015, 14:36:32 UTC | Bump dependency on swh.core | 29 September 2015, 14:41:40 UTC |
bb8cdff | Stefano Zacchiroli | 29 September 2015, 14:38:30 UTC | move db_test.py from swh-storage to swh-core | 29 September 2015, 14:39:10 UTC |
260c379 | Nicolas Dandrimont | 29 September 2015, 14:26:51 UTC | Add swh.storage.api package to setup.py | 29 September 2015, 14:26:51 UTC |
2b46e69 | Nicolas Dandrimont | 29 September 2015, 13:04:43 UTC | swh.storage.api: Serialize/deserialize exceptions This currently uses pickle which breaks the msgpack + dict "abstraction" we settled on. This should be fixed eventually. | 29 September 2015, 13:04:43 UTC |
58f821c | Nicolas Dandrimont | 29 September 2015, 11:52:19 UTC | Update api client and server to msgpack | 29 September 2015, 11:55:22 UTC |
79d6539 | Stefano Zacchiroli | 29 September 2015, 09:16:41 UTC | gitignore: ignore .eggs/ dir | 29 September 2015, 09:16:41 UTC |
b041526 | Antoine R. Dumont | 29 September 2015, 07:23:23 UTC | Refactor - Migrate content_present to content_find with unified api | 29 September 2015, 08:28:05 UTC |
1ddeff4 | Nicolas Dandrimont | 29 September 2015, 08:21:38 UTC | Move remote_storage and api to api.client and api.server Closes T44. | 29 September 2015, 08:21:38 UTC |
1cb39a2 | Nicolas Dandrimont | 28 September 2015, 15:28:42 UTC | sql/swh-schema: unix_path to bytea | 28 September 2015, 15:36:28 UTC |
f660268 | Nicolas Dandrimont | 28 September 2015, 15:14:37 UTC | sql/upgrades/015: drop function before readding it | 28 September 2015, 15:35:16 UTC |
d371898 | Stefano Zacchiroli | 28 September 2015, 14:19:37 UTC | SQL schema diagram: add skipped_content | 28 September 2015, 14:19:37 UTC |
eb7493c | Nicolas Dandrimont | 28 September 2015, 14:01:25 UTC | sql/upgrades/015: bump database version | 28 September 2015, 14:01:25 UTC |
6a4da5c | Nicolas Dandrimont | 28 September 2015, 13:58:56 UTC | sql/swh-func: fix outdated comments | 28 September 2015, 14:00:02 UTC |
13315d2 | Nicolas Dandrimont | 28 September 2015, 13:55:16 UTC | sql/sql-func: Make tests pass again | 28 September 2015, 14:00:02 UTC |
be3910e | Stefano Zacchiroli | 28 September 2015, 13:46:39 UTC | Merge branch 'merge-dir-entries' | 28 September 2015, 13:46:39 UTC |
8870ff4 | Stefano Zacchiroli | 28 September 2015, 13:42:19 UTC | swh-func.sql: add refactoring TODO item | 28 September 2015, 13:46:28 UTC |
3c17d73 | Stefano Zacchiroli | 28 September 2015, 13:40:21 UTC | swh_directory_entry_*_add: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
345293a | Stefano Zacchiroli | 27 September 2015, 17:07:55 UTC | swh_directory_walk_one: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
f4b8b40 | Stefano Zacchiroli | 27 September 2015, 16:27:55 UTC | swh_content_find_dir: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
c08ee26 | Stefano Zacchiroli | 27 September 2015, 14:35:32 UTC | upgrades/015.sql: migration script for dir_list_* table merge | 28 September 2015, 13:46:28 UTC |
f068673 | Stefano Zacchiroli | 27 September 2015, 13:45:24 UTC | SQL schema: merge dir entry arrays into directory This removes tables directory_list_{file,dir,rev}, and add corresponding arrays {dir,file,rev}_entries to the directory table, which was just a set up to now. Indexes migrated accordingly. | 28 September 2015, 13:46:28 UTC |
945bdfb | Nicolas Dandrimont | 28 September 2015, 13:41:14 UTC | sql/sql-func: copy origin from tmp_skipped_content | 28 September 2015, 13:41:14 UTC |
573b8d4 | Nicolas Dandrimont | 28 September 2015, 13:33:20 UTC | swh.storage.storage: Only add missing skipped_content | 28 September 2015, 13:33:20 UTC |
e16565e | Nicolas Dandrimont | 28 September 2015, 13:32:16 UTC | swh.storage.storage: add origin to skipped_content | 28 September 2015, 13:32:16 UTC |
9188980 | Nicolas Dandrimont | 28 September 2015, 12:49:39 UTC | sql/swh-func.sql: remove useless declaration | 28 September 2015, 12:49:39 UTC |
8ff3109 | Nicolas Dandrimont | 28 September 2015, 12:49:12 UTC | swh.storage.storage: Allow storing "skipped" contents | 28 September 2015, 12:49:12 UTC |
09193a5 | Nicolas Dandrimont | 28 September 2015, 12:43:27 UTC | swh.storage.db: map stored procedures for skipped_content | 28 September 2015, 12:43:27 UTC |
9073916 | Nicolas Dandrimont | 28 September 2015, 12:42:39 UTC | sql schema: update skipped_content and add stored procedures for update | 28 September 2015, 12:42:39 UTC |
f1f62bd | Nicolas Dandrimont | 28 September 2015, 09:44:15 UTC | sql/swh-schema: add index on content(sha256) | 28 September 2015, 09:44:15 UTC |
d53dc22 | Nicolas Dandrimont | 28 September 2015, 09:43:48 UTC | sql/swh-schema: add skipped_content table | 28 September 2015, 09:43:48 UTC |
0dd1ece | Stefano Zacchiroli | 27 September 2015, 13:35:38 UTC | SQL: add swh_content_find, to lookup content by checksum(s) Intente use is content-by-content lookup (e.g., in the web app), not mass addition or check for missing contents. We already have swh_content_missing for that. | 27 September 2015, 13:35:38 UTC |
7de609d | Stefano Zacchiroli | 27 September 2015, 12:43:25 UTC | SQL: improve indentation of directory_walk_one to avoid very long lines | 27 September 2015, 12:43:25 UTC |
1ce7f21 | Stefano Zacchiroli | 25 September 2015, 07:43:56 UTC | swh_content_find_occ. query: improve doc string | 25 September 2015, 09:52:02 UTC |
c4e2e78 | Stefano Zacchiroli | 24 September 2015, 20:11:38 UTC | all-in-1 query from sha1 to occurrence information query name: swh_content_find_occurrence given the sha1 of some content, returns info about its origin (type, url) and context (branch, revision, path) | 25 September 2015, 09:52:02 UTC |
785f69b | Stefano Zacchiroli | 24 September 2015, 19:57:23 UTC | find queries: comment SQL code | 25 September 2015, 09:52:02 UTC |
8c5d7f5 | Stefano Zacchiroli | 24 September 2015, 19:38:23 UTC | new queries to find dir/occurrence of some content queries: - swh_content_find_directory(content_id sha1): content_dir - swh_revision_find_occurrence(revision_id sha1_git): occurrence Sample use: zack-linuxgit-arrays=> select sha1 from content offset 100000 limit 1; sha1 -------------------------------------------- \xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1 (1 row) Time: 21,751 ms zack-linuxgit-arrays=> select * from swh_content_find_directory('\xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1'::sha1); directory | path --------------------------------------------+---------------------------- \x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e | drivers/md/dm-log-writes.c (1 row) Time: 729,122 ms zack-linuxgit-arrays=> select id from revision where directory = '\x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e'::sha1_git; id -------------------------------------------- \x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7 (1 row) Time: 395,754 ms zack-linuxgit-arrays=> select * from swh_revision_find_occurrence('\x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7'::sha1_git); origin | branch | revision --------+--------------------+-------------------------------------------- 1 | refs/tags/v4.1-rc3 | \x030bbdbf4c833bc69f502eae58498bc5572db736 (1 row) Time: 22,006 ms zack-linuxgit-arrays=> | 25 September 2015, 09:52:02 UTC |
372ee30 | Nicolas Dandrimont | 25 September 2015, 07:28:34 UTC | Add debian metadata | 25 September 2015, 09:12:50 UTC |
1ea9a1d | Nicolas Dandrimont | 25 September 2015, 07:28:00 UTC | Add missing file README.dev | 25 September 2015, 07:28:00 UTC |
bfb0274 | Nicolas Dandrimont | 25 September 2015, 07:27:50 UTC | Add missing dependency on swh.core | 25 September 2015, 07:27:50 UTC |
5c72222 | Nicolas Dandrimont | 24 September 2015, 13:49:52 UTC | swh.storage.storage: update docs of origin_add_one | 24 September 2015, 13:49:52 UTC |
9b78bfb | Nicolas Dandrimont | 24 September 2015, 13:47:05 UTC | swh.storage.storage: make origin_add_one idempotent | 24 September 2015, 13:48:15 UTC |
ccbc216 | Antoine R. Dumont | 24 September 2015, 13:02:19 UTC | Simplify the api content_present Only True if found, False otherwise. And raises ValueError in case of bad inputs (empty input or no sha1, sha256 keys) | 24 September 2015, 13:02:19 UTC |
f2dac75 | Antoine R. Dumont | 24 September 2015, 12:14:44 UTC | Fix: Escape is only for copy_to as psycopg2 do not escape for copy_to as it does for other queries | 24 September 2015, 12:14:44 UTC |
a8ddcd5 | Antoine R. Dumont | 24 September 2015, 12:06:34 UTC | Improve formatting of the sql query | 24 September 2015, 12:06:34 UTC |
bf19550 | Antoine R. Dumont | 24 September 2015, 12:05:14 UTC | Simplify command that triggers the storage server | 24 September 2015, 12:05:14 UTC |
e1a2df2 | Antoine R. Dumont | 23 September 2015, 09:15:00 UTC | Implement the content_present api | 24 September 2015, 11:40:11 UTC |
4348ff4 | Antoine R. Dumont | 22 September 2015, 12:39:39 UTC | Add a README.dev on how to run | 24 September 2015, 11:40:11 UTC |
0444468 | Antoine R. Dumont | 22 September 2015, 12:39:25 UTC | Open /content/present api | 24 September 2015, 11:40:11 UTC |
95e159c | Stefano Zacchiroli | 23 September 2015, 15:35:42 UTC | query swh_revision_list: add TODOs about ordering | 23 September 2015, 15:35:42 UTC |
f9621ae | Stefano Zacchiroli | 23 September 2015, 15:28:42 UTC | new SQL queries: revision rev-list & revision log | 23 September 2015, 15:28:42 UTC |
d85261c | Stefano Zacchiroli | 23 September 2015, 14:03:43 UTC | swh-func.sql: use ENUM type directory_entry_type instead of TEXT. Adapt swh_directory_walk_one accordingly | 23 September 2015, 14:03:43 UTC |
f61605e | Stefano Zacchiroli | 23 September 2015, 13:15:55 UTC | SQL diagram: add deps occurrences -> revision | 23 September 2015, 13:15:55 UTC |
d88a775 | Nicolas Dandrimont | 23 September 2015, 12:55:18 UTC | swh.storage.api: Add logging to the wsgi app | 23 September 2015, 12:55:18 UTC |
df97c2b | Nicolas Dandrimont | 23 September 2015, 12:54:54 UTC | swh.storage.api: Attempt closing temporary files | 23 September 2015, 12:54:54 UTC |
a6e88b2 | Nicolas Dandrimont | 23 September 2015, 09:55:13 UTC | remote_storage: use proper filename for metadata | 23 September 2015, 09:55:13 UTC |
4287922 | Nicolas Dandrimont | 23 September 2015, 09:51:38 UTC | remote_storage: Post contents as multipart files | 23 September 2015, 09:51:38 UTC |
6c49702 | Nicolas Dandrimont | 23 September 2015, 09:51:17 UTC | Add missing requirements | 23 September 2015, 09:51:17 UTC |
83a5963 | Nicolas Dandrimont | 23 September 2015, 08:42:30 UTC | swh.storage.api: add run_from_webserver wrapper | 23 September 2015, 08:42:30 UTC |
20f043b | Nicolas Dandrimont | 22 September 2015, 15:02:34 UTC | swh.storage.api: Properly escape arbitrary byte sequences in arguments | 22 September 2015, 15:02:34 UTC |
9185046 | Nicolas Dandrimont | 22 September 2015, 14:02:03 UTC | Add setup.py and friends | 22 September 2015, 14:02:37 UTC |
42946e1 | Stefano Zacchiroli | 22 September 2015, 10:17:00 UTC | SQL diagram: add missing edges for releases/revisions | 22 September 2015, 10:17:00 UTC |
49387fb | Stefano Zacchiroli | 22 September 2015, 10:12:33 UTC | SQL diagram: fix target of dir entry -> content arrow | 22 September 2015, 10:12:33 UTC |
51272e2 | Nicolas Dandrimont | 22 September 2015, 09:56:25 UTC | tests: add documentation for Test{,Remote}Storage | 22 September 2015, 09:57:14 UTC |
4ae8648 | Stefano Zacchiroli | 22 September 2015, 09:24:40 UTC | DB schema graph: logical placement of dir entries/lists | 22 September 2015, 09:24:40 UTC |
d9f2995 | Stefano Zacchiroli | 22 September 2015, 09:19:01 UTC | clusters.dot: document how to reference individual columns | 22 September 2015, 09:19:01 UTC |
aa2fbb9 | Stefano Zacchiroli | 22 September 2015, 09:17:58 UTC | SQL schema: add implicit FK dir_entry_dir -> dir | 22 September 2015, 09:17:58 UTC |
687f2c2 | Stefano Zacchiroli | 22 September 2015, 09:15:00 UTC | SQL schema graph: more granular implicit FK arrows in particular: make both arrow sources and arrow destinations point to specific table fields, rather than to whole tables | 22 September 2015, 09:15:03 UTC |
6b36837 | Stefano Zacchiroli | 22 September 2015, 09:06:25 UTC | SQL schema: add implicit FK on directories i.e., port the schema graph after switching to arrays for encoding directories | 22 September 2015, 09:07:05 UTC |
d9b14ab | Nicolas Dandrimont | 22 September 2015, 09:04:10 UTC | tests: make both storage tests inherit from an abstract class This prevents tests from the base class from running twice | 22 September 2015, 09:04:44 UTC |
f70bcd5 | Nicolas Dandrimont | 22 September 2015, 09:03:15 UTC | test_remote_storage: Add code to run the server automatically | 22 September 2015, 09:04:44 UTC |
c005660 | Nicolas Dandrimont | 22 September 2015, 09:01:55 UTC | swh.storage.api: add / route for server testing | 22 September 2015, 09:04:44 UTC |
19cc2d8 | Stefano Zacchiroli | 21 September 2015, 19:07:57 UTC | add requirements.txt, listing psycopg2 as dependency | 21 September 2015, 19:07:57 UTC |
caedb46 | Nicolas Dandrimont | 21 September 2015, 16:09:23 UTC | Add a remote API server and a client for remote storage | 21 September 2015, 16:10:25 UTC |
cc87c6f | Nicolas Dandrimont | 21 September 2015, 16:07:58 UTC | test_storage: don't break if the storage has no objstorage In prevision of the remote storage tests. | 21 September 2015, 16:10:25 UTC |
e07902a | Stefano Zacchiroli | 21 September 2015, 15:24:02 UTC | db.py: drop unused csv module (pep8) this makes "make check" pass again on the whole module | 21 September 2015, 15:24:02 UTC |
37e7281 | Stefano Zacchiroli | 21 September 2015, 15:22:44 UTC | db.py: use single quotes consistently (style) | 21 September 2015, 15:22:44 UTC |
16aca48 | Stefano Zacchiroli | 21 September 2015, 15:21:16 UTC | db: uniform case-ing in SQL query (cosmetic) | 21 September 2015, 15:21:16 UTC |