| 329f4cf | Stefano Zacchiroli | 30 September 2015, 10:11:55 UTC | SQL: nicer API for swh_content_find_* functions the functions no longer fail with internal errors (due to "insert into ... strict"), but rather return NULL in those cases, because no match has been found. | 30 September 2015, 10:14:29 UTC |
| 7fe460f | Stefano Zacchiroli | 30 September 2015, 09:01:43 UTC | bump SQL schema version to 17 include upgrade script from 16 | 30 September 2015, 09:04:30 UTC |
| be72b19 | Stefano Zacchiroli | 30 September 2015, 08:57:00 UTC | sql-func refactoring: single dir*_entry_add function The function is parametric in the type of entries to be added (file/dir/rev). Get rid of the previous 3 (99% overlapping) functions swh_directory_{file,rev,dir}_entry_add functions. Make storage.py use the new function. | 30 September 2015, 09:04:30 UTC |
| 4612bc6 | Antoine R. Dumont | 30 September 2015, 08:57:31 UTC | Refactor - pep8 compliance - Remove unused import | 30 September 2015, 08:57:57 UTC |
| c956386 | Antoine R. Dumont | 30 September 2015, 08:39:42 UTC | Improve output response for find occurrence api | 30 September 2015, 08:56:54 UTC |
| 141d392 | Antoine R. Dumont | 29 September 2015, 15:32:07 UTC | Implement content_find_occurrence up to the db | 30 September 2015, 08:43:30 UTC |
| 3e2fe23 | Antoine R. Dumont | 29 September 2015, 14:54:34 UTC | Refactor - db.py returns None or the content found The triplet of None should be sandboxed to db.py | 30 September 2015, 08:43:30 UTC |
| 91debb1 | Antoine R. Dumont | 29 September 2015, 13:52:57 UTC | Refactor - Permit reuse of the content_find api | 30 September 2015, 08:43:30 UTC |
| 6944ab8 | Antoine R. Dumont | 29 September 2015, 13:18:15 UTC | Refactor - Return result directory Remove the needless yield | 30 September 2015, 08:42:51 UTC |
| 311de1b | Antoine R. Dumont | 29 September 2015, 13:08:20 UTC | Refactor - Reuse | 30 September 2015, 08:42:51 UTC |
| df4bdfa | Antoine R. Dumont | 29 September 2015, 12:16:27 UTC | Open content-find-occurrence | 30 September 2015, 08:42:51 UTC |
| bdfc021 | Stefano Zacchiroli | 30 September 2015, 07:33:27 UTC | test_storage: add collision detection test | 30 September 2015, 07:40:40 UTC |
| 9aa18e4 | Stefano Zacchiroli | 30 September 2015, 07:25:06 UTC | test_api_client: use absolute import to allow rm __init__.py | 30 September 2015, 07:25:06 UTC |
| b4e4b20 | Stefano Zacchiroli | 30 September 2015, 07:18:22 UTC | SQL swh_content_missing: build result incrementally by uniting on contents that appear to be missing when looking at a single checksum algorithm. This is semantically equivalent to the previous approach, but allows to exploit single-column indexes; which are non-redundant, smaller, and hence more likely to be used. This is the "metro ligne 6" approach. | 30 September 2015, 07:18:22 UTC |
| 7a33494 | Stefano Zacchiroli | 29 September 2015, 16:15:56 UTC | db_testing.py: allow to customize DB restore | 29 September 2015, 16:15:56 UTC |
| f88331c | Nicolas Dandrimont | 29 September 2015, 14:51:45 UTC | Bump dependencies on swh.core to >= 0.0.6 | 29 September 2015, 14:51:45 UTC |
| bfe9016 | Nicolas Dandrimont | 29 September 2015, 14:51:25 UTC | debian/control: indent with spaces | 29 September 2015, 14:51:25 UTC |
| e424885 | Nicolas Dandrimont | 29 September 2015, 14:36:32 UTC | Bump dependency on swh.core | 29 September 2015, 14:41:40 UTC |
| bb8cdff | Stefano Zacchiroli | 29 September 2015, 14:38:30 UTC | move db_test.py from swh-storage to swh-core | 29 September 2015, 14:39:10 UTC |
| 260c379 | Nicolas Dandrimont | 29 September 2015, 14:26:51 UTC | Add swh.storage.api package to setup.py | 29 September 2015, 14:26:51 UTC |
| 2b46e69 | Nicolas Dandrimont | 29 September 2015, 13:04:43 UTC | swh.storage.api: Serialize/deserialize exceptions This currently uses pickle which breaks the msgpack + dict "abstraction" we settled on. This should be fixed eventually. | 29 September 2015, 13:04:43 UTC |
| 58f821c | Nicolas Dandrimont | 29 September 2015, 11:52:19 UTC | Update api client and server to msgpack | 29 September 2015, 11:55:22 UTC |
| 79d6539 | Stefano Zacchiroli | 29 September 2015, 09:16:41 UTC | gitignore: ignore .eggs/ dir | 29 September 2015, 09:16:41 UTC |
| b041526 | Antoine R. Dumont | 29 September 2015, 07:23:23 UTC | Refactor - Migrate content_present to content_find with unified api | 29 September 2015, 08:28:05 UTC |
| 1ddeff4 | Nicolas Dandrimont | 29 September 2015, 08:21:38 UTC | Move remote_storage and api to api.client and api.server Closes T44. | 29 September 2015, 08:21:38 UTC |
| 1cb39a2 | Nicolas Dandrimont | 28 September 2015, 15:28:42 UTC | sql/swh-schema: unix_path to bytea | 28 September 2015, 15:36:28 UTC |
| f660268 | Nicolas Dandrimont | 28 September 2015, 15:14:37 UTC | sql/upgrades/015: drop function before readding it | 28 September 2015, 15:35:16 UTC |
| d371898 | Stefano Zacchiroli | 28 September 2015, 14:19:37 UTC | SQL schema diagram: add skipped_content | 28 September 2015, 14:19:37 UTC |
| eb7493c | Nicolas Dandrimont | 28 September 2015, 14:01:25 UTC | sql/upgrades/015: bump database version | 28 September 2015, 14:01:25 UTC |
| 6a4da5c | Nicolas Dandrimont | 28 September 2015, 13:58:56 UTC | sql/swh-func: fix outdated comments | 28 September 2015, 14:00:02 UTC |
| 13315d2 | Nicolas Dandrimont | 28 September 2015, 13:55:16 UTC | sql/sql-func: Make tests pass again | 28 September 2015, 14:00:02 UTC |
| be3910e | Stefano Zacchiroli | 28 September 2015, 13:46:39 UTC | Merge branch 'merge-dir-entries' | 28 September 2015, 13:46:39 UTC |
| 8870ff4 | Stefano Zacchiroli | 28 September 2015, 13:42:19 UTC | swh-func.sql: add refactoring TODO item | 28 September 2015, 13:46:28 UTC |
| 3c17d73 | Stefano Zacchiroli | 28 September 2015, 13:40:21 UTC | swh_directory_entry_*_add: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
| 345293a | Stefano Zacchiroli | 27 September 2015, 17:07:55 UTC | swh_directory_walk_one: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
| f4b8b40 | Stefano Zacchiroli | 27 September 2015, 16:27:55 UTC | swh_content_find_dir: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
| c08ee26 | Stefano Zacchiroli | 27 September 2015, 14:35:32 UTC | upgrades/015.sql: migration script for dir_list_* table merge | 28 September 2015, 13:46:28 UTC |
| f068673 | Stefano Zacchiroli | 27 September 2015, 13:45:24 UTC | SQL schema: merge dir entry arrays into directory This removes tables directory_list_{file,dir,rev}, and add corresponding arrays {dir,file,rev}_entries to the directory table, which was just a set up to now. Indexes migrated accordingly. | 28 September 2015, 13:46:28 UTC |
| 945bdfb | Nicolas Dandrimont | 28 September 2015, 13:41:14 UTC | sql/sql-func: copy origin from tmp_skipped_content | 28 September 2015, 13:41:14 UTC |
| 573b8d4 | Nicolas Dandrimont | 28 September 2015, 13:33:20 UTC | swh.storage.storage: Only add missing skipped_content | 28 September 2015, 13:33:20 UTC |
| e16565e | Nicolas Dandrimont | 28 September 2015, 13:32:16 UTC | swh.storage.storage: add origin to skipped_content | 28 September 2015, 13:32:16 UTC |
| 9188980 | Nicolas Dandrimont | 28 September 2015, 12:49:39 UTC | sql/swh-func.sql: remove useless declaration | 28 September 2015, 12:49:39 UTC |
| 8ff3109 | Nicolas Dandrimont | 28 September 2015, 12:49:12 UTC | swh.storage.storage: Allow storing "skipped" contents | 28 September 2015, 12:49:12 UTC |
| 09193a5 | Nicolas Dandrimont | 28 September 2015, 12:43:27 UTC | swh.storage.db: map stored procedures for skipped_content | 28 September 2015, 12:43:27 UTC |
| 9073916 | Nicolas Dandrimont | 28 September 2015, 12:42:39 UTC | sql schema: update skipped_content and add stored procedures for update | 28 September 2015, 12:42:39 UTC |
| f1f62bd | Nicolas Dandrimont | 28 September 2015, 09:44:15 UTC | sql/swh-schema: add index on content(sha256) | 28 September 2015, 09:44:15 UTC |
| d53dc22 | Nicolas Dandrimont | 28 September 2015, 09:43:48 UTC | sql/swh-schema: add skipped_content table | 28 September 2015, 09:43:48 UTC |
| 0dd1ece | Stefano Zacchiroli | 27 September 2015, 13:35:38 UTC | SQL: add swh_content_find, to lookup content by checksum(s) Intente use is content-by-content lookup (e.g., in the web app), not mass addition or check for missing contents. We already have swh_content_missing for that. | 27 September 2015, 13:35:38 UTC |
| 7de609d | Stefano Zacchiroli | 27 September 2015, 12:43:25 UTC | SQL: improve indentation of directory_walk_one to avoid very long lines | 27 September 2015, 12:43:25 UTC |
| 1ce7f21 | Stefano Zacchiroli | 25 September 2015, 07:43:56 UTC | swh_content_find_occ. query: improve doc string | 25 September 2015, 09:52:02 UTC |
| c4e2e78 | Stefano Zacchiroli | 24 September 2015, 20:11:38 UTC | all-in-1 query from sha1 to occurrence information query name: swh_content_find_occurrence given the sha1 of some content, returns info about its origin (type, url) and context (branch, revision, path) | 25 September 2015, 09:52:02 UTC |
| 785f69b | Stefano Zacchiroli | 24 September 2015, 19:57:23 UTC | find queries: comment SQL code | 25 September 2015, 09:52:02 UTC |
| 8c5d7f5 | Stefano Zacchiroli | 24 September 2015, 19:38:23 UTC | new queries to find dir/occurrence of some content queries: - swh_content_find_directory(content_id sha1): content_dir - swh_revision_find_occurrence(revision_id sha1_git): occurrence Sample use: zack-linuxgit-arrays=> select sha1 from content offset 100000 limit 1; sha1 -------------------------------------------- \xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1 (1 row) Time: 21,751 ms zack-linuxgit-arrays=> select * from swh_content_find_directory('\xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1'::sha1); directory | path --------------------------------------------+---------------------------- \x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e | drivers/md/dm-log-writes.c (1 row) Time: 729,122 ms zack-linuxgit-arrays=> select id from revision where directory = '\x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e'::sha1_git; id -------------------------------------------- \x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7 (1 row) Time: 395,754 ms zack-linuxgit-arrays=> select * from swh_revision_find_occurrence('\x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7'::sha1_git); origin | branch | revision --------+--------------------+-------------------------------------------- 1 | refs/tags/v4.1-rc3 | \x030bbdbf4c833bc69f502eae58498bc5572db736 (1 row) Time: 22,006 ms zack-linuxgit-arrays=> | 25 September 2015, 09:52:02 UTC |
| 372ee30 | Nicolas Dandrimont | 25 September 2015, 07:28:34 UTC | Add debian metadata | 25 September 2015, 09:12:50 UTC |
| 1ea9a1d | Nicolas Dandrimont | 25 September 2015, 07:28:00 UTC | Add missing file README.dev | 25 September 2015, 07:28:00 UTC |
| bfb0274 | Nicolas Dandrimont | 25 September 2015, 07:27:50 UTC | Add missing dependency on swh.core | 25 September 2015, 07:27:50 UTC |
| 5c72222 | Nicolas Dandrimont | 24 September 2015, 13:49:52 UTC | swh.storage.storage: update docs of origin_add_one | 24 September 2015, 13:49:52 UTC |
| 9b78bfb | Nicolas Dandrimont | 24 September 2015, 13:47:05 UTC | swh.storage.storage: make origin_add_one idempotent | 24 September 2015, 13:48:15 UTC |
| ccbc216 | Antoine R. Dumont | 24 September 2015, 13:02:19 UTC | Simplify the api content_present Only True if found, False otherwise. And raises ValueError in case of bad inputs (empty input or no sha1, sha256 keys) | 24 September 2015, 13:02:19 UTC |
| f2dac75 | Antoine R. Dumont | 24 September 2015, 12:14:44 UTC | Fix: Escape is only for copy_to as psycopg2 do not escape for copy_to as it does for other queries | 24 September 2015, 12:14:44 UTC |
| a8ddcd5 | Antoine R. Dumont | 24 September 2015, 12:06:34 UTC | Improve formatting of the sql query | 24 September 2015, 12:06:34 UTC |
| bf19550 | Antoine R. Dumont | 24 September 2015, 12:05:14 UTC | Simplify command that triggers the storage server | 24 September 2015, 12:05:14 UTC |
| e1a2df2 | Antoine R. Dumont | 23 September 2015, 09:15:00 UTC | Implement the content_present api | 24 September 2015, 11:40:11 UTC |
| 4348ff4 | Antoine R. Dumont | 22 September 2015, 12:39:39 UTC | Add a README.dev on how to run | 24 September 2015, 11:40:11 UTC |
| 0444468 | Antoine R. Dumont | 22 September 2015, 12:39:25 UTC | Open /content/present api | 24 September 2015, 11:40:11 UTC |
| 95e159c | Stefano Zacchiroli | 23 September 2015, 15:35:42 UTC | query swh_revision_list: add TODOs about ordering | 23 September 2015, 15:35:42 UTC |
| f9621ae | Stefano Zacchiroli | 23 September 2015, 15:28:42 UTC | new SQL queries: revision rev-list & revision log | 23 September 2015, 15:28:42 UTC |
| d85261c | Stefano Zacchiroli | 23 September 2015, 14:03:43 UTC | swh-func.sql: use ENUM type directory_entry_type instead of TEXT. Adapt swh_directory_walk_one accordingly | 23 September 2015, 14:03:43 UTC |
| f61605e | Stefano Zacchiroli | 23 September 2015, 13:15:55 UTC | SQL diagram: add deps occurrences -> revision | 23 September 2015, 13:15:55 UTC |
| d88a775 | Nicolas Dandrimont | 23 September 2015, 12:55:18 UTC | swh.storage.api: Add logging to the wsgi app | 23 September 2015, 12:55:18 UTC |
| df97c2b | Nicolas Dandrimont | 23 September 2015, 12:54:54 UTC | swh.storage.api: Attempt closing temporary files | 23 September 2015, 12:54:54 UTC |
| a6e88b2 | Nicolas Dandrimont | 23 September 2015, 09:55:13 UTC | remote_storage: use proper filename for metadata | 23 September 2015, 09:55:13 UTC |
| 4287922 | Nicolas Dandrimont | 23 September 2015, 09:51:38 UTC | remote_storage: Post contents as multipart files | 23 September 2015, 09:51:38 UTC |
| 6c49702 | Nicolas Dandrimont | 23 September 2015, 09:51:17 UTC | Add missing requirements | 23 September 2015, 09:51:17 UTC |
| 83a5963 | Nicolas Dandrimont | 23 September 2015, 08:42:30 UTC | swh.storage.api: add run_from_webserver wrapper | 23 September 2015, 08:42:30 UTC |
| 20f043b | Nicolas Dandrimont | 22 September 2015, 15:02:34 UTC | swh.storage.api: Properly escape arbitrary byte sequences in arguments | 22 September 2015, 15:02:34 UTC |
| 9185046 | Nicolas Dandrimont | 22 September 2015, 14:02:03 UTC | Add setup.py and friends | 22 September 2015, 14:02:37 UTC |
| 42946e1 | Stefano Zacchiroli | 22 September 2015, 10:17:00 UTC | SQL diagram: add missing edges for releases/revisions | 22 September 2015, 10:17:00 UTC |
| 49387fb | Stefano Zacchiroli | 22 September 2015, 10:12:33 UTC | SQL diagram: fix target of dir entry -> content arrow | 22 September 2015, 10:12:33 UTC |
| 51272e2 | Nicolas Dandrimont | 22 September 2015, 09:56:25 UTC | tests: add documentation for Test{,Remote}Storage | 22 September 2015, 09:57:14 UTC |
| 4ae8648 | Stefano Zacchiroli | 22 September 2015, 09:24:40 UTC | DB schema graph: logical placement of dir entries/lists | 22 September 2015, 09:24:40 UTC |
| d9f2995 | Stefano Zacchiroli | 22 September 2015, 09:19:01 UTC | clusters.dot: document how to reference individual columns | 22 September 2015, 09:19:01 UTC |
| aa2fbb9 | Stefano Zacchiroli | 22 September 2015, 09:17:58 UTC | SQL schema: add implicit FK dir_entry_dir -> dir | 22 September 2015, 09:17:58 UTC |
| 687f2c2 | Stefano Zacchiroli | 22 September 2015, 09:15:00 UTC | SQL schema graph: more granular implicit FK arrows in particular: make both arrow sources and arrow destinations point to specific table fields, rather than to whole tables | 22 September 2015, 09:15:03 UTC |
| 6b36837 | Stefano Zacchiroli | 22 September 2015, 09:06:25 UTC | SQL schema: add implicit FK on directories i.e., port the schema graph after switching to arrays for encoding directories | 22 September 2015, 09:07:05 UTC |
| d9b14ab | Nicolas Dandrimont | 22 September 2015, 09:04:10 UTC | tests: make both storage tests inherit from an abstract class This prevents tests from the base class from running twice | 22 September 2015, 09:04:44 UTC |
| f70bcd5 | Nicolas Dandrimont | 22 September 2015, 09:03:15 UTC | test_remote_storage: Add code to run the server automatically | 22 September 2015, 09:04:44 UTC |
| c005660 | Nicolas Dandrimont | 22 September 2015, 09:01:55 UTC | swh.storage.api: add / route for server testing | 22 September 2015, 09:04:44 UTC |
| 19cc2d8 | Stefano Zacchiroli | 21 September 2015, 19:07:57 UTC | add requirements.txt, listing psycopg2 as dependency | 21 September 2015, 19:07:57 UTC |
| caedb46 | Nicolas Dandrimont | 21 September 2015, 16:09:23 UTC | Add a remote API server and a client for remote storage | 21 September 2015, 16:10:25 UTC |
| cc87c6f | Nicolas Dandrimont | 21 September 2015, 16:07:58 UTC | test_storage: don't break if the storage has no objstorage In prevision of the remote storage tests. | 21 September 2015, 16:10:25 UTC |
| e07902a | Stefano Zacchiroli | 21 September 2015, 15:24:02 UTC | db.py: drop unused csv module (pep8) this makes "make check" pass again on the whole module | 21 September 2015, 15:24:02 UTC |
| 37e7281 | Stefano Zacchiroli | 21 September 2015, 15:22:44 UTC | db.py: use single quotes consistently (style) | 21 September 2015, 15:22:44 UTC |
| 16aca48 | Stefano Zacchiroli | 21 September 2015, 15:21:16 UTC | db: uniform case-ing in SQL query (cosmetic) | 21 September 2015, 15:21:16 UTC |
| ba5443a | Nicolas Dandrimont | 21 September 2015, 14:02:29 UTC | sql/swh-schema.sql: add indexes on directory lists | 21 September 2015, 14:02:29 UTC |
| b8aa5c4 | Nicolas Dandrimont | 18 September 2015, 13:10:35 UTC | Merge branch 'master' into directory-listing-arrays | 18 September 2015, 13:10:35 UTC |
| e9f08de | Nicolas Dandrimont | 18 September 2015, 10:58:11 UTC | swh.storage.storage: allow listing missing contents per hash | 18 September 2015, 10:58:11 UTC |
| 662c8ee | Nicolas Dandrimont | 18 September 2015, 10:57:03 UTC | swh.storage.storage: add the contents that _are_ missing | 18 September 2015, 10:57:03 UTC |
| 04c1db8 | Nicolas Dandrimont | 18 September 2015, 10:42:14 UTC | swh.storage.storage: content ids are sha1, not sha1_git | 18 September 2015, 10:42:14 UTC |
| 8f97c01 | Nicolas Dandrimont | 18 September 2015, 10:31:25 UTC | Merge branch 'master' into directory-listing-arrays | 18 September 2015, 10:31:25 UTC |
