| 3c66ff6 | Nicolas Dandrimont | 06 October 2015, 15:37:00 UTC | New upstream version 0.0.10 | 06 October 2015, 15:37:00 UTC |
| a4fb8df | Nicolas Dandrimont | 06 October 2015, 12:11:12 UTC | sql: add upgrade script for 21->22 | 06 October 2015, 12:11:12 UTC |
| e80adb8 | Nicolas Dandrimont | 06 October 2015, 11:54:01 UTC | swh-schema: person (name, email) -> bytea Closes T65 in part | 06 October 2015, 12:01:40 UTC |
| 5e7fd18 | Stefano Zacchiroli | 05 October 2015, 20:15:18 UTC | sql/TODO: updated to reflect current status | 05 October 2015, 20:15:18 UTC |
| 04367e1 | Nicolas Dandrimont | 05 October 2015, 16:57:14 UTC | swh.storage.occurrence_add: parse string dates with dateutil | 05 October 2015, 16:57:14 UTC |
| 1f7ca7c | Nicolas Dandrimont | 05 October 2015, 16:41:37 UTC | swh.storage.storage: make occurrence_add support overlapping intervals Closes T45 | 05 October 2015, 16:47:16 UTC |
| 71c4ab6 | Nicolas Dandrimont | 05 October 2015, 16:40:22 UTC | swh.storage.db: plug swh_occurrence_history_add | 05 October 2015, 16:47:16 UTC |
| c91dd71 | Nicolas Dandrimont | 05 October 2015, 16:39:24 UTC | swh.storage.db: proper copy escaping for datetimes, ranges | 05 October 2015, 16:47:16 UTC |
| 3a44ef5 | Nicolas Dandrimont | 05 October 2015, 16:45:58 UTC | sql/upgrades/021: add 020->021 upgrade script | 05 October 2015, 16:47:16 UTC |
| b3aac6a | Nicolas Dandrimont | 05 October 2015, 16:38:05 UTC | swh-func: add swh_occurrence_history_add() | 05 October 2015, 16:38:05 UTC |
| 3c1f401 | Nicolas Dandrimont | 05 October 2015, 14:11:19 UTC | sql/upgrades/020: add upgrade script for 19->20 | 05 October 2015, 14:11:19 UTC |
| 103950b | Nicolas Dandrimont | 05 October 2015, 14:00:51 UTC | sql/swh-func: add table aliases to "where not exists" queries | 05 October 2015, 14:00:51 UTC |
| c4bb88d | Nicolas Dandrimont | 05 October 2015, 08:45:01 UTC | swh.storage.tests: Actually check the error message | 05 October 2015, 08:45:01 UTC |
| bf9253e | Nicolas Dandrimont | 05 October 2015, 08:33:42 UTC | swh.storage.tests: clean the database on teardown | 05 October 2015, 08:42:10 UTC |
| fedd0ff | Stefano Zacchiroli | 05 October 2015, 08:00:30 UTC | storage.content_find: better testing | 05 October 2015, 08:00:30 UTC |
| 96adeaa | Stefano Zacchiroli | 04 October 2015, 18:39:46 UTC | storage.content_find: refactor argument checking also: - avoid overzeaolus assertion in test, to make it more robust - improve method doc string | 04 October 2015, 18:52:00 UTC |
| f1fb64c | Stefano Zacchiroli | 03 October 2015, 12:44:03 UTC | sql/bin/db-upgrade: compute SQL upgrades using git and apgdiff | 03 October 2015, 12:44:03 UTC |
| 07c8f44 | Nicolas Dandrimont | 02 October 2015, 15:18:34 UTC | sql: Improve performance of swh_{directory,revision,release}_missing | 02 October 2015, 15:18:34 UTC |
| 412df3d | Nicolas Dandrimont | 01 October 2015, 17:03:00 UTC | New upstream version 0.0.9 | 01 October 2015, 17:03:00 UTC |
| 08ffeb2 | Nicolas Dandrimont | 01 October 2015, 17:02:10 UTC | Remove atime, mtime, ctime from directory entries | 01 October 2015, 17:02:10 UTC |
| 617de19 | Nicolas Dandrimont | 01 October 2015, 09:32:46 UTC | New upstream version 0.0.8 | 01 October 2015, 09:32:46 UTC |
| 43f5bf9 | Nicolas Dandrimont | 01 October 2015, 09:32:07 UTC | Bump swh.core dependency version | 01 October 2015, 09:32:07 UTC |
| a718e9d | Stefano Zacchiroli | 30 September 2015, 14:17:10 UTC | db.py: drop raises psycopg2.internalerror it is no longer the case | 30 September 2015, 14:17:10 UTC |
| 64a7832 | Stefano Zacchiroli | 30 September 2015, 12:51:01 UTC | objstorage: fix predictable tempfile race when adding objects Before this change, two workers adding the same object will end up racing to write <SHA1>.tmp. That should not lead to corruption, because they are (in principle) going to write the exact same content, but it made the one of the two that arrives last fail the final rename. With this fix they will write in different <SHA1>.RANDOM_STUFF.tmp, and then rename their own file into place. Note that this change slightly increases the chances of having garbage in the object store, so we will need to look for this during GC passes. | 30 September 2015, 12:51:01 UTC |
| e4feb05 | Stefano Zacchiroli | 30 September 2015, 10:31:16 UTC | upgrades/17.sql: add recent changes | 30 September 2015, 10:33:00 UTC |
| 25b0544 | Stefano Zacchiroli | 30 September 2015, 09:56:25 UTC | swh-func.sql: use LANGUAGE sql wherever possible Overall, we can use LANGUAGE sql only in 6 cases (vs 17 functions that remains plpgsql). This is in large part due to the fact that LANGUAGE sql stored procedures cannot non-existent tables, and most notably temporary tables. Closes T55. | 30 September 2015, 10:33:00 UTC |
| cd507ba | Stefano Zacchiroli | 30 September 2015, 10:11:55 UTC | SQL: nicer API for swh_content_find_* functions the functions no longer fail with internal errors (due to "insert into ... strict"), but rather return NULL in those cases, because no match has been found. | 30 September 2015, 10:33:00 UTC |
| d4131b3 | Antoine R. Dumont | 30 September 2015, 09:48:37 UTC | Update docstring | 30 September 2015, 09:48:46 UTC |
| 7fe460f | Stefano Zacchiroli | 30 September 2015, 09:01:43 UTC | bump SQL schema version to 17 include upgrade script from 16 | 30 September 2015, 09:04:30 UTC |
| be72b19 | Stefano Zacchiroli | 30 September 2015, 08:57:00 UTC | sql-func refactoring: single dir*_entry_add function The function is parametric in the type of entries to be added (file/dir/rev). Get rid of the previous 3 (99% overlapping) functions swh_directory_{file,rev,dir}_entry_add functions. Make storage.py use the new function. | 30 September 2015, 09:04:30 UTC |
| 4612bc6 | Antoine R. Dumont | 30 September 2015, 08:57:31 UTC | Refactor - pep8 compliance - Remove unused import | 30 September 2015, 08:57:57 UTC |
| c956386 | Antoine R. Dumont | 30 September 2015, 08:39:42 UTC | Improve output response for find occurrence api | 30 September 2015, 08:56:54 UTC |
| 141d392 | Antoine R. Dumont | 29 September 2015, 15:32:07 UTC | Implement content_find_occurrence up to the db | 30 September 2015, 08:43:30 UTC |
| 3e2fe23 | Antoine R. Dumont | 29 September 2015, 14:54:34 UTC | Refactor - db.py returns None or the content found The triplet of None should be sandboxed to db.py | 30 September 2015, 08:43:30 UTC |
| 91debb1 | Antoine R. Dumont | 29 September 2015, 13:52:57 UTC | Refactor - Permit reuse of the content_find api | 30 September 2015, 08:43:30 UTC |
| 6944ab8 | Antoine R. Dumont | 29 September 2015, 13:18:15 UTC | Refactor - Return result directory Remove the needless yield | 30 September 2015, 08:42:51 UTC |
| 311de1b | Antoine R. Dumont | 29 September 2015, 13:08:20 UTC | Refactor - Reuse | 30 September 2015, 08:42:51 UTC |
| df4bdfa | Antoine R. Dumont | 29 September 2015, 12:16:27 UTC | Open content-find-occurrence | 30 September 2015, 08:42:51 UTC |
| bdfc021 | Stefano Zacchiroli | 30 September 2015, 07:33:27 UTC | test_storage: add collision detection test | 30 September 2015, 07:40:40 UTC |
| 9aa18e4 | Stefano Zacchiroli | 30 September 2015, 07:25:06 UTC | test_api_client: use absolute import to allow rm __init__.py | 30 September 2015, 07:25:06 UTC |
| b4e4b20 | Stefano Zacchiroli | 30 September 2015, 07:18:22 UTC | SQL swh_content_missing: build result incrementally by uniting on contents that appear to be missing when looking at a single checksum algorithm. This is semantically equivalent to the previous approach, but allows to exploit single-column indexes; which are non-redundant, smaller, and hence more likely to be used. This is the "metro ligne 6" approach. | 30 September 2015, 07:18:22 UTC |
| 7a33494 | Stefano Zacchiroli | 29 September 2015, 16:15:56 UTC | db_testing.py: allow to customize DB restore | 29 September 2015, 16:15:56 UTC |
| 8fc078f | Nicolas Dandrimont | 29 September 2015, 14:52:54 UTC | New upstream version 0.0.7 | 29 September 2015, 14:52:54 UTC |
| f88331c | Nicolas Dandrimont | 29 September 2015, 14:51:45 UTC | Bump dependencies on swh.core to >= 0.0.6 | 29 September 2015, 14:51:45 UTC |
| bfe9016 | Nicolas Dandrimont | 29 September 2015, 14:51:25 UTC | debian/control: indent with spaces | 29 September 2015, 14:51:25 UTC |
| d4b8374 | Nicolas Dandrimont | 29 September 2015, 14:43:24 UTC | New upstream version 0.0.6 | 29 September 2015, 14:43:24 UTC |
| e424885 | Nicolas Dandrimont | 29 September 2015, 14:36:32 UTC | Bump dependency on swh.core | 29 September 2015, 14:41:40 UTC |
| bb8cdff | Stefano Zacchiroli | 29 September 2015, 14:38:30 UTC | move db_test.py from swh-storage to swh-core | 29 September 2015, 14:39:10 UTC |
| f2b5c0c | Nicolas Dandrimont | 29 September 2015, 14:27:00 UTC | New upstream version 0.0.5 | 29 September 2015, 14:27:00 UTC |
| 260c379 | Nicolas Dandrimont | 29 September 2015, 14:26:51 UTC | Add swh.storage.api package to setup.py | 29 September 2015, 14:26:51 UTC |
| 2b46e69 | Nicolas Dandrimont | 29 September 2015, 13:04:43 UTC | swh.storage.api: Serialize/deserialize exceptions This currently uses pickle which breaks the msgpack + dict "abstraction" we settled on. This should be fixed eventually. | 29 September 2015, 13:04:43 UTC |
| 58f821c | Nicolas Dandrimont | 29 September 2015, 11:52:19 UTC | Update api client and server to msgpack | 29 September 2015, 11:55:22 UTC |
| 79d6539 | Stefano Zacchiroli | 29 September 2015, 09:16:41 UTC | gitignore: ignore .eggs/ dir | 29 September 2015, 09:16:41 UTC |
| b041526 | Antoine R. Dumont | 29 September 2015, 07:23:23 UTC | Refactor - Migrate content_present to content_find with unified api | 29 September 2015, 08:28:05 UTC |
| 1ddeff4 | Nicolas Dandrimont | 29 September 2015, 08:21:38 UTC | Move remote_storage and api to api.client and api.server Closes T44. | 29 September 2015, 08:21:38 UTC |
| 1cb39a2 | Nicolas Dandrimont | 28 September 2015, 15:28:42 UTC | sql/swh-schema: unix_path to bytea | 28 September 2015, 15:36:28 UTC |
| f660268 | Nicolas Dandrimont | 28 September 2015, 15:14:37 UTC | sql/upgrades/015: drop function before readding it | 28 September 2015, 15:35:16 UTC |
| d371898 | Stefano Zacchiroli | 28 September 2015, 14:19:37 UTC | SQL schema diagram: add skipped_content | 28 September 2015, 14:19:37 UTC |
| eb7493c | Nicolas Dandrimont | 28 September 2015, 14:01:25 UTC | sql/upgrades/015: bump database version | 28 September 2015, 14:01:25 UTC |
| 6a4da5c | Nicolas Dandrimont | 28 September 2015, 13:58:56 UTC | sql/swh-func: fix outdated comments | 28 September 2015, 14:00:02 UTC |
| 13315d2 | Nicolas Dandrimont | 28 September 2015, 13:55:16 UTC | sql/sql-func: Make tests pass again | 28 September 2015, 14:00:02 UTC |
| be3910e | Stefano Zacchiroli | 28 September 2015, 13:46:39 UTC | Merge branch 'merge-dir-entries' | 28 September 2015, 13:46:39 UTC |
| 8870ff4 | Stefano Zacchiroli | 28 September 2015, 13:42:19 UTC | swh-func.sql: add refactoring TODO item | 28 September 2015, 13:46:28 UTC |
| 3c17d73 | Stefano Zacchiroli | 28 September 2015, 13:40:21 UTC | swh_directory_entry_*_add: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
| 345293a | Stefano Zacchiroli | 27 September 2015, 17:07:55 UTC | swh_directory_walk_one: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
| f4b8b40 | Stefano Zacchiroli | 27 September 2015, 16:27:55 UTC | swh_content_find_dir: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
| c08ee26 | Stefano Zacchiroli | 27 September 2015, 14:35:32 UTC | upgrades/015.sql: migration script for dir_list_* table merge | 28 September 2015, 13:46:28 UTC |
| f068673 | Stefano Zacchiroli | 27 September 2015, 13:45:24 UTC | SQL schema: merge dir entry arrays into directory This removes tables directory_list_{file,dir,rev}, and add corresponding arrays {dir,file,rev}_entries to the directory table, which was just a set up to now. Indexes migrated accordingly. | 28 September 2015, 13:46:28 UTC |
| 945bdfb | Nicolas Dandrimont | 28 September 2015, 13:41:14 UTC | sql/sql-func: copy origin from tmp_skipped_content | 28 September 2015, 13:41:14 UTC |
| 573b8d4 | Nicolas Dandrimont | 28 September 2015, 13:33:20 UTC | swh.storage.storage: Only add missing skipped_content | 28 September 2015, 13:33:20 UTC |
| e16565e | Nicolas Dandrimont | 28 September 2015, 13:32:16 UTC | swh.storage.storage: add origin to skipped_content | 28 September 2015, 13:32:16 UTC |
| 9188980 | Nicolas Dandrimont | 28 September 2015, 12:49:39 UTC | sql/swh-func.sql: remove useless declaration | 28 September 2015, 12:49:39 UTC |
| 8ff3109 | Nicolas Dandrimont | 28 September 2015, 12:49:12 UTC | swh.storage.storage: Allow storing "skipped" contents | 28 September 2015, 12:49:12 UTC |
| 09193a5 | Nicolas Dandrimont | 28 September 2015, 12:43:27 UTC | swh.storage.db: map stored procedures for skipped_content | 28 September 2015, 12:43:27 UTC |
| 9073916 | Nicolas Dandrimont | 28 September 2015, 12:42:39 UTC | sql schema: update skipped_content and add stored procedures for update | 28 September 2015, 12:42:39 UTC |
| f1f62bd | Nicolas Dandrimont | 28 September 2015, 09:44:15 UTC | sql/swh-schema: add index on content(sha256) | 28 September 2015, 09:44:15 UTC |
| d53dc22 | Nicolas Dandrimont | 28 September 2015, 09:43:48 UTC | sql/swh-schema: add skipped_content table | 28 September 2015, 09:43:48 UTC |
| 0dd1ece | Stefano Zacchiroli | 27 September 2015, 13:35:38 UTC | SQL: add swh_content_find, to lookup content by checksum(s) Intente use is content-by-content lookup (e.g., in the web app), not mass addition or check for missing contents. We already have swh_content_missing for that. | 27 September 2015, 13:35:38 UTC |
| 7de609d | Stefano Zacchiroli | 27 September 2015, 12:43:25 UTC | SQL: improve indentation of directory_walk_one to avoid very long lines | 27 September 2015, 12:43:25 UTC |
| 1ce7f21 | Stefano Zacchiroli | 25 September 2015, 07:43:56 UTC | swh_content_find_occ. query: improve doc string | 25 September 2015, 09:52:02 UTC |
| c4e2e78 | Stefano Zacchiroli | 24 September 2015, 20:11:38 UTC | all-in-1 query from sha1 to occurrence information query name: swh_content_find_occurrence given the sha1 of some content, returns info about its origin (type, url) and context (branch, revision, path) | 25 September 2015, 09:52:02 UTC |
| 785f69b | Stefano Zacchiroli | 24 September 2015, 19:57:23 UTC | find queries: comment SQL code | 25 September 2015, 09:52:02 UTC |
| 8c5d7f5 | Stefano Zacchiroli | 24 September 2015, 19:38:23 UTC | new queries to find dir/occurrence of some content queries: - swh_content_find_directory(content_id sha1): content_dir - swh_revision_find_occurrence(revision_id sha1_git): occurrence Sample use: zack-linuxgit-arrays=> select sha1 from content offset 100000 limit 1; sha1 -------------------------------------------- \xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1 (1 row) Time: 21,751 ms zack-linuxgit-arrays=> select * from swh_content_find_directory('\xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1'::sha1); directory | path --------------------------------------------+---------------------------- \x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e | drivers/md/dm-log-writes.c (1 row) Time: 729,122 ms zack-linuxgit-arrays=> select id from revision where directory = '\x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e'::sha1_git; id -------------------------------------------- \x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7 (1 row) Time: 395,754 ms zack-linuxgit-arrays=> select * from swh_revision_find_occurrence('\x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7'::sha1_git); origin | branch | revision --------+--------------------+-------------------------------------------- 1 | refs/tags/v4.1-rc3 | \x030bbdbf4c833bc69f502eae58498bc5572db736 (1 row) Time: 22,006 ms zack-linuxgit-arrays=> | 25 September 2015, 09:52:02 UTC |
| 372ee30 | Nicolas Dandrimont | 25 September 2015, 07:28:34 UTC | Add debian metadata | 25 September 2015, 09:12:50 UTC |
| 1ea9a1d | Nicolas Dandrimont | 25 September 2015, 07:28:00 UTC | Add missing file README.dev | 25 September 2015, 07:28:00 UTC |
| bfb0274 | Nicolas Dandrimont | 25 September 2015, 07:27:50 UTC | Add missing dependency on swh.core | 25 September 2015, 07:27:50 UTC |
| 5c72222 | Nicolas Dandrimont | 24 September 2015, 13:49:52 UTC | swh.storage.storage: update docs of origin_add_one | 24 September 2015, 13:49:52 UTC |
| 9b78bfb | Nicolas Dandrimont | 24 September 2015, 13:47:05 UTC | swh.storage.storage: make origin_add_one idempotent | 24 September 2015, 13:48:15 UTC |
| ccbc216 | Antoine R. Dumont | 24 September 2015, 13:02:19 UTC | Simplify the api content_present Only True if found, False otherwise. And raises ValueError in case of bad inputs (empty input or no sha1, sha256 keys) | 24 September 2015, 13:02:19 UTC |
| f2dac75 | Antoine R. Dumont | 24 September 2015, 12:14:44 UTC | Fix: Escape is only for copy_to as psycopg2 do not escape for copy_to as it does for other queries | 24 September 2015, 12:14:44 UTC |
| a8ddcd5 | Antoine R. Dumont | 24 September 2015, 12:06:34 UTC | Improve formatting of the sql query | 24 September 2015, 12:06:34 UTC |
| bf19550 | Antoine R. Dumont | 24 September 2015, 12:05:14 UTC | Simplify command that triggers the storage server | 24 September 2015, 12:05:14 UTC |
| e1a2df2 | Antoine R. Dumont | 23 September 2015, 09:15:00 UTC | Implement the content_present api | 24 September 2015, 11:40:11 UTC |
| 4348ff4 | Antoine R. Dumont | 22 September 2015, 12:39:39 UTC | Add a README.dev on how to run | 24 September 2015, 11:40:11 UTC |
| 0444468 | Antoine R. Dumont | 22 September 2015, 12:39:25 UTC | Open /content/present api | 24 September 2015, 11:40:11 UTC |
| 95e159c | Stefano Zacchiroli | 23 September 2015, 15:35:42 UTC | query swh_revision_list: add TODOs about ordering | 23 September 2015, 15:35:42 UTC |
| f9621ae | Stefano Zacchiroli | 23 September 2015, 15:28:42 UTC | new SQL queries: revision rev-list & revision log | 23 September 2015, 15:28:42 UTC |
| d85261c | Stefano Zacchiroli | 23 September 2015, 14:03:43 UTC | swh-func.sql: use ENUM type directory_entry_type instead of TEXT. Adapt swh_directory_walk_one accordingly | 23 September 2015, 14:03:43 UTC |
| f61605e | Stefano Zacchiroli | 23 September 2015, 13:15:55 UTC | SQL diagram: add deps occurrences -> revision | 23 September 2015, 13:15:55 UTC |
| d88a775 | Nicolas Dandrimont | 23 September 2015, 12:55:18 UTC | swh.storage.api: Add logging to the wsgi app | 23 September 2015, 12:55:18 UTC |
