dad8f67 | Antoine R. Dumont (@ardumont) | 15 October 2015, 14:50:34 UTC | Add gnu.org organization | 15 October 2015, 14:50:34 UTC |
7cbea2a | Antoine R. Dumont (@ardumont) | 15 October 2015, 14:50:19 UTC | Add optional 0.0.0.0 to open dev server | 15 October 2015, 14:50:19 UTC |
04ad426 | Antoine R. Dumont (@ardumont) | 14 October 2015, 14:58:58 UTC | Update sample command line | 14 October 2015, 14:58:58 UTC |
5f511b8 | Stefano Zacchiroli | 14 October 2015, 08:56:24 UTC | storage API client: support argument-less methods | 14 October 2015, 08:56:24 UTC |
166ace6 | Stefano Zacchiroli | 14 October 2015, 08:56:06 UTC | remote storage: expose /stat/counters | 14 October 2015, 08:56:06 UTC |
634c24e | Stefano Zacchiroli | 14 October 2015, 08:48:53 UTC | swh.storage.db: remove overzelous spacing in very short functions cosmetic change | 14 October 2015, 08:48:53 UTC |
d63124a | Stefano Zacchiroli | 14 October 2015, 08:47:50 UTC | new SQL function: swh_stat_counters() returning the number of tuples in various tables expose it through the Python layer | 14 October 2015, 08:47:50 UTC |
19973fa | Antoine R. Dumont (@ardumont) | 13 October 2015, 12:52:13 UTC | SQL function: Add fixme to `ls` directory | 13 October 2015, 12:58:00 UTC |
5002bd5 | Antoine R. Dumont (@ardumont) | 13 October 2015, 12:51:25 UTC | SQL func: Add `ls -lr` on a directory | 13 October 2015, 12:58:00 UTC |
793fb06 | Stefano Zacchiroli | 13 October 2015, 12:45:02 UTC | SQL schema: add index on content(ctime) | 13 October 2015, 12:45:02 UTC |
672dd1c | Nicolas Dandrimont | 13 October 2015, 10:27:01 UTC | objstorage: create files as mode 0o644 | 13 October 2015, 10:27:01 UTC |
3bb88fe | Nicolas Dandrimont | 12 October 2015, 14:50:45 UTC | sql/swh-func: mark SELECT functions as stable This allows inlining of the language sql functions that only select data from the database. | 12 October 2015, 14:51:02 UTC |
3c6b874 | Nicolas Dandrimont | 12 October 2015, 13:53:31 UTC | sql/swh-func: update swh_revision_find_occurrence with swh_revision_list_children | 12 October 2015, 13:53:31 UTC |
2542bdf | Nicolas Dandrimont | 12 October 2015, 13:53:05 UTC | sql/swh-func: add swh_revision_list_children | 12 October 2015, 13:53:05 UTC |
cd8b425 | Nicolas Dandrimont | 12 October 2015, 13:52:37 UTC | sql/swh-schema: add various indexes for reverse queries on contents | 12 October 2015, 13:52:37 UTC |
cefe4d0 | Nicolas Dandrimont | 09 October 2015, 15:44:11 UTC | sql/upgrades/022: update type revision_log_entry | 09 October 2015, 15:44:11 UTC |
ba64d1a | Nicolas Dandrimont | 07 October 2015, 16:49:54 UTC | sql/upgrades/023: add upgrade 022->023 | 07 October 2015, 16:49:54 UTC |
3ddcc6a | Nicolas Dandrimont | 07 October 2015, 16:42:37 UTC | storage: add revision_get | 07 October 2015, 16:43:25 UTC |
a18c612 | Nicolas Dandrimont | 07 October 2015, 16:40:10 UTC | test_storage: use proper types for revisions | 07 October 2015, 16:43:25 UTC |
42f7713 | Nicolas Dandrimont | 07 October 2015, 16:39:36 UTC | db: allow decoding arrays of byteas | 07 October 2015, 16:43:25 UTC |
1350fc9 | Nicolas Dandrimont | 07 October 2015, 16:39:06 UTC | sql: add listing of revisions from tmp_revision | 07 October 2015, 16:43:25 UTC |
c4b1345 | Stefano Zacchiroli | 07 October 2015, 13:52:06 UTC | DB schema chart: include version number | 07 October 2015, 13:52:06 UTC |
dbc238f | Nicolas Dandrimont | 07 October 2015, 13:30:17 UTC | tests: fix flake8 | 07 October 2015, 13:30:17 UTC |
d2495d0 | Nicolas Dandrimont | 07 October 2015, 07:01:55 UTC | sql/upgrades/022: drop default before changing the type. | 07 October 2015, 07:01:55 UTC |
a4fb8df | Nicolas Dandrimont | 06 October 2015, 12:11:12 UTC | sql: add upgrade script for 21->22 | 06 October 2015, 12:11:12 UTC |
e80adb8 | Nicolas Dandrimont | 06 October 2015, 11:54:01 UTC | swh-schema: person (name, email) -> bytea Closes T65 in part | 06 October 2015, 12:01:40 UTC |
5e7fd18 | Stefano Zacchiroli | 05 October 2015, 20:15:18 UTC | sql/TODO: updated to reflect current status | 05 October 2015, 20:15:18 UTC |
04367e1 | Nicolas Dandrimont | 05 October 2015, 16:57:14 UTC | swh.storage.occurrence_add: parse string dates with dateutil | 05 October 2015, 16:57:14 UTC |
1f7ca7c | Nicolas Dandrimont | 05 October 2015, 16:41:37 UTC | swh.storage.storage: make occurrence_add support overlapping intervals Closes T45 | 05 October 2015, 16:47:16 UTC |
71c4ab6 | Nicolas Dandrimont | 05 October 2015, 16:40:22 UTC | swh.storage.db: plug swh_occurrence_history_add | 05 October 2015, 16:47:16 UTC |
c91dd71 | Nicolas Dandrimont | 05 October 2015, 16:39:24 UTC | swh.storage.db: proper copy escaping for datetimes, ranges | 05 October 2015, 16:47:16 UTC |
3a44ef5 | Nicolas Dandrimont | 05 October 2015, 16:45:58 UTC | sql/upgrades/021: add 020->021 upgrade script | 05 October 2015, 16:47:16 UTC |
b3aac6a | Nicolas Dandrimont | 05 October 2015, 16:38:05 UTC | swh-func: add swh_occurrence_history_add() | 05 October 2015, 16:38:05 UTC |
3c1f401 | Nicolas Dandrimont | 05 October 2015, 14:11:19 UTC | sql/upgrades/020: add upgrade script for 19->20 | 05 October 2015, 14:11:19 UTC |
103950b | Nicolas Dandrimont | 05 October 2015, 14:00:51 UTC | sql/swh-func: add table aliases to "where not exists" queries | 05 October 2015, 14:00:51 UTC |
c4bb88d | Nicolas Dandrimont | 05 October 2015, 08:45:01 UTC | swh.storage.tests: Actually check the error message | 05 October 2015, 08:45:01 UTC |
bf9253e | Nicolas Dandrimont | 05 October 2015, 08:33:42 UTC | swh.storage.tests: clean the database on teardown | 05 October 2015, 08:42:10 UTC |
fedd0ff | Stefano Zacchiroli | 05 October 2015, 08:00:30 UTC | storage.content_find: better testing | 05 October 2015, 08:00:30 UTC |
96adeaa | Stefano Zacchiroli | 04 October 2015, 18:39:46 UTC | storage.content_find: refactor argument checking also: - avoid overzeaolus assertion in test, to make it more robust - improve method doc string | 04 October 2015, 18:52:00 UTC |
f1fb64c | Stefano Zacchiroli | 03 October 2015, 12:44:03 UTC | sql/bin/db-upgrade: compute SQL upgrades using git and apgdiff | 03 October 2015, 12:44:03 UTC |
07c8f44 | Nicolas Dandrimont | 02 October 2015, 15:18:34 UTC | sql: Improve performance of swh_{directory,revision,release}_missing | 02 October 2015, 15:18:34 UTC |
08ffeb2 | Nicolas Dandrimont | 01 October 2015, 17:02:10 UTC | Remove atime, mtime, ctime from directory entries | 01 October 2015, 17:02:10 UTC |
43f5bf9 | Nicolas Dandrimont | 01 October 2015, 09:32:07 UTC | Bump swh.core dependency version | 01 October 2015, 09:32:07 UTC |
a718e9d | Stefano Zacchiroli | 30 September 2015, 14:17:10 UTC | db.py: drop raises psycopg2.internalerror it is no longer the case | 30 September 2015, 14:17:10 UTC |
64a7832 | Stefano Zacchiroli | 30 September 2015, 12:51:01 UTC | objstorage: fix predictable tempfile race when adding objects Before this change, two workers adding the same object will end up racing to write <SHA1>.tmp. That should not lead to corruption, because they are (in principle) going to write the exact same content, but it made the one of the two that arrives last fail the final rename. With this fix they will write in different <SHA1>.RANDOM_STUFF.tmp, and then rename their own file into place. Note that this change slightly increases the chances of having garbage in the object store, so we will need to look for this during GC passes. | 30 September 2015, 12:51:01 UTC |
e4feb05 | Stefano Zacchiroli | 30 September 2015, 10:31:16 UTC | upgrades/17.sql: add recent changes | 30 September 2015, 10:33:00 UTC |
25b0544 | Stefano Zacchiroli | 30 September 2015, 09:56:25 UTC | swh-func.sql: use LANGUAGE sql wherever possible Overall, we can use LANGUAGE sql only in 6 cases (vs 17 functions that remains plpgsql). This is in large part due to the fact that LANGUAGE sql stored procedures cannot non-existent tables, and most notably temporary tables. Closes T55. | 30 September 2015, 10:33:00 UTC |
cd507ba | Stefano Zacchiroli | 30 September 2015, 10:11:55 UTC | SQL: nicer API for swh_content_find_* functions the functions no longer fail with internal errors (due to "insert into ... strict"), but rather return NULL in those cases, because no match has been found. | 30 September 2015, 10:33:00 UTC |
d4131b3 | Antoine R. Dumont | 30 September 2015, 09:48:37 UTC | Update docstring | 30 September 2015, 09:48:46 UTC |
7fe460f | Stefano Zacchiroli | 30 September 2015, 09:01:43 UTC | bump SQL schema version to 17 include upgrade script from 16 | 30 September 2015, 09:04:30 UTC |
be72b19 | Stefano Zacchiroli | 30 September 2015, 08:57:00 UTC | sql-func refactoring: single dir*_entry_add function The function is parametric in the type of entries to be added (file/dir/rev). Get rid of the previous 3 (99% overlapping) functions swh_directory_{file,rev,dir}_entry_add functions. Make storage.py use the new function. | 30 September 2015, 09:04:30 UTC |
4612bc6 | Antoine R. Dumont | 30 September 2015, 08:57:31 UTC | Refactor - pep8 compliance - Remove unused import | 30 September 2015, 08:57:57 UTC |
c956386 | Antoine R. Dumont | 30 September 2015, 08:39:42 UTC | Improve output response for find occurrence api | 30 September 2015, 08:56:54 UTC |
141d392 | Antoine R. Dumont | 29 September 2015, 15:32:07 UTC | Implement content_find_occurrence up to the db | 30 September 2015, 08:43:30 UTC |
3e2fe23 | Antoine R. Dumont | 29 September 2015, 14:54:34 UTC | Refactor - db.py returns None or the content found The triplet of None should be sandboxed to db.py | 30 September 2015, 08:43:30 UTC |
91debb1 | Antoine R. Dumont | 29 September 2015, 13:52:57 UTC | Refactor - Permit reuse of the content_find api | 30 September 2015, 08:43:30 UTC |
6944ab8 | Antoine R. Dumont | 29 September 2015, 13:18:15 UTC | Refactor - Return result directory Remove the needless yield | 30 September 2015, 08:42:51 UTC |
311de1b | Antoine R. Dumont | 29 September 2015, 13:08:20 UTC | Refactor - Reuse | 30 September 2015, 08:42:51 UTC |
df4bdfa | Antoine R. Dumont | 29 September 2015, 12:16:27 UTC | Open content-find-occurrence | 30 September 2015, 08:42:51 UTC |
bdfc021 | Stefano Zacchiroli | 30 September 2015, 07:33:27 UTC | test_storage: add collision detection test | 30 September 2015, 07:40:40 UTC |
9aa18e4 | Stefano Zacchiroli | 30 September 2015, 07:25:06 UTC | test_api_client: use absolute import to allow rm __init__.py | 30 September 2015, 07:25:06 UTC |
b4e4b20 | Stefano Zacchiroli | 30 September 2015, 07:18:22 UTC | SQL swh_content_missing: build result incrementally by uniting on contents that appear to be missing when looking at a single checksum algorithm. This is semantically equivalent to the previous approach, but allows to exploit single-column indexes; which are non-redundant, smaller, and hence more likely to be used. This is the "metro ligne 6" approach. | 30 September 2015, 07:18:22 UTC |
7a33494 | Stefano Zacchiroli | 29 September 2015, 16:15:56 UTC | db_testing.py: allow to customize DB restore | 29 September 2015, 16:15:56 UTC |
f88331c | Nicolas Dandrimont | 29 September 2015, 14:51:45 UTC | Bump dependencies on swh.core to >= 0.0.6 | 29 September 2015, 14:51:45 UTC |
bfe9016 | Nicolas Dandrimont | 29 September 2015, 14:51:25 UTC | debian/control: indent with spaces | 29 September 2015, 14:51:25 UTC |
e424885 | Nicolas Dandrimont | 29 September 2015, 14:36:32 UTC | Bump dependency on swh.core | 29 September 2015, 14:41:40 UTC |
bb8cdff | Stefano Zacchiroli | 29 September 2015, 14:38:30 UTC | move db_test.py from swh-storage to swh-core | 29 September 2015, 14:39:10 UTC |
260c379 | Nicolas Dandrimont | 29 September 2015, 14:26:51 UTC | Add swh.storage.api package to setup.py | 29 September 2015, 14:26:51 UTC |
2b46e69 | Nicolas Dandrimont | 29 September 2015, 13:04:43 UTC | swh.storage.api: Serialize/deserialize exceptions This currently uses pickle which breaks the msgpack + dict "abstraction" we settled on. This should be fixed eventually. | 29 September 2015, 13:04:43 UTC |
58f821c | Nicolas Dandrimont | 29 September 2015, 11:52:19 UTC | Update api client and server to msgpack | 29 September 2015, 11:55:22 UTC |
79d6539 | Stefano Zacchiroli | 29 September 2015, 09:16:41 UTC | gitignore: ignore .eggs/ dir | 29 September 2015, 09:16:41 UTC |
b041526 | Antoine R. Dumont | 29 September 2015, 07:23:23 UTC | Refactor - Migrate content_present to content_find with unified api | 29 September 2015, 08:28:05 UTC |
1ddeff4 | Nicolas Dandrimont | 29 September 2015, 08:21:38 UTC | Move remote_storage and api to api.client and api.server Closes T44. | 29 September 2015, 08:21:38 UTC |
1cb39a2 | Nicolas Dandrimont | 28 September 2015, 15:28:42 UTC | sql/swh-schema: unix_path to bytea | 28 September 2015, 15:36:28 UTC |
f660268 | Nicolas Dandrimont | 28 September 2015, 15:14:37 UTC | sql/upgrades/015: drop function before readding it | 28 September 2015, 15:35:16 UTC |
d371898 | Stefano Zacchiroli | 28 September 2015, 14:19:37 UTC | SQL schema diagram: add skipped_content | 28 September 2015, 14:19:37 UTC |
eb7493c | Nicolas Dandrimont | 28 September 2015, 14:01:25 UTC | sql/upgrades/015: bump database version | 28 September 2015, 14:01:25 UTC |
6a4da5c | Nicolas Dandrimont | 28 September 2015, 13:58:56 UTC | sql/swh-func: fix outdated comments | 28 September 2015, 14:00:02 UTC |
13315d2 | Nicolas Dandrimont | 28 September 2015, 13:55:16 UTC | sql/sql-func: Make tests pass again | 28 September 2015, 14:00:02 UTC |
be3910e | Stefano Zacchiroli | 28 September 2015, 13:46:39 UTC | Merge branch 'merge-dir-entries' | 28 September 2015, 13:46:39 UTC |
8870ff4 | Stefano Zacchiroli | 28 September 2015, 13:42:19 UTC | swh-func.sql: add refactoring TODO item | 28 September 2015, 13:46:28 UTC |
3c17d73 | Stefano Zacchiroli | 28 September 2015, 13:40:21 UTC | swh_directory_entry_*_add: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
345293a | Stefano Zacchiroli | 27 September 2015, 17:07:55 UTC | swh_directory_walk_one: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
f4b8b40 | Stefano Zacchiroli | 27 September 2015, 16:27:55 UTC | swh_content_find_dir: port to merged dir entries | 28 September 2015, 13:46:28 UTC |
c08ee26 | Stefano Zacchiroli | 27 September 2015, 14:35:32 UTC | upgrades/015.sql: migration script for dir_list_* table merge | 28 September 2015, 13:46:28 UTC |
f068673 | Stefano Zacchiroli | 27 September 2015, 13:45:24 UTC | SQL schema: merge dir entry arrays into directory This removes tables directory_list_{file,dir,rev}, and add corresponding arrays {dir,file,rev}_entries to the directory table, which was just a set up to now. Indexes migrated accordingly. | 28 September 2015, 13:46:28 UTC |
945bdfb | Nicolas Dandrimont | 28 September 2015, 13:41:14 UTC | sql/sql-func: copy origin from tmp_skipped_content | 28 September 2015, 13:41:14 UTC |
573b8d4 | Nicolas Dandrimont | 28 September 2015, 13:33:20 UTC | swh.storage.storage: Only add missing skipped_content | 28 September 2015, 13:33:20 UTC |
e16565e | Nicolas Dandrimont | 28 September 2015, 13:32:16 UTC | swh.storage.storage: add origin to skipped_content | 28 September 2015, 13:32:16 UTC |
9188980 | Nicolas Dandrimont | 28 September 2015, 12:49:39 UTC | sql/swh-func.sql: remove useless declaration | 28 September 2015, 12:49:39 UTC |
8ff3109 | Nicolas Dandrimont | 28 September 2015, 12:49:12 UTC | swh.storage.storage: Allow storing "skipped" contents | 28 September 2015, 12:49:12 UTC |
09193a5 | Nicolas Dandrimont | 28 September 2015, 12:43:27 UTC | swh.storage.db: map stored procedures for skipped_content | 28 September 2015, 12:43:27 UTC |
9073916 | Nicolas Dandrimont | 28 September 2015, 12:42:39 UTC | sql schema: update skipped_content and add stored procedures for update | 28 September 2015, 12:42:39 UTC |
f1f62bd | Nicolas Dandrimont | 28 September 2015, 09:44:15 UTC | sql/swh-schema: add index on content(sha256) | 28 September 2015, 09:44:15 UTC |
d53dc22 | Nicolas Dandrimont | 28 September 2015, 09:43:48 UTC | sql/swh-schema: add skipped_content table | 28 September 2015, 09:43:48 UTC |
0dd1ece | Stefano Zacchiroli | 27 September 2015, 13:35:38 UTC | SQL: add swh_content_find, to lookup content by checksum(s) Intente use is content-by-content lookup (e.g., in the web app), not mass addition or check for missing contents. We already have swh_content_missing for that. | 27 September 2015, 13:35:38 UTC |
7de609d | Stefano Zacchiroli | 27 September 2015, 12:43:25 UTC | SQL: improve indentation of directory_walk_one to avoid very long lines | 27 September 2015, 12:43:25 UTC |
1ce7f21 | Stefano Zacchiroli | 25 September 2015, 07:43:56 UTC | swh_content_find_occ. query: improve doc string | 25 September 2015, 09:52:02 UTC |
c4e2e78 | Stefano Zacchiroli | 24 September 2015, 20:11:38 UTC | all-in-1 query from sha1 to occurrence information query name: swh_content_find_occurrence given the sha1 of some content, returns info about its origin (type, url) and context (branch, revision, path) | 25 September 2015, 09:52:02 UTC |
785f69b | Stefano Zacchiroli | 24 September 2015, 19:57:23 UTC | find queries: comment SQL code | 25 September 2015, 09:52:02 UTC |