573b8d4 | Nicolas Dandrimont | 28 September 2015, 13:33:20 UTC | swh.storage.storage: Only add missing skipped_content | 28 September 2015, 13:33:20 UTC |
e16565e | Nicolas Dandrimont | 28 September 2015, 13:32:16 UTC | swh.storage.storage: add origin to skipped_content | 28 September 2015, 13:32:16 UTC |
9188980 | Nicolas Dandrimont | 28 September 2015, 12:49:39 UTC | sql/swh-func.sql: remove useless declaration | 28 September 2015, 12:49:39 UTC |
8ff3109 | Nicolas Dandrimont | 28 September 2015, 12:49:12 UTC | swh.storage.storage: Allow storing "skipped" contents | 28 September 2015, 12:49:12 UTC |
09193a5 | Nicolas Dandrimont | 28 September 2015, 12:43:27 UTC | swh.storage.db: map stored procedures for skipped_content | 28 September 2015, 12:43:27 UTC |
9073916 | Nicolas Dandrimont | 28 September 2015, 12:42:39 UTC | sql schema: update skipped_content and add stored procedures for update | 28 September 2015, 12:42:39 UTC |
f1f62bd | Nicolas Dandrimont | 28 September 2015, 09:44:15 UTC | sql/swh-schema: add index on content(sha256) | 28 September 2015, 09:44:15 UTC |
d53dc22 | Nicolas Dandrimont | 28 September 2015, 09:43:48 UTC | sql/swh-schema: add skipped_content table | 28 September 2015, 09:43:48 UTC |
0dd1ece | Stefano Zacchiroli | 27 September 2015, 13:35:38 UTC | SQL: add swh_content_find, to lookup content by checksum(s) Intente use is content-by-content lookup (e.g., in the web app), not mass addition or check for missing contents. We already have swh_content_missing for that. | 27 September 2015, 13:35:38 UTC |
7de609d | Stefano Zacchiroli | 27 September 2015, 12:43:25 UTC | SQL: improve indentation of directory_walk_one to avoid very long lines | 27 September 2015, 12:43:25 UTC |
1ce7f21 | Stefano Zacchiroli | 25 September 2015, 07:43:56 UTC | swh_content_find_occ. query: improve doc string | 25 September 2015, 09:52:02 UTC |
c4e2e78 | Stefano Zacchiroli | 24 September 2015, 20:11:38 UTC | all-in-1 query from sha1 to occurrence information query name: swh_content_find_occurrence given the sha1 of some content, returns info about its origin (type, url) and context (branch, revision, path) | 25 September 2015, 09:52:02 UTC |
785f69b | Stefano Zacchiroli | 24 September 2015, 19:57:23 UTC | find queries: comment SQL code | 25 September 2015, 09:52:02 UTC |
8c5d7f5 | Stefano Zacchiroli | 24 September 2015, 19:38:23 UTC | new queries to find dir/occurrence of some content queries: - swh_content_find_directory(content_id sha1): content_dir - swh_revision_find_occurrence(revision_id sha1_git): occurrence Sample use: zack-linuxgit-arrays=> select sha1 from content offset 100000 limit 1; sha1 -------------------------------------------- \xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1 (1 row) Time: 21,751 ms zack-linuxgit-arrays=> select * from swh_content_find_directory('\xdd63ccef43879c3d7ea47c3f637ef2453f7a58a1'::sha1); directory | path --------------------------------------------+---------------------------- \x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e | drivers/md/dm-log-writes.c (1 row) Time: 729,122 ms zack-linuxgit-arrays=> select id from revision where directory = '\x24f14db9cc7aa0b106dcb09c9927dfa6712dfd1e'::sha1_git; id -------------------------------------------- \x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7 (1 row) Time: 395,754 ms zack-linuxgit-arrays=> select * from swh_revision_find_occurrence('\x0450b2d120ed9e6d4ac7a6eade0ad116f69b88f7'::sha1_git); origin | branch | revision --------+--------------------+-------------------------------------------- 1 | refs/tags/v4.1-rc3 | \x030bbdbf4c833bc69f502eae58498bc5572db736 (1 row) Time: 22,006 ms zack-linuxgit-arrays=> | 25 September 2015, 09:52:02 UTC |
372ee30 | Nicolas Dandrimont | 25 September 2015, 07:28:34 UTC | Add debian metadata | 25 September 2015, 09:12:50 UTC |
1ea9a1d | Nicolas Dandrimont | 25 September 2015, 07:28:00 UTC | Add missing file README.dev | 25 September 2015, 07:28:00 UTC |
bfb0274 | Nicolas Dandrimont | 25 September 2015, 07:27:50 UTC | Add missing dependency on swh.core | 25 September 2015, 07:27:50 UTC |
5c72222 | Nicolas Dandrimont | 24 September 2015, 13:49:52 UTC | swh.storage.storage: update docs of origin_add_one | 24 September 2015, 13:49:52 UTC |
9b78bfb | Nicolas Dandrimont | 24 September 2015, 13:47:05 UTC | swh.storage.storage: make origin_add_one idempotent | 24 September 2015, 13:48:15 UTC |
ccbc216 | Antoine R. Dumont | 24 September 2015, 13:02:19 UTC | Simplify the api content_present Only True if found, False otherwise. And raises ValueError in case of bad inputs (empty input or no sha1, sha256 keys) | 24 September 2015, 13:02:19 UTC |
f2dac75 | Antoine R. Dumont | 24 September 2015, 12:14:44 UTC | Fix: Escape is only for copy_to as psycopg2 do not escape for copy_to as it does for other queries | 24 September 2015, 12:14:44 UTC |
a8ddcd5 | Antoine R. Dumont | 24 September 2015, 12:06:34 UTC | Improve formatting of the sql query | 24 September 2015, 12:06:34 UTC |
bf19550 | Antoine R. Dumont | 24 September 2015, 12:05:14 UTC | Simplify command that triggers the storage server | 24 September 2015, 12:05:14 UTC |
e1a2df2 | Antoine R. Dumont | 23 September 2015, 09:15:00 UTC | Implement the content_present api | 24 September 2015, 11:40:11 UTC |
4348ff4 | Antoine R. Dumont | 22 September 2015, 12:39:39 UTC | Add a README.dev on how to run | 24 September 2015, 11:40:11 UTC |
0444468 | Antoine R. Dumont | 22 September 2015, 12:39:25 UTC | Open /content/present api | 24 September 2015, 11:40:11 UTC |
95e159c | Stefano Zacchiroli | 23 September 2015, 15:35:42 UTC | query swh_revision_list: add TODOs about ordering | 23 September 2015, 15:35:42 UTC |
f9621ae | Stefano Zacchiroli | 23 September 2015, 15:28:42 UTC | new SQL queries: revision rev-list & revision log | 23 September 2015, 15:28:42 UTC |
d85261c | Stefano Zacchiroli | 23 September 2015, 14:03:43 UTC | swh-func.sql: use ENUM type directory_entry_type instead of TEXT. Adapt swh_directory_walk_one accordingly | 23 September 2015, 14:03:43 UTC |
f61605e | Stefano Zacchiroli | 23 September 2015, 13:15:55 UTC | SQL diagram: add deps occurrences -> revision | 23 September 2015, 13:15:55 UTC |
d88a775 | Nicolas Dandrimont | 23 September 2015, 12:55:18 UTC | swh.storage.api: Add logging to the wsgi app | 23 September 2015, 12:55:18 UTC |
df97c2b | Nicolas Dandrimont | 23 September 2015, 12:54:54 UTC | swh.storage.api: Attempt closing temporary files | 23 September 2015, 12:54:54 UTC |
a6e88b2 | Nicolas Dandrimont | 23 September 2015, 09:55:13 UTC | remote_storage: use proper filename for metadata | 23 September 2015, 09:55:13 UTC |
4287922 | Nicolas Dandrimont | 23 September 2015, 09:51:38 UTC | remote_storage: Post contents as multipart files | 23 September 2015, 09:51:38 UTC |
6c49702 | Nicolas Dandrimont | 23 September 2015, 09:51:17 UTC | Add missing requirements | 23 September 2015, 09:51:17 UTC |
83a5963 | Nicolas Dandrimont | 23 September 2015, 08:42:30 UTC | swh.storage.api: add run_from_webserver wrapper | 23 September 2015, 08:42:30 UTC |
20f043b | Nicolas Dandrimont | 22 September 2015, 15:02:34 UTC | swh.storage.api: Properly escape arbitrary byte sequences in arguments | 22 September 2015, 15:02:34 UTC |
9185046 | Nicolas Dandrimont | 22 September 2015, 14:02:03 UTC | Add setup.py and friends | 22 September 2015, 14:02:37 UTC |
42946e1 | Stefano Zacchiroli | 22 September 2015, 10:17:00 UTC | SQL diagram: add missing edges for releases/revisions | 22 September 2015, 10:17:00 UTC |
49387fb | Stefano Zacchiroli | 22 September 2015, 10:12:33 UTC | SQL diagram: fix target of dir entry -> content arrow | 22 September 2015, 10:12:33 UTC |
51272e2 | Nicolas Dandrimont | 22 September 2015, 09:56:25 UTC | tests: add documentation for Test{,Remote}Storage | 22 September 2015, 09:57:14 UTC |
4ae8648 | Stefano Zacchiroli | 22 September 2015, 09:24:40 UTC | DB schema graph: logical placement of dir entries/lists | 22 September 2015, 09:24:40 UTC |
d9f2995 | Stefano Zacchiroli | 22 September 2015, 09:19:01 UTC | clusters.dot: document how to reference individual columns | 22 September 2015, 09:19:01 UTC |
aa2fbb9 | Stefano Zacchiroli | 22 September 2015, 09:17:58 UTC | SQL schema: add implicit FK dir_entry_dir -> dir | 22 September 2015, 09:17:58 UTC |
687f2c2 | Stefano Zacchiroli | 22 September 2015, 09:15:00 UTC | SQL schema graph: more granular implicit FK arrows in particular: make both arrow sources and arrow destinations point to specific table fields, rather than to whole tables | 22 September 2015, 09:15:03 UTC |
6b36837 | Stefano Zacchiroli | 22 September 2015, 09:06:25 UTC | SQL schema: add implicit FK on directories i.e., port the schema graph after switching to arrays for encoding directories | 22 September 2015, 09:07:05 UTC |
d9b14ab | Nicolas Dandrimont | 22 September 2015, 09:04:10 UTC | tests: make both storage tests inherit from an abstract class This prevents tests from the base class from running twice | 22 September 2015, 09:04:44 UTC |
f70bcd5 | Nicolas Dandrimont | 22 September 2015, 09:03:15 UTC | test_remote_storage: Add code to run the server automatically | 22 September 2015, 09:04:44 UTC |
c005660 | Nicolas Dandrimont | 22 September 2015, 09:01:55 UTC | swh.storage.api: add / route for server testing | 22 September 2015, 09:04:44 UTC |
19cc2d8 | Stefano Zacchiroli | 21 September 2015, 19:07:57 UTC | add requirements.txt, listing psycopg2 as dependency | 21 September 2015, 19:07:57 UTC |
caedb46 | Nicolas Dandrimont | 21 September 2015, 16:09:23 UTC | Add a remote API server and a client for remote storage | 21 September 2015, 16:10:25 UTC |
cc87c6f | Nicolas Dandrimont | 21 September 2015, 16:07:58 UTC | test_storage: don't break if the storage has no objstorage In prevision of the remote storage tests. | 21 September 2015, 16:10:25 UTC |
e07902a | Stefano Zacchiroli | 21 September 2015, 15:24:02 UTC | db.py: drop unused csv module (pep8) this makes "make check" pass again on the whole module | 21 September 2015, 15:24:02 UTC |
37e7281 | Stefano Zacchiroli | 21 September 2015, 15:22:44 UTC | db.py: use single quotes consistently (style) | 21 September 2015, 15:22:44 UTC |
16aca48 | Stefano Zacchiroli | 21 September 2015, 15:21:16 UTC | db: uniform case-ing in SQL query (cosmetic) | 21 September 2015, 15:21:16 UTC |
ba5443a | Nicolas Dandrimont | 21 September 2015, 14:02:29 UTC | sql/swh-schema.sql: add indexes on directory lists | 21 September 2015, 14:02:29 UTC |
b8aa5c4 | Nicolas Dandrimont | 18 September 2015, 13:10:35 UTC | Merge branch 'master' into directory-listing-arrays | 18 September 2015, 13:10:35 UTC |
e9f08de | Nicolas Dandrimont | 18 September 2015, 10:58:11 UTC | swh.storage.storage: allow listing missing contents per hash | 18 September 2015, 10:58:11 UTC |
662c8ee | Nicolas Dandrimont | 18 September 2015, 10:57:03 UTC | swh.storage.storage: add the contents that _are_ missing | 18 September 2015, 10:57:03 UTC |
04c1db8 | Nicolas Dandrimont | 18 September 2015, 10:42:14 UTC | swh.storage.storage: content ids are sha1, not sha1_git | 18 September 2015, 10:42:14 UTC |
8f97c01 | Nicolas Dandrimont | 18 September 2015, 10:31:25 UTC | Merge branch 'master' into directory-listing-arrays | 18 September 2015, 10:31:25 UTC |
9308a0a | Nicolas Dandrimont | 18 September 2015, 10:30:54 UTC | swh.storage.storage: skip already written content | 18 September 2015, 10:30:54 UTC |
41abf75 | Nicolas Dandrimont | 18 September 2015, 08:01:12 UTC | Merge branch 'master' into directory-listing-arrays | 18 September 2015, 08:01:12 UTC |
2ccb3a5 | Nicolas Dandrimont | 17 September 2015, 15:58:08 UTC | sql/swh-schema.sql: allow storing dangling occurrences and releases | 17 September 2015, 15:58:08 UTC |
af41d20 | Nicolas Dandrimont | 17 September 2015, 15:10:49 UTC | swh.storage.storage: implement occurrence_add | 17 September 2015, 15:10:49 UTC |
cc82413 | Nicolas Dandrimont | 17 September 2015, 14:49:32 UTC | swh.storage.storage: add origin_get and origin_add_one | 17 September 2015, 14:49:32 UTC |
0128716 | Nicolas Dandrimont | 17 September 2015, 14:37:25 UTC | swh.storage.db: Hand-roll CSV to avoid issues with None vs. empty strings | 17 September 2015, 14:37:25 UTC |
01c0a1b | Nicolas Dandrimont | 17 September 2015, 12:17:37 UTC | sql/swh-schema.sql: Bump schema version | 17 September 2015, 12:17:37 UTC |
6f5d47d | Nicolas Dandrimont | 17 September 2015, 12:15:44 UTC | swh.storage.storage: Implement release_add | 17 September 2015, 12:15:44 UTC |
3126a95 | Nicolas Dandrimont | 17 September 2015, 12:15:07 UTC | swh.storage.storage: Implement release_missing | 17 September 2015, 12:15:07 UTC |
0266474 | Nicolas Dandrimont | 17 September 2015, 12:14:19 UTC | swh.storage.storage: shortcut when there are no missing revisions | 17 September 2015, 12:14:19 UTC |
a891d5f | Nicolas Dandrimont | 17 September 2015, 12:13:23 UTC | swh.storage.db: Add more quoting to csv output | 17 September 2015, 12:13:23 UTC |
5cacce7 | Nicolas Dandrimont | 17 September 2015, 12:12:38 UTC | sql/swh-schema.sql: revisions can have the same parent several times | 17 September 2015, 12:12:38 UTC |
3c1e11e | Nicolas Dandrimont | 17 September 2015, 11:52:31 UTC | sql/swh-schema.sql: messages/comments as byteas | 17 September 2015, 11:52:31 UTC |
160f679 | Nicolas Dandrimont | 17 September 2015, 08:57:01 UTC | sql/swh-schema.sql: Add unique indexes on directory_entry for null times | 17 September 2015, 08:57:21 UTC |
ed8032b | Nicolas Dandrimont | 17 September 2015, 08:05:22 UTC | sql/swh-func,schema.sql: Directory listings as arrays | 17 September 2015, 08:05:22 UTC |
b2bcde3 | Nicolas Dandrimont | 17 September 2015, 08:02:16 UTC | swh.storage.db: Use CSV format for copy_to | 17 September 2015, 08:02:16 UTC |
3503aeb | Nicolas Dandrimont | 17 September 2015, 08:01:47 UTC | Bulk addition of releases | 17 September 2015, 08:01:47 UTC |
cc312eb | Nicolas Dandrimont | 17 September 2015, 07:59:54 UTC | Only insert distinct rows in directory_entry_* | 17 September 2015, 07:59:54 UTC |
dc9b73a | Nicolas Dandrimont | 16 September 2015, 14:50:51 UTC | sql/swh-func.sql: Add date offsets to bulk insertion of revisions | 16 September 2015, 14:50:51 UTC |
3c32d31 | Nicolas Dandrimont | 16 September 2015, 14:40:39 UTC | sql/swh-func.sql: add swh_mktemp_release | 16 September 2015, 14:40:39 UTC |
9ce0124 | Nicolas Dandrimont | 16 September 2015, 13:06:11 UTC | Add offsets to the dates that need them in the schema timestamptz doesn't store timezone information, and we need the offset information to be able to recreate git objects | 16 September 2015, 13:06:11 UTC |
6428e6b | Nicolas Dandrimont | 16 September 2015, 10:55:19 UTC | Merge branch 'tmp-directory-add' | 16 September 2015, 10:55:19 UTC |
7eca197 | Nicolas Dandrimont | 16 September 2015, 10:53:20 UTC | Implement revision_add for swh.storage.Storage | 16 September 2015, 10:53:20 UTC |
4042c3b | Nicolas Dandrimont | 16 September 2015, 10:52:42 UTC | swh.storage.storage: implement revision_missing | 16 September 2015, 10:52:42 UTC |
ec33fca | Nicolas Dandrimont | 16 September 2015, 10:51:12 UTC | swh.storage.db: update copy_to to not fail on missing keys | 16 September 2015, 10:51:12 UTC |
06c888b | Nicolas Dandrimont | 16 September 2015, 10:49:32 UTC | sql/swh-schema.sql: Remove historical comment on revision | 16 September 2015, 10:49:32 UTC |
5f404f1 | Nicolas Dandrimont | 16 September 2015, 10:48:41 UTC | sql/swh-schema.sql: add default values and indexes on person | 16 September 2015, 10:48:41 UTC |
b1d6dc1 | Nicolas Dandrimont | 16 September 2015, 10:47:56 UTC | Add a revision-missing stored procedure | 16 September 2015, 10:47:56 UTC |
07fb39c | Nicolas Dandrimont | 16 September 2015, 10:47:02 UTC | Add a revision-specific temporary table creation function | 16 September 2015, 10:47:02 UTC |
cb4f413 | Stefano Zacchiroli | 16 September 2015, 09:24:36 UTC | SQL: better rendeding of PDF version of the schema | 16 September 2015, 09:24:36 UTC |
b8ba308 | Stefano Zacchiroli | 16 September 2015, 08:50:50 UTC | sql/Makefile: properly handle pdf dep on clusters.dot | 16 September 2015, 08:50:55 UTC |
e1f846c | Stefano Zacchiroli | 15 September 2015, 16:38:13 UTC | add back needed __init__.py to non-namespace packages | 15 September 2015, 16:38:13 UTC |
78ea4c3 | Nicolas Dandrimont | 15 September 2015, 16:14:31 UTC | swh.storage.storage: only COPY missing directories | 15 September 2015, 16:14:31 UTC |
42b7a68 | Stefano Zacchiroli | 15 September 2015, 16:08:44 UTC | get rid of __init__.py | 15 September 2015, 16:08:44 UTC |
e4e2cb9 | Nicolas Dandrimont | 15 September 2015, 14:54:47 UTC | swh.storage.db/storage: Allow listing one directory's contents | 15 September 2015, 14:54:47 UTC |
098f66d | Nicolas Dandrimont | 15 September 2015, 14:53:31 UTC | swh.storage.storage: flake8 on directory_missing | 15 September 2015, 14:53:31 UTC |
030afa9 | Nicolas Dandrimont | 15 September 2015, 14:52:52 UTC | swh.storage.storage Update db_transaction to avoid side-effects | 15 September 2015, 14:52:52 UTC |
99ab9d5 | Nicolas Dandrimont | 15 September 2015, 14:51:30 UTC | swh.storage.db: Unwrap byteas directly in the Db layer | 15 September 2015, 14:51:30 UTC |
f4df49b | Nicolas Dandrimont | 15 September 2015, 14:49:58 UTC | sql/swh-func.sql: Add directory listing function | 15 September 2015, 14:49:58 UTC |