06c888b | Nicolas Dandrimont | 16 September 2015, 10:49:32 UTC | sql/swh-schema.sql: Remove historical comment on revision | 16 September 2015, 10:49:32 UTC |
5f404f1 | Nicolas Dandrimont | 16 September 2015, 10:48:41 UTC | sql/swh-schema.sql: add default values and indexes on person | 16 September 2015, 10:48:41 UTC |
b1d6dc1 | Nicolas Dandrimont | 16 September 2015, 10:47:56 UTC | Add a revision-missing stored procedure | 16 September 2015, 10:47:56 UTC |
07fb39c | Nicolas Dandrimont | 16 September 2015, 10:47:02 UTC | Add a revision-specific temporary table creation function | 16 September 2015, 10:47:02 UTC |
78ea4c3 | Nicolas Dandrimont | 15 September 2015, 16:14:31 UTC | swh.storage.storage: only COPY missing directories | 15 September 2015, 16:14:31 UTC |
e4e2cb9 | Nicolas Dandrimont | 15 September 2015, 14:54:47 UTC | swh.storage.db/storage: Allow listing one directory's contents | 15 September 2015, 14:54:47 UTC |
098f66d | Nicolas Dandrimont | 15 September 2015, 14:53:31 UTC | swh.storage.storage: flake8 on directory_missing | 15 September 2015, 14:53:31 UTC |
030afa9 | Nicolas Dandrimont | 15 September 2015, 14:52:52 UTC | swh.storage.storage Update db_transaction to avoid side-effects | 15 September 2015, 14:52:52 UTC |
99ab9d5 | Nicolas Dandrimont | 15 September 2015, 14:51:30 UTC | swh.storage.db: Unwrap byteas directly in the Db layer | 15 September 2015, 14:51:30 UTC |
f4df49b | Nicolas Dandrimont | 15 September 2015, 14:49:58 UTC | sql/swh-func.sql: Add directory listing function | 15 September 2015, 14:49:58 UTC |
8a202df | Nicolas Dandrimont | 15 September 2015, 14:49:19 UTC | sql/swh-schema.sql: remove foreign key on directory_entry_dir | 15 September 2015, 14:49:19 UTC |
f1dadf5 | Nicolas Dandrimont | 15 September 2015, 14:46:47 UTC | sql/swh-schema.sql: Add indexes on directory_entry_* | 15 September 2015, 14:49:05 UTC |
0db9be6 | Nicolas Dandrimont | 15 September 2015, 11:52:46 UTC | sql/swh-func.sql: Cleanup directory_entry-related functions | 15 September 2015, 14:46:17 UTC |
80029af | Nicolas Dandrimont | 14 September 2015, 18:47:45 UTC | Commit first implementation of directory_add | 14 September 2015, 18:47:45 UTC |
cd46701 | Stefano Zacchiroli | 14 September 2015, 17:01:40 UTC | DB testing: update doc and tests to use swh-storage-testdata | 14 September 2015, 17:01:40 UTC |
1123290 | Stefano Zacchiroli | 14 September 2015, 16:48:17 UTC | sql/Makefile: ignore dropdb errors | 14 September 2015, 16:48:17 UTC |
1c44963 | Nicolas Dandrimont | 14 September 2015, 16:39:16 UTC | swh.storage.storage: Add method for directory_missing | 14 September 2015, 16:40:13 UTC |
5e72c51 | Stefano Zacchiroli | 14 September 2015, 16:14:01 UTC | swh.storage: cosmetic fix in content_missing docstring | 14 September 2015, 16:14:01 UTC |
069c812 | Nicolas Dandrimont | 14 September 2015, 15:45:11 UTC | sql/swh-schema.sql: Add length constraints on checksum domains | 14 September 2015, 15:45:11 UTC |
603eec5 | Nicolas Dandrimont | 14 September 2015, 15:12:35 UTC | swh.storage.storage: Add content_missing method to Storage | 14 September 2015, 15:12:35 UTC |
ee0c895 | Nicolas Dandrimont | 14 September 2015, 15:08:47 UTC | swh.storage.tests: refactor storage test | 14 September 2015, 15:08:47 UTC |
f3053e1 | Nicolas Dandrimont | 14 September 2015, 15:05:59 UTC | swh.storage.storage: allow returning data from db_transaction | 14 September 2015, 15:05:59 UTC |
bf9829e | Nicolas Dandrimont | 14 September 2015, 13:48:12 UTC | Update swh.storage.db for the new hash API | 14 September 2015, 13:48:12 UTC |
900c496 | Nicolas Dandrimont | 14 September 2015, 13:43:25 UTC | Update ObjStorage API for the new hashes ObjStorage is now using bytearrays for its keys instead of hex strings | 14 September 2015, 13:43:25 UTC |
a9a8a09 | Nicolas Dandrimont | 14 September 2015, 09:23:47 UTC | Use functools.wraps on decorators functools.wraps preserves the wrapped function's metadata which makes navigation and interactive use nicer | 14 September 2015, 09:23:47 UTC |
f3f2a7f | Nicolas Dandrimont | 14 September 2015, 08:57:24 UTC | Remove origin.parent_id Nested revisions can now be handled through directory_entry_rev | 14 September 2015, 08:57:24 UTC |
4beae32 | Nicolas Dandrimont | 14 September 2015, 08:56:02 UTC | Add foreign key to directory(id) on directory_list_rev | 14 September 2015, 08:56:02 UTC |
e95ab80 | Nicolas Dandrimont | 14 September 2015, 08:49:55 UTC | Add directory_entry_rev and directory_list_rev tables Those tables are useful to let a directory point to another revision, for instance when using submodules or for tarball in tarball. | 14 September 2015, 08:49:55 UTC |
6c54578 | Nicolas Dandrimont | 14 September 2015, 08:38:41 UTC | Add documentation for all the Storage.<>_add methods | 14 September 2015, 08:38:41 UTC |
59420a8 | Nicolas Dandrimont | 14 September 2015, 08:37:59 UTC | Fix spelling errors on occurrence | 14 September 2015, 08:37:59 UTC |
741d8ac | Nicolas Dandrimont | 14 September 2015, 08:36:59 UTC | Update signature for revision_missing | 14 September 2015, 08:36:59 UTC |
6f53bdb | Stefano Zacchiroli | 13 September 2015, 13:32:24 UTC | SQL: add logical clusters to graphviz version of the DB schema | 13 September 2015, 13:32:24 UTC |
cd302ad | Stefano Zacchiroli | 13 September 2015, 13:20:25 UTC | SQL: create domain unix_path for path and path components | 13 September 2015, 13:20:25 UTC |
46f2e02 | Stefano Zacchiroli | 13 September 2015, 13:14:50 UTC | SQL: bump db version SQL: use bytea for checksum types | 13 September 2015, 13:19:45 UTC |
1439e03 | Stefano Zacchiroli | 12 September 2015, 11:30:26 UTC | storage.py: flake8 fixes (cosmetic) | 12 September 2015, 11:30:26 UTC |
48fe79a | Nicolas Dandrimont | 11 September 2015, 14:26:24 UTC | Add skeleton API methods for Storage | 11 September 2015, 14:26:52 UTC |
f5cecc9 | Nicolas Dandrimont | 11 September 2015, 14:25:20 UTC | Fix Storage.content_add reference to HASH_ALGORITHMS | 11 September 2015, 14:26:52 UTC |
60db4f3 | Nicolas Dandrimont | 11 September 2015, 14:24:56 UTC | Rename Storage.add_content to content_add | 11 September 2015, 14:26:46 UTC |
3abed1d | Antoine R. Dumont | 10 September 2015, 15:11:22 UTC | Fix typo | 10 September 2015, 15:25:11 UTC |
d824587 | Antoine R. Dumont | 10 September 2015, 15:10:54 UTC | Add committer_date on revision table | 10 September 2015, 15:25:11 UTC |
b98d564 | Stefano Zacchiroli | 09 September 2015, 15:06:30 UTC | db.Db: refactor copy_to() function to COPY to arbitrary tables as a side effect, change the API of storage.add_content to require more complete dictionaries. In particular, dictionaries passed there need to have an explicit "length" key | 09 September 2015, 15:06:32 UTC |
2916ac2 | Stefano Zacchiroli | 09 September 2015, 13:54:38 UTC | stored proc: generalize mktemp function it can create temporary tables that mimick any existing table | 09 September 2015, 13:54:38 UTC |
f9e05ff | Stefano Zacchiroli | 09 September 2015, 13:11:01 UTC | SQL schema: add default 'visible' for content.status | 09 September 2015, 13:11:01 UTC |
06628dc | Stefano Zacchiroli | 09 September 2015, 13:10:36 UTC | SQL schema: ensure content.ctime is not NULL we had a default, but that is not enough to ensure an (explicit) NULL will not be inserted | 09 September 2015, 13:10:36 UTC |
813fdf7 | Stefano Zacchiroli | 08 September 2015, 20:34:54 UTC | db.py: add @stored_procedure decorator to define in a declarative way Python end points for argument-less stored procedures | 08 September 2015, 20:34:56 UTC |
f5bf0e1 | Stefano Zacchiroli | 08 September 2015, 20:12:39 UTC | swh.storage: add and use @db_transaction decorator | 08 September 2015, 20:12:39 UTC |
d6b946e | Stefano Zacchiroli | 08 September 2015, 20:00:05 UTC | db_testing: do not try to rollback closed connections | 08 September 2015, 20:00:05 UTC |
fbc1ed5 | Stefano Zacchiroli | 08 September 2015, 19:58:30 UTC | refactor db.Db to be a proxy with storage procedure wrappers This way, the only code that actually call into remote stored procedures will be in this module. Other modules will just call into db.Db (in pure Python) and will not need to touch raw SQL. | 08 September 2015, 19:58:30 UTC |
c5449b2 | Stefano Zacchiroli | 08 September 2015, 13:17:54 UTC | db_testing: bug fix: call specific super() methods to *actually* be cooperative with other classes | 08 September 2015, 13:17:56 UTC |
30d75b5 | Stefano Zacchiroli | 08 September 2015, 11:45:48 UTC | document how to run tests that need a working DB this include adding convenience make "test-{db,nodb}" targets to Makefile.local | 08 September 2015, 11:45:50 UTC |
a7c7bdc | Stefano Zacchiroli | 08 September 2015, 11:36:44 UTC | rename swh.storage.bundle -> swh.storage.storage the apparent name duplication is not a big deal, as it is a hidden module anyhow, that is proxied in the swh.storage.Storage class | 08 September 2015, 11:36:46 UTC |
5ae50a5 | Stefano Zacchiroli | 08 September 2015, 11:34:34 UTC | test_storage.py: test add_content() | 08 September 2015, 11:34:34 UTC |
7c55543 | Stefano Zacchiroli | 08 September 2015, 11:33:55 UTC | test_objstorage.py: use assert(Not)In where possible (style) | 08 September 2015, 11:33:59 UTC |
5494a4a | Stefano Zacchiroli | 08 September 2015, 11:33:25 UTC | swh.storage(.db): allow to pass an established connection object to constructor | 08 September 2015, 11:33:25 UTC |
90b50b4 | Stefano Zacchiroli | 08 September 2015, 10:38:07 UTC | test_db.py: (trivial) tests for swh.storage.db | 08 September 2015, 10:38:07 UTC |
d01a389 | Stefano Zacchiroli | 08 September 2015, 10:37:54 UTC | db_testing.py: fixture to ease testing using the DB | 08 September 2015, 10:37:54 UTC |
e8f6b63 | Stefano Zacchiroli | 08 September 2015, 08:27:23 UTC | make swh.storage.tests a proper python module this allows to have (importable) testing helper modules in there | 08 September 2015, 08:27:23 UTC |
d6599ba | Stefano Zacchiroli | 08 September 2015, 08:26:50 UTC | sql/Makefile: add dumpdb target | 08 September 2015, 08:26:50 UTC |
e1bd7d8 | Stefano Zacchiroli | 07 September 2015, 14:48:51 UTC | bin/swh-storage-add-dir: test/bench script to inject a full dir note that this script inject into both the DB and the object storage, whereas bin/swh-objstorage-add-dir only injects into the latter | 07 September 2015, 14:48:51 UTC |
cb62421 | Stefano Zacchiroli | 07 September 2015, 14:47:54 UTC | add_content(): be resilient to duplicates in the input | 07 September 2015, 14:47:54 UTC |
ad9363e | Stefano Zacchiroli | 07 September 2015, 14:43:30 UTC | add_content(): fix the name "tmp_content" for the temp add table This simplifies client code, and also allows to use non-quoted SQL in the stored procedures. To avoid losing generality, this commit also adds "ON COMMIT DROP" to the temporary table creation, so that different transactions can coexist. | 07 September 2015, 14:43:30 UTC |
f19bdd9 | Stefano Zacchiroli | 07 September 2015, 13:57:17 UTC | SQL: add (commented) sha256 index | 07 September 2015, 13:57:46 UTC |
df7b743 | Stefano Zacchiroli | 07 September 2015, 13:56:35 UTC | swh.storage.Storage: proxy for DB+FS with add_content() support | 07 September 2015, 13:57:46 UTC |
d8b04cb | Stefano Zacchiroli | 07 September 2015, 13:56:03 UTC | db.Db: simple proxy for psycopg2 connection | 07 September 2015, 13:57:46 UTC |
8178e97 | Stefano Zacchiroli | 07 September 2015, 13:55:34 UTC | objstorage: idempotent addition, ignore dups get rid of clobber behaviour, too complex for our needs having an idempotent semantic is simpler and cleaner anyhow | 07 September 2015, 13:57:46 UTC |
6c30b8d | Stefano Zacchiroli | 07 September 2015, 13:54:37 UTC | SQL: server-side function to add content entries | 07 September 2015, 13:57:42 UTC |
a742bcb | Stefano Zacchiroli | 06 September 2015, 08:47:52 UTC | SQL: replace left-over TAB characters with spaces (cosmetic) | 06 September 2015, 08:47:52 UTC |
719157c | Stefano Zacchiroli | 06 September 2015, 08:47:05 UTC | SQL: add creation time to content table | 06 September 2015, 08:47:05 UTC |
a2939c6 | Stefano Zacchiroli | 04 September 2015, 14:20:02 UTC | SQL: move gist extension creation to *-init.sql | 04 September 2015, 14:20:02 UTC |
e686c60 | Stefano Zacchiroli | 04 September 2015, 14:16:24 UTC | SQL: reorganize *.sql files and Makefile separate SQL files into: - swh-init: DB wide changes, e.g., language loading, extensions, etc. - swh-schema: DB schema, e.g., tables, views, etc. - swh-data: DB static data, e.g., predefined rows | 04 September 2015, 14:16:24 UTC |
d7268e8 | Stefano Zacchiroli | 04 September 2015, 12:10:39 UTC | objstorage: add support for len() and __iter__() | 04 September 2015, 12:10:39 UTC |
8ab2913 | Stefano Zacchiroli | 04 September 2015, 09:50:43 UTC | objstorage: return object id upon add() note: this was already documented properly in the API doc, but not actually implemented by add() methods | 04 September 2015, 09:50:45 UTC |
006b4c8 | Stefano Zacchiroli | 04 September 2015, 07:45:09 UTC | objstorage: use __contains__ for membership test | 04 September 2015, 07:45:09 UTC |
abb162c | Stefano Zacchiroli | 03 September 2015, 15:33:03 UTC | SQL: rename "references" to "branches" in the occurrence table | 03 September 2015, 15:34:43 UTC |
441fa4d | Stefano Zacchiroli | 03 September 2015, 15:32:44 UTC | SQL: remove out-of-date TODOs | 03 September 2015, 15:32:44 UTC |
d415446 | Stefano Zacchiroli | 03 September 2015, 15:24:59 UTC | SQL: add type of revision (git, tarball, etc.) | 03 September 2015, 15:24:59 UTC |
75768b7 | Stefano Zacchiroli | 03 September 2015, 15:22:15 UTC | SQL: add parent_rank to revision_history this is needed to be able to deterministically rebuild merge commits | 03 September 2015, 15:22:15 UTC |
7bfe3b0 | Stefano Zacchiroli | 03 September 2015, 15:21:45 UTC | SQL: drop foreign key reference from revision to directories rationale: allow to store commits whose tree has not been fetched | 03 September 2015, 15:21:45 UTC |
b0a8a20 | Stefano Zacchiroli | 03 September 2015, 15:05:32 UTC | SQL: add primary key to occurrence_history | 03 September 2015, 15:05:32 UTC |
c606d62 | Stefano Zacchiroli | 03 September 2015, 15:05:17 UTC | SQL: add authority to occurence_history | 03 September 2015, 15:05:17 UTC |
a1f5164 | Stefano Zacchiroli | 03 September 2015, 14:47:40 UTC | SQL: let directory entries reference child dir deferredly | 03 September 2015, 14:47:40 UTC |
9f48e14 | Stefano Zacchiroli | 03 September 2015, 14:46:26 UTC | SQL: switch content primary key to sha1 (from sha1_git) | 03 September 2015, 14:46:26 UTC |
2102d91 | Stefano Zacchiroli | 03 September 2015, 14:31:19 UTC | SQL: rename git_object_id -> sha1_git note: keep 2 different types --- sha1, sha1_git --- as they represent different types of information | 03 September 2015, 14:31:19 UTC |
c8ced16 | Stefano Zacchiroli | 03 September 2015, 14:28:02 UTC | SQL: add status (AKA visibility) to content table | 03 September 2015, 14:28:02 UTC |
9bd8c2c | Stefano Zacchiroli | 03 September 2015, 14:15:38 UTC | SQL: new modeling of file/directories (joint work with Nicolas and Antoine) | 03 September 2015, 14:21:26 UTC |
4f9dd68 | Stefano Zacchiroli | 03 September 2015, 12:03:24 UTC | objstorage: switch docstring conventions to google style see: https://google-styleguide.googlecode.com/svn/trunk/pyguide.html?showone=Comments#Comments | 03 September 2015, 12:03:24 UTC |
b56253a | Stefano Zacchiroli | 03 September 2015, 11:51:02 UTC | objstorage: simplify exc hierarchy to have a top-level Error exc | 03 September 2015, 11:51:02 UTC |
bed8079 | Stefano Zacchiroli | 03 September 2015, 10:21:12 UTC | objstorage: switch to sha1 object ids (from sha1_git) | 03 September 2015, 10:21:17 UTC |
a25d5ab | Stefano Zacchiroli | 03 September 2015, 10:02:33 UTC | import DB schema under sql/ | 03 September 2015, 10:02:33 UTC |
3b493ba | Stefano Zacchiroli | 03 September 2015, 08:51:52 UTC | sync Makefile with py-template | 03 September 2015, 08:51:52 UTC |
c7aeeab | Stefano Zacchiroli | 03 September 2015, 08:35:29 UTC | Makefile: sync with py template | 03 September 2015, 08:35:29 UTC |
3e2b044 | Stefano Zacchiroli | 02 September 2015, 16:46:16 UTC | ObjStorage: add a high-level description of the API | 02 September 2015, 16:46:16 UTC |
798709b | Stefano Zacchiroli | 02 September 2015, 16:40:50 UTC | ObjStorage doc string: refer to ID_HASH_ALGO, instead of hard-coding sha1_git | 02 September 2015, 16:40:50 UTC |
b88abfa | Stefano Zacchiroli | 02 September 2015, 16:34:55 UTC | document in house exceptions | 02 September 2015, 16:34:55 UTC |
f224581 | Stefano Zacchiroli | 02 September 2015, 16:27:45 UTC | bin/swh-objstorage-fsck: fsck-like script for entire dirs | 02 September 2015, 16:27:45 UTC |
7e3d88b | Stefano Zacchiroli | 02 September 2015, 16:23:11 UTC | add get() method(s) | 02 September 2015, 16:23:11 UTC |
df2c667 | Stefano Zacchiroli | 02 September 2015, 16:22:57 UTC | tests: better organization for setUp values also: add an explicit missing_obj_id value | 02 September 2015, 16:22:57 UTC |
b22ba36 | Stefano Zacchiroli | 02 September 2015, 16:14:27 UTC | rename write context manager to _write_obj_file it is a more appropriate name (since it is used also when clobbering), and it also makes clear that it is not supposed to be used from the outside | 02 September 2015, 16:14:35 UTC |
cf5ff76 | Stefano Zacchiroli | 02 September 2015, 16:13:18 UTC | tests: preseed obj_id on add wherever possible | 02 September 2015, 16:13:18 UTC |
15fd4c2 | Stefano Zacchiroli | 02 September 2015, 15:51:04 UTC | ensure we can easily switch to sha1 algo instead of sha1_git, if we need to check() will be more efficient with sha1, and avoid an extra pass on the gz content to compute the length | 02 September 2015, 15:51:06 UTC |