| 2ab7f19 | Nicolas Dandrimont | 12 June 2017, 15:54:11 UTC | New upstream version 0.0.87 | 12 June 2017, 15:54:11 UTC |
| 9c8455e | Nicolas Dandrimont | 12 June 2017, 13:38:13 UTC | bump requirements for new version of scheduler | 12 June 2017, 13:38:13 UTC |
| c5be4dc | Nicolas Dandrimont | 12 June 2017, 13:35:07 UTC | tasks: update to new swh.scheduler.task api | 12 June 2017, 13:35:07 UTC |
| 34f7638 | Nicolas Dandrimont | 06 June 2017, 16:43:43 UTC | New upstream version 0.0.86 | 06 June 2017, 16:43:43 UTC |
| cb855e0 | Nicolas Dandrimont | 06 June 2017, 16:42:01 UTC | Merge tag 'v0.0.85' v0.0.85 - Improve license endpoint's unknown license policy | 06 June 2017, 16:42:01 UTC |
| 0ae9592 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 16:36:38 UTC |
| cbbce73 | Nicolas Dandrimont | 06 June 2017, 16:14:43 UTC | archiver: minimal update for new schema | 06 June 2017, 16:14:58 UTC |
| 5d0d03d | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:55:39 UTC | New upstream version 0.0.85 | 06 June 2017, 15:55:39 UTC |
| 0965a85 | Antoine R. Dumont (@ardumont) | 06 June 2017, 15:52:02 UTC | swh.storage: Clean up dead code Related T721 | 06 June 2017, 15:52:02 UTC |
| 0a7fdb0 | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:33:32 UTC | sql/upgrades: create db upgrade 106->107 Related T721 | 06 June 2017, 13:33:32 UTC |
| 80944cb | Antoine R. Dumont (@ardumont) | 06 June 2017, 13:19:24 UTC | archiver: Make package work for actual objstorage dependency version The head version of the objstorage migrated. Thus the archiver tests broke locally. The test code was fixed to make the local tests pass with the actual head version of the objstorage. But we did not deploy the new objstorage yet, thus the archiver tests break during the packaging phase (of any swh.storage.* package). This commit is a temporary fix to make everyone happy (since we do need to package other storage version for example). This needs to be reverted as soon as we bump the dependency to the new objstorage (> 0.17). Related f681f83 Related b4dfe06 Related 08827dc Related D205 | 06 June 2017, 13:29:23 UTC |
| 0cf935e | Antoine R. Dumont (@ardumont) | 06 June 2017, 12:24:48 UTC | swh.storage: Improve license endpoint's unknown license policy Prior to this commit, the content_fossology_license_add endpoint was not symmetric with other endpoints since it returned the list of licenses it was unable to deal with. Now, it simply adds those unknown licenses in the fossology_license table. And it no longer returns anything (as other endpoints). Related T721 | 06 June 2017, 13:28:32 UTC |
| f681f83 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 02 June 2017, 14:20:45 UTC |
| 3fd7a93 | Antoine R. Dumont (@ardumont) | 02 June 2017, 14:16:47 UTC | New upstream version 0.0.84 | 02 June 2017, 14:16:47 UTC |
| 08827dc | Antoine R. Dumont (@ardumont) | 02 June 2017, 14:13:28 UTC | Revert "swh.storage.archiver: Fix tests" for packaging reason This reverts commit b4dfe06f9f06bab2ce4b6f44ca9c20dec2e7f28f. | 02 June 2017, 14:16:05 UTC |
| 4bcd830 | Antoine R. Dumont (@ardumont) | 02 June 2017, 12:43:42 UTC | sql/upgrades: create db upgrade 105->106 Related T722 | 02 June 2017, 12:43:42 UTC |
| 5ff3979 | Antoine R. Dumont (@ardumont) | 01 June 2017, 14:55:59 UTC | swh.storage: Update db schema to new version Related T722 | 01 June 2017, 14:55:59 UTC |
| a95a3c4 | Antoine R. Dumont (@ardumont) | 01 June 2017, 14:29:57 UTC | Add new entry for language indexer tool Related T722 | 01 June 2017, 14:29:57 UTC |
| 8f7a5c5 | Antoine R. Dumont (@ardumont) | 01 June 2017, 13:31:03 UTC | swh.storage: fossology license endpoints: use idx_configuration_id Related T722 | 01 June 2017, 13:31:03 UTC |
| 7738a76 | Antoine R. Dumont (@ardumont) | 01 June 2017, 12:21:55 UTC | swh.storage: indexer endpoints: Fix filtering missing data issue The filtering on missing data was only using the content's sha1 and not the indexer configuration id. Nothing to fix in production though. We do not use yet the filtering nor have multiple tools for the same indexer though. Related T722 | 01 June 2017, 12:22:35 UTC |
| f18e2df | Antoine R. Dumont (@ardumont) | 01 June 2017, 12:12:08 UTC | swh.storage: ctags endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 12:12:08 UTC |
| a8ce0d9 | Antoine R. Dumont (@ardumont) | 01 June 2017, 11:22:03 UTC | swh.storage.tests: Refactor reading the indexer tools Related T722 | 01 June 2017, 11:22:03 UTC |
| f3600de | Antoine R. Dumont (@ardumont) | 01 June 2017, 10:31:50 UTC | swh.storage: language endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 10:32:22 UTC |
| 45a923b | Antoine R. Dumont (@ardumont) | 01 June 2017, 10:12:36 UTC | swh.storage: mimetype endpoints: use indexer_configuration_id Related T722 | 01 June 2017, 10:16:10 UTC |
| b4dfe06 | Antoine R. Dumont (@ardumont) | 01 June 2017, 08:50:13 UTC | swh.storage.archiver: Fix tests Related D205 | 01 June 2017, 08:55:48 UTC |
| 85fdb1c | Antoine R. Dumont (@ardumont) | 31 May 2017, 16:26:01 UTC | swh-storage: Add indexer_configuration_get endpoint | 31 May 2017, 16:26:49 UTC |
| 8e97754 | Nicolas Dandrimont | 30 May 2017, 11:08:16 UTC | sql/swh-func: use pg_stat_user_tables for more accurate tuple statistics Close T719 | 30 May 2017, 11:09:33 UTC |
| 55495ec | Antoine Pietri | 09 May 2017, 12:31:56 UTC | vault: remove from storage repository | 09 May 2017, 13:17:37 UTC |
| 8644bd4 | Antoine R. Dumont (@ardumont) | 31 March 2017, 10:27:09 UTC | New upstream version 0.0.83 | 31 March 2017, 10:27:09 UTC |
| 76ed458 | Antoine R. Dumont (@ardumont) | 27 March 2017, 13:58:02 UTC | swh.storage: Use aggregate key to filter on missing skipped contents Related T703 Closes D200 | 31 March 2017, 10:26:44 UTC |
| 0b5e938 | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:43:35 UTC | sql/upgrades: add 103 -> 104 Adapt according to review on D200. Use sql/bin/db-upgrade to generate the upgrade script. Related T703 | 30 March 2017, 12:13:00 UTC |
| abeb873 | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:21:57 UTC | Revert "swh.storage: Use upsert scheme on (skipped_)content_add function" This reverts commit 8dddc91a7860f9db04026132c874d809a581f5db. | 27 March 2017, 12:21:57 UTC |
| 8dddc91 | Antoine R. Dumont (@ardumont) | 24 March 2017, 23:36:30 UTC | swh.storage: Use upsert scheme on (skipped_)content_add function Related T703 | 27 March 2017, 12:21:23 UTC |
| ada557c | Antoine R. Dumont (@ardumont) | 27 March 2017, 12:18:16 UTC | swh.storage: Extract key variable for insertion Related T703 | 27 March 2017, 12:21:23 UTC |
| 4ff4307 | Antoine R. Dumont (@ardumont) | 24 March 2017, 23:07:51 UTC | swh.storage: Add checksum blake2s256 Related T703 | 24 March 2017, 23:07:51 UTC |
| c94ba89 | Antoine R. Dumont (@ardumont) | 24 March 2017, 11:57:18 UTC | d/control: Add python3-swh.journal dependency with version | 24 March 2017, 11:57:18 UTC |
| 3fe8924 | Antoine R. Dumont (@ardumont) | 23 March 2017, 14:48:15 UTC | swh.storage.listener: Use swh.journal.serializers.key_to_kafka function Related T529 | 23 March 2017, 14:49:40 UTC |
| 47cb71b | Antoine R. Dumont (@ardumont) | 23 March 2017, 13:47:28 UTC | swh.storage.listener: Send notify data as dict of composite primary key All data are sent as dictionary of composite primary key for each object type. This also aligns the content type data with the skipped_content type data (sending sha1, sha1_git, sha256). Related T529 | 23 March 2017, 14:06:05 UTC |
| 9be297b | Antoine Pietri | 21 March 2017, 17:43:42 UTC | vault: gitfast: add documentation, remove more useless directory explorations | 22 March 2017, 13:02:02 UTC |
| 39db44a | Antoine Pietri | 21 March 2017, 16:54:19 UTC | vault: gitfast: use a single db transaction for all the export | 22 March 2017, 13:02:02 UTC |
| 2f9860c | Antoine Pietri | 21 March 2017, 16:49:15 UTC | vault: gifast: add an lru_cache around get_dir_ents | 22 March 2017, 13:02:02 UTC |
| c0e66af | Antoine Pietri | 21 March 2017, 16:47:33 UTC | vault: gifast: add a reset command when starting a new root | 22 March 2017, 13:02:02 UTC |
| b399957 | Antoine Pietri | 21 March 2017, 14:48:54 UTC | vault: make gitfast explore only modified directories | 22 March 2017, 13:02:02 UTC |
| e50f990 | Antoine Pietri | 22 March 2017, 12:04:55 UTC | storage: fix unused cur= argument in directory_ls | 22 March 2017, 12:04:55 UTC |
| 7eabf17 | Nicolas Dandrimont | 21 March 2017, 14:47:10 UTC | db.py: fix flake8 mistake | 21 March 2017, 14:47:10 UTC |
| 0f382ad | Antoine R. Dumont (@ardumont) | 21 March 2017, 13:50:15 UTC | New upstream version 0.0.82 | 21 March 2017, 13:50:15 UTC |
| 627bde4 | Antoine R. Dumont (@ardumont) | 21 March 2017, 13:22:50 UTC | swh.storage.listener: Subscribe to new origin notifications | 21 March 2017, 13:25:55 UTC |
| 9ed7fbb | Antoine Pietri | 17 March 2017, 11:59:15 UTC | vault: pass the obj_id as a class parameter | 17 March 2017, 14:11:18 UTC |
| 90e4e54 | Antoine Pietri | 17 March 2017, 11:55:58 UTC | vault: coherent API view names | 17 March 2017, 14:11:18 UTC |
| 0f3006f | Nicolas Dandrimont | 16 March 2017, 14:57:51 UTC | sql/upgrades: create db upgrade 101->102 | 16 March 2017, 15:01:40 UTC |
| 795d681 | Nicolas Dandrimont | 16 March 2017, 14:40:10 UTC | sql/swh-func: check for equality on the three columns for swh_content_missing It turns out that PostgreSQL is less naive than the comment implies. It will know how to use the most specific index for the equality query. That should speed us up quite a bit. | 16 March 2017, 14:52:30 UTC |
| 64a740b | Nicolas Dandrimont | 16 March 2017, 14:37:29 UTC | storage: add length to directory listing primitives | 16 March 2017, 14:37:29 UTC |
| 39ec770 | Nicolas Dandrimont | 16 March 2017, 14:17:40 UTC | test_storage: make AlteringSchemaTest depend on a new base class This avoids running all tests three times... | 16 March 2017, 14:20:43 UTC |
| f349d4c | Antoine Pietri | 15 March 2017, 16:38:00 UTC | vault: revision_git: faster toposort without useless copies | 16 March 2017, 13:03:22 UTC |
| ac9a2d2 | Antoine Pietri | 15 March 2017, 16:32:32 UTC | vault: directory builder: actually default invalid permissions to 644 | 16 March 2017, 13:03:22 UTC |
| cb03a7d | Antoine R. Dumont (@ardumont) | 15 March 2017, 13:43:34 UTC | Migrate from swh.core.hashutil to swh.model.hashutil Related T700 | 15 March 2017, 15:12:31 UTC |
| 39280bc | Antoine Pietri | 09 March 2017, 12:53:10 UTC | vault: add a git fast-import cooker | 13 March 2017, 18:09:45 UTC |
| 20556dc | Antoine Pietri | 13 March 2017, 14:40:58 UTC | vault cookers: move repeated functions to base class | 13 March 2017, 15:36:35 UTC |
| 633b1d3 | Antoine R. Dumont (@ardumont) | 13 March 2017, 14:39:32 UTC | archiver_content_updater: Use standard archiver_storage instanciation Related 5cad6d37a915474e34fc252e314e0142fa47a079 | 13 March 2017, 14:41:01 UTC |
| d2d45b2 | Antoine R. Dumont (@ardumont) | 03 March 2017, 09:39:37 UTC | swh.storage.archiver.updater: Add specific configuration filename | 13 March 2017, 13:22:58 UTC |
| ff48a42 | Antoine R. Dumont (@ardumont) | 03 March 2017, 09:36:38 UTC | swh.storage.archiver.updater: Add logging level to INFO | 13 March 2017, 13:22:58 UTC |
| 1dc2069 | Antoine R. Dumont (@ardumont) | 27 February 2017, 12:12:30 UTC | swh.storage.archiver.updater: Call directly content_archive_add This function will drop any existing contents if encountered Related T494 Related T569 | 13 March 2017, 13:22:58 UTC |
| 6136af8 | Antoine R. Dumont (@ardumont) | 25 February 2017, 00:12:02 UTC | d/control: Add swh-journal dependency to swh.storage.archiver Related T494 | 13 March 2017, 13:22:58 UTC |
| 4f1d48c | Antoine R. Dumont (@ardumont) | 25 February 2017, 00:04:51 UTC | Add journal client to update content archiver with new content Related T494 | 13 March 2017, 13:22:58 UTC |
| f711c20 | Antoine Pietri | 09 March 2017, 14:42:23 UTC | vault: more explicit DirectoryBuilder variable names | 09 March 2017, 14:42:33 UTC |
| 7e15b4b | Antoine Pietri | 06 March 2017, 15:21:41 UTC | vault: move cookers into separate files | 09 March 2017, 14:26:21 UTC |
| edd4d83 | Antoine Pietri | 06 March 2017, 12:43:32 UTC | vault: generic cache to allow multiple cooker types and formats | 09 March 2017, 13:01:41 UTC |
| 30b144b | Nicolas Dandrimont | 09 March 2017, 10:15:28 UTC | New upstream version 0.0.81 | 09 March 2017, 10:15:28 UTC |
| 56366fa | Nicolas Dandrimont | 07 March 2017, 16:02:20 UTC | archiver.director: only yield plain content ids, not dicts | 07 March 2017, 16:02:20 UTC |
| 223992d | Nicolas Dandrimont | 07 March 2017, 14:44:19 UTC | archiver.worker: allow disabling the task chaining mechanism | 07 March 2017, 14:44:19 UTC |
| 858b4d1 | Nicolas Dandrimont | 07 March 2017, 14:12:35 UTC | New upstream version 0.0.80 | 07 March 2017, 14:12:35 UTC |
| 543c8a4 | Nicolas Dandrimont | 07 March 2017, 14:11:24 UTC | archiver.storage: add a stub archiver only writing data to logfiles | 07 March 2017, 14:11:24 UTC |
| 5cad6d3 | Nicolas Dandrimont | 07 March 2017, 12:32:14 UTC | archiver.storage: refactor to provide a get_archiver_storage function This will allow us to handle another storage backend for the storage of the archiver data. | 07 March 2017, 12:32:14 UTC |
| 17c31f1 | Nicolas Dandrimont | 07 March 2017, 12:28:11 UTC | test_archiver: clean up after yourself | 07 March 2017, 12:28:49 UTC |
| 96c0a21 | Antoine R. Dumont (@ardumont) | 01 March 2017, 16:19:00 UTC | storage: open content_update endpoint Permits to batch update content rows (with or without optional new columns). Limited to contents (table content only, table skipped_content is not dealt with). Related T692 Closes D185 | 03 March 2017, 09:15:07 UTC |
| eb9130c | Nicolas Dandrimont | 02 March 2017, 15:56:54 UTC | archiver.worker: only get copies from the configured object storages By default we would try to copy objects from all the archives, even those for which we didn't have a configuration. | 02 March 2017, 15:56:54 UTC |
| 9f9570a | Nicolas Dandrimont | 02 March 2017, 15:43:27 UTC | archiver.storage: remove implicit sources_missing from content_archive_add The default value for content copies is "missing", so we don't need to make it explicit. | 02 March 2017, 15:43:27 UTC |
| e1225d1 | Nicolas Dandrimont | 02 March 2017, 15:40:46 UTC | archiver.director: the source objstorage for unknown content ids is implicit | 02 March 2017, 15:40:46 UTC |
| de9a5c3 | Nicolas Dandrimont | 02 March 2017, 15:34:58 UTC | archiver.director: make the standard input reader more resilient to errors | 02 March 2017, 15:34:58 UTC |
| 4b27819 | Antoine R. Dumont (@ardumont) | 27 February 2017, 15:41:49 UTC | Refactor: Unify the content_archive_add with swh.storage.content_add Implementation wise, this uses COPY statement and drop duplicates if encountered for content_archive insertion. | 02 March 2017, 14:53:44 UTC |
| 61af747 | Antoine R. Dumont (@ardumont) | 27 February 2017, 14:26:18 UTC | archiver-storage: Improve content_archive_add function Use the same insertion pattern as swh.storage.content_add. | 02 March 2017, 14:53:31 UTC |
| ca1c529 | Antoine R. Dumont (@ardumont) | 27 February 2017, 13:50:20 UTC | Refactor: Reuse swh.scheduler.get_task function This also has the benefit to hide some celery name (which is an implementation detail from swh.scheduler). | 02 March 2017, 14:53:28 UTC |
| 269a731 | Antoine R. Dumont (@ardumont) | 27 February 2017, 13:39:46 UTC | content_archive_add: Use the right 'missing' status Related: T494 | 02 March 2017, 14:52:21 UTC |
| afb423e | Antoine R. Dumont (@ardumont) | 27 February 2017, 13:38:34 UTC | test: Remove impossible and commented test This use case cannot happen with ArchiverWithRetentionPolicyDirector: - If a row entry is referenced in the archiver db, it's present in the objstorage - And if a row entry is not referenced in the archiver db, it won't be listed as missing since it's the archiver db which is read for listing the contents we want to archive. | 02 March 2017, 14:52:21 UTC |
| 605ca00 | Antoine R. Dumont (@ardumont) | 27 February 2017, 12:12:30 UTC | Refactor: Merge common behavior in director and content updater client Related T494 Related T569 | 02 March 2017, 14:52:14 UTC |
| c38a452 | Antoine R. Dumont (@ardumont) | 25 February 2017, 00:04:51 UTC | archiver.storage: Add content_archive_content_add endpoint Related T494 | 02 March 2017, 14:50:56 UTC |
| e962cdd | Antoine Pietri | 23 February 2017, 16:37:55 UTC | RevisionVaultCooker: factor out tar creation function | 23 February 2017, 16:37:55 UTC |
| 327924d | Antoine Pietri | 22 February 2017, 15:34:54 UTC | RevisionVaultCooker: add naive flatten implementation | 22 February 2017, 15:34:54 UTC |
| 23c1138 | Antoine Pietri | 21 February 2017, 16:35:40 UTC | config: use 5002 as the default storage port | 21 February 2017, 16:35:40 UTC |
| 2feaa1d | Antoine Pietri | 17 February 2017, 15:52:33 UTC | vault: directory cooker: handle symlinks and executables | 20 February 2017, 14:21:27 UTC |
| c2ca9fc | Antoine Pietri | 16 February 2017, 15:28:38 UTC | vault cooker: fix mismatching subclasses method signatures | 16 February 2017, 16:22:08 UTC |
| 28c3eec | Nicolas Dandrimont | 16 February 2017, 12:21:14 UTC | archiver: fix brown paper bag bug for object counter | 16 February 2017, 12:23:23 UTC |
| 7e4f780 | Antoine R. Dumont (@ardumont) | 15 February 2017, 13:35:31 UTC | README.dev: Update dev documentation with updated configuration samples | 15 February 2017, 13:41:20 UTC |
| 601e1cb | Nicolas Dandrimont | 14 February 2017, 18:37:36 UTC | New upstream version 0.0.79 | 14 February 2017, 18:37:36 UTC |
| 5c41ffc | Nicolas Dandrimont | 14 February 2017, 18:37:15 UTC | d/control: remove spurious blank line | 14 February 2017, 18:37:15 UTC |
| a5fa26c | Nicolas Dandrimont | 14 February 2017, 18:25:02 UTC | debian/control: update swh-model requirement | 14 February 2017, 18:25:25 UTC |
| ec8cebf | Nicolas Dandrimont | 14 February 2017, 18:14:13 UTC | converters: normalize timestamps using swh.model To make sure corruptions such as T680 don't happen again, use the same normalization function as swh.model before inserting timestamps into our database. This makes swh.storage reject non-integer timestamp values as well. Update tests to reflect this change. | 14 February 2017, 18:17:33 UTC |
| c6abed2 | Nicolas Dandrimont | 09 February 2017, 17:40:51 UTC | sql/archiver: get the count of objects in each archive Close T672 | 09 February 2017, 17:40:51 UTC |
| 8ce681c | Nicolas Dandrimont | 09 February 2017, 17:40:24 UTC | sql/archiver: move function defs to the functions file | 09 February 2017, 17:40:24 UTC |
| 08b0802 | Antoine Pietri | 09 February 2017, 11:12:09 UTC | requirements: split internal and external requirements in two separate files | 09 February 2017, 14:09:28 UTC |
