| bbef8c3 | Sagar Vemuri | 07 December 2017, 04:43:52 UTC | Log GetCurrentTime failures during Flush and Compaction Summary: `GetCurrentTime()` is used to populate `creation_time` table property during flushes and compactions. It is safe to ignore `GetCurrentTime()` failures here but they should be logged. (Note that `creation_time` property was introduced as part of TTL-based FIFO compaction in #2480.) Tes Plan: `make check` Closes https://github.com/facebook/rocksdb/pull/3231 Differential Revision: D6501935 Pulled By: sagar0 fbshipit-source-id: 376adcf4ab801d3a43ec4453894b9a10909c8eb6 | 07 December 2017, 04:56:53 UTC |
| d51fcb2 | Sagar Vemuri | 07 December 2017, 04:40:46 UTC | Blob DB: Add db_bench options Summary: Adding more BlobDB db_bench options which are needed for benchmarking. Closes https://github.com/facebook/rocksdb/pull/3230 Differential Revision: D6500711 Pulled By: sagar0 fbshipit-source-id: 91d63122905854ef7c9148a0235568719146e6c5 | 07 December 2017, 04:44:12 UTC |
| 53a516a | Andrew Kryczka | 07 December 2017, 03:46:44 UTC | update history for recent commits Summary: I browsed through the history since 5.9 was released and found some changes worth mentioning in HISTORY.md. Closes https://github.com/facebook/rocksdb/pull/3237 Differential Revision: D6506472 Pulled By: ajkr fbshipit-source-id: 627ce9f94ca33df9f0f231a9c5ced3624b05506c | 07 December 2017, 03:56:17 UTC |
| 78d1a5e | Andrew Kryczka | 07 December 2017, 02:49:26 UTC | Preserve overlapping file endpoint invariant Summary: Fix for #2833. - In `DeleteFilesInRange`, use `GetCleanInputsWithinInterval` instead of `GetOverlappingInputs` to make sure we get a clean cut set of files to delete. - In `GetCleanInputsWithinInterval`, support nullptr as `begin_key` or `end_key`. - In `GetOverlappingInputsRangeBinarySearch`, move the assertion for non-empty range away from `ExtendFileRangeWithinInterval`, which should be allowed to return an empty range (via `end_index < begin_index`). Closes https://github.com/facebook/rocksdb/pull/2843 Differential Revision: D5772387 Pulled By: ajkr fbshipit-source-id: e554e8461823c6be82b21a9262a2da02b3957881 | 07 December 2017, 02:56:54 UTC |
| a7d3277 | Yi Wu | 07 December 2017, 01:04:32 UTC | Fix write_callback_test compile error Summary: Rename shadow variable name db_impl. Fixing #3227 Closes https://github.com/facebook/rocksdb/pull/3235 Differential Revision: D6504051 Pulled By: yiwu-arbug fbshipit-source-id: 186c9378dabb11f8d6db56f45c95cc3b029fcb88 | 07 December 2017, 01:12:27 UTC |
| 7f04af3 | Yi Wu | 07 December 2017, 00:38:49 UTC | ldb to allow db with --try_load_options and without an options file Summary: This is to fix tools/check_format_compatible.sh. The tool try to open old versions of rocksdb with the provided options file. When options file is missing (e.g. rocksdb 2.2), it should still proceed with default options. Closes https://github.com/facebook/rocksdb/pull/3232 Differential Revision: D6503955 Pulled By: yiwu-arbug fbshipit-source-id: e44cfcce7ddc7d12cf83466ed3f3fe7624aa78b8 | 07 December 2017, 00:42:26 UTC |
| b5798bd | Yi Wu | 06 December 2017, 23:56:37 UTC | Add missing recent versions to format compatible test Summary: Add recent versions for format compatible test. We should probably update the script to auto include available versions (by looking at include/rocksdb/versions.h and deduce branch names), but we can do it later. Closes https://github.com/facebook/rocksdb/pull/3233 Differential Revision: D6503631 Pulled By: yiwu-arbug fbshipit-source-id: e2b01d1ef6e784ff6ffa1bd75d741755e3c69a8c | 07 December 2017, 00:13:50 UTC |
| 20995c5 | Yi Wu | 06 December 2017, 19:54:45 UTC | Make iterator invalid on Merge error Summary: Since #1665, on merge error, iterator will be set to corrupted status, but it doesn't invalidate the iterator. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3226 Differential Revision: D6499094 Pulled By: yiwu-arbug fbshipit-source-id: 80222930f949e31f90a6feaa37ddc3529b510d2c | 06 December 2017, 19:56:39 UTC |
| 36911f5 | Maysam Yabandeh | 06 December 2017, 17:32:09 UTC | WritePrepared Txn: stress test Summary: Augment the existing MySQLStyleTransactionTest to check for more core case scenarios. The changes showed effective in revealing the bugs reported in https://github.com/facebook/rocksdb/pull/3205 and https://github.com/facebook/rocksdb/pull/3101 Closes https://github.com/facebook/rocksdb/pull/3222 Differential Revision: D6476862 Pulled By: maysamyabandeh fbshipit-source-id: 5068497702d67ffc206a58ed96f8578fbb510137 | 06 December 2017, 17:42:28 UTC |
| 1b0c58d | Flarnie Marchan | 06 December 2017, 02:41:08 UTC | Add Code of Conduct Summary: In the past Facebook didn't promote including a Code of Conduct when creating new projects, and many projects skipped this important document. Let's fix it. :) **why make this change?:** Facebook Open Source provides a Code of Conduct statement for all projects to follow, to promote a welcoming and safe open source community. Exposing the COC via a separate markdown file is a standard being promoted by Github via the Community Profile in order to meet their Open Source Guide's recommended community standards. As you can see, adding this file will improve [the rocksdb community profile](https://github.com/facebook/rocksdb/community) checklist and increase the visibility of our COC. **test plan:** Viewing it on my branch - <img width="1008" alt="screen shot 2017-12-03 at 5 05 45 pm" src="https://user-images.githubusercontent.com/1114467/33532198-66012a56-d84c-11e7-8fab-29ed410bd600.png"> <img width="1015" alt="screen shot 2017-12-03 at 5 05 59 pm" src="https://user-images.githubusercontent.com/1114467/33532199-661813d8-d84c-11e7-941e-94754dd481e5.png"> **issue:** internal task t23481323 Closes https://github.com/facebook/rocksdb/pull/3219 Reviewed By: yiwu-arbug Differential Revision: D6494234 Pulled By: flarnie fbshipit-source-id: 55b59db335cc5546f3a1c968322b9281a3dc3aaf | 06 December 2017, 02:42:35 UTC |
| a37d734 | Adam Novak | 05 December 2017, 23:05:41 UTC | Add ROCKSDB_DISABLE_* environment variables Summary: Should fix #3036. Closes https://github.com/facebook/rocksdb/pull/3042 Differential Revision: D6452921 Pulled By: sagar0 fbshipit-source-id: eaf11e43fee1f8747006530cfc0c7a358f1c2f0f | 05 December 2017, 23:12:46 UTC |
| 4634c73 | Alex Robinson | 04 December 2017, 09:53:49 UTC | Update DBOptions::IncreaseParallelism to use newer background settings Summary: The Options header file recommends using max_background_jobs rather than directly setting max_background_compactions or max_background_flushes. I've personally seen a performance problem where stalls were happening because the one background flushing thread was blocked that was fixed by this change - https://github.com/cockroachdb/cockroach/issues/19699#issuecomment-347672485 Closes https://github.com/facebook/rocksdb/pull/3208 Differential Revision: D6473178 Pulled By: ajkr fbshipit-source-id: 67c892ceb7b1909d251492640cb15a0f2262b7ed | 04 December 2017, 09:56:15 UTC |
| 57056bb | Andrew Kryczka | 02 December 2017, 02:26:15 UTC | gflags in cmake on linux Summary: We should use it if available otherwise the tools builds never work. Thanks to #3212, we can set -DGFLAGS=1 and it'll be independent of the namespace with which gflags was compiled. Closes https://github.com/facebook/rocksdb/pull/3214 Differential Revision: D6462214 Pulled By: ajkr fbshipit-source-id: db4e5f1b905322e3119554a9d01b57532c499384 | 02 December 2017, 02:28:24 UTC |
| 63f1c0a | Andrew Kryczka | 01 December 2017, 18:40:45 UTC | fix gflags namespace Summary: I started adding gflags support for cmake on linux and got frustrated that I'd need to duplicate the build_detect_platform logic, which determines namespace based on attempting compilation. We can do it differently -- use the GFLAGS_NAMESPACE macro if available, and if not, that indicates it's an old gflags version without configurable namespace so we can simply hardcode "google". Closes https://github.com/facebook/rocksdb/pull/3212 Differential Revision: D6456973 Pulled By: ajkr fbshipit-source-id: 3e6d5bde3ca00d4496a120a7caf4687399f5d656 | 01 December 2017, 18:42:05 UTC |
| 18dcf7f | Maysam Yabandeh | 01 December 2017, 07:39:56 UTC | WritePrepared Txn: PreReleaseCallback Summary: Add PreReleaseCallback to be called at the end of WriteImpl but before publishing the sequence number. The callback is used in WritePrepareTxn to i) update the commit map, ii) update the last published sequence number in the 2nd write queue. It also ensures that all the commits will go to the 2nd queue. These changes will ensure that the commit map is updated before the sequence number is published and used by reading snapshots. If we use two write queues, the snapshots will use the seq number published by the 2nd queue. If we use one write queue (the default, the snapshots will use the last seq number in the memtable, which also indicates the last published seq number. Closes https://github.com/facebook/rocksdb/pull/3205 Differential Revision: D6438959 Pulled By: maysamyabandeh fbshipit-source-id: f8b6c434e94bc5f5ab9cb696879d4c23e2577ab9 | 01 December 2017, 07:50:45 UTC |
| 3e40a5e | Nikhil Benesch | 01 December 2017, 06:37:19 UTC | add missing config checks to CMakeLists.txt Summary: Bring CMakeLists.txt back up to parity with build_detect_platform. Closes https://github.com/facebook/rocksdb/pull/3211 Differential Revision: D6452908 Pulled By: ajkr fbshipit-source-id: 93f5f336ad7eff6ecf65dec47bfaf114dd24cfb2 | 01 December 2017, 06:57:00 UTC |
| fad1405 | Andres Suarez | 30 November 2017, 23:22:41 UTC | Remove `import` use from TARGETS Summary: We're moving away from `import`. The equivalent internal construct that gets the directory from `fbcode/` is `package_name()`. This is a Skylark friendly wrapper around [`get_base_path`]. The additional whitespace change is from running `python ./buckifier/buckify_rocksdb.py`. [`get_base_path`]: https://buckbuild.com/function/get_base_path.html Closes https://github.com/facebook/rocksdb/pull/3210 Reviewed By: yiwu-arbug Differential Revision: D6451242 Pulled By: zertosh fbshipit-source-id: 445757261de0ec89d5d332c1ba9af097086326dc | 30 November 2017, 23:27:34 UTC |
| 54095d3 | Yi Wu | 30 November 2017, 18:40:39 UTC | TARGETS file not include tests in opt mode Summary: Do not build the tests in opt mode, since SyncPoint and other test code will not be included. Closes https://github.com/facebook/rocksdb/pull/3204 Differential Revision: D6431154 Pulled By: yiwu-arbug fbshipit-source-id: c404ef042c1a6f679e5c1dc57600b3d8cb52fc28 | 30 November 2017, 18:56:58 UTC |
| ffacaaa | zhangjinpeng1987 | 30 November 2017, 06:48:55 UTC | fix Seek with lower_bound Summary: When Seek a key less than `lower_bound`, should return `lower_bound`. ajkr PTAL Closes https://github.com/facebook/rocksdb/pull/3199 Differential Revision: D6421126 Pulled By: ajkr fbshipit-source-id: a06c825830573e0040630704f6bcb3f7f48626f7 | 30 November 2017, 06:56:29 UTC |
| ed3af9e | Andrew Kryczka | 29 November 2017, 01:20:47 UTC | improve ldb CLI option support Summary: - Made CLI arguments take precedence over options file when both are provided. Note some of the CLI args are not settable via options file, like `--compression_max_dict_bytes`, so it's necessary to allow both ways of providing options simultaneously. - Changed `PrepareOptionsForOpenDB` to update the proper `ColumnFamilyOptions` if one exists for the user's `--column_family_name` argument. I supported this only in the base class, `LDBCommand`, so it works for the general arguments. Will defer adding support for subcommand-specific arguments. - Made the command fail if `--try_load_options` is provided and loading options file returns NotFound. I found the previous behavior of silently continuing confusing. Closes https://github.com/facebook/rocksdb/pull/3144 Differential Revision: D6270544 Pulled By: ajkr fbshipit-source-id: 7c2eac9f9b38720523d74466fb9e78db53561367 | 29 November 2017, 01:28:58 UTC |
| c85f8cc | Andrew Kryczka | 29 November 2017, 01:20:21 UTC | convert null terminator in ascii dump Summary: The ASCII output is almost always useless to me as the first '\0' byte in the key or value causes it to stop printing. Since all characters are already surrounded by spaces, "\ 0" (how we display a backslash followed by a zero) and "\0" (how this PR displays a null terminator) are distinguishable. My assumption is the value of seeing all the bytes outweighs the value of the alignment we had before, where we always had one character followed by one space. Closes https://github.com/facebook/rocksdb/pull/3203 Differential Revision: D6428651 Pulled By: ajkr fbshipit-source-id: aafc978a51e9ea029cfe3e763e2bb0e1751b9ccf | 29 November 2017, 01:28:58 UTC |
| c1ed005 | Prashant D | 28 November 2017, 23:15:16 UTC | tools: Fix coverity issues Summary: tools/ldb_cmd.cc: ``` 310 ignore_unknown_options_ = IsFlagPresent(flags, ARG_IGNORE_UNKNOWN_OPTIONS); CID 1322798 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 5. uninit_member: Non-static class member db_ttl_ is not initialized in this constructor nor in any functions that it calls. 311} ``` Closes https://github.com/facebook/rocksdb/pull/3122 Differential Revision: D6428576 Pulled By: sagar0 fbshipit-source-id: d77f04dd201f7f1d9f59ef88a215ee7ad7b934e9 | 28 November 2017, 23:27:41 UTC |
| 81cf262 | Prashant D | 28 November 2017, 22:31:46 UTC | utilities/backupable : Fix coverity issues Summary: 1. Class BackupMeta ``` 52 : timestamp_(0), size_(0), meta_filename_(meta_filename), CID 1168103 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member sequence_number_ is not initialized in this constructor nor in any functions that it calls. 153 file_infos_(file_infos), env_(env) {} ``` 2. class BackupEngineImpl ``` 513 } 7. uninit_member: Non-static class member latest_backup_id_ is not initialized in this constructor nor in any functions that it calls. CID 1322803 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 9. uninit_member: Non-static class member latest_valid_backup_id_ is not initialized in this constructor nor in any functions that it calls. 514} ``` 3. struct BackupAfterCopyOrCreateWorkItem ``` 368 struct BackupAfterCopyOrCreateWorkItem { 369 std::future<CopyOrCreateResult> result; 1. member_decl: Class member declaration for shared. 370 bool shared; 3. member_decl: Class member declaration for needed_to_copy. 371 bool needed_to_copy; 5. member_decl: Class member declaration for backup_env. 372 Env* backup_env; 373 std::string dst_path_tmp; 374 std::string dst_path; 375 std::string dst_relative; 2. uninit_member: Non-static class member shared is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member needed_to_copy is not initialized in this constructor nor in any functions that it calls. CID 1396122 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 6. uninit_member: Non-static class member backup_env is not initialized in this constructor nor in any functions that it calls. 376 BackupAfterCopyOrCreateWorkItem() {} ``` 4. struct CopyOrCreateWorkItem ``` 318 struct CopyOrCreateWorkItem { 319 std::string src_path; 320 std::string dst_path; 321 std::string contents; 1. member_decl: Class member declaration for src_env. 322 Env* src_env; 3. member_decl: Class member declaration for dst_env. 323 Env* dst_env; 5. member_decl: Class member declaration for sync. 324 bool sync; 7. member_decl: Class member declaration for rate_limiter. 325 RateLimiter* rate_limiter; 9. member_decl: Class member declaration for size_limit. 326 uint64_t size_limit; 327 std::promise<CopyOrCreateResult> result; 328 std::function<void()> progress_callback; 329 2. uninit_member: Non-static class member src_env is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member dst_env is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member sync is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member rate_limiter is not initialized in this constructor nor in any functions that it calls. CID 1396123 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 10. uninit_member: Non-static class member size_limit is not initialized in this constructor nor in any functions that it calls. 330 CopyOrCreateWorkItem() {} ``` 5. struct RestoreAfterCopyOrCreateWorkItem ``` struct RestoreAfterCopyOrCreateWorkItem { 410 std::future<CopyOrCreateResult> result; 1. member_decl: Class member declaration for checksum_value. 411 uint32_t checksum_value; CID 1396153 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member checksum_value is not initialized in this constructor nor in any functions that it calls. 412 RestoreAfterCopyOrCreateWorkItem() {} ``` Closes https://github.com/facebook/rocksdb/pull/3131 Differential Revision: D6428556 Pulled By: sagar0 fbshipit-source-id: a86675444543eff028e3cae6942197a143a112c4 | 28 November 2017, 22:43:28 UTC |
| 75d57a5 | kapitan-k | 28 November 2017, 22:09:17 UTC | C API: Add some block based table options Summary: Closes https://github.com/facebook/rocksdb/pull/3159 Differential Revision: D6428220 Pulled By: sagar0 fbshipit-source-id: 60508d09b5281f54b907a1c40e9631fc08343131 | 28 November 2017, 22:12:44 UTC |
| b45fbc1 | Prashant D | 28 November 2017, 21:15:20 UTC | utilities: Fix coverity issues Summary: ``` utilities/persistent_cache/block_cache_tier_file.cc 64struct CacheRecordHeader { 2. uninit_member: Non-static class member magic_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member crc_ is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member key_size_ is not initialized in this constructor nor in any functions that it calls. CID 1396161 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member val_size_ is not initialized in this constructor nor in any functions that it calls. 65 CacheRecordHeader() {} 66 CacheRecordHeader(const uint32_t magic, const uint32_t key_size, 67 const uint32_t val_size) 68 : magic_(magic), crc_(0), key_size_(key_size), val_size_(val_size) {} 69 1. member_decl: Class member declaration for magic_. 70 uint32_t magic_; 3. member_decl: Class member declaration for crc_. 71 uint32_t crc_; 5. member_decl: Class member declaration for key_size_. 72 uint32_t key_size_; 7. member_decl: Class member declaration for val_size_. 73 uint32_t val_size_; 74}; utilities/simulator_cache/sim_cache.cc: 157 miss_times_(0), CID 1396124 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member stats_ is not initialized in this constructor nor in any functions that it calls. 158 hit_times_(0) {} 159 ``` Closes https://github.com/facebook/rocksdb/pull/3155 Differential Revision: D6427237 Pulled By: sagar0 fbshipit-source-id: 97e493da5fc043c5b9a3e0d33103442cffb75aad | 28 November 2017, 21:27:08 UTC |
| 7b57510 | Prashant D | 28 November 2017, 20:17:11 UTC | utilities: Fix coverity issues in blob_db and col_buf_decoder Summary: utilities/blob_db/blob_db_impl.cc 265 : bdb_options_.blob_dir; 3. uninit_member: Non-static class member env_ is not initialized in this constructor nor in any functions that it calls. 5. uninit_member: Non-static class member ttl_extractor_ is not initialized in this constructor nor in any functions that it calls. 7. uninit_member: Non-static class member open_p1_done_ is not initialized in this constructor nor in any functions that it calls. CID 1418245 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 9. uninit_member: Non-static class member debug_level_ is not initialized in this constructor nor in any functions that it calls. 266} 4. past_the_end: Function end creates an iterator. CID 1418258 (#1 of 1): Using invalid iterator (INVALIDATE_ITERATOR) 5. deref_iterator: Dereferencing iterator file_nums.end() though it is already past the end of its container. utilities/col_buf_decoder.h: nullable_(nullable), 2. uninit_member: Non-static class member remain_runs_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member run_val_ is not initialized in this constructor nor in any functions that it calls. CID 1396134 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 6. uninit_member: Non-static class member last_val_ is not initialized in this constructor nor in any functions that it calls. 46 big_endian_(big_endian) {} Closes https://github.com/facebook/rocksdb/pull/3134 Differential Revision: D6340607 Pulled By: sagar0 fbshipit-source-id: 25c52566e2ff979fe6c7abb0f40c27fc16597054 | 28 November 2017, 20:27:57 UTC |
| cf0d6aa | Tamir Duberstein | 28 November 2017, 20:13:20 UTC | CMake cross platform Java support and add JNI to travis Summary: Rewrite `java/CMakeLists.txt` to take advantage of CMake's cross platform Java support. adamretter Closes https://github.com/facebook/rocksdb/pull/2301 Differential Revision: D5070724 Pulled By: sagar0 fbshipit-source-id: 999aee9bd39da2b24a5fe493a2eb0e9af6072dc7 | 28 November 2017, 20:27:53 UTC |
| 7827935 | Yi Wu | 28 November 2017, 19:42:28 UTC | Blob DB: Add statistics Summary: Adding a list of blob db counters. Also remove WaStats() which doesn't expose the stats and can be substitute by (BLOB_DB_BYTES_WRITTEN / BLOB_DB_BLOB_FILE_BYTES_WRITTEN). Closes https://github.com/facebook/rocksdb/pull/3193 Differential Revision: D6394216 Pulled By: yiwu-arbug fbshipit-source-id: 017508c8ff3fcd7ea7403c64d0f9834b24816803 | 28 November 2017, 19:58:49 UTC |
| 3cf562b | Yi Wu | 28 November 2017, 19:40:40 UTC | Fix IOError on WAL write doesn't propagate to write group follower Summary: This is a simpler version of #3097 by removing all unrelated changes. Fixing the bug where concurrent writes may get Status::OK while it actually gets IOError on WAL write. This happens when multiple writes form a write batch group, and the leader get an IOError while writing to WAL. The leader failed to pass the error to followers in the group, and the followers end up returning Status::OK() while actually writing nothing. The bug only affect writes in a batch group. Future writes after the batch group will correctly return immediately with the IOError. Closes https://github.com/facebook/rocksdb/pull/3201 Differential Revision: D6421644 Pulled By: yiwu-arbug fbshipit-source-id: 1c2a455c5b73f6842423785eb8a9dbfbb191dc0e | 28 November 2017, 19:42:48 UTC |
| 1bdb44d | Andrew Kryczka | 28 November 2017, 19:18:42 UTC | optimize file ingestion checks for range deletion overlap Summary: Before we were checking every file in the level which was unnecessary. We can piggyback onto the code for checking point-key overlap, which already opens all the files that could possibly contain overlapping range deletions. This PR makes us check just the range deletions from those files, so no extra ones will be opened. Closes https://github.com/facebook/rocksdb/pull/3179 Differential Revision: D6358125 Pulled By: ajkr fbshipit-source-id: 00e200770fdb8f3cc6b1b2da232b755e4ba36279 | 28 November 2017, 19:27:02 UTC |
| 022c598 | Jeffrey Dallatezza | 28 November 2017, 19:15:34 UTC | Fix minor typo in comment Summary: mean -> meant Closes https://github.com/facebook/rocksdb/pull/3202 Differential Revision: D6426443 Pulled By: sagar0 fbshipit-source-id: adaf07218580ee6903986fa5686de92f43f420e1 | 28 November 2017, 19:27:02 UTC |
| 4b65cfc | Phani Shekhar Mantripragada | 28 November 2017, 18:35:17 UTC | Support for block_cache num_shards and other config via option string. Summary: Problem: Option string accepts only cache_size as parameter for block_cache which is specified as "block_cache=1M". It doesn't accept other parameters like num_shards etc. Changes : 1) ParseBlockBasedTableOption in block_based_table_factory is edited to accept cache options in the format "block_cache=<cache_size>:<num_shard_bits>:<strict_capacity_limit>:<high_pri_pool_ratio>". Options other than cache_size are optional to maintain backward compatibility. The changes are valid for block_cache_compressed as well. For example, "block_cache=1M:6:true:0.5", "block_cache=1M:6:true", "block_cache=1M:6" and "block_cache=1M" are all valid option strings. 2) Corresponding unit tests are added. Closes https://github.com/facebook/rocksdb/pull/3108 Differential Revision: D6420997 Pulled By: sagar0 fbshipit-source-id: cdea8b785688d2802907974af27225ccc1c0cd43 | 28 November 2017, 18:48:53 UTC |
| 2f09524 | Griffin Smith | 28 November 2017, 18:24:23 UTC | Expose all remaining read and write options via the C API Summary: Expose read and write options via the C API Closes https://github.com/facebook/rocksdb/pull/3185 Differential Revision: D6389658 Pulled By: sagar0 fbshipit-source-id: 1848912750329a476805b3cb2f315e7b71f61472 | 28 November 2017, 18:28:46 UTC |
| b72b3c6 | Maysam Yabandeh | 27 November 2017, 16:49:30 UTC | WritePrepared Txn: Add MultiGet to DB Summary: This patch implements MultiGet API for WritePreparedTxnDB and update the existing unit tests. Closes https://github.com/facebook/rocksdb/pull/3196 Differential Revision: D6401493 Pulled By: maysamyabandeh fbshipit-source-id: 51501a1e32645fc2da8680e77a50035f6530f2cc | 27 November 2017, 16:56:21 UTC |
| f0dde49 | Yi Wu | 24 November 2017, 19:50:42 UTC | Blob DB: Fix GC handling for inlined blob Summary: Garbage collection checks if the offset in blob index matches the offset of the blob value in the file. If it is a mismatch, the value is the current version. However it failed to check if the blob index is an inlined type, which don't even have an offset. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3194 Differential Revision: D6394270 Pulled By: yiwu-arbug fbshipit-source-id: 7c2b9d795f1116f55f4d728086980f9b6e88ea78 | 24 November 2017, 19:56:47 UTC |
| e59cb2a | Maysam Yabandeh | 22 November 2017, 21:52:56 UTC | Add seq_per_batch to WriteWithCallbackTest Summary: Augment WriteWithCallbackTest to also test when seq_per_batch is true. Closes https://github.com/facebook/rocksdb/pull/3195 Differential Revision: D6398143 Pulled By: maysamyabandeh fbshipit-source-id: 7bc4218609355ec20fed25df426a8455ec2390d3 | 22 November 2017, 21:56:44 UTC |
| 5fac472 | Zhongyi Xie | 22 November 2017, 04:06:55 UTC | make compaction_readahead_size_ thread safe Summary: this should fix the failing tsan_check Closes https://github.com/facebook/rocksdb/pull/3192 Differential Revision: D6390004 Pulled By: miasantreble fbshipit-source-id: 6cadfc6f68febb1a77b0abcdb5416570dad926a5 | 22 November 2017, 04:11:38 UTC |
| fe187e7 | Griffin Smith | 21 November 2017, 18:11:26 UTC | Add Elixir to the list of language bindings Summary: Closes https://github.com/facebook/rocksdb/pull/3183 Differential Revision: D6386140 Pulled By: maysamyabandeh fbshipit-source-id: ca71d54edd741c3b7d9676ee2bcf584a5d49bc35 | 21 November 2017, 18:13:14 UTC |
| 8954f83 | Sagar Vemuri | 21 November 2017, 07:19:46 UTC | Blob DB: db_bench flag to control BlobDB's garbage collection Summary: flag: blob_db_enable_gc, to control BlobDb's enable_garbage_collection. Closes https://github.com/facebook/rocksdb/pull/3190 Differential Revision: D6383395 Pulled By: sagar0 fbshipit-source-id: 4134e835150748c425b8187264273a54c6d8381c | 21 November 2017, 07:26:15 UTC |
| d394a6b | anand1976 | 21 November 2017, 05:12:55 UTC | Add a ticker stat for number of keys skipped during iteration Summary: This diff adds a new ticker stat, NUMBER_ITER_SKIP, to count the number of internal keys skipped during iteration. Keys can be skipped due to deletes, or lower sequence number, or higher sequence number than the one requested. Also, fix the issue when StatisticsData is naturally aligned on cacheline boundary, padding becomes a zero size array, which the Windows compiler doesn't like. So add a cacheline worth of padding in that case to keep it happy. We cannot conditionally add padding as gcc doesn't allow using sizeof in preprocessor directives. Closes https://github.com/facebook/rocksdb/pull/3177 Differential Revision: D6353897 Pulled By: anand1976 fbshipit-source-id: 441d5a09af9c4e22e7355242dfc0c7b27aa0a6c2 | 21 November 2017, 05:26:37 UTC |
| 578f36e | Sagar Vemuri | 21 November 2017, 05:06:30 UTC | Blob DB: Remove some redundant log lines Summary: Saw some redundant log lines when trying to benchmark blob db. So, removed the lines from blob_file.cc, and let the lines in blob_db_impl.cc take the lead. Closes https://github.com/facebook/rocksdb/pull/3189 Differential Revision: D6381726 Pulled By: sagar0 fbshipit-source-id: 5f0b1e56fe4bc3b715d89ea9b5749bd935cd0606 | 21 November 2017, 05:11:34 UTC |
| a6c6b8b | Zhongyi Xie | 21 November 2017, 00:31:04 UTC | Revert "No need for Restart Interval for meta blocks" Summary: See [issue 3169](https://github.com/facebook/rocksdb/issues/3169) for more information This reverts commit 593d3de37171d99a761ce2ab34ffa12654acd055. Closes https://github.com/facebook/rocksdb/pull/3188 Differential Revision: D6379271 Pulled By: miasantreble fbshipit-source-id: 88f9ed67ba52237ad9b6f7251db83672b62d7537 | 21 November 2017, 00:42:03 UTC |
| 30285ee | Maysam Yabandeh | 20 November 2017, 21:16:04 UTC | Fix calculating filter partition target size Summary: block_size_deviation is in percentage while the partition size is in bytes. The current code fails to take that into account resulting into very large target size for filter partitions. Closes https://github.com/facebook/rocksdb/pull/3187 Differential Revision: D6376069 Pulled By: maysamyabandeh fbshipit-source-id: 276546fc68f50e0da32c462abb46f6cf676db9b2 | 20 November 2017, 21:26:57 UTC |
| 0996e14 | Giorgio Azzinnaro | 19 November 2017, 17:59:38 UTC | Added ProfaneDB Summary: Added my project [ProfaneDB](https://profanedb.gitlab.io/) which uses RocksDB as the main storage engine Closes https://github.com/facebook/rocksdb/pull/3182 Differential Revision: D6370847 Pulled By: ajkr fbshipit-source-id: ff4bb6bdbc6e42fdb88bd793a84a0e04c129b6ae | 19 November 2017, 18:11:44 UTC |
| 0213990 | Maysam Yabandeh | 18 November 2017, 01:02:13 UTC | Move static variables out of the header file Summary: Static variables in header files will be instantiated in every file that includes the header file. This patch moves some of them from options_helper.h to its .cc files. It also moves the static variable out of the offset_of since the template function could also lead to multiple instantiation perhaps due to inlining. Fixes https://github.com/facebook/rocksdb/issues/3176 Closes https://github.com/facebook/rocksdb/pull/3178 Differential Revision: D6363794 Pulled By: maysamyabandeh fbshipit-source-id: d0a07f061b4d992ab4e0de2706e622131d258fdd | 18 November 2017, 01:12:27 UTC |
| 2d04ed6 | Gustav Davidsson | 17 November 2017, 19:56:41 UTC | Make trash-to-DB size ratio limit configurable Summary: Allow users to configure the trash-to-DB size ratio limit, so that ratelimits for deletes can be enforced even when larger portions of the database are being deleted. Closes https://github.com/facebook/rocksdb/pull/3158 Differential Revision: D6304897 Pulled By: gdavidsson fbshipit-source-id: a28dd13059ebab7d4171b953ed91ce383a84d6b3 | 17 November 2017, 19:58:17 UTC |
| 32e31d4 | Zhongyi Xie | 17 November 2017, 01:46:43 UTC | Make DBOption compaction_readahead_size dynamic Summary: Closes https://github.com/facebook/rocksdb/pull/3004 Differential Revision: D6056141 Pulled By: miasantreble fbshipit-source-id: 56df1630f464fd56b07d25d38161f699e0528b7f | 17 November 2017, 01:57:25 UTC |
| e27f60b | Andrew Kryczka | 16 November 2017, 03:30:35 UTC | distinguish kZSTDNotFinalCompression in compression string Summary: This confused some users who were getting compression type from the logs. Closes https://github.com/facebook/rocksdb/pull/3153 Differential Revision: D6294964 Pulled By: ajkr fbshipit-source-id: 3c813376d33682dc6ccafc9a78df1a2e2528985e | 16 November 2017, 03:41:59 UTC |
| dd49f89 | Yi Wu | 15 November 2017, 22:13:39 UTC | Fix TARGETS lint warnings. Summary: Fix buckifier script and regenerate TARGETS file with no lint warnings. Closes https://github.com/facebook/rocksdb/pull/3170 Differential Revision: D6328993 Pulled By: yiwu-arbug fbshipit-source-id: 17d0e4ed92f676f35fed76659386611cc72b00b2 | 15 November 2017, 22:28:34 UTC |
| bbcd3b0 | Yi Wu | 15 November 2017, 22:13:03 UTC | Suppress valgrind "unimplemented functionality" error Summary: Add ROCKSDB_VALGRIND_RUN macro and suppress false-positive "unimplemented functionality" throw by valgrind for steam hints. Another approach would be add a valgrind suppress file. Valgrind is suppose to print the suppression when given "--gen-suppressions=all" param, which is suppose to be the content for the suppression file. But it doesn't print. Closes https://github.com/facebook/rocksdb/pull/3174 Differential Revision: D6338786 Pulled By: yiwu-arbug fbshipit-source-id: 3559efa5f3b92d40d09ad6ac82bc7b59f86c75aa | 15 November 2017, 22:28:34 UTC |
| 54b4356 | Maysam Yabandeh | 15 November 2017, 16:22:54 UTC | WritePrepared Txn: Refactoring WriteCallback Summary: Refactor the logic around WriteCallback in the write path to clarify when and how exactly we advance the sequence number and making sure it is consistent across the code. Closes https://github.com/facebook/rocksdb/pull/3168 Differential Revision: D6324312 Pulled By: maysamyabandeh fbshipit-source-id: 9a34f479561fdb2a5d01ef6d37a28908d03bbe33 | 15 November 2017, 16:27:06 UTC |
| 53863b7 | Maysam Yabandeh | 15 November 2017, 16:19:57 UTC | WritePrepared Txn: fix bug with Rollback seq Summary: The sequence number was not properly advanced after a rollback marker. The patch extends the existing unit tests to detect the bug and also fixes it. Closes https://github.com/facebook/rocksdb/pull/3157 Differential Revision: D6304291 Pulled By: maysamyabandeh fbshipit-source-id: 1b519c44a5371b802da49c9e32bd00087a8da401 | 15 November 2017, 16:27:06 UTC |
| 175d5d6 | Maysam Yabandeh | 14 November 2017, 16:42:14 UTC | Properly destruct rebuilding_trx_ Summary: When testing rebuilding_trx_ in MemTableInserter might still be set before the tests finishes which would cause ASAN alarms for leaks. This patch deletes the pointers in MemTableInserter destructor. Closes https://github.com/facebook/rocksdb/pull/3162 Differential Revision: D6317113 Pulled By: maysamyabandeh fbshipit-source-id: a68be70709a4fff7ac2b768660119311968f9c21 | 14 November 2017, 16:56:50 UTC |
| 9871ea4 | Yi Wu | 14 November 2017, 05:14:21 UTC | Regression test build binaries with PORTABLE=1 Summary: We hit "Illegal instruction" error in regression test with "shlx" instruction. Setting PORTABLE=1 to resolve it. Closes https://github.com/facebook/rocksdb/pull/3165 Differential Revision: D6321972 Pulled By: yiwu-arbug fbshipit-source-id: cc9fe0dbd4698d1b66a750a0b062f66899862719 | 14 November 2017, 05:26:24 UTC |
| 42564ad | Yi Wu | 14 November 2017, 02:03:57 UTC | Blob DB: not using PinnableSlice move assignment Summary: The current implementation of PinnableSlice move assignment have an issue #3163. We are moving away from it instead of try to get the move assignment right, since it is too tricky. Closes https://github.com/facebook/rocksdb/pull/3164 Differential Revision: D6319201 Pulled By: yiwu-arbug fbshipit-source-id: 8f3279021f3710da4a4caa14fd238ed2df902c48 | 14 November 2017, 02:12:20 UTC |
| 2515266 | Maysam Yabandeh | 11 November 2017, 21:08:22 UTC | WritePrepared Txn: Refactoring TrackKeys Summary: This patch clarifies and refactors the logic around tracked keys in transactions. Closes https://github.com/facebook/rocksdb/pull/3140 Differential Revision: D6290258 Pulled By: maysamyabandeh fbshipit-source-id: 03b50646264cbcc550813c060b180fc7451a55c1 | 11 November 2017, 21:14:20 UTC |
| 2edc92b | Maysam Yabandeh | 11 November 2017, 19:23:43 UTC | WritePrepared Txn: cross-compatibility test Summary: Add tests to ensure that WritePrepared and WriteCommitted policies are cross compatible when the db WAL is empty. This is important when the admin want to switch between the policies. In such case, before the switch the admin needs to empty the WAL by i) committing/rollbacking all the pending transactions, ii) FlushMemTables Closes https://github.com/facebook/rocksdb/pull/3118 Differential Revision: D6227247 Pulled By: maysamyabandeh fbshipit-source-id: bcde3d92c1e89cda3b9cfa69f6a20af5d8993db7 | 11 November 2017, 19:28:37 UTC |
| 857adf3 | Maysam Yabandeh | 11 November 2017, 01:18:01 UTC | WritePrepared Txn: Refactor conf params Summary: Summary of changes: - Move seq_per_batch out of Options - Rename concurrent_prepare to two_write_queues - Add allocate_seq_only_for_data_ Closes https://github.com/facebook/rocksdb/pull/3136 Differential Revision: D6304458 Pulled By: maysamyabandeh fbshipit-source-id: 08e685bfa82bbc41b5b1c5eb7040a8ca6e05e58c | 11 November 2017, 01:28:12 UTC |
| 07c2738 | Andrew Kryczka | 11 November 2017, 00:53:02 UTC | prefer enabling cpu features via -march/-mcpu Summary: If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2. Closes https://github.com/facebook/rocksdb/pull/3156 Differential Revision: D6304703 Pulled By: ajkr fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef | 11 November 2017, 00:57:11 UTC |
| f8e2db0 | Dmitri Smirnov | 10 November 2017, 18:27:35 UTC | Fix crashes, address test issues and adjust windows test script Summary: Add per-exe execution capability Add fix parsing of groups/tests Add timer test exclusion Fix unit tests Ifdef threadpool specific tests that do not pass on Vista threadpool. Remove spurious outout from prefix_test so test case listing works properly. Fix not using standard test directories results in file creation errors in sst_dump_test. BlobDb fixes: In C++ end() iterators can not be dereferenced. They are not valid. When deleting blob_db_ set it to nullptr before any other code executes. Not fixed:. On Windows you can not delete a file while it is open. [ RUN ] BlobDBTest.ReadWhileGC d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied write_batch Should not call front() if there is a chance the container is empty Closes https://github.com/facebook/rocksdb/pull/3152 Differential Revision: D6293274 Pulled By: sagar0 fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956 | 10 November 2017, 18:41:57 UTC |
| eefd75a | Shaohua Li | 10 November 2017, 17:25:26 UTC | Stream Summary: Add a simple policy for NVMe write time life hint Closes https://github.com/facebook/rocksdb/pull/3095 Differential Revision: D6298030 Pulled By: shligit fbshipit-source-id: 9a72a42e32e92193af11599eb71f0cf77448e24d | 10 November 2017, 17:26:24 UTC |
| f1c5eab | kapitan-k | 10 November 2017, 02:06:50 UTC | updated c ingestexternalfileoptions for ingest behind Summary: Closes https://github.com/facebook/rocksdb/pull/3151 Differential Revision: D6293861 Pulled By: ajkr fbshipit-source-id: f8db0a71509d1cd8237f2d377bf9e1bb0464bdbf | 10 November 2017, 02:15:09 UTC |
| 93f69cb | Andrew Kryczka | 10 November 2017, 01:33:01 UTC | use bottommost compression when base level is bottommost Summary: The previous compression type selection caused unexpected behavior when the base level was also the bottommost level. The following sequence of events could happen: - full compaction generates files with `bottommost_compression` type - now base level is bottommost level since all files are in the same level - any compaction causes files to be rewritten `compression_per_level` type since bottommost compression didn't apply to base level I changed the code to make bottommost compression apply to base level. Closes https://github.com/facebook/rocksdb/pull/3141 Differential Revision: D6264614 Pulled By: ajkr fbshipit-source-id: d7aaa8675126896684154a1f2c9034d6214fde82 | 10 November 2017, 01:42:00 UTC |
| 5e9e5a4 | Yi Wu | 09 November 2017, 03:33:12 UTC | Blob DB: Fix race condition between flush and write Summary: A race condition will happen when: * a user thread writes a value, but it hits the write stop condition because there are too many un-flushed memtables, while holding blob_db_impl.write_mutex_. * Flush is triggered and call flush begin listener and try to acquire blob_db_impl.write_mutex_. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3149 Differential Revision: D6279805 Pulled By: yiwu-arbug fbshipit-source-id: 0e3c58afb78795ebe3360a2c69e05651e3908c40 | 09 November 2017, 03:42:22 UTC |
| ca75f0a | Yi Wu | 08 November 2017, 21:08:15 UTC | Blob DB: Fix release build Summary: `compression` shadow the method name in `BlobFile`. Rename it. Closes https://github.com/facebook/rocksdb/pull/3148 Differential Revision: D6274498 Pulled By: yiwu-arbug fbshipit-source-id: 7d293596530998b23b6b8a8940f983f9b6343a98 | 08 November 2017, 21:14:20 UTC |
| 4f9f124 | Yi Wu | 08 November 2017, 01:40:44 UTC | Blob DB: use compression in file header instead of global options Summary: To fix the issue of failing to decompress existing value after reopen DB with a different compression settings. Closes https://github.com/facebook/rocksdb/pull/3142 Differential Revision: D6267260 Pulled By: yiwu-arbug fbshipit-source-id: c7cf7f3e33b0cd25520abf4771cdf9180cc02a5f | 08 November 2017, 01:42:17 UTC |
| 114896c | Andrew Kryczka | 07 November 2017, 21:49:35 UTC | db_bench compression options Summary: - moved existing compression options to `InitializeOptionsGeneral` since they cannot be set through options file - added flag for `zstd_max_train_bytes` which was recently introduced by #3057 Closes https://github.com/facebook/rocksdb/pull/3128 Differential Revision: D6240460 Pulled By: ajkr fbshipit-source-id: 27dbebd86a55de237ba6a45cc79cff9214e82ebc | 07 November 2017, 22:00:03 UTC |
| 65c95d9 | Andrew Kryczka | 07 November 2017, 18:42:47 UTC | support db_bench compact benchmark on bottommost files Summary: Without this option, running the compact benchmark on a DB containing only bottommost files simply returned immediately. Closes https://github.com/facebook/rocksdb/pull/3138 Differential Revision: D6256660 Pulled By: ajkr fbshipit-source-id: e3b64543acd503d821066f4200daa201d4fb3a9d | 07 November 2017, 18:57:24 UTC |
| e03377c | Manuel Ung | 06 November 2017, 18:52:17 UTC | Add lock wait time as a perf context counter Summary: Adds two new counters: `key_lock_wait_count` counts how many times a lock was blocked by another transaction and had to wait, instead of being granted the lock immediately. `key_lock_wait_time` counts the time spent acquiring locks. Closes https://github.com/facebook/rocksdb/pull/3107 Differential Revision: D6217332 Pulled By: lth fbshipit-source-id: 55d4f46da5550c333e523263422fd61d6a46deb9 | 06 November 2017, 18:57:19 UTC |
| be410de | Yi Wu | 04 November 2017, 01:10:36 UTC | Fix PinnableSlice move assignment Summary: After move assignment, we need to re-initialized the moved PinnableSlice. Also update blob_db_impl.cc to not reuse the moved PinnableSlice since it is supposed to be in an undefined state after move. Closes https://github.com/facebook/rocksdb/pull/3127 Differential Revision: D6238585 Pulled By: yiwu-arbug fbshipit-source-id: bd99f2e37406c4f7de160c7dee6a2e8126bc224e | 04 November 2017, 01:13:21 UTC |
| a6d8e30 | Sagar Vemuri | 03 November 2017, 21:34:10 UTC | Remove unnecessary status check in TableCache::NewIterator Summary: While investigating the usage of `new_table_iterator_nanos` perf counter, I saw some code was wrapper around with unnecessary status check ... so removed it. Closes https://github.com/facebook/rocksdb/pull/3120 Differential Revision: D6229181 Pulled By: sagar0 fbshipit-source-id: f8a44fe67f5a05df94553fdb233b21e54e88cc34 | 03 November 2017, 21:42:08 UTC |
| 4c8f336 | Prashant D | 03 November 2017, 21:32:00 UTC | util: Fix coverity issues Summary: util/concurrent_arena.h: CID 1396145 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member free_begin_ is not initialized in this constructor nor in any functions that it calls. 94 Shard() : allocated_and_unused_(0) {} util/dynamic_bloom.cc: 1. Condition hash_func == NULL, taking true branch. CID 1322821 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 3. uninit_member: Non-static class member data_ is not initialized in this constructor nor in any functions that it calls. 47 hash_func_(hash_func == nullptr ? &BloomHash : hash_func) {} 48 util/file_reader_writer.h: 204 private: 205 AlignedBuffer buffer_; member_not_init_in_gen_ctor: The compiler-generated constructor for this class does not initialize buffer_offset_. 206 uint64_t buffer_offset_; CID 1418246 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) member_not_init_in_gen_ctor: The compiler-generated constructor for this class does not initialize buffer_len_. 207 size_t buffer_len_; 208}; util/thread_local.cc: 341#endif CID 1322795 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 3. uninit_member: Non-static class member pthread_key_ is not initialized in this constructor nor in any functions that it calls. 342} 40struct ThreadData { 2. uninit_member: Non-static class member next is not initialized in this constructor nor in any functions that it calls. CID 1400668 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 4. uninit_member: Non-static class member prev is not initialized in this constructor nor in any functions that it calls. 41 explicit ThreadData(ThreadLocalPtr::StaticMeta* _inst) : entries(), inst(_inst) {} 42 std::vector<Entry> entries; 1. member_decl: Class member declaration for next. 43 ThreadData* next; 3. member_decl: Class member declaration for prev. 44 ThreadData* prev; 45 ThreadLocalPtr::StaticMeta* inst; 46}; Closes https://github.com/facebook/rocksdb/pull/3123 Differential Revision: D6233566 Pulled By: sagar0 fbshipit-source-id: aa2068790ea69787a0035c0db39d59b0c25108db | 03 November 2017, 21:42:08 UTC |
| cfb120f | Andrew Kryczka | 03 November 2017, 18:48:12 UTC | fix CopyFile status checks Summary: copied from internal diff D6156261 Closes https://github.com/facebook/rocksdb/pull/3124 Differential Revision: D6230167 Pulled By: ajkr fbshipit-source-id: 17926bb1152d607556364e3aacfec0ef3c115748 | 03 November 2017, 18:57:10 UTC |
| d956169 | Yi Wu | 03 November 2017, 18:15:26 UTC | Fix clang build error Summary: Fix cast from size_t to unsigned int. Closes https://github.com/facebook/rocksdb/pull/3125 Differential Revision: D6232863 Pulled By: yiwu-arbug fbshipit-source-id: 4c6131168b1faec26f7820b2cf4a09c242d323b7 | 03 November 2017, 18:26:54 UTC |
| 2581c0a | Yi Wu | 03 November 2017, 17:22:50 UTC | Blob DB: Fix BlobDBTest::SnapshotAndGarbageCollection asan failure Summary: Fix unreleased snapshot at the end of the test. Closes https://github.com/facebook/rocksdb/pull/3126 Differential Revision: D6232867 Pulled By: yiwu-arbug fbshipit-source-id: 651ca3144fc573ea2ab0ab20f0a752fb4a101d26 | 03 November 2017, 17:26:59 UTC |
| 24ad430 | Andrew Kryczka | 03 November 2017, 05:46:13 UTC | pass key/value samples through zstd compression dictionary generator Summary: Instead of using samples directly, we now support passing the samples through zstd's dictionary generator when `CompressionOptions::zstd_max_train_bytes` is set to nonzero. If set to zero, we will use the samples directly as the dictionary -- same as before. Note this is the first step of #2987, extracted into a separate PR per reviewer request. Closes https://github.com/facebook/rocksdb/pull/3057 Differential Revision: D6116891 Pulled By: ajkr fbshipit-source-id: 70ab13cc4c734fa02e554180eed0618b75255497 | 03 November 2017, 05:56:36 UTC |
| c4c1f96 | Andrew Kryczka | 03 November 2017, 05:16:23 UTC | dynamically change current memtable size Summary: Previously setting `write_buffer_size` with `SetOptions` would only apply to new memtables. An internal user wanted it to take effect immediately, instead of at an arbitrary future point, to prevent OOM. This PR makes the memtable's size mutable, and makes `SetOptions()` mutate it. There is one case when we preserve the old behavior, which is when memtable prefix bloom filter is enabled and the user is increasing the memtable's capacity. That's because the prefix bloom filter's size is fixed and wouldn't work as well on a larger memtable. Closes https://github.com/facebook/rocksdb/pull/3119 Differential Revision: D6228304 Pulled By: ajkr fbshipit-source-id: e44bd9d10a5f8c9d8c464bf7436070bb3eafdfc9 | 03 November 2017, 05:28:10 UTC |
| 30e4e01 | Zhongyi Xie | 03 November 2017, 05:12:17 UTC | add missing else Summary: Closes https://github.com/facebook/rocksdb/pull/3121 Differential Revision: D6229415 Pulled By: miasantreble fbshipit-source-id: 57c7ad2fddf5dd6b8d7e3aaf6f62348151327dfb | 03 November 2017, 05:28:06 UTC |
| 602fe94 | Prashant D | 03 November 2017, 00:43:04 UTC | Fix coverity issues in include/rocksdb Summary: include/rocksdb/metadata.h: struct ColumnFamilyMetaData { CID 1322804 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member file_count is not initialized in this constructor nor in any functions that it calls. struct SstFileMetaData { 2. uninit_member: Non-static class member size is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member smallest_seqno is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member largest_seqno is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member num_reads_sampled is not initialized in this constructor nor in any functions that it calls. CID 1322807 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 10. uninit_member: Non-static class member being_compacted is not initialized in this constructor nor in any functions that it calls. include/rocksdb/sst_file_writer.h: struct ExternalSstFileInfo { 2. uninit_member: Non-static class member sequence_number is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member file_size is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member num_entries is not initialized in this constructor nor in any functions that it calls. CID 1351697 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member version is not initialized in this constructor nor in any functions that it calls. 31 ExternalSstFileInfo() {} include/rocksdb/utilities/transaction.h: explicit Transaction(const TransactionDB* db) {} 2. uninit_member: Non-static class member log_number_ is not initialized in this constructor nor in any functions that it calls. CID 1396133 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member field txn_state_._M_i is not initialized in this constructor nor in any functions that it calls. 473 Transaction() {} Closes https://github.com/facebook/rocksdb/pull/3100 Differential Revision: D6227651 Pulled By: sagar0 fbshipit-source-id: 5caa4a2cf9471d1f9c3c073f81473636e1f0aa14 | 03 November 2017, 00:56:48 UTC |
| 62578d8 | Yi Wu | 03 November 2017, 00:26:46 UTC | Blob DB: Add compaction filter to remove expired blob index entries Summary: After adding expiration to blob index in #3066, we are now able to add a compaction filter to cleanup expired blob index entries. Closes https://github.com/facebook/rocksdb/pull/3090 Differential Revision: D6183812 Pulled By: yiwu-arbug fbshipit-source-id: 9cb03267a9702975290e758c9c176a2c03530b83 | 03 November 2017, 00:27:38 UTC |
| 76c3fbd | Sagar Vemuri | 03 November 2017, 00:16:11 UTC | Add Memtable Read Tier to RocksJava Summary: This options was introduced in the C++ API in #1953 . Closes https://github.com/facebook/rocksdb/pull/3064 Differential Revision: D6139010 Pulled By: sagar0 fbshipit-source-id: 164de11d539d174cf3afe7cd40e667049f44b0bc | 03 November 2017, 00:27:37 UTC |
| 7bfa880 | Yi Wu | 02 November 2017, 22:50:30 UTC | Blob DB: fix snapshot handling Summary: Blob db will keep blob file if data in the file is visible to an active snapshot. Before this patch it checks whether there is an active snapshot has sequence number greater than the earliest sequence in the file. This is problematic since we take snapshot on every read, if it keep having reads, old blob files will not be cleanup. Change to check if there is an active snapshot falls in the range of [earliest_sequence, obsolete_sequence) where obsolete sequence is 1. if data is relocated to another file by garbage collection, it is the latest sequence at the time garbage collection finish 2. otherwise, it is the latest sequence of the file Closes https://github.com/facebook/rocksdb/pull/3087 Differential Revision: D6182519 Pulled By: yiwu-arbug fbshipit-source-id: cdf4c35281f782eb2a9ad6a87b6727bbdff27a45 | 02 November 2017, 22:58:27 UTC |
| f662f8f | Yi Wu | 02 November 2017, 22:47:30 UTC | Blob DB: option to enable garbage collection Summary: Add an option to enable/disable auto garbage collection, where we keep counting how many keys have been evicted by either deletion or compaction and decide whether to garbage collect a blob file. Default disable auto garbage collection for now since the whole logic is not fully tested and we plan to make major change to it. Closes https://github.com/facebook/rocksdb/pull/3117 Differential Revision: D6224756 Pulled By: yiwu-arbug fbshipit-source-id: cdf53bdccec96a4580a2b3a342110ad9e8864dfe | 02 November 2017, 22:58:27 UTC |
| 167ba59 | Yi Wu | 02 November 2017, 20:20:28 UTC | Blob DB: Fix flaky BlobDBTest::GCExpiredKeyWhileOverwriting test Summary: The test intent to wait until key being overwritten until proceed with garbage collection. It failed to wait for `PutUntil` finally finish. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3116 Differential Revision: D6222833 Pulled By: yiwu-arbug fbshipit-source-id: fa9b57a772b92a66cf250b44e7975c43f62f45c5 | 02 November 2017, 20:27:34 UTC |
| 25ac169 | Sagar Vemuri | 02 November 2017, 19:02:42 UTC | Blob DB: Evict oldest blob file when close to blob db size limit Summary: Evict oldest blob file and put it in obsolete_files list when close to blob db size limit. The file will be delete when the `DeleteObsoleteFiles` background job runs next time. For now I set `kEvictOldestFileAtSize` constant, which controls when to evict the oldest file, at 90%. It could be tweaked or made into an option if really needed; I didn't want to expose it as an option pre-maturely as there are already too many :) . Closes https://github.com/facebook/rocksdb/pull/3094 Differential Revision: D6187340 Pulled By: sagar0 fbshipit-source-id: 687f8262101b9301bf964b94025a2fe9d8573421 | 02 November 2017, 19:11:21 UTC |
| 3c208e7 | Prashant D | 02 November 2017, 18:30:51 UTC | HistogramStat: Handle divide by zero situation Summary: The num() might return cur_num as 0 and we are making sure that cur_num will not be 0 down the path. The mult variable is being set to 100.0/cur_num which makes program crash when cur_num is 0. Closes https://github.com/facebook/rocksdb/pull/3105 Differential Revision: D6222594 Pulled By: ajkr fbshipit-source-id: 986154709897ff4dbbeb0e8aa81eb8c0b2a2db76 | 02 November 2017, 18:41:50 UTC |
| 25fbd9a | Maysam Yabandeh | 02 November 2017, 18:06:10 UTC | Remove the experimental notes about partitioning Summary: This patch will remove the existing comments that declare partitioning indexes and filters as experimental. Closes https://github.com/facebook/rocksdb/pull/3115 Differential Revision: D6222227 Pulled By: maysamyabandeh fbshipit-source-id: 6179ec43b22c518494051b674d91c9e1b54d4ac0 | 02 November 2017, 18:14:30 UTC |
| 60d83df | Maysam Yabandeh | 02 November 2017, 18:05:55 UTC | WritePrepared Txn: Move DB class to its own file Summary: Move WritePreparedTxnDB from pessimistic_transaction_db.h to its own header, write_prepared_txn_db.h Closes https://github.com/facebook/rocksdb/pull/3114 Differential Revision: D6220987 Pulled By: maysamyabandeh fbshipit-source-id: 18893fb4fdc6b809fe117dabb544080f9b4a301b | 02 November 2017, 18:14:30 UTC |
| 6778690 | Andrew Kryczka | 02 November 2017, 05:52:17 UTC | fix duplicate definition of GetEntryType() Summary: It's also defined in db/dbformat.cc per 7fe3b32896ecbb21d67ec52fccb713cb9bc6a644 Closes https://github.com/facebook/rocksdb/pull/3111 Differential Revision: D6219140 Pulled By: ajkr fbshipit-source-id: 0f2b14e41457334a4665c6b7e3f42f1a060a0f35 | 02 November 2017, 05:56:17 UTC |
| cd12421 | Andrew Kryczka | 02 November 2017, 04:17:42 UTC | release 5.9 Summary: updated HISTORY.md and version.h for the release. Closes https://github.com/facebook/rocksdb/pull/3110 Differential Revision: D6218645 Pulled By: ajkr fbshipit-source-id: 99ab8473e9088b02d7596e92351cce7a60a99e93 | 02 November 2017, 04:26:14 UTC |
| 02693f6 | Maysam Yabandeh | 02 November 2017, 01:56:25 UTC | WritePrepared Txn: ValidateSnapshot Summary: Implements ValidateSnapshot for WritePrepared txns and also adds a unit test to clarify the contract of this function. Closes https://github.com/facebook/rocksdb/pull/3101 Differential Revision: D6199405 Pulled By: maysamyabandeh fbshipit-source-id: ace509934c307ea5d26f4bbac5f836d7c80fd240 | 02 November 2017, 02:11:09 UTC |
| 7fe3b32 | Mikhail Antonov | 02 November 2017, 01:43:29 UTC | Added support for differential snapshots Summary: The motivation for this PR is to add to RocksDB support for differential (incremental) snapshots, as snapshot of the DB changes between two points in time (one can think of it as diff between to sequence numbers, or the diff D which can be thought of as an SST file or just set of KVs that can be applied to sequence number S1 to get the database to the state at sequence number S2). This feature would be useful for various distributed storages layers built on top of RocksDB, as it should help reduce resources (time and network bandwidth) needed to recover and rebuilt DB instances as replicas in the context of distributed storages. From the API standpoint that would like client app requesting iterator between (start seqnum) and current DB state, and reading the "diff". This is a very draft PR for initial review in the discussion on the approach, i'm going to rework some parts and keep updating the PR. For now, what's done here according to initial discussions: Preserving deletes: - We want to be able to optionally preserve recent deletes for some defined period of time, so that if a delete came in recently and might need to be included in the next incremental snapshot it would't get dropped by a compaction. This is done by adding new param to Options (preserve deletes flag) and new variable to DB Impl where we keep track of the sequence number after which we don't want to drop tombstones, even if they are otherwise eligible for deletion. - I also added a new API call for clients to be able to advance this cutoff seqnum after which we drop deletes; i assume it's more flexible to let clients control this, since otherwise we'd need to keep some kind of timestamp < -- > seqnum mapping inside the DB, which sounds messy and painful to support. Clients could make use of it by periodically calling GetLatestSequenceNumber(), noting the timestamp, doing some calculation and figuring out by how much we need to advance the cutoff seqnum. - Compaction codepath in compaction_iterator.cc has been modified to avoid dropping tombstones with seqnum > cutoff seqnum. Iterator changes: - couple params added to ReadOptions, to optionally allow client to request internal keys instead of user keys (so that client can get the latest value of a key, be it delete marker or a put), as well as min timestamp and min seqnum. TableCache changes: - I modified table_cache code to be able to quickly exclude SST files from iterators heep if creation_time on the file is less then iter_start_ts as passed in ReadOptions. That would help a lot in some DB settings (like reading very recent data only or using FIFO compactions), but not so much for universal compaction with more or less long iterator time span. What's left: - Still looking at how to best plug that inside DBIter codepath. So far it seems that FindNextUserKeyInternal only parses values as UserKeys, and iter->key() call generally returns user key. Can we add new API to DBIter as internal_key(), and modify this internal method to optionally set saved_key_ to point to the full internal key? I don't need to store actual seqnum there, but I do need to store type. Closes https://github.com/facebook/rocksdb/pull/2999 Differential Revision: D6175602 Pulled By: mikhail-antonov fbshipit-source-id: c779a6696ee2d574d86c69cec866a3ae095aa900 | 02 November 2017, 01:56:43 UTC |
| 17731a4 | Maysam Yabandeh | 02 November 2017, 00:23:52 UTC | WritePrepared Txn: Optimize for recoverable state Summary: GetCommitTimeWriteBatch is currently used to store some state as part of commit in 2PC. In MyRocks it is specifically used to store some data that would be needed only during recovery. So it is not need to be stored in memtable right after each commit. This patch enables an optimization to write the GetCommitTimeWriteBatch only to the WAL. The batch will be written to memtable during recovery when the WAL is replayed. To cover the case when WAL is deleted after memtable flush, the batch is also buffered and written to memtable right before each memtable flush. Closes https://github.com/facebook/rocksdb/pull/3071 Differential Revision: D6148023 Pulled By: maysamyabandeh fbshipit-source-id: 2d09bae5565abe2017c0327421010d5c0d55eaa7 | 02 November 2017, 00:26:46 UTC |
| c1cf94c | Maysam Yabandeh | 01 November 2017, 15:41:47 UTC | WritePrepared Txn: sort indexes before batch collapse Summary: The collapse of duplicate keys in write batch needs to sort the indexes of duplicate keys since it only checks the index in the batch with the head of the list of duplicate keys. Closes https://github.com/facebook/rocksdb/pull/3093 Differential Revision: D6186800 Pulled By: maysamyabandeh fbshipit-source-id: abc9ae8c2f1840445a5584f925cf86ecc6f37154 | 01 November 2017, 15:56:57 UTC |
| f6082d1 | Yi Wu | 31 October 2017, 23:33:55 UTC | Blob DB: cleanup unused options Summary: * cleanup num_concurrent_simple_blobs. We don't do concurrent writes (by taking write_mutex_) so it doesn't make sense to have multiple non TTL files open. We can revisit later when we want to improve writes. * cleanup eviction callback. we don't have plan to use it now. * rename s/open_simple_blob_files_/open_non_ttl_file_/ and s/open_blob_files_/open_ttl_files_/ to avoid confusion. Closes https://github.com/facebook/rocksdb/pull/3088 Differential Revision: D6182598 Pulled By: yiwu-arbug fbshipit-source-id: 99e6f5e01fa66d31309cdb06ce48502464bac6ad | 31 October 2017, 23:42:08 UTC |
| f5078dd | Sagar Vemuri | 31 October 2017, 23:27:22 UTC | Blob DB: Initialize all fields in Blob Header, Footer and Record structs Summary: Fixing un-itializations caught by valgrind. Closes https://github.com/facebook/rocksdb/pull/3103 Differential Revision: D6200195 Pulled By: sagar0 fbshipit-source-id: bf35a3fb03eb1d308e4c5ce30dee1e345d7b03b3 | 31 October 2017, 23:42:08 UTC |
| 33c7d4c | Shaohua Li | 31 October 2017, 20:49:25 UTC | Make writable_file_max_buffer_size dynamic Summary: The DBOptions::writable_file_max_buffer_size can be changed dynamically. Closes https://github.com/facebook/rocksdb/pull/3053 Differential Revision: D6152720 Pulled By: shligit fbshipit-source-id: aa0c0cfcfae6a54eb17faadb148d904797c68681 | 31 October 2017, 20:56:35 UTC |
| c1be8d8 | Prashant D | 31 October 2017, 08:24:20 UTC | Fix removed structurally dead return statement Summary: There seems to be a typo mistake in env ReuseWritableFile func where status is being returned twice. Closes https://github.com/facebook/rocksdb/pull/3099 Differential Revision: D6196204 Pulled By: ajkr fbshipit-source-id: abb6e3e1c1e772dd485fc39e7f1b9d502fa188fe | 31 October 2017, 08:26:13 UTC |
| 4d43c6a | Andrew Kryczka | 31 October 2017, 08:22:51 UTC | db_stress snapshot compatibility with reopens Summary: - Release all snapshots before crashing and reopening the DB. Without this, we may attempt to release snapshots from an old DB using a new DB. That tripped an assertion. - Release multiple snapshots in the same operation if needed. Without this, we would sometimes leak snapshots. Closes https://github.com/facebook/rocksdb/pull/3098 Differential Revision: D6194923 Pulled By: ajkr fbshipit-source-id: b9c89bcca7ebcbb6c7802c616f9d1175a005aadf | 31 October 2017, 08:26:08 UTC |
| b7bc9cc | Andrew Kryczka | 30 October 2017, 07:47:15 UTC | fix tracking oldest snapshot for bottom-level compaction Summary: The assertion was caught by `MySQLStyleTransactionTest/MySQLStyleTransactionTest.TransactionStressTest/5` when run in a loop. The caller doesn't track whether the released snapshot is oldest, so let this function handle that case. Closes https://github.com/facebook/rocksdb/pull/3080 Differential Revision: D6185257 Pulled By: ajkr fbshipit-source-id: 4b3015c11db5d31e46521a00af568546ef4558cd | 30 October 2017, 07:55:58 UTC |
