| b45fbc1 | Prashant D | 28 November 2017, 21:15:20 UTC | utilities: Fix coverity issues Summary: ``` utilities/persistent_cache/block_cache_tier_file.cc 64struct CacheRecordHeader { 2. uninit_member: Non-static class member magic_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member crc_ is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member key_size_ is not initialized in this constructor nor in any functions that it calls. CID 1396161 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member val_size_ is not initialized in this constructor nor in any functions that it calls. 65 CacheRecordHeader() {} 66 CacheRecordHeader(const uint32_t magic, const uint32_t key_size, 67 const uint32_t val_size) 68 : magic_(magic), crc_(0), key_size_(key_size), val_size_(val_size) {} 69 1. member_decl: Class member declaration for magic_. 70 uint32_t magic_; 3. member_decl: Class member declaration for crc_. 71 uint32_t crc_; 5. member_decl: Class member declaration for key_size_. 72 uint32_t key_size_; 7. member_decl: Class member declaration for val_size_. 73 uint32_t val_size_; 74}; utilities/simulator_cache/sim_cache.cc: 157 miss_times_(0), CID 1396124 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member stats_ is not initialized in this constructor nor in any functions that it calls. 158 hit_times_(0) {} 159 ``` Closes https://github.com/facebook/rocksdb/pull/3155 Differential Revision: D6427237 Pulled By: sagar0 fbshipit-source-id: 97e493da5fc043c5b9a3e0d33103442cffb75aad | 28 November 2017, 21:27:08 UTC |
| 7b57510 | Prashant D | 28 November 2017, 20:17:11 UTC | utilities: Fix coverity issues in blob_db and col_buf_decoder Summary: utilities/blob_db/blob_db_impl.cc 265 : bdb_options_.blob_dir; 3. uninit_member: Non-static class member env_ is not initialized in this constructor nor in any functions that it calls. 5. uninit_member: Non-static class member ttl_extractor_ is not initialized in this constructor nor in any functions that it calls. 7. uninit_member: Non-static class member open_p1_done_ is not initialized in this constructor nor in any functions that it calls. CID 1418245 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 9. uninit_member: Non-static class member debug_level_ is not initialized in this constructor nor in any functions that it calls. 266} 4. past_the_end: Function end creates an iterator. CID 1418258 (#1 of 1): Using invalid iterator (INVALIDATE_ITERATOR) 5. deref_iterator: Dereferencing iterator file_nums.end() though it is already past the end of its container. utilities/col_buf_decoder.h: nullable_(nullable), 2. uninit_member: Non-static class member remain_runs_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member run_val_ is not initialized in this constructor nor in any functions that it calls. CID 1396134 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 6. uninit_member: Non-static class member last_val_ is not initialized in this constructor nor in any functions that it calls. 46 big_endian_(big_endian) {} Closes https://github.com/facebook/rocksdb/pull/3134 Differential Revision: D6340607 Pulled By: sagar0 fbshipit-source-id: 25c52566e2ff979fe6c7abb0f40c27fc16597054 | 28 November 2017, 20:27:57 UTC |
| cf0d6aa | Tamir Duberstein | 28 November 2017, 20:13:20 UTC | CMake cross platform Java support and add JNI to travis Summary: Rewrite `java/CMakeLists.txt` to take advantage of CMake's cross platform Java support. adamretter Closes https://github.com/facebook/rocksdb/pull/2301 Differential Revision: D5070724 Pulled By: sagar0 fbshipit-source-id: 999aee9bd39da2b24a5fe493a2eb0e9af6072dc7 | 28 November 2017, 20:27:53 UTC |
| 7827935 | Yi Wu | 28 November 2017, 19:42:28 UTC | Blob DB: Add statistics Summary: Adding a list of blob db counters. Also remove WaStats() which doesn't expose the stats and can be substitute by (BLOB_DB_BYTES_WRITTEN / BLOB_DB_BLOB_FILE_BYTES_WRITTEN). Closes https://github.com/facebook/rocksdb/pull/3193 Differential Revision: D6394216 Pulled By: yiwu-arbug fbshipit-source-id: 017508c8ff3fcd7ea7403c64d0f9834b24816803 | 28 November 2017, 19:58:49 UTC |
| 3cf562b | Yi Wu | 28 November 2017, 19:40:40 UTC | Fix IOError on WAL write doesn't propagate to write group follower Summary: This is a simpler version of #3097 by removing all unrelated changes. Fixing the bug where concurrent writes may get Status::OK while it actually gets IOError on WAL write. This happens when multiple writes form a write batch group, and the leader get an IOError while writing to WAL. The leader failed to pass the error to followers in the group, and the followers end up returning Status::OK() while actually writing nothing. The bug only affect writes in a batch group. Future writes after the batch group will correctly return immediately with the IOError. Closes https://github.com/facebook/rocksdb/pull/3201 Differential Revision: D6421644 Pulled By: yiwu-arbug fbshipit-source-id: 1c2a455c5b73f6842423785eb8a9dbfbb191dc0e | 28 November 2017, 19:42:48 UTC |
| 1bdb44d | Andrew Kryczka | 28 November 2017, 19:18:42 UTC | optimize file ingestion checks for range deletion overlap Summary: Before we were checking every file in the level which was unnecessary. We can piggyback onto the code for checking point-key overlap, which already opens all the files that could possibly contain overlapping range deletions. This PR makes us check just the range deletions from those files, so no extra ones will be opened. Closes https://github.com/facebook/rocksdb/pull/3179 Differential Revision: D6358125 Pulled By: ajkr fbshipit-source-id: 00e200770fdb8f3cc6b1b2da232b755e4ba36279 | 28 November 2017, 19:27:02 UTC |
| 022c598 | Jeffrey Dallatezza | 28 November 2017, 19:15:34 UTC | Fix minor typo in comment Summary: mean -> meant Closes https://github.com/facebook/rocksdb/pull/3202 Differential Revision: D6426443 Pulled By: sagar0 fbshipit-source-id: adaf07218580ee6903986fa5686de92f43f420e1 | 28 November 2017, 19:27:02 UTC |
| 4b65cfc | Phani Shekhar Mantripragada | 28 November 2017, 18:35:17 UTC | Support for block_cache num_shards and other config via option string. Summary: Problem: Option string accepts only cache_size as parameter for block_cache which is specified as "block_cache=1M". It doesn't accept other parameters like num_shards etc. Changes : 1) ParseBlockBasedTableOption in block_based_table_factory is edited to accept cache options in the format "block_cache=<cache_size>:<num_shard_bits>:<strict_capacity_limit>:<high_pri_pool_ratio>". Options other than cache_size are optional to maintain backward compatibility. The changes are valid for block_cache_compressed as well. For example, "block_cache=1M:6:true:0.5", "block_cache=1M:6:true", "block_cache=1M:6" and "block_cache=1M" are all valid option strings. 2) Corresponding unit tests are added. Closes https://github.com/facebook/rocksdb/pull/3108 Differential Revision: D6420997 Pulled By: sagar0 fbshipit-source-id: cdea8b785688d2802907974af27225ccc1c0cd43 | 28 November 2017, 18:48:53 UTC |
| 2f09524 | Griffin Smith | 28 November 2017, 18:24:23 UTC | Expose all remaining read and write options via the C API Summary: Expose read and write options via the C API Closes https://github.com/facebook/rocksdb/pull/3185 Differential Revision: D6389658 Pulled By: sagar0 fbshipit-source-id: 1848912750329a476805b3cb2f315e7b71f61472 | 28 November 2017, 18:28:46 UTC |
| b72b3c6 | Maysam Yabandeh | 27 November 2017, 16:49:30 UTC | WritePrepared Txn: Add MultiGet to DB Summary: This patch implements MultiGet API for WritePreparedTxnDB and update the existing unit tests. Closes https://github.com/facebook/rocksdb/pull/3196 Differential Revision: D6401493 Pulled By: maysamyabandeh fbshipit-source-id: 51501a1e32645fc2da8680e77a50035f6530f2cc | 27 November 2017, 16:56:21 UTC |
| f0dde49 | Yi Wu | 24 November 2017, 19:50:42 UTC | Blob DB: Fix GC handling for inlined blob Summary: Garbage collection checks if the offset in blob index matches the offset of the blob value in the file. If it is a mismatch, the value is the current version. However it failed to check if the blob index is an inlined type, which don't even have an offset. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3194 Differential Revision: D6394270 Pulled By: yiwu-arbug fbshipit-source-id: 7c2b9d795f1116f55f4d728086980f9b6e88ea78 | 24 November 2017, 19:56:47 UTC |
| e59cb2a | Maysam Yabandeh | 22 November 2017, 21:52:56 UTC | Add seq_per_batch to WriteWithCallbackTest Summary: Augment WriteWithCallbackTest to also test when seq_per_batch is true. Closes https://github.com/facebook/rocksdb/pull/3195 Differential Revision: D6398143 Pulled By: maysamyabandeh fbshipit-source-id: 7bc4218609355ec20fed25df426a8455ec2390d3 | 22 November 2017, 21:56:44 UTC |
| 5fac472 | Zhongyi Xie | 22 November 2017, 04:06:55 UTC | make compaction_readahead_size_ thread safe Summary: this should fix the failing tsan_check Closes https://github.com/facebook/rocksdb/pull/3192 Differential Revision: D6390004 Pulled By: miasantreble fbshipit-source-id: 6cadfc6f68febb1a77b0abcdb5416570dad926a5 | 22 November 2017, 04:11:38 UTC |
| fe187e7 | Griffin Smith | 21 November 2017, 18:11:26 UTC | Add Elixir to the list of language bindings Summary: Closes https://github.com/facebook/rocksdb/pull/3183 Differential Revision: D6386140 Pulled By: maysamyabandeh fbshipit-source-id: ca71d54edd741c3b7d9676ee2bcf584a5d49bc35 | 21 November 2017, 18:13:14 UTC |
| 8954f83 | Sagar Vemuri | 21 November 2017, 07:19:46 UTC | Blob DB: db_bench flag to control BlobDB's garbage collection Summary: flag: blob_db_enable_gc, to control BlobDb's enable_garbage_collection. Closes https://github.com/facebook/rocksdb/pull/3190 Differential Revision: D6383395 Pulled By: sagar0 fbshipit-source-id: 4134e835150748c425b8187264273a54c6d8381c | 21 November 2017, 07:26:15 UTC |
| d394a6b | anand1976 | 21 November 2017, 05:12:55 UTC | Add a ticker stat for number of keys skipped during iteration Summary: This diff adds a new ticker stat, NUMBER_ITER_SKIP, to count the number of internal keys skipped during iteration. Keys can be skipped due to deletes, or lower sequence number, or higher sequence number than the one requested. Also, fix the issue when StatisticsData is naturally aligned on cacheline boundary, padding becomes a zero size array, which the Windows compiler doesn't like. So add a cacheline worth of padding in that case to keep it happy. We cannot conditionally add padding as gcc doesn't allow using sizeof in preprocessor directives. Closes https://github.com/facebook/rocksdb/pull/3177 Differential Revision: D6353897 Pulled By: anand1976 fbshipit-source-id: 441d5a09af9c4e22e7355242dfc0c7b27aa0a6c2 | 21 November 2017, 05:26:37 UTC |
| 578f36e | Sagar Vemuri | 21 November 2017, 05:06:30 UTC | Blob DB: Remove some redundant log lines Summary: Saw some redundant log lines when trying to benchmark blob db. So, removed the lines from blob_file.cc, and let the lines in blob_db_impl.cc take the lead. Closes https://github.com/facebook/rocksdb/pull/3189 Differential Revision: D6381726 Pulled By: sagar0 fbshipit-source-id: 5f0b1e56fe4bc3b715d89ea9b5749bd935cd0606 | 21 November 2017, 05:11:34 UTC |
| a6c6b8b | Zhongyi Xie | 21 November 2017, 00:31:04 UTC | Revert "No need for Restart Interval for meta blocks" Summary: See [issue 3169](https://github.com/facebook/rocksdb/issues/3169) for more information This reverts commit 593d3de37171d99a761ce2ab34ffa12654acd055. Closes https://github.com/facebook/rocksdb/pull/3188 Differential Revision: D6379271 Pulled By: miasantreble fbshipit-source-id: 88f9ed67ba52237ad9b6f7251db83672b62d7537 | 21 November 2017, 00:42:03 UTC |
| 30285ee | Maysam Yabandeh | 20 November 2017, 21:16:04 UTC | Fix calculating filter partition target size Summary: block_size_deviation is in percentage while the partition size is in bytes. The current code fails to take that into account resulting into very large target size for filter partitions. Closes https://github.com/facebook/rocksdb/pull/3187 Differential Revision: D6376069 Pulled By: maysamyabandeh fbshipit-source-id: 276546fc68f50e0da32c462abb46f6cf676db9b2 | 20 November 2017, 21:26:57 UTC |
| 0996e14 | Giorgio Azzinnaro | 19 November 2017, 17:59:38 UTC | Added ProfaneDB Summary: Added my project [ProfaneDB](https://profanedb.gitlab.io/) which uses RocksDB as the main storage engine Closes https://github.com/facebook/rocksdb/pull/3182 Differential Revision: D6370847 Pulled By: ajkr fbshipit-source-id: ff4bb6bdbc6e42fdb88bd793a84a0e04c129b6ae | 19 November 2017, 18:11:44 UTC |
| 0213990 | Maysam Yabandeh | 18 November 2017, 01:02:13 UTC | Move static variables out of the header file Summary: Static variables in header files will be instantiated in every file that includes the header file. This patch moves some of them from options_helper.h to its .cc files. It also moves the static variable out of the offset_of since the template function could also lead to multiple instantiation perhaps due to inlining. Fixes https://github.com/facebook/rocksdb/issues/3176 Closes https://github.com/facebook/rocksdb/pull/3178 Differential Revision: D6363794 Pulled By: maysamyabandeh fbshipit-source-id: d0a07f061b4d992ab4e0de2706e622131d258fdd | 18 November 2017, 01:12:27 UTC |
| 2d04ed6 | Gustav Davidsson | 17 November 2017, 19:56:41 UTC | Make trash-to-DB size ratio limit configurable Summary: Allow users to configure the trash-to-DB size ratio limit, so that ratelimits for deletes can be enforced even when larger portions of the database are being deleted. Closes https://github.com/facebook/rocksdb/pull/3158 Differential Revision: D6304897 Pulled By: gdavidsson fbshipit-source-id: a28dd13059ebab7d4171b953ed91ce383a84d6b3 | 17 November 2017, 19:58:17 UTC |
| 32e31d4 | Zhongyi Xie | 17 November 2017, 01:46:43 UTC | Make DBOption compaction_readahead_size dynamic Summary: Closes https://github.com/facebook/rocksdb/pull/3004 Differential Revision: D6056141 Pulled By: miasantreble fbshipit-source-id: 56df1630f464fd56b07d25d38161f699e0528b7f | 17 November 2017, 01:57:25 UTC |
| e27f60b | Andrew Kryczka | 16 November 2017, 03:30:35 UTC | distinguish kZSTDNotFinalCompression in compression string Summary: This confused some users who were getting compression type from the logs. Closes https://github.com/facebook/rocksdb/pull/3153 Differential Revision: D6294964 Pulled By: ajkr fbshipit-source-id: 3c813376d33682dc6ccafc9a78df1a2e2528985e | 16 November 2017, 03:41:59 UTC |
| dd49f89 | Yi Wu | 15 November 2017, 22:13:39 UTC | Fix TARGETS lint warnings. Summary: Fix buckifier script and regenerate TARGETS file with no lint warnings. Closes https://github.com/facebook/rocksdb/pull/3170 Differential Revision: D6328993 Pulled By: yiwu-arbug fbshipit-source-id: 17d0e4ed92f676f35fed76659386611cc72b00b2 | 15 November 2017, 22:28:34 UTC |
| bbcd3b0 | Yi Wu | 15 November 2017, 22:13:03 UTC | Suppress valgrind "unimplemented functionality" error Summary: Add ROCKSDB_VALGRIND_RUN macro and suppress false-positive "unimplemented functionality" throw by valgrind for steam hints. Another approach would be add a valgrind suppress file. Valgrind is suppose to print the suppression when given "--gen-suppressions=all" param, which is suppose to be the content for the suppression file. But it doesn't print. Closes https://github.com/facebook/rocksdb/pull/3174 Differential Revision: D6338786 Pulled By: yiwu-arbug fbshipit-source-id: 3559efa5f3b92d40d09ad6ac82bc7b59f86c75aa | 15 November 2017, 22:28:34 UTC |
| 54b4356 | Maysam Yabandeh | 15 November 2017, 16:22:54 UTC | WritePrepared Txn: Refactoring WriteCallback Summary: Refactor the logic around WriteCallback in the write path to clarify when and how exactly we advance the sequence number and making sure it is consistent across the code. Closes https://github.com/facebook/rocksdb/pull/3168 Differential Revision: D6324312 Pulled By: maysamyabandeh fbshipit-source-id: 9a34f479561fdb2a5d01ef6d37a28908d03bbe33 | 15 November 2017, 16:27:06 UTC |
| 53863b7 | Maysam Yabandeh | 15 November 2017, 16:19:57 UTC | WritePrepared Txn: fix bug with Rollback seq Summary: The sequence number was not properly advanced after a rollback marker. The patch extends the existing unit tests to detect the bug and also fixes it. Closes https://github.com/facebook/rocksdb/pull/3157 Differential Revision: D6304291 Pulled By: maysamyabandeh fbshipit-source-id: 1b519c44a5371b802da49c9e32bd00087a8da401 | 15 November 2017, 16:27:06 UTC |
| 175d5d6 | Maysam Yabandeh | 14 November 2017, 16:42:14 UTC | Properly destruct rebuilding_trx_ Summary: When testing rebuilding_trx_ in MemTableInserter might still be set before the tests finishes which would cause ASAN alarms for leaks. This patch deletes the pointers in MemTableInserter destructor. Closes https://github.com/facebook/rocksdb/pull/3162 Differential Revision: D6317113 Pulled By: maysamyabandeh fbshipit-source-id: a68be70709a4fff7ac2b768660119311968f9c21 | 14 November 2017, 16:56:50 UTC |
| 9871ea4 | Yi Wu | 14 November 2017, 05:14:21 UTC | Regression test build binaries with PORTABLE=1 Summary: We hit "Illegal instruction" error in regression test with "shlx" instruction. Setting PORTABLE=1 to resolve it. Closes https://github.com/facebook/rocksdb/pull/3165 Differential Revision: D6321972 Pulled By: yiwu-arbug fbshipit-source-id: cc9fe0dbd4698d1b66a750a0b062f66899862719 | 14 November 2017, 05:26:24 UTC |
| 42564ad | Yi Wu | 14 November 2017, 02:03:57 UTC | Blob DB: not using PinnableSlice move assignment Summary: The current implementation of PinnableSlice move assignment have an issue #3163. We are moving away from it instead of try to get the move assignment right, since it is too tricky. Closes https://github.com/facebook/rocksdb/pull/3164 Differential Revision: D6319201 Pulled By: yiwu-arbug fbshipit-source-id: 8f3279021f3710da4a4caa14fd238ed2df902c48 | 14 November 2017, 02:12:20 UTC |
| 2515266 | Maysam Yabandeh | 11 November 2017, 21:08:22 UTC | WritePrepared Txn: Refactoring TrackKeys Summary: This patch clarifies and refactors the logic around tracked keys in transactions. Closes https://github.com/facebook/rocksdb/pull/3140 Differential Revision: D6290258 Pulled By: maysamyabandeh fbshipit-source-id: 03b50646264cbcc550813c060b180fc7451a55c1 | 11 November 2017, 21:14:20 UTC |
| 2edc92b | Maysam Yabandeh | 11 November 2017, 19:23:43 UTC | WritePrepared Txn: cross-compatibility test Summary: Add tests to ensure that WritePrepared and WriteCommitted policies are cross compatible when the db WAL is empty. This is important when the admin want to switch between the policies. In such case, before the switch the admin needs to empty the WAL by i) committing/rollbacking all the pending transactions, ii) FlushMemTables Closes https://github.com/facebook/rocksdb/pull/3118 Differential Revision: D6227247 Pulled By: maysamyabandeh fbshipit-source-id: bcde3d92c1e89cda3b9cfa69f6a20af5d8993db7 | 11 November 2017, 19:28:37 UTC |
| 857adf3 | Maysam Yabandeh | 11 November 2017, 01:18:01 UTC | WritePrepared Txn: Refactor conf params Summary: Summary of changes: - Move seq_per_batch out of Options - Rename concurrent_prepare to two_write_queues - Add allocate_seq_only_for_data_ Closes https://github.com/facebook/rocksdb/pull/3136 Differential Revision: D6304458 Pulled By: maysamyabandeh fbshipit-source-id: 08e685bfa82bbc41b5b1c5eb7040a8ca6e05e58c | 11 November 2017, 01:28:12 UTC |
| 07c2738 | Andrew Kryczka | 11 November 2017, 00:53:02 UTC | prefer enabling cpu features via -march/-mcpu Summary: If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2. Closes https://github.com/facebook/rocksdb/pull/3156 Differential Revision: D6304703 Pulled By: ajkr fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef | 11 November 2017, 00:57:11 UTC |
| f8e2db0 | Dmitri Smirnov | 10 November 2017, 18:27:35 UTC | Fix crashes, address test issues and adjust windows test script Summary: Add per-exe execution capability Add fix parsing of groups/tests Add timer test exclusion Fix unit tests Ifdef threadpool specific tests that do not pass on Vista threadpool. Remove spurious outout from prefix_test so test case listing works properly. Fix not using standard test directories results in file creation errors in sst_dump_test. BlobDb fixes: In C++ end() iterators can not be dereferenced. They are not valid. When deleting blob_db_ set it to nullptr before any other code executes. Not fixed:. On Windows you can not delete a file while it is open. [ RUN ] BlobDBTest.ReadWhileGC d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied write_batch Should not call front() if there is a chance the container is empty Closes https://github.com/facebook/rocksdb/pull/3152 Differential Revision: D6293274 Pulled By: sagar0 fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956 | 10 November 2017, 18:41:57 UTC |
| eefd75a | Shaohua Li | 10 November 2017, 17:25:26 UTC | Stream Summary: Add a simple policy for NVMe write time life hint Closes https://github.com/facebook/rocksdb/pull/3095 Differential Revision: D6298030 Pulled By: shligit fbshipit-source-id: 9a72a42e32e92193af11599eb71f0cf77448e24d | 10 November 2017, 17:26:24 UTC |
| f1c5eab | kapitan-k | 10 November 2017, 02:06:50 UTC | updated c ingestexternalfileoptions for ingest behind Summary: Closes https://github.com/facebook/rocksdb/pull/3151 Differential Revision: D6293861 Pulled By: ajkr fbshipit-source-id: f8db0a71509d1cd8237f2d377bf9e1bb0464bdbf | 10 November 2017, 02:15:09 UTC |
| 93f69cb | Andrew Kryczka | 10 November 2017, 01:33:01 UTC | use bottommost compression when base level is bottommost Summary: The previous compression type selection caused unexpected behavior when the base level was also the bottommost level. The following sequence of events could happen: - full compaction generates files with `bottommost_compression` type - now base level is bottommost level since all files are in the same level - any compaction causes files to be rewritten `compression_per_level` type since bottommost compression didn't apply to base level I changed the code to make bottommost compression apply to base level. Closes https://github.com/facebook/rocksdb/pull/3141 Differential Revision: D6264614 Pulled By: ajkr fbshipit-source-id: d7aaa8675126896684154a1f2c9034d6214fde82 | 10 November 2017, 01:42:00 UTC |
| 5e9e5a4 | Yi Wu | 09 November 2017, 03:33:12 UTC | Blob DB: Fix race condition between flush and write Summary: A race condition will happen when: * a user thread writes a value, but it hits the write stop condition because there are too many un-flushed memtables, while holding blob_db_impl.write_mutex_. * Flush is triggered and call flush begin listener and try to acquire blob_db_impl.write_mutex_. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3149 Differential Revision: D6279805 Pulled By: yiwu-arbug fbshipit-source-id: 0e3c58afb78795ebe3360a2c69e05651e3908c40 | 09 November 2017, 03:42:22 UTC |
| ca75f0a | Yi Wu | 08 November 2017, 21:08:15 UTC | Blob DB: Fix release build Summary: `compression` shadow the method name in `BlobFile`. Rename it. Closes https://github.com/facebook/rocksdb/pull/3148 Differential Revision: D6274498 Pulled By: yiwu-arbug fbshipit-source-id: 7d293596530998b23b6b8a8940f983f9b6343a98 | 08 November 2017, 21:14:20 UTC |
| 4f9f124 | Yi Wu | 08 November 2017, 01:40:44 UTC | Blob DB: use compression in file header instead of global options Summary: To fix the issue of failing to decompress existing value after reopen DB with a different compression settings. Closes https://github.com/facebook/rocksdb/pull/3142 Differential Revision: D6267260 Pulled By: yiwu-arbug fbshipit-source-id: c7cf7f3e33b0cd25520abf4771cdf9180cc02a5f | 08 November 2017, 01:42:17 UTC |
| 114896c | Andrew Kryczka | 07 November 2017, 21:49:35 UTC | db_bench compression options Summary: - moved existing compression options to `InitializeOptionsGeneral` since they cannot be set through options file - added flag for `zstd_max_train_bytes` which was recently introduced by #3057 Closes https://github.com/facebook/rocksdb/pull/3128 Differential Revision: D6240460 Pulled By: ajkr fbshipit-source-id: 27dbebd86a55de237ba6a45cc79cff9214e82ebc | 07 November 2017, 22:00:03 UTC |
| 65c95d9 | Andrew Kryczka | 07 November 2017, 18:42:47 UTC | support db_bench compact benchmark on bottommost files Summary: Without this option, running the compact benchmark on a DB containing only bottommost files simply returned immediately. Closes https://github.com/facebook/rocksdb/pull/3138 Differential Revision: D6256660 Pulled By: ajkr fbshipit-source-id: e3b64543acd503d821066f4200daa201d4fb3a9d | 07 November 2017, 18:57:24 UTC |
| e03377c | Manuel Ung | 06 November 2017, 18:52:17 UTC | Add lock wait time as a perf context counter Summary: Adds two new counters: `key_lock_wait_count` counts how many times a lock was blocked by another transaction and had to wait, instead of being granted the lock immediately. `key_lock_wait_time` counts the time spent acquiring locks. Closes https://github.com/facebook/rocksdb/pull/3107 Differential Revision: D6217332 Pulled By: lth fbshipit-source-id: 55d4f46da5550c333e523263422fd61d6a46deb9 | 06 November 2017, 18:57:19 UTC |
| be410de | Yi Wu | 04 November 2017, 01:10:36 UTC | Fix PinnableSlice move assignment Summary: After move assignment, we need to re-initialized the moved PinnableSlice. Also update blob_db_impl.cc to not reuse the moved PinnableSlice since it is supposed to be in an undefined state after move. Closes https://github.com/facebook/rocksdb/pull/3127 Differential Revision: D6238585 Pulled By: yiwu-arbug fbshipit-source-id: bd99f2e37406c4f7de160c7dee6a2e8126bc224e | 04 November 2017, 01:13:21 UTC |
| a6d8e30 | Sagar Vemuri | 03 November 2017, 21:34:10 UTC | Remove unnecessary status check in TableCache::NewIterator Summary: While investigating the usage of `new_table_iterator_nanos` perf counter, I saw some code was wrapper around with unnecessary status check ... so removed it. Closes https://github.com/facebook/rocksdb/pull/3120 Differential Revision: D6229181 Pulled By: sagar0 fbshipit-source-id: f8a44fe67f5a05df94553fdb233b21e54e88cc34 | 03 November 2017, 21:42:08 UTC |
| 4c8f336 | Prashant D | 03 November 2017, 21:32:00 UTC | util: Fix coverity issues Summary: util/concurrent_arena.h: CID 1396145 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member free_begin_ is not initialized in this constructor nor in any functions that it calls. 94 Shard() : allocated_and_unused_(0) {} util/dynamic_bloom.cc: 1. Condition hash_func == NULL, taking true branch. CID 1322821 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 3. uninit_member: Non-static class member data_ is not initialized in this constructor nor in any functions that it calls. 47 hash_func_(hash_func == nullptr ? &BloomHash : hash_func) {} 48 util/file_reader_writer.h: 204 private: 205 AlignedBuffer buffer_; member_not_init_in_gen_ctor: The compiler-generated constructor for this class does not initialize buffer_offset_. 206 uint64_t buffer_offset_; CID 1418246 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) member_not_init_in_gen_ctor: The compiler-generated constructor for this class does not initialize buffer_len_. 207 size_t buffer_len_; 208}; util/thread_local.cc: 341#endif CID 1322795 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 3. uninit_member: Non-static class member pthread_key_ is not initialized in this constructor nor in any functions that it calls. 342} 40struct ThreadData { 2. uninit_member: Non-static class member next is not initialized in this constructor nor in any functions that it calls. CID 1400668 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 4. uninit_member: Non-static class member prev is not initialized in this constructor nor in any functions that it calls. 41 explicit ThreadData(ThreadLocalPtr::StaticMeta* _inst) : entries(), inst(_inst) {} 42 std::vector<Entry> entries; 1. member_decl: Class member declaration for next. 43 ThreadData* next; 3. member_decl: Class member declaration for prev. 44 ThreadData* prev; 45 ThreadLocalPtr::StaticMeta* inst; 46}; Closes https://github.com/facebook/rocksdb/pull/3123 Differential Revision: D6233566 Pulled By: sagar0 fbshipit-source-id: aa2068790ea69787a0035c0db39d59b0c25108db | 03 November 2017, 21:42:08 UTC |
| cfb120f | Andrew Kryczka | 03 November 2017, 18:48:12 UTC | fix CopyFile status checks Summary: copied from internal diff D6156261 Closes https://github.com/facebook/rocksdb/pull/3124 Differential Revision: D6230167 Pulled By: ajkr fbshipit-source-id: 17926bb1152d607556364e3aacfec0ef3c115748 | 03 November 2017, 18:57:10 UTC |
| d956169 | Yi Wu | 03 November 2017, 18:15:26 UTC | Fix clang build error Summary: Fix cast from size_t to unsigned int. Closes https://github.com/facebook/rocksdb/pull/3125 Differential Revision: D6232863 Pulled By: yiwu-arbug fbshipit-source-id: 4c6131168b1faec26f7820b2cf4a09c242d323b7 | 03 November 2017, 18:26:54 UTC |
| 2581c0a | Yi Wu | 03 November 2017, 17:22:50 UTC | Blob DB: Fix BlobDBTest::SnapshotAndGarbageCollection asan failure Summary: Fix unreleased snapshot at the end of the test. Closes https://github.com/facebook/rocksdb/pull/3126 Differential Revision: D6232867 Pulled By: yiwu-arbug fbshipit-source-id: 651ca3144fc573ea2ab0ab20f0a752fb4a101d26 | 03 November 2017, 17:26:59 UTC |
| 24ad430 | Andrew Kryczka | 03 November 2017, 05:46:13 UTC | pass key/value samples through zstd compression dictionary generator Summary: Instead of using samples directly, we now support passing the samples through zstd's dictionary generator when `CompressionOptions::zstd_max_train_bytes` is set to nonzero. If set to zero, we will use the samples directly as the dictionary -- same as before. Note this is the first step of #2987, extracted into a separate PR per reviewer request. Closes https://github.com/facebook/rocksdb/pull/3057 Differential Revision: D6116891 Pulled By: ajkr fbshipit-source-id: 70ab13cc4c734fa02e554180eed0618b75255497 | 03 November 2017, 05:56:36 UTC |
| c4c1f96 | Andrew Kryczka | 03 November 2017, 05:16:23 UTC | dynamically change current memtable size Summary: Previously setting `write_buffer_size` with `SetOptions` would only apply to new memtables. An internal user wanted it to take effect immediately, instead of at an arbitrary future point, to prevent OOM. This PR makes the memtable's size mutable, and makes `SetOptions()` mutate it. There is one case when we preserve the old behavior, which is when memtable prefix bloom filter is enabled and the user is increasing the memtable's capacity. That's because the prefix bloom filter's size is fixed and wouldn't work as well on a larger memtable. Closes https://github.com/facebook/rocksdb/pull/3119 Differential Revision: D6228304 Pulled By: ajkr fbshipit-source-id: e44bd9d10a5f8c9d8c464bf7436070bb3eafdfc9 | 03 November 2017, 05:28:10 UTC |
| 30e4e01 | Zhongyi Xie | 03 November 2017, 05:12:17 UTC | add missing else Summary: Closes https://github.com/facebook/rocksdb/pull/3121 Differential Revision: D6229415 Pulled By: miasantreble fbshipit-source-id: 57c7ad2fddf5dd6b8d7e3aaf6f62348151327dfb | 03 November 2017, 05:28:06 UTC |
| 602fe94 | Prashant D | 03 November 2017, 00:43:04 UTC | Fix coverity issues in include/rocksdb Summary: include/rocksdb/metadata.h: struct ColumnFamilyMetaData { CID 1322804 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member file_count is not initialized in this constructor nor in any functions that it calls. struct SstFileMetaData { 2. uninit_member: Non-static class member size is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member smallest_seqno is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member largest_seqno is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member num_reads_sampled is not initialized in this constructor nor in any functions that it calls. CID 1322807 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 10. uninit_member: Non-static class member being_compacted is not initialized in this constructor nor in any functions that it calls. include/rocksdb/sst_file_writer.h: struct ExternalSstFileInfo { 2. uninit_member: Non-static class member sequence_number is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member file_size is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member num_entries is not initialized in this constructor nor in any functions that it calls. CID 1351697 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member version is not initialized in this constructor nor in any functions that it calls. 31 ExternalSstFileInfo() {} include/rocksdb/utilities/transaction.h: explicit Transaction(const TransactionDB* db) {} 2. uninit_member: Non-static class member log_number_ is not initialized in this constructor nor in any functions that it calls. CID 1396133 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member field txn_state_._M_i is not initialized in this constructor nor in any functions that it calls. 473 Transaction() {} Closes https://github.com/facebook/rocksdb/pull/3100 Differential Revision: D6227651 Pulled By: sagar0 fbshipit-source-id: 5caa4a2cf9471d1f9c3c073f81473636e1f0aa14 | 03 November 2017, 00:56:48 UTC |
| 62578d8 | Yi Wu | 03 November 2017, 00:26:46 UTC | Blob DB: Add compaction filter to remove expired blob index entries Summary: After adding expiration to blob index in #3066, we are now able to add a compaction filter to cleanup expired blob index entries. Closes https://github.com/facebook/rocksdb/pull/3090 Differential Revision: D6183812 Pulled By: yiwu-arbug fbshipit-source-id: 9cb03267a9702975290e758c9c176a2c03530b83 | 03 November 2017, 00:27:38 UTC |
| 76c3fbd | Sagar Vemuri | 03 November 2017, 00:16:11 UTC | Add Memtable Read Tier to RocksJava Summary: This options was introduced in the C++ API in #1953 . Closes https://github.com/facebook/rocksdb/pull/3064 Differential Revision: D6139010 Pulled By: sagar0 fbshipit-source-id: 164de11d539d174cf3afe7cd40e667049f44b0bc | 03 November 2017, 00:27:37 UTC |
| 7bfa880 | Yi Wu | 02 November 2017, 22:50:30 UTC | Blob DB: fix snapshot handling Summary: Blob db will keep blob file if data in the file is visible to an active snapshot. Before this patch it checks whether there is an active snapshot has sequence number greater than the earliest sequence in the file. This is problematic since we take snapshot on every read, if it keep having reads, old blob files will not be cleanup. Change to check if there is an active snapshot falls in the range of [earliest_sequence, obsolete_sequence) where obsolete sequence is 1. if data is relocated to another file by garbage collection, it is the latest sequence at the time garbage collection finish 2. otherwise, it is the latest sequence of the file Closes https://github.com/facebook/rocksdb/pull/3087 Differential Revision: D6182519 Pulled By: yiwu-arbug fbshipit-source-id: cdf4c35281f782eb2a9ad6a87b6727bbdff27a45 | 02 November 2017, 22:58:27 UTC |
| f662f8f | Yi Wu | 02 November 2017, 22:47:30 UTC | Blob DB: option to enable garbage collection Summary: Add an option to enable/disable auto garbage collection, where we keep counting how many keys have been evicted by either deletion or compaction and decide whether to garbage collect a blob file. Default disable auto garbage collection for now since the whole logic is not fully tested and we plan to make major change to it. Closes https://github.com/facebook/rocksdb/pull/3117 Differential Revision: D6224756 Pulled By: yiwu-arbug fbshipit-source-id: cdf53bdccec96a4580a2b3a342110ad9e8864dfe | 02 November 2017, 22:58:27 UTC |
| 167ba59 | Yi Wu | 02 November 2017, 20:20:28 UTC | Blob DB: Fix flaky BlobDBTest::GCExpiredKeyWhileOverwriting test Summary: The test intent to wait until key being overwritten until proceed with garbage collection. It failed to wait for `PutUntil` finally finish. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3116 Differential Revision: D6222833 Pulled By: yiwu-arbug fbshipit-source-id: fa9b57a772b92a66cf250b44e7975c43f62f45c5 | 02 November 2017, 20:27:34 UTC |
| 25ac169 | Sagar Vemuri | 02 November 2017, 19:02:42 UTC | Blob DB: Evict oldest blob file when close to blob db size limit Summary: Evict oldest blob file and put it in obsolete_files list when close to blob db size limit. The file will be delete when the `DeleteObsoleteFiles` background job runs next time. For now I set `kEvictOldestFileAtSize` constant, which controls when to evict the oldest file, at 90%. It could be tweaked or made into an option if really needed; I didn't want to expose it as an option pre-maturely as there are already too many :) . Closes https://github.com/facebook/rocksdb/pull/3094 Differential Revision: D6187340 Pulled By: sagar0 fbshipit-source-id: 687f8262101b9301bf964b94025a2fe9d8573421 | 02 November 2017, 19:11:21 UTC |
| 3c208e7 | Prashant D | 02 November 2017, 18:30:51 UTC | HistogramStat: Handle divide by zero situation Summary: The num() might return cur_num as 0 and we are making sure that cur_num will not be 0 down the path. The mult variable is being set to 100.0/cur_num which makes program crash when cur_num is 0. Closes https://github.com/facebook/rocksdb/pull/3105 Differential Revision: D6222594 Pulled By: ajkr fbshipit-source-id: 986154709897ff4dbbeb0e8aa81eb8c0b2a2db76 | 02 November 2017, 18:41:50 UTC |
| 25fbd9a | Maysam Yabandeh | 02 November 2017, 18:06:10 UTC | Remove the experimental notes about partitioning Summary: This patch will remove the existing comments that declare partitioning indexes and filters as experimental. Closes https://github.com/facebook/rocksdb/pull/3115 Differential Revision: D6222227 Pulled By: maysamyabandeh fbshipit-source-id: 6179ec43b22c518494051b674d91c9e1b54d4ac0 | 02 November 2017, 18:14:30 UTC |
| 60d83df | Maysam Yabandeh | 02 November 2017, 18:05:55 UTC | WritePrepared Txn: Move DB class to its own file Summary: Move WritePreparedTxnDB from pessimistic_transaction_db.h to its own header, write_prepared_txn_db.h Closes https://github.com/facebook/rocksdb/pull/3114 Differential Revision: D6220987 Pulled By: maysamyabandeh fbshipit-source-id: 18893fb4fdc6b809fe117dabb544080f9b4a301b | 02 November 2017, 18:14:30 UTC |
| 6778690 | Andrew Kryczka | 02 November 2017, 05:52:17 UTC | fix duplicate definition of GetEntryType() Summary: It's also defined in db/dbformat.cc per 7fe3b32896ecbb21d67ec52fccb713cb9bc6a644 Closes https://github.com/facebook/rocksdb/pull/3111 Differential Revision: D6219140 Pulled By: ajkr fbshipit-source-id: 0f2b14e41457334a4665c6b7e3f42f1a060a0f35 | 02 November 2017, 05:56:17 UTC |
| cd12421 | Andrew Kryczka | 02 November 2017, 04:17:42 UTC | release 5.9 Summary: updated HISTORY.md and version.h for the release. Closes https://github.com/facebook/rocksdb/pull/3110 Differential Revision: D6218645 Pulled By: ajkr fbshipit-source-id: 99ab8473e9088b02d7596e92351cce7a60a99e93 | 02 November 2017, 04:26:14 UTC |
| 02693f6 | Maysam Yabandeh | 02 November 2017, 01:56:25 UTC | WritePrepared Txn: ValidateSnapshot Summary: Implements ValidateSnapshot for WritePrepared txns and also adds a unit test to clarify the contract of this function. Closes https://github.com/facebook/rocksdb/pull/3101 Differential Revision: D6199405 Pulled By: maysamyabandeh fbshipit-source-id: ace509934c307ea5d26f4bbac5f836d7c80fd240 | 02 November 2017, 02:11:09 UTC |
| 7fe3b32 | Mikhail Antonov | 02 November 2017, 01:43:29 UTC | Added support for differential snapshots Summary: The motivation for this PR is to add to RocksDB support for differential (incremental) snapshots, as snapshot of the DB changes between two points in time (one can think of it as diff between to sequence numbers, or the diff D which can be thought of as an SST file or just set of KVs that can be applied to sequence number S1 to get the database to the state at sequence number S2). This feature would be useful for various distributed storages layers built on top of RocksDB, as it should help reduce resources (time and network bandwidth) needed to recover and rebuilt DB instances as replicas in the context of distributed storages. From the API standpoint that would like client app requesting iterator between (start seqnum) and current DB state, and reading the "diff". This is a very draft PR for initial review in the discussion on the approach, i'm going to rework some parts and keep updating the PR. For now, what's done here according to initial discussions: Preserving deletes: - We want to be able to optionally preserve recent deletes for some defined period of time, so that if a delete came in recently and might need to be included in the next incremental snapshot it would't get dropped by a compaction. This is done by adding new param to Options (preserve deletes flag) and new variable to DB Impl where we keep track of the sequence number after which we don't want to drop tombstones, even if they are otherwise eligible for deletion. - I also added a new API call for clients to be able to advance this cutoff seqnum after which we drop deletes; i assume it's more flexible to let clients control this, since otherwise we'd need to keep some kind of timestamp < -- > seqnum mapping inside the DB, which sounds messy and painful to support. Clients could make use of it by periodically calling GetLatestSequenceNumber(), noting the timestamp, doing some calculation and figuring out by how much we need to advance the cutoff seqnum. - Compaction codepath in compaction_iterator.cc has been modified to avoid dropping tombstones with seqnum > cutoff seqnum. Iterator changes: - couple params added to ReadOptions, to optionally allow client to request internal keys instead of user keys (so that client can get the latest value of a key, be it delete marker or a put), as well as min timestamp and min seqnum. TableCache changes: - I modified table_cache code to be able to quickly exclude SST files from iterators heep if creation_time on the file is less then iter_start_ts as passed in ReadOptions. That would help a lot in some DB settings (like reading very recent data only or using FIFO compactions), but not so much for universal compaction with more or less long iterator time span. What's left: - Still looking at how to best plug that inside DBIter codepath. So far it seems that FindNextUserKeyInternal only parses values as UserKeys, and iter->key() call generally returns user key. Can we add new API to DBIter as internal_key(), and modify this internal method to optionally set saved_key_ to point to the full internal key? I don't need to store actual seqnum there, but I do need to store type. Closes https://github.com/facebook/rocksdb/pull/2999 Differential Revision: D6175602 Pulled By: mikhail-antonov fbshipit-source-id: c779a6696ee2d574d86c69cec866a3ae095aa900 | 02 November 2017, 01:56:43 UTC |
| 17731a4 | Maysam Yabandeh | 02 November 2017, 00:23:52 UTC | WritePrepared Txn: Optimize for recoverable state Summary: GetCommitTimeWriteBatch is currently used to store some state as part of commit in 2PC. In MyRocks it is specifically used to store some data that would be needed only during recovery. So it is not need to be stored in memtable right after each commit. This patch enables an optimization to write the GetCommitTimeWriteBatch only to the WAL. The batch will be written to memtable during recovery when the WAL is replayed. To cover the case when WAL is deleted after memtable flush, the batch is also buffered and written to memtable right before each memtable flush. Closes https://github.com/facebook/rocksdb/pull/3071 Differential Revision: D6148023 Pulled By: maysamyabandeh fbshipit-source-id: 2d09bae5565abe2017c0327421010d5c0d55eaa7 | 02 November 2017, 00:26:46 UTC |
| c1cf94c | Maysam Yabandeh | 01 November 2017, 15:41:47 UTC | WritePrepared Txn: sort indexes before batch collapse Summary: The collapse of duplicate keys in write batch needs to sort the indexes of duplicate keys since it only checks the index in the batch with the head of the list of duplicate keys. Closes https://github.com/facebook/rocksdb/pull/3093 Differential Revision: D6186800 Pulled By: maysamyabandeh fbshipit-source-id: abc9ae8c2f1840445a5584f925cf86ecc6f37154 | 01 November 2017, 15:56:57 UTC |
| f6082d1 | Yi Wu | 31 October 2017, 23:33:55 UTC | Blob DB: cleanup unused options Summary: * cleanup num_concurrent_simple_blobs. We don't do concurrent writes (by taking write_mutex_) so it doesn't make sense to have multiple non TTL files open. We can revisit later when we want to improve writes. * cleanup eviction callback. we don't have plan to use it now. * rename s/open_simple_blob_files_/open_non_ttl_file_/ and s/open_blob_files_/open_ttl_files_/ to avoid confusion. Closes https://github.com/facebook/rocksdb/pull/3088 Differential Revision: D6182598 Pulled By: yiwu-arbug fbshipit-source-id: 99e6f5e01fa66d31309cdb06ce48502464bac6ad | 31 October 2017, 23:42:08 UTC |
| f5078dd | Sagar Vemuri | 31 October 2017, 23:27:22 UTC | Blob DB: Initialize all fields in Blob Header, Footer and Record structs Summary: Fixing un-itializations caught by valgrind. Closes https://github.com/facebook/rocksdb/pull/3103 Differential Revision: D6200195 Pulled By: sagar0 fbshipit-source-id: bf35a3fb03eb1d308e4c5ce30dee1e345d7b03b3 | 31 October 2017, 23:42:08 UTC |
| 33c7d4c | Shaohua Li | 31 October 2017, 20:49:25 UTC | Make writable_file_max_buffer_size dynamic Summary: The DBOptions::writable_file_max_buffer_size can be changed dynamically. Closes https://github.com/facebook/rocksdb/pull/3053 Differential Revision: D6152720 Pulled By: shligit fbshipit-source-id: aa0c0cfcfae6a54eb17faadb148d904797c68681 | 31 October 2017, 20:56:35 UTC |
| c1be8d8 | Prashant D | 31 October 2017, 08:24:20 UTC | Fix removed structurally dead return statement Summary: There seems to be a typo mistake in env ReuseWritableFile func where status is being returned twice. Closes https://github.com/facebook/rocksdb/pull/3099 Differential Revision: D6196204 Pulled By: ajkr fbshipit-source-id: abb6e3e1c1e772dd485fc39e7f1b9d502fa188fe | 31 October 2017, 08:26:13 UTC |
| 4d43c6a | Andrew Kryczka | 31 October 2017, 08:22:51 UTC | db_stress snapshot compatibility with reopens Summary: - Release all snapshots before crashing and reopening the DB. Without this, we may attempt to release snapshots from an old DB using a new DB. That tripped an assertion. - Release multiple snapshots in the same operation if needed. Without this, we would sometimes leak snapshots. Closes https://github.com/facebook/rocksdb/pull/3098 Differential Revision: D6194923 Pulled By: ajkr fbshipit-source-id: b9c89bcca7ebcbb6c7802c616f9d1175a005aadf | 31 October 2017, 08:26:08 UTC |
| b7bc9cc | Andrew Kryczka | 30 October 2017, 07:47:15 UTC | fix tracking oldest snapshot for bottom-level compaction Summary: The assertion was caught by `MySQLStyleTransactionTest/MySQLStyleTransactionTest.TransactionStressTest/5` when run in a loop. The caller doesn't track whether the released snapshot is oldest, so let this function handle that case. Closes https://github.com/facebook/rocksdb/pull/3080 Differential Revision: D6185257 Pulled By: ajkr fbshipit-source-id: 4b3015c11db5d31e46521a00af568546ef4558cd | 30 October 2017, 07:55:58 UTC |
| 792ef10 | Yi Wu | 29 October 2017, 04:56:50 UTC | Return Status::InvalidArgument if user request sync write while disabling WAL Summary: write_options.sync = true and write_options.disableWAL is incompatible. When WAL is disabled, we are not able to persist the write immediately. Return an error in this case to avoid misuse of the options. Closes https://github.com/facebook/rocksdb/pull/3086 Differential Revision: D6176822 Pulled By: yiwu-arbug fbshipit-source-id: 1eb10028c14fe7d7c13c8bc12c0ef659f75aa071 | 29 October 2017, 05:11:18 UTC |
| 6a9335d | Andrew Kryczka | 27 October 2017, 22:43:42 UTC | always drop tombstones compacted to bottommost level Summary: Problem was in bottommost compaction, when an L0->L0 compaction happened and L0 was bottommost. Then we'd preserve tombstones according to `Compaction::KeyNotExistsBeyondOutputLevel`, while zeroing seqnum according to `CompactionIterator::PrepareOutput`, thus triggering the assertion in `PrepareOutput`. To fix, we can just drop tombstones in L0->L0 when the output is "bottommost", i.e., the compaction includes the oldest L0 file and there's nothing at lower levels. Closes https://github.com/facebook/rocksdb/pull/3085 Differential Revision: D6175742 Pulled By: ajkr fbshipit-source-id: 8ab19a2e001496f362e9eb0a71757e2f6ecfdb3b | 27 October 2017, 22:56:35 UTC |
| 84a04af | Yi Wu | 27 October 2017, 21:49:40 UTC | TableProperty::oldest_key_time defaults to 0 Summary: We don't propagate TableProperty::oldest_key_time on compaction and just write the default value to SST files. It is more natural to default the value to 0. Also revert db_sst_test back to before #2842. Closes https://github.com/facebook/rocksdb/pull/3079 Differential Revision: D6165702 Pulled By: yiwu-arbug fbshipit-source-id: ca3ce5928d96ae79a5beb12bb7d8c640a71478a0 | 27 October 2017, 22:00:05 UTC |
| 0599315 | Islam AbdelRahman | 27 October 2017, 20:25:54 UTC | Mark files as trash by using .trash extension Summary: SstFileManager move files that need to be deleted into a trash directory. Deprecate this behaviour and instead add ".trash" extension to files that need to be deleted Closes https://github.com/facebook/rocksdb/pull/2970 Differential Revision: D5976805 Pulled By: IslamAbdelRahman fbshipit-source-id: 27374ece4315610b2792c30ffcd50232d4c9a343 | 27 October 2017, 20:27:12 UTC |
| 3ebb7ba | Yi Wu | 27 October 2017, 20:14:34 UTC | Blob DB: update blob file format Summary: Changing blob file format and some code cleanup around the change. The change with blob log format are: * Remove timestamp field in blob file header, blob file footer and blob records. The field is not being use and often confuse with expiration field. * Blob file header now come with column family id, which always equal to default column family id. It leaves room for future support of column family. * Compression field in blob file header now is a standalone byte (instead of compact encode with flags field) * Blob file footer now come with its own crc. * Key length now being uint64_t instead of uint32_t * Blob CRC now checksum both key and value (instead of value only). * Some reordering of the fields. The list of cleanups: * Better inline comments in blob_log_format.h * rename ttlrange_t and snrange_t to ExpirationRange and SequenceRange respectively. * simplify blob_db::Reader * Move crc checking logic to inside blob_log_format.cc Closes https://github.com/facebook/rocksdb/pull/3081 Differential Revision: D6171304 Pulled By: yiwu-arbug fbshipit-source-id: e4373e0d39264441b7e2fbd0caba93ddd99ea2af | 27 October 2017, 20:27:12 UTC |
| 682db81 | Dmitri Smirnov | 27 October 2017, 20:14:07 UTC | Enable cacheline_aligned_alloc() to allocate from jemalloc if enabled. Summary: Reuse WITH_JEMALLOC option in preparation for module search unification. Move jemalloc overrides into a separate .cc Remote obsolete JEMALLOC_NOINIT option. Closes https://github.com/facebook/rocksdb/pull/3078 Differential Revision: D6174826 Pulled By: yiwu-arbug fbshipit-source-id: 9970a0289b4490272d15853920d9d7531af91140 | 27 October 2017, 20:27:12 UTC |
| d9240b5 | Prashant D | 27 October 2017, 18:24:12 UTC | Fix coverity uninitialized fields warnings in lru_cache Summary: Coverity uninitialized member variable warnings in lru_cache Closes https://github.com/facebook/rocksdb/pull/3082 Differential Revision: D6173062 Pulled By: sagar0 fbshipit-source-id: 7bcfc653457bd362d46045d06527838c9a6adad6 | 27 October 2017, 18:26:43 UTC |
| 50e95a6 | Prashant D | 27 October 2017, 18:19:31 UTC | Fix coverity issues column_family, compaction_db/iterator Summary: db/column_family.h : 79 ColumnFamilyHandleInternal() CID 1322806 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member internal_cfd_ is not initialized in this constructor nor in any functions that it calls. 80 : ColumnFamilyHandleImpl(nullptr, nullptr, nullptr) {} db/compacted_db_impl.cc: 18CompactedDBImpl::CompactedDBImpl( 19 const DBOptions& options, const std::string& dbname) 20 : DBImpl(options, dbname) { 2. uninit_member: Non-static class member cfd_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member version_ is not initialized in this constructor nor in any functions that it calls. CID 1396120 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 6. uninit_member: Non-static class member user_comparator_ is not initialized in this constructor nor in any functions that it calls. 21} db/compaction_iterator.cc: 9. uninit_member: Non-static class member current_user_key_sequence_ is not initialized in this constructor nor in any functions that it calls. 11. uninit_member: Non-static class member current_user_key_snapshot_ is not initialized in this constructor nor in any functions that it calls. CID 1419855 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 13. uninit_member: Non-static class member current_key_committed_ is not initialized in this constructor nor in any functions that it calls. Closes https://github.com/facebook/rocksdb/pull/3084 Differential Revision: D6172999 Pulled By: sagar0 fbshipit-source-id: 084d73393faf8022c01359cfb445807b6a782460 | 27 October 2017, 18:26:42 UTC |
| 47166ba | Prashant D | 27 October 2017, 04:00:17 UTC | Fix coverity uninitialized fields warnings Pulled By: ajkr Differential Revision: D6170448 fbshipit-source-id: 5fd6d1608fc0df27c94d9f5059315ce7f79b8f5c | 27 October 2017, 04:11:50 UTC |
| 67b29e2 | Prashant D | 27 October 2017, 03:54:48 UTC | Fix coverity issue for MutableDBOptions default constructor Summary: 228MutableDBOptions::MutableDBOptions() 229 : max_background_jobs(2), 230 base_background_compactions(-1), 231 max_background_compactions(-1), 232 avoid_flush_during_shutdown(false), 233 delayed_write_rate(2 * 1024U * 1024U), 234 max_total_wal_size(0), 235 delete_obsolete_files_period_micros(6ULL * 60 * 60 * 1000000), 236 stats_dump_period_sec(600), 2. uninit_member: Non-static class member bytes_per_sync is not initialized in this constructor nor in any functions that it calls. CID 1419857 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member wal_bytes_per_sync is not initialized in this constructor nor in any functions that it calls. 237 max_open_files(-1) {} Closes https://github.com/facebook/rocksdb/pull/3069 Differential Revision: D6170424 Pulled By: ajkr fbshipit-source-id: 1f94e86b87611ad2330b8b1707911150978d68b8 | 27 October 2017, 03:56:45 UTC |
| 9566738 | Andrew Kryczka | 27 October 2017, 00:14:04 UTC | implement lower bound for iterators Summary: - for `SeekToFirst()`, just convert it to a regular `Seek()` if lower bound is specified - for operations that iterate backwards over user keys (`SeekForPrev`, `SeekToLast`, `Prev`), change `PrevInternal` to check whether user key went below lower bound every time the user key changes -- same approach we use to ensure we stay within a prefix when `prefix_same_as_start=true`. Closes https://github.com/facebook/rocksdb/pull/3074 Differential Revision: D6158654 Pulled By: ajkr fbshipit-source-id: cb0e3a922e2650d2cd4d1c6e1c0f1e8b729ff518 | 27 October 2017, 00:27:42 UTC |
| 5a2a648 | Yi Wu | 26 October 2017, 19:19:43 UTC | Blob DB: Inline small values in base DB Summary: Adding the `min_blob_size` option to allow storing small values in base db (in LSM tree) together with the key. The goal is to improve performance for small values, while taking advantage of blob db's low write amplification for large values. Also adding expiration timestamp to blob index. It will be useful to evict stale blob indexes in base db by adding a compaction filter. I'll work on the compaction filter in future patches. See blob_index.h for the new blob index format. There are 4 cases when writing a new key: * small value w/o TTL: put in base db as normal value (i.e. ValueType::kTypeValue) * small value w/ TTL: put (type, expiration, value) to base db. * large value w/o TTL: write value to blob log and put (type, file, offset, size, compression) to base db. * large value w/TTL: write value to blob log and put (type, expiration, file, offset, size, compression) to base db. Closes https://github.com/facebook/rocksdb/pull/3066 Differential Revision: D6142115 Pulled By: yiwu-arbug fbshipit-source-id: 9526e76e19f0839310a3f5f2a43772a4ad182cd0 | 26 October 2017, 19:30:54 UTC |
| 9b18cc2 | Andrew Kryczka | 25 October 2017, 23:24:29 UTC | single-file bottom-level compaction when snapshot released Summary: When snapshots are held for a long time, files may reach the bottom level containing overwritten/deleted keys. We previously had no mechanism to trigger compaction on such files. This particularly impacted DBs that write to different parts of the keyspace over time, as such files would never be naturally compacted due to second-last level files moving down. This PR introduces a mechanism for bottommost files to be recompacted upon releasing all snapshots that prevent them from dropping their deleted/overwritten keys. - Changed `CompactionPicker` to compact files in `BottommostFilesMarkedForCompaction()`. These are the last choice when picking. Each file will be compacted alone and output to the same level in which it originated. The goal of this type of compaction is to rewrite the data excluding deleted/overwritten keys. - Changed `ReleaseSnapshot()` to recompute the bottom files marked for compaction when the oldest existing snapshot changes, and schedule a compaction if needed. We cache the value that oldest existing snapshot needs to exceed in order for another file to be marked in `bottommost_files_mark_threshold_`, which allows us to avoid recomputing marked files for most snapshot releases. - Changed `VersionStorageInfo` to track the list of bottommost files, which is recomputed every time the version changes by `UpdateBottommostFiles()`. The list of marked bottommost files is first computed in `ComputeBottommostFilesMarkedForCompaction()` when the version changes, but may also be recomputed when `ReleaseSnapshot()` is called. - Extracted core logic of `Compaction::IsBottommostLevel()` into `VersionStorageInfo::RangeMightExistAfterSortedRun()` since logic to check whether a file is bottommost is now necessary outside of compaction. Closes https://github.com/facebook/rocksdb/pull/3009 Differential Revision: D6062044 Pulled By: ajkr fbshipit-source-id: 123d201cf140715a7d5928e8b3cb4f9cd9f7ad21 | 25 October 2017, 23:30:37 UTC |
| 96e3a60 | Sagar Vemuri | 25 October 2017, 23:18:15 UTC | Return write error on reaching blob dir size limit Summary: I found that we continue accepting writes even when the blob db goes beyond the configured blob directory size limit. Now, we return an error for writes on reaching `blob_dir_size` limit and if `is_fifo` is set to false. (We cannot just drop any file when `is_fifo` is true.) Deleting the oldest file when `is_fifo` is true will be handled in a later PR. Closes https://github.com/facebook/rocksdb/pull/3060 Differential Revision: D6136156 Pulled By: sagar0 fbshipit-source-id: 2f11cb3f2eedfa94524fbfa2613dd64bfad7a23c | 25 October 2017, 23:30:37 UTC |
| addfe1e | Islam AbdelRahman | 25 October 2017, 22:08:01 UTC | Fix tombstone scans in SeekForPrev outside prefix Summary: When doing a Seek() or SeekForPrev() we should stop the moment we see a key with a different prefix as start if ReadOptions:: prefix_same_as_start was set to true Right now we don't stop if we encounter a tombstone outside the prefix while executing SeekForPrev() Closes https://github.com/facebook/rocksdb/pull/3067 Differential Revision: D6149638 Pulled By: IslamAbdelRahman fbshipit-source-id: 7f659862d2bf552d3c9104a360c79439ceba2f18 | 25 October 2017, 22:12:00 UTC |
| 386a57e | zach shipko | 24 October 2017, 20:14:00 UTC | Fix build on OpenBSD Summary: A few simple changes to allow RocksDB to be built on OpenBSD. Let me know if any further changes are needed. Closes https://github.com/facebook/rocksdb/pull/3061 Differential Revision: D6138800 Pulled By: ajkr fbshipit-source-id: a13a17b5dc051e6518bd56a8c5efd1d24dd81b0c | 24 October 2017, 20:27:38 UTC |
| 57fcdc2 | zawlazaw | 23 October 2017, 23:34:47 UTC | added missing subcodes and improved error message for missing enum values Summary: Java's `Status.SubCode` was out of sync with `include/rocksdb/status.h:SubCode`. When running out of disc space this led to an `IllegalArgumentException` because of an invalid status code, rather than just returning the corresponding status code without an exception. I added the missing status codes. By this, we keep the behaviour of throwing an `IllegalArgumentException` in case of newly added status codes that are defined in C but not in Java. We could think of an alternative strategy: add in Java another code "UnknownCode" which acts as a catch-all for all those status codes that are not yet mirrored from C to Java. This approach would never throw an exception but simply return a non-OK status-code. I think the current approach of throwing an Exception in case of a C/Java inconsistency is fine, but if you have some opinion on the alternative strategy, then feel free to comment here. Closes https://github.com/facebook/rocksdb/pull/3050 Differential Revision: D6129682 Pulled By: sagar0 fbshipit-source-id: f2bf44caad650837cffdcb1f93eb793b43580c66 | 23 October 2017, 23:42:07 UTC |
| 66a2c44 | Yi Wu | 23 October 2017, 22:22:05 UTC | Add DB::Properties::kEstimateOldestKeyTime Summary: With FIFO compaction we would like to get the oldest data time for monitoring. The problem is we don't have timestamp for each key in the DB. As an approximation, we expose the earliest of sst file "creation_time" property. My plan is to override the property with a more accurate value with blob db, where we actually have timestamp. Closes https://github.com/facebook/rocksdb/pull/2842 Differential Revision: D5770600 Pulled By: yiwu-arbug fbshipit-source-id: 03833c8f10bbfbee62f8ea5c0d03c0cafb5d853a | 23 October 2017, 22:27:27 UTC |
| d2a65c5 | Dmitri Smirnov | 23 October 2017, 21:20:53 UTC | Fix unused var warnings in Release mode Summary: MSVC does not support unused attribute at this time. A separate assignment line fixes the issue probably by being counted as usage for MSVC and it no longer complains about unused var. Closes https://github.com/facebook/rocksdb/pull/3048 Differential Revision: D6126272 Pulled By: maysamyabandeh fbshipit-source-id: 4907865db45fd75a39a15725c0695aaa17509c1f | 23 October 2017, 21:27:04 UTC |
| 63822eb | Maysam Yabandeh | 23 October 2017, 21:20:22 UTC | Enable two write queues for transactions Summary: Enable concurrent_prepare flag for WritePrepared transactions and extend the existing transaction tests with this config. Closes https://github.com/facebook/rocksdb/pull/3046 Differential Revision: D6106534 Pulled By: maysamyabandeh fbshipit-source-id: 88c8d21d45bc492beb0a131caea84a2ac5e7d38c | 23 October 2017, 21:27:04 UTC |
| a02ed12 | Sagar Vemuri | 21 October 2017, 00:09:46 UTC | Exclude DBTest.DynamicFIFOCompactionOptions test under RocksDB Lite Summary: This test shouldn't be enabled under the lite version; and this fixes the failing contrun test due to #3006. Closes https://github.com/facebook/rocksdb/pull/3056 Differential Revision: D6114681 Pulled By: sagar0 fbshipit-source-id: dc5243549ae6b1353cec7edb820c771d95f66dda | 21 October 2017, 00:11:39 UTC |
| 3ef55d2 | Maysam Yabandeh | 20 October 2017, 22:34:23 UTC | Split CompactionFilterWithValueChange Summary: The test currently times out when it is run under tsan. This patch split it into 4 tests. Closes https://github.com/facebook/rocksdb/pull/3047 Differential Revision: D6106515 Pulled By: maysamyabandeh fbshipit-source-id: 03a28cdf8b1c097be2361b1b0cc3dc1acf2b5d63 | 20 October 2017, 22:42:07 UTC |
| d75793d | Andrew Kryczka | 20 October 2017, 22:17:03 UTC | db_stress support long-held snapshots Summary: Add options to `db_stress` (correctness testing tool) to randomly acquire snapshot and release it after some period of time. It's useful for correctness testing of #3009, as well as other parts of compaction that behave differently depending on which snapshots are held. Closes https://github.com/facebook/rocksdb/pull/3038 Differential Revision: D6086501 Pulled By: ajkr fbshipit-source-id: 3ec0d8666c78ac507f1f808887c4ff759ba9b865 | 20 October 2017, 22:26:59 UTC |
| f8b5bb2 | Andrew Kryczka | 20 October 2017, 21:07:42 UTC | remove unused code Summary: fixup 6a541afcc4d1e5b6e6d78e288b9bee3bb2a933b5. This code didn't do anything because (1) `bytes_per_sync` is assigned in `EnvOptions`'s constructor; and (2) `OptimizeForCompactionTableWrite`'s return value was ignored, even though its only purpose is to return something. Closes https://github.com/facebook/rocksdb/pull/3055 Differential Revision: D6114132 Pulled By: ajkr fbshipit-source-id: ea4831770930e9cf83518e13eb2e1934d1f5487c | 20 October 2017, 21:11:52 UTC |
