| 3fe0937 | Zhongyi Xie | 29 January 2018, 22:34:56 UTC | Use block cache to track memory usage when ReadOptions.fill_cache=false Summary: ReadOptions.fill_cache is set in compaction inputs and can be set by users in their queries too. It tells RocksDB not to put a data block used to block cache. The memory used by the data block is, however, not trackable by users. To make the system more manageable, we can cost the block to block cache while using it, and then release it after using. Closes https://github.com/facebook/rocksdb/pull/3333 Differential Revision: D6670230 Pulled By: miasantreble fbshipit-source-id: ab848d3ed286bd081a13ee1903de357b56cbc308 | 29 January 2018, 22:43:10 UTC |
| e2d4b0e | Siying Dong | 29 January 2018, 22:13:05 UTC | db_bench: sanity check CuckooTable with mmap_read option Summary: This is to avoid run time error. Fail the db_bench immediately if cuckoo table is used but mmap_read is not specified. Closes https://github.com/facebook/rocksdb/pull/3420 Differential Revision: D6838284 Pulled By: siying fbshipit-source-id: 20893fa28d40fadc31e4ff154bed02f5a1bad341 | 29 January 2018, 22:27:32 UTC |
| b8eb32f | Mark Isaacson | 29 January 2018, 20:43:56 UTC | Suppress lint in old files Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues. Reviewed By: yiwu-arbug Differential Revision: D6821806 fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c | 29 January 2018, 20:56:42 UTC |
| 9f7ccc8 | Andrew Kryczka | 29 January 2018, 19:30:37 UTC | fix db_bench filluniquerandom key count assertion Summary: It failed every time. I guess people usually ran with assertions disabled. Closes https://github.com/facebook/rocksdb/pull/3422 Differential Revision: D6822984 Pulled By: ajkr fbshipit-source-id: 2e90db75618b26ac1c46ddfa9e03c095c7bf16e3 | 29 January 2018, 19:43:21 UTC |
| 3f666f7 | Mamy Ratsimbazafy | 29 January 2018, 17:51:13 UTC | Add Nim to the list of language bindings Summary: Closes https://github.com/facebook/rocksdb/pull/3428 Differential Revision: D6834061 Pulled By: maysamyabandeh fbshipit-source-id: edca5b5b8330e0fee646c7434b9631da76670240 | 29 January 2018, 17:57:46 UTC |
| 65cd6cd | Ben Darnell | 29 January 2018, 17:48:58 UTC | Rewrite comments on use_fsync option Summary: This replaces a vague warning about the mostly-obsolete ext3 filesystem with a more detailed note about a historical bug in the still-relevant ext4. Fixes #3410 Closes https://github.com/facebook/rocksdb/pull/3421 Differential Revision: D6834881 Pulled By: siying fbshipit-source-id: 7771ef5c89a54c0ac17821680779c48178d0b400 | 29 January 2018, 17:57:46 UTC |
| 46acdc9 | Maysam Yabandeh | 29 January 2018, 15:36:05 UTC | Split HarnessTest_Randomized to avoid timeout Summary: Split HarnessTest_Randomized to two tests Closes https://github.com/facebook/rocksdb/pull/3424 Differential Revision: D6826006 Pulled By: maysamyabandeh fbshipit-source-id: 59c9a11c7da092206effce6e4fa3792f9c66bef2 | 29 January 2018, 15:41:44 UTC |
| 439855a | Yi Wu | 26 January 2018, 23:12:16 UTC | StackableDB optionally take shared ownership of the underlying DB Summary: Allow StackableDB optionally takes a shared_ptr on construction and thus hold shared ownership of the underlying DB. Closes https://github.com/facebook/rocksdb/pull/3423 Differential Revision: D6824163 Pulled By: yiwu-arbug fbshipit-source-id: dbdc30c42e007533a987ef413785e192340f03eb | 26 January 2018, 23:28:44 UTC |
| 4927b4e | Maysam Yabandeh | 26 January 2018, 20:50:48 UTC | Rounddown in FilePrefetchBuffer::Prefetch Summary: FilePrefetchBuffer::Prefetch is currently rounds the offset up which does not fit its new use cases in prefetching index/filter blocks, as it would skips over some the offsets that were requested to be prefetched. This patch rounds down instead. Fixes #3180 Closes https://github.com/facebook/rocksdb/pull/3413 Differential Revision: D6816392 Pulled By: maysamyabandeh fbshipit-source-id: 3aaeaf59c55d72b61dacfae6d4a8e65eccb3c553 | 26 January 2018, 20:57:25 UTC |
| 7fcc1d0 | Sagar Vemuri | 26 January 2018, 19:01:54 UTC | Incorrect Universal Compaction reason Summary: While writing tests for dynamic Universal Compaction options, I found that the compaction reasons we set for size-ratio based and sorted-run based universal compactions are swapped with each other. Fixed it. Closes https://github.com/facebook/rocksdb/pull/3412 Differential Revision: D6820540 Pulled By: sagar0 fbshipit-source-id: 270a188968ba25b2c96a8339904416c4c87ff5b3 | 26 January 2018, 19:12:40 UTC |
| 0e6e405 | Andrew Kryczka | 26 January 2018, 18:46:30 UTC | db_bench support for memtable in-place update Summary: Closes https://github.com/facebook/rocksdb/pull/3416 Differential Revision: D6820606 Pulled By: ajkr fbshipit-source-id: 5035ffb33ade8d50520cafeb685ee8c8fcf1cca8 | 26 January 2018, 18:57:49 UTC |
| d938226 | Sagar Vemuri | 26 January 2018, 05:34:35 UTC | Improve performance of long range scans with readahead Summary: This change improves the performance of iterators doing long range scans (e.g. big/full table scans in MyRocks) by using readahead and prefetching additional data on each disk IO. This prefetching is automatically enabled on noticing more than 2 IOs for the same table file during iteration. The readahead size starts with 8KB and is exponentially increased on each additional sequential IO, up to a max of 256 KB. This helps in cutting down the number of IOs needed to complete the range scan. Constraints: - The prefetched data is stored by the OS in page cache. So this currently works only for non direct-reads use-cases i.e applications which use page cache. (Direct-I/O support will be enabled in a later PR). - This gets currently enabled only when ReadOptions.readahead_size = 0 (which is the default value). Thanks to siying for the original idea and implementation. **Benchmarks:** Data fill: ``` TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=fillrandom -num=1000000000 -compression_type="none" -level_compaction_dynamic_level_bytes ``` Do a long range scan: Seekrandom with large number of nexts ``` TEST_TMPDIR=/data/users/$USER/benchmarks/iter ./db_bench -benchmarks=seekrandom -duration=60 -num=1000000000 -use_existing_db -seek_nexts=10000 -statistics -histogram ``` Page cache was cleared before each experiment with the command: ``` sudo sh -c "echo 3 > /proc/sys/vm/drop_caches" ``` ``` Before: seekrandom : 34020.945 micros/op 29 ops/sec; 32.5 MB/s (1636 of 1999 found) With this change: seekrandom : 8726.912 micros/op 114 ops/sec; 126.8 MB/s (5702 of 6999 found) ``` ~3.9X performance improvement. Also verified with strace and gdb that the readahead size is increasing as expected. ``` strace -e readahead -f -T -t -p <db_bench process pid> ``` Closes https://github.com/facebook/rocksdb/pull/3282 Differential Revision: D6586477 Pulled By: sagar0 fbshipit-source-id: 8a118a0ed4594fbb7f5b1cafb242d7a4033cb58c | 26 January 2018, 05:41:53 UTC |
| 65d4316 | Ben Darnell | 26 January 2018, 02:07:42 UTC | Update comments about default WALRecoveryMode Summary: The default changed in 6a14f7a9766 but this comment was not updated. Closes https://github.com/facebook/rocksdb/pull/3409 Differential Revision: D6808264 Pulled By: maysamyabandeh fbshipit-source-id: 0d7e2a054eb181e9a144fcb783cf0b2c77219bc0 | 26 January 2018, 02:12:08 UTC |
| 1039133 | Siying Dong | 25 January 2018, 22:43:43 UTC | BlockBasedTable::NewDataBlockIterator to always return BlockIter Summary: This is a pre-cleaning up before a major block based table iterator refactoring. BlockBasedTable::NewDataBlockIterator() will always return BlockIter. This simplifies the logic and code and enable further refactoring and optimization. Closes https://github.com/facebook/rocksdb/pull/3398 Differential Revision: D6780165 Pulled By: siying fbshipit-source-id: 273f7dc896724f682c0118fb69a359d9cc4418b4 | 25 January 2018, 22:57:18 UTC |
| c722642 | Yi Wu | 24 January 2018, 00:47:41 UTC | WritePrepared Txn: Fix DBIterator and add test Summary: In DBIter, Prev() calls FindValueForCurrentKey() to search the current value backward. If it finds that there are too many stale value being skipped, it falls back to FindValueForCurrentKeyUsingSeek(), seeking directly to the key with snapshot sequence. After introducing read_callback, however, the key it seeks to might not be visible, according to read_callback. It thus needs to keep searching forward until the first visible value. Closes https://github.com/facebook/rocksdb/pull/3382 Differential Revision: D6756148 Pulled By: yiwu-arbug fbshipit-source-id: 064e39b1eec5e083af1c10142600f26d1d2697be | 24 January 2018, 00:57:11 UTC |
| d6fdd59 | Kamalalochana Subbaiah | 24 January 2018, 00:44:09 UTC | CMake changes for CRC32 Optimization on PowerPC Summary: Closes https://github.com/facebook/rocksdb/pull/2869 Differential Revision: D6791359 Pulled By: ajkr fbshipit-source-id: fdd38df603d84bbcce8d85dd1729d5caa256e6be | 24 January 2018, 00:57:11 UTC |
| 35d8e65 | Yi Wu | 24 January 2018, 00:06:18 UTC | Make Iterator::SeekForPrev pure virtual Summary: To prevent user who implement the Iterator interface fail to implement SeekForPrev by mistake. Closes https://github.com/facebook/rocksdb/pull/3402 Differential Revision: D6790681 Pulled By: yiwu-arbug fbshipit-source-id: bd75b8ced30208982e0a1414d34384d93496827a | 24 January 2018, 00:12:19 UTC |
| d46e832 | Yi Wu | 23 January 2018, 23:08:29 UTC | Assert last reference before destroy ColumnFamilyData Summary: In ColumnFamilySet destructor, assert it hold the last reference to cfd before destroy them. Closes #3112 Closes https://github.com/facebook/rocksdb/pull/3397 Differential Revision: D6777967 Pulled By: yiwu-arbug fbshipit-source-id: 60b19070e0c194b3b6146699140c1d68777866cb | 23 January 2018, 23:12:28 UTC |
| edc2581 | Yi Wu | 23 January 2018, 22:36:54 UTC | DB::DumpSupportInfo should log all supported compression types Summary: DB::DumpSupportInfo should log all supported compression types. Closes #3146 Closes https://github.com/facebook/rocksdb/pull/3396 Differential Revision: D6777019 Pulled By: yiwu-arbug fbshipit-source-id: 5b17f1ffb2d71224e52f7d9c045434746c789fb0 | 23 January 2018, 22:44:12 UTC |
| ec0167e | Nathan VanBenschoten | 23 January 2018, 20:46:12 UTC | Fix WriteBatch rep_ format for RangeDeletion records Summary: This is a small amount of general cleanup I made while experimenting with https://github.com/facebook/rocksdb/issues/3391. Closes https://github.com/facebook/rocksdb/pull/3392 Differential Revision: D6788365 Pulled By: yiwu-arbug fbshipit-source-id: 2716e5aabd5424a4dfdaa954361a62c8eb721ae2 | 23 January 2018, 20:57:32 UTC |
| 0ea7170 | Sagar Vemuri | 23 January 2018, 18:20:57 UTC | Remove old misleading comments Summary: FIFO and Universal compaction options were recently made dynamic, but I forgot to update these comments. These would mislead anyone who is reading the code. Closes https://github.com/facebook/rocksdb/pull/3399 Differential Revision: D6786358 Pulled By: sagar0 fbshipit-source-id: 57cfc412f63deaee29bbd82b863304821d60057d | 23 January 2018, 18:27:41 UTC |
| 7e3d332 | Yi Wu | 23 January 2018, 06:30:44 UTC | Blob DB: dump blob_db_options.min_blob_size Summary: min_blob_size was missing from BlobDBOptions::Dump. Closes https://github.com/facebook/rocksdb/pull/3400 Differential Revision: D6781525 Pulled By: yiwu-arbug fbshipit-source-id: 40d9b391578d7f8c91bd89f4ce2eda5064864c25 | 23 January 2018, 06:41:27 UTC |
| 7291a3f | Siying Dong | 23 January 2018, 00:36:00 UTC | Improve fallocate size in compaction output Summary: Now in leveled compaction, we allocate solely based on output target file size. If the total input size is smaller than the number, we should use the total input size instead. Also, cap the allocate size to 1GB. Closes https://github.com/facebook/rocksdb/pull/3385 Differential Revision: D6762363 Pulled By: siying fbshipit-source-id: e30906f6e9bff3ec847d2166e44cb49c92f98a13 | 23 January 2018, 00:43:46 UTC |
| c615689 | Islam AbdelRahman | 22 January 2018, 22:37:37 UTC | Support skipping bloom filters for SstFileWriter Summary: Add an option for SstFileWriter to skip building bloom filters Closes https://github.com/facebook/rocksdb/pull/3360 Differential Revision: D6709120 Pulled By: IslamAbdelRahman fbshipit-source-id: 964d4bce38822a048691792f447bcfbb4b6bd809 | 22 January 2018, 22:42:18 UTC |
| 6f5ba0b | Bernard Spil | 20 January 2018, 01:07:52 UTC | Fix building on FreeBSD Summary: FreeBSD uses jemalloc as the base malloc implementation. The patch has been functional on FreeBSD as of the MariaDB 10.2 port. Closes https://github.com/facebook/rocksdb/pull/3386 Differential Revision: D6765742 Pulled By: yiwu-arbug fbshipit-source-id: d55dbc082eecf640ef3df9a21f26064ebe6587e8 | 20 January 2018, 01:12:43 UTC |
| f2f034e | Yi Wu | 20 January 2018, 00:19:34 UTC | Blob DB: fix crash when DB full but no candidate file to evict Summary: When blob_files is empty, std::min_element will return blobfiles.end(), which cannot be dereference. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3387 Differential Revision: D6764927 Pulled By: yiwu-arbug fbshipit-source-id: 86f78700132be95760d35ac63480dfd3a8bbe17a | 20 January 2018, 00:26:50 UTC |
| 5568aec | Yi Wu | 19 January 2018, 20:52:48 UTC | Fix DBTest::SoftLimit TSAN failure Summary: Fix data race found by TSAN around WriteStallListener: https://gist.github.com/yiwu-arbug/027d2448b903648f2f0f40b05258d80f Closes https://github.com/facebook/rocksdb/pull/3384 Differential Revision: D6762167 Pulled By: yiwu-arbug fbshipit-source-id: cd3a5c9f806de390bd1af6077ea6dbbc8bcaec09 | 19 January 2018, 20:57:15 UTC |
| 47ad6b8 | Siying Dong | 19 January 2018, 20:26:06 UTC | Add 5.10.fb to tools/check_format_compatible.sh Summary: Closes https://github.com/facebook/rocksdb/pull/3383 Differential Revision: D6762375 Pulled By: siying fbshipit-source-id: dc1e0dc9718ffb59ffe42e2a2c844b67f935a5fb | 19 January 2018, 20:42:07 UTC |
| f1cb83f | Yi Wu | 19 January 2018, 01:32:50 UTC | Fix Flush() keep waiting after flush finish Summary: Flush() call could be waiting indefinitely if min_write_buffer_number_to_merge is used. Consider the sequence: 1. User call Flush() with flush_options.wait = true 2. The manual flush started in the background 3. New memtable become immutable because of writes. The new memtable will not trigger flush if min_write_buffer_number_to_merge is not reached. 4. The manual flush finish. Because of the new memtable created at step 3 not being flush, previous logic of WaitForFlushMemTable() keep waiting, despite the memtables it intent to flush has been flushed. Here instead of checking if there are any more memtables to flush, WaitForFlushMemTable() also check the id of the earliest memtable. If the id is larger than that of latest memtable at the time flush was initiated, it means all the memtable at the time of flush start has all been flush. Closes https://github.com/facebook/rocksdb/pull/3378 Differential Revision: D6746789 Pulled By: yiwu-arbug fbshipit-source-id: 35e698f71c7f90b06337a93e6825f4ea3b619bfa | 19 January 2018, 01:45:16 UTC |
| b987316 | topilski | 18 January 2018, 22:45:48 UTC | Fixed get version on windows, moved throwing exceptions into cc file. Summary: Fixes for msys2 and mingw, hide exceptions into cpp file. Closes https://github.com/facebook/rocksdb/pull/3377 Differential Revision: D6746707 Pulled By: yiwu-arbug fbshipit-source-id: 456b38df80bc48b8386a2cf87f669b5a4f9999a4 | 18 January 2018, 22:56:56 UTC |
| 4decff6 | jonasf | 18 January 2018, 18:41:26 UTC | Add possibility to change ttl on open DB Summary: We have seen cases where it could be good to change TTL on already open DB. Change ttl in TtlCompactionFilterFactory on open db. Next time a filter is created, it will filter accroding to the set TTL. Is this something that could be useful for others? Any downsides? Closes https://github.com/facebook/rocksdb/pull/3292 Differential Revision: D6731993 Pulled By: miasantreble fbshipit-source-id: 73b94d69237b11e8730734389052429d621a6b1e | 18 January 2018, 18:42:15 UTC |
| 46e599f | Andrew Kryczka | 18 January 2018, 01:37:10 UTC | fix live WALs purged while file deletions disabled Summary: When calling `DisableFileDeletions` followed by `GetSortedWalFiles`, we guarantee the files returned by the latter call won't be deleted until after file deletions are re-enabled. However, `GetSortedWalFiles` didn't omit files already planned for deletion via `PurgeObsoleteFiles`, so the guarantee could be broken. We fix it by making `GetSortedWalFiles` wait for the number of pending purges to hit zero if file deletions are disabled. This condition is eventually met since `PurgeObsoleteFiles` is guaranteed to be called for the existing pending purges, and new purges cannot be scheduled while file deletions are disabled. Once the condition is met, `GetSortedWalFiles` simply returns the content of DB and archive directories, which nobody can delete (except for deletion scheduler, for which I plan to fix this bug later) until deletions are re-enabled. Closes https://github.com/facebook/rocksdb/pull/3341 Differential Revision: D6681131 Pulled By: ajkr fbshipit-source-id: 90b1e2f2362ea9ef715623841c0826611a817634 | 18 January 2018, 01:42:04 UTC |
| 266d85f | Andrew Kryczka | 17 January 2018, 06:56:47 UTC | fix DBTest.AutomaticConflictsWithManualCompaction Summary: After af92d4ad112f192693f6017f24f9ae1b00e1f053, only exclusive manual compaction can have conflict. dc360df81ec48e56a5d9cee4adb7f11ef0ca82ac updated the conflict-checking test case accordingly. But we missed the point that exclusive manual compaction can only conflict with automatic compactions scheduled after it, since it waits on pending automatic compactions before it begins running. This PR updates the test case to ensure the automatic compactions are scheduled after the manual compaction starts but before it finishes, thus ensuring a conflict. I also cleaned up the test case to use less space as I saw it cause out-of-space error on travis. Closes https://github.com/facebook/rocksdb/pull/3375 Differential Revision: D6735162 Pulled By: ajkr fbshipit-source-id: 020530a4e150a4786792dce7cec5d66b420cb884 | 17 January 2018, 07:12:00 UTC |
| dc360df | Yi Wu | 17 January 2018, 01:26:29 UTC | Fix multiple build failures Summary: * Fix DBTest.CompactRangeWithEmptyBottomLevel lite build failure * Fix DBTest.AutomaticConflictsWithManualCompaction failure introduce by #3366 * Fix BlockBasedTableTest::IndexUncompressed should be disabled if snappy is disabled * Fix ASAN failure with DBBasicTest::DBClose test Closes https://github.com/facebook/rocksdb/pull/3373 Differential Revision: D6732313 Pulled By: yiwu-arbug fbshipit-source-id: 1eb9b9d9a8d795f56188fa9770db9353f6fdedc5 | 17 January 2018, 01:30:39 UTC |
| bf6f03f | Bartek Wrona | 16 January 2018, 22:13:38 UTC | Issue #3370 Broken CMakeLists.txt Summary: Issue #3370 Simple fixes to make RocksDB project working also as a submodule of other bigger one. Closes https://github.com/facebook/rocksdb/pull/3372 Differential Revision: D6729595 Pulled By: ajkr fbshipit-source-id: eee2589e7a7c4322873dff8510eebd050301c54c | 16 January 2018, 22:26:50 UTC |
| af92d4a | Sunguck Lee | 16 January 2018, 21:10:34 UTC | Avoid too frequent MaybeScheduleFlushOrCompaction() call Summary: If there's manual compaction in the queue, then "HaveManualCompaction(compaction_queue_.front())" will return true, and this cause too frequent MaybeScheduleFlushOrCompaction(). https://github.com/facebook/rocksdb/issues/3198 Closes https://github.com/facebook/rocksdb/pull/3366 Differential Revision: D6729575 Pulled By: ajkr fbshipit-source-id: 96da04f8fd33297b1ccaec3badd9090403da29b0 | 16 January 2018, 21:12:12 UTC |
| d0f1b49 | Anand Ananthabhotla | 16 January 2018, 18:57:56 UTC | Add a Close() method to DB to return status when closing a db Summary: Currently, the only way to close an open DB is to destroy the DB object. There is no way for the caller to know the status. In one instance, the destructor encountered an error due to failure to close a log file on HDFS. In order to prevent silent failures, we add DB::Close() that calls CloseImpl() which must be implemented by its descendants. The main failure point in the destructor is closing the log file. This patch also adds a Close() entry point to Logger in order to get status. When DBOptions::info_log is allocated and owned by the DBImpl, it is explicitly closed by DBImpl::CloseImpl(). Closes https://github.com/facebook/rocksdb/pull/3348 Differential Revision: D6698158 Pulled By: anand1976 fbshipit-source-id: 9468e2892553eb09c4c41b8723f590c0dbd8ab7d | 16 January 2018, 19:08:57 UTC |
| 68829ed | Sagar Vemuri | 13 January 2018, 07:36:27 UTC | Revert Snappy version upgrade Summary: Java static builds are again broken, this time due Snappy version upgrade introduced in 90c1d81975a03b0f8b352ddc614fbc99c2496ddd (#3331). This is due to two reasons: 1. The new Snappy packages should now be downloaded from https://github.com/google/snappy/archive/<pkg.tar.gz> instead of https://github.com/google/snappy/releases/download/<pkg.tar.gz> which we are using now. 1. In addition to the the above URL change, Snappy changed its build from using autotools to CMake based : https://github.com/google/snappy/blame/e69d9f880677f2aa3488c80b953ec4309f0dfa2e/README.md#L65-L72 So more changes are needed if we are going to upgrade to 1.1.7. Hence reverting the version upgrade until we figure them out. Closes https://github.com/facebook/rocksdb/pull/3363 Differential Revision: D6716983 Pulled By: sagar0 fbshipit-source-id: f451a1bc5eb0bb090f4da07bc3e5ba72cf89aefa | 13 January 2018, 07:41:43 UTC |
| 43549c7 | Andrew Kryczka | 12 January 2018, 21:16:39 UTC | Prevent unnecessary calls to PurgeObsoleteFiles Summary: Split `JobContext::HaveSomethingToDelete` into two functions: itself and `JobContext::HaveSomethingToClean`. Now we won't call `DBImpl::PurgeObsoleteFiles` in cases where we really just need to call `JobContext::Clean`. The change is needed because I want to track pending calls to `PurgeObsoleteFiles` for a bug fix, which is much simpler if we only call it after `FindObsoleteFiles` finds files to delete. Closes https://github.com/facebook/rocksdb/pull/3350 Differential Revision: D6690609 Pulled By: ajkr fbshipit-source-id: 61502e7469288afe16a663a1b7df345baeaf246f | 12 January 2018, 21:27:08 UTC |
| ba295cd | Andrew Kryczka | 12 January 2018, 19:03:55 UTC | replace DBTest.HugeNumbersOfLevel with a more targeted test case Summary: This test often causes out-of-space error when run on travis. We don't want such stress tests in our unit test suite. The bug in #596, which this test intends to expose, can be repro'd as long as the bottommost level(s) are empty when CompactRange is called. I rewrote the test to cover this simple case without writing a lot of data. Closes https://github.com/facebook/rocksdb/pull/3362 Differential Revision: D6710417 Pulled By: ajkr fbshipit-source-id: 9a1ec85e738c813ac2fee29f1d5302065ecb54c5 | 12 January 2018, 19:12:09 UTC |
| e446d14 | Sagar Vemuri | 12 January 2018, 18:53:43 UTC | Fix PowerPC dynamic java build Summary: Java build on PPC64le has been broken since a few months, due to #2716. Fixing it with the least amount of changes. (We should cleanup a little around this code when time permits). This should fix the build failures seen in http://140.211.168.68:8080/job/Rocksdb/ . Closes https://github.com/facebook/rocksdb/pull/3359 Differential Revision: D6712938 Pulled By: sagar0 fbshipit-source-id: 3046e8f072180693de2af4762934ec1ace309ca4 | 12 January 2018, 18:57:14 UTC |
| 6d7e3b9 | Andrew Kryczka | 12 January 2018, 04:08:44 UTC | fix Gemfile.lock nokogiri dependencies Summary: I installed the ruby dependencies and ran `bundle update nokogiri`. It depends on a newer version of "mini_portile2" which I missed in 9c2f64e1488d577746686189d4d127dcf3a7f6e0. Now `bundle install` works again. Closes https://github.com/facebook/rocksdb/pull/3361 Differential Revision: D6710164 Pulled By: ajkr fbshipit-source-id: 9a08d6cc6400ef495b715b3d68b04ce3f3367031 | 12 January 2018, 04:11:32 UTC |
| 45828c7 | Peter (Stig) Edwards | 12 January 2018, 02:49:48 UTC | Consider an increase to buffer size when reading option file, from 4K to 8K. Summary: Hello and thank you for RocksDB, While looking into the buffered io used when an `OPTIONS` file is read I noticed the `OPTIONS` files produced by RocksDB 5.8.8 (and head of master) were just over 4096 bytes in size, resulting in the version of glibc I am using (glibc-2.17-196.el7) (on the filesystem used) being passed a 4K buffer for the `fread_unlocked` call and 2 system call reads using a 4096 buffer being used to read the contents of the `OPTIONS` file. If the buffer size is increased to 8192 then 1 system call read is used to read the contents. As I think the buffer size is just used for reading `OPTIONS` files, and I thought it likely that `OPTIONS` files have increased in size (as more options are added), I thought I would suggest an increase. [ If the comments from the top of the `OPTIONS` file are removed, and white space from the start of lines is removed then the size can be reduced to be under 4K, but as more options are added the size seems likely to grow again. ] Create a new database: ``` > ./ldb --create_if_missing --db=/tmp/rdb_tmp put 1 1 OK ``` The OPTIONS file is 4252 bytes: ``` > stat /tmp/rdb_tmp/OPTIONS* | head -n 2 File: ‘/tmp/rdb_tmp/OPTIONS-000005’ Size: 4252 Blocks: 16 IO Block: 4096 regular file ``` Before, the 4096 byte buffer is used from 2 system read calls: ``` > strace -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -A 1 'RocksDB option file' read(3, "# This is a RocksDB option file."..., 4096) = 4096 read(3, "e\n metadata_block_size=4096\n c"..., 4096) = 156 ``` ltrace shows 4096 passed to fread_unlocked ``` > ltrace -S -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -C 3 'RocksDB option file' [pid 51013] fread_unlocked(0x7ffd5fbf2d50, 1, 4096, 0x7fd2e084e780 <unfinished ...> [pid 51013] fstat@SYS(3, 0x7ffd5fbf28f0) = 0 [pid 51013] mmap@SYS(nil, 4096, 3, 34, -1, 0) = 0x7fd2e318c000 [pid 51013] read@SYS(3, "# This is a RocksDB option file."..., 4096) = 4096 [pid 51013] <... fread_unlocked resumed> ) = 4096 ... ``` After, the 8192 byte buffer is used from 1 system read call: ``` > strace -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -A 1 'RocksDB option file' read(3, "# This is a RocksDB option file."..., 8192) = 4252 read(3, "", 4096) = 0 ``` ltrace shows 8192 passed to fread_unlocked ``` > ltrace -S -f ./ldb --try_load_options --db=/tmp/rdb_tmp get DOES_NOT_EXIST 2>&1 | grep -C 3 'RocksDB option file' [pid 146611] fread_unlocked(0x7ffcfba382f0, 1, 8192, 0x7fc4e844e780 <unfinished ...> [pid 146611] fstat@SYS(3, 0x7ffcfba380f0) = 0 [pid 146611] mmap@SYS(nil, 4096, 3, 34, -1, 0) = 0x7fc4eaee0000 [pid 146611] read@SYS(3, "# This is a RocksDB option file."..., 8192) = 4252 [pid 146611] read@SYS(3, "", 4096) = 0 [pid 146611] <... fread_unlocked resumed> ) = 4252 [pid 146611] feof(0x7fc4e844e780) = 1 ``` Closes https://github.com/facebook/rocksdb/pull/3294 Differential Revision: D6653684 Pulled By: ajkr fbshipit-source-id: 222f25f5442fefe1dcec18c700bd9e235bb63491 | 12 January 2018, 02:57:41 UTC |
| 0a7ba0e | Changli Gao | 12 January 2018, 02:48:19 UTC | Fix memleak when DB::DeleteFile() Summary: Because the corresponding read_first_record_cache_ item wasn't erased, memory leaked. Closes https://github.com/facebook/rocksdb/pull/1712 Differential Revision: D4363654 Pulled By: ajkr fbshipit-source-id: 7da1adcfc8c380e4ffe05b8769fc2221ad17a225 | 12 January 2018, 02:57:33 UTC |
| 9c2f64e | Andrew Kryczka | 12 January 2018, 00:51:09 UTC | Update Gemfile.lock Summary: bump nokogiri number Closes https://github.com/facebook/rocksdb/pull/3358 Differential Revision: D6708596 Pulled By: ajkr fbshipit-source-id: 6662c3ba4994374ecf8a13928e915b655a980b70 | 12 January 2018, 00:57:25 UTC |
| 204af1e | Bo Liu | 11 January 2018, 23:35:21 UTC | add WriteBatch::WriteBatch(std::string&&) Summary: to save a string copy for some use cases. The change is pretty straightforward, please feel free to let me know if you want to suggest any tests for it. Closes https://github.com/facebook/rocksdb/pull/3349 Differential Revision: D6706828 Pulled By: yiwu-arbug fbshipit-source-id: 873ce4442937bdc030b395c7f99228eda7f59eb7 | 11 January 2018, 23:43:56 UTC |
| d4da02d | Adam Retter | 11 January 2018, 22:51:26 UTC | Add Jenkins for PPC64le build status badge Summary: Closes https://github.com/facebook/rocksdb/pull/3356 Differential Revision: D6706909 Pulled By: sagar0 fbshipit-source-id: 6e4757d9eceab3e8a6c1b83c1be4108e86576cb2 | 11 January 2018, 22:57:45 UTC |
| a53c571 | Adam Retter | 11 January 2018, 21:21:35 UTC | FreeBSD build support for RocksDB and RocksJava Summary: Tested on a clean FreeBSD 11.01 x64. Closes https://github.com/facebook/rocksdb/pull/1423 Closes https://github.com/facebook/rocksdb/pull/3357 Differential Revision: D6705868 Pulled By: sagar0 fbshipit-source-id: cbccbbdafd4f42922512ca03619a5d5583a425fd | 11 January 2018, 21:29:55 UTC |
| b010116 | Dmitri Smirnov | 11 January 2018, 01:03:12 UTC | Eliminate some redundant block reads. Summary: Re-use metadata for reading Compression Dictionary on BlockBased table open, this saves two reads from disk. This helps to our 999 percentile in 5.6.1 where prefetch buffer is not present. Closes https://github.com/facebook/rocksdb/pull/3354 Differential Revision: D6695753 Pulled By: ajkr fbshipit-source-id: bb8acd9e9e66e65b89c548ab8940570ae360333c | 11 January 2018, 01:11:58 UTC |
| 0c6e8be | Andrew Kryczka | 10 January 2018, 23:36:48 UTC | Fix directory name for db_basic_test Summary: It was using the same directory as `db_options_test` so transiently failed when unit tests were run in parallel. Closes https://github.com/facebook/rocksdb/pull/3352 Differential Revision: D6691649 Pulled By: ajkr fbshipit-source-id: bee433484fec4faedd5cadf2db3c92fdcc99a170 | 10 January 2018, 23:41:46 UTC |
| 1994051 | Anand Ananthabhotla | 10 January 2018, 23:06:29 UTC | Add a BlockBasedTableOption to turn off index block compression. Summary: Add a new bool option index_uncompressed in BlockBasedTableOptions. Closes https://github.com/facebook/rocksdb/pull/3303 Differential Revision: D6686161 Pulled By: anand1976 fbshipit-source-id: 748b46993d48a01e5f89b6bd3e41f06a59ec6054 | 10 January 2018, 23:11:59 UTC |
| bafec6b | Andrew Kryczka | 10 January 2018, 20:14:18 UTC | Fix checkpoint_test directory setup/cleanup Summary: - Change directory name from "db_test" to "checkpoint_test". Previously it used the same directory as `db_test` - Systematically cleanup snapshot and snapshot staging directories before each test. Previously a failed test run caused subsequent runs to fail, particularly when the first failure caused "snapshot.tmp" to not be cleaned up. Closes https://github.com/facebook/rocksdb/pull/3351 Differential Revision: D6691015 Pulled By: ajkr fbshipit-source-id: 4fc2ac2e21ff2617ea0e96297c5132b5f2eefd79 | 10 January 2018, 20:26:49 UTC |
| a478e85 | Siying Dong | 10 January 2018, 02:12:32 UTC | Remove GCC parameter "-march=native" for ARM Summary: Most popular versions of GCC can't identify platform on ARM if "-march=native" is specified. Remove it to unblock most people. Closes https://github.com/facebook/rocksdb/pull/3346 Differential Revision: D6690544 Pulled By: siying fbshipit-source-id: bbaba9fe2645b6b37144b36ea75beeff88992b49 | 10 January 2018, 02:27:03 UTC |
| 677f249 | Sagar Vemuri | 10 January 2018, 00:27:01 UTC | Fix Travis build failures in CMake RocksJava Summary: Fixed RocksJava travis build failure due to a missing file in java/CMakeLists.txt. (from #3332). Closes https://github.com/facebook/rocksdb/pull/3344 Differential Revision: D6686472 Pulled By: sagar0 fbshipit-source-id: dd3281dff1342c3a7235c402890420aa56db0fe3 | 10 January 2018, 00:42:15 UTC |
| 6aa95f4 | Siying Dong | 09 January 2018, 22:46:30 UTC | Fix a wrong log formatting Summary: I experienced weird segfault because of this mismatch of type in log formatting. Fix it. Closes https://github.com/facebook/rocksdb/pull/3345 Differential Revision: D6687224 Pulled By: siying fbshipit-source-id: c51fb1c008b7ebc3efdc353a4adad3e8f5b3e9de | 09 January 2018, 22:58:33 UTC |
| 0f0d2ab | Andrew Kryczka | 09 January 2018, 20:51:10 UTC | fix DBImpl instance variable naming Summary: got confused while reading `FindObsoleteFiles` due to thinking it's a local variable, so renamed it properly Closes https://github.com/facebook/rocksdb/pull/3342 Differential Revision: D6684797 Pulled By: ajkr fbshipit-source-id: a4df0aae1cccce99d4dd4d164aadc85b17707132 | 09 January 2018, 20:56:58 UTC |
| 46ec524 | Yi Wu | 09 January 2018, 18:53:35 UTC | Fix db_bench write being disabled in lite build Summary: The macro was added by mistake in #2372 Closes https://github.com/facebook/rocksdb/pull/3343 Differential Revision: D6681356 Pulled By: yiwu-arbug fbshipit-source-id: 4180172fb0eaef4189c07f219241e0c261c03461 | 09 January 2018, 18:57:29 UTC |
| 00b33c2 | Maysam Yabandeh | 09 January 2018, 16:47:46 UTC | WritePrepared Txn: address some pending TODOs Summary: This patch addresses a couple of minor TODOs for WritePrepared Txn such as double checking some assert statements at runtime as well, skip extra AddPrepared in non-2pc transactions, and safety check for infinite loops. Closes https://github.com/facebook/rocksdb/pull/3302 Differential Revision: D6617002 Pulled By: maysamyabandeh fbshipit-source-id: ef6673c139cb49f64c0879508d2f573b78609aca | 09 January 2018, 16:57:20 UTC |
| 24e2c16 | Chris Lu | 09 January 2018, 01:16:22 UTC | add support for allow_ingest_behind in C API Summary: https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files Need to expose these functions in the C API to be used by Go bindings. Closes https://github.com/facebook/rocksdb/pull/3011 Differential Revision: D6679563 Pulled By: sagar0 fbshipit-source-id: 536f844ddaeb0172c6d7e416d2a75e8f9e57c8ef | 09 January 2018, 01:26:31 UTC |
| 398d72f | Adam Retter | 08 January 2018, 20:20:48 UTC | Add autotune and #getBytesPerSecond() to RocksJava RateLimiter Summary: Closes https://github.com/facebook/rocksdb/pull/3332 Differential Revision: D6667680 Pulled By: ajkr fbshipit-source-id: b2bb6889257850a4eb6f6cbd7106f62df7b82730 | 08 January 2018, 20:30:52 UTC |
| 30a017f | Yi Wu | 06 January 2018, 00:35:49 UTC | Blob DB: avoid having a separate read of checksum Summary: Previously on a blob db read, we are making a read of the blob value, and then make another read to get CRC checksum. I'm combining the two read into one. readrandom db_bench with 1G database with base db size of 13M, value size 1k: `./db_bench --db=/home/yiwu/tmp/db_bench --use_blob_db --value_size=1024 --num=1000000 --benchmarks=readrandom --use_existing_db --cache_size=32000000` master: throughput 234MB/s, get micros p50 5.984 p95 9.998 p99 20.817 p100 787 this PR: throughput 261MB/s, get micros p50 5.157 p95 9.928 p99 20.724 p100 190 Closes https://github.com/facebook/rocksdb/pull/3301 Differential Revision: D6615950 Pulled By: yiwu-arbug fbshipit-source-id: 052410c6d8539ec0cc305d53793bbc8f3616baa3 | 06 January 2018, 00:41:58 UTC |
| 3e955fa | Sagar Vemuri | 05 January 2018, 23:30:12 UTC | Fix zstd/zdict include path for java static build Summary: With the ZSTD dictionary generator support added in #3057 `PORTABLE=1 ROCKSDB_NO_FBCODE=1 make rocksdbjavastatic` fails as it can't find zdict.h. Specifically due to: https://github.com/facebook/rocksdb/blob/e3a06f12d27fd50af7b6c5941973f529601f9a3e/util/compression.h#L39 In java static builds zstd code gets directly downloaded from https://github.com/facebook/zstd , and in there zdict.h is under dictBuilder directory. So, I modified libzstd.a target to use `make install` to collect all the header files into a single location and used that as the zstd's include path. Closes https://github.com/facebook/rocksdb/pull/3260 Differential Revision: D6669850 Pulled By: sagar0 fbshipit-source-id: f8a7562a670e5aed4c4fb6034a921697590d7285 | 05 January 2018, 23:41:46 UTC |
| 84ddbd1 | Dmitri Smirnov | 05 January 2018, 22:52:57 UTC | Make Windows dep switches compatible with other builds Summary: Make dependacies switches compatible with other OS builds TODO: Make find_package work for Windows. Closes https://github.com/facebook/rocksdb/pull/3322 Differential Revision: D6667637 Pulled By: sagar0 fbshipit-source-id: 5afcd7bbfe69465310a4fbc8e589f01e506b95f5 | 05 January 2018, 22:56:54 UTC |
| 90c1d81 | Adam Retter | 05 January 2018, 19:59:01 UTC | Update javastatic dependencies Summary: 1. Snappy 1.1.7 2. LZ4 1.8.0 3. ZSTD 1.3.3 Closes https://github.com/facebook/rocksdb/pull/3331 Differential Revision: D6667933 Pulled By: ajkr fbshipit-source-id: 21c526609df7580481195a389d31f733e2695e65 | 05 January 2018, 20:11:44 UTC |
| 1c9ada5 | Maysam Yabandeh | 04 January 2018, 19:10:03 UTC | Remove assert(s.ok()) from ::DeleteFile Summary: DestroyDB that is used in tests loops over the files returned by ::GetChildren and delete them one by one. Such files might be already deleted in the file system (during DeleteObsoleteFileImpl for example) but will get actually deleted with a delay sometimes before ::DeleteFile is called on the file name. We have some test failures where FaultInjectionTestEnv::DeleteFile fails on assert(s.ok()) during DestroyDB. This patch removes the assert statement to fix that. Closes https://github.com/facebook/rocksdb/pull/3324 Differential Revision: D6659545 Pulled By: maysamyabandeh fbshipit-source-id: 4c9552fbcd494dcf3e61d475c11fc965c4388b2c | 04 January 2018, 19:11:45 UTC |
| ea8ccd2 | Andrew Kryczka | 03 January 2018, 20:37:07 UTC | fix powerpc java static build Summary: added support for C and asm files as required for e612e317409e8a9d74cf05db0bd733403305f459. Closes https://github.com/facebook/rocksdb/pull/3299 Differential Revision: D6612479 Pulled By: ajkr fbshipit-source-id: 6263ed7c1602f249460421825c76b5721f396163 | 03 January 2018, 20:41:37 UTC |
| 00e5e1e | Adam Retter | 03 January 2018, 20:22:40 UTC | Fixes the build on Windows Summary: As discovered during v5.9.2 release, and forward-ported. Closes https://github.com/facebook/rocksdb/pull/3323 Differential Revision: D6657209 Pulled By: siying fbshipit-source-id: b560d9f8ddb89e0ffaff7c895ec80f68ddf7dab4 | 03 January 2018, 20:27:12 UTC |
| ccc095a | Siying Dong | 02 January 2018, 18:34:51 UTC | Speed up BlockTest.BlockReadAmpBitmap Summary: BlockTest.BlockReadAmpBitmap is too slow and times out in some environments. Speed it up by: (1) improve the way the verification is done. With this it is 5 times faster (2) run fewer tests for large blocks. This cut it down by another 10 times. Now it can finish in similar time as other tests. Closes https://github.com/facebook/rocksdb/pull/3313 Differential Revision: D6643711 Pulled By: siying fbshipit-source-id: c2397d666eab5421a78ca87e1e45491e0f832a6d | 02 January 2018, 18:41:28 UTC |
| b5c99cc | burtonli | 22 December 2017, 02:37:27 UTC | Disable onboard cache for compaction output Summary: FILE_FLAG_WRITE_THROUGH is for disabling device on-board cache in windows API, which should be disabled if user doesn't need system cache. There was a perf issue related with this, we found during memtable flush, the high percentile latency jumps significantly. During profiling, we found those high latency (P99.9) read requests got queue-jumped by write requests from memtable flush and takes 80ms or even more time to wait, even when SSD overall IO throughput is relatively low. After enabling FILE_FLAG_WRITE_THROUGH, we rerun the test found high percentile latency drops a lot without observable impact on writes. Scenario 1: 40MB/s + 40MB/s R/W compaction throughput Original | FILE_FLAG_WRITE_THROUGH | Percentage reduction --------------------------------------------------------------- P99.9 | 56.897 ms | 35.593 ms | -37.4% P99 | 3.905 ms | 3.896 ms | -2.8% Scenario 2: 14MB/s + 14MB/s R/W compaction throughput, cohosted with 100+ other rocksdb instances have manually triggered memtable flush operations (memtable is tiny), creating a lot of randomized the small file writes operations during test. Original | FILE_FLAG_WRITE_THROUGH | Percentage reduction --------------------------------------------------------------- P99.9 | 86.227 ms | 50.436 ms | -41.5% P99 | 8.415 ms | 3.356 ms | -60.1% Closes https://github.com/facebook/rocksdb/pull/3225 Differential Revision: D6624174 Pulled By: miasantreble fbshipit-source-id: 321b86aee9d74470840c70e5d0d4fa9880660a91 | 22 December 2017, 02:41:34 UTC |
| f00e176 | Andrew Kryczka | 21 December 2017, 00:10:03 UTC | fix ForwardIterator reference to temporary object Summary: Fixes the following ASAN error: ``` ==2108042==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7fc50ae9b868 at pc 0x7fc5112aff55 bp 0x7fff9eb9dc10 sp 0x7fff9eb9dc08 === How to use this, how to get the raw stack trace, and more: fburl.com/ASAN === READ of size 8 at 0x7fc50ae9b868 thread T0 SCARINESS: 23 (8-byte-read-stack-use-after-scope) #0 rocksdb/dbformat.h:164 rocksdb::InternalKeyComparator::user_comparator() const #1 librocksdb_src_rocksdb_lib.so+0x1429a7d rocksdb::RangeDelAggregator::InitRep(std::vector<...> const&) #2 librocksdb_src_rocksdb_lib.so+0x142ceae rocksdb::RangeDelAggregator::AddTombstones(std::unique_ptr<...>) #3 librocksdb_src_rocksdb_lib.so+0x1382d88 rocksdb::ForwardIterator::RebuildIterators(bool) #4 librocksdb_src_rocksdb_lib.so+0x1382362 rocksdb::ForwardIterator::ForwardIterator(rocksdb::DBImpl*, rocksdb::ReadOptions const&, rocksdb::ColumnFamilyData*, rocksdb::SuperVersion*) #5 librocksdb_src_rocksdb_lib.so+0x11f433f rocksdb::DBImpl::NewIterator(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*) #6 rocksdb/src/include/rocksdb/db.h:382 rocksdb::DB::NewIterator(rocksdb::ReadOptions const&) #7 rocksdb/db_range_del_test.cc:807 rocksdb::DBRangeDelTest_TailingIteratorRangeTombstoneUnsupported_Test::TestBody() #18 rocksdb/db_range_del_test.cc:1006 main Address 0x7fc50ae9b868 is located in stack of thread T0 at offset 104 in frame #0 librocksdb_src_rocksdb_lib.so+0x13825af rocksdb::ForwardIterator::RebuildIterators(bool) ``` Closes https://github.com/facebook/rocksdb/pull/3300 Differential Revision: D6612989 Pulled By: ajkr fbshipit-source-id: e7ea2ed914c1b80a8a29d71d92440a6bd9cbcc80 | 21 December 2017, 00:12:04 UTC |
| 02a2c11 | Maysam Yabandeh | 20 December 2017, 19:31:39 UTC | Blog post for WritePrepared Txn Summary: Blog post to introduce the next generation of transaction engine at RocksDB. Closes https://github.com/facebook/rocksdb/pull/3296 Differential Revision: D6612932 Pulled By: maysamyabandeh fbshipit-source-id: 5bfa91ce84e937f5e4346bbda5a4725d0a7fd131 | 20 December 2017, 19:42:15 UTC |
| 0ef3fdd | Maysam Yabandeh | 20 December 2017, 16:10:19 UTC | Disable need_log_sync on bg err Summary: When there is a background error PreprocessWrite returns without marking the logs synced. If we keep need_log_sync to true, it would try to sync them at the end, which would break the logic. The patch would unset need_log_sync if the logs end up not being marked for sync in PreprocessWrite. Closes https://github.com/facebook/rocksdb/pull/3293 Differential Revision: D6602347 Pulled By: maysamyabandeh fbshipit-source-id: 37ee04209e8dcfd78de891654ce50d0954abeb38 | 20 December 2017, 16:12:24 UTC |
| 58b841b | Wouter Beek | 20 December 2017, 16:02:53 UTC | FIXED: string buffers potentially too small to fit formatted write Summary: This fixes the following warnings when compiled with GCC7: util/transaction_test_util.cc: In static member function ‘static rocksdb::Status rocksdb::RandomTransactionInserter::DBGet(rocksdb::DB*, rocksdb::Transaction*, rocksdb::ReadOptions&, uint16_t, uint64_t, bool, uint64_t*, std::__cxx11::string*, bool*)’: util/transaction_test_util.cc:75:8: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=] Status RandomTransactionInserter::DBGet( ^~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc:84:11: note: ‘snprintf’ output between 5 and 6 bytes into a destination of size 5 snprintf(prefix_buf, sizeof(prefix_buf), "%.4u", set_i + 1); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc: In static member function ‘static rocksdb::Status rocksdb::RandomTransactionInserter::Verify(rocksdb::DB*, uint16_t, uint64_t, bool, rocksdb::Random64*)’: util/transaction_test_util.cc:245:8: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=] Status RandomTransactionInserter::Verify(DB* db, uint16_t num_sets, ^~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc:268:13: note: ‘snprintf’ output between 5 and 6 bytes into a destination of size 5 snprintf(prefix_buf, sizeof(prefix_buf), "%.4u", set_i + 1); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Closes https://github.com/facebook/rocksdb/pull/3295 Differential Revision: D6609411 Pulled By: maysamyabandeh fbshipit-source-id: 33f0add471056eb59db2f8bd4366e6dfbb1a187d | 20 December 2017, 16:12:22 UTC |
| f54d7f5 | yingsu00 | 20 December 2017, 02:20:50 UTC | Port 3 way SSE4.2 crc32c implementation from Folly Summary: **# Summary** RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain. This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used. **# Performance Test Results** We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm. Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s. 1) ReadRandom in db_bench overall metrics PER RUN Algorithm | run | micros/op | ops/sec |Throughput (MB/s) 3-way | 1 | 4.143 | 241387 | 26.7 3-way | 2 | 3.775 | 264872 | 29.3 3-way | 3 | 4.116 | 242929 | 26.9 FastCrc32c|1 | 4.037 | 247727 | 27.4 FastCrc32c|2 | 4.648 | 215166 | 23.8 FastCrc32c|3 | 4.352 | 229799 | 25.4 AVG Algorithm | Average of micros/op | Average of ops/sec | Average of Throughput (MB/s) 3-way | 4.01 | 249,729 | 27.63 FastCrc32c | 4.35 | 230,897 | 25.53 2) Crc32c computation CPU cost (inclusive samples percentage) PER RUN Implementation | run | TotalSamples | Crc32c percentage 3-way | 1 | 4,572,250,000 | 4.37% 3-way | 2 | 3,779,250,000 | 4.62% 3-way | 3 | 4,129,500,000 | 4.48% FastCrc32c | 1 | 4,663,500,000 | 11.24% FastCrc32c | 2 | 4,047,500,000 | 12.34% FastCrc32c | 3 | 4,366,750,000 | 11.68% **# Test Plan** make -j64 corruption_test && ./corruption_test By default it uses 3-way SSE algorithm NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test make clean && DEBUG_LEVEL=0 make -j64 db_bench make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench Closes https://github.com/facebook/rocksdb/pull/3173 Differential Revision: D6330882 Pulled By: yingsu00 fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9 | 20 December 2017, 02:26:49 UTC |
| e763e1b | Yi Wu | 20 December 2017, 00:50:59 UTC | BlobDB: dump blob db options on open Summary: We dump blob db options on blob db open, but it was removed by mistake in #3246. Adding it back. Closes https://github.com/facebook/rocksdb/pull/3298 Differential Revision: D6607177 Pulled By: yiwu-arbug fbshipit-source-id: 2a4aacbfa52fd8f1878dc9e1fbb95fe48faf80c0 | 20 December 2017, 00:57:12 UTC |
| 48cf8da | Yi Wu | 20 December 2017, 00:34:25 UTC | BlobDB: update blob_db_options.bytes_per_sync behavior Summary: Previously, if blob_db_options.bytes_per_sync, there is a background job to call fsync() for every bytes_per_sync bytes written to a blob file. With the change we simply pass bytes_per_sync as env_options_ to blob files so that sync_file_range() will be used instead. Closes https://github.com/facebook/rocksdb/pull/3297 Differential Revision: D6606994 Pulled By: yiwu-arbug fbshipit-source-id: 452424be52e32ba92f5ea603b564e9b88929af47 | 20 December 2017, 00:41:41 UTC |
| 0614942 | Yi Wu | 19 December 2017, 06:13:08 UTC | WritePrepared Txn: Return NotSupported on iterator refresh Summary: A proper implementation of Iterator::Refresh() for WritePreparedTxnDB would require release and acquire another snapshot. Since MyRocks don't make use of Iterator::Refresh(), we just simply mark it as not supported. Closes https://github.com/facebook/rocksdb/pull/3290 Differential Revision: D6599931 Pulled By: yiwu-arbug fbshipit-source-id: 4e1632d967316431424f6e458254ecf9a97567cf | 19 December 2017, 06:29:30 UTC |
| 1563801 | Andrew Kryczka | 19 December 2017, 01:48:49 UTC | blog post for auto-tuned rate limiter Summary: Wrote the blog post. Closes https://github.com/facebook/rocksdb/pull/3289 Differential Revision: D6599031 Pulled By: ajkr fbshipit-source-id: 77ee553196f225f20c56112d2c015b6fa14f1b83 | 19 December 2017, 01:56:50 UTC |
| 2190e96 | Yi Wu | 19 December 2017, 01:48:41 UTC | Remove incorrect comment Summary: We actually create individual compaction filter from compaction filter factory per sub-compaction in `CompactionJob::ProcessKeyValueCompaction`: https://github.com/facebook/rocksdb/blob/master/db/compaction_job.cc#L742 The comment seems incorrect. Closes https://github.com/facebook/rocksdb/pull/3288 Differential Revision: D6598455 Pulled By: yiwu-arbug fbshipit-source-id: a6bc059a9103b87a73ae6ec4bb01ca33f5d48cf5 | 19 December 2017, 01:56:47 UTC |
| 0faa026 | Maysam Yabandeh | 18 December 2017, 22:39:49 UTC | WritePrepared Txn: make buck tests parallel Summary: The TSAN version of tests could take quite long. Make the buck tests parallel to avoid timeouts. Closes https://github.com/facebook/rocksdb/pull/3280 Differential Revision: D6581594 Pulled By: maysamyabandeh fbshipit-source-id: 3f8476d8c69f0183e394fa8a2089dd8d4e90c90c | 18 December 2017, 22:42:09 UTC |
| 78c2eed | Maysam Yabandeh | 18 December 2017, 22:13:45 UTC | fix release order in validateNumberOfEntries Summary: ScopedArenaIterator should be defined after range_del_agg so that it destructs the assigned iterator, which depends on range_del_agg, before it range_del_agg is already destructed. Closes https://github.com/facebook/rocksdb/pull/3281 Differential Revision: D6592332 Pulled By: maysamyabandeh fbshipit-source-id: 89a15d8ed13d0fc856b0c47dce3d91778738dbac | 18 December 2017, 22:27:28 UTC |
| aa6509d | Guo Xiao | 18 December 2017, 20:23:40 UTC | Fix build for linux Summary: * Include `unistd.h` for `sleep(3)` * Include `sys/time.h` for `gettimeofday(3)` * Include `utils/random.h` for `Random64` Error messages: utilities/persistent_cache/hash_table_bench.cc: In constructor ‘rocksdb::HashTableBenchmark::HashTableBenchmark(rocksdb::HashTableImpl<long unsigned int, std::__cxx11::basic_string<char> >*, size_t, size_t, size_t, size_t)’: utilities/persistent_cache/hash_table_bench.cc:76:28: error: ‘sleep’ was not declared in this scope /* sleep override */ sleep(1); ^~~~~ utilities/persistent_cache/hash_table_bench.cc:76:28: note: suggested alternative: ‘strsep’ /* sleep override */ sleep(1); ^~~~~ strsep utilities/persistent_cache/hash_table_bench.cc: In member function ‘void rocksdb::HashTableBenchmark::RunRead()’: utilities/persistent_cache/hash_table_bench.cc:107:5: error: ‘Random64’ was not declared in this scope Random64 rgen(time(nullptr)); ^~~~~~~~ utilities/persistent_cache/hash_table_bench.cc:107:5: note: suggested alternative: ‘random_r’ Random64 rgen(time(nullptr)); ^~~~~~~~ random_r utilities/persistent_cache/hash_table_bench.cc:110:18: error: ‘rgen’ was not declared in this scope size_t k = rgen.Next() % max_prepop_key; ^~~~ utilities/persistent_cache/hash_table_bench.cc: In static member function ‘static uint64_t rocksdb::HashTableBenchmark::NowInMillSec()’: utilities/persistent_cache/hash_table_bench.cc:153:5: error: ‘gettimeofday’ was not declared in this scope gettimeofday(&tv, /*tz=*/nullptr); ^~~~~~~~~~~~ make[2]: *** [CMakeFiles/hash_table_bench.dir/build.make:63: CMakeFiles/hash_table_bench.dir/utilities/persistent_cache/hash_table_bench.cc.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:3346: CMakeFiles/hash_table_bench.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... Closes https://github.com/facebook/rocksdb/pull/3283 Differential Revision: D6594850 Pulled By: ajkr fbshipit-source-id: fd83957338c210cdfd253763347aafd39476824f | 18 December 2017, 20:28:03 UTC |
| a6d3c76 | Maysam Yabandeh | 18 December 2017, 16:03:18 UTC | WritePrepared Txn: non-2pc write in one round Summary: Currently non-2pc writes do the 2nd dummy write to actually commit the transaction. This was necessary to ensure that publishing the commit sequence number will be done only from one queue (the queue that does not write to memtable). This is however not necessary when we have only one write queue, which is actually the setup that would be used by non-2pc writes. This patch eliminates the 2nd write when two_write_queues are disabled by updating the commit map in the 1st write. Closes https://github.com/facebook/rocksdb/pull/3277 Differential Revision: D6575392 Pulled By: maysamyabandeh fbshipit-source-id: 8ab458f7ca506905962f9166026b2ec81e749c46 | 18 December 2017, 16:19:43 UTC |
| fccc12f | Anand Ananthabhotla | 16 December 2017, 02:45:38 UTC | Add a histogram stat for memtable flush Summary: Add a new histogram stat called rocksdb.db.flush.micros for memtable flush Closes https://github.com/facebook/rocksdb/pull/3269 Differential Revision: D6559496 Pulled By: anand1976 fbshipit-source-id: f5c771ba2568630458751795e8c37a493ff9b14d | 16 December 2017, 02:57:00 UTC |
| 95583e1 | Maysam Yabandeh | 16 December 2017, 00:15:58 UTC | db_stress: skip snapshot check if cf is dropped Summary: We added a new verification that ensures a value that snapshot reads when is released is the same as when it was created. This test however fails when the cf is dropped in between. The patch skips the tests if that was the case. Closes https://github.com/facebook/rocksdb/pull/3279 Differential Revision: D6581584 Pulled By: maysamyabandeh fbshipit-source-id: afe37d371c0f91818d2e279b3949b810e112e8eb | 16 December 2017, 00:28:04 UTC |
| 237b292 | Yi Wu | 15 December 2017, 21:18:32 UTC | BlobDB: Remove the need to get sequence number per write Summary: Previously we store sequence number range of each blob files, and use the sequence number range to check if the file can be possibly visible by a snapshot. But it adds complexity to the code, since the sequence number is only available after a write. (The current implementation get sequence number by calling GetLatestSequenceNumber(), which is wrong.) With the patch, we are not storing sequence number range, and check if snapshot_sequence < obsolete_sequence to decide if the file is visible by a snapshot (previously we check if first_sequence <= snapshot_sequence < obsolete_sequence). Closes https://github.com/facebook/rocksdb/pull/3274 Differential Revision: D6571497 Pulled By: yiwu-arbug fbshipit-source-id: ca06479dc1fcd8782f6525b62b7762cd47d61909 | 15 December 2017, 21:27:30 UTC |
| a79c7c0 | Andrew Kryczka | 15 December 2017, 20:22:13 UTC | fix backup meta-file buffer overrun Summary: - check most times after calling snprintf that the buffer didn't fill up. Previously we'd proceed and use `buf_size - len` as the length in subsequent calls, which underflowed as those are unsigned size_t. - replace some memcpys with snprintf for consistency Closes https://github.com/facebook/rocksdb/pull/3255 Differential Revision: D6541464 Pulled By: ajkr fbshipit-source-id: 8610ea6a24f38e0a37c6d17bc65b7c712da6d932 | 15 December 2017, 20:29:16 UTC |
| 5a7e084 | Andrew Kryczka | 14 December 2017, 22:41:59 UTC | fix ThreadStatus for bottom-pri compaction threads Summary: added `ThreadType::BOTTOM_PRIORITY` which is used in the `ThreadStatus` object to indicate the thread is used for bottom-pri compactions. Previously there was a bug where we mislabeled such threads as `ThreadType::LOW_PRIORITY`. Closes https://github.com/facebook/rocksdb/pull/3270 Differential Revision: D6559428 Pulled By: ajkr fbshipit-source-id: 96b1a50a9c19492b1a5fd1b77cf7061a6f9f1d1c | 14 December 2017, 22:57:49 UTC |
| b4d88d7 | Orvid King | 14 December 2017, 19:57:11 UTC | Fix the build with MSVC 2017 Summary: There were a few places where MSVC's implicit truncation warnings were getting triggered, which was causing the MSVC build to fail due to warnings being treated as errors. This resolves the issues by making the truncations in some places explicit, and by making it so there are no truncations of literals. Fixes #3239 Supersedes #3259 Closes https://github.com/facebook/rocksdb/pull/3273 Reviewed By: yiwu-arbug Differential Revision: D6569204 Pulled By: Orvid fbshipit-source-id: c188cf1cf98d9acb6d94b71875041cc81f8ff088 | 14 December 2017, 20:02:22 UTC |
| def6a00 | Siying Dong | 14 December 2017, 18:18:01 UTC | Print out compression type of new SST files in logging Summary: Closes https://github.com/facebook/rocksdb/pull/3264 Differential Revision: D6552768 Pulled By: siying fbshipit-source-id: 6303110aff22f341d5cff41f8d2d4f138a53652d | 14 December 2017, 18:27:43 UTC |
| 6b77c07 | Siying Dong | 14 December 2017, 18:17:22 UTC | NUMBER_BLOCK_COMPRESSED, etc, shouldn't be treated as timer counter Summary: NUMBER_BLOCK_DECOMPRESSED and NUMBER_BLOCK_COMPRESSED are not reported unless the stats level contain detailed timers, which is wrong. They are normal counters. Fix it. Closes https://github.com/facebook/rocksdb/pull/3263 Differential Revision: D6552519 Pulled By: siying fbshipit-source-id: 40899ccea7b2856bb39752616657c0bfd432f6f9 | 14 December 2017, 18:27:43 UTC |
| cd2e5ca | Maysam Yabandeh | 13 December 2017, 19:51:12 UTC | WritePrepared Txn: make db_stress transactional Summary: Add "--use_txn" option to use transactional API in db_stress, default being WRITE_PREPARED policy, which is the main intention of modifying db_stress. It also extend the existing snapshots to verify that before releasing a snapshot a read from it returns the same value as before. Closes https://github.com/facebook/rocksdb/pull/3243 Differential Revision: D6556912 Pulled By: maysamyabandeh fbshipit-source-id: 1ae31465be362d44bd06e635e2e9e49a1da11268 | 13 December 2017, 19:57:29 UTC |
| 546a632 | Maysam Yabandeh | 13 December 2017, 15:55:02 UTC | disableWAL with WriteImplWALOnly Summary: Currently WriteImplWALOnly simply returns when disableWAL is set. This is an incorrect behavior since it does not allocated the sequence number, which is a side-effect of writing to the WAL. This patch fixes the issue. Closes https://github.com/facebook/rocksdb/pull/3262 Differential Revision: D6550974 Pulled By: maysamyabandeh fbshipit-source-id: 745a83ae8f04e7ca6c8ffb247d6ef16c287c52e7 | 13 December 2017, 15:57:44 UTC |
| 35dfbd5 | Maysam Yabandeh | 13 December 2017, 15:52:02 UTC | WritePrepared Txn: GC old_commit_map_ Summary: Garbage collect entries from old_commit_map_ when the corresponding snapshots are released. Closes https://github.com/facebook/rocksdb/pull/3247 Differential Revision: D6528478 Pulled By: maysamyabandeh fbshipit-source-id: 15d1566d85d4ac07036bc0dc47418f6c3228d4bf | 13 December 2017, 15:57:43 UTC |
| 51c2ea0 | Zhongyi Xie | 13 December 2017, 05:06:26 UTC | Reduce heavy hitter for Get operation Summary: This PR addresses the following heavy hitters in `Get` operation by moving calls to `StatisticsImpl::recordTick` from `BlockBasedTable` to `Version::Get` - rocksdb.block.cache.bytes.write - rocksdb.block.cache.add - rocksdb.block.cache.data.miss - rocksdb.block.cache.data.bytes.insert - rocksdb.block.cache.data.add - rocksdb.block.cache.hit - rocksdb.block.cache.data.hit - rocksdb.block.cache.bytes.read The db_bench statistics before and after the change are: |1GB block read|Children |Self |Command |Shared Object |Symbol| |---|---|---|---|---|---| |master: |4.22% |1.31% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| |updated: |0.51% |0.21% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| | |0.14% |0.14% |db_bench |db_bench |[.] rocksdb::GetContext::record_counters| |1MB block read|Children |Self |Command |Shared Object |Symbol| |---|---|---|---|---|---| |master: |3.48% |1.08% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| |updated: |0.80% |0.31% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| | |0.35% |0.35% |db_bench |db_bench |[.] rocksdb::GetContext::record_counters| Closes https://github.com/facebook/rocksdb/pull/3172 Differential Revision: D6330532 Pulled By: miasantreble fbshipit-source-id: 2b492959e00a3db29e9437ecdcc5e48ca4ec5741 | 13 December 2017, 05:11:33 UTC |
| 9089373 | Islam AbdelRahman | 13 December 2017, 02:15:15 UTC | Fix DeleteScheduler::MarkAsTrash() handling existing trash Summary: DeleteScheduler::MarkAsTrash() don't handle existing .trash files correctly This cause rocksdb to not being able to delete existing .trash files on restart Closes https://github.com/facebook/rocksdb/pull/3261 Differential Revision: D6548003 Pulled By: IslamAbdelRahman fbshipit-source-id: c3800639412e587a690062c63076a5a08881e0e6 | 13 December 2017, 02:17:13 UTC |
| 7393ef7 | Yi Wu | 12 December 2017, 20:08:52 UTC | Fix BlockFetcher ASAN error Summary: Some call sites of BlockFetcher create temporary ReadOptions and pass to BlockFetcher. The temporary object will be gone after BlockFetcher construction but BlockFetcher keep its reference, causing stack-use-after-scope. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3258 Differential Revision: D6547152 Pulled By: yiwu-arbug fbshipit-source-id: 6b49e9dd46bb72307f5d8f88ea15faacff35b9bc | 12 December 2017, 20:12:38 UTC |
| 4bcb7fb | Souvik Banerjee | 12 December 2017, 20:07:35 UTC | Update transaction_test_util.cc Summary: Fixes a compile error on gcc 7.2.1 (-Werror=format-truncation=). Closes https://github.com/facebook/rocksdb/pull/3248 Differential Revision: D6546515 Pulled By: yiwu-arbug fbshipit-source-id: bd78cca63f2af376faceccb1838d2d4cc9208fef | 12 December 2017, 20:12:38 UTC |
| e3a06f1 | Yi Wu | 12 December 2017, 19:02:07 UTC | WritePrepared Txn: fix compaction filter snapshot checks Summary: Add snapshot_checker check whenever we need to check sequence against snapshots and decide what to do with an input key. The changes are related to one of: * compaction filter * single delete * delete at bottom level * merge Closes https://github.com/facebook/rocksdb/pull/3251 Differential Revision: D6537850 Pulled By: yiwu-arbug fbshipit-source-id: 3faba40ed5e37779f4a0cb7ae78af9546659c7f2 | 12 December 2017, 19:12:24 UTC |
| a9c8d4e | Siying Dong | 12 December 2017, 02:14:12 UTC | Fix memory issue introduced by 2f1a3a4d748ea92c282a1302b1523adc6d67ce81 Summary: Closes https://github.com/facebook/rocksdb/pull/3256 Differential Revision: D6541714 Pulled By: siying fbshipit-source-id: 40efd89b68587a9d58cfe6f4eebd771c2d9f1542 | 12 December 2017, 02:27:28 UTC |
