| 005c34f | sdong | 18 June 2018, 16:54:06 UTC | Bump up to 5.12.5 | 18 June 2018, 16:55:01 UTC |
| 843738d | sdong | 12 June 2018, 23:59:46 UTC | Fix regression bug of Prev() with upper bound (#3989) Summary: A recent change pushed down the upper bound checking to child iterators. However, this causes the logic of following sequence wrong: Seek(key); if (!Valid()) SeekToLast(); Because !Valid() may be caused by upper bounds, rather than the end of the iterator. In this case SeekToLast() points to totally wrong places. This can cause wrong results, infinite loops, or segfault in some cases. This sequence is called when changing direction from forward to backward. And this by itself also implicitly happen during reseeking optimization in Prev(). Fix this bug by using SeekForPrev() rather than this sequence, as what is already done in prefix extrator case. Closes https://github.com/facebook/rocksdb/pull/3989 Differential Revision: D8385422 Pulled By: siying fbshipit-source-id: 429e869990cfd2dc389421e0836fc496bed67bb4 | 18 June 2018, 16:50:10 UTC |
| cc0a65f | Zhongyi Xie | 20 April 2018, 21:16:35 UTC | bump version to 5.12.4 and update history | 20 April 2018, 21:16:35 UTC |
| 4837609 | Zhongyi Xie | 16 April 2018, 00:19:57 UTC | fix memory leak in two_level_iterator Summary: this PR fixes a few failed contbuild: 1. ASAN memory leak in Block::NewIterator (table/block.cc:429). the proper destruction of first_level_iter_ and second_level_iter_ of two_level_iterator.cc is missing from the code after the refactoring in https://github.com/facebook/rocksdb/pull/3406 2. various unused param errors introduced by https://github.com/facebook/rocksdb/pull/3662 3. updated comment for `ForceReleaseCachedEntry` to emphasize the use of `force_erase` flag. Closes https://github.com/facebook/rocksdb/pull/3718 Reviewed By: maysamyabandeh Differential Revision: D7621192 Pulled By: miasantreble fbshipit-source-id: 476c94264083a0730ded957c29de7807e4f5b146 | 20 April 2018, 21:14:51 UTC |
| 48171df | Maysam Yabandeh | 10 April 2018, 02:37:11 UTC | update history | 10 April 2018, 02:37:11 UTC |
| d4f3052 | Maysam Yabandeh | 09 April 2018, 23:55:11 UTC | bump patch number to 5.12.3 | 09 April 2018, 23:55:11 UTC |
| c87e28d | Maysam Yabandeh | 09 April 2018, 23:17:15 UTC | Fix the memory leak with pinned partitioned filters Summary: The existing unit test did not set the level so the check for pinned partitioned filter/index being properly released from the block cache was not properly exercised as they only take effect in level 0. As a result a memory leak in pinned partitioned filters was hidden. The patch fix the test as well as the bug. Closes https://github.com/facebook/rocksdb/pull/3692 Differential Revision: D7559763 Pulled By: maysamyabandeh fbshipit-source-id: 55eff274945838af983c764a7d71e8daff092e4a | 09 April 2018, 23:53:16 UTC |
| 8969445 | Sagar Vemuri | 23 March 2018, 18:39:50 UTC | Update history and bump patch number | 23 March 2018, 18:39:50 UTC |
| 250d2a6 | Sagar Vemuri | 22 March 2018, 01:31:21 UTC | Shared block cache in RocksJava Summary: Changes to support sharing block cache using the Java API. Previously DB instances could share the block cache only when the same Options instance is passed to all the DB instances. But now, with this change, it is possible to explicitly create a cache and pass it to multiple options instances, to share the block cache. Implementing this for [Rocksandra](https://github.com/instagram/cassandra/tree/rocks_3.0), but this feature has been requested by many java api users over the years. Closes https://github.com/facebook/rocksdb/pull/3623 Differential Revision: D7305794 Pulled By: sagar0 fbshipit-source-id: 03e4e8ed7aeee6f88bada4a8365d4279ede2ad71 | 23 March 2018, 18:33:26 UTC |
| 68d6989 | Sagar Vemuri | 23 March 2018, 00:34:52 UTC | Fsync after writing global seq number in ExternalSstFileIngestionJob Summary: Fsync after writing global sequence number to the ingestion file in ExternalSstFileIngestionJob. Otherwise the file metadata could be incorrect. Closes https://github.com/facebook/rocksdb/pull/3644 Differential Revision: D7373813 Pulled By: sagar0 fbshipit-source-id: 4da2c9e71a8beb5c08b4ac955f288ee1576358b8 | 23 March 2018, 18:15:55 UTC |
| b4fc156 | Andrew Kryczka | 16 March 2018, 21:09:20 UTC | update history and bump patch number | 16 March 2018, 21:09:20 UTC |
| 34ccab0 | Andrew Kryczka | 16 March 2018, 21:08:01 UTC | Fix WAL corruption from checkpoint/backup race condition Summary: `Writer::WriteBuffer` was always called at the beginning of checkpoint/backup. But that log writer has no internal synchronization, which meant the same buffer could be flushed twice in a race condition case, causing a WAL entry to be duplicated. Then subsequent WAL entries would be at unexpected offsets, causing the 32KB block boundaries to be overlapped and manifesting as a corruption. This PR fixes the behavior to only use `WriteBuffer` (via `FlushWAL`) in checkpoint/backup when manual WAL flush is enabled. In that case, users are responsible for providing synchronization between WAL flushes. We can also consider removing the call entirely. Closes https://github.com/facebook/rocksdb/pull/3603 Differential Revision: D7277447 Pulled By: ajkr fbshipit-source-id: 1b15bd7fd930511222b075418c10de0aaa70a35a | 16 March 2018, 21:08:01 UTC |
| 4f51078 | Adam Retter | 08 March 2018, 19:16:46 UTC | Java wrapper for Native Comparators Summary: This is an abstraction for working with custom Comparators implemented in native C++ code from Java. Native code must directly extend `rocksdb::Comparator`. When the native code comparator is compiled into the RocksDB codebase, you can then create a Java Class, and JNI stub to wrap it. Useful if the C++/JNI barrier overhead is too much for your applications comparator performance. An example is provided in `java/rocksjni/native_comparator_wrapper_test.cc` and `java/src/main/java/org/rocksdb/NativeComparatorWrapperTest.java`. Closes https://github.com/facebook/rocksdb/pull/3334 Differential Revision: D7172605 Pulled By: miasantreble fbshipit-source-id: e24b7eb267a3bcb6afa214e0379a1d5e8a2ceabe | 14 March 2018, 23:59:02 UTC |
| 2c3ceec | Pooya Shareghi | 06 March 2018, 18:20:52 UTC | Added bytes XOR merge operator Summary: Closes https://github.com/facebook/rocksdb/pull/575 I fixed the merge conflicts etc. Closes https://github.com/facebook/rocksdb/pull/3065 Differential Revision: D7128233 Pulled By: sagar0 fbshipit-source-id: 2c23a48c9f0432c290b0cd16a12fb691bb37820c | 14 March 2018, 23:58:36 UTC |
| 5ae32b8 | Adam Retter | 02 March 2018, 23:33:08 UTC | Brings the Java API for WriteBatch inline with the C++ API Summary: * Exposes status * Corrects some method naming * Adds missing functionality Closes https://github.com/facebook/rocksdb/pull/3550 Differential Revision: D7140790 Pulled By: sagar0 fbshipit-source-id: cbdab6c5a7ae4f3030fb46739e9060e381b26fa6 | 14 March 2018, 23:58:02 UTC |
| 9e9192c | Adam Retter | 02 March 2018, 18:22:38 UTC | Add TransactionDB and OptimisticTransactionDB to the Java API Summary: Closes https://github.com/facebook/rocksdb/issues/697 Closes https://github.com/facebook/rocksdb/issues/1151 Closes https://github.com/facebook/rocksdb/pull/1298 Differential Revision: D7131402 Pulled By: sagar0 fbshipit-source-id: bcd34ce95ed88cc641786089ff4232df7b2f089f | 14 March 2018, 23:56:35 UTC |
| b04c5d4 | Siying Dong | 08 March 2018, 18:09:59 UTC | Fix a block pinning regression introduced in b555ed30a4a93b80a3ac4781c6721ab988e03b5b Summary: b555ed30a4a93b80a3ac4781c6721ab988e03b5b introduces a regression, which causes blocks always to be pinned in block based iterators. Fix it. Closes https://github.com/facebook/rocksdb/pull/3582 Differential Revision: D7189534 Pulled By: siying fbshipit-source-id: 117dc7a03d0a0e360424db02efb366e12da2be03 | 08 March 2018, 22:05:10 UTC |
| 8d47fc7 | Fosco Marotto | 01 March 2018, 01:40:14 UTC | 5.12.fb release | 01 March 2018, 01:40:14 UTC |
| ec5843d | leviathan1995 | 28 February 2018, 17:49:32 UTC | Comment typo Summary: Closes https://github.com/facebook/rocksdb/pull/3546 Differential Revision: D7111708 Pulled By: ajkr fbshipit-source-id: 522a4a00eb3e34c73afcb86c1f75cd2e90e7608d | 28 February 2018, 17:56:45 UTC |
| 3ae0047 | Andrew Kryczka | 28 February 2018, 01:08:34 UTC | skip CompactRange flush based on memtable contents Summary: CompactRange has a call to Flush because we guarantee that, at the time it's called, all existing keys in the range will be pushed through the user's compaction filter. However, previously the flush was done blindly, so it'd happen even if the memtable does not contain keys in the range specified by the user. This caused unnecessarily many L0 files to be created, leading to write stalls in some cases. This PR checks the memtable's contents, and decides to flush only if it overlaps with `CompactRange`'s range. - Move the memtable overlap check logic from `ExternalSstFileIngestionJob` to `ColumnFamilyData::RangesOverlapWithMemtables` - Reuse the above logic in `CompactRange` and skip flushing if no overlap Closes https://github.com/facebook/rocksdb/pull/3520 Differential Revision: D7018897 Pulled By: ajkr fbshipit-source-id: a3c6b1cfae56687b49dd89ccac7c948e53545934 | 28 February 2018, 01:12:44 UTC |
| c287c09 | Siying Dong | 27 February 2018, 20:32:09 UTC | Update comments in DB::Close() Summary: Closes https://github.com/facebook/rocksdb/pull/3543 Differential Revision: D7093251 Pulled By: siying fbshipit-source-id: 4066b82c95ecb65866c5842d68ab13ab9f85d567 | 27 February 2018, 20:42:31 UTC |
| d633656 | Istvan Szukacs | 26 February 2018, 23:17:22 UTC | Adding CentOS 7 Vagrantfile & build script Summary: I have updated the Vagrantfile to have an entry for CentOS 7. Also created a simple build script which is pretty similar to the one in Beringei. How to test: ``` vagrant up centos7 ``` Todo: Implement -j X for the build. Closes https://github.com/facebook/rocksdb/pull/3530 Differential Revision: D7090739 Pulled By: ajkr fbshipit-source-id: 9f9eda5b507568993543d08de7ce168dfc12282e | 26 February 2018, 23:27:17 UTC |
| ad05cbb | Zhongyi Xie | 26 February 2018, 22:46:12 UTC | DB:Open should fail on tmpfs when use_direct_reads=true Summary: Before: > $ TEST_TMPDIR=/dev/shm ./db_bench -use_direct_reads=true -benchmarks=readrandomwriterandom -num=10000000 -reads=100000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -readwritepercent=50 -key_size=16 -value_size=48 -threads=32 DB path: [/dev/shm/dbbench] put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument db_bench: tpp.c:84: __pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)' failed. put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument After: > TEST_TMPDIR=/dev/shm ./db_bench -use_direct_reads=true -benchmarks=readrandomwriterandom -num=10000000 -reads=100000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -readwritepercent=50 -key_size=16 -value_size=48 -threads=32 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags open error: Not implemented: Direct I/O is not supported by the specified DB. Closes https://github.com/facebook/rocksdb/pull/3539 Differential Revision: D7082658 Pulled By: miasantreble fbshipit-source-id: f9d9c6ec3b5e9e049cab52154940ee101ba4d342 | 26 February 2018, 22:58:06 UTC |
| 7eb292d | Dmitri Smirnov | 26 February 2018, 21:44:25 UTC | Fix a memory leak in WindowsThread Summary: _endthreadex does not return and thus objects for stack destructors do not run. This creates a memory leak. We remove the calls since _enthreadex called automatically after the threadproc returns i.e. thread exits. Closes https://github.com/facebook/rocksdb/pull/3542 Differential Revision: D7088713 Pulled By: ajkr fbshipit-source-id: 749ecafc6a9572f587f76e516547e07734349a54 | 26 February 2018, 21:46:12 UTC |
| dfbe52e | Anand Ananthabhotla | 23 February 2018, 21:50:02 UTC | Fix the Logger::Close() and DBImpl::Close() design pattern Summary: The recent Logger::Close() and DBImpl::Close() implementation rely on calling the CloseImpl() virtual function from the destructor, which will not work. Refactor the implementation to have a private close helper function in derived classes that can be called by both CloseImpl() and the destructor. Closes https://github.com/facebook/rocksdb/pull/3528 Reviewed By: gfosco Differential Revision: D7049303 Pulled By: anand1976 fbshipit-source-id: 76a64cbf403209216dfe4864ecf96b5d7f3db9f4 | 23 February 2018, 21:57:26 UTC |
| 30649dc | Siying Dong | 23 February 2018, 19:30:55 UTC | Have a different function when ROCKSDB_JEMALLOC=0 Summary: Some sanitizer is not happy with parameter name with ROCKSDB_JEMALLOC not set. Use another function instead. Closes https://github.com/facebook/rocksdb/pull/3536 Differential Revision: D7064849 Pulled By: siying fbshipit-source-id: c6ae94e044686176af1259df9172453d52c2f9d5 | 23 February 2018, 19:42:33 UTC |
| 90eca1e | Maysam Yabandeh | 23 February 2018, 02:05:14 UTC | WritePrepared Txn: optimize SubBatchCnt Summary: Make use of the index in WriteBatchWithIndex to also count the number of sub-batches. This eliminates the need to separately scan the batch to count the number of sub-batches once a duplicate key is detected. Closes https://github.com/facebook/rocksdb/pull/3529 Differential Revision: D7049947 Pulled By: maysamyabandeh fbshipit-source-id: 81cbf12c4e662541c772c7265a8f91631e25c7cd | 23 February 2018, 02:12:26 UTC |
| 243220d | Zhongyi Xie | 23 February 2018, 00:31:58 UTC | Update HISTORY.md to 5.12.0 Summary: Closes https://github.com/facebook/rocksdb/pull/3532 Differential Revision: D7062828 Pulled By: miasantreble fbshipit-source-id: d36967a1cfbcaeeeb33b9f0e09e15dea85b08b70 | 23 February 2018, 00:47:01 UTC |
| 4624edc | Siying Dong | 22 February 2018, 21:23:53 UTC | RocksDBOptionsParser::Parse()'s `ignore_unknown_options` argument only ingores options from higher version. Summary: RocksDB should always be able to parse an option file generated using the same or lower version. Unknown option should only happen if it is from a higher version. Change the behavior of RocksDBOptionsParser::Parse()'s behavior with ignore_unknown_options=true so that unknown option from a lower or the same version will never be skipped. Closes https://github.com/facebook/rocksdb/pull/3527 Differential Revision: D7048851 Pulled By: siying fbshipit-source-id: e261caea12f6515611a4a29f39acf2b619df2361 | 22 February 2018, 21:28:12 UTC |
| aba3409 | Igor Sugak | 22 February 2018, 20:36:41 UTC | Back out "[codemod] - comment out unused parameters" Reviewed By: igorsugak fbshipit-source-id: 4a93675cc1931089ddd574cacdb15d228b1e5f37 | 22 February 2018, 20:43:17 UTC |
| f4a030c | David Lai | 22 February 2018, 17:37:17 UTC | - comment out unused parameters Reviewed By: everiq, igorsugak Differential Revision: D7046710 fbshipit-source-id: 8e10b1f1e2aecebbfb229c742e214db887e5a461 | 22 February 2018, 17:44:23 UTC |
| b092977 | Andrew Kryczka | 22 February 2018, 01:33:14 UTC | BackupEngine gluster-friendly file naming convention Summary: Use the rsync tempfile naming convention in our `BackupEngine`. The temp file follows the format, `.<filename>.<suffix>`, which is later renamed to `<filename>`. We fix `tmp` as the `<suffix>` as we don't need to use random bytes for now. The benefit is gluster treats this tempfile naming convention specially and applies hashing only to `<filename>`, so the file won't need to be linked or moved when it's renamed. Our gluster team suggested this will make things operationally easier. Closes https://github.com/facebook/rocksdb/pull/3463 Differential Revision: D6893333 Pulled By: ajkr fbshipit-source-id: fd7622978f4b2487fce33cde40dd3124f16bcaa8 | 22 February 2018, 01:42:07 UTC |
| 828211e | Maysam Yabandeh | 21 February 2018, 21:40:31 UTC | WritePrepared Txn: fix non-emptied PreparedHeap bug Summary: Under a certain sequence of accessing PreparedHeap, there was a bug that would not successfully empty the heap. This would result in performance issues when the heap content is moved to old_prepared_ after max_evicted_seq_ advances the orphan prepared sequence numbers. The patch fixed the bug and add more unit tests. It also does more logging when the unlikely scenarios are faced Closes https://github.com/facebook/rocksdb/pull/3526 Differential Revision: D7038486 Pulled By: maysamyabandeh fbshipit-source-id: f1e40bea558f67b03d2a29131fcb8734c65fce97 | 21 February 2018, 21:42:23 UTC |
| 8ada876 | Sagar Vemuri | 21 February 2018, 03:05:21 UTC | Add rocksdb.iterator.internal-key property Summary: Added a new iterator property: `rocksdb.iterator.internal-key` to get the internal-key (converted to user key) at which the iterator stopped. Closes https://github.com/facebook/rocksdb/pull/3525 Differential Revision: D7033694 Pulled By: sagar0 fbshipit-source-id: d51e6c00f5e9d766c6276ef79774b81c6c5216f8 | 21 February 2018, 03:12:09 UTC |
| e9c31ab | jsteemann | 21 February 2018, 01:34:44 UTC | save redundant key lookup in map of locked keys Summary: In case it is found that a key is already marked as locked in a stripe's map of locked keys, it is not necessary to look it up again using `std::unordered_map<std::string, ...>::at(size_t)`. Instead, we can use the already found position using the iterator produced by the previous `find` operation. Reusing the iterator will avoid having to hash the key again and do additional "random" memory lookups in the map of keys (though the data will very likely sit available in caches here already due to the previous find operation) Closes https://github.com/facebook/rocksdb/pull/3505 Differential Revision: D7036446 Pulled By: sagar0 fbshipit-source-id: cced51547b2bd2d49394f6bc8c5896f09fa80f68 | 21 February 2018, 01:44:44 UTC |
| 1960e73 | Andrew Kryczka | 21 February 2018, 00:42:06 UTC | fix handling of empty string as checkpoint directory Summary: - made `CreateCheckpoint` properly return `InvalidArgument` when called with an empty directory. Previously it triggered an assertion failure due to a bug in the logic. - made `ldb` set empty `checkpoint_dir` if that's what the user specifies, so that we can use it to properly test `CreateCheckpoint` in the future. Differential Revision: D6874562 fbshipit-source-id: dcc1bd41768261d9338987fa7711444289707ed7 | 21 February 2018, 00:44:00 UTC |
| 5263da6 | Igor Sugak | 21 February 2018, 00:41:54 UTC | fix shift UBSAN error in col_buf_encoder.cc Summary: Add a static cast to perform the left shift as with an unsigned type. make ubsan_check Closes https://github.com/facebook/rocksdb/pull/3517 Reviewed By: sagar0 Differential Revision: D7016044 Pulled By: igorsugak fbshipit-source-id: baf72f6197edd8f7220d010b15a23d6de6a72c49 | 21 February 2018, 00:44:00 UTC |
| ab446dc | Po-Chuan Hsieh | 16 February 2018, 18:34:48 UTC | Fix build with USE_RTTI=0 Summary: utilities/column_aware_encoding_util.cc:61:23: error: cannot use dynamic_cast with -fno-rtti table_reader_.reset(dynamic_cast<BlockBasedTable*>(table_reader.release())); ^ 1 error generated. It was added as a [local patch](https://svnweb.freebsd.org/ports/head/databases/rocksdb/files/patch-utilities-column_aware_encoding_util.cc) on FreeBSD since RocksDB 5.8. It also fixes #2707. Closes https://github.com/facebook/rocksdb/pull/3514 Differential Revision: D7005571 Pulled By: siying fbshipit-source-id: 351a9055d21d0accdd7a932e8e7bfcd3c8e22068 | 16 February 2018, 18:41:49 UTC |
| c178da0 | Maysam Yabandeh | 16 February 2018, 16:36:47 UTC | WritePrepared Txn: optimizations for sysbench update_noindex Summary: These are optimization that we applied to improve sysbech's update_noindex performance. 1. Make use of LIKELY compiler hint 2. Move std::atomic so the subclass 3. Make use of skip_prepared in non-2pc transactions. Closes https://github.com/facebook/rocksdb/pull/3512 Differential Revision: D7000075 Pulled By: maysamyabandeh fbshipit-source-id: 1ab8292584df1f6305a4992973fb1b7933632181 | 16 February 2018, 16:42:31 UTC |
| 97307d8 | Mike Kolupaev | 16 February 2018, 15:58:18 UTC | Fix deadlock in ColumnFamilyData::InstallSuperVersion() Summary: Deadlock: a memtable flush holds DB::mutex_ and calls ThreadLocalPtr::Scrape(), which locks ThreadLocalPtr mutex; meanwhile, a thread exit handler locks ThreadLocalPtr mutex and calls SuperVersionUnrefHandle, which tries to lock DB::mutex_. This deadlock is hit all the time on our workload. It blocks our release. In general, the problem is that ThreadLocalPtr takes an arbitrary callback and calls it while holding a lock on a global mutex. The same global mutex is (at least in some cases) locked by almost all ThreadLocalPtr methods, on any instance of ThreadLocalPtr. So, there'll be a deadlock if the callback tries to do anything to any instance of ThreadLocalPtr, or waits for another thread to do so. So, probably the only safe way to use ThreadLocalPtr callbacks is to do only do simple and lock-free things in them. This PR fixes the deadlock by making sure that local_sv_ never holds the last reference to a SuperVersion, and therefore SuperVersionUnrefHandle never has to do any nontrivial cleanup. I also searched for other uses of ThreadLocalPtr to see if they may have similar bugs. There's only one other use, in transaction_lock_mgr.cc, and it looks fine. Closes https://github.com/facebook/rocksdb/pull/3510 Reviewed By: sagar0 Differential Revision: D7005346 Pulled By: al13n321 fbshipit-source-id: 37575591b84f07a891d6659e87e784660fde815f | 16 February 2018, 16:13:34 UTC |
| 0454f78 | Andrew Kryczka | 16 February 2018, 03:30:52 UTC | fix advance reservation of arena block addresses Summary: Calling `std::vector::reserve()` causes memory to be reallocated and then data to be moved. It was called prior to adding every block. This reallocation could be done a huge amount of times, e.g., for users with large index blocks. Instead, we can simply use `std::vector::emplace_back()` in such a way that preserves the no-memory-leak guarantee, while letting the vector decide when to reallocate space. Now I see reallocation/moving happen O(logN) times, rather than O(N) times, where N is the final size of vector. Closes https://github.com/facebook/rocksdb/pull/3508 Differential Revision: D6994228 Pulled By: ajkr fbshipit-source-id: ab7c11e13ff37c8c6c8249be7a79566a4068cd27 | 16 February 2018, 03:41:52 UTC |
| 989d123 | Yi Wu | 16 February 2018, 01:14:08 UTC | Legocastle job to report lite build binary size to scuba Summary: Add a legocastle job to continuously build the last 10 commits every 4 hours and report lite build binary size to scuba. Closes https://github.com/facebook/rocksdb/pull/3511 Differential Revision: D7001730 Pulled By: yiwu-arbug fbshipit-source-id: 7c8ca87c46d663c786a0d32be69ebbe7b19a5eb9 | 16 February 2018, 01:27:24 UTC |
| 8eb1d44 | Maysam Yabandeh | 16 February 2018, 01:12:48 UTC | Unbreak MemTableRep API change Summary: The MemTableRep API was broken by this commit: 813719e9525f647aaebf19ca3d4bb6f1c63e2648 This patch reverts the changes and instead adds InsertKey (and etc.) overloads to extend the MemTableRep API without breaking the existing classes that inherit from it. Closes https://github.com/facebook/rocksdb/pull/3513 Differential Revision: D7004134 Pulled By: maysamyabandeh fbshipit-source-id: e568d91fe1e17dd76c0c1f6c7dd51a18633b1c4f | 16 February 2018, 01:27:24 UTC |
| 4e7a182 | jsteemann | 16 February 2018, 00:43:23 UTC | Several small "fixes" Summary: - removed a few unneeded variables - fused some variable declarations and their assignments - fixed right-trimming code in string_util.cc to not underflow - simplifed an assertion - move non-nullptr check assertion before dereferencing of that pointer - pass an std::string function parameter by const reference instead of by value (avoiding potential copy) Closes https://github.com/facebook/rocksdb/pull/3507 Differential Revision: D7004679 Pulled By: sagar0 fbshipit-source-id: 52944952d9b56dfcac3bea3cd7878e315bb563c4 | 16 February 2018, 00:57:37 UTC |
| c88c57c | Zhongyi Xie | 15 February 2018, 22:11:08 UTC | Tweak external file ingestion seqno logic under universal compaction Summary: Right now it is possible that a file gets assigned to L0 but also assigned the seqno from a higher level which it doesn't fit Under the current impl, it is possibe that seqno in lower levels (Ln) can be equal to smallest seqno of higher levels (Ln-1), which is undesirable from universal compaction's point of view. This should fix the intermittent failure of `ExternalSSTFileBasicTest.IngestFileWithGlobalSeqnoPickedSeqno` Closes https://github.com/facebook/rocksdb/pull/3411 Differential Revision: D6813802 Pulled By: miasantreble fbshipit-source-id: 693d0462fa94725ccfb9d8858743e6d2d9992d14 | 15 February 2018, 22:13:39 UTC |
| 6a30b98 | jsteemann | 15 February 2018, 19:01:42 UTC | fix wrong indentation Summary: Somehow the indentation was incorrect in this file. The only change in this PR is to get it right again in order to make the code more readable. Please reject if you think it's not worth it. Closes https://github.com/facebook/rocksdb/pull/3504 Differential Revision: D6996011 Pulled By: miasantreble fbshipit-source-id: 060514a3a8c910d34bad795b36eb4d278512b154 | 15 February 2018, 19:13:37 UTC |
| ba6ee1f | Fosco Marotto | 14 February 2018, 19:08:39 UTC | Fix 2 more unused reference errors VS2017 Summary: As in #3425 Closes https://github.com/facebook/rocksdb/pull/3497 Differential Revision: D6979588 Pulled By: gfosco fbshipit-source-id: e9fb32d04ad45575dfe9de1d79348d158e474197 | 14 February 2018, 19:12:36 UTC |
| b3c5351 | Siying Dong | 14 February 2018, 00:20:13 UTC | Direct I/O writable file should do fsync in Close() Summary: We don't do fsync() after truncate in direct I/O writeable file (in fact we don't do any fsync ever). This can cause metadata not persistent to disk after the file is generated. We call it instead. Closes https://github.com/facebook/rocksdb/pull/3500 Differential Revision: D6981482 Pulled By: siying fbshipit-source-id: 7e2b591b7e5dd1b96fc0775515b8b9e6092980ef | 14 February 2018, 00:27:11 UTC |
| d08d05c | Igor Sugak | 13 February 2018, 22:07:48 UTC | fix UBSAN errors in fault_injection_test Summary: This fixes shift and signed-integer-overflow UBSAN checks in fault_injection_test by using a larger and unsigned type. Closes https://github.com/facebook/rocksdb/pull/3498 Reviewed By: siying Differential Revision: D6981116 Pulled By: igorsugak fbshipit-source-id: 3688f62cce570534b161e9b5f42109ebc9ae5a2c | 13 February 2018, 22:12:40 UTC |
| dadf016 | Siying Dong | 13 February 2018, 21:44:22 UTC | Rename one of the two LevelIterator Summary: A new LevelIterator was recently created. Rename the old one to make unity build happy. It's also not a good idea to have two classes in the same name anyway. Closes https://github.com/facebook/rocksdb/pull/3499 Differential Revision: D6979325 Pulled By: siying fbshipit-source-id: 3a032d93fe205650a08e92e5262594731ec726bb | 13 February 2018, 21:57:58 UTC |
| 7474861 | Siying Dong | 13 February 2018, 20:05:36 UTC | Suppress UBSAN error in finer guanularity Summary: Now we suppress alignment UBSAN error as a whole. Suppressing 3-way CRC and murmurhash feels a better idea than turning off alignment check as a whole. Closes https://github.com/facebook/rocksdb/pull/3495 Differential Revision: D6971273 Pulled By: siying fbshipit-source-id: 080b59fed6df494b9f622ef7cb5d42d39e6a8cdf | 13 February 2018, 20:18:07 UTC |
| 3c380fd | Fosco Marotto | 13 February 2018, 19:45:24 UTC | Adding blog post for 5.10.2 release Summary: Closes https://github.com/facebook/rocksdb/pull/3464 Differential Revision: D6906184 Pulled By: gfosco fbshipit-source-id: 415934d7b1dd8dd226b6619bfb71781184d55cd9 | 13 February 2018, 19:56:59 UTC |
| b555ed3 | Siying Dong | 13 February 2018, 00:57:56 UTC | Customized BlockBasedTableIterator and LevelIterator Summary: Use a customzied BlockBasedTableIterator and LevelIterator to replace current implementations leveraging two-level-iterator. Hope the customized logic will make code easier to understand. As a side effect, BlockBasedTableIterator reduces the allocation for the data block iterator object, and avoid the virtual function call to it, because we can directly reference BlockIter, a final class. Similarly, LevelIterator reduces virtual function call to the dummy iterator iterating the file metadata. It also enabled further optimization. The upper bound check is also moved from index block to data block. This implementation fits this iterator better. After the change, forwared iterator is slightly optimized to ensure we trim those iterators. The two-level-iterator now is only used by partitioned index, so it is simplified. Closes https://github.com/facebook/rocksdb/pull/3406 Differential Revision: D6809041 Pulled By: siying fbshipit-source-id: 7da3b9b1d3c8e9d9405302c15920af1fcaf50ffa | 13 February 2018, 01:12:25 UTC |
| 8a04ee4 | Maysam Yabandeh | 13 February 2018, 00:27:39 UTC | WritePrepared Txn: use TransactionDBWriteOptimizations (2nd attempt) Summary: TransactionDB::Write can receive some optimization hints from the user. One is to skip the concurrency control mechanism. WritePreparedTxnDB is currently ignoring such hints. This patch optimizes WritePreparedTxnDB::Write for skip_concurrency_control and skip_duplicate_key_check hints. Closes https://github.com/facebook/rocksdb/pull/3496 Differential Revision: D6971784 Pulled By: maysamyabandeh fbshipit-source-id: cbab10ad538fa2b8bcb47e37c77724afe6e30f03 | 13 February 2018, 00:43:40 UTC |
| ee1c802 | Andrew Kryczka | 12 February 2018, 23:34:39 UTC | Add delay before flush in CompactRange to avoid write stalling Summary: - Refactored logic for checking write stall condition to a helper function: `GetWriteStallConditionAndCause`. Now it is decoupled from the logic for updating WriteController / stats in `RecalculateWriteStallConditions`, so we can reuse it for predicting whether write stall will occur. - Updated `CompactRange` to first check whether the one additional immutable memtable / L0 file would cause stalling before it flushes. If so, it waits until that is no longer true. - Updated `bg_cv_` to be signaled on `SetOptions` calls. The stall conditions `CompactRange` cares about can change when (1) flush finishes, (2) compaction finishes, or (3) options dynamically change. The cv was already signaled for (1) and (2) but not yet for (3). Closes https://github.com/facebook/rocksdb/pull/3381 Differential Revision: D6754983 Pulled By: ajkr fbshipit-source-id: 5613e03f1524df7192dc6ae885d40fd8f091d972 | 12 February 2018, 23:42:47 UTC |
| 0a0fad4 | Andrew Kryczka | 12 February 2018, 22:54:50 UTC | db_bench separate options for partition index and filters Summary: Some workloads (like my current benchmarking) may want partitioned indexes without partitioned filters. Particularly, when `-optimize_filters_for_hits=true`, the total index size may be larger than the total filter size, so it can make sense to hold all filters in-memory but not all indexes. Closes https://github.com/facebook/rocksdb/pull/3492 Differential Revision: D6970092 Pulled By: ajkr fbshipit-source-id: b7fa1828e1d13829339aefb90fd56eb7c5337f61 | 12 February 2018, 22:57:13 UTC |
| 3f1bb07 | Zhongyi Xie | 12 February 2018, 21:14:01 UTC | make flush_reason_ atomic to keep TSAN happy Summary: Closes https://github.com/facebook/rocksdb/pull/3487 Differential Revision: D6967098 Pulled By: miasantreble fbshipit-source-id: 48e0accf2e3b3f589ddb797ff8083c8520269bf0 | 12 February 2018, 21:28:18 UTC |
| ef29d2a | Siying Dong | 09 February 2018, 22:50:09 UTC | Explictly fail writes if key or value is not smaller than 4GB Summary: Right now, users will encounter unexpected bahavior if they use key or value larger than 4GB. We should explicitly fail the queriers. Closes https://github.com/facebook/rocksdb/pull/3484 Differential Revision: D6953895 Pulled By: siying fbshipit-source-id: b60491e1af064fc5d52971956661f6c18ceac24f | 09 February 2018, 22:57:54 UTC |
| fe228da | Yi Wu | 09 February 2018, 22:43:56 UTC | WritePrepared Txn: Support merge operator Summary: CompactionIterator invoke MergeHelper::MergeUntil() to do partial merge between snapshot boundaries. Previously it only depend on sequence number to tell snapshot boundary, but we also need to make use of snapshot_checker to verify visibility of the merge operands to the snapshots. For example, say there is a snapshot with seq = 2 but only can see data with seq <= 1. There are three merges, each with seq = 1, 2, 3. A correct compaction output would be (1),(2+3). Without taking snapshot_checker into account when generating merge result, compaction will generate output (1+2),(3). By filtering uncommitted keys with read callback, the read path already take care of merges well and don't need additional updates. Closes https://github.com/facebook/rocksdb/pull/3475 Differential Revision: D6926087 Pulled By: yiwu-arbug fbshipit-source-id: 8f539d6f897cfe29b6dc27a8992f68c2a629d40a | 09 February 2018, 22:57:54 UTC |
| 9fc72d6 | Chinmay Kamat | 09 February 2018, 22:01:10 UTC | Compilation fixes for powerpc build, -Wparentheses-equality error and missing header guards Summary: This pull request contains miscellaneous compilation fixes. Thanks, Chinmay Closes https://github.com/facebook/rocksdb/pull/3462 Differential Revision: D6941424 Pulled By: sagar0 fbshipit-source-id: fe9c26507bf131221f2466740204bff40a15614a | 09 February 2018, 22:12:43 UTC |
| d62af7f | jsteemann | 09 February 2018, 20:53:11 UTC | fix a typo (of a potential vi user) Summary: Closes https://github.com/facebook/rocksdb/pull/3481 Differential Revision: D6939089 Pulled By: siying fbshipit-source-id: ccce3ae10cc5ff50a74b85804afd044b21a3c3e2 | 09 February 2018, 20:58:07 UTC |
| 945f618 | Zhongyi Xie | 09 February 2018, 20:09:55 UTC | log flush reason for better debugging experience Summary: It's always a mystery from the logs why flush was triggered -- user triggered it manually, WriteBufferManager triggered it, logs were full, write buffer was full, etc. This PR logs Flush reason whenever a flush is scheduled. Closes https://github.com/facebook/rocksdb/pull/3401 Differential Revision: D6788142 Pulled By: miasantreble fbshipit-source-id: a867e54d493c06adf5172bd36a180fb3faae3511 | 09 February 2018, 20:12:43 UTC |
| e78715c | Andrew Kryczka | 07 February 2018, 23:42:35 UTC | Eliminate a memcpy for uncompressed blocks Summary: `ReadBlockFromFile` uses a stack buffer to hold small data blocks before passing them to the compression library, which outputs uncompressed data in a heap buffer. In the case of `kNoCompression` there is a `memcpy` to copy from stack buffer to heap buffer. This PR optimizes `ReadBlockFromFile` to skip the stack buffer for files whose blocks are known to be uncompressed. We determine this using the SST file property, "compression_name", if it's available. Closes https://github.com/facebook/rocksdb/pull/3472 Differential Revision: D6920848 Pulled By: ajkr fbshipit-source-id: 5c753e804efc178b9229ae5dbe6a4adc32031f07 | 07 February 2018, 23:57:37 UTC |
| a0931b3 | Siying Dong | 07 February 2018, 22:41:33 UTC | Fix UBSAN Error in WritePreparedTransactionTest Summary: WritePreparedTransactionTest has the UBSAN error because the wrong order of its parent class construction. Fix it. Closes https://github.com/facebook/rocksdb/pull/3478 Differential Revision: D6928975 Pulled By: siying fbshipit-source-id: 13edfd5cb9cf73f1ac5ae3b6f53061d32783733d | 07 February 2018, 22:57:35 UTC |
| 821e0b1 | Siying Dong | 07 February 2018, 22:28:50 UTC | Disable options_settable_test in UBSAN and fix UBSAN failure in blob_… Summary: …db_test options_settable_test won't pass UBSAN so disable it. blob_db_test fails in UBSAN as SnapshotList doesn't initialize all the fields in dummy snapshot. Fix it. I don't understand why only blob_db_test fails though. Closes https://github.com/facebook/rocksdb/pull/3477 Differential Revision: D6928681 Pulled By: siying fbshipit-source-id: e31dd300fcdecdfd4f6af279a0987fd0cdec5122 | 07 February 2018, 22:42:26 UTC |
| 1336a77 | Siying Dong | 07 February 2018, 18:45:46 UTC | Disable alignment check in UBSAN Summary: Disable alignment check in UBSAN for now. Now we can't get signals to meaningful failures. We can reenable it after we figure out how we can suppress failures in finer grain manner. Closes https://github.com/facebook/rocksdb/pull/3473 Differential Revision: D6925971 Pulled By: siying fbshipit-source-id: a0f1a242cde866abbc5c1eeee9ff8d1d7d582ac4 | 07 February 2018, 18:58:01 UTC |
| 8feee28 | Maysam Yabandeh | 06 February 2018, 23:22:58 UTC | Add skip_cc option to TransactionDB::Write Summary: Compared to DB::Write, TransactionDB::Write has the additional overhead of creating and initializing an internal transaction object, as well as the overhead of locking/unlocking the keys. This patch extends the TransactionDB::Write with an skip_cc option to allow the users to indicate that the write batch do not conflict with others and the concurrency control and its overhead can be skipped. TransactionDB::Write by default calls DB::Write when skip_cc is set, which works for WriteCommitted WritePolicy. Any other flavor of TransactionDB that is not compatible with this default behavior (such as WritePreparedTxnDB) can extend ::Write and implement their own approach for taking into account the skip_cc optimization. Closes https://github.com/facebook/rocksdb/pull/3457 Differential Revision: D6877318 Pulled By: maysamyabandeh fbshipit-source-id: 56f4e21db87ff71492db4e376fb7c2b03dfeab6b | 06 February 2018, 23:28:24 UTC |
| 8f8eb4f | Maysam Yabandeh | 06 February 2018, 22:17:18 UTC | Fix leak report by asan on DuplicateKeys test Summary: Deletes the transaction object at the end of the test. Verified by: - COMPILE_WITH_ASAN=1 make -j32 transaction_test - ./transaction_test --gtest_filter="DBA**Duplicate*" Closes https://github.com/facebook/rocksdb/pull/3470 Differential Revision: D6916473 Pulled By: maysamyabandeh fbshipit-source-id: 8303df25408635d5d3ac2b25f309a3d15957c937 | 06 February 2018, 22:26:35 UTC |
| 81736d8 | Yi Wu | 06 February 2018, 22:06:44 UTC | WritePrepared Txn: update compaction_iterator_test and db_iterator_test Summary: Update compaction_iterator_test with write-prepared transaction DB related tests. Transaction related tests are group in CompactionIteratorWithSnapshotCheckerTest. The existing test are duplicated to make them also test with dummy SnapshotChecker that will say every key is visible to every snapshot (this is okay, we still compare sequence number to verify visibility). Merge related tests are disabled and will be revisit in another PR. Existing db_iterator_tests are also duplicated to test with dummy read_callback that will say every key is committed. Closes https://github.com/facebook/rocksdb/pull/3466 Differential Revision: D6909253 Pulled By: yiwu-arbug fbshipit-source-id: 2ae4656b843a55e2e9ff8beecf21f2832f96cd25 | 06 February 2018, 22:12:13 UTC |
| 2f29991 | Zhongyi Xie | 06 February 2018, 21:54:53 UTC | split RandomizedHarnessTest more ways Summary: RandomizedHarnessTest enumerates different combinations of test type, compression type, restart interval, etc. For some combinations it takes very long to finish, causing the test to time out in test infrastructure. This PR split the test input into smaller trunks in the hope that they will fit in the timeout window. Another possibility is to reduce `num_entries` of course Closes https://github.com/facebook/rocksdb/pull/3467 Differential Revision: D6910235 Pulled By: miasantreble fbshipit-source-id: 717246ee5d21a8a48ad82d4d9c04f9051a66f07f | 06 February 2018, 21:58:18 UTC |
| 88d8b2a | Maysam Yabandeh | 06 February 2018, 02:32:54 UTC | WritePrepared Txn: Duplicate Keys, Txn Part Summary: This patch takes advantage of memtable being able to detect duplicate <key,seq> and returning TryAgain to handle duplicate keys in WritePrepared Txns. Through WriteBatchWithIndex's index it detects existence of at least a duplicate key in the write batch. If duplicate key was reported, it then pays the cost of counting the number of sub-patches by iterating over the write batch and pass it to DBImpl::Write. DB will make use of the provided batch_count to assign proper sequence numbers before sending them to the WAL. When later inserting the batch to the memtable, it increases the seq each time memtbale reports a duplicate (a sub-patch in our counting) and tries again. Closes https://github.com/facebook/rocksdb/pull/3455 Differential Revision: D6873699 Pulled By: maysamyabandeh fbshipit-source-id: db8487526c3a5dc1ddda0ea49f0f979b26ae648d | 06 February 2018, 02:43:24 UTC |
| 4b124fb | Anand Ananthabhotla | 05 February 2018, 21:48:25 UTC | Handle error return from WriteBuffer() Summary: There are a couple of places where we swallow any error from WriteBuffer() - in SwitchMemtable() and DBImpl::CloseImpl(). Propagate the error up in those cases rather than ignoring it. Closes https://github.com/facebook/rocksdb/pull/3404 Differential Revision: D6879954 Pulled By: anand1976 fbshipit-source-id: 2ef88b554be5286b0a8bad7384ba17a105395bdb | 05 February 2018, 21:59:34 UTC |
| c340184 | Agam Brahma | 04 February 2018, 02:25:28 UTC | Minor typo in comment (s/pro/pri) Summary: Closes https://github.com/facebook/rocksdb/pull/3460 Differential Revision: D6895365 Pulled By: miasantreble fbshipit-source-id: 04f633d1971b1f542ac28118b738ceb0242a0228 | 04 February 2018, 02:27:14 UTC |
| cb5b8f2 | Mike Kolupaev | 03 February 2018, 05:16:26 UTC | Fix use-after-free in tailing iterator with merge operator Summary: ForwardIterator::SVCleanup() sometimes didn't pin superversion when it was supposed to. See the added test for the scenario. Here's the ASAN output of the added test without the fix (using `COMPILE_WITH_ASAN=1 make`): https://pastebin.com/9rD0Ywws Closes https://github.com/facebook/rocksdb/pull/3415 Differential Revision: D6817414 Pulled By: al13n321 fbshipit-source-id: bc80c44ea78a3a1fa885dfa448a26111f91afb24 | 03 February 2018, 05:26:28 UTC |
| cd5092e | Tamir Duberstein | 02 February 2018, 20:14:42 UTC | Suppress unused warnings Summary: - Use `__unused__` everywhere - Suppress unused warnings in Release mode + This currently affects non-MSVC builds (e.g. mingw64). Closes https://github.com/facebook/rocksdb/pull/3448 Differential Revision: D6885496 Pulled By: miasantreble fbshipit-source-id: f2f6adacec940cc3851a9eee328fafbf61aad211 | 02 February 2018, 20:27:07 UTC |
| a247617 | Ben Clay | 02 February 2018, 18:46:41 UTC | Java: Add copy constructors for various option classes Summary: Add Java-side copy constructors for: - Options - DBOptions - ColumnFamilyOptions - WriteOptions along with unit tests to assert the copy worked. NOTE: Unit tests are failing in travis but it looks like a global timeout issue. These tests pass. Closes https://github.com/facebook/rocksdb/pull/3450 Differential Revision: D6874425 Pulled By: sagar0 fbshipit-source-id: 5bde68ea5b5225e071faea2628bf8bbf10bd65ab | 02 February 2018, 18:57:28 UTC |
| 6e5b341 | Prashant D | 01 February 2018, 22:20:25 UTC | options: Fix coverity issues Summary: options/cf_options.cc: 77 memtable_insert_with_hint_prefix_extractor( CID 1396208 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member info_log_level is not initialized in this constructor nor in any functions that it calls. Closes https://github.com/facebook/rocksdb/pull/3106 Differential Revision: D6874689 Pulled By: sagar0 fbshipit-source-id: b5cd2d13915fd86d87260050f9c5d117615bbe30 | 01 February 2018, 22:27:42 UTC |
| e502839 | Jun Wu | 01 February 2018, 22:15:28 UTC | crc32: suppress -Wimplicit-fallthrough warnings Summary: Workaround a bunch of "implicit-fallthrough" compiler errors, like: ``` util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=] crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset)); ^ util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’ CRCsinglet(crc0, next, -2 * 8); ^~~~~~~~~~ util/crc32c.cc:1017:7: note: here case 1: ``` Closes https://github.com/facebook/rocksdb/pull/3339 Reviewed By: sagar0 Differential Revision: D6874736 Pulled By: quark-zju fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16 | 01 February 2018, 22:27:42 UTC |
| ba8aa8f | Fosco Marotto | 01 February 2018, 21:54:50 UTC | Upgrade Appveyor to VS2017 Summary: Per some discussions, this will switch our Appveyor testing to use Visual Studio 2017. Closes https://github.com/facebook/rocksdb/pull/3445 Differential Revision: D6874918 Pulled By: gfosco fbshipit-source-id: c5a0032ca9f37f0d3baeae35c59d850d528c3176 | 01 February 2018, 21:57:01 UTC |
| b78ed04 | Andrew Kryczka | 01 February 2018, 17:36:01 UTC | fix ReadaheadRandomAccessFile/iterator prefetch bug Summary: `ReadaheadRandomAccessFile` is used by iterators for file reads in several cases, like in compaction when `compaction_readahead_size > 0` or `use_direct_io_for_flush_and_compaction == true`, or in user iterator when `ReadOptions::readahead_size > 0`. `ReadaheadRandomAccessFile` maintains an internal buffer for readahead data. It assumes that, if the buffer's length is less than `ReadaheadRandomAccessFile::readahead_size_`, which is fixed in the constructor, then EOF has been reached so it doesn't try reading further. Recently, d938226af405681c592f25310f41c0c933bcdb19 started calling `RandomAccessFile::Prefetch` with various lengths: 8KB, 16KB, etc. When the `RandomAccessFile` is a `ReadaheadRandomAccessFile`, it triggers the above condition and incorrectly determines EOF. If a block is partially in the readahead buffer and EOF is incorrectly decided, the result is a truncated data block. The problem is reproducible: ``` TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=fillrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -block_size=18384 -use_direct_io_for_flush_and_compaction=true ... put error: Corruption: truncated block read from /data/compaction_bench/dbbench/000014.sst offset 20245, expected 10143 bytes, got 8427 ``` Closes https://github.com/facebook/rocksdb/pull/3454 Differential Revision: D6869405 Pulled By: ajkr fbshipit-source-id: 87001c299e7600a37c0dcccbd0368e0954c929cf | 01 February 2018, 17:42:09 UTC |
| 813719e | Maysam Yabandeh | 01 February 2018, 02:45:49 UTC | WritePrepared Txn: Duplicate Keys, Memtable part Summary: Currently DB does not accept duplicate keys (keys with the same user key and the same sequence number). If Memtable returns false when receiving such keys, we can benefit from this signal to properly increase the sequence number in the rare cases when we have a duplicate key in the write batch written to DB under WritePrepared transactions. Closes https://github.com/facebook/rocksdb/pull/3418 Differential Revision: D6822412 Pulled By: maysamyabandeh fbshipit-source-id: adea3ce5073131cd38ed52b16bea0673b1a19e77 | 01 February 2018, 02:57:07 UTC |
| e62a763 | Yi Wu | 01 February 2018, 01:59:32 UTC | Blob DB: miscellaneous changes Summary: * Expose garbage collection related options * Minor logging and counter name update * Remove unused constants. Closes https://github.com/facebook/rocksdb/pull/3451 Differential Revision: D6867077 Pulled By: yiwu-arbug fbshipit-source-id: 6c3272a9c9d78b125a0bd6b2e56d00d087cdd6c8 | 01 February 2018, 02:13:23 UTC |
| 1edac32 | Andrew Kryczka | 31 January 2018, 22:16:04 UTC | Update rocksdb.read.block.get.micros when block cache disabled Summary: Previously `ReadBlockFromFile` for data blocks was only measured when reading a block to populate block cache. This PR adds the corresponding measurements for users who disabled block cache. Closes https://github.com/facebook/rocksdb/pull/3442 Differential Revision: D6848671 Pulled By: ajkr fbshipit-source-id: bb4bbe1797fa2cc1d9a5bad44891af2b55384b41 | 31 January 2018, 22:26:52 UTC |
| 5eccf0b | Zhongyi Xie | 31 January 2018, 20:07:52 UTC | add -fno-sanitize-recover option to force exit on errors Summary: By default if ubsan detects any problem, it outputs a “runtime error:” message, and in most cases continues executing the program. In order to make test abort on errors, option `-fno-sanitize-recover` is needed. [link](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) Closes https://github.com/facebook/rocksdb/pull/3447 Differential Revision: D6854654 Pulled By: miasantreble fbshipit-source-id: c48e892b0b38307029df38a67adda0e24257e481 | 31 January 2018, 20:13:00 UTC |
| 6efa8e2 | Fosco Marotto | 31 January 2018, 20:04:52 UTC | Update endif/else behavior for unreachable code error on Windows. Summary: Per #3367 Closes https://github.com/facebook/rocksdb/pull/3389 Differential Revision: D6766126 Pulled By: gfosco fbshipit-source-id: e441a15e8aec6747c613d68f4f0621b605eb48a0 | 31 January 2018, 20:13:00 UTC |
| 5400800 | Fosco Marotto | 31 January 2018, 19:50:59 UTC | Work around VS2017 warning for unused reference Summary: For #3407 Closes https://github.com/facebook/rocksdb/pull/3425 Differential Revision: D6836900 Pulled By: gfosco fbshipit-source-id: 7bcaf7a1beeeeabb7c05584f2745e7b4a2473497 | 31 January 2018, 19:58:10 UTC |
| ab5ab36 | Andrew Kryczka | 31 January 2018, 00:41:21 UTC | fix DBTest2.ReadAmpBitmapLiveInCacheAfterDBClose file ID support check Summary: Updated the test case to handle tmpfs mounted at directories different from "/dev/shm/". Closes https://github.com/facebook/rocksdb/pull/3440 Differential Revision: D6848213 Pulled By: ajkr fbshipit-source-id: 465e9dbf0921d0930161f732db6b3766bb030589 | 31 January 2018, 00:50:42 UTC |
| ec225d2 | Maysam Yabandeh | 31 January 2018, 00:19:22 UTC | Make WithParamInterface virtual in transaction_test Summary: Without this patch, ubsan_check is currently failing with this error: ``` utilities/transactions/write_prepared_transaction_test.cc:369:63: runtime error: member call on address 0x0000051649f8 which does not point to an object of type 'WithParamInterface' 0x0000051649f8: note: object has invalid vptr ``` Tested by `COMPILE_WITH_UBSAN=1 make -j32 transaction_test` and running `./write_prepared_transaction_test --gtest_filter=TwoWriteQueues/SnapshotConcurrentAccessTest.SnapshotConcurrentAccessTest1/0` Closes https://github.com/facebook/rocksdb/pull/3444 Differential Revision: D6850087 Pulled By: maysamyabandeh fbshipit-source-id: 5b254da8504b8757f7aec8a820ad464154da1a1d | 31 January 2018, 00:26:56 UTC |
| ab43ff5 | Huachao Huang | 30 January 2018, 21:46:08 UTC | Delete files in multiple ranges at once Summary: Using `DeleteFilesInRange` to delete files in a lot of ranges can be slow, because `VersionSet::LogAndApply` is expensive. This PR adds a new `DeleteFilesInRange` function to delete files in multiple ranges at once. Close https://github.com/facebook/rocksdb/issues/2951 Closes https://github.com/facebook/rocksdb/pull/3431 Differential Revision: D6849228 Pulled By: ajkr fbshipit-source-id: daeedcabd8def4b1d9ee95a58266dee77b5d68cb | 30 January 2018, 21:56:39 UTC |
| 77dc069 | Fosco Marotto | 30 January 2018, 19:06:35 UTC | Change size_t cast in table_test Summary: Fixes this build error on master (macOS): ``` table/table_test.cc:972:27: error: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'unsigned int' [-Werror,-Wshorten-64-to-32] ``` Closes https://github.com/facebook/rocksdb/pull/3434 Reviewed By: maysamyabandeh Differential Revision: D6840354 Pulled By: gfosco fbshipit-source-id: fffac6aefbbdd134ce1299453c5590aa855a5fc8 | 30 January 2018, 19:12:51 UTC |
| f3fe6f8 | Andrew Kryczka | 30 January 2018, 04:59:59 UTC | fix for checkpoint directory with trailing slash(es) Summary: previously if `checkpoint_dir` contained a trailing slash, we'd attempt to create the `.tmp` directory under `checkpoint_dir` due to simply concatenating `checkpoint_dir + ".tmp"`. This failed because `checkpoint_dir` hadn't been created yet and our directory creation is non-recursive. This PR fixes the issue by always creating the `.tmp` directory in the same parent as `checkpoint_dir` by stripping trailing slashes before concatenating. Closes https://github.com/facebook/rocksdb/pull/3275 Differential Revision: D6574952 Pulled By: ajkr fbshipit-source-id: a6daa6777a901eac2460cd0140c9515f7241aefc | 30 January 2018, 05:11:42 UTC |
| 4bdf06e | Yi Wu | 30 January 2018, 02:44:01 UTC | Fix DBFlushTest::ManualFlushWithMinWriteBufferNumberToMerge dead lock Summary: In the test, there can be a dead lock between background flush thread and foreground main thread as following: * background flush thread: - holding db mutex, while - waiting on "DBImpl::FlushMemTableToOutputFile:BeforeInstallSV" sync point. * foreground thread: - waiting for db mutex to write "key2" Fixing by let background flush thread wait without holding db mutex. Closes https://github.com/facebook/rocksdb/pull/3436 Differential Revision: D6841334 Pulled By: yiwu-arbug fbshipit-source-id: b020768ac94e166e40953c5d09e505515a5f244d | 30 January 2018, 02:56:47 UTC |
| 3073b1c | Maysam Yabandeh | 30 January 2018, 01:03:23 UTC | Split SnapshotConcurrentAccessTest into 20 sub tests Summary: SnapshotConcurrentAccessTest sometimes times out when running on the test infra. This patch splits the test into smaller sub-tests to avoid the timeout. It also benefits from lower run-time of each sub-test and increases the coverage of the test. The overall run-time of each final sub-test is at most half of the original test so we should no longer see a timeout. Closes https://github.com/facebook/rocksdb/pull/3435 Differential Revision: D6839427 Pulled By: maysamyabandeh fbshipit-source-id: d53fdb157109e2438ca7fe447d0cf4b71f304bd8 | 30 January 2018, 01:12:55 UTC |
| e6605e5 | Sagar Vemuri | 30 January 2018, 00:32:17 UTC | Tests for dynamic universal compaction options Summary: Added a test for three dynamic universal compaction options, in the realm of read amplification: - size_ratio - min_merge_width - max_merge_width Also updated DynamicUniversalCompactionSizeAmplification by adding a check on compaction reason. Found a bug in compaction reason setting while working on this PR, and fixed in #3412 . TODO for later: Still to add tests for these options: compression_size_percent, stop_style and trivial_move. Closes https://github.com/facebook/rocksdb/pull/3419 Differential Revision: D6822217 Pulled By: sagar0 fbshipit-source-id: 074573fca6389053cbac229891a0163f38bb56c4 | 30 January 2018, 00:42:45 UTC |
| 3fe0937 | Zhongyi Xie | 29 January 2018, 22:34:56 UTC | Use block cache to track memory usage when ReadOptions.fill_cache=false Summary: ReadOptions.fill_cache is set in compaction inputs and can be set by users in their queries too. It tells RocksDB not to put a data block used to block cache. The memory used by the data block is, however, not trackable by users. To make the system more manageable, we can cost the block to block cache while using it, and then release it after using. Closes https://github.com/facebook/rocksdb/pull/3333 Differential Revision: D6670230 Pulled By: miasantreble fbshipit-source-id: ab848d3ed286bd081a13ee1903de357b56cbc308 | 29 January 2018, 22:43:10 UTC |
| e2d4b0e | Siying Dong | 29 January 2018, 22:13:05 UTC | db_bench: sanity check CuckooTable with mmap_read option Summary: This is to avoid run time error. Fail the db_bench immediately if cuckoo table is used but mmap_read is not specified. Closes https://github.com/facebook/rocksdb/pull/3420 Differential Revision: D6838284 Pulled By: siying fbshipit-source-id: 20893fa28d40fadc31e4ff154bed02f5a1bad341 | 29 January 2018, 22:27:32 UTC |
| b8eb32f | Mark Isaacson | 29 January 2018, 20:43:56 UTC | Suppress lint in old files Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues. Reviewed By: yiwu-arbug Differential Revision: D6821806 fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c | 29 January 2018, 20:56:42 UTC |
| 9f7ccc8 | Andrew Kryczka | 29 January 2018, 19:30:37 UTC | fix db_bench filluniquerandom key count assertion Summary: It failed every time. I guess people usually ran with assertions disabled. Closes https://github.com/facebook/rocksdb/pull/3422 Differential Revision: D6822984 Pulled By: ajkr fbshipit-source-id: 2e90db75618b26ac1c46ddfa9e03c095c7bf16e3 | 29 January 2018, 19:43:21 UTC |
| 3f666f7 | Mamy Ratsimbazafy | 29 January 2018, 17:51:13 UTC | Add Nim to the list of language bindings Summary: Closes https://github.com/facebook/rocksdb/pull/3428 Differential Revision: D6834061 Pulled By: maysamyabandeh fbshipit-source-id: edca5b5b8330e0fee646c7434b9631da76670240 | 29 January 2018, 17:57:46 UTC |
| 65cd6cd | Ben Darnell | 29 January 2018, 17:48:58 UTC | Rewrite comments on use_fsync option Summary: This replaces a vague warning about the mostly-obsolete ext3 filesystem with a more detailed note about a historical bug in the still-relevant ext4. Fixes #3410 Closes https://github.com/facebook/rocksdb/pull/3421 Differential Revision: D6834881 Pulled By: siying fbshipit-source-id: 7771ef5c89a54c0ac17821680779c48178d0b400 | 29 January 2018, 17:57:46 UTC |
