Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
f453bcb Jermy Li 18 December 2019, 19:45:18 UTC Add unit tests for concurrent CF iteration and drop (#6180) Summary: improve https://github.com/facebook/rocksdb/issues/6147 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6180 Differential Revision: D19148936 fbshipit-source-id: f691c9879fd51d54e96c1a99670cf85ca4485a89 18 December 2019, 19:54:35 UTC
02193ce sdong 18 December 2019, 18:59:21 UTC Prevent file prefetch when mmap is enabled. (#6206) Summary: Right now, sometimes file prefetching is still on when mmap is enabled. This causes bug of reading wrong data. In this commit, we remove all those possible paths. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6206 Test Plan: make crash_test with compaction_readahead_size, which used to fail. RUn all existing tests. Differential Revision: D19149429 fbshipit-source-id: 9e18ea8c566e416aac9647bdd05afe596634791b 18 December 2019, 19:01:29 UTC
dfb259e Peter Dillinger 18 December 2019, 17:31:05 UTC Fix syntax error (!) in db_crashtest.py (#6207) Summary: Fixes syntax error from https://github.com/facebook/rocksdb/pull/6203 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6207 Test Plan: make blackbox_crash_test -> no more syntax error Differential Revision: D19161752 Pulled By: pdillinger fbshipit-source-id: b3032f296041ab56307762622b9ef6c03a8379aa 18 December 2019, 17:32:52 UTC
c399704 Zhichao Cao 18 December 2019, 08:48:48 UTC Fix: remove the potential dead store variable in block_based_table_reader.cc (#6204) Summary: buf_offset does not need to get the value from req.len for othe final block. It can cause test fail for clan_analyze. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6204 Test Plan: pass make asan_check Differential Revision: D19145335 Pulled By: zhichao-cao fbshipit-source-id: 8f6e74565746381b5c5ef598b97d746517b36e5b 18 December 2019, 09:23:07 UTC
2afea29 anand76 18 December 2019, 04:43:06 UTC Add VerifyChecksum() to db_stress (#6203) Summary: Add an option to db_stress, verify_checksum_one_in, to call DB::VerifyChecksum() once every N ops. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6203 Differential Revision: D19145753 Pulled By: anand1976 fbshipit-source-id: d09edf21f309ad53aa40dd25b7a563d50665fd8b 18 December 2019, 04:44:58 UTC
ce63eda Mike Kolupaev 18 December 2019, 04:07:21 UTC Fix use-after-free and double-deleting files in BackgroundCallPurge() (#6193) Summary: The bad code was: ``` mutex.Lock(); // `mutex` protects `container` for (auto& x : container) { mutex.Unlock(); // do stuff to x mutex.Lock(); } ``` It's incorrect because both `x` and the iterator may become invalid if another thread modifies the container while this thread is not holding the mutex. Broken by https://github.com/facebook/rocksdb/pull/5796 - it replaced a `while (!container.empty())` loop with a `for (auto x : container)`. (RocksDB code does a lot of such unlocking+re-locking of mutexes, and this type of bugs comes up a lot :/ ) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6193 Test Plan: Ran some logdevice integration tests that were crashing without this fix. Differential Revision: D19116874 Pulled By: al13n321 fbshipit-source-id: 9672bc4227c1b68f46f7436db2b96811adb8c703 18 December 2019, 04:08:56 UTC
cbd58af Levi Tamasi 18 December 2019, 03:07:02 UTC Update HISTORY.md with recent BlobDB related changes Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6202 Differential Revision: D19144158 Pulled By: ltamasi fbshipit-source-id: 3e2522ced458568e3a2a045663704e30ab0ac223 18 December 2019, 03:09:21 UTC
9f250dd sdong 18 December 2019, 02:23:26 UTC crash_test: two fixes (#6200) Summary: Fix two crash test issues: 1. sync mode should not run with disable_wal=true 2. disable "compaction_readahead_size" for now. With it on, some block checksum verification failure will happen in compaction paths. Not sure why, but disable it for now to keep the test clean. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6200 Test Plan: Run "make crash_test" and "make crash_test_with_atomic_flush" and see it runs way longer than before the fix without failing. Differential Revision: D19143493 fbshipit-source-id: 438fad52fbda60aafd142e1b65578addbe7d72b1 18 December 2019, 02:25:04 UTC
2d16709 Adam Retter 17 December 2019, 21:52:09 UTC Small tidy and speed up of the travis build (#6181) Summary: Cuts about 30-60 seconds to from each Travis Linux build, and about 15 minutes from each macOS build Pull Request resolved: https://github.com/facebook/rocksdb/pull/6181 Differential Revision: D19098357 Pulled By: pdillinger fbshipit-source-id: 863dd1ab09076ad9b03c2b7914908359628315ae 17 December 2019, 21:56:45 UTC
39fcaf8 解轶伦 17 December 2019, 21:20:42 UTC delete superversions in BackgroundCallPurge (#6146) Summary: I found that CleanupSuperVersion() may block Get() for 30ms+ （per MemTable is 256MB）. Then I found "delete sv" in ~SuperVersion() takes the time. The backtrace looks like this DBImpl::GetImpl() -> DBImpl::ReturnAndCleanupSuperVersion() -> DBImpl::CleanupSuperVersion() : delete sv; -> ~SuperVersion() I think it's better to delete in a background thread, please review it。 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6146 Differential Revision: D18972066 fbshipit-source-id: 0f7b0b70b9bb1e27ad6fc1c8a408fbbf237ae08c 17 December 2019, 21:22:57 UTC
02aa229 Levi Tamasi 17 December 2019, 18:17:27 UTC Set CompactionIterator::valid_ to false when PrepareBlobOutput indicates error Summary: With https://github.com/facebook/rocksdb/pull/6121, errors returned by `PrepareBlobValue` result in `CompactionIterator::status_` being set to `Corruption` or `IOError` as appropriate, however, `valid_` is not set to `false`. The error is eventually propagated in `CompactionJob::ProcessKeyValueCompaction` but only after the main loop completes. Setting `valid_` to `false` upon errors enables us to terminate the loop early and fail the compaction sooner. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6170 Test Plan: Ran `make check` and used `db_bench` in BlobDB mode. fbshipit-source-id: a2ca88a3ca71115e2605bd34a4c795d8a28bef27 17 December 2019, 18:20:16 UTC
1be48cb anand1976 17 December 2019, 04:35:11 UTC Fix crash in Transaction::MultiGet() when num_keys > 32 Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6192 Test Plan: Add a unit test that fails without the fix and passes now make check Differential Revision: D19124781 Pulled By: anand1976 fbshipit-source-id: 8c8cb6fa16c3fc23ec011e168561a13f76bbd783 17 December 2019, 04:39:35 UTC
7678cf2 Yanqin Jin 17 December 2019, 04:00:43 UTC Use Env::LoadEnv to create custom Env objects (#6196) Summary: As title. Previous assumption was that the underlying lib can always return a shared_ptr<Env>. This is too strong. Therefore, we use Env::LoadEnv to relax it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6196 Test Plan: make check Differential Revision: D19133199 Pulled By: riversand963 fbshipit-source-id: c83a0c02a42610d077054f2de1acfc45126b3a75 17 December 2019, 04:03:14 UTC
68d5d82 Maysam Yabandeh 17 December 2019, 02:30:48 UTC Wait for CancelAllBackgroundWork before Close in db stress (#6191) Summary: In https://github.com/facebook/rocksdb/issues/6174 we fixed the stress test to respect the CancelAllBackgroundWork + Close order for WritePrepared transactions. The fix missed to take into account that some invocation of CancelAllBackgroundWork are with wait=false parameter which essentially breaks the order. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6191 Differential Revision: D19102709 Pulled By: maysamyabandeh fbshipit-source-id: f4e7b5fdae47ff1c1ac284ba1cf67d5d3f3d03eb 17 December 2019, 02:33:09 UTC
cddd637 Zhichao Cao 16 December 2019, 23:55:33 UTC Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089) Summary: In the current MultiGet, if the KV-pairs do not belong to the data blocks in the block cache, multiple blocks are read from a SST. It will trigger one block read for each block request and read them in parallel. In some cases, if some data blocks are adjacent in the SST, the reads for these blocks can be combined to a single large read, which can reduce the system calls and reduce the read latency if possible. Considering to fill the block cache, if multiple data blocks are in the same memory buffer, we need to copy them to the heap separately. Therefore, only in the case that 1) data block compression is enabled, and 2) compressed block cache is null, we can do combined read. Otherwise, extra memory copy is needed, which may cause extra overhead. In the current case, data blocks will be uncompressed to a new memory space. Also, in the case that 1) data block compression is enabled, and 2) compressed block cache is null, it is possible the data block is actually not compressed. In the current logic, these data blocks will not be added to the uncompressed_cache. So if memory buffer is shared and the data block is not compressed, the data block are copied to the head and fill the cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6089 Test Plan: Added test case to ParallelIO.MultiGet. Pass make asan_check Differential Revision: D18734668 Pulled By: zhichao-cao fbshipit-source-id: 67c5615ed373e51e42635fd74b36f8f3a66d5da4 17 December 2019, 00:26:03 UTC
bcc372c sdong 16 December 2019, 23:24:26 UTC Add some new options to crash_test (#6176) Summary: Several options are trivially added to crash test and random values are picked. Made simple test run non-dynamic level and normal test run dynamic level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6176 Test Plan: Run crash_test and watch the printing Differential Revision: D19053955 fbshipit-source-id: 958cb43c968541ebd87ed4d91e778bd1d40e7502 16 December 2019, 23:43:13 UTC
2d095b4 Levi Tamasi 16 December 2019, 23:15:42 UTC Update HISTORY.md with the recent memtable trimming fixes Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6194 Differential Revision: D19125292 Pulled By: ltamasi fbshipit-source-id: d41aca2755ec4bec07feedd6b561e8d18606a931 16 December 2019, 23:19:52 UTC
35126dd sdong 16 December 2019, 22:28:06 UTC db_stress: preserve all historic manifest files (#6142) Summary: compaction history is stored in manifest files. Preserve all of them in db_stress would help debugging. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6142 Test Plan: Run db_stress and observe that manifest files are preserved. Run whole crash_test and see how DB directory looks like. Differential Revision: D19047026 fbshipit-source-id: f83c3e0bb5332b1b4768be5dcee56a24f9b760a9 16 December 2019, 22:32:34 UTC
fbda25f Zhichao Cao 16 December 2019, 21:59:21 UTC db_stress: generate the key based on Zipfian distribution (hot key) (#6163) Summary: In the current db_stress, all the keys are generated randomly and follows the uniform distribution. In order to test some corner cases that some key are always updated or read, we need to generate the key based on other distributions. In this PR, the key is generated based on Zipfian distribution and the skewness can be controlled by setting hot_key_alpha (0.8 to 1.5 is suggested). The larger hot_key_alpha is, the more skewed will be. Not that, usually, if hot_key_alpha is larger than 2, there might be only 1 or 2 keys that are generated. If hot_key_alpha is 0, it generate the key follows uniform distribution (random key) Testing plan: pass the db_stress and printed the keys to make sure it follows the distribution. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6163 Differential Revision: D18978480 Pulled By: zhichao-cao fbshipit-source-id: e123b4865477f7478e83fb581f9576bada334680 16 December 2019, 22:01:58 UTC
db7c687 Levi Tamasi 16 December 2019, 21:13:42 UTC Fix a data race related to memtable trimming (#6187) Summary: https://github.com/facebook/rocksdb/pull/6177 introduced a data race involving `MemTableList::InstallNewVersion` and `MemTableList::NumFlushed`. The patch fixes this by caching whether the current version has any memtable history (i.e. flushed memtables that are kept around for transaction conflict checking) in an `std::atomic<bool>` member called `current_has_history_`, similarly to how `current_memory_usage_excluding_last_` is handled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6187 Test Plan: ``` make clean COMPILE_WITH_TSAN=1 make db_test -j24 ./db_test ``` Differential Revision: D19084059 Pulled By: ltamasi fbshipit-source-id: 327a5af9700fb7102baea2cc8903c085f69543b9 16 December 2019, 21:16:31 UTC
a92bd0a Peter Dillinger 16 December 2019, 05:29:43 UTC Optimize memory and CPU for building new Bloom filter (#6175) Summary: The filter bits builder collects all the hashes to add in memory before adding them (because the number of keys is not known until we've walked over all the keys). Existing code uses a std::vector for this, which can mean up to 2x than necessary space allocated (and not freed) and up to ~2x write amplification in memory. Using std::deque uses close to minimal space (for large filters, the only time it matters), no write amplification, frees memory while building, and no need for large contiguous memory area. The only cost is more calls to allocator, which does not appear to matter, at least in benchmark test. For now, this change only applies to the new (format_version=5) Bloom filter implementation, to ease before-and-after comparison downstream. Temporary memory use during build is about the only way the new Bloom filter could regress vs. the old (because of upgrade to 64-bit hash) and that should only matter for full filters. This change should largely mitigate that potential regression. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6175 Test Plan: Using filter_bench with -new_builder option and 6M keys per filter is like large full filter (improvement). 10k keys and no -new_builder is like partitioned filters (about the same). (Corresponding configurations run simultaneously on devserver.) std::vector impl (before) $ /usr/bin/time -v ./filter_bench -impl=2 -quick -new_builder -working_mem_size_mb=1000 - average_keys_per_filter=6000000 Build avg ns/key: 52.2027 Maximum resident set size (kbytes): 1105016 $ /usr/bin/time -v ./filter_bench -impl=2 -quick -working_mem_size_mb=1000 - average_keys_per_filter=10000 Build avg ns/key: 30.5694 Maximum resident set size (kbytes): 1208152 std::deque impl (after) $ /usr/bin/time -v ./filter_bench -impl=2 -quick -new_builder -working_mem_size_mb=1000 - average_keys_per_filter=6000000 Build avg ns/key: 39.0697 Maximum resident set size (kbytes): 1087196 $ /usr/bin/time -v ./filter_bench -impl=2 -quick -working_mem_size_mb=1000 - average_keys_per_filter=10000 Build avg ns/key: 30.9348 Maximum resident set size (kbytes): 1207980 Differential Revision: D19053431 Pulled By: pdillinger fbshipit-source-id: 2888e748723a19d9ea40403934f13cbb8483430c 16 December 2019, 05:31:08 UTC
ad34fab anand76 14 December 2019, 23:38:04 UTC Fix unity test (#6178) Summary: Fix the test failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6178 Differential Revision: D19071208 Pulled By: maysamyabandeh fbshipit-source-id: 71622832ac93ff2663946c546d9642d5b9e3d194 14 December 2019, 23:39:41 UTC
4b97812 Maysam Yabandeh 14 December 2019, 23:17:05 UTC Add long-running snapshots to stress tests (#6171) Summary: Current implementation holds on to 10% of snapshots for 10x longer, and 1% of snapshots 100x longer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6171 Test Plan: ``` make -j32 crash_test Differential Revision: D19038399 Pulled By: maysamyabandeh fbshipit-source-id: 75da2dbb5c47a0b3f37d299b8719e392b73b42c0 14 December 2019, 23:22:40 UTC
bd8404f Levi Tamasi 14 December 2019, 03:09:53 UTC Do not schedule memtable trimming if there is no history (#6177) Summary: We have observed an increase in CPU load caused by frequent calls to `ColumnFamilyData::InstallSuperVersion` from `DBImpl::TrimMemtableHistory` when using `max_write_buffer_size_to_maintain` to limit the amount of memtable history maintained for transaction conflict checking. Part of the issue is that trimming can potentially be scheduled even if there is no memtable history. The patch adds a check that fixes this. See also https://github.com/facebook/rocksdb/pull/6169. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6177 Test Plan: Compared `perf` output for ``` ./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot=1 --threads=32 ``` before and after the change. There is a significant reduction for the call chain `rocksdb::DBImpl::TrimMemtableHistory` -> `rocksdb::ColumnFamilyData::InstallSuperVersion` -> `rocksdb::ThreadLocalPtr::StaticMeta::Scrape` even without https://github.com/facebook/rocksdb/pull/6169. Differential Revision: D19057445 Pulled By: ltamasi fbshipit-source-id: dff81882d7b280e17eda7d9b072a2d4882c50f79 14 December 2019, 03:11:19 UTC
349bd3e Maysam Yabandeh 14 December 2019, 01:56:12 UTC CancelAllBackgroundWork before Close in db stress (#6174) Summary: Close asserts that there is no unreleased snapshots. For WritePrepared transaction, this means that the background work that holds on a snapshot must be canceled first. Update the stress tests to respect the sequence. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6174 Test Plan: ``` make -j32 crash_test Differential Revision: D19057322 Pulled By: maysamyabandeh fbshipit-source-id: c9e9e24f779bbfb0ab72c2717e34576c01bc6362 14 December 2019, 02:22:50 UTC
edbf0e2 Adam Retter 14 December 2019, 00:11:40 UTC Env should also load the native library (#6167) Summary: Closes https://github.com/facebook/rocksdb/issues/6118 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6167 Differential Revision: D19053577 Pulled By: pdillinger fbshipit-source-id: 86aca9a5bec0947a641649b515da17b3cb12bdde 14 December 2019, 00:27:55 UTC
0d2172f Levi Tamasi 13 December 2019, 23:52:38 UTC Make it possible to enable periodic compactions for BlobDB (#6172) Summary: Periodic compactions ensure that even SSTs that do not get picked up otherwise eventually go through compaction; used in conjunction with BlobDB's garbage collection, they enable BlobDB to reclaim space when old blob files are used by such straggling SSTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6172 Test Plan: Ran `make check` and used the BlobDB mode of `db_bench`. Differential Revision: D19045045 Pulled By: ltamasi fbshipit-source-id: 04636ecc4b6cfe8d495bf656faa65d54a5eb1a93 14 December 2019, 00:13:25 UTC
afa2420 anand76 13 December 2019, 22:47:08 UTC Introduce a new storage specific Env API (#5761) Summary: The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc. This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO. The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before. This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection. The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761 Differential Revision: D18868376 Pulled By: anand1976 fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f 13 December 2019, 22:48:41 UTC
58d46d1 Peter Dillinger 13 December 2019, 21:25:14 UTC Add useful idioms to Random API (OneInOpt, PercentTrue) (#6154) Summary: And clean up related code, especially in stress test. (More clean up of db_stress_test_base.cc coming after this.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6154 Test Plan: make check, make blackbox_crash_test for a bit Differential Revision: D18938180 Pulled By: pdillinger fbshipit-source-id: 524d27621b8dbb25f6dff40f1081e7c00630357e 13 December 2019, 22:30:14 UTC
6d54eb3 Levi Tamasi 13 December 2019, 20:45:49 UTC Do not create/install new SuperVersion if nothing was deleted during memtable trim (#6169) Summary: We have observed an increase in CPU load caused by frequent calls to `ColumnFamilyData::InstallSuperVersion` from `DBImpl::TrimMemtableHistory` when using `max_write_buffer_size_to_maintain` to limit the amount of memtable history maintained for transaction conflict checking. As it turns out, this is caused by the code creating and installing a new `SuperVersion` even if no memtables were actually trimmed. The patch adds a check to avoid this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6169 Test Plan: Compared `perf` output for ``` ./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot=1 --threads=32 ``` before and after the change. With the fix, the call chain `rocksdb::DBImpl::TrimMemtableHistory` -> `rocksdb::ColumnFamilyData::InstallSuperVersion` -> `rocksdb::ThreadLocalPtr::StaticMeta::Scrape` no longer registers in the `perf` report. Differential Revision: D19031509 Pulled By: ltamasi fbshipit-source-id: 02686fce594e5b50eba0710e4b28a9b808c8aa20 13 December 2019, 21:29:29 UTC
ac304ad Kefu Chai 13 December 2019, 19:33:59 UTC cmake: do not build tests for Release build and cleanups (#5916) Summary: fixes https://github.com/facebook/rocksdb/issues/2445 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5916 Differential Revision: D19031236 fbshipit-source-id: bc3107b6b25a01958677d7cb411b1f381aae91c6 13 December 2019, 20:48:06 UTC
fec7302 Maysam Yabandeh 13 December 2019, 18:23:01 UTC Enable unordered_write in stress tests (#6164) Summary: With WritePrepared transactions configured with two_write_queues, unordered_write will offer the same guarantees as vanilla rocksdb and thus can be enabled in stress tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6164 Test Plan: ``` make -j32 crash_test_with_txn Differential Revision: D18991899 Pulled By: maysamyabandeh fbshipit-source-id: eece5e96b4169b67d7931e5c0afca88540a113e1 13 December 2019, 18:25:04 UTC
583c695 Levi Tamasi 13 December 2019, 18:11:03 UTC Move out valid blobs from the oldest blob files during compaction (#6121) Summary: The patch adds logic that relocates live blobs from the oldest N non-TTL blob files as they are encountered during compaction (assuming the BlobDB configuration option `enable_garbage_collection` is `true`), where N is defined as the number of immutable non-TTL blob files multiplied by the value of a new BlobDB configuration option called `garbage_collection_cutoff`. (The default value of this parameter is 0.25, that is, by default the valid blobs residing in the oldest 25% of immutable non-TTL blob files are relocated.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6121 Test Plan: Added unit test and tested using the BlobDB mode of `db_bench`. Differential Revision: D18785357 Pulled By: ltamasi fbshipit-source-id: 8c21c512a18fba777ec28765c88682bb1a5e694e 13 December 2019, 18:13:05 UTC
c2029f9 Jermy Li 13 December 2019, 03:02:51 UTC Support concurrent CF iteration and drop (#6147) Summary: It's easy to cause coredump when closing ColumnFamilyHandle with unreleased iterators, especially iterators release is controlled by java GC when using JNI. This patch fixed concurrent CF iteration and drop, we let iterators(actually SuperVersion) hold a ColumnFamilyData reference to prevent the CF from being released too early. fixed https://github.com/facebook/rocksdb/issues/5982 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6147 Differential Revision: D18926378 fbshipit-source-id: 1dff6d068c603d012b81446812368bfee95a5e15 13 December 2019, 03:04:48 UTC
4b74035 myasuka 13 December 2019, 02:06:01 UTC Correct java docs of RocksDB options (#6123) Summary: Correct javadocs of several RocksDB option classes to not mislead RocksJava users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6123 Differential Revision: D18989044 Pulled By: pdillinger fbshipit-source-id: a5ac6a415e5311084b10d973d354e6925788f01e 13 December 2019, 02:10:03 UTC
c4ce8e6 奏之章 12 December 2019, 23:16:13 UTC Fix RangeDeletion bug (#6062) Summary: Read keys from a snapshot that a range deletion were added after the snapshot was created and this range deletion was inside an immutable memtable, we will get wrong key set. More detail rest in codes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6062 Differential Revision: D18966785 Pulled By: pdillinger fbshipit-source-id: 38a60bb1e2d0a1dbfc8ec641617200b6a02b86c3 12 December 2019, 23:18:02 UTC
a844591 Connor 12 December 2019, 22:05:48 UTC wait pending memtable writes on file ingestion or compact range (#6113) Summary: **Summary:** This PR fixes two unordered_write related issues: - ingestion job may skip the necessary memtable flush https://github.com/facebook/rocksdb/issues/6026 - compact range may cause memtable is flushed before pending unordered write finished 1. `CompactRange` triggers memtable flush but doesn't wait for pending-writes 2. there are some pending writes but memtable is already flushed 3. the memtable related WAL is removed( note that the pending-writes were recorded in that WAL). 4. pending-writes write to newer created memtable 5. there is a restart 6. missing the previous pending-writes because WAL is removed but they aren't included in SST. **How to solve:** - Wait pending memtable writes before ingestion job check memtable key range - Wait pending memtable writes before flush memtable. **Note that: `CompactRange` calls `RangesOverlapWithMemtables` too without waiting for pending waits, but I'm not sure whether it affects the correctness.** **Test Plan:** make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6113 Differential Revision: D18895674 Pulled By: maysamyabandeh fbshipit-source-id: da22b4476fc7e06c176020e7cc171eb78189ecaf 12 December 2019, 22:08:02 UTC
814d4e7 sdong 12 December 2019, 22:02:43 UTC Improve instructions to install formatter (#6162) Summary: While the instruction of installing "make format" dependencies works on some platforms, it is hard to use for some others. Improve it a little bit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6162 Test Plan: Run "make format" on an envrionment missing the dependencies and see the instructions printed out Differential Revision: D18970773 fbshipit-source-id: fd21b31053407cc171a6675f781a556a1c3e8945 12 December 2019, 22:04:01 UTC
a796c06 Maysam Yabandeh 12 December 2019, 21:48:50 UTC Fix build breakage from lock_guard error (#6161) Summary: This change fixes a source issue that caused compile time error which breaks build for many fbcode services in that setup. The size() member function of channel is a const member, so member variables accessed within it are implicitly const as well. This caused error when clang fails to resolve to a constructor that takes std::mutex because the suitable constructor got rejected due to loss of constness for its argument. The fix is to add mutable modifier to the lock_ member of channel. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6161 Differential Revision: D18967685 Pulled By: maysamyabandeh fbshipit-source-id: 698b6a5153c3c92eeacb842c467aa28cc350d432 12 December 2019, 21:50:27 UTC
b433bbe Adam Retter 12 December 2019, 19:58:26 UTC Add missing mutable DBOptions to RocksJava (#6152) Summary: As requested in https://github.com/facebook/rocksdb/issues/6127 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6152 Differential Revision: D18955608 Pulled By: pdillinger fbshipit-source-id: 3e1367d944e44d5f1675a422f7dd2451c86feb6f 12 December 2019, 20:01:19 UTC
3b60761 Levi Tamasi 12 December 2019, 19:29:01 UTC Do not update SST <-> blob file mapping if compaction failed Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6156 Test Plan: Extended unit tests. Differential Revision: D18943867 Pulled By: ltamasi fbshipit-source-id: b3669d2dd6af08e987ad1a59d6712ae2514da0b1 12 December 2019, 19:30:45 UTC
8613ee2 Maysam Yabandeh 12 December 2019, 18:34:52 UTC Enable all txn write policies in crash test (#6158) Summary: Currently the default txn write policy in crash tests is WRITE_PREPARED. The patch randomly picks the write policy at the start of the crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6158 Test Plan: ``` make -j32 crash_test_with_txn ``` Differential Revision: D18946307 Pulled By: maysamyabandeh fbshipit-source-id: f77d7a94f99a08791ef9626da153d284bf521950 12 December 2019, 18:43:49 UTC
e1dfe80 Levi Tamasi 12 December 2019, 01:17:56 UTC Mark BlobIndex::DebugString const Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6157 Test Plan: make check Differential Revision: D18944259 Pulled By: ltamasi fbshipit-source-id: 7fb29447b52d801215bd6ab811e229a7fa2c763d 12 December 2019, 01:19:43 UTC
1ad6fa9 Maysam Yabandeh 11 December 2019, 23:59:45 UTC Enable txn in crash tests (#6155) Summary: Start daily crash tests with use_txn flag. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6155 Differential Revision: D18943630 Pulled By: maysamyabandeh fbshipit-source-id: eea99a6ffd5f57fb9651f6ca7dab8fbf70379c87 12 December 2019, 00:01:55 UTC
d0ad3c5 Peter Dillinger 11 December 2019, 23:15:34 UTC Fix c_test:filter for various CACHE_LINE_SIZEs (#6153) Summary: This test was recently updated but failed to account for Bloom schema variance by CACHE_LINE_SIZE. (Since CACHE_LINE_SIZE is not defined in our C code, the test now simply allows a valid result for any CACHE_LINE_SIZE, not just the current one.) Unblock https://github.com/facebook/rocksdb/issues/5932 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6153 Test Plan: ran unit test with builds TEST_CACHE_LINE_SIZE=128, =256, and unset (64 on Intel) Differential Revision: D18936015 Pulled By: pdillinger fbshipit-source-id: e5e3852f95283d34d624632c1ae8d3adb2f2662c 11 December 2019, 23:17:08 UTC
3717a88 奏之章 11 December 2019, 19:25:33 UTC Fix UniversalCompaction trivial move bug (#6067) Summary: `curr.level` is `c->inputs_` index, not real level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6067 Differential Revision: D18935726 fbshipit-source-id: 4354e6e9cd900ca56c96e9d770f0ab6634e45daf 11 December 2019, 19:27:53 UTC
afdc58d ferhat elmas 11 December 2019, 19:02:22 UTC Fix typos in history Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6116 Differential Revision: D18935622 fbshipit-source-id: 59f7a7bc9f0116ae6354ea217896622a34329d3c 11 December 2019, 19:04:46 UTC
05a8631 Yi Wu 11 December 2019, 18:26:35 UTC Remove unused low_pri_write_rate_limiter_ (#6068) Summary: `low_pri_write_rate_limiter_` is not being used. Removing. `WriteController` has an internal low_pri rate limiter which is the real rate limiter for low-pri writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6068 Test Plan: make Differential Revision: D18664120 fbshipit-source-id: dfe3e4de033cf3522b67781b383aad7d0936034c 11 December 2019, 18:28:33 UTC
77565d7 Cheng Chang 11 December 2019, 17:55:20 UTC Add example to show the effect of Get in snapshot isolation (#6059) Summary: Adds example to show the difference of reading from snapshot and from the latest state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6059 Test Plan: cd examples && make transaction_example && ./transaction_example Differential Revision: D18797616 fbshipit-source-id: f17a2cb12187092ea243159e6ccf55790859e0c0 11 December 2019, 17:56:42 UTC
383f507 Yanqin Jin 11 December 2019, 05:53:43 UTC Add SyncWAL to db_stress (#6149) Summary: Add SyncWAL to db_stress. Specify with `-sync_wal_one_in=N` so that it will be called once every N operations on average. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6149 Test Plan: ``` $make db_stress $./db_stress -sync_wal_one_in=100 -ops_per_thread=100000 ``` Differential Revision: D18922529 Pulled By: riversand963 fbshipit-source-id: 4c0b8cb8fa21852722cffd957deddf688f12ea56 11 December 2019, 05:55:25 UTC
7a99162 sdong 11 December 2019, 04:01:25 UTC db_stress: sometimes call CancelAllBackgroundWork() and Close() before closing DB (#6141) Summary: CancelAllBackgroundWork() and Close() are frequently used features but we don't cover it in stress test. Simply execute them before closing the DB with 1/2 chance. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6141 Test Plan: Run "db_stress". Differential Revision: D18900861 fbshipit-source-id: 49b46ccfae120d0f9de3e0543b82fb6d715949d0 11 December 2019, 04:04:52 UTC
984b6e7 Adam Retter 11 December 2019, 04:00:57 UTC Add Visual Studio 2015 to AppVeyor (#5446) Summary: This is required to compile on Windows with Visual Studio 2015, which is used for creating the RocksJava releases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5446 Differential Revision: D18924811 fbshipit-source-id: a183a62e79a2af5aaf59cd08235458a172fe7dcb 11 December 2019, 04:02:31 UTC
a653857 Peter Dillinger 10 December 2019, 23:45:25 UTC Add PauseBackgroundWork() to db_stress (#6148) Summary: Worker thread will occasionally call PauseBackgroundWork(), briefly sleep (to avoid stalling itself) and then call ContinueBackgroundWork(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6148 Test Plan: some running of 'make blackbox_crash_test' with temporary printf output to confirm code occasionally reached. Differential Revision: D18913886 Pulled By: pdillinger fbshipit-source-id: ae9356a803390929f3165dfb6a00194692ba92be 10 December 2019, 23:46:48 UTC
2bb5fc1 Adam Simpkins 10 December 2019, 23:19:24 UTC Add an option to the CMake build to disable building shared libraries (#6122) Summary: Add an option to explicitly disable building shared versions of the RocksDB libraries. The shared libraries cannot be built in cases where some dependencies are only available as static libraries. This allows still building RocksDB in these situations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6122 Differential Revision: D18920740 fbshipit-source-id: d24f66d93c68a1e65635e6e0b663bae62c903bca 10 December 2019, 23:20:50 UTC
2b060c1 Yanqin Jin 10 December 2019, 19:47:20 UTC Use Env::GetChildren() instead of readdir (#6139) Summary: For more portability, switch from readdir to Env::GetChildren() in ldb's manifest_dump subcommand. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6139 Test Plan: ``` $make check ``` Manually check ldb command. Differential Revision: D18898197 Pulled By: riversand963 fbshipit-source-id: 92afca379e9fbe78ab70b2eb40d127daad8df5e2 10 December 2019, 19:49:09 UTC
14c38ba sdong 10 December 2019, 19:40:09 UTC db_stress: sometimes validate compact range data (#6140) Summary: Right now, in db_stress, compact range is simply executed without any immediate data validation. Add a simply validation which compares hash for all keys within the compact range to stay the same against the same snapshot before and after the compaction. Also, randomly tune most knobs of CompactRangeOptions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6140 Test Plan: Run db_stress with "--compact_range_one_in=2000 --compact_range_width=100000000" for a while. Manually ingest some hacky code and observe the error path. Differential Revision: D18900230 fbshipit-source-id: d96e75bc8c38dd5ec702571ffe7cf5f4ea93ee10 10 December 2019, 19:41:50 UTC
1dd3194 Jermy Li 10 December 2019, 19:21:58 UTC Fix compile error "folly/xx.h file not found" on Mac OS (#6145) Summary: Error message when running `make` on Mac OS with master branch (v6.6.0): ``` $ make $DEBUG_LEVEL is 1 Makefile:168: Warning: Compiling in debug mode. Don't use the resulting binary in production third-party/folly/folly/synchronization/WaitOptions.cpp:6:10: fatal error: 'folly/synchronization/WaitOptions.h' file not found #include <folly/synchronization/WaitOptions.h> ^ 1 error generated. third-party/folly/folly/synchronization/ParkingLot.cpp:6:10: fatal error: 'folly/synchronization/ParkingLot.h' file not found #include <folly/synchronization/ParkingLot.h> ^ 1 error generated. third-party/folly/folly/synchronization/DistributedMutex.cpp:6:10: fatal error: 'folly/synchronization/DistributedMutex.h' file not found #include <folly/synchronization/DistributedMutex.h> ^ 1 error generated. third-party/folly/folly/synchronization/AtomicNotification.cpp:6:10: fatal error: 'folly/synchronization/AtomicNotification.h' file not found #include <folly/synchronization/AtomicNotification.h> ^ 1 error generated. third-party/folly/folly/detail/Futex.cpp:6:10: fatal error: 'folly/detail/Futex.h' file not found #include <folly/detail/Futex.h> ^ 1 error generated. GEN util/build_version.cc $DEBUG_LEVEL is 1 Makefile:168: Warning: Compiling in debug mode. Don't use the resulting binary in production third-party/folly/folly/synchronization/WaitOptions.cpp:6:10: fatal error: 'folly/synchronization/WaitOptions.h' file not found #include <folly/synchronization/WaitOptions.h> ^ 1 error generated. third-party/folly/folly/synchronization/ParkingLot.cpp:6:10: fatal error: 'folly/synchronization/ParkingLot.h' file not found #include <folly/synchronization/ParkingLot.h> ^ 1 error generated. third-party/folly/folly/synchronization/DistributedMutex.cpp:6:10: fatal error: 'folly/synchronization/DistributedMutex.h' file not found #include <folly/synchronization/DistributedMutex.h> ^ 1 error generated. third-party/folly/folly/synchronization/AtomicNotification.cpp:6:10: fatal error: 'folly/synchronization/AtomicNotification.h' file not found #include <folly/synchronization/AtomicNotification.h> ^ 1 error generated. third-party/folly/folly/detail/Futex.cpp:6:10: fatal error: 'folly/detail/Futex.h' file not found #include <folly/detail/Futex.h> ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6145 Differential Revision: D18910812 fbshipit-source-id: 5a4475466c2d0601657831a0b48d34316b2f0816 10 December 2019, 19:24:11 UTC
6380df5 Peter Dillinger 10 December 2019, 16:38:23 UTC Vary bloom_bits in db_crashtest (#6103) Summary: Especially with non-integral bits/key now supported, db_crashtest should vary the bloom_bits configuration. The probabilities look like this: 1/2 chance of a uniform int from 0 to 19. This includes overall 1/40 chance of 0 which disables the bloom filter. 1/2 chance of a float from a lognormal distribution with a median of 10. This always produces positive values but with a decent chance of < 1 (overall ~1/40) or > 100 (overall ~1/40), the enforced/coerced implementation limits. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6103 Test Plan: start 'make blackbox_crash_test' several times and look at configuration output Differential Revision: D18734877 Pulled By: pdillinger fbshipit-source-id: 4a38cb057d3b3fc1327f93199f65b9a9ffbd7316 10 December 2019, 16:39:50 UTC
a68dff5 sdong 09 December 2019, 23:48:04 UTC Apply formatter to some recent commits (#6138) Summary: Formatter somehow complains some recent lines changed. Apply them to make the formatter happy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6138 Test Plan: See CI passes. Differential Revision: D18895950 fbshipit-source-id: 7d1696cf3e3a682bc10a30cdca748a23c6565255 09 December 2019, 23:49:49 UTC
a960287 sdong 09 December 2019, 22:36:10 UTC db_stress: Some code style improvements (#6137) Summary: Two changes: 1. Prevent static variables in a header file 2. Add "override" keyword when virtual functions are overridden. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6137 Test Plan: Build db_stress with or without LITE. Differential Revision: D18892007 fbshipit-source-id: 295356427a34473b23ed36d6ed4ef3ae35a32db0 09 December 2019, 22:38:42 UTC
e43d2c4 Peter Dillinger 09 December 2019, 20:19:50 UTC Fix & test rocksdb_filterpolicy_create_bloom_full (#6132) Summary: Add overrides needed in FilterPolicy wrapper to fix rocksdb_filterpolicy_create_bloom_full (see issue https://github.com/facebook/rocksdb/issues/6129). Re-enabled assertion in BloomFilterPolicy::CreateFilter that was being violated. Expanded c_test to identify Bloom filter implementations by FP counts. (Without the fix, updated test will trigger assertion and fail otherwise without the assertion.) Fixes https://github.com/facebook/rocksdb/issues/6129 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6132 Test Plan: updated c_test, also run under valgrind. Differential Revision: D18864911 Pulled By: pdillinger fbshipit-source-id: 08e81d7b5368b08e501cd402ef5583f2650c19fa 09 December 2019, 20:21:14 UTC
3c34782 sdong 09 December 2019, 20:01:37 UTC Fix thread_local_test failure caused by recent io_uring change (#6136) Summary: thread_local_test now fails because it asserts no thread local instance is created when the test started. However, right now a thread local instance might be created when creating PosixEnv as a static variable. Fix the test by relaxing the assumption of starting from 0. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6136 Test Plan: Find an environment where the test fails, and see it passes with the fix applied. Differential Revision: D18889224 fbshipit-source-id: 7946f3bfea81d236f7bb1554076696705b211b92 09 December 2019, 20:03:30 UTC
7e2f831 Ziyue Yang 09 December 2019, 18:53:07 UTC Fix wrong ExtractUserKey usage in BlockBasedTableBuilder::EnterUnbuff… (#6100) Summary: BlockBasedTableBuilder uses ExtractUserKey in EnterUnbuffered. This would cause index filter building error, since user-provided timestamp is supported by ExtractUserKeyAndStripTimestamp, and it's used in Add. This commit changes ExtractUserKey to ExtractUserKeyAndStripTimestamp. A test case is also added by modifying DBBasicTestWithTimestampWithParam_ PutAndGet test in db_basic_test to cover ExtractUserKeyAndStripTimestamp usage in both kBuffered and kUnbuffered state of BlockBasedTableBuilder. Before the ExtractUserKeyAndStripTimstamp fix: ``` $ ./db_basic_test --gtest_filter="*PutAndGet*" Note: Google Test filter = *PutAndGet* [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam [ RUN ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0 db/db_basic_test.cc:2109: Failure db_->Get(ropts, cfh, "key" + std::to_string(j), &value) NotFound: db/db_basic_test.cc:2109: Failure db_->Get(ropts, cfh, "key" + std::to_string(j), &value) NotFound: db/db_basic_test.cc:2109: Failure db_->Get(ropts, cfh, "key" + std::to_string(j), &value) NotFound: db/db_basic_test.cc:2109: Failure db_->Get(ropts, cfh, "key" + std::to_string(j), &value) NotFound: db/db_basic_test.cc:2109: Failure db_->Get(ropts, cfh, "key" + std::to_string(j), &value) NotFound: [ FAILED ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0, where GetParam() = false (1177 ms) [ RUN ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1 [ OK ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1 (1056 ms) [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam (2233 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (2233 ms total) [ PASSED ] 1 test. [ FAILED ] 1 test, listed below: [ FAILED ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0, where GetParam() = false 1 FAILED TEST ``` After the ExtractUserKeyAndStripTimstamp fix: ``` $ ./db_basic_test --gtest_filter="*PutAndGet*" Note: Google Test filter = *PutAndGet* [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam [ RUN ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0 [ OK ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/0 (1417 ms) [ RUN ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1 [ OK ] Timestamp/DBBasicTestWithTimestampWithParam.PutAndGet/1 (1041 ms) [----------] 2 tests from Timestamp/DBBasicTestWithTimestampWithParam (2458 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (2458 ms total) [ PASSED ] 2 tests. ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6100 Differential Revision: D18769654 Pulled By: riversand963 fbshipit-source-id: 76c2cf2c9a5e0d85db95d98e812e6af0c2a15c6b 09 December 2019, 18:57:02 UTC
d1ae2c3 sdong 09 December 2019, 18:23:28 UTC Fix an asan warning caused by the recent io_uring change (#6135) Summary: ASAN reports: internal_repo_rocksdb/repo:db_test - MultiThreaded/MultiThreadedDBTest.MultiThreaded/43: fatal ==2692739==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6130000500ca at pc 0x0000006be780 bp 0x7efef85ccd20 sp 0x7efef85cc4d0 [CONTEXT] === How to use this, how to get the raw stack trace, and more: fburl.com/ASAN === [CONTEXT] READ of size 331 at 0x6130000500ca thread T195 [CONTEXT] #0 db_test_bin+0x6be77f __interceptor_strlen.part.35 [CONTEXT] https://github.com/facebook/rocksdb/issues/1 internal_repo_rocksdb/repo/include/rocksdb/slice.h:55 rocksdb::Slice::Slice(char const*) [CONTEXT] https://github.com/facebook/rocksdb/issues/2 internal_repo_rocksdb/repo/env/io_posix.cc:522 rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::ReadRequest*, unsigned long) I looked at env/io_posix.cc:522 but don't see a reason why the line needs to be there at all, because it is not used before overwritten. So it must be a line that is put there as a bug. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6135 Test Plan: Rerun the same test which passes after the fix. Run all the tests and make sure they all pass. Differential Revision: D18880251 fbshipit-source-id: 3b84ac6a05b67b529c4202e0ceb4c047460f44f2 09 December 2019, 18:25:09 UTC
3a6d943 Peter Dillinger 09 December 2019, 17:22:07 UTC Use SpecialSkipListFactory in RecalculateScoreAfterPicking (#6125) Summary: Test DBTestUniversalCompaction.RecalculateScoreAfterPicking was flaky on ARM, so it now uses SpecialSkipListFactory (like other tests) for predictable memtable flushes. Fixes https://github.com/facebook/rocksdb/issues/5736 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6125 Test Plan: while ./db_universal_compaction_test; do :; done # for a while on ARM and on Intel (both Linux) Differential Revision: D18864821 Pulled By: pdillinger fbshipit-source-id: 2f3ca0ea66ce420dcd6d41b0ec12377112a5a79f 09 December 2019, 17:23:50 UTC
7d79b32 sdong 09 December 2019, 07:49:32 UTC Break db_stress_tool.cc to a list of source files (#6134) Summary: db_stress_tool.cc now is a giant file. In order to main it easier to improve and maintain, break it down to multiple source files. Most classes are turned into their own files. Separate .h and .cc files are created for gflag definiations. Another .h and .cc files are created for some common functions. Some test execution logic that is only loosely related to class StressTest is moved to db_stress_driver.h and db_stress_driver.cc. All the files are located under db_stress_tool/. The directory name is created as such because if we end it with either stress or test, .gitignore will ignore any file under it and makes it prone to issues in developements. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6134 Test Plan: Build under GCC7 with and without LITE on using GNU Make. Build with GCC 4.8. Build with cmake with -DWITH_TOOL=1 Differential Revision: D18876064 fbshipit-source-id: b25d0a7451840f31ac0f5ebb0068785f783fdf7d 09 December 2019, 07:51:01 UTC
bac38c9 suzanwen 09 December 2019, 05:33:23 UTC Isolate building db_bench from tests with `WITH_BENCHMARK_TOOLS` option. (#6098) Summary: Isolate `db_bench` from building tests, out of respect for the related comments. Let building tests yields to `WITH_TEST=ON` AND `CMAKE_BUILD_TYPE=Debug` both, and building `db_bench` yields to `WITH_BENCHMARK_TOOLS=ON`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6098 Test Plan: cmake -DCMAKE_BUILD_TYPE=Debug/Release -DWITH_TESTS=ON/OFF -DWITH_BENCHMARK_TOOLS=ON/OFF -DWITH_TOOLS=ON/OFF && make Differential Revision: D18856891 Pulled By: riversand963 fbshipit-source-id: addbee8ad6abefb877843a313b4630cfab3ce4f0 09 December 2019, 05:34:28 UTC
e3a82bb sdong 08 December 2019, 04:54:27 UTC PosixRandomAccessFile::MultiRead() to use I/O uring if supported (#5881) Summary: Right now, PosixRandomAccessFile::MultiRead() executes read requests in parallel. In this PR, it leverages I/O Uring library to run it in parallel, even when page cache is enabled. This function will fall back if the kernel version doesn't support it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5881 Test Plan: Run the unit test on a kernel version supporting it and make sure all tests pass, and run a unit test on kernel version supporting it and see it pass. Before merging, will also run stress test and see it passes. Differential Revision: D17742266 fbshipit-source-id: e05699c925ac04fdb42379456a4e23e4ebcb803a 08 December 2019, 04:55:52 UTC
6db57bc Peter Dillinger 06 December 2019, 18:25:40 UTC Disable new Bloom filter assertion (#6128) Summary: A longstanding bug in our C interface can trigger this assertion; see issue https://github.com/facebook/rocksdb/issues/6129. Disabling the assertion for now (for 6.6.0) and will re-enable on fix of that bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6128 Differential Revision: D18854899 Pulled By: pdillinger fbshipit-source-id: 9eb5294b9f11b208dc1a8cc148aaa31e47ff892b 06 December 2019, 18:28:02 UTC
ad528fe Peter Dillinger 05 December 2019, 23:45:44 UTC Disable folly_synchronization_distributed_mutex_test on ARM for now (#6126) Summary: This test is crashing on ARM but is not yet production code. Let's not let it block ARM CI. See PR https://github.com/facebook/rocksdb/issues/5932 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6126 Test Plan: ./folly_synchronization_distributed_mutex_test, on Linux/ARM, on Linux/x86_64, and with LITE=1 on Linux/x86_64 (also disabled) Differential Revision: D18836576 Pulled By: pdillinger fbshipit-source-id: d8a36eea2f048e8330411d994435d1c58a15d978 05 December 2019, 23:48:01 UTC
100b5e6 Adam Simpkins 05 December 2019, 23:33:14 UTC Fix build failure for db_stress tool when building with CMake (#6117) Summary: PR https://github.com/facebook/rocksdb/issues/5937 changed the db_stress tool to also require db_stress_tool.cc, and updated the Makefile but not the CMakeLists.txt file. This updates the CMakeLists.txt file so that the CMake build succeeds again. PR https://github.com/facebook/rocksdb/issues/5950 updated the Makefile build to package db_stress_tool.cc into its own librocksdb_stress.a library. I haven't done that here since there didn't really seem to be much benefit: the Makefile-based build does not install this library. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6117 Test Plan: Confirmed the CMake build succeeds on an Ubuntu 18.04 system. Differential Revision: D18835053 Pulled By: riversand963 fbshipit-source-id: 6e2a66834716e73b1eb736d9b7159870defffec5 05 December 2019, 23:34:54 UTC
cdc431e Jim Meyering 05 December 2019, 19:47:38 UTC build_tools/precommit_checker.py: don't hard-code a platform-afflicted python path (#6124) Summary: Use `#!/usr/bin/env python2.7` instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6124 Test Plan: `J=8 make commit_prereq` Differential Revision: D18834668 Pulled By: ltamasi fbshipit-source-id: cec40266cd5bcae8bf6cbe5a564ae78540deccc4 05 December 2019, 19:49:17 UTC
4edb428 Yanqin Jin 04 December 2019, 00:02:47 UTC Make folly-related targets comply with verbosity (#6120) Summary: Before this fix, `make all` will emit full compilation command when building object files in the third-party/folly directory even if default verbosity is 0 (AM_DEFAULT_VERBOSITY). Test Plan (devserver): ``` $make all | tee build.log $make check ``` Check build.log to verify. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6120 Differential Revision: D18795621 Pulled By: riversand963 fbshipit-source-id: 04641a8359cd4fd55034e6e797ed85de29ee2fe2 04 December 2019, 00:04:44 UTC
f32a311 Connor 03 December 2019, 19:55:18 UTC Fix compliation error on GCC4.8.2 (#6106) Summary: ``` In file included from /usr/include/c++/4.8.2/algorithm:62:0, from ./db/merge_context.h:7, from ./db/dbformat.h:16, from ./tools/block_cache_analyzer/block_cache_trace_analyzer.h:12, from tools/block_cache_analyzer/block_cache_trace_analyzer.cc:8: /usr/include/c++/4.8.2/bits/stl_algo.h: In instantiation of ‘_RandomAccessIterator std::__unguarded_partition(_RandomAccessIterator, _RandomAccessIterator, const _Tp&, _Compare) [with _RandomAccessIterator = __gnu_cxx::__normal_iterator<std::pair<std::basic_string<char>, long unsigned int>*, std::vector<std::pair<std::basic_string<char>, long unsigned int> > >; _Tp = std::pair<std::basic_string<char>, long unsigned int>; _Compare = rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1]’: /usr/include/c++/4.8.2/bits/stl_algo.h:2296:78: required from ‘_RandomAccessIterator std::__unguarded_partition_pivot(_RandomAccessIterator, _RandomAccessIterator, _Compare) [with _RandomAccessIterator = __gnu_cxx::__normal_iterator<std::pair<std::basic_string<char>, long unsigned int>*, std::vector<std::pair<std::basic_string<char>, long unsigned int> > >; _Compare = rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1]’ /usr/include/c++/4.8.2/bits/stl_algo.h:2337:62: required from ‘void std::__introsort_loop(_RandomAccessIterator, _RandomAccessIterator, _Size, _Compare) [with _RandomAccessIterator = __gnu_cxx::__normal_iterator<std::pair<std::basic_string<char>, long unsigned int>*, std::vector<std::pair<std::basic_string<char>, long unsigned int> > >; _Size = long int; _Compare = rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1]’ /usr/include/c++/4.8.2/bits/stl_algo.h:5499:44: required from ‘void std::sort(_RAIter, _RAIter, _Compare) [with _RAIter = __gnu_cxx::__normal_iterator<std::pair<std::basic_string<char>, long unsigned int>*, std::vector<std::pair<std::basic_string<char>, long unsigned int> > >; _Compare = rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1’ tools/block_cache_analyzer/block_cache_trace_analyzer.cc:583:79: required from here /usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: error: no match for call to ‘(rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1) (std::pair<std::basic_string<char>, long unsigned int>&, const std::pair<std::basic_string<char>, long unsigned int>&)’ while (__comp(*__first, __pivot)) ^ tools/block_cache_analyzer/block_cache_trace_analyzer.cc:582:9: note: candidates are: [=](std::pair<std::string, uint64_t>& a, ^ In file included from /usr/include/c++/4.8.2/algorithm:62:0, from ./db/merge_context.h:7, from ./db/dbformat.h:16, from ./tools/block_cache_analyzer/block_cache_trace_analyzer.h:12, from tools/block_cache_analyzer/block_cache_trace_analyzer.cc:8: /usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: note: bool (*)(std::pair<std::basic_string<char>, long unsigned int>&, std::pair<std::basic_string<char>, long unsigned int>&) <conversion> while (__comp(*__first, __pivot)) ^ /usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: note: candidate expects 3 arguments, 3 provided tools/block_cache_analyzer/block_cache_trace_analyzer.cc:583:46: note: rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1 std::pair<std::string, uint64_t>& b) { return b.second < a.second; }); ^ tools/block_cache_analyzer/block_cache_trace_analyzer.cc:583:46: note: no known conversion for argument 2 from ‘const std::pair<std::basic_string<char>, long unsigned int>’ to ‘std::pair<std::basic_string<char>, long unsigned int>&’ In file included from /usr/include/c++/4.8.2/algorithm:62:0, from ./db/merge_context.h:7, from ./db/dbformat.h:16, from ./tools/block_cache_analyzer/block_cache_trace_analyzer.h:12, from tools/block_cache_analyzer/block_cache_trace_analyzer.cc:8: /usr/include/c++/4.8.2/bits/stl_algo.h:2266:34: error: no match for call to ‘(rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1) (const std::pair<std::basic_string<char>, long unsigned int>&, std::pair<std::basic_string<char>, long unsigned int>&)’ while (__comp(__pivot, *__last)) ^ tools/block_cache_analyzer/block_cache_trace_analyzer.cc:582:9: note: candidates are: [=](std::pair<std::string, uint64_t>& a, ^ In file included from /usr/include/c++/4.8.2/algorithm:62:0, from ./db/merge_context.h:7, from ./db/dbformat.h:16, from ./tools/block_cache_analyzer/block_cache_trace_analyzer.h:12, from tools/block_cache_analyzer/block_cache_trace_analyzer.cc:8: /usr/include/c++/4.8.2/bits/stl_algo.h:2266:34: note: bool (*)(std::pair<std::basic_string<char>, long unsigned int>&, std::pair<std::basic_string<char>, long unsigned int>&) <conversion> while (__comp(__pivot, *__last)) ^ /usr/include/c++/4.8.2/bits/stl_algo.h:2266:34: note: candidate expects 3 arguments, 3 provided tools/block_cache_analyzer/block_cache_trace_analyzer.cc:583:46: note: rocksdb::BlockCacheTraceAnalyzer::WriteSkewness(const string&, const std::vector<long unsigned int>&, rocksdb::TraceType) const::__lambda1 std::pair<std::string, uint64_t>& b) { return b.second < a.second; }); ^ tools/block_cache_analyzer/block_cache_trace_analyzer.cc:583:46: note: no known conversion for argument 1 from ‘const std::pair<std::basic_string<char>, long unsigned int>’ to ‘std::pair<std::basic_string<char>, long unsigned int>&’ ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6106 Differential Revision: D18783943 Pulled By: riversand963 fbshipit-source-id: cc7fc10565f0210b9eebf46b95cb4950ec0b15fa 03 December 2019, 19:59:21 UTC
fe1147d Yanqin Jin 03 December 2019, 01:43:37 UTC Let DBSecondary close files after catch up (#6114) Summary: After secondary instance replays the logs from primary, certain files become obsolete. The secondary should find these files, evict their table readers from table cache and close them. If this is not done, the secondary will hold on to these files and prevent their space from being freed. Test plan (devserver): ``` $./db_secondary_test --gtest_filter=DBSecondaryTest.SecondaryCloseFiles $make check $./db_stress -ops_per_thread=100000 -enable_secondary=true -threads=32 -secondary_catch_up_one_in=10000 -clear_column_family_one_in=1000 -reopen=100 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6114 Differential Revision: D18769998 Pulled By: riversand963 fbshipit-source-id: 5d1f151567247196164e1b79d8402fa2045b9120 03 December 2019, 01:45:03 UTC
16fa6fd anand76 02 December 2019, 22:58:22 UTC Remove key length assertion LRUHandle::CalcTotalCharge (#6115) Summary: Inserting an entry in the block cache with 0 length key is a valid use case. Remove the assertion in ```LRUHandle::CalcTotalCharge```. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6115 Differential Revision: D18769693 Pulled By: anand1976 fbshipit-source-id: 34cc159650300dda6d7273480640478f28392cda 02 December 2019, 23:00:07 UTC
048472f David Palm 02 December 2019, 18:58:14 UTC Add missing DataBlock-releated functions to the C-API (#6101) Summary: Adds two missing functions to the C-API: - `rocksdb_block_based_options_set_data_block_index_type` - `rocksdb_block_based_options_set_data_block_hash_ratio` This enables users in other languages to enjoy the new(-ish) feature. The changes here are partially overlapping with [another PR](https://github.com/facebook/rocksdb/pull/5630) but are more focused on the DataBlock indexing options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6101 Differential Revision: D18765639 fbshipit-source-id: 4a8947e71b179f26fa1eb83c267dd47ee64ac3b3 02 December 2019, 19:00:09 UTC
e8f997c Peter Dillinger 27 November 2019, 23:05:32 UTC Update comment on max_valid_backups_to_open (#6105) Summary: To reflect changes in PR https://github.com/facebook/rocksdb/issues/6072 This comment also implies that a seemingly valid use-case for max_valid_backups_to_open is flawed: even if you only want to add a new backup without trying to delete, you might need to clean up after a backup creation that never finished. To clean up properly requires opening all backups to get proper ref counts on shared files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6105 Test Plan: code comment only Differential Revision: D18736716 Pulled By: pdillinger fbshipit-source-id: 2447c0000eefe3a4ca606926bfe922a8456b0cb7 27 November 2019, 23:06:58 UTC
09fcf4f Yanqin Jin 27 November 2019, 22:46:38 UTC Fix a potential bug scheduling unnecessary threads (#6104) Summary: RocksDB should decrement the counter `unscheduled_flushes_` as soon as the bg thread is scheduled. Before this fix, the counter is decremented only when the bg thread starts and picks an element from the flush queue. This may cause more than necessary bg threads to be scheduled. Not a correctness issue, but may affect flush thread count. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6104 Test Plan: ``` make check ``` Differential Revision: D18735584 Pulled By: riversand963 fbshipit-source-id: d36272d4a08a494aeeab6200a3cff7a3d1a2dc10 27 November 2019, 22:48:49 UTC
f19faf7 Peter Dillinger 27 November 2019, 21:17:28 UTC Add format_version=5 to db_crashtest (#6102) Summary: format_version=5 enables new Bloom filter. Using 2/5 probability for "latest and greatest" rather than naive 1/4. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6102 Test Plan: start 'make blackbox_crash_test' Differential Revision: D18735685 Pulled By: pdillinger fbshipit-source-id: e81529c8a3f53560d246086ee5f92ee7d79a2eab 27 November 2019, 21:19:11 UTC
a61ec9a Adam Retter 27 November 2019, 21:07:28 UTC Fix BlobDB compilation on older GCC versions Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6094 Differential Revision: D18731951 Pulled By: ltamasi fbshipit-source-id: 5b73c6009c748f6a2a48d4d880b1259980d801d4 27 November 2019, 21:09:09 UTC
9befbe9 Yingchun Lai 27 November 2019, 18:24:12 UTC fix typo (#6099) Summary: fix a typo in struct ReadOptions Pull Request resolved: https://github.com/facebook/rocksdb/pull/6099 Differential Revision: D18729618 fbshipit-source-id: 850a9df71f7c0abebea17feab77b8d5874e8ba0a 27 November 2019, 18:26:42 UTC
0695793 Peter Dillinger 27 November 2019, 18:22:45 UTC Update format_version comment for 6.6.0 Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6097 Differential Revision: D18729661 Pulled By: pdillinger fbshipit-source-id: d2e4a9d6803aad8dd61ececd5c2b861e6f2da73b 27 November 2019, 18:24:16 UTC
c16b087 John Ericson 27 November 2019, 05:40:16 UTC Work around weird unused errors with Mingw (#6075) Summary: From the reset of the code, it looks this this maybe can be unconditionally given the attribute? But I couldn't test with MSVC so I defensively put under CPP. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6075 Differential Revision: D18723749 fbshipit-source-id: 45fc8732c28dd29aab1644225d68f3c6f39bd69b 27 November 2019, 05:42:29 UTC
aa1857e sdong 27 November 2019, 05:38:38 UTC Support options.max_open_files = -1 with periodic_compaction_seconds (#6090) Summary: options.periodic_compaction_seconds isn't supported when options.max_open_files != -1. It's because that the information of file creation time is stored in table properties and are not guaranteed to be loaded unless options.max_open_files = -1. Relax this constraint by storing the information in manifest. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6090 Test Plan: Pass all existing tests; Modify an existing test to force the manifest value to take 0 to simulate backward compatibility case; manually open the DB generated with the change by release 4.2. Differential Revision: D18702268 fbshipit-source-id: 13e0bd94f546498a04f3dc5fc0d9dff5125ec9eb 27 November 2019, 05:39:56 UTC
496a6ae anand76 27 November 2019, 02:59:24 UTC Fix HISTORY.md for 6.6.0 (#6096) Summary: Some of the entries were incorrectly listed under 6.5.0. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6096 Differential Revision: D18722801 Pulled By: gfosco fbshipit-source-id: 18d1187deb6a9d69a8feb68b727d2f720a65f2bc 27 November 2019, 03:04:49 UTC
ca3b6c2 Peter Dillinger 27 November 2019, 02:18:29 UTC Expose and elaborate FilterBuildingContext (#6088) Summary: This change enables custom implementations of FilterPolicy to wrap a variety of NewBloomFilterPolicy and select among them based on contextual information such as table level and compaction style. * Moves FilterBuildingContext to public API and elaborates it with more useful data. (It would be nice to put more general options-like data, but at the time this object is constructed, we are using internal APIs ImmutableCFOptions and MutableCFOptions and don't have easy access to ColumnFamilyOptions that I can tell.) * Renames BloomFilterPolicy::GetFilterBitsBuilderInternal to GetBuilderWithContext, because it's now public. * Plumbs through the table's "level_at_creation" for filter building context. * Simplified some tests by adding GetBuilder() to MockBlockBasedTableTester. * Adds test as DBBloomFilterTest.ContextCustomFilterPolicy, including sample wrapper class LevelAndStyleCustomFilterPolicy. * Fixes a cross-test bug in DBBloomFilterTest.OptimizeFiltersForHits where it does not reset perf context. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6088 Test Plan: make check, valgrind on db_bloom_filter_test Differential Revision: D18697817 Pulled By: pdillinger fbshipit-source-id: 5f987a2d7b07cc7a33670bc08ca6b4ca698c1cf4 27 November 2019, 02:24:10 UTC
6d58ea9 Adam Retter 27 November 2019, 00:55:46 UTC Fix compilation under MSVC VS2015 (#6081) Summary: **NOTE**: this also needs to be back-ported to 6.4.6 and possibly older branches if further releases from them is envisaged. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6081 Differential Revision: D18710107 Pulled By: zhichao-cao fbshipit-source-id: 03260f9316566e2bfc12c7d702d6338bb7941e01 27 November 2019, 02:24:09 UTC
8ae149e Patrick Double 27 November 2019, 00:51:26 UTC Add shared library for musl-libc (#3143) Summary: Add the jni library for musl-libc, specifically for incorporating into Alpine based docker images. The classifier is `musl64`. I have signed the CLA electronically. Pull Request resolved: https://github.com/facebook/rocksdb/pull/3143 Differential Revision: D18719372 fbshipit-source-id: 6189d149310b6436d6def7d808566b0234b23313 27 November 2019, 02:24:09 UTC
d9314a9 Levi Tamasi 27 November 2019, 00:42:44 UTC Refactor and clean up the code that reads a blob from a file (#6093) Summary: This patch factors out the logic that reads a (potentially compressed) blob from a file into a separate helper method `GetRawBlobFromFile`, and cleans up the code a bit. Also, errors during decompression are now logged/propagated to the user by returning a `Status` code of `Corruption`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6093 Test Plan: `make check` Differential Revision: D18716673 Pulled By: ltamasi fbshipit-source-id: 44144bc064cab616862d5643f34384f2bae6eb78 27 November 2019, 00:49:39 UTC
57f3032 Peter Dillinger 26 November 2019, 23:49:16 UTC Allow fractional bits/key in BloomFilterPolicy (#6092) Summary: There's no technological impediment to allowing the Bloom filter bits/key to be non-integer (fractional/decimal) values, and it provides finer control over the memory vs. accuracy trade-off. This is especially handy in using the format_version=5 Bloom filter in place of the old one, because bits_per_key=9.55 provides the same accuracy as the old bits_per_key=10. This change not only requires refining the logic for choosing the best num_probes for a given bits/key setting, it revealed a flaw in that logic. As bits/key gets higher, the best num_probes for a cache-local Bloom filter is closer to bpk / 2 than to bpk * 0.69, the best choice for a standard Bloom filter. For example, at 16 bits per key, the best num_probes is 9 (FP rate = 0.0843%) not 11 (FP rate = 0.0884%). This change fixes and refines that logic (for the format_version=5 Bloom filter only, just in case) based on empirical tests to find accuracy inflection points between each num_probes. Although bits_per_key is now specified as a double, the new Bloom filter converts/rounds this to "millibits / key" for predictable/precise internal computations. Just in case of unforeseen compatibility issues, we round to the nearest whole number bits / key for the legacy Bloom filter, so as not to unlock new behaviors for it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6092 Test Plan: unit tests included Differential Revision: D18711313 Pulled By: pdillinger fbshipit-source-id: 1aa73295f152a995328cb846ef9157ae8a05522a 26 November 2019, 23:59:34 UTC
72daa92 Levi Tamasi 26 November 2019, 21:16:39 UTC Refactor blob file creation logic (#6066) Summary: The patch refactors and cleans up the logic around creating new blob files by moving the common code of `SelectBlobFile` and `SelectBlobFileTTL` to a new helper method `CreateBlobFileAndWriter`, bringing the implementation of `SelectBlobFile` and `SelectBlobFileTTL` into sync, and increasing encapsulation by adding new constructors for `BlobFile` and `BlobLogHeader`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6066 Test Plan: Ran `make check` and used the BlobDB mode of `db_bench` to sanity test both the TTL and the non-TTL code paths. Differential Revision: D18646921 Pulled By: ltamasi fbshipit-source-id: e5705a84807932e31dccab4f49b3e64369cea26d 26 November 2019, 21:28:32 UTC
771e172 John Ericson 26 November 2019, 18:57:29 UTC Use lowercase for shlwapi.lib rpcrt4.lib (#6076) Summary: This fixes MinGW cross compilation from case-sensative file systems, at no harm to MinGW builds on Windows. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6076 Differential Revision: D18710554 fbshipit-source-id: a9f299ac3aa019f7dbc07ed0c4a79e19cf99b488 26 November 2019, 21:28:32 UTC
1bf316e Adam Retter 26 November 2019, 18:53:23 UTC Fix naming of library on PPC64LE (#6080) Summary: **NOTE**: This also needs to be back-ported to be 6.4.6 Fix a regression introduced in f2bf0b2 by https://github.com/facebook/rocksdb/pull/5674 whereby the compiled library would get the wrong name on PPC64LE platforms. On PPC64LE, the regression caused the library to be named `librocksdbjni-linux64.so` instead of `librocksdbjni-linux-ppc64le.so`. This PR corrects the name back to `librocksdbjni-linux-ppc64le.so` and also corrects the ordering of conditional arguments in the Makefile to match the expected order as defined in the documentation for Make. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6080 Differential Revision: D18710351 fbshipit-source-id: d4db87ef378263b57de7f9edce1b7d15644cf9de 26 November 2019, 21:28:32 UTC
7f14519 Adam Retter 26 November 2019, 18:52:04 UTC Small improvements to Docker build for RocksJava (#6079) Summary: * We can reuse downloaded 3rd-party libraries * We can isolate the build to a Docker volume. This is useful for investigating failed builds, as we can examine the volume by assigning it a name during the build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6079 Differential Revision: D18710263 fbshipit-source-id: 93f456ba44b49e48941c43b0c4d53995ecc1f404 26 November 2019, 21:28:31 UTC
4f17d33 Peter Dillinger 26 November 2019, 18:47:25 UTC Remove unused/undefined ImmutableCFOptions() (#6086) Summary: default constructor not used or even defined Pull Request resolved: https://github.com/facebook/rocksdb/pull/6086 Differential Revision: D18695669 Pulled By: pdillinger fbshipit-source-id: 6b6ac46029f4fb6edf1c11ee6ce1d9f172b2eaf2 26 November 2019, 21:28:31 UTC
382b154 Adam Retter 26 November 2019, 18:45:36 UTC Update 3rd-party libraries used by RocksJava (#6084) Summary: * LZ4 1.8.3 -> 1.9.2 * ZSTD 1.4.0 -> 1.4.4 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6084 Differential Revision: D18710224 fbshipit-source-id: a461ef19a473d3480acdc027f627ec3048730692 26 November 2019, 21:28:31 UTC
77eab5c sdong 26 November 2019, 01:11:26 UTC Make default value of options.ttl to be 30 days when it is supported. (#6073) Summary: By default options.ttl is disabled. We believe a better default will be 30 days, which means deleted data the database will be removed from SST files slightly after 30 days, for most of the cases. Make the default UINT64_MAX - 1 to indicate that it is not overridden by users. Change periodic_compaction_seconds to be UINT64_MAX - 1 to UINT64_MAX too to be consistent. Also fix a small bug in the previous periodic_compaction_seconds default code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6073 Test Plan: Add unit tests for it. Differential Revision: D18669626 fbshipit-source-id: 957cd4374cafc1557d45a0ba002010552a378cc8 26 November 2019, 18:00:32 UTC
fcd7e03 Sebastiano Peluso 25 November 2019, 22:18:10 UTC Ignore value of BackupableDBOptions::max_valid_backups_to_open when B… (#6072) Summary: This change ignores the value of BackupableDBOptions::max_valid_backups_to_open when a BackupEngine is not read-only. Issue: https://github.com/facebook/rocksdb/issues/4997 Note on tests: I had to remove test case WriteOnlyEngine of BackupableDBTest because it was not consistent with the new semantic of BackupableDBOptions::max_valid_backups_to_open. Maybe, we should think about adding a new interface for append-only BackupEngines. On the other hand, I changed LimitBackupsOpened test case to use a read-only BackupEngine, and I added a new specific test case for the change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6072 Reviewed By: pdillinger Differential Revision: D18687364 Pulled By: sebastianopeluso fbshipit-source-id: 77bc1f927d623964d59137a93de123bbd719da4e 26 November 2019, 18:00:31 UTC
0bc8744 sdong 25 November 2019, 20:03:06 UTC Update HISTORY.md for forward compatibility (#6085) Summary: https://github.com/facebook/rocksdb/pull/6060 broke forward compatiblity for releases from 3.10 to 4.2. Update HISTORY.md to mention it. Also remove it from the compatibility tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6085 Differential Revision: D18691694 fbshipit-source-id: 4ef903783dc722b8a4d3e8229abbf0f021a114c9 26 November 2019, 18:00:31 UTC
back to top