Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
16 July 2024, 15:56:10 UTC
sort by:
Revision Author Date Message Commit Date
913c664 sdong 16 December 2019, 22:54:02 UTC tt<Replace this line with a title. Use 1 line only, 67 chars or less> Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: 16 December 2019, 22:54:02 UTC
772c07e sdong 18 October 2019, 23:19:19 UTC Fix VerifyChecksum readahead with mmap mode Summary: A recent change introduced readahead inside VerifyChecksum(). However it is not compatible with mmap mode and generated wrong checksum verification failure. Fix it by not enabling readahead in mmap mode. Test Plan: Add a new unit test which failed without the failure. 18 October 2019, 23:19:19 UTC
1f9d7c0 Yi Wu 16 October 2019, 17:39:00 UTC Fix OnFlushCompleted fired before flush result write to MANIFEST (#5908) Summary: When there are concurrent flush job on the same CF, `OnFlushCompleted` can be called before the flush result being install to LSM. Fixing the issue by passing `FlushJobInfo` through `MemTable`, and the thread who commit the flush result can fetch the `FlushJobInfo` and fire `OnFlushCompleted` on behave of the thread actually writing the SST. Fix https://github.com/facebook/rocksdb/issues/5892 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5908 Test Plan: Add new test. The test will fail without the fix. Differential Revision: D17916144 Pulled By: riversand963 fbshipit-source-id: e18df67d9533b5baee52ae3605026cdeb05cbe10 16 October 2019, 17:40:23 UTC
2c9e9f2 Maysam Yabandeh 16 October 2019, 14:56:51 UTC Update HISTORY for SeekForPrev bug fix (#5925) Summary: Update history for the bug fix in https://github.com/facebook/rocksdb/pull/5907 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5925 Differential Revision: D17952605 Pulled By: maysamyabandeh fbshipit-source-id: 609afcbb2e4087f9153822c4d11193a75a7b0e7a 16 October 2019, 14:59:26 UTC
5ef27de Yanqin Jin 15 October 2019, 05:12:31 UTC Fix clang analyzer error (#5924) Summary: Without this PR, clang analyzer complains. ``` $USE_CLANG=1 make analyze db/compaction/compaction_job_test.cc:161:20: warning: The left operand of '==' is a garbage value if (key.type == kTypeBlobIndex) { ~~~~~~~~ ^ 1 warning generated. ``` Test Plan (on devserver) ``` $USE_CLANG=1 make analyze ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5924 Differential Revision: D17923226 Pulled By: riversand963 fbshipit-source-id: 9d1eb769b5e0de7cb3d89dc90d1cfa895db7fdc8 15 October 2019, 05:14:24 UTC
78b28d8 Levi Tamasi 15 October 2019, 00:47:56 UTC Support non-TTL Puts for BlobDB in db_bench (#5921) Summary: Currently, db_bench only supports PutWithTTL operations for BlobDB but not regular Puts. The patch adds support for regular (non-TTL) Puts and also changes the default for blob_db_max_ttl_range to zero, which corresponds to no TTL. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5921 Test Plan: make check ./db_bench -benchmarks=fillrandom -statistics -stats_interval_seconds=1 -duration=90 -num=500000 -use_blob_db=1 -blob_db_file_size=1000000 -target_file_size_base=1000000 (issues Put operations with no TTL) ./db_bench -benchmarks=fillrandom -statistics -stats_interval_seconds=1 -duration=90 -num=500000 -use_blob_db=1 -blob_db_file_size=1000000 -target_file_size_base=1000000 -blob_db_max_ttl_range=86400 (issues PutWithTTL operations with random TTLs in the [0, blob_db_max_ttl_range) interval, as before) Differential Revision: D17919798 Pulled By: ltamasi fbshipit-source-id: b946c3522b836b92b4c157ffbad24f92ba2b0a16 15 October 2019, 00:49:20 UTC
93edd51 Peter Dillinger 14 October 2019, 22:37:12 UTC bloom_test.cc: include <array> (#5920) Summary: Fix build failure on some platforms, reported in issue https://github.com/facebook/rocksdb/issues/5914 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5920 Test Plan: make bloom_test && ./bloom_test Differential Revision: D17918328 Pulled By: pdillinger fbshipit-source-id: b822004d4442de0171db2aeff433677783f7b94e 14 October 2019, 22:38:31 UTC
5f025ea Levi Tamasi 14 October 2019, 22:19:31 UTC BlobDB GC: add SST <-> oldest blob file referenced mapping (#5903) Summary: This is groundwork for adding garbage collection support to BlobDB. The patch adds logic that keeps track of the oldest blob file referred to by each SST file. The oldest blob file is identified during flush/ compaction (similarly to how the range of keys covered by the SST is identified), and persisted in the manifest as a custom field of the new file edit record. Blob indexes with TTL are ignored for the purposes of identifying the oldest blob file (since such blob files are cleaned up by the TTL logic in BlobDB). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5903 Test Plan: Added new unit tests; also ran db_bench in BlobDB mode, inspected the manifest using ldb, and confirmed (by scanning the SST files using sst_dump) that the value of the oldest blob file number field matches the contents of the file for each SST. Differential Revision: D17859997 Pulled By: ltamasi fbshipit-source-id: 21662c137c6259a6af70446faaf3a9912c550e90 14 October 2019, 22:21:01 UTC
a59dc84 Levi Tamasi 14 October 2019, 19:52:15 UTC Move blob_index.h to db/ (#5919) Summary: Extracted from PR https://github.com/facebook/rocksdb/issues/5903 for technical reasons. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5919 Test Plan: make check Differential Revision: D17910132 Pulled By: ltamasi fbshipit-source-id: 6ecbb8d6e84b2a1d1f28575ad48ac3cc65833eb5 14 October 2019, 19:54:05 UTC
231fffd Yanqin Jin 14 October 2019, 19:23:39 UTC Add Env::SanitizeEnvOptions (#5885) Summary: Add Env::SanitizeEnvOptions to allow underlying environments properly configure env options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5885 Test Plan: ``` make check ``` Differential Revision: D17910327 Pulled By: riversand963 fbshipit-source-id: 86a1ac616e485742c35c4a9cc9f1227c529fc00f 14 October 2019, 19:25:00 UTC
a6e615a Maysam Yabandeh 14 October 2019, 17:33:31 UTC Enable partitioned index/filter in stress tests (#5918) Summary: This is the 3rd attempt after the revert of https://github.com/facebook/rocksdb/issues/4020 and https://github.com/facebook/rocksdb/issues/5895 The last bug is fixed https://github.com/facebook/rocksdb/pull/5907 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5918 Test Plan: ``` make -j32 crash_test ``` Differential Revision: D17909489 Pulled By: maysamyabandeh fbshipit-source-id: 7dfb8cf998c2d295c86465dd21734593d277887e 14 October 2019, 17:35:18 UTC
6febfd8 Yanqin Jin 14 October 2019, 16:52:29 UTC OnTableFileCreationCompleted use "(nil)" for empty file during flush (#5905) Summary: Compaction can call OnTableFileCreationCompleted(). If file is empty, "(nil)" is used as the file name. Do the same for flush. Test plan (dev server): ``` make all make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5905 Differential Revision: D17883285 Pulled By: riversand963 fbshipit-source-id: 6565884adbb00e8023d88b17dfb3b6eb92220b59 14 October 2019, 16:54:10 UTC
4e729f9 Maysam Yabandeh 12 October 2019, 03:28:36 UTC Fix SeekForPrev bug with Partitioned Filters and Prefix (#5907) Summary: Partition Filters make use of a top-level index to find the partition that might have the bloom hash of the key. The index is with internal key format (before format version 3). Each partition contains the i) blooms of the keys in that range ii) bloom of prefixes of keys in that range, iii) the bloom of the prefix of the last key in the previous partition. When ::SeekForPrev(key), we first perform a prefix bloom test on the SST file. The partition however is identified using the full internal key, rather than the prefix key. The reason is to be compatible with the internal key format of the top-level index. This creates a corner case. Example: - SST k, Partition N: P1K1, P1K2 - SST k, top-level index: P1K2 - SST k+1, Partition 1: P2K1, P3K1 - SST k+1 top-level index: P3K1 When SeekForPrev(P1K3), it should point us to P1K2. However SST k top-level index would reject P1K3 since it is out of range. One possible fix would be to search with the prefix P1 (instead of full internal key P1K3) however the details of properly comparing prefix with full internal key might get complicated. The fix we apply in this PR is to look into the last partition anyway even if the key is out of range. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5907 Differential Revision: D17889918 Pulled By: maysamyabandeh fbshipit-source-id: 169fd7b3c71dbc08808eae5a8340611ebe5bdc1e 12 October 2019, 03:30:00 UTC
b00761e Andrew Kryczka 12 October 2019, 01:18:14 UTC Fix block cache ID uniqueness for Windows builds (#5844) Summary: Since we do not evict a file's blocks from block cache before that file is deleted, we require a file's cache ID prefix is both unique and non-reusable. However, the Windows functionality we were relying on only guaranteed uniqueness. That meant a newly created file could be assigned the same cache ID prefix as a deleted file. If the newly created file had block offsets matching the deleted file, full cache keys could be exactly the same, resulting in obsolete data blocks returned from cache when trying to read from the new file. We noticed this when running on FAT32 where compaction was writing out of order keys due to reading obsolete blocks from its input files. The functionality is documented as behaving the same on NTFS, although I wasn't able to repro it there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5844 Test Plan: we had a reliable repro of out-of-order keys on FAT32 that was fixed by this change Differential Revision: D17752442 fbshipit-source-id: 95d983f9196cf415f269e19293b97341edbf7e00 12 October 2019, 01:19:31 UTC
bc8b05c Yanqin Jin 11 October 2019, 02:18:19 UTC Revert "Enable partitioned index/filter in stress tests (#5895)" (#5904) Summary: This reverts commit 2f4e288143f1c0c9f5a9e829d1b20ce62aca4224. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5904 Differential Revision: D17871282 Pulled By: riversand963 fbshipit-source-id: d210725f8f3b26d8eac25892094da09d9694337e 11 October 2019, 02:19:39 UTC
ddb62d1 Yanqin Jin 11 October 2019, 01:03:55 UTC Remove a webhook due to potential security concern (#5902) Summary: As title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5902 Differential Revision: D17858150 Pulled By: riversand963 fbshipit-source-id: db2cd8a756faf7b9751b2651a22e1b29ca9fecec 11 October 2019, 01:05:16 UTC
1e9c8d4 Adam Retter 11 October 2019, 00:20:01 UTC Fix the rocksjava release Vagrant build on CentOS (#5901) Summary: Closes https://github.com/facebook/rocksdb/issues/5873 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5901 Differential Revision: D17869585 fbshipit-source-id: 559472486f1d3ac80c0c7df6c421c4b612b9b7f9 11 October 2019, 00:21:18 UTC
4c49e38 Vijay Nadimpalli 10 October 2019, 16:37:38 UTC MultiGet batching in memtable (#5818) Summary: RocksDB has a MultiGet() API that implements batched key lookup for higher performance (https://github.com/facebook/rocksdb/blob/master/include/rocksdb/db.h#L468). Currently, batching is implemented in BlockBasedTableReader::MultiGet() for SST file lookups. One of the ways it improves performance is by pipelining bloom filter lookups (by prefetching required cachelines for all the keys in the batch, and then doing the probe) and thus hiding the cache miss latency. The same concept can be extended to the memtable as well. This PR involves implementing a pipelined bloom filter lookup in DynamicBloom, and implementing MemTable::MultiGet() that can leverage it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5818 Test Plan: Existing tests Performance Test: Ran the below command which fills up the memtable and makes sure there are no flushes and then call multiget. Ran it on master and on the new change and see atleast 1% performance improvement across all the test runs I did. Sometimes the improvement was upto 5%. TEST_TMPDIR=/data/users/$USER/benchmarks/feature/ numactl -C 10 ./db_bench -benchmarks="fillseq,multireadrandom" -num=600000 -compression_type="none" -level_compaction_dynamic_level_bytes -write_buffer_size=200000000 -target_file_size_base=200000000 -max_bytes_for_level_base=16777216 -reads=90000 -threads=1 -compression_type=none -cache_size=4194304000 -batch_size=32 -disable_auto_compactions=true -bloom_bits=10 -cache_index_and_filter_blocks=true -pin_l0_filter_and_index_blocks_in_cache=true -multiread_batched=true -multiread_stride=4 -statistics -memtable_whole_key_filtering=true -memtable_bloom_size_ratio=10 Differential Revision: D17578869 Pulled By: vjnadimpalli fbshipit-source-id: 23dc651d9bf49db11d22375bf435708875a1f192 10 October 2019, 16:39:39 UTC
80ad996 anand76 09 October 2019, 16:25:56 UTC Make the db_stress reopen loop in OperateDb() more robust (#5893) Summary: The loop in OperateDb() is getting quite complicated with the introduction of multiple key operations such as MultiGet and Reseeks. This is resulting in a number of corner cases that hangs db_stress due to synchronization problems during reopen (i.e when -reopen=<> option is specified). This PR makes it more robust by ensuring all db_stress threads vote to reopen the DB the exact same number of times. Most of the changes in this diff are due to indentation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5893 Test Plan: Run crash test Differential Revision: D17823827 Pulled By: anand1976 fbshipit-source-id: ec893829f611ac7cac4057c0d3d99f9ffb6a6dd9 09 October 2019, 16:27:10 UTC
5b12381 katherine 09 October 2019, 14:39:19 UTC Remove deprecated RocksDBCommonHelper and cont_integration.sh (#5889) Summary: As titled. RocksDBCommonHelper contains references to legacy APIs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5889 Differential Revision: D17783179 fbshipit-source-id: dcde82a73a311bfa3300ad69189b3a32727134d1 09 October 2019, 14:40:35 UTC
90e285e Peter Dillinger 09 October 2019, 02:19:43 UTC Fix some implicit conversions in filter_bench (#5894) Summary: Fixed some spots where converting size_t or uint_fast32_t to uint32_t. Wrapped mt19937 in a new Random32 class to avoid future such traps. NB: I tried using Random32::Uniform (std::uniform_int_distribution) in filter_bench instead of fastrange, but that more than doubled the dry run time! So I added fastrange as Random32::Uniformish. ;) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5894 Test Plan: USE_CLANG=1 build, and manual re-run filter_bench Differential Revision: D17825131 Pulled By: pdillinger fbshipit-source-id: 68feee333b5f8193c084ded760e3d6679b405ecd 09 October 2019, 02:22:07 UTC
167cdc9 Yanqin Jin 09 October 2019, 02:17:39 UTC Support custom env in sst_dump (#5845) Summary: This PR allows for the creation of custom env when using sst_dump. If the user does not set options.env or set options.env to nullptr, then sst_dump will automatically try to create a custom env depending on the path to the sst file or db directory. In order to use this feature, the user must call ObjectRegistry::Register() beforehand. Test Plan (on devserver): ``` $make all && make check ``` All tests must pass to ensure this change does not break anything. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5845 Differential Revision: D17678038 Pulled By: riversand963 fbshipit-source-id: 58ecb4b3f75246d52b07c4c924a63ee61c1ee626 09 October 2019, 02:19:12 UTC
2f4e288 Maysam Yabandeh 08 October 2019, 23:48:57 UTC Enable partitioned index/filter in stress tests (#5895) Summary: This is the 2nd attempt after the revert of https://github.com/facebook/rocksdb/pull/4020 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5895 Test Plan: ``` ./tools/db_crashtest.py blackbox --simple --interval=10 --max_key=10000000 ``` Differential Revision: D17822137 Pulled By: maysamyabandeh fbshipit-source-id: 3d148c0d8cc129080410ff859c04b544223c8ea3 08 October 2019, 23:50:21 UTC
e3a93c9 Tomas Kolda 08 October 2019, 21:18:48 UTC Fix crash when background task fails (#5879) Summary: Fixing crash. Full story in issue: https://github.com/facebook/rocksdb/issues/5878 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5879 Differential Revision: D17812299 Pulled By: anand1976 fbshipit-source-id: 14e5a4fc502ade974583da9692d0ed6e5014613a 08 October 2019, 21:20:01 UTC
46ca51d Peter Dillinger 08 October 2019, 03:09:27 UTC filter_bench - a prelim tool for SST filter benchmarking (#5825) Summary: Example: using the tool before and after PR https://github.com/facebook/rocksdb/issues/5784 shows that the refactoring, presumed performance-neutral, actually sped up SST filters by about 3% to 8% (repeatable result): Before: - Dry run ns/op: 22.4725 - Single filter ns/op: 51.1078 - Random filter ns/op: 120.133 After: + Dry run ns/op: 22.2301 + Single filter run ns/op: 47.4313 + Random filter ns/op: 115.9 Only tests filters for the block-based table (full filters and partitioned filters - same implementation; not block-based filters), which seems to be the recommended format/implementation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5825 Differential Revision: D17804987 Pulled By: pdillinger fbshipit-source-id: 0f18a9c254c57f7866030d03e7fa4ba503bac3c5 08 October 2019, 03:10:53 UTC
457bcfd Yanqin Jin 08 October 2019, 00:47:19 UTC Let TestEnv and FaultInjectEnv use Env of choice (#5886) Summary: Instead of hard coding Env::Default in TestEnv and a few other places, use the DBTestBase::env_ that has been deduced from the constructor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5886 Test Plan: ``` make check ``` Differential Revision: D17773029 Pulled By: riversand963 fbshipit-source-id: 7ce4e5175a487e9d281ea2c3aae3c41bffd44629 08 October 2019, 00:48:50 UTC
9905101 lokeshgupta0912 07 October 2019, 19:23:27 UTC Replaced some words (#5877) Summary: improved Vocabulary Pull Request resolved: https://github.com/facebook/rocksdb/pull/5877 Differential Revision: D17753217 Pulled By: anand1976 fbshipit-source-id: f255418534297e537a2735f0a0546c724b8f7c70 07 October 2019, 19:28:09 UTC
da3b284 jsteemann 07 October 2019, 19:21:39 UTC save a few redundant container lookups (#5875) Summary: This PR eliminates repeated lookups in associative or ordered containers when a single lookup suffices. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5875 Differential Revision: D17753172 Pulled By: anand1976 fbshipit-source-id: 796b02b760082521d8c42a1cb65a76bf0e6c1b8e 07 October 2019, 19:28:09 UTC
19a97dd anand76 04 October 2019, 03:52:17 UTC Fix data block upper bound checking for iterator reseek case (#5883) Summary: When an iterator reseek happens with the user specifying a new iterate_upper_bound in ReadOptions, and the new seek position is at the end of the same data block, the Seek() ends up using a stale value of data_block_within_upper_bound_ and may return incorrect results. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5883 Test Plan: Added a new test case DBIteratorTest.IterReseekNewUpperBound. Verified that it failed due to the assertion failure without the fix, and passes with the fix. Differential Revision: D17752740 Pulled By: anand1976 fbshipit-source-id: f9b635ff5d6aeb0e1bef102cf8b2f900efd378e3 04 October 2019, 03:53:29 UTC
9f54446 Peter Dillinger 03 October 2019, 20:17:44 UTC Fix type in shift operation in bloom_test (#5882) Summary: Broken type for shift in PR#5834. Fixing code means fixing expected values in test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5882 Test Plan: thisisthetest Differential Revision: D17746136 Pulled By: pdillinger fbshipit-source-id: d3c456ed30b433d55fcab6fc7d836940fe3b46b8 03 October 2019, 20:19:20 UTC
cca87d7 anand76 03 October 2019, 17:20:17 UTC Fix reopen voting logic in db_stress to prevent hangs (#5876) Summary: When multiple operations are performed in a db_stress thread in one loop iteration, the reopen voting logic needs to take that into account. It was doing that for MultiGet, but a new option was introduced recently to do multiple iterator seeks per iteration, which broke it again. Fix the logic to be more robust and agnostic of the type of operation performed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5876 Test Plan: Run db_stress Differential Revision: D17733590 Pulled By: anand1976 fbshipit-source-id: 787f01abefa1e83bba43e0b4f4abb26699b2089e 03 October 2019, 17:22:26 UTC
9e4913c Peter Dillinger 02 October 2019, 22:31:54 UTC Add FullBloomTest.CorruptFilters,RawSchema (#5834) Summary: There was significant untested logic in FullFilterBitsReader in the handling of serialized Bloom filter bits that cannot be generated by FullFilterBitsBuilder in the current compilation. These now test many of those corner-case behaviors, including bad metadata or filters created with different cache line size than the current compiled-in value. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5834 Test Plan: thisisthetest Differential Revision: D17726372 Pulled By: pdillinger fbshipit-source-id: fb7b8003b5a8e6fb4666fe95206128f3d5835fc7 02 October 2019, 22:33:48 UTC
d783af1 sdong 02 October 2019, 02:25:12 UTC Fix a timer bug in MergingIterator::Seek() caused by #5871 (#5874) Summary: Conflict resolving in 846e05005d78dfd4276cce6753967cb16930aabb ("Revert "Merging iterator to avoid child iterator reseek for some cases") caused some timer misplaced. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5874 Test Plan: See it build. Differential Revision: D17705073 fbshipit-source-id: 9bd3a8dc4901ac33c2c6fc5b1091ffbc56a8529f 02 October 2019, 02:26:18 UTC
9f31df8 Yanqin Jin 01 October 2019, 21:03:16 UTC Fix compilation error (#5872) Summary: Without this fix, compiler complains. ``` $ROCKSDB_NO_FBCODE=1 USE_CLANG=1 make ldb table/block_based/full_filter_block.cc: In constructor ‘rocksdb::FullFilterBlockBuilder::FullFilterBlockBuilder(const rocksdb::SliceTransform*, bool, rocksdb::FilterBitsBuilder*)’: table/block_based/full_filter_block.cc:20:43: error: declaration of ‘prefix_extractor’ shadows a member of 'this' [-Werror=shadow] FilterBitsBuilder* filter_bits_builder) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5872 Test Plan: ``` $ROCKSDB_NO_FBCODE=1 make all ``` Differential Revision: D17690058 Pulled By: riversand963 fbshipit-source-id: 19e3d9bd86e1123847095240e73d30da5d66240e 01 October 2019, 21:07:13 UTC
846e050 sdong 01 October 2019, 18:20:50 UTC Revert "Merging iterator to avoid child iterator reseek for some cases (#5286)" (#5871) Summary: This reverts commit 9fad3e21eb90d215b6719097baba417bc1eeca3c. Iterator verification in stress tests sometimes fail for assertion table/block_based/block_based_table_reader.cc:2973: void rocksdb::BlockBasedTableIterator<TBlockIter, TValue>::FindBlockForward() [with TBlockIter = rocksdb::DataBlockIter; TValue = rocksdb::Slice]: Assertion `!next_block_is_out_of_bound || user_comparator_.Compare(*read_options_.iterate_upper_bound, index_iter_->user_key()) <= 0' failed. It is likely to be linked to https://github.com/facebook/rocksdb/pull/5286 together with https://github.com/facebook/rocksdb/pull/5468 as the former PR makes some child iterator's seek being avoided, so that upper bound condition fails to be updated there. Strictly speaking, the former PR was merged before the latter one, but the latter one feels a more important improvement so I choose to revert the former one for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5871 Differential Revision: D17689196 fbshipit-source-id: 4ded5be68f67bee2782d31a29cb72ea68f59dd8c 01 October 2019, 18:22:41 UTC
503a756 sdong 01 October 2019, 05:14:03 UTC Fix clang analyze warning in db_stress (#5870) Summary: Recent changes trigger clang analyze warning. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5870 Test Plan: "USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j60 analyze" and make sure it passes. Differential Revision: D17682533 fbshipit-source-id: 02716f2a24572550a22db4bbe9b54d4872dfae32 01 October 2019, 05:15:27 UTC
51413e0 Jay Zhuang 30 September 2019, 21:00:37 UTC Fix a compile error (#5864) Summary: ``` tools/block_cache_analyzer/block_cache_trace_analyzer.cc:653:48: error: implicit conversion loses integer precision: 'uint64_t' (aka 'unsigned long long') to 'std::__1::linear_congruential_engine<unsigned int, 48271, 0, 2147483647>::result_type' (aka 'unsigned int') [-Werror,-Wshorten-64-to-32] std::default_random_engine rand_engine(env_->NowMicros()); ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5864 Differential Revision: D17668962 fbshipit-source-id: e08fa58b2a78a8dd8b334862b5714208f696b8ab 30 September 2019, 21:02:19 UTC
69c4ccb sdong 30 September 2019, 19:36:45 UTC Fix three more db_stress bugs (#5867) Summary: Two more bug fixes in db_stress: 1. this is to complete the fix of the regression bug causing overflowing when supporting FLAGS_prefix_size = -1. 2. Fix regression bug in compare iterator itself: (1) when creating control iterator, which used the same read option as the normal iterator by mistake; (2) the logic of comparing has some problems. Fix them. (3) disable validation for lower bound now, which generated some wildly different results. Disabling it to make normal tests pass while investigating it. 3. Cleaning up snapshots in verification failure cases. Memory is leaked otherwise. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5867 Test Plan: Run "make crash_test" for a while and see at least 1 is fixed. Differential Revision: D17671712 fbshipit-source-id: 011f98ea1a72aef23e19ff28656830c78699b402 30 September 2019, 19:38:23 UTC
643df92 Yanqin Jin 28 September 2019, 00:15:48 UTC Explicitly declare atomic flush incompatible with pipelined write (#5860) Summary: Atomic flush is incompatible with pipelined write. At least now. If pipelined write is enabled, a thread performing write can exit the write thread and start inserting into memtables. Consequently a thread performing flush will enter write thread and race with memtable insertion by the former. This will cause undefined result in terms of data persistence. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5860 Test Plan: ``` $make all && make check ``` Differential Revision: D17638944 Pulled By: riversand963 fbshipit-source-id: abc578dc49a5dbe41bc5adcecf448f8e042a6d49 28 September 2019, 00:17:37 UTC
5cd8aaf sdong 27 September 2019, 23:53:49 UTC db_stress: fix run time error when prefix_size = -1 (#5862) Summary: When prefix_size = -1, stress test crashes with run time error because of overflow. Fix it by not using -1 but 7 in prefix scan mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5862 Test Plan: Run python -u tools/db_crashtest.py --simple whitebox --random_kill_odd \ 888887 --compression_type=zstd and see it doesn't crash. Differential Revision: D17642313 fbshipit-source-id: f029e7651498c905af1b1bee6d310ae50cdcda41 27 September 2019, 23:55:57 UTC
679a45d sdong 27 September 2019, 18:09:06 UTC crash_test to do some verification for prefix extractor and iterator bounds. (#5846) Summary: For now, crash_test is not able to report any failure for the logic related to iterator upper, lower bounds or iterators, or reseek. These are features prone to errors. Improve db_stress in several ways: (1) For each iterator run, reseek up to 3 times. (2) For every iterator, create control iterator with upper or lower bound, with total order seek. Compare the results with the iterator. (3) Make simple crash test to avoid prefix size to have more coverage. (4) make prefix_size = 0 a valid size and -1 to indicate disabling prefix extractor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5846 Test Plan: Manually hack the code to create wrong results and see they are caught by the tool. Differential Revision: D17631760 fbshipit-source-id: acd460a177bd2124a5ffd7fff490702dba63030b 27 September 2019, 18:10:44 UTC
5118559 Chen, You 27 September 2019, 17:18:18 UTC Add unordered write option rocksjava (#5839) Summary: Add unordered_write option api and related ut to rocksjava Pull Request resolved: https://github.com/facebook/rocksdb/pull/5839 Differential Revision: D17604446 Pulled By: maysamyabandeh fbshipit-source-id: c6b07e85ca9d5e3a92973ddb6ab2bc079e53c9c1 27 September 2019, 17:19:40 UTC
ae45835 Yanqin Jin 26 September 2019, 23:59:06 UTC Add TryCatchUpWithPrimary to StackableDB (#5855) Summary: as title. Test Plan (on devserver): ``` $make all && make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5855 Differential Revision: D17615125 Pulled By: riversand963 fbshipit-source-id: bd6ed8cf59eafff41f0d1fc044f39e8f3573172a 27 September 2019, 00:00:55 UTC
76e951d sdong 26 September 2019, 23:16:28 UTC Add a unit test to reproduce a corruption bug (#5851) Summary: This is a bug occaionally shows up in crash test, and this unit test is to reproduce it. The bug is following: 1. Database has multiple CFs. 2. Between one DB restart, the last log file is corrupted in the middle (not the tail) 3. During restart, DB crashes between flushes between two CFs. The DB will fail to be opened again with error "SST file is ahead of WALs" Pull Request resolved: https://github.com/facebook/rocksdb/pull/5851 Test Plan: Run the test itself. Differential Revision: D17614721 fbshipit-source-id: 1b0abce49b203a76a039e38e76bc940429975f20 26 September 2019, 23:18:42 UTC
6652c94 Maysam Yabandeh 24 September 2019, 20:58:27 UTC Fix a bug in format_version 3 + partition filters + prefix search (#5835) Summary: Partitioned filters make use of a top-level index to find the partition in which the filter resides. The top-level index has a key per partition. The key is guaranteed to be larger or equal than any key in that partition. When used with format_version 3, which excludes the sequence number form index keys, the separator key in the index could be equal to the prefix of the keys in the next partition. In this way, when searching for the key, the top-level index will lead us to the previous partition, which has no key with that prefix. The prefix bloom test thus returns false, although the prefix exists in the bloom of the next partition. The patch fixes that by a hack: It always adds the prefix of the first key of the next partition to the bloom of the current partition. In this way, in the corner cases that the index will lead us to the previous partition, we still can find the bloom filter there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5835 Differential Revision: D17513585 Pulled By: maysamyabandeh fbshipit-source-id: e2d1ff26c759e6e03875c4d57f4228316ecf50e9 24 September 2019, 21:00:11 UTC
c9932d1 Levi Tamasi 24 September 2019, 17:59:06 UTC Add class comment for Block Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5832 Differential Revision: D17550773 Pulled By: ltamasi fbshipit-source-id: 66972bb008516e55b6fbba58ddd10234346d5d11 24 September 2019, 18:02:11 UTC
02554b3 WangQingping 23 September 2019, 22:02:02 UTC Update HISTORY.md for stop manual compaction Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5837 Differential Revision: D17529753 fbshipit-source-id: 98bbf22c690384b2f440286151dffdaaa744e97c 23 September 2019, 22:03:39 UTC
2367656 Yikun Jiang 23 September 2019, 19:13:24 UTC Remove invalid comparison of va_list and nullptr (#5836) Summary: The comparison of va_list and nullptr is always False under any arch, and will raise invalid operands of types error in aarch64 env (`error: invalid operands of types ‘va_list {aka __va_list}’ and ‘std::nullptr_t’ to binary ‘operator!=’`). This patch removes this invalid assert. Closes: https://github.com/facebook/rocksdb/issues/4277 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5836 Differential Revision: D17532470 fbshipit-source-id: ca98078ecbc6a9416c69de3bd6ffcfa33a0f0185 23 September 2019, 19:15:01 UTC
42f898b Peter Dillinger 23 September 2019, 16:05:18 UTC Fix format-diff.sh detecting changes vs. upstream (#5831) Summary: format-diff.sh, a.k.a. 'make format', would use 'master' to decide which commits are probably unpublished. Much better to use facebook remote master since local master may not be caught up and may have its own unpublished commits. Script now tries to compare against facebook remote master branch (branch pointer is updated with any fetch or pull), because those differences are what would be considered the differences for a pull request. Also, script would compare against *parent* of merge-base with that reference point, which is just wrong since that includes the last published commit. In case of problems, you can now customize the reference point, by setting the FORMAT_UPSTREAM variable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5831 Test Plan: manual Differential Revision: D17528462 Pulled By: pdillinger fbshipit-source-id: 50fdb8795d683bf3c14d449669c1a5299e0dfa8b 23 September 2019, 16:07:00 UTC
e8263db sdong 20 September 2019, 19:00:55 UTC Apply formatter to recent 200+ commits. (#5830) Summary: Further apply formatter to more recent commits. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5830 Test Plan: Run all existing tests. Differential Revision: D17488031 fbshipit-source-id: 137458fd94d56dd271b8b40c522b03036943a2ab 20 September 2019, 19:04:26 UTC
a5fa873 Vijay Nadimpalli 20 September 2019, 17:05:52 UTC Code comment for Version Edit (#5829) Summary: Added comment for Version Edit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5829 Test Plan: Run existing tests Differential Revision: D17486229 Pulled By: vjnadimpalli fbshipit-source-id: b4b31104fadd667356b64bd2dc409b3376ee46ca 20 September 2019, 17:07:42 UTC
c06b54d sdong 19 September 2019, 19:32:33 UTC Apply formatter on recent 45 commits. (#5827) Summary: Some recent commits might not have passed through the formatter. I formatted recent 45 commits. The script hangs for more commits so I stopped there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5827 Test Plan: Run all existing tests. Differential Revision: D17483727 fbshipit-source-id: af23113ee63015d8a43d89a3bc2c1056189afe8f 19 September 2019, 19:34:17 UTC
6ec6a4a Maysam Yabandeh 19 September 2019, 03:24:17 UTC Remove snap_refresh_nanos option (#5826) Summary: The snap_refresh_nanos option didn't bring much benefit. Remove the feature to simplify the code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5826 Differential Revision: D17467147 Pulled By: maysamyabandeh fbshipit-source-id: 4f950b046990d0d1292d7fc04c2ccafaf751c7f0 19 September 2019, 03:26:04 UTC
a9c5e8e Yanqin Jin 18 September 2019, 23:56:25 UTC Refactor deletefile_test.cc (#5822) Summary: Make DeleteFileTest inherit DBTestBase to avoid code duplication. Test Plan (on devserver) ``` $make deletefile_test $./deletefile_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5822 Differential Revision: D17456750 Pulled By: riversand963 fbshipit-source-id: 224e97967da7b98838a98981cd5095d3230a814f 18 September 2019, 23:58:21 UTC
2cbb61e Levi Tamasi 18 September 2019, 22:22:46 UTC Make clang-analyzer happy (#5821) Summary: clang-analyzer has uncovered a bunch of places where the code is relying on pointers being valid and one case (in VectorIterator) where a moved-from object is being used: In file included from db/range_tombstone_fragmenter.cc:17: ./util/vector_iterator.h:23:18: warning: Method called on moved-from object 'keys' of type 'std::vector' current_(keys.size()) { ^~~~~~~~~~~ 1 warning generated. utilities/persistent_cache/block_cache_tier_file.cc:39:14: warning: Called C++ object pointer is null Status s = env->NewRandomAccessFile(filepath, file, opt); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:47:19: warning: Called C++ object pointer is null Status status = env_->GetFileSize(Path(), size); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:290:14: warning: Called C++ object pointer is null Status s = env_->FileExists(Path()); ^~~~~~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:363:35: warning: Called C++ object pointer is null CacheWriteBuffer* const buf = alloc_->Allocate(); ^~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:399:41: warning: Called C++ object pointer is null const uint64_t file_off = buf_doff_ * alloc_->BufferSize(); ^~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:463:33: warning: Called C++ object pointer is null size_t start_idx = lba.off_ / alloc_->BufferSize(); ^~~~~~~~~~~~~~~~~~~~ utilities/persistent_cache/block_cache_tier_file.cc:515:5: warning: Called C++ object pointer is null alloc_->Deallocate(bufs_[i]); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ 7 warnings generated. ar: creating librocksdb_debug.a utilities/memory/memory_test.cc:68:25: warning: Called C++ object pointer is null cache_set->insert(db->GetDBOptions().row_cache.get()); ^~~~~~~~~~~~~~~~~~ 1 warning generated. The patch fixes these by adding assertions and explicitly passing in zero when initializing VectorIterator::current_ (which preserves the existing behavior). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5821 Test Plan: Ran make check and make analyze to make sure the warnings have disappeared. Differential Revision: D17455949 Pulled By: ltamasi fbshipit-source-id: 363619618ea649a0674287f9f3b3393e390571ee 18 September 2019, 22:25:48 UTC
2389aa2 风 18 September 2019, 21:23:40 UTC Remove unneeded unlock statement (#5809) Summary: The dtor will automatically do unlock Pull Request resolved: https://github.com/facebook/rocksdb/pull/5809 Differential Revision: D17453694 Pulled By: ltamasi fbshipit-source-id: 5348bff8e6a620a05ff639a5454e8d82ae98a22d 18 September 2019, 21:26:37 UTC
6a27903 Yanqin Jin 18 September 2019, 18:49:41 UTC Refactor ObsoleteFilesTest to inherit from DBTestBase (#5820) Summary: Make class ObsoleteFilesTest inherit from DBTestBase. Test plan (on devserver): ``` $COMPILE_WITH_ASAN=1 make obsolete_files_test $./obsolete_files_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5820 Differential Revision: D17452348 Pulled By: riversand963 fbshipit-source-id: b09f4581a18022ca2bfd79f2836c0bf7083f5f25 18 September 2019, 18:52:17 UTC
3a408ee Tomas Kolda 18 September 2019, 16:43:10 UTC Adding support for deleteFilesInRanges in JNI (#4031) Summary: It is very useful method call to achieve https://github.com/facebook/rocksdb/wiki/Delete-A-Range-Of-Keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/4031 Differential Revision: D13515418 Pulled By: vjnadimpalli fbshipit-source-id: 930b48e0992ef07fd1edd0b0cb5f780fabb1b4b5 18 September 2019, 16:45:11 UTC
6d072f2 Yanqin Jin 18 September 2019, 00:15:13 UTC Move WAL-closing loop out of original loop (#5804) Summary: Originally the loop of closing WAL in PurgeObsoleteFiles resides inside a loop iterating over the candidate files. It should be moved out. Test plan (devserver) ``` $COMPILE_WITH_ASAN=1 make -j32 all $make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5804 Differential Revision: D17374350 Pulled By: riversand963 fbshipit-source-id: 2bee7343fc0481d9a385a87c7676491522285c96 18 September 2019, 00:17:19 UTC
43a340b sdong 18 September 2019, 00:08:57 UTC Merging iterator to disble reseek optimization in prefix seek (#5815) Summary: We are seeing a bug of wrong results with merging iterator's reseek avoidence feature and prefix extractor. Disable this optimization for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5815 Test Plan: Validated the same MyRocks case was fixed; run all existing tests. Differential Revision: D17430776 fbshipit-source-id: aef664277ba0ab8a2e68331ff0db6ae682535371 18 September 2019, 00:10:29 UTC
a68d814 Yi Wu 17 September 2019, 23:43:07 UTC fast look up purge_queue (#5796) Summary: purge_queue_ maybe contains thousands sst files, for example manual compact a range. If full scan is triggered at the same time and the total sst files number is large, RocksDB will be blocked at https://github.com/facebook/rocksdb/blob/master/db/db_impl_files.cc#L150 for several seconds. In our environment we have 140,000 sst files and the manual compaction delete about 1000 sst files, it blocked about 2 minutes. Commandeering https://github.com/facebook/rocksdb/issues/5290. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5796 Differential Revision: D17357775 Pulled By: riversand963 fbshipit-source-id: 20eacca917355b8de975ccc7b1c9a3e7bd5b201a 17 September 2019, 23:47:55 UTC
9a87ae4 Maysam Yabandeh 17 September 2019, 07:14:29 UTC Use total charge in MaintainPoolSize (#5813) Summary: https://github.com/facebook/rocksdb/issues/5797 charges the block cache with the total of user-provided charge plus the metadata charge. It had a bug where in MaintainPoolSize the user-provided charge was used instead of the total charge. The patch fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5813 Differential Revision: D17412783 Pulled By: maysamyabandeh fbshipit-source-id: 45c0ac9f1e2233760db5ccd61399605cd74edc87 17 September 2019, 07:16:13 UTC
6287f0d sdong 17 September 2019, 04:02:27 UTC Improve readability of DBIter's two seek functions (#5794) Summary: Doing some code reordering in DBIter::Seek() and DBIter::SeekForPrev(). The logic largely remains the same, except slight difference when handling some stats when valid_ = false, where they are not supposed to be used anyway. Also remove prefix_start_key_, which sometimes point a part of seek target, some times prefix_start_buf_, which is confusing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5794 Test Plan: Run all tests. Differential Revision: D17375257 fbshipit-source-id: 7339a23898cecd3a8475bf72340fcd6f82b933c5 17 September 2019, 04:05:07 UTC
6226830 andrew 17 September 2019, 04:00:13 UTC Allow users to stop manual compactions (#3971) Summary: Manual compaction may bring in very high load because sometime the amount of data involved in a compaction could be large, which may affect online service. So it would be good if the running compaction making the server busy can be stopped immediately. In this implementation, stopping manual compaction condition is only checked in slow process. We let deletion compaction and trivial move go through. Pull Request resolved: https://github.com/facebook/rocksdb/pull/3971 Test Plan: add tests at more spots. Differential Revision: D17369043 fbshipit-source-id: 575a624fb992ce0bb07d9443eb209e547740043c 17 September 2019, 04:01:47 UTC
f5a59c4 sdong 17 September 2019, 03:41:16 UTC Update dependencies (#5777) Summary: Update version of dependencies. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5777 Test Plan: make release Differential Revision: D17269421 fbshipit-source-id: e76dbe5389e1d7f811739d3bc1e404b482dfce34 17 September 2019, 03:44:47 UTC
811e403 sdong 17 September 2019, 03:40:44 UTC Dedup IsFileSectorAligned() to fix unity build. (#5812) Summary: Unity build fails because of name conflict of IsFileSectorAligned() after recent refactoring. Consolidate the function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5812 Test Plan: make unity. At least the failure goes away. Also "make all", "make release" and see no regression in normal cases. Differential Revision: D17411403 fbshipit-source-id: 09d5653471ae2c3a4d898e120a024f7dd08d9c9d 17 September 2019, 03:42:07 UTC
6862624 Peter Dillinger 16 September 2019, 23:15:18 UTC Refactor/consolidate legacy Bloom implementation details (#5784) Summary: Refactoring to consolidate implementation details of legacy Bloom filters. This helps to organize and document some related, obscure code. Also added make/cpp var TEST_CACHE_LINE_SIZE so that it's easy to compile and run unit tests for non-native cache line size. (Fixed a related test failure in db_properties_test.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5784 Test Plan: make check, including Recently added Bloom schema unit tests (in ./plain_table_db_test && ./bloom_test), and including with TEST_CACHE_LINE_SIZE=128U and TEST_CACHE_LINE_SIZE=256U. Tested the schema tests with temporary fault injection into new implementations. Some performance testing with modified unit tests suggest a small to moderate improvement in speed. Differential Revision: D17381384 Pulled By: pdillinger fbshipit-source-id: ee42586da996798910fc45ac0b6289147f16d8df 16 September 2019, 23:17:09 UTC
638d239 Maysam Yabandeh 16 September 2019, 22:14:51 UTC Charge block cache for cache internal usage (#5797) Summary: For our default block cache, each additional entry has extra memory overhead. It include LRUHandle (72 bytes currently) and the cache key (two varint64, file id and offset). The usage is not negligible. For example for block_size=4k, the overhead accounts for an extra 2% memory usage for the cache. The patch charging the cache for the extra usage, reducing untracked memory usage outside block cache. The feature is enabled by default and can be disabled by passing kDontChargeCacheMetadata to the cache constructor. This PR builds up on https://github.com/facebook/rocksdb/issues/4258 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5797 Test Plan: - Existing tests are updated to either disable the feature when the test has too much dependency on the old way of accounting the usage or increasing the cache capacity to account for the additional charge of metadata. - The Usage tests in cache_test.cc are augmented to test the cache usage under kFullChargeCacheMetadata. Differential Revision: D17396833 Pulled By: maysamyabandeh fbshipit-source-id: 7684ccb9f8a40ca595e4f5efcdb03623afea0c6f 16 September 2019, 22:26:21 UTC
94d62d7 Levi Tamasi 16 September 2019, 18:38:10 UTC Temporarily disable partitioned index/filter in stress test (#5811) Summary: PR https://github.com/facebook/rocksdb/issues/4020 enabled partitioned indexes/filters in stress tests; however, this causes assertion failures in BatchedOpsStressTest. This patch disables them until we can root cause the failures. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5811 Test Plan: Ran the script and made sure it only uses the binary search index. Differential Revision: D17399366 Pulled By: ltamasi fbshipit-source-id: adb116e6297f9c6ccd7ac15b6a16c9aa91f21ac5 16 September 2019, 18:41:35 UTC
d3a6726 Peter Dillinger 16 September 2019, 18:36:58 UTC Revert changes from PR#5784 accidentally in PR#5780 (#5810) Summary: This will allow us to fix history by having the code changes for PR#5784 properly attributed to it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5810 Differential Revision: D17400231 Pulled By: pdillinger fbshipit-source-id: 2da8b1cdf2533cfedb35b5526eadefb38c291f09 16 September 2019, 18:38:53 UTC
9bd5fce sdong 16 September 2019, 17:49:20 UTC Refactor UniversalCompactionPicker code a little bit (#5639) Summary: Several functions of UniversalCompactionPicker share most of the parameters. Move these functions to a class with those shared arguments as class members. Hopefully this will make code slightly easier to maintain. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5639 Test Plan: Run all existing test. Differential Revision: D16996403 fbshipit-source-id: fffafd1897ab132b420b1dec073542cffb5c44de 16 September 2019, 17:51:11 UTC
54fc617 Peter (Stig) Edwards 16 September 2019, 17:42:55 UTC Mention sst_dump cmd=recompress changes (#5807) Summary: As requested by siying in https://github.com/facebook/rocksdb/pull/5791#issuecomment-531417468 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5807 Differential Revision: D17399349 fbshipit-source-id: 5986c3894f356becd393fee0f1aeadcd9affc798 16 September 2019, 17:45:03 UTC
b931f84 sdong 16 September 2019, 17:31:27 UTC Divide file_reader_writer.h and .cc (#5803) Summary: file_reader_writer.h and .cc contain several files and helper function, and it's hard to navigate. Separate it to multiple files and put them under file/ Pull Request resolved: https://github.com/facebook/rocksdb/pull/5803 Test Plan: Build whole project using make and cmake. Differential Revision: D17374550 fbshipit-source-id: 10efca907721e7a78ed25bbf74dc5410dea05987 16 September 2019, 17:33:51 UTC
915d72d Peter Dillinger 16 September 2019, 16:35:51 UTC Improve accuracy testing for DynamicBloom (#5805) Summary: DynamicBloom unit test now tests non-sequential as well as sequential keys in testing FP rates. Also now verifies larger structures. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5805 Test Plan: thisisthetest Differential Revision: D17398109 Pulled By: pdillinger fbshipit-source-id: 374074206c76d242efa378afc27830448a0e892a 16 September 2019, 16:37:42 UTC
0b59ef5 Ariel Hurdle 16 September 2019, 04:27:44 UTC Add avrio to USERS.md (#5748) Summary: Add Avrio's use case of RocksDB to USERS.md Pull Request resolved: https://github.com/facebook/rocksdb/pull/5748 Differential Revision: D17392483 Pulled By: riversand963 fbshipit-source-id: 673f4e1a5c82079ec8dfb2816db0cc9af9d38341 16 September 2019, 04:29:09 UTC
2ed9162 Peter (Stig) Edwards 13 September 2019, 23:29:16 UTC sst_dump recompress show #blocks compressed and not compressed (#5791) Summary: Closes https://github.com/facebook/rocksdb/issues/1474 Helps show when the 12.5% threshold for GoodCompressionRatio (originally from ldb) is hit. Example output: ``` > ./sst_dump --file=/tmp/test.sst --command=recompress from [] to [] Process /tmp/test.sst Sst file format: block-based Block Size: 16384 Compression: kNoCompression Size: 122579836 Blocks: 2300 Compressed: 0 ( 0.0%) Not compressed (ratio): 2300 (100.0%) Not compressed (abort): 0 ( 0.0%) Compression: kSnappyCompression Size: 46289962 Blocks: 2300 Compressed: 2119 ( 92.1%) Not compressed (ratio): 181 ( 7.9%) Not compressed (abort): 0 ( 0.0%) Compression: kZlibCompression Size: 29689825 Blocks: 2300 Compressed: 2301 (100.0%) Not compressed (ratio): 0 ( 0.0%) Not compressed (abort): 0 ( 0.0%) Unsupported compression type: kBZip2Compression. Compression: kLZ4Compression Size: 44785490 Blocks: 2300 Compressed: 1950 ( 84.8%) Not compressed (ratio): 350 ( 15.2%) Not compressed (abort): 0 ( 0.0%) Compression: kLZ4HCCompression Size: 37498895 Blocks: 2300 Compressed: 2301 (100.0%) Not compressed (ratio): 0 ( 0.0%) Not compressed (abort): 0 ( 0.0%) Unsupported compression type: kXpressCompression. Compression: kZSTD Size: 32208707 Blocks: 2300 Compressed: 2301 (100.0%) Not compressed (ratio): 0 ( 0.0%) Not compressed (abort): 0 ( 0.0%) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5791 Differential Revision: D17347870 fbshipit-source-id: af10849c010b46b20e54162b70123c2805ffe526 13 September 2019, 23:30:41 UTC
bf5dbc1 sdong 13 September 2019, 22:59:16 UTC merging_iterator.cc: Small refactoring (#5793) Summary: 1. Put the similar logic of adding valid iterator to heap and check invalid iterator's status code to the same helper functions. 2. Because of 1, in the changing direction case, move around the places where we check status a little bit so that we can call the helper function there too. The logic would only divert in the case where the iterator is valid but status is not OK, which is not expected to happen. Add an assertion for that. 3. Put the logic of changing direction from forward to backward to a separate function so the unlikely code path is not in Prev(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5793 Test Plan: run all existing tests. Differential Revision: D17374397 fbshipit-source-id: d595ffcf156095c4bd0f5532bacba854482a2332 13 September 2019, 23:01:13 UTC
9763135 Igor Canadi 13 September 2019, 21:48:18 UTC Allow ingesting overlapping files (#5539) Summary: Currently IngestExternalFile() fails when its input files' ranges overlap. This condition doesn't need to hold for files that are to be ingested in L0, though. This commit allows overlapping files and forces their target level to L0. Additionally, ingest job's completion is logged to EventLogger, analogous to flush and compaction jobs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5539 Differential Revision: D17370660 Pulled By: riversand963 fbshipit-source-id: 749a3899b17d1be267a5afd5b0a99d96b38ab2f3 13 September 2019, 21:49:47 UTC
83a6a61 anand76 13 September 2019, 20:48:04 UTC Refactor ArenaWrappedDBIter into separate files (#5801) Summary: Move definition and implementation for ArenaWrappedDBIter into its own .h/.cc files. Also, change inlining of functions to better comply with the Google C++ style guide. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5801 Test Plan: make check Differential Revision: D17371012 Pulled By: anand1976 fbshipit-source-id: c1361abc2851575111e357a63d88be3b3d6cb341 13 September 2019, 20:50:43 UTC
6a17172 Peter Dillinger 13 September 2019, 18:04:52 UTC Clean up + fix build scripts re: USE_SSE= and PORTABLE= (#5800) Summary: In preparing to utilize a new Intel instruction extension, I noticed problems with the existing build script in regard to the existing utilized extensions, either with USE_SSE or PORTABLE flags. * PORTABLE=0 was interpreted the same as PORTABLE=1. Now empty and 0 mean the same. (I guess you were not supposed to set PORTABLE= if you wanted non-portable--except that...) * The Facebook build script extensions would set PORTABLE=1 even if it's already set in a make var or environment. Now it does not override a non-empty setting, so use PORTABLE=0 for fully optimized build, overriding Facebook environment default. * Put in an explanation of the USE_SSE flag where it's used by build_detect_platform, and cleaned up some confusing/redundant associated logic. * If USE_SSE was set and expected intrinsics were not available, build_detect_platform would exit early but build would proceed with broken, incomplete configuration. Now warning is gracefully recovered. * If USE_SSE was set and expected intrinsics were not available, build would still try to use flags like -msse4.2 etc. which could lead to unexpected compilation failure or binary incompatibility. Now those flags are not used if the warning is issued. This should not break or change existing, valid build scripts. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5800 Test Plan: manual case testing Differential Revision: D17369543 Pulled By: pdillinger fbshipit-source-id: 4ee244911680ae71144d272c40aceea548e3ce88 13 September 2019, 18:07:13 UTC
9ba88a1 Lingjing You 13 September 2019, 17:47:47 UTC Update history.md for option memtable_insert_hint_per_batch (#5799) Summary: Update history.md for option memtable_insert_hint_per_batch Pull Request resolved: https://github.com/facebook/rocksdb/pull/5799 Differential Revision: D17369186 fbshipit-source-id: 71d82f9d99d9a52d1475d1b0153670957b6111e9 13 September 2019, 17:51:32 UTC
27f516a Ronak Sisodia 13 September 2019, 17:41:32 UTC Update HISTORY.md for option to make write group size configurable (#5798) Summary: Update HISTORY.md for option to make write group size configurable . Pull Request resolved: https://github.com/facebook/rocksdb/pull/5798 Differential Revision: D17369062 fbshipit-source-id: 390a3fa0b01675e91879486a729cf2cc7624d106 13 September 2019, 17:43:09 UTC
aa2486b Peter Dillinger 13 September 2019, 17:24:38 UTC Refactor some confusing logic in PlainTableReader Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5780 Test Plan: existing plain table unit test Differential Revision: D17368629 Pulled By: pdillinger fbshipit-source-id: f25409cdc2f39ebe8d5cbb599cf820270e6b5d26 13 September 2019, 17:26:36 UTC
1a928c2 Lingjing You 12 September 2019, 23:53:31 UTC Add insert hints for each writebatch (#5728) Summary: Add insert hints for each writebatch so that they can be used in concurrent write, and add write option to enable it. Bench result (qps): `./db_bench --benchmarks=fillseq -allow_concurrent_memtable_write=true -num=4000000 -batch-size=1 -threads=1 -db=/data3/ylj/tmp -write_buffer_size=536870912 -num_column_families=4` master: | batch size \ thread num | 1 | 2 | 4 | 8 | | ----------------------- | ------- | ------- | ------- | ------- | | 1 | 387883 | 220790 | 308294 | 490998 | | 10 | 1397208 | 978911 | 1275684 | 1733395 | | 100 | 2045414 | 1589927 | 1798782 | 2681039 | | 1000 | 2228038 | 1698252 | 1839877 | 2863490 | fillseq with writebatch hint: | batch size \ thread num | 1 | 2 | 4 | 8 | | ----------------------- | ------- | ------- | ------- | ------- | | 1 | 286005 | 223570 | 300024 | 466981 | | 10 | 970374 | 813308 | 1399299 | 1753588 | | 100 | 1962768 | 1983023 | 2676577 | 3086426 | | 1000 | 2195853 | 2676782 | 3231048 | 3638143 | Pull Request resolved: https://github.com/facebook/rocksdb/pull/5728 Differential Revision: D17297240 fbshipit-source-id: b053590a6d77871f1ef2f911a7bd013b3899b26c 13 September 2019, 00:15:18 UTC
a378a4c HouBingjian 12 September 2019, 23:52:41 UTC arm64 crc prefetch optimise (#5773) Summary: prefetch data for following block，avoid cache miss when doing crc caculate I do performance test at kunpeng-920 server(arm-v8, 64core@2.6GHz) ./db_bench --benchmarks=crc32c --block_size=500000000 before optimise : 587313.500 micros/op 1 ops/sec; 811.9 MB/s (500000000 per op) after optimise : 289248.500 micros/op 3 ops/sec; 1648.5 MB/s (500000000 per op) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5773 Differential Revision: D17347339 fbshipit-source-id: bfcd74f0f0eb4b322b959be68019ddcaae1e3341 12 September 2019, 23:59:44 UTC
d35ffd5 Levi Tamasi 12 September 2019, 19:10:05 UTC Temporarily disable hash index in stress tests (#5792) Summary: PR https://github.com/facebook/rocksdb/issues/4020 implicitly enabled the hash index as well in stress/crash tests, resulting in assertion failures in Block. This patch disables the hash index until we can pinpoint the root cause of these issues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5792 Test Plan: Ran tools/db_crashtest.py and made sure it only uses index types 0 and 2 (binary search and partitioned index). Differential Revision: D17346777 Pulled By: ltamasi fbshipit-source-id: b4318f37f1fda3ee1bbff4ef2c2f556ca9e6b551 12 September 2019, 19:11:34 UTC
e8c2e68 Adam Retter 12 September 2019, 01:35:03 UTC Fix RocksDB bug in block_cache_trace_analyzer.cc on Windows (#5786) Summary: This is required to compile on Windows with Visual Studio 2015. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5786 Differential Revision: D17335994 fbshipit-source-id: 8f9568310bc6f697e312b5e24ad465e9084f0011 12 September 2019, 01:36:41 UTC
d05c0fe Ronak Sisodia 12 September 2019, 01:26:22 UTC Option to make write group size configurable (#5759) Summary: The max batch size that we can write to the WAL is controlled by a static manner. So if the leader write is less than 128 KB we will have the batch size as leader write size + 128 KB else the limit will be 1 MB. Both of them are statically defined. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5759 Differential Revision: D17329298 fbshipit-source-id: a3d910629d8d8ca84ea39ad89c2b2d284571ded5 12 September 2019, 01:28:33 UTC
9eb3e1f Shylock Hg 12 September 2019, 01:07:12 UTC Use delete to disable automatic generated methods. (#5009) Summary: Use delete to disable automatic generated methods instead of private, and put the constructor together for more clear.This modification cause the unused field warning, so add unused attribute to disable this warning. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5009 Differential Revision: D17288733 fbshipit-source-id: 8a767ce096f185f1db01bd28fc88fef1cdd921f3 12 September 2019, 01:09:00 UTC
fcda80f Wilfried Goesgens 12 September 2019, 00:58:34 UTC record the timestamp on first configure (#4799) Summary: cmake doesn't re-generate the timestamp on subsequent builds causing rebuilds of the lib This improves compile time turn-arounds if you have rocksdb as a compileable library include, since with the state its now it will re-generate the time stamp .cc file each time you build, and thus re-compile + re-link the rocksdb library though anything in the source actually changed. The original timestamp is recorded into `CMakeCache.txt` and will remain there until you flush this cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4799 Differential Revision: D17290040 fbshipit-source-id: 28357fef3422693c9c19e88fa2873c8db0f662ed 12 September 2019, 01:00:02 UTC
dd2a35f Andrew Kryczka 11 September 2019, 21:11:38 UTC Support partitioned index and filters in stress/crash tests (#4020) Summary: - In `db_stress`, support choosing index type and whether to enable filter partitioning, and randomly set those options in crash test - When partitioned filter is enabled by crash test, force partitioned index to also be enabled since it's a prerequisite Pull Request resolved: https://github.com/facebook/rocksdb/pull/4020 Test Plan: currently this is blocked on fixing the bug that crash test caught: ``` $ TEST_TMPDIR=/data/compaction_bench python ./tools/db_crashtest.py blackbox --simple --interval=10 --max_key=10000000 ... Verification failed for column family 0 key 937501: Value not found: NotFound: Crash-recovery verification failed :( ``` Differential Revision: D8508683 Pulled By: maysamyabandeh fbshipit-source-id: 0337e5d0558bcef26b1f3699f47265a2c1e99629 11 September 2019, 21:13:38 UTC
20dd828 Andrew Kryczka 11 September 2019, 21:04:54 UTC Avoid clock_gettime on pre-10.12 macOS versions (#5570) Summary: On older macOS like 10.10 we saw the following compiler error: ``` /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb/env/env_posix.cc:845:19: error: use of undeclared identifier 'CLOCK_THREAD_CPUTIME_ID' clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts); ^ ``` According to mac's `man clock_gettime`: "These functions first appeared in Mac OSX 10.12". So we should not try to compile it on earlier versions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5570 Test Plan: verified it compiles now on 10.10. Also did some investigation to ensure it does not cause regression on macOS 10.12+, although I do not have access to such an environment to really test. Differential Revision: D17322629 Pulled By: riversand963 fbshipit-source-id: e0a412223854f826b4d83e6d15c3739ff4620d7d 11 September 2019, 21:07:25 UTC
c85c87a tongyingrui 11 September 2019, 19:02:49 UTC test size was wrong in 'fillbatch' benchmark (#5198) Summary: for fillbatch benchmar, the numEntries should be [num_] but not [num_ / 1000] because numEntries is just the total entries we want to test Pull Request resolved: https://github.com/facebook/rocksdb/pull/5198 Differential Revision: D17274664 Pulled By: anand1976 fbshipit-source-id: f96e952babdbac63fb99d14e1254d478a10437be 11 September 2019, 19:04:44 UTC
2becafd anand76 10 September 2019, 21:32:38 UTC Fix Appveyor build due to signed/unsigned comparison Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5788 Test Plan: Travis CI and Appveyor should complete successfully. Differential Revision: D17287422 Pulled By: anand1976 fbshipit-source-id: d9408b692f78be95d0088b29b33f6a8ff40ec97b 10 September 2019, 21:34:37 UTC
eb9026f anand76 10 September 2019, 18:03:31 UTC Add a db_bench benchmark to warm up the row cache Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5707 Differential Revision: D17242698 Pulled By: anand1976 fbshipit-source-id: 5d1bfda3c9e8f56176ae391cae6c91e6262016b8 10 September 2019, 18:06:36 UTC
4d945c5 jsteemann 10 September 2019, 16:40:21 UTC do a bit less work in the normal case (#5695) Summary: i.e. if alive logfile is not being moved to archive while we are in GetSortedWalsOfType() Pull Request resolved: https://github.com/facebook/rocksdb/pull/5695 Differential Revision: D17279489 Pulled By: vjnadimpalli fbshipit-source-id: 02bcf920a75b812edba8b87c6079b4e6fd5e683c 10 September 2019, 16:41:45 UTC
699e1b5 Richard He 10 September 2019, 01:11:02 UTC Added support for SstFileReader JNI interface (#5556) Summary: Feature request as per https://github.com/facebook/rocksdb/issues/5538 issue. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5556 Differential Revision: D17219008 fbshipit-source-id: e31f18dec318416eac9dea8213bab31da96e1f3a 10 September 2019, 01:12:53 UTC
7af6ced Peter Dillinger 09 September 2019, 22:24:54 UTC Fix block allocation bug in new DynamicBloom (#5783) Summary: Bug found by valgrind. New DynamicBloom wasn't allocating in block sizes. New assertion added that probes starting in final word would be in bounds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5783 Test Plan: ROCKSDB_VALGRIND_RUN=1 DISABLE_JEMALLOC=1 valgrind --leak-check=full ./dynamic_bloom_test Differential Revision: D17270623 Pulled By: pdillinger fbshipit-source-id: 1e0407504b875133a771383cd488c70f91be2b87 09 September 2019, 22:26:43 UTC
108c619 Peter Dillinger 09 September 2019, 21:49:39 UTC Add regression test for serialized Bloom filters (#5778) Summary: Check that we don't accidentally change the on-disk format of existing Bloom filter implementations, including for various CACHE_LINE_SIZE (by changing temporarily). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5778 Test Plan: thisisthetest Differential Revision: D17269630 Pulled By: pdillinger fbshipit-source-id: c77017662f010a77603b7d475892b1f0d5563d8b 09 September 2019, 21:51:30 UTC
fbab991 Wilfried Goesgens 09 September 2019, 18:22:28 UTC upgrade gtest 1.7.0 => 1.8.1 for json result writing Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5332 Differential Revision: D17242232 fbshipit-source-id: c0d4646556a1335e51ac7382b986ca7f6ced7b64 09 September 2019, 18:24:11 UTC
back to top