Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
515a279 Yanqin Jin 27 May 2020, 23:38:10 UTC Update version and HISTORY 27 May 2020, 23:38:10 UTC
138a967 Yanqin Jin 17 May 2020, 04:46:21 UTC Fix buck target db_stress_lib in opt mode (#6847) Summary: In buck build with opt mode, target should not include rocksdb_test_lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6847 Test Plan: Watch for internal cont build. Reviewed By: ajkr Differential Revision: D21586803 Pulled By: riversand963 fbshipit-source-id: 76d253c18d16fac6cab86a8c3f6b471ad5b6efb3 18 May 2020, 17:45:56 UTC
98d19c3 Yanqin Jin 13 May 2020, 04:35:08 UTC Do not print u'string' in TARGETS file (#6841) Summary: Before this PR, extra deps passed in from cmd line to buckifier will be parsed and used to populate a dict. Using this dict and printing to TARGETS file will lead to printing u'', disallowed by build tools. This PR removes the u''. Test Plan (local dev server): ``` python buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}' ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841 Reviewed By: siying Differential Revision: D21538155 Pulled By: riversand963 fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349 13 May 2020, 23:15:06 UTC
3f4008b anand76 13 May 2020, 17:11:44 UTC Update HISTORY.md with a missing API change 13 May 2020, 17:11:44 UTC
8f9cc10 sdong 08 May 2020, 21:12:18 UTC Improve ldb consistency checks (#6802) Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0 11 May 2020, 19:38:50 UTC
bfc1b7a Yanqin Jin 08 May 2020, 19:59:02 UTC Fix a few bugs in best-efforts recovery (#6824) Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2 08 May 2020, 21:14:20 UTC
e0fcbf9 Andrew Kryczka 08 May 2020, 19:50:31 UTC Fixup formatting of 15ee2ee "fix swallowed error for file deletion consistency check (#6809)" 08 May 2020, 19:50:31 UTC
f08ee8d anand76 07 May 2020, 22:52:58 UTC Include options.h in table.h (#6823) Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d 08 May 2020, 18:48:48 UTC
60de1e6 anand76 06 May 2020, 23:40:32 UTC Update release version to 6.10 (#6797) Summary: Update HISTORY.md and version.h to 6.10. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6797 Reviewed By: zhichao-cao Differential Revision: D21371390 Pulled By: anand1976 fbshipit-source-id: 6017bca24fc5d12076d1ddaec7783c9b85712d42 08 May 2020, 18:48:17 UTC
18d0c90 Peter Dillinger 05 May 2020, 20:09:12 UTC Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) Summary: This very old test code bug was causing a new valgrind failure in MultiGetDeadlineExceeded Also fix hang in MultiGetDeadlineExceeded by unifying with some logic from another test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6803 Test Plan: run that unit test under valgrind, make check Reviewed By: siying Differential Revision: D21388470 Pulled By: pdillinger fbshipit-source-id: 0ce99d6d5eb8cd3195b17406892c8c5cff5fa5dd 08 May 2020, 17:11:04 UTC
6d6e857 Yanqin Jin 05 May 2020, 17:44:12 UTC Do not swallow error returned from SaveTo() (#6801) Summary: With consistency check enabled, VersionBuilder::SaveTo() may return error once corruption is detected while building versions. We should handle these errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6801 Test Plan: make check Reviewed By: siying Differential Revision: D21385045 Pulled By: riversand963 fbshipit-source-id: 98f6424e2a4699b62befa21e9fe00e70a771118e 08 May 2020, 17:02:02 UTC
3df88b3 anand76 08 May 2020, 00:30:41 UTC Fix race due to delete triggered compaction in Universal compaction mode (#6799) Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a 08 May 2020, 16:02:10 UTC
4584a99 Peter Dillinger 07 May 2020, 22:39:49 UTC Fix false NotFound from batched MultiGet with kHashSearch (#6821) Summary: The error is assigning KeyContext::s to NotFound status in a table reader for a "not found in this table" case, which skips searching in later tables, like only a delete should. (The hash search index iterator is the only one that can return status NotFound even if Valid() == false.) This was detected by intermittent failure in MultiThreadedDBTest.MultiThreaded/5, a kHashSearch configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6821 Test Plan: modified existing unit test to reproduce problem Reviewed By: anand1976 Differential Revision: D21450469 Pulled By: pdillinger fbshipit-source-id: 7478003684d637dbd491cdac81468041a791be2c 08 May 2020, 15:58:06 UTC
cb33efd sdong 04 May 2020, 21:15:55 UTC Avoid Swallowing Some File Consistency Checking Bugs (#6793) Summary: We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump. More places are not fixed and need follow-up. Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793 Test Plan: Add a unit test to cover the reopen case. Reviewed By: riversand963 Differential Revision: D21366525 fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded 08 May 2020, 15:51:41 UTC
15ee2ee Andrew Kryczka 05 May 2020, 21:52:18 UTC fix swallowed error for file deletion consistency check (#6809) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6809 Reviewed By: pdillinger Differential Revision: D21411971 Pulled By: ajkr fbshipit-source-id: 900b6b0370b76e9a3e5e03f968e2ac1bbaab73b8 07 May 2020, 21:35:10 UTC
211088d Cheng Chang 02 May 2020, 00:35:41 UTC Remove redundant update of txn_state_ in transaction Prepare (#6778) Summary: When expiration is set in a pessimistic transaction, `txn_state_` is already updated to `AWAITING_PREPARE` in the `if (expiration_time_ > 0)` block, there is no need to update the state in `if (can_prepare)` block again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6778 Test Plan: make check Reviewed By: lth Differential Revision: D21335319 Pulled By: cheng-chang fbshipit-source-id: 251d634cc7d1a0e86e673a59f0bda8584da5a35f 02 May 2020, 00:37:33 UTC
c8643ed Zhichao Cao 01 May 2020, 07:01:21 UTC Fix multiple CF replay failure in db_bench replay (#6787) Summary: The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787 Test Plan: pass trace replay test. Reviewed By: yhchiang Differential Revision: D21339941 Pulled By: zhichao-cao fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce 01 May 2020, 07:03:38 UTC
6acbbbf Yanqin Jin 01 May 2020, 02:20:55 UTC Add Github Action for some basic sanity test of PR (#6761) Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787 01 May 2020, 02:22:45 UTC
6504ae0 sdong 30 April 2020, 23:59:16 UTC Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) Summary: The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782 Test Plan: Run all existing tests. Reviewed By: zhichao-cao Differential Revision: D21338614 fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685 01 May 2020, 00:01:17 UTC
ef0c3ed Cheng Chang 30 April 2020, 23:21:05 UTC Make users explicitly be aware of prepare before commit (#6775) Summary: In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare. This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775 Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare Reviewed By: lth Differential Revision: D21313270 Pulled By: cheng-chang fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251 30 April 2020, 23:24:20 UTC
079e50d sdong 30 April 2020, 22:34:43 UTC Disallow BlockBasedTableBuilder to set status from non-OK (#6776) Summary: There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future. The solution is to only make it possible to set the status code if the status code to set is not OK. Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776 Test Plan: Run all existing test. Reviewed By: pdillinger Differential Revision: D21314382 fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375 30 April 2020, 22:37:03 UTC
6277e28 sdong 30 April 2020, 22:19:31 UTC Flag CompressionOptions::parallel_threads to be experimental (#6781) Summary: The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781 Reviewed By: pdillinger Differential Revision: D21330678 fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13 30 April 2020, 22:22:06 UTC
ab13d43 anand76 30 April 2020, 21:48:51 UTC Pass a timeout to FileSystem for random reads (#6751) Summary: Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded. For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR. Tests: Update existing unit tests to verify the correct timeout value is being passed Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751 Reviewed By: riversand963 Differential Revision: D21285631 Pulled By: anand1976 fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567 30 April 2020, 21:50:39 UTC
eecd8fb Peter Dillinger 30 April 2020, 19:08:34 UTC Fix assertion that can fail on sst corruption (#6780) Summary: An assertion that a char == a CompressionType (unsigned char) originally cast from a char can fail if the original value is negative, due to numeric promotion. The assertion should pass even if the value is invalid CompressionType, because the callee UncompressBlockContentsForCompressionType checks for that and reports status appropriately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780 Test Plan: Temporarily change kZSTD = 0x88 and see tests fail. Make this change (in addition), and tests pass. Reviewed By: siying Differential Revision: D21328498 Pulled By: pdillinger fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92 30 April 2020, 19:11:00 UTC
fe238e5 Levi Tamasi 30 April 2020, 18:23:32 UTC Keep track of obsolete blob files in VersionSet (#6755) Summary: The patch adds logic to keep track of obsolete blob files. A blob file becomes obsolete when the last `shared_ptr` that points to the corresponding `SharedBlobFileMetaData` object goes away, which, in turn, happens when the last `Version` that contains the blob file is destroyed. No longer needed blob files are added to the obsolete list in `VersionSet` using a custom deleter to avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`. Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored in `JobContext`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21233155 Pulled By: ltamasi fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a 30 April 2020, 18:25:51 UTC
cf34246 Adam Retter 30 April 2020, 17:56:33 UTC Add Slack forum to README (#6773) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773 Reviewed By: siying Differential Revision: D21310229 Pulled By: pdillinger fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf 30 April 2020, 18:00:28 UTC
e619a20 Ziyue Yang 30 April 2020, 17:46:54 UTC Add an option for parallel compression in for db_stress (#6722) Summary: This commit adds an `compression_parallel_threads` option in db_stress. It also fixes the naming of parallel compression option in db_bench to keep it aligned with others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722 Reviewed By: pdillinger Differential Revision: D21091385 fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100 30 April 2020, 17:49:07 UTC
8c69402 Zhichao Cao 30 April 2020, 15:38:18 UTC Fix potential size_t overflow in import_column_family (#6762) Summary: The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762 Test Plan: make asan_check Reviewed By: pdillinger Differential Revision: D21323063 Pulled By: zhichao-cao fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a 30 April 2020, 15:40:42 UTC
b938e60 anand76 30 April 2020, 02:25:56 UTC Fix a couple of bugs in FaultInjectionTestFS (#6777) Summary: Fix the following cases that can cause false alarms in db_stress when read fault injection is enabled - 1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error. 2. Handle the case when the randomly generated string to overwrite the original block is identical to the original. Tests: Run db_stress w/ and wo/ direct IO and fault injection turned on Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777 Reviewed By: zhichao-cao Differential Revision: D21316734 Pulled By: anand1976 fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa 30 April 2020, 02:28:29 UTC
28fe8e4 Peter Dillinger 29 April 2020, 20:42:43 UTC Fix bug in format-diff.sh (#6772) Summary: Nasty bug in which more/different changes would be applied than those shown to user Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772 Test Plan: manual Reviewed By: siying Differential Revision: D21304604 Pulled By: pdillinger fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc 29 April 2020, 20:45:54 UTC
5272305 Derrick Pallas 29 April 2020, 20:06:27 UTC Fix FilterBench when RTTI=0 (#6732) Summary: The dynamic_cast in the filter benchmark causes release mode to fail due to no-rtti. Replace with static_cast_with_check. Signed-off-by: Derrick Pallas <derrick@pallas.us> Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732 Reviewed By: ltamasi Differential Revision: D21304260 Pulled By: pdillinger fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c 29 April 2020, 20:09:23 UTC
8086e5e Peter Dillinger 29 April 2020, 04:35:24 UTC Fix LITE build (#6770) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770 Test Plan: make LITE=1 check Reviewed By: ajkr Differential Revision: D21296261 Pulled By: pdillinger fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f 29 April 2020, 04:37:20 UTC
335ea73 anand76 29 April 2020, 03:03:28 UTC Fix a valgrind failure due to DBBasicTestMultiGetDeadline (#6756) Summary: Fix a valgrind failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756 Test Plan: valgrind_test Reviewed By: pdillinger Differential Revision: D21284660 Pulled By: anand1976 fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80 29 April 2020, 03:06:06 UTC
618bf63 mrambacher 29 April 2020, 01:02:11 UTC Add Functions to OptionTypeInfo (#6422) Summary: Added functions for parsing, serializing, and comparing elements to OptionTypeInfo. These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map. Using these functions, every type can be handled via the map rather than special cased. By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422 Test Plan: pass make check Reviewed By: siying Differential Revision: D21269005 Pulled By: zhichao-cao fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077 29 April 2020, 01:04:26 UTC
b810e62 Peter Dillinger 28 April 2020, 22:23:33 UTC Clarifying comments in db.h (#6768) Summary: And fix a confusingly worded log message Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768 Reviewed By: anand1976 Differential Revision: D21284527 Pulled By: pdillinger fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164 28 April 2020, 22:26:03 UTC
bae6f58 Peter Dillinger 28 April 2020, 21:46:13 UTC Basic MultiGet support for partitioned filters (#6757) Summary: In MultiGet, access each applicable filter partition only once per batch, rather than for each applicable key. Also, * Fix Bloom stats for MultiGet * Fix/refactor MultiGetContext::Range::KeysLeft, including * Add efficient BitsSetToOne implementation * Assert that MultiGetContext::Range does not go beyond shift range Performance test: Generate db: $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true ... Before (middle performing run of three; note some missing Bloom stats): $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 26.403 micros/op 597517 ops/sec; (548427 of 671968 found) rocksdb.block.cache.filter.hit COUNT : 83443275 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 7931450 rocksdb.number.multiget.get COUNT : 385984 rocksdb.number.multiget.keys.read COUNT : 12351488 rocksdb.number.multiget.bytes.read COUNT : 793145000 rocksdb.number.multiget.keys.found COUNT : 7931450 After (middle performing run of three): $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 21.024 micros/op 752963 ops/sec; (705188 of 863968 found) rocksdb.block.cache.filter.hit COUNT : 49856682 rocksdb.bloom.filter.useful COUNT : 45684579 rocksdb.bloom.filter.full.positive COUNT : 10395458 rocksdb.bloom.filter.full.true.positive COUNT : 9908456 rocksdb.number.multiget.get COUNT : 481984 rocksdb.number.multiget.keys.read COUNT : 15423488 rocksdb.number.multiget.bytes.read COUNT : 990845600 rocksdb.number.multiget.keys.found COUNT : 9908456 So that's about 25% higher throughput even for random keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757 Test Plan: unit test included Reviewed By: anand1976 Differential Revision: D21243256 Pulled By: pdillinger fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f 28 April 2020, 21:49:34 UTC
a7f0b27 Peter Dillinger 28 April 2020, 19:27:23 UTC HISTORY.md update for bzip upgrade (#6767) Summary: See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767 Reviewed By: riversand963 Differential Revision: D21283307 Pulled By: pdillinger fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8 28 April 2020, 19:29:31 UTC
4574d75 Peter Dillinger 28 April 2020, 15:24:15 UTC Update HISTORY.md for block cache redundant adds (#6764) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6764 Reviewed By: ltamasi Differential Revision: D21267108 Pulled By: pdillinger fbshipit-source-id: a3dfe2dbe4e8f6309a53eb72903ef58d52308f97 28 April 2020, 15:26:43 UTC
d4398e0 Yanqin Jin 28 April 2020, 05:47:44 UTC Fix timestamp support for MultiGet (#6748) Summary: 1. Avoid nullptr dereference when passing timestamp to KeyContext creation. 2. Construct LookupKey correctly with timestamp when creating MultiGetContext. 3. Compare without timestamp when sorting KeyContexts. Fixes https://github.com/facebook/rocksdb/issues/6745 Test plan (dev server): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6748 Reviewed By: pdillinger Differential Revision: D21258691 Pulled By: riversand963 fbshipit-source-id: 44e65b759c18b9986947783edf03be4f890bb004 28 April 2020, 05:49:56 UTC
4cd859e Cheng Chang 27 April 2020, 23:51:42 UTC Fix build under LITE (#6758) Summary: GetSupportedCompressions needs to be defined under LITE. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6758 Test Plan: build under LITE Reviewed By: zhichao-cao Differential Revision: D21247937 Pulled By: cheng-chang fbshipit-source-id: 880e59d3e107cdd736d16427a68c5641d1318fb4 27 April 2020, 23:55:14 UTC
bea91d5 Levi Tamasi 27 April 2020, 23:42:16 UTC Destroy any ColumnFamilyHandles in BlobDB::Open upon error (#6763) Summary: If an error happens during BlobDBImpl::Open after the base DB has been opened, we need to destroy the `ColumnFamilyHandle`s returned by `DB::Open` to prevent an assertion in `ColumnFamilySet`'s destructor from being hit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6763 Test Plan: Ran `make check` and tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21262643 Pulled By: ltamasi fbshipit-source-id: 60ebc7ab19be66cf37fbe5f6d8957d58470f3d3b 27 April 2020, 23:45:13 UTC
cc8d16e Albert Hse-Lin Chen 27 April 2020, 21:39:54 UTC Fixed minor typo in comment for MergeOperator::FullMergeV2() (#6759) Summary: Fixed minor typo in comment for FullMergeV2(). Last operand up to snapshot should be +4 instead of +3. Signed-off-by: Albert Hse-Lin Chen <hselin@kalista.io> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6759 Reviewed By: cheng-chang Differential Revision: D21260295 Pulled By: zhichao-cao fbshipit-source-id: cc942306f246c8606538feb30bfdf6df9fb6c54e 27 April 2020, 21:44:47 UTC
249eff0 Peter Dillinger 27 April 2020, 20:18:18 UTC Stats for redundant insertions into block cache (#6681) Summary: Since read threads do not coordinate on loading data into block cache, two threads between Lookup and Insert can end up loading and inserting the same data. This is particularly concerning with cache_index_and_filter_blocks since those are hot and more likely to be race targets if ejected from (or not pre-populated in) the cache. Particularly with moves toward disaggregated / network storage, the cost of redundant retrieval might be high, and we should at least have some hard statistics from which we can estimate impact. Example with full filter thrashing "cliff": $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 ... $ ./db_bench --db=/tmp/rocksdbtest-172704/dbbench --use_existing_db --benchmarks=readrandom,stats --num=200000 --cache_index_and_filter_blocks --cache_size=$((130 * 1024 * 1024)) --bloom_bits=10 --threads=16 -statistics 2>&1 | egrep '^rocksdb.block.cache.(.*add|.*redundant)' | grep -v compress | sort rocksdb.block.cache.add COUNT : 14181 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 476 rocksdb.block.cache.data.add COUNT : 12749 rocksdb.block.cache.data.add.redundant COUNT : 18 rocksdb.block.cache.filter.add COUNT : 1003 rocksdb.block.cache.filter.add.redundant COUNT : 217 rocksdb.block.cache.index.add COUNT : 429 rocksdb.block.cache.index.add.redundant COUNT : 241 $ ./db_bench --db=/tmp/rocksdbtest-172704/dbbench --use_existing_db --benchmarks=readrandom,stats --num=200000 --cache_index_and_filter_blocks --cache_size=$((120 * 1024 * 1024)) --bloom_bits=10 --threads=16 -statistics 2>&1 | egrep '^rocksdb.block.cache.(.*add|.*redundant)' | grep -v compress | sort rocksdb.block.cache.add COUNT : 1182223 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 302728 rocksdb.block.cache.data.add COUNT : 31425 rocksdb.block.cache.data.add.redundant COUNT : 12 rocksdb.block.cache.filter.add COUNT : 795455 rocksdb.block.cache.filter.add.redundant COUNT : 130238 rocksdb.block.cache.index.add COUNT : 355343 rocksdb.block.cache.index.add.redundant COUNT : 172478 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6681 Test Plan: Some manual testing (above) and unit test covering key metrics is included Reviewed By: ltamasi Differential Revision: D21134113 Pulled By: pdillinger fbshipit-source-id: c11497b5f00f4ffdfe919823904e52d0a1a91d87 27 April 2020, 20:20:27 UTC
75b13ea Akanksha Mahajan 27 April 2020, 19:33:49 UTC Allow sst_dump to check size of different compression levels and report time (#6634) Summary: Summary : 1. Add two arguments --compression_level_from and --compression_level_to to check the compression size with different compression level in the given range. Users must specify one compression type else it will error out. Both from and to levels must also be specified together. 2. Display the time taken to compress each file with different compressions by default. Test Plan : make -j64 check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6634 Test Plan: make -j64 check Reviewed By: anand1976 Differential Revision: D20810282 Pulled By: akankshamahajan15 fbshipit-source-id: ac9098d3c079a1fad098f6678dbedb4d888a791b 27 April 2020, 19:36:16 UTC
791e571 Peter Dillinger 27 April 2020, 17:45:49 UTC Understand common build variables passed as make variables (#6740) Summary: Some common build variables like USE_CLANG and COMPILE_WITH_UBSAN did not work if specified as make variables, as in `make USE_CLANG=1 check` etc. rather than (in theory less hygienic) `USE_CLANG=1 make check`. This patches Makefile to export some commonly used ones to build_detect_platform so that they work. (I'm skeptical of a broad `export` in Makefile because it's hard to predict how random make variables might affect various invoked tools.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6740 Test Plan: manual / CI Reviewed By: siying Differential Revision: D21229011 Pulled By: pdillinger fbshipit-source-id: b00c69b23eb2a13105bc8d860ce2d1e61ac5a355 27 April 2020, 17:48:49 UTC
3b2f271 Yanqin Jin 27 April 2020, 00:33:50 UTC Update buckifier to unblock future internal release (#6726) Summary: Some recent PRs added new source files or modified TARGETS file manually. During next internal release, executing the following command will revert the manual changes. Update buckifier so that the following command ``` python buckfier/buckify_rocksdb.py ``` does not change TARGETS file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6726 Test Plan: ``` python buckifier/buckify_rocksdb.py ``` Reviewed By: siying Differential Revision: D21098930 Pulled By: riversand963 fbshipit-source-id: e884f507fefef88163363c9097a460c98f1ed850 27 April 2020, 00:35:37 UTC
0a77617 Cheng Chang 25 April 2020, 06:58:13 UTC Disable O_DIRECT in stress test when db directory does not support direct IO (#6727) Summary: In crash test, the db directory might be set to /dev/shm or /tmp, in certain environments such as internal testing infrastructure, neither of these directories support direct IO, so direct IO is never enabled in crash test. This PR sets up SyncPoints in direct IO related code paths to disable O_DIRECT flag in calls to `open`, so the direct IO code paths will be executed, all direct IO related assertions will be checked, but no real direct IO request will be issued to the file system. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6727 Test Plan: export CRASH_TEST_EXT_ARGS="--use_direct_reads=1 --mmap_read=0" make -j24 crash_test Reviewed By: zhichao-cao Differential Revision: D21139250 Pulled By: cheng-chang fbshipit-source-id: db9adfe78d91aa4759835b1af91c5db7b27b62ee 25 April 2020, 07:01:03 UTC
40497a8 Cheng Chang 24 April 2020, 22:30:12 UTC Reduce memory copies when fetching and uncompressing blocks from SST files (#6689) Summary: In https://github.com/facebook/rocksdb/pull/6455, we modified the interface of `RandomAccessFileReader::Read` to be able to get rid of memcpy in direct IO mode. This PR applies the new interface to `BlockFetcher` when reading blocks from SST files in direct IO mode. Without this PR, in direct IO mode, when fetching and uncompressing compressed blocks, `BlockFetcher` will first copy the raw compressed block into `BlockFetcher::compressed_buf_` or `BlockFetcher::stack_buf_` inside `RandomAccessFileReader::Read` depending on the block size. then during uncompressing, it will copy the uncompressed block into `BlockFetcher::heap_buf_`. In this PR, we get rid of the first memcpy and directly uncompress the block from `direct_io_buf_` to `heap_buf_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6689 Test Plan: A new unit test `block_fetcher_test` is added. Reviewed By: anand1976 Differential Revision: D21006729 Pulled By: cheng-chang fbshipit-source-id: 2370b92c24075692423b81277415feb2aed5d980 24 April 2020, 22:32:56 UTC
1758f76 Cheng Chang 24 April 2020, 22:11:42 UTC Fix unused variable of r in release mode (#6750) Summary: In release mode, asserts are not compiled, so `r` is not used, causing compiler warnings. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6750 Test Plan: make check under release mode Reviewed By: anand1976 Differential Revision: D21220365 Pulled By: cheng-chang fbshipit-source-id: fd4afa9843d54af68c4da8660ec61549803e1167 24 April 2020, 22:14:13 UTC
9e7b7e2 anand76 24 April 2020, 20:03:08 UTC Silence false alarms in db_stress fault injection (#6741) Summary: False alarms are caused by codepaths that intentionally swallow IO errors. Tests: make crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6741 Reviewed By: ltamasi Differential Revision: D21181138 Pulled By: anand1976 fbshipit-source-id: 5ccfbc68eb192033488de6269e59c00f2c65ce00 24 April 2020, 20:06:12 UTC
e04f3bc Yanqin Jin 23 April 2020, 23:18:28 UTC Update CURRENT file after best-efforts recovery (#6746) Summary: After a successful recovery, the CURRENT file should be updated to point to the valid MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6746 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21189876 Pulled By: riversand963 fbshipit-source-id: 7537b49988c5c425ebe9505a5cc260de351ad79b 23 April 2020, 23:21:09 UTC
51bdfae Cheng Chang 23 April 2020, 22:17:00 UTC Check alignment of MultiRead requests in direct IO mode (#6739) Summary: Add assertions to check direct IO's alignment requirements in MultiRead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6739 Test Plan: make check Reviewed By: siying Differential Revision: D21143825 Pulled By: cheng-chang fbshipit-source-id: 26f1623b062a1851080771128feac0669a61f5e9 23 April 2020, 22:19:31 UTC
bc51e33 Levi Tamasi 23 April 2020, 20:41:32 UTC Make sure (Shared)BlobFileMetaData are owned by shared_ptrs (#6749) Summary: The patch makes a couple of small cleanups to `SharedBlobFileMetaData` and `BlobFileMetaData`: * It makes the constructors private and introduces factory methods to ensure these objects are always owned by `shared_ptr`s. Note that `SharedBlobFileMetaData` has an additional factory that takes a deleter object; we can utilize this to e.g. notify `VersionSet` when a blob file becomes obsolete (which is exactly when `SharedBlobFileMetaData` is destroyed). * It disables move operations explicitly instead of relying on them being suppressed because of a user-declared destructor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6749 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21206947 Pulled By: ltamasi fbshipit-source-id: 9094c14cc335b3e226f883e5a0df4f87a5cdeb95 23 April 2020, 20:44:29 UTC
ae77880 Ibrahim Jarif 23 April 2020, 19:26:56 UTC Fix some typos in code comments (#6733) Summary: This PR fixes some typos in code comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6733 Reviewed By: siying Differential Revision: D21209037 Pulled By: zhichao-cao fbshipit-source-id: d9274611fab1f5e992998c8c4117b8078c4cbc69 23 April 2020, 19:28:49 UTC
4cbc19d mrambacher 22 April 2020, 00:35:28 UTC Add a ConfigOptions for use in comparing objects and converting to/from strings (#6389) Summary: The methods in convenience.h are used to compare/convert objects to/from strings. There is a mishmash of parameters in use here with more needed in the future. This PR replaces those parameters with a single structure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6389 Reviewed By: siying Differential Revision: D21163707 Pulled By: zhichao-cao fbshipit-source-id: f807b4cc7e2b0af3871536b69546b2604dfa81bd 22 April 2020, 00:38:17 UTC
c1ccd6b anand76 21 April 2020, 21:49:50 UTC Implement deadline support for MultiGet (#6710) Summary: Initial implementation of ReadOptions.deadline for MultiGet. If the request takes longer than the deadline, the keys not yet found will be returned with Status::TimedOut(). This implementation enforces the deadline in DBImpl, which is fairly high level. Its best effort and may not check the deadline after every key lookup, but may do so after a batch of keys. In subsequent stages, we will extend this to passing a timeout down to the FileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6710 Test Plan: Add new unit tests Reviewed By: riversand963 Differential Revision: D21149158 Pulled By: anand1976 fbshipit-source-id: 9f44eecffeb40873f5034ed59a66d21f9f88879e 21 April 2020, 21:51:51 UTC
6ee66cf Tomas Kolda 21 April 2020, 20:14:03 UTC Prevents Table Cache to open same files more times (#6707) Summary: In highly concurrent requests table cache opens same file more times which lowers purpose of max_open_files. Fixes (https://github.com/facebook/rocksdb/issues/6699) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6707 Reviewed By: ltamasi Differential Revision: D21044965 fbshipit-source-id: f6e91d90b60dad86e518b5147021da42460ee1d2 21 April 2020, 20:16:31 UTC
f9155a3 Andrew Kryczka 20 April 2020, 23:30:34 UTC Prevent uninitialized load in `IndexBlockIter` (#6736) Summary: When index block is empty or an error happens while reading it, `Invalidate()` is called rather than `Initialize()`. So `Seek()` must not refer to member variables that are only initialized in `Initialize()` until it is sure `Initialize()` has been called. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6736 Reviewed By: siying Differential Revision: D21139641 Pulled By: ajkr fbshipit-source-id: 71c58cc1adbd795dc3729dd5023bf7df1515ff32 20 April 2020, 23:32:43 UTC
03a1d95 Akanksha Mahajan 20 April 2020, 23:17:25 UTC Set max_background_flushes dynamically (#6701) Summary: 1. Add changes so that max_background_flushes can be set dynamically. 2. Add a testcase DBOptionsTest.SetBackgroundFlushThreads which set the max_background_flushes dynamically using SetDBOptions. TestPlan: 1. make -j64 check 2. Using new testcase DBOptionsTest.SetBackgroundFlushThreads Pull Request resolved: https://github.com/facebook/rocksdb/pull/6701 Reviewed By: ajkr Differential Revision: D21028010 Pulled By: akankshamahajan15 fbshipit-source-id: 5f949e4a8fd3c32537b637947b7ee09a69cfc7c1 20 April 2020, 23:19:02 UTC
31da5e3 Peter Dillinger 20 April 2020, 20:21:34 UTC C++20 compatibility (#6697) Summary: Based on https://github.com/facebook/rocksdb/issues/6648 (CLA Signed), but heavily modified / extended: * Implicit capture of this via [=] deprecated in C++20, and [=,this] not standard before C++20 -> now using explicit capture lists * Implicit copy operator deprecated in gcc 9 -> add explicit '= default' definition * std::random_shuffle deprecated in C++17 and removed in C++20 -> migrated to a replacement in RocksDB random.h API * Add the ability to build with different std version though -DCMAKE_CXX_STANDARD=11/14/17/20 on the cmake command line * Minimal rebuild flag of MSVC is deprecated and is forbidden with /std:c++latest (C++20) * Added MSVC 2019 C++11 & MSVC 2019 C++20 in AppVeyor * Added GCC 9 C++11 & GCC9 C++20 in Travis Pull Request resolved: https://github.com/facebook/rocksdb/pull/6697 Test Plan: make check and CI Reviewed By: cheng-chang Differential Revision: D21020318 Pulled By: pdillinger fbshipit-source-id: 12311be5dbd8675a0e2c817f7ec50fa11c18ab91 20 April 2020, 20:24:25 UTC
fe206f4 sdong 20 April 2020, 19:55:13 UTC crash_test to cover index_type kBinarySearchWithFirstKey (#6721) Summary: Recently index_type kBinarySearchWithFirstKey is improved so that the API guarantee is exactly the same as other types and it is ready for wide production. We should cover it in crash tst. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6721 Test Plan: Run crash_test Reviewed By: anand1976 Differential Revision: D21099781 fbshipit-source-id: fda91eba831d9eacbb140c703e9768bb1701f935 20 April 2020, 19:57:15 UTC
45d2b4e Peter Dillinger 20 April 2020, 18:37:36 UTC Fix tabs and lint-ignores (#6734) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6734 Reviewed By: cheng-chang Differential Revision: D21134556 Pulled By: pdillinger fbshipit-source-id: 3636cc1d1333137b70031f8277458781c21631fb 20 April 2020, 18:39:31 UTC
243852e Yanqin Jin 17 April 2020, 21:36:51 UTC Add IsDirectory() to Env and FS (#6711) Summary: IsDirectory() is a common API to check whether a path is a regular file or directory. POSIX: call stat() and use S_ISDIR(st_mode) Windows: PathIsDirectoryA() and PathIsDirectoryW() HDFS: FileSystem.IsDirectory() Java: File.IsDirectory() ... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6711 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21053520 Pulled By: riversand963 fbshipit-source-id: 680aadfd8ce982b63689190cf31b3145d5a89e27 17 April 2020, 21:39:18 UTC
63d82e5 sdong 17 April 2020, 17:57:17 UTC crash_test to cover small max_open_files (#6719) Summary: RocksDB behavior is different while max_open_files is small or large. Add the coverage to small max_open_files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6719 Test Plan: Run crash_test Reviewed By: pdillinger Differential Revision: D21081021 fbshipit-source-id: e3e211761a9bd25d93d19a61c1f7b62d48cf5e3c 17 April 2020, 18:00:07 UTC
9e6f3ef Nicolas Pépin-Perreault 16 April 2020, 22:49:52 UTC Add RocksIterator::Refresh (#6573) Summary: This PR exposes the `Iterator::Refresh` method to the Java API by adding it on the `RocksIteratorInterface` interface. There are three concrete implementations: `RocksIterator`, `SstFileReaderIterator`, and `WBWIRocksIterator`. For the first two cases, the JNI side simply delegates to the underlying `Iterator::Refresh` method; in the last case, as it doesn't share an ancestor, and per the discussion in https://github.com/facebook/rocksdb/issues/3465, a `Status::NotSupported` exception is thrown. As the last PR had no activity in a while, I'm opening a new one - I'm completely fine with merging the previous PR if it gets completed before this is reviewed. Let me know if there's anything missing or anything else I can do :+1: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6573 Reviewed By: cheng-chang Differential Revision: D20604666 Pulled By: pdillinger fbshipit-source-id: 4de17df1180c3b87b76cfdd77b674b81fc0563f7 16 April 2020, 22:55:26 UTC
9ca49bd Adam Retter 16 April 2020, 22:48:24 UTC Keep building RocksJava on all architectures (#6583) Summary: Adding solid support for multiple architectures was initially triggered by RocksJava users. As such I would like to keep the CI for RocksJava on all architectures, to ensure we don't break backwards compatibility. pdillinger okay let's see how long it takes to complete Travis-CI with this one... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6583 Reviewed By: cheng-chang Differential Revision: D21036718 Pulled By: pdillinger fbshipit-source-id: 97afe0db2e4c575cc0284fdc1d4cc45d5deb2272 16 April 2020, 22:50:45 UTC
5fef0ff Adam Retter 16 April 2020, 22:32:34 UTC Update RocksJava static version of bzip2 (#6714) Summary: Updates the version of bzip2 used for RocksJava static builds. Please, can we also get this cherry-picked to: 1. 6.7.fb 2. 6.8.fb 3. 6.9.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714 Reviewed By: cheng-chang Differential Revision: D21067233 Pulled By: pdillinger fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e 16 April 2020, 22:35:24 UTC
6717ada Andrew Kryczka 16 April 2020, 20:13:25 UTC Fix CF import with overlapping SST files (#6663) Summary: Invariant checking should use internal key comparator rather than `sstableKeyCompare()`. The latter was intended for checking whether a compaction input file's neighboring files need to be included in the same compaction. Using it for invariant checking was leading to false positives for files with overlapping endpoints. Fixes https://github.com/facebook/rocksdb/issues/6647. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6663 Test Plan: regression test Reviewed By: zhichao-cao Differential Revision: D20910466 Pulled By: ajkr fbshipit-source-id: f0b70dad7c4096fce635cab7a36f16e14f74ae3f 16 April 2020, 20:16:06 UTC
73523ba sdong 16 April 2020, 19:09:18 UTC crash_test to cover options.avoid_flush_during_recovery (#6712) Summary: Options.avoid_flush_during_recovery is uncovered in crash_test. Add the coverage with a chance of 1/8, as it is a less frequently used options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6712 Test Plan: Run crash_test and see the option can be used or not used by chance. Reviewed By: ltamasi Differential Revision: D21056566 fbshipit-source-id: c3b1521517cfc204786e6ef8c6acd7fffda64793 16 April 2020, 19:11:45 UTC
5801af4 Yueh-Hsuan Chiang 16 April 2020, 18:10:53 UTC Add env_fault_injection argument to db_stress (#6687) Summary: Add env_fault_injection argument to db_stress. When enabled, FaultInjectionTestEnv will be used instead. Currently this option does not support running with other env setting. This will allow us to later manually produce error when running db_crashtest. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6687 Test Plan: make db_stress -j32 ./db_stress --env_fault_injection ./db_stress --env_fault_injection --hdfs // expect error message Reviewed By: ajkr Differential Revision: D21014683 Pulled By: yhchiang fbshipit-source-id: 0724aeac37efd57adb72a37defe6dbd3bfa8106a 16 April 2020, 18:13:44 UTC
2767972 Cheng Chang 16 April 2020, 18:00:00 UTC Fix warning when O_CLOEXEC is not defined (#6695) Summary: Compilation fails on systems that do not support O_CLOEXEC. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6695 Test Plan: compile without O_CLOEXEC support Reviewed By: anand1976 Differential Revision: D21011850 Pulled By: cheng-chang fbshipit-source-id: f1bf1cce2aa65c7d10b5a9613e941db30e928347 16 April 2020, 18:02:50 UTC
e45673d Mike Kolupaev 16 April 2020, 00:37:23 UTC Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) Summary: Context: Index type `kBinarySearchWithFirstKey` added the ability for sst file iterator to sometimes report a key from index without reading the corresponding data block. This is useful when sst blocks are cut at some meaningful boundaries (e.g. one block per key prefix), and many seeks land between blocks (e.g. for each prefix, the ranges of keys in different sst files are nearly disjoint, so a typical seek needs to read a data block from only one file even if all files have the prefix). But this added a new error condition, which rocksdb code was really not equipped to deal with: `InternalIterator::value()` may fail with an IO error or Status::Incomplete, but it's just a method returning a Slice, with no way to report error instead. Before this PR, this type of error wasn't handled at all (an empty slice was returned), and kBinarySearchWithFirstKey implementation was considered a prototype. Now that we (LogDevice) have experimented with kBinarySearchWithFirstKey for a while and confirmed that it's really useful, this PR is adding the missing error handling. It's a pretty inconvenient situation implementation-wise. The error needs to be reported from InternalIterator when trying to access value. But there are ~700 call sites of `InternalIterator::value()`, most of which either can't hit the error condition (because the iterator is reading from memtable or from index or something) or wouldn't benefit from the deferred loading of the value (e.g. compaction iterator that reads all values anyway). Adding error handling to all these call sites would needlessly bloat the code. So instead I made the deferred value loading optional: only the call sites that may use deferred loading have to call the new method `PrepareValue()` before calling `value()`. The feature is enabled with a new bool argument `allow_unprepared_value` to a bunch of methods that create iterators (it wouldn't make sense to put it in ReadOptions because it's completely internal to iterators, with virtually no user-visible effect). Lmk if you have better ideas. Note that the deferred value loading only happens for *internal* iterators. The user-visible iterator (DBIter) always prepares the value before returning from Seek/Next/etc. We could go further and add an API to defer that value loading too, but that's most likely not useful for LogDevice, so it doesn't seem worth the complexity for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6621 Test Plan: make -j5 check . Will also deploy to some logdevice test clusters and look at stats. Reviewed By: siying Differential Revision: D20786930 Pulled By: al13n321 fbshipit-source-id: 6da77d918bad3780522e918f17f4d5513d3e99ee 16 April 2020, 00:40:44 UTC
610a09c anand76 15 April 2020, 19:09:33 UTC Remove a printf from db_stress that's not useful info (#6705) Summary: This was causing db_crashtest.py to wrongly assume an error by parsing the output. Hopefully this will stabilize the crash tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6705 Test Plan: make blackbox_crash_test Reviewed By: ltamasi Differential Revision: D21043335 Pulled By: anand1976 fbshipit-source-id: 5cddd112b124d4e2ebd11724a17d4ef0f50c1cf8 15 April 2020, 19:13:35 UTC
165560f sdong 15 April 2020, 18:23:45 UTC Two Improvements to tools/check_format_compatible.sh (#6702) Summary: Improve it in two ways: 1. tools/check_format_compatible.sh is not friendly to run outside FB environment. remove the hard-coded http proxy setting. Instead, move it to Legocastle configuration 2. Always disable warning as error, so that older build is more likely to pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6702 Test Plan: Run the test and make sure at least it doesn't break. Reviewed By: riversand963 Differential Revision: D21033329 fbshipit-source-id: 88b4ec1ec49547b772790050a165466bdc4a62a0 15 April 2020, 18:28:11 UTC
234e2ed anand76 14 April 2020, 18:04:39 UTC Fix a couple of bugs in db_stress fault injection (#6700) Summary: 1. Fix a memory leak in FaultInjectionTestFS in the stack trace related code 2. Check status of all MultiGet keys before deciding whether an error was swallowed, instead of assuming an ok status for any key means an undetected error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6700 Test Plan: Run db_stress with asan and fault injection Reviewed By: cheng-chang Differential Revision: D21021498 Pulled By: anand1976 fbshipit-source-id: 489191efd1ab0fa834923a1e1d57253a7a315465 14 April 2020, 18:06:55 UTC
9ae8058 Cheng Chang 14 April 2020, 17:52:37 UTC Suppress file deletion error message in FaultInjectionTestEnv (#6696) Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. This is a follow up PR for https://github.com/facebook/rocksdb/pull/6694. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6696 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D21021875 Pulled By: cheng-chang fbshipit-source-id: 11e3f536df16941a89949ebcd2147cd8dfa3fbe0 14 April 2020, 17:55:10 UTC
3d6d7bc anand76 14 April 2020, 02:56:02 UTC Log CompactOnDeletionCollectorFactory parameters on DB open (#6686) Summary: Log it in the info log to help in troubleshooting. It is logged as follows - ``` 2020/04/10-10:51:39.886662 7ffff7fef340 Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 100 Deletion trigger = 90); ``` Tests: make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6686 Reviewed By: ltamasi Differential Revision: D21002442 Pulled By: anand1976 fbshipit-source-id: 7adf0dbae7f1febcb00ce61fea5097118ede5c6a 14 April 2020, 02:58:04 UTC
38dfa40 Zhichao Cao 14 April 2020, 02:11:19 UTC Add NewFileChecksumGenCrc32cFactory to file checksum (#6688) Summary: Add NewFileChecksumGenCrc32cFactory to file checksum public interface such that applications can use the build in crc32 checksum factory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6688 Test Plan: pass make asan_check Reviewed By: riversand963 Differential Revision: D21006859 Pulled By: zhichao-cao fbshipit-source-id: ea8a45196a8b77c310728ab05f6cc0f49f3baef0 14 April 2020, 02:13:41 UTC
41563b6 Ziyue Yang 13 April 2020, 23:22:19 UTC Fix data racing of BlockBasedTableBuilder::ParallelCompressionRep::first_block (#6640) Summary: BlockBasedTableBuilder::ParallelCompressionRep::first_block can be read in Flush() and written in BGWorkWriteRawBlock() concurrently. This commit fixes the issue by reading first_block out before pushing the block to compression and write. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6640 Test Plan: Run all tests concurrently with TSAN. Reviewed By: cheng-chang Differential Revision: D20851370 fbshipit-source-id: 6f039222e8319d31e15f1b45e05c106527253f72 13 April 2020, 23:24:57 UTC
d9cad3a anand76 13 April 2020, 22:16:14 UTC Suppress file deletion error message in FaultInjectionTestFS (#6694) Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6694 Reviewed By: siying Differential Revision: D20998531 Pulled By: anand1976 fbshipit-source-id: 89cb54a5f5bb664ae6d239c37559f10e14c5ea07 13 April 2020, 22:18:38 UTC
9eca6d6 Andrew Kryczka 13 April 2020, 18:15:20 UTC fix comparison count for format_version=3 indexes (#6650) Summary: In index blocks since `format_version=3`, user keys are written rather than internal keys. When reading such blocks, the comparator is obtained via `InternalKeyComparator::user_comparator()`. That function must not return an unwrapped result as the wrapper class provides accounting logic to populate `PerfContext::user_key_comparison_count`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6650 Test Plan: ran db_bench and verified `PerfContext::user_key_comparison_count` became larger. Reviewed By: cheng-chang Differential Revision: D20866325 Pulled By: ajkr fbshipit-source-id: ad755d46bda31157dacc5b66e532279f19ad538c 13 April 2020, 18:18:37 UTC
79c838e anand76 13 April 2020, 17:58:43 UTC Fix a few bugs in db_stress fault injection (#6693) Summary: Fix the following issues - 1. Output parsing error in db_crashtest.py 2. Memory leak on exit 3. False alarm on filter block read error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6693 Test Plan: asan_crash Reviewed By: cheng-chang Differential Revision: D20990399 Pulled By: anand1976 fbshipit-source-id: 178ee0dd7c69a4bc5db698379db0dedb29281699 13 April 2020, 18:01:03 UTC
eeb3cf3 Yanqin Jin 12 April 2020, 05:02:17 UTC Fix release build (#6690) Summary: Fix release build caused by variable defined but unused. Test plan (devserver) ``` make release ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6690 Reviewed By: cheng-chang Differential Revision: D20980571 Pulled By: riversand963 fbshipit-source-id: c3f3b13f81dce4bdb19876dc2e710d5902ff8a02 12 April 2020, 05:04:04 UTC
5c19a44 anand76 11 April 2020, 00:18:56 UTC Fault injection in db_stress (#6538) Summary: This PR implements a fault injection mechanism for injecting errors in reads in db_stress. The FaultInjectionTestFS is used for this purpose. A thread local structure is used to track the errors, so that each db_stress thread can independently enable/disable error injection and verify observed errors against expected errors. This is initially enabled only for Get and MultiGet, but can be extended to iterator as well once its proven stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6538 Test Plan: crash_test make check Reviewed By: riversand963 Differential Revision: D20714347 Pulled By: anand1976 fbshipit-source-id: d7598321d4a2d72bda0ced57411a337a91d87dc7 11 April 2020, 00:21:26 UTC
0c05624 Yanqin Jin 10 April 2020, 23:03:33 UTC Compaction with timestamp: input boundaries (#6645) Summary: Towards making compaction logic compatible with user timestamp. When computing boundaries and overlapping ranges for inputs of compaction, We need to compare SSTs by user key without timestamp. Test plan (devserver): ``` make check ``` Several individual tests: ``` ./version_set_test --gtest_filter=VersionStorageInfoTimestampTest.GetOverlappingInputs ./db_with_timestamp_compaction_test ./db_with_timestamp_basic_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6645 Reviewed By: ltamasi Differential Revision: D20960012 Pulled By: riversand963 fbshipit-source-id: ad377fa9eb481bf7a8a3e1824aaade48cdc653a4 10 April 2020, 23:05:49 UTC
a0faff1 Akanksha Mahajan 10 April 2020, 22:27:47 UTC Report kFilesMarkedForCompaction for delete triggered compactions (#6680) Summary: Summary : Set manual_compaction false in case of DeleteTriggeredCompaction object so that kFilesMarkedForComapaction can be reported. Added a DeletionTriggeredUniversalCompactionMarking test case for Deletion Triggered compaction in case of Universal Compaction. Test Plan : make check -j64 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6680 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D20945946 Pulled By: akankshamahajan15 fbshipit-source-id: af84e417bd7127652aaae9143c560d1ab3815d25 10 April 2020, 22:30:38 UTC
d600e5b anand76 10 April 2020, 18:45:47 UTC Fix a Centos build failure reported in #6651 (#6656) Summary: Fixes issue https://github.com/facebook/rocksdb/issues/6651 Tests: make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6656 Reviewed By: cheng-chang Differential Revision: D20879084 Pulled By: anand1976 fbshipit-source-id: c2cc508ca2716fcf80dcf9d2ba31c32d211f941e 10 April 2020, 18:47:46 UTC
1be3be5 sdong 10 April 2020, 18:30:49 UTC Auto-Format two recent diffs and add HISTORY.md (#6685) Summary: Two recent diffs can be autoformatted. Also add HISTORY.md entry for https://github.com/facebook/rocksdb/pull/6214 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6685 Test Plan: Run all existing tests Reviewed By: cheng-chang Differential Revision: D20965780 fbshipit-source-id: 195b08d7849513d42fe14073112cd19fdda6af95 10 April 2020, 18:32:44 UTC
f08630b Andrew Kryczka 10 April 2020, 17:43:38 UTC explicitly mark backup interfaces non-extensible (#6654) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6654 Reviewed By: cheng-chang Differential Revision: D20878094 Pulled By: ajkr fbshipit-source-id: 94d2561bdb6ffb7fe3773ca07d475337600a5b57 10 April 2020, 17:51:09 UTC
c8c739a Connor1996 10 April 2020, 17:43:20 UTC Fix sst_dump not able to open ingested file (#6673) Summary: When investigating https://github.com/facebook/rocksdb/issues/6666, we encounter an error for sst_dump to dump an ingested SST file with global seqno. ``` Corruption: An external sst file with version 2 have global seqno property with value ��/, while largest seqno in the file is 0） ``` Same as https://github.com/facebook/rocksdb/pull/5097, it is due to SstFileReader don't know the largest seqno of a file, it will fail this check when it open a file with global seqno. https://github.com/facebook/rocksdb/blob/ca89ac2ba997dfa0e135bd75d4ccf6f5774a7eff/table/block_based_table_reader.cc#L730 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6673 Test Plan: run it manually Reviewed By: cheng-chang Differential Revision: D20937546 Pulled By: ajkr fbshipit-source-id: c3fd04d60916a738533ee1885f3ea844669a9479 10 April 2020, 17:47:46 UTC
9e89ffb Huisheng Liu 10 April 2020, 16:49:38 UTC make iterator return versions between timestamp bounds (#6544) Summary: (Based on Yanqin's idea) Add a new field in readoptions as lower timestamp bound for iterator. When the parameter is not supplied (nullptr), the iterator returns the latest visible version of a record. When it is supplied, the existing timestamp field is the upper bound. Together the two serves as a bounded time window. The iterator returns all versions of a record falling in the window. SeekRandom perf test (10 minutes) on the same development machine ram drive with the same DB data shows no regression (within marge of error). The test is adapted from https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks. base line (commit e860f8840): seekrandom : 7.836 micros/op 4082449 ops/sec; (0 of 73481999 found) This PR: seekrandom : 7.764 micros/op 4120935 ops/sec; (0 of 71303999 found) db_bench --db=r:\rocksdb.github --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --cache_size=2147483648 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=r:\rocksdb.github\WAL_LOG --sync=0 --verify_checksum=1 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --duration=600 --benchmarks=seekrandom --use_existing_db=1 --num=25000000 --threads=32 --allow_concurrent_memtable_write=0 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6544 Reviewed By: ltamasi Differential Revision: D20844069 Pulled By: riversand963 fbshipit-source-id: d97f2bf38a323c8c6a68db213b2d3c694b1c1f74 10 April 2020, 16:51:58 UTC
66a95f0 Luca Giacchino 10 April 2020, 03:45:17 UTC Provide an allocator for new memory type to be used with RocksDB block cache (#6214) Summary: New memory technologies are being developed by various hardware vendors (Intel DCPMM is one such technology currently available). These new memory types require different libraries for allocation and management (such as PMDK and memkind). The high capacities available make it possible to provision large caches (up to several TBs in size), beyond what is achievable with DRAM. The new allocator provided in this PR uses the memkind library to allocate memory on different media. **Performance** We tested the new allocator using db_bench. - For each test, we vary the size of the block cache (relative to the size of the uncompressed data in the database). - The database is filled sequentially. Throughput is then measured with a readrandom benchmark. - We use a uniform distribution as a worst-case scenario. The plot shows throughput (ops/s) relative to a configuration with no block cache and default allocator. For all tests, p99 latency is below 500 us. ![image](https://user-images.githubusercontent.com/26400080/71108594-42479100-2178-11ea-8231-8a775bbc92db.png) **Changes** - Add MemkindKmemAllocator - Add --use_cache_memkind_kmem_allocator db_bench option (to create an LRU block cache with the new allocator) - Add detection of memkind library with KMEM DAX support - Add test for MemkindKmemAllocator **Minimum Requirements** - kernel 5.3.12 - ndctl v67 - https://github.com/pmem/ndctl - memkind v1.10.0 - https://github.com/memkind/memkind **Memory Configuration** The allocator uses the MEMKIND_DAX_KMEM memory kind. Follow the instructions on[ memkind’s GitHub page](https://github.com/memkind/memkind) to set up NVDIMM memory accordingly. Note on memory allocation with NVDIMM memory exposed as system memory. - The MemkindKmemAllocator will only allocate from NVDIMM memory (using memkind_malloc with MEMKIND_DAX_KMEM kind). - The default allocator is not restricted to RAM by default. Based on NUMA node latency, the kernel should allocate from local RAM preferentially, but it’s a kernel decision. numactl --preferred/--membind can be used to allocate preferentially/exclusively from the local RAM node. **Usage** When creating an LRU cache, pass a MemkindKmemAllocator object as argument. For example (replace capacity with the desired value in bytes): ``` #include "rocksdb/cache.h" #include "memory/memkind_kmem_allocator.h" NewLRUCache( capacity /*size_t*/, 6 /*cache_numshardbits*/, false /*strict_capacity_limit*/, false /*cache_high_pri_pool_ratio*/, std::make_shared<MemkindKmemAllocator>()); ``` Refer to [RocksDB’s block cache documentation](https://github.com/facebook/rocksdb/wiki/Block-Cache) to assign the LRU cache as block cache for a database. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6214 Reviewed By: cheng-chang Differential Revision: D19292435 fbshipit-source-id: 7202f47b769e7722b539c86c2ffd669f64d7b4e1 10 April 2020, 03:47:23 UTC
9d6974d Peter Dillinger 09 April 2020, 23:40:49 UTC Temporarily disable ppc64le unit tests in PRs (#6682) Summary: Until Travis gets its act together (https://github.com/facebook/rocksdb/issues/6653) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6682 Test Plan: CI Reviewed By: riversand963 Differential Revision: D20948865 Pulled By: pdillinger fbshipit-source-id: 215de523c91a83d2a159f466b853e700c925ba4f 09 April 2020, 23:42:44 UTC
e860f88 sdong 09 April 2020, 18:20:33 UTC Fix memory corruption caused by new test in options_settable_test (#6676) Summary: https://github.com/facebook/rocksdb/pull/6668 added some new test code but it has a risk of memory corruption. Fix it Pull Request resolved: https://github.com/facebook/rocksdb/pull/6676 Test Plan: Run the test under ASAN and see it passes. Reviewed By: ajkr Differential Revision: D20937108 fbshipit-source-id: 22cc96bb02030df0a37a02e67a2cc37ca31ba22d 09 April 2020, 18:23:32 UTC
6e6f807 Cheng Chang 09 April 2020, 18:17:39 UTC Add two more optimization improvements to HISTORY (#6679) Summary: Although these optimizations are not user facing, still feel it's valuable to call out in HISTORY. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6679 Test Plan: no need Reviewed By: zhichao-cao Differential Revision: D20945916 Pulled By: cheng-chang fbshipit-source-id: f3e790c07f3bcc4a8a74246c4fa232800ddd4438 09 April 2020, 18:19:51 UTC
eb287c7 Yi Wu 09 April 2020, 04:19:49 UTC Fix wrong key being read on ingested file with global seqno and delta encoding (#6669) Summary: On reading an ingested SST file, `DataBlockIter` will replace seqno encoded in a key with global seqno. However, if the original seqno was part of the prefix used for the next key, the global seqno is by mistake used as part of the prefix to construct the next key, causing wrong result being returned. Although at this point it is only software error while data in the file is not corrupted, the issue can further cause compaction output out of order and corrupted result when the ingested SST participated in compaction. Fixing the issue by save the actual seqno and restore it before the key being used as prefix to construct next key. The unit test is by Little-Wallace from https://github.com/facebook/rocksdb/issues/6666. Fixing https://github.com/facebook/rocksdb/issues/6666. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6669 Test Plan: New unit test Signed-off-by: Yi Wu <yiwu@pingcap.com> Reviewed By: cheng-chang Differential Revision: D20931808 Pulled By: ajkr fbshipit-source-id: f01959c35d6a493954dca981663766c7a5a9e8ab 09 April 2020, 04:22:15 UTC
31759a7 Cheng Chang 09 April 2020, 04:17:42 UTC Fix result slice's address for direct io read (#6672) Summary: When aligned_buf is provided, the result slice's starting address should take offset advance into account. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6672 Test Plan: make check Reviewed By: anand1976 Differential Revision: D20934198 Pulled By: cheng-chang fbshipit-source-id: c3475c9c132b92c50d8c7c399fca2e9e76870803 09 April 2020, 04:20:31 UTC
83fc90b Yi Wu 09 April 2020, 03:15:41 UTC Fix info log source file display length (#5824) Summary: Source code path in info log is not truncated to the correct length. Fixing it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5824 Test Plan: Build and run db_bench. Before: ``` 2019/09/18-21:32:34.631181 7fdd42df6700 [_impl/db_impl_write.cc:1654] [default] New memtable created with log file: https://github.com/facebook/rocksdb/issues/9. Immutable memtables: 0. ``` After: ``` 2019/09/18-21:36:09.226532 7f141b5f6700 [/db_impl/db_impl_write.cc:1654] [default] New memtable created with log file: https://github.com/facebook/rocksdb/issues/9. Immutable memtables: 0. ``` Reviewed By: cheng-chang Differential Revision: D17511851 fbshipit-source-id: b2f92c85ce78726c27b7e0e736657fe2f983513e 09 April 2020, 03:18:08 UTC
94f90ac sdong 08 April 2020, 21:37:01 UTC compression related options are not copied back from MutableCFOptions… (#6668) Summary: … to CFOptions https://github.com/facebook/rocksdb/pull/6615 made several compression related options dynamically changeable. They are moved to MutableCFOptions. However, they are not copied back to ColumnFamilyOptions, so the changed values are not written to option files and for some other uses. Fix it by copying them back. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6668 Test Plan: Add a unit test to make sure that when a MutableCFOptions is converted to CFOptions and back to MutableCFOptions, they stay the same. This test would fail without the fix. Reviewed By: ajkr Differential Revision: D20923999 fbshipit-source-id: c3bccd6923b00d677764e2269bed6a95ad7ed780 08 April 2020, 21:40:46 UTC
a91613d CaixinGong 08 April 2020, 21:23:41 UTC Fix readrandom return NotFound after fillrandom in db_bench (#6665) Summary: This commit is fixing a bug that readrandom test returns many NotFound in db_bench from Version 6.2. Pull Request resolved: https://github.com/facebook/rocksdb/issues/6664 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6665 Reviewed By: cheng-chang Differential Revision: D20911298 Pulled By: ajkr fbshipit-source-id: c2658d4dbb35798ccbf67dff6e64923fb731ef81 08 April 2020, 21:27:12 UTC
back to top