c384c08 | Ziyue Yang | 12 May 2020, 16:25:21 UTC | Add tests for compression failure in BlockBasedTableBuilder (#6709) Summary: Currently there is no check for whether BlockBasedTableBuilder will expose compression error status if compression fails during the table building. This commit adds fake faulting compressors and a unit test to test such cases. This check finds 5 bugs, and this commit also fixes them: 1. Not handling compression failure well in BlockBasedTableBuilder::BGWorkWriteRawBlock. 2. verify_compression failing in BlockBasedTableBuilder when used with ZSTD. 3. Wrongly passing the same reference of block contents to BlockBasedTableBuilder::CompressAndVerifyBlock in parallel compression. 4. Wrongly setting block_rep->first_key_in_next_block to nullptr in BlockBasedTableBuilder::EnterUnbuffered when there are still incoming data blocks. 5. Not maintaining variables for compression ratio estimation and first_block in BlockBasedTableBuilder::EnterUnbuffered. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6709 Reviewed By: ajkr Differential Revision: D21236254 fbshipit-source-id: 101f6e62b2bac2b7be72be198adf93cd32a1ff46 | 12 May 2020, 16:27:35 UTC |
3f21807 | yetingsky | 11 May 2020, 21:56:36 UTC | fix typo (#6831) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6831 Reviewed By: cheng-chang Differential Revision: D21499149 Pulled By: zhichao-cao fbshipit-source-id: 2cb76cbf7086677d8cad5c828019e008062f0052 | 11 May 2020, 21:58:25 UTC |
2e93247 | Sagar Vemuri | 11 May 2020, 20:28:16 UTC | Disable a few timer tests (#6833) Summary: Disable `TimerTest.SingleScheduleRepeatedlyTest` and `TimerTest.MultipleScheduleRepeatedlyTest`. This is to help people to not hit any hangs (https://github.com/facebook/rocksdb/issues/6698) during their development process while I investigate further; I could not reproduce the issue on my dev machine yet. Note that timer is not being utilized anywhere yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6833 Test Plan: ``` svemuri@devbig187 ~/rocksdb (timer-disable-test) $ TEST_TMPDIR=/dev/shm ./timer_test [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from TimerTest [ RUN ] TimerTest.SingleScheduleOnceTest [ OK ] TimerTest.SingleScheduleOnceTest (1 ms) [ RUN ] TimerTest.MultipleScheduleOnceTest [ OK ] TimerTest.MultipleScheduleOnceTest (0 ms) [----------] 2 tests from TimerTest (1 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (1 ms total) [ PASSED ] 2 tests. YOU HAVE 2 DISABLED TESTS ``` Reviewed By: pdillinger Differential Revision: D21502474 Pulled By: sagar0 fbshipit-source-id: ac67caee2011fd14ffb2476a8914a6286a4f9abe | 11 May 2020, 20:30:00 UTC |
f0e8731 | Tongliang Liao | 11 May 2020, 17:26:18 UTC | Use GFlags/Snappy config in CMake, with fallback for legacy approach. (#6771) Summary: Related to some discussion in https://github.com/facebook/rocksdb/issues/6179 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6771 Reviewed By: zhichao-cao Differential Revision: D21340117 fbshipit-source-id: a1af0ba4865bb13c8c817851d6f6c4056191b3fe | 11 May 2020, 17:28:49 UTC |
3a1c29d | Derrick Pallas | 09 May 2020, 03:48:17 UTC | Add missing my_pid to fprintf in multi_process_example (#6731) Summary: Signed-off-by: Derrick Pallas <derrick@pallas.us> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6731 Reviewed By: siying Differential Revision: D21137005 Pulled By: riversand963 fbshipit-source-id: a7182e1bec225bc110971f40e2d0e6c3a671c061 | 09 May 2020, 03:49:33 UTC |
a50ea71 | sdong | 08 May 2020, 21:12:18 UTC | Improve ldb consistency checks (#6802) Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0 | 08 May 2020, 21:17:47 UTC |
d9cd335 | sdong | 08 May 2020, 21:11:03 UTC | Suppress UBSAN warning in CRC32 ARM (#6827) Summary: UBSAN shows following warning: util/crc32c_arm64.cc:111:11: runtime error: load of misaligned address 0x00001afcda86 for type 'const uint64_t', which requires 8 byte alignment 0x00001afcda86: note: pointer points here cc c1 2d 00 01 81 40 24 30 66 39 66 30 37 30 63 2d 32 36 63 34 2d 34 62 61 61 2d 38 35 33 31 2d ^ Suppress it just as what we do in x86 CRC. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6827 Test Plan: Run the same UBSAN and see it to pass now. Reviewed By: ltamasi Differential Revision: D21471838 fbshipit-source-id: 02943dd39a7030d2b03e5d894dcb23ed72b6c9c3 | 08 May 2020, 21:14:00 UTC |
e72e216 | Yanqin Jin | 08 May 2020, 19:59:02 UTC | Fix a few bugs in best-efforts recovery (#6824) Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2 | 08 May 2020, 20:01:42 UTC |
1c84660 | Andrew Kryczka | 08 May 2020, 19:36:49 UTC | prototype status check enforcement (#6798) Summary: Tried making Status object enforce that it is checked in some way. In cases it is not checked, `PermitUncheckedError()` must be called explicitly. Added a way to run tests (`ASSERT_STATUS_CHECKED=1 make -j48 check`) on a whitelist. The effort appears significant to get each test to pass with this assertion, so I only fixed up enough to get one test (`options_test`) working and added it to the whitelist. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6798 Reviewed By: pdillinger Differential Revision: D21377404 Pulled By: ajkr fbshipit-source-id: 73236f9c8df38f01cf24ecac4a6d1661b72d077e | 08 May 2020, 19:40:43 UTC |
1282589 | sdong | 08 May 2020, 03:50:14 UTC | Disable "compression_parallel_threads" in crash test for now (#6816) Summary: "compressio_parallel_threads" caused several test failure tests. To keep crash test clean, disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6816 Test Plan: "make crash_test" to make sure the python script doesn't break Reviewed By: zhichao-cao Differential Revision: D21462112 fbshipit-source-id: 9eecc764800da82cd19665dc8b167eacead3310b | 08 May 2020, 03:52:29 UTC |
9426523 | anand76 | 08 May 2020, 00:30:41 UTC | Fix race due to delete triggered compaction in Universal compaction mode (#6799) Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a | 08 May 2020, 00:32:17 UTC |
3730b05 | Andrew Kryczka | 07 May 2020, 23:37:47 UTC | Fixup HISTORY.md for e9ba4ba "validate range tombstone covers positiv… (#6825) Summary: …e range" Moved it from the wrong section (6.10) to the right section (Unreleased). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6825 Reviewed By: zhichao-cao Differential Revision: D21464577 Pulled By: ajkr fbshipit-source-id: a836b4ab10be2464182826f9411c9c424c933b70 | 07 May 2020, 23:40:17 UTC |
f286fb3 | anand76 | 07 May 2020, 22:52:58 UTC | Include options.h in table.h (#6823) Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d | 07 May 2020, 22:55:29 UTC |
b27a144 | Peter Dillinger | 07 May 2020, 22:39:49 UTC | Fix false NotFound from batched MultiGet with kHashSearch (#6821) Summary: The error is assigning KeyContext::s to NotFound status in a table reader for a "not found in this table" case, which skips searching in later tables, like only a delete should. (The hash search index iterator is the only one that can return status NotFound even if Valid() == false.) This was detected by intermittent failure in MultiThreadedDBTest.MultiThreaded/5, a kHashSearch configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6821 Test Plan: modified existing unit test to reproduce problem Reviewed By: anand1976 Differential Revision: D21450469 Pulled By: pdillinger fbshipit-source-id: 7478003684d637dbd491cdac81468041a791be2c | 07 May 2020, 22:41:37 UTC |
e9ba4ba | Andrew Kryczka | 07 May 2020, 18:53:32 UTC | validate range tombstone covers positive range (#6788) Summary: We found some files containing nothing but negative range tombstones, and unsurprisingly their metadata specified a negative range, which made things crash. Time to add a bit of user input validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6788 Reviewed By: zhichao-cao Differential Revision: D21343719 Pulled By: ajkr fbshipit-source-id: f1c16e4c3e9fa150958c8c866176632a3206fb74 | 07 May 2020, 18:55:30 UTC |
ac3ae1d | Levi Tamasi | 07 May 2020, 16:29:21 UTC | Find/purge obsolete blob files (#6807) Summary: The patch extends `FindObsoleteFiles` and `PurgeObsoleteFiles` with support for blob files. The behavior is analogous to SST files: obsolete blob files are put on the "candidates for deletion" list, while live (and pending) files are preserved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6807 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21406249 Pulled By: ltamasi fbshipit-source-id: 1948f71c31927564b61e8af394f50ca3964880d9 | 07 May 2020, 16:32:51 UTC |
1f20df2 | Andrew Kryczka | 07 May 2020, 01:06:04 UTC | cover single level universal in crash test (#6818) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6818 Test Plan: fast whitebox test and verify there are some single-level universal and some multi-level universal runs. ``` $ python ./tools/db_crashtest.py whitebox --simple -max_key=1000000 -value_size_mult=33 -write_buffer_size=524288 -target_file_size_base=524288 -max_bytes_for_level_base=2097152 --duration=120 --interval=10 --ops_per_thread=1000 --random_kill_odd=887 ``` Reviewed By: riversand963 Differential Revision: D21432138 Pulled By: ajkr fbshipit-source-id: 2fc5ba9f3dfa49bb11e81da7dd00a17b476e64d7 | 07 May 2020, 01:08:09 UTC |
c1e1185 | anand76 | 06 May 2020, 23:40:32 UTC | Update release version to 6.10 (#6797) Summary: Update HISTORY.md and version.h to 6.10. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6797 Reviewed By: zhichao-cao Differential Revision: D21371390 Pulled By: anand1976 fbshipit-source-id: 6017bca24fc5d12076d1ddaec7783c9b85712d42 | 06 May 2020, 23:42:37 UTC |
06c3b85 | Levi Tamasi | 06 May 2020, 20:58:25 UTC | Disallow using the base DB's storage directory as blob_dir in BlobDB (#6810) Summary: https://github.com/facebook/rocksdb/pull/6807 extends the logic that identifies and purges obsolete files to blob files handled by RocksDB itself. In order to prevent that from interfering with the current BlobDB code, we need to make sure that `BlobDBOptions::blob_dir` is different from the storage directories used by the base DB. (Note: this is true by default.) The patch adds a check that explicitly disallows this configuration and returns `Status::NotSupported` from `BlobDB::Open` in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6810 Test Plan: Tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21412676 Pulled By: ltamasi fbshipit-source-id: 6630cc7481e48c8bf55d59423b25f14d52ffe681 | 06 May 2020, 21:00:46 UTC |
53f8447 | Neha Deodhar | 06 May 2020, 17:25:58 UTC | Add YugabyteDB to USERS (#6786) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6786 Reviewed By: zhichao-cao Differential Revision: D21407762 Pulled By: pdillinger fbshipit-source-id: 62f74409cc72c6b9ce4107aa515955cfc5eec12a | 06 May 2020, 17:28:29 UTC |
c21c459 | sdong | 06 May 2020, 01:29:50 UTC | Slightly expand converage to file consistency check failure (#6800) Summary: Current DBCompactionTest.ConsistencyFailTest checks DB fails after L0 inconsitency is found. Add slightly more coverage by introducing DBCompactionTest.ConsistencyFailTest2 which checks non-L0 files too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6800 Test Plan: Run the new test. Reviewed By: riversand963 Differential Revision: D21384806 fbshipit-source-id: 36db7b657eed42115283fe2f6afa4c3a31a3b510 | 06 May 2020, 01:31:53 UTC |
394f2bb | mrambacher | 05 May 2020, 22:02:04 UTC | Add OptionTypeInfo::Enum and related methods (#6423) Summary: Add methods and constructors for handling enums to the OptionTypeInfo. This change allows enums to be converted/compared without adding a special "type" to the OptionType. This change addresses a couple of issues: - It allows new enumerated types to be added to the options without editing the OptionType base class (and related methods) - It standardizes the procedure for adding enumerated types to the options, reducing potential mistakes - It moves the enum maps to the location where they are used, allowing them to be static file members rather than global values - It reduces the number of types and cases that need to be handled in the various OptionType methods Pull Request resolved: https://github.com/facebook/rocksdb/pull/6423 Reviewed By: siying Differential Revision: D21408713 Pulled By: zhichao-cao fbshipit-source-id: fc492af285d011822578b95d186a0fce25d35626 | 05 May 2020, 22:04:04 UTC |
a96461d | Andrew Kryczka | 05 May 2020, 21:52:18 UTC | fix swallowed error for file deletion consistency check (#6809) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6809 Reviewed By: pdillinger Differential Revision: D21411971 Pulled By: ajkr fbshipit-source-id: 900b6b0370b76e9a3e5e03f968e2ac1bbaab73b8 | 05 May 2020, 21:54:21 UTC |
2f1700c | Peter Dillinger | 05 May 2020, 20:09:12 UTC | Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) Summary: This very old test code bug was causing a new valgrind failure in MultiGetDeadlineExceeded Also fix hang in MultiGetDeadlineExceeded by unifying with some logic from another test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6803 Test Plan: run that unit test under valgrind, make check Reviewed By: siying Differential Revision: D21388470 Pulled By: pdillinger fbshipit-source-id: 0ce99d6d5eb8cd3195b17406892c8c5cff5fa5dd | 05 May 2020, 20:11:29 UTC |
91bc013 | Cheng Chang | 05 May 2020, 18:07:52 UTC | Refactor level compaction picker (#6804) Summary: 1. refactor out PickFileToCompact to remove duplicated logic. 2. remove redundant checks of `start_level_inputs_.empty()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6804 Test Plan: make check Reviewed By: siying Differential Revision: D21390053 Pulled By: cheng-chang fbshipit-source-id: 185d5987a08bfdaf63f0f245310c6da69878d415 | 05 May 2020, 18:09:29 UTC |
5584595 | Yanqin Jin | 05 May 2020, 17:44:12 UTC | Do not swallow error returned from SaveTo() (#6801) Summary: With consistency check enabled, VersionBuilder::SaveTo() may return error once corruption is detected while building versions. We should handle these errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6801 Test Plan: make check Reviewed By: siying Differential Revision: D21385045 Pulled By: riversand963 fbshipit-source-id: 98f6424e2a4699b62befa21e9fe00e70a771118e | 05 May 2020, 17:46:20 UTC |
5a61e78 | Yanqin Jin | 05 May 2020, 00:43:33 UTC | Fix db_stress when GetLiveFiles() flushes dropped CF (#6805) Summary: Current impl. of db_stress will abort verification and report failure if GetLiveFiles() causes a dropped column family to be flushed. This is not desired. To fix, this PR makes the following change: In GetLiveFiles, if flush is triggered and returns Status::IsColumnFamilyDropped(), then set status to Status::OK(). This is OK because dropped column families will be skipped during the rest of this function, and valid column families will have their live files returned to caller. Test plan (dev server): make check ./db_stress -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 ./db_stress -disable_wal=1 -reopen=0 -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6805 Reviewed By: ltamasi Differential Revision: D21390044 Pulled By: riversand963 fbshipit-source-id: de67846b95a4f1b88aa0a30c3d70c43cc68625b9 | 05 May 2020, 00:45:49 UTC |
a00ddf1 | Levi Tamasi | 04 May 2020, 22:05:34 UTC | Expose the set of live blob files from Version/VersionSet (#6785) Summary: The patch adds logic that returns the set of live blob files from `Version::AddLiveFiles` and `VersionSet::AddLiveFiles` (in addition to live table files), and also cleans up the code a bit, for example, by exposing only the numbers of table files as opposed to the earlier `FileDescriptor`s that no clients used. Moreover, the patch extends the `GetLiveFiles` API so that it also exposes blob files in the current version. Similarly to https://github.com/facebook/rocksdb/pull/6755, this is a building block for identifying and purging obsolete blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6785 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21336210 Pulled By: ltamasi fbshipit-source-id: fc1aede8a49eacd03caafbc5f6f9ce43b6270821 | 04 May 2020, 22:08:13 UTC |
680c416 | sdong | 04 May 2020, 21:15:55 UTC | Avoid Swallowing Some File Consistency Checking Bugs (#6793) Summary: We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump. More places are not fixed and need follow-up. Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793 Test Plan: Add a unit test to cover the reopen case. Reviewed By: riversand963 Differential Revision: D21366525 fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded | 04 May 2020, 21:18:11 UTC |
d9e170d | Mian Qin | 04 May 2020, 17:53:15 UTC | Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper (#6795) Summary: Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper using db_bench. Details changes as follows. 1, add a separate random mode in MixGraph to produce all_random workload. 2, fix power inverse function for generating prefix_dist workload. 3, make sure key_offset in prefix mode is always unsigned. note: Need to carefully choose key_dist_a/b to avoid aliasing. Power inverse function range should be close to overall key space. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6795 Reviewed By: akankshamahajan15 Differential Revision: D21371095 Pulled By: zhichao-cao fbshipit-source-id: 80744381e242392c8c7cf8ac3d68fe67fe876048 | 04 May 2020, 17:55:14 UTC |
211088d | Cheng Chang | 02 May 2020, 00:35:41 UTC | Remove redundant update of txn_state_ in transaction Prepare (#6778) Summary: When expiration is set in a pessimistic transaction, `txn_state_` is already updated to `AWAITING_PREPARE` in the `if (expiration_time_ > 0)` block, there is no need to update the state in `if (can_prepare)` block again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6778 Test Plan: make check Reviewed By: lth Differential Revision: D21335319 Pulled By: cheng-chang fbshipit-source-id: 251d634cc7d1a0e86e673a59f0bda8584da5a35f | 02 May 2020, 00:37:33 UTC |
c8643ed | Zhichao Cao | 01 May 2020, 07:01:21 UTC | Fix multiple CF replay failure in db_bench replay (#6787) Summary: The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787 Test Plan: pass trace replay test. Reviewed By: yhchiang Differential Revision: D21339941 Pulled By: zhichao-cao fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce | 01 May 2020, 07:03:38 UTC |
6acbbbf | Yanqin Jin | 01 May 2020, 02:20:55 UTC | Add Github Action for some basic sanity test of PR (#6761) Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787 | 01 May 2020, 02:22:45 UTC |
6504ae0 | sdong | 30 April 2020, 23:59:16 UTC | Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) Summary: The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782 Test Plan: Run all existing tests. Reviewed By: zhichao-cao Differential Revision: D21338614 fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685 | 01 May 2020, 00:01:17 UTC |
ef0c3ed | Cheng Chang | 30 April 2020, 23:21:05 UTC | Make users explicitly be aware of prepare before commit (#6775) Summary: In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare. This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775 Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare Reviewed By: lth Differential Revision: D21313270 Pulled By: cheng-chang fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251 | 30 April 2020, 23:24:20 UTC |
079e50d | sdong | 30 April 2020, 22:34:43 UTC | Disallow BlockBasedTableBuilder to set status from non-OK (#6776) Summary: There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future. The solution is to only make it possible to set the status code if the status code to set is not OK. Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776 Test Plan: Run all existing test. Reviewed By: pdillinger Differential Revision: D21314382 fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375 | 30 April 2020, 22:37:03 UTC |
6277e28 | sdong | 30 April 2020, 22:19:31 UTC | Flag CompressionOptions::parallel_threads to be experimental (#6781) Summary: The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781 Reviewed By: pdillinger Differential Revision: D21330678 fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13 | 30 April 2020, 22:22:06 UTC |
ab13d43 | anand76 | 30 April 2020, 21:48:51 UTC | Pass a timeout to FileSystem for random reads (#6751) Summary: Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded. For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR. Tests: Update existing unit tests to verify the correct timeout value is being passed Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751 Reviewed By: riversand963 Differential Revision: D21285631 Pulled By: anand1976 fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567 | 30 April 2020, 21:50:39 UTC |
eecd8fb | Peter Dillinger | 30 April 2020, 19:08:34 UTC | Fix assertion that can fail on sst corruption (#6780) Summary: An assertion that a char == a CompressionType (unsigned char) originally cast from a char can fail if the original value is negative, due to numeric promotion. The assertion should pass even if the value is invalid CompressionType, because the callee UncompressBlockContentsForCompressionType checks for that and reports status appropriately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780 Test Plan: Temporarily change kZSTD = 0x88 and see tests fail. Make this change (in addition), and tests pass. Reviewed By: siying Differential Revision: D21328498 Pulled By: pdillinger fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92 | 30 April 2020, 19:11:00 UTC |
fe238e5 | Levi Tamasi | 30 April 2020, 18:23:32 UTC | Keep track of obsolete blob files in VersionSet (#6755) Summary: The patch adds logic to keep track of obsolete blob files. A blob file becomes obsolete when the last `shared_ptr` that points to the corresponding `SharedBlobFileMetaData` object goes away, which, in turn, happens when the last `Version` that contains the blob file is destroyed. No longer needed blob files are added to the obsolete list in `VersionSet` using a custom deleter to avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`. Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored in `JobContext`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21233155 Pulled By: ltamasi fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a | 30 April 2020, 18:25:51 UTC |
cf34246 | Adam Retter | 30 April 2020, 17:56:33 UTC | Add Slack forum to README (#6773) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773 Reviewed By: siying Differential Revision: D21310229 Pulled By: pdillinger fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf | 30 April 2020, 18:00:28 UTC |
e619a20 | Ziyue Yang | 30 April 2020, 17:46:54 UTC | Add an option for parallel compression in for db_stress (#6722) Summary: This commit adds an `compression_parallel_threads` option in db_stress. It also fixes the naming of parallel compression option in db_bench to keep it aligned with others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722 Reviewed By: pdillinger Differential Revision: D21091385 fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100 | 30 April 2020, 17:49:07 UTC |
8c69402 | Zhichao Cao | 30 April 2020, 15:38:18 UTC | Fix potential size_t overflow in import_column_family (#6762) Summary: The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762 Test Plan: make asan_check Reviewed By: pdillinger Differential Revision: D21323063 Pulled By: zhichao-cao fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a | 30 April 2020, 15:40:42 UTC |
b938e60 | anand76 | 30 April 2020, 02:25:56 UTC | Fix a couple of bugs in FaultInjectionTestFS (#6777) Summary: Fix the following cases that can cause false alarms in db_stress when read fault injection is enabled - 1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error. 2. Handle the case when the randomly generated string to overwrite the original block is identical to the original. Tests: Run db_stress w/ and wo/ direct IO and fault injection turned on Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777 Reviewed By: zhichao-cao Differential Revision: D21316734 Pulled By: anand1976 fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa | 30 April 2020, 02:28:29 UTC |
28fe8e4 | Peter Dillinger | 29 April 2020, 20:42:43 UTC | Fix bug in format-diff.sh (#6772) Summary: Nasty bug in which more/different changes would be applied than those shown to user Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772 Test Plan: manual Reviewed By: siying Differential Revision: D21304604 Pulled By: pdillinger fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc | 29 April 2020, 20:45:54 UTC |
5272305 | Derrick Pallas | 29 April 2020, 20:06:27 UTC | Fix FilterBench when RTTI=0 (#6732) Summary: The dynamic_cast in the filter benchmark causes release mode to fail due to no-rtti. Replace with static_cast_with_check. Signed-off-by: Derrick Pallas <derrick@pallas.us> Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732 Reviewed By: ltamasi Differential Revision: D21304260 Pulled By: pdillinger fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c | 29 April 2020, 20:09:23 UTC |
8086e5e | Peter Dillinger | 29 April 2020, 04:35:24 UTC | Fix LITE build (#6770) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770 Test Plan: make LITE=1 check Reviewed By: ajkr Differential Revision: D21296261 Pulled By: pdillinger fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f | 29 April 2020, 04:37:20 UTC |
335ea73 | anand76 | 29 April 2020, 03:03:28 UTC | Fix a valgrind failure due to DBBasicTestMultiGetDeadline (#6756) Summary: Fix a valgrind failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756 Test Plan: valgrind_test Reviewed By: pdillinger Differential Revision: D21284660 Pulled By: anand1976 fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80 | 29 April 2020, 03:06:06 UTC |
618bf63 | mrambacher | 29 April 2020, 01:02:11 UTC | Add Functions to OptionTypeInfo (#6422) Summary: Added functions for parsing, serializing, and comparing elements to OptionTypeInfo. These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map. Using these functions, every type can be handled via the map rather than special cased. By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422 Test Plan: pass make check Reviewed By: siying Differential Revision: D21269005 Pulled By: zhichao-cao fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077 | 29 April 2020, 01:04:26 UTC |
b810e62 | Peter Dillinger | 28 April 2020, 22:23:33 UTC | Clarifying comments in db.h (#6768) Summary: And fix a confusingly worded log message Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768 Reviewed By: anand1976 Differential Revision: D21284527 Pulled By: pdillinger fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164 | 28 April 2020, 22:26:03 UTC |
bae6f58 | Peter Dillinger | 28 April 2020, 21:46:13 UTC | Basic MultiGet support for partitioned filters (#6757) Summary: In MultiGet, access each applicable filter partition only once per batch, rather than for each applicable key. Also, * Fix Bloom stats for MultiGet * Fix/refactor MultiGetContext::Range::KeysLeft, including * Add efficient BitsSetToOne implementation * Assert that MultiGetContext::Range does not go beyond shift range Performance test: Generate db: $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true ... Before (middle performing run of three; note some missing Bloom stats): $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 26.403 micros/op 597517 ops/sec; (548427 of 671968 found) rocksdb.block.cache.filter.hit COUNT : 83443275 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 7931450 rocksdb.number.multiget.get COUNT : 385984 rocksdb.number.multiget.keys.read COUNT : 12351488 rocksdb.number.multiget.bytes.read COUNT : 793145000 rocksdb.number.multiget.keys.found COUNT : 7931450 After (middle performing run of three): $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 21.024 micros/op 752963 ops/sec; (705188 of 863968 found) rocksdb.block.cache.filter.hit COUNT : 49856682 rocksdb.bloom.filter.useful COUNT : 45684579 rocksdb.bloom.filter.full.positive COUNT : 10395458 rocksdb.bloom.filter.full.true.positive COUNT : 9908456 rocksdb.number.multiget.get COUNT : 481984 rocksdb.number.multiget.keys.read COUNT : 15423488 rocksdb.number.multiget.bytes.read COUNT : 990845600 rocksdb.number.multiget.keys.found COUNT : 9908456 So that's about 25% higher throughput even for random keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757 Test Plan: unit test included Reviewed By: anand1976 Differential Revision: D21243256 Pulled By: pdillinger fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f | 28 April 2020, 21:49:34 UTC |
a7f0b27 | Peter Dillinger | 28 April 2020, 19:27:23 UTC | HISTORY.md update for bzip upgrade (#6767) Summary: See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767 Reviewed By: riversand963 Differential Revision: D21283307 Pulled By: pdillinger fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8 | 28 April 2020, 19:29:31 UTC |
4574d75 | Peter Dillinger | 28 April 2020, 15:24:15 UTC | Update HISTORY.md for block cache redundant adds (#6764) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6764 Reviewed By: ltamasi Differential Revision: D21267108 Pulled By: pdillinger fbshipit-source-id: a3dfe2dbe4e8f6309a53eb72903ef58d52308f97 | 28 April 2020, 15:26:43 UTC |
d4398e0 | Yanqin Jin | 28 April 2020, 05:47:44 UTC | Fix timestamp support for MultiGet (#6748) Summary: 1. Avoid nullptr dereference when passing timestamp to KeyContext creation. 2. Construct LookupKey correctly with timestamp when creating MultiGetContext. 3. Compare without timestamp when sorting KeyContexts. Fixes https://github.com/facebook/rocksdb/issues/6745 Test plan (dev server): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6748 Reviewed By: pdillinger Differential Revision: D21258691 Pulled By: riversand963 fbshipit-source-id: 44e65b759c18b9986947783edf03be4f890bb004 | 28 April 2020, 05:49:56 UTC |
4cd859e | Cheng Chang | 27 April 2020, 23:51:42 UTC | Fix build under LITE (#6758) Summary: GetSupportedCompressions needs to be defined under LITE. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6758 Test Plan: build under LITE Reviewed By: zhichao-cao Differential Revision: D21247937 Pulled By: cheng-chang fbshipit-source-id: 880e59d3e107cdd736d16427a68c5641d1318fb4 | 27 April 2020, 23:55:14 UTC |
bea91d5 | Levi Tamasi | 27 April 2020, 23:42:16 UTC | Destroy any ColumnFamilyHandles in BlobDB::Open upon error (#6763) Summary: If an error happens during BlobDBImpl::Open after the base DB has been opened, we need to destroy the `ColumnFamilyHandle`s returned by `DB::Open` to prevent an assertion in `ColumnFamilySet`'s destructor from being hit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6763 Test Plan: Ran `make check` and tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21262643 Pulled By: ltamasi fbshipit-source-id: 60ebc7ab19be66cf37fbe5f6d8957d58470f3d3b | 27 April 2020, 23:45:13 UTC |
cc8d16e | Albert Hse-Lin Chen | 27 April 2020, 21:39:54 UTC | Fixed minor typo in comment for MergeOperator::FullMergeV2() (#6759) Summary: Fixed minor typo in comment for FullMergeV2(). Last operand up to snapshot should be +4 instead of +3. Signed-off-by: Albert Hse-Lin Chen <hselin@kalista.io> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6759 Reviewed By: cheng-chang Differential Revision: D21260295 Pulled By: zhichao-cao fbshipit-source-id: cc942306f246c8606538feb30bfdf6df9fb6c54e | 27 April 2020, 21:44:47 UTC |
249eff0 | Peter Dillinger | 27 April 2020, 20:18:18 UTC | Stats for redundant insertions into block cache (#6681) Summary: Since read threads do not coordinate on loading data into block cache, two threads between Lookup and Insert can end up loading and inserting the same data. This is particularly concerning with cache_index_and_filter_blocks since those are hot and more likely to be race targets if ejected from (or not pre-populated in) the cache. Particularly with moves toward disaggregated / network storage, the cost of redundant retrieval might be high, and we should at least have some hard statistics from which we can estimate impact. Example with full filter thrashing "cliff": $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 ... $ ./db_bench --db=/tmp/rocksdbtest-172704/dbbench --use_existing_db --benchmarks=readrandom,stats --num=200000 --cache_index_and_filter_blocks --cache_size=$((130 * 1024 * 1024)) --bloom_bits=10 --threads=16 -statistics 2>&1 | egrep '^rocksdb.block.cache.(.*add|.*redundant)' | grep -v compress | sort rocksdb.block.cache.add COUNT : 14181 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 476 rocksdb.block.cache.data.add COUNT : 12749 rocksdb.block.cache.data.add.redundant COUNT : 18 rocksdb.block.cache.filter.add COUNT : 1003 rocksdb.block.cache.filter.add.redundant COUNT : 217 rocksdb.block.cache.index.add COUNT : 429 rocksdb.block.cache.index.add.redundant COUNT : 241 $ ./db_bench --db=/tmp/rocksdbtest-172704/dbbench --use_existing_db --benchmarks=readrandom,stats --num=200000 --cache_index_and_filter_blocks --cache_size=$((120 * 1024 * 1024)) --bloom_bits=10 --threads=16 -statistics 2>&1 | egrep '^rocksdb.block.cache.(.*add|.*redundant)' | grep -v compress | sort rocksdb.block.cache.add COUNT : 1182223 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 302728 rocksdb.block.cache.data.add COUNT : 31425 rocksdb.block.cache.data.add.redundant COUNT : 12 rocksdb.block.cache.filter.add COUNT : 795455 rocksdb.block.cache.filter.add.redundant COUNT : 130238 rocksdb.block.cache.index.add COUNT : 355343 rocksdb.block.cache.index.add.redundant COUNT : 172478 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6681 Test Plan: Some manual testing (above) and unit test covering key metrics is included Reviewed By: ltamasi Differential Revision: D21134113 Pulled By: pdillinger fbshipit-source-id: c11497b5f00f4ffdfe919823904e52d0a1a91d87 | 27 April 2020, 20:20:27 UTC |
75b13ea | Akanksha Mahajan | 27 April 2020, 19:33:49 UTC | Allow sst_dump to check size of different compression levels and report time (#6634) Summary: Summary : 1. Add two arguments --compression_level_from and --compression_level_to to check the compression size with different compression level in the given range. Users must specify one compression type else it will error out. Both from and to levels must also be specified together. 2. Display the time taken to compress each file with different compressions by default. Test Plan : make -j64 check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6634 Test Plan: make -j64 check Reviewed By: anand1976 Differential Revision: D20810282 Pulled By: akankshamahajan15 fbshipit-source-id: ac9098d3c079a1fad098f6678dbedb4d888a791b | 27 April 2020, 19:36:16 UTC |
791e571 | Peter Dillinger | 27 April 2020, 17:45:49 UTC | Understand common build variables passed as make variables (#6740) Summary: Some common build variables like USE_CLANG and COMPILE_WITH_UBSAN did not work if specified as make variables, as in `make USE_CLANG=1 check` etc. rather than (in theory less hygienic) `USE_CLANG=1 make check`. This patches Makefile to export some commonly used ones to build_detect_platform so that they work. (I'm skeptical of a broad `export` in Makefile because it's hard to predict how random make variables might affect various invoked tools.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6740 Test Plan: manual / CI Reviewed By: siying Differential Revision: D21229011 Pulled By: pdillinger fbshipit-source-id: b00c69b23eb2a13105bc8d860ce2d1e61ac5a355 | 27 April 2020, 17:48:49 UTC |
3b2f271 | Yanqin Jin | 27 April 2020, 00:33:50 UTC | Update buckifier to unblock future internal release (#6726) Summary: Some recent PRs added new source files or modified TARGETS file manually. During next internal release, executing the following command will revert the manual changes. Update buckifier so that the following command ``` python buckfier/buckify_rocksdb.py ``` does not change TARGETS file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6726 Test Plan: ``` python buckifier/buckify_rocksdb.py ``` Reviewed By: siying Differential Revision: D21098930 Pulled By: riversand963 fbshipit-source-id: e884f507fefef88163363c9097a460c98f1ed850 | 27 April 2020, 00:35:37 UTC |
0a77617 | Cheng Chang | 25 April 2020, 06:58:13 UTC | Disable O_DIRECT in stress test when db directory does not support direct IO (#6727) Summary: In crash test, the db directory might be set to /dev/shm or /tmp, in certain environments such as internal testing infrastructure, neither of these directories support direct IO, so direct IO is never enabled in crash test. This PR sets up SyncPoints in direct IO related code paths to disable O_DIRECT flag in calls to `open`, so the direct IO code paths will be executed, all direct IO related assertions will be checked, but no real direct IO request will be issued to the file system. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6727 Test Plan: export CRASH_TEST_EXT_ARGS="--use_direct_reads=1 --mmap_read=0" make -j24 crash_test Reviewed By: zhichao-cao Differential Revision: D21139250 Pulled By: cheng-chang fbshipit-source-id: db9adfe78d91aa4759835b1af91c5db7b27b62ee | 25 April 2020, 07:01:03 UTC |
40497a8 | Cheng Chang | 24 April 2020, 22:30:12 UTC | Reduce memory copies when fetching and uncompressing blocks from SST files (#6689) Summary: In https://github.com/facebook/rocksdb/pull/6455, we modified the interface of `RandomAccessFileReader::Read` to be able to get rid of memcpy in direct IO mode. This PR applies the new interface to `BlockFetcher` when reading blocks from SST files in direct IO mode. Without this PR, in direct IO mode, when fetching and uncompressing compressed blocks, `BlockFetcher` will first copy the raw compressed block into `BlockFetcher::compressed_buf_` or `BlockFetcher::stack_buf_` inside `RandomAccessFileReader::Read` depending on the block size. then during uncompressing, it will copy the uncompressed block into `BlockFetcher::heap_buf_`. In this PR, we get rid of the first memcpy and directly uncompress the block from `direct_io_buf_` to `heap_buf_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6689 Test Plan: A new unit test `block_fetcher_test` is added. Reviewed By: anand1976 Differential Revision: D21006729 Pulled By: cheng-chang fbshipit-source-id: 2370b92c24075692423b81277415feb2aed5d980 | 24 April 2020, 22:32:56 UTC |
1758f76 | Cheng Chang | 24 April 2020, 22:11:42 UTC | Fix unused variable of r in release mode (#6750) Summary: In release mode, asserts are not compiled, so `r` is not used, causing compiler warnings. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6750 Test Plan: make check under release mode Reviewed By: anand1976 Differential Revision: D21220365 Pulled By: cheng-chang fbshipit-source-id: fd4afa9843d54af68c4da8660ec61549803e1167 | 24 April 2020, 22:14:13 UTC |
9e7b7e2 | anand76 | 24 April 2020, 20:03:08 UTC | Silence false alarms in db_stress fault injection (#6741) Summary: False alarms are caused by codepaths that intentionally swallow IO errors. Tests: make crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6741 Reviewed By: ltamasi Differential Revision: D21181138 Pulled By: anand1976 fbshipit-source-id: 5ccfbc68eb192033488de6269e59c00f2c65ce00 | 24 April 2020, 20:06:12 UTC |
e04f3bc | Yanqin Jin | 23 April 2020, 23:18:28 UTC | Update CURRENT file after best-efforts recovery (#6746) Summary: After a successful recovery, the CURRENT file should be updated to point to the valid MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6746 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21189876 Pulled By: riversand963 fbshipit-source-id: 7537b49988c5c425ebe9505a5cc260de351ad79b | 23 April 2020, 23:21:09 UTC |
51bdfae | Cheng Chang | 23 April 2020, 22:17:00 UTC | Check alignment of MultiRead requests in direct IO mode (#6739) Summary: Add assertions to check direct IO's alignment requirements in MultiRead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6739 Test Plan: make check Reviewed By: siying Differential Revision: D21143825 Pulled By: cheng-chang fbshipit-source-id: 26f1623b062a1851080771128feac0669a61f5e9 | 23 April 2020, 22:19:31 UTC |
bc51e33 | Levi Tamasi | 23 April 2020, 20:41:32 UTC | Make sure (Shared)BlobFileMetaData are owned by shared_ptrs (#6749) Summary: The patch makes a couple of small cleanups to `SharedBlobFileMetaData` and `BlobFileMetaData`: * It makes the constructors private and introduces factory methods to ensure these objects are always owned by `shared_ptr`s. Note that `SharedBlobFileMetaData` has an additional factory that takes a deleter object; we can utilize this to e.g. notify `VersionSet` when a blob file becomes obsolete (which is exactly when `SharedBlobFileMetaData` is destroyed). * It disables move operations explicitly instead of relying on them being suppressed because of a user-declared destructor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6749 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21206947 Pulled By: ltamasi fbshipit-source-id: 9094c14cc335b3e226f883e5a0df4f87a5cdeb95 | 23 April 2020, 20:44:29 UTC |
ae77880 | Ibrahim Jarif | 23 April 2020, 19:26:56 UTC | Fix some typos in code comments (#6733) Summary: This PR fixes some typos in code comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6733 Reviewed By: siying Differential Revision: D21209037 Pulled By: zhichao-cao fbshipit-source-id: d9274611fab1f5e992998c8c4117b8078c4cbc69 | 23 April 2020, 19:28:49 UTC |
4cbc19d | mrambacher | 22 April 2020, 00:35:28 UTC | Add a ConfigOptions for use in comparing objects and converting to/from strings (#6389) Summary: The methods in convenience.h are used to compare/convert objects to/from strings. There is a mishmash of parameters in use here with more needed in the future. This PR replaces those parameters with a single structure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6389 Reviewed By: siying Differential Revision: D21163707 Pulled By: zhichao-cao fbshipit-source-id: f807b4cc7e2b0af3871536b69546b2604dfa81bd | 22 April 2020, 00:38:17 UTC |
c1ccd6b | anand76 | 21 April 2020, 21:49:50 UTC | Implement deadline support for MultiGet (#6710) Summary: Initial implementation of ReadOptions.deadline for MultiGet. If the request takes longer than the deadline, the keys not yet found will be returned with Status::TimedOut(). This implementation enforces the deadline in DBImpl, which is fairly high level. Its best effort and may not check the deadline after every key lookup, but may do so after a batch of keys. In subsequent stages, we will extend this to passing a timeout down to the FileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6710 Test Plan: Add new unit tests Reviewed By: riversand963 Differential Revision: D21149158 Pulled By: anand1976 fbshipit-source-id: 9f44eecffeb40873f5034ed59a66d21f9f88879e | 21 April 2020, 21:51:51 UTC |
6ee66cf | Tomas Kolda | 21 April 2020, 20:14:03 UTC | Prevents Table Cache to open same files more times (#6707) Summary: In highly concurrent requests table cache opens same file more times which lowers purpose of max_open_files. Fixes (https://github.com/facebook/rocksdb/issues/6699) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6707 Reviewed By: ltamasi Differential Revision: D21044965 fbshipit-source-id: f6e91d90b60dad86e518b5147021da42460ee1d2 | 21 April 2020, 20:16:31 UTC |
f9155a3 | Andrew Kryczka | 20 April 2020, 23:30:34 UTC | Prevent uninitialized load in `IndexBlockIter` (#6736) Summary: When index block is empty or an error happens while reading it, `Invalidate()` is called rather than `Initialize()`. So `Seek()` must not refer to member variables that are only initialized in `Initialize()` until it is sure `Initialize()` has been called. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6736 Reviewed By: siying Differential Revision: D21139641 Pulled By: ajkr fbshipit-source-id: 71c58cc1adbd795dc3729dd5023bf7df1515ff32 | 20 April 2020, 23:32:43 UTC |
03a1d95 | Akanksha Mahajan | 20 April 2020, 23:17:25 UTC | Set max_background_flushes dynamically (#6701) Summary: 1. Add changes so that max_background_flushes can be set dynamically. 2. Add a testcase DBOptionsTest.SetBackgroundFlushThreads which set the max_background_flushes dynamically using SetDBOptions. TestPlan: 1. make -j64 check 2. Using new testcase DBOptionsTest.SetBackgroundFlushThreads Pull Request resolved: https://github.com/facebook/rocksdb/pull/6701 Reviewed By: ajkr Differential Revision: D21028010 Pulled By: akankshamahajan15 fbshipit-source-id: 5f949e4a8fd3c32537b637947b7ee09a69cfc7c1 | 20 April 2020, 23:19:02 UTC |
31da5e3 | Peter Dillinger | 20 April 2020, 20:21:34 UTC | C++20 compatibility (#6697) Summary: Based on https://github.com/facebook/rocksdb/issues/6648 (CLA Signed), but heavily modified / extended: * Implicit capture of this via [=] deprecated in C++20, and [=,this] not standard before C++20 -> now using explicit capture lists * Implicit copy operator deprecated in gcc 9 -> add explicit '= default' definition * std::random_shuffle deprecated in C++17 and removed in C++20 -> migrated to a replacement in RocksDB random.h API * Add the ability to build with different std version though -DCMAKE_CXX_STANDARD=11/14/17/20 on the cmake command line * Minimal rebuild flag of MSVC is deprecated and is forbidden with /std:c++latest (C++20) * Added MSVC 2019 C++11 & MSVC 2019 C++20 in AppVeyor * Added GCC 9 C++11 & GCC9 C++20 in Travis Pull Request resolved: https://github.com/facebook/rocksdb/pull/6697 Test Plan: make check and CI Reviewed By: cheng-chang Differential Revision: D21020318 Pulled By: pdillinger fbshipit-source-id: 12311be5dbd8675a0e2c817f7ec50fa11c18ab91 | 20 April 2020, 20:24:25 UTC |
fe206f4 | sdong | 20 April 2020, 19:55:13 UTC | crash_test to cover index_type kBinarySearchWithFirstKey (#6721) Summary: Recently index_type kBinarySearchWithFirstKey is improved so that the API guarantee is exactly the same as other types and it is ready for wide production. We should cover it in crash tst. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6721 Test Plan: Run crash_test Reviewed By: anand1976 Differential Revision: D21099781 fbshipit-source-id: fda91eba831d9eacbb140c703e9768bb1701f935 | 20 April 2020, 19:57:15 UTC |
45d2b4e | Peter Dillinger | 20 April 2020, 18:37:36 UTC | Fix tabs and lint-ignores (#6734) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6734 Reviewed By: cheng-chang Differential Revision: D21134556 Pulled By: pdillinger fbshipit-source-id: 3636cc1d1333137b70031f8277458781c21631fb | 20 April 2020, 18:39:31 UTC |
243852e | Yanqin Jin | 17 April 2020, 21:36:51 UTC | Add IsDirectory() to Env and FS (#6711) Summary: IsDirectory() is a common API to check whether a path is a regular file or directory. POSIX: call stat() and use S_ISDIR(st_mode) Windows: PathIsDirectoryA() and PathIsDirectoryW() HDFS: FileSystem.IsDirectory() Java: File.IsDirectory() ... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6711 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21053520 Pulled By: riversand963 fbshipit-source-id: 680aadfd8ce982b63689190cf31b3145d5a89e27 | 17 April 2020, 21:39:18 UTC |
63d82e5 | sdong | 17 April 2020, 17:57:17 UTC | crash_test to cover small max_open_files (#6719) Summary: RocksDB behavior is different while max_open_files is small or large. Add the coverage to small max_open_files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6719 Test Plan: Run crash_test Reviewed By: pdillinger Differential Revision: D21081021 fbshipit-source-id: e3e211761a9bd25d93d19a61c1f7b62d48cf5e3c | 17 April 2020, 18:00:07 UTC |
9e6f3ef | Nicolas Pépin-Perreault | 16 April 2020, 22:49:52 UTC | Add RocksIterator::Refresh (#6573) Summary: This PR exposes the `Iterator::Refresh` method to the Java API by adding it on the `RocksIteratorInterface` interface. There are three concrete implementations: `RocksIterator`, `SstFileReaderIterator`, and `WBWIRocksIterator`. For the first two cases, the JNI side simply delegates to the underlying `Iterator::Refresh` method; in the last case, as it doesn't share an ancestor, and per the discussion in https://github.com/facebook/rocksdb/issues/3465, a `Status::NotSupported` exception is thrown. As the last PR had no activity in a while, I'm opening a new one - I'm completely fine with merging the previous PR if it gets completed before this is reviewed. Let me know if there's anything missing or anything else I can do :+1: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6573 Reviewed By: cheng-chang Differential Revision: D20604666 Pulled By: pdillinger fbshipit-source-id: 4de17df1180c3b87b76cfdd77b674b81fc0563f7 | 16 April 2020, 22:55:26 UTC |
9ca49bd | Adam Retter | 16 April 2020, 22:48:24 UTC | Keep building RocksJava on all architectures (#6583) Summary: Adding solid support for multiple architectures was initially triggered by RocksJava users. As such I would like to keep the CI for RocksJava on all architectures, to ensure we don't break backwards compatibility. pdillinger okay let's see how long it takes to complete Travis-CI with this one... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6583 Reviewed By: cheng-chang Differential Revision: D21036718 Pulled By: pdillinger fbshipit-source-id: 97afe0db2e4c575cc0284fdc1d4cc45d5deb2272 | 16 April 2020, 22:50:45 UTC |
5fef0ff | Adam Retter | 16 April 2020, 22:32:34 UTC | Update RocksJava static version of bzip2 (#6714) Summary: Updates the version of bzip2 used for RocksJava static builds. Please, can we also get this cherry-picked to: 1. 6.7.fb 2. 6.8.fb 3. 6.9.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714 Reviewed By: cheng-chang Differential Revision: D21067233 Pulled By: pdillinger fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e | 16 April 2020, 22:35:24 UTC |
6717ada | Andrew Kryczka | 16 April 2020, 20:13:25 UTC | Fix CF import with overlapping SST files (#6663) Summary: Invariant checking should use internal key comparator rather than `sstableKeyCompare()`. The latter was intended for checking whether a compaction input file's neighboring files need to be included in the same compaction. Using it for invariant checking was leading to false positives for files with overlapping endpoints. Fixes https://github.com/facebook/rocksdb/issues/6647. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6663 Test Plan: regression test Reviewed By: zhichao-cao Differential Revision: D20910466 Pulled By: ajkr fbshipit-source-id: f0b70dad7c4096fce635cab7a36f16e14f74ae3f | 16 April 2020, 20:16:06 UTC |
73523ba | sdong | 16 April 2020, 19:09:18 UTC | crash_test to cover options.avoid_flush_during_recovery (#6712) Summary: Options.avoid_flush_during_recovery is uncovered in crash_test. Add the coverage with a chance of 1/8, as it is a less frequently used options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6712 Test Plan: Run crash_test and see the option can be used or not used by chance. Reviewed By: ltamasi Differential Revision: D21056566 fbshipit-source-id: c3b1521517cfc204786e6ef8c6acd7fffda64793 | 16 April 2020, 19:11:45 UTC |
5801af4 | Yueh-Hsuan Chiang | 16 April 2020, 18:10:53 UTC | Add env_fault_injection argument to db_stress (#6687) Summary: Add env_fault_injection argument to db_stress. When enabled, FaultInjectionTestEnv will be used instead. Currently this option does not support running with other env setting. This will allow us to later manually produce error when running db_crashtest. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6687 Test Plan: make db_stress -j32 ./db_stress --env_fault_injection ./db_stress --env_fault_injection --hdfs // expect error message Reviewed By: ajkr Differential Revision: D21014683 Pulled By: yhchiang fbshipit-source-id: 0724aeac37efd57adb72a37defe6dbd3bfa8106a | 16 April 2020, 18:13:44 UTC |
2767972 | Cheng Chang | 16 April 2020, 18:00:00 UTC | Fix warning when O_CLOEXEC is not defined (#6695) Summary: Compilation fails on systems that do not support O_CLOEXEC. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6695 Test Plan: compile without O_CLOEXEC support Reviewed By: anand1976 Differential Revision: D21011850 Pulled By: cheng-chang fbshipit-source-id: f1bf1cce2aa65c7d10b5a9613e941db30e928347 | 16 April 2020, 18:02:50 UTC |
e45673d | Mike Kolupaev | 16 April 2020, 00:37:23 UTC | Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) Summary: Context: Index type `kBinarySearchWithFirstKey` added the ability for sst file iterator to sometimes report a key from index without reading the corresponding data block. This is useful when sst blocks are cut at some meaningful boundaries (e.g. one block per key prefix), and many seeks land between blocks (e.g. for each prefix, the ranges of keys in different sst files are nearly disjoint, so a typical seek needs to read a data block from only one file even if all files have the prefix). But this added a new error condition, which rocksdb code was really not equipped to deal with: `InternalIterator::value()` may fail with an IO error or Status::Incomplete, but it's just a method returning a Slice, with no way to report error instead. Before this PR, this type of error wasn't handled at all (an empty slice was returned), and kBinarySearchWithFirstKey implementation was considered a prototype. Now that we (LogDevice) have experimented with kBinarySearchWithFirstKey for a while and confirmed that it's really useful, this PR is adding the missing error handling. It's a pretty inconvenient situation implementation-wise. The error needs to be reported from InternalIterator when trying to access value. But there are ~700 call sites of `InternalIterator::value()`, most of which either can't hit the error condition (because the iterator is reading from memtable or from index or something) or wouldn't benefit from the deferred loading of the value (e.g. compaction iterator that reads all values anyway). Adding error handling to all these call sites would needlessly bloat the code. So instead I made the deferred value loading optional: only the call sites that may use deferred loading have to call the new method `PrepareValue()` before calling `value()`. The feature is enabled with a new bool argument `allow_unprepared_value` to a bunch of methods that create iterators (it wouldn't make sense to put it in ReadOptions because it's completely internal to iterators, with virtually no user-visible effect). Lmk if you have better ideas. Note that the deferred value loading only happens for *internal* iterators. The user-visible iterator (DBIter) always prepares the value before returning from Seek/Next/etc. We could go further and add an API to defer that value loading too, but that's most likely not useful for LogDevice, so it doesn't seem worth the complexity for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6621 Test Plan: make -j5 check . Will also deploy to some logdevice test clusters and look at stats. Reviewed By: siying Differential Revision: D20786930 Pulled By: al13n321 fbshipit-source-id: 6da77d918bad3780522e918f17f4d5513d3e99ee | 16 April 2020, 00:40:44 UTC |
610a09c | anand76 | 15 April 2020, 19:09:33 UTC | Remove a printf from db_stress that's not useful info (#6705) Summary: This was causing db_crashtest.py to wrongly assume an error by parsing the output. Hopefully this will stabilize the crash tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6705 Test Plan: make blackbox_crash_test Reviewed By: ltamasi Differential Revision: D21043335 Pulled By: anand1976 fbshipit-source-id: 5cddd112b124d4e2ebd11724a17d4ef0f50c1cf8 | 15 April 2020, 19:13:35 UTC |
165560f | sdong | 15 April 2020, 18:23:45 UTC | Two Improvements to tools/check_format_compatible.sh (#6702) Summary: Improve it in two ways: 1. tools/check_format_compatible.sh is not friendly to run outside FB environment. remove the hard-coded http proxy setting. Instead, move it to Legocastle configuration 2. Always disable warning as error, so that older build is more likely to pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6702 Test Plan: Run the test and make sure at least it doesn't break. Reviewed By: riversand963 Differential Revision: D21033329 fbshipit-source-id: 88b4ec1ec49547b772790050a165466bdc4a62a0 | 15 April 2020, 18:28:11 UTC |
234e2ed | anand76 | 14 April 2020, 18:04:39 UTC | Fix a couple of bugs in db_stress fault injection (#6700) Summary: 1. Fix a memory leak in FaultInjectionTestFS in the stack trace related code 2. Check status of all MultiGet keys before deciding whether an error was swallowed, instead of assuming an ok status for any key means an undetected error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6700 Test Plan: Run db_stress with asan and fault injection Reviewed By: cheng-chang Differential Revision: D21021498 Pulled By: anand1976 fbshipit-source-id: 489191efd1ab0fa834923a1e1d57253a7a315465 | 14 April 2020, 18:06:55 UTC |
9ae8058 | Cheng Chang | 14 April 2020, 17:52:37 UTC | Suppress file deletion error message in FaultInjectionTestEnv (#6696) Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. This is a follow up PR for https://github.com/facebook/rocksdb/pull/6694. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6696 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D21021875 Pulled By: cheng-chang fbshipit-source-id: 11e3f536df16941a89949ebcd2147cd8dfa3fbe0 | 14 April 2020, 17:55:10 UTC |
3d6d7bc | anand76 | 14 April 2020, 02:56:02 UTC | Log CompactOnDeletionCollectorFactory parameters on DB open (#6686) Summary: Log it in the info log to help in troubleshooting. It is logged as follows - ``` 2020/04/10-10:51:39.886662 7ffff7fef340 Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 100 Deletion trigger = 90); ``` Tests: make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6686 Reviewed By: ltamasi Differential Revision: D21002442 Pulled By: anand1976 fbshipit-source-id: 7adf0dbae7f1febcb00ce61fea5097118ede5c6a | 14 April 2020, 02:58:04 UTC |
38dfa40 | Zhichao Cao | 14 April 2020, 02:11:19 UTC | Add NewFileChecksumGenCrc32cFactory to file checksum (#6688) Summary: Add NewFileChecksumGenCrc32cFactory to file checksum public interface such that applications can use the build in crc32 checksum factory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6688 Test Plan: pass make asan_check Reviewed By: riversand963 Differential Revision: D21006859 Pulled By: zhichao-cao fbshipit-source-id: ea8a45196a8b77c310728ab05f6cc0f49f3baef0 | 14 April 2020, 02:13:41 UTC |
41563b6 | Ziyue Yang | 13 April 2020, 23:22:19 UTC | Fix data racing of BlockBasedTableBuilder::ParallelCompressionRep::first_block (#6640) Summary: BlockBasedTableBuilder::ParallelCompressionRep::first_block can be read in Flush() and written in BGWorkWriteRawBlock() concurrently. This commit fixes the issue by reading first_block out before pushing the block to compression and write. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6640 Test Plan: Run all tests concurrently with TSAN. Reviewed By: cheng-chang Differential Revision: D20851370 fbshipit-source-id: 6f039222e8319d31e15f1b45e05c106527253f72 | 13 April 2020, 23:24:57 UTC |
d9cad3a | anand76 | 13 April 2020, 22:16:14 UTC | Suppress file deletion error message in FaultInjectionTestFS (#6694) Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6694 Reviewed By: siying Differential Revision: D20998531 Pulled By: anand1976 fbshipit-source-id: 89cb54a5f5bb664ae6d239c37559f10e14c5ea07 | 13 April 2020, 22:18:38 UTC |
9eca6d6 | Andrew Kryczka | 13 April 2020, 18:15:20 UTC | fix comparison count for format_version=3 indexes (#6650) Summary: In index blocks since `format_version=3`, user keys are written rather than internal keys. When reading such blocks, the comparator is obtained via `InternalKeyComparator::user_comparator()`. That function must not return an unwrapped result as the wrapper class provides accounting logic to populate `PerfContext::user_key_comparison_count`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6650 Test Plan: ran db_bench and verified `PerfContext::user_key_comparison_count` became larger. Reviewed By: cheng-chang Differential Revision: D20866325 Pulled By: ajkr fbshipit-source-id: ad755d46bda31157dacc5b66e532279f19ad538c | 13 April 2020, 18:18:37 UTC |
79c838e | anand76 | 13 April 2020, 17:58:43 UTC | Fix a few bugs in db_stress fault injection (#6693) Summary: Fix the following issues - 1. Output parsing error in db_crashtest.py 2. Memory leak on exit 3. False alarm on filter block read error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6693 Test Plan: asan_crash Reviewed By: cheng-chang Differential Revision: D20990399 Pulled By: anand1976 fbshipit-source-id: 178ee0dd7c69a4bc5db698379db0dedb29281699 | 13 April 2020, 18:01:03 UTC |
eeb3cf3 | Yanqin Jin | 12 April 2020, 05:02:17 UTC | Fix release build (#6690) Summary: Fix release build caused by variable defined but unused. Test plan (devserver) ``` make release ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6690 Reviewed By: cheng-chang Differential Revision: D20980571 Pulled By: riversand963 fbshipit-source-id: c3f3b13f81dce4bdb19876dc2e710d5902ff8a02 | 12 April 2020, 05:04:04 UTC |
5c19a44 | anand76 | 11 April 2020, 00:18:56 UTC | Fault injection in db_stress (#6538) Summary: This PR implements a fault injection mechanism for injecting errors in reads in db_stress. The FaultInjectionTestFS is used for this purpose. A thread local structure is used to track the errors, so that each db_stress thread can independently enable/disable error injection and verify observed errors against expected errors. This is initially enabled only for Get and MultiGet, but can be extended to iterator as well once its proven stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6538 Test Plan: crash_test make check Reviewed By: riversand963 Differential Revision: D20714347 Pulled By: anand1976 fbshipit-source-id: d7598321d4a2d72bda0ced57411a337a91d87dc7 | 11 April 2020, 00:21:26 UTC |
0c05624 | Yanqin Jin | 10 April 2020, 23:03:33 UTC | Compaction with timestamp: input boundaries (#6645) Summary: Towards making compaction logic compatible with user timestamp. When computing boundaries and overlapping ranges for inputs of compaction, We need to compare SSTs by user key without timestamp. Test plan (devserver): ``` make check ``` Several individual tests: ``` ./version_set_test --gtest_filter=VersionStorageInfoTimestampTest.GetOverlappingInputs ./db_with_timestamp_compaction_test ./db_with_timestamp_basic_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6645 Reviewed By: ltamasi Differential Revision: D20960012 Pulled By: riversand963 fbshipit-source-id: ad377fa9eb481bf7a8a3e1824aaade48cdc653a4 | 10 April 2020, 23:05:49 UTC |