Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
0720483 Tongliang Liao 13 May 2020, 04:05:50 UTC Mark dependencies as PRIVATE and fix missing dependencies in tools. (#6790) Summary: Tools were mistakenly using leaked (PUBLIC) dependencies from main lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6790 Reviewed By: riversand963 Differential Revision: D21471551 fbshipit-source-id: ec43b92e231777e0fcf0f865444391af09d6963b 13 May 2020, 04:07:55 UTC
4a4b8a1 sdong 13 May 2020, 01:21:32 UTC sst_dump to reduce number of file reads (#6836) Summary: sst_dump can issue many file reads from the file system. This doesn't work well with file systems without a OS cache, especially remote file systems. In order to mitigate this problem, several improvements are done: 1. --readahead_size is added, so that users can specify readahead size when scanning the data. 2. Force a 512KB tail readahead, which prevents three I/Os for footer, meta index and property blocks and hopefully index and filter blocks too. 3. Consoldiate SSTDump's I/Os before opening the file for read. Use the same file prefetch buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6836 Test Plan: Add a test that covers this new feature. Reviewed By: pdillinger Differential Revision: D21516607 fbshipit-source-id: 3ae43526286f67b2f4a5bdedfbc92719d579b87e 13 May 2020, 01:23:33 UTC
70aaa9c Stanislav Tkach 12 May 2020, 21:48:14 UTC Expose CancellAllBackgroundWork to C api (#6832) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6832 Reviewed By: cheng-chang Differential Revision: D21498186 Pulled By: riversand963 fbshipit-source-id: 66bb0d7c06af2bf0df3c6a09b61bca2fb81f2dd3 12 May 2020, 21:50:52 UTC
c384c08 Ziyue Yang 12 May 2020, 16:25:21 UTC Add tests for compression failure in BlockBasedTableBuilder (#6709) Summary: Currently there is no check for whether BlockBasedTableBuilder will expose compression error status if compression fails during the table building. This commit adds fake faulting compressors and a unit test to test such cases. This check finds 5 bugs, and this commit also fixes them: 1. Not handling compression failure well in BlockBasedTableBuilder::BGWorkWriteRawBlock. 2. verify_compression failing in BlockBasedTableBuilder when used with ZSTD. 3. Wrongly passing the same reference of block contents to BlockBasedTableBuilder::CompressAndVerifyBlock in parallel compression. 4. Wrongly setting block_rep->first_key_in_next_block to nullptr in BlockBasedTableBuilder::EnterUnbuffered when there are still incoming data blocks. 5. Not maintaining variables for compression ratio estimation and first_block in BlockBasedTableBuilder::EnterUnbuffered. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6709 Reviewed By: ajkr Differential Revision: D21236254 fbshipit-source-id: 101f6e62b2bac2b7be72be198adf93cd32a1ff46 12 May 2020, 16:27:35 UTC
3f21807 yetingsky 11 May 2020, 21:56:36 UTC fix typo (#6831) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6831 Reviewed By: cheng-chang Differential Revision: D21499149 Pulled By: zhichao-cao fbshipit-source-id: 2cb76cbf7086677d8cad5c828019e008062f0052 11 May 2020, 21:58:25 UTC
2e93247 Sagar Vemuri 11 May 2020, 20:28:16 UTC Disable a few timer tests (#6833) Summary: Disable `TimerTest.SingleScheduleRepeatedlyTest` and `TimerTest.MultipleScheduleRepeatedlyTest`. This is to help people to not hit any hangs (https://github.com/facebook/rocksdb/issues/6698) during their development process while I investigate further; I could not reproduce the issue on my dev machine yet. Note that timer is not being utilized anywhere yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6833 Test Plan: ``` svemuri@devbig187 ~/rocksdb (timer-disable-test) $ TEST_TMPDIR=/dev/shm ./timer_test [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from TimerTest [ RUN ] TimerTest.SingleScheduleOnceTest [ OK ] TimerTest.SingleScheduleOnceTest (1 ms) [ RUN ] TimerTest.MultipleScheduleOnceTest [ OK ] TimerTest.MultipleScheduleOnceTest (0 ms) [----------] 2 tests from TimerTest (1 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (1 ms total) [ PASSED ] 2 tests. YOU HAVE 2 DISABLED TESTS ``` Reviewed By: pdillinger Differential Revision: D21502474 Pulled By: sagar0 fbshipit-source-id: ac67caee2011fd14ffb2476a8914a6286a4f9abe 11 May 2020, 20:30:00 UTC
f0e8731 Tongliang Liao 11 May 2020, 17:26:18 UTC Use GFlags/Snappy config in CMake, with fallback for legacy approach. (#6771) Summary: Related to some discussion in https://github.com/facebook/rocksdb/issues/6179 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6771 Reviewed By: zhichao-cao Differential Revision: D21340117 fbshipit-source-id: a1af0ba4865bb13c8c817851d6f6c4056191b3fe 11 May 2020, 17:28:49 UTC
3a1c29d Derrick Pallas 09 May 2020, 03:48:17 UTC Add missing my_pid to fprintf in multi_process_example (#6731) Summary: Signed-off-by: Derrick Pallas <derrick@pallas.us> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6731 Reviewed By: siying Differential Revision: D21137005 Pulled By: riversand963 fbshipit-source-id: a7182e1bec225bc110971f40e2d0e6c3a671c061 09 May 2020, 03:49:33 UTC
a50ea71 sdong 08 May 2020, 21:12:18 UTC Improve ldb consistency checks (#6802) Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0 08 May 2020, 21:17:47 UTC
d9cd335 sdong 08 May 2020, 21:11:03 UTC Suppress UBSAN warning in CRC32 ARM (#6827) Summary: UBSAN shows following warning: util/crc32c_arm64.cc:111:11: runtime error: load of misaligned address 0x00001afcda86 for type 'const uint64_t', which requires 8 byte alignment 0x00001afcda86: note: pointer points here cc c1 2d 00 01 81 40 24 30 66 39 66 30 37 30 63 2d 32 36 63 34 2d 34 62 61 61 2d 38 35 33 31 2d ^ Suppress it just as what we do in x86 CRC. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6827 Test Plan: Run the same UBSAN and see it to pass now. Reviewed By: ltamasi Differential Revision: D21471838 fbshipit-source-id: 02943dd39a7030d2b03e5d894dcb23ed72b6c9c3 08 May 2020, 21:14:00 UTC
e72e216 Yanqin Jin 08 May 2020, 19:59:02 UTC Fix a few bugs in best-efforts recovery (#6824) Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2 08 May 2020, 20:01:42 UTC
1c84660 Andrew Kryczka 08 May 2020, 19:36:49 UTC prototype status check enforcement (#6798) Summary: Tried making Status object enforce that it is checked in some way. In cases it is not checked, `PermitUncheckedError()` must be called explicitly. Added a way to run tests (`ASSERT_STATUS_CHECKED=1 make -j48 check`) on a whitelist. The effort appears significant to get each test to pass with this assertion, so I only fixed up enough to get one test (`options_test`) working and added it to the whitelist. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6798 Reviewed By: pdillinger Differential Revision: D21377404 Pulled By: ajkr fbshipit-source-id: 73236f9c8df38f01cf24ecac4a6d1661b72d077e 08 May 2020, 19:40:43 UTC
1282589 sdong 08 May 2020, 03:50:14 UTC Disable "compression_parallel_threads" in crash test for now (#6816) Summary: "compressio_parallel_threads" caused several test failure tests. To keep crash test clean, disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6816 Test Plan: "make crash_test" to make sure the python script doesn't break Reviewed By: zhichao-cao Differential Revision: D21462112 fbshipit-source-id: 9eecc764800da82cd19665dc8b167eacead3310b 08 May 2020, 03:52:29 UTC
9426523 anand76 08 May 2020, 00:30:41 UTC Fix race due to delete triggered compaction in Universal compaction mode (#6799) Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a 08 May 2020, 00:32:17 UTC
3730b05 Andrew Kryczka 07 May 2020, 23:37:47 UTC Fixup HISTORY.md for e9ba4ba "validate range tombstone covers positiv… (#6825) Summary: …e range" Moved it from the wrong section (6.10) to the right section (Unreleased). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6825 Reviewed By: zhichao-cao Differential Revision: D21464577 Pulled By: ajkr fbshipit-source-id: a836b4ab10be2464182826f9411c9c424c933b70 07 May 2020, 23:40:17 UTC
f286fb3 anand76 07 May 2020, 22:52:58 UTC Include options.h in table.h (#6823) Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d 07 May 2020, 22:55:29 UTC
b27a144 Peter Dillinger 07 May 2020, 22:39:49 UTC Fix false NotFound from batched MultiGet with kHashSearch (#6821) Summary: The error is assigning KeyContext::s to NotFound status in a table reader for a "not found in this table" case, which skips searching in later tables, like only a delete should. (The hash search index iterator is the only one that can return status NotFound even if Valid() == false.) This was detected by intermittent failure in MultiThreadedDBTest.MultiThreaded/5, a kHashSearch configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6821 Test Plan: modified existing unit test to reproduce problem Reviewed By: anand1976 Differential Revision: D21450469 Pulled By: pdillinger fbshipit-source-id: 7478003684d637dbd491cdac81468041a791be2c 07 May 2020, 22:41:37 UTC
e9ba4ba Andrew Kryczka 07 May 2020, 18:53:32 UTC validate range tombstone covers positive range (#6788) Summary: We found some files containing nothing but negative range tombstones, and unsurprisingly their metadata specified a negative range, which made things crash. Time to add a bit of user input validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6788 Reviewed By: zhichao-cao Differential Revision: D21343719 Pulled By: ajkr fbshipit-source-id: f1c16e4c3e9fa150958c8c866176632a3206fb74 07 May 2020, 18:55:30 UTC
ac3ae1d Levi Tamasi 07 May 2020, 16:29:21 UTC Find/purge obsolete blob files (#6807) Summary: The patch extends `FindObsoleteFiles` and `PurgeObsoleteFiles` with support for blob files. The behavior is analogous to SST files: obsolete blob files are put on the "candidates for deletion" list, while live (and pending) files are preserved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6807 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21406249 Pulled By: ltamasi fbshipit-source-id: 1948f71c31927564b61e8af394f50ca3964880d9 07 May 2020, 16:32:51 UTC
1f20df2 Andrew Kryczka 07 May 2020, 01:06:04 UTC cover single level universal in crash test (#6818) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6818 Test Plan: fast whitebox test and verify there are some single-level universal and some multi-level universal runs. ``` $ python ./tools/db_crashtest.py whitebox --simple -max_key=1000000 -value_size_mult=33 -write_buffer_size=524288 -target_file_size_base=524288 -max_bytes_for_level_base=2097152 --duration=120 --interval=10 --ops_per_thread=1000 --random_kill_odd=887 ``` Reviewed By: riversand963 Differential Revision: D21432138 Pulled By: ajkr fbshipit-source-id: 2fc5ba9f3dfa49bb11e81da7dd00a17b476e64d7 07 May 2020, 01:08:09 UTC
c1e1185 anand76 06 May 2020, 23:40:32 UTC Update release version to 6.10 (#6797) Summary: Update HISTORY.md and version.h to 6.10. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6797 Reviewed By: zhichao-cao Differential Revision: D21371390 Pulled By: anand1976 fbshipit-source-id: 6017bca24fc5d12076d1ddaec7783c9b85712d42 06 May 2020, 23:42:37 UTC
06c3b85 Levi Tamasi 06 May 2020, 20:58:25 UTC Disallow using the base DB's storage directory as blob_dir in BlobDB (#6810) Summary: https://github.com/facebook/rocksdb/pull/6807 extends the logic that identifies and purges obsolete files to blob files handled by RocksDB itself. In order to prevent that from interfering with the current BlobDB code, we need to make sure that `BlobDBOptions::blob_dir` is different from the storage directories used by the base DB. (Note: this is true by default.) The patch adds a check that explicitly disallows this configuration and returns `Status::NotSupported` from `BlobDB::Open` in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6810 Test Plan: Tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21412676 Pulled By: ltamasi fbshipit-source-id: 6630cc7481e48c8bf55d59423b25f14d52ffe681 06 May 2020, 21:00:46 UTC
53f8447 Neha Deodhar 06 May 2020, 17:25:58 UTC Add YugabyteDB to USERS (#6786) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6786 Reviewed By: zhichao-cao Differential Revision: D21407762 Pulled By: pdillinger fbshipit-source-id: 62f74409cc72c6b9ce4107aa515955cfc5eec12a 06 May 2020, 17:28:29 UTC
c21c459 sdong 06 May 2020, 01:29:50 UTC Slightly expand converage to file consistency check failure (#6800) Summary: Current DBCompactionTest.ConsistencyFailTest checks DB fails after L0 inconsitency is found. Add slightly more coverage by introducing DBCompactionTest.ConsistencyFailTest2 which checks non-L0 files too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6800 Test Plan: Run the new test. Reviewed By: riversand963 Differential Revision: D21384806 fbshipit-source-id: 36db7b657eed42115283fe2f6afa4c3a31a3b510 06 May 2020, 01:31:53 UTC
394f2bb mrambacher 05 May 2020, 22:02:04 UTC Add OptionTypeInfo::Enum and related methods (#6423) Summary: Add methods and constructors for handling enums to the OptionTypeInfo. This change allows enums to be converted/compared without adding a special "type" to the OptionType. This change addresses a couple of issues: - It allows new enumerated types to be added to the options without editing the OptionType base class (and related methods) - It standardizes the procedure for adding enumerated types to the options, reducing potential mistakes - It moves the enum maps to the location where they are used, allowing them to be static file members rather than global values - It reduces the number of types and cases that need to be handled in the various OptionType methods Pull Request resolved: https://github.com/facebook/rocksdb/pull/6423 Reviewed By: siying Differential Revision: D21408713 Pulled By: zhichao-cao fbshipit-source-id: fc492af285d011822578b95d186a0fce25d35626 05 May 2020, 22:04:04 UTC
a96461d Andrew Kryczka 05 May 2020, 21:52:18 UTC fix swallowed error for file deletion consistency check (#6809) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6809 Reviewed By: pdillinger Differential Revision: D21411971 Pulled By: ajkr fbshipit-source-id: 900b6b0370b76e9a3e5e03f968e2ac1bbaab73b8 05 May 2020, 21:54:21 UTC
2f1700c Peter Dillinger 05 May 2020, 20:09:12 UTC Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) Summary: This very old test code bug was causing a new valgrind failure in MultiGetDeadlineExceeded Also fix hang in MultiGetDeadlineExceeded by unifying with some logic from another test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6803 Test Plan: run that unit test under valgrind, make check Reviewed By: siying Differential Revision: D21388470 Pulled By: pdillinger fbshipit-source-id: 0ce99d6d5eb8cd3195b17406892c8c5cff5fa5dd 05 May 2020, 20:11:29 UTC
91bc013 Cheng Chang 05 May 2020, 18:07:52 UTC Refactor level compaction picker (#6804) Summary: 1. refactor out PickFileToCompact to remove duplicated logic. 2. remove redundant checks of `start_level_inputs_.empty()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6804 Test Plan: make check Reviewed By: siying Differential Revision: D21390053 Pulled By: cheng-chang fbshipit-source-id: 185d5987a08bfdaf63f0f245310c6da69878d415 05 May 2020, 18:09:29 UTC
5584595 Yanqin Jin 05 May 2020, 17:44:12 UTC Do not swallow error returned from SaveTo() (#6801) Summary: With consistency check enabled, VersionBuilder::SaveTo() may return error once corruption is detected while building versions. We should handle these errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6801 Test Plan: make check Reviewed By: siying Differential Revision: D21385045 Pulled By: riversand963 fbshipit-source-id: 98f6424e2a4699b62befa21e9fe00e70a771118e 05 May 2020, 17:46:20 UTC
5a61e78 Yanqin Jin 05 May 2020, 00:43:33 UTC Fix db_stress when GetLiveFiles() flushes dropped CF (#6805) Summary: Current impl. of db_stress will abort verification and report failure if GetLiveFiles() causes a dropped column family to be flushed. This is not desired. To fix, this PR makes the following change: In GetLiveFiles, if flush is triggered and returns Status::IsColumnFamilyDropped(), then set status to Status::OK(). This is OK because dropped column families will be skipped during the rest of this function, and valid column families will have their live files returned to caller. Test plan (dev server): make check ./db_stress -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 ./db_stress -disable_wal=1 -reopen=0 -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6805 Reviewed By: ltamasi Differential Revision: D21390044 Pulled By: riversand963 fbshipit-source-id: de67846b95a4f1b88aa0a30c3d70c43cc68625b9 05 May 2020, 00:45:49 UTC
a00ddf1 Levi Tamasi 04 May 2020, 22:05:34 UTC Expose the set of live blob files from Version/VersionSet (#6785) Summary: The patch adds logic that returns the set of live blob files from `Version::AddLiveFiles` and `VersionSet::AddLiveFiles` (in addition to live table files), and also cleans up the code a bit, for example, by exposing only the numbers of table files as opposed to the earlier `FileDescriptor`s that no clients used. Moreover, the patch extends the `GetLiveFiles` API so that it also exposes blob files in the current version. Similarly to https://github.com/facebook/rocksdb/pull/6755, this is a building block for identifying and purging obsolete blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6785 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21336210 Pulled By: ltamasi fbshipit-source-id: fc1aede8a49eacd03caafbc5f6f9ce43b6270821 04 May 2020, 22:08:13 UTC
680c416 sdong 04 May 2020, 21:15:55 UTC Avoid Swallowing Some File Consistency Checking Bugs (#6793) Summary: We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump. More places are not fixed and need follow-up. Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793 Test Plan: Add a unit test to cover the reopen case. Reviewed By: riversand963 Differential Revision: D21366525 fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded 04 May 2020, 21:18:11 UTC
d9e170d Mian Qin 04 May 2020, 17:53:15 UTC Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper (#6795) Summary: Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper using db_bench. Details changes as follows. 1, add a separate random mode in MixGraph to produce all_random workload. 2, fix power inverse function for generating prefix_dist workload. 3, make sure key_offset in prefix mode is always unsigned. note: Need to carefully choose key_dist_a/b to avoid aliasing. Power inverse function range should be close to overall key space. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6795 Reviewed By: akankshamahajan15 Differential Revision: D21371095 Pulled By: zhichao-cao fbshipit-source-id: 80744381e242392c8c7cf8ac3d68fe67fe876048 04 May 2020, 17:55:14 UTC
211088d Cheng Chang 02 May 2020, 00:35:41 UTC Remove redundant update of txn_state_ in transaction Prepare (#6778) Summary: When expiration is set in a pessimistic transaction, `txn_state_` is already updated to `AWAITING_PREPARE` in the `if (expiration_time_ > 0)` block, there is no need to update the state in `if (can_prepare)` block again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6778 Test Plan: make check Reviewed By: lth Differential Revision: D21335319 Pulled By: cheng-chang fbshipit-source-id: 251d634cc7d1a0e86e673a59f0bda8584da5a35f 02 May 2020, 00:37:33 UTC
c8643ed Zhichao Cao 01 May 2020, 07:01:21 UTC Fix multiple CF replay failure in db_bench replay (#6787) Summary: The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787 Test Plan: pass trace replay test. Reviewed By: yhchiang Differential Revision: D21339941 Pulled By: zhichao-cao fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce 01 May 2020, 07:03:38 UTC
6acbbbf Yanqin Jin 01 May 2020, 02:20:55 UTC Add Github Action for some basic sanity test of PR (#6761) Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787 01 May 2020, 02:22:45 UTC
6504ae0 sdong 30 April 2020, 23:59:16 UTC Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) Summary: The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782 Test Plan: Run all existing tests. Reviewed By: zhichao-cao Differential Revision: D21338614 fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685 01 May 2020, 00:01:17 UTC
ef0c3ed Cheng Chang 30 April 2020, 23:21:05 UTC Make users explicitly be aware of prepare before commit (#6775) Summary: In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare. This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775 Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare Reviewed By: lth Differential Revision: D21313270 Pulled By: cheng-chang fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251 30 April 2020, 23:24:20 UTC
079e50d sdong 30 April 2020, 22:34:43 UTC Disallow BlockBasedTableBuilder to set status from non-OK (#6776) Summary: There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future. The solution is to only make it possible to set the status code if the status code to set is not OK. Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776 Test Plan: Run all existing test. Reviewed By: pdillinger Differential Revision: D21314382 fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375 30 April 2020, 22:37:03 UTC
6277e28 sdong 30 April 2020, 22:19:31 UTC Flag CompressionOptions::parallel_threads to be experimental (#6781) Summary: The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781 Reviewed By: pdillinger Differential Revision: D21330678 fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13 30 April 2020, 22:22:06 UTC
ab13d43 anand76 30 April 2020, 21:48:51 UTC Pass a timeout to FileSystem for random reads (#6751) Summary: Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded. For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR. Tests: Update existing unit tests to verify the correct timeout value is being passed Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751 Reviewed By: riversand963 Differential Revision: D21285631 Pulled By: anand1976 fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567 30 April 2020, 21:50:39 UTC
eecd8fb Peter Dillinger 30 April 2020, 19:08:34 UTC Fix assertion that can fail on sst corruption (#6780) Summary: An assertion that a char == a CompressionType (unsigned char) originally cast from a char can fail if the original value is negative, due to numeric promotion. The assertion should pass even if the value is invalid CompressionType, because the callee UncompressBlockContentsForCompressionType checks for that and reports status appropriately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780 Test Plan: Temporarily change kZSTD = 0x88 and see tests fail. Make this change (in addition), and tests pass. Reviewed By: siying Differential Revision: D21328498 Pulled By: pdillinger fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92 30 April 2020, 19:11:00 UTC
fe238e5 Levi Tamasi 30 April 2020, 18:23:32 UTC Keep track of obsolete blob files in VersionSet (#6755) Summary: The patch adds logic to keep track of obsolete blob files. A blob file becomes obsolete when the last `shared_ptr` that points to the corresponding `SharedBlobFileMetaData` object goes away, which, in turn, happens when the last `Version` that contains the blob file is destroyed. No longer needed blob files are added to the obsolete list in `VersionSet` using a custom deleter to avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`. Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored in `JobContext`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21233155 Pulled By: ltamasi fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a 30 April 2020, 18:25:51 UTC
cf34246 Adam Retter 30 April 2020, 17:56:33 UTC Add Slack forum to README (#6773) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773 Reviewed By: siying Differential Revision: D21310229 Pulled By: pdillinger fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf 30 April 2020, 18:00:28 UTC
e619a20 Ziyue Yang 30 April 2020, 17:46:54 UTC Add an option for parallel compression in for db_stress (#6722) Summary: This commit adds an `compression_parallel_threads` option in db_stress. It also fixes the naming of parallel compression option in db_bench to keep it aligned with others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722 Reviewed By: pdillinger Differential Revision: D21091385 fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100 30 April 2020, 17:49:07 UTC
8c69402 Zhichao Cao 30 April 2020, 15:38:18 UTC Fix potential size_t overflow in import_column_family (#6762) Summary: The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762 Test Plan: make asan_check Reviewed By: pdillinger Differential Revision: D21323063 Pulled By: zhichao-cao fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a 30 April 2020, 15:40:42 UTC
b938e60 anand76 30 April 2020, 02:25:56 UTC Fix a couple of bugs in FaultInjectionTestFS (#6777) Summary: Fix the following cases that can cause false alarms in db_stress when read fault injection is enabled - 1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error. 2. Handle the case when the randomly generated string to overwrite the original block is identical to the original. Tests: Run db_stress w/ and wo/ direct IO and fault injection turned on Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777 Reviewed By: zhichao-cao Differential Revision: D21316734 Pulled By: anand1976 fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa 30 April 2020, 02:28:29 UTC
28fe8e4 Peter Dillinger 29 April 2020, 20:42:43 UTC Fix bug in format-diff.sh (#6772) Summary: Nasty bug in which more/different changes would be applied than those shown to user Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772 Test Plan: manual Reviewed By: siying Differential Revision: D21304604 Pulled By: pdillinger fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc 29 April 2020, 20:45:54 UTC
5272305 Derrick Pallas 29 April 2020, 20:06:27 UTC Fix FilterBench when RTTI=0 (#6732) Summary: The dynamic_cast in the filter benchmark causes release mode to fail due to no-rtti. Replace with static_cast_with_check. Signed-off-by: Derrick Pallas <derrick@pallas.us> Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732 Reviewed By: ltamasi Differential Revision: D21304260 Pulled By: pdillinger fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c 29 April 2020, 20:09:23 UTC
8086e5e Peter Dillinger 29 April 2020, 04:35:24 UTC Fix LITE build (#6770) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770 Test Plan: make LITE=1 check Reviewed By: ajkr Differential Revision: D21296261 Pulled By: pdillinger fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f 29 April 2020, 04:37:20 UTC
335ea73 anand76 29 April 2020, 03:03:28 UTC Fix a valgrind failure due to DBBasicTestMultiGetDeadline (#6756) Summary: Fix a valgrind failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756 Test Plan: valgrind_test Reviewed By: pdillinger Differential Revision: D21284660 Pulled By: anand1976 fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80 29 April 2020, 03:06:06 UTC
618bf63 mrambacher 29 April 2020, 01:02:11 UTC Add Functions to OptionTypeInfo (#6422) Summary: Added functions for parsing, serializing, and comparing elements to OptionTypeInfo. These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map. Using these functions, every type can be handled via the map rather than special cased. By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422 Test Plan: pass make check Reviewed By: siying Differential Revision: D21269005 Pulled By: zhichao-cao fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077 29 April 2020, 01:04:26 UTC
b810e62 Peter Dillinger 28 April 2020, 22:23:33 UTC Clarifying comments in db.h (#6768) Summary: And fix a confusingly worded log message Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768 Reviewed By: anand1976 Differential Revision: D21284527 Pulled By: pdillinger fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164 28 April 2020, 22:26:03 UTC
bae6f58 Peter Dillinger 28 April 2020, 21:46:13 UTC Basic MultiGet support for partitioned filters (#6757) Summary: In MultiGet, access each applicable filter partition only once per batch, rather than for each applicable key. Also, * Fix Bloom stats for MultiGet * Fix/refactor MultiGetContext::Range::KeysLeft, including * Add efficient BitsSetToOne implementation * Assert that MultiGetContext::Range does not go beyond shift range Performance test: Generate db: $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true ... Before (middle performing run of three; note some missing Bloom stats): $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 26.403 micros/op 597517 ops/sec; (548427 of 671968 found) rocksdb.block.cache.filter.hit COUNT : 83443275 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 7931450 rocksdb.number.multiget.get COUNT : 385984 rocksdb.number.multiget.keys.read COUNT : 12351488 rocksdb.number.multiget.bytes.read COUNT : 793145000 rocksdb.number.multiget.keys.found COUNT : 7931450 After (middle performing run of three): $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 21.024 micros/op 752963 ops/sec; (705188 of 863968 found) rocksdb.block.cache.filter.hit COUNT : 49856682 rocksdb.bloom.filter.useful COUNT : 45684579 rocksdb.bloom.filter.full.positive COUNT : 10395458 rocksdb.bloom.filter.full.true.positive COUNT : 9908456 rocksdb.number.multiget.get COUNT : 481984 rocksdb.number.multiget.keys.read COUNT : 15423488 rocksdb.number.multiget.bytes.read COUNT : 990845600 rocksdb.number.multiget.keys.found COUNT : 9908456 So that's about 25% higher throughput even for random keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757 Test Plan: unit test included Reviewed By: anand1976 Differential Revision: D21243256 Pulled By: pdillinger fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f 28 April 2020, 21:49:34 UTC
a7f0b27 Peter Dillinger 28 April 2020, 19:27:23 UTC HISTORY.md update for bzip upgrade (#6767) Summary: See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767 Reviewed By: riversand963 Differential Revision: D21283307 Pulled By: pdillinger fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8 28 April 2020, 19:29:31 UTC
4574d75 Peter Dillinger 28 April 2020, 15:24:15 UTC Update HISTORY.md for block cache redundant adds (#6764) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6764 Reviewed By: ltamasi Differential Revision: D21267108 Pulled By: pdillinger fbshipit-source-id: a3dfe2dbe4e8f6309a53eb72903ef58d52308f97 28 April 2020, 15:26:43 UTC
d4398e0 Yanqin Jin 28 April 2020, 05:47:44 UTC Fix timestamp support for MultiGet (#6748) Summary: 1. Avoid nullptr dereference when passing timestamp to KeyContext creation. 2. Construct LookupKey correctly with timestamp when creating MultiGetContext. 3. Compare without timestamp when sorting KeyContexts. Fixes https://github.com/facebook/rocksdb/issues/6745 Test plan (dev server): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6748 Reviewed By: pdillinger Differential Revision: D21258691 Pulled By: riversand963 fbshipit-source-id: 44e65b759c18b9986947783edf03be4f890bb004 28 April 2020, 05:49:56 UTC
4cd859e Cheng Chang 27 April 2020, 23:51:42 UTC Fix build under LITE (#6758) Summary: GetSupportedCompressions needs to be defined under LITE. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6758 Test Plan: build under LITE Reviewed By: zhichao-cao Differential Revision: D21247937 Pulled By: cheng-chang fbshipit-source-id: 880e59d3e107cdd736d16427a68c5641d1318fb4 27 April 2020, 23:55:14 UTC
bea91d5 Levi Tamasi 27 April 2020, 23:42:16 UTC Destroy any ColumnFamilyHandles in BlobDB::Open upon error (#6763) Summary: If an error happens during BlobDBImpl::Open after the base DB has been opened, we need to destroy the `ColumnFamilyHandle`s returned by `DB::Open` to prevent an assertion in `ColumnFamilySet`'s destructor from being hit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6763 Test Plan: Ran `make check` and tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21262643 Pulled By: ltamasi fbshipit-source-id: 60ebc7ab19be66cf37fbe5f6d8957d58470f3d3b 27 April 2020, 23:45:13 UTC
cc8d16e Albert Hse-Lin Chen 27 April 2020, 21:39:54 UTC Fixed minor typo in comment for MergeOperator::FullMergeV2() (#6759) Summary: Fixed minor typo in comment for FullMergeV2(). Last operand up to snapshot should be +4 instead of +3. Signed-off-by: Albert Hse-Lin Chen <hselin@kalista.io> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6759 Reviewed By: cheng-chang Differential Revision: D21260295 Pulled By: zhichao-cao fbshipit-source-id: cc942306f246c8606538feb30bfdf6df9fb6c54e 27 April 2020, 21:44:47 UTC
249eff0 Peter Dillinger 27 April 2020, 20:18:18 UTC Stats for redundant insertions into block cache (#6681) Summary: Since read threads do not coordinate on loading data into block cache, two threads between Lookup and Insert can end up loading and inserting the same data. This is particularly concerning with cache_index_and_filter_blocks since those are hot and more likely to be race targets if ejected from (or not pre-populated in) the cache. Particularly with moves toward disaggregated / network storage, the cost of redundant retrieval might be high, and we should at least have some hard statistics from which we can estimate impact. Example with full filter thrashing "cliff": $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 ... $ ./db_bench --db=/tmp/rocksdbtest-172704/dbbench --use_existing_db --benchmarks=readrandom,stats --num=200000 --cache_index_and_filter_blocks --cache_size=$((130 * 1024 * 1024)) --bloom_bits=10 --threads=16 -statistics 2>&1 | egrep '^rocksdb.block.cache.(.*add|.*redundant)' | grep -v compress | sort rocksdb.block.cache.add COUNT : 14181 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 476 rocksdb.block.cache.data.add COUNT : 12749 rocksdb.block.cache.data.add.redundant COUNT : 18 rocksdb.block.cache.filter.add COUNT : 1003 rocksdb.block.cache.filter.add.redundant COUNT : 217 rocksdb.block.cache.index.add COUNT : 429 rocksdb.block.cache.index.add.redundant COUNT : 241 $ ./db_bench --db=/tmp/rocksdbtest-172704/dbbench --use_existing_db --benchmarks=readrandom,stats --num=200000 --cache_index_and_filter_blocks --cache_size=$((120 * 1024 * 1024)) --bloom_bits=10 --threads=16 -statistics 2>&1 | egrep '^rocksdb.block.cache.(.*add|.*redundant)' | grep -v compress | sort rocksdb.block.cache.add COUNT : 1182223 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 302728 rocksdb.block.cache.data.add COUNT : 31425 rocksdb.block.cache.data.add.redundant COUNT : 12 rocksdb.block.cache.filter.add COUNT : 795455 rocksdb.block.cache.filter.add.redundant COUNT : 130238 rocksdb.block.cache.index.add COUNT : 355343 rocksdb.block.cache.index.add.redundant COUNT : 172478 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6681 Test Plan: Some manual testing (above) and unit test covering key metrics is included Reviewed By: ltamasi Differential Revision: D21134113 Pulled By: pdillinger fbshipit-source-id: c11497b5f00f4ffdfe919823904e52d0a1a91d87 27 April 2020, 20:20:27 UTC
75b13ea Akanksha Mahajan 27 April 2020, 19:33:49 UTC Allow sst_dump to check size of different compression levels and report time (#6634) Summary: Summary : 1. Add two arguments --compression_level_from and --compression_level_to to check the compression size with different compression level in the given range. Users must specify one compression type else it will error out. Both from and to levels must also be specified together. 2. Display the time taken to compress each file with different compressions by default. Test Plan : make -j64 check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6634 Test Plan: make -j64 check Reviewed By: anand1976 Differential Revision: D20810282 Pulled By: akankshamahajan15 fbshipit-source-id: ac9098d3c079a1fad098f6678dbedb4d888a791b 27 April 2020, 19:36:16 UTC
791e571 Peter Dillinger 27 April 2020, 17:45:49 UTC Understand common build variables passed as make variables (#6740) Summary: Some common build variables like USE_CLANG and COMPILE_WITH_UBSAN did not work if specified as make variables, as in `make USE_CLANG=1 check` etc. rather than (in theory less hygienic) `USE_CLANG=1 make check`. This patches Makefile to export some commonly used ones to build_detect_platform so that they work. (I'm skeptical of a broad `export` in Makefile because it's hard to predict how random make variables might affect various invoked tools.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6740 Test Plan: manual / CI Reviewed By: siying Differential Revision: D21229011 Pulled By: pdillinger fbshipit-source-id: b00c69b23eb2a13105bc8d860ce2d1e61ac5a355 27 April 2020, 17:48:49 UTC
3b2f271 Yanqin Jin 27 April 2020, 00:33:50 UTC Update buckifier to unblock future internal release (#6726) Summary: Some recent PRs added new source files or modified TARGETS file manually. During next internal release, executing the following command will revert the manual changes. Update buckifier so that the following command ``` python buckfier/buckify_rocksdb.py ``` does not change TARGETS file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6726 Test Plan: ``` python buckifier/buckify_rocksdb.py ``` Reviewed By: siying Differential Revision: D21098930 Pulled By: riversand963 fbshipit-source-id: e884f507fefef88163363c9097a460c98f1ed850 27 April 2020, 00:35:37 UTC
0a77617 Cheng Chang 25 April 2020, 06:58:13 UTC Disable O_DIRECT in stress test when db directory does not support direct IO (#6727) Summary: In crash test, the db directory might be set to /dev/shm or /tmp, in certain environments such as internal testing infrastructure, neither of these directories support direct IO, so direct IO is never enabled in crash test. This PR sets up SyncPoints in direct IO related code paths to disable O_DIRECT flag in calls to `open`, so the direct IO code paths will be executed, all direct IO related assertions will be checked, but no real direct IO request will be issued to the file system. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6727 Test Plan: export CRASH_TEST_EXT_ARGS="--use_direct_reads=1 --mmap_read=0" make -j24 crash_test Reviewed By: zhichao-cao Differential Revision: D21139250 Pulled By: cheng-chang fbshipit-source-id: db9adfe78d91aa4759835b1af91c5db7b27b62ee 25 April 2020, 07:01:03 UTC
40497a8 Cheng Chang 24 April 2020, 22:30:12 UTC Reduce memory copies when fetching and uncompressing blocks from SST files (#6689) Summary: In https://github.com/facebook/rocksdb/pull/6455, we modified the interface of `RandomAccessFileReader::Read` to be able to get rid of memcpy in direct IO mode. This PR applies the new interface to `BlockFetcher` when reading blocks from SST files in direct IO mode. Without this PR, in direct IO mode, when fetching and uncompressing compressed blocks, `BlockFetcher` will first copy the raw compressed block into `BlockFetcher::compressed_buf_` or `BlockFetcher::stack_buf_` inside `RandomAccessFileReader::Read` depending on the block size. then during uncompressing, it will copy the uncompressed block into `BlockFetcher::heap_buf_`. In this PR, we get rid of the first memcpy and directly uncompress the block from `direct_io_buf_` to `heap_buf_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6689 Test Plan: A new unit test `block_fetcher_test` is added. Reviewed By: anand1976 Differential Revision: D21006729 Pulled By: cheng-chang fbshipit-source-id: 2370b92c24075692423b81277415feb2aed5d980 24 April 2020, 22:32:56 UTC
1758f76 Cheng Chang 24 April 2020, 22:11:42 UTC Fix unused variable of r in release mode (#6750) Summary: In release mode, asserts are not compiled, so `r` is not used, causing compiler warnings. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6750 Test Plan: make check under release mode Reviewed By: anand1976 Differential Revision: D21220365 Pulled By: cheng-chang fbshipit-source-id: fd4afa9843d54af68c4da8660ec61549803e1167 24 April 2020, 22:14:13 UTC
9e7b7e2 anand76 24 April 2020, 20:03:08 UTC Silence false alarms in db_stress fault injection (#6741) Summary: False alarms are caused by codepaths that intentionally swallow IO errors. Tests: make crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6741 Reviewed By: ltamasi Differential Revision: D21181138 Pulled By: anand1976 fbshipit-source-id: 5ccfbc68eb192033488de6269e59c00f2c65ce00 24 April 2020, 20:06:12 UTC
e04f3bc Yanqin Jin 23 April 2020, 23:18:28 UTC Update CURRENT file after best-efforts recovery (#6746) Summary: After a successful recovery, the CURRENT file should be updated to point to the valid MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6746 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21189876 Pulled By: riversand963 fbshipit-source-id: 7537b49988c5c425ebe9505a5cc260de351ad79b 23 April 2020, 23:21:09 UTC
51bdfae Cheng Chang 23 April 2020, 22:17:00 UTC Check alignment of MultiRead requests in direct IO mode (#6739) Summary: Add assertions to check direct IO's alignment requirements in MultiRead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6739 Test Plan: make check Reviewed By: siying Differential Revision: D21143825 Pulled By: cheng-chang fbshipit-source-id: 26f1623b062a1851080771128feac0669a61f5e9 23 April 2020, 22:19:31 UTC
bc51e33 Levi Tamasi 23 April 2020, 20:41:32 UTC Make sure (Shared)BlobFileMetaData are owned by shared_ptrs (#6749) Summary: The patch makes a couple of small cleanups to `SharedBlobFileMetaData` and `BlobFileMetaData`: * It makes the constructors private and introduces factory methods to ensure these objects are always owned by `shared_ptr`s. Note that `SharedBlobFileMetaData` has an additional factory that takes a deleter object; we can utilize this to e.g. notify `VersionSet` when a blob file becomes obsolete (which is exactly when `SharedBlobFileMetaData` is destroyed). * It disables move operations explicitly instead of relying on them being suppressed because of a user-declared destructor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6749 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21206947 Pulled By: ltamasi fbshipit-source-id: 9094c14cc335b3e226f883e5a0df4f87a5cdeb95 23 April 2020, 20:44:29 UTC
ae77880 Ibrahim Jarif 23 April 2020, 19:26:56 UTC Fix some typos in code comments (#6733) Summary: This PR fixes some typos in code comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6733 Reviewed By: siying Differential Revision: D21209037 Pulled By: zhichao-cao fbshipit-source-id: d9274611fab1f5e992998c8c4117b8078c4cbc69 23 April 2020, 19:28:49 UTC
4cbc19d mrambacher 22 April 2020, 00:35:28 UTC Add a ConfigOptions for use in comparing objects and converting to/from strings (#6389) Summary: The methods in convenience.h are used to compare/convert objects to/from strings. There is a mishmash of parameters in use here with more needed in the future. This PR replaces those parameters with a single structure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6389 Reviewed By: siying Differential Revision: D21163707 Pulled By: zhichao-cao fbshipit-source-id: f807b4cc7e2b0af3871536b69546b2604dfa81bd 22 April 2020, 00:38:17 UTC
c1ccd6b anand76 21 April 2020, 21:49:50 UTC Implement deadline support for MultiGet (#6710) Summary: Initial implementation of ReadOptions.deadline for MultiGet. If the request takes longer than the deadline, the keys not yet found will be returned with Status::TimedOut(). This implementation enforces the deadline in DBImpl, which is fairly high level. Its best effort and may not check the deadline after every key lookup, but may do so after a batch of keys. In subsequent stages, we will extend this to passing a timeout down to the FileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6710 Test Plan: Add new unit tests Reviewed By: riversand963 Differential Revision: D21149158 Pulled By: anand1976 fbshipit-source-id: 9f44eecffeb40873f5034ed59a66d21f9f88879e 21 April 2020, 21:51:51 UTC
6ee66cf Tomas Kolda 21 April 2020, 20:14:03 UTC Prevents Table Cache to open same files more times (#6707) Summary: In highly concurrent requests table cache opens same file more times which lowers purpose of max_open_files. Fixes (https://github.com/facebook/rocksdb/issues/6699) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6707 Reviewed By: ltamasi Differential Revision: D21044965 fbshipit-source-id: f6e91d90b60dad86e518b5147021da42460ee1d2 21 April 2020, 20:16:31 UTC
f9155a3 Andrew Kryczka 20 April 2020, 23:30:34 UTC Prevent uninitialized load in `IndexBlockIter` (#6736) Summary: When index block is empty or an error happens while reading it, `Invalidate()` is called rather than `Initialize()`. So `Seek()` must not refer to member variables that are only initialized in `Initialize()` until it is sure `Initialize()` has been called. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6736 Reviewed By: siying Differential Revision: D21139641 Pulled By: ajkr fbshipit-source-id: 71c58cc1adbd795dc3729dd5023bf7df1515ff32 20 April 2020, 23:32:43 UTC
03a1d95 Akanksha Mahajan 20 April 2020, 23:17:25 UTC Set max_background_flushes dynamically (#6701) Summary: 1. Add changes so that max_background_flushes can be set dynamically. 2. Add a testcase DBOptionsTest.SetBackgroundFlushThreads which set the max_background_flushes dynamically using SetDBOptions. TestPlan: 1. make -j64 check 2. Using new testcase DBOptionsTest.SetBackgroundFlushThreads Pull Request resolved: https://github.com/facebook/rocksdb/pull/6701 Reviewed By: ajkr Differential Revision: D21028010 Pulled By: akankshamahajan15 fbshipit-source-id: 5f949e4a8fd3c32537b637947b7ee09a69cfc7c1 20 April 2020, 23:19:02 UTC
31da5e3 Peter Dillinger 20 April 2020, 20:21:34 UTC C++20 compatibility (#6697) Summary: Based on https://github.com/facebook/rocksdb/issues/6648 (CLA Signed), but heavily modified / extended: * Implicit capture of this via [=] deprecated in C++20, and [=,this] not standard before C++20 -> now using explicit capture lists * Implicit copy operator deprecated in gcc 9 -> add explicit '= default' definition * std::random_shuffle deprecated in C++17 and removed in C++20 -> migrated to a replacement in RocksDB random.h API * Add the ability to build with different std version though -DCMAKE_CXX_STANDARD=11/14/17/20 on the cmake command line * Minimal rebuild flag of MSVC is deprecated and is forbidden with /std:c++latest (C++20) * Added MSVC 2019 C++11 & MSVC 2019 C++20 in AppVeyor * Added GCC 9 C++11 & GCC9 C++20 in Travis Pull Request resolved: https://github.com/facebook/rocksdb/pull/6697 Test Plan: make check and CI Reviewed By: cheng-chang Differential Revision: D21020318 Pulled By: pdillinger fbshipit-source-id: 12311be5dbd8675a0e2c817f7ec50fa11c18ab91 20 April 2020, 20:24:25 UTC
fe206f4 sdong 20 April 2020, 19:55:13 UTC crash_test to cover index_type kBinarySearchWithFirstKey (#6721) Summary: Recently index_type kBinarySearchWithFirstKey is improved so that the API guarantee is exactly the same as other types and it is ready for wide production. We should cover it in crash tst. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6721 Test Plan: Run crash_test Reviewed By: anand1976 Differential Revision: D21099781 fbshipit-source-id: fda91eba831d9eacbb140c703e9768bb1701f935 20 April 2020, 19:57:15 UTC
45d2b4e Peter Dillinger 20 April 2020, 18:37:36 UTC Fix tabs and lint-ignores (#6734) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6734 Reviewed By: cheng-chang Differential Revision: D21134556 Pulled By: pdillinger fbshipit-source-id: 3636cc1d1333137b70031f8277458781c21631fb 20 April 2020, 18:39:31 UTC
243852e Yanqin Jin 17 April 2020, 21:36:51 UTC Add IsDirectory() to Env and FS (#6711) Summary: IsDirectory() is a common API to check whether a path is a regular file or directory. POSIX: call stat() and use S_ISDIR(st_mode) Windows: PathIsDirectoryA() and PathIsDirectoryW() HDFS: FileSystem.IsDirectory() Java: File.IsDirectory() ... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6711 Test Plan: make check Reviewed By: anand1976 Differential Revision: D21053520 Pulled By: riversand963 fbshipit-source-id: 680aadfd8ce982b63689190cf31b3145d5a89e27 17 April 2020, 21:39:18 UTC
63d82e5 sdong 17 April 2020, 17:57:17 UTC crash_test to cover small max_open_files (#6719) Summary: RocksDB behavior is different while max_open_files is small or large. Add the coverage to small max_open_files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6719 Test Plan: Run crash_test Reviewed By: pdillinger Differential Revision: D21081021 fbshipit-source-id: e3e211761a9bd25d93d19a61c1f7b62d48cf5e3c 17 April 2020, 18:00:07 UTC
9e6f3ef Nicolas Pépin-Perreault 16 April 2020, 22:49:52 UTC Add RocksIterator::Refresh (#6573) Summary: This PR exposes the `Iterator::Refresh` method to the Java API by adding it on the `RocksIteratorInterface` interface. There are three concrete implementations: `RocksIterator`, `SstFileReaderIterator`, and `WBWIRocksIterator`. For the first two cases, the JNI side simply delegates to the underlying `Iterator::Refresh` method; in the last case, as it doesn't share an ancestor, and per the discussion in https://github.com/facebook/rocksdb/issues/3465, a `Status::NotSupported` exception is thrown. As the last PR had no activity in a while, I'm opening a new one - I'm completely fine with merging the previous PR if it gets completed before this is reviewed. Let me know if there's anything missing or anything else I can do :+1: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6573 Reviewed By: cheng-chang Differential Revision: D20604666 Pulled By: pdillinger fbshipit-source-id: 4de17df1180c3b87b76cfdd77b674b81fc0563f7 16 April 2020, 22:55:26 UTC
9ca49bd Adam Retter 16 April 2020, 22:48:24 UTC Keep building RocksJava on all architectures (#6583) Summary: Adding solid support for multiple architectures was initially triggered by RocksJava users. As such I would like to keep the CI for RocksJava on all architectures, to ensure we don't break backwards compatibility. pdillinger okay let's see how long it takes to complete Travis-CI with this one... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6583 Reviewed By: cheng-chang Differential Revision: D21036718 Pulled By: pdillinger fbshipit-source-id: 97afe0db2e4c575cc0284fdc1d4cc45d5deb2272 16 April 2020, 22:50:45 UTC
5fef0ff Adam Retter 16 April 2020, 22:32:34 UTC Update RocksJava static version of bzip2 (#6714) Summary: Updates the version of bzip2 used for RocksJava static builds. Please, can we also get this cherry-picked to: 1. 6.7.fb 2. 6.8.fb 3. 6.9.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714 Reviewed By: cheng-chang Differential Revision: D21067233 Pulled By: pdillinger fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e 16 April 2020, 22:35:24 UTC
6717ada Andrew Kryczka 16 April 2020, 20:13:25 UTC Fix CF import with overlapping SST files (#6663) Summary: Invariant checking should use internal key comparator rather than `sstableKeyCompare()`. The latter was intended for checking whether a compaction input file's neighboring files need to be included in the same compaction. Using it for invariant checking was leading to false positives for files with overlapping endpoints. Fixes https://github.com/facebook/rocksdb/issues/6647. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6663 Test Plan: regression test Reviewed By: zhichao-cao Differential Revision: D20910466 Pulled By: ajkr fbshipit-source-id: f0b70dad7c4096fce635cab7a36f16e14f74ae3f 16 April 2020, 20:16:06 UTC
73523ba sdong 16 April 2020, 19:09:18 UTC crash_test to cover options.avoid_flush_during_recovery (#6712) Summary: Options.avoid_flush_during_recovery is uncovered in crash_test. Add the coverage with a chance of 1/8, as it is a less frequently used options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6712 Test Plan: Run crash_test and see the option can be used or not used by chance. Reviewed By: ltamasi Differential Revision: D21056566 fbshipit-source-id: c3b1521517cfc204786e6ef8c6acd7fffda64793 16 April 2020, 19:11:45 UTC
5801af4 Yueh-Hsuan Chiang 16 April 2020, 18:10:53 UTC Add env_fault_injection argument to db_stress (#6687) Summary: Add env_fault_injection argument to db_stress. When enabled, FaultInjectionTestEnv will be used instead. Currently this option does not support running with other env setting. This will allow us to later manually produce error when running db_crashtest. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6687 Test Plan: make db_stress -j32 ./db_stress --env_fault_injection ./db_stress --env_fault_injection --hdfs // expect error message Reviewed By: ajkr Differential Revision: D21014683 Pulled By: yhchiang fbshipit-source-id: 0724aeac37efd57adb72a37defe6dbd3bfa8106a 16 April 2020, 18:13:44 UTC
2767972 Cheng Chang 16 April 2020, 18:00:00 UTC Fix warning when O_CLOEXEC is not defined (#6695) Summary: Compilation fails on systems that do not support O_CLOEXEC. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6695 Test Plan: compile without O_CLOEXEC support Reviewed By: anand1976 Differential Revision: D21011850 Pulled By: cheng-chang fbshipit-source-id: f1bf1cce2aa65c7d10b5a9613e941db30e928347 16 April 2020, 18:02:50 UTC
e45673d Mike Kolupaev 16 April 2020, 00:37:23 UTC Properly report IO errors when IndexType::kBinarySearchWithFirstKey is used (#6621) Summary: Context: Index type `kBinarySearchWithFirstKey` added the ability for sst file iterator to sometimes report a key from index without reading the corresponding data block. This is useful when sst blocks are cut at some meaningful boundaries (e.g. one block per key prefix), and many seeks land between blocks (e.g. for each prefix, the ranges of keys in different sst files are nearly disjoint, so a typical seek needs to read a data block from only one file even if all files have the prefix). But this added a new error condition, which rocksdb code was really not equipped to deal with: `InternalIterator::value()` may fail with an IO error or Status::Incomplete, but it's just a method returning a Slice, with no way to report error instead. Before this PR, this type of error wasn't handled at all (an empty slice was returned), and kBinarySearchWithFirstKey implementation was considered a prototype. Now that we (LogDevice) have experimented with kBinarySearchWithFirstKey for a while and confirmed that it's really useful, this PR is adding the missing error handling. It's a pretty inconvenient situation implementation-wise. The error needs to be reported from InternalIterator when trying to access value. But there are ~700 call sites of `InternalIterator::value()`, most of which either can't hit the error condition (because the iterator is reading from memtable or from index or something) or wouldn't benefit from the deferred loading of the value (e.g. compaction iterator that reads all values anyway). Adding error handling to all these call sites would needlessly bloat the code. So instead I made the deferred value loading optional: only the call sites that may use deferred loading have to call the new method `PrepareValue()` before calling `value()`. The feature is enabled with a new bool argument `allow_unprepared_value` to a bunch of methods that create iterators (it wouldn't make sense to put it in ReadOptions because it's completely internal to iterators, with virtually no user-visible effect). Lmk if you have better ideas. Note that the deferred value loading only happens for *internal* iterators. The user-visible iterator (DBIter) always prepares the value before returning from Seek/Next/etc. We could go further and add an API to defer that value loading too, but that's most likely not useful for LogDevice, so it doesn't seem worth the complexity for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6621 Test Plan: make -j5 check . Will also deploy to some logdevice test clusters and look at stats. Reviewed By: siying Differential Revision: D20786930 Pulled By: al13n321 fbshipit-source-id: 6da77d918bad3780522e918f17f4d5513d3e99ee 16 April 2020, 00:40:44 UTC
610a09c anand76 15 April 2020, 19:09:33 UTC Remove a printf from db_stress that's not useful info (#6705) Summary: This was causing db_crashtest.py to wrongly assume an error by parsing the output. Hopefully this will stabilize the crash tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6705 Test Plan: make blackbox_crash_test Reviewed By: ltamasi Differential Revision: D21043335 Pulled By: anand1976 fbshipit-source-id: 5cddd112b124d4e2ebd11724a17d4ef0f50c1cf8 15 April 2020, 19:13:35 UTC
165560f sdong 15 April 2020, 18:23:45 UTC Two Improvements to tools/check_format_compatible.sh (#6702) Summary: Improve it in two ways: 1. tools/check_format_compatible.sh is not friendly to run outside FB environment. remove the hard-coded http proxy setting. Instead, move it to Legocastle configuration 2. Always disable warning as error, so that older build is more likely to pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6702 Test Plan: Run the test and make sure at least it doesn't break. Reviewed By: riversand963 Differential Revision: D21033329 fbshipit-source-id: 88b4ec1ec49547b772790050a165466bdc4a62a0 15 April 2020, 18:28:11 UTC
234e2ed anand76 14 April 2020, 18:04:39 UTC Fix a couple of bugs in db_stress fault injection (#6700) Summary: 1. Fix a memory leak in FaultInjectionTestFS in the stack trace related code 2. Check status of all MultiGet keys before deciding whether an error was swallowed, instead of assuming an ok status for any key means an undetected error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6700 Test Plan: Run db_stress with asan and fault injection Reviewed By: cheng-chang Differential Revision: D21021498 Pulled By: anand1976 fbshipit-source-id: 489191efd1ab0fa834923a1e1d57253a7a315465 14 April 2020, 18:06:55 UTC
9ae8058 Cheng Chang 14 April 2020, 17:52:37 UTC Suppress file deletion error message in FaultInjectionTestEnv (#6696) Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. This is a follow up PR for https://github.com/facebook/rocksdb/pull/6694. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6696 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D21021875 Pulled By: cheng-chang fbshipit-source-id: 11e3f536df16941a89949ebcd2147cd8dfa3fbe0 14 April 2020, 17:55:10 UTC
3d6d7bc anand76 14 April 2020, 02:56:02 UTC Log CompactOnDeletionCollectorFactory parameters on DB open (#6686) Summary: Log it in the info log to help in troubleshooting. It is logged as follows - ``` 2020/04/10-10:51:39.886662 7ffff7fef340 Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 100 Deletion trigger = 90); ``` Tests: make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6686 Reviewed By: ltamasi Differential Revision: D21002442 Pulled By: anand1976 fbshipit-source-id: 7adf0dbae7f1febcb00ce61fea5097118ede5c6a 14 April 2020, 02:58:04 UTC
38dfa40 Zhichao Cao 14 April 2020, 02:11:19 UTC Add NewFileChecksumGenCrc32cFactory to file checksum (#6688) Summary: Add NewFileChecksumGenCrc32cFactory to file checksum public interface such that applications can use the build in crc32 checksum factory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6688 Test Plan: pass make asan_check Reviewed By: riversand963 Differential Revision: D21006859 Pulled By: zhichao-cao fbshipit-source-id: ea8a45196a8b77c310728ab05f6cc0f49f3baef0 14 April 2020, 02:13:41 UTC
41563b6 Ziyue Yang 13 April 2020, 23:22:19 UTC Fix data racing of BlockBasedTableBuilder::ParallelCompressionRep::first_block (#6640) Summary: BlockBasedTableBuilder::ParallelCompressionRep::first_block can be read in Flush() and written in BGWorkWriteRawBlock() concurrently. This commit fixes the issue by reading first_block out before pushing the block to compression and write. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6640 Test Plan: Run all tests concurrently with TSAN. Reviewed By: cheng-chang Differential Revision: D20851370 fbshipit-source-id: 6f039222e8319d31e15f1b45e05c106527253f72 13 April 2020, 23:24:57 UTC
d9cad3a anand76 13 April 2020, 22:16:14 UTC Suppress file deletion error message in FaultInjectionTestFS (#6694) Summary: The error message is causing problems in the crash tests due to the error parsing logic in db_crashtest.py. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6694 Reviewed By: siying Differential Revision: D20998531 Pulled By: anand1976 fbshipit-source-id: 89cb54a5f5bb664ae6d239c37559f10e14c5ea07 13 April 2020, 22:18:38 UTC
9eca6d6 Andrew Kryczka 13 April 2020, 18:15:20 UTC fix comparison count for format_version=3 indexes (#6650) Summary: In index blocks since `format_version=3`, user keys are written rather than internal keys. When reading such blocks, the comparator is obtained via `InternalKeyComparator::user_comparator()`. That function must not return an unwrapped result as the wrapper class provides accounting logic to populate `PerfContext::user_key_comparison_count`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6650 Test Plan: ran db_bench and verified `PerfContext::user_key_comparison_count` became larger. Reviewed By: cheng-chang Differential Revision: D20866325 Pulled By: ajkr fbshipit-source-id: ad755d46bda31157dacc5b66e532279f19ad538c 13 April 2020, 18:18:37 UTC
79c838e anand76 13 April 2020, 17:58:43 UTC Fix a few bugs in db_stress fault injection (#6693) Summary: Fix the following issues - 1. Output parsing error in db_crashtest.py 2. Memory leak on exit 3. False alarm on filter block read error Pull Request resolved: https://github.com/facebook/rocksdb/pull/6693 Test Plan: asan_crash Reviewed By: cheng-chang Differential Revision: D20990399 Pulled By: anand1976 fbshipit-source-id: 178ee0dd7c69a4bc5db698379db0dedb29281699 13 April 2020, 18:01:03 UTC
back to top