Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
02db03a Andrew Kryczka 09 June 2020, 23:49:07 UTC make L0 index/filter pinned memory usage predictable (#6911) Summary: Memory pinned by `pin_l0_filter_and_index_blocks_in_cache` needs to be predictable based on user config. This PR makes sure we do not pin extra memory for large files generated by intra-L0 (see https://github.com/facebook/rocksdb/issues/6889). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6911 Test Plan: unit test Reviewed By: siying Differential Revision: D21835818 Pulled By: ajkr fbshipit-source-id: a11a088549d06bed8aacc2548d266e5983f0ead4 09 June 2020, 23:51:23 UTC
5abda3b Levi Tamasi 09 June 2020, 22:12:59 UTC Move blob_log_{format,reader,writer}.{cc,h} to db/blob/ (#6960) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6960 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D21958416 Pulled By: ltamasi fbshipit-source-id: 97cf05027b7363014b07836e7f158c23827bd661 09 June 2020, 22:16:05 UTC
edf74d1 Peter Dillinger 09 June 2020, 17:01:12 UTC Add --version and --help to ldb and sst_dump (#6951) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/6951 Test Plan: tests included + manual Reviewed By: ajkr Differential Revision: D21918540 Pulled By: pdillinger fbshipit-source-id: 79d4991f2a831214fc7e477a839ec19dbbace6c5 09 June 2020, 17:04:01 UTC
6a8ddd3 sdong 09 June 2020, 02:32:18 UTC Introduce some Linux build to CircleCI (#6937) Summary: Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ ed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937 Test Plan: Watch CI build results. Reviewed By: pdillinger Differential Revision: D21941605 fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad 09 June 2020, 02:34:31 UTC
1fb3593 anand76 08 June 2020, 23:08:31 UTC Fix a bug in looking up duplicate keys with MultiGet (#6953) Summary: When MultiGet is called with duplicate keys, and the key matches the largest key in an SST file and the value type is merge, only the first instance of the duplicate key is returned with correct results. This is due to the incorrect assumption that if a key in a batch is equal to the largest key in the file, the next key cannot be present in that file. Tests: Add a new unit test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6953 Reviewed By: cheng-chang Differential Revision: D21935898 Pulled By: anand1976 fbshipit-source-id: a2cc327a15150e23fd997546ca64d1c33021cb4c 08 June 2020, 23:11:21 UTC
f5e6494 Levi Tamasi 08 June 2020, 22:59:25 UTC Add convenience method GetFileMetaDataByNumber (#6940) Summary: The patch adds a convenience method `GetFileMetaDataByNumber` that builds on the `FileLocation` functionality introduced recently (see https://github.com/facebook/rocksdb/pull/6862). This method makes it possible to retrieve the `FileMetaData` directly as opposed to having to go through `LevelFiles` and friends. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6940 Test Plan: `make check` Reviewed By: cheng-chang Differential Revision: D21905946 Pulled By: ltamasi fbshipit-source-id: af99e19de21242b2b4a87594a535c6028d16ee72 08 June 2020, 23:01:59 UTC
119b26f Zitan Chen 08 June 2020, 20:56:22 UTC Implement a new subcommand "identify" for sst_dump (#6943) Summary: Implemented a subcommand of sst_dump called identify, which determines whether a file is an SST file or identifies and lists all the SST files in a directory; This update also fixes the problem that sst_dump exits with a success state even if target file/directory does not exist/is not an SST file/is empty/is corrupted. One test is added to sst_dump_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6943 Test Plan: Passed make check and a few manual tests Reviewed By: pdillinger Differential Revision: D21928985 Pulled By: gg814 fbshipit-source-id: 9a8b48e0cf1a0e96b13f42b690aba8ad981afad3 08 June 2020, 20:58:28 UTC
fb08330 Zhichao Cao 08 June 2020, 17:34:47 UTC decouple the dependency of trace_analyzer_test unit test (#6941) Summary: Since gflags use the global variable to store the flags passed in. In the unit test, if we git one flag per unit test, the result is that all the flags are combined together in the following tests. Therefore, it has the dependency. In this PR, we pass the full arguments each time to ensure that the old arguments will be overwritten by the new one such that the dependency is removed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6941 Test Plan: make asan_check. run each unit test in trace_analyzer_test independently and in arbitrary orders. Reviewed By: pdillinger Differential Revision: D21909176 Pulled By: zhichao-cao fbshipit-source-id: dca550a0a4a205c30faa620e258a020a3b5b4e13 08 June 2020, 17:36:43 UTC
3020df9 Yanqin Jin 08 June 2020, 04:54:54 UTC Remove unnecessary inclusion of version_edit.h in env (#6952) Summary: In db_options.c, we should avoid including header files in the `db` directory to avoid introducing unnecessary dependency. The reason why `version_edit.h` has been included in `db_options.cc` is because we need two constants, `kUnknownChecksum` and `kUnknownChecksumFuncName`. We can put these two constants as `constexpr` in the public header `file_checksum.h`. Test plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6952 Reviewed By: zhichao-cao Differential Revision: D21925341 Pulled By: riversand963 fbshipit-source-id: 2902f3b74c97f0cf16c58ad24c095c787c3a40e2 08 June 2020, 04:56:55 UTC
f8c2e5a Levi Tamasi 06 June 2020, 03:08:51 UTC Do not print messages to stderr in VersionBuilder (#6948) Summary: RocksDB is an embedded library; we should not write to the application's console. Note: in each case, the same information is returned in the form of a `Status::Corruption` object. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6948 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D21914965 Pulled By: ltamasi fbshipit-source-id: ae4b66789aa6b659eb8cc2ed4a048187962c86cc 06 June 2020, 03:10:30 UTC
8988f83 Levi Tamasi 06 June 2020, 01:14:10 UTC Fix up a VersionBuilder test case (#6942) Summary: We currently do not have any validation that would ensure that the `FileMetaData` objects are equivalent when a file gets deleted from the LSM tree and then re-added (think trivial moves); however, if we did, this test case would be in violation. The patch changes the values used in the test case so they are consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6942 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D21911366 Pulled By: ltamasi fbshipit-source-id: 2f0486f8337373a6a111b6f28433d70507857104 06 June 2020, 01:16:38 UTC
f941ade Zhichao Cao 06 June 2020, 00:43:31 UTC Clean up the dead code (#6946) Summary: Remove the dead code in table test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6946 Test Plan: run table_test Reviewed By: riversand963 Differential Revision: D21913563 Pulled By: zhichao-cao fbshipit-source-id: c0aa9f3b95dfe87dd7fb2cd4823784f08cb3ddd3 06 June 2020, 00:45:22 UTC
23e446a Zitan Chen 06 June 2020, 00:24:43 UTC Disable OpenForReadOnly tests in the LITE mode (#6947) Summary: Disable two OpenForReadOnly tests in the LITE mode Pull Request resolved: https://github.com/facebook/rocksdb/pull/6947 Test Plan: passed db_test2 Reviewed By: cheng-chang Differential Revision: D21914345 Pulled By: gg814 fbshipit-source-id: 58e81baf5d8cf8adcedaef3966aa3a427bbdf7c2 06 June 2020, 00:28:24 UTC
2e7070b sdong 05 June 2020, 19:14:42 UTC Directly use unit test tempalte buck (#6926) Summary: Make RocksDB run a predefined unit test so that it can be integrated with better tools. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6926 Test Plan: Watch tests Reviewed By: pdillinger Differential Revision: D21866216 fbshipit-source-id: cafca82efdf0b72671be8d30b665e88a75ae6000 05 June 2020, 19:16:33 UTC
98b0cbe anand76 05 June 2020, 18:06:26 UTC Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909) Summary: The ```for``` loop in ```VerifyChecksumInBlocks``` only checks ```index_iter->Valid()``` which could be ```false``` either due to reaching the end of the index or, in case of partitioned index, it could be due to a checksum mismatch error when reading a 2nd level index block. Instead of throwing away the index iterator status, we need to return any errors back to the caller. Tests: Add a test in block_based_table_reader_test.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6909 Reviewed By: pdillinger Differential Revision: D21833922 Pulled By: anand1976 fbshipit-source-id: bc778ebf1121dbbdd768689de5183f07a9f0beae 05 June 2020, 18:08:25 UTC
1bee0fc Cheng Chang 05 June 2020, 17:53:08 UTC Make DestroyDir destroy directories recursively (#6934) Summary: Currently, `DeleteDir` only deletes the directory if there are no other directories under the target dir. This PR makes it delete directories recursively. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6934 Test Plan: Added a new unit test in testutil_test.cc. `make testutil_test` Reviewed By: zhichao-cao Differential Revision: D21884211 Pulled By: cheng-chang fbshipit-source-id: 0b9a48a200f494ee007aef5d1763b4aa331f8b5a 05 June 2020, 17:55:22 UTC
2677bd5 Akanksha Mahajan 05 June 2020, 16:41:03 UTC Add logs and stats in DeleteScheduler (#6927) Summary: Add logs and stats for files marked as trash and files deleted immediately in DeleteScheduler Pull Request resolved: https://github.com/facebook/rocksdb/pull/6927 Test Plan: make check -j64 Reviewed By: riversand963 Differential Revision: D21869068 Pulled By: akankshamahajan15 fbshipit-source-id: e9f673c4fa8049ce648b23c75d742f2f9c6c57a1 05 June 2020, 16:43:04 UTC
aaece2a Peter Dillinger 04 June 2020, 22:44:05 UTC Fix some defects reported by Coverity Scan (#6933) Summary: Confusing checks for null that are never null Pull Request resolved: https://github.com/facebook/rocksdb/pull/6933 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D21885466 Pulled By: pdillinger fbshipit-source-id: 4b48e03c2a33727f2702b0d12292f9fda5a3c475 04 June 2020, 22:46:27 UTC
c7432cc Peter Dillinger 04 June 2020, 22:32:29 UTC Fix more defects reported by Coverity Scan (#6935) Summary: Mostly uninitialized values: some probably written before use, but some seem like bugs. Also, destructor needs to be virtual, and possible use-after-free in test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6935 Test Plan: make check Reviewed By: siying Differential Revision: D21885484 Pulled By: pdillinger fbshipit-source-id: e2e7cb0a0cf196f2b55edd16f0634e81f6cc8e08 04 June 2020, 22:35:08 UTC
a8170d7 Yanqin Jin 04 June 2020, 21:19:05 UTC Close file to avoid file-descriptor leakage (#6936) Summary: When operation on an open file descriptor fails, we should close the file descriptor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6936 Test Plan: make check Reviewed By: pdillinger Differential Revision: D21885458 Pulled By: riversand963 fbshipit-source-id: ba077a76b256a8537f21e22e4ec198f45390bf50 04 June 2020, 21:21:15 UTC
6cbe9d9 sdong 04 June 2020, 21:14:51 UTC Make StringAppendOperatorTest a parameterized test (#6930) Summary: StringAppendOperatorTest right now runs in a mode where RUN_ALL_TESTS() is executed twice for the same test but different settings. This creates a problem with a tool that expects every test to run once. Fix it by using a parameterized test instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6930 Test Plan: Run the test and see it passed. Reviewed By: ltamasi Differential Revision: D21874145 fbshipit-source-id: 55520b2d7f1ba9f3cba1e2d087fe86f43fb06145 04 June 2020, 21:17:11 UTC
31bd2d7 sdong 04 June 2020, 18:41:21 UTC Fix ThreadLocalTest.SequentialReadWriteTest failure when running individually (#6929) Summary: When running ThreadLocalTest.SequentialReadWriteTest individually, the test fails with: ] ./thread_local_test --gtest_filter="*SequentialReadWriteTest*" Note: Google Test filter = *SequentialReadWriteTest* [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from ThreadLocalTest [ RUN ] ThreadLocalTest.SequentialReadWriteTest internal_repo_rocksdb/repo/util/thread_local_test.cc:144: Failure Expected: IDChecker::PeekId() Which is: 3 To be equal to: base_id + 1u Which is: 2 [ FAILED ] ThreadLocalTest.SequentialReadWriteTest (1 ms) [----------] 1 test from ThreadLocalTest (1 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] ThreadLocalTest.SequentialReadWriteTest 1 FAILED TEST It appears that when running as the first test, PeakId() was updated twice. I didn't dig into it why but it doesn't seem to break the contract. Relax the assertion to make it pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6929 Test Plan: Run the test individually and as the whole thread_local_test Reviewed By: riversand963 Differential Revision: D21873999 fbshipit-source-id: 1dcb6a2e9c38b6afd848027308bfe633342b7548 04 June 2020, 18:44:09 UTC
e85cbdb mrambacher 04 June 2020, 18:38:34 UTC Fix two core dumps when files are missing (#6922) Summary: The LDB create and drop column family commands failed to check if theere was a valid database prior to dereferencing it, leading to a core dump. The SstFileDumper prefetch code would dereference a file when the file did not exist as part of the Prefetch code. This dereference was moved inside an st.ok() check. Tests were added for both failure conditions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6922 Reviewed By: gg814 Differential Revision: D21884024 Pulled By: pdillinger fbshipit-source-id: bddd45c299aa9dc7e928c17a37a96521f8c9149e 04 June 2020, 18:40:32 UTC
0b45a68 sdong 04 June 2020, 16:49:44 UTC env_test */RunMany/* tests to run individually (#6931) Summary: When run */RunMany/* tests individually, e.g. ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0, they hang. It's because they insert to background thread pool without initializing them. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6931 Test Plan: Run ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0 by itself and see it passes. Reviewed By: riversand963 Differential Revision: D21875603 fbshipit-source-id: 7f848174c1a660254a2b1f7e11cca5370793ba30 04 June 2020, 16:51:38 UTC
2f32618 Yanqin Jin 04 June 2020, 15:28:44 UTC Fix a typo (bug) when setting error during Flush (#6928) Summary: As title. The prior change to the line is a typo. Fixing it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6928 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D21873587 Pulled By: riversand963 fbshipit-source-id: f4837fc8792d7106bc230b7b499dfbb7a2847430 04 June 2020, 15:30:42 UTC
02df00d Zitan Chen 04 June 2020, 01:55:25 UTC API change: DB::OpenForReadOnly will not write to the file system unless create_if_missing is true (#6900) Summary: DB::OpenForReadOnly will not write anything to the file system (i.e., create directories or files for the DB) unless create_if_missing is true. This change also fixes some subcommands of ldb, which write to the file system even if the purpose is for readonly. Two tests for this updated behavior of DB::OpenForReadOnly are also added. Other minor changes: 1. Updated HISTORY.md to include this API change of DB::OpenForReadOnly; 2. Updated the help information for the put and batchput subcommands of ldb with the option [--create_if_missing]; 3. Updated the comment of Env::DeleteDir to emphasize that it returns OK only if the directory to be deleted is empty. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6900 Test Plan: passed make check; also manually tested a few ldb subcommands Reviewed By: pdillinger Differential Revision: D21822188 Pulled By: gg814 fbshipit-source-id: 604cc0f0d0326a937ee25a32cdc2b512f9a3be6e 04 June 2020, 01:57:49 UTC
055b4d2 sdong 04 June 2020, 01:18:38 UTC Make sure core components not depend on gtest (#6921) Summary: We recently removed the dependencies of core components on gtest. Add a Travis test to make sure it doesn't regress. Change cmake setting so that the gtest related components are only included when tests, benchmarks or stress tools are included in the build. Add this build setting in Travis to confirm it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6921 Test Plan: See Travis passes Reviewed By: ajkr Differential Revision: D21863564 fbshipit-source-id: df26f50a8305a04ff19ffa8069a1857ecee10289 04 June 2020, 01:22:14 UTC
b7c825d Stanislav Tkach 04 June 2020, 00:06:44 UTC Add (some) getters for options to the C API (#6925) Summary: Additionally I have extended the incomplete test added in the https://github.com/facebook/rocksdb/issues/6880. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6925 Reviewed By: ajkr Differential Revision: D21869788 Pulled By: pdillinger fbshipit-source-id: e9db80f259c57ca1bdcbc2c66cb938cb1ac26e48 04 June 2020, 00:08:50 UTC
afa3518 sdong 03 June 2020, 22:53:09 UTC Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) Summary: This reverts commit 8d87e9cea13b09e12d0d22d4fdebdd5bacbb5c6e. Based on offline discussions, it's too early to upgrade to gtest 1.10, as it prevents some developers from using an older version of gtest to integrate to some other systems. Revert it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6923 Reviewed By: pdillinger Differential Revision: D21864799 fbshipit-source-id: d0726b1ff649fc911b9378f1763316200bd363fc 03 June 2020, 22:55:03 UTC
0f85d16 Hans Holmberg 03 June 2020, 21:55:43 UTC Route GetTestDirectory to FileSystem in CompositeEnvWrappers (#6896) Summary: GetTestDirectory implies a file system operation (it creates the default test directory if missing), so it should be routed to the FileSystem rather than the Env. Also remove the GetTestDirectory implementation in the PosixEnv, since it overrides GetTestDirectory in CompositeEnv making it impossible to override with a custom FileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6896 Reviewed By: cheng-chang Differential Revision: D21868984 Pulled By: ajkr fbshipit-source-id: e79bfef758d06dacef727c54b96abe62e78726fd 03 June 2020, 21:57:46 UTC
f005dac hfrt456 03 June 2020, 20:48:06 UTC fix IsDirectory function in env_hdfs.cc (#6917) Summary: fix IsDirectory function for hdfsEnv Pull Request resolved: https://github.com/facebook/rocksdb/pull/6917 Reviewed By: cheng-chang Differential Revision: D21865020 Pulled By: riversand963 fbshipit-source-id: ad69ed564d027b7bbdf4c693dd57cd02622fb3f8 03 June 2020, 20:50:17 UTC
0b8c549 Levi Tamasi 03 June 2020, 20:37:47 UTC Mention the consistency check improvement in HISTORY.md (#6924) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6924 Reviewed By: cheng-chang Differential Revision: D21865662 Pulled By: ltamasi fbshipit-source-id: 83a01bcbb779cfba941154a36a9e735293a93211 03 June 2020, 20:40:41 UTC
ffe08ff Hao Chen 03 June 2020, 19:23:47 UTC correct level information in version_set.cc (#6920) Summary: fix these two issues https://github.com/facebook/rocksdb/issues/6912 and https://github.com/facebook/rocksdb/issues/6667 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6920 Reviewed By: cheng-chang Differential Revision: D21864885 Pulled By: ajkr fbshipit-source-id: 10e21fc1851b67a59d44358f59c64fa5523bd263 03 June 2020, 19:27:13 UTC
22e5c51 Anatoly Zhmur 03 June 2020, 19:22:29 UTC Add zstd_max_train_bytes to c interop (#6796) Summary: Added setting of zstd_max_train_bytes compression option parameter to c interop. rocksdb_options_set_bottommost_compression_options was using bool parameter and thus not exported, updated it to unsigned char and added to c.h as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6796 Reviewed By: cheng-chang Differential Revision: D21611471 Pulled By: ajkr fbshipit-source-id: caaaf153de934837ad9af283c7f8c025ff0b0cf5 03 June 2020, 19:27:12 UTC
0a17d95 mrambacher 03 June 2020, 19:19:54 UTC Add OptionTypeInfo::Vector to parse/serialize vectors (#6424) Summary: The OptionTypeInfo::Vector method allows a vector<T> to be converted to/from strings via the options. The kVectorInt and kVectorCompressionType vectors were replaced with this methodology. As part of this change, the NextToken method was added to the OptionTypeInfo. This method was refactored from code within the StringToMap function. Future types that could use this functionality include the EventListener vectors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6424 Reviewed By: cheng-chang Differential Revision: D21832368 Pulled By: zhichao-cao fbshipit-source-id: e1ca766faff139d54e6e8407a9ec09ece6517439 03 June 2020, 19:23:07 UTC
172adce Lucian Petrut 03 June 2020, 18:59:37 UTC Posix threads (#6865) Summary: Rocksdb is using the c++11 std::threads feature. The issue is that MINGW only supports it when using Posix threads. This change will allow rocksdb::port::WindowsThread to be replaced with std::thread, which in turn will allow Rocksdb to be cross compiled using MINGW. At the same time, we'll have to use GetCurrentProcessId instead of _getpid. Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6865 Reviewed By: cheng-chang Differential Revision: D21864285 Pulled By: ajkr fbshipit-source-id: 0982eed313e7d34d351b1364c1ccc722da473205 03 June 2020, 19:01:57 UTC
43f8a9d Peter Dillinger 03 June 2020, 18:37:12 UTC Some fixes for gcc 4.8 and add to Travis (#6915) Summary: People keep breaking the gcc 4.8 compilation due to different warnings for shadowing member functions with locals. Adding to Travis to keep compatibility. (gcc 4.8 is default on CentOS 7.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6915 Test Plan: local and Travis Reviewed By: siying Differential Revision: D21842894 Pulled By: pdillinger fbshipit-source-id: bdcd4385127ee5d1cc222d87e53fb3695c32a9d4 03 June 2020, 18:39:25 UTC
78e291b Levi Tamasi 03 June 2020, 18:21:16 UTC Improve consistency checks in VersionBuilder (#6901) Summary: The patch cleans up the code and improves the consistency checks around adding/deleting table files in `VersionBuilder`. Namely, it makes the checks stricter and improves them in the following ways: 1) A table file can now only be deleted from the LSM tree using the level it resides on. Earlier, there was some unnecessary wiggle room for trivially moved files (they could be deleted using a lower level number than the actual one). 2) A table file cannot be added to the tree if it is already present in the tree on any level (not just the target level). The earlier code only had an assertion (which is a no-op in release builds) that the newly added file is not already present on the target level. 3) The above consistency checks around state transitions are now mandatory, as opposed to the earlier `CheckConsistencyForDeletes`, which was a no-op in release mode unless `force_consistency_checks` was set to `true`. The rationale here is that assuming that the initial state is consistent, a valid transition leads to a next state that is also consistent; however, an *invalid* transition offers no such guarantee. Hence it makes sense to validate the transitions unconditionally, and save `force_consistency_checks` for the paranoid checks that re-validate the entire state. 4) The new checks build on the mechanism introduced in https://github.com/facebook/rocksdb/pull/6862, which enables us to efficiently look up the location (level and position within level) of files in a `Version` by file number. This makes the consistency checks much more efficient than the earlier `CheckConsistencyForDeletes`, which essentially performed a linear search. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6901 Test Plan: Extended the unit tests and ran: `make check` `make whitebox_crash_test` Reviewed By: ajkr Differential Revision: D21822714 Pulled By: ltamasi fbshipit-source-id: e2b29c8b6da1bf0f59004acc889e4870b2d18215 03 June 2020, 18:24:55 UTC
9360776 Peter Dillinger 03 June 2020, 17:40:13 UTC Fix handling of too-small filter partition size (#6905) Summary: Because ARM and some other platforms have a larger cache line size, they have a larger minimum filter size, which causes recently added PartitionedMultiGet test in db_bloom_filter_test to fail on those platforms. The code would actually end up using larger partitions, because keys_per_partition_ would be 0 and never == number of keys added. The code now attempts to get as close as possible to the small target size, while fully utilizing that filter size, if the target partition size is smaller than the minimum filter size. Also updated the test to break more uniformly across platforms Pull Request resolved: https://github.com/facebook/rocksdb/pull/6905 Test Plan: updated test, tested on ARM Reviewed By: anand1976 Differential Revision: D21840639 Pulled By: pdillinger fbshipit-source-id: 11684b6d35f43d2e98b85ddb2c8dcfd59d670817 03 June 2020, 17:43:01 UTC
2adb7e3 Zhichao Cao 02 June 2020, 22:02:44 UTC Fix potential overflow of unsigned type in for loop (#6902) Summary: x.size() -1 or y - 1 can overflow to an extremely large value when x.size() pr y is 0 when they are unsigned type. The end condition of i in the for loop will be extremely large, potentially causes segment fault. Fix them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6902 Test Plan: pass make asan_check Reviewed By: ajkr Differential Revision: D21843767 Pulled By: zhichao-cao fbshipit-source-id: 5b8b88155ac5a93d86246d832e89905a783bb5a1 02 June 2020, 22:05:07 UTC
556972e Zhichao Cao 02 June 2020, 22:02:25 UTC Replace Status with IOStatus in CopyFile and CreateFile (#6916) Summary: Replace Status with IOStatus in CopyFile and CreateFile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6916 Test Plan: pass make asan_check Reviewed By: cheng-chang Differential Revision: D21843775 Pulled By: zhichao-cao fbshipit-source-id: 524d4a0fcf47f0941b923da0346e0de71607f5f6 02 June 2020, 22:05:06 UTC
bfc9737 sdong 02 June 2020, 20:54:03 UTC Remove gtest dependency in non-test code under utilities/cassandra (#6908) Summary: production code under utilities/cassandra depends on gtest.h. Remove them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6908 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21842606 fbshipit-source-id: a098e0b49c9aeac51cc90a79562ad9897a36122c 02 June 2020, 20:56:29 UTC
38f988d Stanislav Tkach 02 June 2020, 20:44:53 UTC Expose rocksdb_options_copy function to the C API (#6880) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6880 Reviewed By: ajkr Differential Revision: D21842752 Pulled By: pdillinger fbshipit-source-id: eda326f551ddd9cb397681544b9e9799ea614e52 02 June 2020, 20:48:51 UTC
14eca6b Peter Dillinger 02 June 2020, 19:27:59 UTC For ApproximateSizes, pro-rate table metadata size over data blocks (#6784) Summary: The implementation of GetApproximateSizes was inconsistent in its treatment of the size of non-data blocks of SST files, sometimes including and sometimes now. This was at its worst with large portion of table file used by filters and querying a small range that crossed a table boundary: the size estimate would include large filter size. It's conceivable that someone might want only to know the size in terms of data blocks, but I believe that's unlikely enough to ignore for now. Similarly, there's no evidence the internal function AppoximateOffsetOf is used for anything other than a one-sided ApproximateSize, so I intend to refactor to remove redundancy in a follow-up commit. So to fix this, GetApproximateSizes (and implementation details ApproximateSize and ApproximateOffsetOf) now consistently include in their returned sizes a portion of table file metadata (incl filters and indexes) based on the size portion of the data blocks in range. In other words, if a key range covers data blocks that are X% by size of all the table's data blocks, returned approximate size is X% of the total file size. It would technically be more accurate to attribute metadata based on number of keys, but that's not computationally efficient with data available and rarely a meaningful difference. Also includes miscellaneous comment improvements / clarifications. Also included is a new approximatesizerandom benchmark for db_bench. No significant performance difference seen with this change, whether ~700 ops/sec with cache_index_and_filter_blocks and small cache or ~150k ops/sec without cache_index_and_filter_blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6784 Test Plan: Test added to DBTest.ApproximateSizesFilesWithErrorMargin. Old code running new test... [ RUN ] DBTest.ApproximateSizesFilesWithErrorMargin db/db_test.cc:1562: Failure Expected: (size) <= (11 * 100), actual: 9478 vs 1100 Other tests updated to reflect consistent accounting of metadata. Reviewed By: siying Differential Revision: D21334706 Pulled By: pdillinger fbshipit-source-id: 6f86870e45213334fedbe9c73b4ebb1d8d611185 02 June 2020, 19:30:23 UTC
298b00a sdong 02 June 2020, 19:09:48 UTC Reduce dependency on gtest dependency in release code (#6907) Summary: Release code now depends on gtest, indirectly through including "test_util/testharness.h". This creates multiple problems. One important reason is the definition of IGNORE_STATUS_IF_ERROR() in test_util/testharness.h. Move it to sync_point.h instead. Note that utilities/cassandra/format.h still depends on "test_util/testharness.h". This will be resolved in a separate diff. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6907 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21829884 fbshipit-source-id: 9253c19ffde2936f3ae68998210f8e54f645a6e6 02 June 2020, 19:11:24 UTC
8d87e9c Adam Retter 02 June 2020, 03:31:29 UTC Update googletest from 1.8.1 to 1.10.0 (#6808) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6808 Reviewed By: anand1976 Differential Revision: D21483984 Pulled By: pdillinger fbshipit-source-id: 70c5eff2bd54ddba469761d95e4cd4611fb8e598 02 June 2020, 03:33:42 UTC
66942e8 anand76 02 June 2020, 02:41:58 UTC Avoid unnecessary reads of uncompression dictionary in MultiGet (#6906) Summary: We may sometimes read the uncompression dictionary when its not necessary, when we lookup a key in an SST file but the index indicates the key is not present. This can happen with index_type 3. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6906 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D21828944 Pulled By: anand1976 fbshipit-source-id: 7aef4f0a39548d0874eafefd2687006d2652f9bb 02 June 2020, 02:43:37 UTC
02f59ed sdong 01 June 2020, 23:30:45 UTC Find the correct gcov (#6904) Summary: Right now in FB environment, wrong gcov is used. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6904 Test Plan: "make coverage" and watch results. Reviewed By: riversand963 Differential Revision: D21824291 fbshipit-source-id: 666011fd86c36adafa09ebd9eb97742f94fb90bb 01 June 2020, 23:33:05 UTC
bcb9e41 Cheng Chang 01 June 2020, 22:17:58 UTC Explicitly free allocated buffer when status is not ok (#6903) Summary: Currently we rely on `BlockContents` to implicitly free the allocated scratch buffer, but when IO error happens, it doesn't make sense to construct the `BlockContents` which might be corrupted. In the stress test, we find that `assert(req.result.size() == block_size(handle));` fails because of potential IO errors. In this PR, we explicitly free the scratch buffer on error without constructing `BlockContents`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6903 Test Plan: watch stress test Reviewed By: anand1976 Differential Revision: D21823869 Pulled By: cheng-chang fbshipit-source-id: 5603fc80e9bf3f44a9d7250ddebd871afe1eb89f 01 June 2020, 22:19:40 UTC
038e02d zitan 01 June 2020, 19:13:36 UTC Remove extraneous newline from ldb stderr (#6897) Summary: **Summary** Remove the extraneous newline when using ldb tool. For example, the subcommand list_column_families will print an empty line to stderr even if there are no errors. **Test plan** Passed make check; manually tested a few ldb subcommands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6897 Reviewed By: pdillinger Differential Revision: D21819352 Pulled By: gg814 fbshipit-source-id: 5a16a6431bb96684fe97647f4d3ac5bf0ec7fc90 01 June 2020, 19:15:36 UTC
0c56fc4 Peter Dillinger 29 May 2020, 18:24:19 UTC Allow missing "unversioned" python, as in CentOS 8 (#6883) Summary: RocksDB Makefile was assuming existence of 'python' command, which is not present in CentOS 8. We avoid using 'python' if 'python3' is available. Also added fancy logic to format-diff.sh to make clang-format-diff.py for Python2 work even with Python3 only (as some CentOS 8 FB machines come equipped) Also, now use just 'python3' for PYTHON if not found so that an informative "command not found" error will result rather than something weird. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6883 Test Plan: manually tried some variants, 'make check' on a fresh CentOS 8 machine without 'python' executable or Python2 but with clang-format-diff.py for Python2. Reviewed By: gg814 Differential Revision: D21767029 Pulled By: pdillinger fbshipit-source-id: 54761b376b140a3922407bdc462f3572f461d0e9 29 May 2020, 18:29:23 UTC
c5abf78 Andrew Kryczka 28 May 2020, 17:49:02 UTC avoid `IterKey::UpdateInternalKey()` in `BlockIter` (#6843) Summary: `IterKey::UpdateInternalKey()` is an error-prone API as it's incompatible with `IterKey::TrimAppend()`, which is used for decoding delta-encoded internal keys. This PR stops using it in `BlockIter`. Instead, it assigns global seqno in a separate `IterKey`'s buffer when needed. The logic for safely getting a Slice with global seqno properly assigned is encapsulated in `GlobalSeqnoAppliedKey`. `BinarySeek()` is also migrated to use this API (previously it ignored global seqno entirely). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6843 Test Plan: benchmark setup -- single file DBs, in-memory, no compression. "normal_db" created by regular flush; "ingestion_db" created by ingesting a file. Both DBs have same contents. ``` $ TEST_TMPDIR=/dev/shm/normal_db/ ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=10485760000 -disable_auto_compactions=true -compression_type=none -num=1000000 $ ./ldb write_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ --compression_type=no --hex --create_if_missing < <(./sst_dump --command=scan --output_hex --file=/dev/shm/normal_db/dbbench/000007.sst | awk 'began {print "0x" substr($1, 2, length($1) - 2), "==>", "0x" $5} ; /^Sst file format: block-based/ {began=1}') $ ./ldb ingest_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ ``` benchmark run command: ``` TEST_TMPDIR=/dev/shm/$DB/ ./db_bench -benchmarks=seekrandom -seek_nexts=10 -use_existing_db=true -cache_index_and_filter_blocks=false -num=1000000 -cache_size=1048576000 -threads=1 -reads=40000000 ``` results: | DB | code | throughput | |---|---|---| | normal_db | master | 267.9 | | normal_db | PR6843 | 254.2 (-5.1%) | | ingestion_db | master | 259.6 | | ingestion_db | PR6843 | 250.5 (-3.5%) | Reviewed By: pdillinger Differential Revision: D21562604 Pulled By: ajkr fbshipit-source-id: 937596f836930515da8084d11755e1f247dcb264 28 May 2020, 17:51:30 UTC
961c759 Yanqin Jin 28 May 2020, 17:37:57 UTC Add timestamp to delete (#6253) Summary: Preliminary user-timestamp support for delete. If ["a", ts=100] exists, you can delete it by calling `DB::Delete(write_options, key)` in which `write_options.timestamp` points to a `ts` higher than 100. Implementation A new ValueType, i.e. `kTypeDeletionWithTimestamp` is added for deletion marker with timestamp. The reason for a separate `kTypeDeletionWithTimestamp`: RocksDB may drop tombstones (keys with kTypeDeletion) when compacting them to the bottom level. This is OK and useful if timestamp is disabled. When timestamp is enabled, should we still reuse `kTypeDeletion`, we may drop the tombstone with a more recent timestamp, causing deleted keys to re-appear. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6253 Reviewed By: ltamasi Differential Revision: D20995328 Pulled By: riversand963 fbshipit-source-id: a9e5c22968ad76f98e3dc6ee0151265a3f0df619 28 May 2020, 17:40:03 UTC
e3f953a Levi Tamasi 28 May 2020, 17:00:19 UTC Make it possible to look up files by number in VersionStorageInfo (#6862) Summary: Does what it says on the can: the patch adds a hash map to `VersionStorageInfo` that maps file numbers to file locations, i.e. (level, position in level) pairs. This will enable stricter consistency checks in `VersionBuilder`. The patch also fixes all the unit tests that used duplicate file numbers in a version (which would trigger an assertion with the new code). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6862 Test Plan: `make check` `make whitebox_crash_test` Reviewed By: riversand963 Differential Revision: D21670446 Pulled By: ltamasi fbshipit-source-id: 2eac249945cf33d8fb8597b26bfff5221e1a861a 28 May 2020, 17:03:06 UTC
bcefc59 Akanksha Mahajan 27 May 2020, 20:03:08 UTC Allow MultiGet users to limit cumulative value size (#6826) Summary: 1. Add a value_size in read options which limits the cumulative value size of keys read in batches. Once the size exceeds read_options.value_size, all the remaining keys are returned with status Abort without further fetching any key. 2. Add a unit test case MultiGetBatchedValueSizeSimple the reads keys from memory and sst files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6826 Test Plan: 1. make check -j64 2. Add a new unit test case Reviewed By: anand1976 Differential Revision: D21471483 Pulled By: akankshamahajan15 fbshipit-source-id: dea51b8e76d5d1df38ece8cdb29933b1d798b900 27 May 2020, 20:07:14 UTC
9060e6f Adam Retter 27 May 2020, 18:56:53 UTC Add newer WBWI::NewIteratorWithBase functions to RocksJava (#6872) Summary: Exposes the `ReadOptions` arguments to `NewIteratorWithBase`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6872 Reviewed By: ajkr Differential Revision: D21725867 Pulled By: pdillinger fbshipit-source-id: 4079ba590cc13ba7a6244ed91439d89c40a543b6 27 May 2020, 18:59:12 UTC
82a82c7 Cheng Chang 26 May 2020, 22:27:29 UTC Fix potential memory leak of scratch buffer (#6879) Summary: If `req.scratch` is an internally allocated buffer, but `raw_block_contents` is not constructed to own `req.scratch`, then `req.scratch` will be leaked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6879 Test Plan: make asan_check Reviewed By: anand1976 Differential Revision: D21728498 Pulled By: cheng-chang fbshipit-source-id: 8fc6a4f2543918c565ddc16ecfad1807eb9a42cf 26 May 2020, 22:29:04 UTC
bd68bfb Kefu Chai 26 May 2020, 05:50:17 UTC cmake: link env_librados_test against rados (#6855) Summary: otherwise we have FTBFS like: 2020-05-18T15:12:06.400 INFO:tasks.workunit.client.0.smithi032.stdout:[100%] Linking CXX executable env_librados_test 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: CMakeFiles/rocksdb_env_librados_test.dir/utilities/env_librados_test.cc.o: undefined reference to symbol '_ZN8librados7v14_2_05Rados4initEPKc@LIBRADOS_14.2.0' 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: /lib/librados.so.2: error adding symbols: DSO missing from command line 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:collect2: error: ld returned 1 exit status this addresses the regression introduced by 07204837ce8d66e1e6e4893178f3fd040f9c1044, which hides the symbols exposed by `${THIRDPARTY_LIBS}` from consumers of librocksdb Signed-off-by: Kefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6855 Reviewed By: riversand963 Differential Revision: D21621904 fbshipit-source-id: 7022ba4dc0003504401fce6f06547e4d74a32ac0 26 May 2020, 05:53:00 UTC
b464a85 Andrew Kryczka 24 May 2020, 22:25:43 UTC fix transaction rollback in db_stress TestMultiGet (#6873) Summary: There were further uses of `txn` after `RollbackTxn(txn)` leading to stress test errors. Moved the rollback to the end of the function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6873 Test Plan: found a command from the crash test that previously failed immediately under TSAN; verified now it succeeds. ``` ./db_stress --acquire_snapshot_one_in=10000 --allow_concurrent_memtable_write=1 --avoid_flush_during_recovery=0 --avoid_unnecessary_blocking_io=1 --block_size=16384 --bloom_bits=222.913637674 --bottommost_compression_type=none --cache_index_and_filter_blocks=1 --cache_size=1048576 --checkpoint_one_in=0 --checksum_type=kCRC32c --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_style=1 --compaction_ttl=0 --compression_max_dict_bytes=0 --compression_parallel_threads=1 --compression_type=zstd --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --db=/dev/shm/rocksdb/rocksdb_crashtest_whitebox --db_write_buffer_size=1048576 --delpercent=5 --delrangepercent=0 --destroy_db_initially=0 --disable_wal=0 --enable_pipelined_write=0 --flush_one_in=1000000 --format_version=5 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=12 --index_type=2 --key_len_percent_dist=1,30,69 --level_compaction_dynamic_level_bytes=True --log2_keys_per_lock=22 --long_running_snapshots=0 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=1000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --memtablerep=skip_list --mmap_read=1 --mock_direct_io=False --nooverwritepercent=1 --num_levels=1 --open_files=100 --ops_per_thread=200000 --partition_filters=1 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefixpercent=5 --progress_reports=0 --read_fault_one_in=0 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --snapshot_hold_ops=100000 --subcompactions=4 --sync=0 --sync_fault_injection=False --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=0 --txn_write_policy=1 --unordered_write=1 --use_block_based_filter=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=1 --use_merge=1 --use_multiget=1 --use_txn=1 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --write_buffer_size=4194304 --write_dbid_to_manifest=0 --writepercent=35 ``` Reviewed By: pdillinger Differential Revision: D21708338 Pulled By: ajkr fbshipit-source-id: dcf55cddee0a14f429a75e7a8a505acf8025f2b1 24 May 2020, 22:27:24 UTC
b559dba Andrew Kryczka 24 May 2020, 04:21:55 UTC pin image version in circle CI builds (#6876) Summary: somehow the windows-server-2019-vs2019 image changed in a way that made VS 14 2015 the default. This caused an error when we specify VS 16 2019 as the cmake generator. I could not figure out the right arguments/env vars to get the latest VS working so pinned the image to the previous version instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6876 Reviewed By: pdillinger Differential Revision: D21709679 Pulled By: ajkr fbshipit-source-id: 2d16819ad239b4611fa199547744e1c101dc9da0 24 May 2020, 04:23:27 UTC
803a517 Peter Dillinger 23 May 2020, 13:48:40 UTC Misc things for ASSERT_STATUS_CHECKED, also gcc 4.8.5 (#6871) Summary: * Print stack trace on status checked failure * Make folly_synchronization_distributed_mutex_test a parallel test * Disable ldb_test.py and rocksdb_dump_test.sh with ASSERT_STATUS_CHECKED (broken) * Fix shadow warning in random_access_file_reader.h reported by gcc 4.8.5 (ROCKSDB_NO_FBCODE), also https://github.com/facebook/rocksdb/issues/6866 * Work around compiler bug on max_align_t for gcc < 4.9 * Remove an apparently wrong comment in status.h * Use check_some in Travis config (for proper diagnostic output) * Fix ignored Status in loop in options_helper.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/6871 Test Plan: manual, CI Reviewed By: ajkr Differential Revision: D21706619 Pulled By: pdillinger fbshipit-source-id: daf6364173d6689904eb394461a69a11f5bee2cb 23 May 2020, 13:53:37 UTC
bcd3256 Marek Kurdej 22 May 2020, 18:16:43 UTC Fix warning -Wextra-semi. NFC. (#6869) Summary: Minor fix. CLA signed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6869 Reviewed By: ajkr Differential Revision: D21704001 Pulled By: pdillinger fbshipit-source-id: 57fd08114f3234f51f34758e25e708cc70962582 22 May 2020, 18:20:13 UTC
35a25a3 Peter Dillinger 22 May 2020, 18:15:44 UTC Fix/expand ASSERT_STATUS_CHECKED build, add to Travis (#6870) Summary: Fixed some option handling code that recently broke the ASSERT_STATUS_CHECKED build for options_test. Added all other existing tests that pass under ASSERT_STATUS_CHECKED to the whitelist. Added a Travis configuration to run all whitelisted tests with ASSERT_STATUS_CHECKED. (Someday we might enable this check by default in debug builds.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6870 Test Plan: ASSERT_STATUS_CHECKED=1 make check, Travis Reviewed By: ajkr Differential Revision: D21704374 Pulled By: pdillinger fbshipit-source-id: 15daef98136a19d7a6843fa0c9ec08738c2ac693 22 May 2020, 18:17:29 UTC
826295a mrambacher 21 May 2020, 21:46:09 UTC Change autovector to have a reserved size in LITE mode (#6868) Summary: Previously in LITE mode, an autovector did not have a reserved size. When elements were added to the vector, the underlying array could be reallocated. There was a set of code that never expands the autovector and was doing &autovector::back(). When the vector is resized, the old addresses may become invalid, causing a later exception to be thrown. By reserving space in the autovector up front, this problem is eliminated for those uses where the vector will never exceed the initial size. the resize happens, these pointers become invalid, leading to SEGV or other exceptions. This change allows the autovector to be fully populated before we take the address of any of its elements, thereby elminating the potential for a resize. There is comparable code to this change in Version::MultiGet for dealing with the context objects. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6868 Reviewed By: ajkr Differential Revision: D21693505 Pulled By: cheng-chang fbshipit-source-id: e71d516b15e08f202593cb80f2a42f048fc95768 21 May 2020, 21:48:10 UTC
292bcf6 Andrew Kryczka 21 May 2020, 20:55:18 UTC skip direct I/O tests in rocksdb lite (#6867) Summary: Fix a couple places where direct I/O was used even though it is unsupported in lite builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6867 Test Plan: `LITE=1 make check -j48` Reviewed By: pdillinger Differential Revision: D21689185 Pulled By: ajkr fbshipit-source-id: 3eaa3abf69cd7d0bcaabbcad3bb5a26fb8dd7301 21 May 2020, 20:57:17 UTC
38be686 mrambacher 21 May 2020, 17:56:40 UTC Add Struct Type to OptionsTypeInfo (#6425) Summary: Added code for generically handing structs to OptionTypeInfo. A struct is a collection of variables handled by their own map of OptionTypeInfos. Examples of structs include Compaction and Cache options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6425 Reviewed By: siying Differential Revision: D21668789 Pulled By: zhichao-cao fbshipit-source-id: 064b110de39dadf82361ed4663f7ac1a535b0b07 21 May 2020, 17:58:39 UTC
c7aedf1 Peter Dillinger 21 May 2020, 15:10:43 UTC Clean up some code related to file checksums (#6861) Summary: * Add missing unit test for schema stability of FileChecksumGenCrc32c (previously was only comparing to itself) * A lot of clarifying comments * Add some assertions for preconditions * Rename WritableFileWriter::CalculateFileChecksum -> UpdateFileChecksum * Simplify FileChecksumGenCrc32c with shared functions * Implement EndianSwapValue to replace unused EndianTransform And incidentally since I had trouble with 'make check-format' GitHub action disagreeing with local run, * Output full diagnostic information when 'make check-format' fails in CI Pull Request resolved: https://github.com/facebook/rocksdb/pull/6861 Test Plan: new unit test passes before & after other changes Reviewed By: zhichao-cao Differential Revision: D21667115 Pulled By: pdillinger fbshipit-source-id: 6a99970f87605aa024fa540c78cd519ff322c3e6 21 May 2020, 15:12:51 UTC
eb04bb8 anand76 20 May 2020, 21:45:03 UTC Fix a bug in crash_test_with_txn (#6860) Summary: In NoBatchedOpsStress::TestMultiGet, call txn->Get() when transactions are in use. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6860 Test Plan: make crash_test_with_txn Reviewed By: pdillinger Differential Revision: D21667249 Pulled By: anand1976 fbshipit-source-id: 194bd7b9630a8efc3ae29d85422a61214e9e200e 20 May 2020, 21:47:05 UTC
545e14b Zhichao Cao 20 May 2020, 18:53:49 UTC Generate file checksum in SstFileWriter (#6859) Summary: If Option.file_checksum_gen_factory is set, rocksdb generates the file checksum during flush and compaction based on the checksum generator created by the factory and store the checksum and function name in vstorage and Manifest. This PR enable file checksum generation in SstFileWrite and store the checksum and checksum function name in the ExternalSstFileInfo, such that application can use them for other purpose, for example, ingest the file checksum with files in IngestExternalFile(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6859 Test Plan: add unit test and pass make asan_check. Reviewed By: ajkr Differential Revision: D21656247 Pulled By: zhichao-cao fbshipit-source-id: 78a3570c76031d8832e3d2de3d6c79cdf2b675d0 20 May 2020, 18:55:31 UTC
aaafcb8 Peter Dillinger 20 May 2020, 18:35:28 UTC Use in-repo gtest in buck build (#6858) Summary: ... so that we have freedom to upgrade it (see https://github.com/facebook/rocksdb/issues/6808). As a side benefit, gtest will no longer be linked into main library in buck build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6858 Test Plan: fb internal build & link Reviewed By: riversand963 Differential Revision: D21652061 Pulled By: pdillinger fbshipit-source-id: 6018104af944debde576b5beda6c134e737acedb 20 May 2020, 18:37:45 UTC
a1523ef Akanksha Mahajan 20 May 2020, 02:20:29 UTC Status check enforcement for io_posix_test and options_settable_test (#6857) Summary: Added status check enforcement for io_posix_test and options_settable_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6857 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 check Reviewed By: ajkr Differential Revision: D21647904 Pulled By: akankshamahajan15 fbshipit-source-id: b7f2321eb6c141a88cd5e1270ecb7d58f00341af 20 May 2020, 02:22:28 UTC
39b2443 anand76 19 May 2020, 19:41:43 UTC Strengthen MultiGet correctness verification in NoBatchedOpsStress (#6849) Summary: Add MultiGet to VerifyDb and check consistency with Get in TestMultiGet. Test plan - make crash_test ASAN crash test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6849 Reviewed By: pdillinger Differential Revision: D21635011 Pulled By: anand1976 fbshipit-source-id: deb5a79d08fefd8d8010204f1f20b83adc92310e 19 May 2020, 19:47:22 UTC
1551f10 Levi Tamasi 19 May 2020, 16:57:49 UTC Refactor the blob file related logic in VersionBuilder (#6835) Summary: This patch is groundwork for an upcoming change to store the set of linked SSTs in `BlobFileMetaData`. With the current code, a new `BlobFileMetaData` object is created each time a `VersionEdit` touches a certain blob file. This is fine as long as these objects are lightweight and cheap to create; however, with the addition of the linked SST set, it would be very inefficient since the set would have to be copied over and over again. Note that this is the same kind of problem that `VersionBuilder` is solving w/r/t `Version`s and files, and we can apply the same solution; that is, we can accumulate the changes in a different mutable object, and apply the delta in one shot when the changes are committed. The patch does exactly that by adding a new `BlobFileMetaDataDelta` class to `VersionBuilder`. In addition, it turns the existing `GetBlobFileMetaData` helper into `IsBlobFileInVersion` (which is fine since that's the only thing the method's clients care about now), and adds a couple of helper methods that can create a `BlobFileMetaData` object from the `BlobFileMetaData` in the base (if applicable) and the delta when the `Version` is saved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6835 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21505187 Pulled By: ltamasi fbshipit-source-id: d81a48c5f2ca7b79d7124c935332a6bcf3d5d988 19 May 2020, 17:00:04 UTC
0ac0098 mrambacher 19 May 2020, 16:20:21 UTC Make options length longer for sst_dump_test (#6846) Summary: Under MacOS when running with make -j 8 check, the temporary directory generated was > 100 characters. This caused the tests to do nothing under MacOS. Most of them still reported success for doing nothing, but ReadaheadSize was expecting the test to run. By making the option name longer, the tests will no run successfully (and do something!) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6846 Reviewed By: ajkr Differential Revision: D21576032 fbshipit-source-id: b089cde0d598137b572aa8527cc5459085252af7 19 May 2020, 16:22:12 UTC
ada700b Cheng Chang 19 May 2020, 00:23:22 UTC Re-read the whole request in direct IO mode when IO uring returns partial result (#6853) Summary: If both direct IO and IO uring are enabled, when IO uring returns partial result, we'll try to read the remaining part of the request, but the starting address/offset of the remaining part might not be aligned to the block size, in direct IO mode, the unaligned offset causes bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6853 Test Plan: run make check with both direct IO and IO uring enabled, this is covered by one of the continuous tests. Reviewed By: anand1976 Differential Revision: D21603023 Pulled By: cheng-chang fbshipit-source-id: 942f6a11ff21e1892af6c4464e02bab4c707787c 19 May 2020, 00:25:57 UTC
b9d65f5 Cheng Chang 18 May 2020, 15:39:33 UTC Trigger compaction in CompactOnDeletionCollector based on deletion ratio (#6806) Summary: In level compaction, if the total size (even if compensated after taking account of the deletions) of a level hasn't exceeded the limit, but there are lots of deletion entries in some SST files of the level, these files should also be good candidates for compaction. Otherwise, queries for the deleted keys might be slow because they need to go over all the tombstones. This PR adds an option `deletion_ratio` to the factory of `CompactOnDeletionCollector` to configure it to trigger compaction when the ratio of tombstones >= `deletion_ratio`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6806 Test Plan: Added new unit test in `compact_on_deletion_collector_test.cc`. make compact_on_deletion_collector_test && ./compact_on_deletion_collector_test Reviewed By: ajkr Differential Revision: D21511981 Pulled By: cheng-chang fbshipit-source-id: 65a9d0150e8c9c00337787686475252e4535a3e1 18 May 2020, 15:42:05 UTC
d790e60 Yanqin Jin 17 May 2020, 04:46:21 UTC Fix buck target db_stress_lib in opt mode (#6847) Summary: In buck build with opt mode, target should not include rocksdb_test_lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6847 Test Plan: Watch for internal cont build. Reviewed By: ajkr Differential Revision: D21586803 Pulled By: riversand963 fbshipit-source-id: 76d253c18d16fac6cab86a8c3f6b471ad5b6efb3 17 May 2020, 04:48:20 UTC
50ba245 Peter Dillinger 15 May 2020, 20:57:34 UTC Use 'make all' in LITE Travis configuration (#6834) Summary: So that we don't miss LITE compilation errors in tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/6834 Reviewed By: anand1976 Differential Revision: D21503227 Pulled By: pdillinger fbshipit-source-id: 3b2fdf3c4d395354d0ababac06da32addbafb3a5 15 May 2020, 20:59:24 UTC
91b7553 Cheng Chang 15 May 2020, 06:23:32 UTC Enable IO Uring in MultiGet in direct IO mode (#6815) Summary: Currently, in direct IO mode, `MultiGet` retrieves the data blocks one by one instead of in parallel, see `BlockBasedTable::RetrieveMultipleBlocks`. Since direct IO is supported in `RandomAccessFileReader::MultiRead` in https://github.com/facebook/rocksdb/pull/6446, this PR applies `MultiRead` to `MultiGet` so that the data blocks can be retrieved in parallel. Also, in direct IO mode and when data blocks are compressed and need to uncompressed, this PR only allocates one continuous aligned buffer to hold the data blocks, and then directly uncompress the blocks to insert into block cache, there is no longer intermediate copies to scratch buffers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6815 Test Plan: 1. added a new unit test `BlockBasedTableReaderTest::MultiGet`. 2. existing unit tests and stress tests contain tests against `MultiGet` in direct IO mode. Reviewed By: anand1976 Differential Revision: D21426347 Pulled By: cheng-chang fbshipit-source-id: b8446ae0e74152444ef9111e97f8e402ac31b24f 15 May 2020, 06:26:26 UTC
b11a8b1 Yanqin Jin 15 May 2020, 01:48:06 UTC Fix valgrind error by init memory region (#6842) Summary: As title. After allocating a memory buffer, initialize its content to 0s. This fixes valgrind issue introduced in https://github.com/facebook/rocksdb/issues/6709. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6842 Test Plan: ``` $make valgrind_test $valgrind --tool=memcheck --track-origins=yes ./db_test2 --gtest_filter=DBTest2.CompressionFailures ``` Reviewed By: pdillinger Differential Revision: D21551848 Pulled By: riversand963 fbshipit-source-id: e87a6f413e3f3d92d8e23d8ecc4cf93479c6674c 15 May 2020, 01:50:03 UTC
06a2dce Zhichao Cao 14 May 2020, 21:56:10 UTC Move checksum calculation ahead of memory copy (#6844) Summary: Originally, the checksum of appended data in writable file writer is calculated after the data is copied to the buffer. It will not be able to catch the bit flip happens during copy. Move the checksum calculation before it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6844 Test Plan: pass make asan_check Reviewed By: cheng-chang Differential Revision: D21576726 Pulled By: zhichao-cao fbshipit-source-id: 0a062a1f19886f6ea0d4e3f557e6f4b799773254 14 May 2020, 21:58:14 UTC
50d63a2 anand76 13 May 2020, 17:44:16 UTC Fix LITE build failure in compaction_picker_test (#6839) Summary: Fix LITE build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6839 Test Plan: LITE=1 make check Reviewed By: riversand963 Differential Revision: D21535808 Pulled By: anand1976 fbshipit-source-id: fcad961eca08e13fb0c256c92d18c3c1f1165f22 13 May 2020, 17:47:15 UTC
3bea276 Yanqin Jin 13 May 2020, 04:35:08 UTC Do not print u'string' in TARGETS file (#6841) Summary: Before this PR, extra deps passed in from cmd line to buckifier will be parsed and used to populate a dict. Using this dict and printing to TARGETS file will lead to printing u'', disallowed by build tools. This PR removes the u''. Test Plan (local dev server): ``` python buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}' ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841 Reviewed By: siying Differential Revision: D21538155 Pulled By: riversand963 fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349 13 May 2020, 04:37:31 UTC
244797a Tongliang Liao 13 May 2020, 04:16:50 UTC Add `find_dependency()` in cmake config file. (#6791) Summary: Currently when building PyTorch with latest RocksDB we get errors like "missing target Snappy::snappy", because they are simply not there. With old `${VAR}` approach we essentially hard-code the abs path found during RocksDB build, which is: - Not relocatable. - Doesn't work when changed to modern target-based design because that requires target to present when used for expansion. This fix allows cmake to setup imported target, if enabled during RocksDB build, when downstream uses `find_package(RocksDB)`. This is for https://github.com/facebook/rocksdb/issues/6179 tchaikov Please help review, thanks! Pull Request resolved: https://github.com/facebook/rocksdb/pull/6791 Reviewed By: riversand963 Differential Revision: D21471553 fbshipit-source-id: 8d4ff2ab589a97ca6e6ba27e1f17b97a00f06206 13 May 2020, 04:18:29 UTC
0720483 Tongliang Liao 13 May 2020, 04:05:50 UTC Mark dependencies as PRIVATE and fix missing dependencies in tools. (#6790) Summary: Tools were mistakenly using leaked (PUBLIC) dependencies from main lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6790 Reviewed By: riversand963 Differential Revision: D21471551 fbshipit-source-id: ec43b92e231777e0fcf0f865444391af09d6963b 13 May 2020, 04:07:55 UTC
4a4b8a1 sdong 13 May 2020, 01:21:32 UTC sst_dump to reduce number of file reads (#6836) Summary: sst_dump can issue many file reads from the file system. This doesn't work well with file systems without a OS cache, especially remote file systems. In order to mitigate this problem, several improvements are done: 1. --readahead_size is added, so that users can specify readahead size when scanning the data. 2. Force a 512KB tail readahead, which prevents three I/Os for footer, meta index and property blocks and hopefully index and filter blocks too. 3. Consoldiate SSTDump's I/Os before opening the file for read. Use the same file prefetch buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6836 Test Plan: Add a test that covers this new feature. Reviewed By: pdillinger Differential Revision: D21516607 fbshipit-source-id: 3ae43526286f67b2f4a5bdedfbc92719d579b87e 13 May 2020, 01:23:33 UTC
70aaa9c Stanislav Tkach 12 May 2020, 21:48:14 UTC Expose CancellAllBackgroundWork to C api (#6832) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6832 Reviewed By: cheng-chang Differential Revision: D21498186 Pulled By: riversand963 fbshipit-source-id: 66bb0d7c06af2bf0df3c6a09b61bca2fb81f2dd3 12 May 2020, 21:50:52 UTC
c384c08 Ziyue Yang 12 May 2020, 16:25:21 UTC Add tests for compression failure in BlockBasedTableBuilder (#6709) Summary: Currently there is no check for whether BlockBasedTableBuilder will expose compression error status if compression fails during the table building. This commit adds fake faulting compressors and a unit test to test such cases. This check finds 5 bugs, and this commit also fixes them: 1. Not handling compression failure well in BlockBasedTableBuilder::BGWorkWriteRawBlock. 2. verify_compression failing in BlockBasedTableBuilder when used with ZSTD. 3. Wrongly passing the same reference of block contents to BlockBasedTableBuilder::CompressAndVerifyBlock in parallel compression. 4. Wrongly setting block_rep->first_key_in_next_block to nullptr in BlockBasedTableBuilder::EnterUnbuffered when there are still incoming data blocks. 5. Not maintaining variables for compression ratio estimation and first_block in BlockBasedTableBuilder::EnterUnbuffered. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6709 Reviewed By: ajkr Differential Revision: D21236254 fbshipit-source-id: 101f6e62b2bac2b7be72be198adf93cd32a1ff46 12 May 2020, 16:27:35 UTC
3f21807 yetingsky 11 May 2020, 21:56:36 UTC fix typo (#6831) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6831 Reviewed By: cheng-chang Differential Revision: D21499149 Pulled By: zhichao-cao fbshipit-source-id: 2cb76cbf7086677d8cad5c828019e008062f0052 11 May 2020, 21:58:25 UTC
2e93247 Sagar Vemuri 11 May 2020, 20:28:16 UTC Disable a few timer tests (#6833) Summary: Disable `TimerTest.SingleScheduleRepeatedlyTest` and `TimerTest.MultipleScheduleRepeatedlyTest`. This is to help people to not hit any hangs (https://github.com/facebook/rocksdb/issues/6698) during their development process while I investigate further; I could not reproduce the issue on my dev machine yet. Note that timer is not being utilized anywhere yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6833 Test Plan: ``` svemuri@devbig187 ~/rocksdb (timer-disable-test) $ TEST_TMPDIR=/dev/shm ./timer_test [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from TimerTest [ RUN ] TimerTest.SingleScheduleOnceTest [ OK ] TimerTest.SingleScheduleOnceTest (1 ms) [ RUN ] TimerTest.MultipleScheduleOnceTest [ OK ] TimerTest.MultipleScheduleOnceTest (0 ms) [----------] 2 tests from TimerTest (1 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (1 ms total) [ PASSED ] 2 tests. YOU HAVE 2 DISABLED TESTS ``` Reviewed By: pdillinger Differential Revision: D21502474 Pulled By: sagar0 fbshipit-source-id: ac67caee2011fd14ffb2476a8914a6286a4f9abe 11 May 2020, 20:30:00 UTC
f0e8731 Tongliang Liao 11 May 2020, 17:26:18 UTC Use GFlags/Snappy config in CMake, with fallback for legacy approach. (#6771) Summary: Related to some discussion in https://github.com/facebook/rocksdb/issues/6179 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6771 Reviewed By: zhichao-cao Differential Revision: D21340117 fbshipit-source-id: a1af0ba4865bb13c8c817851d6f6c4056191b3fe 11 May 2020, 17:28:49 UTC
3a1c29d Derrick Pallas 09 May 2020, 03:48:17 UTC Add missing my_pid to fprintf in multi_process_example (#6731) Summary: Signed-off-by: Derrick Pallas <derrick@pallas.us> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6731 Reviewed By: siying Differential Revision: D21137005 Pulled By: riversand963 fbshipit-source-id: a7182e1bec225bc110971f40e2d0e6c3a671c061 09 May 2020, 03:49:33 UTC
a50ea71 sdong 08 May 2020, 21:12:18 UTC Improve ldb consistency checks (#6802) Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0 08 May 2020, 21:17:47 UTC
d9cd335 sdong 08 May 2020, 21:11:03 UTC Suppress UBSAN warning in CRC32 ARM (#6827) Summary: UBSAN shows following warning: util/crc32c_arm64.cc:111:11: runtime error: load of misaligned address 0x00001afcda86 for type 'const uint64_t', which requires 8 byte alignment 0x00001afcda86: note: pointer points here cc c1 2d 00 01 81 40 24 30 66 39 66 30 37 30 63 2d 32 36 63 34 2d 34 62 61 61 2d 38 35 33 31 2d ^ Suppress it just as what we do in x86 CRC. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6827 Test Plan: Run the same UBSAN and see it to pass now. Reviewed By: ltamasi Differential Revision: D21471838 fbshipit-source-id: 02943dd39a7030d2b03e5d894dcb23ed72b6c9c3 08 May 2020, 21:14:00 UTC
e72e216 Yanqin Jin 08 May 2020, 19:59:02 UTC Fix a few bugs in best-efforts recovery (#6824) Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2 08 May 2020, 20:01:42 UTC
1c84660 Andrew Kryczka 08 May 2020, 19:36:49 UTC prototype status check enforcement (#6798) Summary: Tried making Status object enforce that it is checked in some way. In cases it is not checked, `PermitUncheckedError()` must be called explicitly. Added a way to run tests (`ASSERT_STATUS_CHECKED=1 make -j48 check`) on a whitelist. The effort appears significant to get each test to pass with this assertion, so I only fixed up enough to get one test (`options_test`) working and added it to the whitelist. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6798 Reviewed By: pdillinger Differential Revision: D21377404 Pulled By: ajkr fbshipit-source-id: 73236f9c8df38f01cf24ecac4a6d1661b72d077e 08 May 2020, 19:40:43 UTC
1282589 sdong 08 May 2020, 03:50:14 UTC Disable "compression_parallel_threads" in crash test for now (#6816) Summary: "compressio_parallel_threads" caused several test failure tests. To keep crash test clean, disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6816 Test Plan: "make crash_test" to make sure the python script doesn't break Reviewed By: zhichao-cao Differential Revision: D21462112 fbshipit-source-id: 9eecc764800da82cd19665dc8b167eacead3310b 08 May 2020, 03:52:29 UTC
9426523 anand76 08 May 2020, 00:30:41 UTC Fix race due to delete triggered compaction in Universal compaction mode (#6799) Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a 08 May 2020, 00:32:17 UTC
3730b05 Andrew Kryczka 07 May 2020, 23:37:47 UTC Fixup HISTORY.md for e9ba4ba "validate range tombstone covers positiv… (#6825) Summary: …e range" Moved it from the wrong section (6.10) to the right section (Unreleased). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6825 Reviewed By: zhichao-cao Differential Revision: D21464577 Pulled By: ajkr fbshipit-source-id: a836b4ab10be2464182826f9411c9c424c933b70 07 May 2020, 23:40:17 UTC
f286fb3 anand76 07 May 2020, 22:52:58 UTC Include options.h in table.h (#6823) Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d 07 May 2020, 22:55:29 UTC
back to top