Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
17 June 2024, 23:04:45 UTC
sort by:
Revision Author Date Message Commit Date
27e593f Adam Retter 16 April 2020, 22:32:34 UTC Update RocksJava static version of bzip2 (#6714) Summary: Updates the version of bzip2 used for RocksJava static builds. Please, can we also get this cherry-picked to: 1. 6.7.fb 2. 6.8.fb 3. 6.9.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/6714 Reviewed By: cheng-chang Differential Revision: D21067233 Pulled By: pdillinger fbshipit-source-id: 8164b7eb99c5ca7b2021ab8c371ba9ded4cb4f7e 16 April 2020, 23:01:15 UTC
d7b6149 Peter Dillinger 13 March 2020, 20:36:45 UTC Use an Amazon S3 bucket for downloading deps (#6526) Summary: After we had a lot of failures with maven.org downloads, we wanted an alternative location for downloading binary dependencies. Hosting them through github would have been good in terms of organizational and network dependencies, but that approach seems to be awkward (fake releases, so would need a 'rocksdb-deps' repo) and strangely complicated for Facebook policy on open source repositories. This commit moves the downloads (that are not officially hosted by others on github) from my personal rocksdb fork to an S3 bucket owned by the Facebook RocksDB AWS account. Facebook employees can access this through an internal tool, and we should be able to grant permission to outside collaborators. Assuming this works out, I will back-port to older branches to stabilize their CI testing as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6526 Test Plan: CI Differential Revision: D20430130 Pulled By: pdillinger fbshipit-source-id: df52394a65e0a57942db3039bdaade8a4d520cb2 16 April 2020, 23:01:08 UTC
6731fef Adam Retter 11 March 2020, 18:43:23 UTC Update to latest Snappy to fix compilation issue on latest MacOS XCode (#6496) Summary: * **macOS version:** 10.15.2 (Catalina) * **XCode/Clang version:** Apple clang version 11.0.0 (clang-1100.0.33.16) Before this bugfix the error generated is: ``` In file included from ./util/compression.h:23: ./snappy-1.1.7/snappy.h:76:59: error: unknown type name 'string'; did you mean 'std::string'? size_t Compress(const char* input, size_t input_length, string* output); ^~~~~~ std::string /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iosfwd:211:65: note: 'std::string' declared here typedef basic_string<char, char_traits<char>, allocator<char> > string; ^ In file included from db/builder.cc:10: In file included from ./db/builder.h:12: In file included from ./db/range_tombstone_fragmenter.h:15: In file included from ./db/pinned_iterators_manager.h:12: In file included from ./table/internal_iterator.h:13: In file included from ./table/format.h:25: In file included from ./options/cf_options.h:14: In file included from ./util/compression.h:23: ./snappy-1.1.7/snappy.h:85:19: error: unknown type name 'string'; did you mean 'std::string'? string* uncompressed); ^~~~~~ std::string /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iosfwd:211:65: note: 'std::string' declared here typedef basic_string<char, char_traits<char>, allocator<char> > string; ^ 2 errors generated. make: *** [jls/db/builder.o] Error 1 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6496 Differential Revision: D20389254 Pulled By: pdillinger fbshipit-source-id: 2864245c8d0dba7b2ab81294241a62f2adf02e20 15 April 2020, 17:05:11 UTC
3482d4d sdong 18 March 2020, 17:57:46 UTC cmake: add option WITH_CORE_TOOLS to exclude tools except ldb and sst_dump (#6506) Summary: ldb and sst_dump are most important tools and they don't dependend on gflags. In cmake, we don't have an way to only build these two tools and exclude other tools. This is inconvenient if the environment has a problem with gflags. Add such an option WITH_CORE_TOOLS. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6506 Test Plan: cmake and build with WITH_TOOLS and without. Differential Revision: D20473029 fbshipit-source-id: 3d730fd14bbae6eeeae7f9cc9aec50a4e488ad72 15 April 2020, 17:04:56 UTC
d75da26 Yanqin Jin 31 March 2020, 04:44:30 UTC Update HISTORY and version 31 March 2020, 04:44:30 UTC
00894a2 Yanqin Jin 31 March 2020, 01:57:28 UTC Use flush time for the props.creation_time for FIFO compaction (#6612) Summary: For FIFO compaction, we use flush time instead of oldest key time as the creation time. This is to prevent FIFO compaction dropping files whose oldest key time is older than TTL but which has newer keys than TTL. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6612 Test Plan: make check Reviewed By: siying Differential Revision: D20748217 Pulled By: riversand963 fbshipit-source-id: 3f7b00a847020760537cdddd12f6fe039e5bc663 31 March 2020, 04:10:29 UTC
bba5e7b sdong 03 March 2020, 00:34:36 UTC Fix data race of GetCreationTimeOfOldestFile() (#6473) Summary: When DBImpl::GetCreationTimeOfOldestFile() calls Version::GetCreationTimeOfOldestFile(), the version is not directly or indirectly referenced, so an event like compaction can race with the operation and cause DBImpl::GetCreationTimeOfOldestFile() to access delocated data. This was caught by an ASAN run: ==268==ERROR: AddressSanitizer: heap-use-after-free on address 0x612000b7d198 at pc 0x000018332913 bp 0x7f391510d310 sp 0x7f391510d308 READ of size 8 at 0x612000b7d198 thread T845 (store_load-33) SCARINESS: 51 (8-byte-read-heap-use-after-free) #0 0x18332912 in rocksdb::Version::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/src/db/version_set.cc:1488 https://github.com/facebook/rocksdb/issues/1 0x1803ddaa in rocksdb::DBImpl::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/src/db/db_impl/db_impl.cc:4499 https://github.com/facebook/rocksdb/issues/2 0xe24ca09 in rocksdb::StackableDB::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/utilities/stackable_db.h:392 ...... 0x612000b7d198 is located 216 bytes inside of 296-byte region [0x612000b7d0c0,0x612000b7d1e8) freed by thread T28 here: ...... https://github.com/facebook/rocksdb/issues/5 0x1832c73f in std::vector<rocksdb::FileMetaData*, std::allocator<rocksdb::FileMetaData*> >::~vector() third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/stl_vector.h:435 https://github.com/facebook/rocksdb/issues/6 0x1832c73f in rocksdb::VersionStorageInfo::~VersionStorageInfo() rocksdb/src/db/version_set.cc:734 https://github.com/facebook/rocksdb/issues/7 0x1832cf42 in rocksdb::Version::~Version() rocksdb/src/db/version_set.cc:758 https://github.com/facebook/rocksdb/issues/8 0x9d1bb5 in rocksdb::Version::Unref() rocksdb/src/db/version_set.cc:2869 https://github.com/facebook/rocksdb/issues/9 0x183e7631 in rocksdb::Compaction::~Compaction() rocksdb/src/db/compaction/compaction.cc:275 https://github.com/facebook/rocksdb/issues/10 0x9e6de6 in std::default_delete<rocksdb::Compaction>::operator()(rocksdb::Compaction*) const third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78 https://github.com/facebook/rocksdb/issues/11 0x9e6de6 in std::unique_ptr<rocksdb::Compaction, std::default_delete<rocksdb::Compaction> >::reset(rocksdb::Compaction*) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:376 https://github.com/facebook/rocksdb/issues/12 0x9e6de6 in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2826 https://github.com/facebook/rocksdb/issues/13 0x9ac3b8 in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2320 https://github.com/facebook/rocksdb/issues/14 0x9abff7 in rocksdb::DBImpl::BGWorkCompaction(void*) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2096 ...... Fix the issue by reference the super version and use the referenced version from it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6473 Test Plan: Run ASAN for all existing tests. Differential Revision: D20196416 fbshipit-source-id: 5f4a7918110fc7b8dd7841932d376bc9d1e59d6f 03 March 2020, 19:12:05 UTC
cfaaaad sdong 24 February 2020, 23:45:47 UTC Buck config: Re-enable liburing under Linux (#6451) Summary: The known bug of liburing has been fixed. Now we can re-enable liburing under Linux Pull Request resolved: https://github.com/facebook/rocksdb/pull/6451 Test Plan: Watch internal CI Differential Revision: D20079009 fbshipit-source-id: 04a6f53a900ff721f9a62a188cf906771b5d68d2 25 February 2020, 02:24:51 UTC
5a868c1 Cheng Chang 24 February 2020, 19:42:37 UTC Update release version to 6.8 (#6450) Summary: Update release version to 6.8 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6450 Test Plan: no code change Differential Revision: D20071889 Pulled By: cheng-chang fbshipit-source-id: 91450aae09b201926469ff32f59ed436366f3b74 24 February 2020, 20:17:36 UTC
43dde33 Peter Dillinger 22 February 2020, 15:59:38 UTC Share kPageSize (and other small tweaks) (#6443) Summary: Make kPageSize extern const size_t (used in draft https://github.com/facebook/rocksdb/issues/6427) Make kLitteEndian constexpr bool Clarify a couple of comments Pull Request resolved: https://github.com/facebook/rocksdb/pull/6443 Test Plan: make check, CI Differential Revision: D20044558 Pulled By: pdillinger fbshipit-source-id: e0c5cc13229c82726280dc0ddcba4078346b8418 22 February 2020, 16:01:36 UTC
942eaba sdong 22 February 2020, 00:56:04 UTC Handle io_uring partial results (#6441) Summary: The logic that handles io_uring partial results was wrong. Fix the logic by putting it into a queue and continue reading. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6441 Test Plan: Make sure this patch fixes the application test case where the bug was discovered; in env_test, add a unit test that simulates partial results and make sure the results are still correct. Differential Revision: D20018616 fbshipit-source-id: 5398a7e34d74c26d52aa69dfd604e93e95d99c62 22 February 2020, 00:57:37 UTC
890d87f Yanqin Jin 21 February 2020, 23:07:55 UTC Some minor fix-ups (#6440) Summary: Cleanup some code without any real change in functionality. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6440 Differential Revision: D20015891 Pulled By: riversand963 fbshipit-source-id: 33e18754b0f002006a6d4805e9aaf84c0c8ad25a 21 February 2020, 23:09:56 UTC
ab65278 Peter Dillinger 21 February 2020, 21:29:41 UTC Misc filter_bench improvements (#6444) Summary: Useful in validating/testing internal fragmentation changes (https://github.com/facebook/rocksdb/issues/6427) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6444 Test Plan: manual (no changes to production code) Differential Revision: D20040076 Pulled By: pdillinger fbshipit-source-id: 32d26f363d2a9ab9f5bebd281dcebd9915ae340e 21 February 2020, 21:31:57 UTC
fcec56e Zaiyang Li 21 February 2020, 19:10:54 UTC Add function to set row cache on rocksdb_options_t (#6442) Summary: Adding a C API function to set `row_cache` on `rocksdb_options_t` as this functionality is missing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6442 Differential Revision: D20036813 Pulled By: riversand963 fbshipit-source-id: c1fa95ea343345fbc1e57961d0d048e0e79be373 21 February 2020, 19:12:42 UTC
d75ce0a sdong 20 February 2020, 22:53:51 UTC Mention rocksdb_namespace.h in HISTORY.md (#6439) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6439 Differential Revision: D20012442 fbshipit-source-id: a7c9569826eac39cd7ea69c90f08a21dd4caa335 20 February 2020, 22:55:19 UTC
362b8d4 Yanqin Jin 20 February 2020, 21:13:53 UTC Fix MANIFEST name assignment (#6426) Summary: Currently, a new MANIFEST file is assigned a new file number when 1) no MANIFEST is open, or 2) current MANIFEST file size exceeds a threshold. This is not sufficient. There are cases when the caller explicitly specifies that a new MANIFEST be created. For example, if user sets options.write_dbid_to_manifest = true, and there are WAL files, then RocksDB will run into an issue during recovery. `DBImpl::Recover()` will call `LogAndApply()` to write dbid. At this point, the db being recovered creates a new MANIFEST, say, MANIFEST-000003. Since there are WALs, `DBImpl::RecoverLogFiles` will be called. Towards the end of this function, we call `LogAndApply(new_descriptor_log=true)`, which explicitly creates a new MANIFEST. However, the manifest_file_number is wrong before this fix. Consequently, RocksDB opens an existing, non-empty file for append, effectively truncating the file to zero. If a crash occurs, then there will be data loss. Test Plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6426 Test Plan: make check Differential Revision: D19951866 Pulled By: riversand963 fbshipit-source-id: 4b1b9fc28d4fe2ac12764b388ef9e61f05e766da 20 February 2020, 22:30:58 UTC
fdf882d sdong 20 February 2020, 20:07:53 UTC Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) Summary: When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433 Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag. Differential Revision: D19977691 fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e 20 February 2020, 20:09:57 UTC
4e33f1e Gaurav Singh 20 February 2020, 18:25:14 UTC simplify user_access_only expression (#6360) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6360 Differential Revision: D19698918 Pulled By: riversand963 fbshipit-source-id: d20ecca541376cccd32fc7afb504ea90021860ee 20 February 2020, 18:27:56 UTC
a993cc3 Remington Brasga 20 February 2020, 01:06:14 UTC Fixed typo in benchmark.sh (#6434) Summary: TB = 1024 * GB Pull Request resolved: https://github.com/facebook/rocksdb/pull/6434 Differential Revision: D19978339 Pulled By: zhichao-cao fbshipit-source-id: 5a89890110b23f0ebda4a95223f66da6736321ac 20 February 2020, 01:08:02 UTC
5a29751 Yanqin Jin 19 February 2020, 00:08:14 UTC Ignore .vs/.vscode dir used by visual studio/ vscode (#6428) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6428 Differential Revision: D19959176 Pulled By: riversand963 fbshipit-source-id: 5b863944663462b246bb82883fa683f75ab33fc1 19 February 2020, 00:10:15 UTC
c6abe30 Andrew Kryczka 18 February 2020, 21:52:09 UTC Fix concurrent full purge and WAL recycling (#5900) Summary: We were removing the file from `log_recycle_files_` before renaming it with `ReuseWritableFile()`. Since `ReuseWritableFile()` occurs outside the DB mutex, it was possible for a concurrent full purge to sneak in and delete the file before it could be renamed. Consequently, `SwitchMemtable()` would fail and the DB would enter read-only mode. The fix is to hold the old file number in `log_recycle_files_` until after the file has been renamed. Full purge uses that list to decide which files to keep, so it can no longer delete a file pending recycling. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5900 Test Plan: new unit test Differential Revision: D19771719 Pulled By: ajkr fbshipit-source-id: 094346349ca3fb499712e62de03905acc30b5ce8 18 February 2020, 21:54:13 UTC
0f9dcb8 Andrew Kryczka 18 February 2020, 19:14:37 UTC Return NotSupported from WriteBatchWithIndex::DeleteRange (#5393) Summary: As discovered in https://github.com/facebook/rocksdb/issues/5260 and https://github.com/facebook/rocksdb/issues/5392, reads on the indexed batch do not account for range tombstones. So, return `Status::NotSupported` from `WriteBatchWithIndex::DeleteRange` until we properly support it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5393 Test Plan: added unit test Differential Revision: D19912360 Pulled By: ajkr fbshipit-source-id: 0bbfc978ea015d64516ca708fce2429abba524cb 18 February 2020, 19:18:25 UTC
3a34575 acelyc111 14 February 2020, 23:53:29 UTC Fix compile error when LZ4 is up to r123 (#6412) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6412 Differential Revision: D19914063 Pulled By: ajkr fbshipit-source-id: 4e401e665d4b449d24c4cdec35a4585eeda95996 14 February 2020, 23:55:23 UTC
dc23c12 Manuel Ung 14 February 2020, 19:20:37 UTC WriteUnPrepared: Untracked keys (#6404) Summary: For write unprepared, some applications may bypass the transaction api, and write keys directly into the write batch. However, since they are not tracked, rollbacks (both for savepoint and transaction) are not aware that these keys have to be rolled back. The fix is to track them in `WriteUnpreparedTxn::untracked_keys_`. This is populated whenever we flush unprepared batches into the DB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6404 Differential Revision: D19842023 Pulled By: lth fbshipit-source-id: a9edfc643d5c905fc89da9a9a9094d30c9b70108 14 February 2020, 19:31:39 UTC
152f8a8 Cheng Chang 14 February 2020, 16:24:09 UTC Remove unnecessary computation of index (#6406) Summary: `index` can be replaced by `iter`, saving the computation of `index++`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6406 Test Plan: make check Differential Revision: D19905056 Pulled By: cheng-chang fbshipit-source-id: add4638959c0d2e4e77a11f3fa04ffabaf0de790 14 February 2020, 16:26:23 UTC
4034e28 Cheng Chang 14 February 2020, 16:15:24 UTC Fail fast in paranoid mode when LoadTableHandlers fail during recovering (#6368) Summary: Previously, when recovering version set, LoadTableHandlers failures are ignored. If paranoid_checks is true, this failure should not be ignored, otherwise, the opened db might be in an inconsistent state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6368 Test Plan: make check Differential Revision: D19713459 Pulled By: cheng-chang fbshipit-source-id: 68cb94f4f2cc43f8b024b14755193cd45cfcad55 14 February 2020, 16:17:10 UTC
29e2443 wolfkdy 14 February 2020, 02:50:04 UTC refine code (#6420) Summary: I create a new branch from the branch new upsteram/master and "git merge --squash". Maybe it will fix everything. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6420 Differential Revision: D19897152 Pulled By: zhichao-cao fbshipit-source-id: 6575d9e3b23e360f42ee1480b43028b5fcc20136 14 February 2020, 02:55:02 UTC
908b1ee Manuel Ung 14 February 2020, 02:50:01 UTC WriteUnPrepared: Fix assertion during recovery (#6419) Summary: During recovery, multiple (un)prepared batches could exist in the same WAL record due to group commit. This breaks an assertion in `MemTableInserter::MarkBeginPrepare`. To fix, reset unprepared_batch_ to false after `MarkEndPrepare`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6419 Differential Revision: D19896148 Pulled By: lth fbshipit-source-id: b1a32ef88f775a0881264a18bd1a4a5b8c85eee3 14 February 2020, 02:52:05 UTC
fb57150 Manuel Ung 14 February 2020, 02:43:16 UTC WriteUnPrepared: Enable WAL during crash recovery (#6418) Summary: Unfortunately, it seems like mysqld reuses xids across machine restarts. When that happens, we could have something like the following happening: ``` BEGIN_PREPARE(unprepared) Put(a) END_PREPARE(xid = 1) -- crash and recover with Put(a) rolled back as it was not prepared BEGIN_PREPARE(prepared) Put(b) END_PREPARE(xid = 1) COMMIT(xid = 1) -- crash and recover with both a, b ``` To solve this, we will have to log the rollback batch into the WAL during recovery. WritePrepared already logs the rollback batch into the WAL, if a rollback happens after prepare, so there is no problem there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6418 Differential Revision: D19896151 Pulled By: lth fbshipit-source-id: 2ff65ddc5fe75efd57736fed4b7cd7a109d26609 14 February 2020, 02:44:39 UTC
ac8e89a sdong 14 February 2020, 02:39:38 UTC Should flush and sync WAL when writing it in DB::Open() (#6417) Summary: A recent fix related to 2pc https://github.com/facebook/rocksdb/pull/6313/ writes something to WAL, but does not flush or sync. This causes assertion failure "impl->TEST_WALBufferIsEmpty()" if manual_wal_flush = true. We should fsync the entry to make sure a second power reset can recover. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6417 Test Plan: Add manual_wal_flush=true case in TransactionTest.DoubleCrashInRecovery and fix a bug in the test so that the bug can be reproduced. It passes with the fix. Differential Revision: D19894537 fbshipit-source-id: f1e84e49e2269f583c6019743118292cd8b6598e 14 February 2020, 02:41:04 UTC
4651677 Cheng Chang 14 February 2020, 01:25:29 UTC Fix flaky test DecreaseNumBgThreads (#6393) Summary: The DecreaseNumBgThreads test keeps failing on Windows in AppVeyor. It fails because it depends on a timed wait for the tasks to be dequeued from the threadpool's internal queue, but within the specified time, the task might have not been scheduled onto the newly created threads. https://github.com/facebook/rocksdb/pull/6232 tries to fix this by waiting for longer time to let the threads scheduled. This PR tries to fix this by replacing the timed wait with a synchronization on the task's internal conditional variable. When the number of threads increases, instead of guessing the time needed for the task to be scheduled, it directly blocks on the conditional variable until the task starts running. But when thread number is reduced, it still does a timed wait, but this does not lead to the flakiness now, will try to remove these timed waits in a future PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6393 Test Plan: Wait to see whether AppVeyor tests pass. Differential Revision: D19890928 Pulled By: cheng-chang fbshipit-source-id: 4e56e4addf625c98c0876e62d9d57a6f0a156f76 14 February 2020, 01:27:18 UTC
acfee40 sdong 14 February 2020, 00:49:29 UTC Remove IO URING compiler flags (#6415) Summary: Since IO Uring feature is not stable. Remove it from buck configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6415 Test Plan: See internal build pass Differential Revision: D19892988 fbshipit-source-id: 7fc01efc2af5ed707fb8e4e4674223aeb83cd5ea 14 February 2020, 00:51:10 UTC
5138764 Huisheng Liu 13 February 2020, 19:19:39 UTC Fix destroydb (#6308) Summary: It's observed on Windows DestroyDB failed to remove the log file because the logger is still alive in sst file manager and holding a handle to the log file. This fix makes sure the logger is released before attempt to clear the database directory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6308 Differential Revision: D19818829 Pulled By: riversand963 fbshipit-source-id: 54c3e6859aadaaba4a49b3e851b73dc35ec7dc6a 13 February 2020, 19:21:27 UTC
df3f33d sdong 13 February 2020, 18:50:03 UTC Fix db_bench LITE build recently broken (#6411) Summary: A recent change https://github.com/facebook/rocksdb/pull/6386 broke LITE build in a trivial way. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6411 Test Plan: Run "LITE=1 make all" Differential Revision: D19871765 fbshipit-source-id: 74f0ad3f8a9d666fbde0da7fd29ba1547a811f77 13 February 2020, 18:52:50 UTC
a676001 Cheng Chang 13 February 2020, 18:18:23 UTC Revert usage of Defer. (#6410) Summary: Seems like this caused the following test failure on AppVeyor: DBTest2.CrashInRecoveryMultipleCF c:\projects\rocksdb\db\db_test_util.cc(107): error: DestroyDB(dbname_, options) IO error: Failed to delete: C:\projects\rocksdb\db_tests\\testrocksdb-3112//db_test2_10791409581227174103/000013.sst: Access is denied. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6410 Test Plan: Wait to see whether the AppVeyor test passes. Differential Revision: D19879872 Pulled By: cheng-chang fbshipit-source-id: 59a9c55ca88566e9210c0b715ecc45a4fd9afe26 13 February 2020, 18:31:32 UTC
6e97d4d sdong 13 February 2020, 02:00:24 UTC By default turn IO Uring off. (#6405) Summary: We realized bugs related to IO Uring. Turn it off by default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6405 Test Plan: Manually run build_tools/build_detect_platform and observe outputs. Differential Revision: D19862792 fbshipit-source-id: 5d5e8e2762997b72a145ae59389ef3d7e4ccd060 13 February 2020, 02:01:49 UTC
e645089 Burton Li 12 February 2020, 22:44:46 UTC db_bench supports for generating random variable sized value. (#6386) Summary: 1. `db_bench` now supports `value_size_distribution_type`, `value_size_min`, `value_size_max` options for generating random variable sized value. 2. Added `blob_db_compression_type` option for BlobDB to enable blob compression. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6386 Differential Revision: D19859406 Pulled By: zhichao-cao fbshipit-source-id: ace52674090023fde15d832392110bf288a8e215 12 February 2020, 22:47:03 UTC
3e49249 anand76 12 February 2020, 01:25:10 UTC Ensure all MultiGet IO errors are propagated to user (#6403) Summary: Unrevert the previous fix to propagate error status, and an additional fix to not treat a memtable lookup MergeInProgress status as an error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6403 Test Plan: Unit tests Tried running stress tests but couldn't repro the stress failure Differential Revision: D19846721 Pulled By: anand1976 fbshipit-source-id: 7db10cccbdc863d9b559497f0a46b608d2488ca4 12 February 2020, 01:27:22 UTC
e412a42 Tomas Kolda 11 February 2020, 22:45:15 UTC JNI direct buffer support for basic operations (#2283) Summary: It is very useful to support direct ByteBuffers in Java. It allows to have zero memory copy and some serializers are using that directly so one do not need to create byte[] array for it. This change also contains some fixes for Windows JNI build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/2283 Differential Revision: D19834971 Pulled By: pdillinger fbshipit-source-id: 44173aa02afc9836c5498c592fd1ea95b6086e8e 11 February 2020, 22:48:30 UTC
28aa09d Yanqin Jin 11 February 2020, 06:24:29 UTC Pass info_log_level to the inner logger of AutoRollLogger (#6388) Summary: Before this fix, the info_log_level passed from CreateLoggerFromOptions() will be ignored by AutoRollLogger::logger_. This PR fixes it by setting the info log level of logger_ during ResetLogger(). Test plan (dev server): ``` COMPILE_WITH_TSAN=1 make all && make check make all && make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6388 Differential Revision: D19828045 Pulled By: riversand963 fbshipit-source-id: e1ac7de3a2090bee53b6c667f71a11f1774163e6 11 February 2020, 06:26:08 UTC
35ed530 anand76 11 February 2020, 06:21:53 UTC Revert "Check KeyContext status in MultiGet (#6387)" (#6401) Summary: This reverts commit d70011bccc9a9383f368e509671d5e4eb8f3a2c2. The commit is causing some stress test failure due to unexpected Status::MergeInProgress() return for some keys. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6401 Differential Revision: D19826623 Pulled By: anand1976 fbshipit-source-id: edd634cede9cb7bdd2cb8f46e662ea709b16d2f1 11 February 2020, 06:23:36 UTC
d3ba398 Cheng Chang 11 February 2020, 02:12:04 UTC Update unit tests for PinnableSlice (#6399) Summary: 1. remove AssertEmpty because calling methods on moved objects is discouraged. 2. add a test to assert that the internal buffer is moved instead of being copied. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6399 Test Plan: make slice_test && ./slice_test USE_CLANG=1 make analyze Differential Revision: D19825372 Pulled By: cheng-chang fbshipit-source-id: 2e26f8ce5ec3edbfce067db045e80bd433e704f4 11 February 2020, 02:13:27 UTC
dafb568 Cheng Chang 11 February 2020, 01:58:03 UTC Add utility class Defer (#6382) Summary: Add a utility class `Defer` to defer the execution of a function until the Defer object goes out of scope. Used in VersionSet:: ProcessManifestWrites as an example. The inline comments for class `Defer` have more details. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6382 Test Plan: `make defer_test version_set_test && ./defer_test && ./version_set_test` Differential Revision: D19797538 Pulled By: cheng-chang fbshipit-source-id: b1a9b7306e4fd4f48ec2ab55783caa561a315f0f 11 February 2020, 01:59:47 UTC
cbf5f3b Levi Tamasi 11 February 2020, 01:03:21 UTC Do not move VersionEdit into AtomicGroupReadBuffer (#6400) Summary: https://github.com/facebook/rocksdb/pull/6383 surfaced an issue with `VersionSet`/`ReactiveVersionSet` and `AtomicGroupReadBuffer::AddEdit` (which was added in https://github.com/facebook/rocksdb/pull/5411): `AddEdit` moves the `VersionEdit` passed to it into `replay_buffer_`, however, the client `VersionSet` classes keep using it afterwards. This *seemed to* work before the refactoring but it really did not: since `VersionEdit` used to have a user-declared destructor, no move constructor/move assignment operator was generated, and the `move` in `AddEdit` was really a copy. The patch makes the copy explicit. Note: it should be possible to rework this logic so that we can get away with the move but for now, this should fix the issue. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6400 Test Plan: `make check` `make analyze` Differential Revision: D19824466 Pulled By: ltamasi fbshipit-source-id: f38033967daf2a39c78dcd6e12978bafe37632b4 11 February 2020, 01:15:42 UTC
4369f2c Zhichao Cao 10 February 2020, 23:42:46 UTC Checksum for each SST file and stores in MANIFEST (#6216) Summary: In the current code base, RocksDB generate the checksum for each block and verify the checksum at usage. Current PR enable SST file checksum. After a SST file is generated by Flush or Compaction, RocksDB generate the SST file checksum and store the checksum value and checksum method name in the vs_info and MANIFEST as part for the FileMetadata. Added the enable_sst_file_checksum to Options to enable or disable file checksum. Added sst_file_checksum to Options such that user can plugin their own SST file checksum calculate method via overriding the SstFileChecksum class. The checksum information inlcuding uint32_t checksum value and a checksum name (string). A new tool is added to LDB such that user can dump out a list of file checksum information from MANIFEST. If user enables the file checksum but does not provide the sst_file_checksum instance, RocksDB will use the default crc32checksum implemented in table/sst_file_checksum_crc32c.h Pull Request resolved: https://github.com/facebook/rocksdb/pull/6216 Test Plan: Added the testing case in table_test and ldb_cmd_test to verify checksum is correct in different level. Pass make asan_check. Differential Revision: D19171461 Pulled By: zhichao-cao fbshipit-source-id: b2e53479eefc5bb0437189eaa1941670e5ba8b87 10 February 2020, 23:52:52 UTC
594e815 sdong 10 February 2020, 23:42:23 UTC Make clang analyze happy with options_test (#6398) Summary: clang analysis shows following warning: options/options_test.cc:1554:24: warning: The left operand of '-' is a garbage value (file_size - 1) / readahead_size + 1); ~~~~~~~~~ ^ Explicitly initialize file_size and add an assertion to make clang analysis happy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6398 Test Plan: Run "make analysis" and see the warning goes away. Differential Revision: D19819662 fbshipit-source-id: 1589ea91c0c8f78242538f01448e4ad0e5fbc219 10 February 2020, 23:50:25 UTC
b2bc1da sdong 10 February 2020, 21:35:04 UTC Try to fix some analysis failures Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6384 Test Plan: Wait and see the analysis result. Differential Revision: D19781072 fbshipit-source-id: 75e7cb6ee619ebd289841eaabea03dd075c09d3b 10 February 2020, 21:37:14 UTC
2e0159e Yutian Li 10 February 2020, 20:31:12 UTC Add error status for no_slowdown & low priority write (#6396) Summary: When `no_slowdown` is enabled, it returns `Status::Incomplete("Write stall")` if a stall would occur. This patch adds descriptive text for when `no_slowdown` and `low_pri` are enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6396 Differential Revision: D19808978 Pulled By: cheng-chang fbshipit-source-id: a53b0d25ed414c821a086531e0222027f925e627 10 February 2020, 20:33:16 UTC
debc4ef Kefu Chai 10 February 2020, 19:29:58 UTC utilities/env_librados: copy use bufferlist::iterator (#6395) Summary: to adapt the change in ceph upstream where the bufferlist::copy() method was removed in https://github.com/ceph/ceph/commit/c724369010a753bd44e11a534d1f42156c4fc12d Signed-off-by: Kefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6395 Differential Revision: D19816815 Pulled By: zhichao-cao fbshipit-source-id: 9210767b91af0ecdcf5dfaa3e70edcaeea55135f 10 February 2020, 19:31:16 UTC
84b41a6 blackyblack 08 February 2020, 01:06:43 UTC Only add gtest target when building with tests (#6377) Summary: External CMAKE projects cannot add own gtest target due to CMP0002 policy. We should not add gtest target unless we build with tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6377 Differential Revision: D19801708 fbshipit-source-id: 662484683e8737e10397ddb0d17705da5243c4f9 08 February 2020, 01:16:44 UTC
d70011b anand76 08 February 2020, 00:46:32 UTC Check KeyContext status in MultiGet (#6387) Summary: Currently, any IO errors and checksum mismatches while reading data blocks, are being ignored by the batched MultiGet. Its only looking at the GetContext state. Fix that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6387 Test Plan: Add unit tests Differential Revision: D19799819 Pulled By: anand1976 fbshipit-source-id: 46133dccbb04e64067b9fe6cda73e282203db969 08 February 2020, 00:48:16 UTC
4e45727 Robert Yang 07 February 2020, 23:18:04 UTC db/write_thread.cc: Initialize state (#6275) Summary: Fixed an error when compiled with -Og: db/write_thread.cc:183:14: error: 'state' may be used uninitialized in this function [-Werror=maybe-uninitialized] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6275 Differential Revision: D19381755 fbshipit-source-id: a90bf3cd4a7248d9d71219e918fc6253deb97e3c 07 February 2020, 23:20:38 UTC
876c2db sdong 07 February 2020, 23:16:29 UTC Allow readahead when reading option files. (#6372) Summary: Right, when reading from option files, no readahead is used and 8KB buffer is used. It might introduce high latency if the file system provide high latency and doesn't do readahead. Instead, introduce a readahead to the file. When calling inside DB, infer the value from options.log_readahead. Otherwise, a default 512KB readahead size is used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6372 Test Plan: Add --log_readahead_size in db_bench. Run it with several options and observe read size from option files using strace. Differential Revision: D19727739 fbshipit-source-id: e6d8053b0a64259abc087f1f388b9cd66fa8a583 07 February 2020, 23:18:26 UTC
b42fa14 Cheng Chang 07 February 2020, 22:24:41 UTC Support move semantics for PinnableSlice (#6374) Summary: It's logically correct for PinnableSlice to support move semantics to transfer ownership of the pinned memory region. This PR adds both move constructor and move assignment to PinnableSlice. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6374 Test Plan: A set of unit tests for the move semantics are added in slice_test. So `make slice_test && ./slice_test`. Differential Revision: D19739254 Pulled By: cheng-chang fbshipit-source-id: f898bd811bb05b2d87384ec58b645e9915e8e0b1 07 February 2020, 22:26:26 UTC
25fbdc5 Chad Austin 07 February 2020, 22:18:24 UTC Fix Buck build on macOS (#6378) Summary: liburing is a Linux-specific dependency, so make sure it's configured in the Linux-only Buck rules. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6378 Test Plan: ``` ~/fbcode $ cp internal_repo_rocksdb/repo/TARGETS rocksdb/src ~/fbcode $ buck build mode/mac eden ``` Reviewed By: chadaustin Differential Revision: D19760039 Pulled By: riversand963 fbshipit-source-id: 2abfce81c8b17965ef76012262cd117708e0294f 07 February 2020, 22:20:12 UTC
752c87a Levi Tamasi 07 February 2020, 21:25:07 UTC Clean up VersionEdit a bit (#6383) Summary: This is a bunch of small improvements to `VersionEdit`. Namely, the patch * Makes the names and order of variables, methods, and code chunks related to the various information elements more consistent, and adds missing getters for the sake of completeness. * Initializes previously uninitialized stack variables. * Marks all getters const to improve const correctness. * Adds in-class initializers and removes the default ctor that would create an object with uninitialized built-in fields and call `Clear` afterwards. * Adds a new type alias for new files and changes the existing `typedef` for deleted files into a type alias as well. * Makes the helper method `DecodeNewFile4From` private. * Switches from long-winded iterator syntax to range based loops in a couple of places. * Fixes a couple of assignments where an integer 0 was assigned to boolean members. * Fixes a getter which used to return a `const std::string` instead of the intended `const std::string&`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6383 Test Plan: make check Differential Revision: D19780537 Pulled By: ltamasi fbshipit-source-id: b0b4f09fee0ec0e7c7b7a6d76bfe5346e91824d0 07 February 2020, 21:27:06 UTC
5f478b9 Cheng Chang 07 February 2020, 21:17:12 UTC Remove outdated comment (#6379) Summary: Since the logic for handling IDENTITY file is now inside `NewDB`, the comment above `NewDB` is no longer relevant. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6379 Test Plan: not needed Differential Revision: D19795440 Pulled By: cheng-chang fbshipit-source-id: 0b1cca87ac6d92474701c46aa4c8d4d708bfa19b 07 February 2020, 21:18:43 UTC
1b4be4c Levi Tamasi 07 February 2020, 20:49:11 UTC BlobDB: ignore trivially moved files when updating the SST<->blob file mapping (#6381) Summary: BlobDB keeps track of the mapping between SSTs and blob files using the `OnFlushCompleted` and `OnCompactionCompleted` callbacks of the `EventListener` interface: upon receiving a flush notification, a link is added between the newly flushed SST and the corresponding blob file; for compactions, links are removed for the inputs and added for the outputs. The earlier code performed this link deletion and addition even for trivially moved files; the new code walks through the two lists together (in a fashion that's similar to merge sort) and skips such files. This should mitigate https://github.com/facebook/rocksdb/issues/6338, wherein an assertion is triggered with the earlier code when a compaction notification for a trivial move precedes the flush notification for the moved SST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6381 Test Plan: make check Differential Revision: D19773729 Pulled By: ltamasi fbshipit-source-id: ae0f273ded061110dd9334e8fb99b0d7786650b0 07 February 2020, 20:50:57 UTC
107a7ca Cheng Chang 07 February 2020, 20:31:28 UTC Remove inappropriate comments (#6371) Summary: The comments are for iterators, not Cleanable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6371 Test Plan: no need Differential Revision: D19727527 Pulled By: cheng-chang fbshipit-source-id: c74aeffa27ea0ce15a36ff6f9694826712cd1c70 07 February 2020, 20:35:24 UTC
0a74e1b Cheng Chang 07 February 2020, 20:29:41 UTC Add status checks during DB::Open (#6380) Summary: Several statuses were not checked during DB::Open. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6380 Test Plan: make check Differential Revision: D19780237 Pulled By: cheng-chang fbshipit-source-id: c8d189d20344bd1607890dd1449345bda2ef96b9 07 February 2020, 20:32:09 UTC
f361ced Yanqin Jin 07 February 2020, 18:50:17 UTC Atomic flush rollback once on failure (#6385) Summary: Before this fix, atomic flush codepath may hit an assertion failure on a specific failure case. If all flush jobs within an atomic flush succeed (they do not write to MANIFEST), but batch writing version edits to MANIFEST fails, then `cfd->imm()->RollbackMemTableFlush()` will be called twice, and the second invocation hits assertion failure `assert(m->flush_in_progress_)` since the first invocation resets the variable `flush_in_progress_` to false already. Test plan (dev server): ``` ./db_flush_test --gtest_filter=DBAtomicFlushTest/DBAtomicFlushTest.RollbackAfterFailToInstallResults make check ``` Both must succeed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6385 Differential Revision: D19782943 Pulled By: riversand963 fbshipit-source-id: 84e1592625e729d1b70fdc8479959387a74cb121 07 February 2020, 18:52:10 UTC
c6f7551 atul 06 February 2020, 18:29:06 UTC Fixing the documentation of the function (#4803) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6354 Differential Revision: D19725459 Pulled By: riversand963 fbshipit-source-id: fded24576251bfa4b289399f0909f1fe43426e28 06 February 2020, 18:30:44 UTC
f5f79f0 Cheng Chang 06 February 2020, 18:14:19 UTC Be able to read compatible leveldb sst files (#6370) Summary: In `DBSSTTest.SSTsWithLdbSuffixHandling`, some sst files are renamed to ldb files, the original intention of the test is to test that the ldb files can be loaded along with the sst files. The original test checks this by `ASSERT_NE("NOT_FOUND", Get(Key(k)))`, but the problem is `Get(Key(k))` returns IO error due to path not found instead of NOT_FOUND, so the success of ASSERT_NE does not mean the key can be retrieved. This PR updates the test to make sure Get(Key(k)) returns the original value. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6370 Test Plan: make db_sst_test && ./db_sst_test Differential Revision: D19726278 Pulled By: cheng-chang fbshipit-source-id: 993127f56457b315e669af4eeb92d6f956b7a4b7 06 February 2020, 18:15:44 UTC
24c9dce sdong 06 February 2020, 04:59:13 UTC Remove include math.h (#6373) Summary: We see some odd errors complaining math. However, it doesn't seem that it is needed to be included. Remove the include of math.h. Just removing it from db_bench doesn't seem to break anything. Replacing sqrt from std::sqrt seems to work for histogram.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/6373 Test Plan: Watch Travis and appveyor to run. Differential Revision: D19730068 fbshipit-source-id: d3ad41defcdd9f51c2da1a3673fb258f5dfacf47 06 February 2020, 05:00:49 UTC
1ed7d9b Mike Kolupaev 04 February 2020, 21:38:00 UTC Avoid lots of calls to Env::GetFileSize() in SstFileManagerImpl when opening DB (#6363) Summary: Before this PR it calls GetFileSize() once for each sst file in the DB. This can take a long time if there are be tens of thousands of sst files (e.g. in thousands of column families), and even longer if Env is talking to some remote service rather than local filesystem. This PR makes DB::Open() use sst file sizes that are already known from manifest (typically almost all files in the DB) and only call GetFileSize() for non-sst or obsolete files. Note that GetFileSize() is also called and checked against manifest in CheckConsistency(), so the calls in SstFileManagerImpl were completely redundant. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6363 Test Plan: deployed to a test cluster, looked at a dump of Env calls (from a custom instrumented Env) - no more thousands of GetFileSize()s. Differential Revision: D19702509 Pulled By: al13n321 fbshipit-source-id: 99f8110620cb2e9d0c092dfcdbb11f3af4ff8b73 04 February 2020, 21:41:53 UTC
3a07323 sdong 04 February 2020, 19:37:07 UTC Consolidate ReadFileToString() (#6366) Summary: It's a minor refactoring. We have two ReadFileToString() but they are very similar. Make the one with Env argument calls the one with FS argument instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6366 Test Plan: Run all existing tests Differential Revision: D19712332 fbshipit-source-id: 5ae6fabf6355938690d95cda52afd1f39e0a7823 04 February 2020, 19:39:23 UTC
69c8614 sdong 04 February 2020, 19:37:03 UTC Avoid to get manifest file size when recovering from it. (#6369) Summary: Right now RocksDB gets manifest file size before recovering from it. The information is available in LogReader. Use it instead to prevent one file system call. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6369 Test Plan: Run all existing tests Differential Revision: D19714872 fbshipit-source-id: 0144be324d403c99e3da875ea2feccc8f64e883d 04 February 2020, 19:39:23 UTC
637e64b Mike Kolupaev 04 February 2020, 09:24:29 UTC Add an option to prevent DB::Open() from querying sizes of all sst files (#6353) Summary: When paranoid_checks is on, DBImpl::CheckConsistency() iterates over all sst files and calls Env::GetFileSize() for each of them. As far as I could understand, this is pretty arbitrary and doesn't affect correctness - if filesystem doesn't corrupt fsynced files, the file sizes will always match; if it does, it may as well corrupt contents as well as sizes, and rocksdb doesn't check contents on open. If there are thousands of sst files, getting all their sizes takes a while. If, on top of that, Env is overridden to use some remote storage instead of local filesystem, it can be *really* slow and overload the remote storage service. This PR adds an option to not do GetFileSize(); instead it does GetChildren() for parent directory to check that all the expected sst files are at least present, but doesn't check their sizes. We can't just disable paranoid_checks instead because paranoid_checks do a few other important things: make the DB read-only on write errors, print error messages on read errors, etc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6353 Test Plan: ran the added sanity check unit test. Will try it out in a LogDevice test cluster where the GetFileSize() calls are causing a lot of trouble. Differential Revision: D19656425 Pulled By: al13n321 fbshipit-source-id: c2c421b367633033760d1f56747bad206d1fbf82 04 February 2020, 09:27:26 UTC
7330ec0 anand76 04 February 2020, 02:15:12 UTC Fix a test failure in error_handler_test (#6367) Summary: Fix an intermittent failure in DBErrorHandlingTest.CompactionManifestWriteError due to a race between background error recovery and the main test thread calling TEST_WaitForCompact(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6367 Test Plan: Run the test using gtest_parallel Differential Revision: D19713802 Pulled By: anand1976 fbshipit-source-id: 29e35dc26e0984fe8334c083e059f4fa1f335d68 04 February 2020, 02:16:52 UTC
f195d8d sdong 04 February 2020, 01:38:50 UTC Use ReadFileToString() to get content from IDENTITY file (#6365) Summary: Right now when reading IDENTITY file, we use a very similar logic as ReadFileToString() while it does an extra file size check, which may be expensive in some file systems. There is no reason to duplicate the logic. Use ReadFileToString() instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6365 Test Plan: RUn all existing tests. Differential Revision: D19709399 fbshipit-source-id: 3bac31f3b2471f98a0d2694278b41e9cd34040fe 04 February 2020, 01:40:49 UTC
36c504b sdong 03 February 2020, 22:11:50 UTC Avoid create directory for every column families (#6358) Summary: A relatively recent regression causes for every CF, create and open directory is called for the DB directory, unless CF has a private directory. This doesn't scale well with large number of column families. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6358 Test Plan: Run all existing tests and see it pass. strace with db_bench --num_column_families and observe it doesn't open directory for number of column families. Differential Revision: D19675141 fbshipit-source-id: da01d9216f1dae3f03d4064fbd88ce71245bd9be 03 February 2020, 22:13:39 UTC
eb4d6af Huisheng Liu 03 February 2020, 21:30:13 UTC Error handler test fix (#6266) Summary: MultiDBCompactionError fails when it verifies the number of files on level 0 and level 1 without waiting for compaction to finish. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6266 Differential Revision: D19701639 Pulled By: riversand963 fbshipit-source-id: e96d511bcde705075f073e0b550cebcd2ecfccdc 03 February 2020, 21:32:53 UTC
7242dae Adam Retter 03 February 2020, 20:28:25 UTC Improve RocksJava Comparator (#6252) Summary: This is a redesign of the API for RocksJava comparators with the aim of improving performance. It also simplifies the class hierarchy. **NOTE**: This breaks backwards compatibility for existing 3rd party Comparators implemented in Java... so we need to consider carefully which release branches this goes into. Previously when implementing a comparator in Java the developer had a choice of subclassing either `DirectComparator` or `Comparator` which would use direct and non-direct byte-buffers resepectively (via `DirectSlice` and `Slice`). In this redesign there we have eliminated the overhead of using the Java Slice classes, and just use `ByteBuffer`s. The `ComparatorOptions` supplied when constructing a Comparator allow you to choose between direct and non-direct byte buffers by setting `useDirect`. In addition, the `ComparatorOptions` now allow you to choose whether a ByteBuffer is reused over multiple comparator calls, by setting `maxReusedBufferSize > 0`. When buffers are reused, ComparatorOptions provides a choice of mutex type by setting `useAdaptiveMutex`. --- [JMH benchmarks previously indicated](https://github.com/facebook/rocksdb/pull/6241#issue-356398306) that the difference between C++ and Java for implementing a comparator was ~7x slowdown in Java. With these changes, when reusing buffers and guarding access to them via mutexes the slowdown is approximately the same. However, these changes offer a new facility to not reuse mutextes, which reduces the slowdown to ~5.5x in Java. We also offer a `thread_local` mechanism for reusing buffers, which reduces slowdown to ~5.2x in Java (closes https://github.com/facebook/rocksdb/pull/4425). These changes also form a good base for further optimisation work such as further JNI lookup caching, and JNI critical. --- These numbers were captured without jemalloc. With jemalloc, the performance improves for all tests, and the Java slowdown reduces to between 4.8x and 5.x. ``` ComparatorBenchmarks.put native_bytewise thrpt 25 124483.795 ± 2032.443 ops/s ComparatorBenchmarks.put native_reverse_bytewise thrpt 25 114414.536 ± 3486.156 ops/s ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 17228.250 ± 1288.546 ops/s ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 16035.865 ± 1248.099 ops/s ComparatorBenchmarks.put java_bytewise_non-direct_reused-64_thread-local thrpt 25 21571.500 ± 871.521 ops/s ComparatorBenchmarks.put java_bytewise_direct_reused-64_adaptive-mutex thrpt 25 23613.773 ± 8465.660 ops/s ComparatorBenchmarks.put java_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 16768.172 ± 5618.489 ops/s ComparatorBenchmarks.put java_bytewise_direct_reused-64_thread-local thrpt 25 23921.164 ± 8734.742 ops/s ComparatorBenchmarks.put java_bytewise_non-direct_no-reuse thrpt 25 17899.684 ± 839.679 ops/s ComparatorBenchmarks.put java_bytewise_direct_no-reuse thrpt 25 22148.316 ± 1215.527 ops/s ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_adaptive-mutex thrpt 25 11311.126 ± 820.602 ops/s ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_non-adaptive-mutex thrpt 25 11421.311 ± 807.210 ops/s ComparatorBenchmarks.put java_reverse_bytewise_non-direct_reused-64_thread-local thrpt 25 11554.005 ± 960.556 ops/s ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_adaptive-mutex thrpt 25 22960.523 ± 1673.421 ops/s ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_non-adaptive-mutex thrpt 25 18293.317 ± 1434.601 ops/s ComparatorBenchmarks.put java_reverse_bytewise_direct_reused-64_thread-local thrpt 25 24479.361 ± 2157.306 ops/s ComparatorBenchmarks.put java_reverse_bytewise_non-direct_no-reuse thrpt 25 7942.286 ± 626.170 ops/s ComparatorBenchmarks.put java_reverse_bytewise_direct_no-reuse thrpt 25 11781.955 ± 1019.843 ops/s ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6252 Differential Revision: D19331064 Pulled By: pdillinger fbshipit-source-id: 1f3b794e6a14162b2c3ffb943e8c0e64a0c03738 03 February 2020, 20:30:13 UTC
800d24d sdong 31 January 2020, 23:42:34 UTC Fix DBTest2.ChangePrefixExtractor LITE build (#6356) Summary: DBTest2.ChangePrefixExtractor fails in LITE build because LITE build doesn't support adaptive build. Fix it by removing the stats check but only check correctness. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6356 Test Plan: Run the test with both of LITE and non-LITE build. Differential Revision: D19669537 fbshipit-source-id: 6d7dd6c8a79f18e80ca1636864b9c71922030d8e 31 January 2020, 23:44:14 UTC
01ab882 Maysam Yabandeh 31 January 2020, 23:07:27 UTC Fix release warning for unused bg_canceled Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6357 Differential Revision: D19670931 Pulled By: maysamyabandeh fbshipit-source-id: d528c4c7f9450f1f38b9d2a36e0d5d0865b39be9 31 January 2020, 23:09:10 UTC
ec49634 sdong 31 January 2020, 19:00:24 UTC Add a unit test for prefix extractor changes (#6323) Summary: Add a unit test for prefix extractor change, including a check that fails due to a bug. Also comment out the partitioned filter case which will fail the test too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6323 Test Plan: Run the test and it passes (and fails if the SeekForPrev() part is uncommented) Differential Revision: D19509744 fbshipit-source-id: 678202ca97b5503e9de73b54b90de9e5ba822b72 31 January 2020, 19:02:03 UTC
2243030 Maysam Yabandeh 31 January 2020, 18:26:58 UTC Cancel bg jobs before deleting WritePrepared DB in stress tests (#6355) Summary: Background jobs in WritePrepared DB might access the db via a snapshot checker callback. The stress tests therefore should cancel background jobs before deleting the db in ::Reopen. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6355 Differential Revision: D19664132 Pulled By: maysamyabandeh fbshipit-source-id: 6060a830e8aad0015c10448286ad37c8a346ac01 31 January 2020, 18:29:11 UTC
3316d29 Maysam Yabandeh 31 January 2020, 15:26:46 UTC Disable recycle_log_file_num when it is incompatible with recovery mode (#6351) Summary: Non-zero recycle_log_file_num is incompatible with kPointInTimeRecovery and kAbsoluteConsistency recovery modes. Currently SanitizeOptions changes the recovery mode to kTolerateCorruptedTailRecords, while to resolve this option conflict it makes more sense to compromise recycle_log_file_num, which is a performance feature, instead of wal_recovery_mode, which is a safety feature. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6351 Differential Revision: D19648931 Pulled By: maysamyabandeh fbshipit-source-id: dd0bf78349edc007518a00c4d63931fd69294ad7 31 January 2020, 15:28:30 UTC
f2fbc5d Yanqin Jin 31 January 2020, 07:08:56 UTC Shorten certain test names to avoid infra failure (#6352) Summary: Unit test names, together with other components, are used to create log files during some internal testing. Overly long names cause infra failure due to file names being too long. Look for internal tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6352 Differential Revision: D19649307 Pulled By: riversand963 fbshipit-source-id: 6f29de096e33c0eaa87d9c8702f810eda50059e7 31 January 2020, 07:10:24 UTC
c9a5e48 Burton Li 31 January 2020, 00:05:44 UTC fix build warnnings on MSVC (#6309) Summary: Fix build warnings on MSVC. siying Pull Request resolved: https://github.com/facebook/rocksdb/pull/6309 Differential Revision: D19455012 Pulled By: ltamasi fbshipit-source-id: 940739f2c92de60e47cc2bed8dd7f921459545a9 31 January 2020, 00:07:26 UTC
90c71aa Peter Dillinger 30 January 2020, 19:00:08 UTC Don't download from (unreliable) maven.org (#6348) Summary: I set up a mirror of our Java deps on github so we can download them through github URLs rather than maven.org, which is proving terribly unreliable from Travis builds. Also sanitized calls to curl, so they are easier to read and appropriately fail on download failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6348 Test Plan: CI Differential Revision: D19633621 Pulled By: pdillinger fbshipit-source-id: 7eb3f730953db2ead758dc94039c040f406790f3 30 January 2020, 19:02:08 UTC
fb05b5a anand76 30 January 2020, 18:53:46 UTC Force a new manifest file if append to current one fails (#6331) Summary: Fix for issue https://github.com/facebook/rocksdb/issues/6316 When an append/sync of the manifest file fails due to an IO error such as NoSpace, we don't always put the DB in read-only mode. This is true for flush and compactions, as well as foreground operatons such as column family add/drop, CompactFiles etc. Subsequent changes to the DB will be recorded in the same manifest file, which would have a corrupted record in the middle due to the previous failure. On next DB::Open(), it will fail to process the full manifest and data will be lost. To fix this, we reset VersionSet::descriptor_log_ on append/sync failure, which will force a new manifest file to be written on the next append. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6331 Test Plan: Add new unit tests in error_handler_test.cc Differential Revision: D19632951 Pulled By: anand1976 fbshipit-source-id: 68d527cb6e59a94cbbbf9f5a17a7f464381d51e3 30 January 2020, 18:56:29 UTC
9e3ace4 Levi Tamasi 30 January 2020, 00:42:16 UTC Add statistics for BlobDB GC (#6296) Summary: The patch adds statistics support to the new BlobDB garbage collection implementation; namely, it adds support for the following (pre-existing) tickers: `BLOB_DB_GC_NUM_FILES`: the number of blob files obsoleted by the GC logic. `BLOB_DB_GC_NUM_NEW_FILES`: the number of new blob files generated by the GC logic. `BLOB_DB_GC_FAILURES`: the number of failed GC passes (where a GC pass is equivalent to a (sub)compaction). `BLOB_DB_GC_NUM_KEYS_RELOCATED`: the number of blobs relocated to new blob files by the GC logic. `BLOB_DB_GC_BYTES_RELOCATED`: the total size of blobs relocated to new blob files. The tickers `BLOB_DB_GC_NUM_KEYS_OVERWRITTEN`, `BLOB_DB_GC_NUM_KEYS_EXPIRED`, `BLOB_DB_GC_BYTES_OVERWRITTEN`, `BLOB_DB_GC_BYTES_EXPIRED`, and `BLOB_DB_GC_MICROS` are not relevant for the new GC logic, and are thus marked deprecated. The patch also adds a couple of log messages that log the number and total size of blobs encountered and relocated during a GC pass, as well as the number of blob files created and obsoleted. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6296 Test Plan: Extended unit tests and used the BlobDB mode of `db_bench`. Differential Revision: D19402513 Pulled By: ltamasi fbshipit-source-id: d53d2bfbf4928a1db1e9346c67ebb9007b8932ec 30 January 2020, 00:46:16 UTC
71874c5 sdong 30 January 2020, 00:42:15 UTC Fix LITE build with DBTest2.AutoPrefixMode1 (#6346) Summary: DBTest2.AutoPrefixMode1 doesn't pass because auto prefix mode is not supported there. Fix it by disabling the test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6346 Test Plan: Run DBTest2.AutoPrefixMode1 in lite mode Differential Revision: D19627486 fbshipit-source-id: fbde75260aeecb7e6fc406e09c19a71a95aa5f08 30 January 2020, 00:43:42 UTC
23dcf27 Peter Dillinger 29 January 2020, 23:57:13 UTC Upload DB dir for all crash tests (#6344) Summary: Difficult to root cause crash test failures without archiving db dir. Now all crash test configurations should save the db dir. Also exit with error code on bad command. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6344 Test Plan: Hmm, how about this: for TARGET in stress_crash asan_crash ubsan_crash tsan_crash; do EMAIL=email ONCALL=oncall TRIGGER=all SUBSCRIBER=sub build_tools/rocksdb-lego-determinator $TARGET > tmp && node -c tmp && grep -q Upload tmp || echo Bad; done Differential Revision: D19625605 Pulled By: pdillinger fbshipit-source-id: cb84aa93ee80b4534f4c61b90f0e0f99a41155d5 29 January 2020, 23:59:07 UTC
02ac6c9 sdong 29 January 2020, 20:55:58 UTC Fix db_bloom_filter_test clang LITE build (#6340) Summary: db_bloom_filter_test break with clang LITE build with following message: db/db_bloom_filter_test.cc:23:29: error: unused variable 'kPlainTable' [-Werror,-Wunused-const-variable] static constexpr PseudoMode kPlainTable = -1; ^ Fix it by moving the declaration out of LITE build Pull Request resolved: https://github.com/facebook/rocksdb/pull/6340 Test Plan: USE_CLANG=1 LITE=1 make db_bloom_filter_test and without LITE=1 Differential Revision: D19609834 fbshipit-source-id: 0e88f5c6759238a94f9880d84c785ac18e7cdd7e 29 January 2020, 20:57:48 UTC
2f973ca Maysam Yabandeh 29 January 2020, 19:39:11 UTC Double Crash in kPointInTimeRecovery with TransactionDB (#6313) Summary: In WritePrepared there could be gap in sequence numbers. This breaks the trick we use in kPointInTimeRecovery which assume the first seq in the log right after the corrupted log is one larger than the last seq we read from the logs. To let this trick keep working, we add a dummy entry with the expected sequence to the first log right after recovery. Also in WriteCommitted, if the log right after the corrupted log is empty, since it has no sequence number to let the sequential trick work, it is assumed as unexpected behavior. This is however expected to happen if we close the db after recovering from a corruption and before writing anything new to it. To remedy that, we apply the same technique by writing a dummy entry to the log that is created after the corrupted log. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6313 Differential Revision: D19458291 Pulled By: maysamyabandeh fbshipit-source-id: 09bc49e574690085df45b034ca863ff315937e2d 29 January 2020, 19:40:55 UTC
a07a9dc Adam Retter 29 January 2020, 16:00:16 UTC Reduce the need to re-download dependencies (#6318) Summary: Both changes are related to RocksJava: 1. Allow dependencies that are already present on the host system due to Maven to be reused in Docker builds. 2. Extend the `make clean-not-downloaded` target to RocksJava, so that libraries needed as dependencies for the test suite are not deleted and re-downloaded unnecessarily. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6318 Differential Revision: D19608742 Pulled By: pdillinger fbshipit-source-id: 25e25649e3e3212b537ac4512b40e2e53dc02ae7 29 January 2020, 16:01:56 UTC
8f2bee6 sdong 28 January 2020, 22:42:21 UTC Add ReadOptions.auto_prefix_mode (#6314) Summary: Add a new option ReadOptions.auto_prefix_mode. When set to true, iterator should return the same result as total order seek, but may choose to do prefix seek internally, based on iterator upper bounds. Also fix two previous bugs when handling prefix extrator changes: (1) reverse iterator should not rely on upper bound to determine prefix. Fix it with skipping prefix check. (2) block-based filter is not handled properly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6314 Test Plan: (1) add a unit test; (2) add the check to stress test and run see whether it can pass at least one run. Differential Revision: D19458717 fbshipit-source-id: 51c1bcc5cdd826c2469af201979a39600e779bce 28 January 2020, 22:44:05 UTC
431fb6c Siying Dong 28 January 2020, 22:38:57 UTC Add Google Group to Issue Template Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6339 Differential Revision: D19608457 fbshipit-source-id: 2adea28b1bd20b85ccafca1aa567030115220ea6 28 January 2020, 22:40:37 UTC
4f6c862 Sagar Vemuri 28 January 2020, 03:57:32 UTC Use the same oldest ancestor time in table properties and manifest Summary: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions passed 96 / 100 times. ``` With the fix: all runs (tried 100, 1000, 10000) succeed. ``` $ TEST_TMPDIR=/dev/shm ~/gtest-parallel/gtest-parallel ./db_compaction_test --gtest_filter=DBCompactionTest.LevelTtlCascadingCompactions --repeat=1000 [1000/1000] DBCompactionTest.LevelTtlCascadingCompactions (1895 ms) ``` Test Plan: Build: ``` COMPILE_WITH_TSAN=1 make db_compaction_test -j100 ``` Without the fix: a few runs out of 100 fail: ``` $ TEST_TMPDIR=/dev/shm KEEP_DB=1 ~/gtest-parallel/gtest-parallel ./db_compaction_test --gtest_filter=DBCompactionTest.LevelTtlCascadingCompactions --repeat=100 ... ... Note: Google Test filter = DBCompactionTest.LevelTtlCascadingCompactions [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from DBCompactionTest [ RUN ] DBCompactionTest.LevelTtlCascadingCompactions db/db_compaction_test.cc:3687: Failure Expected equality of these values: oldest_time Which is: 1580155869 level_to_files[6][0].oldest_ancester_time Which is: 1580155870 DB is still at /dev/shm//db_compaction_test_6337001442947696266 [ FAILED ] DBCompactionTest.LevelTtlCascadingCompactions (1432 ms) [----------] 1 test from DBCompactionTest (1432 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1433 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DBCompactionTest.LevelTtlCascadingCompactions 1 FAILED TEST [80/100] DBCompactionTest.LevelTtlCascadingCompactions returned/aborted with exit code 1 (1489 ms) [100/100] DBCompactionTest.LevelTtlCascadingCompactions (1522 ms) FAILED TESTS (4/100): 1419 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/90) 1434 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/84) 1457 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/82) 1489 ms: ./db_compaction_test DBCompactionTest.LevelTtlCascadingCompactions (try https://github.com/facebook/rocksdb/issues/74) Differential Revision: D19587040 Pulled By: sagar0 fbshipit-source-id: 11191ae9940837643bff47ebe18b299b4be3d950 28 January 2020, 03:58:53 UTC
7aa66c7 sdong 28 January 2020, 01:44:17 UTC Move HISTORY.md entry of hash index fix from 6.7 to unreleased (#6337) Summary: Commits related to hash index fix have been reverted in 6.7.fb branch. Update HISTORY.md to keep it in sync. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6337 Differential Revision: D19593717 fbshipit-source-id: 466178dc6205c9e41ccced41bf281a0952bdc2ca 28 January 2020, 01:45:42 UTC
5b33cfa Andrew Kryczka 27 January 2020, 23:47:11 UTC fix `WriteBufferManager` flush log message (#6335) Summary: It chooses the oldest memtable, not the largest one. This is an important difference for users whose CFs receive non-uniform write rates. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6335 Differential Revision: D19588865 Pulled By: maysamyabandeh fbshipit-source-id: 62ad4325b0182f5f27858584cd73fd5978fb2cec 27 January 2020, 23:49:22 UTC
f10f135 sdong 27 January 2020, 23:41:57 UTC Fix regression bug of hash index with iterator total order seek (#6328) Summary: https://github.com/facebook/rocksdb/pull/6028 introduces a bug for hash index in SST files. If a table reader is created when total order seek is used, prefix_extractor might be passed into table reader as null. While later when prefix seek is used, the same table reader used, hash index is checked but prefix extractor is null and the program would crash. Fix the issue by fixing http://github.com/facebook/rocksdb/pull/6028 in the way that prefix_extractor is preserved but ReadOptions.total_order_seek is checked Also, a null pointer check is added so that a bug like this won't cause segfault in the future. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6328 Test Plan: Add a unit test that would fail without the fix. Stress test that reproduces the crash would pass. Differential Revision: D19586751 fbshipit-source-id: 8de77690167ddf5a77a01e167cf89430b1bfba42 27 January 2020, 23:44:54 UTC
986df37 Peter Dillinger 27 January 2020, 21:13:27 UTC Clean up PartitionedFilterBlockBuilder (#6299) Summary: Remove the redundant PartitionedFilterBlockBuilder::num_added_ and ::NumAdded since the parent class, FullFilterBlockBuilder, already provides them. Also rename filters_in_partition_ and filters_per_partition_ to keys_added_to_partition_ and keys_per_partition_ to improve readability. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6299 Test Plan: make check Differential Revision: D19413278 Pulled By: pdillinger fbshipit-source-id: 04926ee7874477d659cb2b6ae03f2d995fb747e5 27 January 2020, 21:15:14 UTC
bd698e4 Fosco Marotto 24 January 2020, 23:35:04 UTC Update version for next release, 6.7.0 (#6320) Summary: Adjusted history for 6.6.1 and 6.6.2, switched master version to 6.7.0. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6320 Differential Revision: D19499272 Pulled By: gfosco fbshipit-source-id: 2bafb2456951f231e411e9c03aaa4c044f497684 24 January 2020, 23:36:32 UTC
c4bc30e Maysam Yabandeh 24 January 2020, 21:03:19 UTC Implement PinnableSlice::remove_prefix (#6330) Summary: The function was left unimplemented. Although we currently don't have a use for that it was declared with an assert(0) to prevent mistakenly using the remove_prefix of the parent class. The function body with only assert(0) however causes issues with some compiler's warning levels. The patch implements the function to avoid the warning. It also piggybacks some minor code warning for unnecessary semicolons after the function definition.s Pull Request resolved: https://github.com/facebook/rocksdb/pull/6330 Differential Revision: D19559062 Pulled By: maysamyabandeh fbshipit-source-id: 3a022484f688c9abd4556e5412bcc2628ab96a00 24 January 2020, 21:04:53 UTC
f34782a Levi Tamasi 23 January 2020, 23:25:23 UTC Fix the "records dropped" statistics (#6325) Summary: The earlier code used two conflicting definitions for the number of input records going into a compaction, one based on the `rocksdb.num.entries` table property and one based on `CompactionIterationStats`. The first one is correct and in line with how output records are counted, while the second one incorrectly ignores input records in various cases when the `CompactionIterator` advances or reseeks the input iterator (this can happen, amongst other cases, when dealing with `SingleDelete`s, regular `Delete`s, `Merge`s, and compaction filters). This can result in the code undercounting the input records and computing an incorrect value for "records dropped" during the compaction. The patch fixes this by switching over to the correct (table property based) input record count for "records dropped". Pull Request resolved: https://github.com/facebook/rocksdb/pull/6325 Test Plan: Tested using `make check` and `db_bench`. Differential Revision: D19525491 Pulled By: ltamasi fbshipit-source-id: 4340b0b2f41546db8e356db70ca02199e48fa636 23 January 2020, 23:27:22 UTC
0672a6d anand76 23 January 2020, 21:59:48 UTC Fix queue manipulation in WriteThread::BeginWriteStall() (#6322) Summary: When there is a write stall, the active write group leader calls ```BeginWriteStall()``` to walk the queue of writers and remove any with the ```no_slowdown``` option set. There was a bug in the code which updated the back pointer but not the forward pointer (```link_newer```), corrupting the list and causing some threads to wait forever. This PR fixes it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6322 Test Plan: Add a unit test in db_write_test Differential Revision: D19538313 Pulled By: anand1976 fbshipit-source-id: 6fbed819e594913f435886606f5d36f74f235c3a 23 January 2020, 22:01:28 UTC
967a2d9 Maysam Yabandeh 23 January 2020, 17:07:50 UTC Revert "crash_test to enable block-based table hash index (#6310)" (#6327) Summary: This reverts commit 8e309b35bbc00aebf29038ac5936f704d22ea303. The stress tests are failing . Revert it until we figure the root cause. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6327 Differential Revision: D19537657 Pulled By: maysamyabandeh fbshipit-source-id: bf34a5dd720825957729e136e9a5a729a240e61a 23 January 2020, 17:09:17 UTC
back to top