Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
32fda17 sdong 05 November 2019, 18:57:56 UTC Disable error as warning 05 November 2019, 19:05:04 UTC
2594970 sdong 05 November 2019, 17:25:25 UTC Add one more #include<functional> 05 November 2019, 19:05:03 UTC
1b4234f sdong 31 October 2019, 21:17:26 UTC Add some include<functional> 31 October 2019, 21:23:01 UTC
54cecea sdong 31 October 2019, 21:10:18 UTC [FB Internal] Point to the latest tool chain. 31 October 2019, 21:22:10 UTC
9ec602f sdong 18 July 2017, 05:17:47 UTC [fb only] revert unintended change of USE_SSE The previuos change that use gcc-5 set USE_SSE to wrong flag by mistake. Fix it. 18 July 2017, 05:22:50 UTC
6d1896f sdong 17 July 2017, 22:39:37 UTC [FB Only] use gcc-5 18 July 2017, 04:56:58 UTC
9808491 sdong 16 September 2016, 01:11:36 UTC Release 4.11.2 16 September 2016, 01:11:36 UTC
fbd1c98 Andrew Kryczka 15 September 2016, 18:40:48 UTC Fix recovery for WALs without data for all CFs Summary: if one or more CFs had no data in the WAL, the log number that's used by FindObsoleteFiles() wasn't updated. We need to treat this case the same as if the data for that WAL had been flushed. Test Plan: new unit test Reviewers: IslamAbdelRahman, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D63963 16 September 2016, 01:03:33 UTC
c743289 Andrew Kryczka 15 September 2016, 16:55:02 UTC Fix GetSortedWalFiles when log recycling enabled Summary: Previously the sequence number was mistakenly passed in an argument where the log number should go. This caused the reader to assume the old WAL format was used, which is incompatible with the WAL recycling format. Test Plan: new unit test, verified it fails before this change and passes afterwards. Reviewers: yiwu, lightmark, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D63987 16 September 2016, 01:03:24 UTC
2be165b Mike Kolupaev 25 August 2016, 11:39:26 UTC Fix a crash when compaction fails to open a file Summary: We've got a crash with this stack trace: Program terminated with signal SIGTRAP, Trace/breakpoint trap. #0 0x00007fc85f2f4009 in raise () from /usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/libpthread.so.0 #1 0x00000000005c8f61 in facebook::logdevice::handle_sigsegv(int) () at logdevice/server/sigsegv.cpp:159 #2 0x00007fc85f2f4150 in <signal handler called> () at /usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/libpthread.so.0 #3 0x00000000031ed80c in rocksdb::NewReadaheadRandomAccessFile() at util/file_reader_writer.cc:383 #4 0x00000000031ed80c in rocksdb::NewReadaheadRandomAccessFile() at util/file_reader_writer.cc:472 #5 0x00000000031558e7 in rocksdb::TableCache::GetTableReader() at db/table_cache.cc:99 #6 0x0000000003156329 in rocksdb::TableCache::NewIterator() at db/table_cache.cc:198 #7 0x0000000003166568 in rocksdb::VersionSet::MakeInputIterator() at db/version_set.cc:3345 #8 0x000000000324a94f in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) () at db/compaction_job.cc:650 #9 0x000000000324c2f6 in rocksdb::CompactionJob::Run() () at db/compaction_job.cc:530 #10 0x00000000030f5ae5 in rocksdb::DBImpl::BackgroundCompaction() at db/db_impl.cc:3269 #11 0x0000000003108d36 in rocksdb::DBImpl::BackgroundCallCompaction(void*) () at db/db_impl.cc:2970 #12 0x00000000029a2a9a in facebook::logdevice::RocksDBEnv::callback(void*) () at logdevice/server/locallogstore/RocksDBEnv.cpp:26 #13 0x00000000029a2a9a in facebook::logdevice::RocksDBEnv::callback(void*) () at logdevice/server/locallogstore/RocksDBEnv.cpp:30 #14 0x00000000031e7521 in rocksdb::ThreadPool::BGThread() at util/threadpool.cc:230 #15 0x00000000031e7663 in rocksdb::BGThreadWrapper(void*) () at util/threadpool.cc:254 #16 0x00007fc85f2ea7f1 in start_thread () at /usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/libpthread.so.0 #17 0x00007fc85e8fb46d in clone () at /usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/libc.so.6 From looking at the code, probably what happened is this: - `TableCache::GetTableReader()` called `Env::NewRandomAccessFile()`, which dispatched to a `PosixEnv::NewRandomAccessFile()`, where probably an `open()` call failed, so the `NewRandomAccessFile()` left a nullptr in the resulting file, - `TableCache::GetTableReader()` called `NewReadaheadRandomAccessFile()` with that `nullptr` file, - it tried to call file's method and crashed. This diff is a trivial fix to this crash. Test Plan: `make -j check` Reviewers: sdong, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D62451 16 September 2016, 00:59:07 UTC
0f4c98d sdong 30 August 2016, 18:55:08 UTC Increase version to 4.11.1 Summary: After back-porting two bug fixes, increase the version to 4.11.1 30 August 2016, 18:55:08 UTC
c75f4fa Aaron Gao 24 August 2016, 01:20:41 UTC fix data race in NewIndexIterator() in block_based_table_reader.cc Summary: fixed data race described in https://github.com/facebook/rocksdb/issues/1267 and add regression test Test Plan: ./table_test --gtest_filter=BlockBasedTableTest.NewIndexIteratorLeak make all check -j64 core dump before fix. ok after fix. Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: igor, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D62361 30 August 2016, 18:37:23 UTC
56f497b sdong 25 August 2016, 21:43:12 UTC Mitigate regression bug of options.max_successive_merges hit during DB Recovery Summary: After 1b8a2e8fdd1db0dac3cb50228065f8e7e43095f0, DB Pointer is passed to WriteBatchInternal::InsertInto() while DB recovery. This can cause deadlock if options.max_successive_merges hits. In that case DB::Get() will be called. Get() will try to acquire the DB mutex, which is already held by the DB::Open(), causing a deadlock condition. This commit mitigates the problem by not passing the DB pointer unless 2PC is allowed. Test Plan: Add a new test and run it. Reviewers: IslamAbdelRahman, andrewkr, kradhakrishnan, horuff Reviewed By: kradhakrishnan Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D62625 30 August 2016, 18:24:55 UTC
051c88d sdong 15 August 2016, 17:29:29 UTC Compaction stats printing: "batch" => "commit group" Summary: "Batch" is ambiguous in this context. It can mean "write batch" or commit group. Change it to commit group to be clear. Test Plan: Build Reviewers: MarkCallaghan, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D62055 15 August 2016, 18:26:01 UTC
071893e Yueh-Hsuan Chiang 15 August 2016, 16:04:55 UTC Fix a destruction order issue in ThreadStatusUpdater Summary: Env holds a pointer of ThreadStatusUpdater, which will be deleted when Env is deleted. However, in case a rocksdb database is deleted after Env is deleted. Then this will introduce a free-after-use of this ThreadStatusUpdater. This patch fix this by never deleting the ThreadStatusUpdater in Env, which is in general safe as Env is a singleton in most cases. Test Plan: thread_list_test Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59187 15 August 2016, 17:41:53 UTC
7d7234a sdong 05 August 2016, 22:55:34 UTC Change HISTORY.md for release 4.11 Summary: Need to change HISTORY.md for 4.11. 4.10 was not updated either. Update it together. Test Plan: Not needed. Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61665 10 August 2016, 20:19:48 UTC
d641bb8 sdong 04 August 2016, 00:42:06 UTC read_options.background_purge_on_iterator_cleanup to cover forward iterator and log file closing too. Summary: With read_options.background_purge_on_iterator_cleanup=true, File deletion and closing can still happen in forward iterator, or WAL file closing. Cover those cases too. Test Plan: I am adding unit tests. Reviewers: andrewkr, IslamAbdelRahman, yiwu Reviewed By: yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61503 10 August 2016, 20:19:10 UTC
e98b76d sdong 14 July 2016, 04:08:15 UTC A utility function to help users migrate DB after options change Summary: Add a utility function that trigger necessary full compaction and put output to the correct level by looking at new options and old options. Test Plan: Add unit tests for it. Reviewers: andrewkr, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: muthu, sumeet, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60783 05 August 2016, 23:09:33 UTC
4001e79 Yi Wu 03 August 2016, 04:55:26 UTC Ignore write stall triggers when auto-compaction is disabled Summary: My understanding is that the purpose of write stall triggers are to wait for auto-compaction to catch up. Without auto-compaction, we don't need to stall writes. Also with this diff, flush/compaction conditions are recalculated on dynamic option change. Previously the conditions are recalculate only when write stall options are changed. Test Plan: See the new test. Removed two tests that are no longer valid. Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61437 05 August 2016, 23:09:12 UTC
9ae92f5 Adam Retter 29 July 2016, 19:55:54 UTC More granular steps in the Makefile, can help with running all or single Java tests (and with ASAN build - https://github.com/facebook/rocksdb/wiki/JNI-Debugging) (#1237) 29 July 2016, 19:55:54 UTC
7c01d65 Islam AbdelRahman 29 July 2016, 19:47:15 UTC [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv Summary: Remove duplicate cases in LoggerJniCallback::Logv and fix complication error Test Plan: make jclean rocksdbjava jtest -j4 Reviewers: adamretter, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61335 29 July 2016, 19:47:15 UTC
8796934 Adam Retter 29 July 2016, 19:12:03 UTC Added missing Java ReadOptions settings (#1109) * Minor reorganisation * Java API - Added missing ReadOptions settings 29 July 2016, 19:12:03 UTC
5e2c796 Islam AbdelRahman 29 July 2016, 18:42:28 UTC Make DBTest.CompressionStatsTest more deterministic Summary: DBTest.CompressionStatsTest on non_shm test where the storage device is slow DBTest.CompressionStatsTest assumes that a flush happens to check the number of compressed blocks. This is not always true if the Flush is slow, make the test more deterministic by forcing a flush before doing the check Test Plan: Run the test locally Reviewers: andrewkr, yiwu, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61317 29 July 2016, 18:42:28 UTC
557748f Islam AbdelRahman 29 July 2016, 18:02:03 UTC Fix db_stress failure (pass merge_operator even if not used) Summary: db_stress test is now failing because of this scenario - run db_stress with merge_operator enabled (now we have a db with merge operands) - run db_stress with merge_operator disabled (now when we fail to open the db) the solution is to pass the merge_operator to the DB even if we are not going to do any merge operations Test Plan: Check the failure Reviewers: andrewkr, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61311 29 July 2016, 18:02:03 UTC
811ee21 Adam Retter 28 July 2016, 23:54:06 UTC Bugfix to ensure that logging can be achieved from threads that are not known to the JVM (#1106) 28 July 2016, 23:54:06 UTC
afad5bd Andrew Kryczka 28 July 2016, 21:55:22 UTC Simplify thread-local static initialization Summary: The call stack used to look like this during static initialization: #0 0x00000000008032d1 in rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta() (this=0x7ffff683b300) at util/thread_local.cc:172 #1 0x00000000008030a7 in rocksdb::ThreadLocalPtr::Instance() () at util/thread_local.cc:135 #2 0x000000000080310f in rocksdb::ThreadLocalPtr::StaticMeta::Mutex() () at util/thread_local.cc:141 #3 0x0000000000803103 in rocksdb::ThreadLocalPtr::StaticMeta::InitSingletons() () at util/thread_local.cc:139 #4 0x000000000080305d in rocksdb::ThreadLocalPtr::InitSingletons() () at util/thread_local.cc:106 It involves outer/inner classes and the call stacks goes outer->inner->outer->inner, which is too difficult to understand. We can avoid a level of back-and-forth by skipping StaticMeta::InitSingletons(), which doesn't initialize anything beyond what ThreadLocalPtr::Instance() already initializes. Now the call stack looks like this during static initialization: #0 0x00000000008032c5 in rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta() (this=0x7ffff683b300) at util/thread_local.cc:170 #1 0x00000000008030a7 in rocksdb::ThreadLocalPtr::Instance() () at util/thread_local.cc:135 #2 0x000000000080305d in rocksdb::ThreadLocalPtr::InitSingletons() () at util/thread_local.cc:106 Test Plan: unit tests verify StaticMeta::mutex_ is still initialized in DefaultEnv() (StaticMeta::mutex_ is the only variable intended to be initialized via StaticMeta::InitSingletons() which I removed) #0 0x00000000005cee17 in rocksdb::port::Mutex::Mutex(bool) (this=0x7ffff69500b0, adaptive=false) at port/port_posix.cc:52 #1 0x0000000000769cf8 in rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta() (this=0x7ffff6950000) at util/thread_local.cc:168 #2 0x0000000000769a53 in rocksdb::ThreadLocalPtr::Instance() () at util/thread_local.cc:133 #3 0x0000000000769a09 in rocksdb::ThreadLocalPtr::InitSingletons() () at util/thread_local.cc:105 #4 0x0000000000647d98 in rocksdb::Env::Default() () at util/env_posix.cc:845 Reviewers: lightmark, yhchiang, sdong Reviewed By: sdong Subscribers: arahut, IslamAbdelRahman, yiwu, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60813 28 July 2016, 21:55:22 UTC
6920cde Gunnar Kudrjavets 28 July 2016, 04:14:02 UTC Remove an extra apostrophe 28 July 2016, 04:16:11 UTC
e72ea48 Aaron Gao 28 July 2016, 01:45:53 UTC add InDomain regression test Summary: regression tests to make sure seek keys not in domain would not fail assertion Test Plan: ``` [gzh@dev6163.prn2 ~/local/rocksdb] ./prefix_test --gtest_filter=SamePrefixTest.* /tmp/rocksdbtest-112628/prefix_test Note: Google Test filter = SamePrefixTest.* [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from SamePrefixTest [ RUN ] SamePrefixTest.InDomainTest [ OK ] SamePrefixTest.InDomainTest (211 ms) [----------] 1 test from SamePrefixTest (211 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (211 ms total) ``` Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61161 28 July 2016, 01:45:53 UTC
9c8ac14 Gunnar Kudrjavets 27 July 2016, 23:37:12 UTC Avoid duplicate task creation for RocksDB contruns Summary: We rely on a basic evaluation logic: when using an expression `A || B`, the `B` part will only get evaluated when `A` fails. Therefore the task creation tool is guaranteed to run if previous build/test step failed. To indicate the correct return value to shell, the task creation tool will call `exit(1)` which will cause Sandcastle to mark it as a failure. Test Plan: - Land the changes. - Trigger a RocksDB contrun - observe the results. - Look at the results from the nightly runs and fix issues as necessary. Current testing so far has been in isolation for various components. Reviewers: andrewkr, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60909 27 July 2016, 23:37:12 UTC
d4c4542 sdong 25 July 2016, 19:27:06 UTC db_stress shouldn't assert file size 0 if file creation fails Summary: OnTableFileCreated() now is also called when the file creaion fails. In that case, we shouldn't assert the file size is not 0. Test Plan: Run crash test Reviewers: yiwu, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61137 27 July 2016, 19:08:16 UTC
d3bfd33 Anirban Rahut 27 July 2016, 18:58:57 UTC Testing out parallel sandcastle changes Summary: Removing moreutils from sandcastle and adding gnu parallel. Then passing in J= nproc command Test Plan: Testing on sandcastle Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61017 27 July 2016, 18:58:57 UTC
7efd9c2 sdong 27 July 2016, 18:16:55 UTC Increse timeout in some tests Summary: Increase test timeout to some tests to unblock CI. Test Plan: watch how it runs. Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61263 27 July 2016, 18:39:36 UTC
50b8d29 Alexander Jipa 27 July 2016, 03:29:02 UTC fixes 1230: Error:string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command (#1231) quoting the las argument of string REGEXP REPLACE function 27 July 2016, 03:29:02 UTC
5c858dd woonhak.kang 27 July 2016, 02:28:45 UTC fix errata in libnuma test (#1244) 27 July 2016, 02:28:45 UTC
e5b5f12 sdong 27 July 2016, 01:05:30 UTC Change options memtable_prefix_bloom_huge_page_tlb_size => memtable_huge_page_size and cover huge page to memtable too Summary: Extend the option memtable_prefix_bloom_huge_page_tlb_size from just putting memtable bloom filter to huge page to memtable itself too. Test Plan: Run all existing tests. Reviewers: IslamAbdelRahman, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60513 27 July 2016, 01:15:11 UTC
0ce258f sdong 26 July 2016, 01:08:49 UTC Compaction picker to expand output level files for keys cross files' boundary too. Summary: We may wrongly drop delete operation if we pick a file with the entry to be delete, the put entry of the same user key is in the next file in the level, and the next file is not picked. We expand compaction inputs for output level too. Test Plan: Add unit tests that reproduct the bug of dropping delete entry. Change compaction_picker_test to assert the new behavior. Reviewers: IslamAbdelRahman, igor Reviewed By: igor Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61173 27 July 2016, 00:56:36 UTC
ac0d93b Alexander Jipa 26 July 2016, 23:18:49 UTC fixes 1217: rocksdbjni javac and javah execute_processes fail on windows (#1218) picking execute_process command based on OS 26 July 2016, 23:18:49 UTC
1ae4609 Islam AbdelRahman 26 July 2016, 20:54:08 UTC Appveyor badge to show master branch appveyor badge to show master branch 26 July 2016, 20:54:08 UTC
8745f01 Islam AbdelRahman 26 July 2016, 20:18:31 UTC [Fix java build] Stop using non standard std::make_unique Summary: std::make_unique is not standard and not always available, remove it Test Plan: Run "make clean jclean rocksdbjava jtest -j8" on my mac Reviewers: yhchiang, yiwu, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61143 26 July 2016, 20:18:31 UTC
e12270d Wanning Jiang 26 July 2016, 18:15:14 UTC fix previous typo Summary: old typos with FILTER/INDEX_CACHE Test Plan: still pass this unit test Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61185 26 July 2016, 18:15:14 UTC
bbd6a5a Andrew Kryczka 26 July 2016, 18:13:26 UTC ldb restore subcommand Summary: - Added a new subcommand, 'ldb restore', that restores from backup - Made backup_env_uri optional (also for 'ldb backup') because it can use db_env when backup_env isn't provided Test Plan: verify backup and restore commands work: $ ./ldb backup --db=/data/users/andrewkr/rocksdb/db_bench-out/ --num_threads 1 --backup_dir=/data/users/andrewkr/rocksdb/db_bench-out/backup/ $ ./ldb restore --db=/data/users/andrewkr/rocksdb/db_bench-out/restore/ --num_threads 1 --backup_dir=/data/users/andrewkr/rocksdb/db_bench-out/backup/ Reviewers: sdong, wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60849 26 July 2016, 18:13:26 UTC
9498069 Andrew Kryczka 25 July 2016, 23:37:23 UTC Run error-filtering script on diff-triggered tests Summary: this script creates a summary message based on the test type and output. Previously it only ran during contbuild. This diff makes it also run on commit-triggered diffs. Test Plan: will see if it works on diff's sandcastle tests Reviewers: kradhakrishnan, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60879 25 July 2016, 23:37:23 UTC
f8061a2 Islam AbdelRahman 25 July 2016, 23:05:50 UTC Fix Statistics TickersNameMap miss match with Tickers enum Summary: TickersNameMap is not consistent with Tickers enum. this cause us to report wrong statistics and sometimes to access TickersNameMap outside it's boundary causing crashes (in Fb303 statistics) Test Plan: added new unit test Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61083 25 July 2016, 23:05:50 UTC
16e225f Islam AbdelRahman 25 July 2016, 22:31:41 UTC Fix MergeContext::copied_operands_ strings moving Summary: MergeContext::copied_operands contain strings that MergeContext::operand_list_ Slices point to It's possible that when MergeContext::copied_operands grow, these strings are moved and there place in memory is changed, this will cause MergeContext::operand_list_ to point to invalid memory. fix this problem by using unique_ptr<string> instead of string Test Plan: run tests under mac/clang Reviewers: sdong, yiwu Reviewed By: yiwu Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61023 25 July 2016, 22:31:41 UTC
a4955b3 krad 25 July 2016, 21:17:28 UTC Run sandcastle tests in /dev/shm 25 July 2016, 22:10:24 UTC
ae0ad71 Yi Wu 25 July 2016, 22:09:34 UTC Fix flaky DBSSTTEST::DeleteObsoleteFilesPendingOutputs Summary: The test is flaky on Travis in osx environment. The background flush the test wanting to block can run behind the L2 manual compaction, making the test actually blocking the L2 compaction and won't able to proceed. Test Plan: Test run on travis Reviewers: kradhakrishnan, sdong, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61101 25 July 2016, 22:09:34 UTC
b2a8016 zhang-tong 25 July 2016, 21:50:44 UTC Update db_bench_tool.cc (#1239) * Update db_bench_tool.cc * Update db_bench_tool.cc 25 July 2016, 21:50:44 UTC
c665458 Yi Wu 25 July 2016, 18:48:17 UTC Disable two dynamic options tests under lite build Summary: RocksDB lite don't support dynamic options. Disable the two test from lite build, and assert `SetOptions` should return `status::OK`. Test Plan: Run the db_options test under lite build and normal build. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61119 25 July 2016, 18:48:17 UTC
2a6d0cd sdong 19 July 2016, 18:48:00 UTC Ignore stale logs while restarting DBs Summary: Stale log files can be deleted out of order. This can happen for various reasons. One of the reason is that no data is ever inserted to a column family and we have an optimization to update its log number, but not all the old log files are cleaned up (the case shown in the unit tests added). It can also happen when we simply delete multiple log files out of order. This causes data corruption because we simply increase seqID after processing the next row and we may end up with writing data with smaller seqID than what is already flushed to memtables. In DB recovery, for the oldest files we are replaying, if there it contains no data for any column family, we ignore the sequence IDs in the file. Test Plan: Add two unit tests that fail without the fix. Reviewers: IslamAbdelRahman, igor, yiwu Reviewed By: yiwu Subscribers: hermanlee4, yoshinorim, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60891 25 July 2016, 18:47:31 UTC
ee8bf2e Alexander Jipa 24 July 2016, 10:19:00 UTC fixes 1228: rockdbjni loadLibraryFromJarToTemp fails when file is already present (#1232) overriding existing library in tmp folder 24 July 2016, 10:19:00 UTC
f85df12 sdong 22 July 2016, 18:46:13 UTC Re-enable tsan crash white-box test with reduced killing odds Summary: Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds. Add a parameter in crash test script to allow we pass it through an environment variable. Test Plan: Run it manually. Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61053 22 July 2016, 19:51:27 UTC
89e4c48 Islam AbdelRahman 22 July 2016, 19:03:21 UTC Update README.md to include appveyor badge Update README.md to include appveyor badge 22 July 2016, 19:03:21 UTC
b06ca5f Peter (Stig) Edwards 22 July 2016, 18:46:40 UTC ldb load, prefer ifsteam(/dev/stdin) to std::cin (#1207) getline on std::cin can be very inefficient when ldb is loading large values, with high CPU usage in libc _IO_(un)getc, this is because of the performance penalty that comes from synchronizing stdio and iostream buffers. See the reproducers and tests in #1133 . If an ifstream on /dev/stdin is used (when available) then using ldb to load large values can be much more efficient. I thought for ldb load, that this approach is preferable to using <cstdio> or std::ios_base::sync_with_stdio(false). I couldn't think of a use case where ldb load would need to support reading unbuffered input, an alternative approach would be to add support for passing --input_file=/dev/stdin. I have a CLA in place, thanks. The CI tests were failing at the time of https://github.com/facebook/rocksdb/pull/1156, so this change and PR will supersede it. 22 July 2016, 18:46:40 UTC
4ea0ab3 sdong 22 July 2016, 17:42:56 UTC Revert "Remove bashism from `make check` (#1225)" This reverts commit 08ab1d83ac01c5e36673e844f3590f8ae14a6f2b. 22 July 2016, 17:42:56 UTC
12767b3 Alexander Jipa 22 July 2016, 17:17:10 UTC fixes 1220: rocksjni build fails on Windows due to variable-size array declaration (#1223) * fixes 1220: rocksjni build fails on Windows due to variable-size array declaration using new keyword to create variable-size arrays in order to satisfy most of the compilers * fixes 1220: rocksjni build fails on Windows due to variable-size array declaration using unique_ptr keyword to create variable-size arrays in order to satisfy most of the compilers 22 July 2016, 17:17:10 UTC
a9d512a Andrew Kryczka 22 July 2016, 08:13:25 UTC Update .gitignore for internal release 22 July 2016, 08:17:03 UTC
d5a51d4 sdong 19 July 2016, 22:12:46 UTC Need to make sure log file synced before flushing memtable of one column family Summary: Multiput atomiciy is broken across multiple column families if we don't sync WAL before flushing one column family. The WAL file may contain a write batch containing writes to a key to the CF to be flushed and a key to other CF. If we don't sync WAL before flushing, if machine crashes after flushing, the write batch will only be partial recovered. Data to other CFs are lost. Test Plan: Add a new unit test which will fail without the diff. Reviewers: yhchiang, IslamAbdelRahman, igor, yiwu Reviewed By: yiwu Subscribers: yiwu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60915 21 July 2016, 23:29:06 UTC
89f319c Yi Wu 21 July 2016, 21:32:12 UTC Fix unit test which breaks lite build Summary: Comment out assertion of number of table files from lite build. Test Plan: OPT=-DROCKSDB_LITE make check Reviewers: lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60999 21 July 2016, 21:32:12 UTC
b506329 sdong 21 July 2016, 19:53:02 UTC Add unit test not on /dev/shm as part of the pre-commit tests Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system. Test Plan: See the tests launched by Phabricator Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku Reviewed By: gunnarku Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60963 21 July 2016, 19:53:23 UTC
b9a9718 krad 21 July 2016, 18:05:27 UTC Bump next release version Summary: Bump master version to the next potential release version of 4.11 Test Plan: Compile Reviewers: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60993 21 July 2016, 19:22:14 UTC
663afef ryneli 21 July 2016, 18:16:34 UTC Add EnvLibrados - RocksDB Env of RADOS (#1222) EnvLibrados is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all file system related API of default Env. The easiest way to use it is just like following: std::string db_name = "test_db"; std::string config_path = "path/to/ceph/config"; DB* db; Options options; options.env = EnvLibrados(db_name, config_path); Status s = DB::Open(options, kDBPath, &db); Then EnvLibrados will forward all file read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool". There are some options that users could set for EnvLibrados. - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buffer_max_size, EnvLibrados will sync buffer content to RADOS, then clear buffer. - db_pool. Rather than using default pool, users could set their own db pool name - wal_dir. The dir for WAL files. Because RocksDB only has 2-level structure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal_dir is "/wal". - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool" The example of setting options looks like following: db_name = "test_db"; db_pool = db_name+"_pool"; wal_dir = "/wal"; wal_pool = db_name+"_wal_pool"; write_buffer_size = 1 << 20; env_ = new EnvLibrados(db_name, config, db_pool, wal_dir, wal_pool, write_buffer_size); DB* db; Options options; options.env = env_; // The last level dir name should match the dir name in prefix_pool_map options.wal_dir = "/tmp/wal"; // open DB Status s = DB::Open(options, kDBPath, &db); Librados is required to compile EnvLibrados. Then use "$make LIBRADOS=1" to compile RocksDB. If you want to only compile EnvLibrados test, just run "$ make env_librados_test LIBRADOS=1". To run env_librados_test, you need to have a running RADOS cluster with the configure file located in "../ceph/src/ceph.conf" related to "rocksdb/". 21 July 2016, 18:16:34 UTC
32604e6 Yi Wu 21 July 2016, 17:10:41 UTC Fix flush not being commit while writing manifest Summary: Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075. The issue: # Options.max_background_flushes > 1 # Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.) # Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit. # After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted. Test Plan: run the test. Also verify the new test hit deadlock without the fix. Reviewers: sdong, igor, lightmark Reviewed By: lightmark Subscribers: andrewkr, omegaga, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60969 21 July 2016, 17:10:41 UTC
9ab38c4 John Alexander 20 July 2016, 22:28:04 UTC Remove %z Format Specifier and Fix Windows Build of sim_cache.cc (#1224) * Replace %zu format specifier with Windows-compatible macro 'ROCKSDB_PRIszt' * Added "port/port.h" include to sim_cache.cc for call to snprintf(). * Applied cleaner fix to windows build, reverting part of 7bedd94 20 July 2016, 22:28:04 UTC
08ab1d8 Volker Mische 20 July 2016, 21:41:11 UTC Remove bashism from `make check` (#1225) The tests run by `make check` require Bash. On Debian you'd need to run the test as `make SHELL=/bin/bash check`. This commit makes it work on all POSIX compatible shells (tested on Debian with `dash`). 20 July 2016, 21:41:11 UTC
f9b14be sdong 20 July 2016, 18:14:51 UTC Re-enable TSAN crash test but only with black box crash test Summary: I tried on my host and TSAN black box test runs well. I didn't see any problem with white box crash test too but it chance of hitting crash point is too low that it may run almost forever. First re-enable black box crash test to unblock the job. Test Plan: Run it locally. Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60951 20 July 2016, 20:52:25 UTC
68f3eb7 sdong 20 July 2016, 01:32:51 UTC Run release build for CLANG and GCC 4.8.1 in pre-commit tests too Summary: As Title. Test Plan: See how the diff works. Reviewers: kradhakrishnan, andrewkr, gunnarku Reviewed By: gunnarku Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60933 20 July 2016, 20:46:32 UTC
e70020e omegaga 20 July 2016, 18:23:31 UTC Only cache level 0 indexes and filter when opening table reader Summary: In T8216281 we decided to disable prefetching the index and filter during opening table handlers during startup (max_open_files = -1). Test Plan: Rely on `IndexAndFilterBlocksOfNewTableAddedToCache` to guarantee L0 indexes and filters are still cached and change `PinL0IndexAndFilterBlocksTest` to make sure other levels are not cached (maybe add one more test to test we don't cache other levels?) Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59913 20 July 2016, 18:23:31 UTC
7bedd94 krad 19 July 2016, 23:36:25 UTC Build break fix Summary: (1) Integer size correction (mac build break) (2) snprint usage in Windows (windows build break) Test Plan: Build in windows and mac Reviewers: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60927 20 July 2016, 17:56:43 UTC
68a8e6b Islam AbdelRahman 20 July 2016, 16:49:03 UTC Introduce FullMergeV2 (eliminate memcpy from merge operators) Summary: This diff update the code to pin the merge operator operands while the merge operation is done, so that we can eliminate the memcpy cost, to do that we need a new public API for FullMerge that replace the std::deque<std::string> with std::vector<Slice> This diff is stacked on top of D56493 and D56511 In this diff we - Update FullMergeV2 arguments to be encapsulated in MergeOperationInput and MergeOperationOutput which will make it easier to add new arguments in the future - Replace std::deque<std::string> with std::vector<Slice> to pass operands - Replace MergeContext std::deque with std::vector (based on a simple benchmark I ran https://gist.github.com/IslamAbdelRahman/78fc86c9ab9f52b1df791e58943fb187) - Allow FullMergeV2 output to be an existing operand ``` [Everything in Memtable | 10K operands | 10 KB each | 1 operand per key] DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=10000 --num=10000 --disable_auto_compactions --value_size=10240 --write_buffer_size=1000000000 [FullMergeV2] readseq : 0.607 micros/op 1648235 ops/sec; 16121.2 MB/s readseq : 0.478 micros/op 2091546 ops/sec; 20457.2 MB/s readseq : 0.252 micros/op 3972081 ops/sec; 38850.5 MB/s readseq : 0.237 micros/op 4218328 ops/sec; 41259.0 MB/s readseq : 0.247 micros/op 4043927 ops/sec; 39553.2 MB/s [master] readseq : 3.935 micros/op 254140 ops/sec; 2485.7 MB/s readseq : 3.722 micros/op 268657 ops/sec; 2627.7 MB/s readseq : 3.149 micros/op 317605 ops/sec; 3106.5 MB/s readseq : 3.125 micros/op 320024 ops/sec; 3130.1 MB/s readseq : 4.075 micros/op 245374 ops/sec; 2400.0 MB/s ``` ``` [Everything in Memtable | 10K operands | 10 KB each | 10 operand per key] DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=1000 --num=10000 --disable_auto_compactions --value_size=10240 --write_buffer_size=1000000000 [FullMergeV2] readseq : 3.472 micros/op 288018 ops/sec; 2817.1 MB/s readseq : 2.304 micros/op 434027 ops/sec; 4245.2 MB/s readseq : 1.163 micros/op 859845 ops/sec; 8410.0 MB/s readseq : 1.192 micros/op 838926 ops/sec; 8205.4 MB/s readseq : 1.250 micros/op 800000 ops/sec; 7824.7 MB/s [master] readseq : 24.025 micros/op 41623 ops/sec; 407.1 MB/s readseq : 18.489 micros/op 54086 ops/sec; 529.0 MB/s readseq : 18.693 micros/op 53495 ops/sec; 523.2 MB/s readseq : 23.621 micros/op 42335 ops/sec; 414.1 MB/s readseq : 18.775 micros/op 53262 ops/sec; 521.0 MB/s ``` ``` [Everything in Block cache | 10K operands | 10 KB each | 1 operand per key] [FullMergeV2] $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --num=100000 --db="/dev/shm/merge-random-10K-10KB" --cache_size=1000000000 --use_existing_db --disable_auto_compactions readseq : 14.741 micros/op 67837 ops/sec; 663.5 MB/s readseq : 1.029 micros/op 971446 ops/sec; 9501.6 MB/s readseq : 0.974 micros/op 1026229 ops/sec; 10037.4 MB/s readseq : 0.965 micros/op 1036080 ops/sec; 10133.8 MB/s readseq : 0.943 micros/op 1060657 ops/sec; 10374.2 MB/s [master] readseq : 16.735 micros/op 59755 ops/sec; 584.5 MB/s readseq : 3.029 micros/op 330151 ops/sec; 3229.2 MB/s readseq : 3.136 micros/op 318883 ops/sec; 3119.0 MB/s readseq : 3.065 micros/op 326245 ops/sec; 3191.0 MB/s readseq : 3.014 micros/op 331813 ops/sec; 3245.4 MB/s ``` ``` [Everything in Block cache | 10K operands | 10 KB each | 10 operand per key] DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --num=100000 --db="/dev/shm/merge-random-10-operands-10K-10KB" --cache_size=1000000000 --use_existing_db --disable_auto_compactions [FullMergeV2] readseq : 24.325 micros/op 41109 ops/sec; 402.1 MB/s readseq : 1.470 micros/op 680272 ops/sec; 6653.7 MB/s readseq : 1.231 micros/op 812347 ops/sec; 7945.5 MB/s readseq : 1.091 micros/op 916590 ops/sec; 8965.1 MB/s readseq : 1.109 micros/op 901713 ops/sec; 8819.6 MB/s [master] readseq : 27.257 micros/op 36687 ops/sec; 358.8 MB/s readseq : 4.443 micros/op 225073 ops/sec; 2201.4 MB/s readseq : 5.830 micros/op 171526 ops/sec; 1677.7 MB/s readseq : 4.173 micros/op 239635 ops/sec; 2343.8 MB/s readseq : 4.150 micros/op 240963 ops/sec; 2356.8 MB/s ``` Test Plan: COMPILE_WITH_ASAN=1 make check -j64 Reviewers: yhchiang, andrewkr, sdong Reviewed By: sdong Subscribers: lovro, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57075 20 July 2016, 16:49:03 UTC
e70ba4e sdong 08 July 2016, 19:51:24 UTC MemTable::PostProcess() can skip updating num_deletes if the delta is 0 Summary: In many use cases there is no deletes. No need to pay the overhead of atomically updating num_deletes. Test Plan: Run existing test. Reviewers: ngbronson, yiwu, andrewkr, igor Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60555 20 July 2016, 01:10:18 UTC
2a282e5 sdong 15 July 2016, 17:51:59 UTC DBTablePropertiesTest.GetPropertiesOfTablesInRange: Fix Flaky Summary: Summary There is a possibility that there is no L0 file after writing the data. Generate an L0 file to make it work. Test Plan: Run the test many times. Reviewers: andrewkr, yiwu Reviewed By: yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60825 19 July 2016, 22:46:20 UTC
d9cfaa2 krad 22 April 2016, 23:18:33 UTC Persistent Read Cache (6) Persistent cache tier implentation - File layout Summary: Persistent cache tier is the tier abstraction that can work for any block device based device mounted on a file system. The design/implementation can handle any generic block device. Any generic block support is achieved by generalizing the access patten as {io-size, q-depth, direct-io/buffered}. We have specifically tested and adapted the IO path for NVM and SSD. Persistent cache tier consists of there parts : 1) File layout Provides the implementation for handling IO path for reading and writing data (key/value pair). 2) Meta-data Provides the implementation for handling the index for persistent read cache. 3) Implementation It binds (1) and (2) and flushed out the PersistentCacheTier interface This patch provides implementation for (1)(2). Follow up patch will provide (3) and tests. Test Plan: Compile and run check Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57117 19 July 2016, 19:01:46 UTC
9430333 John Alexander 19 July 2016, 16:44:03 UTC New Statistics to track Compression/Decompression (#1197) * Added new statistics and refactored to allow ioptions to be passed around as required to access environment and statistics pointers (and, as a convenient side effect, info_log pointer). * Prevent incrementing compression counter when compression is turned off in options. * Prevent incrementing compression counter when compression is turned off in options. * Added two more supported compression types to test code in db_test.cc * Prevent incrementing compression counter when compression is turned off in options. * Added new StatsLevel that excludes compression timing. * Fixed casting error in coding.h * Fixed CompressionStatsTest for new StatsLevel. * Removed unused variable that was breaking the Linux build 19 July 2016, 16:44:03 UTC
515b11f Alexander Jipa 18 July 2016, 14:37:49 UTC fixes #1210: rocksdb/java/CMakeLists.txt lacks cmake_minimum_required (#1214) specifying minimum cmake version so that no warning has to be suppressed 18 July 2016, 14:37:49 UTC
876cb8b Alexander Jipa 17 July 2016, 07:43:34 UTC fixes #1212: rocksdbjni maven build does not escape slashes in groovy script (#1213) using gmaven property so that script is not broken by embedding unescaped Windows-style path 17 July 2016, 07:43:34 UTC
21c55bd sdong 15 July 2016, 17:18:16 UTC DBTest.DynamicLevelCompressionPerLevel: Tune Threshold Summary: Each SST's file size increases after we add more table properties. Threshold in DBTest.DynamicLevelCompressionPerLevel need to adjust accordingly to avoid occasional failures. Test Plan: Run the test Reviewers: andrewkr, yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60819 15 July 2016, 23:10:09 UTC
4b95253 Yi Wu 15 July 2016, 17:41:36 UTC Refactor cache.cc Summary: Refactor cache.cc so that I can plugin clock cache (D55581). Mainly move `ShardedCache` to separate file, move `LRUHandle` back to cache.cc and rename it lru_cache.cc. Test Plan: make check -j64 Reviewers: lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59655 15 July 2016, 17:41:36 UTC
c6a8665 Igor Canadi 15 July 2016, 11:09:30 UTC Update LANGUAGE-BINDINGS.md 15 July 2016, 11:09:30 UTC
880ee36 Wanning Jiang 14 July 2016, 21:09:31 UTC ldb backup support Summary: add backup support for ldb tool, and use it to run load test for backup on two HDFS envs Test Plan: first generate some db, then compile against load test in fbcode, run load_test --db=<db path> backup --backup_env_uri=<URI of backup env> --backup_dir=<backup directory> --num_threads=<number of thread> Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60633 14 July 2016, 21:09:31 UTC
6797e6f sdong 07 July 2016, 18:13:57 UTC Avoid updating memtable allocated bytes if write_buffer_size is not set Summary: If options.write_buffer_size is not set, nor options.write_buffer_manager, no need to update the bytes allocated counter in MemTableAllocator, which is expensive in parallel memtable insert case. Remove it can improve parallel memtable insert throughput by 10% with write batch size 128. Test Plan: Run benchmarks TEST_TMPDIR=/dev/shm/ ./db_bench --benchmarks=fillrandom -disable_auto_compactions -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -num=10000000 --writes=1000000 -max_background_flushes=16 -max_write_buffer_number=16 --threads=32 --batch_size=128 -allow_concurrent_memtable_write -enable_write_thread_adaptive_yield The throughput grows 10% with the benchmark. Reviewers: andrewkr, yiwu, IslamAbdelRahman, igor, ngbronson Reviewed By: ngbronson Subscribers: ngbronson, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60465 14 July 2016, 02:33:57 UTC
dda6c72 Aaron Gao 14 July 2016, 00:59:25 UTC Add DestroyColumnFamilyHandle(ColumnFamilyHandle**) to db.h Summary: add DestroyColumnFamilyHandle(ColumnFamilyHandle**) to close column family instead of deleting cfh* User should call this to close a cf and then we can detect the deletion in this function. Test Plan: make all check -j64 Reviewers: andrewkr, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60765 14 July 2016, 00:59:25 UTC
56222f5 Andrew Kryczka 13 July 2016, 22:36:22 UTC Avoid FileMetaData copy Summary: as titled Test Plan: unit tests Reviewers: sdong, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60597 13 July 2016, 22:36:22 UTC
15b7a4a Jay Edgar 08 July 2016, 17:52:25 UTC Fixed output size and removed unneeded loop Summary: In Zlib_Compress and BZip2_Compress the calculation for size was slightly off when using compression_foramt_version 2 (which includes the decompressed size in the output). Also there were unnecessary loops around the deflate/BZ2_bzCompress calls. In Zlib_Compress there was also a possible exit from the function after calling deflateInit2 that didn't call deflateEnd. Test Plan: Standard tests Reviewers: sdong, IslamAbdelRahman, igor Reviewed By: igor Subscribers: sdong, IslamAbdelRahman, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60537 13 July 2016, 16:10:06 UTC
6ea41f8 Yi Wu 12 July 2016, 22:30:38 UTC Fix deadlock when trying update options when write stalls Summary: When write stalls because of auto compaction is disabled, or stop write trigger is reached, user may change these two options to unblock writes. Unfortunately we had issue where the write thread will block the attempt to persist the options, thus creating a deadlock. This diff fix the issue and add two test cases to detect such deadlock. Test Plan: Run unit tests. Also, revert db_impl.cc to master (but don't revert `DBImpl::BackgroundCompaction:Finish` sync point) and run db_options_test. Both tests should hit deadlock. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60627 12 July 2016, 22:30:38 UTC
efd013d Jay Edgar 13 June 2016, 16:57:43 UTC Miscellaneous performance improvements Summary: I was investigating performance issues in the SstFileWriter and found all of the following: - The SstFileWriter::Add() function created a local InternalKey every time it was called generating a allocation and free each time. Changed to have an InternalKey member variable that can be reset with the new InternalKey::Set() function. - In SstFileWriter::Add() the smallest_key and largest_key values were assigned the result of a ToString() call, but it is simpler to just assign them directly from the user's key. - The Slice class had no move constructor so each time one was returned from a function a new one had to be allocated, the old data copied to the new, and the old one was freed. I added the move constructor which also required a copy constructor and assignment operator. - The BlockBuilder::CurrentSizeEstimate() function calculates the current estimate size, but was being called 2 or 3 times for each key added. I changed the class to maintain a running estimate (equal to the original calculation) so that the function can return an already calculated value. - The code in BlockBuilder::Add() that calculated the shared bytes between the last key and the new key duplicated what Slice::difference_offset does, so I replaced it with the standard function. - BlockBuilder::Add() had code to copy just the changed portion into the last key value (and asserted that it now matched the new key). It is more efficient just to copy the whole new key over. - Moved this same code up into the 'if (use_delta_encoding_)' since the last key value is only needed when delta encoding is on. - FlushBlockBySizePolicy::BlockAlmostFull calculated a standard deviation value each time it was called, but this information would only change if block_size of block_size_deviation changed, so I created a member variable to hold the value to avoid the calculation each time. - Each PutVarint??() function has a buffer and calls std::string::append(). Two or three calls in a row could share a buffer and a single call to std::string::append(). Some of these will be helpful outside of the SstFileWriter. I'm not 100% the addition of the move constructor is appropriate as I wonder why this wasn't done before - maybe because of compiler compatibility? I tried it on gcc 4.8 and 4.9. Test Plan: The changes should not affect the results so the existing tests should all still work and no new tests were added. The value of the changes was seen by manually testing the SstFileWriter class through MyRocks and adding timing code to identify problem areas. Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59607 12 July 2016, 21:15:32 UTC
e6f68fa omegaga 11 July 2016, 20:55:46 UTC Update Makefile to fix dependency Summary: In D33849 we updated Makefile to generate .d files for all .cc sources. Since we have more types of source files now, this needs to be updated so that this mechanism can work for new files. Test Plan: change a dependent .h file, re-make and see if .o file is recompiled. Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60591 11 July 2016, 20:55:46 UTC
816ae09 Aaron Gao 11 July 2016, 20:33:52 UTC fix test failure Summary: fix Rocksdb Unit Test USER_FAILURE Test Plan: make all check -j64 Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60603 11 July 2016, 20:33:52 UTC
e295da1 Andrew Kryczka 11 July 2016, 18:16:53 UTC Fix Log() doc for default level Summary: as titled Test Plan: none Reviewers: sdong, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60507 11 July 2016, 18:16:53 UTC
8e6b38d Aaron Gao 11 July 2016, 17:43:12 UTC update DB::AddFile to ingest list of sst files Summary: DB::AddFile(std::string file_path) API that allow them to ingest an SST file created using SstFileWriter We want to update this interface to be able to accept a list of files that will be ingested, DB::AddFile(std::vector<std::string> file_path_list). Test Plan: Add test case `AddExternalSstFileList` in `DBSSTTest`. To make sure: 1. files key ranges are not overlapping with each other 2. each file key range dont overlap with the DB key range 3. make sure no snapshots are held Reviewers: andrewkr, sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58587 11 July 2016, 17:43:12 UTC
296545a Yi Wu 09 July 2016, 00:50:51 UTC Fix clang analyzer errors Summary: Fixing erros reported by clang static analyzer. * Removing some unused variables. * Adding assertions to fix false positives reported by clang analyzer. * Adding `__clang_analyzer__` macro to suppress false positive warnings. Test Plan: USE_CLANG=1 OPT=-g make analyze -j64 Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60549 09 July 2016, 00:50:51 UTC
61dbfbb Gunnar Kudrjavets 08 July 2016, 23:02:29 UTC Add release build to RocksDB per-diff/post-commit tests Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening. Test Plan: - `[p]arc diff --preview` - Observe the execution in Sandcastle and make sure that release build and tests are executed. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60441 08 July 2016, 23:02:29 UTC
907f24d sdong 07 July 2016, 21:45:29 UTC Concurrent memtable inserter to update counters and flush state after all inserts Summary: In concurrent memtable insert case, updating counters in MemTable::Add() can count for 5% CPU usage. By batch all the counters and update in the end of the write batch, the CPU overheads are overhead in the use cases where more than one key is updated in one write batch. Test Plan: Write throughput increases 12% with this benchmark setting: TEST_TMPDIR=/dev/shm/ ./db_bench --benchmarks=fillrandom -disable_auto_compactions -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -num=10000000 --writes=1000000 -max_background_flushes=16 -max_write_buffer_number=16 --threads=64 --batch_size=128 -allow_concurrent_memtable_write -enable_write_thread_adaptive_yield Reviewers: andrewkr, IslamAbdelRahman, ngbronson, igor Reviewed By: ngbronson Subscribers: ngbronson, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60495 08 July 2016, 17:19:55 UTC
0f691c4 Wanning Jiang 08 July 2016, 00:51:36 UTC CLI option & Rename() allow overwrite Summary: this test support CLI option to select HdfsEnv/NativeHdfsEnv now. The latter one is default. add test about when Rename(src, target) should overwrite target Test Plan: existing test Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60399 08 July 2016, 00:51:36 UTC
7c19007 Aaron Gao 07 July 2016, 23:34:41 UTC delete unnessary pointer cast in beginInternalTransaction() function Summary: use of dynamic_cast<TransactionImpl*> is unnecessary and also introduce difficulty for fbrocksdb support of TransactionDB Test Plan: ./transaction_test Reviewers: sdong, IslamAbdelRahman, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60501 07 July 2016, 23:34:41 UTC
e1b3ee8 Andrew Kryczka 07 July 2016, 18:35:40 UTC Cleanup auto-roll logger flush-while-rolling test Summary: Use @omegaga's awesome feature to avoid use of callbacks for ensuring SyncPoints happen in a particular thread. Depends on D60375. Test Plan: $ ./auto_roll_logger_test Reviewers: omegaga, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, omegaga, leveldb Differential Revision: https://reviews.facebook.net/D60471 07 July 2016, 18:35:40 UTC
cd4178a omegaga 07 July 2016, 18:29:14 UTC Add a new feature to enforce a sync point only active on a thread Summary: Add markers to sync points. A marked sync point will only be active when it is on the same thread as the marker sync point. Test Plan: Write a unit test to validate. Reviewers: sdong, IslamAbdelRahman, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60375 07 July 2016, 18:29:14 UTC
b954847 Gunnar Kudrjavets 06 July 2016, 23:07:53 UTC Fix release build for MyRocks by using debug-only code only in debug builds Summary: MyRocks release integration build breaks because we treat warnings caused by unused variables as errors. Variable `edit` is only used in debug builds. Therefore we need to guard it using `#ifndef NDEBUG` check. Test Plan: - `[p]arc diff --preview` for the default validation. - Verify that release build fails before this fix and passes after applying it. Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60423 06 July 2016, 23:07:53 UTC
a00bf1b sdong 28 June 2016, 00:42:14 UTC Add More Logging to track total_log_size Summary: We saw instances where total_log_size is off the real value, but I'm not able to reproduce it. Add more logging to help debugging when it happens again. Test Plan: Run the unit test and see the logging. Reviewers: andrewkr, yhchiang, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60081 06 July 2016, 21:29:18 UTC
01f77cb sdong 21 June 2016, 01:01:03 UTC Update USER.md to include more services at Facebook. Test Plan: Not needed. Reviewers: igor, lgalanis, andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60417 06 July 2016, 21:27:08 UTC
eb53c05 Zun Wang 06 July 2016, 19:54:01 UTC Add comment for GetBackupInfo about returned BackupInfos order Summary: #title Test Plan: n/a Reviewers: uddipta, ldemailly, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60393 06 July 2016, 19:54:01 UTC
back to top