https://github.com/facebook/rocksdb

sort by:
Revision Author Date Message Commit Date
36300fb Enable per-request buffer allocation in RandomAccessFile This change impacts only non-buffered I/O on Windows. Currently, there is a buffer per RandomAccessFile instance that is protected by a lock. The reason we maintain the buffer is non-buffered I/O requires an aligned buffer to work. XPerf traces demonstrate that we accumulate a considerable wait time while waiting for that lock. This change enables to set random access buffer size to zero which would indicate a per request allocation. We are expecting that allocation expense would be much less than I/O costs plus wait time due to the fact that the memory heap would tend to re-use page aligned allocations especially with the use of Jemalloc. This change does not affect buffer use as a read_ahead_buffer for compaction purposes. 01 February 2016, 21:14:37 UTC
3b2a1dd Add options.base_background_compactions as a number of compaction threads for low compaction debt Summary: If options.base_background_compactions is given, we try to schedule number of compactions not existing this number, only when L0 files increase to certain number, or pending compaction bytes more than certain threshold, we schedule compactions based on options.max_background_compactions. The watermarks are calculated based on slowdown thresholds. Test Plan: Add new test cases in column_family_test. Adding more unit tests. Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, rven, anthony Reviewed By: anthony Subscribers: leveldb, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D53409 30 January 2016, 00:15:53 UTC
6ee38bb Slowdown of writing to the last memtable should not override stopping Summary: Now slowing down for the last mem table takes priority against some stopping conditions. This is logically confusing. Fix it. Test Plan: Run all existing tests. Reviewers: yhchiang, IslamAbdelRahman, kradhakrishnan, andrewkr, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53529 29 January 2016, 05:52:29 UTC
d6c838f Add SstFileManager (component tracking all SST file in DBs and control the deletion rate) Summary: Add a new class SstFileTracker that will be notified whenever a DB add/delete/move and sst file, it will also replace DeleteScheduler SstFileTracker can be used later to abort writes when we exceed a specific size Test Plan: unit tests Reviewers: rven, anthony, yhchiang, sdong Reviewed By: sdong Subscribers: igor, lovro, march, dhruba Differential Revision: https://reviews.facebook.net/D50469 29 January 2016, 02:35:01 UTC
45768ad transaction allocation perf improvements Summary: Removed a couple of memory allocations Test Plan: changes covered by existing tests Reviewers: rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53523 29 January 2016, 01:32:28 UTC
77926f9 Merge pull request #964 from benoitc/fix/pi2 fix build for raspberry 2 28 January 2016, 18:24:11 UTC
03a5661 fix build for raspberry 2 fix #810 28 January 2016, 10:43:58 UTC
4b50f13 Should not skip bloom filter for L0 during the query. Summary: It's a regression bug caused by e089db40f9c8f2a8af466377ed0f6fd8a3c26456. With the change, if options.optimize_filters_for_hits=true and there are only L0 files (like single level universal compaction), we skip all the files in L0, which is more than necessary. Fix it by always trying to query bloom filter for files in level 0. Test Plan: Add a unit test for it. Reviewers: anthony, rven, yhchiang, IslamAbdelRahman, kradhakrishnan, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53493 28 January 2016, 00:16:39 UTC
eadd221 Merge pull request #959 from koldat/master Fixing generated GenerateBuildVersion.vcxproj on non English locale 27 January 2016, 20:25:47 UTC
d209076 Merge pull request #961 from wingify/master Add Wingify to USERS.md 27 January 2016, 19:53:47 UTC
26c6180 Add Wingify to USERS.md VWO, the flagship product of Wingify, uses RocksDB to populate the list of URLs where clients can run A/B test campaigns. It is also used to provide a way for the clients to see if VWO's Smart Code is installed on a specific URL of their account. 27 January 2016, 19:34:49 UTC
4265f81 Remove util/auto_roll_logger.cc (it was moved to different directory) 27 January 2016, 11:07:31 UTC
d7f22b6 Fixing generated GenerateBuildVersion.vcxproj when one builds on different locale than english. The problem is that date and time CLI utilities generates different format so that REGEX in CMake does not work. 27 January 2016, 09:51:20 UTC
d20915d Disable stats about mutex duration by default Summary: Measuring mutex duration will measure time inside DB mutex, which breaks our best practice. Add a stat level in Statistics class. By default, disable to measure the mutex operations. Test Plan: Add a unit test to make sure it is off by default. Reviewers: rven, anthony, IslamAbdelRahman, kradhakrishnan, andrewkr, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53367 26 January 2016, 22:56:55 UTC
0c433cd Fix issue in Iterator::Seek when using Block based filter block with prefix_extractor Summary: Similar to D53385 we need to check InDomain before checking the filter block. Test Plan: unit tests Reviewers: yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53421 26 January 2016, 22:47:42 UTC
035857a Fix RocksDB lite build Summary: NewMemEnv() is defined in rocksdb lite but just returns nullptr -- would it be better to just not define it so we can catch issues like this at compile-time? Test Plan: $ make clean && OPT="-DTRAVIS -DROCKSDB_LITE" V=1 make -j32 db_test $ ./db_test --gtest_filter='DBTest.MemEnvTest' ... [ PASSED ] 0 tests. Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53427 26 January 2016, 21:15:36 UTC
77ef87c Update fbcode_config4.8.1.sh to use update_dependencies.sh Summary: This patch update fbcode_config4.8.1.sh to get it's dependencies the same way we updated fbcode_config.sh in D53037 as a result zstd is upgraded to 0.4.7 instead of 0.4.5 Test Plan: make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64 make clean && ROCKSDB_FBCODE_BUILD_WITH_481=1 USE_CLANG=1 make check -j64 Reviewers: yhchiang, andrewkr, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53355 26 January 2016, 20:13:27 UTC
955ecf8 Fix an ASAN error in compact_files_test Summary: compact_files_test enables SyncPoint but never disable it before the test terminates. As a result, it might cause heap-use-after-free error when some code path trying to access the static variable of SyncPoint when it has already gone out of scope after the main thread dies. Test Plan: COMPILE_WITH_ASAN=1 make compact_files_test -j32 ./compact_files_test Reviewers: sdong, anthony, kradhakrishnan, rven, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53379 26 January 2016, 19:30:30 UTC
b0afcde Fix bug in block based tables with full filter block and prefix_extractor Summary: Right now when we are creating a BlockBasedTable with fill filter block we add to the filter all the prefixes that are InDomain() based on the prefix_extractor the problem is that when we read a key from the file, we check the filter block for the prefix whether or not it's InDomain() Test Plan: unit tests Reviewers: yhchiang, rven, anthony, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53385 26 January 2016, 19:07:08 UTC
167bd88 [directory includes cleanup] Finish removing util->db dependencies 26 January 2016, 18:49:24 UTC
acd7d58 [directory includes cleanup] Remove util->db dependency for ThreadStatusUtil Summary: We can avoid the dependency by forward-declaring ColumnFamilyData and then treating it as a black box. That means callers of ThreadStatusUtil need to explicitly provide more options, even if they can be derived from the ColumnFamilyData, since ThreadStatusUtil doesn't include the definition. This is part of a series of diffs to eliminate circular dependencies between directories (e.g., db/* files depending on util/* files and vice-versa). Test Plan: $ ./db_test --gtest_filter=DBTest.GetThreadStatus $ make -j32 commit-prereq Reviewers: sdong, yhchiang, IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53361 26 January 2016, 18:49:16 UTC
46f9cd4 [directory includes cleanup] Move cross-function test points Summary: I split the db-specific test points out into a separate file under db/ directory. There were also a few bugs to fix in xfunc.{h,cc} that prevented it from compiling previously; see https://reviews.facebook.net/D36825. Test Plan: compilation works now, below command works, will also run "make xfunc". $ make check ROCKSDB_XFUNC_TEST='managed_new' tests-regexp='DBTest' -j32 Reviewers: sdong, yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53343 26 January 2016, 18:49:05 UTC
22ecb75 Add valgrind to pre-commit sandcastle testing Summary: Initially I removed "valgrind" from the list since it take too much time (3+hr) compared to tsan (40 min) when the tests are run in parallel. It is not effective to run the tests in parallel in sandcastle and tsan takes about 3hrs as well. Adding valgrind to the list. Test Plan: Submit this diff and watch the run Reviewers: sdong, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53337 26 January 2016, 18:40:57 UTC
b7ecf3d Fix intermittent hang in ColumnFamilyTest.FlushAndDropRaceCondition Summary: ColumnFamilyTest.FlushAndDropRaceCondition sometimes hangs because the sync point, "FlushJob::InstallResults", sleeps holding the DB mutex. Fixing it by releasing the mutex before sleeping. Test Plan: seq 1000 |parallel --gnu --eta 't=/dev/shm/rdb-{}; rm -rf $t; mkdir $t && export TEST_TMPDIR=$t; ./column_family_test -gtest_filter=*FlushAndDropRaceCondition* > $t/log-{}' Reviewers: IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53349 26 January 2016, 17:12:20 UTC
38e1d7f ldb to support --column_family option Summary: Add an option --column_family option, so that users can query or update specific column family. Also add an create column family parameter to make unit test easier. Still need to add unit tests. Test Plan: Will add a test case in ldb python test. Reviewers: yhchiang, rven, andrewkr, IslamAbdelRahman, kradhakrishnan, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53265 25 January 2016, 22:58:18 UTC
da33dfe Parameterize DBTest.Randomized Summary: Break down DBTest.Randomized to multiple gtest tests based on config type Test Plan: Run the test and all tests. Make sure configurations are correctly set Reviewers: yhchiang, IslamAbdelRahman, rven, kradhakrishnan, andrewkr, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53247 25 January 2016, 22:49:08 UTC
fb9811e Add a perf context level that doesn't measure time for mutex operations Summary: Timing mutex operations can impact scalability of the system. Add a new perf context level that can measure time counters except for mutex. Test Plan: Add a new unit test case to make sure it is not set. Reviewers: IslamAbdelRahman, rven, kradhakrishnan, yhchiang, anthony Reviewed By: anthony Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53199 25 January 2016, 22:45:03 UTC
f7ef1a6 Include rest of dependencies in dependencies.sh Summary: This diff - Include the rest of the dependencies (kernel-headers, binutils, valgrind) in dependencies.sh - updtade zst to 0.4.7 - It also fix a problem in clang scan build Test Plan: make check USE_CLANG=1 make check USE_CLANG=1 make analyze coverage_test.sh Reviewers: sdong, yhchiang, rven, andrewkr, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53301 25 January 2016, 22:35:04 UTC
3e9209a Updated GetProperty documentation Summary: As titled. Also added the kBaseLevel string, which was missing earlier. Test Plan: built Reviewers: yhchiang, anthony, rven, kradhakrishnan, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53271 25 January 2016, 22:04:55 UTC
40911e0 Run unit tests in parallel to find failing tests Summary: Added make targets parallel_test and parallel_dbtest to run tests in parallel. Each test is run 32 times in parallel. There is a timeout to catch hangs. The test continues after a failure and reports non-zero status on failure Test Plan: Run the two make targets Reviewers: anthony, yhchiang, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53079 25 January 2016, 22:04:27 UTC
2fbc59a Disallow SstFileWriter from creating empty sst files Summary: SstFileWriter may create an sst file with no entries Right now this will fail when being ingested using DB::AddFile() saying that the keys are corrupted Test Plan: make check Reviewers: yhchiang, rven, anthony, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D52815 25 January 2016, 21:47:07 UTC
f53c95f Cosmetic fixes and comments for the reader Summary: Cosmetic fixes and some comments for the script. It is one big hack and hopefully the comments will make it easy to maintain. Test Plan: Run manual tests Reviewers: sdong, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53277 25 January 2016, 18:45:29 UTC
f1ed170 Add tests to make sure new DB or ColumnFamily options are settable through string Summary: Add a test to fail if someone adds a DB options. Test Plan: Run the test, run the test with valgrind. Add an option to DB option in the middle or in the end and make sure it fails. Reviewers: yhchiang, anthony, IslamAbdelRahman, kradhakrishnan, rven, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53097 23 January 2016, 00:18:44 UTC
f57596b Improvements to pre-commit Summary: - UI is enhanced to lists the tests, status and the results - We are using the same pre-commit tool as the make equivalent - No more emails to user on failure - Dropped valgrind from the list since it can be a time hogger (and can hurt scheduling for others) - Patching bug fix - Made the jobs run in parallel in sandcastle Test Plan: Manual test Reviewers: sdong, rven, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53217 Making parallel requests to sandcastle Test Plan: Run manual tests Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53259 22 January 2016, 23:40:32 UTC
538eec0 Update fbcode_config.sh to use latest versions automatically Summary: Update fbcode_config.sh so that It try to use the latest version for dependencies that we are using, after updating the code these libraries where updated ``` Snappy: 1.0.3 => 1.1.3 GFLAGS: 1.6 => 2.1.1 JEMALLOC: 3.6.0 => 4.0.3 ``` I have also updated clang from 3.7 to 3.7.1 ``` Clang 3.7 => 3.7.1 ``` Another change is that we use the same tp2 directory as fbcode, so we dont need to keep changing commit hash every time we need to change a version of a compiler or a library Test Plan: make check -j64 USE_CLANG=1 make check -j64 DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 (make sure it's running) Reviewers: yhchiang, anthony, rven, kradhakrishnan, andrewkr, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53037 22 January 2016, 19:56:32 UTC
8019aa9 improve test for manifest write failure Summary: Improve testing per discussion in D52989 Test Plan: ran test Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53211 22 January 2016, 19:47:59 UTC
bcd4ccb Revert D7809 Summary: Revert the functionaility of D7809 (but I'm keeping the logging and test code). We decided it was dangerous to ignore sync failures based on attempting to read the data written. The read does not tell us whether the data was synced. Test Plan: There was no test for the particular functionaility that was reverted. Keeping the test code from D7809 that tests whether we set the DB to be readonly when paranoid checks are enabled. Reviewers: rven, yhchiang, kradhakrishnan, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52989 22 January 2016, 01:14:43 UTC
b0a15e7 Mechanism to run CI jobs on local branch via commit_prereq Summary: This patch provides a mechanism to run pre commit tests on the local branch before committing. This can help prevent frequent build breaks. The tests can be run in parallel by specifying the J=<..> environment variable. Test Plan: Run manually Reviewers: sdong rven tec CC: leveldb@ Task ID: #9689218 Blame Rev: 22 January 2016, 00:22:21 UTC
bb28887 Cleanup property-related variable names Summary: I noticed these names were quite confusing while updating GetProperty documentation. Test Plan: running "make commit-prereq -j32" Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53073 21 January 2016, 19:38:15 UTC
2928933 Add named constants for remaining properties Summary: There were just these two properties that didn't have any named constant. Test Plan: build and below test $ ./db_properties_test --gtest_filter=DBPropertiesTest.NumImmutableMemTable Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53103 21 January 2016, 18:59:36 UTC
2c2b722 Disable OptionsParserTest.BlockBasedTableOptionsAllFieldsSettable under CLANG Summary: OptionsParserTest.BlockBasedTableOptionsAllFieldsSettable is failiong under CLANG. Disable the test to unblock the build. Test Plan: Run it both of CLANG and GCC Reviewers: kradhakrishnan, rven, andrewkr, anthony, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53157 21 January 2016, 18:56:34 UTC
a300d99 Added sandcastle pre-commit Test Plan: Lately we have been breaking our builds too often. This changes adds the capability to schedule tests in sandcastle for every diff created. This will help us increase the pre-commit testing bar. This patch will dispatch signals to sandcastle to start running tests on the diff. The test failures are reported to the user via email. The user can also manually check the progress of test in sandcastle via the URL provided. Reviewers: sdong, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53001 21 January 2016, 16:53:55 UTC
202be23 Add test that verifies all options in BlockBasedTableOptions is settable through GetBlockBasedTableOptionsFromString() Summary: Add a test OptionsParserTest.BlockBasedTableOptionsAdded, which will fail if a new option is added to BlockBasedTableOptions but is not settable through GetBlockBasedTableOptionsFromString(). Test Plan: Run the test. Also manually remove and add options and make sure it fails. Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, rven, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52953 21 January 2016, 05:59:28 UTC
eceb5cb Split db_test.cc (part 1: properties) Summary: Moved all the tests that verify property correctness into a separate file. The goal is to reduce compile time and complexity of db_test. I didn't add parallelism for db_properties_test, even though these tests were parallelized in db_test, since the file is small enough that it won't matter. Some of these moves may be controversial since it's hard to say whether the test is "verifying property correctness," or "using properties to verify rocksdb's correctness." I'm interested in any opinions. Test Plan: ran db_properties_test, also waiting on "make commit-prereq -j32" Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52995 20 January 2016, 23:17:52 UTC
94918ae db_bench: explicitly clear buffer in compress benchmark Summary: It is reported that in compress benchmark in db_bench, zlib will cause an OOM. The suggestd fix was to clear the buffer. Test Plan: Build and run compress benchmark. Reviewers: IslamAbdelRahman, yhchiang, rven, andrewkr, kradhakrishnan, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52857 20 January 2016, 02:11:46 UTC
fdbff42 Crash test to make kill decision for every kill point Summary: In crash test, when coming to each kill point, we start a random class using seed as current second. With this approach, for every second, the random number used is the same. However, in each second, there are multiple kill points with different frequency. It makes it hard to reason about chance of kill point to trigger. With this commit, we use thread local random seed to generate the random number, so that it will take different values per second, hoping it makes chances of killing much easier to reason about. Also significantly reduce the kill odd to make sure time before kiling is similar as before. Test Plan: Run white box crash test and see the killing happens as expected and the run time time before killing reasonable. Reviewers: kradhakrishnan, IslamAbdelRahman, rven, yhchiang, andrewkr, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52971 20 January 2016, 02:11:24 UTC
39c3e94 Merge pull request #954 from davidbernard/solaris_build Solaris Build 20 January 2016, 02:05:02 UTC
df7c2f3 As per google coding standard replace "using" in option_builder.cc and geodb_impl.cc 20 January 2016, 00:27:29 UTC
12809b4 Revert "Change notification email for travis" This reverts commit eaa563756860fd752f40d0bc17e9e58a99aba483. 19 January 2016, 22:46:39 UTC
34704d5 [easy] Fixed a crash in LogAndApply() when CF creation failed Summary: That line used to dereference `column_family_data`, which is nullptr if we're creating a column family. Test Plan: `make -j check` Reviewers: sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D52881 19 January 2016, 19:46:52 UTC
791dbaf Merge pull request #953 from sselva/master Add Rakuten Marketing to USERS.md 19 January 2016, 19:04:04 UTC
594a5ef Merge pull request #955 from bcbrock/ppc64-build Simple changes to support builds for ppc64[le] consistent with X86 19 January 2016, 17:05:44 UTC
f423f05 Simple changes to support builds for ppc64[le] consistent with X86 These simple changes are required to allow builds on ppc64[le] systems consistent with X86. The Makefile now recognizes both ppc64 and ppc64le, and in the absence of PORTABLE=1, the code will be built analogously to the X86 -march=native. Note that although GCC supports -mcpu=native -mtune=native on POWER, it doesn't work correctly on all systems. This is why we need to get the actual machine model from the AUX vector. 19 January 2016, 15:08:19 UTC
3f12e16 Make alloca.h optional 19 January 2016, 06:17:31 UTC
eaa5637 Change notification email for travis 19 January 2016, 04:59:39 UTC
d78c6b2 Changes for build on solaris Makefile adjust paths for solaris build Makefile enable _GLIBCXX_USE_C99 so that std::to_string is available db_compaction_test.cc Initialise a variable to avoid a compilation error db_impl.cc Include <alloca.h> db_test.cc Include <alloca.h> Environment.java recognise solaris envrionment options_bulder.cc Make log unambiguous geodb_impl.cc Make log and floor unambiguous 19 January 2016, 04:45:21 UTC
2e9fae3 Add Rakuten Marketing to USERS.md 18 January 2016, 23:24:52 UTC
83e1de9 move internal build to use zstd 0.4.5 Summary: update internal build scripts to use zstd 0.4.5. Test Plan: built and ran tests with and without ROCKSDB_FBCODE_BUILD_WITH_481 Reviewers: yhchiang, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52833 15 January 2016, 21:27:35 UTC
aec10f7 Guard falloc.h inclusion to avoid build breaks Summary: Depending on the order of include paths and versions of various headers we may end up in a situation where we'll encounter a build break caused by redefinition of constants. gcc-4.9-glibc-2.20 header update to include/bits/fcntl-linux.h introduced the definitions of FALLOC_FL_* constants. However, linux/falloc.h from kernel-headers also has FALLOC_FL_* constants defined. Therefore during the compilation we'll get "previously defined" errors. Test Plan: Both in the environment where the build break manifests (to make sure that the change fixed the problem) and in the environment where everything builds fine (to make sure that there are no regressions): make clean make -j 32 Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52821 15 January 2016, 06:47:15 UTC
f7ebc2f Update HISTORY.mc for 4.4.0 Summary: Prepare to cut release Test Plan: no code change. Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52827 15 January 2016, 00:57:06 UTC
addd954 Merge pull request #947 from yuslepukhin/align_and_yield Align statistics 14 January 2016, 18:28:40 UTC
ac50fd3 Align statistics Use Yield macro to make it a little more portable between platforms. 13 January 2016, 22:53:23 UTC
b54d4dd tools/sst_dump_tool_imp.h not to depend on "util/testutil.h" Summary: util/testutil.h doesn't seem to be used in tools/sst_dump_tool_imp.h. Remove it. Also move some other include to tools/sst_dump_tool.cc instead. Test Plan: Build with GCC, CLANG and with GCC 4.81 and 4.9. Reviewers: yuslepukhin, yhchiang, rven, anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52791 13 January 2016, 19:34:53 UTC
48a8667 Merge pull request #929 from warrenfalk/fix32 fix a compile error on 32-bit (fixes #634) 12 January 2016, 19:02:36 UTC
d9bca1e Reduce iterator deletion overhead Summary: After introducing Iterator::PinData(), we have extra overhead of deleting the pinned iterators that we track in a std::set This patch avoid inserting to the std::set if we have only one iterator (normal use case when no iterators are pinned) Before this change ``` DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="newiterator" --db="/tmp/rocksdbtest-8616/dbbench" --use_existing_db --disable_auto_compactions newiterator : 1.006 micros/op 994013 ops/sec; newiterator : 0.994 micros/op 1006295 ops/sec; newiterator : 0.990 micros/op 1010422 ops/sec; ``` After change ``` DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="newiterator" --db="/tmp/rocksdbtest-8616/dbbench" --use_existing_db --disable_auto_compactions newiterator : 0.754 micros/op 1326588 ops/sec; newiterator : 0.759 micros/op 1317394 ops/sec; newiterator : 0.691 micros/op 1446704 ops/sec; ``` Test Plan: make check -j64 Reviewers: yhchiang, rven, anthony, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D52761 12 January 2016, 00:48:15 UTC
45d7940 Merge pull request #940 from yuslepukhin/fix_windows_build_signed_unsigned Fix compile error. 12 January 2016, 00:16:17 UTC
20d7902 Fix compile error. Use constructor style initialization instead of a cast for simplicity. 12 January 2016, 00:10:48 UTC
1477dcb Merge pull request #937 from petehunt/master Add Smyte to USERS.md 10 January 2016, 09:56:04 UTC
c7cb107 Add Smyte to USERS.md 09 January 2016, 22:11:44 UTC
df7e3b6 Include <array> in table/plain_table_key_coding.h Summary: <array> is not included in table/plain_table_key_coding.h. It may be the cause of one CLANG build failure. Test Plan: Build it Reviewers: yhchiang, rven, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52725 09 January 2016, 01:34:53 UTC
235b162 Not scheduling more L1->L2 compaction if L0->L1 is pending with higher priority Summary: When L0->L1 is pending, there may be one L1->L2 compaction going on which prevents the L0->L1 compaction from happening. If L1 needs more data to be moved to L2, then we may continue scheduling more L1->L2 compactions. The end result may be that L0->L1 compaction will not happen until L1 size drops to below target size. We can reduce the stalling because of number of L0 files by stopping schedling new L1->L2 compaction when L0's score is higher than L1. Test Plan: Run all existing tests. Reviewers: yhchiang, MarkCallaghan, rven, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52401 08 January 2016, 21:56:57 UTC
9a8e3f7 plain table reader: non-mmap mode to keep two recent buffers Summary: In plain table reader's non-mmap mode, we only keep the most recent read buffer. However, for binary search, it is likely we come back to a location to read. To avoid one pread in such a case, we keep two read buffers. It should cover most of the cases. Test Plan: 1. run tests 2. check the optimization works through strace when running ./table_reader_bench -mmap_read=false --num_keys2=1 -num_keys1=5000 -table_factory=plain_table --iterator --through_db Reviewers: anthony, rven, kradhakrishnan, igor, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D51171 08 January 2016, 18:53:57 UTC
7ece10e DeleteFilesInRange: Mark files to be deleted as being compacted before applying change Summary: While running the myrocks regression suite, I found that while dropping a table soon after inserting rows into it resulted in an assertion failure in CheckConsistencyForDeletes for not finding a file which was recently added or moved. Marking the files to be deleted as being compacted before calling LogAndApplyChange fixed the assertion failures. Test Plan: DBCompactionTest.DeleteFileRange Reviewers: IslamAbdelRahman, anthony, yhchiang, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, yoshinorim, leveldb Differential Revision: https://reviews.facebook.net/D52599 07 January 2016, 22:48:45 UTC
94d9df2 fix an unused function compiler warning in crc32c in 32-bit mode 07 January 2016, 18:27:20 UTC
2f01e10 use static_cast in crc32c instead of c-style cast 07 January 2016, 18:22:09 UTC
601f130 fix shorten-64-to-32 warning in crc32c 07 January 2016, 18:12:15 UTC
f3fb398 Fix BlockBasedTableTest.NoopTransformSeek failure Summary: table_test is failing because we are creating a temp InternalComparator 14:27:28 [ RUN ] BlockBasedTableTest.NoopTransformSeek 14:27:28 pure virtual method called 14:27:28 terminate called without an active exception 14:27:28 /bin/sh: line 7: 2346261 Aborted (core dumped) ./$t Test Plan: make table_test -j64 && ./table_test --gtest_filter="BlockBasedTableTest.NoopTransformSeek" Reviewers: igor, sdong, anthony, rven Reviewed By: rven Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D52671 07 January 2016, 17:48:29 UTC
55b37ef fix a compile error on 32-bit 07 January 2016, 16:51:52 UTC
8c71eb5 Optimize DBIter::Prev() by reducing stack overhead Summary: It looks like we are spending significant amount of time creating std::deque<std::string> every time we do Iterator::Prev() {F921567} By using merge_operands_ as a DBIter data member w create it once and reduce this overhead and see ~30% performance improvement when using Iterator::Prev() on hot data Orignal performance ``` DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readreverse" --db="/dev/shm/bench_prev_opt/" --use_existing_db --disable_auto_compactions readreverse : 0.713 micros/op 1402219 ops/sec; 155.1 MB/s readreverse : 0.609 micros/op 1641386 ops/sec; 181.6 MB/s readreverse : 0.684 micros/op 1461150 ops/sec; 161.6 MB/s readreverse : 0.629 micros/op 1589842 ops/sec; 175.9 MB/s readreverse : 0.647 micros/op 1544530 ops/sec; 170.9 MB/s ``` After optimization ``` DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readreverse" --db="/dev/shm/bench_prev_opt/" --use_existing_db --disable_auto_compactions readreverse : 0.488 micros/op 2051189 ops/sec; 226.9 MB/s readreverse : 0.505 micros/op 1980892 ops/sec; 219.1 MB/s readreverse : 0.541 micros/op 1846971 ops/sec; 204.3 MB/s readreverse : 0.497 micros/op 2013612 ops/sec; 222.8 MB/s readreverse : 0.480 micros/op 2082665 ops/sec; 230.4 MB/s ``` Test Plan: make check -j64 Reviewers: sdong, anthony, rven, igor, yhchiang Reviewed By: yhchiang Subscribers: jkedgar, dhruba Differential Revision: https://reviews.facebook.net/D52563 07 January 2016, 15:59:14 UTC
73c3137 Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set." Summary: This patch reverts commit 57605d7ef3d6108da94f7b5e4846cac8c3747059 as it will cause BlockBasedTableTest.NoopTransformSeek test crashes in some environment. Test Plan: revert the patch Reviewers: anthony, IslamAbdelRahman, rven, kradhakrishnan, sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52623 07 January 2016, 07:33:41 UTC
57605d7 Fixed the bug when both whole_key_filtering and prefix_extractor are set. Summary: When both whole_key_filtering and prefix_extractor are set, RocksDB will mistakenly encode prefix + whole key into the database instead of simply whole key when BlockBasedTable is used. This patch fixes this bug. Test Plan: Add a test in table_test Reviewers: anthony, IslamAbdelRahman, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52233 07 January 2016, 07:01:23 UTC
6935eb2 Add ColumnFamilyHandle::GetDescriptor() Summary: This patch addes ColumnFamilyHandle::GetDescriptor(), which allows developers to obtain the CF options and names of the associated column family given its handle. // Returns the up-to-date descriptor used by the current handle. Since it // returns the up-to-date information, this call might internally locks // and releases DB mutex to access the up-to-date CF options. virtual ColumnFamilyDescriptor GetDescriptor() = 0; Test Plan: augment column_family_test Reviewers: sdong, yoshinorim, IslamAbdelRahman, rven, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D51543 07 January 2016, 02:14:01 UTC
9760c84 fix valgrind failure in backupable_db_test Summary: fix memory leak in test code Test Plan: ran test Reviewers: rven, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52617 07 January 2016, 02:04:58 UTC
b1a3b4c Make ldb automagically determine the file type and use the correct dumping function Summary: This set of changes implements the following design: `ldb` will utilize `--path` parameter which can be used to specify a file name. Tool will then apply some heuristic to determine how to output the data properly. The design decision is not to probe the file content, but use file names to determine what dumping function to call. Usage examples: Understands that path points to a manifest file and dumps it. `./ldb --path=/tmp/test_db/MANIFEST-000023 dump` Understands that path points to a WAL file and dumps it. `./ldb --path=/tmp/test_db/000024.log dump --header` Understands that path points to a SST file and dumps it. `./ldb --path=/tmp/test_db/000007.sst dump` Figures out that none of the supported file types are applicable and outputs an appropriate error message. `./ldb --path=/tmp/cron.log dump` Test Plan: Basics: git diff make clean make -j 32 commit-prereq arc lint More specific testing (done as part of commit-prereq, but can be iterated separately when making isolated changes): make clean make ldb python tools/ldb_test.py make rocksdb_dump make rocksdb_undump sh tools/rocksdb_dump_test.sh Reviewers: rven, IslamAbdelRahman, yhchiang, kradhakrishnan, anthony, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52269 06 January 2016, 22:19:08 UTC
ba83447 Merge pull request #923 from petermattis/pmattis/prefix-may-match Fix index seeking in BlockTableReader::PrefixMayMatch. 06 January 2016, 22:02:10 UTC
da03249 Optimize GetLatestSequenceForKey Summary: DBImpl::GetLatestSequenceForKey() can do memcpy's to load a value that will never be used. This can be optimized by changing all the Get() functions called to optionally not fetch the value (and only fetch the sequencenumber). Test Plan: optimistic_transaction_test and transaction_test Reviewers: anthony Reviewed By: anthony Subscribers: leveldb, dhruba, hermanlee4 Differential Revision: https://reviews.facebook.net/D52227 06 January 2016, 21:43:22 UTC
260c297 Fix index seeking in BlockTableReader::PrefixMayMatch. PrefixMayMatch previously seeked in the prefix index using an internal key with a sequence number of 0. This would cause the prefix index seek to fall off the end if the last key in the index had a user-key greater than or equal to the key being looked for. Falling off the end of the index in turn results in PrefixMayMatch returning false if the index is in memory. 06 January 2016, 21:39:56 UTC
e541dcc Fix issue #921 Summary: See a bug report here: https://github.com/facebook/rocksdb/issues/921 The fix is to not check the shared/ directory if share_table_files is false. We could also check FileExists() before GetChildren(), but that will add extra latency when Env is Hdfs :( Test Plan: added a unit test Reviewers: rven, sdong, IslamAbdelRahman, yhchiang, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52593 06 January 2016, 21:05:24 UTC
51adc54 fix sporadic failure in fault_injection_test Summary: Need to make sure the background task gets scheduled before it goes out of scope. Test Plan: ran test. Will see if sporadic valgrind failures go away. Reviewers: kradhakrishnan Reviewed By: kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52575 06 January 2016, 19:51:53 UTC
a2422f0 fix potential test SleepingTask race condition Summary: Make sure SleepingTask has bene run before it goes out of scope. Test Plan: run test Reviewers: kradhakrishnan Reviewed By: kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52581 06 January 2016, 19:51:09 UTC
1627c4b Merge pull request #918 from mkurdej/fix/assertion-on-no-disk-space Fix failing assertion in logger on Windows when the disk is full. 05 January 2016, 16:09:58 UTC
92d0850 Fix failing assertion in logger on Windows when the disk is full. 05 January 2016, 12:35:14 UTC
7699439 Prevent the user from setting block_restart_interval to less than 1 Summary: If block_restart_interval gets set to less than 1 an assert will be triggered in BlockBuilder::BlockBuilder(). This prevents the user from doing this by silently setting any value less than 1 to 1. Test Plan: Added a test (in BlockBasedTableTest in table_test) that checks invalid values to make sure that they are reset to the expected values. The block_restart_interval value is checked along with block_size_deviation which also silently sets the value if it is outside a specific range. Reviewers: yoshinorim, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D52509 04 January 2016, 22:13:18 UTC
4041903 Enhance db_bench write rate limit Summary: 1) changes tools/{benchmark,run_flash_bench}.sh to optionally use the write rate limit 2) removes code for --writes_per_second and switches the 'background' write rate limit to use --benchmark_write_rate_limit Replaces https://reviews.facebook.net/D49113 Task ID: #9555881 Blame Rev: Test Plan: tools/run_flash_bench.sh Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D52485 04 January 2016, 20:01:27 UTC
3993432 Add Airbnb and Pinterest to USERS.md Summary: as title Test Plan: none Reviewers: sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52503 04 January 2016, 19:44:55 UTC
d74c9f0 DeleteFilesInRange: Clean job context if no files deleted Summary: We need to clean the job context if we end up not deleting any files because no files are in the range specified. Test Plan: DBCompactionTest.DeleteFileRange Reviewers: sdong, anthony, yhchiang, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52467 04 January 2016, 18:55:31 UTC
1dec5b8 Merge pull request #916 from warrenfalk/capi_huge_page_option Expose the option, memtable_prefix_bloom_huge_page_tlb_size, via C API 04 January 2016, 17:24:30 UTC
12fa27b Merge pull request #915 from warrenfalk/capi_full_bloom Fix for #909: Support creation of "full" format bloom filter from C API 04 January 2016, 17:23:33 UTC
0fde291 expose memtable_prefix_bloom_huge_page_tlb_size option to C API 31 December 2015, 14:27:03 UTC
7e81dba Support creation of "full" format bloom filter from C API 31 December 2015, 14:26:49 UTC
back to top