| 20ef64c | krad | 03 September 2015, 21:53:33 UTC | Moving jobs to use gcc-4.9 Summary: Fixed the glitch in Sandcastle and added gcc-4.9 support. Moving the jobs to 4.9 Test Plan: Manually run the script Reviewers: sdon igor CC: leveldb@ Task ID: #6968635 Blame Rev: | 03 September 2015, 22:20:36 UTC |
| 0f1aab6 | agiardullo | 01 September 2015, 23:51:49 UTC | Add SetLockTimeout for Transactions Summary: MyRocks wants to be able to change the lock timeout of a transaction that has already started. Expose existing SetLockTimeout function to users. Test Plan: unit test Reviewers: spetrunia, rven, sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45987 | 03 September 2015, 03:07:19 UTC |
| 14456ae | Igor Canadi | 02 September 2015, 23:05:53 UTC | Fix compile Summary: There was a merge conflict with https://reviews.facebook.net/D45993 Test Plan: make check Reviewers: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D46065 | 02 September 2015, 23:05:53 UTC |
| 76f286c | Igor Canadi | 02 September 2015, 22:36:47 UTC | Optimize bloom filter cache misses Summary: This optimizes the case when (cache_index_and_filter_blocks=1) and bloom filter is not present in the cache. Previously we did: 1. Read meta block from file 2. Read the filter position from the meta block 3. Read the filter Now, we pre-load the filter position on Table::Open(), so we can skip steps (1) and (2) on bloom filter cache miss. Instead of 2 IOs, we do only 1. Test Plan: make check Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D46047 | 02 September 2015, 22:36:47 UTC |
| 0e6e547 | Yueh-Hsuan Chiang | 02 September 2015, 22:07:44 UTC | Fixed a compile warning in rocksjni/loggerjnicallback.cc Summary: This patch fixes the following compile warning. java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’: java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable] const jint rs = env->GetJavaVM(&m_jvm); ^ Test Plan: make rocksdbjavastaticrelease Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45981 | 02 September 2015, 22:07:44 UTC |
| b8a962d | krad | 02 September 2015, 19:16:42 UTC | Adding commands for few more CI jobs. Summary: Added commands to spawn crash, stress, asan, asan_crash, clang CI jobs Test Plan: Manual test Reviewers: sdong CC: leveldb@ Task ID: #6968635 Blame Rev: | 02 September 2015, 21:06:03 UTC |
| 8a2d59a | Igor Canadi | 02 September 2015, 21:04:51 UTC | Add Cloudera's blog post to USERS.md Summary: As title Test Plan: none Reviewers: yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D46053 | 02 September 2015, 21:04:51 UTC |
| 3c9cef1 | Andres Noetzli | 02 September 2015, 20:58:22 UTC | Unified maps with Comparator for sorting, other cleanup Summary: This diff is a collection of cleanups that were initially part of D43179. Additionally it adds a unified way of defining key-value maps that use a Comparator for sorting (this was previously implemented in four different places). Test Plan: make clean check all Reviewers: rven, anthony, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45993 | 02 September 2015, 20:58:22 UTC |
| 3e0a672 | sdong | 02 September 2015, 17:25:20 UTC | Bug fix: table readers created by TableCache::Get() doesn't have latency histogram reported Summary: TableCache::Get() puts parameters in the wrong places so that table readers created by Get() will not have the histogram updated. Test Plan: Will write a unit test for that. Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D46035 | 02 September 2015, 19:57:07 UTC |
| b42cd6b | Igor Canadi | 02 September 2015, 18:49:49 UTC | Remove the need for LATEST_BACKUP in BackupEngine Summary: In the first implementation of BackupEngine, LATEST_BACKUP was the commit point. The backup became committed after the write to LATEST_BACKUP completed. However, we can avoid the need for LATEST_BACKUP. Instead of write to LATEST_BACKUP, the commit point can be the rename from `meta/<backup_id>.tmp` to `meta/<backup_id>`. Once we see that there exists a file `meta/<backup_id>` (without tmp), we can assume that backup is valid. In this diff, we still write out the file LATEST_BACKUP. We need to do this so that we can maintain backward compatibility. However, the new version doesn't depend on this file anymore. We get the latest backup by `ls`-ing `meta` directory. This diff depends on D41925 Test Plan: Adjusted backupable_db_test to this new behavior Reviewers: benj, yhchiang, sdong, AaronFeldman Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42069 | 02 September 2015, 18:49:49 UTC |
| 0f763db | Siying Dong | 02 September 2015, 18:35:17 UTC | Merge pull request #705 from yuslepukhin/rate_limiter_fix Make WinEnv::NowMicros return system time | 02 September 2015, 18:35:17 UTC |
| 20c44fe | Amit Arya | 02 September 2015, 18:17:02 UTC | t6913679: Use fallocate on LOG FILESS Summary: Use fallocate on LOG FILES to Test Plan: make check + ===check with strace=== [arya@devvm1441 ~/rocksdb] strace -e trace=fallocate ./ldb --db=/tmp/test_new scan fallocate(3, 01, 0, 4194304) = 0 Reviewers: sdong, anthony, IslamAbdelRahman, kradhakrishnan, lgalanis, rven, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45969 | 02 September 2015, 18:17:02 UTC |
| f14c336 | Dmitri Smirnov | 02 September 2015, 18:12:07 UTC | Make WinEnv::NowMicros return system time Previous change for the function https://github.com/facebook/rocksdb/commit/555ca3e7b7f06bd01dfd5e04dbb2cef5360f7917#diff-bdc04e0404c2db4fd3ac5118a63eaa4a made use of the QueryPerformanceCounter to return microseconds values that do not repeat as std::chrono::system_clock returned values that made auto_roll_logger_test fail. The interface documentation does not state that we need to return system time describing the return value as a number of microsecs since some moment in time. However, because on Linux it is implemented using gettimeofday various pieces of code (such as GenericRateLimiter) took advantage of that and make use of NowMicros() as a system timestamp. Thus the previous change broke rate_limiter_test on Windows. In addition, the interface name NowMicros() suggests that it is actually a timestamp so people use it as such. This change makes use of the new system call on Windows that returns system time with required precision. This change preserves the fix for auto_roll_logger_test and fixes rate_limiter_test. Note that DBTest.RateLimitingTest still fails due to a separately reported issue. | 02 September 2015, 18:12:07 UTC |
| aad0572 | Yueh-Hsuan Chiang | 01 September 2015, 18:07:43 UTC | Fixed the build issue of rocksdbjavastaticrelease Summary: This patch fixed couple build issues of rocksdbjavastaticrelease. Test Plan: make rocksdbjavastaticrelease Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45915 | 01 September 2015, 18:07:43 UTC |
| 5508122 | Tomislav Novak | 31 August 2015, 23:44:34 UTC | Fix a perf regression in ForwardIterator Summary: I noticed that memtable iterator usually crosses the `iterate_upper_bound` threshold when tailing. Changes introduced in D43833 made `NeedToSeekImmutable` always return true in such case, even when `Seek()` only needs to rewind the memtable iterator. In a test I ran, this caused the "tailing efficiency" (ratio of calls to `Seek()` that only affect the memtable versus all seeks) to drop almost to zero. This diff attempts to fix the regression by using a different flag to indicate that `current_` is over the limit instead of resetting `valid_` in `UpdateCurrent()`. Test Plan: `DBTestTailingIterator.TailingIteratorUpperBound` Reviewers: sdong, rven Reviewed By: rven Subscribers: dhruba, march Differential Revision: https://reviews.facebook.net/D45909 | 01 September 2015, 16:54:30 UTC |
| b722007 | Andres Notzli | 01 September 2015, 06:11:12 UTC | Fix listener_test when using ROCKSDB_MALLOC_USABLE_SIZE Summary: Flushes in listener_test happened to early when ROCKSDB_MALLOC_USABLE_SIZE was active (e.g. when compiling with ROCKSDB_FBCODE_BUILD_WITH_481=1) due to malloc_usable_size() reporting a better estimate (similar to https://reviews.facebook.net/D43317 ). This patch grows the write buffer size slightly to compensate for this. Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make listener_test && ./listener_test Reviewers: rven, anthony, yhchiang, igor, sdong Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45921 | 01 September 2015, 06:11:12 UTC |
| 40cd91b | Yueh-Hsuan Chiang | 01 September 2015, 01:36:17 UTC | Fixed compile warning in rocksdbjava Summary: Fixed the following compile warning in rocksdbjava: java/rocksjni/comparatorjnicallback.cc:20:14: warning: unused variable ‘rs’ [-Wunused-variable] const jint rs = env->GetJavaVM(&m_jvm); ^ java/rocksjni/comparatorjnicallback.cc: In member function ‘JNIEnv* rocksdb::BaseComparatorJniCallback::getJniEnv() const’: java/rocksjni/comparatorjnicallback.cc:45:8: warning: unused variable ‘rs’ [-Wunused-variable] jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL); ^ java/rocksjni/loggerjnicallback.cc: In constructor ‘rocksdb::LoggerJniCallback::LoggerJniCallback(JNIEnv*, jobject)’: java/rocksjni/loggerjnicallback.cc:19:14: warning: unused variable ‘rs’ [-Wunused-variable] const jint rs = env->GetJavaVM(&m_jvm); ^ java/rocksjni/loggerjnicallback.cc: In member function ‘JNIEnv* rocksdb::LoggerJniCallback::getJniEnv() const’: java/rocksjni/loggerjnicallback.cc:33:8: warning: unused variable ‘rs’ [-Wunused-variable] jint rs = m_jvm->AttachCurrentThread(reinterpret_cast<void **>(&env), NULL); ^ Test Plan: make rocksdbjava -j32 Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45891 | 01 September 2015, 01:36:17 UTC |
| 90415cf | Yueh-Hsuan Chiang | 01 September 2015, 01:35:12 UTC | Fixed a compile warning in linux32 environment. Summary: Fixed the following compile warning in linux32 environment. ==> linux32: util/sst_dump_tool.cc: In member function ‘int rocksdb::SstFileReader::ShowAllCompressionSizes(size_t)’: ==> linux32: util/sst_dump_tool.cc:167:50: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘size_t {aka unsigned int}’ [-Wformat=] ==> linux32: fprintf(stdout, "Block Size: %lu\n", block_size); Test Plan: make sst_dump Reviewers: anthony, IslamAbdelRahman, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45885 | 01 September 2015, 01:35:12 UTC |
| 9d6503f | sdong | 31 August 2015, 23:20:40 UTC | Fix arena_test test break using glibc-2.17 Summary: arena_test is failing with glibc-2.17. Make it more robust Test Plan: Run arena_test using both of glibc-2.17 and 2.2 and make sure both passes. Reviewers: yhchiang, rven, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45879 | 31 August 2015, 23:53:23 UTC |
| 77a2861 | agiardullo | 14 August 2015, 23:45:05 UTC | Support static Status messages Summary: Provide a way to specify a detailed static error message for a Status without incurring a memcpy. Let me know what people think of this approach. Test Plan: added simple test Reviewers: igor, yhchiang, rven, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44259 | 31 August 2015, 23:13:29 UTC |
| 18db1e4 | agiardullo | 28 August 2015, 23:09:34 UTC | better db_bench options for transactions Summary: Pessimistic Transaction expiration time checking currently causes a performace regression, Lets disable it in db_bench by default. Also, in order to be able to better tune how much contention we're simulating, added new optinos to set lock timeout and snapshot. Test Plan: run db_bench randomtranansaction Reviewers: sdong, igor, yhchiang, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45831 | 31 August 2015, 22:56:07 UTC |
| 0be2605 | Igor Canadi | 31 August 2015, 20:48:26 UTC | Merge pull request #702 from PraveenSinghRao/remove_spurious remove spurious compression definitions | 31 August 2015, 20:48:26 UTC |
| 8b68954 | Ari Ekmekji | 31 August 2015, 19:59:02 UTC | Add Subcompactions to Universal Compaction Unit Tests Summary: Now that the approach to parallelizing L0-L1 level-based compactions by breaking the compaction job into subcompactions is being extended to apply to universal compactions as well, the unit tests need to account for this and run the universal compaction tests with subcompactions both enabled and disabled. Test Plan: make all && make check Reviewers: sdong, igor, noetzli, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45657 | 31 August 2015, 19:59:02 UTC |
| c6d870f | krad | 31 August 2015, 19:42:56 UTC | Merge branch 'arcpatch-D45741' | 31 August 2015, 19:42:56 UTC |
| 57b3a87 | krad | 31 August 2015, 19:37:26 UTC | Adding sandcastle determinator for RocksDB Summary: This fuels commands to be executed for different sandcastle jobs. This is a nice way to separate RocksDB specific commands from Sandcastle specific job definition. Test Plan: None. Will be tested when we add Sandcastle job. Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45741 | 31 August 2015, 19:37:26 UTC |
| 3d78eb6 | sdong | 26 August 2015, 21:19:31 UTC | Arena usage to be calculated using malloc_usable_size() Summary: malloc_usable_size() gets a better estimation of memory usage. It is already used to calculate block cache memory usage. Use it in arena too. Test Plan: Run all unit tests Reviewers: anthony, kradhakrishnan, rven, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43317 | 31 August 2015, 16:39:27 UTC |
| effd9dd | Andres Noetzli | 29 August 2015, 01:06:32 UTC | Fix deadlock in WAL sync Summary: MarkLogsSynced() was doing `logs_.erase(it++);`. The standard is saying: ``` all iterators and references are invalidated, unless the erased members are at an end (front or back) of the deque (in which case only iterators and references to the erased members are invalidated) ``` Because `it` is an iterator to the first element of the container, it is invalidated, only one iteration is executed and `log.getting_synced = false;` is not being done, so `while (logs_.front().getting_synced)` in `WriteImpl()` is not terminating. Test Plan: make db_bench && ./db_bench --benchmarks=fillsync Reviewers: igor, rven, IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, sdong, tnovak Reviewed By: tnovak Subscribers: kolmike, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45807 | 29 August 2015, 01:06:32 UTC |
| 64f07de | Praveen Rao | 28 August 2015, 18:17:02 UTC | remove spurious compression definitions | 28 August 2015, 18:17:02 UTC |
| 72a9b73 | Andres Noetzli | 28 August 2015, 18:13:20 UTC | Removed unnecessary checks in DBTest.ApproximateMemoryUsage Summary: Just realized that after D45675, part of the code in DBTest.ApproximateMemoryUsage, does not really test anything anymore, so I removed it. Test Plan: make clean all check Reviewers: rven, igor, sdong, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45783 | 28 August 2015, 18:13:20 UTC |
| cb164bf | Venkatesh Radhakrishnan | 28 August 2015, 18:07:07 UTC | Do not delete iterators for immutable memtables. Summary: The immutable memtable iterators are allocated from an arena and there is no benefit from deleting these. Also the immutable memtables themselves will continue to be in memory until the version set containing it is alive. We will not remove immutable memtable iterators over the upper bound. We now add immutable iterators to the test. Test Plan: db_tailing_iter_test.TailingIteratorTrimSeekToNext Reviewers: tnovak, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45597 | 28 August 2015, 18:07:07 UTC |
| 7a0dbdf | sdong | 27 August 2015, 22:40:42 UTC | Add ZSTD (not final format) compression type Summary: Add ZSTD compression type. The same way as adding LZ4. Test Plan: run all tests. Generate files in db_bench. Make sure reads succeed. But the SST files cannot be opened in older versions. Also some other adhoc tests. Reviewers: rven, anthony, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: MarkCallaghan, maykov, yoshinorim, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45747 | 28 August 2015, 18:01:13 UTC |
| e2db15e | Igor Canadi | 28 August 2015, 00:13:55 UTC | Merge pull request #701 from PraveenSinghRao/usewinapi_notcruntime Remove usage of C runtime API that has file handle limitation | 28 August 2015, 00:13:55 UTC |
| e853191 | Andres Noetzli | 27 August 2015, 23:17:08 UTC | Fix DBTest.ApproximateMemoryUsage Summary: This patch fixes two issues in DBTest.ApproximateMemoryUsage: - It was possible that a flush happened between getting the two properties in Phase 1, resulting in different numbers for the properties and failing the assertion. This is fixed by waiting for the flush to finish before getting the properties. - There was a similar issue in Phase 2 and additionally there was an issue that rocksdb.size-all-mem-tables was not monotonically increasing because it was possible that a flush happened just after getting the properties and then another flush just before getting the properties in the next round. In this situation, the reported memory usage decreased. This is fixed by forcing a flush before getting the properties. Note: during testing, I found that kFlushesPerRound does not seem very accurate. I added a TODO for this and it would be great to get some input on what to do there. Test Plan: The first issue can be made more likely to trigger by inserting a `usleep(10000);` between the calls to GetIntProperty() in Phase 1. The second issue can be made more likely to trigger by inserting a `if (r != 0) usleep(10000);` before the calls to GetIntProperty() and a `usleep(10000);` after the calls. Then execute make db_test && ./db_test --gtest_filter=DBTest.ApproximateMemoryUsage Reviewers: rven, yhchiang, igor, sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45675 | 27 August 2015, 23:17:08 UTC |
| 7c916a5 | Siying Dong | 27 August 2015, 17:06:48 UTC | Merge pull request #699 from OpenChannelSSD/to_fb_master Helper functions to support direct IO | 27 August 2015, 17:06:48 UTC |
| 0886f4f | Javier González | 26 August 2015, 13:28:09 UTC | Helper functions to support direct IO Summary: This patch adds the helper functions and variables to allow a backend implementing WritableFile to support direct IO when persisting a memtable. Test Plan: Since there is no upstream implementation of WritableFile supporting direct IO, the new behavior is disabled. Tests should be provided by the backend implementing WritableFile. | 27 August 2015, 06:36:56 UTC |
| 7e32798 | Praveen Rao | 27 August 2015, 01:51:18 UTC | Remove usage of C runtime API that has file handle limitation | 27 August 2015, 01:51:18 UTC |
| 8ef0144 | Yueh-Hsuan Chiang | 27 August 2015, 01:27:23 UTC | Add argument --show_table_properties to db_bench Summary: Add argument --show_table_properties to db_bench -show_table_properties (If true, then per-level table properties will be printed on every stats-interval when stats_interval is set and stats_per_interval is on.) type: bool default: false Test Plan: ./db_bench --show_table_properties=1 --stats_interval=100000 --stats_per_interval=1 ./db_bench --show_table_properties=1 --stats_interval=100000 --stats_per_interval=1 --num_column_families=2 Sample Output: Compaction Stats [column_family_name_000001] Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) Stall(cnt) KeyIn KeyDrop --------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 5 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 86.3 0 17 0.021 0 0 0 L1 5/0 9 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 0 L2 9/0 16 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 0 Sum 17/0 31 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 86.3 0 17 0.021 0 0 0 Int 0/0 0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 83.9 0 2 0.022 0 0 0 Flush(GB): cumulative 0.030, interval 0.004 Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown_soft, 0 leveln_slowdown_hard Level[0]: # data blocks=2571; # entries=84813; raw key size=2035512; raw average key size=24.000000; raw value size=8481300; raw average value size=100.000000; data block size=5690119; index block size=82415; filter block size=0; (estimated) table size=5772534; filter policy name=N/A; Level[1]: # data blocks=4285; # entries=141355; raw key size=3392520; raw average key size=24.000000; raw value size=14135500; raw average value size=100.000000; data block size=9487353; index block size=137377; filter block size=0; (estimated) table size=9624730; filter policy name=N/A; Level[2]: # data blocks=7713; # entries=254439; raw key size=6106536; raw average key size=24.000000; raw value size=25443900; raw average value size=100.000000; data block size=17077893; index block size=247269; filter block size=0; (estimated) table size=17325162; filter policy name=N/A; Level[3]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Level[4]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Level[5]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Level[6]: # data blocks=0; # entries=0; raw key size=0; raw average key size=0.000000; raw value size=0; raw average value size=0.000000; data block size=0; index block size=0; filter block size=0; (estimated) table size=0; filter policy name=N/A; Reviewers: anthony, IslamAbdelRahman, MarkCallaghan, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45651 | 27 August 2015, 01:27:23 UTC |
| 1fb2aba | Yueh-Hsuan Chiang | 26 August 2015, 23:13:56 UTC | ColumnFamilyOptions serialization / deserialization. Summary: This patch adds GetStringFromColumnFamilyOptions(), the inverse function of the existing GetColumnFamilyOptionsFromString(), and improves the implementation of GetColumnFamilyOptionsFromString(). Test Plan: Add a test in options_test.cc Reviewers: igor, sdong, anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: noetzli, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45009 | 26 August 2015, 23:13:56 UTC |
| 5f4166c | Igor Canadi | 26 August 2015, 22:25:59 UTC | ReadaheadRandomAccessFile -- userspace readahead Summary: ReadaheadRandomAccessFile acts as a transparent layer on top of RandomAccessFile. When a Read() request is issued, it issues a much bigger request to the OS and caches the result. When a new request comes in and we already have the data cached, it doesn't have to issue any requests to the OS. We add ReadaheadRandomAccessFile layer only when file is read during compactions. D45105 was incorrectly closed by Phabricator because I committed it to a separate branch (not master), so I'm resubmitting the diff. Test Plan: make check Reviewers: MarkCallaghan, sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45123 | 26 August 2015, 22:25:59 UTC |
| 16ebe3a | Igor Canadi | 26 August 2015, 21:51:38 UTC | Mmap reads should not return error if reading past file Summary: Currently, mmap returns IOError when user tries to read data past the end of the file. This diff changes the behavior. Now, we return just the bytes that we can, and report the size we returned via a Slice result. This is consistent with non-mmap behavior and also pread() system call. This diff is taken out of D45123. Test Plan: make check Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45645 | 26 August 2015, 21:51:38 UTC |
| d286b5d | sdong | 25 August 2015, 20:40:52 UTC | DBIter to out extra keys with higher sequence numbers when changing direction from forward to backward Summary: When DBIter changes iterating direction from forward to backward, it might see some much larger keys with higher sequence ID. With this commit, these rows will be actively filtered out. It should fix existing disabled tests in db_iter_test. This may not be a perfect fix, but it introduces least impact on existing codes, in order to be safe. Test Plan: Enable existing tests and make sure they pass. Add a new test DBIterWithMergeIterTest.InnerMergeIteratorDataRace8. Also run all existing tests. Reviewers: yhchiang, rven, anthony, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D45567 | 26 August 2015, 20:01:39 UTC |
| 3795449 | Andres Noetzli | 26 August 2015, 17:10:26 UTC | Fix DBTest.GetProperty Summary: DBTest.GetProperty was failing occasionally (see task #8131266). The reason was that the test closed the database before the compaction was done. When the test reopened the database, RocksDB would schedule a compaction which in turn created table readers and lead the test to fail the assertion that rocksdb.estimate-table-readers-mem is 0. In most cases, GetIntProperty() of rocksdb.estimate-table-readers-mem happened before the compaction created the table readers, hiding the problem. This patch changes the WaitForFlushMemTable() to WaitForCompact(). WaitForFlushMemTable() is not necessary because it is already being called a couple of lines before without any insertions in-between. Test Plan: Insert `usleep(10000);` just after `Reopen(options);` on line 2333 to make the issue more likely, then run: make db_test && while ./db_test --gtest_filter=DBTest.GetProperty; do true; done Reviewers: rven, yhchiang, anthony, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45603 | 26 August 2015, 17:10:26 UTC |
| a7834a1 | Igor Canadi | 26 August 2015, 00:15:23 UTC | Merge pull request #698 from yuslepukhin/address_noexcept_windows Address noexcept and const integer lambda capture on win | 26 August 2015, 00:15:23 UTC |
| 9ccf1bd | Yueh-Hsuan Chiang | 25 August 2015, 23:45:23 UTC | Correct the comment for GetProperty() API. Summary: "rocksdb.aggregated-table-properties" and "rocksdb.aggregated-table-properties-at-level<N>" should belong to GetProperty() instead of GetIntProperty(), but the comment mistakenly classifies them to GetIntProperty(). This patch fix this comment error. Test Plan: no code change. Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45561 | 25 August 2015, 23:45:23 UTC |
| fbe2c05 | Dmitri Smirnov | 25 August 2015, 23:34:39 UTC | s/NOEXCEPT/ROCKSDB_NOEXCEPT | 25 August 2015, 23:34:39 UTC |
| 6924d75 | Dmitri Smirnov | 25 August 2015, 22:17:14 UTC | Address noexcept and const integer lambda capture VS 2013 does not support noexcept. Complains about usage of ineteger constant within lambda requiring explicit capture. | 25 August 2015, 22:17:14 UTC |
| 2f8d71e | Ari Ekmekji | 25 August 2015, 21:03:10 UTC | Moving sequence number compaction variables from SubCompactionState to CompactionJob Summary: It was pointed out to me that the members of SubCompactionState 'earliest_snapshot', 'latest_snapshot' and 'visible_at_tip' are never modified by the subcompactions, so they can stay as global varaibles instead to make things simpler. Test Plan: make all && make check Reviewers: sdong, igor, noetzli, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45477 | 25 August 2015, 21:03:10 UTC |
| bab9934 | Venkatesh Radhakrishnan | 25 August 2015, 21:02:03 UTC | Fix build failure caused by bad merge. Summary: There was a bad merge during refresh. Test Plan: make -j all; make check Reviewers: sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45555 | 25 August 2015, 21:02:03 UTC |
| 4d28a7d | Venkatesh Radhakrishnan | 25 August 2015, 20:40:58 UTC | Add a whitebox test for deleted file iterators. Summary: We have earlier added a feature to delete file iterators when the current key is over the iterate upper bound. We now add a whitebox test to check if the file iterators were actually deleted. Test Plan: Add check for a range which has deleted iterators. Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45321 | 25 August 2015, 20:40:58 UTC |
| 249fb4f | Venkatesh Radhakrishnan | 25 August 2015, 20:38:35 UTC | Fix use of deleted file iterators with incomplete iterators Summary: After deleting file iterators which are over the iterate upper bound, we also need to check for null pointers in ResetIncompletIterators. Test Plan: db_tailing_iter_test.TailingIteratorTrimSeekToNext Reviewers: tnovak, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45525 | 25 August 2015, 20:38:35 UTC |
| 53b8878 | Igor Canadi | 25 August 2015, 20:32:46 UTC | Add throttling to multi-threaded backups Summary: See internal task t8056182 Test Plan: Added multi-threading in RateLimiter test Reviewers: benj, AaronFeldman Reviewed By: AaronFeldman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45459 | 25 August 2015, 20:32:46 UTC |
| 09d982f | Andres Notzli | 25 August 2015, 19:29:44 UTC | Fix compact_files_example Summary: See task #7983654. The example was triggering an assert in compaction job because the compaction was not marked as manual. With this patch, CompactionPicker::FormCompaction() marks compactions as manual. This patch also fixes a couple of typos, adds optimistic_transaction_example to .gitignore and librocksdb as a dependency for examples. Adding librocksdb as a dependency makes sure that the examples are built with the latest changes in librocksdb. Test Plan: make clean && cd examples && make all && ./compact_files_example Reviewers: rven, sdong, anthony, igor, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45117 | 25 August 2015, 19:29:44 UTC |
| 6996de8 | Yueh-Hsuan Chiang | 25 August 2015, 19:03:54 UTC | Expose per-level aggregated table properties via GetProperty() Summary: This patch adds "rocksdb.aggregated-table-properties" and "rocksdb.aggregated-table-properties-at-levelN", the former returns the aggreated table properties of a column family, while the later returns the aggregated table properties of the specified level N. Test Plan: Added tests in db_test Reviewers: igor, sdong, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45087 | 25 August 2015, 19:03:54 UTC |
| 86d6c3c | agiardullo | 25 August 2015, 17:50:27 UTC | Fix Windows build Summary: wrong filename Test Plan: none Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45531 | 25 August 2015, 18:02:31 UTC |
| 20d1e54 | agiardullo | 21 August 2015, 22:47:21 UTC | Common base class for transactions Summary: As I keep adding new features to transactions, I keep creating more duplicate code. This diff cleans this up by creating a base implementation class for Transaction and OptimisticTransaction to inherit from. The code in TransactionBase.h/.cc is all just copied from elsewhere. The only entertaining part of this class worth looking at is the virtual TryLock method which allows OptimisticTransactions and Transactions to share the same common code for Put/Get/etc. The rest of this diff is mostly red and easy on the eyes. Test Plan: No functionality change. existing tests pass. Reviewers: sdong, jkedgar, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45135 | 25 August 2015, 02:09:43 UTC |
| 2050832 | Andres Noetzli | 25 August 2015, 00:04:18 UTC | Fixing race condition in DBTest.DynamicMemtableOptions Summary: This patch fixes a race condition in DBTEst.DynamicMemtableOptions. In rare cases, it was possible that the main thread would fill up both memtables before the flush job acquired its work. Then, the flush job was flushing both memtables together, producing only one L0 file while the test expected two. Now, the test waits for flushes to finish earlier, to make sure that the memtables are flushed in separate flush jobs. Test Plan: Insert "usleep(10000);" after "IOSTATS_SET_THREAD_POOL_ID(Env::Priority::HIGH);" in BGWorkFlush() to make the issue more likely. Then test with: make db_test && time while ./db_test --gtest_filter=*DynamicMemtableOptions; do true; done Reviewers: rven, sdong, yhchiang, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45429 | 25 August 2015, 00:04:18 UTC |
| e46bcc0 | Igor Canadi | 24 August 2015, 23:43:18 UTC | Remove an extra 's' from cur-size-all-mem-tabless Summary: As title Test Plan: make check Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45447 | 24 August 2015, 23:43:18 UTC |
| 4ab26c5 | Igor Canadi | 24 August 2015, 18:11:12 UTC | Smarter purging during flush Summary: Currently, we only purge duplicate keys and deletions during flush if `earliest_seqno_in_memtable <= newest_snapshot`. This means that the newest snapshot happened before we first created the memtable. This is almost never true for MyRocks and MongoRocks. This patch makes purging during flush able to understand snapshots. The main logic is copied from compaction_job.cc, although the logic over there is much more complicated and extensive. However, we should try to merge the common functionality at some point. I need this patch to implement no_overwrite_i_promise functionality for flush. We'll also need this to support SingleDelete() during Flush(). @yoshinorim requested the feature. Test Plan: make check I had to adjust some unit tests to understand this new behavior Reviewers: yhchiang, yoshinorim, anthony, sdong, noetzli Reviewed By: noetzli Subscribers: yoshinorim, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42087 | 24 August 2015, 18:11:12 UTC |
| 4c81ac0 | Mark Callaghan | 22 August 2015, 19:18:00 UTC | Fix benchmark report script Summary: db_bench output now displays Percentile many times with --statistics after read IO latency histograms were added. So I only need the last one in the report output. Task ID: # Blame Rev: Test Plan: run run_flash_bench.sh Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45093 | 22 August 2015, 19:18:00 UTC |
| b6def58 | Ari Ekmekji | 21 August 2015, 21:25:34 UTC | Changed 'num_subcompactions' to the more accurate 'max_subcompactions' Summary: Up until this point we had DbOptions.num_subcompactions, but it is semantically more correct to call this max_subcompactions since we will schedule *up to* DbOptions.max_subcompactions smaller compactions at a time during a compaction job. I also added a --subcompactions option to db_bench Test Plan: make all make check Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D45069 | 21 August 2015, 21:25:34 UTC |
| c852968 | sdong | 19 August 2015, 01:08:49 UTC | db_iter_test: add more test cases for the data race bug Summary: Add more test cases of data race causing wrong iterating results. Tag tests not passing as DISABLED_ Test Plan: Run the tests Reviewers: igor, rven, IslamAbdelRahman, anthony, kradhakrishnan, yhchiang Reviewed By: yhchiang Subscribers: tnovak, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44907 | 21 August 2015, 19:14:12 UTC |
| 9130873 | sdong | 21 August 2015, 05:33:25 UTC | Add options.new_table_reader_for_compaction_inputs Summary: Currently compaction inputs share the same file descriptor and table reader as other foreground threads. It makes fadvise works less predictable. Add options.new_table_reader_for_compaction_inputs to enforce to create a new file descriptor and new table reader for it. Test Plan: Add the option. Reviewers: rven, anthony, kradhakrishnan, IslamAbdelRahman, igor, yhchiang Reviewed By: igor Subscribers: igor, MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43311 | 21 August 2015, 15:46:29 UTC |
| 07d2d34 | sdong | 14 August 2015, 04:42:20 UTC | Add a counter about estimated pending compaction bytes Summary: Add a counter of estimated bytes the DB needs to compact for all the compactions to finish. Expose it as a DB Property. In the future, we can use threshold of this counter to replace soft rate limit and hard rate limit. A single threshold of estimated compaction debt in bytes will be easier for users to reason about when should slow down and stopping than more abstract soft and hard rate limits. Test Plan: Add unit tests Reviewers: IslamAbdelRahman, yhchiang, rven, kradhakrishnan, anthony, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44205 | 21 August 2015, 05:17:10 UTC |
| 41a0e28 | Mark Callaghan | 21 August 2015, 01:59:10 UTC | Improve defaults for benchmarks Summary: Changes include: * don't sync-on-commit for single writer thread in readwhile... tests * make default block size 8kb rather than 4kb to avoid too small blocks after compression * use snappy instead of zlib to avoid stalls from compression latency * disable statistics * use bytes_per_sync=8M to reduce throughput loss on disk * use open_files=-1 to reduce mutex contention Task ID: # Blame Rev: Test Plan: run benchmark Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44961 | 21 August 2015, 01:59:10 UTC |
| a203b91 | Yueh-Hsuan Chiang | 21 August 2015, 00:18:47 UTC | Fixed a rare deadlock in DBTest.ThreadStatusFlush Summary: Currently, ThreadStatusFlush uses two sync-points to ensure there's a flush currently running when calling GetThreadList(). However, one of the sync-point is inside db-mutex, which could cause deadlock in case there's a DB::Get() call. This patch fix this issue by moving the sync-point to a better place where the flush job does not hold the mutex. Test Plan: db_test Reviewers: igor, sdong, anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45045 | 21 August 2015, 00:18:47 UTC |
| 962aa64 | Siying Dong | 21 August 2015, 00:04:48 UTC | Merge pull request #695 from yuslepukhin/address_windows_build Address windows build issues caused by introducing Subcompaction | 21 August 2015, 00:04:48 UTC |
| 5bf8907 | Dmitri Smirnov | 20 August 2015, 21:14:02 UTC | More indent adjustment. | 20 August 2015, 21:14:02 UTC |
| e2a9f43 | Dmitri Smirnov | 20 August 2015, 21:10:51 UTC | Adjust indent | 20 August 2015, 21:10:51 UTC |
| 6e9a260 | Dmitri Smirnov | 20 August 2015, 21:09:11 UTC | Merge branch 'address_windows_build' of https://github.com/yuslepukhin/rocksdb into address_windows_build | 20 August 2015, 21:09:11 UTC |
| 1cac89c | Dmitri Smirnov | 20 August 2015, 21:08:24 UTC | Address windows build issues Intro SubCompactionState move functionality =delete copy functionality #ifdef SyncPoint in tests for Windows Release builds | 20 August 2015, 21:08:24 UTC |
| f25f06d | Dmitri Smirnov | 20 August 2015, 20:43:07 UTC | Address windows build issues Intro SubCompactionState move functionality =delete copy functionality #ifdef SyncPoint in tests for Windows Release builds | 20 August 2015, 20:43:07 UTC |
| 027ca5b | Islam AbdelRahman | 20 August 2015, 18:47:19 UTC | Total SST files size DB Property Summary: Add a new DB property that calculate the total size of files used by all RocksDB Versions Test Plan: Unittests for the new property Reviewers: igor, yhchiang, anthony, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44799 | 20 August 2015, 18:47:19 UTC |
| b604d25 | Andres Noetzli | 19 August 2015, 23:57:40 UTC | Removing unused variables to fix build Summary: Removing two unused variables that prevented compilation. Test Plan: make all Reviewers: rven, sdong, yhchiang, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44991 | 19 August 2015, 23:57:40 UTC |
| 1b114ee | Venkatesh Radhakrishnan | 19 August 2015, 23:05:51 UTC | Free file iterators for files which are above the iterate upper bound to Improve memory utilization Summary: This diff improves the memory utilization for tailing iterators RocksDB, by freeing file iterators which are over the upper bound. It is an updating on Siying's original diff for improving the memory usage for tailing iterators. The changes for the seek and next path are now complete and a test has been added to exercise these paths while deleting file iterators which are above the upper bound. Test Plan: db_tailing_iter_test.TailingIteratorTrimSeekToNext Reviewers: march, tnovak, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43833 | 19 August 2015, 23:05:51 UTC |
| 3fd70b0 | Islam AbdelRahman | 19 August 2015, 22:02:17 UTC | Rate limit deletes issued by DestroyDB Summary: Update DestroyDB so that all SST files in the first path id go through DeleteScheduler instead of being deleted immediately Test Plan: added a unittest Reviewers: igor, yhchiang, anthony, kradhakrishnan, rven, sdong Reviewed By: sdong Subscribers: jeanxu2012, dhruba Differential Revision: https://reviews.facebook.net/D44955 | 19 August 2015, 22:02:17 UTC |
| df79eaf | Yueh-Hsuan Chiang | 19 August 2015, 20:32:09 UTC | Introduce GetIntProperty("rocksdb.size-all-mem-tables") Summary: Currently, GetIntProperty("rocksdb.cur-size-all-mem-tables") only returns the memory usage by those memtables which have not yet been flushed. This patch introduces GetIntProperty("rocksdb.size-all-mem-tables"), which includes the memory usage by all the memtables, includes those have been flushed but pinned by iterators. Test Plan: Added a test in db_test Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44229 | 19 August 2015, 20:32:09 UTC |
| 888fbdc | sdong | 19 August 2015, 17:08:03 UTC | Remove the contstaint that iterator upper bound needs to be within a prefix Summary: There is a check to fail the iterator if prefix extractor is specified but upper bound is out of the prefix for the seek key. Relax this constraint to allow users to set upper bound to the next prefix of the current one. Test Plan: make commit-prereq Reviewers: igor, anthony, kradhakrishnan, yhchiang, rven Reviewed By: rven Subscribers: tnovak, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44949 | 19 August 2015, 18:03:51 UTC |
| 137c376 | Ari Ekmekji | 19 August 2015, 15:52:22 UTC | Removing variables used only in assertions to prevent build error Summary: A couple variables were declared but only used in assertions which causes issues when building in fbcode. Test Plan: make dbg and make release Reviewers: yhchiang, sdong, igor, anthony, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44937 | 19 August 2015, 15:52:22 UTC |
| b47cc58 | Ari Ekmekji | 18 August 2015, 21:56:31 UTC | Bounding Number of Subcompactions Summary: In D43239 (https://reviews.facebook.net/D43239) the number of subcompactions is set based on the number of L1 files with unique starting keys. In certain cases when this number is very large this causes issues, particularly with the overlap between files since very small output files can be generated. This diff bounds the number of subcompactions to the user option DBOption.num_subcompactions. Test Plan: ./db_test ./db_compaction_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44883 | 18 August 2015, 21:56:31 UTC |
| e58e1b1 | Venkatesh Radhakrishnan | 18 August 2015, 21:40:06 UTC | Make tailing iterator show new entries in memtable. Summary: Reseek mutable_iter if it is invalid in Next and immutable_iter is invalid. Test Plan: DBTestTailingIterator.TailingIteratorSeekToNext Reviewers: tnovak, march, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44865 | 18 August 2015, 21:40:06 UTC |
| 9ec9571 | Yueh-Hsuan Chiang | 18 August 2015, 20:30:18 UTC | DBOptions serialization and deserialization Summary: This patch implements DBOptions deserialization and improve the current implementation of DBOptions serialization by using a static structure that stores the offset of each DBOptions member variables to perform serialization and deserialization instead of using tons of if-then-branch to determine the mapping between string and variables. Test Plan: Added test in options_test.cc Reviewers: igor, anthony, sdong, IslamAbdelRahman Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44097 | 18 August 2015, 20:30:18 UTC |
| b2df20a | Yueh-Hsuan Chiang | 18 August 2015, 20:19:55 UTC | Make HashCuckooRep::ApproximateMemoryUsage() return reasonable estimation. Summary: HashCuckooRep::ApproximateMemoryUsage() previously return std::numeric_limits<size_t>::max() when it cannot accept more entries. This patch makes it return a more reasonable estimation. This change is necessary in order to make GetIntProperty("rocksdb.cur-size-all-mem-tables") handles HashCuckooRep properly in diff https://reviews.facebook.net/D44229. Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44241 | 18 August 2015, 20:19:55 UTC |
| 601b1aa | Ari Ekmekji | 18 August 2015, 19:27:12 UTC | Fixing Failed Assertion in Subcompaction State Diff Summary: In D43239 (https://reviews.facebook.net/D43239) there is an assertion to make sure a subcompaction's output is never empty at the end of execution. This assertion however breaks the build because some tests lead to exactly that scenario. So instead I have altered the logic to handle this case instead of just failing the assertion. The reason that it is possible for a subcompaction's output to be empty is that during a sequential execution of subcompactions, if a user aborts the compaction job then some of the later subcompactions to be executed may have yet to process any keys and therefore have yet to generate output files. This becomes very rare once the subcompactions are executed in parallel, but for now they are still sequential so the case is possible when there is an early termination, as in some of the tests. Test Plan: ./db_test ./db_compaction_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44877 | 18 August 2015, 19:27:12 UTC |
| f0da697 | Ari Ekmekji | 18 August 2015, 18:06:23 UTC | [Parallel L0-L1 Compaction Prep]: Giving Subcompactions Their Own State Summary: In prepration for running multiple threads at the same time during a compaction job, this patch assigns each subcompaction its own state (instead of sharing the one global CompactionState). Each subcompaction then uses this state to update its statistics, keep track of its snapshots, etc. during the course of execution. Then at the end of all the executions the statistics are aggregated across the subcompactions so that the final result is the same as if only one larger compaction had run. Test Plan: ./db_test ./db_compaction_test ./compaction_job_test Reviewers: sdong, anthony, igor, noetzli, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43239 | 18 August 2015, 18:06:23 UTC |
| f32a572 | Andres Notzli | 18 August 2015, 00:34:38 UTC | Simplify querying of merge results Summary: While working on supporting mixing merge operators with single deletes ( https://reviews.facebook.net/D43179 ), I realized that returning and dealing with merge results can be made simpler. Submitting this as a separate diff because it is not directly related to single deletes. Before, callers of merge helper had to retrieve the merge result in one of two ways depending on whether the merge was successful or not (success = result of merge was single kTypeValue). For successful merges, the caller could query the resulting key/value pair and for unsuccessful merges, the result could be retrieved in the form of two deques of keys and values. However, with single deletes, a successful merge does not return a single key/value pair (if merge operands are merged with a single delete, we have to generate a value and keep the original single delete around to make sure that we are not accidentially producing a key overwrite). In addition, the two existing call sites of the merge helper were taking the same actions independently from whether the merge was successful or not, so this patch simplifies that. Test Plan: make clean all check Reviewers: rven, sdong, yhchiang, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43353 | 18 August 2015, 00:34:38 UTC |
| 7261365 | sdong | 13 August 2015, 21:35:54 UTC | Measure file read latency histogram per level Summary: In internal stats, remember read latency histogram, if statistics is enabled. It can be retrieved from DB::GetProperty() with "rocksdb.dbstats" property, if it is enabled. Test Plan: Manually run db_bench and prints out "rocksdb.dbstats" by hand and make sure it prints out as expected Reviewers: igor, IslamAbdelRahman, rven, kradhakrishnan, anthony, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44193 | 15 August 2015, 00:32:42 UTC |
| b7198c3 | Nathan Bronson | 05 August 2015, 23:56:28 UTC | reduce db mutex contention for write batch groups Summary: This diff allows a Writer to join the next write batch group without acquiring any locks. Waiting is performed via a per-Writer mutex, so all of the non-leader writers never need to acquire the db mutex. It is now possible to join a write batch group after the leader has been chosen but before the batch has been constructed. This diff doesn't increase parallelism, but reduces synchronization overheads. For some CPU-bound workloads (no WAL, RAM-sized working set) this can substantially reduce contention on the db mutex in a multi-threaded environment. With T=8 N=500000 in a CPU-bound scenario (see the test plan) this is good for a 33% perf win. Not all scenarios see such a win, but none show a loss. This code is slightly faster even for the single-threaded case (about 2% for the CPU-bound scenario below). Test Plan: 1. unit tests 2. COMPILE_WITH_TSAN=1 make check 3. stress high-contention scenarios with db_bench -benchmarks=fillrandom -threads=$T -batch_size=1 -memtablerep=skip_list -value_size=0 --num=$N -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -disable_auto_compactions --max_write_buffer_number=8 -max_background_flushes=8 --disable_wal --write_buffer_size=160000000 Reviewers: sdong, igor, rven, ljin, yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43887 | 14 August 2015, 17:55:43 UTC |
| 603b6da | sdong | 13 August 2015, 00:24:45 UTC | Add options.compaction_measure_io_stats to print write I/O stats in compactions Summary: Add options.compaction_measure_io_stats to print out / pass to listener accumulated time spent on write calls. Example outputs in info logs: 2015/08/12-16:27:59.463944 7fd428bff700 (Original Log Time 2015/08/12-16:27:59.463922) EVENT_LOG_v1 {"time_micros": 1439422079463897, "job": 6, "event": "compaction_finished", "output_level": 1, "num_output_files": 4, "total_output_size": 6900525, "num_input_records": 111483, "num_output_records": 106877, "file_write_nanos": 15663206, "file_range_sync_nanos": 649588, "file_fsync_nanos": 349614797, "file_prepare_write_nanos": 1505812, "lsm_state": [2, 4, 0, 0, 0, 0, 0]} Add two more counters in iostats_context. Also add a parameter of db_bench. Test Plan: Add a unit test. Also manually verify LOG outputs in db_bench Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44115 | 13 August 2015, 23:52:26 UTC |
| dc9d563 | Islam AbdelRahman | 13 August 2015, 18:17:10 UTC | Change master to 3.14 Summary: Change master version to 3.14 Test Plan: simple change Reviewers: sdong, yhchiang, kradhakrishnan, rven, anthony, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44187 | 13 August 2015, 18:17:10 UTC |
| b78c8e0 | Igor Canadi | 13 August 2015, 17:49:10 UTC | Merge pull request #689 from msb-at-yahoo/add-tools-target Add a 'tools' target. | 13 August 2015, 17:49:10 UTC |
| 9f0dd22 | maurice barnum | 13 August 2015, 17:13:43 UTC | Add a 'tools' target. My use case is to build the rocksdb static library and tools, and ideally I'd like to not spend time building the shared library and other targets that I won't use. | 13 August 2015, 17:19:53 UTC |
| 4637207 | sdong | 06 August 2015, 20:23:02 UTC | Add test case to repro the mispositional iterator in a low-chance data race case Summary: Iterator has a bug: if a child iterator reaches its end, and user issues a Prev(), and just before SeekToLast() of the child iterator is called, some extra rows is added in the end, the position of iterator can be misplaced. Test Plan: Run the tests with or without valgrind Reviewers: rven, yhchiang, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: tnovak, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43671 | 12 August 2015, 17:50:52 UTC |
| 3bd9db4 | Islam AbdelRahman | 12 August 2015, 17:18:59 UTC | [Cleanup] Remove RandomRWFile Summary: RandomRWFile is not used anywhere in out code base, this patch remove RandomRWFile Test Plan: make check -j64 USE_CLANG=1 make all -j64 OPT=-DROCKSDB_LITE make release -j64 Reviewers: sdong, yhchiang, anthony, kradhakrishnan, rven, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44091 | 12 August 2015, 17:18:59 UTC |
| c3466ea | agiardullo | 31 July 2015, 22:18:27 UTC | Have Transactions use WriteBatch::RollbackToSavePoint Summary: Clean up transactions to use the new RollbackToSavePoint api in WriteBatchWithIndex. Note, this diff depends on Pessimistic Transactions diff and ManagedSnapshot diff (D40869 and D43293). Test Plan: unit tests Reviewers: rven, yhchiang, kradhakrishnan, spetrunia, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43371 | 12 August 2015, 00:53:30 UTC |
| 0db807e | agiardullo | 31 July 2015, 03:31:36 UTC | Transaction error statuses Summary: Based on feedback from spetrunia, we should better differentiate error statuses for transaction failures. https://github.com/MySQLOnRocksDB/mysql-5.6/issues/86#issuecomment-124605954 Test Plan: unit tests Reviewers: rven, kradhakrishnan, spetrunia, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43323 | 12 August 2015, 00:52:56 UTC |
| c2f2cb0 | agiardullo | 26 May 2015, 00:37:33 UTC | Pessimistic Transactions Summary: Initial implementation of Pessimistic Transactions. This diff contains the api changes discussed in D38913. This diff is pretty large, so let me know if people would prefer to meet up to discuss it. MyRocks folks: please take a look at the API in include/rocksdb/utilities/transaction[_db].h and let me know if you have any issues. Also, you'll notice a couple of TODOs in the implementation of RollbackToSavePoint(). After chatting with Siying, I'm going to send out a separate diff for an alternate implementation of this feature that implements the rollback inside of WriteBatch/WriteBatchWithIndex. We can then decide which route is preferable. Next, I'm planning on doing some perf testing and then integrating this diff into MongoRocks for further testing. Test Plan: Unit tests, db_bench parallel testing. Reviewers: igor, rven, sdong, yhchiang, yoshinorim Reviewed By: sdong Subscribers: hermanlee4, maykov, spetrunia, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40869 | 12 August 2015, 00:52:23 UTC |
| c2868cb | Islam AbdelRahman | 11 August 2015, 21:47:14 UTC | Use manual_compaction for compaction_job_test Summary: Under certain conditions (disable compression) the compactions that are created in compaction_job_test will pass the trivial_move conditions This will cause problems since we assert that we dont run a compaction if it's a trivial move https://github.com/facebook/rocksdb/blob/master/db/compaction_job.cc#L144-L147 for example when we disable compression, compactions become a valid trivial move and the assert fails https://ci-builds.fb.com/view/rocksdb/job/rocksdb_no_compression/180/console Test Plan: compaction_job_test Reviewers: sdong, yhchiang, noetzli, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43983 | 11 August 2015, 21:47:14 UTC |
| 6b2d570 | agiardullo | 11 August 2015, 20:44:47 UTC | Fix Windows build by adding snapshot_impl to CMakeLists Test Plan: untested Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44049 | 11 August 2015, 20:49:05 UTC |
| e61fafb | Yueh-Hsuan Chiang | 11 August 2015, 20:30:49 UTC | Fixed clang-build error in util/thread_local.cc Summary: This patch fixes the following clang-build error in util/thread_local.cc by using a cleaner macro blocker: 12:26:31 util/thread_local.cc:157:19: error: declaration shadows a static data member of 'rocksdb::ThreadLocalPtr::StaticMeta' [-Werror,-Wshadow] 12:26:31 ThreadData* tls_ = 12:26:31 ^ 12:26:31 util/thread_local.cc:19:66: note: previous declaration is here 12:26:31 __thread ThreadLocalPtr::ThreadData* ThreadLocalPtr::StaticMeta::tls_ = nullptr; 12:26:31 ^ Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44043 | 11 August 2015, 20:30:49 UTC |
| cee1e8a | Islam AbdelRahman | 11 August 2015, 19:19:56 UTC | Parallelize LoadTableHandlers Summary: Add a new option that all LoadTableHandlers to use multiple threads to load files on DB Open and Recover Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 DISABLE_JEMALLOC=1 make all valgrind_check -j64 (still running) Reviewers: yhchiang, anthony, rven, kradhakrishnan, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43755 | 11 August 2015, 19:19:56 UTC |
