| 57b3a87 | krad | 31 August 2015, 19:37:26 UTC | Adding sandcastle determinator for RocksDB Summary: This fuels commands to be executed for different sandcastle jobs. This is a nice way to separate RocksDB specific commands from Sandcastle specific job definition. Test Plan: None. Will be tested when we add Sandcastle job. Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D45741 | 31 August 2015, 19:37:26 UTC |
| 027ca5b | Islam AbdelRahman | 20 August 2015, 18:47:19 UTC | Total SST files size DB Property Summary: Add a new DB property that calculate the total size of files used by all RocksDB Versions Test Plan: Unittests for the new property Reviewers: igor, yhchiang, anthony, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44799 | 20 August 2015, 18:47:19 UTC |
| b604d25 | Andres Noetzli | 19 August 2015, 23:57:40 UTC | Removing unused variables to fix build Summary: Removing two unused variables that prevented compilation. Test Plan: make all Reviewers: rven, sdong, yhchiang, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44991 | 19 August 2015, 23:57:40 UTC |
| 1b114ee | Venkatesh Radhakrishnan | 19 August 2015, 23:05:51 UTC | Free file iterators for files which are above the iterate upper bound to Improve memory utilization Summary: This diff improves the memory utilization for tailing iterators RocksDB, by freeing file iterators which are over the upper bound. It is an updating on Siying's original diff for improving the memory usage for tailing iterators. The changes for the seek and next path are now complete and a test has been added to exercise these paths while deleting file iterators which are above the upper bound. Test Plan: db_tailing_iter_test.TailingIteratorTrimSeekToNext Reviewers: march, tnovak, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43833 | 19 August 2015, 23:05:51 UTC |
| 3fd70b0 | Islam AbdelRahman | 19 August 2015, 22:02:17 UTC | Rate limit deletes issued by DestroyDB Summary: Update DestroyDB so that all SST files in the first path id go through DeleteScheduler instead of being deleted immediately Test Plan: added a unittest Reviewers: igor, yhchiang, anthony, kradhakrishnan, rven, sdong Reviewed By: sdong Subscribers: jeanxu2012, dhruba Differential Revision: https://reviews.facebook.net/D44955 | 19 August 2015, 22:02:17 UTC |
| df79eaf | Yueh-Hsuan Chiang | 19 August 2015, 20:32:09 UTC | Introduce GetIntProperty("rocksdb.size-all-mem-tables") Summary: Currently, GetIntProperty("rocksdb.cur-size-all-mem-tables") only returns the memory usage by those memtables which have not yet been flushed. This patch introduces GetIntProperty("rocksdb.size-all-mem-tables"), which includes the memory usage by all the memtables, includes those have been flushed but pinned by iterators. Test Plan: Added a test in db_test Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44229 | 19 August 2015, 20:32:09 UTC |
| 888fbdc | sdong | 19 August 2015, 17:08:03 UTC | Remove the contstaint that iterator upper bound needs to be within a prefix Summary: There is a check to fail the iterator if prefix extractor is specified but upper bound is out of the prefix for the seek key. Relax this constraint to allow users to set upper bound to the next prefix of the current one. Test Plan: make commit-prereq Reviewers: igor, anthony, kradhakrishnan, yhchiang, rven Reviewed By: rven Subscribers: tnovak, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44949 | 19 August 2015, 18:03:51 UTC |
| 137c376 | Ari Ekmekji | 19 August 2015, 15:52:22 UTC | Removing variables used only in assertions to prevent build error Summary: A couple variables were declared but only used in assertions which causes issues when building in fbcode. Test Plan: make dbg and make release Reviewers: yhchiang, sdong, igor, anthony, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44937 | 19 August 2015, 15:52:22 UTC |
| b47cc58 | Ari Ekmekji | 18 August 2015, 21:56:31 UTC | Bounding Number of Subcompactions Summary: In D43239 (https://reviews.facebook.net/D43239) the number of subcompactions is set based on the number of L1 files with unique starting keys. In certain cases when this number is very large this causes issues, particularly with the overlap between files since very small output files can be generated. This diff bounds the number of subcompactions to the user option DBOption.num_subcompactions. Test Plan: ./db_test ./db_compaction_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44883 | 18 August 2015, 21:56:31 UTC |
| e58e1b1 | Venkatesh Radhakrishnan | 18 August 2015, 21:40:06 UTC | Make tailing iterator show new entries in memtable. Summary: Reseek mutable_iter if it is invalid in Next and immutable_iter is invalid. Test Plan: DBTestTailingIterator.TailingIteratorSeekToNext Reviewers: tnovak, march, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44865 | 18 August 2015, 21:40:06 UTC |
| 9ec9571 | Yueh-Hsuan Chiang | 18 August 2015, 20:30:18 UTC | DBOptions serialization and deserialization Summary: This patch implements DBOptions deserialization and improve the current implementation of DBOptions serialization by using a static structure that stores the offset of each DBOptions member variables to perform serialization and deserialization instead of using tons of if-then-branch to determine the mapping between string and variables. Test Plan: Added test in options_test.cc Reviewers: igor, anthony, sdong, IslamAbdelRahman Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44097 | 18 August 2015, 20:30:18 UTC |
| b2df20a | Yueh-Hsuan Chiang | 18 August 2015, 20:19:55 UTC | Make HashCuckooRep::ApproximateMemoryUsage() return reasonable estimation. Summary: HashCuckooRep::ApproximateMemoryUsage() previously return std::numeric_limits<size_t>::max() when it cannot accept more entries. This patch makes it return a more reasonable estimation. This change is necessary in order to make GetIntProperty("rocksdb.cur-size-all-mem-tables") handles HashCuckooRep properly in diff https://reviews.facebook.net/D44229. Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44241 | 18 August 2015, 20:19:55 UTC |
| 601b1aa | Ari Ekmekji | 18 August 2015, 19:27:12 UTC | Fixing Failed Assertion in Subcompaction State Diff Summary: In D43239 (https://reviews.facebook.net/D43239) there is an assertion to make sure a subcompaction's output is never empty at the end of execution. This assertion however breaks the build because some tests lead to exactly that scenario. So instead I have altered the logic to handle this case instead of just failing the assertion. The reason that it is possible for a subcompaction's output to be empty is that during a sequential execution of subcompactions, if a user aborts the compaction job then some of the later subcompactions to be executed may have yet to process any keys and therefore have yet to generate output files. This becomes very rare once the subcompactions are executed in parallel, but for now they are still sequential so the case is possible when there is an early termination, as in some of the tests. Test Plan: ./db_test ./db_compaction_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44877 | 18 August 2015, 19:27:12 UTC |
| f0da697 | Ari Ekmekji | 18 August 2015, 18:06:23 UTC | [Parallel L0-L1 Compaction Prep]: Giving Subcompactions Their Own State Summary: In prepration for running multiple threads at the same time during a compaction job, this patch assigns each subcompaction its own state (instead of sharing the one global CompactionState). Each subcompaction then uses this state to update its statistics, keep track of its snapshots, etc. during the course of execution. Then at the end of all the executions the statistics are aggregated across the subcompactions so that the final result is the same as if only one larger compaction had run. Test Plan: ./db_test ./db_compaction_test ./compaction_job_test Reviewers: sdong, anthony, igor, noetzli, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43239 | 18 August 2015, 18:06:23 UTC |
| f32a572 | Andres Notzli | 18 August 2015, 00:34:38 UTC | Simplify querying of merge results Summary: While working on supporting mixing merge operators with single deletes ( https://reviews.facebook.net/D43179 ), I realized that returning and dealing with merge results can be made simpler. Submitting this as a separate diff because it is not directly related to single deletes. Before, callers of merge helper had to retrieve the merge result in one of two ways depending on whether the merge was successful or not (success = result of merge was single kTypeValue). For successful merges, the caller could query the resulting key/value pair and for unsuccessful merges, the result could be retrieved in the form of two deques of keys and values. However, with single deletes, a successful merge does not return a single key/value pair (if merge operands are merged with a single delete, we have to generate a value and keep the original single delete around to make sure that we are not accidentially producing a key overwrite). In addition, the two existing call sites of the merge helper were taking the same actions independently from whether the merge was successful or not, so this patch simplifies that. Test Plan: make clean all check Reviewers: rven, sdong, yhchiang, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43353 | 18 August 2015, 00:34:38 UTC |
| 7261365 | sdong | 13 August 2015, 21:35:54 UTC | Measure file read latency histogram per level Summary: In internal stats, remember read latency histogram, if statistics is enabled. It can be retrieved from DB::GetProperty() with "rocksdb.dbstats" property, if it is enabled. Test Plan: Manually run db_bench and prints out "rocksdb.dbstats" by hand and make sure it prints out as expected Reviewers: igor, IslamAbdelRahman, rven, kradhakrishnan, anthony, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44193 | 15 August 2015, 00:32:42 UTC |
| b7198c3 | Nathan Bronson | 05 August 2015, 23:56:28 UTC | reduce db mutex contention for write batch groups Summary: This diff allows a Writer to join the next write batch group without acquiring any locks. Waiting is performed via a per-Writer mutex, so all of the non-leader writers never need to acquire the db mutex. It is now possible to join a write batch group after the leader has been chosen but before the batch has been constructed. This diff doesn't increase parallelism, but reduces synchronization overheads. For some CPU-bound workloads (no WAL, RAM-sized working set) this can substantially reduce contention on the db mutex in a multi-threaded environment. With T=8 N=500000 in a CPU-bound scenario (see the test plan) this is good for a 33% perf win. Not all scenarios see such a win, but none show a loss. This code is slightly faster even for the single-threaded case (about 2% for the CPU-bound scenario below). Test Plan: 1. unit tests 2. COMPILE_WITH_TSAN=1 make check 3. stress high-contention scenarios with db_bench -benchmarks=fillrandom -threads=$T -batch_size=1 -memtablerep=skip_list -value_size=0 --num=$N -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -disable_auto_compactions --max_write_buffer_number=8 -max_background_flushes=8 --disable_wal --write_buffer_size=160000000 Reviewers: sdong, igor, rven, ljin, yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43887 | 14 August 2015, 17:55:43 UTC |
| 603b6da | sdong | 13 August 2015, 00:24:45 UTC | Add options.compaction_measure_io_stats to print write I/O stats in compactions Summary: Add options.compaction_measure_io_stats to print out / pass to listener accumulated time spent on write calls. Example outputs in info logs: 2015/08/12-16:27:59.463944 7fd428bff700 (Original Log Time 2015/08/12-16:27:59.463922) EVENT_LOG_v1 {"time_micros": 1439422079463897, "job": 6, "event": "compaction_finished", "output_level": 1, "num_output_files": 4, "total_output_size": 6900525, "num_input_records": 111483, "num_output_records": 106877, "file_write_nanos": 15663206, "file_range_sync_nanos": 649588, "file_fsync_nanos": 349614797, "file_prepare_write_nanos": 1505812, "lsm_state": [2, 4, 0, 0, 0, 0, 0]} Add two more counters in iostats_context. Also add a parameter of db_bench. Test Plan: Add a unit test. Also manually verify LOG outputs in db_bench Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D44115 | 13 August 2015, 23:52:26 UTC |
| dc9d563 | Islam AbdelRahman | 13 August 2015, 18:17:10 UTC | Change master to 3.14 Summary: Change master version to 3.14 Test Plan: simple change Reviewers: sdong, yhchiang, kradhakrishnan, rven, anthony, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44187 | 13 August 2015, 18:17:10 UTC |
| b78c8e0 | Igor Canadi | 13 August 2015, 17:49:10 UTC | Merge pull request #689 from msb-at-yahoo/add-tools-target Add a 'tools' target. | 13 August 2015, 17:49:10 UTC |
| 9f0dd22 | maurice barnum | 13 August 2015, 17:13:43 UTC | Add a 'tools' target. My use case is to build the rocksdb static library and tools, and ideally I'd like to not spend time building the shared library and other targets that I won't use. | 13 August 2015, 17:19:53 UTC |
| 4637207 | sdong | 06 August 2015, 20:23:02 UTC | Add test case to repro the mispositional iterator in a low-chance data race case Summary: Iterator has a bug: if a child iterator reaches its end, and user issues a Prev(), and just before SeekToLast() of the child iterator is called, some extra rows is added in the end, the position of iterator can be misplaced. Test Plan: Run the tests with or without valgrind Reviewers: rven, yhchiang, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: tnovak, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43671 | 12 August 2015, 17:50:52 UTC |
| 3bd9db4 | Islam AbdelRahman | 12 August 2015, 17:18:59 UTC | [Cleanup] Remove RandomRWFile Summary: RandomRWFile is not used anywhere in out code base, this patch remove RandomRWFile Test Plan: make check -j64 USE_CLANG=1 make all -j64 OPT=-DROCKSDB_LITE make release -j64 Reviewers: sdong, yhchiang, anthony, kradhakrishnan, rven, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44091 | 12 August 2015, 17:18:59 UTC |
| c3466ea | agiardullo | 31 July 2015, 22:18:27 UTC | Have Transactions use WriteBatch::RollbackToSavePoint Summary: Clean up transactions to use the new RollbackToSavePoint api in WriteBatchWithIndex. Note, this diff depends on Pessimistic Transactions diff and ManagedSnapshot diff (D40869 and D43293). Test Plan: unit tests Reviewers: rven, yhchiang, kradhakrishnan, spetrunia, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43371 | 12 August 2015, 00:53:30 UTC |
| 0db807e | agiardullo | 31 July 2015, 03:31:36 UTC | Transaction error statuses Summary: Based on feedback from spetrunia, we should better differentiate error statuses for transaction failures. https://github.com/MySQLOnRocksDB/mysql-5.6/issues/86#issuecomment-124605954 Test Plan: unit tests Reviewers: rven, kradhakrishnan, spetrunia, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43323 | 12 August 2015, 00:52:56 UTC |
| c2f2cb0 | agiardullo | 26 May 2015, 00:37:33 UTC | Pessimistic Transactions Summary: Initial implementation of Pessimistic Transactions. This diff contains the api changes discussed in D38913. This diff is pretty large, so let me know if people would prefer to meet up to discuss it. MyRocks folks: please take a look at the API in include/rocksdb/utilities/transaction[_db].h and let me know if you have any issues. Also, you'll notice a couple of TODOs in the implementation of RollbackToSavePoint(). After chatting with Siying, I'm going to send out a separate diff for an alternate implementation of this feature that implements the rollback inside of WriteBatch/WriteBatchWithIndex. We can then decide which route is preferable. Next, I'm planning on doing some perf testing and then integrating this diff into MongoRocks for further testing. Test Plan: Unit tests, db_bench parallel testing. Reviewers: igor, rven, sdong, yhchiang, yoshinorim Reviewed By: sdong Subscribers: hermanlee4, maykov, spetrunia, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D40869 | 12 August 2015, 00:52:23 UTC |
| c2868cb | Islam AbdelRahman | 11 August 2015, 21:47:14 UTC | Use manual_compaction for compaction_job_test Summary: Under certain conditions (disable compression) the compactions that are created in compaction_job_test will pass the trivial_move conditions This will cause problems since we assert that we dont run a compaction if it's a trivial move https://github.com/facebook/rocksdb/blob/master/db/compaction_job.cc#L144-L147 for example when we disable compression, compactions become a valid trivial move and the assert fails https://ci-builds.fb.com/view/rocksdb/job/rocksdb_no_compression/180/console Test Plan: compaction_job_test Reviewers: sdong, yhchiang, noetzli, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43983 | 11 August 2015, 21:47:14 UTC |
| 6b2d570 | agiardullo | 11 August 2015, 20:44:47 UTC | Fix Windows build by adding snapshot_impl to CMakeLists Test Plan: untested Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44049 | 11 August 2015, 20:49:05 UTC |
| e61fafb | Yueh-Hsuan Chiang | 11 August 2015, 20:30:49 UTC | Fixed clang-build error in util/thread_local.cc Summary: This patch fixes the following clang-build error in util/thread_local.cc by using a cleaner macro blocker: 12:26:31 util/thread_local.cc:157:19: error: declaration shadows a static data member of 'rocksdb::ThreadLocalPtr::StaticMeta' [-Werror,-Wshadow] 12:26:31 ThreadData* tls_ = 12:26:31 ^ 12:26:31 util/thread_local.cc:19:66: note: previous declaration is here 12:26:31 __thread ThreadLocalPtr::ThreadData* ThreadLocalPtr::StaticMeta::tls_ = nullptr; 12:26:31 ^ Test Plan: db_test Reviewers: sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D44043 | 11 August 2015, 20:30:49 UTC |
| cee1e8a | Islam AbdelRahman | 11 August 2015, 19:19:56 UTC | Parallelize LoadTableHandlers Summary: Add a new option that all LoadTableHandlers to use multiple threads to load files on DB Open and Recover Test Plan: make check -j64 COMPILE_WITH_TSAN=1 make check -j64 DISABLE_JEMALLOC=1 make all valgrind_check -j64 (still running) Reviewers: yhchiang, anthony, rven, kradhakrishnan, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43755 | 11 August 2015, 19:19:56 UTC |
| 4249f15 | Andres Notzli | 11 August 2015, 18:46:15 UTC | Removing duplicate code in db_bench/db_stress, fixing typos Summary: While working on single delete support for db_bench, I realized that db_bench/db_stress contain a bunch of duplicate code related to copmression and found some typos. This patch removes duplicate code, typos and a redundant #ifndef in internal_stats.cc. Test Plan: make db_stress && make db_bench && ./db_bench --benchmarks=compress,uncompress Reviewers: yhchiang, sdong, rven, anthony, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43965 | 11 August 2015, 18:46:15 UTC |
| a03085b | Igor Canadi | 11 August 2015, 18:36:12 UTC | Fix linters on non-fb machines Summary: Our linters assume that clang-format is installed at /mnt/vol/engshare/admin/scripts/clang-format and flint is installed at /home/engshare/tools/flint. This makes them fail on non-fb machines. This change will: * if clang-format is not on a specified path, it will try running generic clang-format. Linters will still fail if clang-format is not installed, but this shouldn't be a big issue, since it's pretty easy to install it. * flint will not be run if /home/engshare/tools/flint is not present Test Plan: Made a change on a mac machine. Ran `arc lint`. No failures observed. Reviewers: aekmekji, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D44031 | 11 August 2015, 18:36:12 UTC |
| 1ae2711 | Nathan Bronson | 11 August 2015, 18:08:02 UTC | reduce comparisons by skiplist Summary: Key comparison is the single largest CPU user for CPU-bound workloads. This diff reduces the number of comparisons in two ways. The first is that it moves predecessor array gathering from FindGreaterOrEqual to FindLessThan, so that FindGreaterOrEqual can return immediately if compare_ returns 0. As part of this change I moved the sequential insertion optimization into Insert, to remove the undocumented (and smelly) requirement that prev must be equal to prev_ if it is non-null. The second optimization is that all of the search functions skip calling compare_ when moving to a lower level that has the same Next pointer. With a branching factor of 4 we would expect this to happen 1/4 of the time. On a single-threaded CPU-bound workload (-benchmarks=fillrandom -threads=1 -batch_size=1 -memtablerep=skip_list -value_size=0 --num=1600000 -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -disable_auto_compactions --max_write_buffer_number=8 -max_background_flushes=8 --disable_wal --write_buffer_size=160000000) on my dev server this is good for a 7% perf win. Test Plan: unit tests Reviewers: rven, ljin, yhchiang, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43233 | 11 August 2015, 18:25:22 UTC |
| b47d65b | Yueh-Hsuan Chiang | 11 August 2015, 17:55:27 UTC | Fixed Segmentation Fault in db_stress on OSX. Summary: This patch provides a simplier solution to the memory leak issue identified in patch https://reviews.facebook.net/D43677, where a static function local variable can be used instead of using a global static unique_ptr. Test Plan: run db_stress on mac Reviewers: igor, sdong, anthony, IslamAbdelRahman, maykov Reviewed By: maykov Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43995 | 11 August 2015, 17:55:27 UTC |
| a1581ec | Igor Canadi | 10 August 2015, 20:58:55 UTC | Modernize RocksDB linters Summary: This was motivated by t7518166. checkCpp crashes on db_test.cc because the file is too big :( Couple of changes: * Added clang-format linter. Now we can catch all code that is not formatted correctly. * Added Howtoeven in our list of linters * Replaced cpplint with flint * Removed checkCpp lint. Nobody ownes it and it doesn't work on db_test.cc Test Plan: Made a random lint error and `arc lint`. Saw an error. Reviewers: yhchiang, kradhakrishnan, anthony, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D41949 | 10 August 2015, 20:58:55 UTC |
| a9dcc0a | Islam AbdelRahman | 10 August 2015, 18:30:36 UTC | Fix clang build Summary: https://ci-builds.fb.com/view/rocksdb/job/rocksdb_clang_build/893/console Fixing clang build Test Plan: make clean USE_CLANG=1 make all -j64 Reviewers: sdong, noetzli, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43959 | 10 August 2015, 18:30:36 UTC |
| 2cf0f4f | Yoshinori Matsunobu | 10 August 2015, 16:21:04 UTC | Adding wal_recovery_mode log message Summary: wal_recovery_mode setting was not written to LOG. This diff adds the log message Test Plan: manually checked Reviewers: kradhakrishnan, sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43953 | 10 August 2015, 16:43:30 UTC |
| 68f9343 | Andres Notzli | 08 August 2015, 04:59:51 UTC | Better CompactionJob testing Summary: Changed compaction_job_test to support better/more thorough tests and added two tests. Also changed MockFileContents to order using InternalKeyComparator. Test Plan: make compaction_job_test && ./compaction_job_test; make all && make check Reviewers: sdong, rven, igor, yhchiang, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42837 | 08 August 2015, 04:59:51 UTC |
| 22dcaaf | Islam AbdelRahman | 07 August 2015, 22:37:56 UTC | More accurate time measurement for delete_scheduler_test Summary: Start measuring time spent before BackgroundEmptyTrash starts Test Plan: delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43857 | 07 August 2015, 22:37:56 UTC |
| 0a7ea58 | sdong | 07 August 2015, 21:50:34 UTC | Add auto-build manifest for appveyor Summary: Check in a simple auto-build manfiest so that developers can issue ad-hoc build for Windows in Appveyor. Test Plan: Run it in test branch test_appveyor and it works well. Reviewers: kradhakrishnan, rven, anthony, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43839 | 07 August 2015, 22:37:46 UTC |
| 0093271 | Siying Dong | 07 August 2015, 18:50:45 UTC | Merge pull request #685 from flandr/fix-tls-build Fix OSX + Windows build | 07 August 2015, 18:50:45 UTC |
| ac04a6c | Nate Rosenblum | 07 August 2015, 17:47:05 UTC | Fix OSX + Windows build Commit 257ee89 added a static destruction helper to avoid notional "leaks" of TLS on main thread exit. This helper fails to compile on OS X (and presumably Windows, though I haven't checked), which lacks the __thread storage class StaticMeta::tls_ member. This patch fixes the builds. Do note that the static cleanup mechanism may be somewhat brittle and atexit(3) may be a more suitable approach to releasing the main thread's TLS if it's highly desirable for this memory to not be reported "reachable" by Valgrind at exit. | 07 August 2015, 17:47:05 UTC |
| 16ea1c7 | agiardullo | 07 August 2015, 00:59:05 UTC | simple ManagedSnapshot wrapper Summary: Implemented this simple wrapper for something else I was working on. Seemed like it makes sense to expose it instead of burying it in some random code. Test Plan: added test Reviewers: rven, kradhakrishnan, sdong, yhchiang Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43293 | 07 August 2015, 00:59:05 UTC |
| 257ee89 | Alexey Maykov | 05 August 2015, 22:45:21 UTC | Fixed memory leaks Summary: MyRocks valgrind run was showing memory leaks. The fixes are mostly self-explaining. There is only a single usage of ThreadLocalPtr. Potentially, we may think about replacing this use with thread_local, but it will be a bigger change. Another option to consider is using thread_local instead of __thread in ThreadLocalPtr implementation. This way, tls_ can be stored using std::unique_ptr and no destructor would be required. Test Plan: - make check - MyRocks valgrind run doesn't report leaks Reviewers: rven, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43677 | 06 August 2015, 22:39:12 UTC |
| 254c4fb | sdong | 06 August 2015, 18:11:36 UTC | In HISTORY.md Switch unreleased notes to 3.13 Summary: Prepare release notes for 3.13. Test Plan: Text only Reviewers: yhchiang, IslamAbdelRahman, rven, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43665 | 06 August 2015, 21:02:44 UTC |
| 40f893f | Islam AbdelRahman | 06 August 2015, 17:56:00 UTC | Fix delete_scheduler_test valgrind error Summary: Use shared_ptr instead of deleting in destructor Test Plan: DISABLE_JEMALLOC=1 make delete_scheduler_test -j64 && valgrind --error-exitcode=2 --leak-check=full ./delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43659 | 06 August 2015, 17:56:00 UTC |
| 6a4aaad | sdong | 06 August 2015, 03:51:27 UTC | Avoid type unique_ptr in LogWriterNumber::writer for Windows build break Summary: Visual Studio complains about deque<LogWriterNumber> because LogWriterNumber is non-copyable for its unique_ptr member writer. Move away from it, and do explit free. It is less safe but I can't think of a better way to unblock it. Test Plan: valgrind check test Reviewers: anthony, IslamAbdelRahman, kolmike, rven, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43647 | 06 August 2015, 17:52:41 UTC |
| d7314ba | Andres Noetzli | 06 August 2015, 17:43:28 UTC | Fixing endless loop if seeking to end of key with seq num 0 Summary: When seeking to the last occurrence of a key with sequence number 0, db_iter ends up in an endless loop because it seeks to type kValueTypeForSeek which is larger than kTypeDeletion/kTypeValue. Added test case that triggers the behavior. Test Plan: make clean all check Reviewers: igor, rven, anthony, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43653 | 06 August 2015, 17:43:28 UTC |
| 48e6e9a | sdong | 06 August 2015, 03:44:27 UTC | Add util/delete_scheduler_impl.cc to CMakeLists.txt Summary: util/delete_scheduler_impl.cc is missing CMakeLists.txt, which fails Windows build. Add it. Test Plan: Build in both of Windows and Linux Reviewers: anthony, rven, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43641 | 06 August 2015, 03:56:04 UTC |
| c774245 | sdong | 05 August 2015, 20:14:28 UTC | Add Statistics.getHistogramString() to print more detailed outputs of a histogram Summary: Provide a way for users to know more detailed ditribution of a histogram metrics. Example outputs: Manually add statement fprintf(stdout, "%s\n", dbstats->getHistogramString(SST_READ_MICROS).c_str()); Will print out something like: Count: 989151 Average: 1.7659 StdDev: 1.52 Min: 0.0000 Median: 1.2071 Max: 860.0000 Percentiles: P50: 1.21 P75: 1.70 P99: 5.12 P99.9: 13.67 P99.99: 21.70 ------------------------------------------------------ [ 0, 1 ) 390839 39.513% 39.513% ######## [ 1, 2 ) 500918 50.641% 90.154% ########## [ 2, 3 ) 79358 8.023% 98.177% ## [ 3, 4 ) 6297 0.637% 98.813% [ 4, 5 ) 1712 0.173% 98.986% [ 5, 6 ) 1134 0.115% 99.101% [ 6, 7 ) 1222 0.124% 99.224% [ 7, 8 ) 1529 0.155% 99.379% [ 8, 9 ) 1264 0.128% 99.507% [ 9, 10 ) 988 0.100% 99.607% [ 10, 12 ) 1378 0.139% 99.746% [ 12, 14 ) 1828 0.185% 99.931% [ 14, 16 ) 410 0.041% 99.972% [ 16, 18 ) 72 0.007% 99.980% [ 18, 20 ) 67 0.007% 99.986% [ 20, 25 ) 106 0.011% 99.997% [ 25, 30 ) 24 0.002% 99.999% [ 30, 35 ) 1 0.000% 100.000% [ 250, 300 ) 2 0.000% 100.000% [ 300, 350 ) 1 0.000% 100.000% [ 800, 900 ) 1 0.000% 100.000% Test Plan: Manually add a print in db_bench and make sure it prints out as expected. Will add some codes to cover the function Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43611 | 06 August 2015, 03:05:56 UTC |
| 29b028b | Islam AbdelRahman | 06 August 2015, 02:16:52 UTC | Make DeleteScheduler tests more reliable Summary: Update DeleteScheduler tests so that they verify the used penalties for waiting instead of measuring the time spent which is not reliable Test Plan: make -j64 delete_scheduler_test && ./delete_scheduler_test COMPILE_WITH_TSAN=1 make -j64 delete_scheduler_test && ./delete_scheduler_test COMPILE_WITH_ASAN=1 make -j64 delete_scheduler_test && ./delete_scheduler_test make -j64 db_test && ./db_test --gtest_filter="DBTest.RateLimitedDelete:DBTest.DeleteSchedulerMultipleDBPaths" COMPILE_WITH_TSAN=1 make -j64 db_test && ./db_test --gtest_filter="DBTest.RateLimitedDelete:DBTest.DeleteSchedulerMultipleDBPaths" COMPILE_WITH_ASAN=1 make -j64 db_test && ./db_test --gtest_filter="DBTest.RateLimitedDelete:DBTest.DeleteSchedulerMultipleDBPaths" Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43635 | 06 August 2015, 02:16:52 UTC |
| fca88f8 | sdong | 05 August 2015, 23:04:18 UTC | valgrind_check to exit on test failures Summary: Currently, valgrind_check doesn't fail on test failures, which creates confusion. valgrind_check should fail if test fails. Test Plan: Manually change tests to return test failure or cause memory leak and see valgrind_check has the correct behavior. Reviewers: anthony, yhchiang, IslamAbdelRahman, igor, kradhakrishnan, rven Reviewed By: rven Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43629 | 06 August 2015, 00:46:09 UTC |
| 7d364d0 | Poornima Chozhiyath Raman | 05 August 2015, 23:38:12 UTC | Fix build failure Summary: fix the build failure Test Plan: make all Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43623 | 05 August 2015, 23:38:12 UTC |
| 960d936 | Poornima Chozhiyath Raman | 05 August 2015, 23:16:46 UTC | Add function 'GetInfoLogList()' Summary: The list of info log files of a db can be obtained using the new function. Test Plan: New test in db_test.cc passed. Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: IslamAbdelRahman, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D41715 | 05 August 2015, 23:16:46 UTC |
| 7ccd1c8 | sdong | 05 August 2015, 18:56:19 UTC | Add two unit tests for SyncWAL() Summary: Add two unit tests for SyncWAL(). One makes sure SyncWAL() doesn't block writes in the other thread. Another one makes sure SyncWAL() doesn't wait ongoing writes to finish before being executed. Create a new test file db_wal_test and move two WAL related tests from db_test to here. Test Plan: Run the new tests Reviewers: IslamAbdelRahman, rven, kradhakrishnan, kolmike, tnovak, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43605 | 05 August 2015, 21:27:02 UTC |
| 3ae386e | sdong | 05 August 2015, 19:11:30 UTC | Add statistic histogram "rocksdb.sst.read.micros" Summary: Measure read latency histogram and put in statistics. Compaction inputs are excluded from it when possible (unfortunately usually no possible as we usually take table reader from table cache. Test Plan: Run db_bench and it shows the stats, like: rocksdb.sst.read.micros statistics Percentiles :=> 50 : 1.238522 95 : 2.529740 99 : 3.912180 Reviewers: kradhakrishnan, rven, anthony, IslamAbdelRahman, MarkCallaghan, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43275 | 05 August 2015, 20:02:33 UTC |
| 8ecb51a | sdong | 05 August 2015, 18:15:54 UTC | "make commit-prereq" should clean up rocksjava properly Summary: "make commit-prereq" fails to clean up java, which can cause rocksjava failure. Test Plan: Run commit-prepreq Reviewers: IslamAbdelRahman, rven, kradhakrishnan, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43575 | 05 August 2015, 19:47:24 UTC |
| 9aec75f | Islam AbdelRahman | 05 August 2015, 18:47:07 UTC | Enable DBTest.FlushSchedule under TSAN Summary: This patch will fix the false positive of DBTest.FlushSchedule under TSAN, we dont need to disable this test Test Plan: COMPILE_WITH_TSAN=1 make -j64 db_test && ./db_test --gtest_filter="DBTest.FlushSchedule" Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43599 | 05 August 2015, 18:47:07 UTC |
| bd2fc5f | Islam AbdelRahman | 05 August 2015, 18:45:31 UTC | Fix TSAN for delete_scheduler_test Summary: Fixing TSAN false positive and relaxing the conditions when we are running under TSAN Test Plan: COMPILE_WITH_TSAN=1 make -j64 delete_scheduler_test && ./delete_scheduler_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43593 | 05 August 2015, 18:45:31 UTC |
| 8e01bd1 | sdong | 04 August 2015, 23:50:40 UTC | Fix misplaced position for reversing iterator direction while current key is a merge Summary: While doing forward iterating, if current key is merge, internal iterator position is placed to the next key. If Prev() is called now, needs to do extra Prev() to recover the location. This is second attempt of fixing after reverting ec70fea4c4025351190eba7a02bd09bb5f083790. This time shrink the fix to only merge key is the current key and avoid the reseeking logic for max_iterating skipping Test Plan: enable the two disabled tests and make sure they pass Reviewers: rven, IslamAbdelRahman, kradhakrishnan, tnovak, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43557 | 05 August 2015, 18:08:50 UTC |
| c465071 | Andres Notzli | 05 August 2015, 14:33:27 UTC | Removing duplicate code Summary: While working on https://reviews.facebook.net/D43179 , I found duplicate code in the tests. This patch removes it. Test Plan: make clean all check Reviewers: igor, sdong, rven, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43263 | 05 August 2015, 14:33:27 UTC |
| e06cf1a | Mike Kolupaev | 05 August 2015, 13:06:39 UTC | [wal changes 3/3] method in DB to sync WAL without blocking writers Summary: Subj. We really need this feature. Previous diff D40899 has most of the changes to make this possible, this diff just adds the method. Test Plan: `make check`, the new test fails without this diff; ran with ASAN, TSAN and valgrind. Reviewers: igor, rven, IslamAbdelRahman, anthony, kradhakrishnan, tnovak, yhchiang, sdong Reviewed By: sdong Subscribers: MarkCallaghan, maykov, hermanlee4, yoshinorim, tnovak, dhruba Differential Revision: https://reviews.facebook.net/D40905 | 05 August 2015, 13:06:39 UTC |
| 5dc3e68 | Ari Ekmekji | 05 August 2015, 05:19:07 UTC | Update Tests To Enable Subcompactions Summary: Updated DBTest DBCompactionTest and CompactionJobStatsTest to run compaction-related tests once with subcompactions enabled and once disabled using the TEST_P test type in the Google Test suite. Test Plan: ./db_test ./db_compaction-test ./compaction_job_stats_test Reviewers: sdong, igor, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43443 | 05 August 2015, 05:19:07 UTC |
| c45a57b | Islam AbdelRahman | 05 August 2015, 03:45:27 UTC | Support delete rate limiting Summary: Introduce DeleteScheduler that allow enforcing a rate limit on file deletion Instead of deleting files immediately, files are moved to trash directory and deleted in a background thread that apply sleep penalty between deletes if needed. I have updated PurgeObsoleteFiles and PurgeObsoleteWALFiles to use the delete_scheduler instead of env_->DeleteFile Test Plan: added delete_scheduler_test existing unit tests Reviewers: kradhakrishnan, anthony, rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43221 | 05 August 2015, 03:45:27 UTC |
| 102ac11 | Yueh-Hsuan Chiang | 05 August 2015, 01:12:58 UTC | Update JAVA-HISTORY.md for v3.13 Summary: Update JAVA-HISTORY.md for v3.13 Test Plan: no code change. Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43539 | 05 August 2015, 01:12:58 UTC |
| 3a1d4e6 | Yueh-Hsuan Chiang | 05 August 2015, 00:33:35 UTC | Merge pull request #670 from skunkwerks/fix_osx_shared_library_names Fix shared library names on OSX | 05 August 2015, 00:33:35 UTC |
| f5d072a | Yueh-Hsuan Chiang | 05 August 2015, 00:01:08 UTC | Fixed RocksJava test failure of shouldSetTestCappedPrefixExtractor Summary: Fixed RocksJava test failure of shouldSetTestCappedPrefixExtractor by adding the missing native implementation of useCappedPrefixExtractor. Test Plan: make jclean make rocksdbjava -j32 make jtest Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43551 | 05 August 2015, 00:01:08 UTC |
| f39cbcb | Yueh-Hsuan Chiang | 04 August 2015, 23:54:14 UTC | Merge pull request #654 from adamretter/remove-emptyvalue-compactionfilter RemoveEmptyValueCompactionFilter | 04 August 2015, 23:54:14 UTC |
| f0b5bcc | ashishn | 29 July 2015, 05:15:15 UTC | add support for capped prefix extractor in java | 04 August 2015, 23:46:23 UTC |
| 18ba58a | Yoshinori Matsunobu | 04 August 2015, 15:05:06 UTC | Upgrading jemalloc from 3.6.0 to the latest for fbcode+gcc 4.8.1 Summary: MyRocks is using jemalloc latest version, not 3.6.0. Combining multiple versions (3.6.0 in RocksDB and latest in MyRocks) broke some features -- for example, getting SIGSEGV when heap profiling was enabled. This diff switches to use jemalloc latest, if env variable ROCKSDB_FBCODE_BUILD_WITH_481=1 was set. My understanding is this env was used by MyRocks only so it would be safe to change. Test Plan: building MyRocks then verified jemalloc heap profiling worked Reviewers: igor, rven, yhchiang, jtolmer, maykov, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43479 | 04 August 2015, 23:35:26 UTC |
| ce21afd | Yueh-Hsuan Chiang | 04 August 2015, 23:07:17 UTC | Expose the BackupEngine from the Java API Summary: Merge pull request #665 by adamretter Exposes BackupEngine from C++ to the Java API. Previously only BackupableDB was available Test Plan: BackupEngineTest.java Reviewers: fyrz, igor, ankgup87, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42873 | 04 August 2015, 23:08:54 UTC |
| b0d12a1 | Yueh-Hsuan Chiang | 04 August 2015, 22:51:45 UTC | Merge pull request #569 from adamretter/travis-java-api Another attempt at adding the Java API and tests to the travis build | 04 August 2015, 22:51:45 UTC |
| 241bb2a | Yueh-Hsuan Chiang | 04 August 2015, 22:47:05 UTC | Make DBCompactionTest.SkipStatsUpdateTest more stable. Summary: Make DBCompactionTest.SkipStatsUpdateTest more stable by removing flaky but unnecessary assertion on the size of db as simply checking the random file open count is suffice. Test Plan: db_compaction_test Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43533 | 04 August 2015, 22:47:05 UTC |
| 3424eeb | Yueh-Hsuan Chiang | 04 August 2015, 22:04:19 UTC | Polish HISTORY.md Summary: Polish HISTORY.md Test Plan: no code change. Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43527 | 04 August 2015, 22:04:19 UTC |
| cf3e053 | sdong | 04 August 2015, 19:20:38 UTC | crash_test cleans up directory before testing if TEST_TMPDIR is set Summary: In a recent change, crash_test can put data under TEST_TMPDIR. However, the directory is not cleaned before running the test, which may cause unexpected results. Clean it. Test Plan: Run white and black box crash test against non-existing, or non-empty but not compactible DBs, and make sure it works as expected. Reviewers: kradhakrishnan, rven, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43515 | 04 August 2015, 21:59:28 UTC |
| 24daff6 | Yueh-Hsuan Chiang | 04 August 2015, 20:51:05 UTC | Fix a typo and update HISTORY.md for NewCompactOnDeletionCollectorFactory(). Summary: Fix a typo and update HISTORY.md for NewCompactOnDeletionCollectorFactory(). Test Plan: no code change. Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43521 | 04 August 2015, 20:51:05 UTC |
| 14d0bfa | Yueh-Hsuan Chiang | 04 August 2015, 20:48:16 UTC | Add DBOptions::skip_sats_update_on_db_open Summary: UpdateAccumulatedStats() is used to optimize compaction decision esp. when the number of deletion entries are high, but this function can slowdown DBOpen esp. in disk environment. This patch adds DBOptions::skip_sats_update_on_db_open, which skips UpdateAccumulatedStats() in DB::Open() time when it's set to true. Test Plan: Add DBCompactionTest.SkipStatsUpdateTest Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: tnovak, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42843 | 04 August 2015, 20:48:16 UTC |
| e2a3bfe | sdong | 04 August 2015, 01:11:14 UTC | First half of whitebox_crash_test to keep crashing the same DB Summary: Currently, whitebox crash test is not really executed, because the DB is destroyed after each crash. With this fix, in the first half of the time, DB will keep opening the crashed DB and continue from there. Test Plan: "make whitebox_crash_test" and see the same DB keeps crashing and being reopened. Reviewers: IslamAbdelRahman, yhchiang, rven, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43503 | 04 August 2015, 19:16:44 UTC |
| 2e73bd4 | sdong | 04 August 2015, 18:35:44 UTC | crash_test to put DB under TEST_TMPDIR Summary: Currently crash_test only puts data under /tmp. It is less flexible if we want to cover different file systems or media. Make crash_test to appreciate TEST_TMPDIR so that users can run it against another file system. Test Plan: Run blackbox_crash_test and whitebox_crash_test with or without TEST_TMPDIR set and make sure DBs are put in the right place Reviewers: kradhakrishnan, yhchiang, rven, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43509 | 04 August 2015, 19:10:55 UTC |
| 1205bdb | sdong | 04 August 2015, 00:23:34 UTC | crash_test to cover simply cases Summary: crash_test now only runs complicated options, multiple column families, prefix hash, frequently changing options, many compaction threads, etc. These options are good to cover new features but we loss coverage in most common use cases. Furthermore, by running only for multiple column families, we are not able to create LSM trees that are large enough to cover some stress cases. Make half of crash_test runs the simply tests: single column family, default mem table, one compaction thread, no change options. Test Plan: Run crash_test Reviewers: rven, yhchiang, IslamAbdelRahman, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43461 | 04 August 2015, 19:08:38 UTC |
| d5c0a6d | Boyang Zhang | 04 August 2015, 17:56:49 UTC | Merge branch 'master' of github.com:facebook/rocksdb Fixed memory leak error | 04 August 2015, 17:56:49 UTC |
| 2d41403 | Boyang Zhang | 04 August 2015, 15:42:34 UTC | Made change to fix the memory leak Summary: So I took a look and I used a pointer to TableBuilder. Changed it to a unique_ptr. I think this should work, but I cannot run valgrind correctly on my local machine to test it. Test Plan: Run valgrind, but it's not working locally. It says I'm executing an unrecognized instruction. Reviewers: yhchiang Subscribers: dhruba, sdong Differential Revision: https://reviews.facebook.net/D43485 | 04 August 2015, 17:55:42 UTC |
| 92f7039 | sdong | 04 August 2015, 17:36:08 UTC | fix memory corruption issue in sst_dump --show_compression_sizes Summary: In "sst_dump --show_compression_sizes", a reference of CompressionOptions is kept in TableBuilderOptions, which is destroyed later, causing a memory issue. Test Plan: Run valgrind against SSTDumpToolTest.CompressedSizes and make sure it is fixed Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, rven Reviewed By: rven Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43497 | 04 August 2015, 17:48:44 UTC |
| be8621f | Yueh-Hsuan Chiang | 04 August 2015, 03:51:34 UTC | Fix compile warning in compact_on_deletion_collector in some environment Summary: Fix compile warning in compact_on_deletion_collector some environment Test Plan: make Reviewers: igor, sdong, anthony, IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43467 | 04 August 2015, 03:53:15 UTC |
| 2689430 | Yueh-Hsuan Chiang | 04 August 2015, 03:42:55 UTC | Add CompactOnDeletionCollector in utilities/table_properties_collectors. Summary: This diff adds CompactOnDeletionCollector in utilities/table_properties_collectors, which applies a sliding window to a sst file and mark this file as need-compaction when it observe enough deletion entries within the consecutive keys covered by the sliding window. Test Plan: compact_on_deletion_collector_test Reviewers: igor, anthony, IslamAbdelRahman, kradhakrishnan, yoshinorim, sdong Reviewed By: sdong Subscribers: maykov, dhruba Differential Revision: https://reviews.facebook.net/D41175 | 04 August 2015, 03:42:55 UTC |
| 20b244f | Venkatesh Radhakrishnan | 03 August 2015, 22:53:22 UTC | Fix CompactFiles by adding all necessary files Summary: The compact files API had a bug where some overlapping files are not added. These are files which overlap with files which were added to the compaction input files, but not to the original set of input files. This happens only when there are more than two levels involved in the compaction. An example will illustrate this better. Level 2 has 1 input file 1.sst which spans [20,30]. Level 3 has added file 2.sst which spans [10,25] Level 4 has file 3.sst which spans [35,40] and input file 4.sst which spans [46,50]. The existing code would not add 3.sst to the set of input_files because it only becomes an overlapping file in level 4 and it wasn't one in level 3. When installing the results of the compaction, 3.sst would overlap with output file from the compact files and result in the assertion in version_set.cc:1130 // Must not overlap assert(level <= 0 || level_files->empty() || internal_comparator_->Compare( (*level_files)[level_files->size() - 1]->largest, f->smallest) < 0); This change now adds overlapping files from the current level to the set of input files also so that we don't hit the assertion above. Test Plan: d=/tmp/j; rm -rf $d; seq 1000 | parallel --gnu --eta 'd=/tmp/j/d-{}; mkdir -p $d; TEST_TMPDIR=$d ./db_compaction_test --gtest_filter=*CompactilesOnLevel* --gtest_also_run_disabled_tests >& '$d'/log-{}' Reviewers: igor, yhchiang, sdong Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43437 | 03 August 2015, 22:53:22 UTC |
| 87df629 | Venkatesh Radhakrishnan | 03 August 2015, 22:52:52 UTC | Make SuggestCompactRangeNoTwoLevel0Compactions deterministic Summary: Made SuggestCompactRangeNoTwoLevel0Compactions by forcing a flush after generating a file and waiting for compaction at the end. Test Plan: Run SuggestCompactRangeNoTwoLevel0Compactions Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43449 | 03 August 2015, 22:52:52 UTC |
| 40c6443 | Ari Ekmekji | 03 August 2015, 18:32:14 UTC | Parallelize L0-L1 Compaction: Restructure Compaction Job Summary: As of now compactions involving files from Level 0 and Level 1 are single threaded because the files in L0, although sorted, are not range partitioned like the other levels. This means that during L0-L1 compaction each file from L1 needs to be merged with potentially all the files from L0. This attempt to parallelize the L0-L1 compaction assigns a thread and a corresponding iterator to each L1 file that then considers only the key range found in that L1 file and only the L0 files that have those keys (and only the specific portion of those L0 files in which those keys are found). In this way the overlap is minimized and potentially eliminated between different iterators focusing on the same files. The first step is to restructure the compaction logic to break L0-L1 compactions into multiple, smaller, sequential compactions. Eventually each of these smaller jobs will be run simultaneously. Areas to pay extra attention to are # Correct aggregation of compaction job statistics across multiple threads # Proper opening/closing of output files (make sure each thread's is unique) # Keys that span multiple L1 files # Skewed distributions of keys within L0 files Test Plan: Make and run db_test (newer version has separate compaction tests) and compaction_job_stats_test Reviewers: igor, noetzli, anthony, sdong, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42699 | 03 August 2015, 18:32:14 UTC |
| 47316c2 | sdong | 03 August 2015, 18:01:24 UTC | dump_manifest supports DB with more number of levels Summary: Now ldb dump_manifest refuses to work if there are 20 levels. Extend the limit to 64. Test Plan: Run the tool with 20 number of levels Reviewers: kradhakrishnan, anthony, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42879 | 03 August 2015, 18:02:09 UTC |
| bd852bf | Andres Noetzli | 31 July 2015, 21:11:43 UTC | Fixed typos in db_stress Summary: Fixed typos. Test Plan: None Reviewers: igor, yhchiang, sdong, anthony, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43365 | 31 July 2015, 21:11:43 UTC |
| 544be63 | Andres Noetzli | 31 July 2015, 00:46:47 UTC | Fixing fprintf of non string literal Summary: sst_dump_tool contains two instances of `fprintf`s where the `format` argument is not a string literal. This prevents the code from compiling with some compilers/compiler options because of the potential security risks associated with printing non-literals. Test Plan: make all Reviewers: rven, igor, yhchiang, sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43305 | 31 July 2015, 00:46:47 UTC |
| 193dc97 | Andres Notzli | 30 July 2015, 23:59:03 UTC | Fixing dead code in table_properties_collector_test Summary: There was a bug in table_properties_collector_test that this patch is fixing: `!backward_mode && !test_int_tbl_prop_collector` in TestCustomizedTablePropertiesCollector was never true, so the code in the if-block never got executed. The reason is that the CustomizedTablePropertiesCollector test was skipping tests with `!backward_mode_ && !encode_as_internal`. The reason for skipping the tests is unknown. Test Plan: make table_properties_collector_test && ./table_properties_collector_test Reviewers: rven, igor, yhchiang, anthony, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D43281 | 30 July 2015, 23:59:03 UTC |
| 05d4265 | Boyang Zhang | 30 July 2015, 00:48:52 UTC | Merge branch 'master' of github.com:facebook/rocksdb | 30 July 2015, 00:48:52 UTC |
| 4be6d44 | Boyang Zhang | 24 July 2015, 00:05:33 UTC | Compression sizes option for sst_dump_tool Summary: Added a new feature to sst_dump_tool.cc to allow a user to see the sizes of the different compression algorithms on an .sst file. Usage: ./sst_dump --file=<filename> --show_compression_sizes ./sst_dump --file=<filename> --show_compression_sizes --set_block_size=<block_size> Note: If you do not set a block size, it will default to 16kb Test Plan: manual test and the write a unit test Reviewers: IslamAbdelRahman, anthony, yhchiang, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D42963 | 30 July 2015, 00:42:13 UTC |
| 8161bdb | agiardullo | 11 July 2015, 03:15:45 UTC | WriteBatch Save Points Summary: Support RollbackToSavePoint() in WriteBatch and WriteBatchWithIndex. Support for partial transaction rollback is needed for MyRocks. An alternate implementation of Transaction::RollbackToSavePoint() exists in D40869. However, the other implementation is messier because it is implemented outside of WriteBatch. This implementation is much cleaner and also exposes a potentially useful feature to WriteBatch. Test Plan: Added unit tests Reviewers: IslamAbdelRahman, kradhakrishnan, maykov, yoshinorim, hermanlee4, spetrunia, sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42723 | 29 July 2015, 23:54:23 UTC |
| 7bfae3a | sdong | 29 July 2015, 01:31:58 UTC | tools/db_crashtest2.py should run on the same DB Summary: Crash tests are supposed to restart the same DB after crashing, but it is now opening a different DB. Fix it. It's probably a leftover of https://reviews.facebook.net/D17073 Test Plan: Run the test and make sure the same Db is opened. Reviewers: kradhakrishnan, rven, igor, IslamAbdelRahman, yhchiang, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D43197 | 29 July 2015, 22:50:37 UTC |
| d06c82e | Andres Notzli | 29 July 2015, 02:21:55 UTC | Further cleanup of CompactionJob and MergeHelper Summary: Simplified logic in CompactionJob and removed unused parameter in MergeHelper. Test Plan: make && make check Reviewers: rven, igor, sdong, yhchiang Reviewed By: sdong Subscribers: aekmekji, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42687 | 29 July 2015, 02:21:55 UTC |
| e95c59c | Andres Notzli | 28 July 2015, 23:41:40 UTC | Count number of corrupt keys during compaction Summary: For task #7771355, we would like to log the number of corrupt keys during a compaction. This patch implements and tests the count as part of CompactionJobStats. Test Plan: make && make check Reviewers: rven, igor, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D42921 | 28 July 2015, 23:41:40 UTC |
| 221a94a | Adam Retter | 07 April 2015, 11:30:45 UTC | Another attempt at adding the Java API and tests to the travis build Summary: Adds the Java build and tests to Travis Test Plan: Make sure that Travis still runs (does currently) Reviewers: igor, fyrz, sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D43173 | 28 July 2015, 10:52:11 UTC |
| 1bdfcef | Poornima Chozhiyath Raman | 27 July 2015, 21:25:57 UTC | Fix when output level is 0 of universal compaction with trivial move Summary: Fix for universal compaction with trivial move, when the ouput level is 0. The tests where failing. Fixed by allowing normal compaction when output level is 0. Test Plan: modified test cases run successfully. Reviewers: sdong, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: anthony, kradhakrishnan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D42933 | 27 July 2015, 21:25:57 UTC |
