| 99d3eed | Igor Canadi | 07 June 2014, 00:26:23 UTC | Write Fast-path for single column family Summary: We have a perf regression of Write() even with one column family. Make fast path for single column family to avoid the perf regression. See task #4455480 Test Plan: make check Reviewers: sdong, ljin Reviewed By: sdong, ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18963 | 07 June 2014, 00:26:23 UTC |
| e72b02e | Yueh-Hsuan Chiang | 06 June 2014, 00:09:25 UTC | [Java] Add basic Java binding for rocksdb::Env. Summary: Add basic Java binding for rocksdb::Env. Test Plan: make rocksdbjava make jtest cd java ./jdb_bench.sh --max_background_compactions=1 ./jdb_bench.sh --max_background_compactions=10 Reviewers: sdong, ankgup87, haobo Reviewed By: haobo Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18903 | 06 June 2014, 00:09:25 UTC |
| b92a19a | sdong | 05 June 2014, 22:12:32 UTC | sst_dump: Set dummy prefix extractor for binary search index in block based table Summary: Now sst_dump fails in block based tables if binary search index is used, as it requires a prefix extractor. Add it. Test Plan: Run it against such a file to make sure it fixes the problem. Reviewers: yhchiang, kailiu Reviewed By: kailiu Subscribers: ljin, igor, dhruba, haobo, leveldb Differential Revision: https://reviews.facebook.net/D18927 | 05 June 2014, 22:37:23 UTC |
| 5d87071 | Igor Canadi | 05 June 2014, 20:19:35 UTC | Correctly preallocate files in universal compaction Summary: In universal compaction, MaxFileSizeForLevel is ULLONG_MAX. We've been preallocation files to UULONG_MAX size all these time :) Test Plan: make check Reviewers: dhruba, haobo, ljin, sdong, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18915 | 05 June 2014, 20:19:35 UTC |
| 166cc5b | Yueh-Hsuan Chiang | 05 June 2014, 16:24:55 UTC | Merge pull request #157 from ankgup87/master [Java] Add RestoreBackupableDB | 05 June 2014, 16:24:55 UTC |
| 1a6c1a5 | Ankit Gupta | 05 June 2014, 12:34:38 UTC | Fix build | 05 June 2014, 12:34:38 UTC |
| 2fa0a99 | Ankit Gupta | 05 June 2014, 12:25:08 UTC | Fix build | 05 June 2014, 12:25:08 UTC |
| 0831432 | Ankit Gupta | 05 June 2014, 12:17:53 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 05 June 2014, 12:17:53 UTC |
| 457bae6 | Igor Canadi | 04 June 2014, 16:59:44 UTC | Fix regression test Summary: https://github.com/facebook/rocksdb/commit/388d2054c7077d58ea56e08092382d904bec71c7 added extra line to db_bench output, breaking regression tests. This diff makes it more robust and fixes the issue Test Plan: ran it Reviewers: ljin, sdong Reviewed By: sdong Subscribers: sdong, leveldb Differential Revision: https://reviews.facebook.net/D18897 | 04 June 2014, 16:59:44 UTC |
| 552c49f | Igor Canadi | 03 June 2014, 20:58:44 UTC | Remove upper bound for rate limiting unit test | 03 June 2014, 20:58:44 UTC |
| fd27001 | Igor Canadi | 03 June 2014, 19:28:58 UTC | Fix compile errors on Mac Summary: https://phabricator.fb.com/P11372644 Test Plan: compiles Reviewers: sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18873 | 03 June 2014, 19:28:58 UTC |
| df9069d | sdong | 02 June 2014, 23:38:00 UTC | In DB::NewIterator(), try to allocate the whole iterator tree in an arena Summary: In this patch, try to allocate the whole iterator tree starting from DBIter from an arena 1. ArenaWrappedDBIter is created when serves as the entry point of an iterator tree, with an arena in it. 2. Add an option to create iterator from arena for following iterators: DBIter, MergingIterator, MemtableIterator, all mem table's iterators, all table reader's iterators and two level iterator. 3. MergeIteratorBuilder is created to incrementally build the tree of internal iterators. It is passed to mem table list and version set and add iterators to it. Limitations: (1) Only DB::NewIterator() without tailing uses the arena. Other cases, including readonly DB and compactions are still from malloc (2) Two level iterator itself is allocated in arena, but not iterators inside it. Test Plan: make all check Reviewers: ljin, haobo Reviewed By: haobo Subscribers: leveldb, dhruba, yhchiang, igor Differential Revision: https://reviews.facebook.net/D18513 | 03 June 2014, 00:44:57 UTC |
| 4627966 | sdong | 02 June 2014, 23:52:29 UTC | dynamic_bloom: replace some divide (remainder) operations with shifts in locality mode, and other improvements Summary: This patch changes meaning of options.bloom_locality: 0 means disable cache line optimization and any positive number means use CACHE_LINE_SIZE as block size (the previous behavior is the block size will be CACHE_LINE_SIZE*options.bloom_locality). By doing it, the divide operations inside a block can be replaced by a shift. Performance is improved: https://reviews.facebook.net/P471 Also, improve the basic algorithm in two ways: (1) make sure num of blocks is an odd number (2) rotate bytes after every probe in locality mode. Since the divider is 2^n, unless doing it, we are never able to use all the bits. Improvements of false positive: https://reviews.facebook.net/P459 Test Plan: make all check Reviewers: ljin, haobo Reviewed By: haobo Subscribers: dhruba, yhchiang, igor, leveldb Differential Revision: https://reviews.facebook.net/D18843 | 03 June 2014, 00:36:38 UTC |
| 91ddd58 | Igor Canadi | 03 June 2014, 00:23:55 UTC | Only signal cond variable if need to Summary: At the end of BackgroundCallCompaction(), we call SignalAll(), even though we don't need to. If compaction hasn't done anything and there's another compaction running, there is no need to signal on the condition variable. Doing so creates a tight feedback loop which results in log files like: wait for memtable flush compaction nothing to do wait for memtable flush compaction nothing to do This change eliminates that Test Plan: make check Also: icanadi@dev1440 ~ $ grep "nothing to do" /fast-rocksdb-tmp/rocksdb_test/column_family_test/LOG | wc -l 7435 icanadi@dev1440 ~ $ grep "nothing to do" /fast-rocksdb-tmp/rocksdb_test/column_family_test/LOG | wc -l 372 First version is before the change, second version is after the change. Reviewers: dhruba, ljin, haobo, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18855 | 03 June 2014, 00:23:55 UTC |
| 8cb7ad8 | Igor Canadi | 02 June 2014, 22:33:54 UTC | Flush stale column families less aggressively Summary: We've seen some production issues where column family is detected as stale, although there is only one column family in the system. This is a quick fix that: 1) doesn't flush stale column families if there's only one of them 2) Use 4 as a coefficient instead of 2 for determening when a column family is stale. This will make flushing less aggressive, while still keep a nice dynamic flushing of very stale CFs. Test Plan: make check Reviewers: dhruba, haobo, ljin, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18861 | 02 June 2014, 22:33:54 UTC |
| 593bb2c | sdong | 02 June 2014, 17:12:41 UTC | db_stress to add an option to periodically change background thread pool size. Summary: Add an option to indicates the variation of background threads. If the flag is not 0, for every 100 milliseconds, adjust thread pool size to a value within the range. Test Plan: run db_stress Reviewers: haobo, dhruba, igor, ljin Reviewed By: ljin Subscribers: leveldb, nkg- Differential Revision: https://reviews.facebook.net/D18759 | 02 June 2014, 17:32:09 UTC |
| 388d205 | Lei Jin | 30 May 2014, 21:31:55 UTC | forward iterator Summary: Forward iterator puts everything together in a flat structure instead of a hierarchy of nested iterators. this should simplify the code and provide better performance. It also enables more optimization since all information are accessiable in one place. Init evaluation shows about 6% improvement Test Plan: db_test and db_bench Reviewers: dhruba, igor, tnovak, sdong, haobo Reviewed By: haobo Subscribers: sdong, leveldb Differential Revision: https://reviews.facebook.net/D18795 | 30 May 2014, 21:31:55 UTC |
| f29c62f | Lei Jin | 30 May 2014, 21:09:22 UTC | add an iterator refresh option for SeekRandom Summary: One more option to allow iterator refreshing when using normal iterator Test Plan: ran db_bench Reviewers: haobo, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18849 | 30 May 2014, 21:09:22 UTC |
| fa1b62c | Ankit Gupta | 30 May 2014, 01:33:11 UTC | Merge remote-tracking branch 'upstream/master' | 30 May 2014, 01:33:11 UTC |
| 9899b12 | sdong | 29 May 2014, 17:57:22 UTC | ThreadID printed when Thread terminating in the same format as posix_logger Summary: https://github.com/facebook/rocksdb/commit/220132b65ec17abb037d3e79d5abf6ca8d797b96 correctly fixed the issue of thread ID printing when terminating a thread. Nothing wrong with it. This diff prints the ID in the same way as in PosixLogger::logv() so that users can be more easily to correlates them. Test Plan: run env_test and make sure it prints correctly. Reviewers: igor, haobo, ljin, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18819 | 29 May 2014, 18:11:08 UTC |
| ab3e566 | Yueh-Hsuan Chiang | 29 May 2014, 01:16:29 UTC | [Java] Generalize dis-own native handle and refine dispose framework. Summary: 1. Move disOwnNativeHandle() function from RocksDB to RocksObject to allow other RocksObject to use disOwnNativeHandle() when its ownership of native handle has been transferred. 2. RocksObject now has an abstract implementation of dispose(), which does the following two things. First, it checks whether both isOwningNativeHandle() and isInitialized() return true. If so, it will call the protected abstract function dispose0(), which all the subclasses of RocksObject should implement. Second, it sets nativeHandle_ = 0. This redesign ensure all subclasses of RocksObject have the same dispose behavior. 3. All subclasses of RocksObject now should implement dispose0() instead of dispose(), and dispose0() will be called only when isInitialized() returns true. Test Plan: make rocksdbjava make jtest Reviewers: dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett, haobo Reviewed By: haobo Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18801 | 29 May 2014, 01:16:29 UTC |
| 663627a | Yueh-Hsuan Chiang | 27 May 2014, 21:07:22 UTC | Merge pull request #153 from bpot/fix_jni_segfault Segfault when using BackupableDB with Java bindings | 27 May 2014, 21:07:22 UTC |
| 8e41e54 | Yueh-Hsuan Chiang | 27 May 2014, 20:52:44 UTC | [Java] Makes DbBenchmark takes 0 and 1 as boolean values. Summary: Originally, the boolean arguments in Java DB benchmark only takes true and false as valid input. This diff allows boolean arguments to be set using 0 or 1. Test Plan: make rocksdbjava make jdb_bench java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=1 java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=0 java/jdb_bench.sh --db=/tmp/path-not-exist --use_existing_db=1 Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett Reviewed By: haobo Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18687 | 27 May 2014, 20:52:44 UTC |
| 84cbafa | Ankit Gupta | 27 May 2014, 20:04:16 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 27 May 2014, 20:04:16 UTC |
| 05dd018 | Bob Potter | 27 May 2014, 18:54:02 UTC | BackupableDB: don't keep a reference to the RocksDB object now that we have disOwnNativeObject | 27 May 2014, 18:54:02 UTC |
| f068d2a | Igor Canadi | 23 May 2014, 17:27:56 UTC | Move master version to 3.2 | 23 May 2014, 17:27:56 UTC |
| 3a1cf12 | Igor Canadi | 22 May 2014, 17:24:24 UTC | Run FIFO compaction as part of db_crashtest2.py Summary: As title Test Plan: ran it Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D18783 | 22 May 2014, 17:24:24 UTC |
| 5fe176d | Bob Potter | 20 May 2014, 23:52:27 UTC | Java Bindings: prevent segfault from double delete Give BackupableDB sole responsibility of the native object to prevent RocksDB from also trying to delete it. | 21 May 2014, 23:47:18 UTC |
| 1fd4654 | Igor Canadi | 21 May 2014, 22:25:05 UTC | Update HISTORY.md | 21 May 2014, 22:25:05 UTC |
| 6de6a06 | Igor Canadi | 21 May 2014, 18:43:35 UTC | FIFO compaction style Summary: Introducing new compaction style -- FIFO. FIFO compaction style has write amplification of 1 (+1 for WAL) and it deletes the oldest files when the total DB size exceeds pre-configured values. FIFO compaction style is suited for storing high-frequency event logs. Test Plan: Added a unit test Reviewers: dhruba, haobo, sdong Reviewed By: dhruba Subscribers: alberts, leveldb Differential Revision: https://reviews.facebook.net/D18765 | 21 May 2014, 18:43:35 UTC |
| 220132b | Igor Canadi | 21 May 2014, 17:26:09 UTC | Merge pull request #156 from Chilledheart/print_pthread_info Print pthread_t in a more safe way | 21 May 2014, 17:26:09 UTC |
| 81b498b | Chilledheart | 21 May 2014, 17:24:42 UTC | Print pthread_t in a more safe way | 21 May 2014, 17:24:42 UTC |
| 62d2b92 | Igor Canadi | 21 May 2014, 16:25:43 UTC | Merge pull request #154 from mrsqueeze/master hdfs cleanup and compile test against CDH 4.4. | 21 May 2014, 16:25:43 UTC |
| 260842a | Ankit Gupta | 21 May 2014, 14:59:50 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 21 May 2014, 14:59:50 UTC |
| d271cc5 | Ankit Gupta | 21 May 2014, 14:54:22 UTC | Treat negative values as no limit | 21 May 2014, 14:54:22 UTC |
| 591f712 | Mike Orr | 21 May 2014, 11:54:22 UTC | cleanup exception text | 21 May 2014, 11:54:22 UTC |
| d788bb8 | Mike Orr | 21 May 2014, 11:50:37 UTC | - hdfs cleanup; fix to NewDirectory to comply with definition in env.h - fix compile error with env_test; static casts added | 21 May 2014, 11:50:37 UTC |
| f725e4f | Igor Canadi | 21 May 2014, 00:09:38 UTC | Make RateLimiting unit test less flakey | 21 May 2014, 00:09:38 UTC |
| b2cf95f | Igor Canadi | 20 May 2014, 21:28:51 UTC | Call EnableFileDeletions with false as argument | 20 May 2014, 21:28:51 UTC |
| c2fda55 | Mike Orr | 20 May 2014, 21:22:12 UTC | hdfs cleanup and compile test against CDH 4.4. | 20 May 2014, 21:22:12 UTC |
| bd1105a | sdong | 19 May 2014, 21:25:11 UTC | Print out thread ID while thread terminates for decreased pool size. Summary: Per request from @nkg-, temporarily print thread ID when a thread terminates. It is a temp solution as we try to minimized stderr messages. Test Plan: env_test Reviewers: haobo, igor, dhruba Reviewed By: igor CC: nkg-, leveldb Differential Revision: https://reviews.facebook.net/D18753 | 19 May 2014, 22:18:02 UTC |
| 3df07d1 | sdong | 19 May 2014, 17:40:18 UTC | ThreadPool to allow decrease number of threads and increase of number of threads is to be instantly scheduled Summary: Add a feature to decrease the number of threads in thread pool. Also instantly schedule more threads if number of threads is increased. Here is the way it is implemented: each background thread needs its thread ID. After decreasing number of threads, all threads are woken up. The thread with the largest thread ID will terminate. If there are more threads to terminate, the thread will wake up all threads again. Another change is made so that when number of threads is increased, more threads are created and all previous excessive threads are woken up to do the work. Test Plan: Add a unit test. Reviewers: haobo, dhruba Reviewed By: haobo CC: yhchiang, igor, nkg-, leveldb Differential Revision: https://reviews.facebook.net/D18675 | 19 May 2014, 18:52:12 UTC |
| 5a1b41c | Ankit Gupta | 19 May 2014, 05:41:00 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 19 May 2014, 05:41:00 UTC |
| e87973c | Ankit Gupta | 19 May 2014, 05:40:48 UTC | Removing code from portal.h for setting handle of RestoreOptions and RestoreBackupableDB | 19 May 2014, 05:40:48 UTC |
| 1e56045 | Igor Canadi | 16 May 2014, 20:17:29 UTC | Add .swp to gitignore | 16 May 2014, 20:17:29 UTC |
| a0e9ee5 | Igor Canadi | 15 May 2014, 23:07:05 UTC | Add rapidjson to RocksDB Summary: This diff adds rapidjson (https://code.google.com/p/rapidjson/) to RocksDB repository. First step to adding JSON is to add a JSON parser :) I'm not sure if rapidjson is the right choice. I only considered folly as an alternative. Using folly JSON parser has serious downsides. I tried extracting folly::json from the folly library. However, it depends on folly::dynamic, which basically depends on everything else. I would prefer to avoid adding complete folly library as RocksDB dependency. Folly has a lot of its own dependencies (https://github.com/facebook/folly) and also looks like open source world has some trouble installing it (https://groups.google.com/forum/#!forum/facebook-folly -- 60% of the posts are compile errors). We can discuss this if you think otherwise. RapidJSON does not have any dependencies whatsoever. No boost, no STL. We don't need to compile it since it's all header files. The performance results are also impressive: https://code.google.com/p/rapidjson/wiki/Performance. However, I'll make sure to write code in a way that we can easily switch JSON implementations going forward. Quora thread has some alternatives: http://www.quora.com/What-is-the-best-C-JSON-library Test Plan: none Reviewers: dhruba, haobo, yhchiang, sdong, jamesgpearce Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18729 | 15 May 2014, 23:07:05 UTC |
| 0b3d03d | Kai Liu | 15 May 2014, 21:09:03 UTC | Materialize the hash index Summary: Materialize the hash index to avoid the soaring cpu/flash usage when initializing the database. Test Plan: existing unit tests passed Reviewers: sdong, haobo Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18339 | 15 May 2014, 21:09:03 UTC |
| 4e0602f | sdong | 14 May 2014, 15:53:09 UTC | Remove maximum key_size check in db_bench Summary: Key size limit doesn't seem to be applicable anymore. Remove it. Test Plan: run a couple of tests in db_bench Reviewers: haobo, igor, yhchiang, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D18723 | 15 May 2014, 18:06:37 UTC |
| f3ec191 | Igor Canadi | 15 May 2014, 17:16:58 UTC | Add build status to README.md | 15 May 2014, 17:16:58 UTC |
| 2ca7a98 | Igor Canadi | 15 May 2014, 07:20:34 UTC | Merge pull request #150 from Chilledheart/master Fix errors while building with clang (against libc++) under linux | 15 May 2014, 07:20:34 UTC |
| fa16ef3 | Chilledheart | 15 May 2014, 04:31:03 UTC | Fix errors while building with clang | 15 May 2014, 04:34:53 UTC |
| c07c960 | Igor Canadi | 14 May 2014, 23:23:40 UTC | Expose Status::code() | 14 May 2014, 23:23:40 UTC |
| cdc53dc | Igor Canadi | 14 May 2014, 23:08:02 UTC | Turn off travis notifications | 14 May 2014, 23:08:02 UTC |
| 52783c7 | Igor Canadi | 14 May 2014, 19:40:49 UTC | declare kInline size in arena.cc | 14 May 2014, 19:40:49 UTC |
| a490d2d | Igor Canadi | 14 May 2014, 19:36:21 UTC | Revert "Define kInlineSize in .cc file instead of header" This reverts commit 35a8873aa66d3e92eefaf216d44c965a588aec5a. | 14 May 2014, 19:36:21 UTC |
| 35a8873 | Igor Canadi | 14 May 2014, 19:29:46 UTC | Define kInlineSize in .cc file instead of header | 14 May 2014, 19:29:46 UTC |
| eea7322 | Igor Canadi | 14 May 2014, 19:14:18 UTC | Improve EnvHdfs Summary: Copy improvements from fbcode's version of EnvHdfs to our open-source version. Some very important bug fixes in there. Test Plan: compiles Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18711 | 14 May 2014, 19:14:18 UTC |
| f457444 | Igor Canadi | 14 May 2014, 19:13:50 UTC | Clean up compaction logging Summary: Cleaned up compaction logging a little bit. Now file sizes are easier to read. Also, removed the trailing space. Test Plan: verified that i'm happy with logging output: files_size[#33(seq=101,sz=98KB,0) #31(seq=81,sz=159KB,0) #26(seq=0,sz=637KB,0)] Reviewers: sdong, haobo, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18549 | 14 May 2014, 19:13:50 UTC |
| 3e4a9ec | sdong | 09 May 2014, 18:01:54 UTC | Arena to inline 2KB of data in it. Summary: In order to use arena to a use case that the total allocation size might be small (LogBuffer is already such a case), inline 1KB of data in it, so that it can be mostly in stack or inline in another class. If always inlining 2KB is a concern, I could make it a template to determine what to inline. However, dependents need to changes. Doesn't go with it for now Test Plan: make all check. Reviewers: haobo, igor, yhchiang, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18609 | 14 May 2014, 18:49:01 UTC |
| 1ef31b6 | Igor Canadi | 14 May 2014, 17:44:22 UTC | Merge pull request #143 from mlin/travis-ci Travis CI | 14 May 2014, 17:44:22 UTC |
| 29eef1f | Ankit Gupta | 14 May 2014, 05:40:01 UTC | Add javadoc for BackupableDBOptions constructor params | 14 May 2014, 05:40:01 UTC |
| b830bb9 | Ankit Gupta | 14 May 2014, 05:34:29 UTC | Add newline at end of file | 14 May 2014, 05:34:29 UTC |
| c92425e | Ankit Gupta | 14 May 2014, 05:23:04 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 14 May 2014, 05:23:04 UTC |
| cf0b773 | Ankit Gupta | 14 May 2014, 05:22:21 UTC | Add more options to backupable DB | 14 May 2014, 05:22:21 UTC |
| 036e323 | Ankit Gupta | 14 May 2014, 05:20:58 UTC | Add RestoreBackupableDB and RestoreOptions | 14 May 2014, 05:20:58 UTC |
| 9a0e3ab | Yueh-Hsuan Chiang | 13 May 2014, 23:45:54 UTC | Merge branch 'master' of github.com:facebook/rocksdb into show-reads | 13 May 2014, 23:45:54 UTC |
| e883407 | Yueh-Hsuan Chiang | 13 May 2014, 23:44:53 UTC | [Java] Refined the output of Java DbBenchmark. | 13 May 2014, 23:44:53 UTC |
| 8c2c460 | sdong | 13 May 2014, 21:42:31 UTC | FixedPrefixTransform to include prefix length in its name Summary: As title Test Plan: make all check. Reviewers: haobo, igor, yhchiang Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D18705 | 13 May 2014, 23:08:21 UTC |
| e30dec9 | Yueh-Hsuan Chiang | 13 May 2014, 20:19:50 UTC | [Java] Fixed a bug in Java DB Benchmark where random reads does not consider full key range. Summary: Fixed a bug in Java DB Benchmark where random reads does not consider full key range. Test Plan: make rocksdbjava make jdb_bench cd java jdb_bench.sh --db=/tmp/rocksdb-test --benchmarks=fillseq --use_existing_db=false --num=100000 jdb_bench.sh --db=/tmp/rocksdb-test --benchmarks=readrandom --use_existing_db=true --num=100000 --reads=1000000 Reviewers: haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18693 | 13 May 2014, 20:19:50 UTC |
| 93f2643 | Yueh-Hsuan Chiang | 13 May 2014, 20:08:45 UTC | [Java] Make read benchmarks print out found / not-found information. Summary: Make read benchmarks print out found / not-found information. Test Plan: make rocksdbjava make jdb_bench cd java ./jdb_bench.sh Reviewers: haobo, sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18699 | 13 May 2014, 20:11:26 UTC |
| 26f5dd9 | Igor Canadi | 13 May 2014, 19:30:55 UTC | TablePropertiesCollectorFactory Summary: This diff addresses task #4296714 and rethinks how users provide us with TablePropertiesCollectors as part of Options. Here's description of task #4296714: I'm debugging #4295529 and noticed that our count of user properties kDeletedKeys is wrong. We're sharing one single InternalKeyPropertiesCollector with all Table Builders. In LOG Files, we're outputting number of kDeletedKeys as connected with a single table, while it's actually the total count of deleted keys since creation of the DB. For example, this table has 3155 entries and 1391828 deleted keys. The problem with current approach that we call methods on a single TablePropertiesCollector for all the tables we create. Even worse, we could do it from multiple threads at the same time and TablePropertiesCollector has no way of knowing which table we're calling it for. Good part: Looks like nobody inside Facebook is using Options::table_properties_collectors. This means we should be able to painfully change the API. In this change, I introduce TablePropertiesCollectorFactory. For every table we create, we call `CreateTablePropertiesCollector`, which creates a TablePropertiesCollector for a single table. We then use it sequentially from a single thread, which means it doesn't have to be thread-safe. Test Plan: Added a test in table_properties_collector_test that fails on master (build two tables, assert that kDeletedKeys count is correct for the second one). Also, all other tests Reviewers: sdong, dhruba, haobo, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D18579 | 13 May 2014, 19:30:55 UTC |
| 2082a7d | Yueh-Hsuan Chiang | 13 May 2014, 19:28:47 UTC | [Java] Temporary set the number of BG threads based on the number of BG compactions. Summary: Before the Java binding for Env is ready, Java developers have no way to control the number of background threads. This diff provides a temporary solution where RocksDB.setMaxBackgroundCompactions() will affect the number of background threads. Note that once Env is ready. Changes made in this diff should be reverted. Test Plan: make rocksdbjava make jtest make jdb_bench java/jdb_bench.sh Reviewers: haobo, sdong Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18681 | 13 May 2014, 19:28:47 UTC |
| 1c7799d | Yueh-Hsuan Chiang | 13 May 2014, 00:50:21 UTC | Fixed a file-not-found issue when a log file is moved to archive. Summary: Fixed a file-not-found issue when a log file is moved to archive by doing a missing retry. Test Plan: make db_test export ROCKSDB_TEST=TransactionLogIteratorRace ./db_test Reviewers: sdong, haobo Reviewed By: sdong CC: igor, leveldb Differential Revision: https://reviews.facebook.net/D18669 | 13 May 2014, 00:50:21 UTC |
| d14581f | Yueh-Hsuan Chiang | 12 May 2014, 18:02:25 UTC | [Java] Rename org.rocksdb.Iterator to org.rocksdb.RocksIterator. Summary: Renamed Iterator to RocksIterator to avoid potential confliction with the Java built-in Iterator. Test Plan: make rocksdbjava make jtest Reviewers: haobo, dhruba, sdong, ankgup87 Reviewed By: sdong CC: leveldb, rsumbaly, swapnilghike, zzbennett Differential Revision: https://reviews.facebook.net/D18615 | 12 May 2014, 18:02:25 UTC |
| 3f8da15 | Igor Canadi | 12 May 2014, 17:48:23 UTC | Merge pull request #142 from mlin/build-in-paths-containing-spaces Fix building RocksDB in paths containing spaces | 12 May 2014, 17:48:23 UTC |
| d08073a | Igor Canadi | 11 May 2014, 22:44:32 UTC | Merge pull request #141 from dallasmarlow/master only use MAP_HUGETLB when the kernel supports it | 11 May 2014, 22:44:32 UTC |
| 557fbc9 | Dallas Marlow | 11 May 2014, 14:22:28 UTC | arena spacing | 11 May 2014, 14:22:28 UTC |
| 27c05ea | Mike Lin | 11 May 2014, 04:18:23 UTC | Add a minimal .travis.yml for Travis CI. Some ugly hacks needed to get RocksDB building in Travis' OpenVZ Ubuntu 12.04 environment. | 11 May 2014, 06:33:50 UTC |
| 76596b5 | Mike Lin | 11 May 2014, 04:01:25 UTC | Fix building RocksDB in paths containing spaces -- quote path names in Makefile and build_detect_platform. | 11 May 2014, 04:01:25 UTC |
| 3edc056 | Igor Canadi | 10 May 2014, 18:25:56 UTC | comment | 10 May 2014, 18:25:56 UTC |
| 038a477 | Igor Canadi | 10 May 2014, 17:49:33 UTC | Make it easier to start using RocksDB Summary: This diff is addressing multiple things with a single goal -- to make RocksDB easier to use: * Add some functions to Options that make RocksDB easier to tune. * Add example code for both simple RocksDB and RocksDB with Column Families. * Rewrite our README.md Regarding Options, I took a stab at something we talked about for a long time: * https://www.facebook.com/groups/rocksdb.dev/permalink/563169950448190/ I added functions: * IncreaseParallelism() -- easy, increases the thread pool and max_background_compactions * OptimizeLevelStyleCompaction(memtable_memory_budget) -- the easiest way to optimize rocksdb for less stalls with level style compaction. This is very likely not ideal configuration. Feel free to suggest improvements. I used some of Mark's suggestions from here: https://github.com/facebook/rocksdb/issues/54 * OptimizeUniversalStyleCompaction(memtable_memory_budget) -- optimize for universal compaction. Test Plan: compiled rocksdb. ran examples. Reviewers: dhruba, MarkCallaghan, haobo, sdong, yhchiang Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18621 | 10 May 2014, 17:49:33 UTC |
| 030db3d | Dallas Marlow | 09 May 2014, 22:58:39 UTC | testing | 09 May 2014, 22:58:39 UTC |
| acd17fd | sdong | 09 May 2014, 17:13:04 UTC | Remove unused variable in DBIter Summary: as title Test Plan: Still compile Reviewers: haobo, igor, yhchiang Reviewed By: igor CC: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18603 | 09 May 2014, 20:20:35 UTC |
| deb8940 | Tyler Neely | 09 May 2014, 18:49:25 UTC | have proprocessor choose correct mmap args | 09 May 2014, 18:49:25 UTC |
| fec4269 | Igor Canadi | 09 May 2014, 15:34:18 UTC | Fix more gflag namespace issues | 09 May 2014, 15:41:02 UTC |
| a1068c9 | Igor Canadi | 09 May 2014, 00:25:13 UTC | Make RocksDB work with newer gflags Summary: Newer gflags switched from `google` namespace to `gflags` namespace. See: https://github.com/facebook/rocksdb/issues/139 and https://github.com/facebook/rocksdb/issues/102 Unfortunately, they don't define any macro with their namespace, so we need to actually try to compile gflags with two different namespace to figure out which one is the correct one. Test Plan: works in fbcode environemnt. I'll also try in ubutnu with newer gflags Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18537 | 09 May 2014, 00:25:13 UTC |
| ddd4114 | sdong | 08 May 2014, 20:32:45 UTC | MergingIterator uses autovector instead of vector Summary: Use autovector in MergingIterator so that if there are 4 or less child iterators in it, iterator wrappers are inline, which is more likely to be cache friendly. Based on one test run with a shadow traffic of one product, it reduces CPU of MergingIterator::Seek() by half. Test Plan: make all check Reviewers: haobo, yhchiang, igor, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D18531 | 08 May 2014, 22:01:20 UTC |
| af7453a | Igor Canadi | 08 May 2014, 20:50:49 UTC | autovector::resize Summary: Resize the autovector! Test Plan: test Reviewers: sdong Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18543 | 08 May 2014, 20:50:49 UTC |
| 8e37a29 | Igor Canadi | 08 May 2014, 20:48:39 UTC | Compaction with zero outputs Summary: We had a hypothesis in https://reviews.facebook.net/D18507 that empty-string internal keys might have been caused by compaction filter deleting all the entries. I added a unit test for that case. Unforutnately, everything works as expected. Test Plan: this is a test Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18519 | 08 May 2014, 20:48:39 UTC |
| 1c6a027 | sdong | 08 May 2014, 04:56:40 UTC | HashLinkedList::Iterator: remove an ununsed class variable Summary: This variable is not used. Remove it. Test Plan: build. Reviewers: haobo, igor, yhchiang Reviewed By: haobo CC: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18525 | 08 May 2014, 20:28:52 UTC |
| f41cde3 | dallas marlow | 08 May 2014, 15:45:44 UTC | remove anon mmap allocation flag `MAP_HUGETLB` | 08 May 2014, 15:45:44 UTC |
| b5616da | Igor Canadi | 08 May 2014, 00:45:27 UTC | Fix iOS compile | 08 May 2014, 00:48:31 UTC |
| 768d424 | Igor Canadi | 07 May 2014, 23:52:12 UTC | [fix] SIGSEGV when VersionEdit in MANIFEST is corrupted Summary: This was reported by our customers in task #4295529. Cause: * MANIFEST file contains a VersionEdit, which contains file entries whose 'smallest' and 'largest' internal keys are empty. String with zero characters. Root cause of corruption was not investigated. We should report corruption when this happens. However, we currently SIGSEGV. Here's what happens: * VersionEdit encodes zero-strings happily and stores them in smallest and largest InternalKeys. InternalKey::Encode() does assert when `rep_.empty()`, but we don't assert in production environemnts. Also, we should never assert as a result of DB corruption. * As part of our ConsistencyCheck, we call GetLiveFilesMetaData() * GetLiveFilesMetadata() calls `file->largest.user_key().ToString()` * user_key() function does: 1. assert(size > 8) (ooops, no assert), 2. returns `Slice(internal_key.data(), internal_key.size() - 8)` * since `internal_key.size()` is unsigned int, this call translates to `Slice(whatever, 1298471928561892576182756)`. Bazinga. Fix: * VersionEdit checks if InternalKey is valid in `VersionEdit::GetInternalKey()`. If it's invalid, returns corruption. Lessons learned: * Always keep in mind that even if you `assert()`, production code will continue execution even if assert fails. * Never `assert` based on DB corruption. Assert only if the code should guarantee that assert can't fail. Test Plan: dumped offending manifest. Before: assert. Now: corruption Reviewers: dhruba, haobo, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18507 | 07 May 2014, 23:52:12 UTC |
| 313b2e5 | Igor Canadi | 07 May 2014, 23:51:30 UTC | Better INSTALL.md and Makefile rules Summary: We have a lot of problems with gflags. However, when compiling rocksdb static library, we don't need gflags dependency. Reorganize INSTALL.md such that first-time customers don't need any dependency installed to actually build rocksdb static library. Test Plan: none Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18501 | 07 May 2014, 23:51:30 UTC |
| 9efbd85 | sdong | 06 May 2014, 21:51:33 UTC | fsync directory after creating current file in NewDB() Summary: One of our users reported current file corruption. The machine was rebooted during the time. This is the only think I can think of which could cause current file corruption. Just add this paranoid check. Test Plan: make all check Reviewers: haobo, igor Reviewed By: haobo CC: yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18495 | 07 May 2014, 00:51:33 UTC |
| 3a171dc | sdong | 04 May 2014, 22:52:23 UTC | Pass logger to memtable rep and TLB page allocation error logged to info logs Summary: TLB page allocation errors are now logged to info logs, instead of stderr. In order to do that, mem table rep's factory functions take a info logger now. Test Plan: make all check Reviewers: haobo, igor, yhchiang Reviewed By: yhchiang CC: leveldb, yhchiang, dhruba Differential Revision: https://reviews.facebook.net/D18471 | 05 May 2014, 23:43:37 UTC |
| 044af85 | Igor Canadi | 05 May 2014, 21:44:57 UTC | Update HISTORY.md -- release RocksDB 3.0 | 05 May 2014, 21:44:57 UTC |
| 7984b9b | Igor Canadi | 05 May 2014, 21:30:24 UTC | BackupableDBTest thread-safe Summary: We need to lock accesses to some TestEnv variables. Otherwise we get failures like http://ci-builds.fb.com/job/rocksdb_asan_check/657/console Test Plan: make check Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18489 | 05 May 2014, 21:30:24 UTC |
| 15c3991 | Igor Canadi | 05 May 2014, 19:57:47 UTC | Add comment about ValueType | 05 May 2014, 19:57:47 UTC |
| 9e7d00d | Igor Canadi | 05 May 2014, 18:35:40 UTC | Make rocksdb work with all versions of lz4 Summary: There are some projects in fbcode that define lz4 dependency on r108. We, however, defined dependency on r117. That produced some interesting issues and our build system was not happy. This diff makes rocksdb work with both r108 and r117. Hopefully this will fix our problems. Test Plan: compiled rocksdb with both r108 and r117 lz4 Reviewers: dhruba, sdong, haobo Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18465 | 05 May 2014, 18:35:40 UTC |
