| e525bb1 | Igor Canadi | 29 April 2014, 15:59:36 UTC | Make kMajorVersion and kMinorVersion take version from version macros | 29 April 2014, 15:59:48 UTC |
| 6cb0cb3 | Igor Canadi | 29 April 2014, 15:32:56 UTC | Add version.h | 29 April 2014, 15:33:00 UTC |
| 91ef2ea | Igor Canadi | 29 April 2014, 03:44:33 UTC | Use new DBWithTTL API in tests | 29 April 2014, 03:46:24 UTC |
| f868dcb | Igor Canadi | 29 April 2014, 03:34:20 UTC | Support for adding TTL-ed column family Summary: This enables user to add a TTL column family to normal DB. Next step should be to expand StackableDB and create StackableColumnFamily, such that users can for example add geo-spatial column families to normal DB. Test Plan: added a test Reviewers: dhruba, haobo, ljin Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18201 | 29 April 2014, 03:34:20 UTC |
| 72ff275 | Igor Canadi | 29 April 2014, 03:30:27 UTC | Fix TransactionLogIterator EOF caching Summary: When TransactionLogIterator comes to EOF, it calls UnmarkEOF and continues reading. However, if glibc cached the EOF status of the file, it will get EOF again, even though the new data might have been written to it. This has been causing errors in Mac OS. Test Plan: test passes, was failing before Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18381 | 29 April 2014, 03:30:27 UTC |
| 9895465 | Yueh-Hsuan Chiang | 28 April 2014, 04:46:30 UTC | [Java] Fixed compile error due to the removal of ReadOptions.prefix_seek, minor improvement on DbBenchmark.java. | 28 April 2014, 04:49:45 UTC |
| 7ca06a3 | Igor Canadi | 28 April 2014, 00:17:15 UTC | Merge pull request #130 from donovanhide/master Add rocksdb_open_for_read_only to C API | 28 April 2014, 00:17:15 UTC |
| 4f9fae9 | Donovan Hide | 27 April 2014, 19:57:10 UTC | Add rocksdb_open_for_read_only to C API | 27 April 2014, 19:57:10 UTC |
| fe331c8 | Igor Canadi | 26 April 2014, 21:21:39 UTC | fix valgrind | 26 April 2014, 21:23:15 UTC |
| c489499 | Igor Canadi | 26 April 2014, 21:15:43 UTC | Fix OSX compile | 26 April 2014, 21:15:43 UTC |
| a618691 | Igor Canadi | 25 April 2014, 19:49:29 UTC | Read-only BackupEngine Summary: Read-only BackupEngine can connect to the same backup directory that is already running BackupEngine. That enables some interesting use-cases (i.e. restoring replica from primary's backup directory) Test Plan: added a unit test Reviewers: dhruba, haobo, ljin Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18297 | 25 April 2014, 19:49:29 UTC |
| ccaca59 | Lei Jin | 25 April 2014, 19:23:07 UTC | avoid calling FindFile twice in TwoLevelIterator for PlainTable Summary: this is to reclaim the regression introduced in https://reviews.facebook.net/D17853 Test Plan: make all check Reviewers: igor, haobo, sdong, dhruba, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17985 | 25 April 2014, 19:23:07 UTC |
| d642c60 | Lei Jin | 25 April 2014, 19:22:23 UTC | Check PrefixMayMatch on Seek() Summary: As a follow-up diff for https://reviews.facebook.net/D17805, add optimization to check PrefixMayMatch on Seek() Test Plan: make all check Reviewers: igor, haobo, sdong, yhchiang, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17853 | 25 April 2014, 19:22:23 UTC |
| 3995e80 | Lei Jin | 25 April 2014, 19:21:34 UTC | kill ReadOptions.prefix and .prefix_seek Summary: also add an override option total_order_iteration if you want to use full iterator with prefix_extractor Test Plan: make all check Reviewers: igor, haobo, sdong, yhchiang Reviewed By: haobo CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D17805 | 25 April 2014, 19:21:34 UTC |
| 8ce5492 | Igor Canadi | 25 April 2014, 18:58:02 UTC | Delete superversion and log outside of mutex Summary: As summary. Add two autovectors that get filled up in MakeRoomForWrite and they get deleted outside of mutex Test Plan: make check Reviewers: dhruba, haobo, ljin, sdong Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18249 | 25 April 2014, 18:58:02 UTC |
| ad3cd39 | Igor Canadi | 25 April 2014, 13:51:16 UTC | Column family logging Summary: Now that we have column families involved, we need to add extra context to every log message. They now start with "[column family name] log message" Also added some logging that I think would be useful, like level summary after every flush (I often needed that when going through the logs). Test Plan: make check + ran db_bench to confirm I'm happy with log output Reviewers: dhruba, haobo, ljin, yhchiang, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18303 | 25 April 2014, 13:51:16 UTC |
| 46b3076 | Igor Canadi | 25 April 2014, 13:50:51 UTC | Better stack trace in MAC Summary: Now this gives us the real deal stack trace: Assertion failed: (false), function GetProperty, file db/db_impl.cc, line 4072. Received signal 6 (Abort trap: 6) #0 0x7fff57ce39b9 #1 abort (in libsystem_c.dylib) + 125 #2 basename (in libsystem_c.dylib) + 0 #3 rocksdb::DBImpl::GetProperty(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) (in db_test) (db_impl.cc:4072) #4 rocksdb::_Test_Empty::_Run() (in db_test) (testharness.h:68) #5 rocksdb::_Test_Empty::_RunIt() (in db_test) (db_test.cc:1005) #6 rocksdb::test::RunAllTests() (in db_test) (testharness.cc:60) #7 main (in db_test) (db_test.cc:6697) #8 start (in libdyld.dylib) + 1 Test Plan: added artificial assert, saw great stack trace Reviewers: haobo, dhruba, ljin Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18309 | 25 April 2014, 13:50:51 UTC |
| a82c492 | Yueh-Hsuan Chiang | 25 April 2014, 05:15:25 UTC | [Java] Fixed some errors in the sample and benchmark code. | 25 April 2014, 05:15:25 UTC |
| e537a84 | Yueh-Hsuan Chiang | 24 April 2014, 23:17:00 UTC | [Java] Fixed a compile error caused in some Java version. | 24 April 2014, 23:17:00 UTC |
| 3e0b93b | Yueh-Hsuan Chiang | 24 April 2014, 22:37:27 UTC | [Java] Fixed a compile error. | 24 April 2014, 22:37:27 UTC |
| 0597949 | Yueh-Hsuan Chiang | 24 April 2014, 20:43:57 UTC | [Java] Enable filluniquerandom, readseq, BloomFilter, and 70+ command-line options to DbBenchmark.java Summary: * Add filluniquerandom * Add readseq, implemented using iterator. * Realize most command-line-arguments from db_bench.cc (70+). * Some code are commented out as some of the options in Options not yet have Java bindings. * Add default option to DbBenchmark. * RocksDB will now take the ownership of all c++ raw-pointers from Options, which includes a c++ raw-pointer for Filter. Test Plan: ./jdb_bench.sh --db=/tmp/rocksjava-bench/db --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=/tmp/rocksjava-bench/wal --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --benchmarks=filluniquerandom,readseq,readrandom --use_existing_db=0 --threads=4 Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18267 | 24 April 2014, 20:43:57 UTC |
| 4cd9f58 | Igor Canadi | 24 April 2014, 18:56:37 UTC | Fix corruption test | 24 April 2014, 18:56:41 UTC |
| 478990c | Igor Canadi | 24 April 2014, 18:13:28 UTC | Make CompactionInputErrorParanoid less flakey Summary: I'm getting lots of e-mails with CompactionInputErrorParanoid failing. Most recent example early morning today was: http://ci-builds.fb.com/job/rocksdb_valgrind/562/consoleFull I'm putting a stop to these e-mails. I investigated why the test is flakey and it turns out it's because of non-determinsim of compaction scheduling. If there is a compaction after the last flush, CorruptFile will corrupt the compacted file instead of file at level 0 (as it assumes). That makes `Check(9, 9)` fail big time. I also saw some errors with table file getting outputed to >= 1 levels instead of 0. Also fixed that. Test Plan: Ran corruption_test 100 times without a failure. Previously it usually failed at 10th occurrence. Reviewers: dhruba, haobo, ljin Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18285 | 24 April 2014, 18:13:28 UTC |
| fc3127e | Igor Canadi | 24 April 2014, 17:52:20 UTC | Install stack trace handlers in unit tests Summary: Sometimes, our tests fail because of normal `assert` call. It would be helpful to see stack trace in that case, too. Test Plan: Added `assert(false)` and verified it prints out stack trace Reviewers: haobo, dhruba, sdong, ljin, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18291 | 24 April 2014, 17:52:20 UTC |
| a40970a | Igor Canadi | 24 April 2014, 16:28:11 UTC | Run whitebox test before black box | 24 April 2014, 16:28:16 UTC |
| 472a80a | Igor Canadi | 24 April 2014, 13:46:50 UTC | Initialize verification_failed in db_stress | 24 April 2014, 13:53:11 UTC |
| 2413a06 | Igor Canadi | 24 April 2014, 13:22:58 UTC | Improve stability of db_stress Summary: Currently, whenever DB Verification fails we bail out by calling `exit(1)`. This is kind of bad since it causes unclean shutdown and spew of error log messages like: 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument 05:03:27 pthread lock: Invalid argument This diff adds a new parameter that is set to true when verification fails. It can then use the parameter to bail out safely. Test Plan: Casued artificail failure. Verified that exit was clean. Reviewers: dhruba, haobo, ljin Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18243 | 24 April 2014, 13:22:58 UTC |
| d8fe006 | Yueh-Hsuan Chiang | 24 April 2014, 03:54:35 UTC | [Java] Add Java bindings and Java tests for 30+ rocksdb::Options. Summary: Add Java bindings and Java tests for 30+ rocksdb::Options. Codes are machine-genearted based on include/rocksdb/options.h with manual-polishment. Test Plan: make rocksdbjava make jtest Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18273 | 24 April 2014, 03:54:35 UTC |
| 4de5b84 | sdong | 24 April 2014, 00:51:16 UTC | Fix a bug in IterKey Summary: IterKey set buffer_size_ to a wrong initial value, causing it to always allocate values from heap instead of stack if the key size is smaller. Fix it. Test Plan: make all check Reviewers: haobo, ljin Reviewed By: haobo CC: igor, dhruba, yhchiang, leveldb Differential Revision: https://reviews.facebook.net/D18279 | 24 April 2014, 02:45:22 UTC |
| f9f8965 | Igor Canadi | 23 April 2014, 13:11:35 UTC | Print out stack trace in mac, too Summary: While debugging Mac-only issue with ThreadLocalPtr, this was very useful. Let's print out stack trace in MAC OS, too. Test Plan: Verified that somewhat useful stack trace was generated on mac. Will run PrintStack() on linux, too. Reviewers: ljin, haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18189 | 23 April 2014, 13:11:35 UTC |
| a570740 | sdong | 23 April 2014, 00:17:33 UTC | Expose number of entries in mem tables to users Summary: In this patch, two new DB properties are defined: rocksdb.num-immutable-mem-table and rocksdb.num-entries-imm-mem-tables, from where number of entries in mem tables can be exposed to users Test Plan: Cover the codes in db_test make all check Reviewers: haobo, ljin, igor Reviewed By: igor CC: nkg-, igor, yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18207 | 23 April 2014, 05:13:21 UTC |
| 5f1daf7 | Lei Jin | 23 April 2014, 04:14:25 UTC | get rid of shared_ptr in memtable.cc Summary: Get rid of the devil. Probably won't impact anything on the perf side. Test Plan: make all check Reviewers: igor, haobo, sdong, yhchiang Reviewed By: haobo CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D18153 | 23 April 2014, 04:14:25 UTC |
| 547bb6a | Lei Jin | 23 April 2014, 04:13:34 UTC | simplify ThreadLocalPtr a little bit Summary: make singleton a static member instead of dynamic object. This should also avoid the race on unique_ptr Test Plan: make all check Reviewers: igor, haobo, sdong Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D18177 | 23 April 2014, 04:13:34 UTC |
| 86a0133 | sdong | 23 April 2014, 01:31:55 UTC | PlainTableReader to expose index size to users Summary: This is a temp solution to expose index sizes to users from PlainTableReader before we persistent them to files. In this patch, the memory consumption of indexes used by PlainTableReader will be reported as two user defined properties, so that users can monitor them. Test Plan: Add a unit test. make all check` Reviewers: haobo, ljin Reviewed By: haobo CC: nkg-, yhchiang, igor, ljin, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18195 | 23 April 2014, 02:29:05 UTC |
| 1068d2f | Igor Canadi | 23 April 2014, 01:38:10 UTC | Revert "Better port::Mutex::AssertHeld() and AssertNotHeld()" This reverts commit ddafceb6c2ecb83b7bdf6711ea1c30d97aeb3b8f. | 23 April 2014, 01:38:10 UTC |
| ddafceb | Igor Canadi | 23 April 2014, 00:26:21 UTC | Better port::Mutex::AssertHeld() and AssertNotHeld() Summary: Using ThreadLocalPtr as a flag to determine if a mutex is locked or not enables us to implement AssertNotHeld(). It also makes AssertHeld() actually correct. I had to remove port::Mutex as a dependency for util/thread_local.h, but that's fine since we can just use std::mutex :) Test Plan: make check Reviewers: ljin, dhruba, haobo, sdong, yhchiang Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18171 | 23 April 2014, 00:26:21 UTC |
| 2912340 | Yueh-Hsuan Chiang | 22 April 2014, 20:46:42 UTC | Merge pull request #125 from ankgup87/master [Java] Add bloom filter JNI bindings | 22 April 2014, 20:46:42 UTC |
| 042221b | Ankit Gupta | 22 April 2014, 20:05:40 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 22 April 2014, 20:05:40 UTC |
| 3992aec | Igor Canadi | 22 April 2014, 18:27:33 UTC | Support for column families in TTL DB Summary: This will enable people using TTL DB to do so with multiple column families. They can also specify different TTLs for each one. TODO: Implement CreateColumnFamily() in TTL world. Test Plan: Added a very simple sanity test. Reviewers: dhruba, haobo, ljin, sdong, yhchiang Reviewed By: haobo CC: leveldb, alberts Differential Revision: https://reviews.facebook.net/D17859 | 22 April 2014, 18:27:33 UTC |
| dd9f6f0 | Ankit Gupta | 22 April 2014, 17:51:39 UTC | Fix formatting | 22 April 2014, 17:51:39 UTC |
| e557297 | James Pearce | 22 April 2014, 16:12:19 UTC | New CLA form | 22 April 2014, 16:12:19 UTC |
| 7a5106f | Ankit Gupta | 22 April 2014, 16:01:57 UTC | Add doc | 22 April 2014, 16:01:57 UTC |
| 2214fd8 | Ankit Gupta | 22 April 2014, 15:58:43 UTC | Refactor filter impl | 22 April 2014, 15:58:43 UTC |
| 89cb481 | Ankit Gupta | 22 April 2014, 07:09:40 UTC | Fix doc | 22 April 2014, 07:09:40 UTC |
| 677b0d6 | Ankit Gupta | 22 April 2014, 07:04:56 UTC | Refactor filter impl | 22 April 2014, 07:04:56 UTC |
| 5e797cf | Ankit Gupta | 22 April 2014, 06:56:19 UTC | Change filter implementation | 22 April 2014, 06:56:19 UTC |
| cea2be2 | Ankit Gupta | 22 April 2014, 03:27:09 UTC | Fix formatting | 22 April 2014, 03:27:09 UTC |
| dc4b27a | Ankit Gupta | 22 April 2014, 03:25:30 UTC | Add bloom filters | 22 April 2014, 03:25:30 UTC |
| af6ad11 | Yueh-Hsuan Chiang | 22 April 2014, 00:49:47 UTC | Fix SIGFAULT when running sst_dump on v2.6 db Summary: Fix the sigfault when running sst_dump on v2.6 db. Test Plan: git checkout bba6595b1f3f42cf79bb21c2d5b981ede1cc0063 make clean make db_bench ./db_bench --db=/tmp/some/db --benchmarks=fillseq arc patch D18039 make clean make sst_dump ./sst_dump --file=/tmp/some/db --command=check Reviewers: igor, haobo, sdong Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18039 | 22 April 2014, 00:49:47 UTC |
| c2da9e5 | Igor Canadi | 22 April 2014, 00:45:04 UTC | Flush before Fsync()/Sync() Summary: Calling Fsync()/Sync() on a file should give the guarantee that whatever you written to the file is now persisted. This is currently not the case, since we might have some data left in application cache as we do Fsync()/Sync(). For example, BuildTable() calls Fsync() without the flush, assuming all sst data is now persisted, but it's actually not. This may result in big inconsistencies. Test Plan: no test Reviewers: sdong, dhruba, haobo, ljin, yhchiang Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18159 | 22 April 2014, 00:45:04 UTC |
| ba16c1f | Igor Canadi | 22 April 2014, 00:43:48 UTC | Move benchmark timing to Env::NowNanos() | 22 April 2014, 00:43:51 UTC |
| e316af5 | Yueh-Hsuan Chiang | 21 April 2014, 22:52:59 UTC | [Java] Add Java binding and Java test for ReadOptions. Summary: Add Java binding and test for rocksdb::ReadOptions. Test Plan: make rocksdbjava make jtest Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18129 | 21 April 2014, 22:52:59 UTC |
| d0939cd | Igor Canadi | 21 April 2014, 22:42:04 UTC | Single-threaded asan_crash_test | 21 April 2014, 22:42:28 UTC |
| ef8b8a8 | Yueh-Hsuan Chiang | 21 April 2014, 22:40:46 UTC | [Java] Add Java bindings for memtables and sst format. Summary: Add Java bindings for memtables and sst format. Specifically, add two abstract Java classses --- MemTableConfig and SstFormatConfig. Each MemTable / SST implementation should has its own config class extends MemTableConfig / SstFormatConfig respectively and pass it to Options via setMemTableConfig / setSstConfig. Test Plan: make rocksdbjava make jdb_test make jdb_bench java/jdb_bench.sh \ --benchmarks=fillseq,readrandom,readwhilewriting \ --memtablerep=hash_skiplist \ --use_plain_table=1 \ --key_size=20 \ --prefix_size=12 \ --value_size=100 \ --cache_size=17179869184 \ --disable_wal=0 \ --sync=0 \ Reviewers: haobo, ankgup87, sdong Reviewed By: haobo CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D17997 | 21 April 2014, 22:40:46 UTC |
| 8dc3436 | Igor Canadi | 21 April 2014, 20:01:50 UTC | Rename "benchmark" back to "bench". Also, make `benchharness.cc` not compiled into rocksdb library. | 21 April 2014, 20:12:15 UTC |
| 05c1686 | Igor Canadi | 21 April 2014, 19:55:41 UTC | Relax env_test::AllocateTest | 21 April 2014, 19:56:32 UTC |
| ff1b5df | Pratyush Seth | 21 April 2014, 19:29:55 UTC | Added benchmark functionality on the lines of folly/Benchmark.h Summary: Added benchmark functionality on the lines of folly/Benchmark.h Test Plan: Added unit tests Reviewers: igor, haobo, sdong, ljin, yhchiang, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D17973 | 21 April 2014, 19:29:55 UTC |
| c7076a7 | Igor Canadi | 21 April 2014, 19:12:02 UTC | Fix Allocate test Summary: For some reason, on a subset of our continuous build machines, preallocation is allocating 8 block more than it should be. Let's relax the test a little bit -- now we require the test to allocate *at least* the number of blocks as we told them to. Test Plan: no Reviewers: ljin, haobo, sdong Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18141 | 21 April 2014, 19:12:02 UTC |
| f813279 | Igor Canadi | 21 April 2014, 18:08:30 UTC | Remove TransactionLogIteratorRace when -DNDEBUG | 21 April 2014, 18:08:30 UTC |
| 11e8525 | Igor Canadi | 21 April 2014, 16:59:39 UTC | Merge pull request #124 from ankgup87/master [Java] Add iterator JNI bindings | 21 April 2014, 16:59:39 UTC |
| 0f2d768 | Lei Jin | 21 April 2014, 16:10:12 UTC | hints for narrowing down FindFile range and avoiding checking unrelevant L0 files Summary: The file tree structure in Version is prebuilt and the range of each file is known. On the Get() code path, we do binary search in FindFile() by comparing target key with each file's largest key and also check the range for each L0 file. With some pre-calculated knowledge, each key comparision that has been done can serve as a hint to narrow down further searches: (1) If a key falls within a L0 file's range, we can safely skip the next file if its range does not overlap with the current one. (2) If a key falls within a file's range in level L0 - Ln-1, we should only need to binary search in the next level for files that overlap with the current one. (1) will be able to skip some files depending one the key distribution. (2) can greatly reduce the range of binary search, especially for bottom levels, given that one file most likely only overlaps with N files from the level below (where N is max_bytes_for_level_multiplier). So on level L, we will only look at ~N files instead of N^L files. Some inital results: measured with 500M key DB, when write is light (10k/s = 1.2M/s), this improves QPS ~7% on top of blocked bloom. When write is heavier (80k/s = 9.6M/s), it gives us ~13% improvement. Test Plan: make all check Reviewers: haobo, igor, dhruba, sdong, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17205 | 21 April 2014, 16:10:12 UTC |
| bbdd550 | Ankit Gupta | 20 April 2014, 06:17:42 UTC | Remove getIterator function from portal | 20 April 2014, 06:17:42 UTC |
| 1574e0c | Ankit Gupta | 19 April 2014, 20:21:06 UTC | Add doc | 19 April 2014, 20:21:06 UTC |
| 06b590d | Ankit Gupta | 19 April 2014, 20:13:01 UTC | Add doc | 19 April 2014, 20:13:01 UTC |
| dc28a72 | Ankit Gupta | 19 April 2014, 20:05:21 UTC | Add doc + refactor + fix formatting | 19 April 2014, 20:05:21 UTC |
| 1d6c1e0 | Ankit Gupta | 19 April 2014, 19:55:28 UTC | Add more iterator JNI bindings | 19 April 2014, 19:55:28 UTC |
| eda3984 | Ankit Gupta | 19 April 2014, 10:35:01 UTC | Add more iterator functions | 19 April 2014, 10:35:01 UTC |
| 5bbeefa | Ankit Gupta | 19 April 2014, 10:26:22 UTC | Adding iterator JNI binding | 19 April 2014, 10:26:22 UTC |
| 27d3bc1 | sdong | 18 April 2014, 01:00:58 UTC | Use a different approach to make sure BlockBasedTableReader can use hash index on older files Summary: A recent commit https://github.com/facebook/rocksdb/commit/e37dd216f9384bfdabc6760fa296e8ee28c79d30 makes sure hash index can be used when reading existing files. This patch uses another way to achieve the approach: (1) Currently, always writing kBinarySearch to files, despite of BlockBasedTableOptions.IndexType setting. (2) When reading a file, read out the field, and make sure it is kBinarySearch, while always use index type by users. The reason for doing it is, to reserve kHashSearch property on disk to future. If now we write out binary index for both of kHashSearch and kBinarySearch. We have to use a new flag in the future for hash index on disk, otherwise compatibility would break. Also, we want the real index type and type shown in properties block to be consistent. Test Plan: make all check Reviewers: haobo, kailiu Reviewed By: kailiu CC: igor, ljin, yhchiang, xjin, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18009 | 18 April 2014, 21:09:21 UTC |
| 35c968f | Igor Canadi | 18 April 2014, 20:53:52 UTC | Merge pull request #122 from ankgup87/master [Java] Add statistics JNI bindings | 18 April 2014, 20:53:52 UTC |
| 686fdea | Ankit Gupta | 18 April 2014, 17:48:48 UTC | Fix formatting issues | 18 April 2014, 17:48:48 UTC |
| ebd85e8 | Ankit Gupta | 18 April 2014, 17:47:03 UTC | Fix build | 18 April 2014, 17:47:03 UTC |
| dc291f5 | Ankit Gupta | 18 April 2014, 17:32:14 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb Conflicts: Makefile java/Makefile java/org/rocksdb/Options.java java/rocksjni/portal.h | 18 April 2014, 17:32:14 UTC |
| 1a8abe7 | Igor Canadi | 18 April 2014, 16:45:48 UTC | Merge pull request #120 from jamesgpearce/master Added period | 18 April 2014, 16:45:48 UTC |
| a745089 | James Pearce | 18 April 2014, 16:33:27 UTC | Added period This is a PR to test some tooling; please do not merge without talking to @jamesgpearce :) | 18 April 2014, 16:33:27 UTC |
| 9b2a093 | Yueh-Hsuan Chiang | 18 April 2014, 08:14:29 UTC | [Java] Add Java bindings for 30 options for rocksdb::DBOptions. Summary: 1. Add Java bindings for 30 options for rocksdb::DBOptions. 2. Add org.rocksdb.test.OptionsTest 3. Codes are semi-auto generated, JavaDocs are manually polished. Test Plan: make rocksdbjava make jtest Reviewers: haobo, ankgup87, sdong, dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18015 | 18 April 2014, 08:14:29 UTC |
| bb6fd15 | Yueh-Hsuan Chiang | 18 April 2014, 00:28:51 UTC | [Java] Add a basic binding and test for BackupableDB and StackableDB. Summary: Add a skeleton binding and test for BackupableDB which shows that BackupableDB and RocksDB can share the same JNI calls. Test Plan: make rocksdbjava make jtest Reviewers: haobo, ankgup87, sdong, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17793 | 18 April 2014, 00:28:51 UTC |
| 6517922 | sdong | 17 April 2014, 22:14:04 UTC | Fix bugs introduced by D17961 Summary: D17961 has two bugs: (1) two level iterator fails to populate FileMetaData.table_reader, causing performance regression. (2) table cache handle the !status.ok() case in the wrong place, causing seg fault which shouldn't happen. Test Plan: make all check Reviewers: ljin, igor, haobo Reviewed By: ljin CC: yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D17991 | 18 April 2014, 00:25:28 UTC |
| ce353c2 | Igor Canadi | 17 April 2014, 21:43:42 UTC | Nuke tools/shell Summary: We don't use or build this code Test Plan: builds Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D17979 | 17 April 2014, 21:43:42 UTC |
| 86ae820 | Igor Canadi | 17 April 2014, 21:29:06 UTC | Fix ifdef NDEBUG | 17 April 2014, 21:29:28 UTC |
| fa430bf | sdong | 17 April 2014, 21:07:05 UTC | Minimize accessing multiple objects in Version::Get() Summary: One of our profilings shows that Version::Get() sometimes is slow when getting pointer of user comparators or other global objects. In this patch: (1) we keep pointers of immutable objects in Version to avoid accesses them though option objects or cfd objects (2) table_reader is directly cached in FileMetaData so that table cache don't have to go through handle first to fetch it (3) If level 0 has less than 3 files, skip the filtering logic based on SST tables' key range. Smallest and largest key are stored in separated memory locations, which has potential cache misses Test Plan: make all check Reviewers: haobo, ljin Reviewed By: haobo CC: igor, yhchiang, nkg-, leveldb Differential Revision: https://reviews.facebook.net/D17739 | 17 April 2014, 21:14:00 UTC |
| e37dd21 | Kai Liu | 11 April 2014, 20:55:26 UTC | Index type doesn't have to be persisted Summary: With the recent changes, there is no need to check the property block about the index block type. If user want to use it, they don't really need any disk format change; everything happens in the fly. Also another team encountered an error while reading the index type from properties. Test Plan: ran all the tests Reviewers: sdong CC: Task ID: # Blame Rev: | 17 April 2014, 18:08:12 UTC |
| 62551b1 | Igor Canadi | 17 April 2014, 17:49:58 UTC | Don't compile sync_point if NDEBUG Summary: We don't really need sync_point.o if we're compiling with NDEBUG. This diff depends on D17823 Test Plan: compiles Reviewers: haobo, ljin, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17829 | 17 April 2014, 17:49:58 UTC |
| 5cef458 | sdong | 17 April 2014, 02:30:33 UTC | RocksDB 2.8 to be able to read files generated by 2.6 Summary: From 2.6 to 2.7, property block name is renamed from rocksdb.stats to rocksdb.properties. Older properties were not able to be loaded. In 2.8, we seem to have added some logic that uses property block without checking null pointers, which create segment faults. In this patch, we fix it by: (1) try rocksdb.stats if rocksdb.properties is not found (2) add some null checking before consuming rep->table_properties Test Plan: make sure a file generated in 2.7 couldn't be opened now can be opened. Reviewers: haobo, igor, yhchiang Reviewed By: igor CC: ljin, xjin, dhruba, kailiu, leveldb Differential Revision: https://reviews.facebook.net/D17961 | 17 April 2014, 16:51:43 UTC |
| 136e4ef | Ankit Gupta | 17 April 2014, 05:02:46 UTC | Add doc | 17 April 2014, 05:02:46 UTC |
| d160595 | Ankit Gupta | 17 April 2014, 05:00:44 UTC | Add doc | 17 April 2014, 05:00:44 UTC |
| d3b44f0 | Ankit Gupta | 17 April 2014, 04:55:15 UTC | Fix white spaces | 17 April 2014, 04:55:15 UTC |
| 6b0cc41 | Ankit Gupta | 17 April 2014, 04:39:44 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 17 April 2014, 04:39:44 UTC |
| 5108ed7 | Ankit Gupta | 17 April 2014, 04:39:29 UTC | Add fixed classes | 17 April 2014, 04:39:29 UTC |
| 320ae72 | Ankit Gupta | 17 April 2014, 04:38:33 UTC | Add histogramType for statistics | 17 April 2014, 04:38:33 UTC |
| 161d9e5 | Igor Canadi | 16 April 2014, 22:15:22 UTC | Don't overflow size_t in mac | 16 April 2014, 22:15:22 UTC |
| 5c12f27 | Igor Canadi | 16 April 2014, 16:09:28 UTC | Remove tautological assert | 16 April 2014, 16:09:28 UTC |
| 144066a | Ankit Gupta | 16 April 2014, 01:08:37 UTC | Deleted statisticsType | 16 April 2014, 01:08:37 UTC |
| 495fc80 | Ankit Gupta | 16 April 2014, 00:38:20 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 16 April 2014, 00:38:20 UTC |
| b18d914 | Ankit Gupta | 16 April 2014, 00:37:50 UTC | Change statisticsType to tickerType | 16 April 2014, 00:37:50 UTC |
| faf7691 | Igor Canadi | 16 April 2014, 00:12:18 UTC | Close DB at the end of DontRollEmptyLogs test | 16 April 2014, 00:20:56 UTC |
| 1803ed2 | Igor Canadi | 15 April 2014, 23:31:49 UTC | Fix Mac OS compile | 15 April 2014, 23:31:49 UTC |
| 7d83885 | Igor Canadi | 15 April 2014, 22:59:34 UTC | Fix compile issues when doing make release | 15 April 2014, 23:00:10 UTC |
| 0f40fe4 | sdong | 15 April 2014, 16:06:13 UTC | When creating a new DB, fail it when wal_dir contains existing log files Summary: Current behavior of creating new DB is, if there is existing log files, we will go ahead and replay them on top of empty DB. This is a behavior that no user would expect. With this patch, we will fail the creation if a user creates a DB with existing log files. Test Plan: make all check Reviewers: haobo, igor, ljin Reviewed By: haobo CC: nkg-, yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D17817 | 15 April 2014, 21:01:57 UTC |
| c166615 | Igor Canadi | 15 April 2014, 20:51:07 UTC | Fix compile issues introduced by RocksDBLite | 15 April 2014, 20:51:07 UTC |
