| eea7322 | Igor Canadi | 14 May 2014, 19:14:18 UTC | Improve EnvHdfs Summary: Copy improvements from fbcode's version of EnvHdfs to our open-source version. Some very important bug fixes in there. Test Plan: compiles Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18711 | 14 May 2014, 19:14:18 UTC |
| f457444 | Igor Canadi | 14 May 2014, 19:13:50 UTC | Clean up compaction logging Summary: Cleaned up compaction logging a little bit. Now file sizes are easier to read. Also, removed the trailing space. Test Plan: verified that i'm happy with logging output: files_size[#33(seq=101,sz=98KB,0) #31(seq=81,sz=159KB,0) #26(seq=0,sz=637KB,0)] Reviewers: sdong, haobo, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18549 | 14 May 2014, 19:13:50 UTC |
| 3e4a9ec | sdong | 09 May 2014, 18:01:54 UTC | Arena to inline 2KB of data in it. Summary: In order to use arena to a use case that the total allocation size might be small (LogBuffer is already such a case), inline 1KB of data in it, so that it can be mostly in stack or inline in another class. If always inlining 2KB is a concern, I could make it a template to determine what to inline. However, dependents need to changes. Doesn't go with it for now Test Plan: make all check. Reviewers: haobo, igor, yhchiang, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18609 | 14 May 2014, 18:49:01 UTC |
| 1ef31b6 | Igor Canadi | 14 May 2014, 17:44:22 UTC | Merge pull request #143 from mlin/travis-ci Travis CI | 14 May 2014, 17:44:22 UTC |
| 9a0e3ab | Yueh-Hsuan Chiang | 13 May 2014, 23:45:54 UTC | Merge branch 'master' of github.com:facebook/rocksdb into show-reads | 13 May 2014, 23:45:54 UTC |
| e883407 | Yueh-Hsuan Chiang | 13 May 2014, 23:44:53 UTC | [Java] Refined the output of Java DbBenchmark. | 13 May 2014, 23:44:53 UTC |
| 8c2c460 | sdong | 13 May 2014, 21:42:31 UTC | FixedPrefixTransform to include prefix length in its name Summary: As title Test Plan: make all check. Reviewers: haobo, igor, yhchiang Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D18705 | 13 May 2014, 23:08:21 UTC |
| e30dec9 | Yueh-Hsuan Chiang | 13 May 2014, 20:19:50 UTC | [Java] Fixed a bug in Java DB Benchmark where random reads does not consider full key range. Summary: Fixed a bug in Java DB Benchmark where random reads does not consider full key range. Test Plan: make rocksdbjava make jdb_bench cd java jdb_bench.sh --db=/tmp/rocksdb-test --benchmarks=fillseq --use_existing_db=false --num=100000 jdb_bench.sh --db=/tmp/rocksdb-test --benchmarks=readrandom --use_existing_db=true --num=100000 --reads=1000000 Reviewers: haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18693 | 13 May 2014, 20:19:50 UTC |
| 93f2643 | Yueh-Hsuan Chiang | 13 May 2014, 20:08:45 UTC | [Java] Make read benchmarks print out found / not-found information. Summary: Make read benchmarks print out found / not-found information. Test Plan: make rocksdbjava make jdb_bench cd java ./jdb_bench.sh Reviewers: haobo, sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18699 | 13 May 2014, 20:11:26 UTC |
| 26f5dd9 | Igor Canadi | 13 May 2014, 19:30:55 UTC | TablePropertiesCollectorFactory Summary: This diff addresses task #4296714 and rethinks how users provide us with TablePropertiesCollectors as part of Options. Here's description of task #4296714: I'm debugging #4295529 and noticed that our count of user properties kDeletedKeys is wrong. We're sharing one single InternalKeyPropertiesCollector with all Table Builders. In LOG Files, we're outputting number of kDeletedKeys as connected with a single table, while it's actually the total count of deleted keys since creation of the DB. For example, this table has 3155 entries and 1391828 deleted keys. The problem with current approach that we call methods on a single TablePropertiesCollector for all the tables we create. Even worse, we could do it from multiple threads at the same time and TablePropertiesCollector has no way of knowing which table we're calling it for. Good part: Looks like nobody inside Facebook is using Options::table_properties_collectors. This means we should be able to painfully change the API. In this change, I introduce TablePropertiesCollectorFactory. For every table we create, we call `CreateTablePropertiesCollector`, which creates a TablePropertiesCollector for a single table. We then use it sequentially from a single thread, which means it doesn't have to be thread-safe. Test Plan: Added a test in table_properties_collector_test that fails on master (build two tables, assert that kDeletedKeys count is correct for the second one). Also, all other tests Reviewers: sdong, dhruba, haobo, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D18579 | 13 May 2014, 19:30:55 UTC |
| 2082a7d | Yueh-Hsuan Chiang | 13 May 2014, 19:28:47 UTC | [Java] Temporary set the number of BG threads based on the number of BG compactions. Summary: Before the Java binding for Env is ready, Java developers have no way to control the number of background threads. This diff provides a temporary solution where RocksDB.setMaxBackgroundCompactions() will affect the number of background threads. Note that once Env is ready. Changes made in this diff should be reverted. Test Plan: make rocksdbjava make jtest make jdb_bench java/jdb_bench.sh Reviewers: haobo, sdong Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18681 | 13 May 2014, 19:28:47 UTC |
| 1c7799d | Yueh-Hsuan Chiang | 13 May 2014, 00:50:21 UTC | Fixed a file-not-found issue when a log file is moved to archive. Summary: Fixed a file-not-found issue when a log file is moved to archive by doing a missing retry. Test Plan: make db_test export ROCKSDB_TEST=TransactionLogIteratorRace ./db_test Reviewers: sdong, haobo Reviewed By: sdong CC: igor, leveldb Differential Revision: https://reviews.facebook.net/D18669 | 13 May 2014, 00:50:21 UTC |
| d14581f | Yueh-Hsuan Chiang | 12 May 2014, 18:02:25 UTC | [Java] Rename org.rocksdb.Iterator to org.rocksdb.RocksIterator. Summary: Renamed Iterator to RocksIterator to avoid potential confliction with the Java built-in Iterator. Test Plan: make rocksdbjava make jtest Reviewers: haobo, dhruba, sdong, ankgup87 Reviewed By: sdong CC: leveldb, rsumbaly, swapnilghike, zzbennett Differential Revision: https://reviews.facebook.net/D18615 | 12 May 2014, 18:02:25 UTC |
| 3f8da15 | Igor Canadi | 12 May 2014, 17:48:23 UTC | Merge pull request #142 from mlin/build-in-paths-containing-spaces Fix building RocksDB in paths containing spaces | 12 May 2014, 17:48:23 UTC |
| d08073a | Igor Canadi | 11 May 2014, 22:44:32 UTC | Merge pull request #141 from dallasmarlow/master only use MAP_HUGETLB when the kernel supports it | 11 May 2014, 22:44:32 UTC |
| 557fbc9 | Dallas Marlow | 11 May 2014, 14:22:28 UTC | arena spacing | 11 May 2014, 14:22:28 UTC |
| 27c05ea | Mike Lin | 11 May 2014, 04:18:23 UTC | Add a minimal .travis.yml for Travis CI. Some ugly hacks needed to get RocksDB building in Travis' OpenVZ Ubuntu 12.04 environment. | 11 May 2014, 06:33:50 UTC |
| 76596b5 | Mike Lin | 11 May 2014, 04:01:25 UTC | Fix building RocksDB in paths containing spaces -- quote path names in Makefile and build_detect_platform. | 11 May 2014, 04:01:25 UTC |
| 3edc056 | Igor Canadi | 10 May 2014, 18:25:56 UTC | comment | 10 May 2014, 18:25:56 UTC |
| 038a477 | Igor Canadi | 10 May 2014, 17:49:33 UTC | Make it easier to start using RocksDB Summary: This diff is addressing multiple things with a single goal -- to make RocksDB easier to use: * Add some functions to Options that make RocksDB easier to tune. * Add example code for both simple RocksDB and RocksDB with Column Families. * Rewrite our README.md Regarding Options, I took a stab at something we talked about for a long time: * https://www.facebook.com/groups/rocksdb.dev/permalink/563169950448190/ I added functions: * IncreaseParallelism() -- easy, increases the thread pool and max_background_compactions * OptimizeLevelStyleCompaction(memtable_memory_budget) -- the easiest way to optimize rocksdb for less stalls with level style compaction. This is very likely not ideal configuration. Feel free to suggest improvements. I used some of Mark's suggestions from here: https://github.com/facebook/rocksdb/issues/54 * OptimizeUniversalStyleCompaction(memtable_memory_budget) -- optimize for universal compaction. Test Plan: compiled rocksdb. ran examples. Reviewers: dhruba, MarkCallaghan, haobo, sdong, yhchiang Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18621 | 10 May 2014, 17:49:33 UTC |
| 030db3d | Dallas Marlow | 09 May 2014, 22:58:39 UTC | testing | 09 May 2014, 22:58:39 UTC |
| acd17fd | sdong | 09 May 2014, 17:13:04 UTC | Remove unused variable in DBIter Summary: as title Test Plan: Still compile Reviewers: haobo, igor, yhchiang Reviewed By: igor CC: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18603 | 09 May 2014, 20:20:35 UTC |
| deb8940 | Tyler Neely | 09 May 2014, 18:49:25 UTC | have proprocessor choose correct mmap args | 09 May 2014, 18:49:25 UTC |
| fec4269 | Igor Canadi | 09 May 2014, 15:34:18 UTC | Fix more gflag namespace issues | 09 May 2014, 15:41:02 UTC |
| a1068c9 | Igor Canadi | 09 May 2014, 00:25:13 UTC | Make RocksDB work with newer gflags Summary: Newer gflags switched from `google` namespace to `gflags` namespace. See: https://github.com/facebook/rocksdb/issues/139 and https://github.com/facebook/rocksdb/issues/102 Unfortunately, they don't define any macro with their namespace, so we need to actually try to compile gflags with two different namespace to figure out which one is the correct one. Test Plan: works in fbcode environemnt. I'll also try in ubutnu with newer gflags Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18537 | 09 May 2014, 00:25:13 UTC |
| ddd4114 | sdong | 08 May 2014, 20:32:45 UTC | MergingIterator uses autovector instead of vector Summary: Use autovector in MergingIterator so that if there are 4 or less child iterators in it, iterator wrappers are inline, which is more likely to be cache friendly. Based on one test run with a shadow traffic of one product, it reduces CPU of MergingIterator::Seek() by half. Test Plan: make all check Reviewers: haobo, yhchiang, igor, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D18531 | 08 May 2014, 22:01:20 UTC |
| af7453a | Igor Canadi | 08 May 2014, 20:50:49 UTC | autovector::resize Summary: Resize the autovector! Test Plan: test Reviewers: sdong Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18543 | 08 May 2014, 20:50:49 UTC |
| 8e37a29 | Igor Canadi | 08 May 2014, 20:48:39 UTC | Compaction with zero outputs Summary: We had a hypothesis in https://reviews.facebook.net/D18507 that empty-string internal keys might have been caused by compaction filter deleting all the entries. I added a unit test for that case. Unforutnately, everything works as expected. Test Plan: this is a test Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18519 | 08 May 2014, 20:48:39 UTC |
| 1c6a027 | sdong | 08 May 2014, 04:56:40 UTC | HashLinkedList::Iterator: remove an ununsed class variable Summary: This variable is not used. Remove it. Test Plan: build. Reviewers: haobo, igor, yhchiang Reviewed By: haobo CC: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18525 | 08 May 2014, 20:28:52 UTC |
| f41cde3 | dallas marlow | 08 May 2014, 15:45:44 UTC | remove anon mmap allocation flag `MAP_HUGETLB` | 08 May 2014, 15:45:44 UTC |
| b5616da | Igor Canadi | 08 May 2014, 00:45:27 UTC | Fix iOS compile | 08 May 2014, 00:48:31 UTC |
| 768d424 | Igor Canadi | 07 May 2014, 23:52:12 UTC | [fix] SIGSEGV when VersionEdit in MANIFEST is corrupted Summary: This was reported by our customers in task #4295529. Cause: * MANIFEST file contains a VersionEdit, which contains file entries whose 'smallest' and 'largest' internal keys are empty. String with zero characters. Root cause of corruption was not investigated. We should report corruption when this happens. However, we currently SIGSEGV. Here's what happens: * VersionEdit encodes zero-strings happily and stores them in smallest and largest InternalKeys. InternalKey::Encode() does assert when `rep_.empty()`, but we don't assert in production environemnts. Also, we should never assert as a result of DB corruption. * As part of our ConsistencyCheck, we call GetLiveFilesMetaData() * GetLiveFilesMetadata() calls `file->largest.user_key().ToString()` * user_key() function does: 1. assert(size > 8) (ooops, no assert), 2. returns `Slice(internal_key.data(), internal_key.size() - 8)` * since `internal_key.size()` is unsigned int, this call translates to `Slice(whatever, 1298471928561892576182756)`. Bazinga. Fix: * VersionEdit checks if InternalKey is valid in `VersionEdit::GetInternalKey()`. If it's invalid, returns corruption. Lessons learned: * Always keep in mind that even if you `assert()`, production code will continue execution even if assert fails. * Never `assert` based on DB corruption. Assert only if the code should guarantee that assert can't fail. Test Plan: dumped offending manifest. Before: assert. Now: corruption Reviewers: dhruba, haobo, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18507 | 07 May 2014, 23:52:12 UTC |
| 313b2e5 | Igor Canadi | 07 May 2014, 23:51:30 UTC | Better INSTALL.md and Makefile rules Summary: We have a lot of problems with gflags. However, when compiling rocksdb static library, we don't need gflags dependency. Reorganize INSTALL.md such that first-time customers don't need any dependency installed to actually build rocksdb static library. Test Plan: none Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D18501 | 07 May 2014, 23:51:30 UTC |
| 9efbd85 | sdong | 06 May 2014, 21:51:33 UTC | fsync directory after creating current file in NewDB() Summary: One of our users reported current file corruption. The machine was rebooted during the time. This is the only think I can think of which could cause current file corruption. Just add this paranoid check. Test Plan: make all check Reviewers: haobo, igor Reviewed By: haobo CC: yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18495 | 07 May 2014, 00:51:33 UTC |
| 3a171dc | sdong | 04 May 2014, 22:52:23 UTC | Pass logger to memtable rep and TLB page allocation error logged to info logs Summary: TLB page allocation errors are now logged to info logs, instead of stderr. In order to do that, mem table rep's factory functions take a info logger now. Test Plan: make all check Reviewers: haobo, igor, yhchiang Reviewed By: yhchiang CC: leveldb, yhchiang, dhruba Differential Revision: https://reviews.facebook.net/D18471 | 05 May 2014, 23:43:37 UTC |
| 044af85 | Igor Canadi | 05 May 2014, 21:44:57 UTC | Update HISTORY.md -- release RocksDB 3.0 | 05 May 2014, 21:44:57 UTC |
| 7984b9b | Igor Canadi | 05 May 2014, 21:30:24 UTC | BackupableDBTest thread-safe Summary: We need to lock accesses to some TestEnv variables. Otherwise we get failures like http://ci-builds.fb.com/job/rocksdb_asan_check/657/console Test Plan: make check Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18489 | 05 May 2014, 21:30:24 UTC |
| 15c3991 | Igor Canadi | 05 May 2014, 19:57:47 UTC | Add comment about ValueType | 05 May 2014, 19:57:47 UTC |
| 9e7d00d | Igor Canadi | 05 May 2014, 18:35:40 UTC | Make rocksdb work with all versions of lz4 Summary: There are some projects in fbcode that define lz4 dependency on r108. We, however, defined dependency on r117. That produced some interesting issues and our build system was not happy. This diff makes rocksdb work with both r108 and r117. Hopefully this will fix our problems. Test Plan: compiled rocksdb with both r108 and r117 lz4 Reviewers: dhruba, sdong, haobo Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D18465 | 05 May 2014, 18:35:40 UTC |
| d2569fe | Igor Canadi | 05 May 2014, 18:11:48 UTC | log_and_apply_bench on a new benchmark framework Summary: db_test includes Benchmark for LogAndApply. This diff removes it from db_test and puts it into a separate log_and_apply bench. I just wanted to play around with our new benchmark framework and figure out how it works. I would also like to show you how great it is! I believe right set of microbenchmarks can speed up our productivity a lot and help catch early regressions. Test Plan: no Reviewers: dhruba, haobo, sdong, ljin, yhchiang Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D18261 | 05 May 2014, 18:11:48 UTC |
| 9b17558 | sdong | 05 May 2014, 16:32:56 UTC | PlainTableFactory::PlainTableFactory() to have huge TLB turned off by default Summary: PlainTableFactory::PlainTableFactory() now has Huge TLB page feature turned on by default. Although it is not a public API (which we always turn the feature off now), our unit tests, like db_test sometimes uses it directly, which causes wrong coverage of codes. This patch fix it to allow unit tests to run with the correct setting Test Plan: Run db_test and make sure this feature is not on any more. Reviewers: igor, haobo Reviewed By: igor CC: yhchiang, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D18483 | 05 May 2014, 18:05:54 UTC |
| 6785a52 | Igor Canadi | 05 May 2014, 14:13:48 UTC | Temporary remove perror() calls before we can log from inside of arena | 05 May 2014, 14:13:48 UTC |
| 4a7c747 | sdong | 04 May 2014, 20:55:53 UTC | Revert "Revert "Allow allocating dynamic bloom, plain table indexes and hash linked list from huge page TLB"" And make the default 0 for hash linked list memtable This reverts commit d69dc64be78a8da3ce661454655966d11ff61bb6. | 04 May 2014, 20:56:29 UTC |
| d56959a | Yueh-Hsuan Chiang | 04 May 2014, 20:15:33 UTC | [Java] Use environmental variable JAVA_HOME in Makefile for RocksJava. | 04 May 2014, 20:23:21 UTC |
| db1854d | Igor Canadi | 04 May 2014, 18:39:49 UTC | Declare all DB methods virtual so that StackableDB can override them | 04 May 2014, 18:39:49 UTC |
| d69dc64 | Igor Canadi | 04 May 2014, 15:37:09 UTC | Revert "Allow allocating dynamic bloom, plain table indexes and hash linked list from huge page TLB" This reverts commit 7dafa3a1d7e63d4cc4a7d717ab958efc616a2892. | 04 May 2014, 15:37:09 UTC |
| 41e5cf2 | Benjamin Renard | 03 May 2014, 00:08:55 UTC | Add share_files_with_cheksum option to BackupEngine Summary: added a new option to BackupEngine: if share_files_with_checksum is set to true, sst files are stored in shared_checksum/ and are identified by the triple (file name, checksum, file size) instead of just the file name. This option is targeted at distributed databases that want to backup their primary replica. Test Plan: unit tests and tested backup and restore on a distributed rocksdb Reviewers: igor Reviewed By: igor Differential Revision: https://reviews.facebook.net/D18393 | 03 May 2014, 00:08:55 UTC |
| 77edbfd | Igor Canadi | 02 May 2014, 20:55:33 UTC | link with lz4 Summary: link our fbcode build with lz4 Test Plan: compiles Reviewers: sdong, haobo, dhruba, yhchiang Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D18447 | 02 May 2014, 20:55:33 UTC |
| 4ecfbcf | Igor Canadi | 02 May 2014, 20:24:04 UTC | ApplyToAllCacheEntries Summary: Added a method that executes a callback on every cache entry. Test Plan: added a unit test Reviewers: haobo Reviewed By: haobo CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D18441 | 02 May 2014, 20:24:04 UTC |
| 31d38a6 | Igor Canadi | 02 May 2014, 18:18:55 UTC | Add TARGETS to gitignore | 02 May 2014, 18:18:55 UTC |
| 3b5fa6b | Igor Canadi | 02 May 2014, 14:13:51 UTC | Improve ttl_test Summary: Our valgrind tests are failing because ttl_test is kind of flakey. This diff should fix valgrind issue and make ttl_test less flakey and much faster. Instead of relying on Env::Default() for getting current time, I expose `Env*` to all TTL functions that are interested in time. That way, I can insert a custom test Env which is then used to provide exactly the times we need. That way, we don't need to sleep anymore -- we control the time. Test Plan: ttl_test in normal and valgrind run Reviewers: dhruba, haobo, sdong, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18399 | 02 May 2014, 14:13:51 UTC |
| 82042f4 | Igor Canadi | 01 May 2014, 23:28:23 UTC | Include version in options | 01 May 2014, 23:28:23 UTC |
| d28ed69 | Igor Canadi | 01 May 2014, 19:41:49 UTC | fix release build | 01 May 2014, 19:42:06 UTC |
| d29e48b | Igor Canadi | 01 May 2014, 18:12:35 UTC | fix compile warning | 01 May 2014, 18:12:35 UTC |
| 0afc8bc | Igor Canadi | 01 May 2014, 18:09:32 UTC | xxHash Summary: Originally: https://github.com/facebook/rocksdb/pull/87/files I'm taking over to apply some finishing touches Test Plan: will add tests Reviewers: dhruba, haobo, sdong, yhchiang, ljin Reviewed By: yhchiang CC: leveldb Differential Revision: https://reviews.facebook.net/D18315 | 01 May 2014, 18:09:32 UTC |
| 61955a0 | Yueh-Hsuan Chiang | 01 May 2014, 08:44:46 UTC | [Java] Add RocksObject, the base class of all java objects with a c++ pointer. Summary: Add RocksObject, the base class of all java objects which has a c++ pointer. While the finalizer of a RocksObject will release its c++ resource, it is suggested to call its RocksObject.dispose() to manually release its c++ resource. Existing RocksDB java classes are now extending RocksObject. Test Plan: make rocksdbjava make jtest make jdb_bench Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18411 | 01 May 2014, 08:44:46 UTC |
| 096f5be | Igor Canadi | 30 April 2014, 20:24:52 UTC | Put column family information in LiveFileMetaData Summary: As summary Test Plan: compiles :) Reviewers: dhruba, haobo, sdong, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18405 | 30 April 2014, 20:24:52 UTC |
| a3ab998 | Yueh-Hsuan Chiang | 30 April 2014, 18:54:02 UTC | [Java] Add static method RocksDB.loadLibrary() Summary: Add static method RocksDB.loadLibrary() which loads necessary library files. Test Plan: make rocksdbjava make jtest make jdb_bench java/jdb_bench.sh Reviewers: haobo, dhruba, sdong, ankgup87, rsumbaly, swapnilghike, zzbennett Reviewed By: ankgup87 CC: leveldb Differential Revision: https://reviews.facebook.net/D18375 | 30 April 2014, 18:54:02 UTC |
| 16f1aa7 | Igor Canadi | 30 April 2014, 18:38:01 UTC | Fix signed/unsigned compare | 30 April 2014, 18:38:01 UTC |
| df70047 | Igor Canadi | 30 April 2014, 18:33:40 UTC | Flush stale column families Summary: Added a new option `max_total_wal_size`. Once the total WAL size goes over that, we make an attempt to flush all column families that still have data in the earliest WAL file. By default, I calculate `max_total_wal_size` dynamically, that should be good-enough for non-advanced customers. Test Plan: Added a test Reviewers: dhruba, haobo, sdong, ljin, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18345 | 30 April 2014, 18:33:40 UTC |
| 7dafa3a | sdong | 25 April 2014, 22:45:37 UTC | Allow allocating dynamic bloom, plain table indexes and hash linked list from huge page TLB Summary: Add an option to allocate a piece of memory from huge page TLB. Add options to trigger it in dynamic bloom, plain table indexes andhash linked list hash table. Test Plan: make all check Reviewers: haobo, ljin Reviewed By: haobo CC: nkg-, dhruba, leveldb, igor, yhchiang Differential Revision: https://reviews.facebook.net/D18357 | 30 April 2014, 18:02:26 UTC |
| 66f88c4 | Igor Canadi | 30 April 2014, 15:38:09 UTC | Some fixes as preparation for release | 30 April 2014, 16:03:24 UTC |
| d6d67c0 | Igor Canadi | 29 April 2014, 20:25:23 UTC | More s/us fixes | 30 April 2014, 14:04:36 UTC |
| 2e11e47 | Yueh-Hsuan Chiang | 30 April 2014, 05:05:55 UTC | Merge pull request #131 from ankgup87/master [Java] Add multiget JNI bindings | 30 April 2014, 05:05:55 UTC |
| 046a85a | Igor Canadi | 30 April 2014, 01:26:12 UTC | Merge pull request #132 from kjk/master Improve building of shared library so that it e.g. works with Go bindings | 30 April 2014, 01:26:12 UTC |
| 9d9d296 | Yueh-Hsuan Chiang | 30 April 2014, 00:13:46 UTC | Add a new mem-table representation based on cuckoo hash. Summary: = Major Changes = * Add a new mem-table representation, HashCuckooRep, which is based cuckoo hash. Cuckoo hash uses multiple hash functions. This allows each key to have multiple possible locations in the mem-table. - Put: When insert a key, it will try to find whether one of its possible locations is vacant and store the key. If none of its possible locations are available, then it will kick out a victim key and store at that location. The kicked-out victim key will then be stored at a vacant space of its possible locations or kick-out another victim. In this diff, the kick-out path (known as cuckoo-path) is found using BFS, which guarantees to be the shortest. - Get: Simply tries all possible locations of a key --- this guarantees worst-case constant time complexity. - Time complexity: O(1) for Get, and average O(1) for Put if the fullness of the mem-table is below 80%. - Default using two hash functions, the number of hash functions used by the cuckoo-hash may dynamically increase if it fails to find a short-enough kick-out path. - Currently, HashCuckooRep does not support iteration and snapshots, as our current main purpose of this is to optimize point access. = Minor Changes = * Add IsSnapshotSupported() to DB to indicate whether the current DB supports snapshots. If it returns false, then DB::GetSnapshot() will always return nullptr. Test Plan: Run existing tests. Will develop a test specifically for cuckoo hash in the next diff. Reviewers: sdong, haobo Reviewed By: sdong CC: leveldb, dhruba, igor Differential Revision: https://reviews.facebook.net/D16155 | 30 April 2014, 00:13:46 UTC |
| 2b7cf03 | Krzysztof Kowalczyk | 29 April 2014, 21:29:45 UTC | Update Makefile | 29 April 2014, 21:29:45 UTC |
| f1c9aa6 | Igor Canadi | 29 April 2014, 19:47:48 UTC | More unsigned/signed compare fixes | 29 April 2014, 20:01:06 UTC |
| 38693d9 | Igor Canadi | 29 April 2014, 19:33:57 UTC | Fix more signed/unsigned comparsions | 29 April 2014, 19:40:18 UTC |
| d0b006a | Igor Canadi | 29 April 2014, 19:29:22 UTC | Fix unsigned and signed integer compare | 29 April 2014, 19:29:22 UTC |
| 76c3dbf | Igor Canadi | 29 April 2014, 19:19:54 UTC | Make ttl_test a bit more robust | 29 April 2014, 19:19:54 UTC |
| dd9eb7a | Igor Canadi | 29 April 2014, 17:27:58 UTC | Cache result of ReadFirstRecord() Summary: ReadFirstRecord() reads the actual log file from disk on every call. This diff introduces a cache layer on top of ReadFirstRecord(), which should significantly speed up repeated calls to GetUpdatesSince(). I also cleaned up some stuff, but the whole TransactionLogIterator could use some refactoring, especially if we see increased usage. Test Plan: make check Reviewers: haobo, sdong, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18387 | 29 April 2014, 17:27:58 UTC |
| de825e7 | Igor Canadi | 29 April 2014, 16:17:34 UTC | Create new branch instead of tag in make_new_version | 29 April 2014, 17:22:34 UTC |
| e525bb1 | Igor Canadi | 29 April 2014, 15:59:36 UTC | Make kMajorVersion and kMinorVersion take version from version macros | 29 April 2014, 15:59:48 UTC |
| 6cb0cb3 | Igor Canadi | 29 April 2014, 15:32:56 UTC | Add version.h | 29 April 2014, 15:33:00 UTC |
| d5cb6e7 | Ankit Gupta | 29 April 2014, 15:02:52 UTC | Assert on key size in multiget | 29 April 2014, 15:02:52 UTC |
| 65fba4b | Ankit Gupta | 29 April 2014, 14:53:29 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 29 April 2014, 14:53:29 UTC |
| 91ef2ea | Igor Canadi | 29 April 2014, 03:44:33 UTC | Use new DBWithTTL API in tests | 29 April 2014, 03:46:24 UTC |
| f868dcb | Igor Canadi | 29 April 2014, 03:34:20 UTC | Support for adding TTL-ed column family Summary: This enables user to add a TTL column family to normal DB. Next step should be to expand StackableDB and create StackableColumnFamily, such that users can for example add geo-spatial column families to normal DB. Test Plan: added a test Reviewers: dhruba, haobo, ljin Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18201 | 29 April 2014, 03:34:20 UTC |
| 72ff275 | Igor Canadi | 29 April 2014, 03:30:27 UTC | Fix TransactionLogIterator EOF caching Summary: When TransactionLogIterator comes to EOF, it calls UnmarkEOF and continues reading. However, if glibc cached the EOF status of the file, it will get EOF again, even though the new data might have been written to it. This has been causing errors in Mac OS. Test Plan: test passes, was failing before Reviewers: dhruba, haobo, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18381 | 29 April 2014, 03:30:27 UTC |
| 9895465 | Yueh-Hsuan Chiang | 28 April 2014, 04:46:30 UTC | [Java] Fixed compile error due to the removal of ReadOptions.prefix_seek, minor improvement on DbBenchmark.java. | 28 April 2014, 04:49:45 UTC |
| 7ca06a3 | Igor Canadi | 28 April 2014, 00:17:15 UTC | Merge pull request #130 from donovanhide/master Add rocksdb_open_for_read_only to C API | 28 April 2014, 00:17:15 UTC |
| 0e4e4db | Ankit Gupta | 27 April 2014, 20:05:39 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 27 April 2014, 20:05:39 UTC |
| 4f9fae9 | Donovan Hide | 27 April 2014, 19:57:10 UTC | Add rocksdb_open_for_read_only to C API | 27 April 2014, 19:57:10 UTC |
| fe331c8 | Igor Canadi | 26 April 2014, 21:21:39 UTC | fix valgrind | 26 April 2014, 21:23:15 UTC |
| c489499 | Igor Canadi | 26 April 2014, 21:15:43 UTC | Fix OSX compile | 26 April 2014, 21:15:43 UTC |
| 13b5632 | Ankit Gupta | 26 April 2014, 05:41:04 UTC | Merge branch 'master' of https://github.com/facebook/rocksdb | 26 April 2014, 05:41:04 UTC |
| a515d23 | Ankit Gupta | 26 April 2014, 05:40:16 UTC | Revert makefile | 26 April 2014, 05:40:16 UTC |
| 9e0ab42 | Ankit Gupta | 26 April 2014, 05:39:33 UTC | Fix bug + formatting | 26 April 2014, 05:39:33 UTC |
| 3f8b412 | Ankit Gupta | 26 April 2014, 03:59:16 UTC | Fix formatting | 26 April 2014, 03:59:16 UTC |
| e96e71b | Ankit Gupta | 25 April 2014, 20:57:20 UTC | Multi-get impl and formatting | 25 April 2014, 20:57:20 UTC |
| a618691 | Igor Canadi | 25 April 2014, 19:49:29 UTC | Read-only BackupEngine Summary: Read-only BackupEngine can connect to the same backup directory that is already running BackupEngine. That enables some interesting use-cases (i.e. restoring replica from primary's backup directory) Test Plan: added a unit test Reviewers: dhruba, haobo, ljin Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18297 | 25 April 2014, 19:49:29 UTC |
| ccaca59 | Lei Jin | 25 April 2014, 19:23:07 UTC | avoid calling FindFile twice in TwoLevelIterator for PlainTable Summary: this is to reclaim the regression introduced in https://reviews.facebook.net/D17853 Test Plan: make all check Reviewers: igor, haobo, sdong, dhruba, yhchiang Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17985 | 25 April 2014, 19:23:07 UTC |
| d642c60 | Lei Jin | 25 April 2014, 19:22:23 UTC | Check PrefixMayMatch on Seek() Summary: As a follow-up diff for https://reviews.facebook.net/D17805, add optimization to check PrefixMayMatch on Seek() Test Plan: make all check Reviewers: igor, haobo, sdong, yhchiang, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D17853 | 25 April 2014, 19:22:23 UTC |
| 3995e80 | Lei Jin | 25 April 2014, 19:21:34 UTC | kill ReadOptions.prefix and .prefix_seek Summary: also add an override option total_order_iteration if you want to use full iterator with prefix_extractor Test Plan: make all check Reviewers: igor, haobo, sdong, yhchiang Reviewed By: haobo CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D17805 | 25 April 2014, 19:21:34 UTC |
| 8ce5492 | Igor Canadi | 25 April 2014, 18:58:02 UTC | Delete superversion and log outside of mutex Summary: As summary. Add two autovectors that get filled up in MakeRoomForWrite and they get deleted outside of mutex Test Plan: make check Reviewers: dhruba, haobo, ljin, sdong Reviewed By: ljin CC: leveldb Differential Revision: https://reviews.facebook.net/D18249 | 25 April 2014, 18:58:02 UTC |
| ad3cd39 | Igor Canadi | 25 April 2014, 13:51:16 UTC | Column family logging Summary: Now that we have column families involved, we need to add extra context to every log message. They now start with "[column family name] log message" Also added some logging that I think would be useful, like level summary after every flush (I often needed that when going through the logs). Test Plan: make check + ran db_bench to confirm I'm happy with log output Reviewers: dhruba, haobo, ljin, yhchiang, sdong Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18303 | 25 April 2014, 13:51:16 UTC |
| 46b3076 | Igor Canadi | 25 April 2014, 13:50:51 UTC | Better stack trace in MAC Summary: Now this gives us the real deal stack trace: Assertion failed: (false), function GetProperty, file db/db_impl.cc, line 4072. Received signal 6 (Abort trap: 6) #0 0x7fff57ce39b9 #1 abort (in libsystem_c.dylib) + 125 #2 basename (in libsystem_c.dylib) + 0 #3 rocksdb::DBImpl::GetProperty(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*) (in db_test) (db_impl.cc:4072) #4 rocksdb::_Test_Empty::_Run() (in db_test) (testharness.h:68) #5 rocksdb::_Test_Empty::_RunIt() (in db_test) (db_test.cc:1005) #6 rocksdb::test::RunAllTests() (in db_test) (testharness.cc:60) #7 main (in db_test) (db_test.cc:6697) #8 start (in libdyld.dylib) + 1 Test Plan: added artificial assert, saw great stack trace Reviewers: haobo, dhruba, ljin Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D18309 | 25 April 2014, 13:50:51 UTC |
| a82c492 | Yueh-Hsuan Chiang | 25 April 2014, 05:15:25 UTC | [Java] Fixed some errors in the sample and benchmark code. | 25 April 2014, 05:15:25 UTC |
| e537a84 | Yueh-Hsuan Chiang | 24 April 2014, 23:17:00 UTC | [Java] Fixed a compile error caused in some Java version. | 24 April 2014, 23:17:00 UTC |
