| 91c0148 | Radheshyam Balasundaram | 29 July 2014, 00:14:25 UTC | Minor changes to CuckooTableBuilder Summary: - Copy the key and value to in-memory hash table during Add operation. Also modified cuckoo_table_reader_test to use this. - Store only the user_key in in-memory hash table if it is last level file. - Handle Carryover while chosing unused key in Finish() method in case unused key was never found before Finish() call. Test Plan: cuckoo_table_reader_test --enable_perf cuckoo_table_builder_test valgrind_check asan_check Reviewers: sdong, yhchiang, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20715 | 29 July 2014, 00:14:25 UTC |
| f04356e | sdong | 28 July 2014, 22:28:53 UTC | Add DB::GetIntProperty() to return integer properties to be returned as integers Summary: We have quite some properties that are integers and we are adding more. Add a function to directly return them as an integer, instead of a string Test Plan: Add several unit test checks Reviewers: yhchiang, igor, dhruba, haobo, ljin Reviewed By: ljin Subscribers: yoshinorim, leveldb Differential Revision: https://reviews.facebook.net/D20637 | 28 July 2014, 23:55:57 UTC |
| f678476 | sdong | 28 July 2014, 21:50:16 UTC | Add DB property estimated number of keys Summary: Add a DB property of estimated number of live keys, by adding number of entries of all mem tables and all files, subtracted by all deletions in all files. Test Plan: Add the case in unit tests Reviewers: hobbymanyp, ljin Reviewed By: ljin Subscribers: MarkCallaghan, yoshinorim, leveldb, igor, dhruba Differential Revision: https://reviews.facebook.net/D20631 | 28 July 2014, 22:27:58 UTC |
| 7e8bb71 | Lei Jin | 28 July 2014, 19:27:08 UTC | InternalStats to take cfd on constructor Summary: It has one-to-one relationship with CFD. Take a pointer to CFD on constructor to avoid passing cfd through member functions. Test Plan: make Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20565 | 28 July 2014, 19:27:08 UTC |
| 1bd3431 | Lei Jin | 28 July 2014, 19:22:37 UTC | Change StopWatch interface Summary: So that we can avoid calling NowSecs() in MakeRoomForWrite twice Test Plan: make all check Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20529 | 28 July 2014, 19:22:37 UTC |
| f6ca226 | Lei Jin | 28 July 2014, 19:10:49 UTC | make statistics forward-able Summary: Make StatisticsImpl being able to forward stats to provided statistics implementation. The main purpose is to allow us to collect internal stats in the future even when user supplies custom statistics implementation. It avoids intrumenting 2 sets of stats collection code. One immediate use case is tuning advisor, which needs to collect some internal stats, users may not be interested. Test Plan: ran db_bench and see stats show up at the end of run Will run make all check since some tests rely on statistics Reviewers: yhchiang, sdong, igor Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20145 | 28 July 2014, 19:10:49 UTC |
| 40fa8a4 | Lei Jin | 28 July 2014, 19:05:36 UTC | make statistics forward-able Summary: Make StatisticsImpl being able to forward stats to provided statistics implementation. The main purpose is to allow us to collect internal stats in the future even when user supplies custom statistics implementation. It avoids intrumenting 2 sets of stats collection code. One immediate use case is tuning advisor, which needs to collect some internal stats, users may not be interested. Test Plan: ran db_bench and see stats show up at the end of run Will run make all check since some tests rely on statistics Reviewers: yhchiang, sdong, igor Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20145 | 28 July 2014, 19:05:36 UTC |
| 4a8f0c9 | sdong | 25 July 2014, 21:38:18 UTC | Block::Iter::PrefixSeek() to have an extra check to filter out some false matches Summary: In block based table's hash index checking, when looking for a key that doesn't exist, there is a high chance that a false block is returned because of hash bucket conflicts. In this revision, another check is done to filter out some of those cases: comparing previous key of the block boundary to see whether the target block is what we are looking for. In a favored test setting (bloom filter disabled, 8 L0 files), I saw about 80% improvements. In a non-favored test setting (bloom filter enabled, files are all in L1, files are all cached), I see the performance penalty is less than 3%. Test Plan: make all check Reviewers: haobo, ljin Reviewed By: ljin Subscribers: wuj, leveldb, zagfox, yhchiang Differential Revision: https://reviews.facebook.net/D20595 | 26 July 2014, 00:27:57 UTC |
| 62f9b07 | Radheshyam Balasundaram | 25 July 2014, 23:37:32 UTC | Implementation of CuckooTableReader Summary: Contains: - Implementation of TableReader based on Cuckoo Hashing - Unittests for CuckooTableReader - Performance test for TableReader Test Plan: make cuckoo_table_reader_test ./cuckoo_table_reader_test make valgrind_check make asan_check Reviewers: yhchiang, sdong, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20511 | 25 July 2014, 23:37:32 UTC |
| d650612 | Lei Jin | 25 July 2014, 22:17:06 UTC | expose RateLimiter definition Summary: User gets undefinied error since the definition is not exposed. Also re-enable the db test with only upper bound check Test Plan: db_test, rate_limit_test Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20403 | 25 July 2014, 22:17:06 UTC |
| 28b367d | Igor Canadi | 25 July 2014, 17:29:50 UTC | Initialize next_id | 25 July 2014, 17:29:50 UTC |
| c8e70e6 | Igor Canadi | 25 July 2014, 00:26:48 UTC | Fix valgrind test | 25 July 2014, 00:31:23 UTC |
| 6480717 | Yueh-Hsuan Chiang | 25 July 2014, 00:06:00 UTC | Fixed compaction-related errors where number of input levels are hard-coded. Summary: Fixed compaction-related errors where number of input levels are hard-coded. It's a bug found in compaction branch. This diff will be pushed into master. Test Plan: export ROCKSDB_TESTS=Compact make db_test -j32 ./db_test also passed the tests in compaction branch Reviewers: igor, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20577 | 25 July 2014, 00:06:00 UTC |
| f780f35 | Igor Canadi | 24 July 2014, 20:45:26 UTC | Fix compile warning | 24 July 2014, 20:45:26 UTC |
| 0754d4c | Igor Canadi | 24 July 2014, 20:39:33 UTC | SpatialDB change API Summary: I changed SpatialDB API so that we only specify list of indexes when we create the database. That way, whoever is querying the DB doesn't need to know the full list of indexes and their options. Test Plan: spatial_db_test Reviewers: yinwang Reviewed By: yinwang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20571 | 24 July 2014, 20:39:33 UTC |
| 07a7d87 | Radheshyam Balasundaram | 24 July 2014, 17:07:41 UTC | Addressing TODOs in CuckooTableBuilder Summary: Contains the following changes in CuckooTableBuilder: - Take an extra parameter in constructor to identify last level file. - Implement a better way to identify if a bucket has been inserted into the tree already during BFS search. - Minor typos Test Plan: make cuckoo_table_builder ./cuckoo_table_builder make valgrind_check Reviewers: sdong, igor, yhchiang, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20445 | 24 July 2014, 17:07:41 UTC |
| 4b61a3d | Yueh-Hsuan Chiang | 23 July 2014, 22:49:14 UTC | Merge branch 'ankgup87-master' | 23 July 2014, 22:49:14 UTC |
| 10fc6c7 | Yueh-Hsuan Chiang | 23 July 2014, 22:44:25 UTC | [Java] Add compaction style to options Summary: Add compression type to options make rocksdbjava make sample Reviewers: haobo, yhchiang, sdong, dhruba, rsumbaly, zzbennett, swapnilghike Reviewed By: yhchiang, sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D20463 | 23 July 2014, 22:45:30 UTC |
| 41a6972 | Igor Canadi | 23 July 2014, 20:52:11 UTC | NewIterators in read-only mode Summary: As title. Test Plan: Added test to column_family_test Reviewers: ljin, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20523 | 23 July 2014, 20:52:11 UTC |
| e5f6980 | Igor Canadi | 23 July 2014, 20:49:58 UTC | Fix release compile error | 23 July 2014, 20:50:31 UTC |
| da92745 | Feng Zhu | 23 July 2014, 19:31:11 UTC | Use IterKey instead of string in Block::Iter to reduce malloc Summary: Modify a functioin TrimAppend in dbformat.h: IterKey. Write a test for it in dbformat_test Use IterKey in block::Iter to replace std::string to reduce malloc. Evaluate it using perf record. malloc: 4.26% -> 2.91% free: 3.61% -> 3.08% Test Plan: make all check ./valgrind db_test dbformat_test Reviewers: ljin, haobo, yhchiang, dhruba, igor, sdong Reviewed By: sdong Differential Revision: https://reviews.facebook.net/D20433 | 23 July 2014, 19:31:11 UTC |
| 6296330 | Igor Canadi | 23 July 2014, 18:22:58 UTC | SpatialDB Summary: This diff is adding spatial index support to RocksDB. When creating the DB user specifies a list of spatial indexes. Spatial indexes can cover different areas and have different resolution (i.e. number of tiles). This is useful for supporting different zoom levels. Each element inserted into SpatialDB has: * a bounding box, which determines how will the element be indexed * string blob, which will usually be WKB representation of the polygon (http://en.wikipedia.org/wiki/Well-known_text) * feature set, which is a map of key-value pairs, where value can be int, double, bool, null or a string. FeatureSet will be a set of tags associated with geo elements (for example, 'road': 'highway' and similar) * a list of indexes to insert the element in. For example, small river element will be inserted in index for high zoom level, while country border will be inserted in all indexes (including the index for low zoom level). Each query is executed on single spatial index. Query guarantees that it will return all elements intersecting the specified bounding box, but it might also return some extra non-intersecting elements. Test Plan: Added bunch of unit tests in spatial_db_test Reviewers: dhruba, yinwang Reviewed By: yinwang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20361 | 23 July 2014, 18:22:58 UTC |
| b5c4c0b | Yueh-Hsuan Chiang | 23 July 2014, 18:16:18 UTC | [Java] Add the missing ROCKSDB_JAR variable in Makefile Summary: Add the missing ROCKSDB_JAR variable in Makefile, which is mistakenly removed in https://reviews.facebook.net/D20289. Test Plan: export ROCKSDB_JAR= make rocksdbjava | 23 July 2014, 18:16:18 UTC |
| 6e7e3e4 | Yueh-Hsuan Chiang | 23 July 2014, 18:15:14 UTC | [Java] Update header inclusion of utilities files Summary: Update header inclusion as include/utilities/*.h has been moved to include/rocksdb/utilities/*.h Test Plan: make rocksdbjava Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20517 | 23 July 2014, 18:15:14 UTC |
| 00f56df | Yueh-Hsuan Chiang | 23 July 2014, 17:31:32 UTC | Fixed a compile error in util/options_builder.cc Summary: Fixed the following compile error by replacing pow by shift, as it computes power of 2. util/options_builder.cc:133:14: error: no member named 'pow' in namespace 'std' std::pow(2, std::max(0, std::min(3, level0_stop_writes_trigger - ~~~~~^ 1 error generated. make: *** [util/options_builder.o] Error 1 Test Plan: make success in mac and linux Reviewers: ljin, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20475 | 23 July 2014, 17:31:32 UTC |
| 0e1b478 | Yueh-Hsuan Chiang | 23 July 2014, 17:22:21 UTC | Fixed a bug in Compaction.cc where input_levels_ was not properly resized. Summary: Fixed a bug in Compaction.cc where input_levels_ was not properly resized. Without this fix, there would be invalid access in input_levels_ when more than two levels are involved in one compaction run. This fix will go to master instead of compaction branch. Test Plan: tested in compaction branch. Reviewers: ljin, sdong, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20481 | 23 July 2014, 17:22:21 UTC |
| f82d4a2 | Igor Canadi | 23 July 2014, 14:28:41 UTC | Also bump version in Makefile | 23 July 2014, 14:28:41 UTC |
| 1053358 | Igor Canadi | 23 July 2014, 14:21:54 UTC | Bump the version | 23 July 2014, 14:21:54 UTC |
| 0ff183a | Igor Canadi | 23 July 2014, 14:21:38 UTC | Move include/utilities/*.h to include/rocksdb/utilities/*.h Summary: All public headers need to be under `include/rocksdb` directory. Otherwise, clients include our header files like this: #include <rocksdb/db.h> #include <utilities/backupable_db.h> // still our public header! Also, internally, we include: #include "utilities/backupable/backupable_db.h" // internal header #include "utilities/backupable_db.h" // public header which is confusing. This way, when we install rocksdb as a system library, we can just copy `include/rocksdb` directory to system's header files. We can't really copy `utilities` directory to system's header files. Test Plan: compiles Reviewers: dhruba, ljin, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20409 | 23 July 2014, 14:21:38 UTC |
| e6de021 | sdong | 16 May 2014, 17:35:41 UTC | Add a utility function to guess optimized options based on constraints Summary: Add a function GetOptions(), where based on four parameters users give: read/write amplification threshold, memory budget for mem tables and target DB size, it picks up a compaction style and parameters for them. Background threads are not touched yet. One limit of this algorithm: since compression rate and key/value size are hard to predict, it's hard to predict level 0 file size from write buffer size. Simply make 1:1 ratio here. Sample results: https://reviews.facebook.net/P477 Test Plan: Will add some a unit test where some sample scenarios are given and see they pick the results that make sense Reviewers: yhchiang, dhruba, haobo, igor, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18741 | 22 July 2014, 22:24:21 UTC |
| 250f035 | Yueh-Hsuan Chiang | 22 July 2014, 21:53:12 UTC | [Java] Optimize statistics collector, improve object dependency in RocksObjects Summary: This diff merges pull request 208. Contributor: ankgup87 [Java] Optimize statistics collector * Optimize statistics collector by collecting statistics of multiple DBs in a single thread rather than starting up a new thread for each DB. * Also, fix packaging of jnilib file on OS_X platform. * Diff review: https://reviews.facebook.net/D20265 [Java] Add documentation on interdependency of dispose call of RocksObjects * Remove transferCppRawPointersOwnershipFrom function. - This function was setting opt.filter_ and thus filter_ to be null. This way there is no one holding reference for filter object and can thus be GC'd which is not the intention. Replaced it with storeOptionsInstace which stores options instance. Options class internally holds Filter instance. Thus when Options is GC'd, filter reference will be GC'd automatically. * Added documentation explaining interdependency of Filter, Options and DB. * Diff review: https://reviews.facebook.net/D20379 Test Plan: described in their diff reviews Reviewers: haobo sdong swapnilghike zzbennett rsumbaly yhchiang Reviewed by: yhchiang | 22 July 2014, 21:53:12 UTC |
| 25682d1 | Ankit Gupta | 10 July 2014, 06:13:28 UTC | [Java] Optimize statistics collector, improve object dependency in RocksObjects Summary: This diff merges pull request #208. Contributor: ankgup87 [Java] Optimize statistics collector * Optimize statistics collector by collecting statistics of multiple DBs in a single thread rather than starting up a new thread for each DB. * Also, fix packaging of jnilib file on OS_X platform. * Diff review: https://reviews.facebook.net/D20265 [Java] Add documentation on interdependency of dispose call of RocksObjects * Remove transferCppRawPointersOwnershipFrom function. - This function was setting opt.filter_ and thus filter_ to be null. This way there is no one holding reference for filter object and can thus be GC'd which is not the intention. Replaced it with storeOptionsInstace which stores options instance. Options class internally holds Filter instance. Thus when Options is GC'd, filter reference will be GC'd automatically. * Added documentation explaining interdependency of Filter, Options and DB. * Diff review: https://reviews.facebook.net/D20379 Test Plan: described in their diff reviews Reviewers: haobo sdong swapnilghike zzbennett rsumbaly yhchiang Reviewed by: yhchiang | 22 July 2014, 21:52:50 UTC |
| 2d3d635 | Igor Canadi | 22 July 2014, 19:35:07 UTC | Fix signed-unsigned compare error | 22 July 2014, 19:35:07 UTC |
| f6272e3 | Radheshyam Balasundaram | 22 July 2014, 16:49:04 UTC | Fixing memory leaks in cuckoo_table_builder_test Summary: Fixes some memory leaks in cuckoo_builder_test.cc. This also fixed broken valgrind_check tests Test Plan: make valgrind_check ./cuckoo_builder_test Currently running make check all. I shall update once it is done. Reviewers: ljin, sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20385 | 22 July 2014, 16:49:04 UTC |
| d19aa25 | Yueh-Hsuan Chiang | 22 July 2014, 16:48:37 UTC | [Java] Add Java support for cache sharding. Summary: Add setCacheNumShardBits() and cacheNumShardBits() to Options. This allows developers to control the number of shards for the block cache. Test Plan: make rocksdbjava cd java make db_bench ./jdb_bench.sh --cache_size=1048576 --cache_numshardbits=6 Reviewers: sdong, ljin, ankgup87 Reviewed By: ankgup87 Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19347 | 22 July 2014, 16:48:37 UTC |
| ae7743f | Yueh-Hsuan Chiang | 22 July 2014, 05:41:54 UTC | Fixed some make and linking issues of RocksDBJava Summary: Fixed some make and linking issues of RocksDBJava. Specifically: * Add JAVA_LDFLAGS, which does not include gflags * rocksdbjava library now uses JAVA_LDFLAGS instead of LDFLAGS * java/Makefile now includes build_config.mk * rearrange make rocksdbjava workflow to ensure the library file is correctly included in the jar file. Test Plan: make rocksdbjava make jdb_bench java/jdb_bench.sh Reviewers: dhruba, swapnilghike, zzbennett, rsumbaly, ankgup87 Reviewed By: ankgup87 Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20289 | 22 July 2014, 05:41:54 UTC |
| f6b7e1e | sdong | 17 July 2014, 00:39:18 UTC | Allow user to specify DB path of output file of manual compaction Summary: Add a parameter path_id to DB::CompactRange(), to indicate where the output file should be placed to. Test Plan: add a unit test Reviewers: yhchiang, ljin Reviewed By: ljin Subscribers: xjin, igor, dhruba, MarkCallaghan, leveldb Differential Revision: https://reviews.facebook.net/D20085 | 22 July 2014, 02:06:00 UTC |
| bbe2e91 | Yueh-Hsuan Chiang | 21 July 2014, 22:17:09 UTC | Fixed a compile error of cuckoo_table_builder. Summary: Fixed the following compile error. ./table/cuckoo_table_builder.h:72:22: error: private field 'key_length_' is not used [-Werror,-Wunused-private-field] const unsigned int key_length_; ^ 1 error generated. Test Plan: make Reviewers: sdong, ljin, radheshyamb, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20349 | 21 July 2014, 22:17:09 UTC |
| cf3da89 | Radheshyam Balasundaram | 21 July 2014, 20:26:09 UTC | Adding a new SST table builder based on Cuckoo Hashing Summary: Cuckoo Hashing based SST table builder. Contains: - Cuckoo Hashing logic and file storage logic. - Unit tests for logic Test Plan: make cuckoo_table_builder_test ./cuckoo_table_builder_test make check all Reviewers: yhchiang, igor, sdong, ljin Reviewed By: ljin Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19545 | 21 July 2014, 20:26:09 UTC |
| f6f1533 | Lei Jin | 21 July 2014, 19:57:29 UTC | make internal stats independent of statistics Summary: also make it aware of column family output from db_bench ``` ** Compaction Stats [default] ** Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14 956 0.9 0.0 0.0 0.0 2.7 2.7 0.0 0.0 0.0 111.6 0 0 0 0 24 40 0.612 75.20 492387 0.15 L1 21 2001 2.0 5.7 2.0 3.7 5.3 1.6 5.4 2.6 71.2 65.7 31 43 55 12 82 2 41.242 43.72 41183 1.06 L2 217 18974 1.9 16.5 2.0 14.4 15.1 0.7 15.6 7.4 70.1 64.3 17 182 185 3 241 16 15.052 0.00 0 0.00 L3 1641 188245 1.8 9.1 1.1 8.0 8.5 0.5 15.4 7.4 61.3 57.2 9 75 76 1 152 9 16.887 0.00 0 0.00 L4 4447 449025 0.4 13.4 4.8 8.6 9.1 0.5 4.7 1.9 77.8 52.7 38 79 100 21 176 38 4.639 0.00 0 0.00 Sum 6340 659201 0.0 44.7 10.0 34.7 40.6 6.0 32.0 15.2 67.7 61.6 95 379 416 37 676 105 6.439 118.91 533570 0.22 Int 0 0 0.0 1.2 0.4 0.8 1.3 0.5 5.2 2.7 59.1 65.6 3 7 9 2 20 10 2.003 0.00 0 0.00 Stalls(secs): 75.197 level0_slowdown, 0.000 level0_numfiles, 0.000 memtable_compaction, 43.717 leveln_slowdown Stalls(count): 492387 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 41183 leveln_slowdown ** DB Stats ** Uptime(secs): 202.1 total, 13.5 interval Cumulative writes: 6291456 writes, 6291456 batches, 1.0 writes per batch, 4.90 ingest GB Cumulative WAL: 6291456 writes, 6291456 syncs, 1.00 writes per sync, 4.90 GB written Interval writes: 1048576 writes, 1048576 batches, 1.0 writes per batch, 836.0 ingest MB Interval WAL: 1048576 writes, 1048576 syncs, 1.00 writes per sync, 0.82 MB written Test Plan: ran it Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19917 | 21 July 2014, 19:57:29 UTC |
| 50c2dcb | Feng Zhu | 21 July 2014, 19:01:40 UTC | add options.block_restart_interval in db_bench Summary: Add block_restart_interval in db_bench, default value 16 Test Plan: make Reviewers: sdong Reviewed By: sdong Differential Revision: https://reviews.facebook.net/D20331 | 21 July 2014, 19:01:40 UTC |
| 9c0d84d | Lei Jin | 21 July 2014, 18:03:16 UTC | improve comments for CrateRateLimiter() Summary: Suggested by @dhruba from the other diff, here is the improved comments for parameters of the function Test Plan: none Reviewers: dhruba, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19623 | 21 July 2014, 18:03:16 UTC |
| c1a90b0 | Stanislau Hlebik | 21 July 2014, 17:31:33 UTC | Fix db_bench Summary: Adding check for zero size index Test Plan: ./build_tools/regression_build_test.sh Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20259 | 21 July 2014, 17:31:33 UTC |
| 80a94d0 | Yueh-Hsuan Chiang | 21 July 2014, 16:19:28 UTC | [Java] Fixed the Java compilation error caused by PlainTableFactory API change. Summary: Fixed the Java compilation error caused by PlainTableFactory API change. Test Plan: make rocksdbjava make jdb_bench cd java;./jdb_bench.sh Reviewers: sdong, ljin, ankgup87, swapnilghike, zzbennett, rsumbaly, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20295 | 21 July 2014, 16:19:28 UTC |
| 3769370 | Igor Canadi | 20 July 2014, 16:08:53 UTC | Merge pull request #207 from Chilledheart/develop Fix clang compiler warnings | 20 July 2014, 16:08:53 UTC |
| 54f4e2f | Chilledheart | 20 July 2014, 14:56:40 UTC | Fix clang compiler warnings | 20 July 2014, 14:57:20 UTC |
| f4c49ae | Stanislau Hlebik | 19 July 2014, 00:23:19 UTC | Update HISTORY.md Summary: Updating HISTORY.md to reflect changes in PlainTable index Test Plan: no need Reviewers: sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20229 | 19 July 2014, 00:23:19 UTC |
| 9d70cce | Stanislau Hlebik | 18 July 2014, 23:58:13 UTC | Adding option to save PlainTable index and bloom filter in SST file. Summary: Adding option to save PlainTable index and bloom filter in SST file. If there is no bloom block and/or index block, PlainTableReader builds new ones. Otherwise PlainTableReader just use these blocks. Test Plan: make all check Reviewers: sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19527 | 18 July 2014, 23:58:13 UTC |
| 92d73cb | Stanislau Hlebik | 18 July 2014, 07:08:38 UTC | Add PlainTableOptions Summary: Since we have a lot of options for PlainTable, add a struct PlainTableOptions to manage them Test Plan: make all check Reviewers: sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20175 | 18 July 2014, 07:08:38 UTC |
| 052ddbe | Yueh-Hsuan Chiang | 18 July 2014, 01:01:04 UTC | Add MaxInputLevel() to CompactionPicker Summary: Having if-then branch for different compaction strategies is considered hacky and make CompactionPicker less pluggable. This diff removes two of such if-then branches in version_set.cc by adding MaxInputLevel() to CompactionPicker. // Given the current number of levels, returns the lowest allowed level // for compaction input. virtual int MaxInputLevel(int current_num_levels) const; Test Plan: make db_test export ROCKSDB_TESTS=Compaction ./db_test Reviewers: igor, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19971 | 18 July 2014, 01:01:04 UTC |
| 2f289dc | Yueh-Hsuan Chiang | 18 July 2014, 00:26:12 UTC | Add -Wsign-compare to WARNING_FLAGS in Makefile Summary: Add -Wsign-compare to WARNING_FLAGS in Makefile as not all g++ compiler include -Wsign-compare in -Wall when compiling '.h' file. Test Plan: make -j32 Reviewers: ljin, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D20169 | 18 July 2014, 00:26:12 UTC |
| aac941b | Yueh-Hsuan Chiang | 17 July 2014, 23:37:25 UTC | Fixed a signed and unsigned comparison in Compaction Summary: Fixed a signed and unsigned comparison in Compaction Test Plan: make db_test export ROCKSDB_TESTS=Compaction ./db_test | 17 July 2014, 23:37:25 UTC |
| 0d57e3a | Radheshyam Balasundaram | 17 July 2014, 22:07:05 UTC | Guarding files_ attribute with #ifndef NDEBUG guard in FilePicker class. Summary: Adding guards to files_ attribute of FilePicker class. This attribute is used only in DEBUG mode. This fixes build of static_lib in mac. Test Plan: make static_lib in mac make check all in devserver Reviewers: ljin, igor, sdong Reviewed By: sdong Differential Revision: https://reviews.facebook.net/D20163 | 17 July 2014, 22:07:05 UTC |
| 3178510 | Yueh-Hsuan Chiang | 17 July 2014, 21:36:41 UTC | Allow class Compaction to handle input files from multiple levels. Summary: Allow class Compaction to handle input files from multiple levels. This diff is a subset of https://reviews.facebook.net/D19263 where only db/compaction.cc and db/compaction.h are changed. Test Plan: make db_test export ROCKSDB_TESTS=Compaction ./db_test Reviewers: igor, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19923 | 17 July 2014, 21:36:41 UTC |
| 296e340 | Yueh-Hsuan Chiang | 17 July 2014, 01:12:17 UTC | Add struct CompactionInputFiles to manage compaction input files. Summary: Add struct CompactionInputFiles to manage compaction input files. Test Plan: export ROCKSDB_TESTS=Compact make db_test ./db_test Reviewers: ljin, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20061 | 17 July 2014, 01:12:17 UTC |
| bc6b2ab | Feng Zhu | 17 July 2014, 00:32:30 UTC | enable kHashSearch for blocktable in db_bench Summary: add a flag called use_hash_search in db_bench Test Plan: make all check ./db_bench --use_hash_search=1 Reviewers: ljin, haobo, yhchiang, sdong Reviewed By: sdong Subscribers: igor, dhruba Differential Revision: https://reviews.facebook.net/D20067 | 17 July 2014, 00:32:30 UTC |
| 87895c6 | Feng Zhu | 16 July 2014, 23:56:49 UTC | fix bug in LOG for flush memtable Summary: One line change to fix a bug in the LOG when flush memtable Test Plan: NONE Reviewers: sdong Reviewed By: sdong Differential Revision: https://reviews.facebook.net/D20049 | 16 July 2014, 23:56:49 UTC |
| 1c9f190 | Stanislau Hlebik | 16 July 2014, 21:51:43 UTC | Fix db_test Summary: Added deletion of DBIterators in DBIterator's tests Test Plan: make valgrind_check Reviewers: igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D20043 | 16 July 2014, 21:51:43 UTC |
| 0418e66 | Radheshyam Balasundaram | 16 July 2014, 20:33:02 UTC | Refactoring Version::Get() Summary: Refactoring Version::Get() method to move file picker logic to a separate class. Test Plan: make check all Reviewers: igor, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19713 | 16 July 2014, 20:33:02 UTC |
| c11d604 | Feng Zhu | 16 July 2014, 18:21:30 UTC | store file_indexer info in sequential memory Summary: use arena to allocate space for next_level_index_ and level_rb_ Thus increasing data locality and make Version::Get faster. Benchmark detail Base version: commit d2a727c182338514af955bbcb1d92db4af83b41c command used: ./db_bench --db=/mnt/db/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=2097152 --max_bytes_for_level_base=1073741824 --disable_wal=0 --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --benchmarks=fillseq, readrandom,readrandom,readrandom --use_existing_db=0 --num=52428800 --threads=1 Result: cpu running percentage: Version::Get, improved from 7.98% to 7.42% FileIndexer::GetNextLevelIndex, improved from 1.18% to 0.68%. Test Plan: make all check Reviewers: ljin, haobo, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, igor Differential Revision: https://reviews.facebook.net/D19845 | 16 July 2014, 18:21:30 UTC |
| 1614284 | Igor Canadi | 16 July 2014, 13:45:49 UTC | Fix compressed cache | 16 July 2014, 13:45:49 UTC |
| d916593 | Stanislau Hlebik | 15 July 2014, 23:10:18 UTC | Add Prev() for merge operator Summary: Implement Prev() with merge operator for DBIterator. Request from mongoDB. Task 4673663. Test Plan: make all check Reviewers: sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19743 | 15 July 2014, 23:10:18 UTC |
| 0abaed2 | sdong | 14 July 2014, 22:34:30 UTC | Support multiple DB directories in universal compaction style Summary: This patch adds a target size parameter in options.db_paths and universal compaction will base it to determine which DB path to place a new file. Level-style stays the same. Test Plan: Add new unit tests Reviewers: ljin, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, dhruba, igor, leveldb Differential Revision: https://reviews.facebook.net/D19869 | 15 July 2014, 19:06:28 UTC |
| 20c0563 | Igor Canadi | 15 July 2014, 13:16:32 UTC | Remove stats logger Summary: Browsing through the code, looks like StatsLogger is not used at all! Test Plan: compiles Reviewers: ljin, sdong, yhchiang, dhruba Reviewed By: dhruba Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19827 | 15 July 2014, 13:16:32 UTC |
| dd6c444 | sdong | 14 July 2014, 19:06:22 UTC | Improve Put()'s comment to indicate that the key is overwritten if existing Summary: As title Test Plan: Not needed for comment only. Reviewers: yhchiang, ljin, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: xjin, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19887 | 14 July 2014, 21:43:59 UTC |
| d2a727c | Igor Canadi | 14 July 2014, 16:06:38 UTC | BG -> GB | 14 July 2014, 16:06:38 UTC |
| 591c2a3 | Igor Canadi | 14 July 2014, 14:56:07 UTC | [db stress] Don't drop column families if there's only 1 | 14 July 2014, 14:56:07 UTC |
| ee6b35e | Igor Canadi | 14 July 2014, 13:56:52 UTC | Fix mac compile Summary: We should use PRIu64 instead of "%lu" for portability Test Plan: compiles now Reviewers: ljin, dhruba Reviewed By: dhruba Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19809 | 14 July 2014, 13:56:52 UTC |
| 46f0f6d | Lei Jin | 11 July 2014, 22:03:30 UTC | improve InternalStats output Summary: as title Test Plan: sampe output: Level Files Size(MB) Score Read(GB) Rn(GB) Rnp1(GB) Write(BG) Wnew(GB) RW-Amp W-Amp Rd(MB/s) Wr(MB/s) Rn(cnt) Rnp1(cnt) Wnp1(cnt) Wnew(cnt) Comp(sec) Comp(cnt) Avg(sec) Stall(sec) Stall(cnt) Avg(ms) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15 1024 1.0 0.0 0.0 0.0 8.2 8.2 0.0 0.0 0.0 111.4 0 0 1 1 75 123 0.612 295.94 1939238 0.15 L1 23 2118 2.1 20.9 8.3 12.7 20.0 7.3 5.0 2.4 73.2 69.9 124 141 208 67 293 8 36.582 17.05 16100 1.06 L2 162 15333 1.5 47.0 7.1 40.0 42.6 2.6 12.7 6.0 67.9 61.5 62 457 482 25 709 55 12.898 0.00 0 0.00 L3 985 108065 1.1 37.8 4.0 33.9 36.9 3.0 18.8 9.3 60.1 58.5 41 338 363 25 645 31 20.812 0.00 0 0.00 L4 2788 356033 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 0.000 0.00 0 0.00 Sum 3973 482572 0.0 105.8 19.3 86.5 107.7 21.2 11.1 5.6 62.9 64.0 227 936 1054 118 1723 217 7.938 312.99 1955338 0.16 Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19707 | 11 July 2014, 22:03:30 UTC |
| 178fd6f | Feng Zhu | 11 July 2014, 19:52:41 UTC | use FileLevel in LevelFileNumIterator Summary: Use FileLevel in LevelFileNumIterator, thus use new version of findFile. Old version of findFile function is deleted. Write a function in version_set.cc to generate FileLevel from files_. Add GenerateFileLevelTest in version_set_test.cc Test Plan: make all check Reviewers: ljin, haobo, yhchiang, sdong Reviewed By: sdong Subscribers: igor, dhruba Differential Revision: https://reviews.facebook.net/D19659 | 11 July 2014, 19:52:41 UTC |
| 3b97ee9 | Tomislav Novak | 10 July 2014, 22:14:24 UTC | ForwardIterator seek bugfix Summary: If `NeedToSeekImmutable()` returns false, `SeekInternal()` won't reset the contents of `immutable_min_heap_`. However, since it calls `UpdateCurrent()` unconditionally, if `current_` is one of immutable iterators (previously popped from `immutable_min_heap_`), `UpdateCurrent()` will overwrite it. As a result, if old `current_` in fact pointed to the smallest entry, forward iterator will skip some records. Fix implemented in this diff pushes `current_` back to `immutable_min_heap_` before calling `UpdateCurrent()`. Test Plan: New unit test (courtesy of @lovro): $ ROCKSDB_TESTS=TailingIteratorSeekToSame ./db_test Reviewers: igor, dhruba, haobo, ljin Reviewed By: ljin Subscribers: lovro, leveldb Differential Revision: https://reviews.facebook.net/D19653 | 10 July 2014, 23:46:13 UTC |
| a51fbf5 | Yueh-Hsuan Chiang | 10 July 2014, 20:35:04 UTC | Update HISTORY.md for release 3.3 Summary: Update HISTORY.md for release 3.3 Test Plan: n/a Reviewers: sdong, ljin, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19635 | 10 July 2014, 20:35:04 UTC |
| 3f53d27 | Igor Canadi | 10 July 2014, 20:29:40 UTC | Merge pull request #200 from rdallman/missing_cf C API: create missing cf's, cleanup | 10 July 2014, 20:29:40 UTC |
| 5ff6633 | Igor Canadi | 10 July 2014, 20:19:28 UTC | Fix mac compile | 10 July 2014, 20:19:47 UTC |
| 1fc71a4 | Reed Allman | 10 July 2014, 19:53:46 UTC | C API: create missing cf's, cleanup | 10 July 2014, 19:55:53 UTC |
| 105c1e0 | Tomislav Novak | 10 July 2014, 00:46:18 UTC | ForwardIterator::status() checks all child iterators Summary: Forward iterator only checked `status_` and `mutable_iter_->status()`, which is not sufficient. For example, when reading exclusively from cache (kBlockCacheTier), `mutable_iter_->status()` may return kOk (e.g. there's nothing in the memtable), but one of immutable iterators could be in kIncomplete. In this case, `ForwardIterator::status()` ought to return that status instead of kOk. This diff changes `status()` to also check `imm_iters_`, `l0_iters_`, and `level_iters_`. Test Plan: ROCKSDB_TESTS=TailingIteratorIncomplete ./db_test Reviewers: ljin, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19581 | 10 July 2014, 19:43:12 UTC |
| 01700b6 | sdong | 10 July 2014, 18:37:53 UTC | Update master to version 3.3 Summary: As tittle Test Plan: no need Reviewers: igor, yhchiang, ljin Reviewed By: ljin Subscribers: haobo, dhruba, xjin, leveldb Differential Revision: https://reviews.facebook.net/D19629 | 10 July 2014, 18:59:35 UTC |
| 36de0e5 | sdong | 10 July 2014, 18:35:07 UTC | Add a function to return current perf level Summary: Add a function to return the perf level. It is to allow a wrapper of DB to increase the perf level and restore the original perf level after finishing the function call. Test Plan: Add a verification in db_test Reviewers: yhchiang, igor, ljin Reviewed By: ljin Subscribers: xjin, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D19551 | 10 July 2014, 18:35:48 UTC |
| 30c81e7 | Stanislau Hlebik | 10 July 2014, 18:32:04 UTC | Removing NewTotalOrderPlainTableFactory Summary: Seems like NewTotalOrderPlainTableFactory is useless and is semantically incorrect. Total order mode indicator is prefix_extractor == nullptr, but NewTotalOrderPlainTableFactory doesn't set it to be nullptr. That's why some tests in plain_table_db_tests is incorrect. Test Plan: make all check Reviewers: sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19587 | 10 July 2014, 18:32:04 UTC |
| 536f4b3 | Igor Canadi | 10 July 2014, 16:52:46 UTC | Add missing DB functions to DocumentDB | 10 July 2014, 16:52:46 UTC |
| f0a8be2 | Igor Canadi | 10 July 2014, 16:31:42 UTC | JSON (Document) API sketch Summary: This is a rough sketch of our new document API. Would like to get some thoughts and comments about the high-level architecture and API. I didn't optimize for performance at all. Leaving some low-hanging fruit so that we can be happy when we fix them! :) Currently, bunch of features are not supported at all. Indexes can be only specified when creating database. There is no query planner whatsoever. This will all be added in due time. Test Plan: Added a simple unit test Reviewers: haobo, yhchiang, dhruba, sdong, ljin Reviewed By: ljin Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18747 | 10 July 2014, 16:31:42 UTC |
| 222cf25 | Feng Zhu | 10 July 2014, 06:40:03 UTC | change the init parameter for FileDescriptor Summary: fix a bug in improve_file_key_search, change the parameter for FileDescriptor Test Plan: make all check Reviewers: sdong Reviewed By: sdong Differential Revision: https://reviews.facebook.net/D19611 | 10 July 2014, 06:40:03 UTC |
| 8a7d1fe | Lei Jin | 10 July 2014, 05:46:15 UTC | disable rate limiter test Summary: The test is not stable because it relies on disk and only runs for a short period of time. So misisng a compaction/flush would greatly affect the rate. I am disabling it for now. What do you guys think? Test Plan: make Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19599 | 10 July 2014, 05:46:15 UTC |
| f697cad | Feng Zhu | 10 July 2014, 05:14:39 UTC | create compressed_levels_ in Version, allocate its space using arena. Make Version::Get, Version::FindFile faster Summary: Define CompressedFileMetaData that just contains fd, smallest_slice, largest_slice. Create compressed_levels_ in Version, the space is allocated using arena Thus increase the file meta data locality, speed up "Get" and "FindFile" benchmark with in-memory tmpfs, could have 4% improvement under "random read" and 2% improvement under "read while writing" benchmark command: ./db_bench --db=/mnt/db/rocksdb --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --block_size=4096 --cache_size=17179869184 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=33554432 --max_bytes_for_level_base=1073741824 --disable_wal=0 --sync=0 --disable_data_sync=1 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_grandparent_overlap_factor=10 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --perf_level=0 --benchmarks=readwhilewriting,readwhilewriting,readwhilewriting --use_existing_db=1 --num=52428800 --threads=1 —writes_per_second=81920 Read Random: From 1.8363 ms/op, improve to 1.7587 ms/op. Read while writing: From 2.985 ms/op, improve to 2.924 ms/op. Test Plan: make all check Reviewers: ljin, haobo, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, igor Differential Revision: https://reviews.facebook.net/D19419 | 10 July 2014, 05:14:39 UTC |
| cae316d | Yueh-Hsuan Chiang | 09 July 2014, 20:54:11 UTC | Merge pull request #195 from ankgup87/master [Java] Package .so file with JNI jar | 09 July 2014, 20:54:11 UTC |
| 7082855 | Yueh-Hsuan Chiang | 09 July 2014, 19:46:08 UTC | Some fixes on size compensation logic for deletion entry in compaction Summary: This patch include two fixes: 1. newly created Version will now takes the aggregated stats for average-value-size from the latest Version. 2. compensated size of a file is now computed only for newly created / loaded file, this addresses the issue where files are already sorted by their compensated file size but might sometimes observe some out-of-order due to later update on compensated file size. Test Plan: export ROCKSDB_TESTS=CompactionDele ./db_test Reviewers: ljin, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19557 | 09 July 2014, 19:46:08 UTC |
| 0a4d930 | Ankit Gupta | 09 July 2014, 16:09:08 UTC | Caching methodId and fieldId is fine: v2 | 09 July 2014, 16:09:08 UTC |
| b6caaea | Ankit Gupta | 09 July 2014, 16:06:40 UTC | Caching methodId and fieldId is fine | 09 July 2014, 16:06:40 UTC |
| 21e5226 | Ankit Gupta | 09 July 2014, 16:06:20 UTC | Revert wrong commit | 09 July 2014, 16:06:20 UTC |
| 05bd545 | Ankit Gupta | 09 July 2014, 16:02:05 UTC | Caching methodId and fieldId is fine | 09 July 2014, 16:02:05 UTC |
| ef1aad9 | Lei Jin | 08 July 2014, 22:29:13 UTC | fix one more internal_stats issue Summary: stall count is wrong Test Plan: make release Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19539 | 08 July 2014, 22:29:13 UTC |
| 73d7147 | Lei Jin | 08 July 2014, 22:15:00 UTC | make rate limiter test more reliable Summary: Randomize keys so that compaction actually happens. Change the config so that compaction happens more aggressively. The test takes longer time, but the results are more stable shown by iostat Test Plan: ran it Reviewers: igor, yhchiang Reviewed By: yhchiang Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19533 | 08 July 2014, 22:15:00 UTC |
| 8a9cc78 | Lei Jin | 08 July 2014, 19:48:10 UTC | report correct interval amplification Summary: as title Test Plan: make release Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19515 | 08 July 2014, 19:48:10 UTC |
| 534357c | Lei Jin | 08 July 2014, 19:31:49 UTC | integrate rate limiter into rocksdb Summary: Add option and plugin rate limiter for PosixWritableFile. The rate limiter only applies to flush and compaction. WAL and MANIFEST are excluded from this enforcement. Test Plan: db_test Reviewers: igor, yhchiang, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19425 | 08 July 2014, 19:31:49 UTC |
| 5ef1ba7 | Lei Jin | 08 July 2014, 18:41:57 UTC | generic rate limiter Summary: A generic rate limiter that can be shared by threads and rocksdb instances. Will use this to smooth out write traffic generated by compaction and flush. This will help us get better p99 behavior on flash storage. Test Plan: unit test output ==== Test RateLimiterTest.Rate request size [1 - 1023], limit 10 KB/sec, actual rate: 10.374969 KB/sec, elapsed 2002265 request size [1 - 2047], limit 20 KB/sec, actual rate: 20.771242 KB/sec, elapsed 2002139 request size [1 - 4095], limit 40 KB/sec, actual rate: 41.285299 KB/sec, elapsed 2202424 request size [1 - 8191], limit 80 KB/sec, actual rate: 81.371605 KB/sec, elapsed 2402558 request size [1 - 16383], limit 160 KB/sec, actual rate: 162.541268 KB/sec, elapsed 3303500 Reviewers: yhchiang, igor, sdong Reviewed By: sdong Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D19359 | 08 July 2014, 18:41:57 UTC |
| b278ae8 | Lei Jin | 08 July 2014, 18:40:42 UTC | Apply fractional cascading in ForwardIterator::Seek() Summary: Use search hint to reduce FindFile range thus avoid comparison For a small DB with 50M keys, perf_context counter shows it reduces comparison from 2B to 1.3B for a 15-minute run. No perf change was observed for 1 seek thread, but quite good improvement was seen for 32 seek threads, when CPU was busy. will post detail results when ready Test Plan: db_bench and db_test Reviewers: haobo, sdong, dhruba, igor Reviewed By: igor Subscribers: leveldb Differential Revision: https://reviews.facebook.net/D18879 | 08 July 2014, 18:40:42 UTC |
| 1b95bf7 | Igor Canadi | 08 July 2014, 18:40:11 UTC | Merge pull request #197 from rdallman/update-options C API: update options w/ convenience funcs & fifo compaction | 08 July 2014, 18:40:11 UTC |
| fd3fb4b | Reed Allman | 08 July 2014, 04:12:25 UTC | C API: update options w/ convenience funcs & fifo compaction | 08 July 2014, 17:57:45 UTC |
| db1a10e | Igor Canadi | 08 July 2014, 15:51:41 UTC | Merge pull request #198 from rdallman/cf-compact-range C API: bugfix column_family_compact_range | 08 July 2014, 15:51:41 UTC |
| 5491c48 | Igor Canadi | 08 July 2014, 15:49:49 UTC | Merge pull request #199 from jrobeson/patch-1 facebook accounts are not required for CLA signers | 08 July 2014, 15:49:49 UTC |
