| d2a0562 | Yueh-Hsuan Chiang | 09 April 2015, 23:39:03 UTC | Fix a compilation error in ROCKSDB_LITE in db/internal_stats.h Summary: Fix a compilation error in ROCKSDB_LITE in db/internal_stats.h Other compilation errors will be fixed in a separate diff. Test Plan: make OPT=-DROCKSDB_LITE Reviewers: sdong, igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36807 | 09 April 2015, 23:43:54 UTC |
| 316ec80 | sdong | 09 April 2015, 04:18:05 UTC | fault_injection_test: add a test case to cover log syncing after a log roll Summary: Add a test case: Write some keys without sync, flush, write other keys and do sync. Before flush finishes, host crashes and unsync data is dropped. Tag the new test as disabled since it is not passing. Test Plan: Run the test Reviewers: MarkCallaghan, rven, anthony, igor, kradhakrishnan Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36741 | 09 April 2015, 23:15:42 UTC |
| ed229a0 | Mark Callaghan | 09 April 2015, 22:51:34 UTC | Fixes for readcache-flashcache Summary: This fixes two problems: 1) the env should not be created twice when use_existing_db is false 2) the env dtor should run before cachedev_fd_ is closed. Task ID: # Blame Rev: Test Plan: Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36795 | 09 April 2015, 22:51:34 UTC |
| 91df4e9 | Igor Canadi | 09 April 2015, 22:10:53 UTC | Remove use of whole-archive to include jemalloc Summary: I don't think we need to use whole-archive to include jemalloc. This change only affects our development builds -- it does not affect our open source builds (which don't support jemalloc) or our fbcode third-party2 builds (which use open-source build codepaths). Test Plan: make verify that jemalloc is running by running `MALLOC_CONF="prof:true" ./cache_test` and observing that file was created Reviewers: MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36783 | 09 April 2015, 22:10:53 UTC |
| 84c5bd7 | agiardullo | 09 April 2015, 04:10:35 UTC | Add thread-safety documentation to MemTable and related classes Summary: Other than making some class members private, this is a documentation-only change Test Plan: unit tests Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36567 | 09 April 2015, 04:10:35 UTC |
| ee9bdd3 | sdong | 07 April 2015, 23:54:39 UTC | Script to check whether RocksDB can read DB generated by previous releases and vice versa Summary: Add a script, which checks out changes from a list of tags, build them and load the same data into it. In the last, checkout the target build and make sure it can successfully open DB and read all the data. It is implemented through ldb tool, because ldb tool is available from all previous builds so that we don't have to cross build anything. Test Plan: Run the script. Reviewers: yhchiang, rven, anthony, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36639 | 08 April 2015, 23:04:59 UTC |
| 2b019a1 | krad | 08 April 2015, 21:35:43 UTC | Enabling checksum in repair db as it should have been. Summary: I think the checksum was turned off by mistake. Test Plan: Run make check Reviewers: igor sdong chip CC: Task ID: Blame Rev: | 08 April 2015, 22:52:02 UTC |
| b1bbdd7 | sdong | 08 April 2015, 20:48:02 UTC | Create EnvOptions using sanitized DB Options Summary: Now EnvOptions uses unsanitized DB options. bytes_per_sync is tuned off when rate_limiter is used, but this change doesn't take effort. Test Plan: See different I/O pattern in db_bench running fillseq. Reviewers: yhchiang, kradhakrishnan, rven, anthony, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36723 | 08 April 2015, 21:40:42 UTC |
| edbb08b | Igor Canadi | 08 April 2015, 21:33:07 UTC | Fix Makefile Summary: These two files are test binaries and are not included in TESTS in Makefile. Test Plan: `make clean` now deletes those files, too Reviewers: sdong, kradhakrishnan, meyering Reviewed By: kradhakrishnan, meyering Subscribers: kradhakrishnan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36705 | 08 April 2015, 21:33:07 UTC |
| 199313d | Jim Meyering | 08 April 2015, 17:26:06 UTC | build: create .o files specifically for java-related targets Summary: When building rocksdbjava and rocksdbjavastatic, create -fPIC-enabled binaries in a temporary subdirectory, jl/. * Makefile (java_libobjects): New variable. (java_libobjects): New rule. (CLEAN_FILES): Arrange for "make clean" to remove that temporary dir. (rocksdbjavastatic): Depend on the new variable. Remove useless OPT=... line. (rocksdbjava): Likewise. Test Plan: JAVA_HOME=/usr/local/jdk-7u67-64 PATH=$JAVA_HOME/bin:$PATH \ make rocksdbjavastatic Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36645 | 08 April 2015, 17:26:06 UTC |
| b118238 | sdong | 08 April 2015, 00:31:52 UTC | Trivial move to cover multiple input levels Summary: Now trivial move is only triggered when moving from level n to n+1. With dynamic level base, it is possible that file is moved from level 0 to level n, while levels from 1 to n-1 are empty. Extend trivial move to this case. Test Plan: Add a more unit test of sequential loading. Non-trivial compaction happened without the patch and now doesn't happen. Reviewers: rven, yhchiang, MarkCallaghan, igor Reviewed By: igor Subscribers: leveldb, dhruba, IslamAbdelRahman Differential Revision: https://reviews.facebook.net/D36669 | 08 April 2015, 16:26:40 UTC |
| e7adfe6 | Igor Canadi | 08 April 2015, 15:52:10 UTC | Fix formatting of USERS.md | 08 April 2015, 15:52:10 UTC |
| 4e7543d | Igor Canadi | 08 April 2015, 15:50:37 UTC | Add USERS.md Summary: See the file. Test Plan: none Reviewers: lgalanis, meyering, MarkCallaghan, yhchiang, rven, anthony, kradhakrishnan, jayadev, sdong Reviewed By: jayadev Subscribers: jayadev, rdallman, andybons, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36621 | 08 April 2015, 15:50:37 UTC |
| 58346b9 | krad | 07 April 2015, 01:15:00 UTC | Log writer record format doc. Summary: Added a ASCII doodle to represent the log writer format. Test Plan: None Reviewers: sdong CC: leveldb Task ID: 6179896 Blame Rev: | 07 April 2015, 23:25:56 UTC |
| db6569c | Yueh-Hsuan Chiang | 07 April 2015, 22:25:22 UTC | Fix the compilation error in flashcache.cc on Mac Summary: Fix the following compilation error in flashcache.cc on Mac Undefined symbols for architecture x86_64: "rocksdb::NewFlashcacheAwareEnv(rocksdb::Env*, int)", referenced from: rocksdb::Benchmark::Open(rocksdb::Options*) in db_bench.o Test Plan: make db_bench Reviewers: sdong, igor, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36657 | 07 April 2015, 22:27:23 UTC |
| cba5920 | Jim Meyering | 07 April 2015, 22:19:25 UTC | build: don't use a glob for java/rocksjni/* Summary: * src.mk (JNI_NATIVE_SOURCES): New variable, so we don't have to use a glob in Makefile * Makefile (JNI_NATIVE_SOURCES): Remove glob-using definition, now that the explicit list of sources is in src.mk. Test Plan: Run this: JAVA_HOME=/usr/local/jdk-7u67-64 PATH=$JAVA_HOME/bin:$PATH \ make rocksdbjava Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36633 | 07 April 2015, 22:19:25 UTC |
| c66483c | Igor Canadi | 07 April 2015, 20:22:22 UTC | Fix github issue #563 Summary: As described in https://github.com/facebook/rocksdb/issues/563, we should add minor version to SONAME, since we break ABI with minor releases. I also turned PLATFORM_SHARED_VERSIONED to true by default. This is true in LevelDB and it was switched to false by D15117 for no apparent reason. It should only be false for iOS. Test Plan: `make shared_lib` produced librocksdb.dylib.3.10.0 Reviewers: sdong, yhchiang, meyering Reviewed By: meyering Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36573 | 07 April 2015, 20:22:22 UTC |
| de22c7b | Igor Canadi | 07 April 2015, 18:56:29 UTC | Integrate Jenkins with Phabricator Summary: After this diff, when a user submits a diff from Facebook's VPN network, we'll automatically trigger a jenkins test. Once jenkins test is done, we'll update the diff with test results. Test Plan: Made sure that jenkins build is triggered on `arc diff` and that result is reflected back on the diff Reviewers: sdong, rven, kradhakrishnan, anthony, yhchiang Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36555 | 07 April 2015, 18:56:29 UTC |
| f126140 | Yoshinori Matsunobu | 07 April 2015, 00:16:40 UTC | Fix TSAN build error of D36447 Summary: D36447 caused build error when using COMPILE_WITH_TSAN=1. This diff fixes the error. Test Plan: jenkins Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36579 | 07 April 2015, 00:37:36 UTC |
| 824e646 | Yoshinori Matsunobu | 03 April 2015, 01:14:49 UTC | Adding another NewFlashcacheAwareEnv function to support pre-opened fd Summary: There are some cases when flachcache file descriptor was already allocated (i.e. fb-MySQL). Then NewFlashcacheAwareEnv returns an error at open() because fd was already assigned. This diff adds another function to instantiate FlashcacheAwareEnv, with pre-allocated fd cachedev_fd. Test Plan: Tested with MyRocks using this function, then worked Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba, MarkCallaghan, rven Differential Revision: https://reviews.facebook.net/D36447 | 06 April 2015, 23:50:36 UTC |
| 5e067a7 | Igor Canadi | 06 April 2015, 19:50:44 UTC | Clean up compression logging Summary: Now we add warnings when user configures compression and the compression is not supported. Test Plan: Configured compression to non-supported values. Observed messages in my log: 2015/03/26-12:17:57.586341 7ffb8a496840 [WARN] Compression type chosen for level 2 is not supported: LZ4. RocksDB will not compress data on level 2. 2015/03/26-12:19:10.768045 7f36f15c5840 [WARN] Compression type chosen is not supported: LZ4. RocksDB will not compress data. Reviewers: rven, sdong, yhchiang Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35979 | 06 April 2015, 19:50:44 UTC |
| e3ee98b | Jim Meyering | 06 April 2015, 19:35:25 UTC | run 'make check's rules (and even subtests) in parallel Summary: When GNU parallel is available, "make check" tests are now run in parallel. When /dev/shm is usable, we tell those tests to create temporary files therein. Now, the longest-running single test, db_test, (which is composed of hundreds of sub-tests) is no longer run sequentially: instead, each of its sub-tests is run independently, and can be parallelized along with all other tests. To make that process easier, this change creates a temporary directory, "t/", in which it puts a small script for each of those subtests. The output from each parallel-run test is now saved in t/log-TEST_NAME. When GNU parallel is not available, we run the tests in sequence, just as before. If GNU parallel is available and you don't like the default of running one subtest per core, you can invoke "make J=1 check" to run only one test at a time. Beware: this will take a long time, and it starts with the two longest-running tests, so you will wait for a long time before seeing any results. Instead, if you want to use fewer resources but still see useful progress, try "make J=60% check". That will attempt to ensure that 60% of the cores are occupied by test runs. To watch progress of individual tests (duration, success (PASS-or-FAIL), name), run "make watch-log" in the same directory from another window. That will start with something like this: and when complete should show numbers/names like this: Every 0.1s: sort -k7,7nr -k4,4gr LOG|perl -n -e '@a=split("\t",$_,-1); $t=$a[8]; $t =~ s,^\./,,;' -e '$t =~ s, >.*,,; chomp $t;' -e '$t =~ /.*--gtest_filter=... Wed Apr 1 10:51:42 2015 152.221 PASS t/DBTest.FileCreationRandomFailure 109.280 PASS t/DBTest.EncodeDecompressedBlockSizeTest 82.315 PASS reduce_levels_test 77.812 PASS t/DBTest.CompactionFilterWithValueChange 73.236 PASS backupable_db_test 63.428 PASS deletefile_test 57.248 PASS table_test 55.665 PASS prefix_test 49.816 PASS t/DBTest.RateLimitingTest ... Test Plan: Timings (measured so as to exclude compile and link times): With this change, all tests complete in 2m40s on a system for which nproc prints 32. Prior to this this change, "make check" would take 24.5 minutes on that same system. Here are durations (in seconds) of the longest-running subtests: 152.435 PASS t/DBTest.FileCreationRandomFailure 107.070 PASS t/DBTest.EncodeDecompressedBlockSizeTest 81.391 PASS ./reduce_levels_test 71.587 PASS ./backupable_db_test 61.746 PASS ./deletefile_test 57.960 PASS ./table_test 55.230 PASS ./prefix_test 54.060 PASS t/DBTest.CompactionFilterWithValueChange 48.873 PASS t/DBTest.RateLimitingTest 47.569 PASS ./fault_injection_test 46.593 PASS t/DBTest.Randomized 42.662 PASS t/DBTest.CompactionFilter 31.793 PASS t/DBTest.SparseMerge 30.612 PASS t/DBTest.CompactionFilterV2 25.891 PASS t/DBTest.GroupCommitTest 23.863 PASS t/DBTest.DynamicLevelMaxBytesBase 22.976 PASS ./rate_limiter_test 18.942 PASS t/DBTest.OptimizeFiltersForHits 16.851 PASS ./env_test 15.399 PASS t/DBTest.CompactionFilterV2WithValueChange 14.827 PASS t/DBTest.CompactionFilterV2NULLPrefix Reviewers: igor, sdong, rven, yhchiang, igor.sugak Reviewed By: igor.sugak Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35379 | 06 April 2015, 19:35:25 UTC |
| a45e758 | sdong | 06 April 2015, 18:33:01 UTC | Avoid naming conflict of EntryType Summary: Fix build break on travis build: $ OPT=-DTRAVIS V=1 make unity && make clean && OPT=-DTRAVIS V=1 make db_test && ./db_test ...... In file included from unity.cc:65:0: ./table/plain_table_key_coding.cc: In member function ‘rocksdb::Status rocksdb::PlainTableKeyDecoder::NextPrefixEncodingKey(const char*, const char*, rocksdb::ParsedInternalKey*, rocksdb::Slice*, size_t*, bool*)’: ./table/plain_table_key_coding.cc:224:3: error: reference to ‘EntryType’ is ambiguous EntryType entry_type; ^ In file included from ./db/table_properties_collector.h:9:0, from ./db/builder.h:11, from ./db/builder.cc:10, from unity.cc:1: ./include/rocksdb/table_properties.h:81:6: note: candidates are: enum rocksdb::EntryType enum EntryType { ^ In file included from unity.cc:65:0: ./table/plain_table_key_coding.cc:16:6: note: enum rocksdb::{anonymous}::EntryType enum EntryType : unsigned char { ^ ./table/plain_table_key_coding.cc:231:51: error: ‘entry_type’ was not declared in this scope const char* pos = DecodeSize(key_ptr, limit, &entry_type, &size); ^ make: *** [unity.o] Error 1 Test Plan: OPT=-DTRAVIS V=1 make unity And make sure it doesn't break anymore. Reviewers: yhchiang, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36549 | 06 April 2015, 18:49:13 UTC |
| 3be82bc | Mark Callaghan | 06 April 2015, 17:42:12 UTC | Add p99.9 and p99.99 response time to benchmark report, add new summary report Summary: This adds p99.9 and p99.99 response times to the benchmark report and adds a second report, report2.txt that has tests listed in test order rather than the time in which they were run, so overwrite tests are listed for all thread counts, then update etc. Also changes fillseq to compress all levels to avoid write-amp from rewriting uncompressed files when they reach the first level to compress. Increase max_write_buffer_number to avoid stalls during fillseq and make max_background_flushes agree with max_write_buffer_number. See https://gist.github.com/mdcallag/297ff4316a25cb2988f7 for an example of the new report (report2.txt) Task ID: # Blame Rev: Test Plan: Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36537 | 06 April 2015, 17:42:12 UTC |
| 953a885 | sdong | 06 April 2015, 17:04:30 UTC | A new call back to TablePropertiesCollector to allow users know the entry is add, delete or merge Summary: Currently users have no idea a key is add, delete or merge from TablePropertiesCollector call back. Add a new function to add it. Also refactor the codes so that (1) make table property collector and internal table property collector two separate data structures with the later one now exposed (2) table builders only receive internal table properties Test Plan: Add cases in table_properties_collector_test to cover both of old and new ways of using TablePropertiesCollector. Reviewers: yhchiang, igor.sugak, rven, igor Reviewed By: rven, igor Subscribers: meyering, yoshinorim, maykov, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D35373 | 06 April 2015, 17:27:21 UTC |
| d2a92c1 | Jim Meyering | 03 April 2015, 21:46:35 UTC | avoid returning a number-of-active-keys estimate of nearly 2^64 Summary: If accumulated_num_non_deletions_ were ever smaller than accumulated_num_deletions_, the computation of "accumulated_num_non_deletions_ - accumulated_num_deletions_" would result in a logically "negative" value, but since the two operands are unsigned (uint64_t), the result corresponding to e.g., -1 would 2^64-1. Instead, return 0 in that case. Test Plan: - ensure "make check" still passes - temporarily add an "abort();" call in the new "if"-block, and observe that it fails in some test cases. However, note that this case is triggered only when the two numbers are equal. Thus, no test case triggers the erroneous behavior this change is designed to avoid. If anyone can construct a scenario in which that bug would be triggered, I'll be happy to add a test case. Reviewers: ljin, igor, rven, igor.sugak, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36489 | 03 April 2015, 21:46:35 UTC |
| a7ac6ce | sdong | 03 April 2015, 05:24:50 UTC | Fix level size overflow for options_.level_compaction_dynamic_level_bytes=true Summary: Int is used for level size targets when options_.level_compaction_dynamic_level_bytes=true, which will cause overflow when database grows big. Fix it. Test Plan: Add a new unit test which fails without the fix. Reviewers: rven, yhchiang, MarkCallaghan, igor Reviewed By: igor Subscribers: leveldb, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D36453 | 03 April 2015, 16:04:35 UTC |
| 089509b | sdong | 02 April 2015, 22:13:55 UTC | db_test: clean up sync points in test cleaning up Summary: In some db_test tests sync points are not cleared which will cause unexpected results in the next tests. Clean them up in test cleaning up. Test Plan: Run the same tests that used to fail: build using USE_CLANG=1 and run ./db_test --gtest_filter="DBTest.CompressLevelCompaction:*DBTestUniversalCompactionParallel*" Reviewers: rven, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36429 | 02 April 2015, 23:17:58 UTC |
| afbafea | Venkatesh Radhakrishnan | 02 April 2015, 18:06:30 UTC | Disallow trivial move if compression level is different Summary: Check compression level of start_level with output_compression before allowing trivial move Test Plan: New DBTest CompressLevelCompactionThirdPath added Reviewers: igor, yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36213 | 02 April 2015, 18:06:30 UTC |
| d0695f3 | Venkatesh Radhakrishnan | 01 April 2015, 23:55:08 UTC | Fix crash caused by opening an empty DB in readonly mode Summary: This diff fixes a crash found when an empty database is opened in readonly mode. We now check the number of levels before we open the DB as a compacted DB. Test Plan: DBTest.EmptyCompactedDB Reviewers: igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36327 | 01 April 2015, 23:55:08 UTC |
| 51c8133 | Herman Lee | 01 April 2015, 02:03:18 UTC | Fix make unity build compiler warning about "stats" shadowing global variable Summary: Fix the make unity build. The local stats variable name was shadowing a global stats variable. Test Plan: Run the build OPT=-DTRAVIS V=1 make unity Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36285 | 01 April 2015, 17:48:42 UTC |
| df71c6b | Igor Canadi | 31 March 2015, 15:56:40 UTC | Script to trigger jenkins test Summary: After you run `arc diff`, just run `build_tools/trigger_jenkins_test.sh` and Jenkins will test your diff! Test Plan: Triggered a build to jenkins Reviewers: sdong, rven, IslamAbdelRahman, anthony, yhchiang, meyering Reviewed By: meyering Subscribers: meyering, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36021 | 31 March 2015, 15:56:40 UTC |
| 38a01ed | Tian Xia | 31 March 2015, 00:48:16 UTC | Update COMMIT.md | 31 March 2015, 00:48:16 UTC |
| 76d63b4 | sdong | 30 March 2015, 22:35:18 UTC | Fix one non-determinism of DBTest.DynamicCompactionOptions Summary: After recent change of DBTest.DynamicCompactionOptions, occasionally hit another non-deterministic case where L0 showdown is triggered while timeout should not triggered for hard limit. Fix it by increasing L0 slowdown trigger at the same time. Test Plan: Run the failed test. Reviewers: igor, rven Reviewed By: rven Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D36219 | 30 March 2015, 22:53:44 UTC |
| b23bbaa | sdong | 30 March 2015, 21:04:21 UTC | Universal Compactions with Small Files Summary: With this change, we use L1 and up to store compaction outputs in universal compaction. The compaction pick logic stays the same. Outputs are stored in the largest "level" as possible. If options.num_levels=1, it behaves all the same as now. Test Plan: 1) convert most of existing unit tests for universal comapaction to include the option of one level and multiple levels. 2) add a unit test to cover parallel compaction in universal compaction and run it in one level and multiple levels 3) add unit test to migrate from multiple level setting back to one level setting 4) add a unit test to insert keys to trigger multiple rounds of compactions and verify results. Reviewers: rven, kradhakrishnan, yhchiang, igor Reviewed By: igor Subscribers: meyering, leveldb, MarkCallaghan, dhruba Differential Revision: https://reviews.facebook.net/D34539 | 30 March 2015, 22:12:02 UTC |
| 2511b7d | Igor Canadi | 30 March 2015, 20:05:35 UTC | Makefile minor cleanup Summary: Just couple of small changes: 1. removed signal_test, since it doesn't seem useful and we don't even run it as part of `make check` 2. moved perf_context_test to TESTS instead of PROGRAMS 3. `make release` probably shouldn't compile benchmarks. We currently rely on `make release` building db_bench (via Jenkins), so I left db_bench there. This is just a minor cleanup. We need to rethink our targets since they are a bit messy right now. We can do this during our tech debt week. Test Plan: make release Reviewers: anthony, rven, yhchiang, sdong, meyering Reviewed By: meyering Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36171 | 30 March 2015, 20:05:35 UTC |
| 1bd70fb | Mark Callaghan | 30 March 2015, 19:58:32 UTC | Add --stats_interval_seconds to db_bench Summary: The --stats_interval_seconds determines interval for stats reporting and overrides --stats_interval when set. I also changed tools/benchmark.sh to report stats every 60 seconds so I can avoid trying to figure out a good value for --stats_interval per test and per storage device. Task ID: #6631621 Blame Rev: Test Plan: run tools/run_flash_bench, look at output Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36189 | 30 March 2015, 19:58:32 UTC |
| fd3dbef | Igor Canadi | 30 March 2015, 19:04:10 UTC | Clean up old log files in background threads Summary: Cleaning up log files can do heavy IO, since we call ftruncate() in the destructor. We don't want to call ftruncate() in user threads. This diff moves cleaning to background threads (flush and compaction) Test Plan: make check, will also run valgrind Reviewers: yhchiang, rven, MarkCallaghan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36177 | 30 March 2015, 19:04:10 UTC |
| 99ec241 | Mark Callaghan | 30 March 2015, 18:28:25 UTC | Make the benchmark scripts configurable and add tests Summary: This makes run_flash_bench.sh configurable. Previously it was hardwired for 1B keys and tests ran for 12 hours each. That kept me from using it. This makes it configuable, adds more tests, makes the duration per-test configurable and refactors the test scripts. Adds the seekrandomwhilemerging test to db_bench which is the same as seekrandomwhilewriting except the writer thread does Merge rather than Put. Forces the stall-time column in compaction IO stats to use a fixed format (H:M:S) which makes it easier to scrape and parse. Also adds an option to AppendHumanMicros to force a fixed format. Sometimes automation and humans want different format. Calls thread->stats.AddBytes(bytes); in db_bench for more tests to get the MB/sec summary stats in the output at test end. Adds the average ingest rate to compaction IO stats. Output now looks like: https://gist.github.com/mdcallag/2bd64d18be1b93adc494 More information on the benchmark output is at https://gist.github.com/mdcallag/db43a58bd5ac624f01e1 For benchmark.sh changes default RocksDB configuration to reduce stalls: * min_level_to_compress from 2 to 3 * hard_rate_limit from 2 to 3 * max_grandparent_overlap_factor and max_bytes_for_level_multiplier from 10 to 8 * L0 file count triggers from 4,8,12 to 4,12,20 for (start,stall,stop) Task ID: #6596829 Blame Rev: Test Plan: run tools/run_flash_bench.sh Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36075 | 30 March 2015, 18:28:25 UTC |
| 2158e0f | Igor Canadi | 30 March 2015, 17:03:23 UTC | Fix clang build Summary: as title Test Plan: clang builds Reviewers: leveldb Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D36183 | 30 March 2015, 17:03:23 UTC |
| d61cb0b | Igor Canadi | 30 March 2015, 16:51:11 UTC | db_bench can now disable flashcache for background threads Summary: Most of the approach is copied from WebSQL's MySQL branch. It's nice that we can do this without touching core RocksDB code. Test Plan: Compiles and runs. Didn't test flashback code, as I don't have flashback device and most if it is c/p Reviewers: MarkCallaghan, sdong Reviewed By: sdong Subscribers: rven, lgalanis, kradhakrishnan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35391 | 30 March 2015, 16:51:11 UTC |
| 1c47c43 | Jim Meyering | 28 March 2015, 02:21:50 UTC | build: always attempt to update util/build_version.cc Summary: This fixes two bugs: "make clean" would never remove the generated file, util/build_version.cc, and since D33591, would be regenerated only if it were absent. * Makefile (clean): Remove the generated file. (util/build_version.cc): Depend on the no-prereq FORCE target, so that this target's rules are always run. Since this is a generated file, make it read-only. Also, be sure to remove the temporary file when it is the same as the original. Test Plan: Ensure that we attempt regeneration every time. Make it empty with an up-to-date time stamp and demonstrate that it is rebuilt with the expected content: $ : > util/build_version.cc $ make util/build_version.o GEN util/build_version.cc GEN util/build_version.d GEN util/build_version.cc CC util/build_version.o $ cat util/build_version.cc #include "build_version.h" const char* rocksdb_build_git_sha = "rocksdb_build_git_sha:v3.10-2-gb30e72a"; const char* rocksdb_build_git_date = "rocksdb_build_git_date:2015-03-27"; const char* rocksdb_build_compile_date = __DATE__; Reviewers: igor.sugak, sdong, ljin, igor, rven Reviewed By: rven Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D36087 | 28 March 2015, 02:21:50 UTC |
| e018892 | Herman Lee | 27 March 2015, 01:09:08 UTC | Formalize the DB properties string definitions. Summary: Assign the string properties to const string variables under the DB::Properties namespace. This helps catch typos during compilation and also consolidates the property definition in one place. Test Plan: Run rocksdb unit tests Reviewers: sdong, yoshinorim, igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35991 | 27 March 2015, 21:50:20 UTC |
| cfa5764 | Yueh-Hsuan Chiang | 27 March 2015, 18:32:49 UTC | Make auto_sanity_test always use the db_sanity_test.cc of the newer commit. Summary: Whenever we add new tests in db_sanity_test.cc, the verification test will fail since the old version db_sanity_test.cc does not have the newly added test. This patch makes auto_sanity_test.sh always use the db_sanity_test.cc of the newer commit. As a result, a macro guard is added to allow db_sanity_test.cc to be backward compatible. Test Plan: tools/auto_sanity_check.sh Reviewers: sdong, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35997 | 27 March 2015, 18:32:49 UTC |
| e9fddb7 | Igor Canadi | 27 March 2015, 13:56:48 UTC | Merge pull request #560 from xiaoxichen/patch-1 Fix interger overflow on i386 arch | 27 March 2015, 13:56:48 UTC |
| bcd8a71 | xiaoxichen | 27 March 2015, 00:38:53 UTC | Fix interger overflow on i386 arch The error was: util/logging.cc: In function 'int rocksdb::AppendHumanMicros(uint64_t, char*, int)': error: util/logging.cc:41:39: integer overflow in expression [-Werror=overflow] } else if (micros < 1000000l * 60 * 60) { ^ error: util/logging.cc:41:39: comparison between signed and unsigned integer expressions [-Werror=sign-compare] | 27 March 2015, 00:38:53 UTC |
| 030859e | Igor Canadi | 26 March 2015, 18:22:20 UTC | Dump compression info on startup Summary: It's useful to know if we have compression support or no Test Plan: Observed this in my LOG: 2015/03/26-10:34:35.460681 7f5b322b7840 Snappy supported 2015/03/26-10:34:35.460682 7f5b322b7840 Zlib supported 2015/03/26-10:34:35.460686 7f5b322b7840 Bzip supported 2015/03/26-10:34:35.460687 7f5b322b7840 LZ4 NOT supported Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35955 | 26 March 2015, 18:22:20 UTC |
| 3539e06 | Igor Canadi | 26 March 2015, 14:02:09 UTC | Merge pull request #558 from aamihailov/master fix compilation error (same as fix #284) | 26 March 2015, 14:02:09 UTC |
| a3e4b32 | Alexander.Mikhaylov | 26 March 2015, 08:53:42 UTC | fix compilation error (same as fix #284) [maa@srv2-nskb-devg2 rocksdb-master]$ CXX=/usr/local/CC/gcc-4.7.4/bin/g++ EXTRA_CXXFLAGS=-std=c++11 DISABLE_WARNING_AS_ERROR=1 make db_bench CC db/db_bench.o db/db_bench.cc: In member function 'rocksdb::Slice rocksdb::Benchmark::AllocateKey(std::unique_ptr<const char []>*)': db/db_bench.cc:1434:41: error: use of deleted function 'void std::unique_ptr<_Tp [], _Dp>::reset(_Up) [with _Up = char*; _Tp = const char; _Dp = std::default_delete<const char []>]' In file included from /usr/local/CC/gcc-4.7.4/lib/gcc/x86_64-unknown-linux-gnu/4.7.4/../../../../include/c++/4.7.4/memory:86:0, from ./include/rocksdb/db.h:14, from ./db/dbformat.h:14, from ./db/db_impl.h:21, from db/db_bench.cc:33: | 26 March 2015, 08:53:42 UTC |
| ff1ff7c | Jim Meyering | 25 March 2015, 21:40:41 UTC | TSAN: avoid new link failure with -pg Summary: * Makefile (COMPILE_WITH_TSAN): Avoid a link failure by disabling -pg when building with TSAN enabled. Now that "make check" builds all $(PROGRAMS), it is linking a few programs that were not normally linked before. For example, this would fail to link with the following diagnostic: COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench CCLD log_and_apply_bench ld: /usr/lib/../lib64/gcrt1.o: relocation R_X86_64_32S against `__libc_csu_fini' can not be used when making a shared object; recompile with -fPIC /usr/lib/../lib64/gcrt1.o: error adding symbols: Bad value collect2: error: ld returned 1 exit status Makefile:511: recipe for target 'log_and_apply_bench' failed make: *** [log_and_apply_bench] Error 1 Since removing -pg is sufficient to get past this link failure, and no one cares about profiling TSAN-enabled binaries anyway, we will refrain from linking with -pg when TSAN testing is enabled. Use a new variable, "pg" which is set to "-pg" in most cases, but that is made empty when COMPILE_WITH_TSAN is set. Test Plan: Now, this succeeds: rm -f log_and_apply_bench COMPILE_WITH_TSAN=1 make -j40 log_and_apply_bench Reviewers: igor.sugak, rven, sdong, ljin, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35943 | 25 March 2015, 21:40:41 UTC |
| 39d508e | Yueh-Hsuan Chiang | 25 March 2015, 21:14:26 UTC | Add a missing section title in HISTORY.md Summary: Add a missing section title in HISTORY.md Test Plan: no code change | 25 March 2015, 21:14:26 UTC |
| 727684b | Yueh-Hsuan Chiang | 25 March 2015, 18:08:08 UTC | Fixed a typo in RocksDBSample.java Summary: Fixed a typo in RocksDBSample.java Test Plan: make clean make rocksdbjava -j32 make jtest | 25 March 2015, 18:09:30 UTC |
| ca25e86 | Jim Meyering | 25 March 2015, 05:09:40 UTC | build: cause the "check" rule to depend on $(PROGRAMS) Summary: * Makefile (check): Cause "make check" to build all $(PROGRAMS), so that it verifies that the few benchmark-only source files that not already built via "make check" do compile and link successfully. Test Plan: run "make clean; make check", and verify that table/table_reader_bench.cc is now compiled. Before, it was not, which led to an incomplete fix for a build break. Reviewers: ljin, igor.sugak, rven, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35883 | 25 March 2015, 05:09:40 UTC |
| 986bdc6 | Anurag Indu | 25 March 2015, 00:00:41 UTC | Merge branch 'master' of github.com:facebook/rocksdb | 25 March 2015, 00:00:41 UTC |
| 1e57f2b | Anurag Indu | 24 March 2015, 23:53:49 UTC | Fix build Test Plan: Running make all Reviewers: sdong Reviewed By: sdong Subscribers: rven, yhchiang, igor, meyering, dhruba Differential Revision: https://reviews.facebook.net/D35889 | 25 March 2015, 00:00:28 UTC |
| 2d417e5 | Yueh-Hsuan Chiang | 24 March 2015, 23:39:39 UTC | Update HISTORY.md for 3.10.0 Summary: Update HISTORY.md for 3.10.0 Test Plan: no code chagne. Reviewers: sdong, rven, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35871 | 24 March 2015, 23:39:39 UTC |
| cbd6a20 | Anurag Indu | 24 March 2015, 23:27:39 UTC | Merge branch 'master' of github.com:facebook/rocksdb | 24 March 2015, 23:27:39 UTC |
| 211ca26 | Anurag Indu | 24 March 2015, 22:06:09 UTC | Fixing build issue Summary: Fixing issues with get context function. Test Plan: Run make commit-prereq Reviewers: sdong, meyering, yhchiang Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35853 | 24 March 2015, 23:27:24 UTC |
| 2495f93 | Yueh-Hsuan Chiang | 24 March 2015, 22:39:42 UTC | Merge pull request #555 from pshareghi/staticLZ4 [RocksJava] Added static lz4 support for roccksjavastatic | 24 March 2015, 22:39:42 UTC |
| f06de5f | Yueh-Hsuan Chiang | 24 March 2015, 21:58:00 UTC | Merge pull request #556 from fyrz/RocksJava-Maven-Fix [RocksJava] Maven build fix | 24 March 2015, 21:58:00 UTC |
| cd987c3 | Yueh-Hsuan Chiang | 24 March 2015, 21:50:32 UTC | Fix compile error when NROCKSDB_THREAD_STATUS is not used. Summary: Fix compile error when NROCKSDB_THREAD_STATUS is not used. Test Plan: make dbg OPT=-DNROCKSDB_THREAD_STATUS -j32 Reviewers: sdong, igor, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35847 | 24 March 2015, 21:52:59 UTC |
| 8f104ae | fyrz | 24 March 2015, 21:52:30 UTC | [RocksJava] Maven build fix | 24 March 2015, 21:52:30 UTC |
| 3d1a924 | Anurag Indu | 03 March 2015, 18:59:36 UTC | Adding stats for the merge and filter operation Summary: We have addded new stats and perf_context for measuring the merge and filter operation time consumption. We have bounded all the merge operations within the GUARD statment and collected the total time for these operations in the DB. Test Plan: WIP Reviewers: rven, yhchiang, kradhakrishnan, igor, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D34377 | 24 March 2015, 21:42:04 UTC |
| afc5164 | Yueh-Hsuan Chiang | 24 March 2015, 21:01:12 UTC | Merge pull request #546 from fyrz/RocksJava-MemEnv [RocksJava] Expose MemEnv in RocksJava | 24 March 2015, 21:01:12 UTC |
| 4806cc1 | Pooya Shareghi | 24 March 2015, 17:40:00 UTC | Added static lz4 support for roccksjavastatic Usage: make clean rocksdbjavastatic Dependency: LZ4 To install LZ4: git clone https://github.com/Cyan4973/lz4.git && \ cd lz4/lib && \ make install Alternativel installation method: curl -O https://codeload.github.com/Cyan4973/lz4/tar.gz/r127 mv r127 lz4-r127.tar.gz tar xvzf lz4-r127.tar.gz cd lz4-r127/lib && \ make install Signed-off-by: Pooya Shareghi <shareghi@gmail.com> | 24 March 2015, 20:48:53 UTC |
| 6284eef | Yueh-Hsuan Chiang | 24 March 2015, 19:14:09 UTC | Merge pull request #545 from fyrz/RocksJava-Level-Compression [RocksJava] Add compression per level to API | 24 March 2015, 19:14:09 UTC |
| 248c063 | Yueh-Hsuan Chiang | 23 March 2015, 23:35:04 UTC | Report elapsed time in micros in ThreadStatus instead of start time. Summary: Report elapsed time of a thread operation in micros in ThreadStatus instead of start time of a thread operation in seconds since the Epoch, 1970-01-01 00:00:00 (UTC). Test Plan: ./db_bench --benchmarks=fillrandom --num=100000 --threads=40 \ --max_background_compactions=10 --max_background_flushes=3 \ --thread_status_per_interval=1000 --key_size=16 --value_size=1000 \ --num_column_families=10 Sample Output: ThreadID ThreadType cfName Operation ElapsedTime Stage State 140667724562496 High Pri column_family_name_000002 Flush 772.419 ms FlushJob::WriteLevel0Table 140667728756800 High Pri default Flush 617.845 ms FlushJob::WriteLevel0Table 140667732951104 High Pri column_family_name_000005 Flush 772.078 ms FlushJob::WriteLevel0Table 140667875557440 Low Pri column_family_name_000008 Compaction 1409.216 ms CompactionJob::Install 140667737145408 Low Pri 140667749728320 Low Pri 140667816837184 Low Pri column_family_name_000007 Compaction 1071.815 ms CompactionJob::ProcessKeyValueCompaction 140667787477056 Low Pri column_family_name_000009 Compaction 772.516 ms CompactionJob::ProcessKeyValueCompaction 140667741339712 Low Pri 140667758116928 Low Pri column_family_name_000004 Compaction 620.739 ms CompactionJob::ProcessKeyValueCompaction 140667753922624 Low Pri 140667842003008 Low Pri column_family_name_000006 Compaction 1260.079 ms CompactionJob::ProcessKeyValueCompaction 140667745534016 Low Pri Reviewers: sdong, igor, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35769 | 24 March 2015, 18:32:25 UTC |
| 315abac | Igor Canadi | 24 March 2015, 16:42:38 UTC | Undeprecate GetLiveFiles() Summary: There is no alternative to GetLiveFiles() function Test Plan: none Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35805 | 24 March 2015, 16:42:38 UTC |
| a057bb2 | Yueh-Hsuan Chiang | 23 March 2015, 22:05:58 UTC | Improve ThreadStatusSingleCompaction Summary: Improve ThreadStatusSingleCompaction in two ways: 1. Use SYNC_POINT to ensure compaction won't happen before the test finishes its "Put Phase" instead of using sleep. 2. In Put Phase, it continues until we have sufficient number of L0 files. Note that during the put phase, there won't be any compaction that consumes L0 files because of item 1. Test Plan: ./db_test --gtest_filter="*ThreadStatusSingleCompaction*" Reviewers: sdong, igor, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35727 | 23 March 2015, 22:30:45 UTC |
| 868968c | krad | 23 March 2015, 22:04:48 UTC | Merge branch 'master' of github.com:facebook/rocksdb | 23 March 2015, 22:04:48 UTC |
| 6893914 | krad | 23 March 2015, 20:38:17 UTC | Make SSTDumpTest.GetProperties less noisy Summary: Limiting verbose printing to "command=scan" Test Plan: Run make check and manual testing of sst_dump_test Reviewers: sdong CC: leveldb Task ID: #6575982 Blame Rev: | 23 March 2015, 21:30:11 UTC |
| 8d86562 | Yueh-Hsuan Chiang | 23 March 2015, 20:25:37 UTC | Merge pull request #551 from fyrz/RocksJava-JavaDoc-Fix [RocksJava] Add missing JavaDoc annotations | 23 March 2015, 20:25:37 UTC |
| f8c505b | Yueh-Hsuan Chiang | 23 March 2015, 20:24:37 UTC | Merge pull request #550 from fyrz/RocksJava-Sample-Fix [RocksJava] Java sample bugfix | 23 March 2015, 20:24:37 UTC |
| 46443bf | fyrz | 23 March 2015, 20:17:20 UTC | [RocksJava] Add missing JavaDoc annotations | 23 March 2015, 20:17:20 UTC |
| 864b7e8 | fyrz | 23 March 2015, 20:03:28 UTC | [RocksJava] Java sample bugfix One of the latest commits broke the sample code. This resolves the introduced bug. | 23 March 2015, 20:03:28 UTC |
| 2b2394c | fyrz | 22 March 2015, 18:02:36 UTC | [RocksJava] DBBenchmark option for RocksMemEnv Extended DBBenchmark with option to enable RocksMemEnv. | 23 March 2015, 17:59:32 UTC |
| fd8804f | fyrz | 22 March 2015, 17:30:51 UTC | [RocksJava] Expose MemEnv in RocksJava Summary: In 3.10 the C++ code was extended with a MemEnv implementation. This is now also available in RocksJava. Changes: - Extraced abstract super class Env - Introduced RocksMemEnv - Remove unnecessary disposeInternal method. The disposal of the default environment is managed by C++ so there needs to be no disposeInternal method in Java. - Introduced a RocksMemEnvTest, which is aligned with the C++ equivalent. Test Plan: make rocksdbjava make jtest Reviewers: adamretter, yhchiang, ankgup87 Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35619 | 23 March 2015, 17:59:31 UTC |
| 004b89f | fyrz | 19 March 2015, 22:50:45 UTC | [RocksJava] Add compression per level to API Summary: RocksDB offers the possibility to set different compression types on a per level basis. This shall be also available using RocksJava. Test Plan: make rocksdbjava make jtest Reviewers: adamretter, yhchiang, ankgup87 Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35577 | 23 March 2015, 17:58:56 UTC |
| 51da3aa | Adam Retter | 22 March 2015, 23:33:35 UTC | Merge pull request #536 from fyrz/RocksJava-32-Bit-adjustment [RocksJava] 32-Bit adjustments | 22 March 2015, 23:33:35 UTC |
| b7e88cf | Adam Retter | 22 March 2015, 23:16:02 UTC | Merge pull request #543 from fyrz/RocksJava-Logger-Comment [RocksJava] Enhanced Logger comment | 22 March 2015, 23:16:02 UTC |
| 38d286f | sdong | 20 March 2015, 19:13:18 UTC | Clean-up WAL directory before running db_test Summary: DBTest doesn't clean up wal directory. It might cause failure after a failure test run. Fix it. Test Plan: Run unit tests Try open DB with non-empty db_path/wal. Reviewers: rven, yhchiang, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D35559 | 20 March 2015, 19:34:24 UTC |
| 28bc6de | Igor Sugak | 20 March 2015, 00:32:43 UTC | rocksdb: print status error message when (ASSERT|EXPECT)_OK fails Summary: Modified rocksdb status assertions ASSERT_OK and EXPECT_OK to print error message from Status::ToString() when failed. Test Plan: Modify a test to fail status assertions ASSERT_OK and EXPECT_OK and notice an error message that came from Status::ToString() Reviewers: meyering, sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35469 | 20 March 2015, 00:32:43 UTC |
| 9405b5e | Igor Sugak | 20 March 2015, 00:29:37 UTC | rocksdb: Remove #include "util/string_util.h" from util/testharness.h Summary: 1. Manually deleted #include "util/string_util.h" from util/testharness.h 2. ``` % USE_CLANG=1 make all -j55 -k 2> build.log % perl -naF: -E 'say $F[0] if /: error:/' build.log | sort -u | xargs sed -i '/#include "util\/testharness.h"/i #include "util\/string_util.h"' ``` Test Plan: Make sure make all completes with no errors. ``` % make all -j55 ``` Reviewers: meyering, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35493 | 20 March 2015, 00:29:37 UTC |
| 220d0df | Igor Sugak | 20 March 2015, 00:06:02 UTC | rocksdb: Remove #include "util/random.h" from util/testharness.h Summary: Cleaning util/testharness.h Test Plan: Make completes with no errors. ``` % make all ``` Reviewers: meyering, igor, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35487 | 20 March 2015, 00:06:02 UTC |
| b088c83 | Igor Canadi | 20 March 2015, 00:04:29 UTC | Don't delete files when column family is dropped Summary: To understand the bug read t5943287 and check out the new test in column_family_test (ReadDroppedColumnFamily), iter 0. RocksDB contract allowes you to read a drop column family as long as there is a live reference. However, since our iteration ignores dropped column families, AddLiveFiles() didn't mark files of a dropped column families as live. So we deleted them. In this patch I no longer ignore dropped column families in the iteration. I think this behavior was confusing and it also led to this bug. Now if an iterator client wants to ignore dropped column families, he needs to do it explicitly. Test Plan: Added a new unit test that is failing on master. Unit test succeeds now. Reviewers: sdong, rven, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D32535 | 20 March 2015, 00:04:29 UTC |
| 17ae3fc | Igor Sugak | 19 March 2015, 23:52:59 UTC | rocksdb: initial util/testharness clean up Summary: Deleted some redundant code. More comming. Test Plan: ```lang=bash % USE_CLANG=1 make check ``` Reviewers: meyering, sdong, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35463 | 19 March 2015, 23:52:59 UTC |
| 39f4271 | fyrz | 19 March 2015, 20:58:03 UTC | [RocksJava] Enhanced Logger comment Added information about performance penalties using a custom logger implementation. | 19 March 2015, 21:01:00 UTC |
| 5615e23 | fyrz | 12 March 2015, 21:25:57 UTC | [RocksJava] 32-Bit adjustments Summary: Before this change overflowing size_t values led to a checked Exception. After that change: size_t overflows on 32-Bit architecture throw now an IllegalArgumentException, which removes the necessity for a developer to catch these Exceptions explicitly. This is especially an advantage for developers targeting 64-Bit systems because it is not necessary anymore to catch exceptions which are never thrown on a 64-Bit system. Test Plan: make clean jclean rocksdbjava jtest mvn -f rocksjni.pom package Reviewers: adamretter, yhchiang, ankgup87 Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D34923 | 19 March 2015, 19:45:55 UTC |
| 836bcc2 | Adam Retter | 19 March 2015, 12:51:40 UTC | Merge pull request #532 from fyrz/RocksJava-LevelCompactionDynamicLevelBytes [RocksJava] Added LevelCompactionDynamicLevelBytes to Options | 19 March 2015, 12:51:40 UTC |
| 67d9958 | Igor Sugak | 19 March 2015, 02:12:23 UTC | rocksdb: fix make unity Summary: changed make unity target to use $LIB_SOURCES as a source of library source code. In the old way (using find) table/mock_table.h was added to a list of library objects and this was a course of `make unity` break. `build_tools/unity` contains some redundant code, I deleted it and moved the functionality in the Makefile. Test Plan: Make sure unity completes with no errors. ```lang=bash % make unity ``` Reviewers: sdong, rven, igor, meyering Reviewed By: igor, meyering Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35385 | 19 March 2015, 02:12:23 UTC |
| 52e0f33 | Igor Canadi | 19 March 2015, 01:25:15 UTC | Clean up compactions_in_progress_ Summary: Suprisingly, the only way we use this vector is to keep track of level0 compactions. Thus, I simplified it. Test Plan: make check Reviewers: rven, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35313 | 19 March 2015, 01:25:15 UTC |
| 6b626ff | Igor Sugak | 19 March 2015, 01:18:12 UTC | rocksdb: change db_test::MultiThreadedDBTest as value parameterized test. Summary: This is a simple change to make db_test::MultiThreadedDBTest as value parameterized test. There is a value of creating a separate set of such tests later. Test Plan: ```lang=bash % make db_test % ./make db_test ``` Also with the following command I can execute all db_test in 2:37.87 on my box ``` % ./db_test --gtest_list_tests | sed 's/\# GetParam.*//' | tr -d ' ' | env time parallel --gnu --eta --joblog=LOG -- 'TEST_TMPDIR=/dev/shm/rocksdb-{} ./db_test --gtest_filter="*{}"' ``` Reviewers: igor, rven, meyering, sdong Reviewed By: meyering Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35361 | 19 March 2015, 01:18:12 UTC |
| 9720ea4 | sdong | 19 March 2015, 00:26:29 UTC | A build option to run through all check-in requirements. Summary: Make it easier for people to run all the tests. Test Plan: Run it. Reviewers: rven, yhchiang, igor, MarkCallaghan, IslamAbdelRahman, igor.sugak, anthony, kradhakrishnan, meyering Reviewed By: meyering Subscribers: meyering, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D35319 | 19 March 2015, 00:57:42 UTC |
| 0831a35 | sdong | 18 March 2015, 23:11:02 UTC | Add a DB Property For Number of Deletions in Memtables Summary: Add a DB property for number of deletions in memtables. It can sometimes help people debug slowness because of too many deletes. Test Plan: Add test cases. Reviewers: rven, yhchiang, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D35247 | 19 March 2015, 00:03:59 UTC |
| f7ed654 | Yueh-Hsuan Chiang | 18 March 2015, 22:40:27 UTC | Fix RocksJava test failure due to deprecation of table_cache_remove_scan_count_limit Summary: table_cache_remove_scan_count_limit is marked as deprecated in RocksDB C++. (see rocksdb/options.h). This patch made necessary changes on RocksJava side. Test Plan: make rocksdbjava -j32 make jtest Reviewers: rven, igor, fyrz, adamretter, ankgup87, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35355 | 18 March 2015, 22:40:27 UTC |
| 51301b8 | Igor Canadi | 18 March 2015, 22:35:55 UTC | Enable dynamic changing of rate limiter's bytes_per_second Summary: This feature is going to be useful for mongodb+rocksdb. I'll expose it through mongo's API. Test Plan: added new unit test. also will run TSAN on the new unit test Reviewers: meyering, sdong Reviewed By: meyering, sdong Subscribers: meyering, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35307 | 18 March 2015, 22:35:55 UTC |
| 652db51 | Yueh-Hsuan Chiang | 18 March 2015, 22:08:31 UTC | Fix compilation error in rocksjni/write_batch_test.cc Summary: Fix compilation error in rocksjni/write_batch_test.cc Test Plan: make rocksdbjava -j32 Reviewers: sdong, igor, rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D35343 | 18 March 2015, 22:10:57 UTC |
| dfccc7b | Mark Callaghan | 18 March 2015, 20:50:52 UTC | Add readwhilemerging benchmark Summary: This is like readwhilewriting but uses Merge rather than Put in the writer thread. I am using it for in-progress benchmarks. I don't think the other benchmarks for Merge cover this behavior. The purpose for this test is to measure read performance when readers might have to merge results. This will also benefit from work-in-progress to add skewed key generation. Task ID: # Blame Rev: Test Plan: Revert Plan: Database Impact: Memcache Impact: Other Notes: EImportant: - begin *PUBLIC* platform impact section - Bugzilla: # - end platform impact - Reviewers: igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D35115 | 18 March 2015, 20:50:52 UTC |
| c345d1e | fyrz | 18 March 2015, 20:37:46 UTC | [RocksJava] Integrated changes for D34857 | 18 March 2015, 20:38:17 UTC |
| 1235011 | fyrz | 11 March 2015, 20:50:10 UTC | [RocksJava] Added LevelCompactionDynamicLevelBytes to Options Summary: Added LevelCompactionDynamicLevelBytes to Options. Test Plan: make clean jclean rocksdbjava jtest mvn -f rocksjni.pom package Reviewers: adamretter, ankgup87, yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D34857 | 18 March 2015, 20:38:17 UTC |
