| 6999a23 | sdong | 09 December 2015, 00:39:10 UTC | Include ldb_tools and sst_dump_tools libraries in shared library Summary: Ldb and sst_dump are not included in shared library now. Add it. Test Plan: Build make release make shared_lib Reviewers: igor, kradhakrishnan, rven, yhchiang, IslamAbdelRahman, anthony Reviewed By: IslamAbdelRahman, anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D51735 | 09 December 2015, 19:05:01 UTC |
| 6880d2f | sdong | 09 December 2015, 01:36:59 UTC | Fix fb-only build for gcc 4.8.1 Summary: fb build for gcc 4.8.1 is broken for the ZSTD dependency is not fixed after ea11923550802a3eeebe81f2cc4a9a44cda32121. Fixing it. Test Plan: ROCKSDB_FBCODE_BUILD_WITH_481=1 make -j40 OPT=-g Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D51741 | 09 December 2015, 19:04:06 UTC |
| b46e796 | sdong | 09 December 2015, 00:33:10 UTC | Upgrade to ZSTD 0.4.2 Summary: Change to call the new compression function. Test Plan: build and run db_bench with the compression to make sure it compresses. Reviewers: anthony, rven, kradhakrishnan, IslamAbdelRahman, igor, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D51603 | 09 December 2015, 19:02:49 UTC |
| 7f64b56 | Yueh-Hsuan Chiang | 17 November 2015, 21:53:06 UTC | Update HISTORY.md for RocksDB 4.1 Summary: Update HISTORY.md for RocksDB 4.1 Test Plan: no code change Reviewers: igor, sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan Subscribers: leveldb | 17 November 2015, 21:53:06 UTC |
| f2ebb67 | Yueh-Hsuan Chiang | 17 November 2015, 21:50:54 UTC | Fix Java Makefile Summary: In case rocksdb java package is built using make rocksdbjavastaticrelease, then only those rocksdb binary built under the virtual environments is release build. This patch fix this issue. Test Plan: PORTABLE=1 V=2 make rocksdbjavastaticrelease -j32 and make sure -O2 and -NDEBUG is included when compiling all source files. Reviewers: sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50895 | 17 November 2015, 21:52:32 UTC |
| 75a8bad | Islam AbdelRahman | 09 November 2015, 21:34:49 UTC | Merge pull request #821 from yuslepukhin/continue_windows_warnings Enable C4200, C4702, C4305 windows warnings | 09 November 2015, 21:34:49 UTC |
| 986230b | Nathan Bronson | 09 November 2015, 19:27:44 UTC | Revert "Fix TSAN build for fbcode" Summary: Reverting c745f1d2c4b72a58874605c5da4180a4cb9a0366 because it was based on an incorrect understanding of the correct way to enable TSAN tests (it assumes "make COMPILE_WITH_TSAN=1 check" but in fact only "COMPILE_WITH_TSAN=1 make check" is supported). Test Plan: COMPILE_WITH_TSAN=1 make check Reviewers: kradhakrishnan Reviewed By: kradhakrishnan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50445 | 09 November 2015, 19:51:10 UTC |
| f3ca28a | Yueh-Hsuan Chiang | 08 November 2015, 17:01:26 UTC | Correct the comment of GetApproximateMemoryUsageByType Summary: Correct the comment of GetApproximateMemoryUsageByType. Test Plan: No code change. Reviewers: igor, sdong, anthony, IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50409 | 08 November 2015, 17:02:35 UTC |
| 838676c | Islam AbdelRahman | 07 November 2015, 00:49:38 UTC | Revert "Adding new table properties" Summary: Reverting https://reviews.facebook.net/D34269 for now after I landed it a flaky test started continuously failing, I am almost sure this patch is not related to the test but I will revert it until I figure out why it's failing Test Plan: make check Reviewers: kradhakrishnan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50385 | 07 November 2015, 00:49:38 UTC |
| 7c86d50 | Dmitri Smirnov | 07 November 2015, 00:44:39 UTC | Enable C4305 'identifier' : truncation from 'type1' to 'type2' | 07 November 2015, 00:44:39 UTC |
| 85a2ce9 | Dmitri Smirnov | 06 November 2015, 23:50:11 UTC | Enable C4702 unreachable code | 06 November 2015, 23:50:11 UTC |
| 62aa1b1 | Dmitri Smirnov | 06 November 2015, 23:32:45 UTC | Enable C4200 warning nonstandard extension used : zero-sized array in struct/union | 06 November 2015, 23:32:45 UTC |
| 5b9ce1a | Islam AbdelRahman | 06 November 2015, 20:08:25 UTC | Merge pull request #820 from yuslepukhin/enable_compiler_warnings Enable Windows warnings C4307 C4309 C4512 C4701 | 06 November 2015, 20:08:25 UTC |
| 20f57b1 | Dmitri Smirnov | 06 November 2015, 19:34:06 UTC | Enable Windows warnings C4307 C4309 C4512 C4701 Enable C4307 'operator' : integral constant overflow Longs and ints on Windows are 32-bit hence the overflow Enable C4309 'conversion' : truncation of constant value Enable C4512 'class' : assignment operator could not be generated Enable C4701 Potentially uninitialized local variable 'name' used | 06 November 2015, 19:34:06 UTC |
| 8be568a | Islam AbdelRahman | 06 November 2015, 19:19:01 UTC | Adding new table properties Summary: This diff introduce new table properties that will be written for block based tables These properties are - comparator name - merge operator name - property collectors names Test Plan: - Added a new unit test to verify that these tests are written/read correctly - Running all other tests right now (wont land until all tests finish) Reviewers: rven, kradhakrishnan, igor, sdong, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D34269 | 06 November 2015, 19:19:01 UTC |
| 2b42000 | Nathan Bronson | 02 November 2015, 17:23:39 UTC | incorrect batch group size computation for write throttling Summary: When a write batch can't join a batch group due to the total size of the contained batches, the write controller's GetDelay is passed a size value that includes the rejected batch. Test Plan: make check Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50343 | 06 November 2015, 17:23:55 UTC |
| c745f1d | Nathan Bronson | 06 November 2015, 17:04:28 UTC | Fix TSAN build for fbcode Summary: TSAN builds for gcc 4.9 need a PIC version of the libraries taken from the fbcode platform. This is accomplished by assuming every .a has a _pic.a sibling, and by fixing the third-party2 zlib build. Test Plan: make COMPILE_WITH_TSAN=1 check Reviewers: sdong, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50331 | 06 November 2015, 17:05:12 UTC |
| fe789c5 | agiardullo | 04 November 2015, 23:16:48 UTC | Document SingleDelete Summary: Docuemented what is currently supported by SingleDelete based on its current implementation. Test Plan: n/a Reviewers: sdong, kradhakrishnan, yhchiang, rven Reviewed By: rven Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50205 | 06 November 2015, 01:24:07 UTC |
| e89e5b2 | Islam AbdelRahman | 06 November 2015, 01:21:34 UTC | Merge pull request #818 from yuslepukhin/improve_test_concurrency Improve concurrency when running tests | 06 November 2015, 01:21:34 UTC |
| ae7940b | Venkatesh Radhakrishnan | 06 November 2015, 00:43:54 UTC | Fix regression failure in PrefixTest.PrefixValid Summary: Use IterKey to store prefix_start_ so that it doesn't get freed Test Plan: PrefixTest.PrefixValid Reviewers: anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50289 | 06 November 2015, 00:43:54 UTC |
| 3277d17 | Dmitri Smirnov | 05 November 2015, 22:03:41 UTC | Improve concurrency when running tests PowerShell seems to have a hard time when a flood of async tasks is scheduled at the same time. I speculated that WaitForMultipleObjects() in Windows can only take up to 64 process handles and if you want to handle more than you should write some additional code which can be sub-optimal. I.e to implement Wait-Job -Any. I decided to test that suggestion and introduced a $Concurrency parameter with a default value of 62. So in the new version the script fires up up to $Concurrency value and wait for anything to complete before starting any more processes. This improved matters greatly. Individual tests against ramdrive now run in 8 minutes and all of the 200+ db_tests run in 9 minutes with concurrency values of 8-16. About 48 is required to load a CPU on my box running against HD but that does not improve running times much. Other changes include respect -EnableJE for the individual test exes. Enforce exclusions for the individual tests. | 05 November 2015, 22:03:41 UTC |
| c8e01ef | Satnam Singh | 05 November 2015, 21:30:51 UTC | Delete test iterators Summary: Valgrind reports an issue with the test for GeoIterator. This diff explicitly deletes the two iterators used in this test. Test Plan: This diff is for a test. The test still passes. Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50193 | 05 November 2015, 21:30:51 UTC |
| 9d50afc | Venkatesh Radhakrishnan | 05 November 2015, 21:24:05 UTC | Prefix-based iterating only shows keys in prefix Summary: MyRocks testing found an issue that while iterating over keys that are outside the prefix, sometimes wrong results were seen for keys outside the prefix. We now tighten the range of keys seen with a new read option called prefix_seen_at_start. This remembers the starting prefix and then compares it on a Next for equality of prefix. If they are from a different prefix, it sets valid to false. Test Plan: PrefixTest.PrefixValid Reviewers: IslamAbdelRahman, sdong, yhchiang, anthony Reviewed By: anthony Subscribers: spetrunia, hermanlee4, yoshinorim, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50211 | 05 November 2015, 21:24:05 UTC |
| 14c6e1a | Igor Canadi | 05 November 2015, 20:07:39 UTC | Add write_stress to RocksDB Legocastle runs Summary: As title. Let's run it for 1 hour. Test Plan: How can I test legocastle changes? Reviewers: IslamAbdelRahman, yhchiang, rven, sdong, anthony, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49653 | 05 November 2015, 20:07:39 UTC |
| db3f5e4 | Yueh-Hsuan Chiang | 05 November 2015, 07:47:53 UTC | Update HISTORY.md Summary: Move some of the items in public API changes section to to new features section. Test Plan: no code change. Reviewers: igor, sdong, anthony, IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50229 | 05 November 2015, 07:52:09 UTC |
| 042fb05 | Islam AbdelRahman | 05 November 2015, 05:02:20 UTC | Fix clang Summary: Fix build for clang Test Plan: USE_CLANG=1 make all -j64 make clean make check -j64 Reviewers: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50217 | 05 November 2015, 05:02:20 UTC |
| 2419f43 | Islam AbdelRahman | 05 November 2015, 04:40:57 UTC | Merge pull request #816 from SherlockNoMad/GeoDBTestFix Fix appveyor build failure | 05 November 2015, 04:40:57 UTC |
| 2e45409 | SherlockNoMad | 05 November 2015, 04:10:16 UTC | Fix appveyor build failure | 05 November 2015, 04:10:16 UTC |
| 183cadf | Yueh-Hsuan Chiang | 05 November 2015, 02:53:30 UTC | Add OptionsSanityCheckLevel Summary: This patch introduces OptionsSanityCheckLevel internally to enable sanity check rocksdb options. Utilities API will be added in the follow-up diffs. Test Plan: Added more tests in options_test Reviewers: igor, IslamAbdelRahman, sdong, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49515 | 05 November 2015, 02:53:30 UTC |
| dba5e00 | Yueh-Hsuan Chiang | 04 November 2015, 02:04:41 UTC | Fixed the compile error in RocksDBLite in memory_test.cc Summary: Fixed the following compile error in RocksDBLite: 18:00:33 CC utilities/memory/memory_test.o 18:00:33 utilities/memory/memory_test.cc: In function ‘int main(int, char**)’: 18:00:33 utilities/memory/memory_test.cc:268:66: error: ‘printf’ was not declared in this scope 18:00:33 printf("Skipped in RocksDBLite as utilities are not supported."); 18:00:33 ^ Test Plan: make OPT=-DROCKSDB_LITE memory_test Reviewers: igor, sdong, anthony, IslamAbdelRahman Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D50145 | 04 November 2015, 02:06:23 UTC |
| 7d7ee2b | Yueh-Hsuan Chiang | 04 November 2015, 01:52:17 UTC | Add Memory Insight support to utilities Summary: This patch introduces utilities/memory, which currently includes GetApproximateMemoryUsageByType that reports different types of rocksdb memory usage given a list of input DBs. The API also take care of the case where Cache could be shared across multiple column families / multiple db instances. Currently, it reports memory usage of memtable, table-readers and cache. Test Plan: utilities/memory/memory_test.cc Reviewers: igor, anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49257 | 04 November 2015, 01:52:17 UTC |
| 3ecbab0 | Yueh-Hsuan Chiang | 03 November 2015, 23:54:18 UTC | Add GetAggregatedIntProperty(): returns the aggregated value from all CFs Summary: This patch adds GetAggregatedIntProperty() that returns the aggregated value from all CFs Test Plan: Added a test in db_test Reviewers: igor, sdong, anthony, IslamAbdelRahman, rven Reviewed By: rven Subscribers: rven, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49497 | 03 November 2015, 23:54:18 UTC |
| 93a9667 | Satnam Singh | 03 November 2015, 23:36:14 UTC | Merge branch 'master' of github.com:facebook/rocksdb | 03 November 2015, 23:36:14 UTC |
| c9aef3c | Satnam Singh | 02 November 2015, 21:49:13 UTC | Add RocksDb/GeoDb Iterator interface Summary: This diff is a first step towards an iterator based interface for the SearchRadial method which replaces a vector of GeoObjects with an iterator for GeoObjects. This diff works by just wrapping the iterator for the encapsulated vector of GeoObjects. A future diff could extend this approach by defining an interator in terms of the underlying iteration in SearchRadial which would then remove the need to have an in-memory representation for all the matching GeoObjects. Fixes T8421387 Test Plan: The existing tests have been modified to work with the new interface. Reviewers: IslamAbdelRahman, kradhakrishnan, dhruba, igor Reviewed By: igor Subscribers: igor, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D50031 | 03 November 2015, 23:20:58 UTC |
| f31442f | Islam AbdelRahman | 02 November 2015, 22:56:11 UTC | Merge pull request #803 from SherlockNoMad/SkipFlush Add Option to Skip Flushing in TableBuilder | 02 November 2015, 22:56:11 UTC |
| dcc898b | Igor Canadi | 02 November 2015, 22:22:33 UTC | Merge pull request #812 from yuslepukhin/fix_windows_warnings Enable compiler warnings | 02 November 2015, 22:22:33 UTC |
| df7ed91 | SherlockNoMad | 02 November 2015, 22:12:29 UTC | Fix white space at end of line | 02 November 2015, 22:12:29 UTC |
| a0163c0 | Dmitri Smirnov | 02 November 2015, 22:11:28 UTC | Do not disable compiler warnings: C4101 'identifier' : unreferenced local variable C4189 'identifier' : local variable is initialized but not referenced C4100 'identifier' : unreferenced formal parameter C4296 'operator' : expression is always false | 02 November 2015, 22:11:28 UTC |
| 279c8e0 | Igor Canadi | 02 November 2015, 20:44:27 UTC | Merge pull request #811 from OverlordQ/unused-variable-warning Fix introduced in 2ab7065 was reverted by 18285c1. | 02 November 2015, 20:44:27 UTC |
| affd833 | Brent Garber | 02 November 2015, 20:35:45 UTC | Fix introduced in 2ab7065 was reverted by 18285c1. Corrects: db/memtablerep_bench.cc:135:22: error: ‘FLAGS_env’ defined but not used [-Werror=unused-variable] static rocksdb::Env* FLAGS_env = rocksdb::Env::Default(); ^ cc1plus: all warnings being treated as errors Makefile:1147: recipe for target 'db/memtablerep_bench.o' failed | 02 November 2015, 20:35:45 UTC |
| db68a2c | Igor Canadi | 02 November 2015, 18:09:47 UTC | Merge pull request #806 from yuslepukhin/signed_unsigned_warning Do not suppress C4018 'expression' : signed/unsigned mismatch | 02 November 2015, 18:09:47 UTC |
| ccc8c10 | SherlockNoMad | 31 October 2015, 01:33:01 UTC | Move skip_table_builder_flush to BlockBasedTableOption | 31 October 2015, 01:33:01 UTC |
| eaaf081 | Dmitri Smirnov | 31 October 2015, 00:03:16 UTC | Do not suppress C4018 'expression' : signed/unsigned mismatch The code compiles cleanly for the most part. Fix db_test. Move debug file to testutil library. | 31 October 2015, 00:03:16 UTC |
| ff4499e | Islam AbdelRahman | 30 October 2015, 23:38:10 UTC | Update DB::AddFile() to have less restrictions Summary: Update DB::AddFile() restrictions to be - Key range in loaded table file don't overlap with existing keys or tombstones in DB. - No other writes happen during AddFile call. The updated AddFile() will verify that the file key range don't overlap with any keys or tombstones in the DB, and then add the file to L0 Test Plan: unit tests Reviewers: igor, rven, anthony, kradhakrishnan, sdong Reviewed By: sdong Subscribers: adsharma, ameyag, dhruba Differential Revision: https://reviews.facebook.net/D49233 | 30 October 2015, 23:38:10 UTC |
| 11c71a3 | sdong | 30 October 2015, 21:42:18 UTC | db_bench: --compaction_pri default should be rocksdb::Options().compaction_pri Summary: Currently db_bnech's --compaction_pri default is set to be rocksdb::Options().compaction_style. Change it to rocksdb::Options().compaction_pri. Although, for now both is 0. Test Plan: Build db_bench Reviewers: anthony, rven, IslamAbdelRahman, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49773 | 30 October 2015, 22:02:33 UTC |
| 335e4ce | sdong | 30 October 2015, 18:50:28 UTC | options_test: fix a bug of assertion Summary: new_cf_opt.table_factory->Name() is char*, ASSERT_EQ doesn't work with char* directly. Construct a string using it. Test Plan: Run the test that failed. Reviewers: igor, kradhakrishnan, rven, IslamAbdelRahman, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49767 | 30 October 2015, 18:53:47 UTC |
| 66a3a87 | Siying Dong | 30 October 2015, 16:44:11 UTC | Merge pull request #797 from SherlockNoMad/optionHelper Support PlainTableOption in option_helper.cc | 30 October 2015, 16:44:11 UTC |
| 550af4e | SherlockNoMad | 30 October 2015, 05:41:57 UTC | Fix Travis Build Error | 30 October 2015, 05:41:57 UTC |
| a6dd083 | SherlockNoMad | 30 October 2015, 05:10:25 UTC | Add Option to Skip Flushing in TableBuilder | 30 October 2015, 05:10:25 UTC |
| 2872e0c | Islam AbdelRahman | 30 October 2015, 01:07:37 UTC | Clean and expose CreateLoggerFromOptions Summary: CreateLoggerFromOptions have some parameters like db_log_dir and env, these parameters are redundant since they already exist in DBOptions this patch remove the redundant parameters and expose CreateLoggerFromOptions to users Test Plan: make check Reviewers: igor, anthony, yhchiang, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, hermanlee4 Differential Revision: https://reviews.facebook.net/D49713 | 30 October 2015, 01:07:37 UTC |
| 296c3a1 | sdong | 29 October 2015, 22:52:32 UTC | "make format" in some recent commits Summary: Run "make format" for some recent commits. Test Plan: Build and run tests Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49707 | 30 October 2015, 00:11:14 UTC |
| 6388e7f | Siying Dong | 29 October 2015, 22:02:59 UTC | Merge pull request #798 from yuslepukhin/readahead_buffermanagement Implement smart buffer management in Windows Env. | 29 October 2015, 22:02:59 UTC |
| f4cbb90 | Islam AbdelRahman | 29 October 2015, 20:41:36 UTC | Merge pull request #799 from yuslepukhin/fix_random_generator_compile Fix compilation problem on Windows. | 29 October 2015, 20:41:36 UTC |
| 1277a48 | Dmitri Smirnov | 29 October 2015, 18:34:34 UTC | Fix 80 character limit issue. | 29 October 2015, 18:34:34 UTC |
| ee2c323 | Dmitri Smirnov | 29 October 2015, 18:29:18 UTC | Fix compilation problem on Windows. char is not a valid template parameter for std::uniform_int_distribution according to the standard. Replacing with int should be just fine. | 29 October 2015, 18:29:18 UTC |
| b69b9b6 | SherlockNoMad | 28 October 2015, 17:46:01 UTC | Support PlainTableOption in option_helper | 29 October 2015, 06:01:33 UTC |
| c97667d | Igor Canadi | 28 October 2015, 23:37:39 UTC | Fix RocksDB lite build for write_stress Summary: We don't have access to GetLiveFilesMetadata() in RocksDB lite. If compiling write_stress for lite, I skip the check for leaked files, which depends on this function. Test Plan: OPT=-DROCKSDB_LITE m write_stress Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49647 | 28 October 2015, 23:37:39 UTC |
| 0d720df | Herman Lee | 28 October 2015, 21:07:22 UTC | Use the correct variable when fetching table properties. Summary: An uninitialized parameter was being passed into the call to fetch the table properties during the compaction notification callbacks. Test Plan: Build it with myrocks and verify unit test passed. Run unit tests. Reviewers: rven, yhchiang, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D49635 | 28 October 2015, 23:28:11 UTC |
| 4b66d95 | Igor Canadi | 28 October 2015, 23:15:07 UTC | Write stress test Summary: The goal of this diff is to create a simple stress test with focus on catching: * bugs in compaction/flush processes, especially the ones that cause assertion errors * bugs in the code that deletes obsolete files There are two parts of the test: * write_stress, a binary that writes to the database * write_stress_runner.py, a script that invokes and kills write_stress Here are some interesting parts of write_stress: * Runs with very high concurrency of compactions and flushes (32 threads total) and tries to create a huge amount of small files * The keys written to the database are not uniformly distributed -- there is a 3-character prefix that mutates occasionally (in prefix mutator thread), in such a way that the first character mutates slower than second, which mutates slower than third character. That way, the compaction stress tests some interesting compaction features like trivial moves and bottommost level calculation * There is a thread that creates an iterator, holds it for couple of seconds and then iterates over all keys. This is supposed to test RocksDB's abilities to keep the files alive when there are references to them. * Some writes trigger WAL sync. This is stress testing our WAL sync code. * At the end of the run, we make sure that we didn't leak any of the sst files write_stress_runner.py changes the mode in which we run write_stress and also kills and restarts it. There are some interesting characteristics: * At the beginning we divide the full test runtime into smaller parts -- shorter runtimes (couple of seconds) and longer runtimes (100, 1000) seconds * The first time we run write_stress, we destroy the old DB. Every next time during the test, we use the same DB. * We can run in kill mode or clean-restart mode. Kill mode kills the write_stress violently. * We can run in mode where delete_obsolete_files_with_fullscan is true or false * We can run with low_open_files mode turned on or off. When it's turned on, we configure table cache to only hold a couple of files -- that way we need to reopen files every time we access them. Another goal was to create a stress test without a lot of parameters. So tools/write_stress_runner.py should only take one parameter -- runtime_sec and it should figure out everything else on its own. In a separate diff, I'll add this new test to our nightly legocastle runs. Test Plan: The goal of this test was to retroactively catch the following bugs: D33045, D48201, D46899, D42399. I failed to reproduce D48201, but all others have been caught! When i reverted https://reviews.facebook.net/D33045: ./write_stress --runtime_sec=200 --low_open_files_mode=true Iterator statuts not OK: IO error: /fast-rocksdb-tmp/rocksdb_test/write_stress/089166.sst: No such file or directory When i reverted https://reviews.facebook.net/D42399: python tools/write_stress_runner.py --runtime_sec=5000 Running write_stress, will kill after 5 seconds: ./write_stress --runtime_sec=-1 Running write_stress, will kill after 2 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --delete_obsolete_files_with_fullscan=true Running write_stress, will kill after 7 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false Running write_stress, will kill after 5 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false Running write_stress, will kill after 8 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --low_open_files_mode=true Write to DB failed: IO error: /fast-rocksdb-tmp/rocksdb_test/write_stress/019250.sst: No such file or directory ERROR: write_stress died with exitcode=-6 When i reverted https://reviews.facebook.net/D46899: python tools/write_stress_runner.py --runtime_sec=1000 runtime: 1000 Going to execute write stress for [3, 3, 100, 3, 2, 100, 1, 788] Running write_stress for 3 seconds: ./write_stress --runtime_sec=3 --low_open_files_mode=true Running write_stress for 3 seconds: ./write_stress --runtime_sec=3 --destroy_db=false --delete_obsolete_files_with_fullscan=true Running write_stress, will kill after 100 seconds: ./write_stress --runtime_sec=-1 --destroy_db=false --delete_obsolete_files_with_fullscan=true write_stress: db/db_impl.cc:2070: void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&): Assertion `log.getting_synced' failed. ERROR: write_stress died with exitcode=-6 Reviewers: IslamAbdelRahman, yhchiang, rven, kradhakrishnan, sdong, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49533 | 28 October 2015, 23:15:07 UTC |
| 47414c6 | sdong | 28 October 2015, 18:57:14 UTC | Move include/posix/io_posix.h to util/io_posix.h Summary: include/posix/io_posix.h is not a public API. Although include/posix/ is not a public header directory, it is confusing to put non-public headers to under include/. Move it to util/ to be clearer. Test Plan: Run all tests Reviewers: rven, IslamAbdelRahman, anthony, kradhakrishnan, yhchiang, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49611 | 28 October 2015, 19:15:51 UTC |
| 2889df8 | sdong | 28 October 2015, 18:55:20 UTC | Revert "Avoid to reply on ROCKSDB_FALLOCATE_PRESENT in include/posix/io_posix.h" This reverts commit c37223c0836d1637c628f9cef3acb5b55ad3d51b. | 28 October 2015, 18:55:20 UTC |
| 28c8758 | Siying Dong | 28 October 2015, 18:37:37 UTC | Merge pull request #795 from yuslepukhin/fix_mocktable_id Fix MockTable ID storage | 28 October 2015, 18:37:37 UTC |
| 5c8f2ee | Dmitri Smirnov | 28 October 2015, 17:53:14 UTC | Fix MockTable ID storage On Windows two tests fail that use MockTable: flush_job_test and compaction_job_test with the following message: compaction_job_test_je.exe : Assertion failed: result.size() == 4, file c:\dev\rocksdb\rocksdb\table\mock_table.cc, line 110 Investigation reveals that this failure occurs when a 4 byte ID written to a beginning of the physically open file (main contents remains in a in-memory map) can not be read back. The reason for the failure is that the ID is written directly to a WritableFile bypassing WritableFileWriter. The side effect of that is that pending_sync_ never becomes true so the file is never flushed, however, the direct cause of the failure is that the filesize_ member of the WritableFileWriter remains zero. At Close() the file is truncated to that size and the file becomes empty so the ID can not be read back. | 28 October 2015, 17:53:14 UTC |
| 72d6e75 | Islam AbdelRahman | 28 October 2015, 04:04:00 UTC | Fix WritableFileWriter::Append() return Summary: It looks like WritableFileWriter::Append() was returning OK() even when there is an error Test Plan: make check Reviewers: sdong, yhchiang, anthony, rven, kradhakrishnan, igor Reviewed By: igor Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D49569 | 28 October 2015, 04:04:00 UTC |
| d0a18c2 | Siying Dong | 27 October 2015, 23:51:22 UTC | Merge pull request #786 from aloukissas/unused_param Fix unused parameter warnings in db.h | 27 October 2015, 23:51:22 UTC |
| c37223c | sdong | 27 October 2015, 21:27:48 UTC | Avoid to reply on ROCKSDB_FALLOCATE_PRESENT in include/posix/io_posix.h Summary: include/posix/io_posix.h should not depend on ROCKSDB_FALLOCATE_PRESENT. Remove it. Test Plan: Build it with both of ROCKSDB_FALLOCATE_PRESENT defined and not defined. Reviewers: rven, yhchiang, anthony, kradhakrishnan, IslamAbdelRahman, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49563 | 27 October 2015, 23:48:40 UTC |
| 6fbc4f9 | Dmitri Smirnov | 27 October 2015, 21:44:16 UTC | Implement smart buffer management. introduce a new DBOption random_access_max_buffer_size to limit the size of the random access buffer used for unbuffered access. Implement read ahead buffering when enabled. To that effect propagate compaction_readahead_size and the new option to the env options to make it available for the implementation. Add Hint() override so SetupForCompaction() call would call Hint() readahead can now be setup from both Hint() and EnableReadAhead() Add new option random_access_max_buffer_size support db_bench, options_helper to make it string parsable and the unit test. | 27 October 2015, 21:44:16 UTC |
| d6219e4 | sdong | 27 October 2015, 19:15:55 UTC | Mac build break caused by include/posix/io_posix.h not declearing errno, Summary: Mac build breaks as include/posix/io_posix.h doesn't include errno. Move the exact function declaration to io_posix.cc Test Plan: Run all test. Will run on Mac Reviewers: rven, anthony, yhchiang, IslamAbdelRahman, igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49551 | 27 October 2015, 21:16:19 UTC |
| beb69d4 | Siying Dong | 27 October 2015, 19:13:01 UTC | Merge pull request #765 from PraveenSinghRao/wal_filter Adding wal filter to inspect and filter wal records on recovery | 27 October 2015, 19:13:01 UTC |
| ab0f3b9 | sdong | 26 October 2015, 23:02:32 UTC | crash_test to trigger some less frequent crash point more frequently Summary: crash_test still has a very low chance to hit some crash point. Have another mode for covering them more likely. Test Plan: Run crash_test and see db_stress is called with expected prameters. Reviewers: kradhakrishnan, igor, anthony, rven, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49473 | 27 October 2015, 19:06:06 UTC |
| 7beb743 | Igor Canadi | 27 October 2015, 16:58:07 UTC | Merge pull request #778 from Vaisman/master Error while cmake by building from zip-archive | 27 October 2015, 16:58:07 UTC |
| 4ce117c | Praveen Rao | 27 October 2015, 02:03:34 UTC | Merge branch 'master' into wal_filter | 27 October 2015, 02:03:34 UTC |
| 32cdec6 | Praveen Rao | 27 October 2015, 01:11:18 UTC | Fail recovery if filter provides more records than original and corresponding unit-test, fix naming conventions | 27 October 2015, 01:11:18 UTC |
| 44d4057 | sdong | 23 October 2015, 16:16:46 UTC | Avoid some includes in io_posix.h Summary: IO Posix depends on too many .h files. Move most of them to .cc files. Test Plan: make all Reviewers: anthony, rven, IslamAbdelRahman, yhchiang, kradhakrishnan, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49479 | 27 October 2015, 00:00:25 UTC |
| 2adad23 | Alex Loukissas | 26 October 2015, 23:38:14 UTC | Fix unused parameter warnings. | 26 October 2015, 23:38:37 UTC |
| b0980ff | Alex Loukissas | 26 October 2015, 23:00:51 UTC | Fix unused parameter warnings. | 26 October 2015, 23:32:14 UTC |
| bc898c5 | Alex Loukissas | 26 October 2015, 23:00:51 UTC | Fix unused parameter warnings. | 26 October 2015, 23:00:51 UTC |
| 138876a | Siying Dong | 26 October 2015, 22:01:28 UTC | Merge pull request #746 from ceph/wip-recycle Add Options.recycle_log_file_num for Recycling WAL Files | 26 October 2015, 22:01:28 UTC |
| 581f20f | Islam AbdelRahman | 26 October 2015, 18:50:29 UTC | Add LITE tests to Legocastle Summary: Update rocksdb-lego-determinator to include running make check under ROCKSDB_LITE Test Plan: will be tested after landing in fbcode Reviewers: sdong, yhchiang, igor, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D49065 | 26 October 2015, 18:50:29 UTC |
| 3d56d86 | Vasili Svirski | 24 October 2015, 17:05:59 UTC | Merge remote-tracking branch 'upstream/master' | 24 October 2015, 17:05:59 UTC |
| d691111 | sdong | 23 October 2015, 14:36:22 UTC | include/posix/io_posix.h should have a once declartion Summary: include/posix/io_posix.h doesn't not prevent multiple includes. Need to fix it. It is also breaking unity build. Test Plan: Run unity build and see error go away. Reviewers: rven, igor, IslamAbdelRahman, kradhakrishnan, anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49281 | 23 October 2015, 14:45:00 UTC |
| a6962ed | Igor Canadi | 23 October 2015, 00:14:52 UTC | Merge pull request #783 from yuslepukhin/remove_test_conditional_compilation No need to #ifdef test only code on windows | 23 October 2015, 00:14:52 UTC |
| 3c750b5 | Dmitri Smirnov | 22 October 2015, 22:15:37 UTC | No need to #ifdef test only code on windows | 22 October 2015, 22:15:37 UTC |
| 8c11c5d | Siying Dong | 22 October 2015, 20:33:51 UTC | Merge pull request #768 from OpenChannelSSD/to_fb_master2 Split posix storage backend into Env and library | 22 October 2015, 20:33:51 UTC |
| 6e6dd5f | Javier González | 14 October 2015, 08:14:53 UTC | Split posix storage backend into Env and library Summary: This patch splits the posix storage backend into Env and the actual *File implementations. The motivation is to allow other Envs to use posix as a library. This enables a storage backend different from posix to split its secondary storage between a normal file system partition managed by posix, and it own media. Test Plan: No new functionality is added to posix Env or the library, thus the current tests should suffice. | 22 October 2015, 15:31:31 UTC |
| 980a82e | Alexey Maykov | 22 October 2015, 01:34:25 UTC | Fix a bug in GetApproximateSizes Summary: Need to pass through the memtable parameter. Test Plan: built, tested through myrocks Reviewers: igor, sdong, rven Reviewed By: rven Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D49167 | 22 October 2015, 01:34:39 UTC |
| d0d13eb | Shusen Liu | 20 October 2015, 18:31:27 UTC | fix bug in db_crashtest.py Summary: in tools/db_crashtest.py, cmd_params['db'] by default is a lambda expression, not the actual db_name. fix by get the db_name before passing it to gen_cmd. Test Plan: run `make crashtest` Reviewers: sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D49119 | 21 October 2015, 05:01:11 UTC |
| 01a41af | Vasili Svirski | 21 October 2015, 03:52:10 UTC | Merge remote-tracking branch 'upstream/master' | 21 October 2015, 03:52:10 UTC |
| 5678c05 | Yueh-Hsuan Chiang | 21 October 2015, 00:09:09 UTC | Use DEBUG_LEVEL=0 in make release and make clean Summary: Use DEBUG_LEVEL=0 in make release and make clean Test Plan: make clean make release -j32 Reviewers: MarkCallaghan, sdong, anthony, IslamAbdelRahman, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49125 | 21 October 2015, 00:09:09 UTC |
| ac25fe6 | Siying Dong | 20 October 2015, 23:32:21 UTC | Merge pull request #779 from yuslepukhin/optimize_windows_build Do not build test only code and unit tests in Release builds | 20 October 2015, 23:32:21 UTC |
| e154ee0 | Dmitri Smirnov | 20 October 2015, 20:35:08 UTC | Do not build test only code and unit tests in Release builds Test code errors are currently blocking Windows Release builew We do not want spend time building in Release what we can not run We want to eliminate a source of most frequent errors when people check-in test only code which can not be built in Release. This feature will work only if you invoke msbuild against rocksdb.sln Invoking it against ALL_BUILD target will attempt to build everything. | 20 October 2015, 20:35:08 UTC |
| cd3286f | Vasili Svirski | 20 October 2015, 18:27:48 UTC | Error while cmake by building from zip-archive * add validation is git found * add validation is .git folder exists in project (project zip archive download without .git folder) * get head commit SHA if git found and .git folder exists Tested: * configure project by CMake 3.0.0 successfully (with and without git), with project zip archive (without .git folder) and with project cloned from github * configure project by command: cmake -G "Visual Studio 12 Win64" * build solution by Visual Studio * manually validate that file utils/build_version.cc contains valid head revision value | 20 October 2015, 18:51:19 UTC |
| e3d4e14 | sdong | 20 October 2015, 06:40:15 UTC | DBCompactionTestWithParam.ManualCompaction to verify block cache is not filled in manual compaction Summary: Manual compaction should not fill block cache. Add the verification in unit test Test Plan: Run the test Reviewers: yhchiang, kradhakrishnan, rven, IslamAbdelRahman, anthony, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49089 | 20 October 2015, 17:36:49 UTC |
| 033c6f1 | Shusen Liu | 20 October 2015, 03:58:38 UTC | T7916298, bug fix Summary: dbname => cmd_params['db'] Test Plan: Run `make crash_test` Reviewers: sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D49077 | 20 October 2015, 04:09:35 UTC |
| 7717ad1 | krad | 20 October 2015, 03:13:14 UTC | Adding artifacts to stress_crash CI job Summary: Adding the ability to upload logs and db content to storage after the completion of the job Test Plan: Manual run Reviewers: CC: leveldb@ Task ID: #8754201 Blame Rev: | 20 October 2015, 03:13:14 UTC |
| 0bf656b | Igor Canadi | 20 October 2015, 01:47:59 UTC | Don't spew warnings when flint doesn't exist Summary: Before this diff `arc lint` on non-fb machine issued warnings. Now it doesn't. Test Plan: `arc lint` is quiet. Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D49071 | 20 October 2015, 01:47:59 UTC |
| 6d6776f | sdong | 19 October 2015, 20:07:05 UTC | Log more information for the add file with overlapping range failure Summary: crash_test sometimes fails, hitting the add file overlapping assert. Add information in info logs help us to find the bug. Test Plan: Run all test suites. Do some manual tests to make sure printing is correct. Reviewers: kradhakrishnan, yhchiang, anthony, IslamAbdelRahman, rven, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D49017 | 20 October 2015, 00:31:13 UTC |
| 7951b9b | Praveen Rao | 20 October 2015, 00:03:01 UTC | make field order match initialization order | 20 October 2015, 00:03:01 UTC |
| 90228bb | Siying Dong | 19 October 2015, 22:26:22 UTC | Merge pull request #771 from maximecaron/patch-1 Fix build error using Visual Studio 12 | 19 October 2015, 22:26:22 UTC |
| 2938c5c | Praveen Rao | 19 October 2015, 22:21:33 UTC | merge upstream changes | 19 October 2015, 22:21:33 UTC |
