9887661 | Adam Retter | 17 August 2016, 15:14:30 UTC | Fix the Windows build of RocksDB Java. Similar to https://github.com/facebook/rocksdb/issues/1220 | 17 August 2016, 15:14:30 UTC |
deda159 | Philipp Unterbrunner | 12 August 2016, 23:34:11 UTC | Added min/max/avg data block size output to sst_dump Summary: Added min/max/avg data block size output to sst_dump. Output was added to the end of BlockBasedTable::DumpDataBlocks, so it appears after the data block details, at the very end of the dump file. Test Plan: ``` ./db_bench --benchmarks=fillrandom ./sst_dump --file=/tmp/rocksdbtest-xyz/dbbench/000007.sst --command=raw tail -n 6 /tmp/rocksdbtest-xyz/dbbench/000007_dump.txt ``` ``` Data Block Summary: -------------------------------------- # data blocks: 11336 min data block size: 903 max data block size: 2268 avg data block size: 2245.363356 ``` Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61815 | 12 August 2016, 23:34:11 UTC |
e408e98 | Aaron Gao | 12 August 2016, 21:16:57 UTC | add Name() to Cache Summary: preparation for detecting Cache type. If SimCache, we then may trigger some command like "setSimCapacity()" with setOptions() Test Plan: make all check Reviewers: yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61953 | 12 August 2016, 21:16:57 UTC |
a297643 | Islam AbdelRahman | 12 August 2016, 06:34:19 UTC | Fix valgrind memory leak | 12 August 2016, 06:34:19 UTC |
d11c09d | Islam AbdelRahman | 12 August 2016, 02:10:16 UTC | Eliminate memcpy from ForwardIterator Summary: This diff update ForwardIterator to support pinning keys and values, which will allow DBIter to take advantage of that and eliminate memcpy when executing merge operators This diff is stacked on D61305 Test Plan: existing tests (updated them to test tailing iterator) new test Reviewers: andrewkr, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60009 | 12 August 2016, 02:10:16 UTC |
d367555 | Adam Retter | 14 June 2016, 15:52:25 UTC | Added further Java API options for controlling concurrent writes | 11 August 2016, 23:33:20 UTC |
ebdfe34 | Adam Retter | 14 June 2016, 15:51:56 UTC | Exposed further Java API options for controlling compaction | 11 August 2016, 23:33:20 UTC |
d1be594 | Dhruba Borthakur | 10 August 2016, 23:21:37 UTC | Improve documentation of SliceTransform. Test Plan: make check Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61875 | 11 August 2016, 22:10:09 UTC |
6056d63 | Yueh-Hsuan Chiang | 11 August 2016, 21:54:29 UTC | Improve comment and bug fix for GetOptionsFromMap functions in convenience.h Summary: This diff improves the documentation for GetOptionsFromMap APIs and fixes a bug in GetOptionsFromMap functions in convenience.h where new_options will still be changed when the function call is not successful. Test Plan: options_test Reviewers: IslamAbdelRahman, kradhakrishnan, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61731 | 11 August 2016, 21:54:29 UTC |
76a67cf | Aaron Gao | 11 August 2016, 21:19:33 UTC | support stackableDB as the baseDB of transactionDB Summary: make transactionDB working with StackableDB Test Plan: make all check -j64 Reviewers: andrewkr, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60705 | 11 August 2016, 21:19:33 UTC |
67c1ae8 | krad | 10 August 2016, 20:26:15 UTC | Travis build break fix Summary: Travis times out with persistent_cache_test, there is no way to increase the timeout, so lower the test case for Travis Test Plan: Run on travis and sandcastle Subscribers: leveldb, dhruba, andrewkr Differential Revision: https://reviews.facebook.net/D61863 | 11 August 2016, 20:02:32 UTC |
b693ba6 | Islam AbdelRahman | 11 August 2016, 18:54:17 UTC | Minor PinnedIteratorsManager Refactoring Summary: This diff include these simple change - Rename ReleasePinnedIterators to ReleasePinnedData - Rename PinIteratorIfNeeded to PinIterator - Use std::vector directly in PinnedIteratorsManager instead of std::unique_ptr<std::vector> - Generalize PinnedIteratorsManager by adding PinPtr which can pin any pointer Test Plan: existing tests Reviewers: sdong, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61305 | 11 August 2016, 18:54:17 UTC |
db3dfb1 | Adam Retter | 11 August 2016, 17:09:43 UTC | Fixes for arcanist config (#1271) * Update arc config for ArcanistBaseUnitTestEngine -> ArcanistUnitTestEngine Test Plan: Execute Java test suite Reviewers: yhchiang Differential Revision: https://reviews.facebook.net/D61911 * Fix for arc use of base64 command on Max OS X Test Plan: Run on `arc diff` on Mac OS X and Linux Reviewers: yhchiang Differential Revision: https://reviews.facebook.net/D61917 | 11 August 2016, 17:09:43 UTC |
87c91bd | krad | 05 May 2016, 18:50:30 UTC | Persistent Read Cache (8) Benchmark tooling Summary: Adding benchmark tool for persistent read cache. TODO: Add integration to db_bench Test Plan: Compile Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57711 | 11 August 2016, 00:49:22 UTC |
2914de6 | Aaron Gao | 11 August 2016, 00:42:24 UTC | add sim_cache stats to Statistics Summary: add SIM_BLOCK_CACHE_HIT and SIM_BLOCK_CACHE_MISS tickers. maybe can be combined with Histograms like DB_GET to evaluate the current setting of the size of block cache. Test Plan: make all check Reviewers: sdong, andrewkr, IslamAbdelRahman, yiwu Reviewed By: yiwu Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61803 | 11 August 2016, 00:42:24 UTC |
8b79422 | sdong | 09 August 2016, 17:16:32 UTC | [Proof-Of-Concept] RocksDB Blob Storage with a blob log file. Summary: This is a proof of concept of a RocksDB blob log file. The actual value of the Put() is appended to a blob log using normal data block format, and the handle of the block is written as the value of the key in RocksDB. The prototype only supports Put() and Get(). It doesn't support DB restart, garbage collection, Write() call, iterator, snapshots, etc. Test Plan: Add unit tests. Reviewers: arahut Reviewed By: arahut Subscribers: kradhakrishnan, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61485 | 11 August 2016, 00:05:17 UTC |
4beffe0 | sdong | 10 August 2016, 17:58:15 UTC | Fix test data race in two FaultInjectionTest tests Summary: Background sleeping tasks may conflict with test cleaning up. Wait for the sleeping tasks to finish before ending the test. Test Plan: Run these tests. Reviewers: andrewkr, yiwu Reviewed By: yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61827 | 10 August 2016, 20:56:50 UTC |
821bcb0 | Willem Jan Withagen | 10 August 2016, 20:52:23 UTC | util/arena.cc: FreeBSD: More portable use of mmap(MAP_ANON) (#1254) From the Linux manual: MAP_ANONYMOUS The mapping is not backed by any file; its contents are initialized to zero. The fd and offset arguments are ignored; however, some implementations require fd to be -1 if MAP_ANONYMOUS (or MAP_ANON) is specified, and portable applications should ensure this. FreeBSD is such a case, it wil just return an error. | 10 August 2016, 20:52:23 UTC |
5370f44 | sdong | 10 August 2016, 20:20:55 UTC | Increase RocksDB version Summary: RocksDB 4.11 has been branched out. Now need to move to 4.12. Test Plan: N/A Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61845 | 10 August 2016, 20:25:31 UTC |
56dd034 | sdong | 04 August 2016, 00:42:06 UTC | read_options.background_purge_on_iterator_cleanup to cover forward iterator and log file closing too. Summary: With read_options.background_purge_on_iterator_cleanup=true, File deletion and closing can still happen in forward iterator, or WAL file closing. Cover those cases too. Test Plan: I am adding unit tests. Reviewers: andrewkr, IslamAbdelRahman, yiwu Reviewed By: yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61503 | 10 August 2016, 20:16:41 UTC |
ccecf3f | Islam AbdelRahman | 10 August 2016, 19:37:43 UTC | UniversalCompaction should ignore sorted runs being compacted (when compacting for file num) Summary: If we have total number of sorted runs greater than level0_file_num_compaction_trigger, Universal compaction will always issue a compaction even if the number of sorted runs that are not being compacted is less than level0_file_num_compaction_trigger. This diff changes this behaviour to relay on the `number of sorted runs not being compacted` instead of `total number of sorted runs` Test Plan: New unit test Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61533 | 10 August 2016, 19:37:43 UTC |
1b0069c | Andrew Kryczka | 10 August 2016, 18:08:09 UTC | Remove non-gtest from parallelized tests Summary: compact_on_deletion_collector_test does not support --gtest_list_tests since it isn't gtest, so the full program would run for the target gen_parallel_tests. This caused gen_parallel_tests to take 8+ minutes for tsan and prevented compact_on_deletion_collector_test from running during check_0 since no t/run-* script could be generated. Test Plan: run make check, verify generating t/run-* scripts is fast and ./compact_on_deletion_collector_test is now run Reviewers: IslamAbdelRahman, wanning, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61695 | 10 August 2016, 18:08:09 UTC |
638c49f | sdong | 05 August 2016, 22:55:34 UTC | Change HISTORY.md for release 4.11 Summary: Need to change HISTORY.md for 4.11. 4.10 was not updated either. Update it together. Test Plan: Not needed. Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61665 | 10 August 2016, 17:44:24 UTC |
6b8e9c6 | Jay | 10 August 2016, 16:08:13 UTC | fix vs generator (#1269) | 10 August 2016, 16:08:13 UTC |
c38b075 | Wanning Jiang | 10 August 2016, 04:12:45 UTC | Update HISTORY.md | 10 August 2016, 04:12:45 UTC |
8f399e3 | Wanning Jiang | 10 August 2016, 03:57:40 UTC | Update HISTORY.md | 10 August 2016, 03:57:40 UTC |
98d0b78 | Zongzhi Chen | 10 August 2016, 01:14:13 UTC | Added check_snapshot option in the DB's AddFile function (#1261) * Added check_snapshot option in the DB's AddFile function * change check_snapshot to skip_snapshot_check * add unit test for skip_snapshot_check * Add skip_snapshot_check comment | 10 August 2016, 01:14:13 UTC |
9fd68b7 | Andrew Kryczka | 09 August 2016, 05:38:04 UTC | set travis open file descriptor limit Summary: max is 8192 on mac and this value seems to prevent the tests from running out of open fds while running Test Plan: committed to separate branch to trigger test runs: https://travis-ci.org/facebook/rocksdb/builds/150817199 Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61767 | 09 August 2016, 05:38:04 UTC |
59ddb50 | krad | 08 August 2016, 22:12:48 UTC | Fix travis build break Summary: Travis provides a very low capability resource for testing and the persistent cache tests are too aggressive. It causes the kernel to not have enough time to do file-system meta data sync operation, causing open file descriptors to increase and available space to decrease. This change skips certain tests Test Plan: Run on travis Subscribers: leveldb, dhruba, andrewkr Differential Revision: https://reviews.facebook.net/D61755 | 09 August 2016, 01:04:45 UTC |
f4d9863 | Adam Retter | 06 August 2016, 19:03:47 UTC | Added SetOptions support to RocksJava (#1243) * [refactor] Split Java ColumnFamilyOptions into mutable and immutable and implement any missing immutable options * [feature] Implement RocksDB#setOptions | 06 August 2016, 19:03:47 UTC |
7882cb9 | Yi Wu | 05 August 2016, 23:45:57 UTC | Make DBOptionsTest::EnableAutoCompactionAndTriggerStall less falky Summary: Explicitly flush two times to generate two sst files. Test Plan: run the test. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61671 | 05 August 2016, 23:45:57 UTC |
44f5cc5 | omegaga | 05 August 2016, 22:56:22 UTC | Add time series database (resubmitted) Summary: Implement a time series database that supports DateTieredCompactionStrategy. It wraps a db object and separate SST files in different column families (time windows). Test Plan: Add `date_tiered_test`. Reviewers: dhruba, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61653 | 05 August 2016, 22:56:22 UTC |
7c4615c | sdong | 14 July 2016, 04:08:15 UTC | A utility function to help users migrate DB after options change Summary: Add a utility function that trigger necessary full compaction and put output to the correct level by looking at new options and old options. Test Plan: Add unit tests for it. Reviewers: andrewkr, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: muthu, sumeet, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60783 | 05 August 2016, 22:39:55 UTC |
5bb0a7f | Siying Dong | 05 August 2016, 19:33:33 UTC | Update appveyor.yml | 05 August 2016, 19:33:33 UTC |
86396cc | Siying Dong | 05 August 2016, 19:27:50 UTC | Update appveyor.yml | 05 August 2016, 19:27:50 UTC |
c1db098 | Siying Dong | 05 August 2016, 19:24:24 UTC | Update appveyor.yml | 05 August 2016, 19:24:24 UTC |
7da2eaf | Siying Dong | 05 August 2016, 19:22:42 UTC | Update appveyor.yml | 05 August 2016, 19:22:42 UTC |
34723b4 | Dhruba Borthakur | 04 August 2016, 23:02:50 UTC | Cleanup unused variable pending_fsync_. Summary: Cleanup unused variable pending_fsync_. Test Plan: make check Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61581 | 05 August 2016, 17:31:41 UTC |
7cc0dbd | Andrew Kryczka | 05 August 2016, 06:06:28 UTC | cat all logs in sandcastle output Summary: Since we enabled parallelism in valgrind the logs now go to t/valgrind_log-*, which doesn't match our pattern t/log-*, so those files don't get printed in the sandcastle output. For this regex we really just want to exclude t/run-* (these are shell scripts), so let's do that explicitly. Test Plan: ran the commands locally, will also look at sandcastle results on this diff which will include these changes Reviewers: arahut, sdong, IslamAbdelRahman, kradhakrishnan, wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61557 | 05 August 2016, 06:06:28 UTC |
9253767 | Adam Retter | 05 August 2016, 03:16:16 UTC | Correct geHistogramData() -> getHistogramData() (#1257) | 05 August 2016, 03:16:16 UTC |
f35b16f | sdong | 03 August 2016, 23:02:41 UTC | db_bench add an option of --base_background_compactions Summary: As title. Test Plan: Run the benchmark with and without the parameter. Reviewers: yiwu, andrewkr, lightmark, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61491 | 04 August 2016, 18:06:00 UTC |
c3a4bea | omegaga | 03 August 2016, 22:19:35 UTC | Fix flaky test `ObsoleteFiles` Summary: The test `ObsoleteFiles` failed occasionally on slow device. This problem appears on Travis CI several times. The reason is that we did not wait until compaction jobs are finished in the test, while in slower device the background jobs take longer time to finish. Test Plan: Pass existing tests. Reviewers: yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61479 | 03 August 2016, 22:19:35 UTC |
8234faa | omegaga | 03 August 2016, 21:23:21 UTC | Fix failed test Summary: Fix failed persistent cache test introduced by recent PR to support cross-platform build. Test Plan: Run persistent_cache_test (on multiple platforms) Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61467 | 03 August 2016, 21:23:21 UTC |
4990c0d | Islam AbdelRahman | 03 August 2016, 18:07:53 UTC | Remove deprecated LEVELDB_PLATFORM_POSIX Summary: We have replaced LEVELDB_PLATFORM_POSIX with ROCKSDB_PLATFORM_POSIX in our code replace it in Makefile Test Plan: make check Reviewers: sdong, yiwu, andrewkr, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61365 | 03 August 2016, 18:07:53 UTC |
7323e4c | omegaga | 03 August 2016, 17:41:27 UTC | Fix clang on macOS Summary: Fix problem when casting `size_t` to `uint64_t`. Avoid usage of `size_t` when possible. Test Plan: Pass related test on Travis CI. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61443 | 03 August 2016, 17:41:27 UTC |
ee027fc | Yi Wu | 03 August 2016, 04:55:26 UTC | Ignore write stall triggers when auto-compaction is disabled Summary: My understanding is that the purpose of write stall triggers are to wait for auto-compaction to catch up. Without auto-compaction, we don't need to stall writes. Also with this diff, flush/compaction conditions are recalculated on dynamic option change. Previously the conditions are recalculate only when write stall options are changed. Test Plan: See the new test. Removed two tests that are no longer valid. Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61437 | 03 August 2016, 04:55:26 UTC |
e4609a7 | Dmitri Smirnov | 03 August 2016, 00:15:18 UTC | Fix Windows build issues (#1253) | 03 August 2016, 00:15:18 UTC |
2306167 | omegaga | 02 August 2016, 22:16:39 UTC | Fix clang build failure and refactor unit test Summary: <endian.h> is not platform independent. Switch to our own endianness transformation function instead. Test Plan: Pass Travis CI. Refactor tests to make sure endianness transformation runs properly. Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61389 | 02 August 2016, 22:16:39 UTC |
343304e | Aaron Gao | 02 August 2016, 21:53:29 UTC | Use StopWatch to do statistic job in db_impl_add_file.cc Summary: patch for diff https://reviews.facebook.net/D58587 Also change StopWatch class to add a fifth param named overwrite which decides whether to overwrite *elapse or add on it. Test Plan: make all check -j64 Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61239 | 02 August 2016, 21:53:29 UTC |
cdc4eb6 | Jay Edgar | 25 July 2016, 22:06:11 UTC | Add a GetComparator() function to the ColumnFamilyHandle base class so that the user's comparator can be retrieved. Summary: MyRocks is adding support for the user of the SstFileWriter which needs a comparator. It would be more convenient to get the comparator from the column family (which already has to have it) than to have caller keep track of it. Test Plan: Standard tests (adding one for the new method) Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61155 | 02 August 2016, 21:34:57 UTC |
712dd27 | krad | 01 August 2016, 23:55:01 UTC | Build break fixes Summary: Regex support for c++ is very inconsistent across compilers, converting the logic to simple string manipulation. Test Plan: Local test Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61377 | 02 August 2016, 17:30:29 UTC |
0155c73 | Islam AbdelRahman | 02 August 2016, 00:22:07 UTC | Fix parallel tests `make check -j` Summary: parallel tests are broken because gnu_parallel is reading deprecated options from `/etc/parallel/config` Fix this by passing `--plain` to ignore `/etc/parallel/config` Test Plan: make check -j64 Reviewers: kradhakrishnan, sdong, andrewkr, yiwu, arahut Reviewed By: arahut Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61359 | 02 August 2016, 00:22:07 UTC |
c49ea68 | krad | 01 August 2016, 23:29:33 UTC | Fix to enable running CI jobs locally Summary: We may not have permission on all /dev/shm to fix the sticky bit. Making the sticky bit fix advisory. Test Plan: Run CI job locally Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61371 | 01 August 2016, 23:31:49 UTC |
726c2f7 | krad | 01 August 2016, 22:22:58 UTC | Build break fix Summary: Fixing build break on Mac (1) uint64_t fix (2) O_DIRECT works only for Linux Test Plan: Build and test on Mac and Unix Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61353 | 01 August 2016, 22:24:13 UTC |
d51dc96 | omegaga | 01 August 2016, 21:50:19 UTC | Experiments on column-aware encodings Summary: Experiments on column-aware encodings. Supported features: 1) extract data blocks from SST file and encode with specified encodings; 2) Decode encoded data back into row format; 3) Directly extract data blocks and write in row format (without prefix encoding); 4) Get column distribution statistics for column format; 5) Dump data blocks separated by columns in human-readable format. There is still on-going work on this diff. More refactoring is necessary. Test Plan: Wrote tests in `column_aware_encoding_test.cc`. More tests should be added. Reviewers: sdong Reviewed By: sdong Subscribers: arahut, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60027 | 01 August 2016, 21:50:19 UTC |
c116b47 | krad | 03 May 2016, 18:05:42 UTC | Persistent Read Cache (part 6) Block Cache Tier Implementation Summary: The patch is a continuation of part 5. It glues the abstraction for file layout and metadata, and flush out the implementation of the API. It adds unit tests for the implementation. Test Plan: Run unit tests Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57549 | 01 August 2016, 21:15:14 UTC |
64046e5 | omegaga | 01 August 2016, 19:28:51 UTC | Write a benchmark to emulate time series data Summary: Add a benchmark to `db_bench`. In this benchmark, a write thread will populate time series data in the format of 'id | timestamp', and multiple read threads will randomly retrieve all data from one id at a time. Test Plan: Run the benchmark: `num=134217728;bpl=536870912;mb=67108864;overlap=10;mcz=2;del=300000000;levels=6;ctrig=4;delay=8;stop=12;wbn=3;mbc=20;wbs=134217728;dds=0;sync=0;t=32;vs=800;bs=4096;cs=17179869184;of=500000;wps=0;si=10000000; kir=100000; dir=/data/users/jhli/test/; ./db_bench --benchmarks=timeseries --disable_seek_compaction=1 --mmap_read=0 --statistics=1 --histogram=1 --num=$num --threads=$t --value_size=$vs --block_size=$bs --cache_size=$cs --bloom_bits=10 --cache_numshardbits=6 --open_files=$of --verify_checksum=1 --db=$dir --sync=$sync --disable_wal=0 --compression_type=none --stats_interval=$si --compression_ratio=1 --disable_data_sync=$dds --write_buffer_size=$wbs --target_file_size_base=$mb --max_write_buffer_number=$wbn --max_background_compactions=$mbc --level0_file_num_compaction_trigger=$ctrig --level0_slowdown_writes_trigger=$delay --level0_stop_writes_trigger=$stop --num_levels=$levels --delete_obsolete_files_period_micros=$del --min_level_to_compress=$mcz --max_grandparent_overlap_factor=$overlap --stats_per_interval=1 --max_bytes_for_level_base=$bpl --use_existing_db=0 --key_id_range=$kir` Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: lgalanis, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60651 | 01 August 2016, 19:28:51 UTC |
9ae92f5 | Adam Retter | 29 July 2016, 19:55:54 UTC | More granular steps in the Makefile, can help with running all or single Java tests (and with ASAN build - https://github.com/facebook/rocksdb/wiki/JNI-Debugging) (#1237) | 29 July 2016, 19:55:54 UTC |
7c01d65 | Islam AbdelRahman | 29 July 2016, 19:47:15 UTC | [Fix Java] Remove duplicate cases in LoggerJniCallback::Logv Summary: Remove duplicate cases in LoggerJniCallback::Logv and fix complication error Test Plan: make jclean rocksdbjava jtest -j4 Reviewers: adamretter, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61335 | 29 July 2016, 19:47:15 UTC |
8796934 | Adam Retter | 29 July 2016, 19:12:03 UTC | Added missing Java ReadOptions settings (#1109) * Minor reorganisation * Java API - Added missing ReadOptions settings | 29 July 2016, 19:12:03 UTC |
5e2c796 | Islam AbdelRahman | 29 July 2016, 18:42:28 UTC | Make DBTest.CompressionStatsTest more deterministic Summary: DBTest.CompressionStatsTest on non_shm test where the storage device is slow DBTest.CompressionStatsTest assumes that a flush happens to check the number of compressed blocks. This is not always true if the Flush is slow, make the test more deterministic by forcing a flush before doing the check Test Plan: Run the test locally Reviewers: andrewkr, yiwu, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61317 | 29 July 2016, 18:42:28 UTC |
557748f | Islam AbdelRahman | 29 July 2016, 18:02:03 UTC | Fix db_stress failure (pass merge_operator even if not used) Summary: db_stress test is now failing because of this scenario - run db_stress with merge_operator enabled (now we have a db with merge operands) - run db_stress with merge_operator disabled (now when we fail to open the db) the solution is to pass the merge_operator to the DB even if we are not going to do any merge operations Test Plan: Check the failure Reviewers: andrewkr, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61311 | 29 July 2016, 18:02:03 UTC |
811ee21 | Adam Retter | 28 July 2016, 23:54:06 UTC | Bugfix to ensure that logging can be achieved from threads that are not known to the JVM (#1106) | 28 July 2016, 23:54:06 UTC |
afad5bd | Andrew Kryczka | 28 July 2016, 21:55:22 UTC | Simplify thread-local static initialization Summary: The call stack used to look like this during static initialization: #0 0x00000000008032d1 in rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta() (this=0x7ffff683b300) at util/thread_local.cc:172 #1 0x00000000008030a7 in rocksdb::ThreadLocalPtr::Instance() () at util/thread_local.cc:135 #2 0x000000000080310f in rocksdb::ThreadLocalPtr::StaticMeta::Mutex() () at util/thread_local.cc:141 #3 0x0000000000803103 in rocksdb::ThreadLocalPtr::StaticMeta::InitSingletons() () at util/thread_local.cc:139 #4 0x000000000080305d in rocksdb::ThreadLocalPtr::InitSingletons() () at util/thread_local.cc:106 It involves outer/inner classes and the call stacks goes outer->inner->outer->inner, which is too difficult to understand. We can avoid a level of back-and-forth by skipping StaticMeta::InitSingletons(), which doesn't initialize anything beyond what ThreadLocalPtr::Instance() already initializes. Now the call stack looks like this during static initialization: #0 0x00000000008032c5 in rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta() (this=0x7ffff683b300) at util/thread_local.cc:170 #1 0x00000000008030a7 in rocksdb::ThreadLocalPtr::Instance() () at util/thread_local.cc:135 #2 0x000000000080305d in rocksdb::ThreadLocalPtr::InitSingletons() () at util/thread_local.cc:106 Test Plan: unit tests verify StaticMeta::mutex_ is still initialized in DefaultEnv() (StaticMeta::mutex_ is the only variable intended to be initialized via StaticMeta::InitSingletons() which I removed) #0 0x00000000005cee17 in rocksdb::port::Mutex::Mutex(bool) (this=0x7ffff69500b0, adaptive=false) at port/port_posix.cc:52 #1 0x0000000000769cf8 in rocksdb::ThreadLocalPtr::StaticMeta::StaticMeta() (this=0x7ffff6950000) at util/thread_local.cc:168 #2 0x0000000000769a53 in rocksdb::ThreadLocalPtr::Instance() () at util/thread_local.cc:133 #3 0x0000000000769a09 in rocksdb::ThreadLocalPtr::InitSingletons() () at util/thread_local.cc:105 #4 0x0000000000647d98 in rocksdb::Env::Default() () at util/env_posix.cc:845 Reviewers: lightmark, yhchiang, sdong Reviewed By: sdong Subscribers: arahut, IslamAbdelRahman, yiwu, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60813 | 28 July 2016, 21:55:22 UTC |
6920cde | Gunnar Kudrjavets | 28 July 2016, 04:14:02 UTC | Remove an extra apostrophe | 28 July 2016, 04:16:11 UTC |
e72ea48 | Aaron Gao | 28 July 2016, 01:45:53 UTC | add InDomain regression test Summary: regression tests to make sure seek keys not in domain would not fail assertion Test Plan: ``` [gzh@dev6163.prn2 ~/local/rocksdb] ./prefix_test --gtest_filter=SamePrefixTest.* /tmp/rocksdbtest-112628/prefix_test Note: Google Test filter = SamePrefixTest.* [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from SamePrefixTest [ RUN ] SamePrefixTest.InDomainTest [ OK ] SamePrefixTest.InDomainTest (211 ms) [----------] 1 test from SamePrefixTest (211 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (211 ms total) ``` Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61161 | 28 July 2016, 01:45:53 UTC |
9c8ac14 | Gunnar Kudrjavets | 27 July 2016, 23:37:12 UTC | Avoid duplicate task creation for RocksDB contruns Summary: We rely on a basic evaluation logic: when using an expression `A || B`, the `B` part will only get evaluated when `A` fails. Therefore the task creation tool is guaranteed to run if previous build/test step failed. To indicate the correct return value to shell, the task creation tool will call `exit(1)` which will cause Sandcastle to mark it as a failure. Test Plan: - Land the changes. - Trigger a RocksDB contrun - observe the results. - Look at the results from the nightly runs and fix issues as necessary. Current testing so far has been in isolation for various components. Reviewers: andrewkr, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60909 | 27 July 2016, 23:37:12 UTC |
d4c4542 | sdong | 25 July 2016, 19:27:06 UTC | db_stress shouldn't assert file size 0 if file creation fails Summary: OnTableFileCreated() now is also called when the file creaion fails. In that case, we shouldn't assert the file size is not 0. Test Plan: Run crash test Reviewers: yiwu, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61137 | 27 July 2016, 19:08:16 UTC |
d3bfd33 | Anirban Rahut | 27 July 2016, 18:58:57 UTC | Testing out parallel sandcastle changes Summary: Removing moreutils from sandcastle and adding gnu parallel. Then passing in J= nproc command Test Plan: Testing on sandcastle Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61017 | 27 July 2016, 18:58:57 UTC |
7efd9c2 | sdong | 27 July 2016, 18:16:55 UTC | Increse timeout in some tests Summary: Increase test timeout to some tests to unblock CI. Test Plan: watch how it runs. Reviewers: kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61263 | 27 July 2016, 18:39:36 UTC |
50b8d29 | Alexander Jipa | 27 July 2016, 03:29:02 UTC | fixes 1230: Error:string sub-command REGEX, mode REPLACE needs at least 6 arguments total to command (#1231) quoting the las argument of string REGEXP REPLACE function | 27 July 2016, 03:29:02 UTC |
5c858dd | woonhak.kang | 27 July 2016, 02:28:45 UTC | fix errata in libnuma test (#1244) | 27 July 2016, 02:28:45 UTC |
e5b5f12 | sdong | 27 July 2016, 01:05:30 UTC | Change options memtable_prefix_bloom_huge_page_tlb_size => memtable_huge_page_size and cover huge page to memtable too Summary: Extend the option memtable_prefix_bloom_huge_page_tlb_size from just putting memtable bloom filter to huge page to memtable itself too. Test Plan: Run all existing tests. Reviewers: IslamAbdelRahman, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60513 | 27 July 2016, 01:15:11 UTC |
0ce258f | sdong | 26 July 2016, 01:08:49 UTC | Compaction picker to expand output level files for keys cross files' boundary too. Summary: We may wrongly drop delete operation if we pick a file with the entry to be delete, the put entry of the same user key is in the next file in the level, and the next file is not picked. We expand compaction inputs for output level too. Test Plan: Add unit tests that reproduct the bug of dropping delete entry. Change compaction_picker_test to assert the new behavior. Reviewers: IslamAbdelRahman, igor Reviewed By: igor Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61173 | 27 July 2016, 00:56:36 UTC |
ac0d93b | Alexander Jipa | 26 July 2016, 23:18:49 UTC | fixes 1217: rocksdbjni javac and javah execute_processes fail on windows (#1218) picking execute_process command based on OS | 26 July 2016, 23:18:49 UTC |
1ae4609 | Islam AbdelRahman | 26 July 2016, 20:54:08 UTC | Appveyor badge to show master branch appveyor badge to show master branch | 26 July 2016, 20:54:08 UTC |
8745f01 | Islam AbdelRahman | 26 July 2016, 20:18:31 UTC | [Fix java build] Stop using non standard std::make_unique Summary: std::make_unique is not standard and not always available, remove it Test Plan: Run "make clean jclean rocksdbjava jtest -j8" on my mac Reviewers: yhchiang, yiwu, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61143 | 26 July 2016, 20:18:31 UTC |
e12270d | Wanning Jiang | 26 July 2016, 18:15:14 UTC | fix previous typo Summary: old typos with FILTER/INDEX_CACHE Test Plan: still pass this unit test Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61185 | 26 July 2016, 18:15:14 UTC |
bbd6a5a | Andrew Kryczka | 26 July 2016, 18:13:26 UTC | ldb restore subcommand Summary: - Added a new subcommand, 'ldb restore', that restores from backup - Made backup_env_uri optional (also for 'ldb backup') because it can use db_env when backup_env isn't provided Test Plan: verify backup and restore commands work: $ ./ldb backup --db=/data/users/andrewkr/rocksdb/db_bench-out/ --num_threads 1 --backup_dir=/data/users/andrewkr/rocksdb/db_bench-out/backup/ $ ./ldb restore --db=/data/users/andrewkr/rocksdb/db_bench-out/restore/ --num_threads 1 --backup_dir=/data/users/andrewkr/rocksdb/db_bench-out/backup/ Reviewers: sdong, wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60849 | 26 July 2016, 18:13:26 UTC |
9498069 | Andrew Kryczka | 25 July 2016, 23:37:23 UTC | Run error-filtering script on diff-triggered tests Summary: this script creates a summary message based on the test type and output. Previously it only ran during contbuild. This diff makes it also run on commit-triggered diffs. Test Plan: will see if it works on diff's sandcastle tests Reviewers: kradhakrishnan, lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60879 | 25 July 2016, 23:37:23 UTC |
f8061a2 | Islam AbdelRahman | 25 July 2016, 23:05:50 UTC | Fix Statistics TickersNameMap miss match with Tickers enum Summary: TickersNameMap is not consistent with Tickers enum. this cause us to report wrong statistics and sometimes to access TickersNameMap outside it's boundary causing crashes (in Fb303 statistics) Test Plan: added new unit test Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61083 | 25 July 2016, 23:05:50 UTC |
16e225f | Islam AbdelRahman | 25 July 2016, 22:31:41 UTC | Fix MergeContext::copied_operands_ strings moving Summary: MergeContext::copied_operands contain strings that MergeContext::operand_list_ Slices point to It's possible that when MergeContext::copied_operands grow, these strings are moved and there place in memory is changed, this will cause MergeContext::operand_list_ to point to invalid memory. fix this problem by using unique_ptr<string> instead of string Test Plan: run tests under mac/clang Reviewers: sdong, yiwu Reviewed By: yiwu Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61023 | 25 July 2016, 22:31:41 UTC |
a4955b3 | krad | 25 July 2016, 21:17:28 UTC | Run sandcastle tests in /dev/shm | 25 July 2016, 22:10:24 UTC |
ae0ad71 | Yi Wu | 25 July 2016, 22:09:34 UTC | Fix flaky DBSSTTEST::DeleteObsoleteFilesPendingOutputs Summary: The test is flaky on Travis in osx environment. The background flush the test wanting to block can run behind the L2 manual compaction, making the test actually blocking the L2 compaction and won't able to proceed. Test Plan: Test run on travis Reviewers: kradhakrishnan, sdong, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61101 | 25 July 2016, 22:09:34 UTC |
b2a8016 | zhang-tong | 25 July 2016, 21:50:44 UTC | Update db_bench_tool.cc (#1239) * Update db_bench_tool.cc * Update db_bench_tool.cc | 25 July 2016, 21:50:44 UTC |
c665458 | Yi Wu | 25 July 2016, 18:48:17 UTC | Disable two dynamic options tests under lite build Summary: RocksDB lite don't support dynamic options. Disable the two test from lite build, and assert `SetOptions` should return `status::OK`. Test Plan: Run the db_options test under lite build and normal build. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D61119 | 25 July 2016, 18:48:17 UTC |
2a6d0cd | sdong | 19 July 2016, 18:48:00 UTC | Ignore stale logs while restarting DBs Summary: Stale log files can be deleted out of order. This can happen for various reasons. One of the reason is that no data is ever inserted to a column family and we have an optimization to update its log number, but not all the old log files are cleaned up (the case shown in the unit tests added). It can also happen when we simply delete multiple log files out of order. This causes data corruption because we simply increase seqID after processing the next row and we may end up with writing data with smaller seqID than what is already flushed to memtables. In DB recovery, for the oldest files we are replaying, if there it contains no data for any column family, we ignore the sequence IDs in the file. Test Plan: Add two unit tests that fail without the fix. Reviewers: IslamAbdelRahman, igor, yiwu Reviewed By: yiwu Subscribers: hermanlee4, yoshinorim, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60891 | 25 July 2016, 18:47:31 UTC |
ee8bf2e | Alexander Jipa | 24 July 2016, 10:19:00 UTC | fixes 1228: rockdbjni loadLibraryFromJarToTemp fails when file is already present (#1232) overriding existing library in tmp folder | 24 July 2016, 10:19:00 UTC |
f85df12 | sdong | 22 July 2016, 18:46:13 UTC | Re-enable tsan crash white-box test with reduced killing odds Summary: Tsan crash white-box test was disabled because it never ends. Re-enable it with reduced killing odds. Add a parameter in crash test script to allow we pass it through an environment variable. Test Plan: Run it manually. Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61053 | 22 July 2016, 19:51:27 UTC |
89e4c48 | Islam AbdelRahman | 22 July 2016, 19:03:21 UTC | Update README.md to include appveyor badge Update README.md to include appveyor badge | 22 July 2016, 19:03:21 UTC |
b06ca5f | Peter (Stig) Edwards | 22 July 2016, 18:46:40 UTC | ldb load, prefer ifsteam(/dev/stdin) to std::cin (#1207) getline on std::cin can be very inefficient when ldb is loading large values, with high CPU usage in libc _IO_(un)getc, this is because of the performance penalty that comes from synchronizing stdio and iostream buffers. See the reproducers and tests in #1133 . If an ifstream on /dev/stdin is used (when available) then using ldb to load large values can be much more efficient. I thought for ldb load, that this approach is preferable to using <cstdio> or std::ios_base::sync_with_stdio(false). I couldn't think of a use case where ldb load would need to support reading unbuffered input, an alternative approach would be to add support for passing --input_file=/dev/stdin. I have a CLA in place, thanks. The CI tests were failing at the time of https://github.com/facebook/rocksdb/pull/1156, so this change and PR will supersede it. | 22 July 2016, 18:46:40 UTC |
4ea0ab3 | sdong | 22 July 2016, 17:42:56 UTC | Revert "Remove bashism from `make check` (#1225)" This reverts commit 08ab1d83ac01c5e36673e844f3590f8ae14a6f2b. | 22 July 2016, 17:42:56 UTC |
12767b3 | Alexander Jipa | 22 July 2016, 17:17:10 UTC | fixes 1220: rocksjni build fails on Windows due to variable-size array declaration (#1223) * fixes 1220: rocksjni build fails on Windows due to variable-size array declaration using new keyword to create variable-size arrays in order to satisfy most of the compilers * fixes 1220: rocksjni build fails on Windows due to variable-size array declaration using unique_ptr keyword to create variable-size arrays in order to satisfy most of the compilers | 22 July 2016, 17:17:10 UTC |
a9d512a | Andrew Kryczka | 22 July 2016, 08:13:25 UTC | Update .gitignore for internal release | 22 July 2016, 08:17:03 UTC |
d5a51d4 | sdong | 19 July 2016, 22:12:46 UTC | Need to make sure log file synced before flushing memtable of one column family Summary: Multiput atomiciy is broken across multiple column families if we don't sync WAL before flushing one column family. The WAL file may contain a write batch containing writes to a key to the CF to be flushed and a key to other CF. If we don't sync WAL before flushing, if machine crashes after flushing, the write batch will only be partial recovered. Data to other CFs are lost. Test Plan: Add a new unit test which will fail without the diff. Reviewers: yhchiang, IslamAbdelRahman, igor, yiwu Reviewed By: yiwu Subscribers: yiwu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60915 | 21 July 2016, 23:29:06 UTC |
89f319c | Yi Wu | 21 July 2016, 21:32:12 UTC | Fix unit test which breaks lite build Summary: Comment out assertion of number of table files from lite build. Test Plan: OPT=-DROCKSDB_LITE make check Reviewers: lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60999 | 21 July 2016, 21:32:12 UTC |
b506329 | sdong | 21 July 2016, 19:53:02 UTC | Add unit test not on /dev/shm as part of the pre-commit tests Summary: RocksDB behavior is slightly different between data on tmpfs and normal file systems. Add a test case to run RocksDB on normal file system. Test Plan: See the tests launched by Phabricator Reviewers: kradhakrishnan, IslamAbdelRahman, gunnarku Reviewed By: gunnarku Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60963 | 21 July 2016, 19:53:23 UTC |
b9a9718 | krad | 21 July 2016, 18:05:27 UTC | Bump next release version Summary: Bump master version to the next potential release version of 4.11 Test Plan: Compile Reviewers: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60993 | 21 July 2016, 19:22:14 UTC |
663afef | ryneli | 21 July 2016, 18:16:34 UTC | Add EnvLibrados - RocksDB Env of RADOS (#1222) EnvLibrados is a customized RocksDB Env to use RADOS as the backend file system of RocksDB. It overrides all file system related API of default Env. The easiest way to use it is just like following: std::string db_name = "test_db"; std::string config_path = "path/to/ceph/config"; DB* db; Options options; options.env = EnvLibrados(db_name, config_path); Status s = DB::Open(options, kDBPath, &db); Then EnvLibrados will forward all file read/write operation to the RADOS cluster assigned by config_path. Default pool is db_name+"_pool". There are some options that users could set for EnvLibrados. - write_buffer_size. This variable is the max buffer size for WritableFile. After reaching the buffer_max_size, EnvLibrados will sync buffer content to RADOS, then clear buffer. - db_pool. Rather than using default pool, users could set their own db pool name - wal_dir. The dir for WAL files. Because RocksDB only has 2-level structure (dir_name/file_name), the format of wal_dir is "/dir_name"(CAN'T be "/dir1/dir2"). Default wal_dir is "/wal". - wal_pool. Corresponding pool name for WAL files. Default value is db_name+"_wal_pool" The example of setting options looks like following: db_name = "test_db"; db_pool = db_name+"_pool"; wal_dir = "/wal"; wal_pool = db_name+"_wal_pool"; write_buffer_size = 1 << 20; env_ = new EnvLibrados(db_name, config, db_pool, wal_dir, wal_pool, write_buffer_size); DB* db; Options options; options.env = env_; // The last level dir name should match the dir name in prefix_pool_map options.wal_dir = "/tmp/wal"; // open DB Status s = DB::Open(options, kDBPath, &db); Librados is required to compile EnvLibrados. Then use "$make LIBRADOS=1" to compile RocksDB. If you want to only compile EnvLibrados test, just run "$ make env_librados_test LIBRADOS=1". To run env_librados_test, you need to have a running RADOS cluster with the configure file located in "../ceph/src/ceph.conf" related to "rocksdb/". | 21 July 2016, 18:16:34 UTC |
32604e6 | Yi Wu | 21 July 2016, 17:10:41 UTC | Fix flush not being commit while writing manifest Summary: Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075. The issue: # Options.max_background_flushes > 1 # Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.) # Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit. # After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted. Test Plan: run the test. Also verify the new test hit deadlock without the fix. Reviewers: sdong, igor, lightmark Reviewed By: lightmark Subscribers: andrewkr, omegaga, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60969 | 21 July 2016, 17:10:41 UTC |