Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
14b3f68 Maysam Yabandeh 02 April 2019, 21:43:03 UTC WriteUnPrepared: less virtual in iterator callback (#5049) Summary: WriteUnPrepared adds a virtual function, MaxUnpreparedSequenceNumber, to ReadCallback, which returns 0 unless WriteUnPrepared is enabled and the transaction has uncommitted data written to the DB. Together with snapshot sequence number, this determines the last sequence that is visible to reads. The patch clarifies the guarantees of the GetIterator API in WriteUnPrepared transactions and make use of that to statically initialize the read callback and thus avoid the virtual call. Furthermore it increases the minimum value for min_uncommitted from 0 to 1 as seq 0 is used only for last level keys that are committed in all snapshots. The following benchmark shows +0.26% higher throughput in seekrandom benchmark. Benchmark: ./db_bench --benchmarks=fillrandom --use_existing_db=0 --num=1000000 --db=/dev/shm/dbbench ./db_bench --benchmarks=seekrandom[X10] --use_existing_db=1 --db=/dev/shm/dbbench --num=1000000 --duration=60 --seek_nexts=100 seekrandom [AVG 10 runs] : 20355 ops/sec; 225.2 MB/sec seekrandom [MEDIAN 10 runs] : 20425 ops/sec; 225.9 MB/sec ./db_bench_lessvirtual3 --benchmarks=seekrandom[X10] --use_existing_db=1 --db=/dev/shm/dbbench --num=1000000 --duration=60 --seek_nexts=100 seekrandom [AVG 10 runs] : 20409 ops/sec; 225.8 MB/sec seekrandom [MEDIAN 10 runs] : 20487 ops/sec; 226.6 MB/sec Pull Request resolved: https://github.com/facebook/rocksdb/pull/5049 Differential Revision: D14366459 Pulled By: maysamyabandeh fbshipit-source-id: ebaff8908332a5ae9af7defeadabcb624be660ef 02 April 2019, 21:47:16 UTC
d9d3cac Simon Grätzer 02 April 2019, 18:52:53 UTC Add a missing define to monitoring/iostats_context_imp.h (#5136) Summary: I think when PR https://github.com/facebook/rocksdb/pull/4889 added the `IOSTATS_CPU_TIMER_GUARD` define to this header file, the noop version in the `#else` branch was forgotten. Not sure if this is common, but on my MacOS machine it breaks my build Pull Request resolved: https://github.com/facebook/rocksdb/pull/5136 Differential Revision: D14727727 Pulled By: siying fbshipit-source-id: 1076e56bdbe6ecda01d461b371dabf7f1593a149 02 April 2019, 18:56:18 UTC
ebcc8ae Siying Dong 02 April 2019, 16:57:35 UTC Revert "Avoid per-key upper bound check in BlockBasedTableIterator (#5101)" (#5132) Summary: This reverts commit f29dc1b90641e7f44b14f932e3866c5840391cd5. In BlockBasedTableIterator, index_iter_->key() is sometimes a user key, so it is wrong to call ExtractUserKey() against it. This is a bug introduced by #5101. Temporarily revert the diff to keep the branch clean. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5132 Differential Revision: D14718584 Pulled By: siying fbshipit-source-id: 0ac55dc9b5dbc18c7809092146bdf7eb9364b9ad 02 April 2019, 17:00:38 UTC
fa1b558 xinbenlv 02 April 2019, 01:16:26 UTC Add LevelDB repository link in the Readme Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5134 Differential Revision: D14719068 Pulled By: siying fbshipit-source-id: c09a544f06ff414dbe2f90792aaf2bb5b8550bee 02 April 2019, 01:19:09 UTC
120bc47 Mike Kolupaev 02 April 2019, 00:07:38 UTC Add DBOptions. avoid_unnecessary_blocking_io to defer file deletions (#5043) Summary: Just like ReadOptions::background_purge_on_iterator_cleanup but for ColumnFamilyHandle instead of Iterator. In our use case we sometimes call ColumnFamilyHandle's destructor from low-latency threads, and sometimes it blocks the thread for a few seconds deleting the files. To avoid that, we can either offload ColumnFamilyHandle's destruction to a background thread on our side, or add this option on rocksdb side. This PR does the latter, to be consistent with how we solve exactly the same problem for iterators using background_purge_on_iterator_cleanup option. (EDIT: It's avoid_unnecessary_blocking_io now, and affects both CF drops and iterator destructors.) I'm not quite comfortable with having two separate options (background_purge_on_iterator_cleanup and background_purge_on_cf_cleanup) for such a rarely used thing. Maybe we should merge them? Rename background_purge_on_cf_cleanup to something like delete_files_on_background_threads_only or avoid_blocking_io_in_unexpected_places, and make iterators use it instead of the one in ReadOptions? I can do that here if you guys think it's better. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5043 Differential Revision: D14339233 Pulled By: al13n321 fbshipit-source-id: ccf7efa11c85c9a5b91d969bb55627d0fb01e7b8 02 April 2019, 00:10:40 UTC
127a850 Remington Brasga 29 March 2019, 22:05:29 UTC Fix arena allocation size in NewEmptyInternalIterator (#4905) Summary: NewEmptyInternalIterator with arena mistakenly used EmptyIterator to allocate the size from area but then initialized it to a totally different object: EmptyInternalIterator. The patch fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4905 Differential Revision: D14689840 Pulled By: maysamyabandeh fbshipit-source-id: af64fd8ee93d5a4ad54691c792e5ecc5efabc887 29 March 2019, 22:09:35 UTC
a703f16 Maysam Yabandeh 29 March 2019, 20:09:51 UTC WriteUnPrepared: Enable auto-compaction after max_evicted_seq_ init (#5128) Summary: Compaction would depend on max_evicted_seq_ value. The ::Initialize method should do that after max_evicted_seq_ is properly initialized. The patch also back ports #4853 from WritePrepared txn to WriteUnPrepared. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5128 Differential Revision: D14686562 Pulled By: maysamyabandeh fbshipit-source-id: b2355025712a72676ac3b20a95258adcf4774490 29 March 2019, 20:18:57 UTC
f29dc1b Yi Wu 29 March 2019, 20:07:44 UTC Avoid per-key upper bound check in BlockBasedTableIterator (#5101) Summary: `BlockBasedTableIterator` avoid reading next block on `Next()` if it detects the iterator will be out of bound, by checking against index key. The optimization was added in #2239, and by the time it only check the bound per block. It seems later change make it a per-key check, which introduce unnecessary key comparisons. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5101 Differential Revision: D14678707 Pulled By: siying fbshipit-source-id: 2372446116753c7892ea4cec7b4b49ef87ba463e 29 March 2019, 20:11:46 UTC
09957de Yanqin Jin 29 March 2019, 17:03:02 UTC Update RepeatableThreadTest with MockTimeEnv (#5107) Summary: **This PR updates RepeatableThread::wait, breaking some tests on OS X. The rest of the PR fixes the tests on OS X.** `RepeatableThreadTest.MockEnvTest` uses `MockTimeEnv` and `RepeatableThread`. If `RepeatableThread::wait` calls `TimedWait` with a time smaller than or equal to the current (real) time, `TimedWait` returns immediately on certain platforms, e.g. OS X. #4560 addresses this issue by replacing `TimedWait` with `Wait` in test. This fixes the test but makes test/production code diverge, which is not optimal for test coverage. This PR proposes an alternative fix which unifies test and production code path for `RepeatableThread::wait`. We obtain the current (real) time in seconds and add 10 extra seconds to ensure that `RepeatableThread::wait` invokes `TimedWait` with a time greater than (real) current time. This is to prevent the `TimedWait` function from returning immediately without sleeping and releasing the mutex. If `TimedWait` returns immediately, the mutex will not be released, and `RepeatableThread::TEST_WaitForRun` never has a chance to execute the callback which, in this case, updates the result returned by `mock_env->NowMicros()`. Consequently, `RepeatableThread::wait` cannot break out of the loop, causing test to hang. The extra 10 seconds is a best-effort approach because there seems no reliable and deterministic way to provide the aforementioned guarantee. By the time `RepeatableThread::wait` is called, there is no guarantee that the `delay + mock_env->NowMicros()` will be greater than the current real time. However, 10 seconds should be sufficient in most cases. We will keep an eye for possible flakiness of this test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5107 Differential Revision: D14680885 Pulled By: riversand963 fbshipit-source-id: d1ecbe10e1dacd110bd464cd01e188bfee72b89e 29 March 2019, 17:08:50 UTC
d77476e Yanqin Jin 29 March 2019, 02:16:58 UTC Fix db_stress for custom env (#5122) Summary: Fix some hdfs-related code so that it can compile and run 'db_stress' Pull Request resolved: https://github.com/facebook/rocksdb/pull/5122 Differential Revision: D14675495 Pulled By: riversand963 fbshipit-source-id: cac280479efcf5451982558947eac1732e8bc45a 29 March 2019, 02:20:27 UTC
dae3b55 anand76 28 March 2019, 22:13:02 UTC Smooth the deletion of WAL files (#5116) Summary: WAL files are currently not subject to deletion rate limiting by DeleteScheduler. If the size of the WAL files is significant, this can cause a high delete rate on SSDs that may affect other operations. To fix it, force WAL file deletions to go through the SstFileManager. Original PR for this is #2768 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5116 Differential Revision: D14669437 Pulled By: anand1976 fbshipit-source-id: c5f62d0640cebaa1574de841a1d01e4ce2faadf0 28 March 2019, 22:17:13 UTC
a98317f Siying Dong 28 March 2019, 21:50:06 UTC Option string/map can set merge operator from object registry (#5123) Summary: Allow customized merge operator to be loaded from option file/map/string by allowing users to pre-regiester merge operators to object registry. Also update HISTORY.md and header files for the same feature for comparator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5123 Differential Revision: D14658488 Pulled By: siying fbshipit-source-id: 86ea2fbd2a0a04632d8ea9fceaffefd041f6ae61 28 March 2019, 21:54:29 UTC
106a94a Siying Dong 28 March 2019, 20:11:53 UTC Improve obsolete_files_test (#5125) Summary: We see a failure of obsolete_files_test but aren't able to identify the issue. Improve the test in following way and hope we can debug better next time: 1. Place sync point before automatic compaction runs so race condition will always trigger. 2. Disable sync point before test finishes. 3. ASSERT_OK() instead of ASSERT_TRUE(status.ok()) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5125 Differential Revision: D14669456 Pulled By: siying fbshipit-source-id: dccb7648e334501ad651eb212880096eef1f4ab2 28 March 2019, 20:16:02 UTC
d1edf4e Burton Li 27 March 2019, 23:31:30 UTC Format env_win coding style by google c++ style guide (#5096) Summary: The existing code for env_win src and header file doesn't fully followed the recommended code style (https://google.github.io/styleguide/cppguide.html#Functions). Fix it for better readability. anand1976 siying Pull Request resolved: https://github.com/facebook/rocksdb/pull/5096 Differential Revision: D14585358 Pulled By: anand1976 fbshipit-source-id: 7ce35ffe9e922f5c1421b0bbaa5fce7abad57617 27 March 2019, 23:38:20 UTC
89ab138 Siying Dong 27 March 2019, 23:13:08 UTC Apply automatic formatting to some files (#5114) Summary: Following files were run through automatic formatter: db/db_impl.cc db/db_impl.h db/db_impl_compaction_flush.cc db/db_impl_debug.cc db/db_impl_files.cc db/db_impl_readonly.h db/db_impl_write.cc db/dbformat.cc db/dbformat.h table/block.cc table/block.h table/block_based_filter_block.cc table/block_based_filter_block.h table/block_based_filter_block_test.cc table/block_based_table_builder.cc table/block_based_table_reader.cc table/block_based_table_reader.h table/block_builder.cc table/block_builder.h table/block_fetcher.cc table/block_prefix_index.cc table/block_prefix_index.h table/block_test.cc table/format.cc table/format.h I could easily run all the files, but I don't want people to feel that I'm doing it for lines of code changes :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/5114 Differential Revision: D14633040 Pulled By: siying fbshipit-source-id: 3f346cb53bf21e8c10704400da548dfce1e89a52 27 March 2019, 23:24:45 UTC
1f7f5a5 Siying Dong 27 March 2019, 20:21:27 UTC Run automatic formatter against public header files (#5115) Summary: Automatically format public headers so it looks more consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5115 Differential Revision: D14632854 Pulled By: siying fbshipit-source-id: ce9929ea62f9dcd65c69660b23eed1931cb0ae84 27 March 2019, 20:24:25 UTC
5f6adf3 Siying Dong 27 March 2019, 19:20:43 UTC Fix some variable naming in db/transaction_log_impl.* (#5112) Summary: We follow Google C++ Style which indicates variable names should be all underscore: https://google.github.io/styleguide/cppguide.html#Variable_Names Fix some variable names under db/transaction_log_impl.* Pull Request resolved: https://github.com/facebook/rocksdb/pull/5112 Differential Revision: D14631157 Pulled By: siying fbshipit-source-id: 9525c9b0976b843bca377b03897700d87cc60af8 27 March 2019, 19:27:54 UTC
8c07204 Fosco Marotto 27 March 2019, 18:18:38 UTC Update history and version for 6.1 Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5119 Differential Revision: D14645216 Pulled By: gfosco fbshipit-source-id: f7c83dca22c2486fc5d8697b61638c382889d073 27 March 2019, 18:21:34 UTC
d692415 Yi Wu 27 March 2019, 17:24:16 UTC Fix perf_context.user_key_comparison_count for range scan (#5098) Summary: Currently `perf_context.user_key_comparison_count` is bump only in `InternalKeyComparator`. For places user comparator is used directly the counter is not bump. Fixing the majority of it. Index iterator and filter code also use user comparator directly and don't bump the counter. It is not fixed in this patch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5098 Differential Revision: D14603753 Pulled By: siying fbshipit-source-id: 1cd41035644ca9e49b97a51030a5d1e15f5f3cae 27 March 2019, 17:34:27 UTC
2b4d5ce Siying Dong 27 March 2019, 17:18:56 UTC Remove some "using std::..." from header files. (#5113) Summary: The code convention we are following, Google C++ Style, discourage alias in header files, especially public headers: https://google.github.io/styleguide/cppguide.html#Aliases Remove some of them. Might removed some from .cc files as well to be consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5113 Differential Revision: D14633030 Pulled By: siying fbshipit-source-id: b990edc919d5de60295992284f980195e501d424 27 March 2019, 17:28:21 UTC
9358178 Yanqin Jin 26 March 2019, 23:41:31 UTC Support for single-primary, multi-secondary instances (#4899) Summary: This PR allows RocksDB to run in single-primary, multi-secondary process mode. The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary. Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`. This PR has several components: 1. (Originally in #4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary. 2. (Similar to #4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue. 3. (Originally in #4710 and #4820). Add implementation of the secondary, i.e. `DBImplSecondary`. 3.1 Tail the primary's MANIFEST during recovery. 3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`. 3.3 Tailing WAL will be in a future PR. 4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4899 Differential Revision: D14510945 Pulled By: riversand963 fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886 26 March 2019, 23:45:31 UTC
2a5463a jsteemann 26 March 2019, 23:31:17 UTC remove bundled but unused fbson library (#5108) Summary: fbson library is still included in `third-party` directory, but is not needed by RocksDB anymore. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5108 Differential Revision: D14622272 Pulled By: siying fbshipit-source-id: 52b24ed17d8d870a71364f85e5bac4eafb192df5 26 March 2019, 23:37:52 UTC
01e6bad Shi Feng 26 March 2019, 23:20:52 UTC Introduce CPU timers for iterator seek and next (#5076) Summary: Introduce CPU timers for iterator seek and next operations. Seek counter includes SeekToFirst, SeekToLast and SeekForPrev, w/ the caveat that SeekToLast timer doesn't include some post processing time if upper bound is defined. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5076 Differential Revision: D14525218 Pulled By: fredfsh fbshipit-source-id: 03ba25df3b22b06c072621e4de0eacfa1445f0d9 26 March 2019, 23:32:13 UTC
4774a94 Siying Dong 26 March 2019, 21:15:26 UTC Allow option string to get comparator from object registry (#5106) Summary: Even customized ldb may not be able to read data from some databases if comparator is not standard. We modify option helper to get comparator from object registry so that we can use customized ldb to read non-standard comparator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5106 Differential Revision: D14622107 Pulled By: siying fbshipit-source-id: 151dcb295a35a4c7d54f919cd4e322a89dc601c9 26 March 2019, 21:23:51 UTC
fe2bd19 Siying Dong 26 March 2019, 17:43:22 UTC BlobDB::Open() should put all existing trash files to delete scheduler (#5103) Summary: Right now, BlobDB::Open() fails to put all trash files to delete scheduler, which causes some trash files permanently untracked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5103 Differential Revision: D14606095 Pulled By: siying fbshipit-source-id: 41a9437a2948abb235c0ed85f9a04612d0e50183 26 March 2019, 17:53:19 UTC
75133b1 Yi Wu 26 March 2019, 17:15:43 UTC Fix SstFileReader not able to open ingested file (#5097) Summary: Since `SstFileReader` don't know largest seqno of a file, it will fail this check when it open a file with global seqno: https://github.com/facebook/rocksdb/blob/ca89ac2ba997dfa0e135bd75d4ccf6f5774a7eff/table/block_based_table_reader.cc#L730 Changes: * Pass largest_seqno=kMaxSequenceNumber from `SstFileReader` and allow it to bypass the above check. * `BlockBasedTable::VerifyChecksum` also double check if checksum will match when excluding global seqno (this is to make the new test in sst_table_reader_test pass). Pull Request resolved: https://github.com/facebook/rocksdb/pull/5097 Differential Revision: D14607434 Pulled By: riversand963 fbshipit-source-id: 9008599227c5fccbf9b73fee46b3bf4a1523f023 26 March 2019, 17:25:18 UTC
7ca9eb7 Yi Wu 26 March 2019, 17:09:26 UTC Fix BlockBasedTableIterator construction missing index_key_is_full parameter Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5104 Differential Revision: D14619000 Pulled By: maysamyabandeh fbshipit-source-id: c2895794a3f31b826c149dcb698c1952dacc2332 26 March 2019, 17:13:01 UTC
3c5eed5 Zhongyi Xie 26 March 2019, 06:22:14 UTC remove incorrect assert in `GetUniqueIdFromFile` (#5102) Summary: User report has shown that sometimes `BlockBasedTable::SetupCacheKeyPrefix` would assert when trying to generate an id from the file. The actual cause seems to be hardware related but we might be better off without the incorrect assertion See T42178927 for more information Pull Request resolved: https://github.com/facebook/rocksdb/pull/5102 Differential Revision: D14604677 Pulled By: miasantreble fbshipit-source-id: fcb09207ebdc4fa66e941afbc0523d84797e7ad7 26 March 2019, 06:28:29 UTC
48e7eff Siying Dong 26 March 2019, 02:14:04 UTC Avoid to go through every CF for every ReleaseSnapshot() (#5090) Summary: With https://github.com/facebook/rocksdb/pull/3009 we go through every CF to check whether a bottommost compaction is needed to be triggered. This is done within DB mutex. What we do within DB mutex may heavily influece the write throughput we can achieve, so we always want to minimize work there. Here we try to avoid this for-loop by first check a global threshold. In most of the time, the CF loop can be avoided. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5090 Differential Revision: D14582684 Pulled By: siying fbshipit-source-id: 968f6d9bb6affe1a5ebc4910b418300b076f166f 26 March 2019, 02:18:04 UTC
52e6404 Zhongyi Xie 25 March 2019, 20:19:25 UTC ldb command parsing: allow option values to contain equals signs (#5088) Summary: Right now ldb command doesn't allow cases where option values contain equals sign. For example, ``` ldb --db=/tmp/test scan --from='q=3' --max_keys=1 ``` after parsing, ldb will have one option 'db', 'max_keys' and one flag 'from'. This PR updates the parsing logic so that it now supports the above mentioned cases Pull Request resolved: https://github.com/facebook/rocksdb/pull/5088 Differential Revision: D14600869 Pulled By: miasantreble fbshipit-source-id: c6ef518c74a98d7b6675ea5954ae08b1bda5554e 25 March 2019, 20:23:11 UTC
a4396f9 Rashmi Sharma 21 March 2019, 23:16:40 UTC Make it easier for users to load options from option file and set shared block cache. (#5063) Summary: [RocksDB] Make it easier for users to load options from option file and set shared block cache. Right now, it requires several dynamic casting for users to set the shared block cache to their option struct cast from the option file. If people don't do that, every CF of every DB will generate its own 8MB block cache. It's not a usable setting. So we are dragging every user who loads options from the file into such a mess. Instead, we should allow them to pass their cache object to LoadLatestOptions() and LoadOptionsFromFile(), so that those loaded option structs will have the shared block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5063 Differential Revision: D14518584 Pulled By: rashmishrm fbshipit-source-id: c91430ff9425a0e67d76fc67931d755f491ca5aa 21 March 2019, 23:25:28 UTC
88d85b6 Burton Li 21 March 2019, 22:10:38 UTC fix NowNanos overflow (#5062) Summary: The original implementation of WinEnvIO::NowNanos() has a constant data overflow by: li.QuadPart *= std::nano::den; As a result, the api provides a incorrect result. e.g.: li.QuadPart=13477844301545 std::nano::den=1e9 The fix uses pre-computed nano_seconds_per_period_ to present the nano seconds per performance counter period, in the case if nano::den is divisible by perf_counter_frequency_. Otherwise it falls back to use high_resolution_clock. siying ajkr Pull Request resolved: https://github.com/facebook/rocksdb/pull/5062 Differential Revision: D14426842 Pulled By: anand1976 fbshipit-source-id: 127f1daf423dd4b30edd0dcf8ea0466f468bec12 21 March 2019, 22:18:36 UTC
c84fad7 Maysam Yabandeh 21 March 2019, 16:51:29 UTC Reorder DBIter fields to reduce memory usage (#5078) Summary: The patch reorders DBIter fields to put 1-byte fields together and let the compiler optimize the memory usage by using less 64-bit allocations for bools and enums. This might have a negative side effect of putting the variables that are accessed together into different cache lines and hence increasing the cache misses. Not sure what benchmark would verify that thought. I ran simple, single-threaded seekrandom benchmarks but the variance in the results is too much to be conclusive. ./db_bench --benchmarks=fillrandom --use_existing_db=0 --num=1000000 --db=/dev/shm/dbbench ./db_bench --benchmarks=seekrandom[X10] --use_existing_db=1 --db=/dev/shm/dbbench --num=1000000 --duration=60 --seek_nexts=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5078 Differential Revision: D14562676 Pulled By: maysamyabandeh fbshipit-source-id: 2284655d46e079b6e9a860e94be5defb6f482167 21 March 2019, 16:55:09 UTC
34f8ac0 Levi Tamasi 20 March 2019, 19:24:57 UTC Make adaptivity of LRU cache mutexes configurable (#5054) Summary: The patch adds a new config option to LRUCacheOptions that enables users to choose whether to use an adaptive mutex for the LRU block cache (on platforms where adaptive mutexes are supported). The default is true if RocksDB is compiled with -DROCKSDB_DEFAULT_TO_ADAPTIVE_MUTEX, false otherwise. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5054 Differential Revision: D14542749 Pulled By: ltamasi fbshipit-source-id: 0065715ab6cf91f10444b737fed8c8aee6a8a0d2 20 March 2019, 19:33:44 UTC
1721635 Alexandre Viau 20 March 2019, 18:07:19 UTC LANGUAGE-BINDINGS.md: mention python-rocksdb Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5086 Differential Revision: D14542212 Pulled By: siying fbshipit-source-id: db2f38a3f7c9b64532655a5d4ac4b7715c392883 20 March 2019, 18:10:48 UTC
959f86e anand76 20 March 2019, 17:38:54 UTC Use placement new and delete in autovector (#5080) Summary: The stack buffer in rocksdb::autovector is currently defined as an array of elements of the template type. This results in unnecessary construction of those objects, which can be a significant overhead in some cases. This PR changes the type of the stack buf to char* and uses placement new to construct new objects when they are inserted into the autovector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5080 Differential Revision: D14533221 Pulled By: anand1976 fbshipit-source-id: 9378985c7d03f4e1a28951bdd2403c72f10f23d7 20 March 2019, 17:42:04 UTC
a291f3a Zhongyi Xie 20 March 2019, 00:24:09 UTC Collect compaction stats by priority and dump to info LOG (#5050) Summary: In order to better understand compaction done by different priority thread pool, we now collect compaction stats by priority and also print them to info LOG through stats dump. ``` ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.8 11.3 5.5 5.6 0.1 0.0 0.0 406.4 136.1 42.24 34.96 45 0.939 13M 8865K High 0/0 0.00 KB 0.0 0.0 0.0 0.0 11.4 11.4 0.0 0.0 0.0 76.2 153.00 35.74 12185 0.013 0 0 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/5050 Differential Revision: D14408583 Pulled By: miasantreble fbshipit-source-id: e53746586ea27cb8abc9fec35805bd80ed30f608 20 March 2019, 00:28:19 UTC
e50326f Andrew Audibert 19 March 2019, 21:21:11 UTC Document the interaction between disableWAL and BackupEngine (#5071) Summary: BackupEngine relies on write-ahead logs to back up the memtable. Disabling write-ahead logs can result in backups failing to preserve unflushed keys. This PR updates the documentation to specify this behavior, and suggest always flushing the memtable when write-ahead logs are disabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5071 Differential Revision: D14524124 Pulled By: miasantreble fbshipit-source-id: 635f855f8a42ad60273b5efd226139b511e3e5d5 19 March 2019, 21:58:14 UTC
36c2a7c Wenjie Yang 19 March 2019, 21:19:01 UTC Add an option to filter traces (#5082) Summary: Add an option to filter out READ or WRITE operations while tracing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5082 Differential Revision: D14515083 Pulled By: mrmiywj fbshipit-source-id: 2504c89a9abf1dd629cad44b4104092702d77610 19 March 2019, 21:36:51 UTC
f2f6acb Hiroaki Nakamura 19 March 2019, 16:37:58 UTC Add missing C API for transaction (#5077) Summary: Partly addresses https://github.com/facebook/rocksdb/issues/4999 I verified `make static_lib` runs fine. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5077 Differential Revision: D14521101 Pulled By: maysamyabandeh fbshipit-source-id: ba88e74a51d2d793cac7260d505b1a54254b53af 19 March 2019, 16:43:22 UTC
b45b1cd Shobhit Dayal 18 March 2019, 19:07:35 UTC Feature for sampling and reporting compressibility (#4842) Summary: This is a feature to sample data-block compressibility and and report them as stats. 1 in N (tunable) blocks is sampled for compressibility using two algorithms: 1. lz4 or snappy for fast compression 2. zstd or zlib for slow but higher compression. The stats are reported to the caller as raw-bytes and compressed-bytes. The block continues to be compressed for storage using the specified CompressionType. The db_bench_tool how has a command line option for specifying the sampling rate. It's default value is 0 (no sampling). To test the overhead for a certain value, users can compare the performance of db_bench_tool, varying the sampling rate. It is unlikely to have a noticeable impact for high values like 20. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4842 Differential Revision: D13629011 Pulled By: shobhitdayal fbshipit-source-id: 14ca668bcab6499b2a1734edf848eb62a4f4fafa 18 March 2019, 19:15:34 UTC
20d49da He Zhe 18 March 2019, 18:27:41 UTC utilities: Fix build failure with -Werror=maybe-uninitialized (#5074) Summary: Initialize magic_number to zero to avoid such failure. utilities/blob_db/blob_log_format.cc:91:3: error: 'magic_number' may be used uninitialized in this function [-Werror=maybe-uninitialized] if (magic_number != kMagicNumber) { ^~ Signed-off-by: He Zhe <zhe.he@windriver.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/5074 Differential Revision: D14505514 Pulled By: miasantreble fbshipit-source-id: 4334462958c2b9c5a7c68c6ab24dadf94ad70902 18 March 2019, 18:35:06 UTC
b4fa51d anand76 15 March 2019, 22:15:01 UTC Update bg_error when log flush fails in SwitchMemtable() (#5072) Summary: There is a potential failure case in DBImpl::SwitchMemtable() that is not handled properly. The call to cur_log_writer->WriteBuffer() can fail due to an IO error. In that case, we need to call SetBGError() in order set the background error since the WriteBuffer() failure may result in data loss. Also, the asserts for !new_mem and !new_log are incorrect, as those would have been allocated by the time this failure is detected. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5072 Differential Revision: D14461384 Pulled By: anand1976 fbshipit-source-id: fb59bce9d61378f37d2dfcd28c0b704b0f43c3cf 15 March 2019, 22:19:25 UTC
2263f86 Andrew Kryczka 15 March 2019, 18:58:37 UTC exercise WAL recycling in crash test (#5070) Summary: Since this feature affects the WAL behavior, it seems important our crash-recovery tests cover it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5070 Differential Revision: D14470085 Pulled By: miasantreble fbshipit-source-id: 9b9682a718a926d57d055e0a5ec867efbd2eb9c1 15 March 2019, 19:03:26 UTC
dcde292 Zhichao Cao 15 March 2019, 03:00:11 UTC Add the -try_process_corrupted_trace option to trace_analyzer (#5067) Summary: In the current trace_analyzer implementation, once the trace file has corrupted content, which can be caused by unexpected tracing operations or other reasons, trace_analyzer will print the error and stop analyzing. By adding the -try_process_corrupted_trace option, user can try to process the corrupted trace file and get the analyzing results of the trace records from the beginning to the the first corrupted point in the trace file. Analyzing might fail even this option is enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5067 Differential Revision: D14433037 Pulled By: zhichao-cao fbshipit-source-id: d095233ba371726869af0def0cdee23b69896831 15 March 2019, 03:03:01 UTC
fdc72a5 Zhongyi Xie 12 March 2019, 20:46:12 UTC add OptionType kInt32T and kInt64T Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5061 Differential Revision: D14418581 Pulled By: miasantreble fbshipit-source-id: be7f90e16586666ddd0cce36971e403782ab0892 12 March 2019, 20:49:52 UTC
5a5c049 Andrew Kryczka 12 March 2019, 20:04:20 UTC ldb: set `total_order_seek` for scans (#5066) Summary: Without `total_order_seek=true`, using this command with `prefix_extractor` set skips over lots of keys. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5066 Differential Revision: D14425967 Pulled By: sagar0 fbshipit-source-id: f6f142733258d92604f920615be9266e1fe797f8 12 March 2019, 20:10:39 UTC
8a1ecd1 Yi Wu 09 March 2019, 01:02:00 UTC Fix build failures due to missing JEMALLOC_CXX_THROW macro (#5053) Summary: JEMALLOC_CXX_THROW is not defined for earlier versions of jemalloc (e.g. 3.6), causing builds to fail on some platforms. Fixing it. Closes #4869 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5053 Differential Revision: D14390034 Pulled By: sagar0 fbshipit-source-id: b2b7a03cd377201ef385eb521f65bae85c558055 09 March 2019, 01:06:13 UTC
c4e00cf Maysam Yabandeh 09 March 2019, 00:46:17 UTC Blog post for format_version=4 Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/5055 Differential Revision: D14395944 Pulled By: maysamyabandeh fbshipit-source-id: 385062b59428c132ada4e49b327685ba1f5d30e6 09 March 2019, 00:49:30 UTC
05ebfeb Zhichao Cao 08 March 2019, 22:07:19 UTC Fixed the potential stack overflow of MixGraph in db_bench (#5051) Summary: In the MixGraph benchmark of db_bench, The max buffer size used for value of KV-pair might be extremely large (64MB), which might cause function stack overflow in some platforms, reduced to 1MB. Added the finished ops printing in MixGraph benchmark. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5051 Differential Revision: D14379571 Pulled By: zhichao-cao fbshipit-source-id: 24084fbe38f60f2902d9a40f6bc9a25e4e2c9bb9 08 March 2019, 22:10:17 UTC
62eb2c2 Yi Wu 08 March 2019, 19:15:51 UTC Print data block index options to info log (#5039) Summary: Print data block index type related options to info log Pull Request resolved: https://github.com/facebook/rocksdb/pull/5039 Differential Revision: D14387718 Pulled By: miasantreble fbshipit-source-id: 9df8f82eea83a8344c7d12a712486f656691bc4a 08 March 2019, 19:18:45 UTC
04d3ac4 Maysam Yabandeh 08 March 2019, 17:34:55 UTC Fix tsan compliant on AddPreparedBeforeMax (#5052) Summary: Add a mutex to the test to synchronize before accessing the shared txn object. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5052 Differential Revision: D14386861 Pulled By: maysamyabandeh fbshipit-source-id: 5b32e209840b210c35af53848dc77f489a76c95a 08 March 2019, 17:39:00 UTC
79b6ab4 Levi Tamasi 07 March 2019, 18:15:24 UTC BlobDB: Remove GC interval option (#5044) Summary: Remove BlobDBOptions.garbage_collection_interval_secs for now, since garbage collection is not yet implemented in BlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5044 Differential Revision: D14354046 Pulled By: ltamasi fbshipit-source-id: 2b966b6d1e088ba9462f3ea73e115013562fbc04 07 March 2019, 18:19:05 UTC
04a2631 Maysam Yabandeh 07 March 2019, 15:32:01 UTC WritePrepared: handle adding prepare before max_evicted_seq_ (#5025) Summary: The patch fixes an improbable race condition between AddPrepared from one write queue and AdvanceMaxEvictedSeq from another queue. In this scenario AddPrepared finds prepare_seq lower than max and adding to PrepareHeap as usual while AdvanceMaxEvictedSeq has finished checking PrepareHeap against the future max. Thus when AdvanceMaxEvictedSeq finishes off by updating the max_evicted_seq_, PrepareHeap ends up with a prepared_seq lower than it which breaks the PrepareHeap contract. The fix is that in AddPrepared we check against the future_max_evicted_seq_ instead, which is update before AdvanceMaxEvictedSeq acquire prepare_mutex_ and looks into PrepareHeap. A unit test added to test for the failure scenario. The code is also refactored a bit to remove the duplicate code between AdvanceMaxEvictedSeq and AddPrepared. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5025 Differential Revision: D14249028 Pulled By: maysamyabandeh fbshipit-source-id: 072ea56663f40359662c05fafa6ac524417b0622 07 March 2019, 15:41:15 UTC
703f137 Maysam Yabandeh 07 March 2019, 15:26:36 UTC WritePrepared: Add rollback batch to PreparedHeap (#5026) Summary: The patch adds the sequence number of the rollback patch to the PrepareHeap when two_write_queues is enabled. Although the current behavior is still correct, the change simplifies reasoning about the code, by having all uncommitted batches registered with the PreparedHeap. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5026 Differential Revision: D14249401 Pulled By: maysamyabandeh fbshipit-source-id: 1e3424edee5cd14e56ee35931ad3c93ed997cd5a 07 March 2019, 15:33:31 UTC
186b3af Andrew Kryczka 04 March 2019, 23:40:26 UTC Use `fallocate` even if hole-punching unsupported (#5023) Summary: The compiler flag `-DROCKSDB_FALLOCATE_PRESENT` was only set when `fallocate`, `FALLOC_FL_KEEP_SIZE`, and `FALLOC_FL_PUNCH_HOLE` were all present. However, the last of the three is not really necessary for the primary `fallocate` use case; furthermore, it was introduced only in later Linux kernel versions (2.6.38+). This PR changes the flag `-DROCKSDB_FALLOCATE_PRESENT` to only require `fallocate` and `FALLOC_FL_KEEP_SIZE` to be present. There is a separate check for `FALLOC_FL_PUNCH_HOLE` only in the place where it is used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5023 Differential Revision: D14248487 Pulled By: siying fbshipit-source-id: a10ed0b902fa755988e957bd2dcec9081ec0502e 04 March 2019, 23:43:17 UTC
a283800 SeterKwok 04 March 2019, 22:51:13 UTC Move some RocksObject into try-with-resources in Test (#5037) Summary: Fix #5008 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5037 Differential Revision: D14302474 Pulled By: riversand963 fbshipit-source-id: dcd9dda5d4d6d459315692f355499a39e546d518 04 March 2019, 22:56:15 UTC
9a31b8d Sagar Vemuri 02 March 2019, 07:01:59 UTC Disable getApproximateSizes test (#5035) Summary: Disabling `org.rocksdb.RocksDBTest.getApproximateSizes` test as it is frequently crashing on travis (#5020). It will be re-enabled once the root-cause is found and fixed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5035 Differential Revision: D14294736 Pulled By: sagar0 fbshipit-source-id: e28bff0d143a58ad6c82991fec3d4cf8c0209995 02 March 2019, 07:05:32 UTC
0920bf4 Siying Dong 01 March 2019, 23:41:55 UTC Revert "Remove PlainTable's feature store_index_in_file (#4914)" (#5034) Summary: This reverts commit ee1818081ff4ca2a49a48cb4ca5b97665b8dcddf. We are not ready to deprecate this feature. revert it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5034 Differential Revision: D14287246 Pulled By: siying fbshipit-source-id: e4beafdeaee1c94364fdaa6ba198218d158339f7 01 March 2019, 23:45:45 UTC
18d2e4b Andrew Kryczka 01 March 2019, 19:14:28 UTC Run db_bench on database generated externally (#5017) Summary: Added an option, `-use_existing_keys`, which can be set to run benchmarks against an arbitrary existing database. Now users can benchmark against their actual database rather than synthetic data. Before the run begins, it loads all the keys into memory, then uses that set of keys rather than synthesizing new ones in `GenerateKeyFromInt`. This is mainly intended for small-scale DBs where the memory consumption is not a concern. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5017 Differential Revision: D14270303 Pulled By: riversand963 fbshipit-source-id: 6328df9dffb5e19170270dd00a69f4bbe424e5ed 01 March 2019, 19:19:03 UTC
aef763b Siying Dong 01 March 2019, 18:39:00 UTC Make statistics's stats_level change thread-safe (#5030) Summary: Right now, users can change statistics.stats_level while DB is running, but TSAN may report data race. We make stats_level_ to be atomic, and access them using accessors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5030 Differential Revision: D14267519 Pulled By: siying fbshipit-source-id: 37d7ebeff7a43a406230143422a16af899163f73 01 March 2019, 18:42:09 UTC
916e524 Fosco Marotto 01 March 2019, 18:28:38 UTC Merge pull request #5031 from gfosco/defsbzl [sync fix] Add defs.bzl 01 March 2019, 18:28:38 UTC
0b80f6b Maysam Yabandeh 01 March 2019, 17:15:46 UTC WritePrepared: script to analyze stress test failures (#5033) Summary: This the hackish script we used to find the root cause of failures in transaction stress tests. It is not well-written and does not require rigorous reviewing but it is better than starting from scratch each time we observe an issue. The stress tests would just say that at which snapshots the sum of all the keys in a set is inconsistent with another set. To help debugging one need to know which key exactly returned inconsistent results. The script looks at the transactions between two conflicting snapshots, and performs thee changes manually to see for which key the read value was inconsistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5033 Differential Revision: D14280362 Pulled By: maysamyabandeh fbshipit-source-id: d5826055c46711460ba81480d96cb5ea082814a5 01 March 2019, 17:18:40 UTC
77ebc82 Maysam Yabandeh 28 February 2019, 23:45:48 UTC Call PreReleaseCallback between WAL and memtable write (#5015) Summary: PreReleaseCallback meant to be called before the writes are visible to the readers. Since the sequence number is known after the WAL write, there is no reason to delay calling PreReleaseCallback to after the memtable write, which would complicates the reader's logic in presence of our memtable writes that are made visible by the other write thread. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5015 Differential Revision: D14221670 Pulled By: maysamyabandeh fbshipit-source-id: a504dd665cf923226d7af09cc8e9c7739a25edc6 28 February 2019, 23:49:11 UTC
68a2f94 Maysam Yabandeh 28 February 2019, 23:20:40 UTC WritePrepared: commit only from the 2nd queue (#5014) Summary: When two_write_queues is enabled we call ::AddPrepared only from the main queue, which writes to both WAL and memtable, and call ::AddCommitted from the 2nd queue, which writes only to WAL. This simplifies the logic by avoiding concurrency between AddPrepared and also between AddCommitted. The patch fixes one case that did not conform with the rule above. This would allow future refactoring. For example AdvaneMaxEvictedSeq, which is invoked by AddCommitted, can be simplified by assuming lack of concurrent calls to it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5014 Differential Revision: D14210493 Pulled By: maysamyabandeh fbshipit-source-id: 6db5ba372a294a568a14caa010576460917a4eab 28 February 2019, 23:23:34 UTC
06ea73d Sagar Vemuri 28 February 2019, 22:43:55 UTC Fix DefaultEnvTest.incBackgroundThreadsIfNeeded test (#5021) Summary: `DefaultEnvTest.incBackgroundThreadsIfNeeded` jtest should assert that the number of threads is greater than or equal to the minimum number of threads. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5021 Differential Revision: D14268311 Pulled By: sagar0 fbshipit-source-id: 01fb32b5b3ce636451d162fa1a2bbc5bd1974682 28 February 2019, 22:48:57 UTC
f83eecf Levi Tamasi 28 February 2019, 19:48:45 UTC Introduce an enum for flag types in LRUHandle (#5024) Summary: Replace the integers used for setting and querying the various flags in LRUHandle with enum values to improve readability. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5024 Differential Revision: D14263429 Pulled By: ltamasi fbshipit-source-id: b1b9ba95635265f122c2b40da73850eaac18227a 28 February 2019, 19:54:24 UTC
b157d3d Fosco Marotto 28 February 2019, 19:35:30 UTC [sync fix] Add defs.bzl 28 February 2019, 19:35:30 UTC
5e298f8 Siying Dong 28 February 2019, 18:14:19 UTC Add two more StatsLevel (#5027) Summary: Statistics cost too much CPU for some use cases. Add two stats levels so that people can choose to skip two types of expensive stats, timers and histograms. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5027 Differential Revision: D14252765 Pulled By: siying fbshipit-source-id: 75ecec9eaa44c06118229df4f80c366115346592 28 February 2019, 18:27:59 UTC
a661c0d Maysam Yabandeh 27 February 2019, 00:52:20 UTC WritePrepared: optimize read path by avoiding virtual (#5018) Summary: The read path includes a callback function, ReadCallback, which would eventually calls IsInSnapshot to figure if a particular seq is in the reading snapshot or not. This callback is virtual, which adds the cost of multiple virtual function call to each read. The first few checks in IsInSnapshot, however, are quite trivial and take care of majority of the cases. The patch moves those to a non-virtual function in the the parent class, ReadCallback, to lower the virtual callback cost. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5018 Differential Revision: D14226562 Pulled By: maysamyabandeh fbshipit-source-id: 6feed5b34f3b082e52092c5ef143e29b49c46b44 27 February 2019, 00:56:19 UTC
bb474e9 Adam Retter 22 February 2019, 22:36:38 UTC Add missing functionality to RocksJava (#4833) Summary: This is my latest round of changes to add missing items to RocksJava. More to come in future PRs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4833 Differential Revision: D14152266 Pulled By: sagar0 fbshipit-source-id: d6cff67e26da06c131491b5cf6911a8cd0db0775 22 February 2019, 22:46:46 UTC
06f378d Siying Dong 22 February 2019, 01:23:05 UTC When closing BlobDB, should first wait for all background tasks (#5005) Summary: When closing a BlobDB, it only waits for background tasks to finish as the last thing, but the background task may access some variables that are destroyed. The fix is to introduce a shutdown function in the timer queue and call the function as the first thing when destorying BlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5005 Differential Revision: D14170342 Pulled By: siying fbshipit-source-id: 081e6a2d99b9765d5956cf6cdfc290c07270c233 22 February 2019, 01:26:01 UTC
c4f5d0a Zhongyi Xie 20 February 2019, 23:46:59 UTC add GetStatsHistory to retrieve stats snapshots (#4748) Summary: This PR adds public `GetStatsHistory` API to retrieve stats history in the form of an std map. The key of the map is the timestamp in microseconds when the stats snapshot is taken, the value is another std map from stats name to stats value (stored in std string). Two DBOptions are introduced: `stats_persist_period_sec` (default 10 minutes) controls the intervals between two snapshots are taken; `max_stats_history_count` (default 10) controls the max number of history snapshots to keep in memory. RocksDB will stop collecting stats snapshots if `stats_persist_period_sec` is set to 0. (This PR is the in-memory part of https://github.com/facebook/rocksdb/pull/4535) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4748 Differential Revision: D13961471 Pulled By: miasantreble fbshipit-source-id: ac836d401ecb84ea92216bf9966f969dedf4ad04 20 February 2019, 23:52:54 UTC
48c8d84 Fosco Marotto 20 February 2019, 18:10:11 UTC Update version and history for 6.0 20 February 2019, 18:10:11 UTC
cf98df3 Maysam Yabandeh 20 February 2019, 03:55:30 UTC Change random seed for txn stress tests on each run (#5004) Summary: Currently the transaction stress tests use thread id as the seed. Since the thread ids are likely to be the same across multiple runs, the seed is thus going to be the same. The patch includes time in calculating the seed to help covering a very different part of state space in each run of the stress tests. To be able to reproduce the bug in case the stress tests failed, it also prints out the time that was used to calculate the seed value. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5004 Differential Revision: D14144356 Pulled By: maysamyabandeh fbshipit-source-id: 728ed522f550fc8b4f5f9f373259c05fe9a54556 20 February 2019, 03:58:55 UTC
0f4244f Maysam Yabandeh 20 February 2019, 00:52:50 UTC WritePrepared: Improve stress tests with slow threads (#4974) Summary: The transaction stress tests, stress a high concurrency scenario. In WritePrepared/WriteUnPrepared we need to also stress the scenarios where an inserting/reading transaction is very slow. This would stress the corner cases that the caching is not sufficient and other slower data structures are engaged. To emulate such cases we make use of slow inserter/verifier threads and also reduce the size of cache data structures. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4974 Differential Revision: D14143070 Pulled By: maysamyabandeh fbshipit-source-id: 81eb674678faf9fae0f654cd60ebcc74e26aeee7 20 February 2019, 00:56:49 UTC
bcdc8c8 Maysam Yabandeh 20 February 2019, 00:09:50 UTC WritePrepared: max_evicted_seq_ update during commit cache lookup (#4955) Summary: max_evicted_seq_ could be updated in the middle of the read in ::IsInSnapshot. The code to be correct in presence of this update would be complicated. The patch simplifies it by checking the value of max_evicted_seq_ before and after looking into commit_cache_ and retries in the unlucky case that it was changed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4955 Differential Revision: D13999556 Pulled By: maysamyabandeh fbshipit-source-id: 7a1bdfa95ea8b5d8d73ddff3263ed31d7297b39c 20 February 2019, 00:14:08 UTC
93f7e7a Siying Dong 19 February 2019, 22:41:16 UTC Temporarily Disable DBTest2.PresetCompressionDict (#5003) Summary: DBTest2.PresetCompressionDict is flaky. Temparily disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5003 Differential Revision: D14139505 Pulled By: siying fbshipit-source-id: ebf1872d364b76b2cb021b489ea2f17ee997116a 19 February 2019, 22:44:12 UTC
7d23210 Yanqin Jin 19 February 2019, 22:02:30 UTC Separate crash test with atomic flush (#4945) Summary: Currently crash test covers cases with and without atomic flush, but takes too long to finish. Therefore it may be a better idea to put crash test with atomic flush in a separate set of tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4945 Differential Revision: D13947548 Pulled By: riversand963 fbshipit-source-id: 177c6de865290fd650b0103408339eaa3f801d8c 19 February 2019, 22:08:39 UTC
3c5d1b1 Michael Liu 19 February 2019, 21:36:04 UTC Apply modernize-use-override (3) Summary: Use C++11’s override and remove virtual where applicable. Change are automatically generated. bypass-lint drop-conflicts Reviewed By: igorsugak Differential Revision: D14131816 fbshipit-source-id: f20e7f7cecf2e699d70f5fa036f72c0e3f59b50e 19 February 2019, 21:39:49 UTC
ed995c6 Zhongyi Xie 19 February 2019, 20:12:25 UTC add whole key bloom filter support in memtables (#4985) Summary: MyRocks calls `GetForUpdate` on `INSERT`, for unique key check, and in almost all cases GetForUpdate returns empty result. For such cases, whole key bloom filter is helpful. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4985 Differential Revision: D14118257 Pulled By: miasantreble fbshipit-source-id: d35cb7109c62fd5ad541a26968e3a3e16d3e85ea 19 February 2019, 20:15:39 UTC
c2affcc Siying Dong 16 February 2019, 00:56:58 UTC Header logger should call LogHeader() (#4980) Summary: The info log header feature never worked well, because log level Header was not translated to Logger::LogHeader() call. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4980 Differential Revision: D14087283 Pulled By: siying fbshipit-source-id: 7e7d03ce35fa8d13d4ee549f46f7326f7bc0006d 16 February 2019, 00:59:36 UTC
26a33ee Siying Dong 16 February 2019, 00:30:23 UTC flush_job logs data size too (#4979) Summary: Right now when a flush is triggered, the memory consumption is logged but data size is not. It's useful to log both when we debug unexpected small flushed file size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4979 Differential Revision: D14071979 Pulled By: siying fbshipit-source-id: 0cd60449c5205eb00e0fbc299084418f609904ed 16 February 2019, 00:33:19 UTC
4db46aa Siying Dong 16 February 2019, 00:10:37 UTC Fix LITE Build (#4989) Summary: LITE mode has EventListener to be an empty class. However in db_bench, it is used. When "override" is added to the functions, the build breaks. Fix it by keeping the listener empty in LITE mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4989 Differential Revision: D14108132 Pulled By: siying fbshipit-source-id: 80121aab35b1120e502b37b782301dd700692697 16 February 2019, 00:13:11 UTC
3231a2e Aubin Sanyal 15 February 2019, 17:48:44 UTC Deprecate ttl option from CompactionOptionsFIFO (#4965) Summary: We introduced ttl option in CompactionOptionsFIFO when ttl-based file deletion (compaction) was supported only as part of FIFO Compaction. But with the extension of ttl semantics even to Level compaction, CompactionOptionsFIFO.ttl can now be deprecated. Instead we will start using ColumnFamilyOptions.ttl for FIFO compaction as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4965 Differential Revision: D14072960 Pulled By: sagar0 fbshipit-source-id: c98cc2ae695a28136295787cd88d36a220fc219e 15 February 2019, 17:51:41 UTC
ca89ac2 Michael Liu 14 February 2019, 21:52:47 UTC Apply modernize-use-override (2nd iteration) Summary: Use C++11’s override and remove virtual where applicable. Change are automatically generated. Reviewed By: Orvid Differential Revision: D14090024 fbshipit-source-id: 1e9432e87d2657e1ff0028e15370a85d1739ba2a 14 February 2019, 22:41:36 UTC
c8c8104 Andrew Kryczka 14 February 2019, 18:16:12 UTC Dictionary compression for files written by SstFileWriter (#4978) Summary: If `CompressionOptions::max_dict_bytes` and/or `CompressionOptions::zstd_max_train_bytes` are set, `SstFileWriter` will now generate files respecting those options. I refactored the logic a bit for deciding when to use dictionary compression. Previously we plumbed `is_bottommost_level` down to the table builder and used that. However it was kind of confusing in `SstFileWriter`'s context since we don't know what level the file will be ingested to. Instead, now the higher-level callers (e.g., flush, compaction, file writer) are responsible for building the right `CompressionOptions` to give the table builder. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4978 Differential Revision: D14060763 Pulled By: ajkr fbshipit-source-id: dc802c327896df2b319dc162d6acc82b9cdb452a 14 February 2019, 19:23:55 UTC
4fc4420 Yanqin Jin 14 February 2019, 02:23:10 UTC Avoid using kInAtomicGroup tag for single-cf op (#4981) Summary: if an operation just involves a single column family, then we do not have to set the kInAtomicGroup tag when writing to MANIFEST. This change can fix a compatibility test failure, i.e. 5.15 and earlier cannot recognize kInAtomicGroup tag. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4981 Differential Revision: D14072687 Pulled By: riversand963 fbshipit-source-id: 46b0c61e399f16c6b7169de0b33430d0ed90d6d4 14 February 2019, 02:33:42 UTC
34b55dd Andrew Kryczka 14 February 2019, 00:37:38 UTC Fix no compression CI test config (#4982) Summary: We should strip `-DZSTD` to prevent ZSTD from being used in the no compression tests, similarly to how we prevent all other compression libraries from being used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4982 Differential Revision: D14075349 Pulled By: ajkr fbshipit-source-id: 8bd861516cf28a568c2b701ad33d0bb658db93b2 14 February 2019, 00:47:01 UTC
51a9041 Philip Jameson 13 February 2019, 18:54:25 UTC Add load statements to rocksdb TARGETS files Reviewed By: siying Differential Revision: D13993686 fbshipit-source-id: 0c55e8952307bcf457c1d78d527a0c86b59628e8 13 February 2019, 22:08:21 UTC
5af9446 Yanqin Jin 13 February 2019, 18:28:22 UTC Remove Lua compaction filter from RocksDB main repo (#4971) Summary: as title. For people who continue to need Lua compaction filter, you can copy the include/rocksdb/utilities/rocks_lua/lua_compaction_filter.h and utilities/lua/rocks_lua_compaction_filter.cc to your own codebase. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4971 Differential Revision: D14047468 Pulled By: riversand963 fbshipit-source-id: 9ad1a6484a7c94e478f1e108127a3184e4069f70 13 February 2019, 20:42:44 UTC
a69d4de Yanqin Jin 13 February 2019, 03:07:25 UTC Atomic ingest (#4895) Summary: Make file ingestion atomic. as title. Ingesting external SST files into multiple column families should be atomic. If a crash occurs and db reopens, either all column families have successfully ingested the files before the crash, or non of the ingestions have any effect on the state of the db. Also add unit tests for atomic ingestion. Note that the unit test here does not cover the case of incomplete atomic group in the MANIFEST, which is covered in VersionSetTest already. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4895 Differential Revision: D13718245 Pulled By: riversand963 fbshipit-source-id: 7df97cc483af73ad44dd6993008f99b083852198 13 February 2019, 03:16:17 UTC
33b3323 Adam Retter 13 February 2019, 00:31:53 UTC Add Java multiGet API for returning List<byte[]> (#1570) Summary: Closes https://github.com/facebook/rocksdb/pull/1570 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4797 Differential Revision: D13961770 Pulled By: sagar0 fbshipit-source-id: e34fd6250d0cd3ebb0bd688e8801fe8947fd464d 13 February 2019, 01:04:48 UTC
49ddd7e Siying Dong 13 February 2019, 00:19:56 UTC Stats should be logged in INFO level (#4977) Summary: Previously, stats were logged in warning level. This was done in that way because people reported that it wasn't logged in MyRocks. However, later we learned that it turns out to be due to a bug in MyRocks, which is fixed in https://github.com/facebook/mysql-5.6/commit/79bb705e74b239d7030b724ea6bbd635eceec531 Now we revert the stats logging to INFO level, so that it doesn't pollute the warning level logging. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4977 Differential Revision: D14058485 Pulled By: siying fbshipit-source-id: 19fab323c19d9bc88184287f209551f9a77ca0e6 13 February 2019, 00:54:55 UTC
eafb09a Sagar Vemuri 12 February 2019, 21:55:40 UTC Fix issues found by Clang Analyzer (#4976) Summary: Fix issues found by Clang Analyzer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4976 Differential Revision: D14054211 Pulled By: sagar0 fbshipit-source-id: ec2053bae43af3b2ff3425306824c677e3ba70c2 12 February 2019, 21:59:44 UTC
c5a64cf Yanqin Jin 12 February 2019, 20:01:55 UTC Avoid fsync on the same directory in atomic flush (#4817) Summary: In `DBImpl::AtomicFlushMemTablesToOutputFiles`, we need to call fsync only once on the same data directory. If two column families share a common directory for their data, we call fsync only once. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4817 Differential Revision: D13543689 Pulled By: riversand963 fbshipit-source-id: 4701d77c96a47802fbf6cb9f3337ee65d46b95f5 12 February 2019, 20:28:36 UTC
62f70f6 Andrew Kryczka 12 February 2019, 03:42:25 UTC Reduce scope of compression dictionary to single SST (#4952) Summary: Our previous approach was to train one compression dictionary per compaction, using the first output SST to train a dictionary, and then applying it on subsequent SSTs in the same compaction. While this was great for minimizing CPU/memory/I/O overhead, it did not achieve good compression ratios in practice. In our most promising potential use case, moderate reductions in a dictionary's scope make a major difference on compression ratio. So, this PR changes compression dictionary to be scoped per-SST. It accepts the tradeoff during table building to use more memory and CPU. Important changes include: - The `BlockBasedTableBuilder` has a new state when dictionary compression is in-use: `kBuffered`. In that state it accumulates uncompressed data in-memory whenever `Add` is called. - After accumulating target file size bytes or calling `BlockBasedTableBuilder::Finish`, a `BlockBasedTableBuilder` moves to the `kUnbuffered` state. The transition (`EnterUnbuffered()`) involves sampling the buffered data, training a dictionary, and compressing/writing out all buffered data. In the `kUnbuffered` state, a `BlockBasedTableBuilder` behaves the same as before -- blocks are compressed/written out as soon as they fill up. - Samples are now whole uncompressed data blocks, except the final sample may be a partial data block so we don't breach the user's configured `max_dict_bytes` or `zstd_max_train_bytes`. The dictionary trainer is supposed to work better when we pass it real units of compression. Previously we were passing 64-byte KV samples which was not realistic. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4952 Differential Revision: D13967980 Pulled By: ajkr fbshipit-source-id: 82bea6f7537e1529c7a1a4cdee84585f5949300f 12 February 2019, 03:47:32 UTC
79496d7 Peter (Stig) Edwards 12 February 2019, 01:46:49 UTC Increment NUMBER_BLOCK_NOT_COMPRESSED when !GoodCompressionRatio (#4929) Summary: See https://github.com/facebook/rocksdb/issues/4884 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4929 Differential Revision: D14028333 Pulled By: sagar0 fbshipit-source-id: eed12bceae85385a34aaa6dd303bf0f53c4c7b06 12 February 2019, 01:56:23 UTC
d6b9b3b Maysam Yabandeh 11 February 2019, 23:59:30 UTC Enhance transaction_test_util with delays (#4970) Summary: Enhance ::Insert and ::Verify test functions to add artificial delay between prepare and commit, and take snapshot and reads respectively. A future PR will make use of these to improve stress tests to test against long-running transactions as well as long-running backup jobs. Also randomly sets set_snapshot to false for inserters to skip setting the snapshot in the initialization phase and let the snapshot be taken later explicitly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4970 Differential Revision: D14031342 Pulled By: maysamyabandeh fbshipit-source-id: b52b453751f0b25b81b23c48892bc1d152464cab 12 February 2019, 00:02:37 UTC
576d2d6 Maysam Yabandeh 11 February 2019, 22:56:50 UTC WritePrepared: relax assert in compaction iterator (#4969) Summary: If IsInSnapshot(seq2, snapshot) determines that the snapshot is released, the future queries IsInSnapshot(seq1, snapshot) could still return a definitive answer of true if for example seq1 is too old that is determined visible in all snapshots. This violates a recently added assert statement to compaction iterator. The patch relaxes the assert. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4969 Differential Revision: D14030998 Pulled By: maysamyabandeh fbshipit-source-id: 6db53db0e37d0a20e8997ef2c1004b8627614ab9 11 February 2019, 23:01:46 UTC
back to top