| 5a6ad9d | Andrew Kryczka | 05 October 2017, 19:32:55 UTC | release build treat warnings as errors Summary: fixing warnings is important, especially for release code. Closes https://github.com/facebook/rocksdb/pull/2971 Differential Revision: D5980596 Pulled By: ajkr fbshipit-source-id: 04f4ea3fb005dcda33d60342e4361e380bc4dfb1 | 05 October 2017, 19:41:52 UTC |
| 4e3c3d8 | Maysam Yabandeh | 05 October 2017, 14:28:37 UTC | WritePrepared Txn: duplicate keys Summary: With WriteCommitted, when the write batch has duplicate keys, the txn db simply inserts them to the db with different seq numbers and let the db ignore/merge the duplicate values at the read time. With WritePrepared all the entries of the batch are inserted with the same seq number which prevents us from benefiting from this simple solution. This patch applies a hackish solution to unblock the end-to-end testing. The hack is to be replaced with a proper solution soon. The patch simply detects the duplicate key insertions, and mark the previous one as obsolete. Then before writing to the db it rewrites the batch eliminating the obsolete keys. This would incur a memcpy cost. Furthermore handing duplicate merge would require to do FullMerge instead of simply ignoring the previous value, which is not handled by this patch. Closes https://github.com/facebook/rocksdb/pull/2969 Differential Revision: D5976337 Pulled By: maysamyabandeh fbshipit-source-id: 114e65b66f137d8454ff2d1d782b8c05da95f989 | 05 October 2017, 14:41:02 UTC |
| 1026e79 | Andrew Kryczka | 05 October 2017, 02:02:22 UTC | rate limit auto-tuning Summary: Dynamic adjustment of rate limit according to demand for background I/O. It increases by a factor when limiter is drained too frequently, and decreases by the same factor when limiter is not drained frequently enough. The parameters for this behavior are fixed in `GenericRateLimiter::Tune`. Other changes: - make rate limiter's `Env*` configurable for testing - track num drain intervals in RateLimiter so we don't have to rely on stats, which may be shared across different DB instances from the ones that share the RateLimiter. Closes https://github.com/facebook/rocksdb/pull/2899 Differential Revision: D5858704 Pulled By: ajkr fbshipit-source-id: cc2bac30f85e7f6fd63655d0a6732ef9ed7403b1 | 05 October 2017, 02:15:01 UTC |
| 75f7f42 | Adam Kupczyk | 05 October 2017, 01:03:29 UTC | Added CPU prefetch for skiplist Summary: This change causes following changes result of test: ./db_bench --writes 10000000 --benchmarks="fillrandom" --compression_type none from fillrandom : 3.177 micros/op 314804 ops/sec; 34.8 MB/s to fillrandom : 2.777 micros/op 360087 ops/sec; 39.8 MB/s Closes https://github.com/facebook/rocksdb/pull/2961 Differential Revision: D5977822 Pulled By: yiwu-arbug fbshipit-source-id: 1ea77707bffa978b1592b0c5d0fe76bfa1930f8d | 05 October 2017, 01:12:52 UTC |
| 88ed1f6 | Manuel Ung | 04 October 2017, 16:44:18 UTC | Allow upgrades from nullptr to some merge operator Summary: Currently, RocksDB does not allow reopening a preexisting DB with no merge operator defined, with a merge operator defined. This means that if a DB ever want to add a merge operator, there's no way to do so currently. Fix this by adding a new verification type `kByNameAllowFromNull` which will allow old values to be nullptr, and new values to be non-nullptr. Closes https://github.com/facebook/rocksdb/pull/2958 Differential Revision: D5961131 Pulled By: lth fbshipit-source-id: 06179bebd0d90db3d43690b5eb7345e2d5bab1eb | 04 October 2017, 16:57:23 UTC |
| 5b2cb64 | Andrew Kryczka | 03 October 2017, 23:25:42 UTC | Prevent threads from respawning during joining Summary: Previously the thread pool might be non-empty after joining since concurrent submissions could spawn new threads. This problem didn't affect our background flush/compaction thread pools because the `shutting_down_` flag prevented new jobs from being submitted during/after joining. But I wanted to be able to reuse the `ThreadPool` without such external synchronization. Closes https://github.com/facebook/rocksdb/pull/2953 Differential Revision: D5951920 Pulled By: ajkr fbshipit-source-id: 0efec7d0056d36d1338367da75e8b0c089bbc973 | 03 October 2017, 23:27:28 UTC |
| 8218870 | Andrew Kryczka | 03 October 2017, 23:24:24 UTC | pin L0 filters/indexes for compaction outputs Summary: We need to tell the iterator the compaction output file's level so it can apply proper optimizations, like pinning filter and index blocks when user enables `pin_l0_filter_and_index_blocks_in_cache` and the output file's level is zero. Closes https://github.com/facebook/rocksdb/pull/2949 Differential Revision: D5945597 Pulled By: ajkr fbshipit-source-id: 2389decf9026ffaa32d45801a77d002529f64a62 | 03 October 2017, 23:27:28 UTC |
| 283d607 | Maysam Yabandeh | 03 October 2017, 21:40:50 UTC | fix valgrind leak report in unit test Summary: I cannot locally reproduce the valgrind leak report but based on my code inspection not deleting txn1 might be the reason. ``` ==197848== 2,990 (544 direct, 2,446 indirect) bytes in 1 blocks are definitely lost in loss record 15 of 16 ==197848== at 0x4C2D06F: operator new(unsigned long) (in /usr/local/fbcode/gcc-5-glibc-2.23/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==197848== by 0x7D5B31: rocksdb::WritePreparedTxnDB::BeginTransaction(rocksdb::WriteOptions const&, rocksdb::TransactionOptions const&, rocksdb::Transaction*) (pessimistic_transaction_db.cc:173) ==197848== by 0x7D80C1: rocksdb::PessimisticTransactionDB::Initialize(std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> > const&) (pessimistic_transaction_db.cc:115) ==197848== by 0x7DC42F: rocksdb::WritePreparedTxnDB::Initialize(std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> > const&) (pessimistic_transaction_db.cc:151) ==197848== by 0x7D8CA0: rocksdb::TransactionDB::WrapDB(rocksdb::DB*, rocksdb::TransactionDBOptions const&, std::vector<unsigned long, std::allocator<unsigned long> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> > const&, rocksdb::TransactionDB**) (pessimistic_transaction_db.cc:275) ==197848== by 0x7D9F26: rocksdb::TransactionDB::Open(rocksdb::DBOptions const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::TransactionDB**) (pessimistic_transaction_db.cc:227) ==197848== by 0x7DB349: rocksdb::TransactionDB::Open(rocksdb::Options const&, rocksdb::TransactionDBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::TransactionDB**) (pessimistic_transaction_db.cc:198) ==197848== by 0x52ABD2: rocksdb::TransactionTest::ReOpenNoDelete() (transaction_test.h:87) ==197848== by 0x51F7B8: rocksdb::WritePreparedTransactionTest_BasicRecoveryTest_Test::TestBody() (write_prepared_transaction_test.cc:843) ==197848== by 0x857557: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest-all.cc:3824) ==197848== by 0x857557: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest-all.cc:3860) ==197848== by 0x84E7EB: testing::Test::Run() [clone .part.485] (gtest-all.cc:3897) ==197848== by 0x84E9BC: Run (gtest-all.cc:3888) ==197848== by 0x84E9BC: testing::TestInfo::Run() [clone .part.486] (gtest-all.cc:4072) ``` Closes https://github.com/facebook/rocksdb/pull/2963 Differential Revision: D5968856 Pulled By: maysamyabandeh fbshipit-source-id: 2ac512bbcad37dc8eeeffe4f363978913354180c | 03 October 2017, 21:58:07 UTC |
| 377e004 | Sagar Vemuri | 03 October 2017, 20:36:53 UTC | Fix DBOptionsTest.SetBytesPerSync test when run with no compression Summary: Also made the test more easier to understand: - changed the value size to ~1MB. - switched to NoCompression. We don't anyway need compression in this test for dynamic options. The test failures started happening starting from: #2893 . Closes https://github.com/facebook/rocksdb/pull/2957 Differential Revision: D5959392 Pulled By: sagar0 fbshipit-source-id: 2d55641e429246328bc6d10fcb9ef540d6ce07da | 03 October 2017, 20:42:11 UTC |
| 92ccae7 | Yi Wu | 03 October 2017, 18:08:11 UTC | speedup 'make check' Summary: Make SnapshotConcurrentAccessTest run in the beginning of the queue. Test Plan `make all check -j64` on devserver Closes https://github.com/facebook/rocksdb/pull/2962 Differential Revision: D5965871 Pulled By: yiwu-arbug fbshipit-source-id: 8cb5a47c2468be0fbbb929226a143ec5848bfaa9 | 03 October 2017, 19:11:49 UTC |
| d1cab2b | Yi Wu | 03 October 2017, 16:08:07 UTC | Add ValueType::kTypeBlobIndex Summary: Add kTypeBlobIndex value type, which will be used by blob db only, to insert a (key, blob_offset) KV pair. The purpose is to 1. Make it possible to open existing rocksdb instance as blob db. Existing value will be of kTypeIndex type, while value inserted by blob db will be of kTypeBlobIndex. 2. Make rocksdb able to detect if the db contains value written by blob db, if so return error. 3. Make it possible to have blob db optionally store value in SST file (with kTypeValue type) or as a blob value (with kTypeBlobIndex type). The root db (DBImpl) basically pretended kTypeBlobIndex are normal value on write. On Get if is_blob is provided, return whether the value read is of kTypeBlobIndex type, or return Status::NotSupported() status if is_blob is not provided. On scan allow_blob flag is pass and if the flag is true, return wether the value is of kTypeBlobIndex type via iter->IsBlob(). Changes on blob db side will be in a separate patch. Closes https://github.com/facebook/rocksdb/pull/2886 Differential Revision: D5838431 Pulled By: yiwu-arbug fbshipit-source-id: 3c5306c62bc13bb11abc03422ec5cbcea1203cca | 03 October 2017, 16:11:23 UTC |
| 880411f | Andrew Kryczka | 03 October 2017, 03:40:23 UTC | disable populating block cache for in-place updates Summary: There's no point populating the block cache during this read. The key we read is guaranteed to be overwritten with a new `kValueType` key immediately afterwards, so can't be accessed again. A user was seeing high turnover of data blocks, at least partially due to this. Closes https://github.com/facebook/rocksdb/pull/2959 Differential Revision: D5961672 Pulled By: ajkr fbshipit-source-id: e7cb27c156c5db3b32af355c780efb99dbdf087c | 03 October 2017, 03:41:24 UTC |
| d27258d | Maysam Yabandeh | 03 October 2017, 02:46:42 UTC | WritePrepared Txn: Rollback Summary: Implement the rollback of WritePrepared txns. For each modified value, it reads the value before the txn and write it back. This would cancel out the effect of transaction. It also remove the rolled back txn from prepared heap. Closes https://github.com/facebook/rocksdb/pull/2946 Differential Revision: D5937575 Pulled By: maysamyabandeh fbshipit-source-id: a6d3c47f44db3729f44b287a80f97d08dc4e888d | 03 October 2017, 02:59:27 UTC |
| bb38cd0 | Sagar Vemuri | 02 October 2017, 23:08:53 UTC | Limit number of merge operands in Cassandra merge operator Summary: Now that RocksDB supports conditional merging during point lookups (introduced in #2923), Cassandra value merge operator can be updated to pass in a limit. The limit needs to be passed in from the Cassandra code. Closes https://github.com/facebook/rocksdb/pull/2947 Differential Revision: D5938454 Pulled By: sagar0 fbshipit-source-id: d64a72d53170d8cf202b53bd648475c3952f7d7f | 02 October 2017, 23:11:40 UTC |
| cf51d3e | Aliaksei Sandryhaila | 02 October 2017, 22:11:40 UTC | Remove an "unused" variable Summary: PR 2893 introduced a variable that is only used in TEST_SYNC_POINT_CALLBACK. When RocksDB is not built in debug mode, this method is not compiled in, and the variable is unused, which triggers a compiler error. This patch reverts the corresponding part of #2893. Closes https://github.com/facebook/rocksdb/pull/2956 Reviewed By: yiwu-arbug Differential Revision: D5955679 Pulled By: asandryh fbshipit-source-id: ac4a8e85b22da7f02efb117cd2e4a6e07ba73390 | 02 October 2017, 22:26:29 UTC |
| 983028f | Adam Retter | 02 October 2017, 18:06:26 UTC | RocksJava build target for Docker on ppc64le Summary: This enables us to crossbuild pcc64le RocksJava binaries with a suitably old version of glibc (2.17) on CentOS 7. Closes https://github.com/facebook/rocksdb/pull/2491 Differential Revision: D5955301 Pulled By: sagar0 fbshipit-source-id: 69ef9746f1dc30ffde4063dc764583d8c7ae937e | 02 October 2017, 18:11:56 UTC |
| 2a3363d | Siying Dong | 02 October 2017, 16:40:00 UTC | ldb dump can print histogram of value size Summary: Make "ldb dump --count_only" print histogram of value size. Also, fix a bug that "ldb dump --path=<db_path>" doesn't work. Closes https://github.com/facebook/rocksdb/pull/2944 Differential Revision: D5954527 Pulled By: siying fbshipit-source-id: c620a444ec544258b8d113f5f663c375dd53d6be | 02 October 2017, 16:41:17 UTC |
| 593d3de | Zhongyi Xie | 30 September 2017, 03:25:31 UTC | No need for Restart Interval for meta blocks Summary: In SST files, restart interval helps us search in data blocks. However, some meta blocks will be read sequentially, so there's no need for restart points. Restart interval will introduce extra space in the block (https://github.com/facebook/rocksdb/blob/master/table/block_builder.cc#L80). We will see if we can remove this redundant space. (Maybe set restart interval to infinite.) Closes https://github.com/facebook/rocksdb/pull/2940 Differential Revision: D5930139 Pulled By: miasantreble fbshipit-source-id: 92b1b23c15cffa90378343ac846b713623b19c21 | 30 September 2017, 03:26:20 UTC |
| 2b22baf | Maysam Yabandeh | 29 September 2017, 18:33:32 UTC | Add a template for issues Summary: This template reminds the users to use issues only for bug reports. The template is written according to the github guidelines at https://help.github.com/articles/creating-an-issue-template-for-your-repository/ Closes https://github.com/facebook/rocksdb/pull/2948 Differential Revision: D5943558 Pulled By: maysamyabandeh fbshipit-source-id: c83b5d211ea8e334107141967689b2f0c453bbc9 | 29 September 2017, 18:41:28 UTC |
| ab0542f | Maysam Yabandeh | 29 September 2017, 14:55:22 UTC | Fix for when block.cache_handle is nullptr Summary: When using with compressed cache it is possible that the status is ok but the block is not actually added to the block cache. The patch takes this case into account. Closes https://github.com/facebook/rocksdb/pull/2945 Differential Revision: D5937613 Pulled By: maysamyabandeh fbshipit-source-id: 5428cf1115e5046b3d01ab78d26cb181122af4c6 | 29 September 2017, 14:56:55 UTC |
| 5df172d | Andrew Kryczka | 29 September 2017, 01:07:42 UTC | fix deletion-triggered compaction in table builder Summary: It was broken when `NotifyCollectTableCollectorsOnFinish` was introduced. That function called `Finish` on each of the `TablePropertiesCollector`s, and `CompactOnDeletionCollector::Finish()` was resetting all its internal state. Then, when we checked whether compaction is necessary, the flag had already been cleared. Fixed above issue by avoiding resetting internal state during `Finish()`. Multiple calls to `Finish()` are allowed, but callers cannot invoke `AddUserKey()` on the collector after any finishes. Closes https://github.com/facebook/rocksdb/pull/2936 Differential Revision: D5918659 Pulled By: ajkr fbshipit-source-id: 4f05e9d80e50ee762ba1e611d8d22620029dca6b | 29 September 2017, 01:17:30 UTC |
| 385049b | Maysam Yabandeh | 28 September 2017, 23:43:04 UTC | WritePrepared Txn: Recovery Summary: Recover txns from the WAL. Also added some unit tests. Closes https://github.com/facebook/rocksdb/pull/2901 Differential Revision: D5859596 Pulled By: maysamyabandeh fbshipit-source-id: 6424967b231388093b4effffe0a3b1b7ec8caeb0 | 28 September 2017, 23:56:45 UTC |
| 8c724f5 | Yu Shu | 28 September 2017, 23:09:08 UTC | Default one to rocksdb:x64-windows Summary: The default one will try to install rocksdb:x86-windows, which would lead to failing of the build at the last step (CMake Error, Rocksdb only supports x64). Because it will try to install a serials of x86 version package, and those cannot proceed to rocksdb:x86-windows building. By using rocksdb:x64-windows, we can make sure to install x64 version. Tested on Win10 x64. Closes https://github.com/facebook/rocksdb/pull/2941 Differential Revision: D5937139 Pulled By: sagar0 fbshipit-source-id: 15637fe23df59326a0e607bd4d5c48733e20bae3 | 28 September 2017, 23:12:24 UTC |
| 93c2b91 | Sagar Vemuri | 28 September 2017, 22:49:05 UTC | Introduce conditional merge-operator invocation in point lookups Summary: For every merge operand encountered for a key in the read path we now have the ability to decide whether to look further (to retrieve more merge operands for the key) or stop and invoke the merge operator to return the value. The user needs to override `ShouldMerge()` method with a condition to terminate search when true to avail this facility. This has a couple of advantages: 1. It helps in limiting the number of merge operands that are looked at to compute a value as part of a user Get operation. 2. It allows to peek at a merge key-value to see if further merge operands need to look at. Example: Limiting the number of merge operands that are looked at: Lets say you have 10 merge operands for a key spread over various levels. If you only want RocksDB to look at the latest two merge operands instead of all 10 to compute the value, it is now possible with this PR. You can set the condition in `ShouldMerge()` to return true when the size of the operand list is 2. Look at the example implementation in the unit test. Without this PR, a Get might look at all the 10 merge operands in different levels before invoking the merge-operator. Added a new unit test. Made sure that there is no perf regression by running benchmarks. Command line to Load data: ``` TEST_TMPDIR=/dev/shm ./db_bench --benchmarks="mergerandom" --merge_operator="uint64add" --num=10000000 ... mergerandom : 12.861 micros/op 77757 ops/sec; 8.6 MB/s ( updates:10000000) ``` **ReadRandomMergeRandom bechmark results:** Command line: ``` TEST_TMPDIR=/dev/shm ./db_bench --benchmarks="readrandommergerandom" --merge_operator="uint64add" --num=10000000 ``` Base -- Without this code change (on commit fc7476b): ``` readrandommergerandom : 38.586 micros/op 25916 ops/sec; (reads:3001599 merges:6998401 total:10000000 hits:842235 maxlength:8) ``` With this code change: ``` readrandommergerandom : 38.653 micros/op 25870 ops/sec; (reads:3001599 merges:6998401 total:10000000 hits:842235 maxlength:8) ``` Closes https://github.com/facebook/rocksdb/pull/2923 Differential Revision: D5898239 Pulled By: sagar0 fbshipit-source-id: daefa325019f77968639a75c851d46352c2303ef | 28 September 2017, 22:58:49 UTC |
| a48a398 | Aliaksei Sandryhaila | 28 September 2017, 21:22:06 UTC | Use RAII instead of pointers in cf_info_map Summary: There is no need for smart pointers in cf_info_map, so use RAII. This should also placate valgrind. Closes https://github.com/facebook/rocksdb/pull/2943 Differential Revision: D5932941 Pulled By: asandryh fbshipit-source-id: 2c37df88573a9df2557880a31193926e4425e054 | 28 September 2017, 21:26:47 UTC |
| c705866 | Maysam Yabandeh | 28 September 2017, 17:13:11 UTC | Blog post for 5.8 release Summary: Closes https://github.com/facebook/rocksdb/pull/2942 Differential Revision: D5932858 Pulled By: maysamyabandeh fbshipit-source-id: e11f52a0b08d65149bb49d99d1dbc82cb5a96fa0 | 28 September 2017, 17:14:09 UTC |
| c2f6e45 | Andrew Kryczka | 28 September 2017, 06:57:07 UTC | prevent nullptr dereference in table reader error case Summary: A user encountered segfault on the call to `CacheDependencies()`, probably because `NewIndexIterator()` failed before populating `*index_entry`. Let's avoid the call in that case. Closes https://github.com/facebook/rocksdb/pull/2939 Differential Revision: D5928611 Pulled By: ajkr fbshipit-source-id: 484be453dbb00e5e160e9c6a1bc933df7d80f574 | 28 September 2017, 07:12:34 UTC |
| 6a541af | Quinn Jarrell | 28 September 2017, 00:37:08 UTC | Make bytes_per_sync and wal_bytes_per_sync mutable Summary: SUMMARY Moves the bytes_per_sync and wal_bytes_per_sync options from immutableoptions to mutable options. Also if wal_bytes_per_sync is changed, the wal file and memtables are flushed. TEST PLAN ran make check all passed Two new tests SetBytesPerSync, SetWalBytesPerSync check that after issuing setoptions with a new value for the var, the db options have the new value. Closes https://github.com/facebook/rocksdb/pull/2893 Reviewed By: yiwu-arbug Differential Revision: D5845814 Pulled By: TheRushingWookie fbshipit-source-id: 93b52d779ce623691b546679dcd984a06d2ad1bd | 28 September 2017, 00:49:45 UTC |
| ec48e5c | Yi Wu | 27 September 2017, 17:24:42 UTC | Add TransactionDB::SingleDelete() Summary: Looks like the API is simply missing. Adding it. Closes https://github.com/facebook/rocksdb/pull/2937 Differential Revision: D5919955 Pulled By: yiwu-arbug fbshipit-source-id: 6e2e9c96c29882b0bb4113d1f8efb72bffc57878 | 27 September 2017, 17:27:26 UTC |
| 0806801 | Sagar Vemuri | 26 September 2017, 23:27:28 UTC | DestroyDB API Summary: Expose DestroyDB API in RocksJava. Closes https://github.com/facebook/rocksdb/pull/2934 Differential Revision: D5914775 Pulled By: sagar0 fbshipit-source-id: 84af6ea0d2bccdcfb9fe8c07b2f87373f0d5bab6 | 26 September 2017, 23:42:11 UTC |
| aa67bae | Maysam Yabandeh | 26 September 2017, 21:16:12 UTC | Break down PinnedDataIteratorRandomized Summary: Its timing out under tsan. Closes https://github.com/facebook/rocksdb/pull/2928 Differential Revision: D5911766 Pulled By: maysamyabandeh fbshipit-source-id: 2faacc07752ac8713a3a2abb5a4c4b7ae3bdf208 | 26 September 2017, 21:27:30 UTC |
| 4748911 | Siying Dong | 25 September 2017, 22:46:52 UTC | Add LogDevice to USERS.md Summary: Closes https://github.com/facebook/rocksdb/pull/2927 Differential Revision: D5906613 Pulled By: siying fbshipit-source-id: 607401e05b27508c816c700864fe81514606e4ef | 25 September 2017, 22:56:40 UTC |
| 1d6700f | Zhongyi Xie | 23 September 2017, 00:24:14 UTC | Add test kPointInTimeRecoveryCFConsistency Summary: Context/problem: - CFs may be flushed at different times - A WAL can only be deleted after all CFs have flushed beyond end of that WAL. - Point-in-time recovery might stop upon reaching the first corruption. - Some CFs may have already flushed beyond that point, while others haven't. We should fail the Open() instead of proceeding with inconsistent CFs. Closes https://github.com/facebook/rocksdb/pull/2900 Differential Revision: D5863281 Pulled By: miasantreble fbshipit-source-id: 180dbaf83d96c804cff49b3c406312a4ae61313e | 23 September 2017, 00:26:36 UTC |
| be97dbb | Yi Wu | 22 September 2017, 22:24:42 UTC | Fix WritePreparedTransactionTest::SeqAdvanceTest ASAN failure Summary: Closes https://github.com/facebook/rocksdb/pull/2922 Differential Revision: D5895310 Pulled By: yiwu-arbug fbshipit-source-id: 52c635a25d22478ec1eca49b6817551202babac2 | 22 September 2017, 22:26:42 UTC |
| 4708a68 | Andrew Kryczka | 22 September 2017, 19:37:59 UTC | Repair DBs with trailing slash in name Summary: Problem: - `DB::SanitizeOptions` strips trailing slash from `wal_dir` but not `dbname` - We check whether `wal_dir` and `dbname` refer to the same directory using string equality: https://github.com/facebook/rocksdb/blob/master/db/repair.cc#L258 - Providing `dbname` with trailing slash causes default `wal_dir` to be misidentified as a separate directory. - Then the repair tries to add all SST files to the `VersionEdit` twice (once for `dbname` dir, once for `wal_dir`) and fails with coredump. Solution: - Add a new `Env` function, `AreFilesSame`, which uses device and inode number to check whether files are the same. It's currently only implemented in `PosixEnv`. - Migrate repair to use `AreFilesSame` to check whether `dbname` and `wal_dir` are same. If unsupported, falls back to string comparison. Closes https://github.com/facebook/rocksdb/pull/2827 Differential Revision: D5761349 Pulled By: ajkr fbshipit-source-id: c839d548678b742af1166d60b09abd94e5476238 | 22 September 2017, 19:42:22 UTC |
| fc7476b | Andrew Kryczka | 22 September 2017, 00:53:48 UTC | fix populating range deletions in forward iterator Summary: fixes #2902 Closes https://github.com/facebook/rocksdb/pull/2917 Differential Revision: D5887175 Pulled By: ajkr fbshipit-source-id: 364e292c636a3238bfc53b0fb9a01ff2f82dcbb9 | 22 September 2017, 00:56:38 UTC |
| c8f3606 | Sagar Vemuri | 22 September 2017, 00:15:27 UTC | Expose LoadLatestOptions, LoadOptionsFromFile and GetLatestOptionsFileName APIs in RocksJava Summary: JNI wrappers for LoadLatestOptions, LoadOptionsFromFile and GetLatestOptionsFileName APIs. Closes https://github.com/facebook/rocksdb/pull/2898 Differential Revision: D5857934 Pulled By: sagar0 fbshipit-source-id: 68b79e83eab8de9416e3f1fef73e11cf7947e90a | 22 September 2017, 00:29:13 UTC |
| 96a13b4 | Sagar Vemuri | 21 September 2017, 23:39:22 UTC | Use jemalloc in rocksdbjni library built via vagrant Summary: Problem: During RocksJava performance testing we found that the rocksdb jni library is not built with jemalloc; instead it was getting built with the default glibc malloc. We saw quite a bit of memory bloat due to this. Addressed this by installing jemalloc-devel package in the vm that we use to build release jars. Closes https://github.com/facebook/rocksdb/pull/2916 Differential Revision: D5887018 Pulled By: sagar0 fbshipit-source-id: ace0b5d60234b3a30dcd5d39633e7827a5982a50 | 21 September 2017, 23:42:06 UTC |
| 65a9cd6 | PhaniShekhar | 21 September 2017, 22:50:46 UTC | Use L1 size as estimate for L0 size in LevelCompactionBuilder::GetPathID Summary: Fix for [2461](https://github.com/facebook/rocksdb/issues/2461). Problem: When using multiple db_paths setting with RocksDB, RocksDB incorrectly calculates the size of L1 in LevelCompactionBuilder::GetPathId. max_bytes_for_level_base is used as L0 size and L1 size is calculated as (L0 size * max_bytes_for_level_multiplier). However, L1 size should be max_bytes_for_level_base. Solution: Use max_bytes_for_level_base as L1 size. Also, use L1 size as the estimated size of L0. Closes https://github.com/facebook/rocksdb/pull/2903 Differential Revision: D5885442 Pulled By: maysamyabandeh fbshipit-source-id: 036da1c9298d173b9b80479cc6661ee4b7a951f6 | 21 September 2017, 22:57:58 UTC |
| 8fc3de3 | Andrew Kryczka | 21 September 2017, 18:03:53 UTC | make rate limiter a general option Summary: it's unsupported in options file, so the flag should be respected by db_bench even when an options file is provided. Closes https://github.com/facebook/rocksdb/pull/2910 Differential Revision: D5869836 Pulled By: ajkr fbshipit-source-id: f67f591ae083e95e989f86b6fad50765d2e3d855 | 21 September 2017, 18:11:00 UTC |
| 1480e6f | Yi Wu | 21 September 2017, 16:47:14 UTC | Fix TransactionTest::SeqAdvanceTest ASAN failure Summary: The test didn't delete txn before creating a new one. Closes https://github.com/facebook/rocksdb/pull/2913 Differential Revision: D5880236 Pulled By: yiwu-arbug fbshipit-source-id: 7a4fcaada3d86332292754502cd8f4341143bf4f | 21 September 2017, 16:56:54 UTC |
| 3fc08fa | Sagar Vemuri | 20 September 2017, 17:23:51 UTC | Expose max_background_jobs option in RocksJava Summary: This option was introduced in the C++ API in RocksDB 5.6 in bb01c1880c0c89a6cf338e22fd54e7e25b7d12ba . Now, exposing it through RocksJava API. Closes https://github.com/facebook/rocksdb/pull/2908 Differential Revision: D5864224 Pulled By: sagar0 fbshipit-source-id: 140aa55dcf74b14e4d11219d996735c7fdddf513 | 20 September 2017, 17:26:37 UTC |
| 8ae8168 | Yao Zongyou | 19 September 2017, 18:56:45 UTC | Update cmake_minimum_required to 2.8.12. Summary: Hello, current master branch declares cmake_minimum_required (VERSION 2.8.11) but cmake gives the following error: [ 6%] CMake Error at CMakeLists.txt:658 (install): install TARGETS given unknown argument "INCLUDES". CMake Error at src/CMakeLists.txt:658 (install): install TARGETS given unknown argument "INCLUDES". because this argument not supported on CMake versions prior 2.8.12 Closes https://github.com/facebook/rocksdb/pull/2904 Differential Revision: D5863430 Pulled By: yiwu-arbug fbshipit-source-id: 0f7230e080add472ad4b87836b3104ea0b971a38 | 19 September 2017, 19:01:09 UTC |
| b4596c6 | Yi Wu | 19 September 2017, 18:45:42 UTC | Fix Get does not return super version on error Summary: This is caught when I was testing #2886. Closes https://github.com/facebook/rocksdb/pull/2907 Differential Revision: D5863153 Pulled By: yiwu-arbug fbshipit-source-id: 8c54759ba1a0dc101f24ab50423e35731300612d | 19 September 2017, 19:01:09 UTC |
| 34ebadf | Orgad Shaneh | 19 September 2017, 17:15:13 UTC | Fix MinGW build Summary: snprintf is defined as _snprintf, which doesn't exist in the std namespace. Closes https://github.com/facebook/rocksdb/pull/2298 Differential Revision: D5070457 Pulled By: yiwu-arbug fbshipit-source-id: 6e1659ac3e86170653b174578da5a8ed16812cbb | 19 September 2017, 17:28:26 UTC |
| e4234fb | Pengchao Wang | 18 September 2017, 23:19:06 UTC | collecting kValue type tombstone Summary: In our testing cluster, we found large amount tombstone has been promoted to kValue type from kMerge after reaching the top level of compaction. Since we used to only collecting tombstone in merge operator, those tombstones can never be collected. This PR addresses the issue by adding a GC step in compaction filter, which is only for kValue type records. Since those record already reached the top of compaction (no earlier data exists) we can safely remove them in compaction filter without worrying old data appears. This PR also removes an old optimization in cassandra merge operator for single merge operands. We need to do GC even on a single operand, so the optimation does not make sense anymore. Closes https://github.com/facebook/rocksdb/pull/2855 Reviewed By: sagar0 Differential Revision: D5806445 Pulled By: wpc fbshipit-source-id: 6eb25629d4ce917eb5e8b489f64a6aa78c7d270b | 18 September 2017, 23:27:12 UTC |
| 60beefd | Maysam Yabandeh | 18 September 2017, 21:36:53 UTC | WritePrepared Txn: Advance seq one per batch Summary: By default the seq number in DB is increased once per written key. WritePrepared txns requires the seq to be increased once per the entire batch so that the seq would be used as the prepare timestamp by which the transaction is identified. Also we need to increase seq for the commit marker since it would give a unique id to the commit timestamp of transactions. Two unit tests are added to verify our understanding of how the seq should be increased. The recovery path requires much more work and is left to another patch. Closes https://github.com/facebook/rocksdb/pull/2885 Differential Revision: D5837843 Pulled By: maysamyabandeh fbshipit-source-id: a08960b93d727e1cf438c254d0c2636fb133cc1c | 18 September 2017, 21:45:08 UTC |
| c57050b | Maysam Yabandeh | 16 September 2017, 00:09:48 UTC | Use the default copy constructor in Options Summary: Our current implementation of (semi-)copy constructor of DBOptions and ColumnFamilyOptions seems to intend value by value copy, which is what the default copy constructor does anyway. Moreover not using the default constructor has the risk of forgetting to add newly added options. As an example, allow_2pc seems to be forgotten in the copy constructor which was causing one of the unit tests not seeing its effect. Closes https://github.com/facebook/rocksdb/pull/2888 Differential Revision: D5846368 Pulled By: maysamyabandeh fbshipit-source-id: 1ee92a2aeae93886754b7bc039c3411ea2458683 | 16 September 2017, 00:15:10 UTC |
| c319792 | Siying Dong | 16 September 2017, 00:02:37 UTC | Directly refernce perf_context internally. Summary: After 7f6c02dda16471c2ed3318e9e7156f2b8d13bf46, the same get_perf_context() is called both of internally and externally. However, I found internally this is not got inlined. I don't know why this is the case, but directly referencing perf_context is the logical way to do. Closes https://github.com/facebook/rocksdb/pull/2892 Differential Revision: D5843789 Pulled By: siying fbshipit-source-id: b49777d8809f35847699291bb7f8ea2754c3af49 | 16 September 2017, 00:15:10 UTC |
| 6b3c71f | Yi Wu | 15 September 2017, 18:45:33 UTC | Fix DBImpl::NotifyOnCompactionCompleted data race Summary: Access of `cfd->current()` needs to hold db mutex. The data race is caught by TSAN but hard to reproduce: https://gist.github.com/yiwu-arbug/0fc6dc0de915297a1740aa9610be9373 Closes https://github.com/facebook/rocksdb/pull/2894 Differential Revision: D5843884 Pulled By: yiwu-arbug fbshipit-source-id: 0a30a421bc96f51840821538ad6453dc0815a942 | 15 September 2017, 18:56:31 UTC |
| f47b4ee | Yi Wu | 15 September 2017, 18:20:16 UTC | Fix memory leak in OptionsTest::OptionsComposeDecompose Summary: Fixing asan error. Closes https://github.com/facebook/rocksdb/pull/2887 Differential Revision: D5838895 Pulled By: yiwu-arbug fbshipit-source-id: 1662ce9856eb5e6877675347dc2240f2acb6fae8 | 15 September 2017, 18:37:37 UTC |
| 382277d | Ben Clay | 15 September 2017, 01:18:49 UTC | JNI support for ReadOptions::iterate_upper_bound Summary: Plumbed ReadOptions::iterate_upper_bound through JNI. Made the following design choices: * Used Slice instead of AbstractSlice due to the anticipated usecase (key / key prefix). Can change this if anyone disagrees. * Used Slice instead of raw byte[] which seemed cleaner but necessitated the package-private handle-based Slice constructor. Followed WriteBatch as an example. * We need a copy constructor for ReadOptions, as we create one base ReadOptions for a particular usecase and clone -> change the iterate_upper_bound on each slice operation. Shallow copy seemed cleanest. * Hold a reference to the upper bound slice on ReadOptions, in contrast to Snapshot. Signed a Facebook CLA this morning. Closes https://github.com/facebook/rocksdb/pull/2872 Differential Revision: D5824446 Pulled By: sagar0 fbshipit-source-id: 74fc51313a10a81ecd348625e2a50ca5b7766888 | 15 September 2017, 01:28:20 UTC |
| edcbb36 | Siying Dong | 15 September 2017, 00:43:41 UTC | Three code-level optimization to Iterator::Next() Summary: Three small optimizations: (1) iter_->IsKeyPinned() shouldn't be called if read_options.pin_data is not true. This may trigger function call all the way down the iterator tree. (2) reuse the iterator key object in DBIter::FindNextUserEntryInternal(). The constructor of the class has some overheads. (3) Move the switching direction logic in MergingIterator::Next() to a separate function. These three in total improves readseq performance by about 3% in my benchmark setting. Closes https://github.com/facebook/rocksdb/pull/2880 Differential Revision: D5829252 Pulled By: siying fbshipit-source-id: 991aea10c6d6c3b43769cb4db168db62954ad1e3 | 15 September 2017, 00:57:31 UTC |
| 885b1c6 | Siying Dong | 14 September 2017, 22:41:19 UTC | Two small refactoring for better inlining Summary: Move uncommon code paths in RangeDelAggregator::ShouldDelete() and IterKey::EnlargeBufferIfNeeded() to a separate function, so that the inlined strcuture can be more optimized. Optimize it because these places show up in CPU profiling, though minimum. The performance is really hard measure. I ran db_bench with readseq benchmark against in-memory DB many times. The variation is big, but it seems to show 1% improvements. Closes https://github.com/facebook/rocksdb/pull/2877 Differential Revision: D5828123 Pulled By: siying fbshipit-source-id: 41a49e229f91e9f8409f85cc6f0dc70e31334e4b | 14 September 2017, 22:41:49 UTC |
| ffac683 | Oleksandr Anyshchenko | 14 September 2017, 21:04:40 UTC | Added save points for transactions C API Summary: Added possibility to set save points in transactions and then rollback to them Closes https://github.com/facebook/rocksdb/pull/2876 Differential Revision: D5825829 Pulled By: yiwu-arbug fbshipit-source-id: 62168992340bbcddecdaea3baa2a678475d1429d | 14 September 2017, 21:18:59 UTC |
| 9a970c8 | Yi Wu | 14 September 2017, 00:21:35 UTC | Fix WriteBatchWithIndex::GetFromBatchAndDB not allowing StackableDB Summary: Closes https://github.com/facebook/rocksdb/pull/2881 Differential Revision: D5829682 Pulled By: yiwu-arbug fbshipit-source-id: abb8fa14b58cea7c416282f9be19e8b1a7961c6e | 14 September 2017, 00:26:35 UTC |
| a843df6 | Yi Wu | 13 September 2017, 22:56:57 UTC | Fix use-after-free in c_tset Summary: Fix asan error introduce by #2823 Closes https://github.com/facebook/rocksdb/pull/2879 Differential Revision: D5828454 Pulled By: yiwu-arbug fbshipit-source-id: 50777855667f4e7b634279a654c3bfa01a1ac729 | 13 September 2017, 23:12:02 UTC |
| 2d6e421 | Sagar Vemuri | 13 September 2017, 22:50:38 UTC | Remove 'experimental' comment around level_compaction_dynamic_level_bytes option Summary: Remove misleading 'experimental' comment around `level_compaction_dynamic_level_bytes` option. This is not experimental anymore and is ready for wider adoption. MyRocks is already using it in production. Closes https://github.com/facebook/rocksdb/pull/2878 Differential Revision: D5828890 Pulled By: sagar0 fbshipit-source-id: fffb45f4999f689b7eca326e4f4caf472d40c5a9 | 13 September 2017, 22:56:24 UTC |
| 464fb36 | Andrew Kryczka | 13 September 2017, 22:31:34 UTC | fix hanging after CompactFiles with L0 overlap Summary: Bug report: https://www.facebook.com/groups/rocksdb.dev/permalink/1389452781153232/ Non-empty `level0_compactions_in_progress_` was aborting `CompactFiles` after incrementing `bg_compaction_scheduled_`, and in that case we never decremented it. This blocked future compactions and prevented DB close as we wait for scheduled compactions to finish/abort during close. I eliminated `CompactFiles`'s dependency on `level0_compactions_in_progress_`. Since it takes a contiguous span of L0 files -- through the last L0 file if any L1+ files are included -- it's fine to run in parallel with other compactions involving L0. We make the same assumption in intra-L0 compaction. Closes https://github.com/facebook/rocksdb/pull/2849 Differential Revision: D5780440 Pulled By: ajkr fbshipit-source-id: 15b15d3faf5a699aed4b82a58352d4a7bb23e027 | 13 September 2017, 22:41:38 UTC |
| 09713a6 | Maysam Yabandeh | 13 September 2017, 18:56:27 UTC | WritePrepared Txn: Lock-free CommitMap Summary: We had two proposals for lock-free commit maps. This patch implements the latter one that was simpler. We can later experiment with both proposals. In this impl each entry is an std::atomic of uint64_t, which are accessed via memory_order_acquire/release. In x86_64 arch this is compiled to simple reads and writes from memory. Closes https://github.com/facebook/rocksdb/pull/2861 Differential Revision: D5800724 Pulled By: maysamyabandeh fbshipit-source-id: 41abae9a4a5df050a8eb696c43de11c2770afdda | 13 September 2017, 19:12:11 UTC |
| 72e4190 | Oleksandr Anyshchenko | 13 September 2017, 18:56:19 UTC | Additions for `OptimisticTransactionDB` in C API Summary: Added some bindings for `OptimisticTransactionDB` in C API Closes https://github.com/facebook/rocksdb/pull/2823 Differential Revision: D5820672 Pulled By: yiwu-arbug fbshipit-source-id: 7efd17f619cc0741feddd2050b8fc856f9288350 | 13 September 2017, 19:12:11 UTC |
| 9d115d3 | Andrew Kryczka | 13 September 2017, 18:48:16 UTC | regression test for missing init options Summary: test the `DBOptions(const Options&)` and `ColumnFamilyOptions(const Options&)` constructors. Actually this'll work better once we refactor `RandomInitDBOptions` / `RandomInitCFOptions` to use the authoritative sources of struct members: `db_options_type_info` / `cf_options_type_info` (internal task T21804189 for this). Closes https://github.com/facebook/rocksdb/pull/2873 Differential Revision: D5817141 Pulled By: ajkr fbshipit-source-id: 8567c20feced9d1751fdf1f4383e2af30f7e3591 | 13 September 2017, 18:56:35 UTC |
| f615f56 | gladiator | 13 September 2017, 00:51:53 UTC | fix missing manual_wal_flush for DBOptions ctor Summary: currently `ImmutableDBOptions::Dump` use default value for `concurrent_prepare` and `manual_wal_flush`, because DBOptions ctor does not init those member variables. so in LOG file, it will be ``` Options.concurrent_prepare: 0 Options.manual_wal_flush: 0 ``` Closes https://github.com/facebook/rocksdb/pull/2864 Differential Revision: D5816240 Pulled By: ajkr fbshipit-source-id: 82335e8bcae3dceedc6a99224e7998de5fad1e50 | 13 September 2017, 01:01:08 UTC |
| 5785b1f | Amy Xu | 13 September 2017, 00:16:44 UTC | Fix naming in InternalKey Summary: - Switched all instances of SetMinPossibleForUserKey and SetMaxPossibleForUserKey in accordance to InternalKeyComparator's comparison logic Closes https://github.com/facebook/rocksdb/pull/2868 Differential Revision: D5804152 Pulled By: axxufb fbshipit-source-id: 80be35e04f2e8abc35cc64abe1fecb03af24e183 | 13 September 2017, 00:17:42 UTC |
| 82860bd | Bernhard M. Wiedemann | 13 September 2017, 00:12:07 UTC | Use cmake TIMESTAMP function Summary: because it is not only platform independent but also allows to override the build date This helps to make ceph builds reproducible (that includes a fork of rockdb in a submodule) Also adds UTC flag, to be independent of timezone. Requires cmake-2.8.11+ from 2013 Closes https://github.com/facebook/rocksdb/pull/2848 Differential Revision: D5820189 Pulled By: yiwu-arbug fbshipit-source-id: e3e8c1550e10e238c173f6c5d9ba15f71ad3ce28 | 13 September 2017, 00:17:42 UTC |
| 2d30aaa | Maysam Yabandeh | 12 September 2017, 21:48:19 UTC | Exclude incompatible options in test Summary: options.enable_pipelined_write and options.concurrent_prepare are incompatible and should not be set together. Closes https://github.com/facebook/rocksdb/pull/2875 Differential Revision: D5818358 Pulled By: maysamyabandeh fbshipit-source-id: dad862508f00817ab302f8b61729accf38315fb8 | 12 September 2017, 21:58:46 UTC |
| f5148ad | Andrew Kryczka | 12 September 2017, 20:12:58 UTC | support opening zero backups during engine init Summary: There are internal users who open BackupEngine for writing new backups only, and they don't care whether old backups can be read or not. The condition `BackupableDBOptions::max_valid_backups_to_open == 0` should be supported (previously in df74b775e6e3034714bb75b42aa8f97e9a47dd81 I made the mistake of choosing 0 as a special value to disable the limit). Closes https://github.com/facebook/rocksdb/pull/2819 Differential Revision: D5751599 Pulled By: ajkr fbshipit-source-id: e73ac19eb5d756d6b68601eae8e43407ee4f2752 | 12 September 2017, 20:26:34 UTC |
| 3c42807 | Archit Mishra | 12 September 2017, 18:41:13 UTC | do not call merge when checking to see if key exists Summary: Changes: * added check for value before merge is called on code path that should check if key exists Closes https://github.com/facebook/rocksdb/pull/2814 Reviewed By: IslamAbdelRahman Differential Revision: D5743966 Pulled By: armishra fbshipit-source-id: 6ac4283bc510c8ca50827d87ef0ba631f2b33b18 | 12 September 2017, 19:02:53 UTC |
| 025b85b | Andrew Kryczka | 12 September 2017, 18:16:24 UTC | speedup DBTest.EncodeDecompressedBlockSizeTest Summary: it sometimes takes more than 10 minutes (i.e., times out) on our internal CI. mainly because bzip is super slow. so I reduced the amount of work it tries to do. Closes https://github.com/facebook/rocksdb/pull/2856 Differential Revision: D5795883 Pulled By: ajkr fbshipit-source-id: e69f986ae60b44ecc26b6b024abd0f13bdf3a3c5 | 12 September 2017, 18:26:47 UTC |
| 044a71e | zawlazaw | 12 September 2017, 17:41:48 UTC | Add iterator's SeekForPrev functionality to the java-api Summary: As discussed in #2742 , this pull-requests brings the iterator's [SeekForPrev()](https://github.com/facebook/rocksdb/wiki/SeekForPrev) functionality to the java-api. It affects all locations in the code where previously only Seek() was supported. All code changes are essentially a copy & paste of the already existing implementations for Seek(). **Please Note**: the changes to the C++ code were applied without fully understanding its effect, so please take a closer look. However, since Seek() and SeekForPrev() provide exactly the same signature, I do not expect any mistake here. The java-tests are extended by new tests for the additional functionality. Compilation (`make rocksdbjavastatic`) and test (`java/make test`) run without errors. Closes https://github.com/facebook/rocksdb/pull/2747 Differential Revision: D5721011 Pulled By: sagar0 fbshipit-source-id: c1f951cddc321592c70dd2d32bc04892f3f119f8 | 12 September 2017, 17:56:29 UTC |
| 64b6452 | Siying Dong | 11 September 2017, 18:53:22 UTC | Make InternalKeyComparator final and directly use it in merging iterator Summary: Merging iterator invokes InternalKeyComparator.Compare() frequently to heap merge. By making InternalKeyComparator final and merging iterator to directly use InternalKeyComparator rather than through Iterator interface, we can give compiler a choice to avoid one more virtual function call if possible. I ran readseq benchmark in memory-only use case to make sure the performance at least doesn't regress. I have to disable the final key word in debug build, as a hack test class depends on overriding the class. Closes https://github.com/facebook/rocksdb/pull/2860 Differential Revision: D5800461 Pulled By: siying fbshipit-source-id: ab876f22a09bb5c560740911412336e0e25ccb53 | 11 September 2017, 19:04:21 UTC |
| 2dd22e5 | Siying Dong | 11 September 2017, 18:43:17 UTC | Make DBIter class final Summary: DBIter is referenced in ArenaWrappedDBIter, which is a simple wrapper. If DBIter is final, some virtual function call can be avoided. Some functions can even be inlined, like DBIter.value() to ArenaWrappedDBIter.value() and DBIter.key() to ArenaWrappedDBIter.key(). The performance gain is hard to measure. I just ran the memory-only benchmark for readseq and saw it didn't regress. There shouldn't be any harm doing it. Just give compiler more choices. Closes https://github.com/facebook/rocksdb/pull/2859 Differential Revision: D5799888 Pulled By: siying fbshipit-source-id: 829788f91310c40282dcfb7e412e6ef489931143 | 11 September 2017, 19:04:21 UTC |
| 2a59150 | Huachao Huang | 11 September 2017, 18:31:47 UTC | Fix missing BYTES_PER_WRITE for pipeline write Summary: Closes https://github.com/facebook/rocksdb/pull/2862 Differential Revision: D5805638 Pulled By: yiwu-arbug fbshipit-source-id: 72d38c74395690023a719f400daff01527645a17 | 11 September 2017, 18:41:27 UTC |
| f46464d | Maysam Yabandeh | 11 September 2017, 15:58:52 UTC | write-prepared txn: call IsInSnapshot Summary: This patch instruments the read path to verify each read value against an optional ReadCallback class. If the value is rejected, the reader moves on to the next value. The WritePreparedTxn makes use of this feature to skip sequence numbers that are not in the read snapshot. Closes https://github.com/facebook/rocksdb/pull/2850 Differential Revision: D5787375 Pulled By: maysamyabandeh fbshipit-source-id: 49d808b3062ab35e7ae98ad388f659757794184c | 11 September 2017, 16:14:48 UTC |
| 9a4df72 | Maysam Yabandeh | 08 September 2017, 22:53:51 UTC | WritePrepared Txn: CommitBatch Summary: Implements CommitBatch and CommitWithoutPrepare for WritePreparedTxn Closes https://github.com/facebook/rocksdb/pull/2854 Differential Revision: D5793999 Pulled By: maysamyabandeh fbshipit-source-id: d8b9858221162c6ac7a1f6912cbd3481d0d8a503 | 08 September 2017, 22:56:39 UTC |
| fce6c89 | Maysam Yabandeh | 08 September 2017, 21:32:30 UTC | Advance max evicted seq in coarser granularity Summary: This patch advances the max_evicted_seq_ is larger granularities to reduce the overhead of updating the relevant data structures. It also refactor the related code and adds testing to that. As part of this patch some of the TODOs for removing usage of non-static const members are also addressed. Closes https://github.com/facebook/rocksdb/pull/2844 Differential Revision: D5772928 Pulled By: maysamyabandeh fbshipit-source-id: f4fcc2948be69c034f10812cf922ce5ab82ef98c | 08 September 2017, 21:41:22 UTC |
| dcd36a6 | Yi Wu | 08 September 2017, 17:57:12 UTC | Make it explicit blob db doesn't support CF Summary: Blob db doesn't currently support column families. Return NotSupported status explicitly. Closes https://github.com/facebook/rocksdb/pull/2825 Differential Revision: D5757438 Pulled By: yiwu-arbug fbshipit-source-id: 44de9408fd032c98e8ae337d4db4ed37169bd9fa | 08 September 2017, 18:11:04 UTC |
| 3cd7ea2 | Andrew Kryczka | 08 September 2017, 01:13:53 UTC | rename stall-related internal stats Summary: Some of these names, like `MEMTABLE_COMPACTION`, did not mean anything. Tried to give them descriptive names. Closes https://github.com/facebook/rocksdb/pull/2852 Differential Revision: D5782822 Pulled By: ajkr fbshipit-source-id: f2695c4124af4073da4492d7135bae2411220f3a | 08 September 2017, 01:26:18 UTC |
| 0e99323 | Siying Dong | 07 September 2017, 21:11:15 UTC | Fix CLANG Analyze Summary: clang analyze shows warnings after we upgrade the CLANG version. Fix them. Closes https://github.com/facebook/rocksdb/pull/2839 Differential Revision: D5769060 Pulled By: siying fbshipit-source-id: 3f8e4df715590d8984f6564b608fa08cfdfa5f14 | 07 September 2017, 21:28:06 UTC |
| ba3c58c | Kefu Chai | 07 September 2017, 19:32:50 UTC | specify SSE42 'target' attribute for Fast_CRC32() Summary: if we enable SSE42 globally when compiling the tree for preparing a portable binary, which could be running on CPU w/o SSE42 instructions even the GCC on the building host is able to emit SSE42 code, this leads to illegal instruction errors on machines not supporting SSE42. to solve this problem, crc32 detects the supported instruction at runtime, and selects the supported CRC32 implementation according to the result of `cpuid`. but intrinics like "_mm_crc32_u64()" will not be available unless the "target" machine is appropriately specified in the command line, like "-msse42", or using the "target" attribute. we could pass "-msse42" only when compiling crc32c.cc, and allow the compiler to generate the SSE42 instructions, but we are still at the risk of executing illegal instructions on machines does not support SSE42 if the compiler emits code that is not guarded by our runtime detection. and we need to do the change in both Makefile and CMakefile. or, we can use GCC's "target" attribute to enable the machine specific instructions on certain function. in this way, we have finer grained control of the used "target". and no need to change the makefiles. so we don't need to duplicate the changes on both makefile and cmake as the previous approach. this problem surfaces when preparing a package for GNU/Linux distribution, and we only applies to optimization for SSE42, so using a feature only available on GCC/Clang is not that formidable. Closes https://github.com/facebook/rocksdb/pull/2807 Differential Revision: D5786084 Pulled By: siying fbshipit-source-id: bca5c0f877b8d6fb55f58f8f122254a26422843d | 07 September 2017, 19:40:57 UTC |
| 7e19a57 | Maysam Yabandeh | 07 September 2017, 19:15:49 UTC | Remove unused TransactionCallback Summary: TransactionCallback was never used. Remove it to avoid confusion. Closes https://github.com/facebook/rocksdb/pull/2853 Differential Revision: D5787219 Pulled By: maysamyabandeh fbshipit-source-id: e2b6a89537e3770a269ad38be71c4b0b160a88ac | 07 September 2017, 19:17:18 UTC |
| 10ddd59 | Andrew Kryczka | 06 September 2017, 18:28:19 UTC | fix CompactFiles inclusion of older L0 files Summary: if we're moving any L0 files down, we need to include older L0 files since they may contain older versions of the keys being moved down. Closes https://github.com/facebook/rocksdb/pull/2845 Differential Revision: D5773800 Pulled By: ajkr fbshipit-source-id: 9f0770a8eaaeea4c87df2e7a2a1d65bf9d7f4f7e | 06 September 2017, 18:42:25 UTC |
| 79810e2 | Maysam Yabandeh | 05 September 2017, 22:28:29 UTC | Skip write_prepared_transaction_test in travis Summary: The patch skips write_prepared_transaction_test from travis as they time out there. They are still covered in daily runs of tests. Closes https://github.com/facebook/rocksdb/pull/2836 Differential Revision: D5767203 Pulled By: maysamyabandeh fbshipit-source-id: 51045ef98a745197136e14b2ec02fc6f38081b75 | 05 September 2017, 22:29:52 UTC |
| ab95e29 | Yi Wu | 01 September 2017, 21:08:19 UTC | Fix memory leak on blob db open Summary: Fixes #2820 Closes https://github.com/facebook/rocksdb/pull/2826 Differential Revision: D5757527 Pulled By: yiwu-arbug fbshipit-source-id: f495b63700495aeaade30a1da5e3675848f3d72f | 01 September 2017, 21:13:51 UTC |
| 37ae8cc | Maysam Yabandeh | 01 September 2017, 18:19:47 UTC | Signal progress of the test to avoid timeout Summary: Closes https://github.com/facebook/rocksdb/pull/2824 Differential Revision: D5756457 Pulled By: maysamyabandeh fbshipit-source-id: dff53e945d8ac4ffe6775a2176424fd1a27fc189 | 01 September 2017, 18:28:52 UTC |
| 3b23b1d | Andrew Kryczka | 01 September 2017, 17:58:04 UTC | fix HistogramWindowingImpl copy-{assignment,constructor} Summary: their arguments had a typo. Closes https://github.com/facebook/rocksdb/pull/2821 Differential Revision: D5752408 Pulled By: ajkr fbshipit-source-id: f2d84489c0c615b12a790f04c42d35cc0dccb02d | 01 September 2017, 18:11:56 UTC |
| 3b9a000 | Andrew Kryczka | 01 September 2017, 17:46:26 UTC | fix inclusive-exclusiveness of histogram ToString Summary: I spent too much time thinking about histograms lately and realized boundary values fall into the lower bucket, not the upper bucket. It's because we're using `std::map::lower_bound` here: https://github.com/facebook/rocksdb/blob/867fe92e5e65ce501069aa22c538757acfaade34/monitoring/histogram.cc#L53. Fixed histogram's `ToString()` to reflect this. Closes https://github.com/facebook/rocksdb/pull/2817 Differential Revision: D5751159 Pulled By: ajkr fbshipit-source-id: 67432bb45849eec9b5bcc0d095551dbc0ee81766 | 01 September 2017, 17:56:34 UTC |
| 0ec90a7 | Dmitri Smirnov | 31 August 2017, 23:42:05 UTC | Add -DPORTABLE=1 to MSVC CI build Summary: Add -DPORTABLE=1 port::cacheline_aligned_alloc() has arguments swapped which prevents every single test from running. Closes https://github.com/facebook/rocksdb/pull/2815 Differential Revision: D5751661 Pulled By: siying fbshipit-source-id: e0857d6e138ec46035b3c23d7c3c751901a0a4a0 | 31 August 2017, 23:42:48 UTC |
| 4a90cbf | Andrew Kryczka | 31 August 2017, 23:21:15 UTC | Include statistics pointer in ImmutableDBOptions::Dump Summary: useful when debugging to tell whether a DB has stats enabled, and whether a stats object is shared across DBs. Closes https://github.com/facebook/rocksdb/pull/2813 Differential Revision: D5741755 Pulled By: ajkr fbshipit-source-id: 9b9d51dee77d14d415cd5da985d8d61b5b3837c3 | 31 August 2017, 23:27:04 UTC |
| b97685a | Andrew Kryczka | 31 August 2017, 22:30:35 UTC | fix backup engine when latest backup corrupt Summary: Backup engine is intentionally openable even when some backups are corrupt. Previously the engine could write new backups as long as the most recent backup wasn't corrupt. This PR makes the backup engine able to create new backups even when the most recent one is corrupt. We now maintain two ID instance variables: - `latest_backup_id_` is used when creating backup to choose the new ID - `latest_valid_backup_id_` is used when restoring latest backup since we want most recent valid one Closes https://github.com/facebook/rocksdb/pull/2804 Differential Revision: D5734148 Pulled By: ajkr fbshipit-source-id: db440707b31df2c7b084188aa5f6368449e10bcf | 31 August 2017, 22:41:49 UTC |
| 725bc40 | Yi Wu | 31 August 2017, 22:19:20 UTC | update dependencies.sh Summary: Update dependencies.sh. Also update tbb to 4.3, which is the latest available in TP2. Closes https://github.com/facebook/rocksdb/pull/2812 Differential Revision: D5741394 Pulled By: yiwu-arbug fbshipit-source-id: cafa0b7179f9a44669e5ccace818a02b42336781 | 31 August 2017, 22:26:24 UTC |
| e612e31 | Kamalalochana Subbaiah | 31 August 2017, 21:02:06 UTC | Updated CRC32 Power Optimization Changes Summary: Support for PowerPC Architecture Detecting AltiVec Support Closes https://github.com/facebook/rocksdb/pull/2716 Differential Revision: D5606836 Pulled By: siying fbshipit-source-id: 720262453b1546e5fdbbc668eff56848164113f3 | 31 August 2017, 21:16:30 UTC |
| 8fa4d10 | Dmitri Smirnov | 31 August 2017, 17:18:58 UTC | Try to switch to Stduio 2017 Summary: Closes https://github.com/facebook/rocksdb/pull/2802 Differential Revision: D5746710 Pulled By: siying fbshipit-source-id: daa621ba5fccb84c0d6cdb7755c5e09319c45cb4 | 31 August 2017, 17:30:27 UTC |
| 825a22c | Pengchao Wang | 31 August 2017, 17:10:47 UTC | garbage collect tombstones in merge operator Summary: Remove cassandra tombstone when reaching the max compaction level (full merge). if all columns collected key will be removed in next compaction via compaction filter Closes https://github.com/facebook/rocksdb/pull/2791 Reviewed By: sagar0 Differential Revision: D5722465 Pulled By: wpc fbshipit-source-id: 61e9898a5686551653a16383255aeaab3197e65e | 31 August 2017, 17:11:54 UTC |
| 26ac24f | Maysam Yabandeh | 31 August 2017, 16:27:14 UTC | Add more unit test to write_prepared txns Summary: Closes https://github.com/facebook/rocksdb/pull/2798 Differential Revision: D5724173 Pulled By: maysamyabandeh fbshipit-source-id: fb6b782d933fb4be315b1a231a6a67a66fdc9c96 | 31 August 2017, 16:41:27 UTC |
| 06b37ee | Sagar Vemuri | 31 August 2017, 00:16:48 UTC | Set defaults for high-pri and low-pri thread pools in regression test script Summary: **Summary**: Set defaults for high-pri and low-pri thread pools in regression test script. **Reason for this change**: With #2680 , high-pri and low-pri thread pools get different numbers than before if `num_high_pri_threads` and `num_low_pri_threads` options are not explicitly passed to db_bench in regression test script ... leading to a false-positive regression. **Test Plan**: REMOTE_HOST=udb1671.prn3 TEST_MODE=1 FBSOURCE=~/fbsource ~/fbsource/fbcode/rocks/tools/debug_regression_test.sh viewstate (with very minor changes to the internals). Observe P50 and P99 which showed up as regressions in our graphs. Stats with the commit prior to #2680 , ie. 4f81ab3 : seekrandomwhilewriting : 75.096 micros/op 13316 ops/sec; 168.6 MB/s (7499074 of 7500000 found) Microseconds per seek: Count: 120000000 Average: 1197.7254 StdDev: 33.35 Min: 187 Median: 980.5292 Max: 1816424 Percentiles: **P50: 980.53** P75: 1494.57 **P99: 4185.64** P99.9: 7800.11 P99.99: 15039.64 Stats at #2680, ie. at commit dce6d5a (false-positive regression): seekrandomwhilewriting : 85.330 micros/op 11719 ops/sec; 148.4 MB/s (7499073 of 7500000 found) Microseconds per seek: Count: 120000000 Average: 1362.3261 StdDev: 27.86 Min: 185 Median: 1088.1915 Max: 652760 Percentiles: **P50: 1088.19** P75: 1658.12 **P99: 5361.15** P99.9: 7997.95 P99.99: 11730.07 Stats with the current change on top of dce6d5a : seekrandomwhilewriting : 77.780 micros/op 12856 ops/sec; 162.8 MB/s (7499102 of 7500000 found) Microseconds per seek: Count: 120000000 Average: 1226.6744 StdDev: 17.16 Min: 185 Median: 994.2956 Max: 2553530 Percentiles: **P50: 994.30** P75: 1513.68 **P99: 4284.30** P99.9: 9338.64 P99.99: 23008.86 Closes https://github.com/facebook/rocksdb/pull/2801 Differential Revision: D5742338 Pulled By: sagar0 fbshipit-source-id: cc5d727c1a131f2a7070d1bb892efbe929b976ff | 31 August 2017, 00:29:34 UTC |
| c10cf16 | Andrew Kryczka | 30 August 2017, 23:33:44 UTC | Dump non-final ZSTD compression type support Summary: Closes https://github.com/facebook/rocksdb/pull/2810 Differential Revision: D5739947 Pulled By: ajkr fbshipit-source-id: 09f99718b6b083c2711dcf17f7b68c305f3fd261 | 30 August 2017, 23:41:24 UTC |
| 8a6708f | Artem Danilov | 30 August 2017, 22:20:48 UTC | Extend property map with compaction stats Summary: This branch extends existing property map which keeps values in doubles to keep values in strings so that it can be used to provide wider range of properties. The immediate need for that is to provide IO stall stats in an easy parseable way to MyRocks which is also part of this branch. Closes https://github.com/facebook/rocksdb/pull/2794 Differential Revision: D5717676 Pulled By: Tema fbshipit-source-id: e34ba5b79ba774697f7b97ce1138d8fd55471b8a | 30 August 2017, 22:26:55 UTC |
| dc5f29f | Changli Gao | 30 August 2017, 22:12:21 UTC | EnvWrapper: Forward more functions Summary: Closes https://github.com/facebook/rocksdb/pull/2789 Differential Revision: D5738335 Pulled By: ajkr fbshipit-source-id: f371303c42b144d0a0424e9304b0df545f073ad1 | 30 August 2017, 22:26:50 UTC |
| 266ac24 | Maysam Yabandeh | 30 August 2017, 21:12:03 UTC | Bumping version to 5.8 Summary: Closes https://github.com/facebook/rocksdb/pull/2738 Differential Revision: D5736261 Pulled By: maysamyabandeh fbshipit-source-id: 49d27e9ccd786c4056a3d586a060fe460ea883ac | 30 August 2017, 21:26:12 UTC |
