ac329cc | sdong | 21 February 2018, 23:12:10 UTC | Update HISTORY.md | 21 February 2018, 23:12:10 UTC |
3300ca0 | Sagar Vemuri | 21 February 2018, 03:05:21 UTC | Add rocksdb.iterator.internal-key property Summary: Added a new iterator property: `rocksdb.iterator.internal-key` to get the internal-key (converted to user key) at which the iterator stopped. Closes https://github.com/facebook/rocksdb/pull/3525 Differential Revision: D7033694 Pulled By: sagar0 fbshipit-source-id: d51e6c00f5e9d766c6276ef79774b81c6c5216f8 | 21 February 2018, 18:20:59 UTC |
aa3b8bb | Sagar Vemuri | 16 February 2018, 22:18:14 UTC | Bump version to 5.10.3 | 16 February 2018, 22:18:14 UTC |
0493719 | Mike Kolupaev | 16 February 2018, 15:58:18 UTC | Fix deadlock in ColumnFamilyData::InstallSuperVersion() Summary: Deadlock: a memtable flush holds DB::mutex_ and calls ThreadLocalPtr::Scrape(), which locks ThreadLocalPtr mutex; meanwhile, a thread exit handler locks ThreadLocalPtr mutex and calls SuperVersionUnrefHandle, which tries to lock DB::mutex_. This deadlock is hit all the time on our workload. It blocks our release. In general, the problem is that ThreadLocalPtr takes an arbitrary callback and calls it while holding a lock on a global mutex. The same global mutex is (at least in some cases) locked by almost all ThreadLocalPtr methods, on any instance of ThreadLocalPtr. So, there'll be a deadlock if the callback tries to do anything to any instance of ThreadLocalPtr, or waits for another thread to do so. So, probably the only safe way to use ThreadLocalPtr callbacks is to do only do simple and lock-free things in them. This PR fixes the deadlock by making sure that local_sv_ never holds the last reference to a SuperVersion, and therefore SuperVersionUnrefHandle never has to do any nontrivial cleanup. I also searched for other uses of ThreadLocalPtr to see if they may have similar bugs. There's only one other use, in transaction_lock_mgr.cc, and it looks fine. Closes https://github.com/facebook/rocksdb/pull/3510 Reviewed By: sagar0 Differential Revision: D7005346 Pulled By: al13n321 fbshipit-source-id: 37575591b84f07a891d6659e87e784660fde815f | 16 February 2018, 19:34:46 UTC |
7ad7940 | Siying Dong | 14 February 2018, 00:20:13 UTC | Direct I/O writable file should do fsync in Close() Summary: We don't do fsync() after truncate in direct I/O writeable file (in fact we don't do any fsync ever). This can cause metadata not persistent to disk after the file is generated. We call it instead. Closes https://github.com/facebook/rocksdb/pull/3500 Differential Revision: D6981482 Pulled By: siying fbshipit-source-id: 7e2b591b7e5dd1b96fc0775515b8b9e6092980ef | 14 February 2018, 00:52:07 UTC |
257eb0a | Wouter Beek | 20 December 2017, 16:02:53 UTC | FIXED: string buffers potentially too small to fit formatted write Summary: This fixes the following warnings when compiled with GCC7: util/transaction_test_util.cc: In static member function ‘static rocksdb::Status rocksdb::RandomTransactionInserter::DBGet(rocksdb::DB*, rocksdb::Transaction*, rocksdb::ReadOptions&, uint16_t, uint64_t, bool, uint64_t*, std::__cxx11::string*, bool*)’: util/transaction_test_util.cc:75:8: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=] Status RandomTransactionInserter::DBGet( ^~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc:84:11: note: ‘snprintf’ output between 5 and 6 bytes into a destination of size 5 snprintf(prefix_buf, sizeof(prefix_buf), "%.4u", set_i + 1); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc: In static member function ‘static rocksdb::Status rocksdb::RandomTransactionInserter::Verify(rocksdb::DB*, uint16_t, uint64_t, bool, rocksdb::Random64*)’: util/transaction_test_util.cc:245:8: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=] Status RandomTransactionInserter::Verify(DB* db, uint16_t num_sets, ^~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc:268:13: note: ‘snprintf’ output between 5 and 6 bytes into a destination of size 5 snprintf(prefix_buf, sizeof(prefix_buf), "%.4u", set_i + 1); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Closes https://github.com/facebook/rocksdb/pull/3295 Differential Revision: D6609411 Pulled By: maysamyabandeh fbshipit-source-id: 33f0add471056eb59db2f8bd4366e6dfbb1a187d | 13 February 2018, 21:37:32 UTC |
54c1984 | Jun Wu | 01 February 2018, 22:15:28 UTC | crc32: suppress -Wimplicit-fallthrough warnings Summary: Workaround a bunch of "implicit-fallthrough" compiler errors, like: ``` util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=] crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset)); ^ util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’ CRCsinglet(crc0, next, -2 * 8); ^~~~~~~~~~ util/crc32c.cc:1017:7: note: here case 1: ``` Closes https://github.com/facebook/rocksdb/pull/3339 Reviewed By: sagar0 Differential Revision: D6874736 Pulled By: quark-zju fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16 | 12 February 2018, 20:58:31 UTC |
7ac5446 | Mark Isaacson | 29 January 2018, 20:43:56 UTC | Suppress lint in old files Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues. Reviewed By: yiwu-arbug Differential Revision: D6821806 fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c | 31 January 2018, 01:15:39 UTC |
c1e70e7 | Yi Wu | 30 January 2018, 19:13:10 UTC | Bump version to 5.10.2 | 30 January 2018, 19:13:10 UTC |
d91f420 | Yi Wu | 26 January 2018, 23:12:16 UTC | StackableDB optionally take shared ownership of the underlying DB Summary: Allow StackableDB optionally takes a shared_ptr on construction and thus hold shared ownership of the underlying DB. Closes https://github.com/facebook/rocksdb/pull/3423 Differential Revision: D6824163 Pulled By: yiwu-arbug fbshipit-source-id: dbdc30c42e007533a987ef413785e192340f03eb | 30 January 2018, 19:12:37 UTC |
1454a5c | Sagar Vemuri | 12 January 2018, 18:53:43 UTC | Fix PowerPC dynamic java build Summary: Java build on PPC64le has been broken since a few months, due to #2716. Fixing it with the least amount of changes. (We should cleanup a little around this code when time permits). This should fix the build failures seen in http://140.211.168.68:8080/job/Rocksdb/ . Closes https://github.com/facebook/rocksdb/pull/3359 Differential Revision: D6712938 Pulled By: sagar0 fbshipit-source-id: 3046e8f072180693de2af4762934ec1ace309ca4 | 19 January 2018, 02:02:26 UTC |
d7defe4 | Yi Wu | 19 January 2018, 02:02:03 UTC | Bump version to 5.10.1 | 19 January 2018, 02:02:03 UTC |
e7e0aa7 | Yi Wu | 19 January 2018, 01:32:50 UTC | Fix Flush() keep waiting after flush finish Summary: Flush() call could be waiting indefinitely if min_write_buffer_number_to_merge is used. Consider the sequence: 1. User call Flush() with flush_options.wait = true 2. The manual flush started in the background 3. New memtable become immutable because of writes. The new memtable will not trigger flush if min_write_buffer_number_to_merge is not reached. 4. The manual flush finish. Because of the new memtable created at step 3 not being flush, previous logic of WaitForFlushMemTable() keep waiting, despite the memtables it intent to flush has been flushed. Here instead of checking if there are any more memtables to flush, WaitForFlushMemTable() also check the id of the earliest memtable. If the id is larger than that of latest memtable at the time flush was initiated, it means all the memtable at the time of flush start has all been flush. Closes https://github.com/facebook/rocksdb/pull/3378 Differential Revision: D6746789 Pulled By: yiwu-arbug fbshipit-source-id: 35e698f71c7f90b06337a93e6825f4ea3b619bfa | 19 January 2018, 01:52:02 UTC |
6c1a724 | Adam Retter | 11 January 2018, 21:21:35 UTC | FreeBSD build support for RocksDB and RocksJava Summary: Tested on a clean FreeBSD 11.01 x64. Closes https://github.com/facebook/rocksdb/pull/1423 Closes https://github.com/facebook/rocksdb/pull/3357 Differential Revision: D6705868 Pulled By: sagar0 fbshipit-source-id: cbccbbdafd4f42922512ca03619a5d5583a425fd | 11 January 2018, 21:42:22 UTC |
c43d91e | Andrew Kryczka | 03 January 2018, 20:37:07 UTC | fix powerpc java static build Summary: added support for C and asm files as required for e612e317409e8a9d74cf05db0bd733403305f459. Closes https://github.com/facebook/rocksdb/pull/3299 Differential Revision: D6612479 Pulled By: ajkr fbshipit-source-id: 6263ed7c1602f249460421825c76b5721f396163 | 11 January 2018, 20:41:44 UTC |
cf2c88c | Sagar Vemuri | 05 January 2018, 23:30:12 UTC | Fix zstd/zdict include path for java static build Summary: With the ZSTD dictionary generator support added in #3057 `PORTABLE=1 ROCKSDB_NO_FBCODE=1 make rocksdbjavastatic` fails as it can't find zdict.h. Specifically due to: https://github.com/facebook/rocksdb/blob/e3a06f12d27fd50af7b6c5941973f529601f9a3e/util/compression.h#L39 In java static builds zstd code gets directly downloaded from https://github.com/facebook/zstd , and in there zdict.h is under dictBuilder directory. So, I modified libzstd.a target to use `make install` to collect all the header files into a single location and used that as the zstd's include path. Closes https://github.com/facebook/rocksdb/pull/3260 Differential Revision: D6669850 Pulled By: sagar0 fbshipit-source-id: f8a7562a670e5aed4c4fb6034a921697590d7285 | 11 January 2018, 01:50:53 UTC |
f897be8 | Yi Wu | 09 January 2018, 18:53:35 UTC | Fix db_bench write being disabled in lite build Summary: The macro was added by mistake in #2372 Closes https://github.com/facebook/rocksdb/pull/3343 Differential Revision: D6681356 Pulled By: yiwu-arbug fbshipit-source-id: 4180172fb0eaef4189c07f219241e0c261c03461 | 11 January 2018, 01:50:03 UTC |
8e42c3f | Siying Dong | 10 January 2018, 02:12:32 UTC | Remove GCC parameter "-march=native" for ARM Summary: Most popular versions of GCC can't identify platform on ARM if "-march=native" is specified. Remove it to unblock most people. Closes https://github.com/facebook/rocksdb/pull/3346 Differential Revision: D6690544 Pulled By: siying fbshipit-source-id: bbaba9fe2645b6b37144b36ea75beeff88992b49 | 11 January 2018, 01:49:56 UTC |
0d2818d | Siying Dong | 09 January 2018, 22:46:30 UTC | Fix a wrong log formatting Summary: I experienced weird segfault because of this mismatch of type in log formatting. Fix it. Closes https://github.com/facebook/rocksdb/pull/3345 Differential Revision: D6687224 Pulled By: siying fbshipit-source-id: c51fb1c008b7ebc3efdc353a4adad3e8f5b3e9de | 11 January 2018, 01:49:43 UTC |
f7285ce | sdong | 27 December 2017, 21:19:17 UTC | Fix HISTORY.md | 27 December 2017, 21:19:53 UTC |
76698fe | yingsu00 | 20 December 2017, 02:20:50 UTC | Port 3 way SSE4.2 crc32c implementation from Folly Summary: **# Summary** RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain. This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used. **# Performance Test Results** We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm. Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s. 1) ReadRandom in db_bench overall metrics PER RUN Algorithm | run | micros/op | ops/sec |Throughput (MB/s) 3-way | 1 | 4.143 | 241387 | 26.7 3-way | 2 | 3.775 | 264872 | 29.3 3-way | 3 | 4.116 | 242929 | 26.9 FastCrc32c|1 | 4.037 | 247727 | 27.4 FastCrc32c|2 | 4.648 | 215166 | 23.8 FastCrc32c|3 | 4.352 | 229799 | 25.4 AVG Algorithm | Average of micros/op | Average of ops/sec | Average of Throughput (MB/s) 3-way | 4.01 | 249,729 | 27.63 FastCrc32c | 4.35 | 230,897 | 25.53 2) Crc32c computation CPU cost (inclusive samples percentage) PER RUN Implementation | run | TotalSamples | Crc32c percentage 3-way | 1 | 4,572,250,000 | 4.37% 3-way | 2 | 3,779,250,000 | 4.62% 3-way | 3 | 4,129,500,000 | 4.48% FastCrc32c | 1 | 4,663,500,000 | 11.24% FastCrc32c | 2 | 4,047,500,000 | 12.34% FastCrc32c | 3 | 4,366,750,000 | 11.68% **# Test Plan** make -j64 corruption_test && ./corruption_test By default it uses 3-way SSE algorithm NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test make clean && DEBUG_LEVEL=0 make -j64 db_bench make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench Closes https://github.com/facebook/rocksdb/pull/3173 Differential Revision: D6330882 Pulled By: yingsu00 fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9 | 27 December 2017, 21:16:32 UTC |
9196e80 | Zhongyi Xie | 13 December 2017, 05:06:26 UTC | Reduce heavy hitter for Get operation Summary: This PR addresses the following heavy hitters in `Get` operation by moving calls to `StatisticsImpl::recordTick` from `BlockBasedTable` to `Version::Get` - rocksdb.block.cache.bytes.write - rocksdb.block.cache.add - rocksdb.block.cache.data.miss - rocksdb.block.cache.data.bytes.insert - rocksdb.block.cache.data.add - rocksdb.block.cache.hit - rocksdb.block.cache.data.hit - rocksdb.block.cache.bytes.read The db_bench statistics before and after the change are: |1GB block read|Children |Self |Command |Shared Object |Symbol| |---|---|---|---|---|---| |master: |4.22% |1.31% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| |updated: |0.51% |0.21% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| | |0.14% |0.14% |db_bench |db_bench |[.] rocksdb::GetContext::record_counters| |1MB block read|Children |Self |Command |Shared Object |Symbol| |---|---|---|---|---|---| |master: |3.48% |1.08% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| |updated: |0.80% |0.31% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| | |0.35% |0.35% |db_bench |db_bench |[.] rocksdb::GetContext::record_counters| Closes https://github.com/facebook/rocksdb/pull/3172 Differential Revision: D6330532 Pulled By: miasantreble fbshipit-source-id: 2b492959e00a3db29e9437ecdcc5e48ca4ec5741 | 15 December 2017, 00:52:06 UTC |
243ca14 | Siying Dong | 14 December 2017, 18:18:01 UTC | Print out compression type of new SST files in logging Summary: Closes https://github.com/facebook/rocksdb/pull/3264 Differential Revision: D6552768 Pulled By: siying fbshipit-source-id: 6303110aff22f341d5cff41f8d2d4f138a53652d | 15 December 2017, 00:51:26 UTC |
1c3c1c4 | Siying Dong | 14 December 2017, 18:17:22 UTC | NUMBER_BLOCK_COMPRESSED, etc, shouldn't be treated as timer counter Summary: NUMBER_BLOCK_DECOMPRESSED and NUMBER_BLOCK_COMPRESSED are not reported unless the stats level contain detailed timers, which is wrong. They are normal counters. Fix it. Closes https://github.com/facebook/rocksdb/pull/3263 Differential Revision: D6552519 Pulled By: siying fbshipit-source-id: 40899ccea7b2856bb39752616657c0bfd432f6f9 | 15 December 2017, 00:51:13 UTC |
3e40def | Orvid King | 14 December 2017, 19:57:11 UTC | Fix the build with MSVC 2017 Summary: There were a few places where MSVC's implicit truncation warnings were getting triggered, which was causing the MSVC build to fail due to warnings being treated as errors. This resolves the issues by making the truncations in some places explicit, and by making it so there are no truncations of literals. Fixes #3239 Supersedes #3259 Closes https://github.com/facebook/rocksdb/pull/3273 Reviewed By: yiwu-arbug Differential Revision: D6569204 Pulled By: Orvid fbshipit-source-id: c188cf1cf98d9acb6d94b71875041cc81f8ff088 | 15 December 2017, 00:49:25 UTC |
766f062 | Siying Dong | 11 December 2017, 23:34:51 UTC | Switch version to 5.10 Summary: Closes https://github.com/facebook/rocksdb/pull/3252 Differential Revision: D6539373 Pulled By: siying fbshipit-source-id: ce7c3d3fe625852179055295da9cf7bc80755025 | 11 December 2017, 23:44:51 UTC |
9a27ac5 | Yi Wu | 11 December 2017, 21:46:57 UTC | Fix drop column family data race Summary: A data race is caught by tsan_crash test between compaction and DropColumnFamily: https://gist.github.com/yiwu-arbug/5a2b4baae05eeb99ae1719b650f30a44 Compaction checks if the column family has been dropped on each key input, while user can issue DropColumnFamily which updates cfd->dropped_, causing the data race. Fixing it by making cfd->dropped_ an atomic. Closes https://github.com/facebook/rocksdb/pull/3250 Differential Revision: D6535991 Pulled By: yiwu-arbug fbshipit-source-id: 5571df020beae7fa7db6fff5ad0d598f49962895 | 11 December 2017, 21:57:48 UTC |
fcc8a65 | Zhongyi Xie | 11 December 2017, 21:12:12 UTC | Make Universal compaction options dynamic Summary: Let me know if more test coverage is needed Closes https://github.com/facebook/rocksdb/pull/3213 Differential Revision: D6457165 Pulled By: miasantreble fbshipit-source-id: 3f944abff28aa7775237f1c4f61c64ccbad4eea9 | 11 December 2017, 21:27:06 UTC |
250a51a | Yi Wu | 11 December 2017, 20:01:22 UTC | BlobDB: refactor DB open logic Summary: Refactor BlobDB open logic. List of changes: Major: * On reopen, mark blob files found as immutable, do not use them for writing new keys. * Not to scan the whole file to find file footer. Instead just seek to the end of the file and try to read footer. Minor: * Move most of the real logic from blob_db.cc to blob_db_impl.cc. * Not to hold shared_ptr of event listeners in global maps in blob_db.cc * Some changes to BlobFile interface. * Improve logging and error handling. Closes https://github.com/facebook/rocksdb/pull/3246 Differential Revision: D6526147 Pulled By: yiwu-arbug fbshipit-source-id: 9dc4cdd63359a2f9b696af817086949da8d06952 | 11 December 2017, 20:12:38 UTC |
6a183d1 | Prashant D | 11 December 2017, 19:48:47 UTC | Fix coverity issues compaction_job, compaction_picker Summary: db/compaction_job.cc: ReportStartedCompaction(compaction); CID 1419863 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member bottommost_level_ is not initialized in this constructor nor in any functions that it calls. db/compaction_picker_universal.cc: 7struct InputFileInfo { 2. uninit_member: Non-static class member level is not initialized in this constructor nor in any functions that it calls. CID 1405355 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member index is not initialized in this constructor nor in any functions that it calls. 38 InputFileInfo() : f(nullptr) {} db/dbformat.h: ParsedInternalKey() 84 : sequence(kMaxSequenceNumber) // Make code analyzer happy CID 1168095 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member type is not initialized in this constructor nor in any functions that it calls. 85 {} // Intentionally left uninitialized (for speed) Closes https://github.com/facebook/rocksdb/pull/3091 Differential Revision: D6534558 Pulled By: yiwu-arbug fbshipit-source-id: 5ada975956196d267b3f149386842af71eda7553 | 11 December 2017, 19:57:15 UTC |
e3814a8 | Andrew Kryczka | 08 December 2017, 00:09:25 UTC | revert fbcode build behavior Summary: Closes https://github.com/facebook/rocksdb/pull/3242 Differential Revision: D6514255 Pulled By: ajkr fbshipit-source-id: c39fa8e745866b052649d02bf339e794d77e96a3 | 08 December 2017, 00:12:52 UTC |
fe608e3 | Dmitri Smirnov | 07 December 2017, 21:41:35 UTC | Fix a race condition in WindowsThread (port::Thread) Summary: Fix a race condition when we create a thread and immediately destroy This case should be supported. What happens is that the thread function needs the Data instance to actually run but has no shared ownership and must rely on the WindowsThread instance to continue existing. To address this we change unique_ptr to shared_ptr and then acquire an additional refcount for the threadproc which destroys it just before the thread exit. We choose to allocate shared_ptr instance on the heap as this allows the original thread to continue w/o waiting for the new thread to start running. Closes https://github.com/facebook/rocksdb/pull/3240 Differential Revision: D6511324 Pulled By: yiwu-arbug fbshipit-source-id: 4633ff7996daf4d287a9fe34f60c1dd28cf4ff36 | 07 December 2017, 21:42:53 UTC |
34aa245 | Prashant D | 07 December 2017, 19:52:12 UTC | Fix coverity issues version, write_batch Summary: db/version_builder.cc: 117 base_vstorage_->InternalComparator(); CID 1351713 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member field level_zero_cmp_.internal_comparator is not initialized in this constructor nor in any functions that it calls. db/version_edit.h: 145 FdWithKeyRange() 146 : fd(), 147 smallest_key(), 148 largest_key() { CID 1418254 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member file_metadata is not initialized in this constructor nor in any functions that it calls. 149 } db/version_set.cc: 120 } CID 1322789 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 4. uninit_member: Non-static class member curr_file_level_ is not initialized in this constructor nor in any functions that it calls. 121 } db/write_batch.cc: 939 assert(cf_mems_); CID 1419862 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 3. uninit_member: Non-static class member rebuilding_trx_seq_ is not initialized in this constructor nor in any functions that it calls. 940 } Closes https://github.com/facebook/rocksdb/pull/3092 Differential Revision: D6505666 Pulled By: yiwu-arbug fbshipit-source-id: fd2c68948a0280772691a419d72ac7e190951d86 | 07 December 2017, 19:57:36 UTC |
baff91c | Prashant D | 07 December 2017, 19:50:49 UTC | table: Fix coverity issues Summary: table/block.cc: 420 } CID 1396127 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 7. uninit_member: Non-static class member restart_offset_ is not initialized in this constructor nor in any functions that it calls. 421} table/block_based_table_builder.cc: CID 1418259 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 7. uninit_member: Non-static class member compressed_cache_key_prefix_size is not initialized in this constructor nor in any functions that it calls. table/block_based_table_reader.h: 3. uninit_member: Non-static class member index_type is not initialized in this constructor nor in any functions that it calls. CID 1396147 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 5. uninit_member: Non-static class member hash_index_allow_collision is not initialized in this constructor nor in any functions that it calls. 413 global_seqno(kDisableGlobalSequenceNumber) {} 414 table/cuckoo_table_reader.cc: 55 if (hash_funs == user_props.end()) { 56 status_ = Status::Corruption("Number of hash functions not found"); 5. uninit_member: Non-static class member is_last_level_ is not initialized in this constructor nor in any functions that it calls. 7. uninit_member: Non-static class member identity_as_first_hash_ is not initialized in this constructor nor in any functions that it calls. 9. uninit_member: Non-static class member use_module_hash_ is not initialized in this constructor nor in any functions that it calls. 11. uninit_member: Non-static class member num_hash_func_ is not initialized in this constructor nor in any functions that it calls. 13. uninit_member: Non-static class member key_length_ is not initialized in this constructor nor in any functions that it calls. 15. uninit_member: Non-static class member user_key_length_ is not initialized in this constructor nor in any functions that it calls. 17. uninit_member: Non-static class member value_length_ is not initialized in this constructor nor in any functions that it calls. 19. uninit_member: Non-static class member bucket_length_ is not initialized in this constructor nor in any functions that it calls. 21. uninit_member: Non-static class member cuckoo_block_size_ is not initialized in this constructor nor in any functions that it calls. 23. uninit_member: Non-static class member cuckoo_block_bytes_minus_one_ is not initialized in this constructor nor in any functions that it calls. CID 1322785 (#2 of 2): Uninitialized scalar field (UNINIT_CTOR) 25. uninit_member: Non-static class member table_size_ is not initialized in this constructor nor in any functions that it calls. 57 return; table/plain_table_index.h: 2. uninit_member: Non-static class member index_size_ is not initialized in this constructor nor in any functions that it calls. CID 1322801 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member sub_index_size_ is not initialized in this constructor nor in any functions that it calls. 128 huge_page_tlb_size_(huge_page_tlb_size) {} 129 Closes https://github.com/facebook/rocksdb/pull/3113 Differential Revision: D6505719 Pulled By: yiwu-arbug fbshipit-source-id: 38f44d8f9dfefb4c2e25d83b8df25a5201c75618 | 07 December 2017, 19:57:36 UTC |
2e3a009 | Andrew Kryczka | 07 December 2017, 19:07:44 UTC | fix ASAN for DeleteFilesInRange test case Summary: error message was ``` ==3095==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffd18216c40 at pc 0x0000005edda1 bp 0x7ffd18215550 sp 0x7ffd18214d00 ... Address 0x7ffd18216c40 is located in stack of thread T0 at offset 1952 in frame #0 internal_repo_rocksdb/db_compaction_test.cc:1520 rocksdb::DBCompactionTest_DeleteFileRangeFileEndpointsOverlapBug_Test::TestBody() ``` It was unsafe to have slices referring to the temporary string objects' buffers, as those strings were destroyed before the slices were used. Fixed it by assigning the strings returned by `Key()` to local variables. Closes https://github.com/facebook/rocksdb/pull/3238 Differential Revision: D6507864 Pulled By: ajkr fbshipit-source-id: dd07de1a0070c6748c1ab4f3d7bd31f9a81889d0 | 07 December 2017, 19:12:43 UTC |
e1c569c | Yi Wu | 07 December 2017, 06:54:11 UTC | Fix clang-analyzer false-positive on ldb_cmd.cc Summary: clang-analyzer complaint about db_ being nullptr, but it couldn't be because it checks exec_stats before proceed. Add an assert to get around the false-positive. Test Plan `make analyze` Closes https://github.com/facebook/rocksdb/pull/3236 Differential Revision: D6505417 Pulled By: yiwu-arbug fbshipit-source-id: e5b65764ea994dd9e4bab3e697b97dc70dc22cab | 07 December 2017, 06:58:46 UTC |
bbef8c3 | Sagar Vemuri | 07 December 2017, 04:43:52 UTC | Log GetCurrentTime failures during Flush and Compaction Summary: `GetCurrentTime()` is used to populate `creation_time` table property during flushes and compactions. It is safe to ignore `GetCurrentTime()` failures here but they should be logged. (Note that `creation_time` property was introduced as part of TTL-based FIFO compaction in #2480.) Tes Plan: `make check` Closes https://github.com/facebook/rocksdb/pull/3231 Differential Revision: D6501935 Pulled By: sagar0 fbshipit-source-id: 376adcf4ab801d3a43ec4453894b9a10909c8eb6 | 07 December 2017, 04:56:53 UTC |
d51fcb2 | Sagar Vemuri | 07 December 2017, 04:40:46 UTC | Blob DB: Add db_bench options Summary: Adding more BlobDB db_bench options which are needed for benchmarking. Closes https://github.com/facebook/rocksdb/pull/3230 Differential Revision: D6500711 Pulled By: sagar0 fbshipit-source-id: 91d63122905854ef7c9148a0235568719146e6c5 | 07 December 2017, 04:44:12 UTC |
53a516a | Andrew Kryczka | 07 December 2017, 03:46:44 UTC | update history for recent commits Summary: I browsed through the history since 5.9 was released and found some changes worth mentioning in HISTORY.md. Closes https://github.com/facebook/rocksdb/pull/3237 Differential Revision: D6506472 Pulled By: ajkr fbshipit-source-id: 627ce9f94ca33df9f0f231a9c5ced3624b05506c | 07 December 2017, 03:56:17 UTC |
78d1a5e | Andrew Kryczka | 07 December 2017, 02:49:26 UTC | Preserve overlapping file endpoint invariant Summary: Fix for #2833. - In `DeleteFilesInRange`, use `GetCleanInputsWithinInterval` instead of `GetOverlappingInputs` to make sure we get a clean cut set of files to delete. - In `GetCleanInputsWithinInterval`, support nullptr as `begin_key` or `end_key`. - In `GetOverlappingInputsRangeBinarySearch`, move the assertion for non-empty range away from `ExtendFileRangeWithinInterval`, which should be allowed to return an empty range (via `end_index < begin_index`). Closes https://github.com/facebook/rocksdb/pull/2843 Differential Revision: D5772387 Pulled By: ajkr fbshipit-source-id: e554e8461823c6be82b21a9262a2da02b3957881 | 07 December 2017, 02:56:54 UTC |
a7d3277 | Yi Wu | 07 December 2017, 01:04:32 UTC | Fix write_callback_test compile error Summary: Rename shadow variable name db_impl. Fixing #3227 Closes https://github.com/facebook/rocksdb/pull/3235 Differential Revision: D6504051 Pulled By: yiwu-arbug fbshipit-source-id: 186c9378dabb11f8d6db56f45c95cc3b029fcb88 | 07 December 2017, 01:12:27 UTC |
7f04af3 | Yi Wu | 07 December 2017, 00:38:49 UTC | ldb to allow db with --try_load_options and without an options file Summary: This is to fix tools/check_format_compatible.sh. The tool try to open old versions of rocksdb with the provided options file. When options file is missing (e.g. rocksdb 2.2), it should still proceed with default options. Closes https://github.com/facebook/rocksdb/pull/3232 Differential Revision: D6503955 Pulled By: yiwu-arbug fbshipit-source-id: e44cfcce7ddc7d12cf83466ed3f3fe7624aa78b8 | 07 December 2017, 00:42:26 UTC |
b5798bd | Yi Wu | 06 December 2017, 23:56:37 UTC | Add missing recent versions to format compatible test Summary: Add recent versions for format compatible test. We should probably update the script to auto include available versions (by looking at include/rocksdb/versions.h and deduce branch names), but we can do it later. Closes https://github.com/facebook/rocksdb/pull/3233 Differential Revision: D6503631 Pulled By: yiwu-arbug fbshipit-source-id: e2b01d1ef6e784ff6ffa1bd75d741755e3c69a8c | 07 December 2017, 00:13:50 UTC |
20995c5 | Yi Wu | 06 December 2017, 19:54:45 UTC | Make iterator invalid on Merge error Summary: Since #1665, on merge error, iterator will be set to corrupted status, but it doesn't invalidate the iterator. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3226 Differential Revision: D6499094 Pulled By: yiwu-arbug fbshipit-source-id: 80222930f949e31f90a6feaa37ddc3529b510d2c | 06 December 2017, 19:56:39 UTC |
36911f5 | Maysam Yabandeh | 06 December 2017, 17:32:09 UTC | WritePrepared Txn: stress test Summary: Augment the existing MySQLStyleTransactionTest to check for more core case scenarios. The changes showed effective in revealing the bugs reported in https://github.com/facebook/rocksdb/pull/3205 and https://github.com/facebook/rocksdb/pull/3101 Closes https://github.com/facebook/rocksdb/pull/3222 Differential Revision: D6476862 Pulled By: maysamyabandeh fbshipit-source-id: 5068497702d67ffc206a58ed96f8578fbb510137 | 06 December 2017, 17:42:28 UTC |
1b0c58d | Flarnie Marchan | 06 December 2017, 02:41:08 UTC | Add Code of Conduct Summary: In the past Facebook didn't promote including a Code of Conduct when creating new projects, and many projects skipped this important document. Let's fix it. :) **why make this change?:** Facebook Open Source provides a Code of Conduct statement for all projects to follow, to promote a welcoming and safe open source community. Exposing the COC via a separate markdown file is a standard being promoted by Github via the Community Profile in order to meet their Open Source Guide's recommended community standards. As you can see, adding this file will improve [the rocksdb community profile](https://github.com/facebook/rocksdb/community) checklist and increase the visibility of our COC. **test plan:** Viewing it on my branch - <img width="1008" alt="screen shot 2017-12-03 at 5 05 45 pm" src="https://user-images.githubusercontent.com/1114467/33532198-66012a56-d84c-11e7-8fab-29ed410bd600.png"> <img width="1015" alt="screen shot 2017-12-03 at 5 05 59 pm" src="https://user-images.githubusercontent.com/1114467/33532199-661813d8-d84c-11e7-941e-94754dd481e5.png"> **issue:** internal task t23481323 Closes https://github.com/facebook/rocksdb/pull/3219 Reviewed By: yiwu-arbug Differential Revision: D6494234 Pulled By: flarnie fbshipit-source-id: 55b59db335cc5546f3a1c968322b9281a3dc3aaf | 06 December 2017, 02:42:35 UTC |
a37d734 | Adam Novak | 05 December 2017, 23:05:41 UTC | Add ROCKSDB_DISABLE_* environment variables Summary: Should fix #3036. Closes https://github.com/facebook/rocksdb/pull/3042 Differential Revision: D6452921 Pulled By: sagar0 fbshipit-source-id: eaf11e43fee1f8747006530cfc0c7a358f1c2f0f | 05 December 2017, 23:12:46 UTC |
4634c73 | Alex Robinson | 04 December 2017, 09:53:49 UTC | Update DBOptions::IncreaseParallelism to use newer background settings Summary: The Options header file recommends using max_background_jobs rather than directly setting max_background_compactions or max_background_flushes. I've personally seen a performance problem where stalls were happening because the one background flushing thread was blocked that was fixed by this change - https://github.com/cockroachdb/cockroach/issues/19699#issuecomment-347672485 Closes https://github.com/facebook/rocksdb/pull/3208 Differential Revision: D6473178 Pulled By: ajkr fbshipit-source-id: 67c892ceb7b1909d251492640cb15a0f2262b7ed | 04 December 2017, 09:56:15 UTC |
57056bb | Andrew Kryczka | 02 December 2017, 02:26:15 UTC | gflags in cmake on linux Summary: We should use it if available otherwise the tools builds never work. Thanks to #3212, we can set -DGFLAGS=1 and it'll be independent of the namespace with which gflags was compiled. Closes https://github.com/facebook/rocksdb/pull/3214 Differential Revision: D6462214 Pulled By: ajkr fbshipit-source-id: db4e5f1b905322e3119554a9d01b57532c499384 | 02 December 2017, 02:28:24 UTC |
63f1c0a | Andrew Kryczka | 01 December 2017, 18:40:45 UTC | fix gflags namespace Summary: I started adding gflags support for cmake on linux and got frustrated that I'd need to duplicate the build_detect_platform logic, which determines namespace based on attempting compilation. We can do it differently -- use the GFLAGS_NAMESPACE macro if available, and if not, that indicates it's an old gflags version without configurable namespace so we can simply hardcode "google". Closes https://github.com/facebook/rocksdb/pull/3212 Differential Revision: D6456973 Pulled By: ajkr fbshipit-source-id: 3e6d5bde3ca00d4496a120a7caf4687399f5d656 | 01 December 2017, 18:42:05 UTC |
18dcf7f | Maysam Yabandeh | 01 December 2017, 07:39:56 UTC | WritePrepared Txn: PreReleaseCallback Summary: Add PreReleaseCallback to be called at the end of WriteImpl but before publishing the sequence number. The callback is used in WritePrepareTxn to i) update the commit map, ii) update the last published sequence number in the 2nd write queue. It also ensures that all the commits will go to the 2nd queue. These changes will ensure that the commit map is updated before the sequence number is published and used by reading snapshots. If we use two write queues, the snapshots will use the seq number published by the 2nd queue. If we use one write queue (the default, the snapshots will use the last seq number in the memtable, which also indicates the last published seq number. Closes https://github.com/facebook/rocksdb/pull/3205 Differential Revision: D6438959 Pulled By: maysamyabandeh fbshipit-source-id: f8b6c434e94bc5f5ab9cb696879d4c23e2577ab9 | 01 December 2017, 07:50:45 UTC |
3e40a5e | Nikhil Benesch | 01 December 2017, 06:37:19 UTC | add missing config checks to CMakeLists.txt Summary: Bring CMakeLists.txt back up to parity with build_detect_platform. Closes https://github.com/facebook/rocksdb/pull/3211 Differential Revision: D6452908 Pulled By: ajkr fbshipit-source-id: 93f5f336ad7eff6ecf65dec47bfaf114dd24cfb2 | 01 December 2017, 06:57:00 UTC |
fad1405 | Andres Suarez | 30 November 2017, 23:22:41 UTC | Remove `import` use from TARGETS Summary: We're moving away from `import`. The equivalent internal construct that gets the directory from `fbcode/` is `package_name()`. This is a Skylark friendly wrapper around [`get_base_path`]. The additional whitespace change is from running `python ./buckifier/buckify_rocksdb.py`. [`get_base_path`]: https://buckbuild.com/function/get_base_path.html Closes https://github.com/facebook/rocksdb/pull/3210 Reviewed By: yiwu-arbug Differential Revision: D6451242 Pulled By: zertosh fbshipit-source-id: 445757261de0ec89d5d332c1ba9af097086326dc | 30 November 2017, 23:27:34 UTC |
54095d3 | Yi Wu | 30 November 2017, 18:40:39 UTC | TARGETS file not include tests in opt mode Summary: Do not build the tests in opt mode, since SyncPoint and other test code will not be included. Closes https://github.com/facebook/rocksdb/pull/3204 Differential Revision: D6431154 Pulled By: yiwu-arbug fbshipit-source-id: c404ef042c1a6f679e5c1dc57600b3d8cb52fc28 | 30 November 2017, 18:56:58 UTC |
ffacaaa | zhangjinpeng1987 | 30 November 2017, 06:48:55 UTC | fix Seek with lower_bound Summary: When Seek a key less than `lower_bound`, should return `lower_bound`. ajkr PTAL Closes https://github.com/facebook/rocksdb/pull/3199 Differential Revision: D6421126 Pulled By: ajkr fbshipit-source-id: a06c825830573e0040630704f6bcb3f7f48626f7 | 30 November 2017, 06:56:29 UTC |
ed3af9e | Andrew Kryczka | 29 November 2017, 01:20:47 UTC | improve ldb CLI option support Summary: - Made CLI arguments take precedence over options file when both are provided. Note some of the CLI args are not settable via options file, like `--compression_max_dict_bytes`, so it's necessary to allow both ways of providing options simultaneously. - Changed `PrepareOptionsForOpenDB` to update the proper `ColumnFamilyOptions` if one exists for the user's `--column_family_name` argument. I supported this only in the base class, `LDBCommand`, so it works for the general arguments. Will defer adding support for subcommand-specific arguments. - Made the command fail if `--try_load_options` is provided and loading options file returns NotFound. I found the previous behavior of silently continuing confusing. Closes https://github.com/facebook/rocksdb/pull/3144 Differential Revision: D6270544 Pulled By: ajkr fbshipit-source-id: 7c2eac9f9b38720523d74466fb9e78db53561367 | 29 November 2017, 01:28:58 UTC |
c85f8cc | Andrew Kryczka | 29 November 2017, 01:20:21 UTC | convert null terminator in ascii dump Summary: The ASCII output is almost always useless to me as the first '\0' byte in the key or value causes it to stop printing. Since all characters are already surrounded by spaces, "\ 0" (how we display a backslash followed by a zero) and "\0" (how this PR displays a null terminator) are distinguishable. My assumption is the value of seeing all the bytes outweighs the value of the alignment we had before, where we always had one character followed by one space. Closes https://github.com/facebook/rocksdb/pull/3203 Differential Revision: D6428651 Pulled By: ajkr fbshipit-source-id: aafc978a51e9ea029cfe3e763e2bb0e1751b9ccf | 29 November 2017, 01:28:58 UTC |
c1ed005 | Prashant D | 28 November 2017, 23:15:16 UTC | tools: Fix coverity issues Summary: tools/ldb_cmd.cc: ``` 310 ignore_unknown_options_ = IsFlagPresent(flags, ARG_IGNORE_UNKNOWN_OPTIONS); CID 1322798 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 5. uninit_member: Non-static class member db_ttl_ is not initialized in this constructor nor in any functions that it calls. 311} ``` Closes https://github.com/facebook/rocksdb/pull/3122 Differential Revision: D6428576 Pulled By: sagar0 fbshipit-source-id: d77f04dd201f7f1d9f59ef88a215ee7ad7b934e9 | 28 November 2017, 23:27:41 UTC |
81cf262 | Prashant D | 28 November 2017, 22:31:46 UTC | utilities/backupable : Fix coverity issues Summary: 1. Class BackupMeta ``` 52 : timestamp_(0), size_(0), meta_filename_(meta_filename), CID 1168103 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member sequence_number_ is not initialized in this constructor nor in any functions that it calls. 153 file_infos_(file_infos), env_(env) {} ``` 2. class BackupEngineImpl ``` 513 } 7. uninit_member: Non-static class member latest_backup_id_ is not initialized in this constructor nor in any functions that it calls. CID 1322803 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 9. uninit_member: Non-static class member latest_valid_backup_id_ is not initialized in this constructor nor in any functions that it calls. 514} ``` 3. struct BackupAfterCopyOrCreateWorkItem ``` 368 struct BackupAfterCopyOrCreateWorkItem { 369 std::future<CopyOrCreateResult> result; 1. member_decl: Class member declaration for shared. 370 bool shared; 3. member_decl: Class member declaration for needed_to_copy. 371 bool needed_to_copy; 5. member_decl: Class member declaration for backup_env. 372 Env* backup_env; 373 std::string dst_path_tmp; 374 std::string dst_path; 375 std::string dst_relative; 2. uninit_member: Non-static class member shared is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member needed_to_copy is not initialized in this constructor nor in any functions that it calls. CID 1396122 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 6. uninit_member: Non-static class member backup_env is not initialized in this constructor nor in any functions that it calls. 376 BackupAfterCopyOrCreateWorkItem() {} ``` 4. struct CopyOrCreateWorkItem ``` 318 struct CopyOrCreateWorkItem { 319 std::string src_path; 320 std::string dst_path; 321 std::string contents; 1. member_decl: Class member declaration for src_env. 322 Env* src_env; 3. member_decl: Class member declaration for dst_env. 323 Env* dst_env; 5. member_decl: Class member declaration for sync. 324 bool sync; 7. member_decl: Class member declaration for rate_limiter. 325 RateLimiter* rate_limiter; 9. member_decl: Class member declaration for size_limit. 326 uint64_t size_limit; 327 std::promise<CopyOrCreateResult> result; 328 std::function<void()> progress_callback; 329 2. uninit_member: Non-static class member src_env is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member dst_env is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member sync is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member rate_limiter is not initialized in this constructor nor in any functions that it calls. CID 1396123 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 10. uninit_member: Non-static class member size_limit is not initialized in this constructor nor in any functions that it calls. 330 CopyOrCreateWorkItem() {} ``` 5. struct RestoreAfterCopyOrCreateWorkItem ``` struct RestoreAfterCopyOrCreateWorkItem { 410 std::future<CopyOrCreateResult> result; 1. member_decl: Class member declaration for checksum_value. 411 uint32_t checksum_value; CID 1396153 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member checksum_value is not initialized in this constructor nor in any functions that it calls. 412 RestoreAfterCopyOrCreateWorkItem() {} ``` Closes https://github.com/facebook/rocksdb/pull/3131 Differential Revision: D6428556 Pulled By: sagar0 fbshipit-source-id: a86675444543eff028e3cae6942197a143a112c4 | 28 November 2017, 22:43:28 UTC |
75d57a5 | kapitan-k | 28 November 2017, 22:09:17 UTC | C API: Add some block based table options Summary: Closes https://github.com/facebook/rocksdb/pull/3159 Differential Revision: D6428220 Pulled By: sagar0 fbshipit-source-id: 60508d09b5281f54b907a1c40e9631fc08343131 | 28 November 2017, 22:12:44 UTC |
b45fbc1 | Prashant D | 28 November 2017, 21:15:20 UTC | utilities: Fix coverity issues Summary: ``` utilities/persistent_cache/block_cache_tier_file.cc 64struct CacheRecordHeader { 2. uninit_member: Non-static class member magic_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member crc_ is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member key_size_ is not initialized in this constructor nor in any functions that it calls. CID 1396161 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member val_size_ is not initialized in this constructor nor in any functions that it calls. 65 CacheRecordHeader() {} 66 CacheRecordHeader(const uint32_t magic, const uint32_t key_size, 67 const uint32_t val_size) 68 : magic_(magic), crc_(0), key_size_(key_size), val_size_(val_size) {} 69 1. member_decl: Class member declaration for magic_. 70 uint32_t magic_; 3. member_decl: Class member declaration for crc_. 71 uint32_t crc_; 5. member_decl: Class member declaration for key_size_. 72 uint32_t key_size_; 7. member_decl: Class member declaration for val_size_. 73 uint32_t val_size_; 74}; utilities/simulator_cache/sim_cache.cc: 157 miss_times_(0), CID 1396124 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member stats_ is not initialized in this constructor nor in any functions that it calls. 158 hit_times_(0) {} 159 ``` Closes https://github.com/facebook/rocksdb/pull/3155 Differential Revision: D6427237 Pulled By: sagar0 fbshipit-source-id: 97e493da5fc043c5b9a3e0d33103442cffb75aad | 28 November 2017, 21:27:08 UTC |
7b57510 | Prashant D | 28 November 2017, 20:17:11 UTC | utilities: Fix coverity issues in blob_db and col_buf_decoder Summary: utilities/blob_db/blob_db_impl.cc 265 : bdb_options_.blob_dir; 3. uninit_member: Non-static class member env_ is not initialized in this constructor nor in any functions that it calls. 5. uninit_member: Non-static class member ttl_extractor_ is not initialized in this constructor nor in any functions that it calls. 7. uninit_member: Non-static class member open_p1_done_ is not initialized in this constructor nor in any functions that it calls. CID 1418245 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 9. uninit_member: Non-static class member debug_level_ is not initialized in this constructor nor in any functions that it calls. 266} 4. past_the_end: Function end creates an iterator. CID 1418258 (#1 of 1): Using invalid iterator (INVALIDATE_ITERATOR) 5. deref_iterator: Dereferencing iterator file_nums.end() though it is already past the end of its container. utilities/col_buf_decoder.h: nullable_(nullable), 2. uninit_member: Non-static class member remain_runs_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member run_val_ is not initialized in this constructor nor in any functions that it calls. CID 1396134 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 6. uninit_member: Non-static class member last_val_ is not initialized in this constructor nor in any functions that it calls. 46 big_endian_(big_endian) {} Closes https://github.com/facebook/rocksdb/pull/3134 Differential Revision: D6340607 Pulled By: sagar0 fbshipit-source-id: 25c52566e2ff979fe6c7abb0f40c27fc16597054 | 28 November 2017, 20:27:57 UTC |
cf0d6aa | Tamir Duberstein | 28 November 2017, 20:13:20 UTC | CMake cross platform Java support and add JNI to travis Summary: Rewrite `java/CMakeLists.txt` to take advantage of CMake's cross platform Java support. adamretter Closes https://github.com/facebook/rocksdb/pull/2301 Differential Revision: D5070724 Pulled By: sagar0 fbshipit-source-id: 999aee9bd39da2b24a5fe493a2eb0e9af6072dc7 | 28 November 2017, 20:27:53 UTC |
7827935 | Yi Wu | 28 November 2017, 19:42:28 UTC | Blob DB: Add statistics Summary: Adding a list of blob db counters. Also remove WaStats() which doesn't expose the stats and can be substitute by (BLOB_DB_BYTES_WRITTEN / BLOB_DB_BLOB_FILE_BYTES_WRITTEN). Closes https://github.com/facebook/rocksdb/pull/3193 Differential Revision: D6394216 Pulled By: yiwu-arbug fbshipit-source-id: 017508c8ff3fcd7ea7403c64d0f9834b24816803 | 28 November 2017, 19:58:49 UTC |
3cf562b | Yi Wu | 28 November 2017, 19:40:40 UTC | Fix IOError on WAL write doesn't propagate to write group follower Summary: This is a simpler version of #3097 by removing all unrelated changes. Fixing the bug where concurrent writes may get Status::OK while it actually gets IOError on WAL write. This happens when multiple writes form a write batch group, and the leader get an IOError while writing to WAL. The leader failed to pass the error to followers in the group, and the followers end up returning Status::OK() while actually writing nothing. The bug only affect writes in a batch group. Future writes after the batch group will correctly return immediately with the IOError. Closes https://github.com/facebook/rocksdb/pull/3201 Differential Revision: D6421644 Pulled By: yiwu-arbug fbshipit-source-id: 1c2a455c5b73f6842423785eb8a9dbfbb191dc0e | 28 November 2017, 19:42:48 UTC |
1bdb44d | Andrew Kryczka | 28 November 2017, 19:18:42 UTC | optimize file ingestion checks for range deletion overlap Summary: Before we were checking every file in the level which was unnecessary. We can piggyback onto the code for checking point-key overlap, which already opens all the files that could possibly contain overlapping range deletions. This PR makes us check just the range deletions from those files, so no extra ones will be opened. Closes https://github.com/facebook/rocksdb/pull/3179 Differential Revision: D6358125 Pulled By: ajkr fbshipit-source-id: 00e200770fdb8f3cc6b1b2da232b755e4ba36279 | 28 November 2017, 19:27:02 UTC |
022c598 | Jeffrey Dallatezza | 28 November 2017, 19:15:34 UTC | Fix minor typo in comment Summary: mean -> meant Closes https://github.com/facebook/rocksdb/pull/3202 Differential Revision: D6426443 Pulled By: sagar0 fbshipit-source-id: adaf07218580ee6903986fa5686de92f43f420e1 | 28 November 2017, 19:27:02 UTC |
4b65cfc | Phani Shekhar Mantripragada | 28 November 2017, 18:35:17 UTC | Support for block_cache num_shards and other config via option string. Summary: Problem: Option string accepts only cache_size as parameter for block_cache which is specified as "block_cache=1M". It doesn't accept other parameters like num_shards etc. Changes : 1) ParseBlockBasedTableOption in block_based_table_factory is edited to accept cache options in the format "block_cache=<cache_size>:<num_shard_bits>:<strict_capacity_limit>:<high_pri_pool_ratio>". Options other than cache_size are optional to maintain backward compatibility. The changes are valid for block_cache_compressed as well. For example, "block_cache=1M:6:true:0.5", "block_cache=1M:6:true", "block_cache=1M:6" and "block_cache=1M" are all valid option strings. 2) Corresponding unit tests are added. Closes https://github.com/facebook/rocksdb/pull/3108 Differential Revision: D6420997 Pulled By: sagar0 fbshipit-source-id: cdea8b785688d2802907974af27225ccc1c0cd43 | 28 November 2017, 18:48:53 UTC |
2f09524 | Griffin Smith | 28 November 2017, 18:24:23 UTC | Expose all remaining read and write options via the C API Summary: Expose read and write options via the C API Closes https://github.com/facebook/rocksdb/pull/3185 Differential Revision: D6389658 Pulled By: sagar0 fbshipit-source-id: 1848912750329a476805b3cb2f315e7b71f61472 | 28 November 2017, 18:28:46 UTC |
b72b3c6 | Maysam Yabandeh | 27 November 2017, 16:49:30 UTC | WritePrepared Txn: Add MultiGet to DB Summary: This patch implements MultiGet API for WritePreparedTxnDB and update the existing unit tests. Closes https://github.com/facebook/rocksdb/pull/3196 Differential Revision: D6401493 Pulled By: maysamyabandeh fbshipit-source-id: 51501a1e32645fc2da8680e77a50035f6530f2cc | 27 November 2017, 16:56:21 UTC |
f0dde49 | Yi Wu | 24 November 2017, 19:50:42 UTC | Blob DB: Fix GC handling for inlined blob Summary: Garbage collection checks if the offset in blob index matches the offset of the blob value in the file. If it is a mismatch, the value is the current version. However it failed to check if the blob index is an inlined type, which don't even have an offset. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3194 Differential Revision: D6394270 Pulled By: yiwu-arbug fbshipit-source-id: 7c2b9d795f1116f55f4d728086980f9b6e88ea78 | 24 November 2017, 19:56:47 UTC |
e59cb2a | Maysam Yabandeh | 22 November 2017, 21:52:56 UTC | Add seq_per_batch to WriteWithCallbackTest Summary: Augment WriteWithCallbackTest to also test when seq_per_batch is true. Closes https://github.com/facebook/rocksdb/pull/3195 Differential Revision: D6398143 Pulled By: maysamyabandeh fbshipit-source-id: 7bc4218609355ec20fed25df426a8455ec2390d3 | 22 November 2017, 21:56:44 UTC |
5fac472 | Zhongyi Xie | 22 November 2017, 04:06:55 UTC | make compaction_readahead_size_ thread safe Summary: this should fix the failing tsan_check Closes https://github.com/facebook/rocksdb/pull/3192 Differential Revision: D6390004 Pulled By: miasantreble fbshipit-source-id: 6cadfc6f68febb1a77b0abcdb5416570dad926a5 | 22 November 2017, 04:11:38 UTC |
fe187e7 | Griffin Smith | 21 November 2017, 18:11:26 UTC | Add Elixir to the list of language bindings Summary: Closes https://github.com/facebook/rocksdb/pull/3183 Differential Revision: D6386140 Pulled By: maysamyabandeh fbshipit-source-id: ca71d54edd741c3b7d9676ee2bcf584a5d49bc35 | 21 November 2017, 18:13:14 UTC |
8954f83 | Sagar Vemuri | 21 November 2017, 07:19:46 UTC | Blob DB: db_bench flag to control BlobDB's garbage collection Summary: flag: blob_db_enable_gc, to control BlobDb's enable_garbage_collection. Closes https://github.com/facebook/rocksdb/pull/3190 Differential Revision: D6383395 Pulled By: sagar0 fbshipit-source-id: 4134e835150748c425b8187264273a54c6d8381c | 21 November 2017, 07:26:15 UTC |
d394a6b | anand1976 | 21 November 2017, 05:12:55 UTC | Add a ticker stat for number of keys skipped during iteration Summary: This diff adds a new ticker stat, NUMBER_ITER_SKIP, to count the number of internal keys skipped during iteration. Keys can be skipped due to deletes, or lower sequence number, or higher sequence number than the one requested. Also, fix the issue when StatisticsData is naturally aligned on cacheline boundary, padding becomes a zero size array, which the Windows compiler doesn't like. So add a cacheline worth of padding in that case to keep it happy. We cannot conditionally add padding as gcc doesn't allow using sizeof in preprocessor directives. Closes https://github.com/facebook/rocksdb/pull/3177 Differential Revision: D6353897 Pulled By: anand1976 fbshipit-source-id: 441d5a09af9c4e22e7355242dfc0c7b27aa0a6c2 | 21 November 2017, 05:26:37 UTC |
578f36e | Sagar Vemuri | 21 November 2017, 05:06:30 UTC | Blob DB: Remove some redundant log lines Summary: Saw some redundant log lines when trying to benchmark blob db. So, removed the lines from blob_file.cc, and let the lines in blob_db_impl.cc take the lead. Closes https://github.com/facebook/rocksdb/pull/3189 Differential Revision: D6381726 Pulled By: sagar0 fbshipit-source-id: 5f0b1e56fe4bc3b715d89ea9b5749bd935cd0606 | 21 November 2017, 05:11:34 UTC |
a6c6b8b | Zhongyi Xie | 21 November 2017, 00:31:04 UTC | Revert "No need for Restart Interval for meta blocks" Summary: See [issue 3169](https://github.com/facebook/rocksdb/issues/3169) for more information This reverts commit 593d3de37171d99a761ce2ab34ffa12654acd055. Closes https://github.com/facebook/rocksdb/pull/3188 Differential Revision: D6379271 Pulled By: miasantreble fbshipit-source-id: 88f9ed67ba52237ad9b6f7251db83672b62d7537 | 21 November 2017, 00:42:03 UTC |
30285ee | Maysam Yabandeh | 20 November 2017, 21:16:04 UTC | Fix calculating filter partition target size Summary: block_size_deviation is in percentage while the partition size is in bytes. The current code fails to take that into account resulting into very large target size for filter partitions. Closes https://github.com/facebook/rocksdb/pull/3187 Differential Revision: D6376069 Pulled By: maysamyabandeh fbshipit-source-id: 276546fc68f50e0da32c462abb46f6cf676db9b2 | 20 November 2017, 21:26:57 UTC |
0996e14 | Giorgio Azzinnaro | 19 November 2017, 17:59:38 UTC | Added ProfaneDB Summary: Added my project [ProfaneDB](https://profanedb.gitlab.io/) which uses RocksDB as the main storage engine Closes https://github.com/facebook/rocksdb/pull/3182 Differential Revision: D6370847 Pulled By: ajkr fbshipit-source-id: ff4bb6bdbc6e42fdb88bd793a84a0e04c129b6ae | 19 November 2017, 18:11:44 UTC |
0213990 | Maysam Yabandeh | 18 November 2017, 01:02:13 UTC | Move static variables out of the header file Summary: Static variables in header files will be instantiated in every file that includes the header file. This patch moves some of them from options_helper.h to its .cc files. It also moves the static variable out of the offset_of since the template function could also lead to multiple instantiation perhaps due to inlining. Fixes https://github.com/facebook/rocksdb/issues/3176 Closes https://github.com/facebook/rocksdb/pull/3178 Differential Revision: D6363794 Pulled By: maysamyabandeh fbshipit-source-id: d0a07f061b4d992ab4e0de2706e622131d258fdd | 18 November 2017, 01:12:27 UTC |
2d04ed6 | Gustav Davidsson | 17 November 2017, 19:56:41 UTC | Make trash-to-DB size ratio limit configurable Summary: Allow users to configure the trash-to-DB size ratio limit, so that ratelimits for deletes can be enforced even when larger portions of the database are being deleted. Closes https://github.com/facebook/rocksdb/pull/3158 Differential Revision: D6304897 Pulled By: gdavidsson fbshipit-source-id: a28dd13059ebab7d4171b953ed91ce383a84d6b3 | 17 November 2017, 19:58:17 UTC |
32e31d4 | Zhongyi Xie | 17 November 2017, 01:46:43 UTC | Make DBOption compaction_readahead_size dynamic Summary: Closes https://github.com/facebook/rocksdb/pull/3004 Differential Revision: D6056141 Pulled By: miasantreble fbshipit-source-id: 56df1630f464fd56b07d25d38161f699e0528b7f | 17 November 2017, 01:57:25 UTC |
e27f60b | Andrew Kryczka | 16 November 2017, 03:30:35 UTC | distinguish kZSTDNotFinalCompression in compression string Summary: This confused some users who were getting compression type from the logs. Closes https://github.com/facebook/rocksdb/pull/3153 Differential Revision: D6294964 Pulled By: ajkr fbshipit-source-id: 3c813376d33682dc6ccafc9a78df1a2e2528985e | 16 November 2017, 03:41:59 UTC |
dd49f89 | Yi Wu | 15 November 2017, 22:13:39 UTC | Fix TARGETS lint warnings. Summary: Fix buckifier script and regenerate TARGETS file with no lint warnings. Closes https://github.com/facebook/rocksdb/pull/3170 Differential Revision: D6328993 Pulled By: yiwu-arbug fbshipit-source-id: 17d0e4ed92f676f35fed76659386611cc72b00b2 | 15 November 2017, 22:28:34 UTC |
bbcd3b0 | Yi Wu | 15 November 2017, 22:13:03 UTC | Suppress valgrind "unimplemented functionality" error Summary: Add ROCKSDB_VALGRIND_RUN macro and suppress false-positive "unimplemented functionality" throw by valgrind for steam hints. Another approach would be add a valgrind suppress file. Valgrind is suppose to print the suppression when given "--gen-suppressions=all" param, which is suppose to be the content for the suppression file. But it doesn't print. Closes https://github.com/facebook/rocksdb/pull/3174 Differential Revision: D6338786 Pulled By: yiwu-arbug fbshipit-source-id: 3559efa5f3b92d40d09ad6ac82bc7b59f86c75aa | 15 November 2017, 22:28:34 UTC |
54b4356 | Maysam Yabandeh | 15 November 2017, 16:22:54 UTC | WritePrepared Txn: Refactoring WriteCallback Summary: Refactor the logic around WriteCallback in the write path to clarify when and how exactly we advance the sequence number and making sure it is consistent across the code. Closes https://github.com/facebook/rocksdb/pull/3168 Differential Revision: D6324312 Pulled By: maysamyabandeh fbshipit-source-id: 9a34f479561fdb2a5d01ef6d37a28908d03bbe33 | 15 November 2017, 16:27:06 UTC |
53863b7 | Maysam Yabandeh | 15 November 2017, 16:19:57 UTC | WritePrepared Txn: fix bug with Rollback seq Summary: The sequence number was not properly advanced after a rollback marker. The patch extends the existing unit tests to detect the bug and also fixes it. Closes https://github.com/facebook/rocksdb/pull/3157 Differential Revision: D6304291 Pulled By: maysamyabandeh fbshipit-source-id: 1b519c44a5371b802da49c9e32bd00087a8da401 | 15 November 2017, 16:27:06 UTC |
175d5d6 | Maysam Yabandeh | 14 November 2017, 16:42:14 UTC | Properly destruct rebuilding_trx_ Summary: When testing rebuilding_trx_ in MemTableInserter might still be set before the tests finishes which would cause ASAN alarms for leaks. This patch deletes the pointers in MemTableInserter destructor. Closes https://github.com/facebook/rocksdb/pull/3162 Differential Revision: D6317113 Pulled By: maysamyabandeh fbshipit-source-id: a68be70709a4fff7ac2b768660119311968f9c21 | 14 November 2017, 16:56:50 UTC |
9871ea4 | Yi Wu | 14 November 2017, 05:14:21 UTC | Regression test build binaries with PORTABLE=1 Summary: We hit "Illegal instruction" error in regression test with "shlx" instruction. Setting PORTABLE=1 to resolve it. Closes https://github.com/facebook/rocksdb/pull/3165 Differential Revision: D6321972 Pulled By: yiwu-arbug fbshipit-source-id: cc9fe0dbd4698d1b66a750a0b062f66899862719 | 14 November 2017, 05:26:24 UTC |
42564ad | Yi Wu | 14 November 2017, 02:03:57 UTC | Blob DB: not using PinnableSlice move assignment Summary: The current implementation of PinnableSlice move assignment have an issue #3163. We are moving away from it instead of try to get the move assignment right, since it is too tricky. Closes https://github.com/facebook/rocksdb/pull/3164 Differential Revision: D6319201 Pulled By: yiwu-arbug fbshipit-source-id: 8f3279021f3710da4a4caa14fd238ed2df902c48 | 14 November 2017, 02:12:20 UTC |
2515266 | Maysam Yabandeh | 11 November 2017, 21:08:22 UTC | WritePrepared Txn: Refactoring TrackKeys Summary: This patch clarifies and refactors the logic around tracked keys in transactions. Closes https://github.com/facebook/rocksdb/pull/3140 Differential Revision: D6290258 Pulled By: maysamyabandeh fbshipit-source-id: 03b50646264cbcc550813c060b180fc7451a55c1 | 11 November 2017, 21:14:20 UTC |
2edc92b | Maysam Yabandeh | 11 November 2017, 19:23:43 UTC | WritePrepared Txn: cross-compatibility test Summary: Add tests to ensure that WritePrepared and WriteCommitted policies are cross compatible when the db WAL is empty. This is important when the admin want to switch between the policies. In such case, before the switch the admin needs to empty the WAL by i) committing/rollbacking all the pending transactions, ii) FlushMemTables Closes https://github.com/facebook/rocksdb/pull/3118 Differential Revision: D6227247 Pulled By: maysamyabandeh fbshipit-source-id: bcde3d92c1e89cda3b9cfa69f6a20af5d8993db7 | 11 November 2017, 19:28:37 UTC |
857adf3 | Maysam Yabandeh | 11 November 2017, 01:18:01 UTC | WritePrepared Txn: Refactor conf params Summary: Summary of changes: - Move seq_per_batch out of Options - Rename concurrent_prepare to two_write_queues - Add allocate_seq_only_for_data_ Closes https://github.com/facebook/rocksdb/pull/3136 Differential Revision: D6304458 Pulled By: maysamyabandeh fbshipit-source-id: 08e685bfa82bbc41b5b1c5eb7040a8ca6e05e58c | 11 November 2017, 01:28:12 UTC |
07c2738 | Andrew Kryczka | 11 November 2017, 00:53:02 UTC | prefer enabling cpu features via -march/-mcpu Summary: If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2. Closes https://github.com/facebook/rocksdb/pull/3156 Differential Revision: D6304703 Pulled By: ajkr fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef | 11 November 2017, 00:57:11 UTC |
f8e2db0 | Dmitri Smirnov | 10 November 2017, 18:27:35 UTC | Fix crashes, address test issues and adjust windows test script Summary: Add per-exe execution capability Add fix parsing of groups/tests Add timer test exclusion Fix unit tests Ifdef threadpool specific tests that do not pass on Vista threadpool. Remove spurious outout from prefix_test so test case listing works properly. Fix not using standard test directories results in file creation errors in sst_dump_test. BlobDb fixes: In C++ end() iterators can not be dereferenced. They are not valid. When deleting blob_db_ set it to nullptr before any other code executes. Not fixed:. On Windows you can not delete a file while it is open. [ RUN ] BlobDBTest.ReadWhileGC d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied write_batch Should not call front() if there is a chance the container is empty Closes https://github.com/facebook/rocksdb/pull/3152 Differential Revision: D6293274 Pulled By: sagar0 fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956 | 10 November 2017, 18:41:57 UTC |
eefd75a | Shaohua Li | 10 November 2017, 17:25:26 UTC | Stream Summary: Add a simple policy for NVMe write time life hint Closes https://github.com/facebook/rocksdb/pull/3095 Differential Revision: D6298030 Pulled By: shligit fbshipit-source-id: 9a72a42e32e92193af11599eb71f0cf77448e24d | 10 November 2017, 17:26:24 UTC |
f1c5eab | kapitan-k | 10 November 2017, 02:06:50 UTC | updated c ingestexternalfileoptions for ingest behind Summary: Closes https://github.com/facebook/rocksdb/pull/3151 Differential Revision: D6293861 Pulled By: ajkr fbshipit-source-id: f8db0a71509d1cd8237f2d377bf9e1bb0464bdbf | 10 November 2017, 02:15:09 UTC |
93f69cb | Andrew Kryczka | 10 November 2017, 01:33:01 UTC | use bottommost compression when base level is bottommost Summary: The previous compression type selection caused unexpected behavior when the base level was also the bottommost level. The following sequence of events could happen: - full compaction generates files with `bottommost_compression` type - now base level is bottommost level since all files are in the same level - any compaction causes files to be rewritten `compression_per_level` type since bottommost compression didn't apply to base level I changed the code to make bottommost compression apply to base level. Closes https://github.com/facebook/rocksdb/pull/3141 Differential Revision: D6264614 Pulled By: ajkr fbshipit-source-id: d7aaa8675126896684154a1f2c9034d6214fde82 | 10 November 2017, 01:42:00 UTC |
5e9e5a4 | Yi Wu | 09 November 2017, 03:33:12 UTC | Blob DB: Fix race condition between flush and write Summary: A race condition will happen when: * a user thread writes a value, but it hits the write stop condition because there are too many un-flushed memtables, while holding blob_db_impl.write_mutex_. * Flush is triggered and call flush begin listener and try to acquire blob_db_impl.write_mutex_. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3149 Differential Revision: D6279805 Pulled By: yiwu-arbug fbshipit-source-id: 0e3c58afb78795ebe3360a2c69e05651e3908c40 | 09 November 2017, 03:42:22 UTC |