Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
ac329cc sdong 21 February 2018, 23:12:10 UTC Update HISTORY.md 21 February 2018, 23:12:10 UTC
3300ca0 Sagar Vemuri 21 February 2018, 03:05:21 UTC Add rocksdb.iterator.internal-key property Summary: Added a new iterator property: `rocksdb.iterator.internal-key` to get the internal-key (converted to user key) at which the iterator stopped. Closes https://github.com/facebook/rocksdb/pull/3525 Differential Revision: D7033694 Pulled By: sagar0 fbshipit-source-id: d51e6c00f5e9d766c6276ef79774b81c6c5216f8 21 February 2018, 18:20:59 UTC
aa3b8bb Sagar Vemuri 16 February 2018, 22:18:14 UTC Bump version to 5.10.3 16 February 2018, 22:18:14 UTC
0493719 Mike Kolupaev 16 February 2018, 15:58:18 UTC Fix deadlock in ColumnFamilyData::InstallSuperVersion() Summary: Deadlock: a memtable flush holds DB::mutex_ and calls ThreadLocalPtr::Scrape(), which locks ThreadLocalPtr mutex; meanwhile, a thread exit handler locks ThreadLocalPtr mutex and calls SuperVersionUnrefHandle, which tries to lock DB::mutex_. This deadlock is hit all the time on our workload. It blocks our release. In general, the problem is that ThreadLocalPtr takes an arbitrary callback and calls it while holding a lock on a global mutex. The same global mutex is (at least in some cases) locked by almost all ThreadLocalPtr methods, on any instance of ThreadLocalPtr. So, there'll be a deadlock if the callback tries to do anything to any instance of ThreadLocalPtr, or waits for another thread to do so. So, probably the only safe way to use ThreadLocalPtr callbacks is to do only do simple and lock-free things in them. This PR fixes the deadlock by making sure that local_sv_ never holds the last reference to a SuperVersion, and therefore SuperVersionUnrefHandle never has to do any nontrivial cleanup. I also searched for other uses of ThreadLocalPtr to see if they may have similar bugs. There's only one other use, in transaction_lock_mgr.cc, and it looks fine. Closes https://github.com/facebook/rocksdb/pull/3510 Reviewed By: sagar0 Differential Revision: D7005346 Pulled By: al13n321 fbshipit-source-id: 37575591b84f07a891d6659e87e784660fde815f 16 February 2018, 19:34:46 UTC
7ad7940 Siying Dong 14 February 2018, 00:20:13 UTC Direct I/O writable file should do fsync in Close() Summary: We don't do fsync() after truncate in direct I/O writeable file (in fact we don't do any fsync ever). This can cause metadata not persistent to disk after the file is generated. We call it instead. Closes https://github.com/facebook/rocksdb/pull/3500 Differential Revision: D6981482 Pulled By: siying fbshipit-source-id: 7e2b591b7e5dd1b96fc0775515b8b9e6092980ef 14 February 2018, 00:52:07 UTC
257eb0a Wouter Beek 20 December 2017, 16:02:53 UTC FIXED: string buffers potentially too small to fit formatted write Summary: This fixes the following warnings when compiled with GCC7: util/transaction_test_util.cc: In static member function ‘static rocksdb::Status rocksdb::RandomTransactionInserter::DBGet(rocksdb::DB*, rocksdb::Transaction*, rocksdb::ReadOptions&, uint16_t, uint64_t, bool, uint64_t*, std::__cxx11::string*, bool*)’: util/transaction_test_util.cc:75:8: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=] Status RandomTransactionInserter::DBGet( ^~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc:84:11: note: ‘snprintf’ output between 5 and 6 bytes into a destination of size 5 snprintf(prefix_buf, sizeof(prefix_buf), "%.4u", set_i + 1); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc: In static member function ‘static rocksdb::Status rocksdb::RandomTransactionInserter::Verify(rocksdb::DB*, uint16_t, uint64_t, bool, rocksdb::Random64*)’: util/transaction_test_util.cc:245:8: error: ‘snprintf’ output may be truncated before the last format character [-Werror=format-truncation=] Status RandomTransactionInserter::Verify(DB* db, uint16_t num_sets, ^~~~~~~~~~~~~~~~~~~~~~~~~ util/transaction_test_util.cc:268:13: note: ‘snprintf’ output between 5 and 6 bytes into a destination of size 5 snprintf(prefix_buf, sizeof(prefix_buf), "%.4u", set_i + 1); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Closes https://github.com/facebook/rocksdb/pull/3295 Differential Revision: D6609411 Pulled By: maysamyabandeh fbshipit-source-id: 33f0add471056eb59db2f8bd4366e6dfbb1a187d 13 February 2018, 21:37:32 UTC
54c1984 Jun Wu 01 February 2018, 22:15:28 UTC crc32: suppress -Wimplicit-fallthrough warnings Summary: Workaround a bunch of "implicit-fallthrough" compiler errors, like: ``` util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=] crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset)); ^ util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’ CRCsinglet(crc0, next, -2 * 8); ^~~~~~~~~~ util/crc32c.cc:1017:7: note: here case 1: ``` Closes https://github.com/facebook/rocksdb/pull/3339 Reviewed By: sagar0 Differential Revision: D6874736 Pulled By: quark-zju fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16 12 February 2018, 20:58:31 UTC
7ac5446 Mark Isaacson 29 January 2018, 20:43:56 UTC Suppress lint in old files Summary: Grandfather in super old lint issues to make a clean slate for moving forward that allows us to have stronger enforcement on new issues. Reviewed By: yiwu-arbug Differential Revision: D6821806 fbshipit-source-id: 22797d31ec58e9eb0255d3b66fedfcfcb0dc127c 31 January 2018, 01:15:39 UTC
c1e70e7 Yi Wu 30 January 2018, 19:13:10 UTC Bump version to 5.10.2 30 January 2018, 19:13:10 UTC
d91f420 Yi Wu 26 January 2018, 23:12:16 UTC StackableDB optionally take shared ownership of the underlying DB Summary: Allow StackableDB optionally takes a shared_ptr on construction and thus hold shared ownership of the underlying DB. Closes https://github.com/facebook/rocksdb/pull/3423 Differential Revision: D6824163 Pulled By: yiwu-arbug fbshipit-source-id: dbdc30c42e007533a987ef413785e192340f03eb 30 January 2018, 19:12:37 UTC
1454a5c Sagar Vemuri 12 January 2018, 18:53:43 UTC Fix PowerPC dynamic java build Summary: Java build on PPC64le has been broken since a few months, due to #2716. Fixing it with the least amount of changes. (We should cleanup a little around this code when time permits). This should fix the build failures seen in http://140.211.168.68:8080/job/Rocksdb/ . Closes https://github.com/facebook/rocksdb/pull/3359 Differential Revision: D6712938 Pulled By: sagar0 fbshipit-source-id: 3046e8f072180693de2af4762934ec1ace309ca4 19 January 2018, 02:02:26 UTC
d7defe4 Yi Wu 19 January 2018, 02:02:03 UTC Bump version to 5.10.1 19 January 2018, 02:02:03 UTC
e7e0aa7 Yi Wu 19 January 2018, 01:32:50 UTC Fix Flush() keep waiting after flush finish Summary: Flush() call could be waiting indefinitely if min_write_buffer_number_to_merge is used. Consider the sequence: 1. User call Flush() with flush_options.wait = true 2. The manual flush started in the background 3. New memtable become immutable because of writes. The new memtable will not trigger flush if min_write_buffer_number_to_merge is not reached. 4. The manual flush finish. Because of the new memtable created at step 3 not being flush, previous logic of WaitForFlushMemTable() keep waiting, despite the memtables it intent to flush has been flushed. Here instead of checking if there are any more memtables to flush, WaitForFlushMemTable() also check the id of the earliest memtable. If the id is larger than that of latest memtable at the time flush was initiated, it means all the memtable at the time of flush start has all been flush. Closes https://github.com/facebook/rocksdb/pull/3378 Differential Revision: D6746789 Pulled By: yiwu-arbug fbshipit-source-id: 35e698f71c7f90b06337a93e6825f4ea3b619bfa 19 January 2018, 01:52:02 UTC
6c1a724 Adam Retter 11 January 2018, 21:21:35 UTC FreeBSD build support for RocksDB and RocksJava Summary: Tested on a clean FreeBSD 11.01 x64. Closes https://github.com/facebook/rocksdb/pull/1423 Closes https://github.com/facebook/rocksdb/pull/3357 Differential Revision: D6705868 Pulled By: sagar0 fbshipit-source-id: cbccbbdafd4f42922512ca03619a5d5583a425fd 11 January 2018, 21:42:22 UTC
c43d91e Andrew Kryczka 03 January 2018, 20:37:07 UTC fix powerpc java static build Summary: added support for C and asm files as required for e612e317409e8a9d74cf05db0bd733403305f459. Closes https://github.com/facebook/rocksdb/pull/3299 Differential Revision: D6612479 Pulled By: ajkr fbshipit-source-id: 6263ed7c1602f249460421825c76b5721f396163 11 January 2018, 20:41:44 UTC
cf2c88c Sagar Vemuri 05 January 2018, 23:30:12 UTC Fix zstd/zdict include path for java static build Summary: With the ZSTD dictionary generator support added in #3057 `PORTABLE=1 ROCKSDB_NO_FBCODE=1 make rocksdbjavastatic` fails as it can't find zdict.h. Specifically due to: https://github.com/facebook/rocksdb/blob/e3a06f12d27fd50af7b6c5941973f529601f9a3e/util/compression.h#L39 In java static builds zstd code gets directly downloaded from https://github.com/facebook/zstd , and in there zdict.h is under dictBuilder directory. So, I modified libzstd.a target to use `make install` to collect all the header files into a single location and used that as the zstd's include path. Closes https://github.com/facebook/rocksdb/pull/3260 Differential Revision: D6669850 Pulled By: sagar0 fbshipit-source-id: f8a7562a670e5aed4c4fb6034a921697590d7285 11 January 2018, 01:50:53 UTC
f897be8 Yi Wu 09 January 2018, 18:53:35 UTC Fix db_bench write being disabled in lite build Summary: The macro was added by mistake in #2372 Closes https://github.com/facebook/rocksdb/pull/3343 Differential Revision: D6681356 Pulled By: yiwu-arbug fbshipit-source-id: 4180172fb0eaef4189c07f219241e0c261c03461 11 January 2018, 01:50:03 UTC
8e42c3f Siying Dong 10 January 2018, 02:12:32 UTC Remove GCC parameter "-march=native" for ARM Summary: Most popular versions of GCC can't identify platform on ARM if "-march=native" is specified. Remove it to unblock most people. Closes https://github.com/facebook/rocksdb/pull/3346 Differential Revision: D6690544 Pulled By: siying fbshipit-source-id: bbaba9fe2645b6b37144b36ea75beeff88992b49 11 January 2018, 01:49:56 UTC
0d2818d Siying Dong 09 January 2018, 22:46:30 UTC Fix a wrong log formatting Summary: I experienced weird segfault because of this mismatch of type in log formatting. Fix it. Closes https://github.com/facebook/rocksdb/pull/3345 Differential Revision: D6687224 Pulled By: siying fbshipit-source-id: c51fb1c008b7ebc3efdc353a4adad3e8f5b3e9de 11 January 2018, 01:49:43 UTC
f7285ce sdong 27 December 2017, 21:19:17 UTC Fix HISTORY.md 27 December 2017, 21:19:53 UTC
76698fe yingsu00 20 December 2017, 02:20:50 UTC Port 3 way SSE4.2 crc32c implementation from Folly Summary: **# Summary** RocksDB uses SSE crc32 intrinsics to calculate the crc32 values but it does it in single way fashion (not pipelined on single CPU core). Intel's whitepaper () published an algorithm that uses 3-way pipelining for the crc32 intrinsics, then use pclmulqdq intrinsic to combine the values. Because pclmulqdq has overhead on its own, this algorithm will show perf gains on buffers larger than 216 bytes, which makes RocksDB a perfect user, since most of the buffers RocksDB call crc32c on is over 4KB. Initial db_bench show tremendous CPU gain. This change uses the 3-way SSE algorithm by default. The old SSE algorithm is now behind a compiler tag NO_THREEWAY_CRC32C. If user compiles the code with NO_THREEWAY_CRC32C=1 then the old SSE Crc32c algorithm would be used. If the server does not have SSE4.2 at the run time the slow way (Non SSE) will be used. **# Performance Test Results** We ran the FillRandom and ReadRandom benchmarks in db_bench. ReadRandom is the point of interest here since it calculates the CRC32 for the in-mem buffers. We did 3 runs for each algorithm. Before this change the CRC32 value computation takes about 11.5% of total CPU cost, and with the new 3-way algorithm it reduced to around 4.5%. The overall throughput also improved from 25.53MB/s to 27.63MB/s. 1) ReadRandom in db_bench overall metrics PER RUN Algorithm | run | micros/op | ops/sec |Throughput (MB/s) 3-way | 1 | 4.143 | 241387 | 26.7 3-way | 2 | 3.775 | 264872 | 29.3 3-way | 3 | 4.116 | 242929 | 26.9 FastCrc32c|1 | 4.037 | 247727 | 27.4 FastCrc32c|2 | 4.648 | 215166 | 23.8 FastCrc32c|3 | 4.352 | 229799 | 25.4 AVG Algorithm | Average of micros/op | Average of ops/sec | Average of Throughput (MB/s) 3-way | 4.01 | 249,729 | 27.63 FastCrc32c | 4.35 | 230,897 | 25.53 2) Crc32c computation CPU cost (inclusive samples percentage) PER RUN Implementation | run |  TotalSamples | Crc32c percentage 3-way   | 1    |  4,572,250,000 | 4.37% 3-way   | 2    |  3,779,250,000 | 4.62% 3-way   | 3    |  4,129,500,000 | 4.48% FastCrc32c     | 1    |  4,663,500,000 | 11.24% FastCrc32c     | 2    |  4,047,500,000 | 12.34% FastCrc32c     | 3    |  4,366,750,000 | 11.68% **# Test Plan** make -j64 corruption_test && ./corruption_test By default it uses 3-way SSE algorithm NO_THREEWAY_CRC32C=1 make -j64 corruption_test && ./corruption_test make clean && DEBUG_LEVEL=0 make -j64 db_bench make clean && DEBUG_LEVEL=0 NO_THREEWAY_CRC32C=1 make -j64 db_bench Closes https://github.com/facebook/rocksdb/pull/3173 Differential Revision: D6330882 Pulled By: yingsu00 fbshipit-source-id: 8ec3d89719533b63b536a736663ca6f0dd4482e9 27 December 2017, 21:16:32 UTC
9196e80 Zhongyi Xie 13 December 2017, 05:06:26 UTC Reduce heavy hitter for Get operation Summary: This PR addresses the following heavy hitters in `Get` operation by moving calls to `StatisticsImpl::recordTick` from `BlockBasedTable` to `Version::Get` - rocksdb.block.cache.bytes.write - rocksdb.block.cache.add - rocksdb.block.cache.data.miss - rocksdb.block.cache.data.bytes.insert - rocksdb.block.cache.data.add - rocksdb.block.cache.hit - rocksdb.block.cache.data.hit - rocksdb.block.cache.bytes.read The db_bench statistics before and after the change are: |1GB block read|Children |Self |Command |Shared Object |Symbol| |---|---|---|---|---|---| |master: |4.22% |1.31% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| |updated: |0.51% |0.21% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| | |0.14% |0.14% |db_bench |db_bench |[.] rocksdb::GetContext::record_counters| |1MB block read|Children |Self |Command |Shared Object |Symbol| |---|---|---|---|---|---| |master: |3.48% |1.08% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| |updated: |0.80% |0.31% |db_bench |db_bench |[.] rocksdb::StatisticsImpl::recordTick| | |0.35% |0.35% |db_bench |db_bench |[.] rocksdb::GetContext::record_counters| Closes https://github.com/facebook/rocksdb/pull/3172 Differential Revision: D6330532 Pulled By: miasantreble fbshipit-source-id: 2b492959e00a3db29e9437ecdcc5e48ca4ec5741 15 December 2017, 00:52:06 UTC
243ca14 Siying Dong 14 December 2017, 18:18:01 UTC Print out compression type of new SST files in logging Summary: Closes https://github.com/facebook/rocksdb/pull/3264 Differential Revision: D6552768 Pulled By: siying fbshipit-source-id: 6303110aff22f341d5cff41f8d2d4f138a53652d 15 December 2017, 00:51:26 UTC
1c3c1c4 Siying Dong 14 December 2017, 18:17:22 UTC NUMBER_BLOCK_COMPRESSED, etc, shouldn't be treated as timer counter Summary: NUMBER_BLOCK_DECOMPRESSED and NUMBER_BLOCK_COMPRESSED are not reported unless the stats level contain detailed timers, which is wrong. They are normal counters. Fix it. Closes https://github.com/facebook/rocksdb/pull/3263 Differential Revision: D6552519 Pulled By: siying fbshipit-source-id: 40899ccea7b2856bb39752616657c0bfd432f6f9 15 December 2017, 00:51:13 UTC
3e40def Orvid King 14 December 2017, 19:57:11 UTC Fix the build with MSVC 2017 Summary: There were a few places where MSVC's implicit truncation warnings were getting triggered, which was causing the MSVC build to fail due to warnings being treated as errors. This resolves the issues by making the truncations in some places explicit, and by making it so there are no truncations of literals. Fixes #3239 Supersedes #3259 Closes https://github.com/facebook/rocksdb/pull/3273 Reviewed By: yiwu-arbug Differential Revision: D6569204 Pulled By: Orvid fbshipit-source-id: c188cf1cf98d9acb6d94b71875041cc81f8ff088 15 December 2017, 00:49:25 UTC
766f062 Siying Dong 11 December 2017, 23:34:51 UTC Switch version to 5.10 Summary: Closes https://github.com/facebook/rocksdb/pull/3252 Differential Revision: D6539373 Pulled By: siying fbshipit-source-id: ce7c3d3fe625852179055295da9cf7bc80755025 11 December 2017, 23:44:51 UTC
9a27ac5 Yi Wu 11 December 2017, 21:46:57 UTC Fix drop column family data race Summary: A data race is caught by tsan_crash test between compaction and DropColumnFamily: https://gist.github.com/yiwu-arbug/5a2b4baae05eeb99ae1719b650f30a44 Compaction checks if the column family has been dropped on each key input, while user can issue DropColumnFamily which updates cfd->dropped_, causing the data race. Fixing it by making cfd->dropped_ an atomic. Closes https://github.com/facebook/rocksdb/pull/3250 Differential Revision: D6535991 Pulled By: yiwu-arbug fbshipit-source-id: 5571df020beae7fa7db6fff5ad0d598f49962895 11 December 2017, 21:57:48 UTC
fcc8a65 Zhongyi Xie 11 December 2017, 21:12:12 UTC Make Universal compaction options dynamic Summary: Let me know if more test coverage is needed Closes https://github.com/facebook/rocksdb/pull/3213 Differential Revision: D6457165 Pulled By: miasantreble fbshipit-source-id: 3f944abff28aa7775237f1c4f61c64ccbad4eea9 11 December 2017, 21:27:06 UTC
250a51a Yi Wu 11 December 2017, 20:01:22 UTC BlobDB: refactor DB open logic Summary: Refactor BlobDB open logic. List of changes: Major: * On reopen, mark blob files found as immutable, do not use them for writing new keys. * Not to scan the whole file to find file footer. Instead just seek to the end of the file and try to read footer. Minor: * Move most of the real logic from blob_db.cc to blob_db_impl.cc. * Not to hold shared_ptr of event listeners in global maps in blob_db.cc * Some changes to BlobFile interface. * Improve logging and error handling. Closes https://github.com/facebook/rocksdb/pull/3246 Differential Revision: D6526147 Pulled By: yiwu-arbug fbshipit-source-id: 9dc4cdd63359a2f9b696af817086949da8d06952 11 December 2017, 20:12:38 UTC
6a183d1 Prashant D 11 December 2017, 19:48:47 UTC Fix coverity issues compaction_job, compaction_picker Summary: db/compaction_job.cc: ReportStartedCompaction(compaction); CID 1419863 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member bottommost_level_ is not initialized in this constructor nor in any functions that it calls. db/compaction_picker_universal.cc: 7struct InputFileInfo { 2. uninit_member: Non-static class member level is not initialized in this constructor nor in any functions that it calls. CID 1405355 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member index is not initialized in this constructor nor in any functions that it calls. 38 InputFileInfo() : f(nullptr) {} db/dbformat.h: ParsedInternalKey() 84 : sequence(kMaxSequenceNumber) // Make code analyzer happy CID 1168095 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member type is not initialized in this constructor nor in any functions that it calls. 85 {} // Intentionally left uninitialized (for speed) Closes https://github.com/facebook/rocksdb/pull/3091 Differential Revision: D6534558 Pulled By: yiwu-arbug fbshipit-source-id: 5ada975956196d267b3f149386842af71eda7553 11 December 2017, 19:57:15 UTC
e3814a8 Andrew Kryczka 08 December 2017, 00:09:25 UTC revert fbcode build behavior Summary: Closes https://github.com/facebook/rocksdb/pull/3242 Differential Revision: D6514255 Pulled By: ajkr fbshipit-source-id: c39fa8e745866b052649d02bf339e794d77e96a3 08 December 2017, 00:12:52 UTC
fe608e3 Dmitri Smirnov 07 December 2017, 21:41:35 UTC Fix a race condition in WindowsThread (port::Thread) Summary: Fix a race condition when we create a thread and immediately destroy This case should be supported. What happens is that the thread function needs the Data instance to actually run but has no shared ownership and must rely on the WindowsThread instance to continue existing. To address this we change unique_ptr to shared_ptr and then acquire an additional refcount for the threadproc which destroys it just before the thread exit. We choose to allocate shared_ptr instance on the heap as this allows the original thread to continue w/o waiting for the new thread to start running. Closes https://github.com/facebook/rocksdb/pull/3240 Differential Revision: D6511324 Pulled By: yiwu-arbug fbshipit-source-id: 4633ff7996daf4d287a9fe34f60c1dd28cf4ff36 07 December 2017, 21:42:53 UTC
34aa245 Prashant D 07 December 2017, 19:52:12 UTC Fix coverity issues version, write_batch Summary: db/version_builder.cc: 117 base_vstorage_->InternalComparator(); CID 1351713 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member field level_zero_cmp_.internal_comparator is not initialized in this constructor nor in any functions that it calls. db/version_edit.h: 145 FdWithKeyRange() 146 : fd(), 147 smallest_key(), 148 largest_key() { CID 1418254 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member file_metadata is not initialized in this constructor nor in any functions that it calls. 149 } db/version_set.cc: 120 } CID 1322789 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 4. uninit_member: Non-static class member curr_file_level_ is not initialized in this constructor nor in any functions that it calls. 121 } db/write_batch.cc: 939 assert(cf_mems_); CID 1419862 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 3. uninit_member: Non-static class member rebuilding_trx_seq_ is not initialized in this constructor nor in any functions that it calls. 940 } Closes https://github.com/facebook/rocksdb/pull/3092 Differential Revision: D6505666 Pulled By: yiwu-arbug fbshipit-source-id: fd2c68948a0280772691a419d72ac7e190951d86 07 December 2017, 19:57:36 UTC
baff91c Prashant D 07 December 2017, 19:50:49 UTC table: Fix coverity issues Summary: table/block.cc: 420 } CID 1396127 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 7. uninit_member: Non-static class member restart_offset_ is not initialized in this constructor nor in any functions that it calls. 421} table/block_based_table_builder.cc: CID 1418259 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 7. uninit_member: Non-static class member compressed_cache_key_prefix_size is not initialized in this constructor nor in any functions that it calls. table/block_based_table_reader.h: 3. uninit_member: Non-static class member index_type is not initialized in this constructor nor in any functions that it calls. CID 1396147 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 5. uninit_member: Non-static class member hash_index_allow_collision is not initialized in this constructor nor in any functions that it calls. 413 global_seqno(kDisableGlobalSequenceNumber) {} 414 table/cuckoo_table_reader.cc: 55 if (hash_funs == user_props.end()) { 56 status_ = Status::Corruption("Number of hash functions not found"); 5. uninit_member: Non-static class member is_last_level_ is not initialized in this constructor nor in any functions that it calls. 7. uninit_member: Non-static class member identity_as_first_hash_ is not initialized in this constructor nor in any functions that it calls. 9. uninit_member: Non-static class member use_module_hash_ is not initialized in this constructor nor in any functions that it calls. 11. uninit_member: Non-static class member num_hash_func_ is not initialized in this constructor nor in any functions that it calls. 13. uninit_member: Non-static class member key_length_ is not initialized in this constructor nor in any functions that it calls. 15. uninit_member: Non-static class member user_key_length_ is not initialized in this constructor nor in any functions that it calls. 17. uninit_member: Non-static class member value_length_ is not initialized in this constructor nor in any functions that it calls. 19. uninit_member: Non-static class member bucket_length_ is not initialized in this constructor nor in any functions that it calls. 21. uninit_member: Non-static class member cuckoo_block_size_ is not initialized in this constructor nor in any functions that it calls. 23. uninit_member: Non-static class member cuckoo_block_bytes_minus_one_ is not initialized in this constructor nor in any functions that it calls. CID 1322785 (#2 of 2): Uninitialized scalar field (UNINIT_CTOR) 25. uninit_member: Non-static class member table_size_ is not initialized in this constructor nor in any functions that it calls. 57 return; table/plain_table_index.h: 2. uninit_member: Non-static class member index_size_ is not initialized in this constructor nor in any functions that it calls. CID 1322801 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member sub_index_size_ is not initialized in this constructor nor in any functions that it calls. 128 huge_page_tlb_size_(huge_page_tlb_size) {} 129 Closes https://github.com/facebook/rocksdb/pull/3113 Differential Revision: D6505719 Pulled By: yiwu-arbug fbshipit-source-id: 38f44d8f9dfefb4c2e25d83b8df25a5201c75618 07 December 2017, 19:57:36 UTC
2e3a009 Andrew Kryczka 07 December 2017, 19:07:44 UTC fix ASAN for DeleteFilesInRange test case Summary: error message was ``` ==3095==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffd18216c40 at pc 0x0000005edda1 bp 0x7ffd18215550 sp 0x7ffd18214d00 ... Address 0x7ffd18216c40 is located in stack of thread T0 at offset 1952 in frame #0 internal_repo_rocksdb/db_compaction_test.cc:1520 rocksdb::DBCompactionTest_DeleteFileRangeFileEndpointsOverlapBug_Test::TestBody() ``` It was unsafe to have slices referring to the temporary string objects' buffers, as those strings were destroyed before the slices were used. Fixed it by assigning the strings returned by `Key()` to local variables. Closes https://github.com/facebook/rocksdb/pull/3238 Differential Revision: D6507864 Pulled By: ajkr fbshipit-source-id: dd07de1a0070c6748c1ab4f3d7bd31f9a81889d0 07 December 2017, 19:12:43 UTC
e1c569c Yi Wu 07 December 2017, 06:54:11 UTC Fix clang-analyzer false-positive on ldb_cmd.cc Summary: clang-analyzer complaint about db_ being nullptr, but it couldn't be because it checks exec_stats before proceed. Add an assert to get around the false-positive. Test Plan `make analyze` Closes https://github.com/facebook/rocksdb/pull/3236 Differential Revision: D6505417 Pulled By: yiwu-arbug fbshipit-source-id: e5b65764ea994dd9e4bab3e697b97dc70dc22cab 07 December 2017, 06:58:46 UTC
bbef8c3 Sagar Vemuri 07 December 2017, 04:43:52 UTC Log GetCurrentTime failures during Flush and Compaction Summary: `GetCurrentTime()` is used to populate `creation_time` table property during flushes and compactions. It is safe to ignore `GetCurrentTime()` failures here but they should be logged. (Note that `creation_time` property was introduced as part of TTL-based FIFO compaction in #2480.) Tes Plan: `make check` Closes https://github.com/facebook/rocksdb/pull/3231 Differential Revision: D6501935 Pulled By: sagar0 fbshipit-source-id: 376adcf4ab801d3a43ec4453894b9a10909c8eb6 07 December 2017, 04:56:53 UTC
d51fcb2 Sagar Vemuri 07 December 2017, 04:40:46 UTC Blob DB: Add db_bench options Summary: Adding more BlobDB db_bench options which are needed for benchmarking. Closes https://github.com/facebook/rocksdb/pull/3230 Differential Revision: D6500711 Pulled By: sagar0 fbshipit-source-id: 91d63122905854ef7c9148a0235568719146e6c5 07 December 2017, 04:44:12 UTC
53a516a Andrew Kryczka 07 December 2017, 03:46:44 UTC update history for recent commits Summary: I browsed through the history since 5.9 was released and found some changes worth mentioning in HISTORY.md. Closes https://github.com/facebook/rocksdb/pull/3237 Differential Revision: D6506472 Pulled By: ajkr fbshipit-source-id: 627ce9f94ca33df9f0f231a9c5ced3624b05506c 07 December 2017, 03:56:17 UTC
78d1a5e Andrew Kryczka 07 December 2017, 02:49:26 UTC Preserve overlapping file endpoint invariant Summary: Fix for #2833. - In `DeleteFilesInRange`, use `GetCleanInputsWithinInterval` instead of `GetOverlappingInputs` to make sure we get a clean cut set of files to delete. - In `GetCleanInputsWithinInterval`, support nullptr as `begin_key` or `end_key`. - In `GetOverlappingInputsRangeBinarySearch`, move the assertion for non-empty range away from `ExtendFileRangeWithinInterval`, which should be allowed to return an empty range (via `end_index < begin_index`). Closes https://github.com/facebook/rocksdb/pull/2843 Differential Revision: D5772387 Pulled By: ajkr fbshipit-source-id: e554e8461823c6be82b21a9262a2da02b3957881 07 December 2017, 02:56:54 UTC
a7d3277 Yi Wu 07 December 2017, 01:04:32 UTC Fix write_callback_test compile error Summary: Rename shadow variable name db_impl. Fixing #3227 Closes https://github.com/facebook/rocksdb/pull/3235 Differential Revision: D6504051 Pulled By: yiwu-arbug fbshipit-source-id: 186c9378dabb11f8d6db56f45c95cc3b029fcb88 07 December 2017, 01:12:27 UTC
7f04af3 Yi Wu 07 December 2017, 00:38:49 UTC ldb to allow db with --try_load_options and without an options file Summary: This is to fix tools/check_format_compatible.sh. The tool try to open old versions of rocksdb with the provided options file. When options file is missing (e.g. rocksdb 2.2), it should still proceed with default options. Closes https://github.com/facebook/rocksdb/pull/3232 Differential Revision: D6503955 Pulled By: yiwu-arbug fbshipit-source-id: e44cfcce7ddc7d12cf83466ed3f3fe7624aa78b8 07 December 2017, 00:42:26 UTC
b5798bd Yi Wu 06 December 2017, 23:56:37 UTC Add missing recent versions to format compatible test Summary: Add recent versions for format compatible test. We should probably update the script to auto include available versions (by looking at include/rocksdb/versions.h and deduce branch names), but we can do it later. Closes https://github.com/facebook/rocksdb/pull/3233 Differential Revision: D6503631 Pulled By: yiwu-arbug fbshipit-source-id: e2b01d1ef6e784ff6ffa1bd75d741755e3c69a8c 07 December 2017, 00:13:50 UTC
20995c5 Yi Wu 06 December 2017, 19:54:45 UTC Make iterator invalid on Merge error Summary: Since #1665, on merge error, iterator will be set to corrupted status, but it doesn't invalidate the iterator. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3226 Differential Revision: D6499094 Pulled By: yiwu-arbug fbshipit-source-id: 80222930f949e31f90a6feaa37ddc3529b510d2c 06 December 2017, 19:56:39 UTC
36911f5 Maysam Yabandeh 06 December 2017, 17:32:09 UTC WritePrepared Txn: stress test Summary: Augment the existing MySQLStyleTransactionTest to check for more core case scenarios. The changes showed effective in revealing the bugs reported in https://github.com/facebook/rocksdb/pull/3205 and https://github.com/facebook/rocksdb/pull/3101 Closes https://github.com/facebook/rocksdb/pull/3222 Differential Revision: D6476862 Pulled By: maysamyabandeh fbshipit-source-id: 5068497702d67ffc206a58ed96f8578fbb510137 06 December 2017, 17:42:28 UTC
1b0c58d Flarnie Marchan 06 December 2017, 02:41:08 UTC Add Code of Conduct Summary: In the past Facebook didn't promote including a Code of Conduct when creating new projects, and many projects skipped this important document. Let's fix it. :) **why make this change?:** Facebook Open Source provides a Code of Conduct statement for all projects to follow, to promote a welcoming and safe open source community. Exposing the COC via a separate markdown file is a standard being promoted by Github via the Community Profile in order to meet their Open Source Guide's recommended community standards. As you can see, adding this file will improve [the rocksdb community profile](https://github.com/facebook/rocksdb/community) checklist and increase the visibility of our COC. **test plan:** Viewing it on my branch - <img width="1008" alt="screen shot 2017-12-03 at 5 05 45 pm" src="https://user-images.githubusercontent.com/1114467/33532198-66012a56-d84c-11e7-8fab-29ed410bd600.png"> <img width="1015" alt="screen shot 2017-12-03 at 5 05 59 pm" src="https://user-images.githubusercontent.com/1114467/33532199-661813d8-d84c-11e7-941e-94754dd481e5.png"> **issue:** internal task t23481323 Closes https://github.com/facebook/rocksdb/pull/3219 Reviewed By: yiwu-arbug Differential Revision: D6494234 Pulled By: flarnie fbshipit-source-id: 55b59db335cc5546f3a1c968322b9281a3dc3aaf 06 December 2017, 02:42:35 UTC
a37d734 Adam Novak 05 December 2017, 23:05:41 UTC Add ROCKSDB_DISABLE_* environment variables Summary: Should fix #3036. Closes https://github.com/facebook/rocksdb/pull/3042 Differential Revision: D6452921 Pulled By: sagar0 fbshipit-source-id: eaf11e43fee1f8747006530cfc0c7a358f1c2f0f 05 December 2017, 23:12:46 UTC
4634c73 Alex Robinson 04 December 2017, 09:53:49 UTC Update DBOptions::IncreaseParallelism to use newer background settings Summary: The Options header file recommends using max_background_jobs rather than directly setting max_background_compactions or max_background_flushes. I've personally seen a performance problem where stalls were happening because the one background flushing thread was blocked that was fixed by this change - https://github.com/cockroachdb/cockroach/issues/19699#issuecomment-347672485 Closes https://github.com/facebook/rocksdb/pull/3208 Differential Revision: D6473178 Pulled By: ajkr fbshipit-source-id: 67c892ceb7b1909d251492640cb15a0f2262b7ed 04 December 2017, 09:56:15 UTC
57056bb Andrew Kryczka 02 December 2017, 02:26:15 UTC gflags in cmake on linux Summary: We should use it if available otherwise the tools builds never work. Thanks to #3212, we can set -DGFLAGS=1 and it'll be independent of the namespace with which gflags was compiled. Closes https://github.com/facebook/rocksdb/pull/3214 Differential Revision: D6462214 Pulled By: ajkr fbshipit-source-id: db4e5f1b905322e3119554a9d01b57532c499384 02 December 2017, 02:28:24 UTC
63f1c0a Andrew Kryczka 01 December 2017, 18:40:45 UTC fix gflags namespace Summary: I started adding gflags support for cmake on linux and got frustrated that I'd need to duplicate the build_detect_platform logic, which determines namespace based on attempting compilation. We can do it differently -- use the GFLAGS_NAMESPACE macro if available, and if not, that indicates it's an old gflags version without configurable namespace so we can simply hardcode "google". Closes https://github.com/facebook/rocksdb/pull/3212 Differential Revision: D6456973 Pulled By: ajkr fbshipit-source-id: 3e6d5bde3ca00d4496a120a7caf4687399f5d656 01 December 2017, 18:42:05 UTC
18dcf7f Maysam Yabandeh 01 December 2017, 07:39:56 UTC WritePrepared Txn: PreReleaseCallback Summary: Add PreReleaseCallback to be called at the end of WriteImpl but before publishing the sequence number. The callback is used in WritePrepareTxn to i) update the commit map, ii) update the last published sequence number in the 2nd write queue. It also ensures that all the commits will go to the 2nd queue. These changes will ensure that the commit map is updated before the sequence number is published and used by reading snapshots. If we use two write queues, the snapshots will use the seq number published by the 2nd queue. If we use one write queue (the default, the snapshots will use the last seq number in the memtable, which also indicates the last published seq number. Closes https://github.com/facebook/rocksdb/pull/3205 Differential Revision: D6438959 Pulled By: maysamyabandeh fbshipit-source-id: f8b6c434e94bc5f5ab9cb696879d4c23e2577ab9 01 December 2017, 07:50:45 UTC
3e40a5e Nikhil Benesch 01 December 2017, 06:37:19 UTC add missing config checks to CMakeLists.txt Summary: Bring CMakeLists.txt back up to parity with build_detect_platform. Closes https://github.com/facebook/rocksdb/pull/3211 Differential Revision: D6452908 Pulled By: ajkr fbshipit-source-id: 93f5f336ad7eff6ecf65dec47bfaf114dd24cfb2 01 December 2017, 06:57:00 UTC
fad1405 Andres Suarez 30 November 2017, 23:22:41 UTC Remove `import` use from TARGETS Summary: We're moving away from `import`. The equivalent internal construct that gets the directory from `fbcode/` is `package_name()`. This is a Skylark friendly wrapper around [`get_base_path`]. The additional whitespace change is from running `python ./buckifier/buckify_rocksdb.py`. [`get_base_path`]: https://buckbuild.com/function/get_base_path.html Closes https://github.com/facebook/rocksdb/pull/3210 Reviewed By: yiwu-arbug Differential Revision: D6451242 Pulled By: zertosh fbshipit-source-id: 445757261de0ec89d5d332c1ba9af097086326dc 30 November 2017, 23:27:34 UTC
54095d3 Yi Wu 30 November 2017, 18:40:39 UTC TARGETS file not include tests in opt mode Summary: Do not build the tests in opt mode, since SyncPoint and other test code will not be included. Closes https://github.com/facebook/rocksdb/pull/3204 Differential Revision: D6431154 Pulled By: yiwu-arbug fbshipit-source-id: c404ef042c1a6f679e5c1dc57600b3d8cb52fc28 30 November 2017, 18:56:58 UTC
ffacaaa zhangjinpeng1987 30 November 2017, 06:48:55 UTC fix Seek with lower_bound Summary: When Seek a key less than `lower_bound`, should return `lower_bound`. ajkr PTAL Closes https://github.com/facebook/rocksdb/pull/3199 Differential Revision: D6421126 Pulled By: ajkr fbshipit-source-id: a06c825830573e0040630704f6bcb3f7f48626f7 30 November 2017, 06:56:29 UTC
ed3af9e Andrew Kryczka 29 November 2017, 01:20:47 UTC improve ldb CLI option support Summary: - Made CLI arguments take precedence over options file when both are provided. Note some of the CLI args are not settable via options file, like `--compression_max_dict_bytes`, so it's necessary to allow both ways of providing options simultaneously. - Changed `PrepareOptionsForOpenDB` to update the proper `ColumnFamilyOptions` if one exists for the user's `--column_family_name` argument. I supported this only in the base class, `LDBCommand`, so it works for the general arguments. Will defer adding support for subcommand-specific arguments. - Made the command fail if `--try_load_options` is provided and loading options file returns NotFound. I found the previous behavior of silently continuing confusing. Closes https://github.com/facebook/rocksdb/pull/3144 Differential Revision: D6270544 Pulled By: ajkr fbshipit-source-id: 7c2eac9f9b38720523d74466fb9e78db53561367 29 November 2017, 01:28:58 UTC
c85f8cc Andrew Kryczka 29 November 2017, 01:20:21 UTC convert null terminator in ascii dump Summary: The ASCII output is almost always useless to me as the first '\0' byte in the key or value causes it to stop printing. Since all characters are already surrounded by spaces, "\ 0" (how we display a backslash followed by a zero) and "\0" (how this PR displays a null terminator) are distinguishable. My assumption is the value of seeing all the bytes outweighs the value of the alignment we had before, where we always had one character followed by one space. Closes https://github.com/facebook/rocksdb/pull/3203 Differential Revision: D6428651 Pulled By: ajkr fbshipit-source-id: aafc978a51e9ea029cfe3e763e2bb0e1751b9ccf 29 November 2017, 01:28:58 UTC
c1ed005 Prashant D 28 November 2017, 23:15:16 UTC tools: Fix coverity issues Summary: tools/ldb_cmd.cc: ``` 310 ignore_unknown_options_ = IsFlagPresent(flags, ARG_IGNORE_UNKNOWN_OPTIONS); CID 1322798 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 5. uninit_member: Non-static class member db_ttl_ is not initialized in this constructor nor in any functions that it calls. 311} ``` Closes https://github.com/facebook/rocksdb/pull/3122 Differential Revision: D6428576 Pulled By: sagar0 fbshipit-source-id: d77f04dd201f7f1d9f59ef88a215ee7ad7b934e9 28 November 2017, 23:27:41 UTC
81cf262 Prashant D 28 November 2017, 22:31:46 UTC utilities/backupable : Fix coverity issues Summary: 1. Class BackupMeta ``` 52 : timestamp_(0), size_(0), meta_filename_(meta_filename), CID 1168103 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member sequence_number_ is not initialized in this constructor nor in any functions that it calls. 153 file_infos_(file_infos), env_(env) {} ``` 2. class BackupEngineImpl ``` 513 } 7. uninit_member: Non-static class member latest_backup_id_ is not initialized in this constructor nor in any functions that it calls. CID 1322803 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 9. uninit_member: Non-static class member latest_valid_backup_id_ is not initialized in this constructor nor in any functions that it calls. 514} ``` 3. struct BackupAfterCopyOrCreateWorkItem ``` 368 struct BackupAfterCopyOrCreateWorkItem { 369 std::future<CopyOrCreateResult> result; 1. member_decl: Class member declaration for shared. 370 bool shared; 3. member_decl: Class member declaration for needed_to_copy. 371 bool needed_to_copy; 5. member_decl: Class member declaration for backup_env. 372 Env* backup_env; 373 std::string dst_path_tmp; 374 std::string dst_path; 375 std::string dst_relative; 2. uninit_member: Non-static class member shared is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member needed_to_copy is not initialized in this constructor nor in any functions that it calls. CID 1396122 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 6. uninit_member: Non-static class member backup_env is not initialized in this constructor nor in any functions that it calls. 376 BackupAfterCopyOrCreateWorkItem() {} ``` 4. struct CopyOrCreateWorkItem ``` 318 struct CopyOrCreateWorkItem { 319 std::string src_path; 320 std::string dst_path; 321 std::string contents; 1. member_decl: Class member declaration for src_env. 322 Env* src_env; 3. member_decl: Class member declaration for dst_env. 323 Env* dst_env; 5. member_decl: Class member declaration for sync. 324 bool sync; 7. member_decl: Class member declaration for rate_limiter. 325 RateLimiter* rate_limiter; 9. member_decl: Class member declaration for size_limit. 326 uint64_t size_limit; 327 std::promise<CopyOrCreateResult> result; 328 std::function<void()> progress_callback; 329 2. uninit_member: Non-static class member src_env is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member dst_env is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member sync is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member rate_limiter is not initialized in this constructor nor in any functions that it calls. CID 1396123 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 10. uninit_member: Non-static class member size_limit is not initialized in this constructor nor in any functions that it calls. 330 CopyOrCreateWorkItem() {} ``` 5. struct RestoreAfterCopyOrCreateWorkItem ``` struct RestoreAfterCopyOrCreateWorkItem { 410 std::future<CopyOrCreateResult> result; 1. member_decl: Class member declaration for checksum_value. 411 uint32_t checksum_value; CID 1396153 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member checksum_value is not initialized in this constructor nor in any functions that it calls. 412 RestoreAfterCopyOrCreateWorkItem() {} ``` Closes https://github.com/facebook/rocksdb/pull/3131 Differential Revision: D6428556 Pulled By: sagar0 fbshipit-source-id: a86675444543eff028e3cae6942197a143a112c4 28 November 2017, 22:43:28 UTC
75d57a5 kapitan-k 28 November 2017, 22:09:17 UTC C API: Add some block based table options Summary: Closes https://github.com/facebook/rocksdb/pull/3159 Differential Revision: D6428220 Pulled By: sagar0 fbshipit-source-id: 60508d09b5281f54b907a1c40e9631fc08343131 28 November 2017, 22:12:44 UTC
b45fbc1 Prashant D 28 November 2017, 21:15:20 UTC utilities: Fix coverity issues Summary: ``` utilities/persistent_cache/block_cache_tier_file.cc 64struct CacheRecordHeader { 2. uninit_member: Non-static class member magic_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member crc_ is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member key_size_ is not initialized in this constructor nor in any functions that it calls. CID 1396161 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member val_size_ is not initialized in this constructor nor in any functions that it calls. 65 CacheRecordHeader() {} 66 CacheRecordHeader(const uint32_t magic, const uint32_t key_size, 67 const uint32_t val_size) 68 : magic_(magic), crc_(0), key_size_(key_size), val_size_(val_size) {} 69 1. member_decl: Class member declaration for magic_. 70 uint32_t magic_; 3. member_decl: Class member declaration for crc_. 71 uint32_t crc_; 5. member_decl: Class member declaration for key_size_. 72 uint32_t key_size_; 7. member_decl: Class member declaration for val_size_. 73 uint32_t val_size_; 74}; utilities/simulator_cache/sim_cache.cc: 157 miss_times_(0), CID 1396124 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member stats_ is not initialized in this constructor nor in any functions that it calls. 158 hit_times_(0) {} 159 ``` Closes https://github.com/facebook/rocksdb/pull/3155 Differential Revision: D6427237 Pulled By: sagar0 fbshipit-source-id: 97e493da5fc043c5b9a3e0d33103442cffb75aad 28 November 2017, 21:27:08 UTC
7b57510 Prashant D 28 November 2017, 20:17:11 UTC utilities: Fix coverity issues in blob_db and col_buf_decoder Summary: utilities/blob_db/blob_db_impl.cc 265 : bdb_options_.blob_dir; 3. uninit_member: Non-static class member env_ is not initialized in this constructor nor in any functions that it calls. 5. uninit_member: Non-static class member ttl_extractor_ is not initialized in this constructor nor in any functions that it calls. 7. uninit_member: Non-static class member open_p1_done_ is not initialized in this constructor nor in any functions that it calls. CID 1418245 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 9. uninit_member: Non-static class member debug_level_ is not initialized in this constructor nor in any functions that it calls. 266} 4. past_the_end: Function end creates an iterator. CID 1418258 (#1 of 1): Using invalid iterator (INVALIDATE_ITERATOR) 5. deref_iterator: Dereferencing iterator file_nums.end() though it is already past the end of its container. utilities/col_buf_decoder.h: nullable_(nullable), 2. uninit_member: Non-static class member remain_runs_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member run_val_ is not initialized in this constructor nor in any functions that it calls. CID 1396134 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 6. uninit_member: Non-static class member last_val_ is not initialized in this constructor nor in any functions that it calls. 46 big_endian_(big_endian) {} Closes https://github.com/facebook/rocksdb/pull/3134 Differential Revision: D6340607 Pulled By: sagar0 fbshipit-source-id: 25c52566e2ff979fe6c7abb0f40c27fc16597054 28 November 2017, 20:27:57 UTC
cf0d6aa Tamir Duberstein 28 November 2017, 20:13:20 UTC CMake cross platform Java support and add JNI to travis Summary: Rewrite `java/CMakeLists.txt` to take advantage of CMake's cross platform Java support. adamretter Closes https://github.com/facebook/rocksdb/pull/2301 Differential Revision: D5070724 Pulled By: sagar0 fbshipit-source-id: 999aee9bd39da2b24a5fe493a2eb0e9af6072dc7 28 November 2017, 20:27:53 UTC
7827935 Yi Wu 28 November 2017, 19:42:28 UTC Blob DB: Add statistics Summary: Adding a list of blob db counters. Also remove WaStats() which doesn't expose the stats and can be substitute by (BLOB_DB_BYTES_WRITTEN / BLOB_DB_BLOB_FILE_BYTES_WRITTEN). Closes https://github.com/facebook/rocksdb/pull/3193 Differential Revision: D6394216 Pulled By: yiwu-arbug fbshipit-source-id: 017508c8ff3fcd7ea7403c64d0f9834b24816803 28 November 2017, 19:58:49 UTC
3cf562b Yi Wu 28 November 2017, 19:40:40 UTC Fix IOError on WAL write doesn't propagate to write group follower Summary: This is a simpler version of #3097 by removing all unrelated changes. Fixing the bug where concurrent writes may get Status::OK while it actually gets IOError on WAL write. This happens when multiple writes form a write batch group, and the leader get an IOError while writing to WAL. The leader failed to pass the error to followers in the group, and the followers end up returning Status::OK() while actually writing nothing. The bug only affect writes in a batch group. Future writes after the batch group will correctly return immediately with the IOError. Closes https://github.com/facebook/rocksdb/pull/3201 Differential Revision: D6421644 Pulled By: yiwu-arbug fbshipit-source-id: 1c2a455c5b73f6842423785eb8a9dbfbb191dc0e 28 November 2017, 19:42:48 UTC
1bdb44d Andrew Kryczka 28 November 2017, 19:18:42 UTC optimize file ingestion checks for range deletion overlap Summary: Before we were checking every file in the level which was unnecessary. We can piggyback onto the code for checking point-key overlap, which already opens all the files that could possibly contain overlapping range deletions. This PR makes us check just the range deletions from those files, so no extra ones will be opened. Closes https://github.com/facebook/rocksdb/pull/3179 Differential Revision: D6358125 Pulled By: ajkr fbshipit-source-id: 00e200770fdb8f3cc6b1b2da232b755e4ba36279 28 November 2017, 19:27:02 UTC
022c598 Jeffrey Dallatezza 28 November 2017, 19:15:34 UTC Fix minor typo in comment Summary: mean -> meant Closes https://github.com/facebook/rocksdb/pull/3202 Differential Revision: D6426443 Pulled By: sagar0 fbshipit-source-id: adaf07218580ee6903986fa5686de92f43f420e1 28 November 2017, 19:27:02 UTC
4b65cfc Phani Shekhar Mantripragada 28 November 2017, 18:35:17 UTC Support for block_cache num_shards and other config via option string. Summary: Problem: Option string accepts only cache_size as parameter for block_cache which is specified as "block_cache=1M". It doesn't accept other parameters like num_shards etc. Changes : 1) ParseBlockBasedTableOption in block_based_table_factory is edited to accept cache options in the format "block_cache=<cache_size>:<num_shard_bits>:<strict_capacity_limit>:<high_pri_pool_ratio>". Options other than cache_size are optional to maintain backward compatibility. The changes are valid for block_cache_compressed as well. For example, "block_cache=1M:6:true:0.5", "block_cache=1M:6:true", "block_cache=1M:6" and "block_cache=1M" are all valid option strings. 2) Corresponding unit tests are added. Closes https://github.com/facebook/rocksdb/pull/3108 Differential Revision: D6420997 Pulled By: sagar0 fbshipit-source-id: cdea8b785688d2802907974af27225ccc1c0cd43 28 November 2017, 18:48:53 UTC
2f09524 Griffin Smith 28 November 2017, 18:24:23 UTC Expose all remaining read and write options via the C API Summary: Expose read and write options via the C API Closes https://github.com/facebook/rocksdb/pull/3185 Differential Revision: D6389658 Pulled By: sagar0 fbshipit-source-id: 1848912750329a476805b3cb2f315e7b71f61472 28 November 2017, 18:28:46 UTC
b72b3c6 Maysam Yabandeh 27 November 2017, 16:49:30 UTC WritePrepared Txn: Add MultiGet to DB Summary: This patch implements MultiGet API for WritePreparedTxnDB and update the existing unit tests. Closes https://github.com/facebook/rocksdb/pull/3196 Differential Revision: D6401493 Pulled By: maysamyabandeh fbshipit-source-id: 51501a1e32645fc2da8680e77a50035f6530f2cc 27 November 2017, 16:56:21 UTC
f0dde49 Yi Wu 24 November 2017, 19:50:42 UTC Blob DB: Fix GC handling for inlined blob Summary: Garbage collection checks if the offset in blob index matches the offset of the blob value in the file. If it is a mismatch, the value is the current version. However it failed to check if the blob index is an inlined type, which don't even have an offset. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3194 Differential Revision: D6394270 Pulled By: yiwu-arbug fbshipit-source-id: 7c2b9d795f1116f55f4d728086980f9b6e88ea78 24 November 2017, 19:56:47 UTC
e59cb2a Maysam Yabandeh 22 November 2017, 21:52:56 UTC Add seq_per_batch to WriteWithCallbackTest Summary: Augment WriteWithCallbackTest to also test when seq_per_batch is true. Closes https://github.com/facebook/rocksdb/pull/3195 Differential Revision: D6398143 Pulled By: maysamyabandeh fbshipit-source-id: 7bc4218609355ec20fed25df426a8455ec2390d3 22 November 2017, 21:56:44 UTC
5fac472 Zhongyi Xie 22 November 2017, 04:06:55 UTC make compaction_readahead_size_ thread safe Summary: this should fix the failing tsan_check Closes https://github.com/facebook/rocksdb/pull/3192 Differential Revision: D6390004 Pulled By: miasantreble fbshipit-source-id: 6cadfc6f68febb1a77b0abcdb5416570dad926a5 22 November 2017, 04:11:38 UTC
fe187e7 Griffin Smith 21 November 2017, 18:11:26 UTC Add Elixir to the list of language bindings Summary: Closes https://github.com/facebook/rocksdb/pull/3183 Differential Revision: D6386140 Pulled By: maysamyabandeh fbshipit-source-id: ca71d54edd741c3b7d9676ee2bcf584a5d49bc35 21 November 2017, 18:13:14 UTC
8954f83 Sagar Vemuri 21 November 2017, 07:19:46 UTC Blob DB: db_bench flag to control BlobDB's garbage collection Summary: flag: blob_db_enable_gc, to control BlobDb's enable_garbage_collection. Closes https://github.com/facebook/rocksdb/pull/3190 Differential Revision: D6383395 Pulled By: sagar0 fbshipit-source-id: 4134e835150748c425b8187264273a54c6d8381c 21 November 2017, 07:26:15 UTC
d394a6b anand1976 21 November 2017, 05:12:55 UTC Add a ticker stat for number of keys skipped during iteration Summary: This diff adds a new ticker stat, NUMBER_ITER_SKIP, to count the number of internal keys skipped during iteration. Keys can be skipped due to deletes, or lower sequence number, or higher sequence number than the one requested. Also, fix the issue when StatisticsData is naturally aligned on cacheline boundary, padding becomes a zero size array, which the Windows compiler doesn't like. So add a cacheline worth of padding in that case to keep it happy. We cannot conditionally add padding as gcc doesn't allow using sizeof in preprocessor directives. Closes https://github.com/facebook/rocksdb/pull/3177 Differential Revision: D6353897 Pulled By: anand1976 fbshipit-source-id: 441d5a09af9c4e22e7355242dfc0c7b27aa0a6c2 21 November 2017, 05:26:37 UTC
578f36e Sagar Vemuri 21 November 2017, 05:06:30 UTC Blob DB: Remove some redundant log lines Summary: Saw some redundant log lines when trying to benchmark blob db. So, removed the lines from blob_file.cc, and let the lines in blob_db_impl.cc take the lead. Closes https://github.com/facebook/rocksdb/pull/3189 Differential Revision: D6381726 Pulled By: sagar0 fbshipit-source-id: 5f0b1e56fe4bc3b715d89ea9b5749bd935cd0606 21 November 2017, 05:11:34 UTC
a6c6b8b Zhongyi Xie 21 November 2017, 00:31:04 UTC Revert "No need for Restart Interval for meta blocks" Summary: See [issue 3169](https://github.com/facebook/rocksdb/issues/3169) for more information This reverts commit 593d3de37171d99a761ce2ab34ffa12654acd055. Closes https://github.com/facebook/rocksdb/pull/3188 Differential Revision: D6379271 Pulled By: miasantreble fbshipit-source-id: 88f9ed67ba52237ad9b6f7251db83672b62d7537 21 November 2017, 00:42:03 UTC
30285ee Maysam Yabandeh 20 November 2017, 21:16:04 UTC Fix calculating filter partition target size Summary: block_size_deviation is in percentage while the partition size is in bytes. The current code fails to take that into account resulting into very large target size for filter partitions. Closes https://github.com/facebook/rocksdb/pull/3187 Differential Revision: D6376069 Pulled By: maysamyabandeh fbshipit-source-id: 276546fc68f50e0da32c462abb46f6cf676db9b2 20 November 2017, 21:26:57 UTC
0996e14 Giorgio Azzinnaro 19 November 2017, 17:59:38 UTC Added ProfaneDB Summary: Added my project [ProfaneDB](https://profanedb.gitlab.io/) which uses RocksDB as the main storage engine Closes https://github.com/facebook/rocksdb/pull/3182 Differential Revision: D6370847 Pulled By: ajkr fbshipit-source-id: ff4bb6bdbc6e42fdb88bd793a84a0e04c129b6ae 19 November 2017, 18:11:44 UTC
0213990 Maysam Yabandeh 18 November 2017, 01:02:13 UTC Move static variables out of the header file Summary: Static variables in header files will be instantiated in every file that includes the header file. This patch moves some of them from options_helper.h to its .cc files. It also moves the static variable out of the offset_of since the template function could also lead to multiple instantiation perhaps due to inlining. Fixes https://github.com/facebook/rocksdb/issues/3176 Closes https://github.com/facebook/rocksdb/pull/3178 Differential Revision: D6363794 Pulled By: maysamyabandeh fbshipit-source-id: d0a07f061b4d992ab4e0de2706e622131d258fdd 18 November 2017, 01:12:27 UTC
2d04ed6 Gustav Davidsson 17 November 2017, 19:56:41 UTC Make trash-to-DB size ratio limit configurable Summary: Allow users to configure the trash-to-DB size ratio limit, so that ratelimits for deletes can be enforced even when larger portions of the database are being deleted. Closes https://github.com/facebook/rocksdb/pull/3158 Differential Revision: D6304897 Pulled By: gdavidsson fbshipit-source-id: a28dd13059ebab7d4171b953ed91ce383a84d6b3 17 November 2017, 19:58:17 UTC
32e31d4 Zhongyi Xie 17 November 2017, 01:46:43 UTC Make DBOption compaction_readahead_size dynamic Summary: Closes https://github.com/facebook/rocksdb/pull/3004 Differential Revision: D6056141 Pulled By: miasantreble fbshipit-source-id: 56df1630f464fd56b07d25d38161f699e0528b7f 17 November 2017, 01:57:25 UTC
e27f60b Andrew Kryczka 16 November 2017, 03:30:35 UTC distinguish kZSTDNotFinalCompression in compression string Summary: This confused some users who were getting compression type from the logs. Closes https://github.com/facebook/rocksdb/pull/3153 Differential Revision: D6294964 Pulled By: ajkr fbshipit-source-id: 3c813376d33682dc6ccafc9a78df1a2e2528985e 16 November 2017, 03:41:59 UTC
dd49f89 Yi Wu 15 November 2017, 22:13:39 UTC Fix TARGETS lint warnings. Summary: Fix buckifier script and regenerate TARGETS file with no lint warnings. Closes https://github.com/facebook/rocksdb/pull/3170 Differential Revision: D6328993 Pulled By: yiwu-arbug fbshipit-source-id: 17d0e4ed92f676f35fed76659386611cc72b00b2 15 November 2017, 22:28:34 UTC
bbcd3b0 Yi Wu 15 November 2017, 22:13:03 UTC Suppress valgrind "unimplemented functionality" error Summary: Add ROCKSDB_VALGRIND_RUN macro and suppress false-positive "unimplemented functionality" throw by valgrind for steam hints. Another approach would be add a valgrind suppress file. Valgrind is suppose to print the suppression when given "--gen-suppressions=all" param, which is suppose to be the content for the suppression file. But it doesn't print. Closes https://github.com/facebook/rocksdb/pull/3174 Differential Revision: D6338786 Pulled By: yiwu-arbug fbshipit-source-id: 3559efa5f3b92d40d09ad6ac82bc7b59f86c75aa 15 November 2017, 22:28:34 UTC
54b4356 Maysam Yabandeh 15 November 2017, 16:22:54 UTC WritePrepared Txn: Refactoring WriteCallback Summary: Refactor the logic around WriteCallback in the write path to clarify when and how exactly we advance the sequence number and making sure it is consistent across the code. Closes https://github.com/facebook/rocksdb/pull/3168 Differential Revision: D6324312 Pulled By: maysamyabandeh fbshipit-source-id: 9a34f479561fdb2a5d01ef6d37a28908d03bbe33 15 November 2017, 16:27:06 UTC
53863b7 Maysam Yabandeh 15 November 2017, 16:19:57 UTC WritePrepared Txn: fix bug with Rollback seq Summary: The sequence number was not properly advanced after a rollback marker. The patch extends the existing unit tests to detect the bug and also fixes it. Closes https://github.com/facebook/rocksdb/pull/3157 Differential Revision: D6304291 Pulled By: maysamyabandeh fbshipit-source-id: 1b519c44a5371b802da49c9e32bd00087a8da401 15 November 2017, 16:27:06 UTC
175d5d6 Maysam Yabandeh 14 November 2017, 16:42:14 UTC Properly destruct rebuilding_trx_ Summary: When testing rebuilding_trx_ in MemTableInserter might still be set before the tests finishes which would cause ASAN alarms for leaks. This patch deletes the pointers in MemTableInserter destructor. Closes https://github.com/facebook/rocksdb/pull/3162 Differential Revision: D6317113 Pulled By: maysamyabandeh fbshipit-source-id: a68be70709a4fff7ac2b768660119311968f9c21 14 November 2017, 16:56:50 UTC
9871ea4 Yi Wu 14 November 2017, 05:14:21 UTC Regression test build binaries with PORTABLE=1 Summary: We hit "Illegal instruction" error in regression test with "shlx" instruction. Setting PORTABLE=1 to resolve it. Closes https://github.com/facebook/rocksdb/pull/3165 Differential Revision: D6321972 Pulled By: yiwu-arbug fbshipit-source-id: cc9fe0dbd4698d1b66a750a0b062f66899862719 14 November 2017, 05:26:24 UTC
42564ad Yi Wu 14 November 2017, 02:03:57 UTC Blob DB: not using PinnableSlice move assignment Summary: The current implementation of PinnableSlice move assignment have an issue #3163. We are moving away from it instead of try to get the move assignment right, since it is too tricky. Closes https://github.com/facebook/rocksdb/pull/3164 Differential Revision: D6319201 Pulled By: yiwu-arbug fbshipit-source-id: 8f3279021f3710da4a4caa14fd238ed2df902c48 14 November 2017, 02:12:20 UTC
2515266 Maysam Yabandeh 11 November 2017, 21:08:22 UTC WritePrepared Txn: Refactoring TrackKeys Summary: This patch clarifies and refactors the logic around tracked keys in transactions. Closes https://github.com/facebook/rocksdb/pull/3140 Differential Revision: D6290258 Pulled By: maysamyabandeh fbshipit-source-id: 03b50646264cbcc550813c060b180fc7451a55c1 11 November 2017, 21:14:20 UTC
2edc92b Maysam Yabandeh 11 November 2017, 19:23:43 UTC WritePrepared Txn: cross-compatibility test Summary: Add tests to ensure that WritePrepared and WriteCommitted policies are cross compatible when the db WAL is empty. This is important when the admin want to switch between the policies. In such case, before the switch the admin needs to empty the WAL by i) committing/rollbacking all the pending transactions, ii) FlushMemTables Closes https://github.com/facebook/rocksdb/pull/3118 Differential Revision: D6227247 Pulled By: maysamyabandeh fbshipit-source-id: bcde3d92c1e89cda3b9cfa69f6a20af5d8993db7 11 November 2017, 19:28:37 UTC
857adf3 Maysam Yabandeh 11 November 2017, 01:18:01 UTC WritePrepared Txn: Refactor conf params Summary: Summary of changes: - Move seq_per_batch out of Options - Rename concurrent_prepare to two_write_queues - Add allocate_seq_only_for_data_ Closes https://github.com/facebook/rocksdb/pull/3136 Differential Revision: D6304458 Pulled By: maysamyabandeh fbshipit-source-id: 08e685bfa82bbc41b5b1c5eb7040a8ca6e05e58c 11 November 2017, 01:28:12 UTC
07c2738 Andrew Kryczka 11 November 2017, 00:53:02 UTC prefer enabling cpu features via -march/-mcpu Summary: If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2. Closes https://github.com/facebook/rocksdb/pull/3156 Differential Revision: D6304703 Pulled By: ajkr fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef 11 November 2017, 00:57:11 UTC
f8e2db0 Dmitri Smirnov 10 November 2017, 18:27:35 UTC Fix crashes, address test issues and adjust windows test script Summary: Add per-exe execution capability Add fix parsing of groups/tests Add timer test exclusion Fix unit tests Ifdef threadpool specific tests that do not pass on Vista threadpool. Remove spurious outout from prefix_test so test case listing works properly. Fix not using standard test directories results in file creation errors in sst_dump_test. BlobDb fixes: In C++ end() iterators can not be dereferenced. They are not valid. When deleting blob_db_ set it to nullptr before any other code executes. Not fixed:. On Windows you can not delete a file while it is open. [ RUN ] BlobDBTest.ReadWhileGC d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied write_batch Should not call front() if there is a chance the container is empty Closes https://github.com/facebook/rocksdb/pull/3152 Differential Revision: D6293274 Pulled By: sagar0 fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956 10 November 2017, 18:41:57 UTC
eefd75a Shaohua Li 10 November 2017, 17:25:26 UTC Stream Summary: Add a simple policy for NVMe write time life hint Closes https://github.com/facebook/rocksdb/pull/3095 Differential Revision: D6298030 Pulled By: shligit fbshipit-source-id: 9a72a42e32e92193af11599eb71f0cf77448e24d 10 November 2017, 17:26:24 UTC
f1c5eab kapitan-k 10 November 2017, 02:06:50 UTC updated c ingestexternalfileoptions for ingest behind Summary: Closes https://github.com/facebook/rocksdb/pull/3151 Differential Revision: D6293861 Pulled By: ajkr fbshipit-source-id: f8db0a71509d1cd8237f2d377bf9e1bb0464bdbf 10 November 2017, 02:15:09 UTC
93f69cb Andrew Kryczka 10 November 2017, 01:33:01 UTC use bottommost compression when base level is bottommost Summary: The previous compression type selection caused unexpected behavior when the base level was also the bottommost level. The following sequence of events could happen: - full compaction generates files with `bottommost_compression` type - now base level is bottommost level since all files are in the same level - any compaction causes files to be rewritten `compression_per_level` type since bottommost compression didn't apply to base level I changed the code to make bottommost compression apply to base level. Closes https://github.com/facebook/rocksdb/pull/3141 Differential Revision: D6264614 Pulled By: ajkr fbshipit-source-id: d7aaa8675126896684154a1f2c9034d6214fde82 10 November 2017, 01:42:00 UTC
5e9e5a4 Yi Wu 09 November 2017, 03:33:12 UTC Blob DB: Fix race condition between flush and write Summary: A race condition will happen when: * a user thread writes a value, but it hits the write stop condition because there are too many un-flushed memtables, while holding blob_db_impl.write_mutex_. * Flush is triggered and call flush begin listener and try to acquire blob_db_impl.write_mutex_. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3149 Differential Revision: D6279805 Pulled By: yiwu-arbug fbshipit-source-id: 0e3c58afb78795ebe3360a2c69e05651e3908c40 09 November 2017, 03:42:22 UTC
back to top