Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
dd49f89 Yi Wu 15 November 2017, 22:13:39 UTC Fix TARGETS lint warnings. Summary: Fix buckifier script and regenerate TARGETS file with no lint warnings. Closes https://github.com/facebook/rocksdb/pull/3170 Differential Revision: D6328993 Pulled By: yiwu-arbug fbshipit-source-id: 17d0e4ed92f676f35fed76659386611cc72b00b2 15 November 2017, 22:28:34 UTC
bbcd3b0 Yi Wu 15 November 2017, 22:13:03 UTC Suppress valgrind "unimplemented functionality" error Summary: Add ROCKSDB_VALGRIND_RUN macro and suppress false-positive "unimplemented functionality" throw by valgrind for steam hints. Another approach would be add a valgrind suppress file. Valgrind is suppose to print the suppression when given "--gen-suppressions=all" param, which is suppose to be the content for the suppression file. But it doesn't print. Closes https://github.com/facebook/rocksdb/pull/3174 Differential Revision: D6338786 Pulled By: yiwu-arbug fbshipit-source-id: 3559efa5f3b92d40d09ad6ac82bc7b59f86c75aa 15 November 2017, 22:28:34 UTC
54b4356 Maysam Yabandeh 15 November 2017, 16:22:54 UTC WritePrepared Txn: Refactoring WriteCallback Summary: Refactor the logic around WriteCallback in the write path to clarify when and how exactly we advance the sequence number and making sure it is consistent across the code. Closes https://github.com/facebook/rocksdb/pull/3168 Differential Revision: D6324312 Pulled By: maysamyabandeh fbshipit-source-id: 9a34f479561fdb2a5d01ef6d37a28908d03bbe33 15 November 2017, 16:27:06 UTC
53863b7 Maysam Yabandeh 15 November 2017, 16:19:57 UTC WritePrepared Txn: fix bug with Rollback seq Summary: The sequence number was not properly advanced after a rollback marker. The patch extends the existing unit tests to detect the bug and also fixes it. Closes https://github.com/facebook/rocksdb/pull/3157 Differential Revision: D6304291 Pulled By: maysamyabandeh fbshipit-source-id: 1b519c44a5371b802da49c9e32bd00087a8da401 15 November 2017, 16:27:06 UTC
175d5d6 Maysam Yabandeh 14 November 2017, 16:42:14 UTC Properly destruct rebuilding_trx_ Summary: When testing rebuilding_trx_ in MemTableInserter might still be set before the tests finishes which would cause ASAN alarms for leaks. This patch deletes the pointers in MemTableInserter destructor. Closes https://github.com/facebook/rocksdb/pull/3162 Differential Revision: D6317113 Pulled By: maysamyabandeh fbshipit-source-id: a68be70709a4fff7ac2b768660119311968f9c21 14 November 2017, 16:56:50 UTC
9871ea4 Yi Wu 14 November 2017, 05:14:21 UTC Regression test build binaries with PORTABLE=1 Summary: We hit "Illegal instruction" error in regression test with "shlx" instruction. Setting PORTABLE=1 to resolve it. Closes https://github.com/facebook/rocksdb/pull/3165 Differential Revision: D6321972 Pulled By: yiwu-arbug fbshipit-source-id: cc9fe0dbd4698d1b66a750a0b062f66899862719 14 November 2017, 05:26:24 UTC
42564ad Yi Wu 14 November 2017, 02:03:57 UTC Blob DB: not using PinnableSlice move assignment Summary: The current implementation of PinnableSlice move assignment have an issue #3163. We are moving away from it instead of try to get the move assignment right, since it is too tricky. Closes https://github.com/facebook/rocksdb/pull/3164 Differential Revision: D6319201 Pulled By: yiwu-arbug fbshipit-source-id: 8f3279021f3710da4a4caa14fd238ed2df902c48 14 November 2017, 02:12:20 UTC
2515266 Maysam Yabandeh 11 November 2017, 21:08:22 UTC WritePrepared Txn: Refactoring TrackKeys Summary: This patch clarifies and refactors the logic around tracked keys in transactions. Closes https://github.com/facebook/rocksdb/pull/3140 Differential Revision: D6290258 Pulled By: maysamyabandeh fbshipit-source-id: 03b50646264cbcc550813c060b180fc7451a55c1 11 November 2017, 21:14:20 UTC
2edc92b Maysam Yabandeh 11 November 2017, 19:23:43 UTC WritePrepared Txn: cross-compatibility test Summary: Add tests to ensure that WritePrepared and WriteCommitted policies are cross compatible when the db WAL is empty. This is important when the admin want to switch between the policies. In such case, before the switch the admin needs to empty the WAL by i) committing/rollbacking all the pending transactions, ii) FlushMemTables Closes https://github.com/facebook/rocksdb/pull/3118 Differential Revision: D6227247 Pulled By: maysamyabandeh fbshipit-source-id: bcde3d92c1e89cda3b9cfa69f6a20af5d8993db7 11 November 2017, 19:28:37 UTC
857adf3 Maysam Yabandeh 11 November 2017, 01:18:01 UTC WritePrepared Txn: Refactor conf params Summary: Summary of changes: - Move seq_per_batch out of Options - Rename concurrent_prepare to two_write_queues - Add allocate_seq_only_for_data_ Closes https://github.com/facebook/rocksdb/pull/3136 Differential Revision: D6304458 Pulled By: maysamyabandeh fbshipit-source-id: 08e685bfa82bbc41b5b1c5eb7040a8ca6e05e58c 11 November 2017, 01:28:12 UTC
07c2738 Andrew Kryczka 11 November 2017, 00:53:02 UTC prefer enabling cpu features via -march/-mcpu Summary: If possible, use -march or -mcpu to get enable all features available on the local CPU or architecture. Only if this is impossible, we will manually set -msse4.2. It should be safe as there'll be a warning printed if `USE_SSE` is set and the provided flags are insufficient to support SSE4.2. Closes https://github.com/facebook/rocksdb/pull/3156 Differential Revision: D6304703 Pulled By: ajkr fbshipit-source-id: 030a53491263300cae7fafb429114d87acc828ef 11 November 2017, 00:57:11 UTC
f8e2db0 Dmitri Smirnov 10 November 2017, 18:27:35 UTC Fix crashes, address test issues and adjust windows test script Summary: Add per-exe execution capability Add fix parsing of groups/tests Add timer test exclusion Fix unit tests Ifdef threadpool specific tests that do not pass on Vista threadpool. Remove spurious outout from prefix_test so test case listing works properly. Fix not using standard test directories results in file creation errors in sst_dump_test. BlobDb fixes: In C++ end() iterators can not be dereferenced. They are not valid. When deleting blob_db_ set it to nullptr before any other code executes. Not fixed:. On Windows you can not delete a file while it is open. [ RUN ] BlobDBTest.ReadWhileGC d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied d:\dev\rocksdb\rocksdb\utilities\blob_db\blob_db_test.cc(75): error: DestroyBlobDB(dbname_, options, bdb_options) IO error: Failed to delete: d:/mnt/db\testrocksdb-17444/blob_db_test/blob_dir/000001.blob: Permission denied write_batch Should not call front() if there is a chance the container is empty Closes https://github.com/facebook/rocksdb/pull/3152 Differential Revision: D6293274 Pulled By: sagar0 fbshipit-source-id: 318c3717c22087fae13b18715dffb24565dbd956 10 November 2017, 18:41:57 UTC
eefd75a Shaohua Li 10 November 2017, 17:25:26 UTC Stream Summary: Add a simple policy for NVMe write time life hint Closes https://github.com/facebook/rocksdb/pull/3095 Differential Revision: D6298030 Pulled By: shligit fbshipit-source-id: 9a72a42e32e92193af11599eb71f0cf77448e24d 10 November 2017, 17:26:24 UTC
f1c5eab kapitan-k 10 November 2017, 02:06:50 UTC updated c ingestexternalfileoptions for ingest behind Summary: Closes https://github.com/facebook/rocksdb/pull/3151 Differential Revision: D6293861 Pulled By: ajkr fbshipit-source-id: f8db0a71509d1cd8237f2d377bf9e1bb0464bdbf 10 November 2017, 02:15:09 UTC
93f69cb Andrew Kryczka 10 November 2017, 01:33:01 UTC use bottommost compression when base level is bottommost Summary: The previous compression type selection caused unexpected behavior when the base level was also the bottommost level. The following sequence of events could happen: - full compaction generates files with `bottommost_compression` type - now base level is bottommost level since all files are in the same level - any compaction causes files to be rewritten `compression_per_level` type since bottommost compression didn't apply to base level I changed the code to make bottommost compression apply to base level. Closes https://github.com/facebook/rocksdb/pull/3141 Differential Revision: D6264614 Pulled By: ajkr fbshipit-source-id: d7aaa8675126896684154a1f2c9034d6214fde82 10 November 2017, 01:42:00 UTC
5e9e5a4 Yi Wu 09 November 2017, 03:33:12 UTC Blob DB: Fix race condition between flush and write Summary: A race condition will happen when: * a user thread writes a value, but it hits the write stop condition because there are too many un-flushed memtables, while holding blob_db_impl.write_mutex_. * Flush is triggered and call flush begin listener and try to acquire blob_db_impl.write_mutex_. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3149 Differential Revision: D6279805 Pulled By: yiwu-arbug fbshipit-source-id: 0e3c58afb78795ebe3360a2c69e05651e3908c40 09 November 2017, 03:42:22 UTC
ca75f0a Yi Wu 08 November 2017, 21:08:15 UTC Blob DB: Fix release build Summary: `compression` shadow the method name in `BlobFile`. Rename it. Closes https://github.com/facebook/rocksdb/pull/3148 Differential Revision: D6274498 Pulled By: yiwu-arbug fbshipit-source-id: 7d293596530998b23b6b8a8940f983f9b6343a98 08 November 2017, 21:14:20 UTC
4f9f124 Yi Wu 08 November 2017, 01:40:44 UTC Blob DB: use compression in file header instead of global options Summary: To fix the issue of failing to decompress existing value after reopen DB with a different compression settings. Closes https://github.com/facebook/rocksdb/pull/3142 Differential Revision: D6267260 Pulled By: yiwu-arbug fbshipit-source-id: c7cf7f3e33b0cd25520abf4771cdf9180cc02a5f 08 November 2017, 01:42:17 UTC
114896c Andrew Kryczka 07 November 2017, 21:49:35 UTC db_bench compression options Summary: - moved existing compression options to `InitializeOptionsGeneral` since they cannot be set through options file - added flag for `zstd_max_train_bytes` which was recently introduced by #3057 Closes https://github.com/facebook/rocksdb/pull/3128 Differential Revision: D6240460 Pulled By: ajkr fbshipit-source-id: 27dbebd86a55de237ba6a45cc79cff9214e82ebc 07 November 2017, 22:00:03 UTC
65c95d9 Andrew Kryczka 07 November 2017, 18:42:47 UTC support db_bench compact benchmark on bottommost files Summary: Without this option, running the compact benchmark on a DB containing only bottommost files simply returned immediately. Closes https://github.com/facebook/rocksdb/pull/3138 Differential Revision: D6256660 Pulled By: ajkr fbshipit-source-id: e3b64543acd503d821066f4200daa201d4fb3a9d 07 November 2017, 18:57:24 UTC
e03377c Manuel Ung 06 November 2017, 18:52:17 UTC Add lock wait time as a perf context counter Summary: Adds two new counters: `key_lock_wait_count` counts how many times a lock was blocked by another transaction and had to wait, instead of being granted the lock immediately. `key_lock_wait_time` counts the time spent acquiring locks. Closes https://github.com/facebook/rocksdb/pull/3107 Differential Revision: D6217332 Pulled By: lth fbshipit-source-id: 55d4f46da5550c333e523263422fd61d6a46deb9 06 November 2017, 18:57:19 UTC
be410de Yi Wu 04 November 2017, 01:10:36 UTC Fix PinnableSlice move assignment Summary: After move assignment, we need to re-initialized the moved PinnableSlice. Also update blob_db_impl.cc to not reuse the moved PinnableSlice since it is supposed to be in an undefined state after move. Closes https://github.com/facebook/rocksdb/pull/3127 Differential Revision: D6238585 Pulled By: yiwu-arbug fbshipit-source-id: bd99f2e37406c4f7de160c7dee6a2e8126bc224e 04 November 2017, 01:13:21 UTC
a6d8e30 Sagar Vemuri 03 November 2017, 21:34:10 UTC Remove unnecessary status check in TableCache::NewIterator Summary: While investigating the usage of `new_table_iterator_nanos` perf counter, I saw some code was wrapper around with unnecessary status check ... so removed it. Closes https://github.com/facebook/rocksdb/pull/3120 Differential Revision: D6229181 Pulled By: sagar0 fbshipit-source-id: f8a44fe67f5a05df94553fdb233b21e54e88cc34 03 November 2017, 21:42:08 UTC
4c8f336 Prashant D 03 November 2017, 21:32:00 UTC util: Fix coverity issues Summary: util/concurrent_arena.h: CID 1396145 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member free_begin_ is not initialized in this constructor nor in any functions that it calls. 94 Shard() : allocated_and_unused_(0) {} util/dynamic_bloom.cc: 1. Condition hash_func == NULL, taking true branch. CID 1322821 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 3. uninit_member: Non-static class member data_ is not initialized in this constructor nor in any functions that it calls. 47 hash_func_(hash_func == nullptr ? &BloomHash : hash_func) {} 48 util/file_reader_writer.h: 204 private: 205 AlignedBuffer buffer_; member_not_init_in_gen_ctor: The compiler-generated constructor for this class does not initialize buffer_offset_. 206 uint64_t buffer_offset_; CID 1418246 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) member_not_init_in_gen_ctor: The compiler-generated constructor for this class does not initialize buffer_len_. 207 size_t buffer_len_; 208}; util/thread_local.cc: 341#endif CID 1322795 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 3. uninit_member: Non-static class member pthread_key_ is not initialized in this constructor nor in any functions that it calls. 342} 40struct ThreadData { 2. uninit_member: Non-static class member next is not initialized in this constructor nor in any functions that it calls. CID 1400668 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 4. uninit_member: Non-static class member prev is not initialized in this constructor nor in any functions that it calls. 41 explicit ThreadData(ThreadLocalPtr::StaticMeta* _inst) : entries(), inst(_inst) {} 42 std::vector<Entry> entries; 1. member_decl: Class member declaration for next. 43 ThreadData* next; 3. member_decl: Class member declaration for prev. 44 ThreadData* prev; 45 ThreadLocalPtr::StaticMeta* inst; 46}; Closes https://github.com/facebook/rocksdb/pull/3123 Differential Revision: D6233566 Pulled By: sagar0 fbshipit-source-id: aa2068790ea69787a0035c0db39d59b0c25108db 03 November 2017, 21:42:08 UTC
cfb120f Andrew Kryczka 03 November 2017, 18:48:12 UTC fix CopyFile status checks Summary: copied from internal diff D6156261 Closes https://github.com/facebook/rocksdb/pull/3124 Differential Revision: D6230167 Pulled By: ajkr fbshipit-source-id: 17926bb1152d607556364e3aacfec0ef3c115748 03 November 2017, 18:57:10 UTC
d956169 Yi Wu 03 November 2017, 18:15:26 UTC Fix clang build error Summary: Fix cast from size_t to unsigned int. Closes https://github.com/facebook/rocksdb/pull/3125 Differential Revision: D6232863 Pulled By: yiwu-arbug fbshipit-source-id: 4c6131168b1faec26f7820b2cf4a09c242d323b7 03 November 2017, 18:26:54 UTC
2581c0a Yi Wu 03 November 2017, 17:22:50 UTC Blob DB: Fix BlobDBTest::SnapshotAndGarbageCollection asan failure Summary: Fix unreleased snapshot at the end of the test. Closes https://github.com/facebook/rocksdb/pull/3126 Differential Revision: D6232867 Pulled By: yiwu-arbug fbshipit-source-id: 651ca3144fc573ea2ab0ab20f0a752fb4a101d26 03 November 2017, 17:26:59 UTC
24ad430 Andrew Kryczka 03 November 2017, 05:46:13 UTC pass key/value samples through zstd compression dictionary generator Summary: Instead of using samples directly, we now support passing the samples through zstd's dictionary generator when `CompressionOptions::zstd_max_train_bytes` is set to nonzero. If set to zero, we will use the samples directly as the dictionary -- same as before. Note this is the first step of #2987, extracted into a separate PR per reviewer request. Closes https://github.com/facebook/rocksdb/pull/3057 Differential Revision: D6116891 Pulled By: ajkr fbshipit-source-id: 70ab13cc4c734fa02e554180eed0618b75255497 03 November 2017, 05:56:36 UTC
c4c1f96 Andrew Kryczka 03 November 2017, 05:16:23 UTC dynamically change current memtable size Summary: Previously setting `write_buffer_size` with `SetOptions` would only apply to new memtables. An internal user wanted it to take effect immediately, instead of at an arbitrary future point, to prevent OOM. This PR makes the memtable's size mutable, and makes `SetOptions()` mutate it. There is one case when we preserve the old behavior, which is when memtable prefix bloom filter is enabled and the user is increasing the memtable's capacity. That's because the prefix bloom filter's size is fixed and wouldn't work as well on a larger memtable. Closes https://github.com/facebook/rocksdb/pull/3119 Differential Revision: D6228304 Pulled By: ajkr fbshipit-source-id: e44bd9d10a5f8c9d8c464bf7436070bb3eafdfc9 03 November 2017, 05:28:10 UTC
30e4e01 Zhongyi Xie 03 November 2017, 05:12:17 UTC add missing else Summary: Closes https://github.com/facebook/rocksdb/pull/3121 Differential Revision: D6229415 Pulled By: miasantreble fbshipit-source-id: 57c7ad2fddf5dd6b8d7e3aaf6f62348151327dfb 03 November 2017, 05:28:06 UTC
602fe94 Prashant D 03 November 2017, 00:43:04 UTC Fix coverity issues in include/rocksdb Summary: include/rocksdb/metadata.h: struct ColumnFamilyMetaData { CID 1322804 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member file_count is not initialized in this constructor nor in any functions that it calls. struct SstFileMetaData { 2. uninit_member: Non-static class member size is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member smallest_seqno is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member largest_seqno is not initialized in this constructor nor in any functions that it calls. 8. uninit_member: Non-static class member num_reads_sampled is not initialized in this constructor nor in any functions that it calls. CID 1322807 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 10. uninit_member: Non-static class member being_compacted is not initialized in this constructor nor in any functions that it calls. include/rocksdb/sst_file_writer.h: struct ExternalSstFileInfo { 2. uninit_member: Non-static class member sequence_number is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member file_size is not initialized in this constructor nor in any functions that it calls. 6. uninit_member: Non-static class member num_entries is not initialized in this constructor nor in any functions that it calls. CID 1351697 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 8. uninit_member: Non-static class member version is not initialized in this constructor nor in any functions that it calls. 31 ExternalSstFileInfo() {} include/rocksdb/utilities/transaction.h: explicit Transaction(const TransactionDB* db) {} 2. uninit_member: Non-static class member log_number_ is not initialized in this constructor nor in any functions that it calls. CID 1396133 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member field txn_state_._M_i is not initialized in this constructor nor in any functions that it calls. 473 Transaction() {} Closes https://github.com/facebook/rocksdb/pull/3100 Differential Revision: D6227651 Pulled By: sagar0 fbshipit-source-id: 5caa4a2cf9471d1f9c3c073f81473636e1f0aa14 03 November 2017, 00:56:48 UTC
62578d8 Yi Wu 03 November 2017, 00:26:46 UTC Blob DB: Add compaction filter to remove expired blob index entries Summary: After adding expiration to blob index in #3066, we are now able to add a compaction filter to cleanup expired blob index entries. Closes https://github.com/facebook/rocksdb/pull/3090 Differential Revision: D6183812 Pulled By: yiwu-arbug fbshipit-source-id: 9cb03267a9702975290e758c9c176a2c03530b83 03 November 2017, 00:27:38 UTC
76c3fbd Sagar Vemuri 03 November 2017, 00:16:11 UTC Add Memtable Read Tier to RocksJava Summary: This options was introduced in the C++ API in #1953 . Closes https://github.com/facebook/rocksdb/pull/3064 Differential Revision: D6139010 Pulled By: sagar0 fbshipit-source-id: 164de11d539d174cf3afe7cd40e667049f44b0bc 03 November 2017, 00:27:37 UTC
7bfa880 Yi Wu 02 November 2017, 22:50:30 UTC Blob DB: fix snapshot handling Summary: Blob db will keep blob file if data in the file is visible to an active snapshot. Before this patch it checks whether there is an active snapshot has sequence number greater than the earliest sequence in the file. This is problematic since we take snapshot on every read, if it keep having reads, old blob files will not be cleanup. Change to check if there is an active snapshot falls in the range of [earliest_sequence, obsolete_sequence) where obsolete sequence is 1. if data is relocated to another file by garbage collection, it is the latest sequence at the time garbage collection finish 2. otherwise, it is the latest sequence of the file Closes https://github.com/facebook/rocksdb/pull/3087 Differential Revision: D6182519 Pulled By: yiwu-arbug fbshipit-source-id: cdf4c35281f782eb2a9ad6a87b6727bbdff27a45 02 November 2017, 22:58:27 UTC
f662f8f Yi Wu 02 November 2017, 22:47:30 UTC Blob DB: option to enable garbage collection Summary: Add an option to enable/disable auto garbage collection, where we keep counting how many keys have been evicted by either deletion or compaction and decide whether to garbage collect a blob file. Default disable auto garbage collection for now since the whole logic is not fully tested and we plan to make major change to it. Closes https://github.com/facebook/rocksdb/pull/3117 Differential Revision: D6224756 Pulled By: yiwu-arbug fbshipit-source-id: cdf53bdccec96a4580a2b3a342110ad9e8864dfe 02 November 2017, 22:58:27 UTC
167ba59 Yi Wu 02 November 2017, 20:20:28 UTC Blob DB: Fix flaky BlobDBTest::GCExpiredKeyWhileOverwriting test Summary: The test intent to wait until key being overwritten until proceed with garbage collection. It failed to wait for `PutUntil` finally finish. Fixing it. Closes https://github.com/facebook/rocksdb/pull/3116 Differential Revision: D6222833 Pulled By: yiwu-arbug fbshipit-source-id: fa9b57a772b92a66cf250b44e7975c43f62f45c5 02 November 2017, 20:27:34 UTC
25ac169 Sagar Vemuri 02 November 2017, 19:02:42 UTC Blob DB: Evict oldest blob file when close to blob db size limit Summary: Evict oldest blob file and put it in obsolete_files list when close to blob db size limit. The file will be delete when the `DeleteObsoleteFiles` background job runs next time. For now I set `kEvictOldestFileAtSize` constant, which controls when to evict the oldest file, at 90%. It could be tweaked or made into an option if really needed; I didn't want to expose it as an option pre-maturely as there are already too many :) . Closes https://github.com/facebook/rocksdb/pull/3094 Differential Revision: D6187340 Pulled By: sagar0 fbshipit-source-id: 687f8262101b9301bf964b94025a2fe9d8573421 02 November 2017, 19:11:21 UTC
3c208e7 Prashant D 02 November 2017, 18:30:51 UTC HistogramStat: Handle divide by zero situation Summary: The num() might return cur_num as 0 and we are making sure that cur_num will not be 0 down the path. The mult variable is being set to 100.0/cur_num which makes program crash when cur_num is 0. Closes https://github.com/facebook/rocksdb/pull/3105 Differential Revision: D6222594 Pulled By: ajkr fbshipit-source-id: 986154709897ff4dbbeb0e8aa81eb8c0b2a2db76 02 November 2017, 18:41:50 UTC
25fbd9a Maysam Yabandeh 02 November 2017, 18:06:10 UTC Remove the experimental notes about partitioning Summary: This patch will remove the existing comments that declare partitioning indexes and filters as experimental. Closes https://github.com/facebook/rocksdb/pull/3115 Differential Revision: D6222227 Pulled By: maysamyabandeh fbshipit-source-id: 6179ec43b22c518494051b674d91c9e1b54d4ac0 02 November 2017, 18:14:30 UTC
60d83df Maysam Yabandeh 02 November 2017, 18:05:55 UTC WritePrepared Txn: Move DB class to its own file Summary: Move WritePreparedTxnDB from pessimistic_transaction_db.h to its own header, write_prepared_txn_db.h Closes https://github.com/facebook/rocksdb/pull/3114 Differential Revision: D6220987 Pulled By: maysamyabandeh fbshipit-source-id: 18893fb4fdc6b809fe117dabb544080f9b4a301b 02 November 2017, 18:14:30 UTC
6778690 Andrew Kryczka 02 November 2017, 05:52:17 UTC fix duplicate definition of GetEntryType() Summary: It's also defined in db/dbformat.cc per 7fe3b32896ecbb21d67ec52fccb713cb9bc6a644 Closes https://github.com/facebook/rocksdb/pull/3111 Differential Revision: D6219140 Pulled By: ajkr fbshipit-source-id: 0f2b14e41457334a4665c6b7e3f42f1a060a0f35 02 November 2017, 05:56:17 UTC
cd12421 Andrew Kryczka 02 November 2017, 04:17:42 UTC release 5.9 Summary: updated HISTORY.md and version.h for the release. Closes https://github.com/facebook/rocksdb/pull/3110 Differential Revision: D6218645 Pulled By: ajkr fbshipit-source-id: 99ab8473e9088b02d7596e92351cce7a60a99e93 02 November 2017, 04:26:14 UTC
02693f6 Maysam Yabandeh 02 November 2017, 01:56:25 UTC WritePrepared Txn: ValidateSnapshot Summary: Implements ValidateSnapshot for WritePrepared txns and also adds a unit test to clarify the contract of this function. Closes https://github.com/facebook/rocksdb/pull/3101 Differential Revision: D6199405 Pulled By: maysamyabandeh fbshipit-source-id: ace509934c307ea5d26f4bbac5f836d7c80fd240 02 November 2017, 02:11:09 UTC
7fe3b32 Mikhail Antonov 02 November 2017, 01:43:29 UTC Added support for differential snapshots Summary: The motivation for this PR is to add to RocksDB support for differential (incremental) snapshots, as snapshot of the DB changes between two points in time (one can think of it as diff between to sequence numbers, or the diff D which can be thought of as an SST file or just set of KVs that can be applied to sequence number S1 to get the database to the state at sequence number S2). This feature would be useful for various distributed storages layers built on top of RocksDB, as it should help reduce resources (time and network bandwidth) needed to recover and rebuilt DB instances as replicas in the context of distributed storages. From the API standpoint that would like client app requesting iterator between (start seqnum) and current DB state, and reading the "diff". This is a very draft PR for initial review in the discussion on the approach, i'm going to rework some parts and keep updating the PR. For now, what's done here according to initial discussions: Preserving deletes: - We want to be able to optionally preserve recent deletes for some defined period of time, so that if a delete came in recently and might need to be included in the next incremental snapshot it would't get dropped by a compaction. This is done by adding new param to Options (preserve deletes flag) and new variable to DB Impl where we keep track of the sequence number after which we don't want to drop tombstones, even if they are otherwise eligible for deletion. - I also added a new API call for clients to be able to advance this cutoff seqnum after which we drop deletes; i assume it's more flexible to let clients control this, since otherwise we'd need to keep some kind of timestamp < -- > seqnum mapping inside the DB, which sounds messy and painful to support. Clients could make use of it by periodically calling GetLatestSequenceNumber(), noting the timestamp, doing some calculation and figuring out by how much we need to advance the cutoff seqnum. - Compaction codepath in compaction_iterator.cc has been modified to avoid dropping tombstones with seqnum > cutoff seqnum. Iterator changes: - couple params added to ReadOptions, to optionally allow client to request internal keys instead of user keys (so that client can get the latest value of a key, be it delete marker or a put), as well as min timestamp and min seqnum. TableCache changes: - I modified table_cache code to be able to quickly exclude SST files from iterators heep if creation_time on the file is less then iter_start_ts as passed in ReadOptions. That would help a lot in some DB settings (like reading very recent data only or using FIFO compactions), but not so much for universal compaction with more or less long iterator time span. What's left: - Still looking at how to best plug that inside DBIter codepath. So far it seems that FindNextUserKeyInternal only parses values as UserKeys, and iter->key() call generally returns user key. Can we add new API to DBIter as internal_key(), and modify this internal method to optionally set saved_key_ to point to the full internal key? I don't need to store actual seqnum there, but I do need to store type. Closes https://github.com/facebook/rocksdb/pull/2999 Differential Revision: D6175602 Pulled By: mikhail-antonov fbshipit-source-id: c779a6696ee2d574d86c69cec866a3ae095aa900 02 November 2017, 01:56:43 UTC
17731a4 Maysam Yabandeh 02 November 2017, 00:23:52 UTC WritePrepared Txn: Optimize for recoverable state Summary: GetCommitTimeWriteBatch is currently used to store some state as part of commit in 2PC. In MyRocks it is specifically used to store some data that would be needed only during recovery. So it is not need to be stored in memtable right after each commit. This patch enables an optimization to write the GetCommitTimeWriteBatch only to the WAL. The batch will be written to memtable during recovery when the WAL is replayed. To cover the case when WAL is deleted after memtable flush, the batch is also buffered and written to memtable right before each memtable flush. Closes https://github.com/facebook/rocksdb/pull/3071 Differential Revision: D6148023 Pulled By: maysamyabandeh fbshipit-source-id: 2d09bae5565abe2017c0327421010d5c0d55eaa7 02 November 2017, 00:26:46 UTC
c1cf94c Maysam Yabandeh 01 November 2017, 15:41:47 UTC WritePrepared Txn: sort indexes before batch collapse Summary: The collapse of duplicate keys in write batch needs to sort the indexes of duplicate keys since it only checks the index in the batch with the head of the list of duplicate keys. Closes https://github.com/facebook/rocksdb/pull/3093 Differential Revision: D6186800 Pulled By: maysamyabandeh fbshipit-source-id: abc9ae8c2f1840445a5584f925cf86ecc6f37154 01 November 2017, 15:56:57 UTC
f6082d1 Yi Wu 31 October 2017, 23:33:55 UTC Blob DB: cleanup unused options Summary: * cleanup num_concurrent_simple_blobs. We don't do concurrent writes (by taking write_mutex_) so it doesn't make sense to have multiple non TTL files open. We can revisit later when we want to improve writes. * cleanup eviction callback. we don't have plan to use it now. * rename s/open_simple_blob_files_/open_non_ttl_file_/ and s/open_blob_files_/open_ttl_files_/ to avoid confusion. Closes https://github.com/facebook/rocksdb/pull/3088 Differential Revision: D6182598 Pulled By: yiwu-arbug fbshipit-source-id: 99e6f5e01fa66d31309cdb06ce48502464bac6ad 31 October 2017, 23:42:08 UTC
f5078dd Sagar Vemuri 31 October 2017, 23:27:22 UTC Blob DB: Initialize all fields in Blob Header, Footer and Record structs Summary: Fixing un-itializations caught by valgrind. Closes https://github.com/facebook/rocksdb/pull/3103 Differential Revision: D6200195 Pulled By: sagar0 fbshipit-source-id: bf35a3fb03eb1d308e4c5ce30dee1e345d7b03b3 31 October 2017, 23:42:08 UTC
33c7d4c Shaohua Li 31 October 2017, 20:49:25 UTC Make writable_file_max_buffer_size dynamic Summary: The DBOptions::writable_file_max_buffer_size can be changed dynamically. Closes https://github.com/facebook/rocksdb/pull/3053 Differential Revision: D6152720 Pulled By: shligit fbshipit-source-id: aa0c0cfcfae6a54eb17faadb148d904797c68681 31 October 2017, 20:56:35 UTC
c1be8d8 Prashant D 31 October 2017, 08:24:20 UTC Fix removed structurally dead return statement Summary: There seems to be a typo mistake in env ReuseWritableFile func where status is being returned twice. Closes https://github.com/facebook/rocksdb/pull/3099 Differential Revision: D6196204 Pulled By: ajkr fbshipit-source-id: abb6e3e1c1e772dd485fc39e7f1b9d502fa188fe 31 October 2017, 08:26:13 UTC
4d43c6a Andrew Kryczka 31 October 2017, 08:22:51 UTC db_stress snapshot compatibility with reopens Summary: - Release all snapshots before crashing and reopening the DB. Without this, we may attempt to release snapshots from an old DB using a new DB. That tripped an assertion. - Release multiple snapshots in the same operation if needed. Without this, we would sometimes leak snapshots. Closes https://github.com/facebook/rocksdb/pull/3098 Differential Revision: D6194923 Pulled By: ajkr fbshipit-source-id: b9c89bcca7ebcbb6c7802c616f9d1175a005aadf 31 October 2017, 08:26:08 UTC
b7bc9cc Andrew Kryczka 30 October 2017, 07:47:15 UTC fix tracking oldest snapshot for bottom-level compaction Summary: The assertion was caught by `MySQLStyleTransactionTest/MySQLStyleTransactionTest.TransactionStressTest/5` when run in a loop. The caller doesn't track whether the released snapshot is oldest, so let this function handle that case. Closes https://github.com/facebook/rocksdb/pull/3080 Differential Revision: D6185257 Pulled By: ajkr fbshipit-source-id: 4b3015c11db5d31e46521a00af568546ef4558cd 30 October 2017, 07:55:58 UTC
792ef10 Yi Wu 29 October 2017, 04:56:50 UTC Return Status::InvalidArgument if user request sync write while disabling WAL Summary: write_options.sync = true and write_options.disableWAL is incompatible. When WAL is disabled, we are not able to persist the write immediately. Return an error in this case to avoid misuse of the options. Closes https://github.com/facebook/rocksdb/pull/3086 Differential Revision: D6176822 Pulled By: yiwu-arbug fbshipit-source-id: 1eb10028c14fe7d7c13c8bc12c0ef659f75aa071 29 October 2017, 05:11:18 UTC
6a9335d Andrew Kryczka 27 October 2017, 22:43:42 UTC always drop tombstones compacted to bottommost level Summary: Problem was in bottommost compaction, when an L0->L0 compaction happened and L0 was bottommost. Then we'd preserve tombstones according to `Compaction::KeyNotExistsBeyondOutputLevel`, while zeroing seqnum according to `CompactionIterator::PrepareOutput`, thus triggering the assertion in `PrepareOutput`. To fix, we can just drop tombstones in L0->L0 when the output is "bottommost", i.e., the compaction includes the oldest L0 file and there's nothing at lower levels. Closes https://github.com/facebook/rocksdb/pull/3085 Differential Revision: D6175742 Pulled By: ajkr fbshipit-source-id: 8ab19a2e001496f362e9eb0a71757e2f6ecfdb3b 27 October 2017, 22:56:35 UTC
84a04af Yi Wu 27 October 2017, 21:49:40 UTC TableProperty::oldest_key_time defaults to 0 Summary: We don't propagate TableProperty::oldest_key_time on compaction and just write the default value to SST files. It is more natural to default the value to 0. Also revert db_sst_test back to before #2842. Closes https://github.com/facebook/rocksdb/pull/3079 Differential Revision: D6165702 Pulled By: yiwu-arbug fbshipit-source-id: ca3ce5928d96ae79a5beb12bb7d8c640a71478a0 27 October 2017, 22:00:05 UTC
0599315 Islam AbdelRahman 27 October 2017, 20:25:54 UTC Mark files as trash by using .trash extension Summary: SstFileManager move files that need to be deleted into a trash directory. Deprecate this behaviour and instead add ".trash" extension to files that need to be deleted Closes https://github.com/facebook/rocksdb/pull/2970 Differential Revision: D5976805 Pulled By: IslamAbdelRahman fbshipit-source-id: 27374ece4315610b2792c30ffcd50232d4c9a343 27 October 2017, 20:27:12 UTC
3ebb7ba Yi Wu 27 October 2017, 20:14:34 UTC Blob DB: update blob file format Summary: Changing blob file format and some code cleanup around the change. The change with blob log format are: * Remove timestamp field in blob file header, blob file footer and blob records. The field is not being use and often confuse with expiration field. * Blob file header now come with column family id, which always equal to default column family id. It leaves room for future support of column family. * Compression field in blob file header now is a standalone byte (instead of compact encode with flags field) * Blob file footer now come with its own crc. * Key length now being uint64_t instead of uint32_t * Blob CRC now checksum both key and value (instead of value only). * Some reordering of the fields. The list of cleanups: * Better inline comments in blob_log_format.h * rename ttlrange_t and snrange_t to ExpirationRange and SequenceRange respectively. * simplify blob_db::Reader * Move crc checking logic to inside blob_log_format.cc Closes https://github.com/facebook/rocksdb/pull/3081 Differential Revision: D6171304 Pulled By: yiwu-arbug fbshipit-source-id: e4373e0d39264441b7e2fbd0caba93ddd99ea2af 27 October 2017, 20:27:12 UTC
682db81 Dmitri Smirnov 27 October 2017, 20:14:07 UTC Enable cacheline_aligned_alloc() to allocate from jemalloc if enabled. Summary: Reuse WITH_JEMALLOC option in preparation for module search unification. Move jemalloc overrides into a separate .cc Remote obsolete JEMALLOC_NOINIT option. Closes https://github.com/facebook/rocksdb/pull/3078 Differential Revision: D6174826 Pulled By: yiwu-arbug fbshipit-source-id: 9970a0289b4490272d15853920d9d7531af91140 27 October 2017, 20:27:12 UTC
d9240b5 Prashant D 27 October 2017, 18:24:12 UTC Fix coverity uninitialized fields warnings in lru_cache Summary: Coverity uninitialized member variable warnings in lru_cache Closes https://github.com/facebook/rocksdb/pull/3082 Differential Revision: D6173062 Pulled By: sagar0 fbshipit-source-id: 7bcfc653457bd362d46045d06527838c9a6adad6 27 October 2017, 18:26:43 UTC
50e95a6 Prashant D 27 October 2017, 18:19:31 UTC Fix coverity issues column_family, compaction_db/iterator Summary: db/column_family.h : 79 ColumnFamilyHandleInternal() CID 1322806 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 2. uninit_member: Non-static class member internal_cfd_ is not initialized in this constructor nor in any functions that it calls. 80 : ColumnFamilyHandleImpl(nullptr, nullptr, nullptr) {} db/compacted_db_impl.cc: 18CompactedDBImpl::CompactedDBImpl( 19 const DBOptions& options, const std::string& dbname) 20 : DBImpl(options, dbname) { 2. uninit_member: Non-static class member cfd_ is not initialized in this constructor nor in any functions that it calls. 4. uninit_member: Non-static class member version_ is not initialized in this constructor nor in any functions that it calls. CID 1396120 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR) 6. uninit_member: Non-static class member user_comparator_ is not initialized in this constructor nor in any functions that it calls. 21} db/compaction_iterator.cc: 9. uninit_member: Non-static class member current_user_key_sequence_ is not initialized in this constructor nor in any functions that it calls. 11. uninit_member: Non-static class member current_user_key_snapshot_ is not initialized in this constructor nor in any functions that it calls. CID 1419855 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 13. uninit_member: Non-static class member current_key_committed_ is not initialized in this constructor nor in any functions that it calls. Closes https://github.com/facebook/rocksdb/pull/3084 Differential Revision: D6172999 Pulled By: sagar0 fbshipit-source-id: 084d73393faf8022c01359cfb445807b6a782460 27 October 2017, 18:26:42 UTC
47166ba Prashant D 27 October 2017, 04:00:17 UTC Fix coverity uninitialized fields warnings Pulled By: ajkr Differential Revision: D6170448 fbshipit-source-id: 5fd6d1608fc0df27c94d9f5059315ce7f79b8f5c 27 October 2017, 04:11:50 UTC
67b29e2 Prashant D 27 October 2017, 03:54:48 UTC Fix coverity issue for MutableDBOptions default constructor Summary: 228MutableDBOptions::MutableDBOptions() 229 : max_background_jobs(2), 230 base_background_compactions(-1), 231 max_background_compactions(-1), 232 avoid_flush_during_shutdown(false), 233 delayed_write_rate(2 * 1024U * 1024U), 234 max_total_wal_size(0), 235 delete_obsolete_files_period_micros(6ULL * 60 * 60 * 1000000), 236 stats_dump_period_sec(600), 2. uninit_member: Non-static class member bytes_per_sync is not initialized in this constructor nor in any functions that it calls. CID 1419857 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 4. uninit_member: Non-static class member wal_bytes_per_sync is not initialized in this constructor nor in any functions that it calls. 237 max_open_files(-1) {} Closes https://github.com/facebook/rocksdb/pull/3069 Differential Revision: D6170424 Pulled By: ajkr fbshipit-source-id: 1f94e86b87611ad2330b8b1707911150978d68b8 27 October 2017, 03:56:45 UTC
9566738 Andrew Kryczka 27 October 2017, 00:14:04 UTC implement lower bound for iterators Summary: - for `SeekToFirst()`, just convert it to a regular `Seek()` if lower bound is specified - for operations that iterate backwards over user keys (`SeekForPrev`, `SeekToLast`, `Prev`), change `PrevInternal` to check whether user key went below lower bound every time the user key changes -- same approach we use to ensure we stay within a prefix when `prefix_same_as_start=true`. Closes https://github.com/facebook/rocksdb/pull/3074 Differential Revision: D6158654 Pulled By: ajkr fbshipit-source-id: cb0e3a922e2650d2cd4d1c6e1c0f1e8b729ff518 27 October 2017, 00:27:42 UTC
5a2a648 Yi Wu 26 October 2017, 19:19:43 UTC Blob DB: Inline small values in base DB Summary: Adding the `min_blob_size` option to allow storing small values in base db (in LSM tree) together with the key. The goal is to improve performance for small values, while taking advantage of blob db's low write amplification for large values. Also adding expiration timestamp to blob index. It will be useful to evict stale blob indexes in base db by adding a compaction filter. I'll work on the compaction filter in future patches. See blob_index.h for the new blob index format. There are 4 cases when writing a new key: * small value w/o TTL: put in base db as normal value (i.e. ValueType::kTypeValue) * small value w/ TTL: put (type, expiration, value) to base db. * large value w/o TTL: write value to blob log and put (type, file, offset, size, compression) to base db. * large value w/TTL: write value to blob log and put (type, expiration, file, offset, size, compression) to base db. Closes https://github.com/facebook/rocksdb/pull/3066 Differential Revision: D6142115 Pulled By: yiwu-arbug fbshipit-source-id: 9526e76e19f0839310a3f5f2a43772a4ad182cd0 26 October 2017, 19:30:54 UTC
9b18cc2 Andrew Kryczka 25 October 2017, 23:24:29 UTC single-file bottom-level compaction when snapshot released Summary: When snapshots are held for a long time, files may reach the bottom level containing overwritten/deleted keys. We previously had no mechanism to trigger compaction on such files. This particularly impacted DBs that write to different parts of the keyspace over time, as such files would never be naturally compacted due to second-last level files moving down. This PR introduces a mechanism for bottommost files to be recompacted upon releasing all snapshots that prevent them from dropping their deleted/overwritten keys. - Changed `CompactionPicker` to compact files in `BottommostFilesMarkedForCompaction()`. These are the last choice when picking. Each file will be compacted alone and output to the same level in which it originated. The goal of this type of compaction is to rewrite the data excluding deleted/overwritten keys. - Changed `ReleaseSnapshot()` to recompute the bottom files marked for compaction when the oldest existing snapshot changes, and schedule a compaction if needed. We cache the value that oldest existing snapshot needs to exceed in order for another file to be marked in `bottommost_files_mark_threshold_`, which allows us to avoid recomputing marked files for most snapshot releases. - Changed `VersionStorageInfo` to track the list of bottommost files, which is recomputed every time the version changes by `UpdateBottommostFiles()`. The list of marked bottommost files is first computed in `ComputeBottommostFilesMarkedForCompaction()` when the version changes, but may also be recomputed when `ReleaseSnapshot()` is called. - Extracted core logic of `Compaction::IsBottommostLevel()` into `VersionStorageInfo::RangeMightExistAfterSortedRun()` since logic to check whether a file is bottommost is now necessary outside of compaction. Closes https://github.com/facebook/rocksdb/pull/3009 Differential Revision: D6062044 Pulled By: ajkr fbshipit-source-id: 123d201cf140715a7d5928e8b3cb4f9cd9f7ad21 25 October 2017, 23:30:37 UTC
96e3a60 Sagar Vemuri 25 October 2017, 23:18:15 UTC Return write error on reaching blob dir size limit Summary: I found that we continue accepting writes even when the blob db goes beyond the configured blob directory size limit. Now, we return an error for writes on reaching `blob_dir_size` limit and if `is_fifo` is set to false. (We cannot just drop any file when `is_fifo` is true.) Deleting the oldest file when `is_fifo` is true will be handled in a later PR. Closes https://github.com/facebook/rocksdb/pull/3060 Differential Revision: D6136156 Pulled By: sagar0 fbshipit-source-id: 2f11cb3f2eedfa94524fbfa2613dd64bfad7a23c 25 October 2017, 23:30:37 UTC
addfe1e Islam AbdelRahman 25 October 2017, 22:08:01 UTC Fix tombstone scans in SeekForPrev outside prefix Summary: When doing a Seek() or SeekForPrev() we should stop the moment we see a key with a different prefix as start if ReadOptions:: prefix_same_as_start was set to true Right now we don't stop if we encounter a tombstone outside the prefix while executing SeekForPrev() Closes https://github.com/facebook/rocksdb/pull/3067 Differential Revision: D6149638 Pulled By: IslamAbdelRahman fbshipit-source-id: 7f659862d2bf552d3c9104a360c79439ceba2f18 25 October 2017, 22:12:00 UTC
386a57e zach shipko 24 October 2017, 20:14:00 UTC Fix build on OpenBSD Summary: A few simple changes to allow RocksDB to be built on OpenBSD. Let me know if any further changes are needed. Closes https://github.com/facebook/rocksdb/pull/3061 Differential Revision: D6138800 Pulled By: ajkr fbshipit-source-id: a13a17b5dc051e6518bd56a8c5efd1d24dd81b0c 24 October 2017, 20:27:38 UTC
57fcdc2 zawlazaw 23 October 2017, 23:34:47 UTC added missing subcodes and improved error message for missing enum values Summary: Java's `Status.SubCode` was out of sync with `include/rocksdb/status.h:SubCode`. When running out of disc space this led to an `IllegalArgumentException` because of an invalid status code, rather than just returning the corresponding status code without an exception. I added the missing status codes. By this, we keep the behaviour of throwing an `IllegalArgumentException` in case of newly added status codes that are defined in C but not in Java. We could think of an alternative strategy: add in Java another code "UnknownCode" which acts as a catch-all for all those status codes that are not yet mirrored from C to Java. This approach would never throw an exception but simply return a non-OK status-code. I think the current approach of throwing an Exception in case of a C/Java inconsistency is fine, but if you have some opinion on the alternative strategy, then feel free to comment here. Closes https://github.com/facebook/rocksdb/pull/3050 Differential Revision: D6129682 Pulled By: sagar0 fbshipit-source-id: f2bf44caad650837cffdcb1f93eb793b43580c66 23 October 2017, 23:42:07 UTC
66a2c44 Yi Wu 23 October 2017, 22:22:05 UTC Add DB::Properties::kEstimateOldestKeyTime Summary: With FIFO compaction we would like to get the oldest data time for monitoring. The problem is we don't have timestamp for each key in the DB. As an approximation, we expose the earliest of sst file "creation_time" property. My plan is to override the property with a more accurate value with blob db, where we actually have timestamp. Closes https://github.com/facebook/rocksdb/pull/2842 Differential Revision: D5770600 Pulled By: yiwu-arbug fbshipit-source-id: 03833c8f10bbfbee62f8ea5c0d03c0cafb5d853a 23 October 2017, 22:27:27 UTC
d2a65c5 Dmitri Smirnov 23 October 2017, 21:20:53 UTC Fix unused var warnings in Release mode Summary: MSVC does not support unused attribute at this time. A separate assignment line fixes the issue probably by being counted as usage for MSVC and it no longer complains about unused var. Closes https://github.com/facebook/rocksdb/pull/3048 Differential Revision: D6126272 Pulled By: maysamyabandeh fbshipit-source-id: 4907865db45fd75a39a15725c0695aaa17509c1f 23 October 2017, 21:27:04 UTC
63822eb Maysam Yabandeh 23 October 2017, 21:20:22 UTC Enable two write queues for transactions Summary: Enable concurrent_prepare flag for WritePrepared transactions and extend the existing transaction tests with this config. Closes https://github.com/facebook/rocksdb/pull/3046 Differential Revision: D6106534 Pulled By: maysamyabandeh fbshipit-source-id: 88c8d21d45bc492beb0a131caea84a2ac5e7d38c 23 October 2017, 21:27:04 UTC
a02ed12 Sagar Vemuri 21 October 2017, 00:09:46 UTC Exclude DBTest.DynamicFIFOCompactionOptions test under RocksDB Lite Summary: This test shouldn't be enabled under the lite version; and this fixes the failing contrun test due to #3006. Closes https://github.com/facebook/rocksdb/pull/3056 Differential Revision: D6114681 Pulled By: sagar0 fbshipit-source-id: dc5243549ae6b1353cec7edb820c771d95f66dda 21 October 2017, 00:11:39 UTC
3ef55d2 Maysam Yabandeh 20 October 2017, 22:34:23 UTC Split CompactionFilterWithValueChange Summary: The test currently times out when it is run under tsan. This patch split it into 4 tests. Closes https://github.com/facebook/rocksdb/pull/3047 Differential Revision: D6106515 Pulled By: maysamyabandeh fbshipit-source-id: 03a28cdf8b1c097be2361b1b0cc3dc1acf2b5d63 20 October 2017, 22:42:07 UTC
d75793d Andrew Kryczka 20 October 2017, 22:17:03 UTC db_stress support long-held snapshots Summary: Add options to `db_stress` (correctness testing tool) to randomly acquire snapshot and release it after some period of time. It's useful for correctness testing of #3009, as well as other parts of compaction that behave differently depending on which snapshots are held. Closes https://github.com/facebook/rocksdb/pull/3038 Differential Revision: D6086501 Pulled By: ajkr fbshipit-source-id: 3ec0d8666c78ac507f1f808887c4ff759ba9b865 20 October 2017, 22:26:59 UTC
f8b5bb2 Andrew Kryczka 20 October 2017, 21:07:42 UTC remove unused code Summary: fixup 6a541afcc4d1e5b6e6d78e288b9bee3bb2a933b5. This code didn't do anything because (1) `bytes_per_sync` is assigned in `EnvOptions`'s constructor; and (2) `OptimizeForCompactionTableWrite`'s return value was ignored, even though its only purpose is to return something. Closes https://github.com/facebook/rocksdb/pull/3055 Differential Revision: D6114132 Pulled By: ajkr fbshipit-source-id: ea4831770930e9cf83518e13eb2e1934d1f5487c 20 October 2017, 21:11:52 UTC
57fd4a8 Andrew Kryczka 20 October 2017, 20:49:36 UTC update HISTORY with recent changes Summary: We should mention these: - `EventListener::OnStallConditionsChanged()` in 01542400a87ad130aed790bf895c029082745cbe - `DeleteRange()` fix in 966b32b57c673774ea677fe3a40d255e6c8916ba Closes https://github.com/facebook/rocksdb/pull/3054 Differential Revision: D6113989 Pulled By: ajkr fbshipit-source-id: d5e058e1ab07570df22936e8d5939fb30fb4d381 20 October 2017, 20:56:49 UTC
ee2b1ec raistlin 20 October 2017, 17:04:28 UTC Fix unstable floating point exception Summary: Fix unstable floating point exception, tested on Windows, 64-bit build. The problem appeared in `SetCapacity()` method at line `high_pri_pool_capacity_ = capacity_ * high_pri_pool_ratio_;` `high_pri_pool_ratio_` was not initialized at that moment, because `SetHighPriorityPoolRatio()` is called after `SetCapacity()`. So, `high_pri_pool_ratio_` contained garbage, which caused "Floating point exception" sometimes. Closes https://github.com/facebook/rocksdb/pull/3052 Differential Revision: D6111161 Pulled By: yiwu-arbug fbshipit-source-id: d170329111ad12b4bf9bbcf37bcb6411523438ae 20 October 2017, 17:12:49 UTC
f0804db Sagar Vemuri 19 October 2017, 22:19:20 UTC Make FIFO compaction options dynamically configurable Summary: ColumnFamilyOptions::compaction_options_fifo and all its sub-fields can be set dynamically now. Some of the ways in which the fifo compaction options can be set are: - `SetOptions({{"compaction_options_fifo", "{max_table_files_size=1024}"}})` - `SetOptions({{"compaction_options_fifo", "{ttl=600;}"}})` - `SetOptions({{"compaction_options_fifo", "{max_table_files_size=1024;ttl=600;}"}})` - `SetOptions({{"compaction_options_fifo", "{max_table_files_size=51;ttl=49;allow_compaction=true;}"}})` Most of the code has been made generic enough so that it could be reused later to make universal options (and other such nested defined-types) dynamic with very few lines of parsing/serializing code changes. Introduced a few new functions like `ParseStruct`, `SerializeStruct` and `GetStringFromStruct`. The duplicate code in `GetStringFromDBOptions` and `GetStringFromColumnFamilyOptions` has been moved into `GetStringFromStruct`. So they become just simple wrappers now. Closes https://github.com/facebook/rocksdb/pull/3006 Differential Revision: D6058619 Pulled By: sagar0 fbshipit-source-id: 1e8f78b3374ca5249bb4f3be8a6d3bb4cbc52f92 19 October 2017, 22:26:36 UTC
ebab2e2 Dmitri Smirnov 19 October 2017, 17:48:47 UTC Enable MSVC W4 with a few exceptions. Fix warnings and bugs Summary: Closes https://github.com/facebook/rocksdb/pull/3018 Differential Revision: D6079011 Pulled By: yiwu-arbug fbshipit-source-id: 988a721e7e7617967859dba71d660fc69f4dff57 19 October 2017, 17:57:12 UTC
b749994 Sagar Vemuri 18 October 2017, 21:30:44 UTC Update RocksDB Authors File Summary: Update RocksDB Authors File. Reviewed By: yiwu-arbug Differential Revision: D6075453 fbshipit-source-id: dff52f483aab33c41de391f145a8273acfd6cbde 18 October 2017, 21:42:10 UTC
7deed2b Gihwan Oh 18 October 2017, 19:18:43 UTC Fix a typo in a comment Summary: instad of for specific level -> instead of a specific level Closes https://github.com/facebook/rocksdb/pull/3040 Differential Revision: D6090811 Pulled By: sagar0 fbshipit-source-id: 499edef0a6f596c448f61791e6aca8f5cce08e9c 18 October 2017, 19:32:28 UTC
7e38238 Maysam Yabandeh 18 October 2017, 16:09:31 UTC WritePrepared Txn: Disable GC during recovery Summary: Disables GC during recovery of a WritePrepared txn db to avoid GCing uncommitted key values. Closes https://github.com/facebook/rocksdb/pull/2980 Differential Revision: D6000191 Pulled By: maysamyabandeh fbshipit-source-id: fc4d522c643d24ebf043f811fe4ecd0dd0294675 18 October 2017, 16:11:50 UTC
7891af8 Nikhil Benesch 18 October 2017, 05:09:01 UTC expose a hook to skip tables during iteration Summary: As discussed on the mailing list (["Skipping entire SSTs while iterating"](https://groups.google.com/forum/#!topic/rocksdb/ujHCJVLrHlU)), this patch adds a `table_filter` to `ReadOptions` that allows specifying a callback to be executed during iteration before each table in the database is scanned. The callback is passed the table's properties; the table is scanned iff the callback returns true. This can be used in conjunction with a `TablePropertiesCollector` to dramatically speed up scans by skipping tables that are known to contain irrelevant data for the scan at hand. We're using this [downstream in CockroachDB](https://github.com/cockroachdb/cockroach/blob/master/pkg/storage/engine/db.cc#L2009-L2022) already. With this feature, under ideal conditions, we can reduce the time of an incremental backup in from hours to seconds. FYI, the first commit in this PR fixes a segfault that I unfortunately have not figured out how to reproduce outside of CockroachDB. I'm hoping you accept it on the grounds that it is not correct to return 8-byte aligned memory from a call to `malloc` on some 64-bit platforms; one correct approach is to infer the necessary alignment from `std::max_align_t`, as done here. As noted in the first commit message, the bug is tickled by having a`std::function` in `struct ReadOptions`. That is, the following patch alone is enough to cause RocksDB to segfault when run from CockroachDB on Darwin. ```diff --- a/include/rocksdb/options.h +++ b/include/rocksdb/options.h @@ -1546,6 +1546,13 @@ struct ReadOptions { // Default: false bool ignore_range_deletions; + // A callback to determine whether relevant keys for this scan exist in a + // given table based on the table's properties. The callback is passed the + // properties of each table during iteration. If the callback returns false, + // the table will not be scanned. + // Default: empty (every table will be scanned) + std::function<bool(const TableProperties&)> table_filter; + ReadOptions(); ReadOptions(bool cksum, bool cache); }; ``` /cc danhhz Closes https://github.com/facebook/rocksdb/pull/2265 Differential Revision: D5054262 Pulled By: yiwu-arbug fbshipit-source-id: dd6b28f2bba6cb8466250d8c5c542d3c92785476 18 October 2017, 05:12:00 UTC
eaaef91 Yi Wu 18 October 2017, 00:24:25 UTC Blob DB: Store blob index as kTypeBlobIndex in base db Summary: Blob db insert blob index to base db as kTypeBlobIndex type, to tell apart values written by plain rocksdb or blob db. This is to make it possible to migrate from existing rocksdb to blob db. Also with the patch blob db garbage collection get away from OptimisticTransaction. Instead it use a custom write callback to achieve similar behavior as OptimisticTransaction. This is because we need to pass the is_blob_index flag to DBImpl::Get but OptimisticTransaction don't support it. Closes https://github.com/facebook/rocksdb/pull/3000 Differential Revision: D6050044 Pulled By: yiwu-arbug fbshipit-source-id: 61dc72ab9977625e75f78cd968e7d8a3976e3632 18 October 2017, 00:28:11 UTC
0552029 Yi Wu 17 October 2017, 19:11:52 UTC Blob DB: not writing sequence number as blob record footer Summary: Previously each time we write a blob we write blog_record_header + key + value + blob_record_footer to blob log. The footer only contains a sequence and a crc for the sequence number. The sequence number was used in garbage collection to verify the value is recent. After #2703 we moved to use optimistic transaction and no longer use sequence number from the footer. Remove the footer altogether. There's another usage of sequence number and we are keeping it: Each blob log file keep track of sequence number range of keys in it, and use it to check if it is reference by a snapshot, before being deleted. Closes https://github.com/facebook/rocksdb/pull/3005 Differential Revision: D6057585 Pulled By: yiwu-arbug fbshipit-source-id: d6da53c457a316e9723f359a1b47facfc3ffe090 17 October 2017, 19:13:08 UTC
966b32b zhangjinpeng1987 17 October 2017, 18:09:29 UTC fix delete range bug Summary: Fix this [issue](https://github.com/facebook/rocksdb/issues/2989). ajkr PTAL Close #2989 Closes https://github.com/facebook/rocksdb/pull/3017 Differential Revision: D6078541 Pulled By: yiwu-arbug fbshipit-source-id: ef3db87b37b9156f83ca468aa39dea1f6dbde49d 17 October 2017, 18:13:19 UTC
c0208df Nikhil Benesch 17 October 2017, 18:07:35 UTC arena: derive alignment unit from std::max_align_t Summary: As raised in #2265, the arena allocator will return memory that is improperly aligned to store a `std::function` on macOS. Oddly, I'm unable to tickle this bug without adding a `std::function` field to `struct ReadOptions`—but my proposal in #2265 does exactly that. In any case, here's a simple reproduction. Apply this bogus patch to get a `std::function` into `struct ReadOptions` ``` --- a/include/rocksdb/options.h +++ b/include/rocksdb/options.h @@ -1035,6 +1035,8 @@ struct ReadOptions { // Default: 0 uint64_t max_skippable_internal_keys; + std::function<void()> foo; + ReadOptions(); ReadOptions(bool cksum, bool cache); }; ``` then compile `db_properties_test` *with ubsan* and run `ReadLatencyHistogramByLevel`: ``` $ make COMPILE_WITH_UBSAN=1 db_properties_test $ ./db_properties_test --gtest_filter=DBPropertiesTest.ReadLatencyHistogramByLevel ``` ubsan will complain about several misaligned accesses: ``` Note: Google Test filter = DBPropertiesTest.ReadLatencyHistogramByLevel [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from DBPropertiesTest [ RUN ] DBPropertiesTest.ReadLatencyHistogramByLevel util/coding.h:372:12: runtime error: load of misaligned address 0x00010d85516c for type 'const unsigned long', which requires 8 byte alignment 0x00010d85516c: note: pointer points here 01 00 34 57 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 78 24 82 0a 01 00 00 00 ^ util/coding.h:362:3: runtime error: store to misaligned address 0x7fff5733fac4 for type 'unsigned long', which requires 8 byte alignment 0x7fff5733fac4: note: pointer points here 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 1d 96 0d 01 00 00 00 ^ util/coding.h:372:12: runtime error: load of misaligned address 0x00010d85516c for type 'const unsigned long', which requires 8 byte alignment 0x00010d85516c: note: pointer points here 01 00 34 57 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 78 24 82 0a 01 00 00 00 ^ version_set.cc:854: runtime error: constructor call on misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ version_set.cc:512: runtime error: constructor call on misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ version_set.cc:505: runtime error: constructor call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: constructor call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: constructor call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ functional:1583: runtime error: constructor call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1585:9: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1585:9: runtime error: store to misaligned address 0x00010dbfa648 for type '__base *' (aka '__base<void ()> *'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:864:29: runtime error: upcast of misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:521:12: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:521:12: runtime error: load of misaligned address 0x00010dbfa5d8 for type 'rocksdb::TableCache *', which requires 16 byte alignment 0x00010dbfa5d8: note: pointer points here 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 ^ db/version_set.cc:522:9: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:9: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:522:24: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:38: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:57: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:522:57: runtime error: load of misaligned address 0x00010dbfa678 for type 'rocksdb::RangeDelAggregator *', which requires 16 byte alignment 0x00010dbfa678: note: pointer points here 01 00 00 00 d0 a1 bf 0d 01 00 00 00 00 00 00 00 00 00 00 00 f8 db 70 0a 01 00 00 00 00 00 00 00 ^ db/version_set.cc:523:54: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:523:54: runtime error: load of misaligned address 0x00010dbfa668 for type 'rocksdb::HistogramImpl *', which requires 16 byte alignment 0x00010dbfa668: note: pointer points here 01 00 00 00 c8 88 a5 0d 01 00 00 00 00 00 00 00 01 00 00 00 d0 a1 bf 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:524:9: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:524:47: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:524:62: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/table_cache.cc:228:33: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ table/block_based_table_reader.cc:1554:41: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ table/block_based_table_reader.cc:1396:21: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ include/rocksdb/options.h:931:8: runtime error: reference binding to misaligned address 0x00010dbfa628 for type 'const std::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1584:13: runtime error: load of misaligned address 0x00010dbfa648 for type '__base *const' (aka '__base<void ()> *const'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c8 a5 97 0d 01 00 00 00 38 36 9b 0d ^ table/block_based_table_reader.cc:1555:24: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/table_cache.cc:244:54: runtime error: load of misaligned address 0x00010dbfa618 for type 'const bool', which requires 16 byte alignment 0x00010dbfa618: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/table_cache.cc:246:49: runtime error: reference binding to misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:532:12: runtime error: member access within misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ db/version_set.cc:532:12: runtime error: member access within misaligned address 0x00010dbfa5e8 for type 'const rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ db/version_set.cc:532:26: runtime error: load of misaligned address 0x00010dbfa5f8 for type 'const rocksdb::Slice *const', which requires 16 byte alignment 0x00010dbfa5f8: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ version_set.cc:493: runtime error: member call on misaligned address 0x00010dbfa5c8 for type 'rocksdb::(anonymous namespace)::LevelFileIteratorState', which requires 16 byte alignment 0x00010dbfa5c8: note: pointer points here 00 00 00 00 a0 db 70 0a 01 00 00 00 00 00 00 00 00 00 00 00 90 14 98 0d 01 00 00 00 00 00 00 00 ^ version_set.cc:493: runtime error: member call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: member call on misaligned address 0x00010dbfa5e8 for type 'rocksdb::ReadOptions', which requires 16 byte alignment 0x00010dbfa5e8: note: pointer points here 00 00 00 00 01 01 ff ff ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ options.h:931: runtime error: member call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ functional:1765: runtime error: member call on misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1766:9: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1766:9: runtime error: load of misaligned address 0x00010dbfa648 for type '__base *' (aka '__base<void ()> *'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c8 a5 97 0d 01 00 00 00 38 36 9b 0d ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1766:27: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1768:14: runtime error: member access within misaligned address 0x00010dbfa628 for type 'std::__1::function<void ()>', which requires 16 byte alignment 0x00010dbfa628: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ^ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/functional:1768:14: runtime error: load of misaligned address 0x00010dbfa648 for type '__base *' (aka '__base<void ()> *'), which requires 16 byte alignment 0x00010dbfa648: note: pointer points here 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c8 a5 97 0d 01 00 00 00 38 36 9b 0d ^ [ OK ] DBPropertiesTest.ReadLatencyHistogramByLevel (1599 ms) [----------] 1 test from DBPropertiesTest (1599 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1599 ms total) [ PASSED ] 1 test. ``` So it seems the root cause is that the internal implementation of `std::function` on macOS (and perhaps with libc++ generally?) requires 16-byte aligned memory, but the arena allocator only guarantees that the returned memory will be `sizeof(void*)` aligned, which is only 8-byte alignment on my machine. This patch solves the problem by adjusting the allocator to derive the necessary alignment from `alignof(std::max_align_t)`, which is properly 16 bytes on my machine. As I mentioned in #2265, none of RocksDB's tests will cause this unaligned access to actually abort the process, but, on macOS, linking CockroachDB against a version of RocksDB with the above patch and letting it run for just a few seconds will cause a SIGABRT. ``` Process 19792 stopped * thread #2, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) frame #0: 0x0000000004f5e78f cockroach`DBNewIter + 95 cockroach`DBNewIter: -> 0x4f5e78f <+95>: callq *0x28(%rax) 0x4f5e792 <+98>: jmp 0x4f5e79e ; <+110> 0x4f5e794 <+100>: movq -0x50(%rbp), %rcx 0x4f5e798 <+104>: movq %rax, %rdi (lldb) bt * thread #2, stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) * frame #0: 0x0000000004f5e78f cockroach`DBNewIter + 95 ``` I'd get you a backtrace, but [Go doesn't include cgo debug information on macOS](https://github.com/golang/go/issues/6942). I've also tried building against libc++ on Linux, where debug information would be available, but I can't seem to trigger the bug there. In any case, this PR both fixes the segfault in CockroachDB and fixes the warnings reported by ubsan. Closes https://github.com/facebook/rocksdb/pull/2347 Differential Revision: D5108596 Pulled By: yiwu-arbug fbshipit-source-id: bd5e4323b2ce915ed4fe78e123cb8996aec75a00 17 October 2017, 18:13:19 UTC
b8cea7c Changli Gao 17 October 2017, 16:58:24 UTC VersionBuilder: Erase with iterators for better performance Summary: Closes https://github.com/facebook/rocksdb/pull/3007 Differential Revision: D6077701 Pulled By: yiwu-arbug fbshipit-source-id: a6fd5b8a23f4feb1660b9ce027f651a7e90352b3 17 October 2017, 17:12:37 UTC
f7843f3 codeeply 17 October 2017, 16:50:16 UTC Move ~Comparator define to comparator.h Summary: When I impl my own comparator, and build in release mode. The following compile error occurs. undefined reference to `typeinfo for rocksdb::Comparator' This fix allows users build with RTTI off when has their own comparator. Closes https://github.com/facebook/rocksdb/pull/3008 Differential Revision: D6077354 Pulled By: yiwu-arbug fbshipit-source-id: 914c26dbab72f0ad1f0e15f8666a3fb2f10bfed8 17 October 2017, 16:58:13 UTC
8e63cad Yi Wu 17 October 2017, 15:49:11 UTC fix lite build Summary: * make `checksum_type_string_map` available for lite * comment out `FilesPerLevel` in lite mode. * travis and legocastle lite build also build `all` target and run tests Closes https://github.com/facebook/rocksdb/pull/3015 Differential Revision: D6069822 Pulled By: yiwu-arbug fbshipit-source-id: 9fe92ac220e711e9e6ed4e921bd25ef4314796a0 17 October 2017, 15:57:09 UTC
7318952 Andrew Kryczka 17 October 2017, 01:29:50 UTC db_bench randomtransaction print throughput Summary: print throughput in MB/s upon finishing randomtransaction benchmark Closes https://github.com/facebook/rocksdb/pull/3016 Differential Revision: D6070426 Pulled By: ajkr fbshipit-source-id: 69df43beed4c374a36d826e761ca3a83e1fdcbf5 17 October 2017, 01:42:25 UTC
60c09f5 Yi Wu 13 October 2017, 21:41:07 UTC print more table_options to info log Summary: print more table_options to info log Closes https://github.com/facebook/rocksdb/pull/3003 Differential Revision: D6054490 Pulled By: yiwu-arbug fbshipit-source-id: 8e6f96e08bdc906077b6c62ade419d7cb739110f 13 October 2017, 21:42:26 UTC
10ba50e Yi Wu 13 October 2017, 21:36:36 UTC Blob DB: Move BlobFile definition to a separate file Summary: simply move BlobFile definition from blob_db_impl.h to blob_file.h. Closes https://github.com/facebook/rocksdb/pull/3002 Differential Revision: D6050143 Pulled By: yiwu-arbug fbshipit-source-id: a8fb6e094fe39bdeace6279569834bc65aa64a34 13 October 2017, 21:42:26 UTC
31d3e41 Yi Wu 13 October 2017, 01:19:10 UTC PinnableSlice move assignment Summary: Allow `std::move(pinnable_slice)`. Closes https://github.com/facebook/rocksdb/pull/2997 Differential Revision: D6036782 Pulled By: yiwu-arbug fbshipit-source-id: 583fb0419a97e437ff530f4305822341cd3381fa 13 October 2017, 01:28:24 UTC
cc67b22 Sagar Vemuri 12 October 2017, 23:45:01 UTC Add OptionsUtil class to java/CMakeLists.txt Summary: Adding OptionsUtil java class and options_util.cc to java/CMakeLists.txt, which were missed accidentally when they were introduced in #2898. Closes https://github.com/facebook/rocksdb/pull/2985 Differential Revision: D6015878 Pulled By: sagar0 fbshipit-source-id: 1abbd46db4aebad1e07ea53523eacbdcb12823e1 12 October 2017, 23:57:05 UTC
560e984 Adam Retter 12 October 2017, 18:06:51 UTC Added CompactionFilterFactory support to RocksJava Summary: This PR also includes some cleanup, bugfixes and refactoring of the Java API. However these are really pre-cursors on the road to CompactionFilterFactory support. Closes https://github.com/facebook/rocksdb/pull/1241 Differential Revision: D6012778 Pulled By: sagar0 fbshipit-source-id: 0774465940ee99001a78906e4fed4ef57068ad5c 12 October 2017, 18:12:16 UTC
8dd0a7e Andrew Kryczka 11 October 2017, 22:11:08 UTC add comment in SuperVersion referencing logic Summary: The referencing logic is super confusing so added a comment at the part that took me longest to figure out. Closes https://github.com/facebook/rocksdb/pull/2996 Differential Revision: D6034969 Pulled By: ajkr fbshipit-source-id: 9cc2e744c1f79d6d57d378f86ed59238a5f583db 11 October 2017, 22:12:31 UTC
fb4ae4d Yi Wu 11 October 2017, 21:48:28 UTC fix DBImpl::NewInternalIterator super-version leak on failure Summary: Close #2955 Closes https://github.com/facebook/rocksdb/pull/2960 Differential Revision: D5962872 Pulled By: yiwu-arbug fbshipit-source-id: a6472d5c015bea3dc476c572ff5a5c90259e6059 11 October 2017, 21:57:43 UTC
019aa70 Kefu Chai 11 October 2017, 19:14:53 UTC cmake: pass "-msse4.2" to when building crc32c.cc if HAVE_SSE42 Summary: it turns out that, with older GCC shipped from centos7, the SSE42 intrinsics are not available even with "target" specified. so we need to pass "-msse42" for checking compiler's sse4.2 support and for building crc32c.cc which uses sse4.2 intrinsics for crc32. Signed-off-by: Kefu Chai <tchaikov@gmail.com> Closes https://github.com/facebook/rocksdb/pull/2950 Differential Revision: D6032298 Pulled By: siying fbshipit-source-id: 124c946321043661b3fb0a70b6cdf4c9c5126ab4 11 October 2017, 19:26:46 UTC
back to top