sort by:
Revision Author Date Message Commit Date
22e5c51 Add zstd_max_train_bytes to c interop (#6796) Summary: Added setting of zstd_max_train_bytes compression option parameter to c interop. rocksdb_options_set_bottommost_compression_options was using bool parameter and thus not exported, updated it to unsigned char and added to c.h as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6796 Reviewed By: cheng-chang Differential Revision: D21611471 Pulled By: ajkr fbshipit-source-id: caaaf153de934837ad9af283c7f8c025ff0b0cf5 03 June 2020, 19:27:12 UTC
0a17d95 Add OptionTypeInfo::Vector to parse/serialize vectors (#6424) Summary: The OptionTypeInfo::Vector method allows a vector<T> to be converted to/from strings via the options. The kVectorInt and kVectorCompressionType vectors were replaced with this methodology. As part of this change, the NextToken method was added to the OptionTypeInfo. This method was refactored from code within the StringToMap function. Future types that could use this functionality include the EventListener vectors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6424 Reviewed By: cheng-chang Differential Revision: D21832368 Pulled By: zhichao-cao fbshipit-source-id: e1ca766faff139d54e6e8407a9ec09ece6517439 03 June 2020, 19:23:07 UTC
172adce Posix threads (#6865) Summary: Rocksdb is using the c++11 std::threads feature. The issue is that MINGW only supports it when using Posix threads. This change will allow rocksdb::port::WindowsThread to be replaced with std::thread, which in turn will allow Rocksdb to be cross compiled using MINGW. At the same time, we'll have to use GetCurrentProcessId instead of _getpid. Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6865 Reviewed By: cheng-chang Differential Revision: D21864285 Pulled By: ajkr fbshipit-source-id: 0982eed313e7d34d351b1364c1ccc722da473205 03 June 2020, 19:01:57 UTC
43f8a9d Some fixes for gcc 4.8 and add to Travis (#6915) Summary: People keep breaking the gcc 4.8 compilation due to different warnings for shadowing member functions with locals. Adding to Travis to keep compatibility. (gcc 4.8 is default on CentOS 7.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6915 Test Plan: local and Travis Reviewed By: siying Differential Revision: D21842894 Pulled By: pdillinger fbshipit-source-id: bdcd4385127ee5d1cc222d87e53fb3695c32a9d4 03 June 2020, 18:39:25 UTC
78e291b Improve consistency checks in VersionBuilder (#6901) Summary: The patch cleans up the code and improves the consistency checks around adding/deleting table files in `VersionBuilder`. Namely, it makes the checks stricter and improves them in the following ways: 1) A table file can now only be deleted from the LSM tree using the level it resides on. Earlier, there was some unnecessary wiggle room for trivially moved files (they could be deleted using a lower level number than the actual one). 2) A table file cannot be added to the tree if it is already present in the tree on any level (not just the target level). The earlier code only had an assertion (which is a no-op in release builds) that the newly added file is not already present on the target level. 3) The above consistency checks around state transitions are now mandatory, as opposed to the earlier `CheckConsistencyForDeletes`, which was a no-op in release mode unless `force_consistency_checks` was set to `true`. The rationale here is that assuming that the initial state is consistent, a valid transition leads to a next state that is also consistent; however, an *invalid* transition offers no such guarantee. Hence it makes sense to validate the transitions unconditionally, and save `force_consistency_checks` for the paranoid checks that re-validate the entire state. 4) The new checks build on the mechanism introduced in https://github.com/facebook/rocksdb/pull/6862, which enables us to efficiently look up the location (level and position within level) of files in a `Version` by file number. This makes the consistency checks much more efficient than the earlier `CheckConsistencyForDeletes`, which essentially performed a linear search. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6901 Test Plan: Extended the unit tests and ran: `make check` `make whitebox_crash_test` Reviewed By: ajkr Differential Revision: D21822714 Pulled By: ltamasi fbshipit-source-id: e2b29c8b6da1bf0f59004acc889e4870b2d18215 03 June 2020, 18:24:55 UTC
9360776 Fix handling of too-small filter partition size (#6905) Summary: Because ARM and some other platforms have a larger cache line size, they have a larger minimum filter size, which causes recently added PartitionedMultiGet test in db_bloom_filter_test to fail on those platforms. The code would actually end up using larger partitions, because keys_per_partition_ would be 0 and never == number of keys added. The code now attempts to get as close as possible to the small target size, while fully utilizing that filter size, if the target partition size is smaller than the minimum filter size. Also updated the test to break more uniformly across platforms Pull Request resolved: https://github.com/facebook/rocksdb/pull/6905 Test Plan: updated test, tested on ARM Reviewed By: anand1976 Differential Revision: D21840639 Pulled By: pdillinger fbshipit-source-id: 11684b6d35f43d2e98b85ddb2c8dcfd59d670817 03 June 2020, 17:43:01 UTC
2adb7e3 Fix potential overflow of unsigned type in for loop (#6902) Summary: x.size() -1 or y - 1 can overflow to an extremely large value when x.size() pr y is 0 when they are unsigned type. The end condition of i in the for loop will be extremely large, potentially causes segment fault. Fix them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6902 Test Plan: pass make asan_check Reviewed By: ajkr Differential Revision: D21843767 Pulled By: zhichao-cao fbshipit-source-id: 5b8b88155ac5a93d86246d832e89905a783bb5a1 02 June 2020, 22:05:07 UTC
556972e Replace Status with IOStatus in CopyFile and CreateFile (#6916) Summary: Replace Status with IOStatus in CopyFile and CreateFile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6916 Test Plan: pass make asan_check Reviewed By: cheng-chang Differential Revision: D21843775 Pulled By: zhichao-cao fbshipit-source-id: 524d4a0fcf47f0941b923da0346e0de71607f5f6 02 June 2020, 22:05:06 UTC
bfc9737 Remove gtest dependency in non-test code under utilities/cassandra (#6908) Summary: production code under utilities/cassandra depends on gtest.h. Remove them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6908 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21842606 fbshipit-source-id: a098e0b49c9aeac51cc90a79562ad9897a36122c 02 June 2020, 20:56:29 UTC
38f988d Expose rocksdb_options_copy function to the C API (#6880) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6880 Reviewed By: ajkr Differential Revision: D21842752 Pulled By: pdillinger fbshipit-source-id: eda326f551ddd9cb397681544b9e9799ea614e52 02 June 2020, 20:48:51 UTC
14eca6b For ApproximateSizes, pro-rate table metadata size over data blocks (#6784) Summary: The implementation of GetApproximateSizes was inconsistent in its treatment of the size of non-data blocks of SST files, sometimes including and sometimes now. This was at its worst with large portion of table file used by filters and querying a small range that crossed a table boundary: the size estimate would include large filter size. It's conceivable that someone might want only to know the size in terms of data blocks, but I believe that's unlikely enough to ignore for now. Similarly, there's no evidence the internal function AppoximateOffsetOf is used for anything other than a one-sided ApproximateSize, so I intend to refactor to remove redundancy in a follow-up commit. So to fix this, GetApproximateSizes (and implementation details ApproximateSize and ApproximateOffsetOf) now consistently include in their returned sizes a portion of table file metadata (incl filters and indexes) based on the size portion of the data blocks in range. In other words, if a key range covers data blocks that are X% by size of all the table's data blocks, returned approximate size is X% of the total file size. It would technically be more accurate to attribute metadata based on number of keys, but that's not computationally efficient with data available and rarely a meaningful difference. Also includes miscellaneous comment improvements / clarifications. Also included is a new approximatesizerandom benchmark for db_bench. No significant performance difference seen with this change, whether ~700 ops/sec with cache_index_and_filter_blocks and small cache or ~150k ops/sec without cache_index_and_filter_blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6784 Test Plan: Test added to DBTest.ApproximateSizesFilesWithErrorMargin. Old code running new test... [ RUN ] DBTest.ApproximateSizesFilesWithErrorMargin db/db_test.cc:1562: Failure Expected: (size) <= (11 * 100), actual: 9478 vs 1100 Other tests updated to reflect consistent accounting of metadata. Reviewed By: siying Differential Revision: D21334706 Pulled By: pdillinger fbshipit-source-id: 6f86870e45213334fedbe9c73b4ebb1d8d611185 02 June 2020, 19:30:23 UTC
298b00a Reduce dependency on gtest dependency in release code (#6907) Summary: Release code now depends on gtest, indirectly through including "test_util/testharness.h". This creates multiple problems. One important reason is the definition of IGNORE_STATUS_IF_ERROR() in test_util/testharness.h. Move it to sync_point.h instead. Note that utilities/cassandra/format.h still depends on "test_util/testharness.h". This will be resolved in a separate diff. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6907 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21829884 fbshipit-source-id: 9253c19ffde2936f3ae68998210f8e54f645a6e6 02 June 2020, 19:11:24 UTC
8d87e9c Update googletest from 1.8.1 to 1.10.0 (#6808) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6808 Reviewed By: anand1976 Differential Revision: D21483984 Pulled By: pdillinger fbshipit-source-id: 70c5eff2bd54ddba469761d95e4cd4611fb8e598 02 June 2020, 03:33:42 UTC
66942e8 Avoid unnecessary reads of uncompression dictionary in MultiGet (#6906) Summary: We may sometimes read the uncompression dictionary when its not necessary, when we lookup a key in an SST file but the index indicates the key is not present. This can happen with index_type 3. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6906 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D21828944 Pulled By: anand1976 fbshipit-source-id: 7aef4f0a39548d0874eafefd2687006d2652f9bb 02 June 2020, 02:43:37 UTC
02f59ed Find the correct gcov (#6904) Summary: Right now in FB environment, wrong gcov is used. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6904 Test Plan: "make coverage" and watch results. Reviewed By: riversand963 Differential Revision: D21824291 fbshipit-source-id: 666011fd86c36adafa09ebd9eb97742f94fb90bb 01 June 2020, 23:33:05 UTC
bcb9e41 Explicitly free allocated buffer when status is not ok (#6903) Summary: Currently we rely on `BlockContents` to implicitly free the allocated scratch buffer, but when IO error happens, it doesn't make sense to construct the `BlockContents` which might be corrupted. In the stress test, we find that `assert(req.result.size() == block_size(handle));` fails because of potential IO errors. In this PR, we explicitly free the scratch buffer on error without constructing `BlockContents`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6903 Test Plan: watch stress test Reviewed By: anand1976 Differential Revision: D21823869 Pulled By: cheng-chang fbshipit-source-id: 5603fc80e9bf3f44a9d7250ddebd871afe1eb89f 01 June 2020, 22:19:40 UTC
038e02d Remove extraneous newline from ldb stderr (#6897) Summary: **Summary** Remove the extraneous newline when using ldb tool. For example, the subcommand list_column_families will print an empty line to stderr even if there are no errors. **Test plan** Passed make check; manually tested a few ldb subcommands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6897 Reviewed By: pdillinger Differential Revision: D21819352 Pulled By: gg814 fbshipit-source-id: 5a16a6431bb96684fe97647f4d3ac5bf0ec7fc90 01 June 2020, 19:15:36 UTC
0c56fc4 Allow missing "unversioned" python, as in CentOS 8 (#6883) Summary: RocksDB Makefile was assuming existence of 'python' command, which is not present in CentOS 8. We avoid using 'python' if 'python3' is available. Also added fancy logic to format-diff.sh to make clang-format-diff.py for Python2 work even with Python3 only (as some CentOS 8 FB machines come equipped) Also, now use just 'python3' for PYTHON if not found so that an informative "command not found" error will result rather than something weird. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6883 Test Plan: manually tried some variants, 'make check' on a fresh CentOS 8 machine without 'python' executable or Python2 but with clang-format-diff.py for Python2. Reviewed By: gg814 Differential Revision: D21767029 Pulled By: pdillinger fbshipit-source-id: 54761b376b140a3922407bdc462f3572f461d0e9 29 May 2020, 18:29:23 UTC
c5abf78 avoid `IterKey::UpdateInternalKey()` in `BlockIter` (#6843) Summary: `IterKey::UpdateInternalKey()` is an error-prone API as it's incompatible with `IterKey::TrimAppend()`, which is used for decoding delta-encoded internal keys. This PR stops using it in `BlockIter`. Instead, it assigns global seqno in a separate `IterKey`'s buffer when needed. The logic for safely getting a Slice with global seqno properly assigned is encapsulated in `GlobalSeqnoAppliedKey`. `BinarySeek()` is also migrated to use this API (previously it ignored global seqno entirely). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6843 Test Plan: benchmark setup -- single file DBs, in-memory, no compression. "normal_db" created by regular flush; "ingestion_db" created by ingesting a file. Both DBs have same contents. ``` $ TEST_TMPDIR=/dev/shm/normal_db/ ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=10485760000 -disable_auto_compactions=true -compression_type=none -num=1000000 $ ./ldb write_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ --compression_type=no --hex --create_if_missing < <(./sst_dump --command=scan --output_hex --file=/dev/shm/normal_db/dbbench/000007.sst | awk 'began {print "0x" substr($1, 2, length($1) - 2), "==>", "0x" $5} ; /^Sst file format: block-based/ {began=1}') $ ./ldb ingest_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ ``` benchmark run command: ``` TEST_TMPDIR=/dev/shm/$DB/ ./db_bench -benchmarks=seekrandom -seek_nexts=10 -use_existing_db=true -cache_index_and_filter_blocks=false -num=1000000 -cache_size=1048576000 -threads=1 -reads=40000000 ``` results: | DB | code | throughput | |---|---|---| | normal_db | master | 267.9 | | normal_db | PR6843 | 254.2 (-5.1%) | | ingestion_db | master | 259.6 | | ingestion_db | PR6843 | 250.5 (-3.5%) | Reviewed By: pdillinger Differential Revision: D21562604 Pulled By: ajkr fbshipit-source-id: 937596f836930515da8084d11755e1f247dcb264 28 May 2020, 17:51:30 UTC
961c759 Add timestamp to delete (#6253) Summary: Preliminary user-timestamp support for delete. If ["a", ts=100] exists, you can delete it by calling `DB::Delete(write_options, key)` in which `write_options.timestamp` points to a `ts` higher than 100. Implementation A new ValueType, i.e. `kTypeDeletionWithTimestamp` is added for deletion marker with timestamp. The reason for a separate `kTypeDeletionWithTimestamp`: RocksDB may drop tombstones (keys with kTypeDeletion) when compacting them to the bottom level. This is OK and useful if timestamp is disabled. When timestamp is enabled, should we still reuse `kTypeDeletion`, we may drop the tombstone with a more recent timestamp, causing deleted keys to re-appear. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6253 Reviewed By: ltamasi Differential Revision: D20995328 Pulled By: riversand963 fbshipit-source-id: a9e5c22968ad76f98e3dc6ee0151265a3f0df619 28 May 2020, 17:40:03 UTC
e3f953a Make it possible to look up files by number in VersionStorageInfo (#6862) Summary: Does what it says on the can: the patch adds a hash map to `VersionStorageInfo` that maps file numbers to file locations, i.e. (level, position in level) pairs. This will enable stricter consistency checks in `VersionBuilder`. The patch also fixes all the unit tests that used duplicate file numbers in a version (which would trigger an assertion with the new code). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6862 Test Plan: `make check` `make whitebox_crash_test` Reviewed By: riversand963 Differential Revision: D21670446 Pulled By: ltamasi fbshipit-source-id: 2eac249945cf33d8fb8597b26bfff5221e1a861a 28 May 2020, 17:03:06 UTC
bcefc59 Allow MultiGet users to limit cumulative value size (#6826) Summary: 1. Add a value_size in read options which limits the cumulative value size of keys read in batches. Once the size exceeds read_options.value_size, all the remaining keys are returned with status Abort without further fetching any key. 2. Add a unit test case MultiGetBatchedValueSizeSimple the reads keys from memory and sst files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6826 Test Plan: 1. make check -j64 2. Add a new unit test case Reviewed By: anand1976 Differential Revision: D21471483 Pulled By: akankshamahajan15 fbshipit-source-id: dea51b8e76d5d1df38ece8cdb29933b1d798b900 27 May 2020, 20:07:14 UTC
9060e6f Add newer WBWI::NewIteratorWithBase functions to RocksJava (#6872) Summary: Exposes the `ReadOptions` arguments to `NewIteratorWithBase`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6872 Reviewed By: ajkr Differential Revision: D21725867 Pulled By: pdillinger fbshipit-source-id: 4079ba590cc13ba7a6244ed91439d89c40a543b6 27 May 2020, 18:59:12 UTC
82a82c7 Fix potential memory leak of scratch buffer (#6879) Summary: If `req.scratch` is an internally allocated buffer, but `raw_block_contents` is not constructed to own `req.scratch`, then `req.scratch` will be leaked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6879 Test Plan: make asan_check Reviewed By: anand1976 Differential Revision: D21728498 Pulled By: cheng-chang fbshipit-source-id: 8fc6a4f2543918c565ddc16ecfad1807eb9a42cf 26 May 2020, 22:29:04 UTC
bd68bfb cmake: link env_librados_test against rados (#6855) Summary: otherwise we have FTBFS like: 2020-05-18T15:12:06.400 INFO:tasks.workunit.client.0.smithi032.stdout:[100%] Linking CXX executable env_librados_test 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: CMakeFiles/rocksdb_env_librados_test.dir/utilities/env_librados_test.cc.o: undefined reference to symbol '_ZN8librados7v14_2_05Rados4initEPKc@LIBRADOS_14.2.0' 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: /lib/librados.so.2: error adding symbols: DSO missing from command line 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:collect2: error: ld returned 1 exit status this addresses the regression introduced by 07204837ce8d66e1e6e4893178f3fd040f9c1044, which hides the symbols exposed by `${THIRDPARTY_LIBS}` from consumers of librocksdb Signed-off-by: Kefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6855 Reviewed By: riversand963 Differential Revision: D21621904 fbshipit-source-id: 7022ba4dc0003504401fce6f06547e4d74a32ac0 26 May 2020, 05:53:00 UTC
b464a85 fix transaction rollback in db_stress TestMultiGet (#6873) Summary: There were further uses of `txn` after `RollbackTxn(txn)` leading to stress test errors. Moved the rollback to the end of the function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6873 Test Plan: found a command from the crash test that previously failed immediately under TSAN; verified now it succeeds. ``` ./db_stress --acquire_snapshot_one_in=10000 --allow_concurrent_memtable_write=1 --avoid_flush_during_recovery=0 --avoid_unnecessary_blocking_io=1 --block_size=16384 --bloom_bits=222.913637674 --bottommost_compression_type=none --cache_index_and_filter_blocks=1 --cache_size=1048576 --checkpoint_one_in=0 --checksum_type=kCRC32c --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_style=1 --compaction_ttl=0 --compression_max_dict_bytes=0 --compression_parallel_threads=1 --compression_type=zstd --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --db=/dev/shm/rocksdb/rocksdb_crashtest_whitebox --db_write_buffer_size=1048576 --delpercent=5 --delrangepercent=0 --destroy_db_initially=0 --disable_wal=0 --enable_pipelined_write=0 --flush_one_in=1000000 --format_version=5 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=12 --index_type=2 --key_len_percent_dist=1,30,69 --level_compaction_dynamic_level_bytes=True --log2_keys_per_lock=22 --long_running_snapshots=0 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=1000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --memtablerep=skip_list --mmap_read=1 --mock_direct_io=False --nooverwritepercent=1 --num_levels=1 --open_files=100 --ops_per_thread=200000 --partition_filters=1 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefixpercent=5 --progress_reports=0 --read_fault_one_in=0 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --snapshot_hold_ops=100000 --subcompactions=4 --sync=0 --sync_fault_injection=False --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=0 --txn_write_policy=1 --unordered_write=1 --use_block_based_filter=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=1 --use_merge=1 --use_multiget=1 --use_txn=1 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --write_buffer_size=4194304 --write_dbid_to_manifest=0 --writepercent=35 ``` Reviewed By: pdillinger Differential Revision: D21708338 Pulled By: ajkr fbshipit-source-id: dcf55cddee0a14f429a75e7a8a505acf8025f2b1 24 May 2020, 22:27:24 UTC
b559dba pin image version in circle CI builds (#6876) Summary: somehow the windows-server-2019-vs2019 image changed in a way that made VS 14 2015 the default. This caused an error when we specify VS 16 2019 as the cmake generator. I could not figure out the right arguments/env vars to get the latest VS working so pinned the image to the previous version instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6876 Reviewed By: pdillinger Differential Revision: D21709679 Pulled By: ajkr fbshipit-source-id: 2d16819ad239b4611fa199547744e1c101dc9da0 24 May 2020, 04:23:27 UTC
803a517 Misc things for ASSERT_STATUS_CHECKED, also gcc 4.8.5 (#6871) Summary: * Print stack trace on status checked failure * Make folly_synchronization_distributed_mutex_test a parallel test * Disable ldb_test.py and rocksdb_dump_test.sh with ASSERT_STATUS_CHECKED (broken) * Fix shadow warning in random_access_file_reader.h reported by gcc 4.8.5 (ROCKSDB_NO_FBCODE), also https://github.com/facebook/rocksdb/issues/6866 * Work around compiler bug on max_align_t for gcc < 4.9 * Remove an apparently wrong comment in status.h * Use check_some in Travis config (for proper diagnostic output) * Fix ignored Status in loop in options_helper.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/6871 Test Plan: manual, CI Reviewed By: ajkr Differential Revision: D21706619 Pulled By: pdillinger fbshipit-source-id: daf6364173d6689904eb394461a69a11f5bee2cb 23 May 2020, 13:53:37 UTC
bcd3256 Fix warning -Wextra-semi. NFC. (#6869) Summary: Minor fix. CLA signed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6869 Reviewed By: ajkr Differential Revision: D21704001 Pulled By: pdillinger fbshipit-source-id: 57fd08114f3234f51f34758e25e708cc70962582 22 May 2020, 18:20:13 UTC
35a25a3 Fix/expand ASSERT_STATUS_CHECKED build, add to Travis (#6870) Summary: Fixed some option handling code that recently broke the ASSERT_STATUS_CHECKED build for options_test. Added all other existing tests that pass under ASSERT_STATUS_CHECKED to the whitelist. Added a Travis configuration to run all whitelisted tests with ASSERT_STATUS_CHECKED. (Someday we might enable this check by default in debug builds.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6870 Test Plan: ASSERT_STATUS_CHECKED=1 make check, Travis Reviewed By: ajkr Differential Revision: D21704374 Pulled By: pdillinger fbshipit-source-id: 15daef98136a19d7a6843fa0c9ec08738c2ac693 22 May 2020, 18:17:29 UTC
826295a Change autovector to have a reserved size in LITE mode (#6868) Summary: Previously in LITE mode, an autovector did not have a reserved size. When elements were added to the vector, the underlying array could be reallocated. There was a set of code that never expands the autovector and was doing &autovector::back(). When the vector is resized, the old addresses may become invalid, causing a later exception to be thrown. By reserving space in the autovector up front, this problem is eliminated for those uses where the vector will never exceed the initial size. the resize happens, these pointers become invalid, leading to SEGV or other exceptions. This change allows the autovector to be fully populated before we take the address of any of its elements, thereby elminating the potential for a resize. There is comparable code to this change in Version::MultiGet for dealing with the context objects. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6868 Reviewed By: ajkr Differential Revision: D21693505 Pulled By: cheng-chang fbshipit-source-id: e71d516b15e08f202593cb80f2a42f048fc95768 21 May 2020, 21:48:10 UTC
292bcf6 skip direct I/O tests in rocksdb lite (#6867) Summary: Fix a couple places where direct I/O was used even though it is unsupported in lite builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6867 Test Plan: `LITE=1 make check -j48` Reviewed By: pdillinger Differential Revision: D21689185 Pulled By: ajkr fbshipit-source-id: 3eaa3abf69cd7d0bcaabbcad3bb5a26fb8dd7301 21 May 2020, 20:57:17 UTC
38be686 Add Struct Type to OptionsTypeInfo (#6425) Summary: Added code for generically handing structs to OptionTypeInfo. A struct is a collection of variables handled by their own map of OptionTypeInfos. Examples of structs include Compaction and Cache options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6425 Reviewed By: siying Differential Revision: D21668789 Pulled By: zhichao-cao fbshipit-source-id: 064b110de39dadf82361ed4663f7ac1a535b0b07 21 May 2020, 17:58:39 UTC
c7aedf1 Clean up some code related to file checksums (#6861) Summary: * Add missing unit test for schema stability of FileChecksumGenCrc32c (previously was only comparing to itself) * A lot of clarifying comments * Add some assertions for preconditions * Rename WritableFileWriter::CalculateFileChecksum -> UpdateFileChecksum * Simplify FileChecksumGenCrc32c with shared functions * Implement EndianSwapValue to replace unused EndianTransform And incidentally since I had trouble with 'make check-format' GitHub action disagreeing with local run, * Output full diagnostic information when 'make check-format' fails in CI Pull Request resolved: https://github.com/facebook/rocksdb/pull/6861 Test Plan: new unit test passes before & after other changes Reviewed By: zhichao-cao Differential Revision: D21667115 Pulled By: pdillinger fbshipit-source-id: 6a99970f87605aa024fa540c78cd519ff322c3e6 21 May 2020, 15:12:51 UTC
eb04bb8 Fix a bug in crash_test_with_txn (#6860) Summary: In NoBatchedOpsStress::TestMultiGet, call txn->Get() when transactions are in use. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6860 Test Plan: make crash_test_with_txn Reviewed By: pdillinger Differential Revision: D21667249 Pulled By: anand1976 fbshipit-source-id: 194bd7b9630a8efc3ae29d85422a61214e9e200e 20 May 2020, 21:47:05 UTC
545e14b Generate file checksum in SstFileWriter (#6859) Summary: If Option.file_checksum_gen_factory is set, rocksdb generates the file checksum during flush and compaction based on the checksum generator created by the factory and store the checksum and function name in vstorage and Manifest. This PR enable file checksum generation in SstFileWrite and store the checksum and checksum function name in the ExternalSstFileInfo, such that application can use them for other purpose, for example, ingest the file checksum with files in IngestExternalFile(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6859 Test Plan: add unit test and pass make asan_check. Reviewed By: ajkr Differential Revision: D21656247 Pulled By: zhichao-cao fbshipit-source-id: 78a3570c76031d8832e3d2de3d6c79cdf2b675d0 20 May 2020, 18:55:31 UTC
aaafcb8 Use in-repo gtest in buck build (#6858) Summary: ... so that we have freedom to upgrade it (see https://github.com/facebook/rocksdb/issues/6808). As a side benefit, gtest will no longer be linked into main library in buck build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6858 Test Plan: fb internal build & link Reviewed By: riversand963 Differential Revision: D21652061 Pulled By: pdillinger fbshipit-source-id: 6018104af944debde576b5beda6c134e737acedb 20 May 2020, 18:37:45 UTC
a1523ef Status check enforcement for io_posix_test and options_settable_test (#6857) Summary: Added status check enforcement for io_posix_test and options_settable_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6857 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 check Reviewed By: ajkr Differential Revision: D21647904 Pulled By: akankshamahajan15 fbshipit-source-id: b7f2321eb6c141a88cd5e1270ecb7d58f00341af 20 May 2020, 02:22:28 UTC
39b2443 Strengthen MultiGet correctness verification in NoBatchedOpsStress (#6849) Summary: Add MultiGet to VerifyDb and check consistency with Get in TestMultiGet. Test plan - make crash_test ASAN crash test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6849 Reviewed By: pdillinger Differential Revision: D21635011 Pulled By: anand1976 fbshipit-source-id: deb5a79d08fefd8d8010204f1f20b83adc92310e 19 May 2020, 19:47:22 UTC
1551f10 Refactor the blob file related logic in VersionBuilder (#6835) Summary: This patch is groundwork for an upcoming change to store the set of linked SSTs in `BlobFileMetaData`. With the current code, a new `BlobFileMetaData` object is created each time a `VersionEdit` touches a certain blob file. This is fine as long as these objects are lightweight and cheap to create; however, with the addition of the linked SST set, it would be very inefficient since the set would have to be copied over and over again. Note that this is the same kind of problem that `VersionBuilder` is solving w/r/t `Version`s and files, and we can apply the same solution; that is, we can accumulate the changes in a different mutable object, and apply the delta in one shot when the changes are committed. The patch does exactly that by adding a new `BlobFileMetaDataDelta` class to `VersionBuilder`. In addition, it turns the existing `GetBlobFileMetaData` helper into `IsBlobFileInVersion` (which is fine since that's the only thing the method's clients care about now), and adds a couple of helper methods that can create a `BlobFileMetaData` object from the `BlobFileMetaData` in the base (if applicable) and the delta when the `Version` is saved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6835 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21505187 Pulled By: ltamasi fbshipit-source-id: d81a48c5f2ca7b79d7124c935332a6bcf3d5d988 19 May 2020, 17:00:04 UTC
0ac0098 Make options length longer for sst_dump_test (#6846) Summary: Under MacOS when running with make -j 8 check, the temporary directory generated was > 100 characters. This caused the tests to do nothing under MacOS. Most of them still reported success for doing nothing, but ReadaheadSize was expecting the test to run. By making the option name longer, the tests will no run successfully (and do something!) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6846 Reviewed By: ajkr Differential Revision: D21576032 fbshipit-source-id: b089cde0d598137b572aa8527cc5459085252af7 19 May 2020, 16:22:12 UTC
ada700b Re-read the whole request in direct IO mode when IO uring returns partial result (#6853) Summary: If both direct IO and IO uring are enabled, when IO uring returns partial result, we'll try to read the remaining part of the request, but the starting address/offset of the remaining part might not be aligned to the block size, in direct IO mode, the unaligned offset causes bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6853 Test Plan: run make check with both direct IO and IO uring enabled, this is covered by one of the continuous tests. Reviewed By: anand1976 Differential Revision: D21603023 Pulled By: cheng-chang fbshipit-source-id: 942f6a11ff21e1892af6c4464e02bab4c707787c 19 May 2020, 00:25:57 UTC
b9d65f5 Trigger compaction in CompactOnDeletionCollector based on deletion ratio (#6806) Summary: In level compaction, if the total size (even if compensated after taking account of the deletions) of a level hasn't exceeded the limit, but there are lots of deletion entries in some SST files of the level, these files should also be good candidates for compaction. Otherwise, queries for the deleted keys might be slow because they need to go over all the tombstones. This PR adds an option `deletion_ratio` to the factory of `CompactOnDeletionCollector` to configure it to trigger compaction when the ratio of tombstones >= `deletion_ratio`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6806 Test Plan: Added new unit test in `compact_on_deletion_collector_test.cc`. make compact_on_deletion_collector_test && ./compact_on_deletion_collector_test Reviewed By: ajkr Differential Revision: D21511981 Pulled By: cheng-chang fbshipit-source-id: 65a9d0150e8c9c00337787686475252e4535a3e1 18 May 2020, 15:42:05 UTC
d790e60 Fix buck target db_stress_lib in opt mode (#6847) Summary: In buck build with opt mode, target should not include rocksdb_test_lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6847 Test Plan: Watch for internal cont build. Reviewed By: ajkr Differential Revision: D21586803 Pulled By: riversand963 fbshipit-source-id: 76d253c18d16fac6cab86a8c3f6b471ad5b6efb3 17 May 2020, 04:48:20 UTC
50ba245 Use 'make all' in LITE Travis configuration (#6834) Summary: So that we don't miss LITE compilation errors in tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/6834 Reviewed By: anand1976 Differential Revision: D21503227 Pulled By: pdillinger fbshipit-source-id: 3b2fdf3c4d395354d0ababac06da32addbafb3a5 15 May 2020, 20:59:24 UTC
91b7553 Enable IO Uring in MultiGet in direct IO mode (#6815) Summary: Currently, in direct IO mode, `MultiGet` retrieves the data blocks one by one instead of in parallel, see `BlockBasedTable::RetrieveMultipleBlocks`. Since direct IO is supported in `RandomAccessFileReader::MultiRead` in https://github.com/facebook/rocksdb/pull/6446, this PR applies `MultiRead` to `MultiGet` so that the data blocks can be retrieved in parallel. Also, in direct IO mode and when data blocks are compressed and need to uncompressed, this PR only allocates one continuous aligned buffer to hold the data blocks, and then directly uncompress the blocks to insert into block cache, there is no longer intermediate copies to scratch buffers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6815 Test Plan: 1. added a new unit test `BlockBasedTableReaderTest::MultiGet`. 2. existing unit tests and stress tests contain tests against `MultiGet` in direct IO mode. Reviewed By: anand1976 Differential Revision: D21426347 Pulled By: cheng-chang fbshipit-source-id: b8446ae0e74152444ef9111e97f8e402ac31b24f 15 May 2020, 06:26:26 UTC
b11a8b1 Fix valgrind error by init memory region (#6842) Summary: As title. After allocating a memory buffer, initialize its content to 0s. This fixes valgrind issue introduced in https://github.com/facebook/rocksdb/issues/6709. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6842 Test Plan: ``` $make valgrind_test $valgrind --tool=memcheck --track-origins=yes ./db_test2 --gtest_filter=DBTest2.CompressionFailures ``` Reviewed By: pdillinger Differential Revision: D21551848 Pulled By: riversand963 fbshipit-source-id: e87a6f413e3f3d92d8e23d8ecc4cf93479c6674c 15 May 2020, 01:50:03 UTC
06a2dce Move checksum calculation ahead of memory copy (#6844) Summary: Originally, the checksum of appended data in writable file writer is calculated after the data is copied to the buffer. It will not be able to catch the bit flip happens during copy. Move the checksum calculation before it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6844 Test Plan: pass make asan_check Reviewed By: cheng-chang Differential Revision: D21576726 Pulled By: zhichao-cao fbshipit-source-id: 0a062a1f19886f6ea0d4e3f557e6f4b799773254 14 May 2020, 21:58:14 UTC
50d63a2 Fix LITE build failure in compaction_picker_test (#6839) Summary: Fix LITE build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6839 Test Plan: LITE=1 make check Reviewed By: riversand963 Differential Revision: D21535808 Pulled By: anand1976 fbshipit-source-id: fcad961eca08e13fb0c256c92d18c3c1f1165f22 13 May 2020, 17:47:15 UTC
3bea276 Do not print u'string' in TARGETS file (#6841) Summary: Before this PR, extra deps passed in from cmd line to buckifier will be parsed and used to populate a dict. Using this dict and printing to TARGETS file will lead to printing u'', disallowed by build tools. This PR removes the u''. Test Plan (local dev server): ``` python buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}' ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841 Reviewed By: siying Differential Revision: D21538155 Pulled By: riversand963 fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349 13 May 2020, 04:37:31 UTC
244797a Add `find_dependency()` in cmake config file. (#6791) Summary: Currently when building PyTorch with latest RocksDB we get errors like "missing target Snappy::snappy", because they are simply not there. With old `${VAR}` approach we essentially hard-code the abs path found during RocksDB build, which is: - Not relocatable. - Doesn't work when changed to modern target-based design because that requires target to present when used for expansion. This fix allows cmake to setup imported target, if enabled during RocksDB build, when downstream uses `find_package(RocksDB)`. This is for https://github.com/facebook/rocksdb/issues/6179 tchaikov Please help review, thanks! Pull Request resolved: https://github.com/facebook/rocksdb/pull/6791 Reviewed By: riversand963 Differential Revision: D21471553 fbshipit-source-id: 8d4ff2ab589a97ca6e6ba27e1f17b97a00f06206 13 May 2020, 04:18:29 UTC
0720483 Mark dependencies as PRIVATE and fix missing dependencies in tools. (#6790) Summary: Tools were mistakenly using leaked (PUBLIC) dependencies from main lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6790 Reviewed By: riversand963 Differential Revision: D21471551 fbshipit-source-id: ec43b92e231777e0fcf0f865444391af09d6963b 13 May 2020, 04:07:55 UTC
4a4b8a1 sst_dump to reduce number of file reads (#6836) Summary: sst_dump can issue many file reads from the file system. This doesn't work well with file systems without a OS cache, especially remote file systems. In order to mitigate this problem, several improvements are done: 1. --readahead_size is added, so that users can specify readahead size when scanning the data. 2. Force a 512KB tail readahead, which prevents three I/Os for footer, meta index and property blocks and hopefully index and filter blocks too. 3. Consoldiate SSTDump's I/Os before opening the file for read. Use the same file prefetch buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6836 Test Plan: Add a test that covers this new feature. Reviewed By: pdillinger Differential Revision: D21516607 fbshipit-source-id: 3ae43526286f67b2f4a5bdedfbc92719d579b87e 13 May 2020, 01:23:33 UTC
70aaa9c Expose CancellAllBackgroundWork to C api (#6832) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6832 Reviewed By: cheng-chang Differential Revision: D21498186 Pulled By: riversand963 fbshipit-source-id: 66bb0d7c06af2bf0df3c6a09b61bca2fb81f2dd3 12 May 2020, 21:50:52 UTC
c384c08 Add tests for compression failure in BlockBasedTableBuilder (#6709) Summary: Currently there is no check for whether BlockBasedTableBuilder will expose compression error status if compression fails during the table building. This commit adds fake faulting compressors and a unit test to test such cases. This check finds 5 bugs, and this commit also fixes them: 1. Not handling compression failure well in BlockBasedTableBuilder::BGWorkWriteRawBlock. 2. verify_compression failing in BlockBasedTableBuilder when used with ZSTD. 3. Wrongly passing the same reference of block contents to BlockBasedTableBuilder::CompressAndVerifyBlock in parallel compression. 4. Wrongly setting block_rep->first_key_in_next_block to nullptr in BlockBasedTableBuilder::EnterUnbuffered when there are still incoming data blocks. 5. Not maintaining variables for compression ratio estimation and first_block in BlockBasedTableBuilder::EnterUnbuffered. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6709 Reviewed By: ajkr Differential Revision: D21236254 fbshipit-source-id: 101f6e62b2bac2b7be72be198adf93cd32a1ff46 12 May 2020, 16:27:35 UTC
3f21807 fix typo (#6831) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6831 Reviewed By: cheng-chang Differential Revision: D21499149 Pulled By: zhichao-cao fbshipit-source-id: 2cb76cbf7086677d8cad5c828019e008062f0052 11 May 2020, 21:58:25 UTC
2e93247 Disable a few timer tests (#6833) Summary: Disable `TimerTest.SingleScheduleRepeatedlyTest` and `TimerTest.MultipleScheduleRepeatedlyTest`. This is to help people to not hit any hangs (https://github.com/facebook/rocksdb/issues/6698) during their development process while I investigate further; I could not reproduce the issue on my dev machine yet. Note that timer is not being utilized anywhere yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6833 Test Plan: ``` svemuri@devbig187 ~/rocksdb (timer-disable-test) $ TEST_TMPDIR=/dev/shm ./timer_test [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from TimerTest [ RUN ] TimerTest.SingleScheduleOnceTest [ OK ] TimerTest.SingleScheduleOnceTest (1 ms) [ RUN ] TimerTest.MultipleScheduleOnceTest [ OK ] TimerTest.MultipleScheduleOnceTest (0 ms) [----------] 2 tests from TimerTest (1 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (1 ms total) [ PASSED ] 2 tests. YOU HAVE 2 DISABLED TESTS ``` Reviewed By: pdillinger Differential Revision: D21502474 Pulled By: sagar0 fbshipit-source-id: ac67caee2011fd14ffb2476a8914a6286a4f9abe 11 May 2020, 20:30:00 UTC
f0e8731 Use GFlags/Snappy config in CMake, with fallback for legacy approach. (#6771) Summary: Related to some discussion in https://github.com/facebook/rocksdb/issues/6179 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6771 Reviewed By: zhichao-cao Differential Revision: D21340117 fbshipit-source-id: a1af0ba4865bb13c8c817851d6f6c4056191b3fe 11 May 2020, 17:28:49 UTC
3a1c29d Add missing my_pid to fprintf in multi_process_example (#6731) Summary: Signed-off-by: Derrick Pallas <derrick@pallas.us> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6731 Reviewed By: siying Differential Revision: D21137005 Pulled By: riversand963 fbshipit-source-id: a7182e1bec225bc110971f40e2d0e6c3a671c061 09 May 2020, 03:49:33 UTC
a50ea71 Improve ldb consistency checks (#6802) Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0 08 May 2020, 21:17:47 UTC
d9cd335 Suppress UBSAN warning in CRC32 ARM (#6827) Summary: UBSAN shows following warning: util/crc32c_arm64.cc:111:11: runtime error: load of misaligned address 0x00001afcda86 for type 'const uint64_t', which requires 8 byte alignment 0x00001afcda86: note: pointer points here cc c1 2d 00 01 81 40 24 30 66 39 66 30 37 30 63 2d 32 36 63 34 2d 34 62 61 61 2d 38 35 33 31 2d ^ Suppress it just as what we do in x86 CRC. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6827 Test Plan: Run the same UBSAN and see it to pass now. Reviewed By: ltamasi Differential Revision: D21471838 fbshipit-source-id: 02943dd39a7030d2b03e5d894dcb23ed72b6c9c3 08 May 2020, 21:14:00 UTC
e72e216 Fix a few bugs in best-efforts recovery (#6824) Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2 08 May 2020, 20:01:42 UTC
1c84660 prototype status check enforcement (#6798) Summary: Tried making Status object enforce that it is checked in some way. In cases it is not checked, `PermitUncheckedError()` must be called explicitly. Added a way to run tests (`ASSERT_STATUS_CHECKED=1 make -j48 check`) on a whitelist. The effort appears significant to get each test to pass with this assertion, so I only fixed up enough to get one test (`options_test`) working and added it to the whitelist. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6798 Reviewed By: pdillinger Differential Revision: D21377404 Pulled By: ajkr fbshipit-source-id: 73236f9c8df38f01cf24ecac4a6d1661b72d077e 08 May 2020, 19:40:43 UTC
1282589 Disable "compression_parallel_threads" in crash test for now (#6816) Summary: "compressio_parallel_threads" caused several test failure tests. To keep crash test clean, disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6816 Test Plan: "make crash_test" to make sure the python script doesn't break Reviewed By: zhichao-cao Differential Revision: D21462112 fbshipit-source-id: 9eecc764800da82cd19665dc8b167eacead3310b 08 May 2020, 03:52:29 UTC
9426523 Fix race due to delete triggered compaction in Universal compaction mode (#6799) Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a 08 May 2020, 00:32:17 UTC
3730b05 Fixup HISTORY.md for e9ba4ba "validate range tombstone covers positiv… (#6825) Summary: …e range" Moved it from the wrong section (6.10) to the right section (Unreleased). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6825 Reviewed By: zhichao-cao Differential Revision: D21464577 Pulled By: ajkr fbshipit-source-id: a836b4ab10be2464182826f9411c9c424c933b70 07 May 2020, 23:40:17 UTC
f286fb3 Include options.h in table.h (#6823) Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d 07 May 2020, 22:55:29 UTC
b27a144 Fix false NotFound from batched MultiGet with kHashSearch (#6821) Summary: The error is assigning KeyContext::s to NotFound status in a table reader for a "not found in this table" case, which skips searching in later tables, like only a delete should. (The hash search index iterator is the only one that can return status NotFound even if Valid() == false.) This was detected by intermittent failure in MultiThreadedDBTest.MultiThreaded/5, a kHashSearch configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6821 Test Plan: modified existing unit test to reproduce problem Reviewed By: anand1976 Differential Revision: D21450469 Pulled By: pdillinger fbshipit-source-id: 7478003684d637dbd491cdac81468041a791be2c 07 May 2020, 22:41:37 UTC
e9ba4ba validate range tombstone covers positive range (#6788) Summary: We found some files containing nothing but negative range tombstones, and unsurprisingly their metadata specified a negative range, which made things crash. Time to add a bit of user input validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6788 Reviewed By: zhichao-cao Differential Revision: D21343719 Pulled By: ajkr fbshipit-source-id: f1c16e4c3e9fa150958c8c866176632a3206fb74 07 May 2020, 18:55:30 UTC
ac3ae1d Find/purge obsolete blob files (#6807) Summary: The patch extends `FindObsoleteFiles` and `PurgeObsoleteFiles` with support for blob files. The behavior is analogous to SST files: obsolete blob files are put on the "candidates for deletion" list, while live (and pending) files are preserved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6807 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21406249 Pulled By: ltamasi fbshipit-source-id: 1948f71c31927564b61e8af394f50ca3964880d9 07 May 2020, 16:32:51 UTC
1f20df2 cover single level universal in crash test (#6818) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6818 Test Plan: fast whitebox test and verify there are some single-level universal and some multi-level universal runs. ``` $ python ./tools/db_crashtest.py whitebox --simple -max_key=1000000 -value_size_mult=33 -write_buffer_size=524288 -target_file_size_base=524288 -max_bytes_for_level_base=2097152 --duration=120 --interval=10 --ops_per_thread=1000 --random_kill_odd=887 ``` Reviewed By: riversand963 Differential Revision: D21432138 Pulled By: ajkr fbshipit-source-id: 2fc5ba9f3dfa49bb11e81da7dd00a17b476e64d7 07 May 2020, 01:08:09 UTC
c1e1185 Update release version to 6.10 (#6797) Summary: Update HISTORY.md and version.h to 6.10. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6797 Reviewed By: zhichao-cao Differential Revision: D21371390 Pulled By: anand1976 fbshipit-source-id: 6017bca24fc5d12076d1ddaec7783c9b85712d42 06 May 2020, 23:42:37 UTC
06c3b85 Disallow using the base DB's storage directory as blob_dir in BlobDB (#6810) Summary: https://github.com/facebook/rocksdb/pull/6807 extends the logic that identifies and purges obsolete files to blob files handled by RocksDB itself. In order to prevent that from interfering with the current BlobDB code, we need to make sure that `BlobDBOptions::blob_dir` is different from the storage directories used by the base DB. (Note: this is true by default.) The patch adds a check that explicitly disallows this configuration and returns `Status::NotSupported` from `BlobDB::Open` in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6810 Test Plan: Tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21412676 Pulled By: ltamasi fbshipit-source-id: 6630cc7481e48c8bf55d59423b25f14d52ffe681 06 May 2020, 21:00:46 UTC
53f8447 Add YugabyteDB to USERS (#6786) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6786 Reviewed By: zhichao-cao Differential Revision: D21407762 Pulled By: pdillinger fbshipit-source-id: 62f74409cc72c6b9ce4107aa515955cfc5eec12a 06 May 2020, 17:28:29 UTC
c21c459 Slightly expand converage to file consistency check failure (#6800) Summary: Current DBCompactionTest.ConsistencyFailTest checks DB fails after L0 inconsitency is found. Add slightly more coverage by introducing DBCompactionTest.ConsistencyFailTest2 which checks non-L0 files too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6800 Test Plan: Run the new test. Reviewed By: riversand963 Differential Revision: D21384806 fbshipit-source-id: 36db7b657eed42115283fe2f6afa4c3a31a3b510 06 May 2020, 01:31:53 UTC
394f2bb Add OptionTypeInfo::Enum and related methods (#6423) Summary: Add methods and constructors for handling enums to the OptionTypeInfo. This change allows enums to be converted/compared without adding a special "type" to the OptionType. This change addresses a couple of issues: - It allows new enumerated types to be added to the options without editing the OptionType base class (and related methods) - It standardizes the procedure for adding enumerated types to the options, reducing potential mistakes - It moves the enum maps to the location where they are used, allowing them to be static file members rather than global values - It reduces the number of types and cases that need to be handled in the various OptionType methods Pull Request resolved: https://github.com/facebook/rocksdb/pull/6423 Reviewed By: siying Differential Revision: D21408713 Pulled By: zhichao-cao fbshipit-source-id: fc492af285d011822578b95d186a0fce25d35626 05 May 2020, 22:04:04 UTC
a96461d fix swallowed error for file deletion consistency check (#6809) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6809 Reviewed By: pdillinger Differential Revision: D21411971 Pulled By: ajkr fbshipit-source-id: 900b6b0370b76e9a3e5e03f968e2ac1bbaab73b8 05 May 2020, 21:54:21 UTC
2f1700c Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) Summary: This very old test code bug was causing a new valgrind failure in MultiGetDeadlineExceeded Also fix hang in MultiGetDeadlineExceeded by unifying with some logic from another test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6803 Test Plan: run that unit test under valgrind, make check Reviewed By: siying Differential Revision: D21388470 Pulled By: pdillinger fbshipit-source-id: 0ce99d6d5eb8cd3195b17406892c8c5cff5fa5dd 05 May 2020, 20:11:29 UTC
91bc013 Refactor level compaction picker (#6804) Summary: 1. refactor out PickFileToCompact to remove duplicated logic. 2. remove redundant checks of `start_level_inputs_.empty()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6804 Test Plan: make check Reviewed By: siying Differential Revision: D21390053 Pulled By: cheng-chang fbshipit-source-id: 185d5987a08bfdaf63f0f245310c6da69878d415 05 May 2020, 18:09:29 UTC
5584595 Do not swallow error returned from SaveTo() (#6801) Summary: With consistency check enabled, VersionBuilder::SaveTo() may return error once corruption is detected while building versions. We should handle these errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6801 Test Plan: make check Reviewed By: siying Differential Revision: D21385045 Pulled By: riversand963 fbshipit-source-id: 98f6424e2a4699b62befa21e9fe00e70a771118e 05 May 2020, 17:46:20 UTC
5a61e78 Fix db_stress when GetLiveFiles() flushes dropped CF (#6805) Summary: Current impl. of db_stress will abort verification and report failure if GetLiveFiles() causes a dropped column family to be flushed. This is not desired. To fix, this PR makes the following change: In GetLiveFiles, if flush is triggered and returns Status::IsColumnFamilyDropped(), then set status to Status::OK(). This is OK because dropped column families will be skipped during the rest of this function, and valid column families will have their live files returned to caller. Test plan (dev server): make check ./db_stress -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 ./db_stress -disable_wal=1 -reopen=0 -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6805 Reviewed By: ltamasi Differential Revision: D21390044 Pulled By: riversand963 fbshipit-source-id: de67846b95a4f1b88aa0a30c3d70c43cc68625b9 05 May 2020, 00:45:49 UTC
a00ddf1 Expose the set of live blob files from Version/VersionSet (#6785) Summary: The patch adds logic that returns the set of live blob files from `Version::AddLiveFiles` and `VersionSet::AddLiveFiles` (in addition to live table files), and also cleans up the code a bit, for example, by exposing only the numbers of table files as opposed to the earlier `FileDescriptor`s that no clients used. Moreover, the patch extends the `GetLiveFiles` API so that it also exposes blob files in the current version. Similarly to https://github.com/facebook/rocksdb/pull/6755, this is a building block for identifying and purging obsolete blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6785 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21336210 Pulled By: ltamasi fbshipit-source-id: fc1aede8a49eacd03caafbc5f6f9ce43b6270821 04 May 2020, 22:08:13 UTC
680c416 Avoid Swallowing Some File Consistency Checking Bugs (#6793) Summary: We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump. More places are not fixed and need follow-up. Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793 Test Plan: Add a unit test to cover the reopen case. Reviewed By: riversand963 Differential Revision: D21366525 fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded 04 May 2020, 21:18:11 UTC
d9e170d Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper (#6795) Summary: Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper using db_bench. Details changes as follows. 1, add a separate random mode in MixGraph to produce all_random workload. 2, fix power inverse function for generating prefix_dist workload. 3, make sure key_offset in prefix mode is always unsigned. note: Need to carefully choose key_dist_a/b to avoid aliasing. Power inverse function range should be close to overall key space. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6795 Reviewed By: akankshamahajan15 Differential Revision: D21371095 Pulled By: zhichao-cao fbshipit-source-id: 80744381e242392c8c7cf8ac3d68fe67fe876048 04 May 2020, 17:55:14 UTC
211088d Remove redundant update of txn_state_ in transaction Prepare (#6778) Summary: When expiration is set in a pessimistic transaction, `txn_state_` is already updated to `AWAITING_PREPARE` in the `if (expiration_time_ > 0)` block, there is no need to update the state in `if (can_prepare)` block again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6778 Test Plan: make check Reviewed By: lth Differential Revision: D21335319 Pulled By: cheng-chang fbshipit-source-id: 251d634cc7d1a0e86e673a59f0bda8584da5a35f 02 May 2020, 00:37:33 UTC
c8643ed Fix multiple CF replay failure in db_bench replay (#6787) Summary: The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787 Test Plan: pass trace replay test. Reviewed By: yhchiang Differential Revision: D21339941 Pulled By: zhichao-cao fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce 01 May 2020, 07:03:38 UTC
6acbbbf Add Github Action for some basic sanity test of PR (#6761) Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787 01 May 2020, 02:22:45 UTC
6504ae0 Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) Summary: The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782 Test Plan: Run all existing tests. Reviewed By: zhichao-cao Differential Revision: D21338614 fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685 01 May 2020, 00:01:17 UTC
ef0c3ed Make users explicitly be aware of prepare before commit (#6775) Summary: In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare. This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775 Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare Reviewed By: lth Differential Revision: D21313270 Pulled By: cheng-chang fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251 30 April 2020, 23:24:20 UTC
079e50d Disallow BlockBasedTableBuilder to set status from non-OK (#6776) Summary: There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future. The solution is to only make it possible to set the status code if the status code to set is not OK. Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776 Test Plan: Run all existing test. Reviewed By: pdillinger Differential Revision: D21314382 fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375 30 April 2020, 22:37:03 UTC
6277e28 Flag CompressionOptions::parallel_threads to be experimental (#6781) Summary: The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781 Reviewed By: pdillinger Differential Revision: D21330678 fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13 30 April 2020, 22:22:06 UTC
ab13d43 Pass a timeout to FileSystem for random reads (#6751) Summary: Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded. For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR. Tests: Update existing unit tests to verify the correct timeout value is being passed Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751 Reviewed By: riversand963 Differential Revision: D21285631 Pulled By: anand1976 fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567 30 April 2020, 21:50:39 UTC
eecd8fb Fix assertion that can fail on sst corruption (#6780) Summary: An assertion that a char == a CompressionType (unsigned char) originally cast from a char can fail if the original value is negative, due to numeric promotion. The assertion should pass even if the value is invalid CompressionType, because the callee UncompressBlockContentsForCompressionType checks for that and reports status appropriately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780 Test Plan: Temporarily change kZSTD = 0x88 and see tests fail. Make this change (in addition), and tests pass. Reviewed By: siying Differential Revision: D21328498 Pulled By: pdillinger fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92 30 April 2020, 19:11:00 UTC
fe238e5 Keep track of obsolete blob files in VersionSet (#6755) Summary: The patch adds logic to keep track of obsolete blob files. A blob file becomes obsolete when the last `shared_ptr` that points to the corresponding `SharedBlobFileMetaData` object goes away, which, in turn, happens when the last `Version` that contains the blob file is destroyed. No longer needed blob files are added to the obsolete list in `VersionSet` using a custom deleter to avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`. Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored in `JobContext`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21233155 Pulled By: ltamasi fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a 30 April 2020, 18:25:51 UTC
cf34246 Add Slack forum to README (#6773) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773 Reviewed By: siying Differential Revision: D21310229 Pulled By: pdillinger fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf 30 April 2020, 18:00:28 UTC
e619a20 Add an option for parallel compression in for db_stress (#6722) Summary: This commit adds an `compression_parallel_threads` option in db_stress. It also fixes the naming of parallel compression option in db_bench to keep it aligned with others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722 Reviewed By: pdillinger Differential Revision: D21091385 fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100 30 April 2020, 17:49:07 UTC
8c69402 Fix potential size_t overflow in import_column_family (#6762) Summary: The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762 Test Plan: make asan_check Reviewed By: pdillinger Differential Revision: D21323063 Pulled By: zhichao-cao fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a 30 April 2020, 15:40:42 UTC
b938e60 Fix a couple of bugs in FaultInjectionTestFS (#6777) Summary: Fix the following cases that can cause false alarms in db_stress when read fault injection is enabled - 1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error. 2. Handle the case when the randomly generated string to overwrite the original block is identical to the original. Tests: Run db_stress w/ and wo/ direct IO and fault injection turned on Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777 Reviewed By: zhichao-cao Differential Revision: D21316734 Pulled By: anand1976 fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa 30 April 2020, 02:28:29 UTC
28fe8e4 Fix bug in format-diff.sh (#6772) Summary: Nasty bug in which more/different changes would be applied than those shown to user Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772 Test Plan: manual Reviewed By: siying Differential Revision: D21304604 Pulled By: pdillinger fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc 29 April 2020, 20:45:54 UTC
5272305 Fix FilterBench when RTTI=0 (#6732) Summary: The dynamic_cast in the filter benchmark causes release mode to fail due to no-rtti. Replace with static_cast_with_check. Signed-off-by: Derrick Pallas <derrick@pallas.us> Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732 Reviewed By: ltamasi Differential Revision: D21304260 Pulled By: pdillinger fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c 29 April 2020, 20:09:23 UTC
back to top