298b00a | sdong | 02 June 2020, 19:09:48 UTC | Reduce dependency on gtest dependency in release code (#6907) Summary: Release code now depends on gtest, indirectly through including "test_util/testharness.h". This creates multiple problems. One important reason is the definition of IGNORE_STATUS_IF_ERROR() in test_util/testharness.h. Move it to sync_point.h instead. Note that utilities/cassandra/format.h still depends on "test_util/testharness.h". This will be resolved in a separate diff. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6907 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21829884 fbshipit-source-id: 9253c19ffde2936f3ae68998210f8e54f645a6e6 | 02 June 2020, 19:11:24 UTC |
8d87e9c | Adam Retter | 02 June 2020, 03:31:29 UTC | Update googletest from 1.8.1 to 1.10.0 (#6808) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6808 Reviewed By: anand1976 Differential Revision: D21483984 Pulled By: pdillinger fbshipit-source-id: 70c5eff2bd54ddba469761d95e4cd4611fb8e598 | 02 June 2020, 03:33:42 UTC |
66942e8 | anand76 | 02 June 2020, 02:41:58 UTC | Avoid unnecessary reads of uncompression dictionary in MultiGet (#6906) Summary: We may sometimes read the uncompression dictionary when its not necessary, when we lookup a key in an SST file but the index indicates the key is not present. This can happen with index_type 3. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6906 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D21828944 Pulled By: anand1976 fbshipit-source-id: 7aef4f0a39548d0874eafefd2687006d2652f9bb | 02 June 2020, 02:43:37 UTC |
02f59ed | sdong | 01 June 2020, 23:30:45 UTC | Find the correct gcov (#6904) Summary: Right now in FB environment, wrong gcov is used. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6904 Test Plan: "make coverage" and watch results. Reviewed By: riversand963 Differential Revision: D21824291 fbshipit-source-id: 666011fd86c36adafa09ebd9eb97742f94fb90bb | 01 June 2020, 23:33:05 UTC |
bcb9e41 | Cheng Chang | 01 June 2020, 22:17:58 UTC | Explicitly free allocated buffer when status is not ok (#6903) Summary: Currently we rely on `BlockContents` to implicitly free the allocated scratch buffer, but when IO error happens, it doesn't make sense to construct the `BlockContents` which might be corrupted. In the stress test, we find that `assert(req.result.size() == block_size(handle));` fails because of potential IO errors. In this PR, we explicitly free the scratch buffer on error without constructing `BlockContents`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6903 Test Plan: watch stress test Reviewed By: anand1976 Differential Revision: D21823869 Pulled By: cheng-chang fbshipit-source-id: 5603fc80e9bf3f44a9d7250ddebd871afe1eb89f | 01 June 2020, 22:19:40 UTC |
038e02d | zitan | 01 June 2020, 19:13:36 UTC | Remove extraneous newline from ldb stderr (#6897) Summary: **Summary** Remove the extraneous newline when using ldb tool. For example, the subcommand list_column_families will print an empty line to stderr even if there are no errors. **Test plan** Passed make check; manually tested a few ldb subcommands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6897 Reviewed By: pdillinger Differential Revision: D21819352 Pulled By: gg814 fbshipit-source-id: 5a16a6431bb96684fe97647f4d3ac5bf0ec7fc90 | 01 June 2020, 19:15:36 UTC |
0c56fc4 | Peter Dillinger | 29 May 2020, 18:24:19 UTC | Allow missing "unversioned" python, as in CentOS 8 (#6883) Summary: RocksDB Makefile was assuming existence of 'python' command, which is not present in CentOS 8. We avoid using 'python' if 'python3' is available. Also added fancy logic to format-diff.sh to make clang-format-diff.py for Python2 work even with Python3 only (as some CentOS 8 FB machines come equipped) Also, now use just 'python3' for PYTHON if not found so that an informative "command not found" error will result rather than something weird. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6883 Test Plan: manually tried some variants, 'make check' on a fresh CentOS 8 machine without 'python' executable or Python2 but with clang-format-diff.py for Python2. Reviewed By: gg814 Differential Revision: D21767029 Pulled By: pdillinger fbshipit-source-id: 54761b376b140a3922407bdc462f3572f461d0e9 | 29 May 2020, 18:29:23 UTC |
c5abf78 | Andrew Kryczka | 28 May 2020, 17:49:02 UTC | avoid `IterKey::UpdateInternalKey()` in `BlockIter` (#6843) Summary: `IterKey::UpdateInternalKey()` is an error-prone API as it's incompatible with `IterKey::TrimAppend()`, which is used for decoding delta-encoded internal keys. This PR stops using it in `BlockIter`. Instead, it assigns global seqno in a separate `IterKey`'s buffer when needed. The logic for safely getting a Slice with global seqno properly assigned is encapsulated in `GlobalSeqnoAppliedKey`. `BinarySeek()` is also migrated to use this API (previously it ignored global seqno entirely). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6843 Test Plan: benchmark setup -- single file DBs, in-memory, no compression. "normal_db" created by regular flush; "ingestion_db" created by ingesting a file. Both DBs have same contents. ``` $ TEST_TMPDIR=/dev/shm/normal_db/ ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=10485760000 -disable_auto_compactions=true -compression_type=none -num=1000000 $ ./ldb write_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ --compression_type=no --hex --create_if_missing < <(./sst_dump --command=scan --output_hex --file=/dev/shm/normal_db/dbbench/000007.sst | awk 'began {print "0x" substr($1, 2, length($1) - 2), "==>", "0x" $5} ; /^Sst file format: block-based/ {began=1}') $ ./ldb ingest_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ ``` benchmark run command: ``` TEST_TMPDIR=/dev/shm/$DB/ ./db_bench -benchmarks=seekrandom -seek_nexts=10 -use_existing_db=true -cache_index_and_filter_blocks=false -num=1000000 -cache_size=1048576000 -threads=1 -reads=40000000 ``` results: | DB | code | throughput | |---|---|---| | normal_db | master | 267.9 | | normal_db | PR6843 | 254.2 (-5.1%) | | ingestion_db | master | 259.6 | | ingestion_db | PR6843 | 250.5 (-3.5%) | Reviewed By: pdillinger Differential Revision: D21562604 Pulled By: ajkr fbshipit-source-id: 937596f836930515da8084d11755e1f247dcb264 | 28 May 2020, 17:51:30 UTC |
961c759 | Yanqin Jin | 28 May 2020, 17:37:57 UTC | Add timestamp to delete (#6253) Summary: Preliminary user-timestamp support for delete. If ["a", ts=100] exists, you can delete it by calling `DB::Delete(write_options, key)` in which `write_options.timestamp` points to a `ts` higher than 100. Implementation A new ValueType, i.e. `kTypeDeletionWithTimestamp` is added for deletion marker with timestamp. The reason for a separate `kTypeDeletionWithTimestamp`: RocksDB may drop tombstones (keys with kTypeDeletion) when compacting them to the bottom level. This is OK and useful if timestamp is disabled. When timestamp is enabled, should we still reuse `kTypeDeletion`, we may drop the tombstone with a more recent timestamp, causing deleted keys to re-appear. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6253 Reviewed By: ltamasi Differential Revision: D20995328 Pulled By: riversand963 fbshipit-source-id: a9e5c22968ad76f98e3dc6ee0151265a3f0df619 | 28 May 2020, 17:40:03 UTC |
e3f953a | Levi Tamasi | 28 May 2020, 17:00:19 UTC | Make it possible to look up files by number in VersionStorageInfo (#6862) Summary: Does what it says on the can: the patch adds a hash map to `VersionStorageInfo` that maps file numbers to file locations, i.e. (level, position in level) pairs. This will enable stricter consistency checks in `VersionBuilder`. The patch also fixes all the unit tests that used duplicate file numbers in a version (which would trigger an assertion with the new code). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6862 Test Plan: `make check` `make whitebox_crash_test` Reviewed By: riversand963 Differential Revision: D21670446 Pulled By: ltamasi fbshipit-source-id: 2eac249945cf33d8fb8597b26bfff5221e1a861a | 28 May 2020, 17:03:06 UTC |
bcefc59 | Akanksha Mahajan | 27 May 2020, 20:03:08 UTC | Allow MultiGet users to limit cumulative value size (#6826) Summary: 1. Add a value_size in read options which limits the cumulative value size of keys read in batches. Once the size exceeds read_options.value_size, all the remaining keys are returned with status Abort without further fetching any key. 2. Add a unit test case MultiGetBatchedValueSizeSimple the reads keys from memory and sst files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6826 Test Plan: 1. make check -j64 2. Add a new unit test case Reviewed By: anand1976 Differential Revision: D21471483 Pulled By: akankshamahajan15 fbshipit-source-id: dea51b8e76d5d1df38ece8cdb29933b1d798b900 | 27 May 2020, 20:07:14 UTC |
9060e6f | Adam Retter | 27 May 2020, 18:56:53 UTC | Add newer WBWI::NewIteratorWithBase functions to RocksJava (#6872) Summary: Exposes the `ReadOptions` arguments to `NewIteratorWithBase`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6872 Reviewed By: ajkr Differential Revision: D21725867 Pulled By: pdillinger fbshipit-source-id: 4079ba590cc13ba7a6244ed91439d89c40a543b6 | 27 May 2020, 18:59:12 UTC |
82a82c7 | Cheng Chang | 26 May 2020, 22:27:29 UTC | Fix potential memory leak of scratch buffer (#6879) Summary: If `req.scratch` is an internally allocated buffer, but `raw_block_contents` is not constructed to own `req.scratch`, then `req.scratch` will be leaked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6879 Test Plan: make asan_check Reviewed By: anand1976 Differential Revision: D21728498 Pulled By: cheng-chang fbshipit-source-id: 8fc6a4f2543918c565ddc16ecfad1807eb9a42cf | 26 May 2020, 22:29:04 UTC |
bd68bfb | Kefu Chai | 26 May 2020, 05:50:17 UTC | cmake: link env_librados_test against rados (#6855) Summary: otherwise we have FTBFS like: 2020-05-18T15:12:06.400 INFO:tasks.workunit.client.0.smithi032.stdout:[100%] Linking CXX executable env_librados_test 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: CMakeFiles/rocksdb_env_librados_test.dir/utilities/env_librados_test.cc.o: undefined reference to symbol '_ZN8librados7v14_2_05Rados4initEPKc@LIBRADOS_14.2.0' 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:/usr/bin/ld: /lib/librados.so.2: error adding symbols: DSO missing from command line 2020-05-18T15:12:06.620 INFO:tasks.workunit.client.0.smithi032.stderr:collect2: error: ld returned 1 exit status this addresses the regression introduced by 07204837ce8d66e1e6e4893178f3fd040f9c1044, which hides the symbols exposed by `${THIRDPARTY_LIBS}` from consumers of librocksdb Signed-off-by: Kefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6855 Reviewed By: riversand963 Differential Revision: D21621904 fbshipit-source-id: 7022ba4dc0003504401fce6f06547e4d74a32ac0 | 26 May 2020, 05:53:00 UTC |
b464a85 | Andrew Kryczka | 24 May 2020, 22:25:43 UTC | fix transaction rollback in db_stress TestMultiGet (#6873) Summary: There were further uses of `txn` after `RollbackTxn(txn)` leading to stress test errors. Moved the rollback to the end of the function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6873 Test Plan: found a command from the crash test that previously failed immediately under TSAN; verified now it succeeds. ``` ./db_stress --acquire_snapshot_one_in=10000 --allow_concurrent_memtable_write=1 --avoid_flush_during_recovery=0 --avoid_unnecessary_blocking_io=1 --block_size=16384 --bloom_bits=222.913637674 --bottommost_compression_type=none --cache_index_and_filter_blocks=1 --cache_size=1048576 --checkpoint_one_in=0 --checksum_type=kCRC32c --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_style=1 --compaction_ttl=0 --compression_max_dict_bytes=0 --compression_parallel_threads=1 --compression_type=zstd --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --db=/dev/shm/rocksdb/rocksdb_crashtest_whitebox --db_write_buffer_size=1048576 --delpercent=5 --delrangepercent=0 --destroy_db_initially=0 --disable_wal=0 --enable_pipelined_write=0 --flush_one_in=1000000 --format_version=5 --get_current_wal_file_one_in=0 --get_live_files_one_in=1000000 --get_sorted_wal_files_one_in=0 --index_block_restart_interval=12 --index_type=2 --key_len_percent_dist=1,30,69 --level_compaction_dynamic_level_bytes=True --log2_keys_per_lock=22 --long_running_snapshots=0 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=1000000 --max_key_len=3 --max_manifest_file_size=1073741824 --max_write_batch_group_size_bytes=1048576 --max_write_buffer_number=3 --memtablerep=skip_list --mmap_read=1 --mock_direct_io=False --nooverwritepercent=1 --num_levels=1 --open_files=100 --ops_per_thread=200000 --partition_filters=1 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefixpercent=5 --progress_reports=0 --read_fault_one_in=0 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --snapshot_hold_ops=100000 --subcompactions=4 --sync=0 --sync_fault_injection=False --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=0 --txn_write_policy=1 --unordered_write=1 --use_block_based_filter=0 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=1 --use_merge=1 --use_multiget=1 --use_txn=1 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --write_buffer_size=4194304 --write_dbid_to_manifest=0 --writepercent=35 ``` Reviewed By: pdillinger Differential Revision: D21708338 Pulled By: ajkr fbshipit-source-id: dcf55cddee0a14f429a75e7a8a505acf8025f2b1 | 24 May 2020, 22:27:24 UTC |
b559dba | Andrew Kryczka | 24 May 2020, 04:21:55 UTC | pin image version in circle CI builds (#6876) Summary: somehow the windows-server-2019-vs2019 image changed in a way that made VS 14 2015 the default. This caused an error when we specify VS 16 2019 as the cmake generator. I could not figure out the right arguments/env vars to get the latest VS working so pinned the image to the previous version instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6876 Reviewed By: pdillinger Differential Revision: D21709679 Pulled By: ajkr fbshipit-source-id: 2d16819ad239b4611fa199547744e1c101dc9da0 | 24 May 2020, 04:23:27 UTC |
803a517 | Peter Dillinger | 23 May 2020, 13:48:40 UTC | Misc things for ASSERT_STATUS_CHECKED, also gcc 4.8.5 (#6871) Summary: * Print stack trace on status checked failure * Make folly_synchronization_distributed_mutex_test a parallel test * Disable ldb_test.py and rocksdb_dump_test.sh with ASSERT_STATUS_CHECKED (broken) * Fix shadow warning in random_access_file_reader.h reported by gcc 4.8.5 (ROCKSDB_NO_FBCODE), also https://github.com/facebook/rocksdb/issues/6866 * Work around compiler bug on max_align_t for gcc < 4.9 * Remove an apparently wrong comment in status.h * Use check_some in Travis config (for proper diagnostic output) * Fix ignored Status in loop in options_helper.cc Pull Request resolved: https://github.com/facebook/rocksdb/pull/6871 Test Plan: manual, CI Reviewed By: ajkr Differential Revision: D21706619 Pulled By: pdillinger fbshipit-source-id: daf6364173d6689904eb394461a69a11f5bee2cb | 23 May 2020, 13:53:37 UTC |
bcd3256 | Marek Kurdej | 22 May 2020, 18:16:43 UTC | Fix warning -Wextra-semi. NFC. (#6869) Summary: Minor fix. CLA signed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6869 Reviewed By: ajkr Differential Revision: D21704001 Pulled By: pdillinger fbshipit-source-id: 57fd08114f3234f51f34758e25e708cc70962582 | 22 May 2020, 18:20:13 UTC |
35a25a3 | Peter Dillinger | 22 May 2020, 18:15:44 UTC | Fix/expand ASSERT_STATUS_CHECKED build, add to Travis (#6870) Summary: Fixed some option handling code that recently broke the ASSERT_STATUS_CHECKED build for options_test. Added all other existing tests that pass under ASSERT_STATUS_CHECKED to the whitelist. Added a Travis configuration to run all whitelisted tests with ASSERT_STATUS_CHECKED. (Someday we might enable this check by default in debug builds.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6870 Test Plan: ASSERT_STATUS_CHECKED=1 make check, Travis Reviewed By: ajkr Differential Revision: D21704374 Pulled By: pdillinger fbshipit-source-id: 15daef98136a19d7a6843fa0c9ec08738c2ac693 | 22 May 2020, 18:17:29 UTC |
826295a | mrambacher | 21 May 2020, 21:46:09 UTC | Change autovector to have a reserved size in LITE mode (#6868) Summary: Previously in LITE mode, an autovector did not have a reserved size. When elements were added to the vector, the underlying array could be reallocated. There was a set of code that never expands the autovector and was doing &autovector::back(). When the vector is resized, the old addresses may become invalid, causing a later exception to be thrown. By reserving space in the autovector up front, this problem is eliminated for those uses where the vector will never exceed the initial size. the resize happens, these pointers become invalid, leading to SEGV or other exceptions. This change allows the autovector to be fully populated before we take the address of any of its elements, thereby elminating the potential for a resize. There is comparable code to this change in Version::MultiGet for dealing with the context objects. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6868 Reviewed By: ajkr Differential Revision: D21693505 Pulled By: cheng-chang fbshipit-source-id: e71d516b15e08f202593cb80f2a42f048fc95768 | 21 May 2020, 21:48:10 UTC |
292bcf6 | Andrew Kryczka | 21 May 2020, 20:55:18 UTC | skip direct I/O tests in rocksdb lite (#6867) Summary: Fix a couple places where direct I/O was used even though it is unsupported in lite builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6867 Test Plan: `LITE=1 make check -j48` Reviewed By: pdillinger Differential Revision: D21689185 Pulled By: ajkr fbshipit-source-id: 3eaa3abf69cd7d0bcaabbcad3bb5a26fb8dd7301 | 21 May 2020, 20:57:17 UTC |
38be686 | mrambacher | 21 May 2020, 17:56:40 UTC | Add Struct Type to OptionsTypeInfo (#6425) Summary: Added code for generically handing structs to OptionTypeInfo. A struct is a collection of variables handled by their own map of OptionTypeInfos. Examples of structs include Compaction and Cache options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6425 Reviewed By: siying Differential Revision: D21668789 Pulled By: zhichao-cao fbshipit-source-id: 064b110de39dadf82361ed4663f7ac1a535b0b07 | 21 May 2020, 17:58:39 UTC |
c7aedf1 | Peter Dillinger | 21 May 2020, 15:10:43 UTC | Clean up some code related to file checksums (#6861) Summary: * Add missing unit test for schema stability of FileChecksumGenCrc32c (previously was only comparing to itself) * A lot of clarifying comments * Add some assertions for preconditions * Rename WritableFileWriter::CalculateFileChecksum -> UpdateFileChecksum * Simplify FileChecksumGenCrc32c with shared functions * Implement EndianSwapValue to replace unused EndianTransform And incidentally since I had trouble with 'make check-format' GitHub action disagreeing with local run, * Output full diagnostic information when 'make check-format' fails in CI Pull Request resolved: https://github.com/facebook/rocksdb/pull/6861 Test Plan: new unit test passes before & after other changes Reviewed By: zhichao-cao Differential Revision: D21667115 Pulled By: pdillinger fbshipit-source-id: 6a99970f87605aa024fa540c78cd519ff322c3e6 | 21 May 2020, 15:12:51 UTC |
eb04bb8 | anand76 | 20 May 2020, 21:45:03 UTC | Fix a bug in crash_test_with_txn (#6860) Summary: In NoBatchedOpsStress::TestMultiGet, call txn->Get() when transactions are in use. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6860 Test Plan: make crash_test_with_txn Reviewed By: pdillinger Differential Revision: D21667249 Pulled By: anand1976 fbshipit-source-id: 194bd7b9630a8efc3ae29d85422a61214e9e200e | 20 May 2020, 21:47:05 UTC |
545e14b | Zhichao Cao | 20 May 2020, 18:53:49 UTC | Generate file checksum in SstFileWriter (#6859) Summary: If Option.file_checksum_gen_factory is set, rocksdb generates the file checksum during flush and compaction based on the checksum generator created by the factory and store the checksum and function name in vstorage and Manifest. This PR enable file checksum generation in SstFileWrite and store the checksum and checksum function name in the ExternalSstFileInfo, such that application can use them for other purpose, for example, ingest the file checksum with files in IngestExternalFile(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6859 Test Plan: add unit test and pass make asan_check. Reviewed By: ajkr Differential Revision: D21656247 Pulled By: zhichao-cao fbshipit-source-id: 78a3570c76031d8832e3d2de3d6c79cdf2b675d0 | 20 May 2020, 18:55:31 UTC |
aaafcb8 | Peter Dillinger | 20 May 2020, 18:35:28 UTC | Use in-repo gtest in buck build (#6858) Summary: ... so that we have freedom to upgrade it (see https://github.com/facebook/rocksdb/issues/6808). As a side benefit, gtest will no longer be linked into main library in buck build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6858 Test Plan: fb internal build & link Reviewed By: riversand963 Differential Revision: D21652061 Pulled By: pdillinger fbshipit-source-id: 6018104af944debde576b5beda6c134e737acedb | 20 May 2020, 18:37:45 UTC |
a1523ef | Akanksha Mahajan | 20 May 2020, 02:20:29 UTC | Status check enforcement for io_posix_test and options_settable_test (#6857) Summary: Added status check enforcement for io_posix_test and options_settable_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6857 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 check Reviewed By: ajkr Differential Revision: D21647904 Pulled By: akankshamahajan15 fbshipit-source-id: b7f2321eb6c141a88cd5e1270ecb7d58f00341af | 20 May 2020, 02:22:28 UTC |
39b2443 | anand76 | 19 May 2020, 19:41:43 UTC | Strengthen MultiGet correctness verification in NoBatchedOpsStress (#6849) Summary: Add MultiGet to VerifyDb and check consistency with Get in TestMultiGet. Test plan - make crash_test ASAN crash test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6849 Reviewed By: pdillinger Differential Revision: D21635011 Pulled By: anand1976 fbshipit-source-id: deb5a79d08fefd8d8010204f1f20b83adc92310e | 19 May 2020, 19:47:22 UTC |
1551f10 | Levi Tamasi | 19 May 2020, 16:57:49 UTC | Refactor the blob file related logic in VersionBuilder (#6835) Summary: This patch is groundwork for an upcoming change to store the set of linked SSTs in `BlobFileMetaData`. With the current code, a new `BlobFileMetaData` object is created each time a `VersionEdit` touches a certain blob file. This is fine as long as these objects are lightweight and cheap to create; however, with the addition of the linked SST set, it would be very inefficient since the set would have to be copied over and over again. Note that this is the same kind of problem that `VersionBuilder` is solving w/r/t `Version`s and files, and we can apply the same solution; that is, we can accumulate the changes in a different mutable object, and apply the delta in one shot when the changes are committed. The patch does exactly that by adding a new `BlobFileMetaDataDelta` class to `VersionBuilder`. In addition, it turns the existing `GetBlobFileMetaData` helper into `IsBlobFileInVersion` (which is fine since that's the only thing the method's clients care about now), and adds a couple of helper methods that can create a `BlobFileMetaData` object from the `BlobFileMetaData` in the base (if applicable) and the delta when the `Version` is saved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6835 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21505187 Pulled By: ltamasi fbshipit-source-id: d81a48c5f2ca7b79d7124c935332a6bcf3d5d988 | 19 May 2020, 17:00:04 UTC |
0ac0098 | mrambacher | 19 May 2020, 16:20:21 UTC | Make options length longer for sst_dump_test (#6846) Summary: Under MacOS when running with make -j 8 check, the temporary directory generated was > 100 characters. This caused the tests to do nothing under MacOS. Most of them still reported success for doing nothing, but ReadaheadSize was expecting the test to run. By making the option name longer, the tests will no run successfully (and do something!) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6846 Reviewed By: ajkr Differential Revision: D21576032 fbshipit-source-id: b089cde0d598137b572aa8527cc5459085252af7 | 19 May 2020, 16:22:12 UTC |
ada700b | Cheng Chang | 19 May 2020, 00:23:22 UTC | Re-read the whole request in direct IO mode when IO uring returns partial result (#6853) Summary: If both direct IO and IO uring are enabled, when IO uring returns partial result, we'll try to read the remaining part of the request, but the starting address/offset of the remaining part might not be aligned to the block size, in direct IO mode, the unaligned offset causes bug. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6853 Test Plan: run make check with both direct IO and IO uring enabled, this is covered by one of the continuous tests. Reviewed By: anand1976 Differential Revision: D21603023 Pulled By: cheng-chang fbshipit-source-id: 942f6a11ff21e1892af6c4464e02bab4c707787c | 19 May 2020, 00:25:57 UTC |
b9d65f5 | Cheng Chang | 18 May 2020, 15:39:33 UTC | Trigger compaction in CompactOnDeletionCollector based on deletion ratio (#6806) Summary: In level compaction, if the total size (even if compensated after taking account of the deletions) of a level hasn't exceeded the limit, but there are lots of deletion entries in some SST files of the level, these files should also be good candidates for compaction. Otherwise, queries for the deleted keys might be slow because they need to go over all the tombstones. This PR adds an option `deletion_ratio` to the factory of `CompactOnDeletionCollector` to configure it to trigger compaction when the ratio of tombstones >= `deletion_ratio`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6806 Test Plan: Added new unit test in `compact_on_deletion_collector_test.cc`. make compact_on_deletion_collector_test && ./compact_on_deletion_collector_test Reviewed By: ajkr Differential Revision: D21511981 Pulled By: cheng-chang fbshipit-source-id: 65a9d0150e8c9c00337787686475252e4535a3e1 | 18 May 2020, 15:42:05 UTC |
d790e60 | Yanqin Jin | 17 May 2020, 04:46:21 UTC | Fix buck target db_stress_lib in opt mode (#6847) Summary: In buck build with opt mode, target should not include rocksdb_test_lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6847 Test Plan: Watch for internal cont build. Reviewed By: ajkr Differential Revision: D21586803 Pulled By: riversand963 fbshipit-source-id: 76d253c18d16fac6cab86a8c3f6b471ad5b6efb3 | 17 May 2020, 04:48:20 UTC |
50ba245 | Peter Dillinger | 15 May 2020, 20:57:34 UTC | Use 'make all' in LITE Travis configuration (#6834) Summary: So that we don't miss LITE compilation errors in tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/6834 Reviewed By: anand1976 Differential Revision: D21503227 Pulled By: pdillinger fbshipit-source-id: 3b2fdf3c4d395354d0ababac06da32addbafb3a5 | 15 May 2020, 20:59:24 UTC |
91b7553 | Cheng Chang | 15 May 2020, 06:23:32 UTC | Enable IO Uring in MultiGet in direct IO mode (#6815) Summary: Currently, in direct IO mode, `MultiGet` retrieves the data blocks one by one instead of in parallel, see `BlockBasedTable::RetrieveMultipleBlocks`. Since direct IO is supported in `RandomAccessFileReader::MultiRead` in https://github.com/facebook/rocksdb/pull/6446, this PR applies `MultiRead` to `MultiGet` so that the data blocks can be retrieved in parallel. Also, in direct IO mode and when data blocks are compressed and need to uncompressed, this PR only allocates one continuous aligned buffer to hold the data blocks, and then directly uncompress the blocks to insert into block cache, there is no longer intermediate copies to scratch buffers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6815 Test Plan: 1. added a new unit test `BlockBasedTableReaderTest::MultiGet`. 2. existing unit tests and stress tests contain tests against `MultiGet` in direct IO mode. Reviewed By: anand1976 Differential Revision: D21426347 Pulled By: cheng-chang fbshipit-source-id: b8446ae0e74152444ef9111e97f8e402ac31b24f | 15 May 2020, 06:26:26 UTC |
b11a8b1 | Yanqin Jin | 15 May 2020, 01:48:06 UTC | Fix valgrind error by init memory region (#6842) Summary: As title. After allocating a memory buffer, initialize its content to 0s. This fixes valgrind issue introduced in https://github.com/facebook/rocksdb/issues/6709. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6842 Test Plan: ``` $make valgrind_test $valgrind --tool=memcheck --track-origins=yes ./db_test2 --gtest_filter=DBTest2.CompressionFailures ``` Reviewed By: pdillinger Differential Revision: D21551848 Pulled By: riversand963 fbshipit-source-id: e87a6f413e3f3d92d8e23d8ecc4cf93479c6674c | 15 May 2020, 01:50:03 UTC |
06a2dce | Zhichao Cao | 14 May 2020, 21:56:10 UTC | Move checksum calculation ahead of memory copy (#6844) Summary: Originally, the checksum of appended data in writable file writer is calculated after the data is copied to the buffer. It will not be able to catch the bit flip happens during copy. Move the checksum calculation before it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6844 Test Plan: pass make asan_check Reviewed By: cheng-chang Differential Revision: D21576726 Pulled By: zhichao-cao fbshipit-source-id: 0a062a1f19886f6ea0d4e3f557e6f4b799773254 | 14 May 2020, 21:58:14 UTC |
50d63a2 | anand76 | 13 May 2020, 17:44:16 UTC | Fix LITE build failure in compaction_picker_test (#6839) Summary: Fix LITE build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6839 Test Plan: LITE=1 make check Reviewed By: riversand963 Differential Revision: D21535808 Pulled By: anand1976 fbshipit-source-id: fcad961eca08e13fb0c256c92d18c3c1f1165f22 | 13 May 2020, 17:47:15 UTC |
3bea276 | Yanqin Jin | 13 May 2020, 04:35:08 UTC | Do not print u'string' in TARGETS file (#6841) Summary: Before this PR, extra deps passed in from cmd line to buckifier will be parsed and used to populate a dict. Using this dict and printing to TARGETS file will lead to printing u'', disallowed by build tools. This PR removes the u''. Test Plan (local dev server): ``` python buckifier/buckify_rocksdb.py '{"fake": {"extra_deps": [":test_dep", "//fake/module:mock1"], "extra_compiler_flags": ["-Os", "-DROCKSDB_LITE"]}}' ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6841 Reviewed By: siying Differential Revision: D21538155 Pulled By: riversand963 fbshipit-source-id: 09403668a4aa1a15bad7dac229c2bc8ce8ee1349 | 13 May 2020, 04:37:31 UTC |
244797a | Tongliang Liao | 13 May 2020, 04:16:50 UTC | Add `find_dependency()` in cmake config file. (#6791) Summary: Currently when building PyTorch with latest RocksDB we get errors like "missing target Snappy::snappy", because they are simply not there. With old `${VAR}` approach we essentially hard-code the abs path found during RocksDB build, which is: - Not relocatable. - Doesn't work when changed to modern target-based design because that requires target to present when used for expansion. This fix allows cmake to setup imported target, if enabled during RocksDB build, when downstream uses `find_package(RocksDB)`. This is for https://github.com/facebook/rocksdb/issues/6179 tchaikov Please help review, thanks! Pull Request resolved: https://github.com/facebook/rocksdb/pull/6791 Reviewed By: riversand963 Differential Revision: D21471553 fbshipit-source-id: 8d4ff2ab589a97ca6e6ba27e1f17b97a00f06206 | 13 May 2020, 04:18:29 UTC |
0720483 | Tongliang Liao | 13 May 2020, 04:05:50 UTC | Mark dependencies as PRIVATE and fix missing dependencies in tools. (#6790) Summary: Tools were mistakenly using leaked (PUBLIC) dependencies from main lib. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6790 Reviewed By: riversand963 Differential Revision: D21471551 fbshipit-source-id: ec43b92e231777e0fcf0f865444391af09d6963b | 13 May 2020, 04:07:55 UTC |
4a4b8a1 | sdong | 13 May 2020, 01:21:32 UTC | sst_dump to reduce number of file reads (#6836) Summary: sst_dump can issue many file reads from the file system. This doesn't work well with file systems without a OS cache, especially remote file systems. In order to mitigate this problem, several improvements are done: 1. --readahead_size is added, so that users can specify readahead size when scanning the data. 2. Force a 512KB tail readahead, which prevents three I/Os for footer, meta index and property blocks and hopefully index and filter blocks too. 3. Consoldiate SSTDump's I/Os before opening the file for read. Use the same file prefetch buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6836 Test Plan: Add a test that covers this new feature. Reviewed By: pdillinger Differential Revision: D21516607 fbshipit-source-id: 3ae43526286f67b2f4a5bdedfbc92719d579b87e | 13 May 2020, 01:23:33 UTC |
70aaa9c | Stanislav Tkach | 12 May 2020, 21:48:14 UTC | Expose CancellAllBackgroundWork to C api (#6832) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6832 Reviewed By: cheng-chang Differential Revision: D21498186 Pulled By: riversand963 fbshipit-source-id: 66bb0d7c06af2bf0df3c6a09b61bca2fb81f2dd3 | 12 May 2020, 21:50:52 UTC |
c384c08 | Ziyue Yang | 12 May 2020, 16:25:21 UTC | Add tests for compression failure in BlockBasedTableBuilder (#6709) Summary: Currently there is no check for whether BlockBasedTableBuilder will expose compression error status if compression fails during the table building. This commit adds fake faulting compressors and a unit test to test such cases. This check finds 5 bugs, and this commit also fixes them: 1. Not handling compression failure well in BlockBasedTableBuilder::BGWorkWriteRawBlock. 2. verify_compression failing in BlockBasedTableBuilder when used with ZSTD. 3. Wrongly passing the same reference of block contents to BlockBasedTableBuilder::CompressAndVerifyBlock in parallel compression. 4. Wrongly setting block_rep->first_key_in_next_block to nullptr in BlockBasedTableBuilder::EnterUnbuffered when there are still incoming data blocks. 5. Not maintaining variables for compression ratio estimation and first_block in BlockBasedTableBuilder::EnterUnbuffered. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6709 Reviewed By: ajkr Differential Revision: D21236254 fbshipit-source-id: 101f6e62b2bac2b7be72be198adf93cd32a1ff46 | 12 May 2020, 16:27:35 UTC |
3f21807 | yetingsky | 11 May 2020, 21:56:36 UTC | fix typo (#6831) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6831 Reviewed By: cheng-chang Differential Revision: D21499149 Pulled By: zhichao-cao fbshipit-source-id: 2cb76cbf7086677d8cad5c828019e008062f0052 | 11 May 2020, 21:58:25 UTC |
2e93247 | Sagar Vemuri | 11 May 2020, 20:28:16 UTC | Disable a few timer tests (#6833) Summary: Disable `TimerTest.SingleScheduleRepeatedlyTest` and `TimerTest.MultipleScheduleRepeatedlyTest`. This is to help people to not hit any hangs (https://github.com/facebook/rocksdb/issues/6698) during their development process while I investigate further; I could not reproduce the issue on my dev machine yet. Note that timer is not being utilized anywhere yet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6833 Test Plan: ``` svemuri@devbig187 ~/rocksdb (timer-disable-test) $ TEST_TMPDIR=/dev/shm ./timer_test [==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from TimerTest [ RUN ] TimerTest.SingleScheduleOnceTest [ OK ] TimerTest.SingleScheduleOnceTest (1 ms) [ RUN ] TimerTest.MultipleScheduleOnceTest [ OK ] TimerTest.MultipleScheduleOnceTest (0 ms) [----------] 2 tests from TimerTest (1 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (1 ms total) [ PASSED ] 2 tests. YOU HAVE 2 DISABLED TESTS ``` Reviewed By: pdillinger Differential Revision: D21502474 Pulled By: sagar0 fbshipit-source-id: ac67caee2011fd14ffb2476a8914a6286a4f9abe | 11 May 2020, 20:30:00 UTC |
f0e8731 | Tongliang Liao | 11 May 2020, 17:26:18 UTC | Use GFlags/Snappy config in CMake, with fallback for legacy approach. (#6771) Summary: Related to some discussion in https://github.com/facebook/rocksdb/issues/6179 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6771 Reviewed By: zhichao-cao Differential Revision: D21340117 fbshipit-source-id: a1af0ba4865bb13c8c817851d6f6c4056191b3fe | 11 May 2020, 17:28:49 UTC |
3a1c29d | Derrick Pallas | 09 May 2020, 03:48:17 UTC | Add missing my_pid to fprintf in multi_process_example (#6731) Summary: Signed-off-by: Derrick Pallas <derrick@pallas.us> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6731 Reviewed By: siying Differential Revision: D21137005 Pulled By: riversand963 fbshipit-source-id: a7182e1bec225bc110971f40e2d0e6c3a671c061 | 09 May 2020, 03:49:33 UTC |
a50ea71 | sdong | 08 May 2020, 21:12:18 UTC | Improve ldb consistency checks (#6802) Summary: When using ldb, users cannot turn on force consistency check in most commands, while they cannot use checksonsistnecy with --try_load_options. The change fixes both by: 1. checkconsistency now calls OpenDB() so that it gets all the options loading and sanitized options logic 2. use options.check_consistency_checks = true by default, and add a --disable_consistency_checks to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6802 Test Plan: Add a new unit test. Some manual tests with corrupted DBs. Reviewed By: pdillinger Differential Revision: D21388051 fbshipit-source-id: 8d122732d391b426e3982a1c3232a8e3763ffad0 | 08 May 2020, 21:17:47 UTC |
d9cd335 | sdong | 08 May 2020, 21:11:03 UTC | Suppress UBSAN warning in CRC32 ARM (#6827) Summary: UBSAN shows following warning: util/crc32c_arm64.cc:111:11: runtime error: load of misaligned address 0x00001afcda86 for type 'const uint64_t', which requires 8 byte alignment 0x00001afcda86: note: pointer points here cc c1 2d 00 01 81 40 24 30 66 39 66 30 37 30 63 2d 32 36 63 34 2d 34 62 61 61 2d 38 35 33 31 2d ^ Suppress it just as what we do in x86 CRC. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6827 Test Plan: Run the same UBSAN and see it to pass now. Reviewed By: ltamasi Differential Revision: D21471838 fbshipit-source-id: 02943dd39a7030d2b03e5d894dcb23ed72b6c9c3 | 08 May 2020, 21:14:00 UTC |
e72e216 | Yanqin Jin | 08 May 2020, 19:59:02 UTC | Fix a few bugs in best-efforts recovery (#6824) Summary: 1. Update column_family_memtables_ to point to latest column_family_set in version_set after recovery. 2. Normalize file paths passed by application so that directories end with '/' or '\\'. 3. In addition to missing files, corrupted files are also ignored in best-efforts recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6824 Test Plan: COMPILE_WITH_ASAN=1 make check Reviewed By: anand1976 Differential Revision: D21463905 Pulled By: riversand963 fbshipit-source-id: c48db8843cc93c8c1c7139c474b64e6f775307d2 | 08 May 2020, 20:01:42 UTC |
1c84660 | Andrew Kryczka | 08 May 2020, 19:36:49 UTC | prototype status check enforcement (#6798) Summary: Tried making Status object enforce that it is checked in some way. In cases it is not checked, `PermitUncheckedError()` must be called explicitly. Added a way to run tests (`ASSERT_STATUS_CHECKED=1 make -j48 check`) on a whitelist. The effort appears significant to get each test to pass with this assertion, so I only fixed up enough to get one test (`options_test`) working and added it to the whitelist. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6798 Reviewed By: pdillinger Differential Revision: D21377404 Pulled By: ajkr fbshipit-source-id: 73236f9c8df38f01cf24ecac4a6d1661b72d077e | 08 May 2020, 19:40:43 UTC |
1282589 | sdong | 08 May 2020, 03:50:14 UTC | Disable "compression_parallel_threads" in crash test for now (#6816) Summary: "compressio_parallel_threads" caused several test failure tests. To keep crash test clean, disable it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6816 Test Plan: "make crash_test" to make sure the python script doesn't break Reviewed By: zhichao-cao Differential Revision: D21462112 fbshipit-source-id: 9eecc764800da82cd19665dc8b167eacead3310b | 08 May 2020, 03:52:29 UTC |
9426523 | anand76 | 08 May 2020, 00:30:41 UTC | Fix race due to delete triggered compaction in Universal compaction mode (#6799) Summary: Delete triggered compaction in universal compaction mode was causing a corruption when scheduled in parallel with other compactions. 1. When num_levels = 1, a file marked for compaction may be picked along with all older files in L0, without checking if any of them are already being compaction. This can cause unpredictable results like resurrection of older versions of keys or deleted keys. 2. When num_levels > 1, a delete triggered compaction would not get scheduled if it overlaps with a running regular compaction. However, the reverse is not true. This is due to the fact that in ```UniversalCompactionBuilder::CalculateSortedRuns```, it assumes that entire sorted runs are picked for compaction and only checks the first file in a sorted run to determine conflicts. This is violated by a delete triggered compaction as it works on a subset of a sorted run. Fix the bug for num_levels > 1, and disable the feature for now when num_levels = 1. After disabling this feature, files would still get marked for compaction, but no compaction would get scheduled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6799 Reviewed By: pdillinger Differential Revision: D21431286 Pulled By: anand1976 fbshipit-source-id: ae9f0bdb1d6ae2f10284847db731c23f43af164a | 08 May 2020, 00:32:17 UTC |
3730b05 | Andrew Kryczka | 07 May 2020, 23:37:47 UTC | Fixup HISTORY.md for e9ba4ba "validate range tombstone covers positiv… (#6825) Summary: …e range" Moved it from the wrong section (6.10) to the right section (Unreleased). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6825 Reviewed By: zhichao-cao Differential Revision: D21464577 Pulled By: ajkr fbshipit-source-id: a836b4ab10be2464182826f9411c9c424c933b70 | 07 May 2020, 23:40:17 UTC |
f286fb3 | anand76 | 07 May 2020, 22:52:58 UTC | Include options.h in table.h (#6823) Summary: https://github.com/facebook/rocksdb/issues/6389 replaced the #include of options.h in table.h with forward declarations, which is causing some build failures in RocksDB users in 6.10. Remove the forward declarations and #include options.h as recommended by the style guide - https://google.github.io/styleguide/cppguide.html#Forward_Declarations Pull Request resolved: https://github.com/facebook/rocksdb/pull/6823 Reviewed By: riversand963 Differential Revision: D21464078 Pulled By: anand1976 fbshipit-source-id: 6033ee2544d279690f57bb0db91bc83816cee11d | 07 May 2020, 22:55:29 UTC |
b27a144 | Peter Dillinger | 07 May 2020, 22:39:49 UTC | Fix false NotFound from batched MultiGet with kHashSearch (#6821) Summary: The error is assigning KeyContext::s to NotFound status in a table reader for a "not found in this table" case, which skips searching in later tables, like only a delete should. (The hash search index iterator is the only one that can return status NotFound even if Valid() == false.) This was detected by intermittent failure in MultiThreadedDBTest.MultiThreaded/5, a kHashSearch configuration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6821 Test Plan: modified existing unit test to reproduce problem Reviewed By: anand1976 Differential Revision: D21450469 Pulled By: pdillinger fbshipit-source-id: 7478003684d637dbd491cdac81468041a791be2c | 07 May 2020, 22:41:37 UTC |
e9ba4ba | Andrew Kryczka | 07 May 2020, 18:53:32 UTC | validate range tombstone covers positive range (#6788) Summary: We found some files containing nothing but negative range tombstones, and unsurprisingly their metadata specified a negative range, which made things crash. Time to add a bit of user input validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6788 Reviewed By: zhichao-cao Differential Revision: D21343719 Pulled By: ajkr fbshipit-source-id: f1c16e4c3e9fa150958c8c866176632a3206fb74 | 07 May 2020, 18:55:30 UTC |
ac3ae1d | Levi Tamasi | 07 May 2020, 16:29:21 UTC | Find/purge obsolete blob files (#6807) Summary: The patch extends `FindObsoleteFiles` and `PurgeObsoleteFiles` with support for blob files. The behavior is analogous to SST files: obsolete blob files are put on the "candidates for deletion" list, while live (and pending) files are preserved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6807 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21406249 Pulled By: ltamasi fbshipit-source-id: 1948f71c31927564b61e8af394f50ca3964880d9 | 07 May 2020, 16:32:51 UTC |
1f20df2 | Andrew Kryczka | 07 May 2020, 01:06:04 UTC | cover single level universal in crash test (#6818) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6818 Test Plan: fast whitebox test and verify there are some single-level universal and some multi-level universal runs. ``` $ python ./tools/db_crashtest.py whitebox --simple -max_key=1000000 -value_size_mult=33 -write_buffer_size=524288 -target_file_size_base=524288 -max_bytes_for_level_base=2097152 --duration=120 --interval=10 --ops_per_thread=1000 --random_kill_odd=887 ``` Reviewed By: riversand963 Differential Revision: D21432138 Pulled By: ajkr fbshipit-source-id: 2fc5ba9f3dfa49bb11e81da7dd00a17b476e64d7 | 07 May 2020, 01:08:09 UTC |
c1e1185 | anand76 | 06 May 2020, 23:40:32 UTC | Update release version to 6.10 (#6797) Summary: Update HISTORY.md and version.h to 6.10. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6797 Reviewed By: zhichao-cao Differential Revision: D21371390 Pulled By: anand1976 fbshipit-source-id: 6017bca24fc5d12076d1ddaec7783c9b85712d42 | 06 May 2020, 23:42:37 UTC |
06c3b85 | Levi Tamasi | 06 May 2020, 20:58:25 UTC | Disallow using the base DB's storage directory as blob_dir in BlobDB (#6810) Summary: https://github.com/facebook/rocksdb/pull/6807 extends the logic that identifies and purges obsolete files to blob files handled by RocksDB itself. In order to prevent that from interfering with the current BlobDB code, we need to make sure that `BlobDBOptions::blob_dir` is different from the storage directories used by the base DB. (Note: this is true by default.) The patch adds a check that explicitly disallows this configuration and returns `Status::NotSupported` from `BlobDB::Open` in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6810 Test Plan: Tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21412676 Pulled By: ltamasi fbshipit-source-id: 6630cc7481e48c8bf55d59423b25f14d52ffe681 | 06 May 2020, 21:00:46 UTC |
53f8447 | Neha Deodhar | 06 May 2020, 17:25:58 UTC | Add YugabyteDB to USERS (#6786) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6786 Reviewed By: zhichao-cao Differential Revision: D21407762 Pulled By: pdillinger fbshipit-source-id: 62f74409cc72c6b9ce4107aa515955cfc5eec12a | 06 May 2020, 17:28:29 UTC |
c21c459 | sdong | 06 May 2020, 01:29:50 UTC | Slightly expand converage to file consistency check failure (#6800) Summary: Current DBCompactionTest.ConsistencyFailTest checks DB fails after L0 inconsitency is found. Add slightly more coverage by introducing DBCompactionTest.ConsistencyFailTest2 which checks non-L0 files too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6800 Test Plan: Run the new test. Reviewed By: riversand963 Differential Revision: D21384806 fbshipit-source-id: 36db7b657eed42115283fe2f6afa4c3a31a3b510 | 06 May 2020, 01:31:53 UTC |
394f2bb | mrambacher | 05 May 2020, 22:02:04 UTC | Add OptionTypeInfo::Enum and related methods (#6423) Summary: Add methods and constructors for handling enums to the OptionTypeInfo. This change allows enums to be converted/compared without adding a special "type" to the OptionType. This change addresses a couple of issues: - It allows new enumerated types to be added to the options without editing the OptionType base class (and related methods) - It standardizes the procedure for adding enumerated types to the options, reducing potential mistakes - It moves the enum maps to the location where they are used, allowing them to be static file members rather than global values - It reduces the number of types and cases that need to be handled in the various OptionType methods Pull Request resolved: https://github.com/facebook/rocksdb/pull/6423 Reviewed By: siying Differential Revision: D21408713 Pulled By: zhichao-cao fbshipit-source-id: fc492af285d011822578b95d186a0fce25d35626 | 05 May 2020, 22:04:04 UTC |
a96461d | Andrew Kryczka | 05 May 2020, 21:52:18 UTC | fix swallowed error for file deletion consistency check (#6809) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6809 Reviewed By: pdillinger Differential Revision: D21411971 Pulled By: ajkr fbshipit-source-id: 900b6b0370b76e9a3e5e03f968e2ac1bbaab73b8 | 05 May 2020, 21:54:21 UTC |
2f1700c | Peter Dillinger | 05 May 2020, 20:09:12 UTC | Fix failure to write output in SpecialEnv::GetCurrentTime (#6803) Summary: This very old test code bug was causing a new valgrind failure in MultiGetDeadlineExceeded Also fix hang in MultiGetDeadlineExceeded by unifying with some logic from another test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6803 Test Plan: run that unit test under valgrind, make check Reviewed By: siying Differential Revision: D21388470 Pulled By: pdillinger fbshipit-source-id: 0ce99d6d5eb8cd3195b17406892c8c5cff5fa5dd | 05 May 2020, 20:11:29 UTC |
91bc013 | Cheng Chang | 05 May 2020, 18:07:52 UTC | Refactor level compaction picker (#6804) Summary: 1. refactor out PickFileToCompact to remove duplicated logic. 2. remove redundant checks of `start_level_inputs_.empty()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6804 Test Plan: make check Reviewed By: siying Differential Revision: D21390053 Pulled By: cheng-chang fbshipit-source-id: 185d5987a08bfdaf63f0f245310c6da69878d415 | 05 May 2020, 18:09:29 UTC |
5584595 | Yanqin Jin | 05 May 2020, 17:44:12 UTC | Do not swallow error returned from SaveTo() (#6801) Summary: With consistency check enabled, VersionBuilder::SaveTo() may return error once corruption is detected while building versions. We should handle these errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6801 Test Plan: make check Reviewed By: siying Differential Revision: D21385045 Pulled By: riversand963 fbshipit-source-id: 98f6424e2a4699b62befa21e9fe00e70a771118e | 05 May 2020, 17:46:20 UTC |
5a61e78 | Yanqin Jin | 05 May 2020, 00:43:33 UTC | Fix db_stress when GetLiveFiles() flushes dropped CF (#6805) Summary: Current impl. of db_stress will abort verification and report failure if GetLiveFiles() causes a dropped column family to be flushed. This is not desired. To fix, this PR makes the following change: In GetLiveFiles, if flush is triggered and returns Status::IsColumnFamilyDropped(), then set status to Status::OK(). This is OK because dropped column families will be skipped during the rest of this function, and valid column families will have their live files returned to caller. Test plan (dev server): make check ./db_stress -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 ./db_stress -disable_wal=1 -reopen=0 -ops_per_thread=1000 -get_live_files_one_in=100 -clear_column_family_one_in=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6805 Reviewed By: ltamasi Differential Revision: D21390044 Pulled By: riversand963 fbshipit-source-id: de67846b95a4f1b88aa0a30c3d70c43cc68625b9 | 05 May 2020, 00:45:49 UTC |
a00ddf1 | Levi Tamasi | 04 May 2020, 22:05:34 UTC | Expose the set of live blob files from Version/VersionSet (#6785) Summary: The patch adds logic that returns the set of live blob files from `Version::AddLiveFiles` and `VersionSet::AddLiveFiles` (in addition to live table files), and also cleans up the code a bit, for example, by exposing only the numbers of table files as opposed to the earlier `FileDescriptor`s that no clients used. Moreover, the patch extends the `GetLiveFiles` API so that it also exposes blob files in the current version. Similarly to https://github.com/facebook/rocksdb/pull/6755, this is a building block for identifying and purging obsolete blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6785 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21336210 Pulled By: ltamasi fbshipit-source-id: fc1aede8a49eacd03caafbc5f6f9ce43b6270821 | 04 May 2020, 22:08:13 UTC |
680c416 | sdong | 04 May 2020, 21:15:55 UTC | Avoid Swallowing Some File Consistency Checking Bugs (#6793) Summary: We are swallowing some file consistency checking failures. This is not expected. We are fixing two cases: DB reopen and manifest dump. More places are not fixed and need follow-up. Error from CheckConsistencyForDeletes() is also swallowed, which is not fixed in this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6793 Test Plan: Add a unit test to cover the reopen case. Reviewed By: riversand963 Differential Revision: D21366525 fbshipit-source-id: eb438a322237814e8d5125f916a3c6de97f39ded | 04 May 2020, 21:18:11 UTC |
d9e170d | Mian Qin | 04 May 2020, 17:53:15 UTC | Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper (#6795) Summary: Fix issues for reproducing synthetic ZippyDB workloads in the FAST20' paper using db_bench. Details changes as follows. 1, add a separate random mode in MixGraph to produce all_random workload. 2, fix power inverse function for generating prefix_dist workload. 3, make sure key_offset in prefix mode is always unsigned. note: Need to carefully choose key_dist_a/b to avoid aliasing. Power inverse function range should be close to overall key space. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6795 Reviewed By: akankshamahajan15 Differential Revision: D21371095 Pulled By: zhichao-cao fbshipit-source-id: 80744381e242392c8c7cf8ac3d68fe67fe876048 | 04 May 2020, 17:55:14 UTC |
211088d | Cheng Chang | 02 May 2020, 00:35:41 UTC | Remove redundant update of txn_state_ in transaction Prepare (#6778) Summary: When expiration is set in a pessimistic transaction, `txn_state_` is already updated to `AWAITING_PREPARE` in the `if (expiration_time_ > 0)` block, there is no need to update the state in `if (can_prepare)` block again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6778 Test Plan: make check Reviewed By: lth Differential Revision: D21335319 Pulled By: cheng-chang fbshipit-source-id: 251d634cc7d1a0e86e673a59f0bda8584da5a35f | 02 May 2020, 00:37:33 UTC |
c8643ed | Zhichao Cao | 01 May 2020, 07:01:21 UTC | Fix multiple CF replay failure in db_bench replay (#6787) Summary: The multiple CF hash map is not passed to the multi-thread worker. When using multi-thread replay for multiple CFs, it will cause segment fault. Pass the cf_map to the argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6787 Test Plan: pass trace replay test. Reviewed By: yhchiang Differential Revision: D21339941 Pulled By: zhichao-cao fbshipit-source-id: 434482b492287e6722c7cd5a706f057c5ec170ce | 01 May 2020, 07:03:38 UTC |
6acbbbf | Yanqin Jin | 01 May 2020, 02:20:55 UTC | Add Github Action for some basic sanity test of PR (#6761) Summary: Add Github Action to perform some basic sanity check for PR, inclding the following. 1) Buck TARGETS file. On the one hand, The TARGETS file is used for internal buck, and we do not manually update it. On the other hand, we need to run the buckifier scripts to update TARGETS whenever new files are added, etc. With this Github Action, we make sure that every PR does not forget this step. The GH Action uses a Makefile target called check-buck-targets. Users can manually run `make check-buck-targets` on local machine. 2) Code format We use clang-format-diff.py to format our code. The GH Action in this PR makes sure this step is not skipped. The checking script build_tools/format-diff.sh assumes that `clang-format-diff.py` is executable. On host running GH Action, it is difficult to download `clang-format-diff.py` and make it executable. Therefore, we modified build_tools/format-diff.sh to handle the case in which there is a non-executable clang-format-diff.py file in the top-level rocksdb repo directory. Test Plan (Github and devserver): Watch for Github Action result in the `Checks` tab. On dev server ``` make check-format make check-buck-targets make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6761 Test Plan: Watch for Github Action result in the `Checks` tab. Reviewed By: pdillinger Differential Revision: D21260209 Pulled By: riversand963 fbshipit-source-id: c646e2f37c6faf9f0614b68aa0efc818cff96787 | 01 May 2020, 02:22:45 UTC |
6504ae0 | sdong | 30 April 2020, 23:59:16 UTC | Remove the support of setting CompressionOptions.parallel_threads from string for now (#6782) Summary: The current way of implementing CompressionOptions.parallel_threads introduces a format change. We plan to change CompressionOptions's serailization format to a new JSON-like format, which would be another format change. We would like to consolidate the two format changes into one, rather than making some users to change twice. Hold CompressionOptions.parallel_threads from being supported by option string for now. Will add it back after the general CompressionOptions's format change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6782 Test Plan: Run all existing tests. Reviewed By: zhichao-cao Differential Revision: D21338614 fbshipit-source-id: bca2dac3cb37d4e6e64b52cbbe8ea749cd848685 | 01 May 2020, 00:01:17 UTC |
ef0c3ed | Cheng Chang | 30 April 2020, 23:21:05 UTC | Make users explicitly be aware of prepare before commit (#6775) Summary: In current commit protocol of pessimistic transaction, if the transaction is not prepared before commit, the commit protocol implicitly assumes that the user wants to commit without prepare. This PR adds TransactionOptions::skip_prepare, the default value is `true` because if set to `false`, all existing users who commit without prepare need to update their code to set skip_prepare to true. Although this does not force the user to explicitly express their intention of skip_prepare, it at least lets the user be aware of the assumption of being able to commit without prepare. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6775 Test Plan: added a new unit test TransactionTest::CommitWithoutPrepare Reviewed By: lth Differential Revision: D21313270 Pulled By: cheng-chang fbshipit-source-id: 3d95b7c9b2d6cdddc09bdd66c561bc4fae8c3251 | 30 April 2020, 23:24:20 UTC |
079e50d | sdong | 30 April 2020, 22:34:43 UTC | Disallow BlockBasedTableBuilder to set status from non-OK (#6776) Summary: There is no systematic mechanism to prevent BlockBasedTableBuilder's status to be set from non-OK to OK. Adding a mechanism to force this will help us prevent failures in the future. The solution is to only make it possible to set the status code if the status code to set is not OK. Since the status code passed to CompressAndVerifyBlock() is changed, a mini refactoring is done too so that the output arguments are changed from reference to pointers, based on Google C++ Style. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6776 Test Plan: Run all existing test. Reviewed By: pdillinger Differential Revision: D21314382 fbshipit-source-id: 27000c10f1e4c121661e026548d6882066409375 | 30 April 2020, 22:37:03 UTC |
6277e28 | sdong | 30 April 2020, 22:19:31 UTC | Flag CompressionOptions::parallel_threads to be experimental (#6781) Summary: The feature of CompressionOptions::parallel_threads is still not yet mature. Mention it to be experimental in the comments for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6781 Reviewed By: pdillinger Differential Revision: D21330678 fbshipit-source-id: d7dd7d099fb002a5c6a5d8da689ce5ee08a9eb13 | 30 April 2020, 22:22:06 UTC |
ab13d43 | anand76 | 30 April 2020, 21:48:51 UTC | Pass a timeout to FileSystem for random reads (#6751) Summary: Calculate ```IOOptions::timeout``` using ```ReadOptions::deadline``` and pass it to ```FileSystem::Read/FileSystem::MultiRead```. This allows us to impose a tighter bound on the time taken by Get/MultiGet on FileSystem/Envs that support IO timeouts. Even on those that don't support, check in ```RandomAccessFileReader::Read``` and ```MultiRead``` and return ```Status::TimedOut()``` if the deadline is exceeded. For now, TableReader creation, which might do file opens and reads, are not covered. It will be implemented in another PR. Tests: Update existing unit tests to verify the correct timeout value is being passed Pull Request resolved: https://github.com/facebook/rocksdb/pull/6751 Reviewed By: riversand963 Differential Revision: D21285631 Pulled By: anand1976 fbshipit-source-id: d89af843e5a91ece866e87aa29438b52a65a8567 | 30 April 2020, 21:50:39 UTC |
eecd8fb | Peter Dillinger | 30 April 2020, 19:08:34 UTC | Fix assertion that can fail on sst corruption (#6780) Summary: An assertion that a char == a CompressionType (unsigned char) originally cast from a char can fail if the original value is negative, due to numeric promotion. The assertion should pass even if the value is invalid CompressionType, because the callee UncompressBlockContentsForCompressionType checks for that and reports status appropriately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6780 Test Plan: Temporarily change kZSTD = 0x88 and see tests fail. Make this change (in addition), and tests pass. Reviewed By: siying Differential Revision: D21328498 Pulled By: pdillinger fbshipit-source-id: 61caf8d815581ce49261ecb7ab0f396e9ac4bb92 | 30 April 2020, 19:11:00 UTC |
fe238e5 | Levi Tamasi | 30 April 2020, 18:23:32 UTC | Keep track of obsolete blob files in VersionSet (#6755) Summary: The patch adds logic to keep track of obsolete blob files. A blob file becomes obsolete when the last `shared_ptr` that points to the corresponding `SharedBlobFileMetaData` object goes away, which, in turn, happens when the last `Version` that contains the blob file is destroyed. No longer needed blob files are added to the obsolete list in `VersionSet` using a custom deleter to avoid unnecessary coupling between `SharedBlobFileMetaData` and `VersionSet`. Obsolete blob files are returned by `VersionSet::GetObsoleteFiles` and stored in `JobContext`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6755 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21233155 Pulled By: ltamasi fbshipit-source-id: 47757e06fdc0127f27ed57f51abd27893d9a7b7a | 30 April 2020, 18:25:51 UTC |
cf34246 | Adam Retter | 30 April 2020, 17:56:33 UTC | Add Slack forum to README (#6773) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6773 Reviewed By: siying Differential Revision: D21310229 Pulled By: pdillinger fbshipit-source-id: c0d52d0c51121d307d7d5c1374abc7bf78b0c4cf | 30 April 2020, 18:00:28 UTC |
e619a20 | Ziyue Yang | 30 April 2020, 17:46:54 UTC | Add an option for parallel compression in for db_stress (#6722) Summary: This commit adds an `compression_parallel_threads` option in db_stress. It also fixes the naming of parallel compression option in db_bench to keep it aligned with others. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6722 Reviewed By: pdillinger Differential Revision: D21091385 fbshipit-source-id: c9ba8c4e5cc327ff9e6094a6dc6a15fcff70f100 | 30 April 2020, 17:49:07 UTC |
8c69402 | Zhichao Cao | 30 April 2020, 15:38:18 UTC | Fix potential size_t overflow in import_column_family (#6762) Summary: The issue is reported in https://github.com/facebook/rocksdb/issues/6753 . size_t is unsigned and if sorted_file.size() is 0, the end condition of i will be extremely large, cause segment fault in sorted_files[i] and sorted_files[i+1]. Added condition to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6762 Test Plan: make asan_check Reviewed By: pdillinger Differential Revision: D21323063 Pulled By: zhichao-cao fbshipit-source-id: 56ce59201949ed319448228553202b8642c2cc3a | 30 April 2020, 15:40:42 UTC |
b938e60 | anand76 | 30 April 2020, 02:25:56 UTC | Fix a couple of bugs in FaultInjectionTestFS (#6777) Summary: Fix the following cases that can cause false alarms in db_stress when read fault injection is enabled - 1. Turn off corruption/truncation when direct IO is enabled. Since the actual IO size is larger than block size due to alignment requirements, the corruption may not result in a detectable error. 2. Handle the case when the randomly generated string to overwrite the original block is identical to the original. Tests: Run db_stress w/ and wo/ direct IO and fault injection turned on Pull Request resolved: https://github.com/facebook/rocksdb/pull/6777 Reviewed By: zhichao-cao Differential Revision: D21316734 Pulled By: anand1976 fbshipit-source-id: bf0e6468043063ca81ff877d4bf71d3f296c77aa | 30 April 2020, 02:28:29 UTC |
28fe8e4 | Peter Dillinger | 29 April 2020, 20:42:43 UTC | Fix bug in format-diff.sh (#6772) Summary: Nasty bug in which more/different changes would be applied than those shown to user Pull Request resolved: https://github.com/facebook/rocksdb/pull/6772 Test Plan: manual Reviewed By: siying Differential Revision: D21304604 Pulled By: pdillinger fbshipit-source-id: 7e20740e513c9c300d1522511290a025b35abedc | 29 April 2020, 20:45:54 UTC |
5272305 | Derrick Pallas | 29 April 2020, 20:06:27 UTC | Fix FilterBench when RTTI=0 (#6732) Summary: The dynamic_cast in the filter benchmark causes release mode to fail due to no-rtti. Replace with static_cast_with_check. Signed-off-by: Derrick Pallas <derrick@pallas.us> Addition by peterd: Remove unnecessary 2nd template arg on all static_cast_with_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6732 Reviewed By: ltamasi Differential Revision: D21304260 Pulled By: pdillinger fbshipit-source-id: 6e8eb437c4ca5a16dbbfa4053d67c4ad55f1608c | 29 April 2020, 20:09:23 UTC |
8086e5e | Peter Dillinger | 29 April 2020, 04:35:24 UTC | Fix LITE build (#6770) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6770 Test Plan: make LITE=1 check Reviewed By: ajkr Differential Revision: D21296261 Pulled By: pdillinger fbshipit-source-id: b6075cc13a6d6db48617b7e0e9ebeea9364dfd9f | 29 April 2020, 04:37:20 UTC |
335ea73 | anand76 | 29 April 2020, 03:03:28 UTC | Fix a valgrind failure due to DBBasicTestMultiGetDeadline (#6756) Summary: Fix a valgrind failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6756 Test Plan: valgrind_test Reviewed By: pdillinger Differential Revision: D21284660 Pulled By: anand1976 fbshipit-source-id: 39bf1bd130b6adb585ddbf2f9aa2f53dbf666f80 | 29 April 2020, 03:06:06 UTC |
618bf63 | mrambacher | 29 April 2020, 01:02:11 UTC | Add Functions to OptionTypeInfo (#6422) Summary: Added functions for parsing, serializing, and comparing elements to OptionTypeInfo. These functions allow all of the special cases that could not be handled directly in the map of OptionTypeInfo to be moved into the map. Using these functions, every type can be handled via the map rather than special cased. By adding these functions, the code for handling options can become more standardized (fewer special cases) and (eventually) handled completely by common classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6422 Test Plan: pass make check Reviewed By: siying Differential Revision: D21269005 Pulled By: zhichao-cao fbshipit-source-id: 9ba71c721a38ebf9ee88259d60bd81b3282b9077 | 29 April 2020, 01:04:26 UTC |
b810e62 | Peter Dillinger | 28 April 2020, 22:23:33 UTC | Clarifying comments in db.h (#6768) Summary: And fix a confusingly worded log message Pull Request resolved: https://github.com/facebook/rocksdb/pull/6768 Reviewed By: anand1976 Differential Revision: D21284527 Pulled By: pdillinger fbshipit-source-id: f03c1422c229a901c3a65e524740452349626164 | 28 April 2020, 22:26:03 UTC |
bae6f58 | Peter Dillinger | 28 April 2020, 21:46:13 UTC | Basic MultiGet support for partitioned filters (#6757) Summary: In MultiGet, access each applicable filter partition only once per batch, rather than for each applicable key. Also, * Fix Bloom stats for MultiGet * Fix/refactor MultiGetContext::Range::KeysLeft, including * Add efficient BitsSetToOne implementation * Assert that MultiGetContext::Range does not go beyond shift range Performance test: Generate db: $ ./db_bench --benchmarks=fillrandom --num=15000000 --cache_index_and_filter_blocks -bloom_bits=10 -partition_index_and_filters=true ... Before (middle performing run of three; note some missing Bloom stats): $ ./db_bench --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 26.403 micros/op 597517 ops/sec; (548427 of 671968 found) rocksdb.block.cache.filter.hit COUNT : 83443275 rocksdb.bloom.filter.useful COUNT : 0 rocksdb.bloom.filter.full.positive COUNT : 0 rocksdb.bloom.filter.full.true.positive COUNT : 7931450 rocksdb.number.multiget.get COUNT : 385984 rocksdb.number.multiget.keys.read COUNT : 12351488 rocksdb.number.multiget.bytes.read COUNT : 793145000 rocksdb.number.multiget.keys.found COUNT : 7931450 After (middle performing run of three): $ ./db_bench_new --use-existing-db --benchmarks=multireadrandom --num=15000000 --cache_index_and_filter_blocks --bloom_bits=10 --threads=16 --cache_size=20000000 -partition_index_and_filters -batch_size=32 -multiread_batched -statistics --duration=20 2>&1 | egrep 'micros/op|block.cache.filter.hit|bloom.filter.(full|use)|number.multiget' multireadrandom : 21.024 micros/op 752963 ops/sec; (705188 of 863968 found) rocksdb.block.cache.filter.hit COUNT : 49856682 rocksdb.bloom.filter.useful COUNT : 45684579 rocksdb.bloom.filter.full.positive COUNT : 10395458 rocksdb.bloom.filter.full.true.positive COUNT : 9908456 rocksdb.number.multiget.get COUNT : 481984 rocksdb.number.multiget.keys.read COUNT : 15423488 rocksdb.number.multiget.bytes.read COUNT : 990845600 rocksdb.number.multiget.keys.found COUNT : 9908456 So that's about 25% higher throughput even for random keys Pull Request resolved: https://github.com/facebook/rocksdb/pull/6757 Test Plan: unit test included Reviewed By: anand1976 Differential Revision: D21243256 Pulled By: pdillinger fbshipit-source-id: 5644a1468d9e8c8575be02f4e04bc5d62dbbb57f | 28 April 2020, 21:49:34 UTC |
a7f0b27 | Peter Dillinger | 28 April 2020, 19:27:23 UTC | HISTORY.md update for bzip upgrade (#6767) Summary: See https://github.com/facebook/rocksdb/issues/6714 and https://github.com/facebook/rocksdb/issues/6703 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6767 Reviewed By: riversand963 Differential Revision: D21283307 Pulled By: pdillinger fbshipit-source-id: 8463bec725669d13846c728ad4b5bde43f9a84f8 | 28 April 2020, 19:29:31 UTC |
4574d75 | Peter Dillinger | 28 April 2020, 15:24:15 UTC | Update HISTORY.md for block cache redundant adds (#6764) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6764 Reviewed By: ltamasi Differential Revision: D21267108 Pulled By: pdillinger fbshipit-source-id: a3dfe2dbe4e8f6309a53eb72903ef58d52308f97 | 28 April 2020, 15:26:43 UTC |
d4398e0 | Yanqin Jin | 28 April 2020, 05:47:44 UTC | Fix timestamp support for MultiGet (#6748) Summary: 1. Avoid nullptr dereference when passing timestamp to KeyContext creation. 2. Construct LookupKey correctly with timestamp when creating MultiGetContext. 3. Compare without timestamp when sorting KeyContexts. Fixes https://github.com/facebook/rocksdb/issues/6745 Test plan (dev server): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6748 Reviewed By: pdillinger Differential Revision: D21258691 Pulled By: riversand963 fbshipit-source-id: 44e65b759c18b9986947783edf03be4f890bb004 | 28 April 2020, 05:49:56 UTC |
4cd859e | Cheng Chang | 27 April 2020, 23:51:42 UTC | Fix build under LITE (#6758) Summary: GetSupportedCompressions needs to be defined under LITE. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6758 Test Plan: build under LITE Reviewed By: zhichao-cao Differential Revision: D21247937 Pulled By: cheng-chang fbshipit-source-id: 880e59d3e107cdd736d16427a68c5641d1318fb4 | 27 April 2020, 23:55:14 UTC |
bea91d5 | Levi Tamasi | 27 April 2020, 23:42:16 UTC | Destroy any ColumnFamilyHandles in BlobDB::Open upon error (#6763) Summary: If an error happens during BlobDBImpl::Open after the base DB has been opened, we need to destroy the `ColumnFamilyHandle`s returned by `DB::Open` to prevent an assertion in `ColumnFamilySet`'s destructor from being hit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6763 Test Plan: Ran `make check` and tested using the BlobDB mode of `db_bench`. Reviewed By: riversand963 Differential Revision: D21262643 Pulled By: ltamasi fbshipit-source-id: 60ebc7ab19be66cf37fbe5f6d8957d58470f3d3b | 27 April 2020, 23:45:13 UTC |
cc8d16e | Albert Hse-Lin Chen | 27 April 2020, 21:39:54 UTC | Fixed minor typo in comment for MergeOperator::FullMergeV2() (#6759) Summary: Fixed minor typo in comment for FullMergeV2(). Last operand up to snapshot should be +4 instead of +3. Signed-off-by: Albert Hse-Lin Chen <hselin@kalista.io> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6759 Reviewed By: cheng-chang Differential Revision: D21260295 Pulled By: zhichao-cao fbshipit-source-id: cc942306f246c8606538feb30bfdf6df9fb6c54e | 27 April 2020, 21:44:47 UTC |