Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
d9ed290 Siying Dong 25 June 2020, 00:06:33 UTC Add CircleCI gadget CircleCI is stably running. Need to add a gadget. Also since Circle builds some Windows and Linux, rename Travis and Appveyor builds to their names. 25 June 2020, 00:06:33 UTC
9cc2519 sdong 24 June 2020, 23:20:55 UTC Test CircleCI with CLANG-10 (#7025) Summary: It's useful to build RocksDB using a more recent clang version in CI. Add a CircleCI build and fix some issues with it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7025 Test Plan: See all tests pass. Reviewed By: pdillinger Differential Revision: D22215700 fbshipit-source-id: 914a729c2cd3f3ac4a627cc0ac58d4691dca2168 24 June 2020, 23:22:49 UTC
50d6969 sdong 24 June 2020, 20:39:26 UTC Fix unity build broken by #7007 (#7024) Summary: https://github.com/facebook/rocksdb/pull/7007 broken the unity build. Fix it by moving the const inside the function Pull Request resolved: https://github.com/facebook/rocksdb/pull/7024 Test Plan: make unity and see it to build. Reviewed By: zhichao-cao Differential Revision: D22212028 fbshipit-source-id: 5daff7383b691808164d4745ab543238502d946b 24 June 2020, 20:40:48 UTC
83a4dd1 Zhichao Cao 24 June 2020, 18:03:09 UTC Fix the memory leak in Env_basic_test (#7017) Summary: Fix the memory leak broken asan and other test introduced by https://github.com/facebook/rocksdb/issues/6830 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7017 Test Plan: pass asan_check Reviewed By: siying Differential Revision: D22190289 Pulled By: zhichao-cao fbshipit-source-id: 03a095f698b4f9d72fd9374191b17c890d7c2b56 24 June 2020, 18:05:24 UTC
8efb5cf Andrew Kryczka 23 June 2020, 23:24:15 UTC add SstFileManager to crash test (#6993) Summary: SstFileManager is already supported in the stress test as of https://github.com/facebook/rocksdb/issues/6454. This PR enables the SstFileManager in some of the crash test runs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6993 Reviewed By: riversand963 Differential Revision: D22084406 Pulled By: ajkr fbshipit-source-id: 78b8642682e7570ff6ec3a1c3ccd9940f4362289 23 June 2020, 23:27:20 UTC
9f21d08 Levi Tamasi 23 June 2020, 20:41:03 UTC Move kNoExpiration to blob_db.h (#7018) Summary: The constant `kNoExpiration` is currently defined in an internal/implementation header (`blob_log_format.h`); the patch moves it to the public header `blob_db.h` so it is accessible to users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7018 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D22191354 Pulled By: ltamasi fbshipit-source-id: 98c8012a83b999a3f1a30e955ce6bb71ba29dc5c 23 June 2020, 20:45:06 UTC
5b2bbac Peter Dillinger 22 June 2020, 20:30:57 UTC Minimize memory internal fragmentation for Bloom filters (#6427) Summary: New experimental option BBTO::optimize_filters_for_memory builds filters that maximize their use of "usable size" from malloc_usable_size, which is also used to compute block cache charges. Rather than always "rounding up," we track state in the BloomFilterPolicy object to mix essentially "rounding down" and "rounding up" so that the average FP rate of all generated filters is the same as without the option. (YMMV as heavily accessed filters might be unluckily lower accuracy.) Thus, the option near-minimizes what the block cache considers as "memory used" for a given target Bloom filter false positive rate and Bloom filter implementation. There are no forward or backward compatibility issues with this change, though it only works on the format_version=5 Bloom filter. With Jemalloc, we see about 10% reduction in memory footprint (and block cache charge) for Bloom filters, but 1-2% increase in storage footprint, due to encoding efficiency losses (FP rate is non-linear with bits/key). Why not weighted random round up/down rather than state tracking? By only requiring malloc_usable_size, we don't actually know what the next larger and next smaller usable sizes for the allocator are. We pick a requested size, accept and use whatever usable size it has, and use the difference to inform our next choice. This allows us to narrow in on the right balance without tracking/predicting usable sizes. Why not weight history of generated filter false positive rates by number of keys? This could lead to excess skew in small filters after generating a large filter. Results from filter_bench with jemalloc (irrelevant details omitted): (normal keys/filter, but high variance) $ ./filter_bench -quick -impl=2 -average_keys_per_filter=30000 -vary_key_count_ratio=0.9 Build avg ns/key: 29.6278 Number of filters: 5516 Total size (MB): 200.046 Reported total allocated memory (MB): 220.597 Reported internal fragmentation: 10.2732% Bits/key stored: 10.0097 Average FP rate %: 0.965228 $ ./filter_bench -quick -impl=2 -average_keys_per_filter=30000 -vary_key_count_ratio=0.9 -optimize_filters_for_memory Build avg ns/key: 30.5104 Number of filters: 5464 Total size (MB): 200.015 Reported total allocated memory (MB): 200.322 Reported internal fragmentation: 0.153709% Bits/key stored: 10.1011 Average FP rate %: 0.966313 (very few keys / filter, optimization not as effective due to ~59 byte internal fragmentation in blocked Bloom filter representation) $ ./filter_bench -quick -impl=2 -average_keys_per_filter=1000 -vary_key_count_ratio=0.9 Build avg ns/key: 29.5649 Number of filters: 162950 Total size (MB): 200.001 Reported total allocated memory (MB): 224.624 Reported internal fragmentation: 12.3117% Bits/key stored: 10.2951 Average FP rate %: 0.821534 $ ./filter_bench -quick -impl=2 -average_keys_per_filter=1000 -vary_key_count_ratio=0.9 -optimize_filters_for_memory Build avg ns/key: 31.8057 Number of filters: 159849 Total size (MB): 200 Reported total allocated memory (MB): 208.846 Reported internal fragmentation: 4.42297% Bits/key stored: 10.4948 Average FP rate %: 0.811006 (high keys/filter) $ ./filter_bench -quick -impl=2 -average_keys_per_filter=1000000 -vary_key_count_ratio=0.9 Build avg ns/key: 29.7017 Number of filters: 164 Total size (MB): 200.352 Reported total allocated memory (MB): 221.5 Reported internal fragmentation: 10.5552% Bits/key stored: 10.0003 Average FP rate %: 0.969358 $ ./filter_bench -quick -impl=2 -average_keys_per_filter=1000000 -vary_key_count_ratio=0.9 -optimize_filters_for_memory Build avg ns/key: 30.7131 Number of filters: 160 Total size (MB): 200.928 Reported total allocated memory (MB): 200.938 Reported internal fragmentation: 0.00448054% Bits/key stored: 10.1852 Average FP rate %: 0.963387 And from db_bench (block cache) with jemalloc: $ ./db_bench -db=/dev/shm/dbbench.no_optimize -benchmarks=fillrandom -format_version=5 -value_size=90 -bloom_bits=10 -num=2000000 -threads=8 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=false $ ./db_bench -db=/dev/shm/dbbench -benchmarks=fillrandom -format_version=5 -value_size=90 -bloom_bits=10 -num=2000000 -threads=8 -optimize_filters_for_memory -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=false $ (for FILE in /dev/shm/dbbench.no_optimize/*.sst; do ./sst_dump --file=$FILE --show_properties | grep 'filter block' ; done) | awk '{ t += $4; } END { print t; }' 17063835 $ (for FILE in /dev/shm/dbbench/*.sst; do ./sst_dump --file=$FILE --show_properties | grep 'filter block' ; done) | awk '{ t += $4; } END { print t; }' 17430747 $ #^ 2.1% additional filter storage $ ./db_bench -db=/dev/shm/dbbench.no_optimize -use_existing_db -benchmarks=readrandom,stats -statistics -bloom_bits=10 -num=2000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=false -duration=10 -cache_index_and_filter_blocks -cache_size=1000000000 rocksdb.block.cache.index.add COUNT : 33 rocksdb.block.cache.index.bytes.insert COUNT : 8440400 rocksdb.block.cache.filter.add COUNT : 33 rocksdb.block.cache.filter.bytes.insert COUNT : 21087528 rocksdb.bloom.filter.useful COUNT : 4963889 rocksdb.bloom.filter.full.positive COUNT : 1214081 rocksdb.bloom.filter.full.true.positive COUNT : 1161999 $ #^ 1.04 % observed FP rate $ ./db_bench -db=/dev/shm/dbbench -use_existing_db -benchmarks=readrandom,stats -statistics -bloom_bits=10 -num=2000000 -compaction_style=2 -fifo_compaction_max_table_files_size_mb=10000 -fifo_compaction_allow_compaction=false -optimize_filters_for_memory -duration=10 -cache_index_and_filter_blocks -cache_size=1000000000 rocksdb.block.cache.index.add COUNT : 33 rocksdb.block.cache.index.bytes.insert COUNT : 8448592 rocksdb.block.cache.filter.add COUNT : 33 rocksdb.block.cache.filter.bytes.insert COUNT : 18220328 rocksdb.bloom.filter.useful COUNT : 5360933 rocksdb.bloom.filter.full.positive COUNT : 1321315 rocksdb.bloom.filter.full.true.positive COUNT : 1262999 $ #^ 1.08 % observed FP rate, 13.6% less memory usage for filters (Due to specific key density, this example tends to generate filters that are "worse than average" for internal fragmentation. "Better than average" cases can show little or no improvement.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6427 Test Plan: unit test added, 'make check' with gcc, clang and valgrind Reviewed By: siying Differential Revision: D22124374 Pulled By: pdillinger fbshipit-source-id: f3e3aa152f9043ddf4fae25799e76341d0d8714e 22 June 2020, 20:32:07 UTC
1092f19 Matthew Von-Maszewski 22 June 2020, 20:25:41 UTC Make EncryptEnv inheritable (#6830) Summary: EncryptEnv class is both declared and defined within env_encryption.cc. This makes it really tough to derive new classes from that base. This branch moves declaration of the class to rocksdb/env_encryption.h. The change facilitates making new encryption modules (such as an upcoming openssl AES CTR pull request) possible / easy. The only coding change was to add the EncryptEnv object to env_basic_test.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6830 Reviewed By: riversand963 Differential Revision: D21706593 Pulled By: ajkr fbshipit-source-id: 64d2da95a1569ceeb9b1549c3bec5404cf4c89f0 22 June 2020, 20:27:16 UTC
d739318 Zhichao Cao 22 June 2020, 17:18:43 UTC Fix double define in IO_tracer (#7007) Summary: Fix the following error "./trace_replay/io_tracer.h:20:20: error: redefinition of ‘const unsigned int rocksdb::{anonymous}::kCharSize’ const unsigned int kCharSize = 1; ^~~~~~~~~ In file included from unity.cc:177: trace_replay/block_cache_tracer.cc:22:20: note: ‘const unsigned int rocksdb::{anonymous}::kCharSize’ previously defined here const unsigned int kCharSize = 1;" Pull Request resolved: https://github.com/facebook/rocksdb/pull/7007 Reviewed By: akankshamahajan15 Differential Revision: D22142618 Pulled By: zhichao-cao fbshipit-source-id: e6dcd51ccc21d1f58df52cdc7a1c88e54cf4f6e8 22 June 2020, 17:20:13 UTC
096beb7 sdong 20 June 2020, 00:10:31 UTC Remove CircleCI clang build's verbose output (#7000) Summary: As CirclrCI build's clang build is stable, verbose flag is less useful. On the other hand, the long outputs might create other problems. A non-reproducible failure "make: write error: stdout" might be related to it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7000 Test Plan: Watch the run Reviewed By: pdillinger Differential Revision: D22118870 fbshipit-source-id: a4157a4282adddcb0c55c0e9e53b2d9ce18bda66 20 June 2020, 00:11:55 UTC
dea4063 sdong 19 June 2020, 23:57:14 UTC Remove an assertion in FlushAfterIntraL0CompactionCheckConsistencyFail (#7003) Summary: FlushAfterIntraL0CompactionCheckConsistencyFail is flakey. It sometimes fails with: db/db_compaction_test.cc:5186: Failure Expected equality of these values: 10 NumTableFilesAtLevel(0) Which is: 3 I don't see a clear reason why the assertion would always be true. The necessarily of the assertion is not clear either. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7003 Test Plan: See the test still builds. Reviewed By: riversand963 Differential Revision: D22129753 fbshipit-source-id: 42f0bb05e32b369e8d726bfd3e35c29cf52fe008 19 June 2020, 23:58:29 UTC
25a0d0c Peter Dillinger 19 June 2020, 23:16:57 UTC Fix block checksum for >=4GB, refactor (#6978) Summary: Although RocksDB falls over in various other ways with KVs around 4GB or more, this change fixes how XXH32 and XXH64 were being called by the block checksum code to support >= 4GB in case that should ever happen, or the code copied for other uses. This change is not a schema compatibility issue because the checksum verification code would checksum the first (block_size + 1) mod 2^32 bytes while the checksum construction code would checksum the first block_size mod 2^32 plus the compression type byte, meaning the XXH32/64 checksums for >=4GB block would not match about 255/256 times. While touching this code, I refactored to consolidate redundant implementations, improving diagnostics and performance tracking in some cases. Also used less confusing language in those diagnostics. Makes https://github.com/facebook/rocksdb/issues/6875 obsolete. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6978 Test Plan: I was able to write a test for this using an SST file writer and VerifyChecksum in a reader. The test fails before the fix, though I'm leaving the test disabled because I don't think it's worth the expense of running regularly. Reviewed By: gg814 Differential Revision: D22143260 Pulled By: pdillinger fbshipit-source-id: 982993d16134e8c50bea2269047f901c1783726e 19 June 2020, 23:18:24 UTC
d76eed4 Andrew Kryczka 19 June 2020, 23:03:56 UTC minor fixes for stress/crash contruns (#7006) Summary: Avoid using `cf_consistency` together with `enable_compaction_filter` as the former heavily uses snapshots while the latter is incompatible with snapshots. Also fix a clang-analyze error for a write to a variable that is never read. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7006 Reviewed By: zhichao-cao Differential Revision: D22141679 Pulled By: ajkr fbshipit-source-id: 1840ae238168818a9ab5973f90fd78c067399447 19 June 2020, 23:05:17 UTC
88b4210 Peter Dillinger 19 June 2020, 22:26:05 UTC Remove racially charged terms "whitelist" and "blacklist" (#7008) Summary: We don't need them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7008 Test Plan: "make check" and ensure "make crash_test" starts Reviewed By: ajkr Differential Revision: D22143838 Pulled By: pdillinger fbshipit-source-id: 72c8e16603abc59f4954e304466bc4dc1f58f94e 19 June 2020, 22:27:32 UTC
a607f3e Zhichao Cao 19 June 2020, 05:05:27 UTC Fix unused variable failure (#7004) Summary: pass make check, run db_stress Pull Request resolved: https://github.com/facebook/rocksdb/pull/7004 Reviewed By: ajkr Differential Revision: D22132617 Pulled By: zhichao-cao fbshipit-source-id: d65397967e213206ec5efcb767bbdda8a575662a 19 June 2020, 05:06:51 UTC
f4583f7 Kefu Chai 19 June 2020, 00:58:24 UTC add WITH_EXAMPLES options to cmake and cleanups. (#6580) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6580 Reviewed By: cheng-chang Differential Revision: D21846336 Pulled By: ajkr fbshipit-source-id: e5bb0152a0876061d4ff158e7144eb9cc5a88cad 19 June 2020, 01:00:04 UTC
552fd76 Akanksha Mahajan 18 June 2020, 17:44:32 UTC Add IOTracer reader, writer classes for reading/writing IO operations in a binary file (#6958) Summary: 1. As part of IOTracing project, Add a class IOTracer, IOTraceReader and IOTracerWriter that writes the file operations information in a binary file. IOTrace Record contains record information and right now it contains access_timestamp, file_operation, file_name, io_status, len, offset and later other options will be added when file system APIs will be call IOTracer. 2. Add few unit test cases that verify that reading and writing to a IO Trace file is working properly and before start trace and after ending trace nothing is added to the binary file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6958 Test Plan: 1. make check -j64 2. New testcases for IOTracer. Reviewed By: anand1976 Differential Revision: D21943375 Pulled By: akankshamahajan15 fbshipit-source-id: 3532204e2a3eab0104bf411ab142e3fdd4fbce54 18 June 2020, 17:46:11 UTC
d6b7b77 sdong 18 June 2020, 17:15:16 UTC Fix a bug that causes iterator to return wrong result in a rare data race (#6973) Summary: The bug fixed in https://github.com/facebook/rocksdb/pull/1816/ is now applicable to iterator too. This was not an issue but https://github.com/facebook/rocksdb/pull/2886 caused the regression. If a put and DB flush happens just between iterator to get latest sequence number and getting super version, empty result for the key or an older value can be returned, which is wrong. Fix it in the same way as the fix in https://github.com/facebook/rocksdb/issues/1816, that is to get the sequence number after referencing the super version. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6973 Test Plan: Will run stress tests for a while to make sure there is no general regression. Reviewed By: ajkr Differential Revision: D22029348 fbshipit-source-id: 94390f93630906796d6e2fec321f44a920953fd1 18 June 2020, 17:16:38 UTC
569b87e Yanqin Jin 18 June 2020, 17:07:42 UTC Fail recovery when MANIFEST record checksum mismatch (#6996) Summary: https://github.com/facebook/rocksdb/issues/5411 refactored `VersionSet::Recover` but introduced a bug, explained as follows. Before, once a checksum mismatch happens, `reporter` will set `s` to be non-ok. Therefore, Recover will stop processing the MANIFEST any further. ``` // Correct // Inside Recover LogReporter reporter; reporter.status = &s; log::Reader reader(..., reporter); while (reader.ReadRecord() && s.ok()) { ... } ``` The bug is that, the local variable `s` in `ReadAndRecover` won't be updated by `reporter` while reading the MANIFEST. It is possible that the reader sees a checksum mismatch in a record, but `ReadRecord` retries internally read and finds the next valid record. The mismatched record will be ignored and no error is reported. ``` // Incorrect // Inside Recover LogReporter reporter; reporter.status = &s; log::Reader reader(..., reporter); s = ReadAndRecover(reader, ...); // Inside ReadAndRecover Status s; // Shadows the s in Recover. while (reader.ReadRecord() && s.ok()) { ... } ``` `LogReporter` can use a separate `log_read_status` to track the errors while reading the MANIFEST. RocksDB can process more MANIFEST entries only if `log_read_status.ok()`. Test plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6996 Reviewed By: ajkr Differential Revision: D22105746 Pulled By: riversand963 fbshipit-source-id: b22f717a423457a41ca152a242abbb64cf91fc38 18 June 2020, 17:09:12 UTC
775dc62 Andrew Kryczka 18 June 2020, 16:51:14 UTC add `CompactionFilter` to stress/crash tests (#6988) Summary: Added a `CompactionFilter` that is aware of the stress test's expected state. It only drops key versions that are already covered according to the expected state. It is incompatible with snapshots (same as all `CompactionFilter`s), so disables all snapshot-related features when used in the crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6988 Test Plan: running a minified blackbox crash test ``` $ TEST_TMPDIR=/dev/shm python tools/db_crashtest.py blackbox --max_key=1000000 -write_buffer_size=1048576 -max_bytes_for_level_base=4194304 -target_file_size_base=1048576 -value_size_mult=33 --interval=10 --duration=3600 ``` Reviewed By: anand1976 Differential Revision: D22072888 Pulled By: ajkr fbshipit-source-id: 727b9d7a90d5eab18be0ec6cd5a810712ac13320 18 June 2020, 16:54:55 UTC
312f23c Andrew Kryczka 18 June 2020, 16:50:05 UTC build fixes for GNU/kFreeBSD (#6992) Summary: Upstream https://salsa.debian.org/mariadb-team/mariadb-10.4/-/blob/master/debian/patches/rocksdb-kfreebsd.patch by jrtc27. Fixes https://github.com/facebook/rocksdb/issues/5223. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6992 Reviewed By: zhichao-cao Differential Revision: D22084150 Pulled By: ajkr fbshipit-source-id: 1822311ba16f112a15065b2180ce89d36af9cafc 18 June 2020, 16:51:28 UTC
223b57e sdong 17 June 2020, 21:19:35 UTC Fix the bug that compressed cache is disabled in read-only DBs (#6990) Summary: Compressed block cache is disabled in https://github.com/facebook/rocksdb/pull/4650 for no good reason. Re-enable it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6990 Test Plan: Add a unit test to make sure a general function works with read-only DB + compressed block cache. Reviewed By: ltamasi Differential Revision: D22072755 fbshipit-source-id: 2a55df6363de23a78979cf6c747526359e5dc7a1 17 June 2020, 21:30:54 UTC
94d0452 Zitan Chen 17 June 2020, 17:55:42 UTC Store DB identity and DB session ID in SST files (#6983) Summary: `db_id` and `db_session_id` are now part of the table properties for all formats and stored in SST files. This adds about 99 bytes to each new SST file. The `TablePropertiesNames` for these two identifiers are `rocksdb.creating.db.identity` and `rocksdb.creating.session.identity`. In addition, SST files generated from SstFileWriter and Repairer have DB identity “SST Writer” and “DB Repairer”, respectively. Their DB session IDs are generated in the same way as `DB::GetDbSessionId`. A table property test is added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6983 Test Plan: make check and some manual tests. Reviewed By: zhichao-cao Differential Revision: D22048826 Pulled By: gg814 fbshipit-source-id: afdf8c11424a6f509b5c0b06dafad584a80103c9 17 June 2020, 17:57:40 UTC
742b452 Andrew Kryczka 17 June 2020, 04:44:19 UTC update minor version for 6.11 release (#6994) Summary: The 6.11.fb branch is already cut so I will also backport this PR to that branch. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6994 Reviewed By: riversand963 Differential Revision: D22084532 Pulled By: ajkr fbshipit-source-id: 0b025f738cc31c65c673cbf89302359e88a34d19 17 June 2020, 04:46:05 UTC
b7bab48 Yanqin Jin 16 June 2020, 19:56:43 UTC Fix a bug of overwriting return code (#6989) Summary: In best-efforts recovery, an error that is not Corruption or IOError::kNotFound or IOError::kPathNotFound will be overwritten silently. Fix this by checking all non-ok cases and return early. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6989 Test Plan: make check Reviewed By: ajkr Differential Revision: D22071418 Pulled By: riversand963 fbshipit-source-id: 5a4ea5dfb1a41f41c7a3fdaf62b163007b42f04b 16 June 2020, 19:59:35 UTC
9bfd46d Yanqin Jin 15 June 2020, 21:09:53 UTC Let best-efforts recovery ignore CURRENT file (#6970) Summary: Best-efforts recovery does not check the content of CURRENT file to determine which MANIFEST to recover from. However, it still checks the presence of CURRENT file to determine whether to create a new DB during `open()`. Therefore, we can tweak the logic in `open()` a little bit so that best-efforts recovery does not rely on CURRENT file at all. Test plan (dev server): make check ./db_basic_test --gtest_filter=DBBasicTest.RecoverWithNoCurrentFile Pull Request resolved: https://github.com/facebook/rocksdb/pull/6970 Reviewed By: anand1976 Differential Revision: D22013990 Pulled By: riversand963 fbshipit-source-id: db552a1868c60ed70e1f7cd252a3a076eb8ea58f 15 June 2020, 21:11:24 UTC
aa8f133 Levi Tamasi 15 June 2020, 21:06:47 UTC Fix uninitialized memory read in table_test (#6980) Summary: When using parameterized tests, `gtest` sometimes prints the test parameters. If no other printing method is available, it essentially produces a hex dump of the object. This can cause issues with valgrind with types like `TestArgs` in `table_test`, where the object layout has gaps (with uninitialized contents) due to the members' alignment requirements. The patch fixes the uninitialized reads by providing an `operator<<` for `TestArgs` and also makes sure all members are initialized (in a consistent order) on all code paths. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6980 Test Plan: `valgrind --leak-check=full ./table_test` Reviewed By: siying Differential Revision: D22045536 Pulled By: ltamasi fbshipit-source-id: 6f5920ac28c712d0aa88162fffb80172ed769c32 15 June 2020, 21:08:12 UTC
88db97b Zitan Chen 15 June 2020, 17:45:03 UTC Add a DB Session ID (#6959) Summary: Added DB::GetDbSessionId by using the same format and machinery as DB::GetDbIdentity. The DB Session ID is generated (and therefore, updated) each time a DB object is opened. It is written to the LOG file right after the line of “DB SUMMARY”. A test for the uniqueness, for different openings and during the same opening, is also added. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6959 Test Plan: Passed make check Reviewed By: zhichao-cao Differential Revision: D21951721 Pulled By: gg814 fbshipit-source-id: 958a48a612db49a39998ea703cded45987d3fa8b 15 June 2020, 17:47:02 UTC
9c24a5c Zhen Li 13 June 2020, 20:26:03 UTC Fix persistent cache on windows (#6932) Summary: Persistent cache feature caused rocks db crash on windows. I posted a issue for it, https://github.com/facebook/rocksdb/issues/6919. I found this is because no "persistent_cache_key_prefix" is generated for persistent cache. Looking repo history, "GetUniqueIdFromFile" is not implemented on Windows. So my fix is adding "NewId()" function in "persistent_cache" and using it to generate prefix for persistent cache. In this PR, i also re-enable related test cases defined in "db_test2" and "persistent_cache_test" for windows. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6932 Test Plan: 1. run related test cases in "db_test2" and "persistent_cache_test" on windows and see it passed. 2. manually run db_bench.exe with "read_cache_path" and verified. Reviewed By: riversand963 Differential Revision: D21911608 Pulled By: cheng-chang fbshipit-source-id: cdfd938d54a385edbb2836b13aaa1d39b0a6f1c2 13 June 2020, 20:28:31 UTC
f7613e2 Cheng Chang 13 June 2020, 20:23:40 UTC Make it able to lower cpu priority to specific level in threadpool (#6969) Summary: `Env::LowerThreadPoolCPUPriority` takes a new parameter `CpuPriority` to be able to lower to a specific priority such as `CpuPriority::kIdle`, previously, the priority is always lowered to `CpuPriority::kLow`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6969 Test Plan: unit test `EnvPosixTest::LowerThreadPoolCpuPriority` added to `env_test.cc`. Reviewed By: siying Differential Revision: D22011169 Pulled By: cheng-chang fbshipit-source-id: 568878c24a924912e35cef00c552d4a63431cdf4 13 June 2020, 20:25:20 UTC
15d9f28 Yanqin Jin 13 June 2020, 02:24:11 UTC Add stress test for best-efforts recovery (#6819) Summary: Add crash test for the case of best-efforts recovery. After a certain amount of time, we kill the db_stress process, randomly delete some certain table files and restart db_stress. Given the randomness of file deletion, it is difficult to verify against a reference for data correctness. Therefore, we just check that the db can restart successfully. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6819 Test Plan: ``` ./db_stress -best_efforts_recovery=true -disable_wal=1 -reopen=0 ./db_stress -best_efforts_recovery=true -disable_wal=0 -skip_verifydb=1 -verify_db_one_in=0 -continuous_verification_interval=0 make crash_test_with_best_efforts_recovery ``` Reviewed By: anand1976 Differential Revision: D21436753 Pulled By: riversand963 fbshipit-source-id: 0b3605c922a16c37ed17d5ab6682ca4240e47926 13 June 2020, 02:27:46 UTC
bacd6ed Levi Tamasi 13 June 2020, 00:29:22 UTC Turn HarnessTest in table_test into a parameterized test (#6974) Summary: `HarnessTest` in `table_test.cc` currently tests many parameter combinations sequentially in a loop. This is problematic from a testing perspective, since if the test fails, we have no way of knowing how many/which combinations have failed. It can also cause timeouts on our test system due to the sheer number of combinations tested. (Specifically, the parallel compression threads parameter added by https://github.com/facebook/rocksdb/pull/6262 seems to have been the last straw.) There is some DIY code there that splits the load among eight test cases but that does not appear to be sufficient anymore. Instead, the patch turns `HarnessTest` into a parameterized test, so all the parameter combinations can be tested separately and potentially concurrently. It also cleans up the tests a little, fixes `RandomizedLongDB`, which did not get updated when the parallel compression threads parameter was added, and turns `FooterTests` into a standalone test case (since it does not actually need a fixture class). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6974 Test Plan: `make check` Reviewed By: siying Differential Revision: D22029572 Pulled By: ltamasi fbshipit-source-id: 51baea670771c33928f2eb3902bd69dcf540aa41 13 June 2020, 00:31:06 UTC
7e2ac0c sdong 13 June 2020, 00:04:35 UTC Reduce test coverage in older VS versions (#6966) Summary: With Appveyor we run the same set of tests for older versions of VS as the latest version. It creates extra hanging which we don't plan to investigate. Instead, minimize tests run there. The full tests on Windows are already covered in CircleCI. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6966 Test Plan: Watch appveyor runs. Reviewed By: pdillinger Differential Revision: D22025383 fbshipit-source-id: 079dff9e8213bc750a47f4add90fdbf18de9d737 13 June 2020, 00:05:47 UTC
d63f86e Zhen Li 12 June 2020, 23:19:56 UTC fix build with 'USE_HDFS' on windows (#6950) Summary: Build with "USE_HDFS" failed with below errors on Windows. This PR is trying to fix them Severity Code Description Project File Line Suppression State Error (active) E0020 identifier "ssize_t" is undefined rocksdb D:\Git\rocksdb\rocksdb\env\env_hdfs.cc 127 Error (active) E1696 cannot open source file "sys/time.h" rocksdb D:\Git\rocksdb\rocksdb\env\env_hdfs.cc 15 Error C2065 'pthread_t': undeclared identifier rocksdb d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h 166 Error C3861 'pthread_self': identifier not found rocksdb d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h 167 Error C1083 Cannot open include file: 'sys/time.h': No such file or directory rocksdb d:\git\rocksdb\rocksdb\env\env_hdfs.cc 15 Error C2065 'pthread_t': undeclared identifier db_bench d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h 166 Error C3861 'pthread_self': identifier not found db_bench d:\git\rocksdb\rocksdb\hdfs\env_hdfs.h 167 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6950 Test Plan: 1. manually test build with "USE_HDFS" on Windows, verified HDFS Env related function by db_bench.exe. D:\Git\rocksdb\build\Debug>db_bench.exe --hdfs="abfs://test@rdbtest2.dfs.core.windows.net" --num=100 --benchmarks="fillseq,readseq,fillseekseq" --db="abfs://test@rdbtest2.dfs.core.windows.net/test" 2020-06-05 20:42:21,102 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2020-06-05 20:42:22,646 WARN utils.SSLSocketFactoryEx: Failed to load OpenSSL. Falling back to the JSSE default. Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 6.10 Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 100 Prefix: 0 bytes Keys per prefix: 0 RawSize: 0.0 MB (estimated) FileSize: 0.0 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: Snappy Compression sampling rate: 0 Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test] fillseq : 1138.350 micros/op 877 ops/sec; 0.1 MB/s DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test] readseq : 63.580 micros/op 15627 ops/sec; 1.7 MB/s DB path: [abfs://test@rdbtest2.dfs.core.windows.net/test] fillseekseq : 45.615 micros/op 21762 ops/sec; Reviewed By: cheng-chang Differential Revision: D21964806 Pulled By: riversand963 fbshipit-source-id: 9d7413178ece0113d11bc4398583f7d0590d5dbd 12 June 2020, 23:21:50 UTC
9810f40 sdong 12 June 2020, 22:48:32 UTC Circle CI's clang build to really use clang (#6965) Summary: The CircleCI's Clang flavor has a bug that doesn't really use CLANG. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6965 Test Plan: See CI results. Reviewed By: pdillinger Differential Revision: D22025355 fbshipit-source-id: e86922b9152e9f5732e5099d0ce41da9226ff806 12 June 2020, 22:50:34 UTC
af58d92 Andrew Kryczka 12 June 2020, 18:12:55 UTC update HISTORY.md for 6.11 release (#6972) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6972 Reviewed By: zhichao-cao Differential Revision: D22021953 Pulled By: ajkr fbshipit-source-id: 4debbafe45b5939fd28549230eebf6006eb43440 12 June 2020, 18:15:06 UTC
8383363 Levi Tamasi 12 June 2020, 16:52:14 UTC Maintain the set of linked SSTs in BlobFileMetaData (#6945) Summary: The `FileMetaData` objects associated with table files already contain the number of the oldest blob file referenced by the SST in question. This patch adds the inverse mapping to `BlobFileMetaData`, namely the set of table file numbers for which the oldest blob file link points to the given blob file (these are referred to as *linked SSTs*). This mapping will be used by the GC logic. Implementation-wise, the patch builds on the `BlobFileMetaDataDelta` functionality introduced in https://github.com/facebook/rocksdb/pull/6835: newly linked/unlinked SSTs are accumulated in `BlobFileMetaDataDelta`, and the changes to the linked SST set are applied in one shot when the new `Version` is saved. The patch also reworks the blob file related consistency checks in `VersionBuilder` so they validate the consistency of the forward table file -> blob file links and the backward blob file -> table file links for blob files that are part of the `Version`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6945 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21912228 Pulled By: ltamasi fbshipit-source-id: c5bc7acf6e729a8fccbb12672dd5cd00f6f000f8 12 June 2020, 16:54:39 UTC
717749f Yanqin Jin 12 June 2020, 01:39:21 UTC Fail point-in-time WAL recovery upon IOError reading WAL (#6963) Summary: If `options.wal_recovery_mode == WALRecoveryMode::kPointInTimeRecovery`, RocksDB stops replaying WAL once hitting an error and discards the rest of the WAL. This can lead to data loss if the error occurs at an offset smaller than the last sync'ed offset. Ideally, RocksDB point-in-time recovery should permit recovery if the error occurs after last synced offset while fail recovery if error occurs before the last synced offset. However, RocksDB does not track the synced offset of WALs. Consequently, RocksDB does not know whether an error occurs before or after the last synced offset. An error can be one of the following. - WAL record checksum mismatch. This can result from both corruption of synced data and dropping of unsynced data during shutdown. We cannot be sure which one. In order not to defeat the original motivation to permit the latter case, we keep the original behavior of point-in-time WAL recovery. - IOError. This means the WAL can be bad, an indicator of whole file becoming unavailable, not to mention synced part of the WAL. Therefore, we choose to modify the behavior of point-in-time recovery and fail the database recovery. Test plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6963 Reviewed By: ajkr Differential Revision: D22011083 Pulled By: riversand963 fbshipit-source-id: f9cbf29a37dc5cc40d3fa62f89eed1ad67ca1536 12 June 2020, 01:42:10 UTC
d854aba Levi Tamasi 12 June 2020, 01:29:51 UTC Revisit the handling of the case when a file is re-added to the same level (#6939) Summary: https://github.com/facebook/rocksdb/pull/6901 subtly changed the handling of the corner case when a table file is deleted from a level, then re-added to the same level. (Note: this should be extremely rare; one scenario that comes to mind is a trivial move followed by a call to `ReFitLevel` that moves the file back to the original level.) Before that change, a new `FileMetaData` object was created as a result of this sequence; after the change, the original `FileMetaData` was essentially resurrected (since the deletion and the addition simply cancel each other out with the change). This patch restores the original behavior, which is more intuitive considering the interface, and in sync with how trivial moves are handled. (Also note that `FileMetaData` contains some mutable data members, the values of which might be different in the resurrected object and the freshly created one.) The PR also fixes a bug in this area: with the original pre-6901 code, `VersionBuilder` would add the same file twice to the same level in the scenario described above. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6939 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D21905580 Pulled By: ltamasi fbshipit-source-id: da07ae45384ecf3c6c53506d106432d88a7ec9df 12 June 2020, 01:32:18 UTC
722ebba Levi Tamasi 11 June 2020, 23:32:51 UTC Turn DBTest2.CompressionFailures into a parameterized test (#6968) Summary: `DBTest2.CompressionFailures` currently tests many configurations sequentially using nested loops, which often leads to timeouts in our test system. The patch turns it into a parameterized test instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6968 Test Plan: `make check` Reviewed By: siying Differential Revision: D22006954 Pulled By: ltamasi fbshipit-source-id: f71f2f7108086b7651ecfce3d79a7fab24620b2c 11 June 2020, 23:35:39 UTC
b3585a1 Zhichao Cao 11 June 2020, 21:25:01 UTC Ingest SST files with checksum information (#6891) Summary: Application can ingest SST files with file checksum information, such that during ingestion, DB is able to check data integrity and identify of the SST file. The PR introduces generate_and_verify_file_checksum to IngestExternalFileOption to control if the ingested checksum information should be verified with the generated checksum. 1. If generate_and_verify_file_checksum options is *FALSE*: *1)* if DB does not enable SST file checksum, the checksum information ingested will be ignored; *2)* if DB enables the SST file checksum and the checksum function name matches the checksum function name in DB, we trust the ingested checksum, store it in Manifest. If the checksum function name does not match, we treat that as an error and fail the IngestExternalFile() call. 2. If generate_and_verify_file_checksum options is *TRUE*: *1)* if DB does not enable SST file checksum, the checksum information ingested will be ignored; *2)* if DB enable the SST file checksum, we will use the checksum generator from DB to calculate the checksum for each ingested SST files after they are copied or moved. Then, compare the checksum results with the ingested checksum information: _A)_ if the checksum function name does not match, _verification always report true_ and we store the DB generated checksum information in Manifest. _B)_ if the checksum function name mach, and checksum match, ingestion continues and stores the checksum information in the Manifest. Otherwise, terminate file ingestion and report file corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6891 Test Plan: added unit test, pass make asan_check Reviewed By: pdillinger Differential Revision: D21935988 Pulled By: zhichao-cao fbshipit-source-id: 7b55f486632db467e76d72602218d0658aa7f6ed 11 June 2020, 21:27:36 UTC
fbe2d25 Levi Tamasi 10 June 2020, 21:02:07 UTC Use a per-thread path for the export directory in import_column_family_test (#6962) Summary: This is required so that the test cases can safely be run in parallel. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6962 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D21980060 Pulled By: ltamasi fbshipit-source-id: 616b7a0b686155d3874848b9098c67ad3f47efcc 10 June 2020, 21:04:07 UTC
e6be168 Andrew Kryczka 10 June 2020, 20:56:25 UTC save a key comparison in block seeks (#6646) Summary: This saves up to two key comparisons in block seeks. The first key comparison saved is a redundant key comparison against the restart key where the linear scan starts. This comparison is saved in all cases except when the found key is in the first restart interval. The second key comparison saved is a redundant key comparison against the restart key where the linear scan ends. This is only saved in cases where all keys in the restart interval are less than the target (probability roughly `1/restart_interval`). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6646 Test Plan: ran a benchmark with mostly default settings and counted key comparisons before: `user_key_comparison_count = 19399529` after: `user_key_comparison_count = 18431498` setup command: ``` $ TEST_TMPDIR=/dev/shm/dbbench ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -level_compaction_dynamic_level_bytes=true -num=10000000 ``` benchmark command: ``` $ TEST_TMPDIR=/dev/shm/dbbench/ ./db_bench -use_existing_db=true -benchmarks=readrandom -disable_auto_compactions=true -num=10000000 -compression_type=none -reads=1000000 -perf_level=3 ``` Reviewed By: pdillinger Differential Revision: D20849707 Pulled By: ajkr fbshipit-source-id: 1f01c5cd99ea771fd27974046e37b194f1cdcfac 10 June 2020, 20:58:39 UTC
02db03a Andrew Kryczka 09 June 2020, 23:49:07 UTC make L0 index/filter pinned memory usage predictable (#6911) Summary: Memory pinned by `pin_l0_filter_and_index_blocks_in_cache` needs to be predictable based on user config. This PR makes sure we do not pin extra memory for large files generated by intra-L0 (see https://github.com/facebook/rocksdb/issues/6889). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6911 Test Plan: unit test Reviewed By: siying Differential Revision: D21835818 Pulled By: ajkr fbshipit-source-id: a11a088549d06bed8aacc2548d266e5983f0ead4 09 June 2020, 23:51:23 UTC
5abda3b Levi Tamasi 09 June 2020, 22:12:59 UTC Move blob_log_{format,reader,writer}.{cc,h} to db/blob/ (#6960) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6960 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D21958416 Pulled By: ltamasi fbshipit-source-id: 97cf05027b7363014b07836e7f158c23827bd661 09 June 2020, 22:16:05 UTC
edf74d1 Peter Dillinger 09 June 2020, 17:01:12 UTC Add --version and --help to ldb and sst_dump (#6951) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/6951 Test Plan: tests included + manual Reviewed By: ajkr Differential Revision: D21918540 Pulled By: pdillinger fbshipit-source-id: 79d4991f2a831214fc7e477a839ec19dbbace6c5 09 June 2020, 17:04:01 UTC
6a8ddd3 sdong 09 June 2020, 02:32:18 UTC Introduce some Linux build to CircleCI (#6937) Summary: Moving towards the long term goal of moving most CI build to CircleCI when possible, add some Linux tests in CircleCI. This is not all what we can include to CircleCI. For example, Java builds are not includ ed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6937 Test Plan: Watch CI build results. Reviewed By: pdillinger Differential Revision: D21941605 fbshipit-source-id: db6aead3c45f523386d4fb30d224cfde573cccad 09 June 2020, 02:34:31 UTC
1fb3593 anand76 08 June 2020, 23:08:31 UTC Fix a bug in looking up duplicate keys with MultiGet (#6953) Summary: When MultiGet is called with duplicate keys, and the key matches the largest key in an SST file and the value type is merge, only the first instance of the duplicate key is returned with correct results. This is due to the incorrect assumption that if a key in a batch is equal to the largest key in the file, the next key cannot be present in that file. Tests: Add a new unit test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6953 Reviewed By: cheng-chang Differential Revision: D21935898 Pulled By: anand1976 fbshipit-source-id: a2cc327a15150e23fd997546ca64d1c33021cb4c 08 June 2020, 23:11:21 UTC
f5e6494 Levi Tamasi 08 June 2020, 22:59:25 UTC Add convenience method GetFileMetaDataByNumber (#6940) Summary: The patch adds a convenience method `GetFileMetaDataByNumber` that builds on the `FileLocation` functionality introduced recently (see https://github.com/facebook/rocksdb/pull/6862). This method makes it possible to retrieve the `FileMetaData` directly as opposed to having to go through `LevelFiles` and friends. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6940 Test Plan: `make check` Reviewed By: cheng-chang Differential Revision: D21905946 Pulled By: ltamasi fbshipit-source-id: af99e19de21242b2b4a87594a535c6028d16ee72 08 June 2020, 23:01:59 UTC
119b26f Zitan Chen 08 June 2020, 20:56:22 UTC Implement a new subcommand "identify" for sst_dump (#6943) Summary: Implemented a subcommand of sst_dump called identify, which determines whether a file is an SST file or identifies and lists all the SST files in a directory; This update also fixes the problem that sst_dump exits with a success state even if target file/directory does not exist/is not an SST file/is empty/is corrupted. One test is added to sst_dump_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6943 Test Plan: Passed make check and a few manual tests Reviewed By: pdillinger Differential Revision: D21928985 Pulled By: gg814 fbshipit-source-id: 9a8b48e0cf1a0e96b13f42b690aba8ad981afad3 08 June 2020, 20:58:28 UTC
fb08330 Zhichao Cao 08 June 2020, 17:34:47 UTC decouple the dependency of trace_analyzer_test unit test (#6941) Summary: Since gflags use the global variable to store the flags passed in. In the unit test, if we git one flag per unit test, the result is that all the flags are combined together in the following tests. Therefore, it has the dependency. In this PR, we pass the full arguments each time to ensure that the old arguments will be overwritten by the new one such that the dependency is removed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6941 Test Plan: make asan_check. run each unit test in trace_analyzer_test independently and in arbitrary orders. Reviewed By: pdillinger Differential Revision: D21909176 Pulled By: zhichao-cao fbshipit-source-id: dca550a0a4a205c30faa620e258a020a3b5b4e13 08 June 2020, 17:36:43 UTC
3020df9 Yanqin Jin 08 June 2020, 04:54:54 UTC Remove unnecessary inclusion of version_edit.h in env (#6952) Summary: In db_options.c, we should avoid including header files in the `db` directory to avoid introducing unnecessary dependency. The reason why `version_edit.h` has been included in `db_options.cc` is because we need two constants, `kUnknownChecksum` and `kUnknownChecksumFuncName`. We can put these two constants as `constexpr` in the public header `file_checksum.h`. Test plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6952 Reviewed By: zhichao-cao Differential Revision: D21925341 Pulled By: riversand963 fbshipit-source-id: 2902f3b74c97f0cf16c58ad24c095c787c3a40e2 08 June 2020, 04:56:55 UTC
f8c2e5a Levi Tamasi 06 June 2020, 03:08:51 UTC Do not print messages to stderr in VersionBuilder (#6948) Summary: RocksDB is an embedded library; we should not write to the application's console. Note: in each case, the same information is returned in the form of a `Status::Corruption` object. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6948 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D21914965 Pulled By: ltamasi fbshipit-source-id: ae4b66789aa6b659eb8cc2ed4a048187962c86cc 06 June 2020, 03:10:30 UTC
8988f83 Levi Tamasi 06 June 2020, 01:14:10 UTC Fix up a VersionBuilder test case (#6942) Summary: We currently do not have any validation that would ensure that the `FileMetaData` objects are equivalent when a file gets deleted from the LSM tree and then re-added (think trivial moves); however, if we did, this test case would be in violation. The patch changes the values used in the test case so they are consistent. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6942 Test Plan: `make check` Reviewed By: zhichao-cao Differential Revision: D21911366 Pulled By: ltamasi fbshipit-source-id: 2f0486f8337373a6a111b6f28433d70507857104 06 June 2020, 01:16:38 UTC
f941ade Zhichao Cao 06 June 2020, 00:43:31 UTC Clean up the dead code (#6946) Summary: Remove the dead code in table test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6946 Test Plan: run table_test Reviewed By: riversand963 Differential Revision: D21913563 Pulled By: zhichao-cao fbshipit-source-id: c0aa9f3b95dfe87dd7fb2cd4823784f08cb3ddd3 06 June 2020, 00:45:22 UTC
23e446a Zitan Chen 06 June 2020, 00:24:43 UTC Disable OpenForReadOnly tests in the LITE mode (#6947) Summary: Disable two OpenForReadOnly tests in the LITE mode Pull Request resolved: https://github.com/facebook/rocksdb/pull/6947 Test Plan: passed db_test2 Reviewed By: cheng-chang Differential Revision: D21914345 Pulled By: gg814 fbshipit-source-id: 58e81baf5d8cf8adcedaef3966aa3a427bbdf7c2 06 June 2020, 00:28:24 UTC
2e7070b sdong 05 June 2020, 19:14:42 UTC Directly use unit test tempalte buck (#6926) Summary: Make RocksDB run a predefined unit test so that it can be integrated with better tools. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6926 Test Plan: Watch tests Reviewed By: pdillinger Differential Revision: D21866216 fbshipit-source-id: cafca82efdf0b72671be8d30b665e88a75ae6000 05 June 2020, 19:16:33 UTC
98b0cbe anand76 05 June 2020, 18:06:26 UTC Check iterator status BlockBasedTableReader::VerifyChecksumInBlocks() (#6909) Summary: The ```for``` loop in ```VerifyChecksumInBlocks``` only checks ```index_iter->Valid()``` which could be ```false``` either due to reaching the end of the index or, in case of partitioned index, it could be due to a checksum mismatch error when reading a 2nd level index block. Instead of throwing away the index iterator status, we need to return any errors back to the caller. Tests: Add a test in block_based_table_reader_test.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6909 Reviewed By: pdillinger Differential Revision: D21833922 Pulled By: anand1976 fbshipit-source-id: bc778ebf1121dbbdd768689de5183f07a9f0beae 05 June 2020, 18:08:25 UTC
1bee0fc Cheng Chang 05 June 2020, 17:53:08 UTC Make DestroyDir destroy directories recursively (#6934) Summary: Currently, `DeleteDir` only deletes the directory if there are no other directories under the target dir. This PR makes it delete directories recursively. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6934 Test Plan: Added a new unit test in testutil_test.cc. `make testutil_test` Reviewed By: zhichao-cao Differential Revision: D21884211 Pulled By: cheng-chang fbshipit-source-id: 0b9a48a200f494ee007aef5d1763b4aa331f8b5a 05 June 2020, 17:55:22 UTC
2677bd5 Akanksha Mahajan 05 June 2020, 16:41:03 UTC Add logs and stats in DeleteScheduler (#6927) Summary: Add logs and stats for files marked as trash and files deleted immediately in DeleteScheduler Pull Request resolved: https://github.com/facebook/rocksdb/pull/6927 Test Plan: make check -j64 Reviewed By: riversand963 Differential Revision: D21869068 Pulled By: akankshamahajan15 fbshipit-source-id: e9f673c4fa8049ce648b23c75d742f2f9c6c57a1 05 June 2020, 16:43:04 UTC
aaece2a Peter Dillinger 04 June 2020, 22:44:05 UTC Fix some defects reported by Coverity Scan (#6933) Summary: Confusing checks for null that are never null Pull Request resolved: https://github.com/facebook/rocksdb/pull/6933 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D21885466 Pulled By: pdillinger fbshipit-source-id: 4b48e03c2a33727f2702b0d12292f9fda5a3c475 04 June 2020, 22:46:27 UTC
c7432cc Peter Dillinger 04 June 2020, 22:32:29 UTC Fix more defects reported by Coverity Scan (#6935) Summary: Mostly uninitialized values: some probably written before use, but some seem like bugs. Also, destructor needs to be virtual, and possible use-after-free in test Pull Request resolved: https://github.com/facebook/rocksdb/pull/6935 Test Plan: make check Reviewed By: siying Differential Revision: D21885484 Pulled By: pdillinger fbshipit-source-id: e2e7cb0a0cf196f2b55edd16f0634e81f6cc8e08 04 June 2020, 22:35:08 UTC
a8170d7 Yanqin Jin 04 June 2020, 21:19:05 UTC Close file to avoid file-descriptor leakage (#6936) Summary: When operation on an open file descriptor fails, we should close the file descriptor. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6936 Test Plan: make check Reviewed By: pdillinger Differential Revision: D21885458 Pulled By: riversand963 fbshipit-source-id: ba077a76b256a8537f21e22e4ec198f45390bf50 04 June 2020, 21:21:15 UTC
6cbe9d9 sdong 04 June 2020, 21:14:51 UTC Make StringAppendOperatorTest a parameterized test (#6930) Summary: StringAppendOperatorTest right now runs in a mode where RUN_ALL_TESTS() is executed twice for the same test but different settings. This creates a problem with a tool that expects every test to run once. Fix it by using a parameterized test instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6930 Test Plan: Run the test and see it passed. Reviewed By: ltamasi Differential Revision: D21874145 fbshipit-source-id: 55520b2d7f1ba9f3cba1e2d087fe86f43fb06145 04 June 2020, 21:17:11 UTC
31bd2d7 sdong 04 June 2020, 18:41:21 UTC Fix ThreadLocalTest.SequentialReadWriteTest failure when running individually (#6929) Summary: When running ThreadLocalTest.SequentialReadWriteTest individually, the test fails with: ] ./thread_local_test --gtest_filter="*SequentialReadWriteTest*" Note: Google Test filter = *SequentialReadWriteTest* [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from ThreadLocalTest [ RUN ] ThreadLocalTest.SequentialReadWriteTest internal_repo_rocksdb/repo/util/thread_local_test.cc:144: Failure Expected: IDChecker::PeekId() Which is: 3 To be equal to: base_id + 1u Which is: 2 [ FAILED ] ThreadLocalTest.SequentialReadWriteTest (1 ms) [----------] 1 test from ThreadLocalTest (1 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] ThreadLocalTest.SequentialReadWriteTest 1 FAILED TEST It appears that when running as the first test, PeakId() was updated twice. I didn't dig into it why but it doesn't seem to break the contract. Relax the assertion to make it pass. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6929 Test Plan: Run the test individually and as the whole thread_local_test Reviewed By: riversand963 Differential Revision: D21873999 fbshipit-source-id: 1dcb6a2e9c38b6afd848027308bfe633342b7548 04 June 2020, 18:44:09 UTC
e85cbdb mrambacher 04 June 2020, 18:38:34 UTC Fix two core dumps when files are missing (#6922) Summary: The LDB create and drop column family commands failed to check if theere was a valid database prior to dereferencing it, leading to a core dump. The SstFileDumper prefetch code would dereference a file when the file did not exist as part of the Prefetch code. This dereference was moved inside an st.ok() check. Tests were added for both failure conditions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6922 Reviewed By: gg814 Differential Revision: D21884024 Pulled By: pdillinger fbshipit-source-id: bddd45c299aa9dc7e928c17a37a96521f8c9149e 04 June 2020, 18:40:32 UTC
0b45a68 sdong 04 June 2020, 16:49:44 UTC env_test */RunMany/* tests to run individually (#6931) Summary: When run */RunMany/* tests individually, e.g. ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0, they hang. It's because they insert to background thread pool without initializing them. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6931 Test Plan: Run ChrootEnvWithDirectIO/EnvPosixTestWithParam.RunMany/0 by itself and see it passes. Reviewed By: riversand963 Differential Revision: D21875603 fbshipit-source-id: 7f848174c1a660254a2b1f7e11cca5370793ba30 04 June 2020, 16:51:38 UTC
2f32618 Yanqin Jin 04 June 2020, 15:28:44 UTC Fix a typo (bug) when setting error during Flush (#6928) Summary: As title. The prior change to the line is a typo. Fixing it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6928 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D21873587 Pulled By: riversand963 fbshipit-source-id: f4837fc8792d7106bc230b7b499dfbb7a2847430 04 June 2020, 15:30:42 UTC
02df00d Zitan Chen 04 June 2020, 01:55:25 UTC API change: DB::OpenForReadOnly will not write to the file system unless create_if_missing is true (#6900) Summary: DB::OpenForReadOnly will not write anything to the file system (i.e., create directories or files for the DB) unless create_if_missing is true. This change also fixes some subcommands of ldb, which write to the file system even if the purpose is for readonly. Two tests for this updated behavior of DB::OpenForReadOnly are also added. Other minor changes: 1. Updated HISTORY.md to include this API change of DB::OpenForReadOnly; 2. Updated the help information for the put and batchput subcommands of ldb with the option [--create_if_missing]; 3. Updated the comment of Env::DeleteDir to emphasize that it returns OK only if the directory to be deleted is empty. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6900 Test Plan: passed make check; also manually tested a few ldb subcommands Reviewed By: pdillinger Differential Revision: D21822188 Pulled By: gg814 fbshipit-source-id: 604cc0f0d0326a937ee25a32cdc2b512f9a3be6e 04 June 2020, 01:57:49 UTC
055b4d2 sdong 04 June 2020, 01:18:38 UTC Make sure core components not depend on gtest (#6921) Summary: We recently removed the dependencies of core components on gtest. Add a Travis test to make sure it doesn't regress. Change cmake setting so that the gtest related components are only included when tests, benchmarks or stress tools are included in the build. Add this build setting in Travis to confirm it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6921 Test Plan: See Travis passes Reviewed By: ajkr Differential Revision: D21863564 fbshipit-source-id: df26f50a8305a04ff19ffa8069a1857ecee10289 04 June 2020, 01:22:14 UTC
b7c825d Stanislav Tkach 04 June 2020, 00:06:44 UTC Add (some) getters for options to the C API (#6925) Summary: Additionally I have extended the incomplete test added in the https://github.com/facebook/rocksdb/issues/6880. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6925 Reviewed By: ajkr Differential Revision: D21869788 Pulled By: pdillinger fbshipit-source-id: e9db80f259c57ca1bdcbc2c66cb938cb1ac26e48 04 June 2020, 00:08:50 UTC
afa3518 sdong 03 June 2020, 22:53:09 UTC Revert "Update googletest from 1.8.1 to 1.10.0 (#6808)" (#6923) Summary: This reverts commit 8d87e9cea13b09e12d0d22d4fdebdd5bacbb5c6e. Based on offline discussions, it's too early to upgrade to gtest 1.10, as it prevents some developers from using an older version of gtest to integrate to some other systems. Revert it for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6923 Reviewed By: pdillinger Differential Revision: D21864799 fbshipit-source-id: d0726b1ff649fc911b9378f1763316200bd363fc 03 June 2020, 22:55:03 UTC
0f85d16 Hans Holmberg 03 June 2020, 21:55:43 UTC Route GetTestDirectory to FileSystem in CompositeEnvWrappers (#6896) Summary: GetTestDirectory implies a file system operation (it creates the default test directory if missing), so it should be routed to the FileSystem rather than the Env. Also remove the GetTestDirectory implementation in the PosixEnv, since it overrides GetTestDirectory in CompositeEnv making it impossible to override with a custom FileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6896 Reviewed By: cheng-chang Differential Revision: D21868984 Pulled By: ajkr fbshipit-source-id: e79bfef758d06dacef727c54b96abe62e78726fd 03 June 2020, 21:57:46 UTC
f005dac hfrt456 03 June 2020, 20:48:06 UTC fix IsDirectory function in env_hdfs.cc (#6917) Summary: fix IsDirectory function for hdfsEnv Pull Request resolved: https://github.com/facebook/rocksdb/pull/6917 Reviewed By: cheng-chang Differential Revision: D21865020 Pulled By: riversand963 fbshipit-source-id: ad69ed564d027b7bbdf4c693dd57cd02622fb3f8 03 June 2020, 20:50:17 UTC
0b8c549 Levi Tamasi 03 June 2020, 20:37:47 UTC Mention the consistency check improvement in HISTORY.md (#6924) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6924 Reviewed By: cheng-chang Differential Revision: D21865662 Pulled By: ltamasi fbshipit-source-id: 83a01bcbb779cfba941154a36a9e735293a93211 03 June 2020, 20:40:41 UTC
ffe08ff Hao Chen 03 June 2020, 19:23:47 UTC correct level information in version_set.cc (#6920) Summary: fix these two issues https://github.com/facebook/rocksdb/issues/6912 and https://github.com/facebook/rocksdb/issues/6667 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6920 Reviewed By: cheng-chang Differential Revision: D21864885 Pulled By: ajkr fbshipit-source-id: 10e21fc1851b67a59d44358f59c64fa5523bd263 03 June 2020, 19:27:13 UTC
22e5c51 Anatoly Zhmur 03 June 2020, 19:22:29 UTC Add zstd_max_train_bytes to c interop (#6796) Summary: Added setting of zstd_max_train_bytes compression option parameter to c interop. rocksdb_options_set_bottommost_compression_options was using bool parameter and thus not exported, updated it to unsigned char and added to c.h as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6796 Reviewed By: cheng-chang Differential Revision: D21611471 Pulled By: ajkr fbshipit-source-id: caaaf153de934837ad9af283c7f8c025ff0b0cf5 03 June 2020, 19:27:12 UTC
0a17d95 mrambacher 03 June 2020, 19:19:54 UTC Add OptionTypeInfo::Vector to parse/serialize vectors (#6424) Summary: The OptionTypeInfo::Vector method allows a vector<T> to be converted to/from strings via the options. The kVectorInt and kVectorCompressionType vectors were replaced with this methodology. As part of this change, the NextToken method was added to the OptionTypeInfo. This method was refactored from code within the StringToMap function. Future types that could use this functionality include the EventListener vectors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6424 Reviewed By: cheng-chang Differential Revision: D21832368 Pulled By: zhichao-cao fbshipit-source-id: e1ca766faff139d54e6e8407a9ec09ece6517439 03 June 2020, 19:23:07 UTC
172adce Lucian Petrut 03 June 2020, 18:59:37 UTC Posix threads (#6865) Summary: Rocksdb is using the c++11 std::threads feature. The issue is that MINGW only supports it when using Posix threads. This change will allow rocksdb::port::WindowsThread to be replaced with std::thread, which in turn will allow Rocksdb to be cross compiled using MINGW. At the same time, we'll have to use GetCurrentProcessId instead of _getpid. Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6865 Reviewed By: cheng-chang Differential Revision: D21864285 Pulled By: ajkr fbshipit-source-id: 0982eed313e7d34d351b1364c1ccc722da473205 03 June 2020, 19:01:57 UTC
43f8a9d Peter Dillinger 03 June 2020, 18:37:12 UTC Some fixes for gcc 4.8 and add to Travis (#6915) Summary: People keep breaking the gcc 4.8 compilation due to different warnings for shadowing member functions with locals. Adding to Travis to keep compatibility. (gcc 4.8 is default on CentOS 7.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6915 Test Plan: local and Travis Reviewed By: siying Differential Revision: D21842894 Pulled By: pdillinger fbshipit-source-id: bdcd4385127ee5d1cc222d87e53fb3695c32a9d4 03 June 2020, 18:39:25 UTC
78e291b Levi Tamasi 03 June 2020, 18:21:16 UTC Improve consistency checks in VersionBuilder (#6901) Summary: The patch cleans up the code and improves the consistency checks around adding/deleting table files in `VersionBuilder`. Namely, it makes the checks stricter and improves them in the following ways: 1) A table file can now only be deleted from the LSM tree using the level it resides on. Earlier, there was some unnecessary wiggle room for trivially moved files (they could be deleted using a lower level number than the actual one). 2) A table file cannot be added to the tree if it is already present in the tree on any level (not just the target level). The earlier code only had an assertion (which is a no-op in release builds) that the newly added file is not already present on the target level. 3) The above consistency checks around state transitions are now mandatory, as opposed to the earlier `CheckConsistencyForDeletes`, which was a no-op in release mode unless `force_consistency_checks` was set to `true`. The rationale here is that assuming that the initial state is consistent, a valid transition leads to a next state that is also consistent; however, an *invalid* transition offers no such guarantee. Hence it makes sense to validate the transitions unconditionally, and save `force_consistency_checks` for the paranoid checks that re-validate the entire state. 4) The new checks build on the mechanism introduced in https://github.com/facebook/rocksdb/pull/6862, which enables us to efficiently look up the location (level and position within level) of files in a `Version` by file number. This makes the consistency checks much more efficient than the earlier `CheckConsistencyForDeletes`, which essentially performed a linear search. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6901 Test Plan: Extended the unit tests and ran: `make check` `make whitebox_crash_test` Reviewed By: ajkr Differential Revision: D21822714 Pulled By: ltamasi fbshipit-source-id: e2b29c8b6da1bf0f59004acc889e4870b2d18215 03 June 2020, 18:24:55 UTC
9360776 Peter Dillinger 03 June 2020, 17:40:13 UTC Fix handling of too-small filter partition size (#6905) Summary: Because ARM and some other platforms have a larger cache line size, they have a larger minimum filter size, which causes recently added PartitionedMultiGet test in db_bloom_filter_test to fail on those platforms. The code would actually end up using larger partitions, because keys_per_partition_ would be 0 and never == number of keys added. The code now attempts to get as close as possible to the small target size, while fully utilizing that filter size, if the target partition size is smaller than the minimum filter size. Also updated the test to break more uniformly across platforms Pull Request resolved: https://github.com/facebook/rocksdb/pull/6905 Test Plan: updated test, tested on ARM Reviewed By: anand1976 Differential Revision: D21840639 Pulled By: pdillinger fbshipit-source-id: 11684b6d35f43d2e98b85ddb2c8dcfd59d670817 03 June 2020, 17:43:01 UTC
2adb7e3 Zhichao Cao 02 June 2020, 22:02:44 UTC Fix potential overflow of unsigned type in for loop (#6902) Summary: x.size() -1 or y - 1 can overflow to an extremely large value when x.size() pr y is 0 when they are unsigned type. The end condition of i in the for loop will be extremely large, potentially causes segment fault. Fix them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6902 Test Plan: pass make asan_check Reviewed By: ajkr Differential Revision: D21843767 Pulled By: zhichao-cao fbshipit-source-id: 5b8b88155ac5a93d86246d832e89905a783bb5a1 02 June 2020, 22:05:07 UTC
556972e Zhichao Cao 02 June 2020, 22:02:25 UTC Replace Status with IOStatus in CopyFile and CreateFile (#6916) Summary: Replace Status with IOStatus in CopyFile and CreateFile. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6916 Test Plan: pass make asan_check Reviewed By: cheng-chang Differential Revision: D21843775 Pulled By: zhichao-cao fbshipit-source-id: 524d4a0fcf47f0941b923da0346e0de71607f5f6 02 June 2020, 22:05:06 UTC
bfc9737 sdong 02 June 2020, 20:54:03 UTC Remove gtest dependency in non-test code under utilities/cassandra (#6908) Summary: production code under utilities/cassandra depends on gtest.h. Remove them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6908 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21842606 fbshipit-source-id: a098e0b49c9aeac51cc90a79562ad9897a36122c 02 June 2020, 20:56:29 UTC
38f988d Stanislav Tkach 02 June 2020, 20:44:53 UTC Expose rocksdb_options_copy function to the C API (#6880) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6880 Reviewed By: ajkr Differential Revision: D21842752 Pulled By: pdillinger fbshipit-source-id: eda326f551ddd9cb397681544b9e9799ea614e52 02 June 2020, 20:48:51 UTC
14eca6b Peter Dillinger 02 June 2020, 19:27:59 UTC For ApproximateSizes, pro-rate table metadata size over data blocks (#6784) Summary: The implementation of GetApproximateSizes was inconsistent in its treatment of the size of non-data blocks of SST files, sometimes including and sometimes now. This was at its worst with large portion of table file used by filters and querying a small range that crossed a table boundary: the size estimate would include large filter size. It's conceivable that someone might want only to know the size in terms of data blocks, but I believe that's unlikely enough to ignore for now. Similarly, there's no evidence the internal function AppoximateOffsetOf is used for anything other than a one-sided ApproximateSize, so I intend to refactor to remove redundancy in a follow-up commit. So to fix this, GetApproximateSizes (and implementation details ApproximateSize and ApproximateOffsetOf) now consistently include in their returned sizes a portion of table file metadata (incl filters and indexes) based on the size portion of the data blocks in range. In other words, if a key range covers data blocks that are X% by size of all the table's data blocks, returned approximate size is X% of the total file size. It would technically be more accurate to attribute metadata based on number of keys, but that's not computationally efficient with data available and rarely a meaningful difference. Also includes miscellaneous comment improvements / clarifications. Also included is a new approximatesizerandom benchmark for db_bench. No significant performance difference seen with this change, whether ~700 ops/sec with cache_index_and_filter_blocks and small cache or ~150k ops/sec without cache_index_and_filter_blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6784 Test Plan: Test added to DBTest.ApproximateSizesFilesWithErrorMargin. Old code running new test... [ RUN ] DBTest.ApproximateSizesFilesWithErrorMargin db/db_test.cc:1562: Failure Expected: (size) <= (11 * 100), actual: 9478 vs 1100 Other tests updated to reflect consistent accounting of metadata. Reviewed By: siying Differential Revision: D21334706 Pulled By: pdillinger fbshipit-source-id: 6f86870e45213334fedbe9c73b4ebb1d8d611185 02 June 2020, 19:30:23 UTC
298b00a sdong 02 June 2020, 19:09:48 UTC Reduce dependency on gtest dependency in release code (#6907) Summary: Release code now depends on gtest, indirectly through including "test_util/testharness.h". This creates multiple problems. One important reason is the definition of IGNORE_STATUS_IF_ERROR() in test_util/testharness.h. Move it to sync_point.h instead. Note that utilities/cassandra/format.h still depends on "test_util/testharness.h". This will be resolved in a separate diff. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6907 Test Plan: Run all existing tests. Reviewed By: ajkr Differential Revision: D21829884 fbshipit-source-id: 9253c19ffde2936f3ae68998210f8e54f645a6e6 02 June 2020, 19:11:24 UTC
8d87e9c Adam Retter 02 June 2020, 03:31:29 UTC Update googletest from 1.8.1 to 1.10.0 (#6808) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6808 Reviewed By: anand1976 Differential Revision: D21483984 Pulled By: pdillinger fbshipit-source-id: 70c5eff2bd54ddba469761d95e4cd4611fb8e598 02 June 2020, 03:33:42 UTC
66942e8 anand76 02 June 2020, 02:41:58 UTC Avoid unnecessary reads of uncompression dictionary in MultiGet (#6906) Summary: We may sometimes read the uncompression dictionary when its not necessary, when we lookup a key in an SST file but the index indicates the key is not present. This can happen with index_type 3. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6906 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D21828944 Pulled By: anand1976 fbshipit-source-id: 7aef4f0a39548d0874eafefd2687006d2652f9bb 02 June 2020, 02:43:37 UTC
02f59ed sdong 01 June 2020, 23:30:45 UTC Find the correct gcov (#6904) Summary: Right now in FB environment, wrong gcov is used. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6904 Test Plan: "make coverage" and watch results. Reviewed By: riversand963 Differential Revision: D21824291 fbshipit-source-id: 666011fd86c36adafa09ebd9eb97742f94fb90bb 01 June 2020, 23:33:05 UTC
bcb9e41 Cheng Chang 01 June 2020, 22:17:58 UTC Explicitly free allocated buffer when status is not ok (#6903) Summary: Currently we rely on `BlockContents` to implicitly free the allocated scratch buffer, but when IO error happens, it doesn't make sense to construct the `BlockContents` which might be corrupted. In the stress test, we find that `assert(req.result.size() == block_size(handle));` fails because of potential IO errors. In this PR, we explicitly free the scratch buffer on error without constructing `BlockContents`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6903 Test Plan: watch stress test Reviewed By: anand1976 Differential Revision: D21823869 Pulled By: cheng-chang fbshipit-source-id: 5603fc80e9bf3f44a9d7250ddebd871afe1eb89f 01 June 2020, 22:19:40 UTC
038e02d zitan 01 June 2020, 19:13:36 UTC Remove extraneous newline from ldb stderr (#6897) Summary: **Summary** Remove the extraneous newline when using ldb tool. For example, the subcommand list_column_families will print an empty line to stderr even if there are no errors. **Test plan** Passed make check; manually tested a few ldb subcommands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6897 Reviewed By: pdillinger Differential Revision: D21819352 Pulled By: gg814 fbshipit-source-id: 5a16a6431bb96684fe97647f4d3ac5bf0ec7fc90 01 June 2020, 19:15:36 UTC
0c56fc4 Peter Dillinger 29 May 2020, 18:24:19 UTC Allow missing "unversioned" python, as in CentOS 8 (#6883) Summary: RocksDB Makefile was assuming existence of 'python' command, which is not present in CentOS 8. We avoid using 'python' if 'python3' is available. Also added fancy logic to format-diff.sh to make clang-format-diff.py for Python2 work even with Python3 only (as some CentOS 8 FB machines come equipped) Also, now use just 'python3' for PYTHON if not found so that an informative "command not found" error will result rather than something weird. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6883 Test Plan: manually tried some variants, 'make check' on a fresh CentOS 8 machine without 'python' executable or Python2 but with clang-format-diff.py for Python2. Reviewed By: gg814 Differential Revision: D21767029 Pulled By: pdillinger fbshipit-source-id: 54761b376b140a3922407bdc462f3572f461d0e9 29 May 2020, 18:29:23 UTC
c5abf78 Andrew Kryczka 28 May 2020, 17:49:02 UTC avoid `IterKey::UpdateInternalKey()` in `BlockIter` (#6843) Summary: `IterKey::UpdateInternalKey()` is an error-prone API as it's incompatible with `IterKey::TrimAppend()`, which is used for decoding delta-encoded internal keys. This PR stops using it in `BlockIter`. Instead, it assigns global seqno in a separate `IterKey`'s buffer when needed. The logic for safely getting a Slice with global seqno properly assigned is encapsulated in `GlobalSeqnoAppliedKey`. `BinarySeek()` is also migrated to use this API (previously it ignored global seqno entirely). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6843 Test Plan: benchmark setup -- single file DBs, in-memory, no compression. "normal_db" created by regular flush; "ingestion_db" created by ingesting a file. Both DBs have same contents. ``` $ TEST_TMPDIR=/dev/shm/normal_db/ ./db_bench -benchmarks=fillrandom,compact -write_buffer_size=10485760000 -disable_auto_compactions=true -compression_type=none -num=1000000 $ ./ldb write_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ --compression_type=no --hex --create_if_missing < <(./sst_dump --command=scan --output_hex --file=/dev/shm/normal_db/dbbench/000007.sst | awk 'began {print "0x" substr($1, 2, length($1) - 2), "==>", "0x" $5} ; /^Sst file format: block-based/ {began=1}') $ ./ldb ingest_extern_sst ./tmp.sst --db=/dev/shm/ingestion_db/dbbench/ ``` benchmark run command: ``` TEST_TMPDIR=/dev/shm/$DB/ ./db_bench -benchmarks=seekrandom -seek_nexts=10 -use_existing_db=true -cache_index_and_filter_blocks=false -num=1000000 -cache_size=1048576000 -threads=1 -reads=40000000 ``` results: | DB | code | throughput | |---|---|---| | normal_db | master | 267.9 | | normal_db | PR6843 | 254.2 (-5.1%) | | ingestion_db | master | 259.6 | | ingestion_db | PR6843 | 250.5 (-3.5%) | Reviewed By: pdillinger Differential Revision: D21562604 Pulled By: ajkr fbshipit-source-id: 937596f836930515da8084d11755e1f247dcb264 28 May 2020, 17:51:30 UTC
961c759 Yanqin Jin 28 May 2020, 17:37:57 UTC Add timestamp to delete (#6253) Summary: Preliminary user-timestamp support for delete. If ["a", ts=100] exists, you can delete it by calling `DB::Delete(write_options, key)` in which `write_options.timestamp` points to a `ts` higher than 100. Implementation A new ValueType, i.e. `kTypeDeletionWithTimestamp` is added for deletion marker with timestamp. The reason for a separate `kTypeDeletionWithTimestamp`: RocksDB may drop tombstones (keys with kTypeDeletion) when compacting them to the bottom level. This is OK and useful if timestamp is disabled. When timestamp is enabled, should we still reuse `kTypeDeletion`, we may drop the tombstone with a more recent timestamp, causing deleted keys to re-appear. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6253 Reviewed By: ltamasi Differential Revision: D20995328 Pulled By: riversand963 fbshipit-source-id: a9e5c22968ad76f98e3dc6ee0151265a3f0df619 28 May 2020, 17:40:03 UTC
e3f953a Levi Tamasi 28 May 2020, 17:00:19 UTC Make it possible to look up files by number in VersionStorageInfo (#6862) Summary: Does what it says on the can: the patch adds a hash map to `VersionStorageInfo` that maps file numbers to file locations, i.e. (level, position in level) pairs. This will enable stricter consistency checks in `VersionBuilder`. The patch also fixes all the unit tests that used duplicate file numbers in a version (which would trigger an assertion with the new code). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6862 Test Plan: `make check` `make whitebox_crash_test` Reviewed By: riversand963 Differential Revision: D21670446 Pulled By: ltamasi fbshipit-source-id: 2eac249945cf33d8fb8597b26bfff5221e1a861a 28 May 2020, 17:03:06 UTC
bcefc59 Akanksha Mahajan 27 May 2020, 20:03:08 UTC Allow MultiGet users to limit cumulative value size (#6826) Summary: 1. Add a value_size in read options which limits the cumulative value size of keys read in batches. Once the size exceeds read_options.value_size, all the remaining keys are returned with status Abort without further fetching any key. 2. Add a unit test case MultiGetBatchedValueSizeSimple the reads keys from memory and sst files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6826 Test Plan: 1. make check -j64 2. Add a new unit test case Reviewed By: anand1976 Differential Revision: D21471483 Pulled By: akankshamahajan15 fbshipit-source-id: dea51b8e76d5d1df38ece8cdb29933b1d798b900 27 May 2020, 20:07:14 UTC
9060e6f Adam Retter 27 May 2020, 18:56:53 UTC Add newer WBWI::NewIteratorWithBase functions to RocksJava (#6872) Summary: Exposes the `ReadOptions` arguments to `NewIteratorWithBase`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6872 Reviewed By: ajkr Differential Revision: D21725867 Pulled By: pdillinger fbshipit-source-id: 4079ba590cc13ba7a6244ed91439d89c40a543b6 27 May 2020, 18:59:12 UTC
82a82c7 Cheng Chang 26 May 2020, 22:27:29 UTC Fix potential memory leak of scratch buffer (#6879) Summary: If `req.scratch` is an internally allocated buffer, but `raw_block_contents` is not constructed to own `req.scratch`, then `req.scratch` will be leaked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6879 Test Plan: make asan_check Reviewed By: anand1976 Differential Revision: D21728498 Pulled By: cheng-chang fbshipit-source-id: 8fc6a4f2543918c565ddc16ecfad1807eb9a42cf 26 May 2020, 22:29:04 UTC
back to top