Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
fbed72f Andrew Kryczka 21 January 2021, 20:41:08 UTC bump version and update HISTORY.md for 6.15.4 21 January 2021, 20:41:08 UTC
1420cbf Andrew Kryczka 21 January 2021, 16:47:06 UTC workaround race conditions during `PeriodicWorkScheduler` registration (#7888) Summary: This provides a workaround for two race conditions that will be fixed in a more sophisticated way later. This PR: (1) Makes the client serialize calls to `Timer::Start()` and `Timer::Shutdown()` (see https://github.com/facebook/rocksdb/issues/7711). The long-term fix will be to make those functions thread-safe. (2) Makes `PeriodicWorkScheduler` atomically add/cancel work together with starting/shutting down its `Timer`. The long-term fix will be for `Timer` API to offer more specialized APIs so the client will not need to synchronize. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7888 Test Plan: ran the repro provided in https://github.com/facebook/rocksdb/issues/7881 Reviewed By: jay-zhuang Differential Revision: D25990891 Pulled By: ajkr fbshipit-source-id: a97fdaebbda6d7db7ddb1b146738b68c16c5be38 21 January 2021, 20:40:16 UTC
6002cce Ramkumar Vadivelu 07 January 2021, 17:32:27 UTC Bump the version file to 6.15.3 and update HISTORY.md with fix information. 07 January 2021, 17:32:27 UTC
c9e00cc Adam Retter 30 December 2020, 21:38:43 UTC Attempt to fix build errors around missing compression library includes (#7803) Summary: This fixes an issue introduced in https://github.com/facebook/rocksdb/pull/7769 that caused many errors about missing compression libraries to be displayed during compilation, although compilation actually succeeded. This PR fixes the compilation so the compression libraries are only introduced where strictly needed. It likely needs to be merged into the same branches as https://github.com/facebook/rocksdb/pull/7769 which I think are: 1. master 2. 6.15.fb 3. 6.16.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/7803 Reviewed By: ramvadiv Differential Revision: D25733743 Pulled By: pdillinger fbshipit-source-id: 6c04f6864b2ff4a345841d791a89b19e0e3f5bf7 07 January 2021, 17:27:26 UTC
ca80827 Ramkumar Vadivelu 22 December 2020, 17:43:58 UTC Bump the version file to 6.15.2 and update HISTORY.md with fix information. Also, fix a merge issue with .circleci/config.yml (missing install-cmake-on-macos in commands section) 22 December 2020, 17:43:58 UTC
12c87b1 Adam Retter 19 December 2020, 00:11:17 UTC Fix various small build issues, Java API naming (#7776) Summary: * Compatibility with older GCC. * Compatibility with older jemalloc libraries. * Remove Docker warning when building i686 binaries. * Fix case inconsistency in Java API naming (potential update to HISTORY.md deferred) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7776 Reviewed By: akankshamahajan15 Differential Revision: D25607235 Pulled By: pdillinger fbshipit-source-id: 7ab0fb7fa7a34e97ed0bec991f5081acb095777d 22 December 2020, 17:29:16 UTC
b39b6d7 Adam Retter 17 November 2020, 23:27:51 UTC Fix jemalloc compliation problem on macOS (#7624) Summary: Closes https://github.com/facebook/rocksdb/issues/7269 I have only tested this on macOS, let's see what CI makes of it for the other platforms... Pull Request resolved: https://github.com/facebook/rocksdb/pull/7624 Reviewed By: ajkr Differential Revision: D24834305 Pulled By: pdillinger fbshipit-source-id: ba818d8424297ccebd18ed854b044764c2dbab5f 22 December 2020, 17:27:57 UTC
ceee8ad Adam Retter 16 December 2020, 23:58:56 UTC Fix failing RocksJava test compilation and add CI (#7769) Summary: * Fixes a Java test compilation issue on macOS * Cleans up CircleCI RocksDBJava build config * Adds CircleCI for RocksDBJava on MacOS * Ensures backwards compatibility with older macOS via CircleCI * Fixes RocksJava static builds ordering * Adds missing RocksJava static builds to CircleCI for Mac and Linux * Improves parallelism in RocksJava builds * Reduces the size of the machines used for RocksJava CircleCI as they don't need to be so large (Saves credits) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7769 Reviewed By: akankshamahajan15 Differential Revision: D25601293 Pulled By: pdillinger fbshipit-source-id: 0a0bb9906f65438fe143487d78e37e1947364d08 22 December 2020, 17:22:47 UTC
c43be4f anand76 17 November 2020, 02:41:43 UTC Use default FileSystem in GenerateUniqueId (#7672) Summary: Use ```FileSystem::Default``` to read ```/proc/sys/kernel/uuid```, so it works for ```Envs``` with remote ```FileSystem``` as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7672 Reviewed By: riversand963 Differential Revision: D24998702 Pulled By: anand1976 fbshipit-source-id: fa95c1d70f0e4ed17561201f047aa055046d06c3 11 December 2020, 03:14:32 UTC
d15cc91 Neil Mitchell 07 December 2020, 18:26:20 UTC Make the TARGETS file Starlark compliant (#7743) Summary: Buck TARGETS files are sometimes parsed with Python, and sometimes with Starlark - this TARGETS file was not Starlark compliant. In Starlark you can't have a top-level if in a TARGETS file, but you can have a ternary `a if b else c`. Therefore I converted TARGETS, and updated the generator for it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7743 Reviewed By: pdillinger Differential Revision: D25342587 Pulled By: ndmitchell fbshipit-source-id: 88cbe8632071a45a3ea8675812967614c62c78d1 10 December 2020, 19:12:58 UTC
eea9a02 anand76 10 December 2020, 00:57:54 UTC Ensure that MultiGet works properly with compressed cache (#7756) Summary: Ensure that when direct IO is enabled and a compressed block cache is configured, MultiGet inserts compressed data blocks into the compressed block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7756 Test Plan: Add unit test to db_basic_test Reviewed By: cheng-chang Differential Revision: D25416240 Pulled By: anand1976 fbshipit-source-id: 75d57526370c9c0a45ff72651f3278dbd8a9086f 10 December 2020, 06:57:35 UTC
4b879ee Andrew Kryczka 01 December 2020, 23:00:23 UTC update HISTORY.md and bump version to 6.15.1 01 December 2020, 23:00:23 UTC
cac1fc4 Andrew Kryczka 01 December 2020, 02:10:18 UTC Fix kPointInTimeRecovery handling of truncated WAL (#7701) Summary: WAL may be truncated to an incomplete record due to crash while writing the last record or corruption. In the former case, no hole will be produced since no ACK'd data was lost. In the latter case, a hole could be produced without this PR since we proceeded to recover the next WAL as if nothing happened. This PR changes the record reading code to always report a corruption for incomplete records in `kPointInTimeRecovery` mode, and the upper layer will only ignore them if the next WAL has consecutive seqnum (i.e., we are guaranteed no hole). While this solves the hole problem for the case of incomplete records, the possibility is still there if the WAL is corrupted by truncation to an exact record boundary. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7701 Test Plan: Interestingly there already was a test for this case (`DBWALTestWithParams.kPointInTimeRecovery`); it just had a typo bug in the verification that prevented it from noticing holes in recovery. Reviewed By: anand1976 Differential Revision: D25111765 Pulled By: ajkr fbshipit-source-id: 5e330b13b1ee2b5be096cea9d0ff6075843e57b6 01 December 2020, 22:59:42 UTC
a5a5be4 anand76 20 November 2020, 06:37:55 UTC Fix initialization order of DBOptions and kHostnameForDbHostId (#7702) Summary: Fix initialization order of DBOptions and kHostnameForDbHostId by making the initialization of the latter static rather than dynamic. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7702 Reviewed By: ajkr Differential Revision: D25111633 Pulled By: anand1976 fbshipit-source-id: 7afad834a66e40bcd8694a43b40d378695212224 21 November 2020, 15:13:37 UTC
46923e5 Ramkumar Vadivelu 17 November 2020, 22:18:54 UTC Fixed a formatting issue in HISTORY.md 17 November 2020, 22:18:54 UTC
518431d Yanqin Jin 17 November 2020, 08:43:20 UTC cherry-picked PR-7680 and fixed HISTORY.md 17 November 2020, 15:16:30 UTC
5bd1258 Ramkumar Vadivelu 15 November 2020, 20:36:24 UTC Update release history to 6.15 (#7673) Summary: Update release history to 6.15 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7673 Test Plan: No code change Reviewed By: ajkr Differential Revision: D24971069 Pulled By: ramvadiv fbshipit-source-id: 5cb3f5cbc1b19beb580ea8095acdef72cc092905 15 November 2020, 20:37:24 UTC
1aae417 Cheng Chang 13 November 2020, 21:36:00 UTC Do not track WAL in MANIFEST when fsync is disabled in a test (#7669) Summary: If fsync is disabled in a unit test, then do not track WAL in MANIFEST, because on DB recovery, the WAL might be missing because the directory is not fsynced. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7669 Test Plan: Tests with fsync enabled should pass. Reviewed By: riversand963 Differential Revision: D24941431 Pulled By: cheng-chang fbshipit-source-id: ab3ff0f90769795cfb4e4d6dcf084ea5545d1975 13 November 2020, 21:37:14 UTC
9aa1b1d Yanqin Jin 13 November 2020, 19:51:24 UTC Hack to load OPTIONS file for read_amp_bytes_per_bit (#7659) Summary: A temporary hack to work around a bug in 6.10, 6.11, 6.12, 6.13 and 6.14. The bug will write out 8 bytes to OPTIONS file from the starting address of BlockBasedTableOptions.read_amp_bytes_per_bit which is actually a uint32. Consequently, the value of read_amp_bytes_per_bit written in the OPTIONS file is wrong. From 6.15, RocksDB will try to parse the read_amp_bytes_per_bit from OPTIONS file as a uint32. To be able to load OPTIONS file generated by affected releases before the fix, we need to manually parse read_amp_bytes_per_bit with this hack. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7659 Test Plan: Generate a db with current 6.14.fb (head at https://github.com/facebook/rocksdb/commit/b6db05dbb5364c658c5401a8078d73697bb5f31d). Maybe use db_stress. Checkout this PR, run ``` ~/rocksdb/ldb --db=. --try_load_options --ignore_unknown_options idump --count_only ``` Expect success, and should not see ``` Failed: Invalid argument: Error parsing read_amp_bytes_per_bit:17179869184 ``` Also make check Reviewed By: anand1976 Differential Revision: D24954752 Pulled By: riversand963 fbshipit-source-id: c7b802fc3e52acd050a4fc1cd475016122234394 13 November 2020, 19:52:50 UTC
e300ce2 Akanksha Mahajan 13 November 2020, 06:08:03 UTC Update option "allow_data_in_errors" in BuildOptions (#7665) Summary: "allow_data_in_errors" is not updated in BuildOptions. So it would assume default value when BuildOptions is called. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7665 Test Plan: make check -j64 Reviewed By: zhichao-cao Differential Revision: D24929100 Pulled By: akankshamahajan15 fbshipit-source-id: dd6225a6c9f13b20027ff1b6de8e79801b57b3f7 13 November 2020, 06:09:17 UTC
60af964 Peter Dillinger 13 November 2020, 04:45:02 UTC Experimental (production candidate) SST schema for Ribbon filter (#7658) Summary: Added experimental public API for Ribbon filter: NewExperimentalRibbonFilterPolicy(). This experimental API will take a "Bloom equivalent" bits per key, and configure the Ribbon filter for the same FP rate as Bloom would have but ~30% space savings. (Note: optimize_filters_for_memory is not yet implemented for Ribbon filter. That can be added with no effect on schema.) Internally, the Ribbon filter is configured using a "one_in_fp_rate" value, which is 1 over desired FP rate. For example, use 100 for 1% FP rate. I'm expecting this will be used in the future for configuring Bloom-like filters, as I expect people to more commonly hold constant the filter accuracy and change the space vs. time trade-off, rather than hold constant the space (per key) and change the accuracy vs. time trade-off, though we might make that available. ### Benchmarking ``` $ ./filter_bench -impl=2 -quick -m_keys_total_max=200 -average_keys_per_filter=100000 -net_includes_hashing Building... Build avg ns/key: 34.1341 Number of filters: 1993 Total size (MB): 238.488 Reported total allocated memory (MB): 262.875 Reported internal fragmentation: 10.2255% Bits/key stored: 10.0029 ---------------------------- Mixed inside/outside queries... Single filter net ns/op: 18.7508 Random filter net ns/op: 258.246 Average FP rate %: 0.968672 ---------------------------- Done. (For more info, run with -legend or -help.) $ ./filter_bench -impl=3 -quick -m_keys_total_max=200 -average_keys_per_filter=100000 -net_includes_hashing Building... Build avg ns/key: 130.851 Number of filters: 1993 Total size (MB): 168.166 Reported total allocated memory (MB): 183.211 Reported internal fragmentation: 8.94626% Bits/key stored: 7.05341 ---------------------------- Mixed inside/outside queries... Single filter net ns/op: 58.4523 Random filter net ns/op: 363.717 Average FP rate %: 0.952978 ---------------------------- Done. (For more info, run with -legend or -help.) ``` 168.166 / 238.488 = 0.705 -> 29.5% space reduction 130.851 / 34.1341 = 3.83x construction time for this Ribbon filter vs. lastest Bloom filter (could make that as little as about 2.5x for less space reduction) ### Working around a hashing "flaw" bloom_test discovered a flaw in the simple hashing applied in StandardHasher when num_starts == 1 (num_slots == 128), showing an excessively high FP rate. The problem is that when many entries, on the order of number of hash bits or kCoeffBits, are associated with the same start location, the correlation between the CoeffRow and ResultRow (for efficiency) can lead to a solution that is "universal," or nearly so, for entries mapping to that start location. (Normally, variance in start location breaks the effective association between CoeffRow and ResultRow; the same value for CoeffRow is effectively different if start locations are different.) Without kUseSmash and with num_starts > 1 (thus num_starts ~= num_slots), this flaw should be completely irrelevant. Even with 10M slots, the chances of a single slot having just 16 (or more) entries map to it--not enough to cause an FP problem, which would be local to that slot if it happened--is 1 in millions. This spreadsheet formula shows that: =1/(10000000*(1 - POISSON(15, 1, TRUE))) As kUseSmash==false (the setting for Standard128RibbonBitsBuilder) is intended for CPU efficiency of filters with many more entries/slots than kCoeffBits, a very reasonable work-around is to disallow num_starts==1 when !kUseSmash, by making the minimum non-zero number of slots 2*kCoeffBits. This is the work-around I've applied. This also means that the new Ribbon filter schema (Standard128RibbonBitsBuilder) is not space-efficient for less than a few hundred entries. Because of this, I have made it fall back on constructing a Bloom filter, under existing schema, when that is more space efficient for small filters. (We can change this in the future if we want.) TODO: better unit tests for this case in ribbon_test, and probably update StandardHasher for kUseSmash case so that it can scale nicely to small filters. ### Other related changes * Add Ribbon filter to stress/crash test * Add Ribbon filter to filter_bench as -impl=3 * Add option string support, as in "filter_policy=experimental_ribbon:5.678;" where 5.678 is the Bloom equivalent bits per key. * Rename internal mode BloomFilterPolicy::kAuto to kAutoBloom * Add a general BuiltinFilterBitsBuilder::CalculateNumEntry based on binary searching CalculateSpace (inefficient), so that subclasses (especially experimental ones) don't have to provide an efficient implementation inverting CalculateSpace. * Minor refactor FastLocalBloomBitsBuilder for new base class XXH3pFilterBitsBuilder shared with new Standard128RibbonBitsBuilder, which allows the latter to fall back on Bloom construction in some extreme cases. * Mostly updated bloom_test for Ribbon filter, though a test like FullBloomTest::Schema is a next TODO to ensure schema stability (in case this becomes production-ready schema as it is). * Add some APIs to ribbon_impl.h for configuring Ribbon filters. Although these are reasonably covered by bloom_test, TODO more unit tests in ribbon_test * Added a "tool" FindOccupancyForSuccessRate to ribbon_test to get data for constructing the linear approximations in GetNumSlotsFor95PctSuccess. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7658 Test Plan: Some unit tests updated but other testing is left TODO. This is considered experimental but laying down schema compatibility as early as possible in case it proves production-quality. Also tested in stress/crash test. Reviewed By: jay-zhuang Differential Revision: D24899349 Pulled By: pdillinger fbshipit-source-id: 9715f3e6371c959d923aea8077c9423c7a9f82b8 13 November 2020, 04:46:14 UTC
bbbb5a2 Levi Tamasi 13 November 2020, 02:57:20 UTC Add options for integrated blob GC (#7661) Summary: This patch simply adds a couple of options that will enable users to configure garbage collection when using the integrated BlobDB implementation. The actual GC logic will be added in a separate step. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7661 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24906544 Pulled By: ltamasi fbshipit-source-id: ee0e056a712a4b4475cd90de8b27d969bd61b7e1 13 November 2020, 02:58:44 UTC
76ef894 Yanqin Jin 13 November 2020, 02:43:30 UTC Add full_history_ts_low_ to FlushJob (#7655) Summary: https://github.com/facebook/rocksdb/issues/7556 enables `CompactionIterator` to perform garbage collection during compaction according to a lower bound (user-defined) timestamp `full_history_ts_low_`. This PR adds a data member `full_history_ts_low_` of type `std::string` to `FlushJob`, and `full_history_ts_low_` does not change during flush. `FlushJob` will pass a pointer to this data member to the `CompactionIterator` used during flush. Also refactored flush_job_test.cc to re-use some existing code, which is actually the majority of this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7655 Test Plan: make check Reviewed By: ltamasi Differential Revision: D24933340 Pulled By: riversand963 fbshipit-source-id: 2e584bfd0cf6e5c295ab1af264e68e9d6a12fca3 13 November 2020, 02:44:34 UTC
bb69b4c Levi Tamasi 13 November 2020, 01:31:28 UTC Fix InternalStats::DumpCFStats (#7666) Summary: https://github.com/facebook/rocksdb/pull/7461 accidentally broke `InternalStats::DumpCFStats` by making `DumpCFFileHistogram` overwrite the output of `DumpCFStatsNoFileHistogram` instead of appending to it, resulting in only the file histogram related information getting logged. The patch fixes this by reverting to appending in `DumpCFFileHistogram`. Fixes https://github.com/facebook/rocksdb/issues/7664 . Pull Request resolved: https://github.com/facebook/rocksdb/pull/7666 Test Plan: Ran `make check` and checked the info log of `db_bench`. Reviewed By: riversand963 Differential Revision: D24929051 Pulled By: ltamasi fbshipit-source-id: 636a3d5ebb5ce23de4f3fe4f03ad3f16cb2858f8 13 November 2020, 01:33:04 UTC
cf9d8e4 Yanqin Jin 12 November 2020, 19:40:52 UTC Add full_history_ts_low_ to CompactionJob (#7657) Summary: https://github.com/facebook/rocksdb/issues/7556 enables `CompactionIterator` to perform garbage collection during compaction according to a lower bound (user-defined) timestamp `full_history_ts_low_`. This PR adds a data member `full_history_ts_low_` of type `std::string` to `CompactionJob`, and `full_history_ts_low_` does not change during compaction. `CompactionJob` will pass a pointer to this data member to the `CompactionIterator` used during compaction. Also refactored compaction_job_test.cc to re-use some existing code, which is actually the majority of this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7657 Test Plan: make check Reviewed By: ltamasi Differential Revision: D24913803 Pulled By: riversand963 fbshipit-source-id: 11ad5329ddac365667152e7b3b02f84182c0ca8e 12 November 2020, 19:43:24 UTC
0dc437d Levi Tamasi 12 November 2020, 16:48:14 UTC Clean up CompactionProxy (#7662) Summary: `CompactionProxy` is currently both a concrete class used for actual `Compaction`s and a base class that `FakeCompaction` (which is used in `compaction_iterator_test`) is derived from. This is bad from an OO design standpoint, and also results in `FakeCompaction` containing an (uninitialized and unused) `Compaction*` member. The patch fixes this by making `CompactionProxy` a pure interface and introducing a separate concrete class `RealCompaction` for non-test/non-fake compactions. It also removes an unused parameter from the virtual method `level`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7662 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24907680 Pulled By: ltamasi fbshipit-source-id: c100ecb1beef4b0ada35e799116c5bda71719ee7 12 November 2020, 16:49:35 UTC
2400cd6 Yanqin Jin 12 November 2020, 16:36:50 UTC Update HISTORY.md for PR6069 (#7663) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7663 Reviewed By: ajkr Differential Revision: D24913081 Pulled By: riversand963 fbshipit-source-id: 704f427812f2b4f92e16d6cbc93be64d730d1cf9 12 November 2020, 16:38:41 UTC
ec346da Andrew Kryczka 12 November 2020, 04:30:58 UTC Always apply bottommost_compression_opts when enabled (#7633) Summary: Previously, even when `bottommost_compression_opts`'s `enabled` flag was set, it only took effect when `bottommost_compression` was also set to something other than `kDisableCompressionOption`. This wasn't documented and, if we kept the old behavior, it'd make things complicated like the migration instructions in https://github.com/facebook/rocksdb/issues/7619. We can simplify the API by making `bottommost_compression_opts` always take effect when its `enabled` flag is set. Fixes https://github.com/facebook/rocksdb/issues/7631. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7633 Reviewed By: ltamasi Differential Revision: D24710358 Pulled By: ajkr fbshipit-source-id: bbbdf9c1b53c63a4239d902cc3f5a11da1874647 12 November 2020, 04:32:28 UTC
c442f68 mrambacher 11 November 2020, 23:09:14 UTC Create a Customizable class to load classes and configurations (#6590) Summary: The Customizable class is an extension of the Configurable class and allows instances to be created by a name/ID. Classes that extend customizable can define their Type (e.g. "TableFactory", "Cache") and a method to instantiate them (TableFactory::CreateFromString). Customizable objects can be registered with the ObjectRegistry and created dynamically. Future PRs will make more types of objects extend Customizable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6590 Reviewed By: cheng-chang Differential Revision: D24841553 Pulled By: zhichao-cao fbshipit-source-id: d0c2132bd932e971cbfe2c908ca2e5db30c5e155 11 November 2020, 23:10:41 UTC
8b6b6ae Yanqin Jin 11 November 2020, 15:58:15 UTC Refactor with VersionEditHandler (#6581) Summary: Added a few classes in the same class hierarchy to remove code duplication and refactor the logic of reading and processing MANIFEST files. New classes are as follows. ``` class VersionEditHandlerBase; class ListColumnFamiliesHandler : VersionEditHandlerBase; class FileChecksumRetriever : VersionEditHandlerBase; class DumpManifestHandler : VersionEditHandler; ``` Classes that already existed before this PR are as follows. ``` class VersionEditHandler : VersionEditHandlerBase; ``` With these classes, refactored functions: `VersionSet::Recover()`, `VersionSet::ListColumnFamilies()`, `VersionSet::DumpManifest()`, `GetFileChecksumFromManifest()`. Test Plan (devserver): ``` make check COMPILE_WITH_ASAN=1 make check ``` These refactored code, especially recovery-related logic, will be tested intensively by all existing unit tests and stress tests. For example, run ``` make crash_test ``` Verified 3 successful runs on devserver. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6581 Reviewed By: ajkr Differential Revision: D20616217 Pulled By: riversand963 fbshipit-source-id: 048c7743aa4be2623ccd0cc3e61c0027e604e78b 11 November 2020, 16:00:14 UTC
c57f914 Peter Dillinger 11 November 2020, 07:41:05 UTC Use NPHash64 in more places (#7632) Summary: Since the hashes should not be persisted in output_validator nor mock_env. Also updated NPHash64 to use 64-bit seed, and comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7632 Test Plan: make check, and new build setting that enables modification to NPHash64, to check for behavior depending on specific values. Added that setting to one of the CircleCI configurations. Reviewed By: jay-zhuang Differential Revision: D24833780 Pulled By: pdillinger fbshipit-source-id: 02a57652ccf1ac105fbca79e77875bb7bf7c071f 11 November 2020, 07:42:13 UTC
bcba372 Yanqin Jin 11 November 2020, 01:16:04 UTC Report if unpinnable value encountered during backward iteration (#7618) Summary: There is an undocumented behavior about a certain combination of options and operations. - inplace_update_support = true, and - call `SeekForPrev()`, `SeekToLast()`, and/or `Prev()` on unflushed data. We should stop the backward iteration and report an error of `Status::NotSupported`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7618 Test Plan: make check Reviewed By: pdillinger Differential Revision: D24769619 Pulled By: riversand963 fbshipit-source-id: 81d199fa55ed4739ab10e719cc345a992238ccbb 11 November 2020, 01:17:39 UTC
18aee7d Jay Zhuang 10 November 2020, 22:51:38 UTC Fix a seek issue with prefix extractor and timestamp (#7644) Summary: During seek, prefix compare should not include timestamp. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7644 Test Plan: added unittest Reviewed By: riversand963 Differential Revision: D24772066 Pulled By: jay-zhuang fbshipit-source-id: 3982655a8bf8da256a738e8497b73b3d9bdac92e 10 November 2020, 22:53:13 UTC
16d103d Huisheng Liu 10 November 2020, 19:13:10 UTC fix read_amp_bytes_per_bit field size (#7651) Summary: The field in BlockBasedTableOptions is 4 bytes: // Default: 0 (disabled) uint32_t read_amp_bytes_per_bit = 0; Pull Request resolved: https://github.com/facebook/rocksdb/pull/7651 Reviewed By: ltamasi Differential Revision: D24844994 Pulled By: riversand963 fbshipit-source-id: e2695e55532256ef8996dd6939cad06987a80293 10 November 2020, 19:14:48 UTC
2026051 Akanksha Mahajan 10 November 2020, 18:48:46 UTC Fix crash test to run in DEBUG_LEVEL=0 mode in tmpfs (#7643) Summary: crash tests donot run in DEBUG_MODE=0 on tmpfs when use_direct_reads/use_direct_io_for_flush_and_compaction is set randomly because direct I/O is not supported on tmpfs and tests exit. Fix: Sanitize direct I/O read options in DEBUG_LEVEL=0 so that crash tests can run in tmpfs. When mmap_reads is set, direct I/O reads options are unset so we can sanitize direct I/O reads options in case of tmpfs as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7643 Test Plan: 1. export DEBUG_LEVEL=0; export TEST_TMPDIR="/dev/shm"; export CRASH_TEST_EXT_ARGS="--use_direct_reads=1 --mmap_read=0"; make crash_test -j64 2. In DEBUG_LEVEL=1 mode: make crash_test -j64 Reviewed By: jay-zhuang Differential Revision: D24766550 Pulled By: akankshamahajan15 fbshipit-source-id: 021720b2343c12c72004f84b26147625d3991d9e 10 November 2020, 18:50:34 UTC
9f1c84c Yanqin Jin 10 November 2020, 02:19:42 UTC Fix a bug in compaction iterator with timestamp (#7645) Summary: https://github.com/facebook/rocksdb/issues/7556 introduced support for compaction iterator to perform timestamp-aware garbage collection. However, there was a bug. The comparison between `ikey_.user_key` and `current_user_key_` should happen before `key_ = current_key_.SetInternalKey(key_, &ikey_);` (line 336 of compaction_iterator.cc). Otherwise, after this line, `current_key_` is always the same as `ikey_.user_key`. This PR also re-arranged the order of some data members because some of them are state variables of `CompactionIterator` while others are inputs from callers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7645 Test Plan: make check Reviewed By: ltamasi Differential Revision: D24845028 Pulled By: riversand963 fbshipit-source-id: c7e79914832701462b86867e8463cd463b6c0c25 10 November 2020, 02:23:31 UTC
c3911f1 Cheng Chang 09 November 2020, 16:42:41 UTC Track WAL in MANIFEST: Track deleted WALs in MANIFEST after recovering from the WALs (#7649) Summary: After replaying the WALs, the memtables are flushed synchronously to L0 instead of being flushed in background. Currently, we only track WAL obsoletion events in the code path of background flush jobs. This PR tracks these events in RecoverLogFiles. After this change, we can enable `track_and_verify_wal_in_manifest` in `db_stress`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7649 Test Plan: `python tools/db_crashtest.py whitebox` Reviewed By: riversand963 Differential Revision: D24824501 Pulled By: cheng-chang fbshipit-source-id: 207129f7b845c50b333680ce6818a68a2fad54b9 09 November 2020, 18:25:43 UTC
5e794b0 Cheng Chang 08 November 2020, 05:54:55 UTC Fix a recovery corner case (#7621) Summary: Consider the following sequence of events: 1. Db flushed an SST with file number N, appended to MANIFEST, and tried to sync the MANIFEST. 2. Syncing MANIFEST failed and db crashed. 3. Db tried to recover with this MANIFEST. In the meantime, no entry about the newly-flushed SST was found in the MANIFEST. Therefore, RocksDB replayed WAL and tried to flush to an SST file reusing the same file number N. This failed because file system does not support overwrite. Then Db deleted this file. 4. Db crashed again. 5. Db tried to recover. When db read the MANIFEST, there was an entry referencing N.sst. This could happen probably because the append in step 1 finally reached the MANIFEST and became visible. Since N.sst had been deleted in step 3, recovery failed. It is possible that N.sst created in step 1 is valid. Although step 3 would still fail since the MANIFEST was not synced properly in step 1 and 2, deleting N.sst would make it impossible for the db to recover even if the remaining part of MANIFEST was appended and visible after step 5. After this PR, in step 3, immediately after recovering from MANIFEST, a new MANIFEST is created, then we find that N.sst is not referenced in the MANIFEST, so we delete it, and we'll not reuse N as file number. Then in step 5, since the new MANIFEST does not contain N.sst, the recovery failure situation in step 5 won't happen. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7621 Test Plan: 1. some tests are updated, because these tests assume that new MANIFEST is created after WAL recovery. 2. a new unit test is added in db_basic_test to simulate step 3. Reviewed By: riversand963 Differential Revision: D24668144 Pulled By: cheng-chang fbshipit-source-id: 90d7487fbad2bc3714f5ede46ea949895b15ae3b 08 November 2020, 06:23:27 UTC
8b8a2e9 Peter Dillinger 08 November 2020, 00:39:14 UTC Ribbon: major re-work of hashing, seeds, and more (#7635) Summary: * Fully optimized StandardHasher, in terms of efficiently generating Start, CoeffRow, and ResultRow from a stock hash value, with sufficient independence between them to have no measurably degraded behavior. (Degraded behavior would be an FP rate higher than explainable by 2^-b and, if using a 32-bit stock hash function, expected stock hash collisions.) Details in code comments. * Our standard 64-bit and 32-bit hash functions do not exhibit sufficient independence on sequential seeds (for one Ribbon construction attempt to have independent probability from the next). I have worked around this in the Ribbon code by "pre-mixing" "ordinal seeds," sequentially tried and appropriate for storage in persisted metadata, into "raw seeds," ready for application and appropriate for in-memory storage. This way the pre-mixing step (though fast) is only applied on loading or configuring the structure, not on each query or banding add. * Fix a subtle flaw in which backtracking not clearing ResultRow data could lead to elevated FP rate on keys that were backtracked on and should (for generality) exhibit the same FP rate as novel keys. * Added a basic test for PhsfQuery and construction algorithms (map or "retrieval structure" rather than set or filter), and made a few trivial related fixes. * Better random configuration generation in unit tests * Some other minor cleanup / clarification / etc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7635 Test Plan: unit tests included Reviewed By: jay-zhuang Differential Revision: D24738978 Pulled By: pdillinger fbshipit-source-id: f9d03599d9e2ca3e30e9d3e7d81cd936b56f76f0 08 November 2020, 01:22:54 UTC
1e40696 Cheng Chang 07 November 2020, 00:30:44 UTC Track WAL in MANIFEST: LogAndApply WAL events to MANIFEST (#7601) Summary: When a WAL is synced, an edit is written to MANIFEST. After flushing memtables, the obsoleted WALs are piggybacked to MANIFEST while writing the new L0 files to MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7601 Test Plan: `track_and_verify_wals_in_manifest` is enabled by default for all tests extending `DBBasicTest`, and in db_stress_test. Unit test `wal_edit_test`, `version_edit_test`, and `version_set_test` are also updated. Watch all tests to pass. Reviewed By: ltamasi Differential Revision: D24553957 Pulled By: cheng-chang fbshipit-source-id: 66a569ff1bdced38e22900bd240b73113906e040 07 November 2020, 01:22:36 UTC
1ce105d Cheng Chang 06 November 2020, 22:25:32 UTC Disable fsync in DBMergeOperatorTest to save test time (#7640) Summary: The test often times out in internal test infra. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7640 Test Plan: watch test to pass internally Reviewed By: anand1976 Differential Revision: D24764928 Pulled By: cheng-chang fbshipit-source-id: 587f2afc97f52909837943fd938a86ca94544b2c 06 November 2020, 23:24:17 UTC
cdc7ba3 Cheng Chang 06 November 2020, 22:21:59 UTC DBTablePropertiesTest often times out in internal test infra (#7639) Summary: In this test, after flushing memtable, it will read directly from the sst files, so `env_do_fsync` was `true` to ensure that the flushed sst files can be read afterwards. Considering that the test does not last long, the data should be available in os buffer even without fsync, so this PR tries to disable fsync to reduce test time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7639 Test Plan: watch the test to pass in internal infra Reviewed By: anand1976 Differential Revision: D24764689 Pulled By: cheng-chang fbshipit-source-id: ef827611a3eaca04201e4280ae801d6c8e60c138 06 November 2020, 22:25:14 UTC
da42ece Cheng Chang 06 November 2020, 22:16:07 UTC Skip fsync in txn tests (#7641) Summary: The tests often times out in internal infra, skipping fsync should reduce test time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7641 Test Plan: watch existing tests to pass Reviewed By: anand1976 Differential Revision: D24765098 Pulled By: cheng-chang fbshipit-source-id: c62bf8110361aee901918d632cf4772435d05e8d 06 November 2020, 22:25:14 UTC
4c2aef0 Cheng Chang 06 November 2020, 17:57:09 UTC ColumnFamilyTest often times out in internal test infra (#7638) Summary: Tries to fix by skipping fsync. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7638 Test Plan: watch the tests to pass Reviewed By: jay-zhuang Differential Revision: D24764355 Pulled By: cheng-chang fbshipit-source-id: 9c21b177709025ca1943066d94da89324ed47655 06 November 2020, 18:25:20 UTC
8154336 Cheng Chang 06 November 2020, 17:38:02 UTC Disable fsync in db_range_del_test (#7637) Summary: This test often times out in internal test infra. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7637 Test Plan: watch test to pass Reviewed By: ajkr Differential Revision: D24763939 Pulled By: cheng-chang fbshipit-source-id: 6564ee2ef637e9faf6688d4b6a5d74a72a51c5e8 06 November 2020, 18:25:20 UTC
1f62721 cheng-chang 05 November 2020, 00:47:42 UTC Simplify a test case in Java ReadOnlyTest (#7608) Summary: The original test nests a lot of `try` blocks. This PR flattens these blocks into independent blocks, so that each `try` block closes the DB before opening the next DB instance. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7608 Test Plan: watch the existing java tests to pass Reviewed By: zhichao-cao Differential Revision: D24611621 Pulled By: cheng-chang fbshipit-source-id: d486c5d37ac25d4b860d739ef2cdd58e6064d42d 05 November 2020, 00:49:17 UTC
c9c9709 Xie Yanbo 05 November 2020, 00:07:39 UTC Update clang-format-diff.py (#7609) Summary: `llvm-mirror/clang` is archived. Get the `clang-format-diff.py` file from the active source. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7609 Reviewed By: ajkr Differential Revision: D24711608 Pulled By: pdillinger fbshipit-source-id: b115d8765ff23fbb8190290a170de21565daba84 05 November 2020, 00:09:01 UTC
b6d8e36 Yanqin Jin 04 November 2020, 18:43:17 UTC Compute NeedCompact() after table builder Finish() (#7627) Summary: In `BuildTable()`, we call `builder->Finish()` before evaluating `builder->NeedCompact()`. However, we call `builder->NeedCompact()` before `builder->Finish()` in compaction job. This can be wrong because the table properties collectors may rely on the success of `Finish()` to provide correct result for `NeedCompact()`. Test plan (on devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/7627 Reviewed By: ajkr Differential Revision: D24728741 Pulled By: riversand963 fbshipit-source-id: 5a0dce244e14eb1106c4f87021e6bebca82b486e 04 November 2020, 18:44:56 UTC
fde0cd7 Yanqin Jin 04 November 2020, 04:33:45 UTC Add API to verify whole sst file checksum (#7578) Summary: Existing API `VerifyChecksum()` allows application to verify sst files' block checksums. Since whole file, user-specified checksum is tracked in MANIFEST, we can expose a new API to verify sst files' file checksums. ``` // Compute table file checksums if applicable and compare with MANIFEST. // Returns OK if no file has mismatching whole-file checksum. Status DB::VerifyFileChecksums(const ReadOptions& /*read_options*/); ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7578 Test Plan: make check Reviewed By: pdillinger Differential Revision: D24436783 Pulled By: riversand963 fbshipit-source-id: 52b51519b842f2b3c4e3351998a97c86cbec85b3 04 November 2020, 04:34:56 UTC
06a92fc Akanksha Mahajan 03 November 2020, 21:54:01 UTC Add "max_write_buffer_size_to_maintain" to crash test (#7634) Summary: Add "max_write_buffer_size_to_maintain" to crash test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7634 Test Plan: make crash_test -j64 Reviewed By: zhichao-cao Differential Revision: D24710401 Pulled By: akankshamahajan15 fbshipit-source-id: 89e0412aaa56b2ef5a75603971b82f4b0b494ab7 03 November 2020, 21:55:18 UTC
746909c Peter Dillinger 03 November 2020, 20:45:18 UTC Ribbon: InterleavedSolutionStorage (#7598) Summary: The core algorithms for InterleavedSolutionStorage and the implementation SerializableInterleavedSolution make Ribbon fast for filter queries. Example output from new unit test: Simple outside query, hot, incl hashing, ns/key: 117.796 Interleaved outside query, hot, incl hashing, ns/key: 42.2655 Bloom outside query, hot, incl hashing, ns/key: 24.0071 Also includes misc cleanup of previous Ribbon code and comments. Some TODOs and FIXMEs remain for futher work / investigation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7598 Test Plan: unit tests included (integration work and tests coming later) Reviewed By: jay-zhuang Differential Revision: D24559209 Pulled By: pdillinger fbshipit-source-id: fea483cd354ba782aea3e806f2bc96e183d59441 03 November 2020, 20:46:36 UTC
0b94468 Yanqin Jin 03 November 2020, 17:47:06 UTC Avoid skipping a test in db_wal_test (#7628) Summary: Recent test report shows that some tests have been skipped. For DBWALTest that inherits from DBTestBase, the following will always be true, since `env_` is an instance of `SpecialEnv`, not `Env::Default()`. Thus the test will always be skipped. ``` if (options.env != Env::Default()) { ROCKSDB_GTEST_SKIP("Test requires default environment"); return; } ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7628 Test Plan: ./db_wal_test --gtest_filter=DBWALTest.TruncateLastLogAfterRecoverWithoutFlush MEM_ENV=1 ./db_wal_test --gtest_filter=DBWALTest.TruncateLastLogAfterRecoverWithoutFlush make check Reviewed By: jay-zhuang Differential Revision: D24693006 Pulled By: riversand963 fbshipit-source-id: 7f2a772492a0f11bff17bbf5e9f493e9e9a1c125 03 November 2020, 17:48:16 UTC
881e0dc Jay Zhuang 03 November 2020, 17:44:21 UTC Fix MultiGet unable to query timestamp data issue (#7589) Summary: The filter query key should not contain timestamp. The timestamp is stripped for Get(), but not MultiGet(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7589 Reviewed By: riversand963 Differential Revision: D24494661 Pulled By: jay-zhuang fbshipit-source-id: fc5ff40f9d683a89a760c6ff0ab3aed05a70c317 03 November 2020, 17:45:41 UTC
c992eb1 Yanqin Jin 03 November 2020, 03:47:00 UTC Avoid skipping a test in db_test2 (#7629) Summary: Test report shows that this test has been skipped recently due to a condition that will never meet. `env_` is not equal to `Env::Default()` for DBTest2 that inherits from DBTestBase. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7629 Test Plan: make check ./db_test2 --gtest_filter=DBTest2.PinnableSliceAndMmapReads Reviewed By: jay-zhuang Differential Revision: D24693317 Pulled By: riversand963 fbshipit-source-id: b1bbd5c1e05a6fa57c1de0d74462b69e3c2d5215 03 November 2020, 03:48:23 UTC
1adbceb Andrew Kryczka 03 November 2020, 03:20:15 UTC Expand effect of dictionary settings in `ColumnFamilyOptions::compression_opts` (#7619) Summary: In dictionary compression's initial implementation, in order to save CPU overhead, we only enabled it for bottom level under the assumption that the vast majority of data is stored there. At that time, there was no such thing as `ColumnFamilyOptions::bottommost_compression_opts`, so we just hardcoded disabling dictionary compression in flush and compactions to non-bottommost level. Now, we have users who generate all their files through flush and are considering using dictionary compression. To support such a use case, this PR expands the scope of `ColumnFamilyOptions::compression_opts` to additionally include flushed files and files generated by compaction to a non-bottommost level. Users can still get the old behavior by moving their dictionary settings to `ColumnFamilyOptions::bottommost_compression_opts` and explicitly enabling both that and `ColumnFamilyOptions::bottommost_compression`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7619 Reviewed By: ltamasi Differential Revision: D24665610 Pulled By: ajkr fbshipit-source-id: 656b90bce1033fe21c71e09af931ef5bde3e464c 03 November 2020, 03:21:11 UTC
a388c8c Andrew Kryczka 30 October 2020, 21:01:52 UTC Add recent fixes to HISTORY.md (#7617) Summary: The recently reverted behavior changes were released to at least one place internally, so we should mention the reverts in release notes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7617 Reviewed By: akankshamahajan15 Differential Revision: D24654343 Pulled By: ajkr fbshipit-source-id: eb64b2797d8508cd95a2dc2698122c1be29ce817 30 October 2020, 21:03:35 UTC
30beece mrambacher 30 October 2020, 01:43:03 UTC Return NotFound from TableFactory configuration errors during options loading (#7615) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7615 Reviewed By: riversand963 Differential Revision: D24637054 Pulled By: ajkr fbshipit-source-id: 7da20d44289eaa2387af4edf8c3c48057425cc1c 30 October 2020, 01:44:24 UTC
6773901 Akanksha Mahajan 29 October 2020, 22:50:27 UTC Add 6.14 branch to check_format_compatible.sh (#7613) Summary: Add 6.14 to check_format_compatible.sh Pull Request resolved: https://github.com/facebook/rocksdb/pull/7613 Test Plan: ./tools/check_format_compatible.sh Reviewed By: riversand963 Differential Revision: D24628535 Pulled By: akankshamahajan15 fbshipit-source-id: a8bf1d5505a1fcc8a5bedc5ff4fdf33a22c3f2e6 29 October 2020, 22:51:50 UTC
7eb2824 mrambacher 29 October 2020, 20:45:17 UTC Revert LoadLatestOptions handling of ignore_unknown_options if versions differ (#7612) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7612 Reviewed By: zhichao-cao Differential Revision: D24627054 Pulled By: riversand963 fbshipit-source-id: 451b4da742e3e84c7442bc7cc4959d39089b89d0 29 October 2020, 20:46:26 UTC
394210f Yanqin Jin 29 October 2020, 06:20:23 UTC Remove unused includes (#7604) Summary: This is a PR generated **semi-automatically** by an internal tool to remove unused includes and `using` statements. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7604 Test Plan: make check Reviewed By: ajkr Differential Revision: D24579392 Pulled By: riversand963 fbshipit-source-id: c4bfa6c6b08da1de186690d37eb73d8fff45aecd 29 October 2020, 06:22:27 UTC
99a0305 Jermy Li 29 October 2020, 01:11:52 UTC java: correct method name RocksDB.GetColumnFamilyMetaData() (#7606) Summary: update GetColumnFamilyMetaData() to getColumnFamilyMetaData() Pull Request resolved: https://github.com/facebook/rocksdb/pull/7606 Reviewed By: zhichao-cao Differential Revision: D24610298 Pulled By: cheng-chang fbshipit-source-id: d24f9b65478da1456f50747637dc95688af874de 29 October 2020, 01:13:27 UTC
ea347d8 Zhichao Cao 28 October 2020, 23:46:04 UTC Updated GenerateOneFileChecksum to use requested_checksum_func_name (#7586) Summary: CreateFileChecksumGenerator may uses requested_checksum_func_name in generator context to decide which generator will be used. GenerateOneFileChecksum has not being updated to use it, which will always get the generator when the name is empty. Fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7586 Test Plan: make check Reviewed By: riversand963 Differential Revision: D24491989 Pulled By: zhichao-cao fbshipit-source-id: d9fdfdd431240f0a9a2e781ddbd48a7d6c609aad 28 October 2020, 23:47:12 UTC
2404f8b jsteemann 28 October 2020, 20:45:21 UTC slightly improve jemalloc allocator API header (#7592) Summary: Fix a few typos and avoid a potential nullptr dereference. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7592 Reviewed By: zhichao-cao Differential Revision: D24582111 Pulled By: riversand963 fbshipit-source-id: 51e9260e8cad1fcdedd310c889f0faeec6efd937 28 October 2020, 20:47:12 UTC
248d10f vdimir 28 October 2020, 18:09:52 UTC Fix typo in arena.cc (#7593) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7593 Reviewed By: zhichao-cao Differential Revision: D24576218 Pulled By: riversand963 fbshipit-source-id: a3d77191362ca696ae9df643f97f4ab5b7ecff12 28 October 2020, 18:11:17 UTC
793e9b7 darionyaphet 28 October 2020, 17:47:18 UTC Remove duplicate close (#7594) Summary: Because `Close()` have called in `Destroy()` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7594 Reviewed By: zhichao-cao Differential Revision: D24576407 Pulled By: riversand963 fbshipit-source-id: eba70d73375fd47dd78ca64c6a1fab3628448276 28 October 2020, 17:48:53 UTC
9a690a7 Ramkumar Vadivelu 28 October 2020, 17:11:13 UTC In ParseInternalKey(), include corrupt key info in Status (#7515) Summary: Fixes Issue https://github.com/facebook/rocksdb/issues/7497 When allow_data_in_errors db_options is set, log error key details in `ParseInternalKey()` Have fixed most of the calls. Have few TODOs still pending - because have to make more deeper changes to pass in the allow_data_in_errors flag. Will do those in a separate PR later. Tests: - make check - some of the existing tests that exercise the "internal key too small" condition are: dbformat_test, cuckoo_table_builder_test - some of the existing tests that exercise the corrupted key path are: corruption_test, merge_helper_test, compaction_iterator_test Example of new status returns: - Key too small - `Corrupted Key: Internal Key too small. Size=5` - Corrupt key with allow_data_in_errors option set to false: `Corrupted Key: '<redacted>' seq:3, type:3` - Corrupt key with allow_data_in_errors option set to true: `Corrupted Key: '61' seq:3, type:3` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7515 Reviewed By: ajkr Differential Revision: D24240264 Pulled By: ramvadiv fbshipit-source-id: bc48f5d4475ac19d7713e16df37505b31aac42e7 28 October 2020, 17:12:58 UTC
6c2c063 Andrew Kryczka 28 October 2020, 16:59:12 UTC Require only one `Logger::Logv()` implementation (#7605) Summary: A user who extended `Logger` recently pointed out it is unusual to require they implement the two-argument `Logv()` overload when they've already implemented the three-argument `Logv()` overload. I agree with that and think we can fix it by only calling the two-argument overload from the default implementation of the three-argument overload. Then when the three-argument overload is overridden, RocksDB would not rely on the two-argument overload. Only `Logger::LogHeader()` needed adjustment to achieve this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7605 Reviewed By: riversand963 Differential Revision: D24584749 Pulled By: ajkr fbshipit-source-id: 9aabe040ac761c4c0dbebc4be046967403ecaf21 28 October 2020, 17:00:51 UTC
0e2e675 Peter Dillinger 27 October 2020, 18:56:51 UTC Give instructions instead of broken 2to3 for clang-format-diff.py (#7603) Summary: My previous change to use lib2to3 to migrate clang-format-diff.py for Python 2 only works if there's nothing to reformat. Instead, give instructions to download to REPO_ROOT. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7603 Test Plan: Try the instructions on a fresh CentOS 8 devserver Reviewed By: riversand963 Differential Revision: D24569608 Pulled By: pdillinger fbshipit-source-id: 1410ba163e016b226e883dec93fae3df9ed0eab2 27 October 2020, 18:58:17 UTC
f35f7f2 mrambacher 27 October 2020, 17:31:34 UTC Fix many tests to run with MEM_ENV and ENCRYPTED_ENV; Introduce a MemoryFileSystem class (#7566) Summary: This PR does a few things: 1. The MockFileSystem class was split out from the MockEnv. This change would theoretically allow a MockFileSystem to be used by other Environments as well (if we created a means of constructing one). The MockFileSystem implements a FileSystem in its entirety and does not rely on any Wrapper implementation. 2. Make the RocksDB test suite work when MOCK_ENV=1 and ENCRYPTED_ENV=1 are set. To accomplish this, a few things were needed: - The tests that tried to use the "wrong" environment (Env::Default() instead of env_) were updated - The MockFileSystem was changed to support the features it was missing or mishandled (such as recursively deleting files in a directory or supporting renaming of a directory). 3. Updated the test framework to have a ROCKSDB_GTEST_SKIP macro. This can be used to flag tests that are skipped. Currently, this defaults to doing nothing (marks the test as SUCCESS) but will mark the tests as SKIPPED when RocksDB is upgraded to a version of gtest that supports this (gtest-1.10). I have run a full "make check" with MEM_ENV, ENCRYPTED_ENV, both, and neither under both MacOS and RedHat. A few tests were disabled/skipped for the MEM/ENCRYPTED cases. The error_handler_fs_test fails/hangs for MEM_ENV (presumably a timing problem) and I will introduce another PR/issue to track that problem. (I will also push a change to disable those tests soon). There is one more test in DBTest2 that also fails which I need to investigate or skip before this PR is merged. Theoretically, this PR should also allow the test suite to run against an Env loaded from the registry, though I do not have one to try it with currently. Finally, once this is accepted, it would be nice if there was a CircleCI job to run these tests on a checkin so this effort does not become stale. I do not know how to do that, so if someone could write that job, it would be appreciated :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7566 Reviewed By: zhichao-cao Differential Revision: D24408980 Pulled By: jay-zhuang fbshipit-source-id: 911b1554a4d0da06fd51feca0c090a4abdcb4a5f 27 October 2020, 17:33:09 UTC
6134ce6 Yanqin Jin 27 October 2020, 01:20:43 UTC Perform post-flush updates of memtable list in a callback (#6069) Summary: Currently, the following interleaving of events can lead to SuperVersion containing both immutable memtables as well as the resulting L0. This can cause Get to return incorrect result if there are merge operands. This may also affect other operations such as single deletes. ``` time main_thr bg_flush_thr bg_compact_thr compact_thr set_opts_thr 0 | WriteManifest:0 1 | issue compact 2 | wait 3 | Merge(counter) 4 | issue flush 5 | wait 6 | WriteManifest:1 7 | wake up 8 | write manifest 9 | wake up 10 | Get(counter) 11 | remove imm V ``` The reason behind is that: one bg flush thread's installing new `Version` can be batched and performed by another thread that is the "leader" MANIFEST writer. This bg thread removes the memtables from current super version only after `LogAndApply` returns. After the leader MANIFEST writer signals (releasing mutex) this bg flush thread, it is possible that another thread sees this cf with both memtables (whose data have been flushed to the newest L0) and the L0 before this bg flush thread removes the memtables. To address this issue, each bg flush thread can pass a callback function to `LogAndApply`. The callback is responsible for removing the memtables. Therefore, the leader MANIFEST writer can call this callback and remove the memtables before releasing the mutex. Test plan (devserver) ``` $make merge_test $./merge_test --gtest_filter=MergeTest.MergeWithCompactionAndFlush $make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6069 Reviewed By: cheng-chang Differential Revision: D18790894 Pulled By: riversand963 fbshipit-source-id: e41bd600c0448b4f4b2deb3f7677f95e3076b4ed 27 October 2020, 01:23:01 UTC
a7a04b6 Levi Tamasi 26 October 2020, 20:50:03 UTC Integrate BlobFileBuilder into the compaction process (#7573) Summary: Similarly to how https://github.com/facebook/rocksdb/issues/7345 integrated blob file writing into the flush process, the patch adds support for writing blob files to the compaction logic. Namely, if `enable_blob_files` is set, large values encountered during compaction are extracted to blob files and replaced with blob indexes. The resulting blob files are then logged to the MANIFEST as part of the compaction job's `VersionEdit` and added to the `Version` alongside any table files written by the compaction. Any errors during blob file building fail the compaction job. There will be a separate follow-up patch to perform blob garbage collection during compactions. In addition, the patch continues to chip away at the mess around computing various compaction related statistics by eliminating some code duplication and by making the `num_output_files` and `bytes_written` stats more consistent for flushes, compactions, and recovery. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7573 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24404696 Pulled By: ltamasi fbshipit-source-id: 21216af3a172ad3ce8f85d11cd30923784ae426c 26 October 2020, 20:51:55 UTC
25d54c7 Peter Dillinger 26 October 2020, 03:43:04 UTC Ribbon: initial (general) algorithms and basic unit test (#7491) Summary: This is intended as the first commit toward a near-optimal alternative to static Bloom filters for SSTs. Stephan Walzer and I have agreed upon the name "Ribbon" for a PHSF based on his linear system construction in "Efficient Gauss Elimination for Near-Quadratic Matrices with One Short Random Block per Row, with Applications" ("SGauss") and my much faster "on the fly" algorithm for gaussian elimination (or for this linear system, "banding"), which can be faster than peeling while also more compact and flexible. See util/ribbon_alg.h for more detailed introduction and background. RIBBON = Rapid Incremental Boolean Banding ON-the-fly This commit just adds generic (templatized) core algorithms and a basic unit test showing some features, including the ability to construct structures within 2.5% space overhead vs. information theoretic lower bound. (Compare to cache-local Bloom filter's ~50% space overhead -> ~30% reduction anticipated.) This commit does not include the storage scheme necessary to make queries fast, especially for filter queries, nor fractional "result bits", but there is some description already and those implementations will come soon. Nor does this commit add FilterPolicy support, for use in SST files, but that will also come soon. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7491 Reviewed By: jay-zhuang Differential Revision: D24517954 Pulled By: pdillinger fbshipit-source-id: 0119ee597e250d7e0edd38ada2ba50d755606fa7 26 October 2020, 03:44:49 UTC
6595267 Yanqin Jin 24 October 2020, 05:58:05 UTC Allow compaction iterator to perform garbage collection (#7556) Summary: Add a threshold timestamp, full_history_ts_low_ of type `std::string*` to `CompactionIterator`, so that RocksDB can also perform garbage collection during compaction. * If full_history_ts_low_ is nullptr, then compaction iterator does not perform GC, preserving all timestamp history for all keys. Compaction iterator will treat user key with different timestamps as different user keys. * If full_history_ts_low_ is not nullptr, then compaction iterator performs GC. GC will look at keys older than `*full_history_ts_low_` and determine their eligibility based on factors including snapshots. Current rules of GC: * If an internal key is in the same snapshot as a previous counterpart with the same user key, and this key is eligible for GC, and the key is not single-delete or merge operand, then this key can be dropped. Note that the previous internal key cannot be a merge operand either. * If a tombstone is the most recent one in the earliest snapshot and it is eligible for GC, and keyNotExistsBeyondLevel() is true, then this tombstone can be dropped. * If a tombstone is the most recent one in a snapshot and it is eligible for GC, and the compaction is at bottommost level, then all other older internal keys of the same user key must also be eligible for GC, thus can be dropped * Single-delete, delete-range and merge are not currently supported. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7556 Test Plan: make check Reviewed By: ltamasi Differential Revision: D24507728 Pulled By: riversand963 fbshipit-source-id: 3c09c7301f41eed76dfcf4d1527e68cf6e0a8bb3 24 October 2020, 05:59:46 UTC
1b22432 Cheng Chang 24 October 2020, 05:48:00 UTC Track WAL in MANIFEST: persist WALs to and recover WALs from MANIFEST (#7256) Summary: This PR makes it able to `LogAndApply` `VersionEdit`s related to WALs, and also be able to `Recover` from MANIFEST with WAL related `VersionEdit`s. The `VersionEdit`s related to WAL are treated similarly as those related to column family operations, they are not applied to versions, but can be in a commit group. Mixing WAL related `VersionEdit`s with other types of edits will make logic in `ProcessManifestWrite` more complicated, so `VersionEdit`s related to WAL can either be WAL additions or deletions, like column family add and drop. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7256 Test Plan: a set of unit tests are added in `version_set_test.cc` Reviewed By: riversand963 Differential Revision: D23123238 Pulled By: cheng-chang fbshipit-source-id: 246be2ed4744fd03fa2738aba408aaa611d0379c 24 October 2020, 05:49:51 UTC
a16d1b2 Peter Dillinger 23 October 2020, 21:08:52 UTC Add Encode/DecodeFixedGeneric, coding_lean.h (#7587) Summary: To minimize dependencies for Ribbon filter code in progress, core part of coding.h for fixed sizes has been moved to coding_lean.h. Also, generic versions of these functions have been added to math128.h (since the generic versions are likely only to be used along with Unsigned128). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7587 Test Plan: Unit tests added for new functions Reviewed By: jay-zhuang Differential Revision: D24486718 Pulled By: pdillinger fbshipit-source-id: a69768f742379689442135fa52237c01dfe2647e 23 October 2020, 21:11:15 UTC
b1cdb8c jmn 23 October 2020, 00:30:11 UTC add StartTrace and EndTrace to stackable_db (#7585) Summary: In addition to trace block cache access, we want to support trace queries on MySQL. To achieve that StartTrace and EndTrace need to be added to the stackable_db.h Pull Request resolved: https://github.com/facebook/rocksdb/pull/7585 Reviewed By: zhichao-cao Differential Revision: D24482306 Pulled By: nmjnmjnmj fbshipit-source-id: de641b4837c64cd33b44b5bebaeae5d1527c8c31 23 October 2020, 00:31:54 UTC
d8ec0a7 Zhichao Cao 23 October 2020, 00:04:39 UTC Make FileType Public and Replace kLogFile with kWalFile (#7580) Summary: As suggested by pdillinger ,The name of kLogFile is misleading, in some tests, kLogFile is defined as info log. Replace it with kWalFile and move it to public, which will be used in https://github.com/facebook/rocksdb/issues/7523 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7580 Test Plan: make check Reviewed By: riversand963 Differential Revision: D24485420 Pulled By: zhichao-cao fbshipit-source-id: 955e3dacc1021bb590fde93b0a568ffe9ad80799 23 October 2020, 00:06:20 UTC
1c78e4b Ziyue Yang 22 October 2020, 18:03:10 UTC Make parallel compression optimization code tidier (#6888) Summary: This commit makes https://github.com/facebook/rocksdb/issues/6262's code change tidier and easier to understand by: 1. Wrapping parallel compression initialization and termination into common methods; 2. Wrapping BlockRep initialization, push/pop into common methods; 3. Wrapping file size estimation into common methods; 4. Fixing function declarations that use non-const reference; 5. Fixing some uninitialized variables; 6. Fixing first_block data race; 7. Making BlockRep::status check in BlockBasedTableBuilder::Finish only present if ok(); 8. Making assert(ok()) in BlockBasedTableBuilder::CompressAndVerifyBlock only present in non-parallel compression mode. In parallel compression mode, compression will abort if status is not OK; 9. Eliminating potential data race caused by BlockBasedTableBuilder::GetStatus() and BlockBasedTableBuilder::GetIOStatus() by returning status copy instead of unprotected reference. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6888 Reviewed By: ajkr Differential Revision: D21957110 Pulled By: jay-zhuang fbshipit-source-id: 3a29892f249209513f030349756cecd7736eae80 22 October 2020, 18:05:25 UTC
eef27d0 Akanksha Mahajan 22 October 2020, 03:16:30 UTC Bug fix to remove function calling in assert statement (#7581) Summary: Remove function calling in assert statement as assert is a no op in opt build and that function might not be called. This causes hang in closing RocksDB when refit level is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7581 Test Plan: make check -j64 Reviewed By: riversand963 Differential Revision: D24466420 Pulled By: akankshamahajan15 fbshipit-source-id: 97db4ec5a95ae693c3290e176a3c12a9b1ad2f6d 22 October 2020, 03:18:06 UTC
5227b31 Cheng Chang 21 October 2020, 21:02:00 UTC Fix unchecked statuses for transaction_test (#7572) Summary: When `ASSERT_STATUS_CHECKED` is enabled, `transaction_test` does not pass without this PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7572 Test Plan: `ASSERT_STATUS_CHECKED=1 make -j32 transaction_test && ./transaction_test` Reviewed By: zhichao-cao Differential Revision: D24404319 Pulled By: cheng-chang fbshipit-source-id: 13689035995366ab06d8eada3ea404e45fef8bc5 21 October 2020, 21:03:59 UTC
73dbe10 Cheng Chang 20 October 2020, 20:17:17 UTC Fix write_batch_test when ASSERT_STATUS_CHECKED=1 (#7575) Summary: Without this PR, `ASSERT_STATUS_CHECKED=1 make -j32 write_batch_test && ./write_batch_test` fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7575 Test Plan: ASSERT_STATUS_CHECKED=1 make -j32 write_batch_test && ./write_batch_test Reviewed By: zhichao-cao Differential Revision: D24411442 Pulled By: cheng-chang fbshipit-source-id: f67dc43c44d6afcc6d7e5ff15c6ae9bbf4dfc943 20 October 2020, 20:18:41 UTC
1eda625 mrambacher 20 October 2020, 18:51:51 UTC Revert `Status`es returned from pre-`Configurable` options functions (#7563) Summary: Further refinement of the earlier PR. Now the Status is NotFound with a subcode of PathNotFound. Also the existing functions for options parsing/loading are reverted to return InvalidArgument no matter in which way the user-provided arguments are deemed invalid. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7563 Reviewed By: zhichao-cao Differential Revision: D24422491 Pulled By: ajkr fbshipit-source-id: ba6b237cd0584d3f925c5ba0d349aeb8c250af67 20 October 2020, 18:53:28 UTC
fc9b416 Cheng Chang 20 October 2020, 18:48:48 UTC Fix typo in db_wal_test (#7571) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7571 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D24392577 Pulled By: cheng-chang fbshipit-source-id: c94f92db48270d0e215aa0f2782b0ff2e31bf708 20 October 2020, 18:50:30 UTC
b4cd51d Akanksha Mahajan 20 October 2020, 16:11:50 UTC Fix for stress test failure (#7574) Summary: Ignore read error in 'FilePrefetchBuffer::TryReadFromCache' as status is ignored and bool value is returned. Return error if prefetch fails in 'PrefetchTail' as we have planned to return Prefetch failures to users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7574 Test Plan: make check -j64, python -u tools/db_crashtest.py blackbox Reviewed By: zhichao-cao Differential Revision: D24408825 Pulled By: akankshamahajan15 fbshipit-source-id: feebda771415998253fbe54632f13e6e75b7a243 20 October 2020, 16:13:42 UTC
00751e4 anand76 19 October 2020, 18:37:05 UTC Add a host location property to TableProperties (#7479) Summary: This PR adds support for writing a location identifier of the DB host to SST files as a table property. By default, the hostname is used, but can be overridden by the user. There have been some recent corruptions in files written by ```SstFileWriter``` before checksumming, so this property can be used to trace it back to the writing host and checking the host for hardware isues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7479 Test Plan: Add new unit tests Reviewed By: pdillinger Differential Revision: D24340671 Pulled By: anand1976 fbshipit-source-id: 2038949fd8d160c0633ccb4f9da77740f19fa2a2 19 October 2020, 18:38:48 UTC
0ea7db7 Cheng Chang 19 October 2020, 17:12:53 UTC Abstract out LockManager interface (#7532) Summary: In order to be able to introduce more locking protocols, we need to abstract out the locking subsystem in TransactionDB into a set of interfaces. PR https://github.com/facebook/rocksdb/pull/7013 introduces interface `LockTracker`. This PR is a follow up to take the first step to abstract out a `LockManager` interface. Further modifications to the interface may be needed when introducing the first implementation of range lock. But the idea here is to put the range lock implementation based on range tree under the `utilities/transactions/lock/range/range_tree`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7532 Test Plan: point_lock_manager_test Reviewed By: ajkr Differential Revision: D24238731 Pulled By: cheng-chang fbshipit-source-id: 2a9458cd8b3fb008d9529dbc4d3b28c24631f463 19 October 2020, 17:14:42 UTC
ed90e2a Stanislav Tkach 16 October 2020, 18:01:50 UTC Add getters to the C API for env, universal compaction options and fifo compaction options (#7501) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7501 Reviewed By: ltamasi Differential Revision: D24344109 Pulled By: pdillinger fbshipit-source-id: d9a2b1b1cc8c8d8a96f13b8ae6814380caa10c96 16 October 2020, 18:04:01 UTC
f4ade82 Adam Retter 16 October 2020, 15:59:18 UTC Fix the CI badge for ppc64le Jenkins (#7561) Summary: Fixes the URL for the badge and link. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7561 Reviewed By: ltamasi Differential Revision: D24336928 Pulled By: pdillinger fbshipit-source-id: 41e04a5b036458e303aac3ae3b6129f572f0e9ce 16 October 2020, 16:00:56 UTC
e8cb32e Levi Tamasi 15 October 2020, 20:02:44 UTC Introduce BlobFileCache and add support for blob files to Get() (#7540) Summary: The patch adds blob file support to the `Get` API by extending `Version` so that whenever a blob reference is read from a file, the blob is retrieved from the corresponding blob file and passed back to the caller. (This is assuming the blob reference is valid and the blob file is actually part of the given `Version`.) It also introduces a cache of `BlobFileReader`s called `BlobFileCache` that enables sharing `BlobFileReader`s between callers. `BlobFileCache` uses the same backing cache as `TableCache`, so `max_open_files` (if specified) limits the total number of open (table + blob) files. TODO: proactively open/cache blob files and pin the cache handles of the readers in the metadata objects similarly to what `VersionBuilder::LoadTableHandlers` does for table files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7540 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D24260219 Pulled By: ltamasi fbshipit-source-id: a8a2a4f11d3d04d6082201b52184bc4d7b0857ba 15 October 2020, 20:04:47 UTC
fa2a8cd Adam Retter 15 October 2020, 08:11:39 UTC Update zstd dependency to 1.4.5 (#7560) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7560 Reviewed By: akankshamahajan15 Differential Revision: D24323048 Pulled By: zhichao-cao fbshipit-source-id: 2094ef089c37f570d4ea30d28d2f46f9fa1ce0f1 15 October 2020, 08:13:26 UTC
a8c89cc mrambacher 15 October 2020, 05:27:12 UTC Test for LoadLatestOptions (#7554) Summary: Make LoadLatestOptions return PathNotFound if the options file does not exist. Added tests for the LoadOptions related methods. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7554 Reviewed By: akankshamahajan15 Differential Revision: D24298985 Pulled By: zhichao-cao fbshipit-source-id: c9ae3cb12fc4a5bbef07743e1c1300f98a2441b3 15 October 2020, 05:28:55 UTC
ccbf468 Adam Retter 15 October 2020, 05:22:01 UTC Small JNI improvements (#7371) Summary: * Avoid some unnecessary array copy operations on read/write * Remove some duplicated code * Don't leak arrays on some exceptions * Fixed some doc comments Pull Request resolved: https://github.com/facebook/rocksdb/pull/7371 Reviewed By: jay-zhuang Differential Revision: D24312932 Pulled By: pdillinger fbshipit-source-id: 422fe6b98bbdb922a148922ac0d2d965c715176e 15 October 2020, 05:23:56 UTC
1a83f5a Stanislav Tkach 15 October 2020, 00:50:06 UTC Expose BackupableDBOptions in the C API (#7550) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7550 Reviewed By: jay-zhuang Differential Revision: D24315343 Pulled By: ajkr fbshipit-source-id: fc7855b630a50c00dcb940241942295932732f39 15 October 2020, 00:51:47 UTC
05fba96 Tomasz Posluszny 14 October 2020, 21:37:55 UTC Make RocksDB instance responsible for closing associated ColumnFamilyHandle instances (#7428) Summary: - Takes the burden off developer to close ColumnFamilyHandle instances before closing RocksDB instance - The change is backward-compatible ---- Previously the pattern for working with Column Families was: ```java try (final ColumnFamilyOptions cfOpts = new ColumnFamilyOptions().optimizeUniversalStyleCompaction()) { // list of column family descriptors, first entry must always be default column family final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList( new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOpts), new ColumnFamilyDescriptor("my-first-columnfamily".getBytes(), cfOpts) ); // a list which will hold the handles for the column families once the db is opened final List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<>(); try (final DBOptions options = new DBOptions() .setCreateIfMissing(true) .setCreateMissingColumnFamilies(true); final RocksDB db = RocksDB.open(options, "path/to/do", cfDescriptors, columnFamilyHandleList)) { try { // do something } finally { // NOTE user must explicitly frees the column family handles before freeing the db for (final ColumnFamilyHandle columnFamilyHandle : columnFamilyHandleList) { columnFamilyHandle.close(); } } // frees the column family options } } // frees the db and the db options ``` With the changes in this PR, the Java user no longer has to worry about manually closing the Column Families, which allows them to write simpler symmetrical create/free oriented code like this: ```java try (final ColumnFamilyOptions cfOpts = new ColumnFamilyOptions().optimizeUniversalStyleCompaction()) { // list of column family descriptors, first entry must always be default column family final List<ColumnFamilyDescriptor> cfDescriptors = Arrays.asList( new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, cfOpts), new ColumnFamilyDescriptor("my-first-columnfamily".getBytes(), cfOpts) ); // a list which will hold the handles for the column families once the db is opened final List<ColumnFamilyHandle> columnFamilyHandleList = new ArrayList<>(); try (final DBOptions options = new DBOptions() .setCreateIfMissing(true) .setCreateMissingColumnFamilies(true); final RocksDB db = RocksDB.open(options, "path/to/do", cfDescriptors, columnFamilyHandleList)) { // do something } // frees the column family options, then frees the db and the db options } } ``` **NOTE**: The changes in this PR are backwards API compatible, which means existing code using the original approach will also continue to function correctly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7428 Reviewed By: cheng-chang Differential Revision: D24063348 Pulled By: jay-zhuang fbshipit-source-id: 648d7526669923128c863ead94516bf4d50ac658 14 October 2020, 21:39:14 UTC
850cc0d Akanksha Mahajan 14 October 2020, 19:43:14 UTC Fix for clang_analyzer build failure in table_test (#7553) Summary: fix for clang_analyzer build failure in table_test because of potential memory leak of memtable in case of ASSERT failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7553 Test Plan: USE_CLANG=1 make analyze; make check -j64 Reviewed By: jay-zhuang Differential Revision: D24295042 Pulled By: akankshamahajan15 fbshipit-source-id: e9ea184367970fff3b520e33f3ceebf28d66ac8d 14 October 2020, 19:44:59 UTC
6528ecc Tomasz Posluszny 14 October 2020, 18:32:10 UTC Add event listeners to RocksJava (#7425) Summary: Allows adding event listeners in RocksJava. * Adds listeners getter and setter in `Options` and `DBOptions` classes. * Adds `EventListener` Java interface and base class for implementing custom event listener callbacks - `AbstractEventListener`, which has an underlying native callback class implementing C++ `EventListener` class. * `AbstractEventListener` class has mechanism for selectively enabling its callback methods in order to prevent invoking Java method if it is not implemented. This decreases performance cost in case only subset of event listener callback methods is needed - the JNI code for remaining "no-op" callbacks is not executed. * The code is covered by unit tests in `EventListenerTest.java`, there are also tests added for setting/getting listeners field in `OptionsTest.java` and `DBOptionsTest.java`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7425 Reviewed By: pdillinger Differential Revision: D24063390 Pulled By: jay-zhuang fbshipit-source-id: 508c359538983d6b765e70d9989c351794a944ee 14 October 2020, 18:33:52 UTC
b99fe1a Zhichao Cao 14 October 2020, 17:44:09 UTC Remove the status.PermitUncheckedError() from WriteGroup Destructor (#7555) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7555 Test Plan: ASSERT_STATUS_CHECKED=1 make -j48 error_handler_fs_test Reviewed By: riversand963 Differential Revision: D24299387 Pulled By: zhichao-cao fbshipit-source-id: 6c8aa91c4b6e2bc82580b8d2264c177068f5a32c 14 October 2020, 17:47:58 UTC
db87afb Akanksha Mahajan 14 October 2020, 17:43:37 UTC Return error if Get/Multi() fails in Prefetching Filter blocks (#7543) Summary: Right now all I/O failures under PartitionFilterBlock::CacheDependencies() is swallowed. Return error in case prefetch fails. On returning error in PartitionedFilterBlockReader::CacheDependencies was causing stress test failure because PrefetchBuffer is initialized with enable_ = true, as result when PosixMmapReadableFile::Read is called from Prefetch, scratch is ignored causing buffer to fill with garbage values. Initializing prefetch buffer by CreatePrefetchBuffer that sets enable_ with !ioptions.allow_mmap_reads fixed the problem as it returns without prefetching data if allow_mmap_reads is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7543 Test Plan: make check -j64; python -u tools/db_crashtest.py --simple blackbox Reviewed By: anand1976 Differential Revision: D24284596 Pulled By: akankshamahajan15 fbshipit-source-id: f3f0fd44b59dcf60645730436f28564a07884868 14 October 2020, 17:45:36 UTC
7b65666 Akanksha Mahajan 14 October 2020, 17:14:12 UTC Update IOTrace operations in stackable_db.h (#7514) Summary: Update IOTrace operations in stackabledb.h and also trace few other IO operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7514 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D24151202 Pulled By: akankshamahajan15 fbshipit-source-id: 112cd3d2041f8c6398b7b0ba1a783b8c93224d4a 14 October 2020, 17:16:15 UTC
c87c3a4 Jay Zhuang 14 October 2020, 00:58:48 UTC Add a missing bug fix in HISTORY.md (#7549) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7549 Reviewed By: ajkr, zhichao-cao Differential Revision: D24292032 Pulled By: jay-zhuang fbshipit-source-id: 0442283386ae20d10410a8d013a431d7cd282b22 14 October 2020, 01:00:17 UTC
back to top