Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
17 June 2024, 23:04:45 UTC
sort by:
Revision Author Date Message Commit Date
4d57a39 sdong 18 November 2021, 22:47:16 UTC Improve HISTORY.md 18 November 2021, 22:47:16 UTC
224da7f sdong 18 November 2021, 22:46:21 UTC Release 6.26.1 18 November 2021, 22:46:40 UTC
9674b84 Adam Retter 16 November 2021, 19:27:45 UTC Fix an error on GCC 4.8.5 where -Werror=unused-parameter fails (#9145) 16 November 2021, 19:27:45 UTC
7448618 Adam Retter 10 November 2021, 01:20:54 UTC Remove invalid RocksJava native entry (#9147) Summary: It seems that an incorrect native source file entry was introduced in https://github.com/facebook/rocksdb/pull/8999. For some reason it appears that CI was not run against that PR, and so the problem was not detected. This PR fixes the problem by removing the invalid entry, allowing RocksJava to build correctly again. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9147 Reviewed By: pdillinger Differential Revision: D32300976 fbshipit-source-id: dbd763b806bacf0fc08f4deaf07c63d0a266c4cf 10 November 2021, 17:56:44 UTC
52626f1 Adam Retter 08 November 2021, 16:19:54 UTC Restore Java 7 Compatibility (#9103) Summary: RocksDB should still compile on Java 7. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9103 Reviewed By: pdillinger Differential Revision: D32067561 Pulled By: mrambacher fbshipit-source-id: bbe9c18c8007ab3e113de4add56a84c9bde61c8e 10 November 2021, 00:54:25 UTC
7c25388 Peter Dillinger 29 October 2021, 15:17:24 UTC Fix EnvLibrados and add to CI (#9088) Summary: This feature was not part of any common or CI build, so no surprise it broke. Now we can at least ensure compilation. I don't know how to run the test successfully (missing config file) so it is bypassed for now. Fixes https://github.com/facebook/rocksdb/issues/9078 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9088 Test Plan: CI Reviewed By: mrambacher Differential Revision: D32009467 Pulled By: pdillinger fbshipit-source-id: 3e0d1e5fde7f0ece703d48a81479e1cc7392c25c 29 October 2021, 16:32:56 UTC
f72fd58 Yanqin Jin 21 October 2021, 04:33:32 UTC Fix atomic flush waiting forever for MANIFEST write (#9034) Summary: In atomic flush, concurrent background flush threads will commit to the MANIFEST one by one, in the order of the IDs of their picked memtables for all included column families. Each time, a background flush thread decides whether to wait based on two criteria: - Is db stopped? If so, don't wait. - Am I the one to commit the currently earliest memtable? If so, don't wait and ready to go. When atomic flush was implemented, error writing to or syncing the MANIFEST would cause the db to be stopped. Therefore, this background thread does not have to check for the background error while waiting. If there has been such an error, `DBStopped()` would have been true, and this thread will **not** wait forever. After we improved error handling, RocksDB may map an IOError while writing to MANIFEST to a soft error, if there is no WAL. This requires the background threads to check for background error while waiting. Otherwise, a background flush thread may wait forever. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9034 Test Plan: make check Reviewed By: zhichao-cao Differential Revision: D31639225 Pulled By: riversand963 fbshipit-source-id: e9ab07c4d8f2eade238adeefe3e42dd9a5a3ebbd 21 October 2021, 04:34:47 UTC
633f069 sdong 20 October 2021, 22:31:07 UTC Update Release Version to 6.26 (#9059) Summary: Before cutting release branch 6.26, update version.h and release notes Pull Request resolved: https://github.com/facebook/rocksdb/pull/9059 Reviewed By: ajkr Differential Revision: D31805126 fbshipit-source-id: ae85ccf06ec756fa21163161f53fd0b728e6e32e 20 October 2021, 22:32:01 UTC
0a73ada leipeng 20 October 2021, 21:04:58 UTC remove unused local obj and simpilify comple code (#9052) Summary: This PR does not change code sematics, it just changes for: 1. local obj `nonmem_w` and `lfile` are unused 2. null check for `delete ptr` is unnecessary 3. use `unique_ptr::reset` instead of `release` + `delete` Pull Request resolved: https://github.com/facebook/rocksdb/pull/9052 Reviewed By: zhichao-cao Differential Revision: D31801661 Pulled By: anand1976 fbshipit-source-id: 16a77d45da8c8833bf5bf3bce546bb3711b335df 20 October 2021, 21:08:05 UTC
0c53b41 leipeng 20 October 2021, 21:03:44 UTC db_impl_write.cc: use stats_ instead of immutable_db_options_.stats (#9053) Summary: This PR has no semantic changes, just to make code shorter. `stats_` has value same with `immutable_db_options_.stats`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9053 Reviewed By: zhichao-cao Differential Revision: D31801603 Pulled By: anand1976 fbshipit-source-id: cbd8fe478d3e90ae078ace49b4f2eb9bb028ccf6 20 October 2021, 21:04:59 UTC
4217d1b Andrew Kryczka 20 October 2021, 20:15:33 UTC Support `GetMapProperty()` with "rocksdb.dbstats" (#9057) Summary: This PR supports querying `GetMapProperty()` with "rocksdb.dbstats" to get the DB-level stats in a map format. It only reports cumulative stats over the DB lifetime and, as such, does not update the baseline for interval stats. Like other map properties, the string keys are not (yet) exposed in the public API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9057 Test Plan: new unit test Reviewed By: zhichao-cao Differential Revision: D31781495 Pulled By: ajkr fbshipit-source-id: 6f77d3aee8b4b1a015061b8c260a123859ceaf9b 20 October 2021, 20:17:00 UTC
c66b442 sdong 20 October 2021, 17:03:03 UTC Incremental Space Amp Compactions in Universal Style (#8655) Summary: This commit introduces incremental compaction in univeral style for space amplification. This follows the first improvement mentioned in https://rocksdb.org/blog/2021/04/12/universal-improvements.html . The implemention simply picks up files about size of max_compaction_bytes to compact and execute if the penalty is not too big. More optimizations can be done in the future, e.g. prioritizing between this compaction and other types. But for now, the feature is supposed to be functional and can often reduce frequency of full compactions, although it can introduce penalty. In order to add cut files more efficiently so that more files from upper levels can be included, SST file cutting threshold (for current file + overlapping parent level files) is set to 1.5X of target file size. A 2MB target file size will generate files like this: https://gist.github.com/siying/29d2676fba417404f3c95e6c013c7de8 Number of files indeed increases but it is not out of control. Two set of write benchmarks are run: 1. For ingestion rate limited scenario, we can see full compaction is mostly eliminated: https://gist.github.com/siying/959bc1186066906831cf4c808d6e0a19 . The write amp increased from 7.7 to 9.4, as expected. After applying file cutting, the number is improved to 8.9. In another benchmark, the write amp is even better with the incremental approach: https://gist.github.com/siying/d1c16c286d7c59c4d7bba718ca198163 2. For ingestion rate unlimited scenario, incremental compaction turns out to be too expensive most of the time and is not executed, as expected. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8655 Test Plan: Add unit tests to the functionality. Reviewed By: ajkr Differential Revision: D31787034 fbshipit-source-id: ce813e63b15a61d5a56e97bf8902a1b28e011beb 20 October 2021, 17:04:13 UTC
6d93b87 Zhichao Cao 19 October 2021, 22:53:16 UTC Add lowest_used_cache_tier to ImmutableDBOptions to enable or disable Secondary Cache (#9050) Summary: Currently, if Secondary Cache is provided to the lru cache, it is used by default. We add CacheTier to advanced_options.h to describe the cache tier we used. Add a `lowest_used_cache_tier` option to `DBOptions` (immutable) and pass it to BlockBasedTableReader to decide if secondary cache will be used or not. By default it is `CacheTier::kNonVolatileTier`, which means, we always use both block cache (kVolatileTier) and secondary cache (kNonVolatileTier). By set it to `CacheTier::kVolatileTier`, the DB will not use the secondary cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9050 Test Plan: added new tests Reviewed By: anand1976 Differential Revision: D31744769 Pulled By: zhichao-cao fbshipit-source-id: a0575ebd23e1c6dfcfc2b4c8578764e73b15bce6 19 October 2021, 22:54:23 UTC
f20b07c Jay Zhuang 19 October 2021, 22:21:53 UTC Add "Java API Changes" session in HISTORY (#9055) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9055 Reviewed By: ajkr Differential Revision: D31765398 Pulled By: jay-zhuang fbshipit-source-id: 77ed67d69415c9fbbfc1132b15310b293e3939c6 19 October 2021, 22:23:06 UTC
f053851 sdong 19 October 2021, 19:47:20 UTC Ignore non-overlapping levels when determinig grandparent files (#9051) Summary: Right now, when picking a compaction, grand parent files are from output_level + 1. This usually works, but if the level doesn't have any overlapping file, it will be more efficient to go further down. This is because the files are likely to be trivial moved further and might create a violation of max_compaction_bytes. This situation can naturally happen and might happen even more with TTL compactions. There is no harm to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9051 Test Plan: Run existing tests and see it passes. Also briefly run crash test. Reviewed By: ajkr Differential Revision: D31748829 fbshipit-source-id: 52b99ab4284dc816d22f34406d528a3c98ff6719 19 October 2021, 19:48:18 UTC
b234a3f Peter Dillinger 19 October 2021, 19:35:17 UTC Improve data block construction performance (#9040) Summary: ... by bypassing tracking of last_key in BlockBuilder when last_key is already known (for BlockBasedTableBuilder::data_block). I tried extracting a base class of BlockBuilder without the last_key tracking at all, but that became complicated by NewFlushBlockPolicy() in the public API referencing BlockBuilder, which would need to be the base class, and I don't want to replace nearly all the internal references to BlockBuilder. Possible follow-up: * Investigate / consider using AddWithLastKey in more places This improvement should stack with https://github.com/facebook/rocksdb/issues/9039 Pull Request resolved: https://github.com/facebook/rocksdb/pull/9040 Test Plan: TEST_TMPDIR=/dev/shm/rocksdb1 ./db_bench -benchmarks=fillseq -memtablerep=vector -allow_concurrent_memtable_write=false -num=50000000 Compiled with DEBUG_LEVEL=0 Test vs. control runs simulaneous for better accuracy, units = ops/sec Run 1: 278929 vs. 267799 (+4.2%) Run 2: 281836 vs. 267432 (+5.4%) Run 3: 278279 vs. 270454 (+2.9%) (This benchmark is chosen to have detectable signal-to-noise, not to represent expected improvement percent on real workloads.) Reviewed By: mrambacher Differential Revision: D31706033 Pulled By: pdillinger fbshipit-source-id: 8a50fe6fefdd67b6d7665ffa687bbdcf5ad0d5ec 19 October 2021, 19:36:21 UTC
0534393 Peter Dillinger 19 October 2021, 18:50:50 UTC Fix stress/crash test handling of SST unique IDs (#9054) Summary: Was not handling the case of OnTableFileCreated invoked for table file NOT created. Also improved error reporting and caught a missing status check. Also strengthened the db_stress listener to require file_size > 0 when status.ok(). We would be violating the API contract if status is OK and we didn't create a valid SST file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9054 Test Plan: make blackbox_crash_test for a while Reviewed By: zhichao-cao Differential Revision: D31765200 Pulled By: pdillinger fbshipit-source-id: 7c527f5531bc239a5efd7a7b018545d480f926e2 19 October 2021, 18:52:07 UTC
8fb3fe8 mrambacher 19 October 2021, 17:42:04 UTC Allow unregistered options to be ignored in DBOptions from files (#9045) Summary: Adds changes to DBOptions (comparable to ColumnFamilyOptions) to allow some option values to be ignored on rehydration from the Options file. This is necessary for some customizable classes that were not registered with the ObjectRegistry but are saved/restored from the Options file. All tests pass. Will run check_format_compatible.sh shortly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9045 Reviewed By: zhichao-cao Differential Revision: D31761664 Pulled By: mrambacher fbshipit-source-id: 300c2251639cce2b223481c3bb2a63877b1f3766 19 October 2021, 17:43:04 UTC
8d615a2 Alan Paxton 19 October 2021, 16:20:56 UTC New-style blob option bindings, Java option getter and improve/fix option parsing (#8999) Summary: Implementation of https://github.com/facebook/rocksdb/issues/8221, plus/including extension of Java options API to allow the get() of options from RocksDB. The extension allows more comprehensive testing of options at the Java side, by validating that the options are set at the C++ side. Variations on methods: MutableColumnFamilyOptions.MutableColumnFamilyOptionsBuilder getOptions() MutableDBOptions.MutableDBOptionsBuilder getDBOptions() retrieve the options via RocksDB C++ interfaces, and parse the resulting string into one of the Java-style option objects. This necessitated generalising the parsing of option strings in Java, which now parses the full range of option strings returned by the C++ interface, rather than a useful subset. This necessitates the list-separator being changed to :(colon) from , (comma). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8999 Reviewed By: jay-zhuang Differential Revision: D31655487 Pulled By: ltamasi fbshipit-source-id: c38e98145c81c61dc38238b0df580db176ce4efd 19 October 2021, 16:21:52 UTC
ad5325a Peter Dillinger 19 October 2021, 06:28:28 UTC Experimental support for SST unique IDs (#8990) Summary: * New public header unique_id.h and function GetUniqueIdFromTableProperties which computes a universally unique identifier based on table properties of table files from recent RocksDB versions. * Generation of DB session IDs is refactored so that they are guaranteed unique in the lifetime of a process running RocksDB. (SemiStructuredUniqueIdGen, new test included.) Along with file numbers, this enables SST unique IDs to be guaranteed unique among SSTs generated in a single process, and "better than random" between processes. See https://github.com/pdillinger/unique_id * In addition to public API producing 'external' unique IDs, there is a function for producing 'internal' unique IDs, with functions for converting between the two. In short, the external ID is "safe" for things people might do with it, and the internal ID enables more "power user" features for the future. Specifically, the external ID goes through a hashing layer so that any subset of bits in the external ID can be used as a hash of the full ID, while also preserving uniqueness guarantees in the first 128 bits (bijective both on first 128 bits and on full 192 bits). Intended follow-up: * Use the internal unique IDs in cache keys. (Avoid conflicts with https://github.com/facebook/rocksdb/issues/8912) (The file offset can be XORed into the third 64-bit value of the unique ID.) * Publish the external unique IDs in FileStorageInfo (https://github.com/facebook/rocksdb/issues/8968) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8990 Test Plan: Unit tests added, and checking of unique ids in stress test. NOTE in stress test we do not generate nearly enough files to thoroughly stress uniqueness, but the test trims off pieces of the ID to check for uniqueness so that we can infer (with some assumptions) stronger properties in the aggregate. Reviewed By: zhichao-cao, mrambacher Differential Revision: D31582865 Pulled By: pdillinger fbshipit-source-id: 1f620c4c86af9abe2a8d177b9ccf2ad2b9f48243 19 October 2021, 06:32:01 UTC
aa21896 anand76 19 October 2021, 06:00:57 UTC Add property_bag to FileOptions (#9030) Summary: Add a property_bag option in FileOptions for direct FileSystem users to pass custom properties to the provider in APIs such as NewRandomAccessFile, NewWritableFile etc. This field will be ignored/not populated by RocksDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9030 Reviewed By: zhichao-cao Differential Revision: D31630643 Pulled By: anand1976 fbshipit-source-id: 1e1ddc5e2933ecada99a94eada5f309b674a03e8 19 October 2021, 06:03:19 UTC
f0841d4 Giuseppe Ottaviano 19 October 2021, 04:23:28 UTC Fix out-of-bounds access in MultiDBParallelOpenTest (#9046) Summary: `dbs` should not be cleared, as it is reused later when reopening the DBs, so we have an out-of-bounds access with `dbnames[dbnum]`. The values left in the vector don't need to be reset, as the db pointer is an out parameter for `DB::Open`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9046 Reviewed By: pdillinger Differential Revision: D31738263 Pulled By: ot fbshipit-source-id: c619e947b8d3dbc3d896f29971f093d3e3c794d3 19 October 2021, 04:25:45 UTC
314de7e Jay Zhuang 19 October 2021, 03:31:32 UTC Make `DB::Close()` thread-safe (#8970) Summary: If `DB::Close()` is called in multi-thread env, the resource could be double released, which causes exception or assert. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8970 Test Plan: Test with multi-thread benchmark, with each thread try to close the DB at the end. Reviewed By: pdillinger Differential Revision: D31242042 Pulled By: jay-zhuang fbshipit-source-id: a61276b1b61e07732e375554106946aea86a23eb 19 October 2021, 03:32:35 UTC
86cf726 Alan Paxton 19 October 2021, 00:19:04 UTC keyMayExist() supports ByteBuffer (#9013) Summary: closes https://github.com/facebook/rocksdb/issues/7917 Implemented ByteBuffer API variants of Java keyMayExist() uniformly with and without column families, read options and return data values. Implemented 2 supporting C++ JNI methods. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9013 Reviewed By: mrambacher Differential Revision: D31665989 Pulled By: jay-zhuang fbshipit-source-id: 8adc1730217dba38d6fa7b31d788650a33e28af1 19 October 2021, 00:20:07 UTC
53a0ab2 Jay Zhuang 18 October 2021, 22:14:01 UTC Deflaky ObsoleteFilesTest (#9049) Summary: WaitForFlushMemTable() may only wait for mem flush but not background flush finishing. The the obsoleted file may not be purged yet. https://github.com/facebook/rocksdb/blob/fcaa7ff6381fe6052b37a1d013b14960ea23ac17/db/db_impl/db_impl_compaction_flush.cc#L2200-L2203 Use WaitForCompact() instead to wait for background flush job. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9049 Test Plan: `gtest-parallel ./obsolete_files_test --gtest_filter=ObsoleteFilesTest.DeleteObsoleteOptionsFile -r 1000` Reviewed By: zhichao-cao Differential Revision: D31737343 Pulled By: jay-zhuang fbshipit-source-id: 82276ebeae7c7c75a733d3e1fd1c130d45e4761f 18 October 2021, 22:15:23 UTC
b4326b5 Jay Zhuang 18 October 2021, 19:21:25 UTC Fix gcc-11 compile error (#9043) Summary: gcc11 added new static check. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9043 Test Plan: Added CI for gcc11 build Reviewed By: zhichao-cao Differential Revision: D31716005 Pulled By: jay-zhuang fbshipit-source-id: 9f53be6f2f9e58e39b83359f6bbf66f945d57429 18 October 2021, 19:22:37 UTC
908a999 Peter Dillinger 18 October 2021, 17:59:55 UTC Fix COMMIT_ID in regression_test.sh (#9047) Summary: https://github.com/facebook/rocksdb/issues/8031 broke internal tests. This should fix but also preserve the intended capability of getting git commit id when hg not used Pull Request resolved: https://github.com/facebook/rocksdb/pull/9047 Test Plan: already broken ¯\\_(ツ)_/¯ Reviewed By: zhichao-cao Differential Revision: D31732198 Pulled By: pdillinger fbshipit-source-id: 7dba8531ddca55a6de5e04978a1a1601aae4cee9 18 October 2021, 18:01:06 UTC
9d66d6d Peter Dillinger 18 October 2021, 15:34:45 UTC Two performance improvements in BlockBuilder (#9039) Summary: Primarily, this change reserves space in the std::string for building the next block once a block is finished, using `block_size` as reservation size. Note: also tried reusing same std::string in the common "unbuffered" path but that showed no benefit or regression. Secondarily, this slightly reduces the work in resetting `restarts_`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9039 Test Plan: TEST_TMPDIR=/dev/shm/rocksdb1 ./db_bench -benchmarks=fillseq -memtablerep=vector -allow_concurrent_memtable_write=false -num=50000000 Compiled with DEBUG_LEVEL=0 Test vs. control runs simulaneous for better accuracy, units = ops/sec Run 1, Primary change only: 292697 vs. 280267 (+4.4%) Run 2, Primary change only: 288763 vs. 279621 (+3.3%) Run 1, Secondary change only: 260065 vs. 254232 (+2.3%) Run 2, Secondary change only: 275925 vs. 272248 (+1.4%) Run 1, Both changes: 284890 vs. 270372 (+5.3%) Run 2, Both changes: 263511 vs. 258188 (+2.0%) Reviewed By: zhichao-cao Differential Revision: D31701253 Pulled By: pdillinger fbshipit-source-id: 7e40810afbb98e6b6446955e77bda59e69b19ffd 18 October 2021, 15:35:38 UTC
3ffb3ba Peter Dillinger 16 October 2021, 17:03:19 UTC Add (Live)FileStorageInfo API (#8968) Summary: New classes FileStorageInfo and LiveFileStorageInfo and 'experimental' function DB::GetLiveFilesStorageInfo, which is intended to largely replace several fragmented DB functions needed to create checkpoints and backups. This function is now used to create checkpoints and backups, because it fixes many (probably not all) of the prior complexities of checkpoint not having atomic access to DB metadata. This also ensures strong functional test coverage of the new API. Specifically, much of the old CheckpointImpl::CreateCustomCheckpoint has been migrated to and updated in DBImpl::GetLiveFilesStorageInfo, with the former now calling the latter. Also, the class FileStorageInfo in metadata.h compatibly replaces BackupFileInfo and serves as a new base class for SstFileMetaData. Some old fields of SstFileMetaData are still provided (for now) but deprecated. Although FileStorageInfo::directory is accurate when using db_paths and/or cf_paths, these have never been supported by Checkpoint nor BackupEngine and still are not. This change does now detect these cases and return NotSupported when appropriate. (More work needed for support.) Somehow this change broke ProgressCallbackDuringBackup, but the progress_callback logic was dubious to begin with because it would call the callback based on copy buffer size, not size actually copied. Logic and test updated to track size actually copied per-thread. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8968 Test Plan: tests updated. DB::GetLiveFilesStorageInfo mostly tested by use in CheckpointImpl. DBTest.SnapshotFiles updated to also test GetLiveFilesStorageInfo, including reading the data after DB close. Added CheckpointTest.CheckpointWithDbPath (NotSupported). Reviewed By: siying Differential Revision: D31242045 Pulled By: pdillinger fbshipit-source-id: b183d1ce9799e220daaefd6b3b5365d98de676c0 16 October 2021, 17:04:32 UTC
678ba5e matthewvon 15 October 2021, 20:05:17 UTC SyncPoint::Process thrashes heap ... fix it (#9023) Summary: The first parameter of SyncPoint::Process is "const std::string&". The majority, maybe all, of the actual calls to this function use a "const char *". The conversion before entering the function requires a construction of a std::string object on the heap. This std::object is then typically not needed because first use of the string is a rocksdb::Slice which has a less costly conversion of char * to slice. Example: We have a load and iterate test. The test loads 10m keys and iterates most via 10 rocksdb::Iterator objects. We used TCMALLOC to gather information about allocation and space usage during iterators. - Before this PR: test took 32 min 17 sec - After this PR: test took 1 min 14 sec The TCMALLOC top object list before this PR: <pre> Total: 5105999 objects 5003717 98.0% 98.0% 5009471 98.1% rocksdb::DBIter::MergeValuesNewToOld (inline) 20260 0.4% 98.4% 20260 0.4% std::__cxx11::basic_string::_M_mutate 15214 0.3% 98.7% 15214 0.3% rocksdb::UncompressBlockContentsForCompressionType (inline) 13408 0.3% 99.0% 13408 0.3% std::_Rb_tree::_M_emplace_hint_unique [clone .constprop.416] (inline) 12957 0.3% 99.2% 12957 0.3% std::_Rb_tree::_M_emplace_hint_unique [clone .constprop.405] (inline) 9327 0.2% 99.4% 9327 0.2% std::_Rb_tree::_M_copy (inline) 7691 0.2% 99.5% 9919 0.2% JVM_FindSignal 2859 0.1% 99.6% 2859 0.1% rocksdb::Cleanable::RegisterCleanup 2844 0.1% 99.7% 2844 0.1% std::map::operator[] (inline) </pre> The "MergeValuesNewToOld (inline)" objects are the #define wrappers to SyncPoint::Process. We discovered this in a 5.18 rocksdb release. There TCMALLOC was more specific that std::basic_string was being constructed. I believe that was before SyncPoint::Process was declared inline in subsequent releases. The TCMALLOC top object list after this PR: <pre> Total: 104911 objects 45090 43.0% 43.0% 45090 43.0% rocksdb::Cleanable::RegisterCleanup 29995 28.6% 71.6% 29995 28.6% rocksdb::LRUCacheShard::Insert 15229 14.5% 86.1% 15229 14.5% rocksdb::UncompressBlockContentsForCompressionType (inline) 4373 4.2% 90.3% 4551 4.3% JVM_FindSignal 2881 2.7% 93.0% 2881 2.7% rocksdb::::ReadBlockFromFile (inline) 1162 1.1% 94.1% 1176 1.1% rocksdb::BlockFetcher::ReadBlockContents (inline) 1036 1.0% 95.1% 1036 1.0% std::__cxx11::basic_string::_M_mutate 869 0.8% 95.9% 869 0.8% std::vector::_M_realloc_insert (inline) 806 0.8% 96.7% 806 0.8% SnmpAgent::GetVariables (inline) </pre> Pull Request resolved: https://github.com/facebook/rocksdb/pull/9023 Reviewed By: pdillinger Differential Revision: D31610907 Pulled By: mrambacher fbshipit-source-id: 574ff51b639dd46ad253a8e664a575f06b7cc85d 15 October 2021, 20:30:29 UTC
ab67557 jsteemann 14 October 2021, 23:13:27 UTC add IsSyncThreadSafe() override to EncryptedWritableFile (#8993) Summary: EncryptedWritableFile is derived from FSWritableFile, which implements the `IsSyncThreadSafe()` function as bool IsSyncThreadSafe() const { return false; } EncryptedWritableFile does not override this method from the base class, so the `IsSyncThreadSafe()` function on an EncryptedWritableFile will always return false. This change adds an override of `IsSyncThreadSafe()` to EncryptedWritableFile so that the latter will now ask its underlying `file_` object for the thread-safety of sync operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8993 Reviewed By: jay-zhuang Differential Revision: D31613123 Pulled By: ajkr fbshipit-source-id: b18625e21a9911744eef3215c29913490e4b6001 14 October 2021, 23:14:26 UTC
fbb09cf Levi Tamasi 14 October 2021, 23:00:02 UTC Mention a new BlobDB option in a couple of comments (#9038) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9038 Reviewed By: riversand963 Differential Revision: D31663906 fbshipit-source-id: a84ff07a1239f0c2eaeb3b776620fd0f7d4331bc 14 October 2021, 23:01:13 UTC
97b30de leipeng 14 October 2021, 21:44:15 UTC improve-histogram-performance: remove valueIndexMap_ (#8625) Summary: `valueIndexMap_` in histogram is redundant and search in `valueIndexMap_` is slower than search in `bucketValues_`. this PR delete `valueIndexMap_` and search in `bucketValues_` by `std::lower_bound` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8625 Reviewed By: zhichao-cao Differential Revision: D31613386 Pulled By: ajkr fbshipit-source-id: d7415d724f5c8f41f80cbe82afd7467cfad6f009 14 October 2021, 21:45:20 UTC
c246c9c mikael 14 October 2021, 21:37:37 UTC Remove -Wshorten-64-to-32 FreeBSD on aarch64 as it breaks compilation. (#9010) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9010 Reviewed By: pdillinger Differential Revision: D31613434 Pulled By: ajkr fbshipit-source-id: 305a84fa715d0121fa65abaea3d32bad41233957 14 October 2021, 21:38:47 UTC
140db3c Ubuntu 14 October 2021, 20:23:01 UTC Fix rate limit for MixGraph (#9027) Summary: Fix race conditions of the read and write limiters. Close https://github.com/facebook/rocksdb/issues/8215 . Pull Request resolved: https://github.com/facebook/rocksdb/pull/9027 Reviewed By: ajkr Differential Revision: D31645361 Pulled By: zhichao-cao fbshipit-source-id: 8ea7731991da422eecff2790c1e32db44c751965 14 October 2021, 20:24:29 UTC
e5aa7de Huachao Huang 14 October 2021, 19:25:53 UTC Fix format script for Ubuntu (#9028) Summary: I get `clang-format-diff` after running `apt install clang-format` on Ubuntu instead of `clang-format-diff.py`. So I think it makes sense to make the format script compatible with this behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9028 Reviewed By: ajkr Differential Revision: D31634041 Pulled By: jay-zhuang fbshipit-source-id: b936de791ddcafa6ff304039ef33936e1e04864d 14 October 2021, 19:27:00 UTC
4c277ab leipeng 14 October 2021, 19:00:23 UTC MergingIterator: rearrange fields to reduce paddings (#9024) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9024 Reviewed By: pdillinger Differential Revision: D31614752 Pulled By: ajkr fbshipit-source-id: ef19ae243127f992e982a5a3b8ddefe7946246f8 14 October 2021, 19:01:56 UTC
f5526af Alan Paxton 14 October 2021, 18:46:59 UTC Fix multiget throwing NPE for num of keys > 70k (#9012) Summary: closes https://github.com/facebook/rocksdb/issues/8039 Unnecessary use of multiple local JNI references at the same time, 1 per key, was limiting the size of the key array. The local references don't need to be held simultaneously, so if we rearrange the code we can make it work for bigger key arrays. Incidentally, make errors throw helpful exception messages rather than returning a null pointer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9012 Reviewed By: mrambacher Differential Revision: D31580862 Pulled By: jay-zhuang fbshipit-source-id: ce05831d52ede332e1b20e74d2dc621d219b9616 14 October 2021, 18:48:12 UTC
ffc48b6 Andrew Kryczka 14 October 2021, 16:33:16 UTC Update HISTORY.md for #9009 (#9036) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9036 Reviewed By: zhichao-cao Differential Revision: D31640901 Pulled By: ajkr fbshipit-source-id: 0b1e6e36094a74bb7906af44e29ecbeaa258de58 14 October 2021, 16:36:32 UTC
3741810 anand76 13 October 2021, 20:17:54 UTC Set perf_level to kEnableTimeExceptForMutex in regression script (#8031) Summary: Set the perf_level in ```tools/regression_test.sh``` in order to exercise ```PerfContext``` counters in regression tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8031 Test Plan: Manually run the script Reviewed By: ajkr Differential Revision: D31508269 Pulled By: anand1976 fbshipit-source-id: 20ddfd1cbca37f1439eed2870086a86d90653b44 13 October 2021, 20:18:48 UTC
4bfd415 Giuseppe Ottaviano 13 October 2021, 03:38:36 UTC Fix sequence number bump logic in multi-CF SST ingestion (#9005) Summary: The code in `IngestExternalFiles()` that bumps the DB's sequence number depending on what seqnos were assigned to the files has 3 bugs: 1) There is an assertion that the sequence number is increased in all the affected column families, but this is unnecessary, it is fine if some files can stick to a lower sequence number. It is very easy to hit the assertion: it is sufficient to insert 2 files in 2 CFs, one which overlaps the CF and one that doesn't (for example the CF is empty). The line added in the `IngestFilesIntoMultipleColumnFamilies_Success` test makes the assertion fail. 2) SetLastSequence() is called with the sum of all the bumps across CFs, but we should take the maximum instead, as all CFs start with the current seqno and bump it independently. 3) The code above is accidentally under a `#ifndef NDEBUG`, so it doesn't run in optimized builds, so some files may be assigned seqnos from the future. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9005 Test Plan: Added line in `IngestFilesIntoMultipleColumnFamilies_Success` that triggers the assertion, verified that the test (and all the others) pass after the fix. Reviewed By: ajkr Differential Revision: D31597892 Pulled By: ot fbshipit-source-id: c2d3237f90290df1178736ace8653a9623f5a770 13 October 2021, 03:39:52 UTC
b4e59a4 Levi Tamasi 12 October 2021, 18:33:36 UTC Add a benchmarking wrapper script for BlobDB (#9015) Summary: The patch adds a new BlobDB benchmarking script called `run_blob_bench.sh`. It is a thin wrapper around `benchmark.sh` (similarly to `run_flash_bench.sh`): it actually calls `benchmark.sh` a number of times, cycling through six workloads, two write-only ones (bulk load and overwrite), two read/write ones (point lookups while writing, range scans while writing), and two read-only ones (point lookups and range scans). Note: this is a simpler/cleaned up/reworked version of the script used to produce the benchmark results in http://rocksdb.org/blog/2021/05/26/integrated-blob-db.html . The new version takes advantage of several recent `benchmark.sh` improvements like the ability to pass in arbitrary `db_bench` options or the possibility of using a job ID. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9015 Test Plan: Ran the script manually with different parameter combinations. Reviewed By: riversand963 Differential Revision: D31555277 Pulled By: ltamasi fbshipit-source-id: 0e151b2f7b2cf6f66ed7f95455571492ad7ea87f 12 October 2021, 18:36:03 UTC
7cc52cd Levi Tamasi 12 October 2021, 17:18:42 UTC Update HISTORY for PR 8994 (#9017) Summary: Also, expand on/clarify a comment in `VersionStorageInfoTest`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9017 Reviewed By: riversand963 Differential Revision: D31566130 Pulled By: ltamasi fbshipit-source-id: 1d30c7af084c4de7b2030bc6c768838d65746010 12 October 2021, 17:19:56 UTC
22d4dc5 Giuseppe Ottaviano 12 October 2021, 07:14:41 UTC Fix race in WriteBufferManager (#9009) Summary: EndWriteStall has a data race: `queue_.empty()` is checked outside of the mutex, so once we enter the critical section another thread may already have cleared the list, and accessing the `front()` is undefined behavior (and causes interesting crashes under high concurrency). This PR fixes the bug, and also rewrites the logic to make it easier to reason about it. It also fixes another subtle bug: if some writers are stalled and `SetBufferSize(0)` is called, which disables the WBM, the writer are not unblocked because of an early `enabled()` check in `EndWriteStall()`. It doesn't significantly change the locking behavior, as before writers won't lock unless entering a stall condition, and `FreeMem` almost always locks if stalling is allowed, but that is inevitable with the current design. Liveness is guaranteed by the fact that if some writes are blocked, eventually all writes will be blocked due to `stall_active_`, and eventually all memory is freed. While at it, do a couple of optimizations: - In `WBMStallInterface::Signal()` signal the CV only after releasing the lock. Signaling under the lock is a common pitfall, as it causes the woken-up thread to immediately go back to sleep because the mutex is still locked by the awaker. - Move all allocations and deallocations outside of the lock. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9009 Test Plan: ``` USE_CLANG=1 make -j64 all check ``` Reviewed By: akankshamahajan15 Differential Revision: D31550668 Pulled By: ot fbshipit-source-id: 5125387c3dc7ecaaa2b8bbc736e58c4156698580 12 October 2021, 07:16:21 UTC
e113916 Yanqin Jin 12 October 2021, 01:12:48 UTC Inline an empty destructor (#9004) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9004 Inline an empty destructor Reviewed By: ltamasi Differential Revision: D31525561 fbshipit-source-id: 3b9e37f06b0c70529a5d2d660de21ea335c73611 12 October 2021, 01:14:10 UTC
1a79839 Yanqin Jin 12 October 2021, 01:12:48 UTC Some code cleanup (#9003) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/9003 cleanup some code before real work. Reviewed By: ltamasi Differential Revision: D31525563 fbshipit-source-id: 44558b3594f2200adc7d8621b08b06c77e358a27 12 October 2021, 01:14:10 UTC
3e1bf77 Levi Tamasi 12 October 2021, 01:00:44 UTC Make it possible to force the garbage collection of the oldest blob files (#8994) Summary: The current BlobDB garbage collection logic works by relocating the valid blobs from the oldest blob files as they are encountered during compaction, and cleaning up blob files once they contain nothing but garbage. However, with sufficiently skewed workloads, it is theoretically possible to end up in a situation when few or no compactions get scheduled for the SST files that contain references to the oldest blob files, which can lead to increased space amp due to the lack of GC. In order to efficiently handle such workloads, the patch adds a new BlobDB configuration option called `blob_garbage_collection_force_threshold`, which signals to BlobDB to schedule targeted compactions for the SST files that keep alive the oldest batch of blob files if the overall ratio of garbage in the given blob files meets the threshold *and* all the given blob files are eligible for GC based on `blob_garbage_collection_age_cutoff`. (For example, if the new option is set to 0.9, targeted compactions will get scheduled if the sum of garbage bytes meets or exceeds 90% of the sum of total bytes in the oldest blob files, assuming all affected blob files are below the age-based cutoff.) The net result of these targeted compactions is that the valid blobs in the oldest blob files are relocated and the oldest blob files themselves cleaned up (since *all* SST files that rely on them get compacted away). These targeted compactions are similar to periodic compactions in the sense that they force certain SST files that otherwise would not get picked up to undergo compaction and also in the sense that instead of merging files from multiple levels, they target a single file. (Note: such compactions might still include neighboring files from the same level due to the need of having a "clean cut" boundary but they never include any files from any other level.) This functionality is currently only supported with the leveled compaction style and is inactive by default (since the default value is set to 1.0, i.e. 100%). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8994 Test Plan: Ran `make check` and tested using `db_bench` and the stress/crash tests. Reviewed By: riversand963 Differential Revision: D31489850 Pulled By: ltamasi fbshipit-source-id: 44057d511726a0e2a03c5d9313d7511b3f0c4eab 12 October 2021, 01:03:01 UTC
a282eff Andrew Kryczka 11 October 2021, 23:22:10 UTC Protect existing files in `FaultInjectionTest{Env,FS}::ReopenWritableFile()` (#8995) Summary: `FaultInjectionTest{Env,FS}::ReopenWritableFile()` functions were accidentally deleting WALs from previous `db_stress` runs causing verification to fail. They were operating under the assumption that `ReopenWritableFile()` would delete any existing file. It was a reasonable assumption considering the `{Env,FileSystem}::ReopenWritableFile()` documentation stated that would happen. The only problem was neither the implementations we offer nor the "real" clients in RocksDB code followed that contract. So, this PR updates the contract as well as fixing the fault injection client usage. The fault injection change exposed that `ExternalSSTFileBasicTest.SyncFailure` was relying on a fault injection `Env` dropping unsynced data written by a regular `Env`. I changed that test to make its `SstFileWriter` use fault injection `Env`, and also implemented `LinkFile()` in fault injection so the unsynced data is tracked under the new name. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8995 Test Plan: - Verified it fixes the following failure: ``` $ ./db_stress --clear_column_family_one_in=0 --column_families=1 --db=/dev/shm/rocksdb_crashtest_whitebox --delpercent=5 --expected_values_dir=/dev/shm/rocksdb_crashtest_expected --iterpercent=0 --key_len_percent_dist=1,30,69 --max_key=100000 --max_key_len=3 --nooverwritepercent=1 --ops_per_thread=1000 --prefixpercent=0 --readpercent=60 --reopen=0 --target_file_size_base=1048576 --test_batches_snapshots=0 --write_buffer_size=1048576 --writepercent=35 --value_size_mult=33 -threads=1 ... $ ./db_stress --avoid_flush_during_recovery=1 --clear_column_family_one_in=0 --column_families=1 --db=/dev/shm/rocksdb_crashtest_whitebox --delpercent=5 --destroy_db_initially=0 --expected_values_dir=/dev/shm/rocksdb_crashtest_expected --iterpercent=10 --key_len_percent_dist=1,30,69 --max_bytes_for_level_base=4194304 --max_key=100000 --max_key_len=3 --nooverwritepercent=1 --open_files=-1 --open_metadata_write_fault_one_in=8 --open_write_fault_one_in=16 --ops_per_thread=1000 --prefix_size=-1 --prefixpercent=0 --readpercent=50 --sync=1 --target_file_size_base=1048576 --test_batches_snapshots=0 --write_buffer_size=1048576 --writepercent=35 --value_size_mult=33 -threads=1 ... Verification failed for column family 0 key 000000000000001300000000000000857878787878 (1143): Value not found: NotFound: Crash-recovery verification failed :( ... ``` - `make check -j48` Reviewed By: ltamasi Differential Revision: D31495388 Pulled By: ajkr fbshipit-source-id: 7886ccb6a07cb8b78ad7b6c1c341ccf40bb68385 11 October 2021, 23:23:18 UTC
ee239df Andrew Kryczka 11 October 2021, 20:04:25 UTC Initialize cache dumper `DumpUnit` in constructor (#9014) Summary: Should fix clang-analyze: ``` utilities/cache_dump_load_impl.cc:296:38: warning: The left operand of '!=' is a garbage value while (io_s.ok() && dump_unit.type != CacheDumpUnitType::kFooter) { ~~~~~~~~~~~~~~ ^ ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/9014 Reviewed By: zhichao-cao Differential Revision: D31546912 Pulled By: ajkr fbshipit-source-id: a2e0dc7874e8c1c6abf190862b5d49e6a6ad6d01 11 October 2021, 20:05:35 UTC
6c3bf83 Hui Xiao 08 October 2021, 23:59:21 UTC Update HISTORY.md for #8428 (#9001) Summary: Context: HISTORY.md was not properly updated along with the change in https://github.com/facebook/rocksdb/pull/8428, where we introduced a change of accounting compression dictionary buffering memory and an extra condition of triggering data unbuffering. Updated HISTORY.md for https://github.com/facebook/rocksdb/pull/8428 in 6.25.0 HISTORY.md section. Updated blog post https://rocksdb.org/blog/2021/05/31/dictionary-compression.html. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9001 Reviewed By: ajkr Differential Revision: D31517836 Pulled By: hx235 fbshipit-source-id: 01f6b30de4e1ff6b315aa8221139d9b700c7c629 09 October 2021, 00:00:30 UTC
418831c anand76 08 October 2021, 20:38:40 UTC Fix wrong tmp dir name in fbcode stress test lego command (#9000) Summary: Change the directory to /dev/shm/rocksdb_fbcode_crash_test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/9000 Reviewed By: akankshamahajan15 Differential Revision: D31511886 Pulled By: anand1976 fbshipit-source-id: 4e48fd2338638fd9223b55d5f7496c26d331bfc7 08 October 2021, 20:39:41 UTC
bcd049c Zhichao Cao 08 October 2021, 17:29:06 UTC Ingest external SST files with Temperature hints (#8949) Summary: Add the file temperature to `IngestExternalFileArg` such that when SST files are ingested, user is able to assign the temperature to each SST file. If the temperature vector is empty or its size does not match the file name vector size, all ingested SST files will be assigned with `Temperature::unKnown`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8949 Test Plan: add the new test and make check Reviewed By: siying Differential Revision: D31127852 Pulled By: zhichao-cao fbshipit-source-id: 141a81f0f7b473d88f4ab0cb2a21a114cbc6f83c 08 October 2021, 17:32:24 UTC
2f1296e anand76 08 October 2021, 17:26:49 UTC Allow more flexible invocation of fbcode_stress_crash (#8985) Summary: Instead of hardcoding the stress test type and some args, allow it to be passed through env variables. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8985 Reviewed By: akankshamahajan15 Differential Revision: D31349495 Pulled By: anand1976 fbshipit-source-id: 585c8fcb0232d0a95925b1a8c4e42a0940227e8b 08 October 2021, 17:28:05 UTC
c0ec58e Andrew Kryczka 07 October 2021, 22:23:14 UTC stop populating unused/invalid MergingIterator heaps (#8975) Summary: I was looking at https://github.com/facebook/rocksdb/issues/2636 and got very confused that `MergingIterator::AddIterator()` is populating `min_heap_` with dangling pointers. There is justification in the comments that `min_heap_` will be cleared before it's used, but it'd be cleaner to not populate it with dangling pointers in the first place. Also made similar change in the constructor for consistency, although the pointers there would not be dangling, just unused. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8975 Test Plan: rely on existing tests Reviewed By: pdillinger, hx235 Differential Revision: D31273767 Pulled By: ajkr fbshipit-source-id: 127ca9dd1f82f77f55dd0c3f19511de3282fc229 07 October 2021, 22:26:08 UTC
fcaa7ff Andrew Kryczka 07 October 2021, 22:22:34 UTC Cancel manual compactions waiting on automatic compactions to drain (#8991) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8991 Test Plan: the new test hangs forever without this fix and passes with this fix. Reviewed By: hx235 Differential Revision: D31456419 Pulled By: ajkr fbshipit-source-id: a82c0e5560b6e6153089dccd8e46163c61b07bff 07 October 2021, 22:23:55 UTC
8717c26 Kajetan Janiak 07 October 2021, 22:22:26 UTC Warning about incompatible options with level_compaction_dynamic_level_bytes (#8329) Summary: This change introduces warnings instead of a silent override when trying to use level_compaction_dynamic_level_bytes with multiple cf_paths/db_paths. I have completed the CLA. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8329 Reviewed By: hx235 Differential Revision: D31399713 Pulled By: ajkr fbshipit-source-id: 29c6fe5258d1f739b4590ecd44aee44f55415595 07 October 2021, 22:23:55 UTC
b632ed0 Zhichao Cao 07 October 2021, 21:57:02 UTC Add file temperature related counter and bytes stats to and io_stats (#8710) Summary: For tiered storage project, we need to know the block read count and read bytes of files with different temperature. Add FileIOByTemperature to IOStatsContext and collect the bytes read and read count from different temperature files through the RandomAccessFileReader. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8710 Test Plan: make check, add the testing cases Reviewed By: siying Differential Revision: D30582400 Pulled By: zhichao-cao fbshipit-source-id: d83173de594374fc8404af5ce93a6a9be72c7141 07 October 2021, 21:58:41 UTC
699f450 Zhichao Cao 07 October 2021, 18:40:20 UTC Introduce a mechanism to dump out blocks from block cache and re-insert to secondary cache (#8912) Summary: Background: Cache warming up will cause potential read performance degradation due to reading blocks from storage to the block cache. Since in production, the workload and access pattern to a certain DB is stable, it is a potential solution to dump out the blocks belonging to a certain DB to persist storage (e.g., to a file) and bulk-load the blocks to Secondary cache before the DB is relaunched. For example, when migrating a DB form host A to host B, it will take a short period of time, the access pattern to blocks in the block cache will not change much. It is efficient to dump out the blocks of certain DB, migrate to the destination host and insert them to the Secondary cache before we relaunch the DB. Design: we introduce the interface of CacheDumpWriter and CacheDumpRead for user to store the blocks dumped out from block cache. RocksDB will encode all the information and send the string to the writer. User can implement their own writer it they want. CacheDumper and CacheLoad are introduced to save the blocks and load the blocks respectively. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8912 Test Plan: add new tests to lru_cache_test and pass make check. Reviewed By: pdillinger Differential Revision: D31452871 Pulled By: zhichao-cao fbshipit-source-id: 11ab4f5d03e383f476947116361d54188d36ec48 07 October 2021, 18:42:31 UTC
fe994bb Ramkumar Vadivelu 07 October 2021, 18:20:44 UTC Misc doc fixes (#8983) Summary: - Update few stale GitHub wiki link references from rocksdb.org - Update the API comments for ignore_range_deletions Pull Request resolved: https://github.com/facebook/rocksdb/pull/8983 Reviewed By: ajkr Differential Revision: D31355965 Pulled By: ramvadiv fbshipit-source-id: 245ac4a6913976dd82afa308bc4aae6bff3d788c 07 October 2021, 18:22:17 UTC
53e595d mrambacher 06 October 2021, 14:47:28 UTC Cleanup multiple implementations of VectorIterator (#8901) Summary: There were three implementations of VectorIterator (util/vector_iterator, test_util/testutil.h and LoggingForwardVectorIterator). Merged them into one class to increase code coverage/testing and reduce duplication. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8901 Reviewed By: pdillinger Differential Revision: D31022673 Pulled By: mrambacher fbshipit-source-id: 8e3acbd2dfd60b4df609d02cc72846de2389d531 06 October 2021, 14:48:31 UTC
d16ceba Yanqin Jin 05 October 2021, 18:20:36 UTC Point bzip2 download address to http://sourceware.org/pub/bzip2 (#8986) Summary: Download bzip2 from `https://sourceware.org/pub/bzip2` to `http://sourceware.org/pub/bzip2` to resolve curl's ca verification error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8986 Reviewed By: akankshamahajan15 Differential Revision: D31387038 Pulled By: riversand963 fbshipit-source-id: 510fdb9530e63639cd5d20339f3f3cbf720068e9 05 October 2021, 18:21:46 UTC
a776406 Stefan Roesch 05 October 2021, 17:50:27 UTC Add file operation callbacks to SequentialFileReader (#8982) Summary: This change adds File IO Notifications to the SequentialFileReader The SequentialFileReader is extended with a listener parameter. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8982 Test Plan: A new test EventListenerTest::OnWALOperationTest has been added. The test verifies that during restore the sequential file reader is called and the notifications are fired. Reviewed By: riversand963 Differential Revision: D31320844 Pulled By: shrfb fbshipit-source-id: 040b24da7c010d7c14ebb5c6460fae9a19b8c168 05 October 2021, 17:51:59 UTC
7872298 mrambacher 04 October 2021, 12:29:21 UTC Fix LITE mode builds on MacOs (#8981) Summary: On MacOS, there were errors building in LITE mode related to unused private member variables: In file included from ./db/compaction/compaction_job.h:20: ./db/blob/blob_file_completion_callback.h:87:19: error: private field ‘sst_file_manager_’ is not used [-Werror,-Wunused-private-field] SstFileManager* sst_file_manager_; ^ ./db/blob/blob_file_completion_callback.h:88:22: error: private field ‘mutex_’ is not used [-Werror,-Wunused-private-field] InstrumentedMutex* mutex_; ^ ./db/blob/blob_file_completion_callback.h:89:17: error: private field ‘error_handler_’ is not used [-Werror,-Wunused-private-field] ErrorHandler* error_handler_; This PR resolves those build issues by removing the values as members in LITE mode and fixing the constructor to ignore the input values in LITE mode (otherwise we get unused parameter warnings). Tested by validating compiles without warnings. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8981 Reviewed By: akankshamahajan15 Differential Revision: D31320141 Pulled By: mrambacher fbshipit-source-id: d67875ebbd39a9555e4f09b2d37159566dd8a085 04 October 2021, 12:30:26 UTC
2cdaf5c Yanqin Jin 02 October 2021, 00:21:39 UTC Add additional checks for three existing unit tests (#8973) Summary: With test sync points, we can assert on the equality of iterator value in three existing unit tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8973 Test Plan: ``` gtest-parallel -r 1000 ./db_test2 --gtest_filter=DBTest2.IterRaceFlush2:DBTest2.IterRaceFlush1:DBTest2.IterRefreshRaceFlush ``` make check Reviewed By: akankshamahajan15 Differential Revision: D31256340 Pulled By: riversand963 fbshipit-source-id: a9440767ab383e0ec61bd43ffa8fbec4ba562ea2 02 October 2021, 00:22:37 UTC
84d71f3 Akanksha Mahajan 01 October 2021, 23:46:48 UTC Enable SingleDelete with user defined ts in db_bench and crash tests (#8971) Summary: Enable SingleDelete with user defined timestamp in db_bench, db_stress and crash test Pull Request resolved: https://github.com/facebook/rocksdb/pull/8971 Test Plan: 1. For db_stress, ran the command for full duration: i) python3 -u tools/db_crashtest.py --enable_ts whitebox --nooverwritepercent=100 ii) make crash_test_with_ts 2. For db_bench, ran: ./db_bench -benchmarks=randomreplacekeys -user_timestamp_size=8 -use_single_deletes=true Reviewed By: riversand963 Differential Revision: D31246558 Pulled By: akankshamahajan15 fbshipit-source-id: 29cd8740c9921341e52f09242fca3c44d75a12b7 01 October 2021, 23:48:01 UTC
e36b9da byronhe 01 October 2021, 23:09:35 UTC Update USERS.md (#8923) Summary: fix typo Pull Request resolved: https://github.com/facebook/rocksdb/pull/8923 Reviewed By: mrambacher Differential Revision: D31003331 Pulled By: zhichao-cao fbshipit-source-id: 00cfcac247621b8bc6d43a3d45c6a11c9dece5b0 01 October 2021, 23:10:35 UTC
7f08a85 sdong 01 October 2021, 21:41:58 UTC Remove IOSTATS_ADD_IF_POSITIVE() (#8984) Summary: IOSTATS_ADD_IF_POSITIVE() doesn't seem to a macro that aims to improve performance but does the opposite. The counter to add is almost always positive so the if is just a waste. Furthermore, adding to a thread local variable seemse to be much cheaper than an if condition if branch prediction has a possibility to be wrong. Remove the macro. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8984 Test Plan: See CI completes. Reviewed By: anand1976 Differential Revision: D31348163 fbshipit-source-id: 30af6d45e1aa8bbc09b2c046206cce6f67f4777a 01 October 2021, 21:43:00 UTC
e5bfb91 Pradeep Ambati 30 September 2021, 22:11:49 UTC List blob files when using command - list_live_files_metadata (#8976) Summary: The ldb list_live_files_metadata command does not print any information about blob files currently. We would like to add this functionality. Note that list_live_files_metadata has two different modes of operation: the one shown above, which shows the LSM tree structure, and another one, which can be enabled using the flag --sort_by_filename and simply lists the files in numerical order regardless of level. We would like to show blob files in both modes. Changes: 1. Using GetAllColumnFamilyMetaData API instead of GetLiveFilesMetaData API for fetching live files data. Testing: 1. Created a sample rocksdb instance using dbbench command (this creates both SST and blob files) 2. Checked if the blob files are listed or not by using ldb commands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8976 Reviewed By: ltamasi Differential Revision: D31316061 Pulled By: pradeepambati fbshipit-source-id: d15cdea192febf7a45f28deee2ba40615d3d84ab 30 September 2021, 22:13:11 UTC
1953b63 Peter (Stig) Edwards 30 September 2021, 21:16:23 UTC ErrorExit if num<1000 for fillsync and fill100K (#8391) Summary: This is to avoid an exception and core dump when running db_bench -benchmarks fillsync -num 999 https://github.com/facebook/rocksdb/issues/8390 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8391 Reviewed By: pdillinger Differential Revision: D29139688 Pulled By: mrambacher fbshipit-source-id: b9e306728ad25a7aac75f6154699aa852bc07bd1 30 September 2021, 21:17:36 UTC
532ff33 anand76 30 September 2021, 20:24:13 UTC Don't ignore deletion rate limit if WAL dir is different (#8967) Summary: If WAL dir is different from the DB dir, we should still honor the SstFileManager deletion rate limit for SST files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8967 Test Plan: Add a new unit test in db_sst_test Reviewed By: pdillinger Differential Revision: D31220116 Pulled By: anand1976 fbshipit-source-id: bcde8a53a7d728e15e597fb5d07ee86c1b38bd28 30 September 2021, 20:26:31 UTC
2acffec Yanqin Jin 29 September 2021, 21:25:43 UTC Add comments for MultiGetBlob() and checks for MultiRead() (#8972) Summary: Add comments for MultiGetBlob() that input argument `offsets` must be sorted. In addition, add assertion for this condition in debug build. Repeat the same for RandomAccessFileReader::MultiRead(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8972 Test Plan: make check Reviewed By: pdillinger Differential Revision: D31253205 Pulled By: riversand963 fbshipit-source-id: 98758229b8052f3aeb319d5584026b4de2d220a2 29 September 2021, 21:27:19 UTC
61a63ae Peter Dillinger 29 September 2021, 18:19:13 UTC Add/improve misc comments (#8963) Summary: Fill in some missing info; fix some incorrect info. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8963 Test Plan: comments only Reviewed By: mrambacher Differential Revision: D31211183 Pulled By: pdillinger fbshipit-source-id: 783ff6673791c01d44c3ed92d4398c64ae5a5005 29 September 2021, 18:20:53 UTC
13ae16c mrambacher 29 September 2021, 11:01:57 UTC Cleanup includes in dbformat.h (#8930) Summary: This header file was including everything and the kitchen sink when it did not need to. This resulted in many places including this header when they needed other pieces instead. Cleaned up this header to only include what was needed and fixed up the remaining code to include what was now missing. Hopefully, this sort of code hygiene cleanup will speed up the builds... Pull Request resolved: https://github.com/facebook/rocksdb/pull/8930 Reviewed By: pdillinger Differential Revision: D31142788 Pulled By: mrambacher fbshipit-source-id: 6b45de3f300750c79f751f6227dece9cfd44085d 29 September 2021, 11:04:40 UTC
559943c Andrew Kryczka 28 September 2021, 21:12:23 UTC Refactor expected state in stress/crash test (#8913) Summary: This is a precursor refactoring to enable an upcoming feature: persistence failure correctness testing. - Changed `--expected_values_path` to `--expected_values_dir` and migrated "db_crashtest.py" to use the new flag. For persistence failure correctness testing there are multiple possible correct states since unsynced data is allowed to be dropped. Making it possible to restore all these possible correct states will eventually involve files containing snapshots of expected values and DB trace files. - The expected values directory is managed by an `ExpectedStateManager` instance. Managing expected state files is separated out of `SharedState` to prevent `SharedState` from becoming too complex when the new files and features (snapshotting, tracing, and restoring) are introduced. - Migrated expected values file access/management out of `SharedState` into a separate class called `ExpectedState`. This is not exposed directly to the test but rather the `ExpectedState` for the latest values file is accessed via a pass-through API on `ExpectedStateManager`. This forces the test to always access the single latest `ExpectedState`. - Changed the initialization of the latest expected values file to use a tempfile followed by rename, and also add cleanup logic for possible stranded tempfiles. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8913 Test Plan: run in several ways; try to make sure it's not obviously broken. - crashtest blackbox without TEST_TMPDIR ``` $ python3 tools/db_crashtest.py blackbox --simple --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --max_key=100000 --value_size_mult=33 --compression_type=none --duration=120 --interval=10 --compression_type=none --blob_compression_type=none ``` - crashtest blackbox with TEST_TMPDIR ``` $ TEST_TMPDIR=/dev/shm python3 tools/db_crashtest.py blackbox --simple --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --max_key=100000 --value_size_mult=33 --compression_type=none --duration=120 --interval=10 --compression_type=none --blob_compression_type=none ``` - crashtest whitebox with TEST_TMPDIR ``` $ TEST_TMPDIR=/dev/shm python3 tools/db_crashtest.py whitebox --simple --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --max_key=100000 --value_size_mult=33 --compression_type=none --duration=120 --interval=10 --compression_type=none --blob_compression_type=none --random_kill_odd=88887 ``` - db_stress without expected_values_dir ``` $ ./db_stress --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --max_key=100000 --value_size_mult=33 --compression_type=none --ops_per_thread=10000 --clear_column_family_one_in=0 --destroy_db_initially=true ``` - db_stress with expected_values_dir and manual corruption ``` $ ./db_stress --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --max_key=100000 --value_size_mult=33 --compression_type=none --ops_per_thread=10000 --clear_column_family_one_in=0 --destroy_db_initially=true --expected_values_dir=./ // modify one byte in "./LATEST.state" $ ./db_stress --write_buffer_size=1048576 --target_file_size_base=1048576 --max_bytes_for_level_base=4194304 --max_key=100000 --value_size_mult=33 --compression_type=none --ops_per_thread=10000 --clear_column_family_one_in=0 --destroy_db_initially=false --expected_values_dir=./ ... Verification failed for column family 0 key 0000000000000000 (0): Value not found: NotFound: ... ``` Reviewed By: riversand963 Differential Revision: D30921951 Pulled By: ajkr fbshipit-source-id: babfe218062e55d018c9b046536c0289fb78f41c 28 September 2021, 21:13:33 UTC
6b34eb0 Jay Zhuang 28 September 2021, 20:59:15 UTC Add remote compaction read/write bytes statistics (#8939) Summary: Add basic read/write bytes statistics on the primary side: `REMOTE_COMPACT_READ_BYTES` `REMOTE_COMPACT_WRITE_BYTES` Fixed existing statistics missing some IO for remote compaction. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8939 Test Plan: CI Reviewed By: ajkr Differential Revision: D31074672 Pulled By: jay-zhuang fbshipit-source-id: c57afdba369990185008ffaec7e3fe7c62e8902f 28 September 2021, 21:00:37 UTC
d6bd1a0 Hui Xiao 28 September 2021, 19:33:03 UTC Support "level_at_creation" in TablePropertiesCollectorFactory::Context (#8919) Summary: Context: Exposing the level of the sst file (i.e, table) where it is created in `TablePropertiesCollectorFactory::Context` allows users of `TablePropertiesCollectorFactory` to customize some implementation details of `TablePropertiesCollectorFactory` and `TablePropertiesCollector` based on the level of creation. For example, `TablePropertiesCollector::NeedCompact()` can return different values based on level of creation. - Declared an extra field `level_at_creation` in `TablePropertiesCollectorFactory::Context` - Allowed `level_at_creation` to be passed in as an argument in `IntTblPropCollectorFactory::CreateIntTblPropCollector()` and `UserKeyTablePropertiesCollectorFactory::CreateIntTblPropCollector()`, the latter of which is an internal wrapper of user's passed-in `TablePropertiesCollectorFactory::CreateTablePropertiesCollector()` used in table-building process - Called `IntTblPropCollectorFactory::CreateIntTblPropCollector()` with `level_at_creation` passed into both `BlockBasedTableBuilder` and `PlainTableBuilder` - `PlainTableBuilder` previously did not capture `level_at_creation` from `TableBuilderOptions` in `PlainTableFactory`. In order for it to call the method with this parameter, this PR also made `PlainTableBuilder` capture `level_at_creation` as a required parameter - Called `IntTblPropCollectorFactory::CreateIntTblPropCollector()` with `level_at_creation` its overridden functions in its derived classes, including `RegularKeysStartWithAFactory::CreateIntTblPropCollector()` in `table_properties_collector_test.cc`, `SstFileWriterPropertiesCollectorFactory::CreateIntTblPropCollector()` in `sst_file_writer_collectors.h` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8919 Test Plan: - Passed the added assertion for `context.level_at_creation` - Passed existing tests - Run `Make` to make sure adding a required parameter to `PlainTableBuilder`'s constructor does not break anything Reviewed By: anand1976 Differential Revision: D30951729 Pulled By: hx235 fbshipit-source-id: c4a0173b0d9344a4cf47e1b987d759c1c73cb474 28 September 2021, 19:35:24 UTC
7fd68b7 mrambacher 28 September 2021, 12:30:32 UTC Make WalFilter, SstPartitionerFactory, FileChecksumGenFactory, and TableProperties Customizable (#8638) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8638 Reviewed By: zhichao-cao Differential Revision: D31024729 Pulled By: mrambacher fbshipit-source-id: 954c04ccab0b8dee64050a27aadf78ed119106c0 28 September 2021, 12:32:02 UTC
b88109d sdong 28 September 2021, 04:28:24 UTC Pollute buffer before calling Read() (#8955) Summary: Add a paranoid check where in case FileSystem layer doesn't fill the buffer but returns succeed, checksum is unlikely to match even if buffer contains a previous block. The byte modified is not useful anyway, so it isn't expect to change any behavior. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8955 Test Plan: See existing CI to pass. Reviewed By: pdillinger Differential Revision: D31183966 fbshipit-source-id: dcc4de429e18131873f783b90d3be55d7eb44a1f 28 September 2021, 04:30:28 UTC
345f4c9 dependabot[bot] 27 September 2021, 22:19:49 UTC Bump nokogiri from 1.11.4 to 1.12.5 in /docs (#8965) Summary: Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.11.4 to 1.12.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/sparklemotion/nokogiri/releases">nokogiri's releases</a>.</em></p> <blockquote> <h2>1.12.5 / 2021-09-27</h2> <h3>Security</h3> <p>[JRuby] Address CVE-2021-41098 (<a href="https://github.com/sparklemotion/nokogiri/security/advisories/GHSA-2rr5-8q37-2w7h">GHSA-2rr5-8q37-2w7h</a>).</p> <p>In Nokogiri v1.12.4 and earlier, on JRuby only, the SAX parsers resolve external entities (XXE) by default. This fix turns off entity-resolution-by-default in the JRuby SAX parsers to match the CRuby SAX parsers' behavior.</p> <p>CRuby users are not affected by this CVE.</p> <h3>Fixed</h3> <ul> <li>[CRuby] <code>Document#to_xhtml</code> properly serializes self-closing tags in libxml &gt; 2.9.10. A behavior change introduced in libxml 2.9.11 resulted in emitting start and and tags (e.g., <code>&lt;br&gt;&lt;/br&gt;</code>) instead of a self-closing tag (e.g., <code>&lt;br/&gt;</code>) in previous Nokogiri versions. [<a href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2324">https://github.com/facebook/rocksdb/issues/2324</a>]</li> </ul> <hr /> <p>SHA256 checksums:</p> <pre><code>36bfa3a07aced069b3f3c9b39d9fb62cb0728d284d02b079404cd55780beaeff nokogiri-1.12.5-arm64-darwin.gem 16b1a9ddbb70a9c998462912a5972097cbc79c3e01eb373906886ef8a469f589 nokogiri-1.12.5-java.gem 218dcc6edd1b49cc6244b5f88afb978739bb2f3f166c271557fe5f51e4bc713c nokogiri-1.12.5-x64-mingw32.gem e33bb919d64c16d931a5f26dc880969e587d225cfa97e6b56e790fb52179f527 nokogiri-1.12.5-x86-linux.gem e13c2ed011b8346fbd589e96fe3542d763158bc2c7ad0f4f55f6d801afd1d9ff nokogiri-1.12.5-x86-mingw32.gem 1ed64f7db7c1414b87fce28029f2a10128611d2037e0871ba298d00f9a00edd6 nokogiri-1.12.5-x86_64-darwin.gem 0868c8d0a147904d4dedaaa05af5f06656f2d3c67e4432601718559bf69d6cea nokogiri-1.12.5-x86_64-linux.gem 2b20905942acc580697c8c496d0d1672ab617facb9d30d156b3c7676e67902ec nokogiri-1.12.5.gem </code></pre> <h2>1.12.4 / 2021-08-29</h2> <h3>Notable fix: Namespace inheritance</h3> <p>Namespace behavior when reparenting nodes has historically been poorly specified and the behavior diverged between CRuby and JRuby. As a result, making this behavior consistent in v1.12.0 introduced a breaking change.</p> <p>This patch release reverts the Builder behavior present in v1.12.0..v1.12.3 but keeps the Document behavior. This release also introduces a Document attribute to allow affected users to easily change this behavior for their legacy code without invasive changes.</p> <h4>Compensating Feature in XML::Document</h4> <p>This release of Nokogiri introduces a new <code>Document</code> boolean attribute, <code>namespace_inheritance</code>, which controls whether children should inherit a namespace when they are reparented. <code>Nokogiri::XML:Document</code> defaults this attribute to <code>false</code> meaning &quot;do not inherit,&quot; thereby making explicit the behavior change introduced in v1.12.0.</p> <p>CRuby users who desire the pre-v1.12.0 behavior may set <code>document.namespace_inheritance = true</code> before reparenting nodes.</p> <p>See <a href="https://nokogiri.org/rdoc/Nokogiri/XML/Document.html#namespace_inheritance-instance_method">https://nokogiri.org/rdoc/Nokogiri/XML/Document.html#namespace_inheritance-instance_method</a> for example usage.</p> <h4>Fix for XML::Builder</h4> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md">nokogiri's changelog</a>.</em></p> <blockquote> <h2>1.12.5 / 2021-09-27</h2> <h3>Security</h3> <p>[JRuby] Address CVE-2021-41098 (<a href="https://github.com/sparklemotion/nokogiri/security/advisories/GHSA-2rr5-8q37-2w7h">GHSA-2rr5-8q37-2w7h</a>).</p> <p>In Nokogiri v1.12.4 and earlier, on JRuby only, the SAX parsers resolve external entities (XXE) by default. This fix turns off entity-resolution-by-default in the JRuby SAX parsers to match the CRuby SAX parsers' behavior.</p> <p>CRuby users are not affected by this CVE.</p> <h3>Fixed</h3> <ul> <li>[CRuby] <code>Document#to_xhtml</code> properly serializes self-closing tags in libxml &gt; 2.9.10. A behavior change introduced in libxml 2.9.11 resulted in emitting start and and tags (e.g., <code>&lt;br&gt;&lt;/br&gt;</code>) instead of a self-closing tag (e.g., <code>&lt;br/&gt;</code>) in previous Nokogiri versions. [<a href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2324">https://github.com/facebook/rocksdb/issues/2324</a>]</li> </ul> <h2>1.12.4 / 2021-08-29</h2> <h3>Notable fix: Namespace inheritance</h3> <p>Namespace behavior when reparenting nodes has historically been poorly specified and the behavior diverged between CRuby and JRuby. As a result, making this behavior consistent in v1.12.0 introduced a breaking change.</p> <p>This patch release reverts the Builder behavior present in v1.12.0..v1.12.3 but keeps the Document behavior. This release also introduces a Document attribute to allow affected users to easily change this behavior for their legacy code without invasive changes.</p> <h4>Compensating Feature in XML::Document</h4> <p>This release of Nokogiri introduces a new <code>Document</code> boolean attribute, <code>namespace_inheritance</code>, which controls whether children should inherit a namespace when they are reparented. <code>Nokogiri::XML:Document</code> defaults this attribute to <code>false</code> meaning &quot;do not inherit,&quot; thereby making explicit the behavior change introduced in v1.12.0.</p> <p>CRuby users who desire the pre-v1.12.0 behavior may set <code>document.namespace_inheritance = true</code> before reparenting nodes.</p> <p>See <a href="https://nokogiri.org/rdoc/Nokogiri/XML/Document.html#namespace_inheritance-instance_method">https://nokogiri.org/rdoc/Nokogiri/XML/Document.html#namespace_inheritance-instance_method</a> for example usage.</p> <h4>Fix for XML::Builder</h4> <p>However, recognizing that we want <code>Builder</code>-created children to inherit namespaces, Builder now will set <code>namespace_inheritance=true</code> on the underlying document for both JRuby and CRuby. This means that, on CRuby, the pre-v1.12.0 behavior is restored.</p> <p>Users who want to turn this behavior off may pass a keyword argument to the Builder constructor like so:</p> <pre lang="ruby"><code>Nokogiri::XML::Builder.new(namespace_inheritance: false) </code></pre> <p>See <a href="https://nokogiri.org/rdoc/Nokogiri/XML/Builder.html#label-Namespace+inheritance">https://nokogiri.org/rdoc/Nokogiri/XML/Builder.html#label-Namespace+inheritance</a> for example usage.</p> <h4>Downstream gem maintainers</h4> <p>Note that any downstream gems may want to specifically omit Nokogiri v1.12.0--v1.12.3 from their dependency specification if they rely on child namespace inheritance:</p> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/sparklemotion/nokogiri/commit/47f6a461fdc3e375b30522259e48569fb578dece"><code>47f6a46</code></a> version bump to v1.12.5</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/2a0ac88518fdd1509d14c4cbdb9784c73dd8a839"><code>2a0ac88</code></a> update CHANGELOG</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/6b6063782cefc42e527dc967c6119125cae0042d"><code>6b60637</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2329">https://github.com/facebook/rocksdb/issues/2329</a> from sparklemotion/flavorjones-GHSA-2rr5-8q37-2w7h_1...</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/4bd943cae3039c51c3f54de9cd76abbfb647666b"><code>4bd943c</code></a> fix(jruby): SAX parser uses an entity resolver</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/f943ee4108b007d225e00c3ac7da00df17b81b1a"><code>f943ee4</code></a> refactor(jruby): handle errors more consistently</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/27901227488ea7e439777cfc907e52c68622e6a3"><code>2790122</code></a> format: test files</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/01e1618f7551ae3c32c1a5790c1004c18a46b316"><code>01e1618</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2327">https://github.com/facebook/rocksdb/issues/2327</a> from sparklemotion/2324-xhtml-self-closing-tags_v1.12.x</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/a0180c72c55c44b8e0db3a98040bd5f115742817"><code>a0180c7</code></a> fix: HTML4::Document.to_xhtml self-closing tags</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/564ac1787303332e0b2b92311ff6f1b30a893eae"><code>564ac17</code></a> release v1.12.4</li> <li><a href="https://github.com/sparklemotion/nokogiri/commit/4d5754baede4fc98cd4f12754f479bd228b6b55b"><code>4d5754b</code></a> backport <a href="https://github-redirect.dependabot.com/sparklemotion/nokogiri/issues/2320">https://github.com/facebook/rocksdb/issues/2320</a></li> <li>Additional commits viewable in <a href="https://github.com/sparklemotion/nokogiri/compare/v1.11.4...v1.12.5">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=nokogiri&package-manager=bundler&previous-version=1.11.4&new-version=1.12.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebook/rocksdb/network/alerts). </details> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8965 Reviewed By: akankshamahajan15 Differential Revision: D31217632 Pulled By: ltamasi fbshipit-source-id: c98c5a42f29eb45164a266edd91569737595ab2a 27 September 2021, 22:21:05 UTC
a2f29ce Akanksha Mahajan 27 September 2021, 21:57:08 UTC Update History.md for SingleDelete with user defined timestamp (#8964) Summary: Update History.md for SingleDelete with user defined timestamp Pull Request resolved: https://github.com/facebook/rocksdb/pull/8964 Reviewed By: zhichao-cao Differential Revision: D31216214 Pulled By: akankshamahajan15 fbshipit-source-id: 0520132c75fe8f6823d154e41585b0df3086c04d 27 September 2021, 21:58:30 UTC
78afb4d Akanksha Mahajan 27 September 2021, 18:49:35 UTC Support SingleDelete for user-defined timestamps (#8921) Summary: Added support for SingleDelete for user-defined timestamps. Users can now Get and Iterate over keys deleted with SingleDelete. It also includes changes in CompactionIterator which preserves the same user key with different timestamps, unless the timestamp is below a certain threshold full_history_ts_low. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8921 Test Plan: Added new unit tests Reviewed By: riversand963 Differential Revision: D31098191 Pulled By: akankshamahajan15 fbshipit-source-id: 78a59ef4b4884ae324fcd10f56e62a27d5ee2f49 27 September 2021, 18:51:07 UTC
0774d64 Peter Dillinger 27 September 2021, 18:42:10 UTC Fix some lint warnings reported on 6.25 (#8945) Summary: Fix some lint warnings Pull Request resolved: https://github.com/facebook/rocksdb/pull/8945 Test Plan: existing tests, linters Reviewed By: zhichao-cao Differential Revision: D31103824 Pulled By: pdillinger fbshipit-source-id: 4dd9b0c30fa50e588107ac6ed392b2dfb507a5d4 27 September 2021, 18:43:20 UTC
b59b757 ricky 27 September 2021, 17:37:09 UTC More clear error message on uncompressing block (#8934) Summary: The origin error message of uncompressing block is confusing, which may result from either build support or data corruption. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8934 Reviewed By: ltamasi Differential Revision: D31112588 Pulled By: pdillinger fbshipit-source-id: 1cbf2d4fbcb0ef376cf942246d06f48cb603f852 27 September 2021, 17:38:16 UTC
e0f697d mrambacher 27 September 2021, 14:42:36 UTC Make SliceTransform into a Customizable class (#8641) Summary: Made SliceTransform into a Customizable class. Would be nice to write a test that stored and used a custom transform in an SST table. There are a set of tests (DBBlockFliterTest.PrefixExtractor*, SamePrefixTest.InDomainTest, PrefixTest.PrefixAndWholeKeyTest that run the same with or without a SliceTransform/PrefixFilter. Is this expected? Pull Request resolved: https://github.com/facebook/rocksdb/pull/8641 Reviewed By: zhichao-cao Differential Revision: D31142793 Pulled By: mrambacher fbshipit-source-id: bb08672fccbfdc263dcae21f25a62307e1facda1 27 September 2021, 14:43:47 UTC
b92cef2 Yanqin Jin 25 September 2021, 05:13:19 UTC Sort per-file blob read requests by offset (#8953) Summary: `RandomAccessFileReader::MultiRead()` tries to merge requests in direct IO, assuming input IO requests are sorted by offsets. Add a test in direct IO mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8953 Test Plan: make check Reviewed By: ltamasi Differential Revision: D31183546 Pulled By: riversand963 fbshipit-source-id: 5d043ec68e2daa47a3149066150afd41ee3d73e6 25 September 2021, 05:14:30 UTC
6d424be Andrew Kryczka 25 September 2021, 01:27:44 UTC Temporarily set experimental_mempurge_threshold=0 in crash test (#8958) Summary: For now, disable it since the below command indicates it can cause a failure. Running that command with `-experimental_mempurge_threshold=0` has been running successfully for several minutes, whereas before it failed in seconds. ``` $ while rm -rf /dev/shm/single_stress && ./db_stress --clear_column_family_one_in=0 --column_families=1 --db=/dev/shm/single_stress --experimental_mempurge_threshold=5.493146827397074 --flush_one_in=10000 --reopen=0 --write_buffer_size=262144 --value_size_mult=33 --max_write_buffer_number=3 -ops_per_thread=10000; do : ; done ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8958 Reviewed By: ltamasi Differential Revision: D31187059 Pulled By: ajkr fbshipit-source-id: 04d5bfb4fcc4f5b66233e691427dfd940c67037f 25 September 2021, 01:29:48 UTC
791bff5 Andrew Kryczka 24 September 2021, 23:52:30 UTC Prevent deadlock in db_stress with DbStressCompactionFilter (#8956) Summary: The cyclic dependency was: - `StressTest::OperateDb()` locks the mutex for key 'k' - `StressTest::OperateDb()` calls a function like `PauseBackgroundWork()`, which waits for pending compaction to complete. - The pending compaction reaches key `k` and `DbStressCompactionFilter::FilterV2()` calls `Lock()` on that key's mutex, which hangs forever. The cycle can be broken by using a new function, `port::Mutex::TryLock()`, which returns immediately upon failure to acquire a lock. In that case `DbStressCompactionFilter::FilterV2()` can just decide to keep the key. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8956 Reviewed By: riversand963 Differential Revision: D31183718 Pulled By: ajkr fbshipit-source-id: 329e4a31ce43085af174cf367ef560b5a04399c5 24 September 2021, 23:54:02 UTC
0595101 Jay Zhuang 24 September 2021, 17:22:22 UTC Add microbench build support for fbcode (#8954) Summary: For internal build enviroment only. Developer could run the microbenchmark without `ROCKSDB_NO_FBCODE=1`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8954 Test Plan: `$ make microbench` on dev server Reviewed By: ajkr Differential Revision: D31163717 Pulled By: jay-zhuang fbshipit-source-id: 1ff59f660ca05afd0fd5c7c7dcdfd831ac365462 24 September 2021, 17:23:35 UTC
7c6a7e8 sdong 23 September 2021, 18:59:33 UTC FaultInjectionTestFS::InjectThreadSpecificReadError() should not corrupt mmaped bytes (#8952) Summary: Right now FaultInjectionTestFS::InjectThreadSpecificReadError() might try to corrupt return bytes, but these bytes might be from mmapped files, which would cause segfault. Instead FaultInjectionTestFS::InjectThreadSpecificReadError() should never corrupt data unless it is in caller's buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8952 Test Plan: See db_stress still runs and make sure in a test run failurs are still injected in non-mmap cases. Reviewed By: ajkr, ltamasi Differential Revision: D31147318 fbshipit-source-id: 9484a64ff2aaa36685557203f449286e694e65f9 23 September 2021, 19:00:47 UTC
b25f2af Hui Xiao 23 September 2021, 02:35:05 UTC Return Status::NotSupported() in RateLimiter::GetTotalPendingRequests default impl (#8950) Summary: Context: After more discussion, a fix in https://github.com/facebook/rocksdb/issues/8938 might turn out to be too restrictive for the case where `GetTotalPendingRequests` might be invoked on RateLimiter classes that does not support the recently added API `RateLimiter::GetTotalPendingRequests` (https://github.com/facebook/rocksdb/issues/8890) due to the `assert(false)` in https://github.com/facebook/rocksdb/issues/8938. Furthermore, sentinel value like `-1` proposed in https://github.com/facebook/rocksdb/issues/8938 is easy to be ignored and unchecked. Therefore we decided to adopt `Status::NotSupported()`, which is also a convention of adding new API to public header in RocksDB. - Changed return value type of `RateLimiter::GetTotalPendingRequests` in related declaration/definition - Passed in pointer argument to hold the output instead of returning it as before - Adapted to the changes above in calling `RateLimiter::GetTotalPendingRequests` in test - Minor improvement to `TEST_F(RateLimiterTest, GetTotalPendingRequests)`: added failure message for assertion and replaced repetitive statements with a loop Pull Request resolved: https://github.com/facebook/rocksdb/pull/8950 Reviewed By: ajkr, pdillinger Differential Revision: D31128450 Pulled By: hx235 fbshipit-source-id: 282ac9c4f3dacaa0aec6d0a993161f77ad47a040 23 September 2021, 02:36:06 UTC
be206db Levi Tamasi 22 September 2021, 23:39:13 UTC Deflake MySQLStyleTransactionTest.TransactionStressTest in "status checked" mode (#8947) Summary: There is a corner case when using WriteUnprepared transactions when `WriteUnpreparedTxn::Get` returns `Status::TryAgain` instead of propagating the result of `GetFromBatchAndDB`. The patch adds `PermitUncheckedError` to make the `ASSERT_STATUS_CHECKED` build pass in this case as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8947 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D31125422 Pulled By: ltamasi fbshipit-source-id: 42de51dcfa9384e032244c2b4d3f40e9a4111194 22 September 2021, 23:40:25 UTC
c988e47 sdong 22 September 2021, 23:19:57 UTC Add HISTORY.md entry to a recent bug fix. (#8948) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8948 Reviewed By: anand1976 Differential Revision: D31127368 fbshipit-source-id: a374cb0baf88c3e15cd587a8f31e8a2d84432928 22 September 2021, 23:23:08 UTC
58444ea Hui Xiao 22 September 2021, 04:27:26 UTC Make RateLimiter::GetTotalPendingRequest() non pure virtual for backward compability (#8938) Summary: Context/Summary: https://github.com/facebook/rocksdb/pull/8890 added a public API `RateLimiter::GetTotalPendingRequest()` but mistakenly marked it as pure virtual, forcing RateLimiter's derived classes to implement this function and breaking backward compatibility. This PR makes `RateLimiter::GetTotalPendingRequest()` as non-pure virtual method by providing a trivial implementation in rate_limiter.h Pull Request resolved: https://github.com/facebook/rocksdb/pull/8938 Test Plan: Passing existing tests Reviewed By: pdillinger Differential Revision: D31100661 Pulled By: hx235 fbshipit-source-id: 06eff1005156a6e5a881e393b2c5b2ad706897d8 22 September 2021, 04:29:26 UTC
9320067 sdong 21 September 2021, 21:47:09 UTC Improve fault injection to MultiRead (#8937) Summary: Several improvements to MultiRead: 1. Fix a bug in stress test which causes false positive when both MultiRead() return and individual read request have failure injected. 2. Add two more types of fault that should be handled: empty read results and checksum mismatch 3. Add a message indicating which type of fault is injected 4. Increase the failure rate Pull Request resolved: https://github.com/facebook/rocksdb/pull/8937 Reviewed By: anand1976 Differential Revision: D31085930 fbshipit-source-id: 3a04994a3cadebf9a64d25e1fe12b14b7a272fba 21 September 2021, 21:48:15 UTC
fcce1f2 sdong 21 September 2021, 19:21:03 UTC RandomAccessFileReader::MultiRead() should not return read bytes not read (#8941) Summary: Right now, if underlying read returns fewer bytes than asked for, RandomAccessFileReader::MultiRead() still returns those in the buffer to upper layer. This can be a surprise to upper layer. This is unlikely to cause incorrect data. To cause incorrect data, checksum checking in upper layer should pass with short reads, whose chance is low. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8941 Test Plan: Run stress tests for a while Reviewed By: anand1976 Differential Revision: D31085780 fbshipit-source-id: 999adf2d6c2712f1323d14bb68b678df59969973 21 September 2021, 19:22:22 UTC
1de5886 sdong 21 September 2021, 18:25:41 UTC Fix flaky ldb_cmd_test tests caused by file deletions during validation (#8942) Summary: In FileChecksumTestHelper::VerifyEachFileChecksum(), we query the file list, and then for each file in the list verify the checksum. However, compaction can delete those files in the mean time and cause failures. To prevent it from happening, disable file deletion during the validation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8942 Test Plan: Run exsiting test and see it doesn't fail. Reviewed By: pdillinger Differential Revision: D31086488 fbshipit-source-id: 554608f36d2dd3bf0a20dfc4039c68bd8533d7f8 21 September 2021, 18:27:38 UTC
5268cdc Peter Dillinger 21 September 2021, 18:12:19 UTC Finish BackupEngine migration to IOStatus (#8940) Summary: Updates a few remaining functions that should have been updated from Status -> IOStatus, and adds to HISTORY for the overall change including https://github.com/facebook/rocksdb/issues/8820. This change is for inclusion in version 6.25. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8940 Test Plan: CI Reviewed By: zhichao-cao Differential Revision: D31085029 Pulled By: pdillinger fbshipit-source-id: 91557c6a39ef1d90357d4f4dcd79af0645d87c7b 21 September 2021, 18:13:17 UTC
6924869 mrambacher 21 September 2021, 15:53:03 UTC Make SystemClock into a Customizable Class (#8636) Summary: Made SystemClock into a Customizable class, complete with CreateFromString. Cleaned up some of the existing SystemClock implementations that were redundant (NoSleep was the same as the internal one for MockEnv). Changed MockEnv construction to allow Clock to be passed to the Memory/MockFileSystem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8636 Reviewed By: zhichao-cao Differential Revision: D30483360 Pulled By: mrambacher fbshipit-source-id: cd0e3a876c39f8c98fe13374c06e8edbd5b9f2a1 21 September 2021, 16:23:48 UTC
d497cdf Peter Dillinger 20 September 2021, 17:38:03 UTC Update version to 6.25.0 (#8935) Summary: for release Pull Request resolved: https://github.com/facebook/rocksdb/pull/8935 Test Plan: CI Reviewed By: ajkr Differential Revision: D31056726 Pulled By: pdillinger fbshipit-source-id: 6fd022c39c19c35f10a2367df45dd2deb43df510 20 September 2021, 18:22:41 UTC
99fe4c5 anand76 18 September 2021, 16:31:57 UTC Add a gflag for IO uring enable/disable (#8931) Summary: In case of IO uring bugs, we need to provide a way for users to turn it off. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8931 Test Plan: Manually run db_bench with/without the option and verify the behavior Reviewed By: pdillinger Differential Revision: D31040252 Pulled By: anand1976 fbshipit-source-id: 56f2537d6ac8488c9e126296d8190ad9e0158f70 18 September 2021, 17:24:56 UTC
back to top