Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
c3034fc Lucian Grijincu 09 August 2021, 23:04:26 UTC rocksdb: don't call LZ4_loadDictHC with null dictionary Summary: UBSAN revealed a pointer underflow when `LZ4HC_init_internal` is called with a null `start`. Reviewed By: ajkr Differential Revision: D30181874 fbshipit-source-id: ca9bbac1a85c58782871d7f153af733b000cc66c 09 August 2021, 23:13:49 UTC
2dea3dd Andrew Kryczka 09 August 2021, 18:34:21 UTC update HISTORY.md release date 09 August 2021, 18:34:36 UTC
37c4996 Levi Tamasi 06 August 2021, 23:26:04 UTC Fix the sorting of KeyContexts for batched MultiGet (#8633) Summary: `CompareKeyContext::operator()` on the trunk has a bug: when comparing column family IDs, `lhs` is used for both sides of the comparison. This results in the `KeyContext`s getting sorted solely based on key, which in turn means that keys with the same column family do not necessarily form a single range in the sorted list. This violates an assumption of the batched `MultiGet` logic, leading to the same column family showing up multiple times in the list of `MultiGetColumnFamilyData`. The end result is the code attempting to check out the thread-local `SuperVersion` for the same CF multiple times, causing an assertion violation in debug builds and memory corruption/crash in release builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8633 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D30169182 Pulled By: ltamasi fbshipit-source-id: a47710652df7e95b14b40fb710924c11a8478023 09 August 2021, 17:59:11 UTC
53da604 Andrew Kryczka 05 August 2021, 00:27:29 UTC update HISTORY.md and version.h for 6.23.3 05 August 2021, 00:27:29 UTC
60f5a22 Andrew Kryczka 05 August 2021, 00:24:06 UTC Do not attempt to rename non-existent info log (#8622) Summary: Previously we attempted to rename "LOG" to "LOG.old.*" without checking its existence first. "LOG" had no reason to exist in a new DB. Errors in renaming a non-existent "LOG" were swallowed via `PermitUncheckedError()` so things worked. However the storage service's error monitoring was detecting all these benign rename failures. So it is better to fix it. Also with this PR we can now distinguish rename failure for other reasons and return them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8622 Test Plan: new unit test Reviewed By: akankshamahajan15 Differential Revision: D30115189 Pulled By: ajkr fbshipit-source-id: e2f337ffb2bd171be0203172abc8e16e7809b170 05 August 2021, 00:26:19 UTC
2898067 Levi Tamasi 04 August 2021, 20:20:04 UTC Mention PR 8585 in HISTORY.md 04 August 2021, 20:47:17 UTC
238a9c3 Levi Tamasi 04 August 2021, 20:07:49 UTC Bump version number to 6.23.2 04 August 2021, 20:07:49 UTC
2511b42 Yanqin Jin 03 August 2021, 20:30:05 UTC Fix NotifyOnFlushCompleted() for atomic flush (#8585) Summary: PR https://github.com/facebook/rocksdb/issues/5908 added `flush_jobs_info_` to `FlushJob` to make sure `OnFlushCompleted()` is called after committing flush results to MANIFEST. However, `flush_jobs_info_` is not updated in atomic flush, causing `NotifyOnFlushCompleted()` to skip `OnFlushCompleted()`. This PR fixes this, in a similar way to https://github.com/facebook/rocksdb/issues/5908 that handles regular flush. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8585 Test Plan: make check Reviewed By: jay-zhuang Differential Revision: D29913720 Pulled By: riversand963 fbshipit-source-id: 4ff023c98372fa2c93188d4a5c8a4e9ffa0f4dda 04 August 2021, 18:24:41 UTC
1920121 Levi Tamasi 03 August 2021, 23:13:41 UTC Mention PR 8605 in HISTORY.md (#8619) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8619 Reviewed By: riversand963 Differential Revision: D30081937 Pulled By: ltamasi fbshipit-source-id: 57505957ae2c22d4b194aa28cb3fd261b3b39919 04 August 2021, 18:20:31 UTC
7c393c3 Levi Tamasi 03 August 2021, 01:10:57 UTC Fix a race in ColumnFamilyData::UnrefAndTryDelete (#8605) Summary: The `ColumnFamilyData::UnrefAndTryDelete` code currently on the trunk unlocks the DB mutex before destroying the `ThreadLocalPtr` holding the per-thread `SuperVersion` pointers when the only remaining reference is the back reference from `super_version_`. The idea behind this was to break the circular dependency between `ColumnFamilyData` and `SuperVersion`: when the penultimate reference goes away, `ColumnFamilyData` can clean up the `SuperVersion`, which can in turn clean up `ColumnFamilyData`. (Assuming there is a `SuperVersion` and it is not referenced by anything else.) However, unlocking the mutex throws a wrench in this plan by making it possible for another thread to jump in and take another reference to the `ColumnFamilyData`, keeping the object alive in a zombie `ThreadLocalPtr`-less state. This can cause issues like https://github.com/facebook/rocksdb/issues/8440 , https://github.com/facebook/rocksdb/issues/8382 , and might also explain the `was_last_ref` assertion failures from the `ColumnFamilySet` destructor we sometimes observe during close in our stress tests. Digging through the archives, this unlocking goes way back to 2014 (or earlier). The original rationale was that `SuperVersionUnrefHandle` used to lock the mutex so it can call `SuperVersion::Cleanup`; however, this logic turned out to be deadlock-prone. https://github.com/facebook/rocksdb/pull/3510 fixed the deadlock but left the unlocking in place. https://github.com/facebook/rocksdb/pull/6147 then introduced the circular dependency and associated cleanup logic described above (in order to enable iterators to keep the `ColumnFamilyData` for dropped column families alive), and moved the unlocking-relocking snippet to its present location in `UnrefAndTryDelete`. Finally, https://github.com/facebook/rocksdb/pull/7749 fixed a memory leak but apparently exacerbated the race by (otherwise correctly) switching to `UnrefAndTryDelete` in `SuperVersion::Cleanup`. The patch simply eliminates the unlocking and relocking, which has been unnecessary ever since https://github.com/facebook/rocksdb/issues/3510 made `SuperVersionUnrefHandle` lock-free. This closes the window during which another thread could increase the reference count, and hopefully fixes the issues above. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8605 Test Plan: Ran `make check` and stress tests locally. Reviewed By: pdillinger Differential Revision: D30051035 Pulled By: ltamasi fbshipit-source-id: 8fe559e4b4ad69fc142579f8bc393ef525918528 04 August 2021, 18:00:18 UTC
7513e90 Jay Zhuang 22 July 2021, 20:46:33 UTC Update HISTORY.md and version.h for 6.23.1 22 July 2021, 20:46:33 UTC
bf117c5 Jay Zhuang 22 July 2021, 20:41:48 UTC Fix an race condition during multiple DB opening (#8574) Summary: ObjectLibrary is shared between multiple DB instances, the Register() could have race condition. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8574 Test Plan: pass the failed test Reviewed By: ajkr Differential Revision: D29855096 Pulled By: jay-zhuang fbshipit-source-id: 541eed0bd495d2c963d858d81e7eabf1ba16153c 22 July 2021, 20:43:40 UTC
c04a86a Jay Zhuang 17 July 2021, 00:51:28 UTC Update HISTORY.md and version.h 6.23 release (#8552) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8552 Reviewed By: ajkr Differential Revision: D29746828 Pulled By: jay-zhuang fbshipit-source-id: 17d564895ae9cb675d455e73626b9a6717db6279 17 July 2021, 00:52:14 UTC
3455ab0 Merlin Mao 17 July 2021, 00:03:56 UTC Remove extra double quote in options.h (#8550) Summary: There is an extra " in options.h (`"index block""`) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8550 Test Plan: None Reviewed By: jay-zhuang Differential Revision: D29746077 Pulled By: autopear fbshipit-source-id: 2e5117296e5414b7c7440d990926bc1e567a0b4f 17 July 2021, 00:05:25 UTC
39a07c9 sdong 16 July 2021, 23:08:14 UTC DB Stress Reopen write failure to skip WAL (#8548) Summary: When DB Stress enables write failure in reopen, WAL files are also created with a wrapper writalbe file which buffers write until fsync. However, crash test currently expects all writes to WAL is persistent. This is at odd with the unsynced bytes dropped. To work it around temporarily, we disable WAL write failure for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8548 Test Plan: Run db_stress. Manual printf to make sure only WAL files are skipped. Reviewed By: jay-zhuang Differential Revision: D29745095 fbshipit-source-id: 1879dd2c01abad7879ca243ee94570ec47c347f3 16 July 2021, 23:09:33 UTC
a379dae Jay Zhuang 16 July 2021, 22:36:00 UTC Minor Makefile update to exclude microbench as dependency (#8523) Summary: Otherwise the build may report warning about missing `benchmark.h` for some targets, the error won't break the build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8523 Test Plan: `make blackbox_ubsan_crash_test` on a machine without benchmark lib installed. Reviewed By: pdillinger Differential Revision: D29682478 Pulled By: jay-zhuang fbshipit-source-id: e1261fbcda46bc6bd3cd39b7b03b7f78927d0430 16 July 2021, 22:36:51 UTC
ac37bfd mrambacher 16 July 2021, 22:04:29 UTC Allow CreateFromString to work on complex URIs (#8547) Summary: Some URIs for creating instances (ala SecondaryCache) use complex URIs like (cache://name;prop=value). These URIs were treated as name-value properties. With this change, if the URI does not contain an "id=XX" setting, it will be treated as a single string value (and not an ID and map of name-value properties). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8547 Reviewed By: anand1976 Differential Revision: D29741386 Pulled By: mrambacher fbshipit-source-id: 0621f62bec3a6699a7b66c7c0b5634b2856653aa 16 July 2021, 22:05:45 UTC
df5dc73 Peter Dillinger 16 July 2021, 21:12:06 UTC Don't hold DB mutex for block cache entry stat scans (#8538) Summary: I previously didn't notice the DB mutex was being held during block cache entry stat scans, probably because I primarily checked for read performance regressions, because they require the block cache and are traditionally latency-sensitive. This change does some refactoring to avoid holding DB mutex and to avoid triggering and waiting for a scan in GetProperty("rocksdb.cfstats"). Some tests have to be updated because now the stats collector is populated in the Cache aggressively on DB startup rather than lazily. (I hope to clean up some of this added complexity in the future.) This change also ensures proper treatment of need_out_of_mutex for non-int DB properties. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8538 Test Plan: Added unit test logic that uses sync points to fail if the DB mutex is held during a scan, covering the various ways that a scan might be triggered. Performance test - the known impact to holding the DB mutex is on TransactionDB, and the easiest way to see the impact is to hack the scan code to almost always miss and take an artificially long time scanning. Here I've injected an unconditional 5s sleep at the call to ApplyToAllEntries. Before (hacked): $ TEST_TMPDIR=/dev/shm ./db_bench.base_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 433.219 micros/op 2308 ops/sec; 0.1 MB/s ( transactions:78999 aborts:0) rocksdb.db.write.micros P50 : 16.135883 P95 : 36.622503 P99 : 66.036115 P100 : 5000614.000000 COUNT : 149677 SUM : 8364856 $ TEST_TMPDIR=/dev/shm ./db_bench.base_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 448.802 micros/op 2228 ops/sec; 0.1 MB/s ( transactions:75999 aborts:0) rocksdb.db.write.micros P50 : 16.629221 P95 : 37.320607 P99 : 72.144341 P100 : 5000871.000000 COUNT : 143995 SUM : 13472323 Notice the 5s P100 write time. After (hacked): $ TEST_TMPDIR=/dev/shm ./db_bench.new_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 303.645 micros/op 3293 ops/sec; 0.1 MB/s ( transactions:98999 aborts:0) rocksdb.db.write.micros P50 : 16.061871 P95 : 33.978834 P99 : 60.018017 P100 : 616315.000000 COUNT : 187619 SUM : 4097407 $ TEST_TMPDIR=/dev/shm ./db_bench.new_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 310.383 micros/op 3221 ops/sec; 0.1 MB/s ( transactions:96999 aborts:0) rocksdb.db.write.micros P50 : 16.270026 P95 : 35.786844 P99 : 64.302878 P100 : 603088.000000 COUNT : 183819 SUM : 4095918 P100 write is now ~0.6s. Not good, but it's the same even if I completely bypass all the scanning code: $ TEST_TMPDIR=/dev/shm ./db_bench.new_skip -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 311.365 micros/op 3211 ops/sec; 0.1 MB/s ( transactions:96999 aborts:0) rocksdb.db.write.micros P50 : 16.274362 P95 : 36.221184 P99 : 68.809783 P100 : 649808.000000 COUNT : 183819 SUM : 4156767 $ TEST_TMPDIR=/dev/shm ./db_bench.new_skip -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 308.395 micros/op 3242 ops/sec; 0.1 MB/s ( transactions:97999 aborts:0) rocksdb.db.write.micros P50 : 16.106222 P95 : 37.202403 P99 : 67.081875 P100 : 598091.000000 COUNT : 185714 SUM : 4098832 No substantial difference. Reviewed By: siying Differential Revision: D29738847 Pulled By: pdillinger fbshipit-source-id: 1c5c155f5a1b62e4fea0fd4eeb515a8b7474027b 16 July 2021, 21:13:08 UTC
1e5b631 sdong 16 July 2021, 19:27:00 UTC db_bench seekrandom with multiDB should only create iterators queried (#7818) Summary: Right now, db_bench with seekrandom and multiple DB setup creates iterator for all DBs just to query one of them. It's different from most real workloads. Fix it by only creating iterators that will be queried. Also fix a bug that DBs are not destroyed in multi-DB mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7818 Test Plan: Run db_bench with single/multiDB X using/not using tailing iterator with ASAN build, and validate the behavior is expected. Reviewed By: ajkr Differential Revision: D25720226 fbshipit-source-id: c2ff7ff7120e5ba64287a30b057c5d29b2cbe20b 16 July 2021, 19:28:10 UTC
0229a88 Baptiste Lemaire 16 July 2021, 17:19:33 UTC Crashtest mempurge (#8545) Summary: Add `experiemental_allow_mempurge` flag support for `db_stress` and `db_crashtest.py`, with a `false` default value. I succesfully tested locally both `whitebox` and `blackbox` crash tests with `experiemental_allow_mempurge` flag set as true. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8545 Reviewed By: akankshamahajan15 Differential Revision: D29734513 Pulled By: bjlemaire fbshipit-source-id: 24316c0eccf6caf409e95c035f31d822c66714ae 16 July 2021, 17:20:22 UTC
42ba60b Mark Rambacher 16 July 2021, 14:57:47 UTC Make EncryptionProvider and BlockCipher into Customizable objects (#8354) Summary: Made the EncryptionProvider and BlockCipher classes inherit from Customizable. Added/fixed the CreateFromString method to these classes to create instances from builtin or registered classes. Added tests to verify that instances can be registered and retrieved as appropriate. Added the ability to configure the builtin (CTR, ROT13) classes from configurable properties. Added the appropriate tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8354 Reviewed By: zhichao-cao Differential Revision: D29558949 Pulled By: mrambacher fbshipit-source-id: c20286b32d179777e060f51a58943e9b0cf81d04 16 July 2021, 14:58:51 UTC
aeb913d Peter Dillinger 16 July 2021, 06:49:12 UTC Standardize on GCC for TSAN conditional compilation (#8543) Summary: In https://github.com/facebook/rocksdb/issues/8539 I accidentally only checked for GCC TSAN, which is what I tested locally, while CircleCI and FB CI use clang TSAN. Related: other existing code like in stack_trace.cc only check for clang TSAN. I've now standardized these to the GCC convention in port/lang.h, so now #ifdef __SANITIZE_THREAD__ can check for any TSAN (assuming lang.h include) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8543 Test Plan: Put an assert(false) in slice_test and look for the NOTE about "signal-unsafe call", both GCC and clang. Eventually, CircleCI TSAN in https://github.com/facebook/rocksdb/issues/8538 Reviewed By: zhichao-cao Differential Revision: D29728483 Pulled By: pdillinger fbshipit-source-id: 8a3b8015c2ed48078214c3ee17146a2c3f11c9f7 16 July 2021, 06:50:00 UTC
b678cb1 zaorangyang 16 July 2021, 01:05:58 UTC The formal parameter types of CompressionOptions constructor should b… (#8510) Summary: …e consistent with the member variables's Pull Request resolved: https://github.com/facebook/rocksdb/pull/8510 Reviewed By: ajkr Differential Revision: D29654067 Pulled By: mrambacher fbshipit-source-id: 908baaddfb20c266db7c5aca6a87971393d62ee6 16 July 2021, 01:06:50 UTC
206845c Baptiste Lemaire 16 July 2021, 00:48:17 UTC Mempurge support for wal (#8528) Summary: In this PR, `mempurge` is made compatible with the Write Ahead Log: in case of recovery, the DB is now capable of recovering the data that was "mempurged" and kept in the `imm()` list of immutable memtables. The twist was to add a uint64_t to the `memtable` struct to store the number of the earliest log file containing entries from the `memtable`. When a `Flush` operation is replaced with a `MemPurge`, the `VersionEdit` (which usually contains the new min log file number to pick up for recovery and the level 0 file path of the newly created SST file) is no longer appended to the manifest log, and every time the `deleteWal` method is called, a check is made on the list of immutable memtables. This PR also includes a unit test that verifies that no data is lost upon Reopening of the database when the mempurge feature is activated. This extensive unit test includes two column families, with valid data contained in the imm() at time of "crash"/reopening (recovery). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8528 Reviewed By: pdillinger Differential Revision: D29701097 Pulled By: bjlemaire fbshipit-source-id: 072a900fb6ccc1edcf5eef6caf88f3060238edf9 16 July 2021, 00:49:13 UTC
4e4ec16 longlijian 16 July 2021, 00:22:49 UTC Replace the namespace "rocksdb" to "ROCKSDB_NAMESPACE" (#8531) Summary: For more detail can reference the https://github.com/facebook/rocksdb/issues/6433 (https://github.com/facebook/rocksdb/pull/6433) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8531 Reviewed By: siying Differential Revision: D29717057 Pulled By: ajkr fbshipit-source-id: 3ccad9501e5612590e54a7cf8c447118f323c7f4 16 July 2021, 00:23:39 UTC
5ad3227 Peter Dillinger 15 July 2021, 23:08:19 UTC Work around falsely reported data race on LRUHandle::flags (#8539) Summary: Some bits are mutated and read while holding a lock, other immutable bits (esp. secondary cache compatibility) can be read by arbitrary threads without holding a lock. AFAIK, this doesn't cause an issue on any architecture we care about, because you will get some legitimate version of the value that includes the initialization, as long as synchronization guarantees the initialization happens before the read. I've only seen this in https://github.com/facebook/rocksdb/issues/8538 so far, but it should be fixed regardless. Otherwise, we'll surely get these false reports again some time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8539 Test Plan: some local TSAN test runs and in CircleCI Reviewed By: zhichao-cao Differential Revision: D29720262 Pulled By: pdillinger fbshipit-source-id: 365fd7e565577c648815161f71b339bcb5ce12d5 15 July 2021, 23:09:18 UTC
31193a7 Jay Zhuang 15 July 2021, 20:36:48 UTC Add missing steps for cmake build (#8524) Summary: Some cmake and test configuration are set in pre-steps enviroment variables. Add the missing steps. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8524 Test Plan: CI pass Reviewed By: siying Differential Revision: D29682731 Pulled By: jay-zhuang fbshipit-source-id: afda1acf6a7b76989db450442b0b27f387388b9d 15 July 2021, 20:37:49 UTC
803a40d longlijian 14 July 2021, 23:04:01 UTC Delete legacy code not used any more. (#8508) Summary: The removed function in this PR, just only have declared and dose not have any reference used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8508 Reviewed By: mrambacher Differential Revision: D29649033 Pulled By: jay-zhuang fbshipit-source-id: df98143b73d6c184a2a60c9f7ea2548a065ee35d 14 July 2021, 23:04:56 UTC
8700332 hongrubb 14 July 2021, 01:12:03 UTC Fix Get() return status when block cache is disabled (#8485) Summary: This PR is for https://github.com/facebook/rocksdb/issues/8453 We need to update `s = biter.status();` when `biter.status().IsIncomplete()` is true. By doing this, can fix the problem in issue. Besides, we still need to update `db_statistics` in `get_context.ReportCounters()` before return back. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8485 Reviewed By: jay-zhuang Differential Revision: D29604835 Pulled By: ajkr fbshipit-source-id: c7f2f1cd058223ce1b507ec05d57cf264b9c9710 14 July 2021, 01:13:24 UTC
7b9ecd4 sherriiiliu 13 July 2021, 19:39:42 UTC fix several MSVC build errors (#8519) Summary: Fixed a few MSVC (VCToolsVersion=14.0) build errors and warnings * `DEFINE_string` is a macro and VC compiler complains that it cannot put [ifdef-inside-define](https://stackoverflow.com/questions/5586429/ifdef-inside-define) * `sleep()` is not a recognizable function. Use `FLAGS_env->SleepForMicroseconds` instead * Define precise type in comparison to avoid mismatch warning Pull Request resolved: https://github.com/facebook/rocksdb/pull/8519 Reviewed By: jay-zhuang Differential Revision: D29683086 fbshipit-source-id: 8c80941472089f8daba84ae29597e75e603850e4 13 July 2021, 19:40:43 UTC
e8e911a dependabot[bot] 13 July 2021, 00:04:40 UTC Bump addressable from 2.7.0 to 2.8.0 in /docs (#8515) Summary: Bumps [addressable](https://github.com/sporkmonger/addressable) from 2.7.0 to 2.8.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md">addressable's changelog</a>.</em></p> <blockquote> <h1>Addressable 2.8.0</h1> <ul> <li>fixes ReDoS vulnerability in Addressable::Template#match</li> <li>no longer replaces <code>+</code> with spaces in queries for non-http(s) schemes</li> <li>fixed encoding ipv6 literals</li> <li>the <code>:compacted</code> flag for <code>normalized_query</code> now dedupes parameters</li> <li>fix broken <code>escape_component</code> alias</li> <li>dropping support for Ruby 2.0 and 2.1</li> <li>adding Ruby 3.0 compatibility for development tasks</li> <li>drop support for <code>rack-mount</code> and remove Addressable::Template#generate</li> <li>performance improvements</li> <li>switch CI/CD to GitHub Actions</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/sporkmonger/addressable/commit/6469a232c0f1892809ff66737370c765d574e16c"><code>6469a23</code></a> Updating gemspec again</li> <li><a href="https://github.com/sporkmonger/addressable/commit/24336385de0261571b3adaad0431459edb420c79"><code>2433638</code></a> Merge branch 'main' of github.com:sporkmonger/addressable into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/e9c76b889789c75d7073c17b0ab557635d3f6704"><code>e9c76b8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/378">https://github.com/facebook/rocksdb/issues/378</a> from ashmaroli/flat-map</li> <li><a href="https://github.com/sporkmonger/addressable/commit/56c5cf7ece9223ff4240e07078cc26d3adbbbd30"><code>56c5cf7</code></a> Update the gemspec</li> <li><a href="https://github.com/sporkmonger/addressable/commit/c1fed1ca0a44c448e74d761fd44ed94869199807"><code>c1fed1c</code></a> Require a non-vulnerable rake</li> <li><a href="https://github.com/sporkmonger/addressable/commit/0d8a3127e35886ce9284810a7f2438bff6b43cbc"><code>0d8a312</code></a> Adding note about ReDoS vulnerability</li> <li><a href="https://github.com/sporkmonger/addressable/commit/89c76130ce255c601f642a018cb5fb5a80e679a7"><code>89c7613</code></a> Merge branch 'template-regexp' into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/cf8884f815c96b646c796f707bf768cf6eb65543"><code>cf8884f</code></a> Note about alias fix</li> <li><a href="https://github.com/sporkmonger/addressable/commit/bb03f7112e8e478240a0f96e1cc7428159b41586"><code>bb03f71</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/371">https://github.com/facebook/rocksdb/issues/371</a> from charleystran/add_missing_encode_component_doc_entry</li> <li><a href="https://github.com/sporkmonger/addressable/commit/6d1d8094a66cbf932ecf69db6850bc9edaf86de0"><code>6d1d809</code></a> Adding note about :compacted normalization</li> <li>Additional commits viewable in <a href="https://github.com/sporkmonger/addressable/compare/addressable-2.7.0...addressable-2.8.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=addressable&package-manager=bundler&previous-version=2.7.0&new-version=2.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebook/rocksdb/network/alerts). </details> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8515 Reviewed By: jay-zhuang Differential Revision: D29668988 Pulled By: ajkr fbshipit-source-id: c4b7abd4a879a7b562cb8ba745088dba6644f503 13 July 2021, 00:06:07 UTC
a53d6d2 Peter Dillinger 12 July 2021, 23:59:22 UTC Improve support for valgrind error on reachable (#8503) Summary: MyRocks apparently uses valgrind to check for unreachable unfreed data, which is stricter than our valgrind checks. Internal ref: D29257815 This patch adds valgrind support to STATIC_AVOID_DESTRUCTION so that it's not reported with those stricter checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8503 Test Plan: make valgrind_test Also, with modified VALGRIND_OPTS (see Makefile), more kinds of failures seen before than after this commit. Reviewed By: ajkr, yizhang82 Differential Revision: D29597784 Pulled By: pdillinger fbshipit-source-id: 360de157a176aec4d1be99ca20d160ecd47c0873 13 July 2021, 00:00:27 UTC
da90e23 mrambacher 12 July 2021, 19:17:32 UTC Improvements to benchmark.sh script (#8346) Summary: 1. Fix printing of stats when there are no writes (wamp=0). Previously had a div0 error 2. Added multireadrandom command as a valid target 3. Added ability to pass additional command line options to db_bench. Now can say things like benchmark.sh readrandom --mmap_read and the option will be passed to db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8346 Reviewed By: zhichao-cao Differential Revision: D29500436 Pulled By: mrambacher fbshipit-source-id: 54e90708aae9133be3a903e35efdf8f8abbd86fa 12 July 2021, 19:18:17 UTC
955b80e bjlemaire 12 July 2021, 17:41:28 UTC Add WARN/INFO for mempurge output status. (#8514) Summary: The MemPurge output status can either be an Abort if the mempurge is aborted due to the new_mem memtable reaching more than the target capacity (currently 60%), or for other reasons. As a result, in the log, we want to differentiate between an abort status, which in this PR only leads to a ROCKS_LOG_INFO, and any other status, which in this PR leads to a ROCKS_LOG_WARN. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8514 Reviewed By: pdillinger Differential Revision: D29662446 Pulled By: bjlemaire fbshipit-source-id: c9bec8e238ebc7ecb14fbbddf580e6887e281c16 12 July 2021, 17:42:14 UTC
0b75b22 Dmitry Vorobev 12 July 2021, 16:04:40 UTC Implement missing Handler methods in ColumnFamilyCollector. (#8456) Summary: When db is open as secondary, there are basically 2 step process: 1) Collect column families from wal log 2) Apply changes to Memtable In case primary db is TransactionDB instance, wal log will contain some additional data, like noop, etc. ColumnFamilyCollector doesn't implement methods to handle these, so it fails to open a wal log written by TransactionDB. (Everything works fine with standard DB::Open). Memtable recovery process knows how to handle such wal logs, so only missing piece seems to be ColumnFamilyCollector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8456 Reviewed By: ajkr Differential Revision: D29455945 Pulled By: mrambacher fbshipit-source-id: 5b29560fcbc008e17e95d0dc4b07558f3d63e26f 12 July 2021, 16:23:45 UTC
bbdc4f2 Myth 12 July 2021, 16:04:08 UTC Fix a minor issue in checkpoint test case (#8483) Summary: A very simple change :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8483 Reviewed By: ajkr Differential Revision: D29558904 Pulled By: mrambacher fbshipit-source-id: bbe68c20c861103726cb6231ca3fb8fbe1e5a546 12 July 2021, 16:09:09 UTC
c866561 mrambacher 12 July 2021, 16:03:41 UTC Make FlushBlockPolicyFactory into a Customizable class (#8432) Summary: Add ability to treat FlushBlockPolicyFactory as a Customizable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8432 Reviewed By: zhichao-cao Differential Revision: D29558941 Pulled By: mrambacher fbshipit-source-id: 4a791af941ea4a65fc2f1fdfb1d7a95f42ca6774 12 July 2021, 16:04:59 UTC
5afd1e3 Adam Retter 12 July 2021, 12:04:23 UTC Correct CVS -> CSV typo (#8513) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8513 Reviewed By: jay-zhuang Differential Revision: D29654066 Pulled By: mrambacher fbshipit-source-id: b8f492fe21edd37fe1f1c5a4a0e9153f58bbf3e2 12 July 2021, 12:05:16 UTC
d1b70b0 anand76 12 July 2021, 05:37:01 UTC Avoid passing existing BG error to WriteStatusCheck (#8511) Summary: In ```DBImpl::WriteImpl()```, we call ```PreprocessWrite()``` which, among other things, checks the BG error and returns it set. This return status is later on passed to ```WriteStatusCheck()```, which calls ```SetBGError()```. This results in a spurious call, and info logs, on every user write request. We should avoid passing the ```PreprocessWrite()``` return status to ```WriteStatusCheck()```, as the former would have called ```SetBGError()``` already if it encountered any new errors, such as error when creating a new WAL file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8511 Test Plan: Run existing tests Reviewed By: zhichao-cao Differential Revision: D29639917 Pulled By: anand1976 fbshipit-source-id: 19234163969e1645dbeb273712aaf5cd9ea2b182 12 July 2021, 05:37:52 UTC
837705a Baptiste Lemaire 10 July 2021, 00:16:00 UTC Make mempurge a background process (equivalent to in-memory compaction). (#8505) Summary: In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype. In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...). Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge. MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`. The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505 Reviewed By: pdillinger Differential Revision: D29619283 Pulled By: bjlemaire fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95 10 July 2021, 00:23:59 UTC
bb485e9 qieqieplus 09 July 2021, 22:44:57 UTC Add ribbon filter to C API (#8486) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8486 Reviewed By: jay-zhuang Differential Revision: D29625501 Pulled By: pdillinger fbshipit-source-id: e6e2a455ae62a71f3a202278a751b9bba17ad03c 09 July 2021, 23:22:48 UTC
5dd18a8 Jay Zhuang 09 July 2021, 00:50:55 UTC Add micro-benchmark support (#8493) Summary: Add google benchmark for microbench. Add ribbon_bench for benchmark ribbon filter vs. other filters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493 Test Plan: added test to CI To run the benchmark on devhost: Install benchmark: `$ sudo dnf install google-benchmark-devel` Build and run: `$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench` or with cmake: `$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench` Reviewed By: pdillinger Differential Revision: D29589649 Pulled By: jay-zhuang fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff 09 July 2021, 01:22:45 UTC
f127d45 sdong 08 July 2021, 17:50:08 UTC Add comments to options.bottommost_compression (#8415) Summary: Add comments to options.bottommost_compression for options.num_levels=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8415 Reviewed By: ajkr Differential Revision: D29181997 fbshipit-source-id: 5f0f49470f75d796320ecb24d5dc4ef4eb6fbe0f 08 July 2021, 17:50:59 UTC
ac3f3f3 longlijian 08 July 2021, 17:07:37 UTC Eliminate compiler complaining, which the return type of the function… (#8498) Summary: … should be uint64_t. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8498 Reviewed By: jay-zhuang Differential Revision: D29605064 Pulled By: ajkr fbshipit-source-id: e431448ac9d8a37ae83679c4cc5732e29fe49de4 08 July 2021, 17:09:05 UTC
b1a53db sdong 07 July 2021, 23:20:40 UTC FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync() to recover… (#8501) Summary: … small overwritten files. If a file is overwritten with renamed and the parent path is not synced, FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync() will delete the file. However, RocksDB relies on file renaming to be atomic no matter whether the parent directory is synced or not, and the current behavior breaks the assumption and caused some false positive: https://github.com/facebook/rocksdb/pull/8489 Since the atomic renaming is used in CURRENT files, to fix the problem, in FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync(), we recover the state of overwritten file if the file is small. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8501 Test Plan: Run stress test for a while and see it doesn't break. Reviewed By: anand1976 Differential Revision: D29594384 fbshipit-source-id: 589b5c2f0a9d2aca53752d7bdb0231efa5b3ae92 07 July 2021, 23:23:23 UTC
ed8eb43 Andrew Kryczka 07 July 2021, 18:13:09 UTC Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475) Summary: Various tests had disabled valgrind due to it slowing down and timing out (as is the case right now) the CI runs. Where a test was disabled with no comment, I assumed slowness was the cause. For these tests that were slow under valgrind, as well as the ones identified in https://github.com/facebook/rocksdb/issues/8352, this PR moves them behind the compiler flag `-DROCKSDB_FULL_VALGRIND_RUN`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8475 Test Plan: running `make full_valgrind_test`, `make valgrind_test`, `make check`; will verify they appear working correctly Reviewed By: jay-zhuang Differential Revision: D29504843 Pulled By: ajkr fbshipit-source-id: 2aac90749cfbd30d5ce11cb29a07a1b9314eeea7 07 July 2021, 18:14:05 UTC
714ce50 Baptiste Lemaire 07 July 2021, 01:47:59 UTC Fix clang_analyzer failure (#8492) Summary: Previously, the following command: ```USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j$(nproc) analyze``` was raising an error/warning the new_mem could potentially be a `nullptr`. This error appeared due to code changes from https://github.com/facebook/rocksdb/issues/8454, including an if-statement containing "`... && new_mem != nullptr && ...`", which made the analyzer believe that past this `if`-statement, a `new_mem==nullptr` was a possible scenario. This code patch simply introduces `assert`s and removes this condition in the `if`-statement. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8492 Reviewed By: jay-zhuang Differential Revision: D29571275 Pulled By: bjlemaire fbshipit-source-id: 75d72246b70ebbbae7dea11ccb5778686d8bcbea 07 July 2021, 01:48:56 UTC
df4197c anand76 06 July 2021, 23:44:40 UTC Bypass buffer in TestFSWritableFile if direct IO is enabled (#8490) Summary: ```TestFSWritableFile``` buffers data in ```Append``` in order to simulate unsynced data loss on crash. This is only required for buffered IO and should be disabled for direct IO. Otherwise, it causes crash tests to assert on the buffer address alignment - ```db_stress: env/io_posix.cc:1194: virtual rocksdb::IOStatus rocksdb::PosixWritableFile::Append(const rocksdb::Slice&, const rocksdb::IOOptions&, rocksdb::IODebugContext*): Assertion `IsSectorAligned(data.data(), GetRequiredBufferAlignment())' failed.```. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8490 Reviewed By: zhichao-cao Differential Revision: D29565080 Pulled By: anand1976 fbshipit-source-id: 682831fd66ed3b9597caa74fc453e22dfaf9b973 06 July 2021, 23:46:16 UTC
fcd8088 anand76 06 July 2021, 21:15:25 UTC Temporarily disable file deletion after open failure in db_stress (#8489) Summary: Write and metadata error injection during DB open was enabled in https://github.com/facebook/rocksdb/issues/8474. This causes crash tests to fail very frequently due to another fault injection feature that deletes files created after the last dir sync during DB open. In real life, a similar failure would happen if the FS returns error on the CURRENT file rename, but the rename actually succeeded and got partially persisted (dir entry for the old CURRENT file got removed, but the entry for the new one is not persisted). Temporarily disable the fault injection feature until we figure out the likelihood of this bug happening and the proper way to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8489 Test Plan: Stress test can open the DB successfully Reviewed By: siying Differential Revision: D29564516 Pulled By: anand1976 fbshipit-source-id: ffd1650715ea3c5bf7131936b0ca6fcf66f4e14e 06 July 2021, 21:16:57 UTC
f33611d sdong 06 July 2021, 18:04:04 UTC Stress test to inject read failures in DB reopen (#8476) Summary: Inject read failures in DB reopen, just as what we do for metadata writes and writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8476 Test Plan: Some manual tests and make sure failures are triggered. Reviewed By: anand1976 Differential Revision: D29507283 fbshipit-source-id: d04da0163973447041038bd87701686a417c4e0c 06 July 2021, 18:05:27 UTC
1ae026c Levi Tamasi 06 July 2021, 17:13:40 UTC Partially revert the "apply subrange of table property collectors" change (#8465) Summary: We ended up using a different approach for tracking the amount of garbage in blob files (see e.g. https://github.com/facebook/rocksdb/pull/8450), so the ability to apply only a range of table property collectors is now unnecessary. The patch reverts this part of https://github.com/facebook/rocksdb/pull/8298 while keeping the cleanup done in that PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8465 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29399921 Pulled By: ltamasi fbshipit-source-id: af64816c357d0829b9d7ba8ca1477038138f6f0a 06 July 2021, 17:14:32 UTC
570248a mrambacher 06 July 2021, 16:17:13 UTC Make SecondaryCache Customizable (#8480) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8480 Reviewed By: zhichao-cao Differential Revision: D29528740 Pulled By: mrambacher fbshipit-source-id: fd0f70d15f66611c8498257a9973f7e98ca13839 06 July 2021, 16:18:08 UTC
9dc887e Baptiste Lemaire 02 July 2021, 12:22:03 UTC Memtable "MemPurge" prototype (#8454) Summary: Implement an experimental feature called "MemPurge", which consists in purging "garbage" bytes out of a memtable and reuse the memtable struct instead of making it immutable and eventually flushing its content to storage. The prototype is by default deactivated and is not intended for use. It is intended for correctness and validation testing. At the moment, the "MemPurge" feature can be switched on by using the `options.experimental_allow_mempurge` flag. For this early stage, when the allow_mempurge flag is set to `true`, all the flush operations will be rerouted to perform a MemPurge. This is a temporary design decision that will give us the time to explore meaningful heuristics to use MemPurge at the right time for relevant workloads . Moreover, the current MemPurge operation only supports `Puts`, `Deletes`, `DeleteRange` operations, and handles `Iterators` as well as `CompactionFilter`s that are invoked at flush time . Three unit tests are added to `db_flush_test.cc` to test if MemPurge works correctly (and checks that the previously mentioned operations are fully supported thoroughly tested). One noticeable design decision is the timing of the MemPurge operation in the memtable workflow: for this prototype, the mempurge happens when the memtable is switched (and usually made immutable). This is an inefficient process because it implies that the entirety of the MemPurge operation happens while holding the db_mutex. Future commits will make the MemPurge operation a background task (akin to the regular flush operation) and aim at drastically enhancing the performance of this operation. The MemPurge is also not fully "WAL-compatible" yet, but when the WAL is full, or when the regular MemPurge operation fails (or when the purged memtable still needs to be flushed), a regular flush operation takes place. Later commits will also correct these behaviors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8454 Reviewed By: anand1976 Differential Revision: D29433971 Pulled By: bjlemaire fbshipit-source-id: 6af48213554e35048a7e03816955100a80a26dc5 02 July 2021, 12:23:02 UTC
c76778e Akanksha Mahajan 02 July 2021, 02:17:21 UTC Call OnCompactionCompleted API in case of DisableManualCompaction (#8469) Summary: Call OnCompactionCompleted API in case of DisableManualCompaction() with updated Status::Incomplete Pull Request resolved: https://github.com/facebook/rocksdb/pull/8469 Reviewed By: ajkr Differential Revision: D29475517 Pulled By: akankshamahajan15 fbshipit-source-id: a1726c5e6ee18c0b5097ea04f5e6975fbe108055 02 July 2021, 02:18:55 UTC
b207377 Peter (Stig) Edwards 02 July 2021, 01:41:20 UTC Add -report_open_timing to db_bench (#8464) Summary: Hello and thanks for RocksDB, This PR adds support for ```-report_open_timing true``` to ```db_bench```. It can be useful when tuning RocksDB on filesystem/env with high latencies for file level operations (create/delete/rename...) seen during ```((Optimistic)Transaction)DB::Open```. Some examples: ``` > db_bench -benchmarks updaterandom -num 1 -db /dev/shm/db_bench > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 3.90133 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 3.33414 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true 2>&1 | grep -A1 OpenDb OpenDb: 6.05423 milliseconds > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 4.06859 milliseconds > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 2.85794 milliseconds > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true 2>&1 | grep OpenDb OpenDb: 6.46376 milliseconds > db_bench -benchmarks updaterandom -num 1 -db /clustered_fs/db_bench > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 3.79805 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 3.00174 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true 2>&1 | grep OpenDb OpenDb: 24.8732 milliseconds ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8464 Reviewed By: hx235 Differential Revision: D29398096 Pulled By: zhichao-cao fbshipit-source-id: 8f05dc3284f084612a3f30234e39e1c37548f50c 02 July 2021, 01:42:19 UTC
a95a776 Zhichao Cao 01 July 2021, 21:15:49 UTC Inject fatal write failures to db_stress when DB is running (#8479) Summary: add the injest_error_severity to control if it is a retryable IO Error or a fatal or unrecoverable error. Use a flag to indicate, if fatal error comes, the flag is set and db is stopped (but not corrupted). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8479 Test Plan: run ./db_stress --reopen=0 --read_fault_one_in=1000 --write_fault_one_in=5 --disable_wal=true --write_buffer_size=3000000 -writepercent=5 -readpercent=50 --injest_error_severity=2 --column_families=1, make check Reviewed By: anand1976 Differential Revision: D29524271 Pulled By: zhichao-cao fbshipit-source-id: 1aa9fb9b5655b0adba6f5ad12005ca8c074c795b 01 July 2021, 21:16:47 UTC
41d3215 anand76 01 July 2021, 19:22:01 UTC Enable crash test to run using fbcode components (#8471) Summary: Add a new test ```fbcode_crash_test``` to rocksdb-lego-determinator. This test allows the crash test to be run on Facebook Sandcastle infra using fbcode components. Also use the default Env in db_stress to access the expected values path as it requires a memory mapped file and may not work with custom Envs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8471 Reviewed By: ajkr Differential Revision: D29474722 Pulled By: anand1976 fbshipit-source-id: 7d086d82dd7091ae48e08cb4ace763ce3e3b87ef 01 July 2021, 19:23:01 UTC
d45b837 mrambacher 01 July 2021, 18:52:18 UTC Fix TSAN issue (#8477) Summary: Added mutex to fix TSAN issue Pull Request resolved: https://github.com/facebook/rocksdb/pull/8477 Reviewed By: zhichao-cao Differential Revision: D29517053 Pulled By: mrambacher fbshipit-source-id: 661ccb1f495b7d34874a79e0a3d7aea1123d6047 01 July 2021, 18:53:18 UTC
ba224b7 sdong 30 June 2021, 23:45:44 UTC Stress Test to inject write failures in reopen (#8474) Summary: Previously Stress can inject metadata write failures when reopening a DB. We extend it to file append too, in the same way. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8474 Test Plan: manually run crash test with various setting and make sure the failures are triggered as expected. Reviewed By: zhichao-cao Differential Revision: D29503116 fbshipit-source-id: e73a446e80ccbd09301a579280e56ff949381fab 30 June 2021, 23:46:41 UTC
41c4b66 mrambacher 30 June 2021, 21:08:19 UTC Fix PrepareOptions for Customizable Classes (#8468) Summary: Added the Customizable::ConfigureNewObject method. The method will configure the object if options are found and invoke PrepareOptions if the flag is set properly. Added tests to test that PrepareOptions is properly called and to test if PrepareOptions fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8468 Reviewed By: zhichao-cao Differential Revision: D29494703 Pulled By: mrambacher fbshipit-source-id: d5767dee5d7a98620ac66190262101cd0aa9d2b7 30 June 2021, 21:09:36 UTC
a0cbb69 anand76 30 June 2021, 20:28:45 UTC Fix assertion failure when releasing a handle after secondary cache lookup fails (#8470) Summary: When the secondary cache lookup fails, we may still allocate a handle and charge the cache for metadata usage. If the cache is full, this can cause the usage to go over capacity. Later, when a (unrelated) handle is released, it trips up an assertion that checks that usage is less than capacity. To prevent this assertion failure, don't charge the cache for a failed secondary cache lookup. Tests: Run crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/8470 Reviewed By: zhichao-cao Differential Revision: D29474713 Pulled By: anand1976 fbshipit-source-id: 27191969c95470a7b070d292b458efce71395bf2 30 June 2021, 20:29:30 UTC
93a7389 Jay Zhuang 29 June 2021, 18:46:44 UTC Add statistics support on CompactionService remote side (#8368) Summary: Add statistics option on CompactionService remote side. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8368 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D28944427 Pulled By: jay-zhuang fbshipit-source-id: 2a19217f4a69b6e511af87eed12391860ef00c5e 29 June 2021, 18:48:14 UTC
3503f28 Jay Zhuang 29 June 2021, 17:41:22 UTC Add sub-compaction support for RemoteCompaction (#8364) Summary: Change the job_id for remote compaction interface, which will include both internal compaction job_id, also a sub_compaction_job_id. It is not a backward compatible change. The user needs to update interface during upgrade. (We will avoid backward incompatible change after the feature is not experimental.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8364 Reviewed By: ajkr Differential Revision: D28917301 Pulled By: jay-zhuang fbshipit-source-id: 6d72a21f652bb517ad6954d0387b496797fc4e11 29 June 2021, 17:42:19 UTC
89f66d4 mrambacher 29 June 2021, 16:07:10 UTC Add customizable_util.h to the public API (#8301) Summary: Useful for allowing new classes to create and manage Customizable objects without using internal APIs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8301 Reviewed By: zhichao-cao Differential Revision: D29428303 Pulled By: mrambacher fbshipit-source-id: 3d33d5197cc8379fe35b54d3d169f91f0dfe7a47 29 June 2021, 16:08:57 UTC
bac3994 Andrew Kryczka 29 June 2021, 15:38:57 UTC jemalloc_helper: Limit the mm_malloc.h hack to glibc on linux (#8425) Summary: Original author: kraj (https://github.com/facebook/rocksdb/issues/8413) We have a hack to ensure clang's `posix_memalign()` hack works to be compatible with glibc's `posix_memalign()` declaration. Our side of the hack is irrelevant and should be omitted when not using glibc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8425 Reviewed By: mrambacher Differential Revision: D29239029 Pulled By: ajkr fbshipit-source-id: 12b900f50a4823b880a6558f25d8590dbfc0aa26 29 June 2021, 15:40:02 UTC
390c524 Lucian Petrut 29 June 2021, 13:51:11 UTC Allow using WindowsThread with Mingw (#8108) Summary: Allow using WindowsThread with Mingw Most Mingw builds require Posix threads in order to use std::thread. As per https://github.com/facebook/rocksdb/issues/7764, this is not always the case. That being considered, we're going to improve the Mingw thread model checks. Closes: https://github.com/facebook/rocksdb/issues/7764 Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8108 Reviewed By: jay-zhuang Differential Revision: D27365778 Pulled By: mrambacher fbshipit-source-id: 2c15b1f04ae90e1e3a25a33e86ceb779224a9529 29 June 2021, 13:52:08 UTC
373e3a1 mrambacher 28 June 2021, 19:27:39 UTC Fix Immutable Customizable Serialization (#8457) Summary: If a Customizable option was not mutable, it would still appear in the list of mutable options when serialized. This meant that when the immutable options were used to configure another immutable object, an "option not changeable" status would be returned. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8457 Reviewed By: zhichao-cao Differential Revision: D29428298 Pulled By: mrambacher fbshipit-source-id: 3945b0b822f8e5955a7c5590fe64dfd5bc1fe6a0 28 June 2021, 19:28:28 UTC
be21908 mrambacher 28 June 2021, 15:12:32 UTC Add BlobMetaData retrieval methods (#8273) Summary: Added BlobMetaData to ColumnFamilyMetaData and LiveBlobMetaData and DB API GetLiveBlobMetaData to retrieve it. First pass at struct. More tests and maybe fields to come... Pull Request resolved: https://github.com/facebook/rocksdb/pull/8273 Reviewed By: ltamasi Differential Revision: D29102400 Pulled By: mrambacher fbshipit-source-id: 8a2383a4446328be6b91dced9841fdd3dfc80b73 28 June 2021, 15:13:29 UTC
6f9ed59 anand76 28 June 2021, 06:53:47 UTC Allow db_stress to use a secondary cache (#8455) Summary: Add a ```-secondary_cache_uri``` to db_stress to allow the user to specify a custom ```SecondaryCache``` object from the object registry. Also allow db_crashtest.py to be run with an alternate db_stress location. Together, these changes will allow us to run db_stress using FB internal components. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8455 Reviewed By: zhichao-cao Differential Revision: D29371972 Pulled By: anand1976 fbshipit-source-id: dd1b1fd80ebbedc11aa63d9246ea6ae49edb77c4 28 June 2021, 06:54:39 UTC
be8199c Akanksha Mahajan 25 June 2021, 17:44:55 UTC Run Merge with Integrated BlobDB in stress, crash and db_bench (#8461) Summary: Run Merge with Intergrated BlobDB in stress tests, crash tests and db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8461 Test Plan: 1. python3 -u tools/db_crashtest.py --simple whitebox ---use_merge=1 --enable_blob_files=1 2. ./db_bench --benchmarks="readwhilemerging" --merge_operator=uint64add --enable_blob_files=true Reviewed By: ltamasi Differential Revision: D29394824 Pulled By: akankshamahajan15 fbshipit-source-id: 0a8e492b13129673e088fb8af3402ab678bb473a 25 June 2021, 17:45:52 UTC
a904c62 Zhichao Cao 25 June 2021, 07:46:33 UTC Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412) Summary: In PR https://github.com/facebook/rocksdb/issues/7523 , checksum handoff is introduced in RocksDB for WAL, Manifest, and SST files. When user enable checksum handoff for a certain type of file, before the data is written to the lower layer storage system, we calculate the checksum (crc32c) of each piece of data and pass the checksum down with the data, such that data verification can be down by the lower layer storage system if it has the capability. However, it cannot cover the whole lifetime of the data in the memory and also it potentially introduces extra checksum calculation overhead. In this PR, we introduce a new interface in WritableFileWriter::Append, which allows the caller be able to pass the data and the checksum (crc32c) together. In this way, WritableFileWriter can directly use the pass-in checksum (crc32c) to generate the checksum of data being passed down to the storage system. It saves the calculation overhead and achieves higher protection coverage. When a new checksum is added with the data, we use Crc32cCombine https://github.com/facebook/rocksdb/issues/8305 to combine the existing checksum and the new checksum. To avoid the segmenting of data by rate-limiter before it is stored, rate-limiter is called enough times to accumulate enough credits for a certain write. This design only support Manifest and WAL which use log_writer in the current stage. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8412 Test Plan: make check, add new testing cases. Reviewed By: anand1976 Differential Revision: D29151545 Pulled By: zhichao-cao fbshipit-source-id: 75e2278c5126cfd58393c67b1efd18dcc7a30772 25 June 2021, 07:47:17 UTC
3d844df Andrew Kryczka 25 June 2021, 04:04:05 UTC add missing fields to `GetLiveFilesMetaData()` (#8460) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8460 Reviewed By: jay-zhuang Differential Revision: D29381865 Pulled By: ajkr fbshipit-source-id: 47ba54c25f3cc039d72ea32e1df20875795683b3 25 June 2021, 04:05:03 UTC
95d0ee9 Akanksha Mahajan 25 June 2021, 01:10:32 UTC Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445) Summary: Provide support for Merge operation with base values during Compaction in IntegratedBlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8445 Test Plan: Add new unit test Reviewed By: ltamasi Differential Revision: D29343949 Pulled By: akankshamahajan15 fbshipit-source-id: 844f6f02f93388a11e6e08bda7bb3a2a28e47c70 25 June 2021, 01:11:30 UTC
66b62a1 Levi Tamasi 25 June 2021, 00:24:41 UTC Update HISTORY.md for PR 8450 (#8458) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8458 Reviewed By: zhichao-cao Differential Revision: D29378728 Pulled By: ltamasi fbshipit-source-id: d5a40b1414500f53823763be5c2bfce8db04daf8 25 June 2021, 00:26:32 UTC
68d8b28 Levi Tamasi 24 June 2021, 23:11:07 UTC Log the amount of blob garbage generated by compactions in the MANIFEST (#8450) Summary: The patch builds on `BlobGarbageMeter` and `BlobCountingIterator` (introduced in https://github.com/facebook/rocksdb/issues/8426 and https://github.com/facebook/rocksdb/issues/8443 respectively) and ties it all together. It measures the amount of garbage generated by a compaction and logs the corresponding `BlobFileGarbage` records as part of the compaction job's `VersionEdit`. Note: in order to have accurate results, `kRemoveAndSkipUntil` for compaction filters is implemented using iteration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8450 Test Plan: Ran `make check` and the crash test script. Reviewed By: jay-zhuang Differential Revision: D29338207 Pulled By: ltamasi fbshipit-source-id: 4381c432ac215139439f6d6fb801a6c0e4d8c128 24 June 2021, 23:11:56 UTC
75741eb Peter (Stig) Edwards 24 June 2021, 18:56:03 UTC Add more ops to: db_bench -report_file_operations (#8448) Summary: Hello and thanks for RocksDB, Here is a PR to add file deletes, renames and ```Flush()```, ```Sync()```, ```Fsync()``` and ```Close()``` to file ops report. The reason is to help tune RocksDB options when using an env/filesystem with high latencies for file level ("metadata") operations, typically seen during ```DB::Open``` (```db_bench -num 0``` also see https://github.com/facebook/rocksdb/pull/7203 where IOTracing does not trace ```DB::Open```). Before: ``` > db_bench -benchmarks updaterandom -num 0 -report_file_operations true ... Entries: 0 ... Num files opened: 12 Num Read(): 6 Num Append(): 8 Num bytes read: 6216 Num bytes written: 6289 ``` After: ``` > db_bench -benchmarks updaterandom -num 0 -report_file_operations true ... Entries: 0 ... Num files opened: 12 Num files deleted: 3 Num files renamed: 4 Num Flush(): 10 Num Sync(): 5 Num Fsync(): 1 Num Close(): 2 Num Read(): 6 Num Append(): 8 Num bytes read: 6216 Num bytes written: 6289 ``` Before: ``` > db_bench -benchmarks updaterandom -report_file_operations true ... Entries: 1000000 ... Num files opened: 18 Num Read(): 396339 Num Append(): 1000058 Num bytes read: 892030224 Num bytes written: 187569238 ``` After: ``` > db_bench -benchmarks updaterandom -report_file_operations true ... Entries: 1000000 ... Num files opened: 18 Num files deleted: 5 Num files renamed: 4 Num Flush(): 1000068 Num Sync(): 9 Num Fsync(): 1 Num Close(): 6 Num Read(): 396339 Num Append(): 1000058 Num bytes read: 892030224 Num bytes written: 187569238 ``` Another example showing how using ```DB::OpenForReadOnly``` reduces file operations compared to ```((Optimistic)Transaction)DB::Open```: ``` > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -readonly true -report_file_operations true ... Entries: 0 ... Num files opened: 8 Num files deleted: 0 Num files renamed: 0 Num Flush(): 0 Num Sync(): 0 Num Fsync(): 0 Num Close(): 0 Num Read(): 13 Num Append(): 0 Num bytes read: 374 Num bytes written: 0 ``` ``` > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_file_operations true ... Entries: 0 ... Num files opened: 14 Num files deleted: 3 Num files renamed: 4 Num Flush(): 14 Num Sync(): 5 Num Fsync(): 1 Num Close(): 3 Num Read(): 11 Num Append(): 10 Num bytes read: 7291 Num bytes written: 7357 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8448 Reviewed By: anand1976 Differential Revision: D29333818 Pulled By: zhichao-cao fbshipit-source-id: a06a8c87f799806462319115195b3e94faf5f542 24 June 2021, 18:56:51 UTC
d44ef2e Levi Tamasi 23 June 2021, 20:27:50 UTC Remove obsolete method VersionSet::VerifyCompactionFileConsistency (#8449) Summary: `VersionSet::VerifyCompactionFileConsistency` was superseded by the LSM tree consistency checks introduced in https://github.com/facebook/rocksdb/pull/6901, which are more comprehensive, more efficient, and are performed unconditionally even in release builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8449 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D29337441 Pulled By: ltamasi fbshipit-source-id: a05324f88e3400e27e6a00406c878a6276e0c9cc 23 June 2021, 20:28:34 UTC
6adc39e Levi Tamasi 23 June 2021, 17:24:39 UTC Add an internal iterator that can measure the inflow of blobs (#8443) Summary: Follow-up to https://github.com/facebook/rocksdb/issues/8426 . The patch adds a new kind of `InternalIterator` that wraps another one and passes each key-value encountered to `BlobGarbageMeter` as inflow. This iterator will be used as an input iterator for compactions when the input SSTs reference blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8443 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29311987 Pulled By: ltamasi fbshipit-source-id: b4493b4c0c0c2e3c2ecc33c8969a5ef02de5d9d8 23 June 2021, 17:25:47 UTC
3f20925 Baptiste Lemaire 23 June 2021, 02:06:44 UTC Add list live files metadata (#8446) Summary: Add an argument to ldb to dump live file names, column families, and levels, `list_live_files_metadata`. The output shows all active SST file names, sorted first by column family and then by level. For each level the SST files are sorted alphabetically. Typically, the output looks like this: ``` ./ldb --db=/tmp/test_db list_live_files_metadata Live SST Files: ===== Column Family: default ===== ---------- level 0 ---------- /tmp/test_db/000069.sst ---------- level 1 ---------- /tmp/test_db/000064.sst /tmp/test_db/000065.sst /tmp/test_db/000066.sst /tmp/test_db/000071.sst ---------- level 2 ---------- /tmp/test_db/000038.sst /tmp/test_db/000039.sst /tmp/test_db/000052.sst /tmp/test_db/000067.sst /tmp/test_db/000070.sst ------------------------------ ``` Second, a flag was added `--sort_by_filename`, to change the layout of the output. When this flag is added to the command, the output shows all active SST files sorted by name, in front of which the LSM level and the column family are mentioned. With the same example, the following command would return: ``` ./ldb --db=/tmp/test_db list_live_files_metadata --sort_by_filename Live SST Files: /tmp/test_db/000038.sst : level 2, column family 'default' /tmp/test_db/000039.sst : level 2, column family 'default' /tmp/test_db/000052.sst : level 2, column family 'default' /tmp/test_db/000064.sst : level 1, column family 'default' /tmp/test_db/000065.sst : level 1, column family 'default' /tmp/test_db/000066.sst : level 1, column family 'default' /tmp/test_db/000067.sst : level 2, column family 'default' /tmp/test_db/000069.sst : level 0, column family 'default' /tmp/test_db/000070.sst : level 2, column family 'default' /tmp/test_db/000071.sst : level 1, column family 'default' ------------------------------ ``` Thus, the user can either request to show the files by levels, or sorted by filenames. This PR includes a simple Python unit test that makes sure the file name and level printed out by this new feature matches the one found with an existing feature, `dump_live_file`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8446 Reviewed By: akankshamahajan15 Differential Revision: D29320080 Pulled By: bjlemaire fbshipit-source-id: 01fb7b5637c59010d74c80730a28d815994e7009 23 June 2021, 02:07:46 UTC
3ab0eae Levi Tamasi 23 June 2021, 00:33:15 UTC Deflake BlobDBTest.SnapshotAndGarbageCollection (#8444) Summary: This test case has been failing occasionally due to automatic compactions kicking in, resulting in GC generating additional blob files that the test did not expect. Disabling automatic compactions to get rid of this flakiness. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8444 Test Plan: `gtest-parallel --repeat=1000 ./blob_db_test --gtest_filter="BlobDBTest.SnapshotAndGarbageCollection"` Reviewed By: jay-zhuang Differential Revision: D29316987 Pulled By: ltamasi fbshipit-source-id: 9815d189ed7d63890622768675a01990e3680221 23 June 2021, 00:34:03 UTC
005c046 Levi Tamasi 22 June 2021, 20:36:40 UTC Update HISTORY.md for PR 8437 (#8442) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8442 Reviewed By: akankshamahajan15 Differential Revision: D29306432 Pulled By: ltamasi fbshipit-source-id: 0fe55736368ac024b2f51213b65f4b06d71d60e6 22 June 2021, 20:37:22 UTC
0a1aed4 Baptiste Lemaire 22 June 2021, 18:45:14 UTC Fix double slashes in user-provided db path. (#8439) Summary: At the moment, the following command : "`./ --db=mypath/ dump_file_files`" returns a series of erronous names with double slashes, ie: "`mypath//000xxx.sst`", including manifest file names with double slashes "`mypath//MANIFEST-00XXX`", whereas "`./ --db=mypath dump_file_files`" correctly returns "`mypath/000xxx.sst`" and "`mypath/MANIFEST-00XXX`". This (very short) PR simply checks if there is a need to add or remove any '`/`' character when the `db_path` and `manifest_filename`/sst `filenames` are concatenated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8439 Reviewed By: akankshamahajan15 Differential Revision: D29301349 Pulled By: bjlemaire fbshipit-source-id: 3e9e58f9749d278b654ae838fcee13ad698705a8 22 June 2021, 18:46:25 UTC
f89423a Jay Zhuang 22 June 2021, 18:09:11 UTC Revert "Revert "Snapshot release triggered compaction without multiple tombstones (#8357)" (#8410)" (#8438) Summary: This reverts commit 25be1ed66a354ee1d665d7a28eb20d36afc75e90. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8438 Test Plan: Run the impacted mysql test 40 times Reviewed By: ajkr Differential Revision: D29286247 Pulled By: jay-zhuang fbshipit-source-id: d3bd056971a19a8b012d5d0295fa045c012b3c04 22 June 2021, 18:10:03 UTC
b511b04 Andrew Kryczka 22 June 2021, 17:29:15 UTC Fixup HISTORY.md for 6.22 release (#8441) Summary: `git diff origin/6.21.fb origin/6.22.fb -- HISTORY.md` looked odd. This PR fixes it up by moving items from 6.21.0 to 6.22.0 that were never in any 6.21 release. Also mentioned the background stat collection fix under 6.22 (previously it was mentioned under 6.21 patch releases only). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8441 Reviewed By: jay-zhuang Differential Revision: D29304812 Pulled By: ajkr fbshipit-source-id: 2a928a9518a1d6615321d5c2d1e22b17cbb59093 22 June 2021, 17:30:10 UTC
cbb3b25 Levi Tamasi 22 June 2021, 16:48:50 UTC Print blob file checksums as hex (#8437) Summary: Currently, blob file checksums are incorrectly dumped as raw bytes in the `ldb manifest_dump` output (i.e. they are not printed as hex). The patch fixes this and also updates some test cases to reflect that the checksum value field in `BlobFileAddition` and `SharedBlobFileMetaData` contains the raw checksum and not a hex string. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8437 Test Plan: `make check` Tested using `ldb manifest_dump` Reviewed By: akankshamahajan15 Differential Revision: D29284170 Pulled By: ltamasi fbshipit-source-id: d11cfb3435b14cd73c8a3d3eb14fa0f9fa1d2228 22 June 2021, 16:49:44 UTC
54d73d6 Jay Zhuang 22 June 2021, 16:16:46 UTC Fix DeleteFilesInRange may cause inconsistent compaction error (#8434) Summary: `DeleteFilesInRange()` marks deleting files to `being_compacted` before deleting, which may cause ongoing compactions report corruption exception or ASSERT for debug build. Adding the missing `ComputeCompactionScore()` when `being_compacted` is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8434 Test Plan: Unittest Reviewed By: ajkr Differential Revision: D29276127 Pulled By: jay-zhuang fbshipit-source-id: f5b223e3c1fc6d821e100e3f3442bc70c1d50cf7 22 June 2021, 16:17:37 UTC
065bea1 Levi Tamasi 22 June 2021, 05:24:23 UTC Add a class for measuring the amount of garbage generated during compaction (#8426) Summary: This is part of an alternative approach to https://github.com/facebook/rocksdb/issues/8316. Unlike that approach, this one relies on key-values getting processed one by one during compaction, and does not involve persistence. Specifically, the patch adds a class `BlobGarbageMeter` that can track the number and total size of blobs in a (sub)compaction's input and output on a per-blob file basis. This information can then be used to compute the amount of additional garbage generated by the compaction for any given blob file by subtracting the "outflow" from the "inflow." Note: this patch only adds `BlobGarbageMeter` and associated unit tests. I plan to hook up this class to the input and output of `CompactionIterator` in a subsequent PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8426 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29242250 Pulled By: ltamasi fbshipit-source-id: 597e50ad556540e413a50e804ba15bc044d809bb 22 June 2021, 05:25:30 UTC
a50da40 anand76 22 June 2021, 04:22:57 UTC Fix a tsan warning due to reading flags in LRUHandle without holding a mutex (#8433) Summary: Tsan complains due to a perceived race condition in accessing LRUHandle flags. One thread calls ```LRUHandle::SetHit()``` from ```LRUCacheShard::Lookup()```, while another thread calls ```LRUHandle::IsPending()``` from ```LRUCacheShard::IsReady()```. The latter call is from ```MultiGet```. It doesn't actually have to call ```IsReady``` since a null value indicates the cache handle is not ready, so its sufficient to check for a null value. Also modify ```IsReady``` to acquire the LRU shard mutex. Tests: 1. make check 2. Run tsan_crash Pull Request resolved: https://github.com/facebook/rocksdb/pull/8433 Reviewed By: zhichao-cao Differential Revision: D29278030 Pulled By: anand1976 fbshipit-source-id: 0c9fed56d12eda853e72dadebe75038361bd257f 22 June 2021, 04:23:56 UTC
e9b627f lorinlee 21 June 2021, 20:40:12 UTC Fix comments in Status (#8429) Summary: Signed-off-by: lorinlee <lorinlee1996@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8429 Reviewed By: zhichao-cao Differential Revision: D29256538 Pulled By: ajkr fbshipit-source-id: fa483861148d82119f4c8775d0cf5f6c71b06215 21 June 2021, 20:41:08 UTC
6f33901 Andrew Kryczka 21 June 2021, 20:14:44 UTC Skip c_test and env_test when ASSERT_STATUS_CHECKED=1 (#8430) Summary: - `c_test` fails because `rocksdb_compact_range()` swallows a `Status`. - `env_test` fails because `ReadRequest`s to `MultiRead()` do not have their `Status`es checked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8430 Test Plan: `ASSERT_STATUS_CHECKED=1 make -j48 check` Reviewed By: jay-zhuang Differential Revision: D29257473 Pulled By: ajkr fbshipit-source-id: e02127f971703744be7de85f0a028e4664c79577 21 June 2021, 20:15:31 UTC
7ed69fb Andrew Kryczka 19 June 2021, 19:11:44 UTC Update HISTORY.md and version.h 6.22 release (#8427) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8427 Reviewed By: zhichao-cao Differential Revision: D29246916 Pulled By: ajkr fbshipit-source-id: ccd44ca1a6dd5101dc37f19b8e1fe6c0e3883e0a 19 June 2021, 19:12:46 UTC
19a8926 mwish 19 June 2021, 00:14:01 UTC typo: fix typo in db/write_thread's state (#8423) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8423 Reviewed By: mrambacher Differential Revision: D29232587 Pulled By: jay-zhuang fbshipit-source-id: 04d4937cf0605cbf341a920d1305369a7b8f0574 19 June 2021, 00:14:51 UTC
82a70e1 Zhichao Cao 18 June 2021, 22:02:59 UTC Trace MultiGet Keys and CF_IDs to the trace file (#8421) Summary: Tracing the MultiGet information including timestamp, keys, and CF_IDs to the trace file for analyzing and replay. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8421 Test Plan: make check, add test to trace_analyzer_test Reviewed By: anand1976 Differential Revision: D29221195 Pulled By: zhichao-cao fbshipit-source-id: 30c677d6c39ab31ef4bbdf7e0d1fa1fd79f295ff 18 June 2021, 22:04:05 UTC
d96b012 Akanksha Mahajan 18 June 2021, 19:31:08 UTC Change Windows image back to stable (#8424) Summary: Since windows timeout issue has been fixed. Change the image back to stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8424 Test Plan: Check CircleCI jobs Reviewed By: zhichao-cao Differential Revision: D29235219 Pulled By: akankshamahajan15 fbshipit-source-id: 2c111f96e216dac4ae3d7ec3b4cdd8e459575d37 18 June 2021, 19:32:01 UTC
8ea0a2c anand76 18 June 2021, 16:35:03 UTC Parallelize secondary cache lookup in MultiGet (#8405) Summary: Implement the ```WaitAll()``` interface in ```LRUCache``` to allow callers to issue multiple lookups in parallel and wait for all of them to complete. Modify ```MultiGet``` to use this to parallelize the secondary cache lookups in order to reduce the overall latency. A call to ```cache->Lookup()``` returns a handle that has an incomplete value (nullptr), and the caller can call ```cache->IsReady()``` to check whether the lookup is complete, and pass a vector of handles to ```WaitAll``` to wait for completion. If any of the lookups fail, ```MultiGet``` will read the block from the SST file. Another change in this PR is to rename ```SecondaryCacheHandle``` to ```SecondaryCacheResultHandle``` as it more accurately describes the return result of the secondary cache lookup, which is more like a future. Tests: 1. Add unit tests in lru_cache_test 2. Benchmark results with no secondary cache configured Master - ``` readrandom : 41.175 micros/op 388562 ops/sec; 106.7 MB/s (7277999 of 7277999 found) readrandom : 41.217 micros/op 388160 ops/sec; 106.6 MB/s (7274999 of 7274999 found) multireadrandom : 10.309 micros/op 1552082 ops/sec; (28908992 of 28908992 found) multireadrandom : 10.321 micros/op 1550218 ops/sec; (29081984 of 29081984 found) ``` This PR - ``` readrandom : 41.158 micros/op 388723 ops/sec; 106.8 MB/s (7290999 of 7290999 found) readrandom : 41.185 micros/op 388463 ops/sec; 106.7 MB/s (7287999 of 7287999 found) multireadrandom : 10.277 micros/op 1556801 ops/sec; (29346944 of 29346944 found) multireadrandom : 10.253 micros/op 1560539 ops/sec; (29274944 of 29274944 found) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8405 Reviewed By: zhichao-cao Differential Revision: D29190509 Pulled By: anand1976 fbshipit-source-id: 6f8eff6246712af8a297cfe22ea0d1c3b2a01bb0 18 June 2021, 16:35:59 UTC
e817bc9 Baptiste Lemaire 18 June 2021, 11:56:43 UTC Added memtable garbage statistics (#8411) Summary: **Summary**: 2 new statistics counters are added to RocksDB: `MEMTABLE_PAYLOAD_BYTES_AT_FLUSH` and `MEMTABLE_GARBAGE_BYTES_AT_FLUSH`. The former tracks how many raw bytes of useful data are present on the memtable at flush time, whereas the latter is tracks how many of these raw bytes are considered garbage, meaning that they ended up not being imported on the SSTables resulting from the flush operations. **Unit test**: run `make db_flush_test -j$(nproc); ./db_flush_test` to run the unit test. This executable includes 3 tests, that test support and correct stat calculations for workloads with inserts, deletes, and DeleteRanges. The parameters are set such that the workloads are performed on a single memtable, and a single SSTable is created as a result of the flush operation. The flush operation is manually called in the test file. The tests verify that the values of these 2 statistics counters introduced in this PR can be exactly predicted, showing that we have a full understanding of the underlying operations. **Performance testing**: `./db_bench -statistics -benchmarks=fillrandom -num=10000000` repeated 10 times. Timing done using "date" function in a bash script. _Results_: Original Rocksdb fork: mean 66.6 sec, std 1.18 sec. This feature branch: mean 67.4 sec, std 1.35 sec. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8411 Reviewed By: akankshamahajan15 Differential Revision: D29150629 Pulled By: bjlemaire fbshipit-source-id: 7b3c2e86d50c6aa34fa50fd134282eacb543a5b1 18 June 2021, 11:57:27 UTC
5ba1b6e Akanksha Mahajan 18 June 2021, 04:55:42 UTC Cache warming data blocks during flush (#8242) Summary: This PR prepopulates warm/hot data blocks which are already in memory into block cache at the time of flush. On a flush, the data block that is in memory (in memtables) get flushed to the device. If using Direct IO, additional IO is incurred to read this data back into memory again, which is avoided by enabling newly added option. Right now, this is enabled only for flush for data blocks. We plan to expand this option to cover compactions in the future and for other types of blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8242 Test Plan: Add new unit test Reviewed By: anand1976 Differential Revision: D28521703 Pulled By: akankshamahajan15 fbshipit-source-id: 7219d6958821cedce689a219c3963a6f1a9d5f05 18 June 2021, 04:56:47 UTC
d53f7ff 聂佩轩 17 June 2021, 22:59:53 UTC Add DeteleRange support for DBWithTTL (#8384) Summary: This commit is for enabling `DBWithTTL` to use `DeteleRange` which it cannot before. As (int32_t)Timestamp is suffixed to values in `DBWithTTL`, there is no reason that it cannot use the common used api. I added `DeleteRangeCF` in `DBWithTTLImpl::Write` so that we can use `DeteleRange` normally. When we run code like `dbWithTtl->DeleteRange(start, end)`, it executes`WriteBatchInternal::DeleteRange` internally. Intended to fix https://github.com/facebook/rocksdb/issues/7218 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8384 Test Plan: added corresponded testing logic to existing unit test Reviewed By: jay-zhuang Differential Revision: D29176734 fbshipit-source-id: 6874ed979fc08e1d138149d03653e43a75f0e0e6 17 June 2021, 23:00:50 UTC
865a251 Peter Dillinger 17 June 2021, 19:28:00 UTC Mark Ribbon filter and optimize_filters_for_memory as production (#8408) Summary: Marked the Ribbon filter and optimize_filters_for_memory features as production-ready, each enabling memory savings for Bloom-like filters. Use `NewRibbonFilterPolicy` in place of `NewBloomFilterPolicy` to use Ribbon filters instead of Bloom, or `ribbonfilter` in place of `bloomfilter` in configuration string. Some small refactoring in db_stress. Removed/refactored unused code in db_bench, in part preparing for future default possibly being different from "disabled." Pull Request resolved: https://github.com/facebook/rocksdb/pull/8408 Test Plan: Lots of prior automated, ad-hoc, and "real world" testing. Updated tests for new API names. Quick db_bench test: bloom fillrandom 77730 ops/sec rocksdb.block.cache.filter.bytes.insert COUNT : 89929384 ribbon fillrandom 71492 ops/sec rocksdb.block.cache.filter.bytes.insert COUNT : 64531384 Reviewed By: mrambacher Differential Revision: D29140805 Pulled By: pdillinger fbshipit-source-id: d742c922722421678f95ad85eeb0aaebc9f5e49a 17 June 2021, 19:29:16 UTC
575ea26 anand76 17 June 2021, 19:04:58 UTC Don't log a warning if file system doesn't support ReopenWritableFile() (#8414) Summary: RocksDB logs a warning if WAL truncation on DB open fails. Its possible that on some file systems, truncation is not required and they would return ```Status::NotSupported()``` for ```ReopenWritableFile```. Don't log a warning in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8414 Reviewed By: akankshamahajan15 Differential Revision: D29181738 Pulled By: anand1976 fbshipit-source-id: 6e01e9117e1e4c1d67daa4dcee7fa59d06e057a7 17 June 2021, 19:05:40 UTC
back to top