https://github.com/facebook/rocksdb
Revision 4361d6d16380f619833d58225183cbfbb2c7a1dd authored by Baptiste Lemaire on 26 July 2021, 18:55:27 UTC, committed by Facebook GitHub Bot on 26 July 2021, 18:56:29 UTC
Summary: Add `experimental_mempurge_policy` option flag and introduce two new `MemPurge` (Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE. `ALWAYS`: every flush will first go through a `MemPurge` process. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on. `ALWAYS` is designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput). `ALTERNATE`: a flush is transformed into a `MemPurge` except if one of the memtables being flushed is the product of a previous `MemPurge`. `ALTERNATE` is a good tradeoff between reduction in number of L0 SST files created and performance. `ALTERNATE` perform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios. This PR also includes support for `experimental_mempurge_policy` in `db_bench`. Testing was done locally by replacing all the `MemPurge` policies of the unit tests with `ALTERNATE`, as well as local testing with `db_crashtest.py` `whitebox` and `blackbox`. Overall, if an `ALWAYS` mempurge policy passes the tests, there is no reasons why an `ALTERNATE` policy would fail, and therefore the mempurge policy was set to `ALWAYS` for all mempurge unit tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8583 Reviewed By: pdillinger Differential Revision: D29888050 Pulled By: bjlemaire fbshipit-source-id: e2cf26646d66679f6f5fb29842624615610759c1
1 parent daf7e77
Tip revision: 4361d6d16380f619833d58225183cbfbb2c7a1dd authored by Baptiste Lemaire on 26 July 2021, 18:55:27 UTC
Add simple heuristics for experimental mempurge. (#8583)
Add simple heuristics for experimental mempurge. (#8583)
Tip revision: 4361d6d
File | Mode | Size |
---|---|---|
.circleci | ||
.github | ||
buckifier | ||
build_tools | ||
cache | ||
cmake | ||
coverage | ||
db | ||
db_stress_tool | ||
docs | ||
env | ||
examples | ||
file | ||
fuzz | ||
hdfs | ||
include | ||
java | ||
logging | ||
memory | ||
memtable | ||
microbench | ||
monitoring | ||
options | ||
plugin | ||
port | ||
table | ||
test_util | ||
third-party | ||
tools | ||
trace_replay | ||
util | ||
utilities | ||
.clang-format | -rw-r--r-- | 138 bytes |
.gitignore | -rw-r--r-- | 1.0 KB |
.lgtm.yml | -rw-r--r-- | 67 bytes |
.travis.yml | -rw-r--r-- | 8.1 KB |
.watchmanconfig | -rw-r--r-- | 130 bytes |
AUTHORS | -rw-r--r-- | 322 bytes |
CMakeLists.txt | -rw-r--r-- | 48.9 KB |
CODE_OF_CONDUCT.md | -rw-r--r-- | 3.3 KB |
CONTRIBUTING.md | -rw-r--r-- | 706 bytes |
COPYING | -rw-r--r-- | 17.7 KB |
DEFAULT_OPTIONS_HISTORY.md | -rw-r--r-- | 1.5 KB |
DUMP_FORMAT.md | -rw-r--r-- | 763 bytes |
HISTORY.md | -rw-r--r-- | 174.7 KB |
INSTALL.md | -rw-r--r-- | 7.8 KB |
LANGUAGE-BINDINGS.md | -rw-r--r-- | 1.2 KB |
LICENSE.Apache | -rw-r--r-- | 11.1 KB |
LICENSE.leveldb | -rw-r--r-- | 1.5 KB |
Makefile | -rw-r--r-- | 88.1 KB |
PLUGINS.md | -rw-r--r-- | 320 bytes |
README.md | -rw-r--r-- | 2.0 KB |
ROCKSDB_LITE.md | -rw-r--r-- | 1.0 KB |
TARGETS | -rw-r--r-- | 57.8 KB |
USERS.md | -rw-r--r-- | 7.6 KB |
Vagrantfile | -rw-r--r-- | 1017 bytes |
WINDOWS_PORT.md | -rw-r--r-- | 12.5 KB |
appveyor.yml | -rw-r--r-- | 3.3 KB |
defs.bzl | -rw-r--r-- | 1.9 KB |
issue_template.md | -rw-r--r-- | 294 bytes |
src.mk | -rw-r--r-- | 42.1 KB |
thirdparty.inc | -rw-r--r-- | 7.8 KB |
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...