https://github.com/facebook/rocksdb
Revision 4361d6d16380f619833d58225183cbfbb2c7a1dd authored by Baptiste Lemaire on 26 July 2021, 18:55:27 UTC, committed by Facebook GitHub Bot on 26 July 2021, 18:56:29 UTC
Summary: Add `experimental_mempurge_policy` option flag and introduce two new `MemPurge` (Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE. `ALWAYS`: every flush will first go through a `MemPurge` process. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on. `ALWAYS` is designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput). `ALTERNATE`: a flush is transformed into a `MemPurge` except if one of the memtables being flushed is the product of a previous `MemPurge`. `ALTERNATE` is a good tradeoff between reduction in number of L0 SST files created and performance. `ALTERNATE` perform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios. This PR also includes support for `experimental_mempurge_policy` in `db_bench`. Testing was done locally by replacing all the `MemPurge` policies of the unit tests with `ALTERNATE`, as well as local testing with `db_crashtest.py` `whitebox` and `blackbox`. Overall, if an `ALWAYS` mempurge policy passes the tests, there is no reasons why an `ALTERNATE` policy would fail, and therefore the mempurge policy was set to `ALWAYS` for all mempurge unit tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8583 Reviewed By: pdillinger Differential Revision: D29888050 Pulled By: bjlemaire fbshipit-source-id: e2cf26646d66679f6f5fb29842624615610759c1
1 parent daf7e77
Tip revision: 4361d6d16380f619833d58225183cbfbb2c7a1dd authored by Baptiste Lemaire on 26 July 2021, 18:55:27 UTC
Add simple heuristics for experimental mempurge. (#8583)
Add simple heuristics for experimental mempurge. (#8583)
Tip revision: 4361d6d
| File | Mode | Size |
|---|---|---|
| .circleci | ||
| .github | ||
| buckifier | ||
| build_tools | ||
| cache | ||
| cmake | ||
| coverage | ||
| db | ||
| db_stress_tool | ||
| docs | ||
| env | ||
| examples | ||
| file | ||
| fuzz | ||
| hdfs | ||
| include | ||
| java | ||
| logging | ||
| memory | ||
| memtable | ||
| microbench | ||
| monitoring | ||
| options | ||
| plugin | ||
| port | ||
| table | ||
| test_util | ||
| third-party | ||
| tools | ||
| trace_replay | ||
| util | ||
| utilities | ||
| .clang-format | -rw-r--r-- | 138 bytes |
| .gitignore | -rw-r--r-- | 1.0 KB |
| .lgtm.yml | -rw-r--r-- | 67 bytes |
| .travis.yml | -rw-r--r-- | 8.1 KB |
| .watchmanconfig | -rw-r--r-- | 130 bytes |
| AUTHORS | -rw-r--r-- | 322 bytes |
| CMakeLists.txt | -rw-r--r-- | 48.9 KB |
| CODE_OF_CONDUCT.md | -rw-r--r-- | 3.3 KB |
| CONTRIBUTING.md | -rw-r--r-- | 706 bytes |
| COPYING | -rw-r--r-- | 17.7 KB |
| DEFAULT_OPTIONS_HISTORY.md | -rw-r--r-- | 1.5 KB |
| DUMP_FORMAT.md | -rw-r--r-- | 763 bytes |
| HISTORY.md | -rw-r--r-- | 174.7 KB |
| INSTALL.md | -rw-r--r-- | 7.8 KB |
| LANGUAGE-BINDINGS.md | -rw-r--r-- | 1.2 KB |
| LICENSE.Apache | -rw-r--r-- | 11.1 KB |
| LICENSE.leveldb | -rw-r--r-- | 1.5 KB |
| Makefile | -rw-r--r-- | 88.1 KB |
| PLUGINS.md | -rw-r--r-- | 320 bytes |
| README.md | -rw-r--r-- | 2.0 KB |
| ROCKSDB_LITE.md | -rw-r--r-- | 1.0 KB |
| TARGETS | -rw-r--r-- | 57.8 KB |
| USERS.md | -rw-r--r-- | 7.6 KB |
| Vagrantfile | -rw-r--r-- | 1017 bytes |
| WINDOWS_PORT.md | -rw-r--r-- | 12.5 KB |
| appveyor.yml | -rw-r--r-- | 3.3 KB |
| defs.bzl | -rw-r--r-- | 1.9 KB |
| issue_template.md | -rw-r--r-- | 294 bytes |
| src.mk | -rw-r--r-- | 42.1 KB |
| thirdparty.inc | -rw-r--r-- | 7.8 KB |
Computing file changes ...
