Revision d3d20dcdca9dc79d893a03dfa611fb1055c28f96 authored by Sagar Vemuri on 11 April 2019, 02:24:25 UTC, committed by Facebook Github Bot on 11 April 2019, 02:31:18 UTC
Summary:
Introducing Periodic Compactions.

This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted.  And also, of course, it helps to cleanup data older than certain threshold.

- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).

This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166

Differential Revision: D14884441

Pulled By: sagar0

fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
1 parent ef0fc1b
History
File Mode Size
advisor
dump
rdb
CMakeLists.txt -rw-r--r-- 544 bytes
Dockerfile -rw-r--r-- 81 bytes
analyze_txn_stress_test.sh -rwxr-xr-x 3.7 KB
auto_sanity_test.sh -rwxr-xr-x 2.7 KB
benchmark.sh -rwxr-xr-x 17.0 KB
benchmark_leveldb.sh -rwxr-xr-x 5.1 KB
blob_dump.cc -rw-r--r-- 3.3 KB
check_format_compatible.sh -rwxr-xr-x 6.5 KB
db_bench.cc -rw-r--r-- 813 bytes
db_bench_tool.cc -rw-r--r-- 220.4 KB
db_bench_tool_test.cc -rw-r--r-- 9.6 KB
db_crashtest.py -rw-r--r-- 14.3 KB
db_repl_stress.cc -rw-r--r-- 4.5 KB
db_sanity_test.cc -rw-r--r-- 8.4 KB
db_stress.cc -rw-r--r-- 144.3 KB
dbench_monitor -rwxr-xr-x 2.6 KB
generate_random_db.sh -rwxr-xr-x 734 bytes
ingest_external_sst.sh -rwxr-xr-x 331 bytes
ldb.cc -rw-r--r-- 572 bytes
ldb_cmd.cc -rw-r--r-- 104.7 KB
ldb_cmd_impl.h -rw-r--r-- 15.6 KB
ldb_cmd_test.cc -rw-r--r-- 3.9 KB
ldb_test.py -rw-r--r-- 26.1 KB
ldb_tool.cc -rw-r--r-- 4.7 KB
pflag -rwxr-xr-x 4.0 KB
reduce_levels_test.cc -rw-r--r-- 5.2 KB
regression_test.sh -rwxr-xr-x 15.8 KB
report_lite_binary_size.sh -rwxr-xr-x 1.2 KB
rocksdb_dump_test.sh -rwxr-xr-x 364 bytes
run_flash_bench.sh -rwxr-xr-x 13.5 KB
run_leveldb.sh -rwxr-xr-x 6.2 KB
sample-dump.dmp -rw-r--r-- 100 bytes
sst_dump.cc -rw-r--r-- 581 bytes
sst_dump_test.cc -rw-r--r-- 7.0 KB
sst_dump_tool.cc -rw-r--r-- 23.3 KB
sst_dump_tool_imp.h -rw-r--r-- 2.9 KB
trace_analyzer.cc -rw-r--r-- 709 bytes
trace_analyzer_test.cc -rw-r--r-- 27.5 KB
trace_analyzer_tool.cc -rw-r--r-- 68.1 KB
trace_analyzer_tool.h -rw-r--r-- 9.8 KB
verify_random_db.sh -rwxr-xr-x 1.0 KB
write_external_sst.sh -rwxr-xr-x 457 bytes
write_stress.cc -rw-r--r-- 10.9 KB
write_stress_runner.py -rw-r--r-- 2.3 KB

back to top