Revision d3d20dcdca9dc79d893a03dfa611fb1055c28f96 authored by Sagar Vemuri on 11 April 2019, 02:24:25 UTC, committed by Facebook Github Bot on 11 April 2019, 02:31:18 UTC
Summary:
Introducing Periodic Compactions.

This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted.  And also, of course, it helps to cleanup data older than certain threshold.

- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).

This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166

Differential Revision: D14884441

Pulled By: sagar0

fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
1 parent ef0fc1b
History
File Mode Size
adaptive_table_factory.cc -rw-r--r-- 4.8 KB
adaptive_table_factory.h -rw-r--r-- 1.8 KB
block.cc -rw-r--r-- 32.2 KB
block.h -rw-r--r-- 19.7 KB
block_based_filter_block.cc -rw-r--r-- 8.1 KB
block_based_filter_block.h -rw-r--r-- 4.7 KB
block_based_filter_block_test.cc -rw-r--r-- 8.3 KB
block_based_table_builder.cc -rw-r--r-- 46.3 KB
block_based_table_builder.h -rw-r--r-- 5.8 KB
block_based_table_factory.cc -rw-r--r-- 24.6 KB
block_based_table_factory.h -rw-r--r-- 8.0 KB
block_based_table_reader.cc -rw-r--r-- 133.7 KB
block_based_table_reader.h -rw-r--r-- 28.9 KB
block_builder.cc -rw-r--r-- 7.1 KB
block_builder.h -rw-r--r-- 2.8 KB
block_fetcher.cc -rw-r--r-- 9.2 KB
block_fetcher.h -rw-r--r-- 3.4 KB
block_prefix_index.cc -rw-r--r-- 7.7 KB
block_prefix_index.h -rw-r--r-- 2.2 KB
block_test.cc -rw-r--r-- 20.4 KB
bloom_block.cc -rw-r--r-- 700 bytes
bloom_block.h -rw-r--r-- 1.0 KB
cleanable_test.cc -rw-r--r-- 6.3 KB
cuckoo_table_builder.cc -rw-r--r-- 19.7 KB
cuckoo_table_builder.h -rw-r--r-- 4.6 KB
cuckoo_table_builder_test.cc -rw-r--r-- 27.3 KB
cuckoo_table_factory.cc -rw-r--r-- 2.6 KB
cuckoo_table_factory.h -rw-r--r-- 2.9 KB
cuckoo_table_reader.cc -rw-r--r-- 13.6 KB
cuckoo_table_reader.h -rw-r--r-- 3.0 KB
cuckoo_table_reader_test.cc -rw-r--r-- 19.8 KB
data_block_footer.cc -rw-r--r-- 1.8 KB
data_block_footer.h -rw-r--r-- 848 bytes
data_block_hash_index.cc -rw-r--r-- 3.1 KB
data_block_hash_index.h -rw-r--r-- 4.9 KB
data_block_hash_index_test.cc -rw-r--r-- 23.5 KB
filter_block.h -rw-r--r-- 5.4 KB
flush_block_policy.cc -rw-r--r-- 3.1 KB
format.cc -rw-r--r-- 15.1 KB
format.h -rw-r--r-- 10.8 KB
full_filter_bits_builder.h -rw-r--r-- 2.8 KB
full_filter_block.cc -rw-r--r-- 7.5 KB
full_filter_block.h -rw-r--r-- 5.1 KB
full_filter_block_test.cc -rw-r--r-- 7.0 KB
get_context.cc -rw-r--r-- 12.0 KB
get_context.h -rw-r--r-- 4.5 KB
index_builder.cc -rw-r--r-- 8.7 KB
index_builder.h -rw-r--r-- 16.7 KB
internal_iterator.h -rw-r--r-- 5.2 KB
iter_heap.h -rw-r--r-- 1.2 KB
iterator.cc -rw-r--r-- 6.6 KB
iterator_wrapper.h -rw-r--r-- 3.2 KB
merger_test.cc -rw-r--r-- 4.5 KB
merging_iterator.cc -rw-r--r-- 12.8 KB
merging_iterator.h -rw-r--r-- 2.1 KB
meta_blocks.cc -rw-r--r-- 19.5 KB
meta_blocks.h -rw-r--r-- 6.2 KB
mock_table.cc -rw-r--r-- 4.4 KB
mock_table.h -rw-r--r-- 5.8 KB
partitioned_filter_block.cc -rw-r--r-- 13.0 KB
partitioned_filter_block.h -rw-r--r-- 4.3 KB
partitioned_filter_block_test.cc -rw-r--r-- 13.2 KB
persistent_cache_helper.cc -rw-r--r-- 4.2 KB
persistent_cache_helper.h -rw-r--r-- 1.5 KB
persistent_cache_options.h -rw-r--r-- 1.0 KB
plain_table_builder.cc -rw-r--r-- 9.7 KB
plain_table_builder.h -rw-r--r-- 4.6 KB
plain_table_factory.cc -rw-r--r-- 8.7 KB
plain_table_factory.h -rw-r--r-- 8.4 KB
plain_table_index.cc -rw-r--r-- 7.3 KB
plain_table_index.h -rw-r--r-- 7.2 KB
plain_table_key_coding.cc -rw-r--r-- 17.4 KB
plain_table_key_coding.h -rw-r--r-- 7.4 KB
plain_table_reader.cc -rw-r--r-- 24.4 KB
plain_table_reader.h -rw-r--r-- 8.4 KB
scoped_arena_iterator.h -rw-r--r-- 1.6 KB
sst_file_reader.cc -rw-r--r-- 2.8 KB
sst_file_reader_test.cc -rw-r--r-- 5.0 KB
sst_file_writer.cc -rw-r--r-- 10.9 KB
sst_file_writer_collectors.h -rw-r--r-- 3.1 KB
table_builder.h -rw-r--r-- 6.2 KB
table_properties.cc -rw-r--r-- 10.4 KB
table_properties_internal.h -rw-r--r-- 1.0 KB
table_reader.h -rw-r--r-- 4.7 KB
table_reader_bench.cc -rw-r--r-- 12.4 KB
table_test.cc -rw-r--r-- 140.6 KB
two_level_iterator.cc -rw-r--r-- 6.3 KB
two_level_iterator.h -rw-r--r-- 1.7 KB

back to top