Revision d3d20dcdca9dc79d893a03dfa611fb1055c28f96 authored by Sagar Vemuri on 11 April 2019, 02:24:25 UTC, committed by Facebook Github Bot on 11 April 2019, 02:31:18 UTC
Summary:
Introducing Periodic Compactions.

This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted.  And also, of course, it helps to cleanup data older than certain threshold.

- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).

This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166

Differential Revision: D14884441

Pulled By: sagar0

fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
1 parent ef0fc1b
History
File Mode Size
backupablejni.cc -rw-r--r-- 11.7 KB
backupenginejni.cc -rw-r--r-- 8.8 KB
cassandra_compactionfilterjni.cc -rw-r--r-- 980 bytes
cassandra_value_operator.cc -rw-r--r-- 1.5 KB
checkpoint.cc -rw-r--r-- 2.3 KB
clock_cache.cc -rw-r--r-- 1.4 KB
columnfamilyhandle.cc -rw-r--r-- 2.5 KB
compact_range_options.cc -rw-r--r-- 6.4 KB
compaction_filter.cc -rw-r--r-- 1007 bytes
compaction_filter_factory.cc -rw-r--r-- 1.3 KB
compaction_filter_factory_jnicallback.cc -rw-r--r-- 2.5 KB
compaction_filter_factory_jnicallback.h -rw-r--r-- 1.2 KB
compaction_job_info.cc -rw-r--r-- 6.6 KB
compaction_job_stats.cc -rw-r--r-- 10.4 KB
compaction_options.cc -rw-r--r-- 3.4 KB
compaction_options_fifo.cc -rw-r--r-- 2.5 KB
compaction_options_universal.cc -rw-r--r-- 6.4 KB
comparator.cc -rw-r--r-- 2.0 KB
comparatorjnicallback.cc -rw-r--r-- 9.7 KB
comparatorjnicallback.h -rw-r--r-- 3.2 KB
compression_options.cc -rw-r--r-- 4.8 KB
env.cc -rw-r--r-- 6.7 KB
env_options.cc -rw-r--r-- 8.0 KB
filter.cc -rw-r--r-- 1.4 KB
ingest_external_file_options.cc -rw-r--r-- 6.3 KB
iterator.cc -rw-r--r-- 5.7 KB
jnicallback.cc -rw-r--r-- 1.6 KB
jnicallback.h -rw-r--r-- 897 bytes
loggerjnicallback.cc -rw-r--r-- 8.6 KB
loggerjnicallback.h -rw-r--r-- 1.6 KB
lru_cache.cc -rw-r--r-- 1.6 KB
memory_util.cc -rw-r--r-- 3.6 KB
memtablejni.cc -rw-r--r-- 3.4 KB
merge_operator.cc -rw-r--r-- 2.6 KB
native_comparator_wrapper_test.cc -rw-r--r-- 1.3 KB
optimistic_transaction_db.cc -rw-r--r-- 9.6 KB
optimistic_transaction_options.cc -rw-r--r-- 2.4 KB
options.cc -rw-r--r-- 199.4 KB
options_util.cc -rw-r--r-- 4.3 KB
persistent_cache.cc -rw-r--r-- 1.8 KB
portal.h -rw-r--r-- 224.9 KB
ratelimiterjni.cc -rw-r--r-- 4.2 KB
remove_emptyvalue_compactionfilterjni.cc -rw-r--r-- 897 bytes
restorejni.cc -rw-r--r-- 1.3 KB
rocks_callback_object.cc -rw-r--r-- 1.3 KB
rocksdb_exception_test.cc -rw-r--r-- 2.5 KB
rocksjni.cc -rw-r--r-- 95.2 KB
slice.cc -rw-r--r-- 11.2 KB
snapshot.cc -rw-r--r-- 898 bytes
sst_file_manager.cc -rw-r--r-- 8.3 KB
sst_file_writerjni.cc -rw-r--r-- 8.9 KB
statistics.cc -rw-r--r-- 7.7 KB
statisticsjni.cc -rw-r--r-- 1001 bytes
statisticsjni.h -rw-r--r-- 1.0 KB
table.cc -rw-r--r-- 6.5 KB
table_filter.cc -rw-r--r-- 873 bytes
table_filter_jnicallback.cc -rw-r--r-- 2.2 KB
table_filter_jnicallback.h -rw-r--r-- 1.0 KB
thread_status.cc -rw-r--r-- 4.1 KB
trace_writer.cc -rw-r--r-- 813 bytes
trace_writer_jnicallback.cc -rw-r--r-- 3.5 KB
trace_writer_jnicallback.h -rw-r--r-- 1.1 KB
transaction.cc -rw-r--r-- 57.9 KB
transaction_db.cc -rw-r--r-- 15.8 KB
transaction_db_options.cc -rw-r--r-- 5.5 KB
transaction_log.cc -rw-r--r-- 2.7 KB
transaction_notifier.cc -rw-r--r-- 1.5 KB
transaction_notifier_jnicallback.cc -rw-r--r-- 1.3 KB
transaction_notifier_jnicallback.h -rw-r--r-- 1.5 KB
transaction_options.cc -rw-r--r-- 6.1 KB
ttl.cc -rw-r--r-- 6.4 KB
wal_filter.cc -rw-r--r-- 780 bytes
wal_filter_jnicallback.cc -rw-r--r-- 4.6 KB
wal_filter_jnicallback.h -rw-r--r-- 1.4 KB
write_batch.cc -rw-r--r-- 19.7 KB
write_batch_test.cc -rw-r--r-- 6.5 KB
write_batch_with_index.cc -rw-r--r-- 26.1 KB
write_buffer_manager.cc -rw-r--r-- 1.4 KB
writebatchhandlerjnicallback.cc -rw-r--r-- 15.1 KB
writebatchhandlerjnicallback.h -rw-r--r-- 3.3 KB

back to top