Revision d3d20dcdca9dc79d893a03dfa611fb1055c28f96 authored by Sagar Vemuri on 11 April 2019, 02:24:25 UTC, committed by Facebook Github Bot on 11 April 2019, 02:31:18 UTC
Summary:
Introducing Periodic Compactions.

This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted.  And also, of course, it helps to cleanup data older than certain threshold.

- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).

This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166

Differential Revision: D14884441

Pulled By: sagar0

fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
1 parent ef0fc1b
History
File Mode Size
2014-03-27-how-to-backup-rocksdb.markdown -rw-r--r-- 7.6 KB
2014-03-27-how-to-persist-in-memory-rocksdb-database.markdown -rw-r--r-- 3.4 KB
2014-04-02-the-1st-rocksdb-local-meetup-held-on-march-27-2014.markdown -rw-r--r-- 2.1 KB
2014-04-07-rocksdb-2-8-release.markdown -rw-r--r-- 1.6 KB
2014-04-21-indexing-sst-files-for-better-lookup-performance.markdown -rw-r--r-- 3.8 KB
2014-05-14-lock.markdown -rw-r--r-- 9.3 KB
2014-05-19-rocksdb-3-0-release.markdown -rw-r--r-- 662 bytes
2014-05-22-rocksdb-3-1-release.markdown -rw-r--r-- 577 bytes
2014-06-23-plaintable-a-new-file-format.markdown -rw-r--r-- 3.0 KB
2014-06-27-avoid-expensive-locks-in-get.markdown -rw-r--r-- 4.7 KB
2014-06-27-rocksdb-3-2-release.markdown -rw-r--r-- 1.0 KB
2014-07-29-rocksdb-3-3-release.markdown -rw-r--r-- 1.6 KB
2014-09-12-cuckoo.markdown -rw-r--r-- 4.4 KB
2014-09-12-new-bloom-filter-format.markdown -rw-r--r-- 3.1 KB
2014-09-15-rocksdb-3-5-release.markdown -rw-r--r-- 1.3 KB
2015-01-16-migrating-from-leveldb-to-rocksdb-2.markdown -rw-r--r-- 3.1 KB
2015-02-24-reading-rocksdb-options-from-a-file.markdown -rw-r--r-- 1.7 KB
2015-02-27-write-batch-with-index.markdown -rw-r--r-- 3.8 KB
2015-04-22-integrating-rocksdb-with-mongodb-2.markdown -rw-r--r-- 902 bytes
2015-06-12-rocksdb-in-osquery.markdown -rw-r--r-- 502 bytes
2015-07-15-rocksdb-2015-h2-roadmap.markdown -rw-r--r-- 3.0 KB
2015-07-17-spatial-indexing-in-rocksdb.markdown -rw-r--r-- 4.8 KB
2015-07-22-rocksdb-is-now-available-in-windows-platform.markdown -rw-r--r-- 1.0 KB
2015-07-23-dynamic-level.markdown -rw-r--r-- 4.3 KB
2015-10-27-getthreadlist.markdown -rw-r--r-- 5.0 KB
2015-11-10-use-checkpoints-for-efficient-snapshots.markdown -rw-r--r-- 2.0 KB
2015-11-16-analysis-file-read-latency-by-level.markdown -rw-r--r-- 12.1 KB
2016-01-29-compaction_pri.markdown -rw-r--r-- 6.9 KB
2016-02-24-rocksdb-4-2-release.markdown -rw-r--r-- 1.3 KB
2016-02-25-rocksdb-ama.markdown -rw-r--r-- 731 bytes
2016-03-07-rocksdb-options-file.markdown -rw-r--r-- 1.2 KB
2016-04-26-rocksdb-4-5-1-released.markdown -rw-r--r-- 2.8 KB
2016-07-26-rocksdb-4-8-released.markdown -rw-r--r-- 3.5 KB
2016-09-28-rocksdb-4-11-2-released.markdown -rw-r--r-- 2.2 KB
2017-01-06-rocksdb-5-0-1-released.markdown -rw-r--r-- 2.0 KB
2017-02-07-rocksdb-5-1-2-released.markdown -rw-r--r-- 662 bytes
2017-02-17-bulkoad-ingest-sst-file.markdown -rw-r--r-- 2.3 KB
2017-03-02-rocksdb-5-2-1-released.markdown -rw-r--r-- 1.3 KB
2017-05-12-partitioned-index-filter.markdown -rw-r--r-- 3.6 KB
2017-05-14-core-local-stats.markdown -rw-r--r-- 5.4 KB
2017-05-26-rocksdb-5-4-5-released.markdown -rw-r--r-- 2.4 KB
2017-06-26-17-level-based-changes.markdown -rw-r--r-- 5.6 KB
2017-06-29-rocksdb-5-5-1-released.markdown -rw-r--r-- 1.0 KB
2017-07-25-rocksdb-5-6-1-released.markdown -rw-r--r-- 1.9 KB
2017-08-24-pinnableslice.markdown -rw-r--r-- 2.2 KB
2017-08-25-flushwal.markdown -rw-r--r-- 2.9 KB
2017-09-28-rocksdb-5-8-released.markdown -rw-r--r-- 1.9 KB
2017-12-18-17-auto-tuned-rate-limiter.markdown -rw-r--r-- 2.3 KB
2017-12-19-write-prepared-txn.markdown -rw-r--r-- 5.6 KB
2018-02-05-rocksdb-5-10-2-released.markdown -rw-r--r-- 1.5 KB
2018-08-01-rocksdb-tuning-advisor.markdown -rw-r--r-- 2.8 KB
2018-08-23-data-block-hash-index.markdown -rw-r--r-- 9.0 KB
2018-11-21-delete-range.markdown -rw-r--r-- 15.5 KB
2019-03-08-format-version-4.markdown -rw-r--r-- 2.7 KB

back to top