Skip to main content

Browse the archive

Revision d3d20dcdca9dc79d893a03dfa611fb1055c28f96 authored by Sagar Vemuri on 11 April 2019, 02:24:25 UTC, committed by Facebook Github Bot on 11 April 2019, 02:31:18 UTC 
Periodic Compactions (#5166)
 
Summary:
Introducing Periodic Compactions.

This feature allows all the files in a CF to be periodically compacted. It could help in catching any corruptions that could creep into the DB proactively as every file is constantly getting re-compacted.  And also, of course, it helps to cleanup data older than certain threshold.

- Introduced a new option `periodic_compaction_time` to control how long a file can live without being compacted in a CF.
- This works across all levels.
- The files are put in the same level after going through the compaction. (Related files in the same level are picked up as `ExpandInputstoCleanCut` is used).
- Compaction filters, if any, are invoked as usual.
- A new table property, `file_creation_time`, is introduced to implement this feature. This property is set to the time at which the SST file was created (and that time is given by the underlying Env/OS).

This feature can be enabled on its own, or in conjunction with `ttl`. It is possible to set a different time threshold for the bottom level when used in conjunction with ttl. Since `ttl` works only on 0 to last but one levels, you could set `ttl` to, say, 1 day, and `periodic_compaction_time` to, say, 7 days. Since `ttl < periodic_compaction_time` all files in last but one levels keep getting picked up based on ttl, and almost never based on periodic_compaction_time. The files in the bottom level get picked up for compaction based on `periodic_compaction_time`.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/5166

Differential Revision: D14884441

Pulled By: sagar0

fbshipit-source-id: 408426cbacb409c06386a98632dcf90bfa1bda47
1 parent ef0fc1b
Raw File
Vagrantfile 
# Vagrant file
Vagrant.configure("2") do |config|

  config.vm.provider "virtualbox" do |v|
    v.memory = 4096
    v.cpus = 2
  end

  config.vm.define "ubuntu14" do |box|
    box.vm.box = "ubuntu/trusty64"
  end

  config.vm.define "centos65" do |box|
    box.vm.box = "chef/centos-6.5"
  end

  config.vm.define "centos7" do |box|
    box.vm.box = "centos/7"
    box.vm.provision "shell", path: "build_tools/setup_centos7.sh"
  end

  config.vm.define "FreeBSD10" do |box|
    box.vm.guest = :freebsd
    box.vm.box = "robin/freebsd-10"
    # FreeBSD does not support 'mount_virtualbox_shared_folder', use NFS
    box.vm.synced_folder ".", "/vagrant", :nfs => true, id: "vagrant-root"
    box.vm.network "private_network", ip: "10.0.1.10"

    # build everything after creating VM, skip using --no-provision
    box.vm.provision "shell", inline: <<-SCRIPT
      pkg install -y gmake clang35
      export CXX=/usr/local/bin/clang++35
      cd /vagrant
      gmake clean
      gmake all OPT=-g
    SCRIPT
  end

end
back to top