https://github.com/facebook/rocksdb
- HEAD
- refs/heads/2.2.fb.branch
- refs/heads/2.3.fb.branch
- refs/heads/2.4.fb.branch
- refs/heads/2.5.fb.branch
- refs/heads/2.6.fb.branch
- refs/heads/2.7
- refs/heads/2.7.fb.branch
- refs/heads/2.8.1.fb
- refs/heads/2.8.fb
- refs/heads/2.8.fb.trunk
- refs/heads/3.0.fb
- refs/heads/3.0.fb.branch
- refs/heads/3.1.fb
- refs/heads/3.10.fb
- refs/heads/3.11.fb
- refs/heads/3.12.fb
- refs/heads/3.13.fb
- refs/heads/3.2.fb
- refs/heads/3.3.fb
- refs/heads/3.4.fb
- refs/heads/3.5.fb
- refs/heads/3.6.fb
- refs/heads/3.7.fb
- refs/heads/3.8.fb
- refs/heads/3.9.fb
- refs/heads/4.0.fb
- refs/heads/4.1.fb
- refs/heads/4.10.fb
- refs/heads/4.11.fb
- refs/heads/4.12.fb
- refs/heads/4.13.fb
- refs/heads/4.2.fb
- refs/heads/4.3.fb
- refs/heads/4.4.fb
- refs/heads/4.5.fb
- refs/heads/4.6.fb
- refs/heads/4.7.fb
- refs/heads/4.8.fb
- refs/heads/4.9.fb
- refs/heads/5.0.fb
- refs/heads/5.1.fb
- refs/heads/5.10.fb
- refs/heads/5.11.fb
- refs/heads/5.12.fb
- refs/heads/5.13.fb
- refs/heads/5.13.fb.myrocks
- refs/heads/5.14.fb
- refs/heads/5.14.fb.myrocks
- refs/heads/5.15.fb
- refs/heads/5.16.fb
- refs/heads/5.17.fb
- refs/heads/5.17.fb.myrocks
- refs/heads/5.18.fb
- refs/heads/5.2.fb
- refs/heads/5.3.fb
- refs/heads/5.4.fb
- refs/heads/5.5.fb
- refs/heads/5.6.fb
- refs/heads/5.7.fb
- refs/heads/5.7.fb.myrocks
- refs/heads/5.8.3
- refs/heads/5.8.fb
- refs/heads/5.9.fb
- refs/heads/5.9.fb.myrocks
- refs/heads/6.0.fb
- refs/heads/6.0.fb.myrocks
- refs/heads/6.1.fb
- refs/heads/6.1.fb.myrocks
- refs/heads/6.1.fb.prod201905
- refs/heads/6.10.fb
- refs/heads/6.11.fb
- refs/heads/6.12.fb
- refs/heads/6.13.fb
- refs/heads/6.13.fb.laser
- refs/heads/6.14.fb
- refs/heads/6.14.fb.laser
- refs/heads/6.15.fb
- refs/heads/6.16.fb
- refs/heads/6.17.fb
- refs/heads/6.17.fb.laser
- refs/heads/6.18.fb
- refs/heads/6.19.fb
- refs/heads/6.2.fb
- refs/heads/6.20.fb
- refs/heads/6.21.fb
- refs/heads/6.22-history.md-fixup
- refs/heads/6.22.fb
- refs/heads/6.23.fb
- refs/heads/6.24.fb
- refs/heads/6.25.fb
- refs/heads/6.26.fb
- refs/heads/6.27.fb
- refs/heads/6.28.fb
- refs/heads/6.29.fb
- refs/heads/6.3.fb
- refs/heads/6.3.fb.myrocks
- refs/heads/6.3.fb.myrocks2
- refs/heads/6.3fb
- refs/heads/6.4.fb
- refs/heads/6.5.fb
- refs/heads/6.6.fb
- refs/heads/6.7.fb
- refs/heads/6.8.fb
- refs/heads/6.9.fb
- refs/heads/7.0.fb
- refs/heads/7.1.fb
- refs/heads/7.10.fb
- refs/heads/7.2.fb
- refs/heads/7.3.fb
- refs/heads/7.4.fb
- refs/heads/7.5.fb
- refs/heads/7.6.fb
- refs/heads/7.7.fb
- refs/heads/7.8.fb
- refs/heads/7.9.fb
- refs/heads/8.0.fb
- refs/heads/8.1.fb
- refs/heads/8.10.fb
- refs/heads/8.11.2_zippydb
- refs/heads/8.11.fb
- refs/heads/8.11.fb_zippydb
- refs/heads/8.2.fb
- refs/heads/8.3.fb
- refs/heads/8.4.fb
- refs/heads/8.5.fb
- refs/heads/8.6.fb
- refs/heads/8.7.fb
- refs/heads/8.8.fb
- refs/heads/8.9.fb
- refs/heads/9.0.fb
- refs/heads/9.1.fb
- refs/heads/9.1.fb.myrocks
- refs/heads/9.2.fb
- refs/heads/9.3.fb
- refs/heads/9.4.fb
- refs/heads/adaptive
- refs/heads/ajkr-patch-1
- refs/heads/ajkr-patch-2
- refs/heads/blob_shadow
- refs/heads/bottom-pri-level
- refs/heads/bugfix-build-detect
- refs/heads/checksum_readahead_mmap_fix
- refs/heads/draft-myrocks-and-fbcode-8.0.fb
- refs/heads/feature/debug-rocksdbjavastatic
- refs/heads/feature/travis-arm64
- refs/heads/fix-release-notes
- refs/heads/fix-win2022-build
- refs/heads/fix-write-batch-comment
- refs/heads/format_compatible_4
- refs/heads/getmergeops
- refs/heads/gh-pages-old
- refs/heads/history-update
- refs/heads/hotfix/lambda-capture
- refs/heads/improve-support
- refs/heads/jijiew-patch-1
- refs/heads/katherinez-patch-1
- refs/heads/katherinez-patch-2
- refs/heads/main
- refs/heads/master
- refs/heads/mdcallag_benchmark_oct22
- refs/heads/nvm_cache_proto
- refs/heads/pr-sanity-check-as-GHAction
- refs/heads/pr/11267
- refs/heads/pr/6062
- refs/heads/ramvadiv-patch-1
- refs/heads/release_fix
- refs/heads/revert-10606-7.6.1
- refs/heads/ribbon_bloom_hybrid
- refs/heads/scaffold
- refs/heads/siying-patch-1
- refs/heads/siying-patch-10
- refs/heads/siying-patch-2
- refs/heads/siying-patch-3
- refs/heads/siying-patch-4
- refs/heads/siying-patch-5
- refs/heads/siying-patch-6
- refs/heads/siying-patch-7
- refs/heads/siying-patch-8
- refs/heads/skip_memtable_flush
- refs/heads/testing_ppc_build
- refs/heads/tests
- refs/heads/unschedule_issue_test_base
- refs/heads/unused-var
- refs/heads/v6.6.4
- refs/heads/xxhash_merge_base
- refs/heads/yiwu_stackable
- refs/heads/yuslepukhin
- refs/remotes/origin/5.13.fb
- refs/tags/2.5.fb
- refs/tags/2.6.fb
- refs/tags/3.0.fb
- refs/tags/do-not-use-me2
- refs/tags/rocksdb-3.1
- refs/tags/rocksdb-3.10.2
- refs/tags/rocksdb-3.11
- refs/tags/rocksdb-3.11.1
- refs/tags/rocksdb-3.11.2
- refs/tags/rocksdb-3.2
- refs/tags/rocksdb-3.3
- refs/tags/rocksdb-3.4
- refs/tags/rocksdb-3.5
- refs/tags/rocksdb-3.5.1
- refs/tags/rocksdb-3.6.1
- refs/tags/rocksdb-3.6.2
- refs/tags/rocksdb-3.7
- refs/tags/rocksdb-3.8
- refs/tags/rocksdb-3.9
- refs/tags/rocksdb-3.9.1
- refs/tags/rocksdb-4.1
- refs/tags/rocksdb-5.10.2
- refs/tags/rocksdb-5.10.3
- refs/tags/rocksdb-5.10.4
- refs/tags/rocksdb-5.11.2
- refs/tags/rocksdb-5.11.3
- refs/tags/rocksdb-5.14.3
- refs/tags/rocksdb-5.2.1
- refs/tags/rocksdb-5.3.3
- refs/tags/rocksdb-5.3.4
- refs/tags/rocksdb-5.3.5
- refs/tags/rocksdb-5.3.6
- refs/tags/rocksdb-5.4.10
- refs/tags/rocksdb-5.4.5
- refs/tags/rocksdb-5.4.6
- refs/tags/rocksdb-5.5.2
- refs/tags/rocksdb-5.5.3
- refs/tags/rocksdb-5.5.4
- refs/tags/rocksdb-5.5.5
- refs/tags/rocksdb-5.5.6
- refs/tags/rocksdb-5.6.1
- refs/tags/rocksdb-5.6.2
- refs/tags/rocksdb-5.7.1
- refs/tags/rocksdb-5.7.2
- refs/tags/rocksdb-5.7.3
- refs/tags/rocksdb-5.7.5
- refs/tags/rocksdb-5.8.6
- refs/tags/rocksdb-5.8.7
- refs/tags/rocksdb-5.8.8
- refs/tags/rocksdb-5.9.2
- refs/tags/v4.0
- refs/tags/v4.1
- refs/tags/v5.10.2
- refs/tags/v5.10.3
- refs/tags/v5.10.4
- refs/tags/v5.11.2
- refs/tags/v5.11.3
- refs/tags/v5.13.3
- refs/tags/v5.14.3
- refs/tags/v5.15.10
- refs/tags/v5.18.3
- refs/tags/v5.2.1
- refs/tags/v5.3.3
- refs/tags/v5.3.4
- refs/tags/v5.3.5
- refs/tags/v5.3.6
- refs/tags/v5.4.10
- refs/tags/v5.4.5
- refs/tags/v5.4.6
- refs/tags/v5.5.2
- refs/tags/v5.5.3
- refs/tags/v5.5.4
- refs/tags/v5.5.5
- refs/tags/v5.5.6
- refs/tags/v5.6.1
- refs/tags/v5.6.2
- refs/tags/v5.7.1
- refs/tags/v5.7.2
- refs/tags/v5.7.3
- refs/tags/v5.7.5
- refs/tags/v5.8.6
- refs/tags/v5.8.7
- refs/tags/v5.8.8
- refs/tags/v5.9.2
- refs/tags/v6.0.1
- refs/tags/v6.0.2
- refs/tags/v6.1.1
- refs/tags/v6.1.2
- refs/tags/v6.10.1
- refs/tags/v6.10.2
- refs/tags/v6.11.4
- refs/tags/v6.11.6
- refs/tags/v6.12.6
- refs/tags/v6.12.7
- refs/tags/v6.13.2
- refs/tags/v6.13.3
- refs/tags/v6.14.5
- refs/tags/v6.14.6
- refs/tags/v6.15.4
- refs/tags/v6.15.5
- refs/tags/v6.16.3
- refs/tags/v6.16.4
- refs/tags/v6.17.3
- refs/tags/v6.2.2
- refs/tags/v6.2.4
- refs/tags/v6.20.3
- refs/tags/v6.22.1
- refs/tags/v6.25.3
- refs/tags/v6.26.1
- refs/tags/v6.28.2
- refs/tags/v6.29.3
- refs/tags/v6.29.4
- refs/tags/v6.29.5
- refs/tags/v6.3.6
- refs/tags/v6.4.6
- refs/tags/v6.5.2
- refs/tags/v6.5.3
- refs/tags/v6.6.3
- refs/tags/v6.6.4
- refs/tags/v6.7.3
- refs/tags/v6.8.1
- refs/tags/v7.0.1
- refs/tags/v7.0.2
- refs/tags/v7.0.4
- refs/tags/v7.2.0
- refs/tags/v7.2.2
- refs/tags/v7.5.3
- refs/tags/v7.7.2
- refs/tags/v7.9.2
- refs/tags/v8.0.0
- refs/tags/v8.11.4
- refs/tags/v8.3.2
- refs/tags/v8.3.3
- refs/tags/v8.4.4
- refs/tags/v8.5.3
- refs/tags/v8.6.7
- refs/tags/v8.7.3
- refs/tags/v9.0.1
- refs/tags/v9.1.1
- refs/tags/v9.2.1
- refs/tags/v9.3.1
- refs/tags/v9.4.0
- v9.1.0
- v9.0.0
- v8.9.1
- v8.8.1
- v8.5.4
- v8.11.3
- v8.10.2
- v8.10.0
- v8.1.1
- v7.8.3
- v7.7.8
- v7.7.3
- v7.6.0
- v7.4.5
- v7.4.4
- v7.4.3
- v7.3.1
- v7.10.2
- v7.1.2
- v7.1.1
- v7.0.3
- v6.27.3
- v6.26.0
- v6.25.1
- v6.24.2
- v6.23.3
- v6.23.2
- v6.19.3
- v6.15.2
- v5.8
- v5.5.1
- v5.4.7
- v5.18.4
- v5.17.2
- v5.16.6
- v5.14.2
- v5.13.4
- v5.13.2
- v5.13.1
- v5.12.5
- v5.12.4
- v5.12.3
- v5.12.2
- v5.1.4
- v5.1.3
- v5.1.2
- v5.0.2
- v5.0.1
- v4.9
- v4.8
- v4.6.1
- v4.5.1
- v4.4.1
- v4.4
- v4.3.1
- v4.3
- v4.2
- v4.13.5
- v4.13
- v4.11.2
- v3.9
- v3.8
- v3.7
- v3.6.1
- v3.5
- v3.4
- v3.3
- v3.2
- v3.13.1
- v3.13
- v3.12.1
- v3.12
- v3.11
- v3.10
- v3.1
- v3.0
- v2.8
- v2.7
- v2.6
- v2.5
- v2.4
- v2.3
- v2.2
- v2.1
- v2.0
- v1.5.9.1
- v1.5.8.2
- v1.5.8.1
- v1.5.8
- v1.5.7
- rocksdb-5.8
- rocksdb-5.4.7
- rocksdb-5.1.4
- rocksdb-5.1.3
- rocksdb-5.1.2
- rocksdb-5.0.2
- rocksdb-5.0.1
- rocksdb-4.9
- rocksdb-4.8
- rocksdb-4.6.1
- rocksdb-4.5.1
- rocksdb-4.4.1
- rocksdb-4.4
- rocksdb-4.3.1
- rocksdb-4.3
- rocksdb-4.2
- rocksdb-4.13.5
- rocksdb-4.13
- rocksdb-4.11.2
- rocksdb-3.10.1
- blob_st_lvl-pre
- 2.8.fb
- 2.7.fb
- 2.4.fb
- 2.3.fb
- 2.2.fb
- 2.1.fb
- 2.0.fb
- 1.5.9.fb
- 1.5.9.2.fb
- 1.5.9.1.fb
- 1.5.8.fb
- 1.5.8.2.fb
- 1.5.8.1.fb
- 1.5.7.fb
Take a new snapshot of a software origin
If the archived software origin currently browsed is not synchronized with its upstream version (for instance when new commits have been issued), you can explicitly request Software Heritage to take a new snapshot of it.
Use the form below to proceed. Once a request has been submitted and accepted, it will be processed as soon as possible. You can then check its processing state by visiting this dedicated page.Processing "take a new snapshot" request ...
Permalinks
To reference or cite the objects present in the Software Heritage archive, permalinks based on SoftWare Hash IDentifiers (SWHIDs) must be used.
Select below a type of object currently browsed in order to display its associated SWHID and permalink.
Revision | Author | Date | Message | Commit Date |
---|---|---|---|---|
0ac4a21 | Peter Dillinger | 25 March 2020, 17:35:48 UTC | Update check_format_compatible.sh for default format_version=4 Summary: And add releases that should have been added before (6.6 - 6.8) Test Plan: facebook automated test (so much easier than running myself) | 25 March 2020, 17:42:33 UTC |
93b80ca | Peter Dillinger | 25 March 2020, 04:19:56 UTC | Update default BBTO::format_version from 2 to 4 (#6582) Summary: Version 4 has been around long enough, for compatibility and extensive validation, that it should be default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6582 Test Plan: CI (w.r.t. changing the default; format_version=4 is well tested and massively in production at Facebook) Reviewed By: siying Differential Revision: D20625233 Pulled By: pdillinger fbshipit-source-id: 2f83ed874cffa4a39bc7a66cdf3833b978fbb948 | 25 March 2020, 04:22:21 UTC |
ccf7676 | Yanqin Jin | 25 March 2020, 03:57:53 UTC | Update a few scripts to be python3 compatible (#6525) Summary: There are a few scripts with python3 compatibility issues that were not detected by automated tool before. Update them now. Test Plan (devserver): python2 tools/ldb_test.py python3 tools/ldb_test.py python2 tools/write_stress_runner.py --runtime_sec=30 python3 tools/write_stress_runner.py --runtime_sec=30 python2 tools/db_crashtest.py --simple --interval=2 --duration=10 blackbox python3 tools/db_crashtest.py --simple --interval=2 --duration=10 blackbox python2 tools/db_crashtest.py --simple --duration=10 --random_kill_odd=1000 --ops_per_thread=1000 whitebox python3 tools/db_crashtest.py --simple --duration=10 --random_kill_odd=1000 --ops_per_thread=1000 whitebox Pull Request resolved: https://github.com/facebook/rocksdb/pull/6525 Reviewed By: cheng-chang Differential Revision: D20627820 Pulled By: riversand963 fbshipit-source-id: 4b25a7bd4d001c7f868be8b640ef876523be6ca3 | 25 March 2020, 04:00:27 UTC |
6fd0ed4 | sdong | 25 March 2020, 03:20:33 UTC | CompactRange() to use bottom pool when goes to bottommost level (#6593) Summary: In automatic compaction, if a compaction is bottommost, it goes to bottom thread pool. We should do the same for manual compaction too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6593 Test Plan: Add a unit test. See all existing tests pass. Reviewed By: ajkr Differential Revision: D20637408 fbshipit-source-id: cb03031e8f895085f7acf6d2d65e69e84c9ddef3 | 25 March 2020, 03:24:32 UTC |
ceeca75 | akankshamahajan | 24 March 2020, 18:24:53 UTC | Create a thread in DeleteScheduler only when rate limit enabled (#6564) Summary: Create a thread in DeleteScheduler only when delete rate limit is set because when there is no rate limit on deletion, a thread per DeleteScheduler consumes unnecessary resources. Test Plan: make -j64 check Reviewed By: riversand963 Differential Revision: D20538138 Pulled By: akankshamahajan15 fbshipit-source-id: 137499e810e817156345c30d627f8678b9adadf7 | 24 March 2020, 18:29:51 UTC |
a6ce5c8 | Huisheng Liu | 24 March 2020, 18:21:10 UTC | multiget support for timestamps (#6483) Summary: Add timestamp support for MultiGet(). timestamp from readoptions is honored, and timestamps can be returned along with values. MultiReadRandom perf test (10 minutes) on the same development machine ram drive with the same DB data shows no regression (within marge of error). The test is adapted from https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks. base line (commit 17bef7d3a): multireadrandom : 104.173 micros/op 307167 ops/sec; (5462999 of 5462999 found) This PR: multireadrandom : 104.199 micros/op 307095 ops/sec; (5307999 of 5307999 found) .\db_bench --db=r:\rocksdb.github --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --cache_size=2147483648 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=r:\rocksdb.github\WAL_LOG --sync=0 --verify_checksum=1 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --duration=600 --benchmarks=multireadrandom --use_existing_db=1 --num=25000000 --threads=32 --allow_concurrent_memtable_write=0 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6483 Reviewed By: anand1976 Differential Revision: D20498373 Pulled By: riversand963 fbshipit-source-id: 8505f22bc40fd791bc7dd05e48d7e67c91edb627 | 24 March 2020, 18:24:09 UTC |
921cdd3 | sdong | 24 March 2020, 17:15:10 UTC | Fix bug that number of table loading threads is set as a boolean (#6576) Summary: When applying a new version in non DB open case, optimize_filters_for_hits is used for max_threads, which is clearly a bug. It is not clear what the indented value in the first place, but it value 1 makes sense here, which would create no extra threads. This bug is not expected to cause user visible problems, assuming C++ implicitly cast bool to 0 or 1. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6576 Test Plan: Run all exsiting test. Reviewed By: ajkr Differential Revision: D20602467 fbshipit-source-id: 40b2cd8619aba09ae9242b36c415464db3c9b737 | 24 March 2020, 17:17:40 UTC |
a9d168c | anand76 | 24 March 2020, 04:50:42 UTC | Simplify migration to FileSystem API (#6552) Summary: The current Env/FileSystem API separation has a couple of issues - 1. It requires the user to specify 2 options - ```Options::env``` and ```Options::file_system``` - which means they have to make code changes to benefit from the new APIs. Furthermore, there is a risk of accessing the same APIs in two different ways, through Env in the old way and through FileSystem in the new way. The two may not always match, for example, if env is ```PosixEnv``` and FileSystem is a custom implementation. Any stray RocksDB calls to env will use the ```PosixEnv``` implementation rather than the file_system implementation. 2. There needs to be a simple way for the FileSystem developer to instantiate an Env for backward compatibility purposes. This PR solves the above issues and simplifies the migration in the following ways - 1. Embed a shared_ptr to the ```FileSystem``` in the ```Env```, and remove ```Options::file_system``` as a configurable option. This way, no code changes will be required in application code to benefit from the new API. The default Env constructor uses a ```LegacyFileSystemWrapper``` as the embedded ```FileSystem```. 1a. - This also makes it more robust by ensuring that even if RocksDB has some stray calls to Env APIs rather than FileSystem, they will go through the same object and thus there is no risk of getting out of sync. 2. Provide a ```NewCompositeEnv()``` API that can be used to construct a PosixEnv with a custom FileSystem implementation. This eliminates an indirection to call Env APIs, and relieves the FileSystem developer of the burden of having to implement wrappers for the Env APIs. 3. Add a couple of missing FileSystem APIs - ```SanitizeEnvOptions()``` and ```NewLogger()``` Tests: 1. New unit tests 2. make check and make asan_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6552 Reviewed By: riversand963 Differential Revision: D20592038 Pulled By: anand1976 fbshipit-source-id: c3801ad4153f96d21d5a3ae26c92ba454d1bf1f7 | 24 March 2020, 04:54:21 UTC |
43aee93 | Cheng Chang | 24 March 2020, 03:12:38 UTC | Initialize scratch to nullptr explicitly to make clang analyzer happy (#6577) Summary: `scratch` is not initialized in `Align` because it will be set outside of it. But clang analyzer is strict on initializing it before return. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6577 Test Plan: make analyze Reviewed By: siying Differential Revision: D20607303 Pulled By: cheng-chang fbshipit-source-id: 2843d759345a057a8e122178d30b90deff0f9b2a | 24 March 2020, 03:15:27 UTC |
d300d10 | Zhichao Cao | 24 March 2020, 01:47:34 UTC | Fix the MultiGet testing failure in Circleci (#6578) Summary: The MultiGet test in db_basic_test fails in CircleCI vs2019. The reason is that even Snappy compression is enabled, the first compression type is still kNoCompression. This PR checks the list and ensure that only when compression is enable and the compression type is valid, compression will be enabled. Such that, it will not fail the combined read test in MultiGet. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6578 Test Plan: make check, db_basic_test. Reviewed By: anand1976 Differential Revision: D20607529 Pulled By: zhichao-cao fbshipit-source-id: dcead264d5c2da105912c18caad34b8510bb04b0 | 24 March 2020, 01:51:09 UTC |
617f479 | Yanqin Jin | 23 March 2020, 19:59:04 UTC | Fix LITE build (#6575) Summary: Fix LITE build by excluding some unit tests that use features not supported in LITE. ``` db/db_basic_test.cc:1778:8: error: ‘void rocksdb::{anonymous}::TableFileListener::OnTableFileCreated(const rocksdb::TableFileCreationInfo&)’ marked ‘override’, but does not override void OnTableFileCreated(const TableFileCreationInfo& info) override { ^~~~~~~~~~~~~~~~~~ make: *** [db/db_basic_test.o] Error 1 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6575 Reviewed By: ltamasi Differential Revision: D20598598 Pulled By: riversand963 fbshipit-source-id: 367f7cb2500360ad57030b138a94c0f731a04339 | 23 March 2020, 20:05:36 UTC |
5c6346c | Zhichao Cao | 23 March 2020, 17:23:42 UTC | Revert "Added the safe-to-ignore tag to version_edit (#6530)" (#6569) Summary: This reverts commit e10553f2a6c5d89ee17a7b44f7b9a6125ae5c735. Pass make asan_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6569 Reviewed By: riversand963 Differential Revision: D20574319 Pulled By: zhichao-cao fbshipit-source-id: ce36981a21596f5f2e14da6a59a2bb3619509a8b | 23 March 2020, 17:27:47 UTC |
fb09ef0 | Yanqin Jin | 21 March 2020, 02:17:54 UTC | Attempt to recover from db with missing table files (#6334) Summary: There are situations when RocksDB tries to recover, but the db is in an inconsistent state due to SST files referenced in the MANIFEST being missing. In this case, previous RocksDB will just fail the recovery and return a non-ok status. This PR enables another possibility. During recovery, RocksDB checks possible MANIFEST files, and try to recover to the most recent state without missing table file. `VersionSet::Recover()` applies version edits incrementally and "materializes" a version only when this version does not reference any missing table file. After processing the entire MANIFEST, the version created last will be the latest version. `DBImpl::Recover()` calls `VersionSet::Recover()`. Afterwards, WAL replay will *not* be performed. To use this capability, set `options.best_efforts_recovery = true` when opening the db. Best-efforts recovery is currently incompatible with atomic flush. Test plan (on devserver): ``` $make check $COMPILE_WITH_ASAN=1 make all && make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6334 Reviewed By: anand1976 Differential Revision: D19778960 Pulled By: riversand963 fbshipit-source-id: c27ea80f29bc952e7d3311ecf5ee9c54393b40a8 | 21 March 2020, 02:30:48 UTC |
4fc2166 | Cheng Chang | 20 March 2020, 23:15:40 UTC | Support direct IO in RandomAccessFileReader::MultiRead (#6446) Summary: By supporting direct IO in RandomAccessFileReader::MultiRead, the benefits of parallel IO (IO uring) and direct IO can be combined. In direct IO mode, read requests are aligned and merged together before being issued to RandomAccessFile::MultiRead, so blocks in the original requests might share the same underlying buffer, the shared buffers are returned in `aligned_bufs`, which is a new parameter of the `MultiRead` API. For example, suppose alignment requirement for direct IO is 4KB, one request is (offset: 1KB, len: 1KB), another request is (offset: 3KB, len: 1KB), then since they all belong to page (offset: 0, len: 4KB), `MultiRead` only reads the page with direct IO into a buffer on heap, and returns 2 Slices referencing regions in that same buffer. See `random_access_file_reader_test.cc` for more examples. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6446 Test Plan: Added a new test `random_access_file_reader_test.cc`. Reviewed By: anand1976 Differential Revision: D20097518 Pulled By: cheng-chang fbshipit-source-id: ca48a8faf9c3af146465c102ef6b266a363e78d1 | 20 March 2020, 23:33:26 UTC |
5fd152b | Cheng Chang | 20 March 2020, 22:19:00 UTC | Get block size only in direct IO mode (#6522) Summary: When `use_direct_reads` and `use_direct_writes` are `false`, `logical_sector_size_` inside various `*File` implementations are not actually used, so `GetLogicalBlockSize` does not necessarily need to be called for `logical_sector_size_`, just set a default page size. This is a follow up PR for https://github.com/facebook/rocksdb/pull/6457. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6522 Test Plan: make check Reviewed By: siying Differential Revision: D20408885 Pulled By: cheng-chang fbshipit-source-id: f2d3808f41265237e7fa2c0be9f084f8fa97fe3d | 20 March 2020, 22:26:10 UTC |
6c50fe1 | sdong | 20 March 2020, 21:42:47 UTC | Change HashMap::Insert()'s value to a const reference (#6567) Summary: When building RocksDB on VS2015, an error shows up with hash_map.h(39): error C2719: 'value': formal parameter with requested alignment of 8 won't be aligned Making the reference a reference can solve the problem, and there isn't a reason we can't do that, at least for the current use of the hash map. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6567 Test Plan: See CI tests pass. Reviewed By: pdillinger Differential Revision: D20548543 fbshipit-source-id: 255b55d74cf68a0b324e6f504c56608a97ea6276 | 20 March 2020, 21:59:54 UTC |
66cd07c | Peter Dillinger | 20 March 2020, 20:14:10 UTC | Exclude more Travis builds for each pull request (#6557) Summary: This commit fixes an incorrect version of this change that was previously landed. On recently adding ARM64 and PPC64LE builds to Travis, we seem to have hit some parallel build limits that dramatically increased queue times. This change majorly limits the configurations for ARM64 and PPC64LE to build on each pull request, but keeps the large matrix for branch builds. In the process, I changed some previously excluded osx build configurations to happen in branch builds. NB: we might want to move master branch Travis build to daily trigger rather than push trigger to further reduce contention. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6557 Test Plan: Travis only Reviewed By: siying Differential Revision: D20563425 Pulled By: pdillinger fbshipit-source-id: d619eb9f196486ed000364aa40de4661f0b1029d | 20 March 2020, 20:20:19 UTC |
d2e3822 | Ben Mehne | 20 March 2020, 17:51:48 UTC | Make testpilot recognize that these tests have coverage instrumentation Summary: TestPilot uses two flags to determine whether coverage is already instrumented: `fbcode_macros` and `coverage`. Normally, these two tags are added automatically to cpp tests, but this is a fake cpp test, so we must manually add them. The first is easy - `fbcode_macros` is added by the `custom_unittest` library, which is in `fbcode_macros`, so it is appropriate. The second is harder - we need to verify that we should add the macro. We do this using the `coverage.bzl` functions. Reviewed By: siying Differential Revision: D20549040 fbshipit-source-id: d2732b3ec26f3dff065efdf398abe3241075bb2f | 20 March 2020, 18:23:23 UTC |
093ff0b | Peter Dillinger | 19 March 2020, 19:49:58 UTC | Exclude more Travis builds for each pull request (#6557) Summary: On recently adding ARM64 and PPC64LE builds to Travis, we seem to have hit some parallel build limits that dramatically increased queue times. This change majorly limits the configurations for ARM64 and PPC64LE to build on each pull request, but keeps the large matrix for branch builds. In the process, I changed some previously excluded osx build configurations to happen in branch builds. NB: we might want to move master branch Travis build to daily trigger rather than push trigger to further reduce contention. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6557 Test Plan: Travis only Reviewed By: siying Differential Revision: D20524575 Pulled By: pdillinger fbshipit-source-id: babcb2c64e195679e472473a1cbdf42de47231ff | 19 March 2020, 19:53:37 UTC |
e10553f | Zhichao Cao | 19 March 2020, 18:25:50 UTC | Added the safe-to-ignore tag to version_edit (#6530) Summary: Each time RocksDB switches to a new MANIFEST file from old one, it calls WriteCurrentStateToManifest() which writes a 'snapshot' of the current in-memory state of versions to the beginning of the new manifest as a bunch of version edits. We can distinguish these version edits from other version edits written during normal operations with a custom, safe-to-ignore tag. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6530 Test Plan: added test to version_edit_test, pass make asan_check Reviewed By: riversand963 Differential Revision: D20524516 Pulled By: zhichao-cao fbshipit-source-id: f1de102f5499bfa88dae3caa2f32c7f42cf904db | 19 March 2020, 18:30:26 UTC |
4424045 | Levi Tamasi | 19 March 2020, 17:40:31 UTC | Clean up VersionBuilder a bit (#6556) Summary: The whole point of the pimpl idiom is to hide implementation details. Internal helper methods like `CheckConsistency`, `CheckConsistencyForDeletes`, and `MaybeAddFile` do not belong in the public interface of the class. In addition, the patch switches to `unique_ptr` for the implementation object instead of using a raw `delete`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6556 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D20523568 Pulled By: ltamasi fbshipit-source-id: 5bbb0ccebd0c47a33b815398c7f9cfe13bd775ac | 19 March 2020, 17:44:16 UTC |
217ce20 | Levi Tamasi | 19 March 2020, 00:11:06 UTC | Remove GetSortedWalFiles/GetCurrentWalFile from the crash test (#6491) Summary: Currently, `db_stress` tests a randomly picked one of `GetLiveFiles`, `GetSortedWalFiles`, and `GetCurrentWalFile` with a 1/N chance when the command line parameter `get_live_files_and_wal_files_one_in` is specified. The problem is that `GetSortedWalFiles` and `GetCurrentWalFile` are unreliable in the sense that they can return errors if another thread removes a WAL file while they are executing (which is a perfectly plausible and legitimate scenario). The patch splits this command line parameter into three (one for each API), and changes the crash test script so that only `GetLiveFiles` is tested during our continuous crash test runs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6491 Test Plan: ``` make check python tools/db_crashtest.py whitebox ``` Reviewed By: siying Differential Revision: D20312200 Pulled By: ltamasi fbshipit-source-id: e7c3481eddfe3bd3d5349476e34abc9eee5b7dc8 | 19 March 2020, 00:14:15 UTC |
8ad4b32 | sdong | 18 March 2020, 17:57:46 UTC | cmake: add option WITH_CORE_TOOLS to exclude tools except ldb and sst_dump (#6506) Summary: ldb and sst_dump are most important tools and they don't dependend on gflags. In cmake, we don't have an way to only build these two tools and exclude other tools. This is inconvenient if the environment has a problem with gflags. Add such an option WITH_CORE_TOOLS. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6506 Test Plan: cmake and build with WITH_TOOLS and without. Differential Revision: D20473029 fbshipit-source-id: 3d730fd14bbae6eeeae7f9cc9aec50a4e488ad72 | 18 March 2020, 18:01:38 UTC |
1df9b01 | Levi Tamasi | 18 March 2020, 16:21:00 UTC | Disable distributed mutex test for valgrind_test (#6553) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6553 Test Plan: ``` $ make valgrind_test -j24 $ ./folly_synchronization_distributed_mutex_test DistributedMutex is not supported in ROCKSDB_LITE, on ARM, or in valgrind_test runs ``` Reviewed By: pdillinger Differential Revision: D20501966 Pulled By: ltamasi fbshipit-source-id: 386ec5f258f89d0781a36c5b390c665787093a74 | 18 March 2020, 16:24:31 UTC |
712bc4b | sdong | 17 March 2020, 19:28:46 UTC | Fix regression bug in partitioned index reseek caused by #6531 (#6551) Summary: https://github.com/facebook/rocksdb/pull/6531 removed some code in partitioned index seek logic. By mistake the logic of storing previous index offset is removed, while the logic of using it is preserved, so that the code might use wrong value to determine reseeking condition. This will trigger a bug, if following a Seek() not going to the last block, SeekToLast() is called, and then Seek() is called which should position the cursor to the block before SeekToLast(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6551 Test Plan: Add a unit test that reproduces the bug. In the same unit test, also some reseek cases are covered to avoid regression. Reviewed By: pdillinger Differential Revision: D20493990 fbshipit-source-id: 3919aa4861c0481ec96844e053048da1a934b91d | 17 March 2020, 19:33:10 UTC |
a8149ae | akankshamahajan | 17 March 2020, 17:59:18 UTC | Allow table/sst_file_reader_test.cc to use custom Env (#6536) Summary: Allowing table/sst_file_reader_test.cc to use custom Env specified by TEST_ENV_URI. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6536 Reviewed By: riversand963 Differential Revision: D20448525 Pulled By: akankshamahajan15 fbshipit-source-id: 74e4d34c8ac4c2743741e78bf599571a4a465459 | 17 March 2020, 18:02:13 UTC |
66ed580 | Yanqin Jin | 17 March 2020, 17:47:36 UTC | Reduce runtime of db_with_timestamp_basic_test (#6546) Summary: Reduce runtime by reducing test scale to avoid test time-outs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6546 Test Plan: time ./db_with_timestamp_basic_test and watch internal tests. Reviewed By: zhichao-cao Differential Revision: D20479292 Pulled By: riversand963 fbshipit-source-id: c9e4a155be7699dd4de60fa531de86d442a3ba0a | 17 March 2020, 17:50:48 UTC |
098dce2 | Yanqin Jin | 17 March 2020, 16:56:15 UTC | Fix compiler warning treated as error (#6547) Summary: Define a private member variable only in debug mode. Without fix, build will fail ``` In file included from table/block_based/partitioned_index_iterator.cc:9: ./table/block_based/partitioned_index_iterator.h:125:32: error: private field 'icomp_' is not used [-Werror,-Wunused-private-field] const InternalKeyComparator& icomp_; ``` Test plan (dev server) 1. make check 2. Make sure fixed in Travis Pull Request resolved: https://github.com/facebook/rocksdb/pull/6547 Reviewed By: siying Differential Revision: D20480027 Pulled By: pdillinger fbshipit-source-id: 288bc94280e240c3136335b6c73eb1ccb0db459d | 17 March 2020, 16:59:28 UTC |
6c595f0 | Peter Dillinger | 17 March 2020, 02:04:23 UTC | Update folly/lang/Align.h (backport to C++11) (#6534) Summary: For s390x support, some updates in newer version of Align.h are needed. Upgrading just that file as best we can, with one addition to Portability.h and tweaking new code in Align.h to use C++11 only (no non-trivial constexpr functions). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6534 Test Plan: CI, further work in PR https://github.com/facebook/rocksdb/issues/6168 Differential Revision: D20445942 Pulled By: pdillinger fbshipit-source-id: 0cef3c367463c71f3123d12cdf287c573af5e342 | 17 March 2020, 02:07:31 UTC |
db02664 | Peter Dillinger | 16 March 2020, 23:57:03 UTC | Remove XXH3(preview) streaming APIs (#6540) Summary: There was an alignment bug in our copy of the streaming APIs for XXH3 (which we dubbed "XXH3p" for "preview" release). Since those APIs are unused and some values for XXH3 have changed since XXH3p, I'm simply removing those APIs, expecting it's better to use finalized XXH3 function if/when we decide to use those APIs (e.g. for checksums). Fixes https://github.com/facebook/rocksdb/issues/6508 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6540 Test Plan: make check Differential Revision: D20479271 Pulled By: pdillinger fbshipit-source-id: 246cf1690d614d3b31042b563d249de32dec1e0d | 17 March 2020, 00:02:00 UTC |
58918d4 | Yanqin Jin | 16 March 2020, 23:54:29 UTC | Use correct Env for DestroyDB in stress test (#6539) Summary: When using custom Env, trying to call DestroyDB() with default Options will fail. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6539 Test Plan: ./db_stress Differential Revision: D20476204 Pulled By: riversand963 fbshipit-source-id: 612c6754660cc9b5bb3e9c2dbb2f6ecd7f648797 | 16 March 2020, 23:57:48 UTC |
488b1e6 | sdong | 16 March 2020, 20:47:28 UTC | Fix an error in db_bench with gcc 4.8 (#6537) Summary: I start to see following failures: tools/db_bench_tool.cc: In constructor ‘rocksdb::NormalDistribution::NormalDistribution(unsigned int, unsigned int)’: tools/db_bench_tool.cc:1528:58: error: declaration of ‘max’ shadows a member of 'this' [-Werror=shadow] NormalDistribution(unsigned int min, unsigned int max) : ^ tools/db_bench_tool.cc:1528:58: error: declaration of ‘min’ shadows a member of 'this' [-Werror=shadow] tools/db_bench_tool.cc: In constructor ‘rocksdb::UniformDistribution::UniformDistribution(unsigned int, unsigned int)’: tools/db_bench_tool.cc:1546:59: error: declaration of ‘max’ shadows a member of 'this' [-Werror=shadow] UniformDistribution(unsigned int min, unsigned int max) : ^ tools/db_bench_tool.cc:1546:59: error: declaration of ‘min’ shadows a member of 'this' [-Werror=shadow] when I build from GCC 4.8. Rename those variables to fix the problem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6537 Test Plan: make all with the compiler that used to show the failure. Differential Revision: D20448741 fbshipit-source-id: 18bcf012dbe020f22f79038a9b08f447befa2574 | 16 March 2020, 20:50:40 UTC |
d669080 | sdong | 16 March 2020, 19:17:34 UTC | De-template block based table iterator (#6531) Summary: Right now block based table iterator is used as both of iterating data for block based table, and for the index iterator for partitioend index. This was initially convenient for introducing a new iterator and block type for new index format, while reducing code change. However, these two usage doesn't go with each other very well. For example, Prev() is never called for partitioned index iterator, and some other complexity is maintained in block based iterators, which is not needed for index iterator but maintainers will always need to reason about it. Furthermore, the template usage is not following Google C++ Style which we are following, and makes a large chunk of code tangled together. This commit separate the two iterators. Right now, here is what it is done: 1. Copy the block based iterator code into partitioned index iterator, and de-template them. 2. Remove some code not needed for partitioned index. The upper bound check and tricks are removed. We never tested performance for those tricks when partitioned index is enabled in the first place. It's unlikelyl to generate performance regression, as creating new partitioned index block is much rarer than data blocks. 3. Separate out the prefetch logic to a helper class and both classes call them. This commit will enable future follow-ups. One direction is that we might separate index iterator interface for data blocks and index blocks, as they are quite different. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6531 Test Plan: build using make and cmake. And build release Differential Revision: D20473108 fbshipit-source-id: e48011783b339a4257c204cc07507b171b834b0f | 16 March 2020, 19:20:50 UTC |
402da45 | Cheng Chang | 14 March 2020, 04:51:26 UTC | Migrate AppVeyor to CircleCI (#6518) Summary: CircleCI is the new recommended CI system internally. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6518 Test Plan: Watch https://app.circleci.com/pipelines/github/facebook/rocksdb Differential Revision: D20454743 Pulled By: cheng-chang fbshipit-source-id: 39031568d6c1d3d25b7fbd78fa9a0e6067ddc47c | 14 March 2020, 04:58:51 UTC |
23eae14 | Cheng Chang | 14 March 2020, 04:51:16 UTC | Destroy DB at the end of each test in db_logical_block_size_cache_test (#6532) Summary: If DB is not deleted, in concurrent test, the tests might fail because of the previously existing DB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6532 Test Plan: make clean && make -j24 LITE=1 db_logical_block_size_cache_test && ./db_logical_block_size_cache_test make clean && make -j24 db_logical_block_size_cache_test && ./db_logical_block_size_cache_test Differential Revision: D20454734 Pulled By: cheng-chang fbshipit-source-id: 8abede2ec1d79c1a4fe1bc95fbda489f8f7ee052 | 14 March 2020, 04:53:38 UTC |
a824727 | Zhichao Cao | 13 March 2020, 23:45:28 UTC | Fix build bug caused by PR 6516 (#6535) Summary: Fix the build corruption caused by PR https://github.com/facebook/rocksdb/issues/6516 Testing plan: make make asan_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6535 Differential Revision: D20448614 Pulled By: zhichao-cao fbshipit-source-id: 4d2a3dae6cdd781fcfe8e28a84ac3f536db1b067 | 13 March 2020, 23:48:03 UTC |
85dbdf2 | Peter Dillinger | 13 March 2020, 20:36:45 UTC | Use an Amazon S3 bucket for downloading deps (#6526) Summary: After we had a lot of failures with maven.org downloads, we wanted an alternative location for downloading binary dependencies. Hosting them through github would have been good in terms of organizational and network dependencies, but that approach seems to be awkward (fake releases, so would need a 'rocksdb-deps' repo) and strangely complicated for Facebook policy on open source repositories. This commit moves the downloads (that are not officially hosted by others on github) from my personal rocksdb fork to an S3 bucket owned by the Facebook RocksDB AWS account. Facebook employees can access this through an internal tool, and we should be able to grant permission to outside collaborators. Assuming this works out, I will back-port to older branches to stabilize their CI testing as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6526 Test Plan: CI Differential Revision: D20430130 Pulled By: pdillinger fbshipit-source-id: df52394a65e0a57942db3039bdaade8a4d520cb2 | 13 March 2020, 20:39:03 UTC |
5c30e6c | Zhichao Cao | 13 March 2020, 18:33:04 UTC | Separate timestamp related test from db_basic_test (#6516) Summary: In some of the test, db_basic_test may cause time out due to its long running time. Separate the timestamp related test from db_basic_test to avoid the potential issue. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6516 Test Plan: pass make asan_check Differential Revision: D20423922 Pulled By: zhichao-cao fbshipit-source-id: d6306f89a8de55b07bf57233e4554c09ef1fe23a | 13 March 2020, 18:37:15 UTC |
674cf41 | sdong | 13 March 2020, 04:39:36 UTC | Divide block_based_table_reader.cc (#6527) Summary: block_based_table_reader.cc is a giant file, which makes it hard for users to navigate the code. Divide the files to multiple files. Some class templates cannot be moved to .cc file. They are moved to .h files. It is still better than including them all in block_based_table_reader.cc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6527 Test Plan: "make all check" and "make release". Also build using cmake. Differential Revision: D20428455 fbshipit-source-id: ca713c698469f07f35bc0c271358c0874ed4eb28 | 13 March 2020, 04:41:50 UTC |
dd7a4a8 | Yuqi Gu | 13 March 2020, 03:58:26 UTC | CI: add Arm support to travis CI matrix (#6436) Summary: This patch based on https://github.com/facebook/rocksdb/issues/5932 offers a better solution to add arm64 to TravisCI matrix. Really thank adamretter for initiating Arm CI setup. Difference comparing to amd64: 1. For CMake, as no official arm64 release ready on Kitware page, a third party (conda-forge) released one is used instead of building from source. The main reason is to save CI time. 2. Explicit export JAVA_HOME on arm64 3. Disable mingw test Signed-off-by: Yuqi Gu <yuqi.gu@arm.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6436 Differential Revision: D20428505 Pulled By: pdillinger fbshipit-source-id: 81ef02435e41480bb71710b783d85ebf452ce926 | 13 March 2020, 04:01:20 UTC |
a8851f2 | Ben Mehne | 13 March 2020, 00:45:04 UTC | Fix coverage for internal_repo_rocksdb Summary: tcc gtest runner need to know the location of the binary in order to collect coverage. We can give them the location in an environment variable. Note that all these tests will break in tpx currently, though this is a bug in rocksdb's wrapper script, not tpx. Reviewed By: siying Differential Revision: D20430043 fbshipit-source-id: c77d5f70bbc28f6011c6f91906bce2ceecc2f167 | 13 March 2020, 00:48:16 UTC |
2ccb794 | Cheng Chang | 12 March 2020, 21:27:51 UTC | Use DestroyColumnFamilyHandle instead of directly deleting column family handle (#6505) Summary: Update example usage of closing column family. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6505 Test Plan: cd examples && make column_families_example && ./column_families_example Differential Revision: D20362100 Pulled By: cheng-chang fbshipit-source-id: 493c5e0068a40b4f237f8f8511cddd22dc15ea5c | 12 March 2020, 21:30:46 UTC |
0d2c8e4 | Cheng Chang | 12 March 2020, 21:10:47 UTC | OpenForReadOnly is not supported in LITE mode (#6523) Summary: In DBLogicalBlockSizeCacheTest, do not test OpenForReadOnly in LITE mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6523 Test Plan: watch test for LITE mode Differential Revision: D20420321 Pulled By: cheng-chang fbshipit-source-id: e45bf6f2800206d6f8ce9af7308e76a08de80643 | 12 March 2020, 21:13:59 UTC |
0772768 | Adam Retter | 12 March 2020, 19:22:03 UTC | Force Java version on Travis CI (#6512) Summary: In the `.travis.yml` file the `jdk: openjdk7` element is ignored when `language: cpp`. So whatever version of the JDK that was installed in the Travis container was used - typically JDK 11. To ensure our RocksJava builds are working, we now instead install and use OpenJDK 8. Ideally we would use OpenJDK 7, as RocksJava supports Java 7, but many of the newer Travis containers don't support Java 7, so Java 8 is the next best thing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6512 Differential Revision: D20388296 Pulled By: pdillinger fbshipit-source-id: 8bbe6b59b70cfab7fe81ff63867d907fefdd2df1 | 12 March 2020, 19:24:51 UTC |
c15e85b | Levi Tamasi | 12 March 2020, 17:58:27 UTC | Move BlobDB related files under db/ to db/blob/ (#6519) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6519 Test Plan: ``` make all make check ``` Differential Revision: D20400691 Pulled By: ltamasi fbshipit-source-id: 20ef911cf1c2c92c7f71ef0b493f9be64f2eef94 | 12 March 2020, 18:00:56 UTC |
07a3f7f | Huisheng Liu | 12 March 2020, 15:39:46 UTC | fix MSVC build failures (#6517) Summary: fix a few build warnings that are treated as failures with more strict MSVC warning settings Pull Request resolved: https://github.com/facebook/rocksdb/pull/6517 Differential Revision: D20401325 Pulled By: pdillinger fbshipit-source-id: b44979dfaafdc7b3b8cb44a565400a99b331dd30 | 12 March 2020, 15:42:39 UTC |
6dea753 | Cheng Chang | 12 March 2020, 04:35:41 UTC | Remove copy of pairs from the for range loop (#6514) Summary: Remove copy of pairs from the for range loop Pull Request resolved: https://github.com/facebook/rocksdb/pull/6514 Test Plan: make check Differential Revision: D20389688 Pulled By: cheng-chang fbshipit-source-id: 1c772091f955be33267514010f3596c61a6f46b5 | 12 March 2020, 04:38:09 UTC |
2d9efc9 | Cheng Chang | 12 March 2020, 01:36:43 UTC | Cache result of GetLogicalBufferSize in Linux (#6457) Summary: In Linux, when reopening DB with many SST files, profiling shows that 100% system cpu time spent for a couple of seconds for `GetLogicalBufferSize`. This slows down MyRocks' recovery time when site is down. This PR introduces two new APIs: 1. `Env::RegisterDbPaths` and `Env::UnregisterDbPaths` lets `DB` tell the env when it starts or stops using its database directories . The `PosixFileSystem` takes this opportunity to set up a cache from database directories to the corresponding logical block sizes. 2. `LogicalBlockSizeCache` is defined only for OS_LINUX to cache the logical block sizes. Other modifications: 1. rename `logical buffer size` to `logical block size` to be consistent with Linux terms. 2. declare `GetLogicalBlockSize` in `PosixHelper` to expose it to `PosixFileSystem`. 3. change the functions `IOError` and `IOStatus` in `env/io_posix.h` to have external linkage since they are used in other translation units too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6457 Test Plan: 1. A new unit test is added for `LogicalBlockSizeCache` in `env/io_posix_test.cc`. 2. A new integration test is added for `DB` operations related to the cache in `db/db_logical_block_size_cache_test.cc`. `make check` Differential Revision: D20131243 Pulled By: cheng-chang fbshipit-source-id: 3077c50f8065c0bffb544d8f49fb10bba9408d04 | 12 March 2020, 01:40:05 UTC |
331e619 | sdong | 11 March 2020, 23:20:13 UTC | Include more information in file lock failure (#6507) Summary: When users fail to open a DB with file lock failure, it is sometimes hard for users to debug. We now include the time the lock is acquired and the thread ID that acquired the lock, to help users debug problems like this. Default Env's thread ID is used. Since type of lockedFiles is changed, rename it to follow naming convention too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6507 Test Plan: Add a unit test and improve an existing test to validate the case. Differential Revision: D20378333 fbshipit-source-id: 312fe0e9733fd1d1e9969c321b90ce523cf4708a | 11 March 2020, 23:23:08 UTC |
37a635c | Levi Tamasi | 11 March 2020, 21:41:34 UTC | Disambiguate CustomFieldTags for the unity build (#6513) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6513 Test Plan: `make unity_test` Differential Revision: D20388919 Pulled By: ltamasi fbshipit-source-id: 88dbceab0723a54ee3939e1644e13dc9a4c70420 | 11 March 2020, 21:45:12 UTC |
8fc20ac | Adam Retter | 11 March 2020, 19:31:06 UTC | Add ppc64le builds to Travis (#6144) Summary: Let's see how this goes... Pull Request resolved: https://github.com/facebook/rocksdb/pull/6144 Differential Revision: D20387515 Pulled By: pdillinger fbshipit-source-id: ba2669348c267141dfddff910b4c2224a22cbb38 | 11 March 2020, 19:33:45 UTC |
65b60db | Adam Retter | 11 March 2020, 18:43:23 UTC | Update to latest Snappy to fix compilation issue on latest MacOS XCode (#6496) Summary: * **macOS version:** 10.15.2 (Catalina) * **XCode/Clang version:** Apple clang version 11.0.0 (clang-1100.0.33.16) Before this bugfix the error generated is: ``` In file included from ./util/compression.h:23: ./snappy-1.1.7/snappy.h:76:59: error: unknown type name 'string'; did you mean 'std::string'? size_t Compress(const char* input, size_t input_length, string* output); ^~~~~~ std::string /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iosfwd:211:65: note: 'std::string' declared here typedef basic_string<char, char_traits<char>, allocator<char> > string; ^ In file included from db/builder.cc:10: In file included from ./db/builder.h:12: In file included from ./db/range_tombstone_fragmenter.h:15: In file included from ./db/pinned_iterators_manager.h:12: In file included from ./table/internal_iterator.h:13: In file included from ./table/format.h:25: In file included from ./options/cf_options.h:14: In file included from ./util/compression.h:23: ./snappy-1.1.7/snappy.h:85:19: error: unknown type name 'string'; did you mean 'std::string'? string* uncompressed); ^~~~~~ std::string /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/iosfwd:211:65: note: 'std::string' declared here typedef basic_string<char, char_traits<char>, allocator<char> > string; ^ 2 errors generated. make: *** [jls/db/builder.o] Error 1 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6496 Differential Revision: D20389254 Pulled By: pdillinger fbshipit-source-id: 2864245c8d0dba7b2ab81294241a62f2adf02e20 | 11 March 2020, 18:46:13 UTC |
00c4ab0 | Adam Retter | 11 March 2020, 15:50:20 UTC | When CMake fails to download a file, display the error message (#6511) Summary: This helps to diagnose errors in the CMake build where it tries to retrieve dependencies. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6511 Differential Revision: D20387392 Pulled By: pdillinger fbshipit-source-id: 7028dfd62704bcc747f39ff864ea9c9bf51cd1be | 11 March 2020, 15:52:46 UTC |
f5bc3b9 | Levi Tamasi | 11 March 2020, 00:24:38 UTC | Split BlobFileState into an immutable and a mutable part (#6502) Summary: It's never too soon to refactor something. The patch splits the recently introduced (`VersionEdit` related) `BlobFileState` into two classes `BlobFileAddition` and `BlobFileGarbage`. The idea is that once blob files are closed, they are immutable, and the only thing that changes is the amount of garbage in them. In the new design, `BlobFileAddition` contains the immutable attributes (currently, the count and total size of all blobs, checksum method, and checksum value), while `BlobFileGarbage` contains the mutable GC-related information elements (count and total size of garbage blobs). This is a better fit for the GC logic and is more consistent with how SST files are handled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6502 Test Plan: `make check` Differential Revision: D20348352 Pulled By: ltamasi fbshipit-source-id: ff93f0121e80ab15e0e0a6525ba0d6af16a0e008 | 11 March 2020, 00:27:26 UTC |
4028eba | Chao Zhao | 10 March 2020, 20:37:37 UTC | Optional sequence number exporting during checkpoint creation (#5528) Summary: Add sequence_number_ptr to the checkpoint interface to expose the sequence number during taking the checkpoint. The number will be consistent with the seq # in rocksdb log. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5528 Test Plan: make check -j64 Reviewed By: Winger1994 Differential Revision: D16080209 fbshipit-source-id: 6dc3c7680287ee97d673c5e61f89aae1f43e33df | 10 March 2020, 20:40:18 UTC |
fd1da22 | Yanqin Jin | 10 March 2020, 01:42:27 UTC | Support options.max_open_files != -1 with FIFO compaction (#6503) Summary: Allow user to specify options.max_open_files != -1 with FIFO compaction. If max_open_files != -1, not all table files are kept open. In the past, FIFO style compaction requires all table files to be open in order to read file creation time from table properties. Later, we added file creation time to MANIFEST, making it possible to read file creation time without opening file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6503 Test Plan: make check Differential Revision: D20353758 Pulled By: riversand963 fbshipit-source-id: ba5c61a648419e47e9ef6d74e0e280e3ee24f296 | 10 March 2020, 01:45:06 UTC |
d93812c | Yanqin Jin | 07 March 2020, 00:21:03 UTC | Iterator with timestamp (#6255) Summary: Preliminary support for iterator with user timestamp. Current implementation does not consider merge operator and reverse iterator. Auto compaction is also disabled in unit tests. Create an iterator with timestamp. ``` ... read_opts.timestamp = &ts; auto* iter = db->NewIterator(read_opts); // target is key without timestamp. for (iter->Seek(target); iter->Valid(); iter->Next()) {} for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {} delete iter; read_opts.timestamp = &ts1; // lower_bound and upper_bound are without timestamp. read_opts.iterate_lower_bound = &lower_bound; read_opts.iterate_upper_bound = &upper_bound; auto* iter1 = db->NewIterator(read_opts); // Do Seek or SeekToFirst() delete iter1; ``` Test plan (dev server) ``` $make check ``` Simple benchmarking (dev server) 1. The overhead introduced by this PR even when timestamp is disabled. key size: 16 bytes value size: 100 bytes Entries: 1000000 Data reside in main memory, and try to stress iterator. Repeated three times on master and this PR. - Seek without next ``` ./db_bench -db=/dev/shm/rocksdbtest-1000 -benchmarks=fillseq,seekrandom -enable_pipelined_write=false -disable_wal=true -format_version=3 ``` master: 159047.0 ops/sec this PR: 158922.3 ops/sec (2% drop in throughput) - Seek and next 10 times ``` ./db_bench -db=/dev/shm/rocksdbtest-1000 -benchmarks=fillseq,seekrandom -enable_pipelined_write=false -disable_wal=true -format_version=3 -seek_nexts=10 ``` master: 109539.3 ops/sec this PR: 107519.7 ops/sec (2% drop in throughput) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6255 Differential Revision: D19438227 Pulled By: riversand963 fbshipit-source-id: b66b4979486f8474619f4aa6bdd88598870b0746 | 07 March 2020, 00:24:27 UTC |
0a0151f | Cheng Chang | 06 March 2020, 22:02:09 UTC | Remove memcpy from RandomAccessFileReader::Read in direct IO mode (#6455) Summary: In direct IO mode, RandomAccessFileReader::Read allocates an internal aligned buffer, and then copies the result into the scratch buffer. If the result is only temporarily used inside a function, there is no need to do the memcpy and just let the result Slice refer to the internally allocated buffer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6455 Test Plan: make check Differential Revision: D20106753 Pulled By: cheng-chang fbshipit-source-id: 44f505843837bba47a56e3fa2c4dd3bd76486b58 | 06 March 2020, 22:05:12 UTC |
f6c2777 | Otto Kekäläinen | 06 March 2020, 20:29:30 UTC | Fix spelling: commited -> committed (#6481) Summary: In most places in the code the variable names are spelled correctly as COMMITTED but in a couple places not. This fixes them and ensures the variable is always called COMMITTED everywhere. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6481 Differential Revision: D20306776 Pulled By: pdillinger fbshipit-source-id: b6c1bfe41db559b4bc6955c530934460c07f7022 | 06 March 2020, 20:45:20 UTC |
e171a21 | Yuqi Gu | 06 March 2020, 01:15:49 UTC | Fix db_wal_test::TruncateLastLogAfterRecoverWithoutFlush failure (#6437) Summary: `TruncateLastLogAfterRecoverWithoutFlush` case depends on fallocate support of underlying file system. On a file system which lacks of this feature, like zfs, it will fail to allocate predefined file size as this test case intends to do; So a check block is added to detect fallocate support and skip test if not. The related work is done by JunHe77. Thanks! Signed-off-by: Yuqi Gu <yuqi.gu@arm.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6437 Differential Revision: D20145032 Pulled By: pdillinger fbshipit-source-id: c8b691dc508e95acfa2a004ddbc07e2faa76680d | 06 March 2020, 01:18:16 UTC |
afb9709 | Cheng Chang | 05 March 2020, 04:12:23 UTC | Skip high levels with no key falling in the range in CompactRange (#6482) Summary: In CompactRange, if there is no key in memtable falling in the specified range, then flush is skipped. This PR extends this skipping logic to SST file levels: it starts compaction from the highest level (starting from L0) that has files with key falling in the specified range, instead of always starts from L0. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6482 Test Plan: A new test ManualCompactionTest::SkipLevel is added. Also updated a test related to statistics of index block cache hit in db_test2, the index cache hit is increased by 1 in this PR because when checking overlap for the key range in L0, OverlapWithLevelIterator will do a seek in the table cache iterator, which will read from the cached index. Also updated db_compaction_test and db_test to use correct range for full compaction. Differential Revision: D20251149 Pulled By: cheng-chang fbshipit-source-id: f822157cf4796972bd5035d9d7178d8dfb7af08b | 05 March 2020, 04:15:25 UTC |
e62fe50 | Zhichao Cao | 04 March 2020, 20:30:34 UTC | Introduce FaultInjectionTestFS to test fault File system instead of Env (#6414) Summary: In the current code base, we can use FaultInjectionTestEnv to simulate the env issue such as file write/read errors, which are used in most of the test. The PR https://github.com/facebook/rocksdb/issues/5761 introduce the File System as a new Env API. This PR implement the FaultInjectionTestFS, which can be used to simulate when File System has issues such as IO error. user can specify any IOStatus error as input, such that FS corresponding actions will return certain error to the caller. A set of ErrorHandlerFSTests are introduced for testing Pull Request resolved: https://github.com/facebook/rocksdb/pull/6414 Test Plan: pass make asan_check, pass error_handler_fs_test. Differential Revision: D20252421 Pulled By: zhichao-cao fbshipit-source-id: e922038f8ce7e6d1da329fd0bba7283c4b779a21 | 04 March 2020, 20:35:05 UTC |
8bbd76e | Fabrice Fontaine | 04 March 2020, 02:06:56 UTC | Check for sys/auxv.h (#6359) Summary: Check for sys/auxv.h and getauxval before using them as they are not always available (for example on uclibc) Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6359 Differential Revision: D20239797 fbshipit-source-id: 175a098094d81545628c2372e7c388e70a32fd48 | 04 March 2020, 02:09:59 UTC |
03dbd11 | Kefu Chai | 03 March 2020, 16:39:14 UTC | s/const auto/const auto&/ when doing loop (#6477) Summary: this silences following warning from clang-11 ``` rocksdb/db/db_impl/db_impl_compaction_flush.cc:1040:21: warning: loop variable 'newf' of type 'const std::pair<int, rocksdb::FileMetaData>' creates a copy from type 'const std::pair<int\ , rocksdb::FileMetaData>' [-Wrange-loop-analysis] for (const auto newf : c->edit()->GetNewFiles()) { ^ rocksdb/db/db_impl/db_impl_compaction_flush.cc:1040:10: note: use reference type 'const std::pair<int, rocksdb::FileMetaData> &' to prevent copying for (const auto newf : c->edit()->GetNewFiles()) { ^~~~~~~~~~~~~~~~~ & ``` Signed-off-by: Kefu Chai <tchaikov@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/6477 Differential Revision: D20211850 Pulled By: ltamasi fbshipit-source-id: 3e89e13a12bba79f1b934d46b7c4c0576cdafb01 | 03 March 2020, 16:41:57 UTC |
48d8d07 | sumeerbhola | 03 March 2020, 04:49:27 UTC | Add missing MutexLock to MockEnv::CreateDir (#6474) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6474 Differential Revision: D20205109 Pulled By: ltamasi fbshipit-source-id: ec136005c63740f5b713ff537b5671ea9b8e217a | 03 March 2020, 04:52:19 UTC |
17bef7d | sdong | 03 March 2020, 00:34:36 UTC | Fix data race of GetCreationTimeOfOldestFile() (#6473) Summary: When DBImpl::GetCreationTimeOfOldestFile() calls Version::GetCreationTimeOfOldestFile(), the version is not directly or indirectly referenced, so an event like compaction can race with the operation and cause DBImpl::GetCreationTimeOfOldestFile() to access delocated data. This was caught by an ASAN run: ==268==ERROR: AddressSanitizer: heap-use-after-free on address 0x612000b7d198 at pc 0x000018332913 bp 0x7f391510d310 sp 0x7f391510d308 READ of size 8 at 0x612000b7d198 thread T845 (store_load-33) SCARINESS: 51 (8-byte-read-heap-use-after-free) #0 0x18332912 in rocksdb::Version::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/src/db/version_set.cc:1488 https://github.com/facebook/rocksdb/issues/1 0x1803ddaa in rocksdb::DBImpl::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/src/db/db_impl/db_impl.cc:4499 https://github.com/facebook/rocksdb/issues/2 0xe24ca09 in rocksdb::StackableDB::GetCreationTimeOfOldestFile(unsigned long*) rocksdb/utilities/stackable_db.h:392 ...... 0x612000b7d198 is located 216 bytes inside of 296-byte region [0x612000b7d0c0,0x612000b7d1e8) freed by thread T28 here: ...... https://github.com/facebook/rocksdb/issues/5 0x1832c73f in std::vector<rocksdb::FileMetaData*, std::allocator<rocksdb::FileMetaData*> >::~vector() third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/stl_vector.h:435 https://github.com/facebook/rocksdb/issues/6 0x1832c73f in rocksdb::VersionStorageInfo::~VersionStorageInfo() rocksdb/src/db/version_set.cc:734 https://github.com/facebook/rocksdb/issues/7 0x1832cf42 in rocksdb::Version::~Version() rocksdb/src/db/version_set.cc:758 https://github.com/facebook/rocksdb/issues/8 0x9d1bb5 in rocksdb::Version::Unref() rocksdb/src/db/version_set.cc:2869 https://github.com/facebook/rocksdb/issues/9 0x183e7631 in rocksdb::Compaction::~Compaction() rocksdb/src/db/compaction/compaction.cc:275 https://github.com/facebook/rocksdb/issues/10 0x9e6de6 in std::default_delete<rocksdb::Compaction>::operator()(rocksdb::Compaction*) const third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:78 https://github.com/facebook/rocksdb/issues/11 0x9e6de6 in std::unique_ptr<rocksdb::Compaction, std::default_delete<rocksdb::Compaction> >::reset(rocksdb::Compaction*) third-party-buck/platform007/build/libgcc/include/c++/trunk/bits/unique_ptr.h:376 https://github.com/facebook/rocksdb/issues/12 0x9e6de6 in rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2826 https://github.com/facebook/rocksdb/issues/13 0x9ac3b8 in rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2320 https://github.com/facebook/rocksdb/issues/14 0x9abff7 in rocksdb::DBImpl::BGWorkCompaction(void*) rocksdb/src/db/db_impl/db_impl_compaction_flush.cc:2096 ...... Fix the issue by reference the super version and use the referenced version from it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6473 Test Plan: Run ASAN for all existing tests. Differential Revision: D20196416 fbshipit-source-id: 5f4a7918110fc7b8dd7841932d376bc9d1e59d6f | 03 March 2020, 00:37:01 UTC |
8d73137 | Zhichao Cao | 03 March 2020, 00:14:00 UTC | Replace Directory with FSDirectory in DB (#6468) Summary: In the current code base, we can use Directory from Env to manage directory (e.g, Fsync()). The PR https://github.com/facebook/rocksdb/issues/5761 introduce the File System as a new Env API. So we further replace the Directory class in DB with FSDirectory such that we can have more IO information from IOStatus returned by FSDirectory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6468 Test Plan: pass make asan_check Differential Revision: D20195261 Pulled By: zhichao-cao fbshipit-source-id: 93962cb9436852bfcfb76e086d9e7babd461cbe1 | 03 March 2020, 00:16:26 UTC |
904a60f | Huisheng Liu | 02 March 2020, 23:58:32 UTC | return timestamp from get (#6409) Summary: Added new Get() methods that return timestamp. Dummy implementation is given so that classes derived from DB don't need to be touched to provide their implementation. MultiGet is not included. ReadRandom perf test (10 minutes) on the same development machine ram drive with the same DB data shows no regression (within marge of error). The test is adapted from https://github.com/facebook/rocksdb/wiki/RocksDB-In-Memory-Workload-Performance-Benchmarks. base line (commit 72ee067b9): 101.712 micros/op 314602 ops/sec; 36.0 MB/s (5658999 of 5658999 found) This PR: 100.288 micros/op 319071 ops/sec; 36.5 MB/s (5674999 of 5674999 found) ./db_bench --db=r:\rocksdb.github --num_levels=6 --key_size=20 --prefix_size=20 --keys_per_prefix=0 --value_size=100 --cache_size=2147483648 --cache_numshardbits=6 --compression_type=none --compression_ratio=1 --min_level_to_compress=-1 --disable_seek_compaction=1 --hard_rate_limit=2 --write_buffer_size=134217728 --max_write_buffer_number=2 --level0_file_num_compaction_trigger=8 --target_file_size_base=134217728 --max_bytes_for_level_base=1073741824 --disable_wal=0 --wal_dir=r:\rocksdb.github\WAL_LOG --sync=0 --verify_checksum=1 --delete_obsolete_files_period_micros=314572800 --max_background_compactions=4 --max_background_flushes=0 --level0_slowdown_writes_trigger=16 --level0_stop_writes_trigger=24 --statistics=0 --stats_per_interval=0 --stats_interval=1048576 --histogram=0 --use_plain_table=1 --open_files=-1 --mmap_read=1 --mmap_write=0 --memtablerep=prefix_hash --bloom_bits=10 --bloom_locality=1 --duration=600 --benchmarks=readrandom --use_existing_db=1 --num=25000000 --threads=32 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6409 Differential Revision: D20200086 Pulled By: riversand963 fbshipit-source-id: 490edd74d924f62bd8ae9c29c2a6bbbb8410ca50 | 03 March 2020, 00:01:00 UTC |
8637bc1 | Levi Tamasi | 02 March 2020, 23:31:56 UTC | Fix the description of unordered_write in db_bench (#6476) Summary: As reported in https://github.com/facebook/rocksdb/issues/6467, the description of the `unordered_write` switch of `db_bench` was incorrect. (Note: the new description is based on https://rocksdb.org/blog/2019/08/15/unordered-write.html). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6476 Test Plan: `db_bench --help` Differential Revision: D20200653 Pulled By: ltamasi fbshipit-source-id: 4c3683fcfa6a069164167af5aaff9974a810c16a | 02 March 2020, 23:34:19 UTC |
5f2f8cd | Yanqin Jin | 02 March 2020, 20:51:17 UTC | Ignore compile_commands.json file (#6472) Summary: Both clangd and cquery-language-server requires a compile_commands.json file to index the project. This file can be ignored by git. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6472 Differential Revision: D20194899 Pulled By: riversand963 fbshipit-source-id: ea1587f2e5d10b7591147073b61efe262a1cf747 | 02 March 2020, 20:54:13 UTC |
9b3c9ef | sdong | 02 March 2020, 19:52:36 UTC | Add --index_with_first_key and --index_shortening_mode to DB bench (#5859) Summary: Some combinatino of --index_with_first_key and --index_shortening_mode can signifcantly improve performance for large values. Expose them in db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5859 Test Plan: Run them with the new options and observe the behavior. Differential Revision: D20104434 fbshipit-source-id: 21d48a732a9caf20b82312c7d7557d747ea3c304 | 02 March 2020, 19:55:28 UTC |
86f1ad7 | sdong | 29 February 2020, 00:40:13 UTC | Add more unit test coverage to MultiRead (#6452) Summary: MultiRead tests in env_test cannot simulate the io_uring case when queries need to be submitted in multiple rounds. Add a new unit test to cover up more requests per MultiRead Pull Request resolved: https://github.com/facebook/rocksdb/pull/6452 Test Plan: Run it and see it pass when liburing is enabled or not enabled. Differential Revision: D20078924 fbshipit-source-id: 6cff7fe345a4c5aa47135186e6181bf00df02b68 | 29 February 2020, 00:42:44 UTC |
051696b | Michael R. Crusoe | 28 February 2020, 22:10:51 UTC | fix some spelling typos (#6464) Summary: Found from Debian's "Lintian" program Pull Request resolved: https://github.com/facebook/rocksdb/pull/6464 Differential Revision: D20162862 Pulled By: zhichao-cao fbshipit-source-id: 06941ee2437b038b2b8045becbe9d2c6fbff3e12 | 28 February 2020, 22:14:03 UTC |
41535d0 | Manuel Ung | 28 February 2020, 19:16:50 UTC | WriteUnPrepared: Pass in correct subbatch count during rollback (#6463) Summary: Today `WriteUnpreparedTxn::RollbackInternal` will write the rollback batch assuming that there is only a single subbatch. However, because untracked_keys_ are currently not deduplicated, it's possible for duplicate keys to exist, and thus split the batch. Also, tracked_keys_ also does not support compators outside of the bytewise comparators, so it's possible for duplicates to occur there as well. To solve this, just pass in the correct subbatch count. Also, removed `WriteUnpreparedRollbackPreReleaseCallback` to unify the Commit/Rollback codepaths some more. Also, fixed a bug in `CommitInternal` where if 1. two_write_queue is true and 2. include_data is true, then `WriteUnpreparedCommitEntryPreReleaseCallback` ends up calling `AddCommitted` on the commit time write batch a second time on the second write. To fix, `WriteUnpreparedCommitEntryPreReleaseCallback` is re-initialized. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6463 Differential Revision: D20150153 Pulled By: lth fbshipit-source-id: df0b42d39406c75af73df995aa1138f0db539cd1 | 28 February 2020, 19:19:32 UTC |
72ee067 | Jermy Li | 27 February 2020, 20:16:27 UTC | fix assert error while db.getDefaultColumnFamily().getDescriptor() (#6006) Summary: Threw assert error at assert(isOwningHandle()) in ColumnFamilyHandle.getDescriptor(), because default CF don't own a handle, due to [RocksDB.getDefaultColumnFamily()](https://github.com/facebook/rocksdb/blob/3a408eeae95614150ac930fc7f244524ed8c6f1c/java/src/main/java/org/rocksdb/RocksDB.java#L3702) called cfHandle.disOwnNativeHandle(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6006 Differential Revision: D19031448 fbshipit-source-id: 2420c45e835bda0e552e919b1b63708472b91538 | 27 February 2020, 20:37:46 UTC |
741decf | Cheng Chang | 26 February 2020, 00:45:43 UTC | Return early on failure when constructing CuckooTableReader (#6453) Summary: If file is not mmaped, CuckooTableReader should not try to read table properties from the file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6453 Test Plan: Added a new unit test Differential Revision: D20103334 Pulled By: cheng-chang fbshipit-source-id: 48539f14d93f6c1ebe12c3df5a14719e9d7b8726 | 26 February 2020, 00:48:28 UTC |
f52db84 | Andrew Kryczka | 26 February 2020, 00:43:33 UTC | support SstFileManager in db_stress (#6454) Summary: Add some flags for configuring an SstFileManager. An SstFileManager is only created when one or more of these flags are set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6454 Test Plan: - ran it a while: ``` $ python ./tools/db_crashtest.py blackbox --simple -max_key=100000 -write_buffer_size=131072 -target_file_size_base=131072 -max_bytes_for_level_base=524288 -value_size_mult=33 --interval=10 -max_background_compactions=4 -max_background_flushes=2 -sst_file_manager_bytes_per_sec=1048576 ``` - verified with strace the SstFileManager is behaving as configured: ``` $ strace -fp `pidof db_stress` -e ftruncate,unlink ... [pid 3074805] ftruncate(9</tmp/rocksdb_crashtest_blackbox6OJywh/000070.sst.trash>, 67423) = 0 [pid 3074805] ftruncate(9</tmp/rocksdb_crashtest_blackbox6OJywh/000070.sst.trash>, 51039) = 0 [pid 3074805] ftruncate(9</tmp/rocksdb_crashtest_blackbox6OJywh/000070.sst.trash>, 34655) = 0 [pid 3074805] ftruncate(9</tmp/rocksdb_crashtest_blackbox6OJywh/000070.sst.trash>, 18271) = 0 [pid 3074805] ftruncate(9</tmp/rocksdb_crashtest_blackbox6OJywh/000070.sst.trash>, 1887) = 0 [pid 3074805] unlink("/tmp/rocksdb_crashtest_blackbox6OJywh/000070.sst.trash") = 0 ... ``` Differential Revision: D20103315 Pulled By: ajkr fbshipit-source-id: b3e1092747157459d244b047947a979b85c98f48 | 26 February 2020, 00:45:30 UTC |
69679e7 | Andrew Kryczka | 25 February 2020, 23:29:17 UTC | Fix range deletion tombstone ingestion with global seqno (#6429) Summary: Original author: jeffrey-xiao If we are writing a global seqno for an ingested file, the range tombstone metablock gets accessed and put into the cache during ingestion preparation. At the time, the global seqno of the ingested file has not yet been determined, so the cached block will not have a global seqno. When the file is ingested and we read its range tombstone metablock, it will be returned from the cache with no global seqno. In that case, we use the actual seqnos stored in the range tombstones, which are all zero, so the tombstones cover nothing. This commit removes global_seqno_ variable from Block. When iterating over a block, the global seqno for the block is determined by the iterator instead of storing this mutable attribute in Block. Additionally, this commit adds a regression test to check that keys are deleted when ingesting a file with a global seqno and range deletion tombstones. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6429 Differential Revision: D19961563 Pulled By: ajkr fbshipit-source-id: 5cf777397fa3e452401f0bf0364b0750492487b7 | 25 February 2020, 23:31:48 UTC |
d87c10c | Levi Tamasi | 25 February 2020, 02:38:00 UTC | Add blob file state to VersionEdit (#6416) Summary: BlobDB currently does not keep track of blob files: no records are written to the manifest when a blob file is added or removed, and upon opening a database, the list of blob files is populated simply based on the contents of the blob directory. This means that lost blob files cannot be detected at the moment. We plan to solve this issue by making blob files a part of `Version`; as a first step, this patch makes it possible to store information about blob files in `VersionEdit`. Currently, this information includes blob file number, total number and size of all blobs, and total number and size of garbage blobs. However, the format is extensible: new fields can be added in both a forward compatible and a forward incompatible manner if needed (similarly to `kNewFile4`). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6416 Test Plan: `make check` Differential Revision: D19894234 Pulled By: ltamasi fbshipit-source-id: f9753e1f2aedf6dadb70c09b345207cb9c58c329 | 25 February 2020, 02:39:53 UTC |
eb367d4 | sdong | 24 February 2020, 23:45:47 UTC | Buck config: Re-enable liburing under Linux (#6451) Summary: The known bug of liburing has been fixed. Now we can re-enable liburing under Linux Pull Request resolved: https://github.com/facebook/rocksdb/pull/6451 Test Plan: Watch internal CI Differential Revision: D20079009 fbshipit-source-id: 04a6f53a900ff721f9a62a188cf906771b5d68d2 | 24 February 2020, 23:47:34 UTC |
b47a714 | Cheng Chang | 24 February 2020, 19:42:37 UTC | Update release version to 6.8 (#6450) Summary: Update release version to 6.8 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6450 Test Plan: no code change Differential Revision: D20071889 Pulled By: cheng-chang fbshipit-source-id: 91450aae09b201926469ff32f59ed436366f3b74 | 24 February 2020, 19:44:47 UTC |
43dde33 | Peter Dillinger | 22 February 2020, 15:59:38 UTC | Share kPageSize (and other small tweaks) (#6443) Summary: Make kPageSize extern const size_t (used in draft https://github.com/facebook/rocksdb/issues/6427) Make kLitteEndian constexpr bool Clarify a couple of comments Pull Request resolved: https://github.com/facebook/rocksdb/pull/6443 Test Plan: make check, CI Differential Revision: D20044558 Pulled By: pdillinger fbshipit-source-id: e0c5cc13229c82726280dc0ddcba4078346b8418 | 22 February 2020, 16:01:36 UTC |
942eaba | sdong | 22 February 2020, 00:56:04 UTC | Handle io_uring partial results (#6441) Summary: The logic that handles io_uring partial results was wrong. Fix the logic by putting it into a queue and continue reading. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6441 Test Plan: Make sure this patch fixes the application test case where the bug was discovered; in env_test, add a unit test that simulates partial results and make sure the results are still correct. Differential Revision: D20018616 fbshipit-source-id: 5398a7e34d74c26d52aa69dfd604e93e95d99c62 | 22 February 2020, 00:57:37 UTC |
890d87f | Yanqin Jin | 21 February 2020, 23:07:55 UTC | Some minor fix-ups (#6440) Summary: Cleanup some code without any real change in functionality. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6440 Differential Revision: D20015891 Pulled By: riversand963 fbshipit-source-id: 33e18754b0f002006a6d4805e9aaf84c0c8ad25a | 21 February 2020, 23:09:56 UTC |
ab65278 | Peter Dillinger | 21 February 2020, 21:29:41 UTC | Misc filter_bench improvements (#6444) Summary: Useful in validating/testing internal fragmentation changes (https://github.com/facebook/rocksdb/issues/6427) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6444 Test Plan: manual (no changes to production code) Differential Revision: D20040076 Pulled By: pdillinger fbshipit-source-id: 32d26f363d2a9ab9f5bebd281dcebd9915ae340e | 21 February 2020, 21:31:57 UTC |
fcec56e | Zaiyang Li | 21 February 2020, 19:10:54 UTC | Add function to set row cache on rocksdb_options_t (#6442) Summary: Adding a C API function to set `row_cache` on `rocksdb_options_t` as this functionality is missing. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6442 Differential Revision: D20036813 Pulled By: riversand963 fbshipit-source-id: c1fa95ea343345fbc1e57961d0d048e0e79be373 | 21 February 2020, 19:12:42 UTC |
d75ce0a | sdong | 20 February 2020, 22:53:51 UTC | Mention rocksdb_namespace.h in HISTORY.md (#6439) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6439 Differential Revision: D20012442 fbshipit-source-id: a7c9569826eac39cd7ea69c90f08a21dd4caa335 | 20 February 2020, 22:55:19 UTC |
362b8d4 | Yanqin Jin | 20 February 2020, 21:13:53 UTC | Fix MANIFEST name assignment (#6426) Summary: Currently, a new MANIFEST file is assigned a new file number when 1) no MANIFEST is open, or 2) current MANIFEST file size exceeds a threshold. This is not sufficient. There are cases when the caller explicitly specifies that a new MANIFEST be created. For example, if user sets options.write_dbid_to_manifest = true, and there are WAL files, then RocksDB will run into an issue during recovery. `DBImpl::Recover()` will call `LogAndApply()` to write dbid. At this point, the db being recovered creates a new MANIFEST, say, MANIFEST-000003. Since there are WALs, `DBImpl::RecoverLogFiles` will be called. Towards the end of this function, we call `LogAndApply(new_descriptor_log=true)`, which explicitly creates a new MANIFEST. However, the manifest_file_number is wrong before this fix. Consequently, RocksDB opens an existing, non-empty file for append, effectively truncating the file to zero. If a crash occurs, then there will be data loss. Test Plan (devserver): make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6426 Test Plan: make check Differential Revision: D19951866 Pulled By: riversand963 fbshipit-source-id: 4b1b9fc28d4fe2ac12764b388ef9e61f05e766da | 20 February 2020, 22:30:58 UTC |
fdf882d | sdong | 20 February 2020, 20:07:53 UTC | Replace namespace name "rocksdb" with ROCKSDB_NAMESPACE (#6433) Summary: When dynamically linking two binaries together, different builds of RocksDB from two sources might cause errors. To provide a tool for user to solve the problem, the RocksDB namespace is changed to a flag which can be overridden in build time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6433 Test Plan: Build release, all and jtest. Try to build with ROCKSDB_NAMESPACE with another flag. Differential Revision: D19977691 fbshipit-source-id: aa7f2d0972e1c31d75339ac48478f34f6cfcfb3e | 20 February 2020, 20:09:57 UTC |
4e33f1e | Gaurav Singh | 20 February 2020, 18:25:14 UTC | simplify user_access_only expression (#6360) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6360 Differential Revision: D19698918 Pulled By: riversand963 fbshipit-source-id: d20ecca541376cccd32fc7afb504ea90021860ee | 20 February 2020, 18:27:56 UTC |
a993cc3 | Remington Brasga | 20 February 2020, 01:06:14 UTC | Fixed typo in benchmark.sh (#6434) Summary: TB = 1024 * GB Pull Request resolved: https://github.com/facebook/rocksdb/pull/6434 Differential Revision: D19978339 Pulled By: zhichao-cao fbshipit-source-id: 5a89890110b23f0ebda4a95223f66da6736321ac | 20 February 2020, 01:08:02 UTC |
5a29751 | Yanqin Jin | 19 February 2020, 00:08:14 UTC | Ignore .vs/.vscode dir used by visual studio/ vscode (#6428) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6428 Differential Revision: D19959176 Pulled By: riversand963 fbshipit-source-id: 5b863944663462b246bb82883fa683f75ab33fc1 | 19 February 2020, 00:10:15 UTC |
c6abe30 | Andrew Kryczka | 18 February 2020, 21:52:09 UTC | Fix concurrent full purge and WAL recycling (#5900) Summary: We were removing the file from `log_recycle_files_` before renaming it with `ReuseWritableFile()`. Since `ReuseWritableFile()` occurs outside the DB mutex, it was possible for a concurrent full purge to sneak in and delete the file before it could be renamed. Consequently, `SwitchMemtable()` would fail and the DB would enter read-only mode. The fix is to hold the old file number in `log_recycle_files_` until after the file has been renamed. Full purge uses that list to decide which files to keep, so it can no longer delete a file pending recycling. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5900 Test Plan: new unit test Differential Revision: D19771719 Pulled By: ajkr fbshipit-source-id: 094346349ca3fb499712e62de03905acc30b5ce8 | 18 February 2020, 21:54:13 UTC |
0f9dcb8 | Andrew Kryczka | 18 February 2020, 19:14:37 UTC | Return NotSupported from WriteBatchWithIndex::DeleteRange (#5393) Summary: As discovered in https://github.com/facebook/rocksdb/issues/5260 and https://github.com/facebook/rocksdb/issues/5392, reads on the indexed batch do not account for range tombstones. So, return `Status::NotSupported` from `WriteBatchWithIndex::DeleteRange` until we properly support it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5393 Test Plan: added unit test Differential Revision: D19912360 Pulled By: ajkr fbshipit-source-id: 0bbfc978ea015d64516ca708fce2429abba524cb | 18 February 2020, 19:18:25 UTC |
3a34575 | acelyc111 | 14 February 2020, 23:53:29 UTC | Fix compile error when LZ4 is up to r123 (#6412) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6412 Differential Revision: D19914063 Pulled By: ajkr fbshipit-source-id: 4e401e665d4b449d24c4cdec35a4585eeda95996 | 14 February 2020, 23:55:23 UTC |
dc23c12 | Manuel Ung | 14 February 2020, 19:20:37 UTC | WriteUnPrepared: Untracked keys (#6404) Summary: For write unprepared, some applications may bypass the transaction api, and write keys directly into the write batch. However, since they are not tracked, rollbacks (both for savepoint and transaction) are not aware that these keys have to be rolled back. The fix is to track them in `WriteUnpreparedTxn::untracked_keys_`. This is populated whenever we flush unprepared batches into the DB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6404 Differential Revision: D19842023 Pulled By: lth fbshipit-source-id: a9edfc643d5c905fc89da9a9a9094d30c9b70108 | 14 February 2020, 19:31:39 UTC |
152f8a8 | Cheng Chang | 14 February 2020, 16:24:09 UTC | Remove unnecessary computation of index (#6406) Summary: `index` can be replaced by `iter`, saving the computation of `index++`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6406 Test Plan: make check Differential Revision: D19905056 Pulled By: cheng-chang fbshipit-source-id: add4638959c0d2e4e77a11f3fa04ffabaf0de790 | 14 February 2020, 16:26:23 UTC |
4034e28 | Cheng Chang | 14 February 2020, 16:15:24 UTC | Fail fast in paranoid mode when LoadTableHandlers fail during recovering (#6368) Summary: Previously, when recovering version set, LoadTableHandlers failures are ignored. If paranoid_checks is true, this failure should not be ignored, otherwise, the opened db might be in an inconsistent state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6368 Test Plan: make check Differential Revision: D19713459 Pulled By: cheng-chang fbshipit-source-id: 68cb94f4f2cc43f8b024b14755193cd45cfcad55 | 14 February 2020, 16:17:10 UTC |
29e2443 | wolfkdy | 14 February 2020, 02:50:04 UTC | refine code (#6420) Summary: I create a new branch from the branch new upsteram/master and "git merge --squash". Maybe it will fix everything. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6420 Differential Revision: D19897152 Pulled By: zhichao-cao fbshipit-source-id: 6575d9e3b23e360f42ee1480b43028b5fcc20136 | 14 February 2020, 02:55:02 UTC |
908b1ee | Manuel Ung | 14 February 2020, 02:50:01 UTC | WriteUnPrepared: Fix assertion during recovery (#6419) Summary: During recovery, multiple (un)prepared batches could exist in the same WAL record due to group commit. This breaks an assertion in `MemTableInserter::MarkBeginPrepare`. To fix, reset unprepared_batch_ to false after `MarkEndPrepare`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6419 Differential Revision: D19896148 Pulled By: lth fbshipit-source-id: b1a32ef88f775a0881264a18bd1a4a5b8c85eee3 | 14 February 2020, 02:52:05 UTC |