https://github.com/facebook/rocksdb
- HEAD
- refs/heads/2.2.fb.branch
- refs/heads/2.3.fb.branch
- refs/heads/2.4.fb.branch
- refs/heads/2.5.fb.branch
- refs/heads/2.6.fb.branch
- refs/heads/2.7
- refs/heads/2.7.fb.branch
- refs/heads/2.8.1.fb
- refs/heads/2.8.fb
- refs/heads/2.8.fb.trunk
- refs/heads/3.0.fb
- refs/heads/3.0.fb.branch
- refs/heads/3.1.fb
- refs/heads/3.10.fb
- refs/heads/3.11.fb
- refs/heads/3.12.fb
- refs/heads/3.13.fb
- refs/heads/3.2.fb
- refs/heads/3.3.fb
- refs/heads/3.4.fb
- refs/heads/3.5.fb
- refs/heads/3.6.fb
- refs/heads/3.7.fb
- refs/heads/3.8.fb
- refs/heads/3.9.fb
- refs/heads/4.0.fb
- refs/heads/4.1.fb
- refs/heads/4.10.fb
- refs/heads/4.11.fb
- refs/heads/4.12.fb
- refs/heads/4.13.fb
- refs/heads/4.2.fb
- refs/heads/4.3.fb
- refs/heads/4.4.fb
- refs/heads/4.5.fb
- refs/heads/4.6.fb
- refs/heads/4.7.fb
- refs/heads/4.8.fb
- refs/heads/4.9.fb
- refs/heads/5.0.fb
- refs/heads/5.1.fb
- refs/heads/5.10.fb
- refs/heads/5.11.fb
- refs/heads/5.12.fb
- refs/heads/5.13.fb
- refs/heads/5.13.fb.myrocks
- refs/heads/5.14.fb
- refs/heads/5.14.fb.myrocks
- refs/heads/5.15.fb
- refs/heads/5.16.fb
- refs/heads/5.17.fb
- refs/heads/5.17.fb.myrocks
- refs/heads/5.18.fb
- refs/heads/5.2.fb
- refs/heads/5.3.fb
- refs/heads/5.4.fb
- refs/heads/5.5.fb
- refs/heads/5.6.fb
- refs/heads/5.7.fb
- refs/heads/5.7.fb.myrocks
- refs/heads/5.8.3
- refs/heads/5.8.fb
- refs/heads/5.9.fb
- refs/heads/5.9.fb.myrocks
- refs/heads/6.0.fb
- refs/heads/6.0.fb.myrocks
- refs/heads/6.1.fb
- refs/heads/6.1.fb.myrocks
- refs/heads/6.1.fb.prod201905
- refs/heads/6.10.fb
- refs/heads/6.11.fb
- refs/heads/6.12.fb
- refs/heads/6.13.fb
- refs/heads/6.13.fb.laser
- refs/heads/6.14.fb
- refs/heads/6.14.fb.laser
- refs/heads/6.15.fb
- refs/heads/6.16.fb
- refs/heads/6.17.fb
- refs/heads/6.17.fb.laser
- refs/heads/6.18.fb
- refs/heads/6.19.fb
- refs/heads/6.2.fb
- refs/heads/6.20.fb
- refs/heads/6.21.fb
- refs/heads/6.22-history.md-fixup
- refs/heads/6.22.fb
- refs/heads/6.23.fb
- refs/heads/6.24.fb
- refs/heads/6.25.fb
- refs/heads/6.26.fb
- refs/heads/6.27.fb
- refs/heads/6.28.fb
- refs/heads/6.29.fb
- refs/heads/6.3.fb
- refs/heads/6.3.fb.myrocks
- refs/heads/6.3.fb.myrocks2
- refs/heads/6.3fb
- refs/heads/6.4.fb
- refs/heads/6.5.fb
- refs/heads/6.6.fb
- refs/heads/6.7.fb
- refs/heads/6.8.fb
- refs/heads/6.9.fb
- refs/heads/7.0.fb
- refs/heads/7.1.fb
- refs/heads/7.10.fb
- refs/heads/7.2.fb
- refs/heads/7.3.fb
- refs/heads/7.4.fb
- refs/heads/7.5.fb
- refs/heads/7.6.fb
- refs/heads/7.7.fb
- refs/heads/7.8.fb
- refs/heads/7.9.fb
- refs/heads/8.0.fb
- refs/heads/8.1.fb
- refs/heads/8.10.fb
- refs/heads/8.11.2_zippydb
- refs/heads/8.11.fb
- refs/heads/8.11.fb_zippydb
- refs/heads/8.2.fb
- refs/heads/8.3.fb
- refs/heads/8.4.fb
- refs/heads/8.5.fb
- refs/heads/8.6.fb
- refs/heads/8.7.fb
- refs/heads/8.8.fb
- refs/heads/8.9.fb
- refs/heads/9.0.fb
- refs/heads/9.1.fb
- refs/heads/9.1.fb.myrocks
- refs/heads/9.2.fb
- refs/heads/9.3.fb
- refs/heads/adaptive
- refs/heads/ajkr-patch-1
- refs/heads/ajkr-patch-2
- refs/heads/blob_shadow
- refs/heads/bottom-pri-level
- refs/heads/bugfix-build-detect
- refs/heads/checksum_readahead_mmap_fix
- refs/heads/draft-myrocks-and-fbcode-8.0.fb
- refs/heads/feature/debug-rocksdbjavastatic
- refs/heads/feature/travis-arm64
- refs/heads/fix-release-notes
- refs/heads/fix-win2022-build
- refs/heads/fix-write-batch-comment
- refs/heads/format_compatible_4
- refs/heads/getmergeops
- refs/heads/gh-pages-old
- refs/heads/history-update
- refs/heads/hotfix/lambda-capture
- refs/heads/improve-support
- refs/heads/jijiew-patch-1
- refs/heads/katherinez-patch-1
- refs/heads/katherinez-patch-2
- refs/heads/main
- refs/heads/master
- refs/heads/mdcallag_benchmark_oct22
- refs/heads/nvm_cache_proto
- refs/heads/pr-sanity-check-as-GHAction
- refs/heads/pr/11267
- refs/heads/pr/6062
- refs/heads/ramvadiv-patch-1
- refs/heads/release_fix
- refs/heads/revert-10606-7.6.1
- refs/heads/ribbon_bloom_hybrid
- refs/heads/scaffold
- refs/heads/siying-patch-1
- refs/heads/siying-patch-10
- refs/heads/siying-patch-2
- refs/heads/siying-patch-3
- refs/heads/siying-patch-4
- refs/heads/siying-patch-5
- refs/heads/siying-patch-6
- refs/heads/siying-patch-7
- refs/heads/siying-patch-8
- refs/heads/skip_memtable_flush
- refs/heads/testing_ppc_build
- refs/heads/tests
- refs/heads/unschedule_issue_test_base
- refs/heads/unused-var
- refs/heads/v6.6.4
- refs/heads/xxhash_merge_base
- refs/heads/yiwu_stackable
- refs/heads/yuslepukhin
- refs/remotes/origin/5.13.fb
- refs/tags/2.5.fb
- refs/tags/2.6.fb
- refs/tags/3.0.fb
- refs/tags/do-not-use-me2
- refs/tags/rocksdb-3.1
- refs/tags/rocksdb-3.10.2
- refs/tags/rocksdb-3.11
- refs/tags/rocksdb-3.11.1
- refs/tags/rocksdb-3.11.2
- refs/tags/rocksdb-3.2
- refs/tags/rocksdb-3.3
- refs/tags/rocksdb-3.4
- refs/tags/rocksdb-3.5
- refs/tags/rocksdb-3.5.1
- refs/tags/rocksdb-3.6.1
- refs/tags/rocksdb-3.6.2
- refs/tags/rocksdb-3.7
- refs/tags/rocksdb-3.8
- refs/tags/rocksdb-3.9
- refs/tags/rocksdb-3.9.1
- refs/tags/rocksdb-4.1
- refs/tags/rocksdb-5.10.2
- refs/tags/rocksdb-5.10.3
- refs/tags/rocksdb-5.10.4
- refs/tags/rocksdb-5.11.2
- refs/tags/rocksdb-5.11.3
- refs/tags/rocksdb-5.14.3
- refs/tags/rocksdb-5.2.1
- refs/tags/rocksdb-5.3.3
- refs/tags/rocksdb-5.3.4
- refs/tags/rocksdb-5.3.5
- refs/tags/rocksdb-5.3.6
- refs/tags/rocksdb-5.4.10
- refs/tags/rocksdb-5.4.5
- refs/tags/rocksdb-5.4.6
- refs/tags/rocksdb-5.5.2
- refs/tags/rocksdb-5.5.3
- refs/tags/rocksdb-5.5.4
- refs/tags/rocksdb-5.5.5
- refs/tags/rocksdb-5.5.6
- refs/tags/rocksdb-5.6.1
- refs/tags/rocksdb-5.6.2
- refs/tags/rocksdb-5.7.1
- refs/tags/rocksdb-5.7.2
- refs/tags/rocksdb-5.7.3
- refs/tags/rocksdb-5.7.5
- refs/tags/rocksdb-5.8.6
- refs/tags/rocksdb-5.8.7
- refs/tags/rocksdb-5.8.8
- refs/tags/rocksdb-5.9.2
- refs/tags/v4.0
- refs/tags/v4.1
- refs/tags/v5.10.2
- refs/tags/v5.10.3
- refs/tags/v5.10.4
- refs/tags/v5.11.2
- refs/tags/v5.11.3
- refs/tags/v5.13.3
- refs/tags/v5.14.3
- refs/tags/v5.15.10
- refs/tags/v5.18.3
- refs/tags/v5.2.1
- refs/tags/v5.3.3
- refs/tags/v5.3.4
- refs/tags/v5.3.5
- refs/tags/v5.3.6
- refs/tags/v5.4.10
- refs/tags/v5.4.5
- refs/tags/v5.4.6
- refs/tags/v5.5.2
- refs/tags/v5.5.3
- refs/tags/v5.5.4
- refs/tags/v5.5.5
- refs/tags/v5.5.6
- refs/tags/v5.6.1
- refs/tags/v5.6.2
- refs/tags/v5.7.1
- refs/tags/v5.7.2
- refs/tags/v5.7.3
- refs/tags/v5.7.5
- refs/tags/v5.8.6
- refs/tags/v5.8.7
- refs/tags/v5.8.8
- refs/tags/v5.9.2
- refs/tags/v6.0.1
- refs/tags/v6.0.2
- refs/tags/v6.1.1
- refs/tags/v6.1.2
- refs/tags/v6.10.1
- refs/tags/v6.10.2
- refs/tags/v6.11.4
- refs/tags/v6.11.6
- refs/tags/v6.12.6
- refs/tags/v6.12.7
- refs/tags/v6.13.2
- refs/tags/v6.13.3
- refs/tags/v6.14.5
- refs/tags/v6.14.6
- refs/tags/v6.15.4
- refs/tags/v6.15.5
- refs/tags/v6.16.3
- refs/tags/v6.16.4
- refs/tags/v6.17.3
- refs/tags/v6.2.2
- refs/tags/v6.2.4
- refs/tags/v6.20.3
- refs/tags/v6.22.1
- refs/tags/v6.25.3
- refs/tags/v6.26.1
- refs/tags/v6.28.2
- refs/tags/v6.29.3
- refs/tags/v6.29.4
- refs/tags/v6.29.5
- refs/tags/v6.3.6
- refs/tags/v6.4.6
- refs/tags/v6.5.2
- refs/tags/v6.5.3
- refs/tags/v6.6.3
- refs/tags/v6.6.4
- refs/tags/v6.7.3
- refs/tags/v6.8.1
- refs/tags/v7.0.1
- refs/tags/v7.0.2
- refs/tags/v7.0.4
- refs/tags/v7.2.0
- refs/tags/v7.2.2
- refs/tags/v7.5.3
- refs/tags/v7.7.2
- refs/tags/v7.9.2
- refs/tags/v8.0.0
- refs/tags/v8.11.4
- refs/tags/v8.3.2
- refs/tags/v8.3.3
- refs/tags/v8.4.4
- refs/tags/v8.5.3
- refs/tags/v8.6.7
- refs/tags/v8.7.3
- refs/tags/v9.0.1
- refs/tags/v9.1.1
- refs/tags/v9.2.1
- v9.1.0
- v9.0.0
- v8.9.1
- v8.8.1
- v8.5.4
- v8.11.3
- v8.10.2
- v8.10.0
- v8.1.1
- v7.8.3
- v7.7.8
- v7.7.3
- v7.6.0
- v7.4.5
- v7.4.4
- v7.4.3
- v7.3.1
- v7.10.2
- v7.1.2
- v7.1.1
- v7.0.3
- v6.27.3
- v6.26.0
- v6.25.1
- v6.24.2
- v6.23.3
- v6.23.2
- v6.19.3
- v6.15.2
- v5.8
- v5.5.1
- v5.4.7
- v5.18.4
- v5.17.2
- v5.16.6
- v5.14.2
- v5.13.4
- v5.13.2
- v5.13.1
- v5.12.5
- v5.12.4
- v5.12.3
- v5.12.2
- v5.1.4
- v5.1.3
- v5.1.2
- v5.0.2
- v5.0.1
- v4.9
- v4.8
- v4.6.1
- v4.5.1
- v4.4.1
- v4.4
- v4.3.1
- v4.3
- v4.2
- v4.13.5
- v4.13
- v4.11.2
- v3.9
- v3.8
- v3.7
- v3.6.1
- v3.5
- v3.4
- v3.3
- v3.2
- v3.13.1
- v3.13
- v3.12.1
- v3.12
- v3.11
- v3.10
- v3.1
- v3.0
- v2.8
- v2.7
- v2.6
- v2.5
- v2.4
- v2.3
- v2.2
- v2.1
- v2.0
- v1.5.9.1
- v1.5.8.2
- v1.5.8.1
- v1.5.8
- v1.5.7
- rocksdb-5.8
- rocksdb-5.4.7
- rocksdb-5.1.4
- rocksdb-5.1.3
- rocksdb-5.1.2
- rocksdb-5.0.2
- rocksdb-5.0.1
- rocksdb-4.9
- rocksdb-4.8
- rocksdb-4.6.1
- rocksdb-4.5.1
- rocksdb-4.4.1
- rocksdb-4.4
- rocksdb-4.3.1
- rocksdb-4.3
- rocksdb-4.2
- rocksdb-4.13.5
- rocksdb-4.13
- rocksdb-4.11.2
- rocksdb-3.10.1
- blob_st_lvl-pre
- 2.8.fb
- 2.7.fb
- 2.4.fb
- 2.3.fb
- 2.2.fb
- 2.1.fb
- 2.0.fb
- 1.5.9.fb
- 1.5.9.2.fb
- 1.5.9.1.fb
- 1.5.8.fb
- 1.5.8.2.fb
- 1.5.8.1.fb
- 1.5.7.fb
Take a new snapshot of a software origin
If the archived software origin currently browsed is not synchronized with its upstream version (for instance when new commits have been issued), you can explicitly request Software Heritage to take a new snapshot of it.
Use the form below to proceed. Once a request has been submitted and accepted, it will be processed as soon as possible. You can then check its processing state by visiting this dedicated page.Processing "take a new snapshot" request ...
Permalinks
To reference or cite the objects present in the Software Heritage archive, permalinks based on SoftWare Hash IDentifiers (SWHIDs) must be used.
Select below a type of object currently browsed in order to display its associated SWHID and permalink.
Revision | Author | Date | Message | Commit Date |
---|---|---|---|---|
2898067 | Levi Tamasi | 04 August 2021, 20:20:04 UTC | Mention PR 8585 in HISTORY.md | 04 August 2021, 20:47:17 UTC |
238a9c3 | Levi Tamasi | 04 August 2021, 20:07:49 UTC | Bump version number to 6.23.2 | 04 August 2021, 20:07:49 UTC |
2511b42 | Yanqin Jin | 03 August 2021, 20:30:05 UTC | Fix NotifyOnFlushCompleted() for atomic flush (#8585) Summary: PR https://github.com/facebook/rocksdb/issues/5908 added `flush_jobs_info_` to `FlushJob` to make sure `OnFlushCompleted()` is called after committing flush results to MANIFEST. However, `flush_jobs_info_` is not updated in atomic flush, causing `NotifyOnFlushCompleted()` to skip `OnFlushCompleted()`. This PR fixes this, in a similar way to https://github.com/facebook/rocksdb/issues/5908 that handles regular flush. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8585 Test Plan: make check Reviewed By: jay-zhuang Differential Revision: D29913720 Pulled By: riversand963 fbshipit-source-id: 4ff023c98372fa2c93188d4a5c8a4e9ffa0f4dda | 04 August 2021, 18:24:41 UTC |
1920121 | Levi Tamasi | 03 August 2021, 23:13:41 UTC | Mention PR 8605 in HISTORY.md (#8619) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8619 Reviewed By: riversand963 Differential Revision: D30081937 Pulled By: ltamasi fbshipit-source-id: 57505957ae2c22d4b194aa28cb3fd261b3b39919 | 04 August 2021, 18:20:31 UTC |
7c393c3 | Levi Tamasi | 03 August 2021, 01:10:57 UTC | Fix a race in ColumnFamilyData::UnrefAndTryDelete (#8605) Summary: The `ColumnFamilyData::UnrefAndTryDelete` code currently on the trunk unlocks the DB mutex before destroying the `ThreadLocalPtr` holding the per-thread `SuperVersion` pointers when the only remaining reference is the back reference from `super_version_`. The idea behind this was to break the circular dependency between `ColumnFamilyData` and `SuperVersion`: when the penultimate reference goes away, `ColumnFamilyData` can clean up the `SuperVersion`, which can in turn clean up `ColumnFamilyData`. (Assuming there is a `SuperVersion` and it is not referenced by anything else.) However, unlocking the mutex throws a wrench in this plan by making it possible for another thread to jump in and take another reference to the `ColumnFamilyData`, keeping the object alive in a zombie `ThreadLocalPtr`-less state. This can cause issues like https://github.com/facebook/rocksdb/issues/8440 , https://github.com/facebook/rocksdb/issues/8382 , and might also explain the `was_last_ref` assertion failures from the `ColumnFamilySet` destructor we sometimes observe during close in our stress tests. Digging through the archives, this unlocking goes way back to 2014 (or earlier). The original rationale was that `SuperVersionUnrefHandle` used to lock the mutex so it can call `SuperVersion::Cleanup`; however, this logic turned out to be deadlock-prone. https://github.com/facebook/rocksdb/pull/3510 fixed the deadlock but left the unlocking in place. https://github.com/facebook/rocksdb/pull/6147 then introduced the circular dependency and associated cleanup logic described above (in order to enable iterators to keep the `ColumnFamilyData` for dropped column families alive), and moved the unlocking-relocking snippet to its present location in `UnrefAndTryDelete`. Finally, https://github.com/facebook/rocksdb/pull/7749 fixed a memory leak but apparently exacerbated the race by (otherwise correctly) switching to `UnrefAndTryDelete` in `SuperVersion::Cleanup`. The patch simply eliminates the unlocking and relocking, which has been unnecessary ever since https://github.com/facebook/rocksdb/issues/3510 made `SuperVersionUnrefHandle` lock-free. This closes the window during which another thread could increase the reference count, and hopefully fixes the issues above. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8605 Test Plan: Ran `make check` and stress tests locally. Reviewed By: pdillinger Differential Revision: D30051035 Pulled By: ltamasi fbshipit-source-id: 8fe559e4b4ad69fc142579f8bc393ef525918528 | 04 August 2021, 18:00:18 UTC |
7513e90 | Jay Zhuang | 22 July 2021, 20:46:33 UTC | Update HISTORY.md and version.h for 6.23.1 | 22 July 2021, 20:46:33 UTC |
bf117c5 | Jay Zhuang | 22 July 2021, 20:41:48 UTC | Fix an race condition during multiple DB opening (#8574) Summary: ObjectLibrary is shared between multiple DB instances, the Register() could have race condition. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8574 Test Plan: pass the failed test Reviewed By: ajkr Differential Revision: D29855096 Pulled By: jay-zhuang fbshipit-source-id: 541eed0bd495d2c963d858d81e7eabf1ba16153c | 22 July 2021, 20:43:40 UTC |
c04a86a | Jay Zhuang | 17 July 2021, 00:51:28 UTC | Update HISTORY.md and version.h 6.23 release (#8552) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8552 Reviewed By: ajkr Differential Revision: D29746828 Pulled By: jay-zhuang fbshipit-source-id: 17d564895ae9cb675d455e73626b9a6717db6279 | 17 July 2021, 00:52:14 UTC |
3455ab0 | Merlin Mao | 17 July 2021, 00:03:56 UTC | Remove extra double quote in options.h (#8550) Summary: There is an extra " in options.h (`"index block""`) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8550 Test Plan: None Reviewed By: jay-zhuang Differential Revision: D29746077 Pulled By: autopear fbshipit-source-id: 2e5117296e5414b7c7440d990926bc1e567a0b4f | 17 July 2021, 00:05:25 UTC |
39a07c9 | sdong | 16 July 2021, 23:08:14 UTC | DB Stress Reopen write failure to skip WAL (#8548) Summary: When DB Stress enables write failure in reopen, WAL files are also created with a wrapper writalbe file which buffers write until fsync. However, crash test currently expects all writes to WAL is persistent. This is at odd with the unsynced bytes dropped. To work it around temporarily, we disable WAL write failure for now. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8548 Test Plan: Run db_stress. Manual printf to make sure only WAL files are skipped. Reviewed By: jay-zhuang Differential Revision: D29745095 fbshipit-source-id: 1879dd2c01abad7879ca243ee94570ec47c347f3 | 16 July 2021, 23:09:33 UTC |
a379dae | Jay Zhuang | 16 July 2021, 22:36:00 UTC | Minor Makefile update to exclude microbench as dependency (#8523) Summary: Otherwise the build may report warning about missing `benchmark.h` for some targets, the error won't break the build. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8523 Test Plan: `make blackbox_ubsan_crash_test` on a machine without benchmark lib installed. Reviewed By: pdillinger Differential Revision: D29682478 Pulled By: jay-zhuang fbshipit-source-id: e1261fbcda46bc6bd3cd39b7b03b7f78927d0430 | 16 July 2021, 22:36:51 UTC |
ac37bfd | mrambacher | 16 July 2021, 22:04:29 UTC | Allow CreateFromString to work on complex URIs (#8547) Summary: Some URIs for creating instances (ala SecondaryCache) use complex URIs like (cache://name;prop=value). These URIs were treated as name-value properties. With this change, if the URI does not contain an "id=XX" setting, it will be treated as a single string value (and not an ID and map of name-value properties). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8547 Reviewed By: anand1976 Differential Revision: D29741386 Pulled By: mrambacher fbshipit-source-id: 0621f62bec3a6699a7b66c7c0b5634b2856653aa | 16 July 2021, 22:05:45 UTC |
df5dc73 | Peter Dillinger | 16 July 2021, 21:12:06 UTC | Don't hold DB mutex for block cache entry stat scans (#8538) Summary: I previously didn't notice the DB mutex was being held during block cache entry stat scans, probably because I primarily checked for read performance regressions, because they require the block cache and are traditionally latency-sensitive. This change does some refactoring to avoid holding DB mutex and to avoid triggering and waiting for a scan in GetProperty("rocksdb.cfstats"). Some tests have to be updated because now the stats collector is populated in the Cache aggressively on DB startup rather than lazily. (I hope to clean up some of this added complexity in the future.) This change also ensures proper treatment of need_out_of_mutex for non-int DB properties. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8538 Test Plan: Added unit test logic that uses sync points to fail if the DB mutex is held during a scan, covering the various ways that a scan might be triggered. Performance test - the known impact to holding the DB mutex is on TransactionDB, and the easiest way to see the impact is to hack the scan code to almost always miss and take an artificially long time scanning. Here I've injected an unconditional 5s sleep at the call to ApplyToAllEntries. Before (hacked): $ TEST_TMPDIR=/dev/shm ./db_bench.base_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 433.219 micros/op 2308 ops/sec; 0.1 MB/s ( transactions:78999 aborts:0) rocksdb.db.write.micros P50 : 16.135883 P95 : 36.622503 P99 : 66.036115 P100 : 5000614.000000 COUNT : 149677 SUM : 8364856 $ TEST_TMPDIR=/dev/shm ./db_bench.base_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 448.802 micros/op 2228 ops/sec; 0.1 MB/s ( transactions:75999 aborts:0) rocksdb.db.write.micros P50 : 16.629221 P95 : 37.320607 P99 : 72.144341 P100 : 5000871.000000 COUNT : 143995 SUM : 13472323 Notice the 5s P100 write time. After (hacked): $ TEST_TMPDIR=/dev/shm ./db_bench.new_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 303.645 micros/op 3293 ops/sec; 0.1 MB/s ( transactions:98999 aborts:0) rocksdb.db.write.micros P50 : 16.061871 P95 : 33.978834 P99 : 60.018017 P100 : 616315.000000 COUNT : 187619 SUM : 4097407 $ TEST_TMPDIR=/dev/shm ./db_bench.new_xxx -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 310.383 micros/op 3221 ops/sec; 0.1 MB/s ( transactions:96999 aborts:0) rocksdb.db.write.micros P50 : 16.270026 P95 : 35.786844 P99 : 64.302878 P100 : 603088.000000 COUNT : 183819 SUM : 4095918 P100 write is now ~0.6s. Not good, but it's the same even if I completely bypass all the scanning code: $ TEST_TMPDIR=/dev/shm ./db_bench.new_skip -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 311.365 micros/op 3211 ops/sec; 0.1 MB/s ( transactions:96999 aborts:0) rocksdb.db.write.micros P50 : 16.274362 P95 : 36.221184 P99 : 68.809783 P100 : 649808.000000 COUNT : 183819 SUM : 4156767 $ TEST_TMPDIR=/dev/shm ./db_bench.new_skip -benchmarks=randomtransaction,stats -cache_index_and_filter_blocks=1 -bloom_bits=10 -partition_index_and_filters=1 -duration=30 -stats_dump_period_sec=12 -cache_size=100000000 -statistics -transaction_db 2>&1 | egrep 'db.db.write.micros|micros/op' randomtransaction : 308.395 micros/op 3242 ops/sec; 0.1 MB/s ( transactions:97999 aborts:0) rocksdb.db.write.micros P50 : 16.106222 P95 : 37.202403 P99 : 67.081875 P100 : 598091.000000 COUNT : 185714 SUM : 4098832 No substantial difference. Reviewed By: siying Differential Revision: D29738847 Pulled By: pdillinger fbshipit-source-id: 1c5c155f5a1b62e4fea0fd4eeb515a8b7474027b | 16 July 2021, 21:13:08 UTC |
1e5b631 | sdong | 16 July 2021, 19:27:00 UTC | db_bench seekrandom with multiDB should only create iterators queried (#7818) Summary: Right now, db_bench with seekrandom and multiple DB setup creates iterator for all DBs just to query one of them. It's different from most real workloads. Fix it by only creating iterators that will be queried. Also fix a bug that DBs are not destroyed in multi-DB mode. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7818 Test Plan: Run db_bench with single/multiDB X using/not using tailing iterator with ASAN build, and validate the behavior is expected. Reviewed By: ajkr Differential Revision: D25720226 fbshipit-source-id: c2ff7ff7120e5ba64287a30b057c5d29b2cbe20b | 16 July 2021, 19:28:10 UTC |
0229a88 | Baptiste Lemaire | 16 July 2021, 17:19:33 UTC | Crashtest mempurge (#8545) Summary: Add `experiemental_allow_mempurge` flag support for `db_stress` and `db_crashtest.py`, with a `false` default value. I succesfully tested locally both `whitebox` and `blackbox` crash tests with `experiemental_allow_mempurge` flag set as true. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8545 Reviewed By: akankshamahajan15 Differential Revision: D29734513 Pulled By: bjlemaire fbshipit-source-id: 24316c0eccf6caf409e95c035f31d822c66714ae | 16 July 2021, 17:20:22 UTC |
42ba60b | Mark Rambacher | 16 July 2021, 14:57:47 UTC | Make EncryptionProvider and BlockCipher into Customizable objects (#8354) Summary: Made the EncryptionProvider and BlockCipher classes inherit from Customizable. Added/fixed the CreateFromString method to these classes to create instances from builtin or registered classes. Added tests to verify that instances can be registered and retrieved as appropriate. Added the ability to configure the builtin (CTR, ROT13) classes from configurable properties. Added the appropriate tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8354 Reviewed By: zhichao-cao Differential Revision: D29558949 Pulled By: mrambacher fbshipit-source-id: c20286b32d179777e060f51a58943e9b0cf81d04 | 16 July 2021, 14:58:51 UTC |
aeb913d | Peter Dillinger | 16 July 2021, 06:49:12 UTC | Standardize on GCC for TSAN conditional compilation (#8543) Summary: In https://github.com/facebook/rocksdb/issues/8539 I accidentally only checked for GCC TSAN, which is what I tested locally, while CircleCI and FB CI use clang TSAN. Related: other existing code like in stack_trace.cc only check for clang TSAN. I've now standardized these to the GCC convention in port/lang.h, so now #ifdef __SANITIZE_THREAD__ can check for any TSAN (assuming lang.h include) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8543 Test Plan: Put an assert(false) in slice_test and look for the NOTE about "signal-unsafe call", both GCC and clang. Eventually, CircleCI TSAN in https://github.com/facebook/rocksdb/issues/8538 Reviewed By: zhichao-cao Differential Revision: D29728483 Pulled By: pdillinger fbshipit-source-id: 8a3b8015c2ed48078214c3ee17146a2c3f11c9f7 | 16 July 2021, 06:50:00 UTC |
b678cb1 | zaorangyang | 16 July 2021, 01:05:58 UTC | The formal parameter types of CompressionOptions constructor should b… (#8510) Summary: …e consistent with the member variables's Pull Request resolved: https://github.com/facebook/rocksdb/pull/8510 Reviewed By: ajkr Differential Revision: D29654067 Pulled By: mrambacher fbshipit-source-id: 908baaddfb20c266db7c5aca6a87971393d62ee6 | 16 July 2021, 01:06:50 UTC |
206845c | Baptiste Lemaire | 16 July 2021, 00:48:17 UTC | Mempurge support for wal (#8528) Summary: In this PR, `mempurge` is made compatible with the Write Ahead Log: in case of recovery, the DB is now capable of recovering the data that was "mempurged" and kept in the `imm()` list of immutable memtables. The twist was to add a uint64_t to the `memtable` struct to store the number of the earliest log file containing entries from the `memtable`. When a `Flush` operation is replaced with a `MemPurge`, the `VersionEdit` (which usually contains the new min log file number to pick up for recovery and the level 0 file path of the newly created SST file) is no longer appended to the manifest log, and every time the `deleteWal` method is called, a check is made on the list of immutable memtables. This PR also includes a unit test that verifies that no data is lost upon Reopening of the database when the mempurge feature is activated. This extensive unit test includes two column families, with valid data contained in the imm() at time of "crash"/reopening (recovery). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8528 Reviewed By: pdillinger Differential Revision: D29701097 Pulled By: bjlemaire fbshipit-source-id: 072a900fb6ccc1edcf5eef6caf88f3060238edf9 | 16 July 2021, 00:49:13 UTC |
4e4ec16 | longlijian | 16 July 2021, 00:22:49 UTC | Replace the namespace "rocksdb" to "ROCKSDB_NAMESPACE" (#8531) Summary: For more detail can reference the https://github.com/facebook/rocksdb/issues/6433 (https://github.com/facebook/rocksdb/pull/6433) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8531 Reviewed By: siying Differential Revision: D29717057 Pulled By: ajkr fbshipit-source-id: 3ccad9501e5612590e54a7cf8c447118f323c7f4 | 16 July 2021, 00:23:39 UTC |
5ad3227 | Peter Dillinger | 15 July 2021, 23:08:19 UTC | Work around falsely reported data race on LRUHandle::flags (#8539) Summary: Some bits are mutated and read while holding a lock, other immutable bits (esp. secondary cache compatibility) can be read by arbitrary threads without holding a lock. AFAIK, this doesn't cause an issue on any architecture we care about, because you will get some legitimate version of the value that includes the initialization, as long as synchronization guarantees the initialization happens before the read. I've only seen this in https://github.com/facebook/rocksdb/issues/8538 so far, but it should be fixed regardless. Otherwise, we'll surely get these false reports again some time. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8539 Test Plan: some local TSAN test runs and in CircleCI Reviewed By: zhichao-cao Differential Revision: D29720262 Pulled By: pdillinger fbshipit-source-id: 365fd7e565577c648815161f71b339bcb5ce12d5 | 15 July 2021, 23:09:18 UTC |
31193a7 | Jay Zhuang | 15 July 2021, 20:36:48 UTC | Add missing steps for cmake build (#8524) Summary: Some cmake and test configuration are set in pre-steps enviroment variables. Add the missing steps. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8524 Test Plan: CI pass Reviewed By: siying Differential Revision: D29682731 Pulled By: jay-zhuang fbshipit-source-id: afda1acf6a7b76989db450442b0b27f387388b9d | 15 July 2021, 20:37:49 UTC |
803a40d | longlijian | 14 July 2021, 23:04:01 UTC | Delete legacy code not used any more. (#8508) Summary: The removed function in this PR, just only have declared and dose not have any reference used. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8508 Reviewed By: mrambacher Differential Revision: D29649033 Pulled By: jay-zhuang fbshipit-source-id: df98143b73d6c184a2a60c9f7ea2548a065ee35d | 14 July 2021, 23:04:56 UTC |
8700332 | hongrubb | 14 July 2021, 01:12:03 UTC | Fix Get() return status when block cache is disabled (#8485) Summary: This PR is for https://github.com/facebook/rocksdb/issues/8453 We need to update `s = biter.status();` when `biter.status().IsIncomplete()` is true. By doing this, can fix the problem in issue. Besides, we still need to update `db_statistics` in `get_context.ReportCounters()` before return back. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8485 Reviewed By: jay-zhuang Differential Revision: D29604835 Pulled By: ajkr fbshipit-source-id: c7f2f1cd058223ce1b507ec05d57cf264b9c9710 | 14 July 2021, 01:13:24 UTC |
7b9ecd4 | sherriiiliu | 13 July 2021, 19:39:42 UTC | fix several MSVC build errors (#8519) Summary: Fixed a few MSVC (VCToolsVersion=14.0) build errors and warnings * `DEFINE_string` is a macro and VC compiler complains that it cannot put [ifdef-inside-define](https://stackoverflow.com/questions/5586429/ifdef-inside-define) * `sleep()` is not a recognizable function. Use `FLAGS_env->SleepForMicroseconds` instead * Define precise type in comparison to avoid mismatch warning Pull Request resolved: https://github.com/facebook/rocksdb/pull/8519 Reviewed By: jay-zhuang Differential Revision: D29683086 fbshipit-source-id: 8c80941472089f8daba84ae29597e75e603850e4 | 13 July 2021, 19:40:43 UTC |
e8e911a | dependabot[bot] | 13 July 2021, 00:04:40 UTC | Bump addressable from 2.7.0 to 2.8.0 in /docs (#8515) Summary: Bumps [addressable](https://github.com/sporkmonger/addressable) from 2.7.0 to 2.8.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md">addressable's changelog</a>.</em></p> <blockquote> <h1>Addressable 2.8.0</h1> <ul> <li>fixes ReDoS vulnerability in Addressable::Template#match</li> <li>no longer replaces <code>+</code> with spaces in queries for non-http(s) schemes</li> <li>fixed encoding ipv6 literals</li> <li>the <code>:compacted</code> flag for <code>normalized_query</code> now dedupes parameters</li> <li>fix broken <code>escape_component</code> alias</li> <li>dropping support for Ruby 2.0 and 2.1</li> <li>adding Ruby 3.0 compatibility for development tasks</li> <li>drop support for <code>rack-mount</code> and remove Addressable::Template#generate</li> <li>performance improvements</li> <li>switch CI/CD to GitHub Actions</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/sporkmonger/addressable/commit/6469a232c0f1892809ff66737370c765d574e16c"><code>6469a23</code></a> Updating gemspec again</li> <li><a href="https://github.com/sporkmonger/addressable/commit/24336385de0261571b3adaad0431459edb420c79"><code>2433638</code></a> Merge branch 'main' of github.com:sporkmonger/addressable into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/e9c76b889789c75d7073c17b0ab557635d3f6704"><code>e9c76b8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/378">https://github.com/facebook/rocksdb/issues/378</a> from ashmaroli/flat-map</li> <li><a href="https://github.com/sporkmonger/addressable/commit/56c5cf7ece9223ff4240e07078cc26d3adbbbd30"><code>56c5cf7</code></a> Update the gemspec</li> <li><a href="https://github.com/sporkmonger/addressable/commit/c1fed1ca0a44c448e74d761fd44ed94869199807"><code>c1fed1c</code></a> Require a non-vulnerable rake</li> <li><a href="https://github.com/sporkmonger/addressable/commit/0d8a3127e35886ce9284810a7f2438bff6b43cbc"><code>0d8a312</code></a> Adding note about ReDoS vulnerability</li> <li><a href="https://github.com/sporkmonger/addressable/commit/89c76130ce255c601f642a018cb5fb5a80e679a7"><code>89c7613</code></a> Merge branch 'template-regexp' into main</li> <li><a href="https://github.com/sporkmonger/addressable/commit/cf8884f815c96b646c796f707bf768cf6eb65543"><code>cf8884f</code></a> Note about alias fix</li> <li><a href="https://github.com/sporkmonger/addressable/commit/bb03f7112e8e478240a0f96e1cc7428159b41586"><code>bb03f71</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/sporkmonger/addressable/issues/371">https://github.com/facebook/rocksdb/issues/371</a> from charleystran/add_missing_encode_component_doc_entry</li> <li><a href="https://github.com/sporkmonger/addressable/commit/6d1d8094a66cbf932ecf69db6850bc9edaf86de0"><code>6d1d809</code></a> Adding note about :compacted normalization</li> <li>Additional commits viewable in <a href="https://github.com/sporkmonger/addressable/compare/addressable-2.7.0...addressable-2.8.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=addressable&package-manager=bundler&previous-version=2.7.0&new-version=2.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `dependabot rebase` will rebase this PR - `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `dependabot merge` will merge this PR after your CI passes on it - `dependabot squash and merge` will squash and merge this PR after your CI passes on it - `dependabot cancel merge` will cancel a previously requested merge and block automerging - `dependabot reopen` will reopen this PR if it is closed - `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/facebook/rocksdb/network/alerts). </details> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8515 Reviewed By: jay-zhuang Differential Revision: D29668988 Pulled By: ajkr fbshipit-source-id: c4b7abd4a879a7b562cb8ba745088dba6644f503 | 13 July 2021, 00:06:07 UTC |
a53d6d2 | Peter Dillinger | 12 July 2021, 23:59:22 UTC | Improve support for valgrind error on reachable (#8503) Summary: MyRocks apparently uses valgrind to check for unreachable unfreed data, which is stricter than our valgrind checks. Internal ref: D29257815 This patch adds valgrind support to STATIC_AVOID_DESTRUCTION so that it's not reported with those stricter checks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8503 Test Plan: make valgrind_test Also, with modified VALGRIND_OPTS (see Makefile), more kinds of failures seen before than after this commit. Reviewed By: ajkr, yizhang82 Differential Revision: D29597784 Pulled By: pdillinger fbshipit-source-id: 360de157a176aec4d1be99ca20d160ecd47c0873 | 13 July 2021, 00:00:27 UTC |
da90e23 | mrambacher | 12 July 2021, 19:17:32 UTC | Improvements to benchmark.sh script (#8346) Summary: 1. Fix printing of stats when there are no writes (wamp=0). Previously had a div0 error 2. Added multireadrandom command as a valid target 3. Added ability to pass additional command line options to db_bench. Now can say things like benchmark.sh readrandom --mmap_read and the option will be passed to db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8346 Reviewed By: zhichao-cao Differential Revision: D29500436 Pulled By: mrambacher fbshipit-source-id: 54e90708aae9133be3a903e35efdf8f8abbd86fa | 12 July 2021, 19:18:17 UTC |
955b80e | bjlemaire | 12 July 2021, 17:41:28 UTC | Add WARN/INFO for mempurge output status. (#8514) Summary: The MemPurge output status can either be an Abort if the mempurge is aborted due to the new_mem memtable reaching more than the target capacity (currently 60%), or for other reasons. As a result, in the log, we want to differentiate between an abort status, which in this PR only leads to a ROCKS_LOG_INFO, and any other status, which in this PR leads to a ROCKS_LOG_WARN. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8514 Reviewed By: pdillinger Differential Revision: D29662446 Pulled By: bjlemaire fbshipit-source-id: c9bec8e238ebc7ecb14fbbddf580e6887e281c16 | 12 July 2021, 17:42:14 UTC |
0b75b22 | Dmitry Vorobev | 12 July 2021, 16:04:40 UTC | Implement missing Handler methods in ColumnFamilyCollector. (#8456) Summary: When db is open as secondary, there are basically 2 step process: 1) Collect column families from wal log 2) Apply changes to Memtable In case primary db is TransactionDB instance, wal log will contain some additional data, like noop, etc. ColumnFamilyCollector doesn't implement methods to handle these, so it fails to open a wal log written by TransactionDB. (Everything works fine with standard DB::Open). Memtable recovery process knows how to handle such wal logs, so only missing piece seems to be ColumnFamilyCollector. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8456 Reviewed By: ajkr Differential Revision: D29455945 Pulled By: mrambacher fbshipit-source-id: 5b29560fcbc008e17e95d0dc4b07558f3d63e26f | 12 July 2021, 16:23:45 UTC |
bbdc4f2 | Myth | 12 July 2021, 16:04:08 UTC | Fix a minor issue in checkpoint test case (#8483) Summary: A very simple change :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8483 Reviewed By: ajkr Differential Revision: D29558904 Pulled By: mrambacher fbshipit-source-id: bbe68c20c861103726cb6231ca3fb8fbe1e5a546 | 12 July 2021, 16:09:09 UTC |
c866561 | mrambacher | 12 July 2021, 16:03:41 UTC | Make FlushBlockPolicyFactory into a Customizable class (#8432) Summary: Add ability to treat FlushBlockPolicyFactory as a Customizable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8432 Reviewed By: zhichao-cao Differential Revision: D29558941 Pulled By: mrambacher fbshipit-source-id: 4a791af941ea4a65fc2f1fdfb1d7a95f42ca6774 | 12 July 2021, 16:04:59 UTC |
5afd1e3 | Adam Retter | 12 July 2021, 12:04:23 UTC | Correct CVS -> CSV typo (#8513) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8513 Reviewed By: jay-zhuang Differential Revision: D29654066 Pulled By: mrambacher fbshipit-source-id: b8f492fe21edd37fe1f1c5a4a0e9153f58bbf3e2 | 12 July 2021, 12:05:16 UTC |
d1b70b0 | anand76 | 12 July 2021, 05:37:01 UTC | Avoid passing existing BG error to WriteStatusCheck (#8511) Summary: In ```DBImpl::WriteImpl()```, we call ```PreprocessWrite()``` which, among other things, checks the BG error and returns it set. This return status is later on passed to ```WriteStatusCheck()```, which calls ```SetBGError()```. This results in a spurious call, and info logs, on every user write request. We should avoid passing the ```PreprocessWrite()``` return status to ```WriteStatusCheck()```, as the former would have called ```SetBGError()``` already if it encountered any new errors, such as error when creating a new WAL file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8511 Test Plan: Run existing tests Reviewed By: zhichao-cao Differential Revision: D29639917 Pulled By: anand1976 fbshipit-source-id: 19234163969e1645dbeb273712aaf5cd9ea2b182 | 12 July 2021, 05:37:52 UTC |
837705a | Baptiste Lemaire | 10 July 2021, 00:16:00 UTC | Make mempurge a background process (equivalent to in-memory compaction). (#8505) Summary: In https://github.com/facebook/rocksdb/issues/8454, I introduced a new process baptized `MemPurge` (memtable garbage collection). This new PR is built upon this past mempurge prototype. In this PR, I made the `mempurge` process a background task, which provides superior performance since the mempurge process does not cling on the db_mutex anymore, and addresses severe restrictions from the past iteration (including a scenario where the past mempurge was failling, when a memtable was mempurged but was still referred to by an iterator/snapshot/...). Now the mempurge process ressembles an in-memory compaction process: the stack of immutable memtables is filtered out, and the useful payload is used to populate an output memtable. If the output memtable is filled at more than 60% capacity (arbitrary heuristic) the mempurge process is aborted and a regular flush process takes place, else the output memtable is kept in the immutable memtable stack. Note that adding this output memtable to the `imm()` memtable stack does not trigger another flush process, so that the flush thread can go to sleep at the end of a successful mempurge. MemPurge is activated by making the `experimental_allow_mempurge` flag `true`. When activated, the `MemPurge` process will always happen when the flush reason is `kWriteBufferFull`. The 3 unit tests confirm that this process supports `Put`, `Get`, `Delete`, `DeleteRange` operators and is compatible with `Iterators` and `CompactionFilters`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8505 Reviewed By: pdillinger Differential Revision: D29619283 Pulled By: bjlemaire fbshipit-source-id: 8a99bee76b63a8211bff1a00e0ae32360aaece95 | 10 July 2021, 00:23:59 UTC |
bb485e9 | qieqieplus | 09 July 2021, 22:44:57 UTC | Add ribbon filter to C API (#8486) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8486 Reviewed By: jay-zhuang Differential Revision: D29625501 Pulled By: pdillinger fbshipit-source-id: e6e2a455ae62a71f3a202278a751b9bba17ad03c | 09 July 2021, 23:22:48 UTC |
5dd18a8 | Jay Zhuang | 09 July 2021, 00:50:55 UTC | Add micro-benchmark support (#8493) Summary: Add google benchmark for microbench. Add ribbon_bench for benchmark ribbon filter vs. other filters. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8493 Test Plan: added test to CI To run the benchmark on devhost: Install benchmark: `$ sudo dnf install google-benchmark-devel` Build and run: `$ ROCKSDB_NO_FBCODE=1 DEBUG_LEVEL=0 make microbench` or with cmake: `$ mkdir build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_BENCHMARK=1 && make microbench` Reviewed By: pdillinger Differential Revision: D29589649 Pulled By: jay-zhuang fbshipit-source-id: 8fed13b562bef4472f161ecacec1ab6b18911dff | 09 July 2021, 01:22:45 UTC |
f127d45 | sdong | 08 July 2021, 17:50:08 UTC | Add comments to options.bottommost_compression (#8415) Summary: Add comments to options.bottommost_compression for options.num_levels=1 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8415 Reviewed By: ajkr Differential Revision: D29181997 fbshipit-source-id: 5f0f49470f75d796320ecb24d5dc4ef4eb6fbe0f | 08 July 2021, 17:50:59 UTC |
ac3f3f3 | longlijian | 08 July 2021, 17:07:37 UTC | Eliminate compiler complaining, which the return type of the function… (#8498) Summary: … should be uint64_t. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8498 Reviewed By: jay-zhuang Differential Revision: D29605064 Pulled By: ajkr fbshipit-source-id: e431448ac9d8a37ae83679c4cc5732e29fe49de4 | 08 July 2021, 17:09:05 UTC |
b1a53db | sdong | 07 July 2021, 23:20:40 UTC | FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync() to recover… (#8501) Summary: … small overwritten files. If a file is overwritten with renamed and the parent path is not synced, FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync() will delete the file. However, RocksDB relies on file renaming to be atomic no matter whether the parent directory is synced or not, and the current behavior breaks the assumption and caused some false positive: https://github.com/facebook/rocksdb/pull/8489 Since the atomic renaming is used in CURRENT files, to fix the problem, in FaultInjectionTestFS::DeleteFilesCreatedAfterLastDirSync(), we recover the state of overwritten file if the file is small. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8501 Test Plan: Run stress test for a while and see it doesn't break. Reviewed By: anand1976 Differential Revision: D29594384 fbshipit-source-id: 589b5c2f0a9d2aca53752d7bdb0231efa5b3ae92 | 07 July 2021, 23:23:23 UTC |
ed8eb43 | Andrew Kryczka | 07 July 2021, 18:13:09 UTC | Move slow valgrind tests behind -DROCKSDB_FULL_VALGRIND_RUN (#8475) Summary: Various tests had disabled valgrind due to it slowing down and timing out (as is the case right now) the CI runs. Where a test was disabled with no comment, I assumed slowness was the cause. For these tests that were slow under valgrind, as well as the ones identified in https://github.com/facebook/rocksdb/issues/8352, this PR moves them behind the compiler flag `-DROCKSDB_FULL_VALGRIND_RUN`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8475 Test Plan: running `make full_valgrind_test`, `make valgrind_test`, `make check`; will verify they appear working correctly Reviewed By: jay-zhuang Differential Revision: D29504843 Pulled By: ajkr fbshipit-source-id: 2aac90749cfbd30d5ce11cb29a07a1b9314eeea7 | 07 July 2021, 18:14:05 UTC |
714ce50 | Baptiste Lemaire | 07 July 2021, 01:47:59 UTC | Fix clang_analyzer failure (#8492) Summary: Previously, the following command: ```USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j$(nproc) analyze``` was raising an error/warning the new_mem could potentially be a `nullptr`. This error appeared due to code changes from https://github.com/facebook/rocksdb/issues/8454, including an if-statement containing "`... && new_mem != nullptr && ...`", which made the analyzer believe that past this `if`-statement, a `new_mem==nullptr` was a possible scenario. This code patch simply introduces `assert`s and removes this condition in the `if`-statement. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8492 Reviewed By: jay-zhuang Differential Revision: D29571275 Pulled By: bjlemaire fbshipit-source-id: 75d72246b70ebbbae7dea11ccb5778686d8bcbea | 07 July 2021, 01:48:56 UTC |
df4197c | anand76 | 06 July 2021, 23:44:40 UTC | Bypass buffer in TestFSWritableFile if direct IO is enabled (#8490) Summary: ```TestFSWritableFile``` buffers data in ```Append``` in order to simulate unsynced data loss on crash. This is only required for buffered IO and should be disabled for direct IO. Otherwise, it causes crash tests to assert on the buffer address alignment - ```db_stress: env/io_posix.cc:1194: virtual rocksdb::IOStatus rocksdb::PosixWritableFile::Append(const rocksdb::Slice&, const rocksdb::IOOptions&, rocksdb::IODebugContext*): Assertion `IsSectorAligned(data.data(), GetRequiredBufferAlignment())' failed.```. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8490 Reviewed By: zhichao-cao Differential Revision: D29565080 Pulled By: anand1976 fbshipit-source-id: 682831fd66ed3b9597caa74fc453e22dfaf9b973 | 06 July 2021, 23:46:16 UTC |
fcd8088 | anand76 | 06 July 2021, 21:15:25 UTC | Temporarily disable file deletion after open failure in db_stress (#8489) Summary: Write and metadata error injection during DB open was enabled in https://github.com/facebook/rocksdb/issues/8474. This causes crash tests to fail very frequently due to another fault injection feature that deletes files created after the last dir sync during DB open. In real life, a similar failure would happen if the FS returns error on the CURRENT file rename, but the rename actually succeeded and got partially persisted (dir entry for the old CURRENT file got removed, but the entry for the new one is not persisted). Temporarily disable the fault injection feature until we figure out the likelihood of this bug happening and the proper way to fix it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8489 Test Plan: Stress test can open the DB successfully Reviewed By: siying Differential Revision: D29564516 Pulled By: anand1976 fbshipit-source-id: ffd1650715ea3c5bf7131936b0ca6fcf66f4e14e | 06 July 2021, 21:16:57 UTC |
f33611d | sdong | 06 July 2021, 18:04:04 UTC | Stress test to inject read failures in DB reopen (#8476) Summary: Inject read failures in DB reopen, just as what we do for metadata writes and writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8476 Test Plan: Some manual tests and make sure failures are triggered. Reviewed By: anand1976 Differential Revision: D29507283 fbshipit-source-id: d04da0163973447041038bd87701686a417c4e0c | 06 July 2021, 18:05:27 UTC |
1ae026c | Levi Tamasi | 06 July 2021, 17:13:40 UTC | Partially revert the "apply subrange of table property collectors" change (#8465) Summary: We ended up using a different approach for tracking the amount of garbage in blob files (see e.g. https://github.com/facebook/rocksdb/pull/8450), so the ability to apply only a range of table property collectors is now unnecessary. The patch reverts this part of https://github.com/facebook/rocksdb/pull/8298 while keeping the cleanup done in that PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8465 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29399921 Pulled By: ltamasi fbshipit-source-id: af64816c357d0829b9d7ba8ca1477038138f6f0a | 06 July 2021, 17:14:32 UTC |
570248a | mrambacher | 06 July 2021, 16:17:13 UTC | Make SecondaryCache Customizable (#8480) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8480 Reviewed By: zhichao-cao Differential Revision: D29528740 Pulled By: mrambacher fbshipit-source-id: fd0f70d15f66611c8498257a9973f7e98ca13839 | 06 July 2021, 16:18:08 UTC |
9dc887e | Baptiste Lemaire | 02 July 2021, 12:22:03 UTC | Memtable "MemPurge" prototype (#8454) Summary: Implement an experimental feature called "MemPurge", which consists in purging "garbage" bytes out of a memtable and reuse the memtable struct instead of making it immutable and eventually flushing its content to storage. The prototype is by default deactivated and is not intended for use. It is intended for correctness and validation testing. At the moment, the "MemPurge" feature can be switched on by using the `options.experimental_allow_mempurge` flag. For this early stage, when the allow_mempurge flag is set to `true`, all the flush operations will be rerouted to perform a MemPurge. This is a temporary design decision that will give us the time to explore meaningful heuristics to use MemPurge at the right time for relevant workloads . Moreover, the current MemPurge operation only supports `Puts`, `Deletes`, `DeleteRange` operations, and handles `Iterators` as well as `CompactionFilter`s that are invoked at flush time . Three unit tests are added to `db_flush_test.cc` to test if MemPurge works correctly (and checks that the previously mentioned operations are fully supported thoroughly tested). One noticeable design decision is the timing of the MemPurge operation in the memtable workflow: for this prototype, the mempurge happens when the memtable is switched (and usually made immutable). This is an inefficient process because it implies that the entirety of the MemPurge operation happens while holding the db_mutex. Future commits will make the MemPurge operation a background task (akin to the regular flush operation) and aim at drastically enhancing the performance of this operation. The MemPurge is also not fully "WAL-compatible" yet, but when the WAL is full, or when the regular MemPurge operation fails (or when the purged memtable still needs to be flushed), a regular flush operation takes place. Later commits will also correct these behaviors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8454 Reviewed By: anand1976 Differential Revision: D29433971 Pulled By: bjlemaire fbshipit-source-id: 6af48213554e35048a7e03816955100a80a26dc5 | 02 July 2021, 12:23:02 UTC |
c76778e | Akanksha Mahajan | 02 July 2021, 02:17:21 UTC | Call OnCompactionCompleted API in case of DisableManualCompaction (#8469) Summary: Call OnCompactionCompleted API in case of DisableManualCompaction() with updated Status::Incomplete Pull Request resolved: https://github.com/facebook/rocksdb/pull/8469 Reviewed By: ajkr Differential Revision: D29475517 Pulled By: akankshamahajan15 fbshipit-source-id: a1726c5e6ee18c0b5097ea04f5e6975fbe108055 | 02 July 2021, 02:18:55 UTC |
b207377 | Peter (Stig) Edwards | 02 July 2021, 01:41:20 UTC | Add -report_open_timing to db_bench (#8464) Summary: Hello and thanks for RocksDB, This PR adds support for ```-report_open_timing true``` to ```db_bench```. It can be useful when tuning RocksDB on filesystem/env with high latencies for file level operations (create/delete/rename...) seen during ```((Optimistic)Transaction)DB::Open```. Some examples: ``` > db_bench -benchmarks updaterandom -num 1 -db /dev/shm/db_bench > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 3.90133 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 3.33414 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /dev/shm/db_bench -use_existing_db true -report_open_timing true 2>&1 | grep -A1 OpenDb OpenDb: 6.05423 milliseconds > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 4.06859 milliseconds > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 2.85794 milliseconds > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_open_timing true 2>&1 | grep OpenDb OpenDb: 6.46376 milliseconds > db_bench -benchmarks updaterandom -num 1 -db /clustered_fs/db_bench > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true -readonly true 2>&1 | grep OpenDb OpenDb: 3.79805 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true -use_secondary_db true 2>&1 | grep OpenDb OpenDb: 3.00174 milliseconds > db_bench -benchmarks updaterandom -num 0 -db /clustered_fs/db_bench -use_existing_db true -report_open_timing true 2>&1 | grep OpenDb OpenDb: 24.8732 milliseconds ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8464 Reviewed By: hx235 Differential Revision: D29398096 Pulled By: zhichao-cao fbshipit-source-id: 8f05dc3284f084612a3f30234e39e1c37548f50c | 02 July 2021, 01:42:19 UTC |
a95a776 | Zhichao Cao | 01 July 2021, 21:15:49 UTC | Inject fatal write failures to db_stress when DB is running (#8479) Summary: add the injest_error_severity to control if it is a retryable IO Error or a fatal or unrecoverable error. Use a flag to indicate, if fatal error comes, the flag is set and db is stopped (but not corrupted). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8479 Test Plan: run ./db_stress --reopen=0 --read_fault_one_in=1000 --write_fault_one_in=5 --disable_wal=true --write_buffer_size=3000000 -writepercent=5 -readpercent=50 --injest_error_severity=2 --column_families=1, make check Reviewed By: anand1976 Differential Revision: D29524271 Pulled By: zhichao-cao fbshipit-source-id: 1aa9fb9b5655b0adba6f5ad12005ca8c074c795b | 01 July 2021, 21:16:47 UTC |
41d3215 | anand76 | 01 July 2021, 19:22:01 UTC | Enable crash test to run using fbcode components (#8471) Summary: Add a new test ```fbcode_crash_test``` to rocksdb-lego-determinator. This test allows the crash test to be run on Facebook Sandcastle infra using fbcode components. Also use the default Env in db_stress to access the expected values path as it requires a memory mapped file and may not work with custom Envs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8471 Reviewed By: ajkr Differential Revision: D29474722 Pulled By: anand1976 fbshipit-source-id: 7d086d82dd7091ae48e08cb4ace763ce3e3b87ef | 01 July 2021, 19:23:01 UTC |
d45b837 | mrambacher | 01 July 2021, 18:52:18 UTC | Fix TSAN issue (#8477) Summary: Added mutex to fix TSAN issue Pull Request resolved: https://github.com/facebook/rocksdb/pull/8477 Reviewed By: zhichao-cao Differential Revision: D29517053 Pulled By: mrambacher fbshipit-source-id: 661ccb1f495b7d34874a79e0a3d7aea1123d6047 | 01 July 2021, 18:53:18 UTC |
ba224b7 | sdong | 30 June 2021, 23:45:44 UTC | Stress Test to inject write failures in reopen (#8474) Summary: Previously Stress can inject metadata write failures when reopening a DB. We extend it to file append too, in the same way. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8474 Test Plan: manually run crash test with various setting and make sure the failures are triggered as expected. Reviewed By: zhichao-cao Differential Revision: D29503116 fbshipit-source-id: e73a446e80ccbd09301a579280e56ff949381fab | 30 June 2021, 23:46:41 UTC |
41c4b66 | mrambacher | 30 June 2021, 21:08:19 UTC | Fix PrepareOptions for Customizable Classes (#8468) Summary: Added the Customizable::ConfigureNewObject method. The method will configure the object if options are found and invoke PrepareOptions if the flag is set properly. Added tests to test that PrepareOptions is properly called and to test if PrepareOptions fails. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8468 Reviewed By: zhichao-cao Differential Revision: D29494703 Pulled By: mrambacher fbshipit-source-id: d5767dee5d7a98620ac66190262101cd0aa9d2b7 | 30 June 2021, 21:09:36 UTC |
a0cbb69 | anand76 | 30 June 2021, 20:28:45 UTC | Fix assertion failure when releasing a handle after secondary cache lookup fails (#8470) Summary: When the secondary cache lookup fails, we may still allocate a handle and charge the cache for metadata usage. If the cache is full, this can cause the usage to go over capacity. Later, when a (unrelated) handle is released, it trips up an assertion that checks that usage is less than capacity. To prevent this assertion failure, don't charge the cache for a failed secondary cache lookup. Tests: Run crash_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/8470 Reviewed By: zhichao-cao Differential Revision: D29474713 Pulled By: anand1976 fbshipit-source-id: 27191969c95470a7b070d292b458efce71395bf2 | 30 June 2021, 20:29:30 UTC |
93a7389 | Jay Zhuang | 29 June 2021, 18:46:44 UTC | Add statistics support on CompactionService remote side (#8368) Summary: Add statistics option on CompactionService remote side. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8368 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D28944427 Pulled By: jay-zhuang fbshipit-source-id: 2a19217f4a69b6e511af87eed12391860ef00c5e | 29 June 2021, 18:48:14 UTC |
3503f28 | Jay Zhuang | 29 June 2021, 17:41:22 UTC | Add sub-compaction support for RemoteCompaction (#8364) Summary: Change the job_id for remote compaction interface, which will include both internal compaction job_id, also a sub_compaction_job_id. It is not a backward compatible change. The user needs to update interface during upgrade. (We will avoid backward incompatible change after the feature is not experimental.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/8364 Reviewed By: ajkr Differential Revision: D28917301 Pulled By: jay-zhuang fbshipit-source-id: 6d72a21f652bb517ad6954d0387b496797fc4e11 | 29 June 2021, 17:42:19 UTC |
89f66d4 | mrambacher | 29 June 2021, 16:07:10 UTC | Add customizable_util.h to the public API (#8301) Summary: Useful for allowing new classes to create and manage Customizable objects without using internal APIs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8301 Reviewed By: zhichao-cao Differential Revision: D29428303 Pulled By: mrambacher fbshipit-source-id: 3d33d5197cc8379fe35b54d3d169f91f0dfe7a47 | 29 June 2021, 16:08:57 UTC |
bac3994 | Andrew Kryczka | 29 June 2021, 15:38:57 UTC | jemalloc_helper: Limit the mm_malloc.h hack to glibc on linux (#8425) Summary: Original author: kraj (https://github.com/facebook/rocksdb/issues/8413) We have a hack to ensure clang's `posix_memalign()` hack works to be compatible with glibc's `posix_memalign()` declaration. Our side of the hack is irrelevant and should be omitted when not using glibc. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8425 Reviewed By: mrambacher Differential Revision: D29239029 Pulled By: ajkr fbshipit-source-id: 12b900f50a4823b880a6558f25d8590dbfc0aa26 | 29 June 2021, 15:40:02 UTC |
390c524 | Lucian Petrut | 29 June 2021, 13:51:11 UTC | Allow using WindowsThread with Mingw (#8108) Summary: Allow using WindowsThread with Mingw Most Mingw builds require Posix threads in order to use std::thread. As per https://github.com/facebook/rocksdb/issues/7764, this is not always the case. That being considered, we're going to improve the Mingw thread model checks. Closes: https://github.com/facebook/rocksdb/issues/7764 Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8108 Reviewed By: jay-zhuang Differential Revision: D27365778 Pulled By: mrambacher fbshipit-source-id: 2c15b1f04ae90e1e3a25a33e86ceb779224a9529 | 29 June 2021, 13:52:08 UTC |
373e3a1 | mrambacher | 28 June 2021, 19:27:39 UTC | Fix Immutable Customizable Serialization (#8457) Summary: If a Customizable option was not mutable, it would still appear in the list of mutable options when serialized. This meant that when the immutable options were used to configure another immutable object, an "option not changeable" status would be returned. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8457 Reviewed By: zhichao-cao Differential Revision: D29428298 Pulled By: mrambacher fbshipit-source-id: 3945b0b822f8e5955a7c5590fe64dfd5bc1fe6a0 | 28 June 2021, 19:28:28 UTC |
be21908 | mrambacher | 28 June 2021, 15:12:32 UTC | Add BlobMetaData retrieval methods (#8273) Summary: Added BlobMetaData to ColumnFamilyMetaData and LiveBlobMetaData and DB API GetLiveBlobMetaData to retrieve it. First pass at struct. More tests and maybe fields to come... Pull Request resolved: https://github.com/facebook/rocksdb/pull/8273 Reviewed By: ltamasi Differential Revision: D29102400 Pulled By: mrambacher fbshipit-source-id: 8a2383a4446328be6b91dced9841fdd3dfc80b73 | 28 June 2021, 15:13:29 UTC |
6f9ed59 | anand76 | 28 June 2021, 06:53:47 UTC | Allow db_stress to use a secondary cache (#8455) Summary: Add a ```-secondary_cache_uri``` to db_stress to allow the user to specify a custom ```SecondaryCache``` object from the object registry. Also allow db_crashtest.py to be run with an alternate db_stress location. Together, these changes will allow us to run db_stress using FB internal components. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8455 Reviewed By: zhichao-cao Differential Revision: D29371972 Pulled By: anand1976 fbshipit-source-id: dd1b1fd80ebbedc11aa63d9246ea6ae49edb77c4 | 28 June 2021, 06:54:39 UTC |
be8199c | Akanksha Mahajan | 25 June 2021, 17:44:55 UTC | Run Merge with Integrated BlobDB in stress, crash and db_bench (#8461) Summary: Run Merge with Intergrated BlobDB in stress tests, crash tests and db_bench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8461 Test Plan: 1. python3 -u tools/db_crashtest.py --simple whitebox ---use_merge=1 --enable_blob_files=1 2. ./db_bench --benchmarks="readwhilemerging" --merge_operator=uint64add --enable_blob_files=true Reviewed By: ltamasi Differential Revision: D29394824 Pulled By: akankshamahajan15 fbshipit-source-id: 0a8e492b13129673e088fb8af3402ab678bb473a | 25 June 2021, 17:45:52 UTC |
a904c62 | Zhichao Cao | 25 June 2021, 07:46:33 UTC | Using existing crc32c checksum in checksum handoff for Manifest and WAL (#8412) Summary: In PR https://github.com/facebook/rocksdb/issues/7523 , checksum handoff is introduced in RocksDB for WAL, Manifest, and SST files. When user enable checksum handoff for a certain type of file, before the data is written to the lower layer storage system, we calculate the checksum (crc32c) of each piece of data and pass the checksum down with the data, such that data verification can be down by the lower layer storage system if it has the capability. However, it cannot cover the whole lifetime of the data in the memory and also it potentially introduces extra checksum calculation overhead. In this PR, we introduce a new interface in WritableFileWriter::Append, which allows the caller be able to pass the data and the checksum (crc32c) together. In this way, WritableFileWriter can directly use the pass-in checksum (crc32c) to generate the checksum of data being passed down to the storage system. It saves the calculation overhead and achieves higher protection coverage. When a new checksum is added with the data, we use Crc32cCombine https://github.com/facebook/rocksdb/issues/8305 to combine the existing checksum and the new checksum. To avoid the segmenting of data by rate-limiter before it is stored, rate-limiter is called enough times to accumulate enough credits for a certain write. This design only support Manifest and WAL which use log_writer in the current stage. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8412 Test Plan: make check, add new testing cases. Reviewed By: anand1976 Differential Revision: D29151545 Pulled By: zhichao-cao fbshipit-source-id: 75e2278c5126cfd58393c67b1efd18dcc7a30772 | 25 June 2021, 07:47:17 UTC |
3d844df | Andrew Kryczka | 25 June 2021, 04:04:05 UTC | add missing fields to `GetLiveFilesMetaData()` (#8460) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8460 Reviewed By: jay-zhuang Differential Revision: D29381865 Pulled By: ajkr fbshipit-source-id: 47ba54c25f3cc039d72ea32e1df20875795683b3 | 25 June 2021, 04:05:03 UTC |
95d0ee9 | Akanksha Mahajan | 25 June 2021, 01:10:32 UTC | Add support for Merge with base value during Compaction in IntegratedBlobDB (#8445) Summary: Provide support for Merge operation with base values during Compaction in IntegratedBlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8445 Test Plan: Add new unit test Reviewed By: ltamasi Differential Revision: D29343949 Pulled By: akankshamahajan15 fbshipit-source-id: 844f6f02f93388a11e6e08bda7bb3a2a28e47c70 | 25 June 2021, 01:11:30 UTC |
66b62a1 | Levi Tamasi | 25 June 2021, 00:24:41 UTC | Update HISTORY.md for PR 8450 (#8458) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8458 Reviewed By: zhichao-cao Differential Revision: D29378728 Pulled By: ltamasi fbshipit-source-id: d5a40b1414500f53823763be5c2bfce8db04daf8 | 25 June 2021, 00:26:32 UTC |
68d8b28 | Levi Tamasi | 24 June 2021, 23:11:07 UTC | Log the amount of blob garbage generated by compactions in the MANIFEST (#8450) Summary: The patch builds on `BlobGarbageMeter` and `BlobCountingIterator` (introduced in https://github.com/facebook/rocksdb/issues/8426 and https://github.com/facebook/rocksdb/issues/8443 respectively) and ties it all together. It measures the amount of garbage generated by a compaction and logs the corresponding `BlobFileGarbage` records as part of the compaction job's `VersionEdit`. Note: in order to have accurate results, `kRemoveAndSkipUntil` for compaction filters is implemented using iteration. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8450 Test Plan: Ran `make check` and the crash test script. Reviewed By: jay-zhuang Differential Revision: D29338207 Pulled By: ltamasi fbshipit-source-id: 4381c432ac215139439f6d6fb801a6c0e4d8c128 | 24 June 2021, 23:11:56 UTC |
75741eb | Peter (Stig) Edwards | 24 June 2021, 18:56:03 UTC | Add more ops to: db_bench -report_file_operations (#8448) Summary: Hello and thanks for RocksDB, Here is a PR to add file deletes, renames and ```Flush()```, ```Sync()```, ```Fsync()``` and ```Close()``` to file ops report. The reason is to help tune RocksDB options when using an env/filesystem with high latencies for file level ("metadata") operations, typically seen during ```DB::Open``` (```db_bench -num 0``` also see https://github.com/facebook/rocksdb/pull/7203 where IOTracing does not trace ```DB::Open```). Before: ``` > db_bench -benchmarks updaterandom -num 0 -report_file_operations true ... Entries: 0 ... Num files opened: 12 Num Read(): 6 Num Append(): 8 Num bytes read: 6216 Num bytes written: 6289 ``` After: ``` > db_bench -benchmarks updaterandom -num 0 -report_file_operations true ... Entries: 0 ... Num files opened: 12 Num files deleted: 3 Num files renamed: 4 Num Flush(): 10 Num Sync(): 5 Num Fsync(): 1 Num Close(): 2 Num Read(): 6 Num Append(): 8 Num bytes read: 6216 Num bytes written: 6289 ``` Before: ``` > db_bench -benchmarks updaterandom -report_file_operations true ... Entries: 1000000 ... Num files opened: 18 Num Read(): 396339 Num Append(): 1000058 Num bytes read: 892030224 Num bytes written: 187569238 ``` After: ``` > db_bench -benchmarks updaterandom -report_file_operations true ... Entries: 1000000 ... Num files opened: 18 Num files deleted: 5 Num files renamed: 4 Num Flush(): 1000068 Num Sync(): 9 Num Fsync(): 1 Num Close(): 6 Num Read(): 396339 Num Append(): 1000058 Num bytes read: 892030224 Num bytes written: 187569238 ``` Another example showing how using ```DB::OpenForReadOnly``` reduces file operations compared to ```((Optimistic)Transaction)DB::Open```: ``` > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -readonly true -report_file_operations true ... Entries: 0 ... Num files opened: 8 Num files deleted: 0 Num files renamed: 0 Num Flush(): 0 Num Sync(): 0 Num Fsync(): 0 Num Close(): 0 Num Read(): 13 Num Append(): 0 Num bytes read: 374 Num bytes written: 0 ``` ``` > db_bench -benchmarks updaterandom -num 1 > db_bench -benchmarks updaterandom -num 0 -use_existing_db true -report_file_operations true ... Entries: 0 ... Num files opened: 14 Num files deleted: 3 Num files renamed: 4 Num Flush(): 14 Num Sync(): 5 Num Fsync(): 1 Num Close(): 3 Num Read(): 11 Num Append(): 10 Num bytes read: 7291 Num bytes written: 7357 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8448 Reviewed By: anand1976 Differential Revision: D29333818 Pulled By: zhichao-cao fbshipit-source-id: a06a8c87f799806462319115195b3e94faf5f542 | 24 June 2021, 18:56:51 UTC |
d44ef2e | Levi Tamasi | 23 June 2021, 20:27:50 UTC | Remove obsolete method VersionSet::VerifyCompactionFileConsistency (#8449) Summary: `VersionSet::VerifyCompactionFileConsistency` was superseded by the LSM tree consistency checks introduced in https://github.com/facebook/rocksdb/pull/6901, which are more comprehensive, more efficient, and are performed unconditionally even in release builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8449 Test Plan: `make check` Reviewed By: ajkr Differential Revision: D29337441 Pulled By: ltamasi fbshipit-source-id: a05324f88e3400e27e6a00406c878a6276e0c9cc | 23 June 2021, 20:28:34 UTC |
6adc39e | Levi Tamasi | 23 June 2021, 17:24:39 UTC | Add an internal iterator that can measure the inflow of blobs (#8443) Summary: Follow-up to https://github.com/facebook/rocksdb/issues/8426 . The patch adds a new kind of `InternalIterator` that wraps another one and passes each key-value encountered to `BlobGarbageMeter` as inflow. This iterator will be used as an input iterator for compactions when the input SSTs reference blob files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8443 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29311987 Pulled By: ltamasi fbshipit-source-id: b4493b4c0c0c2e3c2ecc33c8969a5ef02de5d9d8 | 23 June 2021, 17:25:47 UTC |
3f20925 | Baptiste Lemaire | 23 June 2021, 02:06:44 UTC | Add list live files metadata (#8446) Summary: Add an argument to ldb to dump live file names, column families, and levels, `list_live_files_metadata`. The output shows all active SST file names, sorted first by column family and then by level. For each level the SST files are sorted alphabetically. Typically, the output looks like this: ``` ./ldb --db=/tmp/test_db list_live_files_metadata Live SST Files: ===== Column Family: default ===== ---------- level 0 ---------- /tmp/test_db/000069.sst ---------- level 1 ---------- /tmp/test_db/000064.sst /tmp/test_db/000065.sst /tmp/test_db/000066.sst /tmp/test_db/000071.sst ---------- level 2 ---------- /tmp/test_db/000038.sst /tmp/test_db/000039.sst /tmp/test_db/000052.sst /tmp/test_db/000067.sst /tmp/test_db/000070.sst ------------------------------ ``` Second, a flag was added `--sort_by_filename`, to change the layout of the output. When this flag is added to the command, the output shows all active SST files sorted by name, in front of which the LSM level and the column family are mentioned. With the same example, the following command would return: ``` ./ldb --db=/tmp/test_db list_live_files_metadata --sort_by_filename Live SST Files: /tmp/test_db/000038.sst : level 2, column family 'default' /tmp/test_db/000039.sst : level 2, column family 'default' /tmp/test_db/000052.sst : level 2, column family 'default' /tmp/test_db/000064.sst : level 1, column family 'default' /tmp/test_db/000065.sst : level 1, column family 'default' /tmp/test_db/000066.sst : level 1, column family 'default' /tmp/test_db/000067.sst : level 2, column family 'default' /tmp/test_db/000069.sst : level 0, column family 'default' /tmp/test_db/000070.sst : level 2, column family 'default' /tmp/test_db/000071.sst : level 1, column family 'default' ------------------------------ ``` Thus, the user can either request to show the files by levels, or sorted by filenames. This PR includes a simple Python unit test that makes sure the file name and level printed out by this new feature matches the one found with an existing feature, `dump_live_file`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8446 Reviewed By: akankshamahajan15 Differential Revision: D29320080 Pulled By: bjlemaire fbshipit-source-id: 01fb7b5637c59010d74c80730a28d815994e7009 | 23 June 2021, 02:07:46 UTC |
3ab0eae | Levi Tamasi | 23 June 2021, 00:33:15 UTC | Deflake BlobDBTest.SnapshotAndGarbageCollection (#8444) Summary: This test case has been failing occasionally due to automatic compactions kicking in, resulting in GC generating additional blob files that the test did not expect. Disabling automatic compactions to get rid of this flakiness. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8444 Test Plan: `gtest-parallel --repeat=1000 ./blob_db_test --gtest_filter="BlobDBTest.SnapshotAndGarbageCollection"` Reviewed By: jay-zhuang Differential Revision: D29316987 Pulled By: ltamasi fbshipit-source-id: 9815d189ed7d63890622768675a01990e3680221 | 23 June 2021, 00:34:03 UTC |
005c046 | Levi Tamasi | 22 June 2021, 20:36:40 UTC | Update HISTORY.md for PR 8437 (#8442) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8442 Reviewed By: akankshamahajan15 Differential Revision: D29306432 Pulled By: ltamasi fbshipit-source-id: 0fe55736368ac024b2f51213b65f4b06d71d60e6 | 22 June 2021, 20:37:22 UTC |
0a1aed4 | Baptiste Lemaire | 22 June 2021, 18:45:14 UTC | Fix double slashes in user-provided db path. (#8439) Summary: At the moment, the following command : "`./ --db=mypath/ dump_file_files`" returns a series of erronous names with double slashes, ie: "`mypath//000xxx.sst`", including manifest file names with double slashes "`mypath//MANIFEST-00XXX`", whereas "`./ --db=mypath dump_file_files`" correctly returns "`mypath/000xxx.sst`" and "`mypath/MANIFEST-00XXX`". This (very short) PR simply checks if there is a need to add or remove any '`/`' character when the `db_path` and `manifest_filename`/sst `filenames` are concatenated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8439 Reviewed By: akankshamahajan15 Differential Revision: D29301349 Pulled By: bjlemaire fbshipit-source-id: 3e9e58f9749d278b654ae838fcee13ad698705a8 | 22 June 2021, 18:46:25 UTC |
f89423a | Jay Zhuang | 22 June 2021, 18:09:11 UTC | Revert "Revert "Snapshot release triggered compaction without multiple tombstones (#8357)" (#8410)" (#8438) Summary: This reverts commit 25be1ed66a354ee1d665d7a28eb20d36afc75e90. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8438 Test Plan: Run the impacted mysql test 40 times Reviewed By: ajkr Differential Revision: D29286247 Pulled By: jay-zhuang fbshipit-source-id: d3bd056971a19a8b012d5d0295fa045c012b3c04 | 22 June 2021, 18:10:03 UTC |
b511b04 | Andrew Kryczka | 22 June 2021, 17:29:15 UTC | Fixup HISTORY.md for 6.22 release (#8441) Summary: `git diff origin/6.21.fb origin/6.22.fb -- HISTORY.md` looked odd. This PR fixes it up by moving items from 6.21.0 to 6.22.0 that were never in any 6.21 release. Also mentioned the background stat collection fix under 6.22 (previously it was mentioned under 6.21 patch releases only). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8441 Reviewed By: jay-zhuang Differential Revision: D29304812 Pulled By: ajkr fbshipit-source-id: 2a928a9518a1d6615321d5c2d1e22b17cbb59093 | 22 June 2021, 17:30:10 UTC |
cbb3b25 | Levi Tamasi | 22 June 2021, 16:48:50 UTC | Print blob file checksums as hex (#8437) Summary: Currently, blob file checksums are incorrectly dumped as raw bytes in the `ldb manifest_dump` output (i.e. they are not printed as hex). The patch fixes this and also updates some test cases to reflect that the checksum value field in `BlobFileAddition` and `SharedBlobFileMetaData` contains the raw checksum and not a hex string. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8437 Test Plan: `make check` Tested using `ldb manifest_dump` Reviewed By: akankshamahajan15 Differential Revision: D29284170 Pulled By: ltamasi fbshipit-source-id: d11cfb3435b14cd73c8a3d3eb14fa0f9fa1d2228 | 22 June 2021, 16:49:44 UTC |
54d73d6 | Jay Zhuang | 22 June 2021, 16:16:46 UTC | Fix DeleteFilesInRange may cause inconsistent compaction error (#8434) Summary: `DeleteFilesInRange()` marks deleting files to `being_compacted` before deleting, which may cause ongoing compactions report corruption exception or ASSERT for debug build. Adding the missing `ComputeCompactionScore()` when `being_compacted` is set. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8434 Test Plan: Unittest Reviewed By: ajkr Differential Revision: D29276127 Pulled By: jay-zhuang fbshipit-source-id: f5b223e3c1fc6d821e100e3f3442bc70c1d50cf7 | 22 June 2021, 16:17:37 UTC |
065bea1 | Levi Tamasi | 22 June 2021, 05:24:23 UTC | Add a class for measuring the amount of garbage generated during compaction (#8426) Summary: This is part of an alternative approach to https://github.com/facebook/rocksdb/issues/8316. Unlike that approach, this one relies on key-values getting processed one by one during compaction, and does not involve persistence. Specifically, the patch adds a class `BlobGarbageMeter` that can track the number and total size of blobs in a (sub)compaction's input and output on a per-blob file basis. This information can then be used to compute the amount of additional garbage generated by the compaction for any given blob file by subtracting the "outflow" from the "inflow." Note: this patch only adds `BlobGarbageMeter` and associated unit tests. I plan to hook up this class to the input and output of `CompactionIterator` in a subsequent PR. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8426 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D29242250 Pulled By: ltamasi fbshipit-source-id: 597e50ad556540e413a50e804ba15bc044d809bb | 22 June 2021, 05:25:30 UTC |
a50da40 | anand76 | 22 June 2021, 04:22:57 UTC | Fix a tsan warning due to reading flags in LRUHandle without holding a mutex (#8433) Summary: Tsan complains due to a perceived race condition in accessing LRUHandle flags. One thread calls ```LRUHandle::SetHit()``` from ```LRUCacheShard::Lookup()```, while another thread calls ```LRUHandle::IsPending()``` from ```LRUCacheShard::IsReady()```. The latter call is from ```MultiGet```. It doesn't actually have to call ```IsReady``` since a null value indicates the cache handle is not ready, so its sufficient to check for a null value. Also modify ```IsReady``` to acquire the LRU shard mutex. Tests: 1. make check 2. Run tsan_crash Pull Request resolved: https://github.com/facebook/rocksdb/pull/8433 Reviewed By: zhichao-cao Differential Revision: D29278030 Pulled By: anand1976 fbshipit-source-id: 0c9fed56d12eda853e72dadebe75038361bd257f | 22 June 2021, 04:23:56 UTC |
e9b627f | lorinlee | 21 June 2021, 20:40:12 UTC | Fix comments in Status (#8429) Summary: Signed-off-by: lorinlee <lorinlee1996@gmail.com> Pull Request resolved: https://github.com/facebook/rocksdb/pull/8429 Reviewed By: zhichao-cao Differential Revision: D29256538 Pulled By: ajkr fbshipit-source-id: fa483861148d82119f4c8775d0cf5f6c71b06215 | 21 June 2021, 20:41:08 UTC |
6f33901 | Andrew Kryczka | 21 June 2021, 20:14:44 UTC | Skip c_test and env_test when ASSERT_STATUS_CHECKED=1 (#8430) Summary: - `c_test` fails because `rocksdb_compact_range()` swallows a `Status`. - `env_test` fails because `ReadRequest`s to `MultiRead()` do not have their `Status`es checked. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8430 Test Plan: `ASSERT_STATUS_CHECKED=1 make -j48 check` Reviewed By: jay-zhuang Differential Revision: D29257473 Pulled By: ajkr fbshipit-source-id: e02127f971703744be7de85f0a028e4664c79577 | 21 June 2021, 20:15:31 UTC |
7ed69fb | Andrew Kryczka | 19 June 2021, 19:11:44 UTC | Update HISTORY.md and version.h 6.22 release (#8427) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8427 Reviewed By: zhichao-cao Differential Revision: D29246916 Pulled By: ajkr fbshipit-source-id: ccd44ca1a6dd5101dc37f19b8e1fe6c0e3883e0a | 19 June 2021, 19:12:46 UTC |
19a8926 | mwish | 19 June 2021, 00:14:01 UTC | typo: fix typo in db/write_thread's state (#8423) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/8423 Reviewed By: mrambacher Differential Revision: D29232587 Pulled By: jay-zhuang fbshipit-source-id: 04d4937cf0605cbf341a920d1305369a7b8f0574 | 19 June 2021, 00:14:51 UTC |
82a70e1 | Zhichao Cao | 18 June 2021, 22:02:59 UTC | Trace MultiGet Keys and CF_IDs to the trace file (#8421) Summary: Tracing the MultiGet information including timestamp, keys, and CF_IDs to the trace file for analyzing and replay. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8421 Test Plan: make check, add test to trace_analyzer_test Reviewed By: anand1976 Differential Revision: D29221195 Pulled By: zhichao-cao fbshipit-source-id: 30c677d6c39ab31ef4bbdf7e0d1fa1fd79f295ff | 18 June 2021, 22:04:05 UTC |
d96b012 | Akanksha Mahajan | 18 June 2021, 19:31:08 UTC | Change Windows image back to stable (#8424) Summary: Since windows timeout issue has been fixed. Change the image back to stable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8424 Test Plan: Check CircleCI jobs Reviewed By: zhichao-cao Differential Revision: D29235219 Pulled By: akankshamahajan15 fbshipit-source-id: 2c111f96e216dac4ae3d7ec3b4cdd8e459575d37 | 18 June 2021, 19:32:01 UTC |
8ea0a2c | anand76 | 18 June 2021, 16:35:03 UTC | Parallelize secondary cache lookup in MultiGet (#8405) Summary: Implement the ```WaitAll()``` interface in ```LRUCache``` to allow callers to issue multiple lookups in parallel and wait for all of them to complete. Modify ```MultiGet``` to use this to parallelize the secondary cache lookups in order to reduce the overall latency. A call to ```cache->Lookup()``` returns a handle that has an incomplete value (nullptr), and the caller can call ```cache->IsReady()``` to check whether the lookup is complete, and pass a vector of handles to ```WaitAll``` to wait for completion. If any of the lookups fail, ```MultiGet``` will read the block from the SST file. Another change in this PR is to rename ```SecondaryCacheHandle``` to ```SecondaryCacheResultHandle``` as it more accurately describes the return result of the secondary cache lookup, which is more like a future. Tests: 1. Add unit tests in lru_cache_test 2. Benchmark results with no secondary cache configured Master - ``` readrandom : 41.175 micros/op 388562 ops/sec; 106.7 MB/s (7277999 of 7277999 found) readrandom : 41.217 micros/op 388160 ops/sec; 106.6 MB/s (7274999 of 7274999 found) multireadrandom : 10.309 micros/op 1552082 ops/sec; (28908992 of 28908992 found) multireadrandom : 10.321 micros/op 1550218 ops/sec; (29081984 of 29081984 found) ``` This PR - ``` readrandom : 41.158 micros/op 388723 ops/sec; 106.8 MB/s (7290999 of 7290999 found) readrandom : 41.185 micros/op 388463 ops/sec; 106.7 MB/s (7287999 of 7287999 found) multireadrandom : 10.277 micros/op 1556801 ops/sec; (29346944 of 29346944 found) multireadrandom : 10.253 micros/op 1560539 ops/sec; (29274944 of 29274944 found) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/8405 Reviewed By: zhichao-cao Differential Revision: D29190509 Pulled By: anand1976 fbshipit-source-id: 6f8eff6246712af8a297cfe22ea0d1c3b2a01bb0 | 18 June 2021, 16:35:59 UTC |
e817bc9 | Baptiste Lemaire | 18 June 2021, 11:56:43 UTC | Added memtable garbage statistics (#8411) Summary: **Summary**: 2 new statistics counters are added to RocksDB: `MEMTABLE_PAYLOAD_BYTES_AT_FLUSH` and `MEMTABLE_GARBAGE_BYTES_AT_FLUSH`. The former tracks how many raw bytes of useful data are present on the memtable at flush time, whereas the latter is tracks how many of these raw bytes are considered garbage, meaning that they ended up not being imported on the SSTables resulting from the flush operations. **Unit test**: run `make db_flush_test -j$(nproc); ./db_flush_test` to run the unit test. This executable includes 3 tests, that test support and correct stat calculations for workloads with inserts, deletes, and DeleteRanges. The parameters are set such that the workloads are performed on a single memtable, and a single SSTable is created as a result of the flush operation. The flush operation is manually called in the test file. The tests verify that the values of these 2 statistics counters introduced in this PR can be exactly predicted, showing that we have a full understanding of the underlying operations. **Performance testing**: `./db_bench -statistics -benchmarks=fillrandom -num=10000000` repeated 10 times. Timing done using "date" function in a bash script. _Results_: Original Rocksdb fork: mean 66.6 sec, std 1.18 sec. This feature branch: mean 67.4 sec, std 1.35 sec. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8411 Reviewed By: akankshamahajan15 Differential Revision: D29150629 Pulled By: bjlemaire fbshipit-source-id: 7b3c2e86d50c6aa34fa50fd134282eacb543a5b1 | 18 June 2021, 11:57:27 UTC |
5ba1b6e | Akanksha Mahajan | 18 June 2021, 04:55:42 UTC | Cache warming data blocks during flush (#8242) Summary: This PR prepopulates warm/hot data blocks which are already in memory into block cache at the time of flush. On a flush, the data block that is in memory (in memtables) get flushed to the device. If using Direct IO, additional IO is incurred to read this data back into memory again, which is avoided by enabling newly added option. Right now, this is enabled only for flush for data blocks. We plan to expand this option to cover compactions in the future and for other types of blocks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8242 Test Plan: Add new unit test Reviewed By: anand1976 Differential Revision: D28521703 Pulled By: akankshamahajan15 fbshipit-source-id: 7219d6958821cedce689a219c3963a6f1a9d5f05 | 18 June 2021, 04:56:47 UTC |
d53f7ff | 聂佩轩 | 17 June 2021, 22:59:53 UTC | Add DeteleRange support for DBWithTTL (#8384) Summary: This commit is for enabling `DBWithTTL` to use `DeteleRange` which it cannot before. As (int32_t)Timestamp is suffixed to values in `DBWithTTL`, there is no reason that it cannot use the common used api. I added `DeleteRangeCF` in `DBWithTTLImpl::Write` so that we can use `DeteleRange` normally. When we run code like `dbWithTtl->DeleteRange(start, end)`, it executes`WriteBatchInternal::DeleteRange` internally. Intended to fix https://github.com/facebook/rocksdb/issues/7218 Pull Request resolved: https://github.com/facebook/rocksdb/pull/8384 Test Plan: added corresponded testing logic to existing unit test Reviewed By: jay-zhuang Differential Revision: D29176734 fbshipit-source-id: 6874ed979fc08e1d138149d03653e43a75f0e0e6 | 17 June 2021, 23:00:50 UTC |
865a251 | Peter Dillinger | 17 June 2021, 19:28:00 UTC | Mark Ribbon filter and optimize_filters_for_memory as production (#8408) Summary: Marked the Ribbon filter and optimize_filters_for_memory features as production-ready, each enabling memory savings for Bloom-like filters. Use `NewRibbonFilterPolicy` in place of `NewBloomFilterPolicy` to use Ribbon filters instead of Bloom, or `ribbonfilter` in place of `bloomfilter` in configuration string. Some small refactoring in db_stress. Removed/refactored unused code in db_bench, in part preparing for future default possibly being different from "disabled." Pull Request resolved: https://github.com/facebook/rocksdb/pull/8408 Test Plan: Lots of prior automated, ad-hoc, and "real world" testing. Updated tests for new API names. Quick db_bench test: bloom fillrandom 77730 ops/sec rocksdb.block.cache.filter.bytes.insert COUNT : 89929384 ribbon fillrandom 71492 ops/sec rocksdb.block.cache.filter.bytes.insert COUNT : 64531384 Reviewed By: mrambacher Differential Revision: D29140805 Pulled By: pdillinger fbshipit-source-id: d742c922722421678f95ad85eeb0aaebc9f5e49a | 17 June 2021, 19:29:16 UTC |
575ea26 | anand76 | 17 June 2021, 19:04:58 UTC | Don't log a warning if file system doesn't support ReopenWritableFile() (#8414) Summary: RocksDB logs a warning if WAL truncation on DB open fails. Its possible that on some file systems, truncation is not required and they would return ```Status::NotSupported()``` for ```ReopenWritableFile```. Don't log a warning in such cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8414 Reviewed By: akankshamahajan15 Differential Revision: D29181738 Pulled By: anand1976 fbshipit-source-id: 6e01e9117e1e4c1d67daa4dcee7fa59d06e057a7 | 17 June 2021, 19:05:40 UTC |
298edae | Sidi Mohamed EL AATIFI | 17 June 2021, 19:02:01 UTC | Fix a typo in Javadoc (#8394) Summary: iterateLowerBound Slice representing the lower bound Pull Request resolved: https://github.com/facebook/rocksdb/pull/8394 Reviewed By: ajkr Differential Revision: D29085721 Pulled By: jay-zhuang fbshipit-source-id: a154375879395c48e9bd3794d296e70316894056 | 17 June 2021, 19:02:57 UTC |
d81c2d1 | mrambacher | 17 June 2021, 15:38:30 UTC | Add Inner method and support for wrapped Customizable (#8373) Summary: Inner method added for classes to override to return has-a relationship. CheckedCast expanded to use Inner to return has-a relationship. Future Customizable classes (Env, FileSystem, Statistics, etc) will use this feature Pull Request resolved: https://github.com/facebook/rocksdb/pull/8373 Reviewed By: pdillinger Differential Revision: D29176369 Pulled By: mrambacher fbshipit-source-id: cfb6d7702fa365ca4e40c4a50a19e3a534e5ac43 | 17 June 2021, 15:39:36 UTC |
ecccc63 | Zhichao Cao | 17 June 2021, 01:29:48 UTC | Implementation of Crc32c combine function (#8305) Summary: Implement a function to generate the crc32c of two combined strings. Suppose we have the string 1 (s1) with crc32c checksum crc32c_1 and string 2 (s2) with crc32c checksum crc32c_2, the new string is s1+s2 and its checksum is crc32c_new=Crc32cCombine(crc32c_1, crc32c_2, s2.size). Pull Request resolved: https://github.com/facebook/rocksdb/pull/8305 Test Plan: make check, added new testing case Reviewed By: pdillinger Differential Revision: D28651665 Pulled By: zhichao-cao fbshipit-source-id: c84116108388f11a81f6a217b49f99c70d4ffacf | 17 June 2021, 01:30:34 UTC |
d5bd003 | mrambacher | 16 June 2021, 23:50:43 UTC | Rename ImmutableOptions variables (#8409) Summary: This is the next part of the ImmutableOptions cleanup. After changing the use of ImmutableCFOptions to ImmutableOptions, there were places in the code that had did something like "ImmutableOptions* immutable_cf_options", where "cf" referred to the "old" type. This change simply renames the variables to match the current type. No new functionality is introduced. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8409 Reviewed By: pdillinger Differential Revision: D29166248 Pulled By: mrambacher fbshipit-source-id: 96de97f8e743f5c5160f02246e3ed8269556dc6f | 16 June 2021, 23:51:38 UTC |
25be1ed | Andrew Kryczka | 16 June 2021, 01:14:02 UTC | Revert "Snapshot release triggered compaction without multiple tombstones (#8357)" (#8410) Summary: This reverts commit 9167ece5868fbad452fc5af3e9847988d64bb0e2. It was found to reliably trip a compaction picking conflict assertion in a MyRocks unit test. We don't understand why yet so reverting in the meantime. Pull Request resolved: https://github.com/facebook/rocksdb/pull/8410 Test Plan: `make check -j48` Reviewed By: jay-zhuang Differential Revision: D29150300 Pulled By: ajkr fbshipit-source-id: 2de8664f355d6da015e84e5fec2e3f90f49741c8 | 16 June 2021, 01:15:15 UTC |