Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
16 July 2024, 15:56:10 UTC
sort by:
Revision Author Date Message Commit Date
58ec8cf Andrew Kryczka 17 February 2021, 18:42:56 UTC Backport https://github.com/facebook/rocksdb/pull/7970 17 February 2021, 18:42:56 UTC
553aafa Andrew Kryczka 06 February 2021, 01:14:21 UTC bump version and update HISTORY.md for 6.17.2 06 February 2021, 01:14:21 UTC
1315375 Andrew Kryczka 05 February 2021, 23:55:34 UTC Allow range deletions in `*TransactionDB` only when safe (#7929) Summary: Explicitly reject all range deletions on `TransactionDB` or `OptimisticTransactionDB`, except when the user provides sufficient promises that allow us to proceed safely. The necessary promises are described in the API doc for `TransactionDB::DeleteRange()`. There is currently no way to provide enough promises to make it safe in `OptimisticTransactionDB`. Fixes https://github.com/facebook/rocksdb/issues/7913. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7929 Test Plan: unit tests covering the cases it's permitted/rejected Reviewed By: ltamasi Differential Revision: D26240254 Pulled By: ajkr fbshipit-source-id: 2834a0ce64cc3e4c3799e35b885a5e79c2f4f6d9 06 February 2021, 01:13:26 UTC
a471d31 Zhichao Cao 28 January 2021, 20:27:20 UTC update HISTORY.md and bump version for 6.17.1 28 January 2021, 20:27:20 UTC
8797aea Zhichao Cao 28 January 2021, 01:56:17 UTC Do not set bg error for compaction in retryable IO Error case (#7899) Summary: When retryable IO error occurs during compaction, it is mapped to soft error and set the BG error. However, auto resume is not called to clean the soft error since compaction will reschedule by itself. In this change, When retryable IO error occurs during compaction, BG error is not set. User will be informed the error via EventHelper. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7899 Test Plan: tested with error_handler_fs_test Reviewed By: anand1976 Differential Revision: D26094097 Pulled By: zhichao-cao fbshipit-source-id: c53424f11d237405592cd762f43cbbdf8da8234f 28 January 2021, 20:20:02 UTC
9092ebe Levi Tamasi 21 January 2021, 22:29:52 UTC Remove superfluous 'Unreleased' heading 21 January 2021, 22:29:52 UTC
675f351 Levi Tamasi 21 January 2021, 22:18:25 UTC Update HISTORY.md for PR 7888 (#7890) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7890 Reviewed By: ajkr Differential Revision: D26005509 Pulled By: ltamasi fbshipit-source-id: e7eb732180d447900788d0e3a17dfd1c3f1e708a 21 January 2021, 22:27:06 UTC
11b42f9 Andrew Kryczka 21 January 2021, 16:47:06 UTC workaround race conditions during `PeriodicWorkScheduler` registration (#7888) Summary: This provides a workaround for two race conditions that will be fixed in a more sophisticated way later. This PR: (1) Makes the client serialize calls to `Timer::Start()` and `Timer::Shutdown()` (see https://github.com/facebook/rocksdb/issues/7711). The long-term fix will be to make those functions thread-safe. (2) Makes `PeriodicWorkScheduler` atomically add/cancel work together with starting/shutting down its `Timer`. The long-term fix will be for `Timer` API to offer more specialized APIs so the client will not need to synchronize. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7888 Test Plan: ran the repro provided in https://github.com/facebook/rocksdb/issues/7881 Reviewed By: jay-zhuang Differential Revision: D25990891 Pulled By: ajkr fbshipit-source-id: a97fdaebbda6d7db7ddb1b146738b68c16c5be38 21 January 2021, 20:31:28 UTC
14d173e Adam Retter 21 January 2021, 16:33:55 UTC Fix compilation against musl lib C (#7875) Summary: See https://github.com/percona/PerconaFT/pull/450 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7875 Reviewed By: ajkr Differential Revision: D25938020 Pulled By: jay-zhuang fbshipit-source-id: 9014dbc7b23bf92c5e63bfbdda4565bb0d2f2b58 21 January 2021, 20:13:42 UTC
8b30b8d Levi Tamasi 21 January 2021, 04:27:19 UTC Make blob related VersionEdit tags unignorable (#7886) Summary: BlobFileAddition and BlobFileGarbage should not be in the ignorable tag range, since if they are present in the MANIFEST, users cannot downgrade to a RocksDB version that does not understand them without losing access to the data in the blob files. The patch moves these two tags to the unignorable range; this should still be safe at this point, since the integrated BlobDB project is still work in progress and thus there shouldn't be any ignorable BlobFileAddition/BlobFileGarbage tags out there. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7886 Test Plan: `make check` Reviewed By: cheng-chang Differential Revision: D25980956 Pulled By: ltamasi fbshipit-source-id: 13cf5bd61d77f049b513ecd5ad0be8c637e40a9d 21 January 2021, 18:01:00 UTC
b6471f8 Cheng Chang 20 January 2021, 22:32:16 UTC Update HISTORY.md (#7887) Summary: Mention the forward compatibility fix for WAL related version edits. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7887 Reviewed By: ltamasi Differential Revision: D25982494 Pulled By: cheng-chang fbshipit-source-id: 4be292aa4bf7fbc8a27c0bef1e7a98ad3ea8e1fa 20 January 2021, 23:07:42 UTC
3dd5bc2 Cheng Chang 20 January 2021, 03:26:05 UTC Make it able to ignore WAL related VersionEdits in older versions (#7873) Summary: Although the tags for `WalAddition`, `WalDeletion` are after `kTagSafeIgnoreMask`, to actually be able to skip these entries in older versions of RocksDB, we require that they are encoded with their encoded size as the prefix. This requirement is not met in the current codebase, so a downgraded DB may fail to open if these entries exist in the MANIFEST. If a DB wants to downgrade, and its MANIFEST contains `WalAddition` or `WalDeletion`, it can set `track_and_verify_wals_in_manifest` to `false`, then restart twice, then downgrade. On the first restart, a new MANIFEST will be created with a `WalDeletion` indicating that all previously tracked WALs are removed from MANIFEST. On the second restart, since there is no tracked WALs in MANIFEST now, a new MANIFEST will be created with neither `WalAddition` nor `WalDeletion`. Then the DB can downgrade. Tags for `BlobFileAddition`, `BlobFileGarbage` also have the same problem, but this PR focuses on solving the problem for WAL edits. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7873 Test Plan: Added a `VersionEditTest::IgnorableTags` unit test to verify all entries with tags larger than `kTagSafeIgnoreMask` can actually be skipped and won't affect parsing of other entries. Reviewed By: ajkr Differential Revision: D25935930 Pulled By: cheng-chang fbshipit-source-id: 7a02fdba4311d6084328c14aed110a26d08c3efb 20 January 2021, 16:35:30 UTC
48edcfc Cheng Chang 20 January 2021, 00:08:09 UTC Update HISTORY.md (#7874) Summary: I find that the `track_and_verify_wals_in_manifest` option was only removed from 6.15 branch's HISTORY, but still appears under 6.15 in master branch's HISTORY. It should be moved to 6.16 since that's when the feature should be available. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7874 Reviewed By: jay-zhuang Differential Revision: D25935971 Pulled By: cheng-chang fbshipit-source-id: fe8bf1ec111597f9207e109aa3be65f8f919f1fd 20 January 2021, 16:35:22 UTC
ffe4906 Levi Tamasi 16 January 2021, 02:51:12 UTC Update version to 6.17 (#7871) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7871 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D25932233 Pulled By: ltamasi fbshipit-source-id: 8b80b0638a4f34f21a27ba80b3eda7d75410b2e8 16 January 2021, 02:53:00 UTC
d76a8ee Tomas Kolda 16 January 2021, 01:51:20 UTC Fixing Windows build using CMake (#7854) Summary: Builds were not producing Windows binaries properly in 6.15 branch: ``` 00:00:46.413 Tests run: 11, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.183 sec <<< FAILURE! - in org.rocksdb.EventListenerTest 00:00:46.414 testAllCallbacksInvocation(org.rocksdb.EventListenerTest) Time elapsed: 0.012 sec <<< ERROR! 00:00:46.414 java.lang.UnsatisfiedLinkError: org.rocksdb.test.TestableEventListener.invokeAllCallbacks(J)V 00:00:46.414 at org.rocksdb.test.TestableEventListener.invokeAllCallbacks(Native Method) 00:00:46.414 at org.rocksdb.test.TestableEventListener.invokeAllCallbacks(TestableEventListener.java:19) 00:00:46.414 at org.rocksdb.EventListenerTest.testAllCallbacksInvocation(EventListenerTest.java:436) ``` ``` 00:00:41.497 "D:\j\workspace\RocksDB_Build_Windows\build\java\rocksdbjni_headers.vcxproj" (default target) (3) -> 00:00:41.497 (CustomBuild target) -> 00:00:41.497 CUSTOMBUILD : error : Could not find class file for 'org.rocksdb.TestableEventListener'. [D:\j\workspace\RocksDB_Build_Windows\build\java\rocksdbjni_headers.vcxproj] ``` Also failed on Linux as library was not initialized yet: ``` 00:01:25.103 Running org.rocksdb.NativeComparatorWrapperTest 00:01:25.133 Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.006 sec <<< FAILURE! - in org.rocksdb.NativeComparatorWrapperTest 00:01:25.133 rountrip(org.rocksdb.NativeComparatorWrapperTest) Time elapsed: 0.002 sec <<< ERROR! 00:01:25.133 java.lang.UnsatisfiedLinkError: org.rocksdb.NativeComparatorWrapperTest$NativeStringComparatorWrapper.newStringComparator()J 00:01:25.133 at org.rocksdb.NativeComparatorWrapperTest$NativeStringComparatorWrapper.newStringComparator(Native Method) 00:01:25.133 at org.rocksdb.NativeComparatorWrapperTest$NativeStringComparatorWrapper.initializeNative(NativeComparatorWrapperTest.java:87) 00:01:25.133 at org.rocksdb.RocksCallbackObject.<init>(RocksCallbackObject.java:28) 00:01:25.133 at org.rocksdb.AbstractComparator.<init>(AbstractComparator.java:20) 00:01:25.133 at org.rocksdb.NativeComparatorWrapper.<init>(NativeComparatorWrapper.java:16) 00:01:25.133 at org.rocksdb.NativeComparatorWrapperTest$NativeStringComparatorWrapper.<init>(NativeComparatorWrapperTest.java:82) 00:01:25.133 at org.rocksdb.NativeComparatorWrapperTest.rountrip(NativeComparatorWrapperTest.java:30) ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7854 Reviewed By: jay-zhuang Differential Revision: D25873378 Pulled By: ajkr fbshipit-source-id: 88afb08bfd30edff31f17da063e636df0769cbfe 16 January 2021, 01:53:16 UTC
1001bc0 Tomas Kolda 16 January 2021, 01:03:11 UTC Read Options to support direct slice (#7132) Summary: This request is adding support for using DirectSlice in ReadOptions lower/upper bounds. To be more efficient I have added setLength to DirectSlice so I can just update the length to be used by slice from direct buffer. It is also needed, because when one creates iterator it keep pointer to original slice so setting new slice in options does not help (it needs to reuse existing one). Using this approach one can modify the slice any time during operations with iterator. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7132 Reviewed By: zhichao-cao Differential Revision: D25840092 Pulled By: jay-zhuang fbshipit-source-id: 760167baf61568c9a35138145c4bf9b06824cb71 16 January 2021, 01:05:18 UTC
2fb6d93 darionyaphet 16 January 2021, 00:54:57 UTC Using emplace_back replace push_back (#7568) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7568 Reviewed By: akankshamahajan15 Differential Revision: D24437383 Pulled By: jay-zhuang fbshipit-source-id: 7c9b3c4944b959aa7796c53b410c2b1055dc5641 16 January 2021, 00:56:41 UTC
ac956f2 Tomas Kolda 16 January 2021, 00:30:36 UTC S390 Linux is failing tests ColumnFamilyOptionsTest.cfPaths (#7853) Summary: Fix ColumnFamilyOptionsTest.cfPaths and OptionsTest.cfPaths in 6.15 branch (and probably other branches including master) has_exception variable was not initialized which was causing test failures and incorrect behavior on s390 platform (and maybe others as variable content is undefined). adamretter please take a look. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7853 Reviewed By: akankshamahajan15 Differential Revision: D25901639 Pulled By: jay-zhuang fbshipit-source-id: 151b5db27b495fc6d8ed54c0eccbde2508215ac5 16 January 2021, 00:32:31 UTC
7189ea8 anand76 15 January 2021, 19:15:15 UTC Make regression test load options from file for checkpoint (#7864) Summary: The regression_test.sh script checkpoints the DB directory before running db_bench on it. Specify the --try_load_options when creating the checkpoint in order to load options from the OPTIONS file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7864 Test Plan: manually run db_bench on the checkpoint dir Reviewed By: akankshamahajan15 Differential Revision: D25926960 Pulled By: anand1976 fbshipit-source-id: d3442ae24a7044b474dc80efc9c06bdc6ebe0388 15 January 2021, 19:16:28 UTC
3e6ee9f Adam Retter 13 January 2021, 23:59:36 UTC Update the versions of the test dependencies used for RocksJava (#7805) Summary: Update the versions of the dependencies used for testing RocksJava. pdillinger Please can you add the following to your S3 bucket: 1. https://repo1.maven.org/maven2/junit/junit/4.13.1/junit-4.13.1.jar 2. https://repo1.maven.org/maven2/org/hamcrest/hamcrest/2.2/hamcrest-2.2.jar 3. https://repo1.maven.org/maven2/cglib/cglib/3.3.0/cglib-3.3.0.jar 4. https://repo1.maven.org/maven2/org/assertj/assertj-core/2.9.0/assertj-core-2.9.0.jar Thanks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7805 Reviewed By: akankshamahajan15 Differential Revision: D25906134 Pulled By: jay-zhuang fbshipit-source-id: 1c6c7d461a73abaff1796bb31f0ad90dcbdef1a0 14 January 2021, 00:01:38 UTC
0426d4a Laurent Goujon 13 January 2021, 18:03:27 UTC Fix Java hashCode implementation (#7860) Summary: Classes ColumnFamilyHandle and CapturingWriteBatchHandler.Event have byte array fields as part of their identity, but they do not use the arrays' content to compute the instance's hash, and instead rely on the arrays' identity, causing instances to have different hashcodes although they are equal. The PR addresses it by using the arrays' content to compute the hash, like the equals method does. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7860 Reviewed By: jay-zhuang Differential Revision: D25901327 Pulled By: akankshamahajan15 fbshipit-source-id: 347e7b3d2ba7befe7faa956b033e6421b9d0c235 13 January 2021, 18:04:42 UTC
8e7b068 anand76 12 January 2021, 04:54:58 UTC Make ldb load column family options from OPTIONS file (#7847) Summary: When the --try_load_options is used in conjunction with the --column_family option, ldb incorrectly sets the ColumnFamilyOptions for that column family to defaults. This PR fixes that by retaining from the OPTIONS file and applying command line overrides. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7847 Test Plan: Add a unit test in ldb_cmd_test Reviewed By: ajkr Differential Revision: D25874720 Pulled By: anand1976 fbshipit-source-id: 04bcf23b55e5a30b5b6a59b0e5cb4faef3da7429 12 January 2021, 04:56:34 UTC
77b4bfe Jay Zhuang 11 January 2021, 23:00:55 UTC Disable PeriodicWorkScheduler during RateLimited test (#7810) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7810 Reviewed By: akankshamahajan15 Differential Revision: D25695454 Pulled By: jay-zhuang fbshipit-source-id: 963d11f38a959de7227ba2be15795af2792413a6 11 January 2021, 23:01:52 UTC
fdbebdf Cheng Chang 11 January 2021, 21:33:08 UTC Add note for PR 7789 in history (#7855) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7855 Reviewed By: ajkr Differential Revision: D25872797 Pulled By: cheng-chang fbshipit-source-id: 82159a13f897aaaad5f3c70c7dfa822e073bc623 11 January 2021, 21:34:15 UTC
e4bdf9e Adam Retter 11 January 2021, 21:31:03 UTC Small improvements to CircleCI Windows jobs (#7852) Summary: * Clearer indication of which versions of msbuild and Visual Studio is used * Explicit naming of the build jobs within the Windows workflows Pull Request resolved: https://github.com/facebook/rocksdb/pull/7852 Reviewed By: akankshamahajan15 Differential Revision: D25864444 Pulled By: jay-zhuang fbshipit-source-id: 0d618ad8a8892d5a2575cdfaa59d61a989c4df4b 11 January 2021, 21:32:07 UTC
edfcce3 Jay Zhuang 11 January 2021, 20:46:42 UTC Update github-pages and dependencies (#7850) Summary: The dependencies are updated to resolve github security vulnerabilities warning. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7850 Test Plan: `bundle update` `bundle exec jekyll serve` Reviewed By: ajkr Differential Revision: D25860828 Pulled By: jay-zhuang fbshipit-source-id: 43f9b0a59853c8889daab2993a389a685bf3f0ef 11 January 2021, 20:48:01 UTC
0a9f78d Adam Retter 11 January 2021, 18:29:15 UTC Cleanup Travis CI config (#7848) Summary: We now only use Travis CI for testing RocksDB against Linux on: * ppc64le * arm64 (aarch64) This is just some initial cleanup. I will add further ppc64le and arm64 jobs in a subsequent PR... Pull Request resolved: https://github.com/facebook/rocksdb/pull/7848 Reviewed By: jay-zhuang Differential Revision: D25870782 Pulled By: akankshamahajan15 fbshipit-source-id: d5c264a58d83ab9601790fe89ee0f66772a472f8 11 January 2021, 18:30:28 UTC
eccc47e Jay Zhuang 11 January 2021, 18:15:59 UTC Fix tsan options_test (#7845) Summary: Minor tsan issue that counter could be bumped concurrently: https://app.circleci.com/pipelines/github/facebook/rocksdb/5431/workflows/79312c7c-5815-4f07-8836-94625db8e33e/jobs/81619 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7845 Reviewed By: akankshamahajan15 Differential Revision: D25851472 Pulled By: jay-zhuang fbshipit-source-id: 74cc8797ac503413bec27a30e5d1f055379777e8 11 January 2021, 18:17:57 UTC
a3066ee Jay Zhuang 09 January 2021, 21:22:01 UTC Fix checkpoint_test hang (#7849) Summary: `CheckpointTest.CurrentFileModifiedWhileCheckpointing` could hang because now create checkpoint triggers flush twice. The test should wait both flush done. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7849 Test Plan: `gtest-parallel ./checkpoint_test --gtest_filter=CheckpointTest.CurrentFileModifiedWhileCheckpointing -r 100` Reviewed By: ajkr Differential Revision: D25860713 Pulled By: jay-zhuang fbshipit-source-id: e1c2f23037dedc33e205519f4289a25e77816b41 09 January 2021, 21:26:10 UTC
4926b33 Adam Retter 09 January 2021, 17:42:21 UTC Improvements to Env::GetChildren (#7819) Summary: The main improvement here is to not include `.` or `..` in the results of `Env::GetChildren`. The occurrence of `.` or `..`; it is non-portable, dependent on the Operating System and the File System. See: https://www.gnu.org/software/libc/manual/html_node/Reading_002fClosing-Directory.html There were lots of duplicate checks spread through the RocksDB codebase previously to skip `.` and `..`. This new removes the need for those at the source. Also some minor fixes to `Env::GetChildren`: * Improve error handling in POSIX implementation * Remove unnecessary array allocation on Windows * Fix struct name for Windows Non-UTF-8 API Pull Request resolved: https://github.com/facebook/rocksdb/pull/7819 Reviewed By: ajkr Differential Revision: D25837394 Pulled By: jay-zhuang fbshipit-source-id: 1e137e7218d38b450af9c083f73d5357abcbba2e 09 January 2021, 17:44:34 UTC
8ed680b Akanksha Mahajan 08 January 2021, 21:24:11 UTC Add new API to report dummy entries size in cache in WriteBufferManager (#7837) Summary: Add new API WriteBufferManager::dummy_entries_in_cache_usage() which reports the dummy entries size stored in cache to account for DataBlocks in WriteBufferManager. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7837 Test Plan: Updated test ./write_buffer_manager_test Reviewed By: ajkr Differential Revision: D25794312 Pulled By: akankshamahajan15 fbshipit-source-id: 197f5e8701e3dc57a7df72dab1735624f90daf4b 08 January 2021, 21:26:24 UTC
b2e30bd Cheng Chang 08 January 2021, 07:00:12 UTC Get manifest size again after getting min_log_num during checkpoint (#7836) Summary: Currently, manifest size is determined before getting min_log_num. But between getting manifest size and getting min_log_num, concurrently, a flush might succeed, which will write new records to manifest to make some WALs become outdated, then min_log_num will be correspondingly increased, but the new records in manifest will not be copied into the checkpoint because the manifest's size is determined before them, then the newly outdated WALs will still exist in the checkpoint's manifest, but they are not linked/copied to the checkpoint because their log number is < min_log_num, so a corruption of missing WAL will be reported when restoring from the checkpoint. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7836 Test Plan: make crash_test Reviewed By: ajkr Differential Revision: D25788204 Pulled By: cheng-chang fbshipit-source-id: a4e5acf30f08270b3c0a95304ff559a9e655252f 08 January 2021, 07:02:55 UTC
c22e619 Adam Retter 08 January 2021, 01:11:50 UTC Store test logs as artifacts if the build fails in CircleCI (#7812) Summary: If a workflow fails in CircleCI this will ensure that the `t/` directory is tar'd up and added to the workflow as an artifact. This allows us to download the detailed logs and see what went wrong. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7812 Reviewed By: zhichao-cao Differential Revision: D25761003 Pulled By: jay-zhuang fbshipit-source-id: 41cfd16c6385bfcc9fb35fb63df84f97d4b8b80b 08 January 2021, 01:13:19 UTC
48c0843 Zhichao Cao 08 January 2021, 00:30:06 UTC Treat File Scope Write IO Error the same as Retryable IO Error (#7840) Summary: In RocksDB, when IO error happens, the flags of IOStatus can be set. If the IOStatus is set as "File Scope IO Error", it indicate that the error is constrained in the file level. Since RocksDB does not continues write data to a file when any IO Error happens, File Scope IO Error can be treated the same as Retryable IO Error. Adding the logic to ErrorHandler::SetBGError to include the file scope IO Error in its error handling logic, which is the same as retryable IO Error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7840 Test Plan: added new unit tests in error_handler_fs_test. make check Reviewed By: anand1976 Differential Revision: D25820481 Pulled By: zhichao-cao fbshipit-source-id: 69cabd3d010073e064d6142ce1cabf341b8a6806 08 January 2021, 00:31:33 UTC
cc2a180 mrambacher 07 January 2021, 23:21:51 UTC Add more tests to the ASC pass list (#7834) Summary: Fixed the following to now pass ASC checks: * `ttl_test` * `blob_db_test` * `backupable_db_test`, * `delete_scheduler_test` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7834 Reviewed By: jay-zhuang Differential Revision: D25795398 Pulled By: ajkr fbshipit-source-id: a10037817deda4fc7cbb353a2e00b62ed89b6476 07 January 2021, 23:22:53 UTC
8f7b6c8 DreaMer963 07 January 2021, 03:27:26 UTC fix typo (#7832) Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/7832 Reviewed By: jay-zhuang Differential Revision: D25785459 Pulled By: zhichao-cao fbshipit-source-id: 78658dcb5a5f24141395046f74d7d57f11ad0868 07 January 2021, 03:28:38 UTC
6e0f62f Adam Retter 06 January 2021, 22:14:01 UTC Add more tests to ASSERT_STATUS_CHECKED (3), API change (#7715) Summary: Third batch of adding more tests to ASSERT_STATUS_CHECKED. * db_compaction_filter_test * db_compaction_test * db_dynamic_level_test * db_inplace_update_test * db_sst_test * db_tailing_iter_test * db_io_failure_test Also update GetApproximateSizes APIs to all return Status. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7715 Reviewed By: jay-zhuang Differential Revision: D25806896 Pulled By: pdillinger fbshipit-source-id: 6cb9d62ba5a756c645812754c596ad3995d7c262 06 January 2021, 22:15:02 UTC
5792b73 Zhichao Cao 06 January 2021, 21:16:47 UTC Fixed the swallowed IOStatus in Compaction Job introduced in PR 7718 (#7838) Summary: The IOStatus of TableBuilder is returned by copy the io status from builder->io_status(). pr https://github.com/facebook/rocksdb/issues/7718 swallowed the io status and it will cause the write IO error become non-retryable and no auto resume logic will handle it. Roll back to previous implementation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7838 Test Plan: make check Reviewed By: ajkr Differential Revision: D25795387 Pulled By: zhichao-cao fbshipit-source-id: bc35e69e0b71aa4148a6ed76f073357041b8e372 06 January 2021, 21:18:00 UTC
e628f59 mrambacher 06 January 2021, 18:48:24 UTC Create a CustomEnv class; Add WinFileSystem; Make LegacyFileSystemWrapper private (#7703) Summary: This PR does the following: -> Creates a WinFileSystem class. This class is the Windows equivalent of the PosixFileSystem and will be used on Windows systems. -> Introduces a CustomEnv class. A CustomEnv is an Env that takes a FileSystem as constructor argument. I believe there will only ever be two implementations of this class (PosixEnv and WinEnv). There is still a CustomEnvWrapper class that takes an Env and a FileSystem and wraps the Env calls with the input Env but uses the FileSystem for the FileSystem calls -> Eliminates the public uses of the LegacyFileSystemWrapper. With this change in place, there are effectively the following patterns of Env: - "Base Env classes" (PosixEnv, WinEnv). These classes implement the core Env functions (e.g. Threads) and have a hard-coded input FileSystem. These classes inherit from CompositeEnv, implement the core Env functions (threads) and delegate the FileSystem-like calls to the input file system. - Wrapped Composite Env classes (MemEnv). These classes take in an Env and a FileSystem. The core env functions are re-directed to the wrapped env. The file system calls are redirected to the input file system - Legacy Wrapped Env classes. These classes take in an Env input (but no FileSystem). The core env functions are re-directed to the wrapped env. A "Legacy File System" is created using this env and the file system calls directed to the env itself. With these changes in place, the PosixEnv becomes a singleton -- there is only ever one created. Any other use of the PosixEnv is via another wrapped env. This cleans up some of the issues with the env construction and destruction. Additionally, there were places in the code that required had an Env when they required a FileSystem. Many of these places would wrap the Env with a LegacyFileSystemWrapper instead of using the env->GetFileSystem(). These places were changed, thereby removing layers of additional redirection (LegacyFileSystem --> Env --> Env::FileSystem). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7703 Reviewed By: zhichao-cao Differential Revision: D25762190 Pulled By: anand1976 fbshipit-source-id: 1a088e97fc916f28ac69c149cd1dcad0ab31704b 06 January 2021, 18:49:32 UTC
c1a65a4 mrambacher 04 January 2021, 23:59:52 UTC Make StringEnv, StringSink, StringSource use FS classes (#7786) Summary: Change the StringEnv and related classes to be based on FileSystem APIs rather than the corresponding Env ones. The StringSink and StringSource classes were changed to be based on the corresponding FS file classes. Part of a cleanup to use the newer interfaces. This change also eliminates some of the casts/wrappers to LegacyFile classes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7786 Reviewed By: jay-zhuang Differential Revision: D25761460 Pulled By: anand1976 fbshipit-source-id: 428ae8e32b3db97dbeeca08c9d3bb0d9d4d3a38f 05 January 2021, 00:01:01 UTC
58660bf Jay Zhuang 04 January 2021, 21:11:31 UTC Use mock time for histogram_test (#7799) Summary: `histogram_test` uses real sleep, which depends on the test executing speed, it makes the test unstable. Switching to using a mock time env, it can also increase the test speed (from 10100ms -> 100ms). Pull Request resolved: https://github.com/facebook/rocksdb/pull/7799 Test Plan: run test 10 times, all passed. vs. without fix 3 out 10 test failed: no fix: https://app.circleci.com/pipelines/github/facebook/rocksdb?branch=pull%2F7797 with fix: https://app.circleci.com/pipelines/github/facebook/rocksdb?branch=pull%2F7799 Reviewed By: pdillinger Differential Revision: D25676948 Pulled By: jay-zhuang fbshipit-source-id: 64c273fc299c53283138dbb213386e4b45e8bdc2 04 January 2021, 21:12:40 UTC
225abff Andrew Kryczka 04 January 2021, 19:50:29 UTC Verify file checksum generator name (#7824) Summary: Previously we only had a debug assertion to check the right generator was being used for verification. However a user hit a problem in production where their factory was creating the wrong generator for some files, leading to checksum mismatches. It would have been easier to debug if we verified in optimized builds that the generator with the proper name is used. This PR adds such verification. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7824 Reviewed By: zhichao-cao Differential Revision: D25740254 Pulled By: ajkr fbshipit-source-id: a6231521747605021bad3231484b5d4f99f4044f 04 January 2021, 19:51:50 UTC
159ea47 Dylan Wen 04 January 2021, 19:38:04 UTC Fix typos in comments (#7790) Summary: Hi there, This PR fixes some typos in comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7790 Reviewed By: ajkr Differential Revision: D25684213 Pulled By: zhichao-cao fbshipit-source-id: b77026018cbdd59c9db25aa73edeb359d9962f3e 04 January 2021, 19:39:14 UTC
b8c01ed Andrew Kryczka 04 January 2021, 19:12:08 UTC Support --hex flag in `ldb file_checksum_dump` (#7820) Summary: Prior to this PR it prints the raw bytes which can include non-printable characters. This PR adds the option to print in hex instead. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7820 Test Plan: try it out ``` $ ./ldb file_checksum_dump --hex --db=/tmp/rocksdbtest-9383//db_basic_test_12281129388755189514/ 16, FileChecksumCrc32c, 0xC789D948 ``` Reviewed By: jay-zhuang Differential Revision: D25738072 Pulled By: ajkr fbshipit-source-id: 8cf2856877971756c0495cfa63a9a1281c414dc7 04 January 2021, 19:13:14 UTC
0bad2b4 mrambacher 04 January 2021, 19:07:10 UTC Ignore the OnAddFile Status for SSTFileManager (#7826) Summary: The returned Status is ignored here as some stress tests are failing, presumably when attempting to add an empty file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7826 Reviewed By: jay-zhuang Differential Revision: D25742931 fbshipit-source-id: a1fcd620d9472993a009929306dfc421f93eb43b 04 January 2021, 19:08:28 UTC
61e3244 Andrew Kryczka 04 January 2021, 18:45:15 UTC fix thread status synchronization in thread_list_test (#7825) Summary: The test was flaky because the BG threads could increase `running_count_` up to `job_count_` before applying their thread status updates. Then the test thread would see non-deterministic results when counting threads with each status. The fix is to acquire mutex in test thread so it sees `running_count_` and thread status updated atomically. I think simply reordering the two updates would have been insufficient since the thread status update uses `memory_order_relaxed`. This change happens to also eliminate an undesirable sleep loop. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7825 Test Plan: injected sleeps to verify the failure repros before this PR and does not repro after. Reviewed By: jay-zhuang Differential Revision: D25742409 Pulled By: ajkr fbshipit-source-id: 926a2223fe856e20bc4c0c27df6736ee5cb02c97 04 January 2021, 18:46:24 UTC
bb0f781 Adam Retter 04 January 2021, 17:13:36 UTC Update RocksJava static compression dependencies (#7804) Summary: Updates LZ4 and ZStd to the latest versions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7804 Reviewed By: ajkr Differential Revision: D25733770 Pulled By: pdillinger fbshipit-source-id: ea74ef9eecb57fc47934ef1d4ff950c99ddd5158 04 January 2021, 17:14:56 UTC
81367a4 mrambacher 04 January 2021, 17:04:19 UTC Eliminate the creation of ImmutableDBOptions in WBWI::GetFromBatch (#6851) Summary: 1. Made `WriteBatchWithIndexInternal` into a class that stores the `DB*` or `DBOptions*`. 2. Changed the `GetFromBatch` method to be non-static and use an instance of the class. Added `MergeKey` methods to perform the merge itself and return any status. This change unifies the multiple calls to the `MergeHelper` under a single wrapped API. Closes https://github.com/facebook/rocksdb/issues/6683 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6851 Reviewed By: ajkr Differential Revision: D21706574 Pulled By: pdillinger fbshipit-source-id: 6860bd64d62669aaa591846e914eed3b674e68b1 04 January 2021, 17:05:46 UTC
30cd38c Cheng Chang 31 December 2020, 04:29:56 UTC Increase the txn lock timeout in stress test (#7823) Summary: We recently encounter two cases of txn lock timeout in stress test. It might be caused due to latencies of resource scheduling in the internal infrastructure. Hopefully increasing the timeout can make the related tests less flaky. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7823 Test Plan: watch internal stress test to pass. Reviewed By: siying Differential Revision: D25739233 Pulled By: cheng-chang fbshipit-source-id: 84a5a8ae820db24dacd0cfc05928b26505fab89d 31 December 2020, 04:31:35 UTC
edb0b1f jbosh 30 December 2020, 23:41:49 UTC rocksdb_transaction_get_for_update now exports (#6293) Summary: Added missing ROCKSDB_LIBRARY_API decorator to rocksdb_transaction_get_for_update. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6293 Reviewed By: jay-zhuang Differential Revision: D25234298 Pulled By: ajkr fbshipit-source-id: 8a4817adaec1f445f338c8d8c59d3392925b5721 30 December 2020, 23:42:59 UTC
fd2db79 Adam Retter 30 December 2020, 21:38:43 UTC Attempt to fix build errors around missing compression library includes (#7803) Summary: This fixes an issue introduced in https://github.com/facebook/rocksdb/pull/7769 that caused many errors about missing compression libraries to be displayed during compilation, although compilation actually succeeded. This PR fixes the compilation so the compression libraries are only introduced where strictly needed. It likely needs to be merged into the same branches as https://github.com/facebook/rocksdb/pull/7769 which I think are: 1. master 2. 6.15.fb 3. 6.16.fb Pull Request resolved: https://github.com/facebook/rocksdb/pull/7803 Reviewed By: ramvadiv Differential Revision: D25733743 Pulled By: pdillinger fbshipit-source-id: 6c04f6864b2ff4a345841d791a89b19e0e3f5bf7 30 December 2020, 21:40:10 UTC
01298c8 anand76 30 December 2020, 17:24:04 UTC Return Status from FilePrefetchBuffer::TryReadFromCache() (#7816) Summary: Return the Status from TryReadFromCache() in an argument to make it easier to report prefetch errors to the user. Tests: make crash_test make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/7816 Reviewed By: akankshamahajan15 Differential Revision: D25717222 Pulled By: anand1976 fbshipit-source-id: c320d3c12d4146bda16df78ff6927eee584c1810 30 December 2020, 17:25:09 UTC
d773866 anand76 28 December 2020, 21:37:07 UTC Fix db_bench duration for multireadrandom benchmark (#7817) Summary: The multireadrandom benchmark, when run for a specific number of reads (--reads argument), should base the duration on the actual number of keys read rather than number of batches. Tests: Run db_bench multireadrandom benchmark Pull Request resolved: https://github.com/facebook/rocksdb/pull/7817 Reviewed By: zhichao-cao Differential Revision: D25717230 Pulled By: anand1976 fbshipit-source-id: 13f4d8162268cf9a34918655e60302d0aba3864b 28 December 2020, 21:38:10 UTC
736c6dc cheng-chang 28 December 2020, 00:16:55 UTC Disable BasicLockEscalation if cannot determine whether TSAN is enabled (#7814) Summary: BasicLockEscalation will cause false-positive warnings under TSAN (this is a known issue in TSAN, see details in https://gist.github.com/spetrunia/77274cf2d5848e0a7e090d622695ed4e), skip this test if TSAN is enabled, or if we are not sure whether TSAN is enabled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7814 Test Plan: watch the tsan contrun test to pass. Reviewed By: zhichao-cao Differential Revision: D25708094 Pulled By: cheng-chang fbshipit-source-id: 4fc813ff373301d033d086154cc7bb60a5e95889 28 December 2020, 00:18:00 UTC
44ebc24 Zhichao Cao 27 December 2020, 06:05:42 UTC Add rate_limiter to GenerateOneFileChecksum (#7811) Summary: In GenerateOneFileChecksum(), RocksDB reads the file and computes its checksum. A rate limiter can be passed to the constructor of RandomAccessFileReader so that read I/O can be rate limited. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7811 Test Plan: make check Reviewed By: cheng-chang Differential Revision: D25699896 Pulled By: zhichao-cao fbshipit-source-id: e2688bc1126c543979a3bcf91dda784bd7b74164 27 December 2020, 06:07:24 UTC
601585b Zhichao Cao 25 December 2020, 21:14:38 UTC fix memory leak in db_stress checkpoint test (#7813) Summary: fix memory leak in db_stress checkpoint test. If s is not ok, checkpoint is not deleted, may cause memory leak. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7813 Test Plan: make asan_check Reviewed By: cheng-chang Differential Revision: D25702999 Pulled By: zhichao-cao fbshipit-source-id: 08253b0852835acb8cfd412503cdabf720afb678 25 December 2020, 21:15:48 UTC
55e9968 mrambacher 24 December 2020, 00:54:05 UTC No elide constructors (#7798) Summary: Added "no-elide-constructors to the ASSERT_STATUS_CHECK builds. This flag gives more errors/warnings for some of the Status checks where an inner class checks a Status and later returns it. In this case, without the elide check on, the returned status may not have been checked in the caller, thereby bypassing the checked code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7798 Reviewed By: jay-zhuang Differential Revision: D25680451 Pulled By: pdillinger fbshipit-source-id: c3f14ed9e2a13f0a8c54d839d5fb4d1fc1e93917 24 December 2020, 00:55:53 UTC
30a5ed9 Akanksha Mahajan 23 December 2020, 23:15:14 UTC Update "num_data_read" stat in RetrieveMultipleBlocks (#7770) Summary: RetrieveMultipleBlocks which is used by MultiGet to read data blocks is not updating num_data_read stat in GetContextStats. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7770 Test Plan: make check -j64 Reviewed By: anand1976 Differential Revision: D25538982 Pulled By: akankshamahajan15 fbshipit-source-id: e3daedb035b1be8ab6af6f115cb3793ccc7b1ec6 23 December 2020, 23:16:46 UTC
bdb7e54 cheng-chang 23 December 2020, 19:32:10 UTC Skip WALs according to MinLogNumberToKeep when creating checkpoint (#7789) Summary: In a stress test failure, we observe that a WAL is skipped when creating checkpoint, although its log number >= MinLogNumberToKeep(). This might happen in the following case: 1. when creating the checkpoint, there are 2 column families: CF0 and CF1, and there are 2 WALs: 1, 2; 2. CF0's log number is 1, CF0's active memtable is empty, CF1's log number is 2, CF1's active memtable is not empty, WAL 2 is not empty, the sequence number points to WAL 2; 2. the checkpoint process flushes CF0, since CF0' active memtable is empty, there is no need to SwitchMemtable, thus no new WAL will be created, so CF0's log number is now 2, concurrently, some data is written to CF0 and WAL 2; 3. the checkpoint process flushes CF1, WAL 3 is created and CF1's log number is now 3, CF0's log number is still 2 because CF0 is not empty and WAL 2 contains its unflushed data concurrently written in step 2; 4. the checkpoint process determines that WAL 1 and 2 are no longer needed according to [live_wal_files[i]->StartSequence() >= *sequence_number](https://github.com/facebook/rocksdb/blob/master/utilities/checkpoint/checkpoint_impl.cc#L388), so it skips linking them to the checkpoint directory; 5. but according to `MinLogNumberToKeep()`, WAL 2 still needs to be kept because CF0's log number is 2. If the checkpoint is reopened in read-only mode, and only read from the snapshot with the initial sequence number, then there will be no data loss or data inconsistency. But if the checkpoint is reopened and read from the most recent sequence number, suppose in step 3, there are also data concurrently written to CF1 and WAL 3, then the most recent sequence number refers to the latest entry in WAL 3, so the data written in step 2 should also be visible, but since WAL 2 is discarded, those data are lost. When tracking WAL in MANIFEST is enabled, when reopening the checkpoint, since WAL 2 is still tracked in MANIFEST as alive, but it's missing from the checkpoint directory, a corruption will be reported. This PR makes the checkpoint process to only skip a WAL if its log number < `MinLogNumberToKeep`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7789 Test Plan: watch existing tests to pass. Reviewed By: ajkr Differential Revision: D25662346 Pulled By: cheng-chang fbshipit-source-id: 136471095baa01886cf44809455cf855f24857a0 23 December 2020, 19:33:26 UTC
bd2645b anand76 23 December 2020, 19:24:50 UTC Update regression_test.sh to run multireadrandom benchmark (#7802) Summary: Update the regression_test.sh script to run the multireadrandom benchmark. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7802 Reviewed By: zhichao-cao Differential Revision: D25685482 Pulled By: anand1976 fbshipit-source-id: ef2973b551a1bbdbce198a0adf29fc277f3e65e2 23 December 2020, 19:26:12 UTC
a727efc Peter Dillinger 23 December 2020, 19:06:49 UTC Remove flaky, redundant, and dubious DBTest.SparseMerge (#7800) Summary: This test would occasionally fail like this: WARNING: c:\users\circleci\project\db\db_test.cc(1343): error: Expected: (dbfull()->TEST_MaxNextLevelOverlappingBytes(handles_[1])) <= (20 * 1048576), actual: 33501540 vs 20971520 And being a super old test, it's not structured in a sound way. And it appears that DBTest2.MaxCompactionBytesTest is a better test of what SparseMerge was intended to test. In fact, SparseMerge fails if I set options.max_compaction_bytes = options.target_file_size_base * 1000; Thus, we are removing this negative-value test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7800 Test Plan: Q.E.D. Reviewed By: ajkr Differential Revision: D25693366 Pulled By: pdillinger fbshipit-source-id: 9da07d4dce0559547fc938b2163a2015e956c548 23 December 2020, 19:08:12 UTC
0241819 mrambacher 23 December 2020, 07:44:44 UTC Add more tests for assert status checked (#7524) Summary: Added 10 more tests that pass the ASSERT_STATUS_CHECKED test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7524 Reviewed By: akankshamahajan15 Differential Revision: D24323093 Pulled By: ajkr fbshipit-source-id: 28d4106d0ca1740c3b896c755edf82d504b74801 23 December 2020, 07:45:58 UTC
daab760 Sergei Petrunia 23 December 2020, 03:10:56 UTC Range Locking: Implementation of range locking (#7506) Summary: Range Locking - an implementation based on the locktree library - Add a RangeTreeLockManager and RangeTreeLockTracker which implement range locking using the locktree library. - Point locks are handled as locks on single-point ranges. - Add a unit test: range_locking_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7506 Reviewed By: akankshamahajan15 Differential Revision: D25320703 Pulled By: cheng-chang fbshipit-source-id: f86347384b42ba2b0257d67eca0f45f806b69da7 23 December 2020, 03:12:36 UTC
f4db3e4 sdong 23 December 2020, 00:52:51 UTC Avoid to force PORTABLE mode in tools/regression_test.sh (#7806) Summary: Right now tools/regression_test.sh always builds RocksDB with PORTABLE=1. There isn't a reason for that. Remove it. Users can always specify PORTABLE through envirionement variable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7806 Test Plan: Run tools/regression_test.sh and see it still builds. Reviewed By: ajkr Differential Revision: D25687911 fbshipit-source-id: 1c0b03e5df890babc8b7d8af48b48774d9a4600c 23 December 2020, 00:54:07 UTC
81592d9 Adam Retter 22 December 2020, 23:08:17 UTC Add more tests to ASSERT_STATUS_CHECKED (4) (#7718) Summary: Fourth batch of adding more tests to ASSERT_STATUS_CHECKED. * db_range_del_test * db_write_test * random_access_file_reader_test * merge_test * external_sst_file_test * write_buffer_manager_test * stringappend_test * deletefile_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7718 Reviewed By: pdillinger Differential Revision: D25671608 fbshipit-source-id: 687a794e98a9e0cd5428ead9898ef05ced987c31 22 December 2020, 23:09:39 UTC
41ff125 cheng-chang 22 December 2020, 22:52:29 UTC SyncWAL shouldn't be supported in compacted db (#7788) Summary: `CompactedDB` is a kind of read-only DB, so it shouldn't support `SyncWAL`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7788 Test Plan: watch existing tests to pass. Reviewed By: akankshamahajan15 Differential Revision: D25661209 Pulled By: cheng-chang fbshipit-source-id: 9eb2cc3f73736dcc205c8410e5944aa203f002d3 22 December 2020, 22:53:43 UTC
1022090 Sergei Petrunia 22 December 2020, 22:46:20 UTC Apply the changes from: PS-5501 : Re-license PerconaFT 'locktree' to Apache V2 (#7801) Summary: commit d5178f513c0b4144a5ac9358ec0f6a3b54a28e76 Author: George O. Lorch III <george.lorch@percona.com> Date: Tue Mar 19 12:18:40 2019 -0700 PS-5501 : Re-license PerconaFT 'locktree' to Apache V2 - Fixed some incomplete relicensed files from previous round. - Added missing license text to some. - Relicensed more files to Apache V2 that locktree depends on. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7801 Reviewed By: jay-zhuang Differential Revision: D25682430 Pulled By: cheng-chang fbshipit-source-id: deb8a0de3e76f3638672997bfbd300e2fffbe5f5 22 December 2020, 22:47:41 UTC
9057d0a sdong 22 December 2020, 22:42:06 UTC Minimize Timing Issue in test WALTrashCleanupOnOpen (#7796) Summary: We saw DBWALTestWithParam/DBWALTestWithParam.WALTrashCleanupOnOpen sometimes fail with: db/db_sst_test.cc:575: Failure Expected: (trash_log_count) >= (1), actual: 0 vs 1 The suspicious is that delete scheduling actually deleted all trash files based on rate, but it is not expected. This can be reproduced if we manually add sleep after DB is closed for serveral seconds. Minimize its chance by setting the delete rate to be lowest possible. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7796 Test Plan: The test doesn't fail with the manual sleeping anymore Reviewed By: anand1976 Differential Revision: D25675000 fbshipit-source-id: a39fd05e1a83719c41014e48843792e752368e22 22 December 2020, 22:44:08 UTC
fbac1b3 Akanksha Mahajan 22 December 2020, 18:29:58 UTC Add tests in ASSERT_STATUS_CHECKED (#7793) Summary: add io_tracer_parser_test and prefetch_test under ASSERT_STATUS_CHECKED Pull Request resolved: https://github.com/facebook/rocksdb/pull/7793 Test Plan: ASSERT_STATUS_CHECKED=1 make check -j64 Reviewed By: jay-zhuang Differential Revision: D25673464 Pulled By: akankshamahajan15 fbshipit-source-id: 50e0b6f17160ddda206a521a7b47ee33e699a2d4 22 December 2020, 18:31:13 UTC
4d897e5 Peter Dillinger 22 December 2020, 08:19:44 UTC Migrate away from Travis+Linux+amd64 (#7791) Summary: This disables Linux/amd64 builds in Travis for PRs, and adds a gcc-10+c++20 build in CircleCI, which should fill out sufficient coverage vs. what we had in Travis Fixed a use of std::is_pod, which is deprecated in c++20 Fixed ++ on a volatile in db_repl_stress.cc, with bigger refactoring. Although ++ on this volatile was probably ok with one thread writer and one thread reader, the code was still overly complex. There was a deadcode check for error `if (replThread.no_read < dataPump.no_records)` which can be proven never to happen based on the structure of the code. It infinite loops instead for the case intended to be checked. I just simplified the code for what should be the same checking power. Also most configurations seem to be using make parallelism = 2 * vcores, so fixing / using that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7791 Test Plan: CI and `while ./db_repl_stress; do echo again; done` for a while Reviewed By: siying Differential Revision: D25669834 Pulled By: pdillinger fbshipit-source-id: b2c688053d0b1d52c989903449d3cd27a04130d6 22 December 2020, 08:20:57 UTC
861b0d1 Jay Zhuang 21 December 2020, 23:20:59 UTC Fix Windows build in block_cache_tracer_test (#7795) Summary: The test was added to cmake in https://github.com/facebook/rocksdb/issues/7783 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7795 Reviewed By: akankshamahajan15 Differential Revision: D25671010 Pulled By: jay-zhuang fbshipit-source-id: 2146ff9559cdd7266c4d78476672488c62654a6d 21 December 2020, 23:22:16 UTC
fd0d35d Jay Zhuang 21 December 2020, 16:45:54 UTC Fix block_cache_test failure (#7783) Summary: `block_cache_tracer_test` and `block_cache_trace_analyzer_test` are using the same test directory. Which causes build failure if these 2 tests are running in parallel, for example: https://app.circleci.com/pipelines/github/facebook/rocksdb/5211/workflows/8639afbe-9fec-43e2-a6a4-6d47ea9cbcbe/jobs/74598/tests Pull Request resolved: https://github.com/facebook/rocksdb/pull/7783 Reviewed By: akankshamahajan15 Differential Revision: D25656762 Pulled By: jay-zhuang fbshipit-source-id: 68aa020aa5b4b3bce324315edecb4e1a60cc18e6 21 December 2020, 16:47:08 UTC
a8aeefd Jay Zhuang 19 December 2020, 20:37:43 UTC Update release version to 6.16 (#7782) Summary: Update release version to 6.8 Pull Request resolved: https://github.com/facebook/rocksdb/pull/7782 Reviewed By: siying Differential Revision: D25648579 Pulled By: jay-zhuang fbshipit-source-id: c536d606868b95c5fb2ae8f19c17eb259d67bc51 19 December 2020, 20:39:21 UTC
4d1ac19 Peter Dillinger 19 December 2020, 15:59:08 UTC aggregated-table-properties with GetMapProperty (#7779) Summary: So that we can more easily get aggregate live table data such as total filter, index, and data sizes. Also adds ldb support for getting properties Also fixed some missing/inaccurate related comments in db.h For example: $ ./ldb --db=testdb get_property rocksdb.aggregated-table-properties rocksdb.aggregated-table-properties.data_size: 102871 rocksdb.aggregated-table-properties.filter_size: 0 rocksdb.aggregated-table-properties.index_partitions: 0 rocksdb.aggregated-table-properties.index_size: 2232 rocksdb.aggregated-table-properties.num_data_blocks: 100 rocksdb.aggregated-table-properties.num_deletions: 0 rocksdb.aggregated-table-properties.num_entries: 15000 rocksdb.aggregated-table-properties.num_merge_operands: 0 rocksdb.aggregated-table-properties.num_range_deletions: 0 rocksdb.aggregated-table-properties.raw_key_size: 288890 rocksdb.aggregated-table-properties.raw_value_size: 198890 rocksdb.aggregated-table-properties.top_level_index_size: 0 $ ./ldb --db=testdb get_property rocksdb.aggregated-table-properties-at-level1 rocksdb.aggregated-table-properties-at-level1.data_size: 80909 rocksdb.aggregated-table-properties-at-level1.filter_size: 0 rocksdb.aggregated-table-properties-at-level1.index_partitions: 0 rocksdb.aggregated-table-properties-at-level1.index_size: 1787 rocksdb.aggregated-table-properties-at-level1.num_data_blocks: 81 rocksdb.aggregated-table-properties-at-level1.num_deletions: 0 rocksdb.aggregated-table-properties-at-level1.num_entries: 12466 rocksdb.aggregated-table-properties-at-level1.num_merge_operands: 0 rocksdb.aggregated-table-properties-at-level1.num_range_deletions: 0 rocksdb.aggregated-table-properties-at-level1.raw_key_size: 238210 rocksdb.aggregated-table-properties-at-level1.raw_value_size: 163414 rocksdb.aggregated-table-properties-at-level1.top_level_index_size: 0 $ Pull Request resolved: https://github.com/facebook/rocksdb/pull/7779 Test Plan: Added a test to ldb_test.py Reviewed By: jay-zhuang Differential Revision: D25653103 Pulled By: pdillinger fbshipit-source-id: 2905469a08a64dd6b5510cbd7be2e64d3234d6d3 19 December 2020, 16:00:14 UTC
fbce7a3 Cheng Chang 19 December 2020, 05:33:20 UTC Track WAL obsoletion when updating empty CF's log number (#7781) Summary: In the write path, there is an optimization: when a new WAL is created during SwitchMemtable, we update the internal log number of the empty column families to the new WAL. `FindObsoleteFiles` marks a WAL as obsolete if the WAL's log number is less than `VersionSet::MinLogNumberWithUnflushedData`. After updating the empty column families' internal log number, `VersionSet::MinLogNumberWithUnflushedData` might change, so some WALs might become obsolete to be purged from disk. For example, consider there are 3 column families: 0, 1, 2: 1. initially, all the column families' log number is 1; 2. write some data to cf0, and flush cf0, but the flush is pending; 3. now a new WAL 2 is created; 4. write data to cf1 and WAL 2, now cf0's log number is 1, cf1's log number is 2, cf2's log number is 2 (because cf1 and cf2 are empty, so their log numbers will be set to the highest log number); 5. now cf0's flush hasn't finished, flush cf1, a new WAL 3 is created, and cf1's flush finishes, now cf0's log number is 1, cf1's log number is 3, cf2's log number is 3, since WAL 1 still contains data for the unflushed cf0, no WAL can be deleted from disk; 6. now cf0's flush finishes, cf0's log number is 2 (because when cf0 was switching memtable, WAL 3 does not exist yet), cf1's log number is 3, cf2's log number is 3, so WAL 1 can be purged from disk now, but WAL 2 still cannot because `MinLogNumberToKeep()` is 2; 7. write data to cf2 and WAL 3, because cf0 is empty, its log number is updated to 3, so now cf0's log number is 3, cf1's log number is 3, cf2's log number is 3; 8. now if the background threads want to purge obsolete files from disk, WAL 2 can be purged because `MinLogNumberToKeep()` is 3. But there are only two flush results written to MANIFEST: the first is for flushing cf1, and the `MinLogNumberToKeep` is 1, the second is for flushing cf0, and the `MinLogNumberToKeep` is 2. So without this PR, if the DB crashes at this point and try to recover, `WalSet` will still expect WAL 2 to exist. When WAL tracking is enabled, we assume WALs will only become obsolete after a flush result is written to MANIFEST in `MemtableList::TryInstallMemtableFlushResults` (or its atomic flush counterpart). The above situation breaks this assumption. This PR tracks WAL obsoletion if necessary before updating the empty column families' log numbers. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7781 Test Plan: watch existing tests and stress tests to pass. `make -j48 blackbox_crash_test` on devserver Reviewed By: ltamasi Differential Revision: D25631695 Pulled By: cheng-chang fbshipit-source-id: ca7fff967bdb42204b84226063d909893bc0a4ec 19 December 2020, 05:34:36 UTC
62afa96 Adam Retter 19 December 2020, 00:11:17 UTC Fix various small build issues, Java API naming (#7776) Summary: * Compatibility with older GCC. * Compatibility with older jemalloc libraries. * Remove Docker warning when building i686 binaries. * Fix case inconsistency in Java API naming (potential update to HISTORY.md deferred) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7776 Reviewed By: akankshamahajan15 Differential Revision: D25607235 Pulled By: pdillinger fbshipit-source-id: 7ab0fb7fa7a34e97ed0bec991f5081acb095777d 19 December 2020, 00:12:26 UTC
75e4af1 sdong 18 December 2020, 23:27:40 UTC Update code comment for options.ttl (#7775) Summary: The behavior of options.ttl has been updated long ago but we didn't update the code comments. Also update the periodic compaction's comment. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7775 Test Plan: See it can still build through CI. Reviewed By: ajkr Differential Revision: D25592015 fbshipit-source-id: b1db18b6787e7048ce6aedcbc3bb44493c9fc49b 18 December 2020, 23:29:03 UTC
239d17a Peter Dillinger 18 December 2020, 22:29:48 UTC Support optimize_filters_for_memory for Ribbon filter (#7774) Summary: Primarily this change refactors the optimize_filters_for_memory code for Bloom filters, based on malloc_usable_size, to also work for Ribbon filters. This change also replaces the somewhat slow but general BuiltinFilterBitsBuilder::ApproximateNumEntries with implementation-specific versions for Ribbon (new) and Legacy Bloom (based on a recently deleted version). The reason is to emphasize speed in ApproximateNumEntries rather than 100% accuracy. Justification: ApproximateNumEntries (formerly CalculateNumEntry) is only used by RocksDB for range-partitioned filters, called each time we start to construct one. (In theory, it should be possible to reuse the estimate, but the abstractions provided by FilterPolicy don't really make that workable.) But this is only used as a heuristic estimate for hitting a desired partitioned filter size because of alignment to data blocks, which have various numbers of unique keys or prefixes. The two factors lead us to prioritize reasonable speed over 100% accuracy. optimize_filters_for_memory adds extra complication, because precisely calculating num_entries for some allowed number of bytes depends on state with optimize_filters_for_memory enabled. And the allocator-agnostic implementation of optimize_filters_for_memory, using malloc_usable_size, means we would have to actually allocate memory, many times, just to precisely determine how many entries (keys) could be added and stay below some size budget, for the current state. (In a draft, I got this working, and then realized the balance of speed vs. accuracy was all wrong.) So related to that, I have made CalculateSpace, an internal-only API only used for testing, non-authoritative also if optimize_filters_for_memory is enabled. This simplifies some code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7774 Test Plan: unit test updated, and for FilterSize test, range of tested values is greatly expanded (still super fast) Also tested `db_bench -benchmarks=fillrandom,stats -bloom_bits=10 -num=1000000 -partition_index_and_filters -format_version=5 [-optimize_filters_for_memory] [-use_ribbon_filter]` with temporary debug output of generated filter sizes. Bloom+optimize_filters_for_memory: 1 Filter size: 197 (224 in memory) 134 Filter size: 3525 (3584 in memory) 107 Filter size: 4037 (4096 in memory) Total on disk: 904,506 Total in memory: 918,752 Ribbon+optimize_filters_for_memory: 1 Filter size: 3061 (3072 in memory) 110 Filter size: 3573 (3584 in memory) 58 Filter size: 4085 (4096 in memory) Total on disk: 633,021 (-30.0%) Total in memory: 634,880 (-30.9%) Bloom (no offm): 1 Filter size: 261 (320 in memory) 1 Filter size: 3333 (3584 in memory) 240 Filter size: 3717 (4096 in memory) Total on disk: 895,674 (-1% on disk vs. +offm; known tolerable overhead of offm) Total in memory: 986,944 (+7.4% vs. +offm) Ribbon (no offm): 1 Filter size: 2949 (3072 in memory) 1 Filter size: 3381 (3584 in memory) 167 Filter size: 3701 (4096 in memory) Total on disk: 624,397 (-30.3% vs. Bloom) Total in memory: 690,688 (-30.0% vs. Bloom) Note that optimize_filters_for_memory is even more effective for Ribbon filter than for cache-local Bloom, because it can close the unused memory gap even tighter than Bloom filter, because of 16 byte increments for Ribbon vs. 64 byte increments for Bloom. Reviewed By: jay-zhuang Differential Revision: D25592970 Pulled By: pdillinger fbshipit-source-id: 606fdaa025bb790d7e9c21601e8ea86e10541912 18 December 2020, 22:31:03 UTC
04b3524 Zhichao Cao 17 December 2020, 19:51:04 UTC Inject the random write error to stress test (#7653) Summary: Inject the random write error to stress test, it requires set reopen=0 and disable_wal=true. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7653 Test Plan: pass db_stress and python3 db_crashtest.py blackbox Reviewed By: ajkr Differential Revision: D25354132 Pulled By: zhichao-cao fbshipit-source-id: 44721104eecb416e27f65f854912c40e301dd669 17 December 2020, 19:52:28 UTC
99f5a80 Akanksha Mahajan 17 December 2020, 05:32:00 UTC Fix clang_analyze error (#7777) Summary: Fix clang_analyze error Pull Request resolved: https://github.com/facebook/rocksdb/pull/7777 Test Plan: USE_CLANG=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j64 analyze Reviewed By: jay-zhuang Differential Revision: D25601675 Pulled By: akankshamahajan15 fbshipit-source-id: 30f58cf4d575a2d546c455fb43e856455eb72a07 17 December 2020, 05:34:41 UTC
29d1274 Adam Retter 16 December 2020, 23:58:56 UTC Fix failing RocksJava test compilation and add CI (#7769) Summary: * Fixes a Java test compilation issue on macOS * Cleans up CircleCI RocksDBJava build config * Adds CircleCI for RocksDBJava on MacOS * Ensures backwards compatibility with older macOS via CircleCI * Fixes RocksJava static builds ordering * Adds missing RocksJava static builds to CircleCI for Mac and Linux * Improves parallelism in RocksJava builds * Reduces the size of the machines used for RocksJava CircleCI as they don't need to be so large (Saves credits) Pull Request resolved: https://github.com/facebook/rocksdb/pull/7769 Reviewed By: akankshamahajan15 Differential Revision: D25601293 Pulled By: pdillinger fbshipit-source-id: 0a0bb9906f65438fe143487d78e37e1947364d08 17 December 2020, 00:00:02 UTC
2021392 Burton Li 15 December 2020, 21:50:07 UTC Do not full scan obsolete files on compaction busy (#7739) Summary: When ConcurrentTaskLimiter is enabled and there are too many outstanding compactions, BackgroundCompaction returns Status::Busy(), which shouldn't be treat as compaction failure. This caused performance issue when outstanding compactions reached the limit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7739 Reviewed By: cheng-chang Differential Revision: D25508319 Pulled By: ltamasi fbshipit-source-id: 3b181b16ada0ca3393cfa3a7412985764e79c719 15 December 2020, 21:51:10 UTC
a0e4421 Jay Zhuang 14 December 2020, 22:06:04 UTC Log sst number in Corruption status (#7767) Summary: sst file number in corruption error would be very useful for debugging Pull Request resolved: https://github.com/facebook/rocksdb/pull/7767 Reviewed By: zhichao-cao Differential Revision: D25485872 Pulled By: jay-zhuang fbshipit-source-id: 67315b582cedeefbce6676015303ebe5bf6526a3 14 December 2020, 22:07:52 UTC
1afbd19 Levi Tamasi 14 December 2020, 21:47:17 UTC Add initial blob support to batched MultiGet (#7766) Summary: The patch adds initial support for reading blobs to the batched `MultiGet` API. The current implementation simply retrieves the blob values as the blob indexes are encountered; that is, reads from blob files are currently not batched. (This will be optimized in a separate phase.) In addition, the patch removes some dead code related to BlobDB from the batched `MultiGet` implementation, namely the `is_blob` / `is_blob_index` flags that are passed around in `DBImpl` and `MemTable` / `MemTableListVersion`. These were never hooked up to anything and wouldn't work anyways, since a single flag is not sufficient to communicate the "blobness" of multiple key-values. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7766 Test Plan: `make check` Reviewed By: jay-zhuang Differential Revision: D25479290 Pulled By: ltamasi fbshipit-source-id: 7aba2d290e31876ee592bcf1adfd1018713a8000 14 December 2020, 21:48:22 UTC
003e72b Peter Dillinger 12 December 2020, 06:17:08 UTC Use size_t for filter APIs, protect against overflow (#7726) Summary: Deprecate CalculateNumEntry and replace with ApproximateNumEntries (better name) using size_t instead of int and uint32_t, to minimize confusing casts and bad overflow behavior (possible though probably not realistic). Bloom sizes are now explicitly capped at max size supported by implementations: just under 4GiB for fv=5 Bloom, and just under 512MiB for fv<5 Legacy Bloom. This hardening could help to set up for fuzzing. Also, since RocksDB only uses this information as an approximation for trying to hit certain sizes for partitioned filters, it's more important that the function be reasonably fast than for it to be completely accurate. It's hard enough to be 100% accurate for Ribbon (currently reversing CalculateSpace) that adding optimize_filters_for_memory into the mix is just not worth trying to be 100% accurate for num entries for bytes. Also: - Cleaned up filter_policy.h to remove MSVC warning handling and potentially unsafe use of exception for "not implemented" - Correct the number of entries limit beyond which current Ribbon implementation falls back on Bloom instead. - Consistently use "num_entries" rather than "num_entry" - Remove LegacyBloomBitsBuilder::CalculateNumEntry as it's essentially obsolete from general implementation BuiltinFilterBitsBuilder::CalculateNumEntries. - Fix filter_bench to skip some tests that don't make sense when only one or a small number of filters has been generated. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7726 Test Plan: expanded existing unit tests for CalculateSpace / ApproximateNumEntries. Also manually used filter_bench to verify Legacy and fv=5 Bloom size caps work (much too expensive for unit test). Note that the actual bits per key is below requested due to space cap. $ ./filter_bench -impl=0 -bits_per_key=20 -average_keys_per_filter=256000000 -vary_key_count_ratio=0 -m_keys_total_max=256 -allow_bad_fp_rate ... Total size (MB): 511.992 Bits/key stored: 16.777 ... $ ./filter_bench -impl=2 -bits_per_key=20 -average_keys_per_filter=2000000000 -vary_key_count_ratio=0 -m_keys_total_max=2000 ... Total size (MB): 4096 Bits/key stored: 17.1799 ... $ Reviewed By: jay-zhuang Differential Revision: D25239800 Pulled By: pdillinger fbshipit-source-id: f94e6d065efd31e05ec630ae1a82e6400d8390c4 12 December 2020, 06:18:12 UTC
4917795 Cheng Chang 12 December 2020, 00:07:48 UTC Update SstFileWriter fuzzer to iterate and check all key-value pairs (#7761) Summary: as title Pull Request resolved: https://github.com/facebook/rocksdb/pull/7761 Test Plan: cd fuzz && make sst_file_writer_fuzzer && ./sst_file_writer_fuzzer Reviewed By: pdillinger Differential Revision: D25430802 Pulled By: cheng-chang fbshipit-source-id: 01436307df6f4c434bb608f44e1c8e4a1119f94f 12 December 2020, 00:09:10 UTC
b1ee191 Peter Dillinger 11 December 2020, 19:17:11 UTC Fix memory leak for ColumnFamily drop with live iterator (#7749) Summary: Uncommon bug seen by ASAN with ColumnFamilyTest.LiveIteratorWithDroppedColumnFamily, if the last two references to a ColumnFamilyData are both SuperVersions (during InstallSuperVersion). The fix is to use UnrefAndTryDelete even in SuperVersion::Cleanup but with a parameter to avoid re-entering Cleanup on the same SuperVersion being cleaned up. ColumnFamilyData::Unref is considered unsafe so removed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7749 Test Plan: ./column_family_test --gtest_filter=*LiveIter* --gtest_repeat=100 Reviewed By: jay-zhuang Differential Revision: D25354304 Pulled By: pdillinger fbshipit-source-id: e78f3a3f67c40013b8432f31d0da8bec55c5321c 11 December 2020, 19:18:21 UTC
07c0fc0 Jay Zhuang 11 December 2020, 04:36:44 UTC Fix pyenv-version-name not found issue (#7768) Summary: Remove the workaround for that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7768 Reviewed By: cheng-chang Differential Revision: D25487150 Pulled By: jay-zhuang fbshipit-source-id: 4514dfc64eb56d2dad48f9a24176c21af8107def 11 December 2020, 04:37:51 UTC
fd7d8dc Cheng Chang 10 December 2020, 20:54:34 UTC Do not log unnecessary WAL obsoletion events (#7765) Summary: min_wal_number_to_keep should not be decreasing, if it does not increase, then there is no need to log the WAL obsoletions in MANIFEST since a previous one has been logged. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7765 Test Plan: watch existing tests and stress tests to pass Reviewed By: pdillinger Differential Revision: D25462542 Pulled By: cheng-chang fbshipit-source-id: 0085fcb6edf5cf2b0fc32f9932a7566f508768ff 10 December 2020, 20:55:49 UTC
40f2b65 Azat Khuzhin 10 December 2020, 17:33:53 UTC Eliminate possible race between LockFile() vs UnlockFile() (#7721) Summary: LockFile() accessing LockHoldingInfo (element of locked_files) by reference after mutex_locked_files had been released. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7721 Reviewed By: pdillinger Differential Revision: D25431839 Pulled By: jay-zhuang fbshipit-source-id: eefee93f12a8016a98e2466e442af2605b3e2a5e 10 December 2020, 17:35:11 UTC
8ff6557 Adam Retter 10 December 2020, 05:19:55 UTC Add further tests to ASSERT_STATUS_CHECKED (2) (#7698) Summary: Second batch of adding more tests to ASSERT_STATUS_CHECKED. * external_sst_file_basic_test * checkpoint_test * db_wal_test * db_block_cache_test * db_logical_block_size_cache_test * db_blob_index_test * optimistic_transaction_test * transaction_test * point_lock_manager_test * write_prepared_transaction_test * write_unprepared_transaction_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7698 Reviewed By: cheng-chang Differential Revision: D25441664 Pulled By: pdillinger fbshipit-source-id: 9e78867f32321db5d4833e95eb96c5734526ef00 10 December 2020, 05:21:16 UTC
8e2749f Michael Lee 10 December 2020, 03:23:06 UTC Fix use of positional args in BUCK rules (#7760) Summary: Prefer to use keyword args rather than positional args for Buck rules. This appears to be the only remaining instance for `custom_unittest` Pull Request resolved: https://github.com/facebook/rocksdb/pull/7760 Test Plan: Search for other instances of `custom_unittest` without `name` Reviewed By: cheng-chang Differential Revision: D25439887 Pulled By: mzlee fbshipit-source-id: 518c541a5c01207c7b0c1f7322addf5cc4f09f92 10 December 2020, 03:25:31 UTC
7123990 Manuel Ung 10 December 2020, 03:12:10 UTC Invalidate iterator on transaction clear (#7733) Summary: Some clients do not close their iterators until after the transaction finishes. To handle this case, we will invalidate any iterators on transaction clear. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7733 Reviewed By: cheng-chang Differential Revision: D25261158 Pulled By: lth fbshipit-source-id: b91320f00c54cbe0e6882b794b34f3bb5640dbc0 10 December 2020, 03:13:22 UTC
80159f6 Cheng Chang 10 December 2020, 03:05:14 UTC Carry over min_log_number_to_keep_2pc in new MANIFEST (#7747) Summary: When two phase commit is enabled, `VersionSet::min_log_number_to_keep_2pc` is set during flush. But when a new MANIFEST is created, the `min_log_number_to_keep_2pc` is not carried over to the new MANIFEST. So if a new MANIFEST is created and then DB is reopened, the `min_log_number_to_keep_2pc` will be lost. This may cause DB recovery errors. The bug is reproduced in a new unit test in `version_set_test.cc`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7747 Test Plan: The new unit test in `version_set_test.cc` should pass. Reviewed By: jay-zhuang Differential Revision: D25350661 Pulled By: cheng-chang fbshipit-source-id: eee890d5b19f15769069670692e270ae31044ece 10 December 2020, 03:07:25 UTC
8a1488e anand76 10 December 2020, 00:57:54 UTC Ensure that MultiGet works properly with compressed cache (#7756) Summary: Ensure that when direct IO is enabled and a compressed block cache is configured, MultiGet inserts compressed data blocks into the compressed block cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7756 Test Plan: Add unit test to db_basic_test Reviewed By: cheng-chang Differential Revision: D25416240 Pulled By: anand1976 fbshipit-source-id: 75d57526370c9c0a45ff72651f3278dbd8a9086f 10 December 2020, 01:01:13 UTC
3c2a448 Cheng Chang 10 December 2020, 00:56:26 UTC Add a test for disabling tracking WAL (#7757) Summary: If WAL tracking was enabled, then disabled during reopen, the previously tracked WALs should be removed from MANIFEST. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7757 Test Plan: a new unit test `DBBasicTest.DisableTrackWal` is added. Reviewed By: jay-zhuang Differential Revision: D25410508 Pulled By: cheng-chang fbshipit-source-id: 9d8d9e665066135930a7c1035bb8c2f68bded6a0 10 December 2020, 00:58:26 UTC
89cc06b Cheng Chang 10 December 2020, 00:23:52 UTC Add a new db_map_fuzzer (#7762) Summary: Execute randomly generated operations on both a DB and a std::map, then reopen the DB and make sure that iterating the DB produces the same key-value pairs as iterating through the std::map. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7762 Test Plan: cd fuzz && make db_map_fuzzer && ./db_map_fuzzer Reviewed By: pdillinger Differential Revision: D25437485 Pulled By: cheng-chang fbshipit-source-id: 3a93f7efd046b194193e45d2ab1ad81565510781 10 December 2020, 00:26:35 UTC
efe827b Cheng Chang 10 December 2020, 00:01:07 UTC Always track WAL obsoletion (#7759) Summary: Currently, when a WAL becomes obsolete after flushing, if VersionSet::WalSet does not contain the WAL, we do not track the WAL obsoletion event in MANIFEST. But consider this case: * WAL 10 is synced, a VersionEdit is LogAndApplied to MANIFEST to log this WAL addition event, but the VersionEdit is not applied to WalSet yet since its corresponding ManifestWriter is still pending in the write queue; * Since the above ManifestWriter is blocking, the LogAndApply will block on a conditional variable and release the db mutex, so another LogAndApply can proceed to enqueue other VersionEdits concurrently; * Now flush happens, and WAL 10 becomes obsolete, although WalSet does not contain WAL 10 yet, we should call LogAndApply to enqueue a VersionEdit to indicate the obsoletion of WAL 10; * otherwise, when the queued edit indicating WAL 10 addition is logged to MANIFEST, and DB crashes and reopens, the WAL 10 might have been removed from disk, but it still exists in MANIFEST. This PR changes the behavior to: always `LogAndApply` any WAL addition or obsoletion event, without considering the order issues caused by concurrency, but when applying the edits to `WalSet`, do not add the WALs if they are already obsolete. In this approach, the logical events of WAL addition and obsoletion are always tracked in MANIFEST, so we can inspect the MANIFEST and know all the previous WAL events, but we choose to ignore certain events due to the concurrency issues such as the case above, or the case in https://github.com/facebook/rocksdb/pull/7725. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7759 Test Plan: make check Reviewed By: pdillinger Differential Revision: D25423089 Pulled By: cheng-chang fbshipit-source-id: 9cb9a7fbc1875bf954f2a42f9b6cfd6d49a7b21c 10 December 2020, 00:02:12 UTC
98236fb Sergei Petrunia 09 December 2020, 20:09:46 UTC LockTree library, originally from PerconaFT (#7753) Summary: To be used for implementing Range Locking. Pull Request resolved: https://github.com/facebook/rocksdb/pull/7753 Reviewed By: zhichao-cao Differential Revision: D25378980 Pulled By: cheng-chang fbshipit-source-id: 801a9c5cd92a84654ca2586b73e8f69001e89320 09 December 2020, 20:10:57 UTC
7b2216c Adam Retter 08 December 2020, 23:53:59 UTC Add further tests to ASSERT_STATUS_CHECKED (1) (#7679) Summary: First batch of adding more tests to ASSERT_STATUS_CHECKED. * db_iterator_test * db_memtable_test * db_merge_operator_test * db_merge_operand_test * write_batch_test * write_batch_with_index_test Pull Request resolved: https://github.com/facebook/rocksdb/pull/7679 Reviewed By: ajkr Differential Revision: D25399270 Pulled By: pdillinger fbshipit-source-id: 3017d0a686aec5cd2d743fc2acbbf75df239f3ba 08 December 2020, 23:55:04 UTC
back to top