Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
16 July 2024, 15:56:10 UTC
sort by:
Revision Author Date Message Commit Date
5709e97 Maysam Yabandeh 07 January 2020, 23:33:05 UTC Skip CancelAllBackgroundWork if DBImpl is already closed (#6268) Summary: WritePreparedTxnDB calls CancelAllBackgroundWork in its destructor to avoid dangling references to it from background job's SnapshotChecker callback. However, if the DBImpl is already closed, the info log might be closed with it, which causes memory leak when CancelAllBackgroundWork tries to print to the info log. The patch fixes that by calling CancelAllBackgroundWork only if the db is not closed already. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6268 Differential Revision: D19303439 Pulled By: maysamyabandeh fbshipit-source-id: 4228a6be7e78d43c90630347baa89b008200bd15 07 January 2020, 23:34:27 UTC
1ab1231 wolfkdy 07 January 2020, 22:19:06 UTC parallel occ (#6240) Summary: This is a continuation of https://github.com/facebook/rocksdb/pull/5320/files I open a new mr for these purposes, half a year has past since the old mr is posted so it's almost impossible to fulfill some points below on the old mr, especially 5) 1) add validation modes for optimistic txns 2) modify unittests to test both modes 3) make format 4) refine hash functor 5) push to master Pull Request resolved: https://github.com/facebook/rocksdb/pull/6240 Differential Revision: D19301296 fbshipit-source-id: 5b5b3cbd39558f43947f7d2dec6cd31a06386edb 07 January 2020, 22:20:38 UTC
2fdd808 Huisheng Liu 07 January 2020, 21:54:18 UTC Implement getfreespace for WinEnv (#6265) Summary: A new interface method Env::GetFreeSpace was added in https://github.com/facebook/rocksdb/issues/4164. It needs to be implemented for Windows port. Some error_handler_test cases fail on Windows because recovery cannot succeed without free space being reported. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6265 Differential Revision: D19303065 fbshipit-source-id: 1f1a83e53f334284781cf61feabc996e87b945ca 07 January 2020, 21:56:13 UTC
a8b1085 Yanqin Jin 07 January 2020, 21:45:21 UTC Fix test in LITE mode (#6267) Summary: Currently, the recently-added test DBTest2.SwitchMemtableRaceWithNewManifest fails in LITE mode since SetOptions() returns "Not supported". I do not want to put `#ifndef ROCKSDB_LITE` because it reduces test coverage. Instead, just trigger compaction on a different column family. The bg compaction thread calling LogAndApply() may race with thread calling SwitchMemtable(). Test Plan (dev server): make check OPT=-DROCKSDB_LITE make check or run DBTest2.SwitchMemtableRaceWithNewManifest 100 times. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6267 Differential Revision: D19301309 Pulled By: riversand963 fbshipit-source-id: 88cedcca2f985968ed3bb234d324ffa2aa04ca50 07 January 2020, 21:47:03 UTC
bce5189 Yanqin Jin 07 January 2020, 20:30:40 UTC Fix error message (#6264) Summary: Fix an error message when CURRENT is not found. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6264 Differential Revision: D19300699 Pulled By: riversand963 fbshipit-source-id: 303fa206386a125960ecca1dbdeff07422690caf 07 January 2020, 20:32:20 UTC
3e26a94 Connor1996 07 January 2020, 16:35:12 UTC Add oldest snapshot sequence property (#6228) Summary: Add oldest snapshot sequence property, so we can use `db.GetProperty("rocksdb.oldest-snapshot-sequence")` to get the sequence number of the oldest snapshot. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6228 Differential Revision: D19264145 Pulled By: maysamyabandeh fbshipit-source-id: 67fbe5304d89cbc475bd404e30d1299f7b11c010 07 January 2020, 16:36:44 UTC
1aaa145 Yanqin Jin 07 January 2020, 04:08:24 UTC Fix a data race for cfd->log_number_ (#6249) Summary: A thread calling LogAndApply may release db mutex when calling WriteCurrentStateToManifest() which reads cfd->log_number_. Another thread can call SwitchMemtable() and writes to cfd->log_number_. Solution is to cache the cfd->log_number_ before releasing mutex in LogAndApply. Test Plan (on devserver): ``` $COMPILE_WITH_TSAN=1 make db_stress $./db_stress --acquire_snapshot_one_in=10000 --avoid_unnecessary_blocking_io=1 --block_size=16384 --bloom_bits=16 --bottommost_compression_type=zstd --cache_index_and_filter_blocks=1 --cache_size=1048576 --checkpoint_one_in=1000000 --checksum_type=kxxHash --clear_column_family_one_in=0 --compact_files_one_in=1000000 --compact_range_one_in=1000000 --compaction_ttl=0 --compression_max_dict_bytes=16384 --compression_type=zstd --compression_zstd_max_train_bytes=0 --continuous_verification_interval=0 --db=/dev/shm/rocksdb/rocksdb_crashtest_blackbox --db_write_buffer_size=1048576 --delpercent=5 --delrangepercent=0 --destroy_db_initially=0 --enable_pipelined_write=0 --flush_one_in=1000000 --format_version=5 --get_live_files_and_wal_files_one_in=1000000 --index_block_restart_interval=5 --index_type=0 --log2_keys_per_lock=22 --long_running_snapshots=0 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --max_key=1000000 --max_manifest_file_size=16384 --max_write_batch_group_size_bytes=16 --max_write_buffer_number=3 --memtablerep=skip_list --mmap_read=0 --nooverwritepercent=1 --open_files=500000 --ops_per_thread=100000000 --partition_filters=0 --pause_background_one_in=1000000 --periodic_compaction_seconds=0 --prefixpercent=5 --progress_reports=0 --readpercent=45 --recycle_log_file_num=0 --reopen=20 --set_options_one_in=10000 --snapshot_hold_ops=100000 --subcompactions=2 --sync=1 --target_file_size_base=2097152 --target_file_size_multiplier=2 --test_batches_snapshots=1 --use_direct_io_for_flush_and_compaction=0 --use_direct_reads=0 --use_full_merge_v1=0 --use_merge=0 --use_multiget=1 --verify_checksum=1 --verify_checksum_one_in=1000000 --verify_db_one_in=100000 --write_buffer_size=4194304 --write_dbid_to_manifest=1 --writepercent=35 ``` Then repeat the following multiple times, e.g. 100 after compiling with tsan. ``` $./db_test2 --gtest_filter=DBTest2.SwitchMemtableRaceWithNewManifest ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6249 Differential Revision: D19235077 Pulled By: riversand963 fbshipit-source-id: 79467b52f48739ce7c27e440caa2447a40653173 07 January 2020, 04:09:51 UTC
946c43a Yanqin Jin 06 January 2020, 18:55:53 UTC Improve error msg for SstFileWriter Merge (#6261) Summary: Reword the error message when keys are not added in strict ascending order. Specifically, original error message is not clear when application tries to call SstFileWriter::Merge() with duplicate keys. Test plan (dev server) ``` make check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6261 Differential Revision: D19290398 Pulled By: riversand963 fbshipit-source-id: 4dc30a701414e6894db2eb024e3734470c22b371 06 January 2020, 18:57:22 UTC
edaaa1f Qinfan Wu 06 January 2020, 18:44:48 UTC Add range delete function to C-API (#6259) Summary: It seems that the C-API doesn't expose the range delete functionality at the moment, so add the API. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6259 Differential Revision: D19290320 Pulled By: pdillinger fbshipit-source-id: 3f403a4c3446d2042d55f1ece7cdc9c040f40c27 06 January 2020, 18:46:21 UTC
28e5a9a Maysam Yabandeh 06 January 2020, 18:15:00 UTC Increase max_log_size in FlushJob to 1024 bytes (#6258) Summary: When measure_io_stats_ is enabled, the volume of logging is beyond the default limit of 512 size. The patch allows the EventLoggerStream to change the limit, and also sets it to 1024 for FlushJob. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6258 Differential Revision: D19279269 Pulled By: maysamyabandeh fbshipit-source-id: 3fb5d468dad488f289ac99d713378177eb7504d6 06 January 2020, 18:16:52 UTC
83957dc Maysam Yabandeh 03 January 2020, 21:05:21 UTC Exclude MergeInProgress status from errors in stress tests (#6257) Summary: When called on transactions, MultiGet could return a legit MergeInProgress status. The patch excludes this case from errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6257 Differential Revision: D19275787 Pulled By: maysamyabandeh fbshipit-source-id: f7158229422af015947e592ae066b4273c9fb9a4 03 January 2020, 21:07:35 UTC
7c98d71 Maysam Yabandeh 03 January 2020, 00:43:55 UTC Print before AddErrors in stress tests (#6256) Summary: Stress tests count number of errors and report them at the end. Not all the cases are accompanied with a log line which makes debugging difficult. The patch adds a log line to the remaining cases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6256 Differential Revision: D19268785 Pulled By: maysamyabandeh fbshipit-source-id: bdabcaa5c5c7edcb4ce4f25e38fd8a3fd9c7700b 03 January 2020, 00:46:23 UTC
48a678b Maysam Yabandeh 03 January 2020, 00:13:30 UTC Prevent an incompatible combination of options (#6254) Summary: allow_concurrent_memtable_write is incompatible with non-zero max_successive_merges. Although we check this at runtime, we currently don't prevent the user from setting this combination in options. This has led to stress tests to fail with this combination is tried in ::SetOptions. The patch fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6254 Differential Revision: D19265819 Pulled By: maysamyabandeh fbshipit-source-id: 47f2e2dc26fe0972c7152f4da15dadb9703f1179 03 January 2020, 00:15:06 UTC
83108d2 Peter Dillinger 30 December 2019, 18:13:50 UTC Re-enable level_compaction_dynamic_level_bytes in crash test (#6251) Summary: Was probably a false signal suggesting a problem in https://github.com/facebook/rocksdb/issues/6217 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6251 Test Plan: 'make crash_test' Differential Revision: D19246951 Pulled By: pdillinger fbshipit-source-id: 3e4fafcd9a7cf5f19ffd207b90279ba615145d6f 30 December 2019, 18:15:49 UTC
faebc33 kim.sanghyun 26 December 2019, 19:12:49 UTC Fixed spelling in function comments (#6248) Summary: meareTime() -> measureTime() Pull Request resolved: https://github.com/facebook/rocksdb/pull/6248 Differential Revision: D19231406 Pulled By: riversand963 fbshipit-source-id: 20de4a43a5478b4a3e7853e1fb70b09ccadbf985 26 December 2019, 19:14:11 UTC
95d226d Peter Dillinger 25 December 2019, 02:45:11 UTC Fix a clang analyzer report, and 'analyze' make rule (#6244) Summary: Clang analyzer was falsely reporting on use of txn=nullptr. Added a new const variable so that it can properly prune impossible control flows. Also, 'make analyze' previously required setting USE_CLANG=1 as an environment variable, not a make variable, or else compilation errors like g++: error: unrecognized command line option ‘-Wshorten-64-to-32’ Now USE_CLANG is not required for 'make analyze' (it's implied) and you can do an incremental analysis (recompile what has changed) with 'USE_CLANG=1 make analyze_incremental' Pull Request resolved: https://github.com/facebook/rocksdb/pull/6244 Test Plan: 'make -j24 analyze', 'make crash_test' Differential Revision: D19225950 Pulled By: pdillinger fbshipit-source-id: 14f4039aa552228826a2de62b2671450e0fed3cb 25 December 2019, 02:46:40 UTC
37fd2b9 Peter Dillinger 24 December 2019, 00:31:08 UTC Revert "Generate variable length keys in db_stress (#6165)" and follow-ups (#6243) Summary: This commit is suspected in some crash test failures such as Verification failed for column family 0 key 78438077: Value not found: NotFound: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6243 Test Plan: 'make check' and start 'make crash_test' Differential Revision: D19220495 Pulled By: pdillinger fbshipit-source-id: 6c4709cee80ab4344e06ce360f51e947d79fb3fa 24 December 2019, 00:32:57 UTC
5f55989 Peter Dillinger 22 December 2019, 05:10:31 UTC Disable occasionally failing assertion in TestPrefixScan (#6238) Summary: Seeing crash test failures like db_stress: db_stress_tool/no_batched_ops_stress.cc:271: virtual rocksdb::Status rocksdb::NonBatchedOpsStressTest::TestPrefixScan(rocksdb::ThreadState*, const rocksdb::ReadOptions&, const std::vector<int>&, const std::vector<long int>&): Assertion `count <= GetPrefixKeyCount(prefix.ToString(), upper_bound)' failed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6238 Differential Revision: D19210312 Pulled By: pdillinger fbshipit-source-id: 4d2c35c38f418b408e01c7ba22adf6983ae67d44 22 December 2019, 05:12:11 UTC
22fea0b Peter Dillinger 22 December 2019, 04:56:57 UTC Fix unused variable in release build Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6237 Differential Revision: D19210304 Pulled By: pdillinger fbshipit-source-id: f6f050e995f4d210f812bb1d2020adbd751e1d5a 22 December 2019, 04:58:30 UTC
d4da412 anand76 21 December 2019, 07:10:58 UTC Add Transaction::MultiGet to db_stress (#6227) Summary: Call Transaction::MultiGet from TestMultiGet() in db_stress. We add some Puts/Merges/Deletes into the transaction in order to exercise MultiGetFromBatchAndDB. There is no data verification on read, just check status. There is currently no read data verification in db_stress as it requires synchronization with writes. It needs to be tackled separately. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6227 Test Plan: make crash_test_with_txn Differential Revision: D19204611 Pulled By: anand1976 fbshipit-source-id: 770d0e30d002e88626c264c58103f1d709bb060c 21 December 2019, 07:12:51 UTC
e0f9d11 sdong 21 December 2019, 06:08:44 UTC db_stress should not keep manifest files under checkpoint directory (#6233) Summary: Recently db_stress starts to use a special Env that keeps all manifest files. This should not apply to checkpoint directory and causes test failure like this: Verification failed: Checkpoint gave inconsistent state. Status is IO error: While mkdir: /dev/shm/rocksdb/rocksdb_crashtest_whitebox/.checkpoint27.tmp: File exists Pull Request resolved: https://github.com/facebook/rocksdb/pull/6233 Test Plan: Run crash_test with high chance of checkpoint and make sure it doesn't reproduce. Differential Revision: D19207250 fbshipit-source-id: 12a931379e2e0572bb84aa658b6d03770c8551d4 21 December 2019, 06:10:06 UTC
9d36c06 sdong 21 December 2019, 05:45:01 UTC db_stress: listners to implement all functions (#6197) Summary: Listners are one source of bugs because we frequently release some mutex to invoke them, which introduce race conditions. Implement all callback functions in db_stress's listener class, and randomly sleep. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6197 Test Plan: Run crash_test for a while and see no obvious problem. Differential Revision: D19134015 fbshipit-source-id: b9ea8be9366e4501759119520cd4f204943538f6 21 December 2019, 05:47:06 UTC
79cc8dc sdong 21 December 2019, 05:42:19 UTC db_stress: cover approximate size (#6213) Summary: db_stress to execute DB::GetApproximateSizes() with randomized keys and options. Return value is not validated but error will be reported. Two ways to generate the range keys: (1) two random keys; (2) a small range. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6213 Test Plan: (1) run "make crash_test" for a while; (2) hack the code to ingest some errors to see it is reported. Differential Revision: D19204665 fbshipit-source-id: 652db36f13bcb5a3bd8fe4a10c0aa22a77a0bce2 21 December 2019, 05:43:35 UTC
3160edf anand76 21 December 2019, 05:09:04 UTC Generate variable length keys in db_stress (#6165) Summary: Currently, db_stress generates fixed length keys of 8 bytes. This patch adds the ability to generate variable length keys. Most of the db_stress code continues to work with a numeric key randomly generated, and the numeric key also acts as an index into the values_ array. The numeric key is mapped to a variable length string key in a deterministic way. Furthermore, the ordering is preserved. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6165 Test Plan: run make crash_test Differential Revision: D19204646 Pulled By: anand1976 fbshipit-source-id: d2d46a96615b4832a8be2a981f5913905f0e1ca7 21 December 2019, 05:10:33 UTC
338c149 sdong 21 December 2019, 00:13:19 UTC crash_test to cover bottommost compression and some other changes (#6215) Summary: Several improvements to crash_test/stress_test: (1) Stress_test to support an parameter of bottommost compression (2) Rename those FLAGS_* variables that are not gflags to avoid confusion (3) Crash_test to randomly generate compression type for bottommost compression with half the chance. (4) Stress_test to sanitize unsupported compression type to snappy, so that crash_test to cover all possible compression types and people don't need to worry about they don't support all comrpession types in their environment. (5) In crash_test, when generating db_stress command, sort arguments in alphabeta order, so that it is easier to find value for a specific argument. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6215 Test Plan: Run "make crash_test" for a while and see the botommost option shown in LOG files. Differential Revision: D19171255 fbshipit-source-id: d7001e246c4ff9ee5760776eea0be97738650735 21 December 2019, 00:14:52 UTC
e55c2b3 sdong 20 December 2019, 22:54:30 UTC db_stress: improvements in TestIterator (#6166) Summary: 1. Cover SeekToFirst() and SeekToLast(). 2. Try to record the history of iterator operations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6166 Test Plan: Do some manual changes in the code to cover the failure cases and see the error printing is correct and SeekToFirst() and SeekToLast() sometimes show up. Differential Revision: D19047079 fbshipit-source-id: 1ed616f919fe4d32c0a021fc37932a7bd3063bcd 20 December 2019, 22:56:15 UTC
e697da0 Adam Retter 20 December 2019, 22:25:52 UTC RocksDB#keyMayExist should not assume database values are unicode strings (#6186) Summary: Closes https://github.com/facebook/rocksdb/issues/6183 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6186 Differential Revision: D19201281 Pulled By: pdillinger fbshipit-source-id: 1c96b4ea09e826f91e44b0009eba3de0991d9053 20 December 2019, 22:27:58 UTC
4d3264e Adam Retter 20 December 2019, 21:39:11 UTC Cleanup deprecation warnings and javadoc (#6218) Summary: There are no API changes ;-) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6218 Differential Revision: D19200373 Pulled By: pdillinger fbshipit-source-id: 58d34b01ea53b75a1eccbd72f8b14d6256a7380f 20 December 2019, 21:41:00 UTC
f89dea4 Zhichao Cao 20 December 2019, 20:05:48 UTC db_stress: Added the verification for GetLiveFiles, GetSortedWalFiles and GetCurrentWalFile (#6224) Summary: Add the verification in operateDB to verify GetLiveFiles, GetSortedWalFiles and GetCurrentWalFile. The test will be called every 1 out of N, N is decided by get_live_files_and_wal_files_one_i, whose default is 1000000. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6224 Test Plan: pass db_stress default run. Differential Revision: D19183358 Pulled By: zhichao-cao fbshipit-source-id: 20073cf72ede77a3e0d3cf5f28304f1f605d2b1a 20 December 2019, 20:07:30 UTC
c4fd9cf Yanqin Jin 20 December 2019, 19:27:41 UTC Remove an unnecessary check before running db_stress (#6231) Summary: As title. We can run non-cf-consistency stress tests with verify_db_one_in>0, thus remove the check added previously. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6231 Test Plan: ``` make crash_test ``` Differential Revision: D19198295 Pulled By: riversand963 fbshipit-source-id: e874c701bb03ab76eaab00f059dd4032bb2f537f 20 December 2019, 19:29:06 UTC
1ebaa76 Levi Tamasi 20 December 2019, 18:59:19 UTC Log garbage_collection_cutoff alongside the other BlobDB options Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6229 Differential Revision: D19191195 Pulled By: ltamasi fbshipit-source-id: 2a3c4785299641a46e022fc012460b759a689fce 20 December 2019, 19:00:53 UTC
786c3d4 Levi Tamasi 20 December 2019, 18:25:48 UTC Support BlobDB in db_stress (#6230) Summary: The patch adds support for BlobDB to `db_stress`. Note that BlobDB currently does not support (amongst other features) Column Families or the `SingleDelete` API, so for now, those should be disabled on the command line when running `db_stress` in BlobDB mode (using `-column_families=1` and `-nooverwritepercent=0`, respectively). Also, some BlobDB features that do not go well with the verification logic in `db_stress` like TTL and FIFO eviction are not supported currently. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6230 Test Plan: ``` ./db_stress -max_key=100000 -use_blob_db -column_families=1 -nooverwritepercent=0 -reopen=1 -blob_db_file_size=1000000 -target_file_size_base=1000000 -blob_db_enable_gc -blob_db_gc_cutoff=0.1 -blob_db_min_blob_size=10 -blob_db_bytes_per_sync=16384 ``` Differential Revision: D19191476 Pulled By: ltamasi fbshipit-source-id: 35840452af8c5e6095249c7fd9a53a119a0985fc 20 December 2019, 18:27:56 UTC
670a916 Yanqin Jin 20 December 2019, 16:46:52 UTC Add more verification to db_stress (#6173) Summary: Currently, db_stress performs verification by calling `VerifyDb()` at the end of test and optionally before tests start. In case of corruption or incorrect result, it will be too late. This PR adds more verification in two ways. 1. For cf consistency test, each test thread takes a snapshot and verifies every N ops. N is configurable via `-verify_db_one_in`. This option is not supported in other stress tests. 2. For cf consistency test, we use another background thread in which a secondary instance periodically tails the primary (interval is configurable). We verify the secondary. Once an error is detected, we terminate the test and report. This does not affect other stress tests. Test plan (devserver) ``` $./db_stress -test_cf_consistency -verify_db_one_in=0 -ops_per_thread=100000 -continuous_verification_interval=100 $./db_stress -test_cf_consistency -verify_db_one_in=1000 -ops_per_thread=10000 -continuous_verification_interval=0 $make crash_test ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/6173 Differential Revision: D19047367 Pulled By: riversand963 fbshipit-source-id: aeed584ad71f9310c111445f34975e5ab47a0615 20 December 2019, 16:49:29 UTC
7a7ca8e Levi Tamasi 20 December 2019, 02:03:24 UTC BlobDB: only compare CF IDs when checking whether an API call is for the default CF (#6226) Summary: BlobDB currently only supports using the default column family. The earlier code enforces this by comparing the `ColumnFamilyHandle` passed to the `Get`/`Put`/etc. call with the handle returned by `DefaultColumnFamily` (which, at the end of the day, comes from `DBImpl::default_cf_handle_`). Since other `ColumnFamilyHandle`s can also point to the default column family, this can reject legitimate requests as well. (As an example, with the earlier code, the handle returned by `BlobDB::Open` cannot actually be used in API calls.) The patch fixes this by comparing only the IDs of the column family handles instead of the pointers themselves. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6226 Test Plan: `make check` Differential Revision: D19187461 Pulled By: ltamasi fbshipit-source-id: 54ce2e12ebb1f07e6d1e70e3b1e0213dfa94bda2 20 December 2019, 02:05:49 UTC
1ba92b8 Peter Dillinger 19 December 2019, 23:35:37 UTC Only search specific directories in Python check (#6225) Summary: The new Python syntax check could fail if external entities were cloned or symlinked to a subdir in a rocksdb git clone. (E.g. Facebook internal LITE build.) Only look for Python files in specific subdirs Pull Request resolved: https://github.com/facebook/rocksdb/pull/6225 Test Plan: python tools/check_all_python.py (still 34 files checked) Reviewed By: gfosco Differential Revision: D19186110 Pulled By: pdillinger fbshipit-source-id: 1fefa54e36b32cd5d96d3d1a43e8a2a694c22ea5 19 December 2019, 23:37:30 UTC
f295b09 sdong 19 December 2019, 22:54:48 UTC BlockBasedTable::ApproximateSize() should use total order seek (#6222) Summary: Right now BlockBasedTable::ApproximateSize() uses default setting about whether to use total order seek. There is no reason for that. There is no reason to do any filtering for approximate size boundary key, and it may introduce bugs. Disable it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6222 Test Plan: Run existing tests Differential Revision: D19184787 fbshipit-source-id: 64180660bd2800914fff75104172b61c06f0b1c9 19 December 2019, 22:56:38 UTC
873331f Peter Dillinger 19 December 2019, 22:03:14 UTC Refactor pulling out parts of StressTest::OperateDb (#6195) Summary: Complete some refactoring called for in https://github.com/facebook/rocksdb/issues/6148. Somehow I got some 'make format' in here for files I didn't change, but that should be OK. (I'm not sure why "hide whitespace changes" doesn't seem to help in review.) Not addressed in this PR: some operations simply print to stdout rather than failing on discovering a bad status or inconsistency. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6195 Differential Revision: D19131067 Pulled By: pdillinger fbshipit-source-id: 4f416e6b792023989ef119f385fe122426cb825d 19 December 2019, 22:04:49 UTC
77d5ba7 Maysam Yabandeh 19 December 2019, 18:45:24 UTC Revert "Add kHashSearch to stress tests (#6210)" (#6220) Summary: This reverts commit 54f9092b0c12d99971f340e180d42fffc9f73bc1. It making our daily stress tests fail. Revert it until the issues are fixed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6220 Differential Revision: D19179881 Pulled By: maysamyabandeh fbshipit-source-id: 99de0eaf776567fa81110b9ad2608234a16083ce 19 December 2019, 18:46:55 UTC
9ff569b Peter Dillinger 19 December 2019, 18:22:43 UTC Temporarily disable level_compaction_dynamic_level_bytes in crash test (#6217) Summary: We're seeing assertion violations like this in crash test: db_stress: table/block_based/block_based_table_reader.cc:4129: virtual uint64_t rocksdb::BlockBasedTable::ApproximateSize(const rocksdb::Slice&, const rocksdb::Slice&, rocksdb::TableReaderCaller): Assertion `end_offset >= start_offset' failed.*** And ApproximateSize appears only to be called with the level_compaction_dynamic_level_bytes option. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6217 Test Plan: temporarily put an assert(false) in ApproximateSize and briefly run 'make crash_test' Differential Revision: D19179174 Pulled By: pdillinger fbshipit-source-id: 506e6549aea0da19b363a1a6da04373c364d92e4 19 December 2019, 18:24:49 UTC
5b18729 Peter Dillinger 19 December 2019, 16:29:32 UTC Syntax check python files on testing (#6209) Summary: Adds a python script to syntax check all python files in the repository and report any errors. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6209 Test Plan: 'make check' with and without seeded syntax errors. Also look for "No syntax errors in 34 .py files" on success, and in java_test CI output Differential Revision: D19166756 Pulled By: pdillinger fbshipit-source-id: 537df464b767260d66810b4cf4c9808a026c58a4 19 December 2019, 16:31:11 UTC
54f9092 Maysam Yabandeh 19 December 2019, 02:07:41 UTC Add kHashSearch to stress tests (#6210) Summary: Beside extending index_type to kHashSearch, it clarifies in the code base that this feature is incompatible with index_block_restart_interval > 1. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6210 Test Plan: ``` make -j32 crash_test Differential Revision: D19166567 Pulled By: maysamyabandeh fbshipit-source-id: 3aaf75a70a8b462d372d43aac69dbd10df303ec7 19 December 2019, 02:09:30 UTC
130e710 Levi Tamasi 19 December 2019, 01:44:29 UTC Add BlobDB GC cutoff parameter to db_bench (#6211) Summary: The patch makes it possible to set the BlobDB configuration option `garbage_collection_cutoff` on the command line. In addition, it changes the `db_bench` code so that the default values of BlobDB related parameters are taken from the defaults of the actual BlobDB configuration options (note: this changes the the default of `blob_db_bytes_per_sync`). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6211 Test Plan: Ran `db_bench` with various values of the new parameter. Differential Revision: D19166895 Pulled By: ltamasi fbshipit-source-id: 305ccdf0123b9db032b744715810babdc3e3b7d5 19 December 2019, 01:46:08 UTC
ef91894 sdong 19 December 2019, 01:02:39 UTC Fix potential overflow in CalculateSSTWriteHint() (#6212) Summary: level passed into ColumnFamilyData::CalculateSSTWriteHint() can be smaller than base_level in current version, which would cause overflow. We see ubsan complains: db/compaction/compaction_job.cc:1511:39: runtime error: load of value 4294967295, which is not a valid value for type 'Env::WriteLifeTimeHint' and I hope this commit fixes it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6212 Test Plan: Run existing tests and see them to pass. Differential Revision: D19168442 fbshipit-source-id: bf8fd86f85478ecfa7556db46dc3242de8c83dc9 19 December 2019, 01:04:15 UTC
7da8c06 Peter Dillinger 18 December 2019, 21:34:44 UTC Avoid heading tags in javadocs; fix EnvironmentTest (#6208) Summary: Should fix Travis build error that randomly showed up upon using Java 13 version of javadoc. AdvancedColumnFamilyOptionsInterface.java:257: error: unexpected heading used: <H2>, compared to implicit preceding heading: <H3> According to this reference https://bugs.openjdk.java.net/browse/JDK-8220379 it should work to start at h4, but that didn't work, so avoiding headings should be fine. Also fix Java EnvironmentTest for JDK13. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6208 Test Plan: Travis run on PR (don't have Java 13 handy) Differential Revision: D19163105 Pulled By: pdillinger fbshipit-source-id: 4a9419cbe7ef780fba771b8a1508e1ea80d17b3e 18 December 2019, 21:36:30 UTC
f453bcb Jermy Li 18 December 2019, 19:45:18 UTC Add unit tests for concurrent CF iteration and drop (#6180) Summary: improve https://github.com/facebook/rocksdb/issues/6147 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6180 Differential Revision: D19148936 fbshipit-source-id: f691c9879fd51d54e96c1a99670cf85ca4485a89 18 December 2019, 19:54:35 UTC
02193ce sdong 18 December 2019, 18:59:21 UTC Prevent file prefetch when mmap is enabled. (#6206) Summary: Right now, sometimes file prefetching is still on when mmap is enabled. This causes bug of reading wrong data. In this commit, we remove all those possible paths. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6206 Test Plan: make crash_test with compaction_readahead_size, which used to fail. RUn all existing tests. Differential Revision: D19149429 fbshipit-source-id: 9e18ea8c566e416aac9647bdd05afe596634791b 18 December 2019, 19:01:29 UTC
dfb259e Peter Dillinger 18 December 2019, 17:31:05 UTC Fix syntax error (!) in db_crashtest.py (#6207) Summary: Fixes syntax error from https://github.com/facebook/rocksdb/pull/6203 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6207 Test Plan: make blackbox_crash_test -> no more syntax error Differential Revision: D19161752 Pulled By: pdillinger fbshipit-source-id: b3032f296041ab56307762622b9ef6c03a8379aa 18 December 2019, 17:32:52 UTC
c399704 Zhichao Cao 18 December 2019, 08:48:48 UTC Fix: remove the potential dead store variable in block_based_table_reader.cc (#6204) Summary: buf_offset does not need to get the value from req.len for othe final block. It can cause test fail for clan_analyze. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6204 Test Plan: pass make asan_check Differential Revision: D19145335 Pulled By: zhichao-cao fbshipit-source-id: 8f6e74565746381b5c5ef598b97d746517b36e5b 18 December 2019, 09:23:07 UTC
2afea29 anand76 18 December 2019, 04:43:06 UTC Add VerifyChecksum() to db_stress (#6203) Summary: Add an option to db_stress, verify_checksum_one_in, to call DB::VerifyChecksum() once every N ops. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6203 Differential Revision: D19145753 Pulled By: anand1976 fbshipit-source-id: d09edf21f309ad53aa40dd25b7a563d50665fd8b 18 December 2019, 04:44:58 UTC
ce63eda Mike Kolupaev 18 December 2019, 04:07:21 UTC Fix use-after-free and double-deleting files in BackgroundCallPurge() (#6193) Summary: The bad code was: ``` mutex.Lock(); // `mutex` protects `container` for (auto& x : container) { mutex.Unlock(); // do stuff to x mutex.Lock(); } ``` It's incorrect because both `x` and the iterator may become invalid if another thread modifies the container while this thread is not holding the mutex. Broken by https://github.com/facebook/rocksdb/pull/5796 - it replaced a `while (!container.empty())` loop with a `for (auto x : container)`. (RocksDB code does a lot of such unlocking+re-locking of mutexes, and this type of bugs comes up a lot :/ ) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6193 Test Plan: Ran some logdevice integration tests that were crashing without this fix. Differential Revision: D19116874 Pulled By: al13n321 fbshipit-source-id: 9672bc4227c1b68f46f7436db2b96811adb8c703 18 December 2019, 04:08:56 UTC
cbd58af Levi Tamasi 18 December 2019, 03:07:02 UTC Update HISTORY.md with recent BlobDB related changes Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6202 Differential Revision: D19144158 Pulled By: ltamasi fbshipit-source-id: 3e2522ced458568e3a2a045663704e30ab0ac223 18 December 2019, 03:09:21 UTC
9f250dd sdong 18 December 2019, 02:23:26 UTC crash_test: two fixes (#6200) Summary: Fix two crash test issues: 1. sync mode should not run with disable_wal=true 2. disable "compaction_readahead_size" for now. With it on, some block checksum verification failure will happen in compaction paths. Not sure why, but disable it for now to keep the test clean. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6200 Test Plan: Run "make crash_test" and "make crash_test_with_atomic_flush" and see it runs way longer than before the fix without failing. Differential Revision: D19143493 fbshipit-source-id: 438fad52fbda60aafd142e1b65578addbe7d72b1 18 December 2019, 02:25:04 UTC
2d16709 Adam Retter 17 December 2019, 21:52:09 UTC Small tidy and speed up of the travis build (#6181) Summary: Cuts about 30-60 seconds to from each Travis Linux build, and about 15 minutes from each macOS build Pull Request resolved: https://github.com/facebook/rocksdb/pull/6181 Differential Revision: D19098357 Pulled By: pdillinger fbshipit-source-id: 863dd1ab09076ad9b03c2b7914908359628315ae 17 December 2019, 21:56:45 UTC
39fcaf8 解轶伦 17 December 2019, 21:20:42 UTC delete superversions in BackgroundCallPurge (#6146) Summary: I found that CleanupSuperVersion() may block Get() for 30ms+ （per MemTable is 256MB）. Then I found "delete sv" in ~SuperVersion() takes the time. The backtrace looks like this DBImpl::GetImpl() -> DBImpl::ReturnAndCleanupSuperVersion() -> DBImpl::CleanupSuperVersion() : delete sv; -> ~SuperVersion() I think it's better to delete in a background thread, please review it。 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6146 Differential Revision: D18972066 fbshipit-source-id: 0f7b0b70b9bb1e27ad6fc1c8a408fbbf237ae08c 17 December 2019, 21:22:57 UTC
02aa229 Levi Tamasi 17 December 2019, 18:17:27 UTC Set CompactionIterator::valid_ to false when PrepareBlobOutput indicates error Summary: With https://github.com/facebook/rocksdb/pull/6121, errors returned by `PrepareBlobValue` result in `CompactionIterator::status_` being set to `Corruption` or `IOError` as appropriate, however, `valid_` is not set to `false`. The error is eventually propagated in `CompactionJob::ProcessKeyValueCompaction` but only after the main loop completes. Setting `valid_` to `false` upon errors enables us to terminate the loop early and fail the compaction sooner. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6170 Test Plan: Ran `make check` and used `db_bench` in BlobDB mode. fbshipit-source-id: a2ca88a3ca71115e2605bd34a4c795d8a28bef27 17 December 2019, 18:20:16 UTC
1be48cb anand1976 17 December 2019, 04:35:11 UTC Fix crash in Transaction::MultiGet() when num_keys > 32 Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6192 Test Plan: Add a unit test that fails without the fix and passes now make check Differential Revision: D19124781 Pulled By: anand1976 fbshipit-source-id: 8c8cb6fa16c3fc23ec011e168561a13f76bbd783 17 December 2019, 04:39:35 UTC
7678cf2 Yanqin Jin 17 December 2019, 04:00:43 UTC Use Env::LoadEnv to create custom Env objects (#6196) Summary: As title. Previous assumption was that the underlying lib can always return a shared_ptr<Env>. This is too strong. Therefore, we use Env::LoadEnv to relax it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6196 Test Plan: make check Differential Revision: D19133199 Pulled By: riversand963 fbshipit-source-id: c83a0c02a42610d077054f2de1acfc45126b3a75 17 December 2019, 04:03:14 UTC
68d5d82 Maysam Yabandeh 17 December 2019, 02:30:48 UTC Wait for CancelAllBackgroundWork before Close in db stress (#6191) Summary: In https://github.com/facebook/rocksdb/issues/6174 we fixed the stress test to respect the CancelAllBackgroundWork + Close order for WritePrepared transactions. The fix missed to take into account that some invocation of CancelAllBackgroundWork are with wait=false parameter which essentially breaks the order. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6191 Differential Revision: D19102709 Pulled By: maysamyabandeh fbshipit-source-id: f4e7b5fdae47ff1c1ac284ba1cf67d5d3f3d03eb 17 December 2019, 02:33:09 UTC
cddd637 Zhichao Cao 16 December 2019, 23:55:33 UTC Merge adjacent file block reads in RocksDB MultiGet() and Add uncompressed block to cache (#6089) Summary: In the current MultiGet, if the KV-pairs do not belong to the data blocks in the block cache, multiple blocks are read from a SST. It will trigger one block read for each block request and read them in parallel. In some cases, if some data blocks are adjacent in the SST, the reads for these blocks can be combined to a single large read, which can reduce the system calls and reduce the read latency if possible. Considering to fill the block cache, if multiple data blocks are in the same memory buffer, we need to copy them to the heap separately. Therefore, only in the case that 1) data block compression is enabled, and 2) compressed block cache is null, we can do combined read. Otherwise, extra memory copy is needed, which may cause extra overhead. In the current case, data blocks will be uncompressed to a new memory space. Also, in the case that 1) data block compression is enabled, and 2) compressed block cache is null, it is possible the data block is actually not compressed. In the current logic, these data blocks will not be added to the uncompressed_cache. So if memory buffer is shared and the data block is not compressed, the data block are copied to the head and fill the cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6089 Test Plan: Added test case to ParallelIO.MultiGet. Pass make asan_check Differential Revision: D18734668 Pulled By: zhichao-cao fbshipit-source-id: 67c5615ed373e51e42635fd74b36f8f3a66d5da4 17 December 2019, 00:26:03 UTC
bcc372c sdong 16 December 2019, 23:24:26 UTC Add some new options to crash_test (#6176) Summary: Several options are trivially added to crash test and random values are picked. Made simple test run non-dynamic level and normal test run dynamic level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6176 Test Plan: Run crash_test and watch the printing Differential Revision: D19053955 fbshipit-source-id: 958cb43c968541ebd87ed4d91e778bd1d40e7502 16 December 2019, 23:43:13 UTC
2d095b4 Levi Tamasi 16 December 2019, 23:15:42 UTC Update HISTORY.md with the recent memtable trimming fixes Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6194 Differential Revision: D19125292 Pulled By: ltamasi fbshipit-source-id: d41aca2755ec4bec07feedd6b561e8d18606a931 16 December 2019, 23:19:52 UTC
35126dd sdong 16 December 2019, 22:28:06 UTC db_stress: preserve all historic manifest files (#6142) Summary: compaction history is stored in manifest files. Preserve all of them in db_stress would help debugging. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6142 Test Plan: Run db_stress and observe that manifest files are preserved. Run whole crash_test and see how DB directory looks like. Differential Revision: D19047026 fbshipit-source-id: f83c3e0bb5332b1b4768be5dcee56a24f9b760a9 16 December 2019, 22:32:34 UTC
fbda25f Zhichao Cao 16 December 2019, 21:59:21 UTC db_stress: generate the key based on Zipfian distribution (hot key) (#6163) Summary: In the current db_stress, all the keys are generated randomly and follows the uniform distribution. In order to test some corner cases that some key are always updated or read, we need to generate the key based on other distributions. In this PR, the key is generated based on Zipfian distribution and the skewness can be controlled by setting hot_key_alpha (0.8 to 1.5 is suggested). The larger hot_key_alpha is, the more skewed will be. Not that, usually, if hot_key_alpha is larger than 2, there might be only 1 or 2 keys that are generated. If hot_key_alpha is 0, it generate the key follows uniform distribution (random key) Testing plan: pass the db_stress and printed the keys to make sure it follows the distribution. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6163 Differential Revision: D18978480 Pulled By: zhichao-cao fbshipit-source-id: e123b4865477f7478e83fb581f9576bada334680 16 December 2019, 22:01:58 UTC
db7c687 Levi Tamasi 16 December 2019, 21:13:42 UTC Fix a data race related to memtable trimming (#6187) Summary: https://github.com/facebook/rocksdb/pull/6177 introduced a data race involving `MemTableList::InstallNewVersion` and `MemTableList::NumFlushed`. The patch fixes this by caching whether the current version has any memtable history (i.e. flushed memtables that are kept around for transaction conflict checking) in an `std::atomic<bool>` member called `current_has_history_`, similarly to how `current_memory_usage_excluding_last_` is handled. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6187 Test Plan: ``` make clean COMPILE_WITH_TSAN=1 make db_test -j24 ./db_test ``` Differential Revision: D19084059 Pulled By: ltamasi fbshipit-source-id: 327a5af9700fb7102baea2cc8903c085f69543b9 16 December 2019, 21:16:31 UTC
a92bd0a Peter Dillinger 16 December 2019, 05:29:43 UTC Optimize memory and CPU for building new Bloom filter (#6175) Summary: The filter bits builder collects all the hashes to add in memory before adding them (because the number of keys is not known until we've walked over all the keys). Existing code uses a std::vector for this, which can mean up to 2x than necessary space allocated (and not freed) and up to ~2x write amplification in memory. Using std::deque uses close to minimal space (for large filters, the only time it matters), no write amplification, frees memory while building, and no need for large contiguous memory area. The only cost is more calls to allocator, which does not appear to matter, at least in benchmark test. For now, this change only applies to the new (format_version=5) Bloom filter implementation, to ease before-and-after comparison downstream. Temporary memory use during build is about the only way the new Bloom filter could regress vs. the old (because of upgrade to 64-bit hash) and that should only matter for full filters. This change should largely mitigate that potential regression. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6175 Test Plan: Using filter_bench with -new_builder option and 6M keys per filter is like large full filter (improvement). 10k keys and no -new_builder is like partitioned filters (about the same). (Corresponding configurations run simultaneously on devserver.) std::vector impl (before) $ /usr/bin/time -v ./filter_bench -impl=2 -quick -new_builder -working_mem_size_mb=1000 - average_keys_per_filter=6000000 Build avg ns/key: 52.2027 Maximum resident set size (kbytes): 1105016 $ /usr/bin/time -v ./filter_bench -impl=2 -quick -working_mem_size_mb=1000 - average_keys_per_filter=10000 Build avg ns/key: 30.5694 Maximum resident set size (kbytes): 1208152 std::deque impl (after) $ /usr/bin/time -v ./filter_bench -impl=2 -quick -new_builder -working_mem_size_mb=1000 - average_keys_per_filter=6000000 Build avg ns/key: 39.0697 Maximum resident set size (kbytes): 1087196 $ /usr/bin/time -v ./filter_bench -impl=2 -quick -working_mem_size_mb=1000 - average_keys_per_filter=10000 Build avg ns/key: 30.9348 Maximum resident set size (kbytes): 1207980 Differential Revision: D19053431 Pulled By: pdillinger fbshipit-source-id: 2888e748723a19d9ea40403934f13cbb8483430c 16 December 2019, 05:31:08 UTC
ad34fab anand76 14 December 2019, 23:38:04 UTC Fix unity test (#6178) Summary: Fix the test failure. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6178 Differential Revision: D19071208 Pulled By: maysamyabandeh fbshipit-source-id: 71622832ac93ff2663946c546d9642d5b9e3d194 14 December 2019, 23:39:41 UTC
4b97812 Maysam Yabandeh 14 December 2019, 23:17:05 UTC Add long-running snapshots to stress tests (#6171) Summary: Current implementation holds on to 10% of snapshots for 10x longer, and 1% of snapshots 100x longer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6171 Test Plan: ``` make -j32 crash_test Differential Revision: D19038399 Pulled By: maysamyabandeh fbshipit-source-id: 75da2dbb5c47a0b3f37d299b8719e392b73b42c0 14 December 2019, 23:22:40 UTC
bd8404f Levi Tamasi 14 December 2019, 03:09:53 UTC Do not schedule memtable trimming if there is no history (#6177) Summary: We have observed an increase in CPU load caused by frequent calls to `ColumnFamilyData::InstallSuperVersion` from `DBImpl::TrimMemtableHistory` when using `max_write_buffer_size_to_maintain` to limit the amount of memtable history maintained for transaction conflict checking. Part of the issue is that trimming can potentially be scheduled even if there is no memtable history. The patch adds a check that fixes this. See also https://github.com/facebook/rocksdb/pull/6169. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6177 Test Plan: Compared `perf` output for ``` ./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot=1 --threads=32 ``` before and after the change. There is a significant reduction for the call chain `rocksdb::DBImpl::TrimMemtableHistory` -> `rocksdb::ColumnFamilyData::InstallSuperVersion` -> `rocksdb::ThreadLocalPtr::StaticMeta::Scrape` even without https://github.com/facebook/rocksdb/pull/6169. Differential Revision: D19057445 Pulled By: ltamasi fbshipit-source-id: dff81882d7b280e17eda7d9b072a2d4882c50f79 14 December 2019, 03:11:19 UTC
349bd3e Maysam Yabandeh 14 December 2019, 01:56:12 UTC CancelAllBackgroundWork before Close in db stress (#6174) Summary: Close asserts that there is no unreleased snapshots. For WritePrepared transaction, this means that the background work that holds on a snapshot must be canceled first. Update the stress tests to respect the sequence. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6174 Test Plan: ``` make -j32 crash_test Differential Revision: D19057322 Pulled By: maysamyabandeh fbshipit-source-id: c9e9e24f779bbfb0ab72c2717e34576c01bc6362 14 December 2019, 02:22:50 UTC
edbf0e2 Adam Retter 14 December 2019, 00:11:40 UTC Env should also load the native library (#6167) Summary: Closes https://github.com/facebook/rocksdb/issues/6118 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6167 Differential Revision: D19053577 Pulled By: pdillinger fbshipit-source-id: 86aca9a5bec0947a641649b515da17b3cb12bdde 14 December 2019, 00:27:55 UTC
0d2172f Levi Tamasi 13 December 2019, 23:52:38 UTC Make it possible to enable periodic compactions for BlobDB (#6172) Summary: Periodic compactions ensure that even SSTs that do not get picked up otherwise eventually go through compaction; used in conjunction with BlobDB's garbage collection, they enable BlobDB to reclaim space when old blob files are used by such straggling SSTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6172 Test Plan: Ran `make check` and used the BlobDB mode of `db_bench`. Differential Revision: D19045045 Pulled By: ltamasi fbshipit-source-id: 04636ecc4b6cfe8d495bf656faa65d54a5eb1a93 14 December 2019, 00:13:25 UTC
afa2420 anand76 13 December 2019, 22:47:08 UTC Introduce a new storage specific Env API (#5761) Summary: The current Env API encompasses both storage/file operations, as well as OS related operations. Most of the APIs return a Status, which does not have enough metadata about an error, such as whether its retry-able or not, scope (i.e fault domain) of the error etc., that may be required in order to properly handle a storage error. The file APIs also do not provide enough control over the IO SLA, such as timeout, prioritization, hinting about placement and redundancy etc. This PR separates out the file/storage APIs from Env into a new FileSystem class. The APIs are updated to return an IOStatus with metadata about the error, as well as to take an IOOptions structure as input in order to allow more control over the IO. The user can set both ```options.env``` and ```options.file_system``` to specify that RocksDB should use the former for OS related operations and the latter for storage operations. Internally, a ```CompositeEnvWrapper``` has been introduced that inherits from ```Env``` and redirects individual methods to either an ```Env``` implementation or the ```FileSystem``` as appropriate. When options are sanitized during ```DB::Open```, ```options.env``` is replaced with a newly allocated ```CompositeEnvWrapper``` instance if both env and file_system have been specified. This way, the rest of the RocksDB code can continue to function as before. This PR also ports PosixEnv to the new API by splitting it into two - PosixEnv and PosixFileSystem. PosixEnv is defined as a sub-class of CompositeEnvWrapper, and threading/time functions are overridden with Posix specific implementations in order to avoid an extra level of indirection. The ```CompositeEnvWrapper``` translates ```IOStatus``` return code to ```Status```, and sets the severity to ```kSoftError``` if the io_status is retryable. The error handling code in RocksDB can then recover the DB automatically. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5761 Differential Revision: D18868376 Pulled By: anand1976 fbshipit-source-id: 39efe18a162ea746fabac6360ff529baba48486f 13 December 2019, 22:48:41 UTC
58d46d1 Peter Dillinger 13 December 2019, 21:25:14 UTC Add useful idioms to Random API (OneInOpt, PercentTrue) (#6154) Summary: And clean up related code, especially in stress test. (More clean up of db_stress_test_base.cc coming after this.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6154 Test Plan: make check, make blackbox_crash_test for a bit Differential Revision: D18938180 Pulled By: pdillinger fbshipit-source-id: 524d27621b8dbb25f6dff40f1081e7c00630357e 13 December 2019, 22:30:14 UTC
6d54eb3 Levi Tamasi 13 December 2019, 20:45:49 UTC Do not create/install new SuperVersion if nothing was deleted during memtable trim (#6169) Summary: We have observed an increase in CPU load caused by frequent calls to `ColumnFamilyData::InstallSuperVersion` from `DBImpl::TrimMemtableHistory` when using `max_write_buffer_size_to_maintain` to limit the amount of memtable history maintained for transaction conflict checking. As it turns out, this is caused by the code creating and installing a new `SuperVersion` even if no memtables were actually trimmed. The patch adds a check to avoid this. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6169 Test Plan: Compared `perf` output for ``` ./db_bench -benchmarks=randomtransaction -optimistic_transaction_db=1 -statistics -stats_interval_seconds=1 -duration=90 -num=500000 --max_write_buffer_size_to_maintain=16000000 --transaction_set_snapshot=1 --threads=32 ``` before and after the change. With the fix, the call chain `rocksdb::DBImpl::TrimMemtableHistory` -> `rocksdb::ColumnFamilyData::InstallSuperVersion` -> `rocksdb::ThreadLocalPtr::StaticMeta::Scrape` no longer registers in the `perf` report. Differential Revision: D19031509 Pulled By: ltamasi fbshipit-source-id: 02686fce594e5b50eba0710e4b28a9b808c8aa20 13 December 2019, 21:29:29 UTC
ac304ad Kefu Chai 13 December 2019, 19:33:59 UTC cmake: do not build tests for Release build and cleanups (#5916) Summary: fixes https://github.com/facebook/rocksdb/issues/2445 Pull Request resolved: https://github.com/facebook/rocksdb/pull/5916 Differential Revision: D19031236 fbshipit-source-id: bc3107b6b25a01958677d7cb411b1f381aae91c6 13 December 2019, 20:48:06 UTC
fec7302 Maysam Yabandeh 13 December 2019, 18:23:01 UTC Enable unordered_write in stress tests (#6164) Summary: With WritePrepared transactions configured with two_write_queues, unordered_write will offer the same guarantees as vanilla rocksdb and thus can be enabled in stress tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6164 Test Plan: ``` make -j32 crash_test_with_txn Differential Revision: D18991899 Pulled By: maysamyabandeh fbshipit-source-id: eece5e96b4169b67d7931e5c0afca88540a113e1 13 December 2019, 18:25:04 UTC
583c695 Levi Tamasi 13 December 2019, 18:11:03 UTC Move out valid blobs from the oldest blob files during compaction (#6121) Summary: The patch adds logic that relocates live blobs from the oldest N non-TTL blob files as they are encountered during compaction (assuming the BlobDB configuration option `enable_garbage_collection` is `true`), where N is defined as the number of immutable non-TTL blob files multiplied by the value of a new BlobDB configuration option called `garbage_collection_cutoff`. (The default value of this parameter is 0.25, that is, by default the valid blobs residing in the oldest 25% of immutable non-TTL blob files are relocated.) Pull Request resolved: https://github.com/facebook/rocksdb/pull/6121 Test Plan: Added unit test and tested using the BlobDB mode of `db_bench`. Differential Revision: D18785357 Pulled By: ltamasi fbshipit-source-id: 8c21c512a18fba777ec28765c88682bb1a5e694e 13 December 2019, 18:13:05 UTC
c2029f9 Jermy Li 13 December 2019, 03:02:51 UTC Support concurrent CF iteration and drop (#6147) Summary: It's easy to cause coredump when closing ColumnFamilyHandle with unreleased iterators, especially iterators release is controlled by java GC when using JNI. This patch fixed concurrent CF iteration and drop, we let iterators(actually SuperVersion) hold a ColumnFamilyData reference to prevent the CF from being released too early. fixed https://github.com/facebook/rocksdb/issues/5982 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6147 Differential Revision: D18926378 fbshipit-source-id: 1dff6d068c603d012b81446812368bfee95a5e15 13 December 2019, 03:04:48 UTC
4b74035 myasuka 13 December 2019, 02:06:01 UTC Correct java docs of RocksDB options (#6123) Summary: Correct javadocs of several RocksDB option classes to not mislead RocksJava users. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6123 Differential Revision: D18989044 Pulled By: pdillinger fbshipit-source-id: a5ac6a415e5311084b10d973d354e6925788f01e 13 December 2019, 02:10:03 UTC
c4ce8e6 奏之章 12 December 2019, 23:16:13 UTC Fix RangeDeletion bug (#6062) Summary: Read keys from a snapshot that a range deletion were added after the snapshot was created and this range deletion was inside an immutable memtable, we will get wrong key set. More detail rest in codes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6062 Differential Revision: D18966785 Pulled By: pdillinger fbshipit-source-id: 38a60bb1e2d0a1dbfc8ec641617200b6a02b86c3 12 December 2019, 23:18:02 UTC
a844591 Connor 12 December 2019, 22:05:48 UTC wait pending memtable writes on file ingestion or compact range (#6113) Summary: **Summary:** This PR fixes two unordered_write related issues: - ingestion job may skip the necessary memtable flush https://github.com/facebook/rocksdb/issues/6026 - compact range may cause memtable is flushed before pending unordered write finished 1. `CompactRange` triggers memtable flush but doesn't wait for pending-writes 2. there are some pending writes but memtable is already flushed 3. the memtable related WAL is removed( note that the pending-writes were recorded in that WAL). 4. pending-writes write to newer created memtable 5. there is a restart 6. missing the previous pending-writes because WAL is removed but they aren't included in SST. **How to solve:** - Wait pending memtable writes before ingestion job check memtable key range - Wait pending memtable writes before flush memtable. **Note that: `CompactRange` calls `RangesOverlapWithMemtables` too without waiting for pending waits, but I'm not sure whether it affects the correctness.** **Test Plan:** make check Pull Request resolved: https://github.com/facebook/rocksdb/pull/6113 Differential Revision: D18895674 Pulled By: maysamyabandeh fbshipit-source-id: da22b4476fc7e06c176020e7cc171eb78189ecaf 12 December 2019, 22:08:02 UTC
814d4e7 sdong 12 December 2019, 22:02:43 UTC Improve instructions to install formatter (#6162) Summary: While the instruction of installing "make format" dependencies works on some platforms, it is hard to use for some others. Improve it a little bit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6162 Test Plan: Run "make format" on an envrionment missing the dependencies and see the instructions printed out Differential Revision: D18970773 fbshipit-source-id: fd21b31053407cc171a6675f781a556a1c3e8945 12 December 2019, 22:04:01 UTC
a796c06 Maysam Yabandeh 12 December 2019, 21:48:50 UTC Fix build breakage from lock_guard error (#6161) Summary: This change fixes a source issue that caused compile time error which breaks build for many fbcode services in that setup. The size() member function of channel is a const member, so member variables accessed within it are implicitly const as well. This caused error when clang fails to resolve to a constructor that takes std::mutex because the suitable constructor got rejected due to loss of constness for its argument. The fix is to add mutable modifier to the lock_ member of channel. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6161 Differential Revision: D18967685 Pulled By: maysamyabandeh fbshipit-source-id: 698b6a5153c3c92eeacb842c467aa28cc350d432 12 December 2019, 21:50:27 UTC
b433bbe Adam Retter 12 December 2019, 19:58:26 UTC Add missing mutable DBOptions to RocksJava (#6152) Summary: As requested in https://github.com/facebook/rocksdb/issues/6127 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6152 Differential Revision: D18955608 Pulled By: pdillinger fbshipit-source-id: 3e1367d944e44d5f1675a422f7dd2451c86feb6f 12 December 2019, 20:01:19 UTC
3b60761 Levi Tamasi 12 December 2019, 19:29:01 UTC Do not update SST <-> blob file mapping if compaction failed Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6156 Test Plan: Extended unit tests. Differential Revision: D18943867 Pulled By: ltamasi fbshipit-source-id: b3669d2dd6af08e987ad1a59d6712ae2514da0b1 12 December 2019, 19:30:45 UTC
8613ee2 Maysam Yabandeh 12 December 2019, 18:34:52 UTC Enable all txn write policies in crash test (#6158) Summary: Currently the default txn write policy in crash tests is WRITE_PREPARED. The patch randomly picks the write policy at the start of the crash test. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6158 Test Plan: ``` make -j32 crash_test_with_txn ``` Differential Revision: D18946307 Pulled By: maysamyabandeh fbshipit-source-id: f77d7a94f99a08791ef9626da153d284bf521950 12 December 2019, 18:43:49 UTC
e1dfe80 Levi Tamasi 12 December 2019, 01:17:56 UTC Mark BlobIndex::DebugString const Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6157 Test Plan: make check Differential Revision: D18944259 Pulled By: ltamasi fbshipit-source-id: 7fb29447b52d801215bd6ab811e229a7fa2c763d 12 December 2019, 01:19:43 UTC
1ad6fa9 Maysam Yabandeh 11 December 2019, 23:59:45 UTC Enable txn in crash tests (#6155) Summary: Start daily crash tests with use_txn flag. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6155 Differential Revision: D18943630 Pulled By: maysamyabandeh fbshipit-source-id: eea99a6ffd5f57fb9651f6ca7dab8fbf70379c87 12 December 2019, 00:01:55 UTC
d0ad3c5 Peter Dillinger 11 December 2019, 23:15:34 UTC Fix c_test:filter for various CACHE_LINE_SIZEs (#6153) Summary: This test was recently updated but failed to account for Bloom schema variance by CACHE_LINE_SIZE. (Since CACHE_LINE_SIZE is not defined in our C code, the test now simply allows a valid result for any CACHE_LINE_SIZE, not just the current one.) Unblock https://github.com/facebook/rocksdb/issues/5932 Pull Request resolved: https://github.com/facebook/rocksdb/pull/6153 Test Plan: ran unit test with builds TEST_CACHE_LINE_SIZE=128, =256, and unset (64 on Intel) Differential Revision: D18936015 Pulled By: pdillinger fbshipit-source-id: e5e3852f95283d34d624632c1ae8d3adb2f2662c 11 December 2019, 23:17:08 UTC
3717a88 奏之章 11 December 2019, 19:25:33 UTC Fix UniversalCompaction trivial move bug (#6067) Summary: `curr.level` is `c->inputs_` index, not real level. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6067 Differential Revision: D18935726 fbshipit-source-id: 4354e6e9cd900ca56c96e9d770f0ab6634e45daf 11 December 2019, 19:27:53 UTC
afdc58d ferhat elmas 11 December 2019, 19:02:22 UTC Fix typos in history Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/6116 Differential Revision: D18935622 fbshipit-source-id: 59f7a7bc9f0116ae6354ea217896622a34329d3c 11 December 2019, 19:04:46 UTC
05a8631 Yi Wu 11 December 2019, 18:26:35 UTC Remove unused low_pri_write_rate_limiter_ (#6068) Summary: `low_pri_write_rate_limiter_` is not being used. Removing. `WriteController` has an internal low_pri rate limiter which is the real rate limiter for low-pri writes. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6068 Test Plan: make Differential Revision: D18664120 fbshipit-source-id: dfe3e4de033cf3522b67781b383aad7d0936034c 11 December 2019, 18:28:33 UTC
77565d7 Cheng Chang 11 December 2019, 17:55:20 UTC Add example to show the effect of Get in snapshot isolation (#6059) Summary: Adds example to show the difference of reading from snapshot and from the latest state. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6059 Test Plan: cd examples && make transaction_example && ./transaction_example Differential Revision: D18797616 fbshipit-source-id: f17a2cb12187092ea243159e6ccf55790859e0c0 11 December 2019, 17:56:42 UTC
383f507 Yanqin Jin 11 December 2019, 05:53:43 UTC Add SyncWAL to db_stress (#6149) Summary: Add SyncWAL to db_stress. Specify with `-sync_wal_one_in=N` so that it will be called once every N operations on average. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6149 Test Plan: ``` $make db_stress $./db_stress -sync_wal_one_in=100 -ops_per_thread=100000 ``` Differential Revision: D18922529 Pulled By: riversand963 fbshipit-source-id: 4c0b8cb8fa21852722cffd957deddf688f12ea56 11 December 2019, 05:55:25 UTC
7a99162 sdong 11 December 2019, 04:01:25 UTC db_stress: sometimes call CancelAllBackgroundWork() and Close() before closing DB (#6141) Summary: CancelAllBackgroundWork() and Close() are frequently used features but we don't cover it in stress test. Simply execute them before closing the DB with 1/2 chance. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6141 Test Plan: Run "db_stress". Differential Revision: D18900861 fbshipit-source-id: 49b46ccfae120d0f9de3e0543b82fb6d715949d0 11 December 2019, 04:04:52 UTC
984b6e7 Adam Retter 11 December 2019, 04:00:57 UTC Add Visual Studio 2015 to AppVeyor (#5446) Summary: This is required to compile on Windows with Visual Studio 2015, which is used for creating the RocksJava releases. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5446 Differential Revision: D18924811 fbshipit-source-id: a183a62e79a2af5aaf59cd08235458a172fe7dcb 11 December 2019, 04:02:31 UTC
a653857 Peter Dillinger 10 December 2019, 23:45:25 UTC Add PauseBackgroundWork() to db_stress (#6148) Summary: Worker thread will occasionally call PauseBackgroundWork(), briefly sleep (to avoid stalling itself) and then call ContinueBackgroundWork(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/6148 Test Plan: some running of 'make blackbox_crash_test' with temporary printf output to confirm code occasionally reached. Differential Revision: D18913886 Pulled By: pdillinger fbshipit-source-id: ae9356a803390929f3165dfb6a00194692ba92be 10 December 2019, 23:46:48 UTC
2bb5fc1 Adam Simpkins 10 December 2019, 23:19:24 UTC Add an option to the CMake build to disable building shared libraries (#6122) Summary: Add an option to explicitly disable building shared versions of the RocksDB libraries. The shared libraries cannot be built in cases where some dependencies are only available as static libraries. This allows still building RocksDB in these situations. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6122 Differential Revision: D18920740 fbshipit-source-id: d24f66d93c68a1e65635e6e0b663bae62c903bca 10 December 2019, 23:20:50 UTC
2b060c1 Yanqin Jin 10 December 2019, 19:47:20 UTC Use Env::GetChildren() instead of readdir (#6139) Summary: For more portability, switch from readdir to Env::GetChildren() in ldb's manifest_dump subcommand. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6139 Test Plan: ``` $make check ``` Manually check ldb command. Differential Revision: D18898197 Pulled By: riversand963 fbshipit-source-id: 92afca379e9fbe78ab70b2eb40d127daad8df5e2 10 December 2019, 19:49:09 UTC
14c38ba sdong 10 December 2019, 19:40:09 UTC db_stress: sometimes validate compact range data (#6140) Summary: Right now, in db_stress, compact range is simply executed without any immediate data validation. Add a simply validation which compares hash for all keys within the compact range to stay the same against the same snapshot before and after the compaction. Also, randomly tune most knobs of CompactRangeOptions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/6140 Test Plan: Run db_stress with "--compact_range_one_in=2000 --compact_range_width=100000000" for a while. Manually ingest some hacky code and observe the error path. Differential Revision: D18900230 fbshipit-source-id: d96e75bc8c38dd5ec702571ffe7cf5f4ea93ee10 10 December 2019, 19:41:50 UTC
back to top