Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
0113c61 Vijay Nadimpalli 21 October 2019, 19:07:58 UTC Making platform 007 (gcc 7) default in build_detect_platform.sh (#5947) Summary: Making platform 007 (gcc 7) default in build_detect_platform.sh. Pull Request resolved: https://github.com/facebook/rocksdb/pull/5947 Differential Revision: D18038837 Pulled By: vjnadimpalli fbshipit-source-id: 9ac2ddaa93bf328a416faec028970e039886378e 30 October 2019, 16:58:04 UTC
405017b Andrew Kryczka 28 January 2019, 18:57:57 UTC Add latest toolchain (gcc-8, etc.) build support for fbcode users (#4923) Summary: - When building with internal dependencies, specify this toolchain by setting `ROCKSDB_FBCODE_BUILD_WITH_PLATFORM007=1` - It is not enabled by default. However, it is enabled for TSAN builds in CI since there is a known problem with TSAN in gcc-5: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71090 - I did not add support for Lua since (1) we agreed to deprecate it, and (2) we only have an internal build for v5.3 with this toolchain while that has breaking changes compared to our current version (v5.2). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4923 Differential Revision: D13827226 Pulled By: ajkr fbshipit-source-id: 9aa3388ed3679777cfb15ef8cbcb83c07f62f947 30 October 2019, 16:57:53 UTC
cfdea78 Fosco Marotto 12 November 2018, 19:55:03 UTC Update history and version for 5.16.6 release. 12 November 2018, 19:55:03 UTC
fef38d9 Siying Dong 19 October 2018, 21:38:32 UTC Fix WriteBatchWithIndex's SeekForPrev() (#4559) Summary: WriteBatchWithIndex's SeekForPrev() has a bug that we internally place the position just before the seek key rather than after. This makes the iterator to miss the result that is the same as the seek key. Fix it by position the iterator equal or smaller. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4559 Differential Revision: D10468534 Pulled By: siying fbshipit-source-id: 2fb371ae809c561b60a1c11cef71e1c66fea1f19 19 October 2018, 22:44:58 UTC
4d46173 Andrew Kryczka 16 October 2018, 17:41:36 UTC update HISTORY.md and version number 16 October 2018, 17:41:36 UTC
1e7cca2 anand1976 16 October 2018, 06:20:15 UTC Properly determine a truncated CompactRange stop key (#4496) Summary: When a CompactRange() call for a level is truncated before the end key is reached, because it exceeds max_compaction_bytes, we need to properly set the compaction_end parameter to indicate the stop key. The next CompactRange will use that as the begin key. We set it to the smallest key of the next file in the level after expanding inputs to get a clean cut. Previously, we were setting it before expanding inputs. So we could end up recompacting some files. In a pathological case, where a single key has many entries spanning all the files in the level (possibly due to merge operands without a partial merge operator, thus resulting in compaction output identical to the input), this would result in an endless loop over the same set of files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4496 Differential Revision: D10395026 Pulled By: anand1976 fbshipit-source-id: f0c2f89fee29b4b3be53b6467b53abba8e9146a9 16 October 2018, 17:40:21 UTC
ab21dc6 Andrew Kryczka 15 October 2018, 23:18:55 UTC Avoid per-key linear scan over snapshots in compaction (#4495) Summary: `CompactionIterator::snapshots_` is ordered by ascending seqnum, just like `DBImpl`'s linked list of snapshots from which it was copied. This PR exploits this ordering to make `findEarliestVisibleSnapshot` do binary search rather than linear scan. This can make flush/compaction significantly faster when many snapshots exist since that function is called on every single key. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4495 Differential Revision: D10386470 Pulled By: ajkr fbshipit-source-id: 29734991631227b6b7b677e156ac567690118a8b 16 October 2018, 17:33:02 UTC
fa37825 Anand Ananthabhotla 10 October 2018, 21:10:08 UTC Update version to 5.16.4 Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: 10 October 2018, 21:10:08 UTC
4f6da54 Anand Ananthabhotla 10 October 2018, 05:50:59 UTC Handle mixed slowdown/no_slowdown writer properly (#4475) Summary: There is a bug when the write queue leader is blocked on a write delay/stop, and the queue has writers with WriteOptions::no_slowdown set to true. They are not woken up until the write stall is cleared. The fix introduces a dummy writer inserted at the tail to indicate a write stall and prevent further inserts into the queue, and a condition variable that writers who can tolerate slowdown wait on before adding themselves to the queue. The leader calls WriteThread::BeginWriteStall() to add the dummy writer and then walk the queue to fail any writers with no_slowdown set. Once the stall clears, the leader calls WriteThread::EndWriteStall() to remove the dummy writer and signal the condition variable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4475 Differential Revision: D10285827 Pulled By: anand1976 fbshipit-source-id: 747465e5e7f07a829b1fb0bc1afcd7b93f4ab1a9 10 October 2018, 19:59:34 UTC
98f11d0 Andrew Kryczka 01 October 2018, 08:16:16 UTC Fix CompactFiles support for kDisableCompressionOption (#4438) Summary: Previously `CompactFiles` with `CompressionType::kDisableCompressionOption` caused program to crash on assertion failure. This PR fixes the crash by adding support for that setting. Now, that setting will cause RocksDB to choose compression according to the column family's options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4438 Differential Revision: D10115761 Pulled By: ajkr fbshipit-source-id: a553c6fa76fa5b6f73b0d165d95640da6f454122 02 October 2018, 00:05:59 UTC
7b378c0 Yi Wu 21 September 2018, 16:13:34 UTC Bump version to 5.16.2 21 September 2018, 16:13:34 UTC
e90493a Yi Wu 20 September 2018, 23:50:07 UTC BlobDB: handle IO error on read (#4410) Summary: Fix IO error on read not being handle and crashing the DB. With the fix we properly return the error. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4410 Differential Revision: D9979246 Pulled By: yiwu-arbug fbshipit-source-id: 111a85675067a29c03cb60e9a34103f4ff636694 21 September 2018, 16:11:16 UTC
e36e75d Maysam Yabandeh 18 September 2018, 00:13:58 UTC Fix bug in partition filters with format_version=4 (#4381) Summary: Value delta encoding in format_version 4 requires the differences between the size of two consecutive handles to be sent to BlockBuilder::Add. This applies not only to indexes on blocks but also the indexes on indexes and filters in partitioned indexes and filters respectively. The patch fixes a bug where the partitioned filters would encode the entire size of the handle rather than the difference of the size with the last size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4381 Differential Revision: D9879505 Pulled By: maysamyabandeh fbshipit-source-id: 27a22e49b482b927fbd5629dc310c46d63d4b6d1 21 September 2018, 16:11:04 UTC
895a4ad Sagar Vemuri 17 September 2018, 19:20:30 UTC Bump version and update history 17 September 2018, 19:20:30 UTC
377d4d2 Sagar Vemuri 17 September 2018, 18:44:36 UTC Fix sync-point comment in Block destructor (#4380) Summary: This is a follow up to #4370. The earlier comment is not correct. Thanks to ajkr for pointing this out. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4380 Differential Revision: D9874667 Pulled By: sagar0 fbshipit-source-id: f4e092d86b29c715258210b770643d367e38caae 17 September 2018, 19:02:56 UTC
5271d07 Sagar Vemuri 15 September 2018, 07:05:08 UTC Remove sync point from Block destructor (#4370) Summary: AddressSanitizer: heap-use-after-free in std::__atomic_base<bool>::load(std::memory_order) const ==1798517==ABORTING ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4370 Differential Revision: D9844146 Pulled By: sagar0 fbshipit-source-id: 18a2970b1d504b4f6c8fb04857f26e0f32124dd1 17 September 2018, 17:48:21 UTC
2971348 Yanqin Jin 11 September 2018, 23:20:14 UTC Fix Makefile target 'jtest' on PowerPC (#4357) Summary: Before the fix: On a PowerPC machine, run the following ``` $ make jtest ``` The command will fail due to "undefined symbol: crc32c_ppc". It was caused by 'rocksdbjava' Makefile target not including crc32c_ppc object files when generating the shared lib. The fix is simple. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4357 Differential Revision: D9779474 Pulled By: riversand963 fbshipit-source-id: 3c5ec9068c2b9c796e6500f71cd900267064fd51 17 September 2018, 17:47:45 UTC
abcad80 Yanqin Jin 05 September 2018, 20:41:09 UTC Remove warnings caused by unused variables in jni (#4345) Summary: Test plan ``` $make clean jclean $make -j32 rocksdbjavastatic $make -j32 rocksdbjava ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4345 Differential Revision: D9661256 Pulled By: riversand963 fbshipit-source-id: aed316c53b29d02fbdd3fa1063a3e832b8a66469 17 September 2018, 17:47:14 UTC
b6a8f16 Anand Ananthabhotla 15 September 2018, 01:12:11 UTC Remove trace_analyzer_tool.cc from rocksdb_lib target Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: 15 September 2018, 01:12:11 UTC
eddc3eb Maysam Yabandeh 12 September 2018, 16:42:31 UTC Reduce IndexBlockIter size (#4358) Summary: With #3983 the size of IndexBlockIter was increased. This had resulted in a regression on P50 latencies in one of our benchmarks. The patch reduces IndexBlockIter size be eliminating active_comparator_ field from the class. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4358 Differential Revision: D9781737 Pulled By: maysamyabandeh fbshipit-source-id: 71e2b28d90ff0813db9e04b737ae73e185583c52 12 September 2018, 17:14:07 UTC
73fcd6b Anand Ananthabhotla 06 September 2018, 21:39:52 UTC Fix a lint error due to unspecified move evaluation order (#4348) Summary: In C++ 11, the order of argument and move evaluation in a statement such as below is unspecified - foo(a.b).bar(std::move(a)) The compiler is free to evaluate std::move(a) first, and then a.b is unspecified. In C++ 17, this will be safe if a draft proposal around function chaining rules is accepted. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4348 Differential Revision: D9688810 Pulled By: anand1976 fbshipit-source-id: e4651d0ca03dcf007e50371a0fc72c0d1e710fb4 06 September 2018, 23:41:46 UTC
7981e04 Philip Jameson 29 August 2018, 04:14:52 UTC Grab straggler files to explicitly import AutoHeaders Summary: There were a few files that were missed when AutoHeaders were moved to their own file. Add explicit loads Reviewed By: yfeldblum Differential Revision: D9499942 fbshipit-source-id: 942bf3a683b8961e1b6244136f6337477dcc45af 04 September 2018, 20:09:04 UTC
5279417 Mikhail Antonov 29 August 2018, 18:58:13 UTC Avoiding write stall caused by manual flushes (#4297) Summary: Basically at the moment it seems it's possible to cause write stall by calling flush (either manually vis DB::Flush(), or from Backup Engine directly calling FlushMemTable() while background flush may be already happening. One of the ways to fix it is that in DBImpl::CompactRange() we already check for possible stall and delay flush if needed before we actually proceed to call FlushMemTable(). We can simply move this delay logic to separate method and call it from FlushMemTable. This is draft patch, for first look; need to check tests/update SyncPoints and most certainly would need to add allow_write_stall method to FlushOptions(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4297 Differential Revision: D9420705 Pulled By: mikhail-antonov fbshipit-source-id: f81d206b55e1d7b39e4dc64242fdfbceeea03fcc 31 August 2018, 21:18:59 UTC
8c7b501 Andrew Kryczka 31 August 2018, 19:18:11 UTC Reduce empty SST creation/deletion in compaction (#4336) Summary: This is a followup to #4311. Checking `!RangeDelAggregator::IsEmpty()` before opening a dedicated range tombstone SST did not properly prevent empty SSTs from being generated. That's because it relies on `CollapsedRangeDelMap::Size`, which had an underflow bug when the map was empty. This PR fixes that underflow bug. Also fixed an uninitialized variable in db_stress. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4336 Differential Revision: D9600080 Pulled By: ajkr fbshipit-source-id: bc6980ca79d2cd01b825ebc9dbccd51c1a70cfc7 31 August 2018, 19:35:59 UTC
3c1781d Fenggang Wu 23 August 2018, 17:12:15 UTC DataBlockHashIndex: avoiding expensive iiter->Next when handling hash kNoEntry (#4296) Summary: When returning `kNoEntry` from HashIndex lookup, previously we invalidate the `biter` by set `current_=restarts_`, so that the search can continue to the next block in case the search result may reside in the next block. There is one problem: when we are searching for a missing key, if the search finds a `kNoEntry` and continue the search to the next block, there is also a non-trivial possibility that the HashIndex return `kNoEntry` too, and the expensive index iterator `Next()` will happen several times for nothing. The solution is that if the hash table returns `kNoEntry`, `SeekForGetImpl()` just search the last restart interval for the key. It will stop at the first key that is large than the seek_key, or to the end of the block, and each case will be handled correctly. Microbenchmark script: ``` TEST_TMPDIR=/dev/shm ./db_bench --benchmarks=fillseq,readtocache,readmissing \ --cache_size=20000000000 --use_data_block_hash_index={true|false} ``` `readmissing` performance (lower is better): ``` binary: 3.6098 micros/op hash (before applying diff): 4.1048 micros/op hash (after applying diff): 3.3502 micros/op ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4296 Differential Revision: D9419159 Pulled By: fgwu fbshipit-source-id: 21e3eedcccbc47a249aa8eb4bf405c9def0b8a05 30 August 2018, 22:45:26 UTC
9d5eb40 Fenggang Wu 22 August 2018, 00:06:44 UTC DataBlockHashIndex: fix comment in NumRestarts() (#4286) Summary: Improve the description of the backward compatibility check in NumRestarts() Pull Request resolved: https://github.com/facebook/rocksdb/pull/4286 Differential Revision: D9412490 Pulled By: fgwu fbshipit-source-id: ea7dd5c61d8ff8eacef623b729d4e4fd53cca066 30 August 2018, 22:34:53 UTC
0e35aa9 Sagar Vemuri 27 August 2018, 16:54:32 UTC Download bzip2 packages from Internet Archive (#4306) Summary: Since bzip.org is no longer maintained, download the bzip2 packages from a snapshot taken by the internet archive until we figure out a more credible source. Fixes issue: #4305 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4306 Differential Revision: D9514868 Pulled By: sagar0 fbshipit-source-id: 57c6a141a62e652f94377efc7ca9916b458e68d5 30 August 2018, 22:33:45 UTC
023e9bf Yi Wu 27 August 2018, 17:56:28 UTC BlobDB: Implement DisableFileDeletions (#4314) Summary: `DB::DiableFileDeletions` and `DB::EnableFileDeletions` are used for applications to stop RocksDB background jobs to delete files while they are doing replication. Implement these methods for BlobDB. `DeleteObsolteFiles` now needs to check `disable_file_deletions_` before starting, and will hold `delete_file_mutex_` the whole time while it is running. `DisableFileDeletions` needs to wait on `delete_file_mutex_` for running `DeleteObsolteFiles` job and set `disable_file_deletions_` flag. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4314 Differential Revision: D9501373 Pulled By: yiwu-arbug fbshipit-source-id: 81064c1228f1724eff46da22b50ff765b16292cd 27 August 2018, 18:22:31 UTC
096e5f5 Andrew Kryczka 24 August 2018, 19:21:05 UTC Reduce empty SST creation/deletion during compaction (#4311) Summary: I have a PR to start calling `OnTableFileCreated` for empty SSTs: #4307. However, it is a behavior change so should not go into a patch release. This PR adds back a check to make sure range deletions at least exist before starting file creation. This PR should be safe to backport to earlier versions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4311 Differential Revision: D9493734 Pulled By: ajkr fbshipit-source-id: f0d43cda4cfd904f133cfe3a6eb622f52a9ccbe8 24 August 2018, 21:09:39 UTC
c9a0419 Anand Ananthabhotla 21 August 2018, 21:32:19 UTC Release 5.16 (#4298) Summary: Update HISTORY.md for 5.16. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4298 Differential Revision: D9433868 Pulled By: anand1976 fbshipit-source-id: e7880a1c952210b1e9d7466eed72a6cb5018096b 21 August 2018, 21:43:08 UTC
9e2d5ab Zhichao Cao 21 August 2018, 17:39:41 UTC Adjusted the Makefile of trace_analyzer to isolate the Gflags from other (#4290) Summary: Previously, the trace_analyzer_tool will be complied with other libobjects, which let the GFLAGS of trace_analyzer appear in other tools (e.g., db_bench, rocksdb_dump, and etc.). When using '--help', the help information of trace_analyzer will appear in other tool help information, which will cause confusion issues. Currently, trace_analyzer_tool is built and used only by trace_analyzer and trace_analyzer_test to avoid the issues. Tested with make asan_check. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4290 Differential Revision: D9413163 Pulled By: zhichao-cao fbshipit-source-id: ed5d20c4575a53ca15ff62a2ffe601d5cf278cc4 21 August 2018, 17:47:24 UTC
6d37fdb Fenggang Wu 21 August 2018, 06:04:08 UTC DataBlockHashIndex: Remove the division from EstimateSize() (#4293) Summary: `BlockBasedTableBuilder::Add()` eventually calls `DataBlockHashIndexBuilder::EstimateSize()`. The previous implementation divides the `num_keys` by the `util_ratio_` to get an estimizted `num_buckets`. Such division is expensive as it happens in every `BlockBasedTableBuilder::Add()`. This diff estimates the `num_buckets` by double addition instead of double division. Specifically, in each `Add()`, we add `bucket_per_key_` (inverse of `util_ratio_`) to the current `estimiated_num_buckets_`. The cost is that we are gonna have the `estimated_num_buckets_` stored as one extra field in the DataBlockHashIndexBuilder. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4293 Differential Revision: D9412472 Pulled By: fgwu fbshipit-source-id: 2925c8509a401e7bd3c1ab1d9e9c7244755c277a 21 August 2018, 06:13:50 UTC
7188bd3 Yi Wu 21 August 2018, 05:38:34 UTC BlobDB: Fix expired file not being evicted (#4294) Summary: Fix expired file not being evicted from the DB. We have a background task (previously called `CheckSeqFiles` and I rename it to `EvictExpiredFiles`) to scan and remove expired files, but it only close the files, not marking them as expired. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4294 Differential Revision: D9415984 Pulled By: yiwu-arbug fbshipit-source-id: eff7bf0331c52a7ccdb02318602bff7f64f3ef3d 21 August 2018, 05:42:33 UTC
d5612b4 Siying Dong 21 August 2018, 00:31:28 UTC Two code changes to make "clang analyze" happy (#4292) Summary: Clang analyze is not happy in two pieces of code, with "Potential memory leak". No idea what the problem but slightly changing the code makes clang happy. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4292 Differential Revision: D9413555 Pulled By: siying fbshipit-source-id: 9428c9d3664530c72129feefd135ee63d8386137 21 August 2018, 00:43:41 UTC
dc064f3 Siying Dong 20 August 2018, 23:46:45 UTC Suppress two CLANG Analyze warning (#4291) Summary: Suppress two CLANG analyze warnings. They don't seem to be real bugs Pull Request resolved: https://github.com/facebook/rocksdb/pull/4291 Differential Revision: D9407333 Pulled By: siying fbshipit-source-id: 2ed63d88fa0b217fdccb1572d7508467c2203dc8 20 August 2018, 23:57:38 UTC
d116a17 Yanqin Jin 20 August 2018, 21:54:03 UTC Update recovery code for version edits group commit. (#3945) Summary: During recovery, RocksDB is able to handle version edits that belong to group commits. This PR is a subset of [PR 3752](https://github.com/facebook/rocksdb/pull/3752) Pull Request resolved: https://github.com/facebook/rocksdb/pull/3945 Differential Revision: D8529122 Pulled By: riversand963 fbshipit-source-id: 57cb0f9cc55ecca684a837742d6626dc9c07f37e 20 August 2018, 21:58:00 UTC
90f7449 jsteemann 17 August 2018, 18:53:33 UTC adds missing PopSavePoint method to Transaction (#4256) Summary: Transaction has had methods to deal with SavePoints already, but was missing the PopSavePoint method provided by WriteBatch and WriteBatchWithIndex. This PR adds PopSavePoint to Transaction as well. Having the method on Transaction-level too is useful for applications that repeatedly execute a sequence of operations that normally succeed, but infrequently need to get rolled back. Using SavePoints here is sensible, but as operations normally succeed the application may pile up a lot of useless SavePoints inside a Transaction, leading to slightly increased memory usage for managing the unneeded SavePoints. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4256 Differential Revision: D9326932 Pulled By: yiwu-arbug fbshipit-source-id: 53a0af18a6c7e87feff8a56f1f3eab9df7f371d6 17 August 2018, 18:57:30 UTC
c7cf981 Christian Esken 17 August 2018, 17:52:58 UTC Add CompactRangeOptions for Java (#4220) Summary: Closes https://github.com/facebook/rocksdb/issues/4195 CompactRangeOptions are available the CPP API, but not in the Java API. This PR adds CompactRangeOptions to the Java API and adds an overloaded compactRange() method. See https://github.com/facebook/rocksdb/issues/4195 for the original discussion. This change supports all fields of CompactRangeOptions, including the required enum converters in the JNI portal. Significant changes: - Make CompactRangeOptions available in the compactRange() for Java. - Deprecate other compactRange() methods that have individual option params, like in the CPP code. - Migrate rocksdb_compactrange_helper() to CompactRangeOptions. - Add Java unit tests for CompactRangeOptions. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4220 Differential Revision: D9380007 Pulled By: sagar0 fbshipit-source-id: 6af6c334f221427f1997b33fb24c3986b092fed6 17 August 2018, 17:57:25 UTC
fa4de6e Andrey Zagrebin 17 August 2018, 17:51:30 UTC #3865 followup for fix performance degression introduced by switching order of operands (#4284) Summary: Followup for #4266. There is one more place in **get_context.cc** where **MergeOperator::ShouldMerge** should be called with reversed list of operands. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4284 Differential Revision: D9380008 Pulled By: sagar0 fbshipit-source-id: 70ec26e607e5b88465e1acbdcd6c6171bd76b9f2 17 August 2018, 17:57:25 UTC
9d646a6 Fenggang Wu 17 August 2018, 01:29:13 UTC Add db_bench options of data block hash index (#4281) Summary: Add `--data_block_index_type` and `--data_block_hash_table_util_ratio` option to `db_bench`. `--data_block_index_type` can be either of `binary` (default) or `binary_and_hash`; `--data_block_hash_table_util_ratio` will be a double. The default value is `0.75`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4281 Differential Revision: D9361476 Pulled By: fgwu fbshipit-source-id: dc53e01acef9db81b9eec5e8a96f3bc8ed718c10 17 August 2018, 01:42:46 UTC
889a055 Mikhail Antonov 16 August 2018, 23:39:24 UTC VerifyChecksum() API should preserve options Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4275 Reviewed By: yiwu-arbug Differential Revision: D9369766 Pulled By: mikhail-antonov fbshipit-source-id: d91b64c34cc1976b324a260767fce343fa32afde 16 August 2018, 23:42:29 UTC
9c0c8f5 Siying Dong 16 August 2018, 22:48:55 UTC GetAllKeyVersions() to take an extra argument of `max_num_ikeys`. (#4271) Summary: Right now, `ldb idump` may have memory out of control if there is a big range of tombstones. Add an option to cut maxinum number of keys in GetAllKeyVersions(), and push down --max_num_ikeys from ldb. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4271 Differential Revision: D9369149 Pulled By: siying fbshipit-source-id: 7cbb797b7d2fa16573495a7e84937456d3ff25bf 16 August 2018, 22:57:08 UTC
aeed4f0 Andrey Zagrebin 16 August 2018, 17:44:09 UTC #3865 fix performance regression introduced by MergeOperator.ShouldMerge (#4266) Summary: This PR addresses issue #3865 and implements the following approach to fix it: - adds `MergeContext::GetOperandsDirectionForward` and `MergeContext::GetOperandsDirectionBackward` to query merge operands in a specific order - `MergeContext::GetOperands` becomes a shortcut for `MergeContext::GetOperandsDirectionForward` - pass `MergeContext::GetOperandsDirectionBackward` to `MergeOperator::ShouldMerge` and document the order Pull Request resolved: https://github.com/facebook/rocksdb/pull/4266 Differential Revision: D9360750 Pulled By: sagar0 fbshipit-source-id: 20cb73ff017760b062ecdcf4382560767086e092 16 August 2018, 17:58:05 UTC
19ec44f Fenggang Wu 15 August 2018, 21:27:47 UTC Improve point-lookup performance using a data block hash index (#4174) Summary: Add hash index support to data blocks, which helps to reduce the CPU utilization of point-lookup operations. This feature is backward compatible with the data block created without the hash index. It is disabled by default unless `BlockBasedTableOptions::data_block_index_type` is set to `data_block_index_type = kDataBlockBinaryAndHash.` The DB size would be bigger with the hash index option as a hash table is added at the end of each data block. If the hash utilization ratio is 1:1, the space overhead is one byte per key. The hash table utilization ratio is adjustable using `BlockBasedTableOptions::data_block_hash_table_util_ratio`. A lower utilization ratio will improve more on the point-lookup efficiency, but take more space too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4174 Differential Revision: D8965914 Pulled By: fgwu fbshipit-source-id: 1c6bae5d1fc39c80282d8890a72e9e67bc247198 15 August 2018, 21:30:03 UTC
8ae2bf5 Zhichao Cao 15 August 2018, 01:19:25 UTC Fix the build and test bugs in the Trace_analyzer (#4274) Summary: The wrong options are used in the trace_analyzer_test, removed. The potential loses integer precision are fixed. Pass the specified testing case, make asan_check Pull Request resolved: https://github.com/facebook/rocksdb/pull/4274 Reviewed By: yiwu-arbug Differential Revision: D9327811 Pulled By: zhichao-cao fbshipit-source-id: d62cb18d6586503a490cd323bfc1c672b68b346e 15 August 2018, 01:27:48 UTC
33ad906 jsteemann 14 August 2018, 22:03:57 UTC fix compilation with g++ option `-Wsuggest-override` (#4272) Summary: Fixes compilation warnings (which are turned into compilation errors by default) when compiling with g++ option `-Wsuggest-override`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4272 Differential Revision: D9322556 Pulled By: siying fbshipit-source-id: abd57a29ec8f544bee77c0bb438f31be830b7244 14 August 2018, 22:13:10 UTC
bf07e90 Anand Ananthabhotla 14 August 2018, 21:56:21 UTC Fix db_stress assertion failures on 0 byte SSTs (#4273) Summary: In the OnTableFileCreation() listener, assert on various TableProperties only when file size > 0 bytes. The listener can get called even for 0 byte SSTs which have been deleted. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4273 Differential Revision: D9322738 Pulled By: anand1976 fbshipit-source-id: 17cdfb3d0da946b9a158d7328e5db1c87973956b 14 August 2018, 21:58:26 UTC
d122025 Maysam Yabandeh 14 August 2018, 21:07:12 UTC Extend stress test to format_version 4 (#4265) Summary: Stress tests currently cover format_version 2 and 3. The patch adds 4 as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4265 Differential Revision: D9323185 Pulled By: maysamyabandeh fbshipit-source-id: 54d11e41ecae09bae14cadd7313f07c9a3db5a57 14 August 2018, 21:13:33 UTC
d916a11 Huachao Huang 14 August 2018, 01:34:04 UTC c-api: add some missing options Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4267 Differential Revision: D9309505 Pulled By: anand1976 fbshipit-source-id: eb9fee8037f4ff24dc1cdd5cc5ef41c231a03e1f 14 August 2018, 01:42:30 UTC
f3d91a0 Siying Dong 14 August 2018, 00:31:58 UTC Add a unit test to verify iterators release data blocks after using them (#4170) Summary: Add a unit test to check that iterators release data blocks after it has moved away from it. Verify the same for compaction input iterators. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4170 Differential Revision: D8962513 Pulled By: siying fbshipit-source-id: 05a5b604d7d29887fb488f2cda7286f554a14407 14 August 2018, 00:43:14 UTC
999d955 Zhichao Cao 13 August 2018, 18:32:04 UTC RocksDB Trace Analyzer (#4091) Summary: A framework of trace analyzing for RocksDB After collecting the trace by using the tool of [PR #3837](https://github.com/facebook/rocksdb/pull/3837). User can use the Trace Analyzer to interpret, analyze, and characterize the collected workload. **Input:** 1. trace file 2. Whole keys space file **Statistics:** 1. Access count of each operation (Get, Put, Delete, SingleDelete, DeleteRange, Merge) in each column family. 2. Key hotness (access count) of each one 3. Key space separation based on given prefix 4. Key size distribution 5. Value size distribution if appliable 6. Top K accessed keys 7. QPS statistics including the average QPS and peak QPS 8. Top K accessed prefix 9. The query correlation analyzing, output the number of X after Y and the corresponding average time intervals **Output:** 1. key access heat map (either in the accessed key space or whole key space) 2. trace sequence file (interpret the raw trace file to line base text file for future use) 3. Time serial (The key space ID and its access time) 4. Key access count distritbution 5. Key size distribution 6. Value size distribution (in each intervals) 7. whole key space separation by the prefix 8. Accessed key space separation by the prefix 9. QPS of each operation and each column family 10. Top K QPS and their accessed prefix range **Test:** 1. Added the unit test of analyzing Get, Put, Delete, SingleDelete, DeleteRange, Merge 2. Generated the trace and analyze the trace **Implemented but not tested (due to the limitation of trace_replay):** 1. Analyzing Iterator, supporting Seek() and SeekForPrev() analyzing 2. Analyzing the number of Key found by Get **Future Work:** 1. Support execution time analyzing of each requests 2. Support cache hit situation and block read situation of Get Pull Request resolved: https://github.com/facebook/rocksdb/pull/4091 Differential Revision: D9256157 Pulled By: zhichao-cao fbshipit-source-id: f0ceacb7eedbc43a3eee6e85b76087d7832a8fe6 13 August 2018, 18:44:02 UTC
1b1d264 Yanqin Jin 13 August 2018, 18:06:44 UTC Remove an assersion about file size (#4268) Summary: Due to 4ea56b1bd00b5f8751e5e9733d01ceb33ebd09e3, we should also remove the assersion in stress test. This removal can be temporary, and we can add it back once we figure out the reason for the 0-byte SSTs. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4268 Differential Revision: D9297186 Pulled By: riversand963 fbshipit-source-id: cebba9a68f42e815f8cf24471176d2cfdf962f63 13 August 2018, 18:12:50 UTC
4ea56b1 Anand Ananthabhotla 11 August 2018, 23:44:11 UTC Revert changes in PR #4003 (#4263) Summary: Revert this change. Not generating the OnTableFileCreated() notification for a 0 byte SST on flush breaks the assumption that every OnTableFileCreationStarted() notification is followed by a corresponding OnTableFileCreated(). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4263 Differential Revision: D9285623 Pulled By: anand1976 fbshipit-source-id: 808c3dcd498b4b4f4ed4be947a29a24b2296aa8d 11 August 2018, 23:57:36 UTC
6d75319 Zhichao Cao 11 August 2018, 00:56:11 UTC Add tracing function of Seek() and SeekForPrev() to trace_replay (#4228) Summary: In the current trace_and replay, Get an WriteBatch are traced. This pull request track down the Seek() and SeekForPrev() to the trace file. <target_key, timestamp, column_family_id> are write to the file. Replay of Iterator is not supported in the current implementation. Tested with trace_analyzer. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4228 Differential Revision: D9201381 Pulled By: zhichao-cao fbshipit-source-id: 6f9cc9cb3c20260af741bee065ec35c5c96354ab 11 August 2018, 00:57:40 UTC
76d7720 Zhichao Cao 11 August 2018, 00:23:25 UTC Remove the redundant condition inclusion to avoid confusion (#4254) Summary: The pair of ROCKSDB_LITE condition inclusion is redundant, it is already inside the #ifndef ROCKSDB_LITE. Remove them to void confusion. Tested by make asan_check. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4254 Differential Revision: D9281652 Pulled By: zhichao-cao fbshipit-source-id: 06bf7641ede71391f21f6a3fe37fbd13f0e2a43a 11 August 2018, 00:43:33 UTC
d511f35 Maysam Yabandeh 10 August 2018, 22:14:44 UTC Fix wrong partitioned index size recorded in properties block (#4259) Summary: After refactoring in https://github.com/facebook/rocksdb/pull/4158 the properties block is written after the index block. This breaks the existing logic in estimating the index size in partitioned indexes. The patch fixes that by using the accurate index block size, which is available since by the time we write the properties block, the index block is already written. The patch also fixes an issue in estimating the partition size with format_version=3 which was resulting into partitions smaller than the configured metadata_block_size. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4259 Differential Revision: D9274454 Pulled By: maysamyabandeh fbshipit-source-id: c82d045505cca3e7ed1a44ee1eaa26e4f25a4272 10 August 2018, 22:27:20 UTC
058026a Maysam Yabandeh 10 August 2018, 17:25:28 UTC Fix unity compile error (#4257) Summary: Fix the compile error in "make unity_test" caused by #3983. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4257 Differential Revision: D9271740 Pulled By: maysamyabandeh fbshipit-source-id: 94e56d1675bf8bdc0e94439467eb4f40dd107517 10 August 2018, 17:27:55 UTC
b271f95 Yanqin Jin 10 August 2018, 02:37:47 UTC Fix a TSAN failure (#4250) Summary: TSAN fails due to comparison between signed int and unsigned long. Fix it by static_casting. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4250 Differential Revision: D9256535 Pulled By: riversand963 fbshipit-source-id: c6bad23ff70c6d0ec58e2e85c401ce0ad45de609 10 August 2018, 02:42:32 UTC
caf0f53 Maysam Yabandeh 09 August 2018, 23:49:45 UTC Index value delta encoding (#3983) Summary: Given that index value is a BlockHandle, which is basically an <offset, size> pair we can apply delta encoding on the values. The first value at each index restart interval encoded the full BlockHandle but the rest encode only the size. Refer to IndexBlockIter::DecodeCurrentValue for the detail of the encoding. This reduces the index size which helps using the block cache more efficiently. The feature is enabled with using format_version 4. The feature comes with a bit of cpu overhead which should be paid back by the higher cache hits due to smaller index block size. Results with sysbench read-only using 4k blocks and using 16 index restart interval: Format 2: 19585 rocksdb read-only range=100 Format 3: 19569 rocksdb read-only range=100 Format 4: 19352 rocksdb read-only range=100 Pull Request resolved: https://github.com/facebook/rocksdb/pull/3983 Differential Revision: D8361343 Pulled By: maysamyabandeh fbshipit-source-id: f882ee082322acac32b0072e2bdbb0b5f854e651 09 August 2018, 23:58:40 UTC
63f265c Andrew Kryczka 09 August 2018, 23:45:11 UTC Update FB internal dependencies (#4244) Summary: I noticed we were building against zstd 1.3.0 which is missing optimizations that our fbcode customers have (they're on zstd 1.3.5). Ran `./build_tools/update_dependencies.sh` to catch us up. Omitted the changes it made for gcc-4.8 since it's broken. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4244 Differential Revision: D9230111 Pulled By: ajkr fbshipit-source-id: 3e8ec1d8a961f98ec77c8c6580bde4caacf2d437 09 August 2018, 23:58:40 UTC
72e6949 Anand Ananthabhotla 09 August 2018, 21:41:24 UTC Fix error parsing in build_tools/error_filter.py (#4247) Summary: The error_filter.py script parses the output of the "Build and run" stage of continuous tests to check for errors. It is currently only detecting compile errors and not link errors. This change fixes that. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4247 Differential Revision: D9233735 Pulled By: anand1976 fbshipit-source-id: 16e5a04950891cd9aba5cb3efcb6abc2a2e0d5ae 09 August 2018, 21:43:34 UTC
1b813a9 Georgios Bitzes 09 August 2018, 21:30:38 UTC Make rocksdb::Slice more interoperable with std::string_view (#4242) Summary: This change allows using std::string_view objects directly in the DB API: db->Get(some_string_view_object, ...); The conversion from std::string_view to rocksdb::Slice is done automatically, thanks to the added constructor. I'm stopping short of adding an implicit conversion operator from rocksdb::Slice to std::string_view, as I don't think that's a good idea for PinnableSlices. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4242 Differential Revision: D9224134 Pulled By: anand1976 fbshipit-source-id: f50aad04dd0b01737907c0fb88d495c83a81f4e4 09 August 2018, 21:43:34 UTC
ab22cf3 Dmitri Smirnov 09 August 2018, 21:21:35 UTC Implement Env::NumFileLinks (#4221) Summary: Although delete scheduler implementation allows for the interface not to be supported, the delete_scheduler_test does not allow for that. Address compiler warnings Make sst_dump_test use test directory structure as the current execution directory may not be writiable. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4221 Differential Revision: D9210152 Pulled By: siying fbshipit-source-id: 381a74511e969ecb8089d5c4b4df87dc30c8df63 09 August 2018, 21:29:11 UTC
de7f423 Yanqin Jin 09 August 2018, 21:18:59 UTC Add SST ingestion to ldb (#4205) Summary: We add two subcommands `write_extern_sst` and `ingest_extern_sst` to ldb. This PR avoids changing existing code because we hope to cherry-pick to earlier releases to support compatibility check for external SST file ingestion. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4205 Differential Revision: D9112711 Pulled By: riversand963 fbshipit-source-id: 7cae88380d4de86da8440230e87eca66755648e4 09 August 2018, 21:29:11 UTC
9e8ef45 Andrew Kryczka 09 August 2018, 19:34:44 UTC Link to FB internal build of ZSTD with -fPIC (#4249) Summary: TSAN requires the code is built with -fPIC. This PR links against a libzstd built with -fPIC when necessary, which enables ZSTD compression to be used in TSAN builds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4249 Differential Revision: D9244746 Pulled By: ajkr fbshipit-source-id: 8c6a8fadd6c8643b2077afcbc3626779e1d73b63 09 August 2018, 19:42:43 UTC
b15379d Zhongyi Xie 09 August 2018, 17:58:34 UTC fix use-after-free error involving a temporary string (#4240) Summary: In the current code, `error_msg` is pointing to the inner buffer of a temporary std::string object. When `error_msg` is used to construct the error message, that array is already released. This PR will fix this bug by copying the string to a local variable. Fixes https://github.com/facebook/rocksdb/issues/4239 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4240 Differential Revision: D9204334 Pulled By: miasantreble fbshipit-source-id: 0ac599e166ae0a4ec413e32d8b8853d7c5fba878 09 August 2018, 18:13:10 UTC
7a9a164 Andrew Kryczka 09 August 2018, 17:17:56 UTC Fix db_bench default compression level (#4248) Summary: db_bench's previous default compression level (-1) was not the default compression level in all libraries. In particular, in ZSTD negative values are valid compression levels, while ZSTD's default compression level is three. This PR changes db_bench's default to be RocksDB's library-independent default compression level (see #3895). I also changed a couple other flags to get their default values from an options object directly rather than hardcoding. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4248 Differential Revision: D9235140 Pulled By: ajkr fbshipit-source-id: be4e0722d59fa1968832183db36d1d20fcf11e5b 09 August 2018, 17:28:14 UTC
eb8885a Maysam Yabandeh 09 August 2018, 00:40:05 UTC Return correct usable_size for BlockContents (#4246) Summary: If jemalloc is disabled or the API is incorrectly referenced (jemalloc api on windows have a prefix je_) memory usage is incorrectly reported for all block sizes. This is because sizeof(char) is always 1. sizeof() is calculated at compile time and *(char*) is char. The patch uses the size of the slice to fix that. Fixes #4245 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4246 Differential Revision: D9233958 Pulled By: maysamyabandeh fbshipit-source-id: 9646933b24504e2814c7379f06a31148829c6b4e 09 August 2018, 00:43:00 UTC
853a5cd Anand Ananthabhotla 08 August 2018, 22:50:22 UTC Fix sandcastle rocksdb-contrun-tsan_crash job json (#4243) Summary: Fix the nested quotes for CRASH_TEST_EXT_ARGS, as the generated json could not be parsed by the sandcastle job. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4243 Differential Revision: D9228429 Pulled By: anand1976 fbshipit-source-id: 3c2bcac34870e377949d8a79c55e33b8363b25dd 08 August 2018, 22:57:11 UTC
bbf3033 Jean-Marc Le Roux 08 August 2018, 15:02:43 UTC Fix the build failure with OS_ANDROID (#4232) Summary: sysmacros.h should be included in OS_ANDROID build as well otherwise the compile would complain: error: use of undeclared identifier 'major'. Fixes https://github.com/facebook/rocksdb/issues/4231 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4232 Differential Revision: D9217350 Pulled By: maysamyabandeh fbshipit-source-id: 21f4b62dbbda3163120ac0b38b95d95d35d67dce 08 August 2018, 15:12:02 UTC
d8d66c9 Maysam Yabandeh 08 August 2018, 14:23:10 UTC Simplify DBWithMaxSpaceAllowedRandomized (#4235) Summary: The test has become complicated over the years and hard to reason about the corner cases that makes the test flaky. The patch simplifies the test and also fixes some probable synchronization issues. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4235 Differential Revision: D9187995 Pulled By: maysamyabandeh fbshipit-source-id: 53c7b060f14367e5a9e361014578c26debfe3d27 08 August 2018, 14:27:46 UTC
badfd70 Huachao Huang 07 August 2018, 01:12:25 UTC types: add kEntryBlobIndex for TablePropertiesCollector (#4233) Summary: So that we can act accordingly on blob index entries Pull Request resolved: https://github.com/facebook/rocksdb/pull/4233 Differential Revision: D9190205 Pulled By: yiwu-arbug fbshipit-source-id: e5b84d5b41e44fa7a76762f1f7b0305369bb3a0c 07 August 2018, 01:27:44 UTC
c970358 Yi Wu 07 August 2018, 00:28:40 UTC BlobDB: Can return expiration together with Get() (#4227) Summary: Add API to allow fetching expiration of a key with `Get()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4227 Differential Revision: D9169897 Pulled By: yiwu-arbug fbshipit-source-id: 2a6f216c493dc75731ddcef1daa689b517fab31b 07 August 2018, 00:43:14 UTC
4cb7068 Yi Wu 06 August 2018, 23:45:02 UTC BlobDB: Fix VisibleToActiveSnapshot() (#4236) Summary: There are two issues with `VisibleToActiveSnapshot`: 1. If there are no snapshots, `oldest_snapshot` will be 0 and `VisibleToActiveSnapshot` will always return true. Since the method is used to decide whether it is safe to delete obsolete files, obsolete file won't be able to delete in this case. 2. The `auto` keyword of `auto snapshots = db_impl_->snapshots()` translate to a copy of `const SnapshotList` instead of a reference. Since copy constructor of `SnapshotList` is not defined, using the copy may yield unexpected result. Issue 2 actually hide issue 1 from being catch by tests. During test `snapshots.empty()` can return false while it should actually be empty, and `snapshots.oldest()` return an invalid address, making `oldest_snapshot` being some random large number. The issue was originally reported by BlobDB early adopter at Kuaishou. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4236 Differential Revision: D9188706 Pulled By: yiwu-arbug fbshipit-source-id: a0f2624b927cf9bf28c1bb534784fee5d106f5ea 06 August 2018, 23:57:42 UTC
6175b4b Andrew Kryczka 06 August 2018, 22:23:03 UTC Support dictionary compression in stress/crash tests (#4234) Summary: - Add `--compression_max_dict_bytes` and `--compression_zstd_max_train_bytes` flags to stress test - Randomly enable/disable the above flags in crash test - Set `--compression_type=zstd` in FB-specific crash test runs Pull Request resolved: https://github.com/facebook/rocksdb/pull/4234 Differential Revision: D9187207 Pulled By: ajkr fbshipit-source-id: 8d78cf8d8e1165f2cd1c32e069b73726b5bc1fd2 06 August 2018, 22:27:29 UTC
140f256 Yi Wu 06 August 2018, 18:43:51 UTC BlobDB: Cleanup TTLExtractor interface (#4229) Summary: Cleanup TTLExtractor interface. The original purpose of it is to allow our users keep using existing `Write()` interface but allow it to accept TTL via `TTLExtractor`. However the interface is confusing. Will replace it with something like `WriteWithTTL(batch, ttl)` in the future. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4229 Differential Revision: D9174390 Pulled By: yiwu-arbug fbshipit-source-id: 68201703d784408b851336ab4dd9b84188245b2d 06 August 2018, 18:58:05 UTC
ceb5fea Jingguo Yao 06 August 2018, 17:57:22 UTC Improve FullFilterBitsReader::HashMayMatch's doc (#4202) Summary: HashMayMatch is related to AddKey() instead of CreateFilter(). Also applies some minor Fixes #4191 #4200 #3910 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4202 Differential Revision: D9180945 Pulled By: maysamyabandeh fbshipit-source-id: 6f07b81c5bb9bda5c0273475b486ba8a030471e6 06 August 2018, 18:13:18 UTC
a15354d Gustav Davidsson 05 August 2018, 00:50:01 UTC Expose GetTotalTrashSize in SstFileManager interface (#4206) Summary: Hi, it would be great if we could expose this API, so that LogDevice can use it to track the total size of trash files and alarm if it grows too large in relation to disk size. There's probably other customers that would be interested in this as well. :) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4206 Differential Revision: D9115516 Pulled By: gdavidsson fbshipit-source-id: f34993a940e39cb0a0b544ae8298546499b7e047 05 August 2018, 00:57:48 UTC
1f80277 Yanqin Jin 04 August 2018, 00:34:07 UTC Update JobContext. (#3949) Summary: In the past, we assume that a job modifies a single column family. Therefore, a job can create at most one superversion since each superversion corresponds to one column family. This assumption leads to the fact that a `JobContext` has only one member variable called `superversion_context`. Now we want to support group flush of column families, indicating that each job can create multiple superversions. Therefore, we need to make the following change to accommodate this new feature. Add a vector of `SuperVersionContext` to `JobContext` to support installing superversions for multiple column families in one job context. This PR is a subset of [PR 3752](https://github.com/facebook/rocksdb/pull/3752). Pull Request resolved: https://github.com/facebook/rocksdb/pull/3949 Differential Revision: D8864895 Pulled By: riversand963 fbshipit-source-id: 5937a48817276370d3c8172db9c8aafc826d97ca 04 August 2018, 00:42:34 UTC
2236896 Yanqin Jin 03 August 2018, 20:55:32 UTC Modify verification logic of ObsoleteOptionsFileTest (#4218) Summary: The current verification logic does not consider the case in which multiple threads (foreground and background) may execute `PurgeObsoleteFiles` function simultaneously. Each invocation will trigger the callback adding elements to a vector. Then we verify the elements in the vector, which can fail sometimes. The solution is to give up checking the elements. Instead, we check the number of OPTIONS file in the database dir. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4218 Differential Revision: D9128727 Pulled By: riversand963 fbshipit-source-id: 2b13b705fb21bc0ddd41940c4ec9b6b0c8d88224 03 August 2018, 20:57:40 UTC
fefdac1 Sagar Vemuri 03 August 2018, 18:56:23 UTC Fix lite build failure in db_bench due to trace/replay (#4225) Summary: Fix lite build failure in db_bench due to trace/replay feature. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4225 Differential Revision: D9153303 Pulled By: sagar0 fbshipit-source-id: 9f7a8035429d0dcdbe99616d11389ed7bccf44be 03 August 2018, 18:58:55 UTC
f9373e2 DorianZheng 03 August 2018, 01:42:06 UTC Make sure to call ReleaseFileNumberFromPendingOutputs Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4219 Differential Revision: D9144294 Pulled By: riversand963 fbshipit-source-id: e46b72e5f8a149dc7a0512e38edcd0ddb0150f30 03 August 2018, 01:57:34 UTC
9dbf393 Pooja Malik 02 August 2018, 22:39:25 UTC Rules Advisor: some fixes to support fetching stats from ODS (#4223) Summary: This PR includes fixes for some bugs that I encountered while testing the Optimizer with ODS stats support. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4223 Differential Revision: D9140786 Pulled By: poojam23 fbshipit-source-id: 045cb3f27d075c2042040ac2d561938349419516 02 August 2018, 22:42:42 UTC
892a156 Pooja Malik 01 August 2018, 23:00:58 UTC Advisor: README and blog, and also tests for DBBenchRunner, DatabaseOptions (#4201) Summary: This pull request adds a README file and a blog post for the Advisor tool. It also adds the missing tests for some Optimizer modules. Some comments are added to the classes being tested for improved readability. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4201 Reviewed By: maysamyabandeh Differential Revision: D9125311 Pulled By: poojam23 fbshipit-source-id: aefcf2f06eaa05490cc2834ef5aa6e21f0d1dc55 01 August 2018, 23:13:09 UTC
f8f6983 Andrew Kryczka 01 August 2018, 19:04:17 UTC Skip range deletions at seqno zero when collapsing (#4216) Summary: `CollapsedRangeDelMap` internally uses seqno zero as a sentinel value to denote a gap between range tombstones or the end of range tombstones. It therefore expects to never have consecutive sentinel tombstones. However, since `DeleteRange` is now supported in `SstFileWriter`, an ingested file may contain range tombstones, and that ingested file may be assigned global seqno zero. When such tombstones are added to the collapsed map, they resemble sentinel tombstones due to having seqno zero. Then, the invariant mentioned above about never having consecutive sentinel tombstones can be violated. The symptom of this violation was dereferencing the `end()` iterator (#4204). The fix in this PR is to not add range tombstones with seqno zero to the collapsed map. They're not needed anyways since they can't possibly cover anything (in case of a key and a range tombstone with the same seqno, the key is visible). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4216 Differential Revision: D9121716 Pulled By: ajkr fbshipit-source-id: f5b78a70bea9527354603ea7ac8542a7e2b6a210 01 August 2018, 19:12:02 UTC
12b6cde Sagar Vemuri 01 August 2018, 07:14:43 UTC Trace and Replay for RocksDB (#3837) Summary: A framework for tracing and replaying RocksDB operations. A binary trace file is created by capturing the DB operations, and it can be replayed back at the same rate using db_bench. - Column-families are supported - Multi-threaded tracing is supported. - TraceReader and TraceWriter are exposed to the user, so that tracing to various destinations can be enabled (say, to other messaging/logging services). By default, a FileTraceReader and FileTraceWriter are implemented to capture to a file and replay from it. - This is not yet ideal to be enabled in production due to large performance overhead, but it can be safely tried out in a shadow setup, say, for analyzing RocksDB operations. Currently supported DB operations: - Writes: -- Put -- Merge -- Delete -- SingleDelete -- DeleteRange -- Write - Reads: -- Get (point lookups) Pull Request resolved: https://github.com/facebook/rocksdb/pull/3837 Differential Revision: D7974837 Pulled By: sagar0 fbshipit-source-id: 8ec65aaf336504bc1f6ed0feae67f6ed5ef97a72 01 August 2018, 07:27:08 UTC
ee76171 Fenggang Wu 31 July 2018, 18:40:41 UTC DataBlockHashIndex: Specify that DataBlockHashIndex is not yet implemented in the comment Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4203 Differential Revision: D9090912 Pulled By: fgwu fbshipit-source-id: 6a68be83693ddf2a5c060290382141f0d2fb400b 31 July 2018, 18:43:08 UTC
a1a546a Andrew Kryczka 31 July 2018, 00:51:31 UTC Avoid integer division in filter probing (#4071) Summary: The cache line size was computed dynamically based on the length of the filter bits, and the number of cache-lines encoded in the footer. This calculation had to be dynamic in case users migrate their data between platforms with different cache line sizes. The downside, though, was bloom filter probing became expensive as it did integer mod and division. However, since we know all possible cache line sizes are powers of two, we should be able to use bit shift to find the cache line, and bitwise-and to find the bit within the cache line. To do this, we compute the log-base-two of cache line size in the constructor, and use that in bitwise operations to replace division/mod. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4071 Differential Revision: D8684067 Pulled By: ajkr fbshipit-source-id: 50298872fba5acd01e8269cd7abcc51a095e0f61 31 July 2018, 00:57:44 UTC
8abafb1 Yanqin Jin 31 July 2018, 00:35:29 UTC Generalize parameters generation. (#4046) Summary: Making generation of column families and keys virtual function so that subclasses of StressTest can override them to provide custom parameter generation for more flexibility. This will be useful for future tests. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4046 Differential Revision: D9073382 Pulled By: riversand963 fbshipit-source-id: 2754f0fdfa5c24d95c1f92d4944bc479552fb665 31 July 2018, 00:42:12 UTC
54de568 Yanqin Jin 27 July 2018, 23:00:26 UTC Remove random writes from SST file ingestion (#4172) Summary: RocksDB used to store global_seqno in external SST files written by SstFileWriter. During file ingestion, RocksDB uses `pwrite` to update the `global_seqno`. Since random write is not supported in some non-POSIX compliant file systems, external SST file ingestion is not supported on these file systems. To address this limitation, we no longer update `global_seqno` during file ingestion. Later RocksDB uses the MANIFEST and other information in table properties to deduce global seqno for externally-ingested SST files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4172 Differential Revision: D8961465 Pulled By: riversand963 fbshipit-source-id: 4382ec85270a96be5bc0cf33758ca2b167b05071 27 July 2018, 23:12:23 UTC
a11df58 Fenggang Wu 27 July 2018, 22:35:41 UTC Add DataBlockIndexType option in BlockBasedTableOptions (#4150) Summary: Added DataBlockIndexType option in BlockBasedTableOptions. ``` enum DataBlockIndexType : char { kDataBlockBinarySearch = 0, // traditional block type kDataBlockHashIndex = 1, // additional hash index appended to the end. }; ``` The default type is the traditional binary seek option: `kDataBlockBinarySearch`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4150 Differential Revision: D8895958 Pulled By: fgwu fbshipit-source-id: 480adef48104cf11d30db3bad9a73f98b4a80c10 27 July 2018, 22:42:27 UTC
f5e4635 DorianZheng 27 July 2018, 21:02:07 UTC Protect external file when ingesting (#4099) Summary: If crash happen after a hard link established, Recover function may reuse the file number that has already assigned to the internal file, and this will overwrite the external file. To protect the external file, we have to make sure the file number will never being reused. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4099 Differential Revision: D9034092 Pulled By: riversand963 fbshipit-source-id: 3f1a737440b86aa2ef01673e5013aacbb7c33e28 27 July 2018, 21:13:12 UTC
c33b326 Maysam Yabandeh 27 July 2018, 18:39:47 UTC Correct description of GetColumnFamilyMetaData (#4196) Summary: The inline doc was incorrectly mentioned a return status while the function does not return a value. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4196 Differential Revision: D9030927 Pulled By: maysamyabandeh fbshipit-source-id: 07c34dc6bf521021bf790ac1bfedb676171129ec 27 July 2018, 18:42:37 UTC
e0906eb Maysam Yabandeh 27 July 2018, 16:26:26 UTC Clarify max_total_wal_size's scope (#4194) Summary: max_total_wal_size takes effect only when there are more than one column families. The patch clarify that in the inline docs Closes https://github.com/facebook/rocksdb/issues/4180 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4194 Differential Revision: D9028767 Pulled By: maysamyabandeh fbshipit-source-id: 8d730ca7f15e76e7ee9ff88b2b48030b2d1b7078 27 July 2018, 16:29:44 UTC
134a52e Pooja Malik 27 July 2018, 00:10:59 UTC Optimizer's skeleton: use advisor to optimize config options (#4169) Summary: In https://github.com/facebook/rocksdb/pull/3934 we introduced advisor scripts that make suggestions in the config options based on the log file and stats from a run of rocksdb. The optimizer runs the advisor on a benchmark application in a loop and automatically applies the suggested changes until the config options are optimized. This is a work in progress and the patch is the initial skeleton for the optimizer. The sample application that is run in the loop is currently dbbench. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4169 Reviewed By: maysamyabandeh Differential Revision: D9023671 Pulled By: poojam23 fbshipit-source-id: a6192d475c462cf6eb2b316716f97cb400fcb64d 27 July 2018, 00:13:32 UTC
bdc6abd Yanqin Jin 26 July 2018, 18:11:02 UTC Enable cscope to exclude test source files (#4190) Summary: Usually when using cscope, the query results contain a lot of function calls in test, making it hard to browse. So this PR aims to provide an option to exclude test source files. Add a new PHONY target, tags0, to exclude test source files while using cscope. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4190 Differential Revision: D9015901 Pulled By: riversand963 fbshipit-source-id: ea9a45756ccff5b26344d37e9ff1c02c5d9736d6 26 July 2018, 18:12:29 UTC
fd45495 Siying Dong 26 July 2018, 18:03:44 UTC DBImpl::IngestExternalFile() should grab mutex when releasing file number in failure case (#4189) Summary: 995fcf757319da9cf12eca2df83a6fba4db0ebe4 has a bug: ReleaseFileNumberFromPendingOutputs() added is not protected by the DB mutex. Fix it by grabbing the lock for this operation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4189 Differential Revision: D9015447 Pulled By: siying fbshipit-source-id: b8506e09a96c3f95a6fe32b5ca5fcdb9bee88937 26 July 2018, 18:12:29 UTC
2a81633 Siying Dong 26 July 2018, 00:03:39 UTC Fix bug when seeking backward against an out-of-bound iterator (#4187) Summary: 92ee3350e0ae02c0973af0fbd40fb67b0b958128 introduces an out-of-bound check in BlockBasedTableIterator::Valid(). However, this flag is not reset when re-seeking in backward direction. This caused the iterator to be invalide by mistake. Fix it by always resetting the out-of-bound flag in every seek. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4187 Differential Revision: D8996600 Pulled By: siying fbshipit-source-id: b6235ea614f71381e50e7904c4fb036300604ac1 26 July 2018, 00:14:01 UTC
18f5380 Yanqin Jin 24 July 2018, 20:27:47 UTC Increase version number to 5.16 (#4176) Summary: Given that we have cut 5.15, we should bump the version number to the next version, i.e. 5.16. Also update HISTORY.md cc sagar0 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4176 Differential Revision: D8977965 Pulled By: riversand963 fbshipit-source-id: 481d75d2f446946f0eb2afb7e94ef894c8c87e1e 24 July 2018, 20:43:33 UTC
8805ec2 Fenggang Wu 24 July 2018, 18:42:19 UTC DataBlockHashIndex: Standalone Implementation with Unit Test (#4139) Summary: The first step of the `DataBlockHashIndex` implementation. A string based hash table is implemented and unit-tested. `DataBlockHashIndexBuilder`: `Add()` takes pairs of `<key, restart_index>`, and formats it into a string when `Finish()` is called. `DataBlockHashIndex`: initialized by the formatted string, and can interpret it as a hash table. Lookup for a key is supported by iterator operation. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4139 Reviewed By: sagar0 Differential Revision: D8866764 Pulled By: fgwu fbshipit-source-id: 7f015f0098632c65979a22898a50424384730b10 24 July 2018, 18:43:37 UTC
back to top