Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
6ad033c Siying Dong 26 January 2019, 01:31:06 UTC Fix test name typo in PlainTableDBTest 26 January 2019, 01:31:06 UTC
f184bee Siying Dong 26 January 2019, 01:07:00 UTC PlainTable should avoid copying Get() results from immortal source. (#4924) Summary: https://github.com/facebook/rocksdb/pull/4053 avoids memcopy for Get() results if files are immortable (read-only DB, max_open_files=-1) and the file is ammaped. The same optimization is being applied to PlainTable here. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4924 Differential Revision: D13827749 Pulled By: siying fbshipit-source-id: 1f2cbfc530b40ce08ccd53f95f6e78de4d1c2f96 26 January 2019, 01:12:19 UTC
e1de88c Yanqin Jin 24 January 2019, 19:10:06 UTC Escape '.' by adding a '\' to avoid matching any char Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4912 Differential Revision: D13789449 Pulled By: riversand963 fbshipit-source-id: 0639dae82049b7ac977c8f81851f1c9fdc346705 24 January 2019, 19:25:27 UTC
fc53839 Siying Dong 24 January 2019, 18:11:19 UTC Disallow customized hash function in DynamicBloom (#4915) Summary: I didn't find where customized hash function is used in DynamicBloom. This can only reduce performance. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4915 Differential Revision: D13794452 Pulled By: siying fbshipit-source-id: e38669b11e01444d2d782da11c7decabbd851819 24 January 2019, 18:34:30 UTC
e07aa86 Dmitry Fink 24 January 2019, 05:43:44 UTC Allow full merge when root of history for a key is reached (#4909) Summary: Previously compaction was not collapsing operands for a first key on a layer, even in cases when it was its root of history. Some tests (CompactionJobTest.NonAssocMerge) was actually accounting for that bug, Pull Request resolved: https://github.com/facebook/rocksdb/pull/4909 Differential Revision: D13781169 Pulled By: finik fbshipit-source-id: d2de353ecf05bec39b942cd8d5b97a8dc445f336 24 January 2019, 05:46:10 UTC
8ec3e72 Andrew Kryczka 24 January 2019, 02:11:08 UTC Cache dictionary used for decompressing data blocks (#4881) Summary: - If block cache disabled or not used for meta-blocks, `BlockBasedTableReader::Rep::uncompression_dict` owns the `UncompressionDict`. It is preloaded during `PrefetchIndexAndFilterBlocks`. - If block cache is enabled and used for meta-blocks, block cache owns the `UncompressionDict`, which holds dictionary and digested dictionary when needed. It is never prefetched though there is a TODO for this in the code. The cache key is simply the compression dictionary block handle. - New stats for compression dictionary accesses in block cache: "BLOCK_CACHE_COMPRESSION_DICT_*" and "compression_dict_block_read_count" Pull Request resolved: https://github.com/facebook/rocksdb/pull/4881 Differential Revision: D13663801 Pulled By: ajkr fbshipit-source-id: bdcc54044e180855cdcc57639b493b0e016c9a3f 24 January 2019, 02:15:47 UTC
43defe9 PeifengSi 24 January 2019, 01:58:14 UTC Correct the code comment in Compaction::KeyNotExistsBeyondOutputLevel (#4902) Summary: Even one key falls in a file's range, we can not infer it definitely exists in this file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4902 Differential Revision: D13795018 Pulled By: siying fbshipit-source-id: 590956f727e9440fcdee55ad9541ace934c64914 24 January 2019, 02:00:56 UTC
d94aa2f Siying Dong 24 January 2019, 00:44:02 UTC Make compaction_pri = kMinOverlappingRatio to be default (#4911) Summary: compaction_pri = kMinOverlappingRatio usually provides much better write amplification than the default. https://github.com/facebook/rocksdb/pull/4907 fixes one shortcome of this option. Make it default. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4911 Differential Revision: D13789262 Pulled By: siying fbshipit-source-id: d90acf8c4dede44f00d183ca4c7a210259378269 24 January 2019, 00:47:38 UTC
27054d8 Andrew Kryczka 23 January 2019, 23:36:31 UTC Call NewDataBlockIterator with correct arguments in DB::Get (#4913) Summary: The pointer `get_context` was passed as the value for the boolean argument `index_key_is_full`. Luckily the pointer was always non-null so evaluated to true which is the correct value for the boolean argument. But we were missing out on batch updates to stats since we were not passing anything for the `GetContext*` argument and it defaults to `nullptr`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4913 Differential Revision: D13791449 Pulled By: ajkr fbshipit-source-id: dbe40bf406c64d34cb5298604145d18b9e0ca9be 23 January 2019, 23:39:05 UTC
0cead31 Sagar Vemuri 23 January 2019, 21:26:03 UTC Fix Clang static analyzer warning in db_bench (#4910) Summary: Fixed clang static analyzer warning about division by 0. ``` ar: creating librocksdb_debug.a tools/db_bench_tool.cc:4650:43: warning: Division by zero int pos = static_cast<int>(rand_num % range_); ~~~~~~~~~^~~~~~~~ 1 warning generated. make: *** [analyze] Error 1 ``` This is from the new code I recently merged in ce8e88d2d7a62e2a08c4109aac84cb9e95ed359b. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4910 Differential Revision: D13788037 Pulled By: sagar0 fbshipit-source-id: f48851dca85047c19fbb1a361e25ce643aa4c7ea 23 January 2019, 21:33:02 UTC
5bf9419 Siying Dong 23 January 2019, 21:18:13 UTC CompactionPri = kMinOverlappingRatio also uses compensated file size (#4907) Summary: Right now, CompactionPri = kMinOverlappingRatio provides best write amplification, but it doesn't prioritize files with more tombstones. We combine the two good features: make kMinOverlappingRatio to boost files with lots of tombstones too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4907 Differential Revision: D13788774 Pulled By: siying fbshipit-source-id: 1991cbb495fb76c8b529de69896e38d81ed9d9b3 23 January 2019, 21:21:01 UTC
1eded07 Remington Brasga 23 January 2019, 18:19:27 UTC Bug in Regular Expression in Makefile (#4682) Summary: False-negative about path not existing. The regex is ignoring the "." in front of a path. Example: "./path/to/file" Pull Request resolved: https://github.com/facebook/rocksdb/pull/4682 Differential Revision: D13777110 Pulled By: sagar0 fbshipit-source-id: 9f8173b7581407555fdc055580732aeab37d4ade 23 January 2019, 18:24:10 UTC
cbe0239 Zhongyi Xie 23 January 2019, 06:41:49 UTC add cast to avoid loss of precision error (#4906) Summary: this PR address the following error: > tools/db_bench_tool.cc:4776:68: error: implicit conversion loses integer precision: 'int64_t' (aka 'long') to 'unsigned int' [-Werror,-Wshorten-64-to-32] s = db_with_cfh->db->Put(write_options_, key, gen.Generate(value_size)); Pull Request resolved: https://github.com/facebook/rocksdb/pull/4906 Differential Revision: D13780185 Pulled By: miasantreble fbshipit-source-id: 1c83a77d341099518c72f0f4a63e97ab9c4784b3 23 January 2019, 06:44:17 UTC
08b8cea Siying Dong 23 January 2019, 00:57:40 UTC Deleting Blob files also goes through SstFileManager (#4904) Summary: Right now, deleting blob files is not rate limited, even if SstFileManger is specified. On the other hand, rate limiting blob deletion is not supported. With this change, Blob file deletion will go through SstFileManager too. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4904 Differential Revision: D13772545 Pulled By: siying fbshipit-source-id: bd1b1d0beb26d5167385e00b7ecb8b94b879de84 23 January 2019, 01:00:29 UTC
b2ba068 Philip Jameson 22 January 2019, 23:20:28 UTC Add load() statements to TARGETS files Reviewed By: luciang Differential Revision: D13733578 fbshipit-source-id: 556c115935aa42c1da85ec0e91199b9f198fc467 22 January 2019, 23:24:51 UTC
8189c18 Sagar Vemuri 22 January 2019, 19:10:05 UTC Remove unused Blob WAL filter (#4896) Summary: Remove unused blob WAL filter so that users are not confused. I was initially under the impression that we have WAL Filter support in BlobDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4896 Differential Revision: D13725709 Pulled By: sagar0 fbshipit-source-id: f997d7546e138a474036e88b957907cc714327f1 22 January 2019, 19:13:49 UTC
ce8e88d Zhichao Cao 22 January 2019, 18:40:44 UTC Generate mixed workload with Get, Put, Seek in db_bench (#4788) Summary: Based on the specific workload models (key access distribution, value size distribution, and iterator scan length distribution, the QPS variation), the MixGraph benchmark generate the synthetic workload according to these distributions which can reflect the real-world workload characteristics. After user enable the tracing function, they will get the trace file. By analyzing the trace file with the trace_analyzer tool, user can generate a set of statistic data files including. The *_accessed_key_stats.txt, *-accessed_value_size_distribution.txt, *-iterator_length_distribution.txt, and *-qps_stats.txt are mainly used to fit the Matlab model fitting. After that, user can get the parameters of the workload distributions (the modeling details are described: [here](https://github.com/facebook/rocksdb/wiki/RocksDB-Trace%2C-Replay%2C-and-Analyzer)) The key access distribution follows the The two-term power model. The probability density function is: `f(x) = ax^{b}+c`. The corresponding parameters are key_dist_a, key_dist_b, and key_dist_c in db_bench For the value size distribution and iterator scan length distribution, they both follow the Generalized Pareto Distribution. The probability density function is `f(x) = (1/sigma)(1+k*(x-theta)/sigma))^{-1-1/k)`. The parameters are: value_k, value_theta, value_sigma and iter_k, iter_theta, iter_sigma. For more information about the Generalized Pareto Distribution, users can find the [wiki](https://en.wikipedia.org/wiki/Generalized_Pareto_distribution) and [Matalb page](https://www.mathworks.com/help/stats/generalized-pareto-distribution.html) As for the QPS, it follows the diurnal pattern. So Sine is a good model to fit it. `F(x) = sine_a*sin(sine_b*x + sine_c) + sine_d`. The trace_will tell you the average QPS in the print out resutls, which is sine_d. After user fit the "*-qps_stats.txt" to the Matlab model, user can get the sine_a, sine_b, and sine_c. By using the 4 parameters, user can control the QPS variation including the period, average, changes. To use the bench mark, user can indicate the following parameters as examples: ``` -benchmarks="mixgraph" -key_dist_a=0.002312 -key_dist_b=0.3467 -value_k=0.9233 -value_sigma=226.4092 -iter_k=2.517 -iter_sigma=14.236 -mix_get_ratio=0.7 -mix_put_ratio=0.25 -mix_seek_ratio=0.05 -sine_mix_rate_interval_milliseconds=500 -sine_a=15000 -sine_b=1 -sine_d=20000 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4788 Differential Revision: D13573940 Pulled By: sagar0 fbshipit-source-id: e184c27e07b4f1bc0b436c2be36c5090c1fb0222 22 January 2019, 18:44:26 UTC
16a5ac5 Andrew Kryczka 20 January 2019, 03:15:16 UTC Update HISTORY.md with new use of ZSTD_CDict (#4901) Summary: Mention feature introduced by #4849 in HISTORY.md. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4901 Differential Revision: D13746430 Pulled By: ajkr fbshipit-source-id: f7bdea6f0522ed55428cbc521f8a9f3cd0002d4e 20 January 2019, 03:17:50 UTC
01013ae Andrew Kryczka 19 January 2019, 03:10:17 UTC Digest ZSTD compression dictionary once when writing SST file (#4849) Summary: This is essentially a re-submission of #4251 with a few improvements: - Split `CompressionDict` into two separate classes: `CompressionDict` and `UncompressionDict` - Eliminated `Init` functions. Instead do all initialization work in constructors. - Added test case for parallel DB open, which is the scenario where #4251 failed under TSAN. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4849 Differential Revision: D13606039 Pulled By: ajkr fbshipit-source-id: 08c236059798c710db9cbf545fce0f371232d447 19 January 2019, 03:12:57 UTC
b1ad6eb Yi Wu 19 January 2019, 01:20:13 UTC WritePrepared: fix two versions in compaction see different status for released snapshots (#4890) Summary: Fix how CompactionIterator::findEarliestVisibleSnapshots handles released snapshot. It fixing the two scenarios: Scenario 1: key1 has two values v1 and v2. There're two snapshots s1 and s2 taken after v1 and v2 are committed. Right after compaction output v2, s1 is released. Now findEarliestVisibleSnapshot may see s1 being released, and return the next snapshot, which is s2. That's larger than v2's earliest visible snapshot, which was s1. The fix: the only place we check against last snapshot and current key snapshot is when we decide whether to compact out a value if it is hidden by a later value. In the check if we see current snapshot is even larger than last snapshot, we know last snapshot is released, and we are safe to compact out current key. Scenario 2: key1 has two values v1 and v2. there are two snapshots s1 and s2 taken after v1 and v2 are committed. During compaction before we process the key, s1 is released. When compaction process v2, snapshot checker may return kSnapshotReleased, and the earliest visible snapshot for v2 become s2. When compaction process v1, snapshot checker may return kIsInSnapshot (for WritePrepared transaction, it could be because v1 is still in commit cache). The result will become inconsistent here. The fix: remember the set of released snapshots ever reported by snapshot checker, and ignore them when finding result for findEarliestVisibleSnapshot. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4890 Differential Revision: D13705538 Pulled By: maysamyabandeh fbshipit-source-id: e577f0d9ee1ff5a6035f26859e56902ecc85a5a4 19 January 2019, 01:24:06 UTC
7fd9813 Maysam Yabandeh 18 January 2019, 19:33:30 UTC WritePrepared: commit of delayed prepared entries (#4894) Summary: Here is the order of ops in a commit: 1) update commit cache 2) publish seq, 3) RemovePrepared. In case of a delayed prepared, there will be a gap between when the commit is visible to snapshots until delayed_prepared_ is cleaned up. To tell apart this case from a delayed uncommitted txn from, the commit entry of a delayed prepared is also stored in delayed_prepared_commits_, which is updated before publishing the commit. Also logic in GetSnapshotInternal that ensures that each new snapshot is always larger than max_evicted_seq_ is updated to check against the upcoming value of max_evicted_seq_ rather than its current one. This is because AdvanceMaxEvictedSeq gets the list of snapshots lower than the new max, before updating max_evicted_seq_. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4894 Differential Revision: D13726988 Pulled By: maysamyabandeh fbshipit-source-id: 1e70d78061b50c944c9816bf4b6dac405ab4ccd3 18 January 2019, 19:36:36 UTC
73ff15c tom wang 17 January 2019, 20:03:08 UTC WritePrepared: fix typo in comments Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4891 Differential Revision: D13718016 Pulled By: miasantreble fbshipit-source-id: 90bd372cff453a1c2d104c1cf49731d5dd770c14 17 January 2019, 20:36:36 UTC
dd9eca1 Yanqin Jin 17 January 2019, 19:37:03 UTC Remove unused variable to fix clang compilation err (#4893) Summary: as title. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4893 Differential Revision: D13716733 Pulled By: riversand963 fbshipit-source-id: 6811d6a99fe2094d5344f854e8939f01238b2adb 17 January 2019, 19:57:31 UTC
3cfc751 Sagar Vemuri 16 January 2019, 19:23:42 UTC Remove an unused option (#4888) Summary: Remove `garbage_collection_deletion_size_threshold` as it is not used anywhere. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4888 Differential Revision: D13685982 Pulled By: sagar0 fbshipit-source-id: e08d3017b9a0c8fa99bc332b595ee4ed9db70c87 16 January 2019, 19:48:43 UTC
128f532 Yi Wu 16 January 2019, 17:48:10 UTC WritePrepared: fix issue with snapshot released during compaction (#4858) Summary: Compaction iterator keep a copy of list of live snapshots at the beginning of compaction, and then query snapshot checker to verify if values of a sequence number is visible to these snapshots. However when the snapshot is released in the middle of compaction, the snapshot checker implementation (i.e. WritePreparedSnapshotChecker) may remove info with the snapshot and may report incorrect result, which lead to values being compacted out when it shouldn't. This patch conservatively keep the values if snapshot checker determines that the snapshots is released. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4858 Differential Revision: D13617146 Pulled By: maysamyabandeh fbshipit-source-id: cf18a94f6f61a94bcff73c280f117b224af5fbc3 16 January 2019, 17:55:32 UTC
e79df37 Yanqin Jin 16 January 2019, 17:48:01 UTC Use chrono::time_point instead of time_t (#4868) Summary: By convention, time_t almost always stores the integral number of seconds since 00:00 hours, Jan 1, 1970 UTC, according to http://www.cplusplus.com/reference/ctime/time_t/. We surely want more precision than seconds. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4868 Differential Revision: D13633046 Pulled By: riversand963 fbshipit-source-id: 4e01e23a22e8838023c51a91247a286dbf3a5396 16 January 2019, 17:51:05 UTC
5d4fddf Yi Wu 16 January 2019, 05:32:15 UTC WritePrepared: Fix visible key compacted out by compaction (#4883) Summary: With WritePrepared transaction, flush/compaction can contain uncommitted keys, and those keys can get committed during compaction. If a snapshot is taken before the key is committed, it should not see the key. On the other hand, compaction grab the list of snapshots at its beginning, and only consider those snapshots to dedup keys. Consider the case: ``` seq = 1: put "foo" = "bar" seq = 2: transaction T: delete "foo", prepare seq = 3: compaction start seq = 4: take snapshot S seq = 5: transaction T: commit. ... seq = N: compaction iterator reached key "foo". ``` When compaction start, the list of snapshot is empty. Compaction doesn't take snapshot S into account. When it reached "foo", transaction T is committed. Compaction may think the value "foo=bar" is not visible by any snapshot (which is wrong), and compact the value out. The fix is to explicitly take a snapshot before compaction grabbing the list of snapshots. Compaction will then has to keep keys visible to this snapshot. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4883 Differential Revision: D13668775 Pulled By: maysamyabandeh fbshipit-source-id: 1cab9615f94b7d3e8522cc3d44c3a14c7d4720e4 16 January 2019, 05:34:38 UTC
cad99a6 Maysam Yabandeh 16 January 2019, 02:07:50 UTC WritePrepared: snapshot should be larger than max_evicted_seq_ (#4886) Summary: The AdvanceMaxEvictedSeq algorithm assumes that new snapshots always have sequence number larger than the last max_evicted_seq_. To enforce this assumption we make two changes: i) max is not advanced beyond the last published seq, with the exception that the evicted commit entry itself is not published yet, which is quite rare. ii) When obtaining the snapshot if the max_evicted_seq_ is not published yet, commit a dummy entry so that it waits for it to be published and also increased the latest published seq by one above the max. To test these non-realistic corner cases we create a commit cache with size 1 so that every single commit results into eviction. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4886 Differential Revision: D13685270 Pulled By: maysamyabandeh fbshipit-source-id: 5461bc09c2a9b75798bfcb9853a256c81cdac0b0 16 January 2019, 02:11:52 UTC
7d13f30 Siying Dong 16 January 2019, 00:41:50 UTC Improve Error Message When wal_dir doesn't exist (#4874) Summary: Right now the error mesage when options.wal_dir doesn't exist is not helpful to users. Be more specific Pull Request resolved: https://github.com/facebook/rocksdb/pull/4874 Differential Revision: D13642425 Pulled By: siying fbshipit-source-id: 9a3172ed0f799af233b0f3b2e5e35bc7ce04c7b5 16 January 2019, 00:46:04 UTC
55e03b6 Sagar Vemuri 16 January 2019, 00:25:03 UTC Correct the comment about inlined blob option (#4887) Summary: - Corrected a comment asserting that the values "smaller" than a min_blob_size will be inlined in the base db. - Also fixed the type of ttl_range_secs while dumping blobdb options. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4887 Differential Revision: D13680163 Pulled By: sagar0 fbshipit-source-id: 306c8cf2daa52210ffc334a6924ef44ffdedf887 16 January 2019, 00:36:49 UTC
d50c10e Yi Wu 15 January 2019, 17:13:49 UTC WritePrepared: Fix SmallestUnCommittedSeq() doesn't check delayed_prepared (#4867) Summary: When prepared_txns_ heap is empty, SmallestUnCommittedSeq() should check delayed_prepared_ set as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4867 Differential Revision: D13632134 Pulled By: maysamyabandeh fbshipit-source-id: b0423bb0a58dc95f1e636d5ed3f6e619df801fb7 15 January 2019, 17:17:53 UTC
301da34 Yanqin Jin 12 January 2019, 01:40:44 UTC Make a copy of MutableCFOptions to avoid race condition (#4876) Summary: If we do not do this, then reading MutableCFOptions may have a race condition with SetOptions which modifies MutableCFOptions. Also reserve space in advance for vectors to avoid reallocation changing the address of its elements. Test plan ``` $make clean && make -j32 all check $make clean && COMPILE_WITH_TSAN=1 make -j32 all check $make clean && COMPILE_WITH_ASAN=1 make -j32 all check ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4876 Differential Revision: D13644500 Pulled By: riversand963 fbshipit-source-id: 4b8112c5c819d5a2922bb61ad1521b3d2fb2fd47 12 January 2019, 01:43:37 UTC
4e37251 Siying Dong 12 January 2019, 00:42:26 UTC With ldb --try_load_options and wal_dir doesn't exist, ignore it (#4875) Summary: LDB is frequently used to exam data copied. wal_dir in option file is not modified and it usually points to the path it copied from. The user experience will be better if when ldb sees wal_dir pointed by the option file doesn't exist, rather than fail, just ignore it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4875 Differential Revision: D13643173 Pulled By: siying fbshipit-source-id: 2e64d4ea2ec49a6794b9a706b7fc1ba901128bb8 12 January 2019, 00:48:32 UTC
856ac24 Maysam Yabandeh 11 January 2019, 21:39:26 UTC WritePrepared: fix race condition on GetSnapshotListFromDB (#4872) Summary: Fixes a typo that made mutex_ to remain unlocked when GetSnapshotListFromDB called from WritePreparedTxnDB. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4872 Differential Revision: D13640381 Pulled By: maysamyabandeh fbshipit-source-id: 50f6600568f9092b4b43115f6ebd96e6c7388ad7 11 January 2019, 21:46:23 UTC
6a4ec41 Zhongyi Xie 11 January 2019, 20:02:47 UTC add assert to silence clang warning (#4871) Summary: currently clang analyze fails with the following warning: > utilities/transactions/write_prepared_transaction_test.cc:1451:5: warning: Forming reference to null pointer ASSERT_GT(wp_db->max_evicted_seq_, 0); // max after recovery Pull Request resolved: https://github.com/facebook/rocksdb/pull/4871 Differential Revision: D13638053 Pulled By: miasantreble fbshipit-source-id: b192b0c13c411c58defc9e280b34cdfcab3fa8e3 11 January 2019, 20:17:34 UTC
349c7cc Varadharajan 11 January 2019, 18:22:19 UTC Fix downloaded filename of snappy (#4870) Summary: Build failing due to incorrect filename. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4870 Differential Revision: D13637205 Pulled By: sagar0 fbshipit-source-id: 72da45d51b49bce32f696532ba0656ee0dc2b89f 11 January 2019, 18:29:40 UTC
1fb2e27 Siying Dong 10 January 2019, 21:23:35 UTC Remove some components (#4101) Summary: Remove some components that we never heard people using them. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4101 Differential Revision: D8825431 Pulled By: siying fbshipit-source-id: 97a12ad3cad4ab12c82741a5ba49669aaa854180 10 January 2019, 21:30:09 UTC
d56ac22 Maysam Yabandeh 10 January 2019, 00:09:36 UTC Remove duplicates from SnapshotList::GetAll (#4860) Summary: The vector returned by SnapshotList::GetAll could have duplicate entries if two separate snapshots have the same sequence number. However, when this vector is used in compaction the duplicate entires are of no use and could be safely ignored. Moreover not having duplicate entires simplifies reasoning in the compaction_iterator.cc code. For example when searching for the previous_snap we currently use the snapshot before the current one but the way the code uses that it expects it to be also less than the current snapshot, which would be simpler to read if there is no duplicate entry in the snapshot list. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4860 Differential Revision: D13615502 Pulled By: maysamyabandeh fbshipit-source-id: d45bf01213ead5f39db811f951802da6fcc3332b 10 January 2019, 00:25:42 UTC
75714b4 Yanqin Jin 09 January 2019, 23:51:02 UTC Initialize two members in PerfContext (#4859) Summary: as titled. Currently it's possible to create a local object of type PerfContext since it's part of public API. Then it's safe to initialize the two members to 0. If PerfContext is created as thread-local object, then all members are zero-initialized according to C++ standard. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4859 Differential Revision: D13614504 Pulled By: riversand963 fbshipit-source-id: 406ff548e105a074f379ad1054d56fece5f524a0 09 January 2019, 23:55:03 UTC
ffc9f84 Yanqin Jin 09 January 2019, 01:16:37 UTC Free memory after use Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4857 Differential Revision: D13602688 Pulled By: riversand963 fbshipit-source-id: 993419a6afb982a7a701ff71daebebb4b4a6b265 09 January 2019, 01:19:09 UTC
f3a99e8 Maysam Yabandeh 08 January 2019, 22:43:24 UTC WritePrepared: Report released snapshots in IsInSnapshot (#4856) Summary: Previously IsInSnapshot assumed that the snapshot is valid at the time that the function is called. However there are cases where that might not be valid. Example is background compactions where the compaction algorithm operates with a list of snapshots some of which might be released by the time they are being passed to IsInSnapshot. The patch make two changes to enable the caller to tell difference: i) any live snapshot below max is added to max_committed_seq_, which allows IsInSnapshot to confidently tell whether the passed snapshot is invalid if it below max, ii) extends IsInSnapshot API with a "released" variable that is set true when IsInSnapshot find no such snapshot below max and also find no other way to give a certain return value. In such cases the return value is true but the caller should also check the "released" boolean after the call. In short here is the changes in the API: i) If the snapshot is valid, no change is required. ii) If the snapshot might be invalid, a reference to "released" boolean must be passed to IsInSnapshot. ii-a) If snapshot is above max, IsInSnapshot can figure the return valid using the commit cache. ii-b) otherwise if snapshot is in old_commit_map_, IsInSnapshot can use that to tell if value was visible to the snapshot. ii-c) otherwise it sets "released" to true and returns true as well. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4856 Differential Revision: D13599847 Pulled By: maysamyabandeh fbshipit-source-id: 1752be28667f886a1efec8cae5714b9b7a8f1e0f 08 January 2019, 22:47:29 UTC
8641e9a Siying Dong 08 January 2019, 20:44:56 UTC Non-initial file preloading should always prefetch index and filter (#4852) Summary: https://github.com/facebook/rocksdb/pull/3340 introduces preloading when max_open_files != -1. It doesn't preload index and filter in non-initial file loading case. This is a little bit too complicated to understand. We observed in one MyRocks use case where the filter is expected to be preloaded but is not. To simplify the use case, we simply always prefetch the index and filter. They anyway is expected to be loaded in the file verification phase anyway. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4852 Differential Revision: D13595402 Pulled By: siying fbshipit-source-id: d4d8624eb3e849e20aeb990df2100502d85aff31 08 January 2019, 20:47:34 UTC
cd227d7 Maysam Yabandeh 08 January 2019, 19:23:10 UTC WritePrepared: improve IsInSnapshotEmptyMapTest (#4853) Summary: IsInSnapshotEmptyMapTest tests that IsInSnapshot returns correct value for existing data after a recovery, where max is not zero and yet commit cache is empty. The existing test was preliminary which is improved in this patch. It also increases the db sequence after recovery so that there the snapshot immediately taken after recovery would have a sequence number different than that of max_evicted_seq. This simplifies the logic in IsInSnapshot by not having to consider the special case that an old snapshot might be equal to max_evicted_seq and yet not present in old_commit_map. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4853 Differential Revision: D13595223 Pulled By: maysamyabandeh fbshipit-source-id: 77c12ca8a3f61a47479a93bef2038ff502dc3322 08 January 2019, 19:27:11 UTC
e686caf Yanqin Jin 08 January 2019, 00:44:52 UTC Remove unnecessary assersion in AtomicFlushStressTest::TestCheckpoint (#4846) Summary: as titled. We can remove the assersion because we do not perform verification in AtomicFlushStressTest::TestCheckpoint for similar reasons to TestGet, TestPut, etc. Therefore, we override TestCheckpoint in AtomicFlushStressTest so that the assertion `rand_column_families.size() == rand_keys.size()' is removed, and we do not verify the DB in this function. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4846 Differential Revision: D13583377 Pulled By: riversand963 fbshipit-source-id: 03647f3da67e27a397413fd666e3bb43003bf596 08 January 2019, 00:47:26 UTC
0ed98bf Maysam Yabandeh 07 January 2019, 22:53:26 UTC WritePrepared: fix snapshot sequence in rollback (#4851) Summary: The rollback algorithm in WritePrepared transactions requires reading the values before the transaction start. Currently it uses the prepare_seq -1 as the snapshot sequence number for the read. This is not correct since the passed sequence number must be for a valid snapshot. The patch fixes it by passing kMaxSequenceNumber instead. This is fine since all the writes done by the aborted transaction will be skipped during the read anyway. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4851 Differential Revision: D13592773 Pulled By: maysamyabandeh fbshipit-source-id: ff1bf92ea9909d4cccb173bdff49febc0e9eb7a2 07 January 2019, 22:57:03 UTC
4213552 tom wang 07 January 2019, 21:50:26 UTC modify comments about flush_queue_ Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4850 Differential Revision: D13591940 Pulled By: sagar0 fbshipit-source-id: 617794e0a41d0f4554d40871180b061e84189fc5 07 January 2019, 21:52:59 UTC
a3ab59d Philip Jameson 07 January 2019, 21:35:12 UTC Fix skylark incompatible build files in rocksdb Reviewed By: ttsugriy Differential Revision: D13583867 fbshipit-source-id: 8f218a9ffd9807d386ba0adc966af2a9a48ac64c 07 January 2019, 21:37:40 UTC
cf852fd Yi Wu 05 January 2019, 00:21:44 UTC Minor fix: single delete a blob value is not a mismatch (#4848) Summary: In compaction iterator, if the next value of single delete is a blob value, it should not treated as mismatch. This is only a minor fix and doesn't affect correctness. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4848 Differential Revision: D13585812 Pulled By: yiwu-arbug fbshipit-source-id: 0ff6223fa03a644ac9fd8a2d77f9d6711d0a62b0 05 January 2019, 00:31:02 UTC
9e2c804 Andrew Kryczka 04 January 2019, 19:21:32 UTC Fix point lookup on range tombstone sentinel endpoint (#4829) Summary: Previously for point lookup we decided which file to look into based on user key overlap only. We also did not truncate range tombstones in the point lookup code path. These two ideas did not interact well in cases like this: - L1 has range tombstone [a, c)#1 and point key b#2. The data is split between file1 with range [a#1,1, b#72057594037927935,15], and file2 with range [b#2, c#1]. - L1's file2 gets compacted to L2. - User issues `Get()` for b#3. - L1's file1 is opened and the range tombstone [a, c)#1 is found for b, while no point-key for b is found in L1. - `Get()` assumes that the range tombstone must cover all data in that range in lower levels, so short circuits and returns `NotFound`. The solution to this problem is to not look into files that only overlap with the point lookup at a range tombstone sentinel endpoint. In the above example, this would mean not opening L1's file1 or its tombstones during the `Get()`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4829 Differential Revision: D13561355 Pulled By: ajkr fbshipit-source-id: a13c21c816870a2f5d32a48af6dbd719a7d9d19f 04 January 2019, 19:24:08 UTC
a07175a Yanqin Jin 04 January 2019, 04:53:52 UTC Refactor atomic flush result installation to MANIFEST (#4791) Summary: as titled. Since different bg flush threads can flush different sets of column families (due to column family creation and drop), we decide not to let one thread perform atomic flush result installation for other threads. Bg flush threads will install their atomic flush results sequentially to MANIFEST, using a conditional variable, i.e. atomic_flush_install_cv_ to coordinate. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4791 Differential Revision: D13498930 Pulled By: riversand963 fbshipit-source-id: dd7482fc41f4bd22dad1e1ef7d4764ef424688d7 04 January 2019, 04:56:24 UTC
77a8d4d Yi Wu 04 January 2019, 00:26:31 UTC Detect if Jemalloc is linked with the binary (#4844) Summary: Declare Jemalloc non-standard APIs as weak symbols, so that if Jemalloc is linked with the binary, these symbols will be replaced by Jemalloc's, otherwise they will be nullptr. This is similar to how folly detect jemalloc, but we assume the main program use jemalloc as long as jemalloc is linked: https://github.com/facebook/folly/blob/master/folly/memory/Malloc.h#L147 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4844 Differential Revision: D13574934 Pulled By: yiwu-arbug fbshipit-source-id: 7ea871beb1be7d5a1259cc38f9b78078793db2db 04 January 2019, 00:30:12 UTC
8c79f79 DorianZheng 03 January 2019, 20:36:43 UTC Fix skip WAL for whole write_group when leader's callback fail (#4838) Summary: The original implementation has two problems: 1. https://github.com/facebook/rocksdb/blob/f0dda35d7de1fd56e0b7c96376ca8aff2a6364fd/db/db_impl_write.cc#L478 https://github.com/facebook/rocksdb/blob/f0dda35d7de1fd56e0b7c96376ca8aff2a6364fd/db/write_thread.h#L231 If the callback status of leader of the write_group fails, then the whole write_group will not write to WAL, this may cause data loss. 2. https://github.com/facebook/rocksdb/blob/f0dda35d7de1fd56e0b7c96376ca8aff2a6364fd/db/write_thread.h#L130 The annotation says that Writer.status is the status of memtable inserter, but the original implementation use it for another case which is not consistent with the original design. Looks like we can still reuse Writer.status, but we should modify the annotation, so Writer.status is not only the status of memtable inserter. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4838 Differential Revision: D13574070 Pulled By: yiwu-arbug fbshipit-source-id: a2a2aefcfd329c4c6a91652bf090aaf1ce119c4b 03 January 2019, 20:40:42 UTC
74f7d75 Huachao Huang 03 January 2019, 19:11:09 UTC tools: use provided options instead of the default (#4839) Summary: The current implementation hardcode the default options in different places, which makes it impossible to support other environments (like encrypted environment). Pull Request resolved: https://github.com/facebook/rocksdb/pull/4839 Differential Revision: D13573578 Pulled By: sagar0 fbshipit-source-id: 76b58b4b758902798d10ff2f52d9f39abff015e7 03 January 2019, 19:23:49 UTC
e4feb78 Siying Dong 03 January 2019, 18:54:09 UTC Try to fix DBSSTTest.RateLimitedDelete flakiness (#4840) Summary: DBSSTTest.RateLimitedDelete is flakey. The root cause is not completely identified, but the compaction waiting in the test doesn't strictly wait for compaction cleaning to finish, which may cause test flakiness. Fix it first and see whether the failures still happen. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4840 Differential Revision: D13567273 Pulled By: siying fbshipit-source-id: 6fce38b912aff92a925231e7aa9bb0fef892761a 03 January 2019, 19:05:19 UTC
0732853 Adam Retter 03 January 2019, 18:40:11 UTC Add RocksJava build to AppVeyor Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4843 Differential Revision: D13573607 Pulled By: sagar0 fbshipit-source-id: 0e752992d1d0187cd423f47b53f9b1f80555f8cd 03 January 2019, 18:44:44 UTC
565b5bd Yanqin Jin 03 January 2019, 01:38:27 UTC Add support for read-only db chkpt stress (#4690) Summary: Updated stress test will support testing of db in read-only mode. The user has to make sure that only read/scan operations are enabled. This PR relies on #4681. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4690 Differential Revision: D13102741 Pulled By: riversand963 fbshipit-source-id: f5a36b34db187fe12dd355f7eda161f99d6c75e4 03 January 2019, 01:40:53 UTC
ace543a Andrew Kryczka 02 January 2019, 23:05:41 UTC fix accounting for range tombstones in TableProperties (#4841) Summary: - To be consistent with the accounting of other optypes in `TableProperties`, we should count range tombstones in `TableProperties::num_entries` and `TableProperties::num_deletions`. - Updated assertions in stress test's `OnTableFileCreated` handler to accept files with range tombstones only. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4841 Differential Revision: D13568424 Pulled By: ajkr fbshipit-source-id: 0139d7806494eda20ece67ec460d2458dbbf6026 02 January 2019, 23:08:53 UTC
e8210e4 Tongliang Liao 02 January 2019, 20:47:04 UTC Make FindZLIB consistent with official definitions (#4823) Summary: CMake 3 already has FindZLIB. [https://cmake.org/cmake/help/v3.13/module/FindZLIB.html](https://cmake.org/cmake/help/v3.13/module/FindZLIB.html) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4823 Differential Revision: D13567653 Pulled By: ajkr fbshipit-source-id: e424aac1e5d9af4ee0d293896faedf7c712f7734 02 January 2019, 20:49:57 UTC
b9d6ecc Anand Ananthabhotla 02 January 2019, 19:40:12 UTC Lock free MultiGet (#4754) Summary: Avoid locking the DB mutex in order to reference SuperVersions. Instead, we get the thread local cached SuperVersion for each column family in the list. It depends on finding a sequence number that overlaps with all the open memtables. We start with the latest published sequence number, and if any of the memtables is sealed before we can get all the SuperVersions, the process is repeated. After a few times, give up and lock the DB mutex. Tests: 1. Unit tests 2. make check 3. db_bench - TEST_TMPDIR=/dev/shm ./db_bench -use_existing_db=true -benchmarks=readrandom -write_buffer_size=4194304 -target_file_size_base=4194304 -max_bytes_for_level_base=16777216 -num=5000000 -reads=1000000 -threads=32 -compression_type=none -cache_size=1048576000 -batch_size=1 -bloom_bits=1 readrandom : 0.167 micros/op 5983920 ops/sec; 426.2 MB/s (1000000 of 1000000 found) Multireadrandom with batch size 1: multireadrandom : 0.176 micros/op 5684033 ops/sec; (1000000 of 1000000 found) Pull Request resolved: https://github.com/facebook/rocksdb/pull/4754 Differential Revision: D13363550 Pulled By: anand1976 fbshipit-source-id: 6243e8de7dbd9c8bb490a8eca385da0c855b1dd4 02 January 2019, 19:42:54 UTC
7d65bd5 Faustin Lammler 02 January 2019, 19:15:01 UTC Fix spelling errors (#4827) Summary: Hi, Lintian, the Debian package checker complains about spelling error (spelling-error-in-binary). See https://salsa.debian.org/mariadb-team/mariadb-10.3/-/jobs/98380 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4827 Differential Revision: D13566362 Pulled By: riversand963 fbshipit-source-id: cd4e9212133c73b0591030de6cdedaa47575968d 02 January 2019, 19:17:57 UTC
f8d5c1b Tongliang Liao 02 January 2019, 18:38:53 UTC Import library for DLL (i.e. rocksdb-shared.lib) requires ARCHIVE DESTINATION. (#4832) Summary: `rocksdb-shared.lib` is missing while `rocksdb.lib` and `rocksdb-shared.dll` are installed correctly. Add `ARCHIVE DESTINATION` to fix this issue. Refer to CMake doc for more details: [ https://cmake.org/cmake/help/v3.13/command/install.html#installing-targets](https://cmake.org/cmake/help/v3.13/command/install.html#installing-targets) > ARCHIVE > Static libraries are treated as ARCHIVE targets, except those marked with the FRAMEWORK property on macOS (see FRAMEWORK below.) For DLL platforms (all Windows-based systems including Cygwin), the DLL import library is treated as an ARCHIVE target. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4832 Differential Revision: D13566301 Pulled By: siying fbshipit-source-id: 56e4ef82f7d5c63bd181ddf23b691336ad77881a 02 January 2019, 18:41:45 UTC
ec68091 Yanqin Jin 02 January 2019, 17:56:39 UTC Remove an unused parameter (#4816) Summary: The `flush_reason` parameter in `DBImpl::InstallSuperVersionAndScheduleWork` is not used. Remove it. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4816 Differential Revision: D13543218 Pulled By: riversand963 fbshipit-source-id: 8fc75d49462ce092e85aef0fe0c50936140db153 02 January 2019, 17:59:13 UTC
f0dda35 Siying Dong 29 December 2018, 02:00:00 UTC Preload some files even if options.max_open_files (#3340) Summary: Choose to preload some files if options.max_open_files != -1. This can slightly narrow the gap of performance between options.max_open_files is -1 and a large number. To avoid a significant regression to DB reopen speed if options.max_open_files != -1. Limit the files to preload in DB open time to 16. Pull Request resolved: https://github.com/facebook/rocksdb/pull/3340 Differential Revision: D6686945 Pulled By: siying fbshipit-source-id: 8ec11bbdb46e3d0cdee7b6ad5897a09c5a07869f 29 December 2018, 02:02:28 UTC
46e3209 Burton Li 26 December 2018, 21:56:56 UTC Compaction limiter miscs (#4795) Summary: 1. Remove unused API SubtractCompactionTask(). 2. Assert outstanding tasks drop to zero in ConcurrentTaskLimiterImpl destructor. 3. Remove GetOutstandingTask() check from manual compaction test, as TEST_WaitForCompact() doesn't synced with 'delete prepicked_compaction' in DBImpl::BGWorkCompaction(), which may make the test flaky. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4795 Differential Revision: D13542183 Pulled By: siying fbshipit-source-id: 5eb2a47e62efe4126937149aa0df6e243ebefc33 26 December 2018, 21:59:35 UTC
b1288cd Max 26 December 2018, 17:41:19 UTC Fix typos in comments (#4819) Summary: Fix some typos in comments. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4819 Differential Revision: D13548543 Pulled By: siying fbshipit-source-id: ca2e128fa47bef32892fc3627a7541fd9e2d5c3f 26 December 2018, 17:43:56 UTC
ca32fc1 faust 21 December 2018, 22:52:03 UTC Insane line length detected (#4813) Summary: Lintian, the Debian package checker complains about insane-line-length-in-source-file. Line length is 278 characters (>256) Please see here the error: https://salsa.debian.org/mariadb-team/mariadb-10.3/-/jobs/95739 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4813 Differential Revision: D13539183 Pulled By: miasantreble fbshipit-source-id: 28ad31d1bf23a076b9e4fc9ff62fb0b4c63a65f6 21 December 2018, 22:54:34 UTC
80bf897 Alexander Zinoviev 21 December 2018, 21:15:47 UTC Add a new per level counter for block cache hit (#4796) Summary: Add a new per level counter for block cache hits, increase it by one on every successful attempt to get an entry from cache. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4796 Differential Revision: D13513688 Pulled By: zinoale fbshipit-source-id: 104df038f1232e3356e162eb2d8ca138e34a8281 21 December 2018, 21:20:05 UTC
e0be1bc Andrew Kryczka 21 December 2018, 05:57:18 UTC fix DeleteRange memory leak for mmap and block cache (#4810) Summary: Previously we were cleaning up range tombstone meta-block by calling `ReleaseCachedEntry`, which wouldn't work if `value != nullptr && cache_handle == nullptr`. This happened at least in the case with mmap reads and block cache both enabled. I noticed `NewDataBlockIterator` intends to handle all these cases, so migrated to that instead of `NewUnfragmentedRangeTombstoneIterator`. Also changed the table-opening logic to fail on `ReadRangeDelBlock` failure, since that can cause data corruption. Added a test case to verify this behavior. Note the test case does not fail on `TryReopen` because failure to preload table handlers is not considered critical. However, it does fail on any read involving that file since it cannot return correct data. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4810 Differential Revision: D13534296 Pulled By: ajkr fbshipit-source-id: 55dde1111717cea6ec4bf38418daab81ccef3599 21 December 2018, 05:59:49 UTC
da1c64b Siying Dong 20 December 2018, 20:00:40 UTC Introduce a CPU time counter in perf_context (#4741) Summary: Introduce the first CPU timing counter, perf_context.get_cpu_nanos. This opens a door to more CPU counters in the future. Only Posix Env has it implemented using clock_gettime() with CLOCK_THREAD_CPUTIME_ID. How accurate the counter is depends on the platform. Make PerfStepTimer to take an Env as an argument, and sometimes pass it in. The direct reason is to make the unit tests to use SpecialEnv where we can ingest logic there. But in long term, this is a good change. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4741 Differential Revision: D13287798 Pulled By: siying fbshipit-source-id: 090361049d9d5095d1d1a369fe1338d2e2e1c73f 20 December 2018, 20:03:44 UTC
02bfc58 Abhishek Madan 20 December 2018, 01:18:44 UTC Change is_range_del_table_empty_ flag to atomic (#4801) Summary: To avoid a race on the flag, make it an atomic_bool. This doesn't seem to significantly affect benchmarks. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4801 Differential Revision: D13523845 Pulled By: abhimadan fbshipit-source-id: 3bc29f53c50a4e06cd9f8c6232a4bb221868e055 20 December 2018, 01:21:14 UTC
8bf7320 Abhishek Madan 19 December 2018, 23:43:08 UTC Remove stale TODO (#4800) Summary: This TODO was already addressed, but I forgot to remove it before landing the PR it came from. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4800 Differential Revision: D13522284 Pulled By: abhimadan fbshipit-source-id: 7766bc4f5b54e47d355cf26137ef5e86c604472a 19 December 2018, 23:45:37 UTC
71a69d9 Jakub Tomanik 19 December 2018, 22:01:29 UTC Fix building RocksDB for iOS (#4687) Summary: This PR contains the following fixes: 1. Fixing Makefile to support non-default locations of developer tools 2. Fixing compile error using a patch from https://github.com/facebook/rocksdb/pull/4007 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4687 Differential Revision: D13287263 Pulled By: riversand963 fbshipit-source-id: 4525eb42ba7b6f82af5f9bfb8e52fa4024e27ccc 19 December 2018, 22:13:55 UTC
1b0c9ce Adam Retter 19 December 2018, 21:24:05 UTC Fix Windows broken build error due to non-const override (#4798) Summary: 1) `transaction_base.h` overrides from `transaction.h` with a `const boolean do_validate`. The non-const base declaration, which I cannot see the need for, causes a compilation error on Microsoft Windows. 2) Implicit cast from `double` to `uint64_t` causes a compilation error on Microsoft Windows. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4798 Differential Revision: D13519734 Pulled By: sagar0 fbshipit-source-id: 6e8cb80e9a589b1122e1500c21b8e3a3a472b459 19 December 2018, 21:29:51 UTC
257b458 Adam Retter 19 December 2018, 04:21:22 UTC Update the version of the dependencies used by the RocksJava static build (#4761) Summary: Note that Snappy now requires CMake to build it, so I added a note about RocksJava to the README.md file. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4761 Differential Revision: D13403811 Pulled By: ajkr fbshipit-source-id: 8fcd7e3dc7f7152080364a374d3065472f417eff 19 December 2018, 04:25:43 UTC
671a7eb Yanqin Jin 19 December 2018, 00:43:12 UTC Avoid switching empty memtable in certain cases (#4792) Summary: in certain cases, we do not perform memtable switching if the active memtable of the column family is empty. Two exceptions: 1. In manual flush, if cached_recoverable_state_empty_ is false, then we need to switch memtable due to requirement of transaction. 2. In switch WAL, we need to switch memtable anyway because we have to seal the memtable if the WAL on which it depends will be closed. This change can potentially delay the occurence of write stalls because number of memtables increase more slowly. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4792 Differential Revision: D13499501 Pulled By: riversand963 fbshipit-source-id: 91c9b17ae753578578039f3851667d93610005e1 19 December 2018, 00:47:23 UTC
c15df15 Abhishek Madan 18 December 2018, 22:10:31 UTC Fix unused member compile error Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4793 Differential Revision: D13509363 Pulled By: abhimadan fbshipit-source-id: 530b4765e3335d6ecd016bfaa89645f8aa98c61f 18 December 2018, 22:28:42 UTC
68d949b Andrew Kryczka 18 December 2018, 21:30:56 UTC Enable DeleteRange in stress/crash tests (#4483) Summary: Set `delrangepercent=1` when `test_batches_snapshots=false`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4483 Differential Revision: D10324361 Pulled By: ajkr fbshipit-source-id: 0cde1f1504f9493408a0c6493b976d7e5f5b2d23 18 December 2018, 21:42:49 UTC
81b6b09 Abhishek Madan 18 December 2018, 01:26:56 UTC Remove v1 RangeDelAggregator (#4778) Summary: Now that v2 is fully functional, the v1 aggregator is removed. The v2 aggregator has been renamed. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4778 Differential Revision: D13495930 Pulled By: abhimadan fbshipit-source-id: 9d69500a60a283e79b6c4fa938fc68a8aa4d40d6 18 December 2018, 01:33:46 UTC
311cd8c Fosco Marotto 18 December 2018, 00:27:08 UTC Updated benchmark script (#4134) Summary: When producing the updated performance on flash results for the wiki, these are the updates which were made. https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks Pull Request resolved: https://github.com/facebook/rocksdb/pull/4134 Differential Revision: D13491052 Pulled By: gfosco fbshipit-source-id: dcd92f24659e0917cb1ac54a4446aa8e7aac8b0d 18 December 2018, 00:34:30 UTC
75a333a Adam Retter 18 December 2018, 00:27:00 UTC Fix build on macOS when jemalloc is installed via HomeBrew (#4767) Summary: Closes https://github.com/facebook/rocksdb/issues/2954 This doesn't necessarily mean that Rocks runs correctly with jemalloc on macOS... but at least now the build completes! Pull Request resolved: https://github.com/facebook/rocksdb/pull/4767 Differential Revision: D13495195 Pulled By: sagar0 fbshipit-source-id: d3fda0d6358a93e9803d215e3406f117be367c42 18 December 2018, 00:30:20 UTC
a62c662 Roman Zeyde 17 December 2018, 21:48:53 UTC Support setting options on column families via C bindings (#4785) Summary: Currently, it supports setting options only on the default column family. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4785 Differential Revision: D13491819 Pulled By: ajkr fbshipit-source-id: 75c78bd86222bb05568e538562af84fb53eb4d8d 17 December 2018, 21:52:12 UTC
84001cf Adam Retter 17 December 2018, 21:27:26 UTC Cache dependencies for static build of RocksJava (#4769) Summary: Avoids re-downloading the .tar.gz files for the static build of RocksJava if they are already present. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4769 Differential Revision: D13491919 Pulled By: ajkr fbshipit-source-id: 9265f577e049838dc40335d54f1ff2b4f972c38c 17 December 2018, 21:32:24 UTC
abf931a Abhishek Madan 17 December 2018, 21:12:22 UTC Add compaction logic to RangeDelAggregatorV2 (#4758) Summary: RangeDelAggregatorV2 now supports ShouldDelete calls on snapshot stripes and creation of range tombstone compaction iterators. RangeDelAggregator is no longer used on any non-test code path, and will be removed in a future commit. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4758 Differential Revision: D13439254 Pulled By: abhimadan fbshipit-source-id: fe105bcf8e3d4a2df37a622d5510843cd71b0401 17 December 2018, 21:20:51 UTC
a914a1c Adam Singer 14 December 2018, 22:18:17 UTC Add `getMin`, `getMax`, `getCount`, `getSum` to `HistogramData` class object. (#4742) Summary: Expose common stats min,max,count,sum via statistics JNI. These stats are not fully exposed on the Java side as is, but are available on the native side. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4742 Differential Revision: D13403766 Pulled By: ajkr fbshipit-source-id: 5b70f7bd3fb7490aab73dcbd09f13490fce5c773 14 December 2018, 22:28:44 UTC
4ed3c1e Maysam Yabandeh 14 December 2018, 21:45:21 UTC Fix flaky test DeleteFileRange (#4784) Summary: The test fails sporadically expecting the DB to be empty after DeleteFilesInRange(..., nullptr, nullptr) call which is not. Debugging shows cases where the files are skipped since they are being compacted. The patch fixes the test by waiting for the last CompactRange to finish before calling DeleteFilesInRange. Verified by ``` ~/gtest-parallel/gtest-parallel ./db_compaction_test --gtest_filter=DBCompactionTest.DeleteFileRange --repeat=10000 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4784 Differential Revision: D13469402 Pulled By: maysamyabandeh fbshipit-source-id: 3d8f44abe205b82c69f01e7edf27e1f8098248e1 14 December 2018, 21:47:36 UTC
d6dfe51 Adam Singer 14 December 2018, 19:34:29 UTC Synchronize ticker and histogram metrics for Java API (#4733) Summary: Updating the `HistogramType.java` and `TickerType.java` to expose and correct metrics for statistics callbacks. Moved `NO_ITERATOR_CREATED` to the proper stat name and deprecated `NO_ITERATORS`. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4733 Differential Revision: D13466936 Pulled By: sagar0 fbshipit-source-id: a58d1edcc07c7b68c3525b1aa05828212c89c6c7 14 December 2018, 19:37:05 UTC
8d2b74d Andrew Kryczka 14 December 2018, 06:05:59 UTC Refine db_stress params for atomic flush (#4781) Summary: Separate flag for enabling option from flag for enabling dedicated atomic stress test. I have found setting the former without setting the latter can detect different problems. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4781 Differential Revision: D13463211 Pulled By: ajkr fbshipit-source-id: 054f777885b2dc7d5ea99faafa21d6537eee45fd 14 December 2018, 06:10:38 UTC
3495423 Maysam Yabandeh 14 December 2018, 03:22:07 UTC Fix race condition on options_file_number_ (#4780) Summary: options_file_number_ must be written under db::mutex_ sine its read is protected by mutex_ in ::GetLiveFiles(). However currently it is written in ::RenameTempFileToOptionsFile() which according to its contract must be called without holding db::mutex_. The patch fixes the race condition by also acquitting the mutex_ before writing options_file_number_. Also it does that only if the rename of option file is successful. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4780 Differential Revision: D13461411 Pulled By: maysamyabandeh fbshipit-source-id: 2d5bae96a1f3e969ef2505b737cf2d7ae749787b 14 December 2018, 03:27:38 UTC
4fce44f Yanqin Jin 13 December 2018, 23:10:16 UTC Improve flushing multiple column families (#4708) Summary: If one column family is dropped, we should simply skip it and continue to flush other active ones. Currently we use Status::ShutdownInProgress to notify caller of column families being dropped. In the future, we should consider using a different Status code. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4708 Differential Revision: D13378954 Pulled By: riversand963 fbshipit-source-id: 42f248cdf2d32d4c0f677cd39012694b8f1328ca 13 December 2018, 23:12:40 UTC
67e5b54 Maysam Yabandeh 13 December 2018, 22:44:41 UTC Reduce runtime of compact_on_deletion_collector_test (#4779) Summary: It sometimes times out with it is run with TSAN. The patch reduces the iteration from 50 to 30. This reduces the normal runtime from 5.2 to 3.1 seconds and should similarly address the TSAN timeout problem. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4779 Differential Revision: D13456862 Pulled By: maysamyabandeh fbshipit-source-id: fdc0ad7d781b1c33b771d2415ff5fa2f1b5e2537 13 December 2018, 22:47:08 UTC
2670fe8 DorianZheng 13 December 2018, 22:12:02 UTC Get `CompactionJobInfo` from CompactFiles Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4716 Differential Revision: D13207677 Pulled By: ajkr fbshipit-source-id: d0ccf5a66df6cbb07288b0c5ebad81fd9df3926b 13 December 2018, 22:21:24 UTC
a8b9891 Burton Li 13 December 2018, 21:16:04 UTC Concurrent task limiter for compaction thread control (#4332) Summary: The PR is targeting to resolve the issue of: https://github.com/facebook/rocksdb/issues/3972#issue-330771918 We have a rocksdb created with leveled-compaction with multiple column families (CFs), some of CFs are using HDD to store big and less frequently accessed data and others are using SSD. When there are continuously write traffics going on to all CFs, the compaction thread pool is mostly occupied by those slow HDD compactions, which blocks fully utilize SSD bandwidth. Since atomic write and transaction is needed across CFs, so splitting it to multiple rocksdb instance is not an option for us. With the compaction thread control, we got 30%+ HDD write throughput gain, and also a lot smooth SSD write since less write stall happening. ConcurrentTaskLimiter can be shared with multi-CFs across rocksdb instances, so the feature does not only work for multi-CFs scenarios, but also for multi-rocksdbs scenarios, who need disk IO resource control per tenant. The usage is straight forward: e.g.: // // Enable compaction thread limiter thru ColumnFamilyOptions // std::shared_ptr<ConcurrentTaskLimiter> ctl(NewConcurrentTaskLimiter("foo_limiter", 4)); Options options; ColumnFamilyOptions cf_opt(options); cf_opt.compaction_thread_limiter = ctl; ... // // Compaction thread limiter can be tuned or disabled on-the-fly // ctl->SetMaxOutstandingTask(12); // enlarge to 12 tasks ... ctl->ResetMaxOutstandingTask(); // disable (bypass) thread limiter ctl->SetMaxOutstandingTask(-1); // Same as above ... ctl->SetMaxOutstandingTask(0); // full throttle (0 task) // // Sharing compaction thread limiter among CFs (to resolve multiple storage perf issue) // std::shared_ptr<ConcurrentTaskLimiter> ctl_ssd(NewConcurrentTaskLimiter("ssd_limiter", 8)); std::shared_ptr<ConcurrentTaskLimiter> ctl_hdd(NewConcurrentTaskLimiter("hdd_limiter", 4)); Options options; ColumnFamilyOptions cf_opt_ssd1(options); ColumnFamilyOptions cf_opt_ssd2(options); ColumnFamilyOptions cf_opt_hdd1(options); ColumnFamilyOptions cf_opt_hdd2(options); ColumnFamilyOptions cf_opt_hdd3(options); // SSD CFs cf_opt_ssd1.compaction_thread_limiter = ctl_ssd; cf_opt_ssd2.compaction_thread_limiter = ctl_ssd; // HDD CFs cf_opt_hdd1.compaction_thread_limiter = ctl_hdd; cf_opt_hdd2.compaction_thread_limiter = ctl_hdd; cf_opt_hdd3.compaction_thread_limiter = ctl_hdd; ... // // The limiter is disabled by default (or set to nullptr explicitly) // Options options; ColumnFamilyOptions cf_opt(options); cf_opt.compaction_thread_limiter = nullptr; Pull Request resolved: https://github.com/facebook/rocksdb/pull/4332 Differential Revision: D13226590 Pulled By: siying fbshipit-source-id: 14307aec55b8bd59c8223d04aa6db3c03d1b0c1d 13 December 2018, 21:18:28 UTC
0aa17c1 Maysam Yabandeh 13 December 2018, 00:27:37 UTC Fix flaky test DBCompactionTest::DeleteFileRange (#4776) Summary: The test has been failing sporadically probably because the configured compaction options were actually unused. Verified that by the following: ``` ~/gtest-parallel/gtest-parallel ./db_compaction_test --gtest_filter=DBCompactionTest.DeleteFileRange --repeat=1000 ``` Pull Request resolved: https://github.com/facebook/rocksdb/pull/4776 Differential Revision: D13441052 Pulled By: maysamyabandeh fbshipit-source-id: d35075b9e6cef9b9c9d0d571f9cd72ade8eda55d 13 December 2018, 00:32:14 UTC
4862720 DorianZheng 12 December 2018, 04:25:45 UTC Expose column family id to `FlushJobInfo` Summary: Pull Request resolved: https://github.com/facebook/rocksdb/pull/4772 Differential Revision: D13428923 Pulled By: ajkr fbshipit-source-id: e351e9c5eea97816db25429e129357a8af90712a 12 December 2018, 04:33:42 UTC
ae25546 Siying Dong 11 December 2018, 21:51:08 UTC Direct I/O Close() shouldn't rewrite the last block (#4771) Summary: In Direct I/O case, WritableFileWriter::Close() rewrites the last block again, even if there is nothing new. The reason is that, Close() flushes the buffer. For non-direct I/O case, the buffer is empty in this case so it is a no-op. However, in direct I/O case, the partial data in the last block is kept in the buffer because it needs to be rewritten for the next write. This piece of data is flushed again. This commit fixes it by skipping this write out if `pending_sync_` flag shows that there isn't new data sync last sync. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4771 Differential Revision: D13420426 Pulled By: siying fbshipit-source-id: 9d39ec9a215b1425d4ed40d85e0eba1f5daa75c6 11 December 2018, 21:55:02 UTC
49666d7 Tathagata Das 11 December 2018, 19:51:58 UTC Fix swallowing of exception in Java RocksDB when loading native library (#4728) Summary: This PR fixes #4721. When an exception is caught and thrown as a different exception, then the original exception should be inserted as a cause of the new exception. This bug in RocksDB was swallowing the underlying exception from `NativeLibraryLoader` and throwing the following exception ``` ... Caused by: java.lang.RuntimeException: Unable to load the RocksDB shared libraryjava.nio.channels.ClosedByInterruptException at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:67) at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35) ... 73 more ``` The fix is simple and self-explanatory. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4728 Differential Revision: D13418371 Pulled By: sagar0 fbshipit-source-id: d76c25af2a83a0f8ba62cc8d7b721bfddc85fdf1 11 December 2018, 20:18:44 UTC
cad248f Abhishek Madan 11 December 2018, 19:44:24 UTC Prepare FragmentedRangeTombstoneIterator for use in compaction (#4740) Summary: To support the flush/compaction use cases of RangeDelAggregator in v2, FragmentedRangeTombstoneIterator now supports dropping tombstones that cannot be read in the compaction output file. Furthermore, FragmentedRangeTombstoneIterator supports the "snapshot striping" use case by allowing an iterator to be split by a list of snapshots. RangeDelAggregatorV2 will use these changes in a follow-up change. In the process of making these changes, other miscellaneous cleanups were also done in these files. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4740 Differential Revision: D13287382 Pulled By: abhimadan fbshipit-source-id: f5aeb03e1b3058049b80c02a558ee48f723fa48c 11 December 2018, 20:10:48 UTC
d3daa0d Adam Retter 11 December 2018, 19:04:39 UTC RocksJava must compile on JDK7 (#4768) Summary: Fixes some RocksJava regressions recently introduced, whereby RocksJava would not build on JDK 7. These should have been visible on Travis-CI! Pull Request resolved: https://github.com/facebook/rocksdb/pull/4768 Differential Revision: D13418173 Pulled By: sagar0 fbshipit-source-id: 57bf223188887f84d9e072031af2e0d2c8a69c30 11 December 2018, 19:40:23 UTC
dde3ef1 Sagar Vemuri 11 December 2018, 18:07:12 UTC Change directory where ExternalSSTFileBasicTest runs (#4766) Summary: Change the directory where ExternalSSTFileBasicTest* tests run. **Problem:** Without this change, I spent considerable time chasing around a non-existent issue as ExternalSSTFileTest.* and ExternalSSTFileBasicTest.* create similar directories. Pull Request resolved: https://github.com/facebook/rocksdb/pull/4766 Differential Revision: D13409384 Pulled By: sagar0 fbshipit-source-id: c33e1f4d505dfa6efbc788d6c57cdb680053ded3 11 December 2018, 18:21:37 UTC
f8943ec Adam Retter 11 December 2018, 17:15:37 UTC Fix issues with RocksJava dropColumnFamily (#4770) Summary: Closes https://github.com/facebook/rocksdb/issues/4409 Pull Request resolved: https://github.com/facebook/rocksdb/pull/4770 Differential Revision: D13416802 Pulled By: ajkr fbshipit-source-id: 8a351e9b80dc9eeb6073467fbc67cd2f544917b0 11 December 2018, 17:17:57 UTC
back to top