| 2712d4f | Min Wei | 16 December 2016, 02:11:09 UTC | fix batchresult handle leak remove asserts in AbstractWriteBatch due to WriteBatch usage pattern change | 26 December 2016, 17:31:21 UTC |
| 4e07b08 | Sage Weil | 24 December 2016, 03:03:53 UTC | include/rocksdb/utilities/env_librados: fix typo Summary: Broken by 972f96b3fbae1a4675043bdf4279c9072ad69645 Signed-off-by: Sage Weil <sage@redhat.com> Closes https://github.com/facebook/rocksdb/pull/1719 Differential Revision: D4366123 Pulled By: IslamAbdelRahman fbshipit-source-id: a11e535 | 24 December 2016, 03:09:14 UTC |
| ab48c16 | Yi Wu | 22 December 2016, 22:44:01 UTC | Print cache options to info log Summary: Improve cache options logging to info log. Also print the value of cache_index_and_filter_blocks_with_high_priority. Closes https://github.com/facebook/rocksdb/pull/1709 Differential Revision: D4358776 Pulled By: yiwu-arbug fbshipit-source-id: 8f030a0 | 22 December 2016, 22:54:19 UTC |
| 972f96b | Aaron Gao | 22 December 2016, 20:51:29 UTC | direct io write support Summary: rocksdb direct io support ``` [gzh@dev11575.prn2 ~/rocksdb] ./db_bench -benchmarks=fillseq --num=1000000 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 5.0 Date: Wed Nov 23 13:17:43 2016 CPU: 40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz CPUCache: 25600 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 4.393 micros/op 227639 ops/sec; 25.2 MB/s [gzh@dev11575.prn2 ~/roc Closes https://github.com/facebook/rocksdb/pull/1564 Differential Revision: D4241093 Pulled By: lightmark fbshipit-source-id: 98c29e3 | 22 December 2016, 21:09:19 UTC |
| 989e644 | Islam AbdelRahman | 22 December 2016, 01:35:00 UTC | Remove sst_file_manager option from LITE Summary: Remove sst_file_manager option from LITE Closes https://github.com/facebook/rocksdb/pull/1690 Differential Revision: D4341331 Pulled By: IslamAbdelRahman fbshipit-source-id: 9f9328d | 22 December 2016, 01:54:21 UTC |
| 1beef65 | Islam AbdelRahman | 22 December 2016, 01:32:24 UTC | Fix c_test Summary: addfile phase in c_test could fail because in previous steps we did a DeleteRange. Fix the test by simply moving the addfile phase before DeleteRange Closes https://github.com/facebook/rocksdb/pull/1672 Differential Revision: D4328896 Pulled By: IslamAbdelRahman fbshipit-source-id: 1d946df | 22 December 2016, 01:39:14 UTC |
| 3d69282 | Siying Dong | 21 December 2016, 21:29:26 UTC | persistent_cache: fix two timer Summary: In persistent_cache/block_cache_tier.cc, timers are never restarted, so the latency measured is not correct. Closes https://github.com/facebook/rocksdb/pull/1707 Differential Revision: D4355828 Pulled By: siying fbshipit-source-id: cd5f9e1 | 21 December 2016, 21:39:16 UTC |
| 046099c | ivan | 21 December 2016, 01:17:43 UTC | The array is malloced by backtrace_symbols(), and must be freed Summary: The address of the array of string pointers is returned as the function result of backtrace_symbols(). This array is malloced by backtrace_symbols(), and must be freed by the caller. Closes https://github.com/facebook/rocksdb/pull/1692 Differential Revision: D4355737 Pulled By: IslamAbdelRahman fbshipit-source-id: 5742035 | 21 December 2016, 01:24:12 UTC |
| 6ff2c8d | Yi Wu | 21 December 2016, 00:47:39 UTC | Remove gflags as travis build dependency Summary: apt-get fail to fetch gflags and seems it is in fact not needed. Closes https://github.com/facebook/rocksdb/pull/1705 Differential Revision: D4354555 Pulled By: yiwu-arbug fbshipit-source-id: b68fee3 | 21 December 2016, 01:09:16 UTC |
| 3cd9ed1 | Islam AbdelRahman | 20 December 2016, 18:54:38 UTC | Show sandcastle URL in phabricator Summary: We are passing a string as diff_id which make conduit call fail ``` $ echo '{"diff_id": "20982117", "name":"click here for sandcastle tests for D20982117", "link":"https://our.intern.facebook.com/intern/sandcastle/1984718793/"}' | arc call-conduit differential.updateunitresults {"error":"ERR-CONDUIT-CORE","errorMessage":"ERR-CONDUIT-CORE: Argument 1 passed to EntDiffPropertiesUpdateMutationBuilder::setDiffNumber() must be an instance of int, string given","response":null} ``` fix it by removing double quotes Closes https://github.com/facebook/rocksdb/pull/1700 Differential Revision: D4350227 Pulled By: IslamAbdelRahman fbshipit-source-id: b4504af | 20 December 2016, 19:09:12 UTC |
| 50e305d | Andrew Kryczka | 20 December 2016, 00:44:30 UTC | Collapse range deletions Summary: Added a tombstone-collapsing mode to RangeDelAggregator, which eliminates overlap in the TombstoneMap. In this mode, we can check whether a tombstone covers a user key using upper_bound() (i.e., binary search). However, the tradeoff is the overhead to add tombstones is now higher, so at first I've only enabled it for range scans (compaction/flush/user iterators), where we expect a high number of calls to ShouldDelete() for the same tombstones. Point queries like Get() will still use the linear scan approach. Also in this diff I changed RangeDelAggregator's TombstoneMap to use multimap with user keys instead of map with internal keys. Callers sometimes provided ParsedInternalKey directly, from which it would've required string copying to derive an internal key Slice with which we could search the map. Closes https://github.com/facebook/rocksdb/pull/1614 Differential Revision: D4270397 Pulled By: ajkr fbshipit-source-id: 93092c7 | 20 December 2016, 00:54:12 UTC |
| 5d1457d | Yi Wu | 19 December 2016, 22:00:04 UTC | Dump persistent cache options Summary: Dump persistent cache options Closes https://github.com/facebook/rocksdb/pull/1679 Differential Revision: D4337019 Pulled By: yiwu-arbug fbshipit-source-id: 3812f8a | 19 December 2016, 22:09:12 UTC |
| 7bd725e | Siying Dong | 19 December 2016, 20:25:35 UTC | db_bench: introduce --benchmark_read_rate_limit Summary: Add the parameter in db_bench to help users to measure latency histogram with constant read rate. Closes https://github.com/facebook/rocksdb/pull/1683 Differential Revision: D4341387 Pulled By: siying fbshipit-source-id: 1b4b276 | 19 December 2016, 20:39:11 UTC |
| 2966918 | Siying Dong | 19 December 2016, 19:41:20 UTC | Update Netflix section of USERS.md Summary: The change is based on https://github.com/facebook/rocksdb/issues/1678 Closes https://github.com/facebook/rocksdb/pull/1699 Differential Revision: D4347657 Pulled By: IslamAbdelRahman fbshipit-source-id: 72177c2 | 19 December 2016, 20:09:12 UTC |
| 342370f | Daniel Black | 17 December 2016, 08:05:05 UTC | Simplify MemTable::Update Summary: As suggested by testn in #1650 The Add is at the end of the function. Having a fallthough will result in it being added twice. Closes https://github.com/facebook/rocksdb/pull/1676 Differential Revision: D4331906 Pulled By: yiwu-arbug fbshipit-source-id: 895c4a0 | 17 December 2016, 08:09:13 UTC |
| 1a136c1 | Ding Ma | 17 December 2016, 02:26:56 UTC | Expose file size Summary: add a new function to SstFileWriter that will tell the user how big is there file right now. Closes https://github.com/facebook/rocksdb/pull/1686 Differential Revision: D4338868 Pulled By: mdyuki1016 fbshipit-source-id: c1ee16a | 17 December 2016, 02:39:12 UTC |
| fbff462 | Andrew Kryczka | 17 December 2016, 01:31:15 UTC | Reduce compaction iterator status checks Summary: seems it's expensive to check status since the underlying merge iterator checks status of all its children. so only do it when it's really necessary to get the status before invoking Next(), i.e., when we're advancing to get the first key in the next file. Closes https://github.com/facebook/rocksdb/pull/1691 Differential Revision: D4343446 Pulled By: siying fbshipit-source-id: 70ab315 | 17 December 2016, 01:39:09 UTC |
| bd6cf7b | Jianpeng Ma | 16 December 2016, 21:01:37 UTC | WritableFileWriter: default buffer size equal min(64k,options.writabl? Summary: ?e_file_max_buffer_size) If we overwrite WritableFile and has a buffer which has the same function of buf_. We hope remove the cache function of WritableFileWriter. So using options.writable_file_max_buffer_size = 0 to disable cache function. Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com> Closes https://github.com/facebook/rocksdb/pull/1628 Differential Revision: D4307219 Pulled By: yiwu-arbug fbshipit-source-id: 77a6e26 | 16 December 2016, 21:09:14 UTC |
| fc0c6fd | Yi Wu | 16 December 2016, 19:19:52 UTC | "make format" format diff since last commit from master Summary: Update clang-format script to format diff since last commit from master, instead of just last commit. In our common workflow we usually endup with multiple commits for a single PR. This change make it easier to format all stacking changes. Closes https://github.com/facebook/rocksdb/pull/1684 Differential Revision: D4340597 Pulled By: yiwu-arbug fbshipit-source-id: c18949e | 16 December 2016, 19:24:18 UTC |
| 816c1e3 | Daniel Black | 16 December 2016, 19:17:26 UTC | gcc-7 requires include <functional> for std::function Summary: Fixes compile error: In file included from ./util/statistics.h:17:0, from ./util/stop_watch.h:8, from ./util/perf_step_timer.h:9, from ./util/iostats_context_imp.h:8, from ./util/posix_logger.h:27, from ./port/util_logger.h:18, from ./db/auto_roll_logger.h:15, from db/auto_roll_logger.cc:6: ./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type typedef std::function<void(void*, void*)> FoldFunc; Closes https://github.com/facebook/rocksdb/pull/1656 Differential Revision: D4318702 Pulled By: yiwu-arbug fbshipit-source-id: 8c5d17a | 16 December 2016, 19:24:18 UTC |
| c270735 | Yi Wu | 16 December 2016, 19:00:03 UTC | Iterator should be in corrupted status if merge operator return false Summary: Iterator should be in corrupted status if merge operator return false. Also add test to make sure if max_successive_merges is hit during write, data will not be lost. Closes https://github.com/facebook/rocksdb/pull/1665 Differential Revision: D4322695 Pulled By: yiwu-arbug fbshipit-source-id: b327b05 | 16 December 2016, 19:09:16 UTC |
| a8bf4d6 | Daniel Black | 16 December 2016, 04:11:10 UTC | Make format shows wrong curl command for retrieving clang-format-diff.py Summary: Previously: $ make format Makefile:104: Warning: Compiling in debug mode. Don't use the resulting binary in production build_tools/format-diff.sh You didn't have clang-format-diff.py available in your computer! You can download it by running: curl http://goo.gl/iUW1u2 Makefile:868: recipe for target 'format' failed make: *** [format] Error 128 $ curl http://goo.gl/iUW1u2 > ~/bin/clang-format-diff.py % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 276 0 276 0 0 148 0 --:--:-- 0:00:01 --:--:-- 148m $ more ~/bin/clang-format-diff.py <HTML> <HEAD> <TITLE>Moved Permanently</TITLE> </HEAD> <BODY BGCOLOR="#FFFFFF" TEXT="#000000"> <H1>Moved Permanently</H1> The document has moved <A HREF="https://raw.github.com/leaningtech/duetto-clang/master/tools/clang-format/clang-format-diff.py">here</A>. </BODY> </HTML> Closes https://github.com/facebook/rocksdb/pull/1680 Differential Revision: D4338495 Pulled By: yiwu-arbug fbshipit-source-id: e2b24d8 | 16 December 2016, 04:24:12 UTC |
| 8f5d24a | siddontang | 16 December 2016, 01:14:06 UTC | C API: support get usage and pinned_usage for cache Summary: Closes https://github.com/facebook/rocksdb/pull/1671 Differential Revision: D4327453 Pulled By: yiwu-arbug fbshipit-source-id: bcdbc65 | 16 December 2016, 01:24:17 UTC |
| 0ab6fc1 | Daniel Black | 15 December 2016, 03:20:42 UTC | Gcc-7 buffer size insufficient Summary: Bunch of commits related to insufficient buffer size. Errors in individual commits. Closes https://github.com/facebook/rocksdb/pull/1673 Differential Revision: D4332127 Pulled By: IslamAbdelRahman fbshipit-source-id: 878f73c | 15 December 2016, 03:24:26 UTC |
| b7239bf | Daniel Black | 15 December 2016, 03:15:32 UTC | Gcc 7 fallthrough Summary: hopefully the last of the gcc-7 compile errors Closes https://github.com/facebook/rocksdb/pull/1675 Differential Revision: D4332106 Pulled By: IslamAbdelRahman fbshipit-source-id: 139448c | 15 December 2016, 03:24:25 UTC |
| 477b6ea | Daniel Black | 15 December 2016, 01:05:57 UTC | std::remove_if requires <algorithm> Summary: fixes error (that occurred on gcc-7): error: util/env_basic_test.cc: In member function 'virtual rocksdb::Status rocksdb::NormalizingEnvWrapper::GetChildren(const string&, std::vector<std::__cxx11::basic_string<char> >*)': util/env_basic_test.cc:27:21: error: 'remove_if' is not a member of 'std' result->erase(std::remove_if(result->begin(), result->end(), ^~~ Closes https://github.com/facebook/rocksdb/pull/1674 Differential Revision: D4331221 Pulled By: ajkr fbshipit-source-id: 9bbdc78 | 15 December 2016, 01:09:14 UTC |
| 83f9a6f | Andrew Kryczka | 15 December 2016, 00:29:23 UTC | Fail BackupEngine::Open upon meta-file read error Summary: We used to treat any failure to read a backup's meta-file as if the backup were corrupted; however, we should distinguish corruption errors from errors in the backup Env. This fixes an issue where callers would get inconsistent results from GetBackupInfo() if they called it on an engine that encountered Env error during initialization. Now we fail Initialize() in this case so callers cannot invoke GetBackupInfo() on such engines. Closes https://github.com/facebook/rocksdb/pull/1654 Differential Revision: D4318573 Pulled By: ajkr fbshipit-source-id: f7a7c54 | 15 December 2016, 00:39:15 UTC |
| a79eae4 | Karthikeyan Radhakrishnan | 14 December 2016, 19:48:59 UTC | Add pcache documentation images Summary: Adding images that are used in persistent read cache documentation. Closes https://github.com/facebook/rocksdb/pull/1666 Differential Revision: D4328220 Pulled By: IslamAbdelRahman fbshipit-source-id: 767e0d0 | 14 December 2016, 19:54:22 UTC |
| d71e728 | Islam AbdelRahman | 14 December 2016, 19:09:50 UTC | Print user collected properties in sst_dump Summary: Include a dump of user_collected_properties in sst_dump Closes https://github.com/facebook/rocksdb/pull/1668 Differential Revision: D4325078 Pulled By: IslamAbdelRahman fbshipit-source-id: 226b6d6 | 14 December 2016, 19:24:11 UTC |
| 7004a6f | Jonathan Lee | 14 December 2016, 18:21:54 UTC | Add missing copyright header Summary: "examples/c_simple_example.c" did not have a proper copyright header. Closes https://github.com/facebook/rocksdb/pull/1670 Differential Revision: D4327445 Pulled By: yiwu-arbug fbshipit-source-id: a70389e | 14 December 2016, 18:24:13 UTC |
| 3cdfaec | Adam Retter | 14 December 2016, 18:17:52 UTC | Fixes for MSVC compilation Summary: Closes https://github.com/facebook/rocksdb/pull/1669 Differential Revision: D4327421 Pulled By: yiwu-arbug fbshipit-source-id: 661ee0b | 14 December 2016, 18:24:13 UTC |
| e097222 | Daniel Black | 14 December 2016, 02:33:21 UTC | util/logging.cc: buffer of insufficient size (gcc-7 -Werror=format-length) Summary: util/logging.cc:100:13: error: output may be truncated before the last format character [-Werror=format-length=] std::string NumberToHumanString(int64_t num) { ^~~~~~~~~~~~~~~~~~~ util/logging.cc:106:59: note: format output between 3 and 19 bytes into a destination of size 16 snprintf(buf, sizeof(buf), "%" PRIi64 "K", num / 1000); Closes https://github.com/facebook/rocksdb/pull/1653 Differential Revision: D4318687 Pulled By: yiwu-arbug fbshipit-source-id: 3a5c931 | 14 December 2016, 02:39:14 UTC |
| cfc34d7 | Daniel Black | 14 December 2016, 02:30:21 UTC | Missing break in case in DBTestBase::CurrentOptions Summary: Found by gcc-7 compile error. This appeared to be a fault as these options seems too different. Closes https://github.com/facebook/rocksdb/pull/1667 Differential Revision: D4324174 Pulled By: yiwu-arbug fbshipit-source-id: 0f65383 | 14 December 2016, 02:39:14 UTC |
| bfbcec2 | Daniel Black | 14 December 2016, 02:22:00 UTC | Gcc 7 error expansion to defined Summary: sorry if these gcc-7/clang-4 cleanups are getting tedious. Closes https://github.com/facebook/rocksdb/pull/1658 Differential Revision: D4318792 Pulled By: yiwu-arbug fbshipit-source-id: 8e85891 | 14 December 2016, 02:39:14 UTC |
| 6653e32 | Bassam Tabbara | 13 December 2016, 22:20:17 UTC | build: make it easier to pass PORTABLE Summary: currently when running a portable build we have to do the following PORTABLE=1 make ... this commit adds support for the following make PORTABLE=1 ... this might be seem subtle but it makes PORTABLE like all other makefile args and simplifies invocation from numerous build systems including things like ExternalProject_Add in cmake. Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com> Closes https://github.com/facebook/rocksdb/pull/1643 Differential Revision: D4315870 Pulled By: yiwu-arbug fbshipit-source-id: ee43755 | 13 December 2016, 22:39:17 UTC |
| 67adc93 | Daniel Black | 13 December 2016, 22:21:05 UTC | intentional fallthough (prevents gcc-7/clang-4 error) Summary: db/memtable.cc: In member function 'void rocksdb::MemTable::Update(rocksdb::SequenceNumber, const rocksdb::Slice&, const rocksdb::Slice&)': db/memtable.cc:736:11: error: this statement may fall through [-Werror=implicit-fallthrough=] } ^ db/memtable.cc:738:9: note: here default: ^~~~~~~ cc1plus: all warnings being treated as errors closes #1650 Closes https://github.com/facebook/rocksdb/pull/1655 Differential Revision: D4318696 Pulled By: yiwu-arbug fbshipit-source-id: 1a8981c | 13 December 2016, 22:39:17 UTC |
| 1a146f8 | Islam AbdelRahman | 13 December 2016, 22:03:15 UTC | break Flush wait for dropped CF Summary: In FlushJob we dont do the Flush if the CF is dropped https://github.com/facebook/rocksdb/blob/master/db/flush_job.cc#L184-L188 but inside WaitForFlushMemTable we keep waiting forever even if the CF is dropped. Closes https://github.com/facebook/rocksdb/pull/1664 Differential Revision: D4321032 Pulled By: IslamAbdelRahman fbshipit-source-id: 6e2b25d | 13 December 2016, 22:09:12 UTC |
| c3e5ee7 | Daniel Black | 13 December 2016, 21:52:00 UTC | util/histogram.cc: HistogramStat::toString buffer insufficient Summary: Increased buffer size to 1650. util/histogram.cc: In member function 'std::__cxx11::string rocksdb::HistogramStat::ToString() const': util/histogram.cc:189:13: error: '%.2f' directive output truncated writing between 4 and 313 bytes into a region of size 0 [-Werror=format-length=] std::string HistogramStat::ToString() const { ^~~~~~~~~~~~~ util/histogram.cc:205:30: note: format output between 69 and 1614 bytes into a destination of size 200 Percentile(99.99)); ^ cc1plus: all warnings being treated as errors Makefile:1521: recipe for target 'util/histogram.o' failed Closes https://github.com/facebook/rocksdb/pull/1660 Differential Revision: D4318820 Pulled By: yiwu-arbug fbshipit-source-id: 45ae6ea | 13 December 2016, 22:09:12 UTC |
| 5334d8b | Daniel Black | 13 December 2016, 21:50:22 UTC | table/block_based_table_builder.cc: intentional fallthrough - comment to match gcc pattern Summary: The gcc-7 code for parsing comments (libcpp/lex.c) didn't match the intentional fallthough in this comment. table/block_based_table_builder.cc: In member function 'void rocksdb::BlockBasedTableBuilder::WriteRawBlock(const rocksdb::Slice&, rocksdb::CompressionType, rocksdb::BlockHandle*)': table/block_based_table_builder.cc:754:22: error: this statement may fall through [-Werror=implicit-fallthrough=] assert(false); ^ table/block_based_table_builder.cc:756:7: note: here case kCRC32c: { ^~~~ cc1plus: all warnings being treated as errors Closes https://github.com/facebook/rocksdb/pull/1661 Differential Revision: D4318817 Pulled By: yiwu-arbug fbshipit-source-id: e67d171 | 13 December 2016, 21:54:15 UTC |
| 36d42e6 | Yi Wu | 13 December 2016, 19:42:42 UTC | Disable test to unblock travis build Summary: The two tests keep failing in travis. Disable them and will fix later. Closes https://github.com/facebook/rocksdb/pull/1648 Differential Revision: D4316389 Pulled By: yiwu-arbug fbshipit-source-id: 0a370e7 | 13 December 2016, 19:54:14 UTC |
| b57dd92 | siddontang | 13 December 2016, 19:08:45 UTC | C API: support writebatch delete range Summary: Seem that writebatch delete range can work now, so I add C API for later use. Btw, can we use this feature in production now? Closes https://github.com/facebook/rocksdb/pull/1647 Differential Revision: D4314534 Pulled By: ajkr fbshipit-source-id: e835165 | 13 December 2016, 19:24:18 UTC |
| 2ba59b5 | Islam AbdelRahman | 13 December 2016, 08:47:52 UTC | Disallow ingesting files into dropped CFs Summary: This PR update IngestExternalFile to return an error if we try to ingest a file into a dropped CF. Right now if IngestExternalFile want to flush a memtable, and it's ingesting a file into a dropped CF, it will wait forever since flushing is not possible for the dropped CF Closes https://github.com/facebook/rocksdb/pull/1657 Differential Revision: D4318657 Pulled By: IslamAbdelRahman fbshipit-source-id: ed6ea2b | 13 December 2016, 08:54:14 UTC |
| 1f6f7e3 | Daniel Black | 12 December 2016, 22:32:55 UTC | cast to signed char in ldb_cmd_test for ppc64le Summary: char is unsigned on power by default causing this test to fail with the FF case. ppc64 return 255 while x86 returned -1. Casting works on both platforms. Closes https://github.com/facebook/rocksdb/pull/1500 Differential Revision: D4308775 Pulled By: yiwu-arbug fbshipit-source-id: db3e6e0 | 12 December 2016, 22:39:18 UTC |
| 243975d | Andrew Kryczka | 12 December 2016, 21:18:38 UTC | More accurate error status for BackupEngine::Open Summary: Some users are assuming NotFound means the backup does not exist at the provided path, which is a reasonable assumption. We need to stop returning NotFound for system errors. Depends on #1644 Closes https://github.com/facebook/rocksdb/pull/1645 Differential Revision: D4312233 Pulled By: ajkr fbshipit-source-id: 5343c10 | 12 December 2016, 21:24:21 UTC |
| f0c509e | Andrew Kryczka | 12 December 2016, 20:38:43 UTC | Return finer-granularity status from Env::GetChildren* Summary: It'd be nice to use the error status type to distinguish between user error and system error. For example, GetChildren can fail listing a backup directory's contents either because a bad path was provided (user error) or because an operation failed, e.g., a remote storage service call failed (system error). In the former case, we want to continue and treat the backup directory as empty; in the latter case, we want to immediately propagate the error to the caller. This diff uses NotFound to indicate user error and IOError to indicate system error. Previously IOError indicated both. Closes https://github.com/facebook/rocksdb/pull/1644 Differential Revision: D4312157 Pulled By: ajkr fbshipit-source-id: 51b4f24 | 12 December 2016, 20:54:13 UTC |
| dc64f46 | Andrew Kryczka | 10 December 2016, 00:36:43 UTC | Add db_bench option for stderr logging Summary: The info log file ("LOG") is stored in the db directory by default. When the db is on a distributed env, this is unnecessarily slow. So, I added an option to db_bench to just print the info log messages to stderr. Closes https://github.com/facebook/rocksdb/pull/1641 Differential Revision: D4309348 Pulled By: ajkr fbshipit-source-id: 1e6f851 | 10 December 2016, 00:54:14 UTC |
| 2cabdb8 | Jonathan Lee | 09 December 2016, 19:39:16 UTC | Increase buffer size Summary: When compiling with GCC>=7.0.0, "db/internal_stats.cc" fails to compile as the data being written to the buffer potentially exceeds its size. This fix simply doubles the size of the buffer, thus accommodating the max possible data size. Closes https://github.com/facebook/rocksdb/pull/1635 Differential Revision: D4302162 Pulled By: yiwu-arbug fbshipit-source-id: c76ad59 | 09 December 2016, 19:54:22 UTC |
| 4a17b47 | Jonathan Lee | 09 December 2016, 19:34:32 UTC | Remove unnecessary header include Summary: Remove "util/testharness.h" from list of includes for "db/db_filesnapshot.cc", as it wasn't being used and thus caused an extraneous dependency on gtest. Closes https://github.com/facebook/rocksdb/pull/1634 Differential Revision: D4302146 Pulled By: yiwu-arbug fbshipit-source-id: e900c0b | 09 December 2016, 19:54:21 UTC |
| 8c2b921 | Mike Kolupaev | 09 December 2016, 18:23:58 UTC | Fixed a crash in debug build in flush_job.cc Summary: It was doing `&range_del_iters[0]` on an empty vector. Even though the resulting pointer is never dereferenced, it's still bad for two reasons: * the practical reason: it crashes with `std::out_of_range` exception in our debug build, * the "C++ standard lawyer" reason: it's undefined behavior because, in `std::vector` implementation, it probably "dereferences" a null pointer, which is invalid even though it doesn't actually read the pointed memory, just converts a pointer into a reference (and then flush_job.cc converts it back to pointer); nullptr references are undefined behavior. Closes https://github.com/facebook/rocksdb/pull/1612 Differential Revision: D4265625 Pulled By: al13n321 fbshipit-source-id: db26fb9 | 09 December 2016, 18:39:12 UTC |
| 20ce081 | Islam AbdelRahman | 08 December 2016, 21:30:09 UTC | Fix issue where IngestExternalFile insert blocks in block cache with g_seqno=0 Summary: When we Ingest an external file we open it to read some metadata and first/last key during doing that we insert blocks into the block cache with global_seqno = 0 If we move the file (did not copy it) into the DB, we will use these blocks with the wrong seqno in the read path Closes https://github.com/facebook/rocksdb/pull/1627 Differential Revision: D4293332 Pulled By: yiwu-arbug fbshipit-source-id: 3ce5523 | 08 December 2016, 21:39:18 UTC |
| 5241e0d | Andrew Kryczka | 08 December 2016, 18:58:39 UTC | fix db_bench argument type Summary: Closes https://github.com/facebook/rocksdb/pull/1633 Differential Revision: D4298161 Pulled By: yiwu-arbug fbshipit-source-id: 2c7af35 | 08 December 2016, 19:09:14 UTC |
| c04f6a0 | Jonathan Lee | 08 December 2016, 06:06:59 UTC | Specify shell in makefile Summary: The second variable "SHELL" simply tells make explicitly which shell to use, instead of allowing it to default to "/bin/sh", which may or may not be Bash. However, simply defining the second variable by itself causes make to throw an error concerning a circular definition, as it would be attempting to use the "shell" command while simultaneously trying to set which shell to use. Thus, the first variable "BASH_EXISTS" is defined such that make already knows about "/path/to/bash" before trying to use it to set "SHELL". A more technically correct solution would be to edit the makefile itself to make it compatible with non-bash shells (see the original Issue discussion for details). However, as it seems very few of the people working on this project were building with non-bash shells, I figured this solution would be good enough. Closes https://github.com/facebook/rocksdb/pull/1631 Differential Revision: D4295689 Pulled By: yiwu-arbug fbshipit-source-id: e4f9532 | 08 December 2016, 06:24:15 UTC |
| 45c7ce1 | zhangjinpeng1987 | 08 December 2016, 01:44:35 UTC | CompactRangeOptions C API Summary: Add C API for CompactRangeOptions. Closes https://github.com/facebook/rocksdb/pull/1596 Differential Revision: D4252339 Pulled By: yiwu-arbug fbshipit-source-id: f768f93 | 08 December 2016, 01:54:14 UTC |
| 2c2ba68 | Andrew Kryczka | 07 December 2016, 20:59:59 UTC | db_stress support for range deletions Summary: made db_stress capable of adding range deletions to its db and verifying their correctness. i'll make db_crashtest.py use this option later once the collapsing optimization (https://github.com/facebook/rocksdb/pull/1614) is committed because currently it slows down the test too much. Closes https://github.com/facebook/rocksdb/pull/1625 Differential Revision: D4293939 Pulled By: ajkr fbshipit-source-id: d3beb3a | 07 December 2016, 21:09:24 UTC |
| b821984 | Andrew Kryczka | 07 December 2016, 20:37:09 UTC | DeleteRange read path end-to-end tests Summary: Closes https://github.com/facebook/rocksdb/pull/1592 Differential Revision: D4246260 Pulled By: ajkr fbshipit-source-id: ce03fa2 | 07 December 2016, 20:54:17 UTC |
| 2f4fc53 | Artemiy Kolesnikov | 07 December 2016, 19:42:49 UTC | Compaction::IsTrivialMove relaxing Summary: IsTrivialMove returns true if no input file overlaps with output_level+1 with more than max_compaction_bytes_ bytes. Closes https://github.com/facebook/rocksdb/pull/1619 Differential Revision: D4278338 Pulled By: yiwu-arbug fbshipit-source-id: 994c001 | 07 December 2016, 19:54:11 UTC |
| 1dce75b | dhruba borthakur | 07 December 2016, 19:15:35 UTC | Update USERS.md Summary: Closes https://github.com/facebook/rocksdb/pull/1630 Differential Revision: D4293200 Pulled By: IslamAbdelRahman fbshipit-source-id: 9c81f85 | 07 December 2016, 19:24:14 UTC |
| 304b3c7 | dhruba borthakur | 07 December 2016, 19:14:29 UTC | Update USERS.md Summary: Closes https://github.com/facebook/rocksdb/pull/1629 Differential Revision: D4293183 Pulled By: IslamAbdelRahman fbshipit-source-id: 759ea92 | 07 December 2016, 19:24:14 UTC |
| fa50fff | Andrew Kryczka | 07 December 2016, 02:02:43 UTC | Option to expand range tombstones in db_bench Summary: When enabled, this option replaces range tombstones with a sequence of point tombstones covering the same range. This can be used to A/B test perf of range tombstones vs sequential point tombstones, and help us find the cross-over point, i.e., the size of the range above which range tombstones outperform point tombstones. Closes https://github.com/facebook/rocksdb/pull/1594 Differential Revision: D4246312 Pulled By: ajkr fbshipit-source-id: 3b00b23 | 07 December 2016, 02:09:14 UTC |
| c26a4d8 | Yi Wu | 06 December 2016, 22:29:26 UTC | Fix compile error in trasaction_lock_mgr.cc Summary: Fix error on mac/windows build since they don't recognize `uint`. Closes https://github.com/facebook/rocksdb/pull/1624 Differential Revision: D4287139 Pulled By: yiwu-arbug fbshipit-source-id: b7cc88f | 06 December 2016, 22:39:16 UTC |
| ed8fbdb | Islam AbdelRahman | 06 December 2016, 21:56:17 UTC | Add EventListener::OnExternalFileIngested() event Summary: Add EventListener::OnExternalFileIngested() to allow user to subscribe to external file ingestion events Closes https://github.com/facebook/rocksdb/pull/1623 Differential Revision: D4285844 Pulled By: IslamAbdelRahman fbshipit-source-id: 0b95a88 | 06 December 2016, 22:09:17 UTC |
| 2005c88 | Manuel Ung | 06 December 2016, 01:18:14 UTC | Implement non-exclusive locks Summary: This is an implementation of non-exclusive locks for pessimistic transactions. It is relatively simple and does not prevent starvation (ie. it's possible that request for exclusive access will never be granted if there are always threads holding shared access). It is done by changing `KeyLockInfo` to hold an set a transaction ids, instead of just one, and adding a flag specifying whether this lock is currently held with exclusive access or not. Some implementation notes: - Some lock diagnostic functions had to be updated to return a set of transaction ids for a given lock, eg. `GetWaitingTxn` and `GetLockStatusData`. - Deadlock detection is a bit more complicated since a transaction can now wait on multiple other transactions. A BFS is done in this case, and deadlock detection depth is now just a limit on the number of transactions we visit. - Expirable transactions do not work efficiently with shared locks at the moment, but that's okay for now. Closes https://github.com/facebook/rocksdb/pull/1573 Differential Revision: D4239097 Pulled By: lth fbshipit-source-id: da7c074 | 06 December 2016, 01:39:17 UTC |
| 0b0f235 | Yi Wu | 05 December 2016, 23:55:36 UTC | Mention IngestExternalFile changes in HISTORY.md Summary: I hit the land button too fast and didn't include the line. Closes https://github.com/facebook/rocksdb/pull/1622 Differential Revision: D4281316 Pulled By: yiwu-arbug fbshipit-source-id: c7b38e0 | 06 December 2016, 00:09:11 UTC |
| 23db48e | Yi Wu | 05 December 2016, 23:28:29 UTC | Update HISTORY.md for 5.0 branch Summary: These changes are included in the new branch-cut. Closes https://github.com/facebook/rocksdb/pull/1621 Differential Revision: D4281015 Pulled By: yiwu-arbug fbshipit-source-id: d88858b | 05 December 2016, 23:39:11 UTC |
| beb36d9 | Mike Kolupaev | 05 December 2016, 23:07:01 UTC | Fixed CompactionFilter::Decision::kRemoveAndSkipUntil Summary: Embarassingly enough, the first time I tried to use my new feature in logdevice it crashed with this assertion failure: db/pinned_iterators_manager.h:30: void rocksdb::PinnedIteratorsManager::StartPinning(): Assertion `pinning_enabled == false' failed The issue was that `pinned_iters_mgr_.StartPinning()` was called but `pinned_iters_mgr_.ReleasePinnedData()` wasn't. Closes https://github.com/facebook/rocksdb/pull/1611 Differential Revision: D4265622 Pulled By: al13n321 fbshipit-source-id: 747b10f | 05 December 2016, 23:24:11 UTC |
| 67f37cf | Islam AbdelRahman | 05 December 2016, 22:16:23 UTC | Allow user to specify a CF for SST files generated by SstFileWriter Summary: Allow user to explicitly specify that the generated file by SstFileWriter will be ingested in a specific CF. This allow us to persist the CF id in the generated file Closes https://github.com/facebook/rocksdb/pull/1615 Differential Revision: D4270422 Pulled By: IslamAbdelRahman fbshipit-source-id: 7fb954e | 05 December 2016, 22:24:16 UTC |
| 9053fe2 | Anton Safonov | 05 December 2016, 22:09:35 UTC | Made delete_obsolete_files_period_micros option dynamic Summary: Made delete_obsolete_files_period_micros option dynamic. It can be updating using DB::SetDBOptions(). Closes https://github.com/facebook/rocksdb/pull/1595 Differential Revision: D4246569 Pulled By: tonek fbshipit-source-id: d23f560 | 05 December 2016, 22:24:16 UTC |
| edde954 | Islam AbdelRahman | 02 December 2016, 02:24:13 UTC | fix clang build Summary: override is missing for FilterV2 Closes https://github.com/facebook/rocksdb/pull/1606 Differential Revision: D4263832 Pulled By: IslamAbdelRahman fbshipit-source-id: d8b337a | 02 December 2016, 02:39:10 UTC |
| 56281f3 | Yi Wu | 02 December 2016, 00:50:59 UTC | Add memtable_insert_with_hint_prefix_size option to db_bench Summary: Add memtable_insert_with_hint_prefix_size option to db_bench Closes https://github.com/facebook/rocksdb/pull/1604 Differential Revision: D4260549 Pulled By: yiwu-arbug fbshipit-source-id: cee5ef7 | 02 December 2016, 00:54:16 UTC |
| 4a21b14 | Islam AbdelRahman | 01 December 2016, 21:30:01 UTC | Cache heap::downheap() root comparison (optimize heap cmp call) Summary: Reduce number of comparisons in heap by caching which child node in the first level is smallest (left_child or right_child) So next time we can compare directly against the smallest child I see that the total number of calls to comparator drops significantly when using this optimization Before caching (~2mil key comparison for iterating the DB) ``` $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq" --db="/dev/shm/heap_opt" --use_existing_db --disable_auto_compactions --cache_size=1000000000 --perf_level=2 readseq : 0.338 micros/op 2959201 ops/sec; 327.4 MB/s user_key_comparison_count = 2000008 ``` After caching (~1mil key comparison for iterating the DB) ``` $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq" --db="/dev/shm/heap_opt" --use_existing_db --disable_auto_compactions --cache_size=1000000000 --perf_level=2 readseq : 0.309 micros/op 3236801 ops/sec; 358.1 MB/s user_key_comparison_count = 1000011 ``` It also improves Closes https://github.com/facebook/rocksdb/pull/1600 Differential Revision: D4256027 Pulled By: IslamAbdelRahman fbshipit-source-id: 76fcc66 | 01 December 2016, 21:39:14 UTC |
| e39d080 | Islam AbdelRahman | 01 December 2016, 18:02:12 UTC | Fix travis (compile for clang < 3.9) Summary: Travis fail because it uses clang 3.6 which don't recognize `__attribute__((__no_sanitize__("undefined")))` Closes https://github.com/facebook/rocksdb/pull/1601 Differential Revision: D4257175 Pulled By: IslamAbdelRahman fbshipit-source-id: fb4d1ab | 01 December 2016, 18:09:22 UTC |
| 3f407b0 | Igor Canadi | 01 December 2016, 18:00:56 UTC | Kill flashcache code in RocksDB Summary: Now that we have userspace persisted cache, we don't need flashcache anymore. Closes https://github.com/facebook/rocksdb/pull/1588 Differential Revision: D4245114 Pulled By: igorcanadi fbshipit-source-id: e2c1c72 | 01 December 2016, 18:09:22 UTC |
| b77007d | fangchenliaohui | 01 December 2016, 17:46:43 UTC | Bug: paralle_group status updated in WriteThread::CompleteParallelWorker Summary: Multi-write thread may update the status of the parallel_group in WriteThread::CompleteParallelWorker if the status of Writer is not ok! When copy write status to the paralle_group, the write thread just hold the mutex of the the writer processed by itself. it is useless. The thread should held the the leader of the parallel_group instead. Closes https://github.com/facebook/rocksdb/pull/1598 Differential Revision: D4252335 Pulled By: siying fbshipit-source-id: 3864cf7 | 01 December 2016, 17:54:11 UTC |
| 247d097 | Mike Kolupaev | 01 December 2016, 15:00:17 UTC | Support for range skips in compaction filter Summary: This adds the ability for compaction filter to say "drop this key-value, and also drop everything up to key x". This will cause the compaction to seek input iterator to x, without reading the data. This can make compaction much faster when large consecutive chunks of data are filtered out. See the changes in include/rocksdb/compaction_filter.h for the new API. Along the way this diff also adds ability for compaction filter changing merge operands, similar to how it can change values; we're not going to use this feature, it just seemed easier and cleaner to implement it than to document that it's not implemented :) The diff is not as big as it may seem, about half of the lines are a test. Closes https://github.com/facebook/rocksdb/pull/1599 Differential Revision: D4252092 Pulled By: al13n321 fbshipit-source-id: 41e1e48 | 01 December 2016, 15:09:15 UTC |
| 96fcefb | Panagiotis Ktistakis | 30 November 2016, 19:05:46 UTC | c api: expose option for dynamic level size target Summary: Closes https://github.com/facebook/rocksdb/pull/1587 Differential Revision: D4245923 Pulled By: yiwu-arbug fbshipit-source-id: 6ee7291 | 30 November 2016, 19:24:14 UTC |
| 00197cf | zhangjinpeng1987 | 30 November 2016, 19:05:33 UTC | Add C API to set base_backgroud_compactions Summary: Add C API to set base_backgroud_compactions Closes https://github.com/facebook/rocksdb/pull/1571 Differential Revision: D4245709 Pulled By: yiwu-arbug fbshipit-source-id: 792c6b8 | 30 November 2016, 19:09:13 UTC |
| 5b219ec | Andrew Kryczka | 29 November 2016, 20:10:55 UTC | deleterange end-to-end test improvements for lite/robustness Summary: Closes https://github.com/facebook/rocksdb/pull/1591 Differential Revision: D4246019 Pulled By: ajkr fbshipit-source-id: 0c4aa37 | 29 November 2016, 20:24:13 UTC |
| aad1191 | Anirban Rahut | 29 November 2016, 19:59:51 UTC | pass rocksdb oncall to mysql_mtr_filter otherwise tasks get created w… Summary: …rong owner mysql_mtr_filter script needs proper oncall Closes https://github.com/facebook/rocksdb/pull/1586 Differential Revision: D4245150 Pulled By: anirbanr-fb fbshipit-source-id: fd8577c | 29 November 2016, 20:09:12 UTC |
| e333528 | Andrew Kryczka | 29 November 2016, 18:56:05 UTC | DeleteRange write path end-to-end tests Summary: Closes https://github.com/facebook/rocksdb/pull/1578 Differential Revision: D4241171 Pulled By: ajkr fbshipit-source-id: ce5fd83 | 29 November 2016, 19:09:22 UTC |
| 7784980 | Siying Dong | 29 November 2016, 18:49:47 UTC | Fix mis-reporting of compaction read bytes to the base level Summary: In dynamic leveled compaction, when calculating read bytes, output level bytes may be wronglyl calculated as input level inputs. Fix it. Closes https://github.com/facebook/rocksdb/pull/1475 Differential Revision: D4148412 Pulled By: siying fbshipit-source-id: f2f475a | 29 November 2016, 19:09:22 UTC |
| 3c6b49e | Islam AbdelRahman | 29 November 2016, 18:40:31 UTC | Fix implicit conversion between int64_t to int Summary: Make conversion explicit, implicit conversion breaks the build Closes https://github.com/facebook/rocksdb/pull/1589 Differential Revision: D4245158 Pulled By: IslamAbdelRahman fbshipit-source-id: aaec00d | 29 November 2016, 18:54:15 UTC |
| b3b8756 | Siying Dong | 29 November 2016, 17:06:10 UTC | Remove unused assignment in db/db_iter.cc Summary: "make analyze" complains the assignment is not useful. Remove it. Closes https://github.com/facebook/rocksdb/pull/1581 Differential Revision: D4241697 Pulled By: siying fbshipit-source-id: 178f67a | 29 November 2016, 17:09:14 UTC |
| 4f6e89b | Andrew Kryczka | 29 November 2016, 06:45:35 UTC | Fix range deletion covering key in same SST file Summary: AddTombstones() needs to be before t->Get(), oops :'( Closes https://github.com/facebook/rocksdb/pull/1576 Differential Revision: D4241041 Pulled By: ajkr fbshipit-source-id: 781ceea | 29 November 2016, 06:54:13 UTC |
| a2bf265 | Islam AbdelRahman | 29 November 2016, 02:25:27 UTC | Avoid intentional overflow in GetL0ThresholdSpeedupCompaction Summary: https://github.com/facebook/rocksdb/commit/99c052a34f93d119b75eccdcd489ecd581d48ee9 fixes integer overflow in GetL0ThresholdSpeedupCompaction() by checking if int become -ve. UBSAN will complain about that since this is still an overflow, we can fix the issue by simply using int64_t Closes https://github.com/facebook/rocksdb/pull/1582 Differential Revision: D4241525 Pulled By: IslamAbdelRahman fbshipit-source-id: b3ae21f | 29 November 2016, 02:39:13 UTC |
| 52fd1ff | Islam AbdelRahman | 29 November 2016, 01:46:35 UTC | disable UBSAN for functions with intentional -ve shift / overflow Summary: disable UBSAN for functions with intentional left shift on -ve number / overflow These functions are rocksdb:: Hash FixedLengthColBufEncoder::Append FaultInjectionTest:: Key Closes https://github.com/facebook/rocksdb/pull/1577 Differential Revision: D4240801 Pulled By: IslamAbdelRahman fbshipit-source-id: 3e1caf6 | 29 November 2016, 01:54:12 UTC |
| 1886c43 | Islam AbdelRahman | 29 November 2016, 00:39:49 UTC | Fix CompactionJob::Install division by zero Summary: Fix CompactionJob::Install division by zero Closes https://github.com/facebook/rocksdb/pull/1580 Differential Revision: D4240794 Pulled By: IslamAbdelRahman fbshipit-source-id: 7286721 | 29 November 2016, 00:54:16 UTC |
| 63c30de | Islam AbdelRahman | 29 November 2016, 00:35:21 UTC | fix options_test ubsan Summary: Having -ve value for max_write_buffer_number does not make sense and cause us to do a left shift on a -ve value number Closes https://github.com/facebook/rocksdb/pull/1579 Differential Revision: D4240798 Pulled By: IslamAbdelRahman fbshipit-source-id: bd6267e | 29 November 2016, 00:39:14 UTC |
| 13e66a8 | Islam AbdelRahman | 29 November 2016, 00:31:41 UTC | Fix compaction_job.cc division by zero Summary: Fix division by zero in compaction_job.cc Closes https://github.com/facebook/rocksdb/pull/1575 Differential Revision: D4240818 Pulled By: IslamAbdelRahman fbshipit-source-id: a8bc757 | 29 November 2016, 00:39:13 UTC |
| 01eabf7 | Andrew Kryczka | 28 November 2016, 23:36:54 UTC | Fix double-counted deletion stat Summary: Both the single deletion and the value are included in compaction outputs, so no need to update the stat for the value's deletion yet, otherwise it'd be double-counted. Closes https://github.com/facebook/rocksdb/pull/1574 Differential Revision: D4241181 Pulled By: ajkr fbshipit-source-id: c9aaa15 | 28 November 2016, 23:54:12 UTC |
| 7ffb10f | Andrew Kryczka | 28 November 2016, 19:44:40 UTC | DeleteRange compaction statistics Summary: - "rocksdb.compaction.key.drop.range_del" - number of keys dropped during compaction due to a range tombstone covering them - "rocksdb.compaction.range_del.drop.obsolete" - number of range tombstones dropped due to compaction to bottom level and no snapshot saving them - s/CompactionIteratorStats/CompactionIterationStats/g since this class is no longer specific to CompactionIterator -- it's also updated for range tombstone iteration during compaction - Move the above class into a separate .h file to avoid circular dependency. Closes https://github.com/facebook/rocksdb/pull/1520 Differential Revision: D4187179 Pulled By: ajkr fbshipit-source-id: 10c2103 | 28 November 2016, 19:54:12 UTC |
| 236d4c6 | Mike Kolupaev | 28 November 2016, 18:12:28 UTC | Less linear search in DBIter::Seek() when keys are overwritten a lot Summary: In one deployment we saw high latencies (presumably from slow iterator operations) and a lot of CPU time reported by perf with this stack: ``` rocksdb::MergingIterator::Next rocksdb::DBIter::FindNextUserEntryInternal rocksdb::DBIter::Seek ``` I think what's happening is: 1. we create a snapshot iterator, 2. we do lots of Put()s for the same key x; this creates lots of entries in memtable, 3. we seek the iterator to a key slightly smaller than x, 4. the seek walks over lots of entries in memtable for key x, skipping them because of high sequence numbers. CC IslamAbdelRahman Closes https://github.com/facebook/rocksdb/pull/1413 Differential Revision: D4083879 Pulled By: IslamAbdelRahman fbshipit-source-id: a83ddae | 28 November 2016, 18:24:11 UTC |
| cd7c414 | Siying Dong | 23 November 2016, 17:19:11 UTC | Improve Write Stalling System Summary: Current write stalling system has the problem of lacking of positive feedback if the restricted rate is already too low. Users sometimes stack in very low slowdown value. With the diff, we add a positive feedback (increasing the slowdown value) if we recover from slowdown state back to normal. To avoid the positive feedback to keep the slowdown value to be to high, we add issue a negative feedback every time we are close to the stop condition. Experiments show it is easier to reach a relative balance than before. Also increase level0_stop_writes_trigger default from 24 to 32. Since level0_slowdown_writes_trigger default is 20, stop trigger 24 only gives four files as the buffer time to slowdown writes. In order to avoid stop in four files while 20 files have been accumulated, the slowdown value must be very low, which is amost the same as stop. It also doesn't give enough time for the slowdown value to converge. Increase it to 32 will smooth out the system. Closes https://github.com/facebook/rocksdb/pull/1562 Differential Revision: D4218519 Pulled By: siying fbshipit-source-id: 95e4088 | 23 November 2016, 17:24:15 UTC |
| dfb6fe6 | Yi Wu | 22 November 2016, 22:06:54 UTC | Unified InlineSkipList::Insert algorithm with hinting Summary: This PR is based on nbronson's diff with small modifications to wire it up with existing interface. Comparing to previous version, this approach works better for inserting keys in decreasing order or updating the same key, and impose less restriction to the prefix extractor. ---- Summary from original diff ---- This diff introduces a single InlineSkipList::Insert that unifies the existing sequential insert optimization (prev_), concurrent insertion, and insertion using externally-managed insertion point hints. There's a deep symmetry between insertion hints (cursors) and the concurrent algorithm. In both cases we have partial information from the recent past that is likely but not certain to be accurate. This diff introduces the struct InlineSkipList::Splice, which encodes predecessor and successor information in the same form that was previously only used within a single call to InsertConcurrently. Splice holds information about an insertion point that can be used to levera Closes https://github.com/facebook/rocksdb/pull/1561 Differential Revision: D4217283 Pulled By: yiwu-arbug fbshipit-source-id: 33ee437 | 22 November 2016, 22:09:13 UTC |
| 3068870 | Karthikeyan Radhakrishnan | 22 November 2016, 18:26:08 UTC | Making persistent cache more resilient to filesystem failures Summary: The persistent cache is designed to hop over errors and return key not found. So far, it has shown resilience to write errors, encoding errors, data corruption etc. It is not resilient against disappearing files/directories. This was exposed during testing when multiple instances of persistence cache was started sharing the same directory simulating an unpredictable filesystem environment. This patch - makes the write code path more resilient to errors while creating files - makes the read code path more resilient to handle situation where files are not found - added a test that does negative write/read testing by removing the directory while writes are in progress Closes https://github.com/facebook/rocksdb/pull/1472 Differential Revision: D4143413 Pulled By: kradhakrishnan fbshipit-source-id: fd25e9b | 22 November 2016, 18:39:10 UTC |
| 734e4ac | Andrew Kryczka | 22 November 2016, 05:08:06 UTC | Eliminate redundant cache lookup with range deletion Summary: When we introduced range deletion block, TableCache::Get() and TableCache::NewIterator() each did two table cache lookups, one for range deletion block iterator and another for getting the table reader to which the Get()/NewIterator() is delegated. This extra cache lookup was very CPU-intensive (about 10% overhead in a read-heavy benchmark). We can avoid it by reusing the Cache::Handle created for range deletion block iterator to get the file reader. Closes https://github.com/facebook/rocksdb/pull/1537 Differential Revision: D4201167 Pulled By: ajkr fbshipit-source-id: d33ffd8 | 22 November 2016, 05:24:11 UTC |
| 182b940 | Maysam Yabandeh | 22 November 2016, 01:56:19 UTC | Add WriteOptions.no_slowdown Summary: If the WriteOptions.no_slowdown flag is set AND we need to wait or sleep for the write request, then fail immediately with Status::Incomplete(). Closes https://github.com/facebook/rocksdb/pull/1527 Differential Revision: D4191405 Pulled By: maysamyabandeh fbshipit-source-id: 7f3ce3f | 22 November 2016, 02:09:13 UTC |
| 4118e13 | Karthikeyan Radhakrishnan | 22 November 2016, 01:22:01 UTC | Persistent Cache: Expose stats to user via public API Summary: Exposing persistent cache stats (counters) to the user via public API. Closes https://github.com/facebook/rocksdb/pull/1485 Differential Revision: D4155274 Pulled By: siying fbshipit-source-id: 30a9f50 | 22 November 2016, 01:39:13 UTC |
| f2a8f92 | Siying Dong | 21 November 2016, 22:45:08 UTC | rocks_lua_compaction_filter: add unused attribute to a variable Summary: Release build shows warning without this fix. Closes https://github.com/facebook/rocksdb/pull/1558 Differential Revision: D4215831 Pulled By: yiwu-arbug fbshipit-source-id: 888a755 | 21 November 2016, 22:54:14 UTC |
| 4444256 | Nick Terrell | 21 November 2016, 20:17:06 UTC | Remove use of deprecated LZ4 function Summary: LZ4 1.7.3 emits warnings when calling the deprecated function `LZ4_compress_limitedOutput_continue()`. Starting in r129, LZ4 introduces `LZ4_compress_fast_continue()` as a replacement, and the two functions calls are [exactly equivalent](https://github.com/lz4/lz4/blob/dev/lib/lz4.c#L1408). Closes https://github.com/facebook/rocksdb/pull/1532 Differential Revision: D4199240 Pulled By: siying fbshipit-source-id: 138c2bc | 21 November 2016, 20:24:14 UTC |
| 548d7fb | Changli Gao | 21 November 2016, 20:09:33 UTC | Fix fd leak when using direct IOs Summary: We should close the fd, before overriding it. This bug was introduced by f89caa127baa086cb100976b14da1a531cf0e823 Closes https://github.com/facebook/rocksdb/pull/1553 Differential Revision: D4214101 Pulled By: siying fbshipit-source-id: 0d65de0 | 21 November 2016, 20:24:13 UTC |
