| 8f339ed | Anirban Rahut | 11 January 2017, 00:01:52 UTC | Guarding extra fallocate call with TRAVIS because its not working properly on travis Summary: There is some old code in PosixWritableFile::Close(), which truncates the file to the measured size and then does an extra fallocate with KEEP_SIZE. This is commented as a failsafe because in some cases ftruncate doesn't do the right job (I don't know of an instance of this btw). However doing an fallocate with KEEP_SIZE should not increase the file size. However on Travis Worker which is Docker (likely AUFS ) its not working. There are comments on web that show that the AUFS author had initially not implemented fallocate, and then did it later. So not sure what is the quality of the implementation. Test Plan: Testing on travis and internal sandcastle Reviewers: IslamAbdelRahman | 11 January 2017, 19:43:29 UTC |
| 2172b66 | Sunpoet Po-Chuan Hsieh | 10 January 2017, 19:35:36 UTC | Fix build on FreeBSD Summary: ``` CC utilities/column_aware_encoding_exp.o utilities/column_aware_encoding_exp.cc:149:5: error: use of undeclared identifier 'exit' exit(1); ^ utilities/column_aware_encoding_exp.cc:154:5: error: use of undeclared identifier 'exit' exit(1); ^ utilities/column_aware_encoding_exp.cc:158:5: error: use of undeclared identifier 'exit' exit(1); ^ 3 errors generated. ``` Closes https://github.com/facebook/rocksdb/pull/1754 Differential Revision: D4399044 Pulled By: IslamAbdelRahman fbshipit-source-id: fbab5a2 | 10 January 2017, 19:39:12 UTC |
| 3c233ca | Dmitri Smirnov | 09 January 2017, 23:37:57 UTC | Fix Windows environment issues Summary: Enable directIO on WritableFileImpl::Append with offset being current length of the file. Enable UniqueID tests on Windows, disable others but leeting them to compile. Unique tests are valuable to detect failures on different filesystems and upcoming ReFS. Clear output in WinEnv Getchildren.This is different from previous strategy, do not touch output on failure. Make sure DBTest.OpenWhenOpen works with windows error message Closes https://github.com/facebook/rocksdb/pull/1746 Differential Revision: D4385681 Pulled By: IslamAbdelRahman fbshipit-source-id: c07b702 | 09 January 2017, 23:54:12 UTC |
| 7631734 | Maysam Yabandeh | 09 January 2017, 22:00:25 UTC | Fix the error in ColumnFamiliesTest Summary: In the test the last change to AAAZZZ in handles[1] is deleting it. The result of the get must be NotFound then. Previosuly the test did not check for the return value of Get and assumed that the status is ok. It then move ahead asserting the returned value. The passed-by-reference string value however was not changed (since the key was not found) and the asserted value is what it contained before doing the Get. Closes https://github.com/facebook/rocksdb/pull/1753 Differential Revision: D4390982 Pulled By: maysamyabandeh fbshipit-source-id: dd55a34 | 09 January 2017, 22:09:13 UTC |
| 7a02ad0 | Daniel Black | 09 January 2017, 21:45:04 UTC | Update travis to ubuntu trusty Summary: Distro gflags is now acceptable in build. Add gcc for two linux builds, JOB_NAME=unittests and lite_build. unittests will be built with g++-6. Use apt and ccache during build. Seems more up to date than precise :-) llvm repos for trusty aren't whitelisted yet (ref: https://github.com/travis-ci/apt-source-whitelist/pull/309, https://github.com/travis-ci/apt-source-whitelist/issues/332, https://github.com/travis-ci/apt-source-whitelist/issues/332) Closes https://github.com/facebook/rocksdb/pull/1685 Differential Revision: D4340378 Pulled By: yiwu-arbug fbshipit-source-id: a143b9d | 09 January 2017, 21:54:19 UTC |
| 60c509f | Siying Dong | 09 January 2017, 19:40:20 UTC | Fix valgrind failure in test CurrentFileModifiedWhileCheckpointing2PC Summary: Fix some memory leaks in the test. Also rename the test class name from DBTest to CheckpointTest to avoid confusion. Closes https://github.com/facebook/rocksdb/pull/1752 Differential Revision: D4390355 Pulled By: siying fbshipit-source-id: 0fa388a | 09 January 2017, 19:54:13 UTC |
| d0ba8ec | Maysam Yabandeh | 08 January 2017, 22:08:51 UTC | Revert "PinnableSlice" Summary: This reverts commit 54d94e9c2cc0bf6eeb2a165ada33fa9c174f0b16. The pull request was landed by mistake. Closes https://github.com/facebook/rocksdb/pull/1755 Differential Revision: D4391678 Pulled By: maysamyabandeh fbshipit-source-id: 36d5149 | 08 January 2017, 22:24:12 UTC |
| 54d94e9 | Maysam Yabandeh | 08 January 2017, 21:49:15 UTC | PinnableSlice Summary: Currently the point lookup values are copied to a string provided by the user. This incures an extra memcpy cost. This patch allows doing point lookup via a PinnableSlice which pins the source memory location (instead of copying their content) and releases them after the content is consumed by the user. The old API of Get(string) is translated to the new API underneath. Here is the summary for improvements: 1. value 100 byte: 1.8% regular, 1.2% merge values 2. value 1k byte: 11.5% regular, 7.5% merge values 3. value 10k byte: 26% regular, 29.9% merge values The improvement for merge could be more if we extend this approach to pin the merge output and delay the full merge operation until the user actually needs it. We have put that for future work. PS: Sometimes we observe a small decrease in performance when switching from t5452014 to this patch but with the old Get(string) API. The difference is a little and could be noise. More importantly it is safely cancelled Closes https://github.com/facebook/rocksdb/pull/1732 Differential Revision: D4374613 Pulled By: maysamyabandeh fbshipit-source-id: a077f1a | 08 January 2017, 21:54:13 UTC |
| e04480f | Dmitri Smirnov | 06 January 2017, 22:42:18 UTC | Fix MS warnings. Use ROCKSDB_Prsz for size_t. Summary: Closes https://github.com/facebook/rocksdb/pull/1737 Differential Revision: D4378852 Pulled By: yiwu-arbug fbshipit-source-id: ba8b02d | 07 January 2017, 00:24:14 UTC |
| c081f72 | Yi Wu | 06 January 2017, 22:41:07 UTC | 5.0.1 release blog post Summary: 5.0.1 release blog post Closes https://github.com/facebook/rocksdb/pull/1751 Differential Revision: D4388962 Pulled By: yiwu-arbug fbshipit-source-id: 59559e1 | 06 January 2017, 23:24:15 UTC |
| ac73d75 | Islam AbdelRahman | 06 January 2017, 19:11:51 UTC | Add GetSupportedCompressions() convenience function Summary: This function will return a list of supported compression types in RocksDB This is needed for MyRocks https://github.com/facebook/mysql-5.6/pull/446 Closes https://github.com/facebook/rocksdb/pull/1747 Differential Revision: D4385921 Pulled By: IslamAbdelRahman fbshipit-source-id: 2f5b59f | 06 January 2017, 19:24:14 UTC |
| b104b87 | Andrew Kryczka | 05 January 2017, 18:22:46 UTC | Maintain position in range deletions map Summary: When deletion-collapsing mode is enabled (i.e., for DBIter/CompactionIterator), we maintain position in the tombstone maps across calls to ShouldDelete(). Since iterators often access keys sequentially (or reverse-sequentially), scanning forward/backward from the last position can be faster than binary-searching the map for every key. - When Next() is invoked on an iterator, we use kForwardTraversal to scan forwards, if needed, until arriving at the range deletion containing the next key. - Similarly for Prev(), we use kBackwardTraversal to scan backwards in the range deletion map. - When the iterator seeks, we use kBinarySearch for repositioning - After tombstones are added or before the first ShouldDelete() invocation, the current position is set to invalid, which forces kBinarySearch to be used. - Non-iterator users (i.e., Get()) use kFullScan, which has the same behavior as before---scan the whole map for every key passed to ShouldDelete(). Closes https://github.com/facebook/rocksdb/pull/1701 Differential Revision: D4350318 Pulled By: ajkr fbshipit-source-id: 5129b76 | 05 January 2017, 18:39:12 UTC |
| 640d724 | Yi Wu | 04 January 2017, 02:24:15 UTC | Update db_bench and sst_dump to test with block cache mid-point inser? Summary: ?tion Add flags in db_bench to test with block cache mid-point insertion. Also update sst_dump to dump total block sizes of each type. I find it useful to look at these test db stats and I don't know if we have them elsewhere. Closes https://github.com/facebook/rocksdb/pull/1706 Differential Revision: D4355812 Pulled By: yiwu-arbug fbshipit-source-id: 3e4a348 | 04 January 2017, 02:39:14 UTC |
| 653ac1f | siddontang | 04 January 2017, 02:21:55 UTC | C API: support total_order_mode Summary: Closes https://github.com/facebook/rocksdb/pull/1687 Differential Revision: D4349210 Pulled By: IslamAbdelRahman fbshipit-source-id: 32d0fbd | 04 January 2017, 02:39:14 UTC |
| 85ac1a3 | Adam Retter | 04 January 2017, 02:17:12 UTC | Fix rocksdb::Status::getState Summary: This fixes the Java API for Status#getState use in Native code and also simplifies the implementation of rocksdb::Status::getState. Closes https://github.com/facebook/rocksdb/issues/1688 Closes https://github.com/facebook/rocksdb/pull/1714 Differential Revision: D4364181 Pulled By: yiwu-arbug fbshipit-source-id: 8e073b4 | 04 January 2017, 02:39:14 UTC |
| 76711b6 | Islam AbdelRahman | 04 January 2017, 00:09:38 UTC | Make ExternalSSTFileTest::CompactionDeadlock more deterministic Summary: It's not always true that `ASSERT_EQ(running_threads.load(), 2);` Closes https://github.com/facebook/rocksdb/pull/1736 Differential Revision: D4374091 Pulled By: IslamAbdelRahman fbshipit-source-id: 4f70bbd | 04 January 2017, 02:09:20 UTC |
| c963460 | Islam AbdelRahman | 03 January 2017, 23:42:54 UTC | Fix tests under GCC_481 Summary: This fix the issue with tests failing under GCC 481, I am not sure what is the exact reason Closes https://github.com/facebook/rocksdb/pull/1735 Differential Revision: D4374094 Pulled By: IslamAbdelRahman fbshipit-source-id: b3625bc | 04 January 2017, 01:54:12 UTC |
| 33c86d6 | Andrew Kryczka | 01 January 2017, 19:05:55 UTC | Fix backupable db test Summary: #1733 started using SizeFileBytes(), so our dummy log file implementation should stop asserting that this function isn't called. Closes https://github.com/facebook/rocksdb/pull/1740 Differential Revision: D4376055 Pulled By: ajkr fbshipit-source-id: 2854d89 | 01 January 2017, 19:24:14 UTC |
| e425ec1 | Vincent Lee | 31 December 2016, 18:35:38 UTC | utilities/backupable: backup should limit the copy size of wal. Summary: Since the backup work as snapshot, we should only copy the bytes of the wal while we get the alive files. Closes https://github.com/facebook/rocksdb/pull/1733 Differential Revision: D4373457 Pulled By: ajkr fbshipit-source-id: 389318f | 31 December 2016, 18:54:20 UTC |
| 0712d54 | Maysam Yabandeh | 29 December 2016, 23:48:24 UTC | Delegate Cleanables Summary: Cleanable objects will perform the registered cleanups when they are destructed. We however rather to delay this cleaning like when we are gathering the merge operands. Current approach is to create the Cleanable object on heap (instead of on stack) and delay deleting it. By allowing Cleanables to delegate their cleanups to another cleanable object we can delay the cleaning without however the need to craete the cleanable object on heap and keeping it around. This patch applies this technique for the cleanups of BlockIter and shows improved performance for some in-memory benchmarks: +1.8% for merge worklaod, +6.4% for non-merge workload when the merge operator is specified. https://our.intern.facebook.com/intern/tasks?t=15168163 Non-merge benchmark: TEST_TMPDIR=/dev/shm/v100nocomp/ ./db_bench --benchmarks=fillrandom --num=1000000 -value_size=100 -compression_type=none Reading random with no merge operator specified: TEST_TMPDIR=/dev/shm/v100nocomp/ ./db_bench --benchmarks="read Closes https://github.com/facebook/rocksdb/pull/1711 Differential Revision: D4361163 Pulled By: maysamyabandeh fbshipit-source-id: 9801e07 | 29 December 2016, 23:54:19 UTC |
| d58ef52 | Islam AbdelRahman | 29 December 2016, 22:51:52 UTC | Allow SstFileWriter to Fadvise the file away from page cache Summary: Add `fadvise_trigger` option to `SstFileWriter` If fadvise_trigger is passed with a non-zero value, SstFileWriter will invalidate the os page cache every `fadvise_trigger` bytes for the sst file Closes https://github.com/facebook/rocksdb/pull/1731 Differential Revision: D4371246 Pulled By: IslamAbdelRahman fbshipit-source-id: 91caff1 | 29 December 2016, 23:09:19 UTC |
| 17a4b75 | Siying Dong | 29 December 2016, 02:38:20 UTC | Always fsync the file after file copying Summary: File copying happens when creating checkpoints and bulkloading files from different FS partition. We should fsync the files when copying them to guarantee durability. A side effect will be that the dirty pages in file system buffers won't grow too large. Closes https://github.com/facebook/rocksdb/pull/1728 Differential Revision: D4371083 Pulled By: siying fbshipit-source-id: 579e14c | 29 December 2016, 03:09:16 UTC |
| 2fb70dc | Changli Gao | 29 December 2016, 02:28:49 UTC | examples: Add options_file_example to target all Summary: Closes https://github.com/facebook/rocksdb/pull/1723 Differential Revision: D4372273 Pulled By: IslamAbdelRahman fbshipit-source-id: c8c2ad0 | 29 December 2016, 03:09:16 UTC |
| a738af8 | leipeng | 29 December 2016, 02:25:50 UTC | db/pinned_iterators_manager.h: bugfix Summary: std::unique(beg, end) returns an iterator of unique_end, data behind unique_end should not be accessed. Closes https://github.com/facebook/rocksdb/pull/1726 Differential Revision: D4371076 Pulled By: IslamAbdelRahman fbshipit-source-id: 5564450 | 29 December 2016, 02:54:57 UTC |
| 906523d | Siying Dong | 28 December 2016, 23:48:10 UTC | Add description to the 2PC checkpooint bug in HISTORY.md Summary: Closes https://github.com/facebook/rocksdb/pull/1729 Differential Revision: D4371674 Pulled By: siying fbshipit-source-id: 907e373 | 28 December 2016, 23:54:14 UTC |
| 438f22b | Siying Dong | 28 December 2016, 19:53:29 UTC | Fix bug of Checkpoint loses recent transactions with 2PC Summary: If 2PC is enabled, checkpoint may not copy previous log files that contain uncommitted prepare records. In this diff we keep those files. Closes https://github.com/facebook/rocksdb/pull/1724 Differential Revision: D4368319 Pulled By: siying fbshipit-source-id: cc2c746 | 28 December 2016, 20:24:16 UTC |
| 335981d | Gunnar Kudrjavets | 28 December 2016, 06:21:34 UTC | Fix the directory path for RocksDB repo fbshipit-source-id: 285548d | 28 December 2016, 06:24:16 UTC |
| 548b628 | Gunnar Kudrjavets | 27 December 2016, 23:47:43 UTC | Enable conditionally using adaptive mutexes Summary: To support scenarios where we want all instances of `Mutex` be adaptive we're adding a conditional `#define` so that the desired behavior can be easily enabled. Closes https://github.com/facebook/rocksdb/pull/1710 Differential Revision: D4359863 Pulled By: gunnarku fbshipit-source-id: 2f1e2f8 | 28 December 2016, 00:09:12 UTC |
| 4e07b08 | Sage Weil | 24 December 2016, 03:03:53 UTC | include/rocksdb/utilities/env_librados: fix typo Summary: Broken by 972f96b3fbae1a4675043bdf4279c9072ad69645 Signed-off-by: Sage Weil <sage@redhat.com> Closes https://github.com/facebook/rocksdb/pull/1719 Differential Revision: D4366123 Pulled By: IslamAbdelRahman fbshipit-source-id: a11e535 | 24 December 2016, 03:09:14 UTC |
| ab48c16 | Yi Wu | 22 December 2016, 22:44:01 UTC | Print cache options to info log Summary: Improve cache options logging to info log. Also print the value of cache_index_and_filter_blocks_with_high_priority. Closes https://github.com/facebook/rocksdb/pull/1709 Differential Revision: D4358776 Pulled By: yiwu-arbug fbshipit-source-id: 8f030a0 | 22 December 2016, 22:54:19 UTC |
| 972f96b | Aaron Gao | 22 December 2016, 20:51:29 UTC | direct io write support Summary: rocksdb direct io support ``` [gzh@dev11575.prn2 ~/rocksdb] ./db_bench -benchmarks=fillseq --num=1000000 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 5.0 Date: Wed Nov 23 13:17:43 2016 CPU: 40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz CPUCache: 25600 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 4.393 micros/op 227639 ops/sec; 25.2 MB/s [gzh@dev11575.prn2 ~/roc Closes https://github.com/facebook/rocksdb/pull/1564 Differential Revision: D4241093 Pulled By: lightmark fbshipit-source-id: 98c29e3 | 22 December 2016, 21:09:19 UTC |
| 989e644 | Islam AbdelRahman | 22 December 2016, 01:35:00 UTC | Remove sst_file_manager option from LITE Summary: Remove sst_file_manager option from LITE Closes https://github.com/facebook/rocksdb/pull/1690 Differential Revision: D4341331 Pulled By: IslamAbdelRahman fbshipit-source-id: 9f9328d | 22 December 2016, 01:54:21 UTC |
| 1beef65 | Islam AbdelRahman | 22 December 2016, 01:32:24 UTC | Fix c_test Summary: addfile phase in c_test could fail because in previous steps we did a DeleteRange. Fix the test by simply moving the addfile phase before DeleteRange Closes https://github.com/facebook/rocksdb/pull/1672 Differential Revision: D4328896 Pulled By: IslamAbdelRahman fbshipit-source-id: 1d946df | 22 December 2016, 01:39:14 UTC |
| 3d69282 | Siying Dong | 21 December 2016, 21:29:26 UTC | persistent_cache: fix two timer Summary: In persistent_cache/block_cache_tier.cc, timers are never restarted, so the latency measured is not correct. Closes https://github.com/facebook/rocksdb/pull/1707 Differential Revision: D4355828 Pulled By: siying fbshipit-source-id: cd5f9e1 | 21 December 2016, 21:39:16 UTC |
| 046099c | ivan | 21 December 2016, 01:17:43 UTC | The array is malloced by backtrace_symbols(), and must be freed Summary: The address of the array of string pointers is returned as the function result of backtrace_symbols(). This array is malloced by backtrace_symbols(), and must be freed by the caller. Closes https://github.com/facebook/rocksdb/pull/1692 Differential Revision: D4355737 Pulled By: IslamAbdelRahman fbshipit-source-id: 5742035 | 21 December 2016, 01:24:12 UTC |
| 6ff2c8d | Yi Wu | 21 December 2016, 00:47:39 UTC | Remove gflags as travis build dependency Summary: apt-get fail to fetch gflags and seems it is in fact not needed. Closes https://github.com/facebook/rocksdb/pull/1705 Differential Revision: D4354555 Pulled By: yiwu-arbug fbshipit-source-id: b68fee3 | 21 December 2016, 01:09:16 UTC |
| 3cd9ed1 | Islam AbdelRahman | 20 December 2016, 18:54:38 UTC | Show sandcastle URL in phabricator Summary: We are passing a string as diff_id which make conduit call fail ``` $ echo '{"diff_id": "20982117", "name":"click here for sandcastle tests for D20982117", "link":"https://our.intern.facebook.com/intern/sandcastle/1984718793/"}' | arc call-conduit differential.updateunitresults {"error":"ERR-CONDUIT-CORE","errorMessage":"ERR-CONDUIT-CORE: Argument 1 passed to EntDiffPropertiesUpdateMutationBuilder::setDiffNumber() must be an instance of int, string given","response":null} ``` fix it by removing double quotes Closes https://github.com/facebook/rocksdb/pull/1700 Differential Revision: D4350227 Pulled By: IslamAbdelRahman fbshipit-source-id: b4504af | 20 December 2016, 19:09:12 UTC |
| 50e305d | Andrew Kryczka | 20 December 2016, 00:44:30 UTC | Collapse range deletions Summary: Added a tombstone-collapsing mode to RangeDelAggregator, which eliminates overlap in the TombstoneMap. In this mode, we can check whether a tombstone covers a user key using upper_bound() (i.e., binary search). However, the tradeoff is the overhead to add tombstones is now higher, so at first I've only enabled it for range scans (compaction/flush/user iterators), where we expect a high number of calls to ShouldDelete() for the same tombstones. Point queries like Get() will still use the linear scan approach. Also in this diff I changed RangeDelAggregator's TombstoneMap to use multimap with user keys instead of map with internal keys. Callers sometimes provided ParsedInternalKey directly, from which it would've required string copying to derive an internal key Slice with which we could search the map. Closes https://github.com/facebook/rocksdb/pull/1614 Differential Revision: D4270397 Pulled By: ajkr fbshipit-source-id: 93092c7 | 20 December 2016, 00:54:12 UTC |
| 5d1457d | Yi Wu | 19 December 2016, 22:00:04 UTC | Dump persistent cache options Summary: Dump persistent cache options Closes https://github.com/facebook/rocksdb/pull/1679 Differential Revision: D4337019 Pulled By: yiwu-arbug fbshipit-source-id: 3812f8a | 19 December 2016, 22:09:12 UTC |
| 7bd725e | Siying Dong | 19 December 2016, 20:25:35 UTC | db_bench: introduce --benchmark_read_rate_limit Summary: Add the parameter in db_bench to help users to measure latency histogram with constant read rate. Closes https://github.com/facebook/rocksdb/pull/1683 Differential Revision: D4341387 Pulled By: siying fbshipit-source-id: 1b4b276 | 19 December 2016, 20:39:11 UTC |
| 2966918 | Siying Dong | 19 December 2016, 19:41:20 UTC | Update Netflix section of USERS.md Summary: The change is based on https://github.com/facebook/rocksdb/issues/1678 Closes https://github.com/facebook/rocksdb/pull/1699 Differential Revision: D4347657 Pulled By: IslamAbdelRahman fbshipit-source-id: 72177c2 | 19 December 2016, 20:09:12 UTC |
| 342370f | Daniel Black | 17 December 2016, 08:05:05 UTC | Simplify MemTable::Update Summary: As suggested by testn in #1650 The Add is at the end of the function. Having a fallthough will result in it being added twice. Closes https://github.com/facebook/rocksdb/pull/1676 Differential Revision: D4331906 Pulled By: yiwu-arbug fbshipit-source-id: 895c4a0 | 17 December 2016, 08:09:13 UTC |
| 1a136c1 | Ding Ma | 17 December 2016, 02:26:56 UTC | Expose file size Summary: add a new function to SstFileWriter that will tell the user how big is there file right now. Closes https://github.com/facebook/rocksdb/pull/1686 Differential Revision: D4338868 Pulled By: mdyuki1016 fbshipit-source-id: c1ee16a | 17 December 2016, 02:39:12 UTC |
| fbff462 | Andrew Kryczka | 17 December 2016, 01:31:15 UTC | Reduce compaction iterator status checks Summary: seems it's expensive to check status since the underlying merge iterator checks status of all its children. so only do it when it's really necessary to get the status before invoking Next(), i.e., when we're advancing to get the first key in the next file. Closes https://github.com/facebook/rocksdb/pull/1691 Differential Revision: D4343446 Pulled By: siying fbshipit-source-id: 70ab315 | 17 December 2016, 01:39:09 UTC |
| bd6cf7b | Jianpeng Ma | 16 December 2016, 21:01:37 UTC | WritableFileWriter: default buffer size equal min(64k,options.writabl? Summary: ?e_file_max_buffer_size) If we overwrite WritableFile and has a buffer which has the same function of buf_. We hope remove the cache function of WritableFileWriter. So using options.writable_file_max_buffer_size = 0 to disable cache function. Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com> Closes https://github.com/facebook/rocksdb/pull/1628 Differential Revision: D4307219 Pulled By: yiwu-arbug fbshipit-source-id: 77a6e26 | 16 December 2016, 21:09:14 UTC |
| fc0c6fd | Yi Wu | 16 December 2016, 19:19:52 UTC | "make format" format diff since last commit from master Summary: Update clang-format script to format diff since last commit from master, instead of just last commit. In our common workflow we usually endup with multiple commits for a single PR. This change make it easier to format all stacking changes. Closes https://github.com/facebook/rocksdb/pull/1684 Differential Revision: D4340597 Pulled By: yiwu-arbug fbshipit-source-id: c18949e | 16 December 2016, 19:24:18 UTC |
| 816c1e3 | Daniel Black | 16 December 2016, 19:17:26 UTC | gcc-7 requires include <functional> for std::function Summary: Fixes compile error: In file included from ./util/statistics.h:17:0, from ./util/stop_watch.h:8, from ./util/perf_step_timer.h:9, from ./util/iostats_context_imp.h:8, from ./util/posix_logger.h:27, from ./port/util_logger.h:18, from ./db/auto_roll_logger.h:15, from db/auto_roll_logger.cc:6: ./util/thread_local.h:65:16: error: 'function' in namespace 'std' does not name a template type typedef std::function<void(void*, void*)> FoldFunc; Closes https://github.com/facebook/rocksdb/pull/1656 Differential Revision: D4318702 Pulled By: yiwu-arbug fbshipit-source-id: 8c5d17a | 16 December 2016, 19:24:18 UTC |
| c270735 | Yi Wu | 16 December 2016, 19:00:03 UTC | Iterator should be in corrupted status if merge operator return false Summary: Iterator should be in corrupted status if merge operator return false. Also add test to make sure if max_successive_merges is hit during write, data will not be lost. Closes https://github.com/facebook/rocksdb/pull/1665 Differential Revision: D4322695 Pulled By: yiwu-arbug fbshipit-source-id: b327b05 | 16 December 2016, 19:09:16 UTC |
| a8bf4d6 | Daniel Black | 16 December 2016, 04:11:10 UTC | Make format shows wrong curl command for retrieving clang-format-diff.py Summary: Previously: $ make format Makefile:104: Warning: Compiling in debug mode. Don't use the resulting binary in production build_tools/format-diff.sh You didn't have clang-format-diff.py available in your computer! You can download it by running: curl http://goo.gl/iUW1u2 Makefile:868: recipe for target 'format' failed make: *** [format] Error 128 $ curl http://goo.gl/iUW1u2 > ~/bin/clang-format-diff.py % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 276 0 276 0 0 148 0 --:--:-- 0:00:01 --:--:-- 148m $ more ~/bin/clang-format-diff.py <HTML> <HEAD> <TITLE>Moved Permanently</TITLE> </HEAD> <BODY BGCOLOR="#FFFFFF" TEXT="#000000"> <H1>Moved Permanently</H1> The document has moved <A HREF="https://raw.github.com/leaningtech/duetto-clang/master/tools/clang-format/clang-format-diff.py">here</A>. </BODY> </HTML> Closes https://github.com/facebook/rocksdb/pull/1680 Differential Revision: D4338495 Pulled By: yiwu-arbug fbshipit-source-id: e2b24d8 | 16 December 2016, 04:24:12 UTC |
| 8f5d24a | siddontang | 16 December 2016, 01:14:06 UTC | C API: support get usage and pinned_usage for cache Summary: Closes https://github.com/facebook/rocksdb/pull/1671 Differential Revision: D4327453 Pulled By: yiwu-arbug fbshipit-source-id: bcdbc65 | 16 December 2016, 01:24:17 UTC |
| 0ab6fc1 | Daniel Black | 15 December 2016, 03:20:42 UTC | Gcc-7 buffer size insufficient Summary: Bunch of commits related to insufficient buffer size. Errors in individual commits. Closes https://github.com/facebook/rocksdb/pull/1673 Differential Revision: D4332127 Pulled By: IslamAbdelRahman fbshipit-source-id: 878f73c | 15 December 2016, 03:24:26 UTC |
| b7239bf | Daniel Black | 15 December 2016, 03:15:32 UTC | Gcc 7 fallthrough Summary: hopefully the last of the gcc-7 compile errors Closes https://github.com/facebook/rocksdb/pull/1675 Differential Revision: D4332106 Pulled By: IslamAbdelRahman fbshipit-source-id: 139448c | 15 December 2016, 03:24:25 UTC |
| 477b6ea | Daniel Black | 15 December 2016, 01:05:57 UTC | std::remove_if requires <algorithm> Summary: fixes error (that occurred on gcc-7): error: util/env_basic_test.cc: In member function 'virtual rocksdb::Status rocksdb::NormalizingEnvWrapper::GetChildren(const string&, std::vector<std::__cxx11::basic_string<char> >*)': util/env_basic_test.cc:27:21: error: 'remove_if' is not a member of 'std' result->erase(std::remove_if(result->begin(), result->end(), ^~~ Closes https://github.com/facebook/rocksdb/pull/1674 Differential Revision: D4331221 Pulled By: ajkr fbshipit-source-id: 9bbdc78 | 15 December 2016, 01:09:14 UTC |
| 83f9a6f | Andrew Kryczka | 15 December 2016, 00:29:23 UTC | Fail BackupEngine::Open upon meta-file read error Summary: We used to treat any failure to read a backup's meta-file as if the backup were corrupted; however, we should distinguish corruption errors from errors in the backup Env. This fixes an issue where callers would get inconsistent results from GetBackupInfo() if they called it on an engine that encountered Env error during initialization. Now we fail Initialize() in this case so callers cannot invoke GetBackupInfo() on such engines. Closes https://github.com/facebook/rocksdb/pull/1654 Differential Revision: D4318573 Pulled By: ajkr fbshipit-source-id: f7a7c54 | 15 December 2016, 00:39:15 UTC |
| a79eae4 | Karthikeyan Radhakrishnan | 14 December 2016, 19:48:59 UTC | Add pcache documentation images Summary: Adding images that are used in persistent read cache documentation. Closes https://github.com/facebook/rocksdb/pull/1666 Differential Revision: D4328220 Pulled By: IslamAbdelRahman fbshipit-source-id: 767e0d0 | 14 December 2016, 19:54:22 UTC |
| d71e728 | Islam AbdelRahman | 14 December 2016, 19:09:50 UTC | Print user collected properties in sst_dump Summary: Include a dump of user_collected_properties in sst_dump Closes https://github.com/facebook/rocksdb/pull/1668 Differential Revision: D4325078 Pulled By: IslamAbdelRahman fbshipit-source-id: 226b6d6 | 14 December 2016, 19:24:11 UTC |
| 7004a6f | Jonathan Lee | 14 December 2016, 18:21:54 UTC | Add missing copyright header Summary: "examples/c_simple_example.c" did not have a proper copyright header. Closes https://github.com/facebook/rocksdb/pull/1670 Differential Revision: D4327445 Pulled By: yiwu-arbug fbshipit-source-id: a70389e | 14 December 2016, 18:24:13 UTC |
| 3cdfaec | Adam Retter | 14 December 2016, 18:17:52 UTC | Fixes for MSVC compilation Summary: Closes https://github.com/facebook/rocksdb/pull/1669 Differential Revision: D4327421 Pulled By: yiwu-arbug fbshipit-source-id: 661ee0b | 14 December 2016, 18:24:13 UTC |
| e097222 | Daniel Black | 14 December 2016, 02:33:21 UTC | util/logging.cc: buffer of insufficient size (gcc-7 -Werror=format-length) Summary: util/logging.cc:100:13: error: output may be truncated before the last format character [-Werror=format-length=] std::string NumberToHumanString(int64_t num) { ^~~~~~~~~~~~~~~~~~~ util/logging.cc:106:59: note: format output between 3 and 19 bytes into a destination of size 16 snprintf(buf, sizeof(buf), "%" PRIi64 "K", num / 1000); Closes https://github.com/facebook/rocksdb/pull/1653 Differential Revision: D4318687 Pulled By: yiwu-arbug fbshipit-source-id: 3a5c931 | 14 December 2016, 02:39:14 UTC |
| cfc34d7 | Daniel Black | 14 December 2016, 02:30:21 UTC | Missing break in case in DBTestBase::CurrentOptions Summary: Found by gcc-7 compile error. This appeared to be a fault as these options seems too different. Closes https://github.com/facebook/rocksdb/pull/1667 Differential Revision: D4324174 Pulled By: yiwu-arbug fbshipit-source-id: 0f65383 | 14 December 2016, 02:39:14 UTC |
| bfbcec2 | Daniel Black | 14 December 2016, 02:22:00 UTC | Gcc 7 error expansion to defined Summary: sorry if these gcc-7/clang-4 cleanups are getting tedious. Closes https://github.com/facebook/rocksdb/pull/1658 Differential Revision: D4318792 Pulled By: yiwu-arbug fbshipit-source-id: 8e85891 | 14 December 2016, 02:39:14 UTC |
| 6653e32 | Bassam Tabbara | 13 December 2016, 22:20:17 UTC | build: make it easier to pass PORTABLE Summary: currently when running a portable build we have to do the following PORTABLE=1 make ... this commit adds support for the following make PORTABLE=1 ... this might be seem subtle but it makes PORTABLE like all other makefile args and simplifies invocation from numerous build systems including things like ExternalProject_Add in cmake. Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com> Closes https://github.com/facebook/rocksdb/pull/1643 Differential Revision: D4315870 Pulled By: yiwu-arbug fbshipit-source-id: ee43755 | 13 December 2016, 22:39:17 UTC |
| 67adc93 | Daniel Black | 13 December 2016, 22:21:05 UTC | intentional fallthough (prevents gcc-7/clang-4 error) Summary: db/memtable.cc: In member function 'void rocksdb::MemTable::Update(rocksdb::SequenceNumber, const rocksdb::Slice&, const rocksdb::Slice&)': db/memtable.cc:736:11: error: this statement may fall through [-Werror=implicit-fallthrough=] } ^ db/memtable.cc:738:9: note: here default: ^~~~~~~ cc1plus: all warnings being treated as errors closes #1650 Closes https://github.com/facebook/rocksdb/pull/1655 Differential Revision: D4318696 Pulled By: yiwu-arbug fbshipit-source-id: 1a8981c | 13 December 2016, 22:39:17 UTC |
| 1a146f8 | Islam AbdelRahman | 13 December 2016, 22:03:15 UTC | break Flush wait for dropped CF Summary: In FlushJob we dont do the Flush if the CF is dropped https://github.com/facebook/rocksdb/blob/master/db/flush_job.cc#L184-L188 but inside WaitForFlushMemTable we keep waiting forever even if the CF is dropped. Closes https://github.com/facebook/rocksdb/pull/1664 Differential Revision: D4321032 Pulled By: IslamAbdelRahman fbshipit-source-id: 6e2b25d | 13 December 2016, 22:09:12 UTC |
| c3e5ee7 | Daniel Black | 13 December 2016, 21:52:00 UTC | util/histogram.cc: HistogramStat::toString buffer insufficient Summary: Increased buffer size to 1650. util/histogram.cc: In member function 'std::__cxx11::string rocksdb::HistogramStat::ToString() const': util/histogram.cc:189:13: error: '%.2f' directive output truncated writing between 4 and 313 bytes into a region of size 0 [-Werror=format-length=] std::string HistogramStat::ToString() const { ^~~~~~~~~~~~~ util/histogram.cc:205:30: note: format output between 69 and 1614 bytes into a destination of size 200 Percentile(99.99)); ^ cc1plus: all warnings being treated as errors Makefile:1521: recipe for target 'util/histogram.o' failed Closes https://github.com/facebook/rocksdb/pull/1660 Differential Revision: D4318820 Pulled By: yiwu-arbug fbshipit-source-id: 45ae6ea | 13 December 2016, 22:09:12 UTC |
| 5334d8b | Daniel Black | 13 December 2016, 21:50:22 UTC | table/block_based_table_builder.cc: intentional fallthrough - comment to match gcc pattern Summary: The gcc-7 code for parsing comments (libcpp/lex.c) didn't match the intentional fallthough in this comment. table/block_based_table_builder.cc: In member function 'void rocksdb::BlockBasedTableBuilder::WriteRawBlock(const rocksdb::Slice&, rocksdb::CompressionType, rocksdb::BlockHandle*)': table/block_based_table_builder.cc:754:22: error: this statement may fall through [-Werror=implicit-fallthrough=] assert(false); ^ table/block_based_table_builder.cc:756:7: note: here case kCRC32c: { ^~~~ cc1plus: all warnings being treated as errors Closes https://github.com/facebook/rocksdb/pull/1661 Differential Revision: D4318817 Pulled By: yiwu-arbug fbshipit-source-id: e67d171 | 13 December 2016, 21:54:15 UTC |
| 36d42e6 | Yi Wu | 13 December 2016, 19:42:42 UTC | Disable test to unblock travis build Summary: The two tests keep failing in travis. Disable them and will fix later. Closes https://github.com/facebook/rocksdb/pull/1648 Differential Revision: D4316389 Pulled By: yiwu-arbug fbshipit-source-id: 0a370e7 | 13 December 2016, 19:54:14 UTC |
| b57dd92 | siddontang | 13 December 2016, 19:08:45 UTC | C API: support writebatch delete range Summary: Seem that writebatch delete range can work now, so I add C API for later use. Btw, can we use this feature in production now? Closes https://github.com/facebook/rocksdb/pull/1647 Differential Revision: D4314534 Pulled By: ajkr fbshipit-source-id: e835165 | 13 December 2016, 19:24:18 UTC |
| 2ba59b5 | Islam AbdelRahman | 13 December 2016, 08:47:52 UTC | Disallow ingesting files into dropped CFs Summary: This PR update IngestExternalFile to return an error if we try to ingest a file into a dropped CF. Right now if IngestExternalFile want to flush a memtable, and it's ingesting a file into a dropped CF, it will wait forever since flushing is not possible for the dropped CF Closes https://github.com/facebook/rocksdb/pull/1657 Differential Revision: D4318657 Pulled By: IslamAbdelRahman fbshipit-source-id: ed6ea2b | 13 December 2016, 08:54:14 UTC |
| 1f6f7e3 | Daniel Black | 12 December 2016, 22:32:55 UTC | cast to signed char in ldb_cmd_test for ppc64le Summary: char is unsigned on power by default causing this test to fail with the FF case. ppc64 return 255 while x86 returned -1. Casting works on both platforms. Closes https://github.com/facebook/rocksdb/pull/1500 Differential Revision: D4308775 Pulled By: yiwu-arbug fbshipit-source-id: db3e6e0 | 12 December 2016, 22:39:18 UTC |
| 243975d | Andrew Kryczka | 12 December 2016, 21:18:38 UTC | More accurate error status for BackupEngine::Open Summary: Some users are assuming NotFound means the backup does not exist at the provided path, which is a reasonable assumption. We need to stop returning NotFound for system errors. Depends on #1644 Closes https://github.com/facebook/rocksdb/pull/1645 Differential Revision: D4312233 Pulled By: ajkr fbshipit-source-id: 5343c10 | 12 December 2016, 21:24:21 UTC |
| f0c509e | Andrew Kryczka | 12 December 2016, 20:38:43 UTC | Return finer-granularity status from Env::GetChildren* Summary: It'd be nice to use the error status type to distinguish between user error and system error. For example, GetChildren can fail listing a backup directory's contents either because a bad path was provided (user error) or because an operation failed, e.g., a remote storage service call failed (system error). In the former case, we want to continue and treat the backup directory as empty; in the latter case, we want to immediately propagate the error to the caller. This diff uses NotFound to indicate user error and IOError to indicate system error. Previously IOError indicated both. Closes https://github.com/facebook/rocksdb/pull/1644 Differential Revision: D4312157 Pulled By: ajkr fbshipit-source-id: 51b4f24 | 12 December 2016, 20:54:13 UTC |
| dc64f46 | Andrew Kryczka | 10 December 2016, 00:36:43 UTC | Add db_bench option for stderr logging Summary: The info log file ("LOG") is stored in the db directory by default. When the db is on a distributed env, this is unnecessarily slow. So, I added an option to db_bench to just print the info log messages to stderr. Closes https://github.com/facebook/rocksdb/pull/1641 Differential Revision: D4309348 Pulled By: ajkr fbshipit-source-id: 1e6f851 | 10 December 2016, 00:54:14 UTC |
| 2cabdb8 | Jonathan Lee | 09 December 2016, 19:39:16 UTC | Increase buffer size Summary: When compiling with GCC>=7.0.0, "db/internal_stats.cc" fails to compile as the data being written to the buffer potentially exceeds its size. This fix simply doubles the size of the buffer, thus accommodating the max possible data size. Closes https://github.com/facebook/rocksdb/pull/1635 Differential Revision: D4302162 Pulled By: yiwu-arbug fbshipit-source-id: c76ad59 | 09 December 2016, 19:54:22 UTC |
| 4a17b47 | Jonathan Lee | 09 December 2016, 19:34:32 UTC | Remove unnecessary header include Summary: Remove "util/testharness.h" from list of includes for "db/db_filesnapshot.cc", as it wasn't being used and thus caused an extraneous dependency on gtest. Closes https://github.com/facebook/rocksdb/pull/1634 Differential Revision: D4302146 Pulled By: yiwu-arbug fbshipit-source-id: e900c0b | 09 December 2016, 19:54:21 UTC |
| 8c2b921 | Mike Kolupaev | 09 December 2016, 18:23:58 UTC | Fixed a crash in debug build in flush_job.cc Summary: It was doing `&range_del_iters[0]` on an empty vector. Even though the resulting pointer is never dereferenced, it's still bad for two reasons: * the practical reason: it crashes with `std::out_of_range` exception in our debug build, * the "C++ standard lawyer" reason: it's undefined behavior because, in `std::vector` implementation, it probably "dereferences" a null pointer, which is invalid even though it doesn't actually read the pointed memory, just converts a pointer into a reference (and then flush_job.cc converts it back to pointer); nullptr references are undefined behavior. Closes https://github.com/facebook/rocksdb/pull/1612 Differential Revision: D4265625 Pulled By: al13n321 fbshipit-source-id: db26fb9 | 09 December 2016, 18:39:12 UTC |
| 20ce081 | Islam AbdelRahman | 08 December 2016, 21:30:09 UTC | Fix issue where IngestExternalFile insert blocks in block cache with g_seqno=0 Summary: When we Ingest an external file we open it to read some metadata and first/last key during doing that we insert blocks into the block cache with global_seqno = 0 If we move the file (did not copy it) into the DB, we will use these blocks with the wrong seqno in the read path Closes https://github.com/facebook/rocksdb/pull/1627 Differential Revision: D4293332 Pulled By: yiwu-arbug fbshipit-source-id: 3ce5523 | 08 December 2016, 21:39:18 UTC |
| 5241e0d | Andrew Kryczka | 08 December 2016, 18:58:39 UTC | fix db_bench argument type Summary: Closes https://github.com/facebook/rocksdb/pull/1633 Differential Revision: D4298161 Pulled By: yiwu-arbug fbshipit-source-id: 2c7af35 | 08 December 2016, 19:09:14 UTC |
| c04f6a0 | Jonathan Lee | 08 December 2016, 06:06:59 UTC | Specify shell in makefile Summary: The second variable "SHELL" simply tells make explicitly which shell to use, instead of allowing it to default to "/bin/sh", which may or may not be Bash. However, simply defining the second variable by itself causes make to throw an error concerning a circular definition, as it would be attempting to use the "shell" command while simultaneously trying to set which shell to use. Thus, the first variable "BASH_EXISTS" is defined such that make already knows about "/path/to/bash" before trying to use it to set "SHELL". A more technically correct solution would be to edit the makefile itself to make it compatible with non-bash shells (see the original Issue discussion for details). However, as it seems very few of the people working on this project were building with non-bash shells, I figured this solution would be good enough. Closes https://github.com/facebook/rocksdb/pull/1631 Differential Revision: D4295689 Pulled By: yiwu-arbug fbshipit-source-id: e4f9532 | 08 December 2016, 06:24:15 UTC |
| 45c7ce1 | zhangjinpeng1987 | 08 December 2016, 01:44:35 UTC | CompactRangeOptions C API Summary: Add C API for CompactRangeOptions. Closes https://github.com/facebook/rocksdb/pull/1596 Differential Revision: D4252339 Pulled By: yiwu-arbug fbshipit-source-id: f768f93 | 08 December 2016, 01:54:14 UTC |
| 2c2ba68 | Andrew Kryczka | 07 December 2016, 20:59:59 UTC | db_stress support for range deletions Summary: made db_stress capable of adding range deletions to its db and verifying their correctness. i'll make db_crashtest.py use this option later once the collapsing optimization (https://github.com/facebook/rocksdb/pull/1614) is committed because currently it slows down the test too much. Closes https://github.com/facebook/rocksdb/pull/1625 Differential Revision: D4293939 Pulled By: ajkr fbshipit-source-id: d3beb3a | 07 December 2016, 21:09:24 UTC |
| b821984 | Andrew Kryczka | 07 December 2016, 20:37:09 UTC | DeleteRange read path end-to-end tests Summary: Closes https://github.com/facebook/rocksdb/pull/1592 Differential Revision: D4246260 Pulled By: ajkr fbshipit-source-id: ce03fa2 | 07 December 2016, 20:54:17 UTC |
| 2f4fc53 | Artemiy Kolesnikov | 07 December 2016, 19:42:49 UTC | Compaction::IsTrivialMove relaxing Summary: IsTrivialMove returns true if no input file overlaps with output_level+1 with more than max_compaction_bytes_ bytes. Closes https://github.com/facebook/rocksdb/pull/1619 Differential Revision: D4278338 Pulled By: yiwu-arbug fbshipit-source-id: 994c001 | 07 December 2016, 19:54:11 UTC |
| 1dce75b | dhruba borthakur | 07 December 2016, 19:15:35 UTC | Update USERS.md Summary: Closes https://github.com/facebook/rocksdb/pull/1630 Differential Revision: D4293200 Pulled By: IslamAbdelRahman fbshipit-source-id: 9c81f85 | 07 December 2016, 19:24:14 UTC |
| 304b3c7 | dhruba borthakur | 07 December 2016, 19:14:29 UTC | Update USERS.md Summary: Closes https://github.com/facebook/rocksdb/pull/1629 Differential Revision: D4293183 Pulled By: IslamAbdelRahman fbshipit-source-id: 759ea92 | 07 December 2016, 19:24:14 UTC |
| fa50fff | Andrew Kryczka | 07 December 2016, 02:02:43 UTC | Option to expand range tombstones in db_bench Summary: When enabled, this option replaces range tombstones with a sequence of point tombstones covering the same range. This can be used to A/B test perf of range tombstones vs sequential point tombstones, and help us find the cross-over point, i.e., the size of the range above which range tombstones outperform point tombstones. Closes https://github.com/facebook/rocksdb/pull/1594 Differential Revision: D4246312 Pulled By: ajkr fbshipit-source-id: 3b00b23 | 07 December 2016, 02:09:14 UTC |
| c26a4d8 | Yi Wu | 06 December 2016, 22:29:26 UTC | Fix compile error in trasaction_lock_mgr.cc Summary: Fix error on mac/windows build since they don't recognize `uint`. Closes https://github.com/facebook/rocksdb/pull/1624 Differential Revision: D4287139 Pulled By: yiwu-arbug fbshipit-source-id: b7cc88f | 06 December 2016, 22:39:16 UTC |
| ed8fbdb | Islam AbdelRahman | 06 December 2016, 21:56:17 UTC | Add EventListener::OnExternalFileIngested() event Summary: Add EventListener::OnExternalFileIngested() to allow user to subscribe to external file ingestion events Closes https://github.com/facebook/rocksdb/pull/1623 Differential Revision: D4285844 Pulled By: IslamAbdelRahman fbshipit-source-id: 0b95a88 | 06 December 2016, 22:09:17 UTC |
| 2005c88 | Manuel Ung | 06 December 2016, 01:18:14 UTC | Implement non-exclusive locks Summary: This is an implementation of non-exclusive locks for pessimistic transactions. It is relatively simple and does not prevent starvation (ie. it's possible that request for exclusive access will never be granted if there are always threads holding shared access). It is done by changing `KeyLockInfo` to hold an set a transaction ids, instead of just one, and adding a flag specifying whether this lock is currently held with exclusive access or not. Some implementation notes: - Some lock diagnostic functions had to be updated to return a set of transaction ids for a given lock, eg. `GetWaitingTxn` and `GetLockStatusData`. - Deadlock detection is a bit more complicated since a transaction can now wait on multiple other transactions. A BFS is done in this case, and deadlock detection depth is now just a limit on the number of transactions we visit. - Expirable transactions do not work efficiently with shared locks at the moment, but that's okay for now. Closes https://github.com/facebook/rocksdb/pull/1573 Differential Revision: D4239097 Pulled By: lth fbshipit-source-id: da7c074 | 06 December 2016, 01:39:17 UTC |
| 0b0f235 | Yi Wu | 05 December 2016, 23:55:36 UTC | Mention IngestExternalFile changes in HISTORY.md Summary: I hit the land button too fast and didn't include the line. Closes https://github.com/facebook/rocksdb/pull/1622 Differential Revision: D4281316 Pulled By: yiwu-arbug fbshipit-source-id: c7b38e0 | 06 December 2016, 00:09:11 UTC |
| 23db48e | Yi Wu | 05 December 2016, 23:28:29 UTC | Update HISTORY.md for 5.0 branch Summary: These changes are included in the new branch-cut. Closes https://github.com/facebook/rocksdb/pull/1621 Differential Revision: D4281015 Pulled By: yiwu-arbug fbshipit-source-id: d88858b | 05 December 2016, 23:39:11 UTC |
| beb36d9 | Mike Kolupaev | 05 December 2016, 23:07:01 UTC | Fixed CompactionFilter::Decision::kRemoveAndSkipUntil Summary: Embarassingly enough, the first time I tried to use my new feature in logdevice it crashed with this assertion failure: db/pinned_iterators_manager.h:30: void rocksdb::PinnedIteratorsManager::StartPinning(): Assertion `pinning_enabled == false' failed The issue was that `pinned_iters_mgr_.StartPinning()` was called but `pinned_iters_mgr_.ReleasePinnedData()` wasn't. Closes https://github.com/facebook/rocksdb/pull/1611 Differential Revision: D4265622 Pulled By: al13n321 fbshipit-source-id: 747b10f | 05 December 2016, 23:24:11 UTC |
| 67f37cf | Islam AbdelRahman | 05 December 2016, 22:16:23 UTC | Allow user to specify a CF for SST files generated by SstFileWriter Summary: Allow user to explicitly specify that the generated file by SstFileWriter will be ingested in a specific CF. This allow us to persist the CF id in the generated file Closes https://github.com/facebook/rocksdb/pull/1615 Differential Revision: D4270422 Pulled By: IslamAbdelRahman fbshipit-source-id: 7fb954e | 05 December 2016, 22:24:16 UTC |
| 9053fe2 | Anton Safonov | 05 December 2016, 22:09:35 UTC | Made delete_obsolete_files_period_micros option dynamic Summary: Made delete_obsolete_files_period_micros option dynamic. It can be updating using DB::SetDBOptions(). Closes https://github.com/facebook/rocksdb/pull/1595 Differential Revision: D4246569 Pulled By: tonek fbshipit-source-id: d23f560 | 05 December 2016, 22:24:16 UTC |
| edde954 | Islam AbdelRahman | 02 December 2016, 02:24:13 UTC | fix clang build Summary: override is missing for FilterV2 Closes https://github.com/facebook/rocksdb/pull/1606 Differential Revision: D4263832 Pulled By: IslamAbdelRahman fbshipit-source-id: d8b337a | 02 December 2016, 02:39:10 UTC |
| 56281f3 | Yi Wu | 02 December 2016, 00:50:59 UTC | Add memtable_insert_with_hint_prefix_size option to db_bench Summary: Add memtable_insert_with_hint_prefix_size option to db_bench Closes https://github.com/facebook/rocksdb/pull/1604 Differential Revision: D4260549 Pulled By: yiwu-arbug fbshipit-source-id: cee5ef7 | 02 December 2016, 00:54:16 UTC |
| 4a21b14 | Islam AbdelRahman | 01 December 2016, 21:30:01 UTC | Cache heap::downheap() root comparison (optimize heap cmp call) Summary: Reduce number of comparisons in heap by caching which child node in the first level is smallest (left_child or right_child) So next time we can compare directly against the smallest child I see that the total number of calls to comparator drops significantly when using this optimization Before caching (~2mil key comparison for iterating the DB) ``` $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq" --db="/dev/shm/heap_opt" --use_existing_db --disable_auto_compactions --cache_size=1000000000 --perf_level=2 readseq : 0.338 micros/op 2959201 ops/sec; 327.4 MB/s user_key_comparison_count = 2000008 ``` After caching (~1mil key comparison for iterating the DB) ``` $ DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readseq" --db="/dev/shm/heap_opt" --use_existing_db --disable_auto_compactions --cache_size=1000000000 --perf_level=2 readseq : 0.309 micros/op 3236801 ops/sec; 358.1 MB/s user_key_comparison_count = 1000011 ``` It also improves Closes https://github.com/facebook/rocksdb/pull/1600 Differential Revision: D4256027 Pulled By: IslamAbdelRahman fbshipit-source-id: 76fcc66 | 01 December 2016, 21:39:14 UTC |
| e39d080 | Islam AbdelRahman | 01 December 2016, 18:02:12 UTC | Fix travis (compile for clang < 3.9) Summary: Travis fail because it uses clang 3.6 which don't recognize `__attribute__((__no_sanitize__("undefined")))` Closes https://github.com/facebook/rocksdb/pull/1601 Differential Revision: D4257175 Pulled By: IslamAbdelRahman fbshipit-source-id: fb4d1ab | 01 December 2016, 18:09:22 UTC |
| 3f407b0 | Igor Canadi | 01 December 2016, 18:00:56 UTC | Kill flashcache code in RocksDB Summary: Now that we have userspace persisted cache, we don't need flashcache anymore. Closes https://github.com/facebook/rocksdb/pull/1588 Differential Revision: D4245114 Pulled By: igorcanadi fbshipit-source-id: e2c1c72 | 01 December 2016, 18:09:22 UTC |
| b77007d | fangchenliaohui | 01 December 2016, 17:46:43 UTC | Bug: paralle_group status updated in WriteThread::CompleteParallelWorker Summary: Multi-write thread may update the status of the parallel_group in WriteThread::CompleteParallelWorker if the status of Writer is not ok! When copy write status to the paralle_group, the write thread just hold the mutex of the the writer processed by itself. it is useless. The thread should held the the leader of the parallel_group instead. Closes https://github.com/facebook/rocksdb/pull/1598 Differential Revision: D4252335 Pulled By: siying fbshipit-source-id: 3864cf7 | 01 December 2016, 17:54:11 UTC |
