4fb65f9 | Islam AbdelRahman | 07 April 2017, 00:04:23 UTC | Bump version to 5.3.2 | 07 April 2017, 00:04:23 UTC |
ee33e29 | Islam AbdelRahman | 16 March 2017, 19:06:04 UTC | Support SstFileManager::SetDeleteRateBytesPerSecond() Summary: Update DeleteScheduler component to support changing delete rate in runtime by introducing SstFileManager::SetDeleteRateBytesPerSecond() Closes https://github.com/facebook/rocksdb/pull/1994 Differential Revision: D4719906 Pulled By: IslamAbdelRahman fbshipit-source-id: e6b8d9e | 06 April 2017, 23:55:23 UTC |
5e0bddf | Andrew Kryczka | 05 April 2017, 21:13:25 UTC | bump patch version | 05 April 2017, 21:13:25 UTC |
4f0ef72 | Andrew Kryczka | 05 April 2017, 21:11:33 UTC | revert remove fallocate punch hole | 05 April 2017, 21:11:33 UTC |
491fa69 | Mikhail Antonov | 30 March 2017, 00:13:43 UTC | Added SstFileWriter construtor without explicit comparator to JNI api Summary: Adding API missing after https://github.com/facebook/rocksdb/commit/1ffbdfd9a7637b6517053842386d71df2cd00d9b#diff-b94146418eed4a9c1bf324041b95b279. adamretter IslamAbdelRahman Tested locally. Closes https://github.com/facebook/rocksdb/pull/2028 Differential Revision: D4762817 Pulled By: IslamAbdelRahman fbshipit-source-id: 833f478 | 04 April 2017, 00:47:05 UTC |
4fa68fb | Aaron Gao | 13 March 2017, 18:35:50 UTC | update history.md for fixing the bug that skips keys Summary: Closes https://github.com/facebook/rocksdb/pull/1986 Differential Revision: D4699152 Pulled By: siying fbshipit-source-id: b18c32c | 30 March 2017, 18:19:54 UTC |
76979d8 | Aaron Gao | 28 March 2017, 22:47:54 UTC | delete fallocate with punch_hole Summary: As discuss in this thread: https://www.facebook.com/groups/rocksdb.dev/permalink/1218043868294125/ We remove fallocate with FALLOC_FL_PUNCH_HOLE because the recent bug on xfs in kernel 4.x+ that align file size to page size even with FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE. Closes https://github.com/facebook/rocksdb/pull/2038 Differential Revision: D4779974 Pulled By: siying fbshipit-source-id: 5f54625 | 30 March 2017, 18:14:14 UTC |
bae811f | Siying Dong | 29 March 2017, 18:42:56 UTC | Expose the stalling information through DB::GetProperty() Summary: Add two DB properties: rocksdb.actual_delayed_write_rate and rocksdb.is_write_stooped, for people to know whether current writes are being throttled. Closes https://github.com/facebook/rocksdb/pull/2043 Differential Revision: D4782975 Pulled By: siying fbshipit-source-id: 6b2f5cf | 30 March 2017, 00:37:06 UTC |
b900f6e | Andrew Kryczka | 24 March 2017, 01:27:51 UTC | fix a header include | 24 March 2017, 02:22:30 UTC |
bb14ff7 | Siying Dong | 22 March 2017, 18:13:55 UTC | HISTORY.md for log_size_for_flush in CreateCheckpoint() Summary: Closes https://github.com/facebook/rocksdb/pull/2021 Differential Revision: D4755324 Pulled By: siying fbshipit-source-id: c8d7955 | 22 March 2017, 18:50:25 UTC |
47177be | Siying Dong | 22 March 2017, 00:53:21 UTC | Allow checkpointing without flushing Summary: Add a parameter to Checkpoint::CreateCheckpoint() so that flush can be skipped if total log file size is within a threshold. Closes https://github.com/facebook/rocksdb/pull/1993 Differential Revision: D4719842 Pulled By: siying fbshipit-source-id: 4f9d9e1 | 22 March 2017, 01:49:39 UTC |
93989b0 | Andrew Kryczka | 10 March 2017, 18:46:55 UTC | fix histogram max typecast | 10 March 2017, 18:49:03 UTC |
fe18356 | Andrew Kryczka | 09 March 2017, 20:33:48 UTC | release 5.3 Summary: Closes https://github.com/facebook/rocksdb/pull/1971 Differential Revision: D4683851 Pulled By: ajkr fbshipit-source-id: 967116e | 09 March 2017, 20:39:10 UTC |
5dae019 | Maysam Yabandeh | 09 March 2017, 18:59:06 UTC | Revert "Report cpu usage using time command" Summary: This reverts commit d43adf21bb26c2dd89d2942895c376dd4775455c. The patch has caused problems in regression tests. Will revert it for now until we figure how to debug the problems regression tests. Closes https://github.com/facebook/rocksdb/pull/1975 Differential Revision: D4682880 Pulled By: maysamyabandeh fbshipit-source-id: 84df83a | 09 March 2017, 19:09:13 UTC |
f2817fb | Andrew Kryczka | 09 March 2017, 06:16:45 UTC | avoid ASSERT_EQ(false, ...); Summary: lately it fails on travis due to a compiler bug (see https://github.com/google/googletest/issues/322#issuecomment-125645145). interestingly it seems to affect occurrences of `ASSERT_EQ(false, ...);` but not `ASSERT_EQ(true, ...);`. Closes https://github.com/facebook/rocksdb/pull/1958 Differential Revision: D4680742 Pulled By: ajkr fbshipit-source-id: 291fe41 | 09 March 2017, 06:24:16 UTC |
5b11124 | Andrew Kryczka | 09 March 2017, 06:13:15 UTC | add max to histogram stats Summary: Domas enlightened me about p100 (i.e., max) stats. Let's add them to our histograms. Closes https://github.com/facebook/rocksdb/pull/1968 Differential Revision: D4678716 Pulled By: ajkr fbshipit-source-id: 65e7118 | 09 March 2017, 06:24:15 UTC |
d43adf2 | Maysam Yabandeh | 09 March 2017, 01:50:19 UTC | Report cpu usage using time command Summary: It augments the regression benchmarks with a time command, parses the output, and print them to the SUMMARY.csv file. I tested a variation of the script locally. Any idea how to do run a test that also involves writing to scuba tables? Closes https://github.com/facebook/rocksdb/pull/1967 Differential Revision: D4679470 Pulled By: maysamyabandeh fbshipit-source-id: 44dac30 | 09 March 2017, 01:54:11 UTC |
18fc1bc | Andrew Kryczka | 09 March 2017, 01:43:11 UTC | minor changes for rate limiter test flakiness Summary: the 50%+ drained constraint wasn't working consistently in some of our test environments, maybe their resources are too low. relax the constraints a bit. Closes https://github.com/facebook/rocksdb/pull/1970 Differential Revision: D4679419 Pulled By: ajkr fbshipit-source-id: 3789cd8 | 09 March 2017, 01:54:11 UTC |
12ba00e | Aaron Gao | 09 March 2017, 01:12:04 UTC | Reset DBIter::saved_key_ with proper user key anywhere before pass to DBIter::FindNextUserEntry Summary: fix db_iter bug introduced by [facebook#1413](https://github.com/facebook/rocksdb/pull/1413) Closes https://github.com/facebook/rocksdb/pull/1962 Differential Revision: D4672369 Pulled By: lightmark fbshipit-source-id: 6a22953 | 09 March 2017, 01:24:11 UTC |
c9df05d | Dmitri Smirnov | 09 March 2017, 01:00:04 UTC | Fix random access alignment Summary: This fixes an issue when the most recent readers assume that alignment is always set even if direct io is off. Also adjust slightly appveyor script to run db_basic_test cases concurrently. Closes https://github.com/facebook/rocksdb/pull/1959 Differential Revision: D4671972 Pulled By: IslamAbdelRahman fbshipit-source-id: 1886620 | 09 March 2017, 01:09:11 UTC |
f649915 | Islam AbdelRahman | 08 March 2017, 02:19:02 UTC | Add Bulkoading IngestExternalFile blog post Summary: new blog post for bulkoading Closes https://github.com/facebook/rocksdb/pull/1883 Differential Revision: D4671984 Pulled By: IslamAbdelRahman fbshipit-source-id: 3450860 | 08 March 2017, 02:24:10 UTC |
54b4341 | Maysam Yabandeh | 07 March 2017, 21:48:02 UTC | Builders for partition filter Summary: This is the second split of this pull request: https://github.com/facebook/rocksdb/pull/1891 which includes only the builder part. The testing will be included in the third split, where the reader is also included. Closes https://github.com/facebook/rocksdb/pull/1952 Differential Revision: D4660272 Pulled By: maysamyabandeh fbshipit-source-id: 36b3cf0 | 07 March 2017, 21:54:12 UTC |
97edc72 | Sagar Vemuri | 07 March 2017, 19:50:02 UTC | Add a memtable-only iterator Summary: This PR is to support a way to iterate over all the keys that are just in memtables. Closes https://github.com/facebook/rocksdb/pull/1953 Differential Revision: D4663500 Pulled By: sagar0 fbshipit-source-id: 144e177 | 07 March 2017, 19:54:10 UTC |
7220296 | Leonidas Galanis | 07 March 2017, 19:06:53 UTC | fix db_sst_test flakiness Summary: db_sst_test had been flaky occasionally in the following way: reached_max_space_on_compaction can in very rare cases be 0. This happens when the limit on maximum allowable space set using SetMaxAllowedSpaceUsage is hit during flush for all test db sizes (1,2,4,8 and 10MB).The fix clears the error returned when the the space limit is reached during flush. This ensures that the compaction call back will always be called. The runtime is increased slightly because the 1MB loop writes more data and hits the limit during multiple flushes until compaction is scheduled. Closes https://github.com/facebook/rocksdb/pull/1861 Differential Revision: D4557396 Pulled By: lgalanis fbshipit-source-id: ff778d1 | 07 March 2017, 19:24:13 UTC |
5f65dc8 | Matteo Merli | 07 March 2017, 06:13:53 UTC | Expose DB::DeleteRange and WriteBath::DeleteRange in Java Summary: Added JNI wrapper from `DeleteRange` methods Closes https://github.com/facebook/rocksdb/pull/1951 Differential Revision: D4657746 Pulled By: yiwu-arbug fbshipit-source-id: 3fc7ab8 | 07 March 2017, 06:24:26 UTC |
58b12df | Reid Horuff | 06 March 2017, 23:04:36 UTC | Set logs as getting flushed before releasing lock, race condition fix Summary: Relating to #1903: In MaybeFlushColumnFamilies() we want to modify the 'getting_flushed' flag before releasing the db mutex when SwitchMemtable() is called. The following 2 actions need to be atomic in MaybeFlushColumnFamilies() - getting_flushed is false on oldest log - we determine that all CFs can be flushed to successfully release oldest log - we set getting_flushed = true on the oldest log. ------- - getting_flushed is false on oldest log - we determine that all CFs can NOT be flushed to successfully release oldest log - we set unable_to_flush_oldest_log_ = true on the oldest log. #### In the 2pc case: T1 enters function but is unable to flush all CFs to release log T1 sets unable_to_flush_oldest_log_ = true T1 begins flushing all CFs possible T2 enters function but is unable to flush all CFs to release log T2 sees unable_to_flush_oldes_log_ has been set so exits T3 enters function and will be able to flush all CFs to release oldest log T3 sets getting_flushed = true on oldes Closes https://github.com/facebook/rocksdb/pull/1909 Differential Revision: D4646235 Pulled By: reidHoruff fbshipit-source-id: c8d0447 | 06 March 2017, 23:09:11 UTC |
f8a4ea0 | Siying Dong | 06 March 2017, 17:25:58 UTC | Move db_test and external_sst_file_test out of Travis's MAC OS run Summary: After we have db_basic_test and external_sst_file_basic_test, we don't need to run db_test and external_sst_file_test in Travis's MAC OS run anymore. Move it out. Closes https://github.com/facebook/rocksdb/pull/1940 Differential Revision: D4659361 Pulled By: siying fbshipit-source-id: e64e291 | 06 March 2017, 17:39:15 UTC |
534581a | Maysam Yabandeh | 06 March 2017, 02:01:28 UTC | Fix a bug in tests in options operator= Summary: Note: Using the default operator= is an unsafe approach for Options since it destructs shared_ptr in the same order of their creation, in contrast to destructors which destructs them in the opposite order of creation. One particular problme is that the cache destructor might invoke callback functions that use Option members such as statistics. To work around this problem, we manually call destructor of table_facotry which eventually clears the block cache. Closes https://github.com/facebook/rocksdb/pull/1950 Differential Revision: D4655473 Pulled By: maysamyabandeh fbshipit-source-id: 6c4bbff | 06 March 2017, 02:09:09 UTC |
a2f7a51 | Maysam Yabandeh | 04 March 2017, 02:09:43 UTC | Refactoring Summary: This is the first split of https://github.com/facebook/rocksdb/pull/1891 and will be needed for the upcoming partitioned filter patch. Closes https://github.com/facebook/rocksdb/pull/1949 Differential Revision: D4652152 Pulled By: maysamyabandeh fbshipit-source-id: 9801778 | 04 March 2017, 02:24:12 UTC |
2a5daa0 | Andrew Kryczka | 03 March 2017, 21:17:39 UTC | Add stderr log level for ldb backup commands Summary: Also extracted the common logic into a base class, BackupableCommand. Closes https://github.com/facebook/rocksdb/pull/1939 Differential Revision: D4630121 Pulled By: ajkr fbshipit-source-id: 04bb067 | 03 March 2017, 21:24:15 UTC |
4561275 | Andrew Kryczka | 03 March 2017, 19:00:46 UTC | fix rate limiter test flakiness Summary: fix when elapsed time spans non-integral number of intervals since the rate limiter may still be drained during a partial interval. Closes https://github.com/facebook/rocksdb/pull/1948 Differential Revision: D4651304 Pulled By: ajkr fbshipit-source-id: b1f9e70 | 03 March 2017, 19:09:11 UTC |
7c80a6d | Andrew Kryczka | 03 March 2017, 01:40:24 UTC | Statistic for how often rate limiter is drained Summary: This is the metric I plan to use for adaptive rate limiting. The statistics are updated only if the rate limiter is drained by flush or compaction. I believe (but am not certain) that this is the normal case. The Statistics object is passed in RateLimiter::Request() to avoid requiring changes to client code, which would've been necessary if we passed it in the RateLimiter constructor. Closes https://github.com/facebook/rocksdb/pull/1946 Differential Revision: D4646489 Pulled By: ajkr fbshipit-source-id: d8e0161 | 03 March 2017, 01:54:15 UTC |
0ad5af4 | Andrew Kryczka | 03 March 2017, 01:17:06 UTC | Clarify VerifyBackup behavior Summary: It's non-obvious to users that using the same backup engine for creating/verifying provides better results than using separate backup engines, so add a comment in header. Closes https://github.com/facebook/rocksdb/pull/1942 Differential Revision: D4637865 Pulled By: ajkr fbshipit-source-id: e6efe24 | 03 March 2017, 01:24:11 UTC |
6fb9013 | Aaron Gao | 03 March 2017, 01:11:18 UTC | sanitize readahead when direct read enabled Summary: no readahead: readseq : 8.438 micros/op 118510 ops/sec; 13.1 MB/s sanitize to 10MB: readseq : 6.051 micros/op 165248 ops/sec; 18.3 MB/s Closes https://github.com/facebook/rocksdb/pull/1945 Differential Revision: D4645811 Pulled By: lightmark fbshipit-source-id: 5d63770 | 03 March 2017, 01:24:11 UTC |
f89b389 | Islam AbdelRahman | 03 March 2017, 00:45:07 UTC | Remove skip_table_builder_flush and default it to true Summary: This option is needed to be enabled for Direct IO and I cannot think of a reason where we need to disable it remove it and default it to true Closes https://github.com/facebook/rocksdb/pull/1944 Differential Revision: D4641088 Pulled By: IslamAbdelRahman fbshipit-source-id: d7085b9 | 03 March 2017, 00:54:10 UTC |
cc25398 | Andrew Kryczka | 02 March 2017, 18:45:04 UTC | Use more default options in db_bench Summary: The default behavior was too weird because, previously, we got the L0 file size limit (64MB) from Options default and L1+ file size limit (2MB) from the hardcoded value. We should get both from Options default. Closes https://github.com/facebook/rocksdb/pull/1943 Differential Revision: D4640301 Pulled By: ajkr fbshipit-source-id: fd8c0fd | 02 March 2017, 18:54:11 UTC |
8432bcf | Siying Dong | 02 March 2017, 18:08:49 UTC | Make compaction_pri settable through option string Summary: Closes https://github.com/facebook/rocksdb/pull/1941 Differential Revision: D4637253 Pulled By: siying fbshipit-source-id: a59dcdb | 02 March 2017, 18:24:12 UTC |
d5b607a | Siying Dong | 01 March 2017, 01:32:58 UTC | Make db_wal_test slightly faster Summary: Avoid to run db_wal_test in all the DB test options, and some small changes. Closes https://github.com/facebook/rocksdb/pull/1921 Differential Revision: D4622054 Pulled By: siying fbshipit-source-id: 890fd64 | 01 March 2017, 01:39:10 UTC |
ba4c77b | Siying Dong | 28 February 2017, 22:11:01 UTC | Divide external_sst_file_test Summary: Separate the platform dependent tests from external_sst_file_test. Only those tests need to run on platforms like OSX Closes https://github.com/facebook/rocksdb/pull/1923 Differential Revision: D4622461 Pulled By: siying fbshipit-source-id: d2d6f04 | 28 February 2017, 22:24:11 UTC |
e877afa | Aaron Gao | 28 February 2017, 19:05:08 UTC | Remove bulk loading and auto_roll_logger in rocksdb_lite Summary: shrink lite size Closes https://github.com/facebook/rocksdb/pull/1929 Differential Revision: D4622059 Pulled By: siying fbshipit-source-id: 050b796 | 28 February 2017, 19:09:11 UTC |
90d8355 | xiusir | 28 February 2017, 18:29:44 UTC | Fix the wrong address for PREFETCH in DynamicBloom::Prefetch Summary: - Change data_[b] to data_[b / 8] in DynamicBloom::Prefetch, as b means the b-th bit in data_ and data_[b / 8] is the proper byte in data_. Closes https://github.com/facebook/rocksdb/pull/1935 Differential Revision: D4628696 Pulled By: siying fbshipit-source-id: bc5a0c6 | 28 February 2017, 18:39:11 UTC |
08864df | Islam AbdelRahman | 28 February 2017, 01:36:06 UTC | Move advanced column family options to advanced_options.h Summary: For the sake of making our options simpler, we should keep options.h as simple as possible and move more advanced/less common options to advaned_options.h I started with ColumnFamilyOptions and also did some re-ordering I have moved all ColumnFamilyOptions to advanced_options.h and only left these options in options.h ``` const Comparator* comparator = BytewiseComparator(); std::shared_ptr<MergeOperator> merge_operator = nullptr; const CompactionFilter* compaction_filter = nullptr; std::shared_ptr<CompactionFilterFactory> compaction_filter_factory = nullptr; size_t write_buffer_size = 64 << 20; CompressionType compression; int level0_file_num_compaction_trigger = 4; bool disable_auto_compactions = false; ``` Please feel free to comment on specific options if you think they should be advanced or should not be Closes https://github.com/facebook/rocksdb/pull/1847 Differential Revision: D4519996 Pulled By: IslamAbdelRahman fbshipit-source-id: abebd9a | 28 February 2017, 01:54:14 UTC |
2ca2059 | Peter (Stig) Edwards | 28 February 2017, 01:22:50 UTC | Get unique_ptr to use delete[] for char[] in DumpMallocStats Summary: Avoid mismatched free() / delete / delete [] in DumpMallocStats Closes https://github.com/facebook/rocksdb/pull/1927 Differential Revision: D4622045 Pulled By: siying fbshipit-source-id: 1131b30 | 28 February 2017, 01:39:12 UTC |
253799c | Tamir Duberstein | 28 February 2017, 01:19:09 UTC | Add missing include for `abort()` Summary: Fixes #1233 (again). Closes https://github.com/facebook/rocksdb/pull/1931 Differential Revision: D4625289 Pulled By: ajkr fbshipit-source-id: 70e774e | 28 February 2017, 01:24:13 UTC |
c6d464a | Adam Retter | 28 February 2017, 00:26:12 UTC | Fixed various memory leaks and Java 8 JNI Compatibility Summary: I have manually audited the entire RocksJava code base. Sorry for the large pull-request, I have broken it down into many small atomic commits though. My initial intention was to fix the warnings that appear when running RocksJava on Java 8 with `-Xcheck:jni`, for example when running `make jtest` you would see many errors similar to: ``` WARNING in native method: JNI call made without checking exceptions when required to from CallObjectMethod WARNING in native method: JNI call made without checking exceptions when required to from CallVoidMethod WARNING in native method: JNI call made without checking exceptions when required to from CallStaticVoidMethod ... ``` A few of those warnings still remain, however they seem to come directly from the JVM and are not directly related to RocksJava; I am in contact with the OpenJDK hostpot-dev mailing list about these - http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-February/025981.html. As a result of fixing these, I realised we were not r Closes https://github.com/facebook/rocksdb/pull/1890 Differential Revision: D4591758 Pulled By: siying fbshipit-source-id: 7f7fdf4 | 28 February 2017, 00:39:12 UTC |
be3e556 | Islam AbdelRahman | 27 February 2017, 20:59:34 UTC | Fix unaligned reads in read cache Summary: - Fix unaligned reads in read cache by using RandomAccessFileReader - Allow read cache flags in db_bench Closes https://github.com/facebook/rocksdb/pull/1916 Differential Revision: D4610885 Pulled By: IslamAbdelRahman fbshipit-source-id: 2aa1dc8 | 27 February 2017, 21:09:12 UTC |
8ad0fcd | Siying Dong | 27 February 2017, 20:19:23 UTC | Separate small subset tests in DBTest Summary: Separate a smal subset of tests in DBTest to DBBasicTest. Tests in DBTest don't have to run in CI tests on platforms like OSX, as long as they are covered by Linux. Closes https://github.com/facebook/rocksdb/pull/1924 Differential Revision: D4616702 Pulled By: siying fbshipit-source-id: 13e6549 | 27 February 2017, 20:24:11 UTC |
6c951c4 | Siying Dong | 27 February 2017, 19:04:27 UTC | Run fewer tests in OSX Summary: Travis is short of OSX resource. Try to move platform independent test suites out of OSX Closes https://github.com/facebook/rocksdb/pull/1922 Differential Revision: D4616070 Pulled By: siying fbshipit-source-id: 786342c | 27 February 2017, 19:09:20 UTC |
f7997f1 | Aaron Gao | 24 February 2017, 05:59:26 UTC | add direct I/O to version notes 5.2.0 Summary: let users know this feature is ready Closes https://github.com/facebook/rocksdb/pull/1915 Differential Revision: D4610842 Pulled By: lightmark fbshipit-source-id: d102772 | 24 February 2017, 06:09:12 UTC |
3b8ba70 | Siying Dong | 24 February 2017, 02:36:20 UTC | Fix flaky DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2 Summary: A previous fix to DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2 didn't address the right problem. The problem is L0->L0 compaction is not trivial move in the scenario, not parallel compactions. Fix this. Closes https://github.com/facebook/rocksdb/pull/1911 Differential Revision: D4608955 Pulled By: siying fbshipit-source-id: 7a712cb | 24 February 2017, 02:39:13 UTC |
96c7e15 | Siying Dong | 24 February 2017, 02:35:54 UTC | Fix Java build Summary: The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version. Closes https://github.com/facebook/rocksdb/pull/1912 Differential Revision: D4609854 Pulled By: siying fbshipit-source-id: 3dc04f2 | 24 February 2017, 02:39:13 UTC |
e0b87af | Siying Dong | 23 February 2017, 23:54:47 UTC | Black list some slow valgrind tests Summary: valgrind tests always timeout with parallel run. Black list some slowest ones. It is better to run fewer tests than always have the tests timeout. Closes https://github.com/facebook/rocksdb/pull/1908 Differential Revision: D4607875 Pulled By: siying fbshipit-source-id: 7062664 | 24 February 2017, 00:09:11 UTC |
e67232c | Andrew Kryczka | 23 February 2017, 23:29:41 UTC | Handle failed Finish() in SST file writer Summary: The assertion in Abandon() fails when called after Finish() fails. Finish() already closes the builder so there's no need to call Abandon(). Closes https://github.com/facebook/rocksdb/pull/1901 Differential Revision: D4601373 Pulled By: ajkr fbshipit-source-id: e5678be | 23 February 2017, 23:39:16 UTC |
8efb5ff | Siying Dong | 23 February 2017, 22:53:03 UTC | [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp… Summary: …action The two options, min_partial_merge_operands and verify_checksums_in_compaction, are not seldom used. Remove them to reduce the total number of options. Also remove them from Java and C interface. Closes https://github.com/facebook/rocksdb/pull/1902 Differential Revision: D4601219 Pulled By: siying fbshipit-source-id: aad4cb2 | 23 February 2017, 23:09:12 UTC |
1ba2804 | Siying Dong | 23 February 2017, 20:03:28 UTC | Remove XFunc tests Summary: Xfunc is hardly used. Remove it to keep the code simple. Closes https://github.com/facebook/rocksdb/pull/1905 Differential Revision: D4603220 Pulled By: siying fbshipit-source-id: 731f96d | 23 February 2017, 20:09:11 UTC |
e7d902e | Aaron Gao | 23 February 2017, 19:21:23 UTC | add direct_io and compaction_readahead_size in db_stress Summary: add direct_io and compaction_readahead_size in db_stress test direct_io under db_stress with compaction_readahead_size enabled to capture bugs found in production. `./db_stress --allow_concurrent_memtable_write=0 --use_direct_reads --use_direct_writes --compaction_readahead_size=4096` Closes https://github.com/facebook/rocksdb/pull/1906 Differential Revision: D4604514 Pulled By: IslamAbdelRahman fbshipit-source-id: ebbf0ee | 23 February 2017, 19:39:14 UTC |
1ef5f50 | Aaron Gao | 23 February 2017, 19:17:49 UTC | detect logical sector size Summary: querying logical sector size from the device instead of hardcoding it for linux platform. Closes https://github.com/facebook/rocksdb/pull/1875 Differential Revision: D4591946 Pulled By: ajkr fbshipit-source-id: 4e9805c | 23 February 2017, 19:25:36 UTC |
ed50308 | Andrew Kryczka | 23 February 2017, 18:46:54 UTC | check backup directory exists before listing children Summary: InsertPathnameToSizeBytes() is called on shared/ and shared_checksum/ directories, which only exist for certain configurations. If we try to list a non-existent directory's contents, some Envs will dump an error message. Let's avoid this by checking whether the directory exists before listing its contents. Closes https://github.com/facebook/rocksdb/pull/1895 Differential Revision: D4596301 Pulled By: ajkr fbshipit-source-id: c809679 | 23 February 2017, 18:54:10 UTC |
4d7c06c | Giuseppe Ottaviano | 23 February 2017, 01:40:06 UTC | Make WriteBatchWithIndex moveble Summary: `WriteBatchWithIndex` has an incorrect implicitly-generated move constructor (it will copy the pointer causing a double-free on destruction). Just switch to `unique_ptr` so we get correct move semantics for free. Closes https://github.com/facebook/rocksdb/pull/1899 Differential Revision: D4598896 Pulled By: ajkr fbshipit-source-id: 2373d47 | 23 February 2017, 01:54:11 UTC |
5040414 | Andrew Kryczka | 23 February 2017, 01:34:12 UTC | Gracefully handle previous backup interrupted Summary: As the last step in backup creation, the .tmp directory is renamed omitting the .tmp suffix. In case the process terminates before this, the .tmp directory will be left behind. Even if this happens, we want future backups to succeed, so I added some checks/cleanup for this case. Closes https://github.com/facebook/rocksdb/pull/1896 Differential Revision: D4597323 Pulled By: ajkr fbshipit-source-id: 48900d8 | 23 February 2017, 01:39:12 UTC |
f206af5 | Aaron Gao | 22 February 2017, 22:48:09 UTC | add use_direct_io() to ReadaheadRandomAccessFile Summary: Missing this function will cause RandomAccessFileReader not doing alignment in Direct IO mode, which introduce an IOError: invalid argument. Closes https://github.com/facebook/rocksdb/pull/1900 Differential Revision: D4601261 Pulled By: lightmark fbshipit-source-id: c3eadf1 | 22 February 2017, 22:54:11 UTC |
0824934 | Aaron Gao | 22 February 2017, 18:21:39 UTC | truncate patch Summary: omit the override for the previous commit Closes https://github.com/facebook/rocksdb/pull/1898 Differential Revision: D4598743 Pulled By: lightmark fbshipit-source-id: f98a378 | 22 February 2017, 18:39:11 UTC |
286a36d | Aaron Gao | 22 February 2017, 18:00:25 UTC | posix writablefile truncate Summary: we occasionally missing this call so the file size will be wrong Closes https://github.com/facebook/rocksdb/pull/1894 Differential Revision: D4598446 Pulled By: lightmark fbshipit-source-id: 42b6ef5 | 22 February 2017, 18:09:14 UTC |
f0879e4 | Daniel Black | 22 February 2017, 00:33:54 UTC | Page size isn't always 4k on linux Summary: Some places autodetected. These are the two places that didn't. closes #1498 Still unsure if the following instances of 4 * 1024 need fixing in: util/io_posix.h include/rocksdb/table.h (appears to be blocksize and different) utilities/persistent_cache/block_cache_tier.cc utilities/persistent_cache/persistent_cache_test.h include/rocksdb/env.h util/env_posix.cc db/column_family.cc Closes https://github.com/facebook/rocksdb/pull/1499 Differential Revision: D4593640 Pulled By: yiwu-arbug fbshipit-source-id: efc48de | 22 February 2017, 00:39:14 UTC |
18eeb7b | Mike Kolupaev | 21 February 2017, 23:53:59 UTC | Fix interference between max_total_wal_size and db_write_buffer_size checks Summary: This is a trivial fix for OOMs we've seen a few days ago in logdevice. RocksDB get into the following state: (1) Write throughput is too high for flushes to keep up. Compactions are out of the picture - automatic compactions are disabled, and for manual compactions we don't care that much if they fall behind. We write to many CFs, with only a few L0 sst files in each, so compactions are not needed most of the time. (2) total_log_size_ is consistently greater than GetMaxTotalWalSize(). It doesn't get smaller since flushes are falling ever further behind. (3) Total size of memtables is way above db_write_buffer_size and keeps growing. But the write_buffer_manager_->ShouldFlush() is not checked because (2) prevents it (for no good reason, afaict; this is what this commit fixes). (4) Every call to WriteImpl() hits the MaybeFlushColumnFamilies() path. This keeps flushing the memtables one by one in order of increasing log file number. (5) No write stalling trigger is hit. We rely on max_write_buffer_number Closes https://github.com/facebook/rocksdb/pull/1893 Differential Revision: D4593590 Pulled By: yiwu-arbug fbshipit-source-id: af79c5f | 22 February 2017, 00:09:10 UTC |
1560b2f | Islam AbdelRahman | 21 February 2017, 20:42:52 UTC | Temporarly return deprecated functions to fix MongoRocks build Summary: MongoRocks is still using some deprecated functions, return them temporarily Closes https://github.com/facebook/rocksdb/pull/1892 Differential Revision: D4592451 Pulled By: IslamAbdelRahman fbshipit-source-id: 5e6be3e | 21 February 2017, 20:54:11 UTC |
2a0f3d0 | Aaron Gao | 21 February 2017, 18:11:04 UTC | level compaction expansion Summary: reimplement the compaction expansion on lower level. Considering such a case: input level file: 1[B E] 2[F G] 3[H I] 4 [J M] output level file: 5[A C] 6[D K] 7[L O] If we initially pick file 2, now we will compact file 2 and 6. But we can safely compact 2, 3 and 6 without expanding the output level. The previous code is messy and wrong. In this diff, I first determine the input range [a, b], and output range [c, d], then we get the range [e,f] = [min(a, c), max(b, d] and put all eligible clean-cut files within [e, f] into this compaction. **Note: clean-cut means the files don't have the same user key on the boundaries of some files that are not chosen in this compaction**. Closes https://github.com/facebook/rocksdb/pull/1760 Differential Revision: D4395564 Pulled By: lightmark fbshipit-source-id: 2dc2c5c | 21 February 2017, 18:24:17 UTC |
ebc8a79 | Yulia Kartseva | 18 February 2017, 19:54:49 UTC | alignment is on in ReadaheadRandomAccessFile::Read() Summary: Closes https://github.com/facebook/rocksdb/pull/1857 Differential Revision: D4534518 Pulled By: wat-ze-hex fbshipit-source-id: b456946 | 18 February 2017, 20:09:12 UTC |
381fd32 | Yi Wu | 17 February 2017, 23:15:23 UTC | Remove timeout_hint_us from WriteOptions Summary: The option has been deprecated for two years and has no effect. Removing. Closes https://github.com/facebook/rocksdb/pull/1866 Differential Revision: D4555203 Pulled By: yiwu-arbug fbshipit-source-id: c48f627 | 17 February 2017, 23:24:17 UTC |
fce7a6e | Islam AbdelRahman | 17 February 2017, 21:36:46 UTC | Fail IngestExternalFile when bg_error_ exists Summary: Fail IngestExternalFile() when bg_error_ exists Closes https://github.com/facebook/rocksdb/pull/1881 Differential Revision: D4580621 Pulled By: IslamAbdelRahman fbshipit-source-id: 1194913 | 17 February 2017, 21:39:17 UTC |
a618a16 | Marcin Dlugajczyk | 17 February 2017, 18:47:35 UTC | New subcode for IOError to detect the ESTALE errno Summary: I'd like to propose a patch to expose a new IOError type with subcode kStaleFile to allow to detect when ESTALE error is returned. This allows the rocksdb consumers to handle this error separately from other IOErrors. I've also added a missing string representation for the kDeadlock subcode, I believe calling ToString() on Status object with that subcode would result in an out of band access in the msgs array, Please let me know if you have any questions or would like me to make any changes to this pull request. Closes https://github.com/facebook/rocksdb/pull/1748 Differential Revision: D4387675 Pulled By: IslamAbdelRahman fbshipit-source-id: 67feb13 | 17 February 2017, 18:54:13 UTC |
7ab0051 | Islam AbdelRahman | 17 February 2017, 18:46:32 UTC | Remove deprecated DB::AddFile and DB::CompactRange Summary: Remove functions that we deprecated long time ago in db.h Closes https://github.com/facebook/rocksdb/pull/1878 Differential Revision: D4576521 Pulled By: IslamAbdelRahman fbshipit-source-id: dfddad1 | 17 February 2017, 18:54:13 UTC |
4016673 | notna123 | 17 February 2017, 01:11:13 UTC | Adding Dlang to the list Summary: Closes https://github.com/facebook/rocksdb/pull/1877 Differential Revision: D4576056 Pulled By: IslamAbdelRahman fbshipit-source-id: 0f19275 | 17 February 2017, 01:24:10 UTC |
756c592 | Shu Zhang | 17 February 2017, 01:02:51 UTC | Allow adding external v1 sst file with no global seqno support Summary: This is a follow up fix for https://github.com/facebook/rocksdb/pull/1783. After it, we should be able to ingest external v1 sst files with no global seqno field. Closes https://github.com/facebook/rocksdb/pull/1874 Differential Revision: D4576194 Pulled By: IslamAbdelRahman fbshipit-source-id: 5b34a3e | 17 February 2017, 01:09:12 UTC |
aa0298f | Volker Mische | 16 February 2017, 21:14:53 UTC | Remove trailing whitespace from examples Makefile Summary: The Makefile in the examples directory contained an empty line contain a tab character. This made my Emacs ask on every save `Suspicious line 10. Save anyway? (y or n)` Closes https://github.com/facebook/rocksdb/pull/1872 Differential Revision: D4573881 Pulled By: siying fbshipit-source-id: fb3b4ee | 16 February 2017, 21:24:09 UTC |
db2b4eb | Aaron Gao | 16 February 2017, 18:25:06 UTC | avoid direct io in rocksdb_lite Summary: fix lite bugs disable direct io in lite mode Closes https://github.com/facebook/rocksdb/pull/1870 Differential Revision: D4559866 Pulled By: yiwu-arbug fbshipit-source-id: 3761c51 | 16 February 2017, 18:39:13 UTC |
43e9f01 | Andrew Kryczka | 15 February 2017, 19:16:34 UTC | Fix repair_test on ROCKSDB_LITE Summary: RepairDB isn't included in rocksdb lite, so don't test it. Closes https://github.com/facebook/rocksdb/pull/1873 Differential Revision: D4565094 Pulled By: ajkr fbshipit-source-id: 8cc0898 | 15 February 2017, 19:24:12 UTC |
7106a99 | Xiaofei Du | 15 February 2017, 02:15:05 UTC | Use monotonic time points in write_controller.cc and rate_limiter.cc Summary: NowMicros() provides non-monotonic time. When wall clock is synchronized or changed, the non-monotonicity time points will affect write rate controllers. This patch changes write_controller.cc and rate_limiter.cc to use monotonic time points. Closes https://github.com/facebook/rocksdb/pull/1865 Differential Revision: D4561732 Pulled By: siying fbshipit-source-id: 95ece62 | 15 February 2017, 02:24:24 UTC |
c2247dc | Yi Wu | 14 February 2017, 02:44:21 UTC | Make DBImpl::has_unpersisted_data_ atomic Summary: Seems to me `has_unpersisted_data_` is read from read thread and write from write thread concurrently without synchronization. Making it an atomic. I update the logic not because seeing any problem with it, but it just feel confusing. Closes https://github.com/facebook/rocksdb/pull/1869 Differential Revision: D4555837 Pulled By: yiwu-arbug fbshipit-source-id: eff2ab8 | 14 February 2017, 02:54:13 UTC |
eb912a9 | Sagar Vemuri | 13 February 2017, 18:54:38 UTC | Remove disableDataSync option Summary: Remove disableDataSync, and another similarly named disable_data_sync options. This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods. Closes https://github.com/facebook/rocksdb/pull/1859 Differential Revision: D4541292 Pulled By: sagar0 fbshipit-source-id: 5b3a6ca | 13 February 2017, 19:09:13 UTC |
0227c16 | Adam Retter | 13 February 2017, 06:53:42 UTC | Update static library versions and add checksums Summary: The previous version of zlib is no longer available. I have also updated the versions of the other static libraries and added checkum checks for the downloads; This is related to https://github.com/facebook/rocksdb/issues/1769 Closes https://github.com/facebook/rocksdb/pull/1863 Differential Revision: D4550742 Pulled By: yiwu-arbug fbshipit-source-id: 4414150 | 13 February 2017, 07:09:09 UTC |
b3aae4d | Andrew Kryczka | 11 February 2017, 02:04:05 UTC | Add repair_test to make check Summary: needed so we can proactively find issues like #1858 Closes https://github.com/facebook/rocksdb/pull/1862 Differential Revision: D4545854 Pulled By: ajkr fbshipit-source-id: d77fcb7 | 11 February 2017, 02:09:18 UTC |
421ce7c | Adam Retter | 10 February 2017, 22:58:07 UTC | Add support for JNI Library on Linux on PowerPC. Summary: Closes https://github.com/facebook/rocksdb/issues/1317 Closes https://github.com/facebook/rocksdb/pull/1845 Differential Revision: D4546491 Pulled By: siying fbshipit-source-id: 86190b4 | 10 February 2017, 23:09:10 UTC |
9afa20c | Siying Dong | 10 February 2017, 19:57:23 UTC | Increase build version and HISTORY.md for releasing 5.2 Summary: Also clean up HISTORY.md a little bit. Closes https://github.com/facebook/rocksdb/pull/1854 Differential Revision: D4539556 Pulled By: siying fbshipit-source-id: 567391e | 10 February 2017, 20:09:15 UTC |
a5adda0 | Dmitri Smirnov | 10 February 2017, 18:51:08 UTC | Fix repair issues Summary: Record the first parsed sequence number as the minimum so we can find the true minimum otherwise everything is larger than zero. Fix the comparator name comparision. Closes https://github.com/facebook/rocksdb/pull/1858 Differential Revision: D4544365 Pulled By: ajkr fbshipit-source-id: 439cbc2 | 10 February 2017, 18:54:12 UTC |
b48e477 | Andrew Kryczka | 09 February 2017, 00:07:29 UTC | Consolidate file cutting logic in compaction loop Summary: It was really annoying to have two places (top and bottom of compaction loop) where we cut output files. I had bugs in both DeleteRange and dictionary compression due to updating only one of the two. This diff consolidates the file-cutting logic to the bottom of the compaction loop. Keep in mind that my goal with input_status is to be consistent with the past behavior, even though I'm not sure it's ideal. Closes https://github.com/facebook/rocksdb/pull/1832 Differential Revision: D4503038 Pulled By: ajkr fbshipit-source-id: 7da5213 | 09 February 2017, 00:24:17 UTC |
ac2a77a | Maysam Yabandeh | 08 February 2017, 22:10:55 UTC | Announce the experimetnal two-level index feature in HISTORY.md Summary: Announce the experimetnal two-level index feature in HISTORY.md. Also updated the default for index_per_partition to 1024. Closes https://github.com/facebook/rocksdb/pull/1855 Differential Revision: D4530102 Pulled By: maysamyabandeh fbshipit-source-id: b0fc6ff | 08 February 2017, 22:24:10 UTC |
c4a37dc | Maysam Yabandeh | 08 February 2017, 18:31:59 UTC | Print the missed last layer in cfstats Summary: Printing compaction stats used to operate on two variable: number_levels_: for printing the layer num_levels_to_check: for updating the compaction score After this commit: 361010d44738de48ffc4fd9add70caa0891a0719 these two are mixed up and as a result the last layer might not be printed out: https://fb.facebook.com/groups/rocksdb.internal/permalink/1315716625143616/ number_levels_ was used to decide which layers to print: https://github.com/facebook/rocksdb/blob/672300f47f72b28c164fdf98a08171c09e311205/db/internal_stats.cc#L753 but after the patch it is based on the return value of DumpCFMapStats https://github.com/facebook/rocksdb/blob/361010d44738de48ffc4fd9add70caa0891a0719/db/internal_stats.cc#L929 which returns num_levels_to_check: https://github.com/facebook/rocksdb/blob/361010d44738de48ffc4fd9add70caa0891a0719/db/internal_stats.cc#L917 Closes https://github.com/facebook/rocksdb/pull/1853 Differential Revision: D4529280 Pulled By: maysamyabandeh fbshipit-source-id: 3fd9448 | 08 February 2017, 18:39:15 UTC |
a12818a | Maysam Yabandeh | 08 February 2017, 16:37:18 UTC | Blog post for 5.1.2 release Summary: Closes https://github.com/facebook/rocksdb/pull/1849 Differential Revision: D4528696 Pulled By: maysamyabandeh fbshipit-source-id: c12f3a2 | 08 February 2017, 16:54:20 UTC |
3b4ac80 | Andrew Kryczka | 07 February 2017, 19:36:01 UTC | Clarify ldb column family argument Summary: move the argument description to the right section, make it clearer that the '=' sign is required, and use it in one subcommand where it seemed forgotten before. Closes https://github.com/facebook/rocksdb/pull/1840 Differential Revision: D4515096 Pulled By: ajkr fbshipit-source-id: 7b5b1c1 | 07 February 2017, 19:54:10 UTC |
d70ce7e | Andrew Kryczka | 07 February 2017, 19:34:14 UTC | Move db_bench flags out of unnamed namespace Summary: I want to be able to, e.g., DECLARE_string(statistics_string); in my application such that I can override the default value of statistics_string. For this to work, we need to remove the unnamed namespace containing all the flags, and make sure all variables/functions covered by that namespace are static. Replaces #1828 due to internal tool issues. Closes https://github.com/facebook/rocksdb/pull/1844 Differential Revision: D4515124 Pulled By: ajkr fbshipit-source-id: 23b695e | 07 February 2017, 19:39:12 UTC |
186c7ee | Andrew Kryczka | 07 February 2017, 19:33:14 UTC | Remove incorrect statistics warning Summary: statistics are always concurrently updated regardless of whether a single instance or multiple instances are used. remove the warning since it's unnecessarily preventing people from sharing stats objects across rocksdb instances. replace #1819 which was having some problem with our internal tools. Closes https://github.com/facebook/rocksdb/pull/1826 Differential Revision: D4495970 Pulled By: ajkr fbshipit-source-id: 6601f61 | 07 February 2017, 19:39:12 UTC |
53bb015 | James Sun | 07 February 2017, 18:35:15 UTC | [rocksdb][PR] compaction_style and compaction_pri should output their value as a st… Summary: …ring Replace the numerical output for compaction_style and compaction_pri with strings Closes https://github.com/facebook/rocksdb/pull/1817 Differential Revision: D4482796 Pulled By: highker fbshipit-source-id: 5785768 | 07 February 2017, 18:39:12 UTC |
69d5262 | Maysam Yabandeh | 07 February 2017, 00:29:29 UTC | Two-level Indexes Summary: Partition Index blocks and use a Partition-index as a 2nd level index. The two-level index can be used by setting BlockBasedTableOptions::kTwoLevelIndexSearch as the index type and configuring BlockBasedTableOptions::index_per_partition t15539501 Closes https://github.com/facebook/rocksdb/pull/1814 Differential Revision: D4473535 Pulled By: maysamyabandeh fbshipit-source-id: bffb87e | 07 February 2017, 00:39:12 UTC |
0a4cdde | Dmitri Smirnov | 06 February 2017, 22:43:55 UTC | Windows thread Summary: introduce new methods into a public threadpool interface, - allow submission of std::functions as they allow greater flexibility. - add Joining methods to the implementation to join scheduled and submitted jobs with an option to cancel jobs that did not start executing. - Remove ugly `#ifdefs` between pthread and std implementation, make it uniform. - introduce pimpl for a drop in replacement of the implementation - Introduce rocksdb::port::Thread typedef which is a replacement for std::thread. On Posix Thread defaults as before std::thread. - Implement WindowsThread that allocates memory in a more controllable manner than windows std::thread with a replaceable implementation. - should be no functionality changes. Closes https://github.com/facebook/rocksdb/pull/1823 Differential Revision: D4492902 Pulled By: siying fbshipit-source-id: c74cb11 | 06 February 2017, 22:54:18 UTC |
1aaa898 | Vitaliy Liptchinsky | 06 February 2017, 22:42:38 UTC | Adding GetApproximateMemTableStats method Summary: Added method that returns approx num of entries as well as size for memtables. Closes https://github.com/facebook/rocksdb/pull/1841 Differential Revision: D4511990 Pulled By: VitaliyLi fbshipit-source-id: 9a4576e | 06 February 2017, 22:54:16 UTC |
9fc23c5 | Rhys Parry | 03 February 2017, 21:22:18 UTC | Use gcc-4.9-glibc-2.20-fb python in precommit_checker Summary: The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed soon. Closes https://github.com/facebook/rocksdb/pull/1839 Differential Revision: D4509684 Pulled By: siying fbshipit-source-id: 3efe296 | 03 February 2017, 21:39:18 UTC |
b797e42 | Andrew Kryczka | 03 February 2017, 20:37:02 UTC | Dump compression dictionary meta-block Summary: make sst_dump print size/contents of the dictionary meta-block for easier debugging Closes https://github.com/facebook/rocksdb/pull/1837 Differential Revision: D4506399 Pulled By: ajkr fbshipit-source-id: b9bf668 | 03 February 2017, 20:39:16 UTC |
036d668 | Siying Dong | 03 February 2017, 19:35:22 UTC | Fix wrong result in data race case related to Get() Summary: In theory, Get() can get a wrong result, if it races in a special with with flush. The bug can be reproduced in DBTest2.GetRaceFlush. Fix this bug by getting snapshot after referencing the super version. Closes https://github.com/facebook/rocksdb/pull/1816 Differential Revision: D4475958 Pulled By: siying fbshipit-source-id: bd9e67a | 03 February 2017, 19:39:15 UTC |
574b543 | Islam AbdelRahman | 03 February 2017, 00:38:40 UTC | Rename merger.h -> merging_iterator.h Summary: merger.h was always a confusing name for me, simply give the file a better name Closes https://github.com/facebook/rocksdb/pull/1836 Differential Revision: D4505357 Pulled By: IslamAbdelRahman fbshipit-source-id: 07b28d8 | 03 February 2017, 00:54:19 UTC |