https://github.com/facebook/rocksdb

sort by:
Revision Author Date Message Commit Date
f89b389 Remove skip_table_builder_flush and default it to true Summary: This option is needed to be enabled for Direct IO and I cannot think of a reason where we need to disable it remove it and default it to true Closes https://github.com/facebook/rocksdb/pull/1944 Differential Revision: D4641088 Pulled By: IslamAbdelRahman fbshipit-source-id: d7085b9 03 March 2017, 00:54:10 UTC
cc25398 Use more default options in db_bench Summary: The default behavior was too weird because, previously, we got the L0 file size limit (64MB) from Options default and L1+ file size limit (2MB) from the hardcoded value. We should get both from Options default. Closes https://github.com/facebook/rocksdb/pull/1943 Differential Revision: D4640301 Pulled By: ajkr fbshipit-source-id: fd8c0fd 02 March 2017, 18:54:11 UTC
8432bcf Make compaction_pri settable through option string Summary: Closes https://github.com/facebook/rocksdb/pull/1941 Differential Revision: D4637253 Pulled By: siying fbshipit-source-id: a59dcdb 02 March 2017, 18:24:12 UTC
d5b607a Make db_wal_test slightly faster Summary: Avoid to run db_wal_test in all the DB test options, and some small changes. Closes https://github.com/facebook/rocksdb/pull/1921 Differential Revision: D4622054 Pulled By: siying fbshipit-source-id: 890fd64 01 March 2017, 01:39:10 UTC
ba4c77b Divide external_sst_file_test Summary: Separate the platform dependent tests from external_sst_file_test. Only those tests need to run on platforms like OSX Closes https://github.com/facebook/rocksdb/pull/1923 Differential Revision: D4622461 Pulled By: siying fbshipit-source-id: d2d6f04 28 February 2017, 22:24:11 UTC
e877afa Remove bulk loading and auto_roll_logger in rocksdb_lite Summary: shrink lite size Closes https://github.com/facebook/rocksdb/pull/1929 Differential Revision: D4622059 Pulled By: siying fbshipit-source-id: 050b796 28 February 2017, 19:09:11 UTC
90d8355 Fix the wrong address for PREFETCH in DynamicBloom::Prefetch Summary: - Change data_[b] to data_[b / 8] in DynamicBloom::Prefetch, as b means the b-th bit in data_ and data_[b / 8] is the proper byte in data_. Closes https://github.com/facebook/rocksdb/pull/1935 Differential Revision: D4628696 Pulled By: siying fbshipit-source-id: bc5a0c6 28 February 2017, 18:39:11 UTC
08864df Move advanced column family options to advanced_options.h Summary: For the sake of making our options simpler, we should keep options.h as simple as possible and move more advanced/less common options to advaned_options.h I started with ColumnFamilyOptions and also did some re-ordering I have moved all ColumnFamilyOptions to advanced_options.h and only left these options in options.h ``` const Comparator* comparator = BytewiseComparator(); std::shared_ptr<MergeOperator> merge_operator = nullptr; const CompactionFilter* compaction_filter = nullptr; std::shared_ptr<CompactionFilterFactory> compaction_filter_factory = nullptr; size_t write_buffer_size = 64 << 20; CompressionType compression; int level0_file_num_compaction_trigger = 4; bool disable_auto_compactions = false; ``` Please feel free to comment on specific options if you think they should be advanced or should not be Closes https://github.com/facebook/rocksdb/pull/1847 Differential Revision: D4519996 Pulled By: IslamAbdelRahman fbshipit-source-id: abebd9a 28 February 2017, 01:54:14 UTC
2ca2059 Get unique_ptr to use delete[] for char[] in DumpMallocStats Summary: Avoid mismatched free() / delete / delete [] in DumpMallocStats Closes https://github.com/facebook/rocksdb/pull/1927 Differential Revision: D4622045 Pulled By: siying fbshipit-source-id: 1131b30 28 February 2017, 01:39:12 UTC
253799c Add missing include for `abort()` Summary: Fixes #1233 (again). Closes https://github.com/facebook/rocksdb/pull/1931 Differential Revision: D4625289 Pulled By: ajkr fbshipit-source-id: 70e774e 28 February 2017, 01:24:13 UTC
c6d464a Fixed various memory leaks and Java 8 JNI Compatibility Summary: I have manually audited the entire RocksJava code base. Sorry for the large pull-request, I have broken it down into many small atomic commits though. My initial intention was to fix the warnings that appear when running RocksJava on Java 8 with `-Xcheck:jni`, for example when running `make jtest` you would see many errors similar to: ``` WARNING in native method: JNI call made without checking exceptions when required to from CallObjectMethod WARNING in native method: JNI call made without checking exceptions when required to from CallVoidMethod WARNING in native method: JNI call made without checking exceptions when required to from CallStaticVoidMethod ... ``` A few of those warnings still remain, however they seem to come directly from the JVM and are not directly related to RocksJava; I am in contact with the OpenJDK hostpot-dev mailing list about these - http://mail.openjdk.java.net/pipermail/hotspot-dev/2017-February/025981.html. As a result of fixing these, I realised we were not r Closes https://github.com/facebook/rocksdb/pull/1890 Differential Revision: D4591758 Pulled By: siying fbshipit-source-id: 7f7fdf4 28 February 2017, 00:39:12 UTC
be3e556 Fix unaligned reads in read cache Summary: - Fix unaligned reads in read cache by using RandomAccessFileReader - Allow read cache flags in db_bench Closes https://github.com/facebook/rocksdb/pull/1916 Differential Revision: D4610885 Pulled By: IslamAbdelRahman fbshipit-source-id: 2aa1dc8 27 February 2017, 21:09:12 UTC
8ad0fcd Separate small subset tests in DBTest Summary: Separate a smal subset of tests in DBTest to DBBasicTest. Tests in DBTest don't have to run in CI tests on platforms like OSX, as long as they are covered by Linux. Closes https://github.com/facebook/rocksdb/pull/1924 Differential Revision: D4616702 Pulled By: siying fbshipit-source-id: 13e6549 27 February 2017, 20:24:11 UTC
6c951c4 Run fewer tests in OSX Summary: Travis is short of OSX resource. Try to move platform independent test suites out of OSX Closes https://github.com/facebook/rocksdb/pull/1922 Differential Revision: D4616070 Pulled By: siying fbshipit-source-id: 786342c 27 February 2017, 19:09:20 UTC
f7997f1 add direct I/O to version notes 5.2.0 Summary: let users know this feature is ready Closes https://github.com/facebook/rocksdb/pull/1915 Differential Revision: D4610842 Pulled By: lightmark fbshipit-source-id: d102772 24 February 2017, 06:09:12 UTC
3b8ba70 Fix flaky DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2 Summary: A previous fix to DBTestUniversalCompaction.UniversalCompactionTrivialMoveTest2 didn't address the right problem. The problem is L0->L0 compaction is not trivial move in the scenario, not parallel compactions. Fix this. Closes https://github.com/facebook/rocksdb/pull/1911 Differential Revision: D4608955 Pulled By: siying fbshipit-source-id: 7a712cb 24 February 2017, 02:39:13 UTC
96c7e15 Fix Java build Summary: The PATH update should put the Java path in the beginning, rather than the end. Otherwise, it will be overwritten. Also upgrade the Java version. Closes https://github.com/facebook/rocksdb/pull/1912 Differential Revision: D4609854 Pulled By: siying fbshipit-source-id: 3dc04f2 24 February 2017, 02:39:13 UTC
e0b87af Black list some slow valgrind tests Summary: valgrind tests always timeout with parallel run. Black list some slowest ones. It is better to run fewer tests than always have the tests timeout. Closes https://github.com/facebook/rocksdb/pull/1908 Differential Revision: D4607875 Pulled By: siying fbshipit-source-id: 7062664 24 February 2017, 00:09:11 UTC
e67232c Handle failed Finish() in SST file writer Summary: The assertion in Abandon() fails when called after Finish() fails. Finish() already closes the builder so there's no need to call Abandon(). Closes https://github.com/facebook/rocksdb/pull/1901 Differential Revision: D4601373 Pulled By: ajkr fbshipit-source-id: e5678be 23 February 2017, 23:39:16 UTC
8efb5ff [rocksdb][PR] Remove option min_partial_merge_operands and verify_checksums_in_comp… Summary: …action The two options, min_partial_merge_operands and verify_checksums_in_compaction, are not seldom used. Remove them to reduce the total number of options. Also remove them from Java and C interface. Closes https://github.com/facebook/rocksdb/pull/1902 Differential Revision: D4601219 Pulled By: siying fbshipit-source-id: aad4cb2 23 February 2017, 23:09:12 UTC
1ba2804 Remove XFunc tests Summary: Xfunc is hardly used. Remove it to keep the code simple. Closes https://github.com/facebook/rocksdb/pull/1905 Differential Revision: D4603220 Pulled By: siying fbshipit-source-id: 731f96d 23 February 2017, 20:09:11 UTC
e7d902e add direct_io and compaction_readahead_size in db_stress Summary: add direct_io and compaction_readahead_size in db_stress test direct_io under db_stress with compaction_readahead_size enabled to capture bugs found in production. `./db_stress --allow_concurrent_memtable_write=0 --use_direct_reads --use_direct_writes --compaction_readahead_size=4096` Closes https://github.com/facebook/rocksdb/pull/1906 Differential Revision: D4604514 Pulled By: IslamAbdelRahman fbshipit-source-id: ebbf0ee 23 February 2017, 19:39:14 UTC
1ef5f50 detect logical sector size Summary: querying logical sector size from the device instead of hardcoding it for linux platform. Closes https://github.com/facebook/rocksdb/pull/1875 Differential Revision: D4591946 Pulled By: ajkr fbshipit-source-id: 4e9805c 23 February 2017, 19:25:36 UTC
ed50308 check backup directory exists before listing children Summary: InsertPathnameToSizeBytes() is called on shared/ and shared_checksum/ directories, which only exist for certain configurations. If we try to list a non-existent directory's contents, some Envs will dump an error message. Let's avoid this by checking whether the directory exists before listing its contents. Closes https://github.com/facebook/rocksdb/pull/1895 Differential Revision: D4596301 Pulled By: ajkr fbshipit-source-id: c809679 23 February 2017, 18:54:10 UTC
4d7c06c Make WriteBatchWithIndex moveble Summary: `WriteBatchWithIndex` has an incorrect implicitly-generated move constructor (it will copy the pointer causing a double-free on destruction). Just switch to `unique_ptr` so we get correct move semantics for free. Closes https://github.com/facebook/rocksdb/pull/1899 Differential Revision: D4598896 Pulled By: ajkr fbshipit-source-id: 2373d47 23 February 2017, 01:54:11 UTC
5040414 Gracefully handle previous backup interrupted Summary: As the last step in backup creation, the .tmp directory is renamed omitting the .tmp suffix. In case the process terminates before this, the .tmp directory will be left behind. Even if this happens, we want future backups to succeed, so I added some checks/cleanup for this case. Closes https://github.com/facebook/rocksdb/pull/1896 Differential Revision: D4597323 Pulled By: ajkr fbshipit-source-id: 48900d8 23 February 2017, 01:39:12 UTC
f206af5 add use_direct_io() to ReadaheadRandomAccessFile Summary: Missing this function will cause RandomAccessFileReader not doing alignment in Direct IO mode, which introduce an IOError: invalid argument. Closes https://github.com/facebook/rocksdb/pull/1900 Differential Revision: D4601261 Pulled By: lightmark fbshipit-source-id: c3eadf1 22 February 2017, 22:54:11 UTC
0824934 truncate patch Summary: omit the override for the previous commit Closes https://github.com/facebook/rocksdb/pull/1898 Differential Revision: D4598743 Pulled By: lightmark fbshipit-source-id: f98a378 22 February 2017, 18:39:11 UTC
286a36d posix writablefile truncate Summary: we occasionally missing this call so the file size will be wrong Closes https://github.com/facebook/rocksdb/pull/1894 Differential Revision: D4598446 Pulled By: lightmark fbshipit-source-id: 42b6ef5 22 February 2017, 18:09:14 UTC
f0879e4 Page size isn't always 4k on linux Summary: Some places autodetected. These are the two places that didn't. closes #1498 Still unsure if the following instances of 4 * 1024 need fixing in: util/io_posix.h include/rocksdb/table.h (appears to be blocksize and different) utilities/persistent_cache/block_cache_tier.cc utilities/persistent_cache/persistent_cache_test.h include/rocksdb/env.h util/env_posix.cc db/column_family.cc Closes https://github.com/facebook/rocksdb/pull/1499 Differential Revision: D4593640 Pulled By: yiwu-arbug fbshipit-source-id: efc48de 22 February 2017, 00:39:14 UTC
18eeb7b Fix interference between max_total_wal_size and db_write_buffer_size checks Summary: This is a trivial fix for OOMs we've seen a few days ago in logdevice. RocksDB get into the following state: (1) Write throughput is too high for flushes to keep up. Compactions are out of the picture - automatic compactions are disabled, and for manual compactions we don't care that much if they fall behind. We write to many CFs, with only a few L0 sst files in each, so compactions are not needed most of the time. (2) total_log_size_ is consistently greater than GetMaxTotalWalSize(). It doesn't get smaller since flushes are falling ever further behind. (3) Total size of memtables is way above db_write_buffer_size and keeps growing. But the write_buffer_manager_->ShouldFlush() is not checked because (2) prevents it (for no good reason, afaict; this is what this commit fixes). (4) Every call to WriteImpl() hits the MaybeFlushColumnFamilies() path. This keeps flushing the memtables one by one in order of increasing log file number. (5) No write stalling trigger is hit. We rely on max_write_buffer_number Closes https://github.com/facebook/rocksdb/pull/1893 Differential Revision: D4593590 Pulled By: yiwu-arbug fbshipit-source-id: af79c5f 22 February 2017, 00:09:10 UTC
1560b2f Temporarly return deprecated functions to fix MongoRocks build Summary: MongoRocks is still using some deprecated functions, return them temporarily Closes https://github.com/facebook/rocksdb/pull/1892 Differential Revision: D4592451 Pulled By: IslamAbdelRahman fbshipit-source-id: 5e6be3e 21 February 2017, 20:54:11 UTC
2a0f3d0 level compaction expansion Summary: reimplement the compaction expansion on lower level. Considering such a case: input level file: 1[B E] 2[F G] 3[H I] 4 [J M] output level file: 5[A C] 6[D K] 7[L O] If we initially pick file 2, now we will compact file 2 and 6. But we can safely compact 2, 3 and 6 without expanding the output level. The previous code is messy and wrong. In this diff, I first determine the input range [a, b], and output range [c, d], then we get the range [e,f] = [min(a, c), max(b, d] and put all eligible clean-cut files within [e, f] into this compaction. **Note: clean-cut means the files don't have the same user key on the boundaries of some files that are not chosen in this compaction**. Closes https://github.com/facebook/rocksdb/pull/1760 Differential Revision: D4395564 Pulled By: lightmark fbshipit-source-id: 2dc2c5c 21 February 2017, 18:24:17 UTC
ebc8a79 alignment is on in ReadaheadRandomAccessFile::Read() Summary: Closes https://github.com/facebook/rocksdb/pull/1857 Differential Revision: D4534518 Pulled By: wat-ze-hex fbshipit-source-id: b456946 18 February 2017, 20:09:12 UTC
381fd32 Remove timeout_hint_us from WriteOptions Summary: The option has been deprecated for two years and has no effect. Removing. Closes https://github.com/facebook/rocksdb/pull/1866 Differential Revision: D4555203 Pulled By: yiwu-arbug fbshipit-source-id: c48f627 17 February 2017, 23:24:17 UTC
fce7a6e Fail IngestExternalFile when bg_error_ exists Summary: Fail IngestExternalFile() when bg_error_ exists Closes https://github.com/facebook/rocksdb/pull/1881 Differential Revision: D4580621 Pulled By: IslamAbdelRahman fbshipit-source-id: 1194913 17 February 2017, 21:39:17 UTC
a618a16 New subcode for IOError to detect the ESTALE errno Summary: I'd like to propose a patch to expose a new IOError type with subcode kStaleFile to allow to detect when ESTALE error is returned. This allows the rocksdb consumers to handle this error separately from other IOErrors. I've also added a missing string representation for the kDeadlock subcode, I believe calling ToString() on Status object with that subcode would result in an out of band access in the msgs array, Please let me know if you have any questions or would like me to make any changes to this pull request. Closes https://github.com/facebook/rocksdb/pull/1748 Differential Revision: D4387675 Pulled By: IslamAbdelRahman fbshipit-source-id: 67feb13 17 February 2017, 18:54:13 UTC
7ab0051 Remove deprecated DB::AddFile and DB::CompactRange Summary: Remove functions that we deprecated long time ago in db.h Closes https://github.com/facebook/rocksdb/pull/1878 Differential Revision: D4576521 Pulled By: IslamAbdelRahman fbshipit-source-id: dfddad1 17 February 2017, 18:54:13 UTC
4016673 Adding Dlang to the list Summary: Closes https://github.com/facebook/rocksdb/pull/1877 Differential Revision: D4576056 Pulled By: IslamAbdelRahman fbshipit-source-id: 0f19275 17 February 2017, 01:24:10 UTC
756c592 Allow adding external v1 sst file with no global seqno support Summary: This is a follow up fix for https://github.com/facebook/rocksdb/pull/1783. After it, we should be able to ingest external v1 sst files with no global seqno field. Closes https://github.com/facebook/rocksdb/pull/1874 Differential Revision: D4576194 Pulled By: IslamAbdelRahman fbshipit-source-id: 5b34a3e 17 February 2017, 01:09:12 UTC
aa0298f Remove trailing whitespace from examples Makefile Summary: The Makefile in the examples directory contained an empty line contain a tab character. This made my Emacs ask on every save `Suspicious line 10. Save anyway? (y or n)` Closes https://github.com/facebook/rocksdb/pull/1872 Differential Revision: D4573881 Pulled By: siying fbshipit-source-id: fb3b4ee 16 February 2017, 21:24:09 UTC
db2b4eb avoid direct io in rocksdb_lite Summary: fix lite bugs disable direct io in lite mode Closes https://github.com/facebook/rocksdb/pull/1870 Differential Revision: D4559866 Pulled By: yiwu-arbug fbshipit-source-id: 3761c51 16 February 2017, 18:39:13 UTC
43e9f01 Fix repair_test on ROCKSDB_LITE Summary: RepairDB isn't included in rocksdb lite, so don't test it. Closes https://github.com/facebook/rocksdb/pull/1873 Differential Revision: D4565094 Pulled By: ajkr fbshipit-source-id: 8cc0898 15 February 2017, 19:24:12 UTC
7106a99 Use monotonic time points in write_controller.cc and rate_limiter.cc Summary: NowMicros() provides non-monotonic time. When wall clock is synchronized or changed, the non-monotonicity time points will affect write rate controllers. This patch changes write_controller.cc and rate_limiter.cc to use monotonic time points. Closes https://github.com/facebook/rocksdb/pull/1865 Differential Revision: D4561732 Pulled By: siying fbshipit-source-id: 95ece62 15 February 2017, 02:24:24 UTC
c2247dc Make DBImpl::has_unpersisted_data_ atomic Summary: Seems to me `has_unpersisted_data_` is read from read thread and write from write thread concurrently without synchronization. Making it an atomic. I update the logic not because seeing any problem with it, but it just feel confusing. Closes https://github.com/facebook/rocksdb/pull/1869 Differential Revision: D4555837 Pulled By: yiwu-arbug fbshipit-source-id: eff2ab8 14 February 2017, 02:54:13 UTC
eb912a9 Remove disableDataSync option Summary: Remove disableDataSync, and another similarly named disable_data_sync options. This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods. Closes https://github.com/facebook/rocksdb/pull/1859 Differential Revision: D4541292 Pulled By: sagar0 fbshipit-source-id: 5b3a6ca 13 February 2017, 19:09:13 UTC
0227c16 Update static library versions and add checksums Summary: The previous version of zlib is no longer available. I have also updated the versions of the other static libraries and added checkum checks for the downloads; This is related to https://github.com/facebook/rocksdb/issues/1769 Closes https://github.com/facebook/rocksdb/pull/1863 Differential Revision: D4550742 Pulled By: yiwu-arbug fbshipit-source-id: 4414150 13 February 2017, 07:09:09 UTC
b3aae4d Add repair_test to make check Summary: needed so we can proactively find issues like #1858 Closes https://github.com/facebook/rocksdb/pull/1862 Differential Revision: D4545854 Pulled By: ajkr fbshipit-source-id: d77fcb7 11 February 2017, 02:09:18 UTC
421ce7c Add support for JNI Library on Linux on PowerPC. Summary: Closes https://github.com/facebook/rocksdb/issues/1317 Closes https://github.com/facebook/rocksdb/pull/1845 Differential Revision: D4546491 Pulled By: siying fbshipit-source-id: 86190b4 10 February 2017, 23:09:10 UTC
9afa20c Increase build version and HISTORY.md for releasing 5.2 Summary: Also clean up HISTORY.md a little bit. Closes https://github.com/facebook/rocksdb/pull/1854 Differential Revision: D4539556 Pulled By: siying fbshipit-source-id: 567391e 10 February 2017, 20:09:15 UTC
a5adda0 Fix repair issues Summary: Record the first parsed sequence number as the minimum so we can find the true minimum otherwise everything is larger than zero. Fix the comparator name comparision. Closes https://github.com/facebook/rocksdb/pull/1858 Differential Revision: D4544365 Pulled By: ajkr fbshipit-source-id: 439cbc2 10 February 2017, 18:54:12 UTC
b48e477 Consolidate file cutting logic in compaction loop Summary: It was really annoying to have two places (top and bottom of compaction loop) where we cut output files. I had bugs in both DeleteRange and dictionary compression due to updating only one of the two. This diff consolidates the file-cutting logic to the bottom of the compaction loop. Keep in mind that my goal with input_status is to be consistent with the past behavior, even though I'm not sure it's ideal. Closes https://github.com/facebook/rocksdb/pull/1832 Differential Revision: D4503038 Pulled By: ajkr fbshipit-source-id: 7da5213 09 February 2017, 00:24:17 UTC
ac2a77a Announce the experimetnal two-level index feature in HISTORY.md Summary: Announce the experimetnal two-level index feature in HISTORY.md. Also updated the default for index_per_partition to 1024. Closes https://github.com/facebook/rocksdb/pull/1855 Differential Revision: D4530102 Pulled By: maysamyabandeh fbshipit-source-id: b0fc6ff 08 February 2017, 22:24:10 UTC
c4a37dc Print the missed last layer in cfstats Summary: Printing compaction stats used to operate on two variable: number_levels_: for printing the layer num_levels_to_check: for updating the compaction score After this commit: 361010d44738de48ffc4fd9add70caa0891a0719 these two are mixed up and as a result the last layer might not be printed out: https://fb.facebook.com/groups/rocksdb.internal/permalink/1315716625143616/ number_levels_ was used to decide which layers to print: https://github.com/facebook/rocksdb/blob/672300f47f72b28c164fdf98a08171c09e311205/db/internal_stats.cc#L753 but after the patch it is based on the return value of DumpCFMapStats https://github.com/facebook/rocksdb/blob/361010d44738de48ffc4fd9add70caa0891a0719/db/internal_stats.cc#L929 which returns num_levels_to_check: https://github.com/facebook/rocksdb/blob/361010d44738de48ffc4fd9add70caa0891a0719/db/internal_stats.cc#L917 Closes https://github.com/facebook/rocksdb/pull/1853 Differential Revision: D4529280 Pulled By: maysamyabandeh fbshipit-source-id: 3fd9448 08 February 2017, 18:39:15 UTC
a12818a Blog post for 5.1.2 release Summary: Closes https://github.com/facebook/rocksdb/pull/1849 Differential Revision: D4528696 Pulled By: maysamyabandeh fbshipit-source-id: c12f3a2 08 February 2017, 16:54:20 UTC
3b4ac80 Clarify ldb column family argument Summary: move the argument description to the right section, make it clearer that the '=' sign is required, and use it in one subcommand where it seemed forgotten before. Closes https://github.com/facebook/rocksdb/pull/1840 Differential Revision: D4515096 Pulled By: ajkr fbshipit-source-id: 7b5b1c1 07 February 2017, 19:54:10 UTC
d70ce7e Move db_bench flags out of unnamed namespace Summary: I want to be able to, e.g., DECLARE_string(statistics_string); in my application such that I can override the default value of statistics_string. For this to work, we need to remove the unnamed namespace containing all the flags, and make sure all variables/functions covered by that namespace are static. Replaces #1828 due to internal tool issues. Closes https://github.com/facebook/rocksdb/pull/1844 Differential Revision: D4515124 Pulled By: ajkr fbshipit-source-id: 23b695e 07 February 2017, 19:39:12 UTC
186c7ee Remove incorrect statistics warning Summary: statistics are always concurrently updated regardless of whether a single instance or multiple instances are used. remove the warning since it's unnecessarily preventing people from sharing stats objects across rocksdb instances. replace #1819 which was having some problem with our internal tools. Closes https://github.com/facebook/rocksdb/pull/1826 Differential Revision: D4495970 Pulled By: ajkr fbshipit-source-id: 6601f61 07 February 2017, 19:39:12 UTC
53bb015 [rocksdb][PR] compaction_style and compaction_pri should output their value as a st… Summary: …ring Replace the numerical output for compaction_style and compaction_pri with strings Closes https://github.com/facebook/rocksdb/pull/1817 Differential Revision: D4482796 Pulled By: highker fbshipit-source-id: 5785768 07 February 2017, 18:39:12 UTC
69d5262 Two-level Indexes Summary: Partition Index blocks and use a Partition-index as a 2nd level index. The two-level index can be used by setting BlockBasedTableOptions::kTwoLevelIndexSearch as the index type and configuring BlockBasedTableOptions::index_per_partition t15539501 Closes https://github.com/facebook/rocksdb/pull/1814 Differential Revision: D4473535 Pulled By: maysamyabandeh fbshipit-source-id: bffb87e 07 February 2017, 00:39:12 UTC
0a4cdde Windows thread Summary: introduce new methods into a public threadpool interface, - allow submission of std::functions as they allow greater flexibility. - add Joining methods to the implementation to join scheduled and submitted jobs with an option to cancel jobs that did not start executing. - Remove ugly `#ifdefs` between pthread and std implementation, make it uniform. - introduce pimpl for a drop in replacement of the implementation - Introduce rocksdb::port::Thread typedef which is a replacement for std::thread. On Posix Thread defaults as before std::thread. - Implement WindowsThread that allocates memory in a more controllable manner than windows std::thread with a replaceable implementation. - should be no functionality changes. Closes https://github.com/facebook/rocksdb/pull/1823 Differential Revision: D4492902 Pulled By: siying fbshipit-source-id: c74cb11 06 February 2017, 22:54:18 UTC
1aaa898 Adding GetApproximateMemTableStats method Summary: Added method that returns approx num of entries as well as size for memtables. Closes https://github.com/facebook/rocksdb/pull/1841 Differential Revision: D4511990 Pulled By: VitaliyLi fbshipit-source-id: 9a4576e 06 February 2017, 22:54:16 UTC
9fc23c5 Use gcc-4.9-glibc-2.20-fb python in precommit_checker Summary: The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed soon. Closes https://github.com/facebook/rocksdb/pull/1839 Differential Revision: D4509684 Pulled By: siying fbshipit-source-id: 3efe296 03 February 2017, 21:39:18 UTC
b797e42 Dump compression dictionary meta-block Summary: make sst_dump print size/contents of the dictionary meta-block for easier debugging Closes https://github.com/facebook/rocksdb/pull/1837 Differential Revision: D4506399 Pulled By: ajkr fbshipit-source-id: b9bf668 03 February 2017, 20:39:16 UTC
036d668 Fix wrong result in data race case related to Get() Summary: In theory, Get() can get a wrong result, if it races in a special with with flush. The bug can be reproduced in DBTest2.GetRaceFlush. Fix this bug by getting snapshot after referencing the super version. Closes https://github.com/facebook/rocksdb/pull/1816 Differential Revision: D4475958 Pulled By: siying fbshipit-source-id: bd9e67a 03 February 2017, 19:39:15 UTC
574b543 Rename merger.h -> merging_iterator.h Summary: merger.h was always a confusing name for me, simply give the file a better name Closes https://github.com/facebook/rocksdb/pull/1836 Differential Revision: D4505357 Pulled By: IslamAbdelRahman fbshipit-source-id: 07b28d8 03 February 2017, 00:54:19 UTC
add8b50 Move ThreadLocal implementation into .cc Summary: Closes https://github.com/facebook/rocksdb/pull/1829 Differential Revision: D4502314 Pulled By: siying fbshipit-source-id: f46fac1 02 February 2017, 22:09:12 UTC
71d2496 Fix arc setting for Facebook internal tools Summary: Closes https://github.com/facebook/rocksdb/pull/1834 Differential Revision: D4503469 Pulled By: siying fbshipit-source-id: 761bfa6 02 February 2017, 21:24:16 UTC
f289d9f Fix OSX build break after the fallocate change Summary: The recent update about fallocate failed OSX build. Fix it. Closes https://github.com/facebook/rocksdb/pull/1830 Differential Revision: D4500235 Pulled By: siying fbshipit-source-id: a5f2b40 02 February 2017, 18:39:11 UTC
4a3e7d3 Change the default of delayed slowdown value to 16MB/s Summary: Change the default of delayed slowdown value to 16MB/s and further increase the L0 stop condition to 36 files. Closes https://github.com/facebook/rocksdb/pull/1821 Differential Revision: D4489229 Pulled By: siying fbshipit-source-id: 1003981 02 February 2017, 04:39:17 UTC
0513e21 RangeSync() should work with ROCKSDB_FALLOCATE_PRESENT not set Summary: Closes https://github.com/facebook/rocksdb/pull/1824 Differential Revision: D4493862 Pulled By: siying fbshipit-source-id: c168446 01 February 2017, 18:24:20 UTC
8b369ae Cleaner default options using C++11 in-class init Summary: C++11 in-class initialization is cleaner and makes it the default more explicit to our users and more visible. Use it for ColumnFamilyOptions and DBOptions Closes https://github.com/facebook/rocksdb/pull/1822 Differential Revision: D4490473 Pulled By: IslamAbdelRahman fbshipit-source-id: c493a87 01 February 2017, 02:09:15 UTC
ec79a7b Dedup code in option.cc and db_options.cc Summary: The code in DBOptions::Dump is simply a duplicate of the code in ImmutableDBOptions::Dump and MutableDBOptions.Dump consolidate duplicate code. tested visually Closes https://github.com/facebook/rocksdb/pull/1818 Differential Revision: D4486710 Pulled By: IslamAbdelRahman fbshipit-source-id: 7085189 01 February 2017, 01:39:12 UTC
b96372d improving the C wrapper Summary: - rocksdb_property_int (so that we don't have to parse strings) - and rocksdb_set_options (to allow controlling options via strings) - a few other missing options exposed - a documentation comment fix Closes https://github.com/facebook/rocksdb/pull/1793 Differential Revision: D4456569 Pulled By: yiwu-arbug fbshipit-source-id: 9f1fac1 28 January 2017, 01:39:16 UTC
04c4ec4 Change corruption_test to use 4 bits. Summary: In the patch which LRU cache was made use dynamic shard bits, I changed to 2 shard bits to make the test happy. Look like it is occasionally still unhappy. Change it to 4 shard bits. Closes https://github.com/facebook/rocksdb/pull/1815 Differential Revision: D4475849 Pulled By: siying fbshipit-source-id: 575ff00 27 January 2017, 19:24:16 UTC
2d75cd4 NewLRUCache() to pick number of shard bits based on capacity if not given Summary: If the users use the NewLRUCache() without passing in the number of shard bits, instead of using hard-coded 6, we'll determine it based on capacity. Closes https://github.com/facebook/rocksdb/pull/1584 Differential Revision: D4242517 Pulled By: siying fbshipit-source-id: 86b0f18 27 January 2017, 14:39:12 UTC
f25f1ec Add test DBTest2.GetRaceFlush which can expose a data race bug Summary: A current data race issue in Get() and Flush() can cause a Get() to return wrong results when a flush happened in the middle. Disable the test for now. Closes https://github.com/facebook/rocksdb/pull/1813 Differential Revision: D4472310 Pulled By: siying fbshipit-source-id: 5755ebd 27 January 2017, 00:39:14 UTC
37d4a79 Deserialize custom Statistics object in db_bench Summary: Added -statistics_string to deserialize a Statistics object using the factory functions registered by applications. Closes https://github.com/facebook/rocksdb/pull/1812 Differential Revision: D4469811 Pulled By: ajkr fbshipit-source-id: 2d80862 26 January 2017, 23:24:16 UTC
3b35134 Avoid cache lookups for range deletion meta-block Summary: I added the Cache::Ref() function a couple weeks ago (#1761) to make this feature possible. Like other meta-blocks, rep_->range_del_entry holds a cache handle to pin the range deletion block in uncompressed block cache for the duration of the table reader's lifetime. We can reuse this cache handle to create an iterator over this meta-block without any cache lookup. Ref() is used to increment the cache handle's refcount in case the returned iterator outlives the table reader. Closes https://github.com/facebook/rocksdb/pull/1801 Differential Revision: D4458782 Pulled By: ajkr fbshipit-source-id: 2883f10 26 January 2017, 19:24:13 UTC
94a0c32 Fix LRU Ref() for handles with external references only Summary: For case !handle->InCache() && handle->refs >= 1 (the third case mentioned in lru_cache.h), the key was overwritten by Insert(). In this case, the refcount can still be incremented, and the cache handle will never enter LRU list. Fix Ref() logic for this case. Closes https://github.com/facebook/rocksdb/pull/1808 Differential Revision: D4467656 Pulled By: ajkr fbshipit-source-id: c0784d8 26 January 2017, 18:54:15 UTC
17c1180 Generalize Env registration framework Summary: The Env registration framework supports registering client Envs and selecting which one to instantiate according to a text field. This enabled things like adding the -env_uri argument to db_bench, so the same binary could be reused with different Envs just by changing CLI config. Now this problem has come up again in a non-Env context, as I want to instantiate a client Statistics implementation from db_bench, which is configured entirely via text parameters. Also, in the future we may wish to use it for deserializing client objects when loading OPTIONS file. This diff generalizes the Env registration logic to work with arbitrary types. - Generalized registration and instantiation code by templating them - The entire implementation is in a header file as that's Google style guide's recommendation for template definitions - Pattern match with std::regex_match rather than checking prefix, which was the previous behavior - Rename functions/files to be non-Env-specific Closes https://github.com/facebook/rocksdb/pull/1776 Differential Revision: D4421933 Pulled By: ajkr fbshipit-source-id: 34647d1 26 January 2017, 00:09:14 UTC
07dddd5 EnvPosixTestWithParam should wait for all threads to finish Summary: If we don't wait for the threads to finish after each run, the thread queue may not be empty while the next test starts to run, which can cause unexpected behaviors. Also make some of the relaxed read/write more restrict. Closes https://github.com/facebook/rocksdb/pull/1590 Reviewed By: AsyncDBConnMarkedDownDBException Differential Revision: D4245922 Pulled By: AsyncDBConnMarkedDownDBException fbshipit-source-id: f83b74b 25 January 2017, 23:54:13 UTC
5dad9d6 Avoid logs_ operation out of DB mutex Summary: logs_.back() is called out of DB mutex, which can cause data race. We move the access into the DB mutex protection area. Closes https://github.com/facebook/rocksdb/pull/1774 Reviewed By: AsyncDBConnMarkedDownDBException Differential Revision: D4417472 Pulled By: AsyncDBConnMarkedDownDBException fbshipit-source-id: 2da1f1e 25 January 2017, 23:54:13 UTC
a7b1391 Fix CompactFiles() bug when used with CompactionFilter using SuperVersion Summary: GetAndRefSuperVersion() should not be called again in the same thread before ReturnAndCleanupSuperVersion() is called. If we have a compaction filter that is using DB::Get, This will happen ``` CompactFiles() { GetAndRefSuperVersion() // -- first call .. CompactionFilter() { GetAndRefSuperVersion() // -- second call ReturnAndCleanupSuperVersion() } .. ReturnAndCleanupSuperVersion() } ``` We solve this issue in the same way Iterator is solving it, but using GetReferencedSuperVersion() This was discovered in https://github.com/facebook/mysql-5.6/issues/427 by alxyang Closes https://github.com/facebook/rocksdb/pull/1803 Differential Revision: D4460155 Pulled By: IslamAbdelRahman fbshipit-source-id: 5e54322 25 January 2017, 22:09:13 UTC
616a146 Fix DeleteRange including sentinels in output files Summary: when writing RangeDelAggregator::AddToBuilder, I forgot that there are sentinel tombstones in the middle of the interval map since gaps between real tombstones are represented with sentinels. blame: #1614 Closes https://github.com/facebook/rocksdb/pull/1804 Differential Revision: D4460426 Pulled By: ajkr fbshipit-source-id: 69444b5 25 January 2017, 19:09:12 UTC
c918c4b Update USERS.md add user Pika Summary: add rocksdb users pika Closes https://github.com/facebook/rocksdb/pull/1805 Differential Revision: D4462809 Pulled By: siying fbshipit-source-id: 7289886 25 January 2017, 18:39:17 UTC
03ca2ac Remove function from DBImpl that are not used anywhere Summary: GetAndRefSuperVersionUnlocked ReturnAndCleanupSuperVersionUnlocked GetColumnFamilyHandleUnlocked Are dead code that are not used any where Closes https://github.com/facebook/rocksdb/pull/1802 Differential Revision: D4459948 Pulled By: IslamAbdelRahman fbshipit-source-id: 30fa89d 25 January 2017, 03:24:13 UTC
b0029bc Test merge op covered by range deletion in memtable Summary: It's a test case for #1797. Also got rid of kTypeDeletion in the conditional since we treat it the same as kTypeRangeDeletion. Closes https://github.com/facebook/rocksdb/pull/1800 Differential Revision: D4451300 Pulled By: ajkr fbshipit-source-id: b39dda1 24 January 2017, 21:39:11 UTC
d438e1e Test range deletion block outlives table reader Summary: This test ensures RangeDelAggregator can still access blocks even if it outlives the table readers that created them (detailed description in comments). I plan to optimize away the extra cache lookup we currently do in BlockBasedTable::NewRangeTombstoneIterator(), as it is ~5% CPU in my random read benchmark in a database with 1k tombstones. This test will help make sure nothing breaks in the process. Closes https://github.com/facebook/rocksdb/pull/1739 Differential Revision: D4375954 Pulled By: ajkr fbshipit-source-id: aef9357 24 January 2017, 21:24:14 UTC
fba726e Version librocksdb.so Summary: After make install, I see a directory hierarchy that looks like ``` ./usr ./usr/include ./usr/include/rocksdb ./usr/include/rocksdb/filter_policy.h [..] ./usr/include/rocksdb/iterator.h ./usr/include/rocksdb/utilities ./usr/include/rocksdb/utilities/ldb_cmd_execute_result.h ./usr/include/rocksdb/utilities/lua ./usr/include/rocksdb/utilities/lua/rocks_lua_custom_library.h ./usr/include/rocksdb/utilities/lua/rocks_lua_util.h ./usr/include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h ./usr/include/rocksdb/utilities/backupable_db.h [..] ./usr/include/rocksdb/utilities/env_registry.h [..] ./usr/include/rocksdb/env.h ./usr/lib64 ./usr/lib64/librocksdb.so.5 ./usr/lib64/librocksdb.so.5.0.0 ./usr/lib64/librocksdb.so ./usr/lib64/librocksdb.a ``` Closes https://github.com/facebook/rocksdb/pull/1798 Differential Revision: D4456536 Pulled By: yiwu-arbug fbshipit-source-id: 5494e91 24 January 2017, 19:09:10 UTC
9da4d54 Range deletions unsupported in tailing iterator Summary: change the iterator status to NotSupported as soon as a range tombstone is encountered by a ForwardIterator. Closes https://github.com/facebook/rocksdb/pull/1593 Differential Revision: D4246294 Pulled By: ajkr fbshipit-source-id: aef9f49 23 January 2017, 21:39:12 UTC
f2b4939 fixed typo Summary: I fixed exisit -> exist Closes https://github.com/facebook/rocksdb/pull/1799 Differential Revision: D4451466 Pulled By: yiwu-arbug fbshipit-source-id: b447c3a 23 January 2017, 20:54:13 UTC
973f1b7 memtable: delete merge value for range deleteion Summary: Closes https://github.com/facebook/rocksdb/pull/1797 Differential Revision: D4448004 Pulled By: ajkr fbshipit-source-id: 3ffc27c 23 January 2017, 20:24:14 UTC
aebfd17 fix non-portable behavior in encoder Summary: using ~0UL for mask uses a uint32_t at least in MSVC, but a uint64_t is required for it to work properly Closes https://github.com/facebook/rocksdb/pull/1777 Differential Revision: D4444004 Pulled By: yiwu-arbug fbshipit-source-id: 057cc42 21 January 2017, 00:39:22 UTC
753ff84 Fix get approx size Summary: Fixing GetApproximateSize bug for the case of computing stats for mem tables only. Closes https://github.com/facebook/rocksdb/pull/1795 Differential Revision: D4445507 Pulled By: IslamAbdelRahman fbshipit-source-id: 3905846 20 January 2017, 23:54:12 UTC
d7ea44f Fixup a couple of builds errors on Linux. Summary: The libraries produced on linux are now named librocksdb.a librocksdb.so Other fixes: * Also link with -lrt to avoid linker errors. * Generalize comments at the top to include Linux * Move -lgtest before -lpthread to avoid linker errors * move add_subdirectory(tools) to the end so it picks up the right libraries Closes https://github.com/facebook/rocksdb/pull/1364 Differential Revision: D4444138 Pulled By: yiwu-arbug fbshipit-source-id: f0e2c19 20 January 2017, 21:24:13 UTC
537da37 c: allow set savepoint to writebatch Summary: Allow set SavePoint to WriteBatch in C ABI. Closes https://github.com/facebook/rocksdb/pull/1698 Differential Revision: D4378556 Pulled By: yiwu-arbug fbshipit-source-id: afca746 20 January 2017, 21:24:13 UTC
af6ec4d fix batchresult handle leak Summary: This is related to PR https://github.com/facebook/rocksdb/pull/1642 Sorry about this extra PR, as my workflow was messed up when I checked in another work item by accident. Closes https://github.com/facebook/rocksdb/pull/1681 Differential Revision: D4379513 Pulled By: yiwu-arbug fbshipit-source-id: a668d4c 20 January 2017, 21:24:12 UTC
e29bb93 Zlib 1.2.8 is no longer available, switched to 1.2.10 Summary: Closes https://github.com/facebook/rocksdb/pull/1741 Differential Revision: D4380116 Pulled By: yiwu-arbug fbshipit-source-id: 7cf09df 20 January 2017, 21:24:12 UTC
5ac9731 Fix std::out_of_range when DBOptions::keep_log_file_num is zero Summary: We should validate this option, otherwise we may see std::out_of_range thrown at: db/db_impl.cc:1124 1123 for (unsigned int i = 0; i <= end; i++) { 1124 std::string& to_delete = old_info_log_files.at(i); 1125 std::string full_path_to_delete = 1126 (immutable_db_options_.db_log_dir.empty() Closes https://github.com/facebook/rocksdb/pull/1722 Differential Revision: D4379495 Pulled By: yiwu-arbug fbshipit-source-id: e136552 20 January 2017, 21:24:12 UTC
back to top