https://github.com/facebook/rocksdb

sort by:
Revision Author Date Message Commit Date
1a89d40 correct naming 15 February 2017, 02:46:58 UTC
e20af61 remove job for L0 15 February 2017, 02:32:56 UTC
b2006c4 make format Summary: Test Plan: Reviewers: Subscribers: Tasks: Blame Revision: 15 February 2017, 01:18:52 UTC
ff95ae2 Merge branch 'master' into level_compaction_expansion 14 February 2017, 22:34:40 UTC
c2247dc Make DBImpl::has_unpersisted_data_ atomic Summary: Seems to me `has_unpersisted_data_` is read from read thread and write from write thread concurrently without synchronization. Making it an atomic. I update the logic not because seeing any problem with it, but it just feel confusing. Closes https://github.com/facebook/rocksdb/pull/1869 Differential Revision: D4555837 Pulled By: yiwu-arbug fbshipit-source-id: eff2ab8 14 February 2017, 02:54:13 UTC
eb912a9 Remove disableDataSync option Summary: Remove disableDataSync, and another similarly named disable_data_sync options. This is being done to simplify options, and also because the performance gains of this feature can be achieved by other methods. Closes https://github.com/facebook/rocksdb/pull/1859 Differential Revision: D4541292 Pulled By: sagar0 fbshipit-source-id: 5b3a6ca 13 February 2017, 19:09:13 UTC
0227c16 Update static library versions and add checksums Summary: The previous version of zlib is no longer available. I have also updated the versions of the other static libraries and added checkum checks for the downloads; This is related to https://github.com/facebook/rocksdb/issues/1769 Closes https://github.com/facebook/rocksdb/pull/1863 Differential Revision: D4550742 Pulled By: yiwu-arbug fbshipit-source-id: 4414150 13 February 2017, 07:09:09 UTC
b3aae4d Add repair_test to make check Summary: needed so we can proactively find issues like #1858 Closes https://github.com/facebook/rocksdb/pull/1862 Differential Revision: D4545854 Pulled By: ajkr fbshipit-source-id: d77fcb7 11 February 2017, 02:09:18 UTC
421ce7c Add support for JNI Library on Linux on PowerPC. Summary: Closes https://github.com/facebook/rocksdb/issues/1317 Closes https://github.com/facebook/rocksdb/pull/1845 Differential Revision: D4546491 Pulled By: siying fbshipit-source-id: 86190b4 10 February 2017, 23:09:10 UTC
9afa20c Increase build version and HISTORY.md for releasing 5.2 Summary: Also clean up HISTORY.md a little bit. Closes https://github.com/facebook/rocksdb/pull/1854 Differential Revision: D4539556 Pulled By: siying fbshipit-source-id: 567391e 10 February 2017, 20:09:15 UTC
a5adda0 Fix repair issues Summary: Record the first parsed sequence number as the minimum so we can find the true minimum otherwise everything is larger than zero. Fix the comparator name comparision. Closes https://github.com/facebook/rocksdb/pull/1858 Differential Revision: D4544365 Pulled By: ajkr fbshipit-source-id: 439cbc2 10 February 2017, 18:54:12 UTC
b48e477 Consolidate file cutting logic in compaction loop Summary: It was really annoying to have two places (top and bottom of compaction loop) where we cut output files. I had bugs in both DeleteRange and dictionary compression due to updating only one of the two. This diff consolidates the file-cutting logic to the bottom of the compaction loop. Keep in mind that my goal with input_status is to be consistent with the past behavior, even though I'm not sure it's ideal. Closes https://github.com/facebook/rocksdb/pull/1832 Differential Revision: D4503038 Pulled By: ajkr fbshipit-source-id: 7da5213 09 February 2017, 00:24:17 UTC
ac2a77a Announce the experimetnal two-level index feature in HISTORY.md Summary: Announce the experimetnal two-level index feature in HISTORY.md. Also updated the default for index_per_partition to 1024. Closes https://github.com/facebook/rocksdb/pull/1855 Differential Revision: D4530102 Pulled By: maysamyabandeh fbshipit-source-id: b0fc6ff 08 February 2017, 22:24:10 UTC
c4a37dc Print the missed last layer in cfstats Summary: Printing compaction stats used to operate on two variable: number_levels_: for printing the layer num_levels_to_check: for updating the compaction score After this commit: 361010d44738de48ffc4fd9add70caa0891a0719 these two are mixed up and as a result the last layer might not be printed out: https://fb.facebook.com/groups/rocksdb.internal/permalink/1315716625143616/ number_levels_ was used to decide which layers to print: https://github.com/facebook/rocksdb/blob/672300f47f72b28c164fdf98a08171c09e311205/db/internal_stats.cc#L753 but after the patch it is based on the return value of DumpCFMapStats https://github.com/facebook/rocksdb/blob/361010d44738de48ffc4fd9add70caa0891a0719/db/internal_stats.cc#L929 which returns num_levels_to_check: https://github.com/facebook/rocksdb/blob/361010d44738de48ffc4fd9add70caa0891a0719/db/internal_stats.cc#L917 Closes https://github.com/facebook/rocksdb/pull/1853 Differential Revision: D4529280 Pulled By: maysamyabandeh fbshipit-source-id: 3fd9448 08 February 2017, 18:39:15 UTC
a12818a Blog post for 5.1.2 release Summary: Closes https://github.com/facebook/rocksdb/pull/1849 Differential Revision: D4528696 Pulled By: maysamyabandeh fbshipit-source-id: c12f3a2 08 February 2017, 16:54:20 UTC
3b4ac80 Clarify ldb column family argument Summary: move the argument description to the right section, make it clearer that the '=' sign is required, and use it in one subcommand where it seemed forgotten before. Closes https://github.com/facebook/rocksdb/pull/1840 Differential Revision: D4515096 Pulled By: ajkr fbshipit-source-id: 7b5b1c1 07 February 2017, 19:54:10 UTC
d70ce7e Move db_bench flags out of unnamed namespace Summary: I want to be able to, e.g., DECLARE_string(statistics_string); in my application such that I can override the default value of statistics_string. For this to work, we need to remove the unnamed namespace containing all the flags, and make sure all variables/functions covered by that namespace are static. Replaces #1828 due to internal tool issues. Closes https://github.com/facebook/rocksdb/pull/1844 Differential Revision: D4515124 Pulled By: ajkr fbshipit-source-id: 23b695e 07 February 2017, 19:39:12 UTC
186c7ee Remove incorrect statistics warning Summary: statistics are always concurrently updated regardless of whether a single instance or multiple instances are used. remove the warning since it's unnecessarily preventing people from sharing stats objects across rocksdb instances. replace #1819 which was having some problem with our internal tools. Closes https://github.com/facebook/rocksdb/pull/1826 Differential Revision: D4495970 Pulled By: ajkr fbshipit-source-id: 6601f61 07 February 2017, 19:39:12 UTC
53bb015 [rocksdb][PR] compaction_style and compaction_pri should output their value as a st… Summary: …ring Replace the numerical output for compaction_style and compaction_pri with strings Closes https://github.com/facebook/rocksdb/pull/1817 Differential Revision: D4482796 Pulled By: highker fbshipit-source-id: 5785768 07 February 2017, 18:39:12 UTC
69d5262 Two-level Indexes Summary: Partition Index blocks and use a Partition-index as a 2nd level index. The two-level index can be used by setting BlockBasedTableOptions::kTwoLevelIndexSearch as the index type and configuring BlockBasedTableOptions::index_per_partition t15539501 Closes https://github.com/facebook/rocksdb/pull/1814 Differential Revision: D4473535 Pulled By: maysamyabandeh fbshipit-source-id: bffb87e 07 February 2017, 00:39:12 UTC
0a4cdde Windows thread Summary: introduce new methods into a public threadpool interface, - allow submission of std::functions as they allow greater flexibility. - add Joining methods to the implementation to join scheduled and submitted jobs with an option to cancel jobs that did not start executing. - Remove ugly `#ifdefs` between pthread and std implementation, make it uniform. - introduce pimpl for a drop in replacement of the implementation - Introduce rocksdb::port::Thread typedef which is a replacement for std::thread. On Posix Thread defaults as before std::thread. - Implement WindowsThread that allocates memory in a more controllable manner than windows std::thread with a replaceable implementation. - should be no functionality changes. Closes https://github.com/facebook/rocksdb/pull/1823 Differential Revision: D4492902 Pulled By: siying fbshipit-source-id: c74cb11 06 February 2017, 22:54:18 UTC
1aaa898 Adding GetApproximateMemTableStats method Summary: Added method that returns approx num of entries as well as size for memtables. Closes https://github.com/facebook/rocksdb/pull/1841 Differential Revision: D4511990 Pulled By: VitaliyLi fbshipit-source-id: 9a4576e 06 February 2017, 22:54:16 UTC
9fc23c5 Use gcc-4.9-glibc-2.20-fb python in precommit_checker Summary: The gcc-4.8.1-glibc-2.17 platform is deprecated and will be removed soon. Closes https://github.com/facebook/rocksdb/pull/1839 Differential Revision: D4509684 Pulled By: siying fbshipit-source-id: 3efe296 03 February 2017, 21:39:18 UTC
b797e42 Dump compression dictionary meta-block Summary: make sst_dump print size/contents of the dictionary meta-block for easier debugging Closes https://github.com/facebook/rocksdb/pull/1837 Differential Revision: D4506399 Pulled By: ajkr fbshipit-source-id: b9bf668 03 February 2017, 20:39:16 UTC
036d668 Fix wrong result in data race case related to Get() Summary: In theory, Get() can get a wrong result, if it races in a special with with flush. The bug can be reproduced in DBTest2.GetRaceFlush. Fix this bug by getting snapshot after referencing the super version. Closes https://github.com/facebook/rocksdb/pull/1816 Differential Revision: D4475958 Pulled By: siying fbshipit-source-id: bd9e67a 03 February 2017, 19:39:15 UTC
574b543 Rename merger.h -> merging_iterator.h Summary: merger.h was always a confusing name for me, simply give the file a better name Closes https://github.com/facebook/rocksdb/pull/1836 Differential Revision: D4505357 Pulled By: IslamAbdelRahman fbshipit-source-id: 07b28d8 03 February 2017, 00:54:19 UTC
add8b50 Move ThreadLocal implementation into .cc Summary: Closes https://github.com/facebook/rocksdb/pull/1829 Differential Revision: D4502314 Pulled By: siying fbshipit-source-id: f46fac1 02 February 2017, 22:09:12 UTC
71d2496 Fix arc setting for Facebook internal tools Summary: Closes https://github.com/facebook/rocksdb/pull/1834 Differential Revision: D4503469 Pulled By: siying fbshipit-source-id: 761bfa6 02 February 2017, 21:24:16 UTC
f289d9f Fix OSX build break after the fallocate change Summary: The recent update about fallocate failed OSX build. Fix it. Closes https://github.com/facebook/rocksdb/pull/1830 Differential Revision: D4500235 Pulled By: siying fbshipit-source-id: a5f2b40 02 February 2017, 18:39:11 UTC
4a3e7d3 Change the default of delayed slowdown value to 16MB/s Summary: Change the default of delayed slowdown value to 16MB/s and further increase the L0 stop condition to 36 files. Closes https://github.com/facebook/rocksdb/pull/1821 Differential Revision: D4489229 Pulled By: siying fbshipit-source-id: 1003981 02 February 2017, 04:39:17 UTC
0513e21 RangeSync() should work with ROCKSDB_FALLOCATE_PRESENT not set Summary: Closes https://github.com/facebook/rocksdb/pull/1824 Differential Revision: D4493862 Pulled By: siying fbshipit-source-id: c168446 01 February 2017, 18:24:20 UTC
8b369ae Cleaner default options using C++11 in-class init Summary: C++11 in-class initialization is cleaner and makes it the default more explicit to our users and more visible. Use it for ColumnFamilyOptions and DBOptions Closes https://github.com/facebook/rocksdb/pull/1822 Differential Revision: D4490473 Pulled By: IslamAbdelRahman fbshipit-source-id: c493a87 01 February 2017, 02:09:15 UTC
ec79a7b Dedup code in option.cc and db_options.cc Summary: The code in DBOptions::Dump is simply a duplicate of the code in ImmutableDBOptions::Dump and MutableDBOptions.Dump consolidate duplicate code. tested visually Closes https://github.com/facebook/rocksdb/pull/1818 Differential Revision: D4486710 Pulled By: IslamAbdelRahman fbshipit-source-id: 7085189 01 February 2017, 01:39:12 UTC
b96372d improving the C wrapper Summary: - rocksdb_property_int (so that we don't have to parse strings) - and rocksdb_set_options (to allow controlling options via strings) - a few other missing options exposed - a documentation comment fix Closes https://github.com/facebook/rocksdb/pull/1793 Differential Revision: D4456569 Pulled By: yiwu-arbug fbshipit-source-id: 9f1fac1 28 January 2017, 01:39:16 UTC
04c4ec4 Change corruption_test to use 4 bits. Summary: In the patch which LRU cache was made use dynamic shard bits, I changed to 2 shard bits to make the test happy. Look like it is occasionally still unhappy. Change it to 4 shard bits. Closes https://github.com/facebook/rocksdb/pull/1815 Differential Revision: D4475849 Pulled By: siying fbshipit-source-id: 575ff00 27 January 2017, 19:24:16 UTC
2d75cd4 NewLRUCache() to pick number of shard bits based on capacity if not given Summary: If the users use the NewLRUCache() without passing in the number of shard bits, instead of using hard-coded 6, we'll determine it based on capacity. Closes https://github.com/facebook/rocksdb/pull/1584 Differential Revision: D4242517 Pulled By: siying fbshipit-source-id: 86b0f18 27 January 2017, 14:39:12 UTC
f25f1ec Add test DBTest2.GetRaceFlush which can expose a data race bug Summary: A current data race issue in Get() and Flush() can cause a Get() to return wrong results when a flush happened in the middle. Disable the test for now. Closes https://github.com/facebook/rocksdb/pull/1813 Differential Revision: D4472310 Pulled By: siying fbshipit-source-id: 5755ebd 27 January 2017, 00:39:14 UTC
37d4a79 Deserialize custom Statistics object in db_bench Summary: Added -statistics_string to deserialize a Statistics object using the factory functions registered by applications. Closes https://github.com/facebook/rocksdb/pull/1812 Differential Revision: D4469811 Pulled By: ajkr fbshipit-source-id: 2d80862 26 January 2017, 23:24:16 UTC
3b35134 Avoid cache lookups for range deletion meta-block Summary: I added the Cache::Ref() function a couple weeks ago (#1761) to make this feature possible. Like other meta-blocks, rep_->range_del_entry holds a cache handle to pin the range deletion block in uncompressed block cache for the duration of the table reader's lifetime. We can reuse this cache handle to create an iterator over this meta-block without any cache lookup. Ref() is used to increment the cache handle's refcount in case the returned iterator outlives the table reader. Closes https://github.com/facebook/rocksdb/pull/1801 Differential Revision: D4458782 Pulled By: ajkr fbshipit-source-id: 2883f10 26 January 2017, 19:24:13 UTC
94a0c32 Fix LRU Ref() for handles with external references only Summary: For case !handle->InCache() && handle->refs >= 1 (the third case mentioned in lru_cache.h), the key was overwritten by Insert(). In this case, the refcount can still be incremented, and the cache handle will never enter LRU list. Fix Ref() logic for this case. Closes https://github.com/facebook/rocksdb/pull/1808 Differential Revision: D4467656 Pulled By: ajkr fbshipit-source-id: c0784d8 26 January 2017, 18:54:15 UTC
17c1180 Generalize Env registration framework Summary: The Env registration framework supports registering client Envs and selecting which one to instantiate according to a text field. This enabled things like adding the -env_uri argument to db_bench, so the same binary could be reused with different Envs just by changing CLI config. Now this problem has come up again in a non-Env context, as I want to instantiate a client Statistics implementation from db_bench, which is configured entirely via text parameters. Also, in the future we may wish to use it for deserializing client objects when loading OPTIONS file. This diff generalizes the Env registration logic to work with arbitrary types. - Generalized registration and instantiation code by templating them - The entire implementation is in a header file as that's Google style guide's recommendation for template definitions - Pattern match with std::regex_match rather than checking prefix, which was the previous behavior - Rename functions/files to be non-Env-specific Closes https://github.com/facebook/rocksdb/pull/1776 Differential Revision: D4421933 Pulled By: ajkr fbshipit-source-id: 34647d1 26 January 2017, 00:09:14 UTC
07dddd5 EnvPosixTestWithParam should wait for all threads to finish Summary: If we don't wait for the threads to finish after each run, the thread queue may not be empty while the next test starts to run, which can cause unexpected behaviors. Also make some of the relaxed read/write more restrict. Closes https://github.com/facebook/rocksdb/pull/1590 Reviewed By: AsyncDBConnMarkedDownDBException Differential Revision: D4245922 Pulled By: AsyncDBConnMarkedDownDBException fbshipit-source-id: f83b74b 25 January 2017, 23:54:13 UTC
5dad9d6 Avoid logs_ operation out of DB mutex Summary: logs_.back() is called out of DB mutex, which can cause data race. We move the access into the DB mutex protection area. Closes https://github.com/facebook/rocksdb/pull/1774 Reviewed By: AsyncDBConnMarkedDownDBException Differential Revision: D4417472 Pulled By: AsyncDBConnMarkedDownDBException fbshipit-source-id: 2da1f1e 25 January 2017, 23:54:13 UTC
a7b1391 Fix CompactFiles() bug when used with CompactionFilter using SuperVersion Summary: GetAndRefSuperVersion() should not be called again in the same thread before ReturnAndCleanupSuperVersion() is called. If we have a compaction filter that is using DB::Get, This will happen ``` CompactFiles() { GetAndRefSuperVersion() // -- first call .. CompactionFilter() { GetAndRefSuperVersion() // -- second call ReturnAndCleanupSuperVersion() } .. ReturnAndCleanupSuperVersion() } ``` We solve this issue in the same way Iterator is solving it, but using GetReferencedSuperVersion() This was discovered in https://github.com/facebook/mysql-5.6/issues/427 by alxyang Closes https://github.com/facebook/rocksdb/pull/1803 Differential Revision: D4460155 Pulled By: IslamAbdelRahman fbshipit-source-id: 5e54322 25 January 2017, 22:09:13 UTC
616a146 Fix DeleteRange including sentinels in output files Summary: when writing RangeDelAggregator::AddToBuilder, I forgot that there are sentinel tombstones in the middle of the interval map since gaps between real tombstones are represented with sentinels. blame: #1614 Closes https://github.com/facebook/rocksdb/pull/1804 Differential Revision: D4460426 Pulled By: ajkr fbshipit-source-id: 69444b5 25 January 2017, 19:09:12 UTC
c918c4b Update USERS.md add user Pika Summary: add rocksdb users pika Closes https://github.com/facebook/rocksdb/pull/1805 Differential Revision: D4462809 Pulled By: siying fbshipit-source-id: 7289886 25 January 2017, 18:39:17 UTC
03ca2ac Remove function from DBImpl that are not used anywhere Summary: GetAndRefSuperVersionUnlocked ReturnAndCleanupSuperVersionUnlocked GetColumnFamilyHandleUnlocked Are dead code that are not used any where Closes https://github.com/facebook/rocksdb/pull/1802 Differential Revision: D4459948 Pulled By: IslamAbdelRahman fbshipit-source-id: 30fa89d 25 January 2017, 03:24:13 UTC
b0029bc Test merge op covered by range deletion in memtable Summary: It's a test case for #1797. Also got rid of kTypeDeletion in the conditional since we treat it the same as kTypeRangeDeletion. Closes https://github.com/facebook/rocksdb/pull/1800 Differential Revision: D4451300 Pulled By: ajkr fbshipit-source-id: b39dda1 24 January 2017, 21:39:11 UTC
d438e1e Test range deletion block outlives table reader Summary: This test ensures RangeDelAggregator can still access blocks even if it outlives the table readers that created them (detailed description in comments). I plan to optimize away the extra cache lookup we currently do in BlockBasedTable::NewRangeTombstoneIterator(), as it is ~5% CPU in my random read benchmark in a database with 1k tombstones. This test will help make sure nothing breaks in the process. Closes https://github.com/facebook/rocksdb/pull/1739 Differential Revision: D4375954 Pulled By: ajkr fbshipit-source-id: aef9357 24 January 2017, 21:24:14 UTC
fba726e Version librocksdb.so Summary: After make install, I see a directory hierarchy that looks like ``` ./usr ./usr/include ./usr/include/rocksdb ./usr/include/rocksdb/filter_policy.h [..] ./usr/include/rocksdb/iterator.h ./usr/include/rocksdb/utilities ./usr/include/rocksdb/utilities/ldb_cmd_execute_result.h ./usr/include/rocksdb/utilities/lua ./usr/include/rocksdb/utilities/lua/rocks_lua_custom_library.h ./usr/include/rocksdb/utilities/lua/rocks_lua_util.h ./usr/include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h ./usr/include/rocksdb/utilities/backupable_db.h [..] ./usr/include/rocksdb/utilities/env_registry.h [..] ./usr/include/rocksdb/env.h ./usr/lib64 ./usr/lib64/librocksdb.so.5 ./usr/lib64/librocksdb.so.5.0.0 ./usr/lib64/librocksdb.so ./usr/lib64/librocksdb.a ``` Closes https://github.com/facebook/rocksdb/pull/1798 Differential Revision: D4456536 Pulled By: yiwu-arbug fbshipit-source-id: 5494e91 24 January 2017, 19:09:10 UTC
9da4d54 Range deletions unsupported in tailing iterator Summary: change the iterator status to NotSupported as soon as a range tombstone is encountered by a ForwardIterator. Closes https://github.com/facebook/rocksdb/pull/1593 Differential Revision: D4246294 Pulled By: ajkr fbshipit-source-id: aef9f49 23 January 2017, 21:39:12 UTC
f2b4939 fixed typo Summary: I fixed exisit -> exist Closes https://github.com/facebook/rocksdb/pull/1799 Differential Revision: D4451466 Pulled By: yiwu-arbug fbshipit-source-id: b447c3a 23 January 2017, 20:54:13 UTC
973f1b7 memtable: delete merge value for range deleteion Summary: Closes https://github.com/facebook/rocksdb/pull/1797 Differential Revision: D4448004 Pulled By: ajkr fbshipit-source-id: 3ffc27c 23 January 2017, 20:24:14 UTC
aebfd17 fix non-portable behavior in encoder Summary: using ~0UL for mask uses a uint32_t at least in MSVC, but a uint64_t is required for it to work properly Closes https://github.com/facebook/rocksdb/pull/1777 Differential Revision: D4444004 Pulled By: yiwu-arbug fbshipit-source-id: 057cc42 21 January 2017, 00:39:22 UTC
753ff84 Fix get approx size Summary: Fixing GetApproximateSize bug for the case of computing stats for mem tables only. Closes https://github.com/facebook/rocksdb/pull/1795 Differential Revision: D4445507 Pulled By: IslamAbdelRahman fbshipit-source-id: 3905846 20 January 2017, 23:54:12 UTC
d7ea44f Fixup a couple of builds errors on Linux. Summary: The libraries produced on linux are now named librocksdb.a librocksdb.so Other fixes: * Also link with -lrt to avoid linker errors. * Generalize comments at the top to include Linux * Move -lgtest before -lpthread to avoid linker errors * move add_subdirectory(tools) to the end so it picks up the right libraries Closes https://github.com/facebook/rocksdb/pull/1364 Differential Revision: D4444138 Pulled By: yiwu-arbug fbshipit-source-id: f0e2c19 20 January 2017, 21:24:13 UTC
537da37 c: allow set savepoint to writebatch Summary: Allow set SavePoint to WriteBatch in C ABI. Closes https://github.com/facebook/rocksdb/pull/1698 Differential Revision: D4378556 Pulled By: yiwu-arbug fbshipit-source-id: afca746 20 January 2017, 21:24:13 UTC
af6ec4d fix batchresult handle leak Summary: This is related to PR https://github.com/facebook/rocksdb/pull/1642 Sorry about this extra PR, as my workflow was messed up when I checked in another work item by accident. Closes https://github.com/facebook/rocksdb/pull/1681 Differential Revision: D4379513 Pulled By: yiwu-arbug fbshipit-source-id: a668d4c 20 January 2017, 21:24:12 UTC
e29bb93 Zlib 1.2.8 is no longer available, switched to 1.2.10 Summary: Closes https://github.com/facebook/rocksdb/pull/1741 Differential Revision: D4380116 Pulled By: yiwu-arbug fbshipit-source-id: 7cf09df 20 January 2017, 21:24:12 UTC
5ac9731 Fix std::out_of_range when DBOptions::keep_log_file_num is zero Summary: We should validate this option, otherwise we may see std::out_of_range thrown at: db/db_impl.cc:1124 1123 for (unsigned int i = 0; i <= end; i++) { 1124 std::string& to_delete = old_info_log_files.at(i); 1125 std::string full_path_to_delete = 1126 (immutable_db_options_.db_log_dir.empty() Closes https://github.com/facebook/rocksdb/pull/1722 Differential Revision: D4379495 Pulled By: yiwu-arbug fbshipit-source-id: e136552 20 January 2017, 21:24:12 UTC
4e35ffd cmake: check -momit-leaf-frame-pointer before using it Summary: because not all archs support this option. see https://gcc.gnu.org/onlinedocs/gcc/Option-Summary.html. also do not pass "-fno-omit-frame-pointer" and "-momit-leaf-frame-pointer" to compiler if ${CMAKE_BUILD_TYPE} is "Debug". this matches the behaviour of DEBUG_LEVEL=2 in Makefile. Signed-off-by: Kefu Chai <tchaikov@gmail.com> Closes https://github.com/facebook/rocksdb/pull/1762 Differential Revision: D4444036 Pulled By: yiwu-arbug fbshipit-source-id: 8596fbe 20 January 2017, 21:09:11 UTC
3c0852d Make ingest external file backward compatible Summary: Closes https://github.com/facebook/rocksdb/pull/1783 Differential Revision: D4443463 Pulled By: IslamAbdelRahman fbshipit-source-id: 39d21d6 20 January 2017, 20:09:19 UTC
0e8dfd6 Fix OptimizeForPointLookup() Summary: If users directly call OptimizeForPointLookup(), it is broken as the option isn't compatible with parallel memtable insert. Fix it by using memtable bloomo filter instead. Closes https://github.com/facebook/rocksdb/pull/1791 Differential Revision: D4442836 Pulled By: siying fbshipit-source-id: bf6c9cd 20 January 2017, 18:54:12 UTC
e840213 Change DB::GetApproximateSizes for more flexibility needed for MyRocks Summary: Added an option to GetApproximateSizes to exclude file stats, as MyRocks has those counted exactly and we need only stats from memtables. Closes https://github.com/facebook/rocksdb/pull/1787 Differential Revision: D4441111 Pulled By: IslamAbdelRahman fbshipit-source-id: c11f4c3 20 January 2017, 17:39:11 UTC
9239103 Flush job should release reference current version if sync log failed Summary: Fix the bug when sync log fail, FlushJob::Run() will not be execute and reference to cfd->current() will not be release. Closes https://github.com/facebook/rocksdb/pull/1792 Differential Revision: D4441316 Pulled By: yiwu-arbug fbshipit-source-id: 5523e28 20 January 2017, 07:09:15 UTC
da54d36 Disable IngestExternalFile in ReadOnly mode Summary: Disable IngestExternalFile() in read only mode Closes https://github.com/facebook/rocksdb/pull/1781 Differential Revision: D4439179 Pulled By: IslamAbdelRahman fbshipit-source-id: b7e46e7 19 January 2017, 23:54:19 UTC
5cf176c Fix for 2PC causing WAL to grow too large Summary: Consider the following single column family scenario: prepare in log A commit in log B *WAL is too large, flush all CFs to releast log A* *CFA is on log B so we do not see CFA is depending on log A so no flush is requested* To fix this we must also consider the log containing the prepare section when determining what log a CF is dependent on. Closes https://github.com/facebook/rocksdb/pull/1768 Differential Revision: D4403265 Pulled By: reidHoruff fbshipit-source-id: ce800ff 19 January 2017, 23:39:12 UTC
4a73bb0 Split travis jobs Summary: Some travis jobs are running out of space, splitting to more jobs should reduce the chance of that happening Closes https://github.com/facebook/rocksdb/pull/1789 Differential Revision: D4438039 Pulled By: IslamAbdelRahman fbshipit-source-id: 05787ff 19 January 2017, 21:39:12 UTC
c70d3c7 Enable DBTest.GroupCommit as it runs in a reasonlable time now. Summary: Remove ReRuns as they only waste time. Add env_basic_test to get more foundation coverage. Closes https://github.com/facebook/rocksdb/pull/1788 Differential Revision: D4431433 Pulled By: IslamAbdelRahman fbshipit-source-id: 50d07f8 18 January 2017, 22:24:11 UTC
602c13a Remove fadvise with direct IO read Summary: Remove the logic since we don't use buffer cache with direct IO. Resolve read regression we currently have. Closes https://github.com/facebook/rocksdb/pull/1782 Differential Revision: D4430408 Pulled By: yiwu-arbug fbshipit-source-id: 5557bba 18 January 2017, 20:09:10 UTC
f9d18e2 Fix DeleteRange file boundary correctness issue with max_compaction_bytes Summary: Cockroachdb exposed this bug in #1778. The bug happens when a compaction's output files are ended due to exceeding max_compaction_bytes. In that case we weren't taking into account the next file's start key when deciding how far to extend the current file's max_key. This caused the non-overlapping key-range invariant to be violated. Note this was correctly handled for the usual case of cutting compaction output, which is file size exceeding max_output_file_size. I am not sure why these are two separate code paths, but we can consider refactoring it to prevent such errors in the future. Closes https://github.com/facebook/rocksdb/pull/1784 Differential Revision: D4430235 Pulled By: ajkr fbshipit-source-id: 80af748 18 January 2017, 19:54:22 UTC
3ce091f Add KEEP_DB env var option Summary: When debugging tests, it's useful to preserve the DB to investigate it and check the logs This will allow us to set KEEP_DB=1 to preserve the DB Closes https://github.com/facebook/rocksdb/pull/1759 Differential Revision: D4393826 Pulled By: IslamAbdelRahman fbshipit-source-id: 1bff689 17 January 2017, 21:54:20 UTC
77b4806 Fix 2PC with concurrent memtable insert Summary: If concurrent memtable insert is enabled, and one prepare command and a normal command are grouped into a commit group, the sequence ID will be calculated incorrectly. Closes https://github.com/facebook/rocksdb/pull/1730 Differential Revision: D4371081 Pulled By: siying fbshipit-source-id: cd40c6d 17 January 2017, 19:24:28 UTC
e8a0960 util/thread_local.h: silence a clang-build warning Summary: otherwise clang complains with /home/jenkins/workspace/ceph-master/src/rocksdb/util/thread_local.h:205:5: error: macro expansion producing 'defined' has undefined behavior [-Werror,-Wexpansion-to-defined] ^ /home/jenkins/workspace/ceph-master/src/rocksdb/util/thread_local.h:22:4: note: expanded from macro 'ROCKSDB_SUPPORT_THREAD_LOCAL' !defined(OS_WIN) && !defined(OS_MACOSX) && !defined(IOS_CROSS_COMPILE) ^` Signed-off-by: Kefu Chai <tchaikov@gmail.com> Closes https://github.com/facebook/rocksdb/pull/1757 Differential Revision: D4394140 Pulled By: siying fbshipit-source-id: f0beda0 15 January 2017, 21:24:16 UTC
324a0f9 Follow up for DirectIO refactor Summary: Windows follow up for https://github.com/facebook/rocksdb/commit/dc2584eea0d9965f97d38d4d462a7b57750144d3 Differential Revision: D4420337 Pulled By: IslamAbdelRahman fbshipit-source-id: fedc5b5 15 January 2017, 21:24:16 UTC
bc5d7b7 travis: For linux, do all tests under gcc Summary: apologies yiwu-arbug, I thought I'd included this part of the commit. Seems as though "JOB_NAME=unittests ROCKSDBTESTS_END=db_block_cache_test" has space issues and it did occasional previously too. May need to look up splitting tests more. Closes https://github.com/facebook/rocksdb/pull/1766 Differential Revision: D4417493 Pulled By: siying fbshipit-source-id: 3007ba5 13 January 2017, 23:39:13 UTC
3e6899d change UseDirectIO() to use_direct_io() Summary: also change variable name `direct_io_` to `use_direct_io_` in WritableFile to make it consistent with read path. Closes https://github.com/facebook/rocksdb/pull/1770 Differential Revision: D4416435 Pulled By: lightmark fbshipit-source-id: 4143c53 13 January 2017, 20:09:15 UTC
b5cf68b add overlapping case 12 January 2017, 22:56:13 UTC
d4e07a8 fix warning of unused direct io helper functions Summary: add build guard Closes https://github.com/facebook/rocksdb/pull/1771 Differential Revision: D4410779 Pulled By: siying fbshipit-source-id: 3796c30 12 January 2017, 20:39:14 UTC
dc2584e direct reads refactor Summary: direct IO reads refactoring remove unnecessary classes and unified interfaces tested with db_bench need more change for options and ON/OFF for different files. Since disabled is default, it should be fine now Closes https://github.com/facebook/rocksdb/pull/1636 Differential Revision: D4307189 Pulled By: lightmark fbshipit-source-id: 6991e22 12 January 2017, 00:54:12 UTC
d18dd2c Abort compactions more reliably when closing DB Summary: DB shutdown aborts running compactions by setting an atomic shutting_down=true that CompactionJob periodically checks. Without this PR it checks it before processing every _output_ value. If compaction filter filters everything out, the compaction is uninterruptible. This PR adds checks for shutting_down on every _input_ value (in CompactionIterator and MergeHelper). There's also some minor code cleanup along the way. Closes https://github.com/facebook/rocksdb/pull/1639 Differential Revision: D4306571 Pulled By: yiwu-arbug fbshipit-source-id: f050890 11 January 2017, 23:09:21 UTC
62384eb Guarding extra fallocate call with TRAVIS because its not working pro… Summary: …perly on travis There is some old code in PosixWritableFile::Close(), which truncates the file to the measured size and then does an extra fallocate with KEEP_SIZE. This is commented as a failsafe because in some cases ftruncate doesn't do the right job (I don't know of an instance of this btw). However doing an fallocate with KEEP_SIZE should not increase the file size. However on Travis Worker which is Docker (likely AUFS ) its not working. There are comments on web that show that the AUFS author had initially not implemented fallocate, and then did it later. So not sure what is the quality of the implementation. Closes https://github.com/facebook/rocksdb/pull/1765 Differential Revision: D4401340 Pulled By: anirbanr-fb fbshipit-source-id: e2d8100 11 January 2017, 22:24:13 UTC
9f24629 Performance: Iterate vector by reference Summary: Closes https://github.com/facebook/rocksdb/pull/1763 Differential Revision: D4398796 Pulled By: yiwu-arbug fbshipit-source-id: b82636d 11 January 2017, 18:54:37 UTC
fe395fb Allow incrementing refcount on cache handles Summary: Previously the only way to increment a handle's refcount was to invoke Lookup(), which (1) did hash table lookup to get cache handle, (2) incremented that handle's refcount. For a future DeleteRange optimization, I added a function, Ref(), for when the caller already has a cache handle and only needs to do (2). Closes https://github.com/facebook/rocksdb/pull/1761 Differential Revision: D4397114 Pulled By: ajkr fbshipit-source-id: 9addbe5 11 January 2017, 00:54:20 UTC
f50ed2e fix type conversion 11 January 2017, 00:05:33 UTC
2172b66 Fix build on FreeBSD Summary: ``` CC utilities/column_aware_encoding_exp.o utilities/column_aware_encoding_exp.cc:149:5: error: use of undeclared identifier 'exit' exit(1); ^ utilities/column_aware_encoding_exp.cc:154:5: error: use of undeclared identifier 'exit' exit(1); ^ utilities/column_aware_encoding_exp.cc:158:5: error: use of undeclared identifier 'exit' exit(1); ^ 3 errors generated. ``` Closes https://github.com/facebook/rocksdb/pull/1754 Differential Revision: D4399044 Pulled By: IslamAbdelRahman fbshipit-source-id: fbab5a2 10 January 2017, 19:39:12 UTC
cfafa2f delete comments 10 January 2017, 00:07:29 UTC
6ff7b64 level compaction expansion 10 January 2017, 00:00:15 UTC
3c233ca Fix Windows environment issues Summary: Enable directIO on WritableFileImpl::Append with offset being current length of the file. Enable UniqueID tests on Windows, disable others but leeting them to compile. Unique tests are valuable to detect failures on different filesystems and upcoming ReFS. Clear output in WinEnv Getchildren.This is different from previous strategy, do not touch output on failure. Make sure DBTest.OpenWhenOpen works with windows error message Closes https://github.com/facebook/rocksdb/pull/1746 Differential Revision: D4385681 Pulled By: IslamAbdelRahman fbshipit-source-id: c07b702 09 January 2017, 23:54:12 UTC
7631734 Fix the error in ColumnFamiliesTest Summary: In the test the last change to AAAZZZ in handles[1] is deleting it. The result of the get must be NotFound then. Previosuly the test did not check for the return value of Get and assumed that the status is ok. It then move ahead asserting the returned value. The passed-by-reference string value however was not changed (since the key was not found) and the asserted value is what it contained before doing the Get. Closes https://github.com/facebook/rocksdb/pull/1753 Differential Revision: D4390982 Pulled By: maysamyabandeh fbshipit-source-id: dd55a34 09 January 2017, 22:09:13 UTC
7a02ad0 Update travis to ubuntu trusty Summary: Distro gflags is now acceptable in build. Add gcc for two linux builds, JOB_NAME=unittests and lite_build. unittests will be built with g++-6. Use apt and ccache during build. Seems more up to date than precise :-) llvm repos for trusty aren't whitelisted yet (ref: https://github.com/travis-ci/apt-source-whitelist/pull/309, https://github.com/travis-ci/apt-source-whitelist/issues/332, https://github.com/travis-ci/apt-source-whitelist/issues/332) Closes https://github.com/facebook/rocksdb/pull/1685 Differential Revision: D4340378 Pulled By: yiwu-arbug fbshipit-source-id: a143b9d 09 January 2017, 21:54:19 UTC
60c509f Fix valgrind failure in test CurrentFileModifiedWhileCheckpointing2PC Summary: Fix some memory leaks in the test. Also rename the test class name from DBTest to CheckpointTest to avoid confusion. Closes https://github.com/facebook/rocksdb/pull/1752 Differential Revision: D4390355 Pulled By: siying fbshipit-source-id: 0fa388a 09 January 2017, 19:54:13 UTC
d0ba8ec Revert "PinnableSlice" Summary: This reverts commit 54d94e9c2cc0bf6eeb2a165ada33fa9c174f0b16. The pull request was landed by mistake. Closes https://github.com/facebook/rocksdb/pull/1755 Differential Revision: D4391678 Pulled By: maysamyabandeh fbshipit-source-id: 36d5149 08 January 2017, 22:24:12 UTC
54d94e9 PinnableSlice Summary: Currently the point lookup values are copied to a string provided by the user. This incures an extra memcpy cost. This patch allows doing point lookup via a PinnableSlice which pins the source memory location (instead of copying their content) and releases them after the content is consumed by the user. The old API of Get(string) is translated to the new API underneath. Here is the summary for improvements: 1. value 100 byte: 1.8% regular, 1.2% merge values 2. value 1k byte: 11.5% regular, 7.5% merge values 3. value 10k byte: 26% regular, 29.9% merge values The improvement for merge could be more if we extend this approach to pin the merge output and delay the full merge operation until the user actually needs it. We have put that for future work. PS: Sometimes we observe a small decrease in performance when switching from t5452014 to this patch but with the old Get(string) API. The difference is a little and could be noise. More importantly it is safely cancelled Closes https://github.com/facebook/rocksdb/pull/1732 Differential Revision: D4374613 Pulled By: maysamyabandeh fbshipit-source-id: a077f1a 08 January 2017, 21:54:13 UTC
e04480f Fix MS warnings. Use ROCKSDB_Prsz for size_t. Summary: Closes https://github.com/facebook/rocksdb/pull/1737 Differential Revision: D4378852 Pulled By: yiwu-arbug fbshipit-source-id: ba8b02d 07 January 2017, 00:24:14 UTC
c081f72 5.0.1 release blog post Summary: 5.0.1 release blog post Closes https://github.com/facebook/rocksdb/pull/1751 Differential Revision: D4388962 Pulled By: yiwu-arbug fbshipit-source-id: 59559e1 06 January 2017, 23:24:15 UTC
ac73d75 Add GetSupportedCompressions() convenience function Summary: This function will return a list of supported compression types in RocksDB This is needed for MyRocks https://github.com/facebook/mysql-5.6/pull/446 Closes https://github.com/facebook/rocksdb/pull/1747 Differential Revision: D4385921 Pulled By: IslamAbdelRahman fbshipit-source-id: 2f5b59f 06 January 2017, 19:24:14 UTC
b104b87 Maintain position in range deletions map Summary: When deletion-collapsing mode is enabled (i.e., for DBIter/CompactionIterator), we maintain position in the tombstone maps across calls to ShouldDelete(). Since iterators often access keys sequentially (or reverse-sequentially), scanning forward/backward from the last position can be faster than binary-searching the map for every key. - When Next() is invoked on an iterator, we use kForwardTraversal to scan forwards, if needed, until arriving at the range deletion containing the next key. - Similarly for Prev(), we use kBackwardTraversal to scan backwards in the range deletion map. - When the iterator seeks, we use kBinarySearch for repositioning - After tombstones are added or before the first ShouldDelete() invocation, the current position is set to invalid, which forces kBinarySearch to be used. - Non-iterator users (i.e., Get()) use kFullScan, which has the same behavior as before---scan the whole map for every key passed to ShouldDelete(). Closes https://github.com/facebook/rocksdb/pull/1701 Differential Revision: D4350318 Pulled By: ajkr fbshipit-source-id: 5129b76 05 January 2017, 18:39:12 UTC
640d724 Update db_bench and sst_dump to test with block cache mid-point inser? Summary: ?tion Add flags in db_bench to test with block cache mid-point insertion. Also update sst_dump to dump total block sizes of each type. I find it useful to look at these test db stats and I don't know if we have them elsewhere. Closes https://github.com/facebook/rocksdb/pull/1706 Differential Revision: D4355812 Pulled By: yiwu-arbug fbshipit-source-id: 3e4a348 04 January 2017, 02:39:14 UTC
653ac1f C API: support total_order_mode Summary: Closes https://github.com/facebook/rocksdb/pull/1687 Differential Revision: D4349210 Pulled By: IslamAbdelRahman fbshipit-source-id: 32d0fbd 04 January 2017, 02:39:14 UTC
back to top