https://github.com/facebook/rocksdb

sort by:
Revision Author Date Message Commit Date
2e98ac0 Bump up version to 5.5.5, update HISTORY.md and one more cleaning up 18 July 2017, 19:54:12 UTC
06bc381 Update java/rocksjni.pom 18 July 2017, 19:50:36 UTC
8efefd1 Some more files. 18 July 2017, 19:21:24 UTC
b5a99a7 Update HISTORY.md 18 July 2017, 18:41:17 UTC
832b052 Remove some left-over BSD headers 18 July 2017, 18:38:39 UTC
64266e8 Bump up version to 5.5.4 18 July 2017, 17:46:24 UTC
ed1ccab Add back the LevelDB license file Summary: Closes https://github.com/facebook/rocksdb/pull/2591 Differential Revision: D5432696 Pulled By: siying fbshipit-source-id: a613230ab916de0b279a65ef429ede65460a8db2 18 July 2017, 17:42:29 UTC
90b27c4 Remove the licensing description in CONTRIBUTING.md Summary: Closes https://github.com/facebook/rocksdb/pull/2590 Differential Revision: D5432539 Pulled By: siying fbshipit-source-id: 49902453bba3c95c1fb8354441b2198649e52bf4 18 July 2017, 17:42:18 UTC
5ed2f08 Change RocksDB License Summary: Closes https://github.com/facebook/rocksdb/pull/2589 Differential Revision: D5431502 Pulled By: siying fbshipit-source-id: 8ebf8c87883daa9daa54b2303d11ce01ab1f6f75 18 July 2017, 17:41:51 UTC
7f5f0e0 Bump version to 5.5.3 17 July 2017, 21:31:15 UTC
7778d2a Fix RocksDB Lite build with CLANG Summary: Closes https://github.com/facebook/rocksdb/pull/2419 Differential Revision: D5193976 Pulled By: siying fbshipit-source-id: 62d115edee6043237e9d6ad3c2a05481e162c9eb 17 July 2017, 21:27:24 UTC
9c15f19 Fix db_bench build break with blob db Summary: Lite build does not recognize FLAGS_use_blob_db. Fixing it. Closes https://github.com/facebook/rocksdb/pull/2372 Reviewed By: anirbanr-fb Differential Revision: D5130773 Pulled By: yiwu-arbug fbshipit-source-id: 43131d9d0be5811f2129af562be72cca26369cb3 17 July 2017, 21:24:27 UTC
b082e27 Bump version to 5.5.2 13 July 2017, 18:46:47 UTC
72cf57d Apply CLANG Tidy 27 June 2017, 00:47:20 UTC
769778e bump version to 5.5.1 26 June 2017, 22:24:46 UTC
e2f851e Fix bug that flush doesn't respond to fsync result Summary: With a regression bug was introduced two years ago, by https://github.com/facebook/rocksdb/commit/6e9fbeb27c38329f33ae541302c44c8db8374f8c , we fail to check return status of fsync call. This can cause we miss the information from the file system and can potentially cause corrupted data which we could have been detected. Closes https://github.com/facebook/rocksdb/pull/2495 Reviewed By: ajkr Differential Revision: D5321949 Pulled By: siying fbshipit-source-id: c68117914bb40700198fc37d0e4c63163a8a1031 26 June 2017, 19:59:48 UTC
09fce6b update 5.5 change log Summary: update bug fixed. Closes https://github.com/facebook/rocksdb/pull/2434 Differential Revision: D5218601 Pulled By: lightmark fbshipit-source-id: 1f86b2c93345673612381081537d464e7d12e434 09 June 2017, 18:46:37 UTC
343f70f Switch from CentOS 5 to CentOS 6 for crossbuilding RocksJava Summary: Updates the statically linked libraries from linking against glibc 2.5, to linking against glibc 2.12. Closes https://github.com/facebook/rocksdb/pull/2405 Differential Revision: D5184132 Pulled By: sagar0 fbshipit-source-id: 7a8ad4cf7e737ca62f29e58938bd49fa02114541 05 June 2017, 23:32:19 UTC
cb60842 Revert "cross-platform compatibility improvements" This reverts commit 11c5d4741a1e11a1315d5ca644ce555e07e91f61. 25 May 2017, 01:18:51 UTC
a61e7b5 Revert "remove #include port/port.h in public header file" This reverts commit 4c9d2b10468146f42facb98fbc55d31501036d6a. 25 May 2017, 01:18:24 UTC
1182f63 update buckifer/TARGETS Summary: update targets file for release Closes https://github.com/facebook/rocksdb/pull/2358 Differential Revision: D5115705 Pulled By: lightmark fbshipit-source-id: 96a3c7e15b5807b5d0f5a9bb73850b92754b5794 24 May 2017, 22:40:03 UTC
8fd7fb5 range sync should be enabled Summary: We forgot to add the new flag in internal build script. Add it. Closes https://github.com/facebook/rocksdb/pull/2360 Differential Revision: D5121428 Pulled By: siying fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f 24 May 2017, 22:40:03 UTC
e519921 Fix errors in clang-analyzer builds Summary: Fix build error in db_iter.cc when running clang-analyzer. ``` CC db/db_iter.o db/db_iter.cc:938:21: error: no matching constructor for initialization of 'rocksdb::ParsedInternalKey' ParsedInternalKey ikey(Slice(), 0, 0); ^ ~~~~~~~~~~~~~ ./db/dbformat.h:84:3: note: candidate constructor not viable: no known conversion from 'int' to 'rocksdb::ValueType' for 3rd argument ParsedInternalKey(const Slice& u, const SequenceNumber& seq, ValueType t) ^ ./db/dbformat.h:78:8: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 3 were provided struct ParsedInternalKey { ^ ./db/dbformat.h:78:8: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 3 were provided ./db/dbformat.h:83:3: note: candidate constructor not viable: requires 0 arguments, but 3 were provided ParsedInternalKey() { } // Intentionally left uninitialized (for speed) ^ 1 error generated. ``` Closes https://github.com/facebook/rocksdb/pull/2354 Differential Revision: D5115751 Pulled By: sagar0 fbshipit-source-id: b0e386d4e935e4725b07761c3ca5f7a8cbde3692 24 May 2017, 22:40:03 UTC
a1c62c1 Fix release build on Linux Summary: Release builds are failing on Linux with the error: ``` tools/db_stress.cc: In function ‘int main(int, char**)’: tools/db_stress.cc:2365:12: error: ‘rocksdb::SyncPoint’ has not been declared rocksdb::SyncPoint::GetInstance()->SetCallBack( ^ tools/db_stress.cc:2370:12: error: ‘rocksdb::SyncPoint’ has not been declared rocksdb::SyncPoint::GetInstance()->SetCallBack( ^ tools/db_stress.cc:2375:12: error: ‘rocksdb::SyncPoint’ has not been declared rocksdb::SyncPoint::GetInstance()->EnableProcessing(); ^ make[1]: *** [tools/db_stress.o] Error 1 make[1]: Leaving directory `/data/sandcastle/boxes/trunk-git-rocksdb-public' make: *** [release] Error 2 ``` Closes https://github.com/facebook/rocksdb/pull/2355 Differential Revision: D5113552 Pulled By: sagar0 fbshipit-source-id: 351df707277787da5633ba4a40e52edc7c895dc4 24 May 2017, 22:40:03 UTC
fba9d66 Fix rocksdb.estimate-num-keys DB property underflow Summary: rocksdb.estimate-num-keys is compute from `estimate_num_keys - 2 * estimate_num_deletes`. If `2 * estimate_num_deletes > estimate_num_keys` it will underflow. Fixing it. Closes https://github.com/facebook/rocksdb/pull/2348 Differential Revision: D5109272 Pulled By: yiwu-arbug fbshipit-source-id: e1bfb91346a59b7282a282b615002507e9d7c246 23 May 2017, 20:15:10 UTC
3e86c0f disable direct reads for log and manifest and add direct io to tests Summary: Disable direct reads for log and manifest. Direct reads should not affect sequential_file Also add kDirectIO for option_config_ in db_test_util Closes https://github.com/facebook/rocksdb/pull/2337 Differential Revision: D5100261 Pulled By: lightmark fbshipit-source-id: 0ebfd13b93fa1b8f9acae514ac44f8125a05868b 23 May 2017, 01:41:28 UTC
15ba4d6 Address MS Visual Studio 2017 issue with autovector Summary: This addresses https://github.com/facebook/rocksdb/issues/2262 Closes https://github.com/facebook/rocksdb/pull/2333 Differential Revision: D5097941 Pulled By: siying fbshipit-source-id: fb33582bfe7883ecc3f6da028703982522b5f75f 22 May 2017, 17:57:06 UTC
88c818e Replace deprecated RocksDB#addFile with RocksDB#ingestExternalFile Summary: Previously the Java implementation of `RocksDB#addFile` was both incomplete and not inline with the C++ API. Rather than fix it, as I see that `rocksdb::DB::AddFile` is now deprecated in favour of `rocksdb::DB::IngestExternalFile`, I have removed the old broken implementation and implemented `RocksDB#ingestExternalFile`. Closes https://github.com/facebook/rocksdb/issues/2261 Closes https://github.com/facebook/rocksdb/pull/2291 Differential Revision: D5061264 Pulled By: sagar0 fbshipit-source-id: 85df0899fa1b1fc3535175cac4f52353511d4104 22 May 2017, 17:27:23 UTC
228f49d Fix data races caught by tsan Summary: This fixes the tsan build failures in: - write_callback_test - persistent_cache_test.* Closes https://github.com/facebook/rocksdb/pull/2339 Differential Revision: D5101190 Pulled By: sagar0 fbshipit-source-id: 537e19ed05272b1f34cfbf793aa822b2264a1643 22 May 2017, 17:27:23 UTC
4c9d2b1 remove #include port/port.h in public header file Summary: break internal build Closes https://github.com/facebook/rocksdb/pull/2336 Differential Revision: D5097089 Pulled By: lightmark fbshipit-source-id: 6996cbadeead21074a41e526ea04659190ee61d8 20 May 2017, 23:42:21 UTC
07bdcb9 New WriteImpl to pipeline WAL/memtable write Summary: PipelineWriteImpl is an alternative approach to WriteImpl. In WriteImpl, only one thread is allow to write at the same time. This thread will do both WAL and memtable writes for all write threads in the write group. Pending writers wait in queue until the current writer finishes. In the pipeline write approach, two queue is maintained: one WAL writer queue and one memtable writer queue. All writers (regardless of whether they need to write WAL) will still need to first join the WAL writer queue, and after the house keeping work and WAL writing, they will need to join memtable writer queue if needed. The benefit of this approach is that 1. Writers without memtable writes (e.g. the prepare phase of two phase commit) can exit write thread once WAL write is finish. They don't need to wait for memtable writes in case of group commit. 2. Pending writers only need to wait for previous WAL writer finish to be able to join the write thread, instead of wait also for previous memtable writes. Merging #2056 and #2058 into this PR. Closes https://github.com/facebook/rocksdb/pull/2286 Differential Revision: D5054606 Pulled By: yiwu-arbug fbshipit-source-id: ee5b11efd19d3e39d6b7210937b11cefdd4d1c8d 19 May 2017, 21:26:42 UTC
d746aea Suppress clang-analyzer false positive Summary: Fixing two types of clang-analyzer false positives: * db is deleted and then reopen, and clang-analyzer thinks we are reusing the pointer after it has been deleted. Adding asserts to hint clang-analyzer the pointer is recreated. * ParsedInternalKey is (intentionally) uninitialized. Initialize the struct only when clang-analyzer is running. Closes https://github.com/facebook/rocksdb/pull/2334 Differential Revision: D5093801 Pulled By: yiwu-arbug fbshipit-source-id: f51355382098eb3da5ab9f64e094c6d03e6bdf7d 19 May 2017, 17:56:28 UTC
217b866 column_family_test: EnvCounter::num_new_writable_file_ to be atomic Summary: TSAN shows warning of data race of EnvCounter::num_new_writable_file_. Make it atomic. Closes https://github.com/facebook/rocksdb/pull/2331 Differential Revision: D5089215 Pulled By: siying fbshipit-source-id: 15f6dcfb770a3310cbb6337c22482c8b330daffc 18 May 2017, 20:56:12 UTC
9f839a7 keep util/build_version.cc when make clean Summary: https://github.com/facebook/rocksdb/pull/2264 adding build_version.cc into clean list which breaks fbcode release. we need to keep it when `make clean` Closes https://github.com/facebook/rocksdb/pull/2322 Differential Revision: D5088932 Pulled By: lightmark fbshipit-source-id: ab001424af596e94a6bc1d4186c39edf6ace484f 18 May 2017, 19:26:25 UTC
7eecd40 add emacs tags file - etags Summary: added ctags -e to the tags target in the makefile. It creates an etags file suitable for emacs. Closes https://github.com/facebook/rocksdb/pull/2193 Differential Revision: D4983535 Pulled By: siying fbshipit-source-id: 1077ef0676025b8109df37433572533c9e8fe86e 18 May 2017, 14:56:28 UTC
9bbba4f Remoe unused BlockBasedTable::compaction_optimized_ Summary: BlockBasedTable::compaction_optimized_ is never used but can cause TSAN warning. Remove it. Closes https://github.com/facebook/rocksdb/pull/2324 Differential Revision: D5085533 Pulled By: siying fbshipit-source-id: 2feefce6806d559dfb4ab2989aa3db36752fe25d 18 May 2017, 13:41:23 UTC
f5ba131 Fixed some spelling mistakes Summary: Closes https://github.com/facebook/rocksdb/pull/2314 Differential Revision: D5079601 Pulled By: sagar0 fbshipit-source-id: ae5696fd735718f544435c64c3179c49b8c04349 18 May 2017, 06:12:36 UTC
146b771 Fix mingw compilation with -DNDEBUG Summary: This was exposed by a48a62d, which made NDEBUG the default for cmake builds. Closes https://github.com/facebook/rocksdb/pull/2315 Differential Revision: D5079583 Pulled By: sagar0 fbshipit-source-id: c614e96a40df016a834a62b6236852265e7ee4db 18 May 2017, 05:56:48 UTC
a36220c fix unity test Summary: unity test will fail even if we have the same function names in different anonymous namespaces in different files. Closes https://github.com/facebook/rocksdb/pull/2321 Differential Revision: D5083783 Pulled By: lightmark fbshipit-source-id: 1347aaf866900af30d23cdd4f29c1b96f17352af 18 May 2017, 01:56:55 UTC
0ebdd70 fixed typo Summary: fixed typo Closes https://github.com/facebook/rocksdb/pull/2312 Differential Revision: D5079631 Pulled By: sagar0 fbshipit-source-id: e4c8d1d89b244ee69e9dea1dd013227cc5241026 17 May 2017, 23:41:49 UTC
8032f4c Remove -pie in TSAN Summary: -pic seems to be not working in gcc-5 and it is curently broken. Remove it to fix the build. Closes https://github.com/facebook/rocksdb/pull/2320 Differential Revision: D5082775 Pulled By: siying fbshipit-source-id: 5055f987353f1417643a394e7ce05905670410a4 17 May 2017, 22:56:35 UTC
362ba9b Release RocksDB 5.5.0 Summary: change history.md and version Closes https://github.com/facebook/rocksdb/pull/2317 Differential Revision: D5080484 Pulled By: lightmark fbshipit-source-id: 8d70b3b52dc0d34fefc0d34f91d379c27ac13ed3 17 May 2017, 19:42:20 UTC
ba685a4 Support ingest_behind for IngestExternalFile Summary: First cut for early review; there are few conceptual points to answer and some code structure issues. For conceptual points - - restriction-wise, we're going to disallow ingest_behind if (use_seqno_zero_out=true || disable_auto_compaction=false), the user is responsible to properly open and close DB with required params - we wanted to ingest into reserved bottom most level. Should we fail fast if bottom level isn't empty, or should we attempt to ingest if file fits there key-ranges-wise? - Modifying AssignLevelForIngestedFile seems the place we we'd handle that. On code structure - going to refactor GenerateAndAddExternalFile call in the test class to allow passing instance of IngestionOptions, that's just going to incur lots of changes at callsites. Closes https://github.com/facebook/rocksdb/pull/2144 Differential Revision: D4873732 Pulled By: lightmark fbshipit-source-id: 81cb698106b68ef8797f564453651d50900e153a 17 May 2017, 18:42:42 UTC
01ab7b5 Add ROCKSDB_LIBRARY_API macro to a few C APIs, to fix windows build Summary: Windows build in AppVeyor is broken, I believe due to https://github.com/facebook/rocksdb/pull/2254. Error messages: ``` c_test.obj : error LNK2019: unresolved external symbol rocksdb_get_pinned referenced in function CheckPinGet [C:\projects\rocksdb\build\c_test.vcxproj] c_test.obj : error LNK2019: unresolved external symbol rocksdb_get_pinned_cf referenced in function CheckPinGetCF [C:\projects\rocksdb\build\c_test.vcxproj] c_test.obj : error LNK2019: unresolved external symbol rocksdb_pinnableslice_destroy referenced in function CheckPinGet [C:\projects\rocksdb\build\c_test.vcxproj] c_test.obj : error LNK2019: unresolved external symbol rocksdb_pinnableslice_value referenced in function CheckPinGet [C:\projects\rocksdb\build\c_test.vcxproj] C:\projects\rocksdb\build\Debug\c_test.exe : fatal error LNK1120: 4 unresolved externals [C:\projects\rocksdb\build\c_test.vcxproj] ``` See, for example: https://ci.appveyor.com/project/Facebook/rocksdb/build/1.0.4420 Closes https://github.com/facebook/rocksdb/pull/2309 Differential Revision: D5076992 Pulled By: sagar0 fbshipit-source-id: bf4ca063a53b5a9042ba9f655f7c60c268ea5748 17 May 2017, 17:22:41 UTC
cb9392a add Transactions and Checkpoint to C API Summary: I've added functions to the C API to support Transactions as requested in #1637 and to support Checkpoint. I have also added the corresponding tests to c_test.c For now, the following is omitted: 1. Optimistic Transactions 2. The column family variation of functions Closes https://github.com/facebook/rocksdb/pull/2236 Differential Revision: D4989510 Pulled By: yiwu-arbug fbshipit-source-id: 518cb39f76d5e9ec9690d633fcdc014b98958071 17 May 2017, 05:59:43 UTC
445f123 s/std::snprintf/snprintf Summary: Looks like std::snprintf is not available on all platforms (e.g. MSVC 2010). Change it back to snprintf, where we have a macro in port.h to workaround compatibility. Closes https://github.com/facebook/rocksdb/pull/2308 Differential Revision: D5070988 Pulled By: yiwu-arbug fbshipit-source-id: bedfc1660bab0431c583ad434b7e68265e1211b1 16 May 2017, 19:01:04 UTC
cd593c2 Fix travis java_test Summary: Travis java_test is failing because `make clean` happens after `build_version.cc` is generated. Fixing it. Closes #2299 Closes https://github.com/facebook/rocksdb/pull/2307 Differential Revision: D5070781 Pulled By: yiwu-arbug fbshipit-source-id: 18d3ea11d602048f4786b8c7a41f49c2aee774cd 16 May 2017, 18:21:24 UTC
f720796 fixed typo Summary: fixed exisitng -> existing Closes https://github.com/facebook/rocksdb/pull/2305 Differential Revision: D5070169 Pulled By: yiwu-arbug fbshipit-source-id: 8c8450acf50757b767cf78b78314018395738d96 16 May 2017, 18:07:58 UTC
a48a62d define NDEBUG in CMake non-debug builds Summary: This brings CMake builds further in line with builds that go through the normal Makefile. Closes https://github.com/facebook/rocksdb/pull/2300 Differential Revision: D5064631 Pulled By: yiwu-arbug fbshipit-source-id: 7b2b2d5299f575f87badcf590cc95e040f14d52d 16 May 2017, 18:07:58 UTC
1ca723d C API: support pinnable get Summary: Closes https://github.com/facebook/rocksdb/pull/2254 Differential Revision: D5053590 Pulled By: yiwu-arbug fbshipit-source-id: 2f365a031b3a2947b4fba21d26d4f8f52af9b9f0 16 May 2017, 18:07:58 UTC
2ef15b8 Core-local stats blog post Summary: as titled Closes https://github.com/facebook/rocksdb/pull/2294 Differential Revision: D5070009 Pulled By: siying fbshipit-source-id: 88e41839322e7f7d4b00f7057c58bd8e7d012dad 16 May 2017, 18:07:58 UTC
4f9e69c fix log err Summary: Closes https://github.com/facebook/rocksdb/pull/2206 Differential Revision: D5054222 Pulled By: yiwu-arbug fbshipit-source-id: d8742bda1bf3e76d7b68eeb86df4608031b5cbc8 15 May 2017, 23:15:38 UTC
11c5d47 cross-platform compatibility improvements Summary: We've had a couple CockroachDB users fail to build RocksDB on exotic platforms, so I figured I'd try my hand at solving these issues upstream. The problems stem from a) `USE_SSE=1` being too aggressive about turning on SSE4.2, even on toolchains that don't support SSE4.2 and b) RocksDB attempting to detect support for thread-local storage based on OS, even though it can vary by compiler on the same OS. See the individual commit messages for details. Regarding SSE support, this PR should change virtually nothing for non-CMake based builds. `make`, `PORTABLE=1 make`, `USE_SSE=1 make`, and `PORTABLE=1 USE_SSE=1 make` function exactly as before, except that SSE support will be automatically disabled when a simple SSE4.2-using test program fails to compile, as it does on OpenBSD. (OpenBSD's ports GCC supports SSE4.2, but its binutils do not, so `__SSE_4_2__` is defined but an SSE4.2-using program will fail to assemble.) A warning is emitted in this case. The CMake build is modified to support the same set of options, except that `USE_SSE` is spelled `FORCE_SSE42` because `USE_SSE` is rather useless now that we can automatically detect SSE support, and I figure changing options in the CMake build is less disruptive than changing the non-CMake build. I've tested these changes on all the platforms I can get my hands on (macOS, Windows MSVC, Windows MinGW, and OpenBSD) and it all works splendidly. Let me know if there's anything you object to—I obviously don't mean to break any of your build pipelines in the process of fixing ours downstream. Closes https://github.com/facebook/rocksdb/pull/2199 Differential Revision: D5054042 Pulled By: yiwu-arbug fbshipit-source-id: 938e1fc665c049c02ae15698e1409155b8e72171 15 May 2017, 23:15:38 UTC
d004333 Put lib files into suitable path in RPM package Summary: Currently, the RPM package will install the lib and header files into `/usr/package/lib` and `/usr/package/include` which is not in the default search paths. It is reasonable to install them under `/usr/lib` and `/usr/include` so that no extra configuration is required. Closes https://github.com/facebook/rocksdb/pull/2221 Differential Revision: D5054030 Pulled By: yiwu-arbug fbshipit-source-id: 1d23de5ff21f07e6738c9dfa04429acd7a839143 15 May 2017, 21:05:46 UTC
86d5492 Fix build error with blob DB. Summary: snprintf is in <stdio.h> and not in namespace std. Closes https://github.com/facebook/rocksdb/pull/2287 Reviewed By: anirbanr-fb Differential Revision: D5054752 Pulled By: yiwu-arbug fbshipit-source-id: 356807ec38f3c7d95951cdb41f31a3d3ae0714d4 15 May 2017, 21:05:46 UTC
254c468 Fix the RocksJava Release on Windows Summary: This was previously broken accidentally by https://github.com/facebook/rocksdb/pull/2107 Closes https://github.com/facebook/rocksdb/issues/2293 Closes https://github.com/facebook/rocksdb/pull/2296 Differential Revision: D5061248 Pulled By: sagar0 fbshipit-source-id: 7f58fee754723a7052d2a7f9d3d0369051c3cc5c 15 May 2017, 19:37:01 UTC
7a47b43 Fix .gitignore pattern Summary: `java/**.asc` is not a correct gitignore pattern See https://git-scm.com/docs/gitignore for the list of allowed `**` patterns It seems reasonable to assume that intention is `java/**/*.asc` The reason why it bothers me is the fact that ripgrep parses .gitignore files and complains about invalid pattern https://github.com/BurntSushi/ripgrep Closes https://github.com/facebook/rocksdb/pull/2214 Differential Revision: D5063030 Pulled By: yiwu-arbug fbshipit-source-id: ddd6682b81f03134be15f20fd596130776b69695 15 May 2017, 19:23:10 UTC
fa5a15c Make sure that zstd is statically linked correctly in the Java static build Summary: Closes https://github.com/facebook/rocksdb/issues/2280 Closes https://github.com/facebook/rocksdb/pull/2292 Differential Revision: D5061259 Pulled By: sagar0 fbshipit-source-id: eec89111d114c04beee5870a4eb4b51857754783 15 May 2017, 18:12:08 UTC
3fa9a39 Add GetAllKeyVersions API Summary: - Introduced an include/ file dedicated to db-related debug functions to avoid making db.h more complex - Added debugging function, `GetAllKeyVersions()`, to return a listing of internal data for a range of user keys. The new `struct KeyVersion` exposes data similar to internal key without exposing any internal type. - Migrated the "ldb idump" subcommand to use this function - The API takes an inclusive-exclusive range to match behavior of "ldb idump". This will be quite annoying for users who want to query a single user key's versions :(. Closes https://github.com/facebook/rocksdb/pull/2232 Differential Revision: D4976007 Pulled By: ajkr fbshipit-source-id: cab375da53a7595d6575af2b7e3b776aa3ad793e 12 May 2017, 22:54:06 UTC
1a60982 Simplified instructions for CentOS Summary: Minimal changes for improvements to INSTALL.md for CentOS/RHEL Closes https://github.com/facebook/rocksdb/pull/2266 Differential Revision: D5053530 Pulled By: yiwu-arbug fbshipit-source-id: c05f30299f8efec949b9001a73969d649536ea8a 12 May 2017, 19:23:18 UTC
a5cc7ec Facility for cross-building RocksJava using Docker Summary: As an alternative to Vagrant, we can now also use Docker to cross-build RocksDB. The advantages are: 1. The Docker images are fixed; they include all the latest updates and build tools. 2. The Vagrant image, required scripts that ran for every build that would update CentOS and install the buildtools. This lead to slow repeatable builds, we don't need to do this with Docker as they are already in the provided images. The Docker images I have used have their Docker build files here: https://github.com/evolvedbinary/docker-rocksjava and the images themselves are available from Docker hub: https://hub.docker.com/r/evolvedbinary/rocksjava/ I have added the following targets to the `Makefile`: 1. `rocksdbjavastaticreleasedocker` this uses Docker to perform the cross-builds. It is basically the Docker version of the existing Vagrant `rocksdbjavastaticrelease` target. 2. `rocksdbjavastaticpublishdocker` delegates to `rocksdbjavastaticreleasedocker` and then `rocksdbjavastaticpublishcentral` to upload the artiacts to Maven Central. Equivalent to the existing Vagrant target: `rocksdbjavastaticpublish` Closes https://github.com/facebook/rocksdb/pull/2278 Differential Revision: D5048206 Pulled By: yiwu-arbug fbshipit-source-id: 78fa96ef9d966fe09638ed01de282cd4e31961a9 12 May 2017, 18:41:21 UTC
ccd3ddd Blog post for partitioned index/filters Summary: Closes https://github.com/facebook/rocksdb/pull/2285 Differential Revision: D5053096 Pulled By: maysamyabandeh fbshipit-source-id: c696a6b82391d039c446d16222313f78341c37e8 12 May 2017, 18:01:30 UTC
b145c34 Update blog authors Summary: Closes https://github.com/facebook/rocksdb/pull/2284 Differential Revision: D5052835 Pulled By: maysamyabandeh fbshipit-source-id: afe5fdb4b2d659cf612446a666dbc1d11afc3b5d 12 May 2017, 18:01:30 UTC
bbe9ee7 core-local array type conversions Summary: try to clean up the type conversions and hope it passes on windows. one interesting thing I learned is that bitshift operations are special: in `x << y`, the result type depends only on the type of `x`, unlike most arithmetic operations where the result type depends on both operands' types. Closes https://github.com/facebook/rocksdb/pull/2277 Differential Revision: D5050145 Pulled By: ajkr fbshipit-source-id: f3309e77526ac9612c632bf93a62d99757af9a29 12 May 2017, 16:28:07 UTC
c2be434 Build and link with ZStd when creating the static RocksJava build Summary: Closes https://github.com/facebook/rocksdb/pull/2279 Differential Revision: D5048161 Pulled By: yiwu-arbug fbshipit-source-id: 43742ff93137e0a35ea7e855692c9e9a0cd41968 11 May 2017, 22:27:10 UTC
c61e72c Add missing files of blob_db to CMake file Summary: Some of the file from #2269 didn't add to CMake file. Adding them to fix window build. Closes https://github.com/facebook/rocksdb/pull/2276 Differential Revision: D5043487 Pulled By: yiwu-arbug fbshipit-source-id: 4eba853e9d92574353abce21d77d30e47ce43d3d 11 May 2017, 16:52:22 UTC
3907c94 Fix ColumnFamilyTest:BulkAddDrop Summary: Fix ColumnFamilyTest:BulkAddDrop not deleted CF handles at the end, causing ASAN failure. Closes https://github.com/facebook/rocksdb/pull/2275 Differential Revision: D5040724 Pulled By: yiwu-arbug fbshipit-source-id: 86cd4070c944d01173a3cc36462bb800698af192 11 May 2017, 06:05:44 UTC
cda5fde CoreLocalArray class Summary: Moved the logic for core-local array out of ConcurrentArena and into a separate class because I want to reuse it for core-local stats. Closes https://github.com/facebook/rocksdb/pull/2256 Differential Revision: D5011518 Pulled By: ajkr fbshipit-source-id: a75a7b8f7b7a42fd6273489ada405f14c6be196a 11 May 2017, 01:25:36 UTC
9394966 update TARGETS Summary: address siying's comment in #2272. Closes https://github.com/facebook/rocksdb/pull/2274 Differential Revision: D5039489 Pulled By: ajkr fbshipit-source-id: 3e2d957d3469c13d0e33ededa59320c4c3f24ef6 11 May 2017, 00:57:28 UTC
4e83b80 title: Bulkoading -> title: Bulkloading Summary: Closes https://github.com/facebook/rocksdb/pull/2219 Differential Revision: D4986642 Pulled By: lightmark fbshipit-source-id: c9328991e742768fb5caa0e88e022afb514f0c65 10 May 2017, 23:56:45 UTC
d85ff49 Blob storage pr Summary: The final pull request for Blob Storage. Closes https://github.com/facebook/rocksdb/pull/2269 Differential Revision: D5033189 Pulled By: yiwu-arbug fbshipit-source-id: 6356b683ccd58cbf38a1dc55e2ea400feecd5d06 10 May 2017, 22:14:44 UTC
492fc49 fix readampbitmap tests Summary: fix test failure of ReadAmpBitmap and ReadAmpBitmapLiveInCacheAfterDBClose. test ReadAmpBitmapLiveInCacheAfterDBClose individually and make check Closes https://github.com/facebook/rocksdb/pull/2271 Differential Revision: D5038133 Pulled By: lightmark fbshipit-source-id: 803cd6f45ccfdd14a9d9473c8af311033e164be8 10 May 2017, 19:29:23 UTC
be421b0 portable sched_getcpu calls Summary: - added a feature test in build_detect_platform to check whether sched_getcpu() is available. glibc offers it only on some platforms (e.g., linux but not mac); this way should be easier than maintaining a list of platforms on which it's available. - refactored PhysicalCoreID() to be simpler / less repetitive. ordered the conditional compilation clauses from most-to-least preferred Closes https://github.com/facebook/rocksdb/pull/2272 Differential Revision: D5038093 Pulled By: ajkr fbshipit-source-id: 81d7db3cc620250de220bdeb3194b2b3d7673de7 10 May 2017, 19:29:23 UTC
0f559ab Add NO_UPDATE_BUILD_VERSION option to makefile Summary: When building rocksdb in fbcode using `make`, util/build_version.cc is always updated (gitignore/hgignore doesn't apply because the file is already checked into fbcode). To use the rocksdb makefile from our own makefile, I would like an option to prevent the metadata update, which is of no value for us. Closes https://github.com/facebook/rocksdb/pull/2264 Differential Revision: D5037846 Pulled By: siying fbshipit-source-id: 9fa005725c5ecb31d9cbe2e738cbee209591f08a 10 May 2017, 18:27:40 UTC
3a04a25 Flink state Summary: This is to address the issue reported in https://github.com/facebook/rocksdb/issues/1988 The fix is simple. A typo. Closes https://github.com/facebook/rocksdb/pull/2267 Differential Revision: D5037149 Pulled By: siying fbshipit-source-id: 1bb585c7a753ef77c81c4b92deafbed8e21fe8ff 10 May 2017, 18:12:38 UTC
35df23f Fix suite exclisions Summary: Closes https://github.com/facebook/rocksdb/pull/2250 Differential Revision: D5037141 Pulled By: siying fbshipit-source-id: 53a0ab5553422839bb2df6a7badde54810774a3e 10 May 2017, 18:12:38 UTC
e7cea86 Fixes the CentOS 5 cross-building of RocksJava Summary: Updates to CentOS 5 have been archived as CentOS 5 is EOL. We now pull the updates from the vault. This is a stop gap solution, I will send a PR in a couple days which uses fixed Docker containers (with the updates pre-installed) instead. sagar0 Here you go :-) Closes https://github.com/facebook/rocksdb/pull/2270 Differential Revision: D5033637 Pulled By: sagar0 fbshipit-source-id: a9312dd1bc18bfb8653f06ffa0a1512b4415720d 10 May 2017, 15:57:54 UTC
259a00e unbiase readamp bitmap Summary: Consider BlockReadAmpBitmap with bytes_per_bit = 32. Suppose bytes [a, b) were used, while bytes [a-32, a) and [b+1, b+33) weren't used; more formally, the union of ranges passed to BlockReadAmpBitmap::Mark() contains [a, b) and doesn't intersect with [a-32, a) and [b+1, b+33). Then bits [floor(a/32), ceil(b/32)] will be set, and so the number of useful bytes will be estimated as (ceil(b/32) - floor(a/32)) * 32, which is on average equal to b-a+31. An extreme example: if we use 1 byte from each block, it'll be counted as 32 bytes from each block. It's easy to remove this bias by slightly changing the semantics of the bitmap. Currently each bit represents a byte range [i*32, (i+1)*32). This diff makes each bit represent a single byte: i*32 + X, where X is a random number in [0, 31] generated when bitmap is created. So, e.g., if you read a single byte at random, with probability 31/32 it won't be counted at all, and with probability 1/32 it will be counted as 32 bytes; so, on average it's counted as 1 byte. *But there is one exception: the last bit will always set with the old way.* (*) - assuming read_amp_bytes_per_bit = 32. Closes https://github.com/facebook/rocksdb/pull/2259 Differential Revision: D5035652 Pulled By: lightmark fbshipit-source-id: bd98b1b9b49fbe61f9e3781d07f624e3cbd92356 10 May 2017, 08:49:52 UTC
a620966 port: updated PhysicalCoreID() Summary: Updated PhysicalCoreID() to use sched_getcpu() on x86_64 for glibc >= 2.22. Added a new function named GetCPUID() that calls sched_getcpu(), to avoid repeated code. This change is done as per the comments of PR: https://github.com/facebook/rocksdb/pull/2230 Signed-off-by: Jos Collin <jcollin@redhat.com> Closes https://github.com/facebook/rocksdb/pull/2260 Differential Revision: D5025734 Pulled By: ajkr fbshipit-source-id: f4cca68c12573cafcf8531e7411a1e733bbf8eef 10 May 2017, 02:06:39 UTC
df035b6 Print compaction_options_universal.stop_style in LOG file Summary: Print compaction_options_universal.stop_style in LOG file ./db_bench --benchmarks=fillseq and read the log Closes https://github.com/facebook/rocksdb/pull/2268 Differential Revision: D5032438 Pulled By: lightmark fbshipit-source-id: 0e72fcd96a1caaf3cab20e86d39c75fbebf5ce37 09 May 2017, 22:57:44 UTC
4897eb2 dont skip IO for filter blocks Summary: Based on my experience with linkbench, We should not skip loading bloom filter blocks when they are not available in block cache when using Iterator::Seek Actually I am not sure why this behavior existed in the first place Closes https://github.com/facebook/rocksdb/pull/2255 Differential Revision: D5010721 Pulled By: maysamyabandeh fbshipit-source-id: 0af545a06ac4baeecb248706ec34d009c2480ca4 09 May 2017, 16:52:02 UTC
3f73d54 Add C API to set max_file_opening_threads option Summary: Add `rocksdb_options_set_max_file_opening_threads()` API Closes https://github.com/facebook/rocksdb/pull/2184 Differential Revision: D4923090 Pulled By: lightmark fbshipit-source-id: c4ddce17733d999d426d02f7202b33a46ed6faed 09 May 2017, 05:49:32 UTC
0b69e50 Define CACHE_LINE_SIZE only when it's not defined Summary: RocksDB is compiled as part of MyRocks (MySQL storage engine) build. MySQL already defines `CACHE_LINE_SIZE` and therefore we're getting a conflict. Change RocksDB definition to be more cognizant of this. Closes https://github.com/facebook/rocksdb/pull/2257 Differential Revision: D5013188 Pulled By: gunnarku fbshipit-source-id: cfa76fe99f90dcd82aa09204e2f1f35e07a82b41 08 May 2017, 23:12:28 UTC
2cd0077 Add bulk create/drop column family API Summary: Adding DB::CreateColumnFamilie() and DB::DropColumnFamilies() to bulk create/drop column families. This is to address the problem creating/dropping 1k column families takes minutes. The bottleneck is we persist options files for every single column family create/drop, and it parses the persisted options file for verification, which take a lot CPU time. The new APIs simply create/drop column families individually, and persist options file once at the end. This improves create 1k column families to within ~0.1s. Further improvement can be merge manifest write to one IO. Closes https://github.com/facebook/rocksdb/pull/2248 Differential Revision: D5001578 Pulled By: yiwu-arbug fbshipit-source-id: d4e00bda671451e0b314c13e12ad194b1704aa03 08 May 2017, 06:20:46 UTC
40af238 Object lifetime in cache Summary: Any non-raw-data dependent object must be destructed before the table closes. There was a bug of not doing that for filter object. This patch fixes the bug and adds a unit test to prevent such bugs in future. Closes https://github.com/facebook/rocksdb/pull/2246 Differential Revision: D5001318 Pulled By: maysamyabandeh fbshipit-source-id: 6d8772e58765485868094b92964da82ef9730b6d 06 May 2017, 06:20:01 UTC
fdaefa0 travis: add Windows cross-compilation Summary: - downcase includes for case-sensitive filesystems - give targets the same name (librocksdb) on all platforms With this patch it is possible to cross-compile RocksDB for Windows from a Linux host using mingw. cc yuslepukhin orgads Closes https://github.com/facebook/rocksdb/pull/2107 Differential Revision: D4849784 Pulled By: siying fbshipit-source-id: ad26ed6b4d393851aa6551e6aa4201faba82ef60 06 May 2017, 06:20:01 UTC
a30a696 do not read next datablock if upperbound is reached Summary: Now if we have iterate_upper_bound set, we continue read until get a key >= upper_bound. For a lot of cases that neighboring data blocks have a user key gap between them, our index key will be a user key in the middle to get a shorter size. For example, if we have blocks: [a b c d][f g h] Then the index key for the first block will be 'e'. then if upper bound is any key between 'd' and 'e', for example, d1, d2, ..., d99999999999, we don't have to read the second block and also know that we have done our iteration by reaching the last key that smaller the upper bound already. This diff can reduce RA in most cases. Closes https://github.com/facebook/rocksdb/pull/2239 Differential Revision: D4990693 Pulled By: lightmark fbshipit-source-id: ab30ea2e3c6edf3fddd5efed3c34fcf7739827ff 06 May 2017, 06:20:01 UTC
2d42cf5 Roundup read bytes in ReadaheadRandomAccessFile Summary: Fix alignment in ReadaheadRandomAccessFile Closes https://github.com/facebook/rocksdb/pull/2253 Differential Revision: D5012336 Pulled By: lightmark fbshipit-source-id: 10d2c829520cb787227ef653ef63d5d701725778 05 May 2017, 19:14:14 UTC
264d3f5 Allow IntraL0 compaction in FIFO Compaction Summary: Allow an option for users to do some compaction in FIFO compaction, to pay some write amplification for fewer number of files. Closes https://github.com/facebook/rocksdb/pull/2163 Differential Revision: D4895953 Pulled By: siying fbshipit-source-id: a1ab608dd0627211f3e1f588a2e97159646e1231 05 May 2017, 01:16:13 UTC
8c3a180 Set lower-bound on dynamic level sizes Summary: Changed dynamic leveling to stop setting the base level's size bound below `max_bytes_for_level_base`. Behavior for config where `max_bytes_for_level_base == level0_file_num_compaction_trigger * write_buffer_size` and same amount of data in L0 and base-level: - Before #2027, compaction scoring would favor base-level due to dividing by size smaller than `max_bytes_for_level_base`. - After #2027, L0 and Lbase get equal scores. The disadvantage is L0 is often compacted before reaching the num files trigger since `write_buffer_size` can be bigger than the dynamically chosen base-level size. This increases write-amp. - After this diff, L0 and Lbase still get equal scores. Now it takes `level0_file_num_compaction_trigger` files of size `write_buffer_size` to trigger L0 compaction by size, fixing the write-amp problem above. Closes https://github.com/facebook/rocksdb/pull/2123 Differential Revision: D4861570 Pulled By: ajkr fbshipit-source-id: 467ddef56ed1f647c14d86bb018bcb044c39b964 05 May 2017, 01:16:12 UTC
7c1c8ce Avoid calling fallocate with UINT64_MAX Summary: When user doesn't set a limit on compaction output file size, let's use the sum of the input files' sizes. This will avoid passing UINT64_MAX as fallocate()'s length. Reported in #2249. Test setup: - command: `TEST_TMPDIR=/data/rocksdb-test/ strace -e fallocate ./db_compaction_test --gtest_filter=DBCompactionTest.ManualCompactionUnknownOutputSize` - filesystem: xfs before this diff: `fallocate(10, 01, 0, 1844674407370955160) = -1 ENOSPC (No space left on device)` after this diff: `fallocate(10, 01, 0, 1977) = 0` Closes https://github.com/facebook/rocksdb/pull/2252 Differential Revision: D5007275 Pulled By: ajkr fbshipit-source-id: 4491404a6ae8a41328aede2e2d6f4d9ac3e38880 05 May 2017, 00:43:22 UTC
a45e98a max_open_files dynamic set, follow up Summary: Followup to make 0x40000 a TableCache constant that indicates infinite capacity Closes https://github.com/facebook/rocksdb/pull/2247 Differential Revision: D5001349 Pulled By: lgalanis fbshipit-source-id: ce7bd2e54b0975bb9f8680fdaa0f8bb0e7ae81a2 04 May 2017, 17:42:45 UTC
6b99dbe fix memory alignment with logical sector size Summary: we align the buffer with logical sector size and should not test it with page size, which is usually 4k. Closes https://github.com/facebook/rocksdb/pull/2245 Differential Revision: D5001842 Pulled By: lightmark fbshipit-source-id: a7135fcf6351c6db363e8908956b1e193a4a6291 04 May 2017, 08:30:13 UTC
e7ae4a3 Max open files mutable Summary: Makes max_open_files db option dynamically set-able by SetDBOptions. During the call of SetDBOptions we call SetCapacity on the table cache, which is a LRUCache. Closes https://github.com/facebook/rocksdb/pull/2185 Differential Revision: D4979189 Pulled By: yiwu-arbug fbshipit-source-id: ca7e8dc5e3619c79434f579be4847c0f7e56afda 04 May 2017, 04:13:14 UTC
60847a3 port: updated PhysicalCoreID() Summary: Checked the return value of __get_cpuid(). Implemented the else case where the arch is different from i386 and x86_64. Pulled By: ajkr Differential Revision: D4973496 fbshipit-source-id: c40fdef5840364c2a79b1d11df0db5d4ec3d6a4a 03 May 2017, 20:08:55 UTC
b551104 support PopSavePoint for WriteBatch Summary: Try to fix https://github.com/facebook/rocksdb/issues/1969 Closes https://github.com/facebook/rocksdb/pull/2170 Differential Revision: D4907333 Pulled By: yiwu-arbug fbshipit-source-id: 417b420ff668e6c2fd0dad42a94c57385012edc5 03 May 2017, 17:57:45 UTC
498693c Remove orphaned Java classes Summary: Remove orphaned Java classes that were missed from commit 9385fd7 Closes https://github.com/facebook/rocksdb/issues/2210 Closes https://github.com/facebook/rocksdb/pull/2212 Differential Revision: D4986199 Pulled By: maysamyabandeh fbshipit-source-id: 74e866cd8661c172a4754c2c945e30d650f6fa05 03 May 2017, 04:13:49 UTC
5e2ebf2 travis: add CMake compilation Summary: siying Closes https://github.com/facebook/rocksdb/pull/2233 Differential Revision: D4987113 Pulled By: siying fbshipit-source-id: f07ecd3f7d9f5366a2b1665ce28ba10d74405557 02 May 2017, 22:11:59 UTC
af6fe69 Fix an issue of manual / auto compaction data race Summary: A data race between a manual and an auto compaction can cause a scheduled automatic compaction to be cancelled and never rescheduled again. This may cause a condition of hanging forever. Fix this by always making sure the cancelled compaction is put back to the compaction queue. Closes https://github.com/facebook/rocksdb/pull/2238 Differential Revision: D4984591 Pulled By: siying fbshipit-source-id: 3ab153886403c7b991896dcb2158b96cac12f227 02 May 2017, 22:11:59 UTC
6798d1f Revert "Delete filter before closing the table" Summary: This reverts commit 89833577a80ad7a2cbf6b99c5957f572b3548152. Closes https://github.com/facebook/rocksdb/pull/2240 Differential Revision: D4986982 Pulled By: maysamyabandeh fbshipit-source-id: 56c4c07b7b5b7c6fe122d5c2f2199d221c8510c0 02 May 2017, 20:46:39 UTC
back to top