Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
20d49e4 Yi Wu 25 July 2017, 17:11:29 UTC Bump version to 5.6.1 25 July 2017, 17:11:29 UTC
891ac40 Yi Wu 21 July 2017, 03:28:43 UTC Remove make_new_version.sh Summary: Seems the only function of the script is to create a new branch, which can be done easily. I'm removing it. Closes https://github.com/facebook/rocksdb/pull/2623 Differential Revision: D5468681 Pulled By: yiwu-arbug fbshipit-source-id: 87dea5ecc4c85e06941ccbc36993f7f589063878 21 July 2017, 06:02:01 UTC
81f194e Islam AbdelRahman 14 July 2017, 22:36:20 UTC Make TARGETS file portable Summary: Instead of hard coding the path of the internal repo. Make TARGETS file work anywhere in fbcode Closes https://github.com/facebook/rocksdb/pull/2586 Differential Revision: D5428122 Pulled By: IslamAbdelRahman fbshipit-source-id: 21adec82bfbff14ea93532bee789b5f5bbee5b01 21 July 2017, 00:28:53 UTC
8c4caa5 Islam AbdelRahman 14 July 2017, 18:02:51 UTC Remove arcanist_util directory 21 July 2017, 00:27:39 UTC
1973c95 Siying Dong 18 July 2017, 19:58:57 UTC Update java/rocksjni.pom Summary: Closes https://github.com/facebook/rocksdb/pull/2610 Differential Revision: D5445705 Pulled By: siying fbshipit-source-id: d5f97ffdf5bc18b853c3a106755aca96a650e428 21 July 2017, 00:27:05 UTC
183c3a7 Siying Dong 18 July 2017, 18:46:31 UTC Remove some left-over BSD headers Summary: Closes https://github.com/facebook/rocksdb/pull/2608 Differential Revision: D5444797 Pulled By: siying fbshipit-source-id: 690581d03f37822e059a16085088e8e2d8a45016 21 July 2017, 00:25:34 UTC
8cecbae Siying Dong 17 July 2017, 01:35:39 UTC Add back the LevelDB license file Summary: Closes https://github.com/facebook/rocksdb/pull/2591 Differential Revision: D5432696 Pulled By: siying fbshipit-source-id: a613230ab916de0b279a65ef429ede65460a8db2 21 July 2017, 00:23:55 UTC
6c86638 Siying Dong 16 July 2017, 22:50:14 UTC Remove the licensing description in CONTRIBUTING.md Summary: Closes https://github.com/facebook/rocksdb/pull/2590 Differential Revision: D5432539 Pulled By: siying fbshipit-source-id: 49902453bba3c95c1fb8354441b2198649e52bf4 21 July 2017, 00:23:17 UTC
bd56719 Siying Dong 15 July 2017, 23:03:42 UTC Change RocksDB License Summary: Closes https://github.com/facebook/rocksdb/pull/2589 Differential Revision: D5431502 Pulled By: siying fbshipit-source-id: 8ebf8c87883daa9daa54b2303d11ce01ab1f6f75 21 July 2017, 00:21:13 UTC
396982d Yi Wu 17 July 2017, 21:51:19 UTC Fix column_family_test with LITE build Summary: Fix column_family_test with LITE build. I need this patch to fix 5.6 branch. Closes https://github.com/facebook/rocksdb/pull/2597 Differential Revision: D5437171 Pulled By: yiwu-arbug fbshipit-source-id: 88b9dc5925a6b47af10c1b41bc5b07c4251a84b5 17 July 2017, 22:35:35 UTC
5daf734 Giuseppe Ottaviano 10 July 2017, 19:22:26 UTC Fix undefined behavior in Hash Summary: Instead of ignoring UBSan checks, fix the negative shifts in Hash(). Also add test to make sure the hash values are stable over time. The values were computed before this change, so the test also verifies the correctness of the change. Closes https://github.com/facebook/rocksdb/pull/2546 Differential Revision: D5386369 Pulled By: yiwu-arbug fbshipit-source-id: 6de4b44461a544d6222cc5d72d8cda2c0373d17e 10 July 2017, 19:35:57 UTC
30328d0 Siying Dong 29 June 2017, 04:37:55 UTC Bug fix: Fast CRC Support printing is not honest Summary: 11c5d4741a1e11a1315d5ca644ce555e07e91f61 introduces a bug that IsFastCrc32Supported() returns wrong result. Fix it. Also fix some FB internal scripts. Closes https://github.com/facebook/rocksdb/pull/2513 Differential Revision: D5343802 Pulled By: yiwu-arbug fbshipit-source-id: 057dc7ae3b262fe951413d1190ce60afc788cc05 29 June 2017, 05:30:28 UTC
82c83ab Dmitri Smirnov 20 June 2017, 17:16:24 UTC Implement ReopenWritibaleFile on Windows and other fixes Summary: Make default impl return NoSupported so the db_blob tests exist in a meaningful manner. Replace std::thread to port::Thread Closes https://github.com/facebook/rocksdb/pull/2465 Differential Revision: D5275563 Pulled By: yiwu-arbug fbshipit-source-id: cedf1a18a2c05e20d768c1308b3f3224dbd70ab6 27 June 2017, 23:12:47 UTC
662fb11 Yi Wu 14 June 2017, 20:44:36 UTC Fix blob db compression bug Summary: `CompressBlock()` will return the uncompressed slice (i.e. `Slice(value_unc)`) if compression ratio is not good enough. This is undesired. We need to always assign the compressed slice to `value`. Closes https://github.com/facebook/rocksdb/pull/2447 Differential Revision: D5244682 Pulled By: yiwu-arbug fbshipit-source-id: 6989dd8852c9622822ba9acec9beea02007dff09 27 June 2017, 23:12:38 UTC
f882a9a Yi Wu 14 June 2017, 20:08:54 UTC Update blob_db_test Summary: I'm trying to improve unit test of blob db. I'm rewriting blob db test. In this patch: * Rewrite tests of basic put/write/delete operations. * Add disable_background_tasks to BlobDBOptionsImpl to allow me not running any background job for basic unit tests. * Move DestroyBlobDB out from BlobDBImpl to be a standalone function. * Remove all garbage collection related tests. Will rewrite them in following patch. * Disabled compression test since it is failing. Will fix in a followup patch. Closes https://github.com/facebook/rocksdb/pull/2446 Differential Revision: D5243306 Pulled By: yiwu-arbug fbshipit-source-id: 157c71ad3b699307cb88baa3830e9b6e74f8e939 27 June 2017, 23:12:25 UTC
c4f251e Yi Wu 13 June 2017, 19:37:59 UTC write exact sequence number for each put in write batch Summary: At the beginning of write batch write, grab the latest sequence from base db and assume sequence number will increment by 1 for each put and delete, and write the exact sequence number with each put. This is assuming we are the only writer to increment sequence number (no external file ingestion, etc) and there should be no holes in the sequence number. Also having some minor naming changes. Closes https://github.com/facebook/rocksdb/pull/2402 Differential Revision: D5176134 Pulled By: yiwu-arbug fbshipit-source-id: cb4712ee44478d5a2e5951213a10b72f08fe8c88 27 June 2017, 23:12:08 UTC
b4dadbe Yi Wu 27 June 2017, 21:01:14 UTC Fix TARGETS file tests list Summary: 1. The buckifier script assume each test "foo" comes with a .cc file of the same name (i.e. foo.cc). Update cassandra tests to follow this pattern so that the buckifier script can recognize them. 2. add blob_db_test Closes https://github.com/facebook/rocksdb/pull/2506 Differential Revision: D5331517 Pulled By: yiwu-arbug fbshipit-source-id: 86f3eba471fc621186ab44cbd073b6162cde8e57 27 June 2017, 21:16:36 UTC
37616ec Yi Wu 27 June 2017, 18:25:00 UTC allow numa >= 2.0.8 Summary: Allow numa >= 2.0.8 in buck TARGET file. Closes https://github.com/facebook/rocksdb/pull/2504 Differential Revision: D5330550 Pulled By: yiwu-arbug fbshipit-source-id: 8ffb6167b4ad913877eac16a20a91023b31f8d41 27 June 2017, 21:14:21 UTC
02e3611 Andrew Kryczka 20 June 2017, 20:16:55 UTC Fix cassandra ASAN use-after-free Summary: When we create a column based on the `string::c_str()`, we need to make sure that char array doesn't get deleted when calls to `string::append()` cause the string to expand. Closes https://github.com/facebook/rocksdb/pull/2470 Differential Revision: D5285049 Pulled By: ajkr fbshipit-source-id: f918dd426ff3c024e7a293dcb10448f10b6c98e8 27 June 2017, 19:37:23 UTC
43dcab8 Chen Shen 16 June 2017, 21:12:52 UTC Create a MergeOperator for Cassandra Row Value Summary: This PR implements the MergeOperator for Cassandra Row Values. Closes https://github.com/facebook/rocksdb/pull/2289 Differential Revision: D5055464 Pulled By: scv119 fbshipit-source-id: 45f276ef8cbc4704279202f6a20c64889bc1adef 27 June 2017, 19:36:50 UTC
1059bf7 Ben Torfs 14 June 2017, 23:57:39 UTC Synchronize statistic enumeration values between statistics.h and java API Summary: Closes https://github.com/facebook/rocksdb/pull/2209 Differential Revision: D5251951 Pulled By: sagar0 fbshipit-source-id: 03a73d025a7b4a322bb8d8d86f5d249fcd7dd00e 27 June 2017, 19:36:14 UTC
399baba Sagar Vemuri 12 June 2017, 19:51:19 UTC Update HistogramTypes in the Java API Summary: This diff syncs the Histogram Types in the Java API with the ones in C++ API (`statistics.h`), and brings it up-to-date. I also found that the enum ordering between Java and C++ has gotten out-of-sync, a few years back, with the addition of `SUBCOMPACTION_SETUP_TIME`. So updated the order as well. `READ_NUM_MERGE_OPERANDS` added in #2373 is needed for Cassandra-on-RocksDB work. Closes https://github.com/facebook/rocksdb/pull/2429 Differential Revision: D5215623 Pulled By: sagar0 fbshipit-source-id: bd136698c48197e53693275eb52acc9198ee5a4e 27 June 2017, 19:35:23 UTC
555c42d Siying Dong 27 June 2017, 17:49:34 UTC CLANG Tidy Summary: Closes https://github.com/facebook/rocksdb/pull/2502 Differential Revision: D5326498 Pulled By: siying fbshipit-source-id: 2f0ac6dc6ca5ddb23cecf67a278c086e52646714 27 June 2017, 18:06:09 UTC
55d5bb3 Yi Wu 27 June 2017, 17:48:40 UTC update compatible test Summary: update compatible test to include 5.5 and 5.6 branch. Closes https://github.com/facebook/rocksdb/pull/2501 Differential Revision: D5325220 Pulled By: yiwu-arbug fbshipit-source-id: 5f5271491e6dd2d7b2cf73a7142f38a571553bc4 27 June 2017, 18:05:57 UTC
21c8501 sdong 26 June 2017, 22:39:27 UTC Minor typo in HISTORY.md 26 June 2017, 22:39:52 UTC
35c6efc Yi Wu 26 June 2017, 22:24:13 UTC Fix jni WriteBatchThreadedTest Summary: WriteBatchThreadedTest is failing, at least on Mac. The problem seems to be `wb` is getting GC before we finish write. Explicitly close it seems to fix it. Closes https://github.com/facebook/rocksdb/pull/2482 Differential Revision: D5307379 Pulled By: yiwu-arbug fbshipit-source-id: 8ff7f8170451078c941951f5aafae83afffb7933 26 June 2017, 22:31:44 UTC
3d85c04 Aaron Gao 26 June 2017, 22:13:35 UTC revert perf_context and io_stats to __thread Summary: https://github.com/facebook/rocksdb/pull/2380 introduces a regression by replacing __thread with ThreadLocalPtr. Revert the thread local implementation back. Closes https://github.com/facebook/rocksdb/pull/2485 Differential Revision: D5308050 Pulled By: lightmark fbshipit-source-id: 2676e9c22edf76e8133d3f4c50e2711e11a95480 26 June 2017, 22:31:29 UTC
b6e539f Siying Dong 26 June 2017, 19:42:21 UTC Improve the error message for I/O related errors. Summary: Force people to write something other than file name while returning status for IOError. Closes https://github.com/facebook/rocksdb/pull/2493 Differential Revision: D5321309 Pulled By: siying fbshipit-source-id: 38bcf6c19e80831cd3e300a047e975cbb131d822 26 June 2017, 22:22:30 UTC
df0f181 jsteemann 23 June 2017, 16:32:11 UTC fixed wrong type for "allow_compaction" parameter Summary: should be boolean, not uint64_t MSVC complains about it during compilation with error `include\rocksdb\advanced_options.h(77): warning C4800: 'uint64_t': forcing value to bool 'true' or 'false' (performance warning)` Closes https://github.com/facebook/rocksdb/pull/2487 Differential Revision: D5310685 Pulled By: siying fbshipit-source-id: 719a33b3dba4f711aa72e3f229013c188015dc86 26 June 2017, 19:46:28 UTC
1097ddb Siying Dong 22 June 2017, 22:45:42 UTC Fix Data Race Between CreateColumnFamily() and GetAggregatedIntProperty() Summary: CreateColumnFamily() releases DB mutex after adding column family to the set and install super version (to write option file), so if users call GetAggregatedIntProperty() in the middle, then super version will be null and the process will crash. Fix it by skipping those column families without super version installed. Maybe we should also fix the problem of releasing the lock when reading option file, but it is more risky. so I'm doing a quick and safer fix and we can investigate it later. Closes https://github.com/facebook/rocksdb/pull/2475 Differential Revision: D5298053 Pulled By: siying fbshipit-source-id: 4b3c8f91c60400b163fcc6cda8a0c77723be0ef6 26 June 2017, 19:45:14 UTC
25c4c4a Siying Dong 22 June 2017, 23:16:19 UTC Downgrade option sanitiy check level for prefix_extractor Summary: With c7004840d2f4ad5fc1bdce042902b822492f3a0e, it's safe to open a DB with different prefix extractor. So it's safe to skip prefix extractor check. Closes https://github.com/facebook/rocksdb/pull/2474 Differential Revision: D5294700 Pulled By: siying fbshipit-source-id: eeb500da795eecb29b8c9c56a14cfd4afda12ecc 26 June 2017, 19:44:47 UTC
9cc9e43 Siying Dong 26 June 2017, 19:32:52 UTC Fix bug that flush doesn't respond to fsync result Summary: With a regression bug was introduced two years ago, by https://github.com/facebook/rocksdb/commit/6e9fbeb27c38329f33ae541302c44c8db8374f8c , we fail to check return status of fsync call. This can cause we miss the information from the file system and can potentially cause corrupted data which we could have been detected. Closes https://github.com/facebook/rocksdb/pull/2495 Reviewed By: ajkr Differential Revision: D5321949 Pulled By: siying fbshipit-source-id: c68117914bb40700198fc37d0e4c63163a8a1031 26 June 2017, 19:44:03 UTC
e90cdc2 Siying Dong 21 June 2017, 17:28:54 UTC WriteBufferManager will not trigger flush if much data is already being flushed Summary: Even if hard limit hits, flushing more memtable may not help cap the memory usage if already more than half data is scheduled for flush. Not triggering flush instead. Closes https://github.com/facebook/rocksdb/pull/2469 Differential Revision: D5284249 Pulled By: siying fbshipit-source-id: 8ab7ba1aba56a634dbe72b318fcab2093063972e 21 June 2017, 20:12:59 UTC
a1def6c Sagar Vemuri 13 June 2017, 23:55:08 UTC Allow ignoring unknown options when loading options from a file Summary: Added a flag, `ignore_unknown_options`, to skip unknown options when loading an options file (using `LoadLatestOptions`/`LoadOptionsFromFile`) or while verifying options (using `CheckOptionsCompatibility`). This will help in downgrading the db to an older version. Also added `--ignore_unknown_options` flag to ldb **Example Use case:** In MyRocks, if copying from newer version to older version, it is often impossible to start because of new RocksDB options that don't exist in older version, even though data format is compatible. MyRocks uses these load and verify functions in [ha_rocksdb.cc::check_rocksdb_options_compatibility](https://github.com/facebook/mysql-5.6/blob/e004fd9f416821d043ccc8ad4a345c33ac9953f0/storage/rocksdb/ha_rocksdb.cc#L3348-L3401). **Test Plan:** Updated the unit tests. `make check` ldb: $ ./ldb --db=/tmp/test_db --create_if_missing put a1 b1 OK Now edit /tmp/test_db/<OPTIONS-file> and add an unknown option. Try loading the options now, and it fails: $ ./ldb --db=/tmp/test_db --try_load_options get a1 Failed: Invalid argument: Unrecognized option DBOptions:: abcd Passes with the new --ignore_unknown_options flag $ ./ldb --db=/tmp/test_db --try_load_options --ignore_unknown_options get a1 b1 Closes https://github.com/facebook/rocksdb/pull/2423 Differential Revision: D5212091 Pulled By: sagar0 fbshipit-source-id: 2ec17636feb47dc0351b53a77e5f15ef7cbf2ca7 20 June 2017, 17:03:31 UTC
67bbba5 zhangjinpeng1987 19 June 2017, 18:39:45 UTC fix coredump for release nullptr Summary: Coredump will be triggered when ingest external sst file after delete range. ref https://github.com/facebook/rocksdb/issues/2398 Closes https://github.com/facebook/rocksdb/pull/2463 Differential Revision: D5275599 Pulled By: ajkr fbshipit-source-id: 0828dbc062ea8c74e913877cd63494fd3478a30d 19 June 2017, 18:52:35 UTC
b1934ad Andrew Kryczka 12 June 2017, 19:41:41 UTC cleanup history 12 June 2017, 19:41:41 UTC
d3a7348 Andrew Kryczka 12 June 2017, 19:27:54 UTC update history for 5.6 Summary: - mention range deletion + file ingestion - move post-5.6 stuff into new section Closes https://github.com/facebook/rocksdb/pull/2440 Differential Revision: D5229910 Pulled By: ajkr fbshipit-source-id: 1facfe41993fa1f3b1f6fa7dc77d2b11aa2b317a 12 June 2017, 19:33:13 UTC
a1e5ae6 Siying Dong 12 June 2017, 13:32:01 UTC Fix RocksDB Lite build with CLANG Summary: Closes https://github.com/facebook/rocksdb/pull/2419 Differential Revision: D5193976 Pulled By: siying fbshipit-source-id: 62d115edee6043237e9d6ad3c2a05481e162c9eb 12 June 2017, 18:20:51 UTC
f25ee31 Aaron Gao 09 June 2017, 18:06:34 UTC update 5.5 change log Summary: update bug fixed. Closes https://github.com/facebook/rocksdb/pull/2434 Differential Revision: D5218601 Pulled By: lightmark fbshipit-source-id: 1f86b2c93345673612381081537d464e7d12e434 09 June 2017, 19:27:24 UTC
4222399 Mike Kolupaev 09 June 2017, 02:54:00 UTC Fix crash in PosixWritableFile::Close() when fstat() fails Summary: We had a crash in this code: `fstat()` failed; `file_stats` contained garbage, in particular `file_stats.st_blksize == 6`; the expression `file_stats.st_blocks / (file_stats.st_blksize / 512)` divided by zero. Closes https://github.com/facebook/rocksdb/pull/2420 Differential Revision: D5216110 Pulled By: al13n321 fbshipit-source-id: 6d8fc5e7c4f98c1139e68c7829ebdbac68b0fce0 09 June 2017, 17:35:21 UTC
4b6e3c4 Yi Wu 09 June 2017, 00:28:33 UTC Fix mock_env.cc uninitialized variable Summary: Mingw is complaining about uninitialized variable in mock_env.cc. e.g. https://travis-ci.org/facebook/rocksdb/jobs/240132276 The fix is to initialize the variable. Closes https://github.com/facebook/rocksdb/pull/2428 Differential Revision: D5211306 Pulled By: yiwu-arbug fbshipit-source-id: ee02bf0327dcea8590a2aa087f0176fecaf8621c 09 June 2017, 17:34:30 UTC
6a7d920 Sagar Vemuri 09 June 2017, 00:11:10 UTC Java APIs for put, merge and delete in file ingestion Summary: Adding SSTFileWriter's newly introduced put, merge and delete apis to the Java api. The C++ APIs were first introduced in #2361. Add is deprecated in favor of Put. Merge is especially needed to support streaming for Cassandra-on-RocksDB work in https://issues.apache.org/jira/browse/CASSANDRA-13476. Closes https://github.com/facebook/rocksdb/pull/2392 Differential Revision: D5165091 Pulled By: sagar0 fbshipit-source-id: 6f0ad396a7cbd2e27ca63e702584784dd72acaab 09 June 2017, 17:34:23 UTC
dd6e5c7 Yi Wu 08 June 2017, 19:30:28 UTC Disable DBRangeDelTest::TailingIteratorRangeTombstoneUnsupported for ubsan Summary: UBSAN crashes when it run the test. Disabling it for UBSAN. Closes https://github.com/facebook/rocksdb/pull/2427 Differential Revision: D5210897 Pulled By: yiwu-arbug fbshipit-source-id: 2f5a876807c98d8db79ab9581965f7e6b29d4163 09 June 2017, 17:33:51 UTC
d124230 Maysam Yabandeh 08 June 2017, 17:38:45 UTC fix travis error with init time in mockenv Summary: /home/travis/build/facebook/rocksdb/env/mock_env.cc: In member function ‘virtual void rocksdb::{anonymous}::TestMemLogger::Logv(const char*, va_list)’: /home/travis/build/facebook/rocksdb/env/mock_env.cc:391:53: error: ‘t.tm::tm_year’ may be used uninitialized in this function [-Werror=maybe-uninitialized] static_cast<int>(now_tv.tv_usec)); Closes https://github.com/facebook/rocksdb/pull/2418 Differential Revision: D5193597 Pulled By: maysamyabandeh fbshipit-source-id: 8801a3ef27f33eb419d534f7de747702cdf504a0 09 June 2017, 17:33:40 UTC
550a1df Maysam Yabandeh 06 June 2017, 19:50:56 UTC Fix clang errors by asserting the precondition Summary: USE_CLANG=1 make -j32 analyze The two errors would disappear after the assertion. Closes https://github.com/facebook/rocksdb/pull/2416 Differential Revision: D5193526 Pulled By: maysamyabandeh fbshipit-source-id: 16a21f18f68023f862764dd3ab9e00ca60b0eefa 06 June 2017, 19:56:52 UTC
cc5f933 Maysam Yabandeh 06 June 2017, 19:48:46 UTC Fix concurrency issue with filter_block_set_ Summary: filter_block_set_ access must also be protected with mutex. Closes https://github.com/facebook/rocksdb/pull/2413 Differential Revision: D5193159 Pulled By: maysamyabandeh fbshipit-source-id: 6987fc219d9a65c20b9c7e52151aef4b8e4882e6 06 June 2017, 19:56:52 UTC
2e64f45 Aaron Gao 05 June 2017, 22:55:55 UTC bump version to 5.6 Summary: Bump version to 5.6 beforehand in master Closes https://github.com/facebook/rocksdb/pull/2411 Differential Revision: D5186896 Pulled By: lightmark fbshipit-source-id: 079538e621b1a959c2dc99dada894e9cdb99ef95 05 June 2017, 23:15:21 UTC
afbc2d0 Yi Wu 05 June 2017, 22:34:40 UTC Force travis to build with clang on MacOS Summary: Attempt to force travis to build with clang on MacOS Closes https://github.com/facebook/rocksdb/pull/2408 Differential Revision: D5186635 Pulled By: yiwu-arbug fbshipit-source-id: dbb779eff07b1cb7dbd2092631303cf946316656 05 June 2017, 22:41:57 UTC
b172a3f Sagar Vemuri 05 June 2017, 22:16:37 UTC Fix warnings while generating RocksJava documentation Summary: There are a couple of warnings while building RocksJava, coming from Javadoc generation. ``` Generating target/apidocs/org/rocksdb/RocksDB.html... src/main/java/org/rocksdb/RocksDB.java:2139: warning: no throws for org.rocksdb.RocksDBException public void ingestExternalFile(final List<String> filePathList, ^ src/main/java/org/rocksdb/RocksDB.java:2162: warning: no throws for org.rocksdb.RocksDBException public void ingestExternalFile(final ColumnFamilyHandle columnFamilyHandle, ^ ``` Closes https://github.com/facebook/rocksdb/pull/2396 Differential Revision: D5178388 Pulled By: sagar0 fbshipit-source-id: a0ab6696d6de78d089a9a860a559f64cc320019e 05 June 2017, 22:28:00 UTC
52a7f38 Siying Dong 05 June 2017, 21:42:34 UTC WriteOptions.low_pri which can throttle low pri writes if needed Summary: If ReadOptions.low_pri=true and compaction is behind, the write will either return immediate or be slowed down based on ReadOptions.no_slowdown. Closes https://github.com/facebook/rocksdb/pull/2369 Differential Revision: D5127619 Pulled By: siying fbshipit-source-id: d30e1cff515890af0eff32dfb869d2e4c9545eb0 05 June 2017, 22:02:35 UTC
26a8a80 Adam Retter 05 June 2017, 19:16:02 UTC Switch from CentOS 5 to CentOS 6 for crossbuilding RocksJava Summary: Updates the statically linked libraries from linking against glibc 2.5, to linking against glibc 2.12. Closes https://github.com/facebook/rocksdb/pull/2405 Differential Revision: D5184132 Pulled By: sagar0 fbshipit-source-id: 7a8ad4cf7e737ca62f29e58938bd49fa02114541 05 June 2017, 19:27:24 UTC
dba9f37 Yi Wu 05 June 2017, 19:06:23 UTC Fix db_write_test clang/windows build failure Summary: Fix db_write_test clang/windows build failure. Explicitly cast size_t (unsigned long) to uint32_t (unsigned int). Closes https://github.com/facebook/rocksdb/pull/2407 Differential Revision: D5182995 Pulled By: yiwu-arbug fbshipit-source-id: aba225a9fccb12d5bfbdc2cd6efc11040706a9d2 05 June 2017, 19:27:24 UTC
c7662a4 hyunwoo 05 June 2017, 18:23:31 UTC fixed typo Summary: fixed typo Closes https://github.com/facebook/rocksdb/pull/2376 Differential Revision: D5183630 Pulled By: ajkr fbshipit-source-id: 133cfd0445959e70aa2cd1a12151bf3c0c5c3ac5 05 June 2017, 18:27:34 UTC
7e8d95c Adam Retter 05 June 2017, 01:40:37 UTC Fix the Java build which was broken by a4d9c02 Summary: Closes https://github.com/facebook/rocksdb/pull/2406 Differential Revision: D5181091 Pulled By: ajkr fbshipit-source-id: fd72525da4fb1d50143080a210f8d824cbb968d6 05 June 2017, 01:41:33 UTC
7e5fac2 Aaron Gao 03 June 2017, 00:12:52 UTC remove test dir before exit when current regression is running Summary: clean up the current test dir if the last regression test is still running. Closes https://github.com/facebook/rocksdb/pull/2401 Differential Revision: D5177882 Pulled By: lightmark fbshipit-source-id: 91d899fcc2bde841948eae71af8584d4bdb35468 03 June 2017, 00:26:19 UTC
7f6c02d Aaron Gao 03 June 2017, 00:12:39 UTC using ThreadLocalPtr to hide ROCKSDB_SUPPORT_THREAD_LOCAL from public… Summary: … headers https://github.com/facebook/rocksdb/pull/2199 should not reference RocksDB-specific macros (like ROCKSDB_SUPPORT_THREAD_LOCAL in this case) to public headers, `iostats_context.h` and `perf_context.h`. We shouldn't do that because users have to provide these compiler flags when building their binary with RocksDB. We should hide the thread local global variable inside our implementation and just expose a function api to retrieve these variables. It may break some users for now but good for long term. make check -j64 Closes https://github.com/facebook/rocksdb/pull/2380 Differential Revision: D5177896 Pulled By: lightmark fbshipit-source-id: 6fcdfac57f2e2dcfe60992b7385c5403f6dcb390 03 June 2017, 00:26:19 UTC
138b87e Mike Kolupaev 02 June 2017, 21:56:31 UTC Fix interaction between CompactionFilter::Decision::kRemoveAndSkipUnt… Summary: Fixes the following scenario: 1. Set prefix extractor. Enable bloom filters, with `whole_key_filtering = false`. Use compaction filter that sometimes returns `kRemoveAndSkipUntil`. 2. Do a compaction. 3. Compaction creates an iterator with `total_order_seek = false`, calls `SeekToFirst()` on it, then repeatedly calls `Next()`. 4. At some point compaction filter returns `kRemoveAndSkipUntil`. 5. Compaction calls `Seek(skip_until)` on the iterator. The key that it seeks to happens to have prefix that doesn't match the bloom filter. Since `total_order_seek = false`, iterator becomes invalid, and compaction thinks that it has reached the end. The rest of the compaction input is silently discarded. The fix is to make compaction iterator use `total_order_seek = true`. The implementation for PlainTable is quite awkward. I've made `kRemoveAndSkipUntil` officially incompatible with PlainTable. If you try to use them together, compaction will fail, and DB will enter read-only mode (`bg_error_`). That's not a very graceful way to communicate a misconfiguration, but the alternatives don't seem worth the implementation time and complexity. To be able to check in advance that `kRemoveAndSkipUntil` is not going to be used with PlainTable, we'd need to extend the interface of either `CompactionFilter` or `InternalIterator`. It seems unlikely that anyone will ever want to use `kRemoveAndSkipUntil` with PlainTable: PlainTable probably has very few users, and `kRemoveAndSkipUntil` has only one user so far: us (logdevice). Closes https://github.com/facebook/rocksdb/pull/2349 Differential Revision: D5110388 Pulled By: lightmark fbshipit-source-id: ec29101a99d9dcd97db33923b87f72bce56cc17a 02 June 2017, 22:11:38 UTC
95b0e89 Siying Dong 02 June 2017, 21:13:59 UTC Improve write buffer manager (and allow the size to be tracked in block cache) Summary: Improve write buffer manager in several ways: 1. Size is tracked when arena block is allocated, rather than every allocation, so that it can better track actual memory usage and the tracking overhead is slightly lower. 2. We start to trigger memtable flush when 7/8 of the memory cap hits, instead of 100%, and make 100% much harder to hit. 3. Allow a cache object to be passed into buffer manager and the size allocated by memtable can be costed there. This can help users have one single memory cap across block cache and memtable. Closes https://github.com/facebook/rocksdb/pull/2350 Differential Revision: D5110648 Pulled By: siying fbshipit-source-id: b4238113094bf22574001e446b5d88523ba00017 02 June 2017, 21:26:56 UTC
a4d9c02 Andrew Kryczka 02 June 2017, 19:08:01 UTC Pass CF ID to MemTableRepFactory Summary: Some users want to monitor column family activity in their custom memtable implementations. Previously there was no way to figure out with which column family a memtable is associated. This diff: - adds an overload to MemTableRepFactory::CreateMemTableRep() that provides the CF ID. For compatibility, its default implementation calls the old overload. - updates MemTable to create MemTableRep's using the new overload. Closes https://github.com/facebook/rocksdb/pull/2346 Differential Revision: D5108061 Pulled By: ajkr fbshipit-source-id: 3a1921214a348dd8ea0f54e1cab3b71c3d46d616 02 June 2017, 19:12:06 UTC
f68d88b Yi Wu 02 June 2017, 18:41:29 UTC Fix DBWriteTest::ReturnSequenceNumberMultiThreaded data race Summary: rocksdb::Random is not thread-safe. Have one Random for each thread instead. Closes https://github.com/facebook/rocksdb/pull/2400 Differential Revision: D5173919 Pulled By: yiwu-arbug fbshipit-source-id: 1a99c7b877f3893eb22355af49e321bcad4e53e6 02 June 2017, 18:42:11 UTC
215076e Andrew Kryczka 02 June 2017, 05:14:27 UTC Fix TSAN: avoid arena mode with range deletions Summary: The range deletion meta-block iterators weren't getting cleaned up properly since they don't support arena allocation. I didn't implement arena support since, in the general case, each iterator is used only once and separately from all other iterators, so there should be no benefit to data locality. Anyways, this diff fixes up #2370 by treating range deletion iterators as non-arena-allocated. Closes https://github.com/facebook/rocksdb/pull/2399 Differential Revision: D5171119 Pulled By: ajkr fbshipit-source-id: bef6f5c4c5905a124f4993945aed4bd86e2807d8 02 June 2017, 05:26:49 UTC
3a8a848 Andrew Kryczka 02 June 2017, 00:54:06 UTC account for L0 size in estimated compaction bytes Summary: also changed the `>` in the comparison against `level0_file_num_compaction_trigger` into a `>=` since exactly `level0_file_num_compaction_trigger` can trigger a compaction from L0. Closes https://github.com/facebook/rocksdb/pull/2179 Differential Revision: D4915772 Pulled By: ajkr fbshipit-source-id: e38fec6253de6f9a40e61734615c6670d84038aa 02 June 2017, 00:56:59 UTC
0fae3f5 Andrew Gallagher 02 June 2017, 00:48:17 UTC codemod: format TARGETS with buildifier [5/5] (D5092623) Reviewed By: igorsugak fbshipit-source-id: 906b744c179eb932f5a388b39f93209cecd50a80 02 June 2017, 00:56:59 UTC
8721996 Aaron Gao 01 June 2017, 22:42:02 UTC add checkpoint support for single db in regression test Summary: For level_compaction_style regression test. Closes https://github.com/facebook/rocksdb/pull/2397 Differential Revision: D5168545 Pulled By: lightmark fbshipit-source-id: 195e4d84917e7c261d9f4fbe9aee5d104c9cb9a2 01 June 2017, 22:56:59 UTC
5a9b4d7 Maysam Yabandeh 01 June 2017, 22:30:27 UTC Retire memenv https://github.com/facebook/rocksdb/pull/2082 Summary: This is a manual commit of this PR: Retire InMemoryEnv in favor of MockEnv #2082 With MockEnv doing the same yet being more mature, InMemoryEnv is redundant. Reviewed By: IslamAbdelRahman Differential Revision: D5162323 fbshipit-source-id: 59fd0082a891dc99cc531e4da9d68bf891eae3f5 01 June 2017, 22:41:20 UTC
d601965 Islam AbdelRahman 01 June 2017, 19:31:13 UTC sync internal/external TARGETS 01 June 2017, 19:31:13 UTC
bbaba51 Volker Mische 01 June 2017, 18:26:01 UTC Add missing index type to C-API Summary: When the `TwoLevelIndexSearch` was introduced, it wasn't added to the C-API. Closes https://github.com/facebook/rocksdb/pull/2395 Differential Revision: D5165127 Pulled By: maysamyabandeh fbshipit-source-id: d077f16ab5646c18158d8202a33b0fd076c6c8ad 01 June 2017, 18:27:04 UTC
292edfd Daniel Black 01 June 2017, 17:11:23 UTC travis: test with xcode8.3 (OS X 10.12) Summary: Use later xcode version from https://docs.travis-ci.com/user/osx-ci-environment Closes https://github.com/facebook/rocksdb/pull/2128 Differential Revision: D4907471 Pulled By: yiwu-arbug fbshipit-source-id: debf8e27baef71a5833c845401b1865bc75ac977 01 June 2017, 17:11:50 UTC
0dc3040 Tamir Duberstein 01 June 2017, 05:41:44 UTC db: avoid `#include`ing malloc and jemalloc simultaneously Summary: This fixes a compilation failure on Linux when the system libc is not glibc. jemalloc's configure script incorrectly assumes that glibc is always used on Linux systems, producing glibc-style signatures; when the system libc is e.g. musl, the following error is observed: ``` [ 0%] Building CXX object CMakeFiles/rocksdb.dir/db/db_impl.cc.o In file included from /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb.src/table/block.h:19:0, from /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb.src/db/db_impl.cc:77: /x-tools/x86_64-unknown-linux-musl/x86_64-unknown-linux-musl/sysroot/usr/include/malloc.h:19:8: error: declaration of 'size_t malloc_usable_size(void*)' has a different exception specifier size_t malloc_usable_size(void *); ^~~~~~~~~~~~~~~~~~ In file included from /go/src/github.com/cockroachdb/cockroach/c-deps/rocksdb.src/db/db_impl.cc:20:0: /go/native/x86_64-unknown-linux-musl/jemalloc/include/jemalloc/jemalloc.h:78:33: note: from previous declaration 'size_t malloc_usable_size(void*) throw ()' # define je_malloc_usable_size malloc_usable_size ^ /go/native/x86_64-unknown-linux-musl/jemalloc/include/jemalloc/jemalloc.h:239:41: note: in expansion of macro 'je_malloc_usable_size' JEMALLOC_EXPORT size_t JEMALLOC_NOTHROW je_malloc_usable_size( ^~~~~~~~~~~~~~~~~~~~~ CMakeFiles/rocksdb.dir/build.make:350: recipe for target 'CMakeFiles/rocksdb.dir/db/db_impl.cc.o' failed ``` This works around the issue by rearranging the sources such that jemalloc's headers are never in the same scope as the system's malloc header. The jemalloc issue has been reported as well, see: https://github.com/jemalloc/jemalloc/issues/778. cc tschottdorf Closes https://github.com/facebook/rocksdb/pull/2188 Differential Revision: D5163048 Pulled By: siying fbshipit-source-id: c553125458892def175c1be5682b0330d80b2a0d 01 June 2017, 05:43:02 UTC
9b3ed83 Aaron Gao 31 May 2017, 22:33:02 UTC fix regression test Summary: fix regression test by not reporting stats when building db Closes https://github.com/facebook/rocksdb/pull/2390 Differential Revision: D5159909 Pulled By: lightmark fbshipit-source-id: c3f4b9deb9c6799ff84207fd341c529144f8158d 31 May 2017, 22:41:45 UTC
9c9909b Andrew Kryczka 31 May 2017, 20:43:25 UTC Support ingest file when range deletions exist Summary: Previously we returned NotSupported when ingesting files into a database containing any range deletions. This diff adds the support. - Flush if any memtable contains range deletions overlapping the to-be-ingested file - Place to-be-ingested file before any level that contains range deletions overlapping it. - Added support for `Version` to return iterators over range deletions in a given level. Previously, we piggybacked getting range deletions onto `Version`'s `Get()` / `AddIterator()` functions by passing them a `RangeDelAggregator*`. But file ingestion needs to get iterators over range deletions, not populate an aggregator (since the aggregator does collapsing and doesn't expose the actual ranges). Closes https://github.com/facebook/rocksdb/pull/2370 Differential Revision: D5127648 Pulled By: ajkr fbshipit-source-id: 816faeb9708adfa5287962bafdde717db56e3f1a 31 May 2017, 20:57:19 UTC
ad19eb8 Yi Wu 31 May 2017, 17:45:47 UTC Fixing blob db sequence number handling Summary: Blob db rely on base db returning sequence number through write batch after DB::Write(). However after recent changes to the write path, DB::Writ()e no longer return sequence number in some cases. Fixing it by have WriteBatchInternal::InsertInto() always encode sequence number into write batch. Stacking on #2375. Closes https://github.com/facebook/rocksdb/pull/2385 Differential Revision: D5148358 Pulled By: yiwu-arbug fbshipit-source-id: 8bda0aa07b9334ed03ed381548b39d167dc20c33 31 May 2017, 17:56:45 UTC
51ac91f Siying Dong 31 May 2017, 14:27:40 UTC Histogram of number of merge operands Summary: Add a histogram in statistics to help users understand how many merge operands they merge. Closes https://github.com/facebook/rocksdb/pull/2373 Differential Revision: D5139983 Pulled By: siying fbshipit-source-id: 61b9ba8ca83f358530a4833d68f0103b56a0e182 31 May 2017, 14:41:44 UTC
345878a Yi Wu 31 May 2017, 05:16:32 UTC update blob_db_test Summary: Re-enable blob_db_test with some update: * Commented out delay at the end of GC tests. Will update the logic later with sync point to properly trigger GC. * Added some helper functions. Also update make files to include blob_dump tool. Closes https://github.com/facebook/rocksdb/pull/2375 Differential Revision: D5133793 Pulled By: yiwu-arbug fbshipit-source-id: 95470b26d0c1f9592ba4b7637e027fdd263f425c 31 May 2017, 05:26:13 UTC
cbc821c Aaron Gao 30 May 2017, 23:33:26 UTC change regression rebuild to one level Summary: abandon fillseqdeterministic test locally Closes https://github.com/facebook/rocksdb/pull/2290 Differential Revision: D5151867 Pulled By: lightmark fbshipit-source-id: 4c8a24cc937212ffb5ceb9bfaf7288eb8726d0c1 30 May 2017, 23:41:21 UTC
103d069 Tamir Duberstein 30 May 2017, 18:05:28 UTC Avoid unsupported attributes when not building with UBSAN Summary: yiwu-arbug see individual commits. Closes https://github.com/facebook/rocksdb/pull/2318 Differential Revision: D5141520 Pulled By: yiwu-arbug fbshipit-source-id: 7987c92ab4461eef36afce5a133d3a0ee0c96300 30 May 2017, 18:13:01 UTC
5fd0456 Tamir Duberstein 30 May 2017, 17:32:16 UTC travis: reduce the number of travis builders Summary: This collapses all the "platform dependent" tests into a single travis builder in an effort to reduce overall CI times. These builds currently take a combined 21-23 minutes, but each one has to compile the library, so combining them should yield some time savings (5-10 minutes). Unfortunately the other builders don't duplicate work, so combining them is unlikely to provide benefit. Closes https://github.com/facebook/rocksdb/pull/2306 Differential Revision: D5147850 Pulled By: yiwu-arbug fbshipit-source-id: d947dc8b9f49639fe22f3c8ab9a82a8d730ddddf 30 May 2017, 17:42:01 UTC
2d05002 Sagar Vemuri 27 May 2017, 06:11:15 UTC RocksDB 5.4.5 release blog post Summary: 5.4.5 release blog post. Tag created at: https://github.com/facebook/rocksdb/releases/tag/rocksdb-5.4.5 Closes https://github.com/facebook/rocksdb/pull/2378 Differential Revision: D5141648 Pulled By: sagar0 fbshipit-source-id: 43faadfbd7feb8bb01c3d56127ef3453e23dc28a 27 May 2017, 06:11:56 UTC
7eca90f Sagar Vemuri 27 May 2017, 03:04:32 UTC Update RocksDB blog authors Summary: Adding my name to the authors list so that I can publish a post to rocksdb blog (rocksdb.org). Closes https://github.com/facebook/rocksdb/pull/2379 Differential Revision: D5143582 Pulled By: sagar0 fbshipit-source-id: d85163f8b59aaeb07ac2a1cdd776ae335c7062b9 27 May 2017, 03:12:41 UTC
d03c344 赵星宇 26 May 2017, 21:59:21 UTC update comment of GetNextFile Summary: Closes https://github.com/facebook/rocksdb/pull/2377 Differential Revision: D5141274 Pulled By: lightmark fbshipit-source-id: c237a285b73ad93488c080ea80c71a29a17f1be0 26 May 2017, 22:12:13 UTC
f7bb1a0 Aaron Gao 26 May 2017, 19:05:19 UTC support merge and delete in file ingestion Summary: Previously sst_file_writer only supports kTypeValue, we need kTypeMerge and kTypeDeletion also as user requested. Closes https://github.com/facebook/rocksdb/pull/2361 Differential Revision: D5139402 Pulled By: lightmark fbshipit-source-id: 092a60756d01692539d817a3765ebfd58a8d7f88 26 May 2017, 19:11:21 UTC
c2c62ad Siying Dong 26 May 2017, 18:27:24 UTC Reorder variables of ReadOptions Summary: Reorder variables of ReadOptions so that its size is reduced from 64 to 48 bytes. Closes https://github.com/facebook/rocksdb/pull/2366 Differential Revision: D5124043 Pulled By: siying fbshipit-source-id: 70e9c204c34f97fad011f2fe2297ba292d85df7a 26 May 2017, 18:41:20 UTC
7bb1f5d Sagar Vemuri 26 May 2017, 16:50:49 UTC Increase of compaction threads should be logged at info level instead of a warning Summary: This log message shouldn't be a warning; some services are seeing high warning count due to this. The count for the below line is a few hundreds of millions, as per Logview: ``` [rocksdb/src/db/column_family.cc:729] [checkpoints] Increasing compaction threads because we have 2 level-0 files ``` Closes https://github.com/facebook/rocksdb/pull/2364 Differential Revision: D5123565 Pulled By: sagar0 fbshipit-source-id: a07ce499a4f82f0ebde9cda9f4948fb9df6a734c 26 May 2017, 16:56:13 UTC
6c456ec Sagar Vemuri 26 May 2017, 16:50:45 UTC Clean zstd files Summary: zstd files are downloaded and used as part of JNI build, but are left behind even after doing a `make clean`. This PR updates the `clean` target to remove these zstd files as well. Closes https://github.com/facebook/rocksdb/pull/2365 Differential Revision: D5123537 Pulled By: sagar0 fbshipit-source-id: a8f355da5ba961aa89d5852e35751ffc35de03ea 26 May 2017, 16:56:13 UTC
347e16f Andrew Gallagher 25 May 2017, 22:10:00 UTC codemod: replace `headers = AutoHeaders.*` with `auto_headers` Reviewed By: meyering Differential Revision: D5094332 fbshipit-source-id: 3df2f693def8ca418bc9febe3e20ccf051f2e19d 25 May 2017, 22:12:03 UTC
0be636b Yi Wu 25 May 2017, 21:08:16 UTC Fix db_bench build break with blob db Summary: Lite build does not recognize FLAGS_use_blob_db. Fixing it. Closes https://github.com/facebook/rocksdb/pull/2372 Reviewed By: anirbanr-fb Differential Revision: D5130773 Pulled By: yiwu-arbug fbshipit-source-id: 43131d9d0be5811f2129af562be72cca26369cb3 25 May 2017, 21:11:22 UTC
135ee6a Aaron Gao 25 May 2017, 17:41:00 UTC fix tsan crash data race Summary: rand_ has data race risk TEST_TMPDIR=\/dev\/shm\/rocksdb OPT=-g COMPILE_WITH_TSAN=1 CRASH_TEST_KILL_ODD=1887 make J=1 crash_test Closes https://github.com/facebook/rocksdb/pull/2368 Differential Revision: D5127424 Pulled By: lightmark fbshipit-source-id: b7f4d1430a5769b57da9f99037106749264b2ced 25 May 2017, 17:44:07 UTC
a99fb99 Andrew Kryczka 24 May 2017, 23:26:36 UTC fix column_family_test asan Summary: stop calling Close() at the end of tests holding a compaction pressure token since it causes the write controller to be deleted while it's still needed. these calls were pointless anyways since Close() is already called in the test's destructor. Closes https://github.com/facebook/rocksdb/pull/2367 Differential Revision: D5125906 Pulled By: ajkr fbshipit-source-id: 6cad8673e5546a82ff602ac0ba59cc3f68dbde46 24 May 2017, 23:41:51 UTC
f41bffb Daniel Black 24 May 2017, 22:39:43 UTC travis: clang-3.6 -> clang-4.0 Summary: just realised when I updated the .travis.yml to trusty the llvm repo was still precise. Update this and clang-4.0. Closes https://github.com/facebook/rocksdb/pull/2127 Differential Revision: D4869427 Pulled By: sagar0 fbshipit-source-id: b7f906b6fac28e60cacc6a1f1959d6acf8269906 24 May 2017, 22:42:24 UTC
e761279 Aaron Gao 24 May 2017, 18:49:35 UTC update buckifer/TARGETS Summary: update targets file for release Closes https://github.com/facebook/rocksdb/pull/2358 Differential Revision: D5115705 Pulled By: lightmark fbshipit-source-id: 96a3c7e15b5807b5d0f5a9bb73850b92754b5794 24 May 2017, 18:56:57 UTC
bb01c18 Andrew Kryczka 24 May 2017, 18:25:38 UTC Introduce max_background_jobs mutable option Summary: - `max_background_flushes` and `max_background_compactions` are still supported for backwards compatibility - `base_background_compactions` is completely deprecated. Now we just throttle to one background compaction when there's no pressure. - `max_background_jobs` is added to automatically partition the concurrent background jobs into flushes vs compactions. Currently it's very simple as we just allocate one-fourth of the jobs to flushes, and the remaining can be used for compactions. - The test cases that set `base_background_compactions > 1` needed to be updated. I just grab the pressure token such that the desired number of compactions can be scheduled. Closes https://github.com/facebook/rocksdb/pull/2205 Differential Revision: D4937461 Pulled By: ajkr fbshipit-source-id: df52cbbd497e13bbc9a60560a5ac2a2526b3f1f9 24 May 2017, 18:29:08 UTC
5a25304 Adam Retter 24 May 2017, 18:13:25 UTC Fix the CMakeLists for RocksJava Summary: Closes https://github.com/facebook/rocksdb/pull/2356 Differential Revision: D5122240 Pulled By: sagar0 fbshipit-source-id: 3764797dd261d5b2a04d58b3967969e66c541bbb 24 May 2017, 18:29:08 UTC
41cbb72 Siying Dong 24 May 2017, 16:52:08 UTC options.delayed_write_rate use the rate of rate_limiter by default. Summary: It's hard for RocksDB to come up with a good default of delayed write rate. Use rate given by rate limiter if it is availalbe. This provides the I/O order of magnitude. Closes https://github.com/facebook/rocksdb/pull/2357 Differential Revision: D5115324 Pulled By: siying fbshipit-source-id: 341065ad2211c981fc804011c0f0e59a50c7e754 24 May 2017, 16:58:24 UTC
5068034 Siying Dong 24 May 2017, 16:43:32 UTC range sync should be enabled Summary: We forgot to add the new flag in internal build script. Add it. Closes https://github.com/facebook/rocksdb/pull/2360 Differential Revision: D5121428 Pulled By: siying fbshipit-source-id: af72d48cd855b37df1ce3c1fbb00c80377ba6e4f 24 May 2017, 16:58:24 UTC
02594b5 Sagar Vemuri 24 May 2017, 07:03:14 UTC Fix build errors in blob_dump_tool with GCC 4.8 Summary: Fixing the build errors seen with GCC 4.8.1. ``` Makefile:105: Warning: Compiling in debug mode. Don't use the resulting binary in production utilities/blob_db/blob_dump_tool.cc: In member function ‘rocksdb::Status rocksdb::blob_db::BlobDumpTool::DumpBlobLogFooter(uint64_t, uint64_t*)’: utilities/blob_db/blob_dump_tool.cc:149:42: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " Blob count : %" PRIu64 "\n", footer.GetBlobCount()); ^ utilities/blob_db/blob_dump_tool.cc:149:76: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " Blob count : %" PRIu64 "\n", footer.GetBlobCount()); ^ utilities/blob_db/blob_dump_tool.cc:149:76: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc: In member function ‘rocksdb::Status rocksdb::blob_db::BlobDumpTool::DumpRecord(rocksdb::blob_db::BlobDumpTool::DisplayType, rocksdb::blob_db::BlobDumpTool::DisplayType, uint64_t*)’: utilities/blob_db/blob_dump_tool.cc:161:49: error: expected ‘)’ before ‘PRIx64’ fprintf(stdout, "Read record with offset 0x%" PRIx64 " (%" PRIu64 "):\n", ^ utilities/blob_db/blob_dump_tool.cc:162:27: error: spurious trailing ‘%’ in format [-Werror=format=] *offset, *offset); ^ utilities/blob_db/blob_dump_tool.cc:162:27: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc:176:38: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " blob size : %" PRIu64 "\n", record.GetBlobSize()); ^ utilities/blob_db/blob_dump_tool.cc:176:71: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " blob size : %" PRIu64 "\n", record.GetBlobSize()); ^ utilities/blob_db/blob_dump_tool.cc:176:71: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc:178:38: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " time : %" PRIu64 "\n", record.GetTimeVal()); ^ utilities/blob_db/blob_dump_tool.cc:178:70: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " time : %" PRIu64 "\n", record.GetTimeVal()); ^ utilities/blob_db/blob_dump_tool.cc:178:70: error: too many arguments for format [-Werror=format-extra-args] utilities/blob_db/blob_dump_tool.cc:214:38: error: expected ‘)’ before ‘PRIu64’ fprintf(stdout, " sequence : %" PRIu64 "\n", record.GetSN()); ^ utilities/blob_db/blob_dump_tool.cc:214:65: error: spurious trailing ‘%’ in format [-Werror=format=] fprintf(stdout, " sequence : %" PRIu64 "\n", record.GetSN()); ``` Closes https://github.com/facebook/rocksdb/pull/2359 Differential Revision: D5117684 Pulled By: sagar0 fbshipit-source-id: 7480346bcd96205fcae890927c5e68cf004e87be 24 May 2017, 07:11:36 UTC
52d9e5f Igor Canadi 24 May 2017, 00:07:37 UTC Fix column family seconds_up accounting Summary: `cf_stats_snapshot_.seconds_up` appears to be never updated, unlike `db_stats_snapshot_.seconds_up`, which is updated here: https://github.com/facebook/rocksdb/blob/master/db/internal_stats.cc#L883 This leads to wrong information in the log, for example: ``` ** Compaction Stats [default] ** .... Uptime(secs): 85591.2 total, 85591.2 interval ``` Even though DB's interval is correctly logged as 60 seconds: ``` ** DB Stats ** Uptime(secs): 85591.2 total, 637.8 interval ``` Closes https://github.com/facebook/rocksdb/pull/2338 Differential Revision: D5114131 Pulled By: sagar0 fbshipit-source-id: 85243a38213236ccbb601a7f7aaa8865eaa8083c 24 May 2017, 00:14:04 UTC
7d8207f Sagar Vemuri 23 May 2017, 21:57:32 UTC Fix errors in clang-analyzer builds Summary: Fix build error in db_iter.cc when running clang-analyzer. ``` CC db/db_iter.o db/db_iter.cc:938:21: error: no matching constructor for initialization of 'rocksdb::ParsedInternalKey' ParsedInternalKey ikey(Slice(), 0, 0); ^ ~~~~~~~~~~~~~ ./db/dbformat.h:84:3: note: candidate constructor not viable: no known conversion from 'int' to 'rocksdb::ValueType' for 3rd argument ParsedInternalKey(const Slice& u, const SequenceNumber& seq, ValueType t) ^ ./db/dbformat.h:78:8: note: candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 3 were provided struct ParsedInternalKey { ^ ./db/dbformat.h:78:8: note: candidate constructor (the implicit move constructor) not viable: requires 1 argument, but 3 were provided ./db/dbformat.h:83:3: note: candidate constructor not viable: requires 0 arguments, but 3 were provided ParsedInternalKey() { } // Intentionally left uninitialized (for speed) ^ 1 error generated. ``` Closes https://github.com/facebook/rocksdb/pull/2354 Differential Revision: D5115751 Pulled By: sagar0 fbshipit-source-id: b0e386d4e935e4725b07761c3ca5f7a8cbde3692 23 May 2017, 22:11:42 UTC
85b8569 Sagar Vemuri 23 May 2017, 21:52:13 UTC Fix release build on Linux Summary: Release builds are failing on Linux with the error: ``` tools/db_stress.cc: In function ‘int main(int, char**)’: tools/db_stress.cc:2365:12: error: ‘rocksdb::SyncPoint’ has not been declared rocksdb::SyncPoint::GetInstance()->SetCallBack( ^ tools/db_stress.cc:2370:12: error: ‘rocksdb::SyncPoint’ has not been declared rocksdb::SyncPoint::GetInstance()->SetCallBack( ^ tools/db_stress.cc:2375:12: error: ‘rocksdb::SyncPoint’ has not been declared rocksdb::SyncPoint::GetInstance()->EnableProcessing(); ^ make[1]: *** [tools/db_stress.o] Error 1 make[1]: Leaving directory `/data/sandcastle/boxes/trunk-git-rocksdb-public' make: *** [release] Error 2 ``` Closes https://github.com/facebook/rocksdb/pull/2355 Differential Revision: D5113552 Pulled By: sagar0 fbshipit-source-id: 351df707277787da5633ba4a40e52edc7c895dc4 23 May 2017, 21:57:05 UTC
69ec835 Giuseppe Ottaviano 23 May 2017, 18:36:06 UTC Allow SstFileWriter to use the rate limiter Summary: The default IO priority of WritableFiles is IO_TOTAL, meaning that they will bypass the rate limiter if it's passed in the options. This change allows to pass an io priority in construction, so that by setting IO_LOW or IO_HIGH the rate limit will be honored. It also fixes a minor bug: SstFileWriter's copy and move constructor are not disabled and incorrect, as any copy/move will result in a double free. Switching to unique_ptr makes the object correctly movable and non-copyable as expected. Also fix minor style inconsistencies. Closes https://github.com/facebook/rocksdb/pull/2335 Differential Revision: D5113260 Pulled By: sagar0 fbshipit-source-id: e084236e7ff0b50a56cbeceaa9fedd5e210bf9f8 23 May 2017, 18:42:09 UTC
6cc9aef Andrew Kryczka 23 May 2017, 18:04:25 UTC New API for background work in single thread pool Summary: Previously users could set `max_background_flushes=0` to force rocksdb to use a single thread pool for both background flushes and compactions. That'll no longer be possible since I'm going to deprecate `max_background_flushes` and `max_background_compactions` in favor of a single option. This diff introduces a new way to force a single thread pool: when high-pri pool has zero threads, all background jobs will be submitted to low-pri pool. Note the majority of the code change is adding `Env::GetBackgroundThreads()`, which is necessary to check whether the user has provided a zero-sized thread pool. Closes https://github.com/facebook/rocksdb/pull/2204 Differential Revision: D4936256 Pulled By: ajkr fbshipit-source-id: 929a07a0c0705f7766f5339cd013ff74e90d6e01 23 May 2017, 18:12:27 UTC
back to top