| 49abdea | Manuel Ung | 28 November 2016, 22:12:03 UTC | review comments | 05 December 2016, 23:24:23 UTC |
| e63350e | Manuel Ung | 19 November 2016, 19:34:26 UTC | Use more efficient hash map for deadlock detection Summary: Currently, deadlock cycles are held in std::unordered_map. The problem with it is that it allocates/deallocates memory on every insertion/deletion. This limits throughput since we're doing this expensive operation while holding a global mutex. Fix this by using a vector which caches memory instead. Running the deadlock stress test, this change increased throughput from 39k txns/s -> 49k txns/s. The effect is more noticeable in MyRocks. Closes https://github.com/facebook/rocksdb/pull/1545 Differential Revision: D4205662 Pulled By: lth fbshipit-source-id: ff990e4 | 19 November 2016, 19:39:15 UTC |
| a13bde3 | Siying Dong | 19 November 2016, 03:20:15 UTC | Skip ldb test in Travis Summary: Travis now is building for ldb tests. Disable for now to unblock other tests while we are investigating. Closes https://github.com/facebook/rocksdb/pull/1546 Differential Revision: D4209404 Pulled By: siying fbshipit-source-id: 47edd97 | 19 November 2016, 03:24:13 UTC |
| 73843aa | Siying Dong | 19 November 2016, 03:17:25 UTC | Direct I/O Reads Handle the last sector correctly. Summary: Currently, in the Direct I/O read mode, the last sector of the file, if not full, is not handled correctly. If the return value of pread is not multiplier of kSectorSize, we still go ahead and continue reading, even if the buffer is not aligned. With the commit, if the return value is not multiplier of kSectorSize, and all but the last sector has been read, we simply return. Closes https://github.com/facebook/rocksdb/pull/1550 Differential Revision: D4209609 Pulled By: lightmark fbshipit-source-id: cb0b439 | 19 November 2016, 03:24:13 UTC |
| 9d60151 | Maysam Yabandeh | 19 November 2016, 01:06:37 UTC | Implement PositionedAppend for PosixWritableFile Summary: This patch clarifies the contract of PositionedAppend with some unit tests and also implements it for PosixWritableFile. (Tasks: 14524071) Closes https://github.com/facebook/rocksdb/pull/1514 Differential Revision: D4204907 Pulled By: maysamyabandeh fbshipit-source-id: 06eabd2 | 19 November 2016, 01:24:13 UTC |
| 3f62215 | Andrew Kryczka | 19 November 2016, 00:54:09 UTC | Lazily initialize RangeDelAggregator's map and pinning manager Summary: Since a RangeDelAggregator is created for each read request, these heap-allocating member variables were consuming significant CPU (~3% total) which slowed down request throughput. The map and pinning manager are only necessary when range deletions exist, so we can defer their initialization until the first range deletion is encountered. Currently lazy initialization is done for reads only since reads pass us a single snapshot, which is easier to store on the stack for later insertion into the map than the vector passed to us by flush or compaction. Note the Arena member variable is still expensive, I will figure out what to do with it in a subsequent diff. It cannot be lazily initialized because we currently use this arena even to allocate empty iterators, which is necessary even when no range deletions exist. Closes https://github.com/facebook/rocksdb/pull/1539 Differential Revision: D4203488 Pulled By: ajkr fbshipit-source-id: 3b36279 | 19 November 2016, 01:09:11 UTC |
| 41e77b8 | Kefu Chai | 18 November 2016, 22:41:30 UTC | cmake: s/STEQUAL/STREQUAL/ Summary: Signed-off-by: Kefu Chai <tchaikov@gmail.com> Closes https://github.com/facebook/rocksdb/pull/1540 Differential Revision: D4207564 Pulled By: siying fbshipit-source-id: 567415b | 18 November 2016, 22:54:14 UTC |
| c1038d2 | Islam AbdelRahman | 18 November 2016, 02:31:55 UTC | Release RocksDB 5.0 Summary: Update HISTORY.md and version.h Closes https://github.com/facebook/rocksdb/pull/1536 Differential Revision: D4202987 Pulled By: IslamAbdelRahman fbshipit-source-id: 94985e3 | 18 November 2016, 02:39:15 UTC |
| 635a7bd | Andrew Kryczka | 17 November 2016, 22:30:11 UTC | refactor TableCache Get/NewIterator for single exit points Summary: these functions were too complicated to change with exit points everywhere, so refactored them. btw, please review urgently, this is a prereq to fix the 5.0 perf regression Closes https://github.com/facebook/rocksdb/pull/1534 Differential Revision: D4198972 Pulled By: ajkr fbshipit-source-id: 04ebfb7 | 17 November 2016, 22:39:13 UTC |
| f39452e | Islam AbdelRahman | 17 November 2016, 20:02:40 UTC | Fix heap use after free ASAN/Valgrind Summary: Dont use c_str() of temp std::string in RocksLuaCompactionFilter::Name() Closes https://github.com/facebook/rocksdb/pull/1535 Differential Revision: D4199094 Pulled By: IslamAbdelRahman fbshipit-source-id: e56ce62 | 17 November 2016, 20:24:12 UTC |
| a4eb738 | Siying Dong | 17 November 2016, 18:29:35 UTC | Allow plain table to store index on file with bloom filter disabled Summary: Currently plain table bloom filter is required if storing metadata on file. Remove the constraint. Closes https://github.com/facebook/rocksdb/pull/1525 Differential Revision: D4190977 Pulled By: siying fbshipit-source-id: be60442 | 17 November 2016, 19:09:13 UTC |
| 36e4762 | Yi Wu | 17 November 2016, 06:34:18 UTC | Remove Ticker::SEQUENCE_NUMBER Summary: Remove the ticker count because: * Having to reset the ticker count in WriteImpl is ineffiecent; * It doesn't make sense to have it as a ticker count if multiple db instance share a statistics object. Closes https://github.com/facebook/rocksdb/pull/1531 Differential Revision: D4194442 Pulled By: yiwu-arbug fbshipit-source-id: e2110a9 | 17 November 2016, 06:39:09 UTC |
| 86eb2b9 | Islam AbdelRahman | 17 November 2016, 02:05:19 UTC | Fix src.mk | 17 November 2016, 02:05:19 UTC |
| 0765bab | Andrew Kryczka | 17 November 2016, 01:04:49 UTC | Remove LATEST_BACKUP file Summary: This has been unused since D42069 but kept around for backward compatibility. I think it is unlikely anyone will use a much older version of RocksDB for restore than they use for backup, so I propose removing it. It is also causing recurring confusion, e.g., https://www.facebook.com/groups/rocksdb.dev/permalink/980454015386446/ Ported from https://reviews.facebook.net/D60735 Closes https://github.com/facebook/rocksdb/pull/1529 Differential Revision: D4194199 Pulled By: ajkr fbshipit-source-id: 82f9bf4 | 17 November 2016, 01:24:15 UTC |
| 647eafd | Yueh-Hsuan Chiang | 16 November 2016, 23:27:02 UTC | Introduce Lua Extension: RocksLuaCompactionFilter Summary: This diff includes an implementation of CompactionFilter that allows users to write CompactionFilter in Lua. With this ability, users can dynamically change compaction filter logic without requiring building the rocksdb binary and restarting the database. To compile, WITH_LUA_PATH must be specified to the base directory of lua. Closes https://github.com/facebook/rocksdb/pull/1478 Differential Revision: D4150138 Pulled By: yhchiang fbshipit-source-id: ed84222 | 16 November 2016, 23:39:12 UTC |
| 760ef68 | Andrew Kryczka | 16 November 2016, 21:52:00 UTC | fix deleterange asan issue Summary: pinned_iters_mgr_ pins iterators allocated with arena_, so we should order the instance variable declarations such that the pinned iterators have their destructors executed before the arena is destroyed. Closes https://github.com/facebook/rocksdb/pull/1528 Differential Revision: D4191984 Pulled By: ajkr fbshipit-source-id: 1386f20 | 16 November 2016, 22:09:07 UTC |
| 327085b | Andrew Kryczka | 16 November 2016, 20:02:39 UTC | fix valgrind Summary: Closes https://github.com/facebook/rocksdb/pull/1526 Differential Revision: D4191257 Pulled By: ajkr fbshipit-source-id: d09dc76 | 16 November 2016, 20:09:11 UTC |
| 715591b | Adam Retter | 16 November 2016, 18:36:49 UTC | Ask travis to use JDK 7 Summary: yhchiang This may or may not help Closes https://github.com/facebook/rocksdb/pull/1385 Differential Revision: D4098424 Pulled By: yhchiang fbshipit-source-id: 9f9782e | 16 November 2016, 18:54:12 UTC |
| 972e3ff | Siying Dong | 16 November 2016, 17:24:52 UTC | Enable allow_concurrent_memtable_write and enable_write_thread_adaptive_yield by default Summary: Closes https://github.com/facebook/rocksdb/pull/1496 Differential Revision: D4168080 Pulled By: siying fbshipit-source-id: 056ae62 | 16 November 2016, 17:39:09 UTC |
| 420bdb4 | Siying Dong | 16 November 2016, 05:55:15 UTC | option_change_migration_test: force full compaction when needed Summary: When option_change_migration_test decides to go with a full compaction, we don't force a compaction but allow trivial move. This can cause assert failure if the destination is level 0. Fix it by forcing the full compaction to skip trivial move if the destination level is L0. Closes https://github.com/facebook/rocksdb/pull/1518 Differential Revision: D4183610 Pulled By: siying fbshipit-source-id: dea482b | 16 November 2016, 06:09:34 UTC |
| 1543d5d | Yi Wu | 16 November 2016, 04:05:36 UTC | Report memory usage by memtable insert hints map. Summary: It is hard to measure acutal memory usage by std containers. Even providing a custom allocator will miss count some of the usage. Here we only do a wild guess on its memory usage. Closes https://github.com/facebook/rocksdb/pull/1511 Differential Revision: D4179945 Pulled By: yiwu-arbug fbshipit-source-id: 32ab929 | 16 November 2016, 04:24:13 UTC |
| 018bb2e | Andrew Kryczka | 16 November 2016, 01:33:27 UTC | DeleteRange support for db_bench Summary: Added a few options to configure when to add range tombstones during any benchmark involving writes. Closes https://github.com/facebook/rocksdb/pull/1522 Differential Revision: D4187388 Pulled By: ajkr fbshipit-source-id: 2c8a473 | 16 November 2016, 01:39:47 UTC |
| dc51bd7 | Willem Jan Withagen | 16 November 2016, 01:27:48 UTC | CMakeLists.txt: FreeBSD has jemalloc as default malloc Summary: This will allow reference to `malloc_stats_print` Closes https://github.com/facebook/rocksdb/pull/1516 Differential Revision: D4187258 Pulled By: siying fbshipit-source-id: 34ae9f9 | 16 November 2016, 01:39:47 UTC |
| 48e8bae | Andrew Kryczka | 16 November 2016, 01:18:32 UTC | Decouple data iterator and range deletion iterator in TableCache Summary: Previously we used TableCache::NewIterator() for multiple purposes (data block iterator and range deletion iterator), and returned non-ok status in the data block iterator. In one case where the caller only used the range deletion block iterator (https://github.com/facebook/rocksdb/blob/9e7cf3469bc626b092ec48366d12873ecab22b4e/db/version_set.cc#L965-L973), we didn't check/free the data block iterator containing non-ok status, which caused a valgrind error. So, this diff decouples creation of data block and range deletion block iterators, and updates the callers accordingly. Both functions can return non-ok status in an InternalIterator. Since the non-ok status is returned in an iterator that the callers will definitely use, it should be more usable/less error-prone. Closes https://github.com/facebook/rocksdb/pull/1513 Differential Revision: D4181423 Pulled By: ajkr fbshipit-source-id: 835b8f5 | 16 November 2016, 01:24:28 UTC |
| 4b0aa3c | daoye.ch | 16 November 2016, 00:54:36 UTC | Fix failed compaction_filter_example and add it into make all Summary: Simple patch as title Closes https://github.com/facebook/rocksdb/pull/1512 Differential Revision: D4186994 Pulled By: siying fbshipit-source-id: 880f9b8 | 16 November 2016, 01:09:10 UTC |
| 53b693f | Andrew Kryczka | 15 November 2016, 23:49:15 UTC | ldb support for range delete Summary: Add a subcommand to ldb with which we can delete a range of keys. Closes https://github.com/facebook/rocksdb/pull/1521 Differential Revision: D4186338 Pulled By: ajkr fbshipit-source-id: b8e9861 | 15 November 2016, 23:54:20 UTC |
| 661e4c9 | Andrew Kryczka | 15 November 2016, 23:18:56 UTC | DeleteRange unsupported in non-block-based tables Summary: Return an error from DeleteRange() (or Write() if the user is using the low-level WriteBatch API) if an unsupported table type is configured. Closes https://github.com/facebook/rocksdb/pull/1519 Differential Revision: D4185933 Pulled By: ajkr fbshipit-source-id: abcdf84 | 15 November 2016, 23:24:16 UTC |
| 489d142 | Andrew Kryczka | 15 November 2016, 23:06:03 UTC | DeleteRange interface Summary: Expose DeleteRange() interface since we think the implementation is functionally correct now. Closes https://github.com/facebook/rocksdb/pull/1503 Differential Revision: D4171921 Pulled By: ajkr fbshipit-source-id: 5e21c98 | 15 November 2016, 23:24:16 UTC |
| eba99c2 | Islam AbdelRahman | 15 November 2016, 21:38:24 UTC | Fix min_write_buffer_number_to_merge = 0 bug Summary: It's possible that we set min_write_buffer_number_to_merge to 0. This should never happen Closes https://github.com/facebook/rocksdb/pull/1515 Differential Revision: D4183356 Pulled By: yiwu-arbug fbshipit-source-id: c9d39d7 | 15 November 2016, 21:54:08 UTC |
| 2ef92fe | Joel Marcey | 15 November 2016, 15:39:43 UTC | Remove all instances of relative_url until GitHub pages problem is fixed. I am in email thread with GitHub support about what is happening here. | 15 November 2016, 15:40:18 UTC |
| 91300d0 | Artemiy Kolesnikov | 15 November 2016, 06:45:16 UTC | Dynamic max_total_wal_size option Summary: Closes https://github.com/facebook/rocksdb/pull/1509 Differential Revision: D4176426 Pulled By: yiwu-arbug fbshipit-source-id: b57689d | 15 November 2016, 06:54:17 UTC |
| ec2f647 | Andrew Kryczka | 15 November 2016, 04:20:14 UTC | Consider subcompaction boundaries when updating file boundaries for range deletion Summary: Adjusted AddToBuilder() to take lower_bound and upper_bound, which serve two purposes: (1) only range deletions overlapping with the interval [lower_bound, upper_bound) will be added to the output file, and (2) the output file's boundaries will not be extended before lower_bound or after upper_bound. Our computation of lower_bound/upper_bound consider both subcompaction boundaries and previous/next files within the subcompaction. Test cases are here (level subcompactions: https://gist.github.com/ajkr/63c7eae3e9667c5ebdc0a7efb74ac332, and universal subcompactions: https://gist.github.com/ajkr/5a62af77c4ebe4052a1955c496d51fdb) but can't be included in this diff as they depend on committing the API first. They fail before this change and pass after. Closes https://github.com/facebook/rocksdb/pull/1501 Reviewed By: yhchiang Differential Revision: D4171685 Pulled By: ajkr fbshipit-source-id: ee99db8 | 15 November 2016, 04:24:21 UTC |
| 800e515 | Joel Marcey | 15 November 2016, 04:11:26 UTC | Fix CSS issues again :( I have an email to GitHub support about this. | 15 November 2016, 04:11:26 UTC |
| b952c89 | Yi Wu | 15 November 2016, 03:54:17 UTC | Parallize persistent_cache_test and transaction_test Summary: Parallize persistent_cache_test and transaction_test Closes https://github.com/facebook/rocksdb/pull/1506 Differential Revision: D4179392 Pulled By: IslamAbdelRahman fbshipit-source-id: 05507a1 | 15 November 2016, 04:09:19 UTC |
| 3b192f6 | Andrew Kryczka | 15 November 2016, 01:35:46 UTC | Handle full final subcompaction output file with range deletions Summary: This conditional should only open a new file that's dedicated to range deletions when it's the sole output of the subcompaction. Previously, we created such a file whenever the table builder was nullptr, which would've also been the case whenever the CompactionIterator's final key coincided with the final output table becoming full. Closes https://github.com/facebook/rocksdb/pull/1507 Differential Revision: D4174613 Pulled By: ajkr fbshipit-source-id: 9ffacea | 15 November 2016, 01:54:20 UTC |
| 6c57952 | Andrew Kryczka | 15 November 2016, 01:35:17 UTC | Make range deletion inclusive-exclusive Summary: This makes it easier to implement future optimizations like range collapsing. Closes https://github.com/facebook/rocksdb/pull/1504 Differential Revision: D4172214 Pulled By: ajkr fbshipit-source-id: ac4942f | 15 November 2016, 01:39:13 UTC |
| 425210c | Joel Marcey | 14 November 2016, 15:07:36 UTC | CSS issues are arising on the Github Pages side. Temp fix. Need to figure out why this is still happening that `relative_url` is not prepending the right value at just random times. | 14 November 2016, 15:08:52 UTC |
| 1ea79a7 | Yi Wu | 14 November 2016, 02:58:17 UTC | Optimize sequential insert into memtable - Part 1: Interface Summary: Currently our skip-list have an optimization to speedup sequential inserts from a single stream, by remembering the last insert position. We extend the idea to support sequential inserts from multiple streams, and even tolerate small reordering wihtin each stream. This PR is the interface part adding the following: - Add `memtable_insert_prefix_extractor` to allow specifying prefix for each key. - Add `InsertWithHint()` interface to memtable, to allow underlying implementation to return a hint of insert position, which can be later pass back to optimize inserts. - Memtable will maintain a map from prefix to hints and pass the hint via `InsertWithHint()` if `memtable_insert_prefix_extractor` is non-null. Closes https://github.com/facebook/rocksdb/pull/1419 Differential Revision: D4079367 Pulled By: yiwu-arbug fbshipit-source-id: 3555326 | 14 November 2016, 03:09:18 UTC |
| df5eeb8 | Yi Wu | 13 November 2016, 21:00:52 UTC | Optimize sequential insert into memtable - Part 2: Implementation Summary: Implement a insert hint into skip-list to hint insert position. This is to optimize for the write workload where there are multiple stream of sequential writes. For example, there is a stream of keys of a1, a2, a3... but also b1, b2, b2... Each stream are not neccessary strictly sequential, but can get reorder a little bit. User can specify a prefix extractor and the `SkipListRep` can thus maintan a hint for each of the stream for fast insert into memtable. This is the internal implementation part. See #1419 for the interface part. See inline comments for details. Closes https://github.com/facebook/rocksdb/pull/1449 Differential Revision: D4106781 Pulled By: yiwu-arbug fbshipit-source-id: f4d48c4 | 13 November 2016, 21:09:16 UTC |
| 5ed6508 | Islam AbdelRahman | 13 November 2016, 04:03:39 UTC | Fix SstFileWriter destructor Summary: If user did not call SstFileWriter::Finish() or called Finish() but it failed. We need to abandon the builder, to avoid destructing it while it's open Closes https://github.com/facebook/rocksdb/pull/1502 Differential Revision: D4171660 Pulled By: IslamAbdelRahman fbshipit-source-id: ab6f434 | 13 November 2016, 04:11:19 UTC |
| adb665e | Lijun Tang | 12 November 2016, 23:43:33 UTC | Allowed delayed_write_rate option to be dynamically set. Summary: Closes https://github.com/facebook/rocksdb/pull/1488 Differential Revision: D4157784 Pulled By: siying fbshipit-source-id: f150081 | 12 November 2016, 23:54:11 UTC |
| 307a4e8 | Andrew Kryczka | 12 November 2016, 17:23:05 UTC | sst_dump support for range deletion Summary: Change DumpTable() so we can see the range deletion meta-block. Closes https://github.com/facebook/rocksdb/pull/1505 Differential Revision: D4172227 Pulled By: ajkr fbshipit-source-id: ae35665 | 12 November 2016, 17:39:23 UTC |
| 361010d | Maysam Yabandeh | 12 November 2016, 04:45:47 UTC | Exporting compaction stats in the form of a map Summary: Currently the compaction stats are printed to stdout. We want to export the compaction stats in a map format so that the upper layer apps (e.g., MySQL) could present the stats in any format required by the them. Closes https://github.com/facebook/rocksdb/pull/1477 Differential Revision: D4149836 Pulled By: maysamyabandeh fbshipit-source-id: b3df19f | 12 November 2016, 04:54:14 UTC |
| 672300f | Joel Marcey | 10 November 2016, 22:54:55 UTC | Use relative Urls for stylesheets | 10 November 2016, 22:54:55 UTC |
| b39b2ee | Aaron Gao | 10 November 2016, 19:10:23 UTC | do not call get() in recovery mode Summary: This is a previous fix that has a typo Closes https://github.com/facebook/rocksdb/pull/1487 Differential Revision: D4157381 Pulled By: lightmark fbshipit-source-id: f079be8 | 10 November 2016, 19:24:20 UTC |
| 1ca5f6d | Reid Horuff | 10 November 2016, 18:59:06 UTC | Fix 2PC Recovery SeqId Miscount Summary: Originally sequence ids were calculated, in recovery, based off of the first seqid found if the first log recovered. The working seqid was then incremented from that value based on every insertion that took place. This was faulty because of the potential for missing log files or inserts that skipped the WAL. The current recovery scheme grabs sequence from current recovering batch and increments using memtableinserter to track how many actual inserts take place. This works for 2PC batches as well scenarios where some logs are missing or inserts that skip the WAL. Closes https://github.com/facebook/rocksdb/pull/1486 Differential Revision: D4156064 Pulled By: reidHoruff fbshipit-source-id: a6da8d9 | 10 November 2016, 19:09:22 UTC |
| e095d0c | Sergey Balabanov | 01 November 2016, 23:32:27 UTC | Rocksdb contruns to new Sandcastle API Reviewed By: IslamAbdelRahman Differential Revision: D4114816 fbshipit-source-id: 8082936 | 10 November 2016, 18:54:20 UTC |
| 14c0380 | Anirban Rahut | 10 November 2016, 18:06:06 UTC | Convenience option to parse an internal key on command line Summary: enhancing sst_dump to be able to parse internal key Closes https://github.com/facebook/rocksdb/pull/1482 Differential Revision: D4154175 Pulled By: siying fbshipit-source-id: b0e28b1 | 10 November 2016, 18:09:21 UTC |
| c90fef8 | Andrew Kryczka | 10 November 2016, 06:10:06 UTC | fix open failure with empty wal Summary: Closes https://github.com/facebook/rocksdb/pull/1490 Differential Revision: D4158821 Pulled By: IslamAbdelRahman fbshipit-source-id: 59b73f4 | 10 November 2016, 06:24:26 UTC |
| 4e20c5d | Andrew Kryczka | 09 November 2016, 23:00:16 UTC | Store internal keys in TombstoneMap Summary: This fixes a correctness issue where ranges with same begin key would overwrite each other. This diff uses InternalKey as TombstoneMap's key such that all tombstones have unique keys even when their start keys overlap. We also update TombstoneMap to use an internal key comparator. End-to-end tests pass and are here (https://gist.github.com/ajkr/851ffe4c1b8a15a68d33025be190a7d9) but cannot be included yet since the DeleteRange() API is yet to be checked in. Note both tests failed before this fix. Closes https://github.com/facebook/rocksdb/pull/1484 Differential Revision: D4155248 Pulled By: ajkr fbshipit-source-id: 304b4b9 | 09 November 2016, 23:09:18 UTC |
| a9fb346 | Yueh-Hsuan Chiang | 09 November 2016, 20:20:12 UTC | Fix RocksDB Lite build failure in c_test.cc Summary: Fix the following RocksDB Lite build failure in c_test.cc db/c_test.c:1051:3: error: implicit declaration of function 'fprintf' is invalid in C99 [-Werror,-Wimplicit-function-declaration] fprintf(stderr, "SKIPPED\n"); ^ db/c_test.c:1051:3: error: declaration of built-in function 'fprintf' requires inclusion of the header <stdio.h> [-Werror,-Wbuiltin-requires-header] db/c_test.c:1051:11: error: use of undeclared identifier 'stderr' fprintf(stderr, "SKIPPED\n"); ^ 3 errors generated. Closes https://github.com/facebook/rocksdb/pull/1479 Differential Revision: D4151160 Pulled By: yhchiang fbshipit-source-id: a471a30 | 09 November 2016, 20:24:18 UTC |
| d133b08 | Reid Horuff | 09 November 2016, 20:12:40 UTC | Use correct sequence number when creating memtable Summary: copied from: https://github.com/mdlugajczyk/rocksdb/commit/5ebfd2623a01e69a4cbeae3ed2b788f2a84056ad Opening existing RocksDB attempts recovery from log files, which uses wrong sequence number to create the memtable. This is a regression introduced in change a400336. This change includes a test demonstrating the problem, without the fix the test fails with "Operation failed. Try again.: Transaction could not check for conflicts for operation at SequenceNumber 1 as the MemTable only contains changes newer than SequenceNumber 2. Increasing the value of the max_write_buffer_number_to_maintain option could reduce the frequency of this error" This change is a joint effort by Peter 'Stig' Edwards thatsafunnyname and me. Closes https://github.com/facebook/rocksdb/pull/1458 Differential Revision: D4143791 Pulled By: reidHoruff fbshipit-source-id: 5a25033 | 09 November 2016, 20:24:17 UTC |
| 144cdb8 | Peter (Stig) Edwards | 09 November 2016, 19:15:22 UTC | 16384 as e.g .value for compression_max_dict_bytes Summary: Use 16384 as e.g .value for ldb the --compression_max_dict_bytes option. I think 14 was copy and pasted from the options in the lines above. Closes https://github.com/facebook/rocksdb/pull/1483 Differential Revision: D4154393 Pulled By: siying fbshipit-source-id: ef53a69 | 09 November 2016, 19:24:20 UTC |
| 9bd191d | Islam AbdelRahman | 09 November 2016, 18:37:51 UTC | Fix deadlock between (WriterThread/Compaction/IngestExternalFile) Summary: A deadlock is possible if this happen (1) Writer thread is stopped because it's waiting for compaction to finish (2) Compaction is waiting for current IngestExternalFile() calls to finish (3) IngestExternalFile() is waiting to be able to acquire the writer thread (4) WriterThread is held by stopped writes that are waiting for compactions to finish This patch fix the issue by not incrementing num_running_ingest_file_ except when we acquire the writer thread. This patch include a unittest to reproduce the described scenario Closes https://github.com/facebook/rocksdb/pull/1480 Differential Revision: D4151646 Pulled By: IslamAbdelRahman fbshipit-source-id: 09b39db | 09 November 2016, 18:54:10 UTC |
| a9fae0a | Joel Marcey | 08 November 2016, 23:21:37 UTC | CSS problems again :( Trying to remove baseurl term. | 08 November 2016, 23:22:31 UTC |
| 193221e | Islam AbdelRahman | 08 November 2016, 21:44:38 UTC | Fix Forward Iterator Seek()/SeekToFirst() Summary: In ForwardIterator::SeekInternal(), we may end up passing empty Slice representing an internal key to InternalKeyComparator::Compare. and when we try to extract the user key from this empty Slice, we will create a slice with size = 0 - 8 ( which will overflow and cause us to read invalid memory as well ) Scenarios to reproduce these issues are in the unit tests Closes https://github.com/facebook/rocksdb/pull/1467 Differential Revision: D4136660 Pulled By: lightmark fbshipit-source-id: 151e128 | 08 November 2016, 21:54:31 UTC |
| e48f3f8 | Aaron Gao | 08 November 2016, 21:42:46 UTC | remove tabs and duplicate #include in c api Summary: fix lint error about tabs and duplicate includes. Closes https://github.com/facebook/rocksdb/pull/1476 Differential Revision: D4149646 Pulled By: lightmark fbshipit-source-id: 2e0a632 | 08 November 2016, 21:54:31 UTC |
| 85bd8f5 | Karthik | 08 November 2016, 20:52:17 UTC | Minor fix to GFLAGS usage in persistent cache Summary: The general convention in RocksDB is to use GFLAGS instead of google. Fixing the anomaly. Closes https://github.com/facebook/rocksdb/pull/1470 Differential Revision: D4149213 Pulled By: kradhakrishnan fbshipit-source-id: 2dafa53 | 08 November 2016, 21:09:20 UTC |
| a787527 | Jay Lee | 08 November 2016, 20:36:05 UTC | c: support seek_for_prev Summary: support seek_for_prev in c abi. Closes https://github.com/facebook/rocksdb/pull/1457 Differential Revision: D4135360 Pulled By: lightmark fbshipit-source-id: 61256b0 | 08 November 2016, 20:54:13 UTC |
| 0f17f92 | Joel Marcey | 08 November 2016, 01:03:19 UTC | Make the header links a bit more flexible | 08 November 2016, 01:03:44 UTC |
| cf19f55 | Joel Marcey | 08 November 2016, 00:30:58 UTC | single quotes in feed | 08 November 2016, 00:30:58 UTC |
| 2dc019e | Joel Marcey | 08 November 2016, 00:28:27 UTC | Fix header links | 08 November 2016, 00:28:27 UTC |
| f1aedda | Joel Marcey | 08 November 2016, 00:16:05 UTC | More Jekyll 3.3 fixes | 08 November 2016, 00:16:05 UTC |
| c54cdc3 | Joel Marcey | 08 November 2016, 00:02:48 UTC | More Jekyll 3.3 updates | 08 November 2016, 00:02:48 UTC |
| 2bcaf82 | Joel Marcey | 07 November 2016, 17:12:47 UTC | Update product and feature template for Jekyll 3.3 Key change is using the new `absolute_url` and `relative_url` filters http://jekyllrb.com/news/2016/10/06/jekyll-3-3-is-here/ https://github.com/blog/2277-what-s-new-in-github-pages-with-jekyll-3-3 | 07 November 2016, 17:26:20 UTC |
| 24bceb0 | Adam Retter | 06 November 2016, 09:25:39 UTC | Java API - Implement GetFromBatch and GetFromBatchAndDB in WBWI Summary: Needed for working with `get` after `merge` on a WBWI. Closes https://github.com/facebook/rocksdb/pull/1093 Differential Revision: D4137978 Pulled By: yhchiang fbshipit-source-id: e18d50d | 06 November 2016, 10:44:07 UTC |
| 815f54a | Andrew Kryczka | 05 November 2016, 16:10:51 UTC | Insert range deletion meta-block into block cache Summary: This handles two issues: (1) range deletion iterator sometimes outlives the table reader that created it, in which case the block must not be destroyed during table reader destruction; and (2) we prefer to read these range tombstone meta-blocks from file fewer times. - Extracted cache-populating logic from NewDataBlockIterator() into a separate function: MaybeLoadDataBlockToCache() - Use MaybeLoadDataBlockToCache() to load range deletion meta-block and pin it through the reader's lifetime. This code reuse works since range deletion meta-block has same format as data blocks. - Use NewDataBlockIterator() to create range deletion iterators, which uses block cache if enabled, otherwise reads the block from file. Either way, the underlying block won't disappear until after the iterator is destroyed. Closes https://github.com/facebook/rocksdb/pull/1459 Differential Revision: D4123175 Pulled By: ajkr fbshipit-source-id: 8f64281 | 05 November 2016, 16:24:26 UTC |
| 9e7cf34 | Andrew Kryczka | 04 November 2016, 18:53:38 UTC | DeleteRange user iterator support Summary: Note: reviewed in https://reviews.facebook.net/D65115 - DBIter maintains a range tombstone accumulator. We don't cleanup obsolete tombstones yet, so if the user seeks back and forth, the same tombstones would be added to the accumulator multiple times. - DBImpl::NewInternalIterator() (used to make DBIter's underlying iterator) adds memtable/L0 range tombstones, L1+ range tombstones are added on-demand during NewSecondaryIterator() (see D62205) - DBIter uses ShouldDelete() when advancing to check whether keys are covered by range tombstones Closes https://github.com/facebook/rocksdb/pull/1464 Differential Revision: D4131753 Pulled By: ajkr fbshipit-source-id: be86559 | 04 November 2016, 19:09:22 UTC |
| 5c5d01a | Islam AbdelRahman | 04 November 2016, 18:19:09 UTC | Fix wrong comment (Maximum supported block size) Summary: We can support SST files >2GB but we don't support blocks >2GB Closes https://github.com/facebook/rocksdb/pull/1465 Differential Revision: D4132140 Pulled By: yiwu-arbug fbshipit-source-id: 63bf12d | 04 November 2016, 18:24:14 UTC |
| f998c97 | Andrew Kryczka | 04 November 2016, 01:40:23 UTC | DeleteRange Get support Summary: During Get()/MultiGet(), build up a RangeDelAggregator with range tombstones as we search through live memtable, immutable memtables, and SST files. This aggregator is then used by memtable.cc's SaveValue() and GetContext::SaveValue() to check whether keys are covered. added tests for Get on memtables/files; end-to-end tests mainly in https://reviews.facebook.net/D64761 Closes https://github.com/facebook/rocksdb/pull/1456 Differential Revision: D4111271 Pulled By: ajkr fbshipit-source-id: 6e388d4 | 04 November 2016, 01:54:20 UTC |
| 879f366 | zhangjinpeng1987 | 03 November 2016, 18:00:46 UTC | Add C api for RateLimiter Summary: Add C api for RateLimiter. Closes https://github.com/facebook/rocksdb/pull/1455 Differential Revision: D4116362 Pulled By: yiwu-arbug fbshipit-source-id: cb05a8d | 03 November 2016, 18:09:17 UTC |
| 557034f | Joel Marcey | 03 November 2016, 17:29:33 UTC | Remove all instances of baseurl rocksdb hit the problem that nuclide had. https://github.com/facebook/nuclide/issues/789 https://github.com/facebook/nuclide/pull/793 | 03 November 2016, 17:29:33 UTC |
| 437942e | Yi Wu | 02 November 2016, 22:22:13 UTC | Add avoid_flush_during_shutdown DB option Summary: Add avoid_flush_during_shutdown DB option. Closes https://github.com/facebook/rocksdb/pull/1451 Differential Revision: D4108643 Pulled By: yiwu-arbug fbshipit-source-id: abdaf4d | 02 November 2016, 22:39:18 UTC |
| 2b16d66 | Benoit Girard | 02 November 2016, 04:05:32 UTC | Change max_bytes_for_level_multiplier to double Summary: Closes https://github.com/facebook/rocksdb/pull/1427 Differential Revision: D4094732 Pulled By: yiwu-arbug fbshipit-source-id: b9b79e9 | 02 November 2016, 04:09:23 UTC |
| 16fb044 | Jay Lee | 02 November 2016, 00:02:38 UTC | expose IngestExternalFile to c abi Summary: IngestExternalFile is very useful when doing bulk load. This pr expose this API to c so many bindings can benefit from it too. Closes https://github.com/facebook/rocksdb/pull/1454 Differential Revision: D4113420 Pulled By: yiwu-arbug fbshipit-source-id: 307c6ae | 02 November 2016, 00:09:39 UTC |
| ce22ea9 | Adam Retter | 01 November 2016, 16:55:29 UTC | Fix casts for MSVC Summary: I am not sure if this is the best way to fix this? Closes https://github.com/facebook/rocksdb/pull/1452 Differential Revision: D4109338 Pulled By: yiwu-arbug fbshipit-source-id: ca40809 | 01 November 2016, 17:09:16 UTC |
| 196af03 | Vladislav Vaintroub | 01 November 2016, 08:11:02 UTC | Introduce FAIL_ON_WARNINGS CMake variable (default ON) Summary: If FAIL_ON_WARNINGS is OFF, compilation will succeed if there were compile warnings. Closes https://github.com/facebook/rocksdb/pull/1446 Differential Revision: D4107825 Pulled By: yiwu-arbug fbshipit-source-id: 4ce6e05 | 01 November 2016, 08:24:16 UTC |
| 40a2e40 | Andrew Kryczka | 01 November 2016, 03:35:54 UTC | DeleteRange flush support Summary: Changed BuildTable() (used for flush) to (1) add range tombstones to the aggregator, which is used by CompactionIterator to determine which keys can be removed; and (2) add aggregator's range tombstones to the table that is output for the flush. Closes https://github.com/facebook/rocksdb/pull/1438 Differential Revision: D4100025 Pulled By: ajkr fbshipit-source-id: cb01a70 | 01 November 2016, 03:54:18 UTC |
| d5555d9 | Vladislav Vaintroub | 01 November 2016, 00:22:07 UTC | Fix MSVC compile error in 32 bit compilation Summary: Passing std::atomic<uint64_t> variables to ASSERT_EQ() results in compile error C2718 'const T1': actual parameter with requested alignment of 8 won't be aligned. VS2015 defines std::atomic as specially aligned type ( with 'alignas'), however the compiler does not like declspec(align)ed function arguments. Worked around by casting std::atomic<uint64_t> types to uint64_t in ASSERT_EQ. Closes https://github.com/facebook/rocksdb/pull/1450 Differential Revision: D4106788 Pulled By: yiwu-arbug fbshipit-source-id: 5fb42c3 | 01 November 2016, 00:24:18 UTC |
| da61f34 | Siying Dong | 31 October 2016, 23:03:24 UTC | Print compression and Fast CRC support info as Header level Summary: Currently the compression suppport and fast CRC support information is printed as info level. They should be in the same level as options, which is header level. Also add ZSTD to this printing. Closes https://github.com/facebook/rocksdb/pull/1448 Differential Revision: D4106608 Pulled By: yiwu-arbug fbshipit-source-id: cb9a076 | 31 October 2016, 23:09:13 UTC |
| f9eb567 | Siying Dong | 31 October 2016, 21:50:40 UTC | db_bench: --dump_malloc_stats takes no effect Summary: Fix the bug that --dump_malloc_stats is set before opening the DB. Closes https://github.com/facebook/rocksdb/pull/1447 Differential Revision: D4106001 Pulled By: siying fbshipit-source-id: 4e746da | 31 October 2016, 21:54:26 UTC |
| 6a4faee | ananclub | 31 October 2016, 16:39:00 UTC | fix freebsd build include path err and so & jar file name Summary: Closes https://github.com/facebook/rocksdb/pull/1441 Differential Revision: D4103477 Pulled By: yiwu-arbug fbshipit-source-id: 071a0dc | 31 October 2016, 16:39:16 UTC |
| c90c48d | Siying Dong | 29 October 2016, 23:01:30 UTC | Show More DB Stats in info logs Summary: DB Stats now are truncated if there are too many CFs. Extend the buffer size to allow more to be printed out. Also, separate out malloc to another log line. Closes https://github.com/facebook/rocksdb/pull/1439 Differential Revision: D4100943 Pulled By: yiwu-arbug fbshipit-source-id: 79f7218 | 29 October 2016, 23:09:18 UTC |
| 1b295ac | Siying Dong | 29 October 2016, 22:58:45 UTC | DBTest.GetThreadStatus: Wait for test results for longer Summary: The current 10 millisecond waiting for test results may not be sufficient in some test environments. Increase it to 60 seconds and check the results for every 1 milliseond. Already reviewed: https://reviews.facebook.net/D65457 Closes https://github.com/facebook/rocksdb/pull/1437 Differential Revision: D4099443 Pulled By: siying fbshipit-source-id: cf1f205 | 29 October 2016, 23:09:18 UTC |
| 25f5742 | Nipunn Koorapati | 29 October 2016, 18:54:27 UTC | Update documentation to point at gcc 4.8 Summary: Rocksdb currently has many references to std::map.emplace_back() which is not implemented in gcc 4.7, but valid in gcc 4.8. Confirmed that it did not build with gcc 4.7, but builds fine with gcc 4.8 Closes https://github.com/facebook/rocksdb/pull/1272 Differential Revision: D4101385 Pulled By: IslamAbdelRahman fbshipit-source-id: f6af453 | 29 October 2016, 19:09:17 UTC |
| b50a81a | Aaron Gao | 29 October 2016, 01:17:17 UTC | Add a test for tailing_iterator Summary: A bug that tailingIterator->Seek(target) skips records. I think the bug is in the SeekInternal starting at lines 387: search_left_bound > search_right_bound There are only 2 cases this can happen: (1) target key is smaller than left most file (2) target key is larger than right most file The comment is wrong, there is another possibility that at the higher level there is a big gap such that the file in the lower level fits completely in the gap and then indexer->GetNextLevelIndex returns search_left_bound > search_right_bound I think pointing on the files after and before the gap. details: https://github.com/facebook/rocksdb/issues/1372 fixed this bug with test case added. Closes https://github.com/facebook/rocksdb/pull/1436 Reviewed By: IslamAbdelRahman Differential Revision: D4099313 Pulled By: lightmark fbshipit-source-id: 6a675b3 | 29 October 2016, 01:24:14 UTC |
| 04751d5 | Siying Dong | 29 October 2016, 00:28:57 UTC | L0 compression should follow options.compression_per_level if not empty Summary: Currently, we don't use options.compression_per_level[0] as the compression style for L0 compression type, unless it is None. This behavior doesn't look like on purpose. This diff will make sure L0 compress using the style of options.compression_per_level[0]. Reviewed and accepted in: https://reviews.facebook.net/D65607 Closes https://github.com/facebook/rocksdb/pull/1435 Differential Revision: D4099368 Pulled By: siying fbshipit-source-id: cfbbdcd | 29 October 2016, 00:39:20 UTC |
| 2946cad | Andrew Kryczka | 28 October 2016, 22:44:48 UTC | Improve RangeDelAggregator documentation Summary: as requested in D62259 Closes https://github.com/facebook/rocksdb/pull/1434 Differential Revision: D4099047 Pulled By: ajkr fbshipit-source-id: a258cfb | 28 October 2016, 22:54:21 UTC |
| 0a9fd05 | Islam AbdelRahman | 28 October 2016, 22:35:48 UTC | Update Vagrant file (test internal phabricator workflow) Summary: Add simple comment to Vagrant file Closes https://github.com/facebook/rocksdb/pull/1433 Differential Revision: D4098740 Pulled By: IslamAbdelRahman fbshipit-source-id: 4903bff | 28 October 2016, 22:39:19 UTC |
| fcd1e0b | Islam AbdelRahman | 26 October 2016, 20:45:08 UTC | Make rocksdb work with internal repo fbshipit-source-id: f52d2b6d39668516270c51945fc4e1693e553ff7 | 28 October 2016, 21:59:50 UTC |
| 0aab5e5 | Willem Jan Withagen | 28 October 2016, 17:44:52 UTC | FreeBSD: malloc_usable_size is in <malloc_np.h> (#1428) Signed-off-by: Willem Jan Withagen <wjw@digiware.nl> | 28 October 2016, 17:44:52 UTC |
| 9c0bb7f | Kefu Chai | 28 October 2016, 17:40:47 UTC | cmake: drop "-march=native" from CXX_FLAGS (#1429) this breaks the cross-compiling, and we can not assume that the building machine and the target machine share the same CPU spec. Signed-off-by: Kefu Chai <kchai@redhat.com> | 28 October 2016, 17:40:47 UTC |
| eeb27e1 | Kien-hung Li | 28 October 2016, 17:36:05 UTC | Add handy option to turn on direct I/O in db_bench (#1424) | 28 October 2016, 17:36:05 UTC |
| c6168d1 | Jan Doms | 28 October 2016, 17:33:49 UTC | removed some declarations from c.h which resulted in undefined symbols (#1407) | 28 October 2016, 17:33:49 UTC |
| bc429de | Aaron Gao | 28 October 2016, 17:25:39 UTC | revert fractional cascading in farward iterator Summary: As offline discussion with Siying, revert this since it has bug with seek. Test Plan: make check -j64 Reviewers: yiwu, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D65559 | 28 October 2016, 17:25:39 UTC |
| b9bc7a2 | Andrew Kryczka | 27 October 2016, 17:07:28 UTC | Use skiplist rep for range tombstone memtable Summary: somehow missed committing this update in D62217 Test Plan: make check Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D65361 | 27 October 2016, 17:07:28 UTC |
| 60a2bbb | Kefu Chai | 25 October 2016, 18:31:39 UTC | Makefile: generate util/build_version.cc from .in file (#1384) * util/build_verion.cc.in: add this file, so cmake and make can share the template file for generating util/build_version.cc. * CMakeLists.txt: also, cmake v2.8.11 does not support file(GENERATE ...), so we are using configure_file() for creating build_version.cc. * Makefile: use util/build_verion.cc.in for creating build_version.cc. Signed-off-by: Kefu Chai <tchaikov@gmail.com> | 25 October 2016, 18:31:39 UTC |
| 9ee8406 | Siying Dong | 25 October 2016, 17:23:50 UTC | Disable DBTest.RepeatedWritesToSameKey (#1420) Summary: The verification condition of the test DBTest.RepeatedWritesToSameKey doesn't hold anymore after 3ce3bb3da2486c2c18a332128dda7c05a91abb85. Disable the test for now before we find a way to replace it. Test Plan: Run the test and make sure it is disabled. | 25 October 2016, 17:23:50 UTC |
| f41df30 | sdong | 24 October 2016, 20:16:54 UTC | OptionChangeMigration() to support FIFO compaction Summary: OptionChangeMigration() to support FIFO compaction. If the DB before migration is using FIFO compaction, nothing should be done. If the desitnation option is FIFO options, compact to one single L0 file if the source has more than one levels. Test Plan: Run option_change_migration_test Reviewers: andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D65289 | 25 October 2016, 01:04:32 UTC |
| 2e8004e | Anirban Rahut | 24 October 2016, 23:23:19 UTC | Changing the legocastle run to use valgrind_test instead of _check Summary: valgrind_test is the correct way to run valgrind tests. this is becasue we need to force DISABLE_JEMALLOC Test Plan: Running sandcastle and contrun Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D65451 | 24 October 2016, 23:23:19 UTC |
