| 98279eb | yiwu-arbug | 13 April 2018, 21:04:19 UTC | Change land button to say "Land to master" | 13 April 2018, 21:04:19 UTC |
| 8af04a3 | Fosco Marotto | 08 March 2018, 19:02:15 UTC | Added back needed branch for our integration. | 08 March 2018, 19:02:15 UTC |
| e476d0e | Amy Tai | 08 March 2018, 18:39:15 UTC | Adding stat to count cancelled compactions Summary: Added a stat that counts the number of cancelled compactions. Closes https://github.com/facebook/rocksdb/pull/3574 Differential Revision: D7190259 Pulled By: amytai fbshipit-source-id: d5ce82dc9398da6d6d34023ad4ed8cec909852a3 | 08 March 2018, 18:42:28 UTC |
| a3a3f54 | Bruce Mitchener | 08 March 2018, 18:18:34 UTC | Fix some typos in comments and docs. Summary: Closes https://github.com/facebook/rocksdb/pull/3568 Differential Revision: D7170953 Pulled By: siying fbshipit-source-id: 9cfb8dd88b7266da920c0e0c1e10fb2c5af0641c | 08 March 2018, 18:27:25 UTC |
| a277b0f | Lukas Rist | 08 March 2018, 18:11:18 UTC | Clarification regarding record format Summary: The CRC is actually calculated based on the record type and payload. The wiki should also be updated accordingly and extended with a section on the recyclable record format. Closes https://github.com/facebook/rocksdb/pull/3576 Differential Revision: D7196478 Pulled By: siying fbshipit-source-id: 39f7a0395075cc73e2aa2bfc9e42c85bce35e765 | 08 March 2018, 18:27:25 UTC |
| b560fc9 | Siying Dong | 08 March 2018, 18:09:59 UTC | Fix a block pinning regression introduced in b555ed30a4a93b80a3ac4781c6721ab988e03b5b Summary: b555ed30a4a93b80a3ac4781c6721ab988e03b5b introduces a regression, which causes blocks always to be pinned in block based iterators. Fix it. Closes https://github.com/facebook/rocksdb/pull/3582 Differential Revision: D7189534 Pulled By: siying fbshipit-source-id: 117dc7a03d0a0e360424db02efb366e12da2be03 | 08 March 2018, 18:12:23 UTC |
| e69f6e8 | Sagar Vemuri | 07 March 2018, 23:17:37 UTC | Fix API name in a comment in db.h Summary: ... so that people are not confused. Closes https://github.com/facebook/rocksdb/pull/3580 Differential Revision: D7187175 Pulled By: sagar0 fbshipit-source-id: bce70093d52e38cd24c9432fd708885d7c2c013e | 07 March 2018, 23:27:17 UTC |
| 0de710f | Bruce Mitchener | 07 March 2018, 20:39:19 UTC | Use nullptr instead of NULL / 0 more consistently. Summary: Closes https://github.com/facebook/rocksdb/pull/3569 Differential Revision: D7170968 Pulled By: yiwu-arbug fbshipit-source-id: 308a6b7dd358a04fd9a7de3d927bfd8abd57d348 | 07 March 2018, 20:42:12 UTC |
| f021f1d | Stuart | 07 March 2018, 04:51:30 UTC | Add rocksdb_open_with_ttl function in C API Summary: Change-Id: Ie6f9b10bce459f6bf0ade0e5877264b4e10da3f5 Signed-off-by: Stuart <Stuart.Hu@emc.com> Closes https://github.com/facebook/rocksdb/pull/3553 Differential Revision: D7144833 Pulled By: sagar0 fbshipit-source-id: 815225fa6e560d8a5bc47ffd0a98118b107ce264 | 07 March 2018, 04:57:20 UTC |
| 0a3db28 | amytai | 07 March 2018, 00:13:05 UTC | Disallow compactions if there isn't enough free space Summary: This diff handles cases where compaction causes an ENOSPC error. This does not handle corner cases where another background job is started while compaction is running, and the other background job triggers ENOSPC, although we do allow the user to provision for these background jobs with SstFileManager::SetCompactionBufferSize. It also does not handle the case where compaction has finished and some other background job independently triggers ENOSPC. Usage: Functionality is inside SstFileManager. In particular, users should set SstFileManager::SetMaxAllowedSpaceUsage, which is the reference highwatermark for determining whether to cancel compactions. Closes https://github.com/facebook/rocksdb/pull/3449 Differential Revision: D7016941 Pulled By: amytai fbshipit-source-id: 8965ab8dd8b00972e771637a41b4e6c645450445 | 07 March 2018, 00:27:54 UTC |
| 20c508c | Andrew Kryczka | 06 March 2018, 20:41:37 UTC | Enable subcompactions in manual level-based compaction Summary: This is the simplest way I could think of to speed up `CompactRange`. It works but isn't that optimal because it relies on the same `max_compaction_bytes` and `max_subcompactions` options that are used in other places. If it turns out to be useful we can allow overriding these in `CompactRangeOptions` in the future. Closes https://github.com/facebook/rocksdb/pull/3549 Differential Revision: D7117634 Pulled By: ajkr fbshipit-source-id: d0cd03d6bd0d2fd7ea3fb13cd3b8bf7c47d11e42 | 06 March 2018, 20:43:51 UTC |
| 3462c94 | Fosco Marotto | 06 March 2018, 20:32:35 UTC | Add dual-license info to README.md Summary: From #3417 and after talking to both GitHub and our open source legal team, the recommended approach was to explicitly state the dual-license in the readme. Changing the license files to accommodate the auto-detection is too much of a pain, would involve editing every code file header. Closes https://github.com/facebook/rocksdb/pull/3541 Differential Revision: D7171111 Pulled By: gfosco fbshipit-source-id: 0ee7b134446015228249efe991fa5e76526ca0b0 | 06 March 2018, 20:43:51 UTC |
| 6a3eebb | Andrew Kryczka | 06 March 2018, 20:31:25 UTC | support multiple db_paths in SstFileManager Summary: Now that files scheduled for deletion are kept in the same directory, we don't need to constrain deletion scheduler to `db_paths[0]`. Previously this was done because there was a separate trash directory, and this constraint prevented files from being accidentally copied to another filesystem when they're scheduled for deletion. Closes https://github.com/facebook/rocksdb/pull/3544 Differential Revision: D7093786 Pulled By: ajkr fbshipit-source-id: 202f5c92d925eafebec1281fb95bb5828d33414f | 06 March 2018, 20:43:51 UTC |
| d518fe1 | Fosco Marotto | 06 March 2018, 20:27:07 UTC | uint64_t and size_t changes to compile for iOS Summary: In attempting to build a static lib for use in iOS, I ran in to lots of type errors between uint64_t and size_t. This PR contains the changes I made to get `TARGET_OS=IOS make static_lib` to succeed while also getting Xcode to build successfully with the resulting `librocksdb.a` library imported. This also compiles for me on macOS and tests fine, but I'm really not sure if I made the correct decisions about where to `static_cast` and where to change types. Also up for discussion: is iOS worth supporting? Getting the static lib is just part one, we aren't providing any bridging headers or wrappers like the ObjectiveRocks project, it won't be a great experience. Closes https://github.com/facebook/rocksdb/pull/3503 Differential Revision: D7106457 Pulled By: gfosco fbshipit-source-id: 82ac2073de7e1f09b91f6b4faea91d18bd311f8e | 06 March 2018, 20:43:51 UTC |
| 8bc41f4 | Siying Dong | 06 March 2018, 20:19:15 UTC | Update TARGETS Summary: Watch the build Closes https://github.com/facebook/rocksdb/pull/3533 Differential Revision: D7063777 Pulled By: siying fbshipit-source-id: db9cdfc362a8d281dada6513ab034a6d6f0d552e | 06 March 2018, 20:27:28 UTC |
| c364eb4 | Dmitri Smirnov | 06 March 2018, 19:47:42 UTC | Windows cumulative patch Summary: This patch addressed several issues. Portability including db_test std::thread -> port::Thread Cc: @ and %z to ROCKSDB portable macro. Cc: maysamyabandeh Implement Env::AreFilesSame Make the implementation of file unique number more robust Get rid of C-runtime and go directly to Windows API when dealing with file primitives. Implement GetSectorSize() and aling unbuffered read on the value if available. Adjust Windows Logger for the new interface, implement CloseImpl() Cc: anand1976 Fix test running script issue where $status var was of incorrect scope so the failures were swallowed and not reported. DestroyDB() creates a logger and opens a LOG file in the directory being cleaned up. This holds a lock on the folder and the cleanup is prevented. This fails one of the checkpoin tests. We observe the same in production. We close the log file in this change. Fix DBTest2.ReadAmpBitmapLiveInCacheAfterDBClose failure where the test attempts to open a directory with NewRandomAccessFile which does not work on Windows. Fix DBTest.SoftLimit as it is dependent on thread timing. CC: yiwu-arbug Closes https://github.com/facebook/rocksdb/pull/3552 Differential Revision: D7156304 Pulled By: siying fbshipit-source-id: 43db0a757f1dfceffeb2b7988043156639173f5b | 06 March 2018, 19:57:43 UTC |
| b864bc9 | Yi Wu | 06 March 2018, 19:46:20 UTC | Blob DB: Improve FIFO eviction Summary: Improving blob db FIFO eviction with the following changes, * Change blob_dir_size to max_db_size. Take into account SST file size when computing DB size. * FIFO now only take into account live sst files and live blob files. It is normal for disk usage to go over max_db_size because there are obsolete sst files and blob files pending deletion. * FIFO eviction now also evict TTL blob files that's still open. It doesn't evict non-TTL blob files. * If FIFO is triggered, it will pass an expiration and the current sequence number to compaction filter. Compaction filter will then filter inlined keys to evict those with an earlier expiration and smaller sequence number. So call LSM FIFO. * Compaction filter also filter those blob indexes where corresponding blob file is gone. * Add an event listener to listen compaction/flush event and update sst file size. * Implement DB::Close() to make sure base db, as well as event listener and compaction filter, destruct before blob db. * More blob db statistics around FIFO. * Fix some locking issue when accessing a blob file. Closes https://github.com/facebook/rocksdb/pull/3556 Differential Revision: D7139328 Pulled By: yiwu-arbug fbshipit-source-id: ea5edb07b33dfceacb2682f4789bea61de28bbfa | 06 March 2018, 19:57:42 UTC |
| 0a2354c | Pooya Shareghi | 06 March 2018, 18:20:52 UTC | Added bytes XOR merge operator Summary: Closes https://github.com/facebook/rocksdb/pull/575 I fixed the merge conflicts etc. Closes https://github.com/facebook/rocksdb/pull/3065 Differential Revision: D7128233 Pulled By: sagar0 fbshipit-source-id: 2c23a48c9f0432c290b0cd16a12fb691bb37820c | 06 March 2018, 18:27:36 UTC |
| 62277e1 | Maysam Yabandeh | 06 March 2018, 07:48:23 UTC | WritePrepared Txn: Move DuplicateDetector to util Summary: Move DuplicateDetector and SetComparator to its own header file in util. It would also address a complaint in the unity test. Closes https://github.com/facebook/rocksdb/pull/3567 Differential Revision: D7163268 Pulled By: maysamyabandeh fbshipit-source-id: 6ddf82773473646dbbc1284ae601a78c4907c778 | 06 March 2018, 07:57:12 UTC |
| 9cb4856 | Huachao Huang | 06 March 2018, 01:44:52 UTC | Don't need to UpdateFilesByCompactionPri for kCompactionStyleNone Summary: Closes https://github.com/facebook/rocksdb/pull/3563 Differential Revision: D7154653 Pulled By: ajkr fbshipit-source-id: 4f32fb1b02451a934504c40be22b07fb1f2deb9c | 06 March 2018, 01:57:39 UTC |
| 5d68243 | Andrew Kryczka | 05 March 2018, 21:08:17 UTC | Comment out unused variables Summary: Submitting on behalf of another employee. Closes https://github.com/facebook/rocksdb/pull/3557 Differential Revision: D7146025 Pulled By: ajkr fbshipit-source-id: 495ca5db5beec3789e671e26f78170957704e77e | 05 March 2018, 21:13:41 UTC |
| 1ccdc2c | Pengchao Wang | 05 March 2018, 19:54:59 UTC | Fix vagrant build process Summary: https://blog.github.com/2018-02-23-weak-cryptographic-standards-removed/ Github dropped supporting some weak cryptographic protocols from their website couple of weeks ago which cause our vagrant build process to fail on curl downloading step. This diff force curl use tls v1.2 protocol if it is supported so that it does not rely on the default protocol on different systems. Closes https://github.com/facebook/rocksdb/pull/3561 Differential Revision: D7148575 Pulled By: wpc fbshipit-source-id: b8cecfdfeb2bc8236de2d0d14f044532befec98c | 05 March 2018, 19:57:41 UTC |
| 92b1a68 | Zhongyi Xie | 05 March 2018, 19:10:38 UTC | fix FreeBSD build Summary: Currently FreeBSD build is broken in master and possibly some previous releases due to unrecognized symbol `O_DIRECT`. This PR will fix the build on FreeBSD Closes https://github.com/facebook/rocksdb/pull/3560 Differential Revision: D7148646 Pulled By: miasantreble fbshipit-source-id: 95b6c3d310fa531267c086b2cd40a5ab1c042b5a | 05 March 2018, 19:12:28 UTC |
| 680864a | Maysam Yabandeh | 05 March 2018, 18:48:29 UTC | WritePrepared Txn: Fix bug with duplicate keys during recovery Summary: Fix the following bugs: - During recovery a duplicate key was inserted twice into the write batch of the recovery transaction, once when the memtable returns false (because it was duplicates) and once for the 2nd attempt. This would result into different SubBatch count measured when the recovered transactions is committing. - If a cf is flushed during recovery the memtable is not available to assist in detecting the duplicate key. This could result into not advancing the sequence number when iterating over duplicate keys of a flushed cf and hence inserting the next key with the wrong sequence number. - SubBacthCounter would reset the comparator to default comparator after the first duplicate key. The 2nd duplicate key hence would have gone through a wrong comparator and not being detected. Closes https://github.com/facebook/rocksdb/pull/3562 Differential Revision: D7149440 Pulled By: maysamyabandeh fbshipit-source-id: 91ec317b165f363f5d11ff8b8c47c81cebb8ed77 | 05 March 2018, 18:57:59 UTC |
| 15f55e5 | Sagar Vemuri | 03 March 2018, 00:19:56 UTC | Fix TSAN timeout in MergeOperatorPinningTest.Randomized/x test Summary: [FB - Internal] MergeOperatorPinningTest.Randomized/x tests are frequently failing with timeouts when run with tsan, as they are exceeding 10 minute limit for tests. The tests are in turn getting disabled due to frequent failures. I halved the number of rounds to make the test complete sooner. This reduces the number of testing iterations a little, but it still is much better than totally letting the test be disabled. Closes https://github.com/facebook/rocksdb/pull/3523 Differential Revision: D7031498 Pulled By: sagar0 fbshipit-source-id: 9a694f2176b235259920a42bf24bca5346f7cff1 | 03 March 2018, 00:27:21 UTC |
| db2445a | Adam Retter | 02 March 2018, 23:33:08 UTC | Brings the Java API for WriteBatch inline with the C++ API Summary: * Exposes status * Corrects some method naming * Adds missing functionality Closes https://github.com/facebook/rocksdb/pull/3550 Differential Revision: D7140790 Pulled By: sagar0 fbshipit-source-id: cbdab6c5a7ae4f3030fb46739e9060e381b26fa6 | 02 March 2018, 23:44:10 UTC |
| 1209b6d | Yi Wu | 02 March 2018, 20:54:24 UTC | Blob DB: remove existing garbage collection implementation Summary: Red diff to remove existing implementation of garbage collection. The current approach is reference counting kind of approach and require a lot of effort to get the size counter right on compaction and deletion. I'm going to go with a simple mark-sweep kind of approach and will send another PR for that. CompactionEventListener was added solely for blob db and it adds complexity and overhead to compaction iterator. Removing it as well. Closes https://github.com/facebook/rocksdb/pull/3551 Differential Revision: D7130190 Pulled By: yiwu-arbug fbshipit-source-id: c3a375ad2639a3f6ed179df6eda602372cc5b8df | 02 March 2018, 20:57:23 UTC |
| 2ac988c | Adam Retter | 02 March 2018, 18:22:38 UTC | Add TransactionDB and OptimisticTransactionDB to the Java API Summary: Closes https://github.com/facebook/rocksdb/issues/697 Closes https://github.com/facebook/rocksdb/issues/1151 Closes https://github.com/facebook/rocksdb/pull/1298 Differential Revision: D7131402 Pulled By: sagar0 fbshipit-source-id: bcd34ce95ed88cc641786089ff4232df7b2f089f | 02 March 2018, 18:34:13 UTC |
| d060421 | Maysam Yabandeh | 02 March 2018, 04:33:41 UTC | Fix a leak in prepared_section_completed_ Summary: The zeroed entries were not removed from prepared_section_completed_ map. This patch adds a unit test to show the problem and fixes that by refactoring the code. The new code is more efficient since i) it uses two separate mutex to avoid contention between commit and prepare threads, ii) it uses a sorted vector for maintaining uniq log entires with prepare which avoids a very large heap with many duplicate entries. Closes https://github.com/facebook/rocksdb/pull/3545 Differential Revision: D7106071 Pulled By: maysamyabandeh fbshipit-source-id: b3ae17cb6cd37ef10b6b35e0086c15c758768a48 | 02 March 2018, 04:41:56 UTC |
| bf937cf | Yi Wu | 02 March 2018, 01:50:54 UTC | Add "rocksdb.live-sst-files-size" DB property Summary: Add "rocksdb.live-sst-files-size" DB property which only include files of latest version. Existing "rocksdb.total-sst-files-size" include files from all versions and thus include files that's obsolete but not yet deleted. I'm going to use this new property to cap blob db sst + blob files size. Closes https://github.com/facebook/rocksdb/pull/3548 Differential Revision: D7116939 Pulled By: yiwu-arbug fbshipit-source-id: c6a52e45ce0f24ef78708156e1a923c1dd6bc79a | 02 March 2018, 02:01:10 UTC |
| ec5843d | leviathan1995 | 28 February 2018, 17:49:32 UTC | Comment typo Summary: Closes https://github.com/facebook/rocksdb/pull/3546 Differential Revision: D7111708 Pulled By: ajkr fbshipit-source-id: 522a4a00eb3e34c73afcb86c1f75cd2e90e7608d | 28 February 2018, 17:56:45 UTC |
| 3ae0047 | Andrew Kryczka | 28 February 2018, 01:08:34 UTC | skip CompactRange flush based on memtable contents Summary: CompactRange has a call to Flush because we guarantee that, at the time it's called, all existing keys in the range will be pushed through the user's compaction filter. However, previously the flush was done blindly, so it'd happen even if the memtable does not contain keys in the range specified by the user. This caused unnecessarily many L0 files to be created, leading to write stalls in some cases. This PR checks the memtable's contents, and decides to flush only if it overlaps with `CompactRange`'s range. - Move the memtable overlap check logic from `ExternalSstFileIngestionJob` to `ColumnFamilyData::RangesOverlapWithMemtables` - Reuse the above logic in `CompactRange` and skip flushing if no overlap Closes https://github.com/facebook/rocksdb/pull/3520 Differential Revision: D7018897 Pulled By: ajkr fbshipit-source-id: a3c6b1cfae56687b49dd89ccac7c948e53545934 | 28 February 2018, 01:12:44 UTC |
| c287c09 | Siying Dong | 27 February 2018, 20:32:09 UTC | Update comments in DB::Close() Summary: Closes https://github.com/facebook/rocksdb/pull/3543 Differential Revision: D7093251 Pulled By: siying fbshipit-source-id: 4066b82c95ecb65866c5842d68ab13ab9f85d567 | 27 February 2018, 20:42:31 UTC |
| d633656 | Istvan Szukacs | 26 February 2018, 23:17:22 UTC | Adding CentOS 7 Vagrantfile & build script Summary: I have updated the Vagrantfile to have an entry for CentOS 7. Also created a simple build script which is pretty similar to the one in Beringei. How to test: ``` vagrant up centos7 ``` Todo: Implement -j X for the build. Closes https://github.com/facebook/rocksdb/pull/3530 Differential Revision: D7090739 Pulled By: ajkr fbshipit-source-id: 9f9eda5b507568993543d08de7ce168dfc12282e | 26 February 2018, 23:27:17 UTC |
| ad05cbb | Zhongyi Xie | 26 February 2018, 22:46:12 UTC | DB:Open should fail on tmpfs when use_direct_reads=true Summary: Before: > $ TEST_TMPDIR=/dev/shm ./db_bench -use_direct_reads=true -benchmarks=readrandomwriterandom -num=10000000 -reads=100000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -readwritepercent=50 -key_size=16 -value_size=48 -threads=32 DB path: [/dev/shm/dbbench] put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument db_bench: tpp.c:84: __pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio >= fifo_min_prio && new_prio <= fifo_max_prio)' failed. put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument put error: IO error: While open a file for random read: /dev/shm/dbbench/000007.sst: Invalid argument After: > TEST_TMPDIR=/dev/shm ./db_bench -use_direct_reads=true -benchmarks=readrandomwriterandom -num=10000000 -reads=100000 -write_buffer_size=1048576 -target_file_size_base=1048576 -max_bytes_for_level_base=4194304 -max_background_jobs=12 -readwritepercent=50 -key_size=16 -value_size=48 -threads=32 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags open error: Not implemented: Direct I/O is not supported by the specified DB. Closes https://github.com/facebook/rocksdb/pull/3539 Differential Revision: D7082658 Pulled By: miasantreble fbshipit-source-id: f9d9c6ec3b5e9e049cab52154940ee101ba4d342 | 26 February 2018, 22:58:06 UTC |
| 7eb292d | Dmitri Smirnov | 26 February 2018, 21:44:25 UTC | Fix a memory leak in WindowsThread Summary: _endthreadex does not return and thus objects for stack destructors do not run. This creates a memory leak. We remove the calls since _enthreadex called automatically after the threadproc returns i.e. thread exits. Closes https://github.com/facebook/rocksdb/pull/3542 Differential Revision: D7088713 Pulled By: ajkr fbshipit-source-id: 749ecafc6a9572f587f76e516547e07734349a54 | 26 February 2018, 21:46:12 UTC |
| dfbe52e | Anand Ananthabhotla | 23 February 2018, 21:50:02 UTC | Fix the Logger::Close() and DBImpl::Close() design pattern Summary: The recent Logger::Close() and DBImpl::Close() implementation rely on calling the CloseImpl() virtual function from the destructor, which will not work. Refactor the implementation to have a private close helper function in derived classes that can be called by both CloseImpl() and the destructor. Closes https://github.com/facebook/rocksdb/pull/3528 Reviewed By: gfosco Differential Revision: D7049303 Pulled By: anand1976 fbshipit-source-id: 76a64cbf403209216dfe4864ecf96b5d7f3db9f4 | 23 February 2018, 21:57:26 UTC |
| 30649dc | Siying Dong | 23 February 2018, 19:30:55 UTC | Have a different function when ROCKSDB_JEMALLOC=0 Summary: Some sanitizer is not happy with parameter name with ROCKSDB_JEMALLOC not set. Use another function instead. Closes https://github.com/facebook/rocksdb/pull/3536 Differential Revision: D7064849 Pulled By: siying fbshipit-source-id: c6ae94e044686176af1259df9172453d52c2f9d5 | 23 February 2018, 19:42:33 UTC |
| 90eca1e | Maysam Yabandeh | 23 February 2018, 02:05:14 UTC | WritePrepared Txn: optimize SubBatchCnt Summary: Make use of the index in WriteBatchWithIndex to also count the number of sub-batches. This eliminates the need to separately scan the batch to count the number of sub-batches once a duplicate key is detected. Closes https://github.com/facebook/rocksdb/pull/3529 Differential Revision: D7049947 Pulled By: maysamyabandeh fbshipit-source-id: 81cbf12c4e662541c772c7265a8f91631e25c7cd | 23 February 2018, 02:12:26 UTC |
| 243220d | Zhongyi Xie | 23 February 2018, 00:31:58 UTC | Update HISTORY.md to 5.12.0 Summary: Closes https://github.com/facebook/rocksdb/pull/3532 Differential Revision: D7062828 Pulled By: miasantreble fbshipit-source-id: d36967a1cfbcaeeeb33b9f0e09e15dea85b08b70 | 23 February 2018, 00:47:01 UTC |
| 4624edc | Siying Dong | 22 February 2018, 21:23:53 UTC | RocksDBOptionsParser::Parse()'s `ignore_unknown_options` argument only ingores options from higher version. Summary: RocksDB should always be able to parse an option file generated using the same or lower version. Unknown option should only happen if it is from a higher version. Change the behavior of RocksDBOptionsParser::Parse()'s behavior with ignore_unknown_options=true so that unknown option from a lower or the same version will never be skipped. Closes https://github.com/facebook/rocksdb/pull/3527 Differential Revision: D7048851 Pulled By: siying fbshipit-source-id: e261caea12f6515611a4a29f39acf2b619df2361 | 22 February 2018, 21:28:12 UTC |
| aba3409 | Igor Sugak | 22 February 2018, 20:36:41 UTC | Back out "[codemod] - comment out unused parameters" Reviewed By: igorsugak fbshipit-source-id: 4a93675cc1931089ddd574cacdb15d228b1e5f37 | 22 February 2018, 20:43:17 UTC |
| f4a030c | David Lai | 22 February 2018, 17:37:17 UTC | - comment out unused parameters Reviewed By: everiq, igorsugak Differential Revision: D7046710 fbshipit-source-id: 8e10b1f1e2aecebbfb229c742e214db887e5a461 | 22 February 2018, 17:44:23 UTC |
| b092977 | Andrew Kryczka | 22 February 2018, 01:33:14 UTC | BackupEngine gluster-friendly file naming convention Summary: Use the rsync tempfile naming convention in our `BackupEngine`. The temp file follows the format, `.<filename>.<suffix>`, which is later renamed to `<filename>`. We fix `tmp` as the `<suffix>` as we don't need to use random bytes for now. The benefit is gluster treats this tempfile naming convention specially and applies hashing only to `<filename>`, so the file won't need to be linked or moved when it's renamed. Our gluster team suggested this will make things operationally easier. Closes https://github.com/facebook/rocksdb/pull/3463 Differential Revision: D6893333 Pulled By: ajkr fbshipit-source-id: fd7622978f4b2487fce33cde40dd3124f16bcaa8 | 22 February 2018, 01:42:07 UTC |
| 828211e | Maysam Yabandeh | 21 February 2018, 21:40:31 UTC | WritePrepared Txn: fix non-emptied PreparedHeap bug Summary: Under a certain sequence of accessing PreparedHeap, there was a bug that would not successfully empty the heap. This would result in performance issues when the heap content is moved to old_prepared_ after max_evicted_seq_ advances the orphan prepared sequence numbers. The patch fixed the bug and add more unit tests. It also does more logging when the unlikely scenarios are faced Closes https://github.com/facebook/rocksdb/pull/3526 Differential Revision: D7038486 Pulled By: maysamyabandeh fbshipit-source-id: f1e40bea558f67b03d2a29131fcb8734c65fce97 | 21 February 2018, 21:42:23 UTC |
| 8ada876 | Sagar Vemuri | 21 February 2018, 03:05:21 UTC | Add rocksdb.iterator.internal-key property Summary: Added a new iterator property: `rocksdb.iterator.internal-key` to get the internal-key (converted to user key) at which the iterator stopped. Closes https://github.com/facebook/rocksdb/pull/3525 Differential Revision: D7033694 Pulled By: sagar0 fbshipit-source-id: d51e6c00f5e9d766c6276ef79774b81c6c5216f8 | 21 February 2018, 03:12:09 UTC |
| e9c31ab | jsteemann | 21 February 2018, 01:34:44 UTC | save redundant key lookup in map of locked keys Summary: In case it is found that a key is already marked as locked in a stripe's map of locked keys, it is not necessary to look it up again using `std::unordered_map<std::string, ...>::at(size_t)`. Instead, we can use the already found position using the iterator produced by the previous `find` operation. Reusing the iterator will avoid having to hash the key again and do additional "random" memory lookups in the map of keys (though the data will very likely sit available in caches here already due to the previous find operation) Closes https://github.com/facebook/rocksdb/pull/3505 Differential Revision: D7036446 Pulled By: sagar0 fbshipit-source-id: cced51547b2bd2d49394f6bc8c5896f09fa80f68 | 21 February 2018, 01:44:44 UTC |
| 1960e73 | Andrew Kryczka | 21 February 2018, 00:42:06 UTC | fix handling of empty string as checkpoint directory Summary: - made `CreateCheckpoint` properly return `InvalidArgument` when called with an empty directory. Previously it triggered an assertion failure due to a bug in the logic. - made `ldb` set empty `checkpoint_dir` if that's what the user specifies, so that we can use it to properly test `CreateCheckpoint` in the future. Differential Revision: D6874562 fbshipit-source-id: dcc1bd41768261d9338987fa7711444289707ed7 | 21 February 2018, 00:44:00 UTC |
| 5263da6 | Igor Sugak | 21 February 2018, 00:41:54 UTC | fix shift UBSAN error in col_buf_encoder.cc Summary: Add a static cast to perform the left shift as with an unsigned type. make ubsan_check Closes https://github.com/facebook/rocksdb/pull/3517 Reviewed By: sagar0 Differential Revision: D7016044 Pulled By: igorsugak fbshipit-source-id: baf72f6197edd8f7220d010b15a23d6de6a72c49 | 21 February 2018, 00:44:00 UTC |
| ab446dc | Po-Chuan Hsieh | 16 February 2018, 18:34:48 UTC | Fix build with USE_RTTI=0 Summary: utilities/column_aware_encoding_util.cc:61:23: error: cannot use dynamic_cast with -fno-rtti table_reader_.reset(dynamic_cast<BlockBasedTable*>(table_reader.release())); ^ 1 error generated. It was added as a [local patch](https://svnweb.freebsd.org/ports/head/databases/rocksdb/files/patch-utilities-column_aware_encoding_util.cc) on FreeBSD since RocksDB 5.8. It also fixes #2707. Closes https://github.com/facebook/rocksdb/pull/3514 Differential Revision: D7005571 Pulled By: siying fbshipit-source-id: 351a9055d21d0accdd7a932e8e7bfcd3c8e22068 | 16 February 2018, 18:41:49 UTC |
| c178da0 | Maysam Yabandeh | 16 February 2018, 16:36:47 UTC | WritePrepared Txn: optimizations for sysbench update_noindex Summary: These are optimization that we applied to improve sysbech's update_noindex performance. 1. Make use of LIKELY compiler hint 2. Move std::atomic so the subclass 3. Make use of skip_prepared in non-2pc transactions. Closes https://github.com/facebook/rocksdb/pull/3512 Differential Revision: D7000075 Pulled By: maysamyabandeh fbshipit-source-id: 1ab8292584df1f6305a4992973fb1b7933632181 | 16 February 2018, 16:42:31 UTC |
| 97307d8 | Mike Kolupaev | 16 February 2018, 15:58:18 UTC | Fix deadlock in ColumnFamilyData::InstallSuperVersion() Summary: Deadlock: a memtable flush holds DB::mutex_ and calls ThreadLocalPtr::Scrape(), which locks ThreadLocalPtr mutex; meanwhile, a thread exit handler locks ThreadLocalPtr mutex and calls SuperVersionUnrefHandle, which tries to lock DB::mutex_. This deadlock is hit all the time on our workload. It blocks our release. In general, the problem is that ThreadLocalPtr takes an arbitrary callback and calls it while holding a lock on a global mutex. The same global mutex is (at least in some cases) locked by almost all ThreadLocalPtr methods, on any instance of ThreadLocalPtr. So, there'll be a deadlock if the callback tries to do anything to any instance of ThreadLocalPtr, or waits for another thread to do so. So, probably the only safe way to use ThreadLocalPtr callbacks is to do only do simple and lock-free things in them. This PR fixes the deadlock by making sure that local_sv_ never holds the last reference to a SuperVersion, and therefore SuperVersionUnrefHandle never has to do any nontrivial cleanup. I also searched for other uses of ThreadLocalPtr to see if they may have similar bugs. There's only one other use, in transaction_lock_mgr.cc, and it looks fine. Closes https://github.com/facebook/rocksdb/pull/3510 Reviewed By: sagar0 Differential Revision: D7005346 Pulled By: al13n321 fbshipit-source-id: 37575591b84f07a891d6659e87e784660fde815f | 16 February 2018, 16:13:34 UTC |
| 0454f78 | Andrew Kryczka | 16 February 2018, 03:30:52 UTC | fix advance reservation of arena block addresses Summary: Calling `std::vector::reserve()` causes memory to be reallocated and then data to be moved. It was called prior to adding every block. This reallocation could be done a huge amount of times, e.g., for users with large index blocks. Instead, we can simply use `std::vector::emplace_back()` in such a way that preserves the no-memory-leak guarantee, while letting the vector decide when to reallocate space. Now I see reallocation/moving happen O(logN) times, rather than O(N) times, where N is the final size of vector. Closes https://github.com/facebook/rocksdb/pull/3508 Differential Revision: D6994228 Pulled By: ajkr fbshipit-source-id: ab7c11e13ff37c8c6c8249be7a79566a4068cd27 | 16 February 2018, 03:41:52 UTC |
| 989d123 | Yi Wu | 16 February 2018, 01:14:08 UTC | Legocastle job to report lite build binary size to scuba Summary: Add a legocastle job to continuously build the last 10 commits every 4 hours and report lite build binary size to scuba. Closes https://github.com/facebook/rocksdb/pull/3511 Differential Revision: D7001730 Pulled By: yiwu-arbug fbshipit-source-id: 7c8ca87c46d663c786a0d32be69ebbe7b19a5eb9 | 16 February 2018, 01:27:24 UTC |
| 8eb1d44 | Maysam Yabandeh | 16 February 2018, 01:12:48 UTC | Unbreak MemTableRep API change Summary: The MemTableRep API was broken by this commit: 813719e9525f647aaebf19ca3d4bb6f1c63e2648 This patch reverts the changes and instead adds InsertKey (and etc.) overloads to extend the MemTableRep API without breaking the existing classes that inherit from it. Closes https://github.com/facebook/rocksdb/pull/3513 Differential Revision: D7004134 Pulled By: maysamyabandeh fbshipit-source-id: e568d91fe1e17dd76c0c1f6c7dd51a18633b1c4f | 16 February 2018, 01:27:24 UTC |
| 4e7a182 | jsteemann | 16 February 2018, 00:43:23 UTC | Several small "fixes" Summary: - removed a few unneeded variables - fused some variable declarations and their assignments - fixed right-trimming code in string_util.cc to not underflow - simplifed an assertion - move non-nullptr check assertion before dereferencing of that pointer - pass an std::string function parameter by const reference instead of by value (avoiding potential copy) Closes https://github.com/facebook/rocksdb/pull/3507 Differential Revision: D7004679 Pulled By: sagar0 fbshipit-source-id: 52944952d9b56dfcac3bea3cd7878e315bb563c4 | 16 February 2018, 00:57:37 UTC |
| c88c57c | Zhongyi Xie | 15 February 2018, 22:11:08 UTC | Tweak external file ingestion seqno logic under universal compaction Summary: Right now it is possible that a file gets assigned to L0 but also assigned the seqno from a higher level which it doesn't fit Under the current impl, it is possibe that seqno in lower levels (Ln) can be equal to smallest seqno of higher levels (Ln-1), which is undesirable from universal compaction's point of view. This should fix the intermittent failure of `ExternalSSTFileBasicTest.IngestFileWithGlobalSeqnoPickedSeqno` Closes https://github.com/facebook/rocksdb/pull/3411 Differential Revision: D6813802 Pulled By: miasantreble fbshipit-source-id: 693d0462fa94725ccfb9d8858743e6d2d9992d14 | 15 February 2018, 22:13:39 UTC |
| 6a30b98 | jsteemann | 15 February 2018, 19:01:42 UTC | fix wrong indentation Summary: Somehow the indentation was incorrect in this file. The only change in this PR is to get it right again in order to make the code more readable. Please reject if you think it's not worth it. Closes https://github.com/facebook/rocksdb/pull/3504 Differential Revision: D6996011 Pulled By: miasantreble fbshipit-source-id: 060514a3a8c910d34bad795b36eb4d278512b154 | 15 February 2018, 19:13:37 UTC |
| ba6ee1f | Fosco Marotto | 14 February 2018, 19:08:39 UTC | Fix 2 more unused reference errors VS2017 Summary: As in #3425 Closes https://github.com/facebook/rocksdb/pull/3497 Differential Revision: D6979588 Pulled By: gfosco fbshipit-source-id: e9fb32d04ad45575dfe9de1d79348d158e474197 | 14 February 2018, 19:12:36 UTC |
| b3c5351 | Siying Dong | 14 February 2018, 00:20:13 UTC | Direct I/O writable file should do fsync in Close() Summary: We don't do fsync() after truncate in direct I/O writeable file (in fact we don't do any fsync ever). This can cause metadata not persistent to disk after the file is generated. We call it instead. Closes https://github.com/facebook/rocksdb/pull/3500 Differential Revision: D6981482 Pulled By: siying fbshipit-source-id: 7e2b591b7e5dd1b96fc0775515b8b9e6092980ef | 14 February 2018, 00:27:11 UTC |
| d08d05c | Igor Sugak | 13 February 2018, 22:07:48 UTC | fix UBSAN errors in fault_injection_test Summary: This fixes shift and signed-integer-overflow UBSAN checks in fault_injection_test by using a larger and unsigned type. Closes https://github.com/facebook/rocksdb/pull/3498 Reviewed By: siying Differential Revision: D6981116 Pulled By: igorsugak fbshipit-source-id: 3688f62cce570534b161e9b5f42109ebc9ae5a2c | 13 February 2018, 22:12:40 UTC |
| dadf016 | Siying Dong | 13 February 2018, 21:44:22 UTC | Rename one of the two LevelIterator Summary: A new LevelIterator was recently created. Rename the old one to make unity build happy. It's also not a good idea to have two classes in the same name anyway. Closes https://github.com/facebook/rocksdb/pull/3499 Differential Revision: D6979325 Pulled By: siying fbshipit-source-id: 3a032d93fe205650a08e92e5262594731ec726bb | 13 February 2018, 21:57:58 UTC |
| 7474861 | Siying Dong | 13 February 2018, 20:05:36 UTC | Suppress UBSAN error in finer guanularity Summary: Now we suppress alignment UBSAN error as a whole. Suppressing 3-way CRC and murmurhash feels a better idea than turning off alignment check as a whole. Closes https://github.com/facebook/rocksdb/pull/3495 Differential Revision: D6971273 Pulled By: siying fbshipit-source-id: 080b59fed6df494b9f622ef7cb5d42d39e6a8cdf | 13 February 2018, 20:18:07 UTC |
| 3c380fd | Fosco Marotto | 13 February 2018, 19:45:24 UTC | Adding blog post for 5.10.2 release Summary: Closes https://github.com/facebook/rocksdb/pull/3464 Differential Revision: D6906184 Pulled By: gfosco fbshipit-source-id: 415934d7b1dd8dd226b6619bfb71781184d55cd9 | 13 February 2018, 19:56:59 UTC |
| b555ed3 | Siying Dong | 13 February 2018, 00:57:56 UTC | Customized BlockBasedTableIterator and LevelIterator Summary: Use a customzied BlockBasedTableIterator and LevelIterator to replace current implementations leveraging two-level-iterator. Hope the customized logic will make code easier to understand. As a side effect, BlockBasedTableIterator reduces the allocation for the data block iterator object, and avoid the virtual function call to it, because we can directly reference BlockIter, a final class. Similarly, LevelIterator reduces virtual function call to the dummy iterator iterating the file metadata. It also enabled further optimization. The upper bound check is also moved from index block to data block. This implementation fits this iterator better. After the change, forwared iterator is slightly optimized to ensure we trim those iterators. The two-level-iterator now is only used by partitioned index, so it is simplified. Closes https://github.com/facebook/rocksdb/pull/3406 Differential Revision: D6809041 Pulled By: siying fbshipit-source-id: 7da3b9b1d3c8e9d9405302c15920af1fcaf50ffa | 13 February 2018, 01:12:25 UTC |
| 8a04ee4 | Maysam Yabandeh | 13 February 2018, 00:27:39 UTC | WritePrepared Txn: use TransactionDBWriteOptimizations (2nd attempt) Summary: TransactionDB::Write can receive some optimization hints from the user. One is to skip the concurrency control mechanism. WritePreparedTxnDB is currently ignoring such hints. This patch optimizes WritePreparedTxnDB::Write for skip_concurrency_control and skip_duplicate_key_check hints. Closes https://github.com/facebook/rocksdb/pull/3496 Differential Revision: D6971784 Pulled By: maysamyabandeh fbshipit-source-id: cbab10ad538fa2b8bcb47e37c77724afe6e30f03 | 13 February 2018, 00:43:40 UTC |
| ee1c802 | Andrew Kryczka | 12 February 2018, 23:34:39 UTC | Add delay before flush in CompactRange to avoid write stalling Summary: - Refactored logic for checking write stall condition to a helper function: `GetWriteStallConditionAndCause`. Now it is decoupled from the logic for updating WriteController / stats in `RecalculateWriteStallConditions`, so we can reuse it for predicting whether write stall will occur. - Updated `CompactRange` to first check whether the one additional immutable memtable / L0 file would cause stalling before it flushes. If so, it waits until that is no longer true. - Updated `bg_cv_` to be signaled on `SetOptions` calls. The stall conditions `CompactRange` cares about can change when (1) flush finishes, (2) compaction finishes, or (3) options dynamically change. The cv was already signaled for (1) and (2) but not yet for (3). Closes https://github.com/facebook/rocksdb/pull/3381 Differential Revision: D6754983 Pulled By: ajkr fbshipit-source-id: 5613e03f1524df7192dc6ae885d40fd8f091d972 | 12 February 2018, 23:42:47 UTC |
| 0a0fad4 | Andrew Kryczka | 12 February 2018, 22:54:50 UTC | db_bench separate options for partition index and filters Summary: Some workloads (like my current benchmarking) may want partitioned indexes without partitioned filters. Particularly, when `-optimize_filters_for_hits=true`, the total index size may be larger than the total filter size, so it can make sense to hold all filters in-memory but not all indexes. Closes https://github.com/facebook/rocksdb/pull/3492 Differential Revision: D6970092 Pulled By: ajkr fbshipit-source-id: b7fa1828e1d13829339aefb90fd56eb7c5337f61 | 12 February 2018, 22:57:13 UTC |
| 3f1bb07 | Zhongyi Xie | 12 February 2018, 21:14:01 UTC | make flush_reason_ atomic to keep TSAN happy Summary: Closes https://github.com/facebook/rocksdb/pull/3487 Differential Revision: D6967098 Pulled By: miasantreble fbshipit-source-id: 48e0accf2e3b3f589ddb797ff8083c8520269bf0 | 12 February 2018, 21:28:18 UTC |
| ef29d2a | Siying Dong | 09 February 2018, 22:50:09 UTC | Explictly fail writes if key or value is not smaller than 4GB Summary: Right now, users will encounter unexpected bahavior if they use key or value larger than 4GB. We should explicitly fail the queriers. Closes https://github.com/facebook/rocksdb/pull/3484 Differential Revision: D6953895 Pulled By: siying fbshipit-source-id: b60491e1af064fc5d52971956661f6c18ceac24f | 09 February 2018, 22:57:54 UTC |
| fe228da | Yi Wu | 09 February 2018, 22:43:56 UTC | WritePrepared Txn: Support merge operator Summary: CompactionIterator invoke MergeHelper::MergeUntil() to do partial merge between snapshot boundaries. Previously it only depend on sequence number to tell snapshot boundary, but we also need to make use of snapshot_checker to verify visibility of the merge operands to the snapshots. For example, say there is a snapshot with seq = 2 but only can see data with seq <= 1. There are three merges, each with seq = 1, 2, 3. A correct compaction output would be (1),(2+3). Without taking snapshot_checker into account when generating merge result, compaction will generate output (1+2),(3). By filtering uncommitted keys with read callback, the read path already take care of merges well and don't need additional updates. Closes https://github.com/facebook/rocksdb/pull/3475 Differential Revision: D6926087 Pulled By: yiwu-arbug fbshipit-source-id: 8f539d6f897cfe29b6dc27a8992f68c2a629d40a | 09 February 2018, 22:57:54 UTC |
| 9fc72d6 | Chinmay Kamat | 09 February 2018, 22:01:10 UTC | Compilation fixes for powerpc build, -Wparentheses-equality error and missing header guards Summary: This pull request contains miscellaneous compilation fixes. Thanks, Chinmay Closes https://github.com/facebook/rocksdb/pull/3462 Differential Revision: D6941424 Pulled By: sagar0 fbshipit-source-id: fe9c26507bf131221f2466740204bff40a15614a | 09 February 2018, 22:12:43 UTC |
| d62af7f | jsteemann | 09 February 2018, 20:53:11 UTC | fix a typo (of a potential vi user) Summary: Closes https://github.com/facebook/rocksdb/pull/3481 Differential Revision: D6939089 Pulled By: siying fbshipit-source-id: ccce3ae10cc5ff50a74b85804afd044b21a3c3e2 | 09 February 2018, 20:58:07 UTC |
| 945f618 | Zhongyi Xie | 09 February 2018, 20:09:55 UTC | log flush reason for better debugging experience Summary: It's always a mystery from the logs why flush was triggered -- user triggered it manually, WriteBufferManager triggered it, logs were full, write buffer was full, etc. This PR logs Flush reason whenever a flush is scheduled. Closes https://github.com/facebook/rocksdb/pull/3401 Differential Revision: D6788142 Pulled By: miasantreble fbshipit-source-id: a867e54d493c06adf5172bd36a180fb3faae3511 | 09 February 2018, 20:12:43 UTC |
| e78715c | Andrew Kryczka | 07 February 2018, 23:42:35 UTC | Eliminate a memcpy for uncompressed blocks Summary: `ReadBlockFromFile` uses a stack buffer to hold small data blocks before passing them to the compression library, which outputs uncompressed data in a heap buffer. In the case of `kNoCompression` there is a `memcpy` to copy from stack buffer to heap buffer. This PR optimizes `ReadBlockFromFile` to skip the stack buffer for files whose blocks are known to be uncompressed. We determine this using the SST file property, "compression_name", if it's available. Closes https://github.com/facebook/rocksdb/pull/3472 Differential Revision: D6920848 Pulled By: ajkr fbshipit-source-id: 5c753e804efc178b9229ae5dbe6a4adc32031f07 | 07 February 2018, 23:57:37 UTC |
| a0931b3 | Siying Dong | 07 February 2018, 22:41:33 UTC | Fix UBSAN Error in WritePreparedTransactionTest Summary: WritePreparedTransactionTest has the UBSAN error because the wrong order of its parent class construction. Fix it. Closes https://github.com/facebook/rocksdb/pull/3478 Differential Revision: D6928975 Pulled By: siying fbshipit-source-id: 13edfd5cb9cf73f1ac5ae3b6f53061d32783733d | 07 February 2018, 22:57:35 UTC |
| 821e0b1 | Siying Dong | 07 February 2018, 22:28:50 UTC | Disable options_settable_test in UBSAN and fix UBSAN failure in blob_… Summary: …db_test options_settable_test won't pass UBSAN so disable it. blob_db_test fails in UBSAN as SnapshotList doesn't initialize all the fields in dummy snapshot. Fix it. I don't understand why only blob_db_test fails though. Closes https://github.com/facebook/rocksdb/pull/3477 Differential Revision: D6928681 Pulled By: siying fbshipit-source-id: e31dd300fcdecdfd4f6af279a0987fd0cdec5122 | 07 February 2018, 22:42:26 UTC |
| 1336a77 | Siying Dong | 07 February 2018, 18:45:46 UTC | Disable alignment check in UBSAN Summary: Disable alignment check in UBSAN for now. Now we can't get signals to meaningful failures. We can reenable it after we figure out how we can suppress failures in finer grain manner. Closes https://github.com/facebook/rocksdb/pull/3473 Differential Revision: D6925971 Pulled By: siying fbshipit-source-id: a0f1a242cde866abbc5c1eeee9ff8d1d7d582ac4 | 07 February 2018, 18:58:01 UTC |
| 8feee28 | Maysam Yabandeh | 06 February 2018, 23:22:58 UTC | Add skip_cc option to TransactionDB::Write Summary: Compared to DB::Write, TransactionDB::Write has the additional overhead of creating and initializing an internal transaction object, as well as the overhead of locking/unlocking the keys. This patch extends the TransactionDB::Write with an skip_cc option to allow the users to indicate that the write batch do not conflict with others and the concurrency control and its overhead can be skipped. TransactionDB::Write by default calls DB::Write when skip_cc is set, which works for WriteCommitted WritePolicy. Any other flavor of TransactionDB that is not compatible with this default behavior (such as WritePreparedTxnDB) can extend ::Write and implement their own approach for taking into account the skip_cc optimization. Closes https://github.com/facebook/rocksdb/pull/3457 Differential Revision: D6877318 Pulled By: maysamyabandeh fbshipit-source-id: 56f4e21db87ff71492db4e376fb7c2b03dfeab6b | 06 February 2018, 23:28:24 UTC |
| 8f8eb4f | Maysam Yabandeh | 06 February 2018, 22:17:18 UTC | Fix leak report by asan on DuplicateKeys test Summary: Deletes the transaction object at the end of the test. Verified by: - COMPILE_WITH_ASAN=1 make -j32 transaction_test - ./transaction_test --gtest_filter="DBA**Duplicate*" Closes https://github.com/facebook/rocksdb/pull/3470 Differential Revision: D6916473 Pulled By: maysamyabandeh fbshipit-source-id: 8303df25408635d5d3ac2b25f309a3d15957c937 | 06 February 2018, 22:26:35 UTC |
| 81736d8 | Yi Wu | 06 February 2018, 22:06:44 UTC | WritePrepared Txn: update compaction_iterator_test and db_iterator_test Summary: Update compaction_iterator_test with write-prepared transaction DB related tests. Transaction related tests are group in CompactionIteratorWithSnapshotCheckerTest. The existing test are duplicated to make them also test with dummy SnapshotChecker that will say every key is visible to every snapshot (this is okay, we still compare sequence number to verify visibility). Merge related tests are disabled and will be revisit in another PR. Existing db_iterator_tests are also duplicated to test with dummy read_callback that will say every key is committed. Closes https://github.com/facebook/rocksdb/pull/3466 Differential Revision: D6909253 Pulled By: yiwu-arbug fbshipit-source-id: 2ae4656b843a55e2e9ff8beecf21f2832f96cd25 | 06 February 2018, 22:12:13 UTC |
| 2f29991 | Zhongyi Xie | 06 February 2018, 21:54:53 UTC | split RandomizedHarnessTest more ways Summary: RandomizedHarnessTest enumerates different combinations of test type, compression type, restart interval, etc. For some combinations it takes very long to finish, causing the test to time out in test infrastructure. This PR split the test input into smaller trunks in the hope that they will fit in the timeout window. Another possibility is to reduce `num_entries` of course Closes https://github.com/facebook/rocksdb/pull/3467 Differential Revision: D6910235 Pulled By: miasantreble fbshipit-source-id: 717246ee5d21a8a48ad82d4d9c04f9051a66f07f | 06 February 2018, 21:58:18 UTC |
| 88d8b2a | Maysam Yabandeh | 06 February 2018, 02:32:54 UTC | WritePrepared Txn: Duplicate Keys, Txn Part Summary: This patch takes advantage of memtable being able to detect duplicate <key,seq> and returning TryAgain to handle duplicate keys in WritePrepared Txns. Through WriteBatchWithIndex's index it detects existence of at least a duplicate key in the write batch. If duplicate key was reported, it then pays the cost of counting the number of sub-patches by iterating over the write batch and pass it to DBImpl::Write. DB will make use of the provided batch_count to assign proper sequence numbers before sending them to the WAL. When later inserting the batch to the memtable, it increases the seq each time memtbale reports a duplicate (a sub-patch in our counting) and tries again. Closes https://github.com/facebook/rocksdb/pull/3455 Differential Revision: D6873699 Pulled By: maysamyabandeh fbshipit-source-id: db8487526c3a5dc1ddda0ea49f0f979b26ae648d | 06 February 2018, 02:43:24 UTC |
| 4b124fb | Anand Ananthabhotla | 05 February 2018, 21:48:25 UTC | Handle error return from WriteBuffer() Summary: There are a couple of places where we swallow any error from WriteBuffer() - in SwitchMemtable() and DBImpl::CloseImpl(). Propagate the error up in those cases rather than ignoring it. Closes https://github.com/facebook/rocksdb/pull/3404 Differential Revision: D6879954 Pulled By: anand1976 fbshipit-source-id: 2ef88b554be5286b0a8bad7384ba17a105395bdb | 05 February 2018, 21:59:34 UTC |
| c340184 | Agam Brahma | 04 February 2018, 02:25:28 UTC | Minor typo in comment (s/pro/pri) Summary: Closes https://github.com/facebook/rocksdb/pull/3460 Differential Revision: D6895365 Pulled By: miasantreble fbshipit-source-id: 04f633d1971b1f542ac28118b738ceb0242a0228 | 04 February 2018, 02:27:14 UTC |
| cb5b8f2 | Mike Kolupaev | 03 February 2018, 05:16:26 UTC | Fix use-after-free in tailing iterator with merge operator Summary: ForwardIterator::SVCleanup() sometimes didn't pin superversion when it was supposed to. See the added test for the scenario. Here's the ASAN output of the added test without the fix (using `COMPILE_WITH_ASAN=1 make`): https://pastebin.com/9rD0Ywws Closes https://github.com/facebook/rocksdb/pull/3415 Differential Revision: D6817414 Pulled By: al13n321 fbshipit-source-id: bc80c44ea78a3a1fa885dfa448a26111f91afb24 | 03 February 2018, 05:26:28 UTC |
| cd5092e | Tamir Duberstein | 02 February 2018, 20:14:42 UTC | Suppress unused warnings Summary: - Use `__unused__` everywhere - Suppress unused warnings in Release mode + This currently affects non-MSVC builds (e.g. mingw64). Closes https://github.com/facebook/rocksdb/pull/3448 Differential Revision: D6885496 Pulled By: miasantreble fbshipit-source-id: f2f6adacec940cc3851a9eee328fafbf61aad211 | 02 February 2018, 20:27:07 UTC |
| a247617 | Ben Clay | 02 February 2018, 18:46:41 UTC | Java: Add copy constructors for various option classes Summary: Add Java-side copy constructors for: - Options - DBOptions - ColumnFamilyOptions - WriteOptions along with unit tests to assert the copy worked. NOTE: Unit tests are failing in travis but it looks like a global timeout issue. These tests pass. Closes https://github.com/facebook/rocksdb/pull/3450 Differential Revision: D6874425 Pulled By: sagar0 fbshipit-source-id: 5bde68ea5b5225e071faea2628bf8bbf10bd65ab | 02 February 2018, 18:57:28 UTC |
| 6e5b341 | Prashant D | 01 February 2018, 22:20:25 UTC | options: Fix coverity issues Summary: options/cf_options.cc: 77 memtable_insert_with_hint_prefix_extractor( CID 1396208 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR) 2. uninit_member: Non-static class member info_log_level is not initialized in this constructor nor in any functions that it calls. Closes https://github.com/facebook/rocksdb/pull/3106 Differential Revision: D6874689 Pulled By: sagar0 fbshipit-source-id: b5cd2d13915fd86d87260050f9c5d117615bbe30 | 01 February 2018, 22:27:42 UTC |
| e502839 | Jun Wu | 01 February 2018, 22:15:28 UTC | crc32: suppress -Wimplicit-fallthrough warnings Summary: Workaround a bunch of "implicit-fallthrough" compiler errors, like: ``` util/crc32c.cc:533:7: error: this statement may fall through [-Werror=implicit-fallthrough=] crc = _mm_crc32_u64(crc, *(uint64_t*)(buf + offset)); ^ util/crc32c.cc:1016:9: note: in expansion of macro ‘CRCsinglet’ CRCsinglet(crc0, next, -2 * 8); ^~~~~~~~~~ util/crc32c.cc:1017:7: note: here case 1: ``` Closes https://github.com/facebook/rocksdb/pull/3339 Reviewed By: sagar0 Differential Revision: D6874736 Pulled By: quark-zju fbshipit-source-id: eec9f3bc135e12fca336928d01711006d5c3cb16 | 01 February 2018, 22:27:42 UTC |
| ba8aa8f | Fosco Marotto | 01 February 2018, 21:54:50 UTC | Upgrade Appveyor to VS2017 Summary: Per some discussions, this will switch our Appveyor testing to use Visual Studio 2017. Closes https://github.com/facebook/rocksdb/pull/3445 Differential Revision: D6874918 Pulled By: gfosco fbshipit-source-id: c5a0032ca9f37f0d3baeae35c59d850d528c3176 | 01 February 2018, 21:57:01 UTC |
| b78ed04 | Andrew Kryczka | 01 February 2018, 17:36:01 UTC | fix ReadaheadRandomAccessFile/iterator prefetch bug Summary: `ReadaheadRandomAccessFile` is used by iterators for file reads in several cases, like in compaction when `compaction_readahead_size > 0` or `use_direct_io_for_flush_and_compaction == true`, or in user iterator when `ReadOptions::readahead_size > 0`. `ReadaheadRandomAccessFile` maintains an internal buffer for readahead data. It assumes that, if the buffer's length is less than `ReadaheadRandomAccessFile::readahead_size_`, which is fixed in the constructor, then EOF has been reached so it doesn't try reading further. Recently, d938226af405681c592f25310f41c0c933bcdb19 started calling `RandomAccessFile::Prefetch` with various lengths: 8KB, 16KB, etc. When the `RandomAccessFile` is a `ReadaheadRandomAccessFile`, it triggers the above condition and incorrectly determines EOF. If a block is partially in the readahead buffer and EOF is incorrectly decided, the result is a truncated data block. The problem is reproducible: ``` TEST_TMPDIR=/data/compaction_bench ./db_bench -benchmarks=fillrandom -write_buffer_size=1048576 -target_file_size_base=1048576 -block_size=18384 -use_direct_io_for_flush_and_compaction=true ... put error: Corruption: truncated block read from /data/compaction_bench/dbbench/000014.sst offset 20245, expected 10143 bytes, got 8427 ``` Closes https://github.com/facebook/rocksdb/pull/3454 Differential Revision: D6869405 Pulled By: ajkr fbshipit-source-id: 87001c299e7600a37c0dcccbd0368e0954c929cf | 01 February 2018, 17:42:09 UTC |
| 813719e | Maysam Yabandeh | 01 February 2018, 02:45:49 UTC | WritePrepared Txn: Duplicate Keys, Memtable part Summary: Currently DB does not accept duplicate keys (keys with the same user key and the same sequence number). If Memtable returns false when receiving such keys, we can benefit from this signal to properly increase the sequence number in the rare cases when we have a duplicate key in the write batch written to DB under WritePrepared transactions. Closes https://github.com/facebook/rocksdb/pull/3418 Differential Revision: D6822412 Pulled By: maysamyabandeh fbshipit-source-id: adea3ce5073131cd38ed52b16bea0673b1a19e77 | 01 February 2018, 02:57:07 UTC |
| e62a763 | Yi Wu | 01 February 2018, 01:59:32 UTC | Blob DB: miscellaneous changes Summary: * Expose garbage collection related options * Minor logging and counter name update * Remove unused constants. Closes https://github.com/facebook/rocksdb/pull/3451 Differential Revision: D6867077 Pulled By: yiwu-arbug fbshipit-source-id: 6c3272a9c9d78b125a0bd6b2e56d00d087cdd6c8 | 01 February 2018, 02:13:23 UTC |
| 1edac32 | Andrew Kryczka | 31 January 2018, 22:16:04 UTC | Update rocksdb.read.block.get.micros when block cache disabled Summary: Previously `ReadBlockFromFile` for data blocks was only measured when reading a block to populate block cache. This PR adds the corresponding measurements for users who disabled block cache. Closes https://github.com/facebook/rocksdb/pull/3442 Differential Revision: D6848671 Pulled By: ajkr fbshipit-source-id: bb4bbe1797fa2cc1d9a5bad44891af2b55384b41 | 31 January 2018, 22:26:52 UTC |
| 5eccf0b | Zhongyi Xie | 31 January 2018, 20:07:52 UTC | add -fno-sanitize-recover option to force exit on errors Summary: By default if ubsan detects any problem, it outputs a “runtime error:” message, and in most cases continues executing the program. In order to make test abort on errors, option `-fno-sanitize-recover` is needed. [link](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) Closes https://github.com/facebook/rocksdb/pull/3447 Differential Revision: D6854654 Pulled By: miasantreble fbshipit-source-id: c48e892b0b38307029df38a67adda0e24257e481 | 31 January 2018, 20:13:00 UTC |
| 6efa8e2 | Fosco Marotto | 31 January 2018, 20:04:52 UTC | Update endif/else behavior for unreachable code error on Windows. Summary: Per #3367 Closes https://github.com/facebook/rocksdb/pull/3389 Differential Revision: D6766126 Pulled By: gfosco fbshipit-source-id: e441a15e8aec6747c613d68f4f0621b605eb48a0 | 31 January 2018, 20:13:00 UTC |
| 5400800 | Fosco Marotto | 31 January 2018, 19:50:59 UTC | Work around VS2017 warning for unused reference Summary: For #3407 Closes https://github.com/facebook/rocksdb/pull/3425 Differential Revision: D6836900 Pulled By: gfosco fbshipit-source-id: 7bcaf7a1beeeeabb7c05584f2745e7b4a2473497 | 31 January 2018, 19:58:10 UTC |
| ab5ab36 | Andrew Kryczka | 31 January 2018, 00:41:21 UTC | fix DBTest2.ReadAmpBitmapLiveInCacheAfterDBClose file ID support check Summary: Updated the test case to handle tmpfs mounted at directories different from "/dev/shm/". Closes https://github.com/facebook/rocksdb/pull/3440 Differential Revision: D6848213 Pulled By: ajkr fbshipit-source-id: 465e9dbf0921d0930161f732db6b3766bb030589 | 31 January 2018, 00:50:42 UTC |
| ec225d2 | Maysam Yabandeh | 31 January 2018, 00:19:22 UTC | Make WithParamInterface virtual in transaction_test Summary: Without this patch, ubsan_check is currently failing with this error: ``` utilities/transactions/write_prepared_transaction_test.cc:369:63: runtime error: member call on address 0x0000051649f8 which does not point to an object of type 'WithParamInterface' 0x0000051649f8: note: object has invalid vptr ``` Tested by `COMPILE_WITH_UBSAN=1 make -j32 transaction_test` and running `./write_prepared_transaction_test --gtest_filter=TwoWriteQueues/SnapshotConcurrentAccessTest.SnapshotConcurrentAccessTest1/0` Closes https://github.com/facebook/rocksdb/pull/3444 Differential Revision: D6850087 Pulled By: maysamyabandeh fbshipit-source-id: 5b254da8504b8757f7aec8a820ad464154da1a1d | 31 January 2018, 00:26:56 UTC |
