Skip to main content

Browse the archive

swh:1:snp:5115096b921df712aeb2a08114fede57fb3331fb
sort by:
Revision Author Date Message Commit Date
d903241 sdong 05 November 2019, 18:57:56 UTC Disable error as warning 05 November 2019, 19:04:45 UTC
01475b4 sdong 05 November 2019, 17:25:25 UTC Add one more #include<functional> 05 November 2019, 19:04:44 UTC
de3fdee sdong 31 October 2019, 21:17:26 UTC Add some include<functional> 31 October 2019, 21:24:56 UTC
6afcb0d sdong 31 October 2019, 21:10:18 UTC [FB Internal] Point to the latest tool chain. 31 October 2019, 21:24:29 UTC
d6f3ff3 sdong 18 July 2017, 05:17:47 UTC [fb only] revert unintended change of USE_SSE The previuos change that use gcc-5 set USE_SSE to wrong flag by mistake. Fix it. 18 July 2017, 05:22:40 UTC
4e0ea33 sdong 17 July 2017, 22:39:37 UTC [FB Only] use gcc-5 18 July 2017, 04:55:40 UTC
58d8de1 sdong 07 September 2016, 18:03:54 UTC Upgrade to version 4.10.2 07 September 2016, 18:03:54 UTC
95ef909 sdong 19 July 2016, 18:48:00 UTC Ignore stale logs while restarting DBs Summary: Stale log files can be deleted out of order. This can happen for various reasons. One of the reason is that no data is ever inserted to a column family and we have an optimization to update its log number, but not all the old log files are cleaned up (the case shown in the unit tests added). It can also happen when we simply delete multiple log files out of order. This causes data corruption because we simply increase seqID after processing the next row and we may end up with writing data with smaller seqID than what is already flushed to memtables. In DB recovery, for the oldest files we are replaying, if there it contains no data for any column family, we ignore the sequence IDs in the file. Test Plan: Add two unit tests that fail without the fix. Reviewers: IslamAbdelRahman, igor, yiwu Reviewed By: yiwu Subscribers: hermanlee4, yoshinorim, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60891 07 September 2016, 17:57:32 UTC
1b38103 krad 26 August 2016, 21:53:34 UTC Update version to 4.10.1 26 August 2016, 21:53:34 UTC
4f1f992 krad 26 August 2016, 21:51:39 UTC Revert "Update version to 4.10.1" This reverts commit 89b20d3cce3f2aeabf16d0df7abb43db4e5f0f76. 26 August 2016, 21:51:39 UTC
89b20d3 krad 26 August 2016, 21:46:47 UTC Update version to 4.10.1 26 August 2016, 21:46:47 UTC
a31b8cb sdong 25 August 2016, 21:43:12 UTC Mitigate regression bug of options.max_successive_merges hit during DB Recovery Summary: After 1b8a2e8fdd1db0dac3cb50228065f8e7e43095f0, DB Pointer is passed to WriteBatchInternal::InsertInto() while DB recovery. This can cause deadlock if options.max_successive_merges hits. In that case DB::Get() will be called. Get() will try to acquire the DB mutex, which is already held by the DB::Open(), causing a deadlock condition. This commit mitigates the problem by not passing the DB pointer unless 2PC is allowed. Test Plan: Add a new test and run it. Reviewers: IslamAbdelRahman, andrewkr, kradhakrishnan, horuff Reviewed By: kradhakrishnan Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D62625 26 August 2016, 00:35:38 UTC
2701f5c John Alexander 20 July 2016, 22:28:04 UTC Remove %z Format Specifier and Fix Windows Build of sim_cache.cc (#1224) * Replace %zu format specifier with Windows-compatible macro 'ROCKSDB_PRIszt' * Added "port/port.h" include to sim_cache.cc for call to snprintf(). * Applied cleaner fix to windows build, reverting part of 7bedd94 28 July 2016, 23:07:21 UTC
fdc0675 John Alexander 19 July 2016, 16:44:03 UTC New Statistics to track Compression/Decompression (#1197) * Added new statistics and refactored to allow ioptions to be passed around as required to access environment and statistics pointers (and, as a convenient side effect, info_log pointer). * Prevent incrementing compression counter when compression is turned off in options. * Prevent incrementing compression counter when compression is turned off in options. * Added two more supported compression types to test code in db_test.cc * Prevent incrementing compression counter when compression is turned off in options. * Added new StatsLevel that excludes compression timing. * Fixed casting error in coding.h * Fixed CompressionStatsTest for new StatsLevel. * Removed unused variable that was breaking the Linux build 28 July 2016, 23:06:47 UTC
605cd9b Islam AbdelRahman 25 July 2016, 23:05:50 UTC Fix Statistics TickersNameMap miss match with Tickers enum Summary: TickersNameMap is not consistent with Tickers enum. this cause us to report wrong statistics and sometimes to access TickersNameMap outside it's boundary causing crashes (in Fb303 statistics) Test Plan: added new unit test Reviewers: sdong, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D61083 26 July 2016, 20:42:44 UTC
c7eb50b Yi Wu 21 July 2016, 21:32:12 UTC Fix unit test which breaks lite build Summary: Comment out assertion of number of table files from lite build. Test Plan: OPT=-DROCKSDB_LITE make check Reviewers: lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60999 21 July 2016, 21:32:49 UTC
2462ceb Yi Wu 21 July 2016, 17:10:41 UTC Fix flush not being commit while writing manifest Summary: Fix flush not being commit while writing manifest, which is a recent bug introduced by D60075. The issue: # Options.max_background_flushes > 1 # Background thread A pick up a flush job, flush, then commit to manifest. (Note that mutex is released before writing manifest.) # Background thread B pick up another flush job, flush. When it gets to `MemTableList::InstallMemtableFlushResults`, it notices another thread is commiting, so it quit. # After the first commit, thread A doesn't double check if there are more flush result need to commit, leaving the second flush uncommitted. Test Plan: run the test. Also verify the new test hit deadlock without the fix. Reviewers: sdong, igor, lightmark Reviewed By: lightmark Subscribers: andrewkr, omegaga, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60969 21 July 2016, 20:56:22 UTC
becc230 krad 21 July 2016, 17:59:26 UTC Fix version number 21 July 2016, 17:59:26 UTC
515b11f Alexander Jipa 18 July 2016, 14:37:49 UTC fixes #1210: rocksdb/java/CMakeLists.txt lacks cmake_minimum_required (#1214) specifying minimum cmake version so that no warning has to be suppressed 18 July 2016, 14:37:49 UTC
876cb8b Alexander Jipa 17 July 2016, 07:43:34 UTC fixes #1212: rocksdbjni maven build does not escape slashes in groovy script (#1213) using gmaven property so that script is not broken by embedding unescaped Windows-style path 17 July 2016, 07:43:34 UTC
21c55bd sdong 15 July 2016, 17:18:16 UTC DBTest.DynamicLevelCompressionPerLevel: Tune Threshold Summary: Each SST's file size increases after we add more table properties. Threshold in DBTest.DynamicLevelCompressionPerLevel need to adjust accordingly to avoid occasional failures. Test Plan: Run the test Reviewers: andrewkr, yiwu Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60819 15 July 2016, 23:10:09 UTC
4b95253 Yi Wu 15 July 2016, 17:41:36 UTC Refactor cache.cc Summary: Refactor cache.cc so that I can plugin clock cache (D55581). Mainly move `ShardedCache` to separate file, move `LRUHandle` back to cache.cc and rename it lru_cache.cc. Test Plan: make check -j64 Reviewers: lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59655 15 July 2016, 17:41:36 UTC
c6a8665 Igor Canadi 15 July 2016, 11:09:30 UTC Update LANGUAGE-BINDINGS.md 15 July 2016, 11:09:30 UTC
880ee36 Wanning Jiang 14 July 2016, 21:09:31 UTC ldb backup support Summary: add backup support for ldb tool, and use it to run load test for backup on two HDFS envs Test Plan: first generate some db, then compile against load test in fbcode, run load_test --db=<db path> backup --backup_env_uri=<URI of backup env> --backup_dir=<backup directory> --num_threads=<number of thread> Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60633 14 July 2016, 21:09:31 UTC
6797e6f sdong 07 July 2016, 18:13:57 UTC Avoid updating memtable allocated bytes if write_buffer_size is not set Summary: If options.write_buffer_size is not set, nor options.write_buffer_manager, no need to update the bytes allocated counter in MemTableAllocator, which is expensive in parallel memtable insert case. Remove it can improve parallel memtable insert throughput by 10% with write batch size 128. Test Plan: Run benchmarks TEST_TMPDIR=/dev/shm/ ./db_bench --benchmarks=fillrandom -disable_auto_compactions -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -num=10000000 --writes=1000000 -max_background_flushes=16 -max_write_buffer_number=16 --threads=32 --batch_size=128 -allow_concurrent_memtable_write -enable_write_thread_adaptive_yield The throughput grows 10% with the benchmark. Reviewers: andrewkr, yiwu, IslamAbdelRahman, igor, ngbronson Reviewed By: ngbronson Subscribers: ngbronson, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60465 14 July 2016, 02:33:57 UTC
dda6c72 Aaron Gao 14 July 2016, 00:59:25 UTC Add DestroyColumnFamilyHandle(ColumnFamilyHandle**) to db.h Summary: add DestroyColumnFamilyHandle(ColumnFamilyHandle**) to close column family instead of deleting cfh* User should call this to close a cf and then we can detect the deletion in this function. Test Plan: make all check -j64 Reviewers: andrewkr, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60765 14 July 2016, 00:59:25 UTC
56222f5 Andrew Kryczka 13 July 2016, 22:36:22 UTC Avoid FileMetaData copy Summary: as titled Test Plan: unit tests Reviewers: sdong, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60597 13 July 2016, 22:36:22 UTC
15b7a4a Jay Edgar 08 July 2016, 17:52:25 UTC Fixed output size and removed unneeded loop Summary: In Zlib_Compress and BZip2_Compress the calculation for size was slightly off when using compression_foramt_version 2 (which includes the decompressed size in the output). Also there were unnecessary loops around the deflate/BZ2_bzCompress calls. In Zlib_Compress there was also a possible exit from the function after calling deflateInit2 that didn't call deflateEnd. Test Plan: Standard tests Reviewers: sdong, IslamAbdelRahman, igor Reviewed By: igor Subscribers: sdong, IslamAbdelRahman, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60537 13 July 2016, 16:10:06 UTC
6ea41f8 Yi Wu 12 July 2016, 22:30:38 UTC Fix deadlock when trying update options when write stalls Summary: When write stalls because of auto compaction is disabled, or stop write trigger is reached, user may change these two options to unblock writes. Unfortunately we had issue where the write thread will block the attempt to persist the options, thus creating a deadlock. This diff fix the issue and add two test cases to detect such deadlock. Test Plan: Run unit tests. Also, revert db_impl.cc to master (but don't revert `DBImpl::BackgroundCompaction:Finish` sync point) and run db_options_test. Both tests should hit deadlock. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60627 12 July 2016, 22:30:38 UTC
efd013d Jay Edgar 13 June 2016, 16:57:43 UTC Miscellaneous performance improvements Summary: I was investigating performance issues in the SstFileWriter and found all of the following: - The SstFileWriter::Add() function created a local InternalKey every time it was called generating a allocation and free each time. Changed to have an InternalKey member variable that can be reset with the new InternalKey::Set() function. - In SstFileWriter::Add() the smallest_key and largest_key values were assigned the result of a ToString() call, but it is simpler to just assign them directly from the user's key. - The Slice class had no move constructor so each time one was returned from a function a new one had to be allocated, the old data copied to the new, and the old one was freed. I added the move constructor which also required a copy constructor and assignment operator. - The BlockBuilder::CurrentSizeEstimate() function calculates the current estimate size, but was being called 2 or 3 times for each key added. I changed the class to maintain a running estimate (equal to the original calculation) so that the function can return an already calculated value. - The code in BlockBuilder::Add() that calculated the shared bytes between the last key and the new key duplicated what Slice::difference_offset does, so I replaced it with the standard function. - BlockBuilder::Add() had code to copy just the changed portion into the last key value (and asserted that it now matched the new key). It is more efficient just to copy the whole new key over. - Moved this same code up into the 'if (use_delta_encoding_)' since the last key value is only needed when delta encoding is on. - FlushBlockBySizePolicy::BlockAlmostFull calculated a standard deviation value each time it was called, but this information would only change if block_size of block_size_deviation changed, so I created a member variable to hold the value to avoid the calculation each time. - Each PutVarint??() function has a buffer and calls std::string::append(). Two or three calls in a row could share a buffer and a single call to std::string::append(). Some of these will be helpful outside of the SstFileWriter. I'm not 100% the addition of the move constructor is appropriate as I wonder why this wasn't done before - maybe because of compiler compatibility? I tried it on gcc 4.8 and 4.9. Test Plan: The changes should not affect the results so the existing tests should all still work and no new tests were added. The value of the changes was seen by manually testing the SstFileWriter class through MyRocks and adding timing code to identify problem areas. Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59607 12 July 2016, 21:15:32 UTC
e6f68fa omegaga 11 July 2016, 20:55:46 UTC Update Makefile to fix dependency Summary: In D33849 we updated Makefile to generate .d files for all .cc sources. Since we have more types of source files now, this needs to be updated so that this mechanism can work for new files. Test Plan: change a dependent .h file, re-make and see if .o file is recompiled. Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60591 11 July 2016, 20:55:46 UTC
816ae09 Aaron Gao 11 July 2016, 20:33:52 UTC fix test failure Summary: fix Rocksdb Unit Test USER_FAILURE Test Plan: make all check -j64 Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60603 11 July 2016, 20:33:52 UTC
e295da1 Andrew Kryczka 11 July 2016, 18:16:53 UTC Fix Log() doc for default level Summary: as titled Test Plan: none Reviewers: sdong, lightmark Reviewed By: lightmark Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60507 11 July 2016, 18:16:53 UTC
8e6b38d Aaron Gao 11 July 2016, 17:43:12 UTC update DB::AddFile to ingest list of sst files Summary: DB::AddFile(std::string file_path) API that allow them to ingest an SST file created using SstFileWriter We want to update this interface to be able to accept a list of files that will be ingested, DB::AddFile(std::vector<std::string> file_path_list). Test Plan: Add test case `AddExternalSstFileList` in `DBSSTTest`. To make sure: 1. files key ranges are not overlapping with each other 2. each file key range dont overlap with the DB key range 3. make sure no snapshots are held Reviewers: andrewkr, sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D58587 11 July 2016, 17:43:12 UTC
296545a Yi Wu 09 July 2016, 00:50:51 UTC Fix clang analyzer errors Summary: Fixing erros reported by clang static analyzer. * Removing some unused variables. * Adding assertions to fix false positives reported by clang analyzer. * Adding `__clang_analyzer__` macro to suppress false positive warnings. Test Plan: USE_CLANG=1 OPT=-g make analyze -j64 Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60549 09 July 2016, 00:50:51 UTC
61dbfbb Gunnar Kudrjavets 08 July 2016, 23:02:29 UTC Add release build to RocksDB per-diff/post-commit tests Summary: To make sure that we'll have additional verification for release builds, define a new category and add `make release` to per-diff/post-commit tests. This should in theory prevent the release MyRocks integration builds breaks from happening. Test Plan: - `[p]arc diff --preview` - Observe the execution in Sandcastle and make sure that release build and tests are executed. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60441 08 July 2016, 23:02:29 UTC
907f24d sdong 07 July 2016, 21:45:29 UTC Concurrent memtable inserter to update counters and flush state after all inserts Summary: In concurrent memtable insert case, updating counters in MemTable::Add() can count for 5% CPU usage. By batch all the counters and update in the end of the write batch, the CPU overheads are overhead in the use cases where more than one key is updated in one write batch. Test Plan: Write throughput increases 12% with this benchmark setting: TEST_TMPDIR=/dev/shm/ ./db_bench --benchmarks=fillrandom -disable_auto_compactions -level0_slowdown_writes_trigger=9999 -level0_stop_writes_trigger=9999 -num=10000000 --writes=1000000 -max_background_flushes=16 -max_write_buffer_number=16 --threads=64 --batch_size=128 -allow_concurrent_memtable_write -enable_write_thread_adaptive_yield Reviewers: andrewkr, IslamAbdelRahman, ngbronson, igor Reviewed By: ngbronson Subscribers: ngbronson, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60495 08 July 2016, 17:19:55 UTC
0f691c4 Wanning Jiang 08 July 2016, 00:51:36 UTC CLI option & Rename() allow overwrite Summary: this test support CLI option to select HdfsEnv/NativeHdfsEnv now. The latter one is default. add test about when Rename(src, target) should overwrite target Test Plan: existing test Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60399 08 July 2016, 00:51:36 UTC
7c19007 Aaron Gao 07 July 2016, 23:34:41 UTC delete unnessary pointer cast in beginInternalTransaction() function Summary: use of dynamic_cast<TransactionImpl*> is unnecessary and also introduce difficulty for fbrocksdb support of TransactionDB Test Plan: ./transaction_test Reviewers: sdong, IslamAbdelRahman, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60501 07 July 2016, 23:34:41 UTC
e1b3ee8 Andrew Kryczka 07 July 2016, 18:35:40 UTC Cleanup auto-roll logger flush-while-rolling test Summary: Use @omegaga's awesome feature to avoid use of callbacks for ensuring SyncPoints happen in a particular thread. Depends on D60375. Test Plan: $ ./auto_roll_logger_test Reviewers: omegaga, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, omegaga, leveldb Differential Revision: https://reviews.facebook.net/D60471 07 July 2016, 18:35:40 UTC
cd4178a omegaga 07 July 2016, 18:29:14 UTC Add a new feature to enforce a sync point only active on a thread Summary: Add markers to sync points. A marked sync point will only be active when it is on the same thread as the marker sync point. Test Plan: Write a unit test to validate. Reviewers: sdong, IslamAbdelRahman, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60375 07 July 2016, 18:29:14 UTC
b954847 Gunnar Kudrjavets 06 July 2016, 23:07:53 UTC Fix release build for MyRocks by using debug-only code only in debug builds Summary: MyRocks release integration build breaks because we treat warnings caused by unused variables as errors. Variable `edit` is only used in debug builds. Therefore we need to guard it using `#ifndef NDEBUG` check. Test Plan: - `[p]arc diff --preview` for the default validation. - Verify that release build fails before this fix and passes after applying it. Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60423 06 July 2016, 23:07:53 UTC
a00bf1b sdong 28 June 2016, 00:42:14 UTC Add More Logging to track total_log_size Summary: We saw instances where total_log_size is off the real value, but I'm not able to reproduce it. Add more logging to help debugging when it happens again. Test Plan: Run the unit test and see the logging. Reviewers: andrewkr, yhchiang, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60081 06 July 2016, 21:29:18 UTC
01f77cb sdong 21 June 2016, 01:01:03 UTC Update USER.md to include more services at Facebook. Test Plan: Not needed. Reviewers: igor, lgalanis, andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60417 06 July 2016, 21:27:08 UTC
eb53c05 Zun Wang 06 July 2016, 19:54:01 UTC Add comment for GetBackupInfo about returned BackupInfos order Summary: #title Test Plan: n/a Reviewers: uddipta, ldemailly, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60393 06 July 2016, 19:54:01 UTC
32df973 sdong 21 June 2016, 01:01:03 UTC Add options.write_buffer_manager: control total memtable size across DB instances Summary: Add option write_buffer_manager to help users control total memory spent on memtables across multiple DB instances. Test Plan: Add a new unit test. Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: adela, benj, sumeet, muthu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59925 06 July 2016, 01:11:25 UTC
5aaef91 Aaron Gao 06 July 2016, 01:09:59 UTC group multiple batch of flush into one manifest file (one call to LogAndApply) Summary: Currently, if several flush outputs are committed together, we issue each manifest write per batch (1 batch = 1 flush = 1 sst file = 1+ continuous memtables). Each manifest write requires one fsync and one fsync to parent directory. In some cases, it becomes the bottleneck of write. We should batch them and write in one manifest write when possible. Test Plan: ` ./db_bench -benchmarks="fillseq" -max_write_buffer_number=16 -max_background_flushes=16 -disable_auto_compactions=true -min_write_buffer_number_to_merge=1 -write_buffer_size=65536 -level0_stop_writes_trigger=10000 -level0_slowdown_writes_trigger=10000` **Before** ``` Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 4.9 Date: Fri Jul 1 15:38:17 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 166.277 micros/op 6014 ops/sec; 0.7 MB/s ``` **After** ``` Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags RocksDB: version 4.9 Date: Fri Jul 1 15:35:05 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Write rate: 0 bytes/second Compression: Snappy Memtablerep: skip_list Perf Level: 1 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags DB path: [/tmp/rocksdbtest-112628/dbbench] fillseq : 52.328 micros/op 19110 ops/sec; 2.1 MB/s ``` Reviewers: andrewkr, IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: igor, andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60075 06 July 2016, 01:09:59 UTC
a45ee83 omegaga 05 July 2016, 18:57:14 UTC Fix a bug that accesses invalid address in iterator cleanup function Summary: Reported in T11889874. When registering the cleanup function we should copy the option so that we can still access it if ReadOptions is deleted. Test Plan: Add a unit test to reproduce this bug. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60087 05 July 2016, 18:57:14 UTC
38fae9e Lucas Qiu 05 July 2016, 05:58:35 UTC fix typos in HISTORY.md (#1192) 05 July 2016, 05:58:35 UTC
1a11c93 Andrew Kryczka 04 July 2016, 22:42:33 UTC Disable some persistent cache tests on linux/clang 04 July 2016, 22:42:33 UTC
9b5adea Islam AbdelRahman 02 July 2016, 08:56:57 UTC Add More Logging to track total_log_size Summary: We saw instances where total_log_size is off the real value, but I'm not able to reproduce it. Add more logging to help debugging when it happens again. Test Plan: Run the unit test and see the logging. Reviewers: andrewkr, yhchiang, igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60081 02 July 2016, 08:56:57 UTC
95d96ee Wanning Jiang 01 July 2016, 22:28:08 UTC remove LockFile Summary: LockFile is unnecessary in unit test Test Plan: env_basic_test.cc Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60285 01 July 2016, 22:28:08 UTC
ff45d1b Wanning Jiang 01 July 2016, 18:36:33 UTC if read only backup engine can't find meta dirs, return NotFound() instead of IOError() Summary: Read only backup engine return NotFound() on missing meta dir (for e2e test) Test Plan: backupable_db_test Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60273 01 July 2016, 18:36:33 UTC
cb2476a Aaron Gao 01 July 2016, 07:16:29 UTC fix rate limiter to avoid starvation Summary: The current implementation of rate limiter has the possibility to introduce resource starvation when change its limit. This diff aims to fix this problem by consuming request bytes partially. Test Plan: ``` ./rate_limiter_test [==========] Running 4 tests from 1 test case. [----------] Global test environment set-up. [----------] 4 tests from RateLimiterTest [ RUN ] RateLimiterTest.OverflowRate [ OK ] RateLimiterTest.OverflowRate (0 ms) [ RUN ] RateLimiterTest.StartStop [ OK ] RateLimiterTest.StartStop (0 ms) [ RUN ] RateLimiterTest.Rate request size [1 - 1023], limit 10 KB/sec, actual rate: 10.355712 KB/sec, elapsed 2.00 seconds request size [1 - 1023], limit 20 KB/sec, actual rate: 19.136564 KB/sec, elapsed 2.00 seconds request size [1 - 2047], limit 20 KB/sec, actual rate: 20.783976 KB/sec, elapsed 2.10 seconds request size [1 - 2047], limit 40 KB/sec, actual rate: 39.308144 KB/sec, elapsed 2.10 seconds request size [1 - 4095], limit 40 KB/sec, actual rate: 40.318349 KB/sec, elapsed 2.20 seconds request size [1 - 4095], limit 80 KB/sec, actual rate: 79.667396 KB/sec, elapsed 2.20 seconds request size [1 - 8191], limit 80 KB/sec, actual rate: 81.807158 KB/sec, elapsed 2.30 seconds request size [1 - 8191], limit 160 KB/sec, actual rate: 160.659761 KB/sec, elapsed 2.20 seconds request size [1 - 16383], limit 160 KB/sec, actual rate: 160.700990 KB/sec, elapsed 3.00 seconds request size [1 - 16383], limit 320 KB/sec, actual rate: 317.639481 KB/sec, elapsed 2.50 seconds [ OK ] RateLimiterTest.Rate (22618 ms) [ RUN ] RateLimiterTest.LimitChangeTest [COMPLETE] request size 10 KB, new limit 20KB/sec, refill period 1000 ms [COMPLETE] request size 10 KB, new limit 5KB/sec, refill period 1000 ms [COMPLETE] request size 20 KB, new limit 40KB/sec, refill period 1000 ms [COMPLETE] request size 20 KB, new limit 10KB/sec, refill period 1000 ms [COMPLETE] request size 40 KB, new limit 80KB/sec, refill period 1000 ms [COMPLETE] request size 40 KB, new limit 20KB/sec, refill period 1000 ms [COMPLETE] request size 80 KB, new limit 160KB/sec, refill period 1000 ms [COMPLETE] request size 80 KB, new limit 40KB/sec, refill period 1000 ms [COMPLETE] request size 160 KB, new limit 320KB/sec, refill period 1000 ms [COMPLETE] request size 160 KB, new limit 80KB/sec, refill period 1000 ms [ OK ] RateLimiterTest.LimitChangeTest (5002 ms) [----------] 4 tests from RateLimiterTest (27620 ms total) [----------] Global test environment tear-down [==========] 4 tests from 1 test case ran. (27621 ms total) [ PASSED ] 4 tests. ``` Reviewers: sdong, IslamAbdelRahman, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60207 01 July 2016, 07:16:29 UTC
6b71676 Andrew Kryczka 01 July 2016, 01:34:29 UTC Run env_basic_test on Env::Default Summary: Previously we couldn't run env_basic_test on Env::Default (PosixEnv on our platforms) since GetChildren*() behavior was inconsistent with our other Envs. We can normalize the output of GetChildren*() such that these test cases work on PosixEnv too. Test Plan: ran env_basic_test Reviewers: wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59943 01 July 2016, 01:34:29 UTC
9eb0b53 Andrew Kryczka 01 July 2016, 01:33:49 UTC Move env_basic_test cleanup to TearDown Summary: move cleanup to TearDown and handle directories, so cleanup will happen even if a test fails in the middle. Test Plan: ./env_basic_test Reviewers: wanning Reviewed By: wanning Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60243 01 July 2016, 01:33:49 UTC
1fe3bf8 Islam AbdelRahman 30 June 2016, 21:34:50 UTC Re-enable linux on travis Summary: Travis clang issue on linux is fixed now, re-enable linux on travis Test Plan: https://travis-ci.org/facebook/rocksdb/builds/141098948 Reviewers: yiwu, sdong, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60189 30 June 2016, 21:34:50 UTC
4369279 John Alexander 30 June 2016, 01:44:22 UTC Fixed Minor Bug on Windows Build and db_bench_tool.cc (#1189) * Fixed Windows build error in CMakeLists.txt and perf_level error in db_bench_tool.cc * Changed hard-coded perf levels in db_bench_tool.cc to enum values from perf_level.h * Replaced remaining FLAGS_perf_level > 0 30 June 2016, 01:44:22 UTC
95c1924 Wanning Jiang 29 June 2016, 22:33:27 UTC writable file close before reset Summary: during backup, writable file should call close() before reset() Test Plan: backupable_db_test.cc Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60195 29 June 2016, 22:33:27 UTC
197b832 dhruba borthakur 29 June 2016, 21:36:25 UTC Update USERS.md 29 June 2016, 21:36:25 UTC
bdb1d19 Gunnar Kudrjavets 29 June 2016, 17:49:25 UTC Fix UBSan build break caused by variable not initialized Summary: UBSan is unhappy because `cfd` is not initialized. This breaks UBSan build which in turn breaks MyRocks continuous integration with RocksDB which in turns makes me unhappy :-) Fix this. Test Plan: - `[p]arc diff --preview` + Sandcastle. - Verify that `COMPILE_WITH_UBSAN=1 OPT=-g make J=1 ubsan_check` gets past the break. Reviewers: andrewkr, hermanlee4, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60117 29 June 2016, 17:49:25 UTC
b726bf5 Willem Jan Withagen 29 June 2016, 14:35:17 UTC FreeBSD does not have std::to_string (#1190) Submitted-by: Willem Jan Withagen <wjw@digiware.nl> 29 June 2016, 14:35:17 UTC
faa7eb3 Yueh-Hsuan Chiang 28 June 2016, 21:10:24 UTC Improve regression_test.sh Summary: This diff makes the following improvement in regression_test.sh: 1. Add NUM_OPS and DELETE_TEST_PATH to regression_test.sh: * NUM_OPS: The number of operations that will be issued in EACH thread. Default: $NUM_KEYS / $NUM_THREADS * DELETE_TEST_PATH: If true, then the test directory will be deleted after the script ends. Default: 0 2. Add more information in SUMMARY.csv 3. Fix a bug in regression_test.sh where each thread in fillseq will all issue $NUM_KEYS writes. 4. Add --deletes in db_bench, which allows us to control the number of deletes instead of must using FLAGS_num. Test Plan: run regression test with and without DELETE_TEST_PATH and NUM_OPS Reviewers: yiwu, sdong, IslamAbdelRahman, gunnarku Reviewed By: gunnarku Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D60039 28 June 2016, 21:10:24 UTC
c4cef07 sdong 24 June 2016, 22:54:26 UTC Update DBTestUniversalCompaction.UniversalCompactionSingleSortedRun to use max_size_amplification_percent = 0 Summary: With max_size_amplification_percent = 0 to make sure that DBTestUniversalCompaction.UniversalCompactionSingleSortedRun tests the configuration to compact to one single sorted run. Test Plan: Run all existing tests Reviewers: yhchiang, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D60021 27 June 2016, 22:19:27 UTC
892e9d3 Reid Horuff 23 June 2016, 19:20:48 UTC make transaction WriteOptions modifiable 27 June 2016, 19:53:30 UTC
4f2b094 charsyam 25 June 2016, 07:29:40 UTC fix simple typos (#1183) 25 June 2016, 07:29:40 UTC
3b7ed67 Andrew Kryczka 24 June 2016, 23:29:43 UTC ColumnFamilyOptions API [CF + RepairDB part 3/3] Summary: Overload RepairDB to take vector-of-ColumnFamilyDescriptor, which tells us CF name + options. Also takes a ColumnFamilyOptions for unspecified column families encountered during the repair. One potentially confusing thing is that we store options in the constructor and don't invoke AddColumnFamily() until discovering the CF in ScanTable. This is because we don't know the CF ID until we find a table belonging to that CF. Depends on D59781. Test Plan: $ ./repair_test Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59853 24 June 2016, 23:29:43 UTC
56ac686 Andrew Kryczka 24 June 2016, 20:12:13 UTC Detect column family from properties [CF + RepairDB part 2/3] Summary: This diff uses the CF ID and CF name properties in the SST file to associate recovered data with the proper column family. Depends on D59775. - In ScanTable(), create column families in VersionSet each time a new one is discovered (via reading SST file properties) - In ConvertLogToTable(), dump an SST file for every column family with data in the WAL - In AddTables(), make a VersionEdit per-column family that adds all of that CF's tables Test Plan: $ ./repair_test Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59781 24 June 2016, 20:12:13 UTC
3fc713e Wanning Jiang 24 June 2016, 19:31:58 UTC delete 2nd level children for default env Summary: ensure no 2nd level children under test_dir_ Test Plan: env_basic_test on 4 envs Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59979 24 June 2016, 19:31:58 UTC
343507a Andrew Kryczka 24 June 2016, 18:19:40 UTC Refactor to use VersionSet [CF + RepairDB part 1/3] Summary: To support column families, it is easiest to use VersionSet to manage our column families (if we don't have Versions then ColumnFamilyData always behaves as a dummy column family). This diff only refactors the existing repair logic to use VersionSet; the next two parts will add support for multiple column families. Test Plan: $ ./repair_test Reviewers: yhchiang, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59775 24 June 2016, 18:19:40 UTC
aa432be Gunnar Kudrjavets 23 June 2016, 18:52:31 UTC Workarounds for continuous build implementation 24 June 2016, 02:38:29 UTC
8cd9f04 Gunnar Kudrjavets 23 June 2016, 19:08:44 UTC Test change to verify new commit detection 23 June 2016, 19:08:44 UTC
8a4ee7e Gunnar Kudrjavets 23 June 2016, 18:52:31 UTC Trivial change to test cont. build 23 June 2016, 18:53:38 UTC
af6248d Islam AbdelRahman 23 June 2016, 01:28:38 UTC Fix max_bytes_for_level_base comment Summary: Fix max_bytes_for_level_base comment Test Plan: none Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59955 23 June 2016, 01:28:38 UTC
0d7b261 Wanning Jiang 22 June 2016, 23:16:21 UTC add tests to env_basic_test.cc Summary: test NativeHdfsEnv Test Plan: env_basic_test.cc Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59889 22 June 2016, 23:16:21 UTC
6576fa0 Gunnar Kudrjavets 22 June 2016, 18:05:52 UTC Fix minor typos and PHP source file name used to trigger the builds Summary: Fix two minor typos and update the file name which is used to trigger the runs in case new changes have been committed. Test Plan: - Testing with a private Sandcastle instance. Reviewers: sdong, mung Reviewed By: mung Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59919 22 June 2016, 18:05:52 UTC
c4e19b7 omegaga 22 June 2016, 01:41:23 UTC Add a read option to enable background purge when cleaning up iterators Summary: Add a read option `background_purge_on_iterator_cleanup` to avoid deleting files in foreground when destroying iterators. Instead, a job is scheduled in high priority queue and would be executed in a separate background thread. Test Plan: Add a variant of PurgeObsoleteFileTest. Turn on background purge option in the new test, and use sleeping task to ensure files are deleted in background. Reviewers: IslamAbdelRahman, sdong Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59499 22 June 2016, 01:41:23 UTC
fa813f7 Islam AbdelRahman 22 June 2016, 00:57:59 UTC Update DB::AddFile() to ingest the file to the lowest possible level Summary: DB::AddFile() right now always add the ingested file to L0 update the logic to add the file to the lowest possible level Test Plan: unit tests Reviewers: jkedgar, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D59637 22 June 2016, 00:57:59 UTC
d6b79e2 Islam AbdelRahman 22 June 2016, 00:30:21 UTC Remove filter_deletes from crash_test Summary: filter_deletes option was removed, remove it from crash_test to fix it Test Plan: make crash_test Reviewers: yhchiang, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59901 22 June 2016, 00:30:21 UTC
a52e4d7 Gunnar Kudrjavets 21 June 2016, 18:38:54 UTC Framework for enabling continuous RocksDB build and tests Summary: The main PHP code churn is caused by extracting the common code from `FacebookArcanistConfiguration.php` and `FacebookOldArcanistConfiguration.php` into `RocksDBCommonDeterminator.php`. This is necessary both for reducing the duplication of code and making sure that we can execute the common core logic separately from continuous runs. The main logic in `RocksDBCommonDeterminator.php` remains quite the same with the exception of some things: - Adding separation between the cases when a diff is submitted //vs.// when the code is triggered from a continuous run. There are certain actions which we should do in a case of diff only. - Adding reporting - now the person who authored the diff will receive e-mail notifications if any of the jobs have failed. - Enabling assertions and making sure that we'll terminate on failure. This is an internal code used by competent engineers, so instead of `if (!condition) { echo "Something"; exit(1); }` for every invariant I think that `assert(condition)` provides better readability with the same behavior. Especially taking into account that we're talking about things which shouldn't ever happen. Enabling this entire process will be triggered internally and will be a subject of a separate code review. We should discuss the details of triggering continuous RocksDB build and tests on that diff. Test Plan: - Make sure that `[p]arc diff` scenario isn't broken by verifying that tests validating this diff will pass. - Private testing of triggering the continuous build script. - Once the changes will land then author an internal job which will use the script and verify its validity. Reviewers: sdong, yhchiang, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59811 21 June 2016, 18:38:54 UTC
f9bd667 Islam AbdelRahman 21 June 2016, 17:47:57 UTC Makefile warning for invalid paths in make_config.mk Summary: Update Makefile to show warnings when we have invalid paths in our make_config.mk file sample output ``` $ make static_lib -j64 Makefile:150: Warning: /mnt/gvfs/third-party2/libgcc/53e0eac8911888a105aa98b9a35fe61cf1d8b278/4.9.x/gcc-4.9-glibc-2.20/024dbc3/libs dont exist Makefile:150: Warning: /mnt/gvfs/third-party2/llvm-fb/b91de48a4974ec839946d824402b098d43454cef/stable/centos6-native/7aaccbe/../../src/clang/tools/scan-build/scan-build dont exist GEN util/build_version.cc ``` Test Plan: check that warning is printed visually Reviewers: sdong, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59523 21 June 2016, 17:47:57 UTC
88a2776 Islam AbdelRahman 20 June 2016, 18:26:25 UTC Update SstFileWriter to use bottommost_compression if avaliable Summary: SstFileWriter ignore Options::bottommost_compression, update it to use bottommost_compression if available Test Plan: make check -j64 verified used compression using ./sst_dump Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D59841 20 June 2016, 18:26:25 UTC
e87d5df goroutine 20 June 2016, 16:46:46 UTC TiKV use-case (#1172) 20 June 2016, 16:46:46 UTC
7b79238 sdong 08 June 2016, 00:25:06 UTC Deprectate filter_deletes Summary: filter_deltes is not a frequently used feature. Remove it. Test Plan: Run all test suites. Reviewers: igor, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59427 17 June 2016, 17:30:47 UTC
4939fc3 sdong 16 June 2016, 21:05:34 UTC Bulk load mode shouldn't stop ingest Summary: We introduced default slow down and stop condition, but didn't reset it in bulk load mode. Fix it. Test Plan: N/A Reviewers: igor, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59757 17 June 2016, 16:58:50 UTC
3a2bccc Yueh-Hsuan Chiang 17 June 2016, 11:52:53 UTC Fixed a crash bug that incorrectly parse deprecated options in options_helper Summary: Fixed a crash bug that incorrectly parse deprecated options in options_helper Test Plan: run db_bench with an old options file with memtable_prefix_bloom_probes ./db_bench --options_file=AN_OLD_OPTIONS_FILE --num=100 --benchmarks=fillseq Reviewers: sdong, IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59787 17 June 2016, 11:53:30 UTC
cf8adc9 Yueh-Hsuan Chiang 17 June 2016, 11:36:52 UTC Allow arcanist_util to work with both new and old arc versions Summary: Allow arcanist_util to work with both new and old arc versions. The diff is based on Adam Retter's pull request https://github.com/facebook/rocksdb/pull/1168 Many thanks to Adam to initiate this work Test Plan: run arc lint and arc diff using different arc versions Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, andrewkr, adamretter, igor Reviewed By: adamretter Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59697 17 June 2016, 11:36:52 UTC
30a24f2 Islam AbdelRahman 16 June 2016, 23:21:41 UTC Add InternalStats and logging for AddFile() Summary: We dont report the bytes that we ingested from AddFile which make the write amplification numbers incorrect Update InternalStats and add logging for AddFile() Test Plan: Make sure the code compile and existing tests pass Reviewers: lightmark, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59763 16 June 2016, 23:21:41 UTC
d26a848 Islam AbdelRahman 15 June 2016, 19:08:46 UTC Temporarily disable travis on linux Summary: Travis is broken on linux, disable it temporarily Test Plan: https://travis-ci.org/facebook/rocksdb/builds/137721712 Reviewers: yhchiang, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59673 15 June 2016, 19:08:46 UTC
249e796 sdong 13 June 2016, 22:00:02 UTC Fix Flaky DBCompactionTest.SkipStatsUpdateTest Summary: DBCompactionTest.SkipStatsUpdateTest sometimes fails. I don't see any verification related to the deletes issued. Remove them to avoid the uncertainty. Test Plan: Run the test. Reviewers: IslamAbdelRahman, andrewkr, yhchiang Reviewed By: yhchiang Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59613 15 June 2016, 19:00:51 UTC
162c917 krad 14 June 2016, 23:13:08 UTC Make sandcastle access secure Summary: Making access credentials for sandcastle configurable Test Plan: Submit for review Reviewers: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D59649 15 June 2016, 17:55:29 UTC
8366e10 Islam AbdelRahman 15 June 2016, 07:24:33 UTC Fix clang build Summary: Fix clang build Test Plan: USE_CLANG=1 make check -j64 Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59667 15 June 2016, 07:24:33 UTC
0babce5 sdong 13 June 2016, 22:34:26 UTC Move away from enum char value -1 Summary: char is not signed in some platforms. Having negative values confuse those compilers. Test Plan: Run all existing tests. Reviewers: andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59619 15 June 2016, 00:07:34 UTC
812dbfb Islam AbdelRahman 14 June 2016, 19:27:46 UTC Optimize BlockIter::Prev() by caching decoded entries Summary: Right now the way we do BlockIter::Prev() is like this - Go to the beginning of the restart interval - Keep moving forward (and decoding keys using ParseNextKey()) until we reach the desired key This can be optimized by caching the decoded entries in the first pass and reusing them in consecutive BlockIter::Prev() calls Before caching ``` DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readreverse" --db="/dev/shm/bench_prev_opt/" --use_existing_db --disable_auto_compactions DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.413 micros/op 2423972 ops/sec; 268.2 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.414 micros/op 2413867 ops/sec; 267.0 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.410 micros/op 2440881 ops/sec; 270.0 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.414 micros/op 2417298 ops/sec; 267.4 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.413 micros/op 2421682 ops/sec; 267.9 MB/s ``` After caching ``` DEBUG_LEVEL=0 make db_bench -j64 && ./db_bench --benchmarks="readreverse" --db="/dev/shm/bench_prev_opt/" --use_existing_db --disable_auto_compactions DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.324 micros/op 3088955 ops/sec; 341.7 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.335 micros/op 2980999 ops/sec; 329.8 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.341 micros/op 2929681 ops/sec; 324.1 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.344 micros/op 2908490 ops/sec; 321.8 MB/s DB path: [/dev/shm/bench_prev_opt/] readreverse : 0.338 micros/op 2958404 ops/sec; 327.3 MB/s ``` Test Plan: COMPILE_WITH_ASAN=1 make check -j64 Reviewers: andrewkr, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D59463 14 June 2016, 19:27:46 UTC
550bf89 krad 14 June 2016, 19:20:38 UTC Minor bug fix with log name 14 June 2016, 19:20:58 UTC
886af59 Islam AbdelRahman 14 June 2016, 19:18:32 UTC Fix examples/Makefile jemalloc error Summary: compile and link with jemalloc in examples binaries Test Plan: cd examples make -j64 Reviewers: andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D59457 14 June 2016, 19:18:32 UTC
e3b1e3d Adam Retter 14 June 2016, 16:51:14 UTC Expose save points in Java WriteBatch and WBWI (#1092) * Java API - Expose SetSavePoint and RollbackToRestorePoint for WriteBatch and WriteBatchWithIndex * Minor cleanup 14 June 2016, 16:51:14 UTC
f5177c7 Islam AbdelRahman 13 June 2016, 23:22:14 UTC Remove wasteful instrumentation in FullMerge (stacked on D59577) Summary: [ This diff is stacked on top of D59577 ] We keep calling timer.ElapsedNanos() on every call to MergeOperator::FullMerge even when statistics are disabled, this is wasteful. I run the readseq benchmark on a DB containing 100K merge operands for 100K keys (1 operand per key) with 1GB block cache I see slight performance improvment Original results ``` $ ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=100000 --num=100000 --db="/dev/shm/100K_merge_compacted/" --cache_size=1073741824 --use_existing_db --disable_auto_compactions ------------------------------------------------ DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.498 micros/op 2006597 ops/sec; 222.0 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.295 micros/op 3393627 ops/sec; 375.4 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.285 micros/op 3511155 ops/sec; 388.4 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.286 micros/op 3500470 ops/sec; 387.2 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.283 micros/op 3530751 ops/sec; 390.6 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.289 micros/op 3464811 ops/sec; 383.3 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.277 micros/op 3612814 ops/sec; 399.7 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.283 micros/op 3539640 ops/sec; 391.6 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.285 micros/op 3503766 ops/sec; 387.6 MB/s ``` After patch ``` $ ./db_bench --benchmarks="readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq,readseq" --merge_operator="max" --merge_keys=100000 --num=100000 --db="/dev/shm/100K_merge_compacted/" --cache_size=1073741824 --use_existing_db --disable_auto_compactions ------------------------------------------------ DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.476 micros/op 2100119 ops/sec; 232.3 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.278 micros/op 3600887 ops/sec; 398.4 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.275 micros/op 3636698 ops/sec; 402.3 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.271 micros/op 3691661 ops/sec; 408.4 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.273 micros/op 3661534 ops/sec; 405.1 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.276 micros/op 3627106 ops/sec; 401.3 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.272 micros/op 3682635 ops/sec; 407.4 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.266 micros/op 3758331 ops/sec; 415.8 MB/s DB path: [/dev/shm/100K_merge_compacted/] readseq : 0.266 micros/op 3761907 ops/sec; 416.2 MB/s ``` Test Plan: make check -j64 Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59583 13 June 2016, 23:22:14 UTC
97fd2a6 Gunnar Kudrjavets 13 June 2016, 23:20:15 UTC Remove dead Jenkins code and support `arc diff --preview` in RocksDB Summary: Two changes here: - Remove dead Jenkins related code which is no longer relevant. - Support `arc diff --preview`. Currently it doesn't work because a step which applies a diff assumes that a revision has been created. Which in case of `--preview` isn't. Therefore diff can't be applied and validation fails. Solution is to use `--nocommit` because for validation purposes performing a commit isn't necessary. Test Plan: - Current changes are submitted using `arc diff --preview`. - All the pre-commit verification tests passed. Reviewers: kradhakrishnan, sdong Reviewed By: sdong Subscribers: leveldb, andrewkr, jtolmer, dhruba Differential Revision: https://reviews.facebook.net/D59571 13 June 2016, 23:20:15 UTC
7c919de Islam AbdelRahman 13 June 2016, 23:17:26 UTC Reuse TimedFullMerge instead of FullMerge + instrumentation Summary: We have alot of code duplication whenever we call FullMerge we keep duplicating the instrumentation and statistics code This is a simple diff to refactor the code to use TimedFullMerge instead of FullMerge Test Plan: COMPILE_WITH_ASAN=1 make check -j64 Reviewers: andrewkr, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D59577 13 June 2016, 23:17:26 UTC
back to top