Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
17 June 2024, 23:04:45 UTC
sort by:
Revision Author Date Message Commit Date
8d7926a Andrew Kryczka 07 May 2016, 00:29:53 UTC Increment patch number for 4.6.1 07 May 2016, 00:29:53 UTC
9ed8041 Andrew Kryczka 26 April 2016, 19:33:30 UTC Retrieve file size from proper Env Summary: When db_env_ != backup_env_, InsertPathnameToSizeBytes() would use the wrong Env during backup creation. This happened because this function used backup_env_ instead of db_env_ to get WAL/data file sizes. This diff adds an argument to InsertPathnameToSizeBytes() indicating which Env to use. Test Plan: ran @anirbanb's BackupTestTool Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57159 07 May 2016, 00:28:15 UTC
15ec7d4 Andrew Kryczka 12 March 2016, 21:51:57 UTC Update change log for 4.6 release Summary: as titled Test Plan: N/A Reviewers: sdong, kradhakrishnan, anthony, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55323 14 March 2016, 18:11:09 UTC
0612f47 Islam AbdelRahman 12 March 2016, 03:01:12 UTC Aggregate hot Iterator counters in LocalStatistics (DBIter::Next perf regression) Summary: This patch bump the counters in the frequent code path DBIter::Next() / DBIter::Prev() in a local data members and send them to Statistics when the iterator is destroyed A better solution will be to have thread_local implementation for Statistics New performance ``` readseq : 0.035 micros/op 28597881 ops/sec; 3163.7 MB/s 1,851,568,819 stalled-cycles-frontend # 31.29% frontend cycles idle [49.86%] 884,929,823 stalled-cycles-backend # 14.95% backend cycles idle [50.21%] readreverse : 0.071 micros/op 14077393 ops/sec; 1557.3 MB/s 3,239,575,993 stalled-cycles-frontend # 27.36% frontend cycles idle [49.96%] 1,558,253,983 stalled-cycles-backend # 13.16% backend cycles idle [50.14%] ``` Existing performance ``` readreverse : 0.174 micros/op 5732342 ops/sec; 634.1 MB/s 20,570,209,389 stalled-cycles-frontend # 70.71% frontend cycles idle [50.01%] 18,422,816,837 stalled-cycles-backend # 63.33% backend cycles idle [50.04%] readseq : 0.119 micros/op 8400537 ops/sec; 929.3 MB/s 15,634,225,844 stalled-cycles-frontend # 79.07% frontend cycles idle [49.96%] 14,227,427,453 stalled-cycles-backend # 71.95% backend cycles idle [50.09%] ``` Test Plan: unit tests Reviewers: yhchiang, sdong, igor Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55107 14 March 2016, 18:11:09 UTC
187cb23 Andrew Kryczka 11 March 2016, 02:16:21 UTC Cleanup stale manifests outside of full purge Summary: - Keep track of obsolete manifests in VersionSet - Updated FindObsoleteFiles() to put obsolete manifests in the JobContext for later use by PurgeObsoleteFiles() - Added test case that verifies a stale manifest is deleted by a non-full purge Test Plan: $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation Reviewers: IslamAbdelRahman, yoshinorim, sdong Reviewed By: sdong Subscribers: andrewkr, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D55269 14 March 2016, 18:11:09 UTC
848cf93 Yi Wu 11 March 2016, 01:35:19 UTC Cache to have an option to fail Cache::Insert() when full Summary: Cache to have an option to fail Cache::Insert() when full. Update call sites to check status and handle error. I totally have no idea what's correct behavior of all the call sites when they encounter error. Please let me know if you see something wrong or more unit test is needed. Test Plan: make check -j32, see tests pass. Reviewers: anthony, yhchiang, andrewkr, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54705 14 March 2016, 18:11:09 UTC
c5cfed2 Yueh-Hsuan Chiang 10 March 2016, 22:34:28 UTC Update compaction score right after CompactFiles forms a compaction Summary: This is a follow-up patch of https://reviews.facebook.net/D54891. As the information about files being compacted will also be used when making compaction decision, it is necessary to update the compaction score when a compaction plan has been made but not yet execute. This patch adds a missing call to update the compaction score in CompactFiles(). Test Plan: compact_files_test Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55227 14 March 2016, 18:11:09 UTC
aa3f02d Yueh-Hsuan Chiang 09 March 2016, 00:46:41 UTC Improve comment in compaction.h and compaction_picker.h Summary: ReleaseCompactionFiles must be called when DB mutex is held, but the documentation is mission. Test Plan: no code change Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54987 09 March 2016, 00:46:41 UTC
2200295 agiardullo 04 March 2016, 00:33:26 UTC optimistic transactions support for reinitialization Summary: Extend optimization in D53835 to optimistic transactions for completeness. Test Plan: added test Reviewers: sdong, IslamAbdelRahman, horuff, jkedgar Reviewed By: horuff Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55059 08 March 2016, 03:03:09 UTC
badd6b7 Islam AbdelRahman 07 March 2016, 23:56:16 UTC Ignore db_test2 Summary: Add db_test2 to .gitignore Test Plan: make sure db_test2 dont show in "git status" Reviewers: sdong, yhchiang, andrewkr, anthony Reviewed By: anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D55191 07 March 2016, 23:56:16 UTC
200080e agiardullo 03 March 2016, 23:36:26 UTC Improve snapshot handling for Transaction reinitialization Summary: Previously, reusing a transaction (by passing it as an argument to BeginTransaction) would not clear the transaction's snapshot. This is not a clear, well-definited behavior. Test Plan: improved test Reviewers: sdong, IslamAbdelRahman, horuff, jkedgar Reviewed By: jkedgar Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55053 07 March 2016, 21:28:11 UTC
171c8e8 Islam AbdelRahman 07 March 2016, 20:57:19 UTC Update dependencies / Fix Clang Summary: Update dependencies and fix broken clang symlink version changes ``` zstd 0.4.7 => 0.5.1 jemalloc 4.0.3 => master ``` Test Plan: make check -j64 USE_CLANG=1 make check -j64 ROCKSDB_FBCODE_BUILD_WITH_481=1 make check -j64 Reviewers: sdong, yhchiang, anthony, andrewkr Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D55179 07 March 2016, 20:57:19 UTC
294bdf9 sdong 03 March 2016, 21:18:56 UTC Change Property name from "rocksdb.current_version_number" to "rocksdb.current-super-version-number" Summary: I realized I again is wrong about the naming convention. Let me change it to the correct one. Test Plan: Run unit tests. Reviewers: IslamAbdelRahman, kradhakrishnan, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D55041 05 March 2016, 02:15:29 UTC
bf1c408 Yueh-Hsuan Chiang 05 March 2016, 00:03:31 UTC Use pure if-then check instead of assert in EraseColumnFamilyInfo Summary: Use pure if-then check instead of assert in EraseColumnFamilyInfo when the specified column family does not found in the cf_info_map_. So the second deletion will be no op instead of crash. Test Plan: existing test. Reviewers: sdong, anthony, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55023 05 March 2016, 00:03:31 UTC
a7d4eb2 Yueh-Hsuan Chiang 04 March 2016, 22:24:52 UTC Fix a bug where flush does not happen when a manual compaction is running Summary: Currently, when rocksdb tries to run manual compaction to refit data into a level, there's a ReFitLevel() process that requires no bg work is currently running. When RocksDB plans to ReFitLevel(), it will do the following: 1. pause scheduling new bg work. 2. wait until all bg work finished 3. do the ReFitLevel() 4. unpause scheduling new bg work. However, as it pause scheduling new bg work at step one and waiting for all bg work finished in step 2, RocksDB will stop flushing until all bg work is done (which could take a long time.) This patch fix this issue by changing the way ReFitLevel() pause the background work: 1. pause scheduling compaction. 2. wait until all bg work finished. 3. pause scheduling flush 4. do ReFitLevel() 5. unpause both flush and compaction. The major difference is that. We only pause scheduling compaction in step 1 and wait for all bg work finished in step 2. This prevent flush being blocked for a long time. Although there's a very rare case that ReFitLevel() might be in starvation in step 2, but it's less likely the case as flush typically finish very fast. Test Plan: existing test. Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55029 04 March 2016, 22:24:52 UTC
68189f7 Gunnar Kudrjavets 04 March 2016, 20:32:11 UTC Update benchmarks used to measure subcompaction performance Summary: After closely working with Mark, Siying, and Yueh-Hsuan this set of changes reflects the updates needed to measure RocksDB subcompaction performance in a correct manner. The essence of the benchmark is executing `fillrandom` followed by `compact` with the correct set of options for various number of subcompactions specified. Test Plan: Tested internally to verify correctness and reliability. Reviewers: sdong, yhchiang, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D55089 04 March 2016, 20:32:11 UTC
dfe96c7 Islam AbdelRahman 04 March 2016, 02:25:07 UTC Fix WriteLevel0TableForRecovery file delete protection Summary: The call to ``` CaptureCurrentFileNumberInPendingOutputs() ``` should be before ``` versions_->NewFileNumber() ``` Right now we are not actually protecting the file from being deleted Test Plan: make check Reviewers: sdong, anthony, yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D54645 04 March 2016, 02:25:07 UTC
451678c Igor Canadi 03 March 2016, 23:24:25 UTC Merge pull request #1025 from SherlockNoMad/BuildFix Fix Windows build 03 March 2016, 23:24:25 UTC
58ecd91 SherlockNoMad 03 March 2016, 23:08:24 UTC Fix Windows build 03 March 2016, 23:08:24 UTC
501927f Andrew Kryczka 03 March 2016, 21:32:20 UTC [backupable db] Remove file size embedded in name workaround Summary: Now that we get sizes efficiently, we no longer need the workaround to embed file size in filename. Test Plan: $ ./backupable_db_test Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55035 03 March 2016, 21:32:20 UTC
ef204df sdong 03 March 2016, 00:26:47 UTC Compaction always needs to be removed from level0_compactions_in_progress_ for universal compaction Summary: We always put compaction to level0_compactions_in_progress_ for universal compaction, so we should also remove it. The bug causes assert failure when running manual compaction. Test Plan: TEST_TMPDIR=/dev/shm/ ./db_bench --benchmarks=fillrandom,compact --subcompactions=16 --compaction_style=1 always fails on my host. After the fix, it doesn't fail any more. Reviewers: IslamAbdelRahman, andrewkr, kradhakrishnan, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D55017 03 March 2016, 05:23:28 UTC
e79ad9e sdong 01 March 2016, 02:38:03 UTC Add Iterator Property rocksdb.iterator.version_number Summary: We want to provide a way to detect whether an iterator is stale and needs to be recreated. Add a iterator property to return version number. Test Plan: Add two unit tests for it. Reviewers: IslamAbdelRahman, yhchiang, anthony, kradhakrishnan, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54921 03 March 2016, 00:23:59 UTC
19ea40f sdong 02 March 2016, 19:21:26 UTC Subcompaction boundary keys should not terminate after an empty level Summary: Now we skip to add boundary keys to subcompaction candidates since we see an empty level. This makes subcompaction almost disabled for universal compaction. We should consider all files instead. Test Plan: Run existing tests. Reviewers: IslamAbdelRahman, andrewkr, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D55005 02 March 2016, 23:45:07 UTC
deb08b8 Omar Sandoval 02 March 2016, 18:34:14 UTC Add parsing of missing DB options Summary: There are a few options in struct DBOptions that aren't handled by options_helper.cc. Add those missing options so they can be used by GetDBOptionsFromString() and friends. Test Plan: Updated options_test.cc, reran all tests. Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54603 02 March 2016, 18:34:14 UTC
f8e90e8 Andrew Kryczka 02 March 2016, 03:33:33 UTC Get file attributes in bulk for VerifyBackup and CreateNewBackup Summary: For VerifyBackup(), backup files can be spread across "shared/", "shared_checksum/", and "private/" subdirectories, so we have to bulk get all three. For CreateNewBackup(), we make two separate bulk calls: one for the data files and one for WAL files. There is also a new helper function, ExtendPathnameToSizeBytes(), that translates the file attributes vector to a map. I decided to leave GetChildrenFileAttributes()'s (from D53781) return type as vector to keep it consistent with GetChildren(). Depends on D53781. Test Plan: verified relevant unit tests $ ./backupable_db_test Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53919 02 March 2016, 03:33:33 UTC
12fd9b1 sdong 29 February 2016, 23:02:32 UTC Change BlockBasedTableOptions.format_version default to 2 Summary: BlockBasedTableOptions.format_version = 2 uses better encoding format. Now it's the time to make it default. Test Plan: Run all existing tests. Reviewers: igor, yhchiang, anthony, kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54879 01 March 2016, 22:28:52 UTC
4572a2d sdong 01 March 2016, 21:47:36 UTC Update current version to 4.6 Summary: 4.5 is already cut, we can now increase the version in 4.6. Test Plan: Not needed. Reviewers: anthony, kradhakrishnan, andrewkr, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54963 01 March 2016, 22:26:44 UTC
74b6607 sdong 01 March 2016, 20:56:21 UTC Rename iterator property "rocksdb.iterator.is.key.pinned" => "rocksdb.iterator.is-key-pinned" Summary: Rename iterator property to folow property naming convention. Test Plan: Run all existing tests. Reviewers: andrewkr, anthony, yhchiang, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54957 01 March 2016, 21:47:12 UTC
6743135 Islam AbdelRahman 01 March 2016, 20:05:29 UTC Fix DB::AddFile() issue when PurgeObsoleteFiles() is called Summary: In some situations the DB will scan all existing files in the DB path and delete the ones that are Obsolete. If this happen during adding an external sst file. this could cause the file to be deleted while we are adding it. This diff fix this issue Test Plan: unit test to reproduce the bug existing unit tests Reviewers: sdong, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D54627 01 March 2016, 20:05:29 UTC
432f3ad sdong 01 March 2016, 17:34:50 UTC Add DB Property "rocksdb.current_version_number" Summary: Add a DB Property "rocksdb.current_version_number" for users to monitor version changes and stale iterators. Test Plan: Add a unit test. Reviewers: andrewkr, yhchiang, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54927 01 March 2016, 18:55:40 UTC
b5b1db1 sdong 29 February 2016, 23:15:23 UTC Recompute compaction score after scheduling manual compaction Summary: After we made manual compaction runnable concurrently with automaticallly compaction, we need to run ComputeCompactionScore() to prepare a coming compaction picking call before the compaction finishes. Test Plan: Run existing tests. Reviewers: yhchiang, IslamAbdelRahman, andrewkr, kradhakrishnan, anthony, igor Reviewed By: igor Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54891 01 March 2016, 01:17:51 UTC
5ea9aa3 agiardullo 03 February 2016, 03:19:17 UTC TransactionDB:ReinitializeTransaction Summary: Add function to reinitialize a transaction object so that it can be reused. This is an optimization so users can potentially avoid reallocating transaction objects. Test Plan: added tests Reviewers: yhchiang, kradhakrishnan, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: jkedgar, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53835 01 March 2016, 00:27:32 UTC
1f59541 sdong 27 February 2016, 01:13:39 UTC Introduce Iterator::GetProperty() and replace Iterator::IsKeyPinned() Summary: Add Iterator::GetProperty(), a way for users to communicate with iterator, and turn Iterator::IsKeyPinned() with it. As a follow-up, I'll ask a property as the version number attached to the iterator Test Plan: Rerun existing tests and add a negative test case. Reviewers: yhchiang, andrewkr, kradhakrishnan, anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54783 29 February 2016, 22:01:31 UTC
6778941 Igor Canadi 29 February 2016, 21:12:57 UTC Merge pull request #1020 from gongsu832/master Modified Makefile and build_tools/build_detect_platform to compile on… 29 February 2016, 21:12:57 UTC
69c471b Andrew Kryczka 29 February 2016, 20:56:55 UTC Handle concurrent manifest update and backup creation Summary: Fixed two related race conditions in backup creation. (1) CreateNewBackup() uses DB::DisableFileDeletions() to prevent table files from being deleted while it is copying; however, the MANIFEST file could still rotate during this time. The fix is to stop deleting the old manifest in the rotation logic. It will be deleted safely later when PurgeObsoleteFiles() runs (can only happen when file deletions are enabled). (2) CreateNewBackup() did not account for the CURRENT file being mutable. This is significant because the files returned by GetLiveFiles() contain a particular manifest filename, but the manifest to which CURRENT refers can change at any time. This causes problems when CURRENT changes between the call to GetLiveFiles() and when it's copied to the backup directory. To workaround this, I manually forge a CURRENT file referring to the manifest filename returned in GetLiveFiles(). (2) also applies to the checkpointing code, so let me know if this approach is good and I'll make the same change there. Test Plan: new test for roll manifest during backup creation. running the test before this change: $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation ... IO error: /tmp/rocksdbtest-9383/backupable_db/MANIFEST-000001: No such file or directory running the test after this change: $ ./backupable_db_test --gtest_filter=BackupableDBTest.ChangeManifestDuringBackupCreation ... [ RUN ] BackupableDBTest.ChangeManifestDuringBackupCreation [ OK ] BackupableDBTest.ChangeManifestDuringBackupCreation (2836 ms) Reviewers: IslamAbdelRahman, anthony, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54711 29 February 2016, 20:56:55 UTC
3373c81 root 29 February 2016, 20:02:52 UTC Modify build_tools/build_detect_platform to detect and set -march=z10 on Linux s390x. 29 February 2016, 20:02:52 UTC
9905090 root 29 February 2016, 19:45:55 UTC Merge branch 'master' of https://github.com/gongsu832/rocksdb Rebase for -momit-leaf-frame-pointer fix. 29 February 2016, 19:45:55 UTC
3492889 Igor Canadi 29 February 2016, 17:32:16 UTC Merge pull request #1019 from javacruft/wip-omit-leaf-frame-pointer-archs build: Improve -momit-leaf-frame-pointer usage 29 February 2016, 17:32:16 UTC
7ca731b James Page 29 February 2016, 09:03:24 UTC build: Improve -momit-leaf-frame-pointer usage -momit-leaf-frame-pointer is only supported on certain archs. Detect this automatically based on which flags the compiler understands. 29 February 2016, 09:22:24 UTC
21f17aa root 26 February 2016, 23:03:07 UTC Modified Makefile and build_tools/build_detect_platform to compile on Linux s390x. 26 February 2016, 23:03:07 UTC
8800975 sdong 25 February 2016, 23:42:26 UTC Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more robust Summary: Based on thread scheduling, DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels can fail to flush enough files to trigger expected compactions. Fix it by waiting for flush after inserting each key. There are failrue reported: db/db_universal_compaction_test.cc:1134: Failure Expected: (NumTableFilesAtLevel(options.num_levels - 1, 1)) > (0), actual: 0 vs 0 but I can't repro it. Try to fix the bug and see whether it goes away. Test Plan: Run the test multiple time. Reviewers: IslamAbdelRahman, anthony, andrewkr, kradhakrishnan, yhchiang Reviewed By: yhchiang Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54747 26 February 2016, 19:59:31 UTC
cd3fe67 agiardullo 26 February 2016, 01:29:39 UTC Remove stale TODO Summary: This was fixed by 0c2bd5cb Test Plan: n/a Reviewers: gabijs Reviewed By: gabijs Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54753 26 February 2016, 01:44:35 UTC
69c98f0 Andrew Kryczka 24 February 2016, 18:32:11 UTC Reorder instance variables in backup test for proper destruction order Summary: As titled. This fixes the tsan error caused by logger_ being used in backup_engine_'s destructor. It does not fix the transient unit test failure, which is caused by MANIFEST file changing while backup is happening. Test Plan: verified the tsan error no longer happens on either success or failure. $ COMPILE_WITH_TSAN=1 make -j32 backupable_db_test $ while ./backupable_db_test --gtest_filter=BackupableDBTest.CorruptionsTest ; do : ; done Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54669 24 February 2016, 18:32:11 UTC
82f15fb sdong 23 February 2016, 22:54:05 UTC Add test to make sure DropColumnFamily doesn't impact existing iterators Summary: Add a test case in ColumnFamilyTest.ReadDroppedColumnFamily to make sure existing iterator is not impacted by column family dropping. Test Plan: N/A Reviewers: igor, yhchiang, anthony, andrewkr, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54657 24 February 2016, 18:25:38 UTC
38201b3 sdong 23 February 2016, 01:29:28 UTC Fix assert failure when DBImpl::SyncWAL() conflicts with log rolling Summary: DBImpl::SyncWAL() releases db mutex before calling DBImpl::MarkLogsSynced(), while inside DBImpl::MarkLogsSynced() we assert there is none or one outstanding log file. However, a memtable switch can happen in between and causing two or outstanding logs there, failing the assert. The diff adds a unit test that repros the issue and fix the assert so that the unit test passes. Test Plan: Run the new tests. Reviewers: anthony, kolmike, yhchiang, IslamAbdelRahman, kradhakrishnan, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54621 23 February 2016, 19:42:15 UTC
2568985 Yi Wu 23 February 2016, 18:26:24 UTC IOStatsContext::ToString() add option to exclude zero counters Summary: similar to D52809 add option to exclude zero counters. Test Plan: [yiwu@dev4504.prn1 ~/rocksdb] ./iostats_context_test [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from IOStatsContextTest [ RUN ] IOStatsContextTest.ToString [ OK ] IOStatsContextTest.ToString (0 ms) [----------] 1 test from IOStatsContextTest (0 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (0 ms total) [ PASSED ] 1 test. Reviewers: anthony, yhchiang, andrewkr, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54591 23 February 2016, 18:26:24 UTC
b046916 Andrew Kryczka 23 February 2016, 05:32:19 UTC Redo SyncPoints for flush while rolling test Summary: There was a race condition in the test where the rolling thread acquired the mutex before the flush thread pinned the logger. Rather than add more complicated synchronization to fix it, I followed Siying's suggestion to use SyncPoint in the test code. Comments in the LoadDependency() invocation explain the reason for each of the sync points. Test Plan: Ran test 1000 times for tsan/asan. Will wait for all sandcastle tests to finish before committing since this is a tricky test. Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54615 23 February 2016, 05:32:19 UTC
291ae4c Yueh-Hsuan Chiang 23 February 2016, 00:33:26 UTC Revert "Revert "Fixed the bug when both whole_key_filtering and prefix_extractor are set."" Summary: This reverts commit 73c31377bbcd300061245138dbaf782fedada9ba, which mistakenly reverts 73c31377bbcd300061245138dbaf782fedada9ba that fixes a bug when both whole_key_filtering and prefix_extractor are set Test Plan: revert the patch Reviewers: anthony, IslamAbdelRahman, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52707 23 February 2016, 00:33:26 UTC
eef63ef Mike Kolupaev 22 February 2016, 21:54:58 UTC Fixed CompactFiles() spuriously failing or corrupting DB Summary: We started getting two kinds of crashes since we started using `DB::CompactFiles()`: (1) `CompactFiles()` fails saying something like "/data/logdevice/4440/shard12/012302.sst: No such file or directory", and presumably makes DB read-only, (2) DB fails to open saying "Corruption: Can't access /267000.sst: IO error: /data/logdevice/4440/shard1/267000.sst: No such file or directory". AFAICT, both can be explained by background thread deleting compaction output as "obsolete" while it's being written, before it's committed to manifest. If it ends up committed to the manifest, we get (2); if compaction notices the disappearance and fails, we get (1). The internal tasks t10068021 and t10134177 have some details about the investigation that led to this. Test Plan: `make -j check`; the new test fails to reopen the DB without the fix Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba, sdong Differential Revision: https://reviews.facebook.net/D54561 22 February 2016, 21:54:58 UTC
79ca039 Yueh-Hsuan Chiang 19 February 2016, 22:42:24 UTC Relax the check condition of prefix_extractor in CheckOptionsCompatibility Summary: Relax the check condition of prefix_extractor in CheckOptionsCompatibility by allowing changing value from non-nullptr to nullptr or nullptr to non-nullptr. Test Plan: options_test options_util_test Reviewers: sdong, anthony, IslamAbdelRahman, kradhakrishnan, gunnarku Reviewed By: gunnarku Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54477 19 February 2016, 22:42:24 UTC
4b1b4b8 Igor Canadi 19 February 2016, 22:27:18 UTC Merge pull request #1004 from yuslepukhin/child_attr Implement ConsistentChildrenAttribute on Windows 19 February 2016, 22:27:18 UTC
9ea2968 Dmitri Smirnov 19 February 2016, 22:20:34 UTC Implement ConsistentChildrenAttribute by using default implementation for now as it works. 19 February 2016, 22:20:34 UTC
c7f1a8a sdong 19 February 2016, 21:47:19 UTC Fix LITE build thread_local_test Summary: Recent change break thread_local_test by introducing exception, which is disabled in LITE build. Fix it by disabling exception handling in LITE build. Test Plan: Build with both of LITE and non-LITE Reviewers: anthony, IslamAbdelRahman, yhchiang, kradhakrishnan, andrewkr Reviewed By: andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54513 19 February 2016, 21:57:18 UTC
0914f0c Igor Canadi 19 February 2016, 21:12:36 UTC Merge pull request #1003 from yuslepukhin/fix_mutexlock_pthread_build This addresses build issues on Windows 19 February 2016, 21:12:36 UTC
d37d348 Dmitri Smirnov 19 February 2016, 20:29:54 UTC This addresses build issues on Windows https://github.com/facebook/rocksdb/issues/1002 19 February 2016, 20:29:54 UTC
d825fc7 Andrew Kryczka 19 February 2016, 02:03:53 UTC Use condition variable in log roller test Summary: Previously I just slept until the flush_thread was "probably" ready since proper synchronization in test cases seemed like overkill. But then tsan complained about it, so I did the synchronization (mostly) properly now. Test Plan: $ COMPILE_WITH_TSAN=1 make -j32 auto_roll_logger_test $ ./auto_roll_logger_test Reviewers: anthony, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54399 19 February 2016, 02:03:53 UTC
6b2a047 Islam AbdelRahman 18 February 2016, 19:25:19 UTC Fix SstFileManager uninitialized data member Summary: I have introduced max_allowed_space_ but did not initialize it Test Plan: make check Reviewers: sdong, yhchiang, anthony Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D54357 18 February 2016, 19:25:19 UTC
a3db93c Yueh-Hsuan Chiang 18 February 2016, 07:32:14 UTC Remove the SyncPoint usage in the destructor of PosixEnv Summary: Remove the SyncPoint usage in the destructor of PosixEnv as none of any active tests is using it. SyncPoint is a test-only utility class, and it's a static varible. As a result, using SyncPoint in the destructor of PosixEnv will make default Env depends on SyncPoint. Removing such dependency could solve the problem crash issue only reproducable in Mac environment. Test Plan: OPT=-DTRAVIS V=1 make -j4 check on Mac environment Reviewers: sdong, anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54333 18 February 2016, 07:32:14 UTC
df9ba6d Islam AbdelRahman 17 February 2016, 23:20:23 UTC Introduce SstFileManager::SetMaxAllowedSpaceUsage() to cap disk space usage Summary: Introude SstFileManager::SetMaxAllowedSpaceUsage() that can be used to limit the maximum space usage allowed for RocksDB. When this limit is exceeded WriteImpl() will fail and return Status::Aborted() Test Plan: unit testing Reviewers: yhchiang, anthony, andrewkr, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53763 17 February 2016, 23:20:23 UTC
3943d16 Andrew Kryczka 17 February 2016, 20:06:45 UTC Fix race conditions in auto-rolling logger Summary: For GetLogFileSize() and Flush(), they previously did not follow the synchronization pattern for accessing logger_. This meant ResetLogger() could cause logger_ destruction while the unsynchronized functions were accessing it, causing a segfault. Also made the mutex instance variable mutable so we can preserve GetLogFileSize()'s const-ness. Test Plan: new test case, it's quite ugly because both threads need to access one of the functions with SyncPoints (PosixLogger::Flush()), and also special handling is needed to prevent the mutex and sync points from conflicting. Reviewers: kradhakrishnan, IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54237 17 February 2016, 20:06:45 UTC
d733dd5 Andrew Kryczka 17 February 2016, 19:57:04 UTC [build] Fix env_win.cc compiler errors Summary: I broke it in D53781. Test Plan: tried the same code in util/env_posix.cc and it compiled successfully Reviewers: sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54303 17 February 2016, 19:57:04 UTC
cf38e56 Jonathan Wiepert 17 February 2016, 19:49:52 UTC Fix broken appveyor build caused by D53991 Test Plan: pushed remote as D53991_appveyor_test branch. Appveyor run: https://ci.appveyor.com/project/Facebook/rocksdb/build/1.0.651 shows the file is now found and the run is back to the state before the initial change: https://ci.appveyor.com/project/Facebook/rocksdb/build/1.0.620 Reviewers: sdong, igor Reviewed By: igor Subscribers: igor, dhruba Differential Revision: https://reviews.facebook.net/D54309 17 February 2016, 19:49:52 UTC
351252b Igor Canadi 17 February 2016, 17:06:07 UTC Merge pull request #998 from fengjian0106/master fix ios build error 17 February 2016, 17:06:07 UTC
1336052 fengjian 17 February 2016, 12:22:40 UTC fix ios build error 17 February 2016, 12:22:40 UTC
d08d502 agiardullo 10 February 2016, 02:24:41 UTC Fix transaction locking Summary: Broke transaction locking in 4.4 in D52197. Will cherry-pick this change into 4.4 (which hasn't yet been fully released). Repro'd using db_bench. Test Plan: unit tests and db_Bench Reviewers: sdong, yhchiang, kradhakrishnan, ngbronson Reviewed By: ngbronson Subscribers: ngbronson, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54021 17 February 2016, 01:15:05 UTC
730a422 Yueh-Hsuan Chiang 16 February 2016, 22:55:24 UTC Improve the documentation of LoadLatestOptions Summary: Improve the documentation of LoadLatestOptions Test Plan: No code change Reviewers: anthony, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54087 16 February 2016, 22:55:24 UTC
a7b6f07 reid horuff 16 February 2016, 20:39:55 UTC Improve write_with_callback_test to sync WAL Summary: Currently write_with_callback_test does not test with WAL syncing enabled. This addresses that. Test Plan: write_with_callback_test Reviewers: anthony Reviewed By: anthony Subscribers: leveldb, dhruba, hermanlee4 Differential Revision: https://reviews.facebook.net/D54255 16 February 2016, 22:04:14 UTC
5bcf952 reid horuff 11 February 2016, 20:19:21 UTC Fix WriteImpl empty batch hanging issue Summary: There is an issue in DBImpl::WriteImpl where if an empty writebatch comes in and sync=true then the logs will be marked as being synced yet the sync never actually happens because there is no data in the writebatch. This causes the next incoming batch to hang while waiting for the logs to complete syncing. This fix syncs logs even if the writebatch is empty. Test Plan: DoubleEmptyBatch unit test in transaction_test. Reviewers: yoshinorim, hermanlee4, sdong, ngbronson, anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54057 16 February 2016, 20:21:33 UTC
871cc5f Jonathan Wiepert 16 February 2016, 20:16:47 UTC fix build without gflags Test Plan: Built and ran with gflags: % ./db_bench LevelDB: version 4.5 Date: Tue Feb 16 12:04:23 2016 CPU: 40 * Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz ... And without gflags: % ./db_bench Please install gflags to run rocksdb tools % Reviewers: sdong, igor Reviewed By: igor Subscribers: igor, dhruba Differential Revision: https://reviews.facebook.net/D54243 16 February 2016, 20:16:47 UTC
c90d63a Igor Canadi 16 February 2016, 19:24:40 UTC can_unlock set but not used Test Plan: I couldn't repro, but I hope this fixes it. See the error here: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_ubuntu1404_rocksdb_compile_6e9fd902d5cb25aef992363efa128640affd5196_16_02_11_04_33_37/0?type=T Reviewers: yhchiang, andrewkr, sdong, anthony Reviewed By: anthony Subscribers: meyering, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54123 16 February 2016, 19:24:40 UTC
4437150 Mike Kolupaev 16 February 2016, 19:11:16 UTC Fixed a segfault when compaction fails Summary: We've hit it today. Test Plan: `make -j check`; didn't reproduce the issue Reviewers: yhchiang Reviewed By: yhchiang Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D54219 16 February 2016, 19:11:16 UTC
2f084d3 Igor Canadi 16 February 2016, 18:26:25 UTC Merge pull request #992 from jofusa/jdennison/options-typo-fix fixes typo in options logging 16 February 2016, 18:26:25 UTC
7bd284c Jonathan Wiepert 16 February 2016, 14:17:31 UTC Separeate main from bench functionality to allow cusomizations Summary: Isolate db_bench functionality from main so custom benchmark code can be written and managed Test Plan: Tested commands ./build_tools/regression_build_test.sh ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --reads=500 --writes=500 ./db_bench --db=/tmp/rocksdbtest-12321/dbbench --stats_interval_seconds=1 --num=1000 --merge_keys=100 --numdistinct=100 --num_column_families=3 --num_hot_column_families=1 ./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --usee_uint64_comparator=true --batch-size=5 Test Results - https://phabricator.fb.com/P56130387 Additional tests for: ./db_bench --duration=60 --value_size=50 --seek_nexts=10 --reverse_iterator=true --use_uint64_comparator=true --batch_size=5 --key_size=8 --merge_operator=put ./db_bench --stats_interval_seconds=1 --num=1000 --bloom_locality=1 --seed=5 --threads=5 --merge_operator=uint64add Results: https://phabricator.fb.com/P56130607 Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: dhruba Differential Revision: https://reviews.facebook.net/D53991 16 February 2016, 14:17:31 UTC
bd5f842 John Dennison 15 February 2016, 23:55:29 UTC fixes typo in options logging 15 February 2016, 23:55:29 UTC
1c868d6 Andrew Kryczka 12 February 2016, 22:17:14 UTC Fix includes for env_test Summary: Previously compilation failed when ROCKSDB_NO_FBCODE=1 because fcntl.h wasn't included for open(). Related issue: https://github.com/facebook/rocksdb/issues/977 Test Plan: verified below command works now: $ make clean && ROCKSDB_NO_FBCODE=1 ROCKSDB_DISABLE_FALLOCATE=1 make -j32 env_test Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54135 12 February 2016, 22:17:14 UTC
545a193 krad 07 February 2016, 08:42:37 UTC Add J to commit_prereq so comilation/execution happens in parallel Summary: Users are confused on how to get the parallel compilation going. This can help wire the parallelism. Test Plan: Run manually Reviewers: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53931 12 February 2016, 20:09:07 UTC
5bb7371 Andrew Kryczka 12 February 2016, 18:49:35 UTC [build] Evaluate test names only when db_test exists Summary: as titled, this will prevent the error that was printed because test_names was evaluated before db_test was built. Test Plan: verified below command works and no longer prints errors: $ make release -j32 verified below command still finds the right tests: $ make J=32 parallel_check Reviewers: igor Reviewed By: igor Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54117 12 February 2016, 18:49:35 UTC
6a2b4fc Andrew Kryczka 12 February 2016, 01:00:01 UTC Add flag to forcibly disable fallocate Summary: see https://github.com/facebook/rocksdb/issues/977; there are issues with fallocate() on certain filesystems/kernel versions that can lead it to pre- allocating blocks but never freeing them, even if they're unused. Test Plan: verified build commands omit DROCKSDB_FALLOCATE_PRESENT when this env variable is set. without disabling it: $ ROCKSDB_NO_FBCODE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $? 0 with disabling it: $ ROCKSDB_NO_FBCODE=1 DISABLE_FALLOCATE=1 make -n env_test | grep -q DROCKSDB_FALLOCATE_PRESENT ; echo $? 1 Reviewers: kradhakrishnan, yhchiang, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54069 12 February 2016, 01:00:01 UTC
92a9ccf sdong 11 February 2016, 21:45:53 UTC Add a new compaction priority that picks file whose overlapping ratio is smallest Summary: Add a new compaction priority as following: For every file, we calculate total size of files overalapping with the file in the next level, over the file's size itself. The file with smallest ratio will be picked first. My "db_bench --fillrandom" shows about 5% less compaction than kOldestSmallestSeqFirst if --hard_pending_compaction_bytes_limit value to keep LSM tree in shape. If not limiting hard_pending_compaction_bytes_limit, improvement is only 1% or 2%. Test Plan: Add a unit test Reviewers: andrewkr, kradhakrishnan, anthony, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54075 11 February 2016, 23:59:19 UTC
3dc3d1c Igor Canadi 11 February 2016, 18:22:40 UTC Merge pull request #984 from petermattis/pmattis/comparator-iterate-upper-bound Use user_comparator when comparing against iterate_upper_bound. 11 February 2016, 18:22:40 UTC
239aaf2 Peter Mattis 11 February 2016, 13:26:25 UTC Use user_comparator when comparing against iterate_upper_bound. Fixes #983. 11 February 2016, 13:47:16 UTC
9081003 Yueh-Hsuan Chiang 11 February 2016, 00:56:01 UTC Fixed a dependency issue of ThreadLocalPtr Summary: When a child thread that uses ThreadLocalPtr, ThreadLocalPtr::OnThreadExit will be called when that child thread is destroyed. However, OnThreadExit will try to access a static singleton of ThreadLocalPtr, which will be destroyed when the main thread exit. As a result, when a child thread that uses ThreadLocalPtr exits AFTER the main thread exits, illegal memory access will occur. This diff includes a test that reproduce this legacy bug. ==2095206==ERROR: AddressSanitizer: heap-use-after-free on address 0x608000007fa0 at pc 0x959b79 bp 0x7f5fa7426b60 sp 0x7f5fa7426b58 READ of size 8 at 0x608000007fa0 thread T1 This patch fix this issue by having the thread local mutex never be deleted (but will leak small piece of memory at the end.) The patch also describe a better solution (thread_local) in the comment that requires gcc 4.8.1 and in latest clang as a future work once we agree to move toward gcc 4.8. Test Plan: COMPILE_WITH_ASAN=1 make thread_local_test -j32 ./thread_local_test --gtest_filter="*MainThreadDiesFirst" Reviewers: anthony, hermanlee4, sdong Reviewed By: sdong Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53013 11 February 2016, 00:56:01 UTC
337671b Gunnar Kudrjavets 10 February 2016, 23:30:47 UTC Add universal compaction benchmarks to run_flash_bench.sh Summary: Implement a benchmark for universal compaction based on the feature description (see below), in-person discussions, and reading source code: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide https://github.com/facebook/rocksdb/wiki/Universal-Compaction https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#universal-compaction Universal compaction benchmark is based on `overwrite` benchmark, adding compaction specific options to it, and executing it for different values of subcompaction to understand the impact of scaling out subcompactions for a particular scenario. Test Plan: - Execute the benchmark on various machines for multiple iterations to verify the reliability. - Observe the output to make sure that compaction is taking place. - Observe the execution to make sure that arguments passed to `db_bench` are correct. Reviewers: sdong, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54045 10 February 2016, 23:30:47 UTC
3a67bff Yueh-Hsuan Chiang 10 February 2016, 20:06:59 UTC Fix an ASAN error in transaction_test.cc Summary: One test in transaction_test.cc forgets to call SyncPoint::DisableProcessing(). As a result, a program might to access the SyncPoint singleton after it already goes out of scope. This patch fix this error by calling SyncPoint::DisableProcessing(). Test Plan: transaction_test Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, anthony Reviewed By: anthony Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54033 10 February 2016, 20:06:59 UTC
2a04268 Yueh-Hsuan Chiang 10 February 2016, 00:26:20 UTC Temporarily disable unstable tests in memory_test.cc Summary: memory_test.cc has some tests that are not unstable but hard to reproduce, and the cause is the test itself not the code. Temporarily disable the tests until we have a good fix. Test Plan: memory_test Reviewers: sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D54009 10 February 2016, 00:28:34 UTC
08a78b6 Igor Canadi 09 February 2016, 23:46:18 UTC Merge pull request #979 from facebook/update_licenses Updated all copyright headers to the new format. 09 February 2016, 23:46:18 UTC
21e9581 Baraa Hamodi 09 February 2016, 23:12:00 UTC Updated all copyright headers to the new format. 09 February 2016, 23:12:00 UTC
59b3ee6 Andrew Kryczka 09 February 2016, 22:54:32 UTC Env function for bulk metadata retrieval Summary: Added this new function, which returns filename, size, and modified timestamp for each file in the provided directory. The default implementation retrieves the metadata sequentially using existing functions. In the next diff I'll make HdfsEnv override this function to use libhdfs's bulk get function. This won't work on windows due to the path separator. Test Plan: new unit test $ ./env_test --gtest_filter=EnvPosixTest.ConsistentChildrenMetadata Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: IslamAbdelRahman, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53781 09 February 2016, 22:54:32 UTC
4a8cbf4 Yueh-Hsuan Chiang 09 February 2016, 19:20:22 UTC Allows Get and MultiGet to read directly from SST files. Summary: Add kSstFileTier to ReadTier, which allows Get and MultiGet to read only directly from SST files and skip mem-tables. kSstFileTier = 0x2 // data in SST files. // Note that this ReadTier currently only supports // Get and MultiGet and does not support iterators. Test Plan: add new test in db_test. Reviewers: anthony, IslamAbdelRahman, rven, kradhakrishnan, sdong Reviewed By: sdong Subscribers: igor, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53511 09 February 2016, 19:20:22 UTC
fe93bf9 agiardullo 15 September 2015, 00:11:52 UTC Transaction::UndoGetForUpdate Summary: MyRocks wants to be able to un-lock a key that was just locked by GetForUpdate(). To do this safely, I am now keeping track of the number of reads(for update) and writes for each key in a transaction. UndoGetForUpdate() will only unlock a key if it hasn't been written and the read count reaches 0. Test Plan: more unit tests Reviewers: igor, rven, yhchiang, spetrunia, sdong Reviewed By: spetrunia, sdong Subscribers: spetrunia, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D47043 09 February 2016, 18:46:11 UTC
2608219 sdong 05 February 2016, 22:30:18 UTC crash_test: cover concurrent memtable insert in default crash test Summary: Default crash test uses prefix hash memtable, which is not compatible to concurrent memtable. Allow prefix test run with skip list and use skip list memtable when concurrent insert is used. Test Plan: Run "python -u tools/db_crashtest.py whitebox" and watch sometimes skip list is used. Reviewers: anthony, yhchiang, kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53907 09 February 2016, 16:02:38 UTC
a76e909 sdong 05 February 2016, 22:26:26 UTC Fix LITE db_test build broken by previous commit Summary: Previous commit introduces a test that is not supported in LITE. Fix it. Test Plan: Build the test with ROCKSDB_LITE. Reviewers: kradhakrishnan, IslamAbdelRahman, anthony, yhchiang, andrewkr Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53901 05 February 2016, 22:29:09 UTC
b1887c5 sdong 05 February 2016, 21:46:17 UTC Explictly fail when memtable doesn't support concurrent insert Summary: If users turn on concurrent insert but the memtable doesn't support it, they might see unexcepted crash. Fix it by explicitly fail. Test Plan: Run different setting of stress_test and make sure it fails correctly. Will add a unit test too. Reviewers: anthony, kradhakrishnan, IslamAbdelRahman, yhchiang, andrewkr, ngbronson Reviewed By: ngbronson Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53895 05 February 2016, 22:15:50 UTC
8ed3438 Gunnar Kudrjavets 05 February 2016, 21:20:56 UTC Add option to run fillseq with WAL enabled in addition to WAL disabled Summary: This set of changes is part of the work to introduce benchmark for universal style compaction in RocksDB. It's conceptually separate from the compaction work, so sending it out as a separate diff to get it out of the way. Test Plan: - Run `./tools/run_flash_bench.sh`. - Look at the contents of `report.txt` and `report2.txt` to make sure that data is reported and attributed correctly. - During `db_bench` execution time make sure that the correct flags are passed to `--disable_wal` depending on the benchmark being executed. Reviewers: MarkCallaghan Reviewed By: MarkCallaghan Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D53865 05 February 2016, 21:20:56 UTC
73a9b0f sdong 05 February 2016, 20:10:23 UTC Update version to 4.5 Summary: Time to cut branch for release 4.5. Change the versions. Test Plan: Not needed Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, andrewkr, anthony Reviewed By: anthony Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53883 05 February 2016, 21:05:54 UTC
6f71d3b reid horuff 05 February 2016, 18:44:13 UTC Improve perf of Pessimistic Transaction expirations (and optimistic transactions) Summary: copy from task 8196669: 1) Optimistic transactions do not support batching writes from different threads. 2) Pessimistic transactions do not support batching writes if an expiration time is set. In these 2 cases, we currently do not do any write batching in DBImpl::WriteImpl() because there is a WriteCallback that could decide at the last minute to abort the write. But we could support batching write operations with callbacks if we make sure to process the callbacks correctly. To do this, we would first need to modify write_thread.cc to stop preventing writes with callbacks from being batched together. Then we would need to change DBImpl::WriteImpl() to call all WriteCallback's in a batch, only write the batches that succeed, and correctly set the state of each batch's WriteThread::Writer. Test Plan: Added test WriteWithCallbackTest to write_callback_test.cc which creates multiple client threads and verifies that writes are batched and executed properly. Reviewers: hermanlee4, anthony, ngbronson Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D52863 05 February 2016, 18:44:13 UTC
8e6172b Islam AbdelRahman 05 February 2016, 18:22:37 UTC Add BlockBasedTableOptions::index_block_restart_interval Summary: Add a new option to BlockBasedTableOptions that will allow us to change the restart interval for the index block Test Plan: unit tests Reviewers: yhchiang, anthony, andrewkr, sdong Reviewed By: sdong Subscribers: march, dhruba Differential Revision: https://reviews.facebook.net/D53721 05 February 2016, 18:22:37 UTC
34a40bf sdong 04 February 2016, 23:21:32 UTC Add --allow_concurrent_memtable_write in stress test and run it in crash_test Summary: Add an option of --allow_concurrent_memtable_write in stress test and cover it in crash test Test Plan: Run crash test and make sure three combinations of the two options show up randomly. Reviewers: IslamAbdelRahman, yhchiang, andrewkr, anthony, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53811 05 February 2016, 00:15:18 UTC
73bf330 Siying Dong 04 February 2016, 23:50:21 UTC Merge pull request #973 from yuslepukhin/moveout_testcode Move out of release build some test code 04 February 2016, 23:50:21 UTC
f7c0f4e sdong 04 February 2016, 00:04:10 UTC perf_context.cc and iostats_context.cc use different output macro (fix unity build) Summary: Unity build because perf_context.cc and iostats_context.cc define a different OUTPUT macro. Fix it. Test Plan: Build unity Reviewers: kradhakrishnan, anthony, yhchiang, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: divchenko, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D53769 04 February 2016, 01:27:08 UTC
back to top