https://github.com/facebook/rocksdb

sort by:
Revision Author Date Message Commit Date
7593f5c Fix unit tests issues on Windows 15 April 2016, 00:32:27 UTC
80b74a6 Include ldb tool in the windows build (#914) 14 April 2016, 17:50:34 UTC
7c14d11 Minor fix to java build job Summary: Java build is downloading junit when not available. Without the proxy settings exported that would not work. Test Plan: Run manually Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56751 14 April 2016, 17:46:02 UTC
535af52 BlockBasedTable::PrefixMayMatch() to skip index checking if we can't find a filter block. Summary: In the case where we can't find a filter block, there is not much benefit of doing the binary search and see whether the index key has the prefix. With the change, we blindly return true if we can't get the filter. It also fixes missing row cases for reverse comparator with full bloom. Test Plan: Add a test case that used to fail. Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, hermanlee4, yoshinorim, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56697 14 April 2016, 02:06:48 UTC
09be5ca Minor fix to sandcastle java job definition Summary: System was not able to locate javac Test Plan: Run command manually and observe output Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56709 13 April 2016, 22:59:05 UTC
1aeca97 Release RocksDB 4.7 Summary: Bump the version and update HISTORY.md Test Plan: none Reviewers: yhchiang, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56469 13 April 2016, 22:03:08 UTC
dfc3de8 Split Travis unittests Job Summary: - Update Makefile check_some command to run a subset of the tests - Update travis config to split the unittests job into 2 jobs -- job testing db_test, db_test2 -- job testing the rest of the unittests Test Plan: Run the new travis.yml on my own branch https://travis-ci.org/facebook/rocksdb/builds/122691453 Reviewers: kradhakrishnan, yhchiang, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56673 13 April 2016, 21:22:29 UTC
c2c8fe4 Add Java job for sandcastle Summary: Adding java build job definition to sandcastle. Test Plan: ./build_tools/rocksdb-lego-determinator java_build Reviewers: yhchiang, sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56679 13 April 2016, 18:40:51 UTC
19ef3de Fix ManualCompactionPartial test flakiness Summary: The reason for this test flakiness is that we try to verify that number of files in L0 is 3 after flushing the 3rd file although we may have a compaction running in the background that may finish before we do the check and the 3 L0 files are converted to 1 L1 file Test Plan: Run a modified version of the test that sleep before doing the check Reviewers: sdong, andrewkr, kradhakrishnan, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56643 13 April 2016, 17:38:45 UTC
b345b36 Add a minimum value for the refill bytes per period value Summary: If the user specified a small enough value for the rate limiter's bytes per second, the calculation for the number of refill bytes per period could become zero which would effectively cause the server to hang forever. Test Plan: Existing tests Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56631 13 April 2016, 16:01:42 UTC
dff4c48 BlockBasedTable::PrefixMayMatch: no need to find data block after full bloom checking Summary: Full block checking should be a good enough indication of prefix existance. No need to further check data block. This also fixes wrong results when using prefix bloom and reverse bitwise comparator. Test Plan: Will add a unit test. Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: hermanlee4, yoshinorim, yiwu, kradhakrishnan, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56625 12 April 2016, 23:25:54 UTC
0353b85 Propagate sandcastle run error to UI Summary: Currently the code does not propagate the sandcastle precommit test run error status to UI. This can confuse the developer when searching for errors. With this change, all success should be in green and all errors should be in red Test Plan: Submit the diff (and hopefully something will fail) Reviewers: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56595 12 April 2016, 23:18:37 UTC
b885f33 Parallelize travis jobs Summary: Update travis.yml to split the work into 3 sub jobs - running unittests - Building and testing java - building ROCKSDB_LITE Test Plan: https://travis-ci.org/facebook/rocksdb/builds/122433169 Reviewers: sdong, yhchiang, andrewkr, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56607 12 April 2016, 22:05:56 UTC
71303e0 Update db_bench_tool.cc (#1073) * Update db_bench_tool.cc I fixed the wrong letters, LevelDB -> rocksDB, because I thought of LevelDB as the wrong presentation. the following show my fix : fprintf(stderr, "LevelDB: version %d.%d\n", kMajorVersion, kMinorVersion); -----------------> fprintf(stderr, "rocksDB: version %d.%d\n", kMajorVersion, kMinorVersion); * Update db_bench_tool.cc * Update db_bench_tool.cc 12 April 2016, 21:05:09 UTC
63cf15b Fix option settable tests Summary: In option settable tests, bytes for pointers are not all skipped, so that they may be the same as the special character and cause false positive. Test Plan: Run the test. Manually verify the issue is not there any more. Reviewers: IslamAbdelRahman, andrewkr Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56553 12 April 2016, 21:03:35 UTC
e208575 using java7 in runtime for hdfs env (#1072) 12 April 2016, 04:08:26 UTC
13e6c8e Relax an assertion in Compaction::ShouldStopBefore Summary: In some case, it is possible to have two concesutive SST files might sharing same boundary keys. However, in the assertion in Compaction::ShouldStopBefore, it exclude such possibility. This patch fix this issue by relaxing the assertion to allow the equal case. Test Plan: rocksdb tests Reviewers: IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55875 12 April 2016, 03:15:52 UTC
ae21d71 Fixed a bug in RocksDB Statistics where flush is considered as compaction Summary: Fixed a bug in RocksDB Statistics where flush is considered as compaction Test Plan: unit test Reviewers: sdong, IslamAbdelRahman, rven, kradhakrishnan, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56271 12 April 2016, 02:59:25 UTC
8e0e22f Fix Windows build by replacing strings.h include Summary: strings.h header does not exist on Windows. So, we can try another way to compare strings ignoring case. Test Plan: built and ran: $ ./ldb_cmd_test Reviewers: sdong, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56535 12 April 2016, 02:21:00 UTC
5675d50 Revert travis commit Summary: Revert travis commit pushed unintentionally Test Plan: none Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56559 11 April 2016, 23:46:55 UTC
91f0f1f fix travis 11 April 2016, 23:37:22 UTC
a23c605 Don't run DBOptionsAllFieldsSettable under valgrind Summary: Test DBOptionsAllFieldsSettable sometimes fails under valgrind. Move option settable tests to a separate test file and disable it in valgrind.. Test Plan: Run valgrind test and make sure the test doesn't run. Reviewers: andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56529 11 April 2016, 21:24:55 UTC
30d72ee PrefixTest.PrefixAndWholeKeyTest should run against a different directory from prefix_test Summary: PrefixTest.PrefixAndWholeKeyTest runs against the same directory as prefix_test, which sometimes fail parallel tests. Fix it. Test Plan: Run it in parallel and see it doesn't fail anymore. Reviewers: andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56541 11 April 2016, 20:02:56 UTC
0e3cc2c Add column family info to TableProperties::ToString() Summary: This is used at least by `sst_dump --show_properties` Test Plan: - default CF ``` $ ./sst_dump --show_properties --file=./tmp-db/000007.sst | grep 'column family' column family ID: 0 column family name: default ``` - custom CF ``` $ ./sst_dump --show_properties --file=./tmp-db/000012.sst | grep 'column family' column family ID: 1 column family name: col-fam-1 ``` - no CF ``` $ ./sst_dump --show_properties --file=./tmp-db/000017.sst | grep 'column family' column family ID: N/A column family name: N/A ``` Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56499 09 April 2016, 01:50:18 UTC
2448f80 Make sure that if use_mmap_reads is on use_os_buffer is also on Summary: The code assumes that if use_mmap_reads is on then use_os_buffer is also on. This make sense as by using memory mapped files for reading you are expecting the OS to cache what it needs. Add code to make sure the user does not turn off use_os_buffer when they turn on use_mmap_reads Test Plan: New test: DBTest.MMapAndBufferOptions Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56397 08 April 2016, 21:30:15 UTC
114a1b8 Fix build errors for windows Summary: - Need to use unsigned long long for 64-bit literals on windows - Need size_t for backup meta-file length since clang doesn't let us assign size_t to int Test Plan: backupable_db_test and options_test Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56391 08 April 2016, 20:09:19 UTC
0522990 Improve sst_dump help message Summary: Current Message ``` sst_dump [--command=check|scan|none|raw] [--verify_checksum] --file=data_dir_OR_sst_file [--output_hex] [--input_key_hex] [--from=<user_key>] [--to=<user_key>] [--read_num=NUM] [--show_properties] [--show_compression_sizes] [--show_compression_sizes [--set_block_size=<block_size>]] ``` New message ``` sst_dump --file=<data_dir_OR_sst_file> [--command=check|scan|raw] --file=<data_dir_OR_sst_file> Path to SST file or directory containing SST files --command=check|scan|raw check: Iterate over entries in files but dont print anything except if an error is encounterd (default command) scan: Iterate over entries in files and print them to screen raw: Dump all the table contents to <file_name>_dump.txt --output_hex Can be combined with scan command to print the keys and values in Hex --from=<user_key> Key to start reading from when executing check|scan --to=<user_key> Key to stop reading at when executing check|scan --read_num=<num> Maximum number of entries to read when executing check|scan --verify_checksum Verify file checksum when executing check|scan --input_key_hex Can be combined with --from and --to to indicate that these values are encoded in Hex --show_properties Print table properties after iterating over the file --show_compression_sizes Independent command that will recreate the SST file using 16K block size with different compressions and report the size of the file using such compression --set_block_size=<block_size> Can be combined with --show_compression_sizes to set the block size that will be used when trying different compression algorithms ``` Test Plan: none Reviewers: yhchiang, andrewkr, kradhakrishnan, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56325 08 April 2016, 19:05:02 UTC
0930e5e Update comments on include/rocksdb/perf_context.h Summary: Some grammer mistakes in code comments in include/rocksdb/perf_context.h. Also polish it a liitlebit. Test Plan: Not needed Reviewers: IslamAbdelRahman, yhchiang, yiwu, andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56307 08 April 2016, 18:27:08 UTC
3b977bc instructing people to use java7 for hdfs (#1063) 07 April 2016, 21:34:28 UTC
1518b73 Change default number of cache shard bit to be 6 and max_file_opening_threads to be 16. Summary: Cache shard bit 4 is sometimes too small and 6 is a more common value picked by users. Make that default. It shouldn't hurt much to change options.max_file_opening_threads default to be 16, which will reduce the worst case DB open time. Test Plan: Run all existing tests. Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, igor, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D55047 07 April 2016, 20:55:10 UTC
ada88b6 fix wrong assignment of level0_stop_writes_trigger in spatialdb (#1061) 07 April 2016, 16:02:28 UTC
2391ef7 Embed column family name in SST file Summary: Added the column family name to the properties block. This property is omitted only if the property is unavailable, such as when RepairDB() writes SST files. In a next diff, I will change RepairDB to use this new property for deciding to which column family an existing SST file belongs. If this property is missing, it will add it to the "unknown" column family (same as its existing behavior). Test Plan: New unit test: $ ./db_table_properties_test --gtest_filter=DBTablePropertiesTest.GetColumnFamilyNameProperty Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55605 07 April 2016, 06:10:32 UTC
ab4c623 Don't use version in the error message Summary: We use object `v` in the error message, which is not initialized if the edit is column family manipulation. This doesn't provide much useful info, so this diff is removing it. Instead, it dumps actual VersionEdit contents. Test Plan: compiles. would be great to get tests in version_set_test.cc that cover cases where a file write fails Reviewers: sdong, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56349 06 April 2016, 22:00:15 UTC
d02eb8d Fix unused variable warning Summary: My last diff introduced a warning when compiling under release mode https://reviews.facebook.net/D55539 fix the warning Test Plan: DEBUG_LEVEL=0 make db_bench make check Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56295 04 April 2016, 18:10:46 UTC
9278097 Merge pull request #1056 from facebook/igorcanadi-patch-1 Create LANGUAGE-BINDINGS.md 04 April 2016, 15:08:52 UTC
cc87075 No need to limit to 20 files in UpdateAccumulatedStats() if options.max_open_files=-1 Summary: There is a hardcoded constraint in our statistics collection that prevents reading properties from more than 20 SST files. This means our statistics will be very inaccurate for databases with > 20 files since additional files are just ignored. The purpose of constraining the number of files used is to bound the I/O performed during statistics collection, since these statistics need to be recomputed every time the database reopened. However, this constraint doesn't take into account the case where option "max_open_files" is -1. In that case, all the file metadata has already been read, so MaybeInitializeFileMetaData() won't incur any I/O cost. so this diff gets rid of the 20-file constraint in case max_open_files == -1. Test Plan: write into unit test db/db_properties_test.cc - "ValidateSampleNumber". We generate 20 files with 2 rows and 10 files with 1 row. If max_open_files !=-1, the `rocksdb.estimate-num-keys` should be (10*1 + 10*2)/20 * 30 = 45. Otherwise, it should be the ground truth, 50. {F1089153} Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56253 01 April 2016, 23:19:12 UTC
8a1a603 Eliminate std::deque initialization while iterating over merge operands Summary: This patch is similar to D52563, When we iterate over a DB with merge operands we keep creating std::queue to store the operands, optimize this by reusing merge_operands_ data member Before the patch ``` ./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq" --db="/dev/shm/bench_merge_memcpy_on_the_fly/" --merge_operator="put" --merge_keys=10000 --num=10000 DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] mergerandom : 3.757 micros/op 266141 ops/sec; 29.4 MB/s ( updates:10000) DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.413 micros/op 2423538 ops/sec; 268.1 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.451 micros/op 2219071 ops/sec; 245.5 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.420 micros/op 2382039 ops/sec; 263.5 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.408 micros/op 2452017 ops/sec; 271.3 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] mergerandom : 3.947 micros/op 253376 ops/sec; 28.0 MB/s ( updates:10000) DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.441 micros/op 2266473 ops/sec; 250.7 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.471 micros/op 2122033 ops/sec; 234.8 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.440 micros/op 2271407 ops/sec; 251.3 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.429 micros/op 2331471 ops/sec; 257.9 MB/s ``` with the patch ``` ./db_bench --benchmarks="mergerandom,readseq,readseq,readseq,readseq" --db="/dev/shm/bench_merge_memcpy_on_the_fly/" --merge_operator="put" --merge_keys=10000 --num=10000 DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] mergerandom : 4.080 micros/op 245092 ops/sec; 27.1 MB/s ( updates:10000) DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.308 micros/op 3241843 ops/sec; 358.6 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.312 micros/op 3200408 ops/sec; 354.0 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.332 micros/op 3013962 ops/sec; 333.4 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.300 micros/op 3328017 ops/sec; 368.2 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] mergerandom : 3.973 micros/op 251705 ops/sec; 27.8 MB/s ( updates:10000) DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.320 micros/op 3123752 ops/sec; 345.6 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.335 micros/op 2986641 ops/sec; 330.4 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.339 micros/op 2950047 ops/sec; 326.4 MB/s DB path: [/dev/shm/bench_merge_memcpy_on_the_fly/] readseq : 0.319 micros/op 3131565 ops/sec; 346.4 MB/s ``` Test Plan: make check -j64 Reviewers: yhchiang, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56031 01 April 2016, 22:48:55 UTC
f38540b WriteBatchWithIndex micro optimization Summary: - Put key offset and key size in WriteBatchIndexEntry - Use vector for comparators in WriteBatchEntryComparator I use a slightly modified version of @yoshinorim code to benchmark https://gist.github.com/IslamAbdelRahman/b120f4fba8d6ff7d58d2 For Put I create a transaction that put a 1000000 keys and measure the time spent without commit. For GetForUpdate I read the keys that I added in the Put transaction. Original time: ``` rm -rf /dev/shm/rocksdb-example/ ./txn_bench put 1000000 1000000 OK Ops | took 3.679 seconds ./txn_bench get_for_update 1000000 1000000 OK Ops | took 3.940 seconds ``` New Time ``` rm -rf /dev/shm/rocksdb-example/ ./txn_bench put 1000000 1000000 OK Ops | took 2.727 seconds ./txn_bench get_for_update 1000000 1000000 OK Ops | took 3.880 seconds ``` It looks like there is no significant improvement in GetForUpdate() but we can see ~30% improvement in Put() Test Plan: unittests Reviewers: yhchiang, anthony, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, yoshinorim Differential Revision: https://reviews.facebook.net/D55539 01 April 2016, 22:23:46 UTC
2006540 Merge pull request #1053 from adamretter/benchmark-java-comparator Benchmark Java comparator vs C++ comparator 01 April 2016, 20:53:15 UTC
f2c43a4 Stderr info logger Summary: Adapted a stderr logger from the option tests. Moved it to a separate header so we can reuse it, e.g., from ldb subcommands for faster debugging. This is especially useful to make errors/warnings more visible when running "ldb repair", which involves potential data loss. Test Plan: ran options_test and "ldb repair" $ ./ldb repair --db=./tmp/ [WARN] **** Repaired rocksdb ./tmp/; recovered 1 files; 588bytes. Some data may have been lost. **** OK Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56151 01 April 2016, 18:06:06 UTC
b55e216 Rocksdb backup can store optional application specific metadata Summary: Rocksdb backup engine maintains metadata about backups in separate files. But, there was no way to add extra application specific data to it. Adding support for that. In some use cases, applications decide to restore a backup based on some metadata. This will help those cases to cheaply decide whether to restore or not. Test Plan: Added a unit test. Existing ones are passing Sample meta file for BinaryMetadata test- ``` 1459454043 0 metadata 6162630A64656600676869 2 private/1/MANIFEST-000001 crc32 1184723444 private/1/CURRENT crc32 3505765120 ``` Reviewers: sdong, ldemailly, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, ldemailly Differential Revision: https://reviews.facebook.net/D56007 01 April 2016, 17:56:52 UTC
9b51987 Adding pin_l0_filter_and_index_blocks_in_cache feature and related fixes. Summary: When a block based table file is opened, if prefetch_index_and_filter is true, it will prefetch the index and filter blocks, putting them into the block cache. What this feature adds: when a L0 block based table file is opened, if pin_l0_filter_and_index_blocks_in_cache is true in the options (and prefetch_index_and_filter is true), then the filter and index blocks aren't released back to the block cache at the end of BlockBasedTableReader::Open(). Instead the table reader takes ownership of them, hence pinning them, ie. the LRU cache will never push them out. Meanwhile in the table reader, further accesses will not hit the block cache, thus avoiding lock contention. Test Plan: 'export TEST_TMPDIR=/dev/shm/ && DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32' is OK. I didn't run the Java tests, I don't have Java set up on my devserver. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56133 01 April 2016, 17:42:39 UTC
2feafa3 Change some RocksDB default options Summary: Change some RocksDB default options to make it more friendly to server workloads. Test Plan: Run all existing tests Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: sumeet, muthu, benj, MarkCallaghan, igor, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55941 01 April 2016, 00:12:18 UTC
a558830 Fixed compile warnings in posix_logger.h and coding.h Summary: Fixed the following compile warnings: /Users/yhchiang/rocksdb/util/posix_logger.h:32:11: error: unused variable 'kDebugLogChunkSize' [-Werror,-Wunused-const-variable] const int kDebugLogChunkSize = 128 * 1024; ^ /Users/yhchiang/rocksdb/util/coding.h:24:20: error: unused variable 'kMaxVarint32Length' [-Werror,-Wunused-const-variable] const unsigned int kMaxVarint32Length = 5; ^ 2 errors generated. Test Plan: make clean rocksdb Reviewers: igor, sdong, anthony, IslamAbdelRahman, rven, kradhakrishnan, adamretter Reviewed By: adamretter Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56223 31 March 2016, 23:01:47 UTC
51c9464 Merge pull request #980 from adamretter/java-arm ARM for the Java API 31 March 2016, 20:56:45 UTC
925b5d0 Merge pull request #1054 from DCEngines/magic12 Remove the Magic number 12 used in record size checks 31 March 2016, 04:38:19 UTC
63e8f1b Formatted lines to adhere to 80 char limit 31 March 2016, 02:56:55 UTC
994b3bd Add support for UBsan builds to RocksDB Summary: Undefined Behavior Sanitizer (ubsan) is //a good thing// which will help us to find sneaky bugs with low cost. Please see http://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ for more details and official GCC documentation for more context: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html. Changes itself are quite simple and pretty much imitating whatever is implemented for ASan. Hooking the UBsan validation build to Sandcastle is a separate step and will be dealt as separate diff because code is in internal repository. Test Plan: Make sure that that there no regressions when it comes to builds and test pass rate. Reviewers: leveldb, sdong, IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56049 30 March 2016, 22:59:24 UTC
21700a5 to/from hex refactor Summary: Expose the inverse of ToString(hex=true) on Slice: Slice::DecodeHex Refactor the other implementation of to/from hex in ldb_cmd.h to use the Slice version (Difference between the 2 is whether 0x is expected/produced in front of the hex string or not) Eliminated support for invalid odd length hex string - this is now invalid instead of having 1/2 byte set Added (inverse of HexToString) test for LDBCommand::StringToHex which also indirectly tests Slice::ToString(true) After moving the original implementation from ldb_cmd.h, updated it to much simpler/efficient version (originally/inspired from https://github.com/facebook/wdt/blob/master/util/EncryptionUtils.cpp#L140-L169 ) Test Plan: run tests Reviewers: uddipta, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56121 30 March 2016, 21:36:48 UTC
2442094 Replace kHeader by WriteBatchInternal::kHeader in few more places kHeader was moved from write_batch.cc to header file because it is being used wherever the number "12" was being used to check for record size 30 March 2016, 17:43:00 UTC
3bdbe89 Merge branch 'magic12' of https://github.com/dcengines/rocksdb into magic12 30 March 2016, 17:39:39 UTC
7871152 In all the places where log records are read, there was a check that record.size() should not be less than 12. This "magic number" seems to be the WriteBatch header (8 byte sequence and 4 byte count). Replaced all the places where "12" was used by WriteBatchInternal::kHeader. 30 March 2016, 17:35:22 UTC
e7c64fb Imporve sst_file_manager comment Summary: Improve the comment for sst_file_manager Test Plan: none Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56001 29 March 2016, 20:20:26 UTC
99ffb3d Fix perf_context::merge_operator_time_nanos calculation Summary: We were not measuring the time spent in merge_operator when called from Version::Get() Test Plan: added a unittest Reviewers: sdong, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55905 26 March 2016, 01:29:43 UTC
07bb12d Update internal jemalloc and other versions Summary: Update jemalloc to newer versions, as well as some other dependencies. Only effective for FB internal. Test Plan: See all tests run Reviewers: IslamAbdelRahman, MarkCallaghan Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55947 25 March 2016, 17:42:20 UTC
ad2fdaa Correct a typo in a comment Summary: Correct a typo in a comment Test Plan: No code change. Reviewers: sdong, kradhakrishnan, IslamAbdelRahman Reviewed By: kradhakrishnan, IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55803 25 March 2016, 02:39:13 UTC
be9816b Fix data race issue when sub-compaction is used in CompactionJob Summary: When subcompaction is used, all subcompactions share the same Compaction pointer in CompactionJob while each subcompaction all keeps their mutable stats in SubcompactionState. However, there're still some mutable part that is currently store in the shared Compaction pointer. This patch makes two changes: 1. Make the shared Compaction pointer const so that it can never be modified during the compaction. 2. Move necessary states from Compaction to SubcompactionState. 3. Make functions of Compaction const if the function does not modify its internal state. Test Plan: rocksdb and MyRocks test Reviewers: sdong, kradhakrishnan, andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, yoshinorim, gunnarku, leveldb Differential Revision: https://reviews.facebook.net/D55923 25 March 2016, 02:36:39 UTC
e380253 Merge pull request #1050 from yuslepukhin/support_db_test2 Add support for db_test2 for dev and CI runs 24 March 2016, 00:59:30 UTC
e7cc49c Add support for db_test2 for dev and CI runs 23 March 2016, 23:33:57 UTC
3996770 Add comments to perf_context skip counters Summary: Document the skipped counters in perf context more clearly. Test Plan: Comment only. Reviewers: IslamAbdelRahman, yhchiang, MarkCallaghan Reviewed By: MarkCallaghan Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55833 23 March 2016, 21:26:25 UTC
4e85b74 Make WritableFileWrapper not screw up preallocation Summary: Without this diff, this is what happens to compaction output file if it's a subclass of WritableFileWrapper: - during compaction, all `PrepareWrite()` calls update `last_preallocated_block_` of the `WritableFileWrapper` itself, not of `target_`, since `PrepareWrite()` is not virtual, - `PrepareWrite()` calls `Allocate()`, which is virtual; it does `fallocate()` on `target_`, - after writing data, `target_->Close()` calls `GetPreallocationStatus()` of `target_`; it returns `last_preallocated_block_` of `target_`, which is zero because it was never touched before, - `target_->Close()` doesn't call `ftruncate()`; file remains big. This diff fixes it in a straightforward way, by making the methods virtual. `WritableFileWrapper` ends up having the useless fields `last_preallocated_block_` and `preallocation_block_size_`. I think ideally the preallocation logic should be outside `WritableFile`, the same way as `log_writer.h` and `file_reader_writer.h` moved some non-platform-specific logic out of Env, but that's probably not worth the effort now. Test Plan: `make -j check`; I'm going to deploy it on our test tier and see if it fixes space reclamation problem there Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, sdong Differential Revision: https://reviews.facebook.net/D54681 23 March 2016, 16:14:56 UTC
ec458dc Merge pull request #1047 from PraveenSinghRao/wal_filter_ex Avoid overloaded virtual function 23 March 2016, 04:48:39 UTC
583157f Avoid overloaded virtual function 23 March 2016, 00:10:31 UTC
b9d4fa8 Options settable tests to use a different special charactor Summary: Something changed and the special charactor seems to be conflict with an exisitng value. Change it to unblock the build. Test Plan: Run the test and make sure it passes Reviewers: kradhakrishnan Reviewed By: kradhakrishnan Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55845 22 March 2016, 23:21:53 UTC
60e34ba Merge pull request #1044 from PraveenSinghRao/wal_filter_ex Allow per-column-family processing in wal_filter (publish log numbers for column family to wal_filter, and provide log number in the record callback) 22 March 2016, 23:03:31 UTC
136b8e0 Merge from master 22 March 2016, 19:38:44 UTC
2dcbb3b Addressed review comments 22 March 2016, 19:07:15 UTC
b1fafca Revert "Adding pin_l0_filter_and_index_blocks_in_cache feature." This reverts commit 522de4f59e6314698286cf29d8a325a284d81778. It has bug of index block cleaning up. 21 March 2016, 18:50:42 UTC
5f8741a Revert "Fix failing Java unit test." This reverts commit d7ae42b0f89fd25d8aaed28703059889d145596e. This is reverted as auto buld failure. This commit itself doesn't have any problem. Reverting as it depends on the commit to revert. 21 March 2016, 18:46:57 UTC
43bbb56 tools/check_format_compatible.sh to use consistent version when testing backward and forward compatibility Summary: Test seems to fail if we don't use consistent version between testing forward and backward compatibility. Test Plan: Run the script (with some version removed manually to make it shorter) Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55773 21 March 2016, 18:13:26 UTC
d7ae42b Fix failing Java unit test. Test Plan: sent diff to sdong, passes :) Reviewers: sdong Reviewed By: sdong Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55749 19 March 2016, 15:53:56 UTC
fbbb8a6 Add test for Snapshot 0 Summary: I ran into this assert when stress testing transactions. It's pretty easy to repro. Changing VersionSet::last_sequence_ to start at 1 seems pretty straightforward. We would just need to change the 4 callers of SetLastSequence(), including recovery code. I'd make this change myself, but I do not have enough time to test changes to recovery code-paths this week. But checking in this test case (disabled) for future fixing. Test Plan: n/a Reviewers: yhchiang, kradhakrishnan, andrewkr, anthony, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55311 18 March 2016, 23:16:20 UTC
e182f03 Add unit tests for RepairDB Summary: Basic test cases: - Manifest is lost or corrupt - Manifest refers to too many or too few SST files - SST file is corrupt - Unflushed data is present when RepairDB is called Depends on D55065 for its CreateFile() function in file_utils Test Plan: Ran the tests. Reviewers: IslamAbdelRahman, yhchiang, yoshinorim, sdong Reviewed By: sdong Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55485 18 March 2016, 22:18:42 UTC
7d37186 travis build fixes 18 March 2016, 21:43:22 UTC
fbea4dc Merge pull request #1042 from SherlockNoMad/HistFix Fix in HistogramWindowingImpl 18 March 2016, 21:27:55 UTC
780d2b0 Update format compatible checking tool Summary: After introducing a less forward-compatible change, update the backward compatible checking tool. Test Plan: Run it. Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55695 18 March 2016, 21:07:15 UTC
4f1c74a merge from master 18 March 2016, 19:48:01 UTC
f8c2189 Publish log numbers for column family to wal_filter, and provide log number in the record callback 18 March 2016, 19:32:15 UTC
4475626 Reset block cache in failing unit test. Test Plan: make -j40 check OPT=-g, on both /tmp and /dev/shm Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55701 18 March 2016, 06:13:54 UTC
522de4f Adding pin_l0_filter_and_index_blocks_in_cache feature. Summary: When a block based table file is opened, if prefetch_index_and_filter is true, it will prefetch the index and filter blocks, putting them into the block cache. What this feature adds: when a L0 block based table file is opened, if pin_l0_filter_and_index_blocks_in_cache is true in the options (and prefetch_index_and_filter is true), then the filter and index blocks aren't released back to the block cache at the end of BlockBasedTableReader::Open(). Instead the table reader takes ownership of them, hence pinning them, ie. the LRU cache will never push them out. Meanwhile in the table reader, further accesses will not hit the block cache, thus avoiding lock contention. When the table reader is destroyed, it releases the pinned blocks (if there were any). This has to happen before the cache is destroyed, so I had to introduce a TableReader::Close(), to guarantee the order of destruction. Test Plan: Added two unit tests for this. Existing unit tests run fine (default is pin_l0_filter_and_index_blocks_in_cache=false). DISABLE_JEMALLOC=1 OPT=-g make all valgrind_check -j32 Mac: OK. Linux: with D55287 patched in it's OK. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D54801 17 March 2016, 22:40:01 UTC
be22271 Merge pull request #1041 from yuslepukhin/adjust_for_jemalloc Latest versions of Jemalloc library do not require je_init()/je_unint() 17 March 2016, 21:50:19 UTC
4ecc03c Fix in HistogramWindowingImpl 17 March 2016, 21:28:41 UTC
2ca0994 Latest versions of Jemalloc library do not require je_init()/je_unint() calls. #ifdef in the source code and make this a default build option. 17 March 2016, 18:25:20 UTC
90aff0c Update --max_write_buffer_number for compaction benchmarks Summary: For compactions benchmarks (both level and universal) we'll use `--max_write_buffer_number=4`. For all the other benchmarks which don't customize the value of `--max_background_flushes` we'll continue using `--max_write_buffer_number=8`. Test Plan: To validate basic correctness and command-line options: ``` cd ~/rocksdb NKEYS=10000000 ./tools/run_flash_bench.sh ``` Reviewers: MarkCallaghan Reviewed By: MarkCallaghan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55497 17 March 2016, 17:14:23 UTC
7222410 Forge current file for checkpoint Summary: This fixes a similar issue as D54711: "CURRENT" file can mutate between GetLiveFiles() and copy to the tmp directory, in which case it would reference the wrong manifest filename. To fix this, I forge the "CURRENT" file such that it simply contains the filename for the manifest returned by GetLiveFiles(). - Changed CreateCheckpoint() to forge current file - Added CreateFile() utility function - Added test case that rolls manifest during checkpoint creation Test Plan: $ ./checkpoint_test Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55065 17 March 2016, 17:07:21 UTC
33d5686 Merge pull request #1040 from bureau14/master Fixes warnings and ensure correct int behavior on 32-bit platforms. 17 March 2016, 10:23:30 UTC
02e62eb Fixes warnings and ensure correct int behavior on 32-bit platforms. 16 March 2016, 21:57:57 UTC
9cad568 Merge pull request #1039 from bureau14/master Improve documentation of the allow_os_buffer parameter. 16 March 2016, 17:30:01 UTC
3d29f91 Improve documentation of the allow_os_buffer parameter. 16 March 2016, 14:37:55 UTC
3ff98bd Fix no compression test Summary: DBBlockCacheTest.TestWithCompressedBlockCache is depending on compression using snappy, so this test fail when snappy is not available block this test when we don't have snappy https://ci-builds.fb.com/view/rocksdb/job/rocksdb_no_compression/833/console Test Plan: run the test when compression libraries are not avaliable Reviewers: sdong, yiwu Reviewed By: yiwu Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55413 15 March 2016, 19:17:40 UTC
b9cc42a Merge pull request #1038 from SherlockNoMad/HistFix Fix FB internal CI build failure 15 March 2016, 18:43:06 UTC
f76b260 Fix FB internal CI build failure 15 March 2016, 18:38:15 UTC
774922c Merge pull request #1026 from SherlockNoMad/Hist Histogram Concurrency Improvement and Time-Windowing Support 15 March 2016, 18:27:54 UTC
17b879b Merge pull request #1037 from SherlockNoMad/BuildFix Fix AppVeyor build error 15 March 2016, 18:15:56 UTC
f11b0df Fix AppVeyor build error 15 March 2016, 17:57:33 UTC
6b03f93 Fix the build break on Ubuntu 15.10 when gcc 5.2.1 is used Summary: Using gcc 5.2.1 to compile RocksDB on Ubuntu 15.10 results in a warning about unused variable. Warning is treated as an error and all of it results in a build break. Fix the issue and make sure that everything compiles with that particular configuration. P.S. Lint complains about a non-ASCII character used in the source code ("Wagner-Fischer"). Fix this as well. Test Plan: # Everything compiler cleanly on the system which exhibited the problem. # `make clean && make -j 16` on CentOS. # `make clean && USE_CLANG=1 make -j 16` on CentOS. Reviewers: sdong, igor Reviewed By: igor Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55503 15 March 2016, 17:30:10 UTC
697fab8 Updates to RocksDB subcompaction benchmarking script Summary: Set of updates to the subcompaction benchmark script which are based on our internal discussions. The intent behind the changes is to make sure that the scripts will correctly reflect how we're doing the actual benchmarking. Test Plan: Tested by exercising the full set of compaction benchmarks and validating the execution and consistency of results. Reviewers: MarkCallaghan, sdong, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55461 15 March 2016, 06:09:04 UTC
58379bf remove division from histogramwidowing impl 15 March 2016, 04:41:13 UTC
1a2cc27 ColumnFamilyOptions SanitizeOptions is buggy on 32-bit platforms. Summary: The pre-existing code is trying to clamp between 65,536 and 0, resulting in clamping to 65,536, resulting in very small buffers, resulting in ShouldFlushNow() being true quite easily, resulting in assertion failing and database performance being "not what it should be". https://github.com/facebook/rocksdb/issues/1018 Test Plan: make check Reviewers: sdong, andrewkr, IslamAbdelRahman, yhchiang, igor Reviewed By: igor Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D55455 14 March 2016, 23:21:54 UTC
e778c34 Merge pull request #1035 from bureau14/master Added quasardb to the USERS.md file 14 March 2016, 22:50:55 UTC
back to top