Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
03d7c1c Adam Retter 29 April 2016, 18:56:10 UTC Bugfix to ensure that logging can be achieved from threads that are not known to the JVM 29 April 2016, 18:56:10 UTC
1d2e4ef Reid Horuff 29 April 2016, 01:23:29 UTC ldb support new WAL records 29 April 2016, 18:47:24 UTC
a92049e Yi Wu 29 April 2016, 18:35:00 UTC Added EventListener::OnTableFileCreationStarted() callback Summary: Added EventListener::OnTableFileCreationStarted. EventListener::OnTableFileCreated will be called on failure case. User can check creation status via TableFileCreationInfo::status. Test Plan: unit test. Reviewers: dhruba, yhchiang, ott, sdong Reviewed By: sdong Subscribers: sdong, kradhakrishnan, IslamAbdelRahman, andrewkr, yhchiang, leveldb, ott, dhruba Differential Revision: https://reviews.facebook.net/D56337 29 April 2016, 18:35:00 UTC
e8115ce PraveenSinghRao 29 April 2016, 05:50:26 UTC Revert "Use async file handle for better parallelism (#1049)" (#1105) This reverts commit b54c34742412af0001a69c2f7d909bc05e1ea71f. Revert async file handle change as it causes failures with appveyor 29 April 2016, 05:50:26 UTC
6a14f7a sdong 08 April 2016, 00:40:42 UTC Change several option defaults Summary: Changing several option defaults: options.max_open_files changes from 5000 to -1 options.base_background_compactions changes from max_background_compactions to 1 options.wal_recovery_mode changes from kTolerateCorruptedTailRecords to kTolerateCorruptedTailRecords options.compaction_pri changes from kByCompensatedSize to kByCompensatedSize Test Plan: Write unit tests to see OldDefaults() works as expected. Reviewers: IslamAbdelRahman, yhchiang, igor Reviewed By: igor Subscribers: MarkCallaghan, yiwu, kradhakrishnan, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56427 29 April 2016, 00:50:58 UTC
c6c770a Peter Mattis 28 April 2016, 23:48:03 UTC Use prefix_same_as_start to avoid iteration in FindNextUserEntryInternal. (#1102) This avoids excessive iteration in tombstone fields. 28 April 2016, 23:48:03 UTC
992a8f8 sdong 28 April 2016, 23:04:31 UTC Not enable jemalloc status printing if USE_CLANG=1 Summary: Warning is printed out with USE_CLANG=1 when including jemalloc.h. Disable it in that case. Test Plan: Run db_bench with USE_CLANG=1 and not. Make sure they can all build and jemalloc status is printed out in the case where USE_CLANG is not set. Reviewers: andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57399 28 April 2016, 23:16:14 UTC
029022b Islam AbdelRahman 28 April 2016, 22:59:33 UTC Fix crash_test Summary: crash_test grep for 'fail' string in the output and if found it consider that we failed. Update the output to use something else Test Plan: make crash_test (still running) Reviewers: yhchiang, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57381 28 April 2016, 22:59:33 UTC
a06faa6 Andrew Kryczka 28 April 2016, 22:11:28 UTC Skip PresetCompressionDict test for lite Summary: This test relies on "rocksdb.num-files-at-levelN" property that isn't implemented in rocksdb lite. So we will compile it only for non-lite builds. Test Plan: $ make -j40 check 'OPT=-g -DROCKSDB_LITE' Reviewers: sdong, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57387 28 April 2016, 22:11:28 UTC
e7899c6 Dmitri Smirnov 28 April 2016, 18:39:12 UTC Fix build issue. (#1103) 28 April 2016, 18:39:12 UTC
0f428c5 Andrew Kryczka 28 April 2016, 17:42:10 UTC Fix compression dictionary clang osx error Summary: There was one narrowing conversion in D52287 that only showed up with clang on osx. Test Plan: $ make clean && USE_CLANG=1 DISABLE_JEMALLOC=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j32 check Reviewers: sdong, lightmark, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57357 28 April 2016, 17:42:10 UTC
6d4832a Li Peng 28 April 2016, 09:30:44 UTC Merge pull request #1101 from flyd1005/wip-fix-typo fix typos and remove duplicated words 28 April 2016, 09:30:44 UTC
af70f9a Islam AbdelRahman 28 April 2016, 04:00:22 UTC Fix typo in build_tools/fbcode_config.sh Summary: Fix typo in build_tools/fbcode_config.sh Test Plan: none Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57363 28 April 2016, 04:00:22 UTC
54de13a Andrew Kryczka 28 April 2016, 01:30:04 UTC Fix compression dictionary clang errors Summary: There were a few narrowing conversions that clang didn't like. Test Plan: $ make clean && USE_CLANG=1 DISABLE_JEMALLOC=1 TEST_TMPDIR=/dev/shm/rocksdb OPT=-g make -j32 check Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57351 28 April 2016, 01:30:04 UTC
0850bc5 Islam AbdelRahman 28 April 2016, 01:25:19 UTC Fix build on machines without jemalloc Summary: It looks like we mistakenly enable JEMALLOC even if it's not available on the machine, that's why travis is failing Test Plan: check on my devserver check on my mac Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57345 28 April 2016, 01:25:19 UTC
4032145 Andrew Kryczka 28 April 2016, 00:39:18 UTC Configurable compression in db_bench Summary: Made compression type and dictionary size configurable via environment variables. Depends on D52287. Test Plan: check these options are passed to the db. $ COMPRESSION_MAX_DICT_BYTES=65536 COMPRESSION_TYPE=LZ4 NUM_KEYS=10000000 DB_DIR=./tmp/ WAL_DIR=./tmp/ ./tools/benchmark.sh filluniquerandom ... $ grep Options.compression tmp/LOG 2016/04/22-19:11:30.397829 7f5f263a2980 Options.compression: LZ4 ... 2016/04/22-19:11:30.397837 7f5f263a2980 Options.compression_opts.max_dict_bytes: 65536 Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57141 28 April 2016, 00:39:18 UTC
843d2e3 Andrew Kryczka 28 April 2016, 00:36:03 UTC Shared dictionary compression using reference block Summary: This adds a new metablock containing a shared dictionary that is used to compress all data blocks in the SST file. The size of the shared dictionary is configurable in CompressionOptions and defaults to 0. It's currently only used for zlib/lz4/lz4hc, but the block will be stored in the SST regardless of the compression type if the user chooses a nonzero dictionary size. During compaction, computes the dictionary by randomly sampling the first output file in each subcompaction. It pre-computes the intervals to sample by assuming the output file will have the maximum allowable length. In case the file is smaller, some of the pre-computed sampling intervals can be beyond end-of-file, in which case we skip over those samples and the dictionary will be a bit smaller. After the dictionary is generated using the first file in a subcompaction, it is loaded into the compression library before writing each block in each subsequent file of that subcompaction. On the read path, gets the dictionary from the metablock, if it exists. Then, loads that dictionary into the compression library before reading each block. Test Plan: new unit test Reviewers: yhchiang, IslamAbdelRahman, cyan, sdong Reviewed By: sdong Subscribers: andrewkr, yoshinorim, kradhakrishnan, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D52287 28 April 2016, 00:36:03 UTC
ad573b9 Yueh-Hsuan Chiang 27 April 2016, 23:50:51 UTC Temporarily disable CompactFiles in db_stress in its default setting Summary: As db_stress with CompactFiles possibly catches a previous bug currently, temporarily disable CompactFiles in db_stress in its default setting to allows new bug to be detected while investigating the bug in CompactFiles. Test Plan: crash test Reviewers: sdong, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57333 27 April 2016, 23:50:51 UTC
1c80dfa Sergey Makarenko 27 April 2016, 23:23:33 UTC Print memory allocation counters Summary: Introduced option to dump malloc statistics using new option flag. Added new command line option to db_bench tool to enable this funtionality. Also extended build to support environments with/without jemalloc. Test Plan: 1) Build rocksdb using `make` command. Launch the following command `./db_bench --benchmarks=fillrandom --dump_malloc_stats=true --num=10000000` end verified that jemalloc dump is present in LOG file. 2) Build rocksdb using `DISABLE_JEMALLOC=1 make db_bench -j32` and ran the same db_bench tool and found the following message in LOG file: "Please compile with jemalloc to enable malloc dump". 3) Also built rocksdb using `make` command on MacOS to verify behavior in non-FB environment. Also to debug build configuration change temporary changed AM_DEFAULT_VERBOSITY = 1 in Makefile to see compiler and build tools output. For case 1) -DROCKSDB_JEMALLOC was present in compiler command line. For both 2) and 3) this flag was not present. Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57321 27 April 2016, 23:23:33 UTC
eb73980 Islam AbdelRahman 26 April 2016, 23:07:04 UTC Fix BackupableDBTest Summary: Fix BackupableDBTest.NoDoubleCopy and BackupableDBTest.DifferentEnvs by mocking the db files in db_env instead of backup_env_ Test Plan: make check -j64 Reviewers: sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57273 26 April 2016, 23:07:04 UTC
ac0e54b sdong 21 April 2016, 22:32:06 UTC CompactedDB should not be used if there is outstanding WAL files Summary: CompactedDB skips memtable. So we shouldn't use compacted DB if there is outstanding WAL files. Test Plan: Change to options.max_open_files = -1 perf context test to create a compacted DB, which we shouldn't do. Reviewers: yhchiang, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57057 26 April 2016, 21:22:07 UTC
d719b09 Islam AbdelRahman 26 April 2016, 19:41:07 UTC Introduce PinnedIteratorsManager (Reduce PinData() overhead / Refactor PinData) Summary: While trying to reuse PinData() / ReleasePinnedData() .. to optimize away some memcpys I realized that there is a significant overhead for using PinData() / ReleasePinnedData if they were called many times. This diff refactor the pinning logic by introducing PinnedIteratorsManager a centralized component that will be created once and will be notified whenever we need to Pin an Iterator. This implementation have much less overhead than the original implementation Test Plan: make check -j64 COMPILE_WITH_ASAN=1 make check -j64 Reviewers: yhchiang, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56493 26 April 2016, 19:41:07 UTC
1995e34 Andrew Kryczka 26 April 2016, 19:33:30 UTC Retrieve file size from proper Env Summary: When db_env_ != backup_env_, InsertPathnameToSizeBytes() would use the wrong Env during backup creation. This happened because this function used backup_env_ instead of db_env_ to get WAL/data file sizes. This diff adds an argument to InsertPathnameToSizeBytes() indicating which Env to use. Test Plan: ran @anirbanb's BackupTestTool Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57159 26 April 2016, 19:33:30 UTC
7c14abf Islam AbdelRahman 26 April 2016, 06:02:14 UTC Improve BytewiseComparatorImpl::FindShortestSeparator Summary: The current implementation find the first different byte and try to increment it, if it cannot it return the original key we can improve this by keep going after the first different byte to find the first non 0xFF byte and increment it After trying this patch on some logdevice sst files I see decrease in there index block size by 8.5% Test Plan: existing tests and updated test Reviewers: yhchiang, andrewkr, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56241 26 April 2016, 06:02:14 UTC
f3eb0b5 Islam AbdelRahman 26 April 2016, 01:18:35 UTC Make EventListenerTest.CompactionReasonLevel more deterministic Summary: In this test some times automatic compactions do everything and Manual compaction become a no-op. Update the test to make sure manual compaction is not a no-op Test Plan: run the test Reviewers: andrewkr, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57189 26 April 2016, 01:18:35 UTC
7b78d62 sdong 21 April 2016, 23:58:38 UTC Shouldn't report default column family's compaction stats as DB compaction stats Summary: Now we collect compaction stats per column family, but report default colum family's stat as compaction stats for DB. Fix it by reporting compaction stats per column family instead. Test Plan: Run db_bench with --num_column_families=4 and see the number fixed. Reviewers: IslamAbdelRahman, yhchiang Reviewed By: yhchiang Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57063 25 April 2016, 20:56:59 UTC
995353e Dhruba Borthakur 25 April 2016, 13:27:27 UTC Fix null-pointer-dereference detected by Infer (https://github.com/facebook/infer) Test Plan: make check Reviewers: leveldb, sdong Reviewed By: sdong Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D57165 25 April 2016, 19:09:36 UTC
24110ce Yueh-Hsuan Chiang 25 April 2016, 19:01:01 UTC Correct Statistics FLUSH_WRITE_BYTES Summary: In https://reviews.facebook.net/D56271, we fixed an issue where we consider flush as compaction. However, that makes us mistakenly count FLUSH_WRITE_BYTES twice (one in flush_job and one in db_impl.) This patch removes the one incremented in db_impl. Test Plan: db_test Reviewers: yiwu, andrewkr, IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57111 25 April 2016, 19:01:01 UTC
b71c4e6 dx9 22 April 2016, 23:49:12 UTC Alpine Linux Build (#990) * Musl libc does not provide adaptive mutex. Added feature test for PTHREAD_MUTEX_ADAPTIVE_NP. * Musl libc does not provide backtrace(3). Added a feature check for backtrace(3). * Fixed compiler error. * Musl libc does not implement backtrace(3). Added platform check for libexecinfo. * Alpine does not appear to support gcc -pg option. By default (gcc has PIE option enabled) it fails with: gcc: error: -pie and -pg|p|profile are incompatible when linking When -fno-PIE and -nopie are used it fails with: /usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find gcrt1.o: No such file or directory Added gcc -pg platform test and output PROFILING_FLAGS accordingly. Replaced pg var in Makefile with PROFILING_FLAGS. * fix segfault when TEST_IOCTL_FRIENDLY_TMPDIR is undefined and default candidates are not suitable * use ASSERT_DOUBLE_EQ instead of ASSERT_EQ * When compiled with ROCKSDB_MALLOC_USABLE_SIZE UniversalCompactionFourPaths and UniversalCompactionSecondPathRatio tests fail due to premature memtable flushes on systems with 16-byte alignment. Arena runs out of block space before GenerateNewFile() completes. Increased options.write_buffer_size. 22 April 2016, 23:49:12 UTC
90ffed1 Igor Canadi 22 April 2016, 22:53:32 UTC Update USERS.md with link to LinkedIn blog post (#1088) 22 April 2016, 22:53:32 UTC
99a3bf8 Naitik Shah 22 April 2016, 20:49:59 UTC Merge pull request #1068 from daaku/c-purge-old-backups rocksdb_backup_engine_purge_old_backups for C libraries 22 April 2016, 20:49:59 UTC
b54c347 PraveenSinghRao 22 April 2016, 20:27:33 UTC Use async file handle for better parallelism (#1049) 22 April 2016, 20:27:33 UTC
c146c9b Naitik Shah 22 April 2016, 20:25:05 UTC rocksdb_create_mem_env to allow C libraries to create mem env (#1066) 22 April 2016, 20:25:05 UTC
6da70c5 Naitik Shah 22 April 2016, 20:24:09 UTC expose more options in the c api (#1067) 22 April 2016, 20:24:09 UTC
6f01687 Naitik Shah 22 April 2016, 20:22:21 UTC C rocksdb_create_iterators to expose NewIterators (#1069) 22 April 2016, 20:22:21 UTC
644f978 Yueh-Hsuan Chiang 21 April 2016, 21:47:23 UTC Fix RocksDB Lite build in db_stress Summary: Fix RocksDB Lite build in db_stress Test Plan: OPT=-DROCKSDB_LITE db_stress Reviewers: IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D57045 21 April 2016, 21:47:23 UTC
5bd4022 Islam AbdelRahman 21 April 2016, 17:16:28 UTC Add comparator, merge operator, property collectors to SST file properties (again) Summary: This is the original diff that I have landed and reverted and now I want to land again https://reviews.facebook.net/D34269 For old SST files we will show ``` comparator name: N/A merge operator name: N/A property collectors names: N/A ``` For new SST files with no merge operator name and with no property collectors ``` comparator name: leveldb.BytewiseComparator merge operator name: nullptr property collectors names: [] ``` for new SST files with these properties ``` comparator name: leveldb.BytewiseComparator merge operator name: UInt64AddOperator property collectors names: [DummyPropertiesCollector1,DummyPropertiesCollector2] ``` Test Plan: unittests Reviewers: andrewkr, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56487 21 April 2016, 17:16:28 UTC
7a6045a Andrew Kryczka 21 April 2016, 01:53:37 UTC fix typo in HISTORY.md 21 April 2016, 01:53:37 UTC
73a847e Andrew Kryczka 21 April 2016, 01:46:54 UTC Add per-level compression ratio property Summary: This is needed so we can measure compression ratio improvements achieved by D52287. The property compares raw data size against the total file size for a given level. If the level is empty it should return 0.0. Test Plan: new unit test Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56967 21 April 2016, 01:46:54 UTC
ee221d2 Dmitri Smirnov 20 April 2016, 05:54:24 UTC Introduce XPRESS compresssion on Windows. (#1081) Comparable with Snappy on comp ratio. Implemented using Windows API, does not require external package. Avaiable since Windows 8 and server 2012. Use -DXPRESS=1 with CMake to enable. 20 April 2016, 05:54:24 UTC
874c96a flabby 19 April 2016, 21:51:24 UTC Merge pull request #1083 from flabby/master fix typo in comment of options.h 19 April 2016, 21:51:24 UTC
6cbffd5 Yueh-Hsuan Chiang 19 April 2016, 21:36:09 UTC Enable testing CompactFiles in db_stress Summary: Enable testing CompactFiles in db_stress by adding flag test_compact_files to db_stress. Test Plan: ./db_stress --test_compact_files=1 --compaction_style=0 --allow_concurrent_memtable_write=false --ops_per_thread=100000 ./db_stress --test_compact_files=1 --compaction_style=1 --allow_concurrent_memtable_write=false --ops_per_thread=100000 Sample output (note that it's normal to have some CompactFiles() failed): Stress Test : 491.891 micros/op 65054 ops/sec : Wrote 21.98 MB (0.45 MB/sec) (45% of 3200352 ops) : Wrote 1440728 times : Deleted 441616 times : Single deleted 38181 times : 319251 read and 19025 found the key : Prefix scanned 640520 times : Iterator size sum is 9691415 : Iterated 319704 times : Got errors 0 times : 1323 CompactFiles() succeed : 32 CompactFiles() failed 2016/04/11-15:50:58 Verification successful Reviewers: sdong, IslamAbdelRahman, kradhakrishnan, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56565 19 April 2016, 21:36:09 UTC
b95510d Islam AbdelRahman 19 April 2016, 21:05:48 UTC Fix DBTest.RateLimitedDelete flakiness Summary: We need to enable sync_point processing before creating the SstFileManager to ensure that we are holding the bg delete scheduler thread from running Test Plan: run the test debug using printf Reviewers: sdong, yhchiang, yiwu, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56871 19 April 2016, 21:05:48 UTC
6356b4d Islam AbdelRahman 19 April 2016, 20:57:05 UTC Fix nullptr dereference in adaptive_table Summary: @dulmarod Ran infer on RocksDB and found that we dereference nullptr in adaptive_table https://fb.facebook.com/groups/rocksdb/permalink/1046374415411173/ Test Plan: make check -j64 Reviewers: sdong, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56973 19 April 2016, 20:57:05 UTC
9385fd7 Andrew Kryczka 19 April 2016, 17:32:31 UTC Delete deprecated backup classes in Java Summary: Corresponding change to D56331. Test Plan: Now build succeeds: $ make jclean && make rocksdbjava Reviewers: yhchiang, IslamAbdelRahman, adamretter Reviewed By: adamretter Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56913 19 April 2016, 17:32:31 UTC
a2466c8 Yueh-Hsuan Chiang 18 April 2016, 21:43:33 UTC [db_stress] Make subcompaction random in crash_test Summary: Make subcompaction random in crash_test Test Plan: make crash_test and verify whether subcompaction changes randomly Reviewers: IslamAbdelRahman, kradhakrishnan, yiwu, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56571 18 April 2016, 21:43:33 UTC
c3c389d Andrew Kryczka 18 April 2016, 21:34:45 UTC Fix column label for L0 write sum Summary: This is taken from the "Write(GB)" column in compaction stats, so the units should be GB, not MB. Test Plan: none Reviewers: sdong, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56889 18 April 2016, 21:34:45 UTC
ec84bef Andrew Kryczka 18 April 2016, 20:01:10 UTC New legocastle output parsing Summary: Added a python script to parse combined stdout/stderr of legocastle steps. Previously we just matched words like 'Failure', which didn't work since even our test names matched that pattern. I went through all the legocastle steps to come up with strict failure regexes for the common failure cases. There is also some more complex logic to present gtest failures, since the test name and failure message are not on the same line. There will definitely be error cases that don't match any of these patterns, so we can iterate on it over time. Test Plan: no end-to-end test. I ran the legocastle steps locally and piped to my script, then verified output, e.g., $ set -o pipefail && TEST_TMPDIR=/dev/shm/rocksdb COMPILE_WITH_ASAN=1 OPT=-g make J=1 asan_check |& /usr/facebook/ops/scripts/asan_symbolize.py -d |& python build_tools/error_filter.py asan ==2058029==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000a414 at pc 0x4c12f6 bp 0x7ffcfb7a0520 sp 0x7ffcfb7a0518 Reviewers: kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56691 18 April 2016, 20:01:10 UTC
725184b Yi Wu 18 April 2016, 18:34:11 UTC Fix db_block_cache_test in lite build Summary: D56715 move some of the tests from db_test to db_block_cache_test. Some of them should be disabled in lite build. Test Plan: make check -j32 OPT='-DROCKSDB_LITE' make check -j32 Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56907 18 April 2016, 18:34:11 UTC
290883d Yi Wu 18 April 2016, 17:47:10 UTC Fix lite build Summary: Fix rocksdb lite build after D56715. Test Plan: make -j40 'OPT=-g -DROCKSDB_LITE' Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56895 18 April 2016, 17:47:10 UTC
23089fd sdong 16 April 2016, 00:13:12 UTC write_callback_test: clean test directory before running tests Summary: write_callback_test fails if previous run didn't finish cleanly. Clean the DB before runing the test. Test Plan: Run the test that see it doesn't fail any more. Reviewers: andrewkr, yhchiang, yiwu, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56859 18 April 2016, 17:18:41 UTC
792762c Yi Wu 18 April 2016, 16:42:50 UTC Split db_test.cc Summary: Split db_test.cc into several files. Moving several helper functions into DBTestBase. Test Plan: make check Reviewers: sdong, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: dhruba, andrewkr, kradhakrishnan, yhchiang, leveldb, sdong Differential Revision: https://reviews.facebook.net/D56715 18 April 2016, 16:42:50 UTC
40b840f Andrew Kryczka 18 April 2016, 16:04:14 UTC Delete deprecated *BackupableDB interface for backups Summary: This interface is redundant and has been deprecated for a while. It's also unused internally. Let's delete it. I moved the comments to the corresponding functions in BackupEngine/ BackupEngineReadOnly. This caused the diff tool to not work cleanly. Test Plan: unit tests $ ./backupable_db_test Reviewers: yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56331 18 April 2016, 16:04:14 UTC
6affd45 Yi Wu 18 April 2016, 05:34:56 UTC Make more tests run in parallel Summary: Generate t/run-* scripts to run tests in $PARALLEL_TEST separately, then make check_0 rule execute all of them. Run `time make check` after running `make all`. master: 71 sec with this diff: 63 sec. It seems moving more tests to $PARALLEL_TEST doesn't help improve test time though. Test Plan: Run the following make check J=16 make check J=1 make check make valgrind_check J=1 make valgrind_check J=16 make_valgrind_check Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: leveldb, kradhakrishnan, dhruba, andrewkr, yhchiang Differential Revision: https://reviews.facebook.net/D56805 18 April 2016, 05:34:56 UTC
47833e0 Victor Tyutyunov 16 April 2016, 08:48:17 UTC Merge branch 'master' of github.com:facebook/rocksdb 16 April 2016, 08:48:17 UTC
e5c614e Victor Tyutyunov 12 April 2016, 17:35:15 UTC Fixing snapshot 0 assertion Summary: Solution is not to change db sequence number to start from 1 because 0 value is used in multiple other places. Fix covers only compact_iterator::findEarliestVisibleSnapshot with updated logic to support snapshot's numbering starting from 0. Test Plan: run: make all check it should pass all tests Reviewers: IslamAbdelRahman, sdong Reviewed By: sdong Subscribers: lgalanis, mgalushka, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56601 16 April 2016, 08:47:15 UTC
1b1adeb Victor Tyutyunov 12 April 2016, 17:35:15 UTC Fixing snapshot 0 assertion Test Plan: TBD Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56601 Cosmetic changes and comment update 16 April 2016, 08:45:31 UTC
6d436a3 sdong 16 April 2016, 00:00:56 UTC DBTest.HardLimit made more deterministic Summary: In DBTest.HardLimit, multiple flushes may merge into one, based on thread scheduling. Avoid it by waiting each flush to finish before generating the next one. Test Plan: Run test in parallel several times and see it doesn't fail any more. Reviewers: yhchiang, kradhakrishnan, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: yiwu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56853 16 April 2016, 00:36:57 UTC
994d9bc Islam AbdelRahman 15 April 2016, 23:21:03 UTC Make parallel valgrind watch-log more readable Summary: Before {F1131675} After {F1131681} This will have no effect on normal make check Test Plan: make watch-log Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56847 15 April 2016, 23:21:03 UTC
9d35ae6 sdong 15 April 2016, 22:44:42 UTC Make DBTestUniversalCompaction.IncreaseUniversalCompactionNumLevels more deterministic Summary: DBTestUniversalCompaction, IncreaseUniversalCompactionNumLevels fails one in about 30 runs when running in parallel. We wait for compaction after each flush to make the compaction behavior deterministic. Test Plan: Run the test 1000 times in parallel and it still passes. Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56841 15 April 2016, 23:16:53 UTC
cea8ed9 sdong 15 April 2016, 21:34:53 UTC Fix backupable_db_test test cases that can't run by itself Summary: Several of backupable_db_test fails if running standalone, because of directory missing. Fix it by: (1) garbage collector skips shared directory if it doesn't exit (2) BackupableDBTest.Issue921Test to create the parent directory of the backup directory fist. Test Plan: Run the tests individually and make sure they pass Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56829 15 April 2016, 22:48:57 UTC
4b6833a sdong 14 April 2016, 20:56:29 UTC Rename options.compaction_measure_io_stats to options.report_bg_io_stats and include flush too. Summary: It is useful to print out IO stats in flush jobs too. Extend options.compaction_measure_io_stats to flush jobs and raname it. Test Plan: Try db_bench and see the stats are printed out. Reviewers: yhchiang Reviewed By: yhchiang Subscribers: kradhakrishnan, yiwu, IslamAbdelRahman, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56769 15 April 2016, 17:22:18 UTC
3894603 sdong 14 April 2016, 02:27:52 UTC Allow valgrind_check to run in parallel Summary: Extend "J=<parallel>" to valgrind_check. For DBTest, modify the script to run valgrind. For other tests, prefix launch command with valgrind. Test Plan: Run valgrind_check with J=1 and J>1 and make sure tests run under valgrind. Manually change codes to introduce memory leak and make sure "make watch-log" correctly report it. Reviewers: yhchiang, yiwu, andrewkr, kradhakrishnan, IslamAbdelRahman Reviewed By: kradhakrishnan, IslamAbdelRahman Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56727 15 April 2016, 01:33:04 UTC
c9d668c Dmitri Smirnov 15 April 2016, 00:33:53 UTC Fix unit tests issues on Windows (#1078) 15 April 2016, 00:33:53 UTC
083cadc krad 14 April 2016, 22:04:10 UTC Minor fix to Java sandcastle job definition Summary: proxy config tools are not available in sandcastle environment. Hard coding the proxy information in the job. Test Plan: Run manually Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56775 14 April 2016, 22:05:54 UTC
80b74a6 Sherlock 14 April 2016, 17:50:34 UTC Include ldb tool in the windows build (#914) 14 April 2016, 17:50:34 UTC
7c14d11 krad 14 April 2016, 17:44:15 UTC Minor fix to java build job Summary: Java build is downloading junit when not available. Without the proxy settings exported that would not work. Test Plan: Run manually Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56751 14 April 2016, 17:46:02 UTC
535af52 sdong 13 April 2016, 20:02:33 UTC BlockBasedTable::PrefixMayMatch() to skip index checking if we can't find a filter block. Summary: In the case where we can't find a filter block, there is not much benefit of doing the binary search and see whether the index key has the prefix. With the change, we blindly return true if we can't get the filter. It also fixes missing row cases for reverse comparator with full bloom. Test Plan: Add a test case that used to fail. Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, hermanlee4, yoshinorim, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56697 14 April 2016, 02:06:48 UTC
09be5ca krad 13 April 2016, 22:56:46 UTC Minor fix to sandcastle java job definition Summary: System was not able to locate javac Test Plan: Run command manually and observe output Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56709 13 April 2016, 22:59:05 UTC
1aeca97 Islam AbdelRahman 13 April 2016, 22:03:08 UTC Release RocksDB 4.7 Summary: Bump the version and update HISTORY.md Test Plan: none Reviewers: yhchiang, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56469 13 April 2016, 22:03:08 UTC
dfc3de8 Islam AbdelRahman 13 April 2016, 21:22:29 UTC Split Travis unittests Job Summary: - Update Makefile check_some command to run a subset of the tests - Update travis config to split the unittests job into 2 jobs -- job testing db_test, db_test2 -- job testing the rest of the unittests Test Plan: Run the new travis.yml on my own branch https://travis-ci.org/facebook/rocksdb/builds/122691453 Reviewers: kradhakrishnan, yhchiang, sdong, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56673 13 April 2016, 21:22:29 UTC
c2c8fe4 krad 13 April 2016, 18:38:47 UTC Add Java job for sandcastle Summary: Adding java build job definition to sandcastle. Test Plan: ./build_tools/rocksdb-lego-determinator java_build Reviewers: yhchiang, sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56679 13 April 2016, 18:40:51 UTC
19ef3de Islam AbdelRahman 13 April 2016, 17:38:45 UTC Fix ManualCompactionPartial test flakiness Summary: The reason for this test flakiness is that we try to verify that number of files in L0 is 3 after flushing the 3rd file although we may have a compaction running in the background that may finish before we do the check and the 3 L0 files are converted to 1 L1 file Test Plan: Run a modified version of the test that sleep before doing the check Reviewers: sdong, andrewkr, kradhakrishnan, yhchiang Reviewed By: yhchiang Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56643 13 April 2016, 17:38:45 UTC
b345b36 Jay Edgar 12 April 2016, 22:09:28 UTC Add a minimum value for the refill bytes per period value Summary: If the user specified a small enough value for the rate limiter's bytes per second, the calculation for the number of refill bytes per period could become zero which would effectively cause the server to hang forever. Test Plan: Existing tests Reviewers: sdong, yhchiang, igor Reviewed By: igor Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56631 13 April 2016, 16:01:42 UTC
dff4c48 sdong 12 April 2016, 20:56:24 UTC BlockBasedTable::PrefixMayMatch: no need to find data block after full bloom checking Summary: Full block checking should be a good enough indication of prefix existance. No need to further check data block. This also fixes wrong results when using prefix bloom and reverse bitwise comparator. Test Plan: Will add a unit test. Reviewers: yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: hermanlee4, yoshinorim, yiwu, kradhakrishnan, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56625 12 April 2016, 23:25:54 UTC
0353b85 krad 12 April 2016, 17:37:58 UTC Propagate sandcastle run error to UI Summary: Currently the code does not propagate the sandcastle precommit test run error status to UI. This can confuse the developer when searching for errors. With this change, all success should be in green and all errors should be in red Test Plan: Submit the diff (and hopefully something will fail) Reviewers: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56595 12 April 2016, 23:18:37 UTC
b885f33 Islam AbdelRahman 12 April 2016, 22:05:56 UTC Parallelize travis jobs Summary: Update travis.yml to split the work into 3 sub jobs - running unittests - Building and testing java - building ROCKSDB_LITE Test Plan: https://travis-ci.org/facebook/rocksdb/builds/122433169 Reviewers: sdong, yhchiang, andrewkr, kradhakrishnan Reviewed By: kradhakrishnan Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56607 12 April 2016, 22:05:56 UTC
71303e0 Hyunyoung Lee 12 April 2016, 21:05:09 UTC Update db_bench_tool.cc (#1073) * Update db_bench_tool.cc I fixed the wrong letters, LevelDB -> rocksDB, because I thought of LevelDB as the wrong presentation. the following show my fix : fprintf(stderr, "LevelDB: version %d.%d\n", kMajorVersion, kMinorVersion); -----------------> fprintf(stderr, "rocksDB: version %d.%d\n", kMajorVersion, kMinorVersion); * Update db_bench_tool.cc * Update db_bench_tool.cc 12 April 2016, 21:05:09 UTC
63cf15b sdong 11 April 2016, 21:57:27 UTC Fix option settable tests Summary: In option settable tests, bytes for pointers are not all skipped, so that they may be the same as the special character and cause false positive. Test Plan: Run the test. Manually verify the issue is not there any more. Reviewers: IslamAbdelRahman, andrewkr Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56553 12 April 2016, 21:03:35 UTC
e208575 Shu Zhang 12 April 2016, 04:08:26 UTC using java7 in runtime for hdfs env (#1072) 12 April 2016, 04:08:26 UTC
13e6c8e Yueh-Hsuan Chiang 12 April 2016, 03:15:52 UTC Relax an assertion in Compaction::ShouldStopBefore Summary: In some case, it is possible to have two concesutive SST files might sharing same boundary keys. However, in the assertion in Compaction::ShouldStopBefore, it exclude such possibility. This patch fix this issue by relaxing the assertion to allow the equal case. Test Plan: rocksdb tests Reviewers: IslamAbdelRahman, kradhakrishnan, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55875 12 April 2016, 03:15:52 UTC
ae21d71 Yueh-Hsuan Chiang 12 April 2016, 02:59:25 UTC Fixed a bug in RocksDB Statistics where flush is considered as compaction Summary: Fixed a bug in RocksDB Statistics where flush is considered as compaction Test Plan: unit test Reviewers: sdong, IslamAbdelRahman, rven, kradhakrishnan, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56271 12 April 2016, 02:59:25 UTC
8e0e22f Andrew Kryczka 12 April 2016, 02:21:00 UTC Fix Windows build by replacing strings.h include Summary: strings.h header does not exist on Windows. So, we can try another way to compare strings ignoring case. Test Plan: built and ran: $ ./ldb_cmd_test Reviewers: sdong, yhchiang, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56535 12 April 2016, 02:21:00 UTC
5675d50 Islam AbdelRahman 11 April 2016, 23:46:55 UTC Revert travis commit Summary: Revert travis commit pushed unintentionally Test Plan: none Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56559 11 April 2016, 23:46:55 UTC
91f0f1f Islam AbdelRahman 11 April 2016, 23:37:22 UTC fix travis 11 April 2016, 23:37:22 UTC
a23c605 sdong 11 April 2016, 18:39:51 UTC Don't run DBOptionsAllFieldsSettable under valgrind Summary: Test DBOptionsAllFieldsSettable sometimes fails under valgrind. Move option settable tests to a separate test file and disable it in valgrind.. Test Plan: Run valgrind test and make sure the test doesn't run. Reviewers: andrewkr, IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: kradhakrishnan, yiwu, yhchiang, leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56529 11 April 2016, 21:24:55 UTC
30d72ee sdong 11 April 2016, 19:15:46 UTC PrefixTest.PrefixAndWholeKeyTest should run against a different directory from prefix_test Summary: PrefixTest.PrefixAndWholeKeyTest runs against the same directory as prefix_test, which sometimes fail parallel tests. Fix it. Test Plan: Run it in parallel and see it doesn't fail anymore. Reviewers: andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56541 11 April 2016, 20:02:56 UTC
0e3cc2c Andrew Kryczka 09 April 2016, 01:50:18 UTC Add column family info to TableProperties::ToString() Summary: This is used at least by `sst_dump --show_properties` Test Plan: - default CF ``` $ ./sst_dump --show_properties --file=./tmp-db/000007.sst | grep 'column family' column family ID: 0 column family name: default ``` - custom CF ``` $ ./sst_dump --show_properties --file=./tmp-db/000012.sst | grep 'column family' column family ID: 1 column family name: col-fam-1 ``` - no CF ``` $ ./sst_dump --show_properties --file=./tmp-db/000017.sst | grep 'column family' column family ID: N/A column family name: N/A ``` Reviewers: IslamAbdelRahman Reviewed By: IslamAbdelRahman Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56499 09 April 2016, 01:50:18 UTC
2448f80 Jay Edgar 06 April 2016, 22:53:52 UTC Make sure that if use_mmap_reads is on use_os_buffer is also on Summary: The code assumes that if use_mmap_reads is on then use_os_buffer is also on. This make sense as by using memory mapped files for reading you are expecting the OS to cache what it needs. Add code to make sure the user does not turn off use_os_buffer when they turn on use_mmap_reads Test Plan: New test: DBTest.MMapAndBufferOptions Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56397 08 April 2016, 21:30:15 UTC
114a1b8 Andrew Kryczka 08 April 2016, 20:09:19 UTC Fix build errors for windows Summary: - Need to use unsigned long long for 64-bit literals on windows - Need size_t for backup meta-file length since clang doesn't let us assign size_t to int Test Plan: backupable_db_test and options_test Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56391 08 April 2016, 20:09:19 UTC
0522990 Islam AbdelRahman 08 April 2016, 19:05:02 UTC Improve sst_dump help message Summary: Current Message ``` sst_dump [--command=check|scan|none|raw] [--verify_checksum] --file=data_dir_OR_sst_file [--output_hex] [--input_key_hex] [--from=<user_key>] [--to=<user_key>] [--read_num=NUM] [--show_properties] [--show_compression_sizes] [--show_compression_sizes [--set_block_size=<block_size>]] ``` New message ``` sst_dump --file=<data_dir_OR_sst_file> [--command=check|scan|raw] --file=<data_dir_OR_sst_file> Path to SST file or directory containing SST files --command=check|scan|raw check: Iterate over entries in files but dont print anything except if an error is encounterd (default command) scan: Iterate over entries in files and print them to screen raw: Dump all the table contents to <file_name>_dump.txt --output_hex Can be combined with scan command to print the keys and values in Hex --from=<user_key> Key to start reading from when executing check|scan --to=<user_key> Key to stop reading at when executing check|scan --read_num=<num> Maximum number of entries to read when executing check|scan --verify_checksum Verify file checksum when executing check|scan --input_key_hex Can be combined with --from and --to to indicate that these values are encoded in Hex --show_properties Print table properties after iterating over the file --show_compression_sizes Independent command that will recreate the SST file using 16K block size with different compressions and report the size of the file using such compression --set_block_size=<block_size> Can be combined with --show_compression_sizes to set the block size that will be used when trying different compression algorithms ``` Test Plan: none Reviewers: yhchiang, andrewkr, kradhakrishnan, yiwu, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56325 08 April 2016, 19:05:02 UTC
0930e5e sdong 04 April 2016, 19:26:01 UTC Update comments on include/rocksdb/perf_context.h Summary: Some grammer mistakes in code comments in include/rocksdb/perf_context.h. Also polish it a liitlebit. Test Plan: Not needed Reviewers: IslamAbdelRahman, yhchiang, yiwu, andrewkr Reviewed By: andrewkr Subscribers: leveldb, andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56307 08 April 2016, 18:27:08 UTC
3b977bc Shu Zhang 07 April 2016, 21:34:29 UTC instructing people to use java7 for hdfs (#1063) 07 April 2016, 21:34:28 UTC
1518b73 sdong 07 April 2016, 20:51:47 UTC Change default number of cache shard bit to be 6 and max_file_opening_threads to be 16. Summary: Cache shard bit 4 is sometimes too small and 6 is a more common value picked by users. Make that default. It shouldn't hurt much to change options.max_file_opening_threads default to be 16, which will reduce the worst case DB open time. Test Plan: Run all existing tests. Reviewers: IslamAbdelRahman, yhchiang, kradhakrishnan, igor, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, MarkCallaghan, leveldb, dhruba Differential Revision: https://reviews.facebook.net/D55047 07 April 2016, 20:55:10 UTC
ada88b6 flabby 07 April 2016, 16:02:28 UTC fix wrong assignment of level0_stop_writes_trigger in spatialdb (#1061) 07 April 2016, 16:02:28 UTC
2391ef7 Andrew Kryczka 07 April 2016, 06:10:32 UTC Embed column family name in SST file Summary: Added the column family name to the properties block. This property is omitted only if the property is unavailable, such as when RepairDB() writes SST files. In a next diff, I will change RepairDB to use this new property for deciding to which column family an existing SST file belongs. If this property is missing, it will add it to the "unknown" column family (same as its existing behavior). Test Plan: New unit test: $ ./db_table_properties_test --gtest_filter=DBTablePropertiesTest.GetColumnFamilyNameProperty Reviewers: IslamAbdelRahman, yhchiang, sdong Reviewed By: sdong Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D55605 07 April 2016, 06:10:32 UTC
ab4c623 Igor Canadi 06 April 2016, 22:00:15 UTC Don't use version in the error message Summary: We use object `v` in the error message, which is not initialized if the edit is column family manipulation. This doesn't provide much useful info, so this diff is removing it. Instead, it dumps actual VersionEdit contents. Test Plan: compiles. would be great to get tests in version_set_test.cc that cover cases where a file write fails Reviewers: sdong, yhchiang, andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56349 06 April 2016, 22:00:15 UTC
d02eb8d Islam AbdelRahman 04 April 2016, 18:10:46 UTC Fix unused variable warning Summary: My last diff introduced a warning when compiling under release mode https://reviews.facebook.net/D55539 fix the warning Test Plan: DEBUG_LEVEL=0 make db_bench make check Reviewers: sdong Reviewed By: sdong Subscribers: andrewkr, dhruba Differential Revision: https://reviews.facebook.net/D56295 04 April 2016, 18:10:46 UTC
9278097 Igor Canadi 04 April 2016, 15:08:52 UTC Merge pull request #1056 from facebook/igorcanadi-patch-1 Create LANGUAGE-BINDINGS.md 04 April 2016, 15:08:52 UTC
cc87075 Aaron Gao 01 April 2016, 23:19:12 UTC No need to limit to 20 files in UpdateAccumulatedStats() if options.max_open_files=-1 Summary: There is a hardcoded constraint in our statistics collection that prevents reading properties from more than 20 SST files. This means our statistics will be very inaccurate for databases with > 20 files since additional files are just ignored. The purpose of constraining the number of files used is to bound the I/O performed during statistics collection, since these statistics need to be recomputed every time the database reopened. However, this constraint doesn't take into account the case where option "max_open_files" is -1. In that case, all the file metadata has already been read, so MaybeInitializeFileMetaData() won't incur any I/O cost. so this diff gets rid of the 20-file constraint in case max_open_files == -1. Test Plan: write into unit test db/db_properties_test.cc - "ValidateSampleNumber". We generate 20 files with 2 rows and 10 files with 1 row. If max_open_files !=-1, the `rocksdb.estimate-num-keys` should be (10*1 + 10*2)/20 * 30 = 45. Otherwise, it should be the ground truth, 50. {F1089153} Reviewers: andrewkr Reviewed By: andrewkr Subscribers: andrewkr, dhruba, leveldb Differential Revision: https://reviews.facebook.net/D56253 01 April 2016, 23:19:12 UTC
back to top