Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
29 August 2024, 10:07:27 UTC
sort by:
Revision Author Date Message Commit Date
91b32c0 Swen Kooij 24 November 2013, 04:45:02 UTC Enable position independent code on x64 24 November 2013, 04:45:02 UTC
0c93df9 kailiu 22 November 2013, 01:54:23 UTC Improve the readability of the TableProperties::ToString() 22 November 2013, 01:54:23 UTC
3e35aa6 Siying Dong 22 November 2013, 01:40:39 UTC Revert "Allow users to profile a query and see bottleneck of the query" This reverts commit 3d8ac31d7168c916d6f2f0729eb627b07d8f082b. 22 November 2013, 01:40:39 UTC
3d8ac31 Siying Dong 18 November 2013, 19:32:54 UTC Allow users to profile a query and see bottleneck of the query Summary: Provide a framework to profile a query in detail to figure out latency bottleneck. Currently, in Get(), Put() and iterators, 2-3 simple timing is used. We can easily add more profile counters to the framework later. Test Plan: Enable this profiling in seveal existing tests. Reviewers: haobo, dhruba, kailiu, emayanke, vamsi, igor CC: leveldb Differential Revision: https://reviews.facebook.net/D14001 22 November 2013, 00:29:57 UTC
56589ab kailiu 21 November 2013, 02:42:12 UTC Add TableOptions for BlockBasedTableFactory We are having more and more options to specify for this table so it makes sense to have a TableOptions for future extension. 21 November 2013, 02:42:12 UTC
db2e261 Kai Liu 20 November 2013, 22:55:33 UTC Fix the format in INSTALL.md 20 November 2013, 22:55:33 UTC
f5acb2e Kai Liu 20 November 2013, 22:54:53 UTC Add more guide for mac users. 20 November 2013, 22:54:53 UTC
618250b Kai Liu 20 November 2013, 22:38:17 UTC Update the installation guide for mac users. 20 November 2013, 22:38:17 UTC
1c8b819 kailiu 20 November 2013, 21:45:32 UTC Fix a memory leak happened in table_test 20 November 2013, 21:45:32 UTC
a617227 Haobo Xu 20 November 2013, 07:24:12 UTC [RocksDB] fix prefix_test Summary: user comparator needs to work if either input is prefix only. Test Plan: ./prefix_test --write_buffer_size=100000 --total_prefixes=10000 --items_per_prefix=10 Reviewers: dhruba, igor Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D14241 20 November 2013, 17:16:23 UTC
0142d38 kailiu 20 November 2013, 09:17:44 UTC fix issue #11 URL: https://github.com/facebook/rocksdb/issues/11 20 November 2013, 09:17:44 UTC
6eb5649 kailiu 20 November 2013, 06:00:48 UTC Move flush_block_policy from Options to TableFactory Summary: Previously we introduce a `flush_block_policy_factory` in Options, however, that options is strongly releated to Table based tables. It will make more sense to move it to block based table's own factory class. Test Plan: make check to pass existing tests Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14211 20 November 2013, 06:00:48 UTC
469a9f3 Igor Canadi 20 November 2013, 05:01:48 UTC Fix two nasty use-after-free-bugs Summary: These bugs were caught by ASAN crash test. 1. The first one, in table/filter_block.cc is very nasty. We first reference entries_ and store the reference to Slice prev. Then, we call entries_.append(), which can change the reference. The Slice prev now points to junk. 2. The second one is a bug in a test, so it's not very serious. Once we set read_opts.prefix, we never clear it, so some other function might still reference it. Test Plan: asan crash test now runs more than 5 mins. Before, it failed immediately. I will run the full one, but the full one takes quite some time (5 hours) Reviewers: dhruba, haobo, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14223 20 November 2013, 05:01:48 UTC
8906ab5 Igor Canadi 20 November 2013, 00:44:40 UTC Split asan_check into asan_check and asan_crash_test 20 November 2013, 00:44:40 UTC
92d9050 Igor Canadi 20 November 2013, 00:33:24 UTC make asan_check Summary: Add asan_check rule to Makefile. After we add this, we will create Jenkins run that will check for asan errors! Test Plan: make asan_check Reviewers: dhruba, kailiu, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14205 20 November 2013, 00:33:24 UTC
1415f88 kailiu 20 November 2013, 00:29:42 UTC Improve the "table stats" Summary: The primary motivation of the changes is to make it easier to figure out the inside of the tables. * rename "table stats" to "table properties" since now we have more than "integers" to store in the property block. * Add filter block size to the basic table properties. * Whenever a table is built, we'll log the table properties (the sample output is in Test Plan). * Make an api to expose deleted keys. Test Plan: Passed all existing test. and the sample output of table stats: ================================================================== Basic Properties ------------------------------------------------------------------ # data blocks: 1 # entries: 1 raw key size: 9 raw average key size: 9 raw value size: 9 raw average value size: 0 data block size: 25 index block size: 27 filter block size: 18 (estimated) table size: 70 filter policy: rocksdb.BuiltinBloomFilter ================================================================== User collected properties: InternalKeyPropertiesCollector ------------------------------------------------------------------ kDeletedKeys: 1 ================================================================== Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14187 20 November 2013, 00:29:42 UTC
f045871 Igor Canadi 19 November 2013, 05:33:16 UTC Remove libevent Summary: We don't need that dependency Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14199 19 November 2013, 05:33:16 UTC
ec0acfb Igor Canadi 19 November 2013, 04:56:21 UTC Fix stack overflow Summary: Sure, let me put 8 bytes in that int32_t. Brought to you by ASAN! Test Plan: ttl_test Reviewers: dhruba, haobo, kailiu, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14193 19 November 2013, 04:56:21 UTC
07e8078 Igor Canadi 19 November 2013, 04:33:34 UTC Fix BZip constants Summary: We were using ZLIB constants in BZIP code path. This caused some errors, like: https://github.com/facebook/rocksdb/issues/8 Test Plan: make clean; make check Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14175 19 November 2013, 04:33:34 UTC
e51f55d Igor Canadi 19 November 2013, 04:18:45 UTC Use gcc4.7.1 on CentOS 5.2 Summary: For some reason, snappy on CentOS 5.2 when compiled with gcc 4.8.1 segfaults on strcmp. (!?) Add an if to compile with gcc4.7.1 if you're compiling on CentOS 5.2. Please update your devservers to CentOS 6. Test Plan: make clean; make check on both my devserver (CentOS 6) and dhruba's (CentOS 5.2) Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14169 19 November 2013, 04:18:45 UTC
7f156be Igor Canadi 18 November 2013, 21:50:34 UTC [fbcode] Also add glibc and libgcc includes Summary: We also need to use custom glibc and libgcc includes instead of system ones. Test Plan: 'make clean; make check'. Will also try on @dhruba's dev server. Reviewers: dhruba, kailiu Reviewed By: kailiu CC: leveldb, dhruba Differential Revision: https://reviews.facebook.net/D14157 18 November 2013, 21:50:34 UTC
0d25449 Igor Canadi 18 November 2013, 19:44:17 UTC Compilation.md Summary: Explained what it takes to compile it on linux. Test Plan: - Reviewers: dhruba, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D14151 18 November 2013, 19:44:17 UTC
f611aba Igor Canadi 18 November 2013, 19:40:16 UTC Move the compiler back to 4.8.1 + more small fixes Summary: 1. Moved the compiler back to 4.8.1 and uses Centos 5.2 binaries if OS is Centos 5.2. 2. Fixes this issue: https://github.com/facebook/rocksdb/issues/7 3. We use lot of c++11 features, so we can't pretend we can compile without them. Makes it a first class dependency. 4. Fix blob_store_test, which failes on Ubuntu with "too many files opened" error 5. Removed dependency on port/port_chromium.h, which does not even exist on our system Test Plan: make clean; make check Reviewers: dhruba, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D14145 18 November 2013, 19:40:16 UTC
6c6d5bc kailiu 17 November 2013, 08:44:16 UTC Add a compilation guide to rocksdb Summary: as title. Test Plan: no Reviewers: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14121 18 November 2013, 19:01:16 UTC
31295b0 Dhruba Borthakur 18 November 2013, 18:12:08 UTC Add License message to public header files. Summary: Add License message to public header files. Test Plan: Reviewers: CC: Task ID: # Blame Rev: 18 November 2013, 18:21:35 UTC
37eedfb Igor Canadi 18 November 2013, 18:20:32 UTC Move back to gcc4.7.1 Summary: Dhruba can't compile on gcc4.8.1 so I'm moving temporarily back to 4.7.1 until we figure out what's wrong with 4.8. on his server. Test Plan: It can compile on my devserver, but please 'arc patch' this diff and try compiling on your machine Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14139 18 November 2013, 18:20:32 UTC
ce26e9a Igor Canadi 18 November 2013, 06:05:00 UTC Remove snappy from RocksDB distribution Summary: Argumentation here: https://github.com/facebook/rocksdb/issues/9 Even though we include snappy in the distribution, we do not link with it if we don't have snappy installed on the system. Installing snappy is easy nowadays, just type: sudo apt-get install libsnappy-dev Test Plan: compile on ubuntu Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14133 18 November 2013, 06:05:00 UTC
fc61428 Igor Canadi 18 November 2013, 05:58:16 UTC Include <unistd.h> in db_test Summary: This is the only compile issue in Ubuntu. It might be better to include <unistd.h> only in env_posix and add Truncate function to Env, but since we use truncate only in db_test, I don't think it makes much sense. Test Plan: Rocksdb now compiles on Ubuntu! Reviewers: dhruba, kailiu Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14127 18 November 2013, 05:58:16 UTC
de9ce7d Igor Canadi 17 November 2013, 21:52:55 UTC Upgrading compiler to gcc4.8.1 Summary: Finally did it - the trick was in using --dynamic-linker option. This is first step to running ASAN. All of our code seems to compile just fine on 4.8.1. However, I still left fbcode.471.sh in the 'build_tools/' just in case. Test Plan: make clean; make Reviewers: dhruba, haobo, kailiu, emayanke, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14109 17 November 2013, 21:52:55 UTC
75df72f Kai Liu 17 November 2013, 06:59:22 UTC Change the logic in KeyMayExist() Summary: Previously in KeyMayExist(), if DB::Get() returns non-Status::OK(), we assumes key may not exist. However, as if index block is not in block cache, Status::Incomplete() will return. Worse still, if options::filter_delete is enabled, we may falsely ignore the "delete" operation: https://github.com/facebook/rocksdb/blob/master/db/write_batch.cc#L217-L220 This diff fixes this bug and will let crash-test pass. Test Plan: Ran: ./db_stress --test_batches_snapshots=1 --ops_per_thread=1000000 --threads=32 --write_buffer_size=4194304 --destroy_db_initially=1 --reopen=0 --readpercent=5 --prefixpercent=45 --writepercent=35 --delpercent=5 --iterpercent=10 --db=/home/kailiu/local/newer --max_key=100000000 --disable_seek_compaction=0 --mmap_read=0 --block_size=16384 --cache_size=1048576 --open_files=500000 --verify_checksum=1 --sync=0 --disable_wal=0 --disable_data_sync=0 --target_file_size_base=2097152 --target_file_size_multiplier=2 --max_write_buffer_number=3 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --filter_deletes=1 Previously we'll see crash happens very soon. Reviewers: igor, dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14115 17 November 2013, 09:00:34 UTC
97d8e57 kailiu 17 November 2013, 07:44:39 UTC make util/env_posix.cc work under mac Summary: This diff invoves some more complicated issues in the posix environment. Test Plan: works under mac os. will need to verify dev box. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14061 17 November 2013, 07:44:39 UTC
7604e2f Kai Liu 17 November 2013, 07:21:15 UTC Make the options in table_builder/block_builder less misleading Summary: By original design, the regular `block options` and index `block options` in table_builder is mutable. We can use ChangeOptions to change the options directly. However, with my last change, `BlockBuilder` no longer hold the reference to the index_block_options -- as a result, any changes made after the creation of index block builder will be of no effect. But still the code is very error-prone and developers can easily fall into the trap without aware of it. To avoid this problem from happening in the future, I deleted the `ChangeOptions` and the `index_block_options`, as well as many other changes to make it less misleading. Test Plan: make make check make release Reviewers: dhruba, haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13707 17 November 2013, 07:21:15 UTC
f7a2b97 Kai Liu 17 November 2013, 06:51:09 UTC Reformat CONTRIBUTING.md with less than 80 characters. 17 November 2013, 06:51:09 UTC
72756c7 Igor Canadi 17 November 2013, 01:46:29 UTC Merge pull request #5 from pborreli/typos Fixed typos 17 November 2013, 01:46:29 UTC
f468272 Igor Canadi 16 November 2013, 22:35:04 UTC Added CONTRIBUTING.md 16 November 2013, 22:35:33 UTC
443e04e Pascal Borreli 16 November 2013, 11:21:34 UTC Fixed typos 16 November 2013, 11:21:34 UTC
55baa3d Siying Dong 16 November 2013, 06:23:12 UTC Add an option to table_reader_bench to access the table from DB And Iterating non-existing prefix case. Summary: This patch adds an option to table_reader_bench that queries run against DB level (which has one table). It is useful if user wants to see the extra costs DB level introduces. Test Plan: Run the benchmark with and without the new parameter Reviewers: haobo, dhruba, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13863 16 November 2013, 06:23:12 UTC
3d56b06 Kai Liu 16 November 2013, 06:15:42 UTC Fix typo. 16 November 2013, 06:15:42 UTC
21905dd Igor Canadi 16 November 2013, 00:30:23 UTC Start DeleteFileTest with clean plate Summary: Remove all the files from the test dir before the test. The test failed when there were some old files still in the directory, since it checks the file counts. This is what caused jenkins' test failures. It was running fine on my machine so it was hard to repro. Test Plan: 1. create an extra 000001.log file in the test directory 2. run a ./deletefile_test - test failes 3. patch ./deletefile_test with this 4. test succeeds Reviewers: haobo, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D14097 16 November 2013, 00:30:23 UTC
bd82826 Kai Liu 15 November 2013, 23:36:37 UTC Fix a typo in README 15 November 2013, 23:36:37 UTC
29c931f Igor Canadi 15 November 2013, 06:42:02 UTC Avoid populating live set if we don't need to Summary: Also changed some comments Test Plan: ./deletefile_test Reviewers: haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D14091 15 November 2013, 06:42:02 UTC
a0ce3fd Igor Canadi 15 November 2013, 02:03:57 UTC PurgeObsoleteFiles() unittest Summary: Created a unittest that verifies that automatic deletion performed by PurgeObsoleteFiles() works correctly. Also, few small fixes on the logic part -- call version_set_->GetObsoleteFiles() in FindObsoleteFiles() instead of on some arbitrary positions. Test Plan: Created a unit test Reviewers: dhruba, haobo, nkg- Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D14079 15 November 2013, 02:03:57 UTC
94dde68 Vamsi Ponnekanti 13 November 2013, 22:50:29 UTC [Merge operand meant for key K is being applied on wrong key] Summary: We iterate until we find a different key than original key. ikey is pointing to next key when we break out of loop. After the loop we apply all merge operands meant for original key on the next key! Test Plan: Need to give a build to Marcin to test out. Revert Plan: OK Task ID: #3181932 Reviewers: haobo, emayanke, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D14073 15 November 2013, 01:13:24 UTC
e0ad0f2 Igor Canadi 14 November 2013, 22:05:15 UTC Fix bloom filters Summary: https://reviews.facebook.net/D13167 broke bloom filters. If filter is not in cache, we want to return true (safe thing). Am I right? Test Plan: when benchmarking https://reviews.facebook.net/D14031 I got different results when using bloom filters vs. when not using them. This fixed the issue. I will also be putting this change to the other diff, but that one will probably be in review for longer time. Reviewers: kailiu, dhruba, haobo Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D14085 14 November 2013, 22:05:15 UTC
fda8142 Igor Canadi 13 November 2013, 22:54:54 UTC Delete log files in the correct dir Summary: Log files are stored in wal_dir, not dbname_ Test Plan: deletfile_test Reviewers: nkg- Reviewed By: nkg- CC: leveldb Differential Revision: https://reviews.facebook.net/D14067 13 November 2013, 22:54:54 UTC
80bb81c Kai Liu 13 November 2013, 07:54:31 UTC Add the correct table_factory for tables in table_tests 13 November 2013, 07:54:31 UTC
88ba331 Kai Liu 13 November 2013, 06:46:51 UTC Add the index/filter block cache Summary: This diff leverage the existing block cache and extend it to cache index/filter block. Test Plan: Added new tests in db_test and table_test The correctness is checked by: 1. make check 2. make valgrind_check Performance is test by: 1. 10 times of build_tools/regression_build_test.sh on two versions of rocksdb before/after the code change. Test results suggests no significant difference between them. For the two key operatons `overwrite` and `readrandom`, the average iops are both 20k and ~260k, with very small variance). 2. db_stress. Reviewers: dhruba Reviewed By: dhruba CC: leveldb, haobo, xjin Differential Revision: https://reviews.facebook.net/D13167 13 November 2013, 06:46:51 UTC
aed9f1f Kai Liu 13 November 2013, 05:40:25 UTC The updated sed still doesn't work in mac, revert. 13 November 2013, 05:40:25 UTC
f3b3316 Kai Liu 13 November 2013, 05:26:19 UTC Fix a sed command issue that cannot generated *.d files Summary: The original sed command is not recognized by mac's sed, which generates ".d-e" extension instead of ".d" Test Plan: make clean && make -j32 13 November 2013, 05:26:19 UTC
22e1b04 Kai Liu 13 November 2013, 05:15:49 UTC Quick fix for a string format Summary: Fix one more string format issue that throws warning in mac 13 November 2013, 05:22:32 UTC
35460cc Kai Liu 13 November 2013, 05:02:03 UTC Fix the string format issue Summary: mac and our dev server has totally differnt definition of uint64_t, therefore fixing the warning in mac has actually made code in linux uncompileable. Test Plan: make clean && make -j32 13 November 2013, 05:05:39 UTC
d88d8ec Igor Canadi 13 November 2013, 04:32:07 UTC Fix deleting files Summary: One more fix! In some cases, our filenames start with "/". Apparently, env_ can't handle filenames with double // Test Plan: deletefile_test does not include this line in the LOG anymore: 2013/11/12-18:11:43.150149 7fe4a6fff700 RenameFile logfile #3 FAILED -- IO error: /tmp/rocksdbtest-3574/deletefile_test//000003.log: No such file or directory Reviewers: dhruba, haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D14055 13 November 2013, 04:32:07 UTC
2158776 kailiu 13 November 2013, 04:05:28 UTC Fixing the warning messages captured under mac os # Consider using `git commit -m 'One line title' && arc diff`. # You will save time by running lint and unit in the background. Summary: The work to make sure mac os compiles rocksdb is not completed yet. But at least we can start cleaning some warnings captured only by g++ from mac os.. Test Plan: ran make in mac os Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14049 13 November 2013, 04:05:28 UTC
c3dda72 Igor Canadi 13 November 2013, 00:09:57 UTC Update documentation Summary: Added more options for compaction settings + thread pools. Please check if thread pool description is correct. Test Plan: - Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14043 13 November 2013, 00:09:57 UTC
9df2b21 Igor Canadi 12 November 2013, 20:42:42 UTC Move fast and break things Summary: Broke the compile when I removed purge_log_after_memtable_flush. sorrybus Test Plan: make db_bench works now Reviewers: haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D14037 12 November 2013, 20:42:42 UTC
9bc4a26 Igor Canadi 12 November 2013, 19:53:26 UTC Small changes in Deleting obsolete files Summary: @haobo's suggestions from https://reviews.facebook.net/D13827 Renaming some variables, deprecating purge_log_after_flush, changing for loop into auto for loop. I have not implemented deleting objects outside of mutex yet because it would require a big code change - we would delete object in db_impl, which currently does not know anything about object because it's defined in version_edit.h (FileMetaData). We should do it at some point, though. Test Plan: Ran deletefile_test Reviewers: haobo Reviewed By: haobo CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D14025 12 November 2013, 19:53:26 UTC
dad4255 Igor Canadi 12 November 2013, 18:07:55 UTC Move the comment Summary: Moving the comment per @haobo suggestion. Test Plan: No Reviewers: haobo Reviewed By: haobo CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D14019 12 November 2013, 18:07:55 UTC
4abd219 Igor Canadi 12 November 2013, 05:41:32 UTC Combine two FindObsoleteFiles() Summary: We don't need to call FindObsoleteFiles() twice Test Plan: deletefile_test Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D14007 12 November 2013, 05:41:32 UTC
0ef6285 Kai Liu 12 November 2013, 05:05:16 UTC Don't not suggest flushing data when data block is still empty Summary: This diff fix the bug when the Options::block_size is too small. 12 November 2013, 05:05:16 UTC
94e139f Igor Canadi 12 November 2013, 05:03:41 UTC Fixing failed delete file test Summary: FindObsoleteFiles() has to be called before PurgeObsoleteFiles() because FindObsoleteFiles() sets manifest_file_number, log_number and prev_log_number to valid values. Test Plan: deletefile_test now works Reviewers: dhruba, emayanke, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13995 12 November 2013, 05:03:41 UTC
65e45f0 Igor Canadi 12 November 2013, 05:02:38 UTC Update documentation Summary: Changed leveldb documentation with rocksdb in doc/index.html. Added some of the important options from options.h to doc. Also removed benchmark files and impl.h, since this is all replaced by RocksDB wikis. Test Plan: - Reviewers: dhruba, haobo, kailiu, emayanke, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13977 12 November 2013, 05:02:38 UTC
318a491 Dhruba Borthakur 11 November 2013, 22:42:52 UTC Fix valgrind check by initialising DeletionState. Summary: The valgrind error was introduced by commit 1510339e5257073af82f8c07e6fa1f2a9144e6aa. Initialize DeletionState in constructor. Test Plan: valgrind --leak-check=yes ./deletefile_test Reviewers: igor, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13983 12 November 2013, 00:01:13 UTC
e7c4d82 Kai Liu 10 November 2013, 23:36:30 UTC Fix two bugs that caused 3rd party release failure Summary: * Fix the link to gflags. * Fix a warning for the uninitialized data member. 10 November 2013, 23:36:30 UTC
551ecfa Kai Liu 10 November 2013, 09:17:32 UTC Move down the time consuming tests in table_test Summary: it helps us to better check the tests we really care. Test Plan: make 10 November 2013, 09:17:32 UTC
8a46ecd lovro 07 November 2013, 20:37:58 UTC WriteBatch::Put() overload that gathers key and value from arrays of slices Summary: In our project, when writing to the database, we want to form the value as the concatenation of a small header and a larger payload. It's a shame to have to copy the payload just so we can give RocksDB API a linear view of the value. Since RocksDB makes a copy internally, it's easy to support gather writes. Test Plan: write_batch_test, new test case Reviewers: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13947 09 November 2013, 00:34:32 UTC
1510339 Igor Canadi 08 November 2013, 23:23:46 UTC Speed up FindObsoleteFiles Summary: Here's one solution we discussed on speeding up FindObsoleteFiles. Keep a set of all files in DBImpl and update the set every time we create a file. I probably missed few other spots where we create a file. It might speed things up a bit, but makes code uglier. I don't really like it. Much better approach would be to abstract all file handling to a separate class. Think of it as layer between DBImpl and Env. Having a separate class deal with file namings and deletion would benefit both code cleanliness (especially with huge DBImpl) and speed things up. It will take a huge effort to do this, though. Let's discuss offline today. Test Plan: Ran ./db_stress, verified that files are getting deleted Reviewers: dhruba, haobo, kailiu, emayanke Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D13827 08 November 2013, 23:23:46 UTC
dd218bb Igor Canadi 08 November 2013, 20:23:12 UTC Forgot to change interface everywhere Summary: Changed the name and interface for creating HashSkipListRep. Forgot to change it in db_test. Test Plan: make db_test Reviewers: haobo Reviewed By: haobo Differential Revision: https://reviews.facebook.net/D13965 08 November 2013, 20:23:12 UTC
8b3379d Igor Canadi 08 November 2013, 08:31:09 UTC Implementing DynamicIterator for TransformRepNoLock Summary: What @haobo done with TransformRep, now in TransformRepNoLock. Similar implementation, except that I made DynamicIterator a subclass of Iterator which makes me have less iterator initializations. Test Plan: ./prefix_test. Seeing huge savings vs. TransformRep again! Reviewers: dhruba, haobo, sdong, kailiu Reviewed By: haobo CC: leveldb, haobo Differential Revision: https://reviews.facebook.net/D13953 08 November 2013, 08:31:09 UTC
fd075d6 Kai Liu 08 November 2013, 05:27:21 UTC Provide mechanism to configure when to flush the block Summary: Allow block based table to configure the way flushing the blocks. This feature will allow us to add support for prefix-aligned block. Test Plan: make check Reviewers: dhruba, haobo, sdong, igor Reviewed By: sdong CC: leveldb Differential Revision: https://reviews.facebook.net/D13875 08 November 2013, 05:27:21 UTC
bba6595 Kai Liu 07 November 2013, 23:46:48 UTC Fix the valgrind error Summary: I this bug from valgrind report and found a place that may potentially leak memory. Test Plan: re-ran the valgrind and no error any more Reviewers: emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13959 07 November 2013, 23:46:48 UTC
444cf88 Igor Canadi 07 November 2013, 19:31:56 UTC Flush the log outside of lock Summary: Added a new call LogFlush() that flushes the log contents to the OS buffers. We never call it with lock held. We call it once for every Read/Write and often in compaction/flush process so the frequency should not be a problem. Test Plan: db_test Reviewers: dhruba, haobo, kailiu, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13935 07 November 2013, 19:31:56 UTC
fd20448 Haobo Xu 04 November 2013, 00:32:46 UTC [RocksDB] Generalize prefix-aware iterator to be used for more than one Seek Summary: Added a prefix_seek flag in ReadOptions to indicate that Seek is prefix aware(might not return data with different prefix), and also not bound to a specific prefix. Multiple Seeks and range scans can be invoked on the same iterator. If a specific prefix is specified, this flag will be ignored. Just a quick prototype that works for PrefixHashRep, the new lockless memtable could be easily extended with this support too. Test Plan: test it on Leaf Reviewers: dhruba, kailiu, sdong, igor Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D13929 07 November 2013, 04:45:49 UTC
c2be2cb shamdor 07 November 2013, 02:46:28 UTC WAL log retention policy based on archive size. Summary: Archive cleaning will still happen every WAL_ttl seconds but archived logs will be deleted only if archive size is greater then a WAL_size_limit value. Empty archived logs will be deleted evety WAL_ttl. Test Plan: 1. Unit tests pass. 2. Benchmark. Reviewers: emayanke, dhruba, haobo, sdong, kailiu, igor Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13869 07 November 2013, 02:46:28 UTC
292c2b3 Dhruba Borthakur 06 November 2013, 22:16:22 UTC Fix stress test failure when using mmap-reads. Summary: The mmap-read file->Read() does not use the scratch buffer to read in file-contents. Test Plan: ./db_stress --test_batches_snapshots=1 --ops_per_thread=100000000 --threads=32 --write_buffer_size=4194304 --destroy_db_initially=0 --reopen=0 --readpercent=45 --prefixpercent=5 --writepercent=35 --delpercent=5 --iterpercent=10 --db=/tmp/dhruba --max_key=100000000 --disable_seek_compaction=0 --mmap_read=1 --block_size=16384 --cache_size=1048576 --open_files=500000 --verify_checksum=1 --sync=1 --disable_wal=0 --disable_data_sync=0 --target_file_size_base=2097152 --target_file_size_multiplier=2 --max_write_buffer_number=3 --max_background_compactions=20 --max_bytes_for_level_base=10485760 --filter_deletes=0 Reviewers: haobo, kailiu Reviewed By: kailiu CC: leveldb, kailiu, emayanke Differential Revision: https://reviews.facebook.net/D13923 06 November 2013, 23:40:26 UTC
95a8213 Igor Canadi 06 November 2013, 22:19:46 UTC Log flush every 0 seconds Summary: We have to be able to catch last few log outputs before a crash Test Plan: no Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13917 06 November 2013, 22:19:46 UTC
36409e0 Igor Canadi 06 November 2013, 22:11:52 UTC Fix slow no-io iterator Summary: This fixes #3130525. Dhruba's suggestion and Tnovak's implementation :) The issue was with SkipEmptyDataBlocksForward(), but I also changed SkipEmptyDataBlocksBackward(). Is that OK? Test Plan: Run the logdevice test Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13911 06 November 2013, 22:11:52 UTC
be96f24 Igor Canadi 06 November 2013, 19:55:43 UTC TransformRep - use array instead of unordered_map Summary: I'm sending this diff together with https://reviews.facebook.net/D13881 because it didn't allow me to send only the array one. Here I also replaced unordered_map with just an array of shared_ptrs. This elminated all the locks. I will run the new benchmark and post the results here. Test Plan: db_test Reviewers: dhruba, haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13893 06 November 2013, 19:55:43 UTC
fe4a449 Haobo Xu 23 October 2013, 04:59:44 UTC [RocksDB] prefixhash memtable test Summary: as title, half baked test for prefixhash memtable. Also contains deadlock test option Test Plan: run it Reviewers: igor, dhruba Reviewed By: igor CC: leveldb Differential Revision: https://reviews.facebook.net/D13887 06 November 2013, 07:20:10 UTC
f0b0b28 Kai Liu 06 November 2013, 05:04:22 UTC Remove invalid items in .gitignore 06 November 2013, 05:04:22 UTC
3919058 Dhruba Borthakur 05 November 2013, 22:05:30 UTC Fix failure in rocksdb unit test CompressedCache Summary: The problem was that there was only a single key-value in a block and its compressibility was less than 88%. Rocksdb refuses to compress a block unless its compresses to lesser than 88% of its original size. If a block is not compressed, it does nto get inserted into the compressed block cache. Create the test data so that multiple records fit into the same data block. This increases the compressibility of these data block. Test Plan: ./db_test Reviewers: kailiu, haobo Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13905 06 November 2013, 00:11:34 UTC
7845fd9 Dhruba Borthakur 05 November 2013, 06:02:59 UTC Fixed valgrind error in DBTest.CompressedCache Summary: Fixed valgrind error in DBTest.CompressedCache. This fixes the valgrind error (thanks to Haobo). I am still trying to reproduce the test-failure case deterministically. Test Plan: db_test Reviewers: haobo Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13899 05 November 2013, 19:12:39 UTC
f837f5b Mayank Agarwal 25 October 2013, 02:09:02 UTC Making the transaction log iterator more robust Summary: strict essentially means that we MUST find the startsequence. Thus we should return if starteSequence is not found in the first file in case strict is set. This will take care of ending the iterator in case of permanent gaps due to corruptions in the log files Also created NextImpl function that will have internal variable to distinguish whether Next is being called from StartSequence or by application. Set NotFoudn::gaps status to give an indication of gaps happeneing. Polished the inline documentation at various places Test Plan: * db_repl_stress test * db_test relating to transaction log iterator * fbcode/wormhole/rocksdb/rocks_log_iterator * sigma production machine sigmafio032.prn1 Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13689 05 November 2013, 04:49:03 UTC
b4ad5e8 Dhruba Borthakur 02 September 2013, 06:23:40 UTC Implement a compressed block cache. Summary: Rocksdb can now support a uncompressed block cache, or a compressed block cache or both. Lookups first look for a block in the uncompressed cache, if it is not found only then it is looked up in the compressed cache. If it is found in the compressed cache, then it is uncompressed and inserted into the uncompressed cache. It is possible that the same block resides in the compressed cache as well as the uncompressed cache at the same time. Both caches have their own individual LRU policy. Test Plan: Unit test case attached. Reviewers: kailiu, sdong, haobo, leveldb Reviewed By: haobo CC: xjin, haobo Differential Revision: https://reviews.facebook.net/D12675 01 November 2013, 21:31:35 UTC
1e4375d Piyush Garg 01 November 2013, 20:59:14 UTC Task #3071144 Enhance ldb (db dump tool for leveldb) to report row counters for each row type Summary: Added an option --count_delim=<char> which takes the given character as delimiter ('.' by default) and reports count of each row type found in the db Test Plan: 1. Created test in file (for DBDumperCommand) rocksdb/tools/ldb_test.py which puts various key value pair in db and checks the output using dump --count_delim ,--count_delim="." and --count_delim=",". 2. Created test in file (for InternalDumperCommand) rocksdb/tools/ldb_test.py which puts various key value pair in db and checks the output using dump --count_delim ,--count_delim="." and --count_delim=",". 3. Manually created a database with several keys of several type and verified by running the command ./ldb db=<path> dump --count_delim="<char>" ./ldb db=<path> idump --count_delim="<char>" Reviewers: vamsi, dhruba, emayanke, kailiu Reviewed By: vamsi CC: leveldb Differential Revision: https://reviews.facebook.net/D13815 01 November 2013, 20:59:14 UTC
beeb74b Igor Canadi 01 November 2013, 19:32:27 UTC Move I/O outside of lock Summary: I'm figuring out how Version[Set, Edit, ] classes work and I stumbled on this. It doesn't seem that the comment is accurate anymore. What I read is when the manifest grows too big, create a new file (and not only when we call LogAndApply for the first time). Test Plan: make check (currently running) Reviewers: dhruba, haobo, kailiu, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13839 01 November 2013, 19:32:27 UTC
b572e81 Igor Canadi 31 October 2013, 22:36:40 UTC Flush Log every 5 seconds Summary: This might help with p99 performance, but does not solve the real problem. More discussion on #2947135 Test Plan: make check Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13809 31 October 2013, 22:36:40 UTC
82b7e37 Siying Dong 31 October 2013, 22:26:06 UTC Fix a bug of table_reader_bench Summary: Iterator benchmark case is timed incorrectly. Fix it Test Plan: Run the benchmark Reviewers: haobo, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13845 31 October 2013, 22:26:06 UTC
7caadf2 Siying Dong 31 October 2013, 20:38:54 UTC A very simple benchmark to measure Table implemenation's Get() And Iterator performance Summary: It is a very simple benchmark to measure a Table implementation's Get() and iterator performance if all the data is in memory. Test Plan: N/A Reviewers: dhruba, haobo, kailiu Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13743 31 October 2013, 20:38:54 UTC
8cbe5bb Haobo Xu 27 October 2013, 06:01:26 UTC [RocksDB] Add OnCompactionStart to CompactionFilter class Summary: This is to give application compaction filter a chance to access context information of a specific compaction run. For example, depending on whether a compaction goes through all data files, the application could do things differently. Test Plan: make check Reviewers: dhruba, kailiu, sdong Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13683 31 October 2013, 20:36:43 UTC
b4fab3b Naman Gupta 31 October 2013, 18:51:03 UTC Merge branch 'master' of github.com:facebook/rocksdb into inplace 31 October 2013, 18:51:03 UTC
138a8ee Igor Canadi 31 October 2013, 18:47:22 UTC Fix make release Summary: Don't define if already defined. Test Plan: Running make release in parallel, will not commit if it fails. Reviewers: emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13833 31 October 2013, 18:47:22 UTC
fe25070 Naman Gupta 19 August 2013, 21:12:47 UTC In-place updates for equal keys and similar sized values Summary: Currently for each put, a fresh memory is allocated, and a new entry is added to the memtable with a new sequence number irrespective of whether the key already exists in the memtable. This diff is an attempt to update the value inplace for existing keys. It currently handles a very simple case: 1. Key already exists in the current memtable. Does not inplace update values in immutable memtable or snapshot 2. Latest value type is a 'put' ie kTypeValue 3. New value size is less than existing value, to avoid reallocating memory TODO: For a put of an existing key, deallocate memory take by values, for other value types till a kTypeValue is found, ie. remove kTypeMerge. TODO: Update the transaction log, to allow consistent reload of the memtable. Test Plan: Added a unit test verifying the inplace update. But some other unit tests broken due to invalid sequence number checks. WIll fix them next. Reviewers: xinyaohu, sumeet, haobo, dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D12423 Automatic commit by arc 31 October 2013, 18:27:12 UTC
f03b2df Siying Dong 30 October 2013, 17:52:33 UTC Follow-up Cleaning-up After D13521 Summary: This patch is to address @haobo's comments on D13521: 1. rename Table to be TableReader and make its factory function to be GetTableReader 2. move the compression type selection logic out of TableBuilder but to compaction logic 3. more accurate comments 4. Move stat name constants into BlockBasedTable implementation. 5. remove some uncleaned codes in simple_table_db_test Test Plan: pass test suites. Reviewers: haobo, dhruba, kailiu Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13785 30 October 2013, 17:52:33 UTC
068a819 Siying Dong 29 October 2013, 21:29:03 UTC Fix valgrind_check problem of simple_table_db_test.cc Summary: Two memory issues caused valgrind_check to fail on simple_table_db_test. Fix it Test Plan: make -j32 valgrind_check Reviewers: kailiu, haobo, dhruba Reviewed By: haobo CC: leveldb Differential Revision: https://reviews.facebook.net/D13773 29 October 2013, 21:29:03 UTC
79d8dad Kai Liu 29 October 2013, 04:23:17 UTC Change a typo in method signature 29 October 2013, 04:23:17 UTC
d4eec30 Siying Dong 29 October 2013, 00:54:09 UTC Make "Table" pluggable Summary: This patch makes Table and TableBuilder a abstract class and make all the implementation of the current table into BlockedBasedTable and BlockedBasedTable Builder. Test Plan: Make db_test.cc to work with block based table. Add a new test simple_table_db_test.cc where a different simple table format is implemented. Reviewers: dhruba, haobo, kailiu, emayanke, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13521 29 October 2013, 00:54:09 UTC
8ace6b0 Igor Canadi 29 October 2013, 00:42:33 UTC Run benchmark with no debug Summary: assert(Overlap) significantly slows down the benchmark. Ignore assertions when executing blob_store_bench. Test Plan: Ran the benchmark Reviewers: dhruba, haobo, kailiu Reviewed By: kailiu CC: leveldb Differential Revision: https://reviews.facebook.net/D13737 29 October 2013, 00:42:33 UTC
17991cd Igor Canadi 28 October 2013, 23:31:44 UTC Fix data race in BlobStore benchmark Summary: Apparently C++ doesn't like it if you copy around its atomic<> variables. When running a benchmark for a longer time, benchmark used to stall. Changed WorkerThread in config to WorkerThread*. It works now. Test Plan: Ran benchmark Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13731 28 October 2013, 23:31:44 UTC
994575c Kai Liu 16 October 2013, 18:50:50 UTC Support user-defined table stats collector Summary: 1. Added a new option that support user-defined table stats collection. 2. Added a deleted key stats collector in `utilities` Test Plan: Added a unit test for newly added code. Also ran make check to make sure other tests are not broken. Reviewers: dhruba, haobo Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D13491 28 October 2013, 22:45:14 UTC
7e91b86 Kai Liu 28 October 2013, 21:33:24 UTC Fix a valgrind warning Summary: A latest valgrind test found a recently added unit test has memory leak, which is because DB is not closed at the end of the test. Test Plan: re-run the valgrind locally and make sure there's no memory leakage any more. Reviewers: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D13725 28 October 2013, 21:34:27 UTC
back to top