Skip to main content

Browse the archive

https://github.com/facebook/rocksdb
13 April 2024, 05:39:59 UTC
sort by:
Revision Author Date Message Commit Date
f881d6f Dhruba Borthakur 09 January 2013, 18:26:03 UTC Release 1.5.7.fb Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev: 09 January 2013, 18:26:03 UTC
2e1ad2c Abhishek Kona 08 January 2013, 23:55:06 UTC Remove unnecessary asserts in table/merger.cc Summary: The asserts introduced in https://reviews.facebook.net/D7629 are wrong. The direction of iteration is changed after the function call so they assert's fail. Test Plan: make clean check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7827 09 January 2013, 17:52:45 UTC
d8371ef Kosie van der Merwe 08 January 2013, 20:16:40 UTC Fixing some issues Valgrind found Summary: Found some issues running Valgrind on `db_test` (there are still some outstanding ones) and fixed them. Test Plan: make check ran `valgrind ./db_test` and saw that errors no longer occur Reviewers: dhruba, vamsi, emayanke, sheki Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7803 08 January 2013, 20:16:40 UTC
4d339d7 Kosie van der Merwe 08 January 2013, 19:24:15 UTC Fixed memory leak in ShardedLRUCache Summary: `~ShardedLRUCache()` was empty despite `init()` allocating memory on the heap. Fixed the leak by freeing memory allocated by `init()`. Test Plan: make check Ran valgrind on db_test before and after patch and saw leaked memory went down Reviewers: vamsi, dhruba, emayanke, sheki Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7791 08 January 2013, 19:24:15 UTC
628dc2a Dhruba Borthakur 08 January 2013, 18:03:36 UTC db_bench should use the default value for max_grandparent_overlap_factor. Summary: This was a peformance regression caused by https://reviews.facebook.net/D6729. The default value of max_grandparent_overlap_factor was erroneously set to 0 in db_bench. This was causing compactions to create really really small files because the max_grandparent_overlap_factor was erroneously set to zero in the benchmark. Test Plan: Run --benchmarks=overwrite Reviewers: heyongqiang, emayanke, sheki, MarkCallaghan Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7797 08 January 2013, 19:21:11 UTC
d6e873f Kosie van der Merwe 07 January 2013, 18:11:18 UTC Added clearer error message for failure to create db directory in DBImpl::Recover() Summary: Changed CreateDir() to CreateDirIfMissing() so a directory that already exists now causes and error. Fixed CreateDirIfMissing() and added Env.DirExists() Test Plan: make check to test for regessions Ran the following to test if the error message is not about lock files not existing ./db_bench --db=dir/testdb After creating a file "testdb", ran the following to see if it failed with sane error message: ./db_bench --db=testdb Reviewers: dhruba, emayanke, vamsi, sheki Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D7707 07 January 2013, 18:11:18 UTC
4069f66 Mark Callaghan 03 January 2013, 20:11:50 UTC Add --seed, --read_range to db_bench Summary: Adds the option --seed to db_bench to specify the base for the per-thread RNG. When not set each thread uses the same value across runs of db_bench which defeats IO stress testing. Adds the option --read_range. When set to a value > 1 an iterator is created and each query done for the randomread benchmark will do a range scan for that many rows. When not set or set to 1 the existing behavior (a point lookup) is done. Fixes a bug where a printf format string was missing. Test Plan: run db_bench Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7749 07 January 2013, 17:56:10 UTC
8cd86a7 Kosie van der Merwe 04 January 2013, 01:13:56 UTC Fixing and adding some comments Summary: `MemTableList::Add()` neglected to mention that it took ownership of the reference held by its caller. The comment in `MemTable::Get()` was wrong in describing the format of the key. Test Plan: None Reviewers: dhruba, sheki, emayanke, vamsi Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7755 04 January 2013, 01:13:56 UTC
3f7af03 Abhishek Kona 26 December 2012, 19:51:36 UTC Use a priority queue to merge files. Summary: Use a std::priority_queue in merger.cc instead of doing a o(n) search every time. Currently only the ForwardIteration uses a Priority Queue. Test Plan: make all check Reviewers: dhruba Reviewed By: dhruba CC: emayanke, zshao Differential Revision: https://reviews.facebook.net/D7629 02 January 2013, 21:52:25 UTC
d7d43ae Dhruba Borthakur 31 December 2012, 06:18:52 UTC ExtendOverlappingInputs too slow for large databases. Summary: There was a bug in the ExtendOverlappingInputs method so that the terminating condition for the backward search was incorrect. Test Plan: make clean check Reviewers: sheki, emayanke, MarkCallaghan Reviewed By: MarkCallaghan CC: leveldb Differential Revision: https://reviews.facebook.net/D7725 02 January 2013, 21:19:06 UTC
2fc394a Dhruba Borthakur 27 December 2012, 22:49:24 UTC Do not compile thrift for fbcode build. Summary: 1. The thrift libraries do not need to be built anyore. 2. SSE is dynamically detected via https://github.com/facebook/rocksdb/commit/1aae609b920f8cd4d93ac49798fa96367b9b864c Test Plan: compile and build Reviewers: sheki, emayanke Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7665 28 December 2012, 00:11:11 UTC
5b05417 Dhruba Borthakur 27 December 2012, 20:33:43 UTC Complication error when using gcc 4.7.1. Summary: There is a compilation error while using gcc 4.7.1. util/ldb_cmd.cc:381:3: error: ‘leveldb::ReadOptions::ReadOptions’ names the constructor, not the type util/ldb_cmd.cc:381:37: error: expected ‘;’ before ‘read_options’ util/ldb_cmd.cc:381:49: error: statement cannot resolve address of overloaded function Test Plan: make clean check Reviewers: sheki, emayanke, zshao Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D7659 27 December 2012, 20:38:20 UTC
0f762ac Zheng Shao 26 December 2012, 23:15:54 UTC ldb: Add command "ldb query" to support random read from the database Summary: The queries will come from stdin. One key per line. The output will be in stdout, in the format of "<key> ==> <value>" if found, or "<key>" if not found. "--hex" uses HEX-encoded keys and values in both input and output. Test Plan: ldb query --db=leveldb_db --hex Reviewers: dhruba, emayanke, sheki Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7617 27 December 2012, 04:37:42 UTC
dcece47 Zheng Shao 27 December 2012, 00:30:36 UTC ldb: Fix incorrect arg parsing Summary: We were ignoring additional chars at the end of an arg. This can create confusion, e.g. --disable_wal=0 will act the same as --disable_wal without any warnings. Test Plan: Tried this: [zshao@dev485 ~/git/rocksdb] ./ldb dump --statsAAA Failed: Unknown argument:--statsAAA Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7635 27 December 2012, 04:09:10 UTC
04832db Zheng Shao 26 December 2012, 01:33:15 UTC sst_dump: Fix incorrect cmd args parsing Summary: The current parsing logic ignores any additional chars after the arg. Test Plan: "./sst_dump --verify_checksumAAA" now outputs error. Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7611 26 December 2012, 18:57:04 UTC
396c3aa Abhishek Kona 11 December 2012, 19:57:35 UTC Create a long running test to check GetUpdatesSince. Summary: Create an Executable with * A thread to do Put's * A thread to use GetUpdatesSince. Check if we miss any sequence Numbers. Test Plan: It runs. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7383 21 December 2012, 22:10:06 UTC
127ee2e Zheng Shao 21 December 2012, 07:12:29 UTC manifest_dump: Fix the help message and make it consistent Summary: ldb uses --output_hex so make manifest_dump do the same thing. Test Plan: [zshao@dev485 ~/git/rocksdb] ./manifest_dump --output_hex --file=/data/users/zshao/test_leveldb/MANIFEST-000034 manifest_file_number 42 next_file_number 43 last_sequence 2311567 log_number 36 prev_log_number 0 --- level 0 --- version# 0 --- --- level 1 --- version# 0 --- --- level 2 --- version# 0 --- --- level 3 --- version# 0 --- 5:27788699['0000027F4FBE0000' @ 1 : 1 .. '11CE749602C90000' @ 160642 : 1] 7:27785313['11CE773DA7E00000' @ 160643 : 1 .. '23A4C63EC55D0000' @ 321094 : 1] 9:27784288['23A4D581FCD30000' @ 321095 : 1 .. '3576291D12D00000' @ 481428 : 1] 38:64378271['35762BF0E0CE0000' @ 481429 : 1 .. '5E987E0604700000' @ 852910 : 1] 39:64379046['5E987EB0BDD50000' @ 852911 : 1 .. '87C954330E840000' @ 1224603 : 1] 40:10169201['87C95507E49C0000' @ 1224604 : 1 .. '8E48DC0933B70000' @ 1283317 : 1] 21:27798825['8E48DFB0D7CE0000' @ 1283318 : 1 .. 'A00675F8AD7E0000' @ 1443826 : 1] 23:27793751['A006777536E30000' @ 1443827 : 1 .. 'B1D1787FE8670000' @ 1604553 : 1] 25:27801659['B1D179289BB30000' @ 1604554 : 1 .. 'C396D3A69DCE0000' @ 1765012 : 1] 27:27792661['C396DA1E03B10000' @ 1765013 : 1 .. 'D55C9974FCC10000' @ 1925513 : 1] 29:27789095['D55C9B47CBC00000' @ 1925514 : 1 .. 'E71F67D11CCC0000' @ 2085789 : 1] 31:27793145['E71F7A667E740000' @ 2085790 : 1 .. 'F8D4712EF3D90000' @ 2246454 : 1] 41:11246031['F8D4715916A70000' @ 2246455 : 1 .. 'FFFFFCAE97DF0000' @ 2311567 : 1] --- level 4 --- version# 0 --- --- level 5 --- version# 0 --- --- level 6 --- version# 0 --- Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7575 21 December 2012, 18:45:24 UTC
01d57a3 Zheng Shao 21 December 2012, 07:49:32 UTC sst_dump: Add --output_hex option and output the same format as ldb dump Summary: Now sst_dump has the same option --output_hex as "ldb dump" and also share the same output format. So we can do "sst_dump ... | ldb load ..." for an experiment. Test Plan: [zshao@dev485 ~/git/rocksdb] ./sst_dump --file=/data/users/zshao/test_leveldb/000005.sst --output_hex | head -n 2 0000027F4FBE00000101000000000000 ==> D901000000000000000057596F7520726563656976656420746F6461792773207370656369616C20676966742120436C69636B2041636365707420746F207669657720796F75722047696674206265666F72652069742064697361707065617273210000000000000000 000007F9C2D400000102000000000000 ==> D1010000000000000000544974277320676F6F6420746F206265204B696E67E280A6206F7220517565656E212048657265277320796F75722054756573646179204D79737465727920476966742066726F6D20436173746C6556696C6C65219B7B227A636F6465223A22633566306531633039663764222C227470223A22613275222C227A6B223A22663638663061343262666264303966383435666239626235366365396536643024246234506C345139382E734C4D33522169482D4F31315A64794C4B7A4F4653766D7863746534625F2A3968684E3433786521776C427636504A414355795F70222C227473223A313334343936313737357D0000000000000000 Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7587 21 December 2012, 18:44:19 UTC
70f0f50 Zheng Shao 21 December 2012, 06:56:58 UTC ldb: file_size and write_buffer_size options. Summary: This allows ldb to control the write_buffer_size (which reflects to L0 file size) and file_size (which reflects to L1 file size). Since the target_file_size_ratio is 1 by default, all other levels will also have the same file size as L1. As part of the diff, I also cleaned up some unused code and help messages. Test Plan: ./ldb load --db=/data/users/zshao/test_leveldb --file_size=64000000 --write_buffer_size=32000000 --create_if_missing --input_hex --disable_wal Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7569 21 December 2012, 18:40:38 UTC
1aae609 Abhishek Kona 19 December 2012, 00:52:02 UTC Use CRC32 ss42 instruction. Load it dynamically. Test Plan: make all check Reviewers: dhruba Reviewed By: dhruba CC: leveldb, zshao Differential Revision: https://reviews.facebook.net/D7503 21 December 2012, 18:20:32 UTC
7521a22 Zheng Shao 20 December 2012, 22:25:06 UTC sst_dump: Error message should include the case that compression algorithms are not supported. Summary: It took me almost a day to debug this. :( Although I got to learn the file format as a by-product, this time could be saved if we have better error messages. Test Plan: gmake clean all; sst_dump --hex --file=000005.sst Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7551 20 December 2012, 23:29:51 UTC
551f01f Dhruba Borthakur 20 December 2012, 19:05:41 UTC Unit test to test block format. Summary: This is a standalone unit test to test the format of a block. Test Plan: ./block_test Reviewers: sheki Reviewed By: sheki Differential Revision: https://reviews.facebook.net/D7533 20 December 2012, 22:55:07 UTC
58d1444 Zheng Shao 20 December 2012, 22:30:26 UTC Add libevent include and lib directories Summary: Without this fix, I see failures like this: [zshao@dev1049 /data/users/zshao/rocksdb] . fbcode.gcc471.sh; gmake clean libleveldb.a . . . ./thrift/lib/cpp/async/TEventUtil.h:22:32: fatal error: event.h: No such file or directory Test Plan: . fbcode.gcc471.sh; make clean libleveldb.a Reviewers: dhruba, emayanke, sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7497 20 December 2012, 22:31:54 UTC
f4c2b7c Dhruba Borthakur 18 December 2012, 21:05:39 UTC Enhance ReadOnly mode to process the all committed transactions. Summary: Leveldb has an api OpenForReadOnly() that opens the database in readonly mode. This call had an option to not process the transaction log. This patch removes this option and always processes all transactions that had been committed. It has been done in such a way that it does not create/write to any new files in the process. The invariant of "no-writes" to the leveldb data directory is still true. This enhancement allows multiple threads to open the same database in readonly mode and access all trancations that were committed right upto the OpenForReadOnly call. I changed the public API to match the new semantics because there are no users who are currently using this api. Test Plan: make clean check Reviewers: sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7479 20 December 2012, 00:30:46 UTC
be9b862 Zheng Shao 17 December 2012, 01:06:51 UTC ldb: add "ldb load" command Summary: This command accepts key-value pairs from stdin with the same format of "ldb dump" command. This allows us to try out different compression algorithms/block sizes easily. Test Plan: dump, load, dump, verify the data is the same. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7443 19 December 2012, 09:45:59 UTC
2585979 Abhishek Kona 13 December 2012, 00:29:29 UTC Release 1.5.6 for Java code + Script to automate it. Test Plan: it compiles and deploys. Reviewers: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7341 17 December 2012, 20:11:11 UTC
3d1e92b Dhruba Borthakur 17 December 2012, 05:01:02 UTC Enhancements to rocksdb for better support for replication. Summary: 1. The OpenForReadOnly() call should not lock the db. This is useful so that multiple processes can open the same database concurrently for reading. 2. GetUpdatesSince should not error out if the archive directory does not exist. 3. A new constructor for WriteBatch that can takes a serialized string as a parameter of the constructor. Test Plan: make clean check Reviewers: sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7449 17 December 2012, 19:40:19 UTC
62d4857 Kosie van der Merwe 17 December 2012, 19:26:59 UTC Added meta-database support. Summary: Added kMetaDatabase for meta-databases in db/filename.h along with supporting fuctions. Fixed switch in DBImpl so that it also handles kMetaDatabase. Fixed DestroyDB() that it can handle destroying meta-databases. Test Plan: make check Reviewers: sheki, emayanke, vamsi, dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7245 17 December 2012, 19:26:59 UTC
2f0585f Abhishek Kona 14 December 2012, 21:21:53 UTC Fix a bug. Where DestroyDB deletes a non-existant archive directory. Summary: C tests would fail sometimes as DestroyDB would return a Failure Status message when deleting an archival directory which was not created (WAL_ttl_seconds = 0). Fix: Ignore the Status returned on Deleting Archival Directory. Test Plan: * make check Reviewers: dhruba, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7395 17 December 2012, 18:25:26 UTC
3d9ff0e Zheng Shao 16 December 2012, 23:46:19 UTC ldb: fix dump command to pad HEX output chars with 0. Summary: The old code was omitting the 0 if the char is less than 16. Test Plan: Tried the following program: int main() { unsigned char c = 1; printf("%X\n", c); printf("%02X\n", c); return 0; } The output is: 1 01 Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7437 17 December 2012, 00:55:38 UTC
7dc8bb7 Zheng Shao 16 December 2012, 02:57:31 UTC ldb: support --block_size=<4096|65536|...> and --auto_compaction=<0|1> Summary: This allows us to use ldb to do more experiments like block_size changes. Test Plan: run it by hand. Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7431 16 December 2012, 17:00:14 UTC
c280975 Zheng Shao 16 December 2012, 02:28:36 UTC manifest_dump: Add --hex=1 option Summary: Without this option, manifest_dump does not print binary keys for files in a human-readable way. Test Plan: ./manifest_dump --hex=1 --verbose=0 --file=/data/users/zshao/fdb_comparison/leveldb/fbobj.apprequest-0_0_original/MANIFEST-000002 manifest_file_number 589 next_file_number 590 last_sequence 2311567 log_number 543 prev_log_number 0 --- level 0 --- version# 0 --- 532:1300357['0000455BABE20000' @ 2183973 : 1 .. 'FFFCA5D7ADE20000' @ 2184254 : 1] 536:1308170['000198C75CE30000' @ 2203313 : 1 .. 'FFFCF94A79E30000' @ 2206463 : 1] 542:1321644['0002931AA5E50000' @ 2267055 : 1 .. 'FFF77B31C5E50000' @ 2270754 : 1] 544:1286390['000410A309E60000' @ 2278592 : 1 .. 'FFFE470A73E60000' @ 2289221 : 1] 538:1298778['0006BCF4D8E30000' @ 2217050 : 1 .. 'FFFD77DAF7E30000' @ 2220489 : 1] 540:1282353['00090D5356E40000' @ 2231156 : 1 .. 'FFFFF4625CE40000' @ 2231969 : 1] --- level 1 --- version# 0 --- 510:2112325['000007F9C2D40000' @ 1782099 : 1 .. '146F5B67B8D80000' @ 1905458 : 1] 511:2121742['146F8A3023D60000' @ 1824388 : 1 .. '28BC8FBB9CD40000' @ 1777993 : 1] 512:801631['28BCD396F1DE0000' @ 2080191 : 1 .. '3082DBE9ADDB0000' @ 1989927 : 1] Reviewers: dhruba, sheki, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7425 16 December 2012, 16:58:28 UTC
806d4d9 Abhishek Kona 14 December 2012, 22:01:33 UTC fixing linters. Summary: old version of linters use "lint_engine" instead of "lint.engine" Some bookeeping in gitignore. Reviewers: abhishekk 14 December 2012, 22:05:27 UTC
2ba866e Abhishek Kona 11 December 2012, 20:24:55 UTC GetSequence API in write batch. Summary: WriteBatch is now used by the GetUpdatesSinceAPI. This API is external and will be used by the rocks server. Rocks Server and others will need to know about the Sequence Number in the WriteBatch. This public method will allow for that. Test Plan: make all check. Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7293 13 December 2012, 06:21:10 UTC
d0a3093 Dhruba Borthakur 12 December 2012, 07:54:03 UTC Expose the serialized string that represents a WriteBatch. Summary: Expose the serialized string that represents a WriteBatch. This is helpful to replicate a writebatch operation from one machine to another. Test Plan: make clean check Reviewers: sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7317 13 December 2012, 00:25:52 UTC
f20383f Abhishek Kona 12 December 2012, 06:51:43 UTC Make Java Client compilable. Summary: Debug and ported changes from the Open Source Github repo to our repo. Wrote a script to easy build the java Library. future compiling java lib should just be running this script. Test Plan: it compiles. Reviewers: dhruba, leveldb Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7323 12 December 2012, 22:07:52 UTC
22c2836 Abhishek Kona 10 December 2012, 19:02:07 UTC Fix Bug in Binary Search for files containing a seq no. and delete Archived Log Files during Destroy DB. Summary: * Fixed implementation bug in Binary_Searvch introduced in https://reviews.facebook.net/D7119 * Binary search is also overflow safe. * Delete archive log files and archive dir during DestroyDB Test Plan: make check Reviewers: dhruba CC: kosievdmerwe, emayanke Differential Revision: https://reviews.facebook.net/D7263 12 December 2012, 00:15:02 UTC
24fc379 Dhruba Borthakur 10 December 2012, 23:37:00 UTC An public api to fetch the latest transaction id. Summary: Implement a interface to retrieve the most current transaction id from the database. Test Plan: Added unit test. Reviewers: sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D7269 11 December 2012, 00:04:19 UTC
dcd919a sheki 05 December 2012, 19:32:25 UTC Port various compaction options to Java. Summary: Porting various options, mostly related to Multi-threaded compaction to Java. Test Plan: mvn test. No clear plan on how else test. Reviewers: dhruba Reviewed By: dhruba CC: leveldb, emayanke Differential Revision: https://reviews.facebook.net/D7221 10 December 2012, 18:53:48 UTC
1c6742e Abhishek Kona 08 December 2012, 00:30:22 UTC Refactor GetArchivalDirectoryName to filename.h Summary: filename.h has functions to do similar things. Moving code away from db_impl.cc Test Plan: make check Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7251 10 December 2012, 18:51:07 UTC
38671c4 Dhruba Borthakur 07 December 2012, 00:12:48 UTC Fix a race condition while processing tasks by background threads. Summary: Suppose you submit 100 background tasks one after another. The first enqueu task finds that the queue is empty and wakes up one worker thread. Now suppose that all remaining 99 work items are enqueued, they do not wake up any worker threads because the queue is already non-empty. This causes a situation when there are 99 tasks in the task queue but only one worker thread is processing a task while the remaining worker threads are waiting. The fix is to always wakeup one worker thread while enqueuing a task. I also added a check to count the number of elements in the queue to help in debugging. Test Plan: make clean check. Reviewers: chip Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D7203 10 December 2012, 01:15:27 UTC
768edfa Zheng Shao 05 December 2012, 23:37:03 UTC ldb: Add compression and bloom filter options. Summary: Added the following two options: [--bloom_bits=<int,e.g.:14>] [--compression_type=<no|snappy|zlib|bzip2>] These options will be used when ldb opens the leveldb database. Test Plan: Tried by hand for both success and failure cases. We do need a test framework. Reviewers: dhruba, emayanke, sheki Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7197 07 December 2012, 22:26:37 UTC
8055008 Abhishek Kona 30 November 2012, 01:28:37 UTC GetUpdatesSince API to enable replication. Summary: How it works: * GetUpdatesSince takes a SequenceNumber. * A LogFile with the first SequenceNumber nearest and lesser than the requested Sequence Number is found. * Seek in the logFile till the requested SeqNumber is found. * Return an iterator which contains logic to return record's one by one. Test Plan: * Test case included to check the good code path. * Will update with more test-cases. * Feedback required on test-cases. Reviewers: dhruba, emayanke Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7119 07 December 2012, 19:42:13 UTC
f69e9f3 Kosie van der Merwe 07 December 2012, 18:27:37 UTC Fixed off by 1 in tests. Summary: Added 1 to indices where I shouldn't have so overrun array. Test Plan: make check Reviewers: sheki, emayanke, vamsi, dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D7227 07 December 2012, 18:48:46 UTC
0eb0c9b Kosie van der Merwe 07 December 2012, 18:42:19 UTC Added methods to write small ints to bit streams. Summary: Added BitStreamPutInt() and BitStreamGetInt() which take a stream of chars and can write integers of arbitrary bit sizes to that stream at arbitrary positions. There are also convenience versions of these functions that take std::strings and leveldb::Slices. Test Plan: make check Reviewers: sheki, vamsi, dhruba, emayanke Reviewed By: vamsi CC: leveldb Differential Revision: https://reviews.facebook.net/D7071 07 December 2012, 18:42:19 UTC
c847a31 Dhruba Borthakur 04 December 2012, 17:47:05 UTC Print compaction score for every compaction run. Summary: A compaction is picked based on its score. It is useful to print the compaction score in the LOG because it aids in debugging. If one looks at the logs, one can find out why a compaction was preferred over another. Test Plan: make clean check Differential Revision: https://reviews.facebook.net/D7137 04 December 2012, 18:03:47 UTC
6eb5ed9 Dhruba Borthakur 30 November 2012, 06:39:08 UTC rocksdb README file. Summary: rocksdb README file. Test Plan: Reviewers: CC: Task ID: # Blame Rev: 30 November 2012, 06:39:08 UTC
d4627e6 sheki 26 November 2012, 21:56:45 UTC Move WAL files to archive directory, instead of deleting. Summary: Create a directory "archive" in the DB directory. During DeleteObsolteFiles move the WAL files (*.log) to the Archive directory, instead of deleting. Test Plan: Created a DB using DB_Bench. Reopened it. Checked if files move. Reviewers: dhruba Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D6975 29 November 2012, 01:28:08 UTC
d29f181 Abhishek Kona 29 November 2012, 00:42:36 UTC Fix all the lint errors. Summary: Scripted and removed all trailing spaces and converted all tabs to spaces. Also fixed other lint errors. All lint errors from this point of time should be taken seriously. Test Plan: make all check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D7059 29 November 2012, 01:18:41 UTC
9b83853 Dhruba Borthakur 29 November 2012, 00:09:41 UTC Release 1.5.6.fb Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev: 29 November 2012, 00:09:41 UTC
9a35784 Dhruba Borthakur 27 November 2012, 05:16:21 UTC Delete non-visible keys during a compaction even in the presense of snapshots. Summary: LevelDB should delete almost-new keys when a long-open snapshot exists. The previous behavior is to keep all versions that were created after the oldest open snapshot. This can lead to database size bloat for high-update workloads when there are long-open snapshots and long-open snapshot will be used for logical backup. By "almost new" I mean that the key was updated more than once after the oldest snapshot. If there were two snapshots with seq numbers s1 and s2 (s1 < s2), and if we find two instances of the same key k1 that lie entirely within s1 and s2 (i.e. s1 < k1 < s2), then the earlier version of k1 can be safely deleted because that version is not visible in any snapshot. Test Plan: unit test attached make clean check Differential Revision: https://reviews.facebook.net/D6999 28 November 2012, 23:47:40 UTC
34487af Abhishek Kona 28 November 2012, 06:09:08 UTC Moved FBCode Linter's to LevelDB. Summary: Added FBCODE like linting support to our codebase. Test Plan: arc lint lint's the code. Reviewers: dhruba Reviewed By: dhruba CC: emayanke, leveldb Differential Revision: https://reviews.facebook.net/D7041 28 November 2012, 17:49:01 UTC
3366eda Dhruba Borthakur 28 November 2012, 06:10:20 UTC Print out status at the end of a compaction run. Summary: Print out status at the end of a compaction run. This helps in debugging. Test Plan: make clean check Reviewers: sheki Reviewed By: sheki Differential Revision: https://reviews.facebook.net/D7035 28 November 2012, 06:17:38 UTC
43f5a07 sheki 27 November 2012, 00:58:58 UTC Remove unused varibles. Cause compiler warnings. Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: emayanke Differential Revision: https://reviews.facebook.net/D6993 27 November 2012, 04:55:24 UTC
2a39699 Dhruba Borthakur 26 November 2012, 09:49:50 UTC Assertion failure while running with unit tests with OPT=-g Summary: When we expand the range of keys for a level 0 compaction, we need to invoke ParentFilesInCompaction() only once for the entire range of keys that is being compacted. We were invoking it for each file that was being compacted, but this triggers an assertion because each file's range were contiguous but non-overlapping. I renamed ParentFilesInCompaction to ParentRangeInCompaction to adequately represent that it is the range-of-keys and not individual files that we compact in a single compaction run. Here is the assertion that is fixed by this patch. db_test: db/version_set.cc:585: void leveldb::Version::ExtendOverlappingInputs(int, const leveldb::Slice&, const leveldb::Slice&, std::vector<leveldb::FileMetaData*, std::allocator<leveldb::FileMetaData*> >*, int): Assertion `user_cmp->Compare(flimit, user_begin) >= 0' failed. Test Plan: make clean check OPT=-g Reviewers: sheki Reviewed By: sheki CC: MarkCallaghan, emayanke, leveldb Differential Revision: https://reviews.facebook.net/D6963 26 November 2012, 22:00:39 UTC
7c6f527 Dhruba Borthakur 26 November 2012, 20:01:55 UTC Merge branch 'performance' 26 November 2012, 20:01:55 UTC
e0cd6bf Dhruba Borthakur 21 November 2012, 23:09:46 UTC The c_test was sometimes failing with an assertion. Summary: On fast filesystems (e.g. /dev/shm and ext4), the flushing of memstore to disk was fast and quick, and the background compaction thread was not getting scheduled fast enough to delete obsolete files before the db was closed. This caused the repair method to pick up those files that were not part of the db and the unit test was failing. The fix is to enhance the unti test to run a compaction before closing the database so that all files that are not part of the database are truly deleted from the filesystem. Test Plan: make c_test; ./c_test Reviewers: chip, emayanke, sheki Reviewed By: chip CC: leveldb Differential Revision: https://reviews.facebook.net/D6915 26 November 2012, 19:59:51 UTC
6caf3b8 Chip Turner 21 November 2012, 21:26:32 UTC Fix broken test; some ldb commands can run without a db_ Summary: It would appear our unit tests make use of code from ldb_cmd, and don't always require a valid database handle. D6855 was not aware db_ could sometimes be NULL for such commands, and so it broke reduce_levels_test. This moves the check elsewhere to (at least) fix the 'ldb dump' case of segfaulting when it couldn't open a database. Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D6903 26 November 2012, 19:11:30 UTC
879e45e Chip Turner 21 November 2012, 00:14:04 UTC Fix ldb segfault and use static libsnappy for all builds Summary: Link statically against snappy, using the gvfs one for facebook environments, and the bundled one otherwise. In addition, fix a few minor segfaults in ldb when it couldn't open the database, and update .gitignore to include a few other build artifacts. Test Plan: make check Reviewers: dhruba Reviewed By: dhruba CC: leveldb Differential Revision: https://reviews.facebook.net/D6855 21 November 2012, 19:07:19 UTC
7632fdb Dhruba Borthakur 21 November 2012, 07:07:41 UTC Support taking a configurable number of files from the same level to compact in a single compaction run. Summary: The compaction process takes some files from LevelK and merges it into LevelK+1. The number of files it picks from LevelK was capped such a way that the total amount of data picked does not exceed the maxfilesize of that level. This essentially meant that only one file from LevelK is picked for a single compaction. For bulkloads, we would like to take many many file from LevelK and compact them using a single compaction run. This patch introduces a option called the 'source_compaction_factor' (similar to expanded_compaction_factor). It is a multiplier that is multiplied by the maxfilesize of that level to arrive at the limit that is used to throttle the number of source files from LevelK. For bulk loads, set source_compaction_factor to a very high number so that multiple files from the same level are picked for compaction in a single compaction. The default value of source_compaction_factor is 1, so that we can keep backward compatibilty with existing compaction semantics. Test Plan: make clean check Reviewers: emayanke, sheki Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D6867 21 November 2012, 16:37:03 UTC
fbb73a4 Dhruba Borthakur 20 November 2012, 23:45:41 UTC Support to disable background compactions on a database. Summary: This option is needed for fast bulk uploads. The goal is to load all the data into files in L0 without any interference from background compactions. Test Plan: make clean check Reviewers: sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D6849 21 November 2012, 05:12:06 UTC
3754f2f Dhruba Borthakur 20 November 2012, 17:08:11 UTC A major bug that was not considering the compaction score of the n-1 level. Summary: The method Finalize() recomputes the compaction score of each level and then sorts these score from largest to smallest. The idea is that the level with the largest compaction score will be a better candidate for compaction. There are usually very few levels, and a bubble sort code was used to sort these compaction scores. There existed a bug in the sorting code that skipped looking at the score for the n-1 level. This meant that even if the compaction score of the n-1 level is large, it will not be picked for compaction. This patch fixes the bug and also introduces "asserts" in the code to detect any possible inconsistencies caused by future bugs. This bug existed in the very first code change that introduced multi-threaded compaction to the leveldb code. That version of code was committed on Oct 19th via https://github.com/facebook/leveldb/commit/1ca0584345af85d2dccc434f451218119626d36e Test Plan: make clean check OPT=-g Reviewers: emayanke, sheki, MarkCallaghan Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D6837 20 November 2012, 23:44:21 UTC
dde7089 Dhruba Borthakur 19 November 2012, 22:51:22 UTC Fix asserts Summary: make check OPT=-g fails with the following assert. ==== Test DBTest.ApproximateSizes db_test: db/version_set.cc:765: void leveldb::VersionSet::Builder::CheckConsistencyForDeletes(leveldb::VersionEdit*, int, int): Assertion `found' failed. The assertion was that file #7 that was being deleted did not preexists, but actualy it did pre-exist as shown in the manifest dump shows below. The bug was that we did not check for file existance at the same level. *************************Edit[0] = VersionEdit { Comparator: leveldb.BytewiseComparator } *************************Edit[1] = VersionEdit { LogNumber: 8 PrevLogNumber: 0 NextFile: 9 LastSeq: 80 AddFile: 0 7 8005319 'key000000' @ 1 : 1 .. 'key000079' @ 80 : 1 } *************************Edit[2] = VersionEdit { LogNumber: 8 PrevLogNumber: 0 NextFile: 13 LastSeq: 80 CompactPointer: 0 'key000079' @ 80 : 1 DeleteFile: 0 7 AddFile: 1 9 2101425 'key000000' @ 1 : 1 .. 'key000020' @ 21 : 1 AddFile: 1 10 2101425 'key000021' @ 22 : 1 .. 'key000041' @ 42 : 1 AddFile: 1 11 2101425 'key000042' @ 43 : 1 .. 'key000062' @ 63 : 1 AddFile: 1 12 1701165 'key000063' @ 64 : 1 .. 'key000079' @ 80 : 1 } Test Plan: Reviewers: CC: Task ID: # Blame Rev: 19 November 2012, 22:51:22 UTC
a4b79b6 Dhruba Borthakur 19 November 2012, 21:20:25 UTC Merge branch 'master' into performance 19 November 2012, 21:20:25 UTC
74054fa Dhruba Borthakur 19 November 2012, 21:16:46 UTC Fix compilation error while compiling unit tests with OPT=-g Summary: Fix compilation error while compiling with OPT=-g Test Plan: make clean check OPT=-g Reviewers: CC: Task ID: # Blame Rev: 19 November 2012, 21:16:46 UTC
48dafb2 Dhruba Borthakur 19 November 2012, 20:16:45 UTC Fix compilation error introduced by previous commit 7889e094554dc5bba678a0bfa7fb5eca422c34de Summary: Fix compilation error introduced by previous commit 7889e094554dc5bba678a0bfa7fb5eca422c34de Test Plan: make clean check 19 November 2012, 20:16:45 UTC
7889e09 Dhruba Borthakur 19 November 2012, 19:54:13 UTC Enhance manifest_dump to print each individual edit. Summary: The manifest file contains a series of edits. If the verbose option is switched on, then print each individual edit in the manifest file. This helps in debugging. Test Plan: make clean manifest_dump Reviewers: emayanke, sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D6807 19 November 2012, 20:04:35 UTC
661dc15 Abhishek Kona 19 November 2012, 18:46:36 UTC Fix LDB dumpwal to print the messages as in the file. Summary: StringStream.clear() does not clear the stream. It sets some flags. Who knew? Fixing that is not printing the stuff again and again. Test Plan: ran it on a local db Reviewers: dhruba, emayanke Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D6795 19 November 2012, 20:04:35 UTC
65b035a amayank 16 November 2012, 20:55:21 UTC Fix a coding error in db_test.cc Summary: The new function MinLevelToCompress in db_test.cc was incomplete. It needs to tell the calling function-TEST whether the test has to be skipped or not Test Plan: make all;./db_test Reviewers: dhruba, heyongqiang Reviewed By: dhruba CC: sheki Differential Revision: https://reviews.facebook.net/D6771 19 November 2012, 20:04:35 UTC
30742e1 Abhishek Kona 13 November 2012, 00:45:45 UTC LDB can read WAL. Summary: Add option to read WAL and print a summary for each record. facebook task => #1885013 E.G. Output : ./ldb dump_wal --walfile=/tmp/leveldbtest-5907/dbbench/026122.log --header Sequence,Count,ByteSize 49981,1,100033 49981,1,100033 49982,1,100033 49981,1,100033 49982,1,100033 49983,1,100033 49981,1,100033 49982,1,100033 49983,1,100033 49984,1,100033 49981,1,100033 49982,1,100033 Test Plan: Works run ./ldb read_wal --wal-file=/tmp/leveldbtest-5907/dbbench/000078.log --header Reviewers: dhruba, heyongqiang Reviewed By: dhruba CC: emayanke, leveldb, zshao Differential Revision: https://reviews.facebook.net/D6675 19 November 2012, 20:04:34 UTC
4b622ab Dhruba Borthakur 19 November 2012, 19:54:13 UTC Enhance manifest_dump to print each individual edit. Summary: The manifest file contains a series of edits. If the verbose option is switched on, then print each individual edit in the manifest file. This helps in debugging. Test Plan: make clean manifest_dump Reviewers: emayanke, sheki Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D6807 19 November 2012, 20:02:27 UTC
b648401 Abhishek Kona 19 November 2012, 18:46:36 UTC Fix LDB dumpwal to print the messages as in the file. Summary: StringStream.clear() does not clear the stream. It sets some flags. Who knew? Fixing that is not printing the stuff again and again. Test Plan: ran it on a local db Reviewers: dhruba, emayanke Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D6795 19 November 2012, 19:14:07 UTC
62e7583 Dhruba Borthakur 16 November 2012, 23:28:14 UTC enhance dbstress to simulate hard crash Summary: dbstress has an option to reopen the database. Make it such that the previous handle is not closed before we reopen, this simulates a situation similar to a process crash. Added new api to DMImpl to remove the lock file. Test Plan: run db_stress Reviewers: emayanke Reviewed By: emayanke CC: leveldb Differential Revision: https://reviews.facebook.net/D6777 19 November 2012, 07:16:17 UTC
de278a6 amayank 16 November 2012, 20:55:21 UTC Fix a coding error in db_test.cc Summary: The new function MinLevelToCompress in db_test.cc was incomplete. It needs to tell the calling function-TEST whether the test has to be skipped or not Test Plan: make all;./db_test Reviewers: dhruba, heyongqiang Reviewed By: dhruba CC: sheki Differential Revision: https://reviews.facebook.net/D6771 16 November 2012, 22:56:50 UTC
f5cdf93 Abhishek Kona 13 November 2012, 00:45:45 UTC LDB can read WAL. Summary: Add option to read WAL and print a summary for each record. facebook task => #1885013 E.G. Output : ./ldb dump_wal --walfile=/tmp/leveldbtest-5907/dbbench/026122.log --header Sequence,Count,ByteSize 49981,1,100033 49981,1,100033 49982,1,100033 49981,1,100033 49982,1,100033 49983,1,100033 49981,1,100033 49982,1,100033 49983,1,100033 49984,1,100033 49981,1,100033 49982,1,100033 Test Plan: Works run ./ldb read_wal --wal-file=/tmp/leveldbtest-5907/dbbench/000078.log --header Reviewers: dhruba, heyongqiang Reviewed By: dhruba CC: emayanke, leveldb, zshao Differential Revision: https://reviews.facebook.net/D6675 16 November 2012, 17:09:00 UTC
c3392c9 Dhruba Borthakur 15 November 2012, 06:00:11 UTC The db_stress test should also test multi-threaded compaction. Summary: Create more than one background compaction thread if specified. This code peice is similar to what exists in db_bench. Test Plan: make check Differential Revision: https://reviews.facebook.net/D6753 15 November 2012, 06:01:39 UTC
6c5a4d6 Dhruba Borthakur 15 November 2012, 05:39:52 UTC Merge branch 'master' into performance Conflicts: db/db_impl.h 15 November 2012, 05:39:52 UTC
e988c11 Dhruba Borthakur 14 November 2012, 17:10:46 UTC Enhance db_bench to be able to specify a grandparent_overlap_factor. Summary: The value specified in max_grandparent_overlap_factor is used to limit the file size in a compaction run. This patch makes it configurable when using db_bench. Test Plan: make clean db_bench Reviewers: MarkCallaghan, heyongqiang Reviewed By: heyongqiang CC: leveldb Differential Revision: https://reviews.facebook.net/D6729 15 November 2012, 00:20:13 UTC
0f590af Dhruba Borthakur 14 November 2012, 00:28:11 UTC Push release 1.5.5.fb. Summary: Test Plan: Reviewers: CC: Task ID: # Blame Rev: 14 November 2012, 00:28:11 UTC
33cf6f3 Dhruba Borthakur 14 November 2012, 00:16:21 UTC Make sse compilation optional. Summary: The fbcode compilation was always switching on msse by default. This patch keeps the same behaviour but allows the compilation process to switch off msse if needed. If one does not want to use sse, then do the following: export USE_SSE=0 make clean all Test Plan: make clean all Reviewers: heyongqiang Reviewed By: heyongqiang CC: leveldb Differential Revision: https://reviews.facebook.net/D6717 14 November 2012, 00:25:57 UTC
5d16e50 Dhruba Borthakur 13 November 2012, 23:48:00 UTC Improved CompactionFilter api: pass in a opaque argument to CompactionFilter invocation. Summary: There are applications that operate on multiple leveldb instances. These applications will like to pass in an opaque type for each leveldb instance and this type should be passed back to the application with every invocation of the CompactionFilter api. Test Plan: Enehanced unit test for opaque parameter to CompactionFilter. Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan, sheki, emayanke Differential Revision: https://reviews.facebook.net/D6711 14 November 2012, 00:22:26 UTC
43d9a82 Dhruba Borthakur 13 November 2012, 18:30:00 UTC Fix asserts so that "make check OPT=-g" works on performance branch Summary: Compilation used to fail with the error: db/version_set.cc:1773: error: ‘number_of_files_to_sort_’ is not a member of ‘leveldb::VersionSet’ I created a new method called CheckConsistencyForDeletes() so that all the high cost checking is done only when OPT=-g is specified. I also fixed a bug in PickCompactionBySize that was triggered when OPT=-g was switched on. The base_index in the compaction record was not set correctly. Test Plan: make check OPT=-g Differential Revision: https://reviews.facebook.net/D6687 13 November 2012, 18:40:52 UTC
a785e02 Dhruba Borthakur 12 November 2012, 21:50:55 UTC The db_bench utility was broken in 1.5.4.fb because of a signed-unsigned comparision. Summary: The db_bench utility was broken in 1.5.4.fb because of a signed-unsigned comparision. The static variable FLAGS_min_level_to_compress was recently changed from int to 'unsigned in' but it is initilized to a nagative value -1. The segfault is of this type: Program received signal SIGSEGV, Segmentation fault. Open (this=0x7fffffffdee0) at db/db_bench.cc:939 939 db/db_bench.cc: No such file or directory. (gdb) where Test Plan: run db_bench with no options. Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan, emayanke, sheki Differential Revision: https://reviews.facebook.net/D6663 12 November 2012, 21:59:35 UTC
e626261 amayank 09 November 2012, 21:04:12 UTC Introducing "database reopens" into the stress test. Database will reopen after a specified number of iterations (configurable) of each thread when they will wait for the databse to reopen. Summary: FLAGS_reopen (configurable) specifies the number of times the databse is to be reopened. FLAGS_ops_per_thread is divided into points based on that reopen field. At these points all threads come together to wait for the databse to reopen. Each thread "votes" for the database to reopen and when all have voted, the database reopens. Test Plan: make all;./db_stress Reviewers: dhruba, MarkCallaghan, sheki, asad, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D6627 12 November 2012, 20:26:32 UTC
c64796f heyongqiang 12 November 2012, 06:33:22 UTC Fix test failure of reduce_num_levels Summary: I changed the reduce_num_levels logic to avoid "compactRange()" call if the current number of levels in use (levels that contain files) is smaller than the new num of levels. And that change breaks the assert in reduce_levels_test Test Plan: run reduce_levels_test Reviewers: dhruba, MarkCallaghan Reviewed By: dhruba CC: emayanke, sheki Differential Revision: https://reviews.facebook.net/D6651 12 November 2012, 20:05:38 UTC
9c6c232 Dhruba Borthakur 11 November 2012, 08:20:40 UTC Compilation error while compiling with OPT=-g Summary: make clean check OPT=-g fails leveldb::DBStatistics::getTickerCount(leveldb::Tickers)’: ./db/db_statistics.h:34: error: ‘MAX_NO_TICKERS’ was not declared in this scope util/ldb_cmd.cc:255: warning: left shift count >= width of type Test Plan: make clean check OPT=-g Reviewers: CC: Task ID: # Blame Rev: 11 November 2012, 08:20:40 UTC
0f8e472 Abhishek Kona 09 November 2012, 02:18:34 UTC Metrics: record compaction drop's and bloom filter effectiveness Summary: Record BloomFliter hits and drop off reasons during compaction. Test Plan: Unit tests work. Reviewers: dhruba, heyongqiang Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D6591 09 November 2012, 19:38:45 UTC
20d18a8 heyongqiang 09 November 2012, 02:45:19 UTC disable size compaction in ldb reduce_levels and added compression and file size parameter to it Summary: disable size compaction in ldb reduce_levels, this will avoid compactions rather than the manual comapction, added --compression=none|snappy|zlib|bzip2 and --file_size= per-file size to ldb reduce_levels command Test Plan: run ldb Reviewers: dhruba, MarkCallaghan Reviewed By: dhruba CC: sheki, emayanke Differential Revision: https://reviews.facebook.net/D6597 09 November 2012, 18:14:47 UTC
e00c709 Dhruba Borthakur 09 November 2012, 17:21:11 UTC Preparing for new release 1.5.4.fb Summary: Preparing for new release 1.5.4.fb Test Plan: Reviewers: CC: Task ID: # Blame Rev: 09 November 2012, 17:21:11 UTC
9e97bfd amayank 07 November 2012, 23:35:08 UTC Introducing deletes for stress test Summary: Stress test modified to do deletes and later verify them Test Plan: running the test: db_stress Reviewers: dhruba, heyongqiang, asad, sheki, MarkCallaghan Reviewed By: dhruba Differential Revision: https://reviews.facebook.net/D6567 09 November 2012, 00:55:18 UTC
391885c Abhishek Kona 03 November 2012, 04:02:40 UTC stat's collection in leveldb Summary: Prototype stat's collection. Diff is a good estimate of what the final code will look like. A few assumptions : * Used a global static instance of the statistics object. Plan to pass it to each internal function. Static allows metrics only at app level. * In the Ticker's do not do any locking. Depend on the mutex at each function of LevelDB. If we ever remove the mutex, we should change here too. The other option is use atomic objects anyways as there won't be any contention as they will be always acquired only by one thread. * The counters are dumb, increment through lifecycle. Plan to use ods etc to get last5min stat etc. Test Plan: made changes in db_bench Ran ./db_bench --statistics=1 --num=10000 --cache_size=5000 This will print the cache hit/miss stats. Reviewers: dhruba, heyongqiang Differential Revision: https://reviews.facebook.net/D6441 08 November 2012, 21:55:49 UTC
95dda37 Dhruba Borthakur 01 November 2012, 05:01:57 UTC Move filesize-based-sorting to outside the Mutex Summary: When a new version is created, we sort all the files at every level based on their size. This is necessary because we want to compact the largest file first. The sorting takes quite a bit of CPU. Moved the sorting code to be outside the mutex. Also, the earlier code was sorting files at all levels but we do not need to sort the highest-number level because those files are never the cause of any compaction. To reduce sorting costs, we sort only the first few files in each level because it is likely that those are the only files in that level that will be picked for compaction. At steady state, I have seen that this patch increase throughout from 1500 writes/sec to 1700 writes/sec at the end of a 72 hour run. The cpu saving by not sorting the last level was not distinctive in this test run because there were only 100K files in the highest numbered level. I expect the cpu saving to be significant when the number of files is much higher. This is mostly an early preview and not ready for rigorous review. With this patch, the writs/sec is now bottlenecked not by the sorting code but by GetOverlappingInputs. I am working on a patch to optimize GetOverlappingInputs. Test Plan: make check Reviewers: MarkCallaghan, heyongqiang Reviewed By: heyongqiang Differential Revision: https://reviews.facebook.net/D6411 07 November 2012, 23:39:44 UTC
18cb600 Dhruba Borthakur 07 November 2012, 23:24:47 UTC Fixed compilation error in previous merge. Summary: Fixed compilation error in previous merge. Test Plan: Reviewers: CC: Task ID: # Blame Rev: 07 November 2012, 23:24:47 UTC
8143062 Dhruba Borthakur 07 November 2012, 23:11:37 UTC Merge branch 'master' into performance Conflicts: db/db_impl.cc db/version_set.cc util/options.cc 07 November 2012, 23:11:37 UTC
3fcf533 heyongqiang 06 November 2012, 03:18:49 UTC Add a readonly db Summary: as subject Test Plan: run db_bench readrandom Reviewers: dhruba Reviewed By: dhruba CC: MarkCallaghan, emayanke, sheki Differential Revision: https://reviews.facebook.net/D6495 07 November 2012, 22:19:48 UTC
9b87a2b Dhruba Borthakur 06 November 2012, 17:06:16 UTC Avoid doing a exhaustive search when looking for overlapping files. Summary: The Version::GetOverlappingInputs() is called multiple times in the compaction code path. Eack invocation does a binary search for overlapping files in the specified key range. This patch remembers the offset of an overlapped file when GetOverlappingInputs() is called the first time within a compaction run. Suceeding calls to GetOverlappingInputs() uses the remembered index to avoid the binary search. I measured that 1000 iterations of GetOverlappingInputs takes around 4500 microseconds without this patch. If I use this patch with the hint on every invocation, then 1000 iterations take about 3900 microsecond. Test Plan: make check OPT=-g Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan, emayanke, sheki Differential Revision: https://reviews.facebook.net/D6513 07 November 2012, 19:47:17 UTC
4e413df Abhishek Kona 06 November 2012, 19:21:57 UTC Flush Data at object destruction if disableWal is used. Summary: Added a conditional flush in ~DBImpl to flush. There is still a chance of writes not being persisted if there is a crash (not a clean shutdown) before the DBImpl instance is destroyed. Test Plan: modified db_test to meet the new expectations. Reviewers: dhruba, heyongqiang Differential Revision: https://reviews.facebook.net/D6519 06 November 2012, 23:04:42 UTC
aa42c66 Dhruba Borthakur 06 November 2012, 20:02:18 UTC Fix all warnings generated by -Wall option to the compiler. Summary: The default compilation process now uses "-Wall" to compile. Fix all compilation error generated by gcc. Test Plan: make all check Reviewers: heyongqiang, emayanke, sheki Reviewed By: heyongqiang CC: MarkCallaghan Differential Revision: https://reviews.facebook.net/D6525 06 November 2012, 22:07:31 UTC
5f91868 Dhruba Borthakur 06 November 2012, 00:51:55 UTC Merge branch 'master' into performance Conflicts: db/version_set.cc util/options.cc 06 November 2012, 00:51:55 UTC
cb7a002 Dhruba Borthakur 05 November 2012, 07:47:06 UTC The method GetOverlappingInputs should use binary search. Summary: The method Version::GetOverlappingInputs used a sequential search to map a kay-range to a set of files. But the files are arranged in ascending order of key, so a biary search is more effective. This patch implements Version::GetOverlappingInputsBinarySearch that finds one file that corresponds to the specified key range and then iterates backwards and forwards to find all overlapping files. This patch is critical for making compactions efficient, especially when there are thousands of files in a single level. I measured that 1000 iterations of TEST_MaxNextLevelOverlappingBytes takes 16000 microseconds without this patch. With this patch, the same method takes about 4600 microseconds. Test Plan: Almost all unit tests in db_test uses this method to lookup keys. Reviewers: heyongqiang Reviewed By: heyongqiang CC: MarkCallaghan, emayanke, sheki Differential Revision: https://reviews.facebook.net/D6465 06 November 2012, 00:08:01 UTC
back to top