5de4fff | Andrew Kryczka | 12 August 2017, 23:15:29 UTC | release 5.4.10 | 12 August 2017, 23:15:29 UTC |
9f33284 | Andrew Kryczka | 12 August 2017, 01:01:28 UTC | fix deletion dropping in intra-L0 Summary: `KeyNotExistsBeyondOutputLevel` didn't consider L0 files' key-ranges. So if a key only was covered by older L0 files' key-ranges, we would incorrectly drop deletions of that key. This PR just skips the deletion-dropping optimization when output level is L0. Closes https://github.com/facebook/rocksdb/pull/2726 Differential Revision: D5617286 Pulled By: ajkr fbshipit-source-id: 4bff1396b06d49a828ba4542f249191052915bce | 12 August 2017, 23:15:10 UTC |
fb6f3b6 | sdong | 21 July 2017, 03:52:37 UTC | Bump up version to 5.4.9 | 21 July 2017, 03:52:37 UTC |
b424288 | sdong | 21 July 2017, 00:55:35 UTC | Update headers for more files. | 21 July 2017, 00:55:35 UTC |
96687f0 | sdong | 18 July 2017, 19:35:02 UTC | Update java/rocksjni.pom | 21 July 2017, 00:21:35 UTC |
3327adc | sdong | 18 July 2017, 19:21:24 UTC | Some more files. | 21 July 2017, 00:21:27 UTC |
f63f26f | sdong | 18 July 2017, 18:38:39 UTC | Remove some left-over BSD headers | 21 July 2017, 00:21:15 UTC |
46e923f | Siying Dong | 17 July 2017, 01:35:39 UTC | Add back the LevelDB license file Summary: Closes https://github.com/facebook/rocksdb/pull/2591 Differential Revision: D5432696 Pulled By: siying fbshipit-source-id: a613230ab916de0b279a65ef429ede65460a8db2 | 21 July 2017, 00:21:02 UTC |
b623dca | Siying Dong | 16 July 2017, 22:50:14 UTC | Remove the licensing description in CONTRIBUTING.md Summary: Closes https://github.com/facebook/rocksdb/pull/2590 Differential Revision: D5432539 Pulled By: siying fbshipit-source-id: 49902453bba3c95c1fb8354441b2198649e52bf4 | 21 July 2017, 00:20:55 UTC |
6189230 | Siying Dong | 15 July 2017, 23:03:42 UTC | Change RocksDB License Summary: Closes https://github.com/facebook/rocksdb/pull/2589 Differential Revision: D5431502 Pulled By: siying fbshipit-source-id: 8ebf8c87883daa9daa54b2303d11ce01ab1f6f75 | 21 July 2017, 00:20:44 UTC |
10a12b2 | Siying Dong | 28 April 2017, 00:50:56 UTC | Add GPLv2 as an alternative license. Summary: Closes https://github.com/facebook/rocksdb/pull/2226 Differential Revision: D4967547 Pulled By: siying fbshipit-source-id: dd3b58ae1e7a106ab6bb6f37ab5c88575b125ab4 | 21 July 2017, 00:18:30 UTC |
915c90e | sdong | 17 July 2017, 22:39:37 UTC | [FB Only] use gcc-5 | 18 July 2017, 05:26:24 UTC |
b27c79a | Yi Wu | 13 July 2017, 18:48:42 UTC | Bump version to 5.4.8 | 13 July 2017, 18:48:42 UTC |
70f7fc3 | Andrew Kryczka | 28 June 2017, 22:50:17 UTC | bump patch number | 28 June 2017, 22:50:17 UTC |
2955407 | Siying Dong | 26 June 2017, 19:32:52 UTC | Fix bug that flush doesn't respond to fsync result Summary: With a regression bug was introduced two years ago, by https://github.com/facebook/rocksdb/commit/6e9fbeb27c38329f33ae541302c44c8db8374f8c , we fail to check return status of fsync call. This can cause we miss the information from the file system and can potentially cause corrupted data which we could have been detected. Closes https://github.com/facebook/rocksdb/pull/2495 Reviewed By: ajkr Differential Revision: D5321949 Pulled By: siying fbshipit-source-id: c68117914bb40700198fc37d0e4c63163a8a1031 | 28 June 2017, 22:48:52 UTC |
6e05979 | Yi Wu | 30 May 2017, 21:31:07 UTC | Bump version to 5.4.6 and update HISTORY.md | 30 May 2017, 21:31:07 UTC |
8c1f5c2 | Yi Wu | 25 May 2017, 21:08:16 UTC | Fix db_bench build break with blob db Summary: Lite build does not recognize FLAGS_use_blob_db. Fixing it. Closes https://github.com/facebook/rocksdb/pull/2372 Reviewed By: anirbanr-fb Differential Revision: D5130773 Pulled By: yiwu-arbug fbshipit-source-id: 43131d9d0be5811f2129af562be72cca26369cb3 | 30 May 2017, 20:40:29 UTC |
0ccaba2 | Yi Wu | 23 May 2017, 17:32:02 UTC | Fix rocksdb.estimate-num-keys DB property underflow Summary: rocksdb.estimate-num-keys is compute from `estimate_num_keys - 2 * estimate_num_deletes`. If `2 * estimate_num_deletes > estimate_num_keys` it will underflow. Fixing it. Closes https://github.com/facebook/rocksdb/pull/2348 Differential Revision: D5109272 Pulled By: yiwu-arbug fbshipit-source-id: e1bfb91346a59b7282a282b615002507e9d7c246 | 23 May 2017, 19:27:53 UTC |
4646cd4 | Sagar Vemuri | 19 May 2017, 20:25:59 UTC | Bump version to 5.4.5 and update HISTORY.md | 19 May 2017, 20:25:59 UTC |
2d6abca | Adam Retter | 12 May 2017, 18:23:14 UTC | Facility for cross-building RocksJava using Docker Summary: As an alternative to Vagrant, we can now also use Docker to cross-build RocksDB. The advantages are: 1. The Docker images are fixed; they include all the latest updates and build tools. 2. The Vagrant image, required scripts that ran for every build that would update CentOS and install the buildtools. This lead to slow repeatable builds, we don't need to do this with Docker as they are already in the provided images. The Docker images I have used have their Docker build files here: https://github.com/evolvedbinary/docker-rocksjava and the images themselves are available from Docker hub: https://hub.docker.com/r/evolvedbinary/rocksjava/ I have added the following targets to the `Makefile`: 1. `rocksdbjavastaticreleasedocker` this uses Docker to perform the cross-builds. It is basically the Docker version of the existing Vagrant `rocksdbjavastaticrelease` target. 2. `rocksdbjavastaticpublishdocker` delegates to `rocksdbjavastaticreleasedocker` and then `rocksdbjavastaticpublishcentral` to upload the artiacts to Maven Central. Equivalent to the existing Vagrant target: `rocksdbjavastaticpublish` Closes https://github.com/facebook/rocksdb/pull/2278 Differential Revision: D5048206 Pulled By: yiwu-arbug fbshipit-source-id: 78fa96ef9d966fe09638ed01de282cd4e31961a9 | 18 May 2017, 00:12:02 UTC |
0b129d1 | Adam Retter | 15 May 2017, 17:45:14 UTC | Make sure that zstd is statically linked correctly in the Java static build Summary: Closes https://github.com/facebook/rocksdb/issues/2280 Closes https://github.com/facebook/rocksdb/pull/2292 Differential Revision: D5061259 Pulled By: sagar0 fbshipit-source-id: eec89111d114c04beee5870a4eb4b51857754783 | 18 May 2017, 00:07:55 UTC |
58ef1ca | Adam Retter | 11 May 2017, 22:17:26 UTC | Build and link with ZStd when creating the static RocksJava build Summary: Closes https://github.com/facebook/rocksdb/pull/2279 Differential Revision: D5048161 Pulled By: yiwu-arbug fbshipit-source-id: 43742ff93137e0a35ea7e855692c9e9a0cd41968 | 18 May 2017, 00:07:09 UTC |
1371271 | Yi Wu | 16 May 2017, 18:50:43 UTC | s/std::snprintf/snprintf Summary: Looks like std::snprintf is not available on all platforms (e.g. MSVC 2010). Change it back to snprintf, where we have a macro in port.h to workaround compatibility. Closes https://github.com/facebook/rocksdb/pull/2308 Differential Revision: D5070988 Pulled By: yiwu-arbug fbshipit-source-id: bedfc1660bab0431c583ad434b7e68265e1211b1 | 16 May 2017, 19:05:01 UTC |
ab8129a | Yi Wu | 15 May 2017, 20:40:32 UTC | Fix build error with blob DB. Summary: snprintf is in <stdio.h> and not in namespace std. Closes https://github.com/facebook/rocksdb/pull/2287 Reviewed By: anirbanr-fb Differential Revision: D5054752 Pulled By: yiwu-arbug fbshipit-source-id: 356807ec38f3c7d95951cdb41f31a3d3ae0714d4 | 15 May 2017, 21:21:07 UTC |
87f35fb | Aaron Gao | 12 May 2017, 21:13:16 UTC | fix readamp test type inconsistency | 12 May 2017, 21:14:42 UTC |
9e58371 | Yi Wu | 11 May 2017, 18:43:56 UTC | Bump version to 5.4.4 and update HISTORY.md | 11 May 2017, 18:43:56 UTC |
1d4dc5e | Adam Retter | 10 May 2017, 15:54:46 UTC | Fixes the CentOS 5 cross-building of RocksJava Summary: Updates to CentOS 5 have been archived as CentOS 5 is EOL. We now pull the updates from the vault. This is a stop gap solution, I will send a PR in a couple days which uses fixed Docker containers (with the updates pre-installed) instead. sagar0 Here you go :-) Closes https://github.com/facebook/rocksdb/pull/2270 Differential Revision: D5033637 Pulled By: sagar0 fbshipit-source-id: a9312dd1bc18bfb8653f06ffa0a1512b4415720d | 11 May 2017, 18:37:41 UTC |
b549514 | Yi Wu | 11 May 2017, 16:43:59 UTC | Add missing files of blob_db to CMake file Summary: Some of the file from #2269 didn't add to CMake file. Adding them to fix window build. Closes https://github.com/facebook/rocksdb/pull/2276 Differential Revision: D5043487 Pulled By: yiwu-arbug fbshipit-source-id: 4eba853e9d92574353abce21d77d30e47ce43d3d | 11 May 2017, 18:25:05 UTC |
e8727ff | Anirban Rahut | 10 May 2017, 21:54:35 UTC | Blob storage pr Summary: The final pull request for Blob Storage. Closes https://github.com/facebook/rocksdb/pull/2269 Differential Revision: D5033189 Pulled By: yiwu-arbug fbshipit-source-id: 6356b683ccd58cbf38a1dc55e2ea400feecd5d06 | 11 May 2017, 18:24:46 UTC |
a6e1cf9 | Yi Wu | 11 May 2017, 05:46:24 UTC | Fix ColumnFamilyTest:BulkAddDrop Summary: Fix ColumnFamilyTest:BulkAddDrop not deleted CF handles at the end, causing ASAN failure. Closes https://github.com/facebook/rocksdb/pull/2275 Differential Revision: D5040724 Pulled By: yiwu-arbug fbshipit-source-id: 86cd4070c944d01173a3cc36462bb800698af192 | 11 May 2017, 18:23:14 UTC |
ded1d5a | Yi Wu | 08 May 2017, 05:12:55 UTC | Add bulk create/drop column family API Summary: Adding DB::CreateColumnFamilie() and DB::DropColumnFamilies() to bulk create/drop column families. This is to address the problem creating/dropping 1k column families takes minutes. The bottleneck is we persist options files for every single column family create/drop, and it parses the persisted options file for verification, which take a lot CPU time. The new APIs simply create/drop column families individually, and persist options file once at the end. This improves create 1k column families to within ~0.1s. Further improvement can be merge manifest write to one IO. Closes https://github.com/facebook/rocksdb/pull/2248 Differential Revision: D5001578 Pulled By: yiwu-arbug fbshipit-source-id: d4e00bda671451e0b314c13e12ad194b1704aa03 | 11 May 2017, 18:22:50 UTC |
dc0bbf7 | Siying Dong | 02 May 2017, 22:01:07 UTC | Fix an issue of manual / auto compaction data race Summary: A data race between a manual and an auto compaction can cause a scheduled automatic compaction to be cancelled and never rescheduled again. This may cause a condition of hanging forever. Fix this by always making sure the cancelled compaction is put back to the compaction queue. Closes https://github.com/facebook/rocksdb/pull/2238 Differential Revision: D4984591 Pulled By: siying fbshipit-source-id: 3ab153886403c7b991896dcb2158b96cac12f227 | 11 May 2017, 18:17:26 UTC |
b634fd7 | Aaron Gao | 10 May 2017, 21:18:54 UTC | Bump version to 5.4.3 | 10 May 2017, 21:18:54 UTC |
22f277e | Aaron Gao | 10 May 2017, 18:53:28 UTC | fix readampbitmap tests Summary: fix test failure of ReadAmpBitmap and ReadAmpBitmapLiveInCacheAfterDBClose. test ReadAmpBitmapLiveInCacheAfterDBClose individually and make check Closes https://github.com/facebook/rocksdb/pull/2271 Differential Revision: D5038133 Pulled By: lightmark fbshipit-source-id: 803cd6f45ccfdd14a9d9473c8af311033e164be8 | 10 May 2017, 21:06:54 UTC |
7e62c5d | Aaron Gao | 10 May 2017, 08:32:52 UTC | unbiase readamp bitmap Summary: Consider BlockReadAmpBitmap with bytes_per_bit = 32. Suppose bytes [a, b) were used, while bytes [a-32, a) and [b+1, b+33) weren't used; more formally, the union of ranges passed to BlockReadAmpBitmap::Mark() contains [a, b) and doesn't intersect with [a-32, a) and [b+1, b+33). Then bits [floor(a/32), ceil(b/32)] will be set, and so the number of useful bytes will be estimated as (ceil(b/32) - floor(a/32)) * 32, which is on average equal to b-a+31. An extreme example: if we use 1 byte from each block, it'll be counted as 32 bytes from each block. It's easy to remove this bias by slightly changing the semantics of the bitmap. Currently each bit represents a byte range [i*32, (i+1)*32). This diff makes each bit represent a single byte: i*32 + X, where X is a random number in [0, 31] generated when bitmap is created. So, e.g., if you read a single byte at random, with probability 31/32 it won't be counted at all, and with probability 1/32 it will be counted as 32 bytes; so, on average it's counted as 1 byte. *But there is one exception: the last bit will always set with the old way.* (*) - assuming read_amp_bytes_per_bit = 32. Closes https://github.com/facebook/rocksdb/pull/2259 Differential Revision: D5035652 Pulled By: lightmark fbshipit-source-id: bd98b1b9b49fbe61f9e3781d07f624e3cbd92356 | 10 May 2017, 21:06:54 UTC |
2014cdf | Aaron Gao | 05 May 2017, 22:01:04 UTC | do not read next datablock if upperbound is reached Summary: Now if we have iterate_upper_bound set, we continue read until get a key >= upper_bound. For a lot of cases that neighboring data blocks have a user key gap between them, our index key will be a user key in the middle to get a shorter size. For example, if we have blocks: [a b c d][f g h] Then the index key for the first block will be 'e'. then if upper bound is any key between 'd' and 'e', for example, d1, d2, ..., d99999999999, we don't have to read the second block and also know that we have done our iteration by reaching the last key that smaller the upper bound already. This diff can reduce RA in most cases. Closes https://github.com/facebook/rocksdb/pull/2239 Differential Revision: D4990693 Pulled By: lightmark fbshipit-source-id: ab30ea2e3c6edf3fddd5efed3c34fcf7739827ff | 10 May 2017, 21:06:33 UTC |
459e00b | Aaron Gao | 05 May 2017, 18:58:10 UTC | Roundup read bytes in ReadaheadRandomAccessFile Summary: Fix alignment in ReadaheadRandomAccessFile Closes https://github.com/facebook/rocksdb/pull/2253 Differential Revision: D5012336 Pulled By: lightmark fbshipit-source-id: 10d2c829520cb787227ef653ef63d5d701725778 | 09 May 2017, 22:18:49 UTC |
49412d9 | Aaron Gao | 04 May 2017, 08:21:31 UTC | fix memory alignment with logical sector size Summary: we align the buffer with logical sector size and should not test it with page size, which is usually 4k. Closes https://github.com/facebook/rocksdb/pull/2245 Differential Revision: D5001842 Pulled By: lightmark fbshipit-source-id: a7135fcf6351c6db363e8908956b1e193a4a6291 | 09 May 2017, 22:16:57 UTC |
5a8e732 | Islam AbdelRahman | 08 May 2017, 17:35:44 UTC | Bump version to 5.4.2 | 08 May 2017, 17:35:44 UTC |
f2e68d2 | Andrew Kryczka | 05 May 2017, 00:40:29 UTC | Avoid calling fallocate with UINT64_MAX Summary: When user doesn't set a limit on compaction output file size, let's use the sum of the input files' sizes. This will avoid passing UINT64_MAX as fallocate()'s length. Reported in #2249. Test setup: - command: `TEST_TMPDIR=/data/rocksdb-test/ strace -e fallocate ./db_compaction_test --gtest_filter=DBCompactionTest.ManualCompactionUnknownOutputSize` - filesystem: xfs before this diff: `fallocate(10, 01, 0, 1844674407370955160) = -1 ENOSPC (No space left on device)` after this diff: `fallocate(10, 01, 0, 1977) = 0` Closes https://github.com/facebook/rocksdb/pull/2252 Differential Revision: D5007275 Pulled By: ajkr fbshipit-source-id: 4491404a6ae8a41328aede2e2d6f4d9ac3e38880 | 08 May 2017, 17:32:32 UTC |
b073698 | Yi Wu | 02 May 2017, 04:49:09 UTC | Update HISTORY.md for 5.4.1 | 02 May 2017, 04:49:09 UTC |
f3dc93b | Maysam Yabandeh | 02 May 2017, 00:43:52 UTC | Avoid pinning when row cache is accessed Summary: With row cache being enabled, table cache is doing a short circuit for reading data. This path needs to be updated to take advantage of pinnable slice. In the meanwhile we disabling pinning in this path. Closes https://github.com/facebook/rocksdb/pull/2237 Differential Revision: D4982389 Pulled By: maysamyabandeh fbshipit-source-id: 542630d0cf23cfb1f0c397da82e7053df7966591 | 02 May 2017, 04:46:40 UTC |
30a6d4e | Yi Wu | 28 April 2017, 20:56:40 UTC | Update HISTORY.md for 5.4.1 | 28 April 2017, 20:56:40 UTC |
6d29d8b | Aaron Gao | 26 April 2017, 21:21:04 UTC | add prefetch to PosixRandomAccessFile in buffered io Summary: Every time after a compaction/flush finish, we issue user reads to put the table into block cache which includes a couple of IO that read footer, index blocks, meta block, etc. So we implement Prefetch here to reduce IO. Closes https://github.com/facebook/rocksdb/pull/2196 Differential Revision: D4931782 Pulled By: lightmark fbshipit-source-id: 5a13d58dcab209964352322217193bbf7ff78149 | 28 April 2017, 20:48:23 UTC |
1530f38 | Aaron Gao | 26 April 2017, 22:28:50 UTC | fix WritableFile buffer size in direct IO Summary: �fix the buffer size in case of ppl use buffer size as their block_size. Closes https://github.com/facebook/rocksdb/pull/2198 Differential Revision: D4956878 Pulled By: lightmark fbshipit-source-id: 8bb0dc9c133887aadcd625d5261a3d1110b71473 | 28 April 2017, 20:45:30 UTC |
2b621c1 | Yi Wu | 28 April 2017, 20:36:19 UTC | Bump version to 5.4.1 | 28 April 2017, 20:36:19 UTC |
71acb4c | Yi Wu | 28 April 2017, 19:33:30 UTC | Fix WriteBatchWithIndex address use after scope error Summary: Fix use after scope error caught by ASAN. Closes https://github.com/facebook/rocksdb/pull/2228 Differential Revision: D4968028 Pulled By: yiwu-arbug fbshipit-source-id: a2a266c98634237494ab4fb2d666bc938127aeb2 | 28 April 2017, 20:31:19 UTC |
ebbce5b | Maysam Yabandeh | 22 April 2017, 00:30:30 UTC | Respect deprecated flag in table options Summary: Closes https://github.com/facebook/rocksdb/pull/2197 Differential Revision: D4932434 Pulled By: maysamyabandeh fbshipit-source-id: 6c83c12d6d47e3f0640ab84954944215968f266f | 22 April 2017, 00:52:06 UTC |
fa58674 | Andrew Kryczka | 19 April 2017, 18:44:40 UTC | Change L0 compaction score using level size Summary: The goal is to avoid the problem of small number of L0 files triggering compaction to base level (which increased write-amp), while still allowing L0 compaction-by-size (so intra-L0 compactions cause score to increase). Closes https://github.com/facebook/rocksdb/pull/2172 Differential Revision: D4908552 Pulled By: ajkr fbshipit-source-id: 4b170142b2b368e24bd7948b2a6f24c69fabf73d | 20 April 2017, 00:27:12 UTC |
4623a55 | Maysam Yabandeh | 19 April 2017, 03:19:34 UTC | Re-add index_per_partition but as deprecated Summary: index_per_partition should have deprecated deprecated instead of being removed. It is causing backward compatibility issues. Closes https://github.com/facebook/rocksdb/pull/2173 Differential Revision: D4910947 Pulled By: maysamyabandeh fbshipit-source-id: 5c52939381847d232ede6866606f67f2b4b857ae | 19 April 2017, 03:47:48 UTC |
6df24fc | Yi Wu | 19 April 2017, 00:12:50 UTC | Hide event listeners from lite build Summary: Fixing lite build failure introduce by #2169. Closes https://github.com/facebook/rocksdb/pull/2174 Reviewed By: sagar0 Differential Revision: D4910619 Pulled By: yiwu-arbug fbshipit-source-id: 5213b7b7431cc258688793c8c28153025588d8d9 | 19 April 2017, 01:02:42 UTC |
8a1c349 | Siying Dong | 18 April 2017, 23:30:51 UTC | Add DB:ResetStats() Summary: Add a function to allow users to reset internal stats without restarting the DB. Closes https://github.com/facebook/rocksdb/pull/2167 Differential Revision: D4907939 Pulled By: siying fbshipit-source-id: ab2dd85b88aabe9380da7485320a1d460d3e1f68 | 19 April 2017, 00:22:35 UTC |
48fc484 | Yi Wu | 18 April 2017, 19:00:36 UTC | Blob storage helper methods Summary: Split out interfaces needed for blob storage from #1560, including * CompactionEventListener and OnFlushBegin listener interfaces. * Blob filename support. Closes https://github.com/facebook/rocksdb/pull/2169 Differential Revision: D4905463 Pulled By: yiwu-arbug fbshipit-source-id: 564e73448f1b7a367e5e46216a521e57ea9011b5 | 18 April 2017, 19:44:15 UTC |
1265ed7 | Aaron Gao | 15 April 2017, 01:43:55 UTC | remove warning Summary: st_blocks shows 16 though the right value is 8. This happens occasionally which seems a bug. Closes https://github.com/facebook/rocksdb/pull/2160 Differential Revision: D4893542 Pulled By: lightmark fbshipit-source-id: 68e832586b58bbc6162efbe83ce273f1570d5be3 | 15 April 2017, 02:50:21 UTC |
95c5e2d | Aaron Gao | 15 April 2017, 01:43:32 UTC | readahead backwards from sst end Summary: prefetch some data from the end of the file for each compaction to reduce IO. Closes https://github.com/facebook/rocksdb/pull/2149 Differential Revision: D4880576 Pulled By: lightmark fbshipit-source-id: aa767cd1afc84c541837fbf1ad6c0d45b34d3932 | 15 April 2017, 02:50:09 UTC |
8d7edd5 | Aaron Gao | 13 April 2017, 20:07:33 UTC | change use_direct_writes to use_direct_io_for_flush_and_compaction Summary: Replace Options::use_direct_writes with Options::use_direct_io_for_flush_and_compaction Now if Options::use_direct_io_for_flush_and_compaction = true, we will enable direct io for both reads and writes for flush and compaction job. Whereas Options::use_direct_reads controls user reads like iterator and Get(). Closes https://github.com/facebook/rocksdb/pull/2117 Differential Revision: D4860912 Pulled By: lightmark fbshipit-source-id: d93575a8a5e780cf7e40797287edc425ee648c19 | 14 April 2017, 23:19:53 UTC |
b6f6b73 | Aaron Gao | 13 April 2017, 20:05:50 UTC | add space for buggy kernel warning Summary: add the missing space Closes https://github.com/facebook/rocksdb/pull/2150 Differential Revision: D4880696 Pulled By: lightmark fbshipit-source-id: a4e0ad6a8ea45d6469d3f6c8514fdeb4cf10aaf5 | 14 April 2017, 23:19:41 UTC |
415be22 | Sagar Vemuri | 13 April 2017, 00:30:06 UTC | RocksDB Release 5.4 : Update HISTORY.md and build version. Summary: Closes https://github.com/facebook/rocksdb/pull/2142 Reviewed By: siying Differential Revision: D4874696 Pulled By: sagar0 fbshipit-source-id: 03e6e21735bb74e5a37cc913aabb2c250af558cc | 13 April 2017, 00:36:27 UTC |
3eab41d | Daniel Black | 12 April 2017, 22:04:41 UTC | java dependencies test -s -> use test -d Summary: To correct a build process where the JAVA_TEST_LIBDIR is a symlink to a cache directory. Test -s (size 0) on symlinks returns true, resulting in a mkdir over the top of the symlink resulting in failure. As a solution -d checks if it is a directory (or the symlink refers to a directory), which works in the case of real directories and symlinks to directories. Trivial I know but it was really easy for me to use a symlink here to prevent frequent downloads in a CI environment. Thanks for your consideration. Closes https://github.com/facebook/rocksdb/pull/1917 Differential Revision: D4612263 Pulled By: siying fbshipit-source-id: 4d458f8e1760068cdd6b5eae4bce6e12c400df41 | 12 April 2017, 22:13:41 UTC |
a22ed4e | Siying Dong | 12 April 2017, 22:00:03 UTC | internal_repo_rocksdb to build Java and RocksDB LITE Summary: Build Java and RocksDB LITE as a customized unit test under internal_repo_rocksdb. One thing I'm not sure is that whether these two tests are triggered in every flavor. Reviewed By: IslamAbdelRahman Differential Revision: D4855868 fbshipit-source-id: 82a1628b458744d7692bbd29ef7424cca1294031 | 12 April 2017, 22:13:41 UTC |
9f2cc59 | Islam AbdelRahman | 12 April 2017, 01:17:47 UTC | sync TARGETS file | 12 April 2017, 01:17:47 UTC |
10d7546 | Aaron Gao | 11 April 2017, 23:59:50 UTC | set readahead buffer size from roundup(user_size) + 4k to roundup(use… Summary: Users usually set readahead buffer to a multiple of 4k, more than that, usually a multiple of blocks. So previously we set real buffer size 512 * n + 4k, which may introduce an additional block reading. Closes https://github.com/facebook/rocksdb/pull/2138 Differential Revision: D4871504 Pulled By: lightmark fbshipit-source-id: b070faa51d92e976e8e8468c00692699e585e243 | 12 April 2017, 00:13:33 UTC |
ba7da43 | Aaron Gao | 11 April 2017, 23:53:57 UTC | fix db_stress crash caused by buggy kernel warning Summary: filter the warning out and only print it once. Closes https://github.com/facebook/rocksdb/pull/2137 Differential Revision: D4870925 Pulled By: lightmark fbshipit-source-id: 91b363ce7f70bce88b0780337f408fc4649139b8 | 11 April 2017, 23:56:59 UTC |
6257837 | Siying Dong | 11 April 2017, 23:51:15 UTC | Add ROCKSDB_JAVA_NO_COMPRESSION flag Summary: In some CI test environment, compression libraries can't be successfully built. It still helps to build RocksDB there. Provide such an option to skip to download and build compression libraries. Closes https://github.com/facebook/rocksdb/pull/2135 Differential Revision: D4872617 Pulled By: siying fbshipit-source-id: bb21ac373bc62a2528cdf1ca4547e05fcae86214 | 11 April 2017, 23:56:59 UTC |
6a6723e | Sagar Vemuri | 11 April 2017, 23:11:27 UTC | Move MergeOperatorPinning tests to be with other merge operator tests Summary: Moved MergeOperatorPinning tests from db_test2.cc to db_merge_operator_test.cc. [This is the same code as PR #2104 , which has already been reviewed, but I am creating a new PR as I cannot import from #2104 onto phabricator anymore even after rebasing. I'll close and discard #2104.] Closes https://github.com/facebook/rocksdb/pull/2125 Differential Revision: D4863312 Pulled By: sagar0 fbshipit-source-id: 0f71a7690aa09c1d03ee85ce2bc1d2d89e4f4399 | 11 April 2017, 23:15:06 UTC |
6a8d5c0 | Maysam Yabandeh | 11 April 2017, 20:49:04 UTC | Revert "Report cpu usage using time command" Summary: This reverts commit 97ec8a134968c342cdf7244352220bb292fd950c. Closes https://github.com/facebook/rocksdb/pull/2136 Differential Revision: D4870610 Pulled By: maysamyabandeh fbshipit-source-id: cdbfba135b065562f38f704f350a9a4e63a9a122 | 11 April 2017, 20:57:58 UTC |
8f47a97 | Siying Dong | 11 April 2017, 15:38:22 UTC | File level histogram should be printed per CF, not per DB Summary: Currently level histogram is only printed out for DB stats and for default CF. This is confusing. Change to print for every CF instead. Closes https://github.com/facebook/rocksdb/pull/2126 Differential Revision: D4865373 Pulled By: siying fbshipit-source-id: 1c853e0ac66e00120ee931cabc9daf69ccc2d577 | 11 April 2017, 15:42:03 UTC |
9300ef5 | Manuel Ung | 10 April 2017, 22:47:20 UTC | Fix shared lock upgrades Summary: Upgrading a shared lock was silently succeeding because the actual locking code was skipped. This is because if the keys are tracked, it is assumed that they are already locked and do not require locking. Fix this by recording in tracked keys whether the key was locked exclusively or not. Note that lock downgrades are impossible, which is the behaviour we expect. This fixes facebook/mysql-5.6#587. Closes https://github.com/facebook/rocksdb/pull/2122 Differential Revision: D4861489 Pulled By: IslamAbdelRahman fbshipit-source-id: 58c7ebe7af098bf01b9774b666d3e9867747d8fd | 10 April 2017, 23:06:00 UTC |
1f8b119 | Manuel Ung | 10 April 2017, 22:38:34 UTC | Limit maximum memory used in the WriteBatch representation Summary: Extend TransactionOptions to include max_write_batch_size which determines the maximum size of the writebatch representation. If memory limit is exceeded, the operation will abort with subcode kMemoryLimit. Closes https://github.com/facebook/rocksdb/pull/2124 Differential Revision: D4861842 Pulled By: lth fbshipit-source-id: 46fd172ea67cc90bbba829bf0d70cfab2261c161 | 10 April 2017, 22:42:26 UTC |
97ec8a1 | Maysam Yabandeh | 10 April 2017, 21:49:12 UTC | Report cpu usage using time command Summary: Run the time command before regression tests, parse the output, and add the numbers to the report. Closes https://github.com/facebook/rocksdb/pull/2101 Differential Revision: D4862781 Pulled By: maysamyabandeh fbshipit-source-id: 4a81caa5d14187d67093aad154c8f0ad56aba901 | 10 April 2017, 21:59:31 UTC |
20778f2 | Maysam Yabandeh | 10 April 2017, 19:37:07 UTC | Adding comments to the write path Summary: also did minor refactoring Closes https://github.com/facebook/rocksdb/pull/2115 Differential Revision: D4855818 Pulled By: maysamyabandeh fbshipit-source-id: fbca6ac57e5c6677fffe8354f7291e596a50cb77 | 10 April 2017, 19:43:34 UTC |
7124268 | Sagar Vemuri | 10 April 2017, 18:13:23 UTC | Reduce the number of params needed to construct DBIter Summary: DBIter, and in-turn NewDBIterator and NewArenaWrappedDBIterator, take a bunch of params. They can be reduced by passing in ReadOptions directly instead of passing in every new param separately. It also seems much cleaner as a bunch of the params towards the end seem to be optional. (Recently I introduced max_skippable_internal_keys, which added one more to the already huge count). Idea courtesy IslamAbdelRahman Closes https://github.com/facebook/rocksdb/pull/2116 Differential Revision: D4857128 Pulled By: sagar0 fbshipit-source-id: 7d239df094b94bd9ea79d145cdf825478ac037a8 | 10 April 2017, 18:14:14 UTC |
04abb2b | Willem Jan Withagen | 09 April 2017, 18:15:35 UTC | FreeBSD only requires WITH_JEMALLOC, not the rest Summary: The compiler error: ``` /home/jenkins/workspace/ceph-master/src/rocksdb/db/db_impl.cc:20:10: fatal error: 'jemalloc/jemalloc.h' file not found ^ 1 error generated. ``` But is does compile with the `WITH_JEMALLOC` set. So ignore all the other settings. Closes https://github.com/facebook/rocksdb/pull/2118 Differential Revision: D4858387 Pulled By: yiwu-arbug fbshipit-source-id: 05b982969dcab53669a73a903641e71641c714e7 | 09 April 2017, 18:26:50 UTC |
6173018 | Islam AbdelRahman | 07 April 2017, 23:58:07 UTC | dummy diff Summary: Closes https://github.com/facebook/rocksdb/pull/2114 Differential Revision: D4854860 Pulled By: IslamAbdelRahman fbshipit-source-id: b871c5b9ccc52d20f5ceacdd172dc70b1dbf9110 | 08 April 2017, 00:07:37 UTC |
360e996 | Islam AbdelRahman | 07 April 2017, 21:58:58 UTC | Summary: Remove .arcconfig fbshipit-source-id: fe9f000138776a14c507b24c952ecf50b48d71a9 | 07 April 2017, 21:58:58 UTC |
69a5e64 | Islam AbdelRahman | 07 April 2017, 21:40:30 UTC | Deprecate .arcconfig fbshipit-source-id: 2e88ce2a16d16620aa6ef36aabbf3e91be2fc779 | 07 April 2017, 21:40:30 UTC |
9690653 | Maysam Yabandeh | 07 April 2017, 18:25:42 UTC | Add a verify phase to benchmarks Summary: Check the result of the benchmark againt a specified truth_db, which is expected to be produced using the same benchmark but perhaps on a different commit or with different configs. The verification is simple and assumes that key/values are generated deterministically. This assumption would break if db_bench using rand variable differently from the benchmark that produced truth_db. Currently it is checked to work on fillrandom and readwhilewriting. A param finish_after_writes is added to ensure that the background writing thread will write the same number of entries between two benchmarks. Example: $ TEST_TMPDIR=/dev/shm/truth_db ./db_bench --benchmarks="fillrandom,readwhilewriting" --num=200000 --finish_after_writes=true $ TEST_TMPDIR=/dev/shm/tmpdb ./db_bench --benchmarks="fillrandom,readwhilewriting,verify" --truth_db /dev/shm/truth_db/dbbench --num=200000 --finish_after_writes=true Verifying db <= truth_db... Verifying db >= truth_db... ...Verified Closes https://github.com/facebook/rocksdb/pull/2098 Differential Revision: D4839233 Pulled By: maysamyabandeh fbshipit-source-id: 2f4ed31 | 07 April 2017, 18:39:12 UTC |
dd8f9e3 | Ayappan | 07 April 2017, 17:41:38 UTC | Fix compilation for GCC-5 Summary: Fixes this issue https://github.com/facebook/rocksdb/issues/2108 Closes https://github.com/facebook/rocksdb/pull/2109 Differential Revision: D4851965 Pulled By: yiwu-arbug fbshipit-source-id: 6ee807b | 07 April 2017, 17:54:12 UTC |
c2954f9 | Volker Mische | 07 April 2017, 17:36:09 UTC | Add Travis job to build examples Summary: Currently the examples fail, having a job building them would prevent that from happening. Closes https://github.com/facebook/rocksdb/pull/2007 Differential Revision: D4831570 Pulled By: yiwu-arbug fbshipit-source-id: e38adfa | 07 April 2017, 17:39:15 UTC |
72fc1e9 | Nikhil Benesch | 07 April 2017, 17:26:19 UTC | avoid non-existent O_DIRECT on OpenBSD Summary: OpenBSD doesn't have `O_DIRECT`, so avoid it. (RocksDB compiles successfully on OpenBSD with this patch.) Closes https://github.com/facebook/rocksdb/pull/2106 Differential Revision: D4847833 Pulled By: siying fbshipit-source-id: 214b785 | 07 April 2017, 17:39:15 UTC |
ff97287 | Siying Dong | 07 April 2017, 03:06:34 UTC | Refactor compaction picker code Summary: 1. Move universal compaction picker to separate files compaction_picker_universal.cc and compaction_picker_universal.h. 2. Rename some functions to make the code easier to understand. 3. Move leveled compaction picking code to a dedicated class, so that we we don't need to pass some common variable around when calling functions. It also allowed us to break down LevelCompactionPicker::PickCompaction() to smaller functions. Closes https://github.com/facebook/rocksdb/pull/2100 Differential Revision: D4845948 Pulled By: siying fbshipit-source-id: efa0ab4 | 07 April 2017, 03:09:34 UTC |
9e72939 | Aaron Gao | 07 April 2017, 01:08:53 UTC | only FALLOC_FL_PUNCH_HOLE when ftruncate is buggy Summary: In RocksDB, we sometimes preallocate the estimated space for a file to have better perf with fallocate (if supported). Usually it is a little bit bigger than the real resulting file size. At this time, we have to let the Filesystem reclaim the space not used. Ideally, calling ftruncate to truncate the file to its real size should be enough. HOWEVER, it isn't on tmpfs, which we witness in our case, with some buggy kernel version. ftruncate a file with preallocated space doesn't change number of the blocks used by the file, which means the space not used by the file is not returned to the filesystems. So in this case we need fallocate with FALLOC_FL_PUNCH_HOLE to explicitly reclaim the used blocks. It is a hack to cope with the kernel bug and usually we should not need it. Closes https://github.com/facebook/rocksdb/pull/2102 Differential Revision: D4848934 Pulled By: lightmark fbshipit-source-id: f1b40b5 | 07 April 2017, 01:25:03 UTC |
343b59d | Sagar Vemuri | 06 April 2017, 21:49:13 UTC | Move various string utility functions into string_util Summary: This is an effort to club all string related utility functions into one common place, in string_util, so that it is easier for everyone to know what string processing functions are available. Right now they seem to be spread out across multiple modules, like logging and options_helper. Check the sub-commits for easier reviewing. Closes https://github.com/facebook/rocksdb/pull/2094 Differential Revision: D4837730 Pulled By: sagar0 fbshipit-source-id: 344278a | 06 April 2017, 21:54:12 UTC |
1d068f6 | Ayappan | 06 April 2017, 20:59:24 UTC | Fix CompactRange incorrect buffer release Summary: While running `make jtest` using IBM Java, it fails at compactRangeToLevel with the below error. ``` Run: org.rocksdb.RocksDBTest testing now -> compactRangeToLevel JVMJNCK056E JNI error in ReleaseByteArrayElements: Got memory 0x00003FFF94AA8908 from object 0x00000000000C7F78, releasing from 0x00000000000C7F68 JVMJNCK077E Error detected in org/rocksdb/RocksDB.compactRange0(J[BI[BIZII)V JVMJNCK024E JNI error detected. Aborting. JVMJNCK025I Use -Xcheck:jni:nonfatal to continue running when errors are detected. Fatal error: JNI error Makefile:205: recipe for target 'run_test' failed make[1]: *** [run_test] Error 87 make[1]: Leaving directory '/home/ubuntu/rocksdb/java' Makefile:1542: recipe for target 'jtest' failed make: *** [jtest] Error 2 ``` After checking the code, it is vivid that we are messing up the `ReleaseByteArrayElements` args in `rocksdb_compactrange_helper`. ``` ................. 1959 s = db->CompactRange(compact_options, &begin_slice, &end_slice); 1960 } Closes https://github.com/facebook/rocksdb/pull/2060 Differential Revision: D4831427 Pulled By: yiwu-arbug fbshipit-source-id: dd02037 | 06 April 2017, 21:09:13 UTC |
df6f5a3 | Yi Wu | 06 April 2017, 20:59:31 UTC | Move memtable related files into memtable directory Summary: Move memtable related files into memtable directory. Closes https://github.com/facebook/rocksdb/pull/2087 Differential Revision: D4829242 Pulled By: yiwu-arbug fbshipit-source-id: ca70ab6 | 06 April 2017, 21:09:13 UTC |
107c5f6 | Tamir Duberstein | 06 April 2017, 20:54:49 UTC | CMake: more MinGW fixes Summary: siying this is a resubmission of #2081 with the 4th commit fixed. From that commit message: > Note that the previous use of quotes in PLATFORM_{CC,CXX}FLAGS was incorrect and caused GCC to produce the incorrect define: > > #define ROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE 1 > > This was the cause of the Linux build failure on the previous version of this change. I've tested this locally, and the Linux build succeeds now. Closes https://github.com/facebook/rocksdb/pull/2097 Differential Revision: D4839964 Pulled By: siying fbshipit-source-id: cc51322 | 06 April 2017, 21:09:13 UTC |
d2dce56 | Siying Dong | 06 April 2017, 02:02:00 UTC | Move some files under util/ to separate dirs Summary: Move some files under util/ to new directories env/, monitoring/ options/ and cache/ Closes https://github.com/facebook/rocksdb/pull/2090 Differential Revision: D4833681 Pulled By: siying fbshipit-source-id: 2fd8bef | 06 April 2017, 02:09:16 UTC |
c50e375 | Islam AbdelRahman | 06 April 2017, 00:18:21 UTC | Use a human readable size for level report Summary: Current ``` ** Compaction Stats [default] ** Level Files Size(MB} Score Read(GB} Rn(GB} Rnp1(GB} Write(GB} Wnew(GB} Moved(GB} W-Amp Rd(MB/s} Wr(MB/s} Comp(sec} Comp(cnt} Avg(sec} KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 49.02 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 76.1 1 2 0.322 0 0 Sum 2/0 49.02 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.1 1 2 0.322 0 0 Int 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 76.1 1 2 0.322 0 0 ``` New ``` ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn Key Closes https://github.com/facebook/rocksdb/pull/2055 Differential Revision: D4804576 Pulled By: IslamAbdelRahman fbshipit-source-id: 719be6a | 06 April 2017, 00:24:19 UTC |
ce64b8b | Siying Dong | 06 April 2017, 00:14:05 UTC | Divide db/db_impl.cc Summary: db_impl.cc is too large to manage. Divide db_impl.cc into db/db_impl.cc, db/db_impl_compaction_flush.cc, db/db_impl_files.cc, db/db_impl_open.cc and db/db_impl_write.cc. Closes https://github.com/facebook/rocksdb/pull/2095 Differential Revision: D4838188 Pulled By: siying fbshipit-source-id: c5f3059 | 06 April 2017, 00:24:19 UTC |
02799ad | Aaron Gao | 05 April 2017, 22:56:45 UTC | Revert "delete fallocate with punch_hole" Summary: This reverts commit 0fd574926cc9be7309c2247092d6b337fb022a5d. It breaks tmpfs on kernel 4.0 or earlier. We will wait for the fix before remove this part Closes https://github.com/facebook/rocksdb/pull/2096 Differential Revision: D4839661 Pulled By: lightmark fbshipit-source-id: 574a51f | 05 April 2017, 23:10:09 UTC |
e2a7b20 | Maysam Yabandeh | 05 April 2017, 19:16:13 UTC | Release note for partition filters Summary: I tagged it experimental since the configuring the filter partitions by size will not be ready for this release. Closes https://github.com/facebook/rocksdb/pull/2049 Differential Revision: D4831575 Pulled By: maysamyabandeh fbshipit-source-id: fcc2d25 | 05 April 2017, 19:24:22 UTC |
af256eb | Aaron Gao | 05 April 2017, 19:10:01 UTC | build db every monday Summary: Rebuilding regression db at the first run every monday. Closes https://github.com/facebook/rocksdb/pull/2093 Differential Revision: D4836961 Pulled By: lightmark fbshipit-source-id: 22d6c25 | 05 April 2017, 19:24:21 UTC |
e5a1372 | Dmitri Smirnov | 05 April 2017, 18:22:36 UTC | Rework test running script. Summary: Rework test running script. New options SuiteRun - runs specified executables as google suite test cases in parallel. Run - this option now runs executables in parallel the same as 'tests' RunAll - scans for test executables and attempts to run them all as suites except those mentiones in RunOnly (hardcoded in the script) or specified either in $ExcludeTestCases $ExcludeTestExe Closes https://github.com/facebook/rocksdb/pull/2089 Differential Revision: D4832212 Pulled By: yiwu-arbug fbshipit-source-id: 954990c | 05 April 2017, 18:39:20 UTC |
d659faa | Andrew Kryczka | 05 April 2017, 00:57:27 UTC | Level-based L0->L0 compaction Summary: Level-based L0->L0 compaction operates on spans of files that aren't currently being compacted. It reduces the number of L0 files, thus making write stall conditions harder to reach. - L0->L0 is triggered when base level is unavailable due to pending compactions - L0->L0 always outputs one file of at most `max_level0_burst_file_size` bytes. - Subcompactions are disabled for L0->L0 since we want to output one file. - Input files are chosen as the longest span of available files that will fit within the size limit. This minimizes number of files in L0. Closes https://github.com/facebook/rocksdb/pull/2027 Differential Revision: D4760318 Pulled By: ajkr fbshipit-source-id: 9d07183 | 05 April 2017, 01:09:11 UTC |
a12306f | Jevon Qiao | 04 April 2017, 23:24:08 UTC | Add a notice on gflags installation in INSTALL.md Summary: Add a notice on gflags installation to help people build and install RocksDB from source code correctly. Please see https://github.com/facebook/rocksdb/issues/1775 Closes https://github.com/facebook/rocksdb/pull/2061 Differential Revision: D4831323 Pulled By: yiwu-arbug fbshipit-source-id: 0df1f0e | 04 April 2017, 23:39:25 UTC |
43010a9 | Siying Dong | 04 April 2017, 23:23:00 UTC | Revert "[rocksdb][PR] CMake: more MinGW fixes" fbshipit-source-id: 43b4529 | 04 April 2017, 23:24:26 UTC |
a30b75c | Islam AbdelRahman | 04 April 2017, 23:09:31 UTC | Add buckifier script to github repo Summary: Add buckifier script and TARGETS file to github repo Closes https://github.com/facebook/rocksdb/pull/2083 Differential Revision: D4825822 Pulled By: IslamAbdelRahman fbshipit-source-id: 205f4a7 | 04 April 2017, 23:24:26 UTC |
3450ac8 | Tamir Duberstein | 04 April 2017, 21:53:30 UTC | CMake: more MinGW fixes Summary: See individual commits. yuslepukhin siying Closes https://github.com/facebook/rocksdb/pull/2081 Differential Revision: D4824639 Pulled By: IslamAbdelRahman fbshipit-source-id: 2fc2b00 | 04 April 2017, 22:09:17 UTC |
90cfd46 | Aaron Gao | 04 April 2017, 21:17:16 UTC | update IterKey that can get user key and internal key explicitly Summary: to void future bug that caused by the mix of userkey/internalkey Closes https://github.com/facebook/rocksdb/pull/2084 Differential Revision: D4825889 Pulled By: lightmark fbshipit-source-id: 28411db | 04 April 2017, 21:24:20 UTC |
e2c6c06 | Andrew Kryczka | 04 April 2017, 18:12:47 UTC | add TimedEnv Summary: I've needed Env timing measurements a few times now, so finally built something for it. Closes https://github.com/facebook/rocksdb/pull/2073 Differential Revision: D4811231 Pulled By: ajkr fbshipit-source-id: 218a249 | 04 April 2017, 18:24:12 UTC |