https://github.com/voldemort/voldemort

sort by:
Revision Author Date Message Commit Date
d2190fb Revert "Steps towards automating cluster zone expansion" 09 April 2015, 21:14:16 UTC
a754f35 Merge pull request #223 from gnb/master Steps towards automating cluster zone expansion 09 April 2015, 21:12:19 UTC
46df86e Merge pull request #250 from gnb/roswap2 Improve error messages in ROReplicationHelperCLI 08 April 2015, 23:08:25 UTC
0361984 Improve error messages in ROReplicationHelperCLI Split up one "Unqualified store" error message into three with three separate checks, so that the person who runs this code can actually tell which of those conditions went wrong. 08 April 2015, 21:22:48 UTC
db6de76 Refactor the HDFS fetcher 1) Move some code into a method 2) Allocate the buffer per fetch instead of per file. Tested by fetching 2 directories on HDFS and verified the output. 07 April 2015, 21:52:46 UTC
9b417b6 Incorrectly pushed the logging change to master I had an log info to see where the queries are being sent. This was stashed but not sure how it went to the master. Reverting the change. 07 April 2015, 21:41:45 UTC
f144780 Metadata queries are not sent to same zone Metadata queries for system stores are sent to lowest number node in the cluster instead of the zone. Added a hack to the local pref strategy if the client zone is set, use the zone local routing. The code is very complicated (unnecesarily) did not clean it up as I dont want to run it for all the scenarios and wanted to make a safe fix. 06 April 2015, 22:11:56 UTC
874bef9 RouteToAllStrategy routes to node 0..n RouteToAllStrategy tries the node always in a fixed order. This creates too much metadata queries on the node 0. For zoned cluster, the node with lowest id gets bombarded with too many connections and get queries. Create a shuffled node, when the cluster is initialized and use this in the routing strategy. The random seed is used at the initialization to make it random every time the cluster is re-initialized. 06 April 2015, 19:27:13 UTC
d004dd8 Added more tests for the ClientRequestFormat Added more tests to validate isCompleteResponse for the clientRequestFormat. Noticed that protocolBuffers will break if the server sends in less than 4 bytes of data. 31 March 2015, 16:10:11 UTC
224ce69 Test cases for client request response 1) Validates the request response. 2) Added more validation for missing version timestamps and other issues 3) Added backward compatibility tests. 31 March 2015, 15:41:44 UTC
f1f4853 Merge pull request #246 from gnb/hdfs-fetcher HdfsFetcher fixes 30 March 2015, 20:55:07 UTC
cff89fc Add a destination dir arg to HdfsFetcher main 30 March 2015, 20:01:23 UTC
0b576cd Allow HdfsFetcher to fetch individual files but only from the main(), not when fetch() is invoked by the server. 30 March 2015, 19:59:33 UTC
3a1f460 BnP improvement: - Removed a bunch of redundant constructors that made code unreadble. - Added a min.number.of.records config (defaults to 1) to prevent pushing empty stores. - Improved error handling and reporting in BnP's run function. 27 March 2015, 22:04:47 UTC
258a7c0 add local option for ReadOnlyReplicationHelperCLI 26 March 2015, 01:08:40 UTC
e55cf79 Merge pull request #244 from bhasudha/playing_around_with_rocksdb Rocksdb StorageEngine support for Voldemort 24 March 2015, 06:05:38 UTC
ad87133 Incorporating code review feedbacks - Adding optional read lock for get API - Config change for RocksDb default data directory - minor log fixes 22 March 2015, 19:46:47 UTC
e3e2bf7 Warn on add/delete store in SetMetadata command 1) Currently if you add or delete a store using set metadata the cluster will be in an inconsistent state. Added warning to the server side log if this happens 2) ReplaceNodeCLI does not work correctly if you start the node with empty stores xml. Fixed that. Now it accepts empty stores.xml or the same stores.xml as the other nodes. 3) get stores.xml returns different order different times. Made the ordering constant sorted by the storeName. 4) vadmin.sh meta check stores.xml verify if the store exists and it is queriable on the node. 22 March 2015, 19:23:53 UTC
de59607 Warn on add/delete store in SetMetadata command 1) Currently if you add or delete a store using set metadata the cluster will be in an inconsistent state. Added warning to the server side log if this happens 2) ReplaceNodeCLI does not work correctly if you start the node with empty stores xml. Fixed that. Now it accepts empty stores.xml or the same stores.xml as the other nodes. 3) get stores.xml returns different order different times. Made the ordering constant sorted by the storeName. 4) vadmin.sh meta check stores.xml verify if the store exists and it is queriable on the node. 18 March 2015, 19:51:42 UTC
4758860 Deleting recently added test store 18 March 2015, 05:09:21 UTC
4cba78e Using rocksdbjni jar from maven 18 March 2015, 04:57:44 UTC
2c17d1d New test store New test store with long key and string value. 18 March 2015, 04:28:46 UTC
ab005b9 Add logging messages 18 March 2015, 04:28:46 UTC
fad1e28 Adding Parameterized StorageEngineTest for Rocksdb In this commit; * RocksdbStorageEngineTest that extends AbstractStorageEngineTest * Some fixes to the RocksdbStorageEngine * Adding support for getVersions(ByteArray key) 18 March 2015, 04:28:46 UTC
c5a63e1 Modifying the unit test to be a parameterized test * Now unit test tests both RocksdbStorageEngine and PartitionPrefixedRocksDbStorageEngine * Fixed getALL unit test. 18 March 2015, 04:28:46 UTC
25ff98c Adding test case for getall 18 March 2015, 04:28:46 UTC
5c2f931 Adding more unit test cases 18 March 2015, 04:28:46 UTC
478d8a4 Adding new rocksdbjni jar and librocksdbjni.so 18 March 2015, 04:28:45 UTC
547bd4e Missed to add the PartitionPrefixedRocksDbStorageEngine class in the previous commit. Adding now. 18 March 2015, 04:28:41 UTC
a223c7d Adding basic functional implementation for PartitionPrefixedRocksdbStorageEngine This is similar to PartitionPrefixedBdbStorageEngine. TODO: May need to refactor both storage engines later 18 March 2015, 04:28:41 UTC
8023068 Adding first cut implementations for: * multiVersionPut * Iterators for keys and entries Also adding stubs for other BATCH APIs which can be implemented later based on performance. 18 March 2015, 04:28:41 UTC
7f577e3 Adding basic get after put test to test RocksDB APIs * My tests fail with "java.lang.UnsatisfiedLinkError: no rocksdbjni in java.library.path" . Need to fix this later. 18 March 2015, 04:28:41 UTC
ea60a8d Adding first implementation for delete API 18 March 2015, 04:28:41 UTC
fda545e Adding first impmenetations for getall and put APIs Also added common locking mechanism to be used for write operations 18 March 2015, 04:28:41 UTC
751d9f7 Changed rdb.data.dir to rocksdb.data.dir As suggested by Vinoth Added the RocksDB Storage Config in VoldemortConfig's default list. Added RocksDB's library loading in RocksDbStorageConfiguration Fails for me at runtime because of missing dependencies... will need to be revisited. 18 March 2015, 04:28:41 UTC
b7856c7 Adding v1 implementation for get API May need to change/add more later 18 March 2015, 04:28:41 UTC
2c0707e Added basic functionality in RocksDbStorageConfiguration... untested yet. 18 March 2015, 04:28:40 UTC
0bb0e88 Added some more RDB code.. 18 March 2015, 04:28:40 UTC
c7d3547 Added the rocksdbjni.jar to private-lib and created two empty implementations of necessary interfaces... 18 March 2015, 04:28:40 UTC
19a09dc Releasing Voldemort 1.9.8 17 March 2015, 23:51:10 UTC
41ea5ba Dependency clean up: - Removed tusk, RestHadoopFetcher and related classes and tests. - Now fetching libthrift, catalina-ant and tehuti from Maven. 17 March 2015, 21:54:41 UTC
6a9ea78 Further BnP cleanups: - Removed Azkaban dependency as much as possible. - Standardized on using voldemort.utils.Props as much as possible. - Deleted VoldemortMultiStoreBuildAndPushJob which is not actively used and suffering from code rot. - Added Content-Length header support in BnP HttpHook. - Added more utility functions to voldemort.utils.Props - Safeguards in BnP HttpHook's concurrent code. - Removed System.exit calls from BnP. 17 March 2015, 21:54:41 UTC
e8e6fb3 Rebalance log Logged the rebalance related messages which will be important in troubleshooting if the expansion fails. 14 March 2015, 00:26:50 UTC
66b62ef Print the summary of the stores.xml change When you use the new vadmin.sh meta set stores.xml it will print a summary of all the changes you made to the stores. It will categorize them to the following changes 1) Warn if New stores are being added 2) Warn if Stores are being removed 3) Unchnaged stores For the modified stores, it will add them to atleast one category 1) Replication factor changed 2) Schema changed 3) All other changes This will help you identify the changes to the stores.xml before you update it. The vadmin.sh meta check stores.xml will do a sample key fetch against the store and report if there is an issue. 14 March 2015, 00:26:43 UTC
5695e9d Ignore the config data directory in tar When you run the local test server and if you try tar , unknowingly you are packing all the data files which takes a long amount of time. So ignoring the data directory in the config folder. Sometimes eclipse messes up and drops .gradle and others in the bin directory, so ignoring them in the archive as well. 11 March 2015, 19:20:25 UTC
6128689 fix zone id for vadmin tool restore-from-replica command 06 March 2015, 19:40:22 UTC
394c8d1 Releasing Voldemort 1.9.7 04 March 2015, 23:00:00 UTC
e3e9aee add unit test for slop streaming in host-swap scenario 04 March 2015, 01:44:43 UTC
61c7182 refresh cached cluster object in admin connection verifier for slop pusher jobs 28 February 2015, 00:50:14 UTC
07f4d71 remove mex-connection constraint 27 February 2015, 01:29:28 UTC
0e4fefb cache admin client in admin connection verifier to eliminate threads and connections 27 February 2015, 01:29:28 UTC
be84e5a make vadmin --zone optional for restore-from-replica command 26 February 2015, 20:05:45 UTC
9ce0828 Nio client thread names for multiple factories If a Voldemort client creates multiple factories ( either for isolation or for talking to multiple clusters) the thread names are difficult to track in logs and stack dumps. Now the thread names will contain the factory identifiers in the names. Servers does not pass in Stats object, so the servers hosting this client will continue to be the same. 24 February 2015, 00:30:34 UTC
8562e78 minor fix to Client Shell 19 February 2015, 18:33:23 UTC
017ea62 Many Build and Push improvements: - Set umask for recursive HDFS permissions. - Added some retry logic for an HDFS operation in the HadoopStoreBuilder. - Added BnP Abstract and Http Hook classes for common use cases. - Improved logging and debuggability. - Removed Azkaban dependency from AvroStoreBuilderMapper. - Deleted the deprecated VoldemortBatchIndexJob. 19 February 2015, 02:05:04 UTC
cf0f0c5 Merge pull request #239 from voldemort/topush2 Minor log message fixes 17 February 2015, 18:17:02 UTC
5c717f2 Fix English typo in VoldemortBuildAndPushJob 17 February 2015, 18:00:04 UTC
6c8b7df Fix English typo in ClientRequestExecutorFactory 17 February 2015, 17:57:24 UTC
272cf2c Selector and Admin threads have same name In voldemort selector and admin threads have same name and it is difficult to debug in thread dumps and follow them in the logs. Renamed the threads to use the serviceName, which makes it easier to understand and follow. 14 February 2015, 02:08:59 UTC
e41339f Releasing Voldemort 1.9.6 03 February 2015, 23:21:42 UTC
13e984a Many Build And Push improvements: - Upgraded Azkaban dependency to 2.5.0, and fetch from Maven Central. Removed Azkaban from private-lib. - Upgraded Jackson dependency to 1.9.13. - Fixed BnP hooks default config (other it failed when unspecified). - Rethrow exceptions caught in BnP's run function to maintain previous behavior. - Sanity checks and better error reporting of avro configs. - Cleaned up duplicate references of UndefinedPropertyException. 03 February 2015, 18:56:38 UTC
6cac0af Add string definition for bnp job property keys 02 February 2015, 22:07:19 UTC
2efc0b2 Test fixes 1) Sometimes starting the voldemort server fails with the error ObsoleteVersion Exception. This is an override only used by the tests, so it is safe to modify it. 2) On Java 8, ConsistencyFetcher test fails as the set has a different ordering, probably because of the hashCode implementation difference. In this method ordering is not important so I modified the test to ignore the ordering. 31 January 2015, 01:10:42 UTC
1a9e8df Avoid extra allocation on AvroGenericVersioned The buffer is unnecessarily copied into a new Array which uses extra memory. Used a different API which takes in buffer, offset and length. This is what the previous call we used internally calls into. 31 January 2015, 01:10:42 UTC
e90b885 Added support for custom hooks in the VoldemortBuildAndPushJob. Heartbeat hooks run in a daemon thread. 28 January 2015, 00:29:15 UTC
4a7b365 Better IntelliJ IDEA support in the gradle build. 23 January 2015, 21:45:10 UTC
8bfd182 Read after truncate crashes the JVM If you truncate a read-only store and send read operations get/getall for one of the existent keys it crashes the JVM with SegFault. Some of the data is cached in java collections and it tries to access the memory mapped file which has been closed by the truncate. This causes the JVM to crash. Now the java collections are cleared and the code tries to handle this gracefully. But this is not an ideal code more of a workaround. Fixed spewing of log messages to the standard output, as they are already captured in the log4j loggers. Added unit test for the read after truncate scenario. 22 January 2015, 19:56:41 UTC
8e54d87 Retry on bind exceptions Retry 3 times on bind exceptions before giving up. Most of the tests fail with intermittent bind exceptions. 21 January 2015, 00:24:53 UTC
851625f ync the metadata version on all nodes Sync the metadata version on all nodes by computing the super set of version clock and the timestamp for all the stores. The vector clocks on the nodes have diverged so much because of the bugs, this option just computes the super set and fixes it. 17 January 2015, 02:55:24 UTC
fdbad13 Release voldemort 1.9.5 Releasing voldemort version 1.9.5 12 January 2015, 23:34:27 UTC
bfd1ec7 Version clock increments on each restart Voldemort verion clock increments on each restart and hence rejects future updates. Increment the clock only when necessary. Ideally if the version value already exists, the function should return early, but not sure when cluster.xml, stores.xml and for each stores a key was introduced. And what happens if the keys were not there. So found a safe middle ground to increment the version when the key is not there. 10 January 2015, 00:22:29 UTC
8387603 Cluster state equals and node names Cluster.equals returns true when the outline of the cluster is equal and when the node names are different. But this is not what we want in most cases. Let us say that you have a non-zoned cluster, on 2 different data centers. They are going to have the same node ids, same partitions but they are not equal. Even within the same data center if one cluster.xml has node 1 as A and other cluster.xml has node 1 as B, they are different but currently returns true. So modified the node equivalence to verify the equality. 09 January 2015, 22:41:19 UTC
fd71c8c ReplaceNodeCLI enhancements and tests 1) Made ReplaceNodeCLI unit testing friendly, removed System.exits with throw Exception and quitting in the main method instead of each methods. 2) Added post condition checks to the ReplaceNodeCLI to verify for consistent stores.xml and cluster.xml. Added the testing for ReplaceNode 1) Check if a node can be swapped when it is down 2) Check if a node can be swapped when it is up 3) Check if you can move a hard disk and replace a node down While doing the tests, there are many client threads simulating the traffic to make sure that they dont see any exceptions. 09 January 2015, 22:40:49 UTC
a033aba Enhance meta check to pinpoint different properties Enhance the meta check tool to pinpoint properties that are different. On stats cluster there are many properties and it is hard to see which one is different. So the tool analyzes the property level details and dumps them out. 08 January 2015, 00:13:03 UTC
4b9e652 Format the meta check-version output Formatted the meta check-version output to print the time a particular key was modified. This will help us in debugging the cluster bootstrap issues faster. 07 January 2015, 07:48:06 UTC
0f689dc ClusterVersion might be outdated after host swap Set the cluster version at the end of the host swap to help the client rebootstrap. 07 January 2015, 07:48:06 UTC
00ff96c Share the constant for cluster.xml, stores.xml Share the same Constant from SystemStoreConstants for stores.xml cluster.xml and metadata-versions. Previously each class declared its own constant and searching for the usage within the source was a nightmare. I had to do multiple file searches to piece together information. This will help in the future to make the dependencies easier to track. 07 January 2015, 07:48:05 UTC
b35755b Format only edited lines setting Format only edited lines setting 07 January 2015, 07:48:05 UTC
6e3f03c output the version numbers in meta check output the version numbers in the meta check command. This will help to debug the version numbers more easily. 07 January 2015, 07:48:05 UTC
14e6d54 Refactoring the test cases Further refactored the test cases. Tried to solidify the test case by adding more checks and saw that most of them failed because of existing bugs. So commented them out and left whatever comments I know of in those places. 07 January 2015, 07:48:05 UTC
27387e0 Merge pull request #236 from jwlent55/krati-entry-iteration-bug Fix the Krati entries iterator to properly handle key collisions. 03 January 2015, 09:55:31 UTC
a4b2678 Fix the Krati entries iterator to properly handle key collisions. 30 December 2014, 19:20:05 UTC
4e31f74 Merge pull request #235 from ctasada/issue234 Fixes issue voldemort/voldemort#234 Thanks for the fix, but the transform is not production tested and it has lots of rough edges. We are not using it internally so far. 23 December 2014, 19:37:27 UTC
e8125ef Merge pull request #233 from ctasada/issue232 Fixes voldemort/voldemort#232 23 December 2014, 19:32:18 UTC
b1363b2 Merge pull request #219 from ctasada/release-1.9.0-fixes Script fixes for latest Release - Verified that only windows batch files are affected. 23 December 2014, 19:27:08 UTC
5504d0a Fixes issue voldemort/voldemort#234 23 December 2014, 14:24:19 UTC
815a882 Fixes voldemort/voldemort#232 23 December 2014, 14:19:50 UTC
891dd95 Minor fixes for DeleteKeysCLI Added few minor fixes for DeleteKeyCLI 19 December 2014, 01:14:04 UTC
35414b4 Reformat and correct unit test for ReadOnlyReplicationHelperCLI 18 December 2014, 23:25:47 UTC
43495a8 Delete Keys CLI Add a tool Delete Keys CLI. This tool reads from a keyfile and deletes the key from the supplied stores. The tools are considered to be in human readable format and conversion will be attempted to the appropriate key. First of all understand that The tool also supports the following options 1) --delete-all-versions. If you have more than one value with conflicting versions, the tool will fail, because it may not have the value schema to de-serialize the value and resolve the conflict. The conflict resolution needs to happen before the key is deleted. 2) --nodeid <> --admin-url <>. If you want to delete keys only from a particular node. Use the above options. It is useful when you delete the keys and if a node went down, you want to rerun the tool with that option. 3) --find-keys-exist <> . After the delete you can run with this option to find if any of the keys exist. If the keys are found the tool dumps the version of each of the keys. The tool waits for the number of keys from each store before it completes. 18 December 2014, 02:31:25 UTC
ece26e2 Refactor the traffic generator Refactored the traffic generator into its own class. 18 December 2014, 02:31:25 UTC
c371265 Move slop drain function to server test utils Moved the slop drain function to server test utils. 18 December 2014, 02:31:25 UTC
884b93f Reformat ReadOnlyReplicationHelperCLI output 17 December 2014, 01:01:08 UTC
65de00c Add column names to the info printed by the tool 15 December 2014, 20:57:39 UTC
789a395 Productionize ReadOnlyReplicationHelper - Rename ReadOnlyHostSwapCLI -> REadOnlyReplicationHelper - Create unit test to verify the correctness - Change input from Cluster to BootstrapUrl to fit admin environment - Fix bugs and re-format help menu 13 December 2014, 00:36:09 UTC
6fe2142 RO tool that tells (1)source fileName(2)source node(3)fileName at destination This is important for RO host swap where SRE's need to find out where to copy the from and what to name it on the destination node 13 December 2014, 00:36:09 UTC
5e84a61 create admin api to return read-only storage file list 13 December 2014, 00:36:09 UTC
a920bda Releasing Voldemort 1.9.4 09 December 2014, 01:54:03 UTC
e236910 Refactor Client Traffic Verifier into its own class Refactored the client traffic verifier into its own class. No code change is done. This is just moving code into a new seperate class file and renaming the instantiation and references. Other refactorings will follow this checkin. 09 December 2014, 01:40:30 UTC
f691f67 Rename StoreVerifier to ConnectionVerifier 09 December 2014, 00:16:58 UTC
back to top