| d4f5935 | Abhinay Nagpal | 25 March 2013, 17:26:01 UTC | Minor changes to test case# Please enter the commit message for your changes. Lines starting | 25 March 2013, 17:26:01 UTC |
| 11dff85 | Abhinay Nagpal | 22 March 2013, 21:33:35 UTC | Added test case to test runtime exceptions Cleaned up the try catch logic | 22 March 2013, 22:31:17 UTC |
| 25d6298 | Abhinay Nagpal | 22 March 2013, 00:54:56 UTC | Adding test cases whch simulate intermittent exceptions etc in the hdfsfetcher and simulate retry logic this ensures checksum calculation is robust | 22 March 2013, 22:31:17 UTC |
| 1c606ef | Chinmay Soman | 21 March 2013, 21:40:56 UTC | Adding an extra catch block for Exception and Throwable types. This is used to catch the ClassNotFound exceptions | 22 March 2013, 22:31:17 UTC |
| b8f3a8d | Chinmay Soman | 21 March 2013, 17:34:12 UTC | Using a per file checksum generator in the file copy in HdfsFetcher. This is used to handle the case where we might retry the copy in case of a Filesystem (hdfs) error. | 22 March 2013, 22:31:17 UTC |
| 8f4e8b0 | Jay J Wylie | 22 March 2013, 21:39:15 UTC | Reduce chatter in logs while a node is unavailable (from INFO to DEBUG level output). | 22 March 2013, 21:39:15 UTC |
| 667ed50 | Jay J Wylie | 22 March 2013, 16:56:22 UTC | Fix to log4j set up in bin/run-class.sh that should correctly configure log4j. | 22 March 2013, 16:56:22 UTC |
| d0023e9 | Jay J Wylie | 20 March 2013, 22:34:50 UTC | Added pending release notes for this merge to master. | 20 March 2013, 22:34:50 UTC |
| 9c75277 | Jay J Wylie | 20 March 2013, 20:52:54 UTC | TODOs for refactoring and copyright header updates. | 20 March 2013, 20:52:54 UTC |
| 1a0b8f7 | Jay J Wylie | 20 March 2013, 18:01:13 UTC | Addressed all code review comments for KeySampler and KeyVersionFetcher. Renamed many classes and methods related to FetchStreamRequestHandler. - All sub-classes of FetchStreamRequestHandler have been renamed to have a more consistent nomenclature. - Did some further refactoring in the FullScan* classes to move more work from leaf classes to FullScanFetchRequestHandler.java - moved scan accounting to overall bae class - Added getNodesPartitionIdForKey method to StoreInstance to help with some fetch logic | 20 March 2013, 18:50:15 UTC |
| 96862f2 | Jay J Wylie | 20 March 2013, 17:30:36 UTC | Interim commit to rename a bunch of files. | 20 March 2013, 18:50:14 UTC |
| 92b80c0 | Jay J Wylie | 14 March 2013, 21:55:10 UTC | Addressed review feedback and TODOs for KeyVersionSamplerCLI (and renamed it to KeyVersionFetcherCLI). - mostly usability changes about command line options... - one copyright fix | 20 March 2013, 18:50:14 UTC |
| f5e8f5a | Jay J Wylie | 14 March 2013, 21:09:35 UTC | Addressed all review feedback and TODOs for KeySamplerCLI KeySamplerCLI - added options: --store-names, --partition-ids, --keys-per-second-limit, and --progress-period-ops - got rid of unnecessary (and weird) retry loop. Can add seomthing like that later if needed. - pass all partitions to fetcher now instead of one-at-a-time Also did cosmetic fixes for KeyVersionSamplerCLI and Entropy.java | 20 March 2013, 18:50:14 UTC |
| 96d1513 | Jay J Wylie | 14 March 2013, 17:36:14 UTC | Correctness fixes and significant refactoring of Fetch*StreamRequestHandlers. Expanded AdminFetchTest. Added more common helper methods to common base class of all fetchers FetchStreamRequestHandler. Added abstract base classes for partition-based fetching and non-partition-based fetching: - FetchPartitionStreamRequestHandler (partition-based) - FetchItemsStreamRequestHandler (non-partition-based) Refactored some code up to abstract base classes and made implementations as similar as possible (without heroic efforts) across all fetchers: - FetchEntriesStreamRequestHandler - FetchKeysStreamRequestHandler - FetchPartitionEntriesStreamRequestHandler - FetchPartitionKeysStreamRequestHandler Significant better test coverage in AdminFetchTest - tests fetching keys as well as fetching entries - tests partition-aware and non-partition-aware servers - tests per-partition limits on entries/keys fetched All of this clean up and additional testing lead to minor correctness fixes. Minor other clean ups of comments, override annotations, and fixes for KeySamplerCLI. | 20 March 2013, 18:50:14 UTC |
| 5aca5c0 | Jay J Wylie | 13 March 2013, 17:34:21 UTC | change maxRecords to recordsPerPartition in fetch API and protobuf These are cosmetic changes. The client-side and server-side code does not properly do recordsPerPartition yet. Added a few TODOs in the code too. | 20 March 2013, 18:50:14 UTC |
| a2d9ebb | Jay J Wylie | 13 March 2013, 16:53:11 UTC | remove skipRecords from fetching API and protobuf AFAIK skipRecords was never used. By inspection, the code that would have been exercised if it had been used has never been correct. Removing skipRecords from the code base. Also: - Added a number of TODOs to the code from the reviews - Changed some variable names | 20 March 2013, 18:50:14 UTC |
| d486f2a | Jay J Wylie | 12 March 2013, 19:57:22 UTC | Minor fixes for tests that broke due to changes elsewhere in the code. | 20 March 2013, 18:50:14 UTC |
| 96ed494 | Jay J Wylie | 11 March 2013, 23:03:20 UTC | Minor fix for change to AdminClient src/java/voldemort/client/protocol/admin/AdminClient.java - do not close down AdminStoreClient from queryKeys test/unit/voldemort/client/AdminServiceBasicTest.java - added some additional checks to test to confirm (non)existence of exceptions&values | 20 March 2013, 18:50:14 UTC |
| ad7ca98 | Jay J Wylie | 11 March 2013, 21:46:59 UTC | Many minor tweaks to ConsistencyFix code and related files to address minor review feedback. build.xml - fixed commenting out of 'protobuff' target src/java/voldemort/client/protocol/admin/AdminClient.java - add ClientConfig to constructor. This is needed for AdminStoreClient creation. It is confusing that we need both an AdminClientConfig and ClientConfig, but that is because the *ClientConfig code is so clumsy. - changed ".stop()" methods to ".close()" to be consistent with other interfaces. et cetera - Updated all copyright notices that have changed on this branch since December. This touched a ton of files... - annotated some TODOs with "(refactor)" to make refactoring todos easier to find. | 20 March 2013, 18:50:14 UTC |
| 78e9417 | Jay J Wylie | 08 March 2013, 00:45:28 UTC | Added unit tests for ConsistencyFix, ConsistencyFixWorker, and QueryKeyResult. Many other fixes and cleanup: src/java/voldemort/utils/ConsistencyFix.java - tweak many variable names - add close method to stop adminClient - broke out BadKey to wrap a key with its string representation st failed fixes of badkey's can be dumped in full to file to be retried (without any additional effort) - marked 'parseVersion' as deprecated since, if we do this again, we should dump bytes not strings - track obsolete version exceptions and various statuses in Stats src/java/voldemort/utils/ConsistencyFixCLI.java - clean up of arguments, variable names, etc. - cleanly close down fixer... src/java/voldemort/utils/ConsistencyFixWorker.java - more logger.trace output - minor cleanup test/common/voldemort/TestUtils.java - added getVersioned() helper method test/common/voldemort/config/stores.xml - added consistency-fix store test/unit/voldemort/store/routed/ReadRepairerTest.java - marked all tests as @Test test/unit/voldemort/utils/ConsistencyCheckTest.java - update copyright notice | 20 March 2013, 18:50:14 UTC |
| b0a000d | Jay J Wylie | 07 March 2013, 18:48:19 UTC | Actually adding files KeySamplerCLI and KeyVersionSamplerCLI. | 20 March 2013, 18:50:14 UTC |
| 797eeca | Jay J Wylie | 07 March 2013, 17:56:13 UTC | Documented correct method of compiling protobuffs by hand. | 20 March 2013, 18:50:14 UTC |
| 7669d86 | Jay J Wylie | 07 March 2013, 17:44:15 UTC | Added KeySampler and KeyVersionSampler tools as a first step towards replacing "entropy" tool. Added another argument to bulk fetch operations that specifies maxRecords so that server can fetch a subset of a partition. src/java/voldemort/utils/KeySamplerCLI.java - Samples keys from a cluster src/java/voldemort/utils/KeyVersionSamplerCLI.java - Given file that lists keys per store, samples versions from each "responsible node" for that key src/java/voldemort/client/protocol/admin/AdminClient.java - passed maxRecords through - TODO for future clean up of some types src/java/voldemort/client/protocol/pb/VAdminProto.java - auto generated! src/java/voldemort/server/protocol/admin/AdminServiceRequestHandler.java - white space src/java/voldemort/server/protocol/admin/FetchStreamRequestHandler.java src/java/voldemort/server/protocol/admin/FetchEntriesStreamRequestHandler.java src/java/voldemort/server/protocol/admin/FetchKeysStreamRequestHandler.java - handle maxRecords src/java/voldemort/server/protocol/admin/FetchPartitionKeysStreamRequestHandler.java src/java/voldemort/server/protocol/admin/FetchPartitionEntriesStreamRequestHandler.java - handle maxRecords - fixed usage of skipRecords src/java/voldemort/utils/Entropy.java - added maxRecords src/proto/voldemort-admin.proto - added mac_records to protobuff definition test/unit/voldemort/client/AdminFetchTest.java - added maxRecords field to test | 20 March 2013, 18:50:13 UTC |
| e50ad0a | Jay J Wylie | 04 March 2013, 18:08:32 UTC | Made rebalance --show-plan slightly more verbose and added yet another analysis for cluster balance ("zone primary"). src/java/voldemort/client/rebalance/RebalancePartitionsInfo.java - print out hostname within plan to make it easier to read (rather than having to lookup node ID) src/java/voldemort/utils/ClusterInstance.java - calculate "zone primary" balance to understand which hosted partitions act as pseudo-master when zoned routing is used. | 20 March 2013, 18:50:13 UTC |
| cd62a15 | Jay J Wylie | 26 February 2013, 19:07:25 UTC | Review and cleanup of consistency checker. - added required argument for an output file name for bad keys - changed Reporter to print out 'just the key' to the output file; it outputs more info at DEBUG level in general. - removed 'quiet' option - throw exceptions: - if # partitions differ across clusters - if replication factor is hinky - if isExpired encounters unknown type - main catches exceptions and fails fast - changed system.out debugging to logger.trace | 20 March 2013, 18:50:13 UTC |
| c66a0c2 | Jay J Wylie | 22 February 2013, 19:37:34 UTC | Added more info to cluster dump to track which nodes host 'hot' partitions. | 20 March 2013, 18:50:13 UTC |
| b05967b | Jay J Wylie | 22 February 2013, 16:37:47 UTC | Tweaked Rebalancer --output-dir again to better name interim metadata files for each batch. | 20 March 2013, 18:50:13 UTC |
| 1162dbb | Jay J Wylie | 22 February 2013, 00:17:03 UTC | Changed Rebalancer --output-dir option to append numbers to each .xml file it outputs so that we have access to interim cluster configs. | 20 March 2013, 18:50:13 UTC |
| 433ce02 | Zhongjie Wu | 21 February 2013, 01:58:28 UTC | Refactored Consistency Check | 20 March 2013, 18:50:13 UTC |
| 9d3fb7e | Jay J Wylie | 19 February 2013, 21:42:08 UTC | Added server-put tracking to progress bar. | 20 March 2013, 18:50:13 UTC |
| 104c927 | Jay J Wylie | 13 February 2013, 19:15:53 UTC | Default to printing out BADKEYs from ConsistencyCheck. Cleaned up debug/trace messages in ConsistencyFix.java. | 20 March 2013, 18:50:13 UTC |
| 0c6893c | Jay J Wylie | 12 February 2013, 17:11:17 UTC | Fixed hashmap issues in AdminClient raised during code review. Added '--parse-only' option to ConsistencyFix. src/java/voldemort/client/protocol/admin/AdminClient.java - Added hashCode & equals methods to AdminClient.Nodestore - cleaned up getSocketStore to not leak concurrently created socket stores. src/java/voldemort/utils/ConsistencyFix(CLI).java - added parse only flag which limits that actions of the fixer to bootstrapping and parsing the input file. | 20 March 2013, 18:50:13 UTC |
| 213003a | Jay J Wylie | 11 February 2013, 22:30:17 UTC | Added 'dry-run' option and cleaned up help message. '--dry-run' option goes through all of the read paths (reading files, reading from servers) and calculates what to write where, but does not actually do any writes! Should combine --dry-run with these log4j settings: log4j.logger.voldemort.utils.ConsistencyFix=TRACE log4j.logger.voldemort.utils.ConsistencyFixWorker=DEBUG | 20 March 2013, 18:50:13 UTC |
| eceaf01 | Jay J Wylie | 11 February 2013, 21:58:40 UTC | Code fixes for the fixing of orphans. src/java/voldemort/utils/ConsistencyFix.java - added .trace output for parsing of ugly input - pass the correct key-type into constructor src/java/voldemort/utils/ConsistencyFixWorker.java - substantially more .debug output to trace operation | 20 March 2013, 18:50:13 UTC |
| c29bdf5 | Jay J Wylie | 11 February 2013, 16:34:55 UTC | Missed afile when doing the hand merge of consistency check stuff. | 20 March 2013, 18:50:13 UTC |
| 1d115da | Jay J Wylie | 11 February 2013, 16:14:53 UTC | Actually invoke the BadKeyOrphanReader. Also committing a bunch of TODOs for later cleanup. | 20 March 2013, 18:50:13 UTC |
| f573b53 | Jay J Wylie | 10 February 2013, 22:40:28 UTC | Added basic code for repairing orphaned key,values. src/java/voldemort/utils/ConsistencyFix.java - added BadKeyOrphanReader extends BadKeyReader to consume different input file src/java/voldemort/utils/ConsistencyFixCLI.java - added "orphan-format" flag to indicate that the 'bad-key-file-in' is of orphaned key/values. src/java/voldemort/utils/ConsistencyFixWorker.java - added constructor to take QueryKeyResult of orphaned keys - modified resolveReadConflicts to add orphaned key/values to imaginary nodes for the sake of determine the value/version to be repaired | 20 March 2013, 18:50:13 UTC |
| 8cb8efa | Jay J Wylie | 09 February 2013, 23:41:38 UTC | Added per-server throttling to the Consistency Fixer. Added a map of EventThrottle objects such that repair traffic to each server can be throttled. We care about throttling write rate because of its potential impact on GC and cleaning. | 20 March 2013, 18:50:13 UTC |
| e5ddb34 | Jay J Wylie | 09 February 2013, 00:33:10 UTC | Re-commiting a series of ZWu's commits to the consistency fix in a single batch. The other commits were not made against the same master and so this is easier than trying to figure out what went wrong in the merge/rebase. | 20 March 2013, 18:50:13 UTC |
| 44f1737 | Jay J Wylie | 08 February 2013, 02:24:22 UTC | Complete implementation of consistency fixer. src/java/voldemort/utils/ConsistencyFix.java - added Execute method that orchestrates all the threads - switched pattern of thread execution: - one thread for reading bad keys file that submits to ... - a thread pool of workers that enqueues badkeys for the ... - one thread writing still bad keys - note: construct thread pool with a blocking queue - switched to logging (rather than System.out/err) - added Stats tracking - moved methods that do complicated work (getting & repairing keys) out src/java/voldemort/utils/ConsistencyFixWorker.java - moved methods that do complicated work (getting & repairing keys) in src/java/voldemort/utils/ConsistencyFixCLI.java - added 'progress-bar' option - got rid of 'verbose' option - moved all thread orchestration to ConsistencyFix | 20 March 2013, 18:50:13 UTC |
| 73d62a1 | Jay J Wylie | 07 February 2013, 04:08:32 UTC | Heavily refactored ConsistencyFix. Some key outstanding TODOs remain. src/java/voldemort/utils/ConsistencyFix.java - split out much functionality - switched from having entirely static interfaces to being a normal non-static class src/java/voldemort/utils/ConsistencyFixCLI.java - split out CLI aspect from ConsistencyFix src/java/voldemort/utils/ConsistencyFixKeyGetter.java - split out thread for getting bad keys from ConsistencyFix src/java/voldemort/utils/ConsistencyFixRepairPutter.java - split out thread for repairing bad keys from ConsistencyFix | 20 March 2013, 18:50:13 UTC |
| 2aea4e6 | Jay J Wylie | 07 February 2013, 02:43:03 UTC | Parallelized the consistency fixer. Interim checkin. Need to refactor into separate files. src/java/voldemort/utils/ConsistencyFix.java - 1 thread for reading file of bad keys - 1 thread for writing any bad keys that are not fixed - thread pool for sending gets - thread pool for sending puts | 20 March 2013, 18:50:12 UTC |
| 02e8063 | Jay J Wylie | 06 February 2013, 00:02:57 UTC | Separate simple admin ops from streaming ops and clean up ConsistencyFix src/java/voldemort/client/protocol/admin/AdminClient.java - separated simple StoerOperations from StreamingOperations - note: renamed 'StreamingStoreOperations storeOps' to 'StreamingOperations streamingOps' - pulled the exception handling logic out of queryKeys/repairEntries and reanmed to more generic names (getNodeKey and putNodeKeyValue). src/java/voldemort/client/protocol/admin/QueryKeyResult.java - made CTors public src/java/voldemort/utils/ConsistencyFix.java - significant clean up & refactoring - match changes in voldemortadmintool other - some other TODOS - other files touched due to renames within AdminClient | 20 March 2013, 18:50:12 UTC |
| 3c2fe12 | Jay J Wylie | 05 February 2013, 21:30:16 UTC | Added AdminStoreClient helper class to AdminClient. src/java/voldemort/client/protocol/admin/AdminClient.java - added AdminStoreClient which manages a ClientRequestExecutorPool and caches pertinent SocketStore (Store<ByteArray, byte[], byte[]>) objects. This allows the admin client to re-use connections when doing basic store operations (put, get, etc) against individual servers. - added TODOs about other methods I think should be cleaned up - significantly revised repairEntry - added a queryKey method that uses the AdminStoreclient src/java/voldemort/client/protocol/admin/RepairEntryResult.java - return type for AdminClient.StreamingStoreOperations.repairEntry() src/java/voldemort/utils/ConsistencyFix.java b/src/java/voldemort/utils/ConsistencyFix.java - clean up to keep up with changes to AdminClient interfaces & types - cleaned up TODOs | 20 March 2013, 18:50:12 UTC |
| 79de267 | Jay J Wylie | 04 February 2013, 18:04:02 UTC | Fix errors introduced during rebase process. Rebase with master was ugly. Picked up a bunch of AdminClient refactoring. These changes conflicted with many changes on the persistency-check branch. Had to fix the return type of AdminClient.queryKeys by hand (to iterator<QueryKeyResult>). Added some TODO comments to AdminClient.HelperOperations methods that should move into ClusterInstance. Also updated some copyright dates. | 20 March 2013, 18:50:12 UTC |
| 37d3b76 | Jay J Wylie | 04 February 2013, 16:36:05 UTC | Minor comment/todo clean up. | 20 March 2013, 18:50:12 UTC |
| 83b87bc | Jay J Wylie | 31 January 2013, 22:00:10 UTC | Moved AdminClient.QueryKeyResult into its own file. src/java/voldemort/client/protocol/admin/QueryKeyResult.java - A more complete, proper class based on the inner class that had been in AdminClient.java | 20 March 2013, 18:50:12 UTC |
| 5db6a1d | Jay J Wylie | 29 January 2013, 22:19:06 UTC | Added more TODOs from code review feedback. | 20 March 2013, 18:50:12 UTC |
| 6ba10a1 | Jay J Wylie | 29 January 2013, 21:27:43 UTC | Further refactoring. src/java/voldemort/utils/ConsistencyFix.java - rename inner class VoldemortInstance to ConsistencyFixContext - Drop methods that were duplicated in StoreInstance src/java/voldemort/utils/StoreInstance.java - clean up comments for method getNodeIdListForPartitionIdList | 20 March 2013, 18:50:12 UTC |
| 7e98b10 | Jay J Wylie | 29 January 2013, 21:05:54 UTC | Significant refactoring of util methods. Added helper classes ClusterInstance, StoreInstance, and StoreDefinitionUtil. ClusterInstance wraps up a Cluster and List<StoreDefinition> object and provides helper methods that operate on these objects together. StoreInstance wraps up a Cluster and a StoreDefinition object and provides helper methods that operate on these objects together. StoreDefinitionUtil provides helper method that operate on either StoreDefinition or List<StoreDefinition> objects. Many methods have migrated out of RebalanceUtils into these new classes. | 20 March 2013, 18:50:12 UTC |
| a345a05 | Jay J Wylie | 29 January 2013, 01:13:48 UTC | Minor fix due to merging of persistency-check and rebalance-bug-fix branches | 20 March 2013, 18:50:12 UTC |
| 796aa5d | Jay J Wylie | 25 January 2013, 19:07:51 UTC | Minor tweak to dealing with contiguous runs of partitions. - Hard code 10 repeated attempts to get rid of contiguous partitions. - Added a TODO to do something perfect at some unspecified time in the future... | 20 March 2013, 18:50:12 UTC |
| 10d57ac | Jay J Wylie | 09 January 2013, 17:23:40 UTC | Fixed minor issue with tests in AdminServiceBasicTest. They were not including an appropriate zone list when constructing a cluster. | 20 March 2013, 18:50:12 UTC |
| b8f93b2 | Jay J Wylie | 09 January 2013, 16:07:44 UTC | Update copyright notices. | 20 March 2013, 18:50:12 UTC |
| a1c0724 | Jay J Wylie | 09 January 2013, 15:58:43 UTC | Minor tweaks and refactoring src/java/voldemort/utils/ClusterUtils.java - added method to pretty print partition lists - added methods to determine histogram of contiguous partition run lengths and pretty print - moved verbose pretty printing of cluster details from ClusterRebalanceUtils to here src/java/voldemort/utils/RebalanceClusterUtils.java - randomized order in which zones are processed for shuffle algorithms | 20 March 2013, 18:50:12 UTC |
| 8405330 | Jay J Wylie | 08 January 2013, 21:29:09 UTC | Module level refactor: split methods out of RebalanceUtils and RebalanceClusterUtils.java into new modules NodeUtils.java and ClusterUtils.java. Any helper methods that operate on individual clusters or on individual (sets of) nodes are now in the appropriate util class. | 20 March 2013, 18:50:12 UTC |
| f42a939 | Jay J Wylie | 08 January 2013, 18:37:48 UTC | Module level refactor : split bunch of stuff out of RebalanceUtils.java into new file RebalanceClusterUtils.java All of the algorithms to generate a new cluster.xml that is (more) balanced has been moved into this new util class. | 20 March 2013, 18:50:12 UTC |
| b2cc8f1 | Jay J Wylie | 08 January 2013, 17:35:46 UTC | Addressed code review feedback on rebalance utility and cleaned up code src/java/voldemort/client/rebalance/RebalanceCLI.java - Changed pattern for boolean command line options to be cleaner - Added more options to control when cross zone moves are checked - Added more tests to confirm the specified sub-options for generate are mutually compatible - Revised expansive secondary documentation of various options to cover all options src/java/voldemort/client/rebalance/RebalanceClientConfig.java - minor varaible rename src/java/voldemort/cluster/Cluster.java - add members and methods to track partitions & nodes by zone src/java/voldemort/cluster/Node.java - minor fix src/java/voldemort/utils/RebalanceUtils.java - refactored many of the methods I added to break them into smaller pieces - by moving helper methods into Cluster.java, got rid of lots of redundant code - cleaned up javadoc comments for the methods - now prints out analysis for each try that improves balance - validates num partitions for modified cluster same as original cluster test/unit/voldemort/utils/RebalanceUtilsTest.java - tests for some of the complicated/nuanced helper methods | 20 March 2013, 18:50:12 UTC |
| 8ff3145 | Jay J Wylie | 12 December 2012, 21:24:56 UTC | Improvements to greedy swapping algorithm. The completely greedy algorithm was too expensive. Have balanced the greedy approach with optional randomness & limits on number of swaps attempted in each round. Exposed options all the way through the CLI. | 20 March 2013, 18:50:11 UTC |
| 7c9a739 | Jay J Wylie | 12 December 2012, 18:42:48 UTC | Initial implementation of a greedy swapping algorithm. Greedy swapping is unbelievably expensive because it tests every possible pair-wise partition swap within a zone before making a single swap. The goal is to get the most improvement possible with each swap (to minimize the number of swaps total and so the amount of data movement). | 20 March 2013, 18:50:11 UTC |
| d2ef548 | Jay J Wylie | 11 December 2012, 21:22:14 UTC | complete transition to max/min ratio for balancing cluster. | 20 March 2013, 18:50:11 UTC |
| 4a76cab | Jay J Wylie | 11 December 2012, 17:13:24 UTC | Tweaked rebalance swapping algorithm to improve effectiveness. Cleaned up code. Changed logic of random swapping to iterate over zones while randomly swapping. Because of the dependencies between zones, this appears to work much better. Cleaned up comments, TODOs, and so on to prepare for code review. | 20 March 2013, 18:50:11 UTC |
| 0866bec | Jay J Wylie | 11 December 2012, 16:35:28 UTC | Added more rebalance options and refactored some of the rebalance utils code - added random swap option that swaps random partitions between random nodes within a zone. This is a poor version of "simulated annealing". It produced some good results though in initial tests. - Extended the cross zone partition move code that ensures there are no long contiguous runs of partitions within a single zone. Now, this code is followed by code that can balance the number of partitions in each zone. All of this code now runs in a loop until no further cross zone moves are performed. (A single pass of cross zone moves followed by balancing # of partitions per zone runs the risk of having contiguous partitions again.) - Made most parameters that control these features command line options for the rebalancing --generate option. | 20 March 2013, 18:50:11 UTC |
| 9d8d065 | Jay J Wylie | 07 December 2012, 21:15:18 UTC | Add option to limit the number of contiguous partitions within a zone. - code to better balance partitions between zones is currently added to the main rebalance code. this may not be the right place for this code. - TODO outstanding to figure out if the contig partitions manipulation can be done and then the rest of rebalancing can continue. Currently, the rest of the code gets broken and a separate instantiion of rebalancing needs to be done. | 20 March 2013, 18:50:11 UTC |
| 4dae67f | Jay J Wylie | 07 December 2012, 17:45:02 UTC | Made analyze cluster option much more verbose. Dumps all partition maps and summarizes some aggregate partitions per node information. | 20 March 2013, 18:50:11 UTC |
| 22fd241 | Jay J Wylie | 07 December 2012, 16:40:30 UTC | Add many options to vary how generating partitions works. - turn on/off check to keep primary partitions within same zone - turn on/off check for other partitions that move zones - specify amount of randomness among num partitions per node (w/in zone) | 20 March 2013, 18:50:11 UTC |
| a9a4c22 | Jay J Wylie | 06 December 2012, 18:51:44 UTC | Re-wrote logic for re-balancing partitions in RebalanceUtils.generateMinCluster(). * Separates stealer nodes from donor nodes within each zone based on a goal of evenly distributing partitions within a zone across all nodes within that zone. * Permits rebalancing logic to be run for a target cluster.xml that does not include any new nodes. * Much more verbose output * A few TODOs outstanding for review discussion - Do multiple tries serve any purpose? - Does the change that may make an existing node a stealer to smooth out the partition distribution break any other rebalance logic? - Why does RebalanceUtils.getCrossZoneMoves sometimes return 0? This tool code be sped up by many orders of magnitude for large clusters if we could avoid calling RebalanceUtils.getCrossZoneMoves | 20 March 2013, 18:50:11 UTC |
| a46c699 | Jay J Wylie | 06 December 2012, 00:41:03 UTC | Hack to better balance stealing/donation | 20 March 2013, 18:50:11 UTC |
| 879d774 | Jay J Wylie | 05 December 2012, 23:32:31 UTC | Initial work on fixing out rebalance to more evenly distributed stealing and donating. | 20 March 2013, 18:50:11 UTC |
| 9706394 | Jay J Wylie | 23 January 2013, 16:31:41 UTC | Cleaned up initial implementation of ConsistencyFix for preliminary code review. src/java/voldemort/client/protocol/admin/AdminClient.java - annotated repairEntry method with TODO items to get review feedback on - Created new type QueryKeyResult to get rid of a struct of type Pair<ByteArray, Pair<List<Versioned<byte[]>>, Exception>> src/java/voldemort/utils/ConsistencyFix.java - added output file to list non-fixed keys in - modulate output based on verbose flag - refactored giant method(s) into smaller methods - cleaned up TODO comments for review feedback test/unit/voldemort/client/AdminServiceBasicTest.java - uses QueryKeyResult type src/java/voldemort/VoldemortAdminTool.java - uses QueryKeyResult type | 20 March 2013, 18:50:11 UTC |
| 719fa74 | Jay J Wylie | 22 January 2013, 00:48:46 UTC | Added multiple-key options to ConsistencyFix tool. src/java/voldemort/utils/ConsistencyFix.java - Options for keys: --key : single key --keys : list of keys on command line --key-file : file of keys, one per line. keys are in hexadecimal. - A bunch of TODOs for next step code cleanup, hardening, improving. | 20 March 2013, 18:50:11 UTC |
| 9aef21e | Jay J Wylie | 17 January 2013, 21:59:39 UTC | First step towards a consistency fixer tool. src/java/voldemort/utils/ConsistencyFix.java - CLI and initial implementation of fixer - each invocation fixes one key for one store on one cluster src/java/voldemort/client/protocol/admin/AdminClient.java - added repairEntry method. Very much a work-in-progress. src/java/voldemort/utils/ByteUtils.java - added fromHexString helper function. (Sister function of toHexString) test/unit/voldemort/utils/ByteUtilsTest.java - improved tests for to/from HexString | 20 March 2013, 18:50:11 UTC |
| 386560a | Zhongjie Wu | 15 January 2013, 21:41:00 UTC | Updated ConsistencyCheck.java for new feature | 20 March 2013, 18:50:11 UTC |
| e7d61e7 | Zhongjie Wu | 10 January 2013, 06:07:31 UTC | Added consistency check tool. Support single partition only | 20 March 2013, 18:50:11 UTC |
| 89d2a2e | Zhongjie Wu | 29 January 2013, 00:07:57 UTC | Added non-zoned cluster(multiple bootstrap url) support for consistency check; This uses value hash for comparing values instead of version object | 20 March 2013, 18:50:11 UTC |
| 65662fd | Zhongjie Wu | 19 January 2013, 00:01:55 UTC | fixed a bug where the program does not sweep latest-version-consistent keys correctly | 20 March 2013, 18:50:11 UTC |
| 6422470 | Zhongjie Wu | 17 January 2013, 22:57:24 UTC | wrap up the consistency tool | 20 March 2013, 18:50:11 UTC |
| 76df00d | Zhongjie Wu | 15 January 2013, 21:41:00 UTC | Updated ConsistencyCheck.java for new feature | 20 March 2013, 18:50:10 UTC |
| f54a4f1 | Zhongjie Wu | 10 January 2013, 06:07:31 UTC | Added consistency check tool. Support single partition only | 20 March 2013, 18:50:10 UTC |
| c420a98 | Vinoth Chandar | 08 March 2013, 21:01:08 UTC | Open source release 1.3.0 | 08 March 2013, 21:01:08 UTC |
| 3f0803b | Vinoth Chandar | 08 March 2013, 17:16:24 UTC | Adding ConnectException back as catastrophic exception until the time FD is fixed | 08 March 2013, 17:16:24 UTC |
| ce73194 | Vinoth Chandar | 08 March 2013, 03:26:37 UTC | More Javadoc fixes | 08 March 2013, 03:26:37 UTC |
| 6aa7f4b | Vinoth Chandar | 06 March 2013, 22:02:58 UTC | Adding javadocs + fixing some flaky tests + making ConnectException non catastrophic | 06 March 2013, 22:07:14 UTC |
| 5beaa50 | Vinoth Chandar | 07 February 2013, 21:14:56 UTC | Better BDB default configs | 06 March 2013, 22:07:14 UTC |
| 0530378 | Vinoth Chandar | 21 February 2013, 19:54:46 UTC | Preventing truncate() during failed store opens | 06 March 2013, 22:05:53 UTC |
| d9c63f9 | Vinoth Chandar | 13 February 2013, 21:57:05 UTC | Fix adding a store dynamically, with online retention on | 06 March 2013, 22:05:53 UTC |
| b98b2b2 | Chinmay Soman | 02 March 2013, 23:41:11 UTC | Added a necessary import statement in RepairJob for ReadOnlyStorageConfiguration | 02 March 2013, 23:41:11 UTC |
| 41bf1db | vinoth chandar | 01 March 2013, 18:49:17 UTC | Merge pull request #123 from baepiff/repair_job/mysql delete mysql from blacklist | 01 March 2013, 18:49:17 UTC |
| eb5d559 | vinoth chandar | 01 March 2013, 18:44:48 UTC | Merge pull request #120 from baepiff/restore/mysql Enable mysql to execute restore | 01 March 2013, 18:44:48 UTC |
| 8851652 | Abhinay Nagpal | 01 March 2013, 01:27:01 UTC | Fixed warning message | 01 March 2013, 01:28:08 UTC |
| a9658b9 | Abhinay Nagpal | 28 February 2013, 21:27:31 UTC | Fixed comments | 01 March 2013, 01:28:08 UTC |
| b800b61 | Abhinay Nagpal | 28 February 2013, 21:16:56 UTC | Fixed the build.xml to produce the correct manifest in the voldemort jar | 01 March 2013, 01:28:08 UTC |
| 347650f | Abhinay Nagpal | 28 February 2013, 04:45:45 UTC | Use the resources supplied by the classloader to load the manifest file absolute / relative file sytem paths dont work anymore because when you run this in a servlet container everything is relative to its context hence we use the right method of loading resources we probably need a better key to the implmenetation version - because if there exists another manifest file with the same key in the same servlet container we are doomed as we return its implementation version | 01 March 2013, 01:28:08 UTC |
| a72bcca | Abhinay Nagpal | 27 February 2013, 23:15:04 UTC | Set the map-reduce tokens so that they can be delegated | 28 February 2013, 22:42:47 UTC |
| 75e5bbb | Abhinay Nagpal | 27 February 2013, 22:06:22 UTC | Exposed property names as constants | 28 February 2013, 22:42:46 UTC |
| 3d388a0 | Abhinay Nagpal | 27 February 2013, 19:54:19 UTC | 1) Remove the use of using jars in Hadoop while doing the Build phase 2) Added a prioperty voldemort.distributedcache to indicate apath controlled by azkaban to load jars into distributed cache | 28 February 2013, 22:42:46 UTC |
| abfe406 | Jay Wylie | 27 February 2013, 16:07:17 UTC | Merge pull request #126 from baepiff/CmdUtils/valueof/bug fixed valueof method to handle null value | 27 February 2013, 16:07:17 UTC |
| 8b309a0 | Chinmay Soman | 20 February 2013, 23:08:42 UTC | Updating Version and release_notes for release 1.2.3 | 20 February 2013, 23:08:42 UTC |
| c902517 | Chinmay Soman | 20 February 2013, 21:11:43 UTC | Refactored some try catch blocks. Also returning VoldemortException back from fetch function instead of null. This improves Azkaban visibility. | 20 February 2013, 21:30:21 UTC |
| 0881e1f | Chinmay Soman | 19 February 2013, 22:40:42 UTC | Fixed a small bug in the retry logic for getting the FileSystem | 20 February 2013, 21:30:21 UTC |
