| 5499428 | Chinmay Soman | 25 June 2013, 23:27:21 UTC | Updating build properties and release notes | 25 June 2013, 23:27:21 UTC |
| 2113816 | Chinmay Soman | 25 June 2013, 23:11:46 UTC | Moved logger.error messages to trace | 25 June 2013, 23:12:13 UTC |
| 2991b38 | Chinmay Soman | 25 June 2013, 00:28:43 UTC | Added the max tolerable fatal errors config param | 25 June 2013, 23:12:13 UTC |
| a05cbc1 | Chinmay Soman | 13 June 2013, 23:19:17 UTC | Cleaning up error messages in ThresholdFailureDetector.java | 25 June 2013, 23:12:13 UTC |
| e25f273 | Chinmay Soman | 13 June 2013, 22:46:00 UTC | Modifying Benchmark.java to accept custom timeout values (routing and socket) | 25 June 2013, 23:12:13 UTC |
| b83fddf | Chinmay Soman | 21 May 2013, 22:02:05 UTC | Added the scripts for integration testing and README for FD perf tests | 25 June 2013, 23:12:13 UTC |
| fe23710 | Chinmay Soman | 17 May 2013, 20:37:16 UTC | Sample changes to make FD more robust | 25 June 2013, 23:12:13 UTC |
| 33ea454 | Zhongjie Wu | 24 June 2013, 19:59:11 UTC | bug fixes in coordinator | 24 June 2013, 23:51:34 UTC |
| 93b9495 | Siddharth Singh | 21 June 2013, 19:41:49 UTC | Bump release version | 21 June 2013, 19:41:49 UTC |
| 4d72ef4 | Siddharth Singh | 21 June 2013, 19:35:16 UTC | Add 1.3.4 and 1.4.0 release notes | 21 June 2013, 19:35:16 UTC |
| 155b934 | Siddharth Singh | 19 June 2013, 00:17:53 UTC | Fix comments and typos | 21 June 2013, 15:48:56 UTC |
| 3d8eb81 | Siddharth Singh | 19 June 2013, 00:15:40 UTC | Add separate dummy stores for 2 and 3 zone cluster | 21 June 2013, 15:48:47 UTC |
| 3944a3c | Siddharth Singh | 19 June 2013, 00:15:11 UTC | Cleanup the arguments and add sanity check | 21 June 2013, 15:48:39 UTC |
| 5d1e8e3 | Siddharth Singh | 18 June 2013, 01:00:07 UTC | Step 2: Add logic to handle zoned clusters | 21 June 2013, 15:48:29 UTC |
| d2452a9 | Siddharth Singh | 15 June 2013, 01:19:57 UTC | Step 1 : Generate cluster script cleanup | 21 June 2013, 15:48:14 UTC |
| 882c01f | Jay J Wylie | 20 June 2013, 22:33:40 UTC | Fix RebalanceUtils.getLatestCluster. RebalanceUtils - changed getLatestCluster to examine contents of cluster.xml on different nodes rather than timestamps since timestamps are incomparable across servers. AbstractZonedRebalanceTest - added a specific rebalance test that goes from current -> shuffle and then shuffle -> current. This confirms that rebalance can be invoked repeated times (if need be). | 20 June 2013, 23:11:05 UTC |
| 0c8f0fa | Abhinay Nagpal | 13 June 2013, 23:22:51 UTC | Fixed systemstore constants and got rid of the superfluous zone rep factor | 20 June 2013, 23:11:05 UTC |
| a758c2a | Abhinay Nagpal | 13 June 2013, 21:49:19 UTC | Tests for system store with a 3 zone cluster | 20 June 2013, 23:11:05 UTC |
| 3512cff | Vinoth Chandar | 14 June 2013, 00:35:07 UTC | Modifying RoutedStoreTest to use Consistent Strategy | 20 June 2013, 23:11:05 UTC |
| 9a90156 | Abhinay Nagpal | 13 June 2013, 17:58:27 UTC | Addressed code review comments added more assertions check for getZoneNary and getNodeIdForZoneNary in zone 1 and 2 Added a test for method 'storeRoutingPlan.zoneNAryExists() | 20 June 2013, 23:09:23 UTC |
| 4f135d5 | Abhinay Nagpal | 12 June 2013, 20:38:24 UTC | Test for 3 zones | 20 June 2013, 23:09:23 UTC |
| d2848a1 | Siddharth Singh | 12 June 2013, 21:51:14 UTC | Add dummy-stores.xml + Update the path in script | 20 June 2013, 23:09:23 UTC |
| 803d7dd | Siddharth Singh | 12 June 2013, 21:45:15 UTC | Move rebalance scripts to bin directory Moving the script from top level to the bin directory | 20 June 2013, 23:09:23 UTC |
| 1807fae | Siddharth Singh | 12 June 2013, 21:23:49 UTC | Copy final cluster xml file to output dir When the final cluster.xml is generated it is placed in outputdir/step3 folder. Copy it to outputdir as well. | 20 June 2013, 23:09:23 UTC |
| f9936c1 | Siddharth Singh | 12 June 2013, 03:53:08 UTC | Add rebalance new cluster script | 20 June 2013, 23:09:23 UTC |
| 9d9b47e | Siddharth Singh | 11 June 2013, 22:30:56 UTC | Address code review comments | 20 June 2013, 23:09:23 UTC |
| 407d4e4 | Siddharth Singh | 10 June 2013, 23:47:07 UTC | Edit script to use the new option | 20 June 2013, 23:09:22 UTC |
| ba371fc | Siddharth Singh | 10 June 2013, 23:42:46 UTC | Add an option for choosing zone id for shuffle Add a new option that lets the user choose if they want the shuffle to operate only on one of the zone ids. | 20 June 2013, 23:09:22 UTC |
| c1fb3aa | Siddharth Singh | 10 June 2013, 20:03:21 UTC | Add zone expansion script | 20 June 2013, 23:09:22 UTC |
| aa96bbf | Siddharth Singh | 07 June 2013, 22:18:05 UTC | Write plan to a file inside the output dir | 20 June 2013, 23:09:22 UTC |
| 9ee4ebe | Siddharth Singh | 03 June 2013, 23:27:10 UTC | Print the utility value | 20 June 2013, 23:09:22 UTC |
| f55aace | Jay J Wylie | 07 June 2013, 21:23:36 UTC | Fix final-cluster to be required arg of RebalancePlanCLI | 20 June 2013, 23:09:22 UTC |
| 77f3b69 | Jay J Wylie | 06 June 2013, 20:54:24 UTC | Correctly print RebalanceTask at creation time. | 20 June 2013, 23:09:22 UTC |
| ec9cd1e | Jay J Wylie | 04 June 2013, 16:58:29 UTC | Minor fix to KeyVersionFetcherCLI | 20 June 2013, 23:09:22 UTC |
| 0ef8faf | Jay J Wylie | 04 June 2013, 15:56:24 UTC | Make KeyVersionFetcherCLI have generous timeouts | 20 June 2013, 23:09:22 UTC |
| 3122693 | Siddharth Singh | 30 May 2013, 22:41:44 UTC | Add comments to the tools | 20 June 2013, 23:09:22 UTC |
| e40f793 | Jay J Wylie | 29 May 2013, 22:52:46 UTC | Fix to print one timestamp per-line from KeyVersionFetcherCLI | 20 June 2013, 23:09:22 UTC |
| ffc9b68 | Jay J Wylie | 29 May 2013, 22:35:50 UTC | Make KeyVersionFetcherCLI ZoneNAry aware | 20 June 2013, 23:09:22 UTC |
| 412088d | Jay J Wylie | 29 May 2013, 15:23:26 UTC | Address minor review feedback Mostly, made variable names more clear AdminClient - cleaned up comments & TODO RebalanceController - fix subtle bug in when proxyPause invoked RepartitoinerTest - changed zone expansion test to cover two use cases: when invoked from RepartitoinerCLI with a current cluster versus with an interim cluster. | 20 June 2013, 23:09:22 UTC |
| f816900 | Jay J Wylie | 24 May 2013, 22:55:45 UTC | Fixed typo. | 20 June 2013, 23:09:22 UTC |
| e876dd1 | Jay J Wylie | 24 May 2013, 22:21:43 UTC | Change variable name in RouteToAllStrategy. | 20 June 2013, 23:09:22 UTC |
| 89969de | Jay J Wylie | 24 May 2013, 22:05:19 UTC | Fixed cluster nomenclature: current, interim, final Touched a lot of code to normalize variable names, method names, and comments about clusters: - current : current cluster in prod - interim : current cluster + new nodes/zones w empty partitions - final : final cluster w shuffled partitions and/or populated new nodes/zones Added TODO about voldemort server excepting on invalid cluster xml Dropped TODOs that were unnecessary | 20 June 2013, 23:09:22 UTC |
| 4c276b8 | Jay J Wylie | 24 May 2013, 19:48:30 UTC | Added "progress bar" for rebalance batch plan Added RebalanceBatchPlanProgressBar - progress tracking object for each rebalancing batch - integrated with RebalanceController, AsyncRebalanceTask, and (Donor|Stealer)BasedRebalanceTask Did other general clean up of logging during rebalance to make it appropriately verbose (more verbose in some places, less verbose in others...) | 20 June 2013, 23:09:22 UTC |
| c9b370e | Jay J Wylie | 23 May 2013, 23:04:44 UTC | Add 'proxy pause' between cluster change & rebalance The proxy pause is a window during which clients can pick up the new cluster metadata, and servers can establish proxy bridges, before servers start moving data around for rebalancing. This allows us to observe the cost of proxying separate from rebalance and allows clients to pick up new metadata before any data is moved. Also added some more TODOs about documenting the various rebalanceState methods... | 20 June 2013, 23:09:21 UTC |
| d7f81d7 | Jay J Wylie | 23 May 2013, 18:41:26 UTC | Fix arg checking typo in RebalanceControllerCLI. | 20 June 2013, 23:09:21 UTC |
| 2bf1543 | Jay J Wylie | 23 May 2013, 16:24:26 UTC | Fix potential overflow in AdminClient.waitForCompletion | 20 June 2013, 23:09:21 UTC |
| 81f0a16 | Jay J Wylie | 23 May 2013, 15:58:22 UTC | Fix some bugs I introduced and added more TODOs Fixed overflow introduced in AdminClient.waitForCompletion by passing in Long.MAX_VALUE for duration. Verify cluster store definition in StoreRoutingPlan. This requires working around existing problems with how system stores are handled (the store definition is hard-coded for two zones). Left some TODOs about testing and fixing all of this. Added TODOs about currentCluster vs interimCluster. Need to tweak interface to RebalanceController and RebalancePlan to be consistent with recommended usage (i.e., deploying interimCluster before starting rebalance). Minor tweaks to tests based on above changes. | 20 June 2013, 23:09:21 UTC |
| c794504 | Jay J Wylie | 22 May 2013, 20:43:53 UTC | Minor TODO and copyright cleanup. | 20 June 2013, 23:09:21 UTC |
| 6a283ae | Jay J Wylie | 22 May 2013, 20:22:01 UTC | Fix invalid metadata rate calculation Fixed the calculation of invalid metadata rate in RebalanceUtils. Also added more/different pretty printing of a rebalance plan. | 20 June 2013, 23:09:21 UTC |
| 45903e0 | Jay J Wylie | 22 May 2013, 17:33:05 UTC | Address review of initial RebalanceController work - Removed 'timeout' option for rebalancing since always setting timeouts to infintie (Long.MAX_VALUE) is the recommended practice. - Cleaned up some TODOs and added some javadoc in response to code review from Lei | 20 June 2013, 23:09:21 UTC |
| 92faebd | Jay J Wylie | 22 May 2013, 16:28:06 UTC | Added junit-rebalance and junit-long-rebalance build targets | 20 June 2013, 23:09:21 UTC |
| 10901f7 | Jay J Wylie | 22 May 2013, 16:08:02 UTC | Completed unit tests for zone expansion Added zone expansion unit tests to AbstractZonedRebalanceTest. This required tweaking the helper methods in RebalanceUtils and ClusterTestUtils. | 20 June 2013, 23:09:21 UTC |
| d344725 | Jay J Wylie | 21 May 2013, 23:25:29 UTC | Added more zone expansion test for RebalanceBatchPlan | 20 June 2013, 23:09:21 UTC |
| d4c314f | Jay J Wylie | 21 May 2013, 22:56:13 UTC | RebalanceController uses atomic cluster/store RebalancePlan RebalanceBatchPlan - split storeDefs into currentStoreDefs & finalStoreDefs - clean up some variable naming *RoutingStrategy - marked all override methods with @Override RouteToAllStrategy - Implemented getPartitionList/getReplicatingPartitionList/getMasterPartition - Use a single imaginary parition id ("0") to implement these methods - This allows StoreRoutingPlan to be constructed for this strategy AbstractZonedRebalanceTest - fix proxy put test | 20 June 2013, 23:09:21 UTC |
| 8bbf529 | Jay J Wylie | 21 May 2013, 17:17:43 UTC | Fixes after rebase/dropped propagateCluster method After the rebase with the atomic update of cluster/stores branch, had some clean up to do. Got code running and unit tests passing again. Note that VAdminProto.java had to be regenerated after the merge. Noted that RebalanceUtils.propagate cluster was unnecessary (and potentially dangerous) and so dropped it from the code. | 20 June 2013, 23:09:21 UTC |
| c228e5f | Jay J Wylie | 17 May 2013, 22:26:19 UTC | Added unit tests for RebalancePlan, RebalanceBatchPlan, and AbstractZonedRebalanceTest. Added following rebalance test case coverage: - RebalancePlanTest / RebalanceBatchPlanTest / AbstractZonedRebalanceTest - no-op / shuffle / cluster expansion / zone expansion - 2-zone / 3-zone Also did some clean up of rebalance test utils and rebalance tests in general. | 20 June 2013, 23:09:05 UTC |
| 91c0b18 | Jay J Wylie | 16 May 2013, 15:21:09 UTC | Addressed review feedback on new RebalanceController. | 20 June 2013, 23:09:05 UTC |
| ad4b726 | Jay J Wylie | 15 May 2013, 22:02:03 UTC | Fix to store routing plan to correctly track zone-primaries per-node. | 20 June 2013, 23:09:05 UTC |
| 8bc5e17 | Jay J Wylie | 15 May 2013, 16:53:55 UTC | Minor changes to address second round of review feedback on rebalance plan. | 20 June 2013, 23:09:05 UTC |
| 5244216 | Jay J Wylie | 14 May 2013, 23:08:06 UTC | Address review comments for RebalancePlan RebalanceBatchPlan - factored logic to decide which donor to steal from out of constructBatchPlan. - huge header comment about other policies to consider RebalanceController - dropped comment about historic (arbitrary) sleep command RebalancePlan - added helper hacky method to "clone" cluster RebalanceCluster - TODO to add a .clone() method StoreRoutingPlan - cleaned up javadoc, TODOs, and method names DonorBasedRebalancePusherSlave - set sleep to 30 seconds. Made comments and logger messages consistent with code. StorageEngine - cleaned up javadoc for isPartitionAware() RebalanceUtils - fix validation method to use safe(r) comparison. RebalanceBatchPlanTest - fixed tests to (mostly) only test plan invariants. Prior test code focussed on exact plan details and so was hard-coded to the implementation, rather than the interface. | 20 June 2013, 23:09:05 UTC |
| 4fdeaf2 | Jay J Wylie | 13 May 2013, 18:31:16 UTC | Remove no-longer needed test from RebalancePartitionsInfoTest; annotated another test as mostly failing.. | 20 June 2013, 23:09:05 UTC |
| 8a9a791 | Jay J Wylie | 13 May 2013, 18:09:33 UTC | Clean up Rebalance(Plan|Controller)CLI RebalancePlan(CLI) - set batch size default to INTEGER.MAX_VALUE RebalanceControllerCLI - Cleaned up handling of all optional arguments | 20 June 2013, 23:09:05 UTC |
| 039b7fc | Jay J Wylie | 13 May 2013, 17:43:34 UTC | Renamed Rebalance(Long)Test to NonZonedRebalanceLongTest.java. Added ZonedRebalanceLongTest.java. | 20 June 2013, 23:09:05 UTC |
| a2b950c | Jay J Wylie | 13 May 2013, 17:24:58 UTC | Cleaned up existing TODOs and added some javadoc comments. | 20 June 2013, 23:09:04 UTC |
| 8121c78 | Jay J Wylie | 13 May 2013, 17:02:02 UTC | Renamed RebalanceClusterPlan(Test).java to RebalanceBatchPlan(Test).java. Cleaned up TODOs and comments and variable names in RebalanceBatchPlan. | 20 June 2013, 23:09:04 UTC |
| 1219576 | Jay J Wylie | 13 May 2013, 16:33:12 UTC | Fixed RebalanceClusterPlanTest to be junit 4 and to work as well as possible at this time. Test needs to be re-written to test new planning rather than old planning. | 20 June 2013, 23:09:04 UTC |
| 083a3e8 | Jay J Wylie | 13 May 2013, 16:13:30 UTC | Fix on-wire protocol for RebalancePartitionInfoMap Completed deprecation of member 'attempt' of RebalancePArtitionInfoMap by making it optional and prefixing name with OBSOLETE. This leads to big changes to auto generated file VAdminProto.java. Some other minor tweaks to RebalancePArtitionsInfo and comments as well. | 20 June 2013, 23:09:04 UTC |
| 2b69609 | Jay J Wylie | 10 May 2013, 19:38:10 UTC | Dropping RebalanceNodePlan, the class that sorted lists of RebalancePartitionsInfo by node id (either stealer or donor node), and then flattened the sorted list out and returned it again. AKA the identity operation on a list of RebalancePartitionsInfo. | 20 June 2013, 23:08:23 UTC |
| 42d3868 | Jay J Wylie | 10 May 2013, 19:34:20 UTC | Dropped Rebalance*BatchPlan.java, the classes I thought we needed to encapsulate an executable plan. | 20 June 2013, 23:08:23 UTC |
| d6cd930 | Jay J Wylie | 10 May 2013, 19:31:33 UTC | Interim commit before dropping ExecutableBatch objects RebalanceClusterPlan - removed TODOs about trying to "intelligently" order rebalance tasks RebalanceController - switched to just using REbalancePlan until tasks are actually constructed that are either stealer- or donor-based. RebalancePartitionsInfo - static method for pretty printing a list thereof *Test - clean up to deal with other type & interface changes... | 20 June 2013, 23:08:23 UTC |
| 91be6db | Jay J Wylie | 10 May 2013, 17:59:57 UTC | Interim commit: OrderedClusterTransition has been removed from the code base. | 20 June 2013, 23:08:23 UTC |
| d9e2623 | Jay J Wylie | 10 May 2013, 17:30:19 UTC | interim commit. Some work on taking logic of OrderedClusterTransition and moving it into ExecutableBatches. Logic is broken though, and, more importantly, unnecessary. Given proxy-gets server from local zone and there can only be nominal perf gain from ordering rebalancing tasks at node level, OrderedClusterTransition needs to be removed from teh code, not incorporated into new planning/execution code. | 20 June 2013, 23:08:23 UTC |
| c1c367e | Jay J Wylie | 10 May 2013, 16:09:16 UTC | Continued refactoring and clearing out deprecated code. OrderedClusterTransition - removed unnecessary members: id currentCluster targetCluster rebalanceclusterPlan storeDefs RebalanceController - renamed some member variables for sake of clarity - switched from orderedClusterTransition.getId to batchCount to uniquely ID batches of work... RebalanceNodePlan - added TODOs - removed unnecessary members: attempt storeToReplicaToDeletePartitionList Patched up all tests to not have 'compile time' errors. Have not run tests though... | 20 June 2013, 23:08:23 UTC |
| 70f8259 | Jay J Wylie | 09 May 2013, 22:54:06 UTC | Switched to new RebalanceController based on RebalancePlan and dropped tons of now deprecated code. Notes: - delete of partition-stores during rebalancing has been removed and is no longer an option. - the old RebalanceCLI now only offers the entropy tool features. cannot be dropped until SREs are OK with KeySampler/KeyVersionFetcher tool chain for verifying rebalances. - Dropped RebalanceClientConfig. Configuration parameters are now explicit in pertinent constructors. - Tests are failing or disabled: - donor-based rebalancing is disabled in test suites. Does not currently work with new rebalancing. - RebalanceClusterPlanTest tests mostly fail at this time. - RebalanceTest.serverSideRouting[1] may timeout - ZonedRebalanceTest.testProxyGetDuringRebalancing[1] may timeout OrderedClusterTransition - many small changes to work with all other changes - this class is to-be-deprecated regardless RebalanceClusterPlan - removed tons of deprecated code that permitted old controller to continue to work RebalanceController - Now uses RebalancePlan - No longer offers delete-after-rebalancing feature - Dropped tons of deprecated code... RebalanceTypedBatchPlan RebalanceDonorBasedBatchPlan RebalanceStealerBasedBatchPlan - no longer extends RebalanceClusterPlan - the classes that make a generic plan executable as either stealer- or donor-based - still need to be renamed so purpose of these key classes is more clear RebalancePlanCLI RebalanceControllerCLI - moderate clean up RebalanceUtils - dropped unnecessary helper mehods *Test - Signficant clean up: - use Rebalance*BatchPlan and RebalancePlan - don't check keys deleted since no longer supported | 20 June 2013, 23:08:23 UTC |
| 79ca1e0 | Jay J Wylie | 09 May 2013, 20:04:44 UTC | Unit tests work with new rebalance controller. RebalanceController - helper method to construct RebalancePlan based on cluster read from servers - divide by zero fix - many TODOs and Deprecate annotations... .*RebalanceTask - drop RebalanceclientConfig from construction and direclty pass in the config parameters needed. *Rebalance*Test - do not test donor-based rebalancing. (TODOs to add it back.) - do not test that keys are actually deleted when delete is true in rebalancing. - use the new controller/plan code path in all rebalancing unit tests. | 20 June 2013, 23:08:23 UTC |
| 464df36 | Jay J Wylie | 09 May 2013, 15:39:43 UTC | Initial commit of new rebalance controller. This is an interim commit. The new RebalanceControllerCLI has been added. Portions of the RebalanceController have been updated to use the RebalancePlan. All the tests that should pass, still pass for the old RebalanceController. About to switch tests to use new RebalanceController that works with a RebalancePlan. | 20 June 2013, 23:08:23 UTC |
| 0515ae4 | Jay J Wylie | 08 May 2013, 21:10:52 UTC | Minor changes inr esponse to review feedback and documenting a set of tests that are known to fail at this time. | 20 June 2013, 23:08:23 UTC |
| 8d5d076 | Jay J Wylie | 08 May 2013, 16:24:52 UTC | Refactoring: More class renaming and file reorganization ClusterInstance has been dropped. ClusterInstanceTest is now PartitionBalanceTest and is in tools. ClusterTestUtils is a new util class. All of the getFOOClusterBar and getFOOStoreDef(s)BAR methods have been moved into this test helper. | 20 June 2013, 23:08:23 UTC |
| 54e48c5 | Jay J Wylie | 08 May 2013, 16:04:34 UTC | Refactoring: Renamed some classes and reorganized their location RepartitionCLI -> RepartitionerCLI RepartitionUtils -> Repartitioner PartitionBalance & Repartitioner : voldemort/utils -> voldemort/tools Test classes & files updated to correspond with src/java. | 20 June 2013, 23:08:23 UTC |
| cb5d292 | Jay J Wylie | 08 May 2013, 15:38:43 UTC | Additional minor refactoring of utils and utils tests. | 20 June 2013, 23:08:23 UTC |
| 57686e4 | Jay J Wylie | 08 May 2013, 15:30:16 UTC | Refactoring: cleaned up util methods and moved them to appropriate util classes RebalanceUtils - all dump cluster methods are now here with more consistent interfaces - analyzeInvalidMetadataRate is here (for lack of better place to put it) RepartitionUtils - moved all helper/util methods out of this class Utils - added removeItemsToSplitListEvenly, distributeEvenlyIntoList, and distributeEvenlyIntoMap Moved test methods around to match new locations of util methods. | 20 June 2013, 23:08:23 UTC |
| fdc0eb3 | Jay J Wylie | 07 May 2013, 23:29:09 UTC | Addressing reviews of balance analysis & repartitionerCLI RebalancePlan - print out invalid metadata rate analysis Cluster - added members that allow reverse lookup of partitionID to either ZoneId or NodeId StoreRoutingPlan - Added reverse lookup of nodeID to Zone Primaries hosted. RepartitionCLI - moved constants into RepartitionUtils - Added validation of clusters & storedefs passed in. ClusterUtils - use lookup from Cluster object - changed all contig partition ID methods to call the one method that does all the complicated work. PartitionBalance - Added constants for use in the utility function - Broke the core, complicated method into small, reasonably-sized pieces. - Added verbose print out of ZoneReplicaTypes (to complement old "replica type" print out) RepartitionUtils - Document the recommend default constants for repartitioning - Dump invalid metadata rate - Clean up comments and java doc - dumpInvalidMetadataRate analyzes and pretty-prints how many pseudo-masters per-zone per-unique-store-def lead to invalid metadata exception. | 20 June 2013, 23:08:23 UTC |
| 80dc13a | Jay J Wylie | 03 May 2013, 23:09:04 UTC | copyright updates. | 20 June 2013, 23:08:22 UTC |
| 4f0087f | Jay J Wylie | 03 May 2013, 23:05:18 UTC | Added unit tests for RebalancePlan and for cluster xform & verification methods in RebalanceUtils RebalancePlanTest - Test the core use cases for rebalance planning: - no-op, - rebalance (zone/node topology same, but partition id layout changed), - cluster expansion (zone topology same, but new nodes added & partitions moved to them), - zone expansion (zone topoloigy changes with partitions moved to new zone). RebalanceController - rename method RebalancePlan - added getters for aggregate plan statistics ServerTestUtils - Changed getLocalZonedCluster to accept array of node IDs. Necessary to generalize other test methods. ClusterInstanceTest - pass storage type into helper methods that construct store defs - chnagned all get???Cluster.*() methods to use new getLocalZonedCluster interface. Also added more such helper methods. RebalanceUtilsTest - Test the cluster transformation & verification methods RepartitionUtilsTest - simple clean up | 20 June 2013, 23:08:22 UTC |
| 622ee42 | Jay J Wylie | 03 May 2013, 17:43:14 UTC | Fixed broken junit tests. Documented and commented out other parts of unit tests that rely on donor-based rebalancing or that rely on partition-stores being deleted during rebalance. Both features are currently (necessarily) broken. AdminClient - Fix latent bug that treated node ids as dense/contiguous RebalanceclusterPlan - minor refactor RebalanceController - handle the case of an "empty" plan differently. Because unneccessary moves are optimized out at plan time, some of the small tests now yield plans with no data movement. I.e., primary partition IDs move among nodes, but data does not because all nodes already host all data. This exposed a corner case in which rebalance was successful, but servers never updated their cluster xml with new partitoin ID mapping. - Removed random/unnecessary 10s sleep from control path. Annotated with a TODO for discussion to confirm this is right thing to do. DonorBasedRebalanceAsyncOperation - minor code clarification DonorBasedRebalancePusherSlave - Fixed latent bug (wrong value passed to sleep). Added TODO wondering why this sleep is here in the first place. StorageEngine - How are isPartitoinScanSupported() and isPartitionAware() related? PartitionPrefixedBdbStorageEngine - add isPartitionAware that returns true. Is this correct!? AdminRebalanceTest - Commented out asserts that test deletion during rebalancing ZonedRebalanceTest - Removed donor-based rebalancing tests from this test suite. | 20 June 2013, 23:08:22 UTC |
| 03453dc | Jay J Wylie | 02 May 2013, 15:07:47 UTC | RebalancePlan de-insanification This is a messy commit: 1) There are a ton of TODOs, a ton of code marked to be deprecated, and a lot of gymnastics to continue using data structures that ought to be simplified or removed. All of this is necessary until the RebalanceController is switched to use the RebalancePlan, and until the abortable rebalance work and this work are merged to master. Cleaning up shared data structures is too high risk while on separate branches. 2) Some of the RebalanceTest tests are currently failing. Going to debug and fix in next commit. Commit notes: RebalanceClusterPlan - renamed cluster members to be target & final rather than current & target - determines batch plan at construction time - added batchPlan() method that - is more clear than prior planning method - optimized to do n-ary to n-ary migration within a cluster so that only new zones require cross zone moves RebalancePlan - minor change for method name change StoreRoutingPlan - added internal member & method to track all nary-partition-stores per node - get zone replica type methods that take partition id as input RebalanceUtils - added / refactored suite of methods that validate pairs of clusters (current&target, current&final, target&final) There are tests missing at this point in development: - simple & basic tests of RebalancePlan - test StoreRoutingPlan.getNaryPartitionIds (and possibly other methods too) - tests for RebalanceUtils.validate.* methods | 20 June 2013, 23:08:22 UTC |
| 5e72e52 | Jay J Wylie | 01 May 2013, 16:26:56 UTC | RebalancePlanCLI working for 2-to-3 zone expansion Stealers steal from first donor with desired partition-store. This currently biases all work towards donors with lowest node IDs. This is truly imbalanced plan! RebalanceClusterPlan - Decorated with more TODOs... - Changed stealer to pick first donor with desired partition-store - Added commented-out code section that outlines next step for better planning RebalancePartitionsInfo - Decorated with more TODOs RebalancePlan - Much clean up refactoring. Construction now generates plan. - Methods added to get the plan (getPlan) and to print the plan (toString) RebalanceUtils - Helper method that derives targetCluster from initialCluster and finalCluster. | 20 June 2013, 23:08:22 UTC |
| 857aa57 | Jay J Wylie | 30 April 2013, 18:13:25 UTC | I mangled the merge after a rebase. This commit manually fixes the errors I introduced. Mea culpa. | 20 June 2013, 23:08:22 UTC |
| aee0639 | Jay J Wylie | 30 April 2013, 16:29:33 UTC | Initial work on RebalancePlanCLI. Extracted the planning logic out of RebalanceController. This will reduce cut-and-paste coding, clarify which arguments are needed to tune planning (versus execution of a plan), and isolate rebalance planning logic from execution logic. There are a ton of TODOs added in this commit. As the refactoring continues, and the controller is changed to use the new RebalancePlan, these TODOs will be addressed. This commit has a working RebalancePlanCLI that handles the use cases rebalance planning has historically handled (rebalance in place and cluster expansion). This commit does not address zone expanison. The output of plan statistics is much richer and includes detailed statistics on cross zone moves (from which zone to which zone) and nodes (how much in and how much out). All these statistics are at the partition-store level which should be more informative than prior statistics. Finally, these statistics include an estimation of storage overhead per-node which can be used to assess how much free disk space is required to execute a plan safely. OrderedClusterTransition - TODOs to deprecate this. This logic belongs in the plan, not the execution. RebalanceClusterPlan - TODOs for refactoring - Has logic for doing things "the old way" and the "new way". The new way has simpler data structures nad better encapsulation. - Many more getters for detailed plan stats RebalanceNodePlan - Minor TODOs and some renaming of varaibles RebalancePartitionsInfo - A ton of refactoring TODOs - Fixed all move counting getters to count partition-stores Node - getStateString method to pretty print ID, host name and ports. RepartitionCLI - more argument handling PartitionBalance - getters for partition-store accounting of current cluster+store defs RebalanceUtils - Some methods to deprecate - validation methods for cluster arguments RebalancePlan - New class to encapsulate all rebalancing planning RebalanceTypedBatchPlan RebalanceDonorBasedBatchPlan RebalanceStealerBasedBatchPlan - Sub-classes of RebalancePlan that are needed for converting static plan into executable plan RebalancePlanCLI - CLI for generating a plan MoveMap & MoveMapTest - 2-d map counter used in partitoin-store accounting | 20 June 2013, 23:08:22 UTC |
| 0e17e99 | Jay J Wylie | 25 April 2013, 17:26:46 UTC | copyright fix | 20 June 2013, 23:08:22 UTC |
| 722d9af | Jay J Wylie | 25 April 2013, 17:08:57 UTC | Added tests of repartitioning algorithms. RepartitionCLI - added arg for greedy zone IDs; not bothering exposing it via CLI though ClusterUtils.java - split method that identifies contiguous partition ID runs into two, better tested and now correct, methods PartitionBalance - fix div by zero error RepartitionUtils - Documented the repartition method - Reused the internal balancePrimaryPartitions method in the method that breaks apart contiguous partitoin runs - dropped unused methods - switched greedy algorithms to do either zone-by-zone or cluster as a whole. unless we find cluster-as-a-whole leads to too much data movement, is the behavior that we want. ClusterInstanceTest - made all the get*Cluster and get*StoreDefs methods static and added some more. These are generally useful for tests beyond ClusterInstance. RepartitionUtilsTest - Actually test all of hte repartitioning algorithms! | 20 June 2013, 23:08:22 UTC |
| ce5a73e | Jay J Wylie | 24 April 2013, 20:27:55 UTC | Refactoring. Renamed RebalanceClusterUtils->RepartitionUtils and moved some tests into new file RepartitionUtilsTest. | 20 June 2013, 23:08:22 UTC |
| 92c1139 | Jay J Wylie | 24 April 2013, 17:59:23 UTC | Initial version of repartitioning tool 'RepartitionCLI'. RebalanceCLI - removed all "generate" options and code from this tool (now tools/RepartitionCLI) - removed "analysis" option from this tool (now tools/PartitionAnalysisCLI) - Fixed errors in help/usage docs RepartitionCLI - Removed unnecessary options that restricted partition movement to be 'within a zone'. This decouples the repartition tool from the planning tool. This also assumes that optimizations can be done at plan time to keep all movement within a zone (when possible). - cleaned up option names to be more concise - Added options for target cluster/stores to allow zone expansion to be fully specified PartitionBalance - New class extracted from ClusterInstance - Made all partition analysis zone-aware. This prevents zones with fewer nodes, or partitions, from skewing any analysis. I.e., each zone's balance analysis is now correctly normalized to be comparable with other zones' balance analyses. - added inner ZoneBalanceStats class that makes stats tracking clearer - added getUtility methods to hide the exact utiliyt method from the interfaces - Changed utility methods to sum over all zones and to combine zone primary analysis with nary analysis. RebalanceClusterUtils.java - changed/extended balancePrimaryPartitions to handle zone expansion - Stripped out a bunch of unnecessary options/code. I.e., options/code that tried to minimize cross zone moves during repartitioning (rather than during planning!) - Added some helper methods to clean up repeated code. RebalanceUtilsTest - test new helper methods | 20 June 2013, 23:08:22 UTC |
| 4f1c00b | Jay J Wylie | 23 April 2013, 15:41:52 UTC | Added RepartitionCLI. It is the subset of the RebalanceCLI that does repartitioning. | 20 June 2013, 23:08:22 UTC |
| 34e7a4b | Jay J Wylie | 22 April 2013, 18:39:14 UTC | Partition balance analysis tool for more than two zones. - Started a directory for all of our tools (src/java/voldemort/tools) - PartitionBalance : - broke this util out of ClusterInstance to make it more self-contained - this changed the declaration of a lot of the methods that invoked analyzeBalance and analyzeBalanceVerbose - ClusterInstanceTest : Added a bunch of tests to confirm that partition balance can be constructed (or not) for combinations of 2 & 3 zones | 20 June 2013, 23:08:22 UTC |
| ded4506 | Chinmay Soman | 20 June 2013, 22:06:32 UTC | Removed BannagePeriodFailureDetector parameter from RoutedStoreTest and some beautification changes | 20 June 2013, 22:07:23 UTC |
| 5f796e8 | Chinmay Soman | 19 June 2013, 22:56:56 UTC | Added 3 zone tests to RoutedStoreTest.java, fixed a small bug in GetAllConfigureNodes | 20 June 2013, 22:07:23 UTC |
| c809fba | Siddharth Singh | 20 June 2013, 01:00:19 UTC | Remove unneeded check adminClient doeesn't change in the try block so no check needed in the finally block. | 20 June 2013, 18:05:45 UTC |
| 7daac20 | Siddharth Singh | 20 June 2013, 00:18:08 UTC | Add more locations to gitignore | 20 June 2013, 00:18:28 UTC |
| b8a8a4f | Abhinay Nagpal | 18 June 2013, 21:25:19 UTC | Added comments | 18 June 2013, 21:25:19 UTC |
