Revision a7061474f905fe60a9581dbf4e14431d00b130a2 authored by Felix GV on 12 December 2015, 01:26:55 UTC, committed by Felix GV on 15 December 2015, 02:45:06 UTC
Summary: This new mode provides the capability of pushing to multiple clusters with different number of nodes and different partition assignments. Compatibility: Although this new mode only works if both the BnP job and the Voldemort servers are upgraded, the change can be rolled out gradually without breaking anything. There is a negotiation phase at the beginning of the BnP job which determines if all servers of all clusters are capable and willing (i.e.: configured) of using the new mode. If not all servers are upgraded and enabled, then the BnP job falls back to its old behavior. Likewise, if a server gets a fetch request from a non-upgraded BnP job, it will work just like before. By default, servers answer the negotiation by saying they support the new mode. The old behavior can be forced with the following server-side configuration: readonly.build.primary.replicas.only=false Running in this new mode has several implications: 1. When running in the new mode, store files are stored in the BnP output directory under nested partition directories, rather than in nested node directories. 2. The MR job uses half as many reducers and half as much shuffle bandwidth compared to before. 3. The meta checksum is now done per partition, rather than per node. 4. Instead of having one .metadata file per partition, there is now only a single full-store.metadata file at the root of the output directory. 5. The server-side HdfsFetcher code inspects the metadata file and determines if it should operate in 'build.primary.replicas.only' mode or not. If yes, then the server determines which partitions it needs to fetch on its own, rather than relying on what the BnP job placed in a node-specific directory. 6. The replica type number contained in Read-Only V2 file names is now useless, but we are keeping it in there just to avoid unnecessary changes. 7. When initializing a Read-Only V2 store directory, the server now looks for files named with the incorrect replica type, and if it finds any, it renames them to the replica type expected by this server. Other changes: 1. Added socket port to Node's toString functions. Also made the output of the Node's toString(), briefToString() and getStateString() functions more consistent. 2. Introduced new Protobuf message for the GetConfig admin request. This new message is intended to be a generic way to retrieve any of server config. 3. Refactored VoldemortConfig to provide access to any config by its string key. Also cleaned up a lot of hard-coded strings, which are constants now. 4. Various minor refactorings in BnP code.
1 parent 69fcd3f
voldemort-prod-server.sh
#!/bin/bash
#
# Copyright 2008-2013 LinkedIn, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
#
# Spin up the voldemort server(BDB-JE) with a production worthy JVM config
#
export VOLD_OPTS=" -server -Xms32684m -Xmx32684m -XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 -XX:+AlwaysPreTouch -XX:+UseCompressedOops -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime "
echo "WARNING: This setup spins up a kickass server for production/perftesting. Make SURE that you are running on a server class machine"
echo "Tip: Make sure you either disable swap or give your user permission to mlock the heap"
bin/voldemort-server.sh $@
Computing file changes ...