https://github.com/voldemort/voldemort
Revision 2f72ffb48f1158034da408413c40563cab6b3aec authored by Bhavani Sudha Saktheeswaran on 05 September 2015, 00:44:41 UTC, committed by Bhavani Sudha Saktheeswaran on 25 September 2015, 22:36:11 UTC
- Add Exception communication from server to client Iteration 2: Disk Quota - Random function that generates QuotaException. For testing end to end exception flow. Later this will be replaced by appropriate quota checks Iteration 3: Disk Quota - compute the bytes written to index and data files in each reducer. Add this information to the filename - estimate the total disk size in bytes needed for a <node, store> pair. Done by iterating through all the index and data files and getting theie sizes from the file names, after the job is run. Iteration 4: Disk Quota - In HDFSFetcher, fetch estimated disk size from metadata file - In HDFSFetcher, fetch disk quota limit for <node, store> pair using admin client Minor fix: Add log message Minor fix: Fix exception handling minor fix: Handle Quota Exceptions appropriately minor fix: Fix more exceptions More fixes for Exception logs More exception handling fixes Handling Quota Exception in AdminStoreSwapper Iteration 5: Fix disk estimation logic - Instead of renaing the .data and .index files, add the file size info to corresponding '.checksum' files - Introduce new CheckSumMetadata file that holds the following fields - checksum, data file size, index file size - CheckSumMetadata is similar to ReadOnlyStorageMetadata file in terms of serialization and deserialization minor fix: exclude unnecessary "_" in file names Cleanup debug logs Add quota checks - Determine if quota needs to be checked for the incoming push. For pre existing stores quota is not checked. They will be quota-ed in future. - Check if there is sufficient quota left for a new push. - Filter out stores that push data mistakenly to the Voldemort cluster. This is based on the assumption that for invalid stores, during build phase, a store definition is with 0 quota. All other stores that are onboarded through Nuage will have proper quota set already. More fixes: - Fix Non Quota Exception Handling. Propogates actual error message to client side. - Change getQuota() to getQuotaForNode() in HDFSFetcher. - Add debug log messages - Set zero Quota for stores that are created during Build and Push. We want all new stores to be pre created with appropriate quota via Nuage or via admin client.
1 parent 0af020f
Tip revision: 2f72ffb48f1158034da408413c40563cab6b3aec authored by Bhavani Sudha Saktheeswaran on 05 September 2015, 00:44:41 UTC
Iteration 1: Disk Quota
Iteration 1: Disk Quota
Tip revision: 2f72ffb
File | Mode | Size |
---|---|---|
.settings | ||
bin | ||
clients | ||
config | ||
contrib | ||
docs | ||
example | ||
gradle | ||
private-lib | ||
src | ||
test | ||
voldemort-contrib | ||
.gitignore | -rw-r--r-- | 242 bytes |
CONTRIBUTORS | -rw-r--r-- | 659 bytes |
LICENSE | -rw-r--r-- | 11.1 KB |
NOTES | -rw-r--r-- | 2.5 KB |
NOTICE | -rw-r--r-- | 8.1 KB |
README.md | -rw-r--r-- | 4.4 KB |
build.gradle | -rw-r--r-- | 20.0 KB |
build.xml | -rw-r--r-- | 1.7 KB |
gradle.properties | -rw-r--r-- | 1.1 KB |
gradlew | -rwxr-xr-x | 5.0 KB |
gradlew.bat | -rw-r--r-- | 2.3 KB |
release_notes.txt | -rw-r--r-- | 45.0 KB |
settings.gradle | -rw-r--r-- | 149 bytes |
tomcat-tasks.properties | -rw-r--r-- | 420 bytes |
web.xml | -rw-r--r-- | 1.1 KB |
Computing file changes ...