swh:1:snp:764524290233376de64947417267228741ce2485
Revision 2f72ffb48f1158034da408413c40563cab6b3aec authored by Bhavani Sudha Saktheeswaran on 05 September 2015, 00:44:41 UTC, committed by Bhavani Sudha Saktheeswaran on 25 September 2015, 22:36:11 UTC
- Add Exception communication from server to client

Iteration 2: Disk Quota

- Random function that generates  QuotaException. For testing end to end
  exception flow. Later this will be replaced by appropriate quota
  checks

Iteration 3: Disk Quota

- compute the bytes written to index and data files in each reducer. Add
  this information to the filename
- estimate the total disk size in bytes needed for a <node, store> pair.
  Done by iterating through all the index and data files and getting
  theie sizes from the file names, after the job is run.

Iteration 4: Disk Quota

- In HDFSFetcher, fetch estimated disk size from metadata file
- In HDFSFetcher, fetch disk quota limit for <node, store> pair using admin client

Minor fix: Add log message

Minor fix: Fix exception handling

minor fix: Handle Quota Exceptions appropriately

minor fix: Fix more exceptions

More fixes for Exception logs

More exception handling fixes

Handling Quota Exception in AdminStoreSwapper

Iteration 5: Fix disk estimation logic

- Instead of renaing the .data and .index files, add the file size info to corresponding '.checksum' files
- Introduce new CheckSumMetadata file that holds the following fields - checksum, data file size, index file size
- CheckSumMetadata is similar to ReadOnlyStorageMetadata file in terms of serialization and deserialization

minor fix: exclude unnecessary "_" in file names

Cleanup debug logs

Add quota checks

- Determine if quota needs to be checked for the incoming push. For pre
  existing stores quota is not checked. They will be quota-ed in future.
- Check if there is sufficient quota left for a new push.
- Filter out stores that push data mistakenly to the Voldemort cluster.
  This is based on the assumption that for invalid stores, during build
  phase, a store definition is with 0 quota. All other stores that are
  onboarded through Nuage will have proper quota set already.

More fixes:

- Fix Non Quota Exception Handling.  Propogates actual error message to client
  side.
- Change getQuota() to getQuotaForNode() in HDFSFetcher.
- Add debug log messages
- Set zero Quota for stores that are created during Build and Push. We
  want all new stores to be pre created with appropriate quota via Nuage
  or via admin client.
1 parent 0af020f
History
Tip revision: 99127eab3078eed90faa66cc671259a85cb89464 authored by Felix GV on 24 July 2023, 13:49:47 UTC
Update README.md
Tip revision: 99127ea
File Mode Size
.settings
bin
clients
config
contrib
docs
example
gradle
private-lib
src
test
voldemort-contrib
.gitignore -rw-r--r-- 242 bytes
CONTRIBUTORS -rw-r--r-- 659 bytes
LICENSE -rw-r--r-- 11.1 KB
NOTES -rw-r--r-- 2.5 KB
NOTICE -rw-r--r-- 8.1 KB
README.md -rw-r--r-- 4.4 KB
build.gradle -rw-r--r-- 20.0 KB
build.xml -rw-r--r-- 1.7 KB
gradle.properties -rw-r--r-- 1.1 KB
gradlew -rwxr-xr-x 5.0 KB
gradlew.bat -rw-r--r-- 2.3 KB
release_notes.txt -rw-r--r-- 45.0 KB
settings.gradle -rw-r--r-- 149 bytes
tomcat-tasks.properties -rw-r--r-- 420 bytes
web.xml -rw-r--r-- 1.1 KB

README.md

back to top