https://github.com/voldemort/voldemort
Revision df124091d56be65f4386b9df65e463f26c96f3dd authored by Arunachalam Thirupathi on 18 May 2015, 18:14:36 UTC, committed by Arunachalam Thirupathi on 18 May 2015, 18:49:17 UTC
1) Hdfs Fetcher in 1.0.4 uses ByteRangeInputStream. This class does not override the method read(byte[], int , int). So it defaults to this method from InputStream, which reads a character at a time from the input stream. HttpInputStream for this method creates byte arrays for each read. So if you are download 2 TB data, the server will allocate/free 2 TB data before the data is downloaded. This creates too much garbage and new gen gets full in few milliseconds and GC happens. Though GC are fast, this too much GC causes the latency to spike and causes JVM to run out of Memory. 2) http://svn.apache.org/viewvc?view=revision&revision=1330500 fixed this issue on April 2012 rather knowingly/unknowingly. I tried upgrading to Hadoop latest but it brings in ProtoBuf 2.5.0 and Avro 1.7. When I disabled the dependencies it failed at runtime expecting protobuf 2.5.0 . I enabled only protobuf and it has no runtime dependency on Avro 1.7. But I am saving that fix for a later day. The branch is hadoop_Version_Upgrade which uses Hadoop 2.6.0 and ProtoBuf 2.6.1
1 parent e2d845c
Tip revision: df124091d56be65f4386b9df65e463f26c96f3dd authored by Arunachalam Thirupathi on 18 May 2015, 18:14:36 UTC
RO Hdfs fetcher allocates too much memory
RO Hdfs fetcher allocates too much memory
Tip revision: df12409
| File | Mode | Size |
|---|---|---|
| .settings | ||
| bin | ||
| clients | ||
| config | ||
| contrib | ||
| docs | ||
| example | ||
| gradle | ||
| private-lib | ||
| public-lib | ||
| src | ||
| test | ||
| voldemort-contrib | ||
| .gitignore | -rw-r--r-- | 242 bytes |
| CONTRIBUTORS | -rw-r--r-- | 659 bytes |
| LICENSE | -rw-r--r-- | 11.1 KB |
| NOTES | -rw-r--r-- | 2.5 KB |
| NOTICE | -rw-r--r-- | 8.1 KB |
| README.md | -rw-r--r-- | 4.4 KB |
| build.gradle | -rw-r--r-- | 19.8 KB |
| build.xml | -rw-r--r-- | 1.7 KB |
| gradle.properties | -rw-r--r-- | 1.2 KB |
| gradlew | -rwxr-xr-x | 5.0 KB |
| gradlew.bat | -rw-r--r-- | 2.3 KB |
| release_notes.txt | -rw-r--r-- | 42.2 KB |
| settings.gradle | -rw-r--r-- | 149 bytes |
| tomcat-tasks.properties | -rw-r--r-- | 420 bytes |
| web.xml | -rw-r--r-- | 1.1 KB |
Computing file changes ...
