Revision cd51e2c321d793ac714566efb30f8957f9c5e117 authored by Eyal Farago on 12 October 2017, 12:56:04 UTC, committed by Herman van Hovell on 12 October 2017, 12:56:04 UTC
back-port #19181 to branch-2.2. ## What changes were proposed in this pull request? 1. a test reproducing [SPARK-21907](https://issues.apache.org/jira/browse/SPARK-21907) 2. a fix for the root cause of the issue. `org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill` calls `org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.reset` which may trigger another spill, when this happens the `array` member is already de-allocated but still referenced by the code, this causes the nested spill to fail with an NPE in `org.apache.spark.memory.TaskMemoryManager.getPage`. This patch introduces a reproduction in a test case and a fix, the fix simply sets the in-mem sorter's array member to an empty array before actually performing the allocation. This prevents the spilling code from 'touching' the de-allocated array. ## How was this patch tested? introduced a new test case: `org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorterSuite#testOOMDuringSpill`. Author: Eyal Farago <eyal@nrgene.com> Closes #19481 from eyalfa/SPARK-21907__oom_during_spill__BACKPORT-2.2.
1 parent c5889b5
File | Mode | Size |
---|---|---|
docs | ||
lib | ||
pyspark | ||
test_support | ||
.gitignore | -rw-r--r-- | 49 bytes |
MANIFEST.in | -rw-r--r-- | 1.0 KB |
README.md | -rw-r--r-- | 1.8 KB |
pylintrc | -rw-r--r-- | 13.7 KB |
run-tests | -rwxr-xr-x | 899 bytes |
run-tests.py | -rwxr-xr-x | 8.3 KB |
setup.cfg | -rw-r--r-- | 854 bytes |
setup.py | -rw-r--r-- | 9.5 KB |
Computing file changes ...