https://github.com/apache/spark
Revision ed842ba8e82e845efc38bd115909ce54faef318a authored by ulysses-you on 02 August 2022, 09:05:48 UTC, committed by Hyukjin Kwon on 02 August 2022, 09:06:38 UTC
### What changes were proposed in this pull request?

Explicitly clear final partition buffer if can not find next in `WindowExec`. The same fix in `WindowInPandasExec`

### Why are the changes needed?

We do a repartition after a window, then we need do a local sort after window due to RoundRobinPartitioning shuffle.

The error stack:
```java
ExternalAppendOnlyUnsafeRowArray INFO - Reached spill threshold of 4096 rows, switching to org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter

org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 65536 bytes of memory, got 0
	at org.apache.spark.memory.MemoryConsumer.throwOom(MemoryConsumer.java:157)
	at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:97)
	at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.growPointerArrayIfNecessary(UnsafeExternalSorter.java:352)
	at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.allocateMemoryForRecordIfNecessary(UnsafeExternalSorter.java:435)
	at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:455)
	at org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:138)
	at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:226)
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$10(ShuffleExchangeExec.scala:355)
```

`WindowExec` only clear buffer in `fetchNextPartition` so the final partition buffer miss to clear.

It is not a big problem since we have task completion listener.
```scala
taskContext.addTaskCompletionListener(context -> {
  cleanupResources();
});
```

This bug only affects if the window is not the last operator for this task and the follow operator like sort.

### Does this PR introduce _any_ user-facing change?

yes, bug fix

### How was this patch tested?

N/A

Closes #37358 from ulysses-you/window.

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 1fac870126c289a7ec75f45b6b61c93b9a4965d4)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent 3768ee1
History
Tip revision: ed842ba8e82e845efc38bd115909ce54faef318a authored by ulysses-you on 02 August 2022, 09:05:48 UTC
[SPARK-39932][SQL] WindowExec should clear the final partition buffer
Tip revision: ed842ba

README.md

back to top