https://github.com/apache/spark
Revision 651f5da7d58554ebd4b15c5b0204acf2d08ca439 authored by Dongjoon Hyun on 05 December 2022, 09:09:50 UTC, committed by Dongjoon Hyun on 05 December 2022, 09:20:12 UTC
This PR aims to prevent `getReusablePVCs` from choosing recently created PVCs in the very previous batch by excluding newly created PVCs whose creation time is within `spark.kubernetes.allocation.batch.delay`. In case of slow K8s control plane situation where `spark.kubernetes.allocation.batch.delay` is too short relatively or `spark.kubernetes.executor.enablePollingWithResourceVersion=true` is used, `onNewSnapshots` may not return the full list of executor pods created by the previous batch. This sometimes makes Spark driver think the PVCs in the previous batch are reusable for the next batch. No. Pass the CIs with the newly created test case. Closes #38912 from dongjoon-hyun/SPARK-41388. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit e234cd8276a603ab8a191dd078b11c605b22a50c) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 5e68c98
Tip revision: 651f5da7d58554ebd4b15c5b0204acf2d08ca439 authored by Dongjoon Hyun on 05 December 2022, 09:09:50 UTC
[SPARK-41388][K8S] `getReusablePVCs` should ignore recently created PVCs in the previous batch
[SPARK-41388][K8S] `getReusablePVCs` should ignore recently created PVCs in the previous batch
Tip revision: 651f5da
| File | Mode | Size |
|---|---|---|
| .github | ||
| .idea | ||
| R | ||
| assembly | ||
| bin | ||
| binder | ||
| build | ||
| common | ||
| conf | ||
| core | ||
| data | ||
| dev | ||
| docs | ||
| examples | ||
| external | ||
| graphx | ||
| hadoop-cloud | ||
| launcher | ||
| licenses | ||
| licenses-binary | ||
| mllib | ||
| mllib-local | ||
| project | ||
| python | ||
| repl | ||
| resource-managers | ||
| sbin | ||
| sql | ||
| streaming | ||
| tools | ||
| .asf.yaml | -rw-r--r-- | 1.1 KB |
| .gitattributes | -rw-r--r-- | 130 bytes |
| .gitignore | -rw-r--r-- | 2.0 KB |
| CONTRIBUTING.md | -rw-r--r-- | 997 bytes |
| LICENSE | -rw-r--r-- | 13.1 KB |
| LICENSE-binary | -rw-r--r-- | 22.4 KB |
| NOTICE | -rw-r--r-- | 2.0 KB |
| NOTICE-binary | -rw-r--r-- | 56.5 KB |
| README.md | -rw-r--r-- | 4.4 KB |
| appveyor.yml | -rw-r--r-- | 2.7 KB |
| pom.xml | -rw-r--r-- | 137.5 KB |
| scalastyle-config.xml | -rw-r--r-- | 22.0 KB |
Computing file changes ...
