https://github.com/apache/spark
Revision accfb39e4ddf7f7b54396bd0e35256a04461c693 authored by Dongjoon Hyun on 28 January 2024, 04:24:15 UTC, committed by Dongjoon Hyun on 28 January 2024, 04:25:45 UTC
This PR aims to fix `Master` to reject `/workers/kill/` request if `spark.decommission.enabled` is `false` in order to fix the dangling worker issue. Currently, `spark.decommission.enabled` is `false` by default. So, when a user asks to decommission, only Master marked it `DECOMMISSIONED` while the worker is alive. ``` $ curl -XPOST http://localhost:8080/workers/kill/\?host\=127.0.0.1 ``` **Master UI**  **Worker Log** ``` 24/01/27 18:18:06 WARN Worker: Receive decommission request, but decommission feature is disabled. ``` To be consistent with the existing `Worker` behavior which ignores the request. https://github.com/apache/spark/blob/1787a5261e87e0214a3f803f6534c5e52a0138e6/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L859-L868 No, this is a bug fix. Pass the CI with the newly added test case. No. Closes #44915 from dongjoon-hyun/SPARK-46888. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 20b593811dc02c96c71978851e051d32bf8c3496) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent a2854ba
Tip revision: accfb39e4ddf7f7b54396bd0e35256a04461c693 authored by Dongjoon Hyun on 28 January 2024, 04:24:15 UTC
[SPARK-46888][CORE] Fix `Master` to reject `/workers/kill/` requests if decommission is disabled
[SPARK-46888][CORE] Fix `Master` to reject `/workers/kill/` requests if decommission is disabled
Tip revision: accfb39
| File | Mode | Size |
|---|---|---|
| .github | ||
| R | ||
| assembly | ||
| bin | ||
| binder | ||
| build | ||
| common | ||
| conf | ||
| connector | ||
| core | ||
| data | ||
| dev | ||
| docs | ||
| examples | ||
| graphx | ||
| hadoop-cloud | ||
| launcher | ||
| licenses | ||
| licenses-binary | ||
| mllib | ||
| mllib-local | ||
| project | ||
| python | ||
| repl | ||
| resource-managers | ||
| sbin | ||
| sql | ||
| streaming | ||
| tools | ||
| .asf.yaml | -rw-r--r-- | 1.3 KB |
| .gitattributes | -rw-r--r-- | 130 bytes |
| .gitignore | -rw-r--r-- | 1.8 KB |
| CONTRIBUTING.md | -rw-r--r-- | 997 bytes |
| LICENSE | -rw-r--r-- | 13.0 KB |
| LICENSE-binary | -rw-r--r-- | 22.4 KB |
| NOTICE | -rw-r--r-- | 2.0 KB |
| NOTICE-binary | -rw-r--r-- | 56.5 KB |
| README.md | -rw-r--r-- | 4.5 KB |
| appveyor.yml | -rw-r--r-- | 2.8 KB |
| pom.xml | -rw-r--r-- | 139.1 KB |
| scalastyle-config.xml | -rw-r--r-- | 23.7 KB |
Computing file changes ...
