https://github.com/apache/spark
Revision 2268665663684dd381adf266feb74ac97a53900d authored by Dongjoon Hyun on 02 June 2022, 03:10:01 UTC, committed by Dongjoon Hyun on 02 June 2022, 03:10:08 UTC
### What changes were proposed in this pull request?

This PR aims to avoid the deprecation of `spark.kubernetes.memoryOverheadFactor` from Apache Spark 3.3. In addition, also recovers the documentation which is removed mistakenly at the `deprecation`. `Deprecation` is not a removal.

### Why are the changes needed?

- Apache Spark 3.3.0 RC complains always about `spark.kubernetes.memoryOverheadFactor` because the configuration has the default value (which is not given by the users). There is no way to remove the warnings which means the directional message is not helpful and makes the users confused in a wrong way. In other words, we still get warnings even we use only new configurations or no configuration.
```
22/06/01 23:53:49 WARN SparkConf: The configuration key 'spark.kubernetes.memoryOverheadFactor' has been deprecated as of Spark 3.3.0 and may be removed in the future. Please use spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor
22/06/01 23:53:49 WARN SparkConf: The configuration key 'spark.kubernetes.memoryOverheadFactor' has been deprecated as of Spark 3.3.0 and may be removed in the future. Please use spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor
22/06/01 23:53:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/06/01 23:53:50 WARN SparkConf: The configuration key 'spark.kubernetes.memoryOverheadFactor' has been deprecated as of Spark 3.3.0 and may be removed in the future. Please use spark.driver.memoryOverheadFactor and spark.executor.memoryOverheadFactor
```

- The minimum constraint is slightly different because `spark.kubernetes.memoryOverheadFactor` allowed 0 since Apache Spark 2.4 while new configurations disallow `0`.

- This documentation removal might be too early because the deprecation is not the removal of configuration. This PR recoveres the removed doc and added the following.
```
This will be overridden by the value set by
<code>spark.driver.memoryOverheadFactor</code> and
<code>spark.executor.memoryOverheadFactor</code> explicitly.
```

### Does this PR introduce _any_ user-facing change?

No. This is a consistent with the existing behavior.

### How was this patch tested?

Pass the CIs.

Closes #36744 from dongjoon-hyun/SPARK-39360.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 6d43556089a21b26d1a7590fbe1e25bd1ca7cedd)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 37aa079
History
Tip revision: 2268665663684dd381adf266feb74ac97a53900d authored by Dongjoon Hyun on 02 June 2022, 03:10:01 UTC
[SPARK-39360][K8S] Remove deprecation of `spark.kubernetes.memoryOverheadFactor` and recover doc
Tip revision: 2268665
File Mode Size
workflows
PULL_REQUEST_TEMPLATE -rw-r--r-- 3.0 KB
labeler.yml -rw-r--r-- 4.5 KB

back to top