Revision d23ff11d8726fdc768b4f38fc124ad26f7f556e4 authored by Dongjoon Hyun on 27 July 2024, 04:50:35 UTC, committed by Dongjoon Hyun on 27 July 2024, 04:50:35 UTC
### What changes were proposed in this pull request? This PR aims to support server-side environment variable replacement in REST Submission API. - For example, ephemeral Spark clusters with server-side environment variables can provide backend-resource and information without touching client-side applications and configurations. - The place holder pattern is `{{SERVER_ENVIRONMENT_VARIABLE_NAME}}` style like the following. https://github.com/apache/spark/blob/163e512c53208301a8511310023d930d8b77db96/docs/configuration.md?plain=1#L694 https://github.com/apache/spark/blob/163e512c53208301a8511310023d930d8b77db96/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala#L233-L234 ### Why are the changes needed? A user can submits an environment variable holder like `{{AWS_CA_BUNDLE}}` and `{{AWS_ENDPOINT_URL}}` in order to use server-wide environment variables of Spark Master. ``` $ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" \ AWS_ENDPOINT_URL=ENDPOINT_FOR_THIS_CLUSTER \ sbin/start-master.sh $ sbin/start-worker.sh spark://$(hostname):7077 ``` ``` curl -s -k -XPOST http://localhost:6066/v1/submissions/create \ --header "Content-Type:application/json;charset=UTF-8" \ --data '{ "appResource": "", "sparkProperties": { "spark.master": "spark://localhost:7077", "spark.app.name": "", "spark.submit.deployMode": "cluster", "spark.jars": "/Users/dongjoon/APACHE/spark-merge/examples/target/scala-2.13/jars/spark-examples_2.13-4.0.0-SNAPSHOT.jar" }, "clientSparkVersion": "", "mainClass": "org.apache.spark.examples.SparkPi", "environmentVariables": { "AWS_ACCESS_KEY_ID": "A", "AWS_SECRET_ACCESS_KEY": "B", "AWS_ENDPOINT_URL": "{{AWS_ENDPOINT_URL}}" }, "action": "CreateSubmissionRequest", "appArgs": [ "10000" ] }' ``` - http://localhost:4040/environment/ ![Screenshot 2024-07-26 at 16 58 26](https://github.com/user-attachments/assets/c52daf4e-02ce-4015-bda6-895fb39a39a9) ### Does this PR introduce _any_ user-facing change? No. This is a new feature and disabled by default via `spark.master.rest.enabled (default: false)` ### How was this patch tested? Pass the CIs with newly added test case. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47509 from dongjoon-hyun/SPARK-49033. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 163e512
File | Mode | Size |
---|---|---|
.github | ||
R | ||
assembly | ||
bin | ||
binder | ||
build | ||
common | ||
conf | ||
connect | ||
connector | ||
core | ||
data | ||
dev | ||
docs | ||
examples | ||
graphx | ||
hadoop-cloud | ||
launcher | ||
licenses | ||
licenses-binary | ||
mllib | ||
mllib-local | ||
project | ||
python | ||
repl | ||
resource-managers | ||
sbin | ||
sql | ||
streaming | ||
tools | ||
ui-test | ||
.asf.yaml | -rw-r--r-- | 1.3 KB |
.gitattributes | -rw-r--r-- | 130 bytes |
.gitignore | -rw-r--r-- | 1.9 KB |
CONTRIBUTING.md | -rw-r--r-- | 997 bytes |
LICENSE | -rw-r--r-- | 13.2 KB |
LICENSE-binary | -rw-r--r-- | 21.7 KB |
NOTICE | -rw-r--r-- | 2.0 KB |
NOTICE-binary | -rw-r--r-- | 51.6 KB |
README.md | -rw-r--r-- | 4.3 KB |
pom.xml | -rw-r--r-- | 140.3 KB |
scalastyle-config.xml | -rw-r--r-- | 25.9 KB |
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...