Revision d23ff11d8726fdc768b4f38fc124ad26f7f556e4 authored by Dongjoon Hyun on 27 July 2024, 04:50:35 UTC, committed by Dongjoon Hyun on 27 July 2024, 04:50:35 UTC
### What changes were proposed in this pull request?

This PR aims to support server-side environment variable replacement in REST Submission API.

- For example, ephemeral Spark clusters with server-side environment variables can provide backend-resource and information without touching client-side applications and configurations.

- The place holder pattern is `{{SERVER_ENVIRONMENT_VARIABLE_NAME}}` style like the following.

https://github.com/apache/spark/blob/163e512c53208301a8511310023d930d8b77db96/docs/configuration.md?plain=1#L694

https://github.com/apache/spark/blob/163e512c53208301a8511310023d930d8b77db96/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala#L233-L234

### Why are the changes needed?

A user can submits an environment variable holder like `{{AWS_CA_BUNDLE}}` and `{{AWS_ENDPOINT_URL}}` in order to use server-wide environment variables of Spark Master.

```
$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" \
  AWS_ENDPOINT_URL=ENDPOINT_FOR_THIS_CLUSTER \
  sbin/start-master.sh

$ sbin/start-worker.sh spark://$(hostname):7077
```

```
curl -s -k -XPOST http://localhost:6066/v1/submissions/create \
  --header "Content-Type:application/json;charset=UTF-8" \
  --data '{
          "appResource": "",
          "sparkProperties": {
            "spark.master": "spark://localhost:7077",
            "spark.app.name": "",
            "spark.submit.deployMode": "cluster",
            "spark.jars": "/Users/dongjoon/APACHE/spark-merge/examples/target/scala-2.13/jars/spark-examples_2.13-4.0.0-SNAPSHOT.jar"
          },
          "clientSparkVersion": "",
          "mainClass": "org.apache.spark.examples.SparkPi",
          "environmentVariables": {
            "AWS_ACCESS_KEY_ID": "A",
            "AWS_SECRET_ACCESS_KEY": "B",
            "AWS_ENDPOINT_URL": "{{AWS_ENDPOINT_URL}}"
          },
          "action": "CreateSubmissionRequest",
          "appArgs": [ "10000" ]
  }'
```

- http://localhost:4040/environment/

![Screenshot 2024-07-26 at 16 58 26](https://github.com/user-attachments/assets/c52daf4e-02ce-4015-bda6-895fb39a39a9)

### Does this PR introduce _any_ user-facing change?

No. This is a new feature and disabled by default via `spark.master.rest.enabled (default: false)`

### How was this patch tested?

Pass the CIs with newly added test case.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #47509 from dongjoon-hyun/SPARK-49033.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 163e512
History

README.md

back to top