Revision 9c1678534e9087859aaa33e83d996c68685af0c0 authored by Dongjoon Hyun on 16 February 2023, 05:52:17 UTC, committed by Dongjoon Hyun on 16 February 2023, 05:52:54 UTC
### What changes were proposed in this pull request?

This is found during Apache Spark 3.3.2 docker image publishing. It's not an Apache Spark but important for `docker-image-tool.sh` to provide backward compatibility during cross-building. This PR targets for all **future releases**, Apache Spark 3.4.0/3.3.3/3.2.4.

### Why are the changes needed?

Docker `buildx` v0.10.0 publishes OCI Manifests by default which is not supported by `docker manifest` command like the following.
https://github.com/docker/buildx/issues/1509
```
$ docker manifest inspect apache/spark:v3.3.2
no such manifest: docker.io/apache/spark:v3.3.2
```

Note that the published images are working on both AMD64/ARM64 machines, but `docker manifest` cannot be used. For example, we cannot create `latest` tag.

### Does this PR introduce _any_ user-facing change?

This will fix the regression of Docker `buildx`.

### How was this patch tested?

Manually builds the multi-arch image and check `manifest`.

```
$ docker manifest inspect apache/spark:v3.3.2
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:30ae5023fc384ae3b68d2fb83adde44b1ece05f926cfceecac44204cdc9e79cb",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3444,
         "digest": "sha256:aac13b5b5a681aefa91036d2acae91d30a743c2e78087c6df79af4de46a16e1b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}
```

Closes #40051 from dongjoon-hyun/SPARK-42462.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 2ac70ae5381333aa899d82f6cd4c3bbae524e1c2)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 07b7199
History
File Mode Size
ansible-for-test-node
create-release
deps
sparktestsupport
tests
.gitignore -rw-r--r-- 25 bytes
.rat-excludes -rw-r--r-- 2.2 KB
.scalafmt.conf -rw-r--r-- 1.0 KB
README.md -rw-r--r-- 197 bytes
appveyor-guide.md -rw-r--r-- 8.7 KB
appveyor-install-dependencies.ps1 -rw-r--r-- 3.8 KB
change-scala-version.sh -rwxr-xr-x 3.3 KB
check-license -rwxr-xr-x 2.5 KB
checkstyle-suppressions.xml -rw-r--r-- 2.4 KB
checkstyle.xml -rw-r--r-- 8.3 KB
eslint.json -rw-r--r-- 423 bytes
github_jira_sync.py -rwxr-xr-x 7.0 KB
lint-java -rwxr-xr-x 1.2 KB
lint-js -rwxr-xr-x 1.6 KB
lint-python -rwxr-xr-x 7.6 KB
lint-r -rwxr-xr-x 1.4 KB
lint-r.R -rw-r--r-- 1.3 KB
lint-scala -rwxr-xr-x 925 bytes
make-distribution.sh -rwxr-xr-x 8.6 KB
merge_spark_pr.py -rwxr-xr-x 23.0 KB
mima -rwxr-xr-x 2.0 KB
package-lock.json -rw-r--r-- 81.3 KB
package.json -rw-r--r-- 83 bytes
pip-sanity-check.py -rw-r--r-- 1.2 KB
reformat-python -rwxr-xr-x 1.3 KB
requirements.txt -rw-r--r-- 505 bytes
run-pip-tests -rwxr-xr-x 4.6 KB
run-tests -rwxr-xr-x 1.1 KB
run-tests-jenkins -rwxr-xr-x 1.4 KB
run-tests-jenkins.py -rwxr-xr-x 9.6 KB
run-tests.py -rwxr-xr-x 31.2 KB
sbt-checkstyle -rwxr-xr-x 1.3 KB
scalafmt -rwxr-xr-x 908 bytes
scalastyle -rwxr-xr-x 1.4 KB
test-dependencies.sh -rwxr-xr-x 6.1 KB
tox.ini -rw-r--r-- 1.5 KB

README.md

back to top