Revision 9c1678534e9087859aaa33e83d996c68685af0c0 authored by Dongjoon Hyun on 16 February 2023, 05:52:17 UTC, committed by Dongjoon Hyun on 16 February 2023, 05:52:54 UTC
### What changes were proposed in this pull request? This is found during Apache Spark 3.3.2 docker image publishing. It's not an Apache Spark but important for `docker-image-tool.sh` to provide backward compatibility during cross-building. This PR targets for all **future releases**, Apache Spark 3.4.0/3.3.3/3.2.4. ### Why are the changes needed? Docker `buildx` v0.10.0 publishes OCI Manifests by default which is not supported by `docker manifest` command like the following. https://github.com/docker/buildx/issues/1509 ``` $ docker manifest inspect apache/spark:v3.3.2 no such manifest: docker.io/apache/spark:v3.3.2 ``` Note that the published images are working on both AMD64/ARM64 machines, but `docker manifest` cannot be used. For example, we cannot create `latest` tag. ### Does this PR introduce _any_ user-facing change? This will fix the regression of Docker `buildx`. ### How was this patch tested? Manually builds the multi-arch image and check `manifest`. ``` $ docker manifest inspect apache/spark:v3.3.2 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 3444, "digest": "sha256:30ae5023fc384ae3b68d2fb83adde44b1ece05f926cfceecac44204cdc9e79cb", "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 3444, "digest": "sha256:aac13b5b5a681aefa91036d2acae91d30a743c2e78087c6df79af4de46a16e1b", "platform": { "architecture": "arm64", "os": "linux" } } ] } ``` Closes #40051 from dongjoon-hyun/SPARK-42462. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 2ac70ae5381333aa899d82f6cd4c3bbae524e1c2) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 07b7199
File | Mode | Size |
---|---|---|
ansible-for-test-node | ||
create-release | ||
deps | ||
sparktestsupport | ||
tests | ||
.gitignore | -rw-r--r-- | 25 bytes |
.rat-excludes | -rw-r--r-- | 2.2 KB |
.scalafmt.conf | -rw-r--r-- | 1.0 KB |
README.md | -rw-r--r-- | 197 bytes |
appveyor-guide.md | -rw-r--r-- | 8.7 KB |
appveyor-install-dependencies.ps1 | -rw-r--r-- | 3.8 KB |
change-scala-version.sh | -rwxr-xr-x | 3.3 KB |
check-license | -rwxr-xr-x | 2.5 KB |
checkstyle-suppressions.xml | -rw-r--r-- | 2.4 KB |
checkstyle.xml | -rw-r--r-- | 8.3 KB |
eslint.json | -rw-r--r-- | 423 bytes |
github_jira_sync.py | -rwxr-xr-x | 7.0 KB |
lint-java | -rwxr-xr-x | 1.2 KB |
lint-js | -rwxr-xr-x | 1.6 KB |
lint-python | -rwxr-xr-x | 7.6 KB |
lint-r | -rwxr-xr-x | 1.4 KB |
lint-r.R | -rw-r--r-- | 1.3 KB |
lint-scala | -rwxr-xr-x | 925 bytes |
make-distribution.sh | -rwxr-xr-x | 8.6 KB |
merge_spark_pr.py | -rwxr-xr-x | 23.0 KB |
mima | -rwxr-xr-x | 2.0 KB |
package-lock.json | -rw-r--r-- | 81.3 KB |
package.json | -rw-r--r-- | 83 bytes |
pip-sanity-check.py | -rw-r--r-- | 1.2 KB |
reformat-python | -rwxr-xr-x | 1.3 KB |
requirements.txt | -rw-r--r-- | 505 bytes |
run-pip-tests | -rwxr-xr-x | 4.6 KB |
run-tests | -rwxr-xr-x | 1.1 KB |
run-tests-jenkins | -rwxr-xr-x | 1.4 KB |
run-tests-jenkins.py | -rwxr-xr-x | 9.6 KB |
run-tests.py | -rwxr-xr-x | 31.2 KB |
sbt-checkstyle | -rwxr-xr-x | 1.3 KB |
scalafmt | -rwxr-xr-x | 908 bytes |
scalastyle | -rwxr-xr-x | 1.4 KB |
test-dependencies.sh | -rwxr-xr-x | 6.1 KB |
tox.ini | -rw-r--r-- | 1.5 KB |
Computing file changes ...