Update spark version to 3.5.1 #16325
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Diff for 4744d74:diff --git a/_bashbrew-cat b/_bashbrew-cat
index cd0b29f..e9ff353 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -26,17 +26,17 @@ Architectures: amd64, arm64v8
GitCommit: 028efd4637fb2cf791d5bd9ea70b2fca472de4b7
Directory: 3.5.0/scala2.12-java11-python3-r-ubuntu
-Tags: 3.5.0-scala2.12-java11-python3-ubuntu, 3.5.0-python3, 3.5.0, python3, latest
+Tags: 3.5.0-scala2.12-java11-python3-ubuntu, 3.5.0-python3, 3.5.0
Architectures: amd64, arm64v8
GitCommit: 028efd4637fb2cf791d5bd9ea70b2fca472de4b7
Directory: 3.5.0/scala2.12-java11-python3-ubuntu
-Tags: 3.5.0-scala2.12-java11-r-ubuntu, 3.5.0-r, r
+Tags: 3.5.0-scala2.12-java11-r-ubuntu, 3.5.0-r
Architectures: amd64, arm64v8
GitCommit: 028efd4637fb2cf791d5bd9ea70b2fca472de4b7
Directory: 3.5.0/scala2.12-java11-r-ubuntu
-Tags: 3.5.0-scala2.12-java11-ubuntu, 3.5.0-scala, scala
+Tags: 3.5.0-scala2.12-java11-ubuntu, 3.5.0-scala
Architectures: amd64, arm64v8
GitCommit: 028efd4637fb2cf791d5bd9ea70b2fca472de4b7
Directory: 3.5.0/scala2.12-java11-ubuntu
@@ -46,7 +46,7 @@ Architectures: amd64, arm64v8
GitCommit: 6f68fe0f7051c10f2bf43a50a7decfce2e97baf0
Directory: 3.5.0/scala2.12-java17-python3-r-ubuntu
-Tags: 3.5.0-scala2.12-java17-python3-ubuntu, 3.5.0-java17-python3, 3.5.0-java17, python3-java17
+Tags: 3.5.0-scala2.12-java17-python3-ubuntu, 3.5.0-java17-python3, 3.5.0-java17
Architectures: amd64, arm64v8
GitCommit: 6f68fe0f7051c10f2bf43a50a7decfce2e97baf0
Directory: 3.5.0/scala2.12-java17-python3-ubuntu
@@ -60,3 +60,43 @@ Tags: 3.5.0-scala2.12-java17-ubuntu, 3.5.0-java17-scala
Architectures: amd64, arm64v8
GitCommit: 6f68fe0f7051c10f2bf43a50a7decfce2e97baf0
Directory: 3.5.0/scala2.12-java17-ubuntu
+
+Tags: 3.5.1-scala2.12-java11-python3-r-ubuntu
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java11-python3-r-ubuntu
+
+Tags: 3.5.1-scala2.12-java11-python3-ubuntu, 3.5.1-python3, 3.5.1, python3, latest
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java11-python3-ubuntu
+
+Tags: 3.5.1-scala2.12-java11-r-ubuntu, 3.5.1-r, r
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java11-r-ubuntu
+
+Tags: 3.5.1-scala2.12-java11-ubuntu, 3.5.1-scala, scala
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java11-ubuntu
+
+Tags: 3.5.1-scala2.12-java17-python3-r-ubuntu
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java17-python3-r-ubuntu
+
+Tags: 3.5.1-scala2.12-java17-python3-ubuntu, 3.5.1-java17-python3, 3.5.1-java17, python3-java17
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java17-python3-ubuntu
+
+Tags: 3.5.1-scala2.12-java17-r-ubuntu, 3.5.1-java17-r
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java17-r-ubuntu
+
+Tags: 3.5.1-scala2.12-java17-ubuntu, 3.5.1-java17-scala
+Architectures: amd64, arm64v8
+GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94
+Directory: 3.5.1/scala2.12-java17-ubuntu
diff --git a/_bashbrew-list b/_bashbrew-list
index f93ec40..5d18a67 100644
--- a/_bashbrew-list
+++ b/_bashbrew-list
@@ -22,6 +22,22 @@ spark:3.5.0-scala2.12-java17-python3-r-ubuntu
spark:3.5.0-scala2.12-java17-python3-ubuntu
spark:3.5.0-scala2.12-java17-r-ubuntu
spark:3.5.0-scala2.12-java17-ubuntu
+spark:3.5.1
+spark:3.5.1-java17
+spark:3.5.1-java17-python3
+spark:3.5.1-java17-r
+spark:3.5.1-java17-scala
+spark:3.5.1-python3
+spark:3.5.1-r
+spark:3.5.1-scala
+spark:3.5.1-scala2.12-java11-python3-r-ubuntu
+spark:3.5.1-scala2.12-java11-python3-ubuntu
+spark:3.5.1-scala2.12-java11-r-ubuntu
+spark:3.5.1-scala2.12-java11-ubuntu
+spark:3.5.1-scala2.12-java17-python3-r-ubuntu
+spark:3.5.1-scala2.12-java17-python3-ubuntu
+spark:3.5.1-scala2.12-java17-r-ubuntu
+spark:3.5.1-scala2.12-java17-ubuntu
spark:latest
spark:python3
spark:python3-java17
diff --git a/spark_python3-java17/Dockerfile b/spark_3.5.0-java17/Dockerfile
similarity index 100%
copy from spark_python3-java17/Dockerfile
copy to spark_3.5.0-java17/Dockerfile
diff --git a/spark_r/Dockerfile b/spark_3.5.0-r/Dockerfile
similarity index 100%
copy from spark_r/Dockerfile
copy to spark_3.5.0-r/Dockerfile
diff --git a/spark_scala/Dockerfile b/spark_3.5.0-scala/Dockerfile
similarity index 100%
copy from spark_scala/Dockerfile
copy to spark_3.5.0-scala/Dockerfile
diff --git a/spark_3.5.0-java17-scala/entrypoint.sh b/spark_3.5.0-scala/entrypoint.sh
similarity index 100%
copy from spark_3.5.0-java17-scala/entrypoint.sh
copy to spark_3.5.0-scala/entrypoint.sh
diff --git a/spark_latest/Dockerfile b/spark_3.5.0/Dockerfile
similarity index 100%
copy from spark_latest/Dockerfile
copy to spark_3.5.0/Dockerfile
diff --git a/spark_3.4.1-r/Dockerfile b/spark_3.5.1-java17-r/Dockerfile
similarity index 95%
copy from spark_3.4.1-r/Dockerfile
copy to spark_3.5.1-java17-r/Dockerfile
index 1c9fc38..6f3e867 100644
--- a/spark_3.4.1-r/Dockerfile
+++ b/spark_3.5.1-java17-r/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.4.1-scala2.12-java11-ubuntu
+FROM spark:3.5.1-scala2.12-java17-ubuntu
USER root
diff --git a/spark_3.5.0-java17-scala/Dockerfile b/spark_3.5.1-java17-scala/Dockerfile
similarity index 95%
copy from spark_3.5.0-java17-scala/Dockerfile
copy to spark_3.5.1-java17-scala/Dockerfile
index ed29cba..562d938 100644
--- a/spark_3.5.0-java17-scala/Dockerfile
+++ b/spark_3.5.1-java17-scala/Dockerfile
@@ -36,9 +36,9 @@ RUN set -ex; \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz.asc \
- GPG_KEY=FC3AE3A7EAA1BAC98770840E7E1ABCC53AAA2216
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz.asc \
+ GPG_KEY=FD3E84942E5E6106235A1D25BD356A9F8740E4FF
RUN set -ex; \
export SPARK_TMP="$(mktemp -d)"; \
diff --git a/spark_3.5.0-java17-scala/entrypoint.sh b/spark_3.5.1-java17-scala/entrypoint.sh
similarity index 100%
copy from spark_3.5.0-java17-scala/entrypoint.sh
copy to spark_3.5.1-java17-scala/entrypoint.sh
diff --git a/spark_3.5.0-scala2.12-java17-python3-r-ubuntu/Dockerfile b/spark_3.5.1-scala2.12-java11-python3-r-ubuntu/Dockerfile
similarity index 96%
copy from spark_3.5.0-scala2.12-java17-python3-r-ubuntu/Dockerfile
copy to spark_3.5.1-scala2.12-java11-python3-r-ubuntu/Dockerfile
index fd05497..57c044b 100644
--- a/spark_3.5.0-scala2.12-java17-python3-r-ubuntu/Dockerfile
+++ b/spark_3.5.1-scala2.12-java11-python3-r-ubuntu/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.0-scala2.12-java17-ubuntu
+FROM spark:3.5.1-scala2.12-java11-ubuntu
USER root
diff --git a/spark_3.5.0-scala2.12-java17-python3-r-ubuntu/Dockerfile b/spark_3.5.1-scala2.12-java17-python3-r-ubuntu/Dockerfile
similarity index 96%
copy from spark_3.5.0-scala2.12-java17-python3-r-ubuntu/Dockerfile
copy to spark_3.5.1-scala2.12-java17-python3-r-ubuntu/Dockerfile
index fd05497..59c8024 100644
--- a/spark_3.5.0-scala2.12-java17-python3-r-ubuntu/Dockerfile
+++ b/spark_3.5.1-scala2.12-java17-python3-r-ubuntu/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.0-scala2.12-java17-ubuntu
+FROM spark:3.5.1-scala2.12-java17-ubuntu
USER root
diff --git a/spark_latest/Dockerfile b/spark_latest/Dockerfile
index 78db218..60f8daf 100644
--- a/spark_latest/Dockerfile
+++ b/spark_latest/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.0-scala2.12-java11-ubuntu
+FROM spark:3.5.1-scala2.12-java11-ubuntu
USER root
diff --git a/spark_python3-java17/Dockerfile b/spark_python3-java17/Dockerfile
index e57b4f5..0b9454a 100644
--- a/spark_python3-java17/Dockerfile
+++ b/spark_python3-java17/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.0-scala2.12-java17-ubuntu
+FROM spark:3.5.1-scala2.12-java17-ubuntu
USER root
diff --git a/spark_r/Dockerfile b/spark_r/Dockerfile
index b5fa221..17b6d32 100644
--- a/spark_r/Dockerfile
+++ b/spark_r/Dockerfile
@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-FROM spark:3.5.0-scala2.12-java11-ubuntu
+FROM spark:3.5.1-scala2.12-java11-ubuntu
USER root
diff --git a/spark_scala/Dockerfile b/spark_scala/Dockerfile
index 15f4b31..fbada2d 100644
--- a/spark_scala/Dockerfile
+++ b/spark_scala/Dockerfile
@@ -36,9 +36,9 @@ RUN set -ex; \
# Install Apache Spark
# https://downloads.apache.org/spark/KEYS
-ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz \
- SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz.asc \
- GPG_KEY=FC3AE3A7EAA1BAC98770840E7E1ABCC53AAA2216
+ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz \
+ SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz.asc \
+ GPG_KEY=FD3E84942E5E6106235A1D25BD356A9F8740E4FF
RUN set -ex; \
export SPARK_TMP="$(mktemp -d)"; \Relevant Maintainers:
|
yosifkit
reviewed
Feb 29, 2024
| GitCommit: 4f2d96a415c89cfe0fde89a55e9034d095224c94 | ||
| Directory: ./3.5.1/scala2.12-java11-python3-r-ubuntu | ||
|
|
||
| Tags: 3.5.0-scala2.12-java17-python3-ubuntu, 3.5.0-java17-python3, 3.5.0-java17 |
There was a problem hiding this comment.
Any particular reason to keep the 3.5.0 versions listed here? Removing it here will remove it from the "Supported" section on the Hub readme (and will prevent us from spending cycles rebuilding it on the official build server), but the tag will still be available to users who want it. (See https://github.com/docker-library/official-images#library-definition-files for more detail on this.)
There was a problem hiding this comment.
Merging to get 3.5.1 images building; 3.5.0 tags can be dropped in another PR if they are no longer supported.
There was a problem hiding this comment.
Thanks for reminder, I will submit another PR!
yosifkit
approved these changes
Feb 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch add 3.5.1 version for Apache Spark https://spark.apache.org/docs/3.5.1/ [1]
[1] apache/spark-website#502
[2] apache/spark-docker#59