feat(build): spark 3.5.2 #42
Conversation
| @@ -40,57 +40,25 @@ TERM=xterm-color ./dev/make-distribution.sh \ | |||
| ${HIVE_INSTALL_FLAG:+"-Phive"} \ | |||
| -DskipTests | |||
|
|
|||
| SPARK_MAJOR_VERSION="$(echo "${SPARK_VERSION}" | cut -d '.' -f1)" | |||
There was a problem hiding this comment.
not needed anymore as we no longer use any spark version that is 2.y.z
|
|
||
| if [[ ${SPARK_MAJOR_VERSION} -eq 2 && ${SPARK_MINOR_VERSION} -eq 4 ]]; then # 2.4.z |
There was a problem hiding this comment.
not needed anymore as we no longer use any spark version that is 2.y.z
|
|
||
| if [[ ${SPARK_MAJOR_VERSION} -eq 3 && ${SPARK_MINOR_VERSION} -ge 4 ]]; then # >=3.4 | ||
| # From Spark v3.4.0 onwards, openjdk is not the prefered base image source as it i | ||
| # deprecated and taken over by eclipse-temurin. slim-buster variants are not available | ||
| # on eclipse-temurin at the moment. | ||
| IMAGE_VARIANT="jre" |
There was a problem hiding this comment.
jre-8 no longer works properly with the current setup as it is upgraded to Ubuntu 22, and the python within that version raises an error when user install python packages globally. However, Dockerfile packaged in Spark still installs packages globally (see prev GH action logs here
| SPARK_LABEL="${SPARK_VERSION}" | ||
| TAG_NAME="${SELF_VERSION}_${SPARK_LABEL}_hadoop-${HADOOP_VERSION}_scala-${SCALA_VERSION}_java-${JAVA_VERSION}" | ||
|
|
||
| # ./bin/docker-image-tool.sh \ |
|
|
||
| docker tag "${IMAGE_NAME}:${TAG_NAME}" "${IMAGE_ORG}/${IMAGE_NAME}:${TAG_NAME}" |
There was a problem hiding this comment.
to only use image tags that has self version to prevent overwritten previously working images when rebuild with new base jre images
| - Add Spark 3.5.1 | ||
| - Add Hadoop 3.3.6 | ||
| - Add support for Java 17 for Spark 3.5.1 | ||
| - Fix Ubuntu-based images to use `jre-focal` variant instead of `jre` which was recently upgraded to Ubuntu Jammy to v22.y.z and causing system level python package installation to fail due to [PEP 668](https://issues.apache.org/jira/browse/SPARK-49068) |
There was a problem hiding this comment.
lol i'm pretty sure this guy (Chao Sun) who reported the issue was a singaporean colleague at my previous company
There was a problem hiding this comment.
Time to drop him a message to share with him how to fix
push-images.sh
Outdated
|
|
||
| docker tag "${IMAGE_NAME}:${TAG_NAME}" "${IMAGE_ORG}/${IMAGE_NAME}:${TAG_NAME}" | ||
| docker tag "${IMAGE_NAME}:${TAG_NAME}" "test_${IMAGE_ORG}/${IMAGE_NAME}:${TAG_NAME}" |
There was a problem hiding this comment.
ah thanks for the catch 6c00c0e
See
CHANGELOG.mdfor details