Make CI smarter about when to rebuild the base images #670
Conversation
gerrod3
force-pushed
the
ci-smart-base-build
branch
6 times, most recently
from
9f991f5
to
71b4f04
Compare
|
Current base-images CI results: first build (cache-miss) -> 18 mins, second build (cache-hit) -> 6 mins. The new slow part in the cache-hit is that each container we are checking have to update their repo metadata which is quite slow... Wonder if there is a way to do this once and reuse the same metadata across containers. |
| - name: Check for updates on cached images | ||
| if: steps.cache.outputs.cache-hit == 'true' | ||
| run: | | ||
| sudo podman run --rm --privileged multiarch/qemu-user-static --reset -p yes |
There was a problem hiding this comment.
This line allows us to run/build the arm64 images. Without it podman won't be able to run them.
There was a problem hiding this comment.
That at the very least should be a comment here.
Just more background info for me: This runs a command in a container image named "multiarch/qemu-user-static" that flips a switch somewhere and exits right away?
There was a problem hiding this comment.
Yeah it runs a script in the container that modifies a file on the host to enable support. https://github.com/multiarch/qemu-user-static
| podman build --platform linux/$ARCH --format docker --file images/pulp_ci_centos/Containerfile --tag pulp/pulp-ci-centos9:${TEMP_BASE_TAG}-${ARCH} --build-arg FROM_TAG=${TEMP_BASE_TAG}-${ARCH} . | ||
| for image in "${images[@]}"; do | ||
| echo "Building image ${image}" | ||
| arch=${image:(-5):5} |
There was a problem hiding this comment.
Dangerous assumption. Not all archs out there have 5 chars.
There was a problem hiding this comment.
We are currently only using archs with 5 chars. What would you suggest I change it to?
There was a problem hiding this comment.
Sooo, image can contain dashes. Maybe we can assume arch does not and split on the last "-". Or we use ":" instead as a separator.
Also I'd really prefer to consistently use capitalized (all caps) variable names in shell scripts.
There was a problem hiding this comment.
I switched the separator to use ":" so now we can add support for archs with more than 5 chars.
| podman build --platform linux/$ARCH --format docker --file images/pulp_ci_centos/Containerfile --tag pulp/pulp-ci-centos9:${TEMP_BASE_TAG}-${ARCH} --build-arg FROM_TAG=${TEMP_BASE_TAG}-${ARCH} . | ||
| for image in "${images[@]}"; do | ||
| echo "Building image ${image}" | ||
| arch=${image:(-5):5} |
There was a problem hiding this comment.
Sooo, image can contain dashes. Maybe we can assume arch does not and split on the last "-". Or we use ":" instead as a separator.
Also I'd really prefer to consistently use capitalized (all caps) variable names in shell scripts.
gerrod3
force-pushed
the
ci-smart-base-build
branch
from
71b4f04
to
6e8c2cf
Compare
The base images action now takes the hash of the files used to build the base-image & ci-centos-image and then uses that as the key to pull the images from the cache. If it is a cache miss, then we build the images and upload to the cache like normal. If it is a cache hit then we import the images and check if each one needs to be rebuilt by running
dnf check-update. This should reduce the amount of times we build the base & ci-centos images which is roughly half the CI run time! Also, I removed the temporary build tags as they were superfluous (and confusing).