Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to mobile/third_party/rbe_configs/cc/ #36269

Merged
merged 1 commit into from
Sep 20, 2024

Conversation

asedeno
Copy link
Contributor

@asedeno asedeno commented Sep 20, 2024

This updates files containing paths to older versions of the docker container being used in CI. This matters when clang modules are being used and some dependency (looking at absl) turns on layering checks. Not updating these files prevents abseil from including many standard headers.

These files should be regenerated whenever our base docker container or toolchain is updated.

These files were regenerated following the instructions in https://github.com/bazelbuild/bazel-toolchains

Specifically:

$ ./rbe_configs_gen \
  --bazel_version="6.5.0" \
  --toolchain_container="docker.io/envoyproxy/envoy-build-ubuntu:f94a38f62220a2b017878b790b6ea98a0f6c5f9c" \
  --exec_os="linux" \
  --target_os="linux" \
  --output_src_root="/path/to/envoy-mobile" \
  --output_config_path="third_party/rbe_configs" \
  --cpp_env_json=clang.json \
  --generate_java_configs=False
$ cat clang.json
{"CC": "/opt/llvm/bin/clang", "CXX": "/opt/llvm/bin/clang++"}

Other files were also regenerated, but this commit only contains changes to files with paths.

Risk Level: low
Testing: CI, since this specifically affects the CI.

This updates files containing paths to older versions of the docker
container being used in CI. This matters when clang modules are being
used and some dependency (looking at absl) turns on layering
checks. Not updating these files prevents abseil from including many
standard headers.

These files should be regenerated whenever our base docker container
or toolchain is updated.

These files were regenerated following the instructions in
https://github.com/bazelbuild/bazel-toolchains

Specifically:

```
$ ./rbe_configs_gen \
  --bazel_version="6.5.0" \
  --toolchain_container="docker.io/envoyproxy/envoy-build-ubuntu:f94a38f62220a2b017878b790b6ea98a0f6c5f9c" \
  --exec_os="linux" \
  --target_os="linux" \
  --output_src_root="/path/to/envoy-mobile" \
  --output_config_path="third_party/rbe_configs" \
  --cpp_env_json=clang.json \
  --generate_java_configs=False
$ cat clang.json
{"CC": "/opt/llvm/bin/clang", "CXX": "/opt/llvm/bin/clang++"}
```

Other files were also regenerated, but this commit only contains
changes to files with paths.

Signed-off-by: Alejandro R Sedeño <asedeno@google.com>
Copy link

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #36269 was opened by asedeno.

see: more, trace.

@phlax phlax self-assigned this Sep 20, 2024
@phlax
Copy link
Member

phlax commented Sep 20, 2024

These files should be regenerated whenever our base docker container or toolchain is updated.

the code for updating the build image is here

update-build-image:
if: >-
${{
github.event_name == 'workflow_dispatch'
&& github.event.inputs.task == 'build-image'
}}
name: Update build image (PR)
runs-on: ubuntu-22.04
steps:
- id: appauth
name: Appauth
uses: envoyproxy/toolshed/gh-actions/appauth@actions-v0.2.35
with:
app_id: ${{ secrets.ENVOY_CI_DEP_APP_ID }}
key: ${{ secrets.ENVOY_CI_DEP_APP_KEY }}
- uses: envoyproxy/toolshed/gh-actions/github/checkout@actions-v0.2.35
id: checkout
name: Checkout Envoy repository
with:
config: |
path: envoy
fetch-depth: 0
token: ${{ steps.appauth.outputs.token }}
- uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
name: Checkout Envoy build tools repository
with:
repository: envoyproxy/envoy-build-tools
path: build-tools
fetch-depth: 0
- run: |
shas=(
tag
sha
mobile-sha
gcr-sha)
for sha in "${shas[@]}"; do
current_sha=$(bazel run --config=ci //tools/dependency:build-image-sha "$sha")
echo "${sha}=${current_sha}" >> "$GITHUB_OUTPUT"
done
id: current
name: Current SHAs
working-directory: envoy
- run: |
if [[ -z "$CONTAINER_TAG" ]]; then
# get current build image version
CONTAINER_TAG=$(git log -1 --pretty=format:"%H" "./docker")
fi
echo "tag=${CONTAINER_TAG}" >> "$GITHUB_OUTPUT"
echo "tag_short=${CONTAINER_TAG::7}" >> "$GITHUB_OUTPUT"
env:
CONTAINER_TAG: ${{ inputs.version }}
id: build-tools
name: Build image SHA
working-directory: build-tools
- name: Check Docker SHAs
id: build-images
uses: envoyproxy/toolshed/gh-actions/docker/shas@actions-v0.2.35
with:
images: |
sha: envoyproxy/envoy-build-ubuntu:${{ steps.build-tools.outputs.tag }}
mobile-sha: envoyproxy/envoy-build-ubuntu:mobile-${{ steps.build-tools.outputs.tag }}
gcr-sha: gcr.io/envoy-ci/envoy-build:${{ steps.build-tools.outputs.tag }}
- run: |
SHA_REPLACE=(
"$CURRENT_ENVOY_TAG:$ENVOY_TAG"
"$CURRENT_ENVOY_SHA:${{ fromJSON(steps.build-images.outputs.shas).sha }}"
"$CURRENT_ENVOY_MOBILE_SHA:${{ fromJSON(steps.build-images.outputs.shas).mobile-sha }}"
"$CURRENT_ENVOY_GCR_SHA:${{ fromJSON(steps.build-images.outputs.shas).gcr-sha }}")
echo "replace=${SHA_REPLACE[*]}" >> "$GITHUB_OUTPUT"
name: Find SHAs to replace
id: shas
env:
ENVOY_TAG: ${{ steps.build-tools.outputs.tag }}
CURRENT_ENVOY_TAG: ${{ steps.current.outputs.tag }}
CURRENT_ENVOY_SHA: ${{ steps.current.outputs.sha }}
CURRENT_ENVOY_MOBILE_SHA: ${{ steps.current.outputs.mobile-sha }}
CURRENT_ENVOY_GCR_SHA: ${{ steps.current.outputs.gcr-sha }}
- run: |
echo "${SHA_REPLACE}" | xargs bazel run --config=ci @envoy_toolshed//sha:replace "${PWD}"
env:
SHA_REPLACE: ${{ steps.shas.outputs.replace }}
name: Update SHAs
working-directory: envoy
- name: Create a PR
uses: envoyproxy/toolshed/gh-actions/github/pr@actions-v0.2.35
with:
base: main
body: Created by Envoy dependency bot
branch: dependency-envoy/build-image/latest
committer-name: ${{ env.COMMITTER_NAME }}
committer-email: ${{ env.COMMITTER_EMAIL }}
commit-message: |
deps: Bump build images -> `${{ steps.build-tools.outputs.tag_short }}`
Signed-off-by: ${{ env.COMMITTER_NAME }} <${{ env.COMMITTER_EMAIL }}>
title: 'deps: Bump build images -> `${{ steps.build-tools.outputs.tag_short }}`'
GITHUB_TOKEN: ${{ steps.appauth.outputs.token }}
working-directory: envoy

perhaps we can add something there to do toolchain regeneration at the same time

@asedeno
Copy link
Contributor Author

asedeno commented Sep 20, 2024

I can look into that, with two caveats.

  1. You may be able to deal with faster than I can, as I'm still getting up to speed on how all this works.
  2. I'd like to get this change in sooner rather than later, and not block fixing the broken thing on automating fixing it.

I'm not sure what the automation should look like yet, given the resource exhaustion experienced in #36172. Also, that PR had a series of patches undoing some of the generated changes, some of which were just tidying up, and some of which may not have been needed.

@phlax
Copy link
Member

phlax commented Sep 20, 2024

for sure, out of scope for current change - was more following up on your comment

happy to assist if you want to delve into our ci system

this is one of the easier workflows to change/test/etc so probably not crazy hard to do

Copy link
Member

@phlax phlax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks @asedeno

@phlax phlax merged commit a65a5b6 into envoyproxy:main Sep 20, 2024
31 checks passed
@asedeno asedeno deleted the update_mobile_rbe_modmap branch September 20, 2024 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants