Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI build failure: Bash exited with code '1' failure - Could NOT find ZLIB #86324

Closed
ivanpovazan opened this issue May 16, 2023 · 5 comments · Fixed by #90413
Closed

CI build failure: Bash exited with code '1' failure - Could NOT find ZLIB #86324

ivanpovazan opened this issue May 16, 2023 · 5 comments · Fixed by #90413
Assignees
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Milestone

Comments

@ivanpovazan
Copy link
Member

ivanpovazan commented May 16, 2023

Seen when running runtime-extra-platforms on linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=274202
Build error leg or test failing: Build / linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot / Build Mono LLVM AOT cross compiler
Pull request: #86266

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build Definition Step Name Console log Pull Request
353096 dotnet/runtime Build Mono LLVM AOT cross compiler Log
353048 dotnet/runtime Build Mono LLVM AOT cross compiler Log #89467
352690 dotnet/runtime Build Mono LLVM AOT cross compiler Log #89431
2230039 dotnet-runtime Build product Log
2229927 dotnet-runtime Build product Log
352326 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2229668 dotnet-runtime Build product Log
2229580 dotnet-runtime Build product Log
2229437 dotnet-runtime Build product Log
351937 dotnet/runtime Build Mono LLVM AOT cross compiler Log
351866 dotnet/runtime Build Mono LLVM AOT cross compiler Log
351758 dotnet/runtime Build Mono LLVM AOT cross compiler Log #89223
351723 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2229082 dotnet-runtime Build product Log
351065 dotnet/runtime Build Mono LLVM AOT cross compiler Log
351063 dotnet/runtime Build Mono LLVM AOT cross compiler Log #89421
351019 dotnet/runtime Build product Log
2228885 dotnet-runtime Build product Log
2228802 dotnet-runtime Build product Log
2228665 dotnet-runtime Build product Log
350495 dotnet/runtime Build Mono LLVM AOT cross compiler Log
350271 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2228258 dotnet-runtime Build product Log
349510 dotnet/runtime Build Mono LLVM AOT cross compiler Log
349464 dotnet/runtime Build product Log
2227956 dotnet-runtime Build product Log
2227939 dotnet-runtime Build product Log
349328 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2227837 dotnet-runtime Build product Log
349156 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2227737 dotnet-runtime Build product Log
349144 dotnet/runtime Build product Log
2227669 dotnet-runtime Build product Log
2227649 dotnet-runtime Build product Log
349045 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2227542 dotnet-runtime Build product Log
348654 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2227439 dotnet-runtime Build product Log
348644 dotnet/runtime Build product Log
2227366 dotnet-runtime Build product Log
2227269 dotnet-runtime Build product Log
348291 dotnet/runtime Build Mono LLVM AOT cross compiler Log
347426 dotnet/runtime Build Mono LLVM AOT cross compiler Log
347390 dotnet/runtime Build product Log
2226549 dotnet-runtime Build product Log
2226537 dotnet-runtime Build product Log
346915 dotnet/runtime Build Mono LLVM AOT cross compiler Log
346860 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2226244 dotnet-runtime Build product Log
346479 dotnet/runtime Build Mono LLVM AOT cross compiler Log
346114 dotnet/runtime Build Mono LLVM AOT cross compiler Log
346085 dotnet/runtime Build product Log
2225817 dotnet-runtime Build product Log
2225791 dotnet-runtime Build product Log
345752 dotnet/runtime Build Mono LLVM AOT cross compiler Log
345634 dotnet/runtime Build Mono LLVM AOT cross compiler Log
345419 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2225456 dotnet-runtime Build product Log
344997 dotnet/runtime Build Mono LLVM AOT cross compiler Log
344698 dotnet/runtime Build Mono LLVM AOT cross compiler Log
344678 dotnet/runtime Build product Log
344129 dotnet/runtime Build Mono LLVM AOT cross compiler Log
343954 dotnet/runtime Build Mono LLVM AOT cross compiler Log #88723
343204 dotnet/runtime Build Mono LLVM AOT cross compiler Log
343172 dotnet/runtime Build product Log
2224068 dotnet-runtime Build product Log
2224038 dotnet-runtime Build product Log
342556 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2223560 dotnet-runtime Build product Log
2223513 dotnet-runtime Build product Log
2223490 dotnet-runtime Build product Log
2223435 dotnet-runtime Build product Log
341622 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2223279 dotnet-runtime Build product Log
2223226 dotnet-runtime Build product Log
341572 dotnet/runtime Build product Log
2223186 dotnet-runtime Build product Log
2223134 dotnet-runtime Build product Log
2223121 dotnet-runtime Build product Log
2223110 dotnet-runtime Build product Log
2223079 dotnet-runtime Build product Log
2223049 dotnet-runtime Build product Log
2223043 dotnet-runtime Build product Log
341305 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2223025 dotnet-runtime Build product Log
2223012 dotnet-runtime Build product Log
2222998 dotnet-runtime Build product Log
2222960 dotnet-runtime Build product Log
2222944 dotnet-runtime Build product Log
2222915 dotnet-runtime Build product Log
2222903 dotnet-runtime Build product Log
341090 dotnet/runtime Build Mono LLVM AOT cross compiler Log
341070 dotnet/runtime Build product Log
2222835 dotnet-runtime Build product Log
340960 dotnet/runtime Build Mono LLVM AOT cross compiler Log
2222775 dotnet-runtime Build product Log
2222765 dotnet-runtime Build product Log
340683 dotnet/runtime Build Mono LLVM AOT cross compiler Log
340670 dotnet/runtime Build product Log
2222634 dotnet-runtime Build product Log
Displaying 100 of 119 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 119

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893
Error message validated: Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:28:40 PM UTC

@ivanpovazan ivanpovazan added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels May 16, 2023
@ivanpovazan ivanpovazan added this to the 8.0.0 milestone May 16, 2023
@ghost
Copy link

ghost commented May 16, 2023

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Seen when running runtime-extra-platforms on linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=274202
Build error leg or test failing: Build / linux-arm64 Release AllSubsets_Mono_LLVMFullAot_RuntimeTests llvmfullaot / Build Mono LLVM AOT cross compiler
Pull request: #86266

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}
Author: ivanpovazan
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure-mono, Known Build Error

Milestone: 8.0.0

@directhex
Copy link
Contributor

Nothing to do with ZLIB, that's just CMake lying to you. The true build error here is clang-16: error: unable to execute command: Executable "ld" doesn't exist!, because we aren't passing a value for -fuse-ld=lld via LDFLAGS because we aren't passing a value for -DCMAKE_TOOLCHAIN_FILE= in this specific scenario. Read on for the full analysis.

Once upon a time, we had two code paths in mono.proj - Target Name="BuildMonoRuntime" which compiled a runtime against $(TargetOS)-$(TargetArchitecture), and Target Name="BuildMonoCross" which compiled a cross-compiler (runtime) against $(HostOS)-$(BuildArchitecture) with the codegen targeting $(TargetOS)-$(TargetArchitecture).

So, for example, build for android-arm on an x64 Mac and you'd get BuildMonoRuntime generating an android-arm libmonosgen.so and BuildMonoCross generating an osx-x64 mono-aot-cross. Strict 1:1 mapping (i.e. no consistent way to produce multiple cross-compilers for the same target, or any cross-compiler for a different target than the host)

In 2021 I wrote a new build system, intended to allow a many:many relationship, and part of that was changing BuildMonoCross to deal with the difference between "generate code for RID" and "compile code for RID". I kept the old method working, in order to not break anyone's workflow, but it became deprecated & basically abandoned, other than for these LLVM AOT lanes.

Fast forward to recently, when we started building everything possible with CBL-Mariner. One of the changes forced by Mariner is treating all compiles as cross-compiles - instead of "compile for the host which is always x64 or cross-compile for the OS in /crossrootfs", all architectures are cross-compiles. This is great! However, it runs into a problem with the legacy-BuildMonoCross lanes like this one, because we cannot BuildMonoRuntime using an arm64 crossrootfs and BuildMonoCross using an x64 crossrootfs - we have no Docker images containing both crossroots at once.

This is the patch to enable cross-compiler mode for legacy BuildMonoCross:

diff --git a/src/mono/mono.proj b/src/mono/mono.proj
index de64d7d3440..c187b348c1a 100644
--- a/src/mono/mono.proj
+++ b/src/mono/mono.proj
@@ -665,6 +665,12 @@
   <!-- Build AOT cross compiler (if available) -->
   <Target Name="BuildMonoCross" Condition="'$(BuildMonoAOTCrossCompiler)' == 'true'" DependsOnTargets="BuildMonoRuntime">
 
+    <!-- Legacy cross-compiler mode (build cross compiler for host CPU/OS) -->
+    <PropertyGroup Condition="'$(RealTargetOS)' == '' and '$(RealTargetArchitecture)' == ''">
+      <RealTargetOS>$(HostOS)</RealTargetOS>
+      <RealTargetArchitecture>$(BuildArchitecture)</RealTargetArchitecture>
+    </PropertyGroup>
+
     <!-- iOS/tvOS specific options -->
     <PropertyGroup Condition="'$(TargetstvOS)' == 'true' or '$(TargetsiOS)' == 'true'">
       <!-- FIXME: Disable for simulator -->

But it won't work. Therefore we have three choices:

  1. Don't use Mariner for this lane, revert to whatever we were using before
  2. Create a new multi-targeting Mariner docker image, and use that (passing an overridden crossrootfs property where needed to the appropriate targets, since we assume we use the same property throughout)
  3. Split this job in two - an offset generation job using an arm64 crossrootfs, and the AOT compiler job using an x64 crossrootfs (this is basically the model for official builds, where we generate offsets in eng/pipelines/mono/templates/generate-offsets.yml and consume them in eng/pipelines/common/global-build-job.yml using /p:MonoCrossAOTTargetOS=OSNAMEFROMMONOAOTCROSSDOTPROJ /p:SkipMonoCrossJitConfigure=true /p:BuildMonoAOTCrossCompilerOnly=true

@kotlarmilos
Copy link
Member

Thanks for the explanation. It sounds to me that the 3rd option is aligned with the official builds and may be a path we want to consider. If that makes sense, I can try to split this job into offset generation and the AOT compilation.

@SamMonoRT
Copy link
Member

SamMonoRT commented Aug 11, 2023

@directhex @kotlarmilos @steveisok - Milos reminded me of this while checking on another PR We need to move forward with this, given a lack of desktop full aot llvm lane. Two things we should do asap - 1. Disable this lane (even in extra-platforms) prior to RC1 cutoff. 2. Work on option 3 suggested above for 9.0 (essentially at the earliest once RC1 is out of the way next week). For this we will need some help understanding the cross-compiler/official build.

Moving this to 9.0, but we need to disable the lane prior to rc1 cutoff for cleaner CI to remove the blocking green ci tag

@SamMonoRT SamMonoRT modified the milestones: 8.0.0, 9.0.0 Aug 11, 2023
@steveisok
Copy link
Member

Moving this to 9.0, but we need to disable the lane prior to rc1 cutoff for cleaner CI to remove the blocking green ci tag

@directhex can you please do this before Monday's cutoff?

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 11, 2023
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 11, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants