Failed to find headers with clitools clang on macOS 12.5 (M1)

[shoe42]

On an M1 mac, running macOS 12.5 and with clitools version 13.0.0.0.1.1627064638, if you try and use the stock clitools clang, the build process fails to find some headers - see below log. My current workaround is to use clang from Homebrew's llvm@13.

Failing build:

$ CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX="$(pwd)/../install" -DCMAKE_PREFIX_PATH='/opt/homebrew/opt/llvm@13' -DTERRA_ENABLE_CUDA=OFF ..

Running make a 2nd time to try and just get the errors outputs the following:

$ make -j8
[  0%] Generating luajit/lib/libluajit-5.1.a, luajit/lib/libluajit-5.1.dylib, luajit/bin/luajit-2.1.0-beta3, luajit/include/luajit-2.1/lua.h, luajit/include/luajit-2.1/lualib.h, luajit/include/luajit-2.1/lauxlib.h, luajit/include/luajit-2.1/luaconf.h
==== Building LuaJIT 2.1.0-beta3 ====
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[ 80%] Built target LLVMObjectFiles
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
lj_arch.h:82:10: fatal error: 'TargetConditionals.h' file not found
#include "TargetConditionals.h"
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
HOSTCC    host/minilua.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_fold.o
HOSTCC    host/buildvm_lib.o
CC        lj_gc.o
CC        lj_obj.o
CC        lj_char.o
In file included from host/buildvm_fold.c:In file included from 6host/buildvm_lib.c:
:6host/buildvm.h:9:10: fatal error: 'sys/types.h' file not found
:
host/buildvm.h:9:10: #include <sys/types.h>
         ^~~~~~~~~~~~~
In file included from fatal error: 'sys/types.h' file not found
#include <sys/types.h>
         ^~~~~~~~~~~~~
host/buildvm_peobj.c:9:
host/buildvm.h:9:10: fatal error: 'sys/types.h' file not found
#include <sys/types.h>
         ^~~~~~~~~~~~~
1 error generated.
host/minilua.c:34:10: fatal error: 'math.h' file not found
#include <math.h>
         ^~~~~~~~
1 error generated.
make[4]: *** [host/buildvm_peobj.o] Error 1
make[4]: *** Waiting for unfinished jobs....
In file included from host/buildvm_asm.c:6:
host/buildvm.h:9:10: fatal error: 'sys/types.h' file not found
#include <sys/types.h>
         ^~~~~~~~~~~~~
make[4]: *** [host/buildvm_fold.o] Error 1
In file included from lj_gc.c:12:
In file included from ./lj_obj.h:13:
./lj_def.h:45:10: fatal error: 'string.h' file not found
#include <string.h>
         ^~~~~~~~~~
1 error generated.
make[4]: *** [host/buildvm_lib.o] Error 1
1 error generated.
make[4]: *** [host/buildvm_asm.o] Error 1
In file included from lj_obj.c:9:
In file included from ./lj_obj.h:13:
./lj_def.h:45:10: fatal error: 'string.h' file not found
#include <string.h>
         ^~~~~~~~~~
In file included from lj_char.c:22:
In file included from ./lj_char.h:9:
./lj_def.h:45:10: fatal error: 'string.h' file not found
#include <string.h>
         ^~~~~~~~~~
1 error generated.
1 error generated.
make[4]: *** [lj_char.o] Error 1
make[4]: *** [lj_obj.o] Error 1
1 error generated.
make[4]: *** [lj_gc.o] Error 1
1 error generated.
make[4]: *** [host/minilua.o] Error 1
make[3]: *** [src/luajit] Error 2
make[2]: *** [luajit/lib/libluajit-5.1.a] Error 2
make[1]: *** [CMakeFiles/LuaJIT.dir/all] Error 2
make: *** [all] Error 2

Some other info:

$ clang --version
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ xcodebuild -version
Xcode 13.4.1
Build version 13F100

[elliottslaughter]

Does it help if you build with:

export SDKROOT="$(xcrun --sdk macosx --show-sdk-path)"

[elliottslaughter]

For what it's worth, I confirmed that I get this error on my x86 Mac when I unset SDKROOT (which I normally have set as shown above).

[shoe42]

Looks like that fixes it! I've never had to manually set it before, what part of the build process uses SDKROOT, out of interest?

[elliottslaughter]

A while back, macOS removed /usr/include. We had a big discussion on that in #365, but the end result was that, as best we can tell, SDKROOT is the standard way to specify where the development toolkit is located.

As best I can tell, Apple Clang does not need this to be set, because they add custom code on top of vanilla Clang to auto-detect it. In contrast, if you build a vanilla Clang, it will not (by default) detect this; but it will read SDKROOT if you set it. It may be that Homebrew is setting up a configuration variable in Clang to set this automatically, or else do something the vanilla Clang isn't doing.

But as for the reason why the error occurs with Apple Clang: the auto-detection seems to be finicky. For some reason I don't fully understand, when we use the full path to the C compiler (as we need to do to build LuaJIT), the auto-detection gets shut off. So even though Apple Clang should auto-detect it, you need to set SDKROOT to make the LuaJIT build work. (Or maybe there is something specific to how the LuaJIT build works, I'm not sure. I haven't really dug too deep into this.) Terra itself should build fine because it's just using the Apple Clang compiler as normal.

But at any rate, you will need SDKROOT set in order for Terra itself to work because it has the same issues as a vanilla Clang.

[shoe42]

Gotcha, interesting quirk to note! Thanks for the detailed info, I can't recall if this is mentioned in the build instructions, but if not it might make a good footnote. Looks like clang might have added auto-detection in https://reviews.llvm.org/D109460?id=371426 through xcrun, but whether that works or not I can't test. I can't immediately trigger this issue using a test C file containing some of the problematic headers, so it's certainly a strange problem, but I'll take note of this for the future.

[elliottslaughter]

By the way, I'd be curious how the test suite looks on M1. You could run either ctest -j4 on a CMake build directory or ../build/bin/terra ./run from the tests directory and paste the bottom part of the output (where it shows test pass rate and failing tests).

[shoe42]

I presume running both is redundant, but here's the pass rate and failing tests:

• ctest -j4:

82% tests passed, 98 tests failed out of 549

Total Test time (real) =  15.99 sec

The following tests FAILED:
	  4 - ainline.t (Failed)
	 21 - arrayt.t (Failed)
	 22 - arrayt2.t (Failed)
	 26 - atoi.t (Failed)
	 27 - atomicrmw.t (Failed)
	 28 - avxhadd.t (Failed)
	 30 - benchmark_dgemm.t (Failed)
	 31 - benchmark_fannkuchredux.t (Failed)
	 32 - benchmark_nbody.t (Failed)
	 33 - bf.t (Failed)
	 36 - blocking.t (Failed)
	 37 - blocking2-fixed.t (Failed)
	 38 - blocking2.t (Failed)
	 39 - blocking3.t (Failed)
	 41 - bounce.t (Failed)
	 42 - breaktest.t (Failed)
	 43 - bug.t (Failed)
	 44 - bug2.t (Failed)
	 46 - bug372.t (Failed)
	 48 - bug372b.t (Failed)
	 49 - bug372c.t (Failed)
	 50 - bug372d.t (Failed)
	 51 - bug4.t (Failed)
	 56 - canon2.t (Failed)
	 59 - cconv.t (SEGFAULT)
	 60 - cconv_array.t (Failed)
	 61 - cconv_more.t (Failed)
	 64 - clanginfo.t (Failed)
	 65 - class.t (Failed)
	 66 - class2.t (Failed)
	 67 - class3.t (Failed)
	 69 - class5.t (Failed)
	 70 - class6.t (Failed)
	 72 - clean.t (Failed)
	 73 - cmpxchg.t (Failed)
	 77 - compile_time_array.t (Failed)
	 78 - compile_time_array2.t (Failed)
	 79 - compile_time_array3.t (Failed)
	 90 - coverage3.t (Failed)
	119 - defer.t (Failed)
	120 - deferbreak.t (Failed)
	123 - defergoto.t (Failed)
	125 - dgemm.t (Failed)
	126 - dgemm2.t (Failed)
	127 - dgemm3.t (Failed)
	129 - diffuse.t (Failed)
	130 - dynlib.t (Failed)
	144 - exportdynamic.t (Failed)
	272 - fakeasm.t (Failed)
	299 - gemm.t (Failed)
	301 - gettype.t (Failed)
	305 - gvarfault.t (Failed)
	307 - hello.t (Failed)
	308 - hello2.t (Failed)
	312 - includec.t (Failed)
	334 - linklibrary.t (Failed)
	341 - luaapi.t (Failed)
	342 - luabridge.t (Failed)
	343 - luabridge2.t (Failed)
	358 - malloc.t (Failed)
	359 - mathlib.t (Failed)
	370 - multiterra.t (Failed)
	376 - new.t (Failed)
	387 - objc.t (Failed)
	388 - objc2.t (Failed)
	389 - objtest.t (Failed)
	392 - option_e.t (Failed)
	395 - output.t (Failed)
	423 - printfarray.t (Failed)
	424 - printfloat.t (Failed)
	425 - proxystruct.t (Failed)
	427 - pthreads.t (Failed)
	435 - quote7.t (Failed)
	436 - quote8.t (Failed)
	437 - quote9.t (Failed)
	440 - quoteselect.t (Failed)
	445 - recstruct.t (Failed)
	448 - renaming.t (Failed)
	451 - scc.t (Failed)
	456 - sgemm-old.t (Failed)
	457 - sgemm.t (Failed)
	458 - sgemm3.t (Failed)
	459 - sgemmkernel.t (Failed)
	474 - speed.t (Failed)
	477 - ssimple.t (Failed)
	479 - stattest.t (Failed)
	480 - stdio.t (Failed)
	481 - stencil.t (Failed)
	482 - strerror.t (Failed)
	483 - string.t (Failed)
	496 - symbolmangling.t (Failed)
	518 - testminmax.t (Failed)
	519 - testrequire.t (Failed)
	520 - teststd.t (Failed)
	521 - testvector.t (Failed)
	533 - vararg.t (Failed)
	534 - varargcstring.t (Failed)
	541 - vecobj.t (Failed)

• ../build/bin/terra ./run:

=================
= FAILING tests
=================
sgemm.t
deferbreak.t
bug2.t
testvector.t
multiterra.t
sgemm3.t
cconv_array.t
canon2.t
diffuse.t
pthreads.t
varargcstring.t
speed.t
hello2.t
bug372.t
breaktest.t
cconv_more.t
luaapi.t
compile_time_array.t
malloc.t
gemm.t
strerror.t
vararg.t
avxhadd.t
luabridge2.t
clean.t
quoteselect.t
clanginfo.t
cconv.t
cmpxchg.t
blocking2-fixed.t
renaming.t
includec.t
ssimple.t
quote8.t
mathlib.t
benchmark_fannkuchredux.t
dynlib.t
bug372b.t
linklibrary.t
teststd.t
gvarfault.t
proxystruct.t
objc.t
objtest.t
dgemm2.t
new.t
bounce.t
class3.t
blocking3.t
recstruct.t
blocking.t
sgemmkernel.t
compile_time_array2.t
output.t
benchmark_dgemm.t
exportdynamic.t
objc2.t
ainline.t
luabridge.t
quote7.t
bf.t
atoi.t
bug4.t
atomicrmw.t
quote9.t
bug372c.t
bug.t
class6.t
defergoto.t
dgemm3.t
hello.t
class2.t
blocking2.t
coverage3.t
compile_time_array3.t
vecobj.t
class.t
gettype.t
fakeasm.t
sgemm-old.t
scc.t
testminmax.t
benchmark_nbody.t
arrayt.t
symbolmangling.t
defer.t
bug372d.t
option_e.t
stencil.t
stdio.t
printfarray.t
arrayt2.t
stattest.t
class5.t
printfloat.t
string.t
testrequire.t
dgemm.t
=================

451 tests passed. 98 tests failed.

Are the tests meant to all pass on any platform?

[elliottslaughter]

Hm, this failure rate makes me think you might have forgotten SDKROOT? Or something similar.

We pass 100% of tests on all x86 platforms: macOS, Linux, FreeBSD, and Windows. We also pass 100% of tests on PPC64le Linux.

On ARM64 Linux, the pass rate is around 96% (#565 (comment)). Even in emulation I get 95% pass rate (#565 (comment)).

I'm hoping that there's a simple fix (like SDKROOT) that gets you up to a similar pass rate.

[shoe42]

SDKROOT doesn't seem to fix it - not sure what's going on here. I'll see if I can dig a bit deeper, but at least some seem to be due to missing headers (with SDKROOT set):

stack traceback:
	[C]: in function 'registercfile'
	.../terra/src/terralib.lua:3529: in function 'includecstring'
	./lib/objc.t:10: in main chunk
	[C]: in function 'require'
	testrequire.t:9: in main chunk
<buffer>:1:10: fatal error: 'stdio.h' file not found
#include "stdio.h"
         ^~~~~~~~~
compilation of included c code failed

[elliottslaughter]

I can't explain why SDKROOT wouldn't be working, but that failure mode looks exactly like what I'd expect if SDKROOT weren't set.

You might be able to work around with:

export INCLUDE_PATH="$(xcrun --sdk macosx --show-sdk-path)/usr/include"

That's a different way of specifying the path and shouldn't be necessary if SDKROOT is set. But it might work if SDKROOT isn't.

You built with an LLVM from Homebrew, right? It may also be worth trying a vanilla LLVM (there's a build recipe here: https://github.com/terralang/terra#instructions-for-building-llvm-from-source).

[shoe42]

That seems to fix things up (manually setting INCLUDE_PATH) - just having SDKROOT set doesn't seem to let it find the headers. I'm using the prebuilt LLVM 13 from Homebrew (llvm@13). Pass rate is now much better:

96% tests passed, 21 tests failed out of 549

Total Test time (real) =   4.98 sec

The following tests FAILED:
	 27 - atomicrmw.t (Subprocess aborted)
	 46 - bug372.t (SEGFAULT)
	 48 - bug372b.t (SEGFAULT)
	 49 - bug372c.t (SEGFAULT)
	 50 - bug372d.t (SEGFAULT)
	 56 - canon2.t (Failed)
	 59 - cconv.t (SEGFAULT)
	 60 - cconv_array.t (Failed)
	 61 - cconv_more.t (Failed)
	 65 - class.t (Failed)
	 67 - class3.t (SEGFAULT)
	 70 - class6.t (Failed)
	 90 - coverage3.t (Failed)
	272 - fakeasm.t (Failed)
	301 - gettype.t (Failed)
	342 - luabridge.t (Failed)
	343 - luabridge2.t (Failed)
	423 - printfarray.t (SEGFAULT)
	477 - ssimple.t (Failed)
	533 - vararg.t (Subprocess aborted)
	541 - vecobj.t (Failed)

[elliottslaughter]

Thanks. It's good to know we at least have a path to getting it mostly working.

I'd be surprised if SDKROOT parsing is really dependent on x86, but I suppose it's possible. (If so, probably an oversight.) My only other idea is to try a vanilla source build since Homebrew does sometimes add patches to their packages. (And again, instructions at https://github.com/terralang/terra#instructions-for-building-llvm-from-source .)

[elliottslaughter]

@shoe42 by the way, if you pull from master, you should (hopefully) see a higher test pass rate now that #593 has merged.

[shoe42]

@elliottslaughter apologies for the delay - from the current master branch (23559ea), using self-built LLVM 13 as per the instructions in the Terra docs, it still seems to fail to find the headers. Setting SDKROOT works as expected. Pass rate seems to be better:

$ ctest -j4
.....
98% tests passed, 10 tests failed out of 549

Total Test time (real) =  15.44 sec

The following tests FAILED:
	 27 - atomicrmw.t (Subprocess aborted)
	 46 - bug372.t (SEGFAULT)
	 48 - bug372b.t (SEGFAULT)
	 49 - bug372c.t (SEGFAULT)
	 60 - cconv_array.t (Failed)
	 65 - class.t (SEGFAULT)
	 67 - class3.t (SEGFAULT)
	272 - fakeasm.t (Failed)
	423 - printfarray.t (SEGFAULT)
	533 - vararg.t (Subprocess aborted)

[elliottslaughter]

Thanks for testing.

I should have mentioned #597. On Linux, this means that AArch64 only works with LLVM 11. I suspect this would help you fix most or all of these outstanding failures.