investigate cross-compilation with LLVM tools and without GNU binutils

[tpimh]

I'm not sure that cross compilation is possible with llvm tools at this point. I will research the topic and post my findings in this thread.

[nickdesaulniers]

I don't follow. All of Android is cross compiled via LLVM. Which parts are you referring to specifically? LLVM uses a combination of -triple <target triple> and --sysroot <path to cross crt0.o and friends> .

[tpimh]

Right now we are installing binutils-aarch64-linux-gnu, binutils-arm-linux-gnueabi, binutils-powerpc64le-linux-gnu. These packages provide tools specific for each platform with respective cross-prefixes. My idea is to replace these tools with the ones from llvm-8 package with "llvm-" prefix.
For example, instead of using objcopy, aarch64-linux-gnu-objcopy, arm-linux-gnueabi-objcopy, powerpc64le-linux-gnu-objcopy use llvm-objcopy for all platforms.

[nickdesaulniers]

@rupprecht is looking in to such incompatibilties as well. https://github.com/ClangBuiltLinux/linux/issues?q=is%3Aopen+is%3Aissue+label%3Allvm-tools is our list of known issues (separate from Clang, LLD, or integrated-as).

[tpimh]

In order to make sure there is no binutils installed I do dpkg -r --force-depends binutils binutils-common binutils-x86-64-linux-gnu libbinutils for x86_64, then apt remove binutils-* for all other platforms. At the moment it's not possible to build the kernel with clang's integrated as, so this is not very useful.
However, it's possible to use llvm-ar, llvm-nm, llvm-objdump, llvm-objcopy and llvm-strip.

[tpimh]

After very little modification (replacing about 10 lines of unsupported assembly with nops), I finally have built an x86_64 defconfig kernel without using GNU binutils. It obviously should not be bootable, but it was linked successfully.
Currently the issues are the following:

• expected a symbol reference in '.long' directive in include/trace/events/initcall.h linux#413
• error: expected a symbol reference in cpufeature.h (x86_64 with integrated-as) linux#575
• error: expected assembly-time absolute expression in cpufeature.h (x86_64 with integrated-as) linux#576

There may be other issues like invalid code generation, but they are very unlikely to be detected until a kernel is compiled without stubs.

[nickdesaulniers]

cc @gwelymernans

[tpimh]

Hooray! I have successfully build an arm32_v5 kernel without using GNU binutils: travis build log.

[nathanchance]

Awesome! Something like this should reveal if everything is hermetic:

nathanchance@0e2a26e

I suspect we are still implicitly using ld as the host linker when HOSTCC=clang for certain things, as that is what I ran into:

nathanchance@730028d

ClangBuiltLinux/linux#479

Otherwise, I'd expect only LLVM tools to be used.

[tpimh]

Thank you very much for your suggestions. I was sure that GNU binutils were not used during the build, but still wanted to remove them. Turned out the build was not actually hermetic!

[nickdesaulniers]

great find, filed ClangBuiltLinux/linux#766

[tpimh]

arm32_v5 successfull build with total 4 patches: https://travis-ci.com/ClangBuiltLinux/continuous-integration/jobs/254762048
x86_64 is not so successful; missing readelf, producing a lot of objtool warnings and not booting, definitely needs more investigation: https://travis-ci.com/ClangBuiltLinux/continuous-integration/jobs/254762056

[nickdesaulniers]

with total 4 patches

Which 4/where?

missing readelf,

arch/x86/boot/compressed/Makefile hardcodes readelf, similar to the nm issue on arm.

[tpimh]

These are the patches required for successful build for arm32_v5:

• fix-libelf-detection.patch (fix for HOSTCC always uses its default linker linux#479)
• asm-correct-mrc-operand.patch (fix for operand must be a register in range [r0, r14] or apsr_nzcv in arch/arm/boot/compressed/head.S linux#761)
• replace-solaris-style-asm.patch (fix for "error: expected string in directive" on ARM linux#744)
• use-nm-variable.patch (fix for arm build not hermetic nm linux#766)

[tpimh]

Right now x86_64 boots fine. I also have compiled a bootable ARM64 kernel with the following workarounds:

• Use STRIP=/usr/lib/llvm-11/bin/llvm-strip because of STUBCPY fails with llvm-strip linux#940:

llvm-strip-11: error: unknown argument '-o'

• disable CONFIG_KVM because of too many positional arguments in arm64 kvm linux#939:

arch/arm64/kvm/hyp/entry.S:112:87: error: too many positional arguments
 alternative_insn nop, .inst (0xd500401f | ((0) << 16 | (4) << 5) | ((!!1) << 8)), 4, 1
                                                                                      ^

• disable ARCH_TEGRA_210_SOC because of LLD + ThinLTO fails with "SHF_MERGE section size must be a multiple of sh_entsize" linux#510:

ld.lld-11: error: drivers/built-in.a(soc/tegra/fuse/speedo-tegra210.o):(.init.rodata): SHF_MERGE section size (56) must be a multiple of sh_entsize (16)