ARM64 msys2-runtime

[ZacWalk]

I would like to help get the BASH shell running natively on 'Windows on ARM64'. I see a key dependency will be the msys2-runtime.

I have been able to build various packages for ARM64 using these instructions. I would like to experiment with building newlib-cygwin for ARM using clang; to evaluate how much work is needed.

I was wondering if anyone had already tried this or had some recommendations for setting up a local build. I hope to be able to bring ARM specific code over from newlib where needed.

FYI - @jeremyd2019 @lazka

[mati865]

You'd need to start with adding Cygwin support to LLVM projects first. The target you are using right now is native Windows target buy you need Cygwin target.

There is some support for i686 and x86_64 Cygwin but I don't know if it's still maintained. Starting with x86_64 should be the easiest way to see how much work will it take to build Cygwin with LLVM.

Adding AArch64 Cygwin target will need some work but hopefully it's not much. Porting Cygwin for AArch64 will be bigger task though.

[ZacWalk]

To help me understand. I see that 'grep' has two different packages:

1. An MSYS2 package: MSYS2-packages/grep that depends on MSYS-2.0.dll.
2. MINGW package that does not need MSYS-2.0.dll - and can be built with clang already.

Would a good starting point be: add a 'clang64' arch to the MSYS/Cygwin package and see what goes wrong when building with clang? That should drive out various LLVM missing parts for x86_64? (as a first step before building for ARM)

[mati865]

Would a good starting point be: add a 'clang64' arch to the MSYS/Cygwin package and see what goes wrong when building with clang?

No:

1. we don't even have properly working x86_64 Clang MSYS/package.
2. clang64 is construct purely for MINGW packages, that's why it's in mingw_arch. You'd add aarch64 to arch.
3. I'm 99% sure msys2-runtime won't build even with working Clang on supported architecture like x86_64.

That should drive out various LLVM missing parts for x86_64? (as a first step before building for ARM)

You'd need to somehow obtain working LLVM toolchain (without GCC/Binutils dependencies) targeting x86_64 Cygwin first.

[jon-turney]

3. I'm 99% sure msys2-runtime won't build even with working Clang on supported architecture like x86_64.

Yeah, building cygwin with clang is something nobody is doing currently AFAIK. Although it's probably possible to fix any gcc-isms.

[dscho]

@ZacWalk FWIW Cygwin does not use clang but a custom GCC-based toolchain. And here is the most authoritative answer by the Cygwin project that I found to the question when Cygwin will support Windows/ARM64: https://www.mail-archive.com/cygwin@cygwin.com/msg169879.html

• Toolchains (binutils, gcc, gdb) targeting aarch64-pc-cygwin and aarch64-w64-mingw32.

  I don't know if the mingw toolchain already exists, but if so, the Cygwin toolchain could be derived from there. The differences would
  be basically the same as on x86_64 (sizeof(long), different default link libs, etc)

• Make sure Mingw-w64 for ARM is working as desired, including the special code for Cygwin in there. AFAIK, Mingw-w64 for ARM64 is still in an experimental stage, but I could be wrong.

• Make sure to understand the differences between ARM64 ABI on Windows vs. the default aarch64 ABI (if any).

• At least one person who's fluent in aarch64 assembler who will take a stab at creating matching aarch64 assembler code snippets in places we have x86 and x86_64 assembler today. We might get away with some of the code being in newlib already (memset/memcpy).

• ARM64 Windows and hardware for testing.

I don't see anything of that happen any time soon ¯_(ツ)_/¯

I could imagine that maybe you (or your manager) might be able to do something about the hardware for testing? 😁

[jon-turney]

• Toolchains (binutils, gcc, gdb) targeting aarch64-pc-cygwin and aarch64-w64-mingw32.

I don't think there's any fundamental objection to using a clang-based toolchain for aarch64-cygwin, but 'toolchain needs to exist' is still the problem.

Incidentially, there was some recent work [1] which claims to add aarch64-pe support to binutils.

[1] https://sourceware.org/pipermail/binutils/2021-December/118998.html

[jon-turney]

• Toolchains (binutils, gcc, gdb) targeting aarch64-pc-cygwin and aarch64-w64-mingw32.
  I don't know if the mingw toolchain already exists, but if so, the Cygwin toolchain could be derived from there. The differences would
  be basically the same as on x86_64 (sizeof(long), different default link libs, etc)

See https://cygwin.com/faq.html#faq.programming.64bitporting for a bit more context on the sizeof(long) issue.

[ZacWalk]

Thanks for that input. I will spend some time investigating those 2 approaches:

1. Try building the MSYS-runtime with clang
2. Investigate aarch64-pe support in binutils

I am still learning about the build. My expectation is that clang's WoARM64 support is a little more battle hardened and might lead to better cross compiling options in future. Windows ARM boxes seem quite slow for builds long term.

[mati865]

Try building the MSYS-runtime with clang

You need to create working LLVM toolchain (Clang, compiler-rt, libc++abi, libc++, libunwind, LLD, LLVM) with Cygwin or MSYS target first.

Investigate aarch64-pe support in binutils

It doesn't help at all unless GCC also gains possibility to target AArch64 Windows.

[ZacWalk]

Sorry for the silly question: I was expecting that I could use the mingw-w64-cross-clang-toolchain package as a toolchain.

As an experiment I tried:

pacman -S git base-devel mingw-w64-cross-clang-toolchain autotools 

export CC=clang     
export CC=clang 
export CXX=clang++ 
export LD=lld 
export CHOST=x86_64-w64-mingw32 
export MINGW_CXX=x86_64-w64-mingw32-clang++ 
export MINGW_CC=x86_64-w64-mingw32-clang 
export MINGW32_CC=i686-w64-mingw32-clang 
export MINGW64_CC=x86_64-w64-mingw32-clang

(cd ~/msys2-runtime/winsup && ./autogen.sh)

~/msys2-runtime/configure  --prefix=/usr  --build=${CHOST} --sysconfdir=/etc 

cd ~/msys2-runtime/
LC_ALL=C make

This seemed to configure ok and start a build using clang. The build eventually failed with some compiler command line issues. However a large number of C files did build. I did not get as far as linking.

@mati865 is this a dead end?

[jon-turney]

Sorry for the silly question: I was expecting that I could use the mingw-w64-cross-clang-toolchain package as a toolchain.

Well, you can use it to explore if cygwin can be built with clang, at all.

But you seem to be ignoring hard part of the problem, which is causing the needed toolchain for aarch64-cygwin to exist.

[mati865]

This seemed to configure ok and start a build using clang. The build eventually failed with some compiler command line issues. However a large number of C files did build. I did not get as far as linking.

@mati865 is this a dead end?

This toolchain is the reason why I emphasized you need working LLVM toolchain.
I created hacked it together for only one purpose: to bootstrap LLVM toolchains for CLANG* subsystems.
Problems with it:

• was (and probably still is) crashing a lot with C++ files
• is very outdated and there are a lot of fixes (especially for AArch64) merged in newer releases
• there were many things I didn't bother to fix here because they were not necessary for bootstrapping CLANG* subsystems

Of course you can play with it but sooner or later you will encounter issues.

[Biswa96]

Wasn't there a Fedora repository to cross compile cygwin gcc from Linux? It may help with this porting.

[mati865]

How would it help?
We already have working MSYS GCC. The problem with GCC is it doesn't support AArch64 Windows at all.

[jon-turney]

I guess you are referring to https://copr.fedorainfracloud.org/coprs/yselkowitz/cygwin/

But this is (should be) built from the same gcc sources as a self-hosted cygwin gcc.

[ZacWalk]

I have taken some time to try and understand the choices between:

1. Adding arm64 support to the GCC toolchain
2. Build a new toolchain around clang

There seems a lot of behavior in the existing GCC toolchain that has been built up over a long period. I assume that would be hard to replicate in clang even if the cygwin-runtime could be built.

Does that make it a no-brainer to focus on GCC? Or are there other factors like a desire to move to clang over time?

[ZacWalk]

Quick update on this:

@jon-turney I have reached out to the author of the binutils aarch64-pe patch you mentioned above. This lead to discussions with various GCC contributors at my employer (Microsoft) on how we could help get WoARM64 support into binutils. One of the challenges is getting approval for developer time and an ongoing CI/Testing commitment. My hope is to focus my effort on GCC/Binutils and then to move back to msys2-runtime. Keep you posted.

[jeremyd2019]

Do you happen to know who I could talk to at Microsoft about using WoARM64 in CI/CD applications, for clangarm64? I have been contacting about every cloud I can find that does aarch64 instances about the potential for getting some cloud compute muscle, but even if I get the capacity and even if I get technical issues sorted out so that it actually runs, figuring out the licensing is entirely outside my expertise.

[jeremyd2019]

Something occurred to me that may be a deal-breaker for a native aarch64 cygwin (or msys2). Dynamicbase (ASLR) is required to be set on arm64. Cygwin relies on deterministic address space layout for its fork emulation, so requires dynamicbase to not be set.

Best reference I could find just now was https://learn.microsoft.com/en-us/cpp/build/reference/dynamicbase-use-address-space-layout-randomization?view=msvc-170

Because ASLR can't be disabled on ARM, ARM64, or ARM64EC architectures, /DYNAMICBASE:NO isn't supported for these targets.

[mati865]

Ouch, I had thought Cygwin was eventually able to fix compatibility with ASLR.
This is showstopper then until somebody makes Cygwin work with ASLR.

[jon-turney]

Dynamicbase (ASLR) is required to be set on arm64. Cygwin relies on deterministic address space layout for its fork emulation, so requires dynamicbase to not be set.

There is some problem which ASLR causes, but I'm not sure that's a completely correct statement of the issue: I had the impression that ALSR doen't make a DLL have a different address in every process it's loaded into (which would be fatal to fork emulation) (I can think of a couple of reasons why that wouldn't be a good idea, but I don't know if that's documented behaviour or not...)

I'll see if i can get some more information from people who have more knowledge than me about this.

[lazka]

I remember looking for docs to this, and could not find anything definitive for this problem:

DLL A is loaded in Process X, DLL B in Process Y, both happen to get mapped to the same address space (possible?), what happens if both are loaded in Process Z while X and Y are still running.

(I might be missing something though...)

[jeremyd2019]

When DLLs are dynamicbase, it actually works out mostly alright for cygwin. As I recall, the OS keeps a global per-boot layout of where DLLs were loaded and keeps them in the same place in different processes. If the pool in which dynamicbase DLLs can be loaded gets used up during a boot, though, it falls back to different behavior like what happens when two DLLs are supposed to be loaded in the same space, which results in them being loaded in different places in different processes. I would hope this wouldn't happen with 64-bit address spaces but I think it may be possible if a lot of DLLs are build and loaded on the same boot.

What breaks more definitively is when an EXE is dynamicbase. This seems to quickly break fork emulation, as I discovered when a broken binutils patch resulted in the flag accidentally being set for cygwin PE files. This causes other things like the stack to be located at different addresses between processes.

[jon-turney]

Corinna did some work, so as of Cygwin 3.4.0, ASLR should work.

• You need to use a relatively recent binutils (>=2.36?) for executables to be made with relocations
• Executables marked for high-entropy ASLR break Cygwin's assumptions about address space layout in some way I'm not clever enough to understand

[ZacWalk]

Another one of my very occasional updates: I have been pulling various parts together to make an experimental GNU toolchain.

This allows cross-compiling from Linux to Windows-on-arm64. However, it has some key parts still missing:

• Exception handling and stack unwinding are WIP.
• Probably bugs in supporting the Windows ABI for arm64 correctly.
• Debugging via cv2pdb seems to work but probably has bugs.
• Other various things not even started – for example, atomics.

I am slowly working on Exception handling and stack unwinding. If I can get that working then this toolchain might become useful.

I can't take much credit. I am building on work from several other contributors.