UnexpectedException test failing on ARM ABIs

[triplef]

This test was recently added in #220 and subsequently disabled on ARM (226455b) due to failing on the cross-build ARM CI targets.

I’m able to reproduce it on an arm64 Android emulator with this result in gdb:

Program received signal SIGABRT, Aborted.
0x0000007fbdee5f74 in abort () from target:/apex/com.android.runtime/lib64/bionic/libc.so
(gdb) bt
#0  0x0000007fbdee5f74 in abort () from target:/apex/com.android.runtime/lib64/bionic/libc.so
#1  0x0000007fbe2e7178 in objc_exception_throw (object=0x5555558ee0 <.objc_str_Exception>)
    at eh_personality.c:261
#2  0x0000005555556a4c in main ()
    at UnexpectedException.m:38

Any idea what might be going on here? Interestingly the exception hook is working fine in our app on Android ARM devices and has been for years.

Steps to reproduce with an Android emulator (on macOS ARM host, NDK paths may vary):

adb push libobjc.so /data/local/tmp
adb push Test/UnexpectedException /data/local/tmp
adb push $ANDROID_NDK_ROOT/sources/cxx-stl/llvm-libc++/libs/arm64-v8a/libc++_shared.so /data/local/tmp

# run test
adb shell LD_LIBRARY_PATH=/data/local/tmp/ /data/local/tmp/UnexpectedException

# debug with gdb
adb push $ANDROID_NDK_ROOT/prebuilt/android-arm64/gdbserver/gdbserver /data/local/tmp
adb forward tcp:5039 tcp:5039
adb shell
> cd /data/local/tmp
> LD_LIBRARY_PATH=$PWD ./gdbserver :5039 ./UnexpectedException

# on host machine
$ANDROID_NDK_ROOT/prebuilt/darwin-x86_64/bin/gdb
> target remote :5039
> c

[davidchisnall]

This abort is at the end of the throw function. The unwind library is returning from the unwind function, which happens only if unwinding fails. For some reason, it looks as if it is not reporting the end of the stack as the reason. Can you see what the value of err is? If you define DEBUG_EXCEPTIONS At the top of the file, it will log it (and a load of other info) for you.

[triplef]

These are the logs I get:

Exception caught by C++: 0
Throwing 0x5555558ee0
Throw returned -1115110992

(I commented out the log "Throwing %p, in flight exception: %p" because it doesn’t build as td->lastThrownObject doesn’t exist.)

[triplef]

Sometimes it also returns -1113013840, but seemingly always either that or more often -1115110992.

[davidchisnall]

Hmm, that's deeply strange. That looks like _Unwind_RaiseException is returning something that isn't a valid enum value, and possibly not even a valid integer. I've run this on FreeBSD/AArch64 (which uses the LLVM unwind library), and it passes.

Does Android use the GNU unwinder? Can you look in _Unwind_RaiseException and see if you can see what it thinks it's returning?

[triplef]

Interestingly when running without the debugger the err value seems to be random.

I’m not sure about which unwinder Android uses, and I haven’t been able to locate the sources for _Unwind_RaiseException so far. I’d also be interested to know whether the failure is the same using cross-builds like on the CI targets. If so maybe it’s easier to debug this there?

Unfortunately I don’t have time to dig into this deeper right now, but I wanted to at least document this issue here.

[hmelder]

Can you look in _Unwind_RaiseException and see if you can see what it thinks it's returning?

The error originates from the libgcc implementation of _Unwind_RaiseException and the generated asm code. Somehow all registers from the start of _Unwind_RaiseException are reloaded before returning _URC_END_OF_STACK in unwind.inc:108. As a result the first parameter passed gets returned.

Note that uw_frame_state_for (&cur_context, &fs); returns _URC_END_OF_STACK.

I just requested an account creation for GCC Bugzilla.

Dump of _Unwind_RaiseException

libgcc_s.so.1`_Unwind_RaiseException:
    0xfffff7f27700 <+0>:   sub    sp, sp, #0xc10
    0xfffff7f27704 <+4>:   stp    x29, x30, [sp]
    0xfffff7f27708 <+8>:   mov    x29, sp
    0xfffff7f2770c <+12>:  xpaclri 
    0xfffff7f27710 <+16>:  stp    x21, x22, [sp, #0x40]
    0xfffff7f27714 <+20>:  add    x22, sp, #0xc0
    0xfffff7f27718 <+24>:  add    x21, sp, #0x840
    0xfffff7f2771c <+28>:  stp    x0, x1, [sp, #0x10]
    0xfffff7f27720 <+32>:  add    x1, sp, #0xc10
    0xfffff7f27724 <+36>:  stp    x2, x3, [sp, #0x20]
    0xfffff7f27728 <+40>:  mov    x2, x30
    0xfffff7f2772c <+44>:  stp    x19, x20, [sp, #0x30]
    0xfffff7f27730 <+48>:  mov    x20, x0
    0xfffff7f27734 <+52>:  add    x19, sp, #0x480
    0xfffff7f27738 <+56>:  mov    x0, x22
    0xfffff7f2773c <+60>:  stp    x23, x24, [sp, #0x50]
    0xfffff7f27740 <+64>:  stp    x25, x26, [sp, #0x60]
    0xfffff7f27744 <+68>:  stp    x27, x28, [sp, #0x70]
    0xfffff7f27748 <+72>:  stp    d8, d9, [sp, #0x80]
    0xfffff7f2774c <+76>:  stp    d10, d11, [sp, #0x90]
    0xfffff7f27750 <+80>:  stp    d12, d13, [sp, #0xa0]
    0xfffff7f27754 <+84>:  stp    d14, d15, [sp, #0xb0]
    0xfffff7f27758 <+88>:  bl     0xfffff7f27020 ; uw_init_context_1 at unwind-dw2.c:1324:1
    0xfffff7f2775c <+92>:  mov    x1, x22
    0xfffff7f27760 <+96>:  mov    x0, x19
    0xfffff7f27764 <+100>: mov    x2, #0x3c0 ; =960 
    0xfffff7f27768 <+104>: bl     0xfffff7f13740
    0xfffff7f2776c <+108>: b      0xfffff7f277a0 ; <+160> at unwind.inc:104:14
    0xfffff7f27770 <+112>: cbnz   w2, 0xfffff7f27814 ; <+276> at unwind.inc:113:9
    0xfffff7f27774 <+116>: ldr    x5, [sp, #0xbe0]
    0xfffff7f27778 <+120>: cbz    x5, 0xfffff7f27794 ; <+148> at unwind.inc:127:7
    0xfffff7f2777c <+124>: ldr    x2, [x20]
    0xfffff7f27780 <+128>: blr    x5
    0xfffff7f27784 <+132>: cmp    w0, #0x6
    0xfffff7f27788 <+136>: b.eq   0xfffff7f2781c ; <+284> at unwind.inc:132:18
    0xfffff7f2778c <+140>: cmp    w0, #0x8
    0xfffff7f27790 <+144>: b.ne   0xfffff7f27814 ; <+276> at unwind.inc:113:9
    0xfffff7f27794 <+148>: mov    x1, x21
    0xfffff7f27798 <+152>: mov    x0, x19
    0xfffff7f2779c <+156>: bl     0xfffff7f27264 ; uw_update_context at unwind-dw2.c:1266:1
    0xfffff7f277a0 <+160>: mov    x1, x21
    0xfffff7f277a4 <+164>: mov    x0, x19
    0xfffff7f277a8 <+168>: bl     0xfffff7f25f00 ; uw_frame_state_for at unwind-dw2.c:997:3
    0xfffff7f277ac <+172>: mov    w2, w0
    0xfffff7f277b0 <+176>: mov    w1, #0x1 ; =1 
    0xfffff7f277b4 <+180>: mov    x4, x19
    0xfffff7f277b8 <+184>: mov    x3, x20
    0xfffff7f277bc <+188>: mov    w0, w1
    0xfffff7f277c0 <+192>: cmp    w2, #0x5
    0xfffff7f277c4 <+196>: b.ne   0xfffff7f27770 ; <+112> at unwind.inc:110:10
->  0xfffff7f277c8 <+200>: mov    x4, #0x0 ; =0 
    0xfffff7f277cc <+204>: mov    w0, w2
    0xfffff7f277d0 <+208>: ldp    x29, x30, [sp]
    0xfffff7f277d4 <+212>: ldp    x0, x1, [sp, #0x10]
    0xfffff7f277d8 <+216>: ldp    x2, x3, [sp, #0x20]
    0xfffff7f277dc <+220>: ldp    x19, x20, [sp, #0x30]
    0xfffff7f277e0 <+224>: ldp    x21, x22, [sp, #0x40]
    0xfffff7f277e4 <+228>: ldp    x23, x24, [sp, #0x50]
    0xfffff7f277e8 <+232>: ldp    x25, x26, [sp, #0x60]
    0xfffff7f277ec <+236>: ldp    x27, x28, [sp, #0x70]
    0xfffff7f277f0 <+240>: ldp    d8, d9, [sp, #0x80]
    0xfffff7f277f4 <+244>: ldp    d10, d11, [sp, #0x90]
    0xfffff7f277f8 <+248>: ldp    d12, d13, [sp, #0xa0]
    0xfffff7f277fc <+252>: ldp    d14, d15, [sp, #0xb0]
    0xfffff7f27800 <+256>: add    sp, sp, #0xc10
    0xfffff7f27804 <+260>: cbz    x4, 0xfffff7f27810 ; <+272> at unwind.inc:141:1
    0xfffff7f27808 <+264>: add    sp, sp, x5
    0xfffff7f2780c <+268>: br     x6
    0xfffff7f27810 <+272>: ret    
    0xfffff7f27814 <+276>: mov    w2, #0x3 ; =3 
    0xfffff7f27818 <+280>: b      0xfffff7f277c8 ; <+200> at unwind.inc:141:1
    0xfffff7f2781c <+284>: str    xzr, [x20, #0x10]
    0xfffff7f27820 <+288>: mov    x0, x19
    0xfffff7f27824 <+292>: bl     0xfffff7f13860 ; symbol stub for: pthread_key_create
    0xfffff7f27828 <+296>: mov    x4, x0
    0xfffff7f2782c <+300>: ldr    x3, [sp, #0x7c0]
    0xfffff7f27830 <+304>: mov    x1, x22
    0xfffff7f27834 <+308>: mov    x2, #0x3c0 ; =960 
    0xfffff7f27838 <+312>: mov    x0, x19
    0xfffff7f2783c <+316>: sub    x3, x4, x3, lsr #63
    0xfffff7f27840 <+320>: str    x3, [x20, #0x18]
    0xfffff7f27844 <+324>: bl     0xfffff7f13740
    0xfffff7f27848 <+328>: mov    x2, x21
    0xfffff7f2784c <+332>: mov    x1, x19
    0xfffff7f27850 <+336>: mov    x0, x20
    0xfffff7f27854 <+340>: bl     0xfffff7f273ac ; _Unwind_RaiseException_Phase2 at unwind.inc:41:1
    0xfffff7f27858 <+344>: mov    w2, #0x2 ; =2 
    0xfffff7f2785c <+348>: cmp    w0, #0x7
    0xfffff7f27860 <+352>: b.ne   0xfffff7f277c8 ; <+200> at unwind.inc:141:1
    0xfffff7f27864 <+356>: mov    x1, x19
    0xfffff7f27868 <+360>: mov    x0, x22
    0xfffff7f2786c <+364>: bl     0xfffff7f24a60 ; uw_install_context_1 at unwind-dw2.c:1405:1
    0xfffff7f27870 <+368>: mov    x19, x0
    0xfffff7f27874 <+372>: ldr    x20, [sp, #0x798]
    0xfffff7f27878 <+376>: ldr    x0, [sp, #0x790]
    0xfffff7f2787c <+380>: mov    x1, x20
    0xfffff7f27880 <+384>: bl     0xfffff7f276f0 ; _Unwind_DebugHook at unwind-dw2.c:1382:1
    0xfffff7f27884 <+388>: bl     0xfffff7f2af40 ; __arm_za_disable
    0xfffff7f27888 <+392>: mov    x5, x19
    0xfffff7f2788c <+396>: mov    x6, x20
    0xfffff7f27890 <+400>: mov    x4, #0x1 ; =1 
    0xfffff7f27894 <+404>: b      0xfffff7f277cc ; <+204> at unwind.inc:141:1

Current position right after if (code == _URC_END_OF_STACK).
The return value of uw_frame_state_for is 5 (_URC_END_OF_STACK).

      if (code == _URC_END_OF_STACK)
	/* Hit end of stack with no handler found.  */
	return _URC_END_OF_STACK;

After mov w0, w2 (-> 0xfffff7f277d0 <+208>: ldp x29, x30, [sp])

0xfffff7f27894 <+404>: b      0xfffff7f277cc ; <+204> at unwind.inc:141:1

(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000005
        x1 = 0x0000000000000001
        x2 = 0x0000000000000005
        x3 = 0x0000aaaaaaafa790
        x4 = 0x0000000000000000
        x5 = 0x0000ffffffffe160
        x6 = 0xfffffffffffffff8
        x7 = 0x0000000000000004
        x8 = 0x0000000000000001
        x9 = 0x0000fffff7f402a8
       x10 = 0x0000000000000000
       x11 = 0x0000fffff7f40308
       x12 = 0x0000fffff7ff77c0
       x13 = 0x0000000000000010
       x14 = 0x0000000000000000
       x15 = 0x0000fffff7bc63c0  
       x16 = 0x0000fffff7f40000
       x17 = 0x0000000000000000
       x18 = 0x0000000000000007
       x19 = 0x0000ffffffffe610
       x20 = 0x0000aaaaaaafa790
       x21 = 0x0000ffffffffe9d0
       x22 = 0x0000ffffffffe250
       x23 = 0x0000ffffffffefc8
       x24 = 0x0000fffff7ffdb90  ld-linux-aarch64.so.1`_rtld_global_ro
       x25 = 0x0000000000000000
       x26 = 0x0000fffff7ffe008  _rtld_global
       x27 = 0x0000aaaaaaabfda0  UnexpectedExceptionDebug`__do_global_dtors_aux_fini_array_entry
       x28 = 0x0000000000000000
        fp = 0x0000ffffffffe190
        lr = 0x0000fffff7f277ac  libgcc_s.so.1`_Unwind_RaiseException + 172 at unwind.inc:104:14
        sp = 0x0000ffffffffe190
        pc = 0x0000fffff7f277d0  libgcc_s.so.1`_Unwind_RaiseException + 208 at unwind.inc:141:1
      cpsr = 0x60201000

After ldp x0, x1

(lldb) register read
General Purpose Registers:
        x0 = 0x0000aaaaaaafa790
        x1 = 0x0000000000000000
        x2 = 0x0000000000000058
        x3 = 0x0000000000000000
        x4 = 0x0000000000000000
        x5 = 0x0000ffffffffe160
        x6 = 0xfffffffffffffff8
        x7 = 0x0000000000000004
        x8 = 0x0000000000000001
        x9 = 0x0000fffff7f402a8
       x10 = 0x0000000000000000
       x11 = 0x0000fffff7f40308
       x12 = 0x0000fffff7ff77c0
       x13 = 0x0000000000000010
       x14 = 0x0000000000000000
       x15 = 0x0000fffff7bc63c0  
       x16 = 0x0000fffff7f40000
       x17 = 0x0000000000000000
       x18 = 0x0000000000000007
       x19 = 0x0000ffffffffefb8
       x20 = 0x0000000000000001
       x21 = 0x0000aaaaaaabfda0  UnexpectedExceptionDebug`__do_global_dtors_aux_fini_array_entry
       x22 = 0x0000aaaaaaaa0ae8  UnexpectedExceptionDebug`main at UnexpectedException.m:33
       x23 = 0x0000ffffffffefc8
       x24 = 0x0000fffff7ffdb90  ld-linux-aarch64.so.1`_rtld_global_ro
       x25 = 0x0000000000000000
       x26 = 0x0000fffff7ffe008  _rtld_global
       x27 = 0x0000aaaaaaabfda0  UnexpectedExceptionDebug`__do_global_dtors_aux_fini_array_entry
       x28 = 0x0000000000000000
        fp = 0x0000ffffffffee00
        lr = 0x0000fffff7f76d38  libobjc.so.4.6`objc_exception_throw + 520 at eh_personality.c:256:22
        sp = 0x0000ffffffffeda0
        pc = 0x0000fffff7f27810  libgcc_s.so.1`_Unwind_RaiseException + 272 at unwind.inc:141:1
      cpsr = 0x60201000
(lldb) 

[hmelder]

See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114843

[davidchisnall]

Thanks for root causing this! Let's leave this open until there's a fix, but until then we can recommend that Arm users use LLVM's unwinder instead of GCC's.

[triplef]

How does one choose between the LLVM vs. GCC unwinder, and is the GCC one also ever used when building with Clang?

I’m a bit confused about this issue because we don’t see issues with exception handling on Android in our app (built with Clang and the NDK toolchain) and the exception hook is working fine too, but I was originally able to reproduce this with the test using the NDK toolchain.

[hmelder]

How does one choose between the LLVM vs. GCC unwinder, and is the GCC one also ever used when building with Clang?

Here is an example:

clang-18 test.c -o test -rtlib=compiler-rt --unwindlib=libunwind -fuse-ld=lld-18

[hmelder]

but I was originally able to reproduce this with the test using the NDK toolchain

Yep thats weird. Do you know the NDK version you ran the test on?

[davidchisnall]

I think Android uses the LLVM unwinder. It's typically integrated in the C support files, but precisely where depends on the platform. FreeBSD uses the LLVM one as well, which is why I didn't see these issues.

It's also not always clear which one you're using. On FreeBSD, we ship the LLVM one but with the same name as the GCC one to avoid configure scripts failing.

I'd report this to distros as a bug and tell them that the simple fix is to use the LLVM one instead of the GCC one.

[hmelder]

Just tested it with libunwind-18 on Ubuntu 23.10 and it works as expected:

hugo@ubuntu:/tmp$ clang -L/usr/lib/llvm-18/lib test.c -o test -rtlib=compiler-rt --unwindlib=libunwind -fuse-ld=lld -lunwind
hugo@ubuntu:/tmp$ ./test
RaiseException returned 0x5

I'll test it on Android this evening.

[triplef]

Do you know the NDK version you ran the test on?

Unfortunately no. I think the NDK migrated to the LLVM toolchain over the last couple of releases, so it might have been a release that was still using the GCC unwinder.

Can we detect in CMake which unwinder is used and already enable the test when LLVM is used?

[triplef]

Just found this re. unwinder on Android:

The unwinder APIs are exposed from the platform's libc.so starting with API 30 (Android R).

[hmelder]

Can we detect in CMake which unwinder is used and already enable the test when LLVM is used?

I would enable it unconditionally. The libgcc patch will probably be backported to gcc 13 and maybe gcc 12.

[hmelder]

Just found this re. unwinder on Android:

The unwinder APIs are exposed from the platform's libc.so starting with API 30 (Android R).

Seems like libgcc was removed in NDK r23. Here is the change in clang: https://reviews.llvm.org/D96403

[pinskia]

I would enable it unconditionally. The libgcc patch will probably be backported to gcc 13 and maybe gcc 12.

s/libgcc/gcc/. The bug is not in libgcc directly but rather the code that GCC produces has the bug.

I hope to get it backported in time for the GCC 11.5 release but we will see. I will be posting the patch over the weekend and I doubt it will be reviewed until Monday or later.

[pinskia]

I should note that a few other targets has a similar bug:
powerpc: PR 114846
arm: PR 114847

loongarch had the similar bug but it was fixed in GCC 14: longarch: PR 114848

Those are the "major" targets I tried that had the bug.

[pinskia]

Just an FYI I have posted the GCC patch: https://gcc.gnu.org/pipermail/gcc-patches/2024-April/650080.html .

[hmelder]

Thank you @pinskia!