clang++ 20 Exceptions crash with (recent) libgcc and aarch64 BTI (Linux)

[ozbenh]

Reproduced using clang15 and 18 in Amazon Linux 2023 and clang20 in Fedora42

When building a program with BTI (typically in my case -mbranch-protection=standard, the program crashes in C++ exception processing.

I've pinpointed the cause to clang not putting a bti instruction at the beginning of the exception "catching" code. Older libgcc (I tested gcc11) work fine because they use a ret instruction at the end of __builtin_eh_return() when jumping to the catch handler. With gcc 14 (at least using the version in Amazon Linux 2023) however I get a br instruction which expects the destination to have the appropriate bti instruction.

It works fine with g++

A simple test program demonstrates the issue:

#include <iostream>

int main()
{
	std::cout << "Foo\n";
	try {
		throw 1;
	} catch (int a) 
	{
		std::cout << "Bar\n";
	}
	return 0;
}

Compile with:

$ clang++ -O0 -g2 -mbranch-protection=standard -o test test.cpp`
$ ./test
Foo
Illegal instruction (core dumped)
$ ./test
Foo
Bar

You can see in the generated assembly:

000000000041030c <main>:
  41030c:       d503233f        paciasp
  410310:       d100c3ff        sub     sp, sp, #0x30
  410314:       a9027bfd        stp     x29, x30, [sp, #32]
  410318:       910083fd        add     x29, sp, #0x20
  41031c:       b81fc3bf        stur    wzr, [x29, #-4]
  410320:       90000180        adrp    x0, 440000 <__cxa_begin_catch@CXXABI_1.3>
  410324:       91020000        add     x0, x0, #0x80
  410328:       90000081        adrp    x1, 420000 <_IO_stdin_used>
  41032c:       91004021        add     x1, x1, #0x10
  410330:       97ffff56        bl      410088 <_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc@plt>
  410334:       d2800080        mov     x0, #0x4                        // #4
  410338:       97ffff48        bl      410058 <__cxa_allocate_exception@plt>
  41033c:       52800028        mov     w8, #0x1                        // #1
  410340:       b9000008        str     w8, [x0]
  410344:       f0000161        adrp    x1, 43f000 <GCC_except_table1+0x1ee20>
  410348:       9136a021        add     x1, x1, #0xda8
  41034c:       aa1f03e2        mov     x2, xzr
  410350:       97ffff72        bl      410118 <__cxa_throw@plt>
  410354:       14000022        b       4103dc <_fini>
  __cxa_throw ends up jumping here:
  410358:       2a0103e8        mov     w8, w1     <---- not a BTI instruction, boom !
  41035c:       f9000be0        str     x0, [sp, #16]
  410360:       b9000fe8        str     w8, [sp, #12]
  410364:       14000001        b       410368 <main+0x5c>

 .../...

The same code generated by gcc (11 in this case):

000000000041028c <main>:
  41028c:       d503233f        paciasp
  410290:       a9bd7bfd        stp     x29, x30, [sp, #-48]!
  410294:       910003fd        mov     x29, sp
  410298:       f9000bf3        str     x19, [sp, #16]
  41029c:       90000080        adrp    x0, 420000 <_IO_stdin_used>
  4102a0:       91004001        add     x1, x0, #0x10
  4102a4:       90000180        adrp    x0, 440000 <__cxa_begin_catch@CXXABI_1.3>
  4102a8:       91020000        add     x0, x0, #0x80
  4102ac:       97ffff77        bl      410088 <_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc@plt>
  4102b0:       d2800080        mov     x0, #0x4                        // #4
  4102b4:       97ffff69        bl      410058 <__cxa_allocate_exception@plt>
  4102b8:       52800021        mov     w1, #0x1                        // #1
  4102bc:       b9000001        str     w1, [x0]
  4102c0:       d2800002        mov     x2, #0x0                        // #0
  4102c4:       f0000161        adrp    x1, 43f000 <__FRAME_END__+0x1ee50>
  4102c8:       9136a021        add     x1, x1, #0xda8
  4102cc:       97ffff93        bl      410118 <__cxa_throw@plt>
  __cxa_throw ends up jumping here:
  4102d0:       d503249f        bti     j  <--- here's our BTI
  4102d4:       f100043f        cmp     x1, #0x1
  4102d8:       54000040        b.eq    4102e0 <main+0x54>  // b.none
  4102dc:       97ffff95        bl      410130 <_Unwind_Resume@plt>
  .../...

[llvmbot]

@llvm/issue-subscribers-backend-aarch64

Author: Benjamin Herrenschmidt (ozbenh)

Reproduced using clang15 and 18 in Amazon Linux 2023 and clang20 in Fedora42

When building a program with BTI (typically in my case -mbranch-protection=standard, the program crashes in C++ exception processing.

I've pinpointed the cause to clang not putting a bti instruction at the beginning of the exception "catching" code. Older libgcc (I tested gcc11) work fine because they use a ret instruction at the end of __builtin_eh_return() when jumping to the catch handler. With gcc 14 (at least using the version in Amazon Linux 2023) however I get a br instruction which expects the destination to have the appropriate bti instruction.

It works fine with g++

A simple test program demonstrates the issue:

#include <iostream>

int main()
{
	std::cout << "Foo\n";
	try {
		throw 1;
	} catch (int a) 
	{
		std::cout << "Bar\n";
	}
	return 0;
}

Compile with:

$ clang++ -O0 -g2 -mbranch-protection=standard -o test test.cpp`
$ ./test
Foo
Illegal instruction (core dumped)
$ ./test
Foo
Bar

You can see in the generated assembly:

000000000041030c <main>:
  41030c:       d503233f        paciasp
  410310:       d100c3ff        sub     sp, sp, #<!-- -->0x30
  410314:       a9027bfd        stp     x29, x30, [sp, #<!-- -->32]
  410318:       910083fd        add     x29, sp, #<!-- -->0x20
  41031c:       b81fc3bf        stur    wzr, [x29, #-4]
  410320:       90000180        adrp    x0, 440000 &lt;__cxa_begin_catch@<!-- -->CXXABI_1.3&gt;
  410324:       91020000        add     x0, x0, #<!-- -->0x80
  410328:       90000081        adrp    x1, 420000 &lt;_IO_stdin_used&gt;
  41032c:       91004021        add     x1, x1, #<!-- -->0x10
  410330:       97ffff56        bl      410088 &lt;_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc@<!-- -->plt&gt;
  410334:       d2800080        mov     x0, #<!-- -->0x4                        // #<!-- -->4
  410338:       97ffff48        bl      410058 &lt;__cxa_allocate_exception@<!-- -->plt&gt;
  41033c:       52800028        mov     w8, #<!-- -->0x1                        // #<!-- -->1
  410340:       b9000008        str     w8, [x0]
  410344:       f0000161        adrp    x1, 43f000 &lt;GCC_except_table1+0x1ee20&gt;
  410348:       9136a021        add     x1, x1, #<!-- -->0xda8
  41034c:       aa1f03e2        mov     x2, xzr
  410350:       97ffff72        bl      410118 &lt;__cxa_throw@<!-- -->plt&gt;
  410354:       14000022        b       4103dc &lt;_fini&gt;
  __cxa_throw ends up jumping here:
  410358:       2a0103e8        mov     w8, w1     &lt;---- not a BTI instruction, boom !
  41035c:       f9000be0        str     x0, [sp, #<!-- -->16]
  410360:       b9000fe8        str     w8, [sp, #<!-- -->12]
  410364:       14000001        b       410368 &lt;main+0x5c&gt;

 .../...

The same code generated by gcc (11 in this case):

000000000041028c <main>:
  41028c:       d503233f        paciasp
  410290:       a9bd7bfd        stp     x29, x30, [sp, #-48]!
  410294:       910003fd        mov     x29, sp
  410298:       f9000bf3        str     x19, [sp, #<!-- -->16]
  41029c:       90000080        adrp    x0, 420000 &lt;_IO_stdin_used&gt;
  4102a0:       91004001        add     x1, x0, #<!-- -->0x10
  4102a4:       90000180        adrp    x0, 440000 &lt;__cxa_begin_catch@<!-- -->CXXABI_1.3&gt;
  4102a8:       91020000        add     x0, x0, #<!-- -->0x80
  4102ac:       97ffff77        bl      410088 &lt;_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc@<!-- -->plt&gt;
  4102b0:       d2800080        mov     x0, #<!-- -->0x4                        // #<!-- -->4
  4102b4:       97ffff69        bl      410058 &lt;__cxa_allocate_exception@<!-- -->plt&gt;
  4102b8:       52800021        mov     w1, #<!-- -->0x1                        // #<!-- -->1
  4102bc:       b9000001        str     w1, [x0]
  4102c0:       d2800002        mov     x2, #<!-- -->0x0                        // #<!-- -->0
  4102c4:       f0000161        adrp    x1, 43f000 &lt;__FRAME_END__+0x1ee50&gt;
  4102c8:       9136a021        add     x1, x1, #<!-- -->0xda8
  4102cc:       97ffff93        bl      410118 &lt;__cxa_throw@<!-- -->plt&gt;
  __cxa_throw ends up jumping here:
  4102d0:       d503249f        bti     j  &lt;--- here's our BTI
  4102d4:       f100043f        cmp     x1, #<!-- -->0x1
  4102d8:       54000040        b.eq    4102e0 &lt;main+0x54&gt;  // b.none
  4102dc:       97ffff95        bl      410130 &lt;_Unwind_Resume@<!-- -->plt&gt;
  .../...

[fweimer-rh]

The GCC change is this one, and it's needed for GCS:

commit 426fddcbdad6746fe70e031f707fb07f55dfb405
Author: Szabolcs Nagy <szabolcs.nagy@arm.com>
Date:   Wed Apr 26 14:28:32 2023 +0100

    aarch64: Use br instead of ret for eh_return
    
    The expected way to handle eh_return is to pass the stack adjustment
    offset and landing pad address via
    
      EH_RETURN_STACKADJ_RTX
      EH_RETURN_HANDLER_RTX
    
    to the epilogue that is shared between normal return paths and the
    eh_return paths.  EH_RETURN_HANDLER_RTX is the stack slot of the
    return address that is overwritten with the landing pad in the
    eh_return case and EH_RETURN_STACKADJ_RTX is a register added to sp
    right before return and it is set to 0 in the normal return case.
    
    The issue with this design is that eh_return and normal return may
    require different return sequence but there is no way to distinguish
    the two cases in the epilogue (the stack adjustment may be 0 in the
    eh_return case too).
    
    The reason eh_return and normal return requires different return
    sequence is that control flow integrity hardening may need to treat
    eh_return as a forward-edge transfer (it is not returning to the
    previous stack frame) and normal return as a backward-edge one.
    In case of AArch64 forward-edge is protected by BTI and requires br
    instruction and backward-edge is protected by PAUTH or GCS and
    requires ret (or authenticated ret) instruction.
    
    This patch resolves the issue by introducing EH_RETURN_TAKEN_RTX that
    is a flag set to 1 in the eh_return path and 0 in normal return paths.
    Branching on the EH_RETURN_TAKEN_RTX flag, the right return sequence
    can be used in the epilogue.
    
    The handler could be passed the old way via clobbering the return
    address, but since now the eh_return case can be distinguished, the
    handler can be in a different register than x30 and no stack frame
    is needed for eh_return.
    
    This patch fixes a return to anywhere gadget in the unwinder with
    existing standard branch protection as well as makes EH return
    compatible with the Guarded Control Stack (GCS) extension.
    
    Some tests are adjusted because eh_return no longer prevents pac-ret
    in the normal return path.

I don't know if there was an ABI change accompanying it, or if LLVM just does not implement the existing ABI.

[ozbenh]

I don't know if there was an ABI change accompanying it, or if LLVM just does not implement the existing ABI.

One piece of data here is that gcc 11 (which doesn't have this change) already generates the proper BTI's on exception handlers.

[ozbenh]

Copying here ... the PR above isn't quite fixing it, at least with my backports to clang15 and clang20. The EH gets a bti but the unwinder branches right past it

[ilinpv]

Thanks for the report and for working on the fix with landing pads, it looks aligned with the ABI recommendation:
https://github.com/ARM-software/abi-aa/blob/main/cppabi64/cppabi64.rst#51transferring-control-to-a-landing-pad

[shashforge]

@ozbenh Thanks for spotting the EH_LABEL insertion-point issue — the fix is now All green on check-all!. Writing test cases in progress for window