JIT shim object drops GNU property notes (CET/BTI/PAC) from output binaries

[stratakis]

Bug report

Bug description:

When building with --enable-experimental-jit the jit_shim.o object lacks GNU property notes for hardware security features. The linker removes all such properties from the output binaries when any input object lacks them.

The regression was introduced with 9633c52 which makes the shim being compiled as a regular object file.

You can reproduce it by using ./configure --enable-experimental-jit=yes-off CFLAGS="-fcf-protection" && make -j.

Before the aforementioned commit a readelf -n python | grep 'x86 feature' would show Properties: x86 feature: IBT, SHSTK, whereas now nothing.

This is the case as well for BTI/PAC/GCS for aarch64 when building with #130864 and -mbranch-protection=standard.

My testing included GCC15.2.1 for the interpreter and Clang 21.1.8 for the jit, on Fedora 43.

CPython versions tested on:

3.16, CPython main branch, 3.15

Operating systems tested on:

Linux

[stratakis]

cc @diegorusso

[stratakis]

Digging into it more there are a couple more layers here beyond the property notes.

The stencils need landing pad instructions (endbr64 on x86_64 / BTI or aarch64) on their entry points because JIT code is entered via indirect branches. When CET/BTI is active the CPU will reject the branch if the target of an indirect branch lacks a landing pad.

This doesn't crash now with -fcf-protection (endbr64) or -mbranch-protection(bti), as the missing property note stops the kernel from enabling the hardware protections (if they are there). That is for x86_64, for aarch64 is the same but won't manifest till #130864 is resolved as well.

So just fixing the property note of the shim would make the JIT crash :(

For x86_64 if I pass -fcf-protection via CFLAGS_JIT (e.g. from extracting it via CFLAGS), things work fine, the stenctils get the endbr64 instruction and the shim the correct property note, all tests pass as well, but there is a cost of 4 bytes per stencil and one extra cycle per stencil at runtime (only the first stencil actually needs the instruction since it goes there from an indirect branch, for the rest the instruction becomes NOP).

Now for aarch64, passing -mbranch-protection=standard/bti from CFLAGS_JIT fails with KeyError: '_JIT_ENTRY', the stencil assembly optimizer doesn't handle properly the .note.gnu.property and the .Lsec_end0 label that is added to the assembly by clang. Same for the .ARM.attributes section type.

In summary, the x86_64 fix would be simple, but there may be a performance tradeoff, no idea how much.

The aarch64 fix would be a lot more involved.

[stratakis]

Fed it into Claude, Opus 4.6 and this came out:

diff

diff --git a/Tools/jit/_optimizers.py b/Tools/jit/_optimizers.py
index f192783a559..ff104b2cdd2 100644
--- a/Tools/jit/_optimizers.py
+++ b/Tools/jit/_optimizers.py
@@ -573,6 +573,18 @@ class OptimizerAArch64(Optimizer):  # pylint: disable = too-few-public-methods
     # https://developer.arm.com/documentation/ddi0602/2025-09/Base-Instructions/RET--Return-from-subroutine-
     _re_return = re.compile(r"\s*ret\b")
 
+    def _preprocess(self, text: str) -> str:
+        text = super()._preprocess(text)
+        # Strip .note.gnu.property section and aeabi build attributes emitted
+        # by clang with -mbranch-protection.  The note section contains a
+        # .Lsec_end0: label that breaks the block structure of the optimizer.
+        text = re.sub(r"^\s*\.aeabi_\w+.*\n", "", text, flags=re.MULTILINE)
+        text = re.sub(
+            r"^\s*\.section\s+\.note\.gnu\.property.*?^\.Lsec_end\d+:\n",
+            "", text, flags=re.MULTILINE | re.DOTALL,
+        )
+        return text
+
     supports_small_constants = True
     _re_small_const_1 = re.compile(
         r"\s*(?P<instruction>adrp)\s+(?P<register>x\d\d?),.*(?P<value>_JIT_OP(ARG|ERAND(0|1))_(16|32)).*"
diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py
index ceee383ea68..3d7b2b5110a 100644
--- a/Tools/jit/_targets.py
+++ b/Tools/jit/_targets.py
@@ -569,6 +569,7 @@ def _handle_section(
                 "SHT_NULL",
                 "SHT_STRTAB",
                 "SHT_SYMTAB",
+                "Unknown",
             }, section_type
 
     def _handle_relocation(
diff --git a/configure b/configure
index ecdd6095669..7c6c8f066d8 100755
--- a/configure
+++ b/configure
@@ -11311,6 +11311,19 @@ then :
 
 else case e in #(
   e) as_fn_append CFLAGS_NODIST " $jit_flags"
+                                                       if ${CFLAGS_JIT:+false} :
+then :
+
+             for _cflag in $CFLAGS $CFLAGS_NODIST; do
+               case $_cflag in #(
+  -mbranch-protection=*|-fcf-protection|-fcf-protection=*) :
+    as_fn_append CFLAGS_JIT " $_cflag" ;; #(
+  *) :
+     ;;
+esac
+             done
+
+fi
            REGEN_JIT_COMMAND="\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPLES:-$host} --output-dir . --pyconfig-dir . --cflags=\"$CFLAGS_JIT\" --llvm-version=\"$LLVM_VERSION\" --llvm-tools-install-dir=\"$LLVM_TOOLS_INSTALL_DIR\""
            if test "x$Py_DEBUG" = xtrue
 then :
diff --git a/configure.ac b/configure.ac
index 9863afe5bba..dad035e6387 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2893,6 +2893,17 @@ AS_VAR_IF([jit_flags],
           [],
           [],
           [AS_VAR_APPEND([CFLAGS_NODIST], [" $jit_flags"])
+           dnl If CFLAGS_JIT was not explicitly set, populate it with flags from
+           dnl CFLAGS/CFLAGS_NODIST that affect GNU property notes.  The linker
+           dnl uses AND-merge for these properties, so every object in the link
+           dnl must carry them or they are dropped from the output binary.
+           AS_VAR_IF([CFLAGS_JIT], [], [
+             for _cflag in $CFLAGS $CFLAGS_NODIST; do
+               AS_CASE([$_cflag],
+                 [-mbranch-protection=*|-fcf-protection|-fcf-protection=*],
+                 [AS_VAR_APPEND([CFLAGS_JIT], [" $_cflag"])])
+             done
+           ])
            AS_VAR_SET([REGEN_JIT_COMMAND],
                       ["\$(PYTHON_FOR_REGEN) \$(srcdir)/Tools/jit/build.py ${ARCH_TRIPLES:-$host} --output-dir . --pyconfig-dir . --cflags=\"$CFLAGS_JIT\" --llvm-version=\"$LLVM_VERSION\" --llvm-tools-install-dir=\"$LLVM_TOOLS_INSTALL_DIR\""])
            AS_VAR_IF([Py_DEBUG],

I don't understand the JIT infra code well enough to be confident on this code but I tested it on both x86_64 and aarch64 on top of #130864 with enabling the respective hardware protections and the issue is resolved, JIT works, hardware protections are enabled and the final binary/libpython contain the proper notes.