JIT: `executor->vm_data.valid` assertion failure in `unlink_executor`

[devdanzin]

Crash report

What happened?

Sorry about the messy and convoluted bug report, I hope something interesting might be gleaned from it.

It's possible to non-deterministically abort a special debug JIT build by running the code below, running from a Python executable from a venv where you installed a package in editable mode. This darn requirement for an install in editable mode stumped me for hours!

For this special build, apply this diff (couldn't get it to work in a normal build by changing loop lengths):

index 454c8dde031..9e21c41421a 100644
--- a/Include/internal/pycore_backoff.h
+++ b/Include/internal/pycore_backoff.h
@@ -99,8 +99,8 @@ backoff_counter_triggers(_Py_BackoffCounter counter)
 // Must be larger than ADAPTIVE_COOLDOWN_VALUE, otherwise when JIT code is
 // invalidated we may construct a new trace before the bytecode has properly
 // re-specialized:
-#define JUMP_BACKWARD_INITIAL_VALUE 4095
-#define JUMP_BACKWARD_INITIAL_BACKOFF 12
+#define JUMP_BACKWARD_INITIAL_VALUE 63
+#define JUMP_BACKWARD_INITIAL_BACKOFF 6
 static inline _Py_BackoffCounter
 initial_jump_backoff_counter(void)
 {
@@ -112,8 +112,8 @@ initial_jump_backoff_counter(void)
  * Must be larger than ADAPTIVE_COOLDOWN_VALUE,
  * otherwise when a side exit warms up we may construct
  * a new trace before the Tier 1 code has properly re-specialized. */
-#define SIDE_EXIT_INITIAL_VALUE 4095
-#define SIDE_EXIT_INITIAL_BACKOFF 12
+#define SIDE_EXIT_INITIAL_VALUE 63
+#define SIDE_EXIT_INITIAL_BACKOFF 6

 static inline _Py_BackoffCounter
 initial_temperature_backoff_counter(void)

This MRE must be invoked like this:

/path/to/venv/with/editable/install/bin/python mre.py

Sorry about the long MRE, this code seems stubbornly hard to reduce. MRE:

from random import random

class C_object_v1:

    def __getitem__(self, item):
        return 5
object_v1 = C_object_v1()


int_v4 = -51452741571812374
float_v6 = 46.1
float_v7 = 77.26

class StatefulLen_stateful_len_object_v12:
    def __init__(self):
        self.len_count = 0

    def __len__(self):
        _len = 0 if self.len_count < 70 else 99
        self.len_count += 1
        return _len
stateful_len_object_v12 = StatefulLen_stateful_len_object_v12()

def uop_harness_f1():
    class StatefulIter_uop_harness_f1_6059:

        def __init__(self):
            self._iter_count = 0
            self._iterable = {1, 2, 3}

        def __iter__(self):
            self._iter_count += 1
            if self._iter_count <= 67:
                return iter((None,))
            return iter(self._iterable)
    uop_harness_f1_6059 = StatefulIter_uop_harness_f1_6059()
    evil_iterable = StatefulIter_uop_harness_f1_6059()
    for _ in range(100):
        tuple(evil_iterable)
    object_v1 = globals()['object_v1']

    class SwappedClass_2950:
        pass
    object_v1.__class__ = SwappedClass_2950
    if object_v1:
        pass
    for i_loop_9595 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] * 50:
        res_compareopint = int_v4 > int_v4
    if random() < 0.01:
        object_v1.__class__.get_value = lambda *a, **kw: 'patched!'
for i_f1 in range(500):
    try:
        uop_harness_f1()
    except Exception:
        break

Backtrace:

python: Python/optimizer.c:1436: unlink_executor: Assertion `executor->vm_data.valid' failed.

Program received signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0)
    at ./nptl/pthread_kill.c:44
warning: 44     ./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=0)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (threadid=<optimized out>, signo=6) at ./nptl/pthread_kill.c:89
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:100
#3  0x00007ffff7c4579e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c288cd in __GI_abort () at ./stdlib/abort.c:73
#5  0x00007ffff7c28830 in __assert_fail_base (fmt=<optimized out>, assertion=<optimized out>,
    file=<optimized out>, line=<optimized out>, function=<optimized out>) at ./assert/assert.c:118
#6  0x00007ffff7c3be1f in __assert_fail (assertion=<optimized out>, file=<optimized out>,
    line=<optimized out>, function=<optimized out>) at ./assert/assert.c:127
#7  0x00005555558a2603 in unlink_executor (executor=executor@entry=0x555555f44480)
    at Python/optimizer.c:1436
#8  0x00005555558a31a7 in uop_dealloc (op=op@entry=0x555555f44480) at Python/optimizer.c:271
#9  0x00005555556dc978 in _Py_Dealloc (op=0x555555f44480) at Objects/object.c:3206
#10 0x00007ffff71d59f7 in ?? ()
#11 0x00007fffffffd470 in ?? ()
#12 0x00007ffff71d522b in ?? ()
#13 0x00007fffffffd470 in ?? ()
#14 0x00007ffff71d4017 in ?? ()
#15 0x0000555555e60c70 in ?? ()
#16 0x00007ffff71d4000 in ?? ()
#17 0x0000555555cca298 in _PyRuntime ()
#18 0x00007ffff7fac0b8 in ?? ()
#19 0x00007ffff7fac160 in ?? ()
#20 0x00005555557ca1c0 in _PyEval_EvalFrameDefault (tstate=0xcdcdcdcdcdcdcdcd, frame=0x555555cca298 <_PyRuntime+331064>, throwflag=-842150451)
    at Python/generated_cases.c.h:5598
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Here are the outputs from running with PYTHON_OPT_DEBUG=4 and PYTHON_LLTRACE=4, since they are too long I've attached them as files:
opt_debug2.txt
lltrace2.txt

Found using lafleur.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0a0 (heads/main-dirty:6bf1c0ab349, Jul 21 2025, 20:28:08) [GCC 14.2.0]

[Fidget-Spinner]

I can't reproduce this. I installed https://github.com/pypa/sampleproject in editable mode on a JIT with the patch applied, and ran the reproducer in an infinite while loop for some time in bash. No crash.

[devdanzin]

Please try with https://github.com/devdanzin/lafleur/ installed as editable, with no other packages installed. Seems to do the trick here. Sorry for the bad reproducing steps.

[devdanzin]

OK, here's a reproducer that mimics the effects of having a package installed as editable. Please let me know whether it works.

import sys
from importlib.machinery import all_suffixes as module_suffixes  # Not sure why, but this unused import seems necessary
from importlib.util import spec_from_file_location

class _EditableFinder:
    @classmethod
    def _find_spec(cls, fullname, candidate_path):
        return spec_from_file_location(fullname, candidate_path.with_suffix("pyc"))

sys.meta_path.append(_EditableFinder)

from random import random

class C_object_v1:
    def __getitem__(self, item):
        return 5
object_v1 = C_object_v1()


int_v4 = -51452741571812374
float_v6 = 46.1
float_v7 = 77.26

class StatefulLen_stateful_len_object_v12:
    def __init__(self):
        self.len_count = 0

    def __len__(self):
        _len = 0 if self.len_count < 70 else 99
        self.len_count += 1
        return _len
stateful_len_object_v12 = StatefulLen_stateful_len_object_v12()

def uop_harness_f1():
    class StatefulIter_uop_harness_f1_6059:

        def __init__(self):
            self._iter_count = 0
            self._iterable = {1, 2, 3}

        def __iter__(self):
            self._iter_count += 1
            if self._iter_count <= 67:
                return iter((None,))
            return iter(self._iterable)
    uop_harness_f1_6059 = StatefulIter_uop_harness_f1_6059()
    evil_iterable = StatefulIter_uop_harness_f1_6059()
    for _ in range(100):
        tuple(evil_iterable)
    object_v1 = globals()['object_v1']

    class SwappedClass_2950:
        pass
    object_v1.__class__ = SwappedClass_2950
    if object_v1:
        pass
    for i_loop_9595 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] * 50:
        res_compareopint = int_v4 > int_v4
    if random() < 0.01:
        object_v1.__class__.get_value = lambda *a, **kw: 'patched!'
for i_f1 in range(500):
    try:
        uop_harness_f1()
    except Exception:
        break

[Fidget-Spinner]

Reproduced. Thanks!

[brandtbucher]

I'm making some progress on reducing. Here's what I've got so far:

import gc

class C1:
    pass

O = C1()

gc.set_threshold(1)

for i in range(326):
    for j in []:
        pass
    class C2:
        pass
    O.__class__ = C2
    if O:
        pass

I'm working now on bringing the JIT thresholds back up.

[brandtbucher]

BTW the imports were just to get the GC in a ready-to-crash state. gc.set_threshold(1) is usually my go-to for removing that factor.

[brandtbucher]

Higher JIT thresholds aren't reproducing. Highest I can get is cranking up the number of loops from 326 to 12868 to reproduce with thresholds of 127 instead of 63.

[Fidget-Spinner]

Thanks Brandt for the repro. Based on debugging this, I speculate the following is happening:
The code which crashes is

        case _EXIT_TRACE: {
        ....
            if (exit->executor && !exit->executor->vm_data.valid) {
                exit->temperature = initial_temperature_backoff_counter();
                _PyFrame_SetStackPointer(frame, stack_pointer);
                Py_CLEAR(exit->executor);  // THIS CRASHES
                stack_pointer = _PyFrame_GetStackPointer(frame);
            }

This causes assert(executor->vm_data.valid); to fail. The assert is clearly wrong in this case as we only free the exit trace executor when the data is no longer valid.

I think the correct thing is just to remove the assertion. WDYT Brandt?