Skip to content

[BUG] Traceback reference leak in CPython 3.12.0a6 and above #5724

@yut23

Description

@yut23

Describe the bug

In CPython 3.12.0a6 and newer with CYTHON_FAST_THREAD_STATE=1, traceback objects for Python code called from Cython have their refcounts incremented but never decremented. This is triggered when Cython code calls into a Python function that then raises an exception. The source of the leak is __Pyx_ErrRestoreInState(), which never calls Py_XDECREF on tb:

static CYTHON_INLINE void __Pyx_ErrRestoreInState(PyThreadState *tstate, PyObject *type, PyObject *value, PyObject *tb) {
#if PY_VERSION_HEX >= 0x030C00A6
PyObject *tmp_value;
assert(type == NULL || (value != NULL && type == (PyObject*) Py_TYPE(value)));
CYTHON_UNUSED_VAR(type);
if (value) {
#if CYTHON_COMPILING_IN_CPYTHON
if (unlikely(((PyBaseExceptionObject*) value)->traceback != tb))
#endif
// If this fails, we may lose the traceback but still set the expected exception below.
PyException_SetTraceback(value, tb);
}
tmp_value = tstate->current_exception;
tstate->current_exception = value;
Py_XDECREF(tmp_value);
#else

Code to reproduce the behaviour:

lib.pyx:

def get_spam(data):
    # this will raise a KeyError in the Python code for UserDict.__getitem__()
    return data["spam"]

repro.py:

from collections import UserDict
from lib import get_spam

def main():
    d = UserDict()
    d["lots of data"] = bytearray(1024**2)
    try:
        get_spam(d)
    except KeyError:
        pass

for i in range(10):
    main()

I've uploaded a full reproducer to https://github.com/yut23/cython_3.12_traceback_leak, along with some code I used to track down the leaks.

Expected behaviour

No response

OS

Linux, macOS

Python version

3.12.0a6+

Cython version

3.0.0+

Additional context

We first encountered this when running yt's test suite on Python 3.12, and reported it to CPython in python/cpython#109602. Thanks to @neutrinoceros, @Xarthisius, and @mdboom for their work in simplifying the reproducer!

I'll make a PR with a proposed fix shortly.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions