-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Description
Describe the bug
In CPython 3.12.0a6 and newer with CYTHON_FAST_THREAD_STATE=1
, traceback objects for Python code called from Cython have their refcounts incremented but never decremented. This is triggered when Cython code calls into a Python function that then raises an exception. The source of the leak is __Pyx_ErrRestoreInState()
, which never calls Py_XDECREF
on tb
:
cython/Cython/Utility/Exceptions.c
Lines 178 to 193 in 9827c60
static CYTHON_INLINE void __Pyx_ErrRestoreInState(PyThreadState *tstate, PyObject *type, PyObject *value, PyObject *tb) { | |
#if PY_VERSION_HEX >= 0x030C00A6 | |
PyObject *tmp_value; | |
assert(type == NULL || (value != NULL && type == (PyObject*) Py_TYPE(value))); | |
CYTHON_UNUSED_VAR(type); | |
if (value) { | |
#if CYTHON_COMPILING_IN_CPYTHON | |
if (unlikely(((PyBaseExceptionObject*) value)->traceback != tb)) | |
#endif | |
// If this fails, we may lose the traceback but still set the expected exception below. | |
PyException_SetTraceback(value, tb); | |
} | |
tmp_value = tstate->current_exception; | |
tstate->current_exception = value; | |
Py_XDECREF(tmp_value); | |
#else |
Code to reproduce the behaviour:
lib.pyx:
def get_spam(data):
# this will raise a KeyError in the Python code for UserDict.__getitem__()
return data["spam"]
repro.py:
from collections import UserDict
from lib import get_spam
def main():
d = UserDict()
d["lots of data"] = bytearray(1024**2)
try:
get_spam(d)
except KeyError:
pass
for i in range(10):
main()
I've uploaded a full reproducer to https://github.com/yut23/cython_3.12_traceback_leak, along with some code I used to track down the leaks.
Expected behaviour
No response
OS
Linux, macOS
Python version
3.12.0a6+
Cython version
3.0.0+
Additional context
We first encountered this when running yt's test suite on Python 3.12, and reported it to CPython in python/cpython#109602. Thanks to @neutrinoceros, @Xarthisius, and @mdboom for their work in simplifying the reproducer!
I'll make a PR with a proposed fix shortly.