[3.13] _PyObject_Init() assertion fails with the stable ABI on Python built with assertions (extension built with Python 3.11)

[mgorny]

Crash report

What happened?

Consider the following extension.

C code:

#include <Python.h>

static PyObject *
foo_bar(PyObject *self, PyObject *args)
{
	Py_INCREF(PyExc_TypeError);
	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;
}

static PyMethodDef foomethods[] = {
	{"bar", foo_bar, METH_VARARGS, ""},
	{NULL, NULL, 0, NULL},
};

static PyModuleDef foomodule = {
	PyModuleDef_HEAD_INIT,
	.m_name = "foo",
	.m_doc = "foo test module",
	.m_size = -1,
	.m_methods = foomethods,
};

PyMODINIT_FUNC
PyInit_foo(void)
{
	return PyModule_Create(&foomodule);
}

setup.py:

from setuptools import setup, Extension

setup(name='foo',
      version='0',
      ext_modules=[
          Extension('foo', ['foo.c'], py_limited_api='cp38'),
      ])

If I compile the extension using Python older than 3.12, and then run the method, Python 3.13.0b3 (built --with-assertions) crashes:

$ python3.11 setup.py build_ext -i
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -Wsign-compare -fPIC -I/usr/include/python3.11 -c foo.c -o build/temp.linux-x86_64-cpython-311/foo.o
creating build/lib.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -shared build/temp.linux-x86_64-cpython-311/foo.o -L/usr/lib64 -o build/lib.linux-x86_64-cpython-311/foo.abi3.so
copying build/lib.linux-x86_64-cpython-311/foo.abi3.so -> 
$ python3.13 -c 'import foo; foo.bar()'
python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed.
Aborted (core dumped)

I've been able to bisect it to c32dc47 (CC @markshannon). Originally hit it in extensions using PyO3, and reported to PyO3/pyo3#4311.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0b3 (main, Jul 4 2024, 14:30:57) [GCC 14.1.1 20240622]

Linked PRs

• gh-121528: Fix the process crash when load extension when it's compiled with low version of the python. #121550
• gh-121528: Fix _PyObject_Init() assertion for stable ABI #121725
• [3.13] gh-121528: Fix _PyObject_Init() assertion for stable ABI (GH-121725) #121936

[Zheaoli]

The issue is confirmed, I have found the reason, but I'm not sure this is a bug, I prefer its undefined behavior.

It's caused by the Py_INCREF different behavior between 3.11 and 3.13(or master)

TL;DR;

For the 3.11 version of the Py_INCREF, we add the reference count directly without check FYI https://github.com/python/cpython/blob/3.11/Include/object.h#L491-L504

For the 3.13 version of the Py_INCREF, we add the reference count and check if it exceeds. If it exceeds, this means that this is an immortal object, we do noting except return the function. FYI https://github.com/python/cpython/blob/3.13/Include/object.h#L796-L846

And the function is an inline function, which is means we will expand the function into the binary during the compile. So in 3.11 version, the PyExc_TypeError's reference count is change from 4294967295(the immortal flag) to 4294967296. And the process will fail here https://github.com/python/cpython/pull/116115/files#diff-2a12f738a77b362d74a65949b58c37f2affcd15ba8b1c979b63bd00223b8a456R268

We can compare the assemble code to prove it

For 3.11

0000000000001120 <foo_bar>:
    1120:	48 83 ec 08          	sub    $0x8,%rsp
    1124:	48 8b 05 9d 2e 00 00 	mov    0x2e9d(%rip),%rax        # 3fc8 <PyExc_TypeError@Base>
    112b:	48 8d 35 ce 0e 00 00 	lea    0xece(%rip),%rsi        # 2000 <_fini+0xe9c>
    1132:	48 8b 38             	mov    (%rax),%rdi
    1135:	48 83 07 01          	addq   $0x1,(%rdi)
    1139:	e8 f2 fe ff ff       	call   1030 <PyErr_SetString@plt>
    113e:	31 c0                	xor    %eax,%eax
    1140:	48 83 c4 08          	add    $0x8,%rsp
    1144:	c3                   	ret
    1145:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
    114c:	00 00 00 00 

For 3.13

0000000000001120 <foo_bar>:
    1120:	48 83 ec 08          	sub    $0x8,%rsp
    1124:	48 8b 05 9d 2e 00 00 	mov    0x2e9d(%rip),%rax        # 3fc8 <PyExc_TypeError@Base>
    112b:	48 8b 38             	mov    (%rax),%rdi
    112e:	8b 07                	mov    (%rdi),%eax
    1130:	83 c0 01             	add    $0x1,%eax
    1133:	74 02                	je     1137 <foo_bar+0x17>
    1135:	89 07                	mov    %eax,(%rdi)
    1137:	48 8d 35 c2 0e 00 00 	lea    0xec2(%rip),%rsi        # 2000 <_fini+0xe9c>
    113e:	e8 ed fe ff ff       	call   1030 <PyErr_SetString@plt>
    1143:	31 c0                	xor    %eax,%eax
    1145:	48 83 c4 08          	add    $0x8,%rsp
    1149:	c3                   	ret
    114a:	66 0f 1f 44 00 00    	nopw   0x0(%rax,%rax,1)

The patch to fix this issue is simple ,we just the code back

    if (_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE)  ||  _Py_IsImmortal(typeobj)) {
        Py_INCREF(typeobj);
    }

I'm not sure this patch has side effect or not. I may need @markshannon help

[Zheaoli]

I setup a new disscussion here https://discuss.python.org/t/disable-the-inline-by-default-for-the-function-api-which-is-included-in-the-stable-abi/57832

[Zheaoli]

After dive more deeper, I think maybe its the user code issue.

I think you use Py_INCREF instead of Py_IncRef, Py_INCREF is not part of the stable ABI.

If you use the Py_IncRef I think the code here would be not existed

[davidhewitt]

The general impression I get from wjakob/nanobind#500 is that Py_INCREF is part of the limited API but is not explicitly listed due to tooling limitations. As a macro, it does not feature at the ABI level but its implementation does. Prior to 3.12, the implementation modified object reference counts directly. On 3.12 and later, the limited API implementation calls the private function _Py_IncRef.

[Zheaoli]

The general impression I get from wjakob/nanobind#500 is that Py_INCREF is part of the limited API but is not explicitly listed due to tooling limitations. As a macro, it does not feature at the ABI level but its implementation does. Prior to 3.12, the implementation modified object reference counts directly.

Yes, this is implemented as a Limit API/Stable ABI but actually it's not(can't guarantee the behavior and inlined). I'm not sure we should open a new discussion about this.

Without Py_LIMITED_API defined, some C API functions are inlined or replaced by macros. Defining Py_LIMITED_API disables this inlining, allowing stability as Python’s data structures are improved, but possibly reducing performance.

https://docs.python.org/3/c-api/stable.html#limited-api-scope-and-performance

[Zheaoli]

I think we might need a core to decide this issue should be fixed or not

@vstinner @Fidget-Spinner @markshannon @brettcannon

[alex]

It seems like this will break a large number of existing extensions if it's not resolved.

If the decision is not to fix this, then CPython needs to not load abi3 .sos that are impacted.

[vstinner]

In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call. You can build your C extension with Python 3.12 and the binary should work on Python 3.11 unless you use features specific to Python 3.12. Moreover, the binary (.abi3.so shared library) should work on Python 3.12 and newer.

Example.

setup.py

from setuptools import setup, Extension

setup(name='foo',
      version='0',
      ext_modules=[
          Extension('foo', ['foo.c'], py_limited_api='cp312'),
      ])

Build:

$ python3.12 setup.py build_ext -
(...)
$ ls *.so
foo.abi3.so*

It works on Python 3.11, 3.12, 3.13:

$ python3.11 -c 'import foo; foo.bar()'
TypeError: foo
$ python3.12 -c 'import foo; foo.bar()'
TypeError: foo
$ python3.13 -c 'import foo; foo.bar()'
TypeError: foo

It works even on Python 3.6:

$ python3.6 -c 'import foo; foo.bar()'
TypeError: foo

[alex]

My concern is primarily for the existing abi3 wheels on PyPI. The whole point is that maintainers could publish them and they'd continue to work.

[vstinner]

In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call.

• Issue: PEP 683 (Immortal Objects): Implement Py_INCREF() as function call in limited C API 3.12 #105387
• Discussion: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592
• Issue: Evolving the Stable ABI to use functions for INCREF, DECREF, TYPE capi-workgroup/problems#45

[Zheaoli]

In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call. You can build your C extension with Python 3.12 and the binary should work on Python 3.11 unless you use features specific to Python 3.12. Moreover, the binary (.abi3.so shared library) should work on Python 3.12 and newer.

Yes, the problem is the code which is using Py_INCREF and is compiled based on the Python < 3.12 for stable ABI can not be loaded in Python 3.13.

I'm not sure we should fix this or not.

[vstinner]

PEP 683 promises to not break the backward compatibility for the stable ABI, but the implementation doesn't seem to be fully compatible.

The problem is that _Py_IsImmortal() checks exactly if ref count is equal to _Py_IMMORTAL_REFCNT (refcnt == _Py_IMMORTAL_REFCNT).

This problem remains me the commit 5f6ac2d of the PR gh-112174.

[Zheaoli]

The problem is that _Py_IsImmortal() checks exactly if ref count is equal to _Py_IMMORTAL_REFCNT (refcnt == _Py_IMMORTAL_REFCNT).

Yes, I have mentioned this at here #121528 (comment)

But I think here's another core problem we change the way that we treat the object which is not in the heap and is not immortal object(From just check it to refuse the object by crashing process. I'm not sure this is a correct way.

[Zheaoli]

This problem remains me the commit 5f6ac2d of the PR gh-112174.

Sorry, I'm a little confused about the commit you mentioned because I'm not familiar with free threading. Would you mind give more detail here?

[vstinner]

Another interesting change: commit 0bb0d88 of issue gh-109496 (not directly related).

[vstinner]

Other interesting changes:

• commit bd1e950 of issue Stable ABI doesn't work with PySide (Shiboken) on Python 3.12 #118997 <= the issue is was looking for!
• commit 6199fe3 of issue 3.12.0b2: "Assertion failed: immortal object has less refcnt than expected _Py_IMMORTAL_REFCNT" #105587

[vstinner]

@mgorny:

$ python3.13 -c 'import foo; foo.bar()'
python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed.
Aborted (core dumped)

How did you install this python3.13? It seems to be a debug build of Python. Did you build it manually?

[Zheaoli]

How did you install this python3.13? It seems to be a debug build of Python. Did you build it manually?

it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)

[vstinner]

it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)

To check if you're using a debug build, you can check:

$ python3 -c 'import sys; print("debug build?", hasattr(sys, "gettotalrefcount"))'
debug build? False

I cannot reproduce the issue with a release build.

[vstinner]

The following code is a reference leak, you don't have to INCREF the TypeError exception:

	Py_INCREF(PyExc_TypeError);
	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;

You should just:

	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;

[Zheaoli]

it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)

To check if you're using a debug build, you can check:

$ python3 -c 'import sys; print("debug build?", hasattr(sys, "gettotalrefcount"))'
debug build? False

I cannot reproduce the issue with a release build.

Weird, I can't reproduce this in non-debug build now. Sorry for the previous reply, maybe I mix up something

[mgorny]

We're building --with-assertions on Gentoo, as noted in the first comment of this thread.

The problem right now is that all current versions of PyO3 produce "broken" extensions, and it will take some time before packages update to the very newest version that fixes this, and every single package published on PyPI before that happens will remain broken.

[vstinner]

I wrote PR gh-121725 to fix the assertion. I'm not sure if it's correct. It's an heuristic.

[vstinner]

Fixed by change b826e45 in the main branch. A backport will follow in the 3.13 branch.

Thanks for the bug report @mgorny. Thanks everyone who helped to analyze the root cause.