C API: Add PyTuple_GetItemRef(), similar to PyTuple_GetItem() but return a strong reference

[vstinner]

I propose to add a new PyTuple_GetItemRef() function, similar to PyTuple_GetItem() but return a strong reference instead of a borrowed reference.

Other functions were added recently to return a strong reference:

• PyDict_GetItemRef(), PyDict_GetItemStringRef() -- issue C API: Add PyDict_GetItemRef() function #106004
• PyList_GetItemRef() -- issue Add PyList_GetItemRef, a variant of PyList_GetItem that returns a strong reference #114329
• PyWeakref_GetRef() -- issue C API: Add PyWeakref_GetRef() function #105927
• PyImport_AddModuleRef() -- issue C API: Add PyImport_AddModuleRef() function #105922

I already proposed adding PyTuple_GetItemRef() in 2020, but the idea was rejected: issue gh-86460. Since 2020, Free Threading (PEP 703) was added to Python and borrowed references don't play well with Free Threading.

Linked PRs

• gh-117518: Add PyTuple_GetItemRef() function #117519
• gh-117518: Clarify PyTuple_GetItem() borrowed reference in the doc #117920
• [3.12] gh-117518: Clarify PyTuple_GetItem() borrowed reference in the doc (GH-117920) #118087

[vstinner]

@colesbury @erlend-aasland @serhiy-storchaka: What do you think of adding this function?

[erlend-aasland]

@colesbury @erlend-aasland @serhiy-storchaka: What do you think of adding this function?

I'm not a part of the C API WG, but I'm +1 ;)

[serhiy-storchaka]

I do not think it is needed. Adding it means that there is something wrong with PyTuple_GetItem(), but it is not. Tuples are immutable, so as far as we keep a strong reference to a tuple, weak references to its elements are valid. There is a lot of code that relies on this, not only directly, but function call conventions and PyArg_Parse* API.

Adding PyTuple_GetItemRef() will not solve any problem. Replacing PyTuple_GetItem() with PyTuple_GetItemRef() will have significant performance impact, and in many cases just impossible.

[vstinner]

The problem is not the tuple data structure, the problem here is to promote usage of strong references, rather than borrowed references to prevent crashes caused by dangling pointers. Example where using a borrowed reference is dangerous:

PyObject *tuple = Py_BuildValue("(iii)", 2000, 3000, 4000);
PyObject *number = PyTuple_GetItem(tuple, 0);

Py_DECREF(tuple);
// number is now a dangling pointer

... code using 'number' variable ...

Correct code using only strong references:

PyObject *tuple = Py_BuildValue("(iii)", 2000, 3000, 4000);
PyObject *number = PyTuple_GetItemRef(tuple, 0);

Py_DECREF(tuple);
// number is now a dangling pointer

... code using 'number' variable ...
Py_DECREF(number);

My intent is to have a subset of the C API which does not use borrowed references, or at least, hide functions returning borrowed references, when alternativates using strong references exists. See also PEP 743 – Add Py_COMPAT_API_VERSION to the Python C API.

[vstinner]

There is a lot of code that relies on this, not only directly, but function call conventions and PyArg_Parse* API.

I don't plan to deprecate PyTuple_GetItem(). You can still use it if you are sure that you are holding a strong reference to the container (the tuple) while using its items.

[erlend-aasland]

I suggest opening an issue on the C API WG bug tracker.

[serhiy-storchaka]

If you want to create a strong reference, just call Py_INCREF().

[erlend-aasland]

After re-reading PEP-703 I'm +0. Quoting from the PEP:

Note that some APIs that return borrowed references, such as PyTuple_GetItem, are not problematic because tuples are immutable.

[vstinner]

I already proposed adding PyTuple_GetItemRef() in 2020, but the idea was rejected: issue #86460. Since 2020, Free Threading (PEP 703) was added to Python and borrowed references don't play well with Free Threading.

See also this numpy issue: numpy/numpy#26159

[erlend-aasland]

I already proposed adding PyTuple_GetItemRef() in 2020, but the idea was rejected: issue #86460. Since 2020, Free Threading (PEP 703) was added to Python and borrowed references don't play well with Free Threading.

See also this numpy issue: numpy/numpy#26159

That issue does not mention PyTuple_GetItem at all ;) Consider me -0 now, since there appears to be no third party need for this API.

[vstinner]

The documentation was completed to clarify that it returns a borrow reference.