[C API] Provide a replacement for removed private _Py_Identifier API, or consider reverting the removal

[vstinner]

In Python 3.13, the private _Py_Identifier API was removed: see PR #108593.

Replacements:

• Use a literal bytes string: replace &PyId_abc with "abc", use "String" variant functions. Slower, but works on all Python versions. This solution can be used on code which is not performance sensitive.
• Initialize a global or "static" variable to only create a Python str object once using PyUnicode_FromString(): less convenient, not compatible with multi-phase init API (PEP 489), not compatible with PEP 684 – A Per-Interpreter GIL.

If we want to add a public API for this, I would prefer to only add the bare minimum API, make it as small as possible:

• Py_IDENTIFIER(name) macro.
• PyUnicode_FromId(&PyId_name) function.
• Py_Identifier structure with public members (needed by Py_IDENTIFIER() macro).
• Maybe also: Py_static_string(name, "...") macro for strings which are not valid C identifiers.

It means that _PyObject_GetAttrId(obj, &PyId_abc) must be replaced with:

PyObject *name = PyUnicode_FromId(&Py_abc);
if (name == NULL) { ... handle error... }
attr = PyObject_GetAttr(obj, name);

Since functions like PyObject_GetAttr() don't accept NULL, so you cannot simply write:

attr = PyObject_GetAttr(obj, PyUnicode_FromId(&Py_abc));

See discussion:

• https://discuss.python.org/t/c-api-how-much-private-is-the-private-py-identifier-api/29190 (July 2023)
• https://mail.python.org/archives/list/python-dev@python.org/thread/DNMZAMB4M6RVR76RDZMUK2WRLI6KAAYS/ (Feb 2022)

[encukou]

Would these be interpreter-specific, or shared? Interned? Immortal?

[vstinner]

Would these be interpreter-specific, or shared? Interned? Immortal?

If possible, it should work in all cases: multiple instances of an extension per interpreter and support multiple interpreters per process with one GIL per interpreter.

Currently, _PyUnicode_FromId() interns strings, but interned strings are per interpreter which makes it problematic with Per-Interpreter GIL.

Maybe it's easier to implement if strings are made immortal.

[vstinner]

The following private functions removed in Python 3.13 alpha1 were restored in Python 3.13 alpha2:

• _PyObject_CallMethodId()
• _PyObject_GetAttrId()
• _PyUnicode_FromId()
• _Py_IDENTIFIER()
• _Py_static_string()
• _Py_static_string_init()

[vstinner]

Since the private API is back and no one managed to propose a replacement so far, I suppose that we can keep this API a little bit longer until a replacement is designed (if needed).