How to shrink the size of the C API? Or at least, slow down its growth?

[vstinner]

A side question of issue #52 is: how can we slow down the growth of the C API?

Is there a way to add less functions or even stop adding functions?

Fixing the API requires to add variants of existing functions. Sometimes, the old APIs are deprecated and later removed, but the deprecation is slow: PEP 387 "Backwards Compatibility Policy".

See Statistics of the C API: Functions.

Functions (Python 3.6 => Python 3.12):

• Public functions: 870 (3.6) => 966 (3.12): +96 functions (+11%).
• Private functions:378 (3.6) => 378 (3.12): same! I suppose that this number is only stable thanks to the creation of the internal C API.
• Internal C API: 0 (3.6) => 178 (3.12): +178 functions

Structures (Python 3.6 => Python 3.12):

• Public structures: 110 (3.6) => 102 (3.12): -8 structures. Good! It's smaller! Again, I suppose that it's thanks to the internal C API.
• Internal structures: 0 (3.6) => 170 (3.12): +170 structures

See also issue #39 "Supporting multiple ABI versions at once".

[markshannon]

I thinking the only way to shrink the API is to take care when adding new functions, and aim to deprecate more than we add.
It will be a slow process, as the deprecation period should be many years.

Why is the number of structs an issue? They mostly aren't part of the API.
Internal structs can help clean up APIs and make code more robust.
E.g. changing Py_CODEUNIT to a struct from a typedef makes it safer and clearer to use.

Many of the "public structures" weren't supposed to be part of a public API, they just leaked into Python.h. I think we would all like most, if not all, of them to be private. It just a matter of deprecating them, which takes time and effort.
And any struct we deprecate needs some accessor functions to be added, making the API even bigger 🙁

[koubaa]

Many APIs can be implemented in terms of others. For example, PyRunAnyFile,can be (and is) implemented in terms of PyRun_AnyFileFlags.

For lack of better terms, consider the first category the "sugar" API and the second category the "salt" API.

I think the size of the API, in terms of maintenance burden, should only be measured based on the size of the salt API. The sugar API might one day be developed and shipped in a header-only or a separate binary.

[vstinner]

@markshannon:

Why is the number of structs an issue? They mostly aren't part of the API.

The problem is more structure members which are exposed in the public C API: any change in these members is usually causing a lot of pain.

Type names are fine.

[vstinner]

See also issue #62: How can an user access old removed functions? Can a 3rd party project provide them?

[encukou]

IMO, we should have a better problem description here.

Why is the size of the API a problem? Which numbers are most problematic -- number of exported functions? Documentation entries? C definitions? Struct members?

[vstinner]

I proposed a solution to this problem: PEP 743 – Add Py_COMPAT_API_VERSION to the Python C API. It's a way to get a subset of the API and so a smaller API.

If someone has other ideas to reduce the API size, I suggest opening an issue in the new https://github.com/capi-workgroup/api-evolution project. I close this issue.