Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reference handling #16

Open
encukou opened this issue Oct 10, 2023 · 4 comments
Open

Reference handling #16

encukou opened this issue Oct 10, 2023 · 4 comments
Labels
guideline To be included in guidelines PEP

Comments

@encukou
Copy link
Contributor

encukou commented Oct 10, 2023

Most of this is already in the devguide.

Rules for new API:

Function arguments

Functions must not steal references to their arguments.
IOW, after a function call, the caller still owns the arguments it passed in.

The caller must guarantee that references stay valid for the entire call.

Return values and "output parameters" (*result)

Functions returning references must return a new reference.
IOW, after a function call, the caller owns (a reference to) the return value.

Same rules apply to filling an output parameter (*result, #13).

Borrowed references

We're likely to have exceptions to these rules (#4) which create borrowed references.
These must be documented, and the documentation must explicitly state what the reference is borrowed "from" and how long it is valid for. (Even if it's like None can be borrowed from the interpreter so it's valid until interpreter shutdown.)

@gvanrossum
Copy link
Contributor

Functions must not steal references to their arguments.
IOW, after a function call, the caller still owns the arguments it passed in.

Exception: sometimes an API exists for the sole purpose of transferring ownership. (Canonical example is DECREF, but I'm sure there are some others.)

@gvanrossum
Copy link
Contributor

@encoku The heading "Borrowed references" is followed by a section that is not about borrowed references. Is some text missing, or did you mean to type a different heading, or what?

@encukou
Copy link
Contributor Author

encukou commented Oct 10, 2023

It shouldn't have been under a heading. I moved it to the top.

@encukou encukou added the guideline To be included in guidelines PEP label Oct 11, 2023
@zooba
Copy link
Contributor

zooba commented Oct 11, 2023

Functions must not steal references to their arguments.

With nogil in the mix, we probably also need to specify that callers must guarantee the reference is not lost while the function is executing. There are almost certainly cases today where a native function can borrow all its references and then potentially have them all deallocated during execution. e.g.:

PyObject *do_something(PyObject *mod, PyObject *, PyObject *)
{
    PyObject *mod_dict = PyModule_GetDict(mod); // borrowed
    PyObject *value = PyDict_GetItemString(mod, "key"); // borrowed
    // meanwhile, another thread replaces module.key and value is DECREF'd to zero
    return SomeOtherApi(value);
}

Solving this is a more complicated problem that doesn't fully belong in this issue, but we'll either need to define that the caller is responsible for keeping the reference alive, or the callee is, and we can choose that now.

I propose choosing the caller, at least until we prove that it makes it too hard for people to migrate to nogil and we decide to make it Just Work (usually).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
guideline To be included in guidelines PEP
Projects
None yet
Development

No branches or pull requests

3 participants