Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a variant of PyDict_SetDefault that returns a new reference (instead of a borrowed reference) #112066

Closed
Tracked by #108219
colesbury opened this issue Nov 14, 2023 · 4 comments
Assignees
Labels
3.13 bugs and security fixes topic-C-API topic-free-threading type-feature A feature request or enhancement

Comments

@colesbury
Copy link
Contributor

colesbury commented Nov 14, 2023

The PyDict_SetDefault(mp, key, defaultobj) function returns a borrowed reference to the value corresponding to key. This poses a thread-safety issue particularly for the case where key is already in the dict. In the --disable-gil builds, the returned value may no longer be valid if another thread concurrently modifies the dict.

Proposal (from Victor)

int PyDict_SetDefaultRef(PyObject *dict, PyObject *key, PyObject *default_value, PyObject **value);

The **value pointer is optional. If it is NULL, it is not used.

  • If the key is present in the dict, set *value to a new reference to the current value (if value is not NULL), and return 1.
  • If the key is missing from the dict, insert the key and default_value, and set *value to a new reference to default_value (if value is not NULL), and return 0.
  • On error, set *value to NULL if value is not NULL, and return -1.

Ideally, this new function would be public and part of the stable ABI so that it could be used by all extensions, but even an internal-only function would unblock some of the nogil changes.

EDIT: Updated with @vstinner's proposal

Linked PRs

@colesbury colesbury added type-feature A feature request or enhancement 3.13 bugs and security fixes topic-free-threading labels Nov 14, 2023
@colesbury
Copy link
Contributor Author

cc @vstinner, @encukou

@vstinner
Copy link
Member

Following previous API such as PyDict_GetItemRef() and PyDict_Pop(), I propose the following API:

int PyDict_SetDefaultRef(PyObject *dict, PyObject *key, PyObject *default_value, **value)
  • If key is present in dict, set *value to a new reference to the current value if value is not NULL, and return 1.
  • If key is missing in dict, set key to default_value in dict, set *value to a new reference to the default_value if value is not NULL, and return 0.
  • On error, set *value to NULL if value is not NULL, and return -1.

So value can be NULL if the value is not used.

See also API evolution: Return value conventions.

cc @zooba @serhiy-storchaka @erlend-aasland

@colesbury
Copy link
Contributor Author

@vstinner - sounds good. I updated the issue description with your proposal.

@vstinner
Copy link
Member

Do you want to propose a PR?

colesbury added a commit to colesbury/cpython that referenced this issue Nov 15, 2023
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.
@colesbury colesbury self-assigned this Nov 15, 2023
colesbury added a commit to colesbury/cpython that referenced this issue Nov 17, 2023
…fault`.

This changes a number of internal usages of `PyDict_SetDefault` to
use `PyDict_SetDefaultRef`.
colesbury added a commit that referenced this issue Feb 6, 2024
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
colesbury added a commit to colesbury/cpython that referenced this issue Feb 6, 2024
…fault`.

This changes a number of internal usages of `PyDict_SetDefault` to
use `PyDict_SetDefaultRef`.
colesbury added a commit that referenced this issue Feb 7, 2024
#112211)

This changes a number of internal usages of `PyDict_SetDefault` to use `PyDict_SetDefaultRef`.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
fsc-eriker pushed a commit to fsc-eriker/cpython that referenced this issue Feb 14, 2024
The `PyDict_SetDefaultRef` function is similar to `PyDict_SetDefault`,
but returns a strong reference through the optional `**result` pointer
instead of a borrowed reference.

Co-authored-by: Petr Viktorin <encukou@gmail.com>
fsc-eriker pushed a commit to fsc-eriker/cpython that referenced this issue Feb 14, 2024
…fault`. (python#112211)

This changes a number of internal usages of `PyDict_SetDefault` to use `PyDict_SetDefaultRef`.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
lysnikolaou added a commit to lysnikolaou/cpython that referenced this issue May 7, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 22, 2024
…nGH-118696)

(cherry picked from commit 447edb6)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
lysnikolaou added a commit that referenced this issue May 22, 2024
…18696) (#119430)

(cherry picked from commit 447edb6)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.13 bugs and security fixes topic-C-API topic-free-threading type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

3 participants