Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove #cgo noescape nocallback directives #128

Closed
qmuntal opened this issue Nov 4, 2023 · 4 comments · Fixed by #129
Closed

Remove #cgo noescape nocallback directives #128

qmuntal opened this issue Nov 4, 2023 · 4 comments · Fixed by #129

Comments

@qmuntal
Copy link
Collaborator

qmuntal commented Nov 4, 2023

Thanks to the upstream issue golang/go#63293, reported by @corhere, the Go team has decided to revert some buggy #cgo directives used in this project. We need to stop using them, as they won't compile anymore on tip.

Plain removal is fine, as they are just an optimization to avoid some allocations. Creating a C wrapper that don't allocate is even better.

@dagood @gdams @corhere

@corhere
Copy link
Contributor

corhere commented Nov 6, 2023

We don't necessarily need to stop using them until Go 1.23. The bug is that pre-go1.22 versions of go mod vendor fall over when encountering the offending directives, with no workaround. For most modules that would be a serious issue since any (transitive) importer of the module building their project using either in-support version of Go could be affected. But this is not most modules. The primary consumers are forks of the Go toolchain, so there are no transitive importers of the module. And aside from the private toolchain fork I am maintaining, all consumers are forks of Go 1.22+ which (until it was disabled) understand the directives. We're all necessarily patching the toolchain already to incorporate the OpenSSL bindings, so adding one more patch to the toolchain to re-enable the directives, either by reverting golang/go@607e020 or by special-casing this module so the directives are still disabled for application code, would not add much to the ongoing maintenance cost of our forks.

However:

  • Someone would have to write and test the special-casing patch if reenabling the directives for application code is unacceptable
  • Our forked toolchains and the applications built with them would be the only ones exercising the new cgo directives on Go 1.22, so we'd be on our own if the codegen is buggy
  • Applications would not be able to import golang-fips/openssl/v2 until Go 1.23, though it is unlikely to be an issue in practice
  • Patched Go 1.22 toolchains would be required for development and testing the OpenSSL bindings until Go 1.22 is released in February and the new cgo directives are reenabled on gotip
  • It's still a nontrivial amount of work, and risk, all just for some micro-optimizations which will be generally available in Go 1.23

On the balance, I agree that removing the directives is the right call.

@dagood
Copy link
Collaborator

dagood commented Nov 6, 2023

It's surprising to me how solid the case for it being ok to leave them in. 😄 But I agree, my impression is that the memory improvement from using #cgo directives is significant (#113) but nobody finds it critical (yet), so the risk+cost of patching it back in early doesn't seem worth it.

I'll submit a quick PR to remove the file. (The microsoft/go upstream sync PR is actually currently blocked on this after the upstream revert.)

@qmuntal
Copy link
Collaborator Author

qmuntal commented Nov 6, 2023

But I agree, my impression is that the memory improvement from using #cgo directives is significant (#113)

It is possible to get the same results by using well-crafted C wrappers, as we are already doing in

// These wrappers allocate out_len on the C stack to avoid having to pass a pointer from Go, which would escape to the heap.
. But lers remove the #cgo directives to unblock our toolchain, we can bring back down memory allocations in a later PR.

@dagood
Copy link
Collaborator

dagood commented Nov 6, 2023

Submitted #129 with a simple remove. Will take a stab at writing wrappers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants