The GYP issue: where, how and who to maintain #1791
Comments
|
The nodejs/node#26620 is on progress... So I think we should wait for it? |
|
I think @refack may have lost interest in helping maintain this tooling and we might be needing an alternative solution. We also have gyp.js and ninja.js in the picture, I'm about to put up a pull request for including gyp.js (the 3rd pull request to date) and see if we can work toward an opt-in for it and gradually work to full compatibility. But we need a strategy for maintaining it too, I don't believe @indutny is interested in pursuing these two projects and it'd be good to have a collective space for a group of people to maintain them. For now I was thinking that setting up a new lightweight github org to maintain a collection of gyp and gyp-related tools might be a good approach, at least for the short term. Pull in GYP / GYP3 (or some synthesis of what's been done here, in nodejs/node and refack/GYP), gyp.js and ninja.js if @indutny wants to give them up, and use it as an upstream for node-gyp and anything else using gyp (nodejs/node, libuv?). I'd love to hear others' thoughts on this. |
|
LGTM |
|
I'd be happy to transfer both |
|
My preference would be to put our efforts into gyp.js and get rid of the python-based gyp, at least for add-ons. TBD how to deal with Node's build system. I regret I didn't push my cmake-based fork but I ran out of time. If someone wants to dust it off and take all the credit, you're welcome. :-) |
|
Also related to nodejs/build#1789 |
|
use GN |
|
@bnoordhuis Chrome has given up gyp, we should use GN. like electron. |
|
There's less than zero interest in investing in another Google produced, maintained, and gatekept tool, only to have it abandoned once an entire ecosystem is invested in it, like GYP. This goes especially for one that is only supported on a limited number of platforms. Our preference is toward more open and broadly supported and understood tools |
|
@rvagg Learn from |
|
If it is not because of ecology, what language would you choose? |
|
@zuohuadong There's nothing mandating the use of We were discouraged by Google from using GN (see nodejs/TSC#648 and preceding nodejs/TSC#464). For the thousands of existing addons that do use |
|
@richardlau Such as node-bazel, node-gn, node-cmake one of them. |
@zuohuadong I'd suggest you raise an issue over in the main Node.js repository (or PR directly against the addons docs in the main repository). This issue/repository isn't the place to discuss that. |
|
So... To get back to the issue at hand. My vote is that we pull the GYP3 changes into node-gyp. There are many solid decisions and fixes that went into GYP3 and with the inclusion of pyenv, it will make a more stable platform for continuing our Python 3 compatibility work. In the long run, we should drop the need for Python but in the near term, GYP3 is a better platform for getting to Python 3 compatibility before yearend. |
|
@cclauss |
|
Here's my suggestion for how we view the current status and immediate future plans:
By pushing both GYP3 and gyp.js (and other supporting libraries) out of this repo we can clarify the specific domain that we're focusing on here and push all other issues to those edges. The challenge will be making sure that those edges don't become dead ends, so we'll need to make sure we have active communities around GYP3 and gyp.js because that's really been the biggest problem for node-gyp historically. Fedor doesn't want to maintain gyp.js so we're going to need to pull that into this org (I would have liked a less heavyweight way to manage it but Fedor's preference is to transfer to the nodejs org). I'm fine with vendoring in a GYP3 maintained under @refack's GitHub in absence of any problems managing it that way. In case it's not clear to anyone reading this: this is purely about ecosystem add-on builds and doesn't solve the nodejs/node dependency on GYP, that's a separate discussion but perhaps a more healthy GYP3 helps alleviate that pain. |
|
@rvagg FWIW, I don't have strong preference about transferring or not transferring it. I'll be happy with whatever way of managing you think would be best for it. |
|
@zuohuadong ok, but you realise this is just more fuel for the anti-GN sentiment, right? "it's successor was already deprecated too" isn't correct either, its successor is GN—if you take Google build tooling as canonical for "successor", and some of us aren't blinkered into believing that Node is tightly tied to Google's product decisions, certainly not like Electron which is is hopelessly linked to Chromium codebase. We have a very large and complex ecosystem to manage here, it doesn't turn on a dime. The Electron ecosystem seems to be much more comfortable with the rapid breakage of tying itself to the V8 upgrade cycle, Node takes a much more conservative approach and leans more toward classic server-side upgrade cycles than the rapid Browser change-o-rama. In JavaScript you get "JavaScript one" backward compatibility guarantees, we just don't get that with C++ or build toolchains, it's breakage all the way down and we have to make some very difficult compromises. |
|
@rvagg @nodejs/node-gyp You asked somewhere (I just can't find it) if its time to re-publish node-gyp, and I think it is, or pretty close to. We've got node tests starting to go green when built/tested with py3 over in nodejs/node#29451, and the last trouble was with the node-gyp@5.0.3 in npm, so if we could get a 5.0.4 published that would get us, once npm updates and is re-vendored into node's deps, a py3-working node. At that point there is still remaining work --- figure out what to do in CI, what versions should be default, etc, but it would be quite a milestone! A huge thanks to @cclauss for keeping the pressure up on this, and everyone who has helped out. This is all the shortest, hackiest way of getting py3 support, and in no way derails from the broader question of the gyp fragmentation, the "where, how, and who to maintain" is still very much an issue. |
|
@bnoordhuis @rvagg @Fishrock123 You have publish rights, see ☝️, this needs publishing for py3 capability to make it into npm -- or a pointer to what needs to be fixed before it can be published. |
|
It looks to me like |
|
I don't normally publish this but I could if necessary. I'd prefer for either Ben or Rod to chime in if possible. |
|
nodejs/node has an indirect dependency (through npm) on nodejs/node-gyp. Given that we have 100 days left until Py2 EOL and we gotta publish so that npm picks that up so we can re-update our indirect dependency on ourselves... I would recommend that we first publish a minor release without c6e3b65 and then once we have made more Py3 progress, we bundle up c6e3b65 and other semver-major PRs for a major release. |
|
@Fishrock123 @bnoordhuis I don't know how you'd prefer to manage node-gyp. You could revert c6e3b65 on master after doing a The opposite, more node-like approach is possible, too, to allow master to have semver-majors,and make a 2.x branch and keep semver-majors off of it. /shrug. |
|
will sort out a release asap, I just have a bit of a backlog that I'm working through to mentally get on top of things |
|
aside: I hope this isn't off topic, I can move to another issue if its helpful, but I don't want to fragment the conversation unnecessarily. Just to set the ground rules:
@rvagg What I suggest doing is a staged approach:
|
|
@nodejs/npm As a (the?) major consumer of node-gyp, do you have any comments on the suggestions above? |
|
Complicating things is that we should probably get #1689 back in, which was reverted in #1828 and should be considered semver-major. The problem with it for upstream is that it should probably be considered semver-major for Node too and only go in for a semver-major break, but node-gyp is a transitive dependency via npm so we require some kind of coordination there too. My plan was to start a 5.x branch and manage 5 and 6 at the same time so we can do a proper major and also some patch/minor releases. I don't have strong opinions about any of the Python stuff so will take advice on that. My inclination is to not bother with a patch release, does anyone care enough for the distinction between patch and minor? Are we talking about risk for the changes that are semver-minor? Is it that they're untested and we have low confidence in them? Will a semver-patch prior to a semver-minor actually help matters at all? (i.e. is it really necessary for steps 1 and 2 above to be separate things?). btw I started getting |
|
I'm OK with bundling what I called I think And a 5 & 6 branch seems pretty reasonable to me. |
|
#1893 for a 5.0.4 proposal and #1892 for a 6.0.0. I expect the latter to take a while before it gets out so let's focus on 5.x for now. @sam-github if the Happy to take advice on what else needs doing, otherwise I guess I'll just pump out the v5.0.4. |
|
5.0.4 is out now, has our whole backlog minus the one semver-major that's queued up for a 6.0.0. We have a v5.x branch now so are prepared to do more 5.x releases as needed. |
|
@claudiahdz @isaacs @nodejs/npm Will the next @cclauss there is some precedent for floating node-gyp patches in nodejs, see nodejs/node#c3caf21bd3, so perhaps we could do that in the short-term, though I assume we wouldn't want it actually picked in nodejs/node#v12.x |
|
Done in npm/cli#260 |
|
Back to the original topic of this thread. This is delicate, and I mean no disrespect, but at this stage I don't believe that relying on refack/GYP3 is what's best for this project at this time. It's important that we have a GYP that can receive timely updates and be maintained by more than one or two people. A lot of work has gone in to GYP3 and I'd hate to see that go to waste but my personal preference for node-gyp is for stability and a straightforward path to receiving contributions from people comfortable hacking Python. It seems to me that having GYP in a separate repo would be a good thing, isolating node-gyp's concerns from GYP's concerns. But it's not urgent. We first need to come up with a name for it that's different enough from GYP to indicate that it's not Google's GYP project. "gyp-ng", "gypnext", something like that. Having a separate repo might make it easier to set up proper testing, documentation, a contribution process and even a dedicated team of Python people. We can vendor it in, use submodules or even publish GYP directly to npm and pull it in as a dependency to node-gyp. If someone here is feeling particularly inspired with a new name then speak up and maybe we should get the ball rolling? We should give a proposal to the TSC for action on this front before taking action because this is not a trivial matter and involves Google+legal concerns (the naming thing has come up already). If Refael finds a path to come back to actively contributing to Node.js and wants to re-engage then this would be a perfect place to do so and I'm sure he would be heartily welcomed due to his obvious high level of expertise in the nexus of toolchains+python. But for now, I think we have to move forward on something that we are in more direct control over. |
|
I'm still not up to speed with the git situation, things I don't understand ATM:
In other words, what are the problems to solve?
I like the idea of gyp.js, but my impression is it foundered not on technical difficulties, but once again, on not enough contributors. Everything seems to come down to getting (and keeping) contributors. Both the gyp.js approach @bnoordhuis and @indutny mention or gyp-ng sound like reasonable enough plans to me, but we need people to do the work. |
|
@sam-github good questions, and my comment above didn't include gyp.js, my proposal for which in #1791 (comment) still stands.
node-gyp is the JavaScript tooling wrapper around GYP, it's always vendored it in but when Google dropped it we ended up with two orphaned GYPs, one here and one in nodejs/node, and we've not resolved that relationship, nor have we formalised how we are treating these two codebases. We've just patched things where needed to keep them working. I'd like to at least formalise what GYP is and where it is canonically located or how the relationship between forks is. This confusion keeps on popping up in nodejs/node too. My preference is to separate concerns. We have a GYP codebase that's Python and focused on a particular language (the GYP language) and how it defines an interaciton with toolchains. We have a JavaScript codebase that is focused on the interaction between GYP and Node.js / npm. I'm not at all comfortable hacking on GYP itself. I'm entirely comfortable hacking on the JavaScript in node-gyp and understand the interface down to the GYP layer. I'd like there to be a GYP place where people can go to deal with Python and toolchain concerns. Then node-gyp and nodejs/node could be consumers of that piece of the puzzle, both vendoring in the same Python code.
Because the divergence is now quite large. Refael has done a ton of work in GYP3, so it's an entirely different beast. In parallel, and since then, we've done a lot of work here. I'd love someone to do a reconciliation but I don't know who is going to put in that effort. My preference is to stick with what works and evolve from there, which would mean using the GYP we have in here and perhaps patching in some nodejs/node changes. Then perhaps we could consider GYP3 changes on top of that, but so far it's entirely untested for the purpose of node-gyp so I don't think it's suitable as a starting point.
Yep, people is a problem no matter how we dice it. And we're shackled to GYP at least in the medium term because there's a massive ecosystem built around assumptions of its existence and the way it operates. My proposal in this thread is to accept that, but migrate away from Python over time, investing in gyp.js, which does work for trivial projects (on limited platforms) already. We have way more JavaScript hackers around this org than Python hackers so it seems reasonable to me that people will just show up to fix their little corner cases as we expose it and start suggesting that it's going to eventually replace the Python version. Alternatives include:
The second one of these gets mentioned regularly with the two main suggestions being GN (see comments in this thread for e.g.) and CMake (which I'm more in favour of if we were to make a switch). And maybe we could invest in migration tooling for addon authors if we were to do one of these thing, and even bake them in to our next gen tooling so older stuff still works—this is not even mutually exclusive with the gyp.js as our "next gen" thing, perhaps we have a next-next gen that favours CMake input but is able to parse older binding.gyp with gyp.js and generate appropriate intermediate CMake output. |
|
So, I spent a little time looking at the divergence between nodejs/node:tools/gyp ("tools gyp") and nodejs/node-gyp:gyp ("node-gyp"). There is a lot.... some of it is just textual, or project meta data, but the rest doesn't look easy to reconcile, see https://github.com/nodejs/node-gyp/compare/master...sam-github:compare-to-tools-gyp?expand=1 and https://github.com/nodejs/node/compare/master...sam-github:compare-to-node-gyp?expand=1 As some background, repeating some of what @rvagg explained:
I assume GYP3 was intended to bring the forks together and work for both node itself, and node addons, and just generally, but its untested. A straightforward vendor of node-gyp's gyp/ directory into node's tools/gyp fails to even run, much less build Node.js, which didn't surprise me at all. My thoughts on where we should go: For addons:I agree with @rvagg that we should encourage and hope for gyp.js to develop to the point it can be used to build Node.js addons, but would like to note that the addon community might be moving past gyp, see nodejs/node-addon-api#445 (comment) Still, besides the many many existing addons with gyp configuration that we can't just abandon, I think there is room here for new developments, I'm hopeful that if development continues on the various approaches, one or might become viable enough to become the defacto standard, then later, perhaps even "the" standard (and added to the npm CLI). For node.jsWhat Node.js does to build itself from source is completely unrelated to node-gyp, AFAICT, and while the discussions of "what to do about gyp" for node.js and addons get linked together sometimes, they aren't dependent on each other. I'm not convinced the technical and inertial hurdles to get the node-gyp and node.js forks of gyp merged can be overcome, or even need to be. While tools/gyp is being used for node.js, if any relevant changes land in either fork, they can be cross-ported, which is what happend for Python 3 support. If at some point Node.js gets a new build system (which at this point I think mostly involves someone actually doing the work and making sure it works across all our supported platforms), then tools/gyp would just be deleted, and node-gyp would have the only maintained gyp still in use. |
|
Thanks for this... I am in agreement. I believe that tools/gyp needs a new name (that does not contain the word gyp). An effort should be made in slim tools/gyp down to its bare essentials so that it can either be maintained or rewritten in JavaScript. |
|
Good writeup @sam-github, +1 to everything there I think. I wasn't aware cmake.js was so mature, that's great news frankly and I'd be glad to see more addons head that way. The legacy problem remains though so we should still do our best to make sure node-gyp doesn't represent significant rot and technical debt for Node. I think we've made great progress as a team, particularly this year. Next steps I propose:
I think we're circling around some form of agreement of the general trajectory from there onward. |
|
Given https://github.com/nodejs/gyp-next are we ready to close this issue? |
|
Yep, can someone who's keeping track of gyp-next do a PR to sync us with the latest when it's "ready", we have the recent sync but it looks like even more work has gone in. Once that's done let's just push out a version 7. |
So we have https://github.com/refack/GYP, I don't know what its current state is but I know things have been merged here straight into gyp/ that probably conflict with it so I assume we're in a messy out-of-sync state.
nodejs/admin#247 was closed, I guess @refack doesn't want to transfer it to the org. IMO that's in no way a prerequisite to using it as our GYP upstream so I don't consider that a problem if @refack wants to continue maintaining it under his own name.
I'd just like clarity on where GYP changes are going, who approves them, how we keep things in sync here, with an upstream and with nodejs/node.
@refack do you still have the stomach to continue with maintaining it?
@nodejs/python some input from you folks too would be great. This is well outside my comfort zone and I'd rather node-gyp be concerned with wrapping GYP, not being the primary place it's maintained. That separation of concerns would also make issues and PRs here easier to manage and we have an appropriate place to point people who want to make GYP changes instead of everything getting tangled.
The text was updated successfully, but these errors were encountered: