The Closure Templates to Incremental Dom compiler (SoyToIncrementalDomSrcCompiler) was "Unopensourced"!

[jimbojetlag]

It seems like over 6 months ago Google "unopensourced" (is there really such a verb?) SoyToIncrementalDomSrcCompiler here 2081efa:

GITHUB_BREAKING_CHANGES=The Closure Templates to Incremental Dom compiler no longer exists. It is undergoing a rewrite.

There is no sign of this in closure-template nor incremental-dom documentations.

Would the authors care to explain if the compiler is going through a rewrite, what is the point of "unopensourcing" it? What is the harm of having the compiler source in the master when rewriting is happening?

You understand that when you open source your software, the community relies on it. People started building tools and code on top of your open source software. You cannot just "unopensource" your software with a single commit, leaving no comments in the documentation, nor a roadmap about what is going to happen next.

[iteriani]

The biggest issue on our end was that we had internal references to code generated by tsickle (TS -> Closure). At the same time, we also had a internal rewrite of Incremental DOM itself to TypeScreipt. In general we have no idea how to manage this in open source. Do we just release the .closure.js files?

[jimbojetlag]

Thank you for the update @iteriani!

The biggest issue on our end was that we had internal references to code generated by tsickle (TS -> Closure).

Is this the Typescript code for Closure Template?

At the same time, we also had a internal rewrite of Incremental DOM itself to TypeScreipt. In general we have no idea how to manage this in open source.

I did not know that Incremental DOM itself is going through a rewrite, I thought it was only the compiler. Would that make https://github.com/google/incremental-dom deprecated?

Do we just release the .closure.js files?

I'd say open sourcing .closure.js files would be very welcome, if there is no way to open source the Typescript source itself. May I ask the reason for not being able to open source the Typescript rewrite of IDOM? If there are any internal references in the Typescript version, wouldn't we have the same internal references in the generated .closure.js version?

[iteriani]

1. The current situation is that we have TypeScript files checked in that we transpile to Closure JS files using tsickle and use from the JS code generated from Soy.

2. The TypeScript migration is already done. The source in that repo is up to date with google3.

3. We can just open source the TypeScript source itself, but we have no idea how people are supposed to use them. Inside google3 we have blaze/bazel and it just somehow works.

[jimbojetlag]

We can just open source the TypeScript source itself, but we have no idea how people are supposed to use them. Inside google3 we have blaze/bazel and it just somehow works.

So the Typescript code is not open sourced because of the internal references in the build tools. I assume this is related to this commit of https://github.com/bazelbuild/rules_closure.

If you could open source the Typescript source, and the corresponding .closure.js files, the open source community would be able to come up with an approach to build the source code with available open source tools.

/cc @jplaisted @mprobst

[iteriani]

Correct. Currently the compiler will have references such as

"const idomUtils = goog.require('google3.javascript.template.soy.idomUtils')"

which is an artifact of google3's build setup. I guess what I can do is move this to a java flag and have people configure the goog namespace themselves.

[jimbojetlag]

I guess what I can do is move this to a java flag and have people configure the goog namespace themselves.

That sounds like a good idea. If that namespace is supposed to be project specific, then it should be a parameter of the build system.

[LouisStAmour]

As an outsider who's been watching a lot of BazelCon videos, I'd like to chime in here with my +1.

Bazel is an example of Google leading the way on open sourcing their internal build tools -- they're actively trying to support external uses and even new alpha Google Cloud services have been announced (for remote builds).

From there, well, there are differences between Bazel and Blaze, so it's possible the code won't immediately run under Bazel if it was only tested with Blaze. But I'd bet by now there are ways to get things working externally -- the community is very creative. For example, I managed to figure out enough Bazel to get TypeScript and Closure to work together, in the absence of official documentation on how to do so: https://github.com/bazelbuild/rules_typescript/issues/344

But updating the namespaces for this TS code, it seems to me that a quick shell script could be written to help sync internal commits and rename internal namespaces to the expected external open source ones for the external repo to distribute. I'm sure if there were Travis CI errors, the community would happily offer up PRs. Now, handling external PRs in a timely fashion when merging against an internal codebase, that's something the Bazel team is struggling with right now. They appear to have a lot of Google3 repo code they're actively working on, so it takes effort to transcribe a PR from external sources to internal ones, and to make sure all the internal tests and uses still pass, as the code Bazel is extracted from is still closed source and proprietary due to ... legacy or internal uses, I guess?

The goal, as I understand it, is that Bazel will replace Blaze at some future date, but they didn't have the advantage that some teams have, of starting development in a completely public way first. I'm getting all this from a conference talk - https://www.youtube.com/watch?v=QxZ58YW4XYU

Similarly ... if it were easily done, to mark certain namespaces in Google3 as publicly available and mirroring it onto Github, well, it's better than the alternative, which is waiting for React 19 to get async rendering and perhaps start on perf improvements from Web Components, but they'll be held back from doing so by their legacy internal FB code and the wider community. Vue has made some strides, Stencil too, but I'd like to still see an "official" way forward for simple scripts and toolchains to support modern, Chrome-performant web development using Bazel. Esoteric, I know, but if it's good enough for a few teams at Google, lightweight enough that I can read all of it in an hour or two, I'll certainly consider it. And if adopted by a few teams, well, documentation PRs will inevitably follow as we'll need to tell other co-workers how this tech works. ;-)

For those discovering this issue from Google, see also: @sparhami 's excellent write-up at google/incremental-dom#352 (comment) and documentation at https://github.com/bazelbuild/rules_closure/#closure_js_template_library which in otherwise excellent documentation marks incremental_dom support as experimental for reasons.

[iteriani]

At a high level we're actually lacking the basics here.

We have no idea how to build an application using the Soy Java compiler, the TypeScript compiler, tsickle + clutz, even in a bazel environment. Inside google3, we have proper build rule support and so it's very easy to wire these things together. We have no experience outside.

[iteriani]

Update to this:

We're going to update the github repo to the latest version inside google3 and then put out what we have on github (either TS or tsickle output). After that I'm not really sure what we'd do from then on.

[iteriani]

I've exported everything ~ we have. I still don't know how to make the tsickle/clutz knobs work, but this should close this bug.

[jimbojetlag]

Thank you @iteriani.

I still don't know how to make the tsickle/clutz knobs work

Does this need a new issue to track? Anyone from Google that can help with this?

[iteriani]

Feel free to open an issue. I'm still working on releasing to maven and npm (and docs)

[iteriani]

Our next release puts Incremental DOM + related libraries into

https://github.com/google/closure-templates/blob/master/javascript/soy_requirements_onefile.js

This isn't really that great of a solution, but it should work.

[jimbojetlag]

Thank you @iteriani for your efforts. When should we expect the next maven release?

[iteriani]

It should be or already is out.

…

On Mon, Mar 11, 2019, 8:29 PM jimbojetlag ***@***.***> wrote: Thank you @iteriani <https://github.com/iteriani> for your efforts. When should we expect the next maven release? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#169 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADTBG9Sq0CygAAJRpWa5kj-M4WjeYxUsks5vVx81gaJpZM4YKnuN> .