Support incremental compilation of other JVM languages

[vasilmkd]

Hello everyone 👋🏻

Definitions

Let me start by defining what I mean by "other JVM languages", any programming language that compiles to JVM bytecode and is different from Scala and Java.

I'm mainly considering Kotlin, Groovy and Clojure here. Coincidentally, all three languages have basic plugins that enable simple compilation of source files written in the three languages.

1. sbt-kotlin-plugin and the JetBrains fork.
2. sbt-groovy
3. sbt-clojure

Current state of sbt support for these languages

All three plugins are implemented very similarly, an arbitrary choice has been made to run them either before or after Compile / compile, and they are only concerned with compiling *.kt, *.groovy and *.clj source files respectively. In the case of Kotlin, all Java sources in the same source set (sbt subproject) are also passed in, because Kotlin, like Scala, is able to produce Java stubs and enable mixed Kotlin+Java compilation where source files can circularly refer to each other. I haven't verified if this is the case in the Groovy and Clojure plugins, but I can only assume something like this is possible. In any case, this isn't very relevant, I'm just describing what extent of language compilation support these plugins provide. Finally, just using these plugins for a couple of times in an sbt project, reveals their current biggest shortcoming. They support no incremental compilation. Every time compile is executed, they recompile all *.kt, *.groovy and *.clj sources that these plugins manage in a given sbt subproject. Let me be clear, this is not a criticism of the plugins at all, I'm just pointing out the state of the compile support as it is very relevant to this discussion.

Usage of sbt-kotlin-plugin at JetBrains

As an aside, we make use of sbt-kotlin-plugin at JetBrains to build the Scala Plugin for IntelliJ IDEA. Due to recent changes in the IntelliJ IDEA platform, some specific features need to be implemented in Kotlin source code (mainly where Kotlin coroutines are required), and thus, the Scala Plugin for IDEA .zip distribution contains some classes compiled by kotlinc. sbt-kotlin-plugin, as it stands now is still useful, especially in the CI where we build the Scala Plugin as a batch compilation job, but nevertheless we're facing the issue of Kotlin sources being always recompiled on every compile in sbt.

How are Scala and Java special

In Zinc, Scala and Java have are first-class citizens, both enjoying two very special and useful compilation features.

1. Mixed compilation, where Scala and Java sources are able to circularly reference each other and scalac is able to untangle this and compile the sources correctly, with javac taking care of the Java sources at the end. Please correct me if I'm wrong, I believe this feature is mainly implemented in scalac itself, but nevertheless, Zinc takes it into account and enables this feature at the build tool level.
2. Incremental compilation. I urge people to read the great sbt docs on the topic.

Can we do anything about other JVM languages?

Of course, we have to start with the obvious limitation. Mixed compilation is simply not supported between Scala and any other JVM language which is not Java. If such thing were to exist, it would have to be handcrafted and maintained in scalac. This is out of the scope of this discussion.

I'm more interested in discussing ways to include other JVM languages into the incremental compilation algorithm in Zinc, at the binary dependency level (*.class files). Without loss of generality, I'll be referring to Kotlin in this example, but it applies to the other JVM languages too.

Essentially, users would have to choose something along the lines of CompileOrder.KotlinThenScala or CompileOrder.ScalaThenKotlin. This means that either kotlinc will run first, or scalac+javac will run first, and Kotlin sources will be able to refer to Scala symbols, or vice versa, respectively. When any *.class files get invalidated and their sources recomputed, these would be taken into consideration for invalidating other classes that depend on these sources/outputs.

Some ideas

Here I will list out some basic ideas about how this integration/support might look like. Please feel free to shut any of these down, if you think they are too simplistic/unrealistic.

I was looking at sbt.internal.inc.javac.AnalyzingJavaCompiler. Would it be possible to have an AnalyzingKotlinCompiler, which, like the AnalyzincJavaCompiler, receives a sequence of Kotlin sources, compiles them and runs sbt.internal.inc.classfile.JavaAnalyze$#apply on the generated classes. Again, it's important to stress, I'm not looking for granular Kotlin/Groovy/Clojure incremental compilation at the source level, I'm only aiming for incremental compilation at the JVM bytecode level. This should be enough to create basic Zinc Analysis for the outputs created by the compilers.

Subsequently, the same Zinc Analysis should factor into the incremental compilation algorithm in Zinc, and be used to produce a list of source files, which not only include *.scala and *.java sources, but also *.kt, *.groovy, *.clj or any source types managed by other "analyzing compilers" that will be provided by (sbt) plugins.

Discussion

I've been kind of stuck on this issue for a few months already, without much progress. There is a tracking ticket here. I'm opening this discussion with the hope of gathering as much wisdom from the community as possible and possibly look for a more general integration that will benefit multiple languages. Otherwise, I will probably try to push the ideas from above as far as I will be able to, which probably involves copying and repurposing a lot of Zinc code.

Looking forward to your replies. Thank you in advance.

[eed3si9n]

I was looking at sbt.internal.inc.javac.AnalyzingJavaCompiler. Would it be possible to have an AnalyzingKotlinCompiler, which, like the AnalyzincJavaCompiler, receives a sequence of Kotlin sources, compiles them and runs sbt.internal.inc.classfile.JavaAnalyze$#apply on the generated classes. Again, it's important to stress, I'm not looking for granular Kotlin/Groovy/Clojure incremental compilation at the source level, I'm only aiming for incremental compilation at the JVM bytecode level. This should be enough to create basic Zinc Analysis for the outputs created by the compilers.

Given that at the bytecode level Scala, Java, or Kotlin would look the same, we would still need to track the relationship between the Kotlin source and the generated *.class, so if Kotlin generates example.A and example.B that uses example.A, Zinc can invalidate example/B.kt when example/A.kt changes. But the general strategy of analyzing *.class file instead of injecting custom phases might work ok enough?

The tricky thing about incremental compilation is that if it over-compiles it's a bit annoying, but if it under-compiles, it's broken. So if any of the non-Java languages have type system etc that might not show up as bytecode, but nonetheless important to the UX of the language, it might end up needing all the bells and whistles Zinc adds to scalac. But I guess we can worry about those issues, if it's found to be an issue.

[vasilmkd]

The tricky thing about incremental compilation is that if it over-compiles it's a bit annoying, but if it under-compiles, it's broken.

I completely agree. Which is exactly why I've been postponing even thinking about this problem.

Given that at the bytecode level Scala, Java, or Kotlin would look the same, we would still need to track the relationship between the Kotlin source and the generated *.class, so if Kotlin generates example.A and example.B that uses example.A, Zinc can invalidate example/B.kt when example/A.kt changes. But the general strategy of analyzing *.class file instead of injecting custom phases might work ok enough?

I assume .class -> .java is tracked for Java using JavaAnalyze.apply?

[vasilmkd]

In any case, this indeed might be too risky/unstable to promote as an actual feature of Zinc and maybe should only be confined within sbt plugins.

A problem I'm facing in the experimentation is how to obtain the same AnalysisCallback instance that sbt/Zinc Scala+Java incremental compilation uses, such that sources/classes from other languages can be tracked in the same Zinc Analysis instance as the ones produced from Scala and Java sources.

Or maybe I just need to manually add these things to the analysis...

[eed3si9n]

One potential approach we can think about is treating *.kt as another dialect of Scala instead of Java. In that view you just need to implement the compiler bridge like Scala 2.x or 3.x does and callback from within the compilation process. This would mean that you'd have to split any *.kt target as a standalone JAR, but you can always use sbt-assembly to mash JARs together if you want to.

[vasilmkd]

Hi there. I just wanted to let everyone know, I've implemented tracking of compilation inputs for Kotlin sources, exactly using the JavaAnalyze.apply API. It seems to be working very well at the moment. As soon as any binary dependencies, as well as direct sources are invalidated, the modules containing Kotlin sources will be recompiled.

Some details here. Zinc is able to exactly track to the specific Kotlin source file, only the minimum set of necessary sources to recompile. However, the way kotlinc is invoked in sbt-kotlin-plugin, it is not able to correctly finish this incremental compilation run in every situation. There are some Kotlin language features which seem to require more compiler knowledge to pull off.

Supposedly, these are available through the (still experimental) Kotlin build tools API. We should be able to look into the Kotlin plugin for Gradle for some inspiration on how to continue. For now, for the Scala folks at JetBrains, the current state is enough. We do not have many Kotlin sources, and were mainly interested in avoiding forced recompilation on every sbt compile command. We have achieved this for now. Additional work will be done if/when it becomes necessary.

Thanks everyone for the help!

You can try out the new sbt-kotlin-plugin containing these changes with:

addSbtPlugin("org.jetbrains.scala" % "sbt-kotlin-plugin" % "3.1.1")

[vasilmkd]

The Kotlin compiler generates a target/classes/META-INF/main.kotlin_module binary file during compilation. However, this file is currently not registered anywhere. When remote caching is used, the file gets included in the cache artifact. But when the artifact is restored, this file is skipped and not copied over from the artifact to the target/classes/META-INF directory. Any ideas how to register this file as an official Kotlin compilation product? Thanks in advance.

cc @eed3si9n

[eed3si9n]

When remote caching is used

Is this for sbt 1.x style pullRemoteCache?

[vasilmkd]

Yes, sorry, should have specified.

Essentially, the file is not recorded in copy-resources.txt, and when the cache artifact is unpacked, this file is skipped.

[eed3si9n]

I don't remember all the details, but my guess is that it needs to be specified here? - https://github.com/sbt/sbt/blob/234781bc096df760e576938d599502083e96c708/main/src/main/scala/sbt/internal/RemoteCache.scala#L165-L197