Merge pull request #1246 from eed3si9n/wip/classpath

[2.x] Document classpath changes

src/reference/changes/migrating-from-sbt-1.x.md

@@ -36,6 +36,17 @@ publish / skip := true
 
 In sbt 2.x, bare settings settings should no longer be scoped to `ThisBuild`. One benefit of the new _common settings_ over `ThisBuild` is that it would act in a more predictable delegation. These settings are inserted between plugins settings and those defined in `settings(...)`, meaning they can be used to define settings like `Compile / scalacOptions`, which was not possible with `ThisBuild`.
 
+Migrating to slash syntax
+-------------------------
+
+sbt 1.x supported both the sbt 0.13 style syntax and the slash syntax. sbt 2.x removes the support for the sbt 0.13 syntax, so use the slash syntax for both sbt shell and in `build.sbt`:
+
+```scala
+<project-id> / Config / intask / key
+```
+
+For example, `test:compile` will no longer work on the shell. Use `Test/compile` instead. See [syntactic Scalafix rule for unified slash syntax][syntactic-scalafix-rule-for-unified-slash-syntax] for semi-automated migration of `build.sbt` files.
+
 Cross building sbt plugins
 --------------------------
 
@@ -76,17 +87,6 @@ lazy val plugin = (project in file("plugin"))
   )
 ```
 
-Migrating to slash syntax
--------------------------
-
-sbt 1.x supported both the sbt 0.13 style syntax and the slash syntax. sbt 2.x removes the support for the sbt 0.13 syntax, so use the slash syntax for both sbt shell and in `build.sbt`:
-
-```scala
-<project-id> / Config / intask / key
-```
-
-For example, `test:compile` will no longer work on the shell. Use `Test/compile` instead. See [syntactic Scalafix rule for unified slash syntax][syntactic-scalafix-rule-for-unified-slash-syntax] for semi-automated migration of `build.sbt` files.
-
 Changes to `%%`
 ---------------
 
@@ -102,6 +102,64 @@ libraryDependencies += "org.scala-js" %% "scalajs-dom" % "2.8.0"
 
 Use `.platform(Platform.jvm)` in case where JVM libraries are needed.
 
+
+The PluginCompat technique
+--------------------------
+
+To use the same `*.scala` source but target both sbt 1.x and 2.x, we can create a shim, for example an object named `PluginCompat` in both `src/main/scala-2.12/` and `src/main/scala-3/`.
+
+### Migrating Classpath type
+
+sbt 2.x changed the `Classpath` type to be an alias of the `Seq[Attributed[xsbti.HashedVirtualFileRef]]` type. The following is a shim created to work with classpaths from both sbt 1.x and 2.x.
+
+```scala
+// src/main/scala-3/PluginCompat.scala
+
+package sbtfoo
+
+import java.nio.file.{ Path => NioPath }
+import sbt.*
+import xsbti.{ FileConverter, HashedVirtualFileRef, VirtualFile }
+
+private[sbtfoo] object PluginCompat:
+  type FileRef = HashedVirtualFileRef
+  type Out = VirtualFile
+
+  def toNioPath(a: Attributed[HashedVirtualFileRef])(using conv: FileConverter): NioPath =
+    conv.toPath(a.data)
+  inline def toFile(a: Attributed[HashedVirtualFileRef])(using conv: FileConverter): File =
+    toNioPath(a).toFile()
+  def toNioPaths(cp: Seq[Attributed[HashedVirtualFileRef]])(using conv: FileConverter): Vector[NioPath] =
+    cp.map(toNioPath).toVector
+  inline def toFiles(cp: Seq[Attributed[HashedVirtualFileRef]])(using conv: FileConverter): Vector[File] =
+    toNioPaths(cp).map(_.toFile())
+end PluginCompat
+```
+
+and here's for sbt 1.x:
+
+```scala
+// src/main/scala-2.12/PluginCompat.scala
+
+package sbtfoo
+
+private[sbtfoo] object PluginCompat {
+  type FileRef = java.io.File
+  type Out = java.io.File
+
+  def toNioPath(a: Attributed[File])(implicit conv: FileConverter): NioPath =
+    a.data.toPath()
+  def toFile(a: Attributed[File])(implicit conv: FileConverter): File =
+    a.data
+  def toNioPaths(cp: Seq[Attributed[File]])(implicit conv: FileConverter): Vector[NioPath] =
+    cp.map(_.data.toPath()).toVector
+  def toFiles(cp: Seq[Attributed[File]])(implicit conv: FileConverter): Vector[File] =
+    cp.map(_.data).toVector
+}
+```
+
+Now we can import `PluginCompat.*` and use `toNioPaths(...)` etc to absorb the differences between sbt 1.x and 2.x. The above demonstrates how we can absorb the classpath type change, and convert it into a vector of NIO Paths.
+
   [scala-incompatibility-table]: https://docs.scala-lang.org/scala3/guides/migration/incompatibility-table.html
   [syntactic-scalafix-rule-for-unified-slash-syntax]: https://eed3si9n.com/syntactic-scalafix-rule-for-unified-slash-syntax/
   [tooling-scala2-xsource3]: https://docs.scala-lang.org/scala3/guides/migration/tooling-scala2-xsource3.html

src/reference/changes/sbt-2.0-change-summary.md

@@ -1,3 +1,6 @@
+
+  [Caching Files]: ../concepts/caching.md#caching-files
+
 sbt 2.0 changes (draft)
 =======================
 
@@ -18,6 +21,7 @@ See also [Migrating from sbt 1.x](./migrating-from-sbt-1.x.md).
 - sbt 2.x plugins are published with `_sbt2_3` suffix by [@eed3si9n][@eed3si9n] in [#7671][7671]
 - sbt 2.x adds `platform` setting so `ModuleID`'s `%%` operator can cross build on JVM as well as JS and Native, as opposed to `%%%` operator that was created in a plugin to workaround this issue, by [@eed3si9n][@eed3si9n] in [#6746][6746]
 - Dropped `useCoursier` setting so Coursier cannot be opted out, by [@eed3si9n][@eed3si9n] in [#7712][7712]
+- `Key.Classpath` is changed to be an alias of the `Seq[Attributed[xsbti.HashedVirtualFileRef]]` type, instead of `Seq[Attributed[File]]`. Similarly, some task keys that used to return `File` have changed to return `HashedVirtualFileRef` instead. See [Caching Files].
 
 ### Dropped dreprecations
 

src/reference/concepts/caching.md

@@ -44,6 +44,44 @@ sbt:demo> show someKey
 [success] elapsed time: 0 s, cache 100%, 1 disk cache hit
 ```
 
+### Caching is serializaiton-hard
+
+To participate in the automatic caching, the input keys (e.g. `name` and `version`) must provide a given for `sjsonnew.HashWriter` typeclass and return type must provide a given for `sjsonnew.JsonFormat`. [Contraband](https://www.scala-sbt.org/contraband/) can be used to generate sjson-new codecs.
+
+Caching files
+-------------
+
+Caching files (e.g. `java.io.File`) requires its own consideration, not because it's technically difficult, but mostly because of the ambiguity and assumptions when files are involved. When we say a "file" it could actually mean:
+
+1. Relative path from a well-known location
+2. Materialized actual file
+3. A unique proof of a file, or a content hash
+
+Technically speaking, a `File` just means the file path, so we can deserialize just the filename such as `target/a/b.jar`. This will fail the downstream tasks if they assumed that `target/a/b.jar` would exist in the file system. For clarity, and also for avoiding to capture absolute paths, sbt 2.x provides three separate types for the three cases.
+
+- `xsbti.VirtualFileRef` is used to mean just the relative path, which is equivalent to passing a string
+- `xsbti.VirtualFile` represents a materialized file with contents, which could be a virtual file or a file in your disk
+
+However, for the purpose of hermetic build, neither is great to represent a list of files. Having just the filename alone doesn't guarantee that the file will be the same, and carrying the entire content of the files is too inefficient in a JSON etc.
+
+This is where the mysterious third option, a unique proof of file comes in handy. In addition to the relative path, `HashedVirtualFileRef` tracks the SHA-256 content hash and the file size. This can easily be serialized to JSON yet we can reference the exact file.
+
+### The effect of file creation
+
+There are many tasks that generate file that do not use `VirtualFile` as the return type. For example, `compile` returns `Analysis` instead, and `*.class` file generation happens as a _side effect_ in sbt 1.x.
+
+To participate in caching, we need to declare these effects as something we care about.
+
+```scala
+someKey := Def.cachedTask {
+  val conv = fileConverter.value
+  val out: java.nio.file.Path = createFile(...)
+  val vf: xsbti.VirtualFile = conv.toVirtualFile(out)
+  Def.declareOutput(vf)
+  vf: xsbti.HashedVirtualFileRef
+}
+```
+
 Remote caching
 --------------