Change native builds from Gradle to cmake (tracking issue)

[PeterJohnson]

The current Gradle-based infrastructure for native builds (C++) is bespoke to WPILib and has limited numbers of people who can maintain it. It also has limitations, e.g. it has no built-in support for transitive dependencies. Major updates will eventually be required to either support C++ modules or if Gradle breaks a major component we rely on--the native infrastructure in upstream Gradle has substantially changed, with the new version not supporting our needs; the old version is still available but could be removed in the future.

We would like to move allwpilib, third-party dependencies such as OpenCV, and robot projects to using build tools that are the industry-standard for their respective languages. Gradle is effectively the industry standard for Java, and cmake is the industry standard for C++. It is explicitly not a goal to transition Java builds to cmake. The complicated part is that Java builds (and potentially other languages in the future) require native build artifacts, and we need to be able to build and distribute native binary artifacts as well. The current approach for this is native Maven artifacts--most likely the easiest path forward would be to continue to use Maven for this purpose. Using Maven would also enable gradual transitions of the multiple components of the system (e.g. allwpilib could transition while robot projects are still Gradle-based). JNI support also requires the cmake build to have at least some Java support in order to build the JNI native components for later Java consumption.

Migrating C++ robot projects to a non-Gradle build system will require additional effort. In particular, a language-agnostic standalone deploy tool will be needed (that can be called from any build system) to discover/verify the target platform, deploy the compiled user program, deploy updated shared libraries, and deploy other user files. Robot projects also need the ability to easily target either a cross-build to the Rio or a local (simulation) build, with seamless IDE integration (e.g. today's "Deploy Robot Code" and "Simulate Robot Code" in vscode). They also need to be able to seamlessly pull required dependencies from local folders when no internet access is available. In addition, a new build system will need to support vendordep files (or something equivalent) for communicating potentially complex native library requirements in a single separate file per vendor, so users don't need to edit build files directly in most cases.

For allwpilib, it is important that the "git clone" build experience for local builds is straightforward. Currently (see https://github.com/wpilibsuite/allwpilib#building-wpilib), minimal dependencies are required on all target platforms (Windows, Linux, Mac), and just a single command (gradlew build) will build the entirety of allwpilib from a fresh clone. Supporting cross-builds only requires one additional command (to install the toolchain, which only needs to be done once). Transitioning to cmake for the native build, while maintaining Gradle for the Java build, will require a higher-level build script that builds the native components to artifacts (in the build directory) that Gradle can then pick up. The only new build tool dependency we would want to add would be cmake itself.

TODO for allwpilib:

• Add support to cmake build for pulling third-dependencies such as OpenCV from Maven, rather than using vcpkg or system libraries. However, we don't want to lose the capability for the cmake build to use system libraries instead, as this is critical for builds on random coprocessors without internet access or where we don't provide artifacts for that specific target.
• Add support to cmake build to make cross-builds easy (including toolchain installation)
• Add support to cmake build for publishing built Maven artifacts into a project-local build location consistent with the Maven standard layout (so they can be consumed by Gradle).
• Update Gradle build to pull artifacts from project-local build folder.
• Create top-level build script that runs both cmake and Gradle to support an end-to-end build.

[pjreiniger]

I feel like I must again at least mention that I have bazel set up to build wpilib on the three main OS's for both java and c++, automatically install the 4 toolchains and run cross compilers for all the permutations, stuff to bring in vendor code simply.

I've also experimented with building robotpy completely with bazel with some success, and great success building the other open source vendordeps, including the chroeo suite that uses Rust.

Bazel supports overriding dependencies to point to a different places, including a local copy which can work for offline installs. This is also handy if you are testing wpilib changes against another projects code (like a vendor or a robot project), you can skip all the intermediate build / installs whenever you make a change to the dependency, and build everything in one step.

This is roughly the commands we would need to work with my teams robot in our monorepo. They could easily be wrapped in a VSCode extension, and autodetect the current OS for simulation.

bazel run //y2024/Crescendo:deploy --config=roborio

bazel run //y2024/Crescendo:simulate --config=windows

# To build eveything, which is years of robot projects, Shuffleboard plugins, and python tools.
bazel test //... --config=<whatever>

Other bazel pros:

• The remote caching is absurdly good. Set up a config in your repo, no need to install other tools.
• First class language support for java, c++, and python (sounds familiar) out of the box
• Built specifically with monorepos in mind. There are dozens of articles about companies (including mine) about using bazel with gigantic monorepos and the huge quality of life benefits it brought
• Designed to support multiples languages. i.e. C#, Kotlin, Perl, Ruby, Rust, JavaSript, etc. Those were the easiest to find because https://github.com/bazelbuild owns all of them, but more more exist.

[AustinSchuh]

@PeterJohnson , is there a decent list of the dependencies that are needed?

[spacey-sooty]

@PeterJohnson , is there a decent list of the dependencies that are needed?

Dependencies needed for what?

[pjreiniger]

allwpilib depends on opencv and libssh (for tools). It used to externally depend on googletest, apriltag, and imgui stuff, but those have recently been vendored into the repository.

[spacey-sooty]

It depends on a lot more than that (Jackson etc)

[pjreiniger]

I left off all of the java dependencies. In gradle / cmake / bazel those are all far more trivial than c++ deps. But yes, there are a slew of java deps.

[PeterJohnson]

We will also be adding external deps for wpical (#6915), in particular SuiteSparse and Ceres.