diff --git a/docs/src/guides/languages/postgresql.md b/docs/src/guides/languages/postgresql.md index 793865e29..3c31d5ad1 100644 --- a/docs/src/guides/languages/postgresql.md +++ b/docs/src/guides/languages/postgresql.md @@ -120,7 +120,7 @@ Since we need migration and seed data files, we'll inherit from the `builder` target. The actual image build process is pretty straight-forward and fully defined under the `+BUILD` UDC target. -The only thing it is needed to specify is a few arguments: +The only thing it is needed to specify is a few arguments: * `tag` - the tag of the image, default value `latest`. * `registry` - the registry of the image. diff --git a/docs/src/guides/languages/rust.md b/docs/src/guides/languages/rust.md index b15f880bc..cb16f2ff1 100644 --- a/docs/src/guides/languages/rust.md +++ b/docs/src/guides/languages/rust.md @@ -9,4 +9,216 @@ tags: # :simple-rust: Rust -[TODO](https://github.com/input-output-hk/catalyst-ci/issues/78) + + +## Introduction + + +!!! Tip + If you're just looking for a complete example, + [click here](https://github.com/input-output-hk/catalyst-ci/blob/master/examples/rust/Earthfile). + This guide will provide detailed instructions for how the example was built. + + +This guide will get you started with using the Catalyst CI to work with Rust based projects. + +To begin, clone the Catalyst CI repository: + +```bash +git clone https://github.com/input-output-hk/catalyst-ci.git +``` + +Navigate to `examples/rust` to find a basic Rust project, with the `Earthfile` in it. +This is the `Earthfile` we will be building in this guide. +You can choose to either delete the file and start from scratch, +or read the guide and follow along in the file. + +Also we will take a look how we are setup Rust projects and what configuration is used. + +## Building the Earthfile + + +!!! Note + The below sections will walk through building our `Earthfile` step-by-step. + In each section, only the fragments of the `Earthfile` relative to that section are displayed. + This means that, as you go through each section, you should be cumulatively building the `Earthfile`. + If you get stuck at any point, you can always take a look at the + [example](https://github.com/input-output-hk/catalyst-ci/blob/master/examples/rust/Earthfile). + + +### Prepare base builder + +```Earthfile +VERSION 0.7 + +# Set up our target toolchains, and copy our files. +builder: + FROM ./../../earthly/rust+rust-base + + COPY --dir .cargo .config benches src tests . + COPY Cargo.lock Cargo.toml . + COPY clippy.toml deny.toml rustfmt.toml . + + DO ./../../earthly/rust+SETUP +``` + +The first target `builder` is responsible for preparing an already configured Rust environment, +instal all needed tools and dependencies. + +The fist step of the `builder` target is to prepare a Rust environment via `+rust-base` target. +Next step is to copy source code of the project. +Note that you need to copy only needed files for Rust build process, +any other irrelevant stuff should omitted. +And finally finalize the build with `+SETUP` UDC target. +The `+SETUP` UDC target requires `rust-toolchain.toml` file, +with the specified `channel` option in it. +This `rust-toolchain.toml` file could be specified +via the `toolchain` argument of the `+SETUP` target like this +with defining the specific location of this file with the specific name. +By default `toolchain` setup to `rust-toolchain.toml`. + +### Running checks + +```Earthfile +# Test rust build container - Use best architecture host tools. +check-hosted: + FROM +builder + + DO ./../../earthly/rust+CHECK + +# Test which runs check with all supported host tooling. Needs qemu or rosetta to run. +# Only used to validate tooling is working across host toolsets. +check-all-hosts: + BUILD --platform=linux/amd64 --platform=linux/arm64 +check-hosted + +## Standard CI targets. +## +## These targets are discovered and executed automatically by CI. + +# Run check using the most efficient host tooling +# CI Automated Entry point. +check: + FROM busybox + # This is necessary to pick the correct architecture build to suit the native machine. + # It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation. + # All target implementation of this should follow this pattern. + ARG USERARCH + + IF [ "$USERARCH" == "arm64" ] + BUILD --platform=linux/arm64 +check-hosted + ELSE + BUILD --platform=linux/amd64 +check-hosted + END +``` + +With prepared environment and all data, we're now ready to start operating with the source code and configuration files. +The `check-hosted` target which actually performs all checks and validation +with the help of `+CHECK` UDC target. +The `+CHECK` UDC target performs static checks of the Rust project as +`cargo fmt`, `cargo machete`, `cargo deny` which will validate formatting, +find unused dependencies and any supply chain issues with dependencies. +Here is the list of steps (look at `./earthly/rust/scripts/std_checks.sh`): + +1. `cargo fmtchk` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias), +look at `./earthly/rust/stdcfgs/config.toml`)Checking Rust Code Format. +2. Checking configuration files for consistency. +3. `cargo machete` - Checking for Unused Dependencies. +4. `cargo deny check` - Checking for Supply Chain Issues. + +As it was mentioned above it validates configuration files as +`.cargo/config.toml`, `rustfmt.toml`, `.config/nextest.toml`, `clippy.toml`, `deny.toml` +to be the same as defined in `earthly/rust/stdcfgs` directory of the `catalyst-ci` repo. +So when you are going to setup a new Rust project copy these configuration files +described above to the appropriate location of your Rust project. + +Another targets as `check-all-hosts` and `check` (running on CI) just invoke `check-hosted` +with the specified `--platform`. +It is important to define a `linux` target platform with a proper cpu architecture +for the Rust project when you are building it inside Docker +and check the build process with different scenarios. +The same approach we will see for the another targets of this guide. + +### Build + +```Earthfile +# Build the service. +build-hosted: + FROM +builder + + DO ./../../earthly/rust+BUILD + + DO ./../../earthly/rust+SMOKE_TEST --bin=hello_world + + SAVE ARTIFACT target/$TARGETARCH/doc doc + SAVE ARTIFACT target/$TARGETARCH/release/hello_world hello_world + +# Test which runs check with all supported host tooling. Needs qemu or rosetta to run. +# Only used to validate tooling is working across host toolsets. +build-all-hosts: + BUILD --platform=linux/amd64 --platform=linux/arm64 +build-hosted + +# Run build using the most efficient host tooling +# CI Automated Entry point. +build: + FROM busybox + # This is necessary to pick the correct architecture build to suit the native machine. + # It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation. + # All target implementation of this should follow this pattern. + ARG USERARCH + + IF [ "$USERARCH" == "arm64" ] + BUILD --platform=linux/arm64 +build-hosted + ELSE + BUILD --platform=linux/amd64 +build-hosted + END +``` + +After successful performing checks of the Rust project we can finally build artifacts. +As it was discussed in the previous section, actual job is done with `build-hosted` target, +other targets needs to configure different platform running options. +So we will focus on `build-hosted` target. +Obviously it inherits `builder` target environment and than performs build of the binary. +Important to note that in this particular example we are dealing with the executable Rust project, +so it produces binary as a final artifact. +Another case of the building Rust library we will consider later. +Actual build process is done with `+BUILD` UDC target. +Under this build process we perform different steps of compiling and validating of our Rust project, +here is the list of steps (look at `./earthly/rust/scripts/std_build.sh`): + +1. `cargo build` - Building all code in the workspace. +2. `cargo lint` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias), +look at `./earthly/rust/stdcfgs/config.toml`) +Checking all Clippy Lints in the workspace. +3. `cargo docs` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias), +look at `./earthly/rust/stdcfgs/config.toml`)Checking Documentation can be generated OK. +4. `cargo testci` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias), +look at `./earthly/rust/stdcfgs/config.toml`)Checking Self contained Unit tests all pass. +5. `cargo testdocs` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias), +look at `./earthly/rust/stdcfgs/config.toml`)Checking Documentation tests all pass. +6. `cargo bench` - Checking Benchmarks all run to completion. + +Next steps is mandatory if you are going to produce a binary as an artifact, +for Rust libraries the are not mandatory and could be omitted. +The `+SMOKE_TEST` UDC target checks that produced binary with the specified name (`--bin` argument) +is executable, isn't a busted mess. + +Final step is to provide desired artifacts: docs and binary. + +### Test + +As you already mentioned that running of unit tests is done during the `build` process, +but if you need some integration tests pls follow how it is done for [PostgreSQL builder](./postgresql.md), +Rust will have the same approach. + +### Release and publish + +To prepare a release artifact and publish it to some external container registries +please follow this [guide](./../../onboarding/index.md). +It is pretty strait forward for this builder process, +because as a part of `+build` target we already creating a docker image. + +## Conclusion + +You can see the final `Earthfile` [here](https://github.com/input-output-hk/catalyst-ci/blob/master/examples/rust/Earthfile) +and any other files in the same directory. +We have learnt how to maintain and setup Rust project, as you can see it is pretty simple. diff --git a/earthly/python/Readme.md b/earthly/python/Readme.md index e19e741ab..907e2d8c4 100644 --- a/earthly/python/Readme.md +++ b/earthly/python/Readme.md @@ -1,3 +1,4 @@ + # Python Earthly Build Containers and UDCs This repo defines common python targets and UDCs for use with python. @@ -33,3 +34,4 @@ The directory that contains the Earthfile that invokes this UDC MUST have: Up-to-date by running: * `poetry lock --no-update` : Update lock file, but do not update dependencies; or * `poetry lock` : Update lock file and dependencies as required. + \ No newline at end of file diff --git a/earthly/rust/Earthfile b/earthly/rust/Earthfile index 2d9dac88a..2088eb47a 100644 --- a/earthly/rust/Earthfile +++ b/earthly/rust/Earthfile @@ -1,10 +1,8 @@ # Common Rust UDCs and Builders. VERSION 0.7 -# cspell: words rustup rustc automake autotools xutils miri nextest kani -# cspell: words TARGETPLATFORM TARGETOS TARGETARCH TARGETVARIANT -# cspell: words USERPLATFORM USEROS USERARCH USERVARIANT -# cspell: words ripgrep colordiff rustfmt stdcfgs toolset toolsets readelf +# cspell: words rustup miri nextest ripgrep colordiff rustfmt stdcfgs toolset readelf +# cspell: words TARGETPLATFORM TARGETOS TARGETARCH TARGETVARIANT USERPLATFORM USEROS USERARCH USERVARIANT # Base Rustup build container. rust-base: @@ -17,10 +15,9 @@ rust-base: ARG USERARCH ARG USERVARIANT - # This is our base Host toolset, and rustup. # The ACTUAL version of rust that will be used, and available targets - # is controlled by a `rust-toolchain.toml` file when the RUST_SETUP UDC is run. + # is controlled by a `rust-toolchain.toml` file when the `SETUP` UDC is run. # HOWEVER, It is enforced that the rust version in `rust-toolchain.toml` MUST match this version. FROM rust:1.73-alpine3.18 @@ -33,7 +30,6 @@ rust-base: echo "USERARCH = $USERARCH"; \ echo "USERVARIANT = $USERVARIANT"; - WORKDIR /root # Install necessary packages @@ -84,17 +80,27 @@ rust-base: rust-base-all-hosts: BUILD --platform=linux/amd64 --platform=linux/arm64 +rust-base -# Common Rust setup +CP_SRC: + # Copy the build src using this. + COMMAND + # This can be one directory like `"src/*"` or multiple src separated by `","`: + # eg, "example/Cargo.toml, example/Cargo.lock, example/src" + ARG src="" + + # ONLY copy whats needed to build. Not everything. + # Note: rust-toolchain.toml was already copied when rust was setup. + # Minimizing whats copied reduces needless cache misses. + FOR --sep=", " file IN $src + COPY --dir $file . + END + +# Common Rust setup. # Parameters: # * toolchain : The `rust-toolchain` toml file. SETUP: COMMAND ARG toolchain=./rust-toolchain.toml - # Poetry Installation directory. - # Gives poetry and our poetry project a reliable location. - WORKDIR /build - # Copy our toolchain dependency. COPY $toolchain ./rust-toolchain.toml ENV default_rust_channel=$(rg -oP 'channel\s*=\s*"\K[^"]+' rust-toolchain.toml) @@ -110,22 +116,6 @@ SETUP: cargo --version && \ cargo +nightly --version - -CP_SRC: - # Copy the build src using this. - COMMAND - # This can be one directory like `"src/*"` or multiple src separated by `","`: - # eg, "example/Cargo.toml, example/Cargo.lock, example/src" - ARG src="" - - # ONLY copy whats needed to build. Not everything. - # Note: rust-toolchain.toml was already copied when rust was setup. - # Minimizing whats copied reduces needless cache misses. - FOR --sep=", " file IN $src - COPY --dir $file . - END - - # Steps we do during the `check` CI phase for all Rust programs CHECK: COMMAND @@ -135,11 +125,16 @@ CHECK: # to pass without needing to iterate excessively. RUN /scripts/std_checks.sh +# Step we do during the `build` CI phase for all Rust programs +BUILD: + COMMAND + + RUN /scripts/std_build.sh + # Check if the build executable, isn't a busted mess. SMOKE_TEST: COMMAND - ARG TARGETPLATFORM - ARG bin + ARG --required bin RUN ldd target/$TARGETARCH/release/$bin RUN readelf -p .comment target/$TARGETARCH/release/$bin @@ -147,102 +142,3 @@ SMOKE_TEST: # ALL executables MUST have `--help` as an option. RUN target/$TARGETARCH/release/$bin --help - - -# Set up our target toolchains, and copy our files. -builder: - FROM +rust-base - - DO +SETUP --toolchain=example/rust-toolchain.toml - - DO +CP_SRC --src="example/*" - -# Test rust build container - Use best architecture host tools. -check-hosted: - FROM +builder - - DO +CHECK - -# Test which runs check with all supported host tooling. Needs qemu or rosetta to run. -# Only used to validate tooling is working across host toolsets. -check-all-hosts: - BUILD --platform=linux/amd64 --platform=linux/arm64 +check-hosted - -build-hosted: - ARG TARGETPLATFORM - - # Build the service - FROM +builder - - RUN /scripts/std_build.sh - - DO +SMOKE_TEST --bin=hello_world - - SAVE ARTIFACT target/$TARGETARCH/doc doc - SAVE ARTIFACT target/$TARGETARCH/release/hello_world hello_world - -# Test which runs check with all supported host tooling. Needs qemu or rosetta to run. -# Only used to validate tooling is working across host toolsets. -build-all-hosts: - BUILD --platform=linux/amd64 --platform=linux/arm64 +build-hosted - -## ----------------------------------------------------------------------------- -## -## Standard CI targets. -## -## These targets are discovered and executed automatically by CI. - -# Run check using the most efficient host tooling -# CI Automated Entry point. -check: - FROM busybox - # This is necessary to pick the correct architecture build to suit the native machine. - # It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation. - # All target implementation of this should follow this pattern. - ARG USERARCH - - IF [ "$USERARCH" == "arm64" ] - BUILD --platform=linux/arm64 +check-hosted - ELSE - BUILD --platform=linux/amd64 +check-hosted - END - -# Run build using the most efficient host tooling -# CI Automated Entry point. -build: - FROM busybox - # This is necessary to pick the correct architecture build to suit the native machine. - # It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation. - # All target implementation of this should follow this pattern. - ARG USERARCH - - IF [ "$USERARCH" == "arm64" ] - BUILD --platform=linux/arm64 +build-hosted - ELSE - BUILD --platform=linux/amd64 +build-hosted - END - - -# This step will build any packages we would intend to publish or integration test. -#package: -# FROM scratch - -# Run integration tests on all packages built during the `package` step. -#test: -# FROM scratch - -# Publish packages if all integration tests have passed. -# (Failure to pass tests will prevent packages being published.) -# Don't have a `publish` target if nothing to actually publish. -#publish: -# FROM scratch - -## ----------------------------------------------------------------------------- - -# This step simulates the full CI run for local purposes only. -local-ci-run: - BUILD +check - BUILD +build - # BUILD +package - # BUILD +test - # BUILD +publish diff --git a/earthly/rust/example/README.md b/earthly/rust/example/README.md deleted file mode 100644 index 81a6e9a14..000000000 --- a/earthly/rust/example/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# Example Rust Project - -This is an simple but hopefully full featured example rust project. -It is used to test both our Catalyst-CI Rust building infrastructure and serve as a template for its use. diff --git a/earthly/rust/example/.cargo/config.toml b/examples/rust/.cargo/config.toml similarity index 100% rename from earthly/rust/example/.cargo/config.toml rename to examples/rust/.cargo/config.toml diff --git a/earthly/rust/example/.config/nextest.toml b/examples/rust/.config/nextest.toml similarity index 100% rename from earthly/rust/example/.config/nextest.toml rename to examples/rust/.config/nextest.toml diff --git a/earthly/rust/example/Cargo.lock b/examples/rust/Cargo.lock similarity index 100% rename from earthly/rust/example/Cargo.lock rename to examples/rust/Cargo.lock diff --git a/earthly/rust/example/Cargo.toml b/examples/rust/Cargo.toml similarity index 100% rename from earthly/rust/example/Cargo.toml rename to examples/rust/Cargo.toml diff --git a/examples/rust/Earthfile b/examples/rust/Earthfile new file mode 100644 index 000000000..2420f8321 --- /dev/null +++ b/examples/rust/Earthfile @@ -0,0 +1,82 @@ +VERSION 0.7 + +# cspell: words TARGETARCH USERARCH toolsets rustfmt + +# Set up our target toolchains, and copy our files. +builder: + FROM ./../../earthly/rust+rust-base + + COPY --dir .cargo .config benches src tests . + COPY Cargo.lock Cargo.toml . + COPY clippy.toml deny.toml rustfmt.toml . + + DO ./../../earthly/rust+SETUP + +# Test rust build container - Use best architecture host tools. +check-hosted: + FROM +builder + + DO ./../../earthly/rust+CHECK + +# Test which runs check with all supported host tooling. Needs qemu or rosetta to run. +# Only used to validate tooling is working across host toolsets. +check-all-hosts: + BUILD --platform=linux/amd64 --platform=linux/arm64 +check-hosted + + +# Build the service. +build-hosted: + FROM +builder + + DO ./../../earthly/rust+BUILD + + DO ./../../earthly/rust+SMOKE_TEST --bin=hello_world + + SAVE ARTIFACT target/$TARGETARCH/doc doc + SAVE ARTIFACT target/$TARGETARCH/release/hello_world hello_world + +# Test which runs check with all supported host tooling. Needs qemu or rosetta to run. +# Only used to validate tooling is working across host toolsets. +build-all-hosts: + BUILD --platform=linux/amd64 --platform=linux/arm64 +build-hosted + +## ----------------------------------------------------------------------------- +## +## Standard CI targets. +## +## These targets are discovered and executed automatically by CI. + +# Run check using the most efficient host tooling +# CI Automated Entry point. +check: + FROM busybox + # This is necessary to pick the correct architecture build to suit the native machine. + # It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation. + # All target implementation of this should follow this pattern. + ARG USERARCH + + IF [ "$USERARCH" == "arm64" ] + BUILD --platform=linux/arm64 +check-hosted + ELSE + BUILD --platform=linux/amd64 +check-hosted + END + +# Run build using the most efficient host tooling +# CI Automated Entry point. +build: + FROM busybox + # This is necessary to pick the correct architecture build to suit the native machine. + # It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation. + # All target implementation of this should follow this pattern. + ARG USERARCH + + IF [ "$USERARCH" == "arm64" ] + BUILD --platform=linux/arm64 +build-hosted + ELSE + BUILD --platform=linux/amd64 +build-hosted + END + +# This step simulates the full CI run for local purposes only. +local-ci-run: + BUILD +check + BUILD +build \ No newline at end of file diff --git a/earthly/rust/example/benches/benchmark.rs b/examples/rust/benches/benchmark.rs similarity index 100% rename from earthly/rust/example/benches/benchmark.rs rename to examples/rust/benches/benchmark.rs diff --git a/earthly/rust/example/clippy.toml b/examples/rust/clippy.toml similarity index 100% rename from earthly/rust/example/clippy.toml rename to examples/rust/clippy.toml diff --git a/earthly/rust/example/deny.toml b/examples/rust/deny.toml similarity index 100% rename from earthly/rust/example/deny.toml rename to examples/rust/deny.toml diff --git a/earthly/rust/example/rust-toolchain.toml b/examples/rust/rust-toolchain.toml similarity index 100% rename from earthly/rust/example/rust-toolchain.toml rename to examples/rust/rust-toolchain.toml diff --git a/earthly/rust/example/rustfmt.toml b/examples/rust/rustfmt.toml similarity index 100% rename from earthly/rust/example/rustfmt.toml rename to examples/rust/rustfmt.toml diff --git a/earthly/rust/example/src/lib.rs b/examples/rust/src/lib.rs similarity index 100% rename from earthly/rust/example/src/lib.rs rename to examples/rust/src/lib.rs diff --git a/earthly/rust/example/src/main.rs b/examples/rust/src/main.rs similarity index 100% rename from earthly/rust/example/src/main.rs rename to examples/rust/src/main.rs diff --git a/earthly/rust/example/tests/integration_test.rs b/examples/rust/tests/integration_test.rs similarity index 100% rename from earthly/rust/example/tests/integration_test.rs rename to examples/rust/tests/integration_test.rs