CheriBSD

This repository has commits specific to CheriBSD, to execute self-hosted GitHub runners in ephemeral jails via pot, a BSD jail manager. It makes certain assumptions, that GODEBUG="asyncpreemptoff=1" is necessary to prevent kernel panics, owing to a known bug affecting the Go runtime on CheriBSD. It also copies the libraries for CheriBSD's various ABIs from the host to a sibling pot, later cloned, to ensure that pkg64 and pkg64cb are useable on capability-aware systems like Morello.

Quick start

To install under CheriBSD 23.11:

./install.sh

To initialise a runner via GitHub, there are several ways to use these scripts.

Stand-alone

For the simplest execution:

./config.sh --url {GITHUB_URL} --token {TOKEN}

Automating with GitHub PATs

For use with a personal access token against a specific organisation, there is a script available to check the environment for the variables GITHUB_PAT and GITHUB_ORG:

./get_token.sh
./config.sh --url {GITHUB_URL} --token $GITHUB_TOKEN

Finer control

For fine-grained control over the variables:

POT_MOUNT_BASE=/opt/pot/ \
CHERIBSD_BUILD_ID=23.11 \
RUNNER_NAME={NAME} \
./config.sh --url {GITHUB_URL} --token {TOKEN}

Health checks

To configure healthchecks via crontab -e, add the following in your editor of choice:

* 9 */1 * * /sbin/zpool status -v
* 12 */1 * * /etc/cron.d/scrub-pool.sh
* */1 * * * /etc/cron.d/clean-pots.sh
*/10 * * * * /etc/cron.d/count-pots.sh
*/2 * * * * /etc/cron.d/restart-actions.sh

Development

Testing

This project uses shellspec as a test framework. To run tests locally, shellspec must first be installed on the local system according to the shellspec installation guide.

Tests are defined in the spec directory, and are all files matching the pattern <suite_name>_spec.sh. Mocks, custom matchers and other fixtures can be defined in the spec/support directory.

The pot mock has already been defined and is imported into all tests. By default, all pot subcommands return truthy with no side effects, but these can be overloaded on a per-subcommand basis, as each subcommand has its own stub function. See here for an example of overloading the pot exec subcommand, and here for a list of all subcommand stubs that can be overloaded.

Wrapper scripts to run GitHub actions in a jail on FreeBSD

DISCLAIMER: This uses a third-party implementation of the GitHub Actions protocol and is not officially supported by GitHub.

This repository provides some scripts to run CI driven from GitHub in a FreeBSD jail. It uses pot to manage the jails. The pot is cloned from an immutable ZFS filesystem, runs one action, stops, the underlying ZFS filesystem is destroyed, and then the process repeats. This allows the actions to do whatever they need to as root in the jail (install software, whatever). This is a work in progress, please file issues as you find them.

Installing and configuring a runner

Install with ./install.sh. This will install a pot flavour and the RC scripts required to run the jails. Register a GitHub action by following the directions from GitHub. When you are asked to select an operating system, select Linux. Ignore all of the commands that you are instructed to run except for the one that starts ./config. Paste that command into a root terminal in the directory where you have cloned this repository.

Three environment variables affect this command:

• CHERIBSD_BUILD_ID specifies the build of CheriBSD to use (e.g. 23.11). If not specified this is the latest build of CheriBSD.
• RUNNER_NAME specifies the name to give to this runner. If not specified then this is the {your hostname}-freebsd-{version number}.
• RUNNER_FLAVOURS specifies a space-separated list of flavours that will be applied to the pot. This allows you to provide a set of additional tools that will be made available to runner actions.

This script will create a pot (container) that has a configured GitHub Actions runner inside. The name of the pot is the runner name with any dots replaced with underscores (pot names are not allowed to contain dots), the script will output this on the last line of the output. The runner will have freebsd and freebsd-{version} labels, these can be used to select runners with the runs-on property in the YAML.

The runner's configuration is exported from into runners/{pot name}, allowing the runner to be re-created without having to re-register it with GitHub. This exported configuration includes all of the environment variables that the script used when setting them up.

This pot can be exported and imported on another system using the existing pot commands. This pot will not be run directly by the scripts, it is treated as an immutable prototype and cloned for each invocation.

Installing dependencies

You can modify the pot to install dependencies such as compilers or other tools. You can update the base-system image in the same way.

To make orchestration easier, you should provide your dependencies as one or more pot flavours. These can be injected into the pot by setting the RUNNER_FLAVOURS environment variable as outlined above.

Re-creating a runner

The recreate-runner.sh script re-creates a runner. This expects to find the runners/{pot name} directory containing the configuration. This will read the environment variables that were provided to config.sh from runners/{pot name}/act-config.sh. You can modify this to specify a newer FreeBSD base version (for example, moving from FreeBSD 12.2 to 12.3) or to change the set of flavours that are installed. Note that recreating the runner will not change the labels and so you may need to manually modify labels that refer to the version if you replace a runner with a newer version. If you wish to change the name of the pot when re-creating the runner (for example, to include a version or date) then you must copy or rename this directory.

Note that only one pot with any given name can exist on the system at a time and so you must either rename the pot or destroy the pot before recreating it. Note that you can create the pot on one system, export it, and then import it on your deployment system.

If you have created your runners by providing flavours with all of the dependencies then this script allows you to generate a new version with all dependencies.

Recreating all runners

The recreate-all-runners.sh script will recreate all runners that you have created. This iterates over all of the configs in the runners/ directory and so must be run from the same location as config.sh. This can be run from a cron job to make sure that all runners have picked up the security updates.

Running the runners

The install script will install a gh_actions RC script. This is controlled by two variables in /etc/rc.conf (or any of the equivalent places):

• gh_actions_enable, set this to "YES" to enable starting runners automatically.
• gh_actions_pots is a space-separated list of pot names for the action runners.

Once these are set, service gh_actions start will start all configured runners. service gh_actions stop will stop all runners. Note: There is a bug in the current version of pot that prevents starting a pot immediately after it was forcibly stopped.

Using the runners

The runners appear like any other GitHub Actions runners.

name: Example

# Controls when the workflow will run
on:
  # Triggers the workflow on push
  push:

  # Triggers the workflow on pull-request
  pull_request:

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  # This is just a name
  freebsd:
    # Select any runner with the freebsd-12.2 tag.  This is added automatically
    # to runners that these scripts create from a 12.2 base.
    runs-on: freebsd-12.2
    name: Test in FreeBSD
    steps:
    # Use the existing (NodeJS) checkout action to clone the repo:
    - uses: actions/checkout@v2
    # Add any other GitHub actions here
    - name: Build
      run: {your build commands here}