Creating sandbox environments for debugging

Garden-only lightweight deployment

In most of the cases in order to test something Garden it is sufficient to create a bosh deployment with a single VM running a single job - the garden one. Such a deployment is quick to create, you can create a bunch of them, you can create containers, run processes, run tests, etc. without all the heavy CF machinery.

Eden deployments

Eden is a bosh director that has been already deployed on GCP and we use it for CI purposes and to create our own test deployments. It is easy to get the manifest from an existing deployment, modify it to meet your needs (e.g. mimic the garden configuration in the issue you are after) and deploy it. In order to do that:

1. Target the Eden director

Just navigate to ~/workspace/garden-ci/directors/eden directory, direnv should set all environment variables for you (if it does not, you can force it to do that via direnv allow). You should now be able to list the deployments on Eden via bosh deployments

2. Get yourself a deployment manifest

Here is a very minimal garden-runc manifest:

instance_groups:
  - azs:
      - z1
    instances: 1
    jobs:
      - name: garden
        properties:
          garden:
            debug_listen_address: 0.0.0.0:17013
            listen_address: 0.0.0.0:7777
            listen_network: tcp
            log_level: debug
        release: garden-runc
    name: garden
    networks:
      - name: default
    stemcell: stemcell
    vm_type: garden-small
name: <your-fancy>-garden
releases:
  - name: garden-runc
    version: 1.19.29-rc.4
stemcells:
  - alias: stemcell
    os: ubuntu-bionic
    version: "1.1"
update:
  canaries: 1
  canary_watch_time: 1000-240000
  max_in_flight: 3
  update_watch_time: 1000-240000

In fact, this is the manifest of the clean-garden Eden deployment, as you can see Garden is deployed with all its default properties except for the listen address.

Other interesting deployments you may want to consider as a manifest starting point are:

• containerd-garden - this deployment has Garden configured to use containerd to manage containers, but use the RunRunC OCI interface (i.e. calls the runc binary directly) to run processes. As of the time of writing the article, this is the default configuration is CF-deployment
• nerdful-garden - Garden is configured to use containerd for both container management and running processes (i.e. the RunRunC OCI interface is completely turned off)
• cputhrottle-garden - the experimental CPU throttling feature is turned on

Feel free to look at the rest.

You can also set the version of the garden-runc release to create in order bosh to create a dev release out of your local garden-runc-release version - this is useful when testing changes you implement, here is an example:

......
releases:
- name: garden-runc
  version: create
  url: file:///home/<your-user>/workspace/garden-runc-release

Note: Do not forget to change the name of the deployment if you are editing a manifest of an existing deployment

3. Create the deployments

bosh -d <your-deployment> create-deployment <your-manifest.yml>

You could also provide bosh with additional ops files if needed, here are a bunch of them we found useful over the years: garden-ci/ops

4. Cleaning up

Once you are done fooling around, you could delete your test deployment to save some GCP resources:

bosh -d <your-deployment> delete-deployment

Local bosh-lite deployment

Warning: As of the time of writing the article, creating local bosh-lite deployments does not work on MacOS Big Sur and VirtualBox 6.1 (see this issue for details). The instructions below should otherwise work, take them with a grain of salt though.

1. Create the deployment

Here is the bosh-lite guide, luckily we have most of the things automated:

• The manual part: install Virtuabox, bosh cli, clone https://github.com/cloudfoundry/bosh-deployment to ~/workspace/bosh-deployment
• Create the bosh-lite director

~/workspace/garden-dotfiles/executables/deploy-lite.sh

• Target the newly created director

source ~/workspace/bosh-vbox/bosh-env

• Create the deployment (called garden-runc)

cd ~/workspace/garden-runc-release
./scripts/deploy-lite.sh

The command above would deploy your local garden-runc-release version.

There are a couple of deploy-lite*.sh files under the scripts directory that deploy Garden with different settings, you are welcome to have a look. You can also check the garden-runc-release/manifests directory for useful ops files.

2. Delete the deployment

bosh -d garden-runc delete-deployment

3. Delete the bosh-lite director

BOSH_ACTION=delete ~/workspace/garden-dotfiles/executables/deploy-lite.sh

CF deployments

Lightweight Garden deployments are unfortunately not always sufficient to reproduce a problem, for example you may be after a weird interaction between Garden and Diego. In those cases you would probably need a full CF.

Lite me up

The cheapest and fastest way to have yourself a full-blow CF deployment is lite-me-up - a script that creates a bosh-lite CF deployment on GCP

lite-me-up is capable of creating both director and CF deployment either in one shot, or in consecutive steps. This is useful in situations that you messed up CF so badly that you want to recreate it but you want to reuse the existing director:

cd ~/workspace/garden-ci/directors

./lite-me-up.sh --help
usage: lite-me-up.sh <create|deploy-director|deploy-cf|destroy> <env-name>
run env-vars to see which options you can set on your deployment

lite-me-up can be supplied with optional ops files during creating the director and the CF deployment via environment variables, LITE_ME_UP_CREATE_ENV_EXTRA_ARGS and LITE_ME_UP_CF_DEPLOY_EXTRA_ARGS respectively. We find it useful to create a script per deployment which we source prior running lite-me-up to set them, for example the ops script for the mel-b deployment is here.

lite-me-up would use the versions of bosh-deployment and cf-deployment that are submoduled in garden-ci. If you need to deploy another version of these, make sure to check them out prior running lite-me-up.

1. Creating a director and CF deployment (called cfoo) in one shot

cd ~/workspace/garden-ci/directors
mkdir cfoo
vim cfoo/ops-files # enable whatever ops files you want via LITE_ME_UP_CREATE_ENV_EXTRA_ARGS and LITE_ME_UP_CF_DEPLOY_EXTRA_ARGS
source cfoo/ops-files
./lite-me-up.sh create cfoo

Target the director: source cfoo/bosh-env

The script would put some deployment state files under to cfoo directory, make sure to push it to git in order to be able to delete the deployment after you are done with it.

2. Creating CF deployment only (called cfoo) reusing the existing director

source cfoo/bosh-env
source cfoo/ops-files
./lite-me-up.sh deploy-cf cfoo

Do not forget to push the changes made by the script to git!

3. Destroy the director

source cfoo/bosh-env
./lite-me-up.sh destroy cfoo

Do not forget to push the changes made by the script to git!

Fat me up

Although lite-me-up would create a full-blown CF deployment, it is still a bosh-lite one which differs a bit from the "real thing". If you are desperate for a real-world deployment, you could use fat-me-up - it is quite similar to lite-me-up (see above) but the deployments it creates are bbl ones. It does not support providing ops files via environment variables, you would have to edit the script to suite your needs.

The script would create certain state files, make sure to push them to git in order to be able to clean up after you are done with the deployment.