Ghaf infra new environments

[henrirosten]

Support ghaf-infra deployments as outlined in https://ssrc.atlassian.net/wiki/spaces/SP/pages/1357250598/Ghaf-infra+Environments:

• Allow creating four types of different ghaf-infra instances: 'release', 'prod', 'dev', 'priv':
  • 'release': ephemeral environment manually spun-up and destroyed for the purpose of making Ghaf SLSA L3 release builds. The ‘release’ environment does not use external builders, but spins-up the builder VMs together with the rest of the infra.
  • 'prod': production environment provides the continuous services for Ghaf developers and testers: nix binary cache, as well as the CI environment. Prod environment(s) should run 24/7, minimizing downtime.
  • 'dev': development environment serves as the dev environment for ghaf-infra developers - to be able test infra changes before promoting them to ‘prod’. 'dev' environment(s) are therefore otherwise similar to 'prod', except there's no uptime requirements. As an example, 'prod' and 'dev' share the binary cache storage and key.
  • 'priv': personal environments for ghaf-infra development and testing.

This change also introduces changes to critical resource storage:

• Jenkins artifact storage is moved from environment-specific to persistent storage
• Separate persistent storage for 'release'
• Separate binary cache storage for 'priv' environments

In addition, the changes in this PR enable creating concurrent ghaf-infra instances of the same type. As an example, we could have two 'prod' deployments (with different names) running concurrently. Similarly, ghaf-infra developers can now have many 'priv' environments running concurrently.

Removes terraform-playground.sh: we should now use terraform-init.sh and the instructions in terraform/README.md to achieve the same functionality.

[karim20230]

quite large changes -hoping tested well!

[henrirosten]

quite large changes -hoping tested well!

@karim20230 : thanks for the review. I also struggled as to how to split this in smaller, more digestible pieces. I couldn't figure out how to split the first two commits that are required to align with the new setup. Other commits in this PR are smaller, hopefully easier to understand.

[henrirosten]

Rebased on latest main