Catena-X consortium cloud infrastructure

This repository hosts configuration for setting up the cloud infrastructure for the Catena-X consortium environments. It is mainly terraform based and involves Azure resources.

Structure

The repository follows a specific structure. On the root level, there are directories for each of our kubernetes clusters. The cluster directories are prefixed with a number, that should indicate a potential order, in which clusters should be created, if we start a completely new consortium setup. The order can be necessary, so that managing of clusters can properly be set up.

In addition to the cluster modules, there is a single consortium_cluster module, which defines all the necessary Azure resources to form a complete environment. This module is inteded to define the basis for all consortium environments and can be configured through dedicated variables

How to run

To manage the cloud infrastructure defined in this repository, you need a local installation of terraform CLI. For installation, just follow the instruction for your operating system.

The terraform state files are stored in an Azure Storage Account. To access the credentials, it is recommended to query them via Azure CLI. You can follow the official installation instructions

Terraforming the environments almost always is done by issuing the following commands on your local machine:

# Login with Azure CLI
az login --tenant <catena-x-azure-tenant-id>
# Get credentials for Azure Storage account containing the terraform state files
export ARM_ACCESS_KEY=$(az storage account keys list --resource-group cx-devsecops-tfstates --account-name cxdevsecopstfstate --query '[0].value' -o tsv)

# Change to the environment specific directory
cd <env-specific-dir>

# Init terraform providers - Only necessary on the first run
terraform init
# Upgrade the provider version - Always recommended to upgrade to the latest version of providers matching the specified
# constraints (usually ~> <version> to use the latest patch version)
terraform init -upgrade

# Run terraform plan and apply
terraform plan
terraform apply

Update existing environments

It is encouraged to configure upgrades or changes to environments at the reused terraform module consortium_cluster. This should prevent environments from diverging too much from our standard configuration. After changing the consortium_cluster module, each of the environments has to be reconfigured via terraform. The reconfiguration is done with the command sequence shown in the previous section

NOTE: If there is a valid reason, why a specific environment should not use the standard configuration, consider introducing a variable in the consortium_cluster module and define it with the desired default value. Then overwrite this variable in the module call of main.tf and skip that variable assignment for all other environments, that can follow the standard

Create a new environment

To add a new environment, create a new directory on the top level of this repository. The naming convention is as follows: <incrementing-number>_<environment-name>_cluster example: 01_core_cluster

The terraform config needed for an environment is separated in

• the main.tf file containing the module call and potentially distinct resources for that environment
• the providers.tf file containing the backend config that specifies the state file location (Azure Storage Account)
• the variables.tf file, where we need to specify provider configuration, that can be used in the module call in main.tf

You can copy and paste the mentioned files of an existing environment and adjust is slightly. The adjustments, that need to be made are:

• main.tf
  • choose a module name matching your new environment name
  • overwrite the cluster_name variable to match your new environment name
  • Remove any other specific configuration if you copied it from a non-default environment
  • Remove any additional resource definitions in case you do not need them for the new environment
• providers.tf
  • Adjust container_name in the backend configuration. This should match your environment name
  • Adjust key in the backend configuration. This should be in the form of .tfstate

Before creating the new environment by running terraform, you need to create the Azure Storage Account container, that you specified in the backend configuration in providers.tf. The container can be created, if you navigate to the Storage Account resource on Azure Portal.

As soon as the Azure Storage Account container does exist, you can run the terraform commands:

# Login with Azrue CLI
az login --tenant <catena-x-azure-tenant-id>
# Get credentials for Azure Storage account containing the terraform state files
export ARM_ACCESS_KEY=$(az storage account keys list --resource-group cx-devsecops-tfstates --account-name cxdevsecopstfstate --query '[0].value' -o tsv)

# Change to the environment specific directory
cd <new-env-dir>

# Init terraform providers - Only necessary on the first run
terraform init

# Run terraform plan and apply
terraform plan
terraform apply