Create a PR main.tf template and use tfvars to specify test envs #898

maximenoel8 · 2023-06-16T03:17:36Z

TESTING-DONT-REVIEW-pipeline-pull-request-light.groovy is for testing purpose. DON'T REVIEW

DON"T REVIEW TESTING-DONT-REVIEW-pipeline-pull-request-light.groovy

This PR is a revamp of all main.tf for the PR testing pipelines to manage the differences in tfvars files and have one unique main.tf template common for all the PR testing environments.

Related to https://github.com/SUSE/spacewalk/issues/21471

maximenoel8 · 2023-06-19T01:17:12Z

Create template main.tf for all PR environments and manage specific env variables.

Why ?

I revamp our main.tf configuration for the PR deployment. It's something I wanted to address for a while. This new pipeline is probably the right opportunity.

Normally in terraform project, it's better to separate main.tf (deployment description ) from global variables ( variable.tf ) and dynamic variables ( terraform.tfvars and **.tfvars )

The first step is to remove all variable declarations from main.tf and move it to a variable.tf file ( this file has to be in the same main.tf directory).
The second step I work on was to create a common template-main.tf generic to all uyuniPR-main.tf.

template file : terracumber_config/tf_files/solution1/PR-tests-template-solution-1.tf ( some variables will change depending on the solution )
variable file : terracumber_config/tf_files/solution1/variable.tf ( some variables will change depending on the solution )

I isolated 4 specific environment variables :

environment number : this variable is used to set the host-name. But it could be also use to select the correct environment variable ( see solution 2 )
hyper / hypervisor : configure hypervisor depend on environment number
mac addresses : specific to each module + environment number
additional network : specific to environment number

To manage this variables, I extracted the hard written variables from the main.tf and added it to the variable.tf.
There are two solutions to manage this specific variables :

First solution ( folder `terracumber_config/tf_files/solution1` ):

Have one tfvars for each environment ( enviromnent1.tfvars.json / enviromnent2.tfvars.json / etc ) and declare all the specific settings into this file. You can then call this tvars by using the terraform argument : -var-file="enviromnent2.tfvars"

tfvars example : terracumber_config/tf_files/solution1/env1.tfvars
tfvars.json example. We can also use json format for tfvars: terracumber_config/tf_files/solution1/env2.tfvars.json

Pro :

environment files are easy to identify and don't have too many variables in one file.
no red variable in main.tf
one level map

Con :

need to add support -var-file argument to terracumber-ci ( this support will be needed anyway to manage uyuni / spacewalk differences )
10 tfvars files to manage

Second solution ( folder `terracumber_config/tf_files/solution2` ) :

Have all the environments describe in the default terraform.tfvars.json ( by default used by terraform ). The environment number will be the top key to select the specific settings.

terraform.tfvars example : terracumber_config/tf_files/solution2/terraform.tfvars

Pro :

nothing to add to terracumber-ci
only one file to manage
only need to copy this file into main.tf workspace

Con :

it will be a big file where you can easily make mistakes
two level maps => second level key will be red in main.tf

maximenoel8 · 2023-06-19T02:00:19Z

Create tfvars to manage uyuni and spacewalk configuration

For this variable we will still use the template main tf configuration.
We identified with Jordi some variables that are unique to uyuni and spacewalk.
This variables are described in manager43-pr.tfvars and uyuni-pr.tfvars. The variable declarations are in variable.tf.
During the pipeline we will copy either of this file ( using cp or terracumber-ci ) into the sumaform workspace and then call terraform command using the argument -var-file to overwrite this parameters.

If more variables where to be identified, we can easily add its to this file

FYI, overwrite order in terraform :
variables => terraform.tfvars => tfvars added by using -var-file

maximenoel8 · 2023-06-19T02:01:32Z

Manage variables directly parse during terraform call :

            stage('Deploy') {
                ws(environment_workspace){
                    if(must_test) {
                        // Passing the built repository by parameter using a environment variable to terraform file
                        // TODO: We will need to add a logic to replace the host, when we use IBS for spacewalk
                        env.PULL_REQUEST_REPO= "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${builder_project}:${pull_request_number}/${build_repo}/${arch}"
                        env.MASTER_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${source_project}/${build_repo}/${arch}"
                        env.MASTER_OTHER_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${source_project}:Other/${build_repo}/${arch}"
                        env.MASTER_SUMAFORM_TOOLS_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${sumaform_tools_project}/${build_repo}/${arch}"
                        env.TEST_PACKAGES_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${test_packages_project}/rpm/${arch}"
                        env.UPDATE_REPO = "http://minima-mirror.mgr.prv.suse.net/jordi/some-updates/"
                        if (additional_repo_url == '') {
                            echo "Adding dummy repo for update repo"
                            env.ADDITIONAL_REPO_URL = "http://minima-mirror.mgr.prv.suse.net/jordi/dummy/"
                        } else {
                            echo "Adding ${additional_repo_url}"
                            env.ADDITIONAL_REPO_URL = additional_repo_url
                        }
                        env.SLE_CLIENT_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${source_project}:SLE15-Uyuni-Client-Tools/SLE_15/${arch}"
                        env.RHLIKE_CLIENT_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${source_project}:EL8-Uyuni-Client-Tools/EL_8/${arch}"
                        env.DEBLIKE_CLIENT_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${source_project}:Ubuntu2204-Uyuni-Client-Tools/xUbuntu_22.04/${arch}"
                        env.OPENSUSE_CLIENT_REPO = "http://${fqdn_jenkins_node}/workspace/suma-pr${env_number}/repos/${source_project}:openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/${arch}"

                        // Provision the environment
                        if (terraform_init) {
                            env.TERRAFORM_INIT = '--init'
                        } else {
                            env.TERRAFORM_INIT = ''
                        }
                        sh "set +x; source /home/jenkins/.credentials set -x; export TF_VAR_SLE_CLIENT_REPO=${SLE_CLIENT_REPO};export TF_VAR_RHLIKE_CLIENT_REPO=${RHLIKE_CLIENT_REPO};export TF_VAR_DEBLIKE_CLIENT_REPO=${DEBLIKE_CLIENT_REPO};export TF_VAR_OPENSUSE_CLIENT_REPO=${OPENSUSE_CLIENT_REPO};export TF_VAR_PULL_REQUEST_REPO=${PULL_REQUEST_REPO}; export TF_VAR_MASTER_OTHER_REPO=${MASTER_OTHER_REPO};export TF_VAR_MASTER_SUMAFORM_TOOLS_REPO=${MASTER_SUMAFORM_TOOLS_REPO}; export TF_VAR_TEST_PACKAGES_REPO=${TEST_PACKAGES_REPO}; export TF_VAR_MASTER_REPO=${MASTER_REPO};export TF_VAR_UPDATE_REPO=${UPDATE_REPO};export TF_VAR_ADDITIONAL_REPO_URL=${ADDITIONAL_REPO_URL};export TF_VAR_CUCUMBER_GITREPO=${cucumber_gitrepo}; export TF_VAR_CUCUMBER_BRANCH=${cucumber_ref}; export TERRAFORM=${terraform_bin}; export TERRAFORM_PLUGINS=${terraform_bin_plugins}; ./terracumber-cli ${common_params} --logfile ${resultdirbuild}/sumaform.log ${env.TERRAFORM_INIT} --taint '.*(domain|main_disk).*' --runstep provision"
                        deployed = true
                    }
                }
            }

${fqdn_jenkins_node} => hostname -f done during pipeline
${env_number} => find during pipeline, will determine which environment we are going to use
${builder_project} => depends on uyuni or spacewalk project systemsmanagement:Uyuni:Master:PR
${pull_request_number} => coming from jenkins parameter
${build_repo} => specific to uyuni / spacewalk ( openSUSE_Leap_15.4 or sles15sp4 )
${arch} => hard coded in pipeline ( x86_64 )

=>
Instead of using export to parse the variables, we can directly add this variables to terraform.tfvars.

def terraform_configuation = "SLE_CLIENT_REPO= \"${SLE_CLIENT_REPO} \"\n" +
RHLIKE_CLIENT_REPO= \"${RHLIKE_CLIENT_REPO} \"\n" +
DEBLIKE_CLIENT_REPO= \"${DEBLIKE_CLIENT_REPO} \"\n" +
OPENSUSE_CLIENT_REPO= \"${OPENSUSE_CLIENT_REPO} \"\n" +
PULL_REQUEST_REPO= \"${PULL_REQUEST_REPO} \"\n" +
MASTER_OTHER_REPO= \"${MASTER_OTHER_REPO} \"\n" +
MASTER_SUMAFORM_TOOLS_REPO= \"${MASTER_SUMAFORM_TOOLS_REPO} \"\n" +
TEST_PACKAGES_REPO= \"${TEST_PACKAGES_REPO} \"\n" +
MASTER_REPO= \"${MASTER_REPO} \"\n" +
UPDATE_REPO= \"${UPDATE_REPO} \"\n" +
ADDITIONAL_REPO_URL= \"${ADDITIONAL_REPO_URL} \"\n" +
CUCUMBER_GITREPO= \"${cucumber_gitrepo} \"\n" +
CUCUMBER_BRANCH= \"${cucumber_ref} \"

def terraform_tfvars_file = new File("${jenkins_workspace}/terraform.tfvars")
if (terraform_tfvars_file.exists()) {
  def fileWriter = new FileWriter(terraform_tfvars_file, true) // Append mode
  fileWriter.append(terraform_configuation)
  fileWriter.close()
} else {
  writeFile file: "${jenkins_workspace}/terraform.tfvars", text: terraform_configuation, encoding: "UTF-8"
}

srbarrios · 2023-06-27T08:08:21Z

terracumber_config/tf_files/tfvars/PR-TEST-uyuni.tfvars

@@ -0,0 +1,14 @@
+############ Variable unique to uyuni ########################
+
+IMAGE                  = "opensuse154-ci-pro"


I wasn't aware of this image, what makes it different?

@jordimassaguerpla can give an explanation on this. FMPOV it simply is an adjusted image with already preinstalled packages and adjusted configuration. See https://build.opensuse.org/package/show/systemsmanagement:sumaform:images:libvirt/opensuse154-ci-pr

srbarrios · 2023-06-27T08:09:30Z

terracumber_config/tf_files/tfvars/PR-TEST-manager43.tfvars

+############ Varaibles unique to spacewalk ###########
+
+IMAGE                  = "sles15sp4o"
+GIT_PROFILES_REPO      = "https://github.com/SUSE/spacewalk.git#:testsuite/features/profiles/internal_prv"


For that variable, we will need to diverge it depending of the environment is placed in PRV or NUE. Otherwise the retail tests might fail. But we can let this for a next iteration.

srbarrios

Very nice refactor by using this PR-TEST-main.tf 💯
My only nitpick is that I would not use uppercase in the filename, but this looks great, thank you.

srbarrios · 2023-07-25T07:32:04Z

terracumber_config/tf_files/PR-TEST-main.tf

+        non_os_pool = "http://${var.MIRROR}/distribution/leap/15.4/repo/non-oss/",
+        os_pool = "http://${var.MIRROR}/distribution/leap/15.4/repo/oss/",


I miss something or we are not parametrizing between leap and sles, for Uyuni and SUMA PRs envs?

srbarrios · 2023-07-25T07:32:29Z

terracumber_config/tf_files/PR-TEST-main.tf

+        os_pool = "http://${var.MIRROR}/distribution/leap/15.4/repo/oss/",
+        os_update = var.UPDATE_REPO,
+        os_additional_repo = var.ADDITIONAL_REPO_URL,
+        testing_overlay_devel = "http://${var.MIRROR}/repositories/systemsmanagement:/Uyuni:/Master/images/repo/Testing-Overlay-POOL-x86_64-Media1/",


I miss something or we are not parametrizing between Uyuni and SUMA images for the testing overlay devel?

All repositories will need a revamp too. But I will need Jordi for that. It's just a first step, we will probably still move parameters around. For now, my main focus is to keep the deployment working with all the changes.

srbarrios · 2023-07-25T07:33:09Z

terracumber_config/tf_files/PR-TEST-main.tf

+        non_os_pool = "http://${var.MIRROR}/distribution/leap/15.4/repo/non-oss/",
+        os_pool = "http://${var.MIRROR}/distribution/leap/15.4/repo/oss/",


Same than on the server module

srbarrios · 2023-07-25T07:33:15Z

terracumber_config/tf_files/PR-TEST-main.tf

+        testing_overlay_devel = "http://${var.MIRROR}/repositories/systemsmanagement:/Uyuni:/Master/images/repo/Testing-Overlay-POOL-x86_64-Media1/",
+        proxy_pool = "http://${var.MIRROR}/repositories/systemsmanagement:/Uyuni:/Master/images/repo/Uyuni-Proxy-POOL-x86_64-Media1/",


Same than on the server module

terracumber_config/tf_files/PR-TEST-main.tf

terracumber_config/tf_files/variables/PR-TEST-variable.tf

terracumber_config/tf_files/tfvars/PR-TEST-manager43.tfvars

srbarrios · 2023-07-25T07:44:19Z

jenkins_pipelines/environments/common/pipeline-pull-request-light.groovy

@@ -0,0 +1,208 @@
+def run(params) {


I would guess this "-light" pipeline is just temporary, and we will replace the content of this one inside "pipeline-pull-request.groovy" once it's stable?
Otherwise, we will have an extra pipeline, and a older one not anymore in use.

jenkins_pipelines/environments/common/pipeline-pull-request-light.groovy

srbarrios · 2023-07-25T07:49:33Z

jenkins_pipelines/environments/common/pipeline-pull-request-light.groovy

+                        }
+
+                        // Clean up old results
+                        sh "if [ -d ${resultdir} ];then ./clean-old-results -r ${resultdir};fi"


Being able to have if statements at Groovy level, I don't see a good reason to delegate it to bash.

terracumber_config/tf_files/PR-TEST-main.tf

jenkins_pipelines/environments/common/pipeline-pull-request-light.groovy

srbarrios · 2023-07-25T08:30:51Z

need to add support -var-file argument to terracumber-ci ( this support will be needed anyway to manage uyuni / spacewalk differences )

Maybe instead of having the need of provide one by one all the files for each kind of terraform variables that terraform can handle.
We can have a folder per flavour, and include inside all the terraform variables files that we want, so you just need to pass the folder path.
For the common variable file, well, just use a symbolic link, as we are doing in sumaform on the modules.

maximenoel8 · 2023-12-21T08:19:11Z

Currently running on https://ci.suse.de/view/Manager/view/Uyuni-PRs/job/uyuni-prs-ci-tests-jordi/

terracumber_config/tf_files/variables/PR-testing-variables.tf

srbarrios · 2023-12-21T08:56:10Z

terracumber_config/tf_files/tfvars/PR-testing-manager43.tfvars

+############ Spacewalk unique variables ############
+
+IMAGE                  = "sles15sp4o"
+IMAGES                 = ["rocky9o", "opensuse154o", "sles15sp4o", "ubuntu2204o"]


Rocky9 is correct?
I though we were testing Rocky8 in 4.3 CI

In SUSE-Manager-4.3-PR-tests-env5.tf, I can see we are using rocky9

srbarrios · 2023-12-21T09:09:46Z

jenkins_pipelines/environments/common/pipeline-pull-request.groovy

+                        sh "echo \"############ Repositories variables ############\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo PULL_REQUEST_REPO = \\\"http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${builder_project}:${pull_request_number}/${build_repo}/${arch}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo MASTER_REPO = \\\"http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${source_project}/${build_repo}/${arch}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo MASTER_OTHER_REPO = \\\"http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${other_project}/${other_build_repo}/${arch}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo MASTER_SUMAFORM_TOOLS_REPO = \\\"http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${sumaform_tools_project}/${build_repo}/${arch}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo TEST_PACKAGES_REPO = \\\"http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${test_packages_project}/rpm/${arch}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo UPDATE_REPO = \\\"${update_repo}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"


In my opinion, to avoid that many escaped characters it could be good to keep using env.XXXX
And then, you do something like:
sh "echo MASTER_SUMAFORM_TOOLS_REPO = ${env.MASTER_SUMAFORM_TOOLS_REPO}" >> ${env.resultdir}/sumaform/terraform.tfvars

I will try but I think we probably still need the extra quote ( terraform format)

srbarrios · 2023-12-21T09:16:59Z

jenkins_pipelines/environments/common/pipeline-pull-request.groovy

-                        env.TEST_PACKAGES_REPO = "http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${test_packages_project}/rpm/${arch}"
-                        env.UPDATE_REPO = update_repo
+                        sh "echo \"############ Repositories variables ############\" >> ${env.resultdir}/sumaform/terraform.tfvars"
+                        sh "echo PULL_REQUEST_REPO = \\\"http://${fqdn_jenkins_node}/workspace/${short_product_name}-pr${env_number}/repos/${builder_project}:${pull_request_number}/${build_repo}/${arch}\\\" >> ${env.resultdir}/sumaform/terraform.tfvars"


I did not check, but I would say we need to escape other characters inside that URL 🤔

Not it works correctly :

PULL_REQUEST_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master:PR:8067/openSUSE_Leap_15.5/x86_64" MASTER_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master/openSUSE_Leap_15.5/x86_64" MASTER_OTHER_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master:Other/openSUSE_Leap_15.5/x86_64" MASTER_SUMAFORM_TOOLS_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:sumaform:tools/openSUSE_Leap_15.5/x86_64" TEST_PACKAGES_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Test-Packages:Pool/rpm/x86_64" UPDATE_REPO = "http://minima-mirror-ci-bv.mgr.prv.suse.net/jordi/some-updates/" ADDITIONAL_REPO_URL = "http://minima-mirror-ci-bv.mgr.prv.suse.net/jordi/dummy/" SLE_CLIENT_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master:SLE15-Uyuni-Client-Tools/SLE_15/x86_64" RHLIKE_CLIENT_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master:EL9-Uyuni-Client-Tools/EL_9/x86_64" DEBLIKE_CLIENT_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master:Ubuntu2204-Uyuni-Client-Tools/xUbuntu_22.04/x86_64" OPENSUSE_CLIENT_REPO = "http://jenkins-worker-prs.mgr.prv.suse.net/workspace/suma-pr10/repos/systemsmanagement:Uyuni:Master:openSUSE_Leap_15-Uyuni-Client-Tools/openSUSE_Leap_15.0/x86_64"

srbarrios

That's a hard one to review. I added few comments.
Once you test and it works well... fine for me to merge. (Be careful with the commits coming from a pull, better to fix it before merging too)

maximenoel8 requested a review from jordimassaguerpla June 16, 2023 03:17

maximenoel8 changed the title ~~Create a PR main.tf template and user tfvars to specify test envs~~ Create a PR main.tf template and use tfvars to specify test envs Jun 19, 2023

srbarrios reviewed Jun 27, 2023

View reviewed changes

srbarrios approved these changes Jun 27, 2023

View reviewed changes