Merge pull request #13 from Ontotext-AD/TES-277-automation

TES-277: Packer Automation

CHANGELOG.md

@@ -12,6 +12,14 @@ All notable changes to the Packer template for creating GraphDB Azure VM images
 - Added provisioning of GraphDB backup script.
 - Added keepalive and file max size settings.
 - Improved logging of the GraphDB install script.
+- Added proper prefixes to the variable names in variables.pck.hcl
+- Grouped variables by usage in variables.pck.hcl
+- Added new variables in variables.pck.hcl
+  - `tags` - Additional tags to apply to the build resources
+  - `build_temp_resource_group_name` - Name of the temporary resource group created by Packer during the build
+  - `build_resource_group_name` - Name of a resource group to be used by Packer during the build
+  - `gallery_subscription_id` - ID of the subscription where the Shared Image Gallery is located
+- Updated [create_image_definition.sh](create_image_definition.sh) to just create the definition and to skip invoking `packer build`
 
 ## 1.0.0
 

README.md

@@ -31,20 +31,20 @@ Follow these steps to build an AMI for GraphDB using Packer:
 3. **Edit Variables**:
 
    The Packer configuration allows you to customize various parameters, such as the GraphDB version, Azure build and
-   replication regions, subscription, client and tennat IDs. To do so, create a variables file `variables.pkrvars.hcl`,
+   replication regions, subscription, client and tenant IDs. To do so, create a variables file `variables.pkrvars.hcl`,
    example file:
       ```bash
-      subscription_id              = "<your_azure_subscription_id>"
-      client_id                    = "<your_azure_service_principal_id>"
-      client_secret                = "<your_azure_service_principal_secret>"
-      tenant_id                    = "<your_azure_tenant_id>"
-      primary_location             = "East US"
-      replication_regions          = ["North Europe", "UK South"]
-      image_definition_name        = "10.4.0-x86_64"
-      gdb_version                  = "10.4.0"
-      gallery_resource_group       = "Packer-RG"
-      gallery_name                 = "GraphDB"
-      allowed_inbound_ip_addresses = "<your_public_IP_address>"
+      subscription_id                    = "<your_azure_subscription_id>"
+      tenant_id                          = "<your_azure_tenant_id>"
+      client_id                          = "<your_azure_service_principal_id>"
+      client_secret                      = "<your_azure_service_principal_secret>"
+      build_location                     = "East US"
+      graphdb_version                    = "10.4.0"
+      gallery_image_definition           = "10.4.0-x86_64"
+      gallery_resource_group             = "Packer-RG"
+      gallery_name                       = "GraphDB"
+      gallery_image_replication_regions  = ["North Europe", "UK South"]
+      build_allowed_inbound_ip_addresses = "<your_public_IP_address>"
       ```
 
 4. **Build the AMI**:
@@ -56,7 +56,7 @@ Follow these steps to build an AMI for GraphDB using Packer:
    This command will initiate the Packer build process. Packer will launch an VM instance, install GraphDB,
    and create an image based on the instance.
 
-   Please note that the image definition you specify in `image_definition_name` must exist in the SIG.
+   Please note that the image definition you specify in `gallery_image_definition` must exist in the SIG.
    To automate this process you can use the `create_image_definition.sh` script, which will read the variables in
    `variables.pkrvars.hcl` and create the image definition in the gallery.
    This script utilizes the Azure CLI, so you need to install and configure it beforehand.
@@ -65,10 +65,10 @@ Follow these steps to build an AMI for GraphDB using Packer:
       az_command="az sig image-definition create \
        -g $gallery_resource_group \
        --gallery-name $gallery_name \
-       --gallery-image-definition "$image_definition_name" \
+       --gallery-image-definition "$gallery_image_definition" \
        --publisher Ontotext \
        --offer GraphDB \
-       --sku "$gdb_version" \
+       --sku "$graphdb_version" \
        --os-type Linux \
        --hyper-v-generation v2 \
        --minimum-cpu-core 4 \
@@ -85,39 +85,46 @@ The following points can be customized in a packer variables file `variables.pkr
 
 **Subscription Configuration**
 
-* subscription_id (string): Your Azure subscription ID.
-* tenant_id (string): Your Azure Active Directory tenant ID.
-* client_id (string): The client ID (Service Principal ID) used for authentication.
-* client_secret (string): The client secret (Service Principal Secret) used for authentication.
-* primary_location (string): The primary Azure location you want to use.
+* `subscription_id` (string): Your Azure subscription ID.
+* `tenant_id` (string): Your Azure Active Directory tenant ID.
+* `client_id` (string): The client ID (Service Principal ID) used for authentication.
+* `client_secret` (string): The client secret (Service Principal Secret) used for authentication.
+* `use_azure_cli_auth` (bool): Use the information from an active az login session instead of authenticating with a client secret
 
-**Image Configuration**
+**GraphDB Configuration**
 
-* image_definition_name (string): The name of the x86_64 image to use.
-* gdb_version (string): The version of GraphDB to install.
-* replication_regions (list(string)): A list of Azure regions for replication of the created image.
+* `graphdb_version` (string): The version of GraphDB to install and package as VM
 
-**Gallery Configuration**
+**Build and OS Configuration**
 
-* gallery_resource_group (string): The resource group where the image gallery is located.
-* gallery_name (string): The name of the image gallery.
-* image_replica_count (number): The number of replicas of the Image Version to be created per region. (default is 1).
+* `build_location` (string): The primary Azure location you want to use for building the image
+* `build_temp_resource_group_name` (string): Name of the temporary resource group created by Packer during the build. The default is a random
+  generated name.
+* `build_resource_group_name` (string): Name of a resource group to be used by Packer during the build. Cannot specify this together with
+  build_location or build_temp_resource_group_name.
+* `build_allowed_inbound_ip_addresses` (string): Specify the list of IP addresses and CIDR blocks that should be allowed access to the VM.
+* `build_vm_size` (string): The Azure VM size (default is "Standard_B1ls").
+* `build_os_type` (string): The operating system type (default is "Linux").
+* `build_os_disk_size_gb` (number): size of the OS disk in GB (default is 30). Depends on base VM image limitation,
+  e.g., Ubuntu Server image has `build_os_disk_size_gb = 30`.
 
-**Networking Configuration**
+**Base Image Configuration**
 
-* allowed_inbound_ip_addresses (string): Specify the list of IP addresses and CIDR blocks that should be allowed access to the VM.
+* `base_image_offer` (string): The offer for the base image (default is "0001-com-ubuntu-server-jammy").
+* `base_image_publisher` (string): The publisher for the base image (default is "canonical").
+* `base_image_sku` (string): The SKU for the base image (default is "22_04-lts-gen2").
 
-**OS and Image Defaults**
+**Gallery Configuration**
 
-* os_type (string): The operating system type (default is "Linux").
-* image_offer (string): The offer for the base image (default is "0001-com-ubuntu-server-jammy").
-* image_publisher (string): The publisher for the base image (default is "canonical").
-* image_sku (string): The SKU for the base image (default is "22_04-lts-gen2").
-* vm_size (string): The Azure VM size (default is "Standard_B1ls").
-* shared_gallery_image_version_exclude_from_latest (bool):  If set to true, Virtual Machines deployed from the latest
+* `gallery_subscription_id` (string): ID of the subscription where the Shared Image Gallery is located. Will use subscription_id as default if
+  unspecified
+* `gallery_resource_group` (string): The resource group where the image gallery is located.
+* `gallery_name` (string): The name of the image gallery.
+* `gallery_image_definition` (string): The name of the x86_64 image to use.
+* `gallery_image_replication_regions` (list(string)): A list of Azure regions for replication of the created image.
+* `gallery_image_replica_count` (number): The number of replicas of the Image Version to be created per region. (default is 1).
+* `gallery_image_version_exclude_from_latest` (bool):  If set to true, Virtual Machines deployed from the latest
   version of the Image Definition won't use this Image Version (default is false)
-* os_disk_size_gb (number): size of the OS disk in GB (default is 30). Depends on base VM image limitation,
-  e.g., Ubuntu Server image has `os_disk_size_gb = 30`.
 
 **Provisioning Scripts**: You can replace or modify the provisioning scripts located in the `./files/` directory.
 These scripts and files are copied and executed during the AMI creation process.

azure.pkr.hcl

@@ -1,7 +1,8 @@
 locals {
   timestamp = regex_replace(timestamp(), "[- TZ:]", "")
   # See https://github.com/hashicorp/packer-plugin-azure/issues/65
-  version_timestamp = formatdate("YYYY.MM.DD", timestamp())
+  version_timestamp       = formatdate("YYYY.MM.DD", timestamp())
+  gallery_subscription_id = var.gallery_subscription_id != null ? var.gallery_subscription_id : var.subscription_id
 }
 
 source azure-arm ubuntu-x86-64 {
@@ -10,68 +11,37 @@ source azure-arm ubuntu-x86-64 {
   client_secret      = var.client_secret
   subscription_id    = var.subscription_id
   tenant_id          = var.tenant_id
-  location           = var.primary_location
+
+  location                  = var.build_location
+  temp_resource_group_name  = var.build_temp_resource_group_name
+  build_resource_group_name = var.build_resource_group_name
+
+  image_offer     = var.base_image_offer
+  image_publisher = var.base_image_publisher
+  image_sku       = var.base_image_sku
 
   shared_image_gallery_destination {
-    subscription        = var.subscription_id
+    subscription        = local.gallery_subscription_id
     resource_group      = var.gallery_resource_group
     gallery_name        = var.gallery_name
-    image_name          = var.image_definition_name
+    image_name          = var.gallery_image_definition
     image_version       = local.version_timestamp
-    replication_regions = var.replication_regions
+    replication_regions = var.gallery_image_replication_regions
   }
 
-  shared_image_gallery_replica_count               = var.image_replica_count
-  shared_gallery_image_version_exclude_from_latest = var.shared_gallery_image_version_exclude_from_latest
+  shared_image_gallery_replica_count               = var.gallery_image_replica_count
+  shared_gallery_image_version_exclude_from_latest = var.gallery_image_version_exclude_from_latest
 
-  azure_tags = {
-    GDB_Version      = "${var.gdb_version}"
+  azure_tags = merge({
+    GraphDB_Version  = "${var.graphdb_version}"
     CPU_Architecture = "x86-64"
     Build_Timestamp  = "${local.timestamp}"
-  }
+  }, var.tags)
 
-  communicator              = "ssh"
-  ssh_clear_authorized_keys = true
-  os_type                   = var.os_type
-  os_disk_size_gb           = var.os_disk_size_gb
-  image_offer               = var.image_offer
-  image_publisher           = var.image_publisher
-  image_sku                 = var.image_sku
-
-  vm_size                      = var.vm_size
-  allowed_inbound_ip_addresses = var.allowed_inbound_ip_addresses
+  communicator                 = "ssh"
+  ssh_clear_authorized_keys    = true
+  os_type                      = var.build_os_type
+  os_disk_size_gb              = var.build_os_disk_size_gb
+  vm_size                      = var.build_vm_size
+  allowed_inbound_ip_addresses = var.build_allowed_inbound_ip_addresses
 }
-
-#TODO Not yet supported by Packer https://github.com/hashicorp/packer/issues/12188#issuecomment-1380736642
-#source azure-arm ubuntu-arm64 {
-#  client_id       = var.client_id
-#  client_secret   = var.client_secret
-#  subscription_id = var.subscription_id
-#  tenant_id       = var.tenant_id
-#
-#  location                          = var.primary_location
-#  shared_image_gallery_destination {
-#    subscription        = var.subscription_id
-#    resource_group      = var.gallery_resource_group
-#    gallery_name        = var.gallery_name
-#    image_name          = var.image_name_arm64
-#    image_version       = local.version_timestamp
-#    replication_regions = var.replication_regions
-#  }
-#
-#  azure_tags = {
-#    GDB_Version      = "${var.gdb_version}"
-#    CPU_Architecture = "arm64"
-#    Build_Timestamp  = "${local.timestamp}"
-#  }
-#
-#  communicator    = "ssh"
-#  os_type         = "Linux"
-#  image_offer     = "0001-com-ubuntu-server-jammy"
-#  image_publisher = "canonical"
-#  image_sku       = "22_04-lts-arm64"
-#
-#  vm_size = "Standard_D2ps_v5"
-#
-#  allowed_inbound_ip_addresses = [var.allowed_inbound_ip_addresses]
-#}

build.pkr.hcl

@@ -16,7 +16,7 @@ build {
 
   provisioner "shell" {
     environment_vars = [
-      "GRAPHDB_VERSION=${var.gdb_version}",
+      "GRAPHDB_VERSION=${var.graphdb_version}",
     ]
     inline      = ["sudo -E bash /tmp/install_graphdb.sh"]
     max_retries = 2

create_image_definition.sh

@@ -1,11 +1,11 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 # This script sets up Azure Shared Image Gallery (SIG) image definition and builds an image using Packer.
 # It checks for a variables file, extracts necessary parameters, constructs an Azure CLI command for image definition creation,
 # executes the command, waits for creation completion,
 # and finally uses Packer to build the image based on the specified variables.
 
-set -euxo pipefail
+set -euo pipefail
 
 # Required when running the script on Windows
 alias az=az.cmd || true
@@ -14,49 +14,49 @@ alias az=az.cmd || true
 variables_file="variables.pkrvars.hcl"
 
 # Check if the file exists
-if [ -f "$variables_file" ]; then
-  # Extracts required variables from the variables_file
-  subscription_id=$(grep 'subscription_id' "$variables_file" | cut -d '"' -f 2)
-  echo "Subscription ID: $subscription_id"
-  image_definition_name=$(grep 'image_definition_name' "$variables_file" | cut -d '"' -f 2)
-  echo "Image definition name: $image_definition_name"
-  gdb_version=$(grep 'gdb_version' "$variables_file" | cut -d '"' -f 2)
-  echo "GraphDB version: $gdb_version"
-  gallery_resource_group=$(grep 'gallery_resource_group' "$variables_file" | cut -d '"' -f 2)
-  echo "Resource group: $gallery_resource_group"
-  gallery_name=$(grep 'gallery_name' "$variables_file" | cut -d '"' -f 2)
-  echo "Gallery: $gallery_name"
-
-  # Checks if any of the required variables is empty
-  if [ -z "$image_definition_name" ] || [ -z "$gdb_version" ] || [ -z "$gallery_resource_group" ] || [ -z "$gallery_name" ] || [ -z "$subscription_id" ]; then
-    echo "One or more required variables are not defined in $variables_file."
-    exit 1
-  fi
-
-  # Constructs the az sig image-definition create command
-  az_command="az sig image-definition create \
-       --subscription $subscription_id \
-       -g $gallery_resource_group \
-       --gallery-name $gallery_name \
-       --gallery-image-definition "$image_definition_name" \
-       --publisher Ontotext \
-       --offer GraphDB \
-       --sku "$gdb_version" \
-       --os-type Linux \
-       --hyper-v-generation v2 \
-       --minimum-cpu-core 4 \
-       --maximum-cpu-core 64 \
-       --minimum-memory 4 \
-       --maximum-memory 128 "
-
-  echo "Extracted variables and constructed Azure CLI command:"
-  echo "Creating SIG"
-  eval "$az_command"
-  # Waits for the Shared Image Gallery to be created
-  az sig image-definition wait -i "$image_definition_name" -r "$gallery_name" -g "$gallery_resource_group" --created --subscription $subscription_id
-  echo "Begin building of the Azure VM image"
-  packer build -var-file="variables.pkrvars.hcl" .
-else
+if ! [ -f "$variables_file" ]; then
   echo "The variables file $variables_file does not exist."
   exit 1
 fi
+
+# Extracts required variables from the variables_file
+subscription_id=$(grep 'subscription_id' "$variables_file" | cut -d '"' -f 2)
+echo "Subscription ID: $subscription_id"
+gallery_image_definition=$(grep 'gallery_image_definition' "$variables_file" | cut -d '"' -f 2)
+echo "Image definition name: $gallery_image_definition"
+graphdb_version=$(grep 'graphdb_version' "$variables_file" | cut -d '"' -f 2)
+echo "GraphDB version: $graphdb_version"
+gallery_resource_group=$(grep 'gallery_resource_group' "$variables_file" | cut -d '"' -f 2)
+echo "Resource group: $gallery_resource_group"
+gallery_name=$(grep 'gallery_name' "$variables_file" | cut -d '"' -f 2)
+echo "Gallery: $gallery_name"
+
+# Checks if any of the required variables is empty
+if [ -z "$gallery_image_definition" ] || [ -z "$graphdb_version" ] || [ -z "$gallery_resource_group" ] || [ -z "$gallery_name" ] || [ -z "$subscription_id" ]; then
+  echo "One or more required variables are not defined in $variables_file."
+  exit 1
+fi
+
+# Constructs the az sig image-definition create command
+az_command="az sig image-definition create \
+     --subscription $subscription_id \
+     -g $gallery_resource_group \
+     --gallery-name $gallery_name \
+     --gallery-image-definition "$gallery_image_definition" \
+     --publisher Ontotext \
+     --offer GraphDB \
+     --sku "$graphdb_version" \
+     --os-type Linux \
+     --hyper-v-generation v2 \
+     --minimum-cpu-core 4 \
+     --maximum-cpu-core 64 \
+     --minimum-memory 4 \
+     --maximum-memory 128 "
+
+# TODO: defuk is this :
+echo "Extracted variables and constructed Azure CLI command:"
+echo "Creating SIG"
+eval "$az_command"
+# Waits for the Shared Image Gallery to be created
+az sig image-definition wait -i "$gallery_image_definition" -r "$gallery_name" -g "$gallery_resource_group" --created --subscription $subscription_id
+echo "Begin building of the Azure VM image"