Update Azure Load Test ApiVersion (#984)

* bump load test apiversion * Component updates (#983) * Bump hashicorp/azurerm in /src/infra/monitoring/grafana/terraform/stamps Bumps [hashicorp/azurerm](https://github.com/hashicorp/terraform-provider-azurerm) from 3.58.0 to 3.59.0. - [Release notes](https://github.com/hashicorp/terraform-provider-azurerm/releases) - [Changelog](https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md) - [Commits](hashicorp/terraform-provider-azurerm@v3.58.0...v3.59.0) --- updated-dependencies: - dependency-name: hashicorp/azurerm dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump hashicorp/azurerm in /src/infra/workload/globalresources Bumps [hashicorp/azurerm](https://github.com/hashicorp/terraform-provider-azurerm) from 3.55.0 to 3.59.0. - [Release notes](https://github.com/hashicorp/terraform-provider-azurerm/releases) - [Changelog](https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md) - [Commits](hashicorp/terraform-provider-azurerm@v3.55.0...v3.59.0) --- updated-dependencies: - dependency-name: hashicorp/azurerm dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump hashicorp/azurerm Bumps [hashicorp/azurerm](https://github.com/hashicorp/terraform-provider-azurerm) from 3.55.0 to 3.59.0. - [Release notes](https://github.com/hashicorp/terraform-provider-azurerm/releases) - [Changelog](https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md) - [Commits](hashicorp/terraform-provider-azurerm@v3.55.0...v3.59.0) --- updated-dependencies: - dependency-name: hashicorp/azurerm dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump grafana/grafana in /src/infra/monitoring/grafana Bumps grafana/grafana from 9.5.1 to 9.5.3. --- updated-dependencies: - dependency-name: grafana/grafana dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump hashicorp/azurerm in /src/testing/userload-generator/infra Bumps [hashicorp/azurerm](https://github.com/hashicorp/terraform-provider-azurerm) from 3.55.0 to 3.59.0. - [Release notes](https://github.com/hashicorp/terraform-provider-azurerm/releases) - [Changelog](https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md) - [Commits](hashicorp/terraform-provider-azurerm@v3.55.0...v3.59.0) --- updated-dependencies: - dependency-name: hashicorp/azurerm dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump hashicorp/azurerm in /src/testing/loadtest-locust/infra Bumps [hashicorp/azurerm](https://github.com/hashicorp/terraform-provider-azurerm) from 3.55.0 to 3.59.0. - [Release notes](https://github.com/hashicorp/terraform-provider-azurerm/releases) - [Changelog](https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md) - [Commits](hashicorp/terraform-provider-azurerm@v3.55.0...v3.59.0) --- updated-dependencies: - dependency-name: hashicorp/azurerm dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Bump hashicorp/azurerm in /src/infra/workload/releaseunit Bumps [hashicorp/azurerm](https://github.com/hashicorp/terraform-provider-azurerm) from 3.55.0 to 3.59.0. - [Release notes](https://github.com/hashicorp/terraform-provider-azurerm/releases) - [Changelog](https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md) - [Commits](hashicorp/terraform-provider-azurerm@v3.55.0...v3.59.0) --- updated-dependencies: - dependency-name: hashicorp/azurerm dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * api changes * update api * move to native azurerm * tf update * fix url * more api updates * update * fixes * fix * fixes * no auto stop * fix * fix * fixes * no comp * fixes * updates * fix script * higher criteria * update tf cli * var location * wait for test result * tf down * tf down * or * eq * fix * component back * uncomment * stop timeout run * reset numbers * cleanup * multiregion * eastus2 * maybe norway * missing file * 1 instance * use httpclient * times fix * fix more * shorten * shorten another * update base * single --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sebastian <mail@silenced.eu>
Azure · Jul 20, 2023 · ea7b4bc · ea7b4bc
1 parent a83e7a3
commit ea7b4bc
Show file tree

Hide file tree

Showing 24 changed files with 245 additions and 258 deletions.
diff --git a/.ado/pipelines/config/configuration.yaml b/.ado/pipelines/config/configuration.yaml
@@ -44,11 +44,11 @@ variables:
 
 # Microsoft Azure Load Test (MALT) configuration
 - name: 'azureLoadTestApiVersion' # data plane
-  value: '2022-06-01-preview'
+  value: '2023-04-01-preview'
+- name: 'azureLoadTestLocation'
+  value: 'uksouth'
 - name: 'azureLoadTestEngineInstances'
-  value: 2
-- name: 'azureLoadTestVUsers'
-  value: 100
+  value: 1
 - name: 'azureLoadTestDurationSeconds'
   value: 660
 - name: 'azureLoadTestUserThreads'

diff --git a/.ado/pipelines/config/loadtest-baseline-chaos.json b/.ado/pipelines/config/loadtest-baseline-chaos.json
@@ -1,36 +1,28 @@
-[
-    {
-        "passFailMetrics": {
-            "ece127bc-6271-4a70-93f6-758c56a3f11c": {
-                "clientmetric": "requests_per_sec",
-                "aggregate": "avg",
-                "condition": "<",
-                "value": 1200.0,
-                "result": null,
-                "actualValue": 0.0,
-                "requestName": null,
-                "action": "continue"
-            },
-            "ac6c5e62-647a-4a1a-b001-b961e384891f": {
-                "action": "continue",
-                "actualValue": 0.0,
-                "aggregate": "avg",
-                "clientmetric": "response_time_ms",
-                "condition": ">",
-                "requestName": null,
-                "result": null,
-                "value": 75.0
-              },
-              "a7ab8dec-07e6-4870-911c-4505382b8511": {
-                "action": "continue",
-                "actualValue": 0.0,
-                "aggregate": "percentage",
-                "clientmetric": "error",
-                "condition": ">",
-                "requestName": null,
-                "result": null,
-                "value": 50.0
-              }
+{
+    "passFailMetrics": {
+        "ece127bc-6271-4a70-93f6-758c56a3f11c": {
+            "clientMetric": "requests_per_sec",
+            "aggregate": "avg",
+            "condition": "<",
+            "value": 1200.0,
+            "requestName": null,
+            "action": "continue"
+        },
+        "ac6c5e62-647a-4a1a-b001-b961e384891f": {
+            "action": "continue",
+            "aggregate": "avg",
+            "clientMetric": "response_time_ms",
+            "condition": ">",
+            "requestName": null,
+            "value": 200.0
+        },
+        "a7ab8dec-07e6-4870-911c-4505382b8511": {
+            "action": "continue",
+            "aggregate": "percentage",
+            "clientMetric": "error",
+            "condition": ">",
+            "requestName": null,
+            "value": 50.0
         }
     }
-]
+}
diff --git a/.ado/pipelines/config/loadtest-baseline-normal.json b/.ado/pipelines/config/loadtest-baseline-normal.json
@@ -1,36 +1,28 @@
-[
-    {
-        "passFailMetrics": {
-            "ece127bc-6271-4a70-93f6-758c56a3f12c": {
-                "clientmetric": "requests_per_sec",
-                "aggregate": "avg",
-                "condition": "<",
-                "value": 1200.0,
-                "result": null,
-                "actualValue": 0.0,
-                "requestName": null,
-                "action": "continue"
-            },
-            "ac6c5e62-647a-4a1a-b001-b961e384892f": {
-                "action": "continue",
-                "actualValue": 0.0,
-                "aggregate": "avg",
-                "clientmetric": "response_time_ms",
-                "condition": ">",
-                "requestName": null,
-                "result": null,
-                "value": 75.0
-              },
-              "a7ab8dec-07e6-4870-911c-4505382b85f1": {
-                "action": "continue",
-                "actualValue": 0.0,
-                "aggregate": "percentage",
-                "clientmetric": "error",
-                "condition": ">",
-                "requestName": null,
-                "result": null,
-                "value": 0.0
-              }
+{
+    "passFailMetrics": {
+        "ece127bc-6271-4a70-93f6-758c56a3f12c": {
+            "clientMetric": "requests_per_sec",
+            "aggregate": "avg",
+            "condition": "<",
+            "value": 1200.0,
+            "requestName": null,
+            "action": "continue"
+        },
+        "ac6c5e62-647a-4a1a-b001-b961e384892f": {
+            "action": "continue",
+            "aggregate": "avg",
+            "clientMetric": "response_time_ms",
+            "condition": ">",
+            "requestName": null,
+            "value": 100.0
+        },
+        "a7ab8dec-07e6-4870-911c-4505382b85f1": {
+            "action": "continue",
+            "aggregate": "percentage",
+            "clientMetric": "error",
+            "condition": ">",
+            "requestName": null,
+            "value": 1.0
         }
     }
-]
+}
diff --git a/.ado/pipelines/templates/stages-loadtest-azure.yaml b/.ado/pipelines/templates/stages-loadtest-azure.yaml
@@ -50,7 +50,8 @@ stages:
         customPrefix:              '${{ parameters.customPrefix }}'
         environment:               '$(environment)'
         customAttributes:          '-var=queued_by="$(Build.QueuedBy)"
-                                    -var=branch="$(sourceBranch)"'
+                                    -var=branch="$(sourceBranch)"
+                                    -var=location="$(azureLoadTestLocation)"'
 
     # Parsing the Terraform output for the MALT service deployment
     - template: steps-parse-terraform-output.yaml
@@ -120,8 +121,8 @@ stages:
         arguments:
           -apiEndpoint "$(azureLoadTestDataPlaneURI)" `
           -apiVersion "$(azureLoadTestApiVersion)" `
-          -loadTestDisplayName "$(get-date -f "yyyy-MM-dd hh:ss") load test for build $(Build.BuildId)" `
-          -loadTestDescription "Pipeline-embedded load test for $(Build.BuildId) ($(get-date -f "dd/MM/yyyy hh:ss"))" `
+          -loadTestDisplayName "Load test run for build $(Build.BuildId)" `
+          -loadTestDescription "Pipeline-embedded load test for $(Build.BuildId) ($(get-date -AsUTC -f "yyyy-MM-ddTHH:mm:ssZ"))" `
           -loadTestTargetUrl "$(azureLoadTestTargetFQDN)" `
           -loadTestUserThreads "$(azureLoadTestUserThreads)" `
           -loadTestDurationSeconds "$(azureLoadTestDurationSeconds)" `
@@ -149,13 +150,13 @@ stages:
             foreach($stamp in $releaseUnitInfraDeployOutput.stamp_properties.value) {
               echo "**************************** REGION: $($stamp.location) ****************************"
 
+              echo "*** Adding $($stamp.aks_cluster_id) as app component"
+
               ./src/testing/loadtest-azure/scripts/appcomponents-add-to-loadtest.ps1 `
                 -loadTestId "$(loadTestId)" `
                 -apiEndpoint "$(azureLoadTestDataPlaneURI)" `
                 -apiVersion "$(azureLoadTestApiVersion)" `
                 -resourceId "$($stamp.aks_cluster_id)"
-
-              echo "*** Adding $($stamp.aks_cluster_id)"
             }
 
       # upload a jmx file for the previously created azure load test with an auto-generated testFileId
@@ -177,7 +178,7 @@ stages:
 
       # start azure load test
       - task: AzureCLI@2
-        displayName: 'Run Azure Load Test'
+        displayName: 'Start Azure Load Test'
         inputs:
           azureSubscription: '$(azureServiceConnection)'
           scriptType: pscore
@@ -187,9 +188,8 @@ stages:
             -loadTestId "$(loadTestId)" `
             -apiEndpoint "$(azureLoadTestDataPlaneURI)" `
             -apiVersion "$(azureLoadTestApiVersion)" `
-            -testRunName "$(get-date -f "yyyy-MM-dd hh:ss") Load Test run triggered by ADO" `
+            -testRunName "$(get-date -f "yyyy-MM-ddTHH:mm:ssZ") run triggered by ADO" `
             -testRunDescription "Pipeline executed load test run" `
-            -testRunVUsers "$(azureLoadTestVUsers)" `
             -pipeline $true `
             -verbose:$true
 
@@ -203,9 +203,18 @@ stages:
           scriptLocation: inlineScript
           inlineScript: |
 
+            # set retry wait time in seconds
+            $retryWaitSeconds = 90
+
+            # set timeout in minutes
+            $timeoutMinutes = 45
+
+            #create a variable to sum up time in seconds
+            $totalTime = 0
+
             do {
-              echo "*** Waiting additional 90 seconds for the load test run to complete.."
-              start-sleep -seconds 90
+              echo "*** Waiting additional $retryWaitSeconds seconds for the load test run to complete.."
+              start-sleep -seconds $retryWaitSeconds
               $result = $(System.DefaultWorkingDirectory)/src/testing/loadtest-azure/scripts/loadtest-get-run.ps1 `
                           -apiEndpoint "$(azureLoadTestDataPlaneURI)" `
                           -apiVersion "$(azureLoadTestApiVersion)" `
@@ -217,16 +226,26 @@ stages:
                 throw "*** ERROR: Test run $(testRunId) ended in $($result.status) state."
               } else {
                 # test is still running
-                echo "*** Test Run $(testRunId) is in status $testRunStatus"
+                echo "*** Test Run $(testRunId) is in status $testRunStatus Test Result: $($result.testResult)"
               }
 
-              $resultUrl = ($result).testArtifacts.outputArtifacts.resultUrl.url
-              $logsUrl = ($result).testArtifacts.outputArtifacts.logsUrl.url
+              $totalTime += $retryWaitSeconds
 
-              # todo - timeout?
-            } while ($result.status -ne "DONE")
+              if($totalTime -gt ($timeoutMinutes * 60)) 
+              {
+                echo "*** Test run $(testRunId) did not finish in $timeoutMinutes minutes. Canceling test run..."
+                # Stop run
+                $(System.DefaultWorkingDirectory)/src/testing/loadtest-azure/scripts/loadtest-stop.ps1 `
+                          -apiEndpoint "$(azureLoadTestDataPlaneURI)" `
+                          -apiVersion "$(azureLoadTestApiVersion)" `
+                          -testRunId "$(testRunId)"
+                throw "*** ERROR: Test run $(testRunId) did not finish in $timeoutMinutes minutes. Test Run Status: $testRunStatus Test Result: $($result.testResult)"
+              }
+
+              # Wait until test is done and the rest result (evaluation of the test criteria is done)
+            } while ($result.status -ne "DONE" -or $result.testResult -eq "NOT_APPLICABLE")
 
-            echo "*** Test Run $(testRunId) was completed. Test Run Status: $testRunStatus Test Status: $($result.testResult)"
+            echo "*** Test Run $(testRunId) was completed. Test Run Status: $testRunStatus Test Result: $($result.testResult)"
             echo "*** Portal URL: $($result.portalUrl)"
 
             # throw an error when testResult is FAILED
@@ -235,7 +254,7 @@ stages:
             }
 
       - task: AzureCLI@2
-        displayName: 'RESULTS: Download and convert JMeter results'
+        displayName: 'RESULTS: Download Load Test results'
         condition: succeeded()
         inputs:
           azureSubscription: $(azureServiceConnection)
@@ -252,19 +271,18 @@ stages:
                 -apiVersion "$(azureLoadTestApiVersion)" `
                 -testRunId "$(testRunId)"
 
-              $resultUrl = ($result).testArtifacts.outputArtifacts.resultUrl.url
-              $logsUrl = ($result).testArtifacts.outputArtifacts.logsUrl.url
+              $resultUrl = ($result).testArtifacts.outputArtifacts.resultFileInfo.url
+              $logsUrl = ($result).testArtifacts.outputArtifacts.logsFileInfo.url
 
-              if ( (!$resultUrl) -and (!$logsUrl) ) {
+              if ( (!$resultUrl) -or (!$logsUrl) ) {
                 echo "*** Either resultUrl ($resultUrl) or logsUrl ($logsUrl) is empty. Retry $i/3"
                 Start-Sleep -seconds 15
               } else {
                 echo "*** Received resultUrl ($resultUrl) and logsUrl ($logsUrl)"
-                $i=3 # set to 3 to end the loop
+                $i = 3 # set to 3 to end the loop
               }
-
               $i++
-            } while ( ($i -le 3) -and (($logsUrl) -and ($resultUrl)) )
+            } while ( $i -le 3 )
 
             if ( ($resultUrl) -and ($logsUrl) ) {
               New-Item -Path results -ItemType "directory" -Force
@@ -276,20 +294,22 @@ stages:
               throw "*** ERROR: Either logsUrl ($logsUrl) or resultUrl ($resultUrl) is empty."
             }
 
-            #wget https://raw.githubusercontent.com/Azure-Samples/jmeter-aci-terraform/main/scripts/jtl_junit_converter.py -O $(System.DefaultWorkingDirectory)/junit-converter.py
+            # Conversion of JMeter results to JUnit format currently not being used due to long runtime duration of the script
+            #
+            # wget https://raw.githubusercontent.com/Azure-Samples/jmeter-aci-terraform/main/scripts/jtl_junit_converter.py -O $(System.DefaultWorkingDirectory)/junit-converter.py
 
-            if (Test-Path results/results.zip) {
-              Expand-Archive -Path results/results.zip -DestinationPath results
+            # if (Test-Path results/results.zip) {
+            #   Expand-Archive -Path results/results.zip -DestinationPath results
 
-              # merge multiple csv files
-              # Get-ChildItem -Filter results/*.csv | Import-Csv | Export-Csv results/testreport.csv -NoTypeInformation -Append
+            #   # merge multiple csv files
+            #   Get-ChildItem -Filter results/*.csv | Import-Csv | Export-Csv results/testreport.csv -NoTypeInformation -Append
 
-              #$JMETER_RESULTS="results/testreport.csv"
-              #$JUNIT_RESULTS="output.xml"
-              #python3 junit-converter.py $JMETER_RESULTS $JUNIT_RESULTS
-            } else {
-              echo "Skipping result data conversion. results.zip was not found."
-            }
+            #   $JMETER_RESULTS="results/testreport.csv"
+            #   $JUNIT_RESULTS="output.xml"
+            #   python3 junit-converter.py $JMETER_RESULTS $JUNIT_RESULTS
+            # } else {
+            #   echo "Skipping result data conversion. results.zip was not found."
+            # }
             
         
       - task: PublishTestResults@2
@@ -317,4 +337,5 @@ stages:
           terraformStorageResourceGroupName: '$(terraformResourceGroup)'
           terraformStateFilename:     'terraform-azurelt-${{ parameters.customPrefix }}.state'
           terraformWorkingDirectory:  '${{ parameters.terraformWorkingDirectory }}'
-          customAttributes:           '-var=prefix="${{ parameters.customPrefix }}"'
+          customAttributes:           '-var=prefix="${{ parameters.customPrefix }}"
+                                      -var=location="$(azureLoadTestLocation)"'
diff --git a/src/infra/monitoring/grafana/terraform/globalresources/main.tf b/src/infra/monitoring/grafana/terraform/globalresources/main.tf
@@ -2,7 +2,7 @@ terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
-      version = "3.63.0"
+      version = "3.65.0"
     }
   }
 

diff --git a/src/infra/monitoring/grafana/terraform/stamps/main.tf b/src/infra/monitoring/grafana/terraform/stamps/main.tf
@@ -2,7 +2,7 @@ terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
-      version = "3.63.0"
+      version = "3.65.0"
     }
   }
 

diff --git a/src/infra/workload/globalresources/main.tf b/src/infra/workload/globalresources/main.tf
@@ -2,7 +2,7 @@ terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
-      version = "3.63.0"
+      version = "3.65.0"
     }
     azapi = {
       source  = "Azure/azapi"

diff --git a/src/infra/workload/releaseunit/main.tf b/src/infra/workload/releaseunit/main.tf
@@ -2,11 +2,11 @@ terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
-      version = "3.63.0"
+      version = "3.65.0"
     }
     azapi = {
       source  = "Azure/azapi"
-      version = "1.5.0"
+      version = "1.7.0"
     }
   }
 

diff --git a/src/infra/workload/releaseunit/modules/stamp/main.tf b/src/infra/workload/releaseunit/modules/stamp/main.tf
@@ -2,11 +2,11 @@ terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
-      version = "3.63.0"
+      version = "3.65.0"
     }
     azapi = {
       source  = "Azure/azapi"
-      version = "1.5.0"
+      version = "1.7.0"
     }
   }
 }