fixed triggers

Signed-off-by: avifenesh <aviarchi1994@gmail.com>

.github/DEVELOPER.md

@@ -1,17 +1,25 @@
 # CI/CD Workflow Guide
 
-TODO: Add a description of the CI/CD workflow and its components.
-
 ### Overview
 
 Our CI/CD pipeline tests and builds our project across multiple languages, versions, and environments. This guide outlines the key components and processes of our workflow.
 
 ### Workflow Triggers
 
--   Push to `main` branch
--   Pull requests
--   Scheduled runs (daily)
--   Manual trigger (workflow_dispatch)
+*   Pull requests
+*   Pushes to `main` or release branches (PR merges)
+*   Scheduled runs (daily) - starts CI pipelines for all clients
+*   Manual trigger (`workflow_dispatch`) - a developer can start a client's pipeline or the scheduled one to run all pipelines on demand
+
+<img width="437" alt="Job triggers" src="https://github.com/user-attachments/assets/58bf2b76-d778-4e43-8891-5dcbf0ff9b72">
+
+### Test coverage
+
+There are two levels of testing: the basic one and full (_aka_ `full-matrix`).
+Basic amount of testing is executed on every open and merged PR. The full set of tests is executed by the scheduled job.
+A developer can select the level when starting a job, either scheduled or client's pipeline.
+
+<img width="264" alt="Matrices" src="https://github.com/user-attachments/assets/72857f80-078c-4e11-bcc6-75beb0125a9d">
 
 ### Language-Specific Workflows
 
@@ -21,12 +29,12 @@ Each language has its own workflow file with similar structure but language-spec
 
 #### Matrix Files
 
-While workflows are language-specific, the matrix files are shared across all workflows.
+While workflows are language-specific, the matrix files are shared across all workflows. 
 Workflows are starting by loading the matrix files from the `.github/json_matrices` directory.
 
--   `engine-matrix.json`: Defines the versions of Valkey engine to test against.
--   `build-matrix.json`: Defines the host environments for testing.
--   `supported-languages-version.json`: Defines the supported versions of languages.
+* `engine-matrix.json`: Defines the versions of the engine to test against.
+* `build-matrix.json`: Defines the host environments for testing.
+* `supported-languages-versions.json`: Defines the supported versions of languages.
 
 All matrices have a `run` like field which specifies if the configuration should be tested on every workflow run.
 This allows for flexible control over which configurations are tested in different scenarios, optimizing CI/CD performance and resource usage.
@@ -36,12 +44,16 @@ This allows for flexible control over which configurations are tested in differe
 Defines the versions of Valkey engine to test against:
 
 ```json
-[{ "type": "valkey", "version": "7.2.5", "run": "always" }]
+[
+    { "type": "valkey", "version": "7.2.5", "run": "always" },
+    // ... other configurations
+]
 ```
 
--   `type`: The type of engine (e.g., Valkey, Redis).
--   `version`: The version of the engine.
--   `run`: Specifies if the engine version should be tested on every workflow.
+* `type`: The type of engine (e.g., Valkey, Redis).
+* `version`: Specifies the engine version that the workflow should checkout.
+  For example, "7.2.5" represents a release tag, while "7.0" denotes a branch name. The workflow should use this parameter to checkout the specific release version or branch to build the engine with the appropriate version.
+* `run`: Specifies if the engine version should be tested on every workflow.
 
 #### Build Matrix (build-matrix.json)
 
@@ -58,11 +70,10 @@ Defines the host environments for testing:
     // ... other configurations
 ]
 ```
-
--   `OS`: The operating system of the host.
--   `RUNNER`: The GitHub runner to use.
--   `TARGET`: The target environment.
--   `run`: Specifies which language workflows should use this host configuration, always means run on each workflow trigger.
+* `OS`: The operating system of the host.
+* `RUNNER`: The GitHub runner to use.
+* `TARGET`: The target environment as defined in Rust. To see a list of available targets, run `rustup target list`.
+* `run`: Specifies which language workflows should use this host configuration. The value `always` indicates that the configuration should be used for every workflow trigger.
 
 #### Supported Languages Version (supported-languages-version.json)
 
@@ -74,38 +85,37 @@ Defines the supported versions of languages:
         "language": "java",
         "versions": ["11", "17"],
         "always-run-versions": ["17"]
-    }
+    },
     // ... other configurations
 ]
 ```
-
--   `language`: The language for which the version is supported.
--   `versions`: The full versions supported of the language which will test against scheduled.
--   `always-run-versions`: The versions which will be tested in every workflow run.
+* `language`: The language for which the version is supported.
+* `versions`: The full versions supported of the language which will test against scheduled.
+* `always-run-versions`: The versions that will always be tested, regardless of the workflow trigger.
 
 #### Triggering Workflows
 
-Push to main or create a pull request to run workflows automatically.
-Use workflow_dispatch for manual triggers, accepting inputs of full-matrix which is a boolean value to run all configurations.
-Scheduled runs are triggered daily to ensure regular testing of all configurations.
+- Push to `main` by merging a PR or create a new pull request to run workflows automatically.
+- Use `workflow_dispatch` for manual triggers, accepting a boolean configuration parameter to run all configurations.
+- Scheduled runs are triggered daily to ensure regular testing of all configurations.
 
 ### Mutual vs. Language-Specific Components
 
 #### Mutual
 
-`Matrix files` - `.github/json_matrices`
-`Shared dependencies installation` - `.github/workflows/install-shared-dependencies/action.yml`
-`Linting Rust` - `.github/workflows/lint-rust/action.yml`
+- Matrix files - `.github/json_matrices/`
+- Shared dependencies installation - `.github/workflows/install-shared-dependencies/action.yml`
+- Rust linters - `.github/workflows/lint-rust/action.yml`
 
 #### Language-Specific
 
-`Package manager commands`
-`Testing frameworks`
-`Build processes`
+- Package manager commands
+- Testing frameworks
+- Build processes
 
 ### Customizing Workflows
 
-Modify `[language].yml` files to adjust language-specific steps.
+Modify `<language>.yml` files to adjust language-specific steps.
 Update matrix files to change tested versions or environments.
 Adjust cron schedules in workflow files for different timing of scheduled runs.
 
@@ -119,14 +129,13 @@ We use dynamic matrices for our CI/CD workflows, which are created using the `cr
 2. It reads the `engine-matrix.json`, `build-matrix.json`, and `supported-languages-version.json` files.
 3. It filters the matrices based on the inputs and the event type.
 4. It generates three matrices:
-    - Engine matrix: Defines the types and versions of the engine to test against, for example Valkey 7.2.5.
-    - Host matrix: Defines the host platforms to run the tests on, for example Ubuntu on ARM64.
-    - Language-version matrix: Defines the supported versions of languages, for example python 3.8.
+   - Engine matrix: Defines the types and versions of the engine to test against, for example Valkey 7.2.5.
+   - Host matrix: Defines the host platforms to run the tests on, for example Ubuntu on ARM64.
+   - Language-version matrix: Defines the supported versions of languages, for example python 3.8.
 
 #### Outputs
-
--   `engine-matrix-output`: The generated engine matrix.
--   `host-matrix-output`: The generated host matrix.
--   `language-version-matrix-output`: The generated language version matrix.
+- `engine-matrix-output`: The generated engine matrix.
+- `host-matrix-output`: The generated host matrix.
+- `language-version-matrix-output`: The generated language version matrix.
 
 This dynamic matrix generation allows for flexible and efficient CI/CD workflows, adapting the test configurations based on the type of change and the specific language being tested.

.github/json_matrices/build-matrix.json

@@ -5,36 +5,37 @@
         "RUNNER": "ubuntu-latest",
         "ARCH": "x64",
         "TARGET": "x86_64-unknown-linux-gnu",
-        "PACKAGE_MANAGERS": ["pypi", "npm"],
-        "run": ["always", "python", "node", "java"]
+        "PACKAGE_MANAGERS": ["pypi", "npm", "maven"],
+        "run": "always",
+        "languages": ["python", "node", "java", "go", "dotnet"]
     },
     {
         "OS": "ubuntu",
         "NAMED_OS": "linux",
         "RUNNER": ["self-hosted", "Linux", "ARM64"],
         "ARCH": "arm64",
         "TARGET": "aarch64-unknown-linux-gnu",
-        "PACKAGE_MANAGERS": ["pypi", "npm"],
+        "PACKAGE_MANAGERS": ["pypi", "npm", "maven"],
         "CONTAINER": "2_28",
-        "run": ["python", "node", "java"]
+        "languages": ["python", "node", "java", "go", "dotnet"]
     },
     {
         "OS": "macos",
         "NAMED_OS": "darwin",
         "RUNNER": "macos-12",
         "ARCH": "x64",
         "TARGET": "x86_64-apple-darwin",
-        "PACKAGE_MANAGERS": ["pypi", "npm"],
-        "run": ["python", "node", "java"]
+        "PACKAGE_MANAGERS": ["pypi", "npm", "maven"],
+        "languages": ["python", "node", "java", "go", "dotnet"]
     },
     {
         "OS": "macos",
         "NAMED_OS": "darwin",
         "RUNNER": "macos-latest",
         "ARCH": "arm64",
         "TARGET": "aarch64-apple-darwin",
-        "PACKAGE_MANAGERS": ["pypi", "npm"],
-        "run": ["python", "node", "java"]
+        "PACKAGE_MANAGERS": ["pypi", "npm", "maven"],
+        "languages": ["python", "node", "java", "go", "dotnet"]
     },
     {
         "OS": "ubuntu",
@@ -45,7 +46,7 @@
         "IMAGE": "node:alpine",
         "CONTAINER_OPTIONS": "--user root --privileged --rm",
         "PACKAGE_MANAGERS": ["npm"],
-        "run": ["node"]
+        "languages": ["node"]
     },
     {
         "OS": "ubuntu",
@@ -56,6 +57,15 @@
         "IMAGE": "node:alpine",
         "CONTAINER_OPTIONS": "--user root --privileged",
         "PACKAGE_MANAGERS": ["npm"],
-        "run": ["node"]
+        "languages": ["node"]
+    },
+    {
+        "OS": "amazon-linux",
+        "NAMED_OS": "linux",
+        "RUNNER": "ubuntu-latest",
+        "ARCH": "x64",
+        "TARGET": "x86_64-unknown-linux-gnu",
+        "IMAGE": "amazonlinux:latest",
+        "languages": ["python", "node", "java", "go", "dotnet"]
     }
 ]

.github/json_matrices/supported-languages-versions.json

@@ -11,7 +11,17 @@
     },
     {
         "language": "node",
-        "full-versions": ["16.x", "17.x", "18.x", "19.x", "20.x"],
+        "versions": ["16.x", "17.x", "18.x", "19.x", "20.x"],
         "always-run-versions": ["16.x", "20.x"]
+    },
+    {
+        "language": "dotnet",
+        "versions": ["8.0", "6.0"],
+        "always-run-versions": ["8.0"]
+    },
+    {
+        "language": "go",
+        "versions": ["1.22.0", "1.18.10"],
+        "always-run-versions": ["1.22.0"]
     }
 ]

.github/workflows/create-test-matrices/action.yml

@@ -13,11 +13,6 @@ inputs:
         description: "Run the full matrix"
         required: true
         type: boolean
-    containers:
-    # if set to `true`, "load-host-matrix" step selects blocks with "IMAGE" only from `build-matrix` file, otherwise they are ignored
-        description: "`true` to pick containers to the host list instead of runners"
-        default: false
-        type: boolean
 
 outputs:
     engine-matrix-output:
@@ -53,7 +48,7 @@ runs:
           shell: bash
           run: |
               set -o pipefail
-              [[ ${{ inputs.containers }} == true ]] && CONDITION=".IMAGE?" || CONDITION=".IMAGE == null"
+              [[ "${{ inputs.containers }}" == "true" ]] && CONDITION=".IMAGE?" || CONDITION=".IMAGE == null"
               echo 'Select runners (VMs) to run tests on'
               if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" || "${{ inputs.run-full-matrix }}" == "false" ]]; then
                   echo 'Pick runners marked as '"run": "always"' only - on PR, push or manually triggered job which does not require full matrix'
@@ -72,9 +67,9 @@ runs:
               echo 'Select language (framework/SDK) versions to run tests on'
               if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" || "${{ inputs.run-full-matrix }}" == "false" ]]; then
                   echo 'Pick language versions listed in 'always-run-versions' only - on PR, push or manually triggered job which does not require full matrix'
-                  jq -c '[.[] | select(.language == "${{ inputs.language-name }}") | .["always-run-versions"]][0] // []' < .github/json_matrices/supported-language-versions.json | awk '{ printf "version-matrix=%s\n", $1 }' | tee -a $GITHUB_OUTPUT
+                  jq -c '[.[] | select(.language == "${{ inputs.language-name }}") | .["always-run-versions"]][0] // []' < .github/json_matrices/supported-languages-versions.json | awk '{ printf "version-matrix=%s\n", $1 }' | tee -a $GITHUB_OUTPUT
               else
                   echo 'Pick language versions listed in 'versions' - on cron (schedule) or if manually triggered job requires a full matrix'
-                  jq -c '[.[] | select(.language == "${{ inputs.language-name }}") | .versions][0]' < .github/json_matrices/supported-language-versions.json | awk '{ printf "version-matrix=%s\n", $1 }' | tee -a $GITHUB_OUTPUT
+                  jq -c '[.[] | select(.language == "${{ inputs.language-name }}") | .versions][0]' < .github/json_matrices/supported-languages-versions.json | awk '{ printf "version-matrix=%s\n", $1 }' | tee -a $GITHUB_OUTPUT
               fi
               cat $GITHUB_OUTPUT