limit CI

Signed-off-by: avifenesh <aviarchi1994@gmail.com>

.github/DEVELOPER.md

@@ -0,0 +1,139 @@
+# CI/CD Workflow Guide
+
+TODO: Add a description of the CI/CD workflow and its components.
+
+### Overview
+
+Our CI/CD pipeline tests and builds our project across multiple languages, versions, and environments. This guide outlines the key components and processes of our workflow.
+
+### Workflow Triggers
+
+*   Push to `main` branch
+*   Pull requests
+*   Scheduled runs (daily)
+*   Manual trigger (workflow_dispatch)
+*   Pull request with `Core changes` label
+
+### Language*Specific Workflows
+
+* Python (python.yml)
+
+* Node.js (node.yml)
+
+* Java (java.yml)
+
+Each language has its own workflow file with similar structure but language-specific steps.
+
+### Shared Components
+
+#### Run Field
+
+The `run` field in both engine-matrix.json and build-matrix.json is used to specify when a particular configuration should be executed:
+
+*   In engine*matrix.json:
+
+    *   `"run": "always"`: The engine version will be tested in every workflow run.
+    *   If `run` is omitted: The engine version will only be tested in full matrix runs (e.g., when the `Core changes` label is applied).
+
+*   In build*matrix.json:
+    *   The `run` array specifies which language workflows should use this host configuration.
+    *   `"always"` in the array means the host will be used in every workflow run for the specified languages.
+
+This allows for flexible control over which configurations are tested in different scenarios, optimizing CI/CD performance and resource usage.
+
+#### Engine Matrix (engine-matrix.json)
+
+Defines the versions of Valkey engine to test against:
+
+```json
+[
+    { "type": "valkey", "version": "7.2.5", "run": "always" },
+    { "type": "valkey", "version": "redis-7.0.15" },
+    { "type": "valkey", "version": "redis-6.2.14", "run": "always" }
+]
+```
+
+#### Build Matrix (build-matrix.json)
+
+Defines the host environments for testing:
+
+```json
+[
+    {
+        "OS": "ubuntu",
+        "RUNNER": "ubuntu-latest",
+        "TARGET": "x86_64-unknown-linux-gnu",
+        "run": ["always", "python", "node", "java"]
+    }
+    // ... other configurations
+]
+```
+
+#### Adding New Versions or Hosts
+
+To add a new engine version, update engine-matrix.json.
+To add a new host environment, update build-matrix.json.
+
+Ensure new entries have all required fields:
+
+For engine versions: `type`, `version`, `run` (optional)
+For hosts: `OS`, `RUNNER`, `TARGET`, `run`
+
+#### Triggering Workflows
+
+Push to main or create a pull request to run workflows automatically.
+Add `Core changes` label to a pull request to trigger full matrix tests.
+Use workflow_dispatch for manual triggers.
+
+### Mutual vs. Language-Specific Components
+
+#### Mutual
+
+`Engine matrix`
+`Build matrix`
+`Shared dependencies installation`
+`Linting (Rust)`
+
+#### Language*Specific
+
+`Package manager commands`
+`Testing frameworks`
+`Build processes`
+`Version matrices`
+
+### Customizing Workflows
+
+Modify `[language].yml` files to adjust language-specific steps.
+Update matrix files to change tested versions or environments.
+Adjust cron schedules in workflow files for different timing of scheduled runs.
+
+### Workflow Matrices
+
+We use dynamic matrices for our CI/CD workflows, which are created using the `create-tests-matrices` action. This action is defined in `.github/workflows/create-tests-matrices/action.yml`.
+
+#### How it works
+
+1. The action is called with a `language-name` input.
+2. It checks for the presence of a "Core changes" label on pull requests.
+3. It generates two matrices:
+   - Engine matrix: Defines the versions of Valkey engine to test against.
+   - Host matrix: Defines the host environments for testing.
+
+#### Outputs
+
+- `should_run`: Indicates whether the job should run based on the presence of the "Core changes" label.
+- `engine-matrix-output`: The generated engine matrix.
+- `host-matrix-output`: The generated host matrix.
+
+#### Matrix Generation Logic
+
+- For pull requests and pushes, only configurations marked with `"run": "always"` are included in the matrices.
+- For other events (like scheduled runs), all configurations are included.
+- The host matrix is filtered based on the `language-name` input and the `run` array in the build-matrix.json file.
+
+#### Customizing Matrices
+
+- To modify the engine versions tested, update the `.github/json_matrices/engine-matrix.json` file.
+- To change the host environments, modify the `.github/json_matrices/build-matrix.json` file.
+
+This dynamic matrix generation allows for flexible and efficient CI/CD workflows, adapting the test configurations based on the type of change and the specific language being tested.

.github/json_matrices/build-matrix.json

@@ -5,7 +5,8 @@
         "RUNNER": "ubuntu-latest",
         "ARCH": "x64",
         "TARGET": "x86_64-unknown-linux-gnu",
-        "PACKAGE_MANAGERS": ["pypi", "npm"]
+        "PACKAGE_MANAGERS": ["pypi", "npm"],
+        "run": ["always", "python", "node", "java"]
     },
     {
         "OS": "ubuntu",
@@ -14,23 +15,26 @@
         "ARCH": "arm64",
         "TARGET": "aarch64-unknown-linux-gnu",
         "PACKAGE_MANAGERS": ["pypi", "npm"],
-        "CONTAINER": "2_28"
+        "CONTAINER": "2_28",
+        "run": ["python", "node"]
     },
     {
         "OS": "macos",
         "NAMED_OS": "darwin",
         "RUNNER": "macos-12",
         "ARCH": "x64",
         "TARGET": "x86_64-apple-darwin",
-        "PACKAGE_MANAGERS": ["pypi", "npm"]
+        "PACKAGE_MANAGERS": ["pypi", "npm"],
+        "run": ["python", "node"]
     },
     {
         "OS": "macos",
         "NAMED_OS": "darwin",
         "RUNNER": "macos-latest",
         "ARCH": "arm64",
         "TARGET": "aarch64-apple-darwin",
-        "PACKAGE_MANAGERS": ["pypi", "npm"]
+        "PACKAGE_MANAGERS": ["pypi", "npm"],
+        "run": ["python", "node", "java"]
     },
     {
         "OS": "ubuntu",
@@ -40,7 +44,8 @@
         "RUNNER": ["self-hosted", "Linux", "ARM64"],
         "IMAGE": "node:alpine",
         "CONTAINER_OPTIONS": "--user root --privileged --rm",
-        "PACKAGE_MANAGERS": ["npm"]
+        "PACKAGE_MANAGERS": ["npm"],
+        "run": ["node"]
     },
     {
         "OS": "ubuntu",
@@ -50,6 +55,7 @@
         "RUNNER": "ubuntu-latest",
         "IMAGE": "node:alpine",
         "CONTAINER_OPTIONS": "--user root --privileged",
-        "PACKAGE_MANAGERS": ["npm"]
+        "PACKAGE_MANAGERS": ["npm"],
+        "run": ["node"]
     }
 ]

.github/workflows/create-tests-matrices/action.yml

@@ -0,0 +1,59 @@
+name: Create tests matrices
+
+on:
+    workflow_call:
+        inputs:
+            language-name:
+                description: "Should run the job"
+                required: true
+                default: "false"
+        outputs:
+            should_run:
+                description: "Should run the job"
+                value: ${{ check_should_run.outputs.should_run }}
+            engine-matrix-output:
+                description: "Engine matrix"
+                value: ${{ load-engine-matrix.outputs.matrix }}
+            host-matrix-output:
+                description: "Host matrix"
+                value: ${{ load-host-matrix.outputs.matrix }}
+
+runs:
+    using: "composite"
+    steps:
+        - name: Checkout
+          uses: actions/checkout@v4
+
+        - name: Check for Core changes label
+          id: check_should_run
+          shell: bash
+          run: |
+              if [[ ("${{ github.event.label.name }}" == "Core changes" &&  "${{github.event_name}}" == 'pull_request_target') || "${{github.event_name}}" != 'pull_request_target' ]]; then
+                echo "should_run=true" >> $GITHUB_OUTPUT
+              else
+                echo "should_run=false" >> $GITHUB_OUTPUT
+              fi
+
+        - name: Load engine matrix
+          # We want to run this job only if the previous job outputs should_run=true,
+          # Which means that if the trigger was a label we run this job just if the label was Core changes
+          if: ${{ '$should_run' == 'true' }}
+          id: load-engine-matrix
+          shell: bash
+          run: |
+              if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" ]]; then
+                echo "matrix=$(jq -c '[.[] | select(.run == "always")]' < .github/json_matrices/engine-matrix.json)" >> $GITHUB_OUTPUT
+              else
+                echo "matrix=$(jq -c . < .github/json_matrices/engine-matrix.json)" >> $GITHUB_OUTPUT
+              fi
+
+        - name: load-host-matrix
+        - id: load-host-matrix
+          if: ${{ '$should_run' == 'true' }}
+          shell: bash
+          run: |
+              if [[ "${{ github.event_name }}" == "pull_request" || "${{ github.event_name }}" == "push" ]]; then
+                echo 'matrix={"include":' $(jq '[.[] | select(.run | type == "array" and contains(["always"]))]' .github/json_matrices/build-matrix.json) '}' >> $GITHUB_OUTPUT
+              else
+                echo 'matrix={"include":' $(jq '[.[] | select(.run | type == "array" and contains([${{ inputs.language-name }}]))]' .github/json_matrices/build-matrix.json) '}' >> $GITHUB_OUTPUT
+              fi