TEP-140: Produce Results in Matrix

docs/matrix.md

@@ -17,6 +17,8 @@ weight: 406
   - [Parameters in Matrix.Include.Params](#parameters-in-matrixincludeparams)
   - [Specifying both `params` and `matrix` in a `PipelineTask`](#specifying-both-params-and-matrix-in-a-pipelinetask)
 - [Context Variables](#context-variables)
+  - [Access Matrix Combinations Length](#access-matrix-combinations-length)
+  - [Access Aggregated Results Length](#access-aggregated-results-length)
 - [Results](#results)
   - [Specifying Results in a Matrix](#specifying-results-in-a-matrix)
     - [Results in Matrix.Params](#results-in-matrixparams)
@@ -291,6 +293,38 @@ Similarly to the `Parameters` in the `Params` field, the `Parameters` in the `Ma
 * `Pipeline` name
 * `PipelineTask` retries
 
+
+The following `context` variables allow users to access the `matrix` runtime data. Note: In order to create an ordering dependency, use `runAfter` or `taskResult` consumption as part of the same pipelineTask.
+
+#### Access Matrix Combinations Length
+
+The pipeline authors can access the total number of instances created as part of the `matrix` using the syntax: `tasks.<pipelineTaskName>.matrix.length`.
+
+```yaml
+      - name: matrixed-echo-length
+        runAfter:
+          - matrix-emitting-results
+        params:
+          - name: matrixlength
+            value: $(tasks.matrix-emitting-results.matrix.length)
+```
+
+#### Access Aggregated Results Length
+
+The pipeline authors can access the length of the array of aggregated results that were
+actually produced using the syntax: `tasks.<pipelineTaskName>.matrix.<resultName>.length`. This will allow users to loop over the results produced.
+
+```yaml
+      - name: matrixed-echo-results-length
+        runAfter:
+          - matrix-emitting-results
+        params:
+          - name: matrixlength
+            value: $(tasks.matrix-emitting-results.matrix.a-result.length)
+```
+
+See the full example here: [pr-with-matrix-context-variables]
+
 ## Results
 
 ### Specifying Results in a Matrix
@@ -360,8 +394,51 @@ tasks:
 
 ### Results from fanned out Matrixed PipelineTasks
 
-Emitting `Results` from fanned out `PipelineTasks`  is not currently supported.
-We plan to support emitting `Results` from fanned out `PipelineTasks` in the near future.
+Emitting `Results` from fanned out `PipelineTasks` is now supported. Each fanned out
+`TaskRun` that produces `Result` of type `string` will be aggregated into an `array`
+of `Results` during reconciliation, in which the whole `array` of `Results` can be consumed by another `pipelineTask` using the star notion [*].
+Note: A known limitation is not being able to consume a singular result or specific
+combinations of results produced by a previous fanned out `PipelineTask`.
+
+| Result Type in `taskRef` or `taskSpec` | Parameter Type of Consumer | Specification                                         |
+|----------------------------------------|----------------------------|-------------------------------------------------------|
+| string                                 | array                      | `$(tasks.<pipelineTaskName>.results.<resultName>[*])` |
+| array                                  | Not Supported              | Not Supported                                         |
+| object                                 | Not Supported              | Not Supported                                         |
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: platform-browser-tests
+spec:
+  tasks:
+    - name: matrix-emitting-results
+        matrix:
+          params:
+            - name: platform
+              value:
+                - linux
+                - mac
+                - windows
+            - name: browser
+              value:
+                - chrome
+                - safari
+                - firefox
+        taskRef:
+          name: taskwithresults
+          kind: Task
+      - name: task-consuming-results
+        taskRef:
+          name: echoarrayurl
+          kind: Task
+        params:
+          - name: url
+            value: $(tasks.matrix-emitting-results.results.report-url[*])
+  ...
+```
+See the full example [pr-with-matrix-emitting-results]
 
 
 ## Retries
@@ -851,4 +928,7 @@ status:
 [cel]: https://github.com/tektoncd/experimental/tree/1609827ea81d05c8d00f8933c5c9d6150cd36989/cel
 [pr-with-matrix]: ../examples/v1/pipelineruns/alpha/pipelinerun-with-matrix.yaml
 [pr-with-matrix-and-results]: ../examples/v1/pipelineruns/alpha/pipelinerun-with-matrix-and-results.yaml
+[pr-with-matrix-context-variables]: ../examples/v1/pipelineruns/alpha/pipelinerun-with-matrix-context-variables.yaml
+[pr-with-matrix-emitting-results]: ../examples/v1/pipelineruns/alpha/pipelinerun-with-matrix-emitting-results.yaml
+
 [retries]: pipelines.md#using-the-retries-field

docs/variables.md

@@ -28,10 +28,12 @@ For instructions on using variable substitutions see the relevant section of [th
 | `params["<param name>"][i]` | (see above) |
 | `params.<object-param-name>[*]` | Get the value of the whole object param. This is alpha feature, set `enable-api-fields` to `alpha`  to use it.|
 | `params.<object-param-name>.<individual-key-name>` | Get the value of an individual child of an object param. This is alpha feature, set `enable-api-fields` to `alpha`  to use it. |
+| `tasks.<taskName>.matrix.length` | The length of the `Matrix` combination count. |
 | `tasks.<taskName>.results.<resultName>` | The value of the `Task's` result. Can alter `Task` execution order within a `Pipeline`.) |
 | `tasks.<taskName>.results.<resultName>[i]` | The ith value of the `Task's` array result. Can alter `Task` execution order within a `Pipeline`.) |
 | `tasks.<taskName>.results.<resultName>[*]` | The array value of the `Task's` result. Can alter `Task` execution order within a `Pipeline`. Cannot be used in `script`.) |
 | `tasks.<taskName>.results.<resultName>.key` | The `key` value of the `Task's` object result. Can alter `Task` execution order within a `Pipeline`.) |
+| `tasks.<taskName>.matrix.<resultName>.length` | The length of the matrixed `Task's` results. (Can alter `Task` execution order within a `Pipeline`.) |
 | `workspaces.<workspaceName>.bound` | Whether a `Workspace` has been bound or not. "false" if the `Workspace` declaration has `optional: true` and the Workspace binding was omitted by the PipelineRun. |
 | `context.pipelineRun.name` | The name of the `PipelineRun` that this `Pipeline` is running in. |
 | `context.pipelineRun.namespace` | The namespace of the `PipelineRun` that this `Pipeline` is running in. |

examples/v1/pipelineruns/alpha/pipelinerun-with-matrix-context-variables.yaml

@@ -0,0 +1,123 @@
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: validate-matrix-result-length
+spec:
+  params:
+    - name: matrixlength
+      type: string
+  steps:
+    - name: validate
+      image: alpine
+      args: ["$(params.matrixlength)"]
+      script: |
+        #!/usr/bin/env sh
+        echo "Validating the length of the matrix context variable"
+        echo "The length of the matrix is 3"
+        if [ "$(params.matrixlength)" != 3 ]; then
+          echo "Error: expected matrix to have the length 3 but has length $(params.matrixlength)"
+          exit 1
+        fi
+        echo "Done validating the length of the matrix context variable"
+---
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: validate-matrix-results-length
+spec:
+  params:
+    - name: matrixresultslength-1
+      type: string
+    - name: matrixresultslength-2
+      type: string
+  steps:
+    - name: validate
+      image: alpine
+      script: |
+        #!/usr/bin/env sh
+        echo "Validating the length of the matrix results context variable"
+        echo "The length of the matrix results are $(params.matrixresultslength-1) and $(params.matrixresultslength-2)"
+        if [ "$(params.matrixresultslength-1)" != 3 ]; then
+          echo "Error: expected matrix results to have the length 3 but has length $(params.matrixresultslength-1)"
+          exit 1
+        fi
+        if [ "$(params.matrixresultslength-2)" != 1 ]; then
+          echo "Error: expected matrix results to have the length 1 but has length $(params.matrixresultslength-2)"
+          exit 1
+        fi
+        echo "Done validating the length of the matrix context variable"
+---
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: taskwithresults
+spec:
+  params:
+    - name: IMAGE
+    - name: DIGEST
+      default: ""
+  results:
+    - name: IMAGE-DIGEST
+    - name: IMAGE-URL
+  steps:
+    - name: produce-results
+      image: bash:latest
+      script: |
+        #!/usr/bin/env bash
+        echo "Building image for $(params.IMAGE)"
+        echo -n "$(params.DIGEST)" | sha256sum | tee $(results.IMAGE-DIGEST.path)
+        if [ -z $(params.DIGEST) ]; then
+          echo -n "$(params.DIGEST)" | sha256sum | tee $(results.IMAGE-URL.path)
+        fi
+---
+apiVersion: tekton.dev/v1
+kind: PipelineRun
+metadata:
+  generateName: matrix-context-variables-
+spec:
+  taskRunTemplate:
+    serviceAccountName: "default"
+  pipelineSpec:
+    tasks:
+      - name: matrix-emitting-results
+        matrix:
+          include:
+            - name: build-1
+              params:
+                - name: IMAGE
+                  value: image-1
+                - name: DIGEST
+                  value: path/to/Dockerfile1
+            - name: build-2
+              params:
+                - name: IMAGE
+                  value: image-2
+                - name: DIGEST
+                  value: path/to/Dockerfile2
+            - name: build-3
+              params:
+                - name: IMAGE
+                  value: image-3
+        taskRef:
+          name: taskwithresults
+          kind: Task
+      - name: matrixed-echo-length
+        runAfter:
+          - matrix-emitting-results
+        params:
+          - name: matrixlength
+            value: $(tasks.matrix-emitting-results.matrix.length)
+        taskRef:
+          name: validate-matrix-result-length
+          kind: Task
+      - name: matrixed-echo-results-length
+        runAfter:
+          - matrix-emitting-results
+        params:
+          - name: matrixresultslength-1
+            value: $(tasks.matrix-emitting-results.matrix.IMAGE-DIGEST.length)
+          - name: matrixresultslength-2
+            value: $(tasks.matrix-emitting-results.matrix.IMAGE-URL.length)
+        taskRef:
+          name: validate-matrix-results-length
+          kind: Task

examples/v1/pipelineruns/alpha/pipelinerun-with-matrix-emitting-results.yaml

@@ -0,0 +1,110 @@
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: echostringurl
+spec:
+  params:
+    - name: url
+      type: string
+  steps:
+    - name: echo
+      image: alpine
+      script: |
+        echo "$(params.url)"
+---
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: validate-array-url
+spec:
+  params:
+    - name: url
+      type: array
+  steps:
+    - name: validate
+      image: ubuntu
+      args: ["$(params.url[*])"]
+      script: |
+        #!/usr/bin/env bash
+        args=("$@")
+        URLS=( )
+        URLS[0]="https://api.example/get-report/linux-chrome"
+        URLS[1]="https://api.example/get-report/mac-chrome"
+        URLS[2]="https://api.example/get-report/windows-chrome"
+        URLS[3]="https://api.example/get-report/linux-safari"
+        URLS[4]="https://api.example/get-report/mac-safari"
+        URLS[5]="https://api.example/get-report/windows-safari"
+        URLS[6]="https://api.example/get-report/linux-firefox"
+        URLS[7]="https://api.example/get-report/mac-firefox"
+        URLS[8]="https://api.example/get-report/windows-firefox"
+        for i in "${!URLS[@]}"; do
+            if [ "${URLS[$i]}" != ${args[$i]} ]; then
+              echo "Error: expected url to be ${URLS[$i]}, but got ${args[$i]}"
+              exit 1
+            fi
+            echo "Done validating the url: ${args[$i]}"
+        done
+---
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: task-producing-results
+spec:
+  params:
+    - name: platform
+      default: ""
+    - name: browser
+      default: ""
+  results:
+    - name: report-url
+      type: string
+  steps:
+    - name: produce-report-url
+      image: alpine
+      script: |
+        echo "Running tests on $(params.platform)-$(params.browser)"
+        echo -n "https://api.example/get-report/$(params.platform)-$(params.browser)" | tee $(results.report-url.path)
+---
+apiVersion: tekton.dev/v1
+kind: PipelineRun
+metadata:
+  generateName: platforms-with-results
+spec:
+  taskRunTemplate:
+    serviceAccountName: "default"
+  pipelineSpec:
+    results:
+      - name: pr-result-1
+        value: $(tasks.matrix-emitting-results.results.report-url[*])
+    tasks:
+      - name: matrix-emitting-results
+        matrix:
+          params:
+            - name: platform
+              value:
+                - linux
+                - mac
+                - windows
+            - name: browser
+              value:
+                - chrome
+                - safari
+                - firefox
+        taskRef:
+          name: task-producing-results
+          kind: Task
+      - name: task-consuming-results
+        taskRef:
+          name: validate-array-url
+          kind: Task
+        params:
+          - name: url
+            value: $(tasks.matrix-emitting-results.results.report-url[*])
+      - name: matrix-consuming-results
+        taskRef:
+          name: echostringurl
+          kind: Task
+        matrix:
+          params:
+            - name: url
+              value: $(tasks.matrix-emitting-results.results.report-url[*])

pkg/apis/pipeline/v1/param_types.go

@@ -208,6 +208,29 @@ func (ps Params) extractParamMapArrVals() map[string][]string {
 	return paramsMap
 }
 
+// ParseTaskandResultName parses "task name", "result name" from a Matrix Context Variable
+// Valid Example 1:
+// - Input: tasks.myTask.matrix.length
+// - Output: "myTask", ""
+// Valid Example 2:
+// - Input: tasks.myTask.matrix.ResultName.length
+// - Output: "myTask", "ResultName"
+func (p Param) ParseTaskandResultName() (string, string) {
+	if expressions, ok := p.GetVarSubstitutionExpressions(); ok {
+		for _, expression := range expressions {
+			subExpressions := strings.Split(expression, ".")
+			pipelineTaskName := subExpressions[1]
+			if len(subExpressions) == 4 {
+				return pipelineTaskName, ""
+			} else if len(subExpressions) == 5 {
+				resultName := subExpressions[3]
+				return pipelineTaskName, resultName
+			}
+		}
+	}
+	return "", ""
+}
+
 // Params is a list of Param
 type Params []Param