How to configure mutually exclusive pipelines in workflow?

[niconjjanssens]

Hi,

I'm trying to configure a workflow, containing 2 pipelines that are mutually exclusive.
A simplified example to illustrate:

name: TestMutualExclusive
version: v2.0
workflow:
  TstPip1:
    pipeline: TstPip1
  TstPip2_1:
    depends_on:
    - fork
    conditions:
      check:
      - variable: workflow.TstPip1.build.result
        operator: ge
        value: "1"
    pipeline: TestPip
  TstPip2_2:
    depends_on:
    - fork
    conditions:
      check:
      - variable: workflow.TstPip1.build.result
        operator: ge
        value: "2"
    pipeline: TestPip
  TstPip3:
    depends_on:
    - TstPip2_1
    - TstPip2_2
    pipeline: TestPip

When the pipeline (TstPip1) is executed the values (exported as variables using "worker export ...") are derived which drive the conditions for the middle pipelines (TstPip2_1 and TstPip2_2). The issue is that TstPip3 is only executed, when both TstPip2_1 and TstPip2_2 are executed. But as soon as one of the middle pipelines is not executed (nature of mutually exclusivity), TstPip3 is not executed, not even attempted.

Any suggestions on how to configure such workflow?

Kr,
Nico

[yesnault]

Hi. To be executed, all parent pipelines of a pipeline have to be executed. In your example, if TstPip2_1 or TstPip2_2 is not executed, the pipeline TstPip3 can't be executed.
There is probably another way to do what your want, as :

• having TstPip3_1 and TstPip3_2 (with TstPip3_x using the same pipeline)
• using the pipeline parameter. This feature depends on your use case, but if you don't know them, perhaps you can resolve your use case with them. A pipeline parameter is like an argument of a function, you can set a value of this parameter in the context of the pipeline. And with a pipeline parameter, you can solve the parent execution by using the condition on stage: you will be able to disable a whole stage. With that, your pipeline will be executed even if no job is executed inside

[niconjjanssens]

Hi Yvonnick,

Thanks for sharing this, which confirms that I should not further try to make it work with my current approach. For the time being I would go for the second option.

Reflecting it to my use case., for the time being, I'll go for the second suggestion and put a condition on the Stage. But I encounter a strange behavior.
I tried this by declaring 2 pipeline parameters (say runtime_operation and expected_operation), and in the workflow I define a static value for the "expected_operation" (say "Deploy" for one, and "Upgrade" for the other). The runtime_operation is then filled in with e.g. a {{.workflow.TstPip1.build.operation}}. For the condition to the Stage I defined an advanced (Lua) script like: "return cds_pip_expected_operation == cds_pip_build_runtime_operation". However, the observation was that TstPip3 got executed, but none of these conditional jobs (in TstPip2_1 and TstPip2_2) were executed. When I use only the runtime_operation parameter in the condition (to compare it to the strings "Deploy" or "Upgrade"), then it behaved like expect it for the mutual exlclusivity. So I have a solution to go forward.
Can you clarify why the Lua comparison did not yield the expected result? (Note: when I just put "return true" the Job got executed, so the Lua script got called.)

Actually I thought about the two alternative approaches as generic patterns. But I didn't try them, as both have have their drawbacks when using them into more complex workflows.

The first suggestion basically requires to transform (unfold) from a graph based workflow (withn joins) into a tree-based workflow, and thereby multiplies the pipelines occurences. For N possible (but mutally exclusive) paths (say in my example: TstPip2_1 ... TstPip2_N), this leads to have to declare the subsequent pipeline Nx times. And when having a cascading of such mutually paths, the amount of leaf paths in the tree would become N x M x K ....
Depending on the specific use case, this may be acceptable, or not maintainable.

The second suggestion has the drawback that the pipeline has to be specifically adapted to consider the inclusion cases in workflow, because "context" information bleeds into the pipeline design itself (declaration of pipeline inputs/parameters and conditions on the jobs).
An improvement would be to have some "skip" flag specified at the workflow level for the pipeline context, such that all jobs in the pipeline would be skipped, but yet the pipeline would be considered "executed", so to fulfill the requirment for a potential subsequent join.
Is this something that may be considered in a future release?

Kr,
Nico

[yesnault]

there is another option, a step script using worker exit

worker exit -h
worker exit command lets job finish current step and disabled all further steps

We often discuss about the skip a pipeline, this is not a feature that will happen quickly. This will possible after a refinement of the workflow model that we have to plan after the current work on artifact storage.

About the Run Condition:

• if true, the element is executed (stage / pipeline)
• if false, it's not executed

With a Lua Condition, if the condition return true, it's executed, otherwise, it's not executed.
If you have a yml configuration of a test workflow (can be exported with cdsctl workflow pull command), we can take a look at it to figure out what is wrong.