[Question] Inner job hook feature

[killzoner]

Hi there,

I am investigating using CDS to handle workflows with tasks executed outside of workers.
Imagine I have this schema for a workflow

  - task 1: trigger job using webhook from API A
  - task 2: trigger job using webhook from API B

  - task 3: output ok if task1 AND task2 reported OK

I would need some inner workflow hook available to report from the outside system that the job is OK or KO.
Is there such a capability in the REST API ? Couldn't determine it (saw workflow hooks to trigger one but nothing similar to what i'm searching).

If there is another way to do that using CDS, please feel free to suggest

Thanks in advance

[yesnault]

Hi @killzoner

A CDS workflow can be started by a webhook. You can't add a webhook in a middle of a workflow.

Your use case will not be easy to implement with cds as is, but here is an idea:

• a workflow A triggered by API A, then trigger workflow C
• a workflow B triggered by API B, then trigger workflow C
• a workflow C, checking if workflow A and B is ok in their last run
• you probably need to use cdsctl in your job to get status of workflow A and B from workflow C.

[killzoner]

Hey @yesnault thanks for the quick feedback!

Not sure we are on the same line, how your solution would help me report the job is ok from the external system ?

Maybe i'm mixing CDS terminology, but what i'm referring is 1 workflow is composed of multiple jobs (step 1, step 2, etc.)
These steps are A, B and C.

These jobs would still be created by CDS but have equivalent job in another system:

Job A triggers some batch job on machine M1
Job B triggers some batch job on machine M2

The thing i'm wanting to implement is how machine M1 reports to CDS workflow with job A that it's done and can go to step B, basically steps of the workflow are async and depend on an external system.

In some systems, it's implemented as a callback URL to tell job is done

[yesnault]

What is the maximum duration of the "some batch" launched on the machine M1 / M2 in your example ? (< 1h, > 1h, > 24h) ?

[killzoner]

Average job is most likely in the hour timeframe, but since external batch system could be subject to queued events, this could lead to some days if there is heavy load on the machine / cluster executing the task

[yesnault]

ok, so, two solutions:

• keep the cds job wait for then end of the batch
• or from the host M1 / M2, you can use cdsctl workflow run --node-name XXX --run-number XXX to start a pipeline from an existing run of a workflow.

In both solutions, you have only one workflow, example:

name: test_batchs
version: v2.0
workflow:
  end:
    depends_on:
    - run-batch_on_m1
    - run-batch_on_m2
    when:
    - success
    pipeline: end
  root:
    pipeline: root
  run-batch_on_m1:
    depends_on:
    - root
    when:
    - success
    pipeline: run-batch
  run-batch_on_m2:
    depends_on:
    - root
    when:
    - success
    pipeline: run-batch

[killzoner]

Thanks a bunch @yesnault !
Will probably do some POC using this approach in the coming days and see if I encounter any problem along the way.