Setting up Dagster with GCP Dataflow

[dagsir[bot]]

Hi team,

We are currently investigating the feasibility of using Dagster as our data platform orchestration tool.

Our pipelines are running with <https://cloud.google.com/dataflow|GCP Dataflow> and I am trying to find out how this would work.
I can't find any resource about it and want to ask here if someone has experience with that and could provide some pointers?

---

The question was originally asked in Dagster Slack.

[jamiedemaria]

I'm not very familiar with Dataflow, but one option could be to use the pipes library to kick of jobs in Dataflow from dagster. You can find more information about it here https://docs.dagster.io/guides/dagster-pipes