DagsterExecutionInterruptedError in runs on Kubernetes

[johannkm]

My dagster-k8s runs with the K8sRunLauncher are failing with DagsterExecutionInterruptedError. How can I find the source of the error?

[johannkm]

DagsterExecutionInterruptedError means the process received a SIGINT or SIGTERM, most commonly this means the K8s pod the process was running on was terminated by the Kubernetes cluster. You can use kubectl (or other Kubernetes tools) to investigate.

In the Dagster event log you should see an event with the string

Creating Kubernetes run worker job

followed by a Kubernetes job name and namespace. With those, you can use kubectl to look at the underlying job.

kubectl describe job -n <namespace> <job name>

For example, you might see:

• OOMKilled: the container ran out of memory. You can raise the memory request for the container, or refactor your Job to use less memory

[gibsondan]

kubectl get events -n <your namespace> | grep <your run ID> will also show cluster events for pods related to your job. If the node was scaled down you might see events like:

15m         Normal    ScaleDown              pod/dagster-run-your-run-id-here   deleting pod for node scale down
15m         Normal    Killing                pod/dagster-run-your-run-id-here   Stopping container dagster

[gibsondan]

This can also happen when the kubernetes cluster decides to evict the pod where your run is happening, for example to move it to a new node. Depending on the way your cluster is set up, you may be able to signal via annotations that your pod should not be evicted in this way.

For example, if the cluster autoscaler is evicting your pod, you can apply the following annotation to your pods to prevent them from being evicted (see https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node for more information):

"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

See https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#instance-level-kubernetes-configuration for more information on how to apply annotations to the Dagster run pods. In your Helm chart, config that does this might look something like this:

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      runK8sConfig:
        podTemplateSpecMetadata:
          annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

[domsj-foodpairing]

small note for future readers:

...
        podTemplateSpecMetadata: # raw config for the pod's metadata
          annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: false

you need to add an extra annotations: into the yaml snippet from the post above

[gibsondan]

fixed, thank you!

[atedesch1]

another note, the annotation value false was being converted to False on my application of the helm chart so this fixed it:

...
        podTemplateSpecMetadata: # raw config for the pod's metadata
          annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

[gibsondan]

Fixed the example (again), thank you!

[lafonj]

Does the Agent also trigger this when a code update is detected? We've noticed that many of these happen when we do code updates.

[gibsondan]

The agent doesn't interrupt runs when pods update - it does spin up a new pod on code updates, so it's possible that that is causing some side effect where the cluster wants to start moving other pods around in response to that change.