Logs of k8s job.batch are not shown with garden logs

[yl-raisin]

Bug

Start watching logs of k8s Job defined as Deploy action:

> garden logs -f database-migration -l 4
....
Service logs (from the last '1m' for each service):
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
ℹ [monitors]                → Starting log monitor for Deploy type=helm name=database-migration...
[debug] <No running containers found for ServiceAccount database-migration. Will retry in 10s...>
[debug] <No running containers found for ServiceAccount database-migration. Will retry in 10s...>
[debug] <No running containers found for ServiceAccount database-migration. Will retry in 10s...>
[debug] <No running containers found for ServiceAccount database-migration. Will retry in 10s...>
[debug] <No running containers found for ServiceAccount database-migration. Will retry in 10s...>

Re-deploy job:

> garden deploy database-migration --force
...
Done!

What is happening in the meantime:

> kubectl get pods -w
NAME                       READY   STATUS    RESTARTS   AGE
database-0                 1/1     Running   0          21h
database-migration-49k5n   0/1     Pending   0          0s
database-migration-49k5n   0/1     Pending   0          0s
database-migration-49k5n   0/1     ContainerCreating   0          0s
database-migration-49k5n   1/1     Running             0          0s
database-migration-49k5n   0/1     Completed           0          5s
database-migration-49k5n   0/1     Completed           0          7s
database-migration-49k5n   0/1     Completed           0          8s

Current Behavior

No logs are shown for the job and every 10s it complains with:
[debug] <No running containers found for ServiceAccount database-migration. Will retry in 10s...>

Expected behavior

Logs of the job are shown with garden logs command.

Reproducible example

Deploy action for a helm chart with Job only.

Workaround

Use kubectl logs -f $JOB_POD

Suggested solution(s)

To support Jobs by garden logs because according to docs https://docs.garden.io/using-garden/using-the-cli#logs it shows logs of any Deploy

Additional context

Your environment

• OS:
  Fedora release 39 (Thirty Nine)
• How I'm running Kubernetes:
  EKS

> garden version
garden version: 0.13.27

[stefreak]

@yl-raisin Can you share a Garden configuration yaml with us so we have something to try to reproduce this behaviour?

One workaround I can suggest already is using a Run action, instead of a Deploy action, e.g. https://docs.garden.io/reference/action-types/run/kubernetes-pod

Would that work for you?

[yl-raisin]

hey @stefreak ,

Here you go, I created a simple project that perfectly reproduces this issue: https://github.com/yl-raisin/hello-world-helm-chart/tree/main/

About Run action I would need to investigate how to configure it, thanks for advice.

[yl-raisin]

Also Run action does seem to work with Job as well. Here is a branch of created above simple project where I try to use Run action of type helm-pod: https://github.com/yl-raisin/hello-world-helm-chart/tree/use-helm-run/

It gives me this error:

Error validating spec for Run type=helm-pod name=hello-world:

spec.resource.kind must be one of [Deployment, DaemonSet, StatefulSet]
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[debug] Done flushing all events and log entries.
Failed to resolve Run type=helm-pod name=hello-world: Error: resolve Run type=helm-pod name=hello-world failed: Error: Error validating spec for Run type=helm-pod name=hello-world:

spec.resource.kind must be one of [Deployment, DaemonSet, StatefulSet]