[nomad] Constraint did not meet topology requirement

[GiyoMoon]

TL;DR

I followed the nomad guide and when trying to add a job that makes use of the volumes I get the error "Constraint did not meet topology requirement filtered 1 node".

Expected behavior

The job should be able to use the volume. Maybe it's a bug with the csi driver or the nomad guide is missing something.

Observed behavior

Any jobs that want to make use of a volume can't be created due to constraints not meeting topology requirements.

Minimal working example

Follow the nomad guide 1:1 with the latest csi driver version (v2.10.0) and you should encounter this error.

Log output

+ Job: "volume-name"
+ Task Group: "group-name" (1 create)
  + Task: "task-name" (forces create)

Scheduler dry-run:
- WARNING: Failed to place all allocations.
  Task Group "group-name" (failed to place 1 allocation):
    * Constraint "did not meet topology requirement": 1 nodes excluded by filter

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 test.hcl

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.

Additional information

nomad - 1.9.1
hcloud-csi-driver - v2.10.0

[GiyoMoon]

When printing the volume information, I get:

ID                   = db-vol
Name                 = db-vol
Namespace            = default
External ID          = REDACTED
Plugin ID            = csi.hetzner.cloud
Provider             = csi.hetzner.cloud
Version              = 2.10.0
Capacity             = 10 GiB
Schedulable          = true
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1
Access Mode          = <none>
Attachment Mode      = <none>
Mount Options        = fs_type: ext4 flags: [REDACTED]
Namespace            = default

Topologies
Topology  Segments
01        csi.hetzner.cloud/location=nbg1

Allocations
No allocations placed

Maybe my node needs be assigned to the nbg1 location somehow to meet the constraint? If yes I don't know how to add this, any help would be appreciated 😇 Thanks!

[resmo]

I am the author of the doc and can confirm this issue after updating to csi 2.10.0. A fallback to 2.9.0 resolved the issue for me.

@GiyoMoon could you try using 2.9.0?

[lopcode]

Bumped in to the same issue today, confirmed downgrading to 2.9.0 worked 🤞

[resmo]

I briefly run over the commits and the changes in this PR look suspicious to me: #743

Pinging @apricote to let him know.

Update:
I tried out some things, like adding a custom meta tag, without success...

client {
  enabled = true
  node_class = "client"

  meta {
    "instance.hetzner.cloud/provided-by" = "cloud"
  }

...

[GiyoMoon]

@resmo Hey, thanks for looking into this! I can confirm that 2.9.0 works for me too.

[lukasmetzner]

Hi,

we are in the process of building a Nomad development setup to run csi-driver end-to-end tests on Nomad. During this setup, we encountered this bug. We are currently investigating the issue.

Best Regards
Lukas

[lukasmetzner]

Hi,

we have just released v2.10.1 to revert the breaking changes.

We apologize for any inconvenience this may have caused.

Best regards,
Lukas

[GiyoMoon]

Hey Lukas, thanks! Can confirm that 2.10.1 works :)