how to configure jetstream replicas to share a pvc ?

[databasedav]

the current behavior creates a new pvc per replica, each of which automatically acquires a separate cloud disk, but i want to avoid renting multiple cloud disks, is there any way to point all jetstream replicas to a single pvc ?

[caleblloyd]

I think you'd want to look for a PV Provisioner that does that for you, if that is your goal. Like a Host Path PV Provisioner.

[databasedav]

I see, just to be clear, I'm fine with the provisioning of the cloud disk, and can't use host paths because my hosts have minimal disk + they might be ephemeral

that does that for you

I understand that a special provisioner would accomplish the multiple pvc -> single backing disk mapping, but how is that communicated to the nats configuration? I was imagining needing to set something in nats.jetstream.fileStorage.existingClaim?

[caleblloyd]

You can't, to my knowledge, configure multiple Stateful Set replicas to provision and use a single PVC using volumeClaimTemplates

Might be possible to mount an existing one to multiple replicas, but I wouldn't really recommend that.

[databasedav]

is the issue that the data directory is common between all replicas? can we point each replica to a different data directory? the current storage configuration is common between all replicas, but i can modify the template to take separate config per replica, but would this sort of thing be accepted by the nats server?

[caleblloyd]

JetStream needs fast block based storage. Should not use NFS or other slow file based storage with it. Most fast block based storage in the cloud only works with a single host as a writer

That is why this chart is designed for each replica to get it's own PVC. In the cloud ideally 3 replicas will run in 3 different availability zones, and each replica has exclusive access to a fast block based disk in its availability zone.