User-defined Maintenance Services

[harryjph]

Hi Guys! Thank you for the awesome project.

The River Docs describe how River runs various maintenance services in the background on the leader node. In our application, we want to perform various housekeeping operations as a background loop/daemon, but we want to ensure that only one node runs these services at a time.

One way to do this would be to use Postgres advisory locks, but since River is managing leadership elections and running background tasks on just the leader anyway, it would be great to hook into this to run our own tasks like this too. Is this possible?

[bgentry]

We’ve definitely kicked this idea around and it wouldn’t be technically difficult at all. I think our hesitation is that it’d be an advanced feature with the potential for a misbehaving or badly written service to cause issues within the River client. Also a risk of concentrating too much extra work on the leader node.

In the end that may not be enough to justify holding it back, because as you say there are lots of use cases that would benefit from the solid leader election system that we already have in River.

Thoughts @brandur ?

[harryjph]

Thanks for your response!

Have you considered providing a hook where the user provides a func(ctx context.Context, electedAt time.Time) error or similar? This hook could be invoked in a new goroutine when the leader is elected and the context would be cancelled when the leader resigns. This would present very little possibilities for disrupting the river client. Alternatively, the startstop.Service interface looks like a good existing option which a user could implement and provide as an auxiliary service.

[brandur]

No major objections here.

The one thing on the hook function versus letting people add services supporting start/stop is that there'd also have to be an equivalent "unelected" hook so that consumers wouldn't double start their maintenance service if a new node is elected and the last one steps down.

Given that, it might be safer just to force people to implement the start/stop interface instead because that makes it crystal clear that the equivalent stop function is mandatory.