Pausing Work

[linux2647]

Sometimes a set work is queued that needs to be paused and prevented from being processed. For example, a job may call an API, but that API is down, causing errors and retries. The jobs could simply use the retry or snooze mechanisms, but it would be operationally simpler if an operator could pause the work until the issue is resolved.

In this proposal, work could be paused by queue and/or by kind. Given that river's job model provides both of these attributes for categorizing work, there may be different situations that might require pausing by one or the other or both.

This state would need to be propagated to all workers. Ideally, this state could be maintained even if all workers go away, and new workers that are spun up would still hold paused work. Oban's pause_queue/2 uses some BEAM mechanism to propagate this information, but Go probably doesn't have this OOTB.

Potential solutions follow

One idea is a separate two-column table could track which queues or job kinds that are paused. The first column would indicate the attribute (queue, kind) and the second column would be that attribute's value (default, sort, etc). When querying for work to run, this table could be joined or otherwise filtered against, removing rows that match one of these values. To pause a queue, e.g. the default queue, a row of ('queue', 'default') could be inserted. To resume the queue, the row would be removed. The same idea could be applied to a job kind.

Another idea is to modify the job records themselves, via a column or modifying some other attribute. However, I would caution against this idea. If there are hundreds of thousands of jobs waiting to be resumed, modifying all of those rows would take significant database resources. It could be done, but it would require batching the work appropriately. In addition, a mechanism would be needed to determine how to pause work that is being inserted. It also has the potential for becoming out-of-sync, as some rows may be paused while others aren't.

As an alternative approach, if we discard the requirement of needing to maintain the paused state if all of the workers go away, then this state could be retained in-memory by each worker. The leader could propagate this information to followers via the existing LISTEN/NOTIFY mechanisms used elsewhere in river.

[brandur]

Hi @linux2647, sorry this one fell through the cracks — we try to answer every one question on here, but this one must've come through before I was watching the list, or I archived it by accident.

Short answer on this: pausing is a great idea, and it's coming. @bgentry spent a lot of time thinking about it, and I believe his solution is very close to the first one your proposed.

Thanks for broaching this subject, and the thoughtful list of ideas.

[bgentry]

This was resolved by #301, sorry I forgot to post back until now!