[FEATURE REQUEST] Support setting retry delay when inserting a new job

[rhodeon]

I'm integrating River as part of a bigger task system, and it's been great so far. But there's a particular missing feature that would be nice to have.
Currently, we are able to specify the maximum attempts of retries at both the worker and job level (https://riverqueue.com/docs/job-retries#limiting-retry-attempts):

_, err = riverClient.Insert(ctx, RetryOnceJobArgs{}, &river.InsertOpts{
    // use a max attempts of 5 for this one job even though the job has a
    // default limit of 1:
    MaxAttempts: 5,
})

On the other hand, the job retry delays can be set at the client and worker level (https://riverqueue.com/docs/job-retries#retry-delays), but not at the job level.
It would be great to be able to set the retry delay at job insertion, just like how the maximum attempt is.
A use-case for us is that users are able to create "rules" that control the dispatching of jobs to workers, and these rules can have their max attempts/delays customised.
Is this feature something that can be considered, or even feasible with the current River architecture?

[bgentry]

Hi @rhodeon, you can customize the retry policy either on a per worker type basis or globally. You can use whatever logic you want for that, including basing it on job args that you can control at insertion time.

Let me know if you have a specific use case you think isn’t handled by this!

[rhodeon]

Hey, @bgentry. Thanks for the response!

including basing it on job args that you can control at insertion time.

Great, this works fine. The only issue is that it "pollutes" the arguments of the job by adding the delay there, but that isn't a big deal.

One question, though. Was there a particular reason for not exposing the retry policy under river.InsertOpts, like the maximum attempts is? As they're somewhat related.

[bgentry]

The idea behind the retry policy is to provide more flexibility on the logic for retry delays. Things like exponential vs linear backoff with granular control cannot really be encoded into a database field with the kind of flexibility we were hoping to allow.

Max attempts were actually part of the retry policy originally, but were moved to the database to facilitate other features and because they could be. Although I think this may have originally been done to facilitate incrementing max attempts on snooze, which to be honest I wish we had gone a different direction on (decrementing the current attempt number instead).

A lot of users will drop an embedded or nested struct into their job args & workers types for behaviors they want to share across many parts of their system, maybe that would help keep it manageable and be sufficient for what you need here?

[rhodeon]

Ah, I see. That clears things up. Thanks.

A lot of users will drop an embedded or nested struct into their job args & workers types for behaviors they want to share across many parts of their system, maybe that would help keep it manageable and be sufficient for what you need here?

Yeah, I incidentally decided to do this also. The behaviour needed is fully possible with it.