Phased backoff for non-blocking retries

[jgslima]

Hello, I want to share some ideas and implementations I made here for the non-blocking retries. This is not a feature request, is just idea sharing.

When making retries, I think it is essential to pursue two things:

• to make the first retries quickly (with a small interval), so that for really ephemeral issues, the system recovers quickly and the user experience is less affected.
• to keep making the retries for a long time (about 4h for instance). Because some issues may take more time to go away. For example, an external system may be down, and the operations team may struggle for a while before being able to take it up again.

At first, exponential backoff seems to fit this requirement. However, to achieve the second requirement above, a large number of retry topics are needed. In a way that makes this impractical.

Fixed backoff allows a single retry topic to be used, but adopting fixed backoff brings the problem of inefficiency: in order to have quick retries and insist for a long time, one has to configure an excessive amount of attempts (with an interval of 5s, to keep trying for 4 hours, you need to configure 2880 maxAttempts, which is too much, besides too much CO2).

So I conceived here a custom SleepingBackOffPolicy called PhasedSleepingBackoffPolicy. It's objective is to have an increase in the intervals like exponential, but without an explosion of topics.
You configure for it, a list of phases, with which one being composed of:

• a Duration for the interval between retries.
• a Duration for the whole phase.

For example:

    phases:
    - duration: PT3M
      interval-between-retries: PT5S
    - duration: PT30M
      interval-between-retries: PT45S
    - duration: PT6H
      interval-between-retries: PT2M

For 3 minutes, make retries each 5 seconds, for 30 minutes, make retries each 45 seconds, for 6 hours, make retries every 2 minutes.

The amount of retry topics needed is the amount of phases.

For this to work, only the backoff policy was not enough. I had to make an extension of DefaultDestinationTopicResolver, which delegates part of the decision of the next retry topic to the backoff policy. In the case of phased retries, the current attempt is taken into consideration when deciding in each phase (and consequently for which retry topic) the record currently is. So in terms of component design, there may be a more decoupled approach but anyway it is working fine.

[jgslima]

Another possibility would be for the spring-kafka to have an option to have a single retry topic, for the maxInterval when using exponential backoff.

For instance, for exponential with initialInterval=1000, multiplier=2, maxInterval=8000, and maxAttempts=a large number, you would have retry topics like these:

xxx-retry-1000
xxx-retry-2000
xxx-retry-4000
xxx-retry-8000-0
xxx-retry-8000-1
xxx-retry-8000-2
xxx-retry-8000-3
...
xxx-retry-8000-n

but for 8000-0 and above, it might actually be just one:

xxx-retry-1000
xxx-retry-2000
xxx-retry-4000
xxx-retry-8000

this would make it easier to adopt exponential when retrying for a long time.

[artembilan]

Do I understand it correctly that at the moment this is possible via RetryTemplate and RecoveryCallback chain, where that RecoveryCallback would call the same service method via another RetryTemplate with its own RetryPolicy and BackOffPolicy, and so on?

[garyrussell]

RetryTemplate is blocking retries; this is all about non-blocking retries.

[tomazfernandes]

for instance, I made my DefaultDestinationTopicResolver extension to be aware of, and to interact with, the backoff policy of the topic

Just to get everyone to the same page as to how it currently works, the class that knows about back off values is the DestinationTopicPropertiesFactory, then the created DestinationTopic.Properties instances will be forwarded to the DestinationTopicProcessor, which will then populate the map for DestinationTopicResolver, which has the runtime logic for resolving the topics.

So maybe we can add some property to DestinationTopic.Properties to be looked up by the DTResolver so it can know whether to keep sending to the same topic or go to the next one, or to the DLT.

The current mechanism ignores the downtime (the record goes to the next topic);

Not sure it helps, but we do have a global timeout setting after which the message will go directly to the DLT - in case of an outage for example the record would timeout if so configured.

BackOffContext created when calling start(RetryContext context)

I'm not sure I understand how this is being used - the only time we use the Spring Retry interfaces is to figure out the backoff values early in the configuration phase.

I think the simplest thing to begin with would be to make it so that, depending on how the DestinationTopic.Properties is configured, if the current topic is the last one before the DLT, we keep sending the record to it again until a certain maxAttempts is reached.

We do have a maxAttempts property in DestinationTopic.Properties that currently holds the same value for all topics in a chain - maybe that's a good place to start

[jgslima]

It's tricky; I can see that users might want to choose both behaviors (max attempts Vs. time based).

I agree. I think if we proceed at some point with this "phased" backoff policy, we should go with maxAttempts.

As you guys invited me to make some PR, initially I will create an issue to deal only with the idea of reusing a retry topic for the maxInterval delay of exponential backoffs.

[jgslima]

Created #2496