Have one consumer for retry topics

[psh05]

Currently there is one consumer per topic. If an app consumes from multiple topics which will implement multiple retries and there are multiple instances of this application deployed, then the number of consumers created will be quite large.
e.g. If we have 3 topics, each with 3 retries and 1 DLT. Then we will have 3*4= 12 consumers and if we have 50 instances then thats 12 * 50 = 600 consumers.

@tomazfernandes - Is this possible? or is there a way round this?

[garyrussell]

I received this out-of-band and I suggested starting a discussion. The suggestion is to add an option to consume the main and retry topics from a single container.

[tomazfernandes]

That's interesting, and looks not that hard to implement. Might require some refactoring in the code though, mainly in the RetryTopicConfigurer class.

We do make a few changes in the container properties for the retry topics that might not be desirable for the main one, such as adjusting the pollTimeout and partitionIdle values to make delays more precise - maybe we should disable that in this mode. And using the same container for all topics might interfere with the precision of the delays in general since there are more records to be processed. But that's probably only a matter of properly documenting the behavior.

In general, looks like it'd be a better use of the resources, since currently the backed off containers only keep polling the broker for the heartbeat.

If I understand correctly, we'd just need to set all topics to the same KafkaListenerEndpoint, and the registrar would work from there.

We'd probably also have to look at how this would interfere with other features, such as the RetryTopicNamesProvider which has access to the endpoint and the topic properties that eventually will make each endpoint different. Again, probably a matter of documenting it properly, or making a few adjustments.

Looks good to me, we can discuss it further here, or maybe open an issue already.

Thanks!

[psh05]

Thank you both for the quick reply!

Just thinking of options and flexibility - Would be better to have 2 consumers, 1 for the main topic and 2 for all the retries and DLT? So we don't affect the behaviour or block the main topic (although this might not be an issue)
Also assuming a single consumer, guessing it poll all topics and only process the messages from the retry topics when they are due?

Happy to open an issue!

Thanks!

[tomazfernandes]

To add an example, let's say we have a record that triggers a call that results in a huge payload being sent from the server, which also takes a while to be processed there. Then the processing fails because, well, the payload is too large, or maybe it times out. I've unfortunately seen that happen.

If it's the same container, every time we try to process that record we will wait for the timeout, only to fail again, which would multiply the impact of that bad record in the overall processing.

[garyrussell]

@tomazfernandes thanks for taking the time to review this request.

Yes, it does mean that it might impact the processing rate of the main topic, so I like the idea of the three options, with the existing mechanism being the default.

We can convert this discussion to an issue rather than opening a new one.

I would not be comfortable with back porting such a large change to 2.8.x, and 3.0.x will not be released until late this year (Spring Framework 6.0); we could consider an interim 2.9.x to go with Boot 2.7.x in May.

[tomazfernandes]

Sounds good to me, thanks @garyrussell.

[garyrussell]

#2150

[psh05]

Thanks @garyrussell and @tomazfernandes. Thought I replied - I also like the idea of making it configurable.