What kind of situations can cause duplicate messages in a kafka consumer?

[JSoet]

We had a situation recently where our application received a number of duplicate messages and we're trying to understand how this could've happened. Unfortunately the kafka broker logs were already overwritten by the time we identified the duplicates, so it's likely we'll never be able to understand exactly what happened, but we're trying to understand the different failure points in case it happens again.

I wrote up a couple more specific questions about some questions I had:
#4096
#4095

But I thought I would write up a more general question as well, in case anyone had any more general feedback

Our application is using the high level c++ consumer to consume, and we're using enable.auto.commit=true and enable.auto.offset.store=true. I see the following possibilities of potential places where messages could have been duplicated:

1. The application was disconnected from the broker after it had consumed a number of messages and before the offsets had been committed
   • This is the most likely scenario for duplicates to occur, but it seems that this didn't happen, because we would've seen a message on the consumer side about it being disconnected or a rebalance happening... The one thing here is that we didn't have all logging enabled, we were only logging warnings and errors on the consumer side, so maybe some logging indicating that this happened could've been missed here?
2. When one broker went down it caused the application to rebalance to another broker and some messages were duplicated
   • I'm unclear about the logic when a rebalance happens if uncommitted offsets could then be retried - I've asked in this discussion for a bit more details: What will happen if librdkafka can't commit the offsets to the broker? #4096
3. Our application processed the same message twice after it was consumed from librdkafka
   • Fairly unlikely, we haven't seen any other evidence of this happening
4. A bug in librdkafka - maybe it retrieved the messages from the broker twice, or gave our application the same message twice when consume was called on different threads etc...
   • highly unlikely, I assume there are no known bugs with librdkafka retrieving the same message twice or giving the same message twice?

Anyway, just thought I'd ask a more general question here to see if I'd get any general answer. If anyone has any feedback I'd appreciate it... even if it's just about what log messages I could expect in some of the above situations....