AWS MSK Upgrade problem,consumer recover take 5 mins, And ask for your help.

[lvhuat]

I create a topic like bellow:

Topic: orders  PartitionCount: 1       ReplicationFactor: 3    Configs: min.insync.replicas=2,retention.ms=15552000000,message.format.version=2.2-IV1,unclean.leader.election.enable=false
        Topic: orders   Partition: 0    Leader: 3       Replicas: `3,2,1` Isr: 2,1,3

and consume partition 0 continuously,code like like bellow:

  int init() {
    std::string errstr;
    if (!_conf) {
      _conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);
      KAFKA_SET_MUST_OK(_conf->set("metadata.broker.list", _brokers, errstr));
      KAFKA_SET_MUST_OK(_conf->set("enable.partition.eof", "false", errstr)); // my prod  env consume with this option true, and after replay old messages,the consume it with option true
    }
    KAFKA_SET_MUST_OK(_conf->set("queued.min.messages", "1000000", errstr));
    KAFKA_SET_MUST_OK(
        _conf->set("queued.max.messages.kbytes", "200000", errstr));

    if (!_topic_conf) {
      _topic_conf = RdKafka::Conf::create(RdKafka::Conf::CONF_TOPIC);
    }

    if (!_consumer) {
      _consumer = RdKafka::Consumer::create(_conf, errstr);
      if (!_consumer) {
        std::cerr << "Failed to create consumer: " << errstr << std::endl;
        return -1;
      }
    }

    if (!_topic) {
      _topic =
          RdKafka::Topic::create(_consumer, _topic_name, _topic_conf, errstr);
      if (!_topic) {
        std::cerr << "Failed to create topic: " << errstr << std::endl;
        return -1;
      }
    }

    return 0;
  }

brokers list: 3 brokers

b-1.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092,b-2.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092,b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092

When at upgrade time,the error last for 5 mins long

%3|1676483164.210|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1676483164.107|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 3ms in state CONNECT)
%3|1676483164.245|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1676483164.145|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 3ms in state CONNECT)
%3|1676483164.145|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Disconnected (after 543285108ms in state UP)
%3|1676483164.103|ERROR|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: 3/3 brokers are down
%6|1676483164.103|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Disconnected (after 543285108ms in state UP)
%3|1676483198.150|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 1ms in state CONNECT, 8 identical error(s) suppressed)
%3|1676483198.080|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 1ms in state CONNECT, 8 identical error(s) suppressed)
%3|1676483237.308|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 1ms in state CONNECT, 4 identical error(s) suppressed)
%3|1676483237.378|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 1ms in state CONNECT, 4 identical error(s) suppressed)
%3|1676483277.308|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 2ms in state CONNECT, 4 identical error(s) suppressed)
%3|1676483277.306|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 2ms in state CONNECT, 4 identical error(s) suppressed)
%3|1676483317.305|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 2ms in state CONNECT, 4 identical error(s) suppressed)
%3|1676483317.340|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection refused (after 2ms in state CONNECT, 4 identical error(s) suppressed)
%4|1676483378.340|REQTMOUT|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Timed out 0 in-flight, 0 retry-queued, 2 out-queue, 0 partially-sent requests
%4|1676483439.385|REQTMOUT|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Timed out 0 in-flight, 0 retry-queued, 2 out-queue, 0 partially-sent requests
%3|1676483457.033|ERROR|rdkafka#consumer-2| [thrd:app]: rdkafka#consumer-2: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection timed out (after 129648ms in state CONNECT)
%3|1676483456.950|FAIL|rdkafka#consumer-2| [thrd:b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/]: b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092/3: Connect to ipv4#172.30.37.120:9092 failed: Connection timed out (after 129648ms in state CONNECT)
[2023-02-15 17:50:57.071] [error] query_watermark_offsets(matcher-input-bar-usdt,0) return error 6,Broker: Not leader for partition

• So i suspect that all brokers had been broke down,but i found other topic(with same Replicas and ) recover very fast.
• And i found something strange in logger: the rdkafka only use b-3.matcher-input.<hide>.c3.kafka.us-west-2.amazonaws.com:9092 to connect. (so i don't whether partition failed over failed at server end or consumer didn't choose other address)