% Failed to produce to topic test: Local: Queue full

[sandippatel-incedo]

Facing above Error with default librdkakfka configuration, in C producer ( Ubuntu20.04 - librdkafka version 1.2.1-1ubuntu1)

1. Publishing data at rate of 24,00,000 packets (each packet of ~350 bytes) within 30 seconds
2. Consumer.c is in wait state till 30 seconds and start consuming at every 30 seconds.

Observation:
If a change/increase values in linger.ms or queue.buffering.max.messages, Producer drastically reduce publishing packets (Almost half) ( to change in linger.ms or queue.buffering.max.messages parameter found in multiple discussion )

Request to take a look on above query & guide.

[rolandyoung]

That error will occur if your producer is producing messages faster than the broker(s) are accepting them. This may be due to lack of compute resources on the brokers, lack of network bandwidth between the producer and the brokers, or failure to connect at all to the broker.

You can investigate this by producing at a low message rate and checking that messages are actually getting through to your consumers. Then increase to a moderate message rate and check network and compute resource usage on the producer host and the broker host. Then keep increasing the message rate and see whether the usage of those resources increases proportionally. If the usage of one of the resources stops increasing (and you start seeing the queue full message), then that resource may be insufficient for your message load.