v1.6.0

librdkafka v1.6.0

librdkafka v1.6.0 is feature release:

• KIP-429 Incremental rebalancing with sticky consumer group partition assignor (KIP-54) (by @mhowlett).
• KIP-480 Sticky producer partitioning (sticky.partitioning.linger.ms) - achieves higher throughput and lower latency through sticky selection of random partition (by @abbycriswell).
• AdminAPI: Add support for DeleteRecords(), DeleteGroups() and DeleteConsumerGroupOffsets() (by @gridaphobe)
• KIP-447 Producer scalability for exactly once semantics - allows a single transactional producer to be used for multiple input partitions. Requires Apache Kafka 2.5 or later.
• Transactional producer fixes and improvements, see Transactional Producer fixes below.
• The librdkafka.redist NuGet package now supports Linux ARM64/Aarch64.

Upgrade considerations

• Sticky producer partitioning (sticky.partitioning.linger.ms) is
  enabled by default (10 milliseconds) which affects the distribution of
  randomly partitioned messages, where previously these messages would be
  evenly distributed over the available partitions they are now partitioned
  to a single partition for the duration of the sticky time
  (10 milliseconds by default) before a new random sticky partition
  is selected.
• The new KIP-447 transactional producer scalability guarantees are only
  supported on Apache Kafka 2.5 or later, on earlier releases you will
  need to use one producer per input partition for EOS. This limitation
  is not enforced by the producer or broker.
• Error handling for the transactional producer has been improved, see
  the Transactional Producer fixes below for more information.

Known issues

• The Transactional Producer's API timeout handling is inconsistent with the
  underlying protocol requests, it is therefore strongly recommended that
  applications call rd_kafka_commit_transaction() and
  rd_kafka_abort_transaction() with the timeout_ms parameter
  set to -1, which will use the remaining transaction timeout.

Enhancements

• KIP-107, KIP-204: AdminAPI: Added DeleteRecords() (by @gridaphobe).
• KIP-229: AdminAPI: Added DeleteGroups() (by @gridaphobe).
• KIP-496: AdminAPI: Added DeleteConsumerGroupOffsets().
• KIP-464: AdminAPI: Added support for broker-side default partition count
  and replication factor for CreateTopics().
• Windows: Added ssl.ca.certificate.stores to specify a list of
  Windows Certificate Stores to read CA certificates from, e.g.,
  CA,Root. Root remains the default store.
• Use reentrant rand_r() on supporting platforms which decreases lock
  contention (@azat).
• Added assignor debug context for troubleshooting consumer partition
  assignments.
• Updated to OpenSSL v1.1.1i when building dependencies.
• Update bundled lz4 (used when ./configure --disable-lz4-ext) to v1.9.3
  which has vast performance improvements.
• Added rd_kafka_conf_get_default_topic_conf() to retrieve the
  default topic configuration object from a global configuration object.
• Added conf debugging context to debug - shows set configuration
  properties on client and topic instantiation. Sensitive properties
  are redacted.
• Added rd_kafka_queue_yield() to cancel a blocking queue call.
• Will now log a warning when multiple ClusterIds are seen, which is an
  indication that the client might be erroneously configured to connect to
  multiple clusters which is not supported.
• Added rd_kafka_seek_partitions() to seek multiple partitions to
  per-partition specific offsets.

Fixes

General fixes

• Fix a use-after-free crash when certain coordinator requests were retried.
• The C++ oauthbearer_set_token() function would call free() on
  a new-created pointer, possibly leading to crashes or heap corruption (#3194)

Consumer fixes

• The consumer assignment and consumer group implementations have been
  decoupled, simplified and made more strict and robust. This will sort out
  a number of edge cases for the consumer where the behaviour was previously
  undefined.
• Partition fetch state was not set to STOPPED if OffsetCommit failed.
• The session timeout is now enforced locally also when the coordinator
  connection is down, which was not previously the case.

Transactional Producer fixes

• Transaction commit or abort failures on the broker, such as when the
  producer was fenced by a newer instance, were not propagated to the
  application resulting in failed commits seeming successful.
  This was a critical race condition for applications that had a delay after
  producing messages (or sendings offsets) before committing or
  aborting the transaction. This issue has now been fixed and test coverage
  improved.
• The transactional producer API would return RD_KAFKA_RESP_ERR__STATE
  when API calls were attempted after the transaction had failed, we now
  try to return the error that caused the transaction to fail in the first
  place, such as RD_KAFKA_RESP_ERR__FENCED when the producer has
  been fenced, or RD_KAFKA_RESP_ERR__TIMED_OUT when the transaction
  has timed out.
• Transactional producer retry count for transactional control protocol
  requests has been increased from 3 to infinite, retriable errors
  are now automatically retried by the producer until success or the
  transaction timeout is exceeded. This fixes the case where
  rd_kafka_send_offsets_to_transaction() would fail the current
  transaction into an abortable state when CONCURRENT_TRANSACTIONS was
  returned by the broker (which is a transient error) and the 3 retries
  were exhausted.

Producer fixes

• Calling rd_kafka_topic_new() with a topic config object with
  message.timeout.ms set could sometimes adjust the global linger.ms
  property (if not explicitly configured) which was not desired, this is now
  fixed and the auto adjustment is only done based on the
  default_topic_conf at producer creation.
• rd_kafka_flush() could previously return RD_KAFKA_RESP_ERR__TIMED_OUT
  just as the timeout was reached if the messages had been flushed but
  there were now no more messages. This has been fixed.

Checksums

Release asset checksums:

• v1.6.0.zip SHA256 af6f301a1c35abb8ad2bb0bab0e8919957be26c03a9a10f833c8f97d6c405aa8
• v1.6.0.tar.gz SHA256 3130cbd391ef683dc9acf9f83fe82ff93b8730a1a34d0518e93c250929be9f6b

v1.5.3

librdkafka v1.5.3

librdkafka v1.5.3 is a maintenance release.

Upgrade considerations

• CentOS 6 is now EOL and is no longer included in binary librdkafka packages,
  such as NuGet.

Fixes

General fixes

• Fix a use-after-free crash when certain coordinator requests were retried.

Consumer fixes

• Consumer would not filter out messages for aborted transactions
  if the messages were compressed (#3020).
• Consumer destroy without prior close() could hang in certain
  cgrp states (@gridaphobe, #3127).
• Fix possible null dereference in Message::errstr() (#3140).
• The roundrobin partition assignment strategy could get stuck in an
  endless loop or generate uneven assignments in case the group members
  had asymmetric subscriptions (e.g., c1 subscribes to t1,t2 while c2
  subscribes to t2,t3). (#3159)

Checksums

Release asset checksums:

• v1.5.3.zip SHA256 3f24271232a42f2d5ac8aab3ab1a5ddbf305f9a1ae223c840d17c221d12fe4c1
• v1.5.3.tar.gz SHA256 2105ca01fef5beca10c9f010bc50342b15d5ce6b73b2489b012e6d09a008b7bf

v1.5.2

librdkafka v1.5.2

librdkafka v1.5.2 is a maintenance release.

Upgrade considerations

• The default value for the producer configuration property retries has
  been increased from 2 to infinity, effectively limiting Produce retries to
  only message.timeout.ms.
  As the reasons for the automatic internal retries vary (various broker error
  codes as well as transport layer issues), it doesn't make much sense to limit
  the number of retries for retriable errors, but instead only limit the
  retries based on the allowed time to produce a message.
• The default value for the producer configuration property
  request.timeout.ms has been increased from 5 to 30 seconds to match
  the Apache Kafka Java producer default.
  This change yields increased robustness for broker-side congestion.

Enhancements

• The generated CONFIGURATION.md (through rd_kafka_conf_properties_show()))
  now include all properties and values, regardless if they were included in
  the build, and setting a disabled property or value through
  rd_kafka_conf_set() now returns RD_KAFKA_CONF_INVALID and provides
  a more useful error string saying why the property can't be set.
• Consumer configs on producers and vice versa will now be logged with
  warning messages on client instantiation.

Fixes

Security fixes

• There was an incorrect call to zlib's inflateGetHeader() with
  unitialized memory pointers that could lead to the GZIP header of a fetched
  message batch to be copied to arbitrary memory.
  This function call has now been completely removed since the result was
  not used.
  Reported by Ilja van Sprundel.

General fixes

• rd_kafka_topic_opaque() (used by the C++ API) would cause object
  refcounting issues when used on light-weight (error-only) topic objects
  such as consumer errors (#2693).
• Handle name resolution failures when formatting IP addresses in error logs,
  and increase printed hostname limit to ~256 bytes (was ~60).
• Broker sockets would be closed twice (thus leading to potential race
  condition with fd-reuse in other threads) if a custom socket_cb would
  return error.

Consumer fixes

• The roundrobin partition.assignment.strategy could crash (assert)
  for certain combinations of members and partitions.
  This is a regression in v1.5.0. (#3024)
• The C++ KafkaConsumer destructor did not destroy the underlying
  C rd_kafka_t instance, causing a leak if close() was not used.
• Expose rich error strings for C++ Consumer Message->errstr().
• The consumer could get stuck if an outstanding commit failed during
  rebalancing (#2933).
• Topic authorization errors during fetching are now reported only once (#3072).

Producer fixes

• Topic authorization errors are now properly propagated for produced messages,
  both through delivery reports and as ERR_TOPIC_AUTHORIZATION_FAILED
  return value from produce*() (#2215)
• Treat cluster authentication failures as fatal in the transactional
  producer (#2994).
• The transactional producer code did not properly reference-count partition
  objects which could in very rare circumstances lead to a use-after-free bug
  if a topic was deleted from the cluster when a transaction was using it.
• ERR_KAFKA_STORAGE_ERROR is now correctly treated as a retriable
  produce error (#3026).
• Messages that timed out locally would not fail the ongoing transaction.
  If the application did not take action on failed messages in its delivery
  report callback and went on to commit the transaction, the transaction would
  be successfully committed, simply omitting the failed messages.
• EndTxnRequests (sent on commit/abort) are only retried in allowed
  states (#3041).
  Previously the transaction could hang on commit_transaction() if an abortable
  error was hit and the EndTxnRequest was to be retried.

Note: there was no v1.5.1 librdkafka release

Checksums

Release asset checksums:

• v1.5.2.zip SHA256 de70ebdb74c7ef8c913e9a555e6985bcd4b96eb0c8904572f3c578808e0992e1
• v1.5.2.tar.gz SHA256 ca3db90d04ef81ca791e55e9eed67e004b547b7adedf11df6c24ac377d4840c6

v1.5.0

librdkafka v1.5.0

The v1.5.0 release brings usability improvements, enhancements and fixes to
librdkafka.

Enhancements

• Improved broker connection error reporting with more useful information and
  hints on the cause of the problem.
• Consumer: Propagate errors when subscribing to unavailable topics (#1540)
• Producer: Add batch.size producer configuration property (#638)
• Add topic.metadata.propagation.max.ms to allow newly manually created
  topics to be propagated throughout the cluster before reporting them
  as non-existent. This fixes race issues where CreateTopics() is
  quickly followed by produce().
• Prefer least idle connection for periodic metadata refreshes, et.al.,
  to allow truly idle connections to time out and to avoid load-balancer-killed
  idle connection errors (#2845)
• Added rd_kafka_event_debug_contexts() to get the debug contexts for
  a debug log line (by @wolfchimneyrock).
• Added Test scenarios which define the cluster configuration.
• Added MinGW-w64 builds (@ed-alertedh, #2553)
• ./configure --enable-XYZ now requires the XYZ check to pass,
  and --disable-XYZ disables the feature altogether (@benesch)
• Added rd_kafka_produceva() which takes an array of produce arguments
  for situations where the existing rd_kafka_producev() va-arg approach
  can't be used.
• Added rd_kafka_message_broker_id() to see the broker that a message
  was produced or fetched from, or an error was associated with.
• Added RTT/delay simulation to mock brokers.

Upgrade considerations

• Subscribing to non-existent and unauthorized topics will now propagate
  errors RD_KAFKA_RESP_ERR_UNKNOWN_TOPIC_OR_PART and
  RD_KAFKA_RESP_ERR_TOPIC_AUTHORIZATION_FAILED to the application through
  the standard consumer error (the err field in the message object).
• Consumer will no longer trigger auto creation of topics,
  allow.auto.create.topics=true may be used to re-enable the old deprecated
  functionality.
• The default consumer pre-fetch queue threshold queued.max.messages.kbytes
  has been decreased from 1GB to 64MB to avoid excessive network usage for low
  and medium throughput consumer applications. High throughput consumer
  applications may need to manually set this property to a higher value.
• The default consumer Fetch wait time has been increased from 100ms to 500ms
  to avoid excessive network usage for low throughput topics.
• If OpenSSL is linked statically, or ssl.ca.location=probe is configured,
  librdkafka will probe known CA certificate paths and automatically use the
  first one found. This should alleviate the need to configure
  ssl.ca.location when the statically linked OpenSSL's OPENSSLDIR differs
  from the system's CA certificate path.
• The heuristics for handling Apache Kafka < 0.10 brokers has been removed to
  improve connection error handling for modern Kafka versions.
  Users on Brokers 0.9.x or older should already be configuring
  api.version.request=false and broker.version.fallback=... so there
  should be no functional change.
• The default producer batch accumulation time, linger.ms, has been changed
  from 0.5ms to 5ms to improve batch sizes and throughput while reducing
  the per-message protocol overhead.
  Applications that require lower produce latency than 5ms will need to
  manually set linger.ms to a lower value.
• librdkafka's build tooling now requires Python 3.x (python3 interpreter).

Fixes

General fixes

• The client could crash in rare circumstances on ApiVersion or
  SaslHandshake request timeouts (#2326)
• ./configure --LDFLAGS='a=b, c=d' with arguments containing = are now
  supported (by @sky92zwq).
• ./configure arguments now take precedence over cached configure variables
  from previous invocation.
• Fix theoretical crash on coord request failure.
• Unknown partition error could be triggered for existing partitions when
  additional partitions were added to a topic (@benesch, #2915)
• Quickly refresh topic metadata for desired but non-existent partitions.
  This will speed up the initial discovery delay when new partitions are added
  to an existing topic (#2917).

Consumer fixes

• The roundrobin partition assignor could crash if subscriptions
  where asymmetrical (different sets from different members of the group).
  Thanks to @ankon and @wilmai for identifying the root cause (#2121).
• The consumer assignors could ignore some topics if there were more subscribed
  topics than consumers in taking part in the assignment.
• The consumer would connect to all partition leaders of a topic even
  for partitions that were not being consumed (#2826).
• Initial consumer group joins should now be a couple of seconds quicker
  thanks expedited query intervals (@benesch).
• Fix crash and/or inconsistent subscriptions when using multiple consumers
  (in the same process) with wildcard topics on Windows.
• Don't propagate temporary offset lookup errors to application.
• Immediately refresh topic metadata when partitions are reassigned to other
  brokers, avoiding a fetch stall of up to topic.metadata.refresh.interval.ms. (#2955)
• Memory for batches containing control messages would not be freed when
  using the batch consume APIs (@pf-qiu, #2990).

Producer fixes

• Proper locking for transaction state in EndTxn handler.

Checksums

Release asset checksums:

• v1.5.0.zip SHA256 76a1e83d643405dd1c0e3e62c7872b74e3a96c52be910233e8ec02d501fa33c8
• v1.5.0.tar.gz SHA256 f7fee59fdbf1286ec23ef0b35b2dfb41031c8727c90ced6435b8cf576f23a656

v1.4.4

librdkafka v1.4.4

v1.4.4 is a maintenance release with the following fixes and enhancements:

• Transactional producer could crash on request timeout due to dereferencing
  NULL pointer of non-existent response object.
• Mark rd_kafka_send_offsets_to_transaction() CONCURRENT_TRANSACTION (et.al)
  errors as retriable.
• Fix crash on transactional coordinator FindCoordinator request failure.
• Minimize broker re-connect delay when broker's connection is needed to
  send requests.
• socket.timeout.ms was ignored when transactional.id was set.
• Added RTT/delay simulation to mock brokers.

Note: there was no v1.4.3 librdkafka release

v1.4.2

librdkafka v1.4.2

v1.4.2 is a maintenance release with the following fixes and enhancements:

• Fix produce/consume hang after partition goes away and comes back,
  such as when a topic is deleted and re-created (regression in v1.3.0).
• Consumer: Reset the stored offset when partitions are un-assign()ed (fixes #2782).
  This fixes the case where a manual offset-less commit() or the auto-committer
  would commit a stored offset from a previous assignment before
  a new message was consumed by the application.
• Probe known CA cert paths and set default ssl.ca.location accordingly
  if OpenSSL is statically linked or ssl.ca.location is set to probe.
• Per-partition OffsetCommit errors were unhandled (fixes #2791)
• Seed the PRNG (random number generator) by default, allow application to override with enable.random.seed=false (#2795)
• Fix stack overwrite (of 1 byte) when SaslHandshake MechCnt is zero
• Align bundled c11 threads (tinycthreads) constants to glibc and musl (#2681)
• Fix return value of rd_kafka_test_fatal_error() (by @ckb42)
• Ensure CMake sets disabled defines to zero on Windows (@benesch)
• librdkafka's build tooling now requires Python 3.x (the python3 interpreter).

Note: there was no v1.4.1 librdkafka release

Checksums

Release asset checksums:

• v1.4.2.zip SHA256 ac50da08be69365988bad3d0c46cd87eced9381509d80d3d0b4b50b2fe9b9fa9
• v1.4.2.tar.gz SHA256 3b99a36c082a67ef6295eabd4fb3e32ab0bff7c6b0d397d6352697335f4e57eb

v1.4.0

librdkafka v1.4.0

v1.4.0 is a feature release:

• KIP-98: Transactional Producer API
• KIP-345: Static consumer group membership (by @rnpridgeon)
• KIP-511: Report client software name and version to broker

Transactional Producer API

librdkafka now has complete Exactly-Once-Semantics (EOS) functionality, supporting the idempotent producer (since v1.0.0), a transaction-aware consumer (since v1.2.0) and full producer transaction support (in this release).
This enables developers to create Exactly-Once applications with Apache Kafka.

See the Transactions in Apache Kafka page for an introduction and check the librdkafka transactions example for a complete transactional application example.

Security fixes

Two security issues have been identified in the SASL SCRAM protocol handler:

• The client nonce, which is expected to be a random string, was a static string.
• If sasl.username and sasl.password contained characters that needed escaping, a buffer overflow and heap corruption would occur. This was protected, but too late, by an assertion.

Both of these issues are fixed in this release.

Enhancements

• Add FNV-1a partitioner (by @Manicben, #2724).
  The new fnv1a_random partitioner is compatible with Sarama's NewHashPartitioner partition, easing transition from Sarama to librdkafka-based clients such as confluent-kafka-go.
• Added rd_kafka_error_t / RdKafka::Error complex error type which provides error attributes such as indicating if an error is retriable.
• The builtin mock broker now supports balanced consumer groups.
• Support finding headers in nonstandard directories in CMake (@benesch)
• Improved static library bundles which can now contain most dependencies.
• Documentation, licenses, etc, is now installed by make install
• Bump OpenSSL to v1.0.2u (when auto-building dependencies)

Fixes

General:

• Correct statistics names in docs (@TimWSpence, #2754)
• Wake up broker thread based on next request retry.
  Prior to this fix the next wakeup could be delayed up to 1 second regardless of next retry.
• Treat SSL peer resets as usual Disconnects, making log.connection.close work
• Reset buffer corrid on connection close to honour ApiVers and Sasl request priorities (@xzxxzx401, #2666)
• Cleanup conf object if failing to creat producer or consumer (@fboranek)
• Fix build of rdkafka_example project for windows, when using building it using Visual Studio 2017/2019 (by @Eliyahu-Machluf)
• Minor fix to rdkafka_example usage: add lz4 and zstd compression codec to usage (by @Eliyahu-Machluf)
• Let broker nodename updates propagate as ERR__TRANSPORT rather than ERR__NODE_UPDATE to avoid an extra error code for the application to handle.
• Fix erroneous refcount assert in enq_once_del_source (e.g., on admin operation timeout)
• Producers could get stuck in INIT state after a disconnect until a to-be-retried request timed out or the connection was needed for other purposes (metadata discovery, etc), this is now fixed.

Producer:

• flush() now works with RD_KAFKA_EVENT_DR
• Fix race condition when finding EOS-supporting broker

Consumer:

• Consumers could get stuck after rebalance if assignment was empty
• Enforce session.timeout.ms in the consumer itself (#2631)
• max.poll.interval.ms is now only enforced when using subscribe()
• Fix consumer_lag calculation for transactional topics
• Show fetch/no-fetch reason in topic debugging
• Properly propagate commit errors per partition
• Don't send heartbeats after max.poll.interval.ms is exceeded.
• Honour array size in rd_kafka_event_message_array() to avoid overflow (#2773)

Checksums

Release asset checksums:

• v1.4.0.zip SHA256 eaf954e3b8a2ed98360b2c76f55048ee911964de8aefd8a9e1133418ec9f48dd
• v1.4.0.tar.gz SHA256 ae27ea3f3d0d32d29004e7f709efbba2666c5383a107cc45b3a1949486b2eb84

v1.3.0

librdkafka v1.3.0 release

This is a feature release adding support for KIP-392 Fetch from follower, allowing a consumer to fetch messages from the closest replica to increase throughput and reduce cost.

Features

• KIP-392 - Fetch messages from closest replica / follower (by @mhowlett)
• Added experimental (subject to change or removal) mock broker to make application and librdkafka development testing easier.

Fixes

• Fix consumer_lag in stats when consuming from broker versions <0.11.0.0 (regression in librdkafka v1.2.0).

Checksums

Release asset checksums:

• v1.3.0.zip SHA256 bd3373c462c250ecebea9043fb94597a11bd6e0871d3cde19019433d3f74a99e
• v1.3.0.tar.gz SHA256 465cab533ebc5b9ca8d97c90ab69e0093460665ebaf38623209cf343653c76d2

v1.2.2

librdkafka v1.2.2 release

v1.2.2 fixes the producer performance regression introduced in v1.2.1 which may affect high-throughput producer applications.

Fixes

• Fix producer insert msgq regression in v1.2.1 (#2450).
• Upgrade builtin lz4 to 1.9.2 (CVE-2019-17543, #2598).
• Don't trigger error when broker hostname changes (#2591).
• Less strict message.max.bytes check for individual messages (#993).
• Don't call timespec_get() on OSX (since it was removed in recent XCode) by @maparent .
• configure: add --runstatedir for compatibility with autoconf.
• LZ4 is available from ProduceRequest 0, not 3 (fixes assert in #2480).
• Address 12 code issues identified by Coverity static code analysis.

Enhancements

• Add warnings for inconsistent security configuration.
• Optimizations to hdr histogram (stats) rollover.
• Reorganized examples and added a cleaner consumer example, added minimal C++ producer example.
• Print compression type per message-set when debug=msg

Checksums

Release asset checksums:

• v1.2.2.zip SHA256 7557b37e5133ed4c9b0cbbc3fd721c51be8e934d350d298bd050fcfbc738e551
• v1.2.2.tar.gz SHA256 c5d6eb6ce080431f2996ee7e8e1f4b8f6c61455a1011b922e325e28e88d01b53

v1.2.1

librdkafka v1.2.1 release

Warning: v1.2.1 has a producer performance regression which may affect high-throughput producer applications. We recommend such users to upgrade to v1.3.0

v1.2.1 is a maintenance release:

• Properly handle new Kafka-framed SASL GSSAPI frame semantics on Windows (#2542).
  This bug was introduced in v1.2.0 and broke GSSAPI authentication on Windows.
• Fix msgq (re)insertion code to avoid O(N^2) insert sort operations on retry (#2508)
  The msgq insert code now properly handles interleaved and overlapping
  message range inserts, which may occur during Producer retries for
  high-throughput applications.
• configure: added --disable-c11threads to avoid using libc-provided C11 threads.
• configure: added more autoconf compatibility options to ignore

Checksums

Release asset checksums:

• v1.2.1.zip SHA256 8b5e95318b190f40cbcd4a86d6a59dbe57b54a920d8fdf64d9c850bdf05002ca
• v1.2.1.tar.gz SHA256 f6be27772babfdacbbf2e4c5432ea46c57ef5b7d82e52a81b885e7b804781fd6