Long term plan for ruby-kafka

[sadhu89]

Hello! What are the plans for this library in the medium/long term? After racecar migrated from ruby-kafka to rdkafka-ruby, I saw a few comments about slowing down the development of this library in favour of rdkafka-ruby.
See:

• https://user-images.githubusercontent.com/1410645/101079509-d024b380-3564-11eb-9e1b-e4b9bcf39fcf.png
• Change ruby-kafka to rdkafka-ruby  racecar#97 (comment)

We are trying to choose a library to write to Kafka, and saw those comments recommending rdkafka-ruby instead but it doesn't support IAM roles (A feature we need). So we are favouring ruby-kafka but we are a little bit concerned that the library could become unmaintained in the future.

[mensfeld]

As one of the contributors, I can say this library is pretty much abandoned and should no longer be considered for any production usage lasting long. As @dasch said in the comment and on a few other occasions, there is no point in trying to keep up with Kafka development when other languages use a well-established C binding and it gets more official support.

• racecar moved to librdkafka (Change ruby-kafka to rdkafka-ruby  racecar#97 (comment))
• waterdrop moved to librdkafka (2.0 karafka/waterdrop#106)
• karafka moved to librdkafka (2.0 Roadmap karafka/karafka#431)
• phobos decided to stay (Support rdkafka-ruby implementation phobos/phobos#132

IAM may not be supported but you can still use MSK via SASL + SSL. Here's the setup required to make it work with rdkafka: https://karafka.io/docs/Deployment/#karafka-configuration-for-aws-msk-sasl-ssl

On top of that, I do get reports of users of Karafka 1.4 (ruby-kafka version based) that highlight either design issues, stability issues, or bugs and I do not expect them to be fixed.

Overall: not recommended, I would consider it either in a maintenance mode or fully abandoned.

Also, there is a KIP for this for librdkafka: https://cwiki.apache.org/confluence/display/kafka/kip-86%3a+configurable+sasl+callback+handlers I think it's just a matter of time and priorities.

[dasch]

Lots of folks are still using ruby-kafka, but there's no concerted effort to keep up with Kafka features; it's simply not feasible for such a complex protocol. I recommend using librdkafka based libraries and frameworks instead.

[mensfeld]

@dasch would you be ok with me creating a deprecation notice for the readme recommending moving to rdkafka or rdkafka backed tooling?

[dasch]

Sure