Possibility of using manual partition assignment instead of consumer groups in KafkaBroker #1338
-
Hi! consumer = Consumer(config)
topics = consumer.list_topics().topics
partitions = [TopicPartition(target_topic, partition=partition, offset=OFFSET_END) for partition in list(topics[target_topic].partitions.keys())]
consumer.assign(partitions) In the case this isn't supported right now (by settings the arguments a certain way), faststream could potentially simply things here to something like this ( @broker.assign(topics="some-topic")
def handle1(message: str):
do_something()
@broker.assign(partitions=[TopicPartition(topic="some-topic", partition=0, offset=OFFSET_END), TopicPartition(topic="some-topic", partition=1, offset=OFFSET_END)])
def handle2(message: str):
do_something() An example of where this would be useful is in any case where each consumer application instance needs to process the entire topic, so pretty much the opposite of what consumer groups are used for. A workaround that comes to mind is to set group_id to some random value that's unique to each running instance, but this is a bit hacky, and could potentially still count as a consumer group from the perspective of the broker, which is an issue in certain Kafka offerings where the total number of consumer groups is limited to some small amount, so it's a shame to waste it on this. Here's a link to this exact discussion in the confluent-kafka repo. Great library by the way 👏 |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
Hi! We are planning to add this option in #1319 for confluent and aio-kafka brokers both It should looks like @broker.subscriber(partitions=[
TopicPartition(topic="some-topic", partition=0, offset=OFFSET_END),
TopicPartition(topic="some-topic", partition=1, offset=OFFSET_END),
])
def handle2(message: str):
do_something() We are using |
Beta Was this translation helpful? Give feedback.
Hi! We are planning to add this option in #1319 for confluent and aio-kafka brokers both
It should looks like
We are using
@broker.subscriber
specific options for all ways to subscribe on events