🐛 kafka-python-ng set a default hard limit of 1MB for each message (e…

…vent) sent to Kafka message bus. This limit can now be overriden by setting the new optional 'plugins.messagebus.kafka.settings.max_request_size' configuration directive for 'plugins.messagebus_producers.kafka' used only by 'hermes-server'

ChangeLog.md

@@ -5,6 +5,12 @@ All notable changes to Hermes will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- kafka-python-ng set a default hard limit of 1MB for each message (event) sent to Kafka message bus. This limit can now be overriden by setting the new optional `plugins.messagebus.kafka.settings.max_request_size` configuration directive for `plugins.messagebus_producers.kafka` used only by `hermes-server`.
+
 ## [1.0.0-alpha.5] - 2024-09-05
 
 ### Fixed

plugins/messagebus_producers/kafka/README.md

@@ -45,6 +45,15 @@ hermes:
           # by a "self.check_version()" call.
           api_version: [2, 6, 0]
 
+          # Facultative: Hard limit on the size of a message sent to Kafka.
+          # You should set a higher value if your Kafka messages are likely to
+          # exceed the default of 1MB or if you encountered the error
+          #   "MessageSizeTooLargeError: The message is xxx bytes when
+          #    serialized which is larger than the maximum request size you
+          #    have configured with the max_request_size configuration".
+          # Default: 1048576.
+          max_request_size: 1048576
+
           # Facultative: enables SSL authentication. If set, the 3 options below
           # must be defined
           ssl:

plugins/messagebus_producers/kafka/config-schema-plugin-kafka.yml

@@ -36,6 +36,13 @@ api_version:
   minlength: 1
   schema:
     type: integer
+max_request_size:
+  type: integer
+  required: true
+  empty: false
+  nullable: false
+  default: 1048576
+  min: 1
 ssl:
   type: dict
   required: false

plugins/messagebus_producers/kafka/kafka.py

@@ -52,6 +52,7 @@ def __init__(self, settings: dict[str, Any]):
                     "ssl_certfile": self._settings["ssl"]["certfile"],
                     "ssl_keyfile": self._settings["ssl"]["keyfile"],
                     "ssl_cafile": self._settings["ssl"]["cafile"],
+                    "max_request_size": self._settings["max_request_size"],
                 }
             )