This proposal suggests to create a configuration provider for reading data from environment variables.
Apache Kafka supports pluggable configuration providers for loading configuration data from external sources. They can be used in the configuration of the different components. By default, Kafka provides two configuration providers:
FileConfigProviderfor reading configuration records, as key-value pairs, from a single properties fileDirectoryConfigProviderfor reading one or more configuration files from a specified directory (files are read from start to finish)
In addition, users can implement their own configuration providers.
Strimzi already has one custom configuration provider. Kubernetes Configuration Provider can be used get data from Secrets or Config Maps using Kubernetes client directly from the Kubernetes API. This was proposed in Strimzi Proposal #027.
One of the common ways how to configure applications running on Kubernetes is using environment variables. Environment variables can be set in the Pods (or Deployments, StatefulSets etc.). The values for the environment variables can be either set directly or mapped from other resources. For example from Secrets, Config Maps or from the Downward API. Using environment variables for application configuration also corresponds to the 12 Factor principles.
However, most Kafka components are configured from a properties file and not from environment variables. And Kafka does not support any easy way how to set the individual properties to values from environment variable directly inside the properties file. That means that in order to use the environment variables, you have to generate the properties file and insert the values from the environment variables into it. This is something we do in our own bash scripts in our container images as well.
Having a configuration provider which would load the environment variables might help us to simplify our bash scripts. The whole configuration file might be for example generated by the operator. Instead of the references which it contains today and which are replaced by the environment variables in the bash scripts, it might call the configuration provider directly and no replacements will be needed later in bash. Doing more work in Java instead of in bash would make the code cleaner and testing easier. It would also simplify the API between the operator and the container images.
The configuration provider might be of course useful also to Strimzi users or to Apache Kafka users in general and not just to us. Users might for example use it to load values from environment variables in connector configurations or in client configurations.
We should create a new configuration provider which would load data from environment variables. To use the configuration provider, the user will need to configure their client like this:
config.providers=env
config.providers.env.class=io.strimzi.kafka.EnvVarConfigProvider
option1=${env::FIRST_ENV_VAR}
option2=${env::SECOND_ENV_VAR}The configuration provider should use a separate GitHub repository under the Strimzi GitHub organization.
The repository should be named kafka-env-var-config-provider.
It should be published to Maven Central as io.strimzi:kafka-env-var-config-provider.
It should be also bundled with all out images (Kafka, Kafka Bridge, etc.).
The configuration provider should be compatible with all recent Apache Kafka versions.
This is a separate component which should make it easier for users to use Strimzi based Apache Kafka clusters from their applications. But it has currently no direct impact on the other resources.
No other alternatives were considered.