Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connector instantiation requires information that may not be present at the moment #413

Closed
sujuka99 opened this issue Jan 15, 2024 · 0 comments
Labels
component/pipeline-components Related to pipeline components

Comments

@sujuka99
Copy link
Contributor

sujuka99 commented Jan 15, 2024

Example code:

class CustomApp(StreamsApp):

    def inflate(self) -> list[PipelineComponent]:
        return [self, *self.derive_connectors()]

    def derive_connectors(self) -> list[PipelineComponent]:
        connectors = []
        for topic_name in [...]:
              connectors.append(self.to_connector(topic_name))
        return connectors

    def to_connector(self, topic_name) -> KafkaSinkConnector:
        name = f"{topic_name}"
        return KafkaSinkConnector(
            name=name,
            config=self.config,
            handlers=self.handlers,
            app=KafkaConnectorConfig(
                **{
                    "name": "${pipeline.name}-" + name,  # using `self.full_name` results in an error
                    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",  # Cannot be set in `defaults.yaml`
                }
            ),
        )

When initializing the connector in the above example, self.full_name == pipeline_name-connector_name, but ${pipeline.name}-connector_name is expected, i.e. self.prefix is already substituted.

It can be circumvented by setting the name in KafkaConnectorConfig to KafkaSinkConnector.model_fields["prefix"].default + name, but it is a workaround.

Closely related to #412

@sujuka99 sujuka99 added type/update Update dependencies component/pipeline-components Related to pipeline components and removed type/update Update dependencies labels Jan 15, 2024
@sujuka99 sujuka99 closed this as not planned Won't fix, can't repro, duplicate, stale Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/pipeline-components Related to pipeline components
Projects
None yet
Development

No branches or pull requests

1 participant