openmetadata-dremio-connector

Custom Openmetadata connector to connect data from Dremio to Openmetadata.

This connector uses the sqlalchemy-dremio package to establish a conncetion to Dremio over the arrow flight API.

Configuration

This Connector is currently implemented as CustomDatabase connector and therefore the only way to configure any connection parameters is using the connectionOptions.

connectionOptions:
  hostPort: <host>:<arrowFlightPort>
  username: <username>
  password: <password>
  # optional
  UseEncryption: False
  disableCertificateVerification: True

Local Dev Stack

Requirements:

• Access to a Dremio instance

Steps:

1. Setup python venv and install requirements
2. Start openmetadata local stack

   make local-openmetadata-stack 

3. Create workflow.dremio.yaml as a copy from workflow.dremio.template.yaml
4. Configure workflow.dremio.yaml. Mainly editing connection credentials to Dremio and the jwtToken for OpenMetadata (e.g. from the ingestion bot)
5. Run / Debug ingestion

   python.exe -m metadata ingest -c ./workflow.dremio.yaml

   Or use the provided Run configuration for IntelliJ