diff --git a/site/docs/connectors/index.md b/site/docs/connectors/index.md index 003a286b06..fddd8c02c8 100644 --- a/site/docs/connectors/index.md +++ b/site/docs/connectors/index.md @@ -21,11 +21,11 @@ The connectors that support the exchange and maintenance of metadata help to acc | Type of Connector | Description | |-----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Integration connectors](#integration-connectors) | manage the metadata exchange to a third party technology through an [integration service](/services/omis). | -| [Repository and Event Mapper connectors](#repository-and-event-mapper-connectors) | integrate metadata repositories into the open metadata ecosystem so that they can interact with one or more [open metadata repository cohorts](/services/omrs/cohort). | -| [Open Discovery Services](#open-discovery-services) | analyze the content of [resources](/concepts/resource) in the digital landscape and create annotations that are attached to the resource's [asset](/concepts/asset) metadata element in the open metadata repositories in the form of an [discovery analysis report](/concepts/discovery-analysis-report). | -| [Governance Action Services](#governance-action-services) | perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities as required. | -| [Governance Daemon Connectors](#governance-daemon-connectors) | contain specialist connectors for the governance servers that make active use of open metadata. | +| [Integration connectors](#integration-connectors) | Manage the metadata exchange to a third party technology through an [integration service](/services/omis). | +| [Repository and Event Mapper connectors](#repository-and-event-mapper-connectors) | Integrate metadata repositories into the open metadata ecosystem so that they can interact with one or more [open metadata repository cohorts](/services/omrs/cohort). | +| [Open Discovery Services](#open-discovery-services) | Analyze the content of [resources](/concepts/resource) in the digital landscape and create annotations that are attached to the resource's [asset](/concepts/asset) metadata element in the open metadata repositories in the form of an [discovery analysis report](/concepts/discovery-analysis-report). | +| [Governance Action Services](#governance-action-services) | Perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities as required. | +| [Governance Daemon Connectors](#governance-daemon-connectors) | Contain specialist connectors for the governance servers that make active use of open metadata. | ### Integration Connectors @@ -37,8 +37,8 @@ The *files* integration connectors run in the [Files Integrator Open Metadata In | Files Integration Connectors | Description | |---|---| -| [Data files monitor](/connectors/integration/data-files-monitor-integration-connector) | maintains a `DataFile` asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage). | -| [Data folder monitor](/connectors/integration/data-folder-monitor-integration-connector) | maintains a `DataFolder` asset for the directory. The files and directories underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the files and directories under the monitored directory, it results in an update to the lastModified property of the corresponding DataFolder asset. | +| [Data files monitor](/connectors/integration/data-files-monitor-integration-connector) | Maintains a `DataFile` asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage). | +| [Data folder monitor](/connectors/integration/data-folder-monitor-integration-connector) | Maintains a `DataFolder` asset for the directory. The files and directories underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the files and directories under the monitored directory, it results in an update to the lastModified property of the corresponding DataFolder asset. | #### Cataloguing Databases and their Schemas @@ -46,17 +46,18 @@ The *database* integration connectors run in the [Database Integrator Open Metad | Database Integration Connectors | Description | |--------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [PostgreSQL database connector :material-github:](https://github.com/odpi/egeria-database-connectors/tree/main/egeria-connector-postgres){ target=gh } | automatically maintains the open metadata instances for the databases hosted on a [PostgreSQL server :material-dock-window:](https://www.postgresql.org){ target=psql } This includes the database schemas, tables, columns, primary keys and foreign keys. | -| [JDBC integration connector :material-github:](https://github.com/odpi/egeria-database-connectors/tree/main/jdbc-integration-connector){ target=gh } | automatically maintains the open metadata instances on a database server via JDBC. This includes the database schemas, tables, columns, primary keys and foreign keys. | +| [PostgreSQL database connector :material-github:](https://github.com/odpi/egeria-database-connectors/tree/main/egeria-connector-postgres){ target=gh } | Automatically maintains the open metadata instances for the databases hosted on a [PostgreSQL server :material-dock-window:](https://www.postgresql.org){ target=psql } This includes the database schemas, tables, columns, primary keys and foreign keys. | +| [JDBC integration connector :material-github:](https://github.com/odpi/egeria-database-connectors/tree/main/jdbc-integration-connector){ target=gh } | Automatically maintains the open metadata instances on a database server via JDBC. This includes the database schemas, tables, columns, primary keys and foreign keys. | #### Cataloguing event topics and the structure of their events The *topic* integration connectors run in the [Topic Integrator Open Metadata Integration Service (OMIS)](/services/omis/topic-integrator/overview) hosted in the [integration daemon](/concepts/integration-daemon). -| Topic Integration Connector | Description | -|---|---| -| [Kafka Monitor topic integration connector](/connectors/integration/kafka-monitor-integration-connector) | automatically maintains the open metadata instances for the topics hosted on an [Apache Kafka server :material-dock-window:](https://kafka.apache.org/){ target=kafka }.| -| [Kafka Audit topic integration connector](/connectors/integration/kafka-topics-audit-integration-connector) | Validates that topics that are active in an [Apache Kafka server :material-dock-window:](https://kafka.apache.org/){ target=kafka } are also catalogued in open metadata. Creates an audit log record for each topic that is not catalogued. | +| Topic Integration Connector | Description | +|----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Kafka Monitor topic integration connector](/connectors/integration/kafka-monitor-integration-connector) | Automatically maintains the open metadata instances for the topics hosted on an [Apache Kafka server :material-dock-window:](https://kafka.apache.org/){ target=kafka }. | +| [Kafka Audit topic integration connector](/connectors/integration/kafka-topics-audit-integration-connector) | Validates that topics that are active in an [Apache Kafka server :material-dock-window:](https://kafka.apache.org/){ target=kafka } are also catalogued in open metadata. Creates an audit log record for each topic that is not catalogued. | +| [Strimzi Monitor topic integration connector](/connectors/integration/strimzi-monitor-integration-connector) | Automatically maintains the open metadata instances for the topics hosted in [Strimzi :material-dock-window:](https://strimzi.io){ target=strimzi }. | #### Cataloguing APIs @@ -64,7 +65,7 @@ The *API* integration connectors run in the [API Integrator Open Metadata Integr | API Integration Connectors | Description | |---|---| -| [Open API Monitor integration connector](/connectors/integration/open-api-monitor-integration-connector) | automatically maintains the open metadata instances for the APIs extracted from the Open API Specification extracted from an application.| +| [Open API Monitor integration connector](/connectors/integration/open-api-monitor-integration-connector) | Automatically maintains the open metadata instances for the APIs extracted from the Open API Specification extracted from an application.| #### Populating security enforcement engines @@ -75,14 +76,14 @@ The security integration connectors run in the [Security Integrator Open Metadat The lineage integration connectors run in the [Lineage Integrator OMIS](/services/omis/lineage-integrator/overview) hosted in the [integration daemon](/concepts/integration-daemon). They support [Lineage Management](/features/lineage-management/overview). -| Lineage Integration Connectors | Description | -|--------------------------------------------------------------------------------------------------------------------------------------------|---| -| [Open Lineage Event Receiver integration connector](/connectors/integration/open-lineage-event-receiver-integration-connector) | Connector to receive open lineage events from an event topic and publish them to lineage integration connectors with listeners registered in the same instance of the Lineage Integrator OMIS.| -| [Governance Action to Open Lineage integration connector](/connectors/integration/governance-action-open-lineage-integration-connector) | Connector to listen for governance actions executing in the open metadata ecosystem, generate open lineage events for them and publish them to the integration connectors running in the same instance of Lineage Integrator OMIS that are listening for OpenLineage events. | -| [API-based Open Lineage Log Store integration connector](/connectors/integration/api-based-open-lineage-log-store-integration-connector) | Connector that calls an OpenLineage compliant API to store the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS.| +| Lineage Integration Connectors | Description | +|--------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Open Lineage Event Receiver integration connector](/connectors/integration/open-lineage-event-receiver-integration-connector) | Connector to receive open lineage events from an event topic and publish them to lineage integration connectors with listeners registered in the same instance of the Lineage Integrator OMIS. | +| [Governance Action to Open Lineage integration connector](/connectors/integration/governance-action-open-lineage-integration-connector) | Connector to listen for governance actions executing in the open metadata ecosystem, generate open lineage events for them and publish them to the integration connectors running in the same instance of Lineage Integrator OMIS that are listening for OpenLineage events. | +| [API-based Open Lineage Log Store integration connector](/connectors/integration/api-based-open-lineage-log-store-integration-connector) | Connector that calls an OpenLineage compliant API to store the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. | | [File-based Open Lineage Log Store integration connector](/connectors/integration/file-based-open-lineage-log-store-integration-connector) | Connector that stores the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. Each OpenLineage event is stored in its own file in JSON format. These files are organized according to the namespace and job name in the event. | -| [Open Lineage Cataloguer integration connector](/connectors/integration/open-lineage-cataloguer-integration-connector) | Connector to register an OpenLineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem. | -| [Sample Lineage integration connector](/connectors/integration/sample-lineage-event-receiver-integration-connector) | Connector to register an OpenLineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem. | +| [Open Lineage Cataloguer integration connector](/connectors/integration/open-lineage-cataloguer-integration-connector) | Connector to register an OpenLineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem. | +| [Sample Lineage integration connector](/connectors/integration/sample-lineage-event-receiver-integration-connector) | Sample Connector to listen for lineage events (not in a open lineage format) and catalog the associated assets, processes, schema and lineage. | ??? education "Further information relating to integration connectors" diff --git a/site/docs/connectors/integration/sample-lineage-event-receiver-integration-connector.md b/site/docs/connectors/integration/sample-lineage-event-receiver-integration-connector.md index e21df20856..94a7c7e649 100644 --- a/site/docs/connectors/integration/sample-lineage-event-receiver-integration-connector.md +++ b/site/docs/connectors/integration/sample-lineage-event-receiver-integration-connector.md @@ -122,9 +122,9 @@ Then there is an Input array of assets and an output array of assets. In the inp The Output also describes an event schema. ### Assumptions -- No deletion of assets or processes. Lineage is always adding these -- Where effectivity dates are specified - none is specified -- Only doing asset design lineage, no column lineage or operational lineage. +- No deletion of assets or processes. Lineage is always adding these. +- Where effectivity dates are specified on APIs- none are supplied. +- This is only doing asset design lineage, no column lineage or operational lineage. - All finds are done by qualifiedName - if more than one element is returned - we use the first. - Catalog the assets, schema and process if required. This can result in - deletion, creation and / or updates of schemaAttributes @@ -141,7 +141,7 @@ The Output also describes an event schema. what the process is doing as to which lineage relationships should be created. - The Dataflow relationship qualified name is set to null. This is not appropriate if multiple supply chains create a lineage relationship between an asset and process. Dataflow relationships between the assets and process are [multi-link](https://egeria-project.org/concepts/uni-multi-link/?h=multi+link#multi-link-relationships). -- Though it i possible to supply an array of schemas, currently only the first EventType is processed. See [https://github.com/odpi/egeria/issues/7134](https://github.com/odpi/egeria/issues/7134) has been raised to allow +- Though it is possible to supply an array of schemas, currently only the first EventType is processed. See [https://github.com/odpi/egeria/issues/7134](https://github.com/odpi/egeria/issues/7134) has been raised to allow support of EventTypeLists. This sample will need to be enhanced to make use of these new APIs.