Skip to content

Commit

Permalink
Merge pull request #954 from mandy-chessell/code2024
Browse files Browse the repository at this point in the history
Add short descriptions of the UC connectors
  • Loading branch information
mandy-chessell committed Jul 25, 2024
2 parents 9f881da + ef45528 commit 7d73bb2
Show file tree
Hide file tree
Showing 27 changed files with 812 additions and 64 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,9 @@

# Template Catalog

The template catalog provides descriptions of the [templates](/concepts/template) provided by Egeria, through the [Content Packs](/content-packs).
The template catalog provides descriptions of the [templates](/concepts/template) provided by Egeria, through the [Content Packs](/content-packs).

* [Templates for PostgreSQL](/catalog-templates/postgres-server-templates)
* [Templates for Unity Catalog](/catalog-templates/unity-catalog-templates)

--8<-- "snippets/abbr.md"
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,7 @@ The postgreSQL Server templates are provided through the [Core Content Pack](/co

## The templates in use

![PostgreSQL template in use](postgres-server-template-in-use.svg)
![PostgreSQL template in use](postgres-server-template-in-use.svg)


--8<-- "snippets/abbr.md"
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@

# Unity Catalog Templates

The templates for Unity Catalog are supplied through the [Core Content Pack](/content-packs/core-content-pack/overview).
The templates for Unity Catalog are supplied through the [Core Content Pack](/content-packs/core-content-pack/overview).


--8<-- "snippets/abbr.md"
4 changes: 2 additions & 2 deletions site/docs/concepts/placeholder.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Content-Type: application/json
```
Below is an outline of the default template for cataloguing a PostgreSQL Database Server. This is represented as a [SoftwareServer](/types/0/0040-Software-Servers) asset, in includes a linked [SoftwareCapability](/types/0/0042-Software-Capabilities) for the [database manager (DBMS)](/types/0/0050-Applications-and-Processes). There is also a [connection](/concepts/connection) linked from the asset to define how to create a connector to the PostgreSQL Database Server. Notice that throughout the template, there are placeholder properties:

![PostgreSQL Database Server Template](/templates/postgres-server-catalog-template.svg)
![PostgreSQL Database Server Template](/catalog-templates/postgres-server-catalog-template.svg)

Below is an example of a call to create an asset using the template:

Expand All @@ -99,7 +99,7 @@ Content-Type: application/json
The picture below shows the resulting asset elements, linked back to the elements from the template using the [SourcedFrom](/types/0/0011-Managing-Referenceables).
Notice how the `{{serverName}}` placeholder property is used in each element to create a unique qualifiedName.

![PostgreSQL Database Server Template in use](/templates/postgres-server-template-in-use.svg)
![PostgreSQL Database Server Template in use](/catalog-templates/postgres-server-template-in-use.svg)

The placeholder properties can be used to make the templates easy to use, removing much of the repetitive creation of property values. The result is a consistent set of elements for the asset.

Expand Down
12 changes: 6 additions & 6 deletions site/docs/connectors/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,12 @@ Files provide storage for many types of data. They are organizes into folders (

The Unity Catalog connectors provide a suite of function that integrates a Unity Catalog server into the open metadata ecosystem.

The [Unity Catalog Resource Connector](/connectors/unity-catalog/resource-connector) is a [digital resource connector](/concepts/digital-resource-connector) that acts as a Java client to the Unity Catalog Server REST API. It is used by the other Unity Catalog connectors.
The [Unity Catalog Server Synchronizer](/connectors/unity-catalog/sync-server-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the catalogs defined for a Unity Catalog Server.
The [Unity Catalog Inside Catalog Synchronizer](/connectors/unity-catalog/sync-catalog-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the resources (schemas, tables, volumes and functions) defined within a Catalog found in a Unity Catalog Server.
The [Unity Catalog Server Survey Service](/connectors/unity-catalog/server-survey-sevice) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Unity Catalog Server.
The [Unity Catalog Inside Catalog Survey Service](/connectors/unity-catalog/catalog-survey-sevice) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Catalog inside a Unity Catalog Server.
The [Unity Catalog Inside Schema Survey Service](/connectors/unity-catalog/schema-survey-sevice) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Schema inside a Unity Catalog Server.
* The [Unity Catalog Resource Connector](/connectors/unity-catalog/resource-connector) is a [digital resource connector](/concepts/digital-resource-connector) that acts as a Java client to the Unity Catalog Server REST API. It is used by the other Unity Catalog connectors.
* The [Unity Catalog Server Synchronizer](/connectors/unity-catalog/sync-server-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the catalogs defined for a Unity Catalog Server.
* The [Unity Catalog Inside Catalog Synchronizer](/connectors/unity-catalog/sync-catalog-connector) is an [integration connector](/concepts/integration-connector) that exchanges details about the resources (schemas, tables, volumes and functions) defined within a Catalog found in a Unity Catalog Server.
* The [Unity Catalog Server Survey Service](/connectors/unity-catalog/server-survey-service) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Unity Catalog Server.
* The [Unity Catalog Inside Catalog Survey Service](/connectors/unity-catalog/catalog-survey-service) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Catalog inside a Unity Catalog Server.
* The [Unity Catalog Inside Schema Survey Service](/connectors/unity-catalog/schema-survey-service) is a [Survey Action Service](/concepts/survey-action-service) that surveys the resources defined in a Schema inside a Unity Catalog Server.

### Apache Kafka

Expand Down
6 changes: 5 additions & 1 deletion site/docs/connectors/unity-catalog/catalog-survey-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@

# OSS Unity Catalog Inside Catalog Survey Service

Survey of a catalog within an OSS Unity Catalog Server.
The Catalog Survey Service extracts information about the schemas, tables, functions and volumes managed in a specific catalog within an OSS Unity Catalog Server.

![Connector Operation](catalog-survey-service.svg)

--8<-- "snippets/abbr.md"
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/catalog-survey-service.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
14 changes: 7 additions & 7 deletions site/docs/connectors/unity-catalog/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ The diagram below illustrates the mapping of the Unity Catalog metadata resource

![Type Mapping](unity-catalog-type-mapping.svg)

The templates that implement this mapping are described in [Unity Catalog Templates](/templates/unity-catalog-templates).
The templates that implement this mapping are described in [Unity Catalog Templates](/catalog-templates/unity-catalog-templates).

### Anchor design for Unity Catalog

Expand All @@ -124,12 +124,12 @@ The connectors shipped with Egeria are as follows:

| Connector Name | Connector Type | Purpose |
|----------------------------------------------------------------------------------------------|--------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Unity Catalog Resource Connector](/connector/unity-catalog/resource-connector) | [Digital Resource Connector](/concepts/digital-resource-connector) | Provides wrapper around Unity Catalog's REST API. |
| [Unity Catalog Server Survey](/connector/unity-catalog/server-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Server. |
| [Unity Catalog Catalog Survey](/connector/unity-catalog/catalog-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Catalog. |
| [Unity Catalog Schema Survey](/connector/unity-catalog/schema-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Schema. |
| [Unity Catalog Server Synchronizer](/connector/unity-catalog/sync-server-connector) | [Integration Connector](/concepts/integration-connector) | Bootstraps the cataloguing of a Unity Catalog Server by retrieving the catalogs and configuring the Inside Catalog Connector (below). |
| [Unity Catalog Inside Catalog Synchronizer](/connector/unity-catalog/sync-catalog-connector) | [Integration Connector](/concepts/integration-connector) | Synchronizes the metadata describing a Unity Catalog Server's catalogs, schemas, tables, functions and volumes between Unity Catalog and the Open Metadata Ecosystem. |
| [Unity Catalog Resource Connector](/connectors/unity-catalog/resource-connector) | [Digital Resource Connector](/concepts/digital-resource-connector) | Provides wrapper around Unity Catalog's REST API. |
| [Unity Catalog Server Survey](/connectors/unity-catalog/server-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Server. |
| [Unity Catalog Catalog Survey](/connectors/unity-catalog/catalog-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Catalog. |
| [Unity Catalog Schema Survey](/connectors/unity-catalog/schema-survey-service) | [Survey Action Service](/concepts/survey-action-service) | Surveys the contents of a Unity Catalog Schema. |
| [Unity Catalog Server Synchronizer](/connectors/unity-catalog/sync-server-connector) | [Integration Connector](/concepts/integration-connector) | Bootstraps the cataloguing of a Unity Catalog Server by retrieving the catalogs and configuring the Inside Catalog Connector (below). |
| [Unity Catalog Inside Catalog Synchronizer](/connectors/unity-catalog/sync-catalog-connector) | [Integration Connector](/concepts/integration-connector) | Synchronizes the metadata describing a Unity Catalog Server's catalogs, schemas, tables, functions and volumes between Unity Catalog and the Open Metadata Ecosystem. |


--8<-- "snippets/abbr.md"
1 change: 1 addition & 0 deletions site/docs/connectors/unity-catalog/resource-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@

The *OSS Unity Catalog resource connector* provides a client wrapper to the Open Source version of Unity Catalog. It aims to provide a buffer to the integration connector and the survey service to protect them from changes to the unity catalog API (which are expected).

![Connector Operation](resource-connector.svg)
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/resource-connector.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 5 additions & 1 deletion site/docs/connectors/unity-catalog/schema-survey-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@

# OSS Unity Catalog Schema Survey Service

Survey of a schema within an OSS Unity Catalog Server.
The Schema Survey Service extracts information about the tables, functions and volumes managed in a specific schema within an OSS Unity Catalog Server.

![Connector Operation](schema-survey-service.svg)

--8<-- "snippets/abbr.md"
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/schema-survey-service.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 5 additions & 1 deletion site/docs/connectors/unity-catalog/server-survey-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,8 @@

# OSS Unity Catalog Server Survey Service

Survey for an instance of the OSS Unity Catalog Server to discover the list of catalogs it is managing.
The Server Survey Service extracts information about the catalogs, schemas, tables, functions and volumes managed in a specific catalog within an OSS Unity Catalog Server.

![Connector Operation](server-survey-service.svg)

--8<-- "snippets/abbr.md"
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/server-survey-service.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 7 additions & 1 deletion site/docs/connectors/unity-catalog/sync-catalog-connector.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->

# OSS Unity Catalog (UC) Inside a Catalog Synchronizing Connector
# OSS Unity Catalog (UC) Inside a Catalog Synchronizing Connector

The Unity Catalog Inside Catalog Synchronizer Connector is an [integration connector](/concepts/integration-connector) that synchronizes metadata between catalogs in Unity Catalog and the open metadata ecosystem. If an element originates in Unity Catalog, the copy in open metadata is kept in sync with the unity catalog version and vice versa.

![Connector Operation](sync-catalog-connector.svg)

--8<-- "snippets/abbr.md"
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/sync-catalog-connector.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 6 additions & 2 deletions site/docs/connectors/unity-catalog/sync-server-connector.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,11 @@

# OSS Unity Catalog (UC) Server Synchronizing Connector

The *Unity Catalog OSS Synchronization Connector* exchanges metadata between the open-source version of Unity Catalog and the Open Metadata Ecosystem.
The *Unity Catalog OSS Synchronization Connector* is an [integration connector](/concepts/integration-connector) that exchanges metadata between the open-source version of Unity Catalog and the Open Metadata Ecosystem.

Its work is scoped by the metadata collection that represents Unity Catalog. This is identified by the *metadataSourceQualifiedName* property in the connector's configuration.
It performs two passes each time it is called.
It performs two passes each time it is called.

![Connector Operation](sync-server-connector.svg)

--8<-- "snippets/abbr.md"
4 changes: 4 additions & 0 deletions site/docs/connectors/unity-catalog/sync-server-connector.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 7d73bb2

Please sign in to comment.