Skip to content

Commit

Permalink
Merge pull request #961 from mandy-chessell/code2024
Browse files Browse the repository at this point in the history 
Intitial field mappings for Unity Catalog
  • Loading branch information
@mandy-chessell
mandy-chessell authored Oct 4, 2024
2 parents ea41a16 + 80946cb commit bffee21
Show file tree
Hide file tree
Showing 7 changed files with 193 additions and 1 deletion.
23 changes: 23 additions & 0 deletions site/docs/connectors/unity-catalog/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,29 @@ The diagram below illustrates the mapping of the Unity Catalog metadata resource

![Type Mapping](unity-catalog-type-mapping.svg)

Below is the field mapping from Unity Catalog to these Open Metadata Types:

=== "Catalogs"

--8<-- "snippets/unity-catalog/catalog-mapping.md"

=== "Schemas"

--8<-- "snippets/unity-catalog/schema-mapping.md"

=== "Volumes"

--8<-- "snippets/unity-catalog/volume-mapping.md"

=== "Tables"

--8<-- "snippets/unity-catalog/table-mapping.md"

=== "Functions"

--8<-- "snippets/unity-catalog/function-mapping.md"


The templates that implement this mapping are described in [Unity Catalog Templates](/catalog-templates/unity-catalog-templates).

### Anchor design for Unity Catalog
Expand Down
2 changes: 1 addition & 1 deletion site/docs/connectors/unity-catalog/unity-catalog-type-mapping.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
32 changes: 32 additions & 0 deletions site/snippets/unity-catalog/catalog-mapping.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->


| Unity Catalog Property | Egeria Type and Attribute | Supported in OSS Version |
|---------------------------------|---------------------------------------------------------|--------------------------|
| id | ExternalIdentifier.identifier | Yes |
| name | Catalog.name | Yes |
| comment | Catalog.description | Yes |
| owner | Ownership.owner | No |
| name | ExternalIdentifier.mappingProperties.ucCatalogName | Yes |
| created_at | ExternalIdentifier.externalInstanceCreationTime | Yes |
| created_by | ExternalIdentifier.externalInstanceCreatedBy | No |
| updated_at | ExternalIdentifier.externalInstanceLastUpdateTime | Yes |
| updated_by | ExternalIdentifier.externalInstanceLastUpdatedBy | No |
| catalog_type | PropertyFacet.properties.ucCatalogType | No |
| metastore_id | PropertyFacet.properties.ucMetastoreId | No |
| isolation_mode | PropertyFacet.properties.ucIsolationMode | No |
| accessible_in_current_workspace | PropertyFacet.properties.ucAccessibleInCurrentWorkspace | No |
| browse_only | PropertyFacet.properties.ucBrowseOnly | No |
| securable_type | PropertyFacet.properties.ucSecurableType | No |
| securable_kind | PropertyFacet.properties.ucSecurableKind | No |



| Additional Values | Egeria Type and Attribute | Supported in OSS Version |
|-----------------------------------------------|-----------------------------------------------------------|--------------------------|
| Unity Catalog Catalog: Server URL : full_name | Catalog.qualifiedName | Yes |
| Unity Catalog Catalog | Catalog.deployedImplementationType | Yes |
| Databricks Unity Catalog Catalog: Server URL | Catalog.qualifiedName | No |
| Databricks Unity Catalog Catalog | Catalog.deployedImplementationType | No |
| Server URL | ExternalIdentifier.mappingProperties.serverNetworkAddress | Yes |
35 changes: 35 additions & 0 deletions site/snippets/unity-catalog/function-mapping.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->


| Unity Catalog Property | Egeria Type and Attribute | Supported in OSS Version |
|------------------------|----------------------------------------------------|--------------------------|
| function_id | ExternalIdentifier.identifier | Yes |
| name | DeployedAPI.name | Yes |
| full_name | DeployedAPI.resourceName | Yes |
| comment | DeployedAPI.description | Yes |
| owner | Ownership.owner | No |
| data_source_format | DataAssetEncoding.encoding | Yes |
| created_at | ExternalIdentifier.externalInstanceCreationTime | Yes |
| created_by | ExternalIdentifier.externalInstanceCreatedBy | No |
| updated_at | ExternalIdentifier.externalInstanceLastUpdateTime | Yes |
| updated_by | ExternalIdentifier.externalInstanceLastUpdatedBy | No |
| catalog_name | ExternalIdentifier.mappingProperties.ucCatalogName | Yes |
| schema_name | ExternalIdentifier.mappingProperties.ucSchemaName | Yes |
| storage_location | PropertyFacet.properties.ucStorageLocation | Yes |
| table_type | PropertyFacet.properties.ucTableType | Yes |
| metastore_id | PropertyFacet.properties.ucMetastoreId | No |
| browse_only | PropertyFacet.properties.ucBrowseOnly | No |
| securable_type | PropertyFacet.properties.ucSecurableType | No |
| securable_kind | PropertyFacet.properties.ucSecurableKind | No |
| resource_name | PropertyFacet.properties.ucResourceName | No |



| Additional Values | Egeria Type and Attribute | Supported in OSS Version |
|-----------------------------------------------------------|-----------------------------------------------------------|--------------------------|
| Unity Catalog Function: Server URL : full_name | DeployedAPI.qualifiedName | Yes |
| Unity Catalog Function | DeployedAPI.deployedImplementationType | Yes |
| Databricks Unity Catalog Function: Server URL : full_name | DeployedAPI.qualifiedName | No |
| Databricks Unity Catalog Function | DeployedAPI.deployedImplementationType | No |
| Server URL | ExternalIdentifier.mappingProperties.serverNetworkAddress | Yes |
33 changes: 33 additions & 0 deletions site/snippets/unity-catalog/schema-mapping.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->


| Unity Catalog Property | Egeria Type and Attribute | Supported in OSS Version |
|------------------------|----------------------------------------------------|--------------------------|
| schema_id | ExternalIdentifier.identifier | Yes |
| name | DeployedDatabaseSchema.name | Yes |
| full_name | DeployedDatabaseSchema.resourceName | Yes |
| comment | DeployedDatabaseSchema.description | Yes |
| owner | Ownership.owner | No |
| created_at | ExternalIdentifier.externalInstanceCreationTime | Yes |
| created_by | ExternalIdentifier.externalInstanceCreatedBy | No |
| updated_at | ExternalIdentifier.externalInstanceLastUpdateTime | Yes |
| updated_by | ExternalIdentifier.externalInstanceLastUpdatedBy | No |
| catalog_name | ExternalIdentifier.mappingProperties.ucCatalogName | Yes |
| name | ExternalIdentifier.mappingProperties.ucSchemaName | Yes |
| catalog_type | PropertyFacet.properties.ucCatalogType | No |
| metastore_id | PropertyFacet.properties.ucMetastoreId | No |
| browse_only | PropertyFacet.properties.ucBrowseOnly | No |
| securable_type | PropertyFacet.properties.ucSecurableType | No |
| securable_kind | PropertyFacet.properties.ucSecurableKind | No |
| properties | PropertyFacet.properties | No |



| Additional Values | Egeria Type and Attribute | Supported in OSS Version |
|---------------------------------------------------------|-----------------------------------------------------------|--------------------------|
| Unity Catalog Schema: Server URL : full_name | DeployedDatabaseSchema.qualifiedName | Yes |
| Unity Catalog Schema | DeployedDatabaseSchema.deployedImplementationType | Yes |
| Databricks Unity Catalog Schema: Server URL : full_name | DeployedDatabaseSchema.qualifiedName | No |
| Databricks Unity Catalog Schema | DeployedDatabaseSchema.deployedImplementationType | No |
| Server URL | ExternalIdentifier.mappingProperties.serverNetworkAddress | Yes |
35 changes: 35 additions & 0 deletions site/snippets/unity-catalog/table-mapping.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->


| Unity Catalog Property | Egeria Type and Attribute | Supported in OSS Version |
|------------------------|----------------------------------------------------|--------------------------|
| table_id | ExternalIdentifier.identifier | Yes |
| name | VirtualRelationalTable.name | Yes |
| full_name | VirtualRelationalTable.resourceName | Yes |
| comment | VirtualRelationalTable.description | Yes |
| owner | Ownership.owner | No |
| data_source_format | DataAssetEncoding.encoding | Yes |
| created_at | ExternalIdentifier.externalInstanceCreationTime | Yes |
| created_by | ExternalIdentifier.externalInstanceCreatedBy | No |
| updated_at | ExternalIdentifier.externalInstanceLastUpdateTime | Yes |
| updated_by | ExternalIdentifier.externalInstanceLastUpdatedBy | No |
| catalog_name | ExternalIdentifier.mappingProperties.ucCatalogName | Yes |
| schema_name | ExternalIdentifier.mappingProperties.ucSchemaName | Yes |
| storage_location | PropertyFacet.properties.ucStorageLocation | Yes |
| table_type | PropertyFacet.properties.ucTableType | Yes |
| metastore_id | PropertyFacet.properties.ucMetastoreId | No |
| browse_only | PropertyFacet.properties.ucBrowseOnly | No |
| securable_type | PropertyFacet.properties.ucSecurableType | No |
| securable_kind | PropertyFacet.properties.ucSecurableKind | No |
| resource_name | PropertyFacet.properties.ucResourceName | No |



| Additional Values | Egeria Type and Attribute | Supported in OSS Version |
|--------------------------------------------------------|-----------------------------------------------------------|--------------------------|
| Unity Catalog Table: Server URL : full_name | VirtualRelationalTable.qualifiedName | Yes |
| Unity Catalog Table | VirtualRelationalTable.deployedImplementationType | Yes |
| Databricks Unity Catalog Table: Server URL : full_name | VirtualRelationalTable.qualifiedName | No |
| Databricks Unity Catalog Table | VirtualRelationalTable.deployedImplementationType | No |
| Server URL | ExternalIdentifier.mappingProperties.serverNetworkAddress | Yes |
34 changes: 34 additions & 0 deletions site/snippets/unity-catalog/volume-mapping.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the Egeria project. -->


| Unity Catalog Property | Egeria Type and Attribute | Supported in OSS Version |
|------------------------|--------------------------------------------------------------------------------------------|--------------------------|
| volume_id | ExternalIdentifier.identifier | Yes |
| name | FileFolder.name | Yes |
| full_name | FileFolder.resourceName | Yes |
| comment | FileFolder.description | Yes |
| storage_location | FileFolder.pathName + Endpoint.networkAddress + PropertyFacet.properties.ucStorageLocation | Yes |
| owner | Ownership.owner | No |
| created_at | ExternalIdentifier.externalInstanceCreationTime | Yes |
| created_by | ExternalIdentifier.externalInstanceCreatedBy | No |
| updated_at | ExternalIdentifier.externalInstanceLastUpdateTime | Yes |
| updated_by | ExternalIdentifier.externalInstanceLastUpdatedBy | No |
| catalog_name | ExternalIdentifier.mappingProperties.ucCatalogName | Yes |
| schema_name | ExternalIdentifier.mappingProperties.ucSchemaName | Yes |
| volume_type | PropertyFacet.properties.ucVolumeType | Yes |
| metastore_id | PropertyFacet.properties.ucMetastoreId | No |
| browse_only | PropertyFacet.properties.ucBrowseOnly | No |
| securable_type | PropertyFacet.properties.ucSecurableType | No |
| securable_kind | PropertyFacet.properties.ucSecurableKind | No |
| resource_name | PropertyFacet.properties.ucResourceName | No |



| Additional Values | Egeria Type and Attribute | Supported in OSS Version |
|---------------------------------------------------------|-----------------------------------------------------------|--------------------------|
| Unity Catalog Volume: Server URL : full_name | FileFolder.qualifiedName | Yes |
| Unity Catalog Volume | FileFolder.deployedImplementationType | Yes |
| Databricks Unity Catalog Volume: Server URL : full_name | FileFolder.qualifiedName | No |
| Databricks Unity Catalog Volume | FileFolder.deployedImplementationType | No |
| Server URL | ExternalIdentifier.mappingProperties.serverNetworkAddress | Yes |

0 comments on commit bffee21

Please sign in to comment.