Using this template, you can deploy all the necessary resources in order to have a simple Databricks AWS workspace with Unity Catalog enabled.
This is a one apply template, you will create the base aws resources for a workspace (VPC, subnets, VPC endpoints, S3 Bucket and cross account IAM role) and the unity catalog metastore and cross account role.
In order to run this template, you need to have an account admin identity, preferably with a service principal. Running with a user account also works, but one should not include the account owner in the terraform UC admin or databricks users list as you cannot destroy yourself from the admin list.
When running tf configs for UC resources, due to sometimes requires a few minutes to be ready and you may encounter errors along the way, so you can either wait for the UI to be updated before you apply and patch the next changes; or specifically add depends_on to account level resources. We tried to add the necessary wait times but should you encounter an error just apply again and you should be good to go.
Step 1: Fill in values in
terraform.tfvars; also configure env necessary variables for AWS provider authentication.
Step 2: Run
terraform initandterraform applyto deploy the resources. This will deploy both AWS resources that Unity Catalog requires and Databricks Account Level resources.
| Name | Version |
|---|---|
| aws | ~> 5.0 |
| databricks | >= 1.2.0, < 2.0.0 |
| random | =3.4.1 |
| time | =0.9.1 |
| Name | Version |
|---|---|
| aws | ~> 5.0 |
| databricks.mws | >= 1.2.0, < 2.0.0 |
| databricks.workspace | >= 1.2.0, < 2.0.0 |
| random | =3.4.1 |
| time | =0.9.1 |
| Name | Type |
|---|---|
| databricks_catalog.demo_catalog | resource |
| databricks_cluster.unity_catalog_cluster | resource |
| databricks_grants.unity_catalog_grants | resource |
| databricks_group.admin_group | resource |
| databricks_group.users | resource |
| databricks_group_member.admin_group_member | resource |
| databricks_group_member.my_service_principal | resource |
| databricks_group_member.users_group_members | resource |
| databricks_mws_permission_assignment.add_admin_group | resource |
| databricks_mws_permission_assignment.add_user_group | resource |
| databricks_user.unity_users | resource |
| random_string.naming | resource |
| time_sleep.wait_for_permission_apis | resource |
| aws_caller_identity.current | data source |
| databricks_node_type.smallest | data source |
| databricks_service_principal.admin_service_principal | data source |
| databricks_spark_version.latest_version | data source |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| aws_access_services_role_name | (Optional) Name for the AWS Services role by this module | string |
null |
no |
| aws_profile | (Required) AWS cli profile to be used for authentication with AWS | string |
n/a | yes |
| cidr_block | (Required) CIDR block to be used to create the Databricks VPC | string |
n/a | yes |
| databricks_account_id | (Required) Databricks Account ID | string |
n/a | yes |
| databricks_client_id | (Required) Client ID to authenticate the Databricks provider at the account level | string |
n/a | yes |
| databricks_client_secret | (Required) Client secret to authenticate the Databricks provider at the account level | string |
n/a | yes |
| databricks_metastore_admins | List of Admins to be added at account-level for Unity Catalog. Enter with square brackets and double quotes e.g ["first.admin@domain.com", "second.admin@domain.com"] |
list(string) |
n/a | yes |
| databricks_users | List of Databricks users to be added at account-level for Unity Catalog. Enter with square brackets and double quotes e.g ["first.last@domain.com", "second.last@domain.com"] |
list(string) |
n/a | yes |
| my_username | (Required) Username in the form of an email to be added to the tags and be declared as owner of the assets | string |
n/a | yes |
| region | (Required) AWS region where the assets will be deployed | string |
n/a | yes |
| tags | (Optional) List of tags to be propagated accross all assets in this demo | map(string) |
n/a | yes |
| unity_admin_group | (Required) Name of the admin group. This group will be set as the owner of the Unity Catalog metastore | string |
n/a | yes |
| workspace_name | (Required) Databricks workspace name to be used for deployment | string |
n/a | yes |
| Name | Description |
|---|---|
| databricks_workspace_id | Databricks workspace ID |
| databricks_workspace_url | Databricks workspace URL |