To set up your databricks cluster to reference Key Vault-backed secrets, follow the steps below:
- Set up an Azure Key Vault-backed secret scope.
- Reference your Key Vault-backed keys in your cluster's environment variables:
LOG_ANALYTICS_WORKSPACE_ID={{secrets/<secret-scope-name>/<keyvault-key-name>}}
LOG_ANALYTICS_WORKSPACE_KEY={{secrets/<secret-scope-name>/<keyvault-key-name>}}
- Edit the spark-monitoring.sh initialization script to reference these values:
LOG_ANALYTICS_WORKSPACE_ID=$LOG_ANALYTICS_WORKSPACE_ID
LOG_ANALYTICS_WORKSPACE_KEY=$LOG_ANALYTICS_WORKSPACE_KEY