Skip to content

Latest commit

 

History

History
432 lines (357 loc) · 18.2 KB

GettingStarted.md

File metadata and controls

432 lines (357 loc) · 18.2 KB
Raw

Secrets Store CSI Driver Provider for OCI Vault

Provider for OCI Vault allows you to get secrets stored in OCI Vault and mount them into Kubernetes pods via the Secrets Store CSI driver.

The provider is a gRPC server accessible via the Unix domain socket. It's interface is defined by the Secrets Store CSI driver. Secrets Store CSI Driver requests the provider's API in order to mount secrets onto the pods.

Table of Contents

End User Usage

This section describes steps to deploy and test solution.

Prerequisites

General

Authentication and Authorization

Currently, three modes of authentication is supported. Some AuthN modes are applicable only for a particular variant of cluster.

User Principal

Prepare user principal configuration to access OCI API as per sample template provided in deploy/example/user-auth-config-example.yaml

Refer these documents to understand the properties: ** Required Keys and OCIDs ** SDK Configuration File.

Create a secret in your cluster and in the same namespace of your workload/application

kubectl create secret generic oci-config \
         --from-file=config=user-auth-config-example.yaml \
         --from-file=private-key=./oci/oci_api_key.pem \
         --namespace <workload-namespace>

Instance Principal

Instance principal would work only on OKE cluster. Access should be granted using Access Policies(See Access Policies section).

Workload Identity

Workload Identity works only in OKE Enhanced clusters.

Access should be granted using Access Policies(See Access Policies for Workloads section).

Workload Identity uses a Resource Principal auth, which requires settings a couple of ENV variables on the provider pod, including the region where the cluster is deployed. To achieve this, make sure to specify the provider.oci.auth.types.workload.resourcePrincipalVersion=<version> and provider.oci.auth.types.workload.resourcePrincipalRegion=<region> parameters in the values.yaml for the Helm chart deployment, or as inline parameters.

Access Policies

Access to the vault and secrets should be explicity granted using Policies in case of Instance principal authencation or other users(non owner of vault) or groups of tenancy in case of user principal authentication.

It involves two steps

  • Identification of grantee

    Grantee can be a user, group or dynamic group.
    user and group can be created in a tenancy and have same(static) name throughout its life.
    Dynamic group can hold references to dynamic entities like instances whose name isn't static and assigned at runtime.

    For example, define a dynamic group with matching rules referring to all instances of a compartment.

    Any {instance.compartment.id = 'ocid1.compartment.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'}

    More information on Dynamic groups

  • Creating a policy to grant access

    allow user|group|dynamic-group <username|groupname|dynamic-group-name> to use secret-family in compartment <compartment-name>

    Policy scope can be broadened to Tenancy or restricted to a particular vault as shown below:

    allow dynamic-group dg-name to use secret-family in tenancy

    allow dynamic-group dg-name to use secret-family in compartment c1 where target.vault.id = 'ocid1.vault.oc1..aaaaaaaaaaaaaaa'

    More information on Policy

Access Policies for Workloads

With Workload Identity authentication, only a policy is required, which defines the kubernetes workload the policy works for:

allow any-user to use secret-family in compartment <compartment-name> where ALL {request.principal.type='workload', request.principal.namespace ='<namespace>', request.principal.service_account = 'oci-secrets-store-csi-driver-provider-sa', request.principal.cluster_id = 'ocid1.cluster.oc1....'}

Deployment

Provider and Driver would be deployed as Daemonset. kube-system namespace is preferred, but not restricted.

Provider can be deployed in two ways

Helm

helm repo add oci-provider https://oracle.github.io/oci-secrets-store-csi-driver-provider/charts
helm install oci-provider oci-provider/oci-secrets-store-csi-driver-provider --namespace kube-system

From Code Repository

helm upgrade --install  oci-provider -n kube-system charts/oci-secrets-store-csi-driver-provider

Default values are provided in charts/oci-secrets-store-csi-driver-provider/values.yaml file, can be overridden as per the requirement.

Deployment using yamls

  • Deploy Driver manifests
  • Deploy Provider 
    kubectl apply -f deploy/provider.daemonset.yaml
kubectl apply -f deploy/provider.serviceaccount.yaml

# if user authentication principal is required
kubectl apply -f deploy/provider.roles.yaml

Verification

Verify that provider and driver pods are up and running on each node in the cluster

Note: Use the correct namespace

kubectl get pods --namespace <namespace> --selector='app.kubernetes.io/name in (oci-secrets-store-csi-driver-provider, secrets-store-csi-driver)'

Workload/Application Deployment

It involves deployment of two resources

SecretProviderClass(SPC)

SecretProviderClass is a kind of link between volume and concrete provider. Basically, it contains:

  1. Name of the provider used to retrieve secrets (spec.provider field).
  2. Enumeration of secrets to mount in a single volume (spec.parameters.secrets field).
  3. OCI VaultId (spec.parameters.vaultId field).
  4. Authentication type used to connect to the OCI Vault (spec.parameters.authType field).
  5. Kubernetes Secret holding user principal auth config in case of user auth principal (spec.parameters.authSecretName field) SecretProviderClass is custom K8S resource provided by Secrets Store CSI driver. It's definition is created as part of driver deployment.

Check the Usage page from official docs to learn more about SecretProviderClass.

SecretProviderClass structure

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-test-spc
spec:
  provider: oci 
  parameters: # provider-specific parameters
    secrets: |
      - name: secret1              # Name of the secret in vault
        stage: PREVIOUS
      - name: secret2
        versionNumber: 1           # Version of the secret
        fileName: app1-db-password # Secret will be mounted with this name instead of secret name
    authType: instance             # possible values are: user, instance
    authSecretName: oci-config  # required only for user authType
    vaultId: ocid1.vault.oc1..aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Refer to the sample file provided at deploy/example/secret-provider-class.yaml

Let's describe the provider-specific parameters section:

  1. Field secrets contains an array of secrets to mount.

  2. Each secret represents OCI secret bundle and could have the next attributes:

    1. name - a user-friendly name for the secret. Secret names are unique within a vault. Secret names are case-sensitive.
    2. stage - the rotation state of the secret version. Allowed values are:
      • CURRENT
      • PENDING
      • LATEST
      • PREVIOUS
      • DEPRECATED
    3. versionNumber - the version number of the secret. Should be a positive number.

    Read OCI Secret Versions and Rotation States for more information about versions and stages.

  3. Each secret could be identified with:

    • name and stage
    • name and versionNumber
    • single attribute name (in this case, the default stage CURRENT is used for identification)

  4. fileName - a user-friendly name for a secret. The secret will be mounted with fileName name instead of secret name.

Workload Deployment

  • Define a volume specifying secret store driver(secrets-store.csi.k8s.io) and configure secretProviderClass property referring to the SPC name as shown below

    volumes:
- name: secrets-store-inline
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: my-test-spc

  • Mount the volume onto container

    volumeMounts:
   - name: secrets-store-inline
     mountPath: '/mnt/secrets-store'

Refer to the sample file provided at deploy/example/app-deployment.yaml

More information is available at Usage : https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html

App Verification

Deploy an app with secrets mounted from OCI Vault(Assuming the default namespace)

  1. Create a secret with correct values for user config. Edit deploy/example/user-auth-config-example.yaml file. 
     kubectl create secret generic oci-config \
    --from-file=config=./deploy/example/user-auth-config-example.yaml
  2. Adjust SecretProviderClass to enumerate there secrets stored in particular OCI Vault. Edit deploy/example/secret-provider-class.yaml file. Create example SecretProviderClass with enumerated secrets. 
    kubectl apply -f deploy/example/secret-provider-class.yaml
  3. Create an example app with secrets mounted via SecretProviderClass created above. 
    kubectl apply -f deploy/example/app.deployment.yaml
  4. Step into the app pod and verify secrets. 
    kubectl exec -ti deployment.apps/nginx  -- sh;
ls /mnt/secrets-store/;
# let's assume secrets 'foo' and 'hello' were mounted
cat /mnt/secrets-store/foo;
cat /mnt/secrets-store/hello;

Cleanup

Execute the following to clean up the environment after testing completed:

  • Uninstall app specific resources 
    kubectl delete -f deploy/example/app.deployment.yaml;
kubectl delete -f deploy/example/secret-provider-class.yaml;
# for user principal deployment:
kubectl delete secret oci-config;
  • Uninstall driver and provider specific resources 
    # For Helm based deployment
helm uninstall oci-provider --namespace kube-system;

# For Yaml based deployment
TBD

Logging

No adapters are provided and defaulting to the node logging mechanism.

Additional Features

Secrets Sync

Driver provides Sync as Kubernetes Secret feature. It allows the driver to create a Kubernetes Secret to mirror the mounted content. For example, this feature might be useful for injecting secrets as an environment variables.

Another usecase could be for mounting certificates in the Ingress controller.

By default, the driver has no permission to create K8S secrets. So, to enable secrets sync set Helm value secrets-store-csi-driver.syncSecret.enabled to true. This would enable K8S RBAC role and binding for the driver to allow operations on secrets.

Auto Rotation of Secrets

Driver provides Auto Rotation of mounted contents and synced Kubernetes secret feature. It allows the driver to update mounted secrets as well as Kubernetes secrets periodically in sync with OCI Vault data at the configured rotation frequency.

By default, the driver doesn't enable auto rotation feature. So, to enable set Helm value secrets-store-csi-driver.enableSecretRotation to true.

The default rotation frequency is 2 minutes. To use custom value, set Helm value secrets-store-csi-driver.rotationPollInterval to some permitted value.

For driver official documentation.

Developer Zone or Custom Build

Build Docker image

docker build -t oci-secrets-store-csi-driver-provider -f build/Dockerfile .

For Mac ARM64, to build for linux/amd64

docker buildx build -t --platform=linux/amd64 oci-secrets-store-csi-driver-provider -f build/Dockerfile .

Dependency management

Module vendoring is used to manage 3d-party modules in the project. vendor/ folder contains all 3d-party modules. All changes to those modules should be reflected in the remote VCS repository.

How to introduce new modules or upgrade existing ones?

  1. Once new modules was added or updated, the next command should be executed: 
    go mod vendor
    This command will update sources for that module in vendor/ folder.
  2. Then commit those changes.
  3. Note: When 'sigs.k8s.io/secrets-store-csi-driver' is being upgraded, please make sure you upgrade version for secrets-store-csi-driver in the Chart.yaml

Versioning

Each build publishes 2 artifacts: Docker image and Helm chart. Both of these artifacts use SemVer 2.0.0 for versioning.

That means that developers must increment both Docker image and Helm chart versions, otherwise, the build will fail:

  • Specify the same version in appVersion field in charts/oci-secrets-store-csi-driver-provider/Chart.yaml file;
  • Bump version field in charts/oci-secrets-store-csi-driver-provider/Chart.yaml file.

Note that Docker image version and Helm chart version are independent.

Linter

golangci-lint is used for linting. It is a standalone aggregator for Go linters. Here is the tool's documentation.

Since this tool is standalone, the developers have to control the version themselves.

NOTE: Current version is 1.46.2

CI Setup

GitHub Actions is used to implement Continuous Integration pipeline. Location in the code base: .github/workflows Github workflows:

  1. unit-tests.yaml – Runs unit test cases

  • Functionality:

    • builds binary
    • run unit tests and test coverage reports
    • send report to coveralls

  • triggers:

    • On pushing a commit

  • dependencies:

    • None
  1. build-n-push.yaml – builds and pushes image to image registry
  • Functionality:
    • builds docker image
    • pushes to registry
  • triggers:
    • on workflow_call from e2e tests and release workflows
  • dependencies:
    • unit-tests.yaml
  1. e2e-tests.yaml – Runs end to end test cases
  • Functionality:
    • Creates cluster
    • Creates Vault and Secrets
    • Deploys the provider and sample workload
    • Tests mounted contents with in a workload pod
    • Cleans up created resources
  • triggers:
    • on pull request
  • dependencies:
    • unit-tests.yaml
    • build-n-push.yaml
  • flow:

E2E Pipeline

  1. release.yaml – Release
  • Functionality:
    • Tags the docker image with release version
    • Releases helm charts
  • triggers:
    • on creating a release tag
  • dependencies:
    • unit-tests.yaml
    • build-n-push.yaml
  • flow:

Release Pipeline

Known Issues

FAQ