Table of Contents generated with DocToc
- Credits
- Scenarios
- Issues
- Vault client configuration
- Init Container vault-kubernetes-authenticator
- Init Container vault-kubernetes-synchronizer
- Sidecar vault-kubernetes-token-renewer
- Build
- Demo
- Links
Based on the work of Seth Vargo
Start the Init Container vault-kubernetes-authenticator to authenticate to Vault and get a Vault token.
The Vault token will expire after the given TTL.
Start the Init Container vault-kubernetes-authenticator to authenticate to Vault and get a Vault token.
After successful completion, start the Init Container vault-kubernetes-synchronizer to synchronize secrets to Kubernetes.
The Vault token will expire after the given TTL.
Start the Init Container vault-kubernetes-authenticator to authenticate to Vault and get a Vault token.
After successful completion start the Sidecar Container vault-kubernetes-token-renewer to regularly renew your Vault token.
vault-kubernetes-token-renewer container will be restarted if the token renewal fails (for restartPolicy=always). When the token cannot be renewed (e.g. the token is in the meantime expired):
- let the pod terminate and restart. On restart vault-kubernetes-authenticator will issue a new token. A possible solution could be to use Share Process Namespace between Containers in a Pod (Kubernetes 1.12 beta) and Container Lifecycle Hooks
- let vault-kubernetes-token-renewer re-authenticate and update VAULT_TOKEN_PATH if the token in VAULT_TOKEN_PATH is invalid. The token consumer needs to observe VAULT_TOKEN_PATH for changes (inotify) or read VAULT_TOKEN_PATH on every connect to Vault (isn't a thing because VAULT_TOKEN_PATH is usually in-memory). This can be done independent from the previous case because the token will be valid on after pod creation
removed go.sum from repo due to issue with go version and k8s.io/client-go:
go: verifying k8s.io/client-go@v9.0.0+incompatible: checksum mismatch
The usual environment variables for Vault will be used:
- VAULT_ADDR
- VAULT_CACERT
- VAULT_CAPATH
- VAULT_CLIENT_CERT
- VAULT_CLIENT_KEY
- VAULT_CLIENT_TIMEOUT
- VAULT_SKIP_VERIFY
- VAULT_TLS_SERVER_NAME
- VAULT_WRAP_TTL
- VAULT_MAX_RETRIES
- VAULT_TOKEN
- VAULT_MFA
- VAULT_RATE_LIMIT
see https://godoc.org/github.com/hashicorp/vault/api#Config.ReadEnvironment
the minimal configuration is VAULT_ADDR with VAULT_SKIP_VERIFY=true
- VAULT_ROLE - Required the name of the Vault role to use for authentication.
For a successful Kubernetes authentication the environment variable VAULT_ROLE must be set.
...or...
- VAULT_ROLE_ID - see https://www.vaultproject.io/docs/auth/approle
- VAULT_SECRET_ID - see https://www.vaultproject.io/docs/auth/approle
If the environment variables VAULT_ROLE_ID and VAULT_SECRET_ID are set, AppRole Auth Method will be used, Kubernetes Auth Method otherwise.
-
VAULT_TOKEN_PATH - the destination path on disk to store the token. Usually this is a shared volume.
-
VAULT_AUTH_MOUNT_PATH - the name of the mount where the Kubernetes auth method is enabled. This defaults to kubernetes, but if you changed the mount path you will need to set this value to that path (vault auth enable -path=k8s kubernetes -> VAULT_AUTH_MOUNT_PATH=k8s)
-
SERVICE_ACCOUNT_TOKEN_PATH - the path on disk where the Kubernetes service account jtw token lives. This defaults to /var/run/secrets/kubernetes.io/serviceaccount/token.
-
ALLOW_FAIL - the container will successfully terminate even if the authentication to Vault failed, no token will be written to VAULT_TOKEN_PATH. This condition needs to be handled in the succeeding container. (default: "false")
$ k logs vault-kubernetes-authenticator-5675d58d95-4wd8v -c vault-kubernetes-authenticator
2018/11/26 14:56:29 successfully authenticated to vault
2018/11/26 14:56:29 successfully stored vault token at /home/vault/.vault-token
$ k exec -ti vault-kubernetes-authenticator-5675d58d95-4wd8v sh
~ $ VAULT_TOKEN=$(cat /home/vault/.vault-token)
~ $ echo $VAULT_TOKEN
8Pj0EzFLWQv8uWcjbP9hF1MB
~ $
Depends on Init Container vault-kubernetes-authenticator
-
each Kubernetes secrets created by vault-kubernetes-synchronizer gets the annotation
vault-secret: <vault secret path> -
Existing labels are retained and configured labels are appended to existing ones. If an existing label has the same key as a configured label, the value will be overwritten.
-
obsolete secrets created by vault-kubernetes-synchronizer will be deleted
| Mapping | Vault | Kubernetes |
|---|---|---|
| secret/k8s/first | secret/k8s/first | first |
| secret/k8s/first:third | secret/k8s/first | third |
| ----------------------------- | ----------------------- | ------------- |
| secret/k8s/ | secret/k8s/first | first |
| secret/k8s/second | second |
labels/names in Kubernetes will be validated according to RFC-1123
If you have to encode the secret to put it in Vault, e.g. a Java KeyStore (JKS), then you can use base64 and add the prefix "base64:" to the secret in Vault
Create the secret for Vault as follows:
echo "base64:$(base64 -w0 filename)"
vault-kubernetes-synchronizer will decode the secret from Vault before creating a Kubernetes secret, to prevent double encoding.
-
VAULT_TOKEN_PATH - the destination path on disk to store the token. Usually this is a shared volume.
-
VAULT_SECRETS - comma separated list of secrets (see Secret Mapping)
-
SECRET_PREFIX - prefix for synchronized secrets (e.g. for SECRET_PREFIX="v3t_" Vault secret "first" will get secret "v3t_first" in k8s)
-
SYNCHRONIZER_ANNOTATION - annotation used to track managed secrets (default value
vault-secret). Can be very useful if you need more than onevault-synchronizerinit container in the same namespace. -
SYNCHRONIZER_LABELS - labels will be added to every synchronized secret. Multiple key-value pairs can be separated with a comma. For each key-value pair a key and the equal sign are mandatory. Example:
"k1=v1,k2=v2,k3=,k4"k4 will be ignored because the equal sign is missing.
set ALLOW_FAIL="true" for vault-kubernetes-authenticator
If Vault authentication fails in vault-kubernetes-authenticator and ALLOW_FAIL="true" has been set for vault-kubernetes-authenticator the failed authentication will be handled as follows:
- all secrets in VAULT_SECRETS are available in the namespace (the content of the secrets will not be considered)- vault-kubernetes-synchronizer issues a warning and terminates successfully.
- any secret from VAULT_SECRETS is missing in the namespace vault-secret-synchronizer fails.
Two secrets in Vault:
$ vault kv get secret/k8s/first
====== Metadata ======
...
=== Data ===
Key Value
--- -----
one 12345678
two 23456781
$ vault kv get secret/k8s/second
====== Metadata ======
...
===== Data =====
Key Value
--- -----
green lantern
poison ivy
Configure the two secrets for synchronization with the environment variable VAULT_SECRETS:
$ vi deployment.yaml
...
- name: VAULT_SECRETS
value: secret/data/k8s/first,secret/data/k8s/second
...
$ k logs vault-kubernetes-synchronizer-6875c88858-t6hdw -c vault-kubernetes-authenticator
2018/11/26 14:56:30 successfully authenticated to vault
2018/11/26 14:56:30 successfully stored vault token at /home/vault/.vault-token
$ k logs vault-kubernetes-synchronizer-6875c88858-t6hdw -c vault-kubernetes-synchronizer
2018/11/26 14:56:31 read secret/data/k8s-np/appl-vault-dev-e1/first from vault
2018/11/26 14:56:31 create secret third from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 14:56:31 read secret/data/k8s-np/appl-vault-dev-e1/first from vault
2018/11/26 14:56:31 create secret first from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 14:56:31 read secret/data/k8s-np/appl-vault-dev-e1/second from vault
2018/11/26 14:56:31 create secret second from vault secret secret/data/k8s-np/appl-vault-dev-e1/second
2018/11/26 14:56:31 secrets successfully synchronized
$ k get secrets | grep -e first -e second -e third
first Opaque 2 16m
second Opaque 2 16m
third Opaque 2 16m
$ k describe secrets first second third
Name: first
Namespace: vault-test
Labels: <none>
Annotations: vault-secret=secret/data/k8s/first
Type: Opaque
Data
====
one: 8 bytes
two: 8 bytes
Name: second
Namespace: vault-test
Labels: <none>
Annotations: vault-secret=secret/data/k8s/second
Type: Opaque
Data
====
poison: 3 bytes
green: 7 bytes
Name: third
Namespace: vault-test
Labels: <none>
Annotations: vault-secret=secret/data/k8s/first
Type: Opaque
Data
====
one: 8 bytes
two: 8 bytes
ALLOW_FAIL="false" set for vault-kubernetes-authenticator
$ k logs vault-kubernetes-synchronizer-6875c88858-mbdsp -c vault-kubernetes-authenticator
2018/11/26 15:26:01 authentication failed: login failed with role from environment variable VAULT_ROLE: "k8s-np-appl-vault-dev-e1-auth": Put http://vault-dev-server.appl-vault-dev-e1.svc.cluster.local:8200/v1/auth/k8s-np/login: dial tcp 10.127.21.136:8200: i/o timeout
$ k logs vault-kubernetes-synchronizer-6875c88858-mbdsp -c vault-kubernetes-synchronizer
Error from server (BadRequest): container "vault-kubernetes-synchronizer" in pod "vault-kubernetes-synchronizer-6875c88858-mbdsp" is waiting to start: PodInitializing
$ k get pods
NAME READY STATUS RESTARTS AGE
vault-kubernetes-synchronizer-6875c88858-mbdsp 0/1 Init:CrashLoopBackOff 3 7m40s
ALLOW_FAIL="true" set for vault-kubernetes-authenticator
$ k logs vault-kubernetes-synchronizer-7d5f65895-2pf4j -c vault-kubernetes-authenticator -f
2018/11/26 15:36:53 authentication failed - ALLOW_FAIL is set therefore pod will continue: login failed with role from environment variable VAULT_ROLE: "k8s-np-appl-vault-dev-e1-auth": Put http://vault-dev-server.appl-vault-dev-e1.svc.cluster.local:8200/v1/auth/k8s-np/login: dial tcp 10.127.21.136:8200: i/o timeout
$ k logs vault-kubernetes-synchronizer-7d5f65895-2pf4j -c vault-kubernetes-synchronizer
2018/11/26 15:36:55 check secret second from vault secret secret/data/k8s-np/appl-vault-dev-e1/second
2018/11/26 15:36:55 check secret third from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 15:36:55 check secret first from vault secret secret/data/k8s-np/appl-vault-dev-e1/first
2018/11/26 15:36:55 cannot synchronize secrets - all secrets seems to be available therefore pod creation will continue: could not get vault token: open /home/vault/.vault-token: no such file or directory
$ k get pods
NAME READY STATUS RESTARTS AGE
vault-kubernetes-synchronizer-7d5f65895-2pf4j 1/1 Running 0 5m18s
$ k logs pod/vault-kubernetes-synchronizer-demo-vvzxr -c vault-kubernetes-synchronizer
2019/08/20 14:00:28 Using annotation [ vault-secret ] to detect managed secrets
2019/08/20 14:00:28 failed to prepare synchronization of secrets: Error making API request.
URL: GET http://example.com/v1/sys/mounts
Code: 403. Errors:
* 1 error occurred:
* permission denied
The fix for this is to add read permission to the read permission in the sys/mounts for the SA.
path "sys/mounts" {
capabilities = ["read"]
}
$ k logs pod/vault-kubernetes-synchronizer-sd-67fb88c95b-d7pkb -c vault-kubernetes-authenticator
2019/09/06 08:58:55 successfully authenticated to vault
2019/09/06 08:58:55 successfully stored vault token at /home/vault/.vault-token
$ k logs pod/vault-kubernetes-synchronizer-sd-67fb88c95b-d7pkb -c vault-kubernetes-synchronizer
2019/09/06 09:00:40 Using annotation [ vault-secret ] to detect managed secrets
2019/09/06 09:00:40 failed to prepare synchronization of secrets: strconv.Atoi: parsing "": invalid syntax
The reason for the above error is no versioning enabled for the kv secret engine. The version(1/2) has to be enabled & leaving it blank will cause above issue. Please follow the steps mentioned to fix it.
$ vault secrets list -detailed
Path Plugin Accessor Default TTL Max TTL Force No Cache Replication Seal Wrap Options Description UUID
---- ------ -------- ----------- ------- -------------- ----------- --------- ------- ----------- ----
secret/ kv kv_8210532d system system false replicated false map[] n/a 1dd5df15-8178-7843-6795-f05def3c3db8
$ vault secrets enable -version=1 kv
Success! Enabled the kv secrets engine at: kv/
$ vault kv enable-versioning secret/
Success! Tuned the secrets engine at: secret/
$ vault secrets list -detailed | grep kv
kv/ kv kv_894f5894 system system false replicated false map[version:1] n/a f0736f4d-343d-e32a-b2c5-897bf3552f1f
secret/ kv kv_8210532d system system false replicated false map[version:2] n/a 1dd5df15-8178-7843-6795-f05def3c3db8
Initial synchronized secrets: $ k get secrets | grep ^vault- | grep -v token vault-alpha Opaque 1 26m vault-beta Opaque 1 26m vault-first Opaque 2 26m vault-gamma Opaque 1 26m vault-second Opaque 2 26m vault-third Opaque 2 26m
Add labels for some secrets:
$ for i in alpha beta gamma; do printf "labels of secret %12s: %s\n" vault-$i $(k get secret vault-${i} -o=jsonpath="{.metadata['labels']}"); done
labels of secret vault-alpha: {"batman":"unknown","jocker":"jack_napier","superman":"unknown"}
labels of secret vault-beta: {"batman":"bruce_wayne","joker":"jack_napier"}
labels of secret vault-gamma: {"superman":"kal-el"}
Add SYNCHRONIZER_LABELS to your deployment:
$ vi deployment.yaml
...
- name: SYNCHRONIZER_LABELS
value: batman=bruce_wayne,superman=kal-el
...
> All synchronized secrets will get these labels.
Redeploy and check the labels:
$ for i in alpha beta gamma; do printf "labels of secret %12s: %s\n" vault-$i
> Existing labels are retained or overwritten.
## Example - Custom annotation
Set our custom annotation:
$ vi deployment.yaml ... - name: SYNCHRONIZER_ANNOTATION value: synchronized ...
Deploy and check the annotations:
$ for i in alpha beta gamma; do printf "annotations of secret %12s: %s\n" vault-$i
Change your custom annotation:
$ vi deployment.yaml ... - name: SYNCHRONIZER_ANNOTATION value: vault-kubernetes-synchronizer ...
Deploy and check the logs of your vault-kubernetes-synchronizer pod:
2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/greek/alpha from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-alpha - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/greek/beta from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-beta - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/greek/gamma from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-gamma - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/first from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-first - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/second from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-second - not managed by synchronizer 2021/01/25 11:19:33 read secret/e1-k8s-pfnet-a/scratch-sauterm/first from vault 2021/01/25 11:19:33 WARNING: ignoring secret vault-third - not managed by synchronizer
> Changing the annotation does not work. You have to delete the secrets first.
# Sidecar _vault-kubernetes-token-renewer_
Depends on Init Container _vault-kubernetes-authenticator_
- renew the Vault token regularly
## Configuration
- VAULT_TOKEN_PATH - the destination path on disk to store the token. Usually this is a shared volume.
- VAULT_REAUTH - re-authenticate if the token is invalid (default: "false")
- VAULT_TTL - requested token ttl (can be overwritten by Vault)
> If you set VAULT_REAUTH to "true", you have to provide all necessary environment variable for authentication (see: _vault-kubernetes-authenticator_). The token changes when re-authentication happens and must therefore be read again.
## Example
$ k logs vault-kubernetes-token-renewer-844488f7bc-c6ztf -c vault-kubernetes-authenticator 2018/11/26 14:56:30 successfully authenticated to vault 2018/11/26 14:56:30 successfully stored vault token at /home/vault/.vault-token
$ k logs vault-kubernetes-token-renewer-844488f7bc-c6ztf -c vault-kubernetes-token-renewer 2018/11/26 14:56:32 start renewer loop 2018/11/26 14:56:32 token renewed
# Build
Install [mage](https://magefile.org/)
> The `DOCKER_TARGET` environment variable will be used to tag and push the images. If not set, the images will not be tagged and pushed.
$ export GO111MODULE=on $ export DOCKER_TARGET="registry.example.com/repopath" $ mage buildAllImages
# Demo
- Edit `profile`
cd demo ./deploy.sh profile ... ./delete.sh namespace
# Links
- [Using HashiCorp Vault with Kubernetes (Cloud Next '18)](https://www.youtube.com/watch?v=B16YTeSs1hI)
- [Github - vault-kubernetes-authenticator](https://github.com/sethvargo/vault-kubernetes-authenticator)
- [Vault - Kubernetes Auth Method](https://www.vaultproject.io/docs/auth/kubernetes.html)
- [Kubernetes - Init Containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers)