accessing secrets from hashicorp vault are getting "missing client token" errors

[lvlcalebf]

I spun up a new kops 1.21.1 kubernetes cluster to test the CSI functionality in the vault helm chart so I could follow this and was able to successfully get that working thanks to this. I like the idea of being able to sync vault secrets into kubernetes secrets better than what the vault CSI functionality provides so I'm trying to get that working but I'm getting a 'missing client token' error when I try to use an ExternalSecret

{"level":20,"message_time":"2021-08-27T15:12:00.444Z","pid":19,"hostname":"kes-kubernetes-external-secrets-6fb9d4d77d-fmmb8","msg":"fetching new token from vault for role kes on test-kes"}
{"level":50,"message_time":"2021-08-27T15:12:00.448Z","pid":19,"hostname":"kes-kubernetes-external-secrets-6fb9d4d77d-fmmb8","payload":{"response":{"statusCode":400,"body":{"errors":["missing client token"]}}},"msg":"failure while polling the secret default/hello-vault-service"}
{"level":20,"message_time":"2021-08-27T15:12:00.448Z","pid":19,"hostname":"kes-kubernetes-external-secrets-6fb9d4d77d-fmmb8","msg":"updating status for default/hello-vault-service to: ERROR, missing client token"}

Vault and KES(kubernetes-external-secrets) are both in the default namespace, I've created a role for KES

vault write auth/kubernetes/role/kes \
  bound_service_account_names=kes-kubernetes-external-secrets \
  bound_service_account_namespaces=default \
  policies=internal-app ttl=20m

and verified the clusterrolebindings are correct. I've also created a debug alpine container with curl installed and verified the jwt (/var/run/secrets/kubernetes.io/serviceaccount/token) from the KES pod can successfully log in

curl --request POST --data '{"jwt": "<redacted>","role": "kes"}' http://vault.default:8200/v1/auth/kubernetes/login
{"request_id":"6ae748d0-3bc8-c8a2-4b23-80fb287c1ec0","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"<redacted>","accessor":"<redacted>","policies":["default","internal-app"],"token_policies":["default","internal-app"],"metadata":{"role":"kes","service_account_name":"kes-kubernetes-external-secrets","service_account_namespace":"default","service_account_secret_name":"","service_account_uid":"a25a0a97-eaf8-49bc-bbcb-aa78cdae9ee0"},"lease_duration":1200,"renewable":true,"entity_id":"e63069ec-7e56-92e9-3f7e-130377a9c175","token_type":"service","orphan":true}}

Also, If I reconfigure my vault CSI test to use the KES service account and role it works. Here's some additional info on my environment:

Vault helm install command

helm install vault hashicorp/vault \
    --set "server.dev.enabled=true" \
    --set "injector.enabled=false" \
    --set "csi.enabled=true"

Vault version

/ $ vault version
Vault v1.8.0 (82a99f14eb6133f99a975e653d4dac21c17505c7)

Vault secret I'm trying to access

vault kv put secret/db-pass password="db-secret-password"

Vault kubernetes auth config

vault write auth/kubernetes/config \
    issuer="https://kubernetes.default.svc.cluster.local" \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    disable_iss_validation=true

KES helm install command

helm install kes external-secrets/kubernetes-external-secrets --set env.VAULT_ADDR=http://vault.default:8200

ExternalSecret

cat > external-secret.yaml <<EOF    
apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
  name: hello-vault-service
spec:
  backendType: vault
  vaultMountPoint: test-kes
  vaultRole: kes
  data:
    - name: password
      key: secret/data/db-pass
      property: password
EOF

What am I missing?