Vault Kubernetes Authentication: Injecting Secrets into Pods Without Hardcoding

The Kubernetes Auth Problem

Kubernetes pods need secrets, but you can't put secrets in pod specs — they'd be visible in kubectl describe pod and version control. Kubernetes Secrets are base64-encoded, not encrypted at rest by default, and visible to anyone with kubectl get secret.

Vault's Kubernetes auth method lets pods authenticate using their ServiceAccount JWT tokens — something Kubernetes already provides. The pod proves its identity, Vault validates it against the Kubernetes API, and returns a Vault token scoped to the pod's policies.

---

How Kubernetes Auth Works

1. Pod starts. Kubernetes mounts a ServiceAccount JWT at /var/run/secrets/kubernetes.io/serviceaccount/token.
2. Pod (or sidecar agent) sends the JWT to Vault: POST /v1/auth/kubernetes/login.
3. Vault calls the Kubernetes API (TokenReview) to verify the JWT.
4. Vault checks: does this ServiceAccount name / namespace / pod combination match a configured role?
5. If yes, Vault returns a token with the role's policies attached.

---

Configure Vault for Kubernetes Auth

Step 1: Enable the auth method

vault auth enable kubernetes

Step 2: Configure it to talk to your cluster

Option A: Vault runs inside the same cluster

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc.cluster.local:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

Option B: Vault runs outside the cluster

# Get cluster CA cert
kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > cluster-ca.crt

# Create a ServiceAccount for Vault to use for token reviews
kubectl create serviceaccount vault-auth -n kube-system
kubectl create clusterrolebinding vault-auth-binding \
  --clusterrole=system:auth-delegator \
  --serviceaccount=kube-system:vault-auth

# Get its token (Kubernetes 1.24+ requires explicit token creation)
kubectl create token vault-auth -n kube-system --duration=87600h > reviewer-token.txt

vault write auth/kubernetes/config \
  kubernetes_host="https://api.example.com:6443" \
  kubernetes_ca_cert=@cluster-ca.crt \
  token_reviewer_jwt="$(cat reviewer-token.txt)"

Step 3: Create Vault roles

A Vault role maps a Kubernetes ServiceAccount to one or more Vault policies:

vault write auth/kubernetes/role/myapp \
  bound_service_account_names="myapp-sa" \
  bound_service_account_namespaces="production" \
  policies="myapp-policy" \
  ttl=1h

This says: "Any pod running with ServiceAccount myapp-sa in namespace production gets a token with myapp-policy."

Multiple namespaces and wildcards are supported:

vault write auth/kubernetes/role/myapp \
  bound_service_account_names="myapp-sa" \
  bound_service_account_namespaces="production,staging" \
  policies="myapp-policy" \
  ttl=1h

# Or allow any namespace (careful with this)
vault write auth/kubernetes/role/myapp \
  bound_service_account_names="myapp-sa" \
  bound_service_account_namespaces="*" \
  policies="myapp-policy" \
  ttl=1h

---

Kubernetes ServiceAccount Setup

Create the ServiceAccount your pods will use:

# myapp-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: production

kubectl apply -f myapp-sa.yaml

Reference it in your Deployment:

spec:
  serviceAccountName: myapp-sa
  # ...

---

Option 1: Vault Agent Sidecar (Vault Injector)

The Vault Agent Injector is a mutating admission webhook that automatically injects a Vault Agent sidecar into pods annotated with vault.hashicorp.com/agent-inject: "true".

Install the injector:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --set "injector.enabled=true" \
  --set "server.enabled=false" \
  --set "injector.externalVaultAddr=https://vault.example.com:8200"

Annotate your pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: production
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp"

        # Inject secret as a file
        vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"

        # Optional: template to format the secret file
        vault.hashicorp.com/agent-inject-template-config: |
          {{- with secret "secret/data/myapp/config" -}}
          export DB_PASSWORD="{{ .Data.data.db_password }}"
          export API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: myapp
          image: my-registry/myapp:latest
          command: ["/bin/sh", "-c"]
          args: ["source /vault/secrets/config && exec myapp-binary"]

The injector adds two init containers and one sidecar:

1. vault-agent-init: Authenticates and writes secrets before your app starts
2. vault-agent: Runs alongside your app, renews tokens, re-renders secrets on rotation

Secrets appear at /vault/secrets/<secret-name> inside all containers.

---

Option 2: Vault Secrets Operator (Recommended for Kubernetes-Native)

The Vault Secrets Operator (VSO) is newer and more Kubernetes-native. It syncs Vault secrets into Kubernetes Secrets, which your pods consume normally.

Install VSO:

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace \
  --set defaultVaultConnection.enabled=true \
  --set defaultVaultConnection.address=https://vault.example.com:8200

Configure a VaultAuth resource:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: myapp-auth
  namespace: production
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: myapp
    serviceAccount: myapp-sa

Create a VaultStaticSecret:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: myapp-config
  namespace: production
spec:
  type: kv-v2
  mount: secret
  path: myapp/config
  destination:
    name: myapp-secret       # The Kubernetes Secret to create/update
    create: true
  refreshAfter: 60s           # Re-sync every 60 seconds
  vaultAuthRef: myapp-auth

VSO creates and keeps a Kubernetes Secret named myapp-secret in sync with Vault. Use it in pods like any other Kubernetes Secret:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-secret
        key: db_password

For dynamic secrets (database credentials), use VaultDynamicSecret:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: myapp-db-creds
  namespace: production
spec:
  mount: database
  path: creds/myapp-role
  destination:
    name: myapp-db-secret
    create: true
  renewalPercent: 67          # Renew at 67% of TTL
  vaultAuthRef: myapp-auth

VSO automatically renews the lease before it expires and updates the Kubernetes Secret. Pods mounting the secret see the fresh credentials without restarting.

---

Testing the Authentication

Verify auth works from inside a pod:

# Exec into a pod with the right ServiceAccount
kubectl exec -it myapp-pod -n production -- /bin/sh

# Try to authenticate manually
JWT=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -s \
  --request POST \
  --data "{\"jwt\": \"$JWT\", \"role\": \"myapp\"}" \
  https://vault.example.com:8200/v1/auth/kubernetes/login \
  | jq '.auth.client_token'

# Use the returned token to read a secret
VAULT_TOKEN="<token-from-above>"
curl -s \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  https://vault.example.com:8200/v1/secret/data/myapp/config \
  | jq '.data.data'

---

Common Errors

permission denied on login — The ServiceAccount name or namespace doesn't match the Vault role's bound_service_account_names or bound_service_account_namespaces.

invalid JWT — Vault can't verify the token. Check kubernetes_ca_cert and token_reviewer_jwt in the Vault config. The token reviewer JWT may have expired.

connection refused to Vault — The pod can't reach Vault's address. Check network policies, service mesh mTLS rules, and DNS resolution inside the cluster.

Secrets file empty — The Vault policy doesn't allow read on the KV v2 path. Remember KV v2 data lives at secret/data/..., not secret/....

Token renewal failing with sidecar — Increase the token TTL (ttl= in the role) or check that the sidecar has network access to Vault.