HashiCorp Vault: Secrets Management from First Run to Production

Why Vault Over Environment Variables

Environment variables for secrets have a fundamental problem: they're plaintext, visible to any process running in the same environment, logged by crash reporters, and leaked into CI logs. Vault treats secrets as first-class citizens with audit logging, access policies, automatic rotation, and time-limited leases.

The core model:

• Secrets Engines — backends that store or generate secrets (KV, database, AWS, PKI)
• Auth Methods — how clients prove identity (AppRole, Kubernetes, AWS IAM, OIDC)
• Policies — HCL rules that control what an identity can read or write
• Leases — secrets have TTLs; Vault revokes them automatically when they expire

---

Install and Initialize

# Install on Linux (or use the official Docker image)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vault

# Or with Docker for local dev
docker run --cap-add=IPC_LOCK \
  -e VAULT_DEV_ROOT_TOKEN_ID=root \
  -p 8200:8200 \
  hashicorp/vault

For local development, dev mode starts Vault unsealed with a root token:

vault server -dev -dev-root-token-id=root
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'
vault status

---

KV Secrets Engine

KV (Key-Value) is the simplest engine — stores arbitrary key-value pairs. KV v2 adds versioning and soft deletes.

# Enable KV v2 at the path 'secret/'
vault secrets enable -path=secret kv-v2

# Write a secret
vault kv put secret/myapp/config \
  db_password="s3cur3p@ss" \
  api_key="abc123xyz"

# Read a secret
vault kv get secret/myapp/config

# Read a specific field
vault kv get -field=db_password secret/myapp/config

# List secrets (shows paths, not values)
vault kv list secret/myapp/

# Get a previous version
vault kv get -version=1 secret/myapp/config

# Update (creates a new version, old version preserved)
vault kv patch secret/myapp/config db_password="newp@ssword"

---

Policies: Controlling Access

Policies define what operations an identity can perform. Write them in HCL:

# myapp-policy.hcl
path "secret/data/myapp/*" {
  capabilities = ["read", "list"]
}

path "secret/metadata/myapp/*" {
  capabilities = ["read", "list"]
}

# Deny everything else
path "secret/*" {
  capabilities = ["deny"]
}

# Create the policy
vault policy write myapp-policy myapp-policy.hcl

# List policies
vault policy list

# Read a policy
vault policy read myapp-policy

Capability options: create, read, update, delete, list, deny, sudo.

For KV v2, the actual secret data lives under secret/data/... and metadata at secret/metadata/.... Both need to be addressed in policies.

---

Auth Methods

Token Auth (default)

Tokens are Vault's native auth mechanism. Every auth method ultimately returns a token. For human access:

# Create a token with a policy attached
vault token create -policy=myapp-policy -ttl=1h

# Token lookup
vault token lookup <token>

# Revoke a token
vault token revoke <token>

AppRole (for services and CI/CD)

AppRole is designed for machines and services — a role ID (like a username) and secret ID (like a password):

# Enable AppRole
vault auth enable approle

# Create a role
vault write auth/approle/role/myapp-role \
  token_policies="myapp-policy" \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_ttl=24h \
  secret_id_num_uses=10

# Get the Role ID (static — can be committed to config)
vault read auth/approle/role/myapp-role/role-id

# Generate a Secret ID (dynamic — treat like a password)
vault write -f auth/approle/role/myapp-role/secret-id

# Login with AppRole credentials
vault write auth/approle/login \
  role_id="<role-id>" \
  secret_id="<secret-id>"

The login returns a Vault token your service uses for all subsequent requests.

---

Dynamic Secrets: Database Example

Static credentials (a single password shared across services) are dangerous — one leak compromises everything, rotation breaks running services. Dynamic secrets generate unique credentials per request, with automatic expiry.

# Enable the database secrets engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  allowed_roles="myapp-role" \
  connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/mydb?sslmode=disable" \
  username="vault-admin" \
  password="admin-password"

# Create a role with a SQL template
vault write database/roles/myapp-role \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate credentials
vault read database/creds/myapp-role

This creates a unique PostgreSQL user like v-approle-myapp-role-AbCdEf-1234567890 that automatically expires in 1 hour. Vault revokes it (drops the user) when the lease expires.

---

Reading Secrets in Applications

Vault CLI in scripts

export DB_PASSWORD=$(vault kv get -field=db_password secret/myapp/config)

HTTP API directly

curl \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  http://vault.example.com:8200/v1/secret/data/myapp/config \
  | jq '.data.data.db_password'

Vault Agent (recommended for production)

Vault Agent runs as a sidecar or system daemon, handles authentication and token renewal, and writes secrets to a file or injects into templates:

# vault-agent.hcl
vault {
  address = "https://vault.example.com:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/run/secrets/role-id"
      secret_id_file_path = "/run/secrets/secret-id"
    }
  }

  sink "file" {
    config = {
      path = "/run/vault-token"
    }
  }
}

template {
  source      = "/etc/vault/templates/config.tmpl"
  destination = "/etc/myapp/config.env"
}

Config template:

{{- with secret "secret/data/myapp/config" -}}
DB_PASSWORD={{ .Data.data.db_password }}
API_KEY={{ .Data.data.api_key }}
{{- end -}}

Vault Agent renders this to /etc/myapp/config.env and re-renders whenever the secret changes.

---

Production Hardening

Use Integrated Storage (Raft)

Dev mode uses in-memory storage. For production, use Raft — Vault's built-in HA storage:

# vault.hcl
storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-2.example.com:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3.example.com:8200"
  }
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/vault/tls/vault.crt"
  tls_key_file  = "/vault/tls/vault.key"
}

api_addr     = "https://vault-1.example.com:8200"
cluster_addr = "https://vault-1.example.com:8201"
ui           = true

Auto-Unseal

By default, Vault requires operators to provide unseal keys after a restart (Shamir's secret sharing). For production, use auto-unseal with a cloud KMS:

# AWS KMS auto-unseal
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789:key/abc123"
}

Enable Audit Logging

# Log to file
vault audit enable file file_path=/var/log/vault/audit.log

# Log to syslog
vault audit enable syslog

# List audit devices
vault audit list

Audit logs record every request and response (with sensitive values hashed). If no audit device is working, Vault blocks all requests — this is by design.

Lease Management

# List leases for a path
vault list sys/leases/lookup/database/creds/myapp-role/

# Renew a lease
vault lease renew <lease-id>

# Revoke a lease immediately
vault lease revoke <lease-id>

# Revoke all leases for a path
vault lease revoke -prefix database/creds/myapp-role/

---

Common Patterns

Rotate a static secret without downtime: Write the new value as a new version in KV v2. Running services that cache the old value continue working until the cache expires. New connections get the new value.

Break-glass access: Create a time-limited admin token with vault token create -policy=root -ttl=1h -use-limit=5. Log it in the audit trail. Revoke after use.

Secrets for CI/CD: Use AppRole with secret_id_num_uses=1 — each pipeline run generates a single-use secret ID, eliminating credential reuse across runs.