Mozilla SOPS: Encrypted Secrets in Git for GitOps Workflows That Don't Leak

Secrets in Git: The Problem That Won't Go Away

In 2023, GitGuardian's State of Secrets Sprawl report found over 10 million secrets exposed in public GitHub commits. That number has grown year over year. And those are just the public repositories. The private repos are worse — teams convinced that "private" means "safe" routinely commit plaintext database credentials, API keys, and TLS certificates directly into their repositories.

I've seen it in every GitOps audit I've conducted. Teams adopt ArgoCD or Flux, commit their Kubernetes manifests to Git (which is correct), but then face the question: where do the secrets go? The three common answers are all bad:

1. Plaintext in Git — one leak, one misconfigured access policy, and everything is exposed.
2. External secrets manager only — now your Git repo isn't the single source of truth, and your GitOps model is incomplete.
3. Manual kubectl create secret — congratulations, you've abandoned GitOps entirely.

Mozilla SOPS solves this by encrypting secret values in-place while leaving the keys and structure visible. You get encrypted secrets committed to Git, versioned alongside your infrastructure, decrypted only at deployment time. Your GitOps workflow stays intact. Your secrets stay encrypted at rest.

How SOPS Works

SOPS (Secrets OPerationS) encrypts the values in structured files (YAML, JSON, ENV, INI) while leaving the keys in plaintext. This is a deliberate design choice — you can git diff a SOPS-encrypted file and see which keys changed without exposing the values.

SOPS supports multiple encryption backends:

Backend	Best For
AGE	Local development, small teams, no cloud dependency
AWS KMS	AWS-native teams, IAM-based access control
GCP KMS	GCP-native teams
Azure Key Vault	Azure-native teams
HashiCorp Vault Transit	Multi-cloud, self-hosted
PGP	Legacy workflows (AGE is preferred now)

You can combine multiple backends so that any one of them can decrypt — useful for giving both CI pipelines and developers access with different key material.

Setting Up SOPS With AGE

AGE is the simplest backend to start with. No cloud infrastructure required. No key server to manage. Just a keypair.

Install both tools:

# macOS
brew install sops age

# Linux
curl -sL https://github.com/getsops/sops/releases/download/v3.9.4/sops-v3.9.4.linux.amd64 \
  -o /usr/local/bin/sops && chmod +x /usr/local/bin/sops

curl -sL https://github.com/FiloSottile/age/releases/download/v1.2.0/age-v1.2.0-linux-amd64.tar.gz | \
  tar xz -C /usr/local/bin --strip-components=1

Generate a keypair:

age-keygen -o ~/.config/sops/age/keys.txt
# Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Create a .sops.yaml configuration file in your repo root to define encryption rules:

# .sops.yaml
creation_rules:
  - path_regex: \.enc\.yaml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p,
      age1an2n3n4example_ci_key_here

This tells SOPS to encrypt any file matching *.enc.yaml using the specified AGE public keys. Multiple keys mean multiple recipients can decrypt — add your CI system's public key here.

Encrypting Kubernetes Secrets

Start with a plaintext secret manifest:

# secrets/db-credentials.enc.yaml (before encryption)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  DB_HOST: prod-db.internal.example.com
  DB_USER: app_service
  DB_PASSWORD: "s3cure-p@ssw0rd-here"
  DB_NAME: production_app

Encrypt it:

sops --encrypt --in-place secrets/db-credentials.enc.yaml

The result looks like this — keys visible, values encrypted:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  DB_HOST: ENC[AES256_GCM,data:dGVzdA==,iv:abc123...,tag:def456...]
  DB_USER: ENC[AES256_GCM,data:eHl6,iv:ghi789...,tag:jkl012...]
  DB_PASSWORD: ENC[AES256_GCM,data:bW5v,iv:mno345...,tag:pqr678...]
  DB_NAME: ENC[AES256_GCM,data:c3R1,iv:stu901...,tag:vwx234...]
sops:
  age:
    - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        ...
        -----END AGE ENCRYPTED FILE-----
  lastmodified: "2026-03-22T10:30:00Z"
  mac: ENC[AES256_GCM,data:...]
  version: 3.9.4

This is safe to commit. The SOPS metadata block at the bottom contains the encrypted data key — only someone with the AGE private key (or KMS access) can recover the actual values.

Editing Encrypted Files

SOPS decrypts in-memory, opens your $EDITOR, and re-encrypts on save:

sops secrets/db-credentials.enc.yaml
# Opens decrypted YAML in your editor
# Re-encrypts automatically when you save and close

To view without editing:

sops --decrypt secrets/db-credentials.enc.yaml

Using AWS KMS Instead of AGE

For production environments, use a cloud KMS. The access control is stronger — decryption is tied to IAM roles, and you get audit logs for every decrypt operation.

# .sops.yaml
creation_rules:
  - path_regex: environments/production/.*\.enc\.yaml$
    kms: "arn:aws:kms:us-east-1:123456789:key/abcd-1234-efgh-5678"
  - path_regex: environments/staging/.*\.enc\.yaml$
    kms: "arn:aws:kms:us-east-1:123456789:key/staging-key-id"
  - path_regex: \.enc\.yaml$
    age: "age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p"

Rules are evaluated top-to-bottom. Production secrets use the production KMS key. Staging uses a different key. Everything else falls back to AGE. This means a staging IAM role cannot decrypt production secrets, even if someone copies the file.

ArgoCD Integration With SOPS

ArgoCD doesn't natively decrypt SOPS files. You need the KSOPS plugin or the ArgoCD Vault Plugin. Here's the KSOPS approach, which uses a Kustomize exec plugin:

Install KSOPS in your ArgoCD repo server:

# argocd/repo-server-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
  namespace: argocd
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          env:
            - name: SOPS_AGE_KEY_FILE
              value: /home/argocd/.config/sops/age/keys.txt
            - name: XDG_CONFIG_HOME
              value: /home/argocd/.config
          volumeMounts:
            - name: sops-age-key
              mountPath: /home/argocd/.config/sops/age
      volumes:
        - name: sops-age-key
          secret:
            secretName: argocd-sops-age-key

Create the AGE key secret for ArgoCD:

kubectl create secret generic argocd-sops-age-key \
  --namespace=argocd \
  --from-file=keys.txt=age-key-for-argocd.txt

Then in your application, use a Kustomize generator:

# kustomization.yaml
generators:
  - secret-generator.yaml

# secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: db-credentials-generator
files:
  - secrets/db-credentials.enc.yaml

When ArgoCD syncs, KSOPS decrypts the SOPS file and feeds the plaintext Secret to Kustomize, which applies it to the cluster. The plaintext never touches disk or Git.

Flux Integration With SOPS

Flux has native SOPS support built into the kustomize-controller. No plugins needed.

Create a decryption secret:

flux create secret age flux-sops-age \
  --namespace=flux-system \
  --age-private-key-file=age-key-for-flux.txt

Configure your Kustomization to use it:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: app-secrets
  namespace: flux-system
spec:
  interval: 10m
  path: ./secrets
  prune: true
  sourceRef:
    kind: GitRepository
    name: infrastructure
  decryption:
    provider: sops
    secretRef:
      name: flux-sops-age

Flux will automatically decrypt any SOPS-encrypted files found in the ./secrets path during reconciliation.

Pre-Commit Hook: Prevent Plaintext Commits

The biggest risk with SOPS is someone committing a secret file before encrypting it. A pre-commit hook catches this:

#!/usr/bin/env bash
# .git/hooks/pre-commit

FILES=$(git diff --cached --name-only --diff-filter=ACM | grep '\.enc\.yaml$')

for f in $FILES; do
  if ! grep -q "^sops:" "$f"; then
    echo "ERROR: $f appears to be unencrypted. Run 'sops --encrypt --in-place $f' first."
    exit 1
  fi
done

Better yet, use the pre-commit framework:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/getsops/sops
    rev: v3.9.4
    hooks:
      - id: sops-diff
        name: Detect unencrypted SOPS files
        entry: bash -c 'for f in "$@"; do grep -q "^sops:" "$f" || (echo "UNENCRYPTED: $f" && exit 1); done'
        language: system
        files: '\.enc\.yaml$'

Key Rotation

Rotate keys without re-encrypting every file manually. When you add a new recipient or remove a revoked key, update .sops.yaml and run:

# Update all encrypted files with new key configuration
find . -name "*.enc.yaml" -exec sops updatekeys {} \;

For KMS key rotation, AWS handles the cryptographic rotation automatically. SOPS re-wraps the data key with the new KMS key material on the next encrypt operation. No manual intervention needed.

What Not to Encrypt With SOPS

SOPS is for secrets that need to live alongside infrastructure code. It is not a general-purpose secrets manager. Don't use it for:

• Short-lived tokens — use your cloud provider's identity federation instead
• Secrets that change frequently — the Git churn becomes noise
• Secrets shared across many services — use a centralized vault with dynamic secrets

Use SOPS for database credentials, API keys, TLS certificates, and other long-lived secrets that need to be versioned with the infrastructure that consumes them. For everything else, pair it with HashiCorp Vault or your cloud provider's secrets manager.

Troubleshooting SOPS Decryption Failures

When SOPS can't decrypt, it fails loudly — but the error messages aren't always helpful. Here are the common failure modes and their fixes.

"could not decrypt data key" Error

This means none of the encryption backends could decrypt the data key embedded in the SOPS metadata. Causes:

# Cause 1: AGE private key not found or wrong key
# Check that SOPS_AGE_KEY_FILE points to the right file
echo $SOPS_AGE_KEY_FILE
cat ~/.config/sops/age/keys.txt | head -3

# Cause 2: KMS permissions missing
# Verify the IAM role has kms:Decrypt permission
aws kms decrypt \
  --ciphertext-blob fileb://<(echo "test" | base64 -d) \
  --key-id arn:aws:kms:us-east-1:123456789:key/abcd-1234 \
  2>&1 | head -5

# Cause 3: File was encrypted with a key you don't have
# Check which recipients can decrypt
sops --decrypt --extract '["sops"]["age"]' secrets/db-credentials.enc.yaml
# Compare the recipient public key with your key
grep "public key" ~/.config/sops/age/keys.txt

MAC Mismatch Error

SOPS stores a MAC (message authentication code) that verifies the file hasn't been tampered with. If you manually edit an encrypted file (instead of using sops to edit it), the MAC won't match:

# This will fail with MAC mismatch
vim secrets/db-credentials.enc.yaml  # DON'T DO THIS
sops --decrypt secrets/db-credentials.enc.yaml
# Error: MAC mismatch

# Fix: Re-encrypt with the --ignore-mac flag, then re-save
sops --decrypt --ignore-mac secrets/db-credentials.enc.yaml > /tmp/decrypted.yaml
sops --encrypt /tmp/decrypted.yaml > secrets/db-credentials.enc.yaml
rm /tmp/decrypted.yaml

Never leave decrypted files in temp directories on shared systems. Use sops to edit in-place whenever possible.

CI/CD Pipeline Decryption With SOPS

For automated deployments outside of ArgoCD or Flux, you need SOPS decryption in your CI pipeline. Here's a GitHub Actions workflow that decrypts and applies secrets:

# .github/workflows/deploy-secrets.yml
name: Deploy Secrets
on:
  push:
    branches: [main]
    paths: ["secrets/**"]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/github-actions-deployer
          aws-region: us-east-1

      - name: Install SOPS
        run: |
          curl -sL https://github.com/getsops/sops/releases/download/v3.9.4/sops-v3.9.4.linux.amd64 \
            -o /usr/local/bin/sops && chmod +x /usr/local/bin/sops

      - name: Decrypt and apply secrets
        run: |
          for f in secrets/*.enc.yaml; do
            echo "Applying $f"
            sops --decrypt "$f" | kubectl apply -f -
          done
        env:
          KUBECONFIG: ${{ secrets.KUBECONFIG }}

The decrypted plaintext passes through a pipe and is never written to disk or CI logs. The AWS IAM role used by the GitHub Actions runner has kms:Decrypt permission for the production KMS key, scoped to the specific key ARN.

Auditing SOPS Changes in Git

One of the biggest advantages of encrypted secrets in Git is the audit trail. Every change to a secret produces a Git commit with a diff showing which keys changed (values remain encrypted). Build on this with a script that tracks secret modifications:

#!/bin/bash
# scripts/audit-secret-changes.sh — List all secret changes in the last 30 days
git log --since="30 days ago" --all --oneline --name-only -- '*.enc.yaml' | \
  awk '/^[a-f0-9]/ {commit=$0; next} /\.enc\.yaml/ {print commit " — " $0}'

For compliance requirements, combine this with signed Git commits. If every commit that modifies a *.enc.yaml file must be signed, you have cryptographic proof of who changed which secret and when:

# Verify that all recent secret changes have signed commits
git log --since="30 days ago" --all --format="%H %G? %an %s" -- '*.enc.yaml' | \
  while read hash sig author msg; do
    if [ "$sig" != "G" ]; then
      echo "UNSIGNED: $hash by $author — $msg"
    fi
  done

If unsigned commits appear, investigate. In a mature GitOps workflow, unsigned secret modifications should trigger a security review.

The goal is straightforward: every secret in your GitOps repo is encrypted at rest, decrypted only at deploy time, and auditable through Git history. No more plaintext. No more excuses.

Related Articles

• HashiCorp Vault and Kubernetes: Secrets Management That Actually Works
• Security Headers & Configs: Cheat Sheet
• Kubernetes Security Hardening for Production: The Complete Guide