Container Supply Chain Security With Sigstore and Cosign

You're Running Code You Haven't Verified

In December 2020, the SolarWinds attack demonstrated what a software supply chain compromise looks like at scale — 18,000 organizations running backdoored updates they trusted implicitly. In 2021, the Codecov bash uploader was modified to exfiltrate CI environment variables (including secrets and tokens) from thousands of pipelines. In 2024, the xz-utils backdoor (CVE-2024-3094) nearly compromised every Linux distribution's SSH daemon through a years-long social engineering campaign.

The pattern is clear. Attackers aren't just targeting your application code — they're targeting the supply chain: base images, build tools, dependencies, and the pipelines that assemble them.

If you're pulling container images and running them without verifying who built them, when they were built, and whether they've been tampered with, you're operating on blind trust. Sigstore and Cosign exist to replace that trust with cryptographic proof.

What Sigstore Actually Does

Sigstore is an open-source project (now under the OpenSSF umbrella) that provides three core components:

Component	Purpose
Cosign	Signs and verifies container images and artifacts
Fulcio	Issues short-lived code signing certificates tied to OIDC identity
Rekor	Transparency log that records all signing events immutably

The breakthrough is keyless signing. Instead of managing long-lived GPG or KMS keys (which become their own security liability), Cosign authenticates you via OIDC (GitHub Actions, Google, etc.), Fulcio issues a short-lived certificate, and Rekor records the signature. No keys to rotate. No keys to leak.

Signing Images in Your CI Pipeline

Here's how to sign container images in GitHub Actions using keyless Cosign:

# .github/workflows/build-sign.yml
name: Build and Sign Container Image
on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write  # Required for keyless signing

jobs:
  build-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push image
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Sign the image (keyless)
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

      - name: Generate and attach SBOM
        run: |
          # Generate SBOM with syft
          syft ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }} \
            -o spdx-json > sbom.spdx.json

          # Attach SBOM as a signed attestation
          cosign attest --yes \
            --type spdxjson \
            --predicate sbom.spdx.json \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

The id-token: write permission is essential — it allows the workflow to request an OIDC token from GitHub, which Fulcio uses to issue the signing certificate. The certificate is bound to the workflow identity: the repository, branch, and commit SHA.

Verifying Signed Images

Anyone can verify the image signature:

# Verify the image was signed by a specific GitHub Actions workflow
cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp "https://github.com/your-org/your-repo/.github/workflows/build-sign.yml@refs/heads/main" \
  ghcr.io/your-org/your-repo@sha256:abc123...

This verification confirms three things:

The image was signed (not tampered with after build)
The signer's identity matches your expected CI workflow
The signature is recorded in the Rekor transparency log

If any of these checks fail, the image is untrusted. Do not deploy it.

Verifying SBOM Attestations

The SBOM attestation tells you exactly what's inside the image:

# Verify and extract the SBOM attestation
cosign verify-attestation \
  --type spdxjson \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp "https://github.com/your-org/your-repo/.github/workflows/build-sign.yml@refs/heads/main" \
  ghcr.io/your-org/your-repo@sha256:abc123... | \
  jq -r '.payload' | base64 -d | jq '.predicate'

This gives you a machine-readable inventory of every package in the image. When the next Log4Shell or xz-utils drops, you can query your SBOMs across every image in your registry and know within minutes whether you're affected.

Enforcing Signatures at Admission

Signing images means nothing if your cluster doesn't enforce verification. Use the Sigstore Policy Controller (or Kyverno with Cosign verification) to reject unsigned images at admission time.

Sigstore Policy Controller

Install it:

helm repo add sigstore https://sigstore.github.io/helm-charts
helm repo update

helm install policy-controller sigstore/policy-controller \
  --namespace sigstore-system \
  --create-namespace \
  --set webhook.configurationPolicy.enabled=true

Create a policy that requires images to be signed by your CI pipeline:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-signed-images
spec:
  images:
    - glob: "ghcr.io/your-org/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: "https://github.com/your-org/.*/\\.github/workflows/build-sign\\.yml@refs/heads/main"
        ctlog:
          url: https://rekor.sigstore.dev

Label namespaces to enforce the policy:

kubectl label namespace production policy.sigstore.dev/include=true

Now try deploying an unsigned image to the production namespace:

kubectl run test --image=ghcr.io/your-org/unsigned-app:latest -n production
# Error: admission webhook "policy.sigstore.dev" denied the request:
# validation failed: no matching signatures found

This is exactly what you want. Unsigned code does not run in production. Period.

Kyverno Alternative

If you're already running Kyverno, you can use its built-in Cosign verification:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-signature
      match:
        any:
          - resources:
              kinds: ["Pod"]
      verifyImages:
        - imageReferences: ["ghcr.io/your-org/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/your-org/*"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Threat Model: Supply Chain Attack Scenarios

Let's map this to real attack scenarios:

Scenario 1: Compromised registry. An attacker pushes a malicious image tag to your registry. With signature verification, the admission controller rejects it because the image lacks a valid signature from your CI pipeline.

Scenario 2: Tampered image. An attacker modifies an image layer after it was signed (e.g., injecting a backdoor). The digest changes, the signature no longer matches, verification fails. Deployment blocked.

Scenario 3: Compromised CI pipeline. An attacker modifies your workflow file to inject malicious build steps. The signed image now has a valid signature, but your policy restricts signing to @refs/heads/main. The attacker's PR-branch build won't produce a signature that passes the identity check.

Scenario 4: Dependency confusion. A malicious package enters your build via a typosquatted dependency. The SBOM attestation captures this. Your vulnerability scanner flags it. You catch it before production.

No single control stops every attack. Layered defenses do.

Scanning the Transparency Log

Rekor provides a public, immutable record of every signing event. Use it to audit your signing history:

# Search Rekor for all signatures by your CI identity
rekor-cli search \
  --email "your-org/your-repo/.github/workflows/build-sign.yml@refs/heads/main"

# Get details of a specific log entry
rekor-cli get --uuid <log-entry-uuid> --format json | jq .

If someone signs an image outside your official pipeline, it shows up in Rekor with a different identity. You can alert on this. In a mature setup, any signature from an unexpected identity triggers a security incident.

Implementation Roadmap

Day 1: Install Cosign in your CI pipeline. Sign one image. Verify it locally.
Week 1: Add SBOM generation and attestation to all image builds.
Week 2: Deploy the Sigstore Policy Controller to staging. Enforce signature verification.
Week 3: Roll out enforcement to production. Start in warn mode, then switch to enforce.
Ongoing: Monitor Rekor for unexpected signing events. Update identity policies as workflows change.

Troubleshooting Signature Verification Failures

Signature verification failures in production are stressful — they block deployments. Here's how to systematically debug them.

"no matching signatures" on a Signed Image

This is the most common failure. The image is signed, but the verification parameters don't match. Walk through it step by step:

# Step 1: Confirm the image has a signature at all
cosign tree ghcr.io/your-org/your-repo@sha256:abc123...
# Look for "Signatures" in the output

# Step 2: Inspect the signature's certificate
cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp ".*" \
  ghcr.io/your-org/your-repo@sha256:abc123... 2>&1 | head -30

# Step 3: Extract the actual identity from the certificate
cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp ".*" \
  ghcr.io/your-org/your-repo@sha256:abc123... 2>&1 | \
  jq -r '.[0].optional.Subject'

Common mismatches:

Expected Identity	Actual Identity	Cause
`...build-sign.yml@refs/heads/main`	`...build-sign.yml@refs/heads/feature-x`	Image was built from a non-main branch
`...build-sign.yml@refs/heads/main`	`...build-and-test.yml@refs/heads/main`	Workflow file was renamed
`https://token.actions.githubusercontent.com`	`https://accounts.google.com`	Image was signed with a different OIDC provider

Fix your ClusterImagePolicy or Kyverno policy to match the actual signing identity. If the identity is wrong (e.g., a feature branch), your policy is working correctly — the image shouldn't be deployed.

Signature Verification in Air-Gapped Environments

Not every cluster has internet access to reach rekor.sigstore.dev and fulcio.sigstore.dev. For air-gapped environments, you have two options:

Mirror the Sigstore infrastructure — deploy your own Rekor and Fulcio instances internally. This is the enterprise approach.
Use key-based signing instead of keyless — generate a Cosign key pair and distribute the public key to your clusters:

# Generate a key pair (store the private key securely)
cosign generate-key-pair

# Sign with the private key in CI
cosign sign --key cosign.key \
  ghcr.io/your-org/your-repo@sha256:abc123...

# Verify with the public key (no internet needed)
cosign verify --key cosign.pub \
  ghcr.io/your-org/your-repo@sha256:abc123...

# ClusterImagePolicy for key-based verification (air-gapped)
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-signed-images-airgap
spec:
  images:
    - glob: "registry.internal.example.com/**"
  authorities:
    - key:
        data: |
          -----BEGIN PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
          -----END PUBLIC KEY-----

Key-based signing requires key management (rotation, secure storage), which keyless avoids. But in air-gapped environments, it's the pragmatic choice.

Signing Multi-Architecture Images

If you build multi-arch images with Docker Buildx, you need to sign the manifest list (index), not just individual platform images. Cosign handles this, but the workflow is slightly different:

# .github/workflows/build-sign-multiarch.yml
- name: Build and push multi-arch image
  id: build
  uses: docker/build-push-action@v5
  with:
    context: .
    platforms: linux/amd64,linux/arm64
    push: true
    tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

- name: Sign the multi-arch manifest
  run: |
    # Get the manifest list digest (not a platform-specific digest)
    DIGEST=$(crane digest ghcr.io/${{ github.repository }}:${{ github.sha }})
    cosign sign --yes \
      ghcr.io/${{ github.repository }}@${DIGEST}

The crane digest command fetches the digest of the manifest list. Signing this single digest covers all platform-specific images referenced by the list. Your admission controller verifies the manifest list signature, and any platform image pulled from it is implicitly trusted.

Stop Trusting. Start Verifying.

The software supply chain is the largest unmonitored attack surface in most organizations. Every unsigned image you deploy is an implicit statement: "I trust that nothing between the developer's commit and this running container was tampered with." That's a bold assumption in a world where nation-state actors spend years infiltrating open-source projects.

Sigstore makes verification practical. Cosign makes it automatable. Admission controllers make it enforceable. The tooling exists. The standards are mature. The only thing missing is the decision to stop running unverified code.

Make that decision today. Your future incident responders will thank you.