Flux Notification Controller: Alerting on GitOps Drift and Sync Failures

When your GitOps pipeline breaks silently, you find out the worst way — during an incident. The Flux Notification Controller is your early warning system. Here's everything you need to configure it and stop flying blind.

What the Notification Controller Does

The Notification Controller handles two directions of communication:

• Inbound: Receives webhook events from Git providers to trigger reconciliation
• Outbound: Sends alerts to Slack, Teams, PagerDuty, etc. when Flux events occur

We're focused on outbound alerting here — drift detection and sync failures.

---

Core Resources: The Three-Object Pattern

Every alert setup requires exactly three Kubernetes resources:

Provider → Alert → EventSource (Kustomization/HelmRelease/etc.)

---

Step 1: Create a Provider

Slack:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack-alerts
  namespace: flux-system
spec:
  type: slack
  channel: "#gitops-alerts"
  secretRef:
    name: slack-webhook-url

kubectl create secret generic slack-webhook-url \
  --from-literal=address=https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
  -n flux-system

PagerDuty:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: pagerduty-alerts
  namespace: flux-system
spec:
  type: pagerduty
  secretRef:
    name: pagerduty-token

Generic Webhook (for custom integrations):

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: custom-webhook
  namespace: flux-system
spec:
  type: generic
  address: https://your-endpoint.example.com/flux-events
  secretRef:
    name: webhook-hmac-token

---

Step 2: Create an Alert

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: gitops-failures
  namespace: flux-system
spec:
  providerRef:
    name: slack-alerts
  eventSeverity: error        # Only fire on errors, not info
  eventSources:
    - kind: Kustomization
      name: "*"               # Wildcard catches all Kustomizations
    - kind: HelmRelease
      name: "*"
    - kind: GitRepository
      name: "*"
  exclusionList:
    - ".*validated.*"         # Suppress noisy validation messages

Opinion: Use eventSeverity: error in production. info floods your channel and you'll start ignoring it — defeating the whole point.

---

Alert on Drift Specifically

Flux detects drift when live cluster state diverges from Git. To catch this:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: drift-detection
  namespace: flux-system
spec:
  providerRef:
    name: slack-alerts
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "*"
  summary: "Drift detected in cluster — manual change may have occurred"

Drift events surface as ReconciliationFailed or ValidationFailed in the event stream.

---

Quick Debugging Commands

# Check if your Alert is ready
kubectl get alerts -n flux-system

# Describe for error details
kubectl describe alert gitops-failures -n flux-system

# Check Provider status
kubectl get providers -n flux-system

# View raw Flux events (your alert source)
kubectl get events -n flux-system --sort-by='.lastTimestamp'

# Check notification controller logs
kubectl logs -n flux-system deployment/notification-controller -f

# Manually trigger a reconciliation to test alerting
flux reconcile kustomization flux-system --with-source

---

Common Event Sources Reference

Kind	What it monitors
GitRepository	Git fetch failures, auth issues
Kustomization	Apply failures, drift, validation errors
HelmRelease	Chart failures, upgrade rollbacks
HelmRepository	Index fetch failures
ImageRepository	Image scan failures
ImagePolicy	Policy evaluation failures

---

Multi-Environment Alert Routing

Route critical namespaces to PagerDuty, everything else to Slack:

# Production → PagerDuty
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: prod-critical
  namespace: flux-system
spec:
  providerRef:
    name: pagerduty-alerts
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "production-*"    # Name prefix matching
---
# Non-prod → Slack
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: nonprod-alerts
  namespace: flux-system
spec:
  providerRef:
    name: slack-alerts
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: "staging-*"
    - kind: Kustomization
      name: "dev-*"

---

Test Your Alert Pipeline

Don't wait for a real failure. Break something intentionally:

# Reference a non-existent image to trigger a HelmRelease failure
flux suspend helmrelease my-app -n default

# Or corrupt a configmap reference in your repo, push it, and watch
# The alert should fire within one reconciliation interval

Check alert delivery:

kubectl get alerts -n flux-system -o wide
# Look for "Ready: True" — if False, your Provider config is broken

---

Gotchas to Know

• Wildcard "*" only matches within the Alert's own namespace — cross-namespace alerting requires separate Alert objects per namespace
• Secrets must use the address key for webhook providers — not url, not webhook
• Provider health isn't validated until an event fires — always run a test reconciliation after setup
• eventSeverity: info includes successful syncs — your Slack will look like a firehose

---

TL;DR Checklist

• Create Provider with correct secret key (address)
• Create Alert pointing to that Provider
• Set eventSeverity: error unless you want noise
• Use name: "*" wildcards to catch all resources of a type
• Trigger a test reconciliation to verify delivery
• Check kubectl get alerts -n flux-system — all should be Ready: True

Silent GitOps failures are the worst kind. Five minutes of setup here saves hours of incident response later.