DevOpsil
KubernetesMedium

Fix: Kubernetes ConfigMap/Secret Changes Not Applied

Updated ConfigMap or Secret but pods still use old values

!Symptoms

  • Updated ConfigMap or Secret but pods still use old values
  • Application config has not changed after kubectl apply
  • Environment variables in pod still show old ConfigMap values

?Root Causes

  • Pods do not automatically restart when a ConfigMap or Secret changes
  • ConfigMap mounted as volume may take up to kubelet sync period to update
  • Environment variables from ConfigMap are set at pod creation and never refreshed
  • Deployment did not trigger a rollout because the pod spec did not change

#Diagnosis Steps

  1. 1Verify ConfigMap was updated: `kubectl get cm <name> -o yaml`
  2. 2Check pod creation time: `kubectl get pod <name> -o jsonpath='{.metadata.creationTimestamp}'`
  3. 3Exec into pod and check current values: `kubectl exec <pod> -- env | grep <var>`
  4. 4Check if volume-mounted config files are updated: `kubectl exec <pod> -- cat /path/to/config`

>Fix

  1. 1Restart the deployment to pick up changes: `kubectl rollout restart deployment/<name>`
  2. 2Use a hash annotation to force rollout on ConfigMap change: add configmap hash to pod template annotations
  3. 3Use Reloader (stakater/Reloader) to automatically restart pods on ConfigMap changes
  4. 4For volume mounts, wait for kubelet sync (up to 1-2 minutes) and check again

*Prevention

  • Use a tool like Reloader to auto-restart pods on config changes
  • Include ConfigMap content hash in deployment annotations (Helm does this with `checksum/config`)
  • Use immutable ConfigMaps with versioned names and update the reference
  • Document that config changes require pod restart in your runbooks
  • Use a config management system that handles reloads (e.g., Spring Cloud Config with refresh)

Related Error Messages

config not updatedenvironment variable still shows old valuemount delay