KubernetesMedium
Fix: Kubernetes ConfigMap/Secret Changes Not Applied
Updated ConfigMap or Secret but pods still use old values
!Symptoms
- Updated ConfigMap or Secret but pods still use old values
- Application config has not changed after kubectl apply
- Environment variables in pod still show old ConfigMap values
?Root Causes
- Pods do not automatically restart when a ConfigMap or Secret changes
- ConfigMap mounted as volume may take up to kubelet sync period to update
- Environment variables from ConfigMap are set at pod creation and never refreshed
- Deployment did not trigger a rollout because the pod spec did not change
#Diagnosis Steps
- 1Verify ConfigMap was updated: `kubectl get cm <name> -o yaml`
- 2Check pod creation time: `kubectl get pod <name> -o jsonpath='{.metadata.creationTimestamp}'`
- 3Exec into pod and check current values: `kubectl exec <pod> -- env | grep <var>`
- 4Check if volume-mounted config files are updated: `kubectl exec <pod> -- cat /path/to/config`
>Fix
- 1Restart the deployment to pick up changes: `kubectl rollout restart deployment/<name>`
- 2Use a hash annotation to force rollout on ConfigMap change: add configmap hash to pod template annotations
- 3Use Reloader (stakater/Reloader) to automatically restart pods on ConfigMap changes
- 4For volume mounts, wait for kubelet sync (up to 1-2 minutes) and check again
*Prevention
- Use a tool like Reloader to auto-restart pods on config changes
- Include ConfigMap content hash in deployment annotations (Helm does this with `checksum/config`)
- Use immutable ConfigMaps with versioned names and update the reference
- Document that config changes require pod restart in your runbooks
- Use a config management system that handles reloads (e.g., Spring Cloud Config with refresh)
Related Error Messages
config not updatedenvironment variable still shows old valuemount delay