Fix: GCP IAM Permission Denied on Service Account

!Symptoms

• API calls returning 403 PERMISSION_DENIED
• Terraform apply failing with 'googleapi: Error 403'
• Application unable to access GCP services (GCS, BigQuery, Pub/Sub)
• Workload Identity not mapping to the correct GCP service account

?Root Causes

• IAM role binding missing or applied to wrong resource/project
• Service account doesn't have the required IAM role
• Workload Identity annotation mismatch between KSA and GSA
• Organization policy restricting the API or resource
• API not enabled in the project

#Diagnosis Steps

1. 1Check IAM bindings: `gcloud projects get-iam-policy <project> --flatten='bindings[].members' --filter='bindings.members:<sa-email>'`
2. 2Check which roles are needed: `gcloud iam roles describe roles/<role-name>`
3. 3Verify Workload Identity: `kubectl describe sa <ksa> -n <ns>` — check annotation
4. 4Check API enabled: `gcloud services list --enabled | grep <api>`
5. 5Test with impersonation: `gcloud auth print-access-token --impersonate-service-account=<sa>`

>Fix

1. 1Grant the required role: `gcloud projects add-iam-policy-binding <project> --member=serviceAccount:<sa> --role=roles/<role>`
2. 2Fix Workload Identity binding: `gcloud iam service-accounts add-iam-policy-binding <gsa> --member=serviceAccount:<project>.svc.id.goog[<ns>/<ksa>] --role=roles/iam.workloadIdentityUser`
3. 3Enable the required API: `gcloud services enable <api>.googleapis.com`
4. 4Fix the annotation on the Kubernetes service account
5. 5If org policy blocking, request an exception from the org admin

*Prevention

• Define IAM bindings in Terraform for reproducibility and audit
• Use custom IAM roles with minimum required permissions
• Automate Workload Identity setup as part of cluster provisioning
• Enable Cloud Audit Logs for IAM to track permission changes
• Test IAM permissions in staging before production deployment

Related Error Messages