Fix: PERMISSION_DENIED: The caller does not have permission

A call to a Google Cloud API failed with PERMISSION_DENIED: The caller does not have permission and HTTP 403. Authentication succeeded, Google knows exactly who called, and that identity was refused. The two questions that solve nearly every case are: which principal actually made the call, and which permission did it need on which resource. Engineers regularly lose hours to this error by fixing the IAM of an identity that was never the caller.

What this error means

Google Cloud evaluates every request against several layers. The principal’s IAM bindings on the resource and its ancestors must include a role containing the required permission. Any IAM conditions on those bindings must evaluate true for this request. No deny policy may block the permission. And for some operations, organization policy constraints or a VPC Service Controls perimeter can reject the call even though IAM would allow it. PERMISSION_DENIED is the generic surface for all of these layers, which is why the same message can have five different root causes.

The subtlety is the caller. Code running on GCP uses Application Default Credentials, which resolve in order: the GOOGLE_APPLICATION_CREDENTIALS key file if set, then the gcloud user credentials on a workstation, then the service account attached to the compute resource. A script that works on your laptop under your user account and fails on a VM is not a mystery, it is two different principals with two different sets of roles.

Common causes

• The service account genuinely lacks the role containing the required permission, often after a new service was added and the Terraform grant was forgotten.
• The wrong identity is calling: the default Compute Engine service account instead of the dedicated one, a stale key file overriding the attached account, or on GKE a pod without the Workload Identity annotation chain.
• The grant exists at the wrong level, for example on the project while the resource (bucket, dataset, secret, Pub/Sub topic) has its own IAM policy that does not inherit the needed role.
• An IAM condition on the binding does not match this request, such as a resource name prefix or an expiry that has passed.
• A deny policy on the project or organization explicitly blocks the permission and overrides every allow.
• An org policy constraint or a VPC Service Controls perimeter blocks the operation category, commonly seen when adding external members under domain restricted sharing or calling a protected API from outside the perimeter.

How to fix it

STABILIZE. If a production workload is blocked, first establish who the caller is, because any grant you make before that is a guess. On the affected VM or container, ask the metadata server:

curl -s -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"

For Cloud Run, read the service’s identity directly:

gcloud run services describe SERVICE_NAME \
  --region=REGION \
  --format="value(spec.template.spec.serviceAccountName)"

Also check the environment for a key file that silently overrides the attached account: if GOOGLE_APPLICATION_CREDENTIALS is set, that file’s identity is the caller, not the attached service account.

DIAGNOSE. Find the exact permission that was refused. Recent audit log entries show the method, the principal, and the denial:

gcloud logging read \
  'protoPayload.status.code=7 AND protoPayload.authenticationInfo.principalEmail="SA_EMAIL"' \
  --project=PROJECT_ID --limit=10 --freshness=1h

Then let Policy Troubleshooter tell you which layer said no. It evaluates the full policy hierarchy, including conditions, for one principal, one permission, one resource:

gcloud policy-intelligence troubleshoot-policy iam \
  //cloudresourcemanager.googleapis.com/projects/PROJECT_ID \
  --principal-email=SA_EMAIL \
  --permission=storage.objects.get

Review what the principal currently holds on the project:

gcloud projects get-iam-policy PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:SA_EMAIL" \
  --format="table(bindings.role, bindings.condition.title)"

If the roles look sufficient, check the two layers that override allows. Deny policies:

gcloud iam policies list \
  --attachment-point=cloudresourcemanager.googleapis.com/projects/PROJECT_ID \
  --kind=denypolicies

And if the audit log entry mentions a VPC Service Controls violation or carries a vpcServiceControlsUniqueIdentifier, the block is a perimeter, not IAM, and no role grant will fix it. That identifier can be searched in the org’s audit logs to see exactly which perimeter and rule rejected the request.

FIX. Grant the smallest predefined role that contains the missing permission, on the narrowest resource that works. For example, read access to one bucket belongs on the bucket, not the project:

gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
  --member="serviceAccount:SA_EMAIL" \
  --role="roles/storage.objectViewer"

Project-level grants when the scope genuinely is the project:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:SA_EMAIL" \
  --role="roles/pubsub.publisher"

If the wrong identity was calling, fix the identity instead of widening its access: attach the intended service account to the workload, remove the stale key file, or repair the Workload Identity binding on GKE. If a condition, deny policy, or org policy is the blocker, that control usually exists on purpose. Change the workload to comply, or take the exception request to whoever owns the constraint, rather than weakening it during an incident. Expect any IAM change to take a minute or two to propagate before you judge whether it worked.

How to prevent it

• Manage every IAM binding in infrastructure as code, so grants are reviewable and a recreated service account gets its roles back automatically.
• Give each workload its own service account and stop using the default Compute Engine account, which hides who needs what.
• Ban long-lived key files where you can; attached service accounts and Workload Identity leave no stale credentials to override ADC.
• Test new deployments with Policy Troubleshooter before go-live instead of discovering missing permissions in production.
• Use IAM Recommender to trim over-broad roles gradually, so audits do not force a disruptive cleanup later.
• Document your org policies and perimeters where engineers will find them, since these produce the most confusing denials.

When to call a senior engineer

Call for help when Policy Troubleshooter says access should be allowed but the API still refuses, when the denial involves VPC Service Controls or deny policies spanning an org hierarchy nobody fully maps anymore, or when a Workload Identity chain across GKE, IAM, and ADC fails somewhere you cannot see. Erzon engineers can trace a denied request through every policy layer, fix the actual blocking control, and leave the project with least-privilege IAM that does not break the next deploy.