Debugging AWS IAM Denials Without Guessing at Policy JSON

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering

An AccessDenied error from AWS tells you almost nothing. It names the action and sometimes the resource, but never which of the five places a permission could be blocked actually blocked it. We used to open the IAM console and read policies line by line, sometimes across three accounts. Now we run a handful of CLI commands and get the answer in a few minutes, even for cross-account cases.

The Error That Tells You Nothing

A typical denial looks like this:

An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:iam::123456789012:user/deploy-bot is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::app-artifacts/build.zip" because no identity-based policy allows the s3:PutObject action

That last clause is doing a lot of work. It says no identity-based policy allows it, but permissions in AWS come from more than one layer: the identity policy, a resource policy on the bucket itself, a permissions boundary, a Service Control Policy at the org level, and any session policy attached during role assumption. The error message only ever tells you about the first layer AWS happened to check, and AWS doesn’t always check them in the order you’d assume.

How AWS Actually Evaluates a Request

The order matters more than most people realize:

1. If any applicable SCP explicitly denies the action, the request fails immediately. Nothing else gets evaluated.
2. If a resource policy explicitly denies it, same result.
3. If the identity policy or resource policy explicitly denies it, same result again. Explicit deny always wins, everywhere, no exceptions.
4. If none of the above apply, AWS checks whether the SCP (if any), the identity-based policy, and the resource-based policy (if any) collectively allow the action.
5. If a permissions boundary is attached, the effective permission is the intersection of the identity policy and the boundary. The boundary can only restrict, never grant.
6. If the caller is an assumed role, a session policy passed at AssumeRole time further restricts whatever the role’s own policy allows.

An explicit deny buried in an SCP at the very top of the org will silently override an identity policy that looks completely correct. This is the single most common reason a policy that “should work” doesn’t.

Start With Who You Actually Are

Before touching policy documents, confirm the identity actually making the call:

aws sts get-caller-identity

{
  "UserId": "AIDAEXAMPLE123456",
  "Account": "123456789012",
  "Arn": "arn:aws:iam::123456789012:user/deploy-bot"
}

This sounds obvious, but assumed roles and instance profiles make it easy to debug the wrong principal. If a Lambda function assumes a role, the caller in the error message is the role, not the Lambda’s execution identity. Debugging the execution role’s own permissions instead of the assumed role’s permissions is a common dead end.

Simulate the Call Before Digging Through Policies

iam simulate-principal-policy evaluates a specific action and resource against everything attached to a principal, without making the real API call:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:user/deploy-bot \
  --action-names s3:PutObject \
  --resource-arns arn:aws:s3:::app-artifacts/build.zip

The output includes an EvalDecision of allowed, explicitDeny, or implicitDeny, plus which specific policy statement produced that decision. That last part is the whole point: it names the exact statement, not just the fact that something failed.

Reading the Simulation Output

Three outcomes show up in practice.

implicitDeny means nothing granted the permission. There’s no explicit block, just an absence of any allow. The fix is adding a statement.

explicitDeny means something actively denies it, usually a broader statement like Action: "*" scoped to a condition, or a permissions boundary that caps what the identity policy can grant even if the identity policy itself would allow it. The simulation output names the exact policy ARN and statement ID, so there’s no need to open every attached policy manually.

allowed with the real call still failing means the block isn’t at the identity level at all. That’s the signal to check the resource policy, an SCP, or a session policy.

Debugging Condition Keys With --context-entries

Plenty of real-world policies aren’t unconditional. A policy might only allow s3:GetObject when aws:SourceIp falls in a specific CIDR range, or only allow an action when aws:PrincipalOrgID matches. simulate-principal-policy evaluates conditions too, but only if you supply the context values it would normally get from the request:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:user/deploy-bot \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::app-artifacts/build.zip \
  --context-entries \
    ContextKeyName=aws:SourceIp,ContextKeyValues=203.0.113.42,ContextKeyType=ip

Without the matching --context-entries, a condition-based policy statement gets evaluated as if the condition key were absent, which usually means the statement doesn’t apply and the simulation gives a misleading implicitDeny. This is the step most people skip, and it’s the reason simulation results sometimes don’t match what actually happens in production.

Cross-Account Role Assumption

When the caller assumes a role in a different account, there are two separate trust relationships to check, and a failure at either one produces the same generic AccessDenied.

First, the role’s trust policy has to allow the calling account or principal to assume it:

aws iam get-role --role-name deploy-role \
  --query 'Role.AssumeRolePolicyDocument'

Second, once assumed, the role’s own permissions apply, not the calling identity’s permissions from the source account. It’s easy to confirm the source user has S3 access and miss that the role it assumed doesn’t:

aws sts assume-role \
  --role-arn arn:aws:iam::999999999999:role/deploy-role \
  --role-session-name debug-session

If assume-role itself fails, the problem is the trust policy. If it succeeds but the subsequent S3 call fails, run simulate-principal-policy against the role’s ARN, not the original user’s.

When the Policy Isn’t the Problem

Simulation only evaluates identity-based policies and permissions boundaries. It doesn’t see SCPs or resource-based policies. If simulation says allowed but the real call still fails, check the resource side directly:

aws s3api get-bucket-policy --bucket app-artifacts

And if the account sits inside an AWS Organization, run this from the management account to see what’s actually attached:

aws organizations list-policies-for-target \
  --target-id 123456789012 \
  --filter SERVICE_CONTROL_POLICY

SCPs never show up in simulate-principal-policy output. This is the most common reason a simulation says yes and reality says no, and the reason a policy that worked yesterday can fail today if someone tightened an SCP at the org level without touching a single identity policy.

Cross-Referencing With CloudTrail

For a denial that already happened, CloudTrail records the exact error and the request parameters, which is often faster than reconstructing the call from memory:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject \
  --start-time "$(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%SZ')" \
  --query 'Events[*].CloudTrailEvent' \
  --output text | jq -r '. | fromjson | {errorCode, errorMessage, userIdentity}'

The errorCode field often distinguishes between AccessDenied from an identity policy versus AccessDeniedException from a resource policy or SCP, which the console error message frequently collapses into one generic string.

A Repeatable Debugging Script

We keep this as a small wrapper so nobody has to remember the flag names during an incident:

#!/bin/bash
# iam-check.sh   
aws iam simulate-principal-policy \
  --policy-source-arn "$1" \
  --action-names "$2" \
  --resource-arns "$3" \
  --query 'EvaluationResults[0].[EvalActionName,EvalDecision,EvalDecisionDetails]' \
  --output table

Running ./iam-check.sh <arn> s3:PutObject <bucket-arn> gives a one-line answer instead of a policy read during a deploy failure.

Key Lessons

1. The AccessDenied message only reports the first layer AWS happened to check.
   Identity policy, resource policy, permissions boundary, SCP, and session policy can each independently block a call.
2. Explicit deny always wins, at any layer, with no exceptions.
   An SCP deny at the org root overrides an identity policy that looks entirely correct.
3. Confirm the actual calling identity before reading any policy.
   Assumed roles and instance profiles make it easy to debug the wrong principal.
4. Condition keys need --context-entries to simulate correctly.
   Skipping them produces a misleading implicitDeny on policies that actually work in production.
5. Cross-account calls have two separate trust relationships to check.
   The role’s trust policy and the role’s own permissions fail independently but look identical from the caller’s side.
6. A clean simulation result doesn’t guarantee the real call will succeed.
   Resource policies and SCPs sit outside what simulation checks entirely.

Summary

Layer	Checked by simulate-principal-policy?	Where to look instead
Identity-based policy	Yes	Simulation output names the statement
Permissions boundary	Yes	Simulation output shows the boundary decision
Condition keys	Yes, with --context-entries	Must supply context values manually
Resource policy (bucket, key)	No	get-bucket-policy / resource-specific API
Service Control Policy	No	list-policies-for-target from the management account
Role trust policy (cross-account)	No	get-role on the target role
Session policy (assumed role)	Partial	Re-run simulation with the assumed role’s ARN

Most IAM debugging time goes into reading policies that were never the problem, or into policies AWS evaluated in an order nobody expected. Confirming identity first, simulating the specific call with real context values second, and checking the layers simulation can’t see third turns a policy-reading exercise into a few CLI commands.

Read Next

• Tired of Copying and Pasting AWS Credentials? Say Hello to Single Sign-On
• Managing Multiple Accounts With AWS Control Tower

If you’re working with AWS IAM regularly, follow along on LinkedIn for more of these.

---

Tags:
#AWS   #IAM   #CLI   #DevOps  
#CloudSecurity   #Debugging   #AWSOrganizations   #SRE