Skip to content

Defender — Objective 4: Identify Credential Theft

Finding the Suspicious Event

Start from the most obviously suspicious API call — ListBuckets. A web application has no reason to enumerate all S3 buckets:

find . -type f -exec cat {} \; | jq '.Records[]|select(.eventName=="ListBuckets")'

select() — jq's WHERE clause:

Part What it does
select(condition) Only pass through events where the condition evaluates to true
.eventName=="ListBuckets" Condition: eventName must equal the string "ListBuckets"

This is the jq equivalent of SQL's WHERE eventname = 'ListBuckets'. Without select, every event passes through. With it, only matching events do.


Reading the Event

The ListBuckets event:

{
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAJQMBDNUMIKLZKMF64:d190d14a-2404-45d6-9113-4eda22d7f2c7",
        "arn": "arn:aws:sts::653711331788:assumed-role/level3/d190d14a-2404-45d6-9113-4eda22d7f2c7",
        "accountId": "653711331788",
        "accessKeyId": "<REDACTED_ACCESS_KEY_ID>",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "arn": "arn:aws:iam::653711331788:role/level3"
            }
        }
    },
    "eventTime": "2018-11-28T23:09:28Z",
    "sourceIPAddress": "104.102.221.250",
    "userAgent": "[aws-cli/1.16.19 Python/2.7.10 Darwin/17.7.0 botocore/1.12.9]",
    "eventName": "ListBuckets"
}

Three red flags in this event:

1. Role: level3 — this is the ECS task role

userIdentity.arn shows assumed-role/level3/.... The level3 role is the IAM role attached to the ECS container (the web app). ECS tasks use this role to call AWS services from inside the container. It should only ever appear with AWS-owned source IPs.

2. Source IP: 104.102.221.250 — not an AWS IP

ECS containers run inside AWS infrastructure. When the container calls an AWS API, the request comes from an AWS-owned IP (the underlying EC2 host or Fargate cluster). 104.102.221.250 is not in any AWS IP range — it's an external IP (the exercise says it's nsa.gov, but in a real investigation it would be the attacker's machine or a VPN exit node).

3. User agent: aws-cli/1.16.19 Python/2.7.10 Darwin/17.7.0

Darwin is macOS. The ECS container runs Linux. The ECS application's code wouldn't use the aws-cli directly — it would use an SDK like boto3 or the Go AWS SDK. Seeing the macOS aws-cli user agent means a person on a Mac ran aws s3 ls from their terminal using the stolen credentials.

All three together = definitive credential theft. The credentials are legitimate (the role exists, the keys are valid), but they're being used from outside the AWS environment, by a human, on a personal machine.


Checking the Role's Trust Policy

Confirm the role was only supposed to be used by ECS:

aws --profile target_security iam get-role --role-name level3

aws iam get-role — fetches the role's metadata, including its trust policy.

Part What it does
aws iam get-role Get full details about an IAM role
--role-name level3 The role to look up (by name, not ARN)
--profile target_security Run this against the Target account via the cross-account role

Response:

"AssumeRolePolicyDocument": {
    "Statement": [{
        "Action": "sts:AssumeRole",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Effect": "Allow"
    }]
}

Trust policy explained:

A trust policy is the policy attached to the role that defines who can assume it. It's the "door" — the role's permission policies define what you can do once inside, but the trust policy defines who's allowed to knock.

  • Principal: { Service: "ecs-tasks.amazonaws.com" } — only the ECS service can assume this role
  • No human IAM users, no other accounts, no other services
  • Only ECS tasks can assume this role — so any API call from this role that didn't originate from within ECS is by definition unauthorized

Comparison:

Policy type Attached to Controls
Permission policy Role/user What you can do (which AWS APIs you can call)
Trust policy Role only Who can assume this role (which principals can sts:AssumeRole)

How Credential Theft Detection Works

Normal ECS role behaviour in CloudTrail:

Field Expected value
sourceIPAddress An AWS-owned IP (the EC2 host IP or Fargate IP)
userIdentity.type AssumedRole
userAgent The application framework: python-requests, boto3, aws-sdk-go, etc.
eventName Only the specific APIs the app needs (e.g., GetObject if it reads S3, not ListBuckets)

Stolen credentials being used externally:

Field What you see instead
sourceIPAddress Non-AWS IP — residential, VPN, cloud provider outside AWS
userIdentity.type Still AssumedRole — credentials are valid
userAgent aws-cli on Darwin (macOS) or Windows
eventName Broad recon calls the app would never need: ListBuckets, ListUsers, DescribeInstances

Key insight: CloudTrail cannot flag stolen credentials by itself — it just records API calls. The detection comes from comparing the pattern of use against what's expected. This is what UEBA (User and Entity Behaviour Analytics) tools automate: baselines + anomaly detection.

In manual investigation: look for role identities using APIs from IPs they've never used before, or calling APIs they have no business calling.

The talk to understand this better: Detecting Credential Compromise in AWS — Will Bengston The core idea: establish a baseline of "normal" IPs and user-agents for each role, then alert on deviations.