aws-guardian: detecting and reversing risky AWS changes in seconds

The problem

Most serious cloud incidents start with one API call: a security group opened to the internet, a bucket made public, an administrator policy attached to the wrong identity, or logging switched off so nobody sees what comes next. A posture scanner that runs every few hours finds these eventually. I wanted something that reacts while the change is still seconds old, and that can put it back without waiting for a person, with a scheduled scan behind it for whatever was already there.

aws-guardian is that pipeline. It is a personal project, built from scratch and running in my own AWS account.

How it works

API call  ->  CloudTrail  ->  EventBridge  ->  Lambda  ->  rule matches?
                              (one rule                      |
                               per service)                  +-> finding logged as JSON
                                                             +-> alert published to SNS
                                                             +-> fix applied (enforce mode)

1. CloudTrail records every management API call in the account.
2. EventBridge forwards only the calls a detection cares about. The list of calls is generated from the rule registry, and a unit test fails if the two drift apart.
3. A Python Lambda runs the matching rules. Each rule is a small pure function that takes the CloudTrail event and returns a finding or nothing.
4. Every finding is logged, emailed through SNS, and, in enforce mode, remediated with a single scoped API call.

What it detects

35 rules across six areas, each mapped to a MITRE ATT&CK cloud technique. 20 of them fix the problem automatically; the rest alert only, because the safe response depends on context.

Area	Examples	Rules
Logging and detection	CloudTrail stopped or deleted, GuardDuty disabled, VPC created without flow logs	8
S3	Public access block removed, bucket ACL or policy made public	4
Network	SSH, RDP, database ports or all ports opened to the internet	3
Data exposure	Public EBS or RDS snapshots, public AMIs, KMS key scheduled for deletion	7
IAM	Administrator policy attached, access key created for another user, MFA deactivated	9
Root and sign-in	Root console sign-in, root API activity, sign-in without MFA	4

The full table, with the trigger and response for each rule, is in the detections list.

Design decisions

• Three modes. off alerts only, dry_run also says what it would have fixed, and enforce fixes it. I ran dry-run first and read the alerts before letting it change anything.
• Remediation permissions exist only in enforce mode. In the other modes the Lambda’s role can write its own logs and publish alerts, and nothing else. Terraform attaches the remediation policy only when the mode is enforce.
• Least privilege. The remediation policy has no wildcard actions. It holds one write action per fix and cannot read data, create identities or delete resources.
• Narrow fixes. Opening SSH to the world revokes that one ingress rule and leaves the rest of the security group alone. A public bucket policy is neutralised by re-applying the public access block rather than deleting the policy.
• No feedback loops. The pipeline ignores API calls made by its own role, so undoing a change never raises another finding.
• A failed fix never hides the alert. The finding is still published, with the error attached.

Testing it with real misconfigurations

Unit tests cover every rule with an event that must fire and look-alikes that must not. That proves the logic, but not that the pipeline works in a real account. So I wrote a simulator that performs the misconfigurations for real: each scenario creates its own throwaway resource, misconfigures it, measures the response, and deletes it.

Results from my account in enforce mode. “Reverted after” is measured by the simulator, from making the bad change to observing it undone.

Rule	What the simulator did	Reverted after
NET-001	Opened SSH to the internet	10.2s
NET-002	Opened PostgreSQL to the internet	3.8s
NET-003	Opened every port to the internet	4.0s
LOG-008	Created a VPC with no flow logs	6.2s
S3-001	Deleted a bucket’s public access block	9.3s
IAM-001	Attached AdministratorAccess to a user	3.3s
IAM-002	Added an inline allow-everything policy	3.2s
IAM-003	Created an access key for another user	3.2s
LOG-001	Stopped CloudTrail logging	3.2s
IAM-005	Set a console password for another user	Alert only, detected in 3.0s

All ten scenarios were detected, and all nine with an automatic fix were reverted in about ten seconds or less. The other 25 rules are covered by unit tests but have not yet been exercised against live events.

Finding what was already wrong

The event pipeline only sees changes made after it was deployed. To cover everything that existed before, a second Lambda runs a posture scan on a schedule. It reads the account’s current state in every enabled region and emails one digest, most severe first, with a recommended fix for each check.

It has 16 checks, including root without MFA, console users without MFA, access keys older than 90 days, public buckets, security groups open to the internet, VPCs without flow logs, and public snapshots and images.

• Read-only. The scan reports; it never changes anything. Reversing a change seconds after it was made is safe to automate. Changing something that has been in place for months is a decision for a person.
• Its own role. The scan runs with a separate read-only role, so the responder’s role stays free of read access.
• A failed check is reported, not fatal. One denied API call does not hide the other findings.

Its first run across 17 regions took 11 seconds and returned 53 findings. Nearly all were AWS defaults that are off in every region: default VPCs without flow logs, EBS encryption by default, and GuardDuty.

Limits and next steps

• It covers one account. Multi-account deployment through AWS Organizations is the next step.
• Alerts go to email. Routing by severity to chat or a ticket queue would suit a team.

The code, tests, Terraform and simulator are in the aws-guardian repository.

← all writeups