An S3 Lifecycle Rule Deleted Backups We Thought Were Permanent

By KP | TZoneLabs | DevOps & Cloud Engineering

We went looking for a 45-day-old database backup during an incident review and it wasn’t there. Not corrupted, not slow to restore — gone. An S3 lifecycle rule meant to clean up old application logs had been silently deleting backups for over a month, because both object types shared the same cost-allocation tag.

This post covers how a tag-based lifecycle filter reached objects it was never meant to touch, why nothing alerted us, and what we changed so a deletion like this shows up immediately instead of during the next incident.

What Happened

Nightly backups of our production Postgres database were uploaded to S3 under backups/postgres/. Application logs from the same environment were uploaded to the same bucket under logs/app/. Both upload jobs tagged every object with environment: production for cost-allocation reporting — a convention set up long before either job existed.

A lifecycle rule was added to expire old log objects after 30 days to control storage cost. It was scoped by tag (environment: production) instead of by prefix, because at the time it was written, only the logs were tagged that way. Nobody revisited the filter when the backup job started using the same tag.

Where We Started Looking

Step 1 — Confirm the object is actually missing

aws s3api head-object \
  --bucket prod-storage-backups \
  --key backups/postgres/2026-07-01-full.sql.gz

An error occurred (404) when calling the HeadObject operation: Not Found

Step 2 — Check CloudTrail for who or what deleted it

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=backups/postgres/2026-07-01-full.sql.gz \
  --start-time 2026-06-01

The event source wasn’t a user or a role we recognized. It was s3.amazonaws.com with an event name of LifecycleExpiration — the object hadn’t been deleted by a person, it had aged out.

Step 3 — Read the actual lifecycle configuration

aws s3api get-bucket-lifecycle-configuration --bucket prod-storage-backups

{
  "Rules": [
    {
      "ID": "expire-old-logs",
      "Filter": {
        "Tag": { "Key": "environment", "Value": "production" }
      },
      "Status": "Enabled",
      "Expiration": { "Days": 30 }
    }
  ]
}

The filter had no prefix at all. Any object in the bucket tagged environment: production was in scope, backups included.

Step 4 — Check whether versioning could have saved it

aws s3api get-bucket-versioning --bucket prod-storage-backups

(empty response — versioning was never enabled)

No versioning meant expiration was permanent the moment it ran. There was no noncurrent version to fall back to.

Why the Backups Were Gone

🔴 Root cause: a lifecycle rule filtered by tag, and backups carried the same tag as the logs it was meant to expire.

The rule was written to solve a real problem (log storage cost) but scoped with a tag that wasn’t unique to logs.

The backup upload job adopted the same tag independently, for an unrelated reason, months later.

Lifecycle expiration produces no error, no failed job, no alert. The backup job kept reporting success every night, because writing the backup succeeded — it just didn’t survive 30 days.

Nobody misused a delete permission. The system did exactly what its configuration said, and the configuration was wrong in a way that only showed up when a restore was needed.

The Fix

Immediate: scope the rule to a prefix, not a tag

{
  "Rules": [
    {
      "ID": "expire-old-logs",
      "Filter": { "Prefix": "logs/app/" },
      "Status": "Enabled",
      "Expiration": { "Days": 30 }
    }
  ]
}

Real fix: separate buckets by data class, with Object Lock on backups

aws s3api put-object-lock-configuration \
  --bucket prod-backups-only \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 35 } }
  }'

Splitting backups into their own bucket removes the possibility of a logs-focused rule ever reaching them again. Object Lock in compliance mode means a backup can’t be deleted early, by a lifecycle rule, a misconfigured script, or a person with the right IAM permissions — we cover the permissions side of that in debugging what an IAM policy actually allows, which is worth checking whenever a bucket policy or role touches anything backup-related.

What We Put in Place After

1. Lifecycle rules managed in Terraform, reviewed like code

Rules now live in the same Terraform modules that define the bucket, instead of being added by hand through the console. Reviewing a prefix-scoped filter in a pull request catches the “this also matches backups” problem before it’s applied — the module structure we use for this is the same one in structuring Terraform modules and remote state.

2. An EventBridge alert on any lifecycle expiration under a backup prefix

{
  "source": ["aws.s3"],
  "detail-type": ["Object Deleted"],
  "detail": {
    "reason": ["Lifecycle expiration"],
    "object": { "key": [{ "prefix": "backups/" }] }
  }
}

This routes to the same SNS topic that pages on-call for other production alerts. A backup disappearing is now an incident on the day it happens, not on the day someone needs it.

3. A monthly restore drill, not just a nightly upload check

The backup job’s own success/failure status only tells you the upload worked. It says nothing about whether last month’s file still exists or restores cleanly. We added a scheduled job that pulls a backup at random and restores it to a scratch instance, so a silent gap gets caught within weeks instead of during a real incident.

Key Lessons

Tag-based lifecycle filters can reach objects you never intended.
A tag scoped for one purpose (cost allocation) got reused for another (lifecycle targeting) without anyone connecting the two.
Lifecycle expiration doesn’t fail loudly — it just happens.
There’s no error to catch and no job to see fail. The only signal is the object being gone later.
Backups need write-once protection, not just a retention policy.
Versioning or Object Lock stops a bad rule, a bad script, or a bad actor from being able to delete a backup early at all.
Shared buckets across data classes require filters with zero ambiguity.
A dedicated bucket per data class is easier to reason about than a shared bucket with rules that depend on tagging discipline holding forever.
A successful backup job proves nothing about a backup’s survival.
Only a restore drill confirms the backup you’ll need in six months is actually still there.

Summary

Layer	What Happened	Tool to Check
Lifecycle rule	Tag filter matched backups as well as logs	`get-bucket-lifecycle-configuration`
CloudTrail	Showed system-initiated expiration, not a user action	`cloudtrail lookup-events`
Versioning	Disabled, so expiration was permanent	`get-bucket-versioning`
Backup validation	Upload job reported success; restore was never tested	Scheduled restore drill

The backup job never failed once. The rule that erased its output never failed either. Both were working exactly as configured.