A Typo'd Route53 Zone ID Quietly Exhausted Our ACME Rate Limit, and It Took Down an Unrelated Production Certificate - TZoneLabs A Typo'd Route53 Zone ID Quietly Exhausted Our ACME Rate Limit, and It Took Down an Unrelated Production Certificate

By KP | TZoneLabs | DevOps & Cloud Engineering

Our production API certificate expired on schedule for its routine 60-day renewal, except cert-manager never got a new one issued. The cause had nothing to do with that certificate. A staging Ingress with a typo’d Route53 zone ID had been failing its own renewal silently for weeks, and each failed attempt burned through Let’s Encrypt’s account-wide rate limit until there was none left for anything else on that account, production included.

This covers how a broken staging cert took down an unrelated production one, why cert-manager’s own status didn’t make that obvious, and what we changed so one environment’s misconfiguration can’t spend another environment’s rate limit budget.

What Happened

We use cert-manager with a single ClusterIssuer backed by Let’s Encrypt’s production ACME endpoint, shared across every namespace in the cluster, staging and production alike. A production API’s certificate came up for its automatic renewal and failed. Nobody caught it before it expired, since the previous alert on this had been tuned to check for certificates already expired, not certificates stuck failing to renew.

Where We Started Looking

Step 1 — Check the certificate’s own status

kubectl get certificate -n production api-tls

NAME      READY   SECRET       AGE
api-tls   False   api-tls-crt  58d

READY: False confirmed the renewal had failed, but not why.

Step 2 — Read the certificate’s conditions

kubectl describe certificate api-tls -n production

Status:
  Conditions:
    Message: Waiting for CertificateRequest to complete
    Reason: Pending
    Status: False
    Type: Ready

Generic and unhelpful on its own. The actual failure lives one level down, in the CertificateRequest and the Order it created.

Step 3 — Follow the chain down to the actual Order

kubectl get certificaterequest,order,challenge -n production

NAME                    APPROVED   DENIED   READY   REASON
order.acme.cert-manager.io/api-tls-abc12   False            pending

kubectl describe order api-tls-abc12 -n production

Status:
  Reason: Failed to create Order: too many failed authorizations
    recently: see https://letsencrypt.org/docs/rate-limits/

This was the first line that pointed anywhere real. Not a DNS problem with this certificate. A rate limit, tied to the ACME account, not to this specific hostname.

Step 4 — Find what else on the same account had been failing

kubectl logs -n cert-manager deploy/cert-manager | grep -i "authorization|challenge" | tail -50

The logs showed a different Ingress, in the staging namespace, retrying a DNS-01 challenge on every renewal cycle for the past three weeks and failing every single time.

Why an Unrelated Staging Cert Took Down Production

🔴 Root cause: a wrong Route53 zone ID on a staging Certificate quietly burned a rate limit shared with production.

A staging Certificate’s DNS-01 solver referenced the wrong hosted zone ID, an easy copy-paste mistake between two similarly named zones in the same AWS account.

Every renewal attempt tried to write a TXT record to a zone that didn’t match the domain, the DNS-01 challenge failed, and cert-manager retried on its normal backoff schedule, indefinitely.

Let’s Encrypt counts failed authorizations against the ACME account, not the individual domain. Enough failures from the broken staging cert exhausted the account’s failed-authorization limit entirely.

With that limit hit, every new Order from the same account failed, including the routine, otherwise-healthy renewal for the production certificate.

The staging misconfiguration had been silently failing for weeks with nobody watching it, since it was staging. Its failures were invisible right up until they blocked something that mattered.

The Fix

Immediate: fix the actual broken solver, then wait out the limit

# Before — wrong zone ID, copy-pasted from a similarly named zone
solvers:
  - dns01:
      route53:
        hostedZoneID: Z1D633PJN98FT9  # staging-old.acme.internal
        region: us-east-1

# After — corrected to the zone that actually matches the domain
solvers:
  - dns01:
      route53:
        hostedZoneID: Z2FDTNDATAQYW2  # staging.acme.internal
        region: us-east-1

Let’s Encrypt’s failed-authorization limit resets on a rolling window, not instantly. There was no way to force it clear early. We fixed the actual cause immediately and confirmed the account was clear of new failures, then retried the production certificate once the window passed:

kubectl delete certificaterequest api-tls-abc12 -n production
# cert-manager recreates the CertificateRequest and retries the Order automatically

Real fix: separate ACME accounts per environment

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-env
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-account-key

Two separate ACME accounts, one per environment, each with its own private key and its own rate limit budget. Note this is a second, separate production-endpoint account for the staging environment, not Let’s Encrypt’s staging directory, since staging certs still need to be trusted by real browsers for the team using them. A failure storm in staging now has its own limit to exhaust, and production’s stays untouched.

What We Put in Place After

1. Alert on Ready=False, not just on days-until-expiry

# cert-manager exposes certmanager_certificate_ready_status via its metrics endpoint
- alert: CertificateNotReady
  expr: certmanager_certificate_ready_status{condition="False"} == 1
  for: 1h
  labels:
    severity: warning

A certificate stuck failing to renew for an hour is worth paging on well before it actually expires. The previous alert only fired once expiry was already close, which is too late to catch a rate-limit exhaustion building up over weeks.

2. A pre-apply check on hosted zone IDs

A small script in CI now cross-references every hostedZoneID referenced in a Certificate or Issuer manifest against Route53’s actual zone list before merge, catching a copy-pasted zone ID before it reaches the cluster instead of after it starts failing silently.

3. A dashboard panel on ACME order failures per account

Grouped by ClusterIssuer, so a slow leak of failures in one environment is visible on a graph well before it accumulates into a rate-limit block on another.

Key Lessons

Let’s Encrypt rate limits are per-account, not per-domain.
A failure on one certificate can block issuance for every other certificate sharing the same ACME account.
A shared ClusterIssuer across environments shares its failure budget too.
Separate ACME accounts per environment keep one environment’s misconfiguration from spending another’s limit.
Certificate status alone rarely says why. The Order and Challenge resources do.
kubectl describe down the chain from Certificate to CertificateRequest to Order to find the actual ACME error.
Alert on renewal failure directly, not only on days-until-expiry.
A certificate stuck retrying for weeks gives no expiry-based warning until it’s already too late.
“It’s just staging” doesn’t mean its failures stay contained.
A staging misconfiguration with a shared dependency, an ACME account, a rate limit, a shared cluster resource, can reach into production without anyone touching production at all.

Summary

Layer	What Happened	Tool to Check
Staging Certificate	DNS-01 challenge failing every renewal for three weeks, wrong zone ID	`kubectl logs deploy/cert-manager`
ACME Account	Failed-authorization rate limit exhausted, shared across environments	`kubectl describe order`
Production Certificate	Routine renewal blocked by an unrelated environment’s failures	`kubectl get certificate`
Alerting	Tuned to expiry date, missed weeks of renewal failures	`certmanager_certificate_ready_status`

Production was never the source of the problem. It just happened to share an account with something that was.

A Typo’d Route53 Zone ID Quietly Exhausted Our ACME Rate Limit, and It Took Down an Unrelated Production Certificate