TLS

A Mistyped Route53 Zone ID Exhausted Our ACME Rate Limit — and Killed an Unrelated Cert

by
A Typo’d Route53 Zone ID Quietly Exhausted Our ACME Rate Limit, and It Took Down an Unrelated Production Certificate

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering Our production API certificate expired on schedule for its routine 60-day renewal, except cert-manager never got a new one issued. The cause had nothing to do with that certificate. A staging Ingress with a typo’d Route53 zone ID had been failing its own renewal silently for … Read more

Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

by
Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering At 3 AM, our ingress started rejecting every HTTPS request with a certificate error. Every health check we had was green. Every pod was Running. The cluster looked perfectly healthy because none of our monitoring ever asked the one question that mattered: how many days is … Read more