Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

Our TLS Certificate Expired in Production Because Every Alert We Had Was Watching the Wrong Thing

by

By KP  |  TZoneLabs  |  DevOps & Cloud Engineering

At 3 AM, our ingress started rejecting every HTTPS request with a certificate error. Every health check we had was green. Every pod was Running. The cluster looked perfectly healthy because none of our monitoring ever asked the one question that mattered: how many days is left on this certificate?

We run cert-manager on EKS with Let’s Encrypt, renewing 30 days before expiry through a DNS-01 challenge. That gives a full month of margin. We still burned through all of it without noticing.

What Actually Failed

A DNS provider API token used for the DNS-01 challenge got rotated during a routine credential cleanup. Nobody updated the Secret cert-manager reads it from. The next scheduled renewal, 30 days before expiry, failed authentication against the DNS provider, and cert-manager logged the failure and moved on.

It kept retrying. It kept failing the same way, quietly, for weeks. The Certificate resource sat with Ready: False the entire time. Nothing paged on that condition because nothing was watching it.

Why Every Health Check Missed It

Our readiness and liveness probes hit an internal HTTP endpoint on the pod directly, bypassing TLS termination entirely. Our uptime monitor checked for a 200 response, also over plain HTTP behind the load balancer. Neither one ever negotiated a real TLS handshake against the public certificate, so neither one could ever see it expiring.

The certificate could be one day from expiry or thirty, and every dashboard we had would show the same thing: green.

The Fix: Watch the Certificate, Not Just the Pod

Two changes closed the gap.

First, a blackbox exporter probe against the actual public endpoint, checking the real TLS handshake from outside the cluster:

- job_name: 'blackbox-tls'
  metrics_path: /probe
  params:
    module: [https_2xx]
  static_configs:
    - targets:
        - https://app.example.com
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - target_label: __address__
      replacement: blackbox-exporter:9115

Paired with an alert on probe_ssl_earliest_cert_expiry, firing 14 days out instead of finding out at the moment of expiry:

- alert: TLSCertExpiringSoon
  expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "TLS cert for {{ $labels.instance }} expires in under 14 days"

Second, an alert directly on cert-manager's own state, so a stuck renewal pages someone before it becomes an outage:

- alert: CertManagerCertNotReady
  expr: certmanager_certificate_ready_status{condition="False"} == 1
  for: 6h
  labels:
    severity: critical
  annotations:
    summary: "cert-manager Certificate {{ $labels.name }} has been not-ready for 6h+"

That second one would have caught this incident on day one of the failed renewal, not day thirty.

Key Lessons

  1. A green health check only proves the thing it actually checks.
    Ours checked pod liveness, not certificate validity. Two different failure modes, one dashboard.
  2. Internal health checks that skip TLS termination can't catch TLS failures.
    If the probe never does a real HTTPS handshake against the public endpoint, it will never see the certificate expiring.
  3. A resource stuck in a bad state for weeks is a monitoring gap, not bad luck.
    cert-manager told us Ready: False the entire time. We just weren't listening.
  4. Rotating a credential means finding everything that reads it.
    The DNS token rotation was correct security hygiene. The miss was not tracing where else that token was used.
  5. Alert on the resource that owns the risk, not just the service it protects.
    Watching the Certificate object directly gave us weeks of lead time that watching the app alone never could.

Summary

Gap What Happened Fix
Health checks skip TLS Probes hit HTTP behind the load balancer Blackbox probe against the real public HTTPS endpoint
No cert expiry alerting Nothing tracked days-until-expiry Alert on probe_ssl_earliest_cert_expiry at 14 days
Silent renewal failures cert-manager retried and failed for weeks unnoticed Alert on certmanager_certificate_ready_status
Untracked credential usage DNS token rotated without updating cert-manager's Secret Document every consumer of a credential before rotating it

The certificate didn't fail without warning. It failed without anyone watching the one signal that mattered, for thirty straight days.

Read Next

If you're running Kubernetes in production, follow along on LinkedIn for more incident write-ups like this one.

Tags:
#Kubernetes   #CertManager   #TLS   #Monitoring  
#DevOps   #SRE   #IncidentManagement   #EKS

Leave a Comment