The site was "up." The monitor said so. HTTP 200, response times normal, no alerts.
What the monitor didn't know - what I didn't know - was that our SSL certificate had expired 87 minutes earlier and every user hitting the site was getting a certificate error in their browser. Not a down page. Not a 5xx. A cert error. The kind where browsers show a big red warning screen and most users immediately close the tab.
For a checkout flow, that's about as bad as the server being down. Worse, actually, because at least a down server triggers your uptime alert.
This is the post-mortem.
What happened
We were running Let's Encrypt with certbot and auto-renewal configured. The renewal was supposed to happen when the cert had 30 days left. It had been working fine for about 18 months.
Then it didn't.
The renewal job ran, hit a DNS validation error - our DNS provider had a 30-minute API hiccup that day - and failed silently. Certbot logged the failure, but nobody was watching certbot logs. The retry ran 12 hours later, same issue. Then it was fine. But by then, the "success" window had passed and the cert expired before the next attempt.
Let's Encrypt auto-renewal fails for reasons that feel random at the time:
DNS propagation delays when you're using DNS-01 validation and your DNS provider has latency
Rate limits - Let's Encrypt has per-domain limits (5 failures per hour) that cause subsequent retries to also fail
Firewall or load balancer changes that block the HTTP-01 validation path on port 80
File permission issues on the cert directory after a system update
Webhook or deploy hook failures - the cert renews but the service doesn't reload to pick up the new cert
In our case it was DNS validation timing plus a log nobody was watching. The cert expired at 3:14 PM. The Slack alert - from a user, not a monitor - came in at 4:58 PM.
Why my uptime monitor missed it for four hours
for the cert expiry checks - it connects directly to port 443, validates the full chain, and alerts at 30 days and 7 days with enough context to know what's actually wrong (expiry date, issuer, which check failed).
For the renewal heartbeat: I have certbot's --deploy-hook send a ping to a GrabDiff heartbeat monitor after each successful renewal. If it doesn't ping within 93 days (the Let's Encrypt cert lifetime plus a week), I get alerted. That catches the silent renewal failures before they become a problem.
The six-hour checkout outage cost us - I'd rather not quantify it. The monitoring stack that would have caught it costs $9/month. That math is not complicated.
The broader lesson
SSL expiry is one of the most embarrassing categories of outage because it's entirely predictable. You know the cert will expire. You have the date. The only question is whether you catch it before your users do.
The same is true for domain expiry, for that matter. I've seen teams let their primary domain expire because the renewal email went to a former employee's address and nobody caught it. The monitoring there is trivial - check the WHOIS expiry date, alert 60 days out. But people don't do it until they have to learn the hard way.
If your current monitoring setup would have missed the scenario I described above - HTTP 200 from an expired-cert server - it's worth spending 20 minutes fixing that before you have your own version of this post-mortem to write.
I wrote this mostly to stop myself from having to explain this incident verbally ever again. Now I can just link it.
But seriously - SSL expiry outages are embarrassing in a specific way because they're so avoidable, and I've seen them happen to teams that clearly knew what they were doing otherwise. If you've had your own cert-expiry story (or a renewal failure that was weirder than mine), I'd like to hear it in the comments. Knowing the failure modes other people have hit is the only way to build a monitoring checklist that actually covers the real world.
SOCIAL SHARE CARD GENERATOR