It was 3 AM, and my phone buzzed with an AWS alert I never wanted to see: “$18,452.93 — Forecasted Spend.” My stomach dropped. For a side project with less than 10 users, this was a catastrophe.
The Setup: Like many engineers, I often get drawn to the “latest and greatest.” My side project was a simple internal tool, but I decided to go all-in on a modern serverless architecture. API Gateway, Lambda, DynamoDB — the whole shiny stack. What could go wrong? Everything was behind IAM, locked down, or so I thought.
The Problem: My initial thought was a misconfigured Lambda loop. But after an hour of frantic digging, I found the culprit: a public-facing API endpoint that was supposed to be for internal use only, but was getting hammered by what looked like a botnet doing simple GET requests.
The sheer volume of requests, amplified by Lambda cold starts and API Gateway usage, was generating an insane amount of egress data and compute cycles. The worst part? It wasn’t even a “breach” in the traditional sense; it was just incredibly expensive traffic. My security groups were fine, my IAM roles were perfect for authorized users, but the API itself was simply… open.
Press enter or click to view image in full size
While troubleshooting, it hit me: the entire API didn’t need to be publicly accessible at all. It was an internal tool!
Become a member
My “Security-First” mindset, which usually focused on IAM and WAFs, had completely overlooked the most fundamental principle: if it doesn’t need to be on the public internet, don’t put it there.
The solution wasn’t some complex new AI-driven anomaly detection. It was a simple, “boring” OpenVPN server running on a $5 DigitalOcean droplet. I moved the API behind a private subnet, accessible only via that VPN.
The Lesson Learned: This $18,000 mistake taught me a critical lesson that every DevSecOps engineer needs to engrave in stone:
Public == Cost: If an endpoint is public, it’s a potential cost sink, even if “secure.”
“Boring Tech” is Reliable Tech: Sometimes, the simplest, oldest solutions are the most robust. A VPN isn’t sexy, but it works.
Security is Context: My security was great for authorized public access. It was terrible for unnecessary public access.
Press enter or click to view image in full size
I learned that true “Security-First” isn’t just about hardening endpoints; it’s about reducing the attack surface to zero wherever possible. Don’t put it on the public internet if it doesn’t need to be there. Your wallet (and your sleep) will thank you.