The Midnight Server Meltdown: What To Do When Your Backend Goes Silent
Imagine this: it’s a peaceful night, maybe you’re winding down, enjoying a movie, or catching up on some sleep. Suddenly, your phone rings it's your client or your boss. Their voice is urgent:
“The server is down. Customers can’t access anything. Everything’s super slow!”
Your heart races. You scramble to your desk, half awake but fully alert now. You SSH into the server or open your monitoring dashboard nothing seems obviously broken. You check the logs they’re there, but not helpful. You hit a few endpoints they respond… eventually. What used to take 200ms is now crawling at 10 seconds or, even worse, timing out.
At this point, panic is knocking on your door.
You start digging deeper. CPU usage is spiking, memory is being eaten up, and your server is getting hammered with requests. But they aren’t normal requests they’re overwhelming, coming from a swarm of IPs. That’s when it hits you:
You're under a DDoS attack. Someone, somewhere, decided to bring your server to its knees.
Now the real question is what do you do to prevent this from happening again?
Set Up Rate Limiting
Limit how many requests each IP can make per second using Nginx. This helps slow down bots and flood attempts before they hurt your server.
If your using Nginx consider adding this inside nginx.conf
http {
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;
server {
listen 80;
location /api/ {
limit_req zone=mylimit burst=20 nodelay;
}
}
}
10r/s Limits each IP address to a maximum of 10 requests per second.
zone=mylimit:10m Creates a shared memory zone named
mylimitthat can track up to approximately 160,000 unique IP addresses with 10MB of memory.burst=20 Allows short traffic spikes by permitting up to 20 excess requests to queue temporarily before triggering rate limiting.
Monitor Abusive IPs
Keep an eye on which IPs are getting rate-limited. If the same ones show up repeatedly, it’s a red flag something is trying to break in or overwhelm your system.
Add the following to your nginx.conf to enable logging:
This ensures that NGINX logs blocked or rate-limited requests along with the client’s IP address.
http {
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;
log_format rate_limited '$remote_addr - $request - $status - $http_user_agent';
map $status $log_rate_limit {
503 1;
default 0;
}
access_log /var/log/nginx/ratelimited.log rate_limited if=$log_rate_limit;
server {
listen 80;
location /api/ {
limit_req zone=mylimit burst=20 nodelay;
}
}
}
Automatically Ban Suspicious IPs
Use tools like fail2ban to detect patterns of abuse and block the source IP for a set period. This adds another layer of defense beyond just rate limiting.
sudo apt update
sudo apt install fail2ban
/etc/fail2ban/jail.conf: Default config
/etc/fail2ban/jail.d/: Your custom jail files
/etc/fail2ban/filter.d/: Contains filters
Let’s say we want to block any IP that gets rate-limited too often (503 status in your Nginx log).
1. Create a custom jail
/etc/fail2ban/jail.d/nginx-limit.conf
[nginx-limit]
enabled = true
port = http,https
filter = nginx-limit
logpath = /var/log/nginx/ratelimited.log
maxretry = 10
findtime = 60
bantime = 600
- Ban IPs that match the filter 10 times in 60 seconds
- Ban them for 10 minutes
2. Create the filter file
/etc/fail2ban/filter.d/nginx-limit.conf
[Definition]
failregex = ^<HOST> - .+ - 503 - .*
This regex matches lines like:
192.168.1.5 - /api/endpoint - 503 - curl/7.58.0
It extracts the <HOST> (IP address) and counts the hits.
Start or Restart Fail2Ban
sudo systemctl restart fail2ban
Check Status and Banned IPs
sudo fail2ban-client status
Check specific jail:
sudo fail2ban-client status nginx-limit
Unban a specific IP (if needed):
sudo fail2ban-client set nginx-limit unbanip 192.168.1.5
Use reCAPTCHA to Stop Bots Cold
Especially for sensitive actions like login or guest access, adding Google reCAPTCHA can stop automated scripts in their tracks, without bothering real users much.
Thanks for reading my article 🙌
