Introduction
Modern web applications live on the open internet, which means they’re constantly exposed to unpredictable traffic — from normal users, automated bots, and sometimes attackers. One of the most common threats is a DDoS attack, where a flood of requests overwhelms your server and makes your application unusable.
In this project, I built a real‑time anomaly detection engine that monitors incoming HTTP traffic, learns what “normal” looks like, and automatically reacts when something suspicious happens. It runs alongside a Nextcloud server and watches every request passing through Nginx.
This post walks through how I built it — in simple, beginner‑friendly language — and explains the core ideas behind sliding windows, baselines, anomaly detection, and automated blocking.
Why Build an Anomaly Detector?
Imagine you’re running a cloud storage platform. Everything is fine until suddenly:
One IP starts sending 500 requests per second
Or thousands of IPs spike traffic at the same time
Or a user triggers a burst of 4xx/5xx errors
If you wait for humans to notice, it’s already too late.
A good anomaly detector should:
Watch traffic continuously
Learn normal behavior automatically
Detect unusual spikes
Block malicious IPs
Alert you when something is wrong
Recover automatically when traffic returns to normal
That’s exactly what this project does.
How the System Works (High-Level)
The system has three main components:
Nginx — reverse proxy that writes JSON access logs
Detector Daemon — Python service that analyzes traffic
iptables — firewall used to block suspicious IPs
Here’s the flow:
A user makes a request
Nginx logs it in JSON format
The detector tails the log file in real time
It updates sliding windows of recent traffic
It compares traffic to a rolling baseline
If something looks abnormal, it triggers an action
The dashboard shows live metrics
Everything runs inside Docker.
Sliding Windows (Explained Simply)
A sliding window is just a list of recent events that automatically removes old ones.
Imagine a 60‑second window:
Every time a request comes in, we add its timestamp
Every second, we remove timestamps older than 60 seconds
The number of timestamps left = requests per 60 seconds
This gives us:
Per‑IP request rate
Global request rate
Per‑IP error rate
Global error rate
All in real time.
Why sliding windows?
Because they react instantly.
If an attacker sends 200 requests in 2 seconds, the window sees it immediately.
The Rolling Baseline (How the System Learns)
A baseline is the system’s idea of “normal traffic.”
Instead of hardcoding a threshold, we let the system learn from the last 30 minutes of traffic.
Every second, we record:
global requests per second
global error rate
Every 60 seconds, we compute:
mean (average traffic)
standard deviation (how much traffic usually varies)
This gives us a dynamic baseline that adapts to:
busy hours
quiet hours
natural fluctuations
Why this matters
If your site normally gets 5 requests per second, a spike to 50 is suspicious.
But if your site normally gets 200 requests per second, 50 is nothing.
The baseline makes the detector smart.
How Anomalies Are Detected
The detector uses two rules:
- Z‑Score Rule
If:
Code
(current_rate - baseline_mean) / baseline_stddev > 3
…it’s considered abnormal.
This is a common statistical rule:
3 standard deviations above normal = anomaly.
- Multiplier Rule
If:
Code
current_rate > 5 × baseline_mean
…it’s also an anomaly.
This catches cases where the standard deviation is small.
Error Surge Rule
If an IP suddenly generates lots of 4xx/5xx errors:
Code
ip_error_rate > 3 × baseline_error_mean
…the thresholds for that IP become stricter.
This helps catch brute‑force login attempts or broken bots.
Blocking Suspicious IPs
When an IP is flagged:
The detector inserts an iptables DROP rule
The IP is immediately blocked
An alert is written
The ban is stored with a timer
Unban Schedule
The system automatically unbans IPs:
After 10 minutes
After 30 minutes
After 2 hours
After that → permanent ban
This prevents accidental long‑term blocking of legitimate users.
The Dashboard
To make everything visible, I built a small FastAPI dashboard that shows:
Global requests per second
Error rates
Baseline mean/stddev
Top 10 IPs
Banned IPs
CPU and memory usage
System uptime
It refreshes every 3 seconds and is hosted on a public domain.
Putting It All Together
The final system:
Watches traffic in real time
Learns normal behavior
Detects anomalies
Blocks attackers
Unbans automatically
Logs everything
Shows live metrics
It’s a complete DevSecOps pipeline for traffic monitoring and automated defense.
What I Learned
This project taught me:
How to design real‑time systems
How to use sliding windows efficiently
How to build statistical baselines
How to detect anomalies without ML libraries
How to automate firewall rules
How to build dashboards for observability
How to structure a production‑grade Python daemon
It’s one of the most practical security tools I’ve ever built.
Conclusion
Anomaly detection doesn’t have to be complicated.
With the right architecture — sliding windows, baselines, and simple math — you can build a powerful real‑time defense system.
This project was a great exercise in DevOps, security, and distributed systems thinking.
If you’re learning DevSecOps, I highly recommend building something like this yourself.
If you wait for humans to notice, it’s already too late.
A good anomaly detector should:
SOCIAL SHARE CARD GENERATOR