I just released v1.0.0 of an open-source, production-ready real-time analytics pipeline built with Python. Here's what it does and why you might care.
The Problem
Every SaaS product needs analytics — event tracking, real-time dashboards, time-series aggregations. Most teams either pay for Segment/RudderStack or build their own from scratch. This project is the "build your own" done right.
Architecture
Client → FastAPI → Redis Streams/Kafka → Event Processor → TimescaleDB → WebSocket → Dashboard
Tech Stack
| Component | Technology |
|---|---|
| API Layer | FastAPI (async, auto-docs, WebSocket native) |
| Event Queue | Redis Streams or Apache Kafka |
| Storage | TimescaleDB (PostgreSQL extension for time-series) |
| Real-time | WebSocket with JWT auth + auto-reconnect |
| Metrics | Prometheus + OpenTelemetry |
| Logging | Structured JSON with correlation IDs |
| Deployment | Docker Compose + Kubernetes |
Features
- Async Event Ingestion — REST API + batch endpoints, buffered via Redis Streams or Kafka
- Adaptive Sampling — configurable rate-based sampling per event type
- Data Retention — TTL-based policies with automatic partition management
- Enterprise Security — JWT auth, RBAC (admin/editor/viewer), rate limiting, security headers, correlation IDs
- Live Dashboards — WebSocket push with auto-reconnect
- Observability — Prometheus metrics, OpenTelemetry traces, structured JSON logs
- Production Ready — Docker multi-stage build, Kubernetes manifests, health checks
Quick Start
git clone https://github.com/aman179102/real-time-analytics-pipeline
cd real-time-analytics-pipeline
make install
docker compose up -d postgres redis
make migrate
make run-dev
Testing
150 out of 152 unit tests pass. The 2 excluded tests are pre-existing async timing issues in process_loop — zero regressions introduced.
Enterprise Middleware Pipeline
Every request flows through:
CorrelationMiddleware → AuthMiddleware → SecurityHeadersMiddleware → RateLimitMiddleware → SizeLimiterMiddleware → Router
Try It Out
GitHub: https://github.com/aman179102/real-time-analytics-pipeline
Star the repo if you find it useful! Contributions and feedback are always welcome.
SOCIAL SHARE CARD GENERATOR