Un-Blackboxing vLLM: Building an AI SRE Copilot & FinOps Gateway with SigNoz

Un-Blackboxing vLLM: Building an AI SRE Copilot with SigNoz

When moving from external APIs (like OpenAI) to self-hosted open-source models, developers hit a wall: AI infrastructure is a black box. If an AI Agent hallucinates or costs explode, standard CPU metrics won’t tell you why.

For the Agents of SigNoz Hackathon (Track 1), I solved this by transforming a Kaggle T4 GPU running vLLM into an enterprise-ready system. I built an SRE Copilot and FinOps Gateway, but the real superhero of this architecture is SigNoz Cloud.

Here is how I used OpenTelemetry and SigNoz to make sense of complex AI data.

🏗️ The Tech Stack & Architecture

To treat AI like true enterprise infrastructure, I built three layers of custom observability in Python:

FinOps & SLO Gateway (FastAPI): A proxy that intercepts HTTP traffic, tracks mathematical Latency SLAs, and parses JSON token responses to calculate exact USD costs per internal team.

Hardware Exporter (pynvml): A custom script scraping raw NVIDIA GPU Power, Temperature, and Memory Utilization directly from the Kaggle kernel.

The Engine: vLLM running the Qwen 1.5B model.

All of this telemetry is batched via the OpenTelemetry Collector (otelcol-contrib) and shipped natively into SigNoz Cloud.

🦸‍♂️ SigNoz to the Rescue: The Dashboard

This is where SigNoz becomes the superhero. It takes highly complex, chaotic OpenTelemetry data and unifies it into a single pane of glass using PromQL. I built a custom dashboard to correlate business metrics with hardware physics across three critical sections:

1. 🛡️ Guardrails & SLAs

We enforce a strict 2.0s Latency SLA, using SigNoz to graph our Error Budget dynamically. More importantly, the Gateway runs Zero-Latency Heuristic Guardrails. If a model gets stuck in a hallucination loop, the Gateway detects the lack of semantic diversity. SigNoz instantly visualizes this drop in our "Output Quality Score" metric, alerting us to bad AI behavior without needing a slow, secondary LLM.

1. 💸 LLM FinOps

SigNoz makes FinOps easy. By parsing token usage, we visualize exact costs via this PromQL query: sum by (team) (increase(vllm_finops_cost_dollars_total[1h])) SigNoz instantly graphs this data, showing exactly which internal team is burning the AI budget in real-time.

1. 🖥️ Core Infrastructure

To catch OOM risks, we track Inference Queue Depth and KV Cache Usage. SigNoz allows us to plot these software queues directly alongside raw NVIDIA GPU Utilization, proving exactly when the physical hardware becomes the bottleneck.

🌊 The Stress Test

To prove the pipeline, I built a multi-threaded Python load generator simulating traffic from four teams. Crucially, it injects a "Poison Pill" prompt designed to force the AI into a hallucination loop.

Watching SigNoz react is incredible: the Queue Depth spikes, the SLA Error Budget plummets below zero, and the Output Quality Score crashes the exact second the poison pill hits. Because all this data flows natively into SigNoz, I configured PromQL Alert Rules so my Slack channel is instantly pinged when the Error Budget breaches zero.

By combining custom Python middleware with the power of OpenTelemetry, SigNoz proved it can flawlessly ingest and visualize deep hardware physics alongside complex semantic and financial AI metrics.