Originally published on in production. Version 1.9.x added native sparse+dense hybrid search via the sparse_vectors config, which gives you BM25 keyword matching combined with semantic similarity in a single query — genuinely useful when alert names are exact-match keywords. If you're evaluating alternatives: Weaviate v1.24+ has the generative-openai module built in, which is tempting, but it couples your retrieval and generation layers tightly and makes model swaps painful. Pinecone namespaces work well if you're already in that ecosystem and don't need hybrid search.
Watch out for: Qdrant's default Docker image ships with zero authentication enabled. Always set the QDRANT_SERVICE_API_KEY environment variable and keep port 6333 inside a private subnet. I've seen this misconfiguration in three separate internal tooling audits.
Tip 3: Ingest Runbooks from Confluence or Git with a Lightweight Pipeline
Hash-based change detection keeps your vector store fresh without re-embedding everything on every run.
The ingestion pipeline is where most RAG implementations get lazy and end up paying for it — either in stale data or in runaway embedding API costs. The pattern I use: store a sha256 of each document's content in Redis. On every pipeline run, compare the current hash. If it matches, skip re-embedding entirely. Only new or changed content hits the embedding model.
For Git-based runbooks, enforce a path convention: docs/runbooks/{service}/{alert_name}.md. This lets you extract service and alert_name metadata directly from the file path without parsing file content — simpler and less error-prone. For Confluence, the REST API endpoint /wiki/rest/api/content?type=page&spaceKey=SRE works, and LangChain's ConfluenceLoader (requires atlassian-python-api>=3.41.0) gets you started fast. That said, I moved off it to a custom fetch — you get better metadata control and don't inherit LangChain's chunking decisions.
Here's the full ingestion pipeline with hash-based deduplication and Redis embedding cache:
# rag_ingest.py — Runbook ingestion pipeline with hash-based deduplication
# Deps: qdrant-client>=1.9.0, sentence-transformers>=2.7.0, python-dotenv, redis, tiktoken
import os
import hashlib
import json
from pathlib import Path
from dotenv import load_dotenv
import redis
from qdrant_client import QdrantClient
from qdrant_client.models import (
Distance, VectorParams, PointStruct, Filter,
FieldCondition, MatchValue
)
from sentence_transformers import SentenceTransformer
load_dotenv()
# --- Config ---
QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
QDRANT_API_KEY = os.getenv("QDRANT_API_KEY")
COLLECTION_NAME = "sre_runbooks"
EMBED_MODEL = "BAAI/bge-small-en-v1.5" # 384-dim, fast, good on technical text
CHUNK_SIZE = 512 # tokens
CHUNK_OVERLAP = 64 # token overlap to preserve step continuity
SCORE_THRESHOLD = 0.78 # minimum cosine similarity to surface a result
# --- Clients ---
redis_client = redis.Redis(host="localhost", port=6379, decode_responses=True)
qdrant = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)
model = SentenceTransformer(EMBED_MODEL)
def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
"""Split on word boundaries respecting overlap — avoids mid-step cuts."""
words = text.split()
chunks, i = [], 0
while i < len(words):
chunk = " ".join(words[i:i + size])
chunks.append(chunk)
i += size - overlap # slide with overlap
return chunks
def embed_with_cache(text: str) -> list[float]:
"""Return cached embedding or compute and store it."""
key = f"emb:v1:{hashlib.sha256(text.encode()).hexdigest()}"
cached = redis_client.get(key)
if cached:
return json.loads(cached)
vector = model.encode(text, normalize_embeddings=True).tolist()
redis_client.setex(key, 604800, json.dumps(vector)) # TTL: 7 days
return vector
def ingest_runbook(filepath: Path):
"""Parse path for metadata, chunk content, upsert to Qdrant."""
# Expected path: docs/runbooks/{service}/{alert_name}.md
parts = filepath.parts
service = parts[-2] if len(parts) >= 2 else "unknown"
alert_name = filepath.stem # filename without .md
content = filepath.read_text(encoding="utf-8")
doc_hash = hashlib.sha256(content.encode()).hexdigest()
# Fast change detection via Redis — skip unchanged docs entirely
hash_key = f"doc_hash:{filepath}"
if redis_client.get(hash_key) == doc_hash:
print(f"[SKIP] {filepath} unchanged")
return
chunks = chunk_text(content)
points = []
for idx, chunk in enumerate(chunks):
vector = embed_with_cache(chunk)
point_id = int(hashlib.sha256(f"{filepath}:{idx}".encode()).hexdigest()[:8], 16)
points.append(PointStruct(
id=point_id,
vector=vector,
payload={
"service": service,
"alert_name": alert_name,
"chunk_index": idx,
"source_path": str(filepath),
"doc_hash": doc_hash,
"text": chunk,
}
))
qdrant.upsert(collection_name=COLLECTION_NAME, points=points)
redis_client.set(hash_key, doc_hash) # update change-detection cache
print(f"[OK] Ingested {len(points)} chunks from {filepath}")
def ensure_collection():
"""Create collection if it doesn't exist."""
existing = [c.name for c in qdrant.get_collections().collections]
if COLLECTION_NAME not in existing:
qdrant.create_collection(
collection_name=COLLECTION_NAME,
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
print(f"[INIT] Created collection: {COLLECTION_NAME}")
if __name__ == "__main__":
ensure_collection()
runbook_dir = Path("docs/runbooks")
for md_file in runbook_dir.rglob("*.md"):
ingest_runbook(md_file)
Tip 4: Wire the RAG Query into Your Alerting Workflow
Surface runbook context automatically when an alert fires — not only when someone thinks to ask.
The real value of a RAG pipeline for SRE runbooks isn't a chat interface. It's injecting relevant procedure context into the incident notification itself, before the engineer even opens a terminal. The integration point is your Alertmanager or PagerDuty webhook. When a webhook fires, extract the alertname label (Alertmanager v2 path: .alerts[0].labels.alertname) and use it as the query string to your RAG endpoint.
One PagerDuty-specific gotcha: webhook v3 sends event.data.title as the incident name. Map this field, not event.id, to your query — I've seen this wired wrong in three different integrations and the resulting queries return garbage.
Set a similarity score threshold of 0.78 with cosine distance as your starting point. Below that, return a "matched": false signal so your Slack notification can still fire — just without a runbook attachment. A "no confident match" message is far safer than surfacing a low-confidence wrong runbook. Return the top-3 chunks maximum; more than that and engineers stop reading them.
Here's the FastAPI query endpoint wired to an Alertmanager webhook payload:
# rag_query.py — Query endpoint wired to Alertmanager webhook
# Receives alert payload, returns top-3 runbook chunks above threshold
import os
from fastapi import FastAPI, Request, HTTPException
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
from sentence_transformers import SentenceTransformer
QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
QDRANT_API_KEY = os.getenv("QDRANT_API_KEY")
COLLECTION_NAME = "sre_runbooks"
SCORE_THRESHOLD = 0.78
TOP_K = 3
app = FastAPI()
qdrant = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
@app.post("/query/alert")
async def query_from_alert(request: Request):
"""
Accepts Alertmanager webhook JSON.
Extracts alertname + service label, runs filtered vector search.
Returns top-K chunks or a no-match signal.
"""
body = await request.json()
try:
# Alertmanager v2 webhook schema
alert = body["alerts"][0]
alert_name = alert["labels"]["alertname"] # e.g. "HighMemoryUsage"
service = alert["labels"].get("service", None) # optional label
except (KeyError, IndexError):
raise HTTPException(status_code=400, detail="Invalid Alertmanager payload")
query_text = f"{alert_name} {service or ''}".strip()
query_vector = model.encode(query_text, normalize_embeddings=True).tolist()
# Pre-filter by alert_name metadata before semantic ranking
search_filter = Filter(
must=[FieldCondition(key="alert_name", match=MatchValue(value=alert_name))]
) if alert_name else None
results = qdrant.search(
collection_name=COLLECTION_NAME,
query_vector=query_vector,
query_filter=search_filter,
limit=TOP_K,
score_threshold=SCORE_THRESHOLD, # drop low-confidence results
with_payload=True,
)
if not results:
# Fallback: no confident match — Slack still pages, just without runbook
return {"matched": False, "alert_name": alert_name, "chunks": []}
return {
"matched": True,
"alert_name": alert_name,
"chunks": [
{
"text": r.payload["text"],
"source": r.payload["source_path"],
"score": round(r.score, 4),
"chunk_index": r.payload["chunk_index"],
}
for r in results
],
}
# Example response:
# {
# "matched": true,
# "alert_name": "HighMemoryUsage",
# "chunks": [
# {"text": "Step 1: check OOMKilled pods with kubectl describe...",
# "source": "docs/runbooks/api/HighMemoryUsage.md",
# "score": 0.8912, "chunk_index": 2}
# ]
# }
For Slack delivery, use Block Kit's section block with a mrkdwn text field to render the runbook chunk inline alongside the alert details. Include the source_path and score so engineers immediately know where it came from and how confident the match is.
Tip 5: Evaluate Retrieval Quality Before You Trust It in Production
The silent failure mode is a RAG that returns plausible-but-wrong runbook steps with high confidence.
Most teams evaluate their RAG pipeline by asking "does the LLM answer look right?" That's the wrong question. You need to evaluate whether the retrieved chunks were actually the correct runbook sections before any LLM even sees them. A well-phrased wrong answer is worse than an obvious failure.
Build a golden dataset: 20-30 pairs of (alert_name, expected_runbook_section). Run recall@3 checks — does the correct chunk appear in the top 3 results? That's your baseline metric. For a more structured eval, the .
Tip 7: Control Costs by Caching Embeddings and Limiting Re-Indexing Frequency
Naive re-indexing pipelines multiply embedding costs fast — cache aggressively and schedule smart.
At first glance, embedding costs look trivial. Five hundred runbook pages at roughly 10 chunks each, priced at text-embedding-ada-002's $0.0001 per 1K tokens, works out to about $0.25 per full re-index. That sounds fine. But a naive pipeline that re-embeds everything on every CI merge, or that re-indexes when Confluence sends a webhook for a minor edit, turns that $0.25 into a daily charge. At scale with a self-hosted GPU model, it becomes compute time you're burning for no reason.
The fix is two-layered. First, the Redis embedding cache with key pattern emb:v1:{sha256(chunk_text)} — identical chunk content across different documents or pipeline runs hits the cache, not the model. Include a version prefix (v1) so that when you upgrade your embedding model, you can invalidate the entire cache cleanly by bumping to v2 without touching cache logic. Second, schedule full re-indexes weekly. Run incremental re-indexing (changed documents only, via hash comparison) on every merge to main. This keeps the index current without re-embedding stable content.
One more cost lever: use gRPC instead of HTTP for Qdrant batch upserts. The default HTTP port is 6333, gRPC is 6334. Switching to gRPC gives approximately 30% lower latency on batch operations — not a cost saving directly, but it reduces the wall-clock time your ingestion job runs, which matters if you're paying for the compute that runs it.
SOCIAL SHARE CARD GENERATOR