How I Cut LLM Inference Costs by 78% Without Sacrificing Quality
We were spending $14,200/month on inference for our internal coding assistant and customer support bot. Every request hit a Llama-3.1-70B instance via vLLM, regardless of complexity.
The pain points were immediate:
Cost bleed: 64% of traffic was simple intent classification or RAG lookups. 70B model was overkill.
Latency spikes: P99 hovered at 1.4 seconds. Simple queries queued behind complex reasoning tasks.
Throughput ceiling: ~120 req/s max on g6e.4xlarge instances.
Here's what actually moved the numbers:
- Monthly cost: $14,200 → $3,100 (-78%)
- P99 latency: 1.4s → 0.81s (-42%)
- Throughput: 120 → 450 req/s (+275%)
The Problem With Static Model Selection
Most tutorials compare models in isolation. They run llm.generate() and compare MMLU scores. They don't address production dynamics: variance in query complexity.
A common bad approach is length-based routing:
# BAD: Length-based routing fails on complexity
if len(prompt) < 200:
return call_small_model(prompt)
else:
return call_large_model(prompt)
This fails catastrophically. A 50-token prompt asking for "Refactor this recursive algorithm to iterative with O(1) space complexity" is infinitely more complex than a 500-token prompt asking "Summarize this email." Length correlates poorly with computational difficulty.
The Solution: Dynamic Routing Topology
The paradigm shift is treating your model stack as a tiered compute resource, not a monolith.
We deployed a Qwen2.5-1.5B-Instruct model as a dedicated "Router." It scores every incoming prompt on a semantic complexity scale of 0-10 using a lightweight embedding-based heuristic combined with the small model's self-assessment.
Architecture Overview
User Request
│
▼
┌─────────────────┐
│ Router Model │ ← Qwen2.5-1.5B-Instruct (FP16)
│ (Complexity 0-10)│ Scores complexity in <5ms
────────┬────────┘
│
┌────┴────┐
│ │
Score≤4 Score>4
│ │
▼ ▼
┌───────┐ ┌─────────┐
│ 8B │ │ 70B │ ← Only 15% of traffic hits this
│ Model │ │ Model │
└───────┘ └─────────┘
Router Implementation
from sentence_transformers import SentenceTransformer
import numpy as np
class ComplexityRouter:
def __init__(self):
self.encoder = SentenceTransformer('all-MiniLM-L6-v2')
self.threshold = 4.0
# Complexity seed sentences for few-shot calibration
self.seeds = {
'simple': [
"What is Python?",
"Format this JSON",
"Summarize this paragraph",
],
'complex': [
"Refactor this recursive algorithm to iterative with O(1) space",
"Explain the CAP theorem trade-offs in distributed databases",
"Design a rate limiter with sliding window and token bucket",
]
}
self.seed_embeddings = {
'simple': self.encoder.encode(self.seeds['simple']),
'complex': self.encoder.encode(self.seeds['complex']),
}
def score(self, prompt: str) -> float:
"""Returns complexity score 0-10."""
embedding = self.encoder.encode([prompt])[0]
# Cosine similarity to simple vs complex seeds
sim_simple = max(
np.dot(embedding, s) / (np.linalg.norm(embedding) * np.linalg.norm(s))
for s in self.seed_embeddings['simple']
)
sim_complex = max(
np.dot(embedding, s) / (np.linalg.norm(embedding) * np.linalg.norm(s))
for s in self.seed_embeddings['complex']
)
# Score: 0 = very simple, 10 = very complex
return 10 * (sim_complex / (sim_simple + sim_complex + 0.01))
def route(self, prompt: str) -> str:
score = self.score(prompt)
return 'llama-8b' if score <= self.threshold else 'llama-70b'
Self-Assessment Layer
The router alone isn't enough. The 8B model also self-assesses its confidence:
def generate_with_confidence(model, prompt):
response = model.generate(
prompt,
extra_body={
'response_format': {'type': 'json_object'},
}
)
# Parse confidence from structured output
result = json.loads(response.text)
confidence = result.get('confidence', 0.5)
content = result.get('content', '')
# If confidence is low, escalate to 70B
if confidence < 0.7:
return model_70b.generate(prompt)
return content
Results
| Metric | Before | After | Change |
|---|---|---|---|
| Monthly cost | $14,200 | $3,100 | -78% |
| P99 latency | 1,400ms | 810ms | -42% |
| Max throughput | 120 req/s | 450 req/s | +275% |
| Quality (eval score) | 92.1% | 91.8% | -0.3% |
The traffic split settled at 85/15: 85% of requests routed to the 8B model, 15% to the 70B model. The 8B model handles 94% of queries with zero detectable quality degradation in our eval harness.
The Key Insight
"Your biggest cost isn't the token price; it's the compute wasted on simple queries hitting a 70B parameter model. A 1.5B router pays for itself within 400 requests."
The router model runs on a single CPU core. Its cost is negligible compared to the GPU savings from not sending every query to the 70B model.
Production Tips
Start with length-based routing, then graduate to embedding-based complexity scoring
Monitor the escalation rate — if >30% of traffic hits the 70B model, your threshold is too low
Cache router results for repeated prompts (e.g., system prompts, common queries)
A/B test the 8B output quality weekly against the 70B baseline to catch model drift
Full production architecture guide: https://www.codcompass.com
SOCIAL SHARE CARD GENERATOR