Netflix Headroom: How to Cut AI Agent Costs 10x in Production [2026]

Originally published at . A 10-step agent loop doesn't cost 10x a single call. It costs closer to 50x, because each step re-reads the entire conversation history, tool outputs, and system instructions. Netflix built Headroom to fix exactly this, and workloads without sacrificing output quality.

This isn't a research paper or a toy demo. It's a production system from a company running ML at planet scale. And the patterns inside Headroom are ones any engineering team can steal today.

Why AI Agent Costs Spiral Out of Control

Before I get into what Headroom does, let's talk about why you need it.

LLM inference costs in agentic workflows scale non-linearly. That's the part most teams don't internalize until the invoice lands. Here's the math: if your agent takes 10 steps and you're appending tool outputs and conversation history at each turn, the context window grows roughly like a triangle. Step 1 sends maybe 2,000 tokens. Step 5 sends 12,000. Step 10 sends 25,000+. You're paying for all that accumulated context on every single call.

The majority of your token spend isn't generation. It's input. Most teams I've talked to find that 70-85% of their LLM bill comes from the input/context side. Your agent is re-reading the same system prompt, the same tool definitions, and most of the same conversation history on every turn. Pure waste.

I've seen teams running argues on Dev.to, the root cause is simple: teams feed agents raw, unfiltered data. Every API response, every database query result, every intermediate tool output goes straight into the context window without any pre-processing. It's the LLM equivalent of piping cat output straight to your model and hoping for the best.

What Is Netflix Headroom and How Does It Work?

Headroom is an open-source context optimization layer that Netflix built internally and presented at presented this as a unified system rather than three separate optimizations, and that's the real insight. I've seen plenty of teams apply one of these techniques in isolation and call it a day. Headroom applies all three in a coordinated pipeline. The compounding effect is what gets you to 10x.

Here's the official talk from Open Source Summit NA 2025:

[YOUTUBE:UOWSHg18cL0|Headroom: A Context Optimization Layer for LLM Applications - Tejas Chopra, Netflix, Inc.]

Think of it like a CDN for your LLM calls. A CDN doesn't change the content your server produces. It makes delivering that content cheaper by caching, compressing, and routing intelligently. Headroom does the same thing for context windows.

Context Pruning: The Highest-Leverage Cost Reduction Lever

Context pruning is the single biggest bang-for-your-buck optimization in any discipline. I've seen engineers obsess over prompt wording while completely ignoring prompt size. They're optimizing the wrong variable. A 2,000-token prompt that contains exactly the right context will outperform a 20,000-token prompt padded with irrelevant history. And cost 10x less.

Prompt Caching: Stop Paying for the Same Tokens Twice

Prompt caching is the optimization most teams know about but few implement well. The core idea: if the first 3,000 tokens of your prompt are identical across calls (system prompt + tool definitions + static instructions), you shouldn't pay full price for those tokens every time.

All three major providers now support some form of prompt caching. Anthropic's Claude, Google's Gemini, and OpenAI all offer mechanisms to reuse previously computed attention states for stable prefixes. The savings are real. workloads at scale and sending the same system prompt on every call without cached prefixes, you're lighting money on fire. Full stop.

Tiered Model Routing: Use the Right Model for Each Step

This is the optimization that feels obvious in hindsight but almost nobody does well. Not every step in an agent workflow requires GPT-4-class reasoning. Some steps are classification tasks. Some are simple data extraction. Some are formatting. Sending all of these to your most expensive model is like taking a Ferrari to buy milk.

Tiered model routing means having a router that analyzes each agent step and directs it to the cheapest model capable of handling it:

Task Type	Model Tier	Example Models	Relative Cost
Complex reasoning, multi-step planning	Frontier	GPT-4.1, Claude Sonnet 4.6	1.0x (baseline)
Summarization, moderate analysis	Mid-tier	GPT-4o Mini, Claude Haiku 4.5	0.05-0.1x
Classification, extraction, formatting	Small/local	Gemma 4 12B, Llama 3 8B	0.01-0.02x
Simple routing, intent detection	Tiny/edge	Phi-3, distilled models	<0.01x

Look at the cost column. A step that costs $0.03 on a frontier model might cost $0.001 on a mid-tier model and $0.0001 on a frameworks that already decompose tasks into discrete steps. The router examines the step's requirements (does it need tool calling? long-context reasoning? or is it a simple yes/no decision?) and selects the appropriate tier. Headroom includes this routing logic as a core component.

The key insight from Netflix's approach: routing decisions should be data-driven, not hardcoded. You start by sending everything to the frontier model, log the results, then progressively shift simpler steps to cheaper models while monitoring quality. If quality stays above your threshold, keep shifting. It's A/B testing for model selection.

How Any Team Can Apply These Patterns Without Netflix's Budget

Here's the thing nobody's saying about Headroom: the individual techniques aren't novel. Context pruning, prompt caching, and model routing have all been discussed in the LLMOps community for over a year. What Netflix did is package them into a coherent, production-tested system. But you don't need their system to use their playbook.

Here's how I'd implement this incrementally, based on how I've actually rolled out similar optimizations:

Week 1: Instrument your context windows. Before you optimize anything, measure. Log the token count at each step of your agent loops. Calculate what percentage is system prompt, what's conversation history, what's tool outputs. I promise you'll be shocked at how much redundancy you're carrying. When I first did this on a project, I found that 62% of tokens at step 8 were from tool outputs the model never referenced again.

Week 2: Implement prompt caching. Lowest effort, highest impact. Restructure your prompts so stable content comes first, enable your provider's caching feature, and measure the savings. If you're on Claude, Gemini, or OpenAI, this is a configuration change, not an architecture change.

Week 3: Add basic context pruning. Start with the easy wins: truncate tool outputs to only the fields your model needs, summarize or drop conversation turns older than N steps, conditionally include system prompt sections. Even a crude implementation will cut 30-40% of your token spend.

Week 4: Prototype model routing. Identify 2-3 step types in your agent workflow that clearly don't need frontier-model reasoning. Route those to a cheaper model. Measure quality. Expand from there.

This four-week playbook can realistically get you a 5-8x cost reduction. The remaining push to 10x requires more sophisticated pruning (relevance scoring, semantic deduplication) and fine-tuned routing logic. That's where Headroom's codebase becomes genuinely useful as a reference architecture.

For teams already using frameworks like , many of these optimizations can be implemented as middleware layers or callbacks without restructuring your entire agent pipeline.

Why Netflix Open-Sourcing This Matters Right Now

The LLMOps tooling ecosystem is fragile. Case in point: has published extensively on their ML infrastructure. They run recommendation systems, content understanding models, and personalization engines across 260+ million subscribers. Their cost management is now a first-class engineering discipline. Not an afterthought you bolt on after launch. Part of the architecture from day one.

The Broader LLMOps Stack: What Else Matters for Cost

Headroom addresses the context side of the cost equation, but it's one lever among several. Having built over stuffing. Stop cramming your entire knowledge base into the context window. Use to retrieve only the relevant chunks. This is context pruning applied to external knowledge, and it's one of the most commonly missed optimizations I see.

Structured outputs over free-form generation. When you need the model to return data in a specific format, use without a context optimization layer by 2027. It'll be considered negligent. The same way running production databases without connection pooling is considered negligent today.

The local inference movement — running ), you eliminate API costs entirely for those tiers. I've written about

Vollständiger Original-Artikel

Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

↗ Original-Artikel auf dev.to lesen