Virtual keys per tenant: ditching our custom LLM billing layer

TL;DR: We had 11,247 lines of Python middleware handling per-tenant LLM cost attribution, rate limiting, and provider failover. Replaced about 60% of it with Bifrost's virtual keys and governance features. Some honest gaps remain, which is why this is a writeup and not a sales pitch.

The setup we inherited

Nexus Labs runs enterprise agent automation. Each customer gets isolated workloads. Each workload makes between 200 and 50,000 LLM calls per day across OpenAI, Anthropic, Bedrock, and Vertex.

When I joined, we had a Python middleware doing four things at once: API key rotation per provider, per-tenant rate limits in Redis, cost attribution via request tagging, and fallback logic when a provider returned 429s.

11,247 lines of Python. Three engineers had touched it. Two had left. One of them had encoded their team-internal pricing assumptions inline. Every model deprecation became a sprint.

What we actually needed

Three things, in priority order:

1. Per-customer spend caps that don't require a deploy to update.
2. Provider failover that survives Anthropic going down for 23 minutes (it did, last March).
3. Cost data we don't have to reconstruct from CloudWatch logs.

I evaluated three gateways before picking one. Here is the comparison after running each through a 2-week eval against our actual traffic shape.

LiteLLM was the real contender. Larger community, more battle-tested in production for some workload shapes. Where it lost for us: setting up hierarchical budgets across customer to team to workload tiers required more YAML wrangling than we wanted, and the failover behavior on streaming requests was less predictable under our tests.

Portkey was strong on dashboards. We didn't want a hosted dependency for our cost control path.

What changed

The piece that surprised me most was the virtual keys model. From the docs (governance/virtual-keys), every tenant gets a virtual key. The key carries the budget cap, rate limit, allowed providers, and allowed models. Our orchestrator stopped caring about provider routing entirely.

Config that replaced 4,200 lines of Python:

virtual_keys:
  - id: vk_acme_prod
    customer_id: acme_corp
    budget:
      max_per_month_usd: 12000
      reset_duration: monthly
    rate_limit:
      requests_per_minute: 600
    allowed_providers:
      - openai
      - anthropic
      - bedrock
    fallbacks:
      - provider: openai
        model: gpt-4o
      - provider: anthropic
        model: claude-sonnet-4-6
      - provider: bedrock
        model: anthropic.claude-sonnet-4-6

Our orchestrator now does one thing: pick a virtual key based on tenant. Send the request. Done.

The numbers

Before:

• 11,247 LOC in gateway_middleware/
• p95 added latency from middleware: 47ms
• Mean time to add a new model: 2 days (testing, rollout, monitoring)

After 4 months:

• 4,108 LOC remaining (mostly business logic we still need)
• p95 added latency from Bifrost in front: 8ms
• Mean time to add a new model: under an hour

The latency number was the biggest surprise. Bifrost is Go. Our middleware was Python doing synchronous Redis calls. We knew that was a problem. Solving it wasn't on the roadmap.

Trade-offs and Limitations

This isn't free.

Migration was harder than the docs suggest. Our cost attribution data didn't map cleanly. We had legacy fields like team_internal_billing_code baked into every log. Mapping these to virtual key metadata took a full sprint, and the team still grumbles about it.

Semantic caching is risky for our workload. Our agents call LLMs with tool results embedded in prompts. Two prompts that look 92% similar can require very different responses. We disabled semantic caching for the agent path. Enabled it only for our content generation path, where we saw a 31% hit rate.

MCP gateway integration is newer than the rest. We use it for filesystem access from a customer-facing automation agent. Works fine. But debugging when a tool call fails requires more log digging than the rest of the platform.

No native cost-anomaly alerting yet. Budget caps work. But "this customer's usage spiked 3x in 2 hours" is still wired up via Prometheus alerts and PagerDuty by hand. Portkey has this in their hosted product. If real-time anomaly alerts are your top requirement, weight that.

What I'd tell a peer team

If you have one provider and one customer, you don't need this. Use the provider's SDK.

If you have 3+ providers, multiple customer tiers, and someone on your team has written class CostTrackingMiddleware more than once, evaluate. Spin up the Docker container (quickstart). Point staging traffic at it for a week. Look at the metrics. Decide.

The model is the easy part. Cost attribution is the part that wakes you up at 2am when a customer's bill is wrong.

Further Reading

• Bifrost virtual keys docs
• Budget management hierarchy
• Bifrost GitHub repo
• LiteLLM proxy docs (worth comparing)
• Drop-in replacement notes