Cache-Aware Spawning: What Changed in llm-cli-gateway, a Week On

If your multi-LLM workload sends the same long system prompt or file dump to Claude / Codex / Gemini ten times an hour, you are paying for the same input tokens ten times. Each provider has a cache for exactly this case, and each one expresses the cache differently. This post is about how llm-cli-gateway now uses those caches for you, across all five providers, without you having to re-implement the per-provider cache APIs yourself. I covered , their open-source CLI coding agent powered by Devstral 2. The gateway now wires mistral_request and mistral_request_async alongside the other four providers. Same shape as the rest, sessions through --resume / --continue (which requires [session_logging] enabled = true in ~/.vibe/config.toml, the doctor surfaces this so you do not get an opaque failure), model registry entries, self-update via the vibe binary itself, the same circuit-breaker, approval-gate, flight recorder, metrics, dedup, and durable-job-store plumbing as the others.

The model alias resolution is slightly different. Vibe has no --model flag, so the gateway injects the resolved alias via VIBE_ACTIVE_MODEL instead. That is the only material divergence from the Claude / Codex / Gemini / Grok pattern, and it is documented inline at the call site.

Now five providers, five model families, five vendor lineages (Anthropic, OpenAI, Google, xAI, Mistral). What I noticed running parallel reviews these past few weeks is that the three OpenAI / Anthropic / Google adjacent triangle agreeing on something is not as informative as it looks, because the three model lineages share a lot of training data and a lot of post-training tendencies. I am not pretending this is statistics, it is just how I use these tools in review work, but adding an xAI voice and a Mistral voice means a five-way agreement is sampled from a meaningfully wider distribution than a three-way agreement, and a one-out-of-five dissent (especially from the vendor-outside-the-triangle) is a data point I read rather than a vote I discard.

promptParts: structured prompts, prefix discipline, no API contortions

The change that took most of the engineering is promptParts. The shape is small:

{

  "promptParts": {

    "system": "You are a careful reviewer of TypeScript diffs.",

    "tools":  "<long, stable description of the tools you can call>",

    "context": "<long, stable file dump or repo summary>",

    "task":    "What did the last patch change?"

  }

}

prompt and promptParts are mutually exclusive, you pass exactly one, the runtime check at the top of every handler returns the exact error message provide exactly one of `prompt` or `promptParts` if you pass both (the backticks belong to the error string itself; the messages are part of the public contract and the tests assert them verbatim). The gateway then concatenates the parts in canonical order, system → tools → context → task, with a stable separator, and hands the resulting string to the CLI's positional -p (or equivalent) argument. The stable prefix bytes precede the volatile task tail unchanged across calls, which is enough for each provider's automatic prompt-caching to land on the same content hash each time.

Two specific points worth naming.

First, this is not a request-body translation layer. The gateway does not construct Anthropic / OpenAI / Mistral JSON request bodies; it spawns the CLI binary the same way it always has. The "cache awareness" sits one layer above, in how the input string is composed before the CLI sees it. That keeps the architectural thesis intact (CLI wrapping, not API proxying) while still giving you cache hygiene for free.

Second, for Claude specifically, the gateway does not yet emit explicit cache_control JSON breakpoints. The Claude Code CLI documents --exclude-dynamic-system-prompt-sections and several ENABLE_PROMPT_CACHING_* / DISABLE_PROMPT_CACHING_* environment variables (all listed in ), but the path for injecting per-block cache_control markers via stream-json input is probable rather than verified. The [cache_awareness].emit_anthropic_cache_control flag is reserved in config for the follow-up slice that lands a live smoke test, so the present 1.6.0 release ships "Branch B" (prefix discipline only). That is honest about what works and what is gated on verification.

Third (because I said two and meant three), per-model minimum cacheable token thresholds matter. Anthropic Sonnet 3.5–4.6 caches at 1024 tokens minimum; Opus 4.5+ and Haiku 4.5 require 4096; Haiku 3.5 on Vertex needs 2048. The gateway has a [cache_awareness.min_stable_tokens_for_cache_control] per-family table populated from the under the "Codex field name divergence" section; the gateway's src/codex-json-parser.ts was originally written against the Anthropic-style name). The parser's cache_read_tokens column therefore stays null for Codex rows until a follow-up updates the parser to accept the actual field. The observability surface tolerates this without dividing by zero, and the limitation is also documented in the spec repository. A new .github/workflows/security.yml runs actionlint, zizmor, shellcheck, typos, osv-scanner, gitleaks, ruff, bandit, and lychee on every push and pull request; eslint-plugin-security is wired into the existing eslint config and runs as part of the standard CI lint step. All third-party actions are SHA-pinned; the Python and Go tools are version-pinned (zizmor==1.25.2, ruff==0.14.5, bandit==1.9.4, [email protected]); the gitleaks binary is downloaded and SHA256-verified before execution. Workflows now use least-privilege permissions, defaulting to contents: read and escalating only on the publish jobs that need OIDC for npm provenance / PyPI trusted publishing or gh release upload; every actions/checkout sets persist-credentials: false except the single job that needs the token for the release upload; the release-installer.yml top-level write was narrowed to that one job. Dependabot expanded from github-actions only to also cover npm and pip, with non-security npm bumps grouped so security updates never get delayed behind a batch.

In flight, osv-scanner flagged 26 Go stdlib CVEs in installer/go.mod (pinned to Go 1.22, when the fixes were in 1.23–1.25.x); that has been bumped to 1.25 in lock-step with the release-installer.yml setup-go pin, and re-verified clean. Two test fixtures and one npmjs.com URL needed allowlisting (a deliberate fake bearer token, an npmjs page that Cloudflare bot-protects, and a similar OpenAI help-centre page), each annotated with the specific reason. There are no real findings outstanding.

This is not the kind of work that ships in a marketing line. It is the work that means the next contributor (or me, six months from now) does not accidentally land a workflow with contents: write and a published-to-cache setup-node step on a release-triggered workflow, which is precisely the kind of supply-chain footgun the , and ; the doctor's client_config.claude_desktop_config_present field tells the install agent which path applies.