Spring AI Prompt Caching and Chat Memory: Where the Tokens Go — LLM Cost Control 2/4

Suppose the metrics are in place, each task has the model it actually needs, and every feature has its own client — that was .

Driver #3 — Output and reasoning tokens: the expensive direction

Every token the model generates costs several times more than a token you send it. Reasoning models also generate hidden "thinking" tokens, and these are billed at the same, higher output rate.

The gap is on every price sheet. On , Sonnet 5 is $2 in and $10 out — a 5× difference. (That is introductory pricing to 31 August 2026; the standard $3/$15 keeps the same 5× ratio.)

The exact numbers change often, but the pattern does not: output tokens have cost several times more than input tokens for years. Reasoning makes this worse. A model may use 2,000 thinking tokens to produce a 200-token answer, so you pay for 2,200 output tokens in total — eleven times the text the user actually sees. A verbose model on a busy endpoint can end up costing more than all your input traffic combined.

provides a portable completion-length limit, when supported by the provider. Think of it as a safety net against runaway responses, not a tool for improving quality — a cut-off answer is still billed in full, so pair the limit with prompt instructions that ask for a short answer. The second control is provider-specific and kept isolated in that provider's options object: OpenAI has a .

// Provider-independent token limit 

ChatOptions.Builder capped = ChatOptions.builder()

    .maxTokens(400);



// Provider-specific reasoning control, isolated in one options object

OpenAiChatOptions.Builder lowEffort = OpenAiChatOptions.builder()

    .model("gpt-5-mini")

    .reasoningEffort("low")

    .maxCompletionTokens(400);



// Ollama: disable reasoning for thinking-capable models

OllamaOptions.Builder noThink = OllamaOptions.builder()

    .model("qwen3")

    .think(false);

One 2.0 upgrade note belongs here: — including qwen3 and deepseek-r1 — use reasoning by default. During local development there is no per-token API bill, so it is easy to miss that a prompt pattern has become dependent on lengthy reasoning. Move the same prompts to a provider that bills reasoning tokens, and that hidden reasoning can become part of your completion cost. For simple workloads such as extraction, classification, or reformatting, disable or limit thinking during local development as well. This keeps token usage, latency, and production costs easier to predict.

Driver #4 — Conversation history: you pay for the whole chat, every turn

LLMs have no built-in memory, so "memory" in practice means sending the full conversation history with every request. Every past message is billed again, as if it were new input.

The cost grows faster than you might expect. Assume a 500-token system prompt, roughly 50-token user messages, and roughly 200-token replies. Each finished turn adds about 250 tokens of history, and every later turn has to carry that history too. A message window bounds how much of that history each request includes. Two details must be noted here: the window counts the stored messages, so ten messages means the last five complete user–assistant exchanges, and the current user message always travels outside the window. From turn 6 onwards, every request is therefore the same size: 500 + 1,250 + 50 = 1,800 input tokens.

Without a limit, the cost of each turn grows in a straight line, but the cost of the whole session grows much faster than that: a 20-turn session sends about 58,500 input tokens in total, and a 50-turn support chat about 333,750 — for a single user. The same 50-turn chat with a 10-message window sends about 86,250, roughly a quarter of the unbounded total.

One Spring AI 2.0 detail is worth knowing here: , a sliding window of at most N messages, added through a memory advisor:

@Bean

ChatClient chatClient(ChatClient.Builder builder,

                      ChatMemoryRepository repository) {



    ChatMemory memory = MessageWindowChatMemory.builder()

        .chatMemoryRepository(repository)

        .maxMessages(10)   // overrides the default 20-message window

        .build();

    return builder

        .defaultAdvisors(MessageChatMemoryAdvisor.builder(memory).build())

        .build();

}

The service that injects this client selects the conversation on each call:

chatClient.prompt()

    .user(message)

    // mandatory in Spring AI 2.0 — there is no default conversation id

    .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, sessionId))

    .call()

    .content();

are documented together in Spring AI's is an alternative. It stores history in a vector store and adds back only the messages relevant to the current question, so the input size per turn stays flat no matter how long the session runs. The trade-off: every message has to be turned into an embedding and stored (Part 4), and the history added back changes with each question, which works against caching. Caching is next.

Driver #5 — Repeated static content: paying full price for the same tokens

The system prompt, tool definitions (including schemas), and few-shot examples are often the same across requests, and they are part of the model input context. Without caching, providers generally process and charge these repeated input tokens on every request. Prompt caching reduces this cost: supported providers store already-processed prompt prefixes and apply a lower input-token rate when the same content is reused. Every provider implements caching differently, with its own rules for eligibility, expiration, and configuration.

Anthropic and AWS Bedrock: you choose what to cache

Spring AI exposes named caching strategies — SYSTEM_ONLY, TOOLS_ONLY, SYSTEM_AND_TOOLS, and CONVERSATION_HISTORY — through provider-specific enums for . These strategies define where Spring AI places cache breakpoints while respecting provider limitations, but cache lifetime and expiration remain managed by the underlying provider:

AnthropicChatOptions.builder()

    .cacheOptions(AnthropicCacheOptions.builder()

        .strategy(AnthropicCacheStrategy.SYSTEM_AND_TOOLS)

        .build())

Anthropic prompt caching . With CONVERSATION_HISTORY caching, each tool-calling round adds tool results after the default cache point, so those tool outputs are billed as new, uncached input on later rounds. Enabling cacheToolResults moves the cache point to the last tool result, allowing the next round to read the previous round's (often large) tool output from the cache instead of processing it again. This pairs directly with tool schemas and agent loops (Part 3).

OpenAI: automatic, but no longer free

Prompts of at least 1,024 tokens are cached automatically, with no code changes. Two things changed with the GPT-5.6 family, and , and it no longer falls back to the longest matching prefix before that point. A request can therefore share thousands of identical tokens with the previous one, report zero cached tokens, and pay to write the changing prefix again.

The lever Spring AI gives you is the cache key. Requests that share a prefix should carry the same one:

OpenAiChatOptions.Builder shared = OpenAiChatOptions.builder()

    .model("gpt-5.6-terra")

    .promptCacheKey("support-assistant-v1");

or and partition busier workloads across several keys.

Spring AI 2.0 does not expose explicit cache breakpoints, so on OpenAI the cacheable prefix is whatever your prompt structure gives you — which makes the static-first rule below a cost control, not a style preference. It also reports cache reads and AWS Bedrock Converse. With .multiBlockSystemCaching(true), Spring AI can preserve cacheable system blocks while allowing later changing system content to remain outside the cached prefix.

Structure your prompts this way even before you turn caching on — it is what makes every provider's cache, automatic or explicit, actually work for you.

What's next

There is a limit to how far prompt structure can take you, though. Everything in this part assumed the content was yours: your system prompt, your conversation, your examples. The next part deals with the context your application adds automatically — document chunks from a vector store and tool definitions from every server you connect to. Both arrive as input tokens, both are sent whether the model uses them or not, and both scale with how much you have indexed rather than with how much you need.

Part 3 is coming.