Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer

An exact-match cache misses "how do I reverse a list in Python" when it has already answered "python list reverse". A semantic cache doesn't: it embeds the prompt, finds the closest one it has seen, and replays that answer instead of calling the model. Fewer API calls, lower latency, same answers.

Except for the part where it hands back the wrong one.

"Convert 100 USD to EUR"

"Convert 250 USD to EUR"      cosine similarity: ~0.99

Every mainstream embedding model scores that pair around 0.99. No threshold separates it from a genuine paraphrase, because on the similarity axis the near miss sits closer than most paraphrases do. Raise the threshold and you lose real hits before you lose that one.

So a cache built on a threshold alone will tell someone that 250 dollars is 92 euros. Quickly, with no error, and nothing in the logs.

What Kmemo does about it

.

Calibrate the threshold, don't copy it

ThresholdCalibrator measures the right threshold for your embedding model. The value you found in a blog post was tuned for somebody else's.

Stores, resilience, observability

Embedder and CacheStore are one-method seams, so you can start in memory and move to a vector database without touching the match logic. Redis (RediSearch KNN) and Postgres (pgvector) stores ship, plus an opt-in in-process HNSW store for when the exact scan stops scaling.

The embedder is a network call on every lookup, so Kmemo lets you own its failure:

val cache = SemanticCache(

    embedder = myEmbedder.retrying(maxAttempts = 4),

    embedFailurePolicy = EmbedFailurePolicy.FALL_BACK_TO_COMPUTE,

    negativeCacheSize = 10_000,

)

cache.warm(faqPairs.map { WarmEntry(it.question, it.answer) })

For dashboards and logs, subscribe to the event stream instead of polling stats(). It costs nothing when unused:

val metrics = KmemoMetrics().also { it.bindTo(meterRegistry) }   // kmemo-micrometer

val cache = SemanticCache(embedder, listeners = listOf(metrics, Slf4jCacheListener()))

Integrations

• A Spring Boot starter that auto-configures a SemanticCache bean


• A Spring AI caching Advisor for ChatClient
  


• A LangChain4j caching ChatModel wrapper


• A Ktor server plugin

1.0 is stable under SemVer. If you have run a semantic cache in production and hit a false hit I haven't thought about, I want to hear about it. Open an issue with the pair that broke it.