Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer

An exact-match cache misses "how do I reverse a list in Python" when it has already answered "python list reverse". A semantic cache doesn't: it embeds the prompt, finds the closest one it has seen, and replays that answer instead of calling the model. Fewer API calls, lower latency, same answers.

Except for the part where it hands back the wrong one.

"Convert 100 USD to EUR"

"Convert 250 USD to EUR"      cosine similarity: ~0.99

Every mainstream embedding model scores that pair around 0.99. No threshold separates it from a genuine paraphrase, because on the similarity axis the near miss sits closer than most paraphrases do. Raise the threshold and you lose real hits before you lose that one.

So a cache built on a threshold alone will tell someone that 250 dollars is 92 euros. Quickly, with no error, and nothing in the logs.

What Kmemo does about it

Kmemo treats that as the main event rather than an edge case. Similarity is only the first filter. Candidates that clear it get read as text by a chain of ten guards looking for concrete evidence that the two answers must differ: swapped numbers, mismatched units, different entities, different time references, negation, flipped antonyms, reversed comparisons, or a different kind of answer being asked for.

The defaults follow from the cost asymmetry. A wrong rejection costs one API call. A wrong acceptance costs a wrong answer. So the guards abstain rather than guess.

Quick start

Requires JDK 17+.

dependencies {

    implementation("io.github.nacode-studios:kmemo-core:1.0.0")

}

kmemo-core declares kotlinx-coroutines-core as its only dependency. You bring the embedding source, which is any function from String to FloatArray. Kmemo ships none and depends on no provider SDK.

val cache = SemanticCache(

    embedder = Embedder { text -> openAi.embed(text) },

    store = InMemoryStore(maxEntries = 10_000, ttl = 1.hours),

)



val answer = cache.getOrPut(prompt) { llm.complete(it) }

getOrPut embeds the prompt once and reuses the vector for both the lookup and the write. Concurrent callers asking the same thing get coalesced: the first one computes, the rest wait and are served its answer.

Every miss tells you why

A cache with a 4% hit rate is untunable unless you know what caused the misses, because the fix is opposite for a threshold miss and a guard rejection:

when (val result = cache.lookup(prompt)) {

    is CacheLookup.Hit  -> result.response

    is CacheLookup.Miss -> when (result.reason) {

        MissReason.BELOW_THRESHOLD   -> // traffic repeats less than you assumed, or the threshold is too tight

        MissReason.REJECTED_BY_GUARD -> // a guard found a concrete difference; result.detail says which

        else -> null

    }

}

There's also cache.explain(prompt), a read-only companion that shows every candidate with every guard's verdict. It's what you reach for when a hit you expected didn't happen.

Scopes

Anything that changes what a correct answer looks like belongs in the scope: model, temperature, system prompt, tenant, language. Leave it out and the cache will serve one model's answer to another model's caller.

cache.getOrPut(prompt, scope = "gpt-4o|t=0.0|v3") { llm.complete(it) }

Choosing how strict to be

SemanticCache(embedder)                                    // MatchGuards.standard()

SemanticCache(embedder, guards = MatchGuards.strict())     // trades hit rate for margin

SemanticCache(embedder, guards = MatchGuards.none())       // the naive similarity-only baseline

The guards work outside English too. Curated packs ship for Italian, Spanish, German and French, each measured against a localized near-miss corpus:

SemanticCache(embedder, guards = MatchGuards.standard(Locale.ITALIAN))

What lexical guards can't see

About a third of near misses need world knowledge. Deworming a puppy is not the same as deworming an adult dog. The boiling point of ethanol is not the boiling point of methanol. No amount of token comparison catches those.

For that there's an optional Verifier, typically a cheap model call. It runs only on candidates that already cleared the threshold and every guard, so it costs nothing in the common case, and it fails closed: a timeout or an error rejects rather than serving something unconfirmed.

The numbers

The guards are judged against three labelled corpora with a blind validation split that no guard was tuned against, run as a CI regression gate on every build.

On the blind split, near misses are rejected 67% of the time and paraphrases are kept 88% of the time.

Neither number is 100%, and I would rather publish them than a marketing claim. The near misses that get through are mostly the world-knowledge cases the verifier covers. Reproduce them yourself:

./gradlew :kmemo-core:test --tests '*CorpusTest*'

How the blind splits grow without getting contaminated is written up in docs/CORPUS.md.

Calibrate the threshold, don't copy it

ThresholdCalibrator measures the right threshold for your embedding model. The value you found in a blog post was tuned for somebody else's.

Stores, resilience, observability

Embedder and CacheStore are one-method seams, so you can start in memory and move to a vector database without touching the match logic. Redis (RediSearch KNN) and Postgres (pgvector) stores ship, plus an opt-in in-process HNSW store for when the exact scan stops scaling.

The embedder is a network call on every lookup, so Kmemo lets you own its failure:

val cache = SemanticCache(

    embedder = myEmbedder.retrying(maxAttempts = 4),

    embedFailurePolicy = EmbedFailurePolicy.FALL_BACK_TO_COMPUTE,

    negativeCacheSize = 10_000,

)

cache.warm(faqPairs.map { WarmEntry(it.question, it.answer) })

For dashboards and logs, subscribe to the event stream instead of polling stats(). It costs nothing when unused:

val metrics = KmemoMetrics().also { it.bindTo(meterRegistry) }   // kmemo-micrometer

val cache = SemanticCache(embedder, listeners = listOf(metrics, Slf4jCacheListener()))

Integrations

• A Spring Boot starter that auto-configures a SemanticCache bean


• A Spring AI caching Advisor for ChatClient
  


• A LangChain4j caching ChatModel wrapper


• A Ktor server plugin

examples/ is a runnable demo that needs no API key. It shows a guard catching a live near miss, with a docker-compose for the Redis store.

Try it

• Repo: https://github.com/NaCode-Studios/Kmemo
  


• Maven Central: io.github.nacode-studios:kmemo-core:1.0.0
  


• API docs: https://nacode-studios.github.io/Kmemo/
  


• Apache-2.0

1.0 is stable under SemVer. If you have run a semantic cache in production and hit a false hit I haven't thought about, I want to hear about it. Open an issue with the pair that broke it.