When a user submits a query, the query is converted into an embedding and searched against the vector database to retrieve the relevant documents.
But what happens if the user asks the same or a very similar query again?
This is where semantic caching comes into the picture.
Instead of searching the vector database again, the system stores the previous search result in a cache. A cache is a temporary storage where frequently accessed or recently queried results are stored. When the user asks the same or a semantically similar query again, the system can retrieve the result directly from the cache instead of querying the vector database again.
Benefits
- Saves retrieval time
- Reduces token consumption
- Reduces the number of calls to the vector database
- Reduces the number of calls to the LLM
How Do We Store Results in the Cache?
We can use Redis or Valkey for semantic caching.
These are in-memory databases, which means they store data in RAM instead of disk. Since data is stored in memory, retrieval is much faster compared to traditional databases.
Typically, we store:
- User query
- Related answer
- Metadata
- Embeddings
Example
Suppose a user asks:
"What is today's gold price?"
The query and its corresponding answer are stored in Redis.
Later, another user asks:
"Gold price today?"
Although both queries have the same meaning, Redis cannot directly retrieve the previous answer because it expects the key to match exactly.
This is one of the limitations of using Redis as a simple key-value store.
How Can We Solve This?
One approach is:
- Retrieve all the keys stored in Redis (for example, using
KEYS *). - Generate or retrieve the embedding for each stored query.
- Convert the current user query into an embedding.
- Compare the current query embedding with the stored query embeddings using cosine similarity.
- If the similarity score is above a predefined threshold, retrieve the corresponding answer from Redis.
This allows semantically similar queries to reuse cached results even when the text is different.
Ways to Implement Semantic Caching
Semantic caching can be implemented in two ways:
- Using frameworks such as LangChain
- Using in-memory databases such as Redis, Valkey, or other similar databases
Cache Invalidation
One of the most important aspects of semantic caching is cache invalidation, which determines how long cached data should remain valid before it is automatically removed or refreshed.
For example, suppose a user asks:
"What is today's gold price?"
The answer should only be valid for a limited period. If the application returns yesterday's gold price, the information becomes incorrect.
There is no single solution for cache invalidation. The appropriate strategy depends on the application and the type of data being cached.
Different scenarios need to be considered before deciding when cached data should expire.
When Should In-Memory Databases Be Used?
In-memory databases are well suited for:
- Temporary queries
- Frequently asked questions
- Data that is accessed repeatedly
By understanding the meaning of the query, we can define guardrails to determine which queries should be cached and when the cache should be invalidated.
The main objective is to optimize the RAG pipeline by reducing unnecessary calls to both the vector database and the LLM.
Although it is not possible to eliminate duplicate requests completely, semantic caching can significantly reduce them.
Important Consideration
We should not store every query in an in-memory database.
Only queries that are valuable for caching should be stored because RAM has limited storage capacity. Therefore, an effective caching strategy should carefully decide which queries are worth storing and for how long.