I am trying to use the new AI tools like to develop my applications faster. But I have one problem with them: they don't know how to use the recent versions of the frameworks/libraries I am using. This is normal because the new documentations where not part of their training data.

A trivial solution to this problem would be to just include the new documentation in the context. But this will be costly and make the response slow (and inaccurate sometimes). It's also not possible if I am solving a problem that invloves multiple tools (their docs would exceed the context size of the model).

I need a way to extract only the relevant parts of the documentations and add them to the prompt, I believe this is the idea behind when he was building the first version of : Semantic search denotes search with meaning, as distinguished from lexical search where the search engine looks for literal matches of the query words or variants of them, without understanding the overall meaning of the query.

In short:

• 
  Lexical search: finding literal matches of the query words or variants of them.


• 
  Semantic search: understanding the meaning of the query and finding the relevant parts of the documentations.

I get the Lexical search, it's just looking for the parts that include some specific words. But I still don't get the semantic search, what do you mean by "understanding the meaning of the query"?

Here is a simplified explanation of how semantic search works:

Imagine that we can associate to each word a number that represents its meaning, and that words with similar meanings will have close numbers. Then if we want to test that two words have similar meanings, we can just compare their numbers. Now we can represent the meaning of a sentence (from the query or the documentation) as the list of numbers of each word in the sentence. Then to test if two sentences have similar meanings, we can just compute the distance between their list of numbers. The smaller the distance, the more similar the meanings.

Ok, but how do you come up with the number that represent the meaning of a word, wouldn't it change depending on the context?

Yes it would, the explanation above is just for you to get the idea. In practice, we use a library or an LLM to generate the list of numbers for a chunk of text, called embedding. The embedding would capture the overall meaning of the chunk, taking into account the contextual meaning of the words and the relationship between them. The AI models that are specialized in this are called embedding models. To generate embeddings, we can use an API like .

I will go with the offline approach for this tool, because I am already familiar with Ollama and don't want the tool to require an API key from OpenAI or other service to work. I understand that this will add a new requirement to run the tool, but installing and running Ollama is easy and we can automate it if needed (I am thinking of a setup command that installs all requirements of the tool: Ollama, Git, etc).

Now that we have an idea of how to generate embeddings, here is how we can do semantic search:

Initialization:

1. Split the documentations markdown files into chunks.


2. Generate embeddings for all chunks.


3. Store each chunk and its embedding into a database.

Semantic search:

1. Generate an embedding for the prompt.


2. Query the database for chunks with similar embeddings.

So the next questions we should think about are:

• How to split the documentations into chunks?


• Which database we should use to store embeddings and query them?

Splitting the documentations into chunks

Input: The content of a markdown file

Output: A list of chunks

Constraints:

• The number of tokens in a chunk should not exceed the limit of the embedding model.


• We should not cut a heading or a sentence in the middle.


• We should avoid cutting a paragraph, a code block, a table or a list in the middle as much as possible.


• We should add metadata to each chunk to give context like the name of the framework/library, the version, the headings/titles that lead to the chunk, etc.

"The number of tokens in a chunk", what is a token?

A token is the unit of text used by LLMs, typically representing a word, part of a word, or character.

So we will need a way to count the number of tokens in a chunk, to ensure it does not exceed the limit, right? how?

Yes we will need to count the number of tokens in a chunk. The counting method depends on the embedding model we will choose. We will explore how to use the ollama API to count tokens during the implementation step.

Choosing a database

I would like the tool to be usable offline and easy to install, so the first database that comes to mind is SQLite. I can use an extension like , we can look into that in details once we finish the development of the tool.

Aider

We can configure Aider to automatically include specific files in the context using the read property of in the aider repository to enable integrating aider with external tools. So we could make it call our tool automatically and add the chunks to the context.

Summary

We are building a CLI tool that stores documentations of different frameworks/libraries and allows to do semantic search and extract the relevant parts from them.

Requirements:

• Git: will be used to clone the documentations repository


• Ollama: will be used to generate embeddings

Assumptions / Limitations:

• We are assuming that the documentations are available as markdown files on a Git repository

database:

We choose to go with SQLite for now and add support for other databases in the future.

Next steps

In the next article, we will design the CLI interface and start the implementation of the tool.