Langchain4J musings

I’m coming relatively late to the LLM party, but I rarely come very early in the hype cycle.

and

Ollama’s introduction is even shorter:

Get up and running with large language models.

Run Llama 3.2, Phi 3, Mistral, Gemma 2, and other models. Customize and create your own

—-

The fundamental API model.generate(String) passes the user’s message to the Ollama instance and returns its response. We need to create an endpoint to wrap the call; the details are unimportant.

LangChain4J’s Spring Boot starter automatically creates a ChatLanguageModel from the exact dependency set - here, Ollama. Furthermore, it offers lots of configuration options via Spring Boot.

CODE Kopieren

langchain4j.ollama.chat-model:

  base-url: http://localhost:11434                                       #1

  model-name: llama3.2                                                   #2

1. Point to the running Ollama instance


2. Model to use

When the app starts, LangChain4j creates a bean of type ChatLanguageModel and adds it to the context. Note that the concrete type depends on the dependency found on the classpath.

The Ollama infrastructure

For ease of use, I’ll use Docker, and more specifically Docker Compose. Here’s my Compose file:

CODE Kopieren

services:

  langchain4j:

    build:

      context: .

    environment:

      LANGCHAIN4J_OLLAMA_CHAT_MODEL_BASE_URL: http://ollama:11434        #1

    ports:

      - "8080:8080"

    depends_on:

      - ollama

  ollama:

    image: ollama/ollama                                                 #2

    volumes:

      - ./ollama:/root/.ollama                                           #3

1. Override the URL configured in the JAR to use the Docker container on Docker Compose


2. Use the latest images; it’s not production


3. Keep a copy of the models on the host - see below

As mentioned above, Ollama is a runtime with switchable models. There’s no model by default. To download a model, docker exec into the container and run the following command:

CODE Kopieren

ollama run llama3.2

Be careful, llama3.2 is a whopping 20Gb; for this reason, you want to avoid downloading the model from each docker compose up. This is the reason for the volume mapping above.

Of course, you can substitute llama3.2 with any other smaller model, e.g., tinyllama.

At this point, we can curl our app and see the results:

CODE Kopieren

curl localhost:8080 -d 'Hello I am Nicolas and I am a DevRel'

Enhancing with streaming

The above solution works, but the user experience has room for improvement. The command hangs, and the response comes after several seconds, unlike the traditional OpenAI UI, which streams tokens back to the user.

We can readily replace ChatLanguageModel with StreamingChatLanguageModel to achieve this. Methods are slightly different:

Here’s the relevant code:

CODE Kopieren

data class StructuredMessage(val sessionId: String, val text: String)    //1



interface ChatBot {                                                      //2

    fun talk(@MemoryId sessionId: String, @UserMessage message: String): TokenStream //3-4-5

}



class PromptHandler(private val chatBot: ChatBot) {



    suspend fun handle(req: ServerRequest): ServerResponse {

        val message = req.awaitBody<StructuredMessage>()

        val sink = Sinks.many().unicast().onBackpressureBuffer<String>()

        chatBot.talk(message.sessionId, message.text)                    //6

            .onNext(sink::tryEmitNext)                                   //7

            .onError(sink::tryEmitError)                                 //7

            .onComplete { sink.tryEmitComplete() }                       //7

            .start()

        return ServerResponse.ok().bodyAndAwait(sink.asFlux().asFlow())

    }

}



fun beans() = beans {

    bean {

        coRouter {

            val chatBot = AiServices                                     //8

                .builder(ChatBot::class.java)

                .streamingChatLanguageModel(ref<StreamingChatLanguageModel>())

                .chatMemoryProvider { MessageWindowChatMemory.withMaxMessages(40) }

                .build()

            POST("/")(PromptHandler(chatBot)::handle)

        }

    }

}

1. We need a way to pass a correlation ID to group messages with the same chat history. Given we are using curl and not a browser, we explicitly pass an ID along with the user message


2. Define an interface with no hierarchy requirements. Functions are free-form, but you can set hints


3. 
   @MemoryId marks the correlation ID


4. 
   @UserMessage marks the message sent from the user to the model


5. 
   TokenStream you can subscribe to


6. LangChain4j calls the configured model


7. Pipe the TokenStream to the sink as in our custom implementation


8. Build the ChatBot: AiServices will create the implementation at runtime

Here’s how to use it:

CODE Kopieren

curl -N -H 'Content-Type: application/json' localhost:8080 -d '{ "sessionId": "1", "message": "Hello I am Nicolas and I am a DevRel" }'

curl -N -H 'Content-Type: application/json' localhost:8080 -d '{ "sessionId": "2", "message": "Hello I am Jane Doe and I am a test sample" }'

Adding Retrieval-Augmented Generation

LLMs are only as good as the data they are trained on, and there’s a high chance you want your chatbot to be trained on your own custom data. RAG is the answer to this problem. The idea is to index content ahead of time, store it somewhere, and add the indexed data to the search - called retrieval. For more details, LangChain4j does a great job of . It provides two sources, files and URLs, and an in-memory embedding store. In a regular app, you would index offline and store embeddings in a regular database, but we will do it in memory at startup time. It’s good enough for our prototyping purposes.

CODE Kopieren

class BlogDataLoader(private val embeddingStore: EmbeddingStore<TextSegment>) {



    private val urls = arrayOf(

        "https://blog.frankel.ch/speaking/",

        // Other URLs

    )



    @EventListener(ApplicationStartedEvent::class)                       //1

    fun onApplicationStarted() {

        val parser = TextDocumentParser()

        val documents = urls.map { UrlDocumentLoader.load(it, parser) }

        EmbeddingStoreIngestor.ingest(documents, embeddingStore)

    }

}



fun beans() = beans {

    bean<EmbeddingStore<TextSegment>> {

        InMemoryEmbeddingStore<TextSegment>()                            //2

    }

    bean {

        BlogDataLoader(ref<EmbeddingStore<TextSegment>>())               //3

    }

    bean {

        coRouter {

            val chatBot = AiServices

                .builder(ChatBot::class.java)

                .streamingChatLanguageModel(ref<StreamingChatLanguageModel>())

                .chatMemoryProvider { MessageWindowChatMemory.withMaxMessages(40) }

                .contentRetriever(EmbeddingStoreContentRetriever.from(ref<EmbeddingStore<TextSegment>>())) //4

                .build()

        }

    }

}

1. Run the code when the application starts


2. Define the embedding store. Regular applications should use a persistent data store: LangChain4j supports /










Originally published at A Java Geek on November 10^th, 2024

Vollständiger Original-Bericht
Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
↗ Original-Artikel auf dev.to lesen