How to Serve LLM Completions in Production

Preparations

To start, you need to compile for instructions.

The server is compiled alongside other targets by default.

Once you have the server running, we can continue. We will use PHP Resonance framework.

Troubleshooting

Obtaining Open-Source LLM

I recommend starting either with . You need to download the pretrained weights and convert them into GGUF format before they can be used with supports CPU-only setups, so you don't have to do any additional configuration. It will be slow, but you will still have tokens generated.

Running With a Low VRAM Memory

You can try quantization if you don't have enough VRAM on your GPU to run a specific model. That lowers the response quality and the memory the model needs to use. Llama.cpp has a utility to quantize models:

$ ./quantize ./models/7B/ggml-model-f16.gguf ./models/7B/ggml-model-q4_0.gguf q4_0

10GB of VRAM is enough to run most quantized models.

Starting llama.cpp Server

While writing this tutorial, I had a server started with a command:

$ ./server 

    --model ~/llama-2-7b-chat/ggml-model-q4_0.gguf 

    --n-gpu-layers 200000 

    --ctx-size 2048 

    --parallel 8 

    --cont-batching

    --mlock 

    --port 8081

cont-batching parameter is essential, because it enables continuous batching, which is an optimization technique that allows parallel request.

Without it, even with multiple parallel slots, the server could answer to only one request at a time. cont-batching allows the server to respond to multiple completion requests in parallel.

Configuring Resonance

All you need to do is add a configuration section that specifies the llama.cpp server location:

[llamacpp]

host = 127.0.0.1

port = 8081

Testing

Resonance has built-in commands that connect to llama.cpp and issue requests.

You can send a sample prompt through llamacpp:completion:

$ php ./bin/resonance.php llamacpp:completion "How to write a 'Hello, world' in PHP?"

To write a "Hello, world" in PHP, you can use the following code:



<?php

  echo "Hello, world!";

?>



This will produce a simple "Hello, world!" message when executed.

Programmatic Use

In your class, you need to use server and connect to it with Resonance.