Get models like Phi-2, Mistral, and LLaVA running locally on a Raspberry Pi with Ollama
Ever thought of running your own large language models (LLMs) or vision language models (VLMs) on your own device? You probably did, but the thoughts of setting things up from scratch, having to manage the environment, downloading the right model weights, and the lingering doubt of whether your device can even handle the model has probably given you some pause.
Let’s go one step further than that. Imagine operating your own LLM or VLM on a device no larger than a credit card — a Raspberry Pi. Impossible? Not at all. I mean, I’m writing this post after all, so it definitely is possible.
Possible, yes. But why would you even do it?
LLMs at the edge seem quite far-fetched at this point in time. But this particular niche use case should mature over time, and we will definitely see some cool edge solutions being deployed with an all-local generative AI solution running on-device at the edge.
It’s also about pushing the limits to see what’s possible. If it can be done at this extreme end of the compute scale, then it can be done at any level in between a Raspberry Pi and a big and powerful server GPU.
Traditionally, edge AI has been closely linked with computer vision. Exploring the deployment of LLMs and VLMs at the edge adds an exciting dimension to this field that is just emerging.
Most importantly, I just wanted to do something fun with my recently acquired Raspberry Pi 5.
So, how do we achieve all this on a Raspberry Pi? Using Ollama!
What is Ollama?
There’s also .
2. Running LLMs through the command line
Take a look at . Take a look at the to install it without Docker. It recommends minimally Node.js to be >= 20.10 so we shall follow that. It also recommends Python to be at least 3.11, but Raspbian OS already has that installed for us.
We have to install Node.js first. In the terminal, run
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - &&\
sudo apt-get install -y nodejs
Change the 20.x to a more appropriate version if need be for future readers.
Then run the code block below.
git clone https://github.com/ollama-webui/ollama-webui.git
cd ollama-webui/
# Copying required .env file
cp -RPp example.env .env
# Building Frontend Using Node
npm i
npm run build
# Serving Frontend with the Backend
cd ./backend
pip install -r requirements.txt --break-system-packages
sh start.sh
It’s a slight modification of what is provided on GitHub. Do take note that for simplicity and brevity we are not following best practices like using virtual environments and we are using the — break-system-packages flag. If you encounter an error like uvicorn not being found, restart the terminal session.
If all goes correctly, you should be able to access Ollama Web UI on port 8080 through was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
SOCIAL SHARE CARD GENERATOR