1. Self-Hosted AI: the LLM concepts you need to run models effectively

If you want to run AI models on your own machine and learn the basic concepts with me to do it effectively, then this is the right article :).

This is part one of the series. In the next one we build local AI workflows with n8n and Ollama. This article is the vocabulary we need to pick a model, load it without crashing your machine, and know what each setting does.

The pieces

Running a model locally involves a few separate things that people mix up all the time.

The model is a file on your disk. The server (Ollama, LM Studio) loads that file and runs the math. The API is a local address the server opens so other apps can reach it. The clients are whatever you point at it: a chat window, your code editor, an n8n workflow. You never talk to the model directly, you always go through the server.

Model and inference

The models we run are large language models (LLMs): machine learning models trained on massive text datasets to understand and generate language, built on a neural network type called a transformer (). Everything below is about making inference run well.

Server and API

The server loads and runs the model, and it opens a local API. An API is a set of rules that lets two pieces of software communicate (). Whatever does not fit spills into system RAM and runs on the CPU, which works but is slower.

On Apple Silicon Macs there is no separate VRAM. The CPU and GPU share one pool called unified memory, so your RAM is effectively your VRAM.

Parameters, and how to read a model name

Parameters are the internal variables a model learns during training, and they store the patterns it uses to make predictions (). For us it means the same model ships in smaller versions that need much less memory.

Labels like Q4_K_M or Q8_0 tell you how many bits each weight keeps. Fewer bits, smaller and faster file, slightly less accurate.

Two file formats cover most of what you will meet. GGUF packs a quantized model into a single file and is the native format of llama.cpp, the engine under Ollama and LM Studio. Ollama stores and runs its models as GGUF, and you can import any GGUF from Hugging Face with a one-line Modelfile (). Safetensors is the format Hugging Face uses for full precision weights, which you will mostly meet when a model has not been quantized yet.

Tokens

A token is the smallest unit of text a model processes, usually a word, part of a word, or punctuation (). It holds everything simultaneously: the system prompt, the conversation, pasted files, and the room left for the reply.

When it fills up, the oldest tokens drop out and the model forgets the start of a long chat. And a bigger window costs more memory, since the model keeps a growing cache of the conversation. I keep it at 8K for chat and coding, and raise it only when feeding long documents.

Temperature

Temperature controls the randomness of generated text (: give it a model, a quant, and a context size, and it tells you whether your hardware can take it. When the total goes over your memory, the fix is a smaller quant, a smaller model, or a smaller context.

Capabilities: what a model can and cannot do

Abilities are fixed during training, so check the model's page before relying on one. The tags on ). This powers RAG, the technique behind chatting with your own documents.

A coding model usually cannot see images, and a vision model may be weak at tool use. Match the model to the task.

Knowledge cutoff and hallucination

Two limits to keep in mind with any model, local or not.

A model's knowledge freezes at its training cutoff date. Ask a local model about something from last month and it either does not know or, worse, guesses. Which leads to the second limit: models sometimes state wrong things fluently and confidently, called hallucination. It happens because the model predicts plausible text rather than checking facts. Lower temperature and giving the model the relevant documents (RAG) reduce it, but nothing removes it. Treat any factual claim from a model as a draft to verify.

Local or cloud?

Not every task belongs on a local model. Local wins on privacy and cost, since your data never leaves the machine and tokens are free after download, while cloud models pull ahead on raw capability, huge contexts, and fresh information via web search (, , , , , , , WebCraft: Ollama vs ChatGPT vs Claude.