Build custom AI apps - chatbots, RAG pipelines, and agents - entirely on your own hardware with Dify and Ollama. No monthly fees, no data leaving your network.
What You Need
- A GPU with 12GB+ VRAM (RTX 3060 12GB or better)
- Docker + Docker Compose 2.24.0+
- About 20 minutes
Architecture
| Component | Role |
|---|---|
| Dify | Visual app builder, RAG engine, agent framework, API layer |
| Ollama | Serves local models via OpenAI-compatible API |
| Qwen3 14B | Default model - strong general chat, fits 12GB at Q4 |
Setup
Step 1: Start Ollama
docker run -d --gpus all -p 11434:11434 --name ollama \
-v ollama:/root/.ollama \
ollama/ollama
Pull your default model:
docker exec ollama ollama pull qwen3:14b
Step 2: Start Dify
git clone https://github.com/langgenius/dify.git
cd dify/docker
cp .env.example .env
docker compose up -d
Step 3: Connect Ollama to Dify
- Open http://localhost/install and create your admin account
- Go to Settings > Model Provider
- Click Ollama and fill in:
- Model Name: qwen3:14b
- Base URL: http://host.docker.internal:11434 (Docker Desktop) or http://YOUR_IP:11434 (Linux)
- Click Save
Build Your First App
Chatbot
Studio > Create Application > Chatbot. Select your model, add a system prompt, publish. Your chatbot gets a public URL and API endpoint.
RAG Pipeline
Knowledge > Create Knowledge. Upload documents, choose chunking strategy, create an app that uses this knowledge base. Now your chatbot answers from your documents.
Agent
Studio > Create Application > Agent. Add tools (web search, code interpreter), give it a goal, Dify orchestrates the tool calls.
Cost vs Cloud
| Local | Dify Cloud + OpenAI | |
|---|---|---|
| Monthly | $0 | $59-599 + API usage |
| Hardware | ~$300 once | $0 |
| Data privacy | Stays on your machine | Sent to cloud |
| AI calls | Unlimited, free | Per-token billing |
After about 5 months the GPU has paid for itself versus a mid-tier Dify Cloud plan.
Full guide with detailed troubleshooting and alternatives: https://everylocalai.com/stack/dify-ollama-local-app-builder