Building a Private RAG System: Lessons from a Local-First AI Journal

Most AI apps quietly send your data to the cloud. DiaryGPT does the opposite — and this is the full technical story.

The Problem With AI + Private Data

When you write in a journal, you write the things you'd never say out loud. The last thing you want is that text sitting on someone else's server, used to train a model, or exposed in a breach.

But AI is genuinely useful for journaling. It can find patterns you miss, reflect things back to you, ask questions a blank page never would. The tension is real: you want AI insight without sacrificing privacy.

Most apps solve this by trusting a privacy policy. I wanted a technical guarantee.

So I built DiaryGPT — an AI-powered personal journal where, by default, zero data leaves your machine. Here's exactly how it works.

What DiaryGPT Does

Before the architecture, here's what the app gives you:

• 
  AI mood analysis on every entry — mood, themes, a reflective response, and a follow-up question


• 
  RAG-powered chat — ask "when was I most anxious?" and get answers grounded in your actual entries


• 
  Semantic search — find entries by meaning, not keywords ("times I felt lonely" finds entries with "isolated", "disconnected", "blue")


• 
  Weekly reflection — AI summary of your emotional arc across the week


• 
  Personalized journaling prompts — generated from your recent writing patterns


• 
  Writing streaks and memories — "on this day last year you wrote…"


• 
  AI companion mode — CBT/DBT-grounded reflection with built-in crisis detection (not a replacement for a licensed therapist)


• 
  Mood check-ins — 1–10 logging with history chart


• 
  Voice dictation and voice chat — speak entries, hear responses read back


• 
  Full AES-256-GCM encryption at rest — every diary entry, chat message, and note

The Privacy Architecture

DiaryGPT has two modes. You choose in Settings.

🟢 Local Mode (Default)

Everything runs on your machine. The AI model, the search, the analysis — all local via

The Encryption Layer

Every piece of user content goes through AES-256-GCM encryption before hitting the database.

// Every diary entry, chat message, companion note goes through this

encrypt(text)   // before DB insert

decrypt(text)   // after DB read, before sending to LLM or browser

The encryption key is yours — a 64-character hex string you generate and store in your .env. Without it, the database is unreadable. The server never transmits the key.

The one exception: embedding vectors are stored unencrypted. Cosine similarity requires the raw numbers. The chunk text that generated the embedding is stored separately, encrypted. The security boundary lives at the source text, not the derived vector.

The Technical Stack

Runtime        Node.js + Express

Frontend       Vanilla JS SPA (no build step, no framework)

Auth           JWT + Argon2id password hashing

Encryption     AES-256-GCM (Node.js crypto module)

Storage        SQLite (local default) or PostgreSQL (multi-device)

Vector search  sqlite-vec (local) or pgvector (Postgres)

Embeddings     Ollama nomic-embed-text (local default)

LLM            Ollama (local default) / Groq / OpenAI / Gemini / Anthropic

Streaming      SSE (Server-Sent Events) over POST with ReadableStream

Voice          Browser SpeechRecognition API (free) or Whisper (premium)

The frontend is deliberately no-framework. No React, no build pipeline, no node_modules in the browser. It loads instantly and works offline (except for cloud LLM calls).

LLM Provider Architecture

The LLM layer is a thin factory that routes every call to whatever provider is active:

// services/llm.js

const PROVIDERS = { ollama, anthropic, openai, gemini, groq };



export const streamChat = (history, message, context, onDelta) =>

  PROVIDERS[getConfig().provider].streamChat(history, message, context, onDelta);

Switching providers happens at runtime — no restart needed. Every provider implements the same three-function contract:

analyzeEntry(text)                              // → { mood, themes, reflection, followUpQuestion }

generateText(systemPrompt, userMessage)         // → string

streamChat(history, message, context, onDelta)  // → full string, streams via onDelta

Groq uses the OpenAI SDK pointed at https://api.groq.com/openai/v1. Ollama uses the same SDK pointed at http://localhost:11434/v1. Identical interface, completely different privacy properties.

What I Learned

1. Embeddings and LLMs are completely separate concerns. The model that converts text to numbers has nothing to do with the model that generates answers. You can run Ollama for embeddings and Groq for chat simultaneously. Most people conflate the two.

2. 7B–8B models are good enough for structured diary tasks. Mood detection, theme extraction, journaling prompts — a well-prompted qwen2.5:7b handles all of these reliably. The quality gap versus 70B only shows up in long-form weekly summaries. Use format: json mode in Ollama for structured output; without it, small models will eventually return malformed JSON and break your pipeline silently.

3. Cosine similarity belongs in your database, not a vector database. For a personal app with thousands (not millions) of entries, sqlite-vec and pgvector are more than sufficient. No Pinecone, no Weaviate, no extra infra. The math is simple and fast.

4. SSE over POST is the right call for streaming. The standard advice is to use EventSource, but EventSource is GET-only. Chat requires POST (to send the message body). The fix is fetch + ReadableStream on the client — full control over the stream lifecycle, no awkward query-string payloads.

5. Crisis detection must run before the LLM, not inside it. You cannot rely on an LLM to consistently detect crisis language and respond safely. Keyword matching before the LLM call is not elegant, but it is reliable and auditable. An LLM should never be the first line of defense for someone in crisis — it should never even get the message.

6. The hardest engineering decisions in a privacy-first app are about what not to do. No analytics. No telemetry. No "anonymized" usage data. Every one of those is a useful product feature you give up — and giving them up is the point.

Try It

DiaryGPT is open source. Self-host it, read every line, verify the privacy claims.

🔗 GitHub: https://github.com/rahul70-code/diarygpt

Your diary is yours. The AI should work for you, not harvest from you.

Stack: Node.js · Ollama · SQLite · AES-256-GCM · Vanilla JS

Tags: #LLM #RAG #Privacy #LocalFirst #OpenSource