How I built a self-hosted AI Gamemaster that actually enforces RPG rules (FastAPI & Vue)

Have you ever wondered if an LLM could actually make a capable Gamemaster? If you have ever tried to play or build a text adventure with a standard AI model, you know the architectural frustration. The AI acts like a hyperactive improv partner. If you type 'I jump over the moon and slay the dragon with a spoon', the model simply hallucinates: 'Yes, you do that.' There is no state management, no constraints, and zero real game mechanics. But what if you combined a deterministic, rule-based RPG engine with an omniscient generative AI Gamemaster?

I wanted a genuine RPG experience with a strict Gamemaster, so I built TaleWeaver. It is an open-source, self-hosted RPG engine that generates dynamic worlds on the fly.

Here is how I solved the "LLM Amnesia" and enforced strict game mechanics using a 2-Pass Architecture.

The Problem: Mixing Mechanics and Narrative

LLMs are fantastic at writing atmospheric prose, but they are terrible at tracking state, hitpoints, and inventory items. If you force a single prompt to manage complex math and write a cinematic story simultaneously, the model will hallucinate.

The Solution: The 2-Pass Engine

I split the workload into two distinct processes to keep the game state intact:

1. Pass 1: The Rule Engine

When a user submits an action, a fast, highly logical model evaluates the input against the current game state. It performs hidden D20 stat rolls, checks the inventory, and calculates damage.

• Output: A strict JSON GameEvent that mutates the database (e.g., updating HP, moving an NPC to a new scene, or adding loot).

2. Pass 2: The Gamemaster

Once the database state is safely updated, a creative narrative model takes the raw JSON outcome and translates it into an immersive, atmospheric response.

The Tech Stack

I built this engine to be completely self-hosted and robust:

• Backend: Python and FastAPI. The database is SQLite, fully tenant-ready with UUID-based primary keys.


• Frontend: Vue.js for a snappy, app-like experience.


• Dynamic Cartography: The engine automatically renders a live directed graph map of the discovered scenes using Rough.js.


• TTS Audio: Integrated Gemini 3.1 Flash TTS and ElevenLabs. The engine parses director notes to build dramatic tension directly into the audio output.

Bring Your Own Key (BYOK) & Local Execution

The entire stack is containerized with Docker for a frictionless 1-click setup. It uses the LiteLLM adapter under the hood. You simply plug in your API key for top-tier models, which are highly recommended to handle the complex JSON reasoning.

I also added experimental support for local Ollama execution. It is tough to get smaller local models to consistently output complex nested JSON, but it is a fun challenge to push the boundaries of offline play.

Check out the code

You can find the repository, the complete JSON manifest specifications, and the setup scripts here:

https://github.com/jschm42/taleweaver