Next-Token Prediction: How an AI Actually Writes Text (Not Magic — Just Probability)

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

Ask an AI the same question twice. Get two different answers. That's not a glitch you tolerate — it's the entire mechanism working exactly as designed.

Start below the buzzword: a language model never sees a finished sentence. It only ever answers one tiny question, over and over — given everything written so far, what's the next chunk of text?

One-line mental model: the model outputs a probability over every possible next token → it samples from that distribution instead of always grabbing the top score → the winning token gets glued onto the text → the exact same question runs again from scratch, one token at a time.

The concrete example: finishing one sentence

Say DraftPal, a writing assistant, is finishing: "The cat sat on the ___." It doesn't know the ending. It computes one probability for every possible next token it knows about:

mat    → 41%
chair  → 19%
floor  → 12%
...    → (thousands more, trailing to ~0%)

That's it. That's the entire "intelligence" at this step — a ranked list over the whole vocabulary, built fresh from the text so far.

The part almost everyone skips: it samples, it doesn't grab the top score

Here's the detail that explains half the "weird" behavior people notice about LLMs: the model does not deterministically pick mat because it's the highest score. It samples — a weighted die roll across that entire distribution. 41% wins most of the time. Sometimes chair wins instead. Same model, same prompt, different word — because the die was rolled, not read off a table.

Whatever wins gets glued onto the text, and the whole question — "given everything so far, what's next?" — runs again from scratch, now one token longer. One token, one roll, repeat. That loop, run a few hundred times, is what writes an entire reply.

# pseudocode — the entire generation loop
tokens = tokenize(prompt)
while not done:
    distribution = model(tokens)      # probability over every next token
    next_token = sample(distribution) # NOT always argmax
    tokens.append(next_token)

The chart isn't fixed — it's rebuilt from context every time

Add four words of context before the same question — "write this like a horror story" — and the exact same probability computation comes back totally different: mat collapses under 1%, coffin jumps to 99%. Nothing about the model changed. The input context changed, so the distribution it computes changed.

This one mechanism quietly explains two things developers run into constantly:

• Why the same prompt gives two different replies on two runs. No hidden state, no bug — it's sampling from a distribution, and the die comes up differently.
• What "personalization" actually is. A model doesn't know you. Your prior messages get stuffed back into the context window on every call, which reshapes the same probability chart toward tokens that fit what you've said before. It's context, not memory.

What it costs, and where it fails

The determinism/variety trade-off. Force the model to always take the top slot (greedy decoding, or "temperature 0") and answers get boringly identical every run — useful when you need reproducibility, e.g. structured extraction. Leave sampling on and you get natural variety, at the cost of never getting the exact same output twice.

The compute cost is per token, not per reply. Every single token — not the whole response — costs one full forward pass through the model. A 500-token answer is roughly 500 times more expensive than a 1-token answer, not "a bit more." This is also why streaming feels slow on long outputs: you're watching the loop happen in real time.

No going back — the seed of a hallucination. Once a token is glued onto the context, it is never revised. The model doesn't get to reconsider token 40 after generating token 41. So one confident wrong guess early on doesn't get corrected — the next question is now "given everything so far, including that wrong guess, what's next?" — and the model builds forward on its own mistake. That's the actual mechanical origin of a hallucination: not "the model lied," but "the model committed to a token and the loop only moves forward."

Reframe: this is also the whole story behind LLM-serving latency

If you've ever looked at an inference dashboard, two metrics show up everywhere: TTFT (time-to-first-token) and TPOT (time-per-output-token). This loop is exactly what they're measuring.

• TTFT is the cost of that first forward pass — reading the whole prompt and producing the first probability distribution.
• TPOT is the cost of every subsequent iteration of the loop above — one more forward pass per token, forever, until the model samples a stop token.

That's also why batching exists as a serving technique: since each loop iteration is bottlenecked on loading the model's weights into the GPU's compute units rather than on the arithmetic itself, serving frameworks pack multiple users' next-token requests into the same forward pass so one expensive weight-load produces many tokens at once. And it's why response length is the single biggest lever on cost and latency in any LLM product — you are quite literally paying for the number of times the loop above has to run.

The takeaway

Not a sentence writer. A next-token predictor, running in a loop — one probability chart, one weighted roll, one token glued on, repeated until it samples a stop.

Two devs run the same prompt through the same model. One gets "…sat on the mat," the other gets "…the windowsill." Neither is wrong. That's not inconsistency — that's the mechanism.

Want the full walkthrough with the running example built out end to end? Watch the long-form video. Or the 90-second cut if you just want the core loop.