LLM Sampling Explained: Selecting the Next Token

This is a cross-post, you can find the original article on my Medium

Understanding how LLMs decide what to say next can help you write better prompts and interpret model behavior when generating text. This article breaks down how models assign probabilities to possible next tokens, what log probabilities are, and how the basic sampling methods (greedy vs. probabilistic sampling) work in practice.

A List of Probabilities

In the previous chapter, we have learned that LLMs generate text one token at a time.

So, how does the model decide which token to generate next?

Behind the scenes, the LLM produces a list of all possible next tokens, each paired with its probability.

For example, given the input "How are you? I am ", the model might produce a list like this:

• 
  fine paired with probability 0.7


• 
  good paired with probability 0.2


• 
  bad paired with probability 0.1

Because the list includes every token in the model’s vocabulary, it tends to be quite large.

Technically, the list contains log probabilities—that is, the logarithms of the actual probabilities.

This approach is more numerically stable than working with raw probabilities.

To convert a log probability back to a probability, you simply exponentiate it:

import math



original_prob = 0.7

logprob = math.log(original_prob)

prob = math.exp(logprob)



print(f"Original probability: {original_prob}")

print(f"Log probability: {logprob}")

print(f"Reconstructed probability: {prob}")

This will output:

Original probability: 0.7

Log probability: -0.35667494393873245

Reconstructed probability: 0.7

The OpenAI API lets you retrieve the top log probabilities for the next token, given a prompt:

import math

import os, requests



response = requests.post(

    "https://api.openai.com/v1/chat/completions",

    headers={

        "Authorization": f"Bearer {os.getenv('OPENAI_API_KEY')}",

        "Content-Type": "application/json",

    },

    json={

        "model": "gpt-4o",

        "messages": [

            {"role": "user", "content": "How are you?"}

        ],

        "logprobs": True,

        "top_logprobs": 5

    }

)



response_json = response.json()

logprobs = response_json["choices"][0]["logprobs"]

next_token_logprobs = logprobs["content"][0]["top_logprobs"]



for item in next_token_logprobs:

    token, logprob = item["token"], item["logprob"]

    prob = math.exp(logprob)

    print(token, prob)

This will output something along the lines of:

Thank 0.903825743563041

I'm 0.09526252257393902

I 0.0004998919591426934

Thanks 0.0003893162492314283

Hello 1.9382905474713714e-05

This means the model predicts Thank as the next token with a probability of 0.90, I'm with 0.09, and so on.

Sampling from the List

Now that we have a list of probabilities, how do we use it to generate the next token?

The simplest approach is to use greedy sampling.

This simply means selecting the token with the highest probability:

def greedy_sample(logprobs):

    return max(logprobs, key=lambda item: item["prob"])



next_token_logprobs = [

    {"token": "Apple", "prob": 0.6},

    {"token": "Banana", "prob": 0.3},

    {"token": "Cherry", "prob": 0.1},

]

print(greedy_sample(next_token_logprobs))

This will output:

{'token': 'Apple', 'prob': 0.6}

Another approach is to actually sample from the list.

This involves randomly selecting a token from the list, with each token weighted by its probability.

The higher the probability, the more likely the token will be selected.

import random

from collections import defaultdict



def sample_from_list(logprobs):

    return random.choices(logprobs, weights=[item["prob"] for item in logprobs], k=1)[0]



next_token_logprobs = [

    {"token": "Apple", "prob": 0.6},

    {"token": "Banana", "prob": 0.3},

    {"token": "Cherry", "prob": 0.1},

]



counts = defaultdict(int)

for _ in range(1000):

    counts[sample_from_list(next_token_logprobs)["token"]] += 1



print(counts)

This will output something along the lines of:

{'Apple': 598, 'Banana': 303, 'Cherry': 99}

Note how the counts of every token are roughly proportional to their probabilities.

Greedy sampling has a few clear advantages: it's simple, fast, and fully deterministic.

Nevertheless, it comes with a downside: it always selects the most likely token—even when that token’s probability is relatively low.

As a result, greedy sampling is often associated with repetitive output.

This concern was highlighted in the famous paper The Curious Case of Neural Text Degeneration which shows that greedy sampling—and its close relative, beam search—often leads to repetitive text.

However, that study focused on GPT-2, a model that is outdated by today’s standards.

More recent research paints a more nuanced picture.

For instance, The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism found that greedy sampling actually outperformed more complex methods in some cases.

Similarly, A Thorough Examination of Decoding Methods in the Era of LLMs argues that no single sampling method is the best—it all depends on the task at hand.

In practice, that does seem to hold true.

In short, while probabilistic sampling is typically the default, greedy sampling can be a reasonable—and at times even preferable—alternative.

The discussion around greedy sampling and probabilistic sampling highlights just how shaky the foundations of LLMs are and how quickly the field moves.

We still lack a definitive answer to something as basic as the best sampling method—let alone more complex questions.

If you found this helpful, drop a ❤️ and hit Follow to get more dev insights in your feed!