How LLMLingua compresses prompts: perplexity, deletion, and trust

I build ctxfold, a lossless prompt compressor, so I spend a lot of time thinking about the other school of prompt compression — the one that deletes things on purpose. The most important tool in that school is LLMLingua, from Microsoft Research. Before I publish a measured comparison (that post is coming), I want to explain fairly how LLMLingua actually works, because it's clever, it's genuinely effective at what it targets, and the design choice at its core is the exact opposite of mine.

The one-sentence version

LLMLingua uses a small language model to figure out which tokens in your prompt the big model could have predicted anyway — and deletes them.

That's the whole philosophical move. If a token is highly predictable from its context, removing it costs the downstream LLM very little, because the information it carried was mostly redundant. The measure of "predictable" is perplexity: tokens the small model assigns low perplexity are candidates for deletion.

The original LLMLingua (2023)

The first paper is a coarse-to-fine pipeline with three pieces:

1. Budget controller. Not all parts of a prompt tolerate compression equally. Instructions and the question tend to be information-dense; few-shot demonstrations are usually the flabby part. The budget controller assigns different compression ratios to different sections — it might keep your instruction nearly intact while cutting demonstrations hard, even dropping whole examples.
2. Iterative token-level compression. The surviving text is split into segments and compressed token by token, where each token's perplexity is computed against the already-compressed preceding context. The iteration matters: deleting tokens changes the perplexity of what follows, so a single pass would misjudge dependencies between tokens.
3. Distribution alignment. The small model (GPT2-small or LLaMA-7B scale) is instruction-tuned on data generated by the target LLM, so its sense of "predictable" tracks the big model's, not just its own.

The headline result was up to 20x compression on benchmarks like GSM8K and BBH with little loss in task performance — particularly for in-context learning and reasoning, which makes sense: few-shot demos are exactly where prompts carry the most redundancy per token.

What the output looks like

Compressed prompts from this family are not pretty. Deletion doesn't respect grammar. You get telegraphic text — function words gone, sentences collapsed into keyword runs. Something shaped like this (illustrative, not tool output):

Original:   Sam bought a dozen boxes, each with 30 highlighter pens
            inside, for $10 each box. He rearranged five of these...
Compressed: Sam bought dozen boxes each 30 highlighter pens $10 each.
            rearranged five...

The striking empirical finding — and the reason this line of work exists — is that LLMs read this stuff far better than humans do. The big model reconstructs the missing connective tissue without being asked. Microsoft's own framing is that compressed prompts may be difficult for humans while staying highly effective for LLMs.

LLMLingua-2 (2024): from perplexity to classification

The second generation changes the mechanism entirely. Instead of measuring perplexity with a small causal LM, the team had GPT-4 compress a corpus of prompts, then trained a small bidirectional encoder (XLM-RoBERTa-large) to classify each token: keep or discard, imitating GPT-4's choices. This data-distillation approach is faster (the encoder sees the whole sequence at once, no iterative left-to-right passes) and it fixed a real weakness: causal perplexity only looks backward, but whether a token matters often depends on what comes after it.

There's also LongLLMLingua in between, specialized for long-context and RAG prompts — question-aware compression, document reordering to fight "lost in the middle," and subsequence recovery.

Trying it

The library is a pip install:

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
    use_llmlingua2=True,
)
result = compressor.compress_prompt(
    context, rate=0.33, force_tokens=["\n", "?"]
)
print(result["compressed_prompt"])
print(result["origin_tokens"], "->", result["compressed_tokens"])

The rate is your target: keep a third of the tokens, delete the rest. The force_tokens escape hatch — a list of tokens that must never be deleted — is worth noticing. It exists because sometimes the classifier deletes things you needed. Which brings me to the interesting part.

The trust question

LLMLingua is lossy by design. Once compressed, the original text is unrecoverable — there is no decompressor. You are trusting that what got deleted didn't matter, and that trust is statistical: the classifier learned what GPT-4 usually considered droppable, on the corpora it was trained on.

For prose, summaries, and few-shot demos, that bet pays off, and the benchmark numbers back it up. The open question — the one I care about, because ctxfold lives on the other side of it — is what happens to exact values. SKU codes, prices, quantities, the one log line out of 300 that mattered. A rare identifier is, almost by definition, high-perplexity, which should protect it. But "should" is doing work in that sentence, and the failure mode is silent: nothing tells you a number your task depended on was deleted.

My own tool makes the opposite trade: byte-perfect losslessness, enforced by round-trip tests, at the cost of far smaller ratios (~39% on the formats it targets, vs. up-to-20x). And I recently learned lossless has its own readability limits — my CSV folding is byte-perfect and models still can't read it directly. So neither school gets to claim purity; the real axis is where the information-recovery work happens and who pays for it.

That's the comparison I want to measure rather than argue: same datasets, same exact-match questions, both tools, results whatever they are. Next post.

I write up negative results too — the CSV one is here. ctxfold is on npm.