Documents Aren't Bags of Chunks

One thing kept bothering me about retrieval systems: they often destroy the document before trying to understand it.

Imagine taking a technical specification and cutting it with scissors every five hundred words.

No attention to headings.

No respect for sections.

No care for where a thought begins and ends.

Just cut.

Stack.

Index.

That sounds absurd.

Yet this is remarkably close to what many retrieval pipelines do before a document ever reaches a vector database.

We're asking the wrong question

One of the most common discussions around RAG sounds like this:

What's the optimal chunk size?

256 tokens?

512?

Should chunks overlap?

After watching enough retrieval failures, I started thinking this isn't the right question.

The real question isn't where to split a document.

It's whether every document should be split the same way at all.

Imagine this document

Suppose you have a technical specification called "Rate Limiting Strategy."

It contains four sections:

• Why rate limiting exists


• How the algorithm works


• A code example


• Exceptions and edge cases

Now index it using fixed-size chunks.

The heading lands in one chunk.

The code example ends up in another.

The explanation of edge cases becomes a third.

Each chunk receives its own embedding.

Each becomes an independent point in vector space.

The text survives.

The document, as a coherent unit of meaning, does not.

Retrieval works perfectly

Now someone asks:

"What happens when a user hits the rate limit?"

The retriever returns the chunk containing the code.

From its perspective, everything worked.

The identifiers match.

The embeddings are close.

The similarity score is high.

But something important has disappeared.

The heading explaining what this code belongs to is somewhere else.

The paragraph explaining why the algorithm behaves this way is somewhere else.

The exceptions are somewhere else.

The model receives valid code—but stripped of the reasoning that made the code meaningful.

It can explain what the function does.

It cannot explain why the system behaves that way.

Structure is semantics

It's easy to think of document structure as presentation.

I don't.

A heading tells the reader:

Everything below belongs to the same idea.

A section boundary says:

This thought ends here.

A list says:

These items should be understood together.

Those aren't visual decorations.

They're part of the information itself.

When those signals disappear during indexing, retrieval loses something that humans almost never notice consciously—but rely on constantly while reading.

Meaning doesn't have a fixed size

Eventually I stopped asking:

How should I split this document?

Instead I started asking:

What is the smallest unit that still makes sense on its own?

Sometimes that's a paragraph.

Sometimes it's an entire section.

Sometimes it's a code block together with the explanation above it.

Sometimes it's a diagram and its caption.

An architecture decision record often makes sense only when the decision, its rationale, and its consequences stay together.

An API reference is easier to understand when an endpoint remains connected to its parameters and examples.

Meeting notes become much more useful when decisions stay attached to owners and action items.

The right boundary isn't a character count.

It's a logical boundary.

Retrieval found something. Understanding failed.

One pattern kept repeating itself.

The retrieval engine successfully found text related to the question.

The model confidently explained that text.

And yet the actual answer was still missing.

Not because retrieval failed.

Because the document had already been dismantled before retrieval even began.

A document is an argument

I've gradually stopped thinking of documents as long strings of text.

They're closer to arguments.

Someone deliberately grouped ideas together.

Separated topics.

Added headings.

Built a flow of reasoning.

That structure carries meaning every bit as much as the words themselves.

If I break that structure into arbitrary pieces before indexing it, I shouldn't expect a retrieval engine—or an LLM—to reconstruct it perfectly later.

A document isn't a sequence of tokens.

It's an argument made visible.

Next time I want to explore a different question: why finding a document isn't the same thing as discovering knowledge.