Inside LioranDB's Full-Text Search Segments

A normal secondary index can answer:

status = "active"

It cannot efficiently answer:

documents containing "distributed database"

LioranDB therefore has a dedicated text-segment architecture.

Tokenization

Text is split on non-alphanumeric characters.

Depending on index options, tokens can be normalized to lowercase and filtered through stopwords.

"Building Distributed Databases"



becomes



["building", "distributed", "databases"]

Segment contents

A LioranDB text segment can contain several files and structures:

Term dictionary

Posting lists

Document map

Document-length norms

Optional term positions

Bloom filter

Segment metadata

A posting connects a term to the local documents containing it.

"database" → [doc 2, doc 8, doc 19]

Positions can record where the term appears inside each document.

That enables more advanced query behaviour and phrase-aware features.

Global and local document IDs

Each segment assigns compact local IDs to its documents.

A separate document map translates them back to global document IDs.

This keeps postings smaller while preserving the external identity of the record.

Bloom filters

Each segment also maintains a Bloom filter for terms.

Before reading a segment's postings, the query path can test whether the term might exist there.

A negative answer is definitive.

A positive answer means the segment may contain the term and should be checked.

Query modes

The text query layer supports modes such as:

AND

OR

It also emits scored documents and metrics including:

• Query time


• Postings read


• Candidate documents


• Segments searched

Full-text search is essentially a specialized database living beside the document database.

Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake.

Built by Swaraj Puppalwar under Lioran Group.

Learn more:

• LioranDB


• Lioran Developer Solutions


• Lioran Group