A normal secondary index can answer:



status = "active"






It cannot efficiently answer:



documents containing "distributed database"






LioranDB therefore has a dedicated text-segment architecture.




Tokenization


Text is split on non-alphanumeric characters.

Depending on index options, tokens can be normalized to lowercase...