Building a SQL Database in Rust: Reducing Memory Usage with Spans and String Interning

I have been working on a PostgreSQL compatible database system in Rust and recently made a couple of changes that significantly reduced unnecessary memory allocations.

The first change was in the lexer.

Originally, identifiers were stored as String values inside token variants such as Ident(String) and QuotedIdent(String). This meant every identifier required its own allocation even though the original SQL query already contained that text.

I switched to storing spans instead. A span contains the start and end position along with line and column information. Whenever I need the actual identifier text, I can retrieve it directly from the source query using the span.

Besides reducing allocations, this also improved diagnostics because I always know exactly where a token came from.

The second change was introducing a string interner.

As I started building larger parts of the parser and AST, I noticed that names such as tables, columns, views, and aliases could appear many times throughout a query. Storing the same string repeatedly felt wasteful.

I implemented a simple interner:

pub struct Interner {
    map: HashMap<&'static str, Symbol>,
    strings: Vec<&'static str>,
}

Now identifiers are stored as compact symbols instead of duplicated strings. The actual text is stored only once and can be resolved when needed.

Some benefits of this approach:

• No duplicate allocations for repeated identifiers
• Faster identifier comparisons using integer equality
• Smaller AST nodes
• Better cache locality

The project currently includes a lexer, parser, binder, query planner, optimizer, catalog, and storage engine.

I am still exploring ways to improve performance and memory efficiency, so I would be interested to hear how others have approached similar problems in compilers, interpreters, or database systems.

Repository: https://github.com/musab05/osirisdb