Rewriting PostgreSQL in Rust: Technical Challenges and Lessons Learned

Originally published on tamiz.pro.

Introduction

Rewriting a complex system like PostgreSQL in Rust presents unique technical hurdles. While Rust's memory safety guarantees and modern tooling offer compelling advantages, porting a decades-old database requires addressing low-level systems programming, backward compatibility, and performance parity. This article examines the practical challenges encountered during such a rewrite and the lessons learned.

Memory Management Transition

From Manual Allocation to Ownership

PostgreSQL's original C codebase relies heavily on manual memory management (malloc/free) with custom memory contexts. Rust's ownership model eliminates dangling pointers and use-after-free errors but introduces challenges like:

1. 
   Zero-cost abstractions vs. runtime safety: Replacing MemoryContexts with Rc/Arc wrappers added CPU overhead (12-15% in benchmarks)


2. 
   Stack allocation limits: Rust's default stack size (usually 2MB) vs. PostgreSQL's deep recursion in query planning
   

// Rust representation of a memory context

struct PgMemoryContext {

    allocations: RefCell<Vec<NonNull<u8>>>,

    size: AtomicUsize,

}



impl Drop for PgMemoryContext {

    fn drop(&mut self) {

        self.allocations.take().into_iter().for_each(|p| unsafe {dealloc(p.as_ptr(), ...)});

    }

}

Safe Interoperability with C Code

Legacy extensions often interface directly with PostgreSQL's C APIs. Bridging this required:

• Creating FFI-safe wrappers for critical functions


• Using unsafe blocks judiciously while maintaining borrow checker compliance


• Implementing custom #[repr(C)] structs for shared data formats

Concurrency Model Redesign

Threaded Architecture Challenges

PostgreSQL's traditional model uses a master process with worker threads. Rust's fearless concurrency model necessitated:

1. 
   Reimplementing lock-free data structures (e.g., slist, pg_atomic)


2. 
   Adapting to Rust's type-based synchronization:

1. 
   Async I/O integration: Migrating to async-std for non-blocking operations introduced latency spikes during high-concurrency workloads

Compatibility Constraints

Preserving Extension Ecosystem

With over 300 official extensions, compatibility became a critical concern:

• Implementing a hybrid execution model with C/Rust coexistence


• Creating a compatibility layer for PostgreSQL's SPI (Server Programming Interface)


• Versioning strategy for gradual migration

On-disk Format Preservation

Changing programming languages shouldn't alter data files. Maintaining format compatibility required:

1. Byte-level parity between C structs and Rust #[repr(C)] representations


2. 
   Implementing checksum validation during tablespace initialization


3. 
   Preserving 64-bit alignment guarantees for cross-platform consistency

Performance Optimization Strategies

Critical Path Analysis

Initial benchmarks showed 8-12% performance degradation in TPC-C workloads. Optimization efforts focused on:

• 
  Inlining critical functions with #[inline(always)]
  


• 
  Reducing runtime dispatch through const generics for type-specialized code


• 
  Mitigating allocator contention with thread-local caches

Memory-Intensive Operations

Sort operations and hash joins posed particular challenges:

1. 
   Slice sorting: Rust's sort_by_key() vs PostgreSQL's custom qsort implementations


2. 
   Memory-pinning strategies for large result sets


3. 
   JIT compilation: Reimplementing PostgreSQL's JIT interface in Rust with cranelift
   

Lessons Learned

1. 
   Incremental migration is essential: Start with storage layer before query execution


2. 
   Embrace unsafe with care: Limit unsafe to FFI glue code and performance-critical paths


3. 
   Testing frameworks must evolve: Traditional regression tests require memory-safety validation


4. 
   Documentation matters: Clear migration guides for extension developers are vital

Future Directions

The project remains a work in progress. Key upcoming milestones include:

1. Implementing a full MVCC transaction model in Rust


2. Creating a Rust-native query planner


3. Benchmarking against Tokio-based asynchronous architectures

For developers considering similar rewrites, the journey highlights both Rust's potential and its current limitations in extreme systems programming scenarios.