Stream Postgres WAL to Redis: Real-Time Read Models with pg2redis

Real-time data pipelines are becoming a normal part of application architecture. Product pages need fresh stock counts, order status screens need fast reads, dashboards need recent transactional state, and background services often need to react as soon as data changes.

Postgres is usually the source of truth, but many applications still keep a second read model in Redis for low-latency access. The hard part is keeping Redis in sync without pushing cache update logic into every write path.

The tempting approach is to write to both systems in the same request handler:

type User struct {

    ID        int64     `json:"id"`

    Email     string    `json:"email"`

    FullName  string    `json:"full_name"`

    UpdatedAt time.Time `json:"updated_at"`

}



func SaveUser(ctx context.Context, db *pgxpool.Pool, redisDb *redis.Client, user User) error {

    const upsertUser = `

INSERT INTO users (id, email, full_name)

VALUES ($1, $2, $3)

ON CONFLICT (id) DO UPDATE

SET email = EXCLUDED.email,

    full_name = EXCLUDED.full_name,

    updated_at = now()

RETURNING updated_at`



    if err := db.QueryRow(

        ctx,

        upsertUser,

        user.ID,

        user.Email,

        user.FullName,

    ).Scan(&user.UpdatedAt); err != nil {

        return fmt.Errorf("save user in Postgres: %w", err)

    }



    payload, err := json.Marshal(user)

    if err != nil {

        return fmt.Errorf("marshal user: %w", err)

    }



    key := fmt.Sprintf("user:%d", user.ID)

    if err := redisDb.Set(ctx, key, payload, 0).Err(); err != nil {

        return fmt.Errorf("save user in Redis: %w", err)

    }



    return nil

}

This looks reasonable until the second write fails. If the Postgres upsert succeeds and the Redis SET times out because of a network issue, the source of truth now has the new user and Redis still has the old value or no value at all. The handler returns an error, but the database change already happened.

Retried requests can make this even harder to reason about. A retry may repair Redis, or it may race with another update. Moving the Redis write before the database write only flips the failure mode: Redis can be updated and then the database transaction can fail. A regular Postgres transaction cannot include Redis, so this is not an atomic operation.

This is why cache updates and read-model updates often belong on the committed change stream, not directly in the request path.

That is the problem pg2redis is designed to solve.

pg2redis reads Postgres logical replication changes from the Write-Ahead Log, or WAL, and turns row-level inserts, updates, deletes, and snapshots into Redis commands. You configure what should happen for each table, and pg2redis keeps Redis updated from the database change stream.

The documentation is available at .

Why You Might Need This

Consider a few common cases:

• You keep product, customer, or order read models in Redis.


• You want Redis hashes, sets, sorted sets, streams, or pub/sub notifications to follow Postgres changes.


• You want to initialize Redis from existing tables and then continue streaming new changes.


• You want a focused Postgres-to-Redis pipeline without Kafka, Debezium, or custom cache invalidation code.

There are great tools for large CDC platforms and broad event streaming. If you need schema registries, replayable topic history, and many independent downstream consumers, those tools may be the right fit.

But if the job is "Postgres changes should update Redis", a smaller tool can be easier to deploy, reason about, and operate.

How It Works

At a high level, pg2redis does four things:

1. Connects to Postgres using logical replication.


2. Reads WAL changes through the built-in pgoutput plugin.


3. Expands configured Redis command templates for inserts, updates, deletes, and snapshots.


4. Writes those commands to Redis using batched pipelines.

The mapping is explicit. For example, a row in public.products can become:

tables:

  - public.products:

      insert:

        commands:

          - ["HSET", "product:{id}", "{pairs:*}"]

          - ["SADD", "products:all", "{id}"]

          - ["ZADD", "products:stock", "{stock_quantity}", "{id}"]

      update:

        commands:

          - ["HSET", "product:{id}", "{pairs:*}"]

          - ["ZADD", "products:stock", "{stock_quantity}", "{id}"]

      delete:

        commands:

          - ["DEL", "product:{id}"]

          - ["SREM", "products:all", "{id}"]

          - ["ZREM", "products:stock", "{id}"]

With this configuration, Postgres remains the source of truth. Redis becomes a maintained projection.

Advanced Features Under The Hood

1. Flexible Redis command mapping

pg2redis supports common Redis command families, including:

SET, SETEX, MSET, INCR, DECR, INCRBY, DECRBY, APPEND, DEL, EXPIRE,

HSET, HINCRBY, HDEL,

SADD, SREM,

ZADD, ZINCRBY, ZREM,

XADD, XDEL,

PUBLISH

That means you can model different Redis access patterns from the same database change stream:

• Hash per row with HSET
  


• JSON value per row with SET
  


• Membership indexes with SADD and SREM
  


• Ranked or scored indexes with ZADD and ZREM
  


• Event feeds with XADD
  


• Realtime notifications with PUBLISH
  

Templates can use column values:

["HSET", "order:{id}", "{pairs:*}"]

["SET", "order_json:{id}", "{json:*}"]

["PUBLISH", "order_updates", "{json:id,customer_id,status,total_cents}"]

2. Operation-specific behavior

Inserts, updates, and deletes usually need different Redis behavior. An insert may add a key and index membership. An update may refresh a hash and sorted set score. A delete may remove the key and clean up indexes.

pg2redis lets each operation have its own command list:

insert:

  commands:

    - ["HSET", "customer:{id}", "{pairs:*}"]

    - ["SADD", "customers:all", "{id}"]

update:

  commands:

    - ["HSET", "customer:{id}", "{pairs:*}"]

delete:

  commands:

    - ["DEL", "customer:{id}"]

    - ["SREM", "customers:all", "{id}"]

3. Conditional commands

Sometimes a Redis index should only include rows that match a condition. For example, active products can be maintained as a set:

- command: ["SADD", "products:active", "{id}"]

  condition:

    column: active

    op: "="

    value: "true"

- command: ["SREM", "products:active", "{id}"]

  condition:

    column: active

    op: "="

    value: "false"

Conditions can also compare current and previous values when Postgres provides the old row data. That is useful for status transitions, counters, and selective notifications.

4. Initial snapshots

Redis often starts empty. A streaming-only process can handle new changes, but it will not automatically load data that already exists.

pg2redis can run an initial snapshot first:

snapshot:

  mode: onetime

  batchSize: 100

  parallelWorkers: 2

  abortOnError: true

  config:

    - name: public.customers

      type: full

    - name: public.products

      type: full

    - name: public.orders

      type: full

    - name: public.order_items

      type: full

Snapshot rows use the configured insert commands. After the snapshot finishes, pg2redis continues from the snapshot LSN and streams new WAL changes.

5. Batched Redis writes and retries

pg2redis buffers row changes and flushes them to Redis in pipelines. You can tune the batch size, flush interval, queue depth, and number of flush workers:

flushInterval: 100ms

flushBufferSize: 10

flushQueueDepth: 8

flushWorkers: 2

writeTimeout: 5s

maxWriteQueueSize: 10000

Larger batches reduce round trips. Smaller batches and shorter flush intervals reduce latency.

A practical ordering note: flushWorkers controls how many Redis pipeline batches can be executed at the same time. With flushWorkers > 1, more than one batch can be in flight, so a later batch can finish before an earlier batch. If you need best-effort batch ordering for keys touched by consecutive changes, use flushWorkers: 1. That makes normal batch execution single-worker and FIFO, but it is not a total ordering guarantee.

Note: Right now, ordering is not guaranteed when an entry is retried. The retry tracker schedules failed entries for later and re-enqueues them as smaller write requests, usually one failed row entry at a time. Later WAL entries may already have been flushed to Redis before the retry is applied. For that reason, mappings that depend on exactly once, exactly ordered side effects need extra care. Prefer idempotent writes such as HSET, SADD, ZADD, and DEL for derived state. Be careful with INCRBY, ZINCRBY, XADD, and PUBLISH if delayed or duplicate application would be a problem.

Postgres transaction boundaries are preserved for tracking, not for Redis atomicity. The logical replication listener collects row changes by XID and sends one write request after the Postgres commit. That request carries one LSN, XID, and commit time, and the application does not release that commit point until all row entries are acknowledged or skipped. The Redis writer, however, enqueues those row entries individually and batches them by flushBufferSize or flushInterval. A single Postgres transaction can share a Redis batch with other transactions, or if it is large enough it can be split across more than one Redis batch. For one row change, multiple configured Redis commands are wrapped in MULTI/EXEC; the entire Postgres transaction is not wrapped in one Redis MULTI/EXEC.

Stronger ordering guarantees and better preservation of Postgres transaction boundaries are work in progress and will be addressed in future versions.

Retry behavior is configurable as well:

retryPolicy:

  maxRetries: 10

  maxConnectionRetries: 0

  initialBackoff: 1500ms

  multiplier: 2

  jitter: 0.1

  maxBackoff: 15s

The tool stores processing state so it can continue from a known WAL position. The delivery model is at-least-once, so Redis mappings should be designed to be idempotent where possible.

Getting Started With The Example App

The repository includes a complete e-commerce demo at .

The Docker image is available at alikpgwalk/pg2redis.

Try the pg2redis-example demo if you want to see the full loop: write to Postgres, stream WAL through pg2redis, and read the Redis projection from the API.

If your application already treats Postgres as the source of truth and Redis as a fast read model, this approach keeps that boundary clean. Your write path stays focused on the database, and Redis follows along from the committed WAL stream.