Duplicate processing is not a bug.

It is the default behavior of reliable distributed systems.

Every distributed system eventually faces the same uncomfortable truth:

The moment you introduce:

• Retries


• Failovers


• Message brokers


• Network partitions


• Service crashes

duplicates become inevitable.

Yet many engineers still believe that "exactly-once processing" is something a broker can magically provide.

It isn't.

In this article we'll explore:

• Why exactly-once delivery is mostly a myth


• Why at-least-once is the real-world standard


• How idempotency actually works


• How to build Inbox and Outbox patterns in Go


• How Kafka, PostgreSQL, and Redis fit together in production

🧨 Exactly-Once Is Not a Transport Property

A message broker cannot guarantee exactly-once processing across your entire system.

Exactly-once delivery requires a Single Point of Truth (SPoT) capable of enforcing uniqueness.

Why?

Because delivery, processing, and persistence are separate failure domains.

Even if a broker behaves perfectly:

• Your service can crash after processing


• Your database can commit while ACK fails


• Your consumer can retry unknowingly

This creates unavoidable duplication scenarios.

📊 Delivery Semantics in Real Systems

👉 In practice, everything is at-least-once.

💥 Why Duplicates Happen

Consumer Crash After Processing

func handle(msg Message) error {

    if err := process(msg); err != nil {

        return err

    }



    return ack(msg)

}

`

Failure window:

text

Process Message ✅

Commit Database ✅

Crash Service ❌

ACK Never Sent ❌



Result:

text

Broker thinks processing failed

↓

Message redelivered

↓

Duplicate processing



Network Timeout After Success

`go

err := process(msg)

if err != nil {

return err

}

return broker.Ack(msg)

`

What happens if:

text

ACK sent

↓

Network timeout

↓

Broker never receives ACK



The broker retries.

The operation runs again.

Retry Storms

A temporary latency spike can trigger:

text

Client Retry

↓

Gateway Retry

↓

Service Retry

↓

Consumer Retry



Result:

text

1 failure

↓

50 duplicate executions



🧠 The Real Solution: Idempotency

Instead of preventing duplicates:

text

Wrong Question:

How do I prevent duplicates?



Ask:

text

Correct Question:

How do I make duplicates harmless?



That's where idempotency comes in.

An idempotent operation produces the same final state no matter how many times it executes.

🚨 Naive Payment Service

Let's start with a broken implementation.

`go

func Charge(

ctx context.Context,

req PaymentRequest,

) error {

_, err := db.Exec(ctx, `

    INSERT INTO payments (

        id,

        amount

    )

    VALUES ($1, $2)

`,

    req.ID,

    req.Amount,

)



return err

}

`

Now imagine:

text

Request arrives

↓

Payment inserted

↓

Response lost

↓

Client retries

↓

Payment inserted again



Customer charged twice.

✅ Production Solution #1: Database Unique Constraints

Create an Inbox table.

sql

CREATE TABLE idempotency_keys (

event_id TEXT PRIMARY KEY,

created_at TIMESTAMP DEFAULT now()

);



Attempt to claim the key first.

go

_, err := tx.Exec(ctx,

INSERT INTO idempotency_keys(event_id)

VALUES($1)

`, eventID)

if err != nil {

var pgErr *pgconn.PgError

if errors.As(err, &pgErr) &&

   pgErr.Code == "23505" {



    return nil

}



return err

}

`

The database becomes the source of truth.

⚠️ Common pgx Pitfall

This subtle bug catches many Go teams.

Wrong:

go

import "github.com/jackc/pgconn"



Correct:

go

import "github.com/jackc/pgx/v5/pgconn"



Otherwise:

go

errors.As(err, &pgErr)



silently fails.

Your duplicate protection stops working.

🧪 Testing Duplicate Safety

Let's simulate 100 concurrent requests.

`go

func TestIdempotency(

t *testing.T,

) {

var wg sync.WaitGroup

for i := 0; i < 100; i++ {

    wg.Add(1)



    go func() {

        defer wg.Done()



        processOrder(

            context.Background(),

            "same-event-id",

        )

    }()

}



wg.Wait()

}

`

Expected outcome:

text

100 requests

↓

1 insert succeeds

↓

99 rejected safely



⚡ Redis Optimization

Postgres guarantees correctness.

Redis improves performance.

Use Redis only as:

text

Fast Lock Layer



Not as:

text

Source of Truth



Atomic Lua Guard:

`lua

local key = KEYS[1]

if redis.call("GET", key) then

return 1

end

redis.call(

"SET",

key,

"PENDING",

"EX",

60

)

return 0

`

This protects the database from thundering herds.

📦 Kafka Exactly-Once: What It Actually Means

Many engineers believe:

`text

Kafka Exactly Once

Business Logic Executes Once

`

Wrong.

Kafka guarantees:

text

Producer

↓

Kafka

↓

Consumer



Kafka does NOT guarantee:

text

Producer

↓

Kafka

↓

Consumer

↓

Postgres



The moment you touch an external database:

exactly-once disappears.

⚠️ Disable Auto Commit

Never rely on Kafka auto commits.

Bad:

go

enable.auto.commit=true



Good:

go

consumer, _ := kafka.NewConsumer(

&kafka.ConfigMap{

"enable.auto.commit": false,

},

)



Process first.

Commit later.

`go

err := handleOrder(

ctx,

db,

msg,

)

if err == nil {

consumer.CommitMessage(msg)

}

`

📤 Solving the Dual-Write Problem

This architecture is broken:

text

Insert Order

↓

Publish Event



What if Kafka is down?

text

Order exists

↓

Event lost forever



That's the Dual Write Problem.

🚀 Transactional Outbox Pattern

Store event publishing intent in the same transaction.

sql

CREATE TABLE outbox (

id UUID PRIMARY KEY,

payload JSONB,

status TEXT DEFAULT 'PENDING'

);



go

_, err = tx.Exec(ctx,

INSERT INTO orders(...)

VALUES(...)

`)

_, err = tx.Exec(ctx,

INSERT INTO outbox(...)

VALUES(...)

)

`

Commit both together.

🔄 Outbox Worker

`go

type OutboxWorker struct {

db *pgxpool.Pool

broker Broker

}

func (w *OutboxWorker) Run(

ctx context.Context,

) {

ticker := time.NewTicker(

time.Second,

)

defer ticker.Stop()



for {

    select {

    case <-ctx.Done():

        return



    case <-ticker.C:

        w.processBatch(ctx)

    }

}

}

`

⚡ Scaling Workers Safely

Use PostgreSQL native locking.

sql

SELECT *

FROM outbox

WHERE status = 'PENDING'

FOR UPDATE SKIP LOCKED

LIMIT 10;



Benefits:

• No deadlocks


• No duplicate workers


• Infinite horizontal scaling

💳 The Hardest Part: External Side Effects

Database writes are easy.

External systems are not.

Examples:

• Stripe


• Twilio


• SendGrid


• Payment Gateways

Without idempotency:

text

Duplicate Message

↓

Duplicate Payment

↓

Real Money Lost



Always verify external APIs support idempotency keys.

🛡️ Graceful Shutdown

Kubernetes gives you roughly:

text

30 seconds



before SIGKILL.

Handle shutdown properly.

`go

ctx, stop := signal.NotifyContext(

context.Background(),

syscall.SIGTERM,

)

defer stop()

`

Allow in-flight transactions to finish.

📈 Structured Observability

Track duplicates explicitly.

go

slog.Info(

"duplicate detected",

"event_id",

eventID,

)



If you can't measure duplicates:

you can't prove idempotency works.

🏗️ Final Production Architecture

text

Kafka

│

▼

Inbox Pattern

(Idempotency)

│

▼

PostgreSQL

│

▼

Outbox Pattern

│

▼

Kafka / RabbitMQ



This architecture embraces reality:

`text

At-Least-Once Delivery

+

Idempotency

+

Recovery

Production Reliability

`

🚀 Key Takeaways

• Exactly-once delivery does not exist across distributed systems.


• At-least-once delivery is the industry standard.


• Idempotency transforms duplicates into harmless events.


• PostgreSQL UNIQUE constraints should be the source of truth.


• Redis is an optimization layer, not a consistency layer.


• Kafka auto commits can cause data loss.


• Inbox + Outbox patterns form the backbone of reliable event-driven systems.


• Reliability is not about preventing failures.


• Reliability is about remaining correct despite failures.