Make an Outbox Relay Lease Expire Without Violating Delivery Invariants

Relay A claims outbox row 42 and publishes its message. Before A records completion, its lease expires. Relay B claims row 42 and publishes it again. Both relays behaved according to their local state; the broker received a duplicate.

A lease prevents unlimited concurrent ownership. It cannot create exactly-once delivery across a database and broker without a shared transaction.

The honest contract is:

Every committed outbox event is eventually offered at least once; consumers preserve business invariants under duplicate offers.

State model

pending

  -> leased(owner, token, expires_at)

  -> published



leased --expiry--> claimable

leased --failure--> claimable

A fencing token increments on every claim:

UPDATE outbox

SET lease_owner = :worker,

    lease_token = lease_token + 1,

    lease_expires_at = now() + interval '30 seconds'

WHERE id = :id

  AND published_at IS NULL

  AND (lease_expires_at IS NULL OR lease_expires_at < now())

RETURNING id, event_id, payload, lease_token;

Completion is conditional:

UPDATE outbox

SET published_at = now()

WHERE id = :id

  AND lease_owner = :worker

  AND lease_token = :token

  AND published_at IS NULL;

The token stops stale relay A from marking B's newer lease complete. It does not retract A's already published message.

Minimal event-order simulator

Model actions instead of threads:

state = {

    "token": 0,

    "lease": None,

    "published": False,

    "broker": [],

}





def claim(worker):

    state["token"] += 1

    state["lease"] = (worker, state["token"])

    return state["token"]





def send(worker, token):

    state["broker"].append({"event": "e42", "worker": worker, "token": token})





def complete(worker, token):

    if state["lease"] == (worker, token):

        state["published"] = True





a = claim("A")

send("A", a)

# Time advances; A's lease expires.

b = claim("B")

send("B", b)

complete("A", a)  # fenced out

complete("B", b)



assert len(state["broker"]) == 2

assert state["published"] is True

The duplicate is not a simulator bug. It is the counterexample that defines the consumer requirement.

Consumer invariant

Suppose the event awards a $10 credit. The wrong handler performs an unconditional increment twice. A robust handler stores event identity in the same transaction as the business effect:

BEGIN;



INSERT INTO consumed_events (consumer, event_id)

VALUES ('credit-service', 'e42')

ON CONFLICT DO NOTHING;



-- Continue only when the insert affected one row.

UPDATE accounts SET credit_cents = credit_cents + 1000 WHERE id = :account;



COMMIT;

The invariant is not “the handler ran once.” It is:

For each (consumer, event_id), the credit effect commits at most once.

Other event types need different invariants. A state projection can use monotonic versions. A notification may need a delivery ledger. A cache invalidation can be naturally idempotent. Do not apply one generic deduplication recipe without naming the business effect.

Failure classes to inject

Generate permutations around claim, send, complete, consume, and acknowledge. The acceptance rule should inspect durable state, not log-line counts.

Tradeoffs

Inbox retention must cover the maximum plausible redelivery interval. Deleting deduplication records after one day while a dead-letter queue retains messages for a week reopens old effects.

Limits

This model assumes the source transaction atomically writes domain state and the outbox row. It does not address event ordering across aggregates, broker partition changes, poison messages, or multi-region conflict resolution. Those require additional invariants.

Exactly-once can sometimes be provided inside one technology boundary. Across independently failing database, relay, broker, and consumer boundaries, state the narrower guarantee and test the violating event order. For this protocol, lease expiry after send is the sequence every design must survive.