We hit an at-least-once delivery trap. Here is how we fixed the race conditions.

If you are running an event-driven architecture with message queues like Kafka, you already know the drill, scalability is awesome, but network partitions, out-of-order delivery, and duplicate messages come with the territory.

Our team recently learned this the hard way during a production overhaul. A flood of duplicate webhooks bypassed our application level checks, creating a nasty race condition that left our database in an inconsistent state.

Most enterprise message brokers guarantee at-least-once delivery. That means your consumers will get duplicates eventually. If you are dealing with critical operations—like tracking payments or updating inventory processing a duplicate event will absolutely corrupt your data state.

The Breakdown: When validations fail concurrently

Our issue was a subtle concurrent execution race condition. Two identical update events hit our API nodes just milliseconds apart.

Because the workload was spread across multiple running container instances, both instances ran their database validation checks at the exact same moment. Both saw that the data didn't exist yet, both passed validation, and both written to the database. Our application-level guardrails were completely useless against concurrent requests.

To fix it, we had to implement the Idempotent Consumer Pattern using an atomic locking and verification layer.

Layer 1: Distributed Locking via Redis (SETNX)

Our first line of defense is a fast caching layer using Redis. Before any worker node is allowed to touch an incoming event, it must acquire an atomic distributed lock tied to the unique event_id.

We use the atomic SETNX (Set if Not Exists) command with a strict 30-second TTL to ensure locks clear automatically if a container randomly dies mid-execution:

If a duplicate message lands on worker B while worker A is still processing the original, worker B fails to get the lock and exits gracefully without touching our primary data.

Layer 2: Relational database unique constraints as the final guardrail

Redis handles 99% of the noise, but distributed networks are messy. If the Redis cluster encounters a failover or drops a packet, a duplicate could still leak through. We needed a bulletproof backup.

We added an idempotency_keys table to our core relational database with a strict Unique Constraint on the key_hash column.

Now, when a worker updates the business logic, it must write the event hash to this table inside the exact same database transaction block:

• Worker opens database transaction.


• Worker updates the core business data.


• Worker inserts the event hash into idempotency_keys.


• Worker commits the transaction.

If a duplicate somehow slips past Redis, the database engine catches the duplicate key hash, throws a UniqueConstraintViolationException, and forces an immediate, atomic rollback of the entire transaction.

The Payoff

Decoupling our validation from execution and enforcing atomic operations at both the memory and database levels completely resolved the issue. We threw a stress test of 10,000 concurrent duplicate requests per second at this pipeline, and we achieved 100% data consistency with practically zero performance overhead.