I asked 4 senior Kafka engineers this question on Reddit. Nobody named a tool. So I built one.
You deployed a hotfix at 9:45 PM.
The fix was correct. The status field had been accepting invalid values — "ok", "done", "finished" — from different teams. Converting it to a strict Enum with four valid values was the right call: PENDING, PROCESSING, COMPLETED, FAILED.
Nobody checked the DLQ first.
There were 23,000 payment events sitting in payments.dlq from a consumer failure that afternoon. Your new V2 consumer tries to process them. Each one fails immediately:
com.fasterxml.jackson.databind.exc.InvalidFormatException:
Cannot deserialize value of type `PaymentStatus` from String "pending":
not one of the values accepted for Enum class:
[PENDING, PROCESSING, COMPLETED, FAILED]
PagerDuty fires at 2:00 AM.
You cannot roll back — the bug fix must stay. You cannot delete the messages — they are real payment transactions. You cannot redrive them as-is — the V2 consumer will reject every one.
You need to transform the messages first. And there is no tool for this.
Why Schema Registry Doesn't Solve This
The first suggestion you'll get is "just use Schema Registry."
Schema Registry is a prevention tool. It sits between your producer and Kafka and enforces compatibility for new messages being produced.
It does nothing for messages already sitting in your DLQ.
Those 23,000 messages are stored as bytes inside Kafka. Schema Registry cannot see them. Cannot transform them. Cannot validate them. Cannot redrive them.
This is a recovery problem, not a prevention problem. They are different.
What Senior Engineers Actually Do (This Is The Problem)
I posted this exact scenario on — an open-source Kafka Dead Letter Queue mutation and redrive engine.
Here are the four decisions that matter most.
Decision 1: assign() + seek(), Never subscribe()
This is the most critical Kafka safety decision in the codebase.
// ❌ WRONG - what most tutorials show
consumer.subscribe(List.of("payments.dlq"));
// This JOINS your consumer group.
// Kafka can trigger rebalancing and assign this partition
// to your read-only viewer tool instead of your production consumer.
// Your debug tool just took down your live pipeline.
// ✅ CORRECT - what DLQ Revive does
TopicPartition tp = new TopicPartition("payments.dlq", partition);
consumer.assign(List.of(tp));
consumer.seek(tp, fromOffset);
// Direct partition access. No group membership.
// No rebalance risk. Production consumer untouched.
// NEVER calls commitSync() in view mode.
subscribe() participates in Kafka's consumer group protocol. When you subscribe, Kafka's group coordinator can reassign partitions at any time during a rebalance. Your "read-only" DLQ viewer can suddenly become the assigned consumer for a production partition — reading and potentially skipping messages your application needs.
assign() + seek() bypasses all of this. You specify exactly which partition and offset. You read exactly limit records and stop. The rest of the cluster has no idea you exist.
Decision 2: JSONata for Transformation, Not Groovy
The transformation engine was the most debated architectural decision.
The obvious first choice was Groovy — it's powerful, familiar to Java developers, and can handle any transformation logic.
I rejected it immediately.
User-submitted Groovy executes arbitrary Java code on your backend. A careless engineer could write:
// This is a valid Groovy transformation that will be executed:
Runtime.getRuntime().exec("rm -rf /")
// Or extract your AWS credentials:
System.getenv("AWS_SECRET_ACCESS_KEY")
This is an RCE vulnerability built into the product's core feature.
Drop a comment here or open a GitHub issue. The idempotency design and the JSONata sandbox are the two areas I'm most paranoid about.
Mohammed Saifulhuq — Apache Fineract contributor (SQL injection patch, CI/CD hardening), building DLQ Revive
SOCIAL SHARE CARD GENERATOR