Kafka Consumer Health Checks: Dead or Alive?

Many of you have been there - it’s 3 AM, your phone buzzes, and you’re staring at an alert: “Kafka consumer lag exceeded threshold.”

You stumble to your laptop, check the metrics, and… everything looks fine. The messages are processing. The lag was just a temporary spike from a slow downstream service. You silence the alert and go back to bed, mentally adding another item to your "fix someday" list.

Sound familiar?

Here's what's actually happening: lag tells you how far behind you are, but not whether you're making progress. A consumer can sit at 1000 messages lag for 10 minutes because it's stuck, or because it's processing at exactly the rate messages arrive. From lag alone, you can't tell the difference.

The real question isn't "how much lag?" — it's "are we making progress?"

That’s the problem I found myself wrestling with a while back. After dealing with false positive alerts and delayed detection of real issues, I discovered a simple but brilliant solution from .

Why Traditional Health Checks Fall Short

Let’s talk about common patterns and why they don’t quite work:

The “Always Healthy” Approach

func healthHandler(w http.ResponseWriter, r *http.Request) {

    w.WriteHeader(http.StatusOK)

    w.Write([]byte("OK")) // "I'm alive!" (Are you though?)

}

This tells you nothing. Your consumer could be completely frozen, and Kubernetes (any orchestrator) would happily keep it alive.

The “Ping the Broker” Approach

This is better than nothing — at least you’re verifying connectivity. But connectivity doesn’t mean your consumer is processing messages. Your network might be fine while your consumer group is stuck in an infinite rebalance loop.

The “Lag Threshold” Trap

This is where most teams land. You’re probably already tracking consumer lag through Prometheus or similar tools. You’ve set up alerts when lag exceeds some threshold — maybe 100 messages, maybe 1000, maybe a million.

But here’s the problem: what should that threshold be?

Set it too low → You wake up at 3 AM because a downstream API responded slowly for 30 seconds. The consumer is fine, just briefly overwhelmed. False alarm.

Set it too high → You miss genuine issues until they’ve already cascaded into customer-visible problems. By the time your alert fires, you’re in damage control mode.

The granularity problem: A single stuck POD in your deployment of 50 can go unnoticed if you’re monitoring average lag. That POD quietly fails while the other 49 keep working, masking the issue in your aggregate metrics.

The fundamental tension: fast detection means false positives, reliable alerts mean delayed detection.

You can build sophisticated heuristics or even ML models to detect anomalies, but all of that adds complexity and still detects problems with noticeable delay.

The Key Insight: Progress vs. Position

What we really need is a way to answer a simple question: Is this specific consumer instance making progress?

Instead of measuring lag, we measure progress. Here's the approach:

The Heartbeat: Track the timestamp of the last processed message for each partition. If we're processing messages, we're healthy.

The Verification: If enough time passes without new messages (say, X seconds), we don't panic yet. We query the Kafka broker for the latest offset. Now we can make a decision:

• 
  Consumer Offset < Broker Offset: ❌ UNHEALTHY (there are messages available, but we're not processing them — we're stuck)
  


• 
  Consumer Offset ≥ Broker Offset: ✅ HEALTHY (we're caught up, just waiting for more work — we're idle)
  

This elegantly distinguishes between a stuck consumer and a consumer that's simply idle.

How It Works: Three Scenarios

Let me show you exactly what happens in each case.

Scenario 1: Active Processing (Healthy)

When messages are flowing and your consumer is processing them, health checks are instant:

The broker query reveals there's work to do, but we're not doing it.

Scenario 3: Idle Consumer (Healthy)

The consumer is caught up, just waiting for new messages:

, a lightweight library that works with most popular Kafka clients. The core logic is decoupled from specific client implementations, with adapters provided for , and or implement this pattern yourself, it’s universal across languages and Kafka clients. The principle remains the same.

Try It Out

The repo includes a complete working example. It's the fastest way to see how this works in practice.