Mutex deadlocks in production: the patterns I found in my codebase and how I diagnosed them

It was 11:47 PM and the service wasn't responding. No panic. No error in the logs. Railway showed the container alive, memory stable, CPU at zero. Zero. That's what caught my attention: zero activity on a service that should've been processing queues. I opened a tokio-console session and there it was — four tasks suspended, all waiting on the same MutexGuard. None of them were ever going to move.

That was the first one. Then came the second. Then the third. All three with the same face: total silence, container "healthy," and a chain of locks that was never going to resolve itself.

My thesis is this: mutex deadlocks in async Rust aren't rare or mysterious. They're predictable. They follow patterns. And once you've seen one, you recognize them from a mile away. The problem is that most resources teach you what a deadlock is, not how you diagnose it when it's already in production and you can't just pull a backtrace.

Mutex deadlocks in async Rust: why this isn't a trivial problem

The specific problem with async Rust isn't that locks are hard to understand. It's that tokio::sync::Mutex and std::sync::Mutex behave differently in ways that aren't obvious until something blows up.

When you use std::sync::Mutex inside an async runtime, if a task takes the lock and then does .await, you block the executor's entire thread. Not just your task — the whole thread. Every task running on that worker thread gets suspended. With a single-threaded runtime, the entire program freezes.

// ⚠️ This is poison in async — you're blocking the executor

use std::sync::Mutex;



async fn process_item(state: Arc<Mutex<State>>) {

    let guard = state.lock().unwrap(); // synchronous block

    do_async_io().await;               // while the thread is blocked

    // guard drops here — but the damage is already done

}

With tokio::sync::Mutex the behavior changes: .lock().await suspends the task, not the thread. The executor can keep running other tasks while you wait for the lock. But that doesn't save you from deadlocks — if you have circular dependencies, you're still stuck, just more politely.

What I found in my codebase, after three incidents, is that I had three distinct deadlock patterns. Internally I call them: the Classic Deadly Embrace, the Reentrant Lock, and the Inverted Order Under Pressure.

The three concrete patterns I found and how I reproduced them

Pattern 1: The Classic Deadly Embrace

This is the most well-known one but it still got me. Two tasks, two resources, inverse acquisition order.

// Reproduction of the first real deadlock

// Task A: takes lock_cache, then asks for lock_db

// Task B: takes lock_db, then asks for lock_cache



async fn task_a(

    cache: Arc<Mutex<Cache>>,

    db: Arc<Mutex<DbPool>>,

) {

    let _cache_guard = cache.lock().await;     // Task A takes cache

    tokio::time::sleep(Duration::from_millis(1)).await; // pause = deadlock window

    let _db_guard = db.lock().await;           // Task A waits for db — which Task B holds

}



async fn task_b(

    cache: Arc<Mutex<Cache>>,

    db: Arc<Mutex<DbPool>>,

) {

    let _db_guard = db.lock().await;           // Task B takes db

    tokio::time::sleep(Duration::from_millis(1)).await;

    let _cache_guard = cache.lock().await;     // Task B waits for cache — which Task A holds

}

The fix isn't just "acquire locks in the same order." The real fix is asking yourself whether you need both locks at the same time. In my case, I didn't — I restructured to acquire, operate, release, and only then acquire the second.

// Fixed version: explicit scope, no guard overlap

async fn task_a_fixed(

    cache: Arc<Mutex<Cache>>,

    db: Arc<Mutex<DbPool>>,

) {

    // First we work with cache and release it

    let data = {

        let guard = cache.lock().await;

        guard.get_data()

    }; // guard dropped here



    // Only then do we use db

    let mut db_guard = db.lock().await;

    db_guard.write(data).await;

}

Pattern 2: The Reentrant Lock

This one took me longer because it didn't look like a classic deadlock. A single function, a single mutex. The problem: the function was calling itself (indirectly, through a callback) while it already held the lock.

// The internal callback was calling the same function that already held the lock

async fn process_event(

    state: Arc<Mutex<State>>,

    event: Event,

) {

    let mut guard = state.lock().await;



    // This internal handler calls process_event again

    // with the same Arc<Mutex<State>> — guaranteed deadlock

    guard.run_handlers(&event).await;

}

Rust doesn't have a reentrant RwLock in std, and tokio::sync::Mutex isn't reentrant either. The fix was separating the state the handler needs from the state the main lock holds, or cloning the necessary data before releasing the guard.

// Fix: clone what you need, drop the lock, then run handlers

async fn process_event_fixed(

    state: Arc<Mutex<State>>,

    event: Event,

) {

    // Take what we need and release the lock

    let handlers = {

        let guard = state.lock().await;

        guard.handlers_for(&event).clone() // deliberate clone

    }; // guard dropped



    // Run handlers without holding the lock

    for handler in handlers {

        handler.run(&event).await;

    }

}

Pattern 3: Inverted Order Under Pressure

This is the most treacherous one because the code never fails in development. It only shows up when there's real concurrency, under load, with multiple replicas. I saw it in production when Railway started horizontally scaling the service.

The pattern: you have a lock acquisition order that looks consistent in the code, but under pressure, tasks interleave at exactly the right moment where the effective order inverts. Related to this — in my ) wouldn't have concurrency problems because "it's async." Async doesn't protect you from deadlocks. It changes how they express themselves.

I validated this same intuition when I analyzed — and it eliminated an entire class of problems.

Does Clippy or any static analysis catch deadlocks?

No. Clippy doesn't detect deadlocks in async. Neither does the compiler. It's a runtime behavior problem, not a type problem. There are community proposals to add lock order analysis, but nothing stable yet. The only real tools I have are tokio-console at runtime and explicit timeouts on critical locks. Rust's static analysis is extraordinary for many things — I even validated it against things