The Reconciler Pattern: When a Queued Job Simply Never Runs

TL;DR

• A job dispatched once, at row-creation time, has a silent failure mode: if that dispatch is lost, the row sits pending forever — and never appears in failed_jobs.


• Retries only exist for jobs that ran. A job that never entered the queue has nothing to retry.


• Fix: a reconciler — a scheduled sweep for stale, unclaimed rows that re-dispatches them. The database is the source of truth; the queue is just delivery.

The bug that leaves no trace

Records stuck in pending. No error, no failed job, no log line. The shape of the code:

$status = SyncStatus::create([...]);   // sync_status = 'pending'

ProcessSyncStatus::dispatch($status->id);

One dispatch, at creation. Fine 99% of the time. The 1%:

Only the last row is visible. The first three are invisible failures — dashboards green, data quietly rotting.

The bad mental model: "the queue is durable, so a dispatched job will eventually run." Your database row is the durable thing. The queue is a delivery hint.

Retry ≠ reconcile

Retry is a queue-level mechanism. Reconciliation is a domain-level one. You need both.

The reconciler

Periodically ask the database "does anything look stuck?", and fix it.

  every 15 min

       |

       v

+----------------------------+

|  sweep pending rows        |

|  updated_at < now()-15m    |  <- stale

|  AND claimed_at IS NULL    |  <- never picked up by a worker

+----------------------------+

       | touch(row)   <- so the next tick won't sweep it again

       v

   re-dispatch ProcessSyncStatus

Two predicates carry the weight. Stale: updated_at older than a threshold — fresh pending rows are just working, give the queue time. Unclaimed: the job stamps the row the moment a worker picks it up, so a pending-and-claimed row is in flight right now; leave it alone. Pending and unclaimed after 15 minutes was never delivered.

final class ReconcilePendingSync extends Command

{

    protected $signature = 'sync:reconcile-pending

                            {--minutes=15 : Only sweep rows untouched this long}

                            {--limit=200  : Cap re-dispatches per run}

                            {--dry-run}';



    public function handle(): int

    {

        $rows = SyncStatus::query()

            ->where('sync_status', 'pending')

            ->whereNull('claimed_at')

            ->where('updated_at', '<', now()->subMinutes((int) $this->option('minutes')))

            ->limit((int) $this->option('limit'))

            ->get();



        foreach ($rows as $row) {

            if ($this->option('dry-run')) { continue; }   // ...report only



            $row->touch();                                // don't sweep again next tick

            ProcessSyncStatus::dispatch($row->id);

            Log::info('reconciler: re-dispatched', ['id' => $row->id]);

        }



        return self::SUCCESS;

    }

}

Three things that are easy to get wrong:

• 
  touch() before dispatch. Otherwise the next tick sees the same stale row and dispatches it again — a job multiplier, not a reconciler.


• 
  --limit. If something breaks for a day, the first sweep shouldn't dump 40,000 jobs into Redis.


• 
  Idempotent job. Reconciliation will occasionally re-run something that was fine. That must be a no-op: claim the row at the top of handle(), make the work an upsert.

Tests

Queue::fake() plus a backdated updated_at covers the cases that matter:

function agePendingRow(SyncStatus $row, int $minutes): void

{

    // Bypass Eloquent timestamps to backdate updated_at.

    DB::table('sync_status')->where('id', $row->id)

        ->update(['updated_at' => now()->subMinutes($minutes)]);

}



it('re-dispatches a stale unclaimed pending row', function () {

    Queue::fake();

    $stale = SyncStatus::factory()->create(['sync_status' => 'pending', 'claimed_at' => null]);

    agePendingRow($stale, 60);



    $this->artisan('sync:reconcile-pending', ['--minutes' => 15])->assertSuccessful();



    Queue::assertPushed(ProcessSyncStatus::class, 1);

});



it('skips a pending row already claimed by a worker', function () {

    Queue::fake();

    $claimed = SyncStatus::factory()->create(['sync_status' => 'pending', 'claimed_at' => now()]);

    agePendingRow($claimed, 60);



    $this->artisan('sync:reconcile-pending', ['--minutes' => 15])->assertSuccessful();



    Queue::assertNotPushed(ProcessSyncStatus::class);

});

Plus a third: a fresh pending row must be left alone. Anyone can write a sweeper that re-dispatches everything — the value is in what it refuses to touch.

Takeaway

If a database row implies "a job should be running for this", something needs to check that claim on a schedule. Dispatch-on-create is a happy-path optimisation, not a guarantee.

The rule I'm keeping: the queue delivers, the database decides. Any state that can only be advanced by a queued job needs a reconciler behind it — otherwise "eventually consistent" really means "usually consistent, and silent when it isn't."