Building an Audio Job Queue with GPU Fallback in TypeScript

Introduction

When building AI-powered audio processing — text-to-speech, voice cloning, transcription — you need infrastructure that handles GPU scarcity gracefully. This post shares how we built a persistent audio job queue with segment-level tracking, priority scheduling, and automatic GPU-to-CPU fallback as part of the ORCHESTRATE platform's Sprint 3 Audio Engine.

This is part of our sprint retrospective series:

• Sprint 0: Foundation & Publishing Pipeline


• Sprint 1: Building the Memory System Foundation


• Sprint 2: Content Sourcing & Provenance


• Sprint 3: Production Validation & Pipeline Hardening

The Problem

Audio processing for podcasts involves multiple TTS engines:

• 
  Piper (CPU-only): Fast, lightweight, good for narration


• 
  XTTS v2 (GPU-preferred): Higher quality, voice cloning, but needs GPU

A single podcast episode has dozens of segments. Some need GPU, some don't. GPUs are expensive and scarce. We needed:

1. 
   Persistent job tracking — survive process restarts


2. 
   Segment-level progress — know exactly where a batch stands


3. 
   Priority scheduling — urgent jobs jump the queue


4. 
   GPU fallback — degrade gracefully to CPU when GPUs are busy


5. 
   Failure recovery — retry failed segments without restarting the whole job


6. 
   History cleanup — don't let old jobs fill the database

Architecture: Two Services, One Pattern

We split the solution into two injectable services:

AudioJobQueueService (453 lines)

Manages the job lifecycle:

// Submit a batch of segments for processing

const result = await queue.submitJob({

  batchName: 'Episode 42 Narration',

  priority: 'urgent',

  segments: [

    { text: 'Welcome to the show...', voiceId: 'xtts-v2-clone' },

    { text: 'Today we discuss...', voiceId: 'piper-amy' },

  ],

});



if (!result.ok) {

  console.error(result.code, result.error);

  return;

}



// Track progress

const progress = await queue.getProgress(result.data.job_id);

// { total: 2, completed: 1, failed: 0, processing: 1, pending: 0, eta_ms: 3200 }

Key methods: submitJob, getJobStatus, cancelJob, listJobs, getNextSegment, getProgress, markSegmentComplete, markSegmentFailed, recoverIncompleteJobs, retryFailedSegments, cleanupOldJobs.

AudioJobProcessor (163 lines)

Manages GPU/CPU concurrency slots:

const processor = new AudioJobProcessor({

  gpuScheduler,

  concurrency: {

    maxCpuConcurrent: 4,

    maxGpuConcurrent: 1,

    maxTotalConcurrent: 5,

  },

});



const result = await processor.processNext({

  segment_id: 'seg-001',

  voice_id: 'xtts-v2-clone',

  text: 'Hello world',

  device_hint: 'gpu',

});



if (result.ok) {

  // result.data.device might be 'cpu' with fallback_reason

  console.log(`Processing on ${result.data.device}`);

  if (result.data.fallback_reason) {

    console.log(`GPU unavailable: ${result.data.fallback_reason}`);

  }

}

The Result Pattern: No Exceptions

Every public method returns Result<T, E> per our ADR-028. Never throw:

type Result<T, E = string> =

  | { ok: true; data: T }

  | { ok: false; error: E; code: string };

This makes error handling explicit and composable. No try/catch blocks scattered through calling code. The type system enforces that callers handle both success and failure.

Injectable Dependencies: Testability Without Mocks

Both services use injectable interfaces rather than concrete implementations:

interface JobStoreLike {

  createJob(job: AudioJob): Promise<AudioJob>;

  getNextPendingSegment(): Promise<AudioSegment | null>;

  deleteCompletedJobsBefore(cutoff: string): Promise<{ deletedJobs: number }>;

  // ... 9 more methods

}



interface GpuSchedulerLike {

  acquire(timeout_ms?: number): Promise<{ ok: true; device_id: string } | { ok: false; reason: string }>;

  release(device_id: string): Promise<void>;

}

Tests provide in-memory implementations. Production provides SQLite-backed stores. No mocking framework needed — 89 tests run in under 100ms.

GPU Fallback Strategy

The processNext method implements graceful degradation:

1. Check total concurrent limit


2. If GPU requested: check GPU limit, try acquire


3. If GPU unavailable: fall back to CPU (with reason)


4. If CPU limit reached: return error (caller can retry later)

This means a voice-cloned XTTS segment that would normally use GPU will automatically process on CPU if the GPU is busy — slower but functional.

Failure Recovery

Two recovery mechanisms:

• 
  recoverIncompleteJobs(): Finds segments stuck in 'processing' state (e.g., after a crash) and resets them to 'pending'


• 
  retryFailedSegments(jobId): Retries failed segments up to maxAttempts, escalating permanently failed segments to a review queue

By the Numbers

Methodology: DD TDD

Every feature followed Documentation-Driven Test-Driven Development:

1. Document the intended behavior


2. Write failing tests


3. Implement minimum code to pass


4. Refactor


5. Validate all tests pass

Five AI personas contributed to the retrospective:

• 
  Scrum Ming (facilitator) — delivery metrics


• 
  Owen Pro (product) — podcast roadmap alignment


• 
  Api Endor (backend) — architecture patterns


• 
  Tess Ter (QA) — test coverage gaps


• 
  Aiden Orchestr (AI) — orchestration patterns

What's Next

Six decisions from the retrospective:

1. 
   SQLite integration tests (Sprint 7) — validate real database edge cases


2. 
   GPU hardware smoke tests (Sprint 7) — test real acquire/release cycles


3. 
   Audio queue UI dashboard (Sprint 8) — operator visibility


4. 
   Execution order documentation (Sprint 4) — clarify dependency-driven ordering


5. 
   Podcast episode assembly (Sprint 6) — concatenate segments into full episodes


6. 
   Memory search improvement (Sprint 4) — investigate empty recall results

The audio job queue infrastructure is ready. Sprint 6 will build podcast episode assembly on top of this foundation.

Provenance

Generated by ORCHESTRATE Agile Suite — Sprint 3 Audio Engine Retrospective