Stitching Hundreds of TTS Segments Into an Audiobook Without Seams

Character attribution gets all the attention in multi-voice audiobook work, and fair enough — it's the interesting ML problem. But what makes a generated audiobook feel amateur is almost never the voice casting. It's the seams. A click between paragraphs. A chapter that starts mid-copyright-notice. Volume that jumps when a different speaker takes over. Losing your place because the app re-generated and every segment boundary shifted.

That's assembly engineering. It's unglamorous, and it's where perceived quality actually lives.

EPUB structure lies to you

An EPUB is a zip with an OPF manifest, a spine giving reading order, and usually a nav document (nav.xhtml in EPUB 3, toc.ncx in EPUB 2). Naively, spine items are chapters. They are not.

Real-world breakage, all of which I've hit: one spine item containing five chapters (common in older conversions where the whole book is book.xhtml); one chapter split across four spine items because the converter chunked on file size; spine entries that are cover images, blank pages, or ads for the publisher's other titles; and nav docs pointing at fragment IDs (chapter3.xhtml#ch3), so chapter boundaries live inside files rather than at file boundaries.

So chapter detection is a merge of three signals, in priority order: nav document entries (including fragments), heading structure in the content, and spine order as fallback.

def detect_chapters(epub):
    nav = parse_nav(epub)                    # strongest signal
    if nav and len(nav.entries) >= 3:
        return split_at_anchors(epub, nav.entries)

    # fall back to headings across the flattened spine
    doc = flatten_spine(epub)
    heads = [h for h in doc.headings if h.level <= 2]
    if plausible_chapter_rhythm(heads):
        return split_at(doc, heads)

    return [Chapter(from_spine=item) for item in epub.spine]

plausible_chapter_rhythm is a sanity check worth writing: if "chapters" average 200 words, you split on scene-break headings. If one is 60% of the book, you missed the real boundaries. Reject and fall through.

Front and back matter need explicit handling because nobody wants to listen to a copyright page. Classify by position plus title matching plus length: leading items titled cover/title/copyright/dedication, trailing items titled acknowledgments/about the author/index. Don't silently delete them — mark them skippable and default them off. Some listeners want the author's note. Nobody wants ISBN digits read aloud.

The other thing that must not reach TTS: stranded page numbers, running headers repeated at every page break, and footnote markers. The result was clear.7 becomes "the result was clear seven," and it sounds broken every time.

Stitching without seams

You will generate hundreds of segments — sentence or paragraph level, sometimes across different voices. Concatenating them is where three specific artifacts come from.

1. Sample rate mismatch. Different voices, or the same engine on a different day, can hand back different sample rates. Concatenating 24kHz and 22.05kHz PCM without resampling produces pitch-shifted, chipmunked segments. Normalize everything on ingest — decode to float32, resample to one working rate, consistent channel count — and only encode at the very end.

2. Clicks at boundaries. A click is a discontinuity: segment A ends at amplitude 0.3, segment B starts at -0.2, and that instantaneous jump is broadband noise your ear hears as a tick. Fix by trimming to zero-crossings and applying a short fade:

FADE_MS = 8   # inaudible as a fade, sufficient to kill the click

def join(a, b, sr):
    a = trim_trailing_silence(a, threshold_db=-45)
    b = trim_leading_silence(b, threshold_db=-45)
    a = apply_fade_out(a, ms=FADE_MS, sr=sr)
    b = apply_fade_in(b,  ms=FADE_MS, sr=sr)
    return concat(a, b)

Trim the engine's own leading and trailing silence first — it's inconsistent between segments, and leaving it in produces erratic gaps that read as hesitation. Then re-insert silence deliberately.

3. Wrong pacing. Silence is punctuation. Fixed gaps everywhere sound robotic; no gaps sound breathless. What has worked for me:

within a paragraph, sentence to sentence   ~250-400 ms
paragraph to paragraph                     ~600-900 ms
scene break (blank line / *** )            ~1200-1500 ms
speaker change in dialogue                 ~150-300 ms  (tighter!)
chapter boundary                           ~1500-2000 ms + optional title read

Speaker changes being tighter than paragraph breaks is the counterintuitive one. Real dialogue overlaps and interrupts; a long pause between two characters trading lines destroys the sense of exchange. Multi-voice with generous gaps sounds like two people reading in separate rooms — because that's literally what it is.

Loudness normalization is not peak normalization. Different voices have genuinely different perceived loudness at the same peak amplitude, and peak-normalizing each segment makes it worse. Use an integrated loudness measure (EBU R128 / LUFS), and measure per voice across a whole chapter rather than per segment — per-segment normalization flattens intentional dynamics and makes quiet lines shout. Around −18 LUFS integrated with true peak under about −1.5 dBTP is a reasonable starting point for speech, but audiobook distributors publish their own required windows, so check before mastering.

Getting this stack right end-to-end — EPUB parse, front-matter classification, per-voice loudness, silence grammar — is most of the work behind VoxForge; the character casting was genuinely the easier half.

Playback position that survives regeneration

Last one, and it's the one that gets discovered too late to fix cheaply.

If your playback position is a byte offset or an absolute timestamp into a concatenated file, it is invalid the moment anything upstream changes. Re-cast one character, fix one mispronunciation, swap a TTS model — every downstream offset shifts and your listener is thrown to the wrong place in a nine-hour file.

Anchor position to the text, not the audio:

{
  "chapter_id": "ch07",
  "segment_id": "ch07-p012-s03",
  "offset_ms": 1420,
  "text_hash": "9f2ab1c4"
}

Segment IDs derive from document structure, so they're stable across regeneration. On resume: look up the segment, seek to its current start, add the offset. If text_hash doesn't match, the text itself changed — seek to segment start and accept losing a couple of seconds rather than landing somewhere arbitrary.

This same index is what makes chapter navigation, sentence-level scrubbing, and read-along highlighting possible later. Build it during assembly. Reconstructing text↔audio alignment afterward means forced alignment over hours of audio, and you'll wish you had written the offsets down while you had them.