Better speech synthesis for audiobooks

The Audible team introduced a controllable text‑to‑speech system for audiobooks that uses natural‑language captions to steer voice characteristics, enabling distinct character voices and improved dialogue differentiation. By building a data synthesis pipeline that pairs speech with high‑quality descriptive captions and an automated character‑profiling system that identifies dialogue with 97 % accuracy, the model can follow stage‑direction style prompts at inference. In listening tests the approach raised MUSHRA scores by 3.7 points and yielded statistically significant listening‑experience gains for eight of ten voices across English and Spanish. This demonstrates a practical path for engineers to create long‑form, expressive TTS that maintains consistent character styling throughout a book.

⚡ Key Takeaways

• Dialogue captions generated by the profiling system achieve 97 % accuracy in identifying spoken lines.
• The training pipeline augments conventional text‑speech pairs with expressive captions describing pitch, rate, emotion, and character voice.
• MUSHRA evaluation shows a 3.7‑point improvement in dialogue differentiation versus baseline TTS.
• Inference uses natural‑language prompts such as “speak in an enthusiastic, high‑pitched voice” to control output.
• The method relies on a high‑quality expressive speech dataset; without such data the controllable model cannot learn the caption mappings.

Engineers can now deploy TTS that dynamically adapts voice style per character, reducing listener fatigue and boosting engagement for long‑form content like audiobooks.

✅ Practical Steps

1. Use the described data synthesis pipeline to create a captioned speech dataset that pairs each audio segment with detailed natural‑language descriptions of its acoustic attributes.
2. Integrate the automated character‑profiling component to scan book texts, achieve 97 % dialogue detection accuracy, and generate corresponding style captions for each line.
3. At inference time, feed the controllable‑TTS model the generated captions as prompts (e.g., “enthusiastic, high‑pitched”) to produce character‑specific speech.

Want the full story? Read the original article.