[Paper Review] Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition

Ye Bai, Jingping Chen+53 more|arXiv (Cornell University)|Jul 5, 2024

Speech Recognition and Synthesis4 citations

TL;DR

Seed-ASR proposes a large language model (LLM)-based automatic speech recognition system that leverages audio-conditioned LLM (AcLLM) architecture to improve recognition across diverse speech types, including multiple languages, dialects, accents, and domains. By integrating continuous speech representations with contextual information and employing a stage-wise training recipe—self-supervised pretraining, supervised fine-tuning, context-aware fine-tuning, and reinforcement learning—Seed-ASR achieves 10%-40% relative word error rate reduction over state-of-the-art models on public and internal benchmarks.

ABSTRACT

Modern automatic speech recognition (ASR) model is required to accurately transcribe diverse speech signals (from different domains, languages, accents, etc) given the specific contextual information in various application scenarios. Classic end-to-end models fused with extra language models perform well, but mainly in data matching scenarios and are gradually approaching a bottleneck. In this work, we introduce Seed-ASR, a large language model (LLM) based speech recognition model. Seed-ASR is developed based on the framework of audio conditioned LLM (AcLLM), leveraging the capabilities of LLMs by inputting continuous speech representations together with contextual information into the LLM. Through stage-wise large-scale training and the elicitation of context-aware capabilities in LLM, Seed-ASR demonstrates significant improvement over end-to-end models on comprehensive evaluation sets, including multiple domains, accents/dialects and languages. Additionally, Seed-ASR can be further deployed to support specific needs in various scenarios without requiring extra language models. Compared to recently released large ASR models, Seed-ASR achieves 10%-40% reduction in word (or character, for Chinese) error rates on Chinese and English public test sets, further demonstrating its powerful performance.

Motivation & Objective

• To develop a more accurate and generalizable ASR system capable of handling diverse speech inputs such as multiple languages, dialects, accents, and domains.
• To overcome the limitations of end-to-end ASR models that rely heavily on external language models and struggle with contextual reasoning and out-of-domain generalization.
• To leverage the reasoning and knowledge capabilities of large language models (LLMs) within an audio-conditioned LLM framework to improve transcription accuracy and context awareness.
• To enable unified, customizable deployment across varied application scenarios without requiring additional language models.
• To establish a scalable, stage-wise training recipe that progressively enhances model performance through self-supervised learning, supervised fine-tuning, context-aware adaptation, and reinforcement learning.

Proposed method

• The model uses an audio encoder with ~2 billion parameters to extract continuous speech representations from raw audio inputs.
• These representations are fused with textual context (e.g., dialogue history, video editing context) and fed into a Mixture-of-Experts (MoE) LLM with tens of billions of parameters.
• The framework follows an audio-conditioned LLM (AcLLM) paradigm, where the LLM performs next-token prediction conditioned on both audio features and contextual prompts.
• A stage-wise training recipe is employed: (1) self-supervised pretraining on 20M+ hours of raw audio, (2) supervised fine-tuning on paired ASR data, (3) context-aware fine-tuning using rich contextual signals, and (4) reinforcement learning to align text generation with accurate transcription goals.
• Context-aware fine-tuning explicitly enhances keyword recall by modeling dependencies between speech content and surrounding contextual cues such as meeting history or dialogue flow.
• Reinforcement learning optimizes for transcription quality, especially for semantically critical phrases, by aligning model outputs with reference transcriptions using a reward model.

Experimental results

Research questions

• RQ1 Can an LLM-based ASR system significantly outperform end-to-end models in recognizing diverse speech types, including low-resource languages and regional dialects?
• RQ2 To what extent can context-aware fine-tuning improve keyword recall and transcription consistency in conversational and multi-turn settings?
• RQ3 How effective is a stage-wise training recipe combining self-supervised learning, supervised fine-tuning, context adaptation, and reinforcement learning for ASR?
• RQ4 Can a unified LLM-based architecture replace the need for separate language models while maintaining or improving performance across multiple domains and languages?
• RQ5 How does the model generalize to unseen domains and long-form speech with complex acoustic and semantic variations?

Key findings

• Seed-ASR (ML) achieves a 5.34% WER on English multi-domain evaluation sets, representing a 49% relative reduction compared to the strongest baseline (Whisper Large-v3 at 10.41% WER).
• On English multi-accent and hard-case evaluation sets, Seed-ASR (ML) achieves a 11.26% WER and 87.94% F1 score, outperforming Whisper Large-v3 (21.52% WER, 79.54% F1) and Google USM (22.19% WER, 63.30% F1).
• On multilingual multi-domain evaluation sets, Seed-ASR (ML) achieves a 12.16% WER, a 42% relative improvement over the strongest baseline (Whisper Large-v3 at 20.55% WER).
• On public test sets, Seed-ASR (ML) achieves state-of-the-art performance across 14 languages, with 1.58% WER on Librispeech test-clean, 2.84% on test-other, and 3.11% on Tedlium 3, outperforming all reported baselines.
• For low-resource languages such as Arabic (13.05% WER), Spanish (2.50% WER), and Japanese (3.46% WER), Seed-ASR (ML) shows strong generalization and competitive performance.
• The context-aware fine-tuning stage significantly boosts keyword recall in conversational and context-rich scenarios, demonstrating the model’s ability to leverage contextual information effectively.