[Paper Review] Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition

Nanxin Chen, Shinji Watanabe+2 more|arXiv (Cornell University)|Nov 10, 2019

Speech Recognition and Synthesis23 references53 citations

TL;DR

The paper proposes two non-autoregressive transformer frameworks (A-CMLM and A-FMLM) for ASR, enabling parallel decoding and achieving up to 7x real-time speedup while matching autoregressive transformer performance on Mandarin and Japanese benchmarks.

ABSTRACT

Recently very deep transformers have outperformed conventional bi-directional long short-term memory networks by a large margin in speech recognition. However, to put it into production usage, inference computation cost is still a serious concern in real scenarios. In this paper, we study two different non-autoregressive transformer structure for automatic speech recognition (ASR): A-CMLM and A-FMLM. During training, for both frameworks, input tokens fed to the decoder are randomly replaced by special mask tokens. The network is required to predict the tokens corresponding to those mask tokens by taking both unmasked context and input speech into consideration. During inference, we start from all mask tokens and the network iteratively predicts missing tokens based on partial results. We show that this framework can support different decoding strategies, including traditional left-to-right. A new decoding strategy is proposed as an example, which starts from the easiest predictions to the most difficult ones. Results on Mandarin (Aishell) and Japanese (CSJ) ASR benchmarks show the possibility to train such a non-autoregressive network for ASR. Especially in Aishell, the proposed method outperformed the Kaldi ASR system and it matches the performance of the state-of-the-art autoregressive transformer with 7x speedup. Pretrained models and code will be made available after publication.

Motivation & Objective

• Motivate reducing decoding latency in transformer-based ASR without sacrificing accuracy.
• Adapt masked language modeling ideas to audio input for non-autoregressive decoding.
• Explore training and decoding strategies (A-CMLM, A-FMLM, easy first, mask-predict) to bridge training/inference gaps.
• Evaluate on Mandarin (AIShell) and Japanese (CSJ) to assess performance and speedups.

Proposed method

• Introduce two non-autoregressive transformer frameworks: Audio-Conditional Masked Language Model (A-CMLM) and Audio-Factorized Masked Language Model (A-FMLM).
• Replace decoder history with masked tokens and use input speech with context to predict masked positions.
• In A-CMLM, predict all masked tokens in parallel given unmasked tokens and speech features using a product of conditionals (independence assumption).
• In A-FMLM, mitigate train-inference mismatch by progressive decoding steps (Z_t sets) and a factorized objective across iterations.
• Propose decoding strategies: easy first and mask-predict, including variant formulations and iteration-based refinement.
• Report practical training/decoding details, including length handling via EOS and iteration counts (N=2 during training).
• Benchmark setups use ESPnet, Transformer baselines, and compare against Kaldi systems where applicable.

Experimental results

Research questions

• RQ1 Can non-autoregressive Transformers be effectively trained for ASR by conditioning masked predictions on audio input?
• RQ2 Do A-CMLM and A-FMLM offer competitive accuracy with substantial decoding speedups compared to autoregressive transformers?
• RQ3 What decoding strategies (easy first, mask-predict) best balance accuracy and speed for Mandarin and Japanese ASR?
• RQ4 How does non-autoregressive ASR scale with utterance length and what mitigations exist for long sequences?

Key findings

• A-FMLM achieves up to 7x real-time speedup on AIShell with performance matching autoregressive baselines.
• A-CMLM provides limited gains beyond a few iterations for AIShell, while A-FMLM’s single-iteration setup already delivers strong results.
• On AIShell, several non-autoregressive variants nearly match the autoregressive transformer in CER with substantial speedups.
• On CSJ, A-FMLM outperforms A-CMLM and autoregressive baselines in several configurations, achieving notable CER reductions and speedups.
• Long utterances reveal higher deletion and substitution errors for non-autoregressive models, suggesting limits and opportunities for insertion-based approaches.
• Non-autoregressive decoding supports multiple strategies (left-to-right, easy first, mask-predict), enabling flexible inference.