[Paper Review] Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input

Xingchen Song, Zhiyong Wu+4 more|arXiv (Cornell University)|Oct 28, 2020

Speech Recognition and Synthesis18 references4 citations

TL;DR

This paper proposes a non-autoregressive Transformer ASR model that enhances decoder input with greedy CTC predictions to improve accuracy while maintaining 50x faster inference than autoregressive baselines. By refining CTC outputs instead of using masked or copied encoder states, the model achieves state-of-the-art NAR performance on Aishell-1 and Aishell-2 with only 0.0–0.3% absolute CER degradation compared to strong autoregressive models.

ABSTRACT

Non-autoregressive (NAR) transformer models have achieved significantly inference speedup but at the cost of inferior accuracy compared to autoregressive (AR) models in automatic speech recognition (ASR). Most of the NAR transformers take a fixed-length sequence filled with MASK tokens or a redundant sequence copied from encoder states as decoder input, they cannot provide efficient target-side information thus leading to accuracy degradation. To address this problem, we propose a CTC-enhanced NAR transformer, which generates target sequence by refining predictions of the CTC module. Experimental results show that our method outperforms all previous NAR counterparts and achieves 50x faster decoding speed than a strong AR baseline with only 0.0 ~ 0.3 absolute CER degradation on Aishell-1 and Aishell-2 datasets.

Motivation & Objective

• To address the accuracy degradation in non-autoregressive (NAR) Transformer ASR models caused by weak or indirect target-side context in decoder inputs.
• To improve NAR decoding performance by providing stronger, more relevant target-side information to the decoder.
• To reduce the gap between NAR and autoregressive (AR) models in ASR accuracy while preserving high inference speed.
• To evaluate the effectiveness of CTC-predicted outputs as a direct, informative input for NAR decoder refinement.

Proposed method

• The model uses greedy CTC predictions as the primary decoder input, replacing fixed-length MASK sequences or copied encoder states.
• The decoder refines the CTC predictions in a single forward pass, leveraging both speech context and partial target-side context.
• The method avoids the masking mechanism used in MP-CTC, instead directly using CTC outputs to guide the NAR decoder.
• The CTC module provides length supervision and initial target sequence estimates, which the NAR decoder then corrects.
• The decoder is conditioned on the CTC predictions and encoded audio features, enabling joint optimization of speech and target sequence modeling.
• The approach is trained with a joint CTC and attention loss, ensuring alignment between CTC outputs and final predictions.

Experimental results

Research questions

• RQ1 Can using CTC predictions as decoder input improve the accuracy of non-autoregressive Transformer ASR models?
• RQ2 Does providing direct target-side context via CTC outputs reduce the performance gap between NAR and AR ASR models?
• RQ3 Can a CTC-enhanced NAR decoder achieve faster inference while maintaining competitive accuracy on large-scale datasets?
• RQ4 How does the proposed method compare to existing NAR approaches like ST-NAR and MP-CTC in terms of speed and accuracy?

Key findings

• The proposed NAR Transformer achieves 5.6% CER on Aishell-1 dev set and 6.3% on test set, outperforming all previous NAR models.
• On Aishell-2, the model achieves 7.1% CER on iOS, 8.0% on Mic, and 8.1% on Android, surpassing all prior NAR methods on this large-scale corpus.
• The model achieves 50x faster decoding speed than a strong autoregressive baseline, with only 0.0–0.3% absolute CER degradation.
• The CTC-enhanced decoder input reduces the difficulty of the NAR task by providing meaningful target-side context, improving generalization and alignment.
• The model achieves state-of-the-art performance among NAR models on both Aishell-1 and Aishell-2, demonstrating effectiveness on industrial-scale LVCSR tasks.
• The method outperforms even strong AR baselines in inference speed while maintaining comparable accuracy, making it highly suitable for real-time applications.