[Paper Review] Intermediate Loss Regularization for CTC-based Speech Recognition

Jaesong Lee, Shinji Watanabe|arXiv (Cornell University)|Feb 5, 2021

Speech Recognition and Synthesis39 references7 citations

TL;DR

This paper proposes intermediate CTC loss, a simple auxiliary training objective that applies CTC loss to an intermediate layer of a CTC-based ASR encoder, improving model generalization with minimal computational overhead. The method achieves state-of-the-art CTC-only performance—9.9% WER on WSJ and 5.2% CER on AISHELL-1—using only greedy decoding and no external language model or beam search.

ABSTRACT

We present a simple and efficient auxiliary loss function for automatic speech recognition (ASR) based on the connectionist temporal classification (CTC) objective. The proposed objective, an intermediate CTC loss, is attached to an intermediate layer in the CTC encoder network. This intermediate CTC loss well regularizes CTC training and improves the performance requiring only small modification of the code and small and no overhead during training and inference, respectively. In addition, we propose to combine this intermediate CTC loss with stochastic depth training, and apply this combination to a recently proposed Conformer network. We evaluate the proposed method on various corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character error rate (CER) 5.2% on the AISHELL-1 corpus respectively, based on CTC greedy search without a language model. Especially, the AISHELL-1 task is comparable to other state-of-the-art ASR systems based on auto-regressive decoder with beam search.

Motivation & Objective

• Improve the performance of end-to-end automatic speech recognition (ASR) models based on Connectionist Temporal Classification (CTC) without relying on autoregressive decoders or external language models.
• Address the limitations of CTC, such as strong conditional independence assumptions and suboptimal performance compared to attention-based models, through a lightweight regularization technique.
• Design a training objective that enhances feature learning in deeper layers of the encoder while maintaining inference efficiency.
• Explore the synergy between intermediate CTC loss and other regularization techniques, such as stochastic depth, to further improve model robustness and accuracy.

Proposed method

• Introduce an auxiliary loss function, intermediate CTC loss, computed on the output of an intermediate encoder layer rather than the final layer.
• Train the model using a weighted combination of the standard CTC loss and the intermediate CTC loss, with a hyperparameter to balance the two objectives.
• Apply the intermediate CTC loss to both Transformer and Conformer architectures, which combine self-attention and convolutional layers for improved modeling capacity.
• Combine intermediate CTC loss with stochastic depth regularization, where random dropping of residual blocks during training enhances generalization.
• Use greedy decoding at inference time, ensuring no additional computational cost compared to standard CTC.
• Evaluate the method on multiple benchmarks, including WSJ, AISHELL-1, and TED-LIUM2, under consistent training and decoding protocols.

Experimental results

Research questions

• RQ1 Can applying CTC loss at an intermediate layer of the encoder improve the performance of CTC-based ASR models without increasing inference cost?
• RQ2 How does intermediate CTC loss compare to other auxiliary training strategies in terms of performance gain and training efficiency?
• RQ3 Does combining intermediate CTC loss with stochastic depth lead to further improvements in ASR accuracy?
• RQ4 Can intermediate CTC loss enhance the performance of non-autoregressive ASR methods like Mask CTC, which rely on CTC for initial predictions?
• RQ5 To what extent can CTC-only models achieve state-of-the-art performance without beam search or language models?

Key findings

• The intermediate CTC loss improves ASR performance across multiple architectures, including Transformer and Conformer, with consistent gains on WSJ, AISHELL-1, and TED-LIUM2.
• On the WSJ corpus, the method achieves a word error rate (WER) of 9.9% using only CTC and greedy decoding, outperforming previous non-autoregressive CTC-based methods and approaching autoregressive SOTA results.
• On the AISHELL-1 corpus, the method achieves a character error rate (CER) of 5.2%, surpassing Transformer-based encoder-decoder models and nearing the state-of-the-art autoregressive result of 5.1%.
• The combination of intermediate CTC loss and stochastic depth yields the best performance, demonstrating complementary regularization effects.
• The intermediate CTC loss also improves the performance of Mask CTC-based non-autoregressive ASR, reducing WER from 13.5% to 11.6% on WSJ, showing broader applicability.
• Ablation studies confirm that the default intermediate layer choice (e.g., middle of the network) performs best, outperforming random or lower-layer variants.