[Paper Review] Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition

Somshubra Majumdar, Jagadeesh Balam+4 more|arXiv (Cornell University)|Apr 5, 2021

Speech Recognition and Synthesis29 references43 citations

TL;DR

Citrinet is a deep 1D time-channel separable convolutional CTC ASR model enhanced with squeeze-and-excitation that narrows the performance gap between non-autoregressive CTC models and autoregressive Seq2Seq/RNN-T models without external language models.

ABSTRACT

We propose Citrinet - a new end-to-end convolutional Connectionist Temporal Classification (CTC) based automatic speech recognition (ASR) model. Citrinet is deep residual neural model which uses 1D time-channel separable convolutions combined with sub-word encoding and squeeze-and-excitation. The resulting architecture significantly reduces the gap between non-autoregressive and sequence-to-sequence and transducer models. We evaluate Citrinet on LibriSpeech, TED-LIUM2, AISHELL-1 and Multilingual LibriSpeech (MLS) English speech datasets. Citrinet accuracy on these datasets is close to the best autoregressive Transducer models.

Motivation & Objective

• Motivate bridging the performance gap between non-autoregressive CTC models and autoregressive Seq2Seq/RNN-T models in end-to-end ASR.
• Propose Citrinet, a deep convolutional CTC model augmented with 1D time-channel separable convolutions and squeeze-and-excitation modules.
• Evaluate Citrinet across multiple standard ASR datasets and compare with leading autoregressive models and room-temperature baselines.

Proposed method

• Introduce Citrinet as a deep residual CTC model with prolog/epilog blocks and three mega-blocks of residual QuartzNet-style blocks.
• Employ 1D time-channel separable convolutions with kernel layouts (K4) and a squeeze-and-excitation context module to capture global information.
• Use a tokenization strategy with sub-word units (word-piece) and optional external language model rescoring for evaluation.
• Train with SpecAugment, NovoGrad optimizer, cosine LR schedule, and standard acoustic front-end (80-dim log-Mel features).
• Evaluate variants by varying width (C), depth (R), kernel layouts, and tokenization sizes to study impact on WER/CER.

Experimental results

Research questions

• RQ1 How close can a non-autoregressive CTC model get to autoregressive Seq2Seq/RNN-T models on standard ASR benchmarks without external LMs?
• RQ2 What architectural choices in Citrinet (kernel layout, SE context, width, depth, tokenization) most influence ASR accuracy?
• RQ3 Can Citrinet achieve state-of-the-art-like performance across LibriSpeech, MLS, TED-LIUM 2, and AISHELL-1?
• RQ4 What is the impact of context window size in the SE module on recognition accuracy?
• RQ5 How does pre-training on English MLS transfer to other datasets (TED-LIUM 2, AISHELL-1) and affect fine-tuning results?

Key findings

• Citrinet-1024 achieves WER 6.22% on LibriSpeech test-other without external LM and narrow gaps to state-of-the-art autoregressive models with LM rescoring.
• Citrinet variants (e.g., Citrinet-1024) approach the performance of Transducer models on several datasets without LMs (e.g., MLS English and TED-LIUM 2 results).
• SE context significantly improves accuracy, with larger SE context windows correlating with lower WER/CER across tested setups.
• Kernel width ablation shows an optimal range; too narrow or too wide kernels degrade performance, with 0.75–1.0 scaling producing best results in their setup.
• Increasing model width (C) and depth (R) generally improves performance at the cost of more parameters, with diminishing returns beyond certain configurations.
• Tokenizer lexicon size affects accuracy, where extremely large lexicons harm performance, and very small lexicons can reduce trainability due to CT loss constraints.
• The model benefits from cross-dataset pre-training (MLS English) and fine-tuning on domain-specific data (TED-LIUM 2, AISHELL-1), achieving competitive or superior CER/WER to comparable baselines.