[Paper Review] Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers

Huahuan Zheng, Wenjie Peng+2 more|arXiv (Cornell University)|Jul 7, 2021

Speech Recognition and Synthesis44 references4 citations

TL;DR

This paper advances CTC-CRF-based end-to-end speech recognition by integrating wordpiece modeling units and Conformer neural networks, achieving state-of-the-art performance on Switchboard, Librispeech, and CommonVoice German datasets. Results show that Conformers significantly improve accuracy over BLSTM and VGG-BLSTM, while wordpiece-based systems match phone-based performance on high grapheme-phoneme correspondence languages like German but underperform slightly on English.

ABSTRACT

Automatic speech recognition systems have been largely improved in the past few decades and current systems are mainly hybrid-based and end-to-end-based. The recently proposed CTC-CRF framework inherits the data-efficiency of the hybrid approach and the simplicity of the end-to-end approach. In this paper, we further advance CTC-CRF based ASR technique with explorations on modeling units and neural architectures. Specifically, we investigate techniques to enable the recently developed wordpiece modeling units and Conformer neural networks to be succesfully applied in CTC-CRFs. Experiments are conducted on two English datasets (Switchboard, Librispeech) and a German dataset from CommonVoice. Experimental results suggest that (i) Conformer can improve the recognition performance significantly; (ii) Wordpiece-based systems perform slightly worse compared with phone-based systems for the target language with a low degree of grapheme-phoneme correspondence (e.g. English), while the two systems can perform equally strong when such degree of correspondence is high for the target language (e.g. German).

Motivation & Objective

• To investigate the integration of wordpiece modeling units into CTC-CRF-based automatic speech recognition systems.
• To evaluate the effectiveness of Conformer neural networks in CTC-CRF frameworks for improved ASR performance.
• To analyze the impact of model size, data augmentation, and input features on CTC-CRF performance with Conformers.
• To compare wordpiece-based and phone-based systems across languages with varying grapheme-phoneme correspondence.
• To demonstrate that CTC-CRF can achieve state-of-the-art results with modern architectures and units without requiring a pronunciation lexicon.

Proposed method

• Adopted Conformer neural networks as the acoustic model in the CTC-CRF framework, replacing older BLSTM and VGG-BLSTM architectures.
• Used wordpiece tokenization (e.g., BPE) as the modeling unit, replacing traditional phone-based units.
• Applied ratio-based SpecAug for data augmentation, improving robustness and generalization.
• Employed 3-channel 120-dimensional Fbank features with delta and delta-delta for input, enhancing feature representation.
• Trained models using a learning rate scheduler and optimized hyperparameters for Conformer training in CTC-CRF settings.
• Conducted ablation studies on model size, input features, and data augmentation to isolate performance gains.

Experimental results

Research questions

• RQ1 Can Conformer neural networks improve recognition accuracy in CTC-CRF-based ASR compared to BLSTM and VGG-BLSTM?
• RQ2 How does wordpiece-based modeling compare to phone-based modeling in CTC-CRF systems for languages with low versus high grapheme-phoneme correspondence?
• RQ3 What is the impact of input feature representation and data augmentation on CTC-CRF performance with Conformers?
• RQ4 Does increasing Conformer model size lead to consistent performance gains in CTC-CRF ASR?
• RQ5 Can CTC-CRF frameworks effectively leverage modern neural architectures like Conformers without requiring a pronunciation lexicon?

Key findings

• Conformer-based CTC-CRF models achieved a 5.3% relative WER reduction compared to BLSTM, even with fewer parameters.
• The Conformer-M+ model (51M parameters) achieved a 12.1% WER on Switchboard Eval2000, outperforming both BLSTM and VGG-BLSTM.
• Wordpiece-based systems performed slightly worse than phone-based systems on English datasets (Switchboard and Librispeech), with a 0.4–0.5% absolute WER gap.
• On the German CommonVoice dataset, wordpiece-based and phone-based systems achieved comparable performance, indicating that high grapheme-phoneme correspondence enables wordpiece success.
• SpecAug with ratio-based augmentation reduced WER by 7.1% on Switchboard Switchboard and 9.3% on CH, outperforming frame-wise fixed SpecAug.
• Using 80-dimensional Fbank features instead of 40-dimensional ones reduced WER by 0.3–0.4% across test sets, showing benefit from higher-dimensional features.