[Paper Review] Convolutional Speech Recognition with Pitch and Voice Quality Features

Guillermo Cámbara, Jordi Luque+1 more|arXiv (Cornell University)|Sep 2, 2020

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper proposes integrating pitch, jitter, and shimmer features with mel-frequency spectral coefficients (MFSCs) in a convolutional neural network (Conv GLU) for end-to-end automatic speech recognition. By appending these prosodic and voice quality features to MFSC inputs, the model achieves up to 7% relative WER reduction on Spanish Common Voice and 2.94% on LibriSpeech, demonstrating consistent gains in robustness, especially in diverse or non-clean conditions.

ABSTRACT

The effects of adding pitch and voice quality features such as jitter and shimmer to a state-of-the-art CNN model for Automatic Speech Recognition are studied in this work. Pitch features have been previously used for improving classical HMM and DNN baselines, while jitter and shimmer parameters have proven to be useful for tasks like speaker or emotion recognition. Up to our knowledge, this is the first work combining such pitch and voice quality features with modern convolutional architectures, showing improvements up to 7% and 3% relative WER points, for the publicly available Spanish Common Voice and LibriSpeech 100h datasets, respectively. Particularly, our work combines these features with mel-frequency spectral coefficients (MFSCs) to train a convolutional architecture with Gated Linear Units (Conv GLUs). Such models have shown to yield small word error rates, while being very suitable for parallel processing for online streaming recognition use cases. We have added pitch and voice quality functionality to Facebook's wav2letter speech recognition framework, and we provide with such code and recipes to the community, to carry on with further experiments. Besides, to the best of our knowledge, our Spanish Common Voice recipe is the first public Spanish recipe for wav2letter.

Motivation & Objective

• To investigate whether pitch and voice quality features (jitter, shimmer) improve performance in modern convolutional ASR models.
• To bridge the gap between hand-crafted prosodic features and end-to-end deep learning architectures.
• To provide open-source recipes and code integration for pitch and voice quality features in the wav2letter framework.
• To establish the first public Spanish Common Voice recipe for wav2letter.
• To assess the impact of these features on robustness across diverse speech conditions, such as varying accents and voice pathologies.

Proposed method

• The model uses a Conv GLU architecture with gated linear units to enable deep, parallelizable training and low WER.
• Pitch, jitter, and shimmer features are extracted from raw audio and concatenated with MFSCs at the input layer, without architectural modifications.
• The framework extends Facebook’s wav2letter with integrated pitch and voice quality feature extraction pipelines.
• Experiments are conducted on two public datasets: Spanish Common Voice and LibriSpeech 100h, using standard training and decoding protocols.
• The approach maintains simplicity by avoiding post-processing or specialized filters, focusing on feature concatenation at input level.
• Performance is evaluated using word error rate (WER) across dev-clean, dev-other, test-clean, and test-other splits.

Experimental results

Research questions

• RQ1 Can pitch and voice quality features (jitter, shimmer) improve the performance of modern convolutional ASR models like Conv GLU?
• RQ2 How do these features affect ASR robustness in diverse conditions such as non-clean or accented speech?
• RQ3 What is the relative contribution of pitch, jitter, and shimmer when combined with MFSCs in end-to-end ASR?
• RQ4 Can simple concatenation of prosodic features at the input layer yield significant WER improvements without architectural changes?
• RQ5 How do these features impact models trained on low-resource languages like Spanish, compared to high-resource ones like English?

Key findings

• The addition of pitch, jitter, and shimmer features reduced WER by up to 7% relative on the Spanish Common Voice dataset, particularly benefiting the dev-other and test-other sets.
• On LibriSpeech 100h, the model achieved a 2.94% relative WER improvement on the dev-clean set and 2.87% on test-other when using MFSC+pitch+jitter+shimmer.
• The combination of all three prosodic features (pitch, jitter, shimmer) yielded the most consistent and robust performance across all test sets.
• MFSC+pitch+jitter+shimmer outperformed both MFSC-only and MFSC+pitch baselines, indicating additive value from voice quality features.
• The gains were more pronounced in non-clean and diverse prosody conditions (e.g., dev-other), suggesting enhanced robustness to accent and variability.
• The study confirms that pitch and voice quality features are effective in modern convolutional ASR, even with minimal architectural adaptation.