[Paper Review] Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks

Dimitri Palaz, Ronan Collobert+1 more|arXiv (Cornell University)|Apr 3, 2013

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a convolutional neural network (CNN) that directly estimates phoneme class conditional probabilities from raw speech signals, bypassing traditional feature extraction. The approach achieves comparable or better phoneme recognition accuracy than conventional MFCC-based systems, demonstrating that CNNs can automatically learn relevant acoustic features from raw audio data.

ABSTRACT

In hybrid hidden Markov model/artificial neural networks (HMM/ANN) automatic speech recognition (ASR) system, the phoneme class conditional probabilities are estimated by first extracting acoustic features from the speech signal based on prior knowledge such as, speech perception or/and speech production knowledge, and, then modeling the acoustic features with an ANN. Recent advances in machine learning techniques, more specifically in the field of image processing and text processing, have shown that such divide and conquer strategy (i.e., separating feature extraction and modeling steps) may not be necessary. Motivated from these studies, in the framework of convolutional neural networks (CNNs), this paper investigates a novel approach, where the input to the ANN is raw speech signal and the output is phoneme class conditional probability estimates. On TIMIT phoneme recognition task, we study different ANN architectures to show the benefit of CNNs and compare the proposed approach against conventional approach where, spectral-based feature MFCC is extracted and modeled by a multilayer perceptron. Our studies show that the proposed approach can yield comparable or better phoneme recognition performance when compared to the conventional approach. It indicates that CNNs can learn features relevant for phoneme classification automatically from the raw speech signal.

Motivation & Objective

• To investigate whether convolutional neural networks can learn relevant acoustic features directly from raw speech signals for phoneme classification.
• To compare the performance of raw speech input with conventional MFCC-based features in a CNN framework.
• To evaluate the impact of discriminative decoding using conditional random fields (CRF) versus generative HMM decoding on phoneme recognition accuracy.
• To assess the contribution of max-pooling layers in improving model generalization and reducing parameters.

Proposed method

• The system uses a deep CNN architecture with multiple convolutional and pooling layers to process raw speech waveforms directly as input.
• The network is trained to predict phoneme class conditional probabilities using cross-entropy loss on the TIMIT phoneme recognition task.
• A CRF-based decoding algorithm is applied to model sequential dependencies between phonemes, improving recognition accuracy over HMM decoding.
• Filters in the first convolutional layer are analyzed to understand their learned frequency response characteristics.
• The model is compared against a baseline MLP using MFCC features and standard HMM decoding.
• Hyperparameters such as filter size, pooling, and network depth are tuned to optimize performance while minimizing overfitting.

Experimental results

Research questions

• RQ1 Can a CNN trained on raw speech signals achieve competitive phoneme recognition performance compared to systems using handcrafted features like MFCCs?
• RQ2 What is the contribution of max-pooling layers in improving recognition accuracy and reducing model complexity?
• RQ3 How does CRF-based decoding compare to HMM-based decoding in terms of phoneme recognition accuracy?
• RQ4 Do the filters learned in the first convolutional layer of the CNN correspond to meaningful frequency bands relevant to phoneme classification?

Key findings

• The CNN model using raw speech input achieved a phoneme recognition accuracy of 67.88% on the TIMIT core test set, outperforming the baseline MLP with raw input (38.91%) and approaching the performance of MFCC-based systems.
• Using MFCC features with the same CNN architecture yielded 70.52% accuracy with HMM decoding, slightly outperforming raw input, though the difference was small.
• The CRF decoding strategy improved recognition accuracy to 69.47% with raw input and 71.80% with MFCC features, indicating that discriminative decoding enhances performance over generative HMM decoding.
• Max-pooling layers improved performance: removing them reduced accuracy from 67.60% to 64.96%, while also increasing the number of parameters.
• The filters in the first convolutional layer learned distinct frequency responses, suggesting they act as band-pass filters tuned to perceptually relevant speech bands.
• The results indicate that deep CNNs can automatically learn effective representations from raw speech, reducing reliance on hand-designed features.