[Paper Review] Speech Recognition: Keyword Spotting Through Image Recognition

Sanjay Krishna Gouda, Salil Kanetkar+2 more|arXiv (Cornell University)|Mar 10, 2018

Handwritten Text Recognition TechniquesComputer Science13 references19 citations

TL;DR

This paper proposes transforming speech recognition into image classification by converting audio clips into log-spectrograms, enabling the use of powerful convolutional neural networks (CNNs) for keyword spotting. It demonstrates that virtual adversarial training (VAT) significantly improves model robustness and achieves 92% validation accuracy, showcasing a novel, effective regularization technique for audio recognition with strong performance even under noisy conditions.

ABSTRACT

The problem of identifying voice commands has always been a challenge due to the presence of noise and variability in speed, pitch, etc. We will compare the efficacies of several neural network architectures for the speech recognition problem. In particular, we will build a model to determine whether a one second audio clip contains a particular word (out of a set of 10), an unknown word, or silence. The models to be implemented are a CNN recommended by the Tensorflow Speech Recognition tutorial, a low-latency CNN, and an adversarially trained CNN. The result is a demonstration of how to convert a problem in audio recognition to the better-studied domain of image classification, where the powerful techniques of convolutional neural networks are fully developed. Additionally, we demonstrate the applicability of the technique of Virtual Adversarial Training (VAT) to this problem domain, functioning as a powerful regularizer with promising potential future applications.

Motivation & Objective

• To address the challenge of keyword spotting in noisy, variable speech by transforming audio into image-like spectrograms for application of well-established CNN techniques.
• To evaluate and compare multiple CNN architectures—standard, low-latency, and adversarially trained—for speech recognition under resource-constrained and unrestricted conditions.
• To investigate the effectiveness of Virtual Adversarial Training (VAT) as a regularizer in audio recognition, particularly in improving model generalization and robustness to noise.
• To optimize key spectrogram and network hyperparameters (window size, stride, frequency bins) for improved model accuracy and efficiency.

Proposed method

• Audio clips are converted into log-spectrograms using short-time Fourier transforms, transforming one-dimensional audio signals into two-dimensional image-like representations suitable for 2D CNNs.
• Three CNN models are trained: a standard CNN from the TensorFlow Speech Recognition tutorial, a low-latency CNN optimized for memory and computation efficiency, and a VAT-regularized CNN for improved robustness.
• Virtual Adversarial Training (VAT) is applied to generate adversarial perturbations in the input space without requiring labels, enhancing generalization and reducing overfitting.
• Hyperparameter tuning is performed across spectrogram parameters (window size, stride, frequency bins) and network architecture components to maximize accuracy and efficiency.
• Noise is injected into training data at varying signal-to-noise ratios (0 to 0.5) to evaluate model robustness under real-world conditions.
• Model performance is evaluated on a 10-word keyword recognition task, with unknown words and silence as additional classes.

Experimental results

Research questions

• RQ1 Can transforming audio into spectrograms enable effective application of 2D CNNs to speech recognition tasks traditionally handled by 1D networks?
• RQ2 How does Virtual Adversarial Training compare to standard regularization techniques like dropout in improving keyword spotting accuracy under noisy conditions?
• RQ3 What is the optimal configuration of spectrogram parameters (window size, stride, frequency resolution) for maximizing model accuracy in keyword spotting?
• RQ4 How does the low-latency CNN architecture balance computational efficiency and recognition performance for deployment on resource-constrained devices?
• RQ5 To what extent does data augmentation with background noise improve model robustness in real-world audio recognition scenarios?

Key findings

• The use of log-spectrograms as input enables effective transfer of image-based CNN techniques to audio recognition, significantly improving model performance.
• Virtual Adversarial Training (VAT) achieved a maximum validation accuracy of 92% on a 20% random sample of the data, outperforming standard regularization methods like dropout.
• Increasing the number of frequency bins (from 10 to 40) worsened model performance, indicating that higher spectral resolution does not always improve recognition accuracy.
• Both window size and stride in spectrogram generation have an optimal range: too small or too large values degrade performance due to redundancy or information loss, respectively.
• The model demonstrated robustness to background noise, with only a few percentage points drop in accuracy even at low signal-to-noise ratios (down to 6 dB), suggesting practical viability in real-world settings.
• Hyperparameter tuning was found to be critical, as suboptimal choices in network architecture or spectrogram parameters significantly degraded model accuracy.