[Paper Review] A Monaural Speech Enhancement Method for Robust Small-Footprint Keyword Spotting

Yue Gu, Zhihao Du+2 more|arXiv (Cornell University)|Jun 20, 2019

Speech Recognition and Synthesis18 references4 citations

TL;DR

This paper proposes a joint-optimized, small-footprint monaural speech enhancement method for robust keyword spotting (KWS) using a novel convolutional recurrent network (CRN) and end-to-end training. By integrating a speech enhancement front-end with a CNN-based KWS model and jointly training both via back-propagation, the system achieves significantly improved noise robustness—especially with Mel-spectrogram inputs—while reducing model size and computation compared to BiLSTM-based approaches.

ABSTRACT

Robustness against noise is critical for keyword spotting (KWS) in real-world environments. To improve the robustness, a speech enhancement front-end is involved. Instead of treating the speech enhancement as a separated preprocessing before the KWS system, in this study, a pre-trained speech enhancement front-end and a convolutional neural networks (CNNs) based KWS system are concatenated, where a feature transformation block is used to transform the output from the enhancement front-end into the KWS system's input. The whole model is trained jointly, thus the linguistic and other useful information from the KWS system can be back-propagated to the enhancement front-end to improve its performance. To fit the small-footprint device, a novel convolution recurrent network is proposed, which needs fewer parameters and computation and does not degrade performance. Furthermore, by changing the input features from the power spectrogram to Mel-spectrogram, less computation and better performance are obtained. our experimental results demonstrate that the proposed method significantly improves the KWS system with respect to noise robustness.

Motivation & Objective

• Address the challenge of keyword spotting (KWS) robustness in noisy, real-world environments on resource-limited devices.
• Overcome the limitations of multi-condition training, which requires large models unsuitable for small-footprint deployment.
• Develop a lightweight speech enhancement front-end that reduces parameters and computation without sacrificing performance.
• Improve KWS robustness by jointly training the enhancement and KWS models, enabling linguistic information to guide enhancement.
• Investigate the impact of feature domain (power spectrogram vs. Mel-spectrogram) on model efficiency and performance in KWS tasks.

Proposed method

• Propose a novel convolutional recurrent network (CRN) as a lightweight speech enhancement front-end, replacing computationally heavy BiLSTM models.
• Integrate the enhancement model with a CNN-based KWS system via a feature transformation block that converts enhanced spectrograms into MFCCs.
• Train the entire system end-to-end using joint optimization, allowing gradients from the KWS system to update the enhancement front-end.
• Use ideal ratio masking (IRM) as the speech enhancement objective, minimizing mean squared error (MSE) between predicted and ideal masks.
• Apply element-wise multiplication of the noisy spectrogram with predicted masks to generate enhanced spectrograms.
• Evaluate performance using Mel-spectrograms as input to the KWS system, showing improved efficiency and robustness over power spectrograms.

Experimental results

Research questions

• RQ1 Can a lightweight, small-footprint speech enhancement model improve keyword spotting robustness in noisy environments without increasing computational cost?
• RQ2 Does joint training of the enhancement and KWS models lead to better performance than separate training or multi-condition training?
• RQ3 Is Mel-spectrogram a better input representation than power spectrogram for KWS systems with speech enhancement front-ends?
• RQ4 How sensitive is the enhancement model to the number of phonetic symbols in keywords, and does the choice of feature domain affect this sensitivity?
• RQ5 How well does the proposed model generalize to unseen noise conditions not present in the training data?

Key findings

• The joint-training strategy significantly outperforms both multi-condition training and separate training, achieving 93.17% accuracy on the test set under matched noise conditions.
• The MelCRN32 model achieved the lowest false rejection rate (FRR) of 1.19% and false alarm rate (FAR) of 6.20% among all models, outperforming BiLSTM and PowCRN variants.
• The Mel-spectrogram-based models reduced computation by lowering the number of frequency bins while maintaining or improving performance compared to power spectrograms.
• The joint-trained MelCRN32 model achieved 78.12% accuracy under unmatched noise conditions (100 unseen noises), outperforming BiLSTM and other CRN variants.
• The Mel-spectrogram-based models showed reduced sensitivity to the number of phonetic symbols in keywords, with lower AUC reduction compared to power spectrogram-based models.
• The narrow models (MelCRN16 and PowCRN16) achieved comparable performance to full-size models with significantly fewer parameters and lower computation, making them suitable for small-footprint devices.