[Paper Review] CNN self-attention voice activity detector

Amit Sofer, Shlomo E. Chazan|arXiv (Cornell University)|Mar 6, 2022

Speech and Audio Processing4 citations

TL;DR

This paper proposes a novel CNN self-attention voice activity detection (VAD) model that combines a convolutional neural network (CNN) for spatial feature extraction with a self-attention (SA) encoder for long-range contextual modeling. By processing the entire input signal at once, the model achieves state-of-the-art (SOTA) performance on real-world benchmarks with a lightweight, efficient architecture that outperforms standalone CNN and SA models.

ABSTRACT

In this work we present a novel single-channel Voice Activity Detector (VAD) approach. We utilize a Convolutional Neural Network (CNN) which exploits the spatial information of the noisy input spectrum to extract frame-wise embedding sequence, followed by a Self Attention (SA) Encoder with a goal of finding contextual information from the embedding sequence. Different from previous works which were employed on each frame (with context frames) separately, our method is capable of processing the entire signal at once, and thus enabling long receptive field. We show that the fusion of CNN and SA architectures outperforms methods based solely on CNN and SA. Extensive experimental-study shows that our model outperforms previous models on real-life benchmarks, and provides State Of The Art (SOTA) results with relatively small and lightweight model.

Motivation & Objective

• To address the limitations of traditional signal-processing and deep learning-based VAD methods in noisy and reverberant environments.
• To overcome the restricted receptive field and sequential processing constraints of prior CNN and RNN-based VAD models.
• To improve performance by jointly leveraging spatial features from CNNs and long-range temporal dependencies via self-attention.
• To develop a computationally efficient, end-to-end VAD system capable of processing full-length signals in a single forward pass.
• To demonstrate SOTA performance on real-world benchmarks with a lightweight architecture.

Proposed method

• The model uses a CNN encoder to extract frame-wise embeddings from the Mel-spectrogram of a noisy single-channel input, preserving the input length and spatial structure.
• The resulting embedding sequence is fed into a self-attention (SA) encoder that computes attention weights across all frames to model long-range contextual dependencies.
• The SA encoder computes query-key-value attention over the entire sequence, allowing each frame to attend to relevant frames across the entire signal.
• A final fully connected (FC) layer produces the final VAD output, estimating the probability of speech presence per frame.
• The model is trained end-to-end using cross-entropy loss on labeled speech/noise frames.
• The architecture is designed to process the entire input signal at once, enabling a large effective receptive field without segmentation.

Experimental results

Research questions

• RQ1 Can a hybrid CNN-self-attention architecture outperform standalone CNN or self-attention models in voice activity detection?
• RQ2 Does processing the entire input signal at once, rather than in short segments, improve VAD performance in noisy and reverberant conditions?
• RQ3 To what extent does the self-attention mechanism enhance long-range contextual modeling in VAD compared to local or sequential modeling?
• RQ4 How does the proposed model compare to state-of-the-art VAD systems on real-world, diverse audio benchmarks?
• RQ5 Can a lightweight, single-stream architecture achieve SOTA performance without relying on complex post-processing or multi-stage designs?

Key findings

• The proposed CNN-self-attention model achieves state-of-the-art (SOTA) performance on real-world benchmarks, including LibriSpeech, TIMIT, and NIST datasets.
• On the LibriSpeech test set, the model achieves an EER of 5.05% and AUC of 99.02% when processing the entire input sequence, outperforming FCNN (11.68%/95.7%) and SA-only models (10.41%/96.62%).
• On the Park test set, the model achieves an AUC of 99.04%, surpassing the next-best model (98.80%) by a margin of 0.24 percentage points.
• The ablation study confirms that the fusion of CNN and SA components yields superior performance compared to either component alone, with the full model reducing EER by over 50% compared to the FCNN baseline on LibriSpeech.
• Attention visualization shows that the model correctly attends to relevant speech frames even at a distance, while minimizing attention to non-stationary noise and transient events.
• The model maintains high efficiency, achieving SOTA results with only 560K parameters, and processes the entire signal in a single forward pass, enabling long-range context without segmentation.