[Paper Review] Multi-stream Convolutional Neural Network with Frequency Selection for Robust Speaker Verification

Wei Yao, Chen Shen+2 more|arXiv (Cornell University)|Dec 21, 2020

Speech Recognition and Synthesis52 references4 citations

TL;DR

This paper proposes a multi-stream convolutional neural network (CNN) with frequency selection for robust speaker verification, where each stream processes distinct sub-bands of the frequency spectrum to enhance temporal embedding diversity. The method achieves a 20.53% relative improvement in minimum Decision Cost Function (minDCF) over a single-stream baseline on the VoxCeleb dataset, demonstrating that partial-frequency learning improves robustness and performance.

ABSTRACT

Speaker verification aims to verify whether an input speech corresponds to the claimed speaker, and conventionally, this kind of system is deployed based on single-stream scenario, wherein the feature extractor operates in full frequency range. In this paper, we hypothesize that machine can learn enough knowledge to do classification task when listening to partial frequency range instead of full frequency range, which is so called frequency selection technique, and further propose a novel framework of multi-stream Convolutional Neural Network (CNN) with this technique for speaker verification tasks. The proposed framework accommodates diverse temporal embeddings generated from multiple streams to enhance the robustness of acoustic modeling. For the diversity of temporal embeddings, we consider feature augmentation with frequency selection, which is to manually segment the full-band of frequency into several sub-bands, and the feature extractor of each stream can select which sub-bands to use as target frequency domain. Different from conventional single-stream solution wherein each utterance would only be processed for one time, in this framework, there are multiple streams processing it in parallel. The input utterance for each stream is pre-processed by a frequency selector within specified frequency range, and post-processed by mean normalization. The normalized temporal embeddings of each stream will flow into a pooling layer to generate fused embeddings. We conduct extensive experiments on VoxCeleb dataset, and the experimental results demonstrate that multi-stream CNN significantly outperforms single-stream baseline with 20.53 % of relative improvement in minimum Decision Cost Function (minDCF).

Motivation & Objective

• To investigate whether neural networks can achieve comparable or better performance using partial frequency sub-bands instead of full-band features in speaker verification.
• To enhance acoustic modeling robustness by introducing diversity in temporal embeddings through multi-stream processing of selected frequency sub-bands.
• To develop a practical two-stage training strategy that reduces computational cost while maintaining high performance.
• To validate the effectiveness of frequency selection as a data-augmentation and model-regularization technique in deep speaker embedding learning.

Proposed method

• The framework uses multiple parallel CNN streams, each processing a different frequency sub-band selected via a frequency selector, replacing full-band input.
• Each stream applies mean normalization to its normalized temporal embeddings before pooling to generate fused embeddings.
• The frequency selector manually segments the full-band spectrum into sub-bands (e.g., low, high), and each stream is trained to use only its assigned sub-band.
• A two-stage training process is employed: first, each stream is trained independently; second, optimal fusion weights are searched to combine the streams.
• The model is trained and evaluated using the VoxCeleb dataset with PyTorch, using standard metrics like minDCF and EER.
• The method is designed to improve robustness under noisy, unconstrained conditions by leveraging frequency-domain diversity.

Experimental results

Research questions

• RQ1 Can a neural network achieve competitive performance when trained on partial frequency sub-bands instead of the full frequency band?
• RQ2 Does multi-stream processing with frequency-selected sub-bands improve the diversity and robustness of temporal embeddings in speaker verification?
• RQ3 How does frequency selection compare to conventional single-stream full-band training in terms of minDCF and EER performance?
• RQ4 Can a two-stage training strategy effectively combine multiple frequency-stream models without significant degradation?

Key findings

• The proposed multi-stream CNN with frequency selection achieved a 20.53% relative improvement in minimum Decision Cost Function (minDCF) over the single-stream baseline.
• The method reduced EER by 16.5% relative to the baseline, with the best model achieving 1.781% EER at 500 epochs.
• The ensemble of three full-band models (MS-1) showed slight degradation compared to the single-stream baseline, indicating that naive ensembling is less effective than frequency-aware multi-stream learning.
• The model trained with partial-band features (SS-2: low-frequency, SS-3: high-frequency) showed better generalization than full-band models with different initialization, suggesting sub-band learning is beneficial.
• The relative improvement of the proposed method was consistently above 16% across all training epochs, demonstrating stable and significant gains.
• The two-stage training strategy effectively combined streams with optimal fusion weights, avoiding the computational burden of end-to-end joint training.