[Paper Review] Speech enhancement aided end-to-end multi-task learning for voice activity detection

Xu Tan, Xiao-Lei Zhang|arXiv (Cornell University)|Oct 23, 2020

Speech and Audio Processing26 references4 citations

TL;DR

This paper proposes an end-to-end multi-task learning framework that jointly optimizes speech enhancement (SE) and voice activity detection (VAD) using a novel loss function, VAD-masked scale-invariant source-to-distortion ratio (mSI-SDR). By leveraging VAD predictions to mask the SE output during training, the method enables joint optimization at the objective level, significantly improving VAD robustness in low SNR and noisy environments—especially in babble and human-interference scenarios—while maintaining real-time capability and outperforming both single-task and prior multi-task baselines.

ABSTRACT

Robust voice activity detection (VAD) is a challenging task in low signal-to-noise (SNR) environments. Recent studies show that speech enhancement is helpful to VAD, but the performance improvement is limited. To address this issue, here we propose a speech enhancement aided end-to-end multi-task model for VAD. The model has two decoders, one for speech enhancement and the other for VAD. The two decoders share the same encoder and speech separation network. Unlike the direct thought that takes two separated objectives for VAD and speech enhancement respectively, here we propose a new joint optimization objective -- VAD-masked scale-invariant source-to-distortion ratio (mSI-SDR). mSI-SDR uses VAD information to mask the output of the speech enhancement decoder in the training process. It makes the VAD and speech enhancement tasks jointly optimized not only at the shared encoder and separation network, but also at the objective level. It also satisfies real-time working requirement theoretically. Experimental results show that the multi-task method significantly outperforms its single-task VAD counterpart. Moreover, mSI-SDR outperforms SI-SDR in the same multi-task setting.

Motivation & Objective

• To improve voice activity detection (VAD) robustness in low signal-to-noise ratio (SNR) and noisy environments, where single-task VAD models struggle.
• To address the limited performance gain in prior SE-augmented VAD methods by enabling deeper joint optimization between speech enhancement and VAD.
• To design a novel joint optimization objective that integrates VAD supervision into the speech enhancement loss function, enhancing mutual learning.
• To ensure the model supports real-time inference by preserving the low-latency structure of Conv-TasNet.
• To validate that the proposed multi-task framework with mSI-SDR outperforms both VAD-only and SE-only baselines in both VAD and SE metrics.

Proposed method

• The model uses a shared encoder and temporal convolutional network (TCN) for both speech enhancement and VAD, with two separate decoders for each task.
• The proposed mSI-SDR loss function masks the speech enhancement output using ground-truth and predicted VAD labels during training, focusing optimization on speech-active segments.
• mSI-SDR is defined as the scale-invariant source-to-distortion ratio computed only on frames where the VAD label is active (1), making the SE loss more relevant to VAD performance.
• The architecture is based on Conv-TasNet, ensuring low-latency inference and real-time operation through causal dilated convolutions and cumulative layer normalization.
• The model is trained end-to-end with a multi-task objective combining the mSI-SDR for SE and a binary cross-entropy loss for VAD.
• The framework enables joint optimization not only at the shared encoder and separation network but also at the loss function level, enhancing task alignment.

Experimental results

Research questions

• RQ1 Can joint training of speech enhancement and voice activity detection improve VAD robustness in low SNR environments?
• RQ2 Does integrating VAD supervision into the speech enhancement loss function lead to better performance than independent or cascaded training?
• RQ3 Can the proposed mSI-SDR loss function outperform standard SI-SDR in a multi-task setting for VAD and SE?
• RQ4 Does the multi-task model maintain real-time inference capability while improving performance?
• RQ5 How does the performance of the proposed method compare to single-task VAD and SE baselines across diverse noisy environments?

Key findings

• The proposed Multi-mSS model achieves a 73.77% relative improvement in AUC and a 59.83% relative reduction in EER over the VAD-only model in babble noise at -5 dB SNR.
• In the most challenging environment (babble noise at -5 dB), Multi-mSS achieves a 30.43% relative AUC improvement and a 16.87% relative EER reduction over the Multi-SS baseline.
• The causal version of Multi-mSS maintains strong performance, with minimal degradation compared to the non-causal version, confirming real-time feasibility.
• The speech enhancement metrics (PESQ, STOI, SI-SDR) of Multi-mSS are comparable to the SE-only model, indicating no significant degradation in SE performance.
• The mSI-SDR loss outperforms SI-SDR in the same multi-task setting, demonstrating the effectiveness of VAD-guided masking in SE optimization.
• The model shows the greatest performance gain in speech-shaped noise environments (e.g., babble, café, pedestrians), where the target speech and noise have similar spectral characteristics.