[Paper Review] Speaker activity driven neural speech extraction

Marc Delcroix, Kateřina Žmolíková+3 more|arXiv (Cornell University)|Jan 14, 2021

Speech and Audio Processing29 references4 citations

TL;DR

This paper proposes ADEnet, a neural speech extraction model that uses speaker activity information as an auxiliary clue to extract target speech from single-channel mixtures without requiring enrollment utterances or video. It achieves competitive performance—up to 25% relative WER reduction in ASR—especially in high-overlap conditions when combined with diarization, demonstrating a practical, versatile alternative to enrollment-based methods.

ABSTRACT

Target speech extraction, which extracts the speech of a target speaker in a mixture given auxiliary speaker clues, has recently received increased interest. Various clues have been investigated such as pre-recorded enrollment utterances, direction information, or video of the target speaker. In this paper, we explore the use of speaker activity information as an auxiliary clue for single-channel neural network-based speech extraction. We propose a speaker activity driven speech extraction neural network (ADEnet) and show that it can achieve performance levels competitive with enrollment-based approaches, without the need for pre-recordings. We further demonstrate the potential of the proposed approach for processing meeting-like recordings, where the speaker activity is obtained from a diarization system. We show that this simple yet practical approach can successfully extract speakers after diarization, which results in improved ASR performance, especially in high overlapping conditions, with a relative word error rate reduction of up to 25%.

Motivation & Objective

• To develop a speaker activity-driven speech extraction method that avoids reliance on pre-recorded enrollment utterances.
• To investigate whether speaker activity information can serve as a practical and effective auxiliary clue for single-channel speech extraction.
• To evaluate the integration of ADEnet with diarization systems for improving ASR performance in meeting-like recordings.
• To demonstrate that speaker activity cues alone can enable high-quality speech extraction, even in overlapping speech regions.

Proposed method

• ADEnet is a single-channel neural network that takes a speech mixture and a time-synchronized speaker activity signal as inputs.
• The speaker activity signal indicates, frame-by-frame, whether a target speaker is active, serving as a spatial-temporal mask for the extraction network.
• The network architecture uses a U-Net-like encoder-decoder structure with skip connections to preserve fine-grained spectral details.
• The model is trained end-to-end using a loss function that maximizes the signal-to-distortion ratio (SDR) between the extracted and reference speech.
• The speaker activity signal is obtained from a diarization system (e.g., TS-VAD) in real-world scenarios, enabling plug-and-play integration.
• The method is robust to imperfect activity signals, with performance maintained even when noise or overlap regions are present in the activity cue.

Experimental results

Research questions

• RQ1 Can speaker activity information alone serve as an effective auxiliary clue for neural speech extraction without enrollment utterances?
• RQ2 How does ADEnet perform compared to enrollment-based methods like SpeakerBeam under oracle and noisy activity conditions?
• RQ3 To what extent can ADEnet improve ASR performance when combined with diarization in meeting-like recordings?
• RQ4 How does ADEnet handle overlapping speech regions where multiple speakers are active simultaneously?

Key findings

• ADEnet with oracle speaker activity achieves a higher signal-to-distortion ratio (SDR) of 10.2 dB compared to SpeakerBeam’s 9.4 dB, without requiring enrollment utterances.
• ADEnet-mix with noisy training outperforms both inactivity-masking and baseline SpeakerBeam, especially in noisy or imperfect activity signal conditions.
• When integrated with diarization (TS-VAD), ADEnet reduces cpWER by up to 25% in high-overlap conditions (OV40), achieving 29.8% WER vs. 33.6% without ADEnet.
• The method maintains stable performance across varying quality of speaker activity signals, indicating robustness to diarization errors and overlap regions.
• ADEnet enables effective single-channel speech extraction even in overlapping regions, as shown in qualitative examples of extracted speech.