[Paper Review] Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection

Latané Bullock, Hervé Bredin+1 more|arXiv (Cornell University)|Oct 25, 2019

Speech Recognition and Synthesis21 references85 citations

TL;DR

The paper introduces an LSTM-based overlapped speech detector and a resegmentation module that assigns a second speaker in overlap regions, achieving state-of-the-art DER improvements on AMI Headset mix and introducing a two-stage, overlap-aware diarization pipeline.

ABSTRACT

We address the problem of effectively handling overlapping speech in a diarization system. First, we detail a neural Long Short-Term Memory-based architecture for overlap detection. Secondly, detected overlap regions are exploited in conjunction with a frame-level speaker posterior matrix to make two-speaker assignments for overlapped frames in the resegmentation step. The overlap detection module achieves state-of-the-art performance on the AMI, DIHARD, and ETAPE corpora. We apply overlap-aware resegmentation on AMI, resulting in a 20% relative DER reduction over the baseline system. While this approach is by no means an end-all solution to overlap-aware diarization, it reveals promising directions for handling overlap.

Motivation & Objective

• Motivate robust diarization in the presence of overlapping speech.
• Develop a neural overlapped speech detector (OSD) with end-to-end or MFCC-based features.
• Propose a resegmentation method that assigns a secondary speaker in overlap regions using VB-HMM posteriors.
• Evaluate the impact of overlap detection and assignment on diarization performance (DER).
• Provide reproducible baselines and open-source resources to foster further research.

Proposed method

• Formulate overlapped speech detection as a sequence labeling task with binary labels per frame.
• Train LSTM-based detectors (end-to-end waveform or MFCC features) on fixed-length chunks with data augmentation, including artificial overlap for balance.
• Process test audio with overlapping sliding windows and average scores across windows for thresholding overlap labels.
• Use VB-HMM resegmentation with i-vector posteriors as a soft speaker posterior matrix and assign a second speaker only for overlapped frames.
• Combine the primary speaker sequence from VB-HMM with the secondary speaker assignments to form a union diarization hypothesis.

Fig. 1 : The proposed pipeline for speaker diarization. The baseline incorporates end-to-end neural voice activity detection, speaker change detection, and speaker embeddings, with clustering performed via affinity propagation [ 11 ] . It is available in pyannote.audio toolkit [ 12 ] . The grey boxe

Experimental results

Research questions

• RQ1 Can an end-to-end overlapped speech detector outperform handcrafted-feature baselines across diverse corpora (AMI, DIHARD II, ETAPE) for overlap detection?
• RQ2 Does incorporating overlap-aware resegmentation with a second speaker hypothesis reduce diarization error rate (DER) on challenging datasets like AMI Headset mix?
• RQ3 How does oracle vs. predicted overlap detection and assignment influence DER and error components (false alarm, missed detection, speaker confusion)?

Key findings

• End-to-end overlapped speech detection achieves state-of-the-art precision/recall on AMI, DIHARD II, and ETAPE datasets.
• Overlap-aware resegmentation with a second speaker yields a 20% relative DER reduction on AMI Headset mix (from 29.7% to 23.8%).
• VB-HMM resegmentation reduces speaker confusion marginally; the main gains come from overlap-aware assignment.
• With oracle overlap detection and oracle assignment, DER drops further to 11.8%, highlighting room for improvement mainly in secondary speaker assignment.
• High-precision detection reduces misses but can slightly increase false alarms, and secondary speaker assignment can raise confusion unless detection/assignment are improved.
• On AMI Headset mix, the approach sets a new state-of-the-art in DER and demonstrates the potential of overlap-aware diarization.

Fig. 2 : To increase the number of positive training samples for overlapped speech detection, artificial audio chunks are created by summing two random audio chunks.