[Paper Review] Royalflush Speaker Diarization System for ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challenge

Jingguang Tian, Xinhui Hu+1 more|arXiv (Cornell University)|Feb 10, 2022

Speech and Audio Processing4 citations

TL;DR

This paper presents the Royalflush speaker diarization system for the ICASSP 2022 M2MeT challenge, introducing a multi-channel U-Net-based overlapped speech detection model, a speech separation and verification-based overlapped speech handling method, and three speaker embedding architectures. The system achieves a 6.30% diarization error rate (DER) on the far-field Alimeeting evaluation set, significantly reducing DER from 15.25% in the baseline to 6.40% in the best individual system and further improving via ensemble fusion.

ABSTRACT

This paper describes the Royalflush speaker diarization system submitted to the Multi-channel Multi-party Meeting Transcription Challenge(M2MeT). Our system comprises speech enhancement, overlapped speech detection, speaker embedding extraction, speaker clustering, speech separation and system fusion. In this system, we made three contributions. First, we propose an architecture of combining the multi-channel and U-Net-based models, aiming at utilizing the benefits of these two individual architectures, for far-field overlapped speech detection. Second, in order to use overlapped speech detection model to help speaker diarization, a speech separation based overlapped speech handling approach, in which the speaker verification technique is further applied, is proposed. Third, we explore three speaker embedding methods, and obtained the state-of-the-art performance on the CNCeleb-E test set. With these proposals, our best individual system significantly reduces DER from 15.25% to 6.40%, and the fusion of four systems finally achieves a DER of 6.30% on the far-field Alimeeting evaluation set.

Motivation & Objective

• To improve speaker diarization performance in challenging far-field multi-speaker meeting scenarios with overlapping speech, noise, and reverberation.
• To develop a robust overlapped speech detection (OSD) model leveraging multi-channel audio and U-Net architecture for improved detection in adverse acoustic conditions.
• To enhance diarization accuracy by integrating speech separation with speaker verification for labeling overlapping speech segments.
• To evaluate and compare multiple state-of-the-art speaker embedding models for optimal performance on the CNCeleb-E test set.
• To achieve top performance in the M2MeT challenge through system fusion of multiple sub-systems with diverse architectures and handling strategies.

Proposed method

• A multi-channel U-Net-based architecture is proposed for far-field overlapped speech detection, combining spatial diversity from 8-channel microphone arrays with U-Net’s U-shaped encoder-decoder structure for precise frame-level segmentation.
• Speech enhancement is applied via multi-channel WPE followed by DAS beamforming to produce a single-channel clean signal for downstream processing, excluding the OSD module which uses raw multi-channel input.
• Overlapped speech segments are handled via two methods: a heuristic approach assigning the two closest speaker labels, and a speech separation-based method using Conv-Tasnet to separate mixed signals, followed by speaker verification using cosine similarity with speaker embedding centroids.
• Three different speaker embedding models are trained and evaluated on the CNCeleb dataset, with data augmentation using MUSAN and RIRs, and tempo perturbations via sox.
• System fusion is performed using the DOVER-Lap toolkit, combining outputs from four distinct sub-systems differing in speaker embedding model and overlapped speech handling strategy.
• The final system is trained and evaluated on the far-field Alimeeting dataset, with speech separation data simulated by mixing two random utterances from different speakers at SNRs between -5 dB and 5 dB.

Experimental results

Research questions

• RQ1 Can a multi-channel U-Net-based architecture outperform conventional models in detecting overlapped speech in far-field multi-speaker meetings?
• RQ2 Does integrating speech separation with speaker verification improve diarization accuracy in overlapping speech regions compared to heuristic labeling?
• RQ3 Which speaker embedding model architecture achieves the best performance on the CNCeleb-E evaluation set?
• RQ4 To what extent can system fusion of multiple sub-systems with diverse components reduce the overall diarization error rate (DER)?
• RQ5 How does the proposed system compare to baseline systems in terms of DER on the far-field Alimeeting evaluation set?

Key findings

• The best individual system reduced the diarization error rate (DER) from 15.25% (baseline) to 6.40% on the far-field Alimeeting evaluation set.
• The fusion of four sub-systems using DOVER-Lap achieved a final DER of 6.30% on the same evaluation set, demonstrating the effectiveness of ensemble learning.
• The speech separation-based overlapped speech handling method reduced DER by nearly 1% in absolute terms compared to the heuristic method.
• The proposed multi-channel U-Net-based OSD model improved detection performance in far-field conditions by leveraging spatial and spectral features from multi-channel inputs.
• Among the three speaker embedding models evaluated, the most robust architecture achieved state-of-the-art performance on the CNCeleb-E test set.
• Speech enhancement via WPE and DAS beamforming significantly improved speaker clustering and speech separation performance, but was detrimental to OSD, so it was excluded from the OSD pipeline.