[Paper Review] Multi-scale Speaker Diarization with Dynamic Scale Weighting

Tae Jin Park, Nithin Rao Koluguri+2 more|arXiv (Cornell University)|Mar 30, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a multi-scale speaker diarization system with a dynamic scale weighting mechanism using 1D CNNs to adaptively assign importance to different temporal scales at each time step. By initializing with multi-scale clustering and employing an LSTM-based decoder that estimates speaker labels frame-by-frame, the method achieves state-of-the-art performance with 3.92% DER on CALLHOME and 1.05% DER on AMI MixHeadset, enabling overlap-aware diarization and flexible speaker count estimation without iterative refinement.

ABSTRACT

Speaker diarization systems are challenged by a trade-off between the temporal resolution and the fidelity of the speaker representation. By obtaining a superior temporal resolution with an enhanced accuracy, a multi-scale approach is a way to cope with such a trade-off. In this paper, we propose a more advanced multi-scale diarization system based on a multi-scale diarization decoder. There are two main contributions in this study that significantly improve the diarization performance. First, we use multi-scale clustering as an initialization to estimate the number of speakers and obtain the average speaker representation vector for each speaker and each scale. Next, we propose the use of 1-D convolutional neural networks that dynamically determine the importance of each scale at each time step. To handle a variable number of speakers and overlapping speech, the proposed system can estimate the number of existing speakers. Our proposed system achieves a state-of-art performance on the CALLHOME and AMI MixHeadset datasets, with 3.92% and 1.05% diarization error rates, respectively.

Motivation & Objective

• To address the trade-off between temporal resolution and speaker representation fidelity in speaker diarization.
• To enable overlap-aware diarization without requiring post-processing or resegmentation.
• To support a flexible number of speakers, including unknown or variable speaker counts during inference.
• To improve performance over prior clustering-based and sequence-model-based diarization systems by dynamically weighting multi-scale representations.

Proposed method

• The system uses a pretrained speaker embedding extractor (TitaNet) to generate frame-level embeddings.
• Multi-scale clustering initializes the system by estimating speaker count and average speaker representations across multiple temporal scales.
• A 1D CNN computes dynamic scale weights at each time step based on input embeddings and cluster centroids.
• Weighted cosine similarities between speaker embeddings and cluster representations are computed using scale weights.
• An LSTM-based sequence model predicts binary speaker labels per frame, enabling overlap-aware diarization via multi-dimensional sigmoid output.
• Speaker labels are inferred by averaging sigmoid outputs across all speaker pairs, allowing flexible handling of unknown speaker counts.

Figure 1: Data-flow of the proposed multi-scale speaker diarization system.

Experimental results

Research questions

• RQ1 Can a dynamic scale weighting mechanism improve diarization performance across multiple temporal resolutions?
• RQ2 How does dynamic scale weighting compare to static or optimized scale weights in clustering-based initialization?
• RQ3 Can an end-to-end sequence model with clustering initialization handle variable numbers of speakers without iterative refinement?
• RQ4 To what extent does the proposed system outperform existing state-of-the-art modular and end-to-end diarization systems on standard benchmarks?
• RQ5 Does the dynamic weighting mechanism reduce dependency on hyperparameter tuning in multi-scale diarization?

Key findings

• The proposed Equal-w-MSDD system achieves 3.92% diarization error rate (DER) on the CALLHOME test set, outperforming prior state-of-the-art methods.
• On the AMI MixHeadset dataset, the system achieves 1.05% DER, setting a new state-of-the-art performance.
• The Opt-w-MSDD variant, which uses optimized scale weights, achieves slightly better results than Equal-w-MSDD, but the latter shows robustness with no hyperparameter tuning.
• The system demonstrates stable performance across unknown domains due to reduced sensitivity to scale weight tuning compared to prior clustering-based multi-scale methods.
• The dynamic scale weighting mechanism enables effective overlap-aware diarization without requiring additional resegmentation or iterative refinement steps.
• The system maintains high accuracy even when the finest scale is reduced below 0.5s, though performance degrades slightly at very short scales.

Figure 2: Cosine similarity values from each speaker and each scale are weighted by the scale weights to form a weighted cosine similarity vector.