[Paper Review] Neural Speaker Diarization with Speaker-Wise Chain Rule

Yusuke Fujita, Shinji Watanabe+4 more|arXiv (Cornell University)|Jun 2, 2020

Speech Recognition and Synthesis36 references41 citations

TL;DR

Introduces a speaker-wise conditional inference method (SC-EEND) that decodes speaker activities sequentially using the probabilistic chain rule, enabling variable numbers of speakers and improving DER over prior EEND methods.

ABSTRACT

Speaker diarization is an essential step for processing multi-speaker audio. Although an end-to-end neural diarization (EEND) method achieved state-of-the-art performance, it is limited to a fixed number of speakers. In this paper, we solve this fixed number of speaker issue by a novel speaker-wise conditional inference method based on the probabilistic chain rule. In the proposed method, each speaker's speech activity is regarded as a single random variable, and is estimated sequentially conditioned on previously estimated other speakers' speech activities. Similar to other sequence-to-sequence models, the proposed method produces a variable number of speakers with a stop sequence condition. We evaluated the proposed method on multi-speaker audio recordings of a variable number of speakers. Experimental results show that the proposed method can correctly produce diarization results with a variable number of speakers and outperforms the state-of-the-art end-to-end speaker diarization methods in terms of diarization error rate.

Motivation & Objective

• Address fixed-number limitation in end-to-end neural diarization (EEND).
• Propose a speaker-wise conditional inference framework based on the probabilistic chain rule.
• Enable diarization for a variable number of speakers with a stop condition.
• Incorporate teacher forcing to improve training stability and performance.
• Evaluate on CALLHOME and simulated variable-speaker datasets to compare with EEND and x-vector+AHC methods.

Proposed method

• Model speaker diarization as a joint distribution over per-speaker speech activity random variables and decode sequentially using previously estimated speakers (P(y1,...,yS|X)) via the chain rule.
• Use a speaker-wise conditional neural network (SCNN) that accepts X and the previous speaker’s speech activity to produce z_s,t probabilities.
• Adopt an encoder-decoder architecture with a Transformer encoder and an LSTM-based decoder to generate variable-length speaker outputs.
• Train with permutation-invariant training (PIT) to handle speaker ordering; explore two loss strategies (greedy and two-stage PIT with TF).
• During training, apply teacher forcing by feeding ground-truth previous speaker activity, with two-stage PIT loss to determine optimal speaker order.
• Iterate decoding until a zero-vector is produced to signal no more speakers.

Experimental results

Research questions

• RQ1 Can a speaker-wise chain-rule-based model handle a variable number of speakers while producing accurate diarization?
• RQ2 Does conditioning each speaker on previously estimated speakers improve performance over traditional EEND?
• RQ3 What is the impact of teacher forcing and two-stage PIT on diarization error rate (DER) for variable-speaker scenarios?
• RQ4 How does SC-EEND compare to x-vector+AHC and standard EEND across fixed two-speaker and variable-speaker datasets?

Key findings

• SC-EEND with PIT+TF achieves DER improvements over conventional EEND on two-speaker CALLHOME (8.86% vs 9.70%).
• SC-EEND with PIT+TF achieves strong DER on variable-speaker simulated data, outperforming EEND and Greedy+TF, especially as the number of speakers grows.
• On variable-speaker CALLHOME, SC-EEND with PIT+TF achieves DER 15.75%, better than x-vector+AHC and EEND baselines.
• SC-EEND without TF shows limited gains; TF significantly boosts performance in variable-speaker settings.
• Two-stage PIT loss generally yields slightly better DER than Greedy+TF or single-stage PIT in several experiments.
• SC-EEND demonstrates better speaker counting accuracy than x-vector+AHC, though handling more than four speakers remains challenging.