[Paper Review] End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification

Yusuke Fujita, Shinji Watanabe+3 more|arXiv (Cornell University)|Feb 24, 2020

Speech Recognition and Synthesis52 references43 citations

TL;DR

The paper reframes speaker diarization as end-to-end multi-label frame-wise classification with permutation-free training, and shows self-attention-based EEND outperforms clustering-based methods while handling overlaps.

ABSTRACT

The most common approach to speaker diarization is clustering of speaker embeddings. However, the clustering-based approach has a number of problems; i.e., (i) it is not optimized to minimize diarization errors directly, (ii) it cannot handle speaker overlaps correctly, and (iii) it has trouble adapting their speaker embedding models to real audio recordings with speaker overlaps. To solve these problems, we propose the End-to-End Neural Diarization (EEND), in which a neural network directly outputs speaker diarization results given a multi-speaker recording. To realize such an end-to-end model, we formulate the speaker diarization problem as a multi-label classification problem and introduce a permutation-free objective function to directly minimize diarization errors. Besides its end-to-end simplicity, the EEND method can explicitly handle speaker overlaps during training and inference. Just by feeding multi-speaker recordings with corresponding speaker segment labels, our model can be easily adapted to real conversations. We evaluated our method on simulated speech mixtures and real conversation datasets. The results showed that the EEND method outperformed the state-of-the-art x-vector clustering-based method, while it correctly handled speaker overlaps. We explored the neural network architecture for the EEND method, and found that the self-attention-based neural network was the key to achieving excellent performance. In contrast to conditioning the network only on its previous and next hidden states, as is done using bidirectional long short-term memory (BLSTM), self-attention is directly conditioned on all the frames. By visualizing the attention weights, we show that self-attention captures global speaker characteristics in addition to local speech activity dynamics, making it especially suitable for dealing with the speaker diarization problem.

Motivation & Objective

• Motivate and address limitations of clustering-based diarization methods.
• Formulate speaker diarization as an end-to-end multi-label classification problem.
• Introduce permutation-free training to resolve speaker-label permutations.
• Explore BLSTM and self-attention architectures for E2E diarization.
• Demonstrate effectiveness on simulated mixtures and real conversational data.

Proposed method

• Formulate Y as frame-wise multi-label outputs for C speakers.
• Introduce permutation-free loss to minimize diarization errors across speaker permutations.
• Compare BLSTM-based EEND with Deep Clustering objectives to self-attention-based EEND.
• Use two architectures: BLSTM-EEND with DC loss and SA-EEND with encoder blocks and multi-head self-attention.
• Train on simulated mixtures (SimBeta2, SimLarge) and real data (Real, Comb) and adapt to domain (CALLHOME, CSJ).
• Evaluate with diarization error rate (DER) including overlaps and with collar tolerance.

Experimental results

Research questions

• RQ1 Can end-to-end diarization beat traditional clustering-based methods on simulated and real data?
• RQ2 Does self-attention provide advantages over BLSTM for end-to-end diarization, especially with overlaps?
• RQ3 How effective is permutation-free training in addressing speaker-label permutations across time frames?
• RQ4 How does EEND perform under different overlap conditions and with domain adaptation to real conversations?

Key findings

• Self-attention EEND (SA-EEND) substantially reduces DER over clustering baselines on simulated mixtures, especially under high overlap.
• SA-EEND with SimLarge training achieves DERs of 6.81–6.60% on simulated tests and 14.03% (CALLHOME CH) / 21.84% (CSJ) on real/test sets.
• BLSTM-EEND improves over clustering baselines on simulated data but under real data shows weaker performance than SA-EEND.
• Domain adaptation (CALLHOME) further reduces DER for SA-EEND (e.g., 10.76% w adapt) compared to non-adapted models.
• Multi-condition training (SimLarge, Comb) improves robustness across varying overlap scenarios.
• SA-EEND outperforms x-vector and i-vector clustering baselines in DER across most test sets when trained with appropriate data.