[Paper Review] Encoder-Decoder Based Attractor Calculation for End-to-End Neural Diarization.

Shota Horiguchi, Yusuke Fujita+3 more|arXiv (Cornell University)|Jun 20, 2021

Speech Recognition and Synthesis54 references4 citations

TL;DR

This paper proposes an encoder-decoder-based attractor calculation (EDA) module for end-to-end neural diarization (EEND), enabling flexible speaker count estimation by iteratively generating speaker-wise attractors via an LSTM sequence-to-sequence model. The method improves overlap handling and achieves superior performance over pipeline approaches on simulated and real datasets.

ABSTRACT

This paper investigates an end-to-end neural diarization (EEND) method for an unknown number of speakers. In contrast to the conventional pipeline approach to speaker diarization, EEND methods are better in terms of speaker overlap handling. However, EEND still has a disadvantage in that it cannot deal with a flexible number of speakers. To remedy this problem, we introduce encoder-decoder-based attractor calculation module (EDA) to EEND. Once frame-wise embeddings are obtained, EDA sequentially generates speaker-wise attractors on the basis of a sequence-to-sequence method using an LSTM encoder-decoder. The attractor generation continues until a stopping condition is satisfied; thus, the number of attractors can be flexible. Diarization results are then estimated as dot products of the attractors and embeddings. The embeddings from speaker overlaps result in larger dot product values with multiple attractors; thus, this method can deal with speaker overlaps. Because the maximum number of output speakers is still limited by the training set, we also propose an iterative inference method to remove this restriction. Further, we propose a method that aligns the estimated diarization results with the results of an external speech activity detector, which enables fair comparison against pipeline approaches. Extensive evaluations on simulated and real datasets show that EEND-EDA outperforms the conventional pipeline approach.

Motivation & Objective

• Address the limitation of end-to-end neural diarization (EEND) in handling an unknown and variable number of speakers.
• Overcome the rigidity of fixed speaker count in conventional EEND methods by enabling dynamic attractor generation.
• Improve speaker overlap detection by leveraging multiple attractors that amplify dot products with overlapping embeddings.
• Enable fair comparison with pipeline methods by aligning diarization outputs with external speech activity detection results.
• Extend the upper limit of detectable speakers through an iterative inference method to overcome training set constraints.

Proposed method

• Introduce an encoder-decoder-based attractor calculation (EDA) module that generates speaker-wise attractors sequentially using an LSTM encoder-decoder architecture.
• Use frame-wise embeddings as input and generate attractors iteratively until a stopping condition is met, enabling dynamic speaker count estimation.
• Estimate diarization results via dot products between embeddings and generated attractors, where overlapping speaker embeddings yield higher scores with multiple attractors.
• Implement an iterative inference strategy to extend the maximum detectable speaker count beyond the limit imposed by the training data.
• Align predicted diarization outputs with external speech activity detection results to ensure fair evaluation against pipeline-based systems.
• Train the EEND-EDA model end-to-end with a sequence-to-sequence loss to optimize attractor generation and diarization accuracy.

Experimental results

Research questions

• RQ1 Can an end-to-end diarization system dynamically estimate an unknown number of speakers without prior knowledge of speaker count?
• RQ2 How effectively can an encoder-decoder-based attractor generation mechanism handle speaker overlaps compared to conventional EEND methods?
• RQ3 To what extent does iterative inference extend the practical speaker count limit in EEND systems?
• RQ4 Can the proposed method achieve performance parity or superiority over pipeline approaches when evaluated under fair conditions?
• RQ5 How does attractor-based dot product scoring enhance diarization accuracy in overlapping speech scenarios?

Key findings

• The EEND-EDA model outperforms conventional pipeline approaches on both simulated and real-world diarization datasets.
• The attractor generation mechanism successfully handles speaker overlaps by producing higher dot product scores when multiple attractors align with overlapping embeddings.
• The iterative inference method effectively extends the maximum detectable speaker count beyond the limit observed during training.
• The alignment with external speech activity detection enables fair and consistent benchmarking against pipeline-based systems.
• The EEND-EDA framework demonstrates robustness and flexibility in handling variable speaker counts and complex overlap scenarios.
• The sequence-to-sequence attractor generation process enables end-to-end training with improved generalization for unseen speaker counts.