[论文解读] End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification

Yusuke Fujita, Shinji Watanabe+3 more|arXiv (Cornell University)|Feb 24, 2020

Speech Recognition and Synthesis参考文献 52被引用 43

一句话总结

该论文将说话人分离重述为端到端多标签逐帧分类，采用置换-free训练，且自注意力为基础的EEND优于基于聚类的方法，同时能处理重叠。

ABSTRACT

The most common approach to speaker diarization is clustering of speaker embeddings. However, the clustering-based approach has a number of problems; i.e., (i) it is not optimized to minimize diarization errors directly, (ii) it cannot handle speaker overlaps correctly, and (iii) it has trouble adapting their speaker embedding models to real audio recordings with speaker overlaps. To solve these problems, we propose the End-to-End Neural Diarization (EEND), in which a neural network directly outputs speaker diarization results given a multi-speaker recording. To realize such an end-to-end model, we formulate the speaker diarization problem as a multi-label classification problem and introduce a permutation-free objective function to directly minimize diarization errors. Besides its end-to-end simplicity, the EEND method can explicitly handle speaker overlaps during training and inference. Just by feeding multi-speaker recordings with corresponding speaker segment labels, our model can be easily adapted to real conversations. We evaluated our method on simulated speech mixtures and real conversation datasets. The results showed that the EEND method outperformed the state-of-the-art x-vector clustering-based method, while it correctly handled speaker overlaps. We explored the neural network architecture for the EEND method, and found that the self-attention-based neural network was the key to achieving excellent performance. In contrast to conditioning the network only on its previous and next hidden states, as is done using bidirectional long short-term memory (BLSTM), self-attention is directly conditioned on all the frames. By visualizing the attention weights, we show that self-attention captures global speaker characteristics in addition to local speech activity dynamics, making it especially suitable for dealing with the speaker diarization problem.

研究动机与目标

• 激发并解决基于聚类的分离方法的局限性。
• 将说话人分离表述为端到端的多标签分类问题。
• 引入置换自由训练以解决说话人标签的置换问题。
• 探索BLSTM和自注意力架构在端到端分离中的应用。
• 在仿真混合数据和真实对话数据上证明有效性。

提出的方法

• 将Y表述为C个说话人每帧的多标签输出。
• 引入置换自由损失，以在不同说话人置换下最小化分离误差。
• 将基于BLSTM的EEND与Deep Clustering目标的EEND进行对比到自注意力驱动的EEND。
• 使用两种架构：BLSTM-EEND带DC损失和SA-EEND带编码器块与多头自注意力。
• 在仿真混合数据（SimBeta2、SimLarge）和真实数据（Real、Comb）上进行训练，并进行领域自适应（CALLHOME、CSJ）。
• 使用包含重叠的分离误差率(DER)和带颈部容忍的评估。

实验结果

研究问题

• RQ1 端到端分离能在仿真和真实数据上超越传统的基于聚类的方法吗？
• RQ2 自注意力是否在端到端分离中相对于BLSTM具有优势，尤其是在存在重叠时？
• RQ3 置换自由训练在跨时间帧解决说话人标签置换方面有多大效果？
• RQ4 在不同重叠条件下，EEND在对真实对话进行领域自适应时的表现如何？

主要发现

• 自注意力EEND（SA-EEND）在仿真混合数据上的DER明显低于聚类基线，尤其在高重叠时表现出色。
• SA-EEND结合SimLarge训练在仿真测试上实现DER为6.81–6.60%，在真实/测试集上为14.03%（CALLHOME CH）/ 21.84%（CSJ）。
• BLSTM-EEND在仿真数据上优于聚类基线，但在真实数据上表现不及SA-EEND。
• 领域自适应（CALLHOME）进一步降低SA-EEND的DER（如适应后为10.76%），相比未适应的模型。
• 多条件训练（SimLarge、Comb）在不同重叠场景下提升鲁棒性。
• 在大多数测试集上，SA-EEND在DER方面超越x-vector和i-vector聚类基线，当使用合适数据进行训练时。