[论文解读] Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds

Keisuke Kinoshita, Marc Delcroix+1 more|arXiv (Cornell University)|Oct 26, 2020

Speech Recognition and Synthesis参考文献 21被引用 7

一句话总结

本文提出EEND-vector聚类，一种混合说话人分割框架，结合端到端神经分割（EEND）与聚类，通过为每个音频块生成全局说话人嵌入来实现。通过使用受限聚类算法在块之间聚类这些嵌入，该方法解决了块间标签排列问题，从而在长时长录音和重叠语音场景下实现鲁棒性能，尤其在长时长设置下优于原始EEND。

ABSTRACT

Recent diarization technologies can be categorized into two approaches, i.e., clustering and end-to-end neural approaches, which have different pros and cons. The clustering-based approaches assign speaker labels to speech regions by clustering speaker embeddings such as x-vectors. While it can be seen as a current state-of-the-art approach that works for various challenging data with reasonable robustness and accuracy, it has a critical disadvantage that it cannot handle overlapped speech that is inevitable in natural conversational data. In contrast, the end-to-end neural diarization (EEND), which directly predicts diarization labels using a neural network, was devised to handle the overlapped speech. While the EEND, which can easily incorporate emerging deep-learning technologies, has started outperforming the x-vector clustering approach in some realistic database, it is difficult to make it work for `long' recordings (e.g., recordings longer than 10 minutes) because of, e.g., its huge memory consumption. Block-wise independent processing is also difficult because it poses an inter-block label permutation problem, i.e., an ambiguity of the speaker label assignments between blocks. In this paper, we propose a simple but effective hybrid diarization framework that works with overlapped speech and for long recordings containing an arbitrary number of speakers. It modifies the conventional EEND framework to simultaneously output global speaker embeddings so that speaker clustering can be performed across blocks to solve the permutation problem. With experiments based on simulated noisy reverberant 2-speaker meeting-like data, we show that the proposed framework works significantly better than the original EEND especially when the input data is long.

研究动机与目标

• 为解决端到端神经分割（EEND）在处理长录音时因内存占用高和泛化能力差而产生的局限性。
• 解决分块EEND处理中的块间标签排列问题，该问题导致各段之间说话人标签不一致。
• 在单一框架中结合基于聚类的分割（对长序列具有鲁棒性）与EEND（可处理重叠语音）的优势。
• 实现在长录音（>5分钟）中有效分割，支持任意数量说话人及显著的语音重叠。

提出的方法

• 修改EEND模型，使其输出帧级分割标签以及每个音频块的全局说话人嵌入。
• 使用受限聚类算法（COP-k-means）基于共享的说话人嵌入对块之间的说话人标签进行对齐。
• 在训练过程中引入说话人损失项以提升嵌入质量，超参数λ = 0.01。
• 通过将长录音分割为非重叠块，应用带嵌入输出的EEND，然后在块之间聚类嵌入以解决标签排列问题。
• 使用多头注意力模块（4个头，256个单元）和带有学习率调度的Adam优化器进行模型训练。
• 对最后10个模型检查点取平均，并使用随机初始化运行10次推理以降低方差。

Fig. 1 : Schematic diagram of the proposed diarization framework. The input contains 3 speakers in total (red, green, and blue speakers shown in the waveform in the bottom), but only at most 2 speakers are actively speaking in each chunk.

实验结果

研究问题

• RQ1 结合EEND与聚类的混合分割系统是否能有效处理具有重叠语音的长录音？
• RQ2 所提出的方法是否能缓解分块EEND处理中固有的块间标签排列问题？
• RQ3 所提出的EEND-vector聚类在长时长数据上的性能与原始EEND相比如何？
• RQ4 引入说话人嵌入估计是否能提升聚类准确率和整体分割性能？

主要发现

• 所提出的EEND-vector聚类在长录音（如10分钟混合录音）上显著优于原始EEND，且随着时长增加优势更加明显。
• 该方法在短录音上的性能与原始EEND相当，证实所添加组件未造成性能下降。
• 块间标签排列问题在无聚类的分块EEND处理中导致显著性能下降，表现为仅分块与聚类变体之间的DER显著下降。
• t-SNE可视化结果表明，即使在存在重叠语音的情况下，说话人嵌入仍能良好分离且准确，表明全局嵌入估计可靠。
• 在所有重叠比例范围内，所提方法的DER均显著低于原始EEND，表明对不同水平的说话人重叠具有鲁棒性。
• 10次随机初始化下DER的标准差小于0.2%，表明该方法具有高度稳定性。

Fig. 2 : t-SNE plot of the 26 test speaker’s embedding vectors