[论文解读] TMS: A Temporal Multi-scale Backbone Design for Speaker Embedding

Ruiteng Zhang, Jianguo Wei+7 more|arXiv (Cornell University)|Mar 17, 2022

Speech Recognition and Synthesis被引用 5

一句话总结

本文提出TMS，一种用于说话人嵌入的时序多尺度主干网络设计，通过分离时序建模与通道建模，以极低的计算成本高效整合多尺度特征。通过支持更多具有更大感受野的分支，并采用重参数化策略实现更快的推理速度，TMS在VoxCeleb和CNCeleb数据集上达到最先进性能，EER最高降低1.3%，推理速度相比ECAPA-TDNN提升29–46%。

ABSTRACT

Speaker embedding is an important front-end module to explore discriminative speaker features for many speech applications where speaker information is needed. Current SOTA backbone networks for speaker embedding are designed to aggregate multi-scale features from an utterance with multi-branch network architectures for speaker representation. However, naively adding many branches of multi-scale features with the simple fully convolutional operation could not efficiently improve the performance due to the rapid increase of model parameters and computational complexity. Therefore, in the most current state-of-the-art network architectures, only a few branches corresponding to a limited number of temporal scales could be designed for speaker embeddings. To address this problem, in this paper, we propose an effective temporal multi-scale (TMS) model where multi-scale branches could be efficiently designed in a speaker embedding network almost without increasing computational costs. The new model is based on the conventional TDNN, where the network architecture is smartly separated into two modeling operators: a channel-modeling operator and a temporal multi-branch modeling operator. Adding temporal multi-scale in the temporal multi-branch operator needs only a little bit increase of the number of parameters, and thus save more computational budget for adding more branches with large temporal scales. Moreover, in the inference stage, we further developed a systemic re-parameterization method to convert the TMS-based model into a single-path-based topology in order to increase inference speed. We investigated the performance of the new TMS method for automatic speaker verification (ASV) on in-domain and out-of-domain conditions. Results show that the TMS-based model obtained a significant increase in the performance over the SOTA ASV models, meanwhile, had a faster inference speed.

研究动机与目标

• 为解决现有多分支网络在说话人嵌入中因扩展时序感受野而导致的高参数量与高计算成本问题。
• 在不显著增加计算开销的前提下，实现更深、更宽的多尺度时序分支设计。
• 通过增强多尺度特征学习能力，提升模型在分布外数据上的泛化性能。
• 通过重参数化策略将多分支结构转换为单路径结构，实现推理速度的显著提升。

提出的方法

• TMS模块将标准TDNN分解为两个独立操作：通道建模操作与时序多分支建模操作。
• 时序多分支操作通过使用扩张1D卷积并逐步增大卷积核尺寸，在多个分支中捕捉多尺度时序上下文。
• 通道建模操作采用逐点卷积来建模通道间关系，将其与时序建模解耦，从而降低复杂度。
• 在训练后应用重参数化方法，将多分支结构合并为等效的单路径结构，以实现更快的推理速度。
• 该方法被集成到E-TDNN和Res2Net-based主干网络中（如Rep-E-TMS-TDNN和Rep-A-TMS-TDNN），并引入自适应批量归一化与挤压-激励模块以进一步提升性能。
• 模型采用AM-Softmax损失进行训练，并在VoxCeleb和CNCeleb数据集上，在域内与域外条件下进行评估。

实验结果

研究问题

• RQ1 是否可以使多分支时序建模设计在计算效率上足够高效，以支持大量具有大时序感受野的分支？
• RQ2 将通道建模与时序建模分离，是否能提升多尺度说话人嵌入网络的效率与可扩展性？
• RQ3 对多分支网络进行重参数化，是否能在不损失性能的前提下实现显著的推理加速？
• RQ4 TMS-based模型在分布外数据上的泛化能力是否优于ECAPA-TDNN？
• RQ5 TMS模块中分支数量的变化如何影响不同评估集上的性能表现？

主要发现

• 采用四分支的Rep-E-TMS-TDNN模型在VoxCeleb1-test上达到1.44%的EER，优于基线E-TDNN（1.65%）与ECAPA-TDNN。
• 在分布外的CNCeleb数据集上，TMS模型相比ECAPA-TDNN将EER降低了1.3个百分点，展现出更优的泛化能力。
• 得益于重参数化为单路径拓扑结构，TMS-based模型相比ECAPA-TDNN实现了29–46%的推理速度提升。
• 消融实验表明，四分支的TMS模块显著提升了性能，而SE-Block与AS-Norm进一步增强了模型鲁棒性。
• 四分支模型在VoxCeleb1-test、VoxCeleb1-E与VoxCeleb1-H上的minDCF2值分别为0.1685、0.1626与0.2431，均低于ECAPA-TDNN与E-TDNN基线。
• 重参数化的TMS模型在显著降低推理延迟的同时保持或提升了性能，证明其在实时应用中的有效性。