[论文解读] Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition

Shoukang Hu, Xurong Xie+6 more|arXiv (Cornell University)|Dec 8, 2020

Speech Recognition and Synthesis参考文献 81被引用 4

一句话总结

本文提出了一种完整的贝叶斯框架，用于基于LF-MMI训练的因子化TDNN，通过变分推断对权重参数、激活函数和隐藏层输出中的不确定性进行建模。在多个语音识别任务中，该方法实现了0.4%至1.8%的绝对WER降低，优于采用标准正则化和自适应技术的最先进基线系统。

ABSTRACT

Discriminative training techniques define state-of-the-art performance for automatic speech recognition systems. However, they are inherently prone to overfitting, leading to poor generalization performance when using limited training data. In order to address this issue, this paper presents a full Bayesian framework to account for model uncertainty in sequence discriminative training of factored TDNN acoustic models. Several Bayesian learning based TDNN variant systems are proposed to model the uncertainty over weight parameters and choices of hidden activation functions, or the hidden layer outputs. Efficient variational inference approaches using a few as one single parameter sample ensure their computational cost in both training and evaluation time comparable to that of the baseline TDNN systems. Statistically significant word error rate (WER) reductions of 0.4%-1.8% absolute (5%-11% relative) were obtained over a state-of-the-art 900 hour speed perturbed Switchboard corpus trained baseline LF-MMI factored TDNN system using multiple regularization methods including F-smoothing, L2 norm penalty, natural gradient, model averaging and dropout, in addition to i-Vector plus learning hidden unit contribution (LHUC) based speaker adaptation and RNNLM rescoring. Consistent performance improvements were also obtained on a 450 hour HKUST conversational Mandarin telephone speech recognition task. On a third cross domain adaptation task requiring rapidly porting a 1000 hour LibriSpeech data trained system to a small DementiaBank elderly speech corpus, the proposed Bayesian TDNN LF-MMI systems outperformed the baseline system using direct weight fine-tuning by up to 2.5\% absolute WER reduction.

研究动机与目标

• 解决在训练数据有限时，因子化TDNN进行序列判别性训练过程中出现的过拟合问题。
• 以一种系统化的方式对网络权重、激活函数和隐藏层输出中的不确定性进行建模。
• 开发计算效率高的贝叶斯TDNN变体，其训练和推理成本与标准TDNN相当。
• 将模型不确定性估计无缝集成到序列级MMI目标函数中。
• 在多种ASR场景下，将所提出的贝叶斯方法与最先进正则化和自适应技术进行对比评估。

提出的方法

• 为LF-MMI训练的因子化TDNN构建完整的贝叶斯框架，对权重参数、激活函数和隐藏层输出中的不确定性进行建模。
• 采用变分推断，仅需一次参数采样即可确保训练和推理过程的计算效率。
• 将Dropout重新表述为所提出的贝叶斯TDNN框架的一个特例。
• 引入基于高斯过程的TDNN变体（GP-TDNN2和GP-TDNN3），以对神经网络结构选择中的不确定性进行建模，特别是激活函数的选择。
• 在模型训练和不确定性估计中使用共享的序列级MMI目标函数，确保优化过程的一致性。
• 将该框架应用于标准ASR流水线，包括i-Vector自适应、LHUC和RNNLM重打分，以实现全面评估。

Figure 1: An example TDNN architecture with the option of using standard TDNN, Bayesian TDNN and Gaussian Process TDNN. In this example, the input dimension of all hidden nodes is assumed to be two. $w_{1}$ and $w_{2}$ are the corresponding weights for each input dimension of a hidden node. Conventi

实验结果

研究问题

• RQ1 是否能够在不增加计算成本的前提下，通过完整的贝叶斯框架有效减少LF-MMI训练的TDNN中的过拟合？
• RQ2 在低资源场景下，对权重、激活函数和隐藏单元中的不确定性进行建模，对WER有何影响？
• RQ3 采用极少数样本的变分推断是否能保持与标准TDNN相当的性能，同时实现不确定性感知学习？
• RQ4 与L2正则化、Dropout和模型平均等传统正则化技术相比，所提出的贝叶斯方法在泛化能力方面表现如何？
• RQ5 在微调受限的情况下，贝叶斯TDNN在跨领域自适应性能方面能提升多少？

主要发现

• 所提出的贝叶斯TDNN系统在900小时速度扰动Switchboard基线（最先进的）上，实现了0.4%至1.8%的绝对WER降低（相对降低5%至11%），且具有统计显著性。
• 在450小时的HKUST普通话语料库上，也观察到一致的WER改进，证实了其在语言和领域间的泛化能力。
• 在从LibriSpeech（1000小时）到DementiaBank（10小时）的具有挑战性的跨领域自适应任务中，贝叶斯TDNN在直接微调的基础上将WER降低了1.1%绝对值，表现优于基线。
• GP-TDNN变体展示了对神经网络结构不确定性（特别是最优激活函数选择）的建模能力，这是标准方法所不具备的独特优势。
• 该贝叶斯框架在多种正则化技术、数据增强、说话人自适应和RNNLM重打分的设置下，始终优于最先进系统。
• 由于采用高效的变分推断并仅使用极少的参数样本，贝叶斯系统的计算成本与基线TDNN保持相当。

Figure 2: An example Variational TDNN. Note that the output vector of $l$ -th layer $\mathbf{h}_{t}^{l}$ is used as the input of the inference network $\Phi^{Infer}$ and prior network $\Phi^{Prior}$ to calculate the mean and variance of the latent variable variational and prior distributions respect