[Paper Review] Bayesian Learning of LF-MMI Trained Time Delay Neural Networks for Speech Recognition

Shoukang Hu, Xurong Xie+6 more|arXiv (Cornell University)|Dec 8, 2020

Speech Recognition and Synthesis81 references4 citations

TL;DR

This paper proposes a full Bayesian framework for LF-MMI trained factored TDNNs to model uncertainty in weight parameters, activation functions, and hidden layer outputs using variational inference. It achieves 0.4%-1.8% absolute WER reduction across multiple speech recognition tasks, outperforming state-of-the-art baseline systems with standard regularization and adaptation techniques.

ABSTRACT

Discriminative training techniques define state-of-the-art performance for automatic speech recognition systems. However, they are inherently prone to overfitting, leading to poor generalization performance when using limited training data. In order to address this issue, this paper presents a full Bayesian framework to account for model uncertainty in sequence discriminative training of factored TDNN acoustic models. Several Bayesian learning based TDNN variant systems are proposed to model the uncertainty over weight parameters and choices of hidden activation functions, or the hidden layer outputs. Efficient variational inference approaches using a few as one single parameter sample ensure their computational cost in both training and evaluation time comparable to that of the baseline TDNN systems. Statistically significant word error rate (WER) reductions of 0.4%-1.8% absolute (5%-11% relative) were obtained over a state-of-the-art 900 hour speed perturbed Switchboard corpus trained baseline LF-MMI factored TDNN system using multiple regularization methods including F-smoothing, L2 norm penalty, natural gradient, model averaging and dropout, in addition to i-Vector plus learning hidden unit contribution (LHUC) based speaker adaptation and RNNLM rescoring. Consistent performance improvements were also obtained on a 450 hour HKUST conversational Mandarin telephone speech recognition task. On a third cross domain adaptation task requiring rapidly porting a 1000 hour LibriSpeech data trained system to a small DementiaBank elderly speech corpus, the proposed Bayesian TDNN LF-MMI systems outperformed the baseline system using direct weight fine-tuning by up to 2.5\% absolute WER reduction.

Motivation & Objective

• To address overfitting in sequence discriminative training of factored TDNNs when using limited training data.
• To model uncertainty in network weights, activation functions, and hidden layer outputs in a principled Bayesian manner.
• To develop computationally efficient Bayesian TDNN variants with training and inference costs comparable to standard TDNNs.
• To integrate model uncertainty estimation seamlessly within the sequence-level MMI objective function.
• To evaluate the proposed Bayesian methods against state-of-the-art regularization and adaptation techniques in diverse ASR scenarios.

Proposed method

• Formulates a full Bayesian framework for LF-MMI trained factored TDNNs, modeling uncertainty over weight parameters, activation functions, and hidden layer outputs.
• Employs variational inference with as few as one parameter sample to ensure computational efficiency during training and inference.
• Reformulates dropout as a special case of the proposed Bayesian TDNN framework.
• Introduces Gaussian Process-based TDNN variants (GP-TDNN2 and GP-TDNN3) to model uncertainty in neural architecture choices, particularly activation functions.
• Uses a shared sequence-level MMI objective for both model training and uncertainty estimation, ensuring consistency in optimization.
• Applies the framework to standard ASR pipelines including i-Vector adaptation, LHUC, and RNNLM rescoring for comprehensive evaluation.

Figure 1: An example TDNN architecture with the option of using standard TDNN, Bayesian TDNN and Gaussian Process TDNN. In this example, the input dimension of all hidden nodes is assumed to be two. $w_{1}$ and $w_{2}$ are the corresponding weights for each input dimension of a hidden node. Conventi

Experimental results

Research questions

• RQ1 Can a full Bayesian framework effectively reduce overfitting in LF-MMI trained TDNNs without increasing computational cost?
• RQ2 How does modeling uncertainty in weights, activation functions, and hidden units impact WER in low-data scenarios?
• RQ3 Can variational inference with minimal samples maintain performance comparable to standard TDNNs while enabling uncertainty-aware learning?
• RQ4 How does the proposed Bayesian approach compare to conventional regularization techniques like L2, dropout, and model averaging in terms of generalization?
• RQ5 To what extent can Bayesian TDNNs improve cross-domain adaptation performance, especially when fine-tuning is limited?

Key findings

• The proposed Bayesian TDNN systems achieved statistically significant WER reductions of 0.4%-1.8% absolute (5%-11% relative) over a state-of-the-art 900-hour speed-perturbed Switchboard baseline.
• On the 450-hour HKUST Mandarin corpus, consistent WER improvements were observed, confirming generalization across language and domain.
• In a challenging cross-domain adaptation task from LibriSpeech (1000h) to DementiaBank (10h), the Bayesian TDNN reduced WER by 1.1% absolute over direct fine-tuning, outperforming the baseline.
• The GP-TDNN variants demonstrated the ability to model neural architecture uncertainty, particularly in selecting optimal activation functions, a unique advantage over standard methods.
• The Bayesian framework consistently outperformed state-of-the-art systems using multiple regularization techniques, data augmentation, speaker adaptation, and RNNLM rescoring.
• The computational cost of the Bayesian systems remained comparable to baseline TDNNs due to efficient variational inference with minimal parameter samples.

Figure 2: An example Variational TDNN. Note that the output vector of $l$ -th layer $\mathbf{h}_{t}^{l}$ is used as the input of the inference network $\Phi^{Infer}$ and prior network $\Phi^{Prior}$ to calculate the mean and variance of the latent variable variational and prior distributions respect