[Paper Review] An evaluation of word-level confidence estimation for end-to-end automatic speech recognition

Dan Oneaţă, Alexandru Caranica+2 more|arXiv (Cornell University)|Jan 14, 2021

Speech Recognition and Synthesis45 references4 citations

TL;DR

This paper evaluates word-level confidence estimation in end-to-end automatic speech recognition (ASR) by adapting uncertainty estimation methods to structured, sequence-based outputs. It finds that temperature scaling of logits, followed by negative entropy computation and sum pooling across tokens, yields a strong baseline for word-level confidence scoring, significantly improving AUPR and AUROC on LibriSpeech, TED-LIUM, and CommonVoice datasets.

ABSTRACT

Quantifying the confidence (or conversely the uncertainty) of a prediction is a highly desirable trait of an automatic system, as it improves the robustness and usefulness in downstream tasks. In this paper we investigate confidence estimation for end-to-end automatic speech recognition (ASR). Previous work has addressed confidence measures for lattice-based ASR, while current machine learning research mostly focuses on confidence measures for unstructured deep learning. However, as the ASR systems are increasingly being built upon deep end-to-end methods, there is little work that tries to develop confidence measures in this context. We fill this gap by providing an extensive benchmark of popular confidence methods on four well-known speech datasets. There are two challenges we overcome in adapting existing methods: working on structured data (sequences) and obtaining confidences at a coarser level than the predictions (words instead of tokens). Our results suggest that a strong baseline can be obtained by scaling the logits by a learnt temperature, followed by estimating the confidence as the negative entropy of the predictive distribution and, finally, sum pooling to aggregate at word level.

Motivation & Objective

• To address the lack of confidence estimation methods tailored for end-to-end ASR systems, which are increasingly dominant in modern speech recognition.
• To adapt existing uncertainty estimation techniques—originally designed for unstructured deep learning or lattice-based systems—to sequence-level, word-level confidence scoring in end-to-end ASR.
• To overcome two key challenges: modeling structured sequence outputs and aggregating token-level uncertainty to coarser word-level predictions.
• To benchmark a comprehensive set of confidence estimation methods across multiple standard speech recognition datasets to identify the most effective approach.
• To provide a reproducible, simple, and effective baseline for future research in end-to-end ASR confidence estimation using only readily available token posteriors.

Proposed method

• Apply temperature scaling to the final logits of a pre-trained end-to-end ASR model to improve calibration of token-level probability distributions.
• Compute confidence scores using the negative entropy of the predicted token distribution as a proxy for uncertainty.
• Aggregate token-level confidence scores to word-level confidence via sum pooling, enabling user-relevant error detection in transcription tasks.
• Evaluate additional confidence estimation techniques: Monte Carlo dropout (averaging 64 forward passes) and model ensembling with independently trained models.
• Use the same pre-trained ASR model for inference and confidence scoring to ensure consistency and reproducibility across experiments.
• Optimize temperature parameters on the development split of each dataset and combine improvement techniques (e.g., temperature + dropout) to assess synergistic effects.

Fig. 2 : Fraction of errors as a function of the word length. The fraction of errors is computed as the number of erroneously transcribed words divided by the total number of words, while the word length is measured as number of tokens.

Experimental results

Research questions

• RQ1 Which confidence estimation method—log-probability, negative entropy, or Monte Carlo dropout—performs best at the word level in end-to-end ASR?
• RQ2 How does temperature scaling of logits affect the reliability of confidence scores in end-to-end ASR systems?
• RQ3 Can combining multiple confidence estimation techniques (e.g., temperature scaling + dropout) lead to improved performance over individual methods?
• RQ4 How effective is sum pooling of token-level confidence scores in producing meaningful word-level uncertainty estimates?
• RQ5 Does the performance of a pre-trained ASR model correlate with its ability to generate reliable confidence scores?

Key findings

• Temperature scaling of logits significantly improves confidence estimation performance across all metrics, yielding the largest gain in AUPR and AUROC.
• The combination of temperature scaling and negative entropy features with sum pooling achieved the best overall performance, reaching 85.18% AUROC and 41.15% AUPR on the TED-LIUM test set.
• Monte Carlo dropout with 64 runs improved performance over the baseline, with AUPR increasing from 28.58% to 33.00% when combined with temperature scaling and ensembling.
• Ensembling with multiple independently trained models further improved confidence scoring, especially when combined with temperature scaling, achieving 34.57% AUPR and 81.64% AUROC.
• The pre-trained model's predictive performance correlated with its confidence scoring performance, suggesting that high-accuracy models may naturally support better uncertainty estimation.
• The negative entropy feature showed greater improvement with temperature scaling, while log-probability features benefited more from dropout, indicating feature-specific sensitivity to methodological enhancements.

Fig. 3 : AUPR $e$ performance as a function of the number of dropout runs on the TED test set. The horizontal red line indicates the performance of the model without dropout. The model uses neg-entropy features, sum aggregation and temperature scaling.