[Paper Review] Private Language Model Adaptation for Speech Recognition

Zhe Liu, Ke Li+2 more|arXiv (Cornell University)|Sep 28, 2021

Privacy-Preserving Technologies in Data29 references4 citations

TL;DR

This paper proposes a federated learning (FL)-based approach for private adaptation of neural network language models (NNLMs) in automatic speech recognition (ASR), using token-level confidence scores to mitigate errors in on-device transcribed hypotheses. By weighting the cross-entropy loss according to confidence scores, the method achieves relative 2.6% and 10.8% word error rate (WER) reductions on two evaluation sets, improving ASR accuracy while preserving user privacy.

ABSTRACT

Speech model adaptation is crucial to handle the discrepancy between server-side proxy training data and actual data received on local devices of users. With the use of federated learning (FL), we introduce an efficient approach on continuously adapting neural network language models (NNLMs) on private devices with applications on automatic speech recognition (ASR). To address the potential speech transcription errors in the on-device training corpus, we perform empirical studies on comparing various strategies of leveraging token-level confidence scores to improve the NNLM quality in the FL settings. Experiments show that compared with no model adaptation, the proposed method achieves relative 2.6% and 10.8% word error rate (WER) reductions on two speech evaluation datasets, respectively. We also provide analysis in evaluating privacy guarantees of our presented procedure.

Motivation & Objective

• To address the domain shift between server-side training data and real user speech on local devices in on-device ASR systems.
• To enable private, on-device adaptation of NNLMs without exposing raw user data, leveraging federated learning (FL).
• To improve adaptation quality by mitigating transcription errors in ASR-generated hypotheses using token-level confidence scores.
• To evaluate the privacy guarantees of the proposed FL-based adaptation using Rényi differential privacy (RDP).
• To demonstrate that confidence-based loss weighting outperforms naive training on potentially erroneous transcribed data.

Proposed method

• The method uses federated learning to fine-tune a pre-trained NNLM on-device using user-generated speech transcripts, with model updates aggregated on a central server.
• On-device, the ASR system generates hypotheses for user utterances, which are used as unsupervised training data for the NNLM.
• A confidence classifier estimates the likelihood of correctness for each token in the ASR hypotheses, providing per-token confidence scores.
• The NNLM training objective is modified by weighting the cross-entropy loss using these confidence scores to down-weight low-confidence tokens.
• The approach supports both utterance-level and token-level confidence weighting strategies to improve robustness to transcription errors.
• Privacy is analyzed using Rényi differential privacy (RDP), with L2 gradient clipping and Gaussian noise injection during model aggregation.

Experimental results

Research questions

• RQ1 Can federated learning effectively adapt neural network language models to user-specific speech domains without exposing raw data?
• RQ2 How do different confidence-based weighting strategies (e.g., token-level vs. utterance-level) affect NNLM adaptation quality in the presence of ASR transcription errors?
• RQ3 What is the impact of using confidence scores on the final word error rate (WER) in end-to-end ASR systems?
• RQ4 How do privacy guarantees, measured via Rényi differential privacy, trade off with model performance in the FL-based NNLM adaptation setting?

Key findings

• The proposed method achieves a relative 2.6% WER reduction on the conversation evaluation set and 10.8% on the voice command set compared to non-adapted models.
• Token-level confidence weighting outperforms all other strategies, yielding the best WER performance across both datasets.
• Hard thresholding of utterances based on confidence leads to the best result on the short voice command set, while token-level weighting excels on longer conversations.
• Training from scratch in the FL setting results in ~1% WER degradation compared to fine-tuning from a pre-trained model, indicating the benefit of pre-training.
• Even under strong privacy constraints (ε ≈ 0.9), the adapted models still outperform the non-adapted server-pretrained model in terms of WER, though gains are reduced.
• The privacy analysis using Rényi DP confirms that the method provides meaningful privacy guarantees, with ε values ranging from 0.9 to 248.6 depending on noise levels.