[Paper Review] Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

Yu Yu, Chao-Han Huck Yang+16 more|arXiv (Cornell University)|Sep 26, 2023

Speech Recognition and SynthesisComputer Science3 citations

TL;DR

This paper proposes LoRB, a parameter-efficient low-rank adaptation (LoRA) method for fine-tuning BERT-based language models in speech recognition rescoring. By freezing all pretrained parameters and inserting trainable low-rank matrices (0.08% of total parameters), LoRB achieves performance comparable to full fine-tuning with up to 6× faster training and 32% lower memory usage, while maintaining generalization across domains through a discriminative training objective and correlation-based regularization.

ABSTRACT

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.

Motivation & Objective

• To address the high computational cost of fine-tuning large language models (LLMs) for speech recognition rescoring.
• To enable domain adaptation of pretrained BERT models with minimal parameter updates and no inference latency.
• To improve generalization on unseen domains by mitigating performance degradation common in low-rank fine-tuning.
• To achieve competitive automatic speech recognition performance with significantly reduced training time and memory usage.
• To evaluate the scaling behavior of LoRA in rescoring across model size, data size, and rank configuration.

Proposed method

• LoRA is applied to BERT by inserting a pair of low-rank matrices into each transformer layer, which are trained while all other parameters are frozen.
• A discriminative training objective is used to minimize word error rate (WER), directly optimizing for ASR performance.
• A correlation-based regularization loss is introduced to stabilize representations and prevent degradation in out-of-domain generalization.
• The method is applied to second-pass rescoring, where N-best hypothesis lists are re-scored using the adapted BERT model.
• The approach is evaluated on LibriSpeech and internal datasets across multiple domains and model sizes (5M, 170M, 1B parameters).
• Training is performed with a larger learning rate due to improved stability, enabling faster convergence despite slower initial convergence.

Fig. 1 : Illustration of the Low-Rank adaptation based Rescoring BERT (LoRB).

Experimental results

Research questions

• RQ1 Can low-rank adaptation achieve performance comparable to full fine-tuning in BERT-based speech recognition rescoring with minimal parameter updates?
• RQ2 Does the proposed correlation-based regularization improve generalization to out-of-domain test sets compared to standard LoRA?
• RQ3 How does the training speed and memory usage of LoRA compare to full fine-tuning across different model sizes and data scales?
• RQ4 What is the scaling behavior of LoRA with respect to model size, data size, and rank configuration in rescoring tasks?
• RQ5 Can LoRA maintain performance on non-target domains while adapting to a specific domain with limited in-domain data?

Key findings

• LoRB achieves a relative WER improvement of +0.23% on the non-target Shopping domain compared to the baseline, while using only 0.08% of the parameters updated in full fine-tuning.
• LoRB reduces training time by a factor of 3.6 to 5.4 compared to full fine-tuning, with up to 6× speedup when using a larger learning rate.
• LoRB reduces GPU memory utilization from 87% to 52% during training, significantly improving training efficiency.
• The performance gap between LoRB and full fine-tuning shrinks from -22.3% WER relative at 170M parameters to +2.4% at 1B parameters, indicating better scalability with larger models.
• LoRB exhibits a logarithmic scaling curve with data size, showing diminishing returns beyond 150K training utterances, unlike the linear scaling observed in full fine-tuning.
• LoRB shows improved convergence speed with larger backbone models, achieving 2.74× faster convergence with a 1B-parameter BERT.

Fig. 2 : Wall-clock training time of LoRB, LoRB+ $\mathcal{L}_{cor}$ and Fine-Tuning (FT) when training on messaging data.