[Paper Review] Efficient Adapters for Giant Speech Models

Nanxin Chen, Izhak Shafran+5 more|arXiv (Cornell University)|Jun 13, 2023

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes Two Parallel Adapters (TPA), a novel adapter architecture for Conformer-based giant speech models that inserts parallel residual adapters into both feed-forward networks of each block. TPA achieves performance comparable to full fine-tuning—matching or exceeding it on 50% of speech recognition benchmarks—while updating only 1.4% to 11% of parameters, and enables efficient, task-specific adaptation with a frozen pre-trained encoder.

ABSTRACT

Large pre-trained speech models are widely used as the de-facto paradigm, especially in scenarios when there is a limited amount of labeled data available. However, finetuning all parameters from the self-supervised learned model can be computationally expensive, and becomes infeasiable as the size of the model and the number of downstream tasks scales. In this paper, we propose a novel approach called Two Parallel Adapter (TPA) that is inserted into the conformer-based model pre-trained model instead. TPA is based on systematic studies of the residual adapter, a popular approach for finetuning a subset of parameters. We evaluate TPA on various public benchmarks and experiment results demonstrates its superior performance, which is close to the full finetuning on different datasets and speech tasks. These results show that TPA is an effective and efficient approach for serving large pre-trained speech models. Ablation studies show that TPA can also be pruned, especially for lower blocks.

Motivation & Objective

• Address the computational and storage inefficiency of full fine-tuning in large pre-trained speech models.
• Overcome catastrophic forgetting and overfitting in low-resource scenarios by minimizing parameter updates.
• Develop a general-purpose, efficient adaptation method compatible with diverse speech tasks and multilingual settings.
• Enable scalable deployment of large speech models across multiple downstream tasks without retraining the encoder.
• Investigate structural and architectural factors of residual adapters to design an optimized, prunable, and high-performing variant.

Proposed method

• Propose Two Parallel Adapters (TPA), a dual-branch residual adapter inserted into both feed-forward networks (FFN1 and FFN2) of each Conformer block.
• Use a residual connection with two fully connected layers and ReLU activation, optionally including layer normalization.
• Keep the pre-trained Conformer encoder frozen during training, updating only the TPA and a randomly initialized decoder.
• Apply systematic ablation studies on adapter components (e.g., layer norm, ReLU, depth, width) to identify optimal configurations.
• Implement iterative neuron pruning on TPA components, especially in lower layers, based on activation statistics.
• Use a unified framework where the same frozen encoder serves multiple downstream tasks via task-specific TPA and decoder heads.

Experimental results

Research questions

• RQ1 Can a dual-branch adapter architecture (TPA) outperform standard serial or single-branch residual adapters in speech recognition and translation tasks?
• RQ2 How does the architectural design of TPA (e.g., layer norm, width, depth) affect performance and parameter efficiency?
• RQ3 To what extent can TPA be pruned without performance degradation, especially in lower versus upper blocks?
• RQ4 Can TPA maintain performance close to full fine-tuning while updating only a small fraction of parameters across diverse speech benchmarks?
• RQ5 Is TPA generalizable across multiple speech tasks, including multilingual ASR and multilingual speech translation?

Key findings

• TPA achieves performance within 0.9 BLEU of full fine-tuning on CoVoST 2 multilingual speech translation, updating only 1.4% of parameters.
• On LibriSpeech, TPA achieves a WER of 1.9% on the clean set and 3.7% on the other set, matching full fine-tuning performance with 11% parameter updates.
• On the challenging AMI and Switchboard datasets, TPA matches or slightly underperforms full fine-tuning by less than 0.3 WER, despite updating only 2% of parameters.
• Ablation studies show that up to 60–80% of neurons in the second FFN block can be pruned without performance loss, especially in lower layers.
• Pruning the frozen encoder is less effective, but 20–40% of neurons in blocks 2–10 can be removed without degradation, indicating potential for inference speedup.
• TPA outperforms both serial and parallel adapter variants reported in prior work, demonstrating superior parameter efficiency and accuracy.