[Paper Review] Effects of Layer Freezing on Transferring a Speech Recognition System to Under-resourced Languages

Onno Eberhard, Torsten Zesch|arXiv (Cornell University)|Feb 8, 2021

Speech Recognition and Synthesis4 citations

TL;DR

This paper investigates layer freezing in transfer learning for automatic speech recognition (ASR) on low-resource languages, using Mozilla's DeepSpeech architecture to transfer a pre-trained English model to German and Swiss German. Freezing the first one to three layers significantly improves word error rates (WER) compared to fine-tuning all layers, with minimal gains beyond three frozen layers, demonstrating that early layers learn generalizable features across related languages.

ABSTRACT

In this paper, we investigate the effect of layer freezing on the effectiveness of model transfer in the area of automatic speech recognition. We experiment with Mozilla's DeepSpeech architecture on German and Swiss German speech datasets and compare the results of either training from scratch vs. transferring a pre-trained model. We compare different layer freezing schemes and find that even freezing only one layer already significantly improves results.

Motivation & Objective

• To evaluate the impact of layer freezing on transfer learning effectiveness in ASR for low-resource languages.
• To determine optimal layer freezing strategies when adapting an English pre-trained DeepSpeech model to German and Swiss German.
• To assess whether hierarchical feature learning in deep networks supports freezing early layers during fine-tuning.
• To compare performance across different freezing schemes in terms of WER and CER on limited-resource datasets.

Proposed method

• Fine-tune a pre-trained English DeepSpeech model on German and Swiss German datasets using transfer learning.
• Apply various layer freezing schemes: freezing 0, 1, 2, 3, 4, 5, or 1–5 layers (excluding the final output layer).
• Use the DeepSpeech architecture with 6 layers: 3 fully connected (FC) layers, 1 LSTM, and 2 final FC layers with ReLU and softmax activations.
• Optimize the model using the Adam optimizer and Connectionist Temporal Classification (CTC) loss for sequence-to-sequence alignment.
• Evaluate results using WER and CER on held-out test sets, with KenLM language models trained on mixed text corpora.
• Train and validate using learning curves and epoch-specific metrics to assess convergence and generalization.

Experimental results

Research questions

• RQ1 How does freezing different numbers of layers affect ASR performance when transferring a pre-trained English model to German and Swiss German?
• RQ2 Does freezing early layers (which learn low-level features) yield better results than freezing later layers or fine-tuning all layers?
• RQ3 What is the optimal number of layers to freeze for minimizing WER and CER on low-resource speech recognition tasks?
• RQ4 How do learning curves and validation loss compare across different freezing configurations?
• RQ5 Does freezing more layers lead to diminishing returns or even degradation in performance?

Key findings

• Freezing the first one to three layers yields the best performance, reducing WER from 70% (baseline) to 44% on German and from 74% to 67% on Swiss German.
• The model with layers 1–2 frozen achieved the lowest WER (44%) and CER (0.22) on German, outperforming all other freezing schemes.
• Freezing more than three layers (e.g., 1–5) results in only marginal or no improvement, with slight degradation observed in some cases, suggesting diminishing returns.
• Models with four or five frozen layers still achieve strong performance, indicating that features from the English pre-trained model are generalizable across related languages.
• The learning curves show that freezing early layers leads to faster convergence and lower validation loss, with minimal differences between freezing 1–3 and 1–5 layers.
• Freezing the LSTM layer (layer 4) or the final FC layer (layer 5) does not significantly affect performance, suggesting that these layers may not be as language-specific as expected.