[Paper Review] Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation

Sunwoo Kim, Minje Kim|arXiv (Cornell University)|May 8, 2021

Speech and Audio Processing4 citations

TL;DR

This paper proposes a zero-shot test-time adaptation method for personalized speech enhancement using knowledge distillation, where a compact student model is fine-tuned using pseudo-targets from a large pre-trained teacher model, achieving performance superior to larger generalist models without requiring any clean speech from the test speaker. The approach enables privacy-preserving, efficient model compression with significant performance gains on unseen speakers and noise conditions.

ABSTRACT

In realistic speech enhancement settings for end-user devices, we often encounter only a few speakers and noise types that tend to reoccur in the specific acoustic environment. We propose a novel personalized speech enhancement method to adapt a compact denoising model to the test-time specificity. Our goal in this test-time adaptation is to utilize no clean speech target of the test speaker, thus fulfilling the requirement for zero-shot learning. To complement the lack of clean utterance, we employ the knowledge distillation framework. Instead of the missing clean utterance target, we distill the more advanced denoising results from an overly large teacher model, and use it as the pseudo target to train the small student model. This zero-shot learning procedure circumvents the process of collecting users' clean speech, a process that users are reluctant to comply due to privacy concerns and technical difficulty of recording clean voice. Experiments on various test-time conditions show that the proposed personalization method achieves significant performance gains compared to larger baseline networks trained from a large speaker- and noise-agnostic datasets. In addition, since the compact personalized models can outperform larger general-purpose models, we claim that the proposed method performs model compression with no loss of denoising performance.

Motivation & Objective

• To address the challenge of deploying effective speech enhancement models on resource-constrained devices with personalized performance for specific users.
• To enable zero-shot learning in speech enhancement, avoiding the need for users to provide clean speech samples due to privacy and practicality concerns.
• To develop a test-time adaptation framework that personalizes a compact student model to the test speaker’s voice and acoustic environment using knowledge distillation.
• To achieve model compression that maintains or exceeds the performance of larger generalist models through personalized adaptation.
• To enable efficient, on-device deployment by transferring only the small student model after cloud-based fine-tuning.

Proposed method

• A knowledge distillation framework is used where the student model is trained using the output of a large, pre-trained teacher model as a pseudo-target, instead of requiring actual clean speech.
• The teacher model, trained on a large, diverse dataset, generalizes well across various test conditions and provides high-quality denoising estimates.
• The student model is fine-tuned at test time using the teacher’s predictions as supervision, enabling adaptation to the specific speaker and noise environment without labeled clean data.
• The method supports heterogeneous architectures between teacher and student, allowing flexibility in model design and compression.
• The framework enables on-device inference using only the compact student model, while the computationally heavy distillation process is offloaded to the cloud.
• Personalization is achieved through test-time adaptation, where the student model is optimized for the specific test-time context without retraining from scratch.

Figure 1: An overview of the proposed KD-based PSE process. The estimated clean speech by the student model is compared against the result from a larger teacher model, whose discrepancy is used to fine-tune the student model. This can be done during the test time as it does not require the clean spe

Experimental results

Research questions

• RQ1 Can a compact student model be effectively personalized to a specific speaker and noise environment at test time without access to clean speech data?
• RQ2 Does knowledge distillation from a large, generalist teacher model enable superior performance compared to generalist models of similar or larger size?
• RQ3 Can the proposed zero-shot method achieve model compression without performance loss, even when the student is smaller than the teacher?
• RQ4 How does the performance of the student model vary when fine-tuned using different teacher models with varying architectures and capabilities?
• RQ5 Can the framework be extended to improve performance for underrepresented speaker groups in the training data?

Key findings

• The personalized student models outperform larger generalist models across all test conditions, with the 2×32 CTN student model achieving better performance than the 2×1024 generalist model at -5 dB mixture SNR.
• The personalized student model trained with the CTN teacher outperforms the one trained with the GRU teacher, demonstrating that teacher quality directly impacts student performance.
• The 2×32 personalized model saves 11.47 million parameters and 719 million MACs compared to the 2×1024 generalist model, confirming effective model compression.
• In some cases, the personalized student model even surpasses its teacher model, particularly when the teacher is only slightly larger, due to dedicated adaptation to the test-time context.
• The method enables privacy-preserving personalization, as no user-provided clean speech is required, addressing key concerns in real-world deployment.
• The framework is flexible and can be applied across different model architectures and loss functions, suggesting broad applicability beyond the current setup.

(a) -5 dB Mixture SNR