[Paper Review] One Shot Learning for Speech Separation

Yuan-Kuei Wu, Kuan-Po Huang+2 more|arXiv (Cornell University)|Nov 20, 2020

Speech Recognition and Synthesis26 references4 citations

TL;DR

This paper proposes a meta-learning approach for speech separation using MAML and ANIL to enable fast adaptation to unseen speakers and noisy environments with only one mixture sample. By treating the separator as a task-specific module and the encoder/decoder as feature-reuse components, the model achieves superior generalization over multi-task learning baselines, especially in zero-shot and noisy conditions.

ABSTRACT

Despite the recent success of speech separation models, they fail to separate sources properly while facing different sets of people or noisy environments. To tackle this problem, we proposed to apply meta-learning to the speech separation task. We aimed to find a meta-initialization model, which can quickly adapt to new speakers by seeing only one mixture generated by those people. In this paper, we use model-agnostic meta-learning(MAML) algorithm and almost no inner loop(ANIL) algorithm in Conv-TasNet to achieve this goal. The experiment results show that our model can adapt not only to a new set of speakers but also noisy environments. Furthermore, we found out that the encoder and decoder serve as the feature-reuse layers, while the separator is the task-specific module.

Motivation & Objective

• To address the generalization failure of speech separation models when encountering new speakers or noisy environments.
• To enable rapid adaptation to unseen speaker combinations and acoustic conditions using only one mixture sample.
• To investigate whether meta-learning can improve zero-shot generalization in speech separation beyond multi-task learning.
• To analyze the role of different network components (encoder, separator, decoder) in meta-learning-based adaptation.

Proposed method

• Applied model-agnostic meta-learning (MAML) and almost no inner loop (ANIL) to fine-tune Conv-TasNet for one-shot speech separation.
• Used scale-invariant signal-to-noise ratio (SI-SNR) as the loss function with utterance-level permutation invariant training (uPIT).
• Treated the encoder and decoder as feature-reuse modules and the separator as the task-specific module.
• Trained the meta-learner on a diverse set of speaker mixtures from WSJ0-2mix-meta, then evaluated on unseen datasets including Libri2mix and VCTK.
• Performed fast adaptation by fine-tuning the meta-initialized model on a single mixture from a new speaker set or noisy environment.
• Compared meta-learning with multi-task learning and ablated fine-tuning strategies on different model components.

Experimental results

Research questions

• RQ1 Can meta-learning enable speech separation models to generalize to unseen speaker combinations with only one example?
• RQ2 How does meta-learning compare to multi-task learning in adapting to new noisy environments?
• RQ3 Which components of the Conv-TasNet architecture (encoder, separator, decoder) play the most critical role in fast adaptation?
• RQ4 Does fine-tuning only the separator or the full model yield better performance in meta-learning-based speech separation?

Key findings

• The MAML-based model outperformed multi-task learning baselines on all test sets, achieving the highest SI-SNRi scores on Libri2mix and VCTK datasets with and without noise.
• The ANIL method achieved better performance than standard meta-learning when fine-tuning only the separator, indicating that the separator is the primary task-specific module.
• Fine-tuning only the separator (a_s) yielded better results than fine-tuning the entire model or the autoencoder, confirming the separator's task-specific role.
• The model trained with half the pretraining epochs still outperformed full multi-task fine-tuning, suggesting that full pretraining is not necessary for effective meta-learning.
• The encoder and decoder were found to serve as feature-reuse layers, while the separator was identified as the core task-specific component.
• The meta-learned model achieved strong zero-shot generalization, demonstrating robustness to unseen speakers and noisy conditions.