[Paper Review] Few Shot Speaker Recognition using Deep Neural Networks

Prashant Anand, Ajeet Kumar Singh+2 more|arXiv (Cornell University)|Apr 17, 2019

Speech Recognition and SynthesisComputer Science20 references34 citations

TL;DR

This paper proposes few-shot speaker recognition using CNNs and CapsuleNet with prototypical loss, plus an auto-encoder to map capsule class vectors into a generalized embedding space, evaluated on VoxCeleb1 and VCTK with very short 3-second utterances.

ABSTRACT

The recent advances in deep learning are mostly driven by availability of large amount of training data. However, availability of such data is not always possible for specific tasks such as speaker recognition where collection of large amount of data is not possible in practical scenarios. Therefore, in this paper, we propose to identify speakers by learning from only a few training examples. To achieve this, we use a deep neural network with prototypical loss where the input to the network is a spectrogram. For output, we project the class feature vectors into a common embedding space, followed by classification. Further, we show the effectiveness of capsule net in a few shot learning setting. To this end, we utilize an auto-encoder to learn generalized feature embeddings from class-specific embeddings obtained from capsule network. We provide exhaustive experiments on publicly available datasets and competitive baselines, demonstrating the superiority and generalization ability of the proposed few shot learning pipelines.

Motivation & Objective

• Motivate practical speaker recognition under very limited data and short utterances.
• Propose a few-shot learning pipeline using spectrogram inputs and prototypical loss.
• Evaluate CNN and Capsule Network approaches, with an auto-encoder to enable generalization to unseen speakers.
• Show that prototypical loss improves few-shot performance across architectures.

Proposed method

• Convert audio to single-channel 16 kHz, 16-bit streams and compute 128x300 spectrograms per 3-second utterance.
• Use CNN baselines (VGG-M, ResNet-34) and a modified Capsule Network (CapsuleNet-M) as feature extractors.
• Extend CapsuleNet with an auto-encoder to produce a generalized embedding suitable for prototypical loss.
• Apply prototypical loss to learn class prototypes in embedding space for few-shot classification.
• Incorporate a contractive auto-encoder to produce embeddings from capsule class vectors for the few-shot setting (CapsuleNet-MA).
• Train end-to-end and evaluate under 5-way and 20-way, with 1-shot and 5-shot regimes.

Experimental results

Research questions

• RQ1 Can few-shot learning enable accurate speaker identification from 3-second utterances?
• RQ2 How do CNN and Capsule Network approaches compare under few-shot conditions?
• RQ3 Does mapping capsule-derived class vectors through an auto-encoder facilitate generalization to unseen speakers?
• RQ4 Does prototypical loss improve performance for few-shot speaker recognition across architectures?

Key findings

• ResNet-34 substantially outperform other networks on standard (non-few-shot) VoxCeleb1 subsets, achieving Top-1 90.37% and Top-5 98.13% for 50 classes, and Top-1 71.48% and Top-5 88.45% for 200 classes.
• In few-shot setups, ResNet-34 achieves 1-shot 79.97% and 5-shot 91.50% on 5-way VoxCeleb1, while CapsuleNet-MA attains 53.62% (1-shot) and 82.93% (5-shot) and VGG-M attains 52.42% (1-shot) and 82.10% (5-shot).
• CapsuleNet-MA outperforms VGG-M in several few-shot settings and approaches ResNet performance with fewer parameters, while standard CapsuleNet (CapsuleNet-M) lags behind ResNet but remains competitive with VGG-M.
• On the VCTK corpus, non-few-shot results show CapsuleNet-M 91.95% Top-1 and 98.13% Top-5, VGG-M 95.25% Top-1 and 99.45% Top-5, and ResNet-34 96.91% Top-1 and 99.91% Top-5.
• In few-shot VCTK, CapsuleNet-MA yields 5-way 1-shot 65.26% and 5-way 5-shot 91.28%, while ResNet-34 yields 5-way 1-shot 80.96% and 5-way 5-shot 96.46%.