[Paper Review] Deep Speaker: an End-to-End Neural Speaker Embedding System

Chao Li, Xiaokong Ma+7 more|arXiv (Cornell University)|May 5, 2017

Speech Recognition and Synthesis14 references426 citations

TL;DR

Deep Speaker proposes end-to-end speaker embeddings mapped to a hypersphere using ResCNN or GRU frame extractors with triplet loss and softmax pre-training, achieving substantial improvements over i-vector baselines and transferring across languages.

ABSTRACT

We present Deep Speaker, a neural speaker embedding system that maps utterances to a hypersphere where speaker similarity is measured by cosine similarity. The embeddings generated by Deep Speaker can be used for many tasks, including speaker identification, verification, and clustering. We experiment with ResCNN and GRU architectures to extract the acoustic features, then mean pool to produce utterance-level speaker embeddings, and train using triplet loss based on cosine similarity. Experiments on three distinct datasets suggest that Deep Speaker outperforms a DNN-based i-vector baseline. For example, Deep Speaker reduces the verification equal error rate by 50% (relatively) and improves the identification accuracy by 60% (relatively) on a text-independent dataset. We also present results that suggest adapting from a model trained with Mandarin can improve accuracy for English speaker recognition.

Motivation & Objective

• Motivate a robust end-to-end embedding approach for speaker recognition that directly optimizes utterance-level similarity.
• Develop and compare two deep architectures (ResCNN and GRU) for frame-level feature extraction.
• Learn utterance embeddings via mean pooling and cosine normalization optimized with triplet loss.
• Enhance training with softmax pre-training and hard negative mining across GPUs to improve convergence.
• Demonstrate cross-language transfer (Mandarin-English) and the impact of data scale and enrollment utterances.

Proposed method

• Use two deep architectures for frame-level features: a residual CNN (ResCNN) and a stacked GRU network.
• Apply a sentence-level average pooling to produce utterance representations, followed by an affine projection to a 512-dim embedding with length normalization.
• Train with a cosine-based triplet loss that enforces higher similarity for same-speaker pairs than for different-speaker pairs, using hard negatives found across GPUs.
• Employ softmax pre-training with a classification layer to stabilize/initialize training before triplet-loss fine-tuning.
• Explore hard negative mining by scanning across GPUs to select challenging negatives within mini-batches, enhancing convergence behavior.
• Optionally fuse ResCNN and GRU via embedding or score fusion to improve performance.

Experimental results

Research questions

• RQ1 Does end-to-end embedding learning with triplet loss outperform traditional i-vector baselines on text-independent and text-dependent tasks?
• RQ2 How do ResCNN and GRU architectures compare in yielding robust speaker embeddings for Mandarin and English datasets?
• RQ3 What is the impact of softmax pre-training and cross-GPU hard negative mining on training stability and final accuracy?
• RQ4 How does training data size, enrollment utterance count, and cross-language adaptation affect performance?
• RQ5 Can Deep Speaker transfer across languages (Mandarin to English) and remain effective in text-dependent scenarios?

Key findings

• Deep Speaker reduces verification EER by roughly 50-80% (Mandarin UIDs) and identification error by 60-70% relative to DNN i-vectors.
• Softmax pre-training followed by triplet fine-tuning yields the best performance for both ResCNN and GRU.
• On Mandarin text-independent tasks, ResCNN with softmax+triplet achieves 2.23% EER and 90.53% ACC, vs. 13.79% EER and 51.72% ACC for the DNN i-vector baseline.
• On MTurk English data, finetuned models achieve substantial gains (e.g., EER reductions around 2.4-2.7% with high ACC).
• Enlarging training data from Train50k to Train250k improves performance (e.g., ResCNN: EER from 2.23% to 1.83%; ACC from 90.53% to 92.58%).
• Enrollment utterance count improves EER/ACC with diminishing returns (e.g., 1→5 enrollments yields notable gains).
• Deep Speaker models trained on large(text-independent) data transfer effectively to text-dependent XiaoDu and cross-language MTurk tasks, with fine-tuning further improving results.