[Paper Review] Text-Independent Speaker Verification Using Long Short-Term Memory Networks

Aryan Mobiny, Najarian, Mohammad|arXiv (Cornell University)|May 2, 2018

Speech Recognition and Synthesis18 references7 citations

TL;DR

This paper proposes an end-to-end LSTM-based architecture for text-independent speaker verification that directly models temporal speaker characteristics from MFCC features. By jointly learning background and speaker models in an end-to-end fashion, the method achieves a 22.9% EER on the evaluation set, outperforming GMM-UBM (27.1%) and i-vector + PLDA (23.5%) baselines, particularly excelling with short utterances.

ABSTRACT

In this paper, an architecture based on Long Short-Term Memory Networks has been proposed for the text-independent scenario which is aimed to capture the temporal speaker-related information by operating over traditional speech features. For speaker verification, at first, a background model must be created for speaker representation. Then, in enrollment stage, the speaker models will be created based on the enrollment utterances. For this work, the model will be trained in an end-to-end fashion to combine the first two stages. The main goal of end-to-end training is the model being optimized to be consistent with the speaker verification protocol. The end- to-end training jointly learns the background and speaker models by creating the representation space. The LSTM architecture is trained to create a discrimination space for validating the match and non-match pairs for speaker verification. The proposed architecture demonstrate its superiority in the text-independent compared to other traditional methods.

Motivation & Objective

• To develop an end-to-end deep learning model that captures temporal speaker-specific patterns in text-independent speaker verification.
• To improve speaker verification performance by jointly optimizing background and speaker model learning for the verification protocol.
• To demonstrate the superiority of LSTMs over traditional methods like GMM-UBM and i-vector in handling short utterances.
• To investigate the limitations of LSTMs in modeling long-term speaker variations compared to conventional approaches.

Proposed method

• The model uses LSTM networks to process raw speech features (MFCCs) extracted from 1-second utterances with 60% overlap.
• End-to-end training jointly optimizes the background model and speaker model by learning a shared representation space.
• A Siamese-like loss function is used, combining a contrastive loss with margin-based ranking: $ C_W = Y imes C_{gen} + (1-Y) imes C_{imp} + \lambda \|W\|_2 $.
• The $ C_{gen} $ and $ C_{imp} $ terms are defined as $ \frac{1}{2} D_W^2 $ and $ \frac{1}{2} \max\{0, (M - D_W)\}^2 $, respectively, with $ M $ as the margin.
• The system uses batch normalization and data augmentation during training to stabilize learning and improve generalization.
• For evaluation, the model computes embedding distances and uses a dynamic thresholding strategy based on max and min distances of genuine pairs.

Experimental results

Research questions

• RQ1 Can an end-to-end LSTM architecture effectively learn speaker-specific temporal patterns in text-independent speaker verification?
• RQ2 How does the proposed model compare to traditional methods like GMM-UBM and i-vector in terms of EER across varying utterance lengths?
• RQ3 Does the LSTM-based model maintain superior performance on short utterances compared to conventional models?
• RQ4 Why do traditional models still outperform LSTMs on longer utterances, and what are the limitations of LSTMs in capturing long-term speaker variability?

Key findings

• The proposed LSTM model achieves a 22.9% Equal Error Rate (EER), outperforming GMM-UBM (27.1%) and i-vector + PLDA (23.5%) baselines.
• The model significantly outperforms traditional methods on short utterances (1-second), demonstrating strong performance in low-resource settings.
• For longer utterances, traditional i-vector-based models still surpass the LSTM model, indicating limitations in capturing long-term speaker dependencies.
• The model's performance is sensitive to utterance duration, with a clear performance gap favoring it in short utterance regimes.
• The use of end-to-end training with a contrastive loss function enables effective joint optimization of background and speaker modeling.
• The dynamic pair selection algorithm improves training efficiency by focusing on hard negative samples, enhancing model discrimination.