[Paper Review] Unsupervised Speaker Adaptation using Attention-based Speaker Memory for End-to-End ASR

Leda Sarı, Niko Moritz+2 more|arXiv (Cornell University)|Feb 14, 2020

Speech Recognition and Synthesis30 references4 citations

TL;DR

This paper proposes an unsupervised speaker adaptation method for end-to-end ASR using an attention-based speaker memory inspired by the neural Turing machine. The model stores speaker i-vectors in a memory bank and uses an attention mechanism to dynamically retrieve and combine relevant speaker representations (M-vectors) at frame level, which are appended to acoustic features or hidden layers. The method achieves comparable or better word error rates than i-vector baselines, especially in speaker change scenarios, without requiring test-time speaker embeddings or adaptation data.

ABSTRACT

We propose an unsupervised speaker adaptation method inspired by the neural Turing machine for end-to-end (E2E) automatic speech recognition (ASR). The proposed model contains a memory block that holds speaker i-vectors extracted from the training data and reads relevant i-vectors from the memory through an attention mechanism. The resulting memory vector (M-vector) is concatenated to the acoustic features or to the hidden layer activations of an E2E neural network model. The E2E ASR system is based on the joint connectionist temporal classification and attention-based encoder-decoder architecture. M-vector and i-vector results are compared for inserting them at different layers of the encoder neural network using the WSJ and TED-LIUM2 ASR benchmarks. We show that M-vectors, which do not require an auxiliary speaker embedding extraction system at test time, achieve similar word error rates (WERs) compared to i-vectors for single speaker utterances and significantly lower WERs for utterances in which there are speaker changes.

Motivation & Objective

• To address the degradation in end-to-end ASR performance when test speakers differ from training speakers.
• To develop an unsupervised, frame-level speaker adaptation method that does not require speaker labels or i-vector computation at test time.
• To investigate the effectiveness of adapting intermediate layers in the encoder, not just input features, for speaker adaptation.
• To evaluate the robustness of the method under speaker change conditions, a common real-world challenge.
• To explore the potential of memory-augmented networks for speaker representation learning in end-to-end ASR.

Proposed method

• The model uses a memory bank that stores speaker i-vectors extracted from training data.
• An attention mechanism computes weights over the memory items based on the current acoustic context, producing a weighted combination of i-vectors known as the M-vector.
• The M-vector is concatenated to either the input acoustic features or the output of an intermediate encoder layer before being projected into the network.
• The method is applied to a joint CTC and attention-based encoder-decoder E2E ASR architecture, trained on WSJ and TED-LIUM2 datasets.
• The memory read operation is inspired by the neural Turing machine, enabling dynamic, context-dependent speaker representation adaptation.
• The approach is unsupervised: no test-time speaker information or i-vector extraction is needed, as the memory is fixed from training data.

Experimental results

Research questions

• RQ1 Can an attention-based memory mechanism improve end-to-end ASR performance without requiring speaker labels at test time?
• RQ2 Does adapting intermediate encoder layers yield better performance than adapting only the input features in E2E ASR?
• RQ3 How does the proposed M-vector method compare to i-vector-based adaptation in terms of word error rate, especially under speaker change conditions?
• RQ4 Can a memory-augmented speaker adaptation mechanism achieve robustness to speaker changes without explicit adaptation data?
• RQ5 What is the impact of using speaker-level versus utterance-level i-vectors on adaptation performance, and how does the M-vector method compare?

Key findings

• The M-vector method achieved a test set WER of 11.0% on TED-LIUM2 when adapting the third encoder layer, matching the best i-vector performance.
• For utterances with synthetic speaker changes, the M-vector system achieved a 4% absolute reduction in WER (15.9% → 11.9%) compared to the i-vector baseline.
• The M-vector approach outperformed utterance-level i-vector systems by 6.8% relatively on the test set (11.8% → 11.0% WER).
• Adapting the third encoder layer yielded the best performance for both i-vectors and M-vectors, indicating that intermediate layer adaptation is more effective than input-level adaptation.
• The M-vector method achieved similar performance to speaker-level i-vectors without requiring any test-time i-vector computation, demonstrating its unsupervised and efficient nature.
• The method showed superior robustness to speaker changes, confirming the advantage of frame-level adaptation enabled by the attention-based memory read mechanism.