[Paper Review] Label-Synchronous Neural Transducer for End-to-End ASR

Keqi Deng, Philip C. Woodland|arXiv (Cornell University)|Jul 6, 2023

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes the Label-Synchronous Neural Transducer (LS-Transducer), a streaming end-to-end ASR model that eliminates blank token prediction by generating label-level encoder representations via an Auto-regressive Integrate-and-Fire (AIF) mechanism, enabling direct adaptation of the prediction network using text-only data. The LS-Transducer achieves up to 19% relative WER reduction over standard neural transducers in cross-domain settings, with significant gains in both intra- and cross-domain scenarios.

ABSTRACT

Neural transducers provide a natural way of streaming ASR. However, they augment output sequences with blank tokens which leads to challenges for domain adaptation using text data. This paper proposes a label-synchronous neural transducer (LS-Transducer), which extracts a label-level encoder representation before combining it with the prediction network output. Hence blank tokens are no longer needed and the prediction network can be easily adapted using text data. An Auto-regressive Integrate-and-Fire (AIF) mechanism is proposed to generate the label-level encoder representation while retaining the streaming property. In addition, a streaming joint decoding method is designed to improve ASR accuracy. Experiments show that compared to standard neural transducers, the proposed LS-Transducer gave a 10% relative WER reduction (WERR) for intra-domain Librispeech-100h data, as well as 17% and 19% relative WERRs on cross-domain TED-LIUM2 and AESRC2020 data with an adapted prediction network.

Motivation & Objective

• To address the challenge of domain adaptation in end-to-end ASR, especially when target-domain labeled data is scarce.
• To eliminate the need for blank token prediction in neural transducers, which hinders effective language model adaptation using text-only data.
• To enable the prediction network to function as a standalone language model by decoupling it from blank token generation.
• To maintain streaming inference capability while improving alignment accuracy and robustness to acoustic boundary errors.
• To design a streaming joint decoding method that enhances ASR accuracy without sacrificing latency.

Proposed method

• The LS-Transducer replaces blank token prediction by extracting a label-level encoder representation before fusion with the prediction network output.
• An Auto-regressive Integrate-and-Fire (AIF) mechanism generates the label-level representation by accumulating frame-level encoder outputs using learnable attention weights.
• The AIF mechanism ensures monotonic alignment and splits weights at threshold 1.0 to emit a label representation, improving robustness to inaccurate boundaries.
• The prediction network is decoupled from blank prediction and can be pre-trained or fine-tuned using only text data, mimicking a standard language model.
• A streaming joint decoding method is introduced to refine the search space during beam search, improving accuracy without increasing latency.
• The model retains streaming capability by processing input frames in real time and emitting labels synchronously with the label-level representation.

Fig. 1 : Illustration of the CIF [ 14 ] mechanism. $\bm{\oplus}$ and $\bm{\otimes}$ denote addition and multiplication. $\mathbf{E}=(\bm{e_{1}},\cdots,\bm{e_{T}})$ represents the encoder output and $\bm{\alpha}=(\alpha_{1},\cdots,\alpha_{T})$ denotes predicted weights whose example values $(0.2,0.9,

Experimental results

Research questions

• RQ1 Can a neural transducer be redesigned to eliminate blank token prediction while preserving streaming inference and alignment accuracy?
• RQ2 How can a label-level encoder representation be generated efficiently and robustly for streaming ASR?
• RQ3 Can the prediction network be effectively adapted using only text data when blank tokens are no longer predicted?
• RQ4 Does the proposed AIF mechanism outperform existing methods like Continuous Integrate-and-Fire (CIF) in terms of WER and robustness?
• RQ5 Can the LS-Transducer achieve superior performance in both intra-domain and cross-domain ASR scenarios compared to standard and factorised neural transducer models?

Key findings

• The LS-Transducer achieved a 10% relative WER reduction on intra-domain LibriSpeech-100h data compared to standard neural transducers.
• On cross-domain TED-LIUM2 and AESRC2020 data, the LS-Transducer achieved 17% and 19% relative WER reductions, respectively, with an adapted prediction network.
• The AIF mechanism outperformed the baseline CIF method, reducing WER by 2.8 points on clean LibriSpeech dev set and 2.4 points on other set.
• Pre-training the prediction network in standard Transformer-Pred T-T models degraded performance, while the LS-Transducer naturally leveraged pre-trained language models without degradation.
• The LS-Transducer outperformed both HAT and factorised T-T models by 8.1% to 15.4% relative WER reduction across all test sets, with statistical significance at the 0.1% level.
• The proposed streaming joint decoding method improved accuracy by refining the search space during beam search, contributing to overall performance gains.

Fig. 2 : Illustration of the proposed LS-Transducer. FC is a fully connected layer. $L$ and $V$ are the length and vocabulary size of label-level logits. $\bm{\oplus}$ denotes addition.