[Paper Review] An online sequence-to-sequence model for noisy speech recognition

Chung‐Cheng Chiu, Dieterich Lawson+5 more|arXiv (Cornell University)|Jun 16, 2017

Speech Recognition and Synthesis9 references3 citations

TL;DR

This paper proposes an improved online sequence-to-sequence model using a modified policy gradient training method for end-to-end speech recognition, enabling real-time inference with causal attention. It achieves strong performance on noisy, single-channel mixed speech (Multi-TIMIT), reducing phoneme error rate to 25.9% at 10% mixing proportion, demonstrating robustness to interference while maintaining low latency.

ABSTRACT

Generative models have long been the dominant approach for speech recognition. The success of these models however relies on the use of sophisticated recipes and complicated machinery that is not easily accessible to non-practitioners. Recent innovations in Deep Learning have given rise to an alternative - discriminative models called Sequence-to-Sequence models, that can almost match the accuracy of state of the art generative models. While these models are easy to train as they can be trained end-to-end in a single step, they have a practical limitation that they can only be used for offline recognition. This is because the models require that the entirety of the input sequence be available at the beginning of inference, an assumption that is not valid for instantaneous speech recognition. To address this problem, online sequence-to-sequence models were recently introduced. These models are able to start producing outputs as data arrives, and the model feels confident enough to output partial transcripts. These models, like sequence-to-sequence are causal - the output produced by the model until any time, $t$, affects the features that are computed subsequently. This makes the model inherently more powerful than generative models that are unable to change features that are computed from the data. This paper highlights two main contributions - an improvement to online sequence-to-sequence model training, and its application to noisy settings with mixed speech from two speakers.

Motivation & Objective

• Address the limitation of standard sequence-to-sequence models that require full input before generating output, making them unsuitable for real-time applications.
• Develop a causal, online inference mechanism that generates transcriptions incrementally as audio arrives.
• Improve training stability and performance of online sequence-to-sequence models using a modified policy gradient approach with binary stochastic variables.
• Evaluate the model on noisy, single-channel mixed speech from two speakers, simulating realistic interference scenarios.
• Demonstrate the model’s ability to recognize the target speaker in mixed audio while maintaining low latency and high accuracy.

Proposed method

• Employ a sequence-to-sequence architecture with soft attention, where the decoder generates output tokens autoregressively using context vectors from encoder states.
• Introduce binary stochastic variables to determine when the model should emit a token, enabling online inference by allowing partial outputs during input processing.
• Train the emission decision mechanism using a policy gradient method, with a modified objective to improve training stability and convergence.
• Use unidirectional LSTMs in both encoder and decoder, with a 2-layer architecture and 256 units per layer, for efficient sequence modeling.
• Apply the model to a new dataset, Multi-TIMIT, created by mixing male and female TIMIT utterances at varying volume ratios (10%, 25%, 50%).
• Use standard log Mel filter bank features (123-dimensional) and train end-to-end with cross-entropy loss on token-level predictions.

Experimental results

Research questions

• RQ1 Can an online sequence-to-sequence model with improved training be effectively applied to real-time speech recognition under noisy conditions?
• RQ2 How does the model perform when faced with single-channel mixed speech from two speakers, particularly as interference levels increase?
• RQ3 Does the causal, autoregressive nature of the model allow it to adapt to degraded input without requiring full input sequence?
• RQ4 Can the model maintain low latency while achieving competitive accuracy on noisy speech compared to standard models?
• RQ5 How does the model’s emission timing differ between clean and noisy inputs, and what does this imply about its robustness?

Key findings

• The proposed model achieved a phoneme error rate (PER) of 25.9% on Multi-TIMIT with 10% mixing proportion, outperforming CTC (27.3%) and RNN-Transducer (25.7%) under the same conditions.
• At 25% mixing, the model achieved 32.5% PER, showing a moderate degradation compared to clean data, but still outperforming baseline models.
• At 50% mixing, the model reached 42.9% PER, which is still better than the RNN-Transducer (48.9%), indicating robustness to high interference.
• The model emits tokens later for noisy Multi-TIMIT inputs compared to clean TIMIT, suggesting adaptive timing to handle degraded input.
• Overfitting worsened when each target utterance was paired with multiple confounding speakers, indicating the model’s capacity to memorize specific input-output pairs.
• The model’s causal architecture enables real-time inference and supports multiple transcript generation, enhancing its flexibility for practical deployment.