[Paper Review] Transformer with Bidirectional Decoder for Speech Recognition

Xi Chen, Songyang Zhang+3 more|arXiv (Cornell University)|Aug 11, 2020

Speech Recognition and Synthesis24 references4 citations

TL;DR

This paper proposes a Speech Transformer with Bidirectional Decoder (STBD) that jointly predicts speech tokens in both left-to-right and right-to-left directions using a shared encoder and dual unidirectional decoders. By employing a bidirectional beam search during inference and selecting the highest-scoring hypothesis from both directions, STBD achieves a 3.6% relative CER reduction over unidirectional baselines, with the STBD-Big model reaching 6.64% CER on AISHELL-1 without language model rescoring or data augmentation.

ABSTRACT

Attention-based models have made tremendous progress on end-to-end automatic speech recognition(ASR) recently. However, the conventional transformer-based approaches usually generate the sequence results token by token from left to right, leaving the right-to-left contexts unexploited. In this work, we introduce a bidirectional speech transformer to utilize the different directional contexts simultaneously. Specifically, the outputs of our proposed transformer include a left-to-right target, and a right-to-left target. In inference stage, we use the introduced bidirectional beam search method, which can not only generate left-to-right candidates but also generate right-to-left candidates, and determine the best hypothesis by the score. To demonstrate our proposed speech transformer with a bidirectional decoder(STBD), we conduct extensive experiments on the AISHELL-1 dataset. The results of experiments show that STBD achieves a 3.6\% relative CER reduction(CERR) over the unidirectional speech transformer baseline. Besides, the strongest model in this paper called STBD-Big can achieve 6.64\% CER on the test set, without language model rescoring and any extra data augmentation strategies.

Motivation & Objective

• To address the limitation of conventional left-to-right autoregressive speech transformers that ignore right-to-left context in sequence generation.
• To mitigate exposure bias in autoregressive decoding by incorporating bidirectional context during both training and inference.
• To improve automatic speech recognition performance without relying on external language models or data augmentation.
• To investigate the effectiveness of bidirectional targets and bidirectional beam search in end-to-end ASR.

Proposed method

• The model uses a shared encoder and two unidirectional decoders—one for left-to-right (L2R) and one for right-to-left (R2L) sequence generation—allowing simultaneous prediction in both directions.
• During training, the model optimizes both L2R and R2L targets using cross-entropy loss, enabling the network to learn from bidirectional context.
• Inference employs a bidirectional beam search that maintains separate beams for L2R and R2L decoding, dynamically updating candidates from both directions.
• The final hypothesis is selected based on the highest overall score among all L2R and R2L candidates, improving robustness and reducing error propagation.
• The model shares parameters between the two decoders, maintaining efficiency while enabling bidirectional context learning.
• The method is evaluated on the AISHELL-1 dataset using character-level ASR, with ablation studies on beam size and decoding strategy.

(a)

Experimental results

Research questions

• RQ1 Can joint prediction in both left-to-right and right-to-left directions improve automatic speech recognition performance?
• RQ2 Does bidirectional beam search lead to better hypothesis selection compared to unidirectional beam search?
• RQ3 How effective is the use of bidirectional targets during training in reducing error accumulation during inference?
• RQ4 To what extent does bidirectional decoding improve robustness when no language model or data augmentation is used?

Key findings

• The STBD model achieves a 3.6% relative CER reduction compared to the unidirectional speech transformer baseline on the AISHELL-1 test set.
• The STBD-Big model reaches 6.64% CER on the AISHELL-1 test set without language model rescoring or data augmentation.
• The bidirectional beam search method improves performance over standard unidirectional beam search, even when using the same trained model.
• 49.4% of test sentences were decoded more accurately by the right-to-left decoder, indicating complementary strengths between directions.
• Ablation studies show that increasing beam size improves performance, with gains saturating beyond beam size 2.
• Visualization of attention alignment confirms that the L2R decoder attends along a diagonal from bottom-left to top-right, while the R2L decoder attends from bottom-right to top-left, validating the directional behavior.

(b)