[Paper Review] Deja-vu: Double Feature Presentation in Deep Transformer Networks.

Andros Tjandra, Chunxi Liu+6 more|arXiv (Cornell University)|Oct 23, 2019

Generative Adversarial Networks and Image Synthesis15 references4 citations

TL;DR

This paper proposes a deep Transformer architecture that reuses input features at multiple layers via intermediate model heads and loss functions, enabling the network to re-examine raw features in light of evolving hypotheses. The method achieves 10–20% relative WER improvement on Librispeech and 3.2–13% on a large-scale video dataset by combining multi-depth feature re-use with layer-wise supervision.

ABSTRACT

Deep acoustic models typically receive features in the first layer of the network, and process increasingly abstract representations in the subsequent layers. Here, we propose to feed the input features at multiple depths in the acoustic model. As our motivation is to allow acoustic models to re-examine their input features in light of partial hypotheses we introduce intermediate model heads and loss function. We study this architecture in the context of deep Transformer networks, and we use an attention mechanism over both the previous layer activations and the input features. To train this model's intermediate output hypothesis, we apply the objective function at each layer right before feature re-use. We find that the use of such intermediate losses significantly improves performance by itself, as well as enabling input feature re-use. We present results on both Librispeech, and a large scale video dataset, with relative improvements of 10 - 20% for Librispeech and 3.2 - 13% for videos.

Motivation & Objective

• To improve deep Transformer acoustic models by enabling re-examination of input features at multiple network depths.
• To address the limitation of single-pass feature processing in standard Transformers, where early features are not revisited.
• To explore whether intermediate supervision and feature re-use can enhance representation learning and model performance.
• To evaluate the effectiveness of the proposed architecture on diverse ASR benchmarks, including Librispeech and a large-scale video dataset.

Proposed method

• Introduce intermediate model heads at multiple layers to generate hypotheses based on both previous layer activations and original input features.
• Apply a loss function at each intermediate layer right before feature re-use to supervise the model’s evolving predictions.
• Use an attention mechanism that attends over both the previous layer’s activations and the input features to enable contextualized re-examination.
• Train the network end-to-end with multi-depth supervision, allowing gradients to flow through intermediate outputs.
• Design the architecture to allow feature re-use without modifying the main model’s forward pass, maintaining computational efficiency.

Experimental results

Research questions

• RQ1 Can reusing input features at multiple depths improve performance in deep Transformer ASR models?
• RQ2 Does introducing intermediate supervision via layer-wise losses enhance model accuracy and generalization?
• RQ3 How does attention over both previous activations and input features affect representation quality?
• RQ4 What is the impact of this architecture on diverse ASR datasets, including those with noisy or complex audio?

Key findings

• The proposed architecture achieves a 10–20% relative WER improvement on the Librispeech benchmark compared to standard Transformers.
• On a large-scale video dataset, the model shows a 3.2–13% relative WER reduction, demonstrating strong generalization across domains.
• Intermediate losses alone significantly improve performance, indicating that multi-depth supervision enhances learning dynamics.
• Feature re-use via attention over input and hidden states enables the model to refine hypotheses using raw input in context.