[Paper Review] Phase Conductor on Multi-layered Attentions for Machine Comprehension

Rui Liu, Wei Wei+2 more|arXiv (Cornell University)|Oct 28, 2017

Topic Modeling12 references18 citations

TL;DR

This paper proposes PhaseCond, a multi-phase, multi-layered attention model for machine comprehension that enhances question answering by separating question-aware passage representation and evidence propagation into distinct phases. It improves attention mechanisms by using independent and shared encoders for questions and passages, achieving state-of-the-art performance on SQuAD with 71.85% EM and 81.13% F1.

ABSTRACT

Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and self-matching attention model. Our research proposes phase conductor (PhaseCond) for attention models in two meaningful ways. First, PhaseCond, an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information flow. Second, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspectives. We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model significantly outperforms both state-of-the-art single-layered and multiple-layered attention models. We deepen our results with new findings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.

Motivation & Objective

• To address the limitation of single-phase attention models in capturing long-range dependencies and propagating answer evidence effectively in machine comprehension.
• To investigate whether separating question-aware representation and evidence propagation into distinct phases improves model performance and interpretability.
• To explore the impact of using multiple, diverse question representations (independent and shared encoders) in attention mechanisms for better alignment and feature learning.
• To analyze dynamic changes in attention weights across multiple layers, revealing insights into information flow and degradation in stacked attention mechanisms.

Proposed method

• PhaseCond introduces a two-phase architecture: the question-aware passage representation phase (with stacked question-passage attention layers) and the evidence propagation phase (with stacked self-attention layers).
• Each phase includes fusion layers—outer fusion for concatenating representations across layers in the question-passage phase, and inner fusion for regulating information flow in self-attention layers.
• An improved dot-product attention function is proposed, using three distinct embedding streams: an independent question encoder, a weight-shared question encoder, and a weight-shared passage encoder.
• The model uses a multi-head dot-product attention mechanism where the query is derived from the shared question representation and keys from the passage, with context-aware alignment through learned attention weights.
• The architecture supports stacking multiple layers in each phase, enabling iterative refinement of passage representations and propagation of answer-relevant evidence.
• Visualizations and ablation studies are conducted on SQuAD to analyze attention dynamics across layers, particularly focusing on weight concentration and degradation patterns.

Experimental results

Research questions

• RQ1 Does separating question-aware passage representation and evidence propagation into distinct phases improve performance on machine comprehension tasks?
• RQ2 How does using multiple, perspective-specific question representations (independent and shared) affect attention alignment and model accuracy compared to single-encoder approaches?
• RQ3 What are the dynamic changes in attention weights across multiple layers in question-passage and self-attention phases, and how do they relate to model performance?
• RQ4 Why does adding more layers in the question-passage attention phase lead to performance degradation, while deeper self-attention layers improve results?
• RQ5 To what extent do attention matrices reveal meaningful patterns of evidence concentration and propagation in complex passages?

Key findings

• PhaseCond achieves 71.85% EM and 81.13% F1 on the SQuAD benchmark, significantly outperforming both single-layered and multi-layered attention models.
• Adding a second layer to the question-passage attention phase degrades performance (EM drops from 72.05 to 71.85), indicating that repeated alignment with the same question representation causes overfitting to the question and reduces representation diversity.
• The second layer of self-attention produces sharper alignment weights than the first, suggesting that deeper self-attention layers enhance evidence concentration and propagation through the passage.
• Visualizations show that after the first question-passage attention layer, passage words become increasingly aligned with the question, leading to indistinguishable attention patterns in the second layer, which explains performance degradation.
• In the self-attention phase, attention weights become more focused—e.g., 'Denver Broncos' becomes more concentrated on 'Carolina Panthers' in the second layer, indicating effective propagation of answer-relevant evidence.
• The model reveals that evidence propagation is more effective when done through self-attention layers than through repeated question-passage attention, highlighting the importance of internal passage refinement.