[Paper Review] Task-Specific Attentive Pooling of Phrase Alignments Contributes to Sentence Matching

Wenpeng Yin, Hinrich Schütze|arXiv (Cornell University)|Jan 9, 2017

Topic Modeling15 citations

TL;DR

This paper proposes a task-specific attentive pooling mechanism using Gated Recurrent Units (GRUs) to learn phrase representations of arbitrary granularity and dynamically weigh phrase alignments based on their semantic intensity. The method improves sentence matching by focusing on weaker alignments in textual entailment and stronger alignments in answer selection, outperforming existing models on both tasks through flexible, learnable attention mechanisms.

ABSTRACT

This work studies comparatively two typical sentence matching tasks: textual entailment (TE) and answer selection (AS), observing that weaker phrase alignments are more critical in TE, while stronger phrase alignments deserve more attention in AS. The key to reach this observation lies in phrase detection, phrase representation, phrase alignment, and more importantly how to connect those aligned phrases of different matching degrees with the final classifier. Prior work (i) has limitations in phrase generation and representation, or (ii) conducts alignment at word and phrase levels by handcrafted features or (iii) utilizes a single framework of alignment without considering the characteristics of specific tasks, which limits the framework's effectiveness across tasks. We propose an architecture based on Gated Recurrent Unit that supports (i) representation learning of phrases of arbitrary granularity and (ii) task-specific attentive pooling of phrase alignments between two sentences. Experimental results on TE and AS match our observation and show the effectiveness of our approach.

Motivation & Objective

• To investigate how different intensities of phrase alignments contribute to sentence matching in distinct NLP tasks.
• To address the limitation of existing models that assume stronger alignments are always more important, which fails in tasks like textual entailment.
• To develop a deep learning framework that learns phrase representations of arbitrary granularity and supports task-specific attention over alignments.
• To design a flexible attention mechanism that can adaptively prioritize either strong or weak alignments based on task requirements.
• To demonstrate the effectiveness of this approach on two major sentence matching tasks: textual entailment and answer selection.

Proposed method

• Uses Gated Recurrent Units (GRUs) to learn dense, context-aware representations for phrases of arbitrary length and granularity.
• Applies a two-step attentive pooling mechanism: first, attention weights are computed over aligned phrase pairs based on their matching scores; second, k-min or k-max pooling is applied to select the most relevant phrases.
• Employs task-specific pooling strategies: k-min pooling for textual entailment (to emphasize weak alignments) and k-max pooling for answer selection (to emphasize strong alignments).
• Integrates phrase representations into a unified feature vector with three components: representation similarity, semantic similarity (cosine scores), and additional features like sentence length and word count.
• Uses a soft attention mechanism that allows the model to focus on semantically relevant phrases without relying on handcrafted features or fixed filter sizes.
• Treats phrase alignment as a learnable, differentiable process rather than a pre-defined or heuristic one, enabling end-to-end training.

Experimental results

Research questions

• RQ1 Does the importance of phrase alignment intensity vary across different sentence matching tasks?
• RQ2 Can a single neural architecture dynamically adapt to prioritize either strong or weak alignments depending on the task?
• RQ3 How does learning phrase representations of arbitrary granularity affect alignment quality and downstream performance?
• RQ4 To what extent does task-specific attentive pooling improve performance over fixed attention mechanisms in textual entailment and answer selection?
• RQ5 Can the model effectively identify and focus on semantically critical phrases even when they are not exact lexical matches?

Key findings

• The proposed GRU-based model with task-specific attentive pooling outperforms baseline models on both textual entailment and answer selection tasks.
• k-min-max pooling (a variant of k-min and k-max pooling) achieves the best performance, suggesting that selective attention to critical phrases improves generalization.
• Visualizations confirm that the model correctly identifies key phrases such as 'kids' corresponding to 'young boys' and 'smiling nearby' corresponding to 'a smile', even when surface forms differ.
• The model reduces reliance on surface-level matches by assigning low attention to phrases containing function words like 'and' or 'near', which alter semantic meaning.
• Pooling size k has a significant impact: values above 4 yield competitive results, but performance drops with larger k, indicating optimal selection of top-k phrases is crucial.
• The method achieves superior results compared to ABCNN and AP-CNN, primarily due to its ability to learn and compare phrases of arbitrary length in a shared embedding space.