[Paper Review] Attentive Convolution

Wenpeng Yin, Hinrich Schütze|arXiv (Cornell University)|Oct 2, 2017

Topic Modeling10 citations

TL;DR

This paper proposes ATTCONV, a novel attentive convolution network that integrates attention mechanisms directly into the convolution operation, extending local receptive fields with nonlocal context from distant or external text. By fusing attention with convolution rather than pooling, ATTCONV achieves superior sentence representation learning, outperforming both attentive pooling and competitive RNN-based models across zero-, single-, and multiple-context NLP tasks.

ABSTRACT

In NLP, convolutional neural networks (CNNs) have benefited less than recurrent neural networks (RNNs) from attention mechanisms. We hypothesize that this is because the attention in CNNs has been mainly implemented as attentive pooling (i.e., it is applied to pooling) rather than as attentive convolution (i.e., it is integrated into convolution). Convolution is the differentiator of CNNs in that it can powerfully model the higher-level representation of a word by taking into account its local fixed-size context in the input text t^x. In this work, we propose an attentive convolution network, ATTCONV. It extends the context scope of the convolution operation, deriving higher-level features for a word not only from local context, but also information extracted from nonlocal context by the attention mechanism commonly used in RNNs. This nonlocal context can come (i) from parts of the input text t^x that are distant or (ii) from extra (i.e., external) contexts t^y. Experiments on sentence modeling with zero-context (sentiment analysis), single-context (textual entailment) and multiple-context (claim verification) demonstrate the effectiveness of ATTCONV in sentence representation learning with the incorporation of context. In particular, attentive convolution outperforms attentive pooling and is a strong competitor to popular attentive RNNs.

Motivation & Objective

• To address the limited effectiveness of attention mechanisms in CNNs compared to RNNs in NLP.
• To investigate whether integrating attention directly into the convolution operation—rather than applying it post-convolution—improves representation learning.
• To extend the context scope of CNNs beyond local receptive fields by incorporating nonlocal information via attention.
• To evaluate ATTCONV's performance in diverse NLP tasks involving zero, single, and multiple contexts.
• To demonstrate that attentive convolution can rival or surpass attentive pooling and attentive RNNs in sentence modeling.

Proposed method

• ATTCONV replaces standard convolution with an attentive convolution mechanism that computes weighted features using query, key, and value vectors derived from input tokens.
• The attention mechanism computes context-aware weights over distant or external text, allowing the model to attend to nonlocal information during feature extraction.
• The model integrates these attention-enhanced features into the convolution operation, enabling dynamic context expansion beyond fixed-size local windows.
• It supports multiple context types: zero-context (e.g., sentiment analysis), single-context (e.g., textual entailment), and multiple-context (e.g., claim verification).
• The architecture maintains the inductive bias of CNNs for local pattern learning while augmenting it with global context awareness via attention.
• The method is evaluated end-to-end on sentence modeling tasks with varying context availability.

Experimental results

Research questions

• RQ1 Can integrating attention directly into the convolution operation improve sentence representation learning in CNNs?
• RQ2 How does attentive convolution compare to attentive pooling in modeling sentence-level representations?
• RQ3 Does ATTCONV outperform or rival attention-enhanced RNNs in tasks with varying context availability?
• RQ4 Can ATTCONV effectively leverage nonlocal or external context to improve performance in zero-, single-, and multiple-context NLP tasks?
• RQ5 What is the contribution of nonlocal context from distant or external sources to the final representation in ATTCONV?

Key findings

• ATTCONV outperforms attentive pooling across all evaluated tasks, demonstrating the superiority of integrating attention into convolution rather than applying it after.
• The model achieves competitive performance compared to state-of-the-art attentive RNNs, particularly in tasks requiring long-range or external context integration.
• In claim verification tasks with multiple contexts, ATTCONV effectively leverages external evidence, showing robustness in complex reasoning scenarios.
• The integration of nonlocal context through attention significantly improves representation quality, especially in zero-context and single-context settings.
• ATTCONV maintains the inductive bias of CNNs for local pattern detection while extending their context scope via attention, leading to improved generalization.
• The ablation study confirms that the attentive convolution mechanism is the key contributor to performance gains, not just attention applied post-convolution.