[Paper Review] Learning Context-Sensitive Convolutional Filters for Text Processing

Dinghan Shen, Martin Renqiang Min+2 more|arXiv (Cornell University)|Sep 25, 2017

Topic Modeling30 references4 citations

TL;DR

This paper proposes Adaptive Context-sensitive CNN (ACNN), a novel framework that uses a meta network to generate context-sensitive convolutional filters tailored to each input sentence, enabling dynamic, input-aware feature extraction. The method consistently outperforms standard and attention-augmented CNNs across four NLP tasks—ontology classification, sentiment analysis, answer sentence selection, and paraphrase identification—demonstrating improved modeling flexibility and performance through adaptive filter generation and bidirectional sentence pair modeling.

ABSTRACT

Convolutional neural networks (CNNs) have recently emerged as a popular building block for natural language processing (NLP). Despite their success, most existing CNN models employed in NLP share the same learned (and static) set of filters for all input sentences. In this paper, we consider an approach of using a small meta network to learn context-sensitive convolutional filters for text processing. The role of meta network is to abstract the contextual information of a sentence or document into a set of input-aware filters. We further generalize this framework to model sentence pairs, where a bidirectional filter generation mechanism is introduced to encapsulate co-dependent sentence representations. In our benchmarks on four different tasks, including ontology classification, sentiment analysis, answer sentence selection, and paraphrase identification, our proposed model, a modified CNN with context-sensitive filters, consistently outperforms the standard CNN and attention-based CNN baselines. By visualizing the learned context-sensitive filters, we further validate and rationalize the effectiveness of proposed framework.

Motivation & Objective

• To address the limitation of static, fixed convolutional filters in standard CNNs for NLP, which fail to capture context-dependent semantic variations.
• To enhance modeling flexibility by learning input-aware filters that adapt to the contextual content of each sentence.
• To generalize the framework to sentence pair matching by introducing a bidirectional filter generation mechanism for co-dependent representations.
• To validate the effectiveness of context-sensitive filters through empirical benchmarks and visualization of learned filters.

Proposed method

• A meta network is trained to generate a set of convolutional filters conditioned on the input sentence's contextual representation.
• The generated filters are applied to the input embedding matrix, allowing filter weights to vary per input, enabling context-sensitive feature abstraction.
• For sentence pair modeling, a bidirectional filter generation mechanism is introduced to jointly generate filters for two sentences, capturing interdependent representations.
• The framework is integrated into a CNN architecture and evaluated on multiple NLP tasks, with ablation studies confirming the contribution of the adaptive mechanism.
• The model is further enhanced with multi-perspective attention layers in some experiments to assess complementarity with attention mechanisms.
• Filter visualization via t-SNE is used to analyze learned filters and validate their clustering by label, indicating task-specific feature learning.

Experimental results

Research questions

• RQ1 Can context-sensitive convolutional filters improve performance on NLP tasks compared to standard CNNs with fixed filters?
• RQ2 How does the meta-network-based filter generation mechanism enhance feature abstraction in a context-aware manner?
• RQ3 Does the bidirectional filter generation mechanism for sentence pairs lead to better performance in text matching tasks?
• RQ4 To what extent does the proposed framework improve reasoning capabilities on complex question types, such as 'How' questions?
• RQ5 Can the learned filters be meaningfully visualized to demonstrate their sensitivity to input context and label structure?

Key findings

• The ACNN model outperforms standard CNN and attention-based CNN baselines across all four evaluated tasks: ontology classification, sentiment analysis, answer sentence selection, and paraphrase identification.
• On the WikiQA dataset, the two-way AdaQA model achieved a MAP score of 0.8516, significantly outperforming the Siamese-CNN baseline (0.7960) and Multi-Perspective-CNN (0.8138).
• When combined with multi-perspective attention, the two-way AdaQA model achieved a MAP score of 0.8794 on WikiQA, further improving performance.
• On the Quora Question Pairs dataset, the AdaQA model achieved superior results compared to attentive pooling networks and ABCNN baselines, confirming its effectiveness in sentence matching.
• The model showed the largest relative improvement over standard CNNs on 'How' questions, with a MAP score of 0.579, outperforming NASM (0.524), indicating enhanced reasoning capability.
• t-SNE visualization of filters showed that filters for documents of the same ontology label cluster together, validating that the model learns label-specific, context-sensitive features.