[Paper Review] Attentive Pooling Networks

Cícero Nogueira dos Santos, Ming Tan+2 more|arXiv (Cornell University)|Feb 11, 2016

Topic Modeling25 references324 citations

TL;DR

Attentive Pooling (AP) introduces a two-way attention mechanism that makes the pooling layer pair-aware, improving CNNs and biLSTMs for answer selection across three datasets and achieving state-of-the-art results without handcrafted features.

ABSTRACT

In this work, we propose Attentive Pooling (AP), a two-way attention mechanism for discriminative model training. In the context of pair-wise ranking or classification with neural networks, AP enables the pooling layer to be aware of the current input pair, in a way that information from the two input items can directly influence the computation of each other's representations. Along with such representations of the paired inputs, AP jointly learns a similarity measure over projected segments (e.g. trigrams) of the pair, and subsequently, derives the corresponding attention vector for each input to guide the pooling. Our two-way attention mechanism is a general framework independent of the underlying representation learning, and it has been applied to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in our studies. The empirical results, from three very different benchmark tasks of question answering/answer selection, demonstrate that our proposed models outperform a variety of strong baselines and achieve state-of-the-art performance in all the benchmarks.

Motivation & Objective

• Motivate the need for discriminative pair-wise matching in neural networks beyond one-way attention.
• Propose Attentive Pooling (AP) to jointly learn representations and pair-wise similarity.
• Demonstrate AP as a general mechanism applicable to CNNs and RNNs for answer selection.
• Show that AP improves robustness to long inputs and reduces need for many convolutional filters.

Proposed method

• Define a two-way attention where a learned similarity over projected segments (e.g., trigrams or hidden states) guides pooling.
• Compute an inter-item interaction matrix G via G = tanh(Q^T U A) where Q and A are pairwise representations (from CNN or biLSTM).
• Derive attention vectors for both inputs through column-wise/row-wise pooling and softmax to obtain r^q and r^a.
• Score pairs by cosine similarity between r^q and r^a and train with a hinge ranking loss.
• Apply AP to both AP-CNN and AP-biLSTM architectures and compare to QA-CNN and QA-biLSTM.
• Train with SGD and use negative sampling (50 per question, max-scoring negative for updates).

Experimental results

Research questions

• RQ1 Does two-way attentive pooling improve discriminative training for pair-wise QA tasks compared to one-way attention or no attention?
• RQ2 Can AP be effectively integrated with both CNNs and RNNs (biLSTM) for answer selection?
• RQ3 Does AP enhance robustness to longer inputs and reduce model complexity (fewer filters) while maintaining or improving accuracy?
• RQ4 How does AP perform across diverse datasets with varying lengths and domains (InsuranceQA, TREC-QA, WikiQA)?

Key findings

• AP-CNN and AP-biLSTM outperform their non-attentive counterparts on all three datasets.
• AP-CNN achieves state-of-the-art results on InsuranceQA, TREC-QA, and strong results on WikiQA.
• AP-based models require fewer convolutional filters and training time can be faster (e.g., AP-CNN with 400 filters vs QA-CNN with 4000).
• AP improves robustness to long answers, with accuracy stabilizing for AP-CNN beyond ~90 tokens, unlike QA-CNN.
• Across datasets, AP-CNN consistently improves MAP/Precision metrics over baselines, often surpassing recent state-of-the-art methods.