[Paper Review] Multi-Cast Attention Networks for Retrieval-based Question Answering and Response Prediction

Yi Tay, Luu Anh Tuan+1 more|arXiv (Cornell University)|Jun 3, 2018

Topic ModelingComputer Science52 references22 citations

TL;DR

This paper introduces Multi-Cast Attention Networks (MCAN), a novel neural architecture that repurposes attention not as a pooling mechanism but as a feature augmentation technique by casting scalar features derived from attention matrices into subsequent encoder layers. MCAN achieves state-of-the-art performance on four benchmark datasets, including a 9% improvement on the Ubuntu Dialogue Corpus and the best result to date on TrecQA, by enabling multiple attention variants (e.g., co-attention, intra-attention) and pooling types (e.g., max, mean, alignment) to be applied simultaneously without increasing representation size.

ABSTRACT

Attention is typically used to select informative sub-phrases that are used for prediction. This paper investigates the novel use of attention as a form of feature augmentation, i.e, casted attention. We propose Multi-Cast Attention Networks (MCAN), a new attention mechanism and general model architecture for a potpourri of ranking tasks in the conversational modeling and question answering domains. Our approach performs a series of soft attention operations, each time casting a scalar feature upon the inner word embeddings. The key idea is to provide a real-valued hint (feature) to a subsequent encoder layer and is targeted at improving the representation learning process. There are several advantages to this design, e.g., it allows an arbitrary number of attention mechanisms to be casted, allowing for multiple attention types (e.g., co-attention, intra-attention) and attention variants (e.g., alignment-pooling, max-pooling, mean-pooling) to be executed simultaneously. This not only eliminates the costly need to tune the nature of the co-attention layer, but also provides greater extents of explainability to practitioners. Via extensive experiments on four well-known benchmark datasets, we show that MCAN achieves state-of-the-art performance. On the Ubuntu Dialogue Corpus, MCAN outperforms existing state-of-the-art models by $9\%$. MCAN also achieves the best performing score to date on the well-studied TrecQA dataset.

Motivation & Objective

• To address the limitations of traditional attention mechanisms that are used solely for feature pooling or representation compression in neural ranking models.
• To eliminate the need for architectural engineering by enabling arbitrary, simultaneous application of multiple attention variants and pooling types without increasing representation dimensionality.
• To improve model explainability and performance in retrieval-based question answering and response prediction by providing real-valued hints through casted attention features.
• To develop a unified, general-purpose neural ranking framework applicable across diverse conversational modeling and question answering tasks.

Proposed method

• MCAN introduces a multi-cast attention mechanism that performs multiple soft attention operations, each casting a scalar feature derived from an attention matrix into the next encoder layer.
• Each attention operation is based on a different attention type (e.g., co-attention, intra-attention) and pooling variant (e.g., max-pooling, mean-pooling, alignment-pooling), enabling diverse views of the input sequence pair.
• The method compresses attention matrices into scalar features using techniques such as global pooling or attention-weighted aggregation, which are then injected as hints into subsequent layers.
• The architecture supports an arbitrary number of attention casts, allowing for flexible and efficient integration of multiple attention mechanisms without concatenating representations or increasing dimensionality.
• The model uses a standard encoder (e.g., BiLSTM or Transformer) to process the input, with attention features injected at multiple layers to guide representation learning.
• The final representation is used for ranking or classification, with a learned scoring function applied to document-query or response-pair pairs.

Experimental results

Research questions

• RQ1 Can attention be effectively repurposed not as a pooling mechanism but as a form of feature augmentation to improve representation learning in neural ranking models?
• RQ2 Does enabling multiple simultaneous attention variants (e.g., co-attention, intra-attention) and pooling types (e.g., max, mean, alignment) lead to improved performance without increasing model complexity?
• RQ3 Can the proposed multi-cast attention mechanism achieve state-of-the-art results across diverse retrieval-based tasks such as question answering and dialogue response selection?
• RQ4 How do the casted attention features contribute to model interpretability and explainability in sequence matching tasks?

Key findings

• MCAN achieves a 9% relative improvement over the best existing model on the Ubuntu Dialogue Corpus, setting a new state-of-the-art performance.
• MCAN achieves the best reported result to date on the well-studied TrecQA dataset, outperforming all prior methods.
• The model achieves strong performance on the Community Question Answering (CQA) and Tweet Reply Prediction benchmarks, demonstrating broad applicability across conversational modeling tasks.
• Visualizations of casted attention features show that they are interpretable and correspond to semantically meaningful sub-phrases, enhancing model explainability.
• The multi-cast mechanism allows for the simultaneous use of diverse attention types and pooling strategies without increasing representation size, reducing architectural engineering costs.