[Paper Review] Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processing

Rui Yu, Yifeng Li+2 more|arXiv (Cornell University)|Nov 5, 2022

Topic Modeling4 citations

TL;DR

This paper proposes Tri-Attention, a novel triple-attention mechanism that explicitly models interactions among query, key, and context to improve context-aware representation in NLP. By extending standard Bi-Attention to a three-dimensional tensor operation, Tri-Attention outperforms state-of-the-art models across multiple NLP tasks, including sentence matching, dialogue, and reading comprehension, with consistent gains over both non-attention and attention-based baselines.

ABSTRACT

In natural language processing (NLP), the context of a word or sentence plays an essential role. Contextual information such as the semantic representation of a passage or historical dialogue forms an essential part of a conversation and a precise understanding of the present phrase or sentence. However, the standard attention mechanisms typically generate weights using query and key but ignore context, forming a Bi-Attention framework, despite their great success in modeling sequence alignment. This Bi-Attention mechanism does not explicitly model the interactions between the contexts, queries and keys of target sequences, missing important contextual information and resulting in poor attention performance. Accordingly, a novel and general triple-attention (Tri-Attention) framework expands the standard Bi-Attention mechanism and explicitly interacts query, key, and context by incorporating context as the third dimension in calculating relevance scores. Four variants of Tri-Attention are generated by expanding the two-dimensional vector-based additive, dot-product, scaled dot-product, and bilinear operations in Bi-Attention to the tensor operations for Tri-Attention. Extensive experiments on three NLP tasks demonstrate that Tri-Attention outperforms about 30 state-of-the-art non-attention, standard Bi-Attention, contextual Bi-Attention approaches and pretrained neural language models1.

Motivation & Objective

• To address the limitation of standard Bi-Attention mechanisms, which ignore contextual information and fail to model interactions between queries, keys, and context.
• To develop a general-purpose attention framework that explicitly captures interactions among query, key, and context, emulating human-like contextual attention.
• To improve performance in context-sensitive NLP tasks such as sentence matching, dialogue understanding, and machine reading comprehension by incorporating context directly into attention computation.
• To provide a flexible, stackable architecture with multiple variants based on different relevance scoring methods (additive, dot-product, scaled dot-product, bilinear).

Proposed method

• Proposes a Tri-Attention mechanism that extends standard Bi-Attention by introducing context as a third dimension in attention computation, forming a three-way interaction between query, key, and context.
• Derives four variants of Tri-Attention using tensor algebra: Tri-Attention_Add, Tri-Attention_Dot, Tri-Attention_ScaledDot, and Tri-Attention_Bilinear, each based on different relevance scoring operations.
• Computes a three-dimensional attention matrix by calculating relevance scores between query, key, and context using tensor operations, enabling joint modeling of sequence and contextual interactions.
• Introduces contextual value by computing relevance between context and value, ensuring consistency in the semantic space of the final output.
• Employs a stackable, modular architecture (TAN) that allows dynamic adjustment of the number of Tri-Attention layers for different tasks.
• Applies the Tri-Attention module within deep neural networks for end-to-end training on NLP tasks, with ablation and hyperparameter analysis to validate design choices.

Experimental results

Research questions

• RQ1 Can explicitly modeling interactions between query, key, and context improve attention performance in NLP tasks compared to standard Bi-Attention?
• RQ2 How do different tensor-based relevance scoring methods (additive, dot-product, etc.) affect the performance of the Tri-Attention mechanism?
• RQ3 Does integrating context as a first-class component in attention computation lead to better generalization across diverse NLP tasks?
• RQ4 What is the optimal number of Tri-Attention layers and relevance scoring method for different NLP tasks?
• RQ5 Can Tri-Attention outperform both non-attention and state-of-the-art pretrained language models in context-sensitive NLP tasks?

Key findings

• Tri-Attention outperforms 30 state-of-the-art models, including non-attention, standard Bi-Attention, contextual Bi-Attention, and pretrained language models, across three major NLP tasks: sentence matching (LCQMC), dialogue (Ubuntu), and reading comprehension (RACE).
• The additive-based variant (Tri-Attention_Add) demonstrates superior stability and performance, particularly in identifying subtle semantic differences in sentence pairs.
• On the LCQMC dataset, Tri-Attention correctly classified 151 instances where Bi-Attention and C-BiAttention failed, with 102 of these being negative (semantically different) pairs, indicating improved discrimination on ambiguous cases.
• The number of Tri-Attention layers and the choice of relevance scoring method significantly affect performance, with optimal configurations varying by task, indicating the need for task-specific tuning.
• Case studies show that Tri-Attention effectively leverages contextual information to resolve ambiguity—e.g., distinguishing 'play' vs. 'download' movies—where word-level similarity misleads standard models.
• The framework is generalizable and effective even without BERT pretraining, suggesting its value as a plug-in attention module in diverse architectures.