[Paper Review] Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks

Yinan Zhang, Raphael Tang+1 more|arXiv (Cornell University)|Nov 7, 2019

Topic Modeling20 references4 citations

TL;DR

This paper proposes integrating an explicit pairwise word interaction (PWI) module—originally from pre-transformer models—into BERT to improve performance on English semantic similarity tasks. By adding a constrained, convolutional PWI layer after BERT’s self-attention, the model achieves consistent, statistically significant gains across four benchmark datasets, demonstrating that explicit interaction modeling enhances pretrained transformers even when self-attention already captures global interactions implicitly.

ABSTRACT

In English semantic similarity tasks, classic word embedding-based approaches explicitly model pairwise "interactions" between the word representations of a sentence pair. Transformer-based pretrained language models disregard this notion, instead modeling pairwise word interactions globally and implicitly through their self-attention mechanism. In this paper, we hypothesize that introducing an explicit, constrained pairwise word interaction mechanism to pretrained language models improves their effectiveness on semantic similarity tasks. We validate our hypothesis using BERT on four tasks in semantic textual similarity and answer sentence selection. We demonstrate consistent improvements in quality by adding an explicit pairwise word interaction module to BERT.

Motivation & Objective

• To investigate whether explicit pairwise word interaction mechanisms—historically effective in pre-transformer models—improve performance in modern pretrained transformers like BERT.
• To address the gap in current transformer-based models, which implicitly model word interactions via self-attention, by introducing a constrained, explicit interaction module.
• To evaluate whether combining BERT with a structured PWI module leads to improved performance on semantic textual similarity and answer sentence selection tasks.

Proposed method

• Integrate a deep pairwise word interaction (VDPWI) module after BERT’s final layer, which computes a pairwise distance matrix between word representations of two input sentences.
• Apply a convolutional neural network (CNN) to the pairwise interaction matrix to extract local patterns, treating semantic similarity as a pattern recognition task.
• Use a joint encoding scheme where both sentences are concatenated with [CLS] and [SEP] tokens and processed together through BERT, followed by the PWI module.
• Train the combined model end-to-end using standard fine-tuning procedures, with loss functions tailored to each dataset: Kullback–Leibler divergence for regression tasks (STS-B, SICK), and negative log-likelihood for classification tasks (WikiQA, TrecQA).
• Ablate the use of a BiLSTM in the PWI module to assess its necessity, comparing models with and without it.
• Employ RMSProp for training the PWI module and Adam for BERT fine-tuning, with hyperparameters tuned via grid and random search on development sets.

Experimental results

Research questions

• RQ1 Does adding an explicit, constrained pairwise word interaction module improve BERT’s performance on semantic similarity tasks?
• RQ2 Is the joint encoding scheme—where both sentences are processed together—more effective than separate encoding for models with explicit PWI?
• RQ3 Does the inclusion of a BiLSTM in the PWI module provide a significant performance gain over a simpler CNN-only architecture?
• RQ4 Are the improvements from explicit PWI statistically significant across multiple semantic similarity benchmarks?
• RQ5 Can explicit interaction modeling complement the implicit global attention in BERT, leading to consistent gains without overfitting or degenerate solutions?

Key findings

• Adding an explicit pairwise word interaction module to BERT leads to consistent improvements across all four semantic similarity tasks: STS-B, SICK, WikiQA, and TrecQA.
• The average improvement across all datasets is 0.9 points in performance metrics, with gains statistically significant via one-sided Wilcoxon signed-rank tests (p < 0.05).
• Joint encoding of sentence pairs consistently outperforms separate encoding, with performance gaps up to 14 points on TrecQA; separate encoding leads to degenerate solutions in some cases.
• The inclusion of a BiLSTM in the PWI module does not yield a statistically significant improvement over a CNN-only architecture, suggesting it is not necessary for performance gains.
• The best-performing configurations (e.g., BERT with VDPWI and joint encoding) outperform both the original BERT and the standalone VDPWI model on all datasets, indicating synergistic benefits from combining explicit interaction with pretrained representations.